unwind, arm64: add sframe unwinder for kernel
Subsystems (9)
v9 cover unwind, arm64: add sframe unwinder for kernel
Implement a generic kernel sframe-based [1] unwinder. The main goal is to improve reliable stacktrace on arm64 by unwinding across exception boundaries.
On x86, the ORC unwinder provides reliable stacktrace through similar methodology, but arm64 lacks the necessary support from objtool to create ORC unwind tables.
Currently, there is a sframe unwinder proposed for userspace [2], along with an RFC for an .eh_frame unwinder [5]. This patch series implements a similar SFrame lookup library to [2], but with many simplifications to account for the fact that user access is not necessary. The intent here is to continue to leave the door open for common usage of definitions and algorithms between kernel and user unwind in the future, while at the same time keeping these patches strictly constrained the minimal set of features necessary for arm64 kernel unwind.
Unlike past versions of this series, these patches apply directly to v7.4-rc1 rather than depending upon [2]. To make this possible, I've adopted a number of patches from [2] with significant squashes and changes made in order to coherently suit the kernel unwind use case. I've done my best to maintain appropriate authorship on these changes, and have added 'Co-developed-by' trailers to attribute shared authorship as accurately as possible. Please let me know if you see any discrepencies in my work here.
Currently, only GNU Binutils support sframe. This series relies on the Sframe V3 format, which is supported in binutils 2.46.
This full series is available on github: [4].
Ref:
[1]: https://sourceware.org/binutils/docs/sframe-spec.html
[2]: https://lore.kernel.org/all/[email protected]/
[3]: https://git.kernel.org/pub/scm/linux/kernel/git/rostedt/linux-trace.git/log/?h=sframe/core
[4]: https://github.com/dylanbhatch/linux/tree/sframe-v9
[5]: https://lore.kernel.org/all/[email protected]/
---
Summary of changes since v8:
- Rebase onto v7.4-rc1, resolving merge conflicts in Makefile and arch/arm64/lib/strncmp.S and dropping arch/arm64/lib/clear_page.S (removed upstream).
- (sashiko) Add 'depends on !RUST' to HAVE_UNWIND_KERNEL_SFRAME and keep KBUILD_RUSTFLAGS conditioned only on CONFIG_UNWIND_TABLES.
- (sashiko) Move STACK UNWINDING entry in MAINTAINERS to alphabetical order and add unwind_sframe.h patterns in the patch introducing them.
- (sashiko) Filter out $(CC_FLAGS_SFRAME) in efi/libstub, arm64 pi, and arm64 kvm/hyp/nvhe Makefiles where unwind tables are disabled.
- (sashiko) Guard SFrame header, FDE, and FRE bounds calculations against integer overflow.
- (sashiko) Preserve KASAN, KCSAN, and ftrace exclusions on sframe.o in kernel/unwind/Makefile, and clean up header includes across sframe.h, sframe_debug.h, module.h, and kernel/module/main.c.
- (sashiko) Mark is_kernel_inittext(), __is_kernel_text(), and within_module_mem_type() as __always_inline for noinstr sframe_find().
- (sashiko) Provide generic include/asm-generic/unwind_sframe.h fallback and move <asm/unwind_sframe.h> include to the IP validation patch.
- (sashiko) Update CONFIG_SFRAME_VALIDATION help text and commit log for in-kernel vmlinux and module .sframe validation.
- (sashiko) Place .cfi_startproc before 'bti c' in call_on_irq_stack() using SYM_START().
- (sashiko) Move out-of-line blocks in pmull_gcm_ghash_4x() and pmull_gcm_enc_4x() out of '.subsection 1' and annotate __pi_strrchr with SYM_LEAF_FUNC_START / SYM_LEAF_FUNC_END.
- (sashiko) Restrict SFrame unwinding at KUNWIND_SOURCE_REGS_PC to real on-stack FRAME_META_TYPE_PT_REGS exception frames, drop invalid cross-stack 'fp < sp' check in UNWIND_CFA_RULE_FP_OFFSET, enforce 8-byte alignment in get_word(), and propagate state->unreliable on -ENOENT in do_kunwind().
Summary of changes since v7:
- (sashiko): Put .cfi_startproc right after SYM_START to account for potential alignment bytes ahead of the actual function symbol.
- (sashiko) Use __always_inline for IP validation helper.
- (sashiko) Move .sframe module init and sframe section fields to arch generic code.
- (sashiko) Make sframe_find() noinstr.
- (sashiko) Add preempt_notrace and instrumentation to __sframe_module_find() code path.
- (sashiko) Block tracing and kprobe on the sframe_find code path.
- (shashiko) remove KCOV, KCSAN, and KASAN instrumentation from sframe.o (copying over ORC's shoulder).
Dylan Hatch (6):
arm64, unwind: build kernel with sframe V3 info module, sframe: Add sframe support for modules arm64/sframe: Validate IP addresses
sframe: Add debug helpers with object name arm64, crypto/lib: Annotate leaf functions with CFI info
unwind: arm64: Use sframe to unwind interrupt frames
Jens Remus (1):
sframe: Separate reading of FRE from reading of FRE data words
Josh Poimboeuf (3):
arm64/sframe: Read vmlinux .sframe header
sframe: Add support for reading vmlinux .sframe contents
sframe: Add .sframe validation option
Weinan Liu (1):
arm64: entry: add unwind info for call_on_irq_stack()
MAINTAINERS | 17 +- Makefile | 8 + arch/Kconfig | 51 ++ arch/arm64/Kconfig | 1 + arch/arm64/Makefile | 8 +- arch/arm64/crypto/aes-ce-ccm-core.S | 12 +- arch/arm64/crypto/aes-neonbs-core.S | 16 +- arch/arm64/crypto/ghash-ce-core.S | 76 ++- arch/arm64/crypto/sm4-ce-ccm-core.S | 16 +- arch/arm64/crypto/sm4-ce-cipher-core.S | 4 +- arch/arm64/crypto/sm4-ce-core.S | 44 +- arch/arm64/crypto/sm4-ce-gcm-core.S | 16 +- arch/arm64/crypto/sm4-neon-core.S | 12 +- arch/arm64/include/asm/linkage.h | 32 ++ arch/arm64/include/asm/module.lds.h | 2 + arch/arm64/include/asm/sections.h | 1 + arch/arm64/include/asm/unwind_sframe.h | 32 ++ arch/arm64/kernel/entry.S | 17 +- arch/arm64/kernel/pi/Makefile | 3 + arch/arm64/kernel/setup.c | 2 + arch/arm64/kernel/stacktrace.c | 211 +++++++- arch/arm64/kernel/vdso/Makefile | 2 +- arch/arm64/kernel/vmlinux.lds.S | 14 + arch/arm64/kvm/hyp/nvhe/Makefile | 5 +- arch/arm64/lib/clear_user.S | 4 +- arch/arm64/lib/copy_from_user.S | 4 +- arch/arm64/lib/copy_page.S | 4 +- arch/arm64/lib/copy_to_user.S | 4 +- arch/arm64/lib/memchr.S | 4 +- arch/arm64/lib/memcmp.S | 4 +- arch/arm64/lib/memcpy.S | 8 +- arch/arm64/lib/memset.S | 8 +- arch/arm64/lib/mte.S | 28 +- arch/arm64/lib/strchr.S | 4 +- arch/arm64/lib/strcmp.S | 4 +- arch/arm64/lib/strlen.S | 4 +- arch/arm64/lib/strncmp.S | 4 +- arch/arm64/lib/strnlen.S | 4 +- arch/arm64/lib/strrchr.S | 4 +- arch/arm64/lib/tishift.S | 12 +- drivers/firmware/efi/libstub/Makefile | 2 + include/asm-generic/Kbuild | 1 + include/asm-generic/sections.h | 8 +- include/asm-generic/unwind_sframe.h | 5 + include/asm-generic/vmlinux.lds.h | 15 + include/linux/module.h | 11 +- include/linux/sframe.h | 42 ++ include/linux/unwind_types.h | 34 ++ kernel/module/main.c | 20 + kernel/unwind/Makefile | 9 +- kernel/unwind/sframe.c | 714 +++++++++++++++++++++++++ kernel/unwind/sframe.h | 79 +++ kernel/unwind/sframe_debug.h | 38 ++ 53 files changed, 1501 insertions(+), 183 deletions(-) create mode 100644 arch/arm64/include/asm/unwind_sframe.h create mode 100644 include/asm-generic/unwind_sframe.h create mode 100644 include/linux/sframe.h create mode 100644 include/linux/unwind_types.h create mode 100644 kernel/unwind/sframe.c create mode 100644 kernel/unwind/sframe.h create mode 100644 kernel/unwind/sframe_debug.h
Signature · 1 line
2.56.0.679.gc527c65306-goog
PATCH v9 1/11 arm64, unwind: build kernel with sframe V3 info
To generate a .sframe section, build with -Wa,--gsframe-3 flags. Also, add unwind-table options to ensure the necessary CFI directives are emitted by the compiler, but discard .eh_frame unless CONFIG_UNWIND_TABLES is explicitly enabled.
This will be used for in-kernel reliable stacktrace in cases where the frame pointer alone is insufficient.
Currently, the sframe format only supports arm64, x86_64 and s390x architectures.
---
Changes since v8:
- (conflict resolution) Resolve v7.4-rc1 merge conflict in Makefile around CFLAGS_GCOV moving to scripts/Makefile.gcov.
- (sashiko) Add 'depends on !RUST' to HAVE_UNWIND_KERNEL_SFRAME and keep KBUILD_RUSTFLAGS conditioned only on CONFIG_UNWIND_TABLES.
- (sashiko) Move STACK UNWINDING entry in MAINTAINERS to alphabetical order and defer unwind_sframe.h pattern to the patch introducing it.
- (sashiko) Filter out $(CC_FLAGS_SFRAME) in efi/libstub, arm64 pi, and arm64 kvm/hyp/nvhe Makefiles where unwind tables are disabled.
--- MAINTAINERS | 14 ++++++------ Makefile | 8 +++++++ arch/Kconfig | 32 +++++++++++++++++++++++++++ arch/arm64/Kconfig | 1 + arch/arm64/Makefile | 8 +++++-- arch/arm64/include/asm/module.lds.h | 2 ++ arch/arm64/kernel/pi/Makefile | 3 +++ arch/arm64/kernel/vdso/Makefile | 2 +- arch/arm64/kernel/vmlinux.lds.S | 12 ++++++++++ arch/arm64/kvm/hyp/nvhe/Makefile | 5 +++-- drivers/firmware/efi/libstub/Makefile | 2 ++ include/asm-generic/sections.h | 4 ++++ include/asm-generic/vmlinux.lds.h | 15 +++++++++++++ 13 files changed, 96 insertions(+), 12 deletions(-)
MAINTAINERS +7 -7
@@ -26639,6 +26639,13 @@ L: [email protected] S: Supported F: Documentation/process/stable-kernel-rules.rst +STACK UNWINDING +M: Josh Poimboeuf <[email protected]> +M: Steven Rostedt <[email protected]> +S: Maintained +F: include/linux/unwind*.h +F: kernel/unwind/ + STAGING - ATOMISP DRIVER M: Hans de Goede <[email protected]> M: Mauro Carvalho Chehab <[email protected]>
@@ -29242,13 +29249,6 @@ F: Documentation/driver-api/uio-howto.rst F: drivers/uio/ F: include/linux/uio_driver.h -USERSPACE STACK UNWINDING -M: Josh Poimboeuf <[email protected]> -M: Steven Rostedt <[email protected]> -S: Maintained -F: include/linux/unwind*.h -F: kernel/unwind/ - UTIL-LINUX PACKAGE M: Karel Zak <[email protected]> L: [email protected]
Makefile +8 -0
@@ -1184,6 +1184,14 @@ endif # Ensure compilers do not transform certain loops into calls to wcslen() KBUILD_CFLAGS += -fno-builtin-wcslen +# build with sframe table +ifdef CONFIG_HAVE_UNWIND_KERNEL_SFRAME +CC_FLAGS_SFRAME := -Wa,--gsframe-3 +KBUILD_CFLAGS += $(CC_FLAGS_SFRAME) +KBUILD_AFLAGS += $(CC_FLAGS_SFRAME) +export CC_FLAGS_SFRAME +endif + # change __FILE__ to the relative path to the source directory ifdef building_out_of_srctree CFLAGS_PREFIX_MAP := -fmacro-prefix-map=$(srcroot)/=
arch/Kconfig +32 -0
@@ -507,6 +507,38 @@ config HAVE_UNWIND_USER_FP bool select UNWIND_USER +config AS_SFRAME + bool + +config AS_SFRAME3 + def_bool $(as-instr,.cfi_startproc\n.cfi_endproc,-Wa$(comma)--gsframe-3) + select AS_SFRAME + +config UNWIND_SFRAME_LOOKUP + bool + +config ARCH_SUPPORTS_UNWIND_KERNEL_SFRAME + bool + help + An architecture can select this if it enables the SFrame (Simple + Frame) unwinder for unwinding kernel stack traces. It uses an unwind + table that is directly generated by the toolchain based on DWARF CFI + information. + +config HAVE_UNWIND_KERNEL_SFRAME + bool "Sframe unwinder" + depends on AS_SFRAME3 + depends on 64BIT + depends on ARCH_SUPPORTS_UNWIND_KERNEL_SFRAME + depends on !RUST + select UNWIND_SFRAME_LOOKUP + help + This option enables the SFrame (Simple Frame) unwinder for unwinding + kernel stack traces. It uses an unwind table that is directly + generated by the toolchain based on DWARF CFI information. In + practice, this can provide more reliable stacktrace results than + unwinding with frame pointers alone. + config HAVE_PERF_REGS bool help
arch/arm64/Kconfig +1 -0
@@ -87,6 +87,7 @@ config ARM64 select ARCH_SUPPORTS_SCHED_SMT select ARCH_SUPPORTS_SCHED_CLUSTER select ARCH_SUPPORTS_SCHED_MC + select ARCH_SUPPORTS_UNWIND_KERNEL_SFRAME select ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH select ARCH_WANT_COMPAT_IPC_PARSE_VERSION if COMPAT select ARCH_WANT_DEFAULT_BPF_JIT
arch/arm64/Makefile +6 -2
@@ -58,13 +58,17 @@ KBUILD_CFLAGS += $(call cc-option,-mabi=lp64) KBUILD_AFLAGS += $(call cc-option,-mabi=lp64) # Avoid generating .eh_frame* sections. -ifneq ($(CONFIG_UNWIND_TABLES),y) +ifeq ($(filter y, $(CONFIG_UNWIND_TABLES) $(CONFIG_HAVE_UNWIND_KERNEL_SFRAME)),) KBUILD_CFLAGS += -fno-asynchronous-unwind-tables -fno-unwind-tables KBUILD_AFLAGS += -fno-asynchronous-unwind-tables -fno-unwind-tables -KBUILD_RUSTFLAGS += -Cforce-unwind-tables=n else KBUILD_CFLAGS += -fasynchronous-unwind-tables KBUILD_AFLAGS += -fasynchronous-unwind-tables +endif + +ifneq ($(CONFIG_UNWIND_TABLES),y) +KBUILD_RUSTFLAGS += -Cforce-unwind-tables=n +else KBUILD_RUSTFLAGS += -Cforce-unwind-tables=y -Zuse-sync-unwind=n # Work around rustc bug on compilers without # https://github.com/rust-lang/rust/pull/156973.
arch/arm64/include/asm/module.lds.h +2 -0
@@ -10,5 +10,7 @@ SECTIONS {* put it into the .init allocation. */ .init.eh_frame 0 : { *(.eh_frame) } +#elif defined(CONFIG_HAVE_UNWIND_KERNEL_SFRAME ) + /DISCARD/ : { *(.eh_frame) }#endif }
arch/arm64/kernel/pi/Makefile +3 -0
@@ -18,6 +18,9 @@ CFLAGS_map_range.o += -mstrict-align KBUILD_CFLAGS := $(filter-out $(CC_FLAGS_SCS), $(KBUILD_CFLAGS)) # disable LTO KBUILD_CFLAGS := $(filter-out $(CC_FLAGS_LTO), $(KBUILD_CFLAGS)) +# disable SFrame +KBUILD_CFLAGS := $(filter-out $(CC_FLAGS_SFRAME), $(KBUILD_CFLAGS)) +KBUILD_AFLAGS := $(filter-out $(CC_FLAGS_SFRAME), $(KBUILD_AFLAGS)) hostprogs := relacheck
arch/arm64/kernel/vdso/Makefile +1 -1
@@ -38,7 +38,7 @@ ccflags-y += -DDISABLE_BRANCH_PROFILING -DBUILD_VDSO CC_FLAGS_REMOVE_VDSO := $(CC_FLAGS_FTRACE) -Os $(CC_FLAGS_SCS) \ $(RANDSTRUCT_CFLAGS) $(KSTACK_ERASE_CFLAGS) \ $(GCC_PLUGINS_CFLAGS) \ - $(CC_FLAGS_LTO) $(CC_FLAGS_CFI) \ + $(CC_FLAGS_LTO) $(CC_FLAGS_CFI) $(CC_FLAGS_SFRAME) \ -Wmissing-prototypes -Wmissing-declarations CC_FLAGS_ADD_VDSO := -O2 -mcmodel=tiny -fasynchronous-unwind-tables $(CONFIG_CC_OPT_INLINE_MEMSET)
arch/arm64/kernel/vmlinux.lds.S +12 -0
@@ -145,14 +145,25 @@ jiffies = jiffies_64; #endif #ifdef CONFIG_UNWIND_TABLES + #define UNWIND_DATA_SECTIONS \ .eh_frame : { \ __pi___eh_frame_start = .; \ *(.eh_frame) \ __pi___eh_frame_end = .; \ } +#define SFRAME_DISCARDS + +#elif defined(CONFIG_HAVE_UNWIND_KERNEL_SFRAME) + +#define UNWIND_DATA_SECTIONS +#define SFRAME_DISCARDS *(.eh_frame) + #else + #define UNWIND_DATA_SECTIONS +#define SFRAME_DISCARDS + #endif /*
@@ -184,6 +195,7 @@ SECTIONS *(.interp .dynamic) *(.dynsym .dynstr .hash .gnu.hash) *(.ARM.attributes) + SFRAME_DISCARDS } . = KIMAGE_VADDR;
arch/arm64/kvm/hyp/nvhe/Makefile +3 -2
@@ -97,9 +97,10 @@ quiet_cmd_hyprel = HYPREL $@ quiet_cmd_hypcopy = HYPCOPY $@ cmd_hypcopy = $(OBJCOPY) --prefix-symbols=__kvm_nvhe_ $< $@ -# Remove ftrace and Shadow Call Stack CFLAGS. +# Remove ftrace, Shadow Call Stack, and SFrame flags. # This is equivalent to the 'notrace' and '__noscs' annotations. -KBUILD_CFLAGS := $(filter-out $(CC_FLAGS_FTRACE) $(CC_FLAGS_SCS), $(KBUILD_CFLAGS)) +KBUILD_CFLAGS := $(filter-out $(CC_FLAGS_FTRACE) $(CC_FLAGS_SCS) $(CC_FLAGS_SFRAME), $(KBUILD_CFLAGS)) +KBUILD_AFLAGS := $(filter-out $(CC_FLAGS_SFRAME), $(KBUILD_AFLAGS)) # Starting from 13.0.0 llvm emits SHT_REL section '.llvm.call-graph-profile' # when profile optimization is applied. gen-hyprel does not support SHT_REL and # causes a build failure. Remove profile optimization flags.
drivers/firmware/efi/libstub/Makefile +2 -0
@@ -57,6 +57,8 @@ KBUILD_CFLAGS := $(filter-out $(CC_FLAGS_SCS), $(KBUILD_CFLAGS)) KBUILD_CFLAGS := $(filter-out $(CC_FLAGS_CFI), $(KBUILD_CFLAGS)) # disable LTO KBUILD_CFLAGS := $(filter-out $(CC_FLAGS_LTO), $(KBUILD_CFLAGS)) +# disable SFrame +KBUILD_CFLAGS := $(filter-out $(CC_FLAGS_SFRAME), $(KBUILD_CFLAGS)) # The .data section would be renamed to .data.efistub, therefore, remove # `-fdata-sections` flag from KBUILD_CFLAGS_KERNEL
include/asm-generic/sections.h +4 -0
@@ -31,6 +31,7 @@* __irqentry_text_start, __irqentry_text_end * __softirqentry_text_start, __softirqentry_text_end * __start_opd, __end_opd +* __start_sframe, __end_sframe */ extern char _text[], _stext[], _etext[];extern char _data[], _sdata[], _edata[];
@@ -53,6 +54,9 @@ extern char __ctors_start[], __ctors_end[];/* Start and end of .opd section - used for function descriptors. */ extern char __start_opd[], __end_opd[]; +/* Start and end of .sframe section - used for stack unwinding. */ +extern char __start_sframe[], __end_sframe[]; +/* Start and end of instrumentation protected text section */ extern char __noinstr_text_start[], __noinstr_text_end[];
include/asm-generic/vmlinux.lds.h +15 -0
@@ -491,6 +491,8 @@ *(.rodata1) \ } \ \ + SFRAME \ + \ /* PCI quirks */ \ .pci_fixup : AT(ADDR(.pci_fixup) - LOAD_OFFSET) { \ BOUNDED_SECTION_PRE_LABEL(.pci_fixup_early, _pci_fixups_early, __start, __end) \
@@ -943,6 +945,19 @@#define TRACEDATA #endif +#ifdef CONFIG_HAVE_UNWIND_KERNEL_SFRAME +#define SFRAME \ +/* sframe */ \ + .sframe : AT(ADDR(.sframe) - LOAD_OFFSET) { \ + __start_sframe = .; \ + KEEP(*(.sframe)) \ + KEEP(*(.init.sframe)) \ + __end_sframe = .; \ + } +#else +#define SFRAME +#endif +#ifdef CONFIG_PRINTK_INDEX #define PRINTK_INDEX \ .printk_index : AT(ADDR(.printk_index) - LOAD_OFFSET) { \
Signature · 1 line
2.56.0.679.gc527c65306-goog
PATCH v9 2/11 arm64/sframe: Read vmlinux .sframe header
In reply to Dylan HatchFrom: Josh Poimboeuf <[email protected]>
In preparation for unwinding kernel space stacks with sframe, add basic sframe compile infrastructure and support for reading the .sframe section header. Add init_sframe_table() to read the vmlinux .sframe header at boot.
[ Jens Remus: Add support for SFrame V3. Add support for PC-relative FDE function start offset. Cleanup includes and indentation. ]
[ Dylan Hatch: Repurpose patch for arm64 vmlinux. ]
---
Changes since v8:
- (sashiko) Use check_add_overflow() and check_mul_overflow() in sframe_read_header() to prevent integer overflow on section bounds.
--- MAINTAINERS | 1 + arch/arm64/kernel/setup.c | 2 + include/linux/sframe.h | 32 ++++++++++++++++ kernel/unwind/Makefile | 9 ++++- kernel/unwind/sframe.c | 80 +++++++++++++++++++++++++++++++++++++++ kernel/unwind/sframe.h | 78 ++++++++++++++++++++++++++++++++++++++ 6 files changed, 201 insertions(+), 1 deletion(-) create mode 100644 include/linux/sframe.h create mode 100644 kernel/unwind/sframe.c create mode 100644 kernel/unwind/sframe.h
MAINTAINERS +1 -0
@@ -26643,6 +26643,7 @@ STACK UNWINDING M: Josh Poimboeuf <[email protected]> M: Steven Rostedt <[email protected]> S: Maintained +F: include/linux/sframe.h F: include/linux/unwind*.h F: kernel/unwind/
arch/arm64/kernel/setup.c +2 -0
@@ -33,6 +33,7 @@#include <linux/sched/task.h> #include <linux/scs.h> #include <linux/mm.h> +#include <linux/sframe.h> #include <asm/acpi.h> #include <asm/fixmap.h>
@@ -373,6 +374,7 @@ void __init __no_sanitize_address setup_arch(char **cmdline_p)"This indicates a broken bootloader or old kernel\n" , boot_args[1], boot_args[2], boot_args[3]); } +init_sframe_table (); }static inline bool cpu_can_disable (unsigned int cpu)
include/linux/sframe.h new file +32 -0
@@ -0,0 +1,32 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _LINUX_SFRAME_H +#define _LINUX_SFRAME_H + +#include <linux/init.h> + +#ifdef CONFIG_UNWIND_SFRAME_LOOKUP + +struct sframe_section { +unsigned long sframe_start; +unsigned long sframe_end; + +unsigned long fdes_start; +unsigned long fres_start; +unsigned long fres_end; +unsigned int num_fdes; + +signed char ra_off; +signed char fp_off; +}; + +extern struct sframe_section kernel_sfsec__ro_after_init ; + +void __init init_sframe_table (void ); + +#else /* !CONFIG_UNWIND_SFRAME_LOOKUP */ + +static inline void __init init_sframe_table (void ) {} + +#endif /* CONFIG_UNWIND_SFRAME_LOOKUP */ + +#endif /* _LINUX_SFRAME_H */
kernel/unwind/Makefile +8 -1
@@ -1 +1,8 @@ - obj-$(CONFIG_UNWIND_USER) += user.o deferred.o + obj-$(CONFIG_UNWIND_USER) += user.o deferred.o + + KCOV_INSTRUMENT_sframe.o := n + KASAN_SANITIZE_sframe.o := n + KCSAN_SANITIZE_sframe.o := n + + CFLAGS_REMOVE_sframe.o = $(CC_FLAGS_FTRACE) + obj-$(CONFIG_UNWIND_SFRAME_LOOKUP) += sframe.o
kernel/unwind/sframe.c new file +80 -0
80 lines not on this page · raw message
kernel/unwind/sframe.h new file +78 -0
78 lines not on this page · raw message
Signature · 1 line
2.56.0.679.gc527c65306-goog
PATCH v9 3/11 sframe: Add support for reading vmlinux .sframe contents
In reply to Dylan HatchFrom: Josh Poimboeuf <[email protected]>
In preparation for using sframe to unwind kernel space stacks, add an sframe_find() interface for finding the sframe information associated with a given vmlinux text address.
Treat a RA offset from CFA of zero as indication that the RA is not saved (on the stack). For the topmost frame treat it as indication that the RA is in the link/RA register, such as on arm64 and s390, and obtain it from there. For non-topmost frames treat it as error, as the RA must be saved.
[ Jens Remus: Add initial support for SFrame V3 (limited to regular FDEs). Add support for PC-relative FDE function start offset. Simplify logic by using an internal FDE representation. Rename struct sframe_fre to sframe_fre_internal to align with struct sframe_fde_internal. Cleanup includes. Fix checkpatch errors "spaces required around that ':'". ]
[ Dylan Hatch: Adapt for in-kernel unwinding. Squash parts of patches by Jens Remus <[email protected]>:
- "unwind_user: Enable archs that pass RA in a register"
- "unwind_user: Flexible FP/RA recovery rules"
- "unwind_user: Flexible CFA recovery rules"
- "unwind_user/sframe: Add support for outermost frame indication" ]
---
Changes since v8:
- (sashiko) Use overflow-safe arithmetic and bounds checks in __read_fde(), __find_fde(), and __read_fre().
--- include/linux/sframe.h | 4 +- include/linux/unwind_types.h | 34 ++++ kernel/unwind/sframe.c | 355 +++++++++++++++++++++++++++++++++++ kernel/unwind/sframe.h | 1 + 4 files changed, 393 insertions(+), 1 deletion(-) create mode 100644 include/linux/unwind_types.h
include/linux/sframe.h +3 -1
@@ -3,6 +3,7 @@#define _LINUX_SFRAME_H#include <linux/init.h> +#include <linux/unwind_types.h> #ifdef CONFIG_UNWIND_SFRAME_LOOKUP
@@ -22,11 +23,12 @@ struct sframe_section {extern struct sframe_section kernel_sfsec__ro_after_init ;void __init init_sframe_table (void ); +extern int sframe_find (unsigned long ip,struct unwind_frame *frame);#else /* !CONFIG_UNWIND_SFRAME_LOOKUP */ static inline void __init init_sframe_table (void ) {} - +static inline int sframe_find (unsigned long ip,struct unwind_frame *frame) {return -EINVAL ; }#endif /* CONFIG_UNWIND_SFRAME_LOOKUP */ #endif /* _LINUX_SFRAME_H */
include/linux/unwind_types.h new file +34 -0
@@ -0,0 +1,34 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _LINUX_UNWIND_TYPES_H +#define _LINUX_UNWIND_TYPES_H + +enum unwind_cfa_rule { +UNWIND_CFA_RULE_SP_OFFSET ,/* CFA = SP + offset */ +UNWIND_CFA_RULE_FP_OFFSET ,/* CFA = FP + offset */ +/* REG and DEREF variants not implemented -- flex FDEs not supported. */ +}; + +struct unwind_cfa_rule_data { +enum unwind_cfa_rule rule; +s32 offset; +unsigned int regnum; +}; + +enum unwind_rule { +UNWIND_RULE_RETAIN ,/* entity = entity */ +UNWIND_RULE_CFA_OFFSET_DEREF ,/* entity = *(CFA + offset) */ +}; + +struct unwind_rule_data { +enum unwind_rule rule; +s32 offset; +}; + +struct unwind_frame { +struct unwind_cfa_rule_data cfa; +struct unwind_rule_data ra; +struct unwind_rule_data fp; +bool outermost; +}; + +#endif /* _LINUX_UNWIND_TYPES_H */
kernel/unwind/sframe.c +355 -0
367 lines not on this page · raw message
kernel/unwind/sframe.h +1 -0
@@ -74,5 +74,6 @@ struct sframe_fda_v3 {#define SFRAME_V3_FRE_DATAWORD_COUNT (info) (((info) >> 1) & 0xf)#define SFRAME_V3_FRE_DATAWORD_SIZE (info) (((info) >> 5) & 0x3)#define SFRAME_V3_AARCH64_FRE_MANGLED_RA_P (info) (((info) >> 7) & 0x1) +#define SFRAME_V3_FRE_RA_UNDEFINED_P (info) (SFRAME_V3_FRE_DATAWORD_COUNT(info) == 0)#endif /* _SFRAME_H */
Signature · 1 line
2.56.0.679.gc527c65306-goog
PATCH v9 4/11 sframe: Separate reading of FRE from reading of FRE data words
In reply to Dylan HatchFrom: Jens Remus <[email protected]>
__find_fre() performs linear search for a matching SFrame FRE for a given IP. For that purpose it uses __read_fre(), which reads the whole FRE. That is the variable-size FRE structure as well as the trailing variable-length array of variable-size data words. For the search logic to skip over the FRE it would be sufficient to read the variable-size FRE structure only, which includes the count and size of data words.
Add fields to struct sframe_fre_internal to store the FRE data word's address, count, and size. Change __read_fre() to read the variable- size FRE structure only and populate those new fields. Change __read_fre_datawords() to use those new fields. Change __find_fre() to use __read_fre_datawords() to read the FRE data words only after a matching FRE has been found.
[ Dylan Hatch: Adapt for in-kernel unwinding without flexible FDEs. ]
--- kernel/unwind/sframe.c | 97 +++++++++++++++++++++++++++++++----------- 1 file changed, 73 insertions(+), 24 deletions(-)
kernel/unwind/sframe.c +73 -24
@@ -38,6 +38,9 @@ struct sframe_fre_internal {s32 ra_off;s32 fp_off;u8 info; +unsigned long dw_addr; +unsigned char dw_count; +unsigned char dw_size; }; static __always_inline unsigned char fre_type_to_size(unsigned char fre_type)
@@ -183,6 +186,67 @@ static __always_inline int __find_fde(struct sframe_section *sec, s32 : GET_SIGNED_AND_INC(to, from, size), \ s64 : GET_SIGNED_AND_INC(to, from, size)) +static __always_inline int +__read_default_fre_datawords(struct sframe_section *sec, + struct sframe_fde_internal *fde, + struct sframe_fre_internal *fre) +{ +unsigned char dataword_count = fre->dw_count; +unsigned char dataword_size = fre->dw_size; +unsigned long cur = fre->dw_addr; +s32 cfa_off, ra_off, fp_off; + +GET_AND_INC (cfa_off, cur, dataword_size); + dataword_count--; + + ra_off = sec->ra_off; +if (!ra_off && dataword_count) { + dataword_count--; +GET_AND_INC (ra_off, cur, dataword_size); + } + + fp_off = sec->fp_off; +if (!fp_off && dataword_count) { + dataword_count--; +GET_AND_INC (fp_off, cur, dataword_size); + } + +if (dataword_count) +return -EFAULT ; + + fre->cfa_off = cfa_off; + fre->ra_off = ra_off; + fre->fp_off = fp_off; + +return 0 ; +} + +static __always_inline int +__read_fre_datawords (struct sframe_section *sec, +struct sframe_fde_internal *fde, +struct sframe_fre_internal *fre) +{ +unsigned char fde_type =SFRAME_V3_FDE_TYPE (fde->info2); +unsigned char dataword_count = fre->dw_count; + +if (!dataword_count) { +/* A FRE without data words indicates an outermost frame. */ + fre->cfa_off =0 ; + fre->ra_off =0 ; + fre->fp_off =0 ; + +return 0 ; + } + +switch (fde_type) { +case SFRAME_FDE_TYPE_DEFAULT : +return __read_default_fre_datawords (sec, fde, fre); +/* Flexible FDEs not supported */ +default : +return -EFAULT ; + } +} + static __always_inline int __read_fre(struct sframe_section *sec, struct sframe_fde_internal *fde, unsigned long fre_addr,
@@ -192,7 +256,6 @@ static __always_inline int __read_fre(struct sframe_section *sec,unsigned char fde_pctype =SFRAME_V3_FDE_PCTYPE (fde->info);unsigned char fre_type =SFRAME_V3_FDE_FRE_TYPE (fde->info);unsigned char dataword_count, dataword_size; -s32 cfa_off, ra_off, fp_off;unsigned long cur = fre_addr;unsigned char addr_size;u32 ip_off;
@@ -213,7 +276,7 @@ static __always_inline int __read_fre(struct sframe_section *sec,GET_AND_INC (info, cur,1 ); dataword_count =SFRAME_V3_FRE_DATAWORD_COUNT (info); dataword_size =dataword_size_enum_to_size (SFRAME_V3_FRE_DATAWORD_SIZE (info)); -if (!dataword_size || !dataword_count) +if (!dataword_size)return -EFAULT ; if (cur + (dataword_count * dataword_size) > sec->fres_end ||
@@ -225,30 +288,11 @@ static __always_inline int __read_fre(struct sframe_section *sec,return -EFAULT ; fre->size = addr_size +1 + (dataword_count * dataword_size); - -GET_AND_INC (cfa_off, cur, dataword_size); - dataword_count--; - - ra_off = sec->ra_off; -if (!ra_off && dataword_count) { - dataword_count--; -GET_AND_INC (ra_off, cur, dataword_size); - } - - fp_off = sec->fp_off; -if (!fp_off && dataword_count) { - dataword_count--; -GET_AND_INC (fp_off, cur, dataword_size); - } - -if (dataword_count) -return -EFAULT ; - fre->ip_off = ip_off; - fre->cfa_off = cfa_off; - fre->ra_off = ra_off; - fre->fp_off = fp_off; fre->info = info; + fre->dw_addr = cur; + fre->dw_count = dataword_count; + fre->dw_size = dataword_size;return 0 ; }
@@ -298,6 +342,7 @@ static __always_inline int __find_fre(struct sframe_section *sec,bool which = false;unsigned int i;u32 ip_off; +int ret; ip_off = ip - fde->func_addr;
@@ -338,6 +383,10 @@ static __always_inline int __find_fre(struct sframe_section *sec,return -EINVAL ; fre = prev_fre; + ret =__read_fre_datawords (sec, fde, fre); +if (ret) +return ret; +if (sframe_init_cfa_rule_data (&frame->cfa, fre->info, fre->cfa_off))return -EINVAL ;
Signature · 1 line
2.56.0.679.gc527c65306-goog
PATCH v9 5/11 module, sframe: Add sframe support for modules
In reply to Dylan HatchAdd sframe table to struct module and support sframe PC lookups when an .sframe section can be found on incoming modules. SFRAME_F_FDE_SORTED is not set for module .sframe, so FDEs are sorted right after the sframe header is read.
---
Changes since v8:
- (sashiko) Preserve KASAN_SANITIZE_sframe.o, KCSAN_SANITIZE_sframe.o, and CFLAGS_REMOVE_sframe.o in kernel/unwind/Makefile.
- (sashiko) Replace unused includes in include/linux/sframe.h with 'struct module;' forward declaration and drop redundant includes in arch/arm64/include/asm/module.h and kernel/module/main.c.
--- include/linux/module.h | 5 ++ include/linux/sframe.h | 8 +++ kernel/module/main.c | 20 +++++++ kernel/unwind/sframe.c | 118 +++++++++++++++++++++++++++++++++++++++-- 4 files changed, 146 insertions(+), 5 deletions(-)
include/linux/module.h +5 -0
@@ -25,6 +25,7 @@#include <linux/export.h> #include <linux/rbtree_latch.h> #include <linux/tracepoint-defs.h> +#include <linux/sframe.h> #include <linux/srcu.h> #include <linux/static_call_types.h> #include <linux/dynamic_debug.h>
@@ -581,6 +582,10 @@ struct module {#ifdef CONFIG_DYNAMIC_DEBUG_CORE struct _ddebug_info dyndbg_info;#endif +#ifdef CONFIG_HAVE_UNWIND_KERNEL_SFRAME +struct sframe_section sframe_sec; +bool sframe_init; +#endif }____cacheline_aligned __randomize_layout;#ifndef MODULE_ARCH_INIT #define MODULE_ARCH_INIT {}
include/linux/sframe.h +8 -0
@@ -5,6 +5,8 @@#include <linux/init.h> #include <linux/unwind_types.h> +struct module ; +#ifdef CONFIG_UNWIND_SFRAME_LOOKUP struct sframe_section {
@@ -15,6 +17,7 @@ struct sframe_section {unsigned long fres_start;unsigned long fres_end;unsigned int num_fdes; +bool fdes_sorted;signed char ra_off;signed char fp_off;
@@ -23,12 +26,17 @@ struct sframe_section {extern struct sframe_section kernel_sfsec__ro_after_init ;void __init init_sframe_table (void ); +void sframe_module_init (struct module *mod,void *addr,unsigned int size); +extern int sframe_find (unsigned long ip,struct unwind_frame *frame);#else /* !CONFIG_UNWIND_SFRAME_LOOKUP */ static inline void __init init_sframe_table (void ) {} +static inline void sframe_module_init (struct module *mod,void *addr,unsigned int size) {} +static inline int sframe_find (unsigned long ip,struct unwind_frame *frame) {return -EINVAL ; } +#endif /* CONFIG_UNWIND_SFRAME_LOOKUP */ #endif /* _LINUX_SFRAME_H */
kernel/module/main.c +20 -0
@@ -3024,6 +3024,23 @@ static void module_deallocate(struct module *mod, struct load_info *info)free_mod_mem (mod); } +#ifdef CONFIG_HAVE_UNWIND_KERNEL_SFRAME + +static void setup_sframe (struct module *mod,const struct load_info *info) +{ +unsigned int sframe_size; +void *sframe =section_objs (info,".sframe" ,1 , &sframe_size); + +if (sframe) +sframe_module_init (mod, sframe, sframe_size); +} + +#else + +static void setup_sframe (struct module *mod,const struct load_info *info) {} + +#endif +int __weak module_finalize (const Elf_Ehdr *hdr,const Elf_Shdr *sechdrs,struct module *me)
@@ -3043,6 +3060,9 @@ static int post_relocation(struct module *mod, const struct load_info *info)/* Setup kallsyms-specific fields. */ add_kallsyms (mod, info); +/* Initialize sframe metadata. */ +setup_sframe (mod, info); +/* Arch-specific module finalizing. */ return module_finalize (info->hdr, info->sechdrs, mod); }
kernel/unwind/sframe.c +113 -5
162 lines not on this page · raw message
Signature · 1 line
2.56.0.679.gc527c65306-goog
[email protected]
AI reviewRe: [PATCH v9 05/11] module, sframe: Add sframe support for modules
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider: - [Medium] unwind/sframe: noinstr violation calling rcu_is_watching() in sframe_find() --
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider:
- [Medium] unwind/sframe: noinstr violation calling rcu_is_watching() in sframe_find()
--
commit 777f63a65edfda582b87b92a3b49553c49d8a38c
Author: Dylan Hatch <[email protected]>
module, sframe: Add sframe support for modules
This commit adds SFrame section tracking and FDE sorting to struct module and extends sframe_find() to look up PCs in loaded modules.
Quoted 4 lines
kernel/unwind/sframe.c +0
-0
[ ... ]
Quoted 30 lines
kernel/unwind/sframe.c +25
-1@@ -411,12 +416,53 @@ static __always_inline int __sframe_find(struct sframe_section *sec,return __find_fre (sec, &fde, ip, frame); } -notrace int sframe_find(unsigned long ip, struct unwind_frame *frame) +#ifdef CONFIG_MODULES + +static __always_inline int __sframe_find_module(unsigned long ip, +struct unwind_frame *frame) +{ +struct module *mod; +int ret; + +if (!rcu_is_watching ()) +return -EINVAL ; + +instrumentation_begin (); +scoped_guard (preempt_notrace) { + mod =__module_text_address (ip); +if (mod &&smp_load_acquire (&mod->sframe_init)) + ret =__sframe_find (&mod->sframe_sec, ip, frame); +else + ret = -EINVAL ; + } +instrumentation_end (); + +return ret; +} + +#else
kernel/unwind/sframe.c static __always_inline int __sframe_find(struct sframe_section *sec, · line 453
Quoted 10 lines
kernel/unwind/sframe.c +10
-0+ +static __always_inline int __sframe_find_module(unsigned long ip, +struct unwind_frame *frame) +{ +return -EINVAL ; +} + +#endif + +noinstr int sframe_find(unsigned long ip, struct unwind_frame *frame)
[Severity: Medium]
Does calling rcu_is_watching() before instrumentation_begin() in __sframe_find_module() violate noinstr rules for sframe_find()?
In kernel/rcu/tree.c, rcu_is_watching() is marked notrace rather than noinstr, so it lives in .text and can be instrumented by sanitizers such as KASAN or KCSAN. Calling it from noinstr sframe_find() before instrumentation_begin() triggers an objtool noinstr validation warning when CONFIG_NOINSTR_VALIDATION is enabled and can run instrumented code while unwinding from a context where RCU is not watching.
Using the __always_inline rcu_is_watching_curr_cpu() helper inside the preempt_notrace guard before instrumentation_begin() would keep the check in noinstr context and avoid toggling preemption twice.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=5
PATCH v9 6/11 arm64/sframe: Validate IP addresses
In reply to Dylan HatchValidate the given IP and computed function start address against the known vmlinux and module text address ranges.
This requires arch-specific validation for vmlinux. This is because arm64 keeps .exit.text (normally discarded) and .rodata.text, both of both of which lie outside the bounds of .text and .init.text. Note that .rodata.text contains code that is never executed by the kernel mapping, but for which the toolchain nonetheless generates sframe data, and needs to be considered valid at lookup time.
---
Changes since v8:
- (sashiko) Mark is_kernel_inittext(), __is_kernel_text(), and within_module_mem_type() as __always_inline for noinstr sframe_find().
- (sashiko) Move <asm/unwind_sframe.h> include in sframe.c to this patch, add generic include/asm-generic/unwind_sframe.h fallback, and add unwind_sframe.h file patterns to MAINTAINERS.
--- MAINTAINERS | 2 ++ arch/arm64/include/asm/sections.h | 1 + arch/arm64/include/asm/unwind_sframe.h | 32 +++++++++++++++++++ arch/arm64/kernel/vmlinux.lds.S | 2 ++ include/asm-generic/Kbuild | 1 + include/asm-generic/sections.h | 4 +-- include/asm-generic/unwind_sframe.h | 5 +++ include/linux/module.h | 6 ++-- kernel/unwind/sframe.c | 44 +++++++++++++++++++++++++- 9 files changed, 91 insertions(+), 6 deletions(-) create mode 100644 arch/arm64/include/asm/unwind_sframe.h create mode 100644 include/asm-generic/unwind_sframe.h
MAINTAINERS +2 -0
@@ -26643,6 +26643,8 @@ STACK UNWINDING M: Josh Poimboeuf <[email protected]> M: Steven Rostedt <[email protected]> S: Maintained +F: arch/*/include/asm/unwind_sframe.h +F: include/asm-generic/unwind_sframe.h F: include/linux/sframe.h F: include/linux/unwind*.h F: kernel/unwind/
arch/arm64/include/asm/sections.h +1 -0
@@ -23,6 +23,7 @@ extern char __irqentry_text_start[], __irqentry_text_end[];extern char __mmuoff_data_start[], __mmuoff_data_end[];extern char __entry_tramp_text_start[], __entry_tramp_text_end[];extern char __relocate_new_kernel_start[], __relocate_new_kernel_end[]; +extern char _srodatatext[], _erodatatext[];static inline size_t entry_tramp_text_size (void ) {
arch/arm64/include/asm/unwind_sframe.h new file +32 -0
@@ -0,0 +1,32 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ASM_ARM64_UNWIND_SFRAME_H +#define _ASM_ARM64_UNWIND_SFRAME_H + +#include <linux/module.h> +#include <linux/sframe.h> +#include <asm/sections.h> + +static __always_inline bool sframe_is_kernel_ip_valid(unsigned long ip) +{ +if (__is_kernel_text (ip) ||is_kernel_inittext (ip)) +return true; + +/* .exit.text is retained in vmlinux on arm64. */ +if (ip >= (unsigned long )__exittext_begin && + ip < (unsigned long )__exittext_end) +return true; + +/* +* .rodata.text is never executed from the kernel mapping, but it still +* has sframe data. +*/ +if (ip >= (unsigned long )_srodatatext && + ip < (unsigned long )_erodatatext) +return true; + +return false; +} + +#define sframe_is_kernel_ip_valid sframe_is_kernel_ip_valid + +#endif /* _ASM_ARM64_UNWIND_SFRAME_H */
arch/arm64/kernel/vmlinux.lds.S +2 -0
@@ -245,12 +245,14 @@ SECTIONS /* code sections that are never executed via the kernel mapping */ .rodata.text : { + _srodatatext = .; TRAMP_TEXT HIBERNATE_TEXT KEXEC_TEXT IDMAP_TEXT . = ALIGN(PAGE_SIZE); } + _erodatatext = .; idmap_pg_dir = .; . += PAGE_SIZE;
include/asm-generic/Kbuild +1 -0
@@ -61,6 +61,7 @@ mandatory-y += tlbflush.h mandatory-y += topology.h mandatory-y += trace_clock.h mandatory-y += uaccess.h +mandatory-y += unwind_sframe.h mandatory-y += unwind_user.h mandatory-y += vdso/clocksource.h mandatory-y += vdso/vsyscall.h
include/asm-generic/sections.h +2 -2
@@ -197,7 +197,7 @@ static inline bool is_kernel_ro_after_init(unsigned long addr)* * Returns: true if the address is located in .init.text, false otherwise. */ -static inline bool is_kernel_inittext (unsigned long addr) +static __always_inline bool is_kernel_inittext(unsigned long addr) {return addr >= (unsigned long )_sinittext && addr < (unsigned long )_einittext;
@@ -212,7 +212,7 @@ static inline bool is_kernel_inittext(unsigned long addr)* Returns: true if the address is located in .text, false otherwise. * Note: an internal helper, only check the range of _stext to _etext. */ -static inline bool __is_kernel_text (unsigned long addr) +static __always_inline bool __is_kernel_text(unsigned long addr) {return addr >= (unsigned long )_stext && addr < (unsigned long )_etext;
include/asm-generic/unwind_sframe.h new file +5 -0
@@ -0,0 +1,5 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ASM_GENERIC_UNWIND_SFRAME_H +#define _ASM_GENERIC_UNWIND_SFRAME_H + +#endif /* _ASM_GENERIC_UNWIND_SFRAME_H */
include/linux/module.h +3 -3
@@ -628,9 +628,9 @@ bool __is_module_percpu_address(unsigned long addr, unsigned long *can_addr);bool is_module_percpu_address (unsigned long addr);bool is_module_text_address (unsigned long addr); -static inline bool within_module_mem_type (unsigned long addr, -const struct module *mod, -enum mod_mem_type type) +static __always_inline bool within_module_mem_type(unsigned long addr, +const struct module *mod, +enum mod_mem_type type) {unsigned long base, size;
kernel/unwind/sframe.c +43 -1
68 lines not on this page · raw message
Signature · 1 line
2.56.0.679.gc527c65306-goog
PATCH v9 7/11 sframe: Add debug helpers with object name
In reply to Dylan HatchInclude the object name for kernel .sframe sections, indicated with the format "(<module-name>)" or "(vmlinux)".
---
Changes since v8:
- (sashiko) Include <linux/container_of.h>, <linux/module.h>, <linux/printk.h>, and <linux/stddef.h> in sframe_debug.h.
--- kernel/unwind/sframe.c | 11 +++++++---- kernel/unwind/sframe_debug.h | 38 ++++++++++++++++++++++++++++++++++++ 2 files changed, 45 insertions(+), 4 deletions(-) create mode 100644 kernel/unwind/sframe_debug.h
kernel/unwind/sframe.c +7 -4
@@ -19,6 +19,7 @@#include <asm/unwind_sframe.h> #include "sframe.h" +#include "sframe_debug.h" static bool sframe_init__ro_after_init ;struct sframe_section kernel_sfsec__ro_after_init ;
@@ -516,7 +517,7 @@ static int sframe_read_header(struct sframe_section *sec)if (check_add_overflow (sec->sframe_start,sizeof (struct sframe_header ), &header_end) || header_end >= sec->sframe_end) { -pr_debug ("header struct doesn't fit in section\n" ); +dbg_sec (sec,"header struct doesn't fit in section\n" );return -EINVAL ; }
@@ -526,12 +527,12 @@ static int sframe_read_header(struct sframe_section *sec) shdr->preamble.version != SFRAME_VERSION_3 || !(shdr->preamble.flags & SFRAME_F_FDE_FUNC_START_PCREL) || shdr->auxhdr_len) { -pr_debug ("bad/unsupported sframe header\n" ); +dbg_sec (sec,"bad/unsupported sframe header\n" );return -EINVAL ; }if (!shdr->num_fdes || !shdr->num_fres) { -pr_debug ("no fde/fre entries\n" ); +dbg_sec (sec,"no fde/fre entries\n" );return -EINVAL ; }
@@ -542,7 +543,7 @@ static int sframe_read_header(struct sframe_section *sec) check_add_overflow(header_end, shdr->fres_off, &fres_start) || check_add_overflow(fres_start, shdr->fre_len, &fres_end) || fres_start < fdes_end || fres_end > sec->sframe_end) { -pr_debug ("inconsistent fde/fre offsets\n" ); +dbg_sec (sec,"inconsistent fde/fre offsets\n" );return -EINVAL ; }
@@ -569,6 +570,7 @@ void __init init_sframe_table(void) } sframe_init = true; +dbg_print_header (&kernel_sfsec); }#ifdef CONFIG_MODULES
@@ -629,6 +631,7 @@ void sframe_module_init(struct module *mod, void *addr, unsigned int size)/* Ensure SFrame is initialized when sframe_find() happens */ smp_store_release (&mod->sframe_init, true); + dbg_print_header(sec); }#endif
kernel/unwind/sframe_debug.h new file +38 -0
@@ -0,0 +1,38 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _SFRAME_DEBUG_H +#define _SFRAME_DEBUG_H + +#include <linux/container_of.h> +#include <linux/module.h> +#include <linux/printk.h> +#include <linux/sframe.h> +#include <linux/stddef.h> +#include "sframe.h" + +#ifdef CONFIG_MODULES + +#define objname (sec) ((sec) == &kernel_sfsec \ + ? "vmlinux" \ + : container_of((sec), struct module, sframe_sec)->name) + +#else + +#define objname (...) "vmlinux" + +#endif + +#define dbg_sec (sec, fmt, ...) \ + pr_debug("(%s): " fmt, objname(sec), ##__VA_ARGS__) + +static __always_inline void dbg_print_header(struct sframe_section *sec) +{ +unsigned long fdes_end; + + fdes_end = sec->fdes_start + (sec->num_fdes *sizeof (struct sframe_fde_v3 )); + +dbg_sec (sec,"SEC:0x%lx-0x%lx fdes:0x%lx-0x%lx fres:0x%lx-0x%lx ra_off:%d fp_off:%d\n" , + sec->sframe_start, sec->sframe_end, sec->fdes_start, fdes_end, sec->fres_start, + sec->fres_end, sec->ra_off, sec->fp_off); +} + +#endif /* _SFRAME_DEBUG_H */
Signature · 1 line
2.56.0.679.gc527c65306-goog
PATCH v9 8/11 sframe: Add .sframe validation option
In reply to Dylan HatchFrom: Josh Poimboeuf <[email protected]>
Add a debug feature to validate vmlinux and module .sframe sections at boot and module load time rather than on demand during unwinding.
[ Jens Remus: Add support for SFrame V3. Add support for PC-relative FDE function start offset. Adjust to rename of struct sframe_fre to sframe_fre_internal. ]
[ Dylan Hatch: Adapt this patch for in-kernel sframe sections. ]
---
Changes since v8:
- (sashiko) Update CONFIG_SFRAME_VALIDATION help text and commit message for in-kernel vmlinux and module .sframe validation, and move <asm/unwind_sframe.h> include to "arm64/sframe: Validate IP addresses".
--- arch/Kconfig | 19 +++++++++++ kernel/unwind/sframe.c | 77 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 96 insertions(+)
arch/Kconfig +19 -0
@@ -539,6 +539,25 @@ config HAVE_UNWIND_KERNEL_SFRAME practice, this can provide more reliable stacktrace results than unwinding with frame pointers alone. +config SFRAME_VALIDATION + bool "Enable .sframe section debugging" + depends on UNWIND_SFRAME_LOOKUP + depends on DYNAMIC_DEBUG + help + When initializing the .sframe section for vmlinux or a module, + validate the entire section immediately rather than on demand. + + This is a debug feature which is helpful for rooting out .sframe + section issues. If the .sframe section is corrupt, SFrame unwinding + for that object will be disabled immediately with a warning, and more + information provided in dynamic printks. + + This adds overhead at boot and module load time due to validating + every FDE and FRE in the .sframe section. Not recommended for + general use. + + If unsure, say N. + config HAVE_PERF_REGS bool help
kernel/unwind/sframe.c +77 -0
95 lines not on this page · raw message
Signature · 1 line
2.56.0.679.gc527c65306-goog
PATCH v9 9/11 arm64: entry: add unwind info for call_on_irq_stack()
In reply to Dylan HatchFrom: Weinan Liu <[email protected]>
DWARF CFI (Call Frame Information) specifies how to recover the return address and callee-saved registers at each PC in a given function. Compilers are able to generate the CFI annotations when they compile the code to assembly language. For handcrafted assembly, we need to annotate them by hand.
Frame pointers alone are usually sufficient to recover stack frames (without CFI), except at the exception boundary, where more information is needed to determine if the LR is live.
Since an exception can be taken from call_on_irq_stack(), annotate it with CFI. The actual entry assembly functions are left untouched, since they are not expected to take exceptions themselves.
---
Changes since v8:
- (sashiko) Use SYM_START() with .cfi_startproc before 'bti c' so the FDE covers the landing pad at call_on_irq_stack+0.
--- arch/arm64/kernel/entry.S | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-)
arch/arm64/kernel/entry.S +16 -1
@@ -30,6 +30,12 @@ #include <asm/asm-uaccess.h> #include <asm/unistd.h> +/* + * Do not generate .eh_frame. Only generate .debug_frame and optionally + * .sframe (via assembler option --gsframe[-N]). + */ + .cfi_sections .debug_frame + .macro clear_gp_regs .irp n,0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29 mov x\n, xzr
@@ -881,7 +887,9 @@ NOKPROBE(ret_from_fork) * * Calls func(regs) using this CPU's irq stack and shadow irq stack. */ -SYM_FUNC_START(call_on_irq_stack) +SYM_START(call_on_irq_stack, SYM_L_GLOBAL, SYM_A_ALIGN) + .cfi_startproc + bti c save_and_disable_daif x9 #ifdef CONFIG_SHADOW_CALL_STACK get_current_task x16
@@ -892,6 +900,9 @@ SYM_FUNC_START(call_on_irq_stack) /* Create a frame record to save our LR and SP (implicit in FP) */ stp x29, x30, [sp, #-16]! mov x29, sp + .cfi_def_cfa 29, 16 + .cfi_offset 29, -16 + .cfi_offset 30, -8 ldr_this_cpu x16, irq_stack_ptr, x17
@@ -907,9 +918,13 @@ SYM_FUNC_START(call_on_irq_stack) */ mov sp, x29 ldp x29, x30, [sp], #16 + .cfi_restore 29 + .cfi_restore 30 + .cfi_def_cfa 31, 0 scs_load_current restore_irq x9 ret + .cfi_endproc SYM_FUNC_END(call_on_irq_stack) NOKPROBE(call_on_irq_stack)
Signature · 1 line
2.56.0.679.gc527c65306-goog
PATCH v9 10/11 arm64, crypto/lib: Annotate leaf functions with CFI info
In reply to Dylan HatchDWARF CFI (Call Frame Information) specifies how to recover return address and callee-saved registers for annotated functions. These annotations are generated by the compiler, but for assembly, they must be annotated by hand.
Add simple CFI annotations to assembly leaf functions so that the LR can be recovered by the unwinder when an exception is taken from one of them.
---
Changes since v8:
- (conflict resolution) Drop arch/arm64/lib/clear_page.S (removed upstream in v7.4-rc1) and resolve context conflict in arch/arm64/lib/strncmp.S.
- (sashiko) Move out-of-line blocks in pmull_gcm_ghash_4x() and pmull_gcm_enc_4x() out of '.subsection 1' so their FDEs span the full functions.
- (sashiko) Annotate __pi_strrchr in arch/arm64/lib/strrchr.S with SYM_LEAF_FUNC_START / SYM_LEAF_FUNC_END.
--- arch/arm64/crypto/aes-ce-ccm-core.S | 12 ++-- arch/arm64/crypto/aes-neonbs-core.S | 16 +++--- arch/arm64/crypto/ghash-ce-core.S | 76 +++++++++++++------------- arch/arm64/crypto/sm4-ce-ccm-core.S | 16 +++--- arch/arm64/crypto/sm4-ce-cipher-core.S | 4 +- arch/arm64/crypto/sm4-ce-core.S | 44 +++++++-------- arch/arm64/crypto/sm4-ce-gcm-core.S | 16 +++--- arch/arm64/crypto/sm4-neon-core.S | 12 ++-- arch/arm64/include/asm/linkage.h | 32 +++++++++++ arch/arm64/lib/clear_user.S | 4 +- arch/arm64/lib/copy_from_user.S | 4 +- arch/arm64/lib/copy_page.S | 4 +- arch/arm64/lib/copy_to_user.S | 4 +- arch/arm64/lib/memchr.S | 4 +- arch/arm64/lib/memcmp.S | 4 +- arch/arm64/lib/memcpy.S | 8 +-- arch/arm64/lib/memset.S | 8 +-- arch/arm64/lib/mte.S | 28 +++++----- arch/arm64/lib/strchr.S | 4 +- arch/arm64/lib/strcmp.S | 4 +- arch/arm64/lib/strlen.S | 4 +- arch/arm64/lib/strncmp.S | 4 +- arch/arm64/lib/strnlen.S | 4 +- arch/arm64/lib/strrchr.S | 4 +- arch/arm64/lib/tishift.S | 12 ++-- 25 files changed, 181 insertions(+), 151 deletions(-)
arch/arm64/crypto/aes-ce-ccm-core.S +6 -6
@@ -80,7 +80,7 @@ ret .endm -SYM_FUNC_START_LOCAL(ce_aes_ccm_crypt_tail) +SYM_LEAF_FUNC_START_LOCAL(ce_aes_ccm_crypt_tail) eor v0.16b, v0.16b, v5.16b /* final round mac */ eor v1.16b, v1.16b, v5.16b /* final round enc */
@@ -113,7 +113,7 @@ SYM_INNER_LABEL(ce_aes_ccm_final, SYM_L_LOCAL) eor v0.16b, v0.16b, v1.16b /* en-/decrypt the mac */ 0: st1 {v0.16b}, [x5] /* store result */ ret -SYM_FUNC_END(ce_aes_ccm_crypt_tail) +SYM_LEAF_FUNC_END(ce_aes_ccm_crypt_tail) /* * void ce_aes_ccm_encrypt(u8 out[], u8 const in[], u32 cbytes,
@@ -123,15 +123,15 @@ SYM_FUNC_END(ce_aes_ccm_crypt_tail) * u8 const rk[], u32 rounds, u8 mac[], * u8 ctr[], u8 const final_iv[]); */ -SYM_FUNC_START(ce_aes_ccm_encrypt) +SYM_LEAF_FUNC_START(ce_aes_ccm_encrypt) movi v22.16b, #255 aes_ccm_do_crypt 1 -SYM_FUNC_END(ce_aes_ccm_encrypt) +SYM_LEAF_FUNC_END(ce_aes_ccm_encrypt) -SYM_FUNC_START(ce_aes_ccm_decrypt) +SYM_LEAF_FUNC_START(ce_aes_ccm_decrypt) movi v22.16b, #0 aes_ccm_do_crypt 0 -SYM_FUNC_END(ce_aes_ccm_decrypt) +SYM_LEAF_FUNC_END(ce_aes_ccm_decrypt) .section ".rodata", "a" .align 6
arch/arm64/crypto/aes-neonbs-core.S +8 -8
@@ -381,7 +381,7 @@ ISRM0: .octa 0x0306090c00070a0d01040b0e0205080f /* * void aesbs_convert_key(u8 out[], u32 const rk[], int rounds) */ -SYM_FUNC_START(aesbs_convert_key) +SYM_LEAF_FUNC_START(aesbs_convert_key) ld1 {v7.4s}, [x1], #16 // load round 0 key ld1 {v17.4s}, [x1], #16 // load round 1 key
@@ -426,10 +426,10 @@ SYM_FUNC_START(aesbs_convert_key) eor v17.16b, v17.16b, v7.16b str q17, [x0] ret -SYM_FUNC_END(aesbs_convert_key) +SYM_LEAF_FUNC_END(aesbs_convert_key) .align 4 -SYM_FUNC_START_LOCAL(aesbs_encrypt8) +SYM_LEAF_FUNC_START_LOCAL(aesbs_encrypt8) ldr q9, [bskey], #16 // round 0 key ldr q8, M0SR ldr q24, SR
@@ -489,10 +489,10 @@ SYM_FUNC_START_LOCAL(aesbs_encrypt8) eor v2.16b, v2.16b, v12.16b eor v5.16b, v5.16b, v12.16b ret -SYM_FUNC_END(aesbs_encrypt8) +SYM_LEAF_FUNC_END(aesbs_encrypt8) .align 4 -SYM_FUNC_START_LOCAL(aesbs_decrypt8) +SYM_LEAF_FUNC_START_LOCAL(aesbs_decrypt8) lsl x9, rounds, #7 add bskey, bskey, x9
@@ -554,7 +554,7 @@ SYM_FUNC_START_LOCAL(aesbs_decrypt8) eor v3.16b, v3.16b, v12.16b eor v5.16b, v5.16b, v12.16b ret -SYM_FUNC_END(aesbs_decrypt8) +SYM_LEAF_FUNC_END(aesbs_decrypt8) /* * aesbs_ecb_encrypt(u8 out[], u8 const in[], u8 const rk[], int rounds,
@@ -735,7 +735,7 @@ SYM_FUNC_END(aesbs_cbc_decrypt) * aesbs_xts_decrypt(u8 out[], u8 const in[], u8 const rk[], int rounds, * int blocks, u8 iv[]) */ -SYM_FUNC_START_LOCAL(__xts_crypt8) +SYM_LEAF_FUNC_START_LOCAL(__xts_crypt8) movi v18.2s, #0x1 movi v19.2s, #0x87 uzp1 v18.4s, v18.4s, v19.4s
@@ -766,7 +766,7 @@ SYM_FUNC_START_LOCAL(__xts_crypt8) mov bskey, x2 mov rounds, x3 br x16 -SYM_FUNC_END(__xts_crypt8) +SYM_LEAF_FUNC_END(__xts_crypt8) .macro __xts_crypt, do8, o0, o1, o2, o3, o4, o5, o6, o7 frame_push 0, 32
arch/arm64/crypto/ghash-ce-core.S +37 -39
119 lines not on this page · raw message
arch/arm64/crypto/sm4-ce-ccm-core.S +8 -8
@@ -37,7 +37,7 @@ .align 3 -SYM_FUNC_START(sm4_ce_cbcmac_update) +SYM_LEAF_FUNC_START(sm4_ce_cbcmac_update) /* input: * x0: round key array, CTX * x1: mac
@@ -81,10 +81,10 @@ SYM_FUNC_START(sm4_ce_cbcmac_update) .Lcbcmac_end: st1 {RMAC.16b}, [x1] ret -SYM_FUNC_END(sm4_ce_cbcmac_update) +SYM_LEAF_FUNC_END(sm4_ce_cbcmac_update) .align 3 -SYM_FUNC_START(sm4_ce_ccm_final) +SYM_LEAF_FUNC_START(sm4_ce_ccm_final) /* input: * x0: round key array, CTX * x1: ctr0 (big endian, 128 bit)
@@ -102,10 +102,10 @@ SYM_FUNC_START(sm4_ce_ccm_final) st1 {RMAC.16b}, [x2] ret -SYM_FUNC_END(sm4_ce_ccm_final) +SYM_LEAF_FUNC_END(sm4_ce_ccm_final) .align 3 -SYM_TYPED_FUNC_START(sm4_ce_ccm_enc) +SYM_TYPED_LEAF_FUNC_START(sm4_ce_ccm_enc) /* input: * x0: round key array, CTX * x1: dst
@@ -214,10 +214,10 @@ SYM_TYPED_FUNC_START(sm4_ce_ccm_enc) .Lccm_enc_ret: ret -SYM_FUNC_END(sm4_ce_ccm_enc) +SYM_LEAF_FUNC_END(sm4_ce_ccm_enc) .align 3 -SYM_TYPED_FUNC_START(sm4_ce_ccm_dec) +SYM_TYPED_LEAF_FUNC_START(sm4_ce_ccm_dec) /* input: * x0: round key array, CTX * x1: dst
@@ -326,4 +326,4 @@ SYM_TYPED_FUNC_START(sm4_ce_ccm_dec) .Lccm_dec_ret: ret -SYM_FUNC_END(sm4_ce_ccm_dec) +SYM_LEAF_FUNC_END(sm4_ce_ccm_dec)
arch/arm64/crypto/sm4-ce-cipher-core.S +2 -2
@@ -15,7 +15,7 @@ * void sm4_ce_do_crypt(const u32 *rk, u32 *out, const u32 *in); */ .text -SYM_FUNC_START(sm4_ce_do_crypt) +SYM_LEAF_FUNC_START(sm4_ce_do_crypt) ld1 {v8.4s}, [x2] ld1 {v0.4s-v3.4s}, [x0], #64 rev32 v8.16b, v8.16b
@@ -33,4 +33,4 @@ SYM_FUNC_START(sm4_ce_do_crypt) rev32 v8.16b, v8.16b st1 {v8.4s}, [x1] ret -SYM_FUNC_END(sm4_ce_do_crypt) +SYM_LEAF_FUNC_END(sm4_ce_do_crypt)
arch/arm64/crypto/sm4-ce-core.S +22 -22
140 lines not on this page · raw message
arch/arm64/crypto/sm4-ce-gcm-core.S +8 -8
@@ -259,7 +259,7 @@ #define RH4 v19 .align 3 -SYM_FUNC_START(sm4_ce_pmull_ghash_setup) +SYM_LEAF_FUNC_START(sm4_ce_pmull_ghash_setup) /* input: * x0: round key array, CTX * x1: ghash table
@@ -293,10 +293,10 @@ SYM_FUNC_START(sm4_ce_pmull_ghash_setup) st1 {RH1.16b-RH4.16b}, [x1] ret -SYM_FUNC_END(sm4_ce_pmull_ghash_setup) +SYM_LEAF_FUNC_END(sm4_ce_pmull_ghash_setup) .align 3 -SYM_FUNC_START(pmull_ghash_update) +SYM_LEAF_FUNC_START(pmull_ghash_update) /* input: * x0: ghash table * x1: ghash result
@@ -368,10 +368,10 @@ SYM_FUNC_START(pmull_ghash_update) st1 {RHASH.2d}, [x1] ret -SYM_FUNC_END(pmull_ghash_update) +SYM_LEAF_FUNC_END(pmull_ghash_update) .align 3 -SYM_TYPED_FUNC_START(sm4_ce_pmull_gcm_enc) +SYM_TYPED_LEAF_FUNC_START(sm4_ce_pmull_gcm_enc) /* input: * x0: round key array, CTX * x1: dst
@@ -534,7 +534,7 @@ SYM_TYPED_FUNC_START(sm4_ce_pmull_gcm_enc) st1 {RHASH.2d}, [x5] ret -SYM_FUNC_END(sm4_ce_pmull_gcm_enc) +SYM_LEAF_FUNC_END(sm4_ce_pmull_gcm_enc) #undef RR1 #undef RR3
@@ -582,7 +582,7 @@ SYM_FUNC_END(sm4_ce_pmull_gcm_enc) #define RH3 v20 .align 3 -SYM_TYPED_FUNC_START(sm4_ce_pmull_gcm_dec) +SYM_TYPED_LEAF_FUNC_START(sm4_ce_pmull_gcm_dec) /* input: * x0: round key array, CTX * x1: dst
@@ -726,7 +726,7 @@ SYM_TYPED_FUNC_START(sm4_ce_pmull_gcm_dec) st1 {RHASH.2d}, [x5] ret -SYM_FUNC_END(sm4_ce_pmull_gcm_dec) +SYM_LEAF_FUNC_END(sm4_ce_pmull_gcm_dec) .section ".rodata", "a" .align 4
arch/arm64/crypto/sm4-neon-core.S +6 -6
@@ -257,7 +257,7 @@ .align 3 -SYM_FUNC_START(sm4_neon_crypt) +SYM_LEAF_FUNC_START(sm4_neon_crypt) /* input: * x0: round key array, CTX * x1: dst
@@ -318,10 +318,10 @@ SYM_FUNC_START(sm4_neon_crypt) .Lcrypt_end: ret -SYM_FUNC_END(sm4_neon_crypt) +SYM_LEAF_FUNC_END(sm4_neon_crypt) .align 3 -SYM_FUNC_START(sm4_neon_cbc_dec) +SYM_LEAF_FUNC_START(sm4_neon_cbc_dec) /* input: * x0: round key array, CTX * x1: dst
@@ -435,10 +435,10 @@ SYM_FUNC_START(sm4_neon_cbc_dec) st1 {RIV.16b}, [x3] ret -SYM_FUNC_END(sm4_neon_cbc_dec) +SYM_LEAF_FUNC_END(sm4_neon_cbc_dec) .align 3 -SYM_FUNC_START(sm4_neon_ctr_crypt) +SYM_LEAF_FUNC_START(sm4_neon_ctr_crypt) /* input: * x0: round key array, CTX * x1: dst
@@ -563,4 +563,4 @@ SYM_FUNC_START(sm4_neon_ctr_crypt) stp x7, x8, [x3] ret -SYM_FUNC_END(sm4_neon_ctr_crypt) +SYM_LEAF_FUNC_END(sm4_neon_ctr_crypt)
arch/arm64/include/asm/linkage.h +32 -0
@@ -3,6 +3,12 @@#ifdef __ASSEMBLER__#include <asm/assembler.h> + +/* +* Do not generate .eh_frame. Only generate .debug_frame and optionally +* .sframe (via assembler option --gsframe[-N]). +*/ + .cfi_sections .debug_frame #endif#define __ALIGN .balign CONFIG_FUNCTION_ALIGNMENT
@@ -47,4 +53,30 @@#define __bss_pgtbl __section(".bss..pgtbl") __aligned(PAGE_SIZE) +/* +* SYM_[TYPED_]LEAF_FUNC_[START|END] macros add CFI minimal CFI directives +* allowing .sframe data to be generated for functions which do not modify the +* LR (x30). Unwind data will not be correct if these macros are used on +* non-leaf functions, as additional CFI directives would be necessary in such +* cases. +*/ +#define SYM_LEAF_FUNC_START (name) \ + SYM_START(name, SYM_L_GLOBAL, SYM_A_ALIGN) \ + .cfi_startproc ; \ + bti c ; + +#define SYM_LEAF_FUNC_END (name) \ + .cfi_endproc ; \ + SYM_FUNC_END(name) + +#define SYM_LEAF_FUNC_START_LOCAL (name) \ + SYM_START(name, SYM_L_LOCAL, SYM_A_ALIGN) \ + .cfi_startproc ; \ + bti c ; + +#define SYM_TYPED_LEAF_FUNC_START (name) \ + SYM_TYPED_START(name, SYM_L_GLOBAL, SYM_A_ALIGN) \ + .cfi_startproc ; \ + bti c ; +#endif
arch/arm64/lib/clear_user.S +2 -2
@@ -17,7 +17,7 @@ * Alignment fixed up by hardware. */ -SYM_FUNC_START(__arch_clear_user) +SYM_LEAF_FUNC_START(__arch_clear_user) add x2, x0, x1 #ifdef CONFIG_AS_HAS_MOPS
@@ -68,5 +68,5 @@ USER(7f, sttrb wzr, [x2, #-1]) 8: add x0, x0, #4 // ...or the second word of the 4-7 byte case 9: sub x0, x2, x0 ret -SYM_FUNC_END(__arch_clear_user) +SYM_LEAF_FUNC_END(__arch_clear_user) EXPORT_SYMBOL(__arch_clear_user)
arch/arm64/lib/copy_from_user.S +2 -2
@@ -61,7 +61,7 @@ end .req x5 srcin .req x15 -SYM_FUNC_START(__arch_copy_from_user) +SYM_LEAF_FUNC_START(__arch_copy_from_user) add end, x0, x2 mov srcin, x1 #include "copy_template.S"
@@ -79,5 +79,5 @@ USER(9998f, ldtrb tmp1w, [srcin]) strb tmp1w, [dst], #1 9998: sub x0, end, dst // bytes not copied ret -SYM_FUNC_END(__arch_copy_from_user) +SYM_LEAF_FUNC_END(__arch_copy_from_user) EXPORT_SYMBOL(__arch_copy_from_user)
arch/arm64/lib/copy_page.S +2 -2
@@ -17,7 +17,7 @@ * x0 - dest * x1 - src */ -SYM_FUNC_START(__pi_copy_page) +SYM_LEAF_FUNC_START(__pi_copy_page) #ifdef CONFIG_AS_HAS_MOPS .arch_extension mops alternative_if_not ARM64_HAS_MOPS
@@ -77,6 +77,6 @@ alternative_else_nop_endif stnp x16, x17, [x0, #112 - 256] ret -SYM_FUNC_END(__pi_copy_page) +SYM_LEAF_FUNC_END(__pi_copy_page) SYM_FUNC_ALIAS(copy_page, __pi_copy_page) EXPORT_SYMBOL(copy_page)
arch/arm64/lib/copy_to_user.S +2 -2
@@ -60,7 +60,7 @@ end .req x5 srcin .req x15 -SYM_FUNC_START(__arch_copy_to_user) +SYM_LEAF_FUNC_START(__arch_copy_to_user) add end, x0, x2 mov srcin, x1 #include "copy_template.S"
@@ -79,5 +79,5 @@ USER(9998f, sttrb tmp1w, [dst]) add dst, dst, #1 9998: sub x0, end, dst // bytes not copied ret -SYM_FUNC_END(__arch_copy_to_user) +SYM_LEAF_FUNC_END(__arch_copy_to_user) EXPORT_SYMBOL(__arch_copy_to_user)
arch/arm64/lib/memchr.S +2 -2
@@ -38,7 +38,7 @@ .p2align 4 nop -SYM_FUNC_START(__pi_memchr) +SYM_LEAF_FUNC_START(__pi_memchr) and chrin, chrin, #0xff lsr wordcnt, cntin, #3 cbz wordcnt, L(byte_loop)
@@ -71,6 +71,6 @@ L(found_word): L(not_found): mov result, #0 ret -SYM_FUNC_END(__pi_memchr) +SYM_LEAF_FUNC_END(__pi_memchr) SYM_FUNC_ALIAS_WEAK(memchr, __pi_memchr) EXPORT_SYMBOL_NOKASAN(memchr)
arch/arm64/lib/memcmp.S +2 -2
@@ -32,7 +32,7 @@ #define tmp1 x7 #define tmp2 x8 -SYM_FUNC_START(__pi_memcmp) +SYM_LEAF_FUNC_START(__pi_memcmp) subs limit, limit, 8 b.lo L(less8)
@@ -132,6 +132,6 @@ L(byte_loop): b.eq L(byte_loop) sub result, data1w, data2w ret -SYM_FUNC_END(__pi_memcmp) +SYM_LEAF_FUNC_END(__pi_memcmp) SYM_FUNC_ALIAS_WEAK(memcmp, __pi_memcmp) EXPORT_SYMBOL_NOKASAN(memcmp)
arch/arm64/lib/memcpy.S +4 -4
@@ -57,7 +57,7 @@ The loop tail is handled by always copying 64 bytes from the end. */ -SYM_FUNC_START_LOCAL(__pi_memcpy_generic) +SYM_LEAF_FUNC_START_LOCAL(__pi_memcpy_generic) add srcend, src, count add dstend, dstin, count cmp count, 128
@@ -238,11 +238,11 @@ L(copy64_from_start): stp B_l, B_h, [dstin, 16] stp C_l, C_h, [dstin] ret -SYM_FUNC_END(__pi_memcpy_generic) +SYM_LEAF_FUNC_END(__pi_memcpy_generic) #ifdef CONFIG_AS_HAS_MOPS .arch_extension mops -SYM_FUNC_START(__pi_memcpy) +SYM_LEAF_FUNC_START(__pi_memcpy) alternative_if_not ARM64_HAS_MOPS b __pi_memcpy_generic alternative_else_nop_endif
@@ -252,7 +252,7 @@ alternative_else_nop_endif cpym [dst]!, [src]!, count! cpye [dst]!, [src]!, count! ret -SYM_FUNC_END(__pi_memcpy) +SYM_LEAF_FUNC_END(__pi_memcpy) #else SYM_FUNC_ALIAS(__pi_memcpy, __pi_memcpy_generic) #endif
arch/arm64/lib/memset.S +4 -4
@@ -43,7 +43,7 @@ dst .req x8 tmp3w .req w9 tmp3 .req x9 -SYM_FUNC_START_LOCAL(__pi_memset_generic) +SYM_LEAF_FUNC_START_LOCAL(__pi_memset_generic) mov dst, dstin /* Preserve return value. */ and A_lw, val, #255 orr A_lw, A_lw, A_lw, lsl #8
@@ -202,11 +202,11 @@ SYM_FUNC_START_LOCAL(__pi_memset_generic) ands count, count, zva_bits_x b.ne .Ltail_maybe_long ret -SYM_FUNC_END(__pi_memset_generic) +SYM_LEAF_FUNC_END(__pi_memset_generic) #ifdef CONFIG_AS_HAS_MOPS .arch_extension mops -SYM_FUNC_START(__pi_memset) +SYM_LEAF_FUNC_START(__pi_memset) alternative_if_not ARM64_HAS_MOPS b __pi_memset_generic alternative_else_nop_endif
@@ -216,7 +216,7 @@ alternative_else_nop_endif setm [dst]!, count!, val_x sete [dst]!, count!, val_x ret -SYM_FUNC_END(__pi_memset) +SYM_LEAF_FUNC_END(__pi_memset) #else SYM_FUNC_ALIAS(__pi_memset, __pi_memset_generic) #endif
arch/arm64/lib/mte.S +14 -14
@@ -27,14 +27,14 @@ * Clear the tags in a page * x0 - address of the page to be cleared */ -SYM_FUNC_START(mte_clear_page_tags) +SYM_LEAF_FUNC_START(mte_clear_page_tags) multitag_transfer_size x1, x2 1: stgm xzr, [x0] add x0, x0, x1 tst x0, #(PAGE_SIZE - 1) b.ne 1b ret -SYM_FUNC_END(mte_clear_page_tags) +SYM_LEAF_FUNC_END(mte_clear_page_tags) /* * Zero the page and tags at the same time
@@ -42,7 +42,7 @@ SYM_FUNC_END(mte_clear_page_tags) * Parameters: * x0 - address to the beginning of the page */ -SYM_FUNC_START(mte_zero_clear_page_tags) +SYM_LEAF_FUNC_START(mte_zero_clear_page_tags) and x0, x0, #(1 << MTE_TAG_SHIFT) - 1 // clear the tag mrs x1, dczid_el0 tbnz x1, #4, 2f // Branch if DC GZVA is prohibited
@@ -60,14 +60,14 @@ SYM_FUNC_START(mte_zero_clear_page_tags) tst x0, #(PAGE_SIZE - 1) b.ne 2b ret -SYM_FUNC_END(mte_zero_clear_page_tags) +SYM_LEAF_FUNC_END(mte_zero_clear_page_tags) /* * Copy the tags from the source page to the destination one * x0 - address of the destination page * x1 - address of the source page */ -SYM_FUNC_START(mte_copy_page_tags) +SYM_LEAF_FUNC_START(mte_copy_page_tags) mov x2, x0 mov x3, x1 multitag_transfer_size x5, x6
@@ -78,7 +78,7 @@ SYM_FUNC_START(mte_copy_page_tags) tst x2, #(PAGE_SIZE - 1) b.ne 1b ret -SYM_FUNC_END(mte_copy_page_tags) +SYM_LEAF_FUNC_END(mte_copy_page_tags) /* * Read tags from a user buffer (one tag per byte) and set the corresponding
@@ -89,7 +89,7 @@ SYM_FUNC_END(mte_copy_page_tags) * Returns: * x0 - number of tags read/set */ -SYM_FUNC_START(mte_copy_tags_from_user) +SYM_LEAF_FUNC_START(mte_copy_tags_from_user) mov x3, x1 cbz x2, 2f 1:
@@ -103,7 +103,7 @@ USER(2f, ldtrb w4, [x1]) // exception handling and function return 2: sub x0, x1, x3 // update the number of tags set ret -SYM_FUNC_END(mte_copy_tags_from_user) +SYM_LEAF_FUNC_END(mte_copy_tags_from_user) /* * Get the tags from a kernel address range and write the tag values to the
@@ -114,7 +114,7 @@ SYM_FUNC_END(mte_copy_tags_from_user) * Returns: * x0 - number of tags read/set */ -SYM_FUNC_START(mte_copy_tags_to_user) +SYM_LEAF_FUNC_START(mte_copy_tags_to_user) mov x3, x0 cbz x2, 2f 1:
@@ -129,14 +129,14 @@ USER(2f, sttrb w4, [x0]) // exception handling and function return 2: sub x0, x0, x3 // update the number of tags copied ret -SYM_FUNC_END(mte_copy_tags_to_user) +SYM_LEAF_FUNC_END(mte_copy_tags_to_user) /* * Save the tags in a page * x0 - page address * x1 - tag storage, MTE_PAGE_TAG_STORAGE bytes */ -SYM_FUNC_START(mte_save_page_tags) +SYM_LEAF_FUNC_START(mte_save_page_tags) multitag_transfer_size x7, x5 1: mov x2, #0
@@ -153,14 +153,14 @@ SYM_FUNC_START(mte_save_page_tags) b.ne 1b ret -SYM_FUNC_END(mte_save_page_tags) +SYM_LEAF_FUNC_END(mte_save_page_tags) /* * Restore the tags in a page * x0 - page address * x1 - tag storage, MTE_PAGE_TAG_STORAGE bytes */ -SYM_FUNC_START(mte_restore_page_tags) +SYM_LEAF_FUNC_START(mte_restore_page_tags) multitag_transfer_size x7, x5 1: ldr x2, [x1], #8
@@ -174,4 +174,4 @@ SYM_FUNC_START(mte_restore_page_tags) b.ne 1b ret -SYM_FUNC_END(mte_restore_page_tags) +SYM_LEAF_FUNC_END(mte_restore_page_tags)
arch/arm64/lib/strchr.S +2 -2
@@ -18,7 +18,7 @@ * Returns: * x0 - address of first occurrence of 'c' or 0 */ -SYM_FUNC_START(__pi_strchr) +SYM_LEAF_FUNC_START(__pi_strchr) and w1, w1, #0xff 1: ldrb w2, [x0], #1 cmp w2, w1
@@ -28,7 +28,7 @@ SYM_FUNC_START(__pi_strchr) cmp w2, w1 csel x0, x0, xzr, eq ret -SYM_FUNC_END(__pi_strchr) +SYM_LEAF_FUNC_END(__pi_strchr) SYM_FUNC_ALIAS_WEAK(strchr, __pi_strchr) EXPORT_SYMBOL_NOKASAN(strchr)
arch/arm64/lib/strcmp.S +2 -2
@@ -45,7 +45,7 @@ NUL too in big-endian, byte-reverse the data before the NUL check. */ -SYM_FUNC_START(__pi_strcmp) +SYM_LEAF_FUNC_START(__pi_strcmp) sub off2, src2, src1 mov zeroones, REP8_01 and tmp, src1, 7
@@ -153,6 +153,6 @@ L(tail): L(done): sub result, data1, data2 ret -SYM_FUNC_END(__pi_strcmp) +SYM_LEAF_FUNC_END(__pi_strcmp) SYM_FUNC_ALIAS_WEAK(strcmp, __pi_strcmp) EXPORT_SYMBOL_NOKASAN(strcmp)
arch/arm64/lib/strlen.S +2 -2
@@ -79,7 +79,7 @@ whether the first fetch, which may be misaligned, crosses a page boundary. */ -SYM_FUNC_START(__pi_strlen) +SYM_LEAF_FUNC_START(__pi_strlen) and tmp1, srcin, MIN_PAGE_SIZE - 1 mov zeroones, REP8_01 cmp tmp1, MIN_PAGE_SIZE - 16
@@ -182,6 +182,6 @@ L(page_cross): csel data1, data1, tmp4, eq csel data2, data2, tmp2, eq b L(page_cross_entry) -SYM_FUNC_END(__pi_strlen) +SYM_LEAF_FUNC_END(__pi_strlen) SYM_FUNC_ALIAS_WEAK(strlen, __pi_strlen) EXPORT_SYMBOL_NOKASAN(strlen)
arch/arm64/lib/strncmp.S +2 -2
@@ -45,7 +45,7 @@ #define offset pos #define neg_offset x15 -SYM_FUNC_START(__pi_strncmp) +SYM_LEAF_FUNC_START(__pi_strncmp) cbz limit, L(ret0) eor tmp1, src1, src2 mov zeroones, #REP8_01
@@ -224,6 +224,6 @@ L(misaligned_mid_loop): L(ret0): mov result, #0 ret -SYM_FUNC_END(__pi_strncmp) +SYM_LEAF_FUNC_END(__pi_strncmp) SYM_FUNC_ALIAS_WEAK(strncmp, __pi_strncmp) EXPORT_SYMBOL_NOKASAN(strncmp)
arch/arm64/lib/strnlen.S +2 -2
@@ -47,7 +47,7 @@ limit_wd .req x14 #define REP8_7f 0x7f7f7f7f7f7f7f7f #define REP8_80 0x8080808080808080 -SYM_FUNC_START(__pi_strnlen) +SYM_LEAF_FUNC_START(__pi_strnlen) cbz limit, .Lhit_limit mov zeroones, #REP8_01 bic src, srcin, #15
@@ -142,7 +142,7 @@ SYM_FUNC_START(__pi_strnlen) .Lhit_limit: mov len, limit ret -SYM_FUNC_END(__pi_strnlen) +SYM_LEAF_FUNC_END(__pi_strnlen) SYM_FUNC_ALIAS_WEAK(strnlen, __pi_strnlen) EXPORT_SYMBOL_NOKASAN(strnlen)
arch/arm64/lib/strrchr.S +2 -2
@@ -18,7 +18,7 @@ * Returns: * x0 - address of last occurrence of 'c' or 0 */ -SYM_FUNC_START(__pi_strrchr) +SYM_LEAF_FUNC_START(__pi_strrchr) mov x3, #0 and w1, w1, #0xff 1: ldrb w2, [x0], #1
@@ -29,6 +29,6 @@ SYM_FUNC_START(__pi_strrchr) b 1b 2: mov x0, x3 ret -SYM_FUNC_END(__pi_strrchr) +SYM_LEAF_FUNC_END(__pi_strrchr) SYM_FUNC_ALIAS_WEAK(strrchr, __pi_strrchr) EXPORT_SYMBOL_NOKASAN(strrchr)
arch/arm64/lib/tishift.S +6 -6
@@ -7,7 +7,7 @@ #include <asm/assembler.h> -SYM_FUNC_START(__ashlti3) +SYM_LEAF_FUNC_START(__ashlti3) cbz x2, 1f mov x3, #64 sub x3, x3, x2
@@ -26,10 +26,10 @@ SYM_FUNC_START(__ashlti3) lsl x1, x0, x1 mov x0, x2 ret -SYM_FUNC_END(__ashlti3) +SYM_LEAF_FUNC_END(__ashlti3) EXPORT_SYMBOL(__ashlti3) -SYM_FUNC_START(__ashrti3) +SYM_LEAF_FUNC_START(__ashrti3) cbz x2, 1f mov x3, #64 sub x3, x3, x2
@@ -48,10 +48,10 @@ SYM_FUNC_START(__ashrti3) asr x0, x1, x0 mov x1, x2 ret -SYM_FUNC_END(__ashrti3) +SYM_LEAF_FUNC_END(__ashrti3) EXPORT_SYMBOL(__ashrti3) -SYM_FUNC_START(__lshrti3) +SYM_LEAF_FUNC_START(__lshrti3) cbz x2, 1f mov x3, #64 sub x3, x3, x2
@@ -70,5 +70,5 @@ SYM_FUNC_START(__lshrti3) lsr x0, x1, x0 mov x1, x2 ret -SYM_FUNC_END(__lshrti3) +SYM_LEAF_FUNC_END(__lshrti3) EXPORT_SYMBOL(__lshrti3)
Signature · 1 line
2.56.0.679.gc527c65306-goog
PATCH v9 11/11 unwind: arm64: Use sframe to unwind interrupt frames
In reply to Dylan HatchAdd kunwind_next_regs_sframe() function to unwind by sframe info if present. Use this method at exception boundaries, falling back to frame-pointer unwind only on failure. In such failure cases, the stacktrace is considered unreliable. During normal unwind, prefer frame pointer unwind (for better performance).
This change restores the LR behavior originally introduced in commit c2c6b27b5aa14fa2 ("arm64: stacktrace: unwind exception boundaries"), But later removed in commit 32ed1205682e ("arm64: stacktrace: Skip reporting LR at exception boundaries")
This can be done because the sframe data can be used to determine whether the LR is current for the PC value recovered from pt_regs at the exception boundary.
---
Changes since v8:
- (sashiko) Restrict SFrame unwinding at KUNWIND_SOURCE_REGS_PC to real on-stack FRAME_META_TYPE_PT_REGS exception frames so synthetic pt_regs fall back to frame-pointer unwinding.
- (sashiko) Remove invalid cross-stack 'fp < sp' check in UNWIND_CFA_RULE_FP_OFFSET and enforce 8-byte alignment in get_word().
- (sashiko) Propagate state->unreliable when kunwind_next() terminates with -ENOENT in do_kunwind().
--- arch/arm64/kernel/stacktrace.c | 211 +++++++++++++++++++++++++++++++-- 1 file changed, 198 insertions(+), 13 deletions(-)
arch/arm64/kernel/stacktrace.c +198 -13
@@ -13,6 +13,7 @@#include <linux/sched.h> #include <linux/sched/debug.h> #include <linux/sched/task_stack.h> +#include <linux/sframe.h> #include <linux/stacktrace.h> #include <asm/efi.h>
@@ -26,6 +27,7 @@ enum kunwind_source {KUNWIND_SOURCE_CALLER ,KUNWIND_SOURCE_TASK ,KUNWIND_SOURCE_REGS_PC , + KUNWIND_SOURCE_REGS_LR, }; union unwind_flags {
@@ -45,6 +47,7 @@ union unwind_flags {* @kr_cur: When KRETPROBES is selected, holds the kretprobe instance * associated with the most recently encountered replacement lr * value. +* @unreliable: Stacktrace is unreliable. */ struct kunwind_state {struct unwind_state common;
@@ -56,6 +59,7 @@ struct kunwind_state {enum kunwind_source source;union unwind_flags flags;struct pt_regs *regs; +bool unreliable; };static __always_inline void
@@ -80,12 +84,31 @@ static __always_inline void kunwind_init_from_regs(struct kunwind_state *state, struct pt_regs *regs) { + struct stack_info *info; +kunwind_init (state, current); - state->regs = regs; state->common.fp = regs->regs[29 ]; state->common.pc = regs->pc; state->source =KUNWIND_SOURCE_REGS_PC ; + +/* +* Synthetic pt_regs (e.g. from perf_fetch_caller_regs() or +* ftrace_fill_perf_regs()) do not have a full register state +* consistent with regs->pc and cannot be unwound with SFrame. +* Real exception pt_regs have FRAME_META_TYPE_PT_REGS. +*/ +if (!regs->stackframe.record.fp && !regs->stackframe.record.lr && + regs->stackframe.type ==FRAME_META_TYPE_PT_REGS ) { + info =unwind_find_stack (&state->common, (unsigned long )regs, +sizeof (*regs)); +if (info) { +unwind_consume_stack (&state->common, info, + (unsigned long )regs, +sizeof (*regs)); + state->regs = regs; + } + } }/*
@@ -181,7 +204,6 @@ int kunwind_next_regs_pc(struct kunwind_state *state) state->regs = regs; state->common.pc = regs->pc; state->common.fp = regs->regs[29 ]; - state->regs =NULL ; state->source =KUNWIND_SOURCE_REGS_PC ;return 0 ; }
@@ -244,6 +266,163 @@ kunwind_next_frame_record(struct kunwind_state *state)return 0 ; } +#ifdef CONFIG_HAVE_UNWIND_KERNEL_SFRAME + +static __always_inline struct stack_info * +get_word (struct unwind_state *state,unsigned long *word) +{ +unsigned long addr = *word; +struct stack_info *info; + +if (addr &0x7 ) +return NULL ; + + info =unwind_find_stack (state, addr,sizeof (addr)); +if (!info) +return info; + + *word =READ_ONCE (*(unsigned long *)addr); + +return info; +} + +static __always_inline int +get_consume_word (struct unwind_state *state,unsigned long *word) +{ +struct stack_info *info; +unsigned long addr = *word; + + info =get_word (state, word); +if (!info) +return -EINVAL ; + +unwind_consume_stack (state, info, addr,sizeof (addr)); +return 0 ; +} + +/* +* Unwind from a pt_regs according to sframe. +*/ +static __always_inline int +kunwind_next_regs_sframe (struct kunwind_state *state) +{ +struct unwind_frame frame; +unsigned long cfa, fp, ra; +enum kunwind_source source =KUNWIND_SOURCE_FRAME ; +struct pt_regs *regs = state->regs; + +int err; + +if (WARN_ON_ONCE (state->source !=KUNWIND_SOURCE_REGS_PC )) +return -EINVAL ; +if (WARN_ON_ONCE (!state->regs)) +return -EINVAL ; + +/* FP/SP alignment 8 bytes */ +if (state->common.fp &0x7 ) +return -EINVAL ; + + err =sframe_find (state->common.pc, &frame); +if (err) +return -EINVAL ; + +/* +* A kernel unwind should always end at a FRAME_META_TYPE_FINAL +* frame. There should be no outermost frames within the kernel. +*/ +if (frame.outermost) +return -EINVAL ; + +/* Get the Canonical Frame Address (CFA) */ +switch (frame.cfa.rule) { +case UNWIND_CFA_RULE_SP_OFFSET : + cfa = state->regs->sp; +break ; +case UNWIND_CFA_RULE_FP_OFFSET : + cfa = state->common.fp; +break ; +/* +* UNWIND_CFA_RULE_REG_OFFSET and UNWIND_CFA_RULE_REG_OFFSET_DEREF not +* implemented -- flexible FDEs are not currently generated by assembler +* for arm64. +*/ +default : +WARN_ON_ONCE (1 ); +return -EINVAL ; + } + cfa += frame.cfa.offset; + +/* CFA alignment 16 bytes */ +if (cfa &0xf ) +return -EINVAL ; + +/* +* Consume RA and FP from the stack. The frame record puts FP at a lower +* address than RA, so we read FP first. +*/ +switch (frame.fp.rule) { +case UNWIND_RULE_RETAIN : + fp = state->common.fp; +break ; +/* +* UNWIND_RULE_CFA_OFFSET is currently not used for FP +* (e.g. SFrame cannot represent this rule). +*/ +case UNWIND_RULE_CFA_OFFSET_DEREF : + fp = cfa + frame.fp.offset; +if (!get_word (&state->common, &fp)) +return -EINVAL ; +break ; +/* +* UNWIND_RULE_REG_OFFSET and UNWIND_RULE_REG_OFFSET_DEREF not +* implemented -- flexible FDEs are not currently generated by assembler +* for arm64. +*/ +default : +WARN_ON_ONCE (1 ); +return -EINVAL ; + } + +switch (frame.ra.rule) { +case UNWIND_RULE_RETAIN : + ra = regs->regs[30 ]; + source =KUNWIND_SOURCE_REGS_LR ; +break ; + +/* +* UNWIND_RULE_CFA_OFFSET doesn't make sense for RA. +* The return address cannot legitimately be a stack address. +*/ +case UNWIND_RULE_CFA_OFFSET_DEREF : + ra = cfa + frame.ra.offset; +if (get_consume_word (&state->common, &ra)) +return -EINVAL ; +break ; +/* +* UNWIND_RULE_REG_OFFSET and UNWIND_RULE_REG_OFFSET_DEREF not +* implemented -- flexible FDEs are not currently generated by assembler +* for arm64. +*/ +default : +WARN_ON_ONCE (1 ); +return -EINVAL ; + } + + state->common.pc = ra; + state->common.fp = fp; + + state->source = source; + +return 0 ; +} + +#else /* !CONFIG_HAVE_UNWIND_KERNEL_SFRAME */ + +static __always_inline int +kunwind_next_regs_sframe (struct kunwind_state *state) {return -EINVAL ; } + +#endif /* !CONFIG_HAVE_UNWIND_KERNEL_SFRAME*/ +/* * Unwind from one frame record (A) to the next frame record (B). *
@@ -259,10 +438,23 @@ kunwind_next(struct kunwind_state *state) state->flags.all =0 ;switch (state->source) { +case KUNWIND_SOURCE_REGS_PC : +if (state->regs) + err =kunwind_next_regs_sframe (state); +else + err = -EINVAL ; + +if (err) { +/* Fallback to FP based unwinder */ + err =kunwind_next_frame_record (state); + state->unreliable = true; + } + state->regs =NULL ; +break ;case KUNWIND_SOURCE_FRAME :case KUNWIND_SOURCE_CALLER :case KUNWIND_SOURCE_TASK : -case KUNWIND_SOURCE_REGS_PC : +case KUNWIND_SOURCE_REGS_LR : err =kunwind_next_frame_record (state);break ;default :
@@ -294,7 +486,7 @@ do_kunwind(struct kunwind_state *state, kunwind_consume_fn consume_state,return -EINVAL ; ret =kunwind_next (state);if (ret == -ENOENT ) -return 0 ; +return state->unreliable ? -EINVAL :0 ;if (ret <0 )return ret; }
@@ -393,15 +585,7 @@ noinline noinstr void arch_stack_walk(stack_trace_consume_fn consume_entry, static __always_inline boolarch_reliable_kunwind_consume_entry (const struct kunwind_state *state,void *cookie) { -/* -* At an exception boundary we can reliably consume the saved PC. We do -* not know whether the LR was live when the exception was taken, and -* so we cannot perform the next unwind step reliably. -* -* All that matters is whether the *entire* unwind is reliable, so give -* up as soon as we hit an exception boundary. -*/ -if (state->source ==KUNWIND_SOURCE_REGS_PC ) +if (state->unreliable)return false;return arch_kunwind_consume_entry (state, cookie);
@@ -452,6 +636,7 @@ static const char *state_source_string(const struct kunwind_state *state)case KUNWIND_SOURCE_CALLER :return "C" ;case KUNWIND_SOURCE_TASK :return "T" ;case KUNWIND_SOURCE_REGS_PC :return "P" ; +case KUNWIND_SOURCE_REGS_LR :return "L" ;default :return "U" ; } }
Signature · 1 line
2.56.0.679.gc527c65306-goog
[email protected]
AI reviewRe: [PATCH v9 11/11] unwind: arm64: Use sframe to unwind interrupt frames
Thank you for your contribution! Sashiko AI review found 2 potential issues to consider: - [Medium] arm64: stacktrace: kunwind_next() clears state->regs after fallback - [Low] unwind: arm64: consume…
Thank you for your contribution! Sashiko AI review found 2 potential issues to consider:
- [Medium] arm64: stacktrace: kunwind_next() clears state->regs after fallback
- [Low] unwind: arm64: consume FP stack slot in kunwind_next_regs_sframe()
--
commit e44539400678f653df77eaa983169650892957a4
Author: Dylan Hatch <[email protected]>
unwind: arm64: Use sframe to unwind interrupt frames
This commit adds kunwind_next_regs_sframe() to unwind across exception boundaries using SFrame information, falling back to frame-pointer unwinding and marking the stacktrace unreliable on failure.
Quoted 4 lines
arch/arm64/kernel/stacktrace.c +0
-0
[ ... ]
Quoted 1 line
arch/arm64/kernel/stacktrace.c +0
-0@@ -244,6 +266,163 @@ kunwind_next_frame_record(struct kunwind_state *state)
[ ... ]
Quoted 32 lines
arch/arm64/kernel/stacktrace.c +32
-0+/* +* Consume RA and FP from the stack. The frame record puts FP at a lower +* address than RA, so we read FP first. +*/ +switch (frame.fp.rule) { +case UNWIND_RULE_RETAIN : + fp = state->common.fp; +break ; +/* +* UNWIND_RULE_CFA_OFFSET is currently not used for FP +* (e.g. SFrame cannot represent this rule). +*/ +case UNWIND_RULE_CFA_OFFSET_DEREF : + fp = cfa + frame.fp.offset; +if (!get_word (&state->common, &fp)) +return -EINVAL ; +break ; +/* +* UNWIND_RULE_REG_OFFSET and UNWIND_RULE_REG_OFFSET_DEREF not +* implemented -- flexible FDEs are not currently generated by assembler +* for arm64. +*/ +default : +WARN_ON_ONCE (1 ); +return -EINVAL ; + } + +switch (frame.ra.rule) { +case UNWIND_RULE_RETAIN : + ra = regs->regs[30 ]; + source =KUNWIND_SOURCE_REGS_LR ; +break ;
arch/arm64/kernel/stacktrace.c kunwind_next_frame_record(struct kunwind_state *state) · line 400
Quoted 10 lines
arch/arm64/kernel/stacktrace.c +10
-0+ +/* +* UNWIND_RULE_CFA_OFFSET doesn't make sense for RA. +* The return address cannot legitimately be a stack address. +*/ +case UNWIND_RULE_CFA_OFFSET_DEREF : + ra = cfa + frame.ra.offset; +if (get_consume_word (&state->common, &ra)) +return -EINVAL ; +break ;
[Severity: Low]
Does UNWIND_RULE_CFA_OFFSET_DEREF for frame.fp.rule miss calling unwind_consume_stack() when frame.ra.rule is UNWIND_RULE_RETAIN (or frame.fp.offset >= frame.ra.offset)?
get_word() reads the saved FP from the stack without consuming it, relying on get_consume_word() for frame.ra to advance state->common.stack.low. If RA is retained in LR, the FP slot is never consumed, leaving state->common.stack unadvanced (or failing to transition stacks if CFA + frame.fp.offset lies on the previous stack) while state->unreliable stays false. Note that the stack should only be consumed after both frame.fp and frame.ra checks succeed so state->common.stack remains intact if kunwind_next_regs_sframe() fails and falls back to kunwind_next_frame_record().
[ ... ]
Quoted 7 lines
arch/arm64/kernel/stacktrace.c +3
-0@@ -259,10 +438,23 @@ kunwind_next(struct kunwind_state *state) state->flags.all =0 ;switch (state->source) { +case KUNWIND_SOURCE_REGS_PC : +if (state->regs) + err =kunwind_next_regs_sframe (state);
arch/arm64/kernel/stacktrace.c kunwind_next(struct kunwind_state *state) · line 453
Quoted 10 lines
arch/arm64/kernel/stacktrace.c +10
-0+else + err = -EINVAL ; + +if (err) { +/* Fallback to FP based unwinder */ + err =kunwind_next_frame_record (state); + state->unreliable = true; + } + state->regs =NULL ; +break ;
[Severity: Medium]
Should state->regs be cleared before calling the fallback kunwind_next_frame_record(state)?
When kunwind_next() falls back to kunwind_next_frame_record() (for example on a synthetic pt_regs or an exception in entry.S without SFrame data) and state->common.fp points directly to an on-stack FRAME_META_TYPE_PT_REGS record, kunwind_next_frame_record() calls kunwind_next_regs_pc(), which sets state->regs = regs and state->source = KUNWIND_SOURCE_REGS_PC for that next exception frame. Executing state->regs = NULL afterwards overwrites that new pointer, preventing SFrame unwinding on the next exception boundary.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=11