unwind, arm64: add sframe unwinder for kernel
cover unwind, arm64: add sframe unwinder for kernel
This patchset implements a generic kernel sframe-based [1] unwinder. The main goal is to support reliable stacktraces on arm64.
On x86 orc unwinder provides reliable stacktraces. But arm64 misses the required support from objtool: it cannot generate orc unwind tables for arm64.
Currently, there's already a sframe unwinder proposed for userspace: [2]. Since the sframe unwind table algorithm is similar, these two proposal could integrate common functionality in the future.
There are some incomplete features or challenges:
- The unwinder doesn't yet work with kernel modules. The start_addr of FRE from kernel modules doesn't appear correct, preventing us from unwinding functions from kernel modules.
- Currently, only GCC supports sframe.
Ref:
[1]: https://sourceware.org/binutils/docs/sframe-spec.html
[2]: https://lore.kernel.org/lkml/[email protected]/
Madhavan T. Venkataraman (1):
arm64: Define TIF_PATCH_PENDING for livepatch
Weinan Liu (7):
unwind: build kernel with sframe info
arm64: entry: add unwind info for various kernel entries
unwind: add sframe v2 header
unwind: Implement generic sframe unwinder library
unwind: arm64: Add sframe unwinder on arm64
unwind: arm64: add reliable stacktrace support for arm64
arm64: Enable livepatch for ARM64
Makefile | 6 + arch/Kconfig | 8 + arch/arm64/Kconfig | 3 + arch/arm64/Kconfig.debug | 10 + arch/arm64/include/asm/stacktrace/common.h | 6 + arch/arm64/include/asm/thread_info.h | 4 +- arch/arm64/kernel/entry-common.c | 4 + arch/arm64/kernel/entry.S | 10 + arch/arm64/kernel/setup.c | 2 + arch/arm64/kernel/stacktrace.c | 102 ++++++++++ include/asm-generic/vmlinux.lds.h | 12 ++ include/linux/sframe_lookup.h | 43 +++++ kernel/Makefile | 1 + kernel/sframe.h | 215 +++++++++++++++++++++ kernel/sframe_lookup.c | 196 +++++++++++++++++++ 15 files changed, 621 insertions(+), 1 deletion(-) create mode 100644 include/linux/sframe_lookup.h create mode 100644 kernel/sframe.h create mode 100644 kernel/sframe_lookup.c
Signature · 1 line
2.48.1.262.g85cc9f2d1e-goog
PATCH 1/8 unwind: build kernel with sframe info
Use the -Wa,--gsframe flags to build the code, so GAS will generate a new .sframe section for the stack trace information. Currently, the sframe format only supports arm64 and x86_64 architectures. Add this configuration on arm64 to enable sframe unwinder in the future.
--- Makefile | 6 ++++++ arch/Kconfig | 8 ++++++++ arch/arm64/Kconfig.debug | 10 ++++++++++ include/asm-generic/vmlinux.lds.h | 12 ++++++++++++ 4 files changed, 36 insertions(+)
Makefile +6 -0
@@ -1064,6 +1064,12 @@ ifdef CONFIG_CC_IS_GCC KBUILD_CFLAGS += -fconserve-stack endif +# build with sframe table +ifdef CONFIG_SFRAME_UNWIND_TABLE +KBUILD_CFLAGS += -Wa,--gsframe +KBUILD_AFLAGS += -Wa,--gsframe +endif + # change __FILE__ to the relative path to the source directory ifdef building_out_of_srctree KBUILD_CPPFLAGS += $(call cc-option,-fmacro-prefix-map=$(srcroot)/=)
arch/Kconfig +8 -0
@@ -1736,4 +1736,12 @@ config ARCH_WANTS_PRE_LINK_VMLINUX An architecture can select this if it provides arch/<arch>/tools/Makefile with .arch.vmlinux.o target to be linked into vmlinux. +config AS_HAS_SFRAME_SUPPORT + # Detect availability of the AS option -Wa,--gsframe for generating + # sframe unwind table. + def_bool $(cc-option,-Wa$(comma)--gsframe) + +config SFRAME_UNWIND_TABLE + bool + endmenu
arch/arm64/Kconfig.debug +10 -0
@@ -20,4 +20,14 @@ config ARM64_RELOC_TEST depends on m tristate "Relocation testing module" +config SFRAME_UNWINDER + bool "Sframe unwinder" + depends on AS_HAS_SFRAME_SUPPORT + depends on 64BIT + select SFRAME_UNWIND_TABLE + help + This option enables the sframe (Simple Frame) unwinder for unwinding + kernel stack traces. It uses unwind table that is direclty generated + by toolchain based on DWARF CFI information + source "drivers/hwtracing/coresight/Kconfig"
include/asm-generic/vmlinux.lds.h +12 -0
@@ -469,6 +469,8 @@ defined(CONFIG_AUTOFDO_CLANG) || defined(CONFIG_PROPELLER_CLANG) *(.rodata1) \ } \ \ + SFRAME \ + \ /* PCI quirks */ \ .pci_fixup : AT(ADDR(.pci_fixup) - LOAD_OFFSET) { \ BOUNDED_SECTION_PRE_LABEL(.pci_fixup_early, _pci_fixups_early, __start, __end) \
@@ -886,6 +888,16 @@ defined(CONFIG_AUTOFDO_CLANG) || defined(CONFIG_PROPELLER_CLANG)#define TRACEDATA #endif +#ifdef CONFIG_SFRAME_UNWIND_TABLE +#define SFRAME \ +/* sframe */ \ + .sframe : AT(ADDR(.sframe) - LOAD_OFFSET) { \ + BOUNDED_SECTION_BY(.sframe, _sframe_header) \ + } +#else +#define SFRAME +#endif +#ifdef CONFIG_PRINTK_INDEX #define PRINTK_INDEX \ .printk_index : AT(ADDR(.printk_index) - LOAD_OFFSET) { \
Signature · 1 line
2.48.1.262.g85cc9f2d1e-goog
Re: [PATCH 1/8] unwind: build kernel with sframe info
Quoted 87 lines · Weinan Liu
Use the -Wa,--gsframe flags to build the code, so GAS will generate a new .sframe section for the stack trace information. Currently, the sframe format only supports arm64 and
x86_64architectures. Add this configuration on arm64 to enable sframe unwinder in the future.Signed-off-by Weinan Liu <[email protected]>--- Makefile | 6 ++++++ arch/Kconfig | 8 ++++++++ arch/arm64/Kconfig.debug | 10 ++++++++++ include/asm-generic/vmlinux.lds.h | 12 ++++++++++++ 4 files changed, 36 insertions(+)Makefile +6
-0@@ -1064,6 +1064,12 @@ ifdef CONFIG_CC_IS_GCC KBUILD_CFLAGS += -fconserve-stack endif +# build with sframe table +ifdef CONFIG_SFRAME_UNWIND_TABLE +KBUILD_CFLAGS += -Wa,--gsframe +KBUILD_AFLAGS += -Wa,--gsframe +endif + # change __FILE__ to the relative path to the source directory ifdef building_out_of_srctree KBUILD_CPPFLAGS += $(call cc-option,-fmacro-prefix-map=$(srcroot)/=)arch/Kconfig +8
-0@@ -1736,4 +1736,12 @@ config ARCH_WANTS_PRE_LINK_VMLINUX An architecture can select this if it provides arch/<arch>/tools/Makefile with .arch.vmlinux.o target to be linked into vmlinux. +config AS_HAS_SFRAME_SUPPORT + # Detect availability of the AS option -Wa,--gsframe for generating + # sframe unwind table. + def_bool $(cc-option,-Wa$(comma)--gsframe) + +config SFRAME_UNWIND_TABLE + bool + endmenuarch/arm64/Kconfig.debug +10
-0@@ -20,4 +20,14 @@ config ARM64_RELOC_TEST depends on m tristate "Relocation testing module" +config SFRAME_UNWINDER + bool "Sframe unwinder" + depends on AS_HAS_SFRAME_SUPPORT + depends on 64BIT + select SFRAME_UNWIND_TABLE + help + This option enables the sframe (Simple Frame) unwinder for unwinding + kernel stack traces. It uses unwind table that is direclty generated + by toolchain based on DWARF CFI information + source "drivers/hwtracing/coresight/Kconfig"include/asm-generic/vmlinux.lds.h +5
-0@@ -469,6 +469,8 @@ defined(CONFIG_AUTOFDO_CLANG) || defined(CONFIG_PROPELLER_CLANG) *(.rodata1) \ } \ \ + SFRAME \ + \ /* PCI quirks */ \ .pci_fixup : AT(ADDR(.pci_fixup) - LOAD_OFFSET) { \ BOUNDED_SECTION_PRE_LABEL(.pci_fixup_early, _pci_fixups_early, __start, __end) \@@ -886,6 +888,16 @@ defined(CONFIG_AUTOFDO_CLANG) || defined(CONFIG_PROPELLER_CLANG)#define TRACEDATA #endif +#ifdef CONFIG_SFRAME_UNWIND_TABLE +#define SFRAME \ +/* sframe */ \
include/asm-generic/vmlinux.lds.h defined(CONFIG_AUTOFDO_CLANG) || defined(CONFIG_PROPELLER_CLANG) · line 903
Quoted 10 lines
include/asm-generic/vmlinux.lds.h +7
-0+ .sframe : AT(ADDR(.sframe) - LOAD_OFFSET) { \ + BOUNDED_SECTION_BY(.sframe, _sframe_header) \ + } +#else +#define SFRAME +#endif +#ifdef CONFIG_PRINTK_INDEX #define PRINTK_INDEX \ .printk_index : AT(ADDR(.printk_index) - LOAD_OFFSET) { \
Looks good to me.
Re: [PATCH 1/8] unwind: build kernel with sframe info
In reply to Weinan LiuQuoted 36 lines · Weinan Liu
Use the -Wa,--gsframe flags to build the code, so GAS will generate a new .sframe section for the stack trace information. Currently, the sframe format only supports arm64 and
x86_64architectures. Add this configuration on arm64 to enable sframe unwinder in the future.Signed-off-by Weinan Liu <[email protected]>--- Makefile | 6 ++++++ arch/Kconfig | 8 ++++++++ arch/arm64/Kconfig.debug | 10 ++++++++++ include/asm-generic/vmlinux.lds.h | 12 ++++++++++++ 4 files changed, 36 insertions(+)Makefile +6
-0@@ -1064,6 +1064,12 @@ ifdef CONFIG_CC_IS_GCC KBUILD_CFLAGS += -fconserve-stack endif +# build with sframe table +ifdef CONFIG_SFRAME_UNWIND_TABLE +KBUILD_CFLAGS += -Wa,--gsframe +KBUILD_AFLAGS += -Wa,--gsframe +endif + # change __FILE__ to the relative path to the source directory ifdef building_out_of_srctree KBUILD_CPPFLAGS += $(call cc-option,-fmacro-prefix-map=$(srcroot)/=)arch/Kconfig +0
-0
arch/Kconfig config ARCH_WANTS_PRE_LINK_VMLINUX · line 1743
Quoted 8 lines
arch/Kconfig +5
-0@@ -1736,4 +1736,12 @@ config ARCH_WANTS_PRE_LINK_VMLINUX An architecture can select this if it provides arch/<arch>/tools/Makefile with .arch.vmlinux.o target to be linked into vmlinux. +config AS_HAS_SFRAME_SUPPORT + # Detect availability of the AS option -Wa,--gsframe for generating + # sframe unwind table. + def_bool $(cc-option,-Wa$(comma)--gsframe) +
Since the version of an admissible SFrame section needs to be atleast SFRAME_VERSION_2, it will make sense to include SFrame version check when detecting compatible toolchain.
Quoted 53 lines
arch/Kconfig +3
-0+config SFRAME_UNWIND_TABLE + bool + endmenuarch/arm64/Kconfig.debug +10
-0@@ -20,4 +20,14 @@ config ARM64_RELOC_TEST depends on m tristate "Relocation testing module" +config SFRAME_UNWINDER + bool "Sframe unwinder" + depends on AS_HAS_SFRAME_SUPPORT + depends on 64BIT + select SFRAME_UNWIND_TABLE + help + This option enables the sframe (Simple Frame) unwinder for unwinding + kernel stack traces. It uses unwind table that is direclty generated + by toolchain based on DWARF CFI information + source "drivers/hwtracing/coresight/Kconfig"include/asm-generic/vmlinux.lds.h +12
-0@@ -469,6 +469,8 @@ defined(CONFIG_AUTOFDO_CLANG) || defined(CONFIG_PROPELLER_CLANG) *(.rodata1) \ } \ \ + SFRAME \ + \ /* PCI quirks */ \ .pci_fixup : AT(ADDR(.pci_fixup) - LOAD_OFFSET) { \ BOUNDED_SECTION_PRE_LABEL(.pci_fixup_early, _pci_fixups_early, __start, __end) \@@ -886,6 +888,16 @@ defined(CONFIG_AUTOFDO_CLANG) || defined(CONFIG_PROPELLER_CLANG)#define TRACEDATA #endif +#ifdef CONFIG_SFRAME_UNWIND_TABLE +#define SFRAME \ +/* sframe */ \ + .sframe : AT(ADDR(.sframe) - LOAD_OFFSET) { \ + BOUNDED_SECTION_BY(.sframe, _sframe_header) \ + } +#else +#define SFRAME +#endif +#ifdef CONFIG_PRINTK_INDEX #define PRINTK_INDEX \ .printk_index : AT(ADDR(.printk_index) - LOAD_OFFSET) { \
Josh Poimboeuf
maintainerRe: [PATCH 1/8] unwind: build kernel with sframe info
Quoted 5 lines · Indu Bhagat
Quoted 5 lines
+++ b/arch/Kconfig
arch/Kconfig +1
-0@@ -1736,4 +1736,12 @@ config ARCH_WANTS_PRE_LINK_VMLINUX An architecture can select this if it provides arch/<arch>/tools/Makefile with .arch.vmlinux.o target to be linked into vmlinux. +config AS_HAS_SFRAME_SUPPORT
Quoted 8 lines
arch/Kconfig config ARCH_WANTS_PRE_LINK_VMLINUX · line 1743
Quoted 4 lines
arch/Kconfig +4
-0+ # Detect availability of the AS option -Wa,--gsframe for generating + # sframe unwind table. + def_bool $(cc-option,-Wa$(comma)--gsframe) +Since the version of an admissible SFrame section needs to be atleast
SFRAME_VERSION_2, it will make sense to include SFrame version check when detecting compatible toolchain.
Also, Jens pointed out that checking for '-Wa,--gsframe' isn't sufficient, as it still succeeds for architectures that aren't yet supported:
https://lore.kernel.org/[email protected]
so in my patches I have:
config AS_SFRAME
def_bool $(as-instr,.cfi_sections .sframe\n.cfi_startproc\n.cfi_endproc)
Though I'm not sure how you would add an SFrame version check to that.
Signature · 1 line
Josh
PATCH 2/8 arm64: entry: add unwind info for various kernel entries
In reply to Weinan LiuDWARF CFI (Call Frame Information) specifies how to recover the return address and callee-saved registers at each PC in a given function. Compilers are able to generate the CFI annotations when they compile the code to assembly language. For handcrafted assembly, we need to annotate them by hand.
Annotate CFI unwind info for assembly for interrupt and exception handlers.
--- arch/arm64/kernel/entry.S | 10 ++++++++++ 1 file changed, 10 insertions(+)
arch/arm64/kernel/entry.S +10 -0
@@ -579,7 +579,12 @@ SYM_CODE_START_LOCAL(el\el\ht\()_\regsize\()_\label) .if \el == 0 b ret_to_user .else + .cfi_startproc + .cfi_def_cfa_offset PT_REGS_SIZE + .cfi_offset 29, S_FP - PT_REGS_SIZE + .cfi_offset 30, S_LR - PT_REGS_SIZE b ret_to_kernel + .cfi_endproc .endif SYM_CODE_END(el\el\ht\()_\regsize\()_\label) .endm
@@ -889,6 +894,10 @@ SYM_FUNC_START(call_on_irq_stack) /* Move to the new stack and call the function there */ add sp, x16, #IRQ_STACK_SIZE blr x1 + .cfi_startproc + .cfi_def_cfa 29, 16 + .cfi_offset 29, -16 + .cfi_offset 30, -8 /* * Restore the SP from the FP, and restore the FP and LR from the frame
@@ -898,6 +907,7 @@ SYM_FUNC_START(call_on_irq_stack) ldp x29, x30, [sp], #16 scs_load_current ret + .cfi_endproc SYM_FUNC_END(call_on_irq_stack) NOKPROBE(call_on_irq_stack)
Signature · 1 line
2.48.1.262.g85cc9f2d1e-goog
PATCH 3/8 unwind: add sframe v2 header
In reply to Weinan LiuAdd sframe header so that we know how to access the sframe section generated by compilers.
This is the sframe header file borrowed from the patchset [1] Josh Poimboeuf according to sframe v2 spec [2].
[1]: https://lore.kernel.org/lkml/42c0a99236af65c09c8182e260af7bcf5aa1e158.1730150953.git.jpoimboe@kernel.org/
[2]: https://sourceware.org/binutils/docs/sframe-spec.html
--- kernel/sframe.h | 215 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 215 insertions(+) create mode 100644 kernel/sframe.h
kernel/sframe.h new file +215 -0
@@ -0,0 +1,215 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +/* +* Copyright (C) 2023, Oracle and/or its affiliates. +* +* This file contains definitions for the SFrame stack tracing format, which is +* documented at https://sourceware.org/binutils/docs +*/ +#ifndef _SFRAME_H +#define _SFRAME_H + +#include <linux/types.h> + +#define SFRAME_VERSION_1 1 +#define SFRAME_VERSION_2 2 +#define SFRAME_MAGIC 0xdee2 + +/* Function Descriptor Entries are sorted on PC. */ +#define SFRAME_F_FDE_SORTED 0x1 +/* Frame-pointer based stack tracing. Defined, but not set. */ +#define SFRAME_F_FRAME_POINTER 0x2 + +#define SFRAME_CFA_FIXED_FP_INVALID 0 +#define SFRAME_CFA_FIXED_RA_INVALID 0 + +/* Supported ABIs/Arch. */ +#define SFRAME_ABI_AARCH64_ENDIAN_BIG 1/* AARCH64 big endian. */ +#define SFRAME_ABI_AARCH64_ENDIAN_LITTLE 2/* AARCH64 little endian. */ +#define SFRAME_ABI_AMD64_ENDIAN_LITTLE 3/* AMD64 little endian. */ + +/* SFrame FRE types. */ +#define SFRAME_FRE_TYPE_ADDR1 0 +#define SFRAME_FRE_TYPE_ADDR2 1 +#define SFRAME_FRE_TYPE_ADDR4 2 + +/* +* SFrame Function Descriptor Entry types. +* +* The SFrame format has two possible representations for functions. The +* choice of which type to use is made according to the instruction patterns +* in the relevant program stub. +*/ + +/* Unwinders perform a (PC >= FRE_START_ADDR) to look up a matching FRE. */ +#define SFRAME_FDE_TYPE_PCINC 0 +/* +* Unwinders perform a (PC & FRE_START_ADDR_AS_MASK >= FRE_START_ADDR_AS_MASK) +* to look up a matching FRE. Typical usecases are pltN entries, trampolines +* etc. +*/ +#define SFRAME_FDE_TYPE_PCMASK 1 + +/** +* struct sframe_preamble - SFrame Preamble. +* @magic: Magic number (SFRAME_MAGIC). +* @version: Format version number (SFRAME_VERSION). +* @flags: Various flags. +*/ +struct sframe_preamble { +u16 magic; +u8 version; +u8 flags; +} __packed; + +/** +* struct sframe_header - SFrame Header. +* @preamble: SFrame preamble. +* @abi_arch: Identify the arch (including endianness) and ABI. +* @cfa_fixed_fp_offset: Offset for the Frame Pointer (FP) from CFA may be +* fixed for some ABIs ((e.g, in AMD64 when -fno-omit-frame-pointer is +* used). When fixed, this field specifies the fixed stack frame offset +* and the individual FREs do not need to track it. When not fixed, it +* is set to SFRAME_CFA_FIXED_FP_INVALID, and the individual FREs may +* provide the applicable stack frame offset, if any. +* @cfa_fixed_ra_offset: Offset for the Return Address from CFA is fixed for +* some ABIs. When fixed, this field specifies the fixed stack frame +* offset and the individual FREs do not need to track it. When not +* fixed, it is set to SFRAME_CFA_FIXED_FP_INVALID. +* @auxhdr_len: Number of bytes making up the auxiliary header, if any. +* Some ABI/arch, in the future, may use this space for extending the +* information in SFrame header. Auxiliary header is contained in bytes +* sequentially following the sframe_header. +* @num_fdes: Number of SFrame FDEs in this SFrame section. +* @num_fres: Number of SFrame Frame Row Entries. +* @fre_len: Number of bytes in the SFrame Frame Row Entry section. +* @fdes_off: Offset of SFrame Function Descriptor Entry section. +* @fres_off: Offset of SFrame Frame Row Entry section. +*/ +struct sframe_header { +struct sframe_preamble preamble; +u8 abi_arch; +s8 cfa_fixed_fp_offset; +s8 cfa_fixed_ra_offset; +u8 auxhdr_len; +u32 num_fdes; +u32 num_fres; +u32 fre_len; +u32 fdes_off; +u32 fres_off; +} __packed; + +#define SFRAME_HDR_SIZE (sframe_hdr) \ + ((sizeof(struct sframe_header) + (sframe_hdr).auxhdr_len)) + +/* Two possible keys for executable (instruction) pointers signing. */ +#define SFRAME_AARCH64_PAUTH_KEY_A 0/* Key A. */ +#define SFRAME_AARCH64_PAUTH_KEY_B 1/* Key B. */ + +/** +* struct sframe_fde - SFrame Function Descriptor Entry. +* @start_addr: Function start address. Encoded as a signed offset, +* relative to the current FDE. +* @size: Size of the function in bytes. +* @fres_off: Offset of the first SFrame Frame Row Entry of the function, +* relative to the beginning of the SFrame Frame Row Entry sub-section. +* @fres_num: Number of frame row entries for the function. +* @info: Additional information for deciphering the stack trace +* information for the function. Contains information about SFrame FRE +* type, SFrame FDE type, PAC authorization A/B key, etc. +* @rep_size: Block size for SFRAME_FDE_TYPE_PCMASK +* @padding: Unused +*/ +struct sframe_fde { +s32 start_addr; +u32 size; +u32 fres_off; +u32 fres_num; +u8 info; +u8 rep_size; +u16 padding; +} __packed; + +/* +* 'func_info' in SFrame FDE contains additional information for deciphering +* the stack trace information for the function. In V1, the information is +* organized as follows: +* - 4-bits: Identify the FRE type used for the function. +* - 1-bit: Identify the FDE type of the function - mask or inc. +* - 1-bit: PAC authorization A/B key (aarch64). +* - 2-bits: Unused. +* --------------------------------------------------------------------- +* | Unused | PAC auth A/B key (aarch64) | FDE type | FRE type | +* | | Unused (amd64) | | | +* --------------------------------------------------------------------- +* 8 6 5 4 0 +*/ + +/* Note: Set PAC auth key to SFRAME_AARCH64_PAUTH_KEY_A by default. */ +#define SFRAME_FUNC_INFO (fde_type, fre_enc_type) \ + (((SFRAME_AARCH64_PAUTH_KEY_A & 0x1) << 5) | \ + (((fde_type) & 0x1) << 4) | ((fre_enc_type) & 0xf)) + +#define SFRAME_FUNC_FRE_TYPE (data) ((data) & 0xf) +#define SFRAME_FUNC_FDE_TYPE (data) (((data) >> 4) & 0x1) +#define SFRAME_FUNC_PAUTH_KEY (data) (((data) >> 5) & 0x1) + +/* +* Size of stack frame offsets in an SFrame Frame Row Entry. A single +* SFrame FRE has all offsets of the same size. Offset size may vary +* across frame row entries. +*/ +#define SFRAME_FRE_OFFSET_1B 0 +#define SFRAME_FRE_OFFSET_2B 1 +#define SFRAME_FRE_OFFSET_4B 2 + +/* An SFrame Frame Row Entry can be SP or FP based. */ +#define SFRAME_BASE_REG_FP 0 +#define SFRAME_BASE_REG_SP 1 + +/* +* The index at which a specific offset is presented in the variable length +* bytes of an FRE. +*/ +#define SFRAME_FRE_CFA_OFFSET_IDX 0 +/* +* The RA stack offset, if present, will always be at index 1 in the variable +* length bytes of the FRE. +*/ +#define SFRAME_FRE_RA_OFFSET_IDX 1 +/* +* The FP stack offset may appear at offset 1 or 2, depending on the ABI as RA +* may or may not be tracked. +*/ +#define SFRAME_FRE_FP_OFFSET_IDX 2 + +/* +* 'fre_info' in SFrame FRE contains information about: +* - 1 bit: base reg for CFA +* - 4 bits: Number of offsets (N). A value of up to 3 is allowed to track +* all three of CFA, FP and RA (fixed implicit order). +* - 2 bits: information about size of the offsets (S) in bytes. +* Valid values are SFRAME_FRE_OFFSET_1B, SFRAME_FRE_OFFSET_2B, +* SFRAME_FRE_OFFSET_4B +* - 1 bit: Mangled RA state bit (aarch64 only). +* --------------------------------------------------------------- +* | Mangled-RA (aarch64) | Size of | Number of | base_reg | +* | Unused (amd64) | offsets | offsets | | +* --------------------------------------------------------------- +* 8 7 5 1 0 +*/ + +/* Note: Set mangled_ra_p to zero by default. */ +#define SFRAME_FRE_INFO (base_reg_id, offset_num, offset_size) \ + (((0 & 0x1) << 7) | (((offset_size) & 0x3) << 5) | \ + (((offset_num) & 0xf) << 1) | ((base_reg_id) & 0x1)) + +/* Set the mangled_ra_p bit as indicated. */ +#define SFRAME_FRE_INFO_UPDATE_MANGLED_RA_P (mangled_ra_p, fre_info) \ + ((((mangled_ra_p) & 0x1) << 7) | ((fre_info) & 0x7f)) + +#define SFRAME_FRE_CFA_BASE_REG_ID (data) ((data) & 0x1) +#define SFRAME_FRE_OFFSET_COUNT (data) (((data) >> 1) & 0xf) +#define SFRAME_FRE_OFFSET_SIZE (data) (((data) >> 5) & 0x3) +#define SFRAME_FRE_MANGLED_RA_P (data) (((data) >> 7) & 0x1) + +#endif /* _SFRAME_H */
Signature · 1 line
2.48.1.262.g85cc9f2d1e-goog
Re: [PATCH 3/8] unwind: add sframe v2 header
Quoted 226 lines · Weinan Liu
Add sframe header so that we know how to access the sframe section generated by compilers.
This is the sframe header file borrowed from the patchset [1] Josh Poimboeuf according to sframe v2 spec [2].
[1]: https://lore.kernel.org/lkml/42c0a99236af65c09c8182e260af7bcf5aa1e158.1730150953.git.jpoimboe@kernel.org/
[2]: https://sourceware.org/binutils/docs/sframe-spec.htmlSigned-off-by Weinan Liu <[email protected]>--- kernel/sframe.h | 215 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 215 insertions(+)create mode 100644
kernel/sframe.hkernel/sframe.h new file +205
-0@@ -0,0 +1,215 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +/* +* Copyright (C) 2023, Oracle and/or its affiliates. +* +* This file contains definitions for the SFrame stack tracing format, which is +* documented at https://sourceware.org/binutils/docs +*/ +#ifndef _SFRAME_H +#define _SFRAME_H + +#include <linux/types.h> + +#define SFRAME_VERSION_1 1 +#define SFRAME_VERSION_2 2 +#define SFRAME_MAGIC 0xdee2 + +/* Function Descriptor Entries are sorted on PC. */ +#define SFRAME_F_FDE_SORTED 0x1 +/* Frame-pointer based stack tracing. Defined, but not set. */ +#define SFRAME_F_FRAME_POINTER 0x2 + +#define SFRAME_CFA_FIXED_FP_INVALID 0 +#define SFRAME_CFA_FIXED_RA_INVALID 0 + +/* Supported ABIs/Arch. */ +#define SFRAME_ABI_AARCH64_ENDIAN_BIG 1/* AARCH64 big endian. */ +#define SFRAME_ABI_AARCH64_ENDIAN_LITTLE 2/* AARCH64 little endian. */ +#define SFRAME_ABI_AMD64_ENDIAN_LITTLE 3/* AMD64 little endian. */ + +/* SFrame FRE types. */ +#define SFRAME_FRE_TYPE_ADDR1 0 +#define SFRAME_FRE_TYPE_ADDR2 1 +#define SFRAME_FRE_TYPE_ADDR4 2 + +/* +* SFrame Function Descriptor Entry types. +* +* The SFrame format has two possible representations for functions. The +* choice of which type to use is made according to the instruction patterns +* in the relevant program stub. +*/ + +/* Unwinders perform a (PC >= FRE_START_ADDR) to look up a matching FRE. */ +#define SFRAME_FDE_TYPE_PCINC 0 +/* +* Unwinders perform a (PC & FRE_START_ADDR_AS_MASK >= FRE_START_ADDR_AS_MASK) +* to look up a matching FRE. Typical usecases are pltN entries, trampolines +* etc. +*/ +#define SFRAME_FDE_TYPE_PCMASK 1 + +/** +* struct sframe_preamble - SFrame Preamble. +* @magic: Magic number (SFRAME_MAGIC). +* @version: Format version number (SFRAME_VERSION). +* @flags: Various flags. +*/ +struct sframe_preamble { +u16 magic; +u8 version; +u8 flags; +} __packed; + +/** +* struct sframe_header - SFrame Header. +* @preamble: SFrame preamble. +* @abi_arch: Identify the arch (including endianness) and ABI. +* @cfa_fixed_fp_offset: Offset for the Frame Pointer (FP) from CFA may be +* fixed for some ABIs ((e.g, in AMD64 when -fno-omit-frame-pointer is +* used). When fixed, this field specifies the fixed stack frame offset +* and the individual FREs do not need to track it. When not fixed, it +* is set to SFRAME_CFA_FIXED_FP_INVALID, and the individual FREs may +* provide the applicable stack frame offset, if any. +* @cfa_fixed_ra_offset: Offset for the Return Address from CFA is fixed for +* some ABIs. When fixed, this field specifies the fixed stack frame +* offset and the individual FREs do not need to track it. When not +* fixed, it is set to SFRAME_CFA_FIXED_FP_INVALID. +* @auxhdr_len: Number of bytes making up the auxiliary header, if any. +* Some ABI/arch, in the future, may use this space for extending the +* information in SFrame header. Auxiliary header is contained in bytes +* sequentially following the sframe_header. +* @num_fdes: Number of SFrame FDEs in this SFrame section. +* @num_fres: Number of SFrame Frame Row Entries. +* @fre_len: Number of bytes in the SFrame Frame Row Entry section. +* @fdes_off: Offset of SFrame Function Descriptor Entry section. +* @fres_off: Offset of SFrame Frame Row Entry section. +*/ +struct sframe_header { +struct sframe_preamble preamble; +u8 abi_arch; +s8 cfa_fixed_fp_offset; +s8 cfa_fixed_ra_offset; +u8 auxhdr_len; +u32 num_fdes; +u32 num_fres; +u32 fre_len; +u32 fdes_off; +u32 fres_off; +} __packed; + +#define SFRAME_HDR_SIZE (sframe_hdr) \ + ((sizeof(struct sframe_header) + (sframe_hdr).auxhdr_len)) + +/* Two possible keys for executable (instruction) pointers signing. */ +#define SFRAME_AARCH64_PAUTH_KEY_A 0/* Key A. */ +#define SFRAME_AARCH64_PAUTH_KEY_B 1/* Key B. */ + +/** +* struct sframe_fde - SFrame Function Descriptor Entry. +* @start_addr: Function start address. Encoded as a signed offset, +* relative to the current FDE. +* @size: Size of the function in bytes. +* @fres_off: Offset of the first SFrame Frame Row Entry of the function, +* relative to the beginning of the SFrame Frame Row Entry sub-section. +* @fres_num: Number of frame row entries for the function. +* @info: Additional information for deciphering the stack trace +* information for the function. Contains information about SFrame FRE +* type, SFrame FDE type, PAC authorization A/B key, etc. +* @rep_size: Block size for SFRAME_FDE_TYPE_PCMASK +* @padding: Unused +*/ +struct sframe_fde { +s32 start_addr; +u32 size; +u32 fres_off; +u32 fres_num; +u8 info; +u8 rep_size; +u16 padding; +} __packed; + +/* +* 'func_info' in SFrame FDE contains additional information for deciphering +* the stack trace information for the function. In V1, the information is +* organized as follows: +* - 4-bits: Identify the FRE type used for the function. +* - 1-bit: Identify the FDE type of the function - mask or inc. +* - 1-bit: PAC authorization A/B key (aarch64). +* - 2-bits: Unused. +* --------------------------------------------------------------------- +* | Unused | PAC auth A/B key (aarch64) | FDE type | FRE type | +* | | Unused (amd64) | | | +* --------------------------------------------------------------------- +* 8 6 5 4 0 +*/ + +/* Note: Set PAC auth key to SFRAME_AARCH64_PAUTH_KEY_A by default. */ +#define SFRAME_FUNC_INFO (fde_type, fre_enc_type) \ + (((SFRAME_AARCH64_PAUTH_KEY_A & 0x1) << 5) | \ + (((fde_type) & 0x1) << 4) | ((fre_enc_type) & 0xf)) + +#define SFRAME_FUNC_FRE_TYPE (data) ((data) & 0xf) +#define SFRAME_FUNC_FDE_TYPE (data) (((data) >> 4) & 0x1) +#define SFRAME_FUNC_PAUTH_KEY (data) (((data) >> 5) & 0x1) + +/* +* Size of stack frame offsets in an SFrame Frame Row Entry. A single +* SFrame FRE has all offsets of the same size. Offset size may vary +* across frame row entries. +*/ +#define SFRAME_FRE_OFFSET_1B 0 +#define SFRAME_FRE_OFFSET_2B 1 +#define SFRAME_FRE_OFFSET_4B 2 + +/* An SFrame Frame Row Entry can be SP or FP based. */ +#define SFRAME_BASE_REG_FP 0 +#define SFRAME_BASE_REG_SP 1 + +/* +* The index at which a specific offset is presented in the variable length +* bytes of an FRE. +*/ +#define SFRAME_FRE_CFA_OFFSET_IDX 0 +/* +* The RA stack offset, if present, will always be at index 1 in the variable +* length bytes of the FRE. +*/ +#define SFRAME_FRE_RA_OFFSET_IDX 1 +/* +* The FP stack offset may appear at offset 1 or 2, depending on the ABI as RA +* may or may not be tracked. +*/ +#define SFRAME_FRE_FP_OFFSET_IDX 2 + +/* +* 'fre_info' in SFrame FRE contains information about: +* - 1 bit: base reg for CFA +* - 4 bits: Number of offsets (N). A value of up to 3 is allowed to track +* all three of CFA, FP and RA (fixed implicit order). +* - 2 bits: information about size of the offsets (S) in bytes. +* Valid values are SFRAME_FRE_OFFSET_1B, SFRAME_FRE_OFFSET_2B, +* SFRAME_FRE_OFFSET_4B +* - 1 bit: Mangled RA state bit (aarch64 only). +* --------------------------------------------------------------- +* | Mangled-RA (aarch64) | Size of | Number of | base_reg | +* | Unused (amd64) | offsets | offsets | | +* --------------------------------------------------------------- +* 8 7 5 1 0 +*/ + +/* Note: Set mangled_ra_p to zero by default. */ +#define SFRAME_FRE_INFO (base_reg_id, offset_num, offset_size) \ + (((0 & 0x1) << 7) | (((offset_size) & 0x3) << 5) | \ + (((offset_num) & 0xf) << 1) | ((base_reg_id) & 0x1)) +
kernel/sframe.h line 215
Quoted 10 lines
kernel/sframe.h new file +10
-0+/* Set the mangled_ra_p bit as indicated. */ +#define SFRAME_FRE_INFO_UPDATE_MANGLED_RA_P (mangled_ra_p, fre_info) \ + ((((mangled_ra_p) & 0x1) << 7) | ((fre_info) & 0x7f)) + +#define SFRAME_FRE_CFA_BASE_REG_ID (data) ((data) & 0x1) +#define SFRAME_FRE_OFFSET_COUNT (data) (((data) >> 1) & 0xf) +#define SFRAME_FRE_OFFSET_SIZE (data) (((data) >> 5) & 0x3) +#define SFRAME_FRE_MANGLED_RA_P (data) (((data) >> 7) & 0x1) + +#endif /* _SFRAME_H */
Looks good to me.
Josh Poimboeuf
maintainerRe: [PATCH 3/8] unwind: add sframe v2 header
In reply to Weinan LiuQuoted 8 lines · Weinan Liu
Add sframe header so that we know how to access the sframe section generated by compilers.
This is the sframe header file borrowed from the patchset [1] Josh Poimboeuf according to sframe v2 spec [2].
[1]: https://lore.kernel.org/lkml/42c0a99236af65c09c8182e260af7bcf5aa1e158.1730150953.git.jpoimboe@kernel.org/
[2]: https://sourceware.org/binutils/docs/sframe-spec.html
FWIW, my v4 patchset has a more streamlined version of this file.
https://lore.kernel.org/f27e8463783febfa0dabb0432a3dd6be8ad98412.1737511963.git.jpoimboe@kernel.org
Signature · 1 line
Josh
PATCH 4/8 unwind: Implement generic sframe unwinder library
In reply to Weinan LiuThis change introduces a kernel space unwinder using sframe table for architectures without ORC unwinder support.
The implementation is adapted from Josh's userspace sframe unwinder proposal[1] according to the sframe v2 spec[2].
[1] https://lore.kernel.org/lkml/42c0a99236af65c09c8182e260af7bcf5aa1e158.1730150953.git.jpoimboe@kernel.org/
[2] https://sourceware.org/binutils/docs/sframe-spec.html
--- include/linux/sframe_lookup.h | 43 ++++++++ kernel/Makefile | 1 + kernel/sframe_lookup.c | 196 ++++++++++++++++++++++++++++++++++ 3 files changed, 240 insertions(+) create mode 100644 include/linux/sframe_lookup.h create mode 100644 kernel/sframe_lookup.c
include/linux/sframe_lookup.h new file +43 -0
@@ -0,0 +1,43 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _LINUX_SFRAME_LOOKUP_H +#define _LINUX_SFRAME_LOOKUP_H + +/** +* struct sframe_ip_entry - sframe unwind info for given ip +* @cfa_offset: Offset for the Canonical Frame Address(CFA) from Frame +* Pointer(FP) or Stack Pointer(SP) +* @ra_offset: Offset for the Return Address from CFA. +* @fp_offset: Offset for the Frame Pointer (FP) from CFA. +* @use_fp: Use FP to get next CFA or not +*/ +struct sframe_ip_entry { +int32_t cfa_offset; +int32_t ra_offset; +int32_t fp_offset; +bool use_fp; +}; + +/** +* struct sframe_table - sframe struct of a table +* @sfhdr_p: Pointer to sframe header +* @fde_p: Pointer to the first of sframe frame description entry(FDE). +* @fre_p: Pointer to the first of sframe frame row entry(FRE). +*/ +struct sframe_table { +struct sframe_header *sfhdr_p; +struct sframe_fde *fde_p; +char *fre_p; +}; + +#ifdef CONFIG_SFRAME_UNWINDER +void init_sframe_table (void ); +int sframe_find_pc (unsigned long pc,struct sframe_ip_entry *entry); +#else +static inline void init_sframe_table (void ) {} +static inline int sframe_find_pc (unsigned long pc,struct sframe_ip_entry *entry) +{ +return -EINVAL ; +} +#endif + +#endif /* _LINUX_SFRAME_LOOKUP_H */
kernel/Makefile +1 -0
@@ -131,6 +131,7 @@ obj-$(CONFIG_WATCH_QUEUE) += watch_queue.o obj-$(CONFIG_RESOURCE_KUNIT_TEST) += resource_kunit.o obj-$(CONFIG_SYSCTL_KUNIT_TEST) += sysctl-test.o +obj-$(CONFIG_SFRAME_UNWINDER) += sframe_lookup.o CFLAGS_stackleak.o += $(DISABLE_STACKLEAK_PLUGIN) obj-$(CONFIG_GCC_PLUGIN_STACKLEAK) += stackleak.o
kernel/sframe_lookup.c new file +196 -0
196 lines not on this page · raw message
Signature · 1 line
2.48.1.262.g85cc9f2d1e-goog
Re: [PATCH 4/8] unwind: Implement generic sframe unwinder library
Quoted 29 lines · Weinan Liu
This change introduces a kernel space unwinder using sframe table for architectures without ORC unwinder support.
The implementation is adapted from Josh's userspace sframe unwinder proposal[1] according to the sframe v2 spec[2].
[1] https://lore.kernel.org/lkml/42c0a99236af65c09c8182e260af7bcf5aa1e158.1730150953.git.jpoimboe@kernel.org/
[2] https://sourceware.org/binutils/docs/sframe-spec.htmlSigned-off-by Weinan Liu <[email protected]>--- include/linux/sframe_lookup.h | 43 ++++++++ kernel/Makefile | 1 + kernel/sframe_lookup.c | 196 ++++++++++++++++++++++++++++++++++ 3 files changed, 240 insertions(+)create mode 100644
include/linux/sframe_lookup.hcreate mode 100644kernel/sframe_lookup.cinclude/linux/sframe_lookup.h new file +5
-0@@ -0,0 +1,43 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _LINUX_SFRAME_LOOKUP_H +#define _LINUX_SFRAME_LOOKUP_H + +/**
include/linux/sframe_lookup.h line 15
Quoted 10 lines
include/linux/sframe_lookup.h new file +10
-0+* struct sframe_ip_entry - sframe unwind info for given ip +* @cfa_offset: Offset for the Canonical Frame Address(CFA) from Frame +* Pointer(FP) or Stack Pointer(SP) +* @ra_offset: Offset for the Return Address from CFA. +* @fp_offset: Offset for the Frame Pointer (FP) from CFA. +* @use_fp: Use FP to get next CFA or not +*/ +struct sframe_ip_entry { +int32_t cfa_offset; +int32_t ra_offset;
The ra_offset is not present for x86_64 in SFrame FRE as per the spec. I am wondering whether this struct should change based on the architecture or just set ra_offset calculated from cfa_offset for x86_64.
Quoted 232 lines
include/linux/sframe_lookup.h new file +28
-0+int32_t fp_offset; + bool use_fp; +}; + +/** +* struct sframe_table - sframe struct of a table +* @sfhdr_p: Pointer to sframe header +* @fde_p: Pointer to the first of sframe frame description entry(FDE). +* @fre_p: Pointer to the first of sframe frame row entry(FRE). +*/ +struct sframe_table { +struct sframe_header *sfhdr_p; +struct sframe_fde *fde_p; +char *fre_p; +}; + +#ifdef CONFIG_SFRAME_UNWINDER +void init_sframe_table (void ); +int sframe_find_pc (unsigned long pc,struct sframe_ip_entry *entry); +#else +static inline void init_sframe_table (void ) {} +static inline int sframe_find_pc (unsigned long pc,struct sframe_ip_entry *entry) +{ +return -EINVAL ; +} +#endif + +#endif /* _LINUX_SFRAME_LOOKUP_H */ kernel/Makefile +1
-0@@ -131,6 +131,7 @@ obj-$(CONFIG_WATCH_QUEUE) += watch_queue.o obj-$(CONFIG_RESOURCE_KUNIT_TEST) += resource_kunit.o obj-$(CONFIG_SYSCTL_KUNIT_TEST) += sysctl-test.o +obj-$(CONFIG_SFRAME_UNWINDER) += sframe_lookup.o CFLAGS_stackleak.o += $(DISABLE_STACKLEAK_PLUGIN) obj-$(CONFIG_GCC_PLUGIN_STACKLEAK) += stackleak.okernel/sframe_lookup.c new file +186
-0@@ -0,0 +1,196 @@ +// SPDX-License-Identifier: GPL-2.0-only +#include <linux/module.h> +#include <linux/sort.h> +#include <linux/sframe_lookup.h> +#include <linux/kallsyms.h> +#include "sframe.h" + +#define pr_fmt (fmt) "sframe: " fmt + +extern char __start_sframe_header[]; +extern char __stop_sframe_header[]; + +static bool sframe_init__ro_after_init ; +static struct sframe_table sftbl; + +#define SFRAME_READ_TYPE (in, out, type) \ +({ \ + type __tmp; \ + memcpy(&__tmp, in, sizeof(__tmp)); \ + in += sizeof(__tmp); \ + out = __tmp; \ +}) + +#define SFRAME_READ_ROW_ADDR (in_addr, out_addr, type) \ +({ \ + switch (type) { \ + case SFRAME_FRE_TYPE_ADDR1: \ + SFRAME_READ_TYPE(in_addr, out_addr, u8); \ + break; \ + case SFRAME_FRE_TYPE_ADDR2: \ + SFRAME_READ_TYPE(in_addr, out_addr, u16); \ + break; \ + case SFRAME_FRE_TYPE_ADDR4: \ + SFRAME_READ_TYPE(in_addr, out_addr, u32); \ + break; \ + default: \ + break; \ + } \ +}) + +#define SFRAME_READ_ROW_OFFSETS (in_addr, out_addr, size) \ +({ \ + switch (size) { \ + case 1: \ + SFRAME_READ_TYPE(in_addr, out_addr, s8); \ + break; \ + case 2: \ + SFRAME_READ_TYPE(in_addr, out_addr, s16); \ + break; \ + case 4: \ + SFRAME_READ_TYPE(in_addr, out_addr, s32); \ + break; \ + default: \ + break; \ + } \ +}) + +static struct sframe_fde *find_fde (const struct sframe_table *tbl,unsigned long pc) +{ +int l, r, m, f; +int32_t ip; +struct sframe_fde *fdep; + +if (!tbl || !tbl->sfhdr_p || !tbl->fde_p) +return NULL ; + + ip = (pc - (unsigned long )tbl->sfhdr_p); + +/* Do a binary range search to find the rightmost FDE start_addr < ip */ + l = m = f =0 ; + r = tbl->sfhdr_p->num_fdes; +while (l < r) { + m = l + ((r - l) /2 ); + fdep = tbl->fde_p + m; +if (fdep->start_addr > ip) + r = m; +else + l = m +1 ; + } +/* use l - 1 because l will be the first item fdep->start_addr > ip */ + f = l -1 ; +if (f >= tbl->sfhdr_p->num_fdes || f <0 ) +return NULL ; + fdep = tbl->fde_p + f; +if (ip < fdep->start_addr || ip >= fdep->start_addr + fdep->size) +return NULL ; + +return fdep; +} + +static int find_fre (const struct sframe_table *tbl,unsigned long pc, +const struct sframe_fde *fdep,struct sframe_ip_entry *entry) +{ +int i, offset_size, offset_count; +char *fres, *offsets_loc; +int32_t ip_off; +uint32_t next_row_ip_off; +uint8_t fre_info, fde_type =SFRAME_FUNC_FDE_TYPE (fdep->info), + fre_type =SFRAME_FUNC_FRE_TYPE (fdep->info); + + fres = tbl->fre_p + fdep->fres_off; + +/* Whether PCs in the FREs should be treated as masks or not */ +if (fde_type ==SFRAME_FDE_TYPE_PCMASK ) + ip_off = pc % fdep->rep_size; +else + ip_off = (int32_t )(pc - (unsigned long )tbl->sfhdr_p) - fdep->start_addr; + +if (ip_off <0 || ip_off >= fdep->size) +return -EINVAL ; + +/* +* FRE structure starts by address of the entry with variants length. Use +* two pointers to track current head(fres) and the address of last +* offset(offsets_loc) +*/ +for (i =0 ; i < fdep->fres_num; i++) { +SFRAME_READ_ROW_ADDR (fres, next_row_ip_off, fre_type); +if (ip_off < next_row_ip_off) +break ; +SFRAME_READ_TYPE (fres, fre_info, u8); + offsets_loc = fres; +/* +* jump to the start of next fre +* fres += fre_offets_cnt*offset_size +*/ + fres +=SFRAME_FRE_OFFSET_COUNT (fre_info) <<SFRAME_FRE_OFFSET_SIZE (fre_info); + } + + offset_size =1 <<SFRAME_FRE_OFFSET_SIZE (fre_info); + offset_count =SFRAME_FRE_OFFSET_COUNT (fre_info); + +if (offset_count >0 ) { +SFRAME_READ_ROW_OFFSETS (offsets_loc, entry->cfa_offset, offset_size); + offset_count--; + } +if (offset_count >0 && !entry->ra_offset) { +SFRAME_READ_ROW_OFFSETS (offsets_loc, entry->ra_offset, offset_size); + offset_count--; + } +if (offset_count >0 && !entry->fp_offset) { +SFRAME_READ_ROW_OFFSETS (offsets_loc, entry->fp_offset, offset_size); + offset_count--; + } +if (offset_count) +return -EINVAL ; + + entry->use_fp =SFRAME_FRE_CFA_BASE_REG_ID (fre_info) ==SFRAME_BASE_REG_FP ; + +return 0 ; +} + +int sframe_find_pc (unsigned long pc,struct sframe_ip_entry *entry) +{ +struct sframe_fde *fdep; +struct sframe_table *sftbl_p = &sftbl; +int err; + +if (!sframe_init) +return -EINVAL ; + +memset (entry,0 ,sizeof (*entry)); + entry->ra_offset = sftbl_p->sfhdr_p->cfa_fixed_ra_offset; + entry->fp_offset = sftbl_p->sfhdr_p->cfa_fixed_fp_offset; + + fdep =find_fde (sftbl_p, pc); +if (!fdep) +return -EINVAL ; + err =find_fre (sftbl_p, pc, fdep, entry); +if (err) +return err; + +return 0 ; +} + +void __init init_sframe_table (void ) +{ +size_t sframe_size = (void *)__stop_sframe_header - (void *)__start_sframe_header; +void *sframe_buf = __start_sframe_header; + +if (sframe_size <=0 ) +return ; + sftbl.sfhdr_p = sframe_buf; +if (!sftbl.sfhdr_p || sftbl.sfhdr_p->preamble.magic !=SFRAME_MAGIC || + sftbl.sfhdr_p->preamble.version !=SFRAME_VERSION_2 || + !(sftbl.sfhdr_p->preamble.flags &SFRAME_F_FDE_SORTED )) {
kernel/sframe_lookup.c line 196
Quoted 10 lines
kernel/sframe_lookup.c new file +10
-0+pr_warn ("WARNING: Unable to read sframe header. Disabling unwinder.\n" ); +return ; + } + + sftbl.fde_p = (struct sframe_fde *)(__start_sframe_header +SFRAME_HDR_SIZE (*sftbl.sfhdr_p) + + sftbl.sfhdr_p->fdes_off); + sftbl.fre_p = __start_sframe_header +SFRAME_HDR_SIZE (*sftbl.sfhdr_p) + + sftbl.sfhdr_p->fres_off; + sframe_init = true; +}
Other than the minor suggestion, the code looks good to me.
Re: [PATCH 4/8] unwind: Implement generic sframe unwinder library
Quoted 8 lines · Prasanna Kumar T S M
Quoted 6 lines · Weinan Liu
This change introduces a kernel space unwinder using sframe table for architectures without ORC unwinder support.
The implementation is adapted from Josh's userspace sframe unwinder proposal[1] according to the sframe v2 spec[2].
Quoted 8 lines
Quoted 8 lines
[1] https://lore.kernel.org/lkml/42c0a99236af65c09c8182e260af7bcf5aa1e158.1730150953.git.jpoimboe@kernel.org/
[2] https://sourceware.org/binutils/docs/sframe-spec.htmlSigned-off-by Weinan Liu <[email protected]>---
include/linux/sframe_lookup.h| 43 ++++++++ kernel/Makefile | 1 + kernel/sframe_lookup.c | 196 ++++++++++++++++++++++++++++++++++
Nit: Can this file be placed inside lib/ instead of kernel/ folder?
Quoted 288 lines
Quoted 15 lines
3 files changed, 240 insertions(+)create mode 100644
include/linux/sframe_lookup.hcreate mode 100644kernel/sframe_lookup.cinclude/linux/sframe_lookup.h new file +5
-0@@ -0,0 +1,43 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _LINUX_SFRAME_LOOKUP_H +#define _LINUX_SFRAME_LOOKUP_H + +/** include/linux/sframe_lookup.h line 15
Quoted 10 lines
include/linux/sframe_lookup.h new file +10
-0+* struct sframe_ip_entry - sframe unwind info for given ip +* @cfa_offset: Offset for the Canonical Frame Address(CFA) from Frame +* Pointer(FP) or Stack Pointer(SP) +* @ra_offset: Offset for the Return Address from CFA. +* @fp_offset: Offset for the Frame Pointer (FP) from CFA. +* @use_fp: Use FP to get next CFA or not +*/ +struct sframe_ip_entry { + int32_t cfa_offset; + int32_t ra_offset;The
ra_offsetis not present forx86_64in SFrame FRE as per the spec. I am wondering whether this struct should change based on the architecture or just setra_offsetcalculated fromcfa_offsetforx86_64.Quoted 248 lines
include/linux/sframe_lookup.h new file +22
-0+ int32_t fp_offset; + bool use_fp; +}; + +/** +* struct sframe_table - sframe struct of a table +* @sfhdr_p: Pointer to sframe header +* @fde_p: Pointer to the first of sframe frame description entry(FDE). +* @fre_p: Pointer to the first of sframe frame row entry(FRE). +*/ +struct sframe_table { + struct sframe_header *sfhdr_p; + struct sframe_fde *fde_p; + char *fre_p; +}; + +#ifdef CONFIG_SFRAME_UNWINDER +void init_sframe_table (void ); +int sframe_find_pc (unsigned long pc,struct sframe_ip_entry *entry); +#else +static inline void init_sframe_table (void ) {} +static inline int sframe_find_pc(unsigned long pc, struct
sframe_ip_entry*entry)include/linux/sframe_lookup.h new file +6
-0+{ + return -EINVAL; +} +#endif + +#endif /* _LINUX_SFRAME_LOOKUP_H */ kernel/Makefile +0
-0@@ -131,6 +131,7 @@ obj-$(CONFIG_WATCH_QUEUE) += watch_queue.oobj-$(CONFIG_RESOURCE_KUNIT_TEST) += resource_kunit.o obj-$(CONFIG_SYSCTL_KUNIT_TEST) += sysctl-test.o +obj-$(CONFIG_SFRAME_UNWINDER) += sframe_lookup.o CFLAGS_stackleak.o += $(DISABLE_STACKLEAK_PLUGIN) obj-$(CONFIG_GCC_PLUGIN_STACKLEAK) += stackleak.o
kernel/sframe_lookup.c new file +58
-0@@ -0,0 +1,196 @@ +// SPDX-License-Identifier: GPL-2.0-only +#include <linux/module.h> +#include <linux/sort.h> +#include <linux/sframe_lookup.h> +#include <linux/kallsyms.h> +#include "sframe.h" + +#define pr_fmt (fmt) "sframe: " fmt + +extern char __start_sframe_header[]; +extern char __stop_sframe_header[]; + +static bool sframe_init__ro_after_init ; +static struct sframe_table sftbl; + +#define SFRAME_READ_TYPE (in, out, type) \ +({ \ + type __tmp; \ + memcpy(&__tmp, in, sizeof(__tmp)); \ + in += sizeof(__tmp); \ + out = __tmp; \ +}) + +#define SFRAME_READ_ROW_ADDR (in_addr, out_addr, type) \ +({ \ + switch (type) { \ + case SFRAME_FRE_TYPE_ADDR1: \ + SFRAME_READ_TYPE(in_addr, out_addr, u8); \ + break; \ + case SFRAME_FRE_TYPE_ADDR2: \ + SFRAME_READ_TYPE(in_addr, out_addr, u16); \ + break; \ + case SFRAME_FRE_TYPE_ADDR4: \ + SFRAME_READ_TYPE(in_addr, out_addr, u32); \ + break; \ + default: \ + break; \ + } \ +}) + +#define SFRAME_READ_ROW_OFFSETS (in_addr, out_addr, size) \ +({ \ + switch (size) { \ + case 1: \ + SFRAME_READ_TYPE(in_addr, out_addr, s8); \ + break; \ + case 2: \ + SFRAME_READ_TYPE(in_addr, out_addr, s16); \ + break; \ + case 4: \ + SFRAME_READ_TYPE(in_addr, out_addr, s32); \ + break; \ + default: \ + break; \ + } \ +}) + +static struct sframe_fde *find_fde(const struct sframe_table *tbl,unsigned long pc)
kernel/sframe_lookup.c new file +11
-0+{ + int l, r, m, f; + int32_t ip; + struct sframe_fde *fdep; + + if (!tbl || !tbl->sfhdr_p || !tbl->fde_p) + return NULL; + + ip = (pc - (unsigned long)tbl->sfhdr_p); + + /* Do a binary range search to find the rightmost FDE start_addr< ip */
+ l = m = f = 0;
+ r =tbl->sfhdr_p->num_fdes;
+ while (l < r) {
+ m = l + ((r - l) / 2);
+ fdep =tbl->fde_p+ m;
+ if (fdep->start_addr> ip)
+ r = m;
+ else
+ l = m + 1;
+ }
+ /* use l - 1 because l will be the first itemfdep->start_addr> ip */
+ f = l - 1;
+ if (f >=tbl->sfhdr_p->num_fdes|| f < 0)
+ return NULL;
+ fdep =tbl->fde_p+ f;
+ if (ip <fdep->start_addr|| ip >=fdep->start_addr+fdep->size) + return NULL;kernel/sframe_lookup.c new file +21
-0+ + return fdep; +} + +static int find_fre (const struct sframe_table *tbl,unsigned long pc, + const struct sframe_fde *fdep, struct sframe_ip_entry *entry) +{ + int i, offset_size, offset_count; + char *fres, *offsets_loc; + int32_t ip_off; + uint32_t next_row_ip_off; + uint8_t fre_info, fde_type = SFRAME_FUNC_FDE_TYPE(fdep->info), + fre_type = SFRAME_FUNC_FRE_TYPE(fdep->info); + + fres = tbl->fre_p + fdep->fres_off; + + /* Whether PCs in the FREs should be treated as masks or not */ + if (fde_type == SFRAME_FDE_TYPE_PCMASK) + ip_off = pc % fdep->rep_size; + else + ip_off = (int32_t)(pc - (unsigned long)tbl->sfhdr_p) -
fdep->start_addr;kernel/sframe_lookup.c new file +6
-0+ + if (ip_off < 0 || ip_off >= fdep->size) + return -EINVAL; + + /* +* FRE structure starts by address of the entry with variants length. Use
+ * two pointers to track current head(fres) and the address of last + *offset(offsets_loc)
+ */
+ for (i = 0; i <fdep->fres_num; i++) {
+SFRAME_READ_ROW_ADDR(fres, next_row_ip_off, fre_type); + if (ip_off<next_row_ip_off)
+ break;
+SFRAME_READ_TYPE(fres, fre_info, u8);
+offsets_loc= fres;
+ /*
+ * jump to the start of next fre
+ * fres += fre_offets_cnt*offset_size
+ */
+ fres +=SFRAME_FRE_OFFSET_COUNT(fre_info)<<SFRAME_FRE_OFFSET_SIZE(fre_info);
+ }kernel/sframe_lookup.c new file +6
-0+ + offset_size = 1 << SFRAME_FRE_OFFSET_SIZE(fre_info); + offset_count = SFRAME_FRE_OFFSET_COUNT(fre_info); + + if (offset_count > 0) { + SFRAME_READ_ROW_OFFSETS(offsets_loc, entry->cfa_offset,
offset_size);
+ offset_count--;
+ }
+ if (offset_count> 0 && !entry->ra_offset) {
+SFRAME_READ_ROW_OFFSETS(offsets_loc, entry->ra_offset, offset_size);
+ offset_count--;
+ }
+ if (offset_count> 0 && !entry->fp_offset) {
+SFRAME_READ_ROW_OFFSETS(offsets_loc, entry->fp_offset, offset_size);
+ offset_count--;
+ }
+ if (offset_count)
+ return-EINVAL;
+
+entry->use_fp=SFRAME_FRE_CFA_BASE_REG_ID(fre_info)==SFRAME_BASE_REG_FP;kernel/sframe_lookup.c new file +30
-0+ + return 0; +} + +int sframe_find_pc (unsigned long pc,struct sframe_ip_entry *entry) +{ + struct sframe_fde *fdep; + struct sframe_table *sftbl_p = &sftbl; + int err; + + if (!sframe_init) + return -EINVAL; + + memset(entry, 0, sizeof(*entry)); + entry->ra_offset = sftbl_p->sfhdr_p->cfa_fixed_ra_offset; + entry->fp_offset = sftbl_p->sfhdr_p->cfa_fixed_fp_offset; + + fdep = find_fde(sftbl_p, pc); + if (!fdep) + return -EINVAL; + err = find_fre(sftbl_p, pc, fdep, entry); + if (err) + return err; + + return 0; +} + +void __init init_sframe_table (void ) +{ + size_t sframe_size = (void *)__stop_sframe_header - (void*)__start_sframe_header;
+ void*sframe_buf=__start_sframe_header;
+
+ if (sframe_size<= 0)
+ return;
+ sftbl.sfhdr_p =sframe_buf;
+ if (!sftbl.sfhdr_p ||sftbl.sfhdr_p->preamble.magic!=SFRAME_MAGIC||
+sftbl.sfhdr_p->preamble.version!=SFRAME_VERSION_2|| + !(sftbl.sfhdr_p->preamble.flags &SFRAME_F_FDE_SORTED)) { + pr_warn("WARNING: Unable to read sframe header. Disabling unwinder.\n");
+ return;
+ }
+Quoted 8 lines
+ sftbl.fde_p = (struct
sframe_fde*)(__start_sframe_header +SFRAME_HDR_SIZE(*sftbl.sfhdr_p)
+ +sftbl.sfhdr_p->fdes_off);
+ sftbl.fre_p =__start_sframe_header+SFRAME_HDR_SIZE(*sftbl.sfhdr_p)
+ +sftbl.sfhdr_p->fres_off;
+sframe_init= true;
+}Other than the minor suggestion, the code looks good to me.
Reviewed-by Prasanna Kumar T S M <[email protected]>.
Re: [PATCH 4/8] unwind: Implement generic sframe unwinder library
In reply to Prasanna Kumar T S MQuoted 18 lines · Prasanna Kumar T S M
Quoted 17 lines · Weinan Liu
include/linux/sframe_lookup.h new file +11
-0@@ -0,0 +1,43 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _LINUX_SFRAME_LOOKUP_H +#define _LINUX_SFRAME_LOOKUP_H + +/** +* struct sframe_ip_entry - sframe unwind info for given ip +* @cfa_offset: Offset for the Canonical Frame Address(CFA) from Frame +* Pointer(FP) or Stack Pointer(SP) +* @ra_offset: Offset for the Return Address from CFA. +* @fp_offset: Offset for the Frame Pointer (FP) from CFA. +* @use_fp: Use FP to get next CFA or not
Quoted 8 lines
include/linux/sframe_lookup.h line 15
Quoted 4 lines
include/linux/sframe_lookup.h new file +4
-0+*/ +struct sframe_ip_entry { +int32_t cfa_offset; +int32_t ra_offset;The
ra_offsetis not present forx86_64in SFrame FRE as per the spec. I am wondering whether this struct should change based on the architecture or just setra_offsetcalculated fromcfa_offsetforx86_64.
According to the https://sourceware.org/binutils/docs/sframe-spec.html#AMD64
For x86_64, RA will be stored at a fixed offset from the CFA upon function entry. The ra_offset will set to sfhdr_p->cfa_fixed_ra_offset during the initialization of the sframe_ip_entry within the function sframe_find_pc()
Re: [PATCH 4/8] unwind: Implement generic sframe unwinder library
Quoted 13 lines · Prasanna Kumar T S M
Quoted 12 lines · Prasanna Kumar T S M
Quoted 10 lines · Weinan Liu
This change introduces a kernel space unwinder using sframe table for architectures without ORC unwinder support.
The implementation is adapted from Josh's userspace sframe unwinder proposal[1] according to the sframe v2 spec[2].
[1]
https://lore.kernel.org/lkml/42c0a99236af65c09c8182e260af7bcf5aa1e158.1730150953.git.jpoimboe@kernel.org/
[2] https://sourceware.org/binutils/docs/sframe-spec.html
Quoted 6 lines
Quoted 5 lines
Quoted 5 lines
Signed-off-by Weinan Liu <[email protected]>--- include/linux/sframe_lookup.h | 43 ++++++++ kernel/Makefile | 1 + kernel/sframe_lookup.c | 196 ++++++++++++++++++++++++++++++++++Nit: Can this file be placed inside
lib/instead ofkernel/folder?
This could be integrated with Josh's proposal in the future.
https://lore.kernel.org/lkml/[email protected]/
Either lib/ or kernel/unwind/ are ok to me.
PATCH 5/8 unwind: arm64: Add sframe unwinder on arm64
In reply to Weinan LiuAdd unwind_next_frame_sframe() function to unwind by sframe info. Built with GNU Binutils 2.42 to verify that this sframe unwinder can backtrace correctly on arm64.
--- arch/arm64/include/asm/stacktrace/common.h | 4 ++ arch/arm64/kernel/setup.c | 2 + arch/arm64/kernel/stacktrace.c | 59 ++++++++++++++++++++++ 3 files changed, 65 insertions(+)
arch/arm64/include/asm/stacktrace/common.h +4 -0
@@ -25,6 +25,7 @@ struct stack_info {* @stack: The stack currently being unwound. * @stacks: An array of stacks which can be unwound. * @nr_stacks: The number of stacks in @stacks. +* @cfa: The sp value at the call site of the current function. */ struct unwind_state {unsigned long fp;
@@ -33,6 +34,9 @@ struct unwind_state {struct stack_info stack;struct stack_info *stacks;int nr_stacks; +#ifdef CONFIG_SFRAME_UNWINDER +unsigned long cfa; +#endif };static inline struct stack_info stackinfo_get_unknown (void )
arch/arm64/kernel/setup.c +2 -0
@@ -32,6 +32,7 @@#include <linux/sched/task.h> #include <linux/scs.h> #include <linux/mm.h> +#include <linux/sframe_lookup.h> #include <asm/acpi.h> #include <asm/fixmap.h>
@@ -377,6 +378,7 @@ void __init __no_sanitize_address setup_arch(char **cmdline_p)"This indicates a broken bootloader or old kernel\n" , boot_args[1], boot_args[2], boot_args[3]); } +init_sframe_table (); }static inline bool cpu_can_disable (unsigned int cpu)
arch/arm64/kernel/stacktrace.c +59 -0
84 lines not on this page · raw message
Signature · 1 line
2.48.1.262.g85cc9f2d1e-goog
Re: [PATCH 5/8] unwind: arm64: Add sframe unwinder on arm64
Quoted 136 lines · Weinan Liu
Add
unwind_next_frame_sframe()function to unwind by sframe info. Built with GNU Binutils 2.42 to verify that this sframe unwinder can backtrace correctly on arm64.Signed-off-by Weinan Liu <[email protected]>--- arch/arm64/include/asm/stacktrace/common.h | 4 ++ arch/arm64/kernel/setup.c | 2 + arch/arm64/kernel/stacktrace.c | 59 ++++++++++++++++++++++ 3 files changed, 65 insertions(+)arch/arm64/include/asm/stacktrace/common.h +4
-0@@ -25,6 +25,7 @@ struct stack_info {* @stack: The stack currently being unwound. * @stacks: An array of stacks which can be unwound. * @nr_stacks: The number of stacks in @stacks. +* @cfa: The sp value at the call site of the current function. */ struct unwind_state {unsigned long fp;@@ -33,6 +34,9 @@ struct unwind_state {struct stack_info stack;struct stack_info *stacks;int nr_stacks; +#ifdef CONFIG_SFRAME_UNWINDER +unsigned long cfa; +#endif };static inline struct stack_info stackinfo_get_unknown (void )arch/arm64/kernel/setup.c +2
-0@@ -32,6 +32,7 @@#include <linux/sched/task.h> #include <linux/scs.h> #include <linux/mm.h> +#include <linux/sframe_lookup.h> #include <asm/acpi.h> #include <asm/fixmap.h> @@ -377,6 +378,7 @@ void __init __no_sanitize_address setup_arch(char **cmdline_p)"This indicates a broken bootloader or old kernel\n" , boot_args[1], boot_args[2], boot_args[3]); } +init_sframe_table (); }static inline bool cpu_can_disable (unsigned int cpu)arch/arm64/kernel/stacktrace.c +56
-0@@ -14,6 +14,7 @@#include <linux/sched/debug.h> #include <linux/sched/task_stack.h> #include <linux/stacktrace.h> +#include <linux/sframe_lookup.h> #include <asm/efi.h> #include <asm/irq.h> @@ -242,6 +243,53 @@ kunwind_next_frame_record(struct kunwind_state *state)return 0 ; } +#ifdef CONFIG_SFRAME_UNWINDER +/* +* Unwind to the next frame according to sframe. +*/ +static __always_inline int +unwind_next_frame_sframe (struct unwind_state *state) +{ +unsigned long fp = state->fp, ip = state->pc; +unsigned long base_reg, cfa; +unsigned long pc_addr, fp_addr; +struct sframe_ip_entry entry; +struct stack_info *info; +struct frame_record *record = (struct frame_record *)fp; + +int err; + +/* frame record alignment 8 bytes */ +if (fp &0x7 ) +return -EINVAL ; + + info =unwind_find_stack (state, fp,sizeof (*record)); +if (!info) +return -EINVAL ; + + err =sframe_find_pc (ip, &entry); +if (err) +return -EINVAL ; + +unwind_consume_stack (state, info, fp,sizeof (*record)); + + base_reg = entry.use_fp ? fp : state->cfa; + +/* Set up the initial CFA using fp based info if CFA is not set */ +if (!state->cfa) + cfa = fp - entry.fp_offset; +else + cfa = base_reg + entry.cfa_offset; + fp_addr = cfa + entry.fp_offset; + pc_addr = cfa + entry.ra_offset; + state->cfa = cfa; + state->fp =READ_ONCE (*(unsigned long *)(fp_addr)); + state->pc =READ_ONCE (*(unsigned long *)(pc_addr)); + +return 0 ; +} +#endif +/* * Unwind from one frame record (A) to the next frame record (B). * @@ -261,7 +309,15 @@ kunwind_next(struct kunwind_state *state)case KUNWIND_SOURCE_CALLER :case KUNWIND_SOURCE_TASK :case KUNWIND_SOURCE_REGS_PC : +#ifdef CONFIG_SFRAME_UNWINDER + err =unwind_next_frame_sframe (&state->common); + +/* Fallback to FP based unwinder */ +if (err) err =kunwind_next_frame_record (state); +#else + err =kunwind_next_frame_record (state); +#endif break ;default : err = -EINVAL ;
arch/arm64/kernel/stacktrace.c kunwind_stack_walk(kunwind_consume_fn consume_state, · line 411
Quoted 9 lines
arch/arm64/kernel/stacktrace.c +3
-0@@ -347,6 +403,9 @@ kunwind_stack_walk(kunwind_consume_fn consume_state, .common = { .stacks = stacks, .nr_stacks = ARRAY_SIZE(stacks), +#ifdef CONFIG_SFRAME_UNWINDER + .cfa = 0, +#endif }, };
Looks good to me.
PATCH 6/8 unwind: arm64: add reliable stacktrace support for arm64
In reply to Weinan LiuTo support livepatch, we need to add arch_stack_walk_reliable to support reliable stacktrace according to
https://docs.kernel.org/livepatch/reliable-stacktrace.html#requirements
report stacktrace is not reliable if we are not able to unwind the stack by sframe unwinder and fallback to FP based unwinder
--- arch/arm64/include/asm/stacktrace/common.h | 2 + arch/arm64/kernel/stacktrace.c | 47 +++++++++++++++++++++- 2 files changed, 47 insertions(+), 2 deletions(-)
arch/arm64/include/asm/stacktrace/common.h +2 -0
@@ -26,6 +26,7 @@ struct stack_info {* @stacks: An array of stacks which can be unwound. * @nr_stacks: The number of stacks in @stacks. * @cfa: The sp value at the call site of the current function. +* @unreliable: Stacktrace is unreliable. */ struct unwind_state {unsigned long fp;
@@ -36,6 +37,7 @@ struct unwind_state {int nr_stacks;#ifdef CONFIG_SFRAME_UNWINDER unsigned long cfa; +bool unreliable;#endif };
arch/arm64/kernel/stacktrace.c +45 -2
62 lines not on this page · raw message
Signature · 1 line
2.48.1.262.g85cc9f2d1e-goog
Re: [PATCH 6/8] unwind: arm64: add reliable stacktrace support for arm64
Quoted 91 lines · Weinan Liu
To support livepatch, we need to add
arch_stack_walk_reliableto support reliable stacktrace according to
https://docs.kernel.org/livepatch/reliable-stacktrace.html#requirementsreport stacktrace is not reliable if we are not able to unwind the stack by sframe unwinder and fallback to FP based unwinder
Signed-off-by Weinan Liu <[email protected]>--- arch/arm64/include/asm/stacktrace/common.h | 2 + arch/arm64/kernel/stacktrace.c | 47 +++++++++++++++++++++- 2 files changed, 47 insertions(+), 2 deletions(-)arch/arm64/include/asm/stacktrace/common.h +2
-0@@ -26,6 +26,7 @@ struct stack_info {* @stacks: An array of stacks which can be unwound. * @nr_stacks: The number of stacks in @stacks. * @cfa: The sp value at the call site of the current function. +* @unreliable: Stacktrace is unreliable. */ struct unwind_state {unsigned long fp;@@ -36,6 +37,7 @@ struct unwind_state {int nr_stacks;#ifdef CONFIG_SFRAME_UNWINDER unsigned long cfa; +bool unreliable;#endif };arch/arm64/kernel/stacktrace.c +38
-2@@ -310,11 +310,16 @@ kunwind_next(struct kunwind_state *state)case KUNWIND_SOURCE_TASK :case KUNWIND_SOURCE_REGS_PC :#ifdef CONFIG_SFRAME_UNWINDER - err =unwind_next_frame_sframe (&state->common); +if (!state->common.unreliable) + err =unwind_next_frame_sframe (&state->common);/* Fallback to FP based unwinder */ -if (err) +if (err || state->common.unreliable) { err =kunwind_next_frame_record (state); +/* Mark its stacktrace result as unreliable if it is unwindable via FP */ +if (!err) + state->common.unreliable = true; + }#else err =kunwind_next_frame_record (state);#endif @@ -446,6 +451,44 @@ noinline noinstr void arch_stack_walk(stack_trace_consume_fn consume_entry,kunwind_stack_walk (arch_kunwind_consume_entry, &data, task, regs); } +#ifdef CONFIG_SFRAME_UNWINDER +struct kunwind_reliable_consume_entry_data { +stack_trace_consume_fn consume_entry; +void *cookie; +bool unreliable; +}; + +static __always_inline bool +arch_kunwind_reliable_consume_entry (const struct kunwind_state *state,void *cookie) +{ +struct kunwind_reliable_consume_entry_data *data = cookie; + +if (state->common.unreliable) { + data->unreliable = true; +return false; + } +return data->consume_entry (data->cookie, state->common.pc); +} + +noinline notrace int arch_stack_walk_reliable( +stack_trace_consume_fn consume_entry, +void *cookie,struct task_struct *task) +{ +struct kunwind_reliable_consume_entry_data data = { + .consume_entry = consume_entry, + .cookie = cookie, + .unreliable = false, + }; + +kunwind_stack_walk (arch_kunwind_reliable_consume_entry, &data, task,NULL ); +
arch/arm64/kernel/stacktrace.c noinline noinstr void arch_stack_walk(stack_trace_consume_fn consume_entry, · line 494
Quoted 10 lines
arch/arm64/kernel/stacktrace.c +7
-0+if (data.unreliable) +return -EINVAL ; + +return 0 ; +} +#endif +struct bpf_unwind_consume_entry_data {bool (*consume_entry)(void *cookie,u64 ip,u64 sp,u64 fp);void *cookie;
Why not fold the previous patch and this into one?
But the code looks good to me.
PATCH 7/8 arm64: Define TIF_PATCH_PENDING for livepatch
In reply to Weinan LiuFrom: "Madhavan T. Venkataraman" <[email protected]>
- Define TIF_PATCH_PENDING in arch/arm64/include/asm/thread_info.h for livepatch.
- Check TIF_PATCH_PENDING in do_notify_resume() to patch the current task for livepatch.
--- arch/arm64/include/asm/thread_info.h | 4 +++- arch/arm64/kernel/entry-common.c | 4 ++++ 2 files changed, 7 insertions(+), 1 deletion(-)
arch/arm64/include/asm/thread_info.h +3 -1
@@ -64,6 +64,7 @@ void arch_setup_new_exec(void);#define TIF_UPROBE 4/* uprobe breakpoint or singlestep */ #define TIF_MTE_ASYNC_FAULT 5/* MTE Asynchronous Tag Check Fault */ #define TIF_NOTIFY_SIGNAL 6/* signal notifications exist */ +#define TIF_PATCH_PENDING 7/* pending live patching update */ #define TIF_SYSCALL_TRACE 8/* syscall trace active */ #define TIF_SYSCALL_AUDIT 9/* syscall auditing */ #define TIF_SYSCALL_TRACEPOINT 10/* syscall tracepoint for ftrace */
@@ -99,11 +100,12 @@ void arch_setup_new_exec(void);#define _TIF_MTE_ASYNC_FAULT (1 << TIF_MTE_ASYNC_FAULT)#define _TIF_NOTIFY_SIGNAL (1 << TIF_NOTIFY_SIGNAL)#define _TIF_TSC_SIGSEGV (1 << TIF_TSC_SIGSEGV) +#define _TIF_PATCH_PENDING (1 << TIF_PATCH_PENDING)#define _TIF_WORK_MASK (_TIF_NEED_RESCHED | _TIF_SIGPENDING | \ _TIF_NOTIFY_RESUME | _TIF_FOREIGN_FPSTATE | \ _TIF_UPROBE | _TIF_MTE_ASYNC_FAULT | \ - _TIF_NOTIFY_SIGNAL) + _TIF_NOTIFY_SIGNAL | _TIF_PATCH_PENDING)#define _TIF_SYSCALL_WORK (_TIF_SYSCALL_TRACE | _TIF_SYSCALL_AUDIT | \ _TIF_SYSCALL_TRACEPOINT | _TIF_SECCOMP | \
arch/arm64/kernel/entry-common.c +4 -0
@@ -8,6 +8,7 @@#include <linux/context_tracking.h> #include <linux/kasan.h> #include <linux/linkage.h> +#include <linux/livepatch.h> #include <linux/lockdep.h> #include <linux/ptrace.h> #include <linux/resume_user_mode.h>
@@ -144,6 +145,9 @@ static void do_notify_resume(struct pt_regs *regs, unsigned long thread_flags) (void __user *)NULL, current); } +if (thread_flags & _TIF_PATCH_PENDING) +klp_update_patch_state (current); +if (thread_flags & (_TIF_SIGPENDING | _TIF_NOTIFY_SIGNAL))do_signal (regs);
Signature · 1 line
2.48.1.262.g85cc9f2d1e-goog
Re: [PATCH 7/8] arm64: Define TIF_PATCH_PENDING for livepatch
Quoted 55 lines · Weinan Liu
From: "Madhavan T. Venkataraman" <[email protected]>
- Define
TIF_PATCH_PENDINGinarch/arm64/include/asm/thread_info.hfor livepatch.- Check
TIF_PATCH_PENDINGindo_notify_resume()to patch the current task for livepatch.Signed-off-by Suraj Jitindar Singh <[email protected]>Signed-off-by Madhavan T. Venkataraman <[email protected]>Signed-off-by Weinan Liu <[email protected]>--- arch/arm64/include/asm/thread_info.h | 4 +++- arch/arm64/kernel/entry-common.c | 4 ++++ 2 files changed, 7 insertions(+), 1 deletion(-)arch/arm64/include/asm/thread_info.h +3
-1@@ -64,6 +64,7 @@ void arch_setup_new_exec(void);#define TIF_UPROBE 4/* uprobe breakpoint or singlestep */ #define TIF_MTE_ASYNC_FAULT 5/* MTE Asynchronous Tag Check Fault */ #define TIF_NOTIFY_SIGNAL 6/* signal notifications exist */ +#define TIF_PATCH_PENDING 7/* pending live patching update */ #define TIF_SYSCALL_TRACE 8/* syscall trace active */ #define TIF_SYSCALL_AUDIT 9/* syscall auditing */ #define TIF_SYSCALL_TRACEPOINT 10/* syscall tracepoint for ftrace */ @@ -99,11 +100,12 @@ void arch_setup_new_exec(void);#define _TIF_MTE_ASYNC_FAULT (1 << TIF_MTE_ASYNC_FAULT)#define _TIF_NOTIFY_SIGNAL (1 << TIF_NOTIFY_SIGNAL)#define _TIF_TSC_SIGSEGV (1 << TIF_TSC_SIGSEGV) +#define _TIF_PATCH_PENDING (1 << TIF_PATCH_PENDING)#define _TIF_WORK_MASK (_TIF_NEED_RESCHED | _TIF_SIGPENDING | \ _TIF_NOTIFY_RESUME | _TIF_FOREIGN_FPSTATE | \ _TIF_UPROBE | _TIF_MTE_ASYNC_FAULT | \ - _TIF_NOTIFY_SIGNAL) + _TIF_NOTIFY_SIGNAL | _TIF_PATCH_PENDING)#define _TIF_SYSCALL_WORK (_TIF_SYSCALL_TRACE | _TIF_SYSCALL_AUDIT | \ _TIF_SYSCALL_TRACEPOINT | _TIF_SECCOMP | \arch/arm64/kernel/entry-common.c +1
-0@@ -8,6 +8,7 @@#include <linux/context_tracking.h> #include <linux/kasan.h> #include <linux/linkage.h> +#include <linux/livepatch.h> #include <linux/lockdep.h> #include <linux/ptrace.h> #include <linux/resume_user_mode.h>
arch/arm64/kernel/entry-common.c static void do_notify_resume(struct pt_regs *regs, unsigned long thread_flags) · line 153
Quoted 9 lines
arch/arm64/kernel/entry-common.c +3
-0@@ -144,6 +145,9 @@ static void do_notify_resume(struct pt_regs *regs, unsigned long thread_flags) (void __user *)NULL, current); } +if (thread_flags & _TIF_PATCH_PENDING) +klp_update_patch_state (current); +if (thread_flags & (_TIF_SIGPENDING | _TIF_NOTIFY_SIGNAL))do_signal (regs);
Looks good to me.
Miroslav Benes
maintainerRe: [PATCH 7/8] arm64: Define TIF_PATCH_PENDING for livepatch
In reply to Weinan LiuHi,
Quoted 13 lines
arch/arm64/kernel/entry-common.c +1
-0@@ -8,6 +8,7 @@#include <linux/context_tracking.h> #include <linux/kasan.h> #include <linux/linkage.h> +#include <linux/livepatch.h> #include <linux/lockdep.h> #include <linux/ptrace.h> #include <linux/resume_user_mode.h>
arch/arm64/kernel/entry-common.c static void do_notify_resume(struct pt_regs *regs, unsigned long thread_flags) · line 152
Quoted 8 lines
arch/arm64/kernel/entry-common.c +3
-0@@ -144,6 +145,9 @@ static void do_notify_resume(struct pt_regs *regs, unsigned long thread_flags) (void __user *)NULL, current); } +if (thread_flags & _TIF_PATCH_PENDING) +klp_update_patch_state (current); +if (thread_flags & (_TIF_SIGPENDING | _TIF_NOTIFY_SIGNAL))do_signal (regs);
Just a remark so that it is recorded. Once arm64 is moved to generic entry infra, the hunk will not be needed.
Regards,
Miroslav
PATCH 8/8 arm64: Enable livepatch for ARM64
In reply to Weinan LiuSince SFrame is considered as reliable stacktrace, enable livepatch in arch/arm64/Kconfig
--- arch/arm64/Kconfig | 3 +++ 1 file changed, 3 insertions(+)
arch/arm64/Kconfig +3 -0
@@ -271,6 +271,8 @@ config ARM64 select HAVE_SOFTIRQ_ON_OWN_STACK select USER_STACKTRACE_SUPPORT select VDSO_GETRANDOM + select HAVE_RELIABLE_STACKTRACE if SFRAME_UNWINDER + select HAVE_LIVEPATCH if HAVE_DYNAMIC_FTRACE_WITH_ARGS && HAVE_RELIABLE_STACKTRACE help ARM 64-bit (AArch64) Linux support.
@@ -2498,3 +2500,4 @@ source "drivers/acpi/Kconfig" source "arch/arm64/kvm/Kconfig" +source "kernel/livepatch/Kconfig"
Signature · 1 line
2.48.1.262.g85cc9f2d1e-goog
Re: [PATCH 8/8] arm64: Enable livepatch for ARM64
Quoted 22 lines · Weinan Liu
Since SFrame is considered as reliable stacktrace, enable livepatch in
arch/arm64/KconfigSigned-off-by Weinan Liu <[email protected]>--- arch/arm64/Kconfig | 3 +++ 1 file changed, 3 insertions(+)arch/arm64/Kconfig +2
-0@@ -271,6 +271,8 @@ config ARM64 select HAVE_SOFTIRQ_ON_OWN_STACK select USER_STACKTRACE_SUPPORT select VDSO_GETRANDOM + select HAVE_RELIABLE_STACKTRACE if SFRAME_UNWINDER + select HAVE_LIVEPATCH if HAVE_DYNAMIC_FTRACE_WITH_ARGS && HAVE_RELIABLE_STACKTRACE help ARM 64-bit (AArch64) Linux support.
arch/arm64/Kconfig source "drivers/acpi/Kconfig" · line 2503
Quoted 4 lines
arch/arm64/Kconfig +1
-0@@ -2498,3 +2500,4 @@ source "drivers/acpi/Kconfig" source "arch/arm64/kvm/Kconfig" +source "kernel/livepatch/Kconfig"
Looks good to me.
Re: [PATCH 8/8] arm64: Enable livepatch for ARM64
In reply to Weinan LiuQuoted 22 lines · Weinan Liu
Since SFrame is considered as reliable stacktrace, enable livepatch in
arch/arm64/KconfigSigned-off-by Weinan Liu <[email protected]>--- arch/arm64/Kconfig | 3 +++ 1 file changed, 3 insertions(+)arch/arm64/Kconfig +2
-0@@ -271,6 +271,8 @@ config ARM64 select HAVE_SOFTIRQ_ON_OWN_STACK select USER_STACKTRACE_SUPPORT select VDSO_GETRANDOM + select HAVE_RELIABLE_STACKTRACE if SFRAME_UNWINDER + select HAVE_LIVEPATCH if HAVE_DYNAMIC_FTRACE_WITH_ARGS && HAVE_RELIABLE_STACKTRACE help ARM 64-bit (AArch64) Linux support.
arch/arm64/Kconfig source "drivers/acpi/Kconfig" · line 2503
Quoted 4 lines
arch/arm64/Kconfig +1
-0@@ -2498,3 +2500,4 @@ source "drivers/acpi/Kconfig" source "arch/arm64/kvm/Kconfig" +source "kernel/livepatch/Kconfig"
Will this work for ftrace'd (kprobe'd) function as well?
Steven Rostedt
reviewerRe: [PATCH 8/8] arm64: Enable livepatch for ARM64
Quoted 11 lines · On Fri, 31 Jan 2025 21:38:46 +0530 Prasanna Kumar T S M…
Quoted 11 lines
arch/arm64/Kconfig +2
-0@@ -271,6 +271,8 @@ config ARM64 select HAVE_SOFTIRQ_ON_OWN_STACK select USER_STACKTRACE_SUPPORT select VDSO_GETRANDOM + select HAVE_RELIABLE_STACKTRACE if SFRAME_UNWINDER + select HAVE_LIVEPATCH if HAVE_DYNAMIC_FTRACE_WITH_ARGS && HAVE_RELIABLE_STACKTRACE help ARM 64-bit (AArch64) Linux support.
Quoted 7 lines
arch/arm64/Kconfig source "drivers/acpi/Kconfig" · line 2503
Quoted 5 lines
arch/arm64/Kconfig +1
-0@@ -2498,3 +2500,4 @@ source "drivers/acpi/Kconfig" source "arch/arm64/kvm/Kconfig" +source "kernel/livepatch/Kconfig"Will this work for ftrace'd (kprobe'd) function as well?
What work? The stack walker? It may require updates to the tracing trampolines, but it should be doable just like ORC is.
-- Steve
Quoted 7 lines · Weinan Liu
This patchset implements a generic kernel sframe-based [1] unwinder. The main goal is to support reliable stacktraces on arm64.
On x86 orc unwinder provides reliable stacktraces. But arm64 misses the required support from objtool: it cannot generate orc unwind tables for arm64.
Quoted 8 lines
Currently, there's already a sframe unwinder proposed for userspace: [2]. Since the sframe unwind table algorithm is similar, these two proposal could integrate common functionality in the future.
There are some incomplete features or challenges:
- The unwinder doesn't yet work with kernel modules. Thestart_addrof FRE from kernel modules doesn't appear correct, preventing us from unwinding functions from kernel modules.
I did file https://sourceware.org/bugzilla/show_bug.cgi?id=32589 earlier. It is misleading (and inconvenient) to see all 0s in the dump of non-relocated SFrame section.
I get the sense that while issue 32589 may be causing confusion that the SFrame data is not correct, the problem is elsewhere ?
I can share a fix for 32589 so atleast we can verify that the starting point is sane.
Quoted 38 lines
- Currently, only GCC supports sframe.
Ref:
[1]: https://sourceware.org/binutils/docs/sframe-spec.html
[2]: https://lore.kernel.org/lkml/[email protected]/Madhavan T. Venkataraman (1):
arm64: DefineTIF_PATCH_PENDINGfor livepatchWeinan Liu (7):
unwind: build kernel with sframe info
arm64: entry: add unwind info for various kernel entries
unwind: add sframe v2 header
unwind: Implement generic sframe unwinder library
unwind: arm64: Add sframe unwinder on arm64
unwind: arm64: add reliable stacktrace support for arm64
arm64: Enable livepatch for ARM64Makefile | 6 + arch/Kconfig | 8 + arch/arm64/Kconfig | 3 + arch/arm64/Kconfig.debug | 10 + arch/arm64/include/asm/stacktrace/common.h | 6 + arch/arm64/include/asm/thread_info.h | 4 +- arch/arm64/kernel/entry-common.c | 4 + arch/arm64/kernel/entry.S | 10 + arch/arm64/kernel/setup.c | 2 + arch/arm64/kernel/stacktrace.c | 102 ++++++++++ include/asm-generic/vmlinux.lds.h | 12 ++ include/linux/sframe_lookup.h | 43 +++++ kernel/Makefile | 1 + kernel/sframe.h | 215 +++++++++++++++++++++ kernel/sframe_lookup.c | 196 +++++++++++++++++++ 15 files changed, 621 insertions(+), 1 deletion(-)create mode 100644
include/linux/sframe_lookup.hcreate mode 100644kernel/sframe.hcreate mode 100644kernel/sframe_lookup.c
Not only does objdump show that func_start_addr all 0s, but the func_start_addr in the kernel module's sframe table is also incorrect. I'll prepare a repro so we can investigate this.
Quoted 4 lines
I can share a fix for 32589 so atleast we can verify that the starting point is sane.
Great! Thanks.
This will be helpful for comparing the values from the sframe table with the values we obtain at runtime
Song Liu
maintainerI missed this set before sending my RFC set. If this set works well, we won't need the other set. I will give this one a try.
Thanks,
Song
Quoted 58 lines · Weinan Liu
This patchset implements a generic kernel sframe-based [1] unwinder. The main goal is to support reliable stacktraces on arm64.
On x86 orc unwinder provides reliable stacktraces. But arm64 misses the required support from objtool: it cannot generate orc unwind tables for arm64.
Currently, there's already a sframe unwinder proposed for userspace: [2]. Since the sframe unwind table algorithm is similar, these two proposal could integrate common functionality in the future.
There are some incomplete features or challenges:
- The unwinder doesn't yet work with kernel modules. Thestart_addrof FRE from kernel modules doesn't appear correct, preventing us from unwinding functions from kernel modules.
- Currently, only GCC supports sframe.Ref:
[1]: https://sourceware.org/binutils/docs/sframe-spec.html
[2]: https://lore.kernel.org/lkml/[email protected]/Madhavan T. Venkataraman (1):
arm64: DefineTIF_PATCH_PENDINGfor livepatchWeinan Liu (7):
unwind: build kernel with sframe info
arm64: entry: add unwind info for various kernel entries
unwind: add sframe v2 header
unwind: Implement generic sframe unwinder library
unwind: arm64: Add sframe unwinder on arm64
unwind: arm64: add reliable stacktrace support for arm64
arm64: Enable livepatch for ARM64Makefile | 6 + arch/Kconfig | 8 + arch/arm64/Kconfig | 3 + arch/arm64/Kconfig.debug | 10 + arch/arm64/include/asm/stacktrace/common.h | 6 + arch/arm64/include/asm/thread_info.h | 4 +- arch/arm64/kernel/entry-common.c | 4 + arch/arm64/kernel/entry.S | 10 + arch/arm64/kernel/setup.c | 2 + arch/arm64/kernel/stacktrace.c | 102 ++++++++++ include/asm-generic/vmlinux.lds.h | 12 ++ include/linux/sframe_lookup.h | 43 +++++ kernel/Makefile | 1 + kernel/sframe.h | 215 +++++++++++++++++++++ kernel/sframe_lookup.c | 196 +++++++++++++++++++ 15 files changed, 621 insertions(+), 1 deletion(-) create mode 100644 include/linux/sframe_lookup.h create mode 100644 kernel/sframe.h create mode 100644 kernel/sframe_lookup.c--
2.48.1.262.g85cc9f2d1e-goog
Song Liu
maintainerQuoted 3 lines · Song Liu
I missed this set before sending my RFC set. If this set works well, we won't need the other set. I will give this one a try.
I just realized that llvm doesn't support sframe yet. So we (Meta) still need some sframe-less approach before llvm supports sframe.
IIRC, Google also uses llvm to compile the kernel. Weinan, would you mind share your thoughts on how we can adopt this before llvm supports sframe? (compile arm64 kernel with gcc?)
Thanks,
Song
Quoted 11 lines · Song Liu
Quoted 3 lines · Song Liu
I missed this set before sending my RFC set. If this set works well, we won't need the other set. I will give this one a try.
I just realized that llvm doesn't support sframe yet. So we (Meta) still need some sframe-less approach before llvm supports sframe.
IIRC, Google also uses llvm to compile the kernel. Weinan, would you mind share your thoughts on how we can adopt this before llvm supports sframe? (compile arm64 kernel with gcc?)
Hi Song,
the plan is to start the work on adding sframe support to llvm in parallel to landing these changes upstream. From the initial assessment it shouldn't be too hard.
Thanks
Song Liu
maintainerHi Roman,
Quoted 14 lines · Roman Gushchin
Quoted 11 lines · Song Liu
Quoted 3 lines · Song Liu
I missed this set before sending my RFC set. If this set works well, we won't need the other set. I will give this one a try.
I just realized that llvm doesn't support sframe yet. So we (Meta) still need some sframe-less approach before llvm supports sframe.
IIRC, Google also uses llvm to compile the kernel. Weinan, would you mind share your thoughts on how we can adopt this before llvm supports sframe? (compile arm64 kernel with gcc?)
Quoted 5 lines
Hi Song,
the plan is to start the work on adding sframe support to llvm in parallel to landing these changes upstream. From the initial assessment it shouldn't be too hard.
Thanks for the information!
Song
Quoted 17 lines · Weinan Liu
This patchset implements a generic kernel sframe-based [1] unwinder. The main goal is to support reliable stacktraces on arm64.
On x86 orc unwinder provides reliable stacktraces. But arm64 misses the required support from objtool: it cannot generate orc unwind tables for arm64.
Currently, there's already a sframe unwinder proposed for userspace: [2]. Since the sframe unwind table algorithm is similar, these two proposal could integrate common functionality in the future.
There are some incomplete features or challenges:
- The unwinder doesn't yet work with kernel modules. Thestart_addrof FRE from kernel modules doesn't appear correct, preventing us from unwinding functions from kernel modules.
- Currently, only GCC supports sframe.
Quoted 4 lines
Ref:
[1]: https://sourceware.org/binutils/docs/sframe-spec.html
[2]: https://lore.kernel.org/lkml/[email protected]/
Hi Weinan,
Thanks for working on this.
I tested this set on my setup and faced some issues, here are the details:
Here is my setup [on AWS c6gd.16xlarge instance]:
-------------------------------------------------
[root@ip-172-31-32-86 linux-upstream]# uname -a
Linux ip-172-31-32-86.ec2.internal 6.14.0-rc1+ #1 SMP Tue Feb 4 14:15:55 UTC 2025 aarch64 aarch64 aarch64 GNU/Linux
[root@ip-172-31-32-86 linux-upstream]# git log --onelinee9a702365 (HEAD -> master) arm64: Enable livepatch for ARM645dedc956e arm64: Define TIF_PATCH_PENDING for livepatchba563b31a unwind: arm64: add reliable stacktrace support for arm64d807d392d unwind: arm64: Add sframe unwinder on arm647872f050b unwind: Implement generic sframe unwinder library03d2ad003 unwind: add sframe v2 header5e95cc051 arm64: entry: add unwind info for various kernel entriesfaff6cbc3 unwind: build kernel with sframe info0de63bb7d (origin/master, origin/HEAD) Merge tag 'pull-fix' of git://git.kernel.org/pub/scm/linux/kernel/git/viro/vfs902e09c8a fix braino in "9p: fix ->rename_sem exclusion"f286757b6 Merge tag 'timers-urgent-2025-02-03' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tipa360f3ffd (grafted) Merge tag 'irq-urgent-2025-02-03' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tipbb2784d9a (grafted) jiffies: Cast to unsigned long in secs_to_jiffies() conversion30d61efe1 (grafted) 9p: fix ->rename_sem exclusion
[root@ip-172-31-32-86 linux-upstream]# grep SFRAME .config CONFIG_AS_HAS_SFRAME_SUPPORT=y CONFIG_SFRAME_UNWIND_TABLE=y CONFIG_SFRAME_UNWINDER=y
[root@ip-172-31-32-86 linux-upstream]# grep LIVEPATCH .config CONFIG_HAVE_LIVEPATCH=yCONFIG_LIVEPATCH=yCONFIG_SAMPLE_LIVEPATCH=m
[root@ip-172-31-32-86 linux-upstream]# as --version
GNU assembler version 2.41-50.al2023.0.2
Copyright (C) 2023 Free Software Foundation, Inc.
This program is free software; you may redistribute it under the terms of the GNU General Public License version 3 or later.
This program has absolutely no warranty.
This assembler was configured for a target of `aarch64-amazon-linux'.
[root@ip-172-31-32-86 linux-upstream]# gcc --version
gcc (GCC) 11.4.1 20230605 (Red Hat 11.4.1-2)
Copyright (C) 2021 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Loading the livepatch-sameple module:
-------------------------------------
[root@ip-172-31-32-86 linux-upstream]# kpatch load /lib/modules/6.14.0-rc1+/kernel/samples/livepatch/livepatch-sample.ko loading patch module: /lib/modules/6.14.0-rc1+/kernel/samples/livepatch/livepatch-sample.ko
waiting (up to 15 seconds) for patch transition to complete...
patch transition has stalled!
<4>kpatch: Livepatch process signaling is performed automatically on your system.
<4>kpatch: Skipping manual process signaling.
waiting (up to 60 seconds) for patch transition to complete...
Stalled processes:
340 kdevtmpfs
stack:
[<0>] devtmpfs_work_loop+0x2cc/0x2d8 [<0>] devtmpfsd+0x4c/0x58 [<0>] kthread+0xf0/0x100 [<0>] ret_from_fork+0x10/0x20
module livepatch_sample did not complete its transition, unloading... disabling patch module: livepatch_sample
waiting (up to 15 seconds) for patch transition to complete... transition complete (3 seconds)
unloading patch module: livepatch_sample
<4>kpatch: error: failed to load module livepatch_sample (transition stalled)
Useful messages from kernel log [pr_debug enabled]:
---------------------------------------------------
livepatch: enabling patch 'livepatch_sample'
livepatch: 'livepatch_sample': initializing patching transition
livepatch: 'livepatch_sample': starting patching transition
livepatch: klp_try_switch_task: kdevtmpfs:340 has an unreliable stack
livepatch: klp_try_switch_task: insmod:9226 has an unreliable stack
livepatch: klp_try_switch_task: swapper/63:0 is running [......SNIP.......]
livepatch: klp_try_switch_task: kdevtmpfs:340 has an unreliable stack [......SNIP.......]
livepatch: signaling remaining tasks
livepatch: klp_try_switch_task: kdevtmpfs:340 has an unreliable stack
livepatch: 'livepatch_sample': reversing transition from patching to unpatching
livepatch: 'livepatch_sample': starting unpatching transition
livepatch: klp_try_switch_task: swapper/45:0 is running
livepatch: 'livepatch_sample': completing unpatching transition
livepatch: 'livepatch_sample': unpatching complete
Please let me know if you are aware of this already or if this is expected behaviour with this version. I will try to debug this from my side as well. Also let me know if you need more details for debugging this.
P.S. - I also saw multiple build warning like:
ld: warning: orphan section .eh_frame' from arch/arm64/kernel/entry.o' being placed in section `.eh_frame'
ld: warning: orphan section .init.sframe' from arch/arm64/kernel/pi/lib-fdt.pi.o' being placed in section `.init.sframe'
Thanks,
Puranjay
Quoted 93 lines · Puranjay Mohan
Quoted 17 lines · Weinan Liu
This patchset implements a generic kernel sframe-based [1] unwinder. The main goal is to support reliable stacktraces on arm64.
On x86 orc unwinder provides reliable stacktraces. But arm64 misses the required support from objtool: it cannot generate orc unwind tables for arm64.
Currently, there's already a sframe unwinder proposed for userspace: [2]. Since the sframe unwind table algorithm is similar, these two proposal could integrate common functionality in the future.
There are some incomplete features or challenges:
- The unwinder doesn't yet work with kernel modules. Thestart_addrof FRE from kernel modules doesn't appear correct, preventing us from unwinding functions from kernel modules.
- Currently, only GCC supports sframe.Quoted 4 lines
Ref:
[1]: https://sourceware.org/binutils/docs/sframe-spec.html
[2]: https://lore.kernel.org/lkml/[email protected]/Hi Weinan,
Thanks for working on this.I tested this set on my setup and faced some issues, here are the details:
Here is my setup [on AWS c6gd.16xlarge instance]:
-------------------------------------------------[root@ip-172-31-32-86 linux-upstream]# uname -a
Linux ip-172-31-32-86.ec2.internal 6.14.0-rc1+ #1 SMP Tue Feb 4 14:15:55 UTC 2025 aarch64 aarch64 aarch64 GNU/Linux[root@ip-172-31-32-86 linux-upstream]# git log --oneline
e9a702365(HEAD -> master) arm64: Enable livepatch for ARM645dedc956earm64: DefineTIF_PATCH_PENDINGfor livepatchba563b31aunwind: arm64: add reliable stacktrace support for arm64d807d392dunwind: arm64: Add sframe unwinder on arm647872f050bunwind: Implement generic sframe unwinder library03d2ad003unwind: add sframe v2 header5e95cc051arm64: entry: add unwind info for various kernel entriesfaff6cbc3unwind: build kernel with sframe info0de63bb7d(origin/master, origin/HEAD) Merge tag 'pull-fix' ofgit://git.kernel.org/pub/scm/linux/kernel/git/viro/vfs902e09c8afix braino in "9p: fix ->rename_sem exclusion"f286757b6Merge tag 'timers-urgent-2025-02-03' ofgit://git.kernel.org/pub/scm/linux/kernel/git/tip/tipa360f3ffd(grafted) Merge tag 'irq-urgent-2025-02-03' ofgit://git.kernel.org/pub/scm/linux/kernel/git/tip/tipbb2784d9a(grafted) jiffies: Cast to unsigned long insecs_to_jiffies()conversion30d61efe1(grafted) 9p: fix ->rename_sem exclusion[root@ip-172-31-32-86 linux-upstream]# grep SFRAME .config
CONFIG_AS_HAS_SFRAME_SUPPORT=yCONFIG_SFRAME_UNWIND_TABLE=yCONFIG_SFRAME_UNWINDER=y
[root@ip-172-31-32-86 linux-upstream]# grep LIVEPATCH .configCONFIG_HAVE_LIVEPATCH=yCONFIG_LIVEPATCH=yCONFIG_SAMPLE_LIVEPATCH=m[root@ip-172-31-32-86 linux-upstream]# as --version
GNU assembler version 2.41-50.al2023.0.2
Copyright (C) 2023 Free Software Foundation, Inc.
This program is free software; you may redistribute it under the terms of the GNU General Public License version 3 or later.
This program has absolutely no warranty.
This assembler was configured for a target of `aarch64-amazon-linux'.[root@ip-172-31-32-86 linux-upstream]# gcc --version
gcc (GCC) 11.4.1 20230605 (Red Hat 11.4.1-2)
Copyright (C) 2021 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.Loading the livepatch-sameple module:
-------------------------------------[root@ip-172-31-32-86 linux-upstream]# kpatch load
/lib/modules/6.14.0-rc1+/kernel/samples/livepatch/livepatch-sample.koloading patch module:/lib/modules/6.14.0-rc1+/kernel/samples/livepatch/livepatch-sample.ko
waiting (up to 15 seconds) for patch transition to complete...
patch transition has stalled!
<4>kpatch: Livepatch process signaling is performed automatically on your system.
<4>kpatch: Skipping manual process signaling.
waiting (up to 60 seconds) for patch transition to complete...Stalled processes:
340 kdevtmpfs
stack:[<0>] devtmpfs_work_loop+0x2cc/0x2d8 [<0>] devtmpfsd+0x4c/0x58 [<0>] kthread+0xf0/0x100 [<0>] ret_from_fork+0x10/0x20
Quoted 6 lines
module
livepatch_sampledid not complete its transition, unloading... disabling patch module:livepatch_sample
waiting (up to 15 seconds) for patch transition to complete... transition complete (3 seconds)
unloading patch module:livepatch_sample
<4>kpatch: error: failed to load modulelivepatch_sample(transition stalled)
After some debugging this is what I found:
devtmpfsd() calls devtmpfs_work_loop() which is marked '__noreturn' and has an infinite loop. The compiler puts the bl to devtmpfs_work_loop() as the the last instruction in devtmpfsd() and therefore on entry to devtmpfs_work_loop(), LR points to an instruction beyond devtmpfsd() and this consfuses the unwinder.
ffff800080d9a070 <devtmpfsd>:
ffff800080d9a070: d503201f nop ffff800080d9a074: d503201f nop ffff800080d9a078: d503233f paciasp ffff800080d9a07c: a9be7bfd stp x29, x30, [sp, #-32]! ffff800080d9a080: 910003fd mov x29, sp ffff800080d9a084: f9000bf3 str x19, [sp, #16] ffff800080d9a088: 943378e8 bl ffff800081a78428 <devtmpfs_setup> ffff800080d9a08c: 90006ca1 adrp x1, ffff800081b2e000 <unique_processor_ids+0x3758> ffff800080d9a090: 2a0003f3 mov w19, w0 ffff800080d9a094: 912de021 add x1, x1, #0xb78 ffff800080d9a098: 91002020 add x0, x1, #0x8 ffff800080d9a09c: 97cd2a43 bl ffff8000800e49a8 <complete> ffff800080d9a0a0: 340000d3 cbz w19, ffff800080d9a0b8 <devtmpfsd+0x48> ffff800080d9a0a4: 2a1303e0 mov w0, w19 ffff800080d9a0a8: f9400bf3 ldr x19, [sp, #16] ffff800080d9a0ac: a8c27bfd ldp x29, x30, [sp], #32 ffff800080d9a0b0: d50323bf autiasp ffff800080d9a0b4: d65f03c0 ret ffff800080d9a0b8: 97f06526 bl ffff8000809b3550 <devtmpfs_work_loop> ffff800080d9a0bc: 00000000 udf #0 ffff800080d9a0c0: d503201f nop ffff800080d9a0c4: d503201f nop
find_fde() got pc=0xffff800080d9a0bc which is not in [sfde_func_start_address, sfde_func_size)
output for readelf --sframe for devtmpfsd()
func idx [51825]: pc = 0xffff800080d9a070, size = 76 bytes
STARTPC CFA FP RA
ffff800080d9a070 sp+0 u u
ffff800080d9a07c sp+0 u u[s]
ffff800080d9a080 sp+32 c-32 c-24[s]
ffff800080d9a0b0 sp+0 u u[s]
ffff800080d9a0b4 sp+0 u u
ffff800080d9a0b8 sp+32 c-32 c-24[s]The unwinder and all the related infra is assuming that the return address will be part of a valid function which is not the case here.
I am not sure which component needs to be fixed here, but the following patch(which is a hack) fixes the issue by considering the return address as part of the function descriptor entry.
-- 8< --
kernel/sframe_lookup.c +2 -2
@@ -82,7 +82,7 @@ static struct sframe_fde *find_fde(const struct sframe_table *tbl, unsigned longif (f >= tbl->sfhdr_p->num_fdes || f <0 )return NULL ; fdep = tbl->fde_p + f; -if (ip < fdep->start_addr || ip >= fdep->start_addr + fdep->size) +if (ip < fdep->start_addr || ip > fdep->start_addr + fdep->size)return NULL ;return fdep;
@@ -106,7 +106,7 @@ static int find_fre(const struct sframe_table *tbl, unsigned long pc,else ip_off = (int32_t )(pc - (unsigned long )tbl->sfhdr_p) - fdep->start_addr; -if (ip_off <0 || ip_off >= fdep->size) +if (ip_off <0 || ip_off > fdep->size)return -EINVAL ;/*
-- >8 --
Thanks,
Puranjay
Quoted 78 lines
After some debugging this is what I found:
devtmpfsd()callsdevtmpfs_work_loop()which is marked '__noreturn' and has an infinite loop. The compiler puts thebltodevtmpfs_work_loop()as the the last instruction indevtmpfsd()and therefore on entry todevtmpfs_work_loop(), LR points to an instruction beyonddevtmpfsd()and this consfuses the unwinder.
ffff800080d9a070<devtmpfsd>:ffff800080d9a070: d503201f nop ffff800080d9a074: d503201f nop ffff800080d9a078: d503233f paciasp ffff800080d9a07c: a9be7bfd stp x29, x30, [sp, #-32]! ffff800080d9a080: 910003fd mov x29, sp ffff800080d9a084: f9000bf3 str x19, [sp, #16] ffff800080d9a088: 943378e8 bl ffff800081a78428 <devtmpfs_setup> ffff800080d9a08c: 90006ca1 adrp x1, ffff800081b2e000 <unique_processor_ids+0x3758> ffff800080d9a090: 2a0003f3 mov w19, w0 ffff800080d9a094: 912de021 add x1, x1, #0xb78 ffff800080d9a098: 91002020 add x0, x1, #0x8 ffff800080d9a09c: 97cd2a43 bl ffff8000800e49a8 <complete> ffff800080d9a0a0: 340000d3 cbz w19, ffff800080d9a0b8 <devtmpfsd+0x48> ffff800080d9a0a4: 2a1303e0 mov w0, w19 ffff800080d9a0a8: f9400bf3 ldr x19, [sp, #16] ffff800080d9a0ac: a8c27bfd ldp x29, x30, [sp], #32 ffff800080d9a0b0: d50323bf autiasp ffff800080d9a0b4: d65f03c0 ret ffff800080d9a0b8: 97f06526 bl ffff8000809b3550 <devtmpfs_work_loop> ffff800080d9a0bc: 00000000 udf #0 ffff800080d9a0c0: d503201f nop ffff800080d9a0c4: d503201f nop
find_fde()got pc=0xffff800080d9a0bc which is not in [sfde_func_start_address,sfde_func_size)output for readelf --sframe for
devtmpfsd()func idx [51825]: pc =
0xffff800080d9a070, size = 76 bytesSTARTPC CFA FP RA ffff800080d9a070 sp+0 u u ffff800080d9a07c sp+0 u u[s] ffff800080d9a080 sp+32 c-32 c-24[s] ffff800080d9a0b0 sp+0 u u[s] ffff800080d9a0b4 sp+0 u u ffff800080d9a0b8 sp+32 c-32 c-24[s]The unwinder and all the related infra is assuming that the return address will be part of a valid function which is not the case here.
I am not sure which component needs to be fixed here, but the following patch(which is a hack) fixes the issue by considering the return address as part of the function descriptor entry.
-- 8< --
kernel/sframe_lookup.c +2
-3@@ -82,7 +82,7 @@ static struct sframe_fde *find_fde(const struct sframe_table *tbl, unsigned longif (f >= tbl->sfhdr_p->num_fdes || f <0 )return NULL ; fdep = tbl->fde_p + f; -if (ip < fdep->start_addr || ip >= fdep->start_addr + fdep->size) +if (ip < fdep->start_addr || ip > fdep->start_addr + fdep->size)return NULL ;return fdep;@@ -106,7 +106,7 @@ static int find_fre(const struct sframe_table *tbl, unsigned long pc,else ip_off = (int32_t )(pc - (unsigned long )tbl->sfhdr_p) - fdep->start_addr; -if (ip_off <0 || ip_off >= fdep->size) +if (ip_off <0 || ip_off > fdep->size)return -EINVAL ;/* -- >8 --
Quoted 2 lines
Thanks,
Puranjay
Thank you for reporting this issue.
I just found out that Josh also intentionally uses '>' instead of '>=' for the same reason
https://lore.kernel.org/lkml/20250122225257.h64ftfnorofe7cb4@jpoimboe/T/#m6d70a20ed9f5b3bbe5b24b24b8c5dcc603a79101
QQ, do we need to care the stacktrace after '__noreturn' function?
Quoted 78 lines · Weinan Liu
Quoted 78 lines
After some debugging this is what I found:
devtmpfsd()callsdevtmpfs_work_loop()which is marked '__noreturn' and has an infinite loop. The compiler puts thebltodevtmpfs_work_loop()as the the last instruction indevtmpfsd()and therefore on entry todevtmpfs_work_loop(), LR points to an instruction beyonddevtmpfsd()and this consfuses the unwinder.
ffff800080d9a070<devtmpfsd>:ffff800080d9a070: d503201f nop ffff800080d9a074: d503201f nop ffff800080d9a078: d503233f paciasp ffff800080d9a07c: a9be7bfd stp x29, x30, [sp, #-32]! ffff800080d9a080: 910003fd mov x29, sp ffff800080d9a084: f9000bf3 str x19, [sp, #16] ffff800080d9a088: 943378e8 bl ffff800081a78428 <devtmpfs_setup> ffff800080d9a08c: 90006ca1 adrp x1, ffff800081b2e000 <unique_processor_ids+0x3758> ffff800080d9a090: 2a0003f3 mov w19, w0 ffff800080d9a094: 912de021 add x1, x1, #0xb78 ffff800080d9a098: 91002020 add x0, x1, #0x8 ffff800080d9a09c: 97cd2a43 bl ffff8000800e49a8 <complete> ffff800080d9a0a0: 340000d3 cbz w19, ffff800080d9a0b8 <devtmpfsd+0x48> ffff800080d9a0a4: 2a1303e0 mov w0, w19 ffff800080d9a0a8: f9400bf3 ldr x19, [sp, #16] ffff800080d9a0ac: a8c27bfd ldp x29, x30, [sp], #32 ffff800080d9a0b0: d50323bf autiasp ffff800080d9a0b4: d65f03c0 ret ffff800080d9a0b8: 97f06526 bl ffff8000809b3550 <devtmpfs_work_loop> ffff800080d9a0bc: 00000000 udf #0 ffff800080d9a0c0: d503201f nop ffff800080d9a0c4: d503201f nop
find_fde()got pc=0xffff800080d9a0bc which is not in [sfde_func_start_address,sfde_func_size)output for readelf --sframe for
devtmpfsd()func idx [51825]: pc =
0xffff800080d9a070, size = 76 bytesSTARTPC CFA FP RA ffff800080d9a070 sp+0 u u ffff800080d9a07c sp+0 u u[s] ffff800080d9a080 sp+32 c-32 c-24[s] ffff800080d9a0b0 sp+0 u u[s] ffff800080d9a0b4 sp+0 u u ffff800080d9a0b8 sp+32 c-32 c-24[s]The unwinder and all the related infra is assuming that the return address will be part of a valid function which is not the case here.
I am not sure which component needs to be fixed here, but the following patch(which is a hack) fixes the issue by considering the return address as part of the function descriptor entry.
-- 8< --
kernel/sframe_lookup.c +2
-3@@ -82,7 +82,7 @@ static struct sframe_fde *find_fde(const struct sframe_table *tbl, unsigned longif (f >= tbl->sfhdr_p->num_fdes || f <0 )return NULL ; fdep = tbl->fde_p + f; -if (ip < fdep->start_addr || ip >= fdep->start_addr + fdep->size) +if (ip < fdep->start_addr || ip > fdep->start_addr + fdep->size)return NULL ;return fdep;@@ -106,7 +106,7 @@ static int find_fre(const struct sframe_table *tbl, unsigned long pc,else ip_off = (int32_t )(pc - (unsigned long )tbl->sfhdr_p) - fdep->start_addr; -if (ip_off <0 || ip_off >= fdep->size) +if (ip_off <0 || ip_off > fdep->size)return -EINVAL ;/* -- >8 --
Quoted 8 lines
Quoted 2 lines
Thanks,
PuranjayThank you for reporting this issue.
I just found out that Josh also intentionally uses '>' instead of '>=' for the same reason
https://lore.kernel.org/lkml/20250122225257.h64ftfnorofe7cb4@jpoimboe/T/#m6d70a20ed9f5b3bbe5b24b24b8c5dcc603a79101QQ, do we need to care the stacktrace after '
__noreturn' function?
Yes, I think we should, but others people could add more to this.
I have been testing this series with Kpatch and created a PR that works with this unwinder: https://github.com/dynup/kpatch/pull/1439
For the modules, I think we need per module sframe tables that are initialised when the module is loaded. And the unwinder should use the module specific table if the IP is in a module's code.
Have you already started working on it? if not I would like to help and work on that.
Thanks,
Puranjay
Josh Poimboeuf
maintainerQuoted 8 lines · Puranjay Mohan
Quoted 5 lines · Weinan Liu
Thank you for reporting this issue.
I just found out that Josh also intentionally uses '>' instead of '>=' for the same reason
https://lore.kernel.org/lkml/20250122225257.h64ftfnorofe7cb4@jpoimboe/T/#m6d70a20ed9f5b3bbe5b24b24b8c5dcc603a79101QQ, do we need to care the stacktrace after '
__noreturn' function?Yes, I think we should, but others people could add more to this.
FYI, here's how ORC handles this:
/*
* Find the orc_entry associated with the text address.
*
* For a call frame (as opposed to a signal frame), state->ip points to
* the instruction after the call. That instruction's stack layout
* could be different from the call instruction's layout, for example
* if the call was to a noreturn function. So get the ORC data for the
* call instruction itself.
*/
orc = orc_find(state->signal ? state->ip : state->ip - 1);
and state->signal is only set for exceptions/interrupts (including preemption and page faults) and newly forked tasks which haven't run yet.
Signature · 1 line
Josh
Quoted 10 lines · Puranjay Mohan
Yes, I think we should, but others people could add more to this.
I have been testing this series with Kpatch and created a PR that works with this unwinder: https://github.com/dynup/kpatch/pull/1439
For the modules, I think we need per module sframe tables that are initialised when the module is loaded. And the unwinder should use the module specific table if the IP is in a module's code.
Quoted 5 lines
Have you already started working on it? if not I would like to help and work on that.
Thanks,
Puranjay
Thanks for updating the arm64 kpatch PR so quickly. So we can use kpatch to test this proposal.
I already have a WIP patch to add sframe support to the kernel module. However, it is not yet working. I had trouble unwinding frames for the kernel module using the current algorithm.
Indu has likely identified the issue and will be addressing it from the toolchain side.
Quoted 8 lines · Weinan Liu
I already have a WIP patch to add sframe support to the kernel module. However, it is not yet working. I had trouble unwinding frames for the kernel module using the current algorithm.
Indu has likely identified the issue and will be addressing it from the toolchain side.
I have a working in progress patch that adds sframe support for kernel module.
https://github.com/heuza/linux/tree/sframe_unwinder.rfc
According to the sframe table values I got during runtime testing, looks like the offsets are not correct .
When unwind symbols init_module(0xffff80007b155048) from the kernel module(livepatch-sample.ko), the start_address of the FDE entries in the sframe table of the kernel modules appear incorrect.
For instance, the first FDE's start_addr is reported as -20564. Adding this offset to the module's sframe section address (0xffff80007b15a040) yields 0xffff80007b154fec, which is not within the livepatch-sample.ko memory region(It should be larger than 0xffff80007b155000).
Here are the sframe table values of the livepatch-samples.ko that I print by qemu + gdb.
```
$ /usr/bin/aarch64-linux-gnu-objdump -L --sframe=.sframe ./samples/livepatch/livepatch-sample.ko
./samples/livepatch/livepatch-sample.ko: file format elf64-littleaarch64
Contents of the SFrame section .sframe:
Header :
Version: SFRAME_VERSION_2
Flags: SFRAME_F_FDE_SORTED
Num FDEs: 3
Num FREs: 11
Function Index :
func idx [0]: pc = 0x0, size = 12 bytes
STARTPC CFA FP RA
0000000000000000 sp+0 u u
func idx [1]: pc = 0x0, size = 44 bytes
STARTPC CFA FP RA
0000000000000000 sp+0 u u
000000000000000c sp+0 u u[s]
0000000000000010 sp+16 c-16 c-8[s]
0000000000000024 sp+0 u u[s]
0000000000000028 sp+0 u u
func idx [2]: pc = 0x0, size = 56 bytes
STARTPC CFA FP RA
0000000000000000 sp+0 u u
000000000000000c sp+0 u u[s]
0000000000000010 sp+16 c-16 c-8[s]
0000000000000030 sp+0 u u[s]
0000000000000034 sp+0 u u
(gdb) bt
#0 find_fde (tbl=0xffff80007b157708, pc=18446603338286190664) at kernel/sframe_lookup.c:75
#1 0xffff80008031e260 in sframe_find_pc (pc=18446603338286190664, entry=0xffff800086f83800) at kernel/sframe_lookup.c:175
#2 0xffff800080035a48 in unwind_next_frame_sframe (state=0xffff800086f83828) at arch/arm64/kernel/stacktrace.c:270
#3 kunwind_next (state=0xffff800086f83828) at arch/arm64/kernel/stacktrace.c:332
...
(gdb) lx-symbols
loading vmlinux
scanning for modules in /home/wnliu/kernel
loading @0xffff80007b155000: /home/wnliu/kernel/samples/livepatch/livepatch-sample.ko
loading @0xffff80007b14d000: /home/wnliu/kernel/fs/fat/vfat.ko
loading @0xffff80007b130000: /home/wnliu/kernel/fs/fat/fat.ko
(gdb) p/x *tbl->sfhdr_p
$5 = {preamble = {magic = 0xdee2, version = 0x2, flags = 0x1}, abi_arch = 0x2, cfa_fixed_fp_offset = 0x0, cfa_fixed_ra_offset = 0x0, auxhdr_len = 0x0, num_fdes = 0x3, num_fres = 0xb, fre_len = 0x25, fdes_off = 0x0, fres_off = 0x3c}
(gdb) p/x tbl->sfhdr_p
$6 = 0xffff80007b15a040
(gdb) p *tbl->fde_p
$7 = {start_addr = -20564, size = 12, fres_off = 0, fres_num = 1, info = 0 '\000', rep_size = 0 '\000', padding = 0}
(gdb) p *(tbl->fde_p + 1)
$11 = {start_addr = -20552, size = 44, fres_off = 3, fres_num = 5, info = 0 '\000', rep_size = 0 '\000', padding = 0}
(gdb) p *(tbl->fde_p + 2)
$12 = {start_addr = -20508, size = 56, fres_off = 20, fres_num = 5, info = 0 '\000', rep_size = 0 '\000', padding = 0}
/* -20564 + 0xffff80007b15a040 = 0xffff80007b154fec */
(gdb) info symbol 0xffff80007b154fec
No symbol matches 0xffff80007b154fec
```Josh Poimboeuf
maintainerQuoted 18 lines · Weinan Liu
Quoted 8 lines · Weinan Liu
I already have a WIP patch to add sframe support to the kernel module. However, it is not yet working. I had trouble unwinding frames for the kernel module using the current algorithm.
Indu has likely identified the issue and will be addressing it from the toolchain side.
I have a working in progress patch that adds sframe support for kernel module.
https://github.com/heuza/linux/tree/sframe_unwinder.rfcAccording to the sframe table values I got during runtime testing, looks like the offsets are not correct .
Quoted 7 lines
When unwind symbols
init_module(0xffff80007b155048)from the kernel module(livepatch-sample.ko), thestart_addressof the FDE entries in the sframe table of the kernel modules appear incorrect.
For instance, the first FDE'sstart_addris reported as -20564. Adding this offset to the module's sframe section address (0xffff80007b15a040) yields0xffff80007b154fec, which is not within the livepatch-sample.ko memory region(It should be larger than0xffff80007b155000).
I assume kpatch create-diff-object needs to copy over a subset of the .sframe section. Similar to what kpatch_regenerate_orc_sections() does.
Signature · 1 line
Josh
Quoted 28 lines · Josh Poimboeuf
Quoted 18 lines · Weinan Liu
Quoted 8 lines · Weinan Liu
I already have a WIP patch to add sframe support to the kernel module. However, it is not yet working. I had trouble unwinding frames for the kernel module using the current algorithm.
Indu has likely identified the issue and will be addressing it from the toolchain side.
I have a working in progress patch that adds sframe support for kernel module.
https://github.com/heuza/linux/tree/sframe_unwinder.rfcAccording to the sframe table values I got during runtime testing, looks like the offsets are not correct .
Quoted 7 lines
When unwind symbols
init_module(0xffff80007b155048)from the kernel module(livepatch-sample.ko), thestart_addressof the FDE entries in the sframe table of the kernel modules appear incorrect.
For instance, the first FDE'sstart_addris reported as -20564. Adding this offset to the module's sframe section address (0xffff80007b15a040) yields0xffff80007b154fec, which is not within the livepatch-sample.ko memory region(It should be larger than0xffff80007b155000).
Quoted 2 lines
I assume kpatch create-diff-object needs to copy over a subset of the .sframe section. Similar to what
kpatch_regenerate_orc_sections()does.
You're right that we need to process the sframe section like what kpatch_regenerate_orc_sections() does when building livepatch by kpatch.
However, livepatch-sample.ko is not generated by kpatch. It is built directly from samples/livepatch/livepatch-sample.c by gcc during the kernel build
Quoted 7 lines · Weinan Liu
Quoted 7 lines
I already have a WIP patch to add sframe support to the kernel module. However, it is not yet working. I had trouble unwinding frames for the kernel module using the current algorithm.
Indu has likely identified the issue and will be addressing it from the toolchain side.
Quoted 9 lines
I have a working in progress patch that adds sframe support for kernel module.
https://github.com/heuza/linux/tree/sframe_unwinder.rfcAccording to the sframe table values I got during runtime testing, looks like the offsets are not correct .
I hope to sanitize the fix for 32666 and post upstream soon (I had to address other related issues). Unless fixed, relocating .sframe sections using the .rela.sframe is expected to generate incorrect output.
Quoted 3 lines
When unwind symbols
init_module(0xffff80007b155048)from the kernel module(livepatch-sample.ko), thestart_addressof the FDE entries in the sframe table of the kernel modules appear incorrect.
init_module will apply the relocations on the .sframe section, isnt it ?
Quoted 5 lines
For instance, the first FDE's
start_addris reported as -20564. Adding this offset to the module's sframe section address (0xffff80007b15a040) yields0xffff80007b154fec, which is not within the livepatch-sample.ko memory region(It should be larger than0xffff80007b155000).
Hmm..something seems off here. Having tested a potential fix for 32666 locally, I do not expect the first FDE to show this symptom.
Quoted 72 lines
Here are the sframe table values of the livepatch-samples.ko that I print by qemu + gdb.
``` $ /usr/bin/aarch64-linux-gnu-objdump -L --sframe=.sframe ./samples/livepatch/livepatch-sample.ko ./samples/livepatch/livepatch-sample.ko: file format elf64-littleaarch64 Contents of the SFrame section .sframe: Header : Version: SFRAME_VERSION_2 Flags: SFRAME_F_FDE_SORTED Num FDEs: 3 Num FREs: 11 Function Index : func idx [0]: pc = 0x0, size = 12 bytes STARTPC CFA FP RA 0000000000000000 sp+0 u u func idx [1]: pc = 0x0, size = 44 bytes STARTPC CFA FP RA 0000000000000000 sp+0 u u 000000000000000c sp+0 u u[s] 0000000000000010 sp+16 c-16 c-8[s] 0000000000000024 sp+0 u u[s] 0000000000000028 sp+0 u u func idx [2]: pc = 0x0, size = 56 bytes STARTPC CFA FP RA 0000000000000000 sp+0 u u 000000000000000c sp+0 u u[s] 0000000000000010 sp+16 c-16 c-8[s] 0000000000000030 sp+0 u u[s] 0000000000000034 sp+0 u u (gdb) bt #0 find_fde (tbl=0xffff80007b157708, pc=18446603338286190664) at kernel/sframe_lookup.c:75 #1 0xffff80008031e260 in sframe_find_pc (pc=18446603338286190664, entry=0xffff800086f83800) at kernel/sframe_lookup.c:175 #2 0xffff800080035a48 in unwind_next_frame_sframe (state=0xffff800086f83828) at arch/arm64/kernel/stacktrace.c:270 #3 kunwind_next (state=0xffff800086f83828) at arch/arm64/kernel/stacktrace.c:332 ... (gdb) lx-symbols loading vmlinux scanning for modules in /home/wnliu/kernel loading @0xffff80007b155000: /home/wnliu/kernel/samples/livepatch/livepatch-sample.ko loading @0xffff80007b14d000: /home/wnliu/kernel/fs/fat/vfat.ko loading @0xffff80007b130000: /home/wnliu/kernel/fs/fat/fat.ko (gdb) p/x *tbl->sfhdr_p $5 = {preamble = {magic = 0xdee2, version = 0x2, flags = 0x1}, abi_arch = 0x2, cfa_fixed_fp_offset = 0x0, cfa_fixed_ra_offset = 0x0, auxhdr_len = 0x0, num_fdes = 0x3, num_fres = 0xb, fre_len = 0x25, fdes_off = 0x0, fres_off = 0x3c} (gdb) p/x tbl->sfhdr_p $6 = 0xffff80007b15a040 (gdb) p *tbl->fde_p $7 = {start_addr = -20564, size = 12, fres_off = 0, fres_num = 1, info = 0 '\000', rep_size = 0 '\000', padding = 0} (gdb) p *(tbl->fde_p + 1) $11 = {start_addr = -20552, size = 44, fres_off = 3, fres_num = 5, info = 0 '\000', rep_size = 0 '\000', padding = 0} (gdb) p *(tbl->fde_p + 2) $12 = {start_addr = -20508, size = 56, fres_off = 20, fres_num = 5, info = 0 '\000', rep_size = 0 '\000', padding = 0} /* -20564 + 0xffff80007b15a040 = 0xffff80007b154fec */ (gdb) info symbol 0xffff80007b154fec No symbol matches 0xffff80007b154fec ```
Quoted 29 lines · Indu Bhagat
Quoted 7 lines · Weinan Liu
Quoted 7 lines
I already have a WIP patch to add sframe support to the kernel module. However, it is not yet working. I had trouble unwinding frames for the kernel module using the current algorithm.
Indu has likely identified the issue and will be addressing it from the toolchain side.
Quoted 9 lines
I have a working in progress patch that adds sframe support for kernel module.
https://github.com/heuza/linux/tree/sframe_unwinder.rfcAccording to the sframe table values I got during runtime testing, looks like the offsets are not correct .
I hope to sanitize the fix for 32666 and post upstream soon (I had to address other related issues). Unless fixed, relocating .sframe sections using the .rela.sframe is expected to generate incorrect output.
Quoted 3 lines
When unwind symbols
init_module(0xffff80007b155048)from the kernel module(livepatch-sample.ko), thestart_addressof the FDE entries in the sframe table of the kernel modules appear incorrect.
init_modulewill apply the relocations on the .sframe section, isnt it ?
Quoted 9 lines
Quoted 5 lines
For instance, the first FDE's
start_addris reported as -20564. Adding this offset to the module's sframe section address (0xffff80007b15a040) yields0xffff80007b154fec, which is not within the livepatch-sample.ko memory region(It should be larger than0xffff80007b155000).Hmm..something seems off here. Having tested a potential fix for 32666 locally, I do not expect the first FDE to show this symptom.
Yes, I think init_module will apply the relocation as well. To further investigate, here's the relevant relocation and symbol table information for the kernel module:
Relocation section '.rela.sframe' at offset 0x28350 contains 3 entries:
Offset Info Type Sym. Value Sym. Name + Addend 00000000001c 000100000105 R_AARCH64_PREL32 0000000000000000 .text + 8 000000000030 000100000105 R_AARCH64_PREL32 0000000000000000 .text + 28 000000000044 000100000105 R_AARCH64_PREL32 0000000000000000 .text + 68
Symbol table '.symtab' contains 68 entries:
Num: Value Size Type Bind Vis Ndx Name
0: 0000000000000000 0 NOTYPE LOCAL DEFAULT UND
1: 0000000000000000 0 SECTION LOCAL DEFAULT 1 .text
...
32: 0000000000000008 12 FUNC LOCAL DEFAULT 1 livepatch_exit 33: 0000000000000008 0 NOTYPE LOCAL DEFAULT 3 $d
34: 0000000000000028 44 FUNC LOCAL DEFAULT 1 livepatch_init 35: 0000000000000000 0 NOTYPE LOCAL DEFAULT 9 $d
36: 0000000000000010 0 NOTYPE LOCAL DEFAULT 3 $d
37: 0000000000000068 56 FUNC LOCAL DEFAULT 1 livepatch_cmdlin[...]
...
63: 0000000000000008 12 FUNC GLOBAL DEFAULT 1 cleanup_module 64: 0000000000000000 0 NOTYPE GLOBAL DEFAULT UND klp_enable_patch 65: 0000000000000028 44 FUNC GLOBAL DEFAULT 1 init_module
Quoted 44 lines · Weinan Liu
Quoted 29 lines · Indu Bhagat
Quoted 7 lines · Weinan Liu
Quoted 7 lines
I already have a WIP patch to add sframe support to the kernel module. However, it is not yet working. I had trouble unwinding frames for the kernel module using the current algorithm.
Indu has likely identified the issue and will be addressing it from the toolchain side.
Quoted 9 lines
I have a working in progress patch that adds sframe support for kernel module.
https://github.com/heuza/linux/tree/sframe_unwinder.rfcAccording to the sframe table values I got during runtime testing, looks like the offsets are not correct .
I hope to sanitize the fix for 32666 and post upstream soon (I had to address other related issues). Unless fixed, relocating .sframe sections using the .rela.sframe is expected to generate incorrect output.
Quoted 3 lines
When unwind symbols
init_module(0xffff80007b155048)from the kernel module(livepatch-sample.ko), thestart_addressof the FDE entries in the sframe table of the kernel modules appear incorrect.
init_modulewill apply the relocations on the .sframe section, isnt it ?Quoted 9 lines
Quoted 5 lines
For instance, the first FDE's
start_addris reported as -20564. Adding this offset to the module's sframe section address (0xffff80007b15a040) yields0xffff80007b154fec, which is not within the livepatch-sample.ko memory region(It should be larger than0xffff80007b155000).Hmm..something seems off here. Having tested a potential fix for 32666 locally, I do not expect the first FDE to show this symptom.
Yes, I think
init_modulewill apply the relocation as well. To further investigate, here's the relevant relocation and symbol table information for the kernel module:
Quoted 6 lines
Relocation section '.rela.sframe' at offset
0x28350contains 3 entries:
Offset Info Type Sym. Value Sym. Name + Addend00000000001c000100000105R_AARCH64_PREL320000000000000000 .text + 8 000000000030 000100000105R_AARCH64_PREL320000000000000000 .text + 28 000000000044 000100000105R_AARCH64_PREL320000000000000000 .text + 68
The offsets look OK..
Quoted 15 lines
Symbol table '.symtab' contains 68 entries:
Num: Value Size Type Bind Vis Ndx Name
0: 0000000000000000 0 NOTYPE LOCAL DEFAULT UND
1: 0000000000000000 0 SECTION LOCAL DEFAULT 1 .text
...
32: 0000000000000008 12 FUNC LOCAL DEFAULT 1livepatch_exit33: 0000000000000008 0 NOTYPE LOCAL DEFAULT 3 $d
34: 0000000000000028 44 FUNC LOCAL DEFAULT 1livepatch_init35: 0000000000000000 0 NOTYPE LOCAL DEFAULT 9 $d
36: 0000000000000010 0 NOTYPE LOCAL DEFAULT 3 $d
37: 0000000000000068 56 FUNC LOCAL DEFAULT 1 livepatch_cmdlin[...]
...
63: 0000000000000008 12 FUNC GLOBAL DEFAULT 1cleanup_module64: 0000000000000000 0 NOTYPE GLOBAL DEFAULT UNDklp_enable_patch65: 0000000000000028 44 FUNC GLOBAL DEFAULT 1init_module
Quoted 37 lines · Indu Bhagat
Quoted 36 lines · Weinan Liu
Quoted 35 lines · Indu Bhagat
Quoted 7 lines · Weinan Liu
Quoted 7 lines
I already have a WIP patch to add sframe support to the kernel module. However, it is not yet working. I had trouble unwinding frames for the kernel module using the current algorithm.
Indu has likely identified the issue and will be addressing it from the toolchain side.
Quoted 9 lines
I have a working in progress patch that adds sframe support for kernel module.
https://github.com/heuza/linux/tree/sframe_unwinder.rfcAccording to the sframe table values I got during runtime testing, looks like the offsets are not correct .
I hope to sanitize the fix for 32666 and post upstream soon (I had to address other related issues). Unless fixed, relocating .sframe sections using the .rela.sframe is expected to generate incorrect output.
Quoted 3 lines
When unwind symbols
init_module(0xffff80007b155048)from the kernel module(livepatch-sample.ko), thestart_addressof the FDE entries in the sframe table of the kernel modules appear incorrect.
init_modulewill apply the relocations on the .sframe section, isnt it ?Quoted 5 lines
For instance, the first FDE's
start_addris reported as -20564. Adding this offset to the module's sframe section address (0xffff80007b15a040) yields0xffff80007b154fec, which is not within the livepatch-sample.ko memory region(It should be larger than0xffff80007b155000).
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
Hmm..something seems off here. Having tested a potential fix for 32666 locally, I do not expect the first FDE to show this symptom.
Hi,
Sorry for not responding in the past few days. I was on PTO and was trying to improve my snowboarding technique, I am back now!!
I think what we are seeing is expected behaviour:
| For instance, the first FDE's start_addr is reported as -20564. Adding | this offset to the module's sframe section address (0xffff80007b15a040) | yields 0xffff80007b154fec, which is not within the livepatch-sample.ko | memory region(It should be larger than 0xffff80007b155000).
Let me explain using a __dummy__ example.
Assume Memory layout before relocation:
| Address | Element | Relocation | .... | .... | | 60 | init_module (start address) | | 72 | init_module (end address) | | .... | ..... | | 100 | Sframe section header start address | | 128 | First FDE's start address | RELOC_OP_PREL -> Put init_module address (60) - current address (128)
So, after relocation First FDE's start address has value 60 - 128 = -68
Now, while doing unwinding we Try to add this value to the sframe section header's start address which is in this example 100,
so 100 + (-68) = 32
So, 32 is not within [60, 72], i.e. within init_module.
You can see that it is possible for this value to be less than the start address of the module's memory region when this function's address is very close to the start of the memory region.
The crux is that the offset in the FDE's start address is calculated based on the address of the FDE's start_address and not based on the address of the sframe section.
Thanks,
Puranjay
Quoted 69 lines · Puranjay Mohan
Quoted 37 lines · Indu Bhagat
Quoted 36 lines · Weinan Liu
Quoted 35 lines · Indu Bhagat
Quoted 7 lines · Weinan Liu
Quoted 7 lines
I already have a WIP patch to add sframe support to the kernel module. However, it is not yet working. I had trouble unwinding frames for the kernel module using the current algorithm.
Indu has likely identified the issue and will be addressing it from the toolchain side.
Quoted 9 lines
I have a working in progress patch that adds sframe support for kernel module.
https://github.com/heuza/linux/tree/sframe_unwinder.rfcAccording to the sframe table values I got during runtime testing, looks like the offsets are not correct .
I hope to sanitize the fix for 32666 and post upstream soon (I had to address other related issues). Unless fixed, relocating .sframe sections using the .rela.sframe is expected to generate incorrect output.
Quoted 3 lines
When unwind symbols
init_module(0xffff80007b155048)from the kernel module(livepatch-sample.ko), thestart_addressof the FDE entries in the sframe table of the kernel modules appear incorrect.
init_modulewill apply the relocations on the .sframe section, isnt it ?Quoted 5 lines
For instance, the first FDE's
start_addris reported as -20564. Adding this offset to the module's sframe section address (0xffff80007b15a040) yields0xffff80007b154fec, which is not within the livepatch-sample.ko memory region(It should be larger than0xffff80007b155000).Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
Hmm..something seems off here. Having tested a potential fix for 32666 locally, I do not expect the first FDE to show this symptom.
Hi,
Sorry for not responding in the past few days. I was on PTO and was trying to improve my snowboarding technique, I am back now!!
I think what we are seeing is expected behaviour:
| For instance, the first FDE's start_addr is reported as -20564. Adding | this offset to the module's sframe section address (0xffff80007b15a040) | yields 0xffff80007b154fec, which is not within the livepatch-sample.ko | memory region(It should be larger than 0xffff80007b155000).Let me explain using a
__dummy__example.Assume Memory layout before relocation:
| Address | Element | Relocation | .... | .... | | 60 | init_module (start address) | | 72 | init_module (end address) | | .... | ..... | | 100 | Sframe section header start address | | 128 | First FDE's start address | RELOC_OP_PREL -> Put init_module address (60) - current address (128)
Quoted 2 lines
So, after relocation First FDE's start address has value 60 - 128 = -68
For SFrame FDE function start address is :
"Signed 32-bit integral field denoting the virtual memory address of the described function, for which the SFrame FDE applies. The value encoded in the ‘sfde_func_start_address’ field is the offset in bytes of the function’s start address, from the SFrame section."
So, in your case, after applying the relocations, you will get:
S + A - P = 60 - 128 = -68
This is the distance of the function start address (60) from the current location in SFrame section (128)
But what we intend to store is the distance of the function start address from the start of the SFrame section. So we need to do an additional step for SFrame FDE: Value += r_offset
-68 + 28 = -40
Where 28 is the r_offset in the RELA.
So we really expect a -40 in the relocated SFrame section instead of -68 above. IOW, the RELAs of SFrame section will need an additional step after relocation.
Quoted 18 lines
Now, while doing unwinding we Try to add this value to the sframe section header's start address which is in this example 100,
so 100 + (-68) = 32
So, 32 is not within [60, 72], i.e. within
init_module.You can see that it is possible for this value to be less than the start address of the module's memory region when this function's address is very close to the start of the memory region.
The crux is that the offset in the FDE's start address is calculated based on the address of the FDE's
start_addressand not based on the address of the sframe section.Thanks,
Puranjay
Quoted 83 lines · Indu Bhagat
Quoted 69 lines · Puranjay Mohan
Quoted 37 lines · Indu Bhagat
Quoted 36 lines · Weinan Liu
Quoted 35 lines · Indu Bhagat
Quoted 7 lines · Weinan Liu
Quoted 7 lines
I already have a WIP patch to add sframe support to the kernel module. However, it is not yet working. I had trouble unwinding frames for the kernel module using the current algorithm.
Indu has likely identified the issue and will be addressing it from the toolchain side.
Quoted 9 lines
I have a working in progress patch that adds sframe support for kernel module.
https://github.com/heuza/linux/tree/sframe_unwinder.rfcAccording to the sframe table values I got during runtime testing, looks like the offsets are not correct .
I hope to sanitize the fix for 32666 and post upstream soon (I had to address other related issues). Unless fixed, relocating .sframe sections using the .rela.sframe is expected to generate incorrect output.
Quoted 3 lines
When unwind symbols
init_module(0xffff80007b155048)from the kernel module(livepatch-sample.ko), thestart_addressof the FDE entries in the sframe table of the kernel modules appear incorrect.
init_modulewill apply the relocations on the .sframe section, isnt it ?Quoted 5 lines
For instance, the first FDE's
start_addris reported as -20564. Adding this offset to the module's sframe section address (0xffff80007b15a040) yields0xffff80007b154fec, which is not within the livepatch-sample.ko memory region(It should be larger than0xffff80007b155000).Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
Hmm..something seems off here. Having tested a potential fix for 32666 locally, I do not expect the first FDE to show this symptom.
Hi,
Sorry for not responding in the past few days. I was on PTO and was trying to improve my snowboarding technique, I am back now!!
I think what we are seeing is expected behaviour:
| For instance, the first FDE's start_addr is reported as -20564. Adding | this offset to the module's sframe section address (0xffff80007b15a040) | yields 0xffff80007b154fec, which is not within the livepatch-sample.ko | memory region(It should be larger than 0xffff80007b155000).Let me explain using a
__dummy__example.Assume Memory layout before relocation:
| Address | Element | Relocation | .... | .... | | 60 | init_module (start address) | | 72 | init_module (end address) | | .... | ..... | | 100 | Sframe section header start address | | 128 | First FDE's start address | RELOC_OP_PREL -> Put init_module address (60) - current address (128)Quoted 2 lines
So, after relocation First FDE's start address has value 60 - 128 = -68
For SFrame FDE function start address is :
"Signed 32-bit integral field denoting the virtual memory address of the described function, for which the SFrame FDE applies. The value encoded in the ‘sfde_func_start_address’ field is the offset in bytes of the function’s start address, from the SFrame section."
So, in your case, after applying the relocations, you will get:
S + A - P = 60 - 128 = -68
Quoted 6 lines
This is the distance of the function start address (60) from the current location in SFrame section (128)
But what we intend to store is the distance of the function start address from the start of the SFrame section. So we need to do an additional step for SFrame FDE: Value +=
r_offset
Thanks for the explaination, now it makes sense.
But I couldn't find a relocation type in AARCH64 that does this extra += r_offset along with PREL32.
The kernel's module loader is only doing the R_AARCH64_PREL32 which is why we see this issue.
How is this working even for the kernel itself? or for that matter, any other binary compiled with sframe?
From my limited undestanding, the way to fix this would be to hack the relocator to do this additional step while relocating .sframe sections. Or the 'addend' values in .rela.sframe should already have the +r_offset added to it, then no change to the relocator would be needed.
Quoted 7 lines
-68 + 28 = -40
Where 28 is ther_offsetin the RELA.So we really expect a -40 in the relocated SFrame section instead of -68 above. IOW, the RELAs of SFrame section will need an additional step after relocation.
Thanks,
Puranjay
Quoted 94 lines · Puranjay Mohan
Quoted 83 lines · Indu Bhagat
Quoted 69 lines · Puranjay Mohan
Quoted 37 lines · Indu Bhagat
Quoted 36 lines · Weinan Liu
Quoted 35 lines · Indu Bhagat
Quoted 7 lines · Weinan Liu
Quoted 7 lines
I already have a WIP patch to add sframe support to the kernel module. However, it is not yet working. I had trouble unwinding frames for the kernel module using the current algorithm.
Indu has likely identified the issue and will be addressing it from the toolchain side.
Quoted 9 lines
I have a working in progress patch that adds sframe support for kernel module.
https://github.com/heuza/linux/tree/sframe_unwinder.rfcAccording to the sframe table values I got during runtime testing, looks like the offsets are not correct .
I hope to sanitize the fix for 32666 and post upstream soon (I had to address other related issues). Unless fixed, relocating .sframe sections using the .rela.sframe is expected to generate incorrect output.
Quoted 3 lines
When unwind symbols
init_module(0xffff80007b155048)from the kernel module(livepatch-sample.ko), thestart_addressof the FDE entries in the sframe table of the kernel modules appear incorrect.
init_modulewill apply the relocations on the .sframe section, isnt it ?Quoted 5 lines
For instance, the first FDE's
start_addris reported as -20564. Adding this offset to the module's sframe section address (0xffff80007b15a040) yields0xffff80007b154fec, which is not within the livepatch-sample.ko memory region(It should be larger than0xffff80007b155000).Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
Hmm..something seems off here. Having tested a potential fix for 32666 locally, I do not expect the first FDE to show this symptom.
Hi,
Sorry for not responding in the past few days. I was on PTO and was trying to improve my snowboarding technique, I am back now!!
I think what we are seeing is expected behaviour:
| For instance, the first FDE's start_addr is reported as -20564. Adding | this offset to the module's sframe section address (0xffff80007b15a040) | yields 0xffff80007b154fec, which is not within the livepatch-sample.ko | memory region(It should be larger than 0xffff80007b155000).Let me explain using a
__dummy__example.Assume Memory layout before relocation:
| Address | Element | Relocation | .... | .... | | 60 | init_module (start address) | | 72 | init_module (end address) | | .... | ..... | | 100 | Sframe section header start address | | 128 | First FDE's start address | RELOC_OP_PREL -> Put init_module address (60) - current address (128)Quoted 2 lines
So, after relocation First FDE's start address has value 60 - 128 = -68
For SFrame FDE function start address is :
"Signed 32-bit integral field denoting the virtual memory address of the described function, for which the SFrame FDE applies. The value encoded in the ‘sfde_func_start_address’ field is the offset in bytes of the function’s start address, from the SFrame section."
So, in your case, after applying the relocations, you will get:
S + A - P = 60 - 128 = -68Quoted 6 lines
This is the distance of the function start address (60) from the current location in SFrame section (128)
But what we intend to store is the distance of the function start address from the start of the SFrame section. So we need to do an additional step for SFrame FDE: Value +=
r_offsetThanks for the explaination, now it makes sense.
Quoted 9 lines
But I couldn't find a relocation type in AARCH64 that does this extra +=
r_offsetalong with PREL32.The kernel's module loader is only doing the
R_AARCH64_PREL32which is why we see this issue.How is this working even for the kernel itself? or for that matter, any other binary compiled with sframe?
For the usual executables or shared objects, the calculations are applied by ld.bfd at this time. Hence, the issue manifests in relocatable files.
Quoted 5 lines
From my limited undestanding, the way to fix this would be to hack the
relocator to do this additional step while relocating .sframe sections. Or the 'addend' values in .rela.sframe should already have the +r_offset added to it, then no change to the relocator would be needed.
Of the two, adjusting the addend values in .rela.sframe may be a reasonable way to go about it. Let me try it out in GAS and ld.bfd.
Quoted 10 lines
Quoted 7 lines
-68 + 28 = -40
Where 28 is ther_offsetin the RELA.So we really expect a -40 in the relocated SFrame section instead of -68 above. IOW, the RELAs of SFrame section will need an additional step after relocation.
Thanks,
Puranjay
Quoted 107 lines · Indu Bhagat
Quoted 94 lines · Puranjay Mohan
Quoted 83 lines · Indu Bhagat
Quoted 69 lines · Puranjay Mohan
Quoted 37 lines · Indu Bhagat
Quoted 36 lines · Weinan Liu
Quoted 35 lines · Indu Bhagat
Quoted 7 lines · Weinan Liu
Quoted 7 lines
I already have a WIP patch to add sframe support to the kernel module. However, it is not yet working. I had trouble unwinding frames for the kernel module using the current algorithm.
Indu has likely identified the issue and will be addressing it from the toolchain side.
Quoted 9 lines
I have a working in progress patch that adds sframe support for kernel module.
https://github.com/heuza/linux/tree/sframe_unwinder.rfcAccording to the sframe table values I got during runtime testing, looks like the offsets are not correct .
I hope to sanitize the fix for 32666 and post upstream soon (I had to address other related issues). Unless fixed, relocating .sframe sections using the .rela.sframe is expected to generate incorrect output.
Quoted 3 lines
When unwind symbols
init_module(0xffff80007b155048)from the kernel module(livepatch-sample.ko), thestart_addressof the FDE entries in the sframe table of the kernel modules appear incorrect.
init_modulewill apply the relocations on the .sframe section, isnt it ?Quoted 5 lines
For instance, the first FDE's
start_addris reported as -20564. Adding this offset to the module's sframe section address (0xffff80007b15a040) yields0xffff80007b154fec, which is not within the livepatch- sample.ko memory region(It should be larger than0xffff80007b155000).Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
Hmm..something seems off here. Having tested a potential fix for 32666 locally, I do not expect the first FDE to show this symptom.
Hi,
Sorry for not responding in the past few days. I was on PTO and was trying to improve my snowboarding technique, I am back now!!
I think what we are seeing is expected behaviour:
| For instance, the first FDE's start_addr is reported as -20564. Adding | this offset to the module's sframe section address (0xffff80007b15a040) | yields 0xffff80007b154fec, which is not within the livepatch- sample.ko | memory region(It should be larger than 0xffff80007b155000).Let me explain using a
__dummy__example.Assume Memory layout before relocation:
| Address | Element | Relocation | .... | .... | | 60 | init_module (start address) | | 72 | init_module (end address) | | .... | ..... | | 100 | Sframe section header start address | | 128 | First FDE's start address | RELOC_OP_PREL -> Put init_module address (60) - current address (128)Quoted 2 lines
So, after relocation First FDE's start address has value 60 - 128 = -68
For SFrame FDE function start address is :
"Signed 32-bit integral field denoting the virtual memory address of the described function, for which the SFrame FDE applies. The value encoded in the ‘sfde_func_start_address’ field is the offset in bytes of the function’s start address, from the SFrame section."
So, in your case, after applying the relocations, you will get:
S + A - P = 60 - 128 = -68Quoted 6 lines
This is the distance of the function start address (60) from the current location in SFrame section (128)
But what we intend to store is the distance of the function start address from the start of the SFrame section. So we need to do an additional step for SFrame FDE: Value +=
r_offsetThanks for the explaination, now it makes sense.
Quoted 9 lines
But I couldn't find a relocation type in AARCH64 that does this extra +=
r_offsetalong with PREL32.The kernel's module loader is only doing the
R_AARCH64_PREL32which is why we see this issue.How is this working even for the kernel itself? or for that matter, any other binary compiled with sframe?
For the usual executables or shared objects, the calculations are applied by ld.bfd at this time. Hence, the issue manifests in relocatable files.
Quoted 8 lines
Quoted 5 lines
From my limited undestanding, the way to fix this would be to hack the relocator to do this additional step while relocating .sframe sections. Or the 'addend' values in .rela.sframe should already have the +r_offset added to it, then no change to the relocator would be needed.
Of the two, adjusting the addend values in .rela.sframe may be a reasonable way to go about it. Let me try it out in GAS and ld.bfd.
A fix for this is in the works (being discussed on the binutils@sourceware list). I will keep you posted.
Thanks
Indu
Song Liu
maintainerI run some tests with this set and my RFC set [1]. Most of the test is done with kpatch-build. I tested both Puranjay's version [3] and my version [4].
For gcc 14.2.1, I have seen the following issue with this test [2]. This happens with both upstream and 6.13.2. The livepatch loaded fine, but the system spilled out the following warning quickly.
On the other hand, the same test works with LLVM and my RFC set (LLVM doesn't support SFRAME, and thus doesn't work with this set yet).
Thanks,
Song
[ 81.250437] ------------[ cut here ]------------ [ 81.250818] refcount_t: saturated; leaking memory. [ 81.251201] WARNING: CPU: 0 PID: 95 at lib/refcount.c:22 refcount_warn_saturate+0x6c/0x140 [ 81.251841] Modules linked in: livepatch_special_static(OEK) [ 81.252277] CPU: 0 UID: 0 PID: 95 Comm: bash Tainted: G OE K 6.13.2-00321-g52d2813b4b07 #49 [ 81.253003] Tainted: [O]=OOT_MODULE, [E]=UNSIGNED_MODULE, [K]=LIVEPATCH [ 81.253503] Hardware name: linux,dummy-virt (DT) [ 81.253856] pstate: 634000c5 (nZCv daIF +PAN -UAO +TCO +DIT -SSBS BTYPE=--) [ 81.254383] pc : refcount_warn_saturate+0x6c/0x140 [ 81.254748] lr : refcount_warn_saturate+0x6c/0x140 [ 81.255114] sp : ffff800085a6fc00 [ 81.255371] x29: ffff800085a6fc00 x28: 0000000001200000 x27: ffff0000c2966180 [ 81.255918] x26: 0000000000000000 x25: ffff8000829c0000 x24: ffff0000c2e9b608 [ 81.256462] x23: ffff800083351000 x22: ffff0000c2e9af80 x21: ffff0000c062e140 [ 81.257006] x20: ffff0000c1c10c00 x19: ffff800085a6fd80 x18: ffffffffffffffff [ 81.257544] x17: 0000000000000001 x16: ffffffffffffffff x15: 0000000000000006 [ 81.258083] x14: 0000000000000000 x13: 2e79726f6d656d20 x12: 676e696b61656c20 [ 81.258625] x11: ffff8000829f7d70 x10: 0000000000000147 x9 : ffff8000801546b4 [ 81.259165] x8 : 00000000fffeffff x7 : 00000000ffff0000 x6 : ffff800082f77d70 [ 81.259709] x5 : 80000000ffff0000 x4 : 0000000000000000 x3 : 0000000000000001 [ 81.260257] x2 : ffff8000829f7a88 x1 : ffff8000829f7a88 x0 : 0000000000000026 [ 81.260824] Call trace: [ 81.261015] refcount_warn_saturate+0x6c/0x140 (P) [ 81.261387] __refcount_add.constprop.0+0x60/0x70 [ 81.261748] copy_process+0xfdc/0xfd58 [livepatch_special_static] [ 81.262217] kernel_clone+0x80/0x3e0 [ 81.262499] __do_sys_clone+0x5c/0x88 [ 81.262786] __arm64_sys_clone+0x24/0x38 [ 81.263085] invoke_syscall+0x4c/0x108 [ 81.263378] el0_svc_common.constprop.0+0x44/0xe8 [ 81.263734] do_el0_svc+0x20/0x30 [ 81.263993] el0_svc+0x34/0xf8 [ 81.264231] el0t_64_sync_handler+0x104/0x130 [ 81.264561] el0t_64_sync+0x184/0x188 [ 81.264846] ---[ end trace 0000000000000000 ]--- [ 82.335559] ------------[ cut here ]------------ [ 82.335931] refcount_t: underflow; use-after-free. [ 82.336307] WARNING: CPU: 1 PID: 0 at lib/refcount.c:28 refcount_warn_saturate+0xec/0x140 [ 82.336949] Modules linked in: livepatch_special_static(OEK) [ 82.337389] CPU: 1 UID: 0 PID: 0 Comm: swapper/1 Tainted: G W OE K 6.13.2-00321-g52d2813b4b07 #49 [ 82.338148] Tainted: [W]=WARN, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE, [K]=LIVEPATCH [ 82.338721] Hardware name: linux,dummy-virt (DT) [ 82.339083] pstate: 63400005 (nZCv daif +PAN -UAO +TCO +DIT -SSBS BTYPE=--) [ 82.339617] pc : refcount_warn_saturate+0xec/0x140 [ 82.340007] lr : refcount_warn_saturate+0xec/0x140 [ 82.340378] sp : ffff80008370fe40 [ 82.340637] x29: ffff80008370fe40 x28: 0000000000000000 x27: 0000000000000000 [ 82.341188] x26: 000000000000000a x25: ffff0000fdaf7ab8 x24: 0000000000000014 [ 82.341737] x23: ffff8000829c8d30 x22: ffff80008370ff28 x21: ffff0000fe020000 [ 82.342286] x20: ffff0000c062e140 x19: ffff0000c2e9af80 x18: ffffffffffffffff [ 82.342839] x17: ffff80007b7a0000 x16: ffff800083700000 x15: 0000000000000006 [ 82.343389] x14: 0000000000000000 x13: 2e656572662d7265 x12: 7466612d65737520 [ 82.343944] x11: ffff8000829f7d70 x10: 000000000000016a x9 : ffff8000801546b4 [ 82.344499] x8 : 00000000fffeffff x7 : 00000000ffff0000 x6 : ffff800082f77d70 [ 82.345051] x5 : 80000000ffff0000 x4 : 0000000000000000 x3 : 0000000000000001 [ 82.345604] x2 : ffff8000829f7a88 x1 : ffff8000829f7a88 x0 : 0000000000000026 [ 82.346163] Call trace: [ 82.346359] refcount_warn_saturate+0xec/0x140 (P) [ 82.346736] __put_task_struct+0x130/0x170 [ 82.347063] delayed_put_task_struct+0xbc/0xe8 [ 82.347411] rcu_core+0x20c/0x5f8 [ 82.347680] rcu_core_si+0x14/0x28 [ 82.347952] handle_softirqs+0x124/0x308 [ 82.348260] __do_softirq+0x18/0x20 [ 82.348536] ____do_softirq+0x14/0x28 [ 82.348828] call_on_irq_stack+0x24/0x30 [ 82.349137] do_softirq_own_stack+0x20/0x38 [ 82.349465] __irq_exit_rcu+0xcc/0x108 [ 82.349764] irq_exit_rcu+0x14/0x28 [ 82.350038] el1_interrupt+0x34/0x50 [ 82.350321] el1h_64_irq_handler+0x14/0x20 [ 82.350642] el1h_64_irq+0x6c/0x70 [ 82.350911] default_idle_call+0x30/0xd0 (P) [ 82.351248] do_idle+0x1d0/0x200 [ 82.351506] cpu_startup_entry+0x38/0x48 [ 82.351818] secondary_start_kernel+0x124/0x150 [ 82.352176] __secondary_switched+0xac/0xb0 [ 82.352505] ---[ end trace 0000000000000000 ]---
[1] SFRAME-less livepatch RFC
https://lore.kernel.org/live-patching/[email protected]/
[2] special-static test from kpatch
https://github.com/dynup/kpatch/blob/master/test/integration/linux-6.2.0/special-static.patch
[3] Puranjay's kpatch with arm64 support
https://github.com/puranjaymohan/kpatch/tree/arm64
[4] My version of kpatch with arm64 and LTO support
https://github.com/liu-song-6/kpatch/tree/fb-6.13-v2
Quoted 58 lines · Weinan Liu
This patchset implements a generic kernel sframe-based [1] unwinder. The main goal is to support reliable stacktraces on arm64.
On x86 orc unwinder provides reliable stacktraces. But arm64 misses the required support from objtool: it cannot generate orc unwind tables for arm64.
Currently, there's already a sframe unwinder proposed for userspace: [2]. Since the sframe unwind table algorithm is similar, these two proposal could integrate common functionality in the future.
There are some incomplete features or challenges:
- The unwinder doesn't yet work with kernel modules. Thestart_addrof FRE from kernel modules doesn't appear correct, preventing us from unwinding functions from kernel modules.
- Currently, only GCC supports sframe.Ref:
[1]: https://sourceware.org/binutils/docs/sframe-spec.html
[2]: https://lore.kernel.org/lkml/[email protected]/Madhavan T. Venkataraman (1):
arm64: DefineTIF_PATCH_PENDINGfor livepatchWeinan Liu (7):
unwind: build kernel with sframe info
arm64: entry: add unwind info for various kernel entries
unwind: add sframe v2 header
unwind: Implement generic sframe unwinder library
unwind: arm64: Add sframe unwinder on arm64
unwind: arm64: add reliable stacktrace support for arm64
arm64: Enable livepatch for ARM64Makefile | 6 + arch/Kconfig | 8 + arch/arm64/Kconfig | 3 + arch/arm64/Kconfig.debug | 10 + arch/arm64/include/asm/stacktrace/common.h | 6 + arch/arm64/include/asm/thread_info.h | 4 +- arch/arm64/kernel/entry-common.c | 4 + arch/arm64/kernel/entry.S | 10 + arch/arm64/kernel/setup.c | 2 + arch/arm64/kernel/stacktrace.c | 102 ++++++++++ include/asm-generic/vmlinux.lds.h | 12 ++ include/linux/sframe_lookup.h | 43 +++++ kernel/Makefile | 1 + kernel/sframe.h | 215 +++++++++++++++++++++ kernel/sframe_lookup.c | 196 +++++++++++++++++++ 15 files changed, 621 insertions(+), 1 deletion(-) create mode 100644 include/linux/sframe_lookup.h create mode 100644 kernel/sframe.h create mode 100644 kernel/sframe_lookup.c--
2.48.1.262.g85cc9f2d1e-goog
Josh Poimboeuf
maintainerQuoted 19 lines · Song Liu
[ 81.250437] ------------[ cut here ]------------ [ 81.250818] refcount_t: saturated; leaking memory. [ 81.251201] WARNING: CPU: 0 PID: 95 at lib/refcount.c:22 refcount_warn_saturate+0x6c/0x140 [ 81.251841] Modules linked in: livepatch_special_static(OEK) [ 81.252277] CPU: 0 UID: 0 PID: 95 Comm: bash Tainted: G OE K 6.13.2-00321-g52d2813b4b07 #49 [ 81.253003] Tainted: [O]=OOT_MODULE, [E]=UNSIGNED_MODULE, [K]=LIVEPATCH [ 81.253503] Hardware name: linux,dummy-virt (DT) [ 81.253856] pstate: 634000c5 (nZCv daIF +PAN -UAO +TCO +DIT -SSBS BTYPE=--) [ 81.254383] pc : refcount_warn_saturate+0x6c/0x140 [ 81.254748] lr : refcount_warn_saturate+0x6c/0x140 [ 81.255114] sp : ffff800085a6fc00 [ 81.255371] x29: ffff800085a6fc00 x28: 0000000001200000 x27: ffff0000c2966180 [ 81.255918] x26: 0000000000000000 x25: ffff8000829c0000 x24: ffff0000c2e9b608 [ 81.256462] x23: ffff800083351000 x22: ffff0000c2e9af80 x21: ffff0000c062e140 [ 81.257006] x20: ffff0000c1c10c00 x19: ffff800085a6fd80 x18: ffffffffffffffff [ 81.257544] x17: 0000000000000001 x16: ffffffffffffffff x15: 0000000000000006 [ 81.258083] x14: 0000000000000000 x13: 2e79726f6d656d20 x12: 676e696b61656c20
Quoted 8 lines
[ 81.258625] x11: ffff8000829f7d70 x10: 0000000000000147 x9 : ffff8000801546b4 [ 81.259165] x8 : 00000000fffeffff x7 : 00000000ffff0000 x6 : ffff800082f77d70 [ 81.259709] x5 : 80000000ffff0000 x4 : 0000000000000000 x3 : 0000000000000001 [ 81.260257] x2 : ffff8000829f7a88 x1 : ffff8000829f7a88 x0 : 0000000000000026 [ 81.260824] Call trace: [ 81.261015] refcount_warn_saturate+0x6c/0x140 (P) [ 81.261387] __refcount_add.constprop.0+0x60/0x70 [ 81.261748] copy_process+0xfdc/0xfd58 [livepatch_special_static]
Does that copy_process+0xfdc/0xfd58 resolve to this line in copy_process()?
refcount_inc(¤t->signal->sigcnt);
Maybe the klp rela reference to 'current' is bogus, or resolving to the wrong address somehow?
Signature · 1 line
Josh
Song Liu
maintainerQuoted 30 lines · Josh Poimboeuf
Quoted 19 lines · Song Liu
[ 81.250437] ------------[ cut here ]------------ [ 81.250818] refcount_t: saturated; leaking memory. [ 81.251201] WARNING: CPU: 0 PID: 95 at lib/refcount.c:22 refcount_warn_saturate+0x6c/0x140 [ 81.251841] Modules linked in: livepatch_special_static(OEK) [ 81.252277] CPU: 0 UID: 0 PID: 95 Comm: bash Tainted: G OE K 6.13.2-00321-g52d2813b4b07 #49 [ 81.253003] Tainted: [O]=OOT_MODULE, [E]=UNSIGNED_MODULE, [K]=LIVEPATCH [ 81.253503] Hardware name: linux,dummy-virt (DT) [ 81.253856] pstate: 634000c5 (nZCv daIF +PAN -UAO +TCO +DIT -SSBS BTYPE=--) [ 81.254383] pc : refcount_warn_saturate+0x6c/0x140 [ 81.254748] lr : refcount_warn_saturate+0x6c/0x140 [ 81.255114] sp : ffff800085a6fc00 [ 81.255371] x29: ffff800085a6fc00 x28: 0000000001200000 x27: ffff0000c2966180 [ 81.255918] x26: 0000000000000000 x25: ffff8000829c0000 x24: ffff0000c2e9b608 [ 81.256462] x23: ffff800083351000 x22: ffff0000c2e9af80 x21: ffff0000c062e140 [ 81.257006] x20: ffff0000c1c10c00 x19: ffff800085a6fd80 x18: ffffffffffffffff [ 81.257544] x17: 0000000000000001 x16: ffffffffffffffff x15: 0000000000000006 [ 81.258083] x14: 0000000000000000 x13: 2e79726f6d656d20 x12: 676e696b61656c20Quoted 8 lines
[ 81.258625] x11: ffff8000829f7d70 x10: 0000000000000147 x9 : ffff8000801546b4 [ 81.259165] x8 : 00000000fffeffff x7 : 00000000ffff0000 x6 : ffff800082f77d70 [ 81.259709] x5 : 80000000ffff0000 x4 : 0000000000000000 x3 : 0000000000000001 [ 81.260257] x2 : ffff8000829f7a88 x1 : ffff8000829f7a88 x0 : 0000000000000026 [ 81.260824] Call trace: [ 81.261015] refcount_warn_saturate+0x6c/0x140 (P) [ 81.261387] __refcount_add.constprop.0+0x60/0x70 [ 81.261748] copy_process+0xfdc/0xfd58 [livepatch_special_static]
Quoted 7 lines
Does that
copy_process+0xfdc/0xfd58resolve to this line incopy_process()?refcount_inc(¤t->signal->sigcnt);Maybe the klp rela reference to 'current' is bogus, or resolving to the wrong address somehow?
It resolves the following line.
p->signal->tty = tty_kref_get(current->signal->tty);
I am not quite sure how 'current' should be resolved.
The size of copy_process (0xfd58) is wrong. It is only about 5.5kB in size. Also, the copy_process function in the .ko file looks very broken. I will try a few more things.
Thanks,
Song
Josh Poimboeuf
maintainerQuoted 7 lines · Song Liu
Quoted 7 lines
Quoted 1 line
[ 81.261748] copy_process+0xfdc/0xfd58 [livepatch_special_static]Does that
copy_process+0xfdc/0xfd58resolve to this line incopy_process()?refcount_inc(¤t->signal->sigcnt);
Quoted 8 lines
Quoted 2 lines
Maybe the klp rela reference to 'current' is bogus, or resolving to the wrong address somehow?
It resolves the following line.
p->signal->tty=tty_kref_get(current->signal->tty);I am not quite sure how 'current' should be resolved.
Hm, on arm64 it looks like the value of 'current' is stored in the SP_EL0 register. So I guess that shouldn't need any relocations.
Quoted 3 lines
The size of
copy_process(0xfd58) is wrong. It is only about 5.5kB in size. Also, thecopy_processfunction in the .ko file looks very broken. I will try a few more things.
Ah ok, sounds like it's pretty borked.
Signature · 1 line
Josh
Song Liu
maintainerQuoted 12 lines · Josh Poimboeuf
Quoted 10 lines · Song Liu
Quoted 9 lines
Quoted 1 line
[ 81.261748] copy_process+0xfdc/0xfd58 [livepatch_special_static]Does that
copy_process+0xfdc/0xfd58resolve to this line incopy_process()?refcount_inc(¤t->signal->sigcnt);Maybe the klp rela reference to 'current' is bogus, or resolving to the wrong address somehow?
Quoted 12 lines
Quoted 5 lines
It resolves the following line.
p->signal->tty=tty_kref_get(current->signal->tty);I am not quite sure how 'current' should be resolved.
Hm, on arm64 it looks like the value of 'current' is stored in the
SP_EL0register. So I guess that shouldn't need any relocations.Quoted 3 lines
The size of
copy_process(0xfd58) is wrong. It is only about 5.5kB in size. Also, thecopy_processfunction in the .ko file looks very broken. I will try a few more things.
When I try each step of kpatch-build, the copy_process function looks reasonable (according to gdb-disassemble) in fork.o and output.o. However, copy_process looks weird in livepatch-special-static.o, which is generated by ld:
ld -EL -maarch64linux -z norelro -z noexecstack --no-warn-rwx-segments -T ././kpatch.lds -r -o livepatch-special-static.o ./patch-hook.o ./output.o
I have attached these files to the email. I am not sure whether the email server will let them through.
Indu, does this look like an issue with ld?
Thanks,
Song
Quoted 31 lines · Song Liu
Quoted 12 lines · Josh Poimboeuf
Quoted 10 lines · Song Liu
Quoted 9 lines
Quoted 1 line
[ 81.261748] copy_process+0xfdc/0xfd58 [livepatch_special_static]Does that
copy_process+0xfdc/0xfd58resolve to this line incopy_process()?refcount_inc(¤t->signal->sigcnt);Maybe the klp rela reference to 'current' is bogus, or resolving to the wrong address somehow?
Quoted 12 lines
Quoted 5 lines
It resolves the following line.
p->signal->tty=tty_kref_get(current->signal->tty);I am not quite sure how 'current' should be resolved.
Hm, on arm64 it looks like the value of 'current' is stored in the
SP_EL0register. So I guess that shouldn't need any relocations.Quoted 3 lines
The size of
copy_process(0xfd58) is wrong. It is only about 5.5kB in size. Also, thecopy_processfunction in the .ko file looks very broken. I will try a few more things.When I try each step of kpatch-build, the
copy_processfunction looks reasonable (according to gdb-disassemble) in fork.o and output.o. However,copy_processlooks weird in livepatch-special-static.o, which is generated by ld:
Quoted 6 lines
ld -EL -maarch64linux -z norelro -z noexecstack --no-warn-rwx-segments -T
././kpatch.lds-r -o livepatch-special-static.o ./patch-hook.o ./output.oI have attached these files to the email. I am not sure whether the email server will let them through.
I think, I am missing something here,
I did :
objdump -Dr livepatch-special-static.o | less
and
objdump -Dr output.o | less
and the disassembly of copy_process() looks exactly same.
Quoted 4 lines
Indu, does this look like an issue with ld?
Thanks,
Song
Song Liu
maintainerQuoted 45 lines · Puranjay Mohan
Quoted 31 lines · Song Liu
Quoted 12 lines · Josh Poimboeuf
Quoted 10 lines · Song Liu
Quoted 9 lines
Quoted 1 line
[ 81.261748] copy_process+0xfdc/0xfd58 [livepatch_special_static]Does that
copy_process+0xfdc/0xfd58resolve to this line incopy_process()?refcount_inc(¤t->signal->sigcnt);Maybe the klp rela reference to 'current' is bogus, or resolving to the wrong address somehow?
Quoted 12 lines
Quoted 5 lines
It resolves the following line.
p->signal->tty=tty_kref_get(current->signal->tty);I am not quite sure how 'current' should be resolved.
Hm, on arm64 it looks like the value of 'current' is stored in the
SP_EL0register. So I guess that shouldn't need any relocations.Quoted 3 lines
The size of
copy_process(0xfd58) is wrong. It is only about 5.5kB in size. Also, thecopy_processfunction in the .ko file looks very broken. I will try a few more things.When I try each step of kpatch-build, the
copy_processfunction looks reasonable (according to gdb-disassemble) in fork.o and output.o. However,copy_processlooks weird in livepatch-special-static.o, which is generated by ld:Quoted 6 lines
ld -EL -maarch64linux -z norelro -z noexecstack --no-warn-rwx-segments -T
././kpatch.lds-r -o livepatch-special-static.o ./patch-hook.o ./output.oI have attached these files to the email. I am not sure whether the email server will let them through.
I think, I am missing something here,
I did :
Quoted 7 lines
objdump -Dr livepatch-special-static.o | less
and
objdump -Dr output.o | less
and the disassembly of
copy_process()looks exactly same.
Yeah, objdump does show the same disassembly. However, if I open the file with gdb, and do "disassemble copy_process", the one in livepatch-special-static.o looks very weird.
Thanks,
Song
Josh Poimboeuf
maintainerQuoted 3 lines · Song Liu
Yeah, objdump does show the same disassembly. However, if I open the file with gdb, and do "disassemble
copy_process", the one in livepatch-special-static.o looks very weird.
The symbol table looks ok. I'm not sure why gdb is getting confused, but that could possibly be a red herring. Maybe it doesn't like the -ffunction-sections for some reason.
It's really weird the function length reported by kallsyms is so wrong. Can you share the .ko?
The refcount warning might indicate it's passing some bogus memory to tty_kref_get(). Any chance you have struct randomization enabled?
Are you sure there's no code or .config mismatch between the built kernel and the running kernel?
Ignorant arm64 question: is the module's text further away from slab memory than vmlinux text, thus requiring a different instruction (or GOT/TOC) to access memory further away in the address space?
Signature · 1 line
Josh
Song Liu
maintainerQuoted 6 lines · Josh Poimboeuf
Quoted 3 lines · Song Liu
Yeah, objdump does show the same disassembly. However, if I open the file with gdb, and do "disassemble
copy_process", the one in livepatch-special-static.o looks very weird.
Quoted 6 lines
The symbol table looks ok. I'm not sure why gdb is getting confused, but that could possibly be a red herring. Maybe it doesn't like the -ffunction-sections for some reason.
It's really weird the function length reported by kallsyms is so wrong. Can you share the .ko?
Attached is the .ko file.
Quoted 3 lines
The refcount warning might indicate it's passing some bogus memory to
tty_kref_get(). Any chance you have struct randomization enabled?
No, struct randomization is disabled.
Quoted 2 lines
Are you sure there's no code or .config mismatch between the built kernel and the running kernel?
The test is done in a vm, so the .config used to build the livepatch is exactly the same as the testing kernel .
Quoted 4 lines
Ignorant arm64 question: is the module's text further away from slab memory than vmlinux text, thus requiring a different instruction (or GOT/TOC) to access memory further away in the address space?
It appears to me the module text is very close to vmlinux text:
vmlinux: ffff8000800b4b68 T copy_process
module: ffff80007b0f06d0 t copy_process [livepatch_always_inline_special_static]
I run tests with CONFIG_PAGE_SIZE_64KB=y.
Thanks,
Song
Josh Poimboeuf
maintainerQuoted 8 lines · Song Liu
Quoted 3 lines
Ignorant arm64 question: is the module's text further away from slab memory than vmlinux text, thus requiring a different instruction (or GOT/TOC) to access memory further away in the address space?
It appears to me the module text is very close to vmlinux text:
vmlinux:
ffff8000800b4b68Tcopy_process
module:ffff80007b0f06d0tcopy_process[livepatch_always_inline_special_static]
Hm... the only other thing I can think of is that the klp relas might be wrong somewhere. If you share patched.o and .ko files from the same build I could take a look.
BTW, I realized the wrong function size shown in the WARNING stack trace is probably just due to a kallsyms quirk. It calculates a symbol's size by subtracting its start address from the next symbol's start address. It doesn't actually use the ELF symbol size. So the next symbol after copy_process() in the loaded module's address space might just be far away.
That kallsyms issue has caused other headaches. It really needs to be fixed to use the actual ELF symbol size.
Signature · 1 line
Josh
Song Liu
maintainerHi Josh,
Quoted 6 lines · Josh Poimboeuf
Quoted 4 lines · Song Liu
Quoted 3 lines
Ignorant arm64 question: is the module's text further away from slab memory than vmlinux text, thus requiring a different instruction (or GOT/TOC) to access memory further away in the address space?
Quoted 8 lines
Quoted 4 lines
It appears to me the module text is very close to vmlinux text:
vmlinux:
ffff8000800b4b68Tcopy_process
module:ffff80007b0f06d0tcopy_process[livepatch_always_inline_special_static]Hm... the only other thing I can think of is that the klp relas might be wrong somewhere. If you share patched.o and .ko files from the same build I could take a look.
A tarball with these files is available here:
https://drive.google.com/file/d/1ONB1tC9oK-Z5ShmSXneqWLTjJgC5Xq-C/view?usp=drive_link
Quoted 6 lines
BTW, I realized the wrong function size shown in the WARNING stack trace is probably just due to a kallsyms quirk. It calculates a symbol's size by subtracting its start address from the next symbol's start address. It doesn't actually use the ELF symbol size. So the next symbol after
copy_process()in the loaded module's address space might just be far away.
Yeah, I also think kallsyms logic was the issue here. So it is not the same as the gdb-disassembly issue.
Quoted 2 lines
That kallsyms issue has caused other headaches. It really needs to be fixed to use the actual ELF symbol size.
Maybe we should have a "module_text_end" symbol?
Thanks,
Song
Josh Poimboeuf
maintainerQuoted 14 lines · Song Liu
Hi Josh,
Quoted 11 lines · Josh Poimboeuf
Quoted 8 lines · Song Liu
Quoted 3 lines
Ignorant arm64 question: is the module's text further away from slab memory than vmlinux text, thus requiring a different instruction (or GOT/TOC) to access memory further away in the address space?
It appears to me the module text is very close to vmlinux text:
vmlinux:
ffff8000800b4b68Tcopy_process
module:ffff80007b0f06d0tcopy_process[livepatch_always_inline_special_static]
Quoted 7 lines
Quoted 3 lines
Hm... the only other thing I can think of is that the klp relas might be wrong somewhere. If you share patched.o and .ko files from the same build I could take a look.
A tarball with these files is available here:
https://drive.google.com/file/d/1ONB1tC9oK-Z5ShmSXneqWLTjJgC5Xq-C/view?usp=drive_link
Thanks, I'll take a look.
Quoted 4 lines
Quoted 2 lines
That kallsyms issue has caused other headaches. It really needs to be fixed to use the actual ELF symbol size.
Maybe we should have a "
module_text_end" symbol?
Maybe, though it would be a lot cleaner for kallsyms to just use the actual ELF sizes. And actually I'm thinking that would be a pretty trivial change.
Signature · 1 line
Josh
Josh Poimboeuf
maintainerQuoted 14 lines · Song Liu
Hi Josh,
Quoted 11 lines · Josh Poimboeuf
Quoted 8 lines · Song Liu
Quoted 3 lines
Ignorant arm64 question: is the module's text further away from slab memory than vmlinux text, thus requiring a different instruction (or GOT/TOC) to access memory further away in the address space?
It appears to me the module text is very close to vmlinux text:
vmlinux:
ffff8000800b4b68Tcopy_process
module:ffff80007b0f06d0tcopy_process[livepatch_always_inline_special_static]
Quoted 7 lines
Quoted 3 lines
Hm... the only other thing I can think of is that the klp relas might be wrong somewhere. If you share patched.o and .ko files from the same build I could take a look.
A tarball with these files is available here:
https://drive.google.com/file/d/1ONB1tC9oK-Z5ShmSXneqWLTjJgC5Xq-C/view?usp=drive_link
Poking around the arm64 module code, arch/arm64/kernel/module-plts.c is looking at all the relocations in order to set up the PLT. That also needs to be done for klp relas, or are your patches already doing that?
Signature · 1 line
Josh
Song Liu
maintainerQuoted 20 lines · Josh Poimboeuf
Quoted 18 lines · Song Liu
Hi Josh,
Quoted 14 lines · Josh Poimboeuf
Quoted 8 lines · Song Liu
Quoted 3 lines
Ignorant arm64 question: is the module's text further away from slab memory than vmlinux text, thus requiring a different instruction (or GOT/TOC) to access memory further away in the address space?
It appears to me the module text is very close to vmlinux text:
vmlinux:
ffff8000800b4b68Tcopy_process
module:ffff80007b0f06d0tcopy_process[livepatch_always_inline_special_static]Hm... the only other thing I can think of is that the klp relas might be wrong somewhere. If you share patched.o and .ko files from the same build I could take a look.
Quoted 7 lines
Quoted 3 lines
A tarball with these files is available here:
https://drive.google.com/file/d/1ONB1tC9oK-Z5ShmSXneqWLTjJgC5Xq-C/view?usp=drive_link
Poking around the arm64 module code,
arch/arm64/kernel/module-plts.cis looking at all the relocations in order to set up the PLT. That also needs to be done for klp relas, or are your patches already doing that?
I don't think either version (this set and my RFC) added logic for PLT. There is some rela logic on the kpatch-build side. But I am not sure whether it is sufficient.
Thanks,
Song
Josh Poimboeuf
maintainerQuoted 8 lines · Song Liu
Quoted 3 lines · Josh Poimboeuf
Poking around the arm64 module code,
arch/arm64/kernel/module-plts.cis looking at all the relocations in order to set up the PLT. That also needs to be done for klp relas, or are your patches already doing that?I don't think either version (this set and my RFC) added logic for PLT. There is some rela logic on the kpatch-build side. But I am not sure whether it is sufficient.
The klp relas looked ok. I didn't see any signs of kpatch-build doing anything wrong. AFAICT the problem is that module-plts.c creates PLT entries for regular relas but not klp relas.
Signature · 1 line
Josh
Song Liu
maintainerHi Josh,
Quoted 7 lines · Josh Poimboeuf
Quoted 5 lines · Song Liu
Quoted 3 lines · Josh Poimboeuf
Poking around the arm64 module code,
arch/arm64/kernel/module-plts.cis looking at all the relocations in order to set up the PLT. That also needs to be done for klp relas, or are your patches already doing that?
Quoted 7 lines
Quoted 3 lines
I don't think either version (this set and my RFC) added logic for PLT. There is some rela logic on the kpatch-build side. But I am not sure whether it is sufficient.
The klp relas looked ok. I didn't see any signs of kpatch-build doing anything wrong. AFAICT the problem is that
module-plts.ccreates PLT entries for regular relas but not klp relas.
In my tests (with printk) module-plts.c processes the . klp.rela.vmlinux..text.copy_process section just like any other .rela.* sections. Do we need special handling of the klp.rela.* sections?
Thanks,
Song
Josh Poimboeuf
maintainerQuoted 14 lines · Song Liu
Hi Josh,
Quoted 11 lines · Josh Poimboeuf
Quoted 8 lines · Song Liu
Quoted 3 lines · Josh Poimboeuf
Poking around the arm64 module code,
arch/arm64/kernel/module-plts.cis looking at all the relocations in order to set up the PLT. That also needs to be done for klp relas, or are your patches already doing that?I don't think either version (this set and my RFC) added logic for PLT. There is some rela logic on the kpatch-build side. But I am not sure whether it is sufficient.
Quoted 7 lines
Quoted 3 lines
The klp relas looked ok. I didn't see any signs of kpatch-build doing anything wrong. AFAICT the problem is that
module-plts.ccreates PLT entries for regular relas but not klp relas.In my tests (with printk)
module-plts.cprocesses the . klp.rela.vmlinux..text.copy_process section just like any other .rela.* sections. Do we need special handling of the klp.rela.* sections?
Ok, I see how it works now:
klp_write_section_relocs()
apply_relocate_add()
module_emit_plt_entry()
If that code is working correctly then I'm fresh out of ideas...
Signature · 1 line
Josh
Song Liu
maintainerQuoted 24 lines · Josh Poimboeuf
Quoted 14 lines · Song Liu
Hi Josh,
Quoted 11 lines · Josh Poimboeuf
Quoted 8 lines · Song Liu
Quoted 3 lines · Josh Poimboeuf
Poking around the arm64 module code,
arch/arm64/kernel/module-plts.cis looking at all the relocations in order to set up the PLT. That also needs to be done for klp relas, or are your patches already doing that?I don't think either version (this set and my RFC) added logic for PLT. There is some rela logic on the kpatch-build side. But I am not sure whether it is sufficient.
Quoted 7 lines
Quoted 3 lines
The klp relas looked ok. I didn't see any signs of kpatch-build doing anything wrong. AFAICT the problem is that
module-plts.ccreates PLT entries for regular relas but not klp relas.In my tests (with printk)
module-plts.cprocesses the . klp.rela.vmlinux..text.copy_process section just like any other .rela.* sections. Do we need special handling of the klp.rela.* sections?
Quoted 7 lines
Ok, I see how it works now:
klp_write_section_relocs()
apply_relocate_add()
module_emit_plt_entry()If that code is working correctly then I'm fresh out of ideas...
I tried to dump assembly of copy_process, but couldn't find any clue. I am wondering whether this is an issue with gcc-14.2.1.
Puranjay, could you please try with a different gcc, like some version of gcc-14?
Thanks,
Song
Song Liu
maintainerQuoted 32 lines · Song Liu
Quoted 30 lines · Josh Poimboeuf
Quoted 14 lines · Song Liu
Hi Josh,
Quoted 11 lines · Josh Poimboeuf
Quoted 8 lines · Song Liu
Quoted 3 lines · Josh Poimboeuf
Poking around the arm64 module code,
arch/arm64/kernel/module-plts.cis looking at all the relocations in order to set up the PLT. That also needs to be done for klp relas, or are your patches already doing that?I don't think either version (this set and my RFC) added logic for PLT. There is some rela logic on the kpatch-build side. But I am not sure whether it is sufficient.
Quoted 7 lines
Quoted 3 lines
The klp relas looked ok. I didn't see any signs of kpatch-build doing anything wrong. AFAICT the problem is that
module-plts.ccreates PLT entries for regular relas but not klp relas.In my tests (with printk)
module-plts.cprocesses the . klp.rela.vmlinux..text.copy_process section just like any other .rela.* sections. Do we need special handling of the klp.rela.* sections?Ok, I see how it works now:
klp_write_section_relocs()
apply_relocate_add()
module_emit_plt_entry()
Quoted 7 lines
Quoted 1 line
If that code is working correctly then I'm fresh out of ideas...
I tried to dump assembly of
copy_process, but couldn't find any clue. I am wondering whether this is an issue with gcc-14.2.1.Puranjay, could you please try with a different gcc, like some version of gcc-14?
I figured out why this is broken with gcc-14.
The stack trace points to tty_kref_get(), which is eventually a __refcount_add(). With gcc, there are multiple of constprop versions of the function.
$ readelf -s -W vmlinux | grep __refcount_add 19747: ffff800080872eb0 104 FUNC LOCAL DEFAULT 2 __refcount_add.constprop.0 20153: ffff80008001fcd8 88 FUNC LOCAL DEFAULT 2 __refcount_add.constprop.0 36428: ffff800080e62ac0 100 FUNC LOCAL DEFAULT 2 __refcount_add.constprop.0 45055: ffff8000800bf130 100 FUNC LOCAL DEFAULT 2 __refcount_add.constprop.0 54823: ffff8000801c2a40 100 FUNC LOCAL DEFAULT 2 __refcount_add.constprop.0 180848: ffff8000810a8cb0 100 FUNC LOCAL DEFAULT 2 __refcount_add.constprop.0
The problem is, with gcc-14, these symbols are NOT sorted by their addresses. OTOH, the order in kallsyms is like:
[root@(none) /test-klp]# grep __refcount_add /proc/kallsyms ffff80008001fcd8 t __refcount_add.constprop.0 ffff8000800bf130 t __refcount_add.constprop.0 ffff8000801c2a40 t __refcount_add.constprop.0 ffff800080872eb0 t __refcount_add.constprop.0 ffff800080e62ac0 t __refcount_add.constprop.0 ffff8000810a8cb0 t __refcount_add.constprop.0
kpatch-build uses readelf to get the symtab, and then calculate proper sympos based on this symtab. However, this sympos is different when the kernel applies this KLP. In this case, copy_process calls ffff8000800bf130. kpatch-build gives sympos of 4, based on the output of readelf, but ffff8000800bf130 is actually at position 2 in kallsyms.
gcc-11, OTOH, doesn't have this problem with readelf.
Indu, is this behavior (symbols with same name are not in sorted order from readelf -s) expected? Or is this a bug? I am using this gcc:
$ gcc --version gcc (GCC) 14.2.1 20240801 (Red Hat 14.2.1-1) Copyright (C) 2024 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Thanks,
Song
Josh Poimboeuf
maintainerQuoted 9 lines · Song Liu
Indu, is this behavior (symbols with same name are not in sorted order from readelf -s) expected? Or is this a bug? I am using this gcc:
$ gcc --version gcc (GCC) 14.2.1 20240801 (Red Hat 14.2.1-1) Copyright (C) 2024 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Are you using different binutils versions as well?
It sounds like a linker "issue" to me. I'm not sure if it qualifies as a bug, the linker might be free to layout symbols how it wishes.
Signature · 1 line
Josh
A message that is not in the archive (sent off-list or not yet received): CAPhsuW53DK2vFH-BZeUYN-eythX3NQEbcxrYf6jvBDtDmctRgw@mail.gmail.com
Song Liu
maintainerQuoted 18 lines · Song Liu
Quoted 6 lines · Josh Poimboeuf
Quoted 4 lines · Song Liu
Indu, is this behavior (symbols with same name are not in sorted order from readelf -s) expected? Or is this a bug? I am using this gcc:
Quoted 7 lines
Quoted 5 lines
$ gcc --version gcc (GCC) 14.2.1 20240801 (Red Hat 14.2.1-1) Copyright (C) 2024 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.Are you using different binutils versions as well?
Quoted 8 lines
I am using binutil 2.41, for both gcc -11 and gcc-14.
Quoted 3 lines
It sounds like a linker "issue" to me. I'm not sure if it qualifies as a bug, the linker might be free to layout symbols how it wishes.
We can probably handle that in kpatch-build.
OK, something like the following gets the proper sympos for gcc-14 built kernel.
Thanks,
Song
kpatch-build/lookup.c +0 -0
@@ -479,13 +479,10 @@ static bool lookup_local_symbol(struct
lookup_table *table,
struct object_symbol *sym;
unsigned long sympos = 0;
int i, in_file = 0;+ bool found = false;
memset(result, 0, sizeof(*result));
for_each_obj_symbol(i, sym, table) {- if (sym->bind == STB_LOCAL && !strcmp(sym->name,
- lookup_sym->name))
- sympos++;
-
if (lookup_sym->lookup_table_file_sym == sym) {
in_file = 1;
continue;@@ -499,20 +496,29 @@ static bool lookup_local_symbol(struct lookup_table *table,
if (sym->bind == STB_LOCAL && !strcmp(sym->name,
lookup_sym->name)) {
- if (result->objname)
+ if (found)
ERROR("duplicate local symbol found for %s",
lookup_sym->name);
result->objname = table->objname;
result->addr = sym->addr;
result->size = sym->size;
- result->sympos = sympos;
result->global = false;
result->exported = false;
+ found = true;
}
}+ if (!found)
+ return false;
- return !!result->objname;
+ for_each_obj_symbol(i, sym, table) {
+ if (sym->bind == STB_LOCAL &&
+ !strcmp(sym->name, lookup_sym->name) &&
+ sym->addr < result->addr)
+ sympos++;
+ } + result->sympos = sympos; + return true;
}
static bool lookup_exported_symbol(struct lookup_table *table, char *name,
Quoted 31 lines · Song Liu
Quoted 12 lines · Josh Poimboeuf
Quoted 10 lines · Song Liu
Quoted 9 lines
Quoted 1 line
[ 81.261748] copy_process+0xfdc/0xfd58 [livepatch_special_static]Does that
copy_process+0xfdc/0xfd58resolve to this line incopy_process()?refcount_inc(¤t->signal->sigcnt);Maybe the klp rela reference to 'current' is bogus, or resolving to the wrong address somehow?
Quoted 12 lines
Quoted 5 lines
It resolves the following line.
p->signal->tty=tty_kref_get(current->signal->tty);I am not quite sure how 'current' should be resolved.
Hm, on arm64 it looks like the value of 'current' is stored in the
SP_EL0register. So I guess that shouldn't need any relocations.Quoted 3 lines
The size of
copy_process(0xfd58) is wrong. It is only about 5.5kB in size. Also, thecopy_processfunction in the .ko file looks very broken. I will try a few more things.When I try each step of kpatch-build, the
copy_processfunction looks reasonable (according to gdb-disassemble) in fork.o and output.o. However,copy_processlooks weird in livepatch-special-static.o, which is generated by ld:
Quoted 9 lines
ld -EL -maarch64linux -z norelro -z noexecstack --no-warn-rwx-segments -T
././kpatch.lds-r -o livepatch-special-static.o ./patch-hook.o ./output.oI have attached these files to the email. I am not sure whether the email server will let them through.
Indu, does this look like an issue with ld?
Sorry for the delay.
Looks like there has been progress since and issue may be elsewhere, but:
FWIW, I looked at the .sframe and .rela.sframe sections here, the data does look OK. I noted that there is no .sframe for copy_process () in output.o... I will take a look into it.
Song Liu
maintainerQuoted 43 lines · Indu Bhagat
Quoted 31 lines · Song Liu
Quoted 12 lines · Josh Poimboeuf
Quoted 10 lines · Song Liu
Quoted 9 lines
Quoted 1 line
[ 81.261748] copy_process+0xfdc/0xfd58 [livepatch_special_static]Does that
copy_process+0xfdc/0xfd58resolve to this line incopy_process()?refcount_inc(¤t->signal->sigcnt);Maybe the klp rela reference to 'current' is bogus, or resolving to the wrong address somehow?
Quoted 12 lines
Quoted 5 lines
It resolves the following line.
p->signal->tty=tty_kref_get(current->signal->tty);I am not quite sure how 'current' should be resolved.
Hm, on arm64 it looks like the value of 'current' is stored in the
SP_EL0register. So I guess that shouldn't need any relocations.Quoted 3 lines
The size of
copy_process(0xfd58) is wrong. It is only about 5.5kB in size. Also, thecopy_processfunction in the .ko file looks very broken. I will try a few more things.When I try each step of kpatch-build, the
copy_processfunction looks reasonable (according to gdb-disassemble) in fork.o and output.o. However,copy_processlooks weird in livepatch-special-static.o, which is generated by ld:Quoted 9 lines
ld -EL -maarch64linux -z norelro -z noexecstack --no-warn-rwx-segments -T
././kpatch.lds-r -o livepatch-special-static.o ./patch-hook.o ./output.oI have attached these files to the email. I am not sure whether the email server will let them through.
Indu, does this look like an issue with ld?
Quoted 7 lines
Sorry for the delay.
Looks like there has been progress since and issue may be elsewhere, but:
FWIW, I looked at the .sframe and .rela.sframe sections here, the data does look OK. I noted that there is no .sframe for
copy_process() in output.o... I will take a look into it.
That output.o was generated with a version of kpatch-build that blindly removes all .sframe sections.
Thanks,
Song
Quoted 42 lines · Indu Bhagat
Quoted 31 lines · Song Liu
Quoted 12 lines · Josh Poimboeuf
Quoted 10 lines · Song Liu
Quoted 9 lines
Quoted 1 line
[ 81.261748] copy_process+0xfdc/0xfd58 [livepatch_special_static]Does that
copy_process+0xfdc/0xfd58resolve to this line incopy_process()?refcount_inc(¤t->signal->sigcnt);Maybe the klp rela reference to 'current' is bogus, or resolving to the wrong address somehow?
Quoted 12 lines
Quoted 5 lines
It resolves the following line.
p->signal->tty=tty_kref_get(current->signal->tty);I am not quite sure how 'current' should be resolved.
Hm, on arm64 it looks like the value of 'current' is stored in the
SP_EL0register. So I guess that shouldn't need any relocations.Quoted 3 lines
The size of
copy_process(0xfd58) is wrong. It is only about 5.5kB in size. Also, thecopy_processfunction in the .ko file looks very broken. I will try a few more things.When I try each step of kpatch-build, the
copy_processfunction looks reasonable (according to gdb-disassemble) in fork.o and output.o. However,copy_processlooks weird in livepatch-special-static.o, which is generated by ld:Quoted 9 lines
ld -EL -maarch64linux -z norelro -z noexecstack --no-warn-rwx-segments -T
././kpatch.lds-r -o livepatch-special-static.o ./patch-hook.o ./output.oI have attached these files to the email. I am not sure whether the email server will let them through.
Indu, does this look like an issue with ld?
Quoted 7 lines
Sorry for the delay.
Looks like there has been progress since and issue may be elsewhere, but:
FWIW, I looked at the .sframe and .rela.sframe sections here, the data does look OK. I noted that there is no .sframe for
copy_process() in output.o... I will take a look into it.
Hi Indu,
I saw another issue in my kernel build with sframes enabled (-Wa,--gsframe):
ld: warning: orphan section .init.sframe' from arch/arm64/kernel/pi/lib-fdt.pi.o' being placed in section `.init.sframe' [... Many more similar warnings (.init.sframe) ...]
So, this orphan sections is generated in the build process.
I am using GNU ld version 2.41-50 and gcc (GCC) 11.4.1
Is this section needed for sframes to work? or can we discard .init.sframe section with a patch like following to the linker script:
-- 8< --
include/asm-generic/vmlinux.lds.h +7 -0
@@ -1044,9 +1044,16 @@ defined(CONFIG_AUTOFDO_CLANG) || defined(CONFIG_PROPELLER_CLANG)# define SANITIZER_DISCARDS #endif +#if defined(CONFIG_SFRAME_UNWIND_TABLE ) +#define DISCARD_INIT_SFRAME *(.init.sframe) +#else +#define DISCARD_INIT_SFRAME +#endif +#define COMMON_DISCARDS \ SANITIZER_DISCARDS \ PATCHABLE_DISCARDS \ + DISCARD_INIT_SFRAME \ *(.discard) \ *(.discard.*) \ *(.export_symbol) \
-- >8 --
Thanks,
Puranjay
Quoted 56 lines · Puranjay Mohan
Quoted 42 lines · Indu Bhagat
Quoted 31 lines · Song Liu
Quoted 12 lines · Josh Poimboeuf
Quoted 10 lines · Song Liu
Quoted 9 lines
Quoted 1 line
[ 81.261748] copy_process+0xfdc/0xfd58 [livepatch_special_static]Does that
copy_process+0xfdc/0xfd58resolve to this line incopy_process()?refcount_inc(¤t->signal->sigcnt);Maybe the klp rela reference to 'current' is bogus, or resolving to the wrong address somehow?
Quoted 12 lines
Quoted 5 lines
It resolves the following line.
p->signal->tty=tty_kref_get(current->signal->tty);I am not quite sure how 'current' should be resolved.
Hm, on arm64 it looks like the value of 'current' is stored in the
SP_EL0register. So I guess that shouldn't need any relocations.Quoted 3 lines
The size of
copy_process(0xfd58) is wrong. It is only about 5.5kB in size. Also, thecopy_processfunction in the .ko file looks very broken. I will try a few more things.When I try each step of kpatch-build, the
copy_processfunction looks reasonable (according to gdb-disassemble) in fork.o and output.o. However,copy_processlooks weird in livepatch-special-static.o, which is generated by ld:Quoted 9 lines
ld -EL -maarch64linux -z norelro -z noexecstack --no-warn-rwx-segments -T
././kpatch.lds-r -o livepatch-special-static.o ./patch-hook.o ./output.oI have attached these files to the email. I am not sure whether the email server will let them through.
Indu, does this look like an issue with ld?
Quoted 7 lines
Sorry for the delay.
Looks like there has been progress since and issue may be elsewhere, but:
FWIW, I looked at the .sframe and .rela.sframe sections here, the data does look OK. I noted that there is no .sframe for
copy_process() in output.o... I will take a look into it.Hi Indu,
I saw another issue in my kernel build with sframes enabled (-Wa,--gsframe):
Quoted 8 lines
ld: warning: orphan section
.init.sframe' fromarch/arm64/kernel/pi/lib-fdt.pi.o' being placed in section `.init.sframe' [... Many more similar warnings (.init.sframe) ...]So, this orphan sections is generated in the build process.
I am using GNU ld version 2.41-50 and gcc (GCC) 11.4.1
Is this section needed for sframes to work? or can we discard
No this should not be discarded. This looks like a wrongly-named but valid SFrame section.
Once correctly named as .sframe, the linker should do the right thing. Let me check whats going on..
Quoted 30 lines
.init.sframe section with a patch like following to the linker script:
-- 8< --
include/asm-generic/vmlinux.lds.h +7
-1@@ -1044,9 +1044,16 @@ defined(CONFIG_AUTOFDO_CLANG) || defined(CONFIG_PROPELLER_CLANG)# define SANITIZER_DISCARDS #endif +#if defined(CONFIG_SFRAME_UNWIND_TABLE ) +#define DISCARD_INIT_SFRAME *(.init.sframe) +#else +#define DISCARD_INIT_SFRAME +#endif +#define COMMON_DISCARDS \ SANITIZER_DISCARDS \ PATCHABLE_DISCARDS \ + DISCARD_INIT_SFRAME \ *(.discard) \ *(.discard.*) \ *(.export_symbol) \ -- >8 --Thanks,
Puranjay
Quoted 60 lines · Indu Bhagat
Quoted 59 lines · Puranjay Mohan
Quoted 42 lines · Indu Bhagat
Quoted 31 lines · Song Liu
Quoted 12 lines · Josh Poimboeuf
Quoted 10 lines · Song Liu
Quoted 9 lines
Quoted 1 line
[ 81.261748]
copy_process+0xfdc/0xfd58[livepatch_special_static]Does that
copy_process+0xfdc/0xfd58resolve to this line incopy_process()?
refcount_inc(¤t->signal->sigcnt);Maybe the klp rela reference to 'current' is bogus, or resolving to the wrong address somehow?
Quoted 12 lines
Quoted 5 lines
It resolves the following line.
p->signal->tty=tty_kref_get(current->signal->tty);I am not quite sure how 'current' should be resolved.
Hm, on arm64 it looks like the value of 'current' is stored in the
SP_EL0register. So I guess that shouldn't need any relocations.Quoted 3 lines
The size of
copy_process(0xfd58) is wrong. It is only about 5.5kB in size. Also, thecopy_processfunction in the .ko file looks very broken. I will try a few more things.When I try each step of kpatch-build, the
copy_processfunction looks reasonable (according to gdb-disassemble) in fork.o and output.o. However,copy_processlooks weird in livepatch-special- static.o, which is generated by ld:Quoted 9 lines
ld -EL -maarch64linux -z norelro -z noexecstack --no-warn-rwx-segments -T ././kpatch.lds -r -o livepatch-special-static.o ./patch-hook.o ./output.o
I have attached these files to the email. I am not sure whether the email server will let them through.
Indu, does this look like an issue with ld?
Quoted 7 lines
Sorry for the delay.
Looks like there has been progress since and issue may be elsewhere, but:
FWIW, I looked at the .sframe and .rela.sframe sections here, the data does look OK. I noted that there is no .sframe for
copy_process() in output.o... I will take a look into it.Hi Indu,
I saw another issue in my kernel build with sframes enabled (-Wa,-- gsframe):
ld: warning: orphan section
.init.sframe' fromarch/arm64/kernel/pi/lib-fdt.pi.o' being placed in section `.init.sframe' [... Many more similar warnings (.init.sframe) ...]
Quoted 8 lines
Quoted 5 lines
So, this orphan sections is generated in the build process.
I am using GNU ld version 2.41-50 and gcc (GCC) 11.4.1
Is this section needed for sframes to work? or can we discard
No this should not be discarded. This looks like a wrongly-named but valid SFrame section.
Not wrongly named. --prefix-alloc-sections=.init is expected to do that as .sframe is an allocated section.
Quoted 39 lines
Once correctly named as .sframe, the linker should do the right thing. Let me check whats going on..
Quoted 36 lines
.init.sframe section with a patch like following to the linker script:
-- 8< --
include/asm-generic/vmlinux.lds.h +7
-0@@ -1044,9 +1044,16 @@ defined(CONFIG_AUTOFDO_CLANG) || defined(CONFIG_PROPELLER_CLANG) # define SANITIZER_DISCARDS #endif +#if defined(CONFIG_SFRAME_UNWIND_TABLE ) +#define DISCARD_INIT_SFRAME *(.init.sframe) +#else +#define DISCARD_INIT_SFRAME +#endif + #define COMMON_DISCARDS \ SANITIZER_DISCARDS \ PATCHABLE_DISCARDS \ + DISCARD_INIT_SFRAME \*(.discard) \
*(.discard.*) \
*(.export_symbol) \
-- >8 --
Thanks,
Puranjay
Quoted 72 lines · Indu Bhagat
Quoted 71 lines · Indu Bhagat
Quoted 70 lines · Puranjay Mohan
Quoted 46 lines · Indu Bhagat
Quoted 35 lines · Song Liu
Quoted 14 lines · Josh Poimboeuf
Quoted 12 lines · Song Liu
Quoted 11 lines
Quoted 2 lines
[ 81.261748]
copy_process+0xfdc/0xfd58
[livepatch_special_static]Does that
copy_process+0xfdc/0xfd58resolve to this line incopy_process()?
refcount_inc(¤t->signal->sigcnt);Maybe the klp rela reference to 'current' is bogus, or resolving to the
wrong address somehow?Quoted 12 lines
Quoted 5 lines
It resolves the following line.
p->signal->tty=tty_kref_get(current->signal->tty);I am not quite sure how 'current' should be resolved.
Hm, on arm64 it looks like the value of 'current' is stored in the
SP_EL0register. So I guess that shouldn't need any relocations.Quoted 3 lines
The size of
copy_process(0xfd58) is wrong. It is only about 5.5kB in size. Also, thecopy_processfunction in the .ko file looks very broken. I will try a few more things.When I try each step of kpatch-build, the
copy_processfunction looks reasonable (according to gdb-disassemble) in fork.o and output.o. However,copy_processlooks weird in livepatch-special- static.o,
which is generated by ld:Quoted 9 lines
ld -EL -maarch64linux -z norelro -z noexecstack --no-warn-rwx-segments -T ././kpatch.lds -r -o livepatch-special-static.o ./patch-hook.o ./output.o
I have attached these files to the email. I am not sure whether the email server will let them through.
Indu, does this look like an issue with ld?
Quoted 8 lines
Sorry for the delay.
Looks like there has been progress since and issue may be elsewhere, but:
FWIW, I looked at the .sframe and .rela.sframe sections here, the data does look OK. I noted that there is no .sframe for
copy_process() in output.o... I will take a look into it.Hi Indu,
I saw another issue in my kernel build with sframes enabled (-Wa,-- gsframe):
ld: warning: orphan section
.init.sframe' fromarch/arm64/kernel/pi/lib-fdt.pi.o' being placed in section `.init.sframe'
[... Many more similar warnings (.init.sframe) ...]So, this orphan sections is generated in the build process.
I am using GNU ld version 2.41-50 and gcc (GCC) 11.4.1
Quoted 9 lines
Quoted 5 lines
Quoted 1 line
Is this section needed for sframes to work? or can we discard
No this should not be discarded. This looks like a wrongly-named but valid SFrame section.
Not wrongly named. --prefix-alloc-sections=.init is expected to do that as .sframe is an allocated section.
So, these .init.sframe sections are then moved into .sframe by the linker? (see linker script line below)
Here are some outputs from the build, do they look correct?
One of the objects that were emitting the warning
[ec2-user@ip-172-31-32-86 linux-upstream]$ readelf -SW arch/arm64/kernel/pi/lib-fdt.pi.o | grep sframe
[47] .init.sframe PROGBITS 0000000000000000 003c90 000226 00 A 0 0 8 [48] .rela.init.sframe RELA 0000000000000000 008f08 000180 18 I 51 47 8
Final vmlinux ELF
[ec2-user@ip-172-31-32-86 linux-upstream]$ readelf -SW vmlinux | grep sframe
[ 5] .init.sframe PROGBITS ffff80008118c298 119c298 002a88 00 A 0 0 8 [ 6] .sframe PROGBITS ffff80008118ed20 119ed20 247c45 00 A 0 0 8
[ec2-user@ip-172-31-32-86 linux-upstream]$ readelf --sframe=.sframe vmlinux | head Contents of the SFrame section .sframe:
Header :
Version: SFRAME_VERSION_2
Flags: SFRAME_F_FDE_SORTED
Num FDEs: 51842
Num FREs: 321245Function Index :
Does this also look correct?
[ec2-user@ip-172-31-32-86 linux-upstream]$ readelf --sframe=.init.sframe vmlinux | head Contents of the SFrame section .init.sframe:
Header :
Version: SFRAME_VERSION_2
Flags: NONE
Num FDEs: 16
Num FREs: 50
Function Index :
and the linker script has this line:
.sframe : AT(ADDR(.sframe) - 0) { __start_sframe_header = .; KEEP(*(.sframe)) __stop_sframe_header = .; }
So, do can you suggest the best way to fix these warnings?
Thanks,
Puranjay
Josh Poimboeuf
maintainerQuoted 5 lines · Puranjay Mohan
and the linker script has this line:
.sframe : AT(ADDR(.sframe) - 0) {
__start_sframe_header= .; KEEP(*(.sframe))__stop_sframe_header= .; }So, do can you suggest the best way to fix these warnings?
Just add *(.init.sframe) like so:
.sframe : AT(ADDR(.sframe) - 0) { __start_sframe_header = .; KEEP(*(.sframe) *(.init.sframe)) __stop_sframe_header = .; }
Signature · 1 line
Josh
Josh Poimboeuf
maintainerQuoted 10 lines · Josh Poimboeuf
Quoted 5 lines · Puranjay Mohan
and the linker script has this line:
.sframe : AT(ADDR(.sframe) - 0) {
__start_sframe_header= .; KEEP(*(.sframe))__stop_sframe_header= .; }So, do can you suggest the best way to fix these warnings?
Just add *(.init.sframe) like so:
.sframe : AT(ADDR(.sframe) - 0) {
__start_sframe_header= .;KEEP(*(.sframe) *(.init.sframe))__stop_sframe_header= .; }
Actually each probably needs its own KEEP:
... KEEP(*(.sframe)) KEEP(*(.init.sframe)) ...
or so.
Signature · 1 line
Josh
Quoted 9 lines · Song Liu
I run some tests with this set and my RFC set [1]. Most of the test is done with kpatch-build. I tested both Puranjay's version [3] and my version [4].
For gcc 14.2.1, I have seen the following issue with this test [2]. This happens with both upstream and 6.13.2. The livepatch loaded fine, but the system spilled out the following warning quickly.
In presence of the issue https://sourceware.org/bugzilla/show_bug.cgi?id=32666, I'd expect bad data in SFrame section. Which may be causing this symptom?
To be clear, the issue affects loaded kernel modules. I cannot tell for certain - is there module loading involved in your test ?
Quoted 162 lines
On the other hand, the same test works with LLVM and my RFC set (LLVM doesn't support SFRAME, and thus doesn't work with this set yet).
Thanks,
Song[ 81.250437] ------------[ cut here ]------------ [ 81.250818] refcount_t: saturated; leaking memory. [ 81.251201] WARNING: CPU: 0 PID: 95 at lib/refcount.c:22 refcount_warn_saturate+0x6c/0x140 [ 81.251841] Modules linked in: livepatch_special_static(OEK) [ 81.252277] CPU: 0 UID: 0 PID: 95 Comm: bash Tainted: G OE K 6.13.2-00321-g52d2813b4b07 #49 [ 81.253003] Tainted: [O]=OOT_MODULE, [E]=UNSIGNED_MODULE, [K]=LIVEPATCH [ 81.253503] Hardware name: linux,dummy-virt (DT) [ 81.253856] pstate: 634000c5 (nZCv daIF +PAN -UAO +TCO +DIT -SSBS BTYPE=--) [ 81.254383] pc : refcount_warn_saturate+0x6c/0x140 [ 81.254748] lr : refcount_warn_saturate+0x6c/0x140 [ 81.255114] sp : ffff800085a6fc00 [ 81.255371] x29: ffff800085a6fc00 x28: 0000000001200000 x27: ffff0000c2966180 [ 81.255918] x26: 0000000000000000 x25: ffff8000829c0000 x24: ffff0000c2e9b608 [ 81.256462] x23: ffff800083351000 x22: ffff0000c2e9af80 x21: ffff0000c062e140 [ 81.257006] x20: ffff0000c1c10c00 x19: ffff800085a6fd80 x18: ffffffffffffffff [ 81.257544] x17: 0000000000000001 x16: ffffffffffffffff x15: 0000000000000006 [ 81.258083] x14: 0000000000000000 x13: 2e79726f6d656d20 x12: 676e696b61656c20 [ 81.258625] x11: ffff8000829f7d70 x10: 0000000000000147 x9 : ffff8000801546b4 [ 81.259165] x8 : 00000000fffeffff x7 : 00000000ffff0000 x6 : ffff800082f77d70 [ 81.259709] x5 : 80000000ffff0000 x4 : 0000000000000000 x3 : 0000000000000001 [ 81.260257] x2 : ffff8000829f7a88 x1 : ffff8000829f7a88 x0 : 0000000000000026 [ 81.260824] Call trace: [ 81.261015] refcount_warn_saturate+0x6c/0x140 (P) [ 81.261387] __refcount_add.constprop.0+0x60/0x70 [ 81.261748] copy_process+0xfdc/0xfd58 [livepatch_special_static] [ 81.262217] kernel_clone+0x80/0x3e0 [ 81.262499] __do_sys_clone+0x5c/0x88 [ 81.262786] __arm64_sys_clone+0x24/0x38 [ 81.263085] invoke_syscall+0x4c/0x108 [ 81.263378] el0_svc_common.constprop.0+0x44/0xe8 [ 81.263734] do_el0_svc+0x20/0x30 [ 81.263993] el0_svc+0x34/0xf8 [ 81.264231] el0t_64_sync_handler+0x104/0x130 [ 81.264561] el0t_64_sync+0x184/0x188 [ 81.264846] ---[ end trace 0000000000000000 ]--- [ 82.335559] ------------[ cut here ]------------ [ 82.335931] refcount_t: underflow; use-after-free. [ 82.336307] WARNING: CPU: 1 PID: 0 at lib/refcount.c:28 refcount_warn_saturate+0xec/0x140 [ 82.336949] Modules linked in: livepatch_special_static(OEK) [ 82.337389] CPU: 1 UID: 0 PID: 0 Comm: swapper/1 Tainted: G W OE K 6.13.2-00321-g52d2813b4b07 #49 [ 82.338148] Tainted: [W]=WARN, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE, [K]=LIVEPATCH [ 82.338721] Hardware name: linux,dummy-virt (DT) [ 82.339083] pstate: 63400005 (nZCv daif +PAN -UAO +TCO +DIT -SSBS BTYPE=--) [ 82.339617] pc : refcount_warn_saturate+0xec/0x140 [ 82.340007] lr : refcount_warn_saturate+0xec/0x140 [ 82.340378] sp : ffff80008370fe40 [ 82.340637] x29: ffff80008370fe40 x28: 0000000000000000 x27: 0000000000000000 [ 82.341188] x26: 000000000000000a x25: ffff0000fdaf7ab8 x24: 0000000000000014 [ 82.341737] x23: ffff8000829c8d30 x22: ffff80008370ff28 x21: ffff0000fe020000 [ 82.342286] x20: ffff0000c062e140 x19: ffff0000c2e9af80 x18: ffffffffffffffff [ 82.342839] x17: ffff80007b7a0000 x16: ffff800083700000 x15: 0000000000000006 [ 82.343389] x14: 0000000000000000 x13: 2e656572662d7265 x12: 7466612d65737520 [ 82.343944] x11: ffff8000829f7d70 x10: 000000000000016a x9 : ffff8000801546b4 [ 82.344499] x8 : 00000000fffeffff x7 : 00000000ffff0000 x6 : ffff800082f77d70 [ 82.345051] x5 : 80000000ffff0000 x4 : 0000000000000000 x3 : 0000000000000001 [ 82.345604] x2 : ffff8000829f7a88 x1 : ffff8000829f7a88 x0 : 0000000000000026 [ 82.346163] Call trace: [ 82.346359] refcount_warn_saturate+0xec/0x140 (P) [ 82.346736] __put_task_struct+0x130/0x170 [ 82.347063] delayed_put_task_struct+0xbc/0xe8 [ 82.347411] rcu_core+0x20c/0x5f8 [ 82.347680] rcu_core_si+0x14/0x28 [ 82.347952] handle_softirqs+0x124/0x308 [ 82.348260] __do_softirq+0x18/0x20 [ 82.348536] ____do_softirq+0x14/0x28 [ 82.348828] call_on_irq_stack+0x24/0x30 [ 82.349137] do_softirq_own_stack+0x20/0x38 [ 82.349465] __irq_exit_rcu+0xcc/0x108 [ 82.349764] irq_exit_rcu+0x14/0x28 [ 82.350038] el1_interrupt+0x34/0x50 [ 82.350321] el1h_64_irq_handler+0x14/0x20 [ 82.350642] el1h_64_irq+0x6c/0x70 [ 82.350911] default_idle_call+0x30/0xd0 (P) [ 82.351248] do_idle+0x1d0/0x200 [ 82.351506] cpu_startup_entry+0x38/0x48 [ 82.351818] secondary_start_kernel+0x124/0x150 [ 82.352176] __secondary_switched+0xac/0xb0 [ 82.352505] ---[ end trace 0000000000000000 ]---[1] SFRAME-less livepatch RFC
https://lore.kernel.org/live-patching/[email protected]/
[2] special-static test from kpatch
https://github.com/dynup/kpatch/blob/master/test/integration/linux-6.2.0/special-static.patch
[3] Puranjay's kpatch with arm64 support
https://github.com/puranjaymohan/kpatch/tree/arm64
[4] My version of kpatch with arm64 and LTO support
https://github.com/liu-song-6/kpatch/tree/fb-6.13-v2Quoted 58 lines · Weinan Liu
This patchset implements a generic kernel sframe-based [1] unwinder. The main goal is to support reliable stacktraces on arm64.
On x86 orc unwinder provides reliable stacktraces. But arm64 misses the required support from objtool: it cannot generate orc unwind tables for arm64.
Currently, there's already a sframe unwinder proposed for userspace: [2]. Since the sframe unwind table algorithm is similar, these two proposal could integrate common functionality in the future.
There are some incomplete features or challenges:
- The unwinder doesn't yet work with kernel modules. Thestart_addrof FRE from kernel modules doesn't appear correct, preventing us from unwinding functions from kernel modules.
- Currently, only GCC supports sframe.Ref:
[1]: https://sourceware.org/binutils/docs/sframe-spec.html
[2]: https://lore.kernel.org/lkml/[email protected]/Madhavan T. Venkataraman (1):
arm64: DefineTIF_PATCH_PENDINGfor livepatchWeinan Liu (7):
unwind: build kernel with sframe info
arm64: entry: add unwind info for various kernel entries
unwind: add sframe v2 header
unwind: Implement generic sframe unwinder library
unwind: arm64: Add sframe unwinder on arm64
unwind: arm64: add reliable stacktrace support for arm64
arm64: Enable livepatch for ARM64Makefile | 6 + arch/Kconfig | 8 + arch/arm64/Kconfig | 3 + arch/arm64/Kconfig.debug | 10 + arch/arm64/include/asm/stacktrace/common.h | 6 + arch/arm64/include/asm/thread_info.h | 4 +- arch/arm64/kernel/entry-common.c | 4 + arch/arm64/kernel/entry.S | 10 + arch/arm64/kernel/setup.c | 2 + arch/arm64/kernel/stacktrace.c | 102 ++++++++++ include/asm-generic/vmlinux.lds.h | 12 ++ include/linux/sframe_lookup.h | 43 +++++ kernel/Makefile | 1 + kernel/sframe.h | 215 +++++++++++++++++++++ kernel/sframe_lookup.c | 196 +++++++++++++++++++ 15 files changed, 621 insertions(+), 1 deletion(-)create mode 100644
include/linux/sframe_lookup.hcreate mode 100644kernel/sframe.hcreate mode 100644kernel/sframe_lookup.c--
2.48.1.262.g85cc9f2d1e-goog
Song Liu
maintainerQuoted 12 lines · Indu Bhagat
Quoted 9 lines · Song Liu
I run some tests with this set and my RFC set [1]. Most of the test is done with kpatch-build. I tested both Puranjay's version [3] and my version [4].
For gcc 14.2.1, I have seen the following issue with this test [2]. This happens with both upstream and 6.13.2. The livepatch loaded fine, but the system spilled out the following warning quickly.
Quoted 6 lines
In presence of the issue
https://sourceware.org/bugzilla/show_bug.cgi?id=32666, I'd expect bad data in SFrame section. Which may be causing this symptom?To be clear, the issue affects loaded kernel modules. I cannot tell for certain - is there module loading involved in your test ?
The KLP is a module, I guess that is also affected?
During kpatch-build, we added some logic to drop the .sframe section. I guess this is wrong, as we need the .sframe section when we apply the next KLP. However, I don't think the issue is caused by missing .sframe section.
Thanks,
Song
Josh Poimboeuf
maintainerQuoted 20 lines · Song Liu
Quoted 12 lines · Indu Bhagat
Quoted 9 lines · Song Liu
I run some tests with this set and my RFC set [1]. Most of the test is done with kpatch-build. I tested both Puranjay's version [3] and my version [4].
For gcc 14.2.1, I have seen the following issue with this test [2]. This happens with both upstream and 6.13.2. The livepatch loaded fine, but the system spilled out the following warning quickly.
Quoted 6 lines
In presence of the issue
https://sourceware.org/bugzilla/show_bug.cgi?id=32666, I'd expect bad data in SFrame section. Which may be causing this symptom?To be clear, the issue affects loaded kernel modules. I cannot tell for certain - is there module loading involved in your test ?
Quoted 6 lines
The KLP is a module, I guess that is also affected?
During kpatch-build, we added some logic to drop the .sframe section. I guess this is wrong, as we need the .sframe section when we apply the next KLP. However, I don't think the issue is caused by missing .sframe section.
Right, it looks like it unwound through the patch module just fine.
Signature · 1 line
Josh
Quoted 20 lines · Song Liu
Quoted 12 lines · Indu Bhagat
Quoted 9 lines · Song Liu
I run some tests with this set and my RFC set [1]. Most of the test is done with kpatch-build. I tested both Puranjay's version [3] and my version [4].
For gcc 14.2.1, I have seen the following issue with this test [2]. This happens with both upstream and 6.13.2. The livepatch loaded fine, but the system spilled out the following warning quickly.
Quoted 6 lines
In presence of the issue
https://sourceware.org/bugzilla/show_bug.cgi?id=32666, I'd expect bad data in SFrame section. Which may be causing this symptom?To be clear, the issue affects loaded kernel modules. I cannot tell for certain - is there module loading involved in your test ?
Quoted 6 lines
The KLP is a module, I guess that is also affected?
During kpatch-build, we added some logic to drop the .sframe section. I guess this is wrong, as we need the .sframe section when we apply the next KLP. However, I don't think the issue is caused by missing .sframe section.
Hi, I did the same testing and did not get the Warning.
I am testing on the 6.12.11 kernel with GCC 11.4.1.
Just to verify, the patch we are testing is:
--- >8 ---
kernel/fork.c 2023-01-12 11:21:19.186137466 -0500 renamed from kernel/fork.c 2023-01-12 11:20:05.408700033 -0500 +8 -1
@@ -1700,10 +1700,18 @@ static void posix_cpu_timers_init_group( posix_cputimers_group_init(pct, cpu_limit); } +void kpatch_foo(void) +{ + if (!jiffies) + printk("kpatch copy signal\n"); +} + static int copy_signal(unsigned long clone_flags, struct task_struct *tsk) { struct signal_struct *sig; + kpatch_foo(); + if (clone_flags & CLONE_THREAD) return 0; --- 8< ---
P.S. - I have a downstream patch for create-diff-object to generate .sframe sections for livepatch module, will add it to the PR after some cleanups.
Thanks,
Puranjay
Song Liu
maintainerQuoted 25 lines · Puranjay Mohan
Quoted 22 lines · Song Liu
Quoted 12 lines · Indu Bhagat
Quoted 9 lines · Song Liu
I run some tests with this set and my RFC set [1]. Most of the test is done with kpatch-build. I tested both Puranjay's version [3] and my version [4].
For gcc 14.2.1, I have seen the following issue with this test [2]. This happens with both upstream and 6.13.2. The livepatch loaded fine, but the system spilled out the following warning quickly.
Quoted 6 lines
In presence of the issue
https://sourceware.org/bugzilla/show_bug.cgi?id=32666, I'd expect bad data in SFrame section. Which may be causing this symptom?To be clear, the issue affects loaded kernel modules. I cannot tell for certain - is there module loading involved in your test ?
The KLP is a module, I guess that is also affected?
Quoted 8 lines
Quoted 4 lines
During kpatch-build, we added some logic to drop the .sframe section. I guess this is wrong, as we need the .sframe section when we apply the next KLP. However, I don't think the issue is caused by missing .sframe section.
Hi, I did the same testing and did not get the Warning.
I am testing on the 6.12.11 kernel with GCC 11.4.1.
Could you please also try kernel 6.13.2?
Quoted 1 line
Just to verify, the patch we are testing is:
Yes, this is the test patch.
Quoted 2 lines
--- >8 ---
[...]
Quoted 4 lines
--- 8< ---
P.S. - I have a downstream patch for create-diff-object to generate .sframe sections for livepatch module, will add it to the PR after some cleanups.
Yeah, I think the .sframe section is still needed.
Thanks,
Song
Quoted 41 lines · Song Liu
Quoted 25 lines · Puranjay Mohan
Quoted 22 lines · Song Liu
Quoted 12 lines · Indu Bhagat
Quoted 9 lines · Song Liu
I run some tests with this set and my RFC set [1]. Most of the test is done with kpatch-build. I tested both Puranjay's version [3] and my version [4].
For gcc 14.2.1, I have seen the following issue with this test [2]. This happens with both upstream and 6.13.2. The livepatch loaded fine, but the system spilled out the following warning quickly.
Quoted 6 lines
In presence of the issue
https://sourceware.org/bugzilla/show_bug.cgi?id=32666, I'd expect bad data in SFrame section. Which may be causing this symptom?To be clear, the issue affects loaded kernel modules. I cannot tell for certain - is there module loading involved in your test ?
The KLP is a module, I guess that is also affected?
Quoted 8 lines
Quoted 4 lines
During kpatch-build, we added some logic to drop the .sframe section. I guess this is wrong, as we need the .sframe section when we apply the next KLP. However, I don't think the issue is caused by missing .sframe section.
Hi, I did the same testing and did not get the Warning.
I am testing on the 6.12.11 kernel with GCC 11.4.1.
Could you please also try kernel 6.13.2?
Quoted 1 line
Just to verify, the patch we are testing is:
Yes, this is the test patch.
Quoted 1 line
Quoted 9 lines
Quoted 1 line
--- >8 ---
[...]
Quoted 4 lines
--- 8< ---
P.S. - I have a downstream patch for create-diff-object to generate .sframe sections for livepatch module, will add it to the PR after some cleanups.Yeah, I think the .sframe section is still needed.
Hi Song,
Can you try with this:
https://github.com/puranjaymohan/kpatch/tree/arm64_wip
This has the .sframe logic patch, but it looks as if I wrote that code in a 30 minute leetcode interview. I need to refactor it before I send it for review with the main PR.
Can you test with this branch with your setup?
Thanks,
Puranjay
Song Liu
maintainerQuoted 56 lines · Puranjay Mohan
Quoted 41 lines · Song Liu
Quoted 25 lines · Puranjay Mohan
Quoted 22 lines · Song Liu
Quoted 12 lines · Indu Bhagat
Quoted 9 lines · Song Liu
I run some tests with this set and my RFC set [1]. Most of the test is done with kpatch-build. I tested both Puranjay's version [3] and my version [4].
For gcc 14.2.1, I have seen the following issue with this test [2]. This happens with both upstream and 6.13.2. The livepatch loaded fine, but the system spilled out the following warning quickly.
Quoted 6 lines
In presence of the issue
https://sourceware.org/bugzilla/show_bug.cgi?id=32666, I'd expect bad data in SFrame section. Which may be causing this symptom?To be clear, the issue affects loaded kernel modules. I cannot tell for certain - is there module loading involved in your test ?
The KLP is a module, I guess that is also affected?
Quoted 8 lines
Quoted 4 lines
During kpatch-build, we added some logic to drop the .sframe section. I guess this is wrong, as we need the .sframe section when we apply the next KLP. However, I don't think the issue is caused by missing .sframe section.
Hi, I did the same testing and did not get the Warning.
I am testing on the 6.12.11 kernel with GCC 11.4.1.
Could you please also try kernel 6.13.2?
Quoted 1 line
Just to verify, the patch we are testing is:
Yes, this is the test patch.
Quoted 1 line
Quoted 9 lines
Quoted 1 line
--- >8 ---
[...]
Quoted 4 lines
--- 8< ---
P.S. - I have a downstream patch for create-diff-object to generate .sframe sections for livepatch module, will add it to the PR after some cleanups.Yeah, I think the .sframe section is still needed.
Hi Song,
Quoted 8 lines
Can you try with this:
https://github.com/puranjaymohan/kpatch/tree/arm64_wipThis has the .sframe logic patch, but it looks as if I wrote that code in a 30 minute leetcode interview. I need to refactor it before I send it for review with the main PR.
Can you test with this branch with your setup?
This branch has the same issue as the other branch.
Song
Quoted 31 lines · Song Liu
Quoted 30 lines · Puranjay Mohan
Quoted 26 lines · Song Liu
Quoted 12 lines · Indu Bhagat
Quoted 9 lines · Song Liu
I run some tests with this set and my RFC set [1]. Most of the test is done with kpatch-build. I tested both Puranjay's version [3] and my version [4].
For gcc 14.2.1, I have seen the following issue with this test [2]. This happens with both upstream and 6.13.2. The livepatch loaded fine, but the system spilled out the following warning quickly.
Quoted 6 lines
In presence of the issue
https://sourceware.org/bugzilla/show_bug.cgi?id=32666, I'd expect bad data in SFrame section. Which may be causing this symptom?To be clear, the issue affects loaded kernel modules. I cannot tell for certain - is there module loading involved in your test ?
The KLP is a module, I guess that is also affected?
During kpatch-build, we added some logic to drop the .sframe section. I guess this is wrong, as we need the .sframe section when we apply the next KLP. However, I don't think the issue is caused by missing .sframe section.
Quoted 6 lines
Quoted 3 lines
Hi, I did the same testing and did not get the Warning.
I am testing on the 6.12.11 kernel with GCC 11.4.1.
Could you please also try kernel 6.13.2?
I have tested with 6.13.2, here are the results:
[ec2-user@ip-172-31-32-86 ~]$ uname -r
6.13.2+
My tree looks like:
[ec2-user@ip-172-31-32-86 linux-upstream]$ git log --oneline bdf4086dd (HEAD) arm64: Enable livepatch for ARM64 3904c7058 arm64: Define TIF_PATCH_PENDING for livepatch b5de442a9 unwind: arm64: add reliable stacktrace support for arm64 fdc243fa8 unwind: arm64: Add sframe unwinder on arm64 87681b7c0 unwind: Implement generic sframe unwinder library c8833ba50 unwind: add sframe v2 headerf3f3863cf arm64: entry: add unwind info for various kernel entries 57ab97f05 unwind: build kernel with sframe info 0da4b4b84 Linux 6.13.2
I have also fixed the 87681b7c0 ("unwind: Implement generic sframe unwinder library")
with:
--- >8 ---
kernel/sframe_lookup.c +2 -2
@@ -82,7 +82,7 @@ static struct sframe_fde *find_fde(const struct sframe_table *tbl, unsigned longif (f >= tbl->sfhdr_p->num_fdes || f <0 )return NULL ; fdep = tbl->fde_p + f; -if (ip < fdep->start_addr || ip >= fdep->start_addr + fdep->size) +if (ip < fdep->start_addr || ip > fdep->start_addr + fdep->size)return NULL ;return fdep;
@@ -106,7 +106,7 @@ static int find_fre(const struct sframe_table *tbl, unsigned long pc,else ip_off = (int32_t )(pc - (unsigned long )tbl->sfhdr_p) - fdep->start_addr; -if (ip_off <0 || ip_off >= fdep->size) +if (ip_off <0 || ip_off > fdep->size)return -EINVAL ;/*
--- 8< ---
GCC is gcc (GCC) 11.4.1 20230605 (Red Hat 11.4.1-2)
kpatch is from https://github.com/puranjaymohan/kpatch/tree/arm64_wip
I run the with following:
./kpatch/kpatch-build/kpatch-build --non-replace -d -v \
linux-upstream/vmlinux -s linux-upstream/ -c linux-upstream/.config \kpatch/test/integration/linux-6.2.0/special-static.patch
and my dmesg output is:
[ec2-user@ip-172-31-32-86 ~]$ sudo dmesg
[ 167.202596] livepatch_special_static: loading out-of-tree module taints kernel. [ 167.203217] livepatch_special_static: tainting kernel with TAINT_LIVEPATCH [ 167.203760] livepatch_special_static: module verification failed: signature and/or required key missing - tainting kernel [ 167.205659] livepatch: enabling patch 'livepatch_special_static' [ 167.207358] livepatch: 'livepatch_special_static': starting patching transition [ 168.264901] livepatch: 'livepatch_special_static': patching complete [ 410.641806] livepatch: 'livepatch_special_static': starting unpatching transition [ 412.389369] livepatch: 'livepatch_special_static': unpatching complete
I even ran stress-ng with the livepatch loaded to see if something happens.
P.S. - The livepatch doesn't have copy_process() but only copy_signal(), yours had copy_process() somehow.
Here is the symbol table of the .ko
Symbol table '.symtab' contains 169 entries:
Num: Value Size Type Bind Vis Ndx Name
0: 0000000000000000 0 NOTYPE LOCAL DEFAULT UND
1: 0000000000000000 0 SECTION LOCAL DEFAULT 1 .note.gnu.build-id
2: 0000000000000000 0 SECTION LOCAL DEFAULT 2 .note.Linux
3: 0000000000000000 0 SECTION LOCAL DEFAULT 3 .text
4: 0000000000000000 0 SECTION LOCAL DEFAULT 5 .exit.text
5: 0000000000000000 0 SECTION LOCAL DEFAULT 7 .init.text
6: 0000000000000000 0 SECTION LOCAL DEFAULT 9 .text.copy_signal
7: 0000000000000000 0 SECTION LOCAL DEFAULT 11 .text.kpatch_foo
8: 0000000000000000 0 SECTION LOCAL DEFAULT 13 .altinstructions
9: 0000000000000000 0 SECTION LOCAL DEFAULT 15 __patchable_function_entries
10: 0000000000000000 0 SECTION LOCAL DEFAULT 17 .codetag.alloc_tags
11: 0000000000000000 0 SECTION LOCAL DEFAULT 18 .plt
12: 0000000000000000 0 SECTION LOCAL DEFAULT 19 .init.plt
13: 0000000000000000 0 SECTION LOCAL DEFAULT 20 .text.ftrace_trampoline
14: 0000000000000000 0 SECTION LOCAL DEFAULT 21 .rodata.str1.8
15: 0000000000000000 0 SECTION LOCAL DEFAULT 22 .modinfo
16: 0000000000000000 0 SECTION LOCAL DEFAULT 23 .sframe
17: 0000000000000000 0 SECTION LOCAL DEFAULT 25 .rodata.trace_raw_output_task_newtask.str1.8
18: 0000000000000000 0 SECTION LOCAL DEFAULT 26 .rodata.trace_raw_output_task_rename.str1.8
19: 0000000000000000 0 SECTION LOCAL DEFAULT 27 .rodata.sighand_ctor.str1.8
20: 0000000000000000 0 SECTION LOCAL DEFAULT 28 .rodata.str
21: 0000000000000000 0 SECTION LOCAL DEFAULT 29 .rodata.__mmdrop.str1.8
22: 0000000000000000 0 SECTION LOCAL DEFAULT 30 .rodata.copy_signal.str1.8
23: 0000000000000000 0 SECTION LOCAL DEFAULT 31 .rodata.mm_init.str1.8
24: 0000000000000000 0 SECTION LOCAL DEFAULT 32 .rodata.vm_area_alloc.str1.8
25: 0000000000000000 0 SECTION LOCAL DEFAULT 33 .rodata.dup_mmap.str1.8
26: 0000000000000000 0 SECTION LOCAL DEFAULT 34 .rodata.copy_process.str1.8
27: 0000000000000000 0 SECTION LOCAL DEFAULT 35 .rodata.kernel_clone.str1.8
28: 0000000000000000 0 SECTION LOCAL DEFAULT 36 .rodata.str__task__trace_system_name
29: 0000000000000000 0 SECTION LOCAL DEFAULT 37 .kpatch.strings
30: 0000000000000000 0 SECTION LOCAL DEFAULT 38 .kpatch.funcs
31: 0000000000000000 0 SECTION LOCAL DEFAULT 40 __versions
32: 0000000000000000 0 SECTION LOCAL DEFAULT 41 .bss
33: 0000000000000000 0 SECTION LOCAL DEFAULT 42 .bss.__key.4
34: 0000000000000000 0 SECTION LOCAL DEFAULT 43 .bss.__key.5
35: 0000000000000000 0 SECTION LOCAL DEFAULT 44 .bss.__key.6
36: 0000000000000000 0 SECTION LOCAL DEFAULT 45 .note.GNU-stack
37: 0000000000000000 0 SECTION LOCAL DEFAULT 46 .comment
38: 0000000000000000 0 SECTION LOCAL DEFAULT 47 .data
39: 0000000000000000 0 SECTION LOCAL DEFAULT 49 .exit.data
40: 0000000000000000 0 SECTION LOCAL DEFAULT 51 .init.data
41: 0000000000000000 0 SECTION LOCAL DEFAULT 53 .kpatch.callbacks.pre_patch
42: 0000000000000000 0 SECTION LOCAL DEFAULT 54 .kpatch.callbacks.post_patch
43: 0000000000000000 0 SECTION LOCAL DEFAULT 55 .kpatch.callbacks.pre_unpatch
44: 0000000000000000 0 SECTION LOCAL DEFAULT 56 .kpatch.callbacks.post_unpatch
45: 0000000000000000 0 SECTION LOCAL DEFAULT 57 .kpatch.force
46: 0000000000000000 0 SECTION LOCAL DEFAULT 58 .gnu.linkonce.this_module
47: 0000000000000000 0 SECTION LOCAL DEFAULT 60 .debug_info
48: 0000000000000000 0 SECTION LOCAL DEFAULT 62 .debug_abbrev
49: 0000000000000000 0 SECTION LOCAL DEFAULT 63 .debug_loc
50: 0000000000000000 0 SECTION LOCAL DEFAULT 65 .debug_aranges
51: 0000000000000000 0 SECTION LOCAL DEFAULT 67 .debug_ranges
52: 0000000000000000 0 SECTION LOCAL DEFAULT 69 .debug_line
53: 0000000000000000 0 SECTION LOCAL DEFAULT 71 .debug_str
54: 0000000000000000 0 SECTION LOCAL DEFAULT 72 .debug_frame
55: 0000000000000000 0 FILE LOCAL DEFAULT ABS module-common.c
56: 0000000000000062 52 OBJECT LOCAL DEFAULT 22 __UNIQUE_ID_vermagic547
57: 0000000000000000 0 NOTYPE LOCAL DEFAULT 2 $d
58: 0000000000000000 24 OBJECT LOCAL DEFAULT 2 _note_19
59: 0000000000000018 52 OBJECT LOCAL DEFAULT 2 _note_18
60: 0000000000000000 0 FILE LOCAL DEFAULT ABS patch-hook.c
61: 0000000000000000 0 NOTYPE LOCAL DEFAULT 3 $x
62: 0000000000000000 0 NOTYPE LOCAL DEFAULT 15 $d
63: 0000000000000008 104 FUNC LOCAL DEFAULT 3 patch_free_livepatch
64: 0000000000000000 0 NOTYPE LOCAL DEFAULT 5 $x
65: 0000000000000000 36 FUNC LOCAL DEFAULT 5 patch_exit
66: 0000000000000078 260 FUNC LOCAL DEFAULT 3 patch_free_scaffold
67: 0000000000000000 0 NOTYPE LOCAL DEFAULT 21 $d
68: 0000000000000188 280 FUNC LOCAL DEFAULT 3 patch_find_object_by_name
69: 00000000000002a8 432 FUNC LOCAL DEFAULT 3 add_callbacks_to_patch_objects
70: 0000000000000000 0 NOTYPE LOCAL DEFAULT 7 $x
71: 0000000000000008 584 FUNC LOCAL DEFAULT 7 patch_init
72: 0000000000000000 0 NOTYPE LOCAL DEFAULT 47 $d
73: 0000000000000000 16 OBJECT LOCAL DEFAULT 47 patch_objects
74: 0000000000000000 0 NOTYPE LOCAL DEFAULT 41 $d
75: 0000000000000000 8 OBJECT LOCAL DEFAULT 41 lpatch
76: 0000000000000008 4 OBJECT LOCAL DEFAULT 41 patch_objects_nr
77: 0000000000000000 0 NOTYPE LOCAL DEFAULT 49 $d
78: 0000000000000000 8 OBJECT LOCAL DEFAULT 49 __UNIQUE_ID___addressable_cleanup_module737
79: 0000000000000000 0 NOTYPE LOCAL DEFAULT 51 $d
80: 0000000000000000 8 OBJECT LOCAL DEFAULT 51 __UNIQUE_ID___addressable_init_module736
81: 0000000000000000 12 OBJECT LOCAL DEFAULT 22 __UNIQUE_ID_livepatch739
82: 000000000000000c 12 OBJECT LOCAL DEFAULT 22 __UNIQUE_ID_license738
83: 0000000000000000 0 FILE LOCAL DEFAULT ABS fork.c
84: 0000000000000008 496 FUNC LOCAL DEFAULT 9 copy_signal
85: 0000000000000000 0 NOTYPE LOCAL DEFAULT 25 $d
86: 0000000000000000 0 NOTYPE LOCAL DEFAULT 26 $d
87: 0000000000000000 0 NOTYPE LOCAL DEFAULT 27 $d
88: 0000000000000000 0 NOTYPE LOCAL DEFAULT 29 $d
89: 0000000000000000 0 NOTYPE LOCAL DEFAULT 30 $d
90: 0000000000000000 0 NOTYPE LOCAL DEFAULT 31 $d
91: 0000000000000000 0 NOTYPE LOCAL DEFAULT 32 $d
92: 0000000000000000 0 NOTYPE LOCAL DEFAULT 33 $d
93: 0000000000000128 0 NOTYPE LOCAL DEFAULT 21 $d
94: 0000000000000000 0 NOTYPE LOCAL DEFAULT 34 $d
95: 0000000000000000 0 NOTYPE LOCAL DEFAULT 35 $d
96: 0000000000000000 0 OBJECT LOCAL DEFAULT 42 __key.4
97: 0000000000000000 0 OBJECT LOCAL DEFAULT 43 __key.5
98: 0000000000000000 0 OBJECT LOCAL DEFAULT 44 __key.6
99: 0000000000000000 5 OBJECT LOCAL DEFAULT 36 str__task__trace_system_name
100: 0000000000000000 0 NOTYPE LOCAL DEFAULT 36 $d
101: 0000000000000000 0 NOTYPE LOCAL DEFAULT 28 .L14472^B1
102: 000000000000000e 0 NOTYPE LOCAL DEFAULT 28 .L14472^B2
103: 000000000000001c 0 NOTYPE LOCAL DEFAULT 28 .L14472^B3
104: 000000000000002a 0 NOTYPE LOCAL DEFAULT 28 .L14472^B4
105: 0000000000000038 0 NOTYPE LOCAL DEFAULT 28 .L14472^B5
106: 0000000000000046 0 NOTYPE LOCAL DEFAULT 28 .L14472^B6
107: 0000000000000054 0 NOTYPE LOCAL DEFAULT 28 .L14472^B7
108: 0000000000000062 0 NOTYPE LOCAL DEFAULT 28 .L14472^B8
109: 0000000000000070 0 NOTYPE LOCAL DEFAULT 28 .L14472^B9
110: 000000000000007e 0 NOTYPE LOCAL DEFAULT 28 .L14472^B10
111: 000000000000008c 0 NOTYPE LOCAL DEFAULT 28 .L14472^B11
112: 00000000000000a9 0 NOTYPE LOCAL DEFAULT 28 .L14472^B12
113: 00000000000000c1 0 NOTYPE LOCAL DEFAULT 28 .L14472^B13
114: 00000000000000d9 0 NOTYPE LOCAL DEFAULT 28 .L14472^B14
115: 00000000000000f1 0 NOTYPE LOCAL DEFAULT 28 .L14472^B15
116: 0000000000000109 0 NOTYPE LOCAL DEFAULT 28 .L14472^B16
117: 0000000000000124 0 NOTYPE LOCAL DEFAULT 28 .L14472^B17
118: 0000000000000132 0 NOTYPE LOCAL DEFAULT 28 .L14472^B18
119: 000000000000014d 0 NOTYPE LOCAL DEFAULT 28 .L14472^B19
120: 0000000000000168 0 NOTYPE LOCAL DEFAULT 28 .L14472^B20
121: 0000000000000000 0 FILE LOCAL DEFAULT ABS livepatch-special-static.mod.c
122: 0000000000000000 0 NOTYPE LOCAL DEFAULT 58 $d
123: 0000000000000018 35 OBJECT LOCAL DEFAULT 22 __UNIQUE_ID_srcversion549
124: 000000000000003b 9 OBJECT LOCAL DEFAULT 22 __UNIQUE_ID_depends548
125: 0000000000000044 30 OBJECT LOCAL DEFAULT 22 __UNIQUE_ID_name547
126: 0000000000000000 0 NOTYPE LOCAL DEFAULT 40 $d
127: 0000000000000000 0 OBJECT LOCAL DEFAULT OS [0xff20] .klp.sym.vmlinux.signal_cachep,1
128: 0000000000000000 1216 OBJECT LOCAL DEFAULT 40 ____versions
129: 0000000000000000 0 NOTYPE GLOBAL DEFAULT 55 __kpatch_callbacks_pre_unpatch_end
130: 0000000000000000 0 NOTYPE GLOBAL DEFAULT UND __init_rwsem
131: 0000000000000000 0 NOTYPE GLOBAL DEFAULT UND __list_add_valid_or_report
132: 0000000000000000 0 NOTYPE GLOBAL DEFAULT UND __kmalloc_noprof
133: 0000000000000000 1344 OBJECT GLOBAL DEFAULT 58 __this_module
134: 0000000000000000 0 NOTYPE GLOBAL DEFAULT 54 __kpatch_callbacks_post_patch
135: 0000000000000000 0 NOTYPE GLOBAL DEFAULT 57 __kpatch_force_funcs
136: 0000000000000000 0 NOTYPE GLOBAL DEFAULT 17 __stop_alloc_tags
137: 0000000000000000 0 NOTYPE GLOBAL DEFAULT UND hrtimer_init
138: 0000000000000000 36 FUNC GLOBAL DEFAULT 5 cleanup_module
139: 0000000000000000 0 NOTYPE GLOBAL DEFAULT UND memcpy
140: 0000000000000000 0 NOTYPE GLOBAL DEFAULT UND klp_enable_patch
141: 0000000000000000 0 NOTYPE GLOBAL DEFAULT UND kfree
142: 0000000000000000 0 NOTYPE GLOBAL DEFAULT 53 __kpatch_callbacks_pre_patch
143: 0000000000000008 584 FUNC GLOBAL DEFAULT 7 init_module
144: 0000000000000000 0 NOTYPE GLOBAL DEFAULT 55 __kpatch_callbacks_pre_unpatch
145: 0000000000000000 0 NOTYPE GLOBAL DEFAULT 17 __start_alloc_tags
146: 0000000000000000 0 NOTYPE GLOBAL DEFAULT 38 __kpatch_funcs
147: 0000000000000000 0 NOTYPE GLOBAL DEFAULT UND _printk
148: 0000000000000000 0 NOTYPE GLOBAL DEFAULT 57 __kpatch_force_funcs_end
149: 0000000000000000 0 NOTYPE GLOBAL DEFAULT UND queued_spin_lock_slowpath
150: 0000000000000038 0 NOTYPE GLOBAL DEFAULT 38 __kpatch_funcs_end
151: 0000000000000000 0 NOTYPE GLOBAL DEFAULT 54 __kpatch_callbacks_post_patch_end
152: 0000000000000000 0 NOTYPE GLOBAL DEFAULT 53 __kpatch_callbacks_pre_patch_end
153: 0000000000000000 0 NOTYPE GLOBAL DEFAULT UND __list_del_entry_valid_or_report
154: 0000000000000000 0 NOTYPE GLOBAL DEFAULT UND __mutex_init
155: 0000000000000008 60 FUNC GLOBAL DEFAULT 11 kpatch_foo
156: 0000000000000000 0 NOTYPE GLOBAL DEFAULT 56 __kpatch_callbacks_post_unpatch_end
157: 0000000000000000 0 NOTYPE GLOBAL DEFAULT UND kmem_cache_alloc_noprof
158: 0000000000000000 0 NOTYPE GLOBAL DEFAULT UND __init_waitqueue_head
159: 0000000000000000 0 NOTYPE GLOBAL DEFAULT UND strcmp
160: 0000000000000000 0 NOTYPE GLOBAL DEFAULT UND jiffies
161: 0000000000000000 0 NOTYPE GLOBAL DEFAULT 56 __kpatch_callbacks_post_unpatch
162: 0000000000000000 0 NOTYPE GLOBAL DEFAULT UND __kmalloc_cache_noprof
163: 0000000000000000 0 NOTYPE GLOBAL DEFAULT UND alt_cb_patch_nops
164: 0000000000000000 0 NOTYPE GLOBAL DEFAULT UND kmalloc_caches
165: 0000000000000000 0 NOTYPE GLOBAL DEFAULT OS [0xff20] .klp.sym.vmlinux.it_real_fn,0
166: 0000000000000000 0 NOTYPE GLOBAL DEFAULT OS [0xff20] .klp.sym.vmlinux.posix_cputimers_group_init,0
167: 0000000000000000 0 NOTYPE GLOBAL DEFAULT OS [0xff20] .klp.sym.vmlinux.tty_audit_fork,0
168: 0000000000000000 0 NOTYPE GLOBAL DEFAULT OS [0xff20] .klp.sym.vmlinux.sched_autogroup_fork,0and Sections:
[ec2-user@ip-172-31-32-86 ~]$ readelf -WS livepatch-special-static.ko There are 78 section headers, starting at offset 0x1727d8:
Section Headers:
[Nr] Name Type Address Off Size ES Flg Lk Inf Al [ 0] NULL 0000000000000000 000000 000000 00 0 0 0 [ 1] .note.gnu.build-id NOTE 0000000000000000 000040 000024 00 A 0 0 4 [ 2] .note.Linux NOTE 0000000000000000 000064 00004c 00 A 0 0 4 [ 3] .text PROGBITS 0000000000000000 0000c0 000458 00 AX 0 0 32 [ 4] .rela.text RELA 0000000000000000 000518 000570 18 I 74 3 8 [ 5] .exit.text PROGBITS 0000000000000000 000a88 000024 00 AX 0 0 8 [ 6] .rela.exit.text RELA 0000000000000000 000ab0 000048 18 I 74 5 8 [ 7] .init.text PROGBITS 0000000000000000 000b00 000250 00 AX 0 0 32 [ 8] .rela.init.text RELA 0000000000000000 000d50 000348 18 I 74 7 8 [ 9] .text.copy_signal PROGBITS 0000000000000000 001098 0001f8 00 AX 0 0 8 [10] .rela.text.copy_signal RELA 0000000000000000 001290 000240 18 I 74 9 8 [11] .text.kpatch_foo PROGBITS 0000000000000000 0014d0 000044 00 AX 0 0 8 [12] .rela.text.kpatch_foo RELA 0000000000000000 001518 000078 18 I 74 11 8 [13] .altinstructions PROGBITS 0000000000000000 001590 00000c 00 A 0 0 8 [14] .rela.altinstructions RELA 0000000000000000 0015a0 000030 18 I 74 13 8 [15] __patchable_function_entries PROGBITS 0000000000000010 0015d0 000038 00 WAL 3 0 8 [16] .rela__patchable_function_entries RELA 0000000000000000 001608 0000a8 18 I 74 15 8 [17] .codetag.alloc_tags PROGBITS 0000000000000540 0016b0 000000 00 W 0 0 1 [18] .plt PROGBITS 0000000000000000 0016b0 000001 00 AX 0 0 1 [19] .init.plt PROGBITS 0000000000000000 0016b1 000001 00 A 0 0 1 [20] .text.ftrace_trampoline PROGBITS 0000000000000000 0016b2 000001 00 AX 0 0 1 [21] .rodata.str1.8 PROGBITS 0000000000000000 0016b8 000455 01 AMS 0 0 8 [22] .modinfo PROGBITS 0000000000000000 001b0d 000096 00 A 0 0 1 [23] .sframe PROGBITS 0000000000000000 001ba8 0001ad 00 A 0 0 8 [24] .rela.sframe RELA 0000000000000000 001d58 0000c0 18 I 74 23 8 [25] .rodata.trace_raw_output_task_newtask.str1.8 PROGBITS 0000000000000000 001e18 000032 01 AMS 0 0 8 [26] .rodata.trace_raw_output_task_rename.str1.8 PROGBITS 0000000000000000 001e50 000030 01 AMS 0 0 8 [27] .rodata.sighand_ctor.str1.8 PROGBITS 0000000000000000 001e80 000017 01 AMS 0 0 8 [28] .rodata.str PROGBITS 0000000000000000 001e97 000180 01 AMS 0 0 1 [29] .rodata.__mmdrop.str1.8 PROGBITS 0000000000000000 002018 00006b 01 AMS 0 0 8 [30] .rodata.copy_signal.str1.8 PROGBITS 0000000000000000 002088 00005f 01 AMS 0 0 8 [31] .rodata.mm_init.str1.8 PROGBITS 0000000000000000 0020e8 00000f 01 AMS 0 0 8 [32] .rodata.vm_area_alloc.str1.8 PROGBITS 0000000000000000 0020f8 000014 01 AMS 0 0 8 [33] .rodata.dup_mmap.str1.8 PROGBITS 0000000000000000 002110 000054 01 AMS 0 0 8 [34] .rodata.copy_process.str1.8 PROGBITS 0000000000000000 002168 000017 01 AMS 0 0 8 [35] .rodata.kernel_clone.str1.8 PROGBITS 0000000000000000 002180 000009 01 AMS 0 0 8 [36] .rodata.str__task__trace_system_name PROGBITS 0000000000000000 002190 000005 00 A 0 0 8 [37] .kpatch.strings PROGBITS 0000000000000000 002195 00006c 00 A 0 0 1 [38] .kpatch.funcs PROGBITS 0000000000000000 002208 000038 00 A 0 0 8 [39] .rela.kpatch.funcs RELA 0000000000000000 002240 000048 18 I 74 38 8 [40] __versions PROGBITS 0000000000000000 002288 0004c0 00 A 0 0 8 [41] .bss NOBITS 0000000000000000 002748 00000c 00 WA 0 0 8 [42] .bss.__key.4 NOBITS 0000000000000000 002748 000000 00 WA 0 0 8 [43] .bss.__key.5 NOBITS 0000000000000000 002748 000000 00 WA 0 0 8 [44] .bss.__key.6 NOBITS 0000000000000000 002748 000000 00 WA 0 0 8 [45] .note.GNU-stack PROGBITS 0000000000000000 002748 000000 00 0 0 1 [46] .comment PROGBITS 0000000000000000 002748 00008d 01 MS 0 0 1 [47] .data PROGBITS 0000000000000048 0027d8 000010 00 WA 0 0 8 [48] .rela.data RELA 0000000000000000 0027e8 000030 18 I 74 47 8 [49] .exit.data PROGBITS 0000000000000000 002818 000008 00 WA 0 0 8 [50] .rela.exit.data RELA 0000000000000000 002820 000018 18 I 74 49 8 [51] .init.data PROGBITS 0000000000000000 002838 000008 00 WA 0 0 8 [52] .rela.init.data RELA 0000000000000000 002840 000018 18 I 74 51 8 [53] .kpatch.callbacks.pre_patch PROGBITS 0000000000000000 002858 000008 00 WA 0 0 1 [54] .kpatch.callbacks.post_patch PROGBITS 0000000000000000 002860 000008 00 WA 0 0 1 [55] .kpatch.callbacks.pre_unpatch PROGBITS 0000000000000000 002868 000008 00 WA 0 0 1 [56] .kpatch.callbacks.post_unpatch PROGBITS 0000000000000000 002870 000008 00 WA 0 0 1 [57] .kpatch.force PROGBITS 0000000000000000 002878 000008 00 WA 0 0 1 [58] .gnu.linkonce.this_module PROGBITS 0000000000000000 002880 000540 00 WA 0 0 64 [59] .rela.gnu.linkonce.this_module RELA 0000000000000000 002dc0 000030 18 I 74 58 8 [60] .debug_info PROGBITS 0000000000000000 002df0 06041d 00 0 0 1 [61] .rela.debug_info RELA 0000000000000000 063210 08a390 18 I 74 60 8 [62] .debug_abbrev PROGBITS 0000000000000000 0ed5a0 002392 00 0 0 1 [63] .debug_loc PROGBITS 0000000000000000 0ef932 031dd6 00 0 0 1 [64] .rela.debug_loc RELA 0000000000000000 121708 004e30 18 I 74 63 8 [65] .debug_aranges PROGBITS 0000000000000000 126538 000740 00 0 0 1 [66] .rela.debug_aranges RELA 0000000000000000 126c78 0000d8 18 I 74 65 8 [67] .debug_ranges PROGBITS 0000000000000000 126d50 00a870 00 0 0 1 [68] .rela.debug_ranges RELA 0000000000000000 1315c0 001320 18 I 74 67 8 [69] .debug_line PROGBITS 0000000000000000 1328e0 010623 00 0 0 1 [70] .rela.debug_line RELA 0000000000000000 142f08 000078 18 I 74 69 8 [71] .debug_str PROGBITS 0000000000000000 142f80 02b764 01 MS 0 0 1 [72] .debug_frame PROGBITS 0000000000000000 16e6e8 001900 00 0 0 8 [73] .rela.debug_frame RELA 0000000000000000 16ffe8 000b28 18 I 74 72 8 [74] .symtab SYMTAB 0000000000000000 170b10 000fd8 18 75 129 8 [75] .strtab STRTAB 0000000000000000 171ae8 0006cc 00 0 0 1 [76] .shstrtab STRTAB 0000000000000000 1721b4 000576 00 0 0 1
[77] .klp.rela.vmlinux..text.copy_signal RELA 0000000000000000 172730 0000a8 18 AIo 74 9 8
Thanks,
Puranjay
Song Liu
maintainerOn Thu, Feb 13, 2025 at 12:38 AM Puranjay Mohan <[email protected]> wrote:
[...]
Quoted 3 lines
P.S. - The livepatch doesn't havecopy_process()but onlycopy_signal(), yours hadcopy_process()somehow.
In my build, copy_signal is inlined to copy_process, unless I add noinline. If I do add noinline, the issue will not reproduce.
I tried more combinations. The issue doesn't reproduce if I either
1) add noinline to copy_signal, so we are not patching the whole copy_process function;
or
2) Switch compiler from gcc 14.2.1 to gcc 11.5.0.
So it appears something in gcc 14.2.1 is causing live patch to fail for copy_process().
Thanks,
Song
[...]
Quoted 9 lines · Song Liu
On Thu, Feb 13, 2025 at 12:38 AM Puranjay Mohan <[email protected]> wrote:
[...]Quoted 3 lines
P.S. - The livepatch doesn't havecopy_process()but onlycopy_signal(), yours hadcopy_process()somehow.In my build,
copy_signalis inlined tocopy_process, unless I add noinline. If I do add noinline, the issue will not reproduce.
Quoted 8 lines
I tried more combinations. The issue doesn't reproduce if I either
1) add noinline tocopy_signal, so we are not patching the wholecopy_processfunction;
or
2) Switch compiler from gcc 14.2.1 to gcc 11.5.0.So it appears something in gcc 14.2.1 is causing live patch to fail for
copy_process().
So, can you test your RFC set (without SFRAME) with gcc 14.2.1, so we can be sure that it is not a sframe problem?
And about having the .sframe section in the livepatch module, I realised that this set doesn't include support for reading/using sframe data from any module(livepatches included), so the patch I added for generating .sframe in kpatch is irrelevant because it is a no-op with the current setup.
Thanks,
Puranjay
Song Liu
maintainerQuoted 18 lines · Puranjay Mohan
Quoted 15 lines · Song Liu
On Thu, Feb 13, 2025 at 12:38 AM Puranjay Mohan <[email protected]> wrote:
[...]Quoted 3 lines
P.S. - The livepatch doesn't havecopy_process()but onlycopy_signal(), yours hadcopy_process()somehow.In my build,
copy_signalis inlined tocopy_process, unless I add noinline. If I do add noinline, the issue will not reproduce.I tried more combinations. The issue doesn't reproduce if I either
1) add noinline tocopy_signal, so we are not patching the wholecopy_processfunction;
or
2) Switch compiler from gcc 14.2.1 to gcc 11.5.0.
Quoted 5 lines
Quoted 2 lines
So it appears something in gcc 14.2.1 is causing live patch to fail for
copy_process().So, can you test your RFC set (without SFRAME) with gcc 14.2.1, so we can be sure that it is not a sframe problem?
My RFC set is the same. No issue with gcc 11.5.0; but hits the same WARNING with gcc 14.2.1. My previous tests are all with clang. I didn't see a similar issue there.
Quoted 4 lines
And about having the .sframe section in the livepatch module, I realised that this set doesn't include support for reading/using sframe data from any module(livepatches included), so the patch I added for generating .sframe in kpatch is irrelevant because it is a no-op with the current setup.
Agreed, this issue should be irrelevant to the .sframe section in the .ko file.
Thanks,
Song
Song Liu
maintainerQuoted 21 lines · Puranjay Mohan
Quoted 9 lines · Song Liu
On Thu, Feb 13, 2025 at 12:38 AM Puranjay Mohan <[email protected]> wrote:
[...]Quoted 3 lines
P.S. - The livepatch doesn't havecopy_process()but onlycopy_signal(), yours hadcopy_process()somehow.In my build,
copy_signalis inlined tocopy_process, unless I add noinline. If I do add noinline, the issue will not reproduce.Quoted 8 lines
I tried more combinations. The issue doesn't reproduce if I either
1) add noinline tocopy_signal, so we are not patching the wholecopy_processfunction;
or
2) Switch compiler from gcc 14.2.1 to gcc 11.5.0.So it appears something in gcc 14.2.1 is causing live patch to fail for
copy_process().
Quoted 7 lines
So, can you test your RFC set (without SFRAME) with gcc 14.2.1, so we can be sure that it is not a sframe problem?
And about having the .sframe section in the livepatch module, I realised that this set doesn't include support for reading/using sframe data from any module(livepatches included), so the patch I added for generating .sframe in kpatch is irrelevant because it is a no-op with the current setup.
Puranjay,
Could you please try the following?
1. Use gcc 11.4.1;
2. Add __always_inline to copy_signal();
3. Build kernel, and livepatch with the same test (we need to add __always_inline to the .patch file).
4. Run gdb livepatch-xxx.ko
5. In gdb do disassemble copy_process.
In my tests, both gcc-14.2.1 and gcc-11.5.0 generated a .ko file that looks weird in gdb-disassemble. Specifically, readels shows copy_process is about 5.5kB, but gdb-disassemble only shows 140 bytes or so for copy_process. clang doesn't seem to have this problem.
I am really curious whether you have the same problem in your setup.
Thanks,
Song
Quoted 41 lines · Song Liu
Quoted 21 lines · Puranjay Mohan
Quoted 9 lines · Song Liu
On Thu, Feb 13, 2025 at 12:38 AM Puranjay Mohan <[email protected]> wrote:
[...]Quoted 3 lines
P.S. - The livepatch doesn't havecopy_process()but onlycopy_signal(), yours hadcopy_process()somehow.In my build,
copy_signalis inlined tocopy_process, unless I add noinline. If I do add noinline, the issue will not reproduce.Quoted 8 lines
I tried more combinations. The issue doesn't reproduce if I either
1) add noinline tocopy_signal, so we are not patching the wholecopy_processfunction;
or
2) Switch compiler from gcc 14.2.1 to gcc 11.5.0.So it appears something in gcc 14.2.1 is causing live patch to fail for
copy_process().Quoted 7 lines
So, can you test your RFC set (without SFRAME) with gcc 14.2.1, so we can be sure that it is not a sframe problem?
And about having the .sframe section in the livepatch module, I realised that this set doesn't include support for reading/using sframe data from any module(livepatches included), so the patch I added for generating .sframe in kpatch is irrelevant because it is a no-op with the current setup.
Puranjay,
Could you please try the following?
1. Use gcc 11.4.1;
2. Add__always_inlinetocopy_signal();
3. Build kernel, and livepatch with the same test (we need to add__always_inlineto the .patch file).
4. Run gdb livepatch-xxx.ko
5. In gdb do disassemblecopy_process.
Quoted 8 lines
In my tests, both gcc-14.2.1 and gcc-11.5.0 generated a .ko file that looks weird in gdb-disassemble. Specifically, readels shows
copy_processis about 5.5kB, but gdb-disassemble only shows 140 bytes or so forcopy_process. clang doesn't seem to have this problem.I am really curious whether you have the same problem in your setup.
Hi Song,
I did this test and found the same issue as you (gdb assembly broken), but I can see this issue even without the inlining. I think GDB tried to load the debuginfo and that is somehow broken therefore it fails to disassemblt properly.
But even with inlining, I couldn't see the warning about the refcount with my setup.
Thanks,
Puranjay
Song Liu
maintainerHi Puranjay,
Thanks for running the tests.
On Fri, Feb 14, 2025 at 12:56 AM Puranjay Mohan <[email protected]> wrote:
[...]
Quoted 10 lines
Quoted 3 lines
I am really curious whether you have the same problem in your setup.
Hi Song,
I did this test and found the same issue as you (gdb assembly broken), but I can see this issue even without the inlining. I think GDB tried to load the debuginfo and that is somehow broken therefore it fails to disassemblt properly.
Yes, this matches my observations: gcc-11 generates the .ko that confuses gdb.
I tested with two versions of gdb (10.2 and 14.2), both have the problem. OTOH, lldb is able to disassemble copy_process from a gcc-compiled .ko file properly.
Quoted 2 lines
But even with inlining, I couldn't see the warning about the refcount with my setup.
This also matches my observations. gcc-11 compiled livepatch works fine.
Thanks,
Song
Josh Poimboeuf
maintainerQuoted 4 lines · Puranjay Mohan
I did this test and found the same issue as you (gdb assembly broken), but I can see this issue even without the inlining. I think GDB tried to load the debuginfo and that is somehow broken therefore it fails to disassemblt properly.
I had the same theory about the debuginfo, but I stripped debug sections from that file and gdb still got confused.
Still, the symbol table looked normal, so the gdb issue might be completely separate from the kernel runtime issues.
Signature · 1 line
Josh