Mark Kprobe debug exception paths noinstr
Subsystems (6)
v2 cover Mark Kprobe debug exception paths noinstr
From: Hongyan Xia <[email protected]>
Commit 879a6754d3d1 ("arm64: kprobes: Only handle faults originating from XOL slot") and 23f851ac0078 ("arm64: kprobes: Allow reentering kprobes while single-stepping") band-aided two corner cases in the arm64 kprobe path: Nested page faults inside Kprobe and nested Kprobe SS handling. Although the two patches were taken, Will asked whether we should keep patching things like this each time we find a corner case, and whether making Kprobe noinstr is the correct answer [1].
This series attempts to do exactly that: Making the Kprobe path noinstr. The kprobe handlers become noinstr end to end. Apart from marking functions noinstr, we also force-inline some static inline helper functions because certain compilers still outline them, breaking noinstr.
Note that arm64 has no objtool so at the moment noinstr is more of a contract that must be followed, instead of actually enforcing control flow. Instrumentation can still happen if someone with root privileges does something stupid, but now it is regarded as a bug rather than a legitimate use case that the kernel needs to cover.
[1] https://lore.kernel.org/all/alpuL10h7-OK2hFb@willie-the-truck/
Changed in v2:
- Remove instrumentation_{begin/end}() markers as they do nothing
- Fold __always_inline into the patches that actually need them
- Remove the debug exception path changes unrelated to Kprobe. This series is only about Kprobe for now
- Drop the kprobe_page_fault() patch for now. It's a question whether we even need it at all. Will send a patch for it separately.
Hongyan Xia (4):
arm64/entry: Make debug_exception_enter/exit() noinstr arm64/debug: Make the Kprobe functions noinstr arm64/kprobes: Make the entire Kprobe noinstr Revert "arm64: kprobes: Allow reentering kprobes while
single-stepping"
arch/arm64/include/asm/esr.h | 2 +- arch/arm64/include/asm/kprobes.h | 15 ++------ arch/arm64/include/asm/percpu.h | 2 +- arch/arm64/include/asm/preempt.h | 4 +- arch/arm64/include/asm/ptrace.h | 4 +- arch/arm64/kernel/debug-monitors.c | 6 +-- arch/arm64/kernel/entry-common.c | 13 ++----- arch/arm64/kernel/probes/kprobes.c | 59 ++++++++++-------------------- include/linux/kprobes.h | 8 ++-- 9 files changed, 38 insertions(+), 75 deletions(-)
Signature · 1 line
2.47.3
RFC PATCH v2 1/4 arm64/entry: Make debug_exception_enter/exit() noinstr
From: Hongyan Xia <[email protected]>
Commit 879a6754d3d11e30af24b7dc486f561510d62641 ran into a crash because debug_exception_enter/exit() triggered page faults caused by perf dwarf call graph tracing. That patch was a band-aid on top.
Instead of trying to band-aid all possible paths that can happen during instrumentation or perf tracing, simply force-inline these functions and use the notrace variant of preempt handling to avoid all the complexity.
Drop the RCU_LOCKDEP_WARN(): arm64_enter_el1_dbg() runs first and enters NMI context via ct_nmi_enter(), so RCU is always watching by the time debug_exception_enter() runs.
Also mark __preempt_count_add/sub() __always_inline so the preempt count updates fold into the noinstr callers instead of becoming outlined calls to instrumentable text.
--- arch/arm64/include/asm/preempt.h | 4 ++-- arch/arm64/kernel/entry-common.c | 13 ++++--------- 2 files changed, 6 insertions(+), 11 deletions(-)
arch/arm64/include/asm/preempt.h +2 -2
@@ -41,14 +41,14 @@ static inline bool test_preempt_need_resched(void)return !current_thread_info ()->preempt.need_resched; } -static inline void __preempt_count_add (int val) +static __always_inline void __preempt_count_add(int val) {u32 pc =READ_ONCE (current_thread_info ()->preempt.count); pc += val;WRITE_ONCE (current_thread_info ()->preempt.count, pc); } -static inline void __preempt_count_sub (int val) +static __always_inline void __preempt_count_sub(int val) {u32 pc =READ_ONCE (current_thread_info ()->preempt.count); pc -= val;
arch/arm64/kernel/entry-common.c +4 -9
@@ -292,20 +292,15 @@ static __always_inline void fpsimd_syscall_exit(void)* accidentally schedule in exception context and it will force a warning * if we somehow manage to schedule by accident. */ -static void debug_exception_enter (struct pt_regs *regs) +static __always_inline void debug_exception_enter(struct pt_regs *regs) { -preempt_disable (); - -/* This code is a bit fragile. Test it. */ -RCU_LOCKDEP_WARN (!rcu_is_watching (),"exception_enter didn't work" ); +preempt_disable_notrace (); } -NOKPROBE_SYMBOL (debug_exception_enter); -static void debug_exception_exit (struct pt_regs *regs) +static __always_inline void debug_exception_exit(struct pt_regs *regs) { -preempt_enable_no_resched (); +preempt_enable_no_resched_notrace (); } -NOKPROBE_SYMBOL (debug_exception_exit); UNHANDLED(el1t, 64, sync)UNHANDLED (el1t,64 , irq)
Signature · 1 line
2.47.3
Re: [RFC PATCH v2 1/4] arm64/entry: Make debug_exception_enter/exit() noinstr
Quoted 10 lines · On Thu, 13 Aug 2026 06:49:27 +0000 Hongyan Xia <[email protected]>…
From: Hongyan Xia <[email protected]>
Commit
879a6754d3d11e30af24b7dc486f561510d62641ran into a crash because debug_exception_enter/exit() triggered page faults caused by perf dwarf call graph tracing. That patch was a band-aid on top.Instead of trying to band-aid all possible paths that can happen during instrumentation or perf tracing, simply force-inline these functions and use the notrace variant of preempt handling to avoid all the complexity.
Quoted 7 lines
Drop the
RCU_LOCKDEP_WARN():arm64_enter_el1_dbg()runs first and enters NMI context viact_nmi_enter(), so RCU is always watching by the timedebug_exception_enter()runs.Also mark __preempt_count_add/sub()
__always_inlineso the preempt count updates fold into the noinstr callers instead of becoming outlined calls to instrumentable text.
Looks good to me as a kprobe maintainer.
Thanks,
Quoted 60 lines
Signed-off-by Hongyan Xia <[email protected]>--- arch/arm64/include/asm/preempt.h | 4 ++-- arch/arm64/kernel/entry-common.c | 13 ++++--------- 2 files changed, 6 insertions(+), 11 deletions(-)arch/arm64/include/asm/preempt.h +2
-2@@ -41,14 +41,14 @@ static inline bool test_preempt_need_resched(void)return !current_thread_info ()->preempt.need_resched; } -static inline void __preempt_count_add (int val) +static __always_inline void __preempt_count_add(int val) {u32 pc =READ_ONCE (current_thread_info ()->preempt.count); pc += val;WRITE_ONCE (current_thread_info ()->preempt.count, pc); } -static inline void __preempt_count_sub (int val) +static __always_inline void __preempt_count_sub(int val) {u32 pc =READ_ONCE (current_thread_info ()->preempt.count); pc -= val;arch/arm64/kernel/entry-common.c +4
-9@@ -292,20 +292,15 @@ static __always_inline void fpsimd_syscall_exit(void)* accidentally schedule in exception context and it will force a warning * if we somehow manage to schedule by accident. */ -static void debug_exception_enter (struct pt_regs *regs) +static __always_inline void debug_exception_enter(struct pt_regs *regs) { -preempt_disable (); - -/* This code is a bit fragile. Test it. */ -RCU_LOCKDEP_WARN (!rcu_is_watching (),"exception_enter didn't work" ); +preempt_disable_notrace (); } -NOKPROBE_SYMBOL (debug_exception_enter); -static void debug_exception_exit (struct pt_regs *regs) +static __always_inline void debug_exception_exit(struct pt_regs *regs) { -preempt_enable_no_resched (); +preempt_enable_no_resched_notrace (); } -NOKPROBE_SYMBOL (debug_exception_exit); UNHANDLED(el1t, 64, sync)UNHANDLED (el1t,64 , irq)Signature · 1 line
2.47.3
Signature · 1 line
Masami Hiramatsu (Google) <[email protected]>
RFC PATCH v2 2/4 arm64/debug: Make the Kprobe functions noinstr
In reply to Hongyan XiaFrom: Hongyan Xia <[email protected]>
The Kprobe debug exception path must be run with extra care. NOKPROBE isn't sufficient, as other instrumentation like ftrace still opens a can of worms that is very complex to deal with.
Mark the three main Kprobe entry points noinstr, as well as the debug exception paths that lead to and exit from these entry points. Note that noinstr attribute is stronger and can safely replace NOKPROBE and __kprobe modifiers.
Also mark esr_brk_comment() __always_inline, as clang does not reliably inline plain static inline functions into noinline (noinstr) callers such as call_el1_break_hook().
--- arch/arm64/include/asm/esr.h | 2 +- arch/arm64/include/asm/kprobes.h | 9 +++------ arch/arm64/kernel/debug-monitors.c | 6 ++---- arch/arm64/kernel/probes/kprobes.c | 6 +++--- 4 files changed, 9 insertions(+), 14 deletions(-)
arch/arm64/include/asm/esr.h +1 -1
@@ -437,7 +437,7 @@#ifndef __ASSEMBLER__#include <asm/types.h> -static inline unsigned long esr_brk_comment (unsigned long esr) +static __always_inline unsigned long esr_brk_comment (unsigned long esr) {return esr & ESR_ELx_BRK64_ISS_COMMENT_MASK; }
arch/arm64/include/asm/kprobes.h +3 -6
@@ -48,11 +48,8 @@ void __kprobes *trampoline_probe_handler(struct pt_regs *regs);#endif /* CONFIG_KPROBES */ -int __kprobes kprobe_brk_handler (struct pt_regs *regs, -unsigned long esr); -int __kprobes kprobe_ss_brk_handler (struct pt_regs *regs, -unsigned long esr); -int __kprobes kretprobe_brk_handler (struct pt_regs *regs, -unsigned long esr); +int noinstr kprobe_brk_handler(struct pt_regs *regs, unsigned long esr); +int noinstr kprobe_ss_brk_handler(struct pt_regs *regs, unsigned long esr); +int noinstr kretprobe_brk_handler(struct pt_regs *regs, unsigned long esr);#endif /* _ARM_KPROBES_H */
arch/arm64/kernel/debug-monitors.c +2 -4
@@ -207,7 +207,7 @@ void do_el1_softstep(unsigned long esr, struct pt_regs *regs) }NOKPROBE_SYMBOL (do_el1_softstep); -static int call_el1_break_hook (struct pt_regs *regs,unsigned long esr) +static int noinstr call_el1_break_hook(struct pt_regs *regs, unsigned long esr) {if (esr_brk_comment (esr) ==BUG_BRK_IMM )return bug_brk_handler (regs, esr);
@@ -245,7 +245,6 @@ static int call_el1_break_hook(struct pt_regs *regs, unsigned long esr)return DBG_HOOK_ERROR ; } -NOKPROBE_SYMBOL (call_el1_break_hook);/* * We have already unmasked interrupts and enabled preemption
@@ -261,14 +260,13 @@ void do_el0_brk64(unsigned long esr, struct pt_regs *regs)send_user_sigtrap (TRAP_BRKPT ); } -void do_el1_brk64 (unsigned long esr,struct pt_regs *regs) +void noinstr do_el1_brk64(unsigned long esr, struct pt_regs *regs) {if (call_el1_break_hook (regs, esr) ==DBG_HOOK_HANDLED )return ;die ("Oops - BRK" , regs, esr); } -NOKPROBE_SYMBOL (do_el1_brk64);#ifdef CONFIG_COMPAT void do_bkpt32 (unsigned long esr,struct pt_regs *regs)
arch/arm64/kernel/probes/kprobes.c +3 -3
@@ -350,7 +350,7 @@ int __kprobes kprobe_fault_handler(struct pt_regs *regs, unsigned int fsr)return 0 ; } -int __kprobes +int noinstrkprobe_brk_handler (struct pt_regs *regs,unsigned long esr) {struct kprobe *p, *cur_kprobe;
@@ -394,7 +394,7 @@ kprobe_brk_handler(struct pt_regs *regs, unsigned long esr)return DBG_HOOK_HANDLED ; } -int __kprobes +int noinstrkprobe_ss_brk_handler (struct pt_regs *regs,unsigned long esr) {struct kprobe_ctlblk *kcb =get_kprobe_ctlblk ();
@@ -413,7 +413,7 @@ kprobe_ss_brk_handler(struct pt_regs *regs, unsigned long esr)return DBG_HOOK_ERROR ; } -int __kprobes +int noinstrkretprobe_brk_handler (struct pt_regs *regs,unsigned long esr) {if (regs->pc != (unsigned long )__kretprobe_trampoline)
Signature · 1 line
2.47.3
Re: [RFC PATCH v2 2/4] arm64/debug: Make the Kprobe functions noinstr
Quoted 6 lines · On Thu, 13 Aug 2026 06:49:42 +0000 Hongyan Xia <[email protected]>…
From: Hongyan Xia <[email protected]>
The Kprobe debug exception path must be run with extra care. NOKPROBE isn't sufficient, as other instrumentation like ftrace still opens a can of worms that is very complex to deal with.
Quoted 9 lines
Mark the three main Kprobe entry points noinstr, as well as the debug exception paths that lead to and exit from these entry points. Note that noinstr attribute is stronger and can safely replace NOKPROBE and
__kprobemodifiers.Also mark
esr_brk_comment()__always_inline, as clang does not reliably inline plain static inline functions into noinline (noinstr) callers such ascall_el1_break_hook().
Looks good to me.
Thank you,
Quoted 111 lines
Signed-off-by Hongyan Xia <[email protected]>--- arch/arm64/include/asm/esr.h | 2 +- arch/arm64/include/asm/kprobes.h | 9 +++------ arch/arm64/kernel/debug-monitors.c | 6 ++---- arch/arm64/kernel/probes/kprobes.c | 6 +++--- 4 files changed, 9 insertions(+), 14 deletions(-)arch/arm64/include/asm/esr.h +1
-1@@ -437,7 +437,7 @@#ifndef __ASSEMBLER__#include <asm/types.h> -static inline unsigned long esr_brk_comment (unsigned long esr) +static __always_inline unsigned long esr_brk_comment (unsigned long esr) {return esr & ESR_ELx_BRK64_ISS_COMMENT_MASK; }arch/arm64/include/asm/kprobes.h +3
-6@@ -48,11 +48,8 @@ void __kprobes *trampoline_probe_handler(struct pt_regs *regs);#endif /* CONFIG_KPROBES */ -int __kprobes kprobe_brk_handler (struct pt_regs *regs, -unsigned long esr); -int __kprobes kprobe_ss_brk_handler (struct pt_regs *regs, -unsigned long esr); -int __kprobes kretprobe_brk_handler (struct pt_regs *regs, -unsigned long esr); +int noinstr kprobe_brk_handler(struct pt_regs *regs, unsigned long esr); +int noinstr kprobe_ss_brk_handler(struct pt_regs *regs, unsigned long esr); +int noinstr kretprobe_brk_handler(struct pt_regs *regs, unsigned long esr);#endif /* _ARM_KPROBES_H */ arch/arm64/kernel/debug-monitors.c +2
-4@@ -207,7 +207,7 @@ void do_el1_softstep(unsigned long esr, struct pt_regs *regs) }NOKPROBE_SYMBOL (do_el1_softstep); -static int call_el1_break_hook (struct pt_regs *regs,unsigned long esr) +static int noinstr call_el1_break_hook(struct pt_regs *regs, unsigned long esr) {if (esr_brk_comment (esr) ==BUG_BRK_IMM )return bug_brk_handler (regs, esr);@@ -245,7 +245,6 @@ static int call_el1_break_hook(struct pt_regs *regs, unsigned long esr)return DBG_HOOK_ERROR ; } -NOKPROBE_SYMBOL (call_el1_break_hook);/* * We have already unmasked interrupts and enabled preemption @@ -261,14 +260,13 @@ void do_el0_brk64(unsigned long esr, struct pt_regs *regs)send_user_sigtrap (TRAP_BRKPT ); } -void do_el1_brk64 (unsigned long esr,struct pt_regs *regs) +void noinstr do_el1_brk64(unsigned long esr, struct pt_regs *regs) {if (call_el1_break_hook (regs, esr) ==DBG_HOOK_HANDLED )return ;die ("Oops - BRK" , regs, esr); } -NOKPROBE_SYMBOL (do_el1_brk64);#ifdef CONFIG_COMPAT void do_bkpt32 (unsigned long esr,struct pt_regs *regs)arch/arm64/kernel/probes/kprobes.c +3
-3@@ -350,7 +350,7 @@ int __kprobes kprobe_fault_handler(struct pt_regs *regs, unsigned int fsr)return 0 ; } -int __kprobes +int noinstrkprobe_brk_handler (struct pt_regs *regs,unsigned long esr) {struct kprobe *p, *cur_kprobe;@@ -394,7 +394,7 @@ kprobe_brk_handler(struct pt_regs *regs, unsigned long esr)return DBG_HOOK_HANDLED ; } -int __kprobes +int noinstrkprobe_ss_brk_handler (struct pt_regs *regs,unsigned long esr) {struct kprobe_ctlblk *kcb =get_kprobe_ctlblk ();@@ -413,7 +413,7 @@ kprobe_ss_brk_handler(struct pt_regs *regs, unsigned long esr)return DBG_HOOK_ERROR ; } -int __kprobes +int noinstrkretprobe_brk_handler (struct pt_regs *regs,unsigned long esr) {if (regs->pc != (unsigned long )__kretprobe_trampoline)Signature · 1 line
2.47.3
Signature · 1 line
Masami Hiramatsu (Google) <[email protected]>
RFC PATCH v2 3/4 arm64/kprobes: Make the entire Kprobe noinstr
In reply to Hongyan XiaFrom: Hongyan Xia <[email protected]>
Convert all the sub-functions called by kprobe_brk_handler, kprobe_ss_brk_handler and kretprobe_brk_handler noinstr. After this commit, the entire Kprobe path (anything between Kprobe debug_exception entry and exit) is now noinstr.
One big item that is missing is the instruction simulation path. We leave it as future work as the GCS handling needs extra care.
The __kprobes attribute (notrace + .kprobes.text) is replaced by noinstr, which is a strict superset for these functions.
Also mark the leaf helpers used inside these functions __always_inline, as clang does not reliably inline plain static inline functions into noinline (noinstr) callers.
--- arch/arm64/include/asm/percpu.h | 2 +- arch/arm64/include/asm/ptrace.h | 4 ++-- arch/arm64/kernel/probes/kprobes.c | 30 +++++++++++++++--------------- include/linux/kprobes.h | 8 ++++---- 4 files changed, 22 insertions(+), 22 deletions(-)
arch/arm64/include/asm/percpu.h +1 -1
@@ -29,7 +29,7 @@ static inline unsigned long __hyp_my_cpu_offset(void)return read_sysreg (tpidr_el2); } -static inline unsigned long __kern_my_cpu_offset (void ) +static __always_inline unsigned long __kern_my_cpu_offset (void ) {unsigned long off;
arch/arm64/include/asm/ptrace.h +2 -2
@@ -336,11 +336,11 @@ static inline unsigned long regs_get_kernel_argument(struct pt_regs *regs,struct task_struct ;int valid_user_regs (struct user_pt_regs *regs,struct task_struct *task); -static inline unsigned long instruction_pointer (struct pt_regs *regs) +static __always_inline unsigned long instruction_pointer (struct pt_regs *regs) {return regs->pc; } -static inline void instruction_pointer_set (struct pt_regs *regs, +static __always_inline void instruction_pointer_set(struct pt_regs *regs,unsigned long val) { regs->pc = val;
arch/arm64/kernel/probes/kprobes.c +15 -15
@@ -39,7 +39,7 @@ DEFINE_PER_CPU(struct kprobe *, current_kprobe) = NULL; DEFINE_PER_CPU(struct kprobe_ctlblk, kprobe_ctlblk); -static void __kprobes +static void noinstrpost_kprobe_handler (struct kprobe *,struct kprobe_ctlblk *,struct pt_regs *);void *alloc_insn_page (void )
@@ -170,7 +170,7 @@ void __kprobes arch_remove_kprobe(struct kprobe *p) } } -static void __kprobes save_previous_kprobe (struct kprobe_ctlblk *kcb) +static void noinstr save_previous_kprobe(struct kprobe_ctlblk *kcb) { kcb->prev_kprobe.kp =kprobe_running (); kcb->prev_kprobe.status = kcb->kprobe_status;
@@ -184,7 +184,7 @@ static void __kprobes save_previous_kprobe(struct kprobe_ctlblk *kcb) kcb->prev_kprobe.saved_irqflag = kcb->saved_irqflag; } -static void __kprobes restore_previous_kprobe (struct kprobe_ctlblk *kcb) +static void noinstr restore_previous_kprobe(struct kprobe_ctlblk *kcb) {__this_cpu_write (current_kprobe, kcb->prev_kprobe.kp); kcb->kprobe_status = kcb->prev_kprobe.status;
@@ -197,7 +197,7 @@ static void __kprobes restore_previous_kprobe(struct kprobe_ctlblk *kcb) kcb->saved_irqflag = kcb->prev_kprobe.saved_irqflag; } -static void __kprobes set_current_kprobe (struct kprobe *p) +static void noinstr set_current_kprobe(struct kprobe *p) {__this_cpu_write (current_kprobe, p); }
@@ -207,23 +207,23 @@ static void __kprobes set_current_kprobe(struct kprobe *p)* simple and avoid nesting exceptions. Interrupts do have to be disabled since * the kprobe state is per-CPU and doesn't get migrated. */ -static void __kprobes kprobes_save_local_irqflag (struct kprobe_ctlblk *kcb, -struct pt_regs *regs) +static void noinstr kprobes_save_local_irqflag(struct kprobe_ctlblk *kcb, +struct pt_regs *regs) { kcb->saved_irqflag = regs->pstate &DAIF_MASK ; regs->pstate |=DAIF_MASK ; } -static void __kprobes kprobes_restore_local_irqflag (struct kprobe_ctlblk *kcb, -struct pt_regs *regs) +static void noinstr kprobes_restore_local_irqflag(struct kprobe_ctlblk *kcb, +struct pt_regs *regs) { regs->pstate &= ~DAIF_MASK ; regs->pstate |= kcb->saved_irqflag; } -static void __kprobes setup_singlestep (struct kprobe *p, -struct pt_regs *regs, -struct kprobe_ctlblk *kcb,int reenter) +static void noinstr setup_singlestep(struct kprobe *p, +struct pt_regs *regs, +struct kprobe_ctlblk *kcb,int reenter) {unsigned long slot;
@@ -248,9 +248,9 @@ static void __kprobes setup_singlestep(struct kprobe *p, } } -static int __kprobes reenter_kprobe (struct kprobe *p, -struct pt_regs *regs, -struct kprobe_ctlblk *kcb) +static int noinstr reenter_kprobe(struct kprobe *p, +struct pt_regs *regs, +struct kprobe_ctlblk *kcb) {switch (kcb->kprobe_status) {case KPROBE_HIT_SSDONE :
@@ -278,7 +278,7 @@ static int __kprobes reenter_kprobe(struct kprobe *p,return 1 ; } -static void __kprobes +static void noinstrpost_kprobe_handler (struct kprobe *cur,struct kprobe_ctlblk *kcb,struct pt_regs *regs) {/* return addr restore if non-branching insn */
include/linux/kprobes.h +4 -4
@@ -384,17 +384,17 @@ static inline void kprobe_ftrace_kill(void) {}struct kprobe *get_kprobe (void *addr);/* kprobe_running() will just return the current_kprobe on this CPU */ -static inline struct kprobe *kprobe_running (void ) +static __always_inline struct kprobe *kprobe_running(void) {return __this_cpu_read (current_kprobe); } -static inline void reset_current_kprobe (void ) +static __always_inline void reset_current_kprobe(void) {__this_cpu_write (current_kprobe,NULL ); } -static inline struct kprobe_ctlblk *get_kprobe_ctlblk (void ) +static __always_inline struct kprobe_ctlblk *get_kprobe_ctlblk(void) {return this_cpu_ptr (&kprobe_ctlblk); }
@@ -449,7 +449,7 @@ static inline struct kprobe *get_kprobe(void *addr) {return NULL ; } -static inline struct kprobe *kprobe_running (void ) +static __always_inline struct kprobe *kprobe_running(void) {return NULL ; }
Signature · 1 line
2.47.3
[email protected]
AI reviewRe: [RFC PATCH v2 3/4] arm64/kprobes: Make the entire Kprobe noinstr
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] Functions converted to `noinstr` directly call non-noinstr and instrumentable code without using…
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] Functions converted to noinstr directly call non-noinstr and instrumentable code without using the required instrumentation_begin() / instrumentation_end() wrappers, violating strict non-instrumentation boundaries. --
--- Patch [3]: [RFC PATCH v2 3/4] arm64/kprobes: Make the entire Kprobe noinstr --- commit 3ef2956d1a4e614b73b64bbc64be2a2e20e577e4
Author: Hongyan Xia <[email protected]>
arm64/kprobes: Make the entire Kprobe noinstr
This commit converts the kprobe handling path to use noinstr instead of __kprobes, ensuring the code between debug_exception entry and exit is strictly non-instrumentable. It also marks leaf helpers as __always_inline and explicitly leaves the instruction simulation path as future work.
Quoted 4 lines
arch/arm64/kernel/probes/kprobes.c +0
-0
[ ... ]
Quoted 7 lines
arch/arm64/kernel/probes/kprobes.c +3
-3-static void __kprobes setup_singlestep (struct kprobe *p, -struct pt_regs *regs, -struct kprobe_ctlblk *kcb,int reenter) +static void noinstr setup_singlestep(struct kprobe *p, +struct pt_regs *regs, +struct kprobe_ctlblk *kcb,int reenter) {
[ ... ]
arch/arm64/kernel/probes/kprobes.c
Quoted 9 lines
arch/arm64/kernel/probes/kprobes.c +0
-0if (p->ainsn.xol_insn) {/* prepare for single stepping */ slot = (unsigned long )p->ainsn.xol_insn;kprobes_save_local_irqflag (kcb, regs);instruction_pointer_set (regs, slot); }else {/* insn simulation */ arch_simulate_insn (p, regs);
[Severity: Medium]
Since setup_singlestep() is now marked noinstr, is it safe to call arch_simulate_insn() here without instrumentation_begin() and instrumentation_end()? The commit message notes the simulation path isn't noinstr yet, so calling it directly from a noinstr function could trigger objtool validation failures and issues with KASAN/KCOV.
Quoted 14 lines
arch/arm64/kernel/probes/kprobes.c +3
-3} } -static int __kprobes reenter_kprobe (struct kprobe *p, -struct pt_regs *regs, -struct kprobe_ctlblk *kcb) +static int noinstr reenter_kprobe(struct kprobe *p, +struct pt_regs *regs, +struct kprobe_ctlblk *kcb) {switch (kcb->kprobe_status) {case KPROBE_HIT_SSDONE :case KPROBE_HIT_ACTIVE :kprobes_inc_nmissed_count (p);
arch/arm64/kernel/probes/kprobes.c static void __kprobes setup_singlestep(struct kprobe *p,
Quoted 10 lines
arch/arm64/kernel/probes/kprobes.c +0
-0setup_singlestep (p, regs, kcb,1 );break ;case KPROBE_HIT_SS :case KPROBE_REENTER :pr_warn ("Failed to recover from reentered kprobes.\n" );dump_kprobe (p);BUG ();break ;default :WARN_ON (1 );
[Severity: Medium]
Similarly, reenter_kprobe() is now noinstr but makes calls to instrumentable functions like pr_warn(), dump_kprobe(), BUG(), and WARN_ON(). Should these be wrapped in instrumentation_begin() / instrumentation_end() to prevent instrumentation from running in an un-instrumentable context?
Quoted 13 lines
arch/arm64/kernel/probes/kprobes.c +1
-1return 0 ; }return 1 ; } -static void __kprobes +static void noinstrpost_kprobe_handler (struct kprobe *cur,struct kprobe_ctlblk *kcb,struct pt_regs *regs) {/* return addr restore if non-branching insn */ if (cur->ainsn.xol_restore !=0 )instruction_pointer_set (regs, cur->ainsn.xol_restore);
arch/arm64/kernel/probes/kprobes.c
Quoted 10 lines
arch/arm64/kernel/probes/kprobes.c +0
-0/* restore back original saved kprobe variables and continue */ if (kcb->kprobe_status ==KPROBE_REENTER ) {restore_previous_kprobe (kcb);return ; }/* call post handler */ kcb->kprobe_status =KPROBE_HIT_SSDONE ;if (cur->post_handler) cur->post_handler (cur, regs,0 );
[Severity: Medium]
Does the call to cur->post_handler() also need to be wrapped in instrumentation_begin() / instrumentation_end()? User-supplied post handlers are instrumentable, so executing them directly from a noinstr context might violate strict non-instrumentation boundaries if the probe triggers in early exception entry.
Quoted 3 lines
reset_current_kprobe();
}
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
RFC PATCH v2 4/4 Revert "arm64: kprobes: Allow reentering kprobes while single-stepping"
In reply to Hongyan XiaFrom: Hongyan Xia <[email protected]>
With the debug exception path noinstr from the vectors down to the kprobe handlers, any Kprobe reentry under KPROBE_HIT_SS must be considered a bug now. The special case is no longer needed.
This reverts commit 23f851ac0078a908bf3422d6467ebc1db5828c46.
--- arch/arm64/include/asm/kprobes.h | 6 ------ arch/arm64/kernel/probes/kprobes.c | 23 +---------------------- 2 files changed, 1 insertion(+), 28 deletions(-)
arch/arm64/include/asm/kprobes.h +0 -6
@@ -26,12 +26,6 @@struct prev_kprobe {struct kprobe *kp;unsigned int status; - -/* -* The original DAIF state of the outer kprobe, saved here before -* a nested kprobe overwrites kcb->saved_irqflag during reentry. -*/ -unsigned long saved_irqflag; };/* per-cpu kprobe control block */
arch/arm64/kernel/probes/kprobes.c +1 -22
@@ -174,27 +174,12 @@ static void noinstr save_previous_kprobe(struct kprobe_ctlblk *kcb) { kcb->prev_kprobe.kp =kprobe_running (); kcb->prev_kprobe.status = kcb->kprobe_status; - -/* -* Save the outer kprobe's original DAIF flags before the nested -* kprobe calls kprobes_save_local_irqflag() and overwrites -* kcb->saved_irqflag. Without this, the outer kprobe will restore -* the wrong DAIF state and leave interrupts permanently masked. -*/ - kcb->prev_kprobe.saved_irqflag = kcb->saved_irqflag; } static void noinstr restore_previous_kprobe(struct kprobe_ctlblk *kcb) {__this_cpu_write (current_kprobe, kcb->prev_kprobe.kp); kcb->kprobe_status = kcb->prev_kprobe.status; - -/* -* Restore the outer kprobe's saved_irqflag so that when its -* single-step completes, kprobes_restore_local_irqflag() uses -* the correct original DAIF value. -*/ - kcb->saved_irqflag = kcb->prev_kprobe.saved_irqflag; } static void noinstr set_current_kprobe(struct kprobe *p)
@@ -255,16 +240,10 @@ static int noinstr reenter_kprobe(struct kprobe *p,switch (kcb->kprobe_status) {case KPROBE_HIT_SSDONE :case KPROBE_HIT_ACTIVE : -case KPROBE_HIT_SS : -/* -* A probe can be hit while another kprobe is preparing or -* executing its XOL single-step instruction. This is still a -* recoverable one-level reentry, so handle it in the same way as -* reentry from KPROBE_HIT_ACTIVE or KPROBE_HIT_SSDONE. -*/ kprobes_inc_nmissed_count (p);setup_singlestep (p, regs, kcb,1 );break ; +case KPROBE_HIT_SS :case KPROBE_REENTER :pr_warn ("Failed to recover from reentered kprobes.\n" );dump_kprobe (p);
Signature · 1 line
2.47.3
Will Deacon
maintainer[+Ada]
Quoted 39 lines · Hongyan Xia
From: Hongyan Xia <[email protected]>
Commit
879a6754d3d1("arm64: kprobes: Only handle faults originating from XOL slot") and23f851ac0078("arm64: kprobes: Allow reentering kprobes while single-stepping") band-aided two corner cases in the arm64 kprobe path: Nested page faults inside Kprobe and nested Kprobe SS handling. Although the two patches were taken, Will asked whether we should keep patching things like this each time we find a corner case, and whether making Kprobe noinstr is the correct answer [1].This series attempts to do exactly that: Making the Kprobe path noinstr. The kprobe handlers become noinstr end to end. Apart from marking functions noinstr, we also force-inline some static inline helper functions because certain compilers still outline them, breaking noinstr.
Note that arm64 has no objtool so at the moment noinstr is more of a contract that must be followed, instead of actually enforcing control flow. Instrumentation can still happen if someone with root privileges does something stupid, but now it is regarded as a bug rather than a legitimate use case that the kernel needs to cover.
[1] https://lore.kernel.org/all/alpuL10h7-OK2hFb@willie-the-truck/
Changed in v2:
- Remove instrumentation_{begin/end}() markers as they do nothing
- Fold__always_inlineinto the patches that actually need them
- Remove the debug exception path changes unrelated to Kprobe. This series is only about Kprobe for now
- Drop thekprobe_page_fault()patch for now. It's a question whether we even need it at all. Will send a patch for it separately.Hongyan Xia (4):
arm64/entry: Make debug_exception_enter/exit() noinstr arm64/debug: Make the Kprobe functions noinstr arm64/kprobes: Make the entire Kprobe noinstr Revert "arm64: kprobes: Allow reentering kprobes while
single-stepping"
Quoted 9 lines
arch/arm64/include/asm/esr.h | 2 +- arch/arm64/include/asm/kprobes.h | 15 ++------ arch/arm64/include/asm/percpu.h | 2 +- arch/arm64/include/asm/preempt.h | 4 +- arch/arm64/include/asm/ptrace.h | 4 +- arch/arm64/kernel/debug-monitors.c | 6 +-- arch/arm64/kernel/entry-common.c | 13 ++----- arch/arm64/kernel/probes/kprobes.c | 59 ++++++++++-------------------- include/linux/kprobes.h | 8 ++--
Given the lack of objtool enforcement (as you note above), I wonder whether it would be more robust to do this on a per-compilation unit basis? i.e. disable instrumentation for debug-monitors.c, entry-common.c and kprobes.c?
Unfortunately, I couldn't find a way to do that in the Makefile :/ In which case, perhaps this series is the best we can do, but I thought I'd ask the question in case somebody has any bright ideas.
Will
Quoted 1 line · Will Deacon
[+Ada]
Thanks for bringing this to my attention !
Quoted 17 lines · Hongyan Xia
Quoted 16 lines
From: Hongyan Xia <[email protected]>
Commit
879a6754d3d1("arm64: kprobes: Only handle faults originating from XOL slot") and23f851ac0078("arm64: kprobes: Allow reentering kprobes while single-stepping") band-aided two corner cases in the arm64 kprobe path: Nested page faults inside Kprobe and nested Kprobe SS handling. Although the two patches were taken, Will asked whether we should keep patching things like this each time we find a corner case, and whether making Kprobe noinstr is the correct answer [1].This series attempts to do exactly that: Making the Kprobe path noinstr. The kprobe handlers become noinstr end to end. Apart from marking functions noinstr, we also force-inline some static inline helper functions because certain compilers still outline them, breaking noinstr.
Quoted 5 lines
Quoted 5 lines
Note that arm64 has no objtool so at the moment noinstr is more of a contract that must be followed, instead of actually enforcing control flow. Instrumentation can still happen if someone with root privileges does something stupid, but now it is regarded as a bug rather than a legitimate use case that the kernel needs to cover.
Just to make sure I am not missing something : as far as I can tell, Kprobe checks the section the instruction to be instrumented is in and will refuse to instrument anything within .noinstr.text.[0] Given that this series converts the whole call trees of the Kprobe functions to noinstr, should that not make it safe ?
(My understanding is that Kprobe is the only one checking the noinstr section, all the other instrumentations and sanitizers do not.)
Quoted 26 lines
Quoted 17 lines
[1] https://lore.kernel.org/all/alpuL10h7-OK2hFb@willie-the-truck/
Changed in v2:
- Remove instrumentation_{begin/end}() markers as they do nothing
- Fold__always_inlineinto the patches that actually need them
- Remove the debug exception path changes unrelated to Kprobe. This series is only about Kprobe for now
- Drop thekprobe_page_fault()patch for now. It's a question whether we even need it at all. Will send a patch for it separately.Hongyan Xia (4):
arm64/entry: Make debug_exception_enter/exit() noinstr arm64/debug: Make the Kprobe functions noinstr arm64/kprobes: Make the entire Kprobe noinstr Revert "arm64: kprobes: Allow reentering kprobes while
single-stepping"Quoted 9 lines
arch/arm64/include/asm/esr.h | 2 +- arch/arm64/include/asm/kprobes.h | 15 ++------ arch/arm64/include/asm/percpu.h | 2 +- arch/arm64/include/asm/preempt.h | 4 +- arch/arm64/include/asm/ptrace.h | 4 +- arch/arm64/kernel/debug-monitors.c | 6 +-- arch/arm64/kernel/entry-common.c | 13 ++----- arch/arm64/kernel/probes/kprobes.c | 59 ++++++++++-------------------- include/linux/kprobes.h | 8 ++--
Quoted 4 lines
Given the lack of objtool enforcement (as you note above), I wonder whether it would be more robust to do this on a per-compilation unit basis? i.e. disable instrumentation for
debug-monitors.c,entry-common.candkprobes.c?
I think that marking whole compilations units as noinstr would be great too, though I am not sure how it would affect some other sanitizers/instrumentation.
Quoted 5 lines
Unfortunately, I couldn't find a way to do that in the
Makefile:/ In which case, perhaps this series is the best we can do, but I thought I'd ask the question in case somebody has any bright ideas.Will
I found out the same thing when I initially started working on making sure alternative-callbacks where noinstr/patch safe, so this pushed me to dig a bit further.
There are two parts to this, the first one I believe to be relatively simple :
GCOV, KCOV, KASAN, KCSAN, KMSAN, UBSAN and ftrace all have a way to be turned off for specific compilation units in the Makefiles. We already use them in some places : KCOV_INSTRUMENT_entry-common.o := n.
We could introduce a similar variable to disable all instrumentations, re-using what is already in place : something like NO_INSTRUMENT_entry-common.o := 1, which would be checked before all the other control variables in scripts/Makefile.lib and override them.
The second part is Kprobe itself, which would be much more painful to disable as it does not rely on compiler instrumentation or other build options we could use.
However, Kprobe /does/ check at runtime if it is allowed to probe a location, refusing to do so if it is in .noinstr.text or if the function name is part of its explicit deny list.
So, looking at the shenanigans KVM does for the nVHE hypervisor symbols[1], I think it could be possible to either :
1. move the .text section into .noinstr.text, or
2. add all symbols to the _kprobe_blacklist section, as if adding NOKPROBE_SYMBOL() to all functions within the compilation unit.
I played around by hand with the first idea using objcopy and was able to combine the .text section into an existing .noinstr.text section, preserving both, but that did not update the relocations, so the kernel immediately crashed.
This leads me to think that adding the symbols to the _kprobe_blacklist section with a LD script might be the more straightforward solution.
I have no experience with LD scripts but I would be happy to have a go, if that sounds like something that could work *and* worth the effort. Otherwise I think this series might be the best we can do !
Kind regards,
Ada
[0] : https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/kernel/kprobes.c?id=6628e7d4c6b25c8d18f1345e3a20d21829aae7e4#n2840
[1] : https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/arm64/kvm/hyp/nvhe/Makefile?id=6628e7d4c6b25c8d18f1345e3a20d21829aae7e4#n38
Quoted 27 lines · Ada Couprie Diaz
Quoted 1 line · Will Deacon
[+Ada]
Thanks for bringing this to my attention !
Quoted 17 lines · Hongyan Xia
Quoted 16 lines
From: Hongyan Xia <[email protected]>
Commit
879a6754d3d1("arm64: kprobes: Only handle faults originating from XOL slot") and23f851ac0078("arm64: kprobes: Allow reentering kprobes while single-stepping") band-aided two corner cases in the arm64 kprobe path: Nested page faults inside Kprobe and nested Kprobe SS handling. Although the two patches were taken, Will asked whether we should keep patching things like this each time we find a corner case, and whether making Kprobe noinstr is the correct answer [1].This series attempts to do exactly that: Making the Kprobe path noinstr. The kprobe handlers become noinstr end to end. Apart from marking functions noinstr, we also force-inline some static inline helper functions because certain compilers still outline them, breaking noinstr.
Quoted 5 lines
Quoted 5 lines
Note that arm64 has no objtool so at the moment noinstr is more of a contract that must be followed, instead of actually enforcing control flow. Instrumentation can still happen if someone with root privileges does something stupid, but now it is regarded as a bug rather than a legitimate use case that the kernel needs to cover.
Quoted 3 lines
Just to make sure I am not missing something : as far as I can tell, Kprobe checks the section the instruction to be instrumented is in and will refuse to instrument anything within
.noinstr.text.[0]
Yes, exactly.
Quoted 4 lines
Given that this series converts the whole call trees of the Kprobe functions to
noinstr, should that not make it safe ?
(My understanding is that Kprobe is the only one checking the noinstr section, all the other instrumentations and sanitizers do not.)
This is the sad part. Yes, it refuses to attach to noinstr sections, but noinstr sections could just call normal .text functions which can be instrumented, so we are still not safe.
This is exactly the point of objtool, statically validating that the control flow simply cannot escape .noinstr sections.
Quoted 29 lines
Quoted 29 lines
Quoted 29 lines
[1] https://apc01.safelinks.protection.outlook.com/?
url=https%3A%2F%2Flore.kernel.org%2Fall%2FalpuL10h7-OK2hFb%40willie-
the- truck%2F&data=05%7C02%7Chongyan.xia%40transsion.com%7C964feb00a6854510ed3008df39c11576%7C2e8503a62d0143338e366ab7c8cd7ae2%7C0%7C0%7C639293173603733128%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=co2FrgXxUfoT3bmrblgDn4G4LHhNGfXGSuuNBGrTYF4%3D&reserved=0Changed in v2:
- Remove instrumentation_{begin/end}() markers as they do nothing
- Fold__always_inlineinto the patches that actually need them
- Remove the debug exception path changes unrelated to Kprobe. This series is only about Kprobe for now
- Drop thekprobe_page_fault()patch for now. It's a question whether we even need it at all. Will send a patch for it separately.Hongyan Xia (4):
arm64/entry: Make debug_exception_enter/exit() noinstr arm64/debug: Make the Kprobe functions noinstr arm64/kprobes: Make the entire Kprobe noinstr Revert "arm64: kprobes: Allow reentering kprobes while single-stepping"arch/arm64/include/asm/esr.h | 2 +- arch/arm64/include/asm/kprobes.h | 15 ++------ arch/arm64/include/asm/percpu.h | 2 +- arch/arm64/include/asm/preempt.h | 4 +- arch/arm64/include/asm/ptrace.h | 4 +-
arch/arm64/kernel/debug-monitors.c| 6 +-- arch/arm64/kernel/entry-common.c | 13 ++-----arch/arm64/kernel/probes/kprobes.c| 59 ++++++++++-------------------- include/linux/kprobes.h | 8 ++--
Quoted 7 lines
Quoted 4 lines
Given the lack of objtool enforcement (as you note above), I wonder whether it would be more robust to do this on a per-compilation unit basis? i.e. disable instrumentation for
debug-monitors.c,entry-common.candkprobes.c?I think that marking whole compilations units as
noinstrwould be great too,
That would be cool, indeed.
Quoted 40 lines
though I am not sure how it would affect some other sanitizers/ instrumentation.
Quoted 5 lines
Unfortunately, I couldn't find a way to do that in the
Makefile:/ In which case, perhaps this series is the best we can do, but I thought I'd ask the question in case somebody has any bright ideas.Will
I found out the same thing when I initially started working on making sure alternative-callbacks where
noinstr/patch safe, so this pushed me to dig a bit further.There are two parts to this, the first one I believe to be relatively simple :
GCOV, KCOV, KASAN, KCSAN, KMSAN, UBSAN and ftrace all have a way to be turned off for specific compilation units in the Makefiles. We already use them in some places :KCOV_INSTRUMENT_entry-common.o := n.We could introduce a similar variable to disable all instrumentations, re-using what is already in place : something like
NO_INSTRUMENT_entry-common.o := 1, which would be checked before all the other control variables inscripts/Makefile.liband override them.The second part is Kprobe itself, which would be much more painful to disable as it does not rely on compiler instrumentation or other build options we could use.
However, Kprobe /does/ check at runtime if it is allowed to probe a location, refusing to do so if it is in.noinstr.textor if the function
name is part of its explicit deny list.So, looking at the shenanigans KVM does for the nVHE hypervisor symbols[1], I think it could be possible to either :
1. move the.textsection into.noinstr.text, or
2. add all symbols to the_kprobe_blacklistsection, as if addingNOKPROBE_SYMBOL()to all functions within the compilation unit.
Quoted 6 lines
I played around by hand with the first idea using
objcopyand was able to combine the.textsection into an existing.noinstr.textsection, preserving both, but that did not update the relocations, so the kernel immediately crashed.
This leads me to think that adding the symbols to the_kprobe_blacklistsection with a LD script might be the more straightforward solution.
Apart from linker wizardry, the whole noinstr machinery should still allow small code snippets to be instrumentable, namely instrumentation_{begin/end}() regions. If the whole-compile-unit-noinstr implementation doesn't allow it, it might face resistance upstream.
Quoted 11 lines
I have no experience with LD scripts but I would be happy to have a go, if that sounds like something that could work *and* worth the effort. Otherwise I think this series might be the best we can do !
Kind regards,
Ada[0] : https://apc01.safelinks.protection.outlook.com/?
url=https%3A%2F%2Fgit.kernel.org%2Fpub%2Fscm%2Flinux%2Fkernel%2Fgit%2Ftorvalds%2Flinux.git%2Ftree%2Fkernel%2Fkprobes.c%3Fid%3D6628e7d4c6b25c8d18f1345e3a20d21829aae7e4%23n2840&data=05%7C02%7Chongyan.xia%40transsion.com%7C964feb00a6854510ed3008df39c11576%7C2e8503a62d0143338e366ab7c8cd7ae2%7C0%7C0%7C639293173603788749%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=kE%2Fy%2FzHhDV42qutiIkFqI4emK4vQ1Y2bWf0rQ%2FYYR44%3D&reserved=0
[1] : https://apc01.safelinks.protection.outlook.com/?
url=https%3A%2F%2Fgit.kernel.org%2Fpub%2Fscm%2Flinux%2Fkernel%2Fgit%2Ftorvalds%2Flinux.git%2Ftree%2Farch%2Farm64%2Fkvm%2Fhyp%2Fnvhe%2FMakefile%3Fid%3D6628e7d4c6b25c8d18f1345e3a20d21829aae7e4%23n38&data=05%7C02%7Chongyan.xia%40transsion.com%7C964feb00a6854510ed3008df39c11576%7C2e8503a62d0143338e366ab7c8cd7ae2%7C0%7C0%7C639293173603834481%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=Ocq80oowgwqVrxjfWJWmQZAkk6y%2BWNnY%2B2qgavLhJcQ%3D&reserved=0