arm64: kprobes: Fix some bugs in arm64 kprobes
v2 cover arm64: kprobes: Fix some bugs in arm64 kprobes
Hi,
Here are the v2 patches which fixes kprobe bugs on arm64.
Naresh reported that recently ftracetest crashes kernel, and I found there are 3 different bugs around the crash. In v1 thread, we found one another bug of RCU and debug exception.
- Kprobes on arm64 doesn't recover pstate.D mask after single stepping. This causes a real kernel crash if a kprobe is unexpectedly nested.
- Some symbols which are called from blacklisted function, are not blacklisted.
- Debug exception is not visible to RCU, thus rcu_read_lock() cause a warning inside it.
- Debug exception handlers on arm64 is using rcu_read_lock(), but that is not needed because interrupts are disabled.
This series includes fixes for above bugs.
Thank you,
---
Masami Hiramatsu (4):
arm64: kprobes: Recover pstate.D in single-step exception handler
arm64: unwind: Prohibit probing on return_address()
arm64: Make debug exception handlers visible from RCU
arm64: Remove unneeded rcu_read_lock from debug handlers
arch/arm64/kernel/debug-monitors.c | 14 +++++++----- arch/arm64/kernel/probes/kprobes.c | 41 ++++++------------------------------ arch/arm64/kernel/return_address.c | 4 +++- arch/arm64/kernel/stacktrace.c | 3 +++ arch/arm64/mm/fault.c | 40 +++++++++++++++++++++++++++++++++++ 5 files changed, 61 insertions(+), 41 deletions(-)
--
Masami Hiramatsu (Linaro) <[email protected]>
PATCH v2 1/4 arm64: kprobes: Recover pstate.D in single-step exception handler
On arm64, if a nested kprobes hit, it can crash the kernel with below error message.
[ 152.118921] Unexpected kernel single-step exception at EL1
This is because commit 7419333fa15e ("arm64: kprobe: Always clear pstate.D in breakpoint exception handler") unmask pstate.D for doing single step but does not recover it after single step in the nested kprobes. That is correct *unless* any nested kprobes (single-stepping) runs inside other kprobes user handler.
When the 1st kprobe hits, do_debug_exception() will be called. At this point, debug exception (= pstate.D) must be masked (=1). When the 2nd
(nested) kprobe is hit before single-step of the first kprobe, it
unmask debug exception (pstate.D = 0) and return.
Then, when the 1st kprobe setting up single-step, it saves current DAIF, mask DAIF, enable single-step, and restore DAIF. However, since "D" flag in DAIF is cleared by the 2nd kprobe, the single-step exception happens soon after restoring DAIF.
To solve this issue, this stores all DAIF bits and restore it after single stepping.
---
Changes in v2:
- Save and restore all DAIF flags.
- Operate pstate directly and remove spsr_set_debug_flag().
--- arch/arm64/kernel/probes/kprobes.c | 41 ++++++------------------------------ 1 file changed, 7 insertions(+), 34 deletions(-)
arch/arm64/kernel/probes/kprobes.c +7 -34
@@ -29,6 +29,8 @@#include "decode-insn.h" +#define PSR_DAIF_MASK (PSR_D_BIT | PSR_A_BIT | PSR_I_BIT | PSR_F_BIT) + DEFINE_PER_CPU(struct kprobe *, current_kprobe) = NULL; DEFINE_PER_CPU(struct kprobe_ctlblk, kprobe_ctlblk);
@@ -167,33 +169,6 @@ static void __kprobes set_current_kprobe(struct kprobe *p)__this_cpu_write (current_kprobe, p); } -/* -* When PSTATE.D is set (masked), then software step exceptions can not be -* generated. -* SPSR's D bit shows the value of PSTATE.D immediately before the -* exception was taken. PSTATE.D is set while entering into any exception -* mode, however software clears it for any normal (none-debug-exception) -* mode in the exception entry. Therefore, when we are entering into kprobe -* breakpoint handler from any normal mode then SPSR.D bit is already -* cleared, however it is set when we are entering from any debug exception -* mode. -* Since we always need to generate single step exception after a kprobe -* breakpoint exception therefore we need to clear it unconditionally, when -* we become sure that the current breakpoint exception is for kprobe. -*/ -static void __kprobes -spsr_set_debug_flag (struct pt_regs *regs,int mask) -{ -unsigned long spsr = regs->pstate; - -if (mask) - spsr |=PSR_D_BIT ; -else - spsr &= ~PSR_D_BIT ; - - regs->pstate = spsr; -} -/* * Interrupts need to be disabled before single-step mode is set, and not * reenabled until after single-step mode ends.
@@ -205,17 +180,17 @@ spsr_set_debug_flag(struct pt_regs *regs, int mask)static void __kprobes kprobes_save_local_irqflag (struct kprobe_ctlblk *kcb,struct pt_regs *regs) { - kcb->saved_irqflag = regs->pstate; + kcb->saved_irqflag = regs->pstate &PSR_DAIF_MASK ; regs->pstate |=PSR_I_BIT ; +/* Unmask PSTATE.D for enabling software step exceptions. */ + regs->pstate &= ~PSR_D_BIT ; }static void __kprobes kprobes_restore_local_irqflag (struct kprobe_ctlblk *kcb,struct pt_regs *regs) { -if (kcb->saved_irqflag &PSR_I_BIT ) - regs->pstate |=PSR_I_BIT ; -else - regs->pstate &= ~PSR_I_BIT ; + regs->pstate &= ~PSR_DAIF_MASK ; + regs->pstate |= kcb->saved_irqflag; } static void __kprobes
@@ -252,8 +227,6 @@ static void __kprobes setup_singlestep(struct kprobe *p,set_ss_context (kcb, slot);/* mark pending ss */ -spsr_set_debug_flag (regs,0 ); -/* IRQs and single stepping do not mix well. */ kprobes_save_local_irqflag (kcb, regs);kernel_enable_single_step (regs);
Re: [PATCH v2 1/4] arm64: kprobes: Recover pstate.D in single-step exception handler
Hi!
Quoted 5 lines · Masami Hiramatsu
On arm64, if a nested kprobes hit, it can crash the kernel with below error message.
[ 152.118921] Unexpected kernel single-step exception at EL1
Quoted 7 lines
This is because commit
7419333fa15e("arm64: kprobe: Always clear pstate.D in breakpoint exception handler") unmask pstate.D for doing single step but does not recover it after single step in the nested kprobes.That is correct *unless* any nested kprobes (single-stepping) runs inside other kprobes user handler.
(I don't think this is correct, its just usually invisible as PSTATE.D is normally clear)
Quoted 8 lines
When the 1st kprobe hits,
do_debug_exception()will be called. At this point, debug exception (= pstate.D) must be masked (=1). When the 2nd
(nested) kprobe is hit before single-step of the first kprobe, it
unmask debug exception (pstate.D = 0) and return.
Then, when the 1st kprobe setting up single-step, it saves current DAIF, mask DAIF, enable single-step, and restore DAIF. However, since "D" flag in DAIF is cleared by the 2nd kprobe, the single-step exception happens soon after restoring DAIF.
This is pretty complicated. Just to check I've understood this properly:
Stepping on a kprobe in a kprobe-user's pre_handler will cause the remainder of the handler (the first one) to run with PSTATE.D clear. Once we enable single-step, we start stepping the debug handler, and will never step the original kprobe'd instruction.
This is describing the most complicated way that this problem shows up! (I agree its also the worst)
I can get this to show up with just one kprobe. (function/file names here are meaningless):
| static int wibble(struct seq_file *m, void *discard)
| {
| unsigned long d, flags;
|
| flags = local_daif_save();
|
| kprobe_me();
| d = read_sysreg(daif);
| local_daif_restore(flags);
|
| seq_printf(m, "%lx\n", d);
|
| return 0;
| }plumbed into debugfs, then kicked using the kprobe_example module:
| root@adam:/sys/kernel/debug# cat wibble
| 3c0
| root@adam:/sys/kernel/debug# insmod ~morse/kprobe_example.ko symbol=kprobe_me | [ 69.478098] Planted kprobe at [..] | root@adam:/sys/kernel/debug# cat wibble | [ 71.478935] <kprobe_me> pre_handler: p->addr = [..], pc = [..], pstate = 0x600003c5 | [ 71.488942] <kprobe_me> post_handler: p->addr = [..], pstate = 0x600001c5 | 1c0
| root@adam:/sys/kernel/debug#
This is problem for any code that had debug masked, not just kprobes.
Can we start the commit-message with the simplest description of the problem: kprobes manipulates the interrupted PSTATE for single step, and doesn't restore it.
(trying to understand this bug through kprobe's interaction with itself is hard!)
arch/arm64/kernel/probes/kprobes.c line 32
Quoted 13 lines
To solve this issue, this stores all DAIF bits and restore it after single stepping.
arch/arm64/kernel/probes/kprobes.c +1
-0@@ -29,6 +29,8 @@#include "decode-insn.h" +#define PSR_DAIF_MASK (PSR_D_BIT | PSR_A_BIT | PSR_I_BIT | PSR_F_BIT)
We should probably move this to daifflags.h. Its going to be useful to other series too.
Patch looks good!
(I haven't tried to test the nested kprobes case...)
Thanks,
James
Re: [PATCH v2 1/4] arm64: kprobes: Recover pstate.D in single-step exception handler
Quoted 13 lines · On Tue, 23 Jul 2019 17:03:47 +0100 James Morse <[email protected]> wrote:
Hi!
Quoted 9 lines · Masami Hiramatsu
On arm64, if a nested kprobes hit, it can crash the kernel with below error message.
[ 152.118921] Unexpected kernel single-step exception at EL1This is because commit
7419333fa15e("arm64: kprobe: Always clear pstate.D in breakpoint exception handler") unmask pstate.D for doing single step but does not recover it after single step in the nested kprobes.
Quoted 4 lines
Quoted 2 lines
That is correct *unless* any nested kprobes (single-stepping) runs inside other kprobes user handler.
(I don't think this is correct, its just usually invisible as PSTATE.D is normally clear)
Ah, right.
Quoted 11 lines
Quoted 10 lines
When the 1st kprobe hits,
do_debug_exception()will be called. At this point, debug exception (= pstate.D) must be masked (=1). When the 2nd
(nested) kprobe is hit before single-step of the first kprobe, it
unmask debug exception (pstate.D = 0) and return.
Then, when the 1st kprobe setting up single-step, it saves current DAIF, mask DAIF, enable single-step, and restore DAIF. However, since "D" flag in DAIF is cleared by the 2nd kprobe, the single-step exception happens soon after restoring DAIF.
Quoted 4 lines
This is pretty complicated. Just to check I've understood this properly:
Stepping on a kprobe in a kprobe-user'spre_handlerwill cause the remainder of the handler (the first one) to run with PSTATE.D clear. Once we enable single-step, we start stepping the debug handler, and will never step the original kprobe'd instruction.
Yes, that's correct. I saw the single stepping happens on right after recover the saved daif.
Quoted 33 lines
This is describing the most complicated way that this problem shows up! (I agree its also the worst)
I can get this to show up with just one kprobe. (function/file names here are meaningless):
| static int wibble(struct seq_file *m, void *discard) | { | unsigned long d, flags; | | flags = local_daif_save(); | | kprobe_me(); | d = read_sysreg(daif); | local_daif_restore(flags); | | seq_printf(m, "%lx\n", d); | | return 0; | }plumbed into debugfs, then kicked using the
kprobe_examplemodule:
| root@adam:/sys/kernel/debug# cat wibble
| 3c0| root@adam:/sys/kernel/debug# insmod ~morse/kprobe_example.ko symbol=kprobe_me | [ 69.478098] Planted kprobe at [..] | root@adam:/sys/kernel/debug# cat wibble | [ 71.478935] <kprobe_me> pre_handler: p->addr = [..], pc = [..], pstate = 0x600003c5 | [ 71.488942] <kprobe_me> post_handler: p->addr = [..], pstate = 0x600001c5 | 1c0 | root@adam:/sys/kernel/debug#
Quoted 1 line
This is problem for any code that had debug masked, not just kprobes.
Agreed.
Quoted 3 lines
Can we start the commit-message with the simplest description of the problem: kprobes manipulates the interrupted PSTATE for single step, and doesn't restore it.
Thanks for making it clearer :)
Quoted 14 lines
(trying to understand this bug through kprobe's interaction with itself is hard!)
Quoted 10 lines
To solve this issue, this stores all DAIF bits and restore it after single stepping.
arch/arm64/kernel/probes/kprobes.c +0
-0@@ -29,6 +29,8 @@
Quoted 5 lines
arch/arm64/kernel/probes/kprobes.c line 32
Quoted 3 lines
arch/arm64/kernel/probes/kprobes.c +1
-0#include "decode-insn.h" +#define PSR_DAIF_MASK (PSR_D_BIT | PSR_A_BIT | PSR_I_BIT | PSR_F_BIT)We should probably move this to
daifflags.h. Its going to be useful to other series too.
OK.
Quoted 7 lines
Patch looks good!
Reviewed-by James Morse <[email protected]>Tested-by James Morse <[email protected]>(I haven't tried to test the nested kprobes case...)
OK, I'll update and resend it.
Thank you!
Quoted 5 lines
Thanks,
James
Signature · 1 line
Masami Hiramatsu <[email protected]>
PATCH v2 2/4 arm64: unwind: Prohibit probing on return_address()
In reply to Masami HiramatsuProhibit probing on return_address() and subroutines which is called from return_address(), since the it is invoked from trace_hardirqs_off() which is also kprobe blacklisted.
--- arch/arm64/kernel/return_address.c | 4 +++- arch/arm64/kernel/stacktrace.c | 3 +++ 2 files changed, 6 insertions(+), 1 deletion(-)
arch/arm64/kernel/return_address.c +3 -1
@@ -8,6 +8,7 @@#include <linux/export.h> #include <linux/ftrace.h> +#include <linux/kprobes.h> #include <asm/stack_pointer.h> #include <asm/stacktrace.h>
@@ -17,7 +18,7 @@ struct return_address_data {void *addr; }; -static int save_return_addr (struct stackframe *frame,void *d) +static nokprobe_inline int save_return_addr(struct stackframe *frame, void *d) {struct return_address_data *data = d;
@@ -52,3 +53,4 @@ void *return_address(unsigned int level)return NULL ; }EXPORT_SYMBOL_GPL (return_address); +NOKPROBE_SYMBOL (return_address);
arch/arm64/kernel/stacktrace.c +3 -0
@@ -7,6 +7,7 @@#include <linux/kernel.h> #include <linux/export.h> #include <linux/ftrace.h> +#include <linux/kprobes.h> #include <linux/sched.h> #include <linux/sched/debug.h> #include <linux/sched/task_stack.h>
@@ -73,6 +74,7 @@ int notrace unwind_frame(struct task_struct *tsk, struct stackframe *frame)return 0 ; } +NOKPROBE_SYMBOL (unwind_frame); void notrace walk_stackframe(struct task_struct *tsk, struct stackframe *frame,int (*fn)(struct stackframe *,void *),void *data)
@@ -87,6 +89,7 @@ void notrace walk_stackframe(struct task_struct *tsk, struct stackframe *frame,break ; } } +NOKPROBE_SYMBOL (walk_stackframe);#ifdef CONFIG_STACKTRACE struct stack_trace_data {
Re: [PATCH v2 2/4] arm64: unwind: Prohibit probing on return_address()
Hi,
Quoted 3 lines · Masami Hiramatsu
Prohibit probing on
return_address()and subroutines which is called fromreturn_address(), since the it is invoked fromtrace_hardirqs_off()which is also kprobe blacklisted.
(Nits: "which are called" and "since it is")
Quoted 13 lines
arch/arm64/kernel/return_address.c +1
-0@@ -8,6 +8,7 @@#include <linux/export.h> #include <linux/ftrace.h> +#include <linux/kprobes.h> #include <asm/stack_pointer.h> #include <asm/stacktrace.h>
arch/arm64/kernel/return_address.c struct return_address_data { · line 21
Quoted 5 lines
arch/arm64/kernel/return_address.c +1
-1@@ -17,7 +18,7 @@ struct return_address_data {void *addr; }; -static int save_return_addr (struct stackframe *frame,void *d) +static nokprobe_inline int save_return_addr(struct stackframe *frame, void *d)
This nokprobe_inline ends up as __always_inline if kprobes is enabled. What do we expect the compiler to do with this? save_return_addr is passed as a function-pointer to walk_stackframe()... I don't see how the compiler can inline it!
This would be needed for on_accessible_stack().
Should we cover ftrace_graph_get_ret_stack()?, or is that already in hand?
arch/arm64/kernel/return_address.c void *return_address(unsigned int level) · line 56
Quoted 8 lines
arch/arm64/kernel/return_address.c +1
-0{struct return_address_data *data = d;@@ -52,3 +53,4 @@ void *return_address(unsigned int level)return NULL ; }EXPORT_SYMBOL_GPL (return_address); +NOKPROBE_SYMBOL (return_address);
Thanks,
James
Re: [PATCH v2 2/4] arm64: unwind: Prohibit probing on return_address()
Quoted 8 lines · On Tue, 23 Jul 2019 17:04:21 +0100 James Morse <[email protected]> wrote:
Hi,
Quoted 3 lines · Masami Hiramatsu
Prohibit probing on
return_address()and subroutines which is called fromreturn_address(), since the it is invoked fromtrace_hardirqs_off()which is also kprobe blacklisted.(Nits: "which are called" and "since it is")
Thanks!
Quoted 18 lines
Quoted 18 lines
arch/arm64/kernel/return_address.c +1
-0@@ -8,6 +8,7 @@#include <linux/export.h> #include <linux/ftrace.h> +#include <linux/kprobes.h> #include <asm/stack_pointer.h> #include <asm/stacktrace.h> @@ -17,7 +18,7 @@ struct return_address_data {void *addr; };
Quoted 6 lines
arch/arm64/kernel/return_address.c struct return_address_data { · line 21
Quoted 2 lines
arch/arm64/kernel/return_address.c +1
-1-static int save_return_addr (struct stackframe *frame,void *d) +static nokprobe_inline int save_return_addr(struct stackframe *frame, void *d)This
nokprobe_inlineends up as__always_inlineif kprobes is enabled. What do we expect the compiler to do with this?save_return_addris passed as a function-pointer towalk_stackframe()... I don't see how the compiler can inline it!
Oops, that's my mistake. Then it should be NOKPROBE_SYMBOL.
Quoted 3 lines
This would be needed for
on_accessible_stack().
Should we coverftrace_graph_get_ret_stack()?, or is that already in hand?
No, that is OK. It just covers that the functions which are involved in the kprobe execution path. ftrace_graph_ret_stack() is out of the debug exception handler.
Thank you,
Quoted 13 lines
arch/arm64/kernel/return_address.c void *return_address(unsigned int level) · line 56
Quoted 8 lines
arch/arm64/kernel/return_address.c +1
-0{struct return_address_data *data = d;@@ -52,3 +53,4 @@ void *return_address(unsigned int level)return NULL ; }EXPORT_SYMBOL_GPL (return_address); +NOKPROBE_SYMBOL (return_address);Thanks,
James
Signature · 1 line
Masami Hiramatsu <[email protected]>
PATCH v2 3/4 arm64: Make debug exception handlers visible from RCU
In reply to Masami HiramatsuMake debug exceptions visible from RCU so that synchronize_rcu() correctly track the debug exception handler.
This also introduces sanity checks for user-mode exceptions as same as x86's ist_enter()/ist_exit().
The debug exception can interrupt in idle task. For example, it warns if we put a kprobe on a function called from idle task as below. The warning message showed that the rcu_read_lock() caused this problem. But actually, this means the RCU is lost the context which is already in NMI/IRQ.
/sys/kernel/debug/tracing # echo p default_idle_call >> kprobe_events /sys/kernel/debug/tracing # echo 1 > events/kprobes/enable /sys/kernel/debug/tracing # [ 135.122237]
[ 135.125035] =============================
[ 135.125310] WARNING: suspicious RCU usage
[ 135.125581] 5.2.0-08445-g9187c508bdc7 #20 Not tainted
[ 135.125904] -----------------------------
[ 135.126205] include/linux/rcupdate.h:594 rcu_read_lock() used illegally while idle!
[ 135.126839]
[ 135.126839] other info that might help us debug this:
[ 135.126839]
[ 135.127410]
[ 135.127410] RCU used illegally from idle CPU!
[ 135.127410] rcu_scheduler_active = 2, debug_locks = 1
[ 135.128114] RCU used illegally from extended quiescent state!
[ 135.128555] 1 lock held by swapper/0/0:
[ 135.128944] #0: (____ptrval____) (rcu_read_lock){....}, at: call_break_hook+0x0/0x178
[ 135.130499]
[ 135.130499] stack backtrace:
[ 135.131192] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.2.0-08445-g9187c508bdc7 #20
[ 135.131841] Hardware name: linux,dummy-virt (DT)
[ 135.132224] Call trace:
[ 135.132491] dump_backtrace+0x0/0x140
[ 135.132806] show_stack+0x24/0x30
[ 135.133133] dump_stack+0xc4/0x10c
[ 135.133726] lockdep_rcu_suspicious+0xf8/0x108
[ 135.134171] call_break_hook+0x170/0x178
[ 135.134486] brk_handler+0x28/0x68
[ 135.134792] do_debug_exception+0x90/0x150
[ 135.135051] el1_dbg+0x18/0x8c
[ 135.135260] default_idle_call+0x0/0x44
[ 135.135516] cpu_startup_entry+0x2c/0x30
[ 135.135815] rest_init+0x1b0/0x280
[ 135.136044] arch_call_rest_init+0x14/0x1c
[ 135.136305] start_kernel+0x4d4/0x500
[ 135.136597]So make debug exception visible to RCU can fix this warning.
--- arch/arm64/mm/fault.c | 40 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 40 insertions(+)
arch/arm64/mm/fault.c +40 -0
@@ -777,6 +777,42 @@ void __init hook_debug_fault_code(int nr, debug_fault_info[nr].name = name; } +/* +* In debug exception context, we explicitly disable preemption. +* This serves two purposes: it makes it much less likely that we would +* accidentally schedule in exception context and it will force a warning +* if we somehow manage to schedule by accident. +*/ +static void debug_exception_enter (struct pt_regs *regs) +{ +if (user_mode (regs)) { +RCU_LOCKDEP_WARN (!rcu_is_watching (),"entry code didn't wake RCU" ); + }else { +/* +* We might have interrupted pretty much anything. In +* fact, if we're a debug exception, we can even interrupt +* NMI processing. We don't want in_nmi() to return true, +* but we need to notify RCU. +*/ +rcu_nmi_enter (); + } + +preempt_disable (); + +/* This code is a bit fragile. Test it. */ +RCU_LOCKDEP_WARN (!rcu_is_watching (),"exception_enter didn't work" ); +} +NOKPROBE_SYMBOL (debug_exception_enter); + +static void debug_exception_exit (struct pt_regs *regs) +{ +preempt_enable_no_resched (); + +if (!user_mode (regs)) +rcu_nmi_exit (); +} +NOKPROBE_SYMBOL (debug_exception_exit); +#ifdef CONFIG_ARM64_ERRATUM_1463225 DECLARE_PER_CPU (int, __in_cortex_a76_erratum_1463225_wa);
@@ -824,6 +860,8 @@ asmlinkage void __exception do_debug_exception(unsigned long addr_if_watchpoint,if (interrupts_enabled (regs))trace_hardirqs_off (); +debug_exception_enter (regs); +if (user_mode (regs) && !is_ttbr0_addr (pc))arm64_apply_bp_hardening ();
@@ -832,6 +870,8 @@ asmlinkage void __exception do_debug_exception(unsigned long addr_if_watchpoint, inf->sig, inf->code, (void __user *)pc, esr); } +debug_exception_exit (regs); +if (interrupts_enabled (regs))trace_hardirqs_on (); }
Re: [PATCH v2 3/4] arm64: Make debug exception handlers visible from RCU
Quoted 49 lines · Masami Hiramatsu
Make debug exceptions visible from RCU so that
synchronize_rcu()correctly track the debug exception handler.This also introduces sanity checks for user-mode exceptions as same as x86's ist_enter()/ist_exit().
The debug exception can interrupt in idle task. For example, it warns if we put a kprobe on a function called from idle task as below. The warning message showed that the
rcu_read_lock()caused this problem. But actually, this means the RCU is lost the context which is already in NMI/IRQ.
/sys/kernel/debug/tracing# echo pdefault_idle_call>>kprobe_events/sys/kernel/debug/tracing# echo 1 > events/kprobes/enable/sys/kernel/debug/tracing# [ 135.122237][ 135.125035] ============================= [ 135.125310] WARNING: suspicious RCU usage [ 135.125581] 5.2.0-08445-g9187c508bdc7 #20 Not tainted [ 135.125904] ----------------------------- [ 135.126205] include/linux/rcupdate.h:594 rcu_read_lock() used illegally while idle! [ 135.126839] [ 135.126839] other info that might help us debug this: [ 135.126839] [ 135.127410] [ 135.127410] RCU used illegally from idle CPU! [ 135.127410] rcu_scheduler_active = 2, debug_locks = 1 [ 135.128114] RCU used illegally from extended quiescent state! [ 135.128555] 1 lock held by swapper/0/0: [ 135.128944] #0: (____ptrval____) (rcu_read_lock){....}, at: call_break_hook+0x0/0x178 [ 135.130499] [ 135.130499] stack backtrace: [ 135.131192] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.2.0-08445-g9187c508bdc7 #20 [ 135.131841] Hardware name: linux,dummy-virt (DT) [ 135.132224] Call trace: [ 135.132491] dump_backtrace+0x0/0x140 [ 135.132806] show_stack+0x24/0x30 [ 135.133133] dump_stack+0xc4/0x10c [ 135.133726] lockdep_rcu_suspicious+0xf8/0x108 [ 135.134171] call_break_hook+0x170/0x178 [ 135.134486] brk_handler+0x28/0x68 [ 135.134792] do_debug_exception+0x90/0x150 [ 135.135051] el1_dbg+0x18/0x8c [ 135.135260] default_idle_call+0x0/0x44 [ 135.135516] cpu_startup_entry+0x2c/0x30 [ 135.135815] rest_init+0x1b0/0x280 [ 135.136044] arch_call_rest_init+0x14/0x1c [ 135.136305] start_kernel+0x4d4/0x500 [ 135.136597]
Quoted 4 lines
So make debug exception visible to RCU can fix this warning.
Reported-by Naresh Kamboju <[email protected]>Cc Paul E. McKenney <[email protected]>
From an RCU viewpoint:
Quoted 71 lines
Signed-off-by Masami Hiramatsu <[email protected]>--- arch/arm64/mm/fault.c | 40 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 40 insertions(+)arch/arm64/mm/fault.c +40
-0@@ -777,6 +777,42 @@ void __init hook_debug_fault_code(int nr, debug_fault_info[nr].name = name; } +/* +* In debug exception context, we explicitly disable preemption. +* This serves two purposes: it makes it much less likely that we would +* accidentally schedule in exception context and it will force a warning +* if we somehow manage to schedule by accident. +*/ +static void debug_exception_enter (struct pt_regs *regs) +{ +if (user_mode (regs)) { +RCU_LOCKDEP_WARN (!rcu_is_watching (),"entry code didn't wake RCU" ); + }else { +/* +* We might have interrupted pretty much anything. In +* fact, if we're a debug exception, we can even interrupt +* NMI processing. We don't want in_nmi() to return true, +* but we need to notify RCU. +*/ +rcu_nmi_enter (); + } + +preempt_disable (); + +/* This code is a bit fragile. Test it. */ +RCU_LOCKDEP_WARN (!rcu_is_watching (),"exception_enter didn't work" ); +} +NOKPROBE_SYMBOL (debug_exception_enter); + +static void debug_exception_exit (struct pt_regs *regs) +{ +preempt_enable_no_resched (); + +if (!user_mode (regs)) +rcu_nmi_exit (); +} +NOKPROBE_SYMBOL (debug_exception_exit); +#ifdef CONFIG_ARM64_ERRATUM_1463225 DECLARE_PER_CPU (int, __in_cortex_a76_erratum_1463225_wa);@@ -824,6 +860,8 @@ asmlinkage void __exception do_debug_exception(unsigned long addr_if_watchpoint,if (interrupts_enabled (regs))trace_hardirqs_off (); +debug_exception_enter (regs); +if (user_mode (regs) && !is_ttbr0_addr (pc))arm64_apply_bp_hardening ();@@ -832,6 +870,8 @@ asmlinkage void __exception do_debug_exception(unsigned long addr_if_watchpoint, inf->sig, inf->code, (void __user *)pc, esr); } +debug_exception_exit (regs); +if (interrupts_enabled (regs))trace_hardirqs_on (); }
Re: [PATCH v2 3/4] arm64: Make debug exception handlers visible from RCU
In reply to Masami HiramatsuHi,
Quoted 22 lines · Masami Hiramatsu
Make debug exceptions visible from RCU so that
synchronize_rcu()correctly track the debug exception handler.This also introduces sanity checks for user-mode exceptions as same as x86's ist_enter()/ist_exit().
The debug exception can interrupt in idle task. For example, it warns if we put a kprobe on a function called from idle task as below. The warning message showed that the
rcu_read_lock()caused this problem. But actually, this means the RCU is lost the context which is already in NMI/IRQ.So make debug exception visible to RCU can fix this warning.
arch/arm64/mm/fault.c +0
-0@@ -777,6 +777,42 @@ void __init hook_debug_fault_code(int nr, debug_fault_info[nr].name = name; }
arch/arm64/mm/fault.c void __init hook_debug_fault_code(int nr, · line 789
Quoted 10 lines
arch/arm64/mm/fault.c +10
-0+/* +* In debug exception context, we explicitly disable preemption. +* This serves two purposes: it makes it much less likely that we would +* accidentally schedule in exception context and it will force a warning +* if we somehow manage to schedule by accident. +*/ +static void debug_exception_enter (struct pt_regs *regs) +{ +if (user_mode (regs)) { +RCU_LOCKDEP_WARN (!rcu_is_watching (),"entry code didn't wake RCU" );
Would moving entry.S's context_tracking_user_exit() call to be before do_debug_exception() also fix this?
I don't know the reason its done 'after' debug exception handling. Its always been like
this: commit 6c81fe7925cc4c42 ("arm64: enable context tracking").
arch/arm64/mm/fault.c void __init hook_debug_fault_code(int nr,
Quoted 8 lines
arch/arm64/mm/fault.c +7
-0+ }else { +/* +* We might have interrupted pretty much anything. In +* fact, if we're a debug exception, we can even interrupt +* NMI processing. +* We don't want in_nmi() to return true, +* but we need to notify RCU.
How come? If you interrupted an SError or pseudo-nmi, it already is. Those paths should all be painted no-kprobe, but I'm sure there are gaps. The hw-breakpoints can almost certainly hook them.
arch/arm64/mm/fault.c void __init hook_debug_fault_code(int nr, · line 799
Quoted 2 lines
arch/arm64/mm/fault.c +2
-0+*/ +rcu_nmi_enter ();
Can we interrupt printk()? Do we need printk_nmi_enter()? ... What about ftrace?
Because SError and pseudo-nmi can interrupt interrupt-masked code, we describe them as NMI. The only difference here is these exceptions are synchronous.
I suspect we should make these debug exceptions nmi for EL1. We can then use this for the kprobe-re-entrance stuff so the pre/post hooks don't get run if they interrupted something also described as NMI.
arch/arm64/mm/fault.c void __init hook_debug_fault_code(int nr, · line 805
Quoted 8 lines
arch/arm64/mm/fault.c +8
-0+ } + +preempt_disable (); + +/* This code is a bit fragile. Test it. */ +RCU_LOCKDEP_WARN (!rcu_is_watching (),"exception_enter didn't work" ); +} +NOKPROBE_SYMBOL (debug_exception_enter);
Thanks,
James
Re: [PATCH v2 3/4] arm64: Make debug exception handlers visible from RCU
Quoted 25 lines · On Tue, 23 Jul 2019 18:07:56 +0100 James Morse <[email protected]> wrote:
Hi,
Quoted 22 lines · Masami Hiramatsu
Make debug exceptions visible from RCU so that
synchronize_rcu()correctly track the debug exception handler.This also introduces sanity checks for user-mode exceptions as same as x86's ist_enter()/ist_exit().
The debug exception can interrupt in idle task. For example, it warns if we put a kprobe on a function called from idle task as below. The warning message showed that the
rcu_read_lock()caused this problem. But actually, this means the RCU is lost the context which is already in NMI/IRQ.So make debug exception visible to RCU can fix this warning.
arch/arm64/mm/fault.c +0
-0@@ -777,6 +777,42 @@ void __init hook_debug_fault_code(int nr, debug_fault_info[nr].name = name; }
Quoted 13 lines
arch/arm64/mm/fault.c void __init hook_debug_fault_code(int nr, · line 789
Quoted 10 lines
arch/arm64/mm/fault.c +10
-0+/* +* In debug exception context, we explicitly disable preemption. +* This serves two purposes: it makes it much less likely that we would +* accidentally schedule in exception context and it will force a warning +* if we somehow manage to schedule by accident. +*/ +static void debug_exception_enter (struct pt_regs *regs) +{ +if (user_mode (regs)) { +RCU_LOCKDEP_WARN (!rcu_is_watching (),"entry code didn't wake RCU" );Would moving entry.S's
context_tracking_user_exit()call to be beforedo_debug_exception()also fix this?
It sounds like treating only user context, correct?
This part is just adding assertion, not fixing the problem which Naresh reported.
Quoted 11 lines
I don't know the reason its done 'after' debug exception handling. Its always been like
this: commit6c81fe7925cc4c42("arm64: enable context tracking").Quoted 5 lines
arch/arm64/mm/fault.c +5
-0+ }else { +/* +* We might have interrupted pretty much anything. In +* fact, if we're a debug exception, we can even interrupt +* NMI processing.
Quoted 6 lines
arch/arm64/mm/fault.c void __init hook_debug_fault_code(int nr, · line 796
Quoted 2 lines
arch/arm64/mm/fault.c +2
-0+* We don't want in_nmi() to return true, +* but we need to notify RCU. How come? If you interrupted an SError or pseudo-nmi, it already is. Those paths should all be painted no-kprobe, but I'm sure there are gaps. The hw-breakpoints can almost certainly hook them.
I think that sentense means "we don't want that this code makes in_nmi() to return true" So, if the breakpoint interrupts pNMI/SError context, it is OK that in_nmi() returns true.
Quoted 6 lines
arch/arm64/mm/fault.c void __init hook_debug_fault_code(int nr, · line 798
Quoted 4 lines
arch/arm64/mm/fault.c +2
-0+*/ +rcu_nmi_enter ();Can we interrupt
printk()? Do we needprintk_nmi_enter()? ... What about ftrace?
Good point! As far as I know, we don't use it because ftrace doesn't use printk. But indeed, kprobes user can use printk and they have to call printk_nmi_enter()/exit(), that must be commented in the documentation. Anyway, basically it is user's choice.
Quoted 8 lines
Because SError and pseudo-nmi can interrupt interrupt-masked code, we describe them as NMI. The only difference here is these exceptions are synchronous.
I suspect we should make these debug exceptions nmi for EL1. We can then use this for the kprobe-re-entrance stuff so the pre/post hooks don't get run if they interrupted something also described as NMI.
I'm not sure how it can prevent... anyway because we have to run a single-stepping for recovery, and kprobe already check the reentered kprobes and skip user-handlers in such case.
Thank you,
Quoted 15 lines
arch/arm64/mm/fault.c void __init hook_debug_fault_code(int nr, · line 805
Quoted 10 lines
arch/arm64/mm/fault.c +8
-0+ } + +preempt_disable (); + +/* This code is a bit fragile. Test it. */ +RCU_LOCKDEP_WARN (!rcu_is_watching (),"exception_enter didn't work" ); +} +NOKPROBE_SYMBOL (debug_exception_enter);Thanks,
James
Signature · 1 line
Masami Hiramatsu <[email protected]>
PATCH v2 4/4 arm64: Remove unneeded rcu_read_lock from debug handlers
In reply to Masami HiramatsuRemove rcu_read_lock()/rcu_read_unlock() from debug exception handlers since we are sure those are not preemptible and interrupts are off.
--- arch/arm64/kernel/debug-monitors.c | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-)
arch/arm64/kernel/debug-monitors.c +8 -6
@@ -207,16 +207,16 @@ static int call_step_hook(struct pt_regs *regs, unsigned int esr) list =user_mode (regs) ? &user_step_hook : &kernel_step_hook; -rcu_read_lock (); - +/* +* Since single-step exception disables interrupt, this function is +* entirely not preemptible, and we can use rcu list safely here. +*/ list_for_each_entry_rcu (hook, list, node) { retval = hook->fn (regs, esr);if (retval ==DBG_HOOK_HANDLED )break ; } -rcu_read_unlock (); -return retval; }NOKPROBE_SYMBOL (call_step_hook);
@@ -305,14 +305,16 @@ static int call_break_hook(struct pt_regs *regs, unsigned int esr) list =user_mode (regs) ? &user_break_hook : &kernel_break_hook; -rcu_read_lock (); +/* +* Since brk exception disables interrupt, this function is +* entirely not preemptible, and we can use rcu list safely here. +*/ list_for_each_entry_rcu (hook, list, node) {unsigned int comment = esr & ESR_ELx_BRK64_ISS_COMMENT_MASK;if ((comment & ~hook->mask) == hook->imm) fn = hook->fn; } -rcu_read_unlock ();return fn ?fn (regs, esr) :DBG_HOOK_ERROR ; }
Re: [PATCH v2 4/4] arm64: Remove unneeded rcu_read_lock from debug handlers
Quoted 6 lines · Masami Hiramatsu
Remove rcu_read_lock()/rcu_read_unlock() from debug exception handlers since we are sure those are not preemptible and interrupts are off.
Signed-off-by Masami Hiramatsu <[email protected]>Cc Paul E. McKenney <[email protected]>
From an RCU viewpoint:
Quoted 49 lines
--- arch/arm64/kernel/debug-monitors.c | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-)arch/arm64/kernel/debug-monitors.c +8
-6@@ -207,16 +207,16 @@ static int call_step_hook(struct pt_regs *regs, unsigned int esr) list =user_mode (regs) ? &user_step_hook : &kernel_step_hook; -rcu_read_lock (); - +/* +* Since single-step exception disables interrupt, this function is +* entirely not preemptible, and we can use rcu list safely here. +*/ list_for_each_entry_rcu (hook, list, node) { retval = hook->fn (regs, esr);if (retval ==DBG_HOOK_HANDLED )break ; } -rcu_read_unlock (); -return retval; }NOKPROBE_SYMBOL (call_step_hook);@@ -305,14 +305,16 @@ static int call_break_hook(struct pt_regs *regs, unsigned int esr) list =user_mode (regs) ? &user_break_hook : &kernel_break_hook; -rcu_read_lock (); +/* +* Since brk exception disables interrupt, this function is +* entirely not preemptible, and we can use rcu list safely here. +*/ list_for_each_entry_rcu (hook, list, node) {unsigned int comment = esr & ESR_ELx_BRK64_ISS_COMMENT_MASK;if ((comment & ~hook->mask) == hook->imm) fn = hook->fn; } -rcu_read_unlock ();return fn ?fn (regs, esr) :DBG_HOOK_ERROR ; }