Skip to contentSkip to conversation

arm64: kprobes: Fix some bugs in arm64 kprobes

PATCH Superseded Masami Hiramatsu 3 participants 13 messages Jul 22, 2019 – Jul 24, 2019 arm64

v2 of 4 Superseded Latest is v4 →

lore
Share & tools
lore
#0 lore raw

v2 cover arm64: kprobes: Fix some bugs in arm64 kprobes

Hi,

Here are the v2 patches which fixes kprobe bugs on arm64.

Naresh reported that recently ftracetest crashes kernel, and I found there are 3 different bugs around the crash. In v1 thread, we found one another bug of RCU and debug exception.

- Kprobes on arm64 doesn't recover pstate.D mask after single stepping. This causes a real kernel crash if a kprobe is unexpectedly nested.
- Some symbols which are called from blacklisted function, are not blacklisted.
- Debug exception is not visible to RCU, thus rcu_read_lock() cause a warning inside it.
- Debug exception handlers on arm64 is using rcu_read_lock(), but that is not needed because interrupts are disabled.

This series includes fixes for above bugs.

Thank you,

---

Masami Hiramatsu (4):
arm64: kprobes: Recover pstate.D in single-step exception handler
arm64: unwind: Prohibit probing on return_address()
arm64: Make debug exception handlers visible from RCU
arm64: Remove unneeded rcu_read_lock from debug handlers

 arch/arm64/kernel/debug-monitors.c |   14 +++++++-----
 arch/arm64/kernel/probes/kprobes.c |   41 ++++++------------------------------
 arch/arm64/kernel/return_address.c |    4 +++-
 arch/arm64/kernel/stacktrace.c     |    3 +++
 arch/arm64/mm/fault.c              |   40 +++++++++++++++++++++++++++++++++++
 5 files changed, 61 insertions(+), 41 deletions(-)

--
Masami Hiramatsu (Linaro) <[email protected]>

#1 lore raw

PATCH v2 1/4 arm64: kprobes: Recover pstate.D in single-step exception handler

On arm64, if a nested kprobes hit, it can crash the kernel with below error message.

[  152.118921] Unexpected kernel single-step exception at EL1

This is because commit 7419333fa15e ("arm64: kprobe: Always clear pstate.D in breakpoint exception handler") unmask pstate.D for doing single step but does not recover it after single step in the nested kprobes. That is correct *unless* any nested kprobes (single-stepping) runs inside other kprobes user handler.

When the 1st kprobe hits, do_debug_exception() will be called. At this point, debug exception (= pstate.D) must be masked (=1). When the 2nd
(nested) kprobe is hit before single-step of the first kprobe, it
unmask debug exception (pstate.D = 0) and return.
Then, when the 1st kprobe setting up single-step, it saves current DAIF, mask DAIF, enable single-step, and restore DAIF. However, since "D" flag in DAIF is cleared by the 2nd kprobe, the single-step exception happens soon after restoring DAIF.

To solve this issue, this stores all DAIF bits and restore it after single stepping.

Fixes commit 7419333fa15e ("arm64: kprobe: Always clear pstate.D in breakpoint exception handler")

---
Changes in v2:
- Save and restore all DAIF flags.
- Operate pstate directly and remove spsr_set_debug_flag().

---
 arch/arm64/kernel/probes/kprobes.c |   41 ++++++------------------------------
 1 file changed, 7 insertions(+), 34 deletions(-)
arch/arm64/kernel/probes/kprobes.c +7 -34
index bd5dfffca272..348e02b799a2 100644
@@ -29,6 +29,8 @@
 
 #include "decode-insn.h"
 
+#define PSR_DAIF_MASK	(PSR_D_BIT | PSR_A_BIT | PSR_I_BIT | PSR_F_BIT)
+
 DEFINE_PER_CPU(struct kprobe *, current_kprobe) = NULL;
 DEFINE_PER_CPU(struct kprobe_ctlblk, kprobe_ctlblk);
 
@@ -167,33 +169,6 @@ static void __kprobes set_current_kprobe(struct kprobe *p)
 	__this_cpu_write(current_kprobe, p);
 }
 
-/*
- * When PSTATE.D is set (masked), then software step exceptions can not be
- * generated.
- * SPSR's D bit shows the value of PSTATE.D immediately before the
- * exception was taken. PSTATE.D is set while entering into any exception
- * mode, however software clears it for any normal (none-debug-exception)
- * mode in the exception entry. Therefore, when we are entering into kprobe
- * breakpoint handler from any normal mode then SPSR.D bit is already
- * cleared, however it is set when we are entering from any debug exception
- * mode.
- * Since we always need to generate single step exception after a kprobe
- * breakpoint exception therefore we need to clear it unconditionally, when
- * we become sure that the current breakpoint exception is for kprobe.
- */
-static void __kprobes
-spsr_set_debug_flag(struct pt_regs *regs, int mask)
-{
-	unsigned long spsr = regs->pstate;
-
-	if (mask)
-		spsr |= PSR_D_BIT;
-	else
-		spsr &= ~PSR_D_BIT;
-
-	regs->pstate = spsr;
-}
-
 /*
  * Interrupts need to be disabled before single-step mode is set, and not
  * reenabled until after single-step mode ends.
@@ -205,17 +180,17 @@ spsr_set_debug_flag(struct pt_regs *regs, int mask)
 static void __kprobes kprobes_save_local_irqflag(struct kprobe_ctlblk *kcb,
 						struct pt_regs *regs)
 {
-	kcb->saved_irqflag = regs->pstate;
+	kcb->saved_irqflag = regs->pstate & PSR_DAIF_MASK;
 	regs->pstate |= PSR_I_BIT;
+	/* Unmask PSTATE.D for enabling software step exceptions. */
+	regs->pstate &= ~PSR_D_BIT;
 }
 
 static void __kprobes kprobes_restore_local_irqflag(struct kprobe_ctlblk *kcb,
 						struct pt_regs *regs)
 {
-	if (kcb->saved_irqflag & PSR_I_BIT)
-		regs->pstate |= PSR_I_BIT;
-	else
-		regs->pstate &= ~PSR_I_BIT;
+	regs->pstate &= ~PSR_DAIF_MASK;
+	regs->pstate |= kcb->saved_irqflag;
 }
 
 static void __kprobes
@@ -252,8 +227,6 @@ static void __kprobes setup_singlestep(struct kprobe *p,
 
 		set_ss_context(kcb, slot);	/* mark pending ss */
 
-		spsr_set_debug_flag(regs, 0);
-
 		/* IRQs and single stepping do not mix well. */
 		kprobes_save_local_irqflag(kcb, regs);
 		kernel_enable_single_step(regs);
Reviewed-by Tested-by on v2 1/4
#2 lore raw

Re: [PATCH v2 1/4] arm64: kprobes: Recover pstate.D in single-step exception handler

Hi!

Quoted 5 lines · Masami Hiramatsu

On arm64, if a nested kprobes hit, it can crash the kernel with below error message.

[  152.118921] Unexpected kernel single-step exception at EL1
Quoted 7 lines

This is because commit 7419333fa15e ("arm64: kprobe: Always clear pstate.D in breakpoint exception handler") unmask pstate.D for doing single step but does not recover it after single step in the nested kprobes.

That is correct *unless* any nested kprobes (single-stepping) runs inside other kprobes user handler.

(I don't think this is correct, its just usually invisible as PSTATE.D is normally clear)

Quoted 8 lines

When the 1st kprobe hits, do_debug_exception() will be called. At this point, debug exception (= pstate.D) must be masked (=1). When the 2nd
(nested) kprobe is hit before single-step of the first kprobe, it
unmask debug exception (pstate.D = 0) and return.
Then, when the 1st kprobe setting up single-step, it saves current DAIF, mask DAIF, enable single-step, and restore DAIF. However, since "D" flag in DAIF is cleared by the 2nd kprobe, the single-step exception happens soon after restoring DAIF.

This is pretty complicated. Just to check I've understood this properly:
Stepping on a kprobe in a kprobe-user's pre_handler will cause the remainder of the handler (the first one) to run with PSTATE.D clear. Once we enable single-step, we start stepping the debug handler, and will never step the original kprobe'd instruction.

This is describing the most complicated way that this problem shows up! (I agree its also the worst)

I can get this to show up with just one kprobe. (function/file names here are meaningless):

| static int wibble(struct seq_file *m, void *discard)
| {
|        unsigned long d, flags;
|
|        flags = local_daif_save();
|
|        kprobe_me();
|        d = read_sysreg(daif);
|        local_daif_restore(flags);
|
|        seq_printf(m, "%lx\n", d);
|
|        return 0;
| }

plumbed into debugfs, then kicked using the kprobe_example module:
| root@adam:/sys/kernel/debug# cat wibble
| 3c0

| root@adam:/sys/kernel/debug# insmod ~morse/kprobe_example.ko symbol=kprobe_me
| [   69.478098] Planted kprobe at [..]
| root@adam:/sys/kernel/debug# cat wibble
| [   71.478935] <kprobe_me> pre_handler: p->addr = [..], pc = [..], pstate = 0x600003c5
| [   71.488942] <kprobe_me> post_handler: p->addr = [..], pstate = 0x600001c5
| 1c0

| root@adam:/sys/kernel/debug#

This is problem for any code that had debug masked, not just kprobes.

Can we start the commit-message with the simplest description of the problem: kprobes manipulates the interrupted PSTATE for single step, and doesn't restore it.

(trying to understand this bug through kprobe's interaction with itself is hard!)

arch/arm64/kernel/probes/kprobes.c line 32

Quoted 13 lines

To solve this issue, this stores all DAIF bits and restore it after single stepping.

arch/arm64/kernel/probes/kprobes.c +1 -0
index bd5dfffca272..348e02b799a2 100644
@@ -29,6 +29,8 @@
 
 #include "decode-insn.h"
 
+#define PSR_DAIF_MASK	(PSR_D_BIT | PSR_A_BIT | PSR_I_BIT | PSR_F_BIT)

We should probably move this to daifflags.h. Its going to be useful to other series too.

Patch looks good!

(I haven't tried to test the nested kprobes case...)

Thanks,

James

#3 lore raw

Re: [PATCH v2 1/4] arm64: kprobes: Recover pstate.D in single-step exception handler

Quoted 13 lines · On Tue, 23 Jul 2019 17:03:47 +0100 James Morse <[email protected]> wrote:

Hi!

Quoted 9 lines · Masami Hiramatsu

On arm64, if a nested kprobes hit, it can crash the kernel with below error message.

[  152.118921] Unexpected kernel single-step exception at EL1

This is because commit 7419333fa15e ("arm64: kprobe: Always clear pstate.D in breakpoint exception handler") unmask pstate.D for doing single step but does not recover it after single step in the nested kprobes.

Quoted 4 lines
Quoted 2 lines

That is correct *unless* any nested kprobes (single-stepping) runs inside other kprobes user handler.

(I don't think this is correct, its just usually invisible as PSTATE.D is normally clear)

Ah, right.

Quoted 11 lines
Quoted 10 lines

When the 1st kprobe hits, do_debug_exception() will be called. At this point, debug exception (= pstate.D) must be masked (=1). When the 2nd
(nested) kprobe is hit before single-step of the first kprobe, it
unmask debug exception (pstate.D = 0) and return.
Then, when the 1st kprobe setting up single-step, it saves current DAIF, mask DAIF, enable single-step, and restore DAIF. However, since "D" flag in DAIF is cleared by the 2nd kprobe, the single-step exception happens soon after restoring DAIF.

Quoted 4 lines

This is pretty complicated. Just to check I've understood this properly:
Stepping on a kprobe in a kprobe-user's pre_handler will cause the remainder of the handler (the first one) to run with PSTATE.D clear. Once we enable single-step, we start stepping the debug handler, and will never step the original kprobe'd instruction.

Yes, that's correct. I saw the single stepping happens on right after recover the saved daif.

Quoted 33 lines

This is describing the most complicated way that this problem shows up! (I agree its also the worst)

I can get this to show up with just one kprobe. (function/file names here are meaningless):

| static int wibble(struct seq_file *m, void *discard)
| {
|        unsigned long d, flags;
|
|        flags = local_daif_save();
|
|        kprobe_me();
|        d = read_sysreg(daif);
|        local_daif_restore(flags);
|
|        seq_printf(m, "%lx\n", d);
|
|        return 0;
| }

plumbed into debugfs, then kicked using the kprobe_example module:
| root@adam:/sys/kernel/debug# cat wibble
| 3c0

| root@adam:/sys/kernel/debug# insmod ~morse/kprobe_example.ko symbol=kprobe_me
| [   69.478098] Planted kprobe at [..]
| root@adam:/sys/kernel/debug# cat wibble
| [   71.478935] <kprobe_me> pre_handler: p->addr = [..], pc = [..], pstate = 0x600003c5
| [   71.488942] <kprobe_me> post_handler: p->addr = [..], pstate = 0x600001c5
| 1c0
| root@adam:/sys/kernel/debug#
Quoted 1 line

This is problem for any code that had debug masked, not just kprobes.

Agreed.

Quoted 3 lines

Can we start the commit-message with the simplest description of the problem: kprobes manipulates the interrupted PSTATE for single step, and doesn't restore it.

Thanks for making it clearer :)

Quoted 14 lines

(trying to understand this bug through kprobe's interaction with itself is hard!)

Quoted 10 lines

To solve this issue, this stores all DAIF bits and restore it after single stepping.

arch/arm64/kernel/probes/kprobes.c +0 -0
index bd5dfffca272..348e02b799a2 100644
@@ -29,6 +29,8 @@
 
Quoted 5 lines

arch/arm64/kernel/probes/kprobes.c line 32

Quoted 3 lines
arch/arm64/kernel/probes/kprobes.c +1 -0
 #include "decode-insn.h"
 
+#define PSR_DAIF_MASK	(PSR_D_BIT | PSR_A_BIT | PSR_I_BIT | PSR_F_BIT)

We should probably move this to daifflags.h. Its going to be useful to other series too.

OK.

Quoted 7 lines

Patch looks good!

(I haven't tried to test the nested kprobes case...)

OK, I'll update and resend it.

Thank you!

Quoted 5 lines

Thanks,

James

Signature · 1 line
Masami Hiramatsu <[email protected]>
#4 lore raw

PATCH v2 2/4 arm64: unwind: Prohibit probing on return_address()

In reply to Masami Hiramatsu

Prohibit probing on return_address() and subroutines which is called from return_address(), since the it is invoked from trace_hardirqs_off() which is also kprobe blacklisted.

---
 arch/arm64/kernel/return_address.c |    4 +++-
 arch/arm64/kernel/stacktrace.c     |    3 +++
 2 files changed, 6 insertions(+), 1 deletion(-)
arch/arm64/kernel/return_address.c +3 -1
index b21cba90f82d..7f8a143268b0 100644
@@ -8,6 +8,7 @@
 
 #include <linux/export.h>
 #include <linux/ftrace.h>
+#include <linux/kprobes.h>
 
 #include <asm/stack_pointer.h>
 #include <asm/stacktrace.h>
@@ -17,7 +18,7 @@ struct return_address_data {
 	void *addr;
 };
 
-static int save_return_addr(struct stackframe *frame, void *d)
+static nokprobe_inline int save_return_addr(struct stackframe *frame, void *d)
 {
 	struct return_address_data *data = d;
 
@@ -52,3 +53,4 @@ void *return_address(unsigned int level)
 		return NULL;
 }
 EXPORT_SYMBOL_GPL(return_address);
+NOKPROBE_SYMBOL(return_address);
arch/arm64/kernel/stacktrace.c +3 -0
index 62d395151abe..cd7dab54d17b 100644
@@ -7,6 +7,7 @@
 #include <linux/kernel.h>
 #include <linux/export.h>
 #include <linux/ftrace.h>
+#include <linux/kprobes.h>
 #include <linux/sched.h>
 #include <linux/sched/debug.h>
 #include <linux/sched/task_stack.h>
@@ -73,6 +74,7 @@ int notrace unwind_frame(struct task_struct *tsk, struct stackframe *frame)
 
 	return 0;
 }
+NOKPROBE_SYMBOL(unwind_frame);
 
 void notrace walk_stackframe(struct task_struct *tsk, struct stackframe *frame,
 		     int (*fn)(struct stackframe *, void *), void *data)
@@ -87,6 +89,7 @@ void notrace walk_stackframe(struct task_struct *tsk, struct stackframe *frame,
 			break;
 	}
 }
+NOKPROBE_SYMBOL(walk_stackframe);
 
 #ifdef CONFIG_STACKTRACE
 struct stack_trace_data {
#5 lore raw

Re: [PATCH v2 2/4] arm64: unwind: Prohibit probing on return_address()

Hi,

Quoted 3 lines · Masami Hiramatsu

Prohibit probing on return_address() and subroutines which is called from return_address(), since the it is invoked from trace_hardirqs_off() which is also kprobe blacklisted.

(Nits: "which are called" and "since it is")

Quoted 13 lines
arch/arm64/kernel/return_address.c +1 -0
index b21cba90f82d..7f8a143268b0 100644
@@ -8,6 +8,7 @@
 
 #include <linux/export.h>
 #include <linux/ftrace.h>
+#include <linux/kprobes.h>
 
 #include <asm/stack_pointer.h>
 #include <asm/stacktrace.h>

arch/arm64/kernel/return_address.c struct return_address_data { · line 21

Quoted 5 lines
arch/arm64/kernel/return_address.c +1 -1
@@ -17,7 +18,7 @@ struct return_address_data {
 	void *addr;
 };
 
-static int save_return_addr(struct stackframe *frame, void *d)
+static nokprobe_inline int save_return_addr(struct stackframe *frame, void *d)

This nokprobe_inline ends up as __always_inline if kprobes is enabled. What do we expect the compiler to do with this? save_return_addr is passed as a function-pointer to walk_stackframe()... I don't see how the compiler can inline it!

This would be needed for on_accessible_stack().
Should we cover ftrace_graph_get_ret_stack()?, or is that already in hand?

arch/arm64/kernel/return_address.c void *return_address(unsigned int level) · line 56

Quoted 8 lines
arch/arm64/kernel/return_address.c +1 -0
 {
 	struct return_address_data *data = d;
 
@@ -52,3 +53,4 @@ void *return_address(unsigned int level)
 		return NULL;
 }
 EXPORT_SYMBOL_GPL(return_address);
+NOKPROBE_SYMBOL(return_address);

Thanks,

James

#6 lore raw

Re: [PATCH v2 2/4] arm64: unwind: Prohibit probing on return_address()

Quoted 8 lines · On Tue, 23 Jul 2019 17:04:21 +0100 James Morse <[email protected]> wrote:

Hi,

Quoted 3 lines · Masami Hiramatsu

Prohibit probing on return_address() and subroutines which is called from return_address(), since the it is invoked from trace_hardirqs_off() which is also kprobe blacklisted.

(Nits: "which are called" and "since it is")

Thanks!

Quoted 18 lines
Quoted 18 lines
arch/arm64/kernel/return_address.c +1 -0
index b21cba90f82d..7f8a143268b0 100644
@@ -8,6 +8,7 @@
 
 #include <linux/export.h>
 #include <linux/ftrace.h>
+#include <linux/kprobes.h>
 
 #include <asm/stack_pointer.h>
 #include <asm/stacktrace.h>
@@ -17,7 +18,7 @@ struct return_address_data {
 	void *addr;
 };
 
Quoted 6 lines

arch/arm64/kernel/return_address.c struct return_address_data { · line 21

Quoted 2 lines
arch/arm64/kernel/return_address.c +1 -1
-static int save_return_addr(struct stackframe *frame, void *d)
+static nokprobe_inline int save_return_addr(struct stackframe *frame, void *d)

This nokprobe_inline ends up as __always_inline if kprobes is enabled. What do we expect the compiler to do with this? save_return_addr is passed as a function-pointer to walk_stackframe()... I don't see how the compiler can inline it!

Oops, that's my mistake. Then it should be NOKPROBE_SYMBOL.

Quoted 3 lines

This would be needed for on_accessible_stack().
Should we cover ftrace_graph_get_ret_stack()?, or is that already in hand?

No, that is OK. It just covers that the functions which are involved in the kprobe execution path. ftrace_graph_ret_stack() is out of the debug exception handler.

Thank you,

Quoted 13 lines

arch/arm64/kernel/return_address.c void *return_address(unsigned int level) · line 56

Quoted 8 lines
arch/arm64/kernel/return_address.c +1 -0
 {
 	struct return_address_data *data = d;
 
@@ -52,3 +53,4 @@ void *return_address(unsigned int level)
 		return NULL;
 }
 EXPORT_SYMBOL_GPL(return_address);
+NOKPROBE_SYMBOL(return_address);

Thanks,

James

Signature · 1 line
Masami Hiramatsu <[email protected]>
#7 lore raw

PATCH v2 3/4 arm64: Make debug exception handlers visible from RCU

In reply to Masami Hiramatsu

Make debug exceptions visible from RCU so that synchronize_rcu() correctly track the debug exception handler.

This also introduces sanity checks for user-mode exceptions as same as x86's ist_enter()/ist_exit().

The debug exception can interrupt in idle task. For example, it warns if we put a kprobe on a function called from idle task as below. The warning message showed that the rcu_read_lock() caused this problem. But actually, this means the RCU is lost the context which is already in NMI/IRQ.

/sys/kernel/debug/tracing # echo p default_idle_call >> kprobe_events /sys/kernel/debug/tracing # echo 1 > events/kprobes/enable /sys/kernel/debug/tracing # [ 135.122237]

  [  135.125035] =============================
  [  135.125310] WARNING: suspicious RCU usage
  [  135.125581] 5.2.0-08445-g9187c508bdc7 #20 Not tainted
  [  135.125904] -----------------------------
  [  135.126205] include/linux/rcupdate.h:594 rcu_read_lock() used illegally while idle!
  [  135.126839]
  [  135.126839] other info that might help us debug this:
  [  135.126839]
  [  135.127410]
  [  135.127410] RCU used illegally from idle CPU!
  [  135.127410] rcu_scheduler_active = 2, debug_locks = 1
  [  135.128114] RCU used illegally from extended quiescent state!
  [  135.128555] 1 lock held by swapper/0/0:
  [  135.128944]  #0: (____ptrval____) (rcu_read_lock){....}, at: call_break_hook+0x0/0x178
  [  135.130499]
  [  135.130499] stack backtrace:
  [  135.131192] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.2.0-08445-g9187c508bdc7 #20
  [  135.131841] Hardware name: linux,dummy-virt (DT)
  [  135.132224] Call trace:
  [  135.132491]  dump_backtrace+0x0/0x140
  [  135.132806]  show_stack+0x24/0x30
  [  135.133133]  dump_stack+0xc4/0x10c
  [  135.133726]  lockdep_rcu_suspicious+0xf8/0x108
  [  135.134171]  call_break_hook+0x170/0x178
  [  135.134486]  brk_handler+0x28/0x68
  [  135.134792]  do_debug_exception+0x90/0x150
  [  135.135051]  el1_dbg+0x18/0x8c
  [  135.135260]  default_idle_call+0x0/0x44
  [  135.135516]  cpu_startup_entry+0x2c/0x30
  [  135.135815]  rest_init+0x1b0/0x280
  [  135.136044]  arch_call_rest_init+0x14/0x1c
  [  135.136305]  start_kernel+0x4d4/0x500
  [  135.136597]

So make debug exception visible to RCU can fix this warning.

---
 arch/arm64/mm/fault.c |   40 ++++++++++++++++++++++++++++++++++++++++
 1 file changed, 40 insertions(+)
arch/arm64/mm/fault.c +40 -0
index 9568c116ac7f..a6b244240db6 100644
@@ -777,6 +777,42 @@ void __init hook_debug_fault_code(int nr,
 	debug_fault_info[nr].name	= name;
 }
 
+/*
+ * In debug exception context, we explicitly disable preemption.
+ * This serves two purposes: it makes it much less likely that we would
+ * accidentally schedule in exception context and it will force a warning
+ * if we somehow manage to schedule by accident.
+ */
+static void debug_exception_enter(struct pt_regs *regs)
+{
+	if (user_mode(regs)) {
+		RCU_LOCKDEP_WARN(!rcu_is_watching(), "entry code didn't wake RCU");
+	} else {
+		/*
+		 * We might have interrupted pretty much anything.  In
+		 * fact, if we're a debug exception, we can even interrupt
+		 * NMI processing.  We don't want in_nmi() to return true,
+		 * but we need to notify RCU.
+		 */
+		rcu_nmi_enter();
+	}
+
+	preempt_disable();
+
+	/* This code is a bit fragile.  Test it. */
+	RCU_LOCKDEP_WARN(!rcu_is_watching(), "exception_enter didn't work");
+}
+NOKPROBE_SYMBOL(debug_exception_enter);
+
+static void debug_exception_exit(struct pt_regs *regs)
+{
+	preempt_enable_no_resched();
+
+	if (!user_mode(regs))
+		rcu_nmi_exit();
+}
+NOKPROBE_SYMBOL(debug_exception_exit);
+
 #ifdef CONFIG_ARM64_ERRATUM_1463225
 DECLARE_PER_CPU(int, __in_cortex_a76_erratum_1463225_wa);
 
@@ -824,6 +860,8 @@ asmlinkage void __exception do_debug_exception(unsigned long addr_if_watchpoint,
 	if (interrupts_enabled(regs))
 		trace_hardirqs_off();
 
+	debug_exception_enter(regs);
+
 	if (user_mode(regs) && !is_ttbr0_addr(pc))
 		arm64_apply_bp_hardening();
 
@@ -832,6 +870,8 @@ asmlinkage void __exception do_debug_exception(unsigned long addr_if_watchpoint,
 				 inf->sig, inf->code, (void __user *)pc, esr);
 	}
 
+	debug_exception_exit(regs);
+
 	if (interrupts_enabled(regs))
 		trace_hardirqs_on();
 }
Acked-by on v2 3/4
#8 lore raw

Re: [PATCH v2 3/4] arm64: Make debug exception handlers visible from RCU

Quoted 49 lines · Masami Hiramatsu

Make debug exceptions visible from RCU so that synchronize_rcu() correctly track the debug exception handler.

This also introduces sanity checks for user-mode exceptions as same as x86's ist_enter()/ist_exit().

The debug exception can interrupt in idle task. For example, it warns if we put a kprobe on a function called from idle task as below. The warning message showed that the rcu_read_lock() caused this problem. But actually, this means the RCU is lost the context which is already in NMI/IRQ.

/sys/kernel/debug/tracing # echo p default_idle_call >> kprobe_events /sys/kernel/debug/tracing # echo 1 > events/kprobes/enable /sys/kernel/debug/tracing # [ 135.122237]

  [  135.125035] =============================
  [  135.125310] WARNING: suspicious RCU usage
  [  135.125581] 5.2.0-08445-g9187c508bdc7 #20 Not tainted
  [  135.125904] -----------------------------
  [  135.126205] include/linux/rcupdate.h:594 rcu_read_lock() used illegally while idle!
  [  135.126839]
  [  135.126839] other info that might help us debug this:
  [  135.126839]
  [  135.127410]
  [  135.127410] RCU used illegally from idle CPU!
  [  135.127410] rcu_scheduler_active = 2, debug_locks = 1
  [  135.128114] RCU used illegally from extended quiescent state!
  [  135.128555] 1 lock held by swapper/0/0:
  [  135.128944]  #0: (____ptrval____) (rcu_read_lock){....}, at: call_break_hook+0x0/0x178
  [  135.130499]
  [  135.130499] stack backtrace:
  [  135.131192] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.2.0-08445-g9187c508bdc7 #20
  [  135.131841] Hardware name: linux,dummy-virt (DT)
  [  135.132224] Call trace:
  [  135.132491]  dump_backtrace+0x0/0x140
  [  135.132806]  show_stack+0x24/0x30
  [  135.133133]  dump_stack+0xc4/0x10c
  [  135.133726]  lockdep_rcu_suspicious+0xf8/0x108
  [  135.134171]  call_break_hook+0x170/0x178
  [  135.134486]  brk_handler+0x28/0x68
  [  135.134792]  do_debug_exception+0x90/0x150
  [  135.135051]  el1_dbg+0x18/0x8c
  [  135.135260]  default_idle_call+0x0/0x44
  [  135.135516]  cpu_startup_entry+0x2c/0x30
  [  135.135815]  rest_init+0x1b0/0x280
  [  135.136044]  arch_call_rest_init+0x14/0x1c
  [  135.136305]  start_kernel+0x4d4/0x500
  [  135.136597]
Quoted 4 lines

So make debug exception visible to RCU can fix this warning.

From an RCU viewpoint:

Quoted 71 lines
---
 arch/arm64/mm/fault.c |   40 ++++++++++++++++++++++++++++++++++++++++
 1 file changed, 40 insertions(+)
arch/arm64/mm/fault.c +40 -0
index 9568c116ac7f..a6b244240db6 100644
@@ -777,6 +777,42 @@ void __init hook_debug_fault_code(int nr,
 	debug_fault_info[nr].name	= name;
 }
 
+/*
+ * In debug exception context, we explicitly disable preemption.
+ * This serves two purposes: it makes it much less likely that we would
+ * accidentally schedule in exception context and it will force a warning
+ * if we somehow manage to schedule by accident.
+ */
+static void debug_exception_enter(struct pt_regs *regs)
+{
+	if (user_mode(regs)) {
+		RCU_LOCKDEP_WARN(!rcu_is_watching(), "entry code didn't wake RCU");
+	} else {
+		/*
+		 * We might have interrupted pretty much anything.  In
+		 * fact, if we're a debug exception, we can even interrupt
+		 * NMI processing.  We don't want in_nmi() to return true,
+		 * but we need to notify RCU.
+		 */
+		rcu_nmi_enter();
+	}
+
+	preempt_disable();
+
+	/* This code is a bit fragile.  Test it. */
+	RCU_LOCKDEP_WARN(!rcu_is_watching(), "exception_enter didn't work");
+}
+NOKPROBE_SYMBOL(debug_exception_enter);
+
+static void debug_exception_exit(struct pt_regs *regs)
+{
+	preempt_enable_no_resched();
+
+	if (!user_mode(regs))
+		rcu_nmi_exit();
+}
+NOKPROBE_SYMBOL(debug_exception_exit);
+
 #ifdef CONFIG_ARM64_ERRATUM_1463225
 DECLARE_PER_CPU(int, __in_cortex_a76_erratum_1463225_wa);
 
@@ -824,6 +860,8 @@ asmlinkage void __exception do_debug_exception(unsigned long addr_if_watchpoint,
 	if (interrupts_enabled(regs))
 		trace_hardirqs_off();
 
+	debug_exception_enter(regs);
+
 	if (user_mode(regs) && !is_ttbr0_addr(pc))
 		arm64_apply_bp_hardening();
 
@@ -832,6 +870,8 @@ asmlinkage void __exception do_debug_exception(unsigned long addr_if_watchpoint,
 				 inf->sig, inf->code, (void __user *)pc, esr);
 	}
 
+	debug_exception_exit(regs);
+
 	if (interrupts_enabled(regs))
 		trace_hardirqs_on();
 }
#9 lore raw

Re: [PATCH v2 3/4] arm64: Make debug exception handlers visible from RCU

In reply to Masami Hiramatsu

Hi,

Quoted 22 lines · Masami Hiramatsu

Make debug exceptions visible from RCU so that synchronize_rcu() correctly track the debug exception handler.

This also introduces sanity checks for user-mode exceptions as same as x86's ist_enter()/ist_exit().

The debug exception can interrupt in idle task. For example, it warns if we put a kprobe on a function called from idle task as below. The warning message showed that the rcu_read_lock() caused this problem. But actually, this means the RCU is lost the context which is already in NMI/IRQ.

So make debug exception visible to RCU can fix this warning.

arch/arm64/mm/fault.c +0 -0
index 9568c116ac7f..a6b244240db6 100644
@@ -777,6 +777,42 @@ void __init hook_debug_fault_code(int nr,
 	debug_fault_info[nr].name	= name;
 }
 

arch/arm64/mm/fault.c void __init hook_debug_fault_code(int nr, · line 789

Quoted 10 lines
arch/arm64/mm/fault.c +10 -0
+/*
+ * In debug exception context, we explicitly disable preemption.
+ * This serves two purposes: it makes it much less likely that we would
+ * accidentally schedule in exception context and it will force a warning
+ * if we somehow manage to schedule by accident.
+ */
+static void debug_exception_enter(struct pt_regs *regs)
+{
+	if (user_mode(regs)) {
+		RCU_LOCKDEP_WARN(!rcu_is_watching(), "entry code didn't wake RCU");

Would moving entry.S's context_tracking_user_exit() call to be before do_debug_exception() also fix this?

I don't know the reason its done 'after' debug exception handling. Its always been like
this: commit 6c81fe7925cc4c42 ("arm64: enable context tracking").

arch/arm64/mm/fault.c void __init hook_debug_fault_code(int nr,

Quoted 8 lines
arch/arm64/mm/fault.c +7 -0
+	} else {
+		/*
+		 * We might have interrupted pretty much anything.  In
+		 * fact, if we're a debug exception, we can even interrupt
+		 * NMI processing.
 
+		 * We don't want in_nmi() to return true,
+		 * but we need to notify RCU.

How come? If you interrupted an SError or pseudo-nmi, it already is. Those paths should all be painted no-kprobe, but I'm sure there are gaps. The hw-breakpoints can almost certainly hook them.

arch/arm64/mm/fault.c void __init hook_debug_fault_code(int nr, · line 799

Quoted 2 lines
arch/arm64/mm/fault.c +2 -0
+		 */
+		rcu_nmi_enter();

Can we interrupt printk()? Do we need printk_nmi_enter()? ... What about ftrace?

Because SError and pseudo-nmi can interrupt interrupt-masked code, we describe them as NMI. The only difference here is these exceptions are synchronous.

I suspect we should make these debug exceptions nmi for EL1. We can then use this for the kprobe-re-entrance stuff so the pre/post hooks don't get run if they interrupted something also described as NMI.

arch/arm64/mm/fault.c void __init hook_debug_fault_code(int nr, · line 805

Quoted 8 lines
arch/arm64/mm/fault.c +8 -0
+	}
+
+	preempt_disable();
+
+	/* This code is a bit fragile.  Test it. */
+	RCU_LOCKDEP_WARN(!rcu_is_watching(), "exception_enter didn't work");
+}
+NOKPROBE_SYMBOL(debug_exception_enter);

Thanks,

James

#10 lore raw

Re: [PATCH v2 3/4] arm64: Make debug exception handlers visible from RCU

Quoted 25 lines · On Tue, 23 Jul 2019 18:07:56 +0100 James Morse <[email protected]> wrote:

Hi,

Quoted 22 lines · Masami Hiramatsu

Make debug exceptions visible from RCU so that synchronize_rcu() correctly track the debug exception handler.

This also introduces sanity checks for user-mode exceptions as same as x86's ist_enter()/ist_exit().

The debug exception can interrupt in idle task. For example, it warns if we put a kprobe on a function called from idle task as below. The warning message showed that the rcu_read_lock() caused this problem. But actually, this means the RCU is lost the context which is already in NMI/IRQ.

So make debug exception visible to RCU can fix this warning.

arch/arm64/mm/fault.c +0 -0
index 9568c116ac7f..a6b244240db6 100644
@@ -777,6 +777,42 @@ void __init hook_debug_fault_code(int nr,
 	debug_fault_info[nr].name	= name;
 }
 
Quoted 13 lines

arch/arm64/mm/fault.c void __init hook_debug_fault_code(int nr, · line 789

Quoted 10 lines
arch/arm64/mm/fault.c +10 -0
+/*
+ * In debug exception context, we explicitly disable preemption.
+ * This serves two purposes: it makes it much less likely that we would
+ * accidentally schedule in exception context and it will force a warning
+ * if we somehow manage to schedule by accident.
+ */
+static void debug_exception_enter(struct pt_regs *regs)
+{
+	if (user_mode(regs)) {
+		RCU_LOCKDEP_WARN(!rcu_is_watching(), "entry code didn't wake RCU");

Would moving entry.S's context_tracking_user_exit() call to be before do_debug_exception() also fix this?

It sounds like treating only user context, correct?
This part is just adding assertion, not fixing the problem which Naresh reported.

Quoted 11 lines

I don't know the reason its done 'after' debug exception handling. Its always been like
this: commit 6c81fe7925cc4c42 ("arm64: enable context tracking").

Quoted 5 lines
arch/arm64/mm/fault.c +5 -0
+	} else {
+		/*
+		 * We might have interrupted pretty much anything.  In
+		 * fact, if we're a debug exception, we can even interrupt
+		 * NMI processing.
Quoted 6 lines

arch/arm64/mm/fault.c void __init hook_debug_fault_code(int nr, · line 796

Quoted 2 lines
arch/arm64/mm/fault.c +2 -0
+		 * We don't want in_nmi() to return true,
+		 * but we need to notify RCU.

How come? If you interrupted an SError or pseudo-nmi, it already is. Those paths should all be painted no-kprobe, but I'm sure there are gaps. The hw-breakpoints can almost certainly hook them.

I think that sentense means "we don't want that this code makes in_nmi() to return true" So, if the breakpoint interrupts pNMI/SError context, it is OK that in_nmi() returns true.

Quoted 6 lines

arch/arm64/mm/fault.c void __init hook_debug_fault_code(int nr, · line 798

Quoted 4 lines
arch/arm64/mm/fault.c +2 -0
+		 */
+		rcu_nmi_enter();

Can we interrupt printk()? Do we need printk_nmi_enter()? ... What about ftrace?

Good point! As far as I know, we don't use it because ftrace doesn't use printk. But indeed, kprobes user can use printk and they have to call printk_nmi_enter()/exit(), that must be commented in the documentation. Anyway, basically it is user's choice.

Quoted 8 lines

Because SError and pseudo-nmi can interrupt interrupt-masked code, we describe them as NMI. The only difference here is these exceptions are synchronous.

I suspect we should make these debug exceptions nmi for EL1. We can then use this for the kprobe-re-entrance stuff so the pre/post hooks don't get run if they interrupted something also described as NMI.

I'm not sure how it can prevent... anyway because we have to run a single-stepping for recovery, and kprobe already check the reentered kprobes and skip user-handlers in such case.

Thank you,

Quoted 15 lines

arch/arm64/mm/fault.c void __init hook_debug_fault_code(int nr, · line 805

Quoted 10 lines
arch/arm64/mm/fault.c +8 -0
+	}
+
+	preempt_disable();
+
+	/* This code is a bit fragile.  Test it. */
+	RCU_LOCKDEP_WARN(!rcu_is_watching(), "exception_enter didn't work");
+}
+NOKPROBE_SYMBOL(debug_exception_enter);

Thanks,

James

Signature · 1 line
Masami Hiramatsu <[email protected]>
#11 lore raw

PATCH v2 4/4 arm64: Remove unneeded rcu_read_lock from debug handlers

In reply to Masami Hiramatsu

Remove rcu_read_lock()/rcu_read_unlock() from debug exception handlers since we are sure those are not preemptible and interrupts are off.

---
 arch/arm64/kernel/debug-monitors.c |   14 ++++++++------
 1 file changed, 8 insertions(+), 6 deletions(-)
arch/arm64/kernel/debug-monitors.c +8 -6
index f8719bd30850..48222a4760c2 100644
@@ -207,16 +207,16 @@ static int call_step_hook(struct pt_regs *regs, unsigned int esr)
 
 	list = user_mode(regs) ? &user_step_hook : &kernel_step_hook;
 
-	rcu_read_lock();
-
+	/*
+	 * Since single-step exception disables interrupt, this function is
+	 * entirely not preemptible, and we can use rcu list safely here.
+	 */
 	list_for_each_entry_rcu(hook, list, node)	{
 		retval = hook->fn(regs, esr);
 		if (retval == DBG_HOOK_HANDLED)
 			break;
 	}
 
-	rcu_read_unlock();
-
 	return retval;
 }
 NOKPROBE_SYMBOL(call_step_hook);
@@ -305,14 +305,16 @@ static int call_break_hook(struct pt_regs *regs, unsigned int esr)
 
 	list = user_mode(regs) ? &user_break_hook : &kernel_break_hook;
 
-	rcu_read_lock();
+	/*
+	 * Since brk exception disables interrupt, this function is
+	 * entirely not preemptible, and we can use rcu list safely here.
+	 */
 	list_for_each_entry_rcu(hook, list, node) {
 		unsigned int comment = esr & ESR_ELx_BRK64_ISS_COMMENT_MASK;
 
 		if ((comment & ~hook->mask) == hook->imm)
 			fn = hook->fn;
 	}
-	rcu_read_unlock();
 
 	return fn ? fn(regs, esr) : DBG_HOOK_ERROR;
 }
Acked-by on v2 4/4
#12 lore raw

Re: [PATCH v2 4/4] arm64: Remove unneeded rcu_read_lock from debug handlers

Quoted 6 lines · Masami Hiramatsu

Remove rcu_read_lock()/rcu_read_unlock() from debug exception handlers since we are sure those are not preemptible and interrupts are off.

From an RCU viewpoint:

Quoted 49 lines
---
 arch/arm64/kernel/debug-monitors.c |   14 ++++++++------
 1 file changed, 8 insertions(+), 6 deletions(-)
arch/arm64/kernel/debug-monitors.c +8 -6
index f8719bd30850..48222a4760c2 100644
@@ -207,16 +207,16 @@ static int call_step_hook(struct pt_regs *regs, unsigned int esr)
 
 	list = user_mode(regs) ? &user_step_hook : &kernel_step_hook;
 
-	rcu_read_lock();
-
+	/*
+	 * Since single-step exception disables interrupt, this function is
+	 * entirely not preemptible, and we can use rcu list safely here.
+	 */
 	list_for_each_entry_rcu(hook, list, node)	{
 		retval = hook->fn(regs, esr);
 		if (retval == DBG_HOOK_HANDLED)
 			break;
 	}
 
-	rcu_read_unlock();
-
 	return retval;
 }
 NOKPROBE_SYMBOL(call_step_hook);
@@ -305,14 +305,16 @@ static int call_break_hook(struct pt_regs *regs, unsigned int esr)
 
 	list = user_mode(regs) ? &user_break_hook : &kernel_break_hook;
 
-	rcu_read_lock();
+	/*
+	 * Since brk exception disables interrupt, this function is
+	 * entirely not preemptible, and we can use rcu list safely here.
+	 */
 	list_for_each_entry_rcu(hook, list, node) {
 		unsigned int comment = esr & ESR_ELx_BRK64_ISS_COMMENT_MASK;
 
 		if ((comment & ~hook->mask) == hook->imm)
 			fn = hook->fn;
 	}
-	rcu_read_unlock();
 
 	return fn ? fn(regs, esr) : DBG_HOOK_ERROR;
 }