riscv: entry: Fixup do_trap_break from kernel side
Subsystems (3)
PATCH 1/1 riscv: entry: Fixup do_trap_break from kernel side
From: Guo Ren <[email protected]>
The irqentry_nmi_enter/exit would force the current context into in_interrupt. That would trigger the kernel to dead panic, but the kdb still needs "ebreak" to debug the kernel.
Move irqentry_nmi_enter/exit to exception_enter/exit could correct handle_break of the kernel side.
Before the fixup:
$echo BUG > /sys/kernel/debug/provoke-crash/DIRECT
lkdtm: Performing direct entry BUG
------------[ cut here ]------------ kernel BUG at drivers/misc/lkdtm/bugs.c:78! handle_break, 256. Kernel BUG [#1] Modules linked in: CPU: 0 PID: 104 Comm: echo Not tainted 6.4.0-rc1-00055-g0ca05a4b079f-dirty #30 Hardware name: riscv-virtio,qemu (DT) epc : lkdtm_BUG+0x6/0x8 ra : lkdtm_do_action+0x14/0x1c epc : ffffffff8055c730 ra : ffffffff8087e188 sp : ff200000007dbd40 gp : ffffffff81500878 tp : ff600000028ebac0 t0 : 6500000000000000 t1 : 000000000000006c t2 : 6550203a6d74646b s0 : ff200000007dbd50 s1 : ffffffff814bfc80 a0 : ffffffff814bfc80 a1 : ff6000001ffd8608 a2 : ff6000001ffdb870 a3 : 0000000000000000 a4 : 0000000000000000 a5 : ffffffff8055c72a a6 : 0000000000000032 a7 : 0000000000000038 s2 : 0000000000000004 s3 : 00000000556371a0 s4 : ff200000007dbe70 s5 : ff60000002090000 s6 : 00000000556371a0 s7 : 0000000000000030 s8 : 000000007fffec78 s9 : 0000000000000007 s10: 0000000055637530 s11: 0000000000000001 t3 : ffffffff81513ed7 t4 : ffffffff81513ed7 t5 : ffffffff81513ed8 t6 : ff200000007dbb88 status: 0000000100000120 badaddr: 0000000000000000 cause: 0000000000000003 [<ffffffff8055c730>] lkdtm_BUG+0x6/0x8 Code: 0513 6b05 b097 0031 80e7 e960 b705 1141 e422 0800 (9002) 1141 ---[ end trace 0000000000000000 ]--- Kernel panic - not syncing: Aiee, killing interrupt handler! ---[ end Kernel panic - not syncing: Aiee, killing interrupt handler! ]---
(Dead in the kernel side.)
After the fixup:
$echo BUG > /sys/kernel/debug/provoke-crash/DIRECT
lkdtm: Performing direct entry BUG
------------[ cut here ]------------ kernel BUG at drivers/misc/lkdtm/bugs.c:78! Kernel BUG [#13] Modules linked in: CPU: 0 PID: 129 Comm: echo Tainted: G D 6.4.0-rc1-00055-g0ca05a4b079f-dirty #34 Hardware name: riscv-virtio,qemu (DT) epc : lkdtm_BUG+0x6/0x8 ra : lkdtm_do_action+0x14/0x1c epc : ffffffff8055c71c ra : ffffffff8087e170 sp : ff200000007e3d40 gp : ffffffff81500878 tp : ff600000028ebac0 t0 : 6500000000000000 t1 : 000000000000006c t2 : 6550203a6d74646b s0 : ff200000007e3d50 s1 : ffffffff814bfc80 a0 : ffffffff814bfc80 a1 : ff6000001ffd8608 a2 : ff6000001ffdb870 a3 : 0000000000000000 a4 : 0000000000000000 a5 : ffffffff8055c716 a6 : 0000000000000032 a7 : 0000000000000038 s2 : 0000000000000004 s3 : 00000000556371a0 s4 : ff200000007e3e70 s5 : ff60000002090000 s6 : 00000000556371a0 s7 : 0000000000000030 s8 : 000000007fffec78 s9 : 0000000000000007 s10: 0000000055637530 s11: 0000000000000001 t3 : ffffffff81513ed7 t4 : ffffffff81513ed7 t5 : ffffffff81513ed8 t6 : ff200000007e3b88 status: 0000000100000120 badaddr: 0000000000000000 cause: 0000000000000003 [<ffffffff8055c71c>] lkdtm_BUG+0x6/0x8 Code: 0513 6945 b097 0031 80e7 e920 b705 1141 e422 0800 (9002) 1141 ---[ end trace 0000000000000000 ]--- note: echo[129] exited with irqs disabled Segmentation fault
(Resume to the shell normally.)
--- arch/riscv/kernel/traps.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
arch/riscv/kernel/traps.c +3 -2
@@ -18,6 +18,7 @@#include <linux/irq.h> #include <linux/kexec.h> #include <linux/entry-common.h> +#include <linux/context_tracking.h> #include <asm/asm-prototypes.h> #include <asm/bug.h>
@@ -257,11 +258,11 @@ asmlinkage __visible __trap_section void do_trap_break(struct pt_regs *regs)irqentry_exit_to_user_mode (regs); }else { -irqentry_state_t state =irqentry_nmi_enter (regs); +enum ctx_state prev_state =exception_enter ();handle_break (regs); -irqentry_nmi_exit (regs, state); +exception_exit (prev_state); } }
Signature · 1 line
2.36.1
Quoted 8 lines · On Sat, Jul 01, 2023 at 10:57:07PM -0400, [email protected] wrote:
From: Guo Ren <[email protected]>
The irqentry_nmi_enter/exit would force the current context into
in_interrupt. That would trigger the kernel to dead panic, but the kdb still needs "ebreak" to debug the kernel.Move irqentry_nmi_enter/exit to exception_enter/exit could correct
handle_breakof the kernel side.
<snip>
Quoted 5 lines
Fixes f0bddf50586d ("riscv: entry: Convert to generic entry")Reported-by Daniel Thompson <[email protected]>Signed-off-by Guo Ren <[email protected]>Signed-off-by Guo Ren <[email protected]>
I pushed this though the kgdb test suite that originally found the problem (although it didn't occur to me when I reported it that the problem was nothing to do with kgdb ;-) ). So FWIW:
Daniel.
_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
Quoted 19 lines · Daniel Thompson
Quoted 8 lines · On Sat, Jul 01, 2023 at 10:57:07PM -0400, [email protected] wrote:
From: Guo Ren <[email protected]>
The irqentry_nmi_enter/exit would force the current context into
in_interrupt. That would trigger the kernel to dead panic, but the kdb still needs "ebreak" to debug the kernel.Move irqentry_nmi_enter/exit to exception_enter/exit could correct
handle_breakof the kernel side.<snip>
Quoted 5 lines
Fixes f0bddf50586d ("riscv: entry: Convert to generic entry")Reported-by Daniel Thompson <[email protected]>Signed-off-by Guo Ren <[email protected]>Signed-off-by Guo Ren <[email protected]>
Quoted 5 lines
I pushed this though the kgdb test suite that originally found the problem (although it didn't occur to me when I reported it that the problem was nothing to do with kgdb ;-) ). So FWIW:
Tested-by Daniel Thompson <[email protected]>
Thx for the report & tested-by.
Quoted 3 lines
Daniel.
Signature · 2 lines
Best Regards Guo Ren
Quoted 8 lines · On Sat, Jul 01, 2023 at 10:57:07PM -0400, [email protected] wrote:
From: Guo Ren <[email protected]>
The irqentry_nmi_enter/exit would force the current context into
in_interrupt. That would trigger the kernel to dead panic, but the kdb still needs "ebreak" to debug the kernel.Move irqentry_nmi_enter/exit to exception_enter/exit could correct
handle_breakof the kernel side.
This doesn't explain much if anything :/
I'm confused (probably because I don't know RISC-V very well), what's EBREAK and how does it happen?
Specifically, if EBREAK can happen inside an local_irq_disable() region, then the below change is actively wrong. Any exception/interrupt that can happen while local_irq_disable() must be treated like an NMI.
If that makes kdb unhappy, fix kdb.
Quoted 38 lines
Fixes f0bddf50586d ("riscv: entry: Convert to generic entry")Reported-by Daniel Thompson <[email protected]>Signed-off-by Guo Ren <[email protected]>Signed-off-by Guo Ren <[email protected]>--- arch/riscv/kernel/traps.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)arch/riscv/kernel/traps.c +3
-2@@ -18,6 +18,7 @@#include <linux/irq.h> #include <linux/kexec.h> #include <linux/entry-common.h> +#include <linux/context_tracking.h> #include <asm/asm-prototypes.h> #include <asm/bug.h> @@ -257,11 +258,11 @@ asmlinkage __visible __trap_section void do_trap_break(struct pt_regs *regs)irqentry_exit_to_user_mode (regs); }else { -irqentry_state_t state =irqentry_nmi_enter (regs); +enum ctx_state prev_state =exception_enter ();handle_break (regs); -irqentry_nmi_exit (regs, state); +exception_exit (prev_state); } }Signature · 1 line
2.36.1
_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
Quoted 7 lines · Peter Zijlstra
Quoted 6 lines · On Sat, Jul 01, 2023 at 10:57:07PM -0400, [email protected] wrote:
From: Guo Ren <[email protected]>
The irqentry_nmi_enter/exit would force the current context into
in_interrupt. That would trigger the kernel to dead panic, but the kdb still needs "ebreak" to debug the kernel.
Quoted 7 lines
Quoted 2 lines
Move irqentry_nmi_enter/exit to exception_enter/exit could correct
handle_breakof the kernel side.This doesn't explain much if anything :/
I'm confused (probably because I don't know RISC-V very well), what's EBREAK and how does it happen?
Among other things ebreak is part of the BUG() macro (although it is also used to programmatically enter kgdb).
Quoted 5 lines
Specifically, if EBREAK can happen inside an
local_irq_disable()region, then the below change is actively wrong. Any exception/interrupt that can happen whilelocal_irq_disable()must be treated like an NMI.If that makes kdb unhappy, fix kdb.
The only relationship this problem has to kgdb/kdb is that is was found using the kgdb test suite. However the panic is absolutely nothing to do with kgdb.
I would never normally be so sure regarding the absence of bugs in kgdb but in this case it can be reproduced when kgdb is not enabled in the KConfig which I think puts it in the clear!
Reproduction is simply:
/bin/echo BUG > /sys/kernel/debug/provoke-crash/DIRECT
Above will panic the kernel but, absent options specifically requesting a panic, this should kill the echo process rather than killing the kernel.
Daniel.
_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
Quoted 8 lines · Peter Zijlstra
Quoted 6 lines · On Sat, Jul 01, 2023 at 10:57:07PM -0400, [email protected] wrote:
From: Guo Ren <[email protected]>
The irqentry_nmi_enter/exit would force the current context into
in_interrupt. That would trigger the kernel to dead panic, but the kdb still needs "ebreak" to debug the kernel.
Quoted 7 lines
Quoted 2 lines
Move irqentry_nmi_enter/exit to exception_enter/exit could correct
handle_breakof the kernel side.This doesn't explain much if anything :/
I'm confused (probably because I don't know RISC-V very well), what's EBREAK and how does it happen?
EBREAK is just an instruction of riscv which would rise breakpoint exception.
Quoted 4 lines
Specifically, if EBREAK can happen inside an
local_irq_disable()region, then the below change is actively wrong. Any exception/interrupt that can happen whilelocal_irq_disable()must be treated like an NMI.
When the ebreak happend out of local_irq_disable region, but __nmi_enter forces handle_break() into in_interupt() state. So how about:
arch/riscv/kernel/traps.c +1 -0
@@ -18,6 +18,7 @@#include <linux/irq.h> #include <linux/kexec.h> #include <linux/entry-common.h> +#include <linux/context_tracking.h> #include <asm/asm-prototypes.h> #include <asm/bug.h>
@@ -285,12 +286,18 @@ asmlinkage __visible __trap_section void
do_trap_break(struct pt_regs *regs)
handle_break(regs);
irqentry_exit_to_user_mode(regs);
- } else {
+ } else if (in_interrupt()){
irqentry_state_t state = irqentry_nmi_enter(regs);
handle_break(regs);
irqentry_nmi_exit(regs, state);+ } else {
+ enum ctx_state prev_state = exception_enter();
+
+ handle_break(regs);
+
+ exception_exit(prev_state);
}}
Quoted 33 lines
If that makes kdb unhappy, fix kdb.
Quoted 30 lines
Fixes f0bddf50586d ("riscv: entry: Convert to generic entry")Reported-by Daniel Thompson <[email protected]>Signed-off-by Guo Ren <[email protected]>Signed-off-by Guo Ren <[email protected]>--- arch/riscv/kernel/traps.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)arch/riscv/kernel/traps.c +2
-1@@ -18,6 +18,7 @@#include <linux/irq.h> #include <linux/kexec.h> #include <linux/entry-common.h> +#include <linux/context_tracking.h> #include <asm/asm-prototypes.h> #include <asm/bug.h> @@ -257,11 +258,11 @@ asmlinkage __visible __trap_section void do_trap_break(struct pt_regs *regs)irqentry_exit_to_user_mode (regs); }else { -irqentry_state_t state =irqentry_nmi_enter (regs); +enum ctx_state prev_state =exception_enter ();handle_break (regs);
Quoted 8 lines
Quoted 8 lines
arch/riscv/kernel/traps.c +1
-2-irqentry_nmi_exit (regs, state); +exception_exit (prev_state); } } --2.36.1
--
Best Regards
Guo Ren
Quoted 20 lines · Guo Ren
Quoted 8 lines · Peter Zijlstra
Quoted 6 lines · On Sat, Jul 01, 2023 at 10:57:07PM -0400, [email protected] wrote:
From: Guo Ren <[email protected]>
The irqentry_nmi_enter/exit would force the current context into
in_interrupt. That would trigger the kernel to dead panic, but the kdb still needs "ebreak" to debug the kernel.Quoted 7 lines
Quoted 2 lines
Move irqentry_nmi_enter/exit to exception_enter/exit could correct
handle_breakof the kernel side.This doesn't explain much if anything :/
I'm confused (probably because I don't know RISC-V very well), what's EBREAK and how does it happen?
EBREAK is just an instruction of riscv which would rise breakpoint exception.
Quoted 1 line
Quoted 5 lines
Quoted 3 lines
Specifically, if EBREAK can happen inside an
local_irq_disable()region, then the below change is actively wrong. Any exception/interrupt that can happen whilelocal_irq_disable()must be treated like an NMI.When the ebreak happend out of
local_irq_disableregion, but__nmi_enterforceshandle_break()intoin_interupt()state. So how
And why is that a problem? I think I'm missing something fundamental here...
Quoted 24 lines
about:
arch/riscv/kernel/traps.c +2
-1@@ -18,6 +18,7 @@#include <linux/irq.h> #include <linux/kexec.h> #include <linux/entry-common.h> +#include <linux/context_tracking.h> #include <asm/asm-prototypes.h> #include <asm/bug.h> @@ -285,12 +286,18 @@ asmlinkage __visible __trap_section void do_trap_break(struct pt_regs *regs)handle_break (regs);irqentry_exit_to_user_mode (regs); - }else { + }else if (in_interrupt ()){irqentry_state_t state =irqentry_nmi_enter (regs);handle_break (regs);
arch/riscv/kernel/traps.c asmlinkage __visible __trap_section void · line 303
Quoted 10 lines
arch/riscv/kernel/traps.c +6
-0irqentry_nmi_exit (regs, state); + }else { +enum ctx_state prev_state =exception_enter (); + +handle_break (regs); + +exception_exit (prev_state); } }
That's wrong. If you want to make it conditional, you have to look at !(regs->status & SR_IE) (that's the interrupt enable flag of the interrupted context, right?)
When you hit an EBREAK when IRQs were disabled, you must be NMI like.
But making it conditional like this makes it really hard to write a handler though, it basically must assume it will be NMI contetx (because it can't know) so there is no point in sometimes not doing NMI context.
_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
Quoted 22 lines · Peter Zijlstra
Quoted 20 lines · Guo Ren
Quoted 8 lines · Peter Zijlstra
Quoted 6 lines · On Sat, Jul 01, 2023 at 10:57:07PM -0400, [email protected] wrote:
From: Guo Ren <[email protected]>
The irqentry_nmi_enter/exit would force the current context into
in_interrupt. That would trigger the kernel to dead panic, but the kdb still needs "ebreak" to debug the kernel.Quoted 7 lines
Quoted 2 lines
Move irqentry_nmi_enter/exit to exception_enter/exit could correct
handle_breakof the kernel side.This doesn't explain much if anything :/
I'm confused (probably because I don't know RISC-V very well), what's EBREAK and how does it happen?
EBREAK is just an instruction of riscv which would rise breakpoint exception.
Quoted 1 line
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
Specifically, if EBREAK can happen inside an
local_irq_disable()region, then the below change is actively wrong. Any exception/interrupt that can happen whilelocal_irq_disable()must be treated like an NMI.When the ebreak happend out of
local_irq_disableregion, but__nmi_enterforceshandle_break()intoin_interupt()state. So howAnd why is that a problem? I think I'm missing something fundamental here...
The irqentry_nmi_enter() would force the current context to get in_interrupt=true, although ebreak happens in the context which is in_interrupt=false.
A lot of checking codes, such as:
if (in_interrupt())
panic("Fatal exception in interrupt");It would make the kernel panic, but we don't panic; we want back to the shell. eg:
echo BUG > /sys/kernel/debug/provoke-crash/DIRECT
Quoted 45 lines
Quoted 25 lines
about:
arch/riscv/kernel/traps.c +2
-1@@ -18,6 +18,7 @@#include <linux/irq.h> #include <linux/kexec.h> #include <linux/entry-common.h> +#include <linux/context_tracking.h> #include <asm/asm-prototypes.h> #include <asm/bug.h> @@ -285,12 +286,18 @@ asmlinkage __visible __trap_section void do_trap_break(struct pt_regs *regs)handle_break (regs);irqentry_exit_to_user_mode (regs); - }else { + }else if (in_interrupt ()){irqentry_state_t state =irqentry_nmi_enter (regs);handle_break (regs);arch/riscv/kernel/traps.c asmlinkage __visible __trap_section void · line 303
Quoted 10 lines
arch/riscv/kernel/traps.c +6
-0irqentry_nmi_exit (regs, state); + }else { +enum ctx_state prev_state =exception_enter (); + +handle_break (regs); + +exception_exit (prev_state); } }That's wrong. If you want to make it conditional, you have to look at !(regs->status &
SR_IE) (that's the interrupt enable flag of the interrupted context, right?)When you hit an EBREAK when IRQs were disabled, you must be NMI like.
But making it conditional like this makes it really hard to write a handler though, it basically must assume it will be NMI contetx (because it can't know) so there is no point in sometimes not doing NMI context.
Signature · 2 lines
Best Regards Guo Ren
Quoted 29 lines · Guo Ren
Quoted 28 lines · Peter Zijlstra
Quoted 25 lines · Guo Ren
Quoted 8 lines · Peter Zijlstra
Quoted 6 lines · On Sat, Jul 01, 2023 at 10:57:07PM -0400, [email protected] wrote:
From: Guo Ren <[email protected]>
The irqentry_nmi_enter/exit would force the current context into
in_interrupt. That would trigger the kernel to dead panic, but the kdb still needs "ebreak" to debug the kernel.Quoted 7 lines
Quoted 2 lines
Move irqentry_nmi_enter/exit to exception_enter/exit could correct
handle_breakof the kernel side.This doesn't explain much if anything :/
I'm confused (probably because I don't know RISC-V very well), what's EBREAK and how does it happen?
EBREAK is just an instruction of riscv which would rise breakpoint exception.
Quoted 4 lines
Specifically, if EBREAK can happen inside an
local_irq_disable()region, then the below change is actively wrong. Any exception/interrupt that can happen whilelocal_irq_disable()must be treated like an NMI.When the ebreak happend out of
local_irq_disableregion, but__nmi_enterforceshandle_break()intoin_interupt()state. So how
Quoted 8 lines
Quoted 2 lines
And why is that a problem? I think I'm missing something fundamental here...
The
irqentry_nmi_enter()would force the current context to get in_interrupt=true, although ebreak happens in the context which is in_interrupt=false.
A lot of checking codes, such as:if (in_interrupt()) panic("Fatal exception in interrupt");
Why would you do that?!?
Are you're trying to differentiate between an exception and an interrupt?
You *could* have ebreak in an interrupt, right? So why panic the machine if that happens?
Quoted 3 lines
It would make the kernel panic, but we don't panic; we want back to the shell. eg:
echo BUG >/sys/kernel/debug/provoke-crash/DIRECT
Quoted 40 lines · Peter Zijlstra
Quoted 29 lines · Guo Ren
Quoted 28 lines · Peter Zijlstra
Quoted 25 lines · Guo Ren
Quoted 8 lines · Peter Zijlstra
Quoted 6 lines · On Sat, Jul 01, 2023 at 10:57:07PM -0400, [email protected] wrote:
From: Guo Ren <[email protected]>
The irqentry_nmi_enter/exit would force the current context into
in_interrupt. That would trigger the kernel to dead panic, but the kdb still needs "ebreak" to debug the kernel.Quoted 7 lines
Quoted 2 lines
Move irqentry_nmi_enter/exit to exception_enter/exit could correct
handle_breakof the kernel side.This doesn't explain much if anything :/
I'm confused (probably because I don't know RISC-V very well), what's EBREAK and how does it happen?
EBREAK is just an instruction of riscv which would rise breakpoint exception.
Quoted 4 lines
Specifically, if EBREAK can happen inside an
local_irq_disable()region, then the below change is actively wrong. Any exception/interrupt that can happen whilelocal_irq_disable()must be treated like an NMI.When the ebreak happend out of
local_irq_disableregion, but__nmi_enterforceshandle_break()intoin_interupt()state. So howQuoted 8 lines
Quoted 2 lines
And why is that a problem? I think I'm missing something fundamental here...
The
irqentry_nmi_enter()would force the current context to get in_interrupt=true, although ebreak happens in the context which is in_interrupt=false.
A lot of checking codes, such as:if (in_interrupt()) panic("Fatal exception in interrupt");
Quoted 7 lines
Why would you do that?!?
Are you're trying to differentiate between an exception and an interrupt?
You *could* have ebreak in an interrupt, right? So why panic the machine if that happens?
Do you mean the below patch? Yes, it could fix up.
arch/riscv/kernel/traps.c +0 -2
@@ -85,8 +85,6 @@ void die(struct pt_regs *regs, const char *str)spin_unlock_irqrestore (&die_lock, flags);oops_exit (); -if (in_interrupt ()) -panic ("Fatal exception in interrupt" );if (panic_on_oops)panic ("Fatal exception" );if (ret !=NOTIFY_STOP )
kernel/exit.c +0 -2
@@ -940,8 +940,6 @@ void __noreturn make_task_dead(int signr)struct task_struct *tsk = current;unsigned int limit; -if (unlikely (in_interrupt ())) -panic ("Aiee, killing interrupt handler!" );if (unlikely (!tsk->pid))panic ("Attempted to kill the idle task!" );
But how does x86 deal with it without kernel/exit.c modifcation?
Quoted 7 lines
Quoted 4 lines
It would make the kernel panic, but we don't panic; we want back to the shell. eg:
echo BUG >/sys/kernel/debug/provoke-crash/DIRECT
Signature · 7 lines
Best Regards Guo Ren _______________________________________________ linux-riscv mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-riscv
*thread encromancy*
Quoted 9 lines · On Sat, Jul 01, 2023 at 10:57:07PM -0400, [email protected] wrote:
From: Guo Ren <[email protected]>
The irqentry_nmi_enter/exit would force the current context into
in_interrupt. That would trigger the kernel to dead panic, but the kdb still needs "ebreak" to debug the kernel.Move irqentry_nmi_enter/exit to exception_enter/exit could correct
handle_breakof the kernel side.
Quoted 7 lines
Before the fixup:
$echo BUG >/sys/kernel/debug/provoke-crash/DIRECT
lkdtm: Performing direct entry BUG------------[ cut here ]------------ kernel BUG at drivers/misc/lkdtm/bugs.c:78! [...] Kernel panic - not syncing: Aiee, killing interrupt handler!
This appears to still be unfixed. What's the blocker? The solutions in this thread seem to work...
I'd like to be exercising an Oops path via KUnit (for KCFI), and riscv just instantly falls over instead of thread-killing on the exception.
Thanks!
-Kees
Signature · 1 line
Kees Cook
Quoted 20 lines · Kees Cook
*thread encromancy*
Quoted 9 lines · On Sat, Jul 01, 2023 at 10:57:07PM -0400, [email protected] wrote:
From: Guo Ren <[email protected]>
The irqentry_nmi_enter/exit would force the current context into
in_interrupt. That would trigger the kernel to dead panic, but the kdb still needs "ebreak" to debug the kernel.Move irqentry_nmi_enter/exit to exception_enter/exit could correct
handle_breakof the kernel side.Quoted 7 lines
Before the fixup:
$echo BUG >/sys/kernel/debug/provoke-crash/DIRECT
lkdtm: Performing direct entry BUG------------[ cut here ]------------ kernel BUG at drivers/misc/lkdtm/bugs.c:78! [...] Kernel panic - not syncing: Aiee, killing interrupt handler!
Quoted 5 lines
This appears to still be unfixed. What's the blocker? The solutions in this thread seem to work...
I'd like to be exercising an Oops path via KUnit (for KCFI), and riscv just instantly falls over instead of thread-killing on the exception.
Thanks for reviving this thread. At the time I didn’t fully understand Peter’s point. We should only use the NMI path when the trap occurs with interrupts disabled.
Here’s the updated fix:
do_trap_break(struct pt_regs *regs)
...
irqentry_exit_to_user_mode(regs);
} else {
- irqentry_state_t state = irqentry_nmi_enter(regs);
+ if (regs->status & SR_IE) {
+ enum ctx_state prev_state = exception_enter();
- handle_break(regs);
+ handle_break(regs);
- irqentry_nmi_exit(regs, state);
+ exception_exit(prev_state);
+ } else {
+ irqentry_state_t state = irqentry_nmi_enter(regs);
+
+ handle_break(regs);
+
+ irqentry_nmi_exit(regs, state);
+ }
}
}
If you & Peter have no objection, I’ll post a v2.
Best Regards
GUO Ren
Quoted 54 lines · Guo Ren
Quoted 20 lines · Kees Cook
*thread encromancy*
Quoted 9 lines · On Sat, Jul 01, 2023 at 10:57:07PM -0400, [email protected] wrote:
From: Guo Ren <[email protected]>
The irqentry_nmi_enter/exit would force the current context into
in_interrupt. That would trigger the kernel to dead panic, but the kdb still needs "ebreak" to debug the kernel.Move irqentry_nmi_enter/exit to exception_enter/exit could correct
handle_breakof the kernel side.Quoted 7 lines
Before the fixup:
$echo BUG >/sys/kernel/debug/provoke-crash/DIRECT
lkdtm: Performing direct entry BUG------------[ cut here ]------------ kernel BUG at drivers/misc/lkdtm/bugs.c:78! [...] Kernel panic - not syncing: Aiee, killing interrupt handler!Quoted 5 lines
This appears to still be unfixed. What's the blocker? The solutions in this thread seem to work...
I'd like to be exercising an Oops path via KUnit (for KCFI), and riscv just instantly falls over instead of thread-killing on the exception.
Thanks for reviving this thread. At the time I didn’t fully understand Peter’s point. We should only use the NMI path when the trap occurs with interrupts disabled.
Here’s the updated fix:
do_trap_break(struct pt_regs *regs)
...arch/riscv/kernel/traps.c +11
-3irqentry_exit_to_user_mode (regs); }else { -irqentry_state_t state =irqentry_nmi_enter (regs); +if (regs->status &SR_IE ) { +enum ctx_state prev_state =exception_enter (); -handle_break (regs); +handle_break (regs); -irqentry_nmi_exit (regs, state); +exception_exit (prev_state); + }else { +irqentry_state_t state =irqentry_nmi_enter (regs); + +handle_break (regs); + +irqentry_nmi_exit (regs, state); + } } }
Quoted 1 line
If you & Peter have no objection, I’ll post a v2.
I still don't understand it. This cannot fix anything. Consider:
EBREAK
raw_spin_lock_irq(&your_lock)
EBREAK
So now the first 'works', but the second will crash. Additionally, having the EBREAK context differ so dramatically between invocations seems like a very bad deal to me.
Quoted 57 lines · Peter Zijlstra
Quoted 53 lines · Guo Ren
Quoted 19 lines · Kees Cook
*thread encromancy*
Quoted 9 lines · On Sat, Jul 01, 2023 at 10:57:07PM -0400, [email protected] wrote:
From: Guo Ren <[email protected]>
The irqentry_nmi_enter/exit would force the current context into
in_interrupt. That would trigger the kernel to dead panic, but the kdb still needs "ebreak" to debug the kernel.Move irqentry_nmi_enter/exit to exception_enter/exit could correct
handle_breakof the kernel side.Quoted 7 lines
Before the fixup:
$echo BUG >/sys/kernel/debug/provoke-crash/DIRECT
lkdtm: Performing direct entry BUG------------[ cut here ]------------ kernel BUG at drivers/misc/lkdtm/bugs.c:78! [...] Kernel panic - not syncing: Aiee, killing interrupt handler!Quoted 5 lines
This appears to still be unfixed. What's the blocker? The solutions in this thread seem to work...
I'd like to be exercising an Oops path via KUnit (for KCFI), and riscv just instantly falls over instead of thread-killing on the exception.
Thanks for reviving this thread. At the time I didn’t fully understand Peter’s point. We should only use the NMI path when the trap occurs with interrupts disabled.
Here’s the updated fix:
do_trap_break(struct pt_regs *regs)
...arch/riscv/kernel/traps.c +11
-3irqentry_exit_to_user_mode (regs); }else { -irqentry_state_t state =irqentry_nmi_enter (regs); +if (regs->status &SR_IE ) { +enum ctx_state prev_state =exception_enter (); -handle_break (regs); +handle_break (regs); -irqentry_nmi_exit (regs, state); +exception_exit (prev_state); + }else { +irqentry_state_t state =irqentry_nmi_enter (regs); + +handle_break (regs); + +irqentry_nmi_exit (regs, state); + } } }Quoted 1 line
If you & Peter have no objection, I’ll post a v2.
I still don't understand it. This cannot fix anything. Consider:
Quoted 7 lines
EBREAK
raw_spin_lock_irq(&your_lock)
EBREAKSo now the first 'works', but the second will crash. Additionally, having the EBREAK context differ so dramatically between invocations seems like a very bad deal to me.
To spell it out, the problem that needs fixing is:
-> BUG()
-> ebreak instruction
-> Breakpoint exception
-> do_trap_break()
-> irqentry_nmi_enter()
[ now in_nmi() / in_interrupt() ]
-> report_bug() returns BUG_TRAP_TYPE_BUG
-> die()
-> make_task_dead()
-> panic() because we're in_interrupt()As such, currently on riscv all BUG() simply completely panic() the entire machine, rather than just killing the one task.
How do you think this should be fixed? Here are some ideas but I'm not familiar with generic entry stuff:
* Should we irqentry_nmi_exit() before calling die() for BUG()?
* Should we move the GENERIC_BUG trap instruction to cause illegal instruction exception instead, for which we can write a simpler handler that doesn't need to care about the probe stuff?
Vivian "dramforever" Wang
Quoted 23 lines · Vivian Wang
Quoted 9 lines
I still don't understand it. This cannot fix anything. Consider:
EBREAK
raw_spin_lock_irq(&your_lock)
EBREAKSo now the first 'works', but the second will crash. Additionally, having the EBREAK context differ so dramatically between invocations seems like a very bad deal to me.
To spell it out, the problem that needs fixing is:
->
BUG()
-> ebreak instruction-> Breakpoint exception -> do_trap_break() -> irqentry_nmi_enter() [ now in_nmi() / in_interrupt() ] -> report_bug() returns BUG_TRAP_TYPE_BUG -> die() -> make_task_dead() -> panic() because we're in_interrupt()
Quoted 2 lines
As such, currently on riscv all
BUG()simply completelypanic()the entire machine, rather than just killing the one task.
Hmm, from reading some of the previous emails this morning, I got the impression the problem was with kgdb, not BUG().
Anyway, my argument doesn't change, with the proposed patch:
BUG()
and:
local_irq_disable(); BUG();
will behave quite differently, for no sane reason.
Anyway, BUG()/trap is indeed a bit of magic, the x86 code lives in arch/x86/kernel/traps.c:exc_invalid_op(). And it looks like we do not indeed use NMI-like for this path, although I cannot remember why.
*however* I see your kgdb thing also uses ebreak, whereas on x86 WARN/BUG and kGDB use different exceptions (#UD for WARN/BUG and #BP for gdb). And our #BP handler (exc_int3) very much does NMI for from-kernel.
Same for kprobes, we use #BP/int3 for that, you also have that in EBREAK.
Anyway, you're handling 3 different cases in one exception, which is a bit of a mess, but something like so perhaps?
---
arch/riscv/kernel/traps.c +45 -27
@@ -264,42 +264,58 @@ static inline unsigned long get_break_insn_length(unsigned long pc)return GET_INSN_LENGTH (insn); } -static bool probe_single_step_handler (struct pt_regs *regs) +static void handle_kernel_die (struct pt_regs *regs) { -bool user =user_mode (regs); - -return user ?uprobe_single_step_handler (regs) :kprobe_single_step_handler (regs); +irqentry_state_t state =irqentry_enter (regs); +die (regs,"Kernel BUG" ); +irqentry_exit (regs, state); } -static bool probe_breakpoint_handler (struct pt_regs *regs) +static bool handle_kernel_bug (struct pt_regs *regs) { -bool user =user_mode (regs); +if (report_bug (regs->epc, regs) ==BUG_TRAP_TYPE_WARN || +handle_cfi_failure (regs) ==BUG_TRAP_TYPE_WARN ) { + regs->epc +=get_break_insn_length (regs->epc); +return true; + } -return user ?uprobe_breakpoint_handler (regs) :kprobe_breakpoint_handler (regs); +return false; } -void handle_break (struct pt_regs *regs) +static bool __handle_kernel_break (struct pt_regs *regs) { -if (probe_single_step_handler (regs)) -return ; -if (probe_breakpoint_handler (regs)) +if (kprobe_single_step_handler (regs) || +kprobe_breakpoint_handler (regs)) +return true; + + current->thread.bad_cause = regs->cause; + +#ifdef CONFIG_KGDB +if (notify_die (DIE_TRAP ,"EBREAK" , regs,0 , regs->cause,SIGTRAP ) + ==NOTIFY_STOP ) +return true; +#endif +return false; +} + +static bool handle_kernel_break (struct pt_regs *regs) +{ +irqentry_state_t state =irqentry_nmi_enter (regs); +bool ret =__handle_kernel_break (regs); +irqentry_nmi_exit (regs, state); +return ret; +} + +static void handle_user_break (struct pt_regs *regs) +{ +if (uprobe_single_step_handler (regs) || +uprobe_breakpoint_handler (regs))return ; current->thread.bad_cause = regs->cause; -if (user_mode (regs)) -force_sig_fault (SIGTRAP ,TRAP_BRKPT , (void __user *)regs->epc); -#ifdef CONFIG_KGDB - else if (notify_die(DIE_TRAP, "EBREAK", regs, 0, regs->cause, SIGTRAP) - == NOTIFY_STOP) -return ; -#endif - else if (report_bug(regs->epc, regs) == BUG_TRAP_TYPE_WARN || - handle_cfi_failure(regs) == BUG_TRAP_TYPE_WARN) - regs->epc += get_break_insn_length(regs->epc); - else - die(regs, "Kernel BUG"); +force_sig_fault (SIGTRAP ,TRAP_BRKPT , (void __user *)regs->epc); } asmlinkage __visible __trap_section void do_trap_break(struct pt_regs *regs)
@@ -308,16 +324,18 @@ asmlinkage __visible __trap_section void do_trap_break(struct pt_regs *regs)irqentry_enter_from_user_mode (regs);local_irq_enable (); -handle_break (regs); +handle_user_break (regs);local_irq_disable ();irqentry_exit_to_user_mode (regs); }else { -irqentry_state_t state =irqentry_nmi_enter (regs); +if (handle_kernel_bug (regs)) +return ; -handle_break (regs); +if (handle_kernel_break (regs)) +return ; -irqentry_nmi_exit (regs, state); +handle_kernel_die (regs); } }
Quoted 27 lines · Vivian Wang
Quoted 9 lines · Peter Zijlstra
I still don't understand it. This cannot fix anything. Consider:
EBREAK
raw_spin_lock_irq(&your_lock)
EBREAKSo now the first 'works', but the second will crash. Additionally, having the EBREAK context differ so dramatically between invocations seems like a very bad deal to me.
To spell it out, the problem that needs fixing is:
->
BUG()
-> ebreak instruction-> Breakpoint exception -> do_trap_break() -> irqentry_nmi_enter() [ now in_nmi() / in_interrupt() ] -> report_bug() returns BUG_TRAP_TYPE_BUG -> die() -> make_task_dead() -> panic() because we're in_interrupt()As such, currently on riscv all
BUG()simply completelypanic()the entire machine, rather than just killing the one task.
Quoted 7 lines
How do you think this should be fixed? Here are some ideas but I'm not familiar with generic entry stuff:
* Should we
irqentry_nmi_exit()before callingdie()forBUG()?
* Should we move theGENERIC_BUGtrap instruction to cause illegal instruction exception instead, for which we can write a simpler handler that doesn't need to care about the probe stuff?
Look at how x86 handles UD exceptions.