arm64: entry: Update the IRQ tracing before debug exception unmasked
PATCH 1/1 arm64: entry: Update the IRQ tracing before debug exception unmasked
When running the kgdb test suite, we get the following call trace:
# echo kgdbts=V1 > /sys/module/kgdbts/parameters/kgdbts
DEBUG_LOCKS_WARN_ON(current->hardirqs_enabled)
WARNING: CPU: 10 PID: 697 at kernel/locking/lockdep.c:4793 check_flags.part.0+0x15c/0x180 Modules linked in: CPU: 10 PID: 697 Comm: sh Not tainted 5.6.0-next-20200331-yoctodev-standard+ #341 Hardware name: Marvell OcteonTX CN96XX board (DT) pstate: 604003c9 (nZCv DAIF +PAN -UAO) pc : check_flags.part.0+0x15c/0x180 lr : check_flags.part.0+0x15c/0x180 sp : ffff800017d0f830 x29: ffff800017d0f830 x28: ffff000b936c0000 x27: 00000000c28f5c29 x26: 00000000ffffffff x25: 00000000000003c0 x24: ffff800011af8dd0 x23: 0000000000000000 x22: ffff8000119afdc0 x21: ffff80001198bbe8 x20: ffff800011400018 x19: ffff800012655000 x18: 0000000000000001 x17: 0000000000000000 x16: 0000000000000000 x15: ffff000b936c0470 x14: ffffffffffffffff x13: 0000000000000000 x12: ffff800012655000 x11: ffff800017d0f830 x10: ffff800017d0f830 x9 : 00000000000003c0 x8 : 6e655f7371726964 x7 : 7261683e2d746e65 x6 : ffff8000126552fe x5 : 0000000000000000 x4 : 0000000000000000 x3 : 00000000ffffffff x2 : 0000000000000000 x1 : bbf8ef1cf7dda200 x0 : 0000000000000000 Call trace: check_flags.part.0+0x15c/0x180 lock_is_held_type+0xf0/0x120 rcu_read_lock_sched_held+0x74/0x98 trace_rcu_dyntick+0x1b8/0x1e0 rcu_nmi_enter+0x7c/0xb8 debug_exception_enter+0x68/0xe8 do_debug_exception+0x60/0x150 el1_sync_handler+0xd8/0xf8 el1_sync+0x7c/0x100 el1_irq+0x78/0x180 kgdbts_break_test+0x0/0x40 param_set_kgdbts_var+0x68/0xe8 param_attr_store+0xb8/0x120 module_attr_store+0x2c/0x48 sysfs_kf_write+0x54/0x80 kernfs_fop_write+0x154/0x248 __vfs_write+0x24/0x50 vfs_write+0xec/0x1d8 ksys_write+0x74/0x100 __arm64_sys_write+0x24/0x30 do_el0_svc+0x8c/0x1e8 el0_sync_handler+0x11c/0x198 el0_sync+0x158/0x180 irq event stamp: 76505 hardirqs last enabled at (76505): [<ffff80001009f9ec>] debug_exception_exit+0x54/0x68 hardirqs last disabled at (76504): [<ffff80001009fb04>] debug_exception_enter+0xac/0xe8 softirqs last enabled at (76498): [<ffff8000100817b4>] __do_softirq+0x5a4/0x5ec softirqs last disabled at (76439): [<ffff8000100b6ff4>] irq_exit+0x13c/0x150
The reason is that an IRQ is emitted when doing the single step debug, but in the IRQ handler we would enable the debug exception before updating the IRQ tracing flags. This will cause the debug exception running in an context which the IRQ state and IRQ tracing flags are mismatched. And the debug exception handler only update the IRQ tracing flag to off when it think that the debug emit in a IRQ enabled context. Then we would get the above call trace if any code like check_flags() is called in the debug exception handler's path. So we have to update the IRQ tracing to the correct status before the debug exception unmasked.
--- arch/arm64/kernel/entry.S | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/arm64/kernel/entry.S +1 -1
@@ -592,7 +592,6 @@ SYM_CODE_END(el1_sync) SYM_CODE_START_LOCAL_NOALIGN(el1_irq) kernel_entry 1 gic_prio_irq_setup pmr=x20, tmp=x1 - enable_da_f #ifdef CONFIG_ARM64_PSEUDO_NMI test_irqs_unmasked res=x0, pmr=x20
@@ -605,6 +604,7 @@ SYM_CODE_START_LOCAL_NOALIGN(el1_irq) bl trace_hardirqs_off #endif + enable_da_f irq_handler #ifdef CONFIG_PREEMPTION
Signature · 7 lines
2.25.1.377.g2d2118b814c1 _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Mark Rutland
reviewerQuoted 67 lines · Kevin Hao
When running the kgdb test suite, we get the following call trace:
# echo kgdbts=V1 > /sys/module/kgdbts/parameters/kgdbts
DEBUG_LOCKS_WARN_ON(current->hardirqs_enabled)WARNING: CPU: 10 PID: 697 at kernel/locking/lockdep.c:4793 check_flags.part.0+0x15c/0x180 Modules linked in: CPU: 10 PID: 697 Comm: sh Not tainted 5.6.0-next-20200331-yoctodev-standard+ #341 Hardware name: Marvell OcteonTX CN96XX board (DT) pstate: 604003c9 (nZCv DAIF +PAN -UAO) pc : check_flags.part.0+0x15c/0x180 lr : check_flags.part.0+0x15c/0x180 sp : ffff800017d0f830 x29: ffff800017d0f830 x28: ffff000b936c0000 x27: 00000000c28f5c29 x26: 00000000ffffffff x25: 00000000000003c0 x24: ffff800011af8dd0 x23: 0000000000000000 x22: ffff8000119afdc0 x21: ffff80001198bbe8 x20: ffff800011400018 x19: ffff800012655000 x18: 0000000000000001 x17: 0000000000000000 x16: 0000000000000000 x15: ffff000b936c0470 x14: ffffffffffffffff x13: 0000000000000000 x12: ffff800012655000 x11: ffff800017d0f830 x10: ffff800017d0f830 x9 : 00000000000003c0 x8 : 6e655f7371726964 x7 : 7261683e2d746e65 x6 : ffff8000126552fe x5 : 0000000000000000 x4 : 0000000000000000 x3 : 00000000ffffffff x2 : 0000000000000000 x1 : bbf8ef1cf7dda200 x0 : 0000000000000000 Call trace: check_flags.part.0+0x15c/0x180 lock_is_held_type+0xf0/0x120 rcu_read_lock_sched_held+0x74/0x98 trace_rcu_dyntick+0x1b8/0x1e0 rcu_nmi_enter+0x7c/0xb8 debug_exception_enter+0x68/0xe8 do_debug_exception+0x60/0x150 el1_sync_handler+0xd8/0xf8 el1_sync+0x7c/0x100 el1_irq+0x78/0x180 kgdbts_break_test+0x0/0x40 param_set_kgdbts_var+0x68/0xe8 param_attr_store+0xb8/0x120 module_attr_store+0x2c/0x48 sysfs_kf_write+0x54/0x80 kernfs_fop_write+0x154/0x248 __vfs_write+0x24/0x50 vfs_write+0xec/0x1d8 ksys_write+0x74/0x100 __arm64_sys_write+0x24/0x30 do_el0_svc+0x8c/0x1e8 el0_sync_handler+0x11c/0x198 el0_sync+0x158/0x180 irq event stamp: 76505 hardirqs last enabled at (76505): [<ffff80001009f9ec>] debug_exception_exit+0x54/0x68 hardirqs last disabled at (76504): [<ffff80001009fb04>] debug_exception_enter+0xac/0xe8 softirqs last enabled at (76498): [<ffff8000100817b4>] __do_softirq+0x5a4/0x5ec softirqs last disabled at (76439): [<ffff8000100b6ff4>] irq_exit+0x13c/0x150The reason is that an IRQ is emitted when doing the single step debug, but in the IRQ handler we would enable the debug exception before updating the IRQ tracing flags. This will cause the debug exception running in an context which the IRQ state and IRQ tracing flags are mismatched. And the debug exception handler only update the IRQ tracing flag to off when it think that the debug emit in a IRQ enabled context. Then we would get the above call trace if any code like
check_flags()is called in the debug exception handler's path. So we have to update the IRQ tracing to the correct status before the debug exception unmasked.
Quoted 1 line
Signed-off-by Kevin Hao <[email protected]>
I think that actual issue here is in the debug path:debug_exception_enter() tries to conditionally trace_hardirqs_off(), and this is unsound as it does this based solely on interrupts_enabled(regs).
Here we take a debug exception while the HW state is disabled, but the tracing state is enabled, and I reckon we can also do that elsewhere. for example, if we set a breakpoint in the middle of local_irq_disable(), between the write to DAIF and the update of the irq tracing state. In that case, interrupts_enabled(regs) will be false, but the tracing state will be stale and might say that IRQs are enabled.
Given that possibility, I suspect that we need to unconditionally trace_hardirqs_off() in the debug entry path to make that robust.
Thanks,
Mark.
Quoted 32 lines
--- arch/arm64/kernel/entry.S | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)arch/arm64/kernel/entry.S +1
-1@@ -592,7 +592,6 @@ SYM_CODE_END(el1_sync) SYM_CODE_START_LOCAL_NOALIGN(el1_irq) kernel_entry 1 gic_prio_irq_setup pmr=x20, tmp=x1 - enable_da_f #ifdef CONFIG_ARM64_PSEUDO_NMI test_irqs_unmasked res=x0, pmr=x20@@ -605,6 +604,7 @@ SYM_CODE_START_LOCAL_NOALIGN(el1_irq) bl trace_hardirqs_off #endif + enable_da_f irq_handler #ifdef CONFIG_PREEMPTIONSignature · 7 lines
2.25.1.377.g2d2118b814c1 _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Quoted 82 lines · Mark Rutland
Quoted 67 lines · Kevin Hao
When running the kgdb test suite, we get the following call trace:
# echo kgdbts=V1 > /sys/module/kgdbts/parameters/kgdbts
DEBUG_LOCKS_WARN_ON(current->hardirqs_enabled)WARNING: CPU: 10 PID: 697 at kernel/locking/lockdep.c:4793 check_flags.part.0+0x15c/0x180 Modules linked in: CPU: 10 PID: 697 Comm: sh Not tainted 5.6.0-next-20200331-yoctodev-standard+ #341 Hardware name: Marvell OcteonTX CN96XX board (DT) pstate: 604003c9 (nZCv DAIF +PAN -UAO) pc : check_flags.part.0+0x15c/0x180 lr : check_flags.part.0+0x15c/0x180 sp : ffff800017d0f830 x29: ffff800017d0f830 x28: ffff000b936c0000 x27: 00000000c28f5c29 x26: 00000000ffffffff x25: 00000000000003c0 x24: ffff800011af8dd0 x23: 0000000000000000 x22: ffff8000119afdc0 x21: ffff80001198bbe8 x20: ffff800011400018 x19: ffff800012655000 x18: 0000000000000001 x17: 0000000000000000 x16: 0000000000000000 x15: ffff000b936c0470 x14: ffffffffffffffff x13: 0000000000000000 x12: ffff800012655000 x11: ffff800017d0f830 x10: ffff800017d0f830 x9 : 00000000000003c0 x8 : 6e655f7371726964 x7 : 7261683e2d746e65 x6 : ffff8000126552fe x5 : 0000000000000000 x4 : 0000000000000000 x3 : 00000000ffffffff x2 : 0000000000000000 x1 : bbf8ef1cf7dda200 x0 : 0000000000000000 Call trace: check_flags.part.0+0x15c/0x180 lock_is_held_type+0xf0/0x120 rcu_read_lock_sched_held+0x74/0x98 trace_rcu_dyntick+0x1b8/0x1e0 rcu_nmi_enter+0x7c/0xb8 debug_exception_enter+0x68/0xe8 do_debug_exception+0x60/0x150 el1_sync_handler+0xd8/0xf8 el1_sync+0x7c/0x100 el1_irq+0x78/0x180 kgdbts_break_test+0x0/0x40 param_set_kgdbts_var+0x68/0xe8 param_attr_store+0xb8/0x120 module_attr_store+0x2c/0x48 sysfs_kf_write+0x54/0x80 kernfs_fop_write+0x154/0x248 __vfs_write+0x24/0x50 vfs_write+0xec/0x1d8 ksys_write+0x74/0x100 __arm64_sys_write+0x24/0x30 do_el0_svc+0x8c/0x1e8 el0_sync_handler+0x11c/0x198 el0_sync+0x158/0x180 irq event stamp: 76505 hardirqs last enabled at (76505): [<ffff80001009f9ec>] debug_exception_exit+0x54/0x68 hardirqs last disabled at (76504): [<ffff80001009fb04>] debug_exception_enter+0xac/0xe8 softirqs last enabled at (76498): [<ffff8000100817b4>] __do_softirq+0x5a4/0x5ec softirqs last disabled at (76439): [<ffff8000100b6ff4>] irq_exit+0x13c/0x150The reason is that an IRQ is emitted when doing the single step debug, but in the IRQ handler we would enable the debug exception before updating the IRQ tracing flags. This will cause the debug exception running in an context which the IRQ state and IRQ tracing flags are mismatched. And the debug exception handler only update the IRQ tracing flag to off when it think that the debug emit in a IRQ enabled context. Then we would get the above call trace if any code like
check_flags()is called in the debug exception handler's path. So we have to update the IRQ tracing to the correct status before the debug exception unmasked.Quoted 1 line
Signed-off-by Kevin Hao <[email protected]>I think that actual issue here is in the debug path:
debug_exception_enter()tries to conditionallytrace_hardirqs_off(), and this is unsound as it does this based solely oninterrupts_enabled(regs).Here we take a debug exception while the HW state is disabled, but the tracing state is enabled, and I reckon we can also do that elsewhere. for example, if we set a breakpoint in the middle of
local_irq_disable(), between the write to DAIF and the update of the irq tracing state. In that case,interrupts_enabled(regs)will be false, but the tracing state will be stale and might say that IRQs are enabled.
Quoted 2 lines
Given that possibility, I suspect that we need to unconditionally
trace_hardirqs_off()in the debug entry path to make that robust.
It makes sense. The v2 is coming.
Thanks,
Kevin
Quoted 32 lines
Thanks,
Mark.Quoted 28 lines
--- arch/arm64/kernel/entry.S | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)arch/arm64/kernel/entry.S +1
-1@@ -592,7 +592,6 @@ SYM_CODE_END(el1_sync) SYM_CODE_START_LOCAL_NOALIGN(el1_irq) kernel_entry 1 gic_prio_irq_setup pmr=x20, tmp=x1 - enable_da_f #ifdef CONFIG_ARM64_PSEUDO_NMI test_irqs_unmasked res=x0, pmr=x20@@ -605,6 +604,7 @@ SYM_CODE_START_LOCAL_NOALIGN(el1_irq) bl trace_hardirqs_off #endif + enable_da_f irq_handler #ifdef CONFIG_PREEMPTIONSignature · 1 line
2.25.1.377.g2d2118b814c1
Quoted 4 lines
Quoted 4 lines
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel