Skip to contentSkip to conversation

arm64: entry: Update the IRQ tracing before debug exception unmasked

PATCH In review Kevin Hao 2 participants 3 messages Apr 1, 2020 – Apr 17, 2020 arm64

lore
Share & tools
lore

Kevin Hao

author
#0 lore raw

PATCH 1/1 arm64: entry: Update the IRQ tracing before debug exception unmasked

When running the kgdb test suite, we get the following call trace:

  # echo kgdbts=V1 > /sys/module/kgdbts/parameters/kgdbts

DEBUG_LOCKS_WARN_ON(current->hardirqs_enabled)

  WARNING: CPU: 10 PID: 697 at kernel/locking/lockdep.c:4793 check_flags.part.0+0x15c/0x180
  Modules linked in:
  CPU: 10 PID: 697 Comm: sh Not tainted 5.6.0-next-20200331-yoctodev-standard+ #341
  Hardware name: Marvell OcteonTX CN96XX board (DT)
  pstate: 604003c9 (nZCv DAIF +PAN -UAO)
  pc : check_flags.part.0+0x15c/0x180
  lr : check_flags.part.0+0x15c/0x180
  sp : ffff800017d0f830
  x29: ffff800017d0f830 x28: ffff000b936c0000
  x27: 00000000c28f5c29 x26: 00000000ffffffff
  x25: 00000000000003c0 x24: ffff800011af8dd0
  x23: 0000000000000000 x22: ffff8000119afdc0
  x21: ffff80001198bbe8 x20: ffff800011400018
  x19: ffff800012655000 x18: 0000000000000001
  x17: 0000000000000000 x16: 0000000000000000
  x15: ffff000b936c0470 x14: ffffffffffffffff
  x13: 0000000000000000 x12: ffff800012655000
  x11: ffff800017d0f830 x10: ffff800017d0f830
  x9 : 00000000000003c0 x8 : 6e655f7371726964
  x7 : 7261683e2d746e65 x6 : ffff8000126552fe
  x5 : 0000000000000000 x4 : 0000000000000000
  x3 : 00000000ffffffff x2 : 0000000000000000
  x1 : bbf8ef1cf7dda200 x0 : 0000000000000000
  Call trace:
   check_flags.part.0+0x15c/0x180
   lock_is_held_type+0xf0/0x120
   rcu_read_lock_sched_held+0x74/0x98
   trace_rcu_dyntick+0x1b8/0x1e0
   rcu_nmi_enter+0x7c/0xb8
   debug_exception_enter+0x68/0xe8
   do_debug_exception+0x60/0x150
   el1_sync_handler+0xd8/0xf8
   el1_sync+0x7c/0x100
   el1_irq+0x78/0x180
   kgdbts_break_test+0x0/0x40
   param_set_kgdbts_var+0x68/0xe8
   param_attr_store+0xb8/0x120
   module_attr_store+0x2c/0x48
   sysfs_kf_write+0x54/0x80
   kernfs_fop_write+0x154/0x248
   __vfs_write+0x24/0x50
   vfs_write+0xec/0x1d8
   ksys_write+0x74/0x100
   __arm64_sys_write+0x24/0x30
   do_el0_svc+0x8c/0x1e8
   el0_sync_handler+0x11c/0x198
   el0_sync+0x158/0x180
  irq event stamp: 76505
  hardirqs last  enabled at (76505): [<ffff80001009f9ec>] debug_exception_exit+0x54/0x68
  hardirqs last disabled at (76504): [<ffff80001009fb04>] debug_exception_enter+0xac/0xe8
  softirqs last  enabled at (76498): [<ffff8000100817b4>] __do_softirq+0x5a4/0x5ec
  softirqs last disabled at (76439): [<ffff8000100b6ff4>] irq_exit+0x13c/0x150

The reason is that an IRQ is emitted when doing the single step debug, but in the IRQ handler we would enable the debug exception before updating the IRQ tracing flags. This will cause the debug exception running in an context which the IRQ state and IRQ tracing flags are mismatched. And the debug exception handler only update the IRQ tracing flag to off when it think that the debug emit in a IRQ enabled context. Then we would get the above call trace if any code like check_flags() is called in the debug exception handler's path. So we have to update the IRQ tracing to the correct status before the debug exception unmasked.

---
 arch/arm64/kernel/entry.S | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/arm64/kernel/entry.S +1 -1
index ddcde093c433..0cc65b7b9e1a 100644
@@ -592,7 +592,6 @@ SYM_CODE_END(el1_sync)
 SYM_CODE_START_LOCAL_NOALIGN(el1_irq)
 	kernel_entry 1
 	gic_prio_irq_setup pmr=x20, tmp=x1
-	enable_da_f
 
 #ifdef CONFIG_ARM64_PSEUDO_NMI
 	test_irqs_unmasked	res=x0, pmr=x20
@@ -605,6 +604,7 @@ SYM_CODE_START_LOCAL_NOALIGN(el1_irq)
 	bl	trace_hardirqs_off
 #endif
 
+	enable_da_f
 	irq_handler
 
 #ifdef CONFIG_PREEMPTION
Signature · 7 lines
2.25.1.377.g2d2118b814c1


_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
reviewer
#1 lore raw
Quoted 67 lines · Kevin Hao

When running the kgdb test suite, we get the following call trace:

  # echo kgdbts=V1 > /sys/module/kgdbts/parameters/kgdbts

DEBUG_LOCKS_WARN_ON(current->hardirqs_enabled)

  WARNING: CPU: 10 PID: 697 at kernel/locking/lockdep.c:4793 check_flags.part.0+0x15c/0x180
  Modules linked in:
  CPU: 10 PID: 697 Comm: sh Not tainted 5.6.0-next-20200331-yoctodev-standard+ #341
  Hardware name: Marvell OcteonTX CN96XX board (DT)
  pstate: 604003c9 (nZCv DAIF +PAN -UAO)
  pc : check_flags.part.0+0x15c/0x180
  lr : check_flags.part.0+0x15c/0x180
  sp : ffff800017d0f830
  x29: ffff800017d0f830 x28: ffff000b936c0000
  x27: 00000000c28f5c29 x26: 00000000ffffffff
  x25: 00000000000003c0 x24: ffff800011af8dd0
  x23: 0000000000000000 x22: ffff8000119afdc0
  x21: ffff80001198bbe8 x20: ffff800011400018
  x19: ffff800012655000 x18: 0000000000000001
  x17: 0000000000000000 x16: 0000000000000000
  x15: ffff000b936c0470 x14: ffffffffffffffff
  x13: 0000000000000000 x12: ffff800012655000
  x11: ffff800017d0f830 x10: ffff800017d0f830
  x9 : 00000000000003c0 x8 : 6e655f7371726964
  x7 : 7261683e2d746e65 x6 : ffff8000126552fe
  x5 : 0000000000000000 x4 : 0000000000000000
  x3 : 00000000ffffffff x2 : 0000000000000000
  x1 : bbf8ef1cf7dda200 x0 : 0000000000000000
  Call trace:
   check_flags.part.0+0x15c/0x180
   lock_is_held_type+0xf0/0x120
   rcu_read_lock_sched_held+0x74/0x98
   trace_rcu_dyntick+0x1b8/0x1e0
   rcu_nmi_enter+0x7c/0xb8
   debug_exception_enter+0x68/0xe8
   do_debug_exception+0x60/0x150
   el1_sync_handler+0xd8/0xf8
   el1_sync+0x7c/0x100
   el1_irq+0x78/0x180
   kgdbts_break_test+0x0/0x40
   param_set_kgdbts_var+0x68/0xe8
   param_attr_store+0xb8/0x120
   module_attr_store+0x2c/0x48
   sysfs_kf_write+0x54/0x80
   kernfs_fop_write+0x154/0x248
   __vfs_write+0x24/0x50
   vfs_write+0xec/0x1d8
   ksys_write+0x74/0x100
   __arm64_sys_write+0x24/0x30
   do_el0_svc+0x8c/0x1e8
   el0_sync_handler+0x11c/0x198
   el0_sync+0x158/0x180
  irq event stamp: 76505
  hardirqs last  enabled at (76505): [<ffff80001009f9ec>] debug_exception_exit+0x54/0x68
  hardirqs last disabled at (76504): [<ffff80001009fb04>] debug_exception_enter+0xac/0xe8
  softirqs last  enabled at (76498): [<ffff8000100817b4>] __do_softirq+0x5a4/0x5ec
  softirqs last disabled at (76439): [<ffff8000100b6ff4>] irq_exit+0x13c/0x150

The reason is that an IRQ is emitted when doing the single step debug, but in the IRQ handler we would enable the debug exception before updating the IRQ tracing flags. This will cause the debug exception running in an context which the IRQ state and IRQ tracing flags are mismatched. And the debug exception handler only update the IRQ tracing flag to off when it think that the debug emit in a IRQ enabled context. Then we would get the above call trace if any code like check_flags() is called in the debug exception handler's path. So we have to update the IRQ tracing to the correct status before the debug exception unmasked.

Quoted 1 line

I think that actual issue here is in the debug path:
debug_exception_enter() tries to conditionally trace_hardirqs_off(), and this is unsound as it does this based solely on interrupts_enabled(regs).

Here we take a debug exception while the HW state is disabled, but the tracing state is enabled, and I reckon we can also do that elsewhere. for example, if we set a breakpoint in the middle of local_irq_disable(), between the write to DAIF and the update of the irq tracing state. In that case, interrupts_enabled(regs) will be false, but the tracing state will be stale and might say that IRQs are enabled.

Given that possibility, I suspect that we need to unconditionally trace_hardirqs_off() in the debug entry path to make that robust.

Thanks,
Mark.

Quoted 32 lines
---
 arch/arm64/kernel/entry.S | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/arm64/kernel/entry.S +1 -1
index ddcde093c433..0cc65b7b9e1a 100644
@@ -592,7 +592,6 @@ SYM_CODE_END(el1_sync)
 SYM_CODE_START_LOCAL_NOALIGN(el1_irq)
 	kernel_entry 1
 	gic_prio_irq_setup pmr=x20, tmp=x1
-	enable_da_f
 
 #ifdef CONFIG_ARM64_PSEUDO_NMI
 	test_irqs_unmasked	res=x0, pmr=x20
@@ -605,6 +604,7 @@ SYM_CODE_START_LOCAL_NOALIGN(el1_irq)
 	bl	trace_hardirqs_off
 #endif
 
+	enable_da_f
 	irq_handler
 
 #ifdef CONFIG_PREEMPTION
Signature · 7 lines
2.25.1.377.g2d2118b814c1


_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

author
#2 lore raw
Quoted 82 lines · Mark Rutland
Quoted 67 lines · Kevin Hao

When running the kgdb test suite, we get the following call trace:

  # echo kgdbts=V1 > /sys/module/kgdbts/parameters/kgdbts

DEBUG_LOCKS_WARN_ON(current->hardirqs_enabled)

  WARNING: CPU: 10 PID: 697 at kernel/locking/lockdep.c:4793 check_flags.part.0+0x15c/0x180
  Modules linked in:
  CPU: 10 PID: 697 Comm: sh Not tainted 5.6.0-next-20200331-yoctodev-standard+ #341
  Hardware name: Marvell OcteonTX CN96XX board (DT)
  pstate: 604003c9 (nZCv DAIF +PAN -UAO)
  pc : check_flags.part.0+0x15c/0x180
  lr : check_flags.part.0+0x15c/0x180
  sp : ffff800017d0f830
  x29: ffff800017d0f830 x28: ffff000b936c0000
  x27: 00000000c28f5c29 x26: 00000000ffffffff
  x25: 00000000000003c0 x24: ffff800011af8dd0
  x23: 0000000000000000 x22: ffff8000119afdc0
  x21: ffff80001198bbe8 x20: ffff800011400018
  x19: ffff800012655000 x18: 0000000000000001
  x17: 0000000000000000 x16: 0000000000000000
  x15: ffff000b936c0470 x14: ffffffffffffffff
  x13: 0000000000000000 x12: ffff800012655000
  x11: ffff800017d0f830 x10: ffff800017d0f830
  x9 : 00000000000003c0 x8 : 6e655f7371726964
  x7 : 7261683e2d746e65 x6 : ffff8000126552fe
  x5 : 0000000000000000 x4 : 0000000000000000
  x3 : 00000000ffffffff x2 : 0000000000000000
  x1 : bbf8ef1cf7dda200 x0 : 0000000000000000
  Call trace:
   check_flags.part.0+0x15c/0x180
   lock_is_held_type+0xf0/0x120
   rcu_read_lock_sched_held+0x74/0x98
   trace_rcu_dyntick+0x1b8/0x1e0
   rcu_nmi_enter+0x7c/0xb8
   debug_exception_enter+0x68/0xe8
   do_debug_exception+0x60/0x150
   el1_sync_handler+0xd8/0xf8
   el1_sync+0x7c/0x100
   el1_irq+0x78/0x180
   kgdbts_break_test+0x0/0x40
   param_set_kgdbts_var+0x68/0xe8
   param_attr_store+0xb8/0x120
   module_attr_store+0x2c/0x48
   sysfs_kf_write+0x54/0x80
   kernfs_fop_write+0x154/0x248
   __vfs_write+0x24/0x50
   vfs_write+0xec/0x1d8
   ksys_write+0x74/0x100
   __arm64_sys_write+0x24/0x30
   do_el0_svc+0x8c/0x1e8
   el0_sync_handler+0x11c/0x198
   el0_sync+0x158/0x180
  irq event stamp: 76505
  hardirqs last  enabled at (76505): [<ffff80001009f9ec>] debug_exception_exit+0x54/0x68
  hardirqs last disabled at (76504): [<ffff80001009fb04>] debug_exception_enter+0xac/0xe8
  softirqs last  enabled at (76498): [<ffff8000100817b4>] __do_softirq+0x5a4/0x5ec
  softirqs last disabled at (76439): [<ffff8000100b6ff4>] irq_exit+0x13c/0x150

The reason is that an IRQ is emitted when doing the single step debug, but in the IRQ handler we would enable the debug exception before updating the IRQ tracing flags. This will cause the debug exception running in an context which the IRQ state and IRQ tracing flags are mismatched. And the debug exception handler only update the IRQ tracing flag to off when it think that the debug emit in a IRQ enabled context. Then we would get the above call trace if any code like check_flags() is called in the debug exception handler's path. So we have to update the IRQ tracing to the correct status before the debug exception unmasked.

Quoted 1 line

I think that actual issue here is in the debug path:
debug_exception_enter() tries to conditionally trace_hardirqs_off(), and this is unsound as it does this based solely on interrupts_enabled(regs).

Here we take a debug exception while the HW state is disabled, but the tracing state is enabled, and I reckon we can also do that elsewhere. for example, if we set a breakpoint in the middle of local_irq_disable(), between the write to DAIF and the update of the irq tracing state. In that case, interrupts_enabled(regs) will be false, but the tracing state will be stale and might say that IRQs are enabled.

Quoted 2 lines

Given that possibility, I suspect that we need to unconditionally trace_hardirqs_off() in the debug entry path to make that robust.

It makes sense. The v2 is coming.

Thanks,
Kevin

Quoted 32 lines

Thanks,
Mark.

Quoted 28 lines
---
 arch/arm64/kernel/entry.S | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/arm64/kernel/entry.S +1 -1
index ddcde093c433..0cc65b7b9e1a 100644
@@ -592,7 +592,6 @@ SYM_CODE_END(el1_sync)
 SYM_CODE_START_LOCAL_NOALIGN(el1_irq)
 	kernel_entry 1
 	gic_prio_irq_setup pmr=x20, tmp=x1
-	enable_da_f
 
 #ifdef CONFIG_ARM64_PSEUDO_NMI
 	test_irqs_unmasked	res=x0, pmr=x20
@@ -605,6 +604,7 @@ SYM_CODE_START_LOCAL_NOALIGN(el1_irq)
 	bl	trace_hardirqs_off
 #endif
 
+	enable_da_f
 	irq_handler
 
 #ifdef CONFIG_PREEMPTION
Signature · 1 line
2.25.1.377.g2d2118b814c1
Quoted 4 lines
Quoted 4 lines

_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel