Fix spurious hard lockup events while in debugger
PATCH 1/1 Fix spurious hard lockup events while in debugger
The current touch_nmi_watchdog() function in /kernel/watchdog.c does not always catch all cases when a processor is spinning in the nmi handler inside either KGDB, KDB, or MDB, in particular, the case where a processor is being held by a debugger inside an int1 handler.
The hrtimer_interrupts_saved count can still end up matching the hrtime value in some cases, resulting in the hard lockup detector tagging processors inside a debugger and executing a panic.
The patch below corrects this problem. I did not add this to the touch_nmi_function directly becuase of possible affects on timing issues since the function is widely used by drivers and modules.
I have tested this patch and it fixes the problem for kernel debuggers stopping errant hard lockup events when processors are spinning inside the debugger.
--- kernel/watchdog.c | 7 +++++++ 1 file changed, 7 insertions(+)
kernel/watchdog.c +7 -0
@@ -283,6 +283,13 @@ static bool is_hardlockup(void)__this_cpu_write (hrtimer_interrupts_saved, hrint); return false; } + +void touch_hardlockup_watchdog (void ) +{ +__this_cpu_write (hrtimer_interrupts_saved,0 ); +} +EXPORT_SYMBOL_GPL (touch_hardlockup_watchdog); +#endif static int is_softlockup (unsigned long touch_ts)
Signature · 1 line
1.8.3.1
Quoted 42 lines · Jeff Merkey
The current
touch_nmi_watchdog()function in /kernel/watchdog.c does not always catch all cases when a processor is spinning in the nmi handler inside either KGDB, KDB, or MDB, in particular, the case where a processor is being held by a debugger inside an int1 handler.The
hrtimer_interrupts_savedcount can still end up matching the hrtime value in some cases, resulting in the hard lockup detector tagging processors inside a debugger and executing a panic.The patch below corrects this problem. I did not add this to the
touch_nmi_functiondirectly becuase of possible affects on timing issues since the function is widely used by drivers and modules.I have tested this patch and it fixes the problem for kernel debuggers stopping errant hard lockup events when processors are spinning inside the debugger.
Signed-off-by Jeff Merkey <[email protected]>--- kernel/watchdog.c | 7 +++++++ 1 file changed, 7 insertions(+)kernel/watchdog.c +7
-1@@ -283,6 +283,13 @@ static bool is_hardlockup(void)__this_cpu_write (hrtimer_interrupts_saved, hrint); return false; } + +void touch_hardlockup_watchdog (void ) +{ +__this_cpu_write (hrtimer_interrupts_saved,0 ); +} +EXPORT_SYMBOL_GPL (touch_hardlockup_watchdog); +#endif static int is_softlockup (unsigned long touch_ts) --
Quoted 3 lines
1.8.3.1
I stared at the function that detects hardlockups until my eyes have turned red and the code looks ok. I am still trying to figure why this is happening. I call touch_nmi_watchdog() without the other function and the hard lockup fires off.
I'll try to debug it and determine why it is happening. Having the flag thing for the touch operating independent of the counter may be the clue. I'll look into it a little more today and see if I can figure out why it is happening.
Jeff
Quoted 42 lines · Jeff Merkey
The current
touch_nmi_watchdog()function in /kernel/watchdog.c does not always catch all cases when a processor is spinning in the nmi handler inside either KGDB, KDB, or MDB, in particular, the case where a processor is being held by a debugger inside an int1 handler.The
hrtimer_interrupts_savedcount can still end up matching the hrtime value in some cases, resulting in the hard lockup detector tagging processors inside a debugger and executing a panic.The patch below corrects this problem. I did not add this to the
touch_nmi_functiondirectly becuase of possible affects on timing issues since the function is widely used by drivers and modules.I have tested this patch and it fixes the problem for kernel debuggers stopping errant hard lockup events when processors are spinning inside the debugger.
Signed-off-by Jeff Merkey <[email protected]>--- kernel/watchdog.c | 7 +++++++ 1 file changed, 7 insertions(+)kernel/watchdog.c +7
-1@@ -283,6 +283,13 @@ static bool is_hardlockup(void)__this_cpu_write (hrtimer_interrupts_saved, hrint); return false; } + +void touch_hardlockup_watchdog (void ) +{ +__this_cpu_write (hrtimer_interrupts_saved,0 ); +} +EXPORT_SYMBOL_GPL (touch_hardlockup_watchdog); +#endif static int is_softlockup (unsigned long touch_ts) --
Quoted 3 lines
1.8.3.1
I got to the bottom of it. It's related to the hardware I am using. One of the processors is faulting and hanging due to an existing bug in the hw_breakpoint handler not setting the resume flag (I have previously reported it and submitted a patch). This breaks your code, but there's nothing you can do about it.
There is a severe bug in hw_breakpoint.c that causes int1 recursion and this whole "lazy debug register switching" nonsense does not work properly. I am probably the first person to actually test this code path robustly. I applied the patch that fixes this bug in hw_breakpoint.c and the problem with your code firing off and ignoring the touch flag
went away.
Jeff
Quoted 53 lines · Jeff Merkey
Quoted 42 lines · Jeff Merkey
The current
touch_nmi_watchdog()function in /kernel/watchdog.c does not always catch all cases when a processor is spinning in the nmi handler inside either KGDB, KDB, or MDB, in particular, the case where a processor is being held by a debugger inside an int1 handler.The
hrtimer_interrupts_savedcount can still end up matching the hrtime value in some cases, resulting in the hard lockup detector tagging processors inside a debugger and executing a panic.The patch below corrects this problem. I did not add this to the
touch_nmi_functiondirectly becuase of possible affects on timing issues since the function is widely used by drivers and modules.I have tested this patch and it fixes the problem for kernel debuggers stopping errant hard lockup events when processors are spinning inside the debugger.
Signed-off-by Jeff Merkey <[email protected]>--- kernel/watchdog.c | 7 +++++++ 1 file changed, 7 insertions(+)kernel/watchdog.c +7
-1@@ -283,6 +283,13 @@ static bool is_hardlockup(void)__this_cpu_write (hrtimer_interrupts_saved, hrint); return false; } + +void touch_hardlockup_watchdog (void ) +{ +__this_cpu_write (hrtimer_interrupts_saved,0 ); +} +EXPORT_SYMBOL_GPL (touch_hardlockup_watchdog); +#endif static int is_softlockup (unsigned long touch_ts) --Quoted 3 lines
1.8.3.1
I got to the bottom of it. It's related to the hardware I am using. One of the processors is faulting and hanging due to an existing bug in the
hw_breakpointhandler not setting the resume flag (I have previously reported it and submitted a patch). This breaks your code, but there's nothing you can do about it.
Quoted 7 lines
There is a severe bug in
hw_breakpoint.cthat causes int1 recursion and this whole "lazy debug register switching" nonsense does not work properly. I am probably the first person to actually test this code path robustly. I applied the patch that fixes this bug inhw_breakpoint.cand the problem with your code firing off and ignoring the touch flag
went away.
Ah, good to know. Thanks! I'll drop this patch then.
Cheers,
Don
Quoted 61 lines · Jeff Merkey
Quoted 42 lines · Jeff Merkey
The current
touch_nmi_watchdog()function in /kernel/watchdog.c does not always catch all cases when a processor is spinning in the nmi handler inside either KGDB, KDB, or MDB, in particular, the case where a processor is being held by a debugger inside an int1 handler.The
hrtimer_interrupts_savedcount can still end up matching the hrtime value in some cases, resulting in the hard lockup detector tagging processors inside a debugger and executing a panic.The patch below corrects this problem. I did not add this to the
touch_nmi_functiondirectly becuase of possible affects on timing issues since the function is widely used by drivers and modules.I have tested this patch and it fixes the problem for kernel debuggers stopping errant hard lockup events when processors are spinning inside the debugger.
Signed-off-by Jeff Merkey <[email protected]>--- kernel/watchdog.c | 7 +++++++ 1 file changed, 7 insertions(+)kernel/watchdog.c +7
-1@@ -283,6 +283,13 @@ static bool is_hardlockup(void)__this_cpu_write (hrtimer_interrupts_saved, hrint); return false; } + +void touch_hardlockup_watchdog (void ) +{ +__this_cpu_write (hrtimer_interrupts_saved,0 ); +} +EXPORT_SYMBOL_GPL (touch_hardlockup_watchdog); +#endif static int is_softlockup (unsigned long touch_ts) --Quoted 3 lines
1.8.3.1
I got to the bottom of it. It's related to the hardware I am using. One of the processors is faulting and hanging due to an existing bug in the
hw_breakpointhandler not setting the resume flag (I have previously reported it and submitted a patch). This breaks your code, but there's nothing you can do about it.There is a severe bug in
hw_breakpoint.cthat causes int1 recursion and this whole "lazy debug register switching" nonsense does not work properly. I am probably the first person to actually test this code path robustly. I applied the patch that fixes this bug inhw_breakpoint.cand the problem with your code firing off and ignoring the touch flag
went away.
Quoted 2 lines
Jeff
Wow, I figured it out. What's really needed here is the ability to touch all the processors from just one processor. That's what's missing. This per processor nonsense doesn't fly here. A debugger needs to be able to turn off your stuff (it needs an on/off switch) completely when needed.
I'll submit a patch for that. I'll just maintain the working version in my patch for the debugger so people can get a working, stable, debugable kernel and not a broken one until fixes start showing up in the tree.
Jeff