Skip to contentSkip to conversation

Add hard/soft lockup debugger entry points

PATCH In review Jeff Merkey 3 participants 6 messages Jan 26, 2016

lore
Share & tools
lore

Jeff Merkey

author
#0 lore raw

PATCH v2 1/1 Add hard/soft lockup debugger entry points

This patch adds an export which can be set by system debuggers to direct the hard lockup and soft lockup detector to trigger a breakpoint exception and enter a debugger if one is active. It is assumed that if someone sets this variable, then an breakpoint handler of some sort will be actively loaded or registered via the notify die handler chain.

This addition is extremely useful for debugging hard and soft lockups real time and quickly from a console debugger.

Signed-off-by Jeff Merkey <[email protected]>
---
 kernel/watchdog.c | 10 ++++++++++
 1 file changed, 10 insertions(+)
kernel/watchdog.c +10 -0
index b3ace6e..c28e58c 100644
@@ -23,6 +23,7 @@
 #include <linux/workqueue.h>
 
 #include <asm/irq_regs.h>
+#include <linux/kgdb.h>
 #include <linux/kvm_para.h>
 #include <linux/perf_event.h>
 #include <linux/kthread.h>

@@ -108,6 +109,9 @@ static DEFINE_PER_CPU(struct perf_event *, watchdog_ev);
 #endif
 static unsigned long soft_lockup_nmi_warn;
 
+int debug_watchdog_lockups;
+EXPORT_SYMBOL_GPL(debug_watchdog_lockups);
+
 /* boot commands */
 /*
  * Should we panic when a soft-lockup or hard-lockup occurs:

@@ -358,6 +362,9 @@ static void watchdog_overflow_callback(struct perf_event *event,
 		else
 			dump_stack();
 
+		if (debug_watchdog_lockups)
+			arch_kgdb_breakpoint();
+
 		/*
 		 * Perform all-CPU dump only once to avoid multiple hardlockups
 		 * generating interleaving traces

@@ -478,6 +485,9 @@ static enum hrtimer_restart watchdog_timer_fn(struct hrtimer *hrtimer)
 		else
 			dump_stack();
 
+		if (debug_watchdog_lockups)
+			arch_kgdb_breakpoint();
+
 		if (softlockup_all_cpu_backtrace) {
 			/* Avoid generating two back traces for current
 			 * given that one is already made above
Signature · 1 line
1.8.3.1
#1 lore raw
Quoted 6 lines · Jeff Merkey

This patch adds an export which can be set by system debuggers to direct the hard lockup and soft lockup detector to trigger a breakpoint exception and enter a debugger if one is active. It is assumed that if someone sets this variable, then an breakpoint handler of some sort will be actively loaded or registered via the notify die handler chain.

Quoted 7 lines

This addition is extremely useful for debugging hard and soft lockups real time and quickly from a console debugger.

Signed-off-by Jeff Merkey<[email protected]>
---
  kernel/watchdog.c | 10 ++++++++++
  1 file changed, 10 insertions(+)

You probably should reach out to someone who uses this stuff more regularly - I actually wonder if the kgdb_breakpoint() API is the right thing, though, not the internal arch_kgdb_breakpoint().

Of course any of these strategies also assume you are building the kernel with CONFIG_KGDB set, and I'm pretty sure will cause your build to fail if it isn't. You likely need to guard this stuff locally within watchdog.c for !CONFIG_KGDB.

Signature · 2 lines
Chris Metcalf, EZChip Semiconductor
http://www.ezchip.com
author
#2 lore raw
Quoted 21 lines · Chris Metcalf
Quoted 8 lines · Jeff Merkey

This patch adds an export which can be set by system debuggers to direct the hard lockup and soft lockup detector to trigger a breakpoint exception
and enter a debugger if one is active. It is assumed that if someone sets this variable, then an breakpoint handler of some sort will be actively
loaded or registered via the notify die handler chain.

Quoted 7 lines

This addition is extremely useful for debugging hard and soft lockups real time and quickly from a console debugger.

Signed-off-by Jeff Merkey<[email protected]>
---
  kernel/watchdog.c | 10 ++++++++++
  1 file changed, 10 insertions(+)

You probably should reach out to someone who uses this stuff more regularly - I actually wonder if the kgdb_breakpoint() API is the right thing, though, not the internal arch_kgdb_breakpoint().

Quoted 10 lines

Of course any of these strategies also assume you are building the kernel with CONFIG_KGDB set, and I'm pretty sure will cause your build to fail if it isn't. You likely need to guard this stuff locally within watchdog.c for !CONFIG_KGDB.

--
Chris Metcalf, EZChip Semiconductor
http://www.ezchip.com

I'll go review that code carefully. What would ideal is to have the breakpoint abstraction independent of kgdb entirely. That would not be a simple patch but more involved. We really just need the breakpoint instruction for each arch, not all of kgdb and it should not depend on KGB being enabled. I looked over those includes and this may work, but you are correct, it's not squeaky clean to do it that way. I'll get to work on v3.

:-)

Jeff

build bot
#3 lore raw
In reply to Jeff Merkey
reproduce: wget https://git.kernel.org/cgit/linux/kernel/git/wfg/lkp-tests.git/plain/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross # save the attached .config to linux build tree…

Hi Jeff,

[auto build test ERROR on v4.5-rc1]
[also build test ERROR on next-20160125]
[if your patch is applied to the wrong git tree, please drop us a note to help improving the system]

url:    https://github.com/0day-ci/linux/commits/Jeff-Merkey/Add-hard-soft-lockup-debugger-entry-points/20160127-010930
config: xtensa-allyesconfig (attached as .config)

reproduce:
wget https://git.kernel.org/cgit/linux/kernel/git/wfg/lkp-tests.git/plain/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# save the attached .config to linux build tree
make.cross ARCH=xtensa

All errors (new ones prefixed by >>):

kernel/watchdog.c: In function 'watchdog_timer_fn':

>> kernel/watchdog.c:489:4: error: implicit declaration of function 'arch_kgdb_breakpoint' [-Werror=implicit-function-declaration]
       arch_kgdb_breakpoint();
       ^

cc1: some warnings being treated as errors

vim +/arch_kgdb_breakpoint +489 kernel/watchdog.c

483 if (regs)
484 show_regs(regs);
485 else
486 dump_stack();
487
488 if (debug_watchdog_lockups)

 > 489				arch_kgdb_breakpoint();

490
491 if (softlockup_all_cpu_backtrace) {
492 /* Avoid generating two back traces for current

---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation

build bot
#4 lore raw
In reply to Jeff Merkey
reproduce: wget https://git.kernel.org/cgit/linux/kernel/git/wfg/lkp-tests.git/plain/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross # save the attached .config to linux build tree…

Hi Jeff,

[auto build test ERROR on v4.5-rc1]
[also build test ERROR on next-20160125]
[if your patch is applied to the wrong git tree, please drop us a note to help improving the system]

url:    https://github.com/0day-ci/linux/commits/Jeff-Merkey/Add-hard-soft-lockup-debugger-entry-points/20160127-010930
config: sparc64-defconfig (attached as .config)

reproduce:
wget https://git.kernel.org/cgit/linux/kernel/git/wfg/lkp-tests.git/plain/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# save the attached .config to linux build tree
make.cross ARCH=sparc64

All errors (new ones prefixed by >>):

kernel/built-in.o: In function `watchdog_timer_fn':

>> watchdog.c:(.text+0x736ec): undefined reference to `arch_kgdb_breakpoint'

---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation

author
#5 lore raw
Quoted 21 lines · kbuild test robot

Hi Jeff,

[auto build test ERROR on v4.5-rc1]
[also build test ERROR on next-20160125]
[if your patch is applied to the wrong git tree, please drop us a note to help improving the system]

url:
https://github.com/0day-ci/linux/commits/Jeff-Merkey/Add-hard-soft-lockup-debugger-entry-points/20160127-010930
config: sparc64-defconfig (attached as .config) reproduce:
wget
https://git.kernel.org/cgit/linux/kernel/git/wfg/lkp-tests.git/plain/sbin/make.cross
-O ~/bin/make.cross
chmod +x ~/bin/make.cross
# save the attached .config to linux build tree make.cross ARCH=sparc64

All errors (new ones prefixed by >>):

kernel/built-in.o: In function `watchdog_timer_fn':

Quoted 9 lines
Quoted 2 lines
Quoted 2 lines

watchdog.c:(.text+0x736ec): undefined reference to `arch_kgdb_breakpoint'

---
0-DAY kernel test infrastructure Open Source Technology
Center
https://lists.01.org/pipermail/kbuild-all Intel
Corporation

I love this test robot. What an incredibly useful tool. Well, looks like a I quite a bit of work to do.

Jeff