how to debug a deadlock'ed kernel?
Anyone got any good tips on how to debug a SMP system that is locked up in a deadlock situation in the kernel? I'm working on a kernel module, and after some number of hours of stress testing, the box locks up. None of the sysrq options show anything on the display, though the reBoot option does reboot the system. RedHat 6.2 and its 2.2.14 kernel. Doesn't hang for me on 2.4, so I need to debug it here...
Any hints?
thx.bri.
Quoted 8 lines · On Tue, 11 Dec 2001 13:57:52 -0600 Brian Horton <[email protected]> wrote:
Anyone got any good tips on how to debug a SMP system that is locked up in a deadlock situation in the kernel? I'm working on a kernel module, and after some number of hours of stress testing, the box locks up. None of the sysrq options show anything on the display, though the reBoot option does reboot the system. RedHat 6.2 and its 2.2.14 kernel. Doesn't hang for me on 2.4, so I need to debug it here...
Any hints?
Try using a serial console (activate it in your kernel config and hook up another PC to the serial port - if it's oopsing, you should see the oops over the serial line.)
Also, I believe you can use kdb via serial as well (although I've never tried).
Bruce
Quoted 6 lines · Brian Horton
Anyone got any good tips on how to debug a SMP system that is locked up in a deadlock situation in the kernel? I'm working on a kernel module, and after some number of hours of stress testing, the box locks up. None of the sysrq options show anything on the display, though the reBoot option does reboot the system. RedHat 6.2 and its 2.2.14 kernel. Doesn't hang for me on 2.4, so I need to debug it here...
You might try Keith Owen's kdb. When you lock-up, hit <pause> which brings up a kdb prompt. From there you can do backtraces, memory examination, and disassembly on either processor.
It's often quite helpful to modify your test to narrow down what is making it crash and/or make it happen faster. Reads vs writes, short/long packets, etc.
Signature · 1 line
"Love the dolphins," she advised him. "Write by W.A.S.T.E.."
Quoted 9 lines · Brian Horton
Anyone got any good tips on how to debug a SMP system that is locked up in a deadlock situation in the kernel? I'm working on a kernel module, and after some number of hours of stress testing, the box locks up. None of the sysrq options show anything on the display, though the reBoot option does reboot the system. RedHat 6.2 and its 2.2.14 kernel. Doesn't hang for me on 2.4, so I need to debug it here...
Any hints?
First read about the NMI boot option in Documentation/nmi_watchdog.txt. If you have this turned on and are not oopsing, then the timer (at least) is interrupting. The next step I would take would be to used either kdb (no experience) or kgdb. I have my own version of this if you are interested. It does, however, require an RS232 (serial) connection to a host machine.
I don't know about kdb, but kgdb (my version) uses the NMI to trap the other cpus and also traps NMIs on the way to oopsing.
Signature · 3 lines
George [email protected] High-res-timers: http://sourceforge.net/projects/high-res-timers/ Real time sched: http://sourceforge.net/projects/rtsched/
Thanks, I'll try the nmi_watchdog option out. It appears to not be in the 2.2.14 kernel, but is in a 2.2.19 kernel that I have from RedHat.
Is there a version of kgdb that works with 2.2.x kernels? I only see 2.4 kernels on the web page (http://kgdb.sourceforge.net)
thx.bri.
Quoted 25 lines · george anzinger
Quoted 9 lines · Brian Horton
Anyone got any good tips on how to debug a SMP system that is locked up in a deadlock situation in the kernel? I'm working on a kernel module, and after some number of hours of stress testing, the box locks up. None of the sysrq options show anything on the display, though the reBoot option does reboot the system. RedHat 6.2 and its 2.2.14 kernel. Doesn't hang for me on 2.4, so I need to debug it here...
Any hints?
First read about the NMI boot option in
Documentation/nmi_watchdog.txt. If you have this turned on and are not oopsing, then the timer (at least) is interrupting. The next step I would take would be to used either kdb (no experience) or kgdb. I have my own version of this if you are interested. It does, however, require an RS232 (serial) connection to a host machine.I don't know about kdb, but kgdb (my version) uses the NMI to trap the other cpus and also traps NMIs on the way to oopsing.
--
George [email protected]
High-res-timers: http://sourceforge.net/projects/high-res-timers/
Real time sched: http://sourceforge.net/projects/rtsched/
Quoted 2 lines · Brian Horton
Thanks, I'll try the
nmi_watchdogoption out. It appears to not be in the 2.2.14 kernel, but is in a 2.2.19 kernel that I have from RedHat.
It's there by default in SMP, you just have to enable it with nmiwatchdog=1 at boot (or something). I didn't mention it because it probably won't help your problem: as you can use the magic sysrq keys to reboot, you are not deadlocked with interrupts off, therefore the timer interrupt will keep the NMI watchdog from ever firing. NMI watchdog is mostly of use when you can't even get the capslock light to toggle..
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Anyone got any good tips on how to debug a SMP system that is locked up in a deadlock situation in the kernel? I'm working on a kernel module, and after some number of hours of stress testing, the box locks up. None of the sysrq options show anything on the display, though the reBoot option does reboot the system. RedHat 6.2 and its 2.2.14 kernel. Doesn't hang for me on 2.4, so I need to debug it here...
Signature · 1 line
"Love the dolphins," she advised him. "Write by W.A.S.T.E.."
Yup, the nmi_watchdog option didn't work, but I was able to get information I needed with kdb!
thx! .bri.
Quoted 22 lines · Oliver Xymoron
Quoted 2 lines · Brian Horton
Thanks, I'll try the
nmi_watchdogoption out. It appears to not be in the 2.2.14 kernel, but is in a 2.2.19 kernel that I have from RedHat.It's there by default in SMP, you just have to enable it with nmiwatchdog=1 at boot (or something). I didn't mention it because it probably won't help your problem: as you can use the magic sysrq keys to reboot, you are not deadlocked with interrupts off, therefore the timer interrupt will keep the NMI watchdog from ever firing. NMI watchdog is mostly of use when you can't even get the capslock light to toggle..
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Anyone got any good tips on how to debug a SMP system that is locked up in a deadlock situation in the kernel? I'm working on a kernel module, and after some number of hours of stress testing, the box locks up. None of the sysrq options show anything on the display, though the reBoot option does reboot the system. RedHat 6.2 and its 2.2.14 kernel. Doesn't hang for me on 2.4, so I need to debug it here...
--
"Love the dolphins," she advised him. "Write by W.A.S.T.E.."