hangcheck-timer module
Folks,
Attached is a module, hangcheck-timer. It is used to detect
when the system goes out to lunch for a period of time, such as when a driver like qla2x00 udelays a bunch.
The module sets a timer. When the timer goes off, it then uses
the TSC (warning: portability needed) to determine how much real time has passed.
On a normal system, the real elapsed time will be almost
identical to the expected timer duration. However, if a device decided to udelay for 60 seconds (or some other circumstance), the module takes notice. If the margin of error passes a threshold, the machine is rebooted.
The module is currently used in a cluster environment. After
some time out to lunch, the rest of the cluster will have given up on a machine. If the machine suddenly comes back and assumes it is still "live", bad things can happen.
We can also see use for this in a debugging sense, for kernel
hangs as well as driver code. That's why I'm proposing it for general inclusion.
Comments? Thoughts?
Joel
Building:
The module should happily build against most 2.4 kernels. The
usual module building compile line:
gcc -I /scratch/jlbec/kernel/linux-2.4.20-rc2/include \ -DMODULE -D__KERNEL__ -DLINUX -c -o hangcheck-timer.o \ hangcheck-timer.c
Running:
Load the module with insmod. There are two options.
"hangcheck_tick=<seconds>" specifies the timer timeout, and "hangcheck_margin=<seconds" specifies the margin of error.
Joel
Signature · 8 lines
"Friends may come and go, but enemies accumulate."
- Thomas Jones
Joel Becker
Senior Member of Technical Staff
Oracle Corporation
E-mail: [email protected]
Phone: (650) 506-8127Quoted 30 lines · Joel Becker
Folks,
Attached is a module, hangcheck-timer. It is used to detect
when the system goes out to lunch for a period of time, such as when a driver like qla2x00 udelays a bunch.
The module sets a timer. When the timer goes off, it then uses
the TSC (warning: portability needed) to determine how much real time has passed.
On a normal system, the real elapsed time will be almost
identical to the expected timer duration. However, if a device decided to udelay for 60 seconds (or some other circumstance), the module takes notice. If the margin of error passes a threshold, the machine is rebooted.
The module is currently used in a cluster environment. After
some time out to lunch, the rest of the cluster will have given up on a machine. If the machine suddenly comes back and assumes it is still "live", bad things can happen.
We can also see use for this in a debugging sense, for kernel
hangs as well as driver code. That's why I'm proposing it for general inclusion.
Comments? Thoughts?Joel
Building:
The module should happily build against most 2.4 kernels. The
usual module building compile line:gcc -I /scratch/jlbec/kernel/linux-2.4.20-rc2/include \ -DMODULE -D__KERNEL__ -DLINUX -c -o hangcheck-timer.o \ hangcheck-timer.c
Quoted 7 lines
Running:
Load the module with insmod. There are two options.
"hangcheck_tick=<seconds>" specifies the timer timeout, and "hangcheck_margin=<seconds" specifies the margin of error.Joel
There is already an NMI watchdog that is better than what you propose, because it will also catch cases where something gets stuck with interrupts disabled.
--
Brian Gerst
Quoted 8 lines · Brian Gerst
Quoted 3 lines · Joel Becker
Attached is a module, hangcheck-timer. It is used to detect
when the system goes out to lunch for a period of time, such as when a driver like qla2x00 udelays a bunch.There is already an NMI watchdog that is better than what you propose, because it will also catch cases where something gets stuck with interrupts disabled.
The issue at hand is not permanent hangs. The issue is hangs
that return. Consider a clustering enviornment where the other nodes have given up on the delayed node and clean up after it. When the hang finally ends, the node still thinks it is "alive" and happily scribbles to places it shouldn't.
udelay will not ever trigger the NMI watchdog, as it is running
on the processor, so the cpu timer will run happily. But as far as everything higher up (kernel + userspace), the delay will be unnoticed and bad things can happen.
Joel
Signature · 10 lines
"I'm so tired of being tired, Sure as night will follow day. Most things I worry about Never happen anyway." Joel Becker Senior Member of Technical Staff Oracle Corporation E-mail: [email protected] Phone: (650) 506-8127