ftrace based hard lockup detector
Quoted 6 lines · Frederic Weisbecker
Like the NMI watchdog, this feature try to detect hard lockups by lurking at the non-progress of the timer interrupts.
You can enable it at boot time by passing the
ftrace_hardlockupparameter. I plan to add a debugfs file to enable/disable at runtime.
Quoted 7 lines
When a hardlockup is detected, it will print a backtrace. Perhaps it would be good to print the locks held from lockdep too?
It only support x86 for the moment, because a kind of generic timer interrupt counter is needed on all archs to have it generic.
Signed-off-by Frederic Weisbecker <[email protected]>
Hi Frederic,
This seems like a rewrite of the NMI lockup code. In my debugging, I simply put ftrace_dump in the NMI lockup, which gives me a ftrace dump as soon as NMI detects a lockup. I'm a bit confused at what this gives us over that?
-- Steve
Quoted 14 lines · Steven Rostedt
Quoted 12 lines · Frederic Weisbecker
Like the NMI watchdog, this feature try to detect hard lockups by lurking at the non-progress of the timer interrupts.
You can enable it at boot time by passing the
ftrace_hardlockupparameter. I plan to add a debugfs file to enable/disable at runtime.When a hardlockup is detected, it will print a backtrace. Perhaps it would be good to print the locks held from lockdep too?
It only support x86 for the moment, because a kind of generic timer interrupt counter is needed on all archs to have it generic.
Quoted 8 lines
Quoted 1 line
Signed-off-by Frederic Weisbecker <[email protected]>Hi Frederic,
This seems like a rewrite of the NMI lockup code. In my debugging, I simply put
ftrace_dumpin the NMI lockup, which gives me a ftrace dump as soon as NMI detects a lockup. I'm a bit confused at what this gives us over that?
this is different from the NMI watchdog in a number of ways:
- it works on all platforms and in all situations where the NMI watchdog does not work.
- in theory it can detect hard lockups in situations where the NMI watchdog is disabled, such as suspend/resume or early bootup. (especially early bootup lockups are nasty and the NMI watchdog is enabled relatively late)
- it could be extended to detect 'soft' lockups too - i.e. we could have a one-stop facility to detect all kinds of "kernel does not seem to progress" lockups.
But it's not as complete as the NMI watchdog: it relies on instrumented function calls rolling on and on during the lockup - that's not the case when we get a hard lockup due to a tight, infinite loop somewhere.
Ingo
Quoted 36 lines · Ingo Molnar
Quoted 14 lines · Steven Rostedt
Quoted 12 lines · Frederic Weisbecker
Like the NMI watchdog, this feature try to detect hard lockups by lurking at the non-progress of the timer interrupts.
You can enable it at boot time by passing the
ftrace_hardlockupparameter. I plan to add a debugfs file to enable/disable at runtime.When a hardlockup is detected, it will print a backtrace. Perhaps it would be good to print the locks held from lockdep too?
It only support x86 for the moment, because a kind of generic timer interrupt counter is needed on all archs to have it generic.
Quoted 8 lines
Quoted 1 line
Signed-off-by Frederic Weisbecker <[email protected]>Hi Frederic,
This seems like a rewrite of the NMI lockup code. In my debugging, I simply put
ftrace_dumpin the NMI lockup, which gives me a ftrace dump as soon as NMI detects a lockup. I'm a bit confused at what this gives us over that?this is different from the NMI watchdog in a number of ways:
- it works on all platforms and in all situations where the NMI watchdog does not work.
- in theory it can detect hard lockups in situations where the NMI watchdog is disabled, such as suspend/resume or early bootup. (especially early bootup lockups are nasty and the NMI watchdog is enabled relatively late)
Quoted 7 lines
- it could be extended to detect 'soft' lockups too - i.e. we could have a one-stop facility to detect all kinds of "kernel does not seem to progress" lockups.
But it's not as complete as the NMI watchdog: it relies on instrumented function calls rolling on and on during the lockup - that's not the case when we get a hard lockup due to a tight, infinite loop somewhere.
Ah, OK, the check is in the function tracer. Hmm, my logdev code had an option to enable tracing at early bootup. Instead of using the normal memory alloction for the ring buffer, it needed to use alloc_bootmem. I wonder if it would be worth it to allow for a tracer to do the same if it needs to be allocated early on (before memory is initialized)?
--Steve
Quoted 47 lines · Steven Rostedt
Quoted 36 lines · Ingo Molnar
Quoted 14 lines · Steven Rostedt
Quoted 12 lines · Frederic Weisbecker
Like the NMI watchdog, this feature try to detect hard lockups by lurking at the non-progress of the timer interrupts.
You can enable it at boot time by passing the
ftrace_hardlockupparameter. I plan to add a debugfs file to enable/disable at runtime.When a hardlockup is detected, it will print a backtrace. Perhaps it would be good to print the locks held from lockdep too?
It only support x86 for the moment, because a kind of generic timer interrupt counter is needed on all archs to have it generic.
Quoted 8 lines
Quoted 1 line
Signed-off-by Frederic Weisbecker <[email protected]>Hi Frederic,
This seems like a rewrite of the NMI lockup code. In my debugging, I simply put
ftrace_dumpin the NMI lockup, which gives me a ftrace dump as soon as NMI detects a lockup. I'm a bit confused at what this gives us over that?this is different from the NMI watchdog in a number of ways:
- it works on all platforms and in all situations where the NMI watchdog does not work.
- in theory it can detect hard lockups in situations where the NMI watchdog is disabled, such as suspend/resume or early bootup. (especially early bootup lockups are nasty and the NMI watchdog is enabled relatively late)
Quoted 7 lines
- it could be extended to detect 'soft' lockups too - i.e. we could have a one-stop facility to detect all kinds of "kernel does not seem to progress" lockups.
But it's not as complete as the NMI watchdog: it relies on instrumented function calls rolling on and on during the lockup - that's not the case when we get a hard lockup due to a tight, infinite loop somewhere.
Quoted 5 lines
Ah, OK, the check is in the function tracer. Hmm, my logdev code had an option to enable tracing at early bootup. Instead of using the normal memory alloction for the ring buffer, it needed to use
alloc_bootmem. I wonder if it would be worth it to allow for a tracer to do the same if it needs to be allocated early on (before memory is initialized)?
Yes, very much so!
Especially when using things like dump_trace options, this would be handy.
Ingo
2009/1/19 Steven Rostedt <[email protected]>:
Quoted 7 lines
Ah, OK, the check is in the function tracer. Hmm, my logdev code had an option to enable tracing at early bootup. Instead of using the normal memory alloction for the ring buffer, it needed to use
alloc_bootmem. I wonder if it would be worth it to allow for a tracer to do the same if it needs to be allocated early on (before memory is initialized)?--Steve
Hi Steve,
Yeah I think that would be a good idea.
BTW, tote that for the moment, this hard lockup detector is basic, but I could even make it relying
to the function graph tracer and then dump the whole stack of functions called. I think that could
give a more precise backtrace.... (and more human readable).