Skip to contentSkip to conversation

ftrace based hard lockup detector

PATCH Steven Rostedt 3 participants 5 messages Jan 19, 2009 1 message not in the archive

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

#1 lore raw
Quoted 6 lines · Frederic Weisbecker

Like the NMI watchdog, this feature try to detect hard lockups by lurking at the non-progress of the timer interrupts.

You can enable it at boot time by passing the ftrace_hardlockup parameter. I plan to add a debugfs file to enable/disable at runtime.

Quoted 7 lines

When a hardlockup is detected, it will print a backtrace. Perhaps it would be good to print the locks held from lockdep too?

It only support x86 for the moment, because a kind of generic timer interrupt counter is needed on all archs to have it generic.

Hi Frederic,

This seems like a rewrite of the NMI lockup code. In my debugging, I simply put ftrace_dump in the NMI lockup, which gives me a ftrace dump as soon as NMI detects a lockup. I'm a bit confused at what this gives us over that?

-- Steve

#2 lore raw
Quoted 14 lines · Steven Rostedt
Quoted 12 lines · Frederic Weisbecker

Like the NMI watchdog, this feature try to detect hard lockups by lurking at the non-progress of the timer interrupts.

You can enable it at boot time by passing the ftrace_hardlockup parameter. I plan to add a debugfs file to enable/disable at runtime.

When a hardlockup is detected, it will print a backtrace. Perhaps it would be good to print the locks held from lockdep too?

It only support x86 for the moment, because a kind of generic timer interrupt counter is needed on all archs to have it generic.

Quoted 8 lines
Quoted 1 line

Hi Frederic,

This seems like a rewrite of the NMI lockup code. In my debugging, I simply put ftrace_dump in the NMI lockup, which gives me a ftrace dump as soon as NMI detects a lockup. I'm a bit confused at what this gives us over that?

this is different from the NMI watchdog in a number of ways:

- it works on all platforms and in all situations where the NMI watchdog does not work.

- in theory it can detect hard lockups in situations where the NMI watchdog is disabled, such as suspend/resume or early bootup. (especially early bootup lockups are nasty and the NMI watchdog is enabled relatively late)

- it could be extended to detect 'soft' lockups too - i.e. we could have a one-stop facility to detect all kinds of "kernel does not seem to progress" lockups.

But it's not as complete as the NMI watchdog: it relies on instrumented function calls rolling on and on during the lockup - that's not the case when we get a hard lockup due to a tight, infinite loop somewhere.

Ingo

#3 lore raw
Quoted 36 lines · Ingo Molnar
Quoted 14 lines · Steven Rostedt
Quoted 12 lines · Frederic Weisbecker

Like the NMI watchdog, this feature try to detect hard lockups by lurking at the non-progress of the timer interrupts.

You can enable it at boot time by passing the ftrace_hardlockup parameter. I plan to add a debugfs file to enable/disable at runtime.

When a hardlockup is detected, it will print a backtrace. Perhaps it would be good to print the locks held from lockdep too?

It only support x86 for the moment, because a kind of generic timer interrupt counter is needed on all archs to have it generic.

Quoted 8 lines
Quoted 1 line

Hi Frederic,

This seems like a rewrite of the NMI lockup code. In my debugging, I simply put ftrace_dump in the NMI lockup, which gives me a ftrace dump as soon as NMI detects a lockup. I'm a bit confused at what this gives us over that?

this is different from the NMI watchdog in a number of ways:

- it works on all platforms and in all situations where the NMI watchdog does not work.

- in theory it can detect hard lockups in situations where the NMI watchdog is disabled, such as suspend/resume or early bootup. (especially early bootup lockups are nasty and the NMI watchdog is enabled relatively late)

Quoted 7 lines

- it could be extended to detect 'soft' lockups too - i.e. we could have a one-stop facility to detect all kinds of "kernel does not seem to progress" lockups.

But it's not as complete as the NMI watchdog: it relies on instrumented function calls rolling on and on during the lockup - that's not the case when we get a hard lockup due to a tight, infinite loop somewhere.

Ah, OK, the check is in the function tracer. Hmm, my logdev code had an option to enable tracing at early bootup. Instead of using the normal memory alloction for the ring buffer, it needed to use alloc_bootmem. I wonder if it would be worth it to allow for a tracer to do the same if it needs to be allocated early on (before memory is initialized)?

--Steve

#4 lore raw
Quoted 47 lines · Steven Rostedt
Quoted 36 lines · Ingo Molnar
Quoted 14 lines · Steven Rostedt
Quoted 12 lines · Frederic Weisbecker

Like the NMI watchdog, this feature try to detect hard lockups by lurking at the non-progress of the timer interrupts.

You can enable it at boot time by passing the ftrace_hardlockup parameter. I plan to add a debugfs file to enable/disable at runtime.

When a hardlockup is detected, it will print a backtrace. Perhaps it would be good to print the locks held from lockdep too?

It only support x86 for the moment, because a kind of generic timer interrupt counter is needed on all archs to have it generic.

Quoted 8 lines
Quoted 1 line

Hi Frederic,

This seems like a rewrite of the NMI lockup code. In my debugging, I simply put ftrace_dump in the NMI lockup, which gives me a ftrace dump as soon as NMI detects a lockup. I'm a bit confused at what this gives us over that?

this is different from the NMI watchdog in a number of ways:

- it works on all platforms and in all situations where the NMI watchdog does not work.

- in theory it can detect hard lockups in situations where the NMI watchdog is disabled, such as suspend/resume or early bootup. (especially early bootup lockups are nasty and the NMI watchdog is enabled relatively late)

Quoted 7 lines

- it could be extended to detect 'soft' lockups too - i.e. we could have a one-stop facility to detect all kinds of "kernel does not seem to progress" lockups.

But it's not as complete as the NMI watchdog: it relies on instrumented function calls rolling on and on during the lockup - that's not the case when we get a hard lockup due to a tight, infinite loop somewhere.

Quoted 5 lines

Ah, OK, the check is in the function tracer. Hmm, my logdev code had an option to enable tracing at early bootup. Instead of using the normal memory alloction for the ring buffer, it needed to use alloc_bootmem. I wonder if it would be worth it to allow for a tracer to do the same if it needs to be allocated early on (before memory is initialized)?

Yes, very much so!

Especially when using things like dump_trace options, this would be handy.

Ingo

#5 lore raw
In reply to Steven Rostedt

2009/1/19 Steven Rostedt <[email protected]>:

Quoted 7 lines

Ah, OK, the check is in the function tracer. Hmm, my logdev code had an option to enable tracing at early bootup. Instead of using the normal memory alloction for the ring buffer, it needed to use alloc_bootmem. I wonder if it would be worth it to allow for a tracer to do the same if it needs to be allocated early on (before memory is initialized)?

--Steve

Hi Steve,

Yeah I think that would be a good idea.

BTW, tote that for the moment, this hard lockup detector is basic, but I could even make it relying
to the function graph tracer and then dump the whole stack of functions called. I think that could
give a more precise backtrace.... (and more human readable).