watchdog: print all locks on a softlock
PATCH 1/1 watchdog: print all locks on a softlock
If the CPU hits a softlockup this patch will also have it print the information about all locks being held on the system. This might help determine if a lock is being held too long leading to this problem.
--- kernel/watchdog.c | 1 + 1 file changed, 1 insertion(+)
kernel/watchdog.c +1 -0
@@ -322,6 +322,7 @@ static enum hrtimer_restart watchdog_timer_fn(struct hrtimer *hrtimer) current->comm, task_pid_nr(current));print_modules ();print_irqtrace_events (current); +debug_show_all_locks ();if (regs)show_regs (regs);else
Signature · 1 line
1.9.0
Quoted 3 lines · Eric Paris
If the CPU hits a softlockup this patch will also have it print the information about all locks being held on the system. This might help determine if a lock is being held too long leading to this problem.
I am not sure this helps you. A softlockup is the result of pre-emption disabled, ie the scheduler not being called after 60 seconds. Holding a lock does not disable pre-emption usually. So I don't think this is going to add anything.
Are you trying to debug a hung task? The the hung_task thread checks to see if a task hasn't scheduled in 2 minutes or so. That could be the result of long lock (but that output already dumps the lockdep stuff).
Cheers,
Don
Quoted 26 lines
Signed-off-by Eric Paris <[email protected]>Cc Frederic Weisbecker <[email protected]>Cc Andrew Morton <[email protected]>Cc Don Zickus <[email protected]>Cc Michal Hocko <[email protected]>Cc Ben Zhang <[email protected]>--- kernel/watchdog.c | 1 + 1 file changed, 1 insertion(+)kernel/watchdog.c +1
-0@@ -322,6 +322,7 @@ static enum hrtimer_restart watchdog_timer_fn(struct hrtimer *hrtimer) current->comm, task_pid_nr(current));print_modules ();print_irqtrace_events (current); +debug_show_all_locks ();if (regs)show_regs (regs);else Signature · 1 line
1.9.0
Quoted 5 lines · Don Zickus
Quoted 3 lines · Eric Paris
If the CPU hits a softlockup this patch will also have it print the information about all locks being held on the system. This might help determine if a lock is being held too long leading to this problem.
Quoted 8 lines
I am not sure this helps you. A softlockup is the result of pre-emption disabled, ie the scheduler not being called after 60 seconds. Holding a lock does not disable pre-emption usually. So I don't think this is going to add anything.
Are you trying to debug a hung task? The the
hung_taskthread checks to see if a task hasn't scheduled in 2 minutes or so. That could be the result of long lock (but that output already dumps the lockdep stuff).
There may be some deadlocks that lockdep doesn't detect yet. 2 example:
1) spinlock <-> IPI dependency
CPU 0 CPU 1
--------------------------------------------------------
spin_lock_irq(A)
smp_send_function_single_async(CPU 1, func)
//IPI
func {
spin_lock(1)
}But this should be resolved with a virtual lock on the IPI functions. I should try that.
2) rwlock <-> IPI
CPU 0 CPU 1
--------------------------------------------------------
read_lock(A)
write_lock_irq(A)
smp_send_function_single(CPU 1, func)
//IPI never happens
This one is much trickier.
Anyway those are the only scenario I know of but there may be more. When possible we want to extend lockdep to detect new scenarios of deadlock but we don't have the guarantee that it can detect everything.
So, could be useful...
2014-05-01 22:09 GMT+02:00 Frederic Weisbecker <[email protected]>:
Quoted 29 lines · Don Zickus
Quoted 5 lines · Eric Paris
Quoted 3 lines
If the CPU hits a softlockup this patch will also have it print the information about all locks being held on the system. This might help determine if a lock is being held too long leading to this problem.
Quoted 8 lines
I am not sure this helps you. A softlockup is the result of pre-emption disabled, ie the scheduler not being called after 60 seconds. Holding a lock does not disable pre-emption usually. So I don't think this is going to add anything.
Are you trying to debug a hung task? The the
hung_taskthread checks to see if a task hasn't scheduled in 2 minutes or so. That could be the result of long lock (but that output already dumps the lockdep stuff).There may be some deadlocks that lockdep doesn't detect yet. 2 example:
1) spinlock <-> IPI dependency
CPU 0 CPU 1 -------------------------------------------------------- spin_lock_irq(A) smp_send_function_single_async(CPU 1, func) //IPI func { spin_lock(1) }
Quoted 2 lines
But this should be resolved with a virtual lock on the IPI functions. I should try that.
So actually this one above shouldn't be a problem because the _async version doesn't wait for the IPI to complete. But the below still looks possible.
Quoted 17 lines
2) rwlock <-> IPI
CPU 0 CPU 1
--------------------------------------------------------
read_lock(A)
write_lock_irq(A)
smp_send_function_single(CPU 1, func)
//IPI never happensThis one is much trickier.
Anyway those are the only scenario I know of but there may be more. When possible we want to extend lockdep to detect new scenarios of deadlock but we don't have the guarantee that it can detect everything.
So, could be useful...
Quoted 32 lines · Frederic Weisbecker
Quoted 5 lines · Don Zickus
Quoted 3 lines · Eric Paris
If the CPU hits a softlockup this patch will also have it print the information about all locks being held on the system. This might help determine if a lock is being held too long leading to this problem.
Quoted 8 lines
I am not sure this helps you. A softlockup is the result of pre-emption disabled, ie the scheduler not being called after 60 seconds. Holding a lock does not disable pre-emption usually. So I don't think this is going to add anything.
Are you trying to debug a hung task? The the
hung_taskthread checks to see if a task hasn't scheduled in 2 minutes or so. That could be the result of long lock (but that output already dumps the lockdep stuff).There may be some deadlocks that lockdep doesn't detect yet. 2 example:
1) spinlock <-> IPI dependency
CPU 0 CPU 1 -------------------------------------------------------- spin_lock_irq(A) smp_send_function_single_async(CPU 1, func) //IPI func { spin_lock(1) }But this should be resolved with a virtual lock on the IPI functions. I should try that.
Quoted 8 lines
2) rwlock <-> IPI
CPU 0 CPU 1
--------------------------------------------------------
read_lock(A)
write_lock_irq(A)
smp_send_function_single(CPU 1, func)
//IPI never happens
The hardlockup detector would go off here. And dumping all the cpus in the system (something we don't do today), would show this scenario. I see this scenario a lot during page flushes on RHEL (a lot being once every other month or so).
Cheers,
Don
Quoted 8 lines
This one is much trickier.
Anyway those are the only scenario I know of but there may be more. When possible we want to extend lockdep to detect new scenarios of deadlock but we don't have the guarantee that it can detect everything.
So, could be useful...