Skip to contentSkip to conversation

watchdog: print all locks on a softlock

PATCH In review Eric Paris 3 participants 5 messages May 1, 2014

lore
Share & tools
lore

Eric Paris

author
#0 lore raw

PATCH 1/1 watchdog: print all locks on a softlock

If the CPU hits a softlockup this patch will also have it print the information about all locks being held on the system. This might help determine if a lock is being held too long leading to this problem.

Cc Frederic Weisbecker <[email protected]>
Cc Andrew Morton <[email protected]>
Cc Don Zickus <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Ben Zhang <[email protected]>
---
 kernel/watchdog.c | 1 +
 1 file changed, 1 insertion(+)
kernel/watchdog.c +1 -0
index 516203e..a027240 100644
@@ -322,6 +322,7 @@ static enum hrtimer_restart watchdog_timer_fn(struct hrtimer *hrtimer)
 			current->comm, task_pid_nr(current));
 		print_modules();
 		print_irqtrace_events(current);
+		debug_show_all_locks();
 		if (regs)
 			show_regs(regs);
 		else
Signature · 1 line
1.9.0
#1 lore raw
Quoted 3 lines · Eric Paris

If the CPU hits a softlockup this patch will also have it print the information about all locks being held on the system. This might help determine if a lock is being held too long leading to this problem.

I am not sure this helps you. A softlockup is the result of pre-emption disabled, ie the scheduler not being called after 60 seconds. Holding a lock does not disable pre-emption usually. So I don't think this is going to add anything.

Are you trying to debug a hung task? The the hung_task thread checks to see if a task hasn't scheduled in 2 minutes or so. That could be the result of long lock (but that output already dumps the lockdep stuff).

Cheers,
Don

Quoted 26 lines
Cc Frederic Weisbecker <[email protected]>
Cc Andrew Morton <[email protected]>
Cc Don Zickus <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Ben Zhang <[email protected]>
---
 kernel/watchdog.c | 1 +
 1 file changed, 1 insertion(+)
kernel/watchdog.c +1 -0
index 516203e..a027240 100644
@@ -322,6 +322,7 @@ static enum hrtimer_restart watchdog_timer_fn(struct hrtimer *hrtimer)
 			current->comm, task_pid_nr(current));
 		print_modules();
 		print_irqtrace_events(current);
+		debug_show_all_locks();
 		if (regs)
 			show_regs(regs);
 		else
Signature · 1 line
1.9.0
#2 lore raw
Quoted 5 lines · Don Zickus
Quoted 3 lines · Eric Paris

If the CPU hits a softlockup this patch will also have it print the information about all locks being held on the system. This might help determine if a lock is being held too long leading to this problem.

Quoted 8 lines

I am not sure this helps you. A softlockup is the result of pre-emption disabled, ie the scheduler not being called after 60 seconds. Holding a lock does not disable pre-emption usually. So I don't think this is going to add anything.

Are you trying to debug a hung task? The the hung_task thread checks to see if a task hasn't scheduled in 2 minutes or so. That could be the result of long lock (but that output already dumps the lockdep stuff).

There may be some deadlocks that lockdep doesn't detect yet. 2 example:

1) spinlock <-> IPI dependency

        CPU 0                                            CPU 1
        --------------------------------------------------------
        spin_lock_irq(A)
        smp_send_function_single_async(CPU 1, func)
                                                         //IPI
                                                         func {
                                                            spin_lock(1)
                                                         }

But this should be resolved with a virtual lock on the IPI functions. I should try that.

2) rwlock <-> IPI

CPU 0 CPU 1
--------------------------------------------------------
read_lock(A)
write_lock_irq(A)
smp_send_function_single(CPU 1, func)
//IPI never happens

This one is much trickier.

Anyway those are the only scenario I know of but there may be more. When possible we want to extend lockdep to detect new scenarios of deadlock but we don't have the guarantee that it can detect everything.

So, could be useful...

#3 lore raw

2014-05-01 22:09 GMT+02:00 Frederic Weisbecker <[email protected]>:

Quoted 29 lines · Don Zickus
Quoted 5 lines · Eric Paris
Quoted 3 lines

If the CPU hits a softlockup this patch will also have it print the information about all locks being held on the system. This might help determine if a lock is being held too long leading to this problem.

Quoted 8 lines

I am not sure this helps you. A softlockup is the result of pre-emption disabled, ie the scheduler not being called after 60 seconds. Holding a lock does not disable pre-emption usually. So I don't think this is going to add anything.

Are you trying to debug a hung task? The the hung_task thread checks to see if a task hasn't scheduled in 2 minutes or so. That could be the result of long lock (but that output already dumps the lockdep stuff).

There may be some deadlocks that lockdep doesn't detect yet. 2 example:

1) spinlock <-> IPI dependency

        CPU 0                                            CPU 1
        --------------------------------------------------------
        spin_lock_irq(A)
        smp_send_function_single_async(CPU 1, func)
                                                         //IPI
                                                         func {
                                                            spin_lock(1)
                                                         }
Quoted 2 lines

But this should be resolved with a virtual lock on the IPI functions. I should try that.

So actually this one above shouldn't be a problem because the _async version doesn't wait for the IPI to complete. But the below still looks possible.

Quoted 17 lines

2) rwlock <-> IPI

CPU 0 CPU 1
--------------------------------------------------------
read_lock(A)
write_lock_irq(A)
smp_send_function_single(CPU 1, func)
//IPI never happens

This one is much trickier.

Anyway those are the only scenario I know of but there may be more. When possible we want to extend lockdep to detect new scenarios of deadlock but we don't have the guarantee that it can detect everything.

So, could be useful...

#4 lore raw
In reply to Frederic Weisbecker
Quoted 32 lines · Frederic Weisbecker
Quoted 5 lines · Don Zickus
Quoted 3 lines · Eric Paris

If the CPU hits a softlockup this patch will also have it print the information about all locks being held on the system. This might help determine if a lock is being held too long leading to this problem.

Quoted 8 lines

I am not sure this helps you. A softlockup is the result of pre-emption disabled, ie the scheduler not being called after 60 seconds. Holding a lock does not disable pre-emption usually. So I don't think this is going to add anything.

Are you trying to debug a hung task? The the hung_task thread checks to see if a task hasn't scheduled in 2 minutes or so. That could be the result of long lock (but that output already dumps the lockdep stuff).

There may be some deadlocks that lockdep doesn't detect yet. 2 example:

1) spinlock <-> IPI dependency

        CPU 0                                            CPU 1
        --------------------------------------------------------
        spin_lock_irq(A)
        smp_send_function_single_async(CPU 1, func)
                                                         //IPI
                                                         func {
                                                            spin_lock(1)
                                                         }

But this should be resolved with a virtual lock on the IPI functions. I should try that.

Quoted 8 lines

2) rwlock <-> IPI

CPU 0 CPU 1
--------------------------------------------------------
read_lock(A)
write_lock_irq(A)
smp_send_function_single(CPU 1, func)
//IPI never happens

The hardlockup detector would go off here. And dumping all the cpus in the system (something we don't do today), would show this scenario. I see this scenario a lot during page flushes on RHEL (a lot being once every other month or so).

Cheers,
Don

Quoted 8 lines

This one is much trickier.

Anyway those are the only scenario I know of but there may be more. When possible we want to extend lockdep to detect new scenarios of deadlock but we don't have the guarantee that it can detect everything.

So, could be useful...