Skip to contentSkip to conversation

Fix misrouted interrupts deadlocks

RFC PATCH In review Pavel Emelianov 4 participants 8 messages Nov 10, 2006 – Nov 21, 2006 IRQ

lore
Share & tools
lore
#0 lore raw

RFC PATCH 1/1 Fix misrouted interrupts deadlocks

While testing kernel on machine with "irqpoll" option I've caught such a lockup:

	__do_IRQ()
	   spin_lock(&desc->lock);
           desc->chip->ack(); /* IRQ is ACKed */
	note_interrupt()
	misrouted_irq()
	handle_IRQ_event()
           if (...)
	      local_irq_enable_in_hardirq();
	/* interrupts are enabled from now */
	...
	__do_IRQ() /* same IRQ we've started from */
	   spin_lock(&desc->lock); /* LOCKUP */

Looking at misrouted_irq() code I've found that a potential deadlock like this can also take place:

1CPU:
__do_IRQ()
spin_lock(&desc->lock); /* irq = A */
misrouted_irq()

   for (i = 1; i < NR_IRQS; i++) {
      spin_lock(&desc->lock); /* irq = B */
      if (desc->status & IRQ_INPROGRESS) {

2CPU:
__do_IRQ()
spin_lock(&desc->lock); /* irq = B */
misrouted_irq()

   for (i = 1; i < NR_IRQS; i++) {
      spin_lock(&desc->lock); /* irq = A */
      if (desc->status & IRQ_INPROGRESS) {

As the second lock on booth CPUs is taken before checking that this irq is being handled in another processor this may cause a deadlock. This issue is only theoretical.

I propose the attached patch to fix booth problems: when trying to handle misrouted IRQ active desc->lock may be unlocked.

Please comment.

kernel/irq/spurious.c renamed from kernel/irq/spurious.c.irqlockup +5 -1
@@ -147,7 +147,11 @@ void note_interrupt(unsigned int irq, st
 	if (unlikely(irqfixup)) {
 		/* Don't punish working computers */
 		if ((irqfixup == 2 && irq == 0) || action_ret == IRQ_NONE) {
-			int ok = misrouted_irq(irq);
+			int ok;
+
+			spin_unlock(&desc->lock);
+			ok = misrouted_irq(irq);
+			spin_lock(&desc->lock);
 			if (action_ret == IRQ_NONE)
 				desc->irqs_unhandled -= ok;
 		}
Acked-by on v1 1/1
#1 lore raw

kernel/irq/spurious.c void note_interrupt(unsigned int irq, st · line 154

Quoted 6 lines · Pavel Emelianov
kernel/irq/spurious.c renamed from kernel/irq/spurious.c.irqlockup +5 -1
-                       int ok = misrouted_irq(irq);
+                       int ok;
+
+                       spin_unlock(&desc->lock);
+                       ok = misrouted_irq(irq);
+                       spin_lock(&desc->lock); 

your fix looks reasonable to me - it's a thinko to call misrouted_irq() with the descriptor lock still held. (btw., how did you find it - lockdep spinlock debugging or NMI watchdog?)

Ingo

#2 lore raw
Quoted 11 lines · Ingo Molnar

kernel/irq/spurious.c void note_interrupt(unsigned int irq, st · line 154

Quoted 6 lines · Pavel Emelianov
kernel/irq/spurious.c renamed from kernel/irq/spurious.c.irqlockup +5 -1
-                       int ok = misrouted_irq(irq);
+                       int ok;
+
+                       spin_unlock(&desc->lock);
+                       ok = misrouted_irq(irq);
+                       spin_lock(&desc->lock); 

your fix looks reasonable to me - it's a thinko to call misrouted_irq() with the descriptor lock still held. (btw., how did you find it - lockdep spinlock debugging or NMI watchdog?)

It was NMI watchdog who reported the deadlock. With lockdep turned on it wouldn't be caught - local_irq_enable_in_hardirq() is nothing but a "do { } while (0)" if CONFIG_LOCKDEP=y :)

The second issue (with 2 cpus involved) was discovered by code examining.

Quoted 10 lines

Ingo

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

#3 lore raw
In reply to Pavel Emelianov
Quoted 12 lines · Pavel Emelianov

As the second lock on booth CPUs is taken before checking that this irq is being handled in another processor this may cause a deadlock. This issue is only theoretical.

I propose the attached patch to fix booth problems: when trying to handle misrouted IRQ active desc->lock may be unlocked.

Please comment.

kernel/irq/spurious.c void note_interrupt(unsigned int irq, st · line 154

Quoted 9 lines
kernel/irq/spurious.c renamed from kernel/irq/spurious.c.irqlockup +5 -1
@@ -147,7 +147,11 @@ void note_interrupt(unsigned int irq, st
 	if (unlikely(irqfixup)) {
 		/* Don't punish working computers */
 		if ((irqfixup == 2 && irq == 0) || action_ret == IRQ_NONE) {
-			int ok = misrouted_irq(irq);
+			int ok;
+
+			spin_unlock(&desc->lock);
+			ok = misrouted_irq(irq);
+			spin_lock(&desc->lock);

Hi Pavel,

Till -rc5, I was able to boot a kernel with irqpoll option and in -rc6 I can't. It simply hangs. I suspect it is realted to this change. I have yet to confirm that. But before that one observation.

Not at every place note_interrupt() is called with desc->lock() held. For example, handle_level_irq(). I enabled spin lock debugging and I run into following BUG().

PID hash table entries: 256 (order: 8, 2048 bytes) time.c: Using 3.579545 MHz WALL PM GTOD PIT/TSC timer. time.c: Detected 3000.218 MHz processor.

=====================================
[ BUG: bad unlock balance detected! ]
-------------------------------------
swapper/0 is trying to release lock (&irq_desc_lock_class) at:
[<ffffffff8104c673>] note_interrupt+0x7a/0x22b
but there are no more locks to release!

other info that might help us debug this:
no locks held by swapper/0.

stack backtrace:

Call Trace:
  [<ffffffff8100a6f9>] show_trace+0x34/0x47
  [<ffffffff8100a71e>] dump_stack+0x12/0x17
  [<ffffffff8103caba>] print_unlock_inbalance_bug+0xfb/0x106
  [<ffffffff8103e6e5>] lock_release+0x89/0x128
  [<ffffffff81332d96>] _spin_unlock+0x17/0x20
  [<ffffffff8104c673>] note_interrupt+0x7a/0x22b
  [<ffffffff8104d131>] handle_level_irq+0xab/0xea
  [<ffffffff8100b776>] do_IRQ+0xf4/0x132
  [<ffffffff81009956>] ret_from_intr+0x0/0xf
DWARF2 unwinder stuck at ret_from_intr+0x0/0xf

Leftover inexact backtrace:

  <IRQ>  <EOI>  [<ffffffff8159f61d>] start_kernel+0x178/0x2f6
  [<ffffffff8159f625>] start_kernel+0x180/0x2f6
  [<ffffffff8159f61d>] start_kernel+0x178/0x2f6
  [<ffffffff8159f13e>] _sinittext+0x13e/0x142

BUG: spinlock lockup on CPU#0, swapper/0, ffffffff81586140

Call Trace:
  [<ffffffff8100a6f9>] show_trace+0x34/0x47
  [<ffffffff8100a71e>] dump_stack+0x12/0x17
  [<ffffffff811457c8>] _raw_spin_lock+0xca/0xe8
  [<ffffffff8104d139>] handle_level_irq+0xb3/0xea
  [<ffffffff8100b776>] do_IRQ+0xf4/0x132
  [<ffffffff81009956>] ret_from_intr+0x0/0xf
DWARF2 unwinder stuck at ret_from_intr+0x0/0xf

Leftover inexact backtrace:

  <IRQ>  <EOI>  [<ffffffff8159f61d>] start_kernel+0x178/0x2f6
  [<ffffffff8159f625>] start_kernel+0x180/0x2f6
  [<ffffffff8159f61d>] start_kernel+0x178/0x2f6
  [<ffffffff8159f13e>] _sinittext+0x13e/0x142

Thanks
Vivek

#4 lore raw
Quoted 25 lines · Vivek Goyal
Quoted 12 lines · Pavel Emelianov

As the second lock on booth CPUs is taken before checking that this irq is being handled in another processor this may cause a deadlock. This issue is only theoretical.

I propose the attached patch to fix booth problems: when trying to handle misrouted IRQ active desc->lock may be unlocked.

Please comment.

kernel/irq/spurious.c void note_interrupt(unsigned int irq, st · line 154

Quoted 9 lines
kernel/irq/spurious.c renamed from kernel/irq/spurious.c.irqlockup +5 -1
@@ -147,7 +147,11 @@ void note_interrupt(unsigned int irq, st
 	if (unlikely(irqfixup)) {
 		/* Don't punish working computers */
 		if ((irqfixup == 2 && irq == 0) || action_ret == IRQ_NONE) {
-			int ok = misrouted_irq(irq);
+			int ok;
+
+			spin_unlock(&desc->lock);
+			ok = misrouted_irq(irq);
+			spin_lock(&desc->lock);

Hi Pavel,

Quoted 4 lines

Till -rc5, I was able to boot a kernel with irqpoll option and in -rc6 I can't. It simply hangs. I suspect it is realted to this change. I have yet to confirm that. But before that one observation.

Hi Pavel,

If I backout your changes, everything works fine. So it looks like that the problem I am facing is because of your patch but I don't have a logical explanation yet that why the problem is there. Just realasing a lock which is not currently acquired should not hang the system?

Thanks
Vivek

#5 lore raw
Quoted 31 lines · Vivek Goyal
Quoted 25 lines · Vivek Goyal
Quoted 12 lines · Pavel Emelianov

As the second lock on booth CPUs is taken before checking that this irq is being handled in another processor this may cause a deadlock. This issue is only theoretical.

I propose the attached patch to fix booth problems: when trying to handle misrouted IRQ active desc->lock may be unlocked.

Please comment.

kernel/irq/spurious.c void note_interrupt(unsigned int irq, st · line 154

Quoted 9 lines
kernel/irq/spurious.c renamed from kernel/irq/spurious.c.irqlockup +5 -1
@@ -147,7 +147,11 @@ void note_interrupt(unsigned int irq, st
 	if (unlikely(irqfixup)) {
 		/* Don't punish working computers */
 		if ((irqfixup == 2 && irq == 0) || action_ret == IRQ_NONE) {
-			int ok = misrouted_irq(irq);
+			int ok;
+
+			spin_unlock(&desc->lock);
+			ok = misrouted_irq(irq);
+			spin_lock(&desc->lock);

Hi Pavel,

Quoted 4 lines

Till -rc5, I was able to boot a kernel with irqpoll option and in -rc6 I can't. It simply hangs. I suspect it is realted to this change. I have yet to confirm that. But before that one observation.

Quoted 7 lines

Hi Pavel,

If I backout your changes, everything works fine. So it looks like that the problem I am facing is because of your patch but I don't have a logical explanation yet that why the problem is there. Just realasing a lock which is not currently acquired should not hang the system?

Some more data regarding this issue.

For my system it gets locked in following sequence.

handle_level_irq() {
	spin_unlock(desc->lock)
	......
	note_interrupt() {
		/* Called without desc->lock held */
		spin_unlock(desc->lock)
		misrouted_irq()
		spin_lock(desc->lock)	
	}
	spin_lock(desc->lock) /* Lockup */ 	
}

So basically problems seems to be due to calling conventions of note_interrupt(). In few cases we call it with desc->lock held and in other cases we call it with desc->lock not held.

I guess, note_interrupt() should restore the desc->lock status back to the state of the lock when function was entered so that caller does not lockup.

Thanks
Vivek

#6 lore raw
In reply to Vivek Goyal
Quoted 29 lines · Vivek Goyal
Quoted 23 lines · Vivek Goyal
Quoted 11 lines · Pavel Emelianov

As the second lock on booth CPUs is taken before checking that this irq is being handled in another processor this may cause a deadlock. This issue is only theoretical.

I propose the attached patch to fix booth problems: when trying to handle misrouted IRQ active desc->lock may be unlocked.

Please comment.

kernel/irq/spurious.c void note_interrupt(unsigned int irq, st · line 154

Quoted 9 lines
kernel/irq/spurious.c renamed from kernel/irq/spurious.c.irqlockup +5 -1
@@ -147,7 +147,11 @@ void note_interrupt(unsigned int irq, st
 	if (unlikely(irqfixup)) {
 		/* Don't punish working computers */
 		if ((irqfixup == 2 && irq == 0) || action_ret == IRQ_NONE) {
-			int ok = misrouted_irq(irq);
+			int ok;
+
+			spin_unlock(&desc->lock);
+			ok = misrouted_irq(irq);
+			spin_lock(&desc->lock);

Hi Pavel,

Quoted 4 lines

Till -rc5, I was able to boot a kernel with irqpoll option and in -rc6 I can't. It simply hangs. I suspect it is realted to this change. I have yet to confirm that. But before that one observation.

Quoted 6 lines

Hi Pavel,

If I backout your changes, everything works fine. So it looks like that the problem I am facing is because of your patch but I don't have a logical explanation yet that why the problem is there. Just realasing a lock which is not currently acquired should not hang the system?

Without this patch my kernel hanged in another place. I'll look over the interrupt code again. I suspect that just another fix is required.

#7 lore raw
In reply to Vivek Goyal
Quoted 6 lines

Hi Pavel,

If I backout your changes, everything works fine. So it looks like that the problem I am facing is because of your patch but I don't have a logical explanation yet that why the problem is there. Just realasing a lock which is not currently acquired should not hang the system?

Hm... A simple grep over the code showed that note_interrupt is called w/o desc->lock in all places but __do_IRQ(). And this looks like an error at least for the following reason:
note_interrupt() calls __report_bad_irq() and __report_bad_irq() does require desc->lock to be held. So I suppose that we have to do spin_lock(&desc->lock) before calling note_interrupt(). I'll prepare a patch in a moment.