Fix misrouted interrupts deadlocks
RFC PATCH 1/1 Fix misrouted interrupts deadlocks
While testing kernel on machine with "irqpoll" option I've caught such a lockup:
__do_IRQ()
spin_lock(&desc->lock);
desc->chip->ack(); /* IRQ is ACKed */
note_interrupt()
misrouted_irq()
handle_IRQ_event()
if (...)
local_irq_enable_in_hardirq();
/* interrupts are enabled from now */
...
__do_IRQ() /* same IRQ we've started from */
spin_lock(&desc->lock); /* LOCKUP */Looking at misrouted_irq() code I've found that a potential deadlock like this can also take place:
1CPU:__do_IRQ()
spin_lock(&desc->lock); /* irq = A */misrouted_irq()
for (i = 1; i < NR_IRQS; i++) {
spin_lock(&desc->lock); /* irq = B */
if (desc->status & IRQ_INPROGRESS) {2CPU:__do_IRQ()
spin_lock(&desc->lock); /* irq = B */misrouted_irq()
for (i = 1; i < NR_IRQS; i++) {
spin_lock(&desc->lock); /* irq = A */
if (desc->status & IRQ_INPROGRESS) {As the second lock on booth CPUs is taken before checking that this irq is being handled in another processor this may cause a deadlock. This issue is only theoretical.
I propose the attached patch to fix booth problems: when trying to handle misrouted IRQ active desc->lock may be unlocked.
Please comment.
kernel/irq/spurious.c renamed from kernel/irq/spurious.c.irqlockup +5 -1
@@ -147,7 +147,11 @@ void note_interrupt(unsigned int irq, stif (unlikely (irqfixup)) {/* Don't punish working computers */ if ((irqfixup ==2 && irq ==0 ) || action_ret ==IRQ_NONE ) { -int ok =misrouted_irq (irq); +int ok; + +spin_unlock (&desc->lock); + ok =misrouted_irq (irq); +spin_lock (&desc->lock);if (action_ret ==IRQ_NONE ) desc->irqs_unhandled -= ok; }
kernel/irq/spurious.c void note_interrupt(unsigned int irq, st · line 154
Quoted 6 lines · Pavel Emelianov
kernel/irq/spurious.c renamed from kernel/irq/spurious.c.irqlockup +5
-1-int ok =misrouted_irq (irq); +int ok; + +spin_unlock (&desc->lock); + ok =misrouted_irq (irq); +spin_lock (&desc->lock);
your fix looks reasonable to me - it's a thinko to call misrouted_irq() with the descriptor lock still held. (btw., how did you find it - lockdep spinlock debugging or NMI watchdog?)
Ingo
Quoted 11 lines · Ingo Molnar
kernel/irq/spurious.c void note_interrupt(unsigned int irq, st · line 154
Quoted 6 lines · Pavel Emelianov
kernel/irq/spurious.c renamed from kernel/irq/spurious.c.irqlockup +5
-1-int ok =misrouted_irq (irq); +int ok; + +spin_unlock (&desc->lock); + ok =misrouted_irq (irq); +spin_lock (&desc->lock);your fix looks reasonable to me - it's a thinko to call
misrouted_irq()with the descriptor lock still held. (btw., how did you find it - lockdep spinlock debugging or NMI watchdog?)
It was NMI watchdog who reported the deadlock. With lockdep turned on it wouldn't be caught - local_irq_enable_in_hardirq() is nothing but a "do { } while (0)" if CONFIG_LOCKDEP=y :)
The second issue (with 2 cpus involved) was discovered by code examining.
Quoted 10 lines
Acked-by Ingo Molnar <[email protected]>Ingo
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Quoted 12 lines · Pavel Emelianov
As the second lock on booth CPUs is taken before checking that this irq is being handled in another processor this may cause a deadlock. This issue is only theoretical.
I propose the attached patch to fix booth problems: when trying to handle misrouted IRQ active
desc->lockmay be unlocked.Please comment.
kernel/irq/spurious.c void note_interrupt(unsigned int irq, st · line 154
Quoted 9 lines
kernel/irq/spurious.c renamed from kernel/irq/spurious.c.irqlockup +5
-1@@ -147,7 +147,11 @@ void note_interrupt(unsigned int irq, stif (unlikely (irqfixup)) {/* Don't punish working computers */ if ((irqfixup ==2 && irq ==0 ) || action_ret ==IRQ_NONE ) { -int ok =misrouted_irq (irq); +int ok; + +spin_unlock (&desc->lock); + ok =misrouted_irq (irq); +spin_lock (&desc->lock);
Hi Pavel,
Till -rc5, I was able to boot a kernel with irqpoll option and in -rc6 I can't. It simply hangs. I suspect it is realted to this change. I have yet to confirm that. But before that one observation.
Not at every place note_interrupt() is called with desc->lock() held. For example, handle_level_irq(). I enabled spin lock debugging and I run into following BUG().
PID hash table entries: 256 (order: 8, 2048 bytes) time.c: Using 3.579545 MHz WALL PM GTOD PIT/TSC timer. time.c: Detected 3000.218 MHz processor.
=====================================
[ BUG: bad unlock balance detected! ]
-------------------------------------
swapper/0 is trying to release lock (&irq_desc_lock_class) at:
[<ffffffff8104c673>] note_interrupt+0x7a/0x22b
but there are no more locks to release!
other info that might help us debug this:
no locks held by swapper/0.
stack backtrace: Call Trace: [<ffffffff8100a6f9>] show_trace+0x34/0x47 [<ffffffff8100a71e>] dump_stack+0x12/0x17 [<ffffffff8103caba>] print_unlock_inbalance_bug+0xfb/0x106 [<ffffffff8103e6e5>] lock_release+0x89/0x128 [<ffffffff81332d96>] _spin_unlock+0x17/0x20 [<ffffffff8104c673>] note_interrupt+0x7a/0x22b [<ffffffff8104d131>] handle_level_irq+0xab/0xea [<ffffffff8100b776>] do_IRQ+0xf4/0x132 [<ffffffff81009956>] ret_from_intr+0x0/0xf DWARF2 unwinder stuck at ret_from_intr+0x0/0xf
Leftover inexact backtrace:
<IRQ> <EOI> [<ffffffff8159f61d>] start_kernel+0x178/0x2f6 [<ffffffff8159f625>] start_kernel+0x180/0x2f6 [<ffffffff8159f61d>] start_kernel+0x178/0x2f6 [<ffffffff8159f13e>] _sinittext+0x13e/0x142 BUG: spinlock lockup on CPU#0, swapper/0, ffffffff81586140 Call Trace: [<ffffffff8100a6f9>] show_trace+0x34/0x47 [<ffffffff8100a71e>] dump_stack+0x12/0x17 [<ffffffff811457c8>] _raw_spin_lock+0xca/0xe8 [<ffffffff8104d139>] handle_level_irq+0xb3/0xea [<ffffffff8100b776>] do_IRQ+0xf4/0x132 [<ffffffff81009956>] ret_from_intr+0x0/0xf DWARF2 unwinder stuck at ret_from_intr+0x0/0xf
Leftover inexact backtrace:
<IRQ> <EOI> [<ffffffff8159f61d>] start_kernel+0x178/0x2f6 [<ffffffff8159f625>] start_kernel+0x180/0x2f6 [<ffffffff8159f61d>] start_kernel+0x178/0x2f6 [<ffffffff8159f13e>] _sinittext+0x13e/0x142
Thanks
Vivek
Quoted 25 lines · Vivek Goyal
Quoted 12 lines · Pavel Emelianov
As the second lock on booth CPUs is taken before checking that this irq is being handled in another processor this may cause a deadlock. This issue is only theoretical.
I propose the attached patch to fix booth problems: when trying to handle misrouted IRQ active
desc->lockmay be unlocked.Please comment.
kernel/irq/spurious.c void note_interrupt(unsigned int irq, st · line 154
Quoted 9 lines
kernel/irq/spurious.c renamed from kernel/irq/spurious.c.irqlockup +5
-1@@ -147,7 +147,11 @@ void note_interrupt(unsigned int irq, stif (unlikely (irqfixup)) {/* Don't punish working computers */ if ((irqfixup ==2 && irq ==0 ) || action_ret ==IRQ_NONE ) { -int ok =misrouted_irq (irq); +int ok; + +spin_unlock (&desc->lock); + ok =misrouted_irq (irq); +spin_lock (&desc->lock);Hi Pavel,
Quoted 4 lines
Till -rc5, I was able to boot a kernel with irqpoll option and in -rc6 I can't. It simply hangs. I suspect it is realted to this change. I have yet to confirm that. But before that one observation.
Hi Pavel,
If I backout your changes, everything works fine. So it looks like that the problem I am facing is because of your patch but I don't have a logical explanation yet that why the problem is there. Just realasing a lock which is not currently acquired should not hang the system?
Thanks
Vivek
Quoted 31 lines · Vivek Goyal
Quoted 25 lines · Vivek Goyal
Quoted 12 lines · Pavel Emelianov
As the second lock on booth CPUs is taken before checking that this irq is being handled in another processor this may cause a deadlock. This issue is only theoretical.
I propose the attached patch to fix booth problems: when trying to handle misrouted IRQ active
desc->lockmay be unlocked.Please comment.
kernel/irq/spurious.c void note_interrupt(unsigned int irq, st · line 154
Quoted 9 lines
kernel/irq/spurious.c renamed from kernel/irq/spurious.c.irqlockup +5
-1@@ -147,7 +147,11 @@ void note_interrupt(unsigned int irq, stif (unlikely (irqfixup)) {/* Don't punish working computers */ if ((irqfixup ==2 && irq ==0 ) || action_ret ==IRQ_NONE ) { -int ok =misrouted_irq (irq); +int ok; + +spin_unlock (&desc->lock); + ok =misrouted_irq (irq); +spin_lock (&desc->lock);Hi Pavel,
Quoted 4 lines
Till -rc5, I was able to boot a kernel with irqpoll option and in -rc6 I can't. It simply hangs. I suspect it is realted to this change. I have yet to confirm that. But before that one observation.
Quoted 7 lines
Hi Pavel,
If I backout your changes, everything works fine. So it looks like that the problem I am facing is because of your patch but I don't have a logical explanation yet that why the problem is there. Just realasing a lock which is not currently acquired should not hang the system?
Some more data regarding this issue.
For my system it gets locked in following sequence.
handle_level_irq() {
spin_unlock(desc->lock)
......
note_interrupt() {
/* Called without desc->lock held */
spin_unlock(desc->lock)
misrouted_irq()
spin_lock(desc->lock)
}
spin_lock(desc->lock) /* Lockup */
}So basically problems seems to be due to calling conventions of note_interrupt(). In few cases we call it with desc->lock held and in other cases we call it with desc->lock not held.
I guess, note_interrupt() should restore the desc->lock status back to the state of the lock when function was entered so that caller does not lockup.
Thanks
Vivek
Quoted 29 lines · Vivek Goyal
Quoted 23 lines · Vivek Goyal
Quoted 11 lines · Pavel Emelianov
As the second lock on booth CPUs is taken before checking that this irq is being handled in another processor this may cause a deadlock. This issue is only theoretical.
I propose the attached patch to fix booth problems: when trying to handle misrouted IRQ active
desc->lockmay be unlocked.Please comment.
kernel/irq/spurious.c void note_interrupt(unsigned int irq, st · line 154
Quoted 9 lines
kernel/irq/spurious.c renamed from kernel/irq/spurious.c.irqlockup +5
-1@@ -147,7 +147,11 @@ void note_interrupt(unsigned int irq, stif (unlikely (irqfixup)) {/* Don't punish working computers */ if ((irqfixup ==2 && irq ==0 ) || action_ret ==IRQ_NONE ) { -int ok =misrouted_irq (irq); +int ok; + +spin_unlock (&desc->lock); + ok =misrouted_irq (irq); +spin_lock (&desc->lock);Hi Pavel,
Quoted 4 lines
Till -rc5, I was able to boot a kernel with irqpoll option and in -rc6 I can't. It simply hangs. I suspect it is realted to this change. I have yet to confirm that. But before that one observation.
Quoted 6 lines
Hi Pavel,
If I backout your changes, everything works fine. So it looks like that the problem I am facing is because of your patch but I don't have a logical explanation yet that why the problem is there. Just realasing a lock which is not currently acquired should not hang the system?
Without this patch my kernel hanged in another place. I'll look over the interrupt code again. I suspect that just another fix is required.
Quoted 6 lines
Hi Pavel,
If I backout your changes, everything works fine. So it looks like that the problem I am facing is because of your patch but I don't have a logical explanation yet that why the problem is there. Just realasing a lock which is not currently acquired should not hang the system?
Hm... A simple grep over the code showed that note_interrupt is called w/o desc->lock in all places but __do_IRQ(). And this looks like an error at least for the following reason:note_interrupt() calls __report_bad_irq() and __report_bad_irq() does require desc->lock to be held. So I suppose that we have to do spin_lock(&desc->lock) before calling note_interrupt(). I'll prepare a patch in a moment.