do_IRQ: 0.126 No irq handler for vector (irq -1)
Hello list,
i've 36 servers all running vanilla 3.18.18 kernel which have a very high disk and network load.
Since a few days i encounter regular the following error messages and pretty often completely hanging disk i/o:
[535040.439859] do_IRQ: 0.126 No irq handler for vector (irq -1) [548400.353679] do_IRQ: 2.109 No irq handler for vector (irq -1) [551624.894507] do_IRQ: 4.84 No irq handler for vector (irq -1) [557524.288691] do_IRQ: 1.158 No irq handler for vector (irq -1) [559786.928441] do_IRQ: 3.172 No irq handler for vector (irq -1) [572906.281394] do_IRQ: 3.72 No irq handler for vector (irq -1) [576611.808128] do_IRQ: 3.118 No irq handler for vector (irq -1) [577242.682643] do_IRQ: 2.45 No irq handler for vector (irq -1) [578524.584545] do_IRQ: 5.190 No irq handler for vector (irq -1) [602109.548268] do_IRQ: 3.101 No irq handler for vector (irq -1)
All systems are Single E5 Xeons and I'm running irqbalance on them.
Chipset:
Intel C602J chipset
Is there anything i can do to fix this? Is there may be a kernel patch available?
Thanks!
Greets,
Stefan
Quoted 8 lines · Stefan Priebe - Profihost AG
Hello list,
i've 36 servers all running vanilla 3.18.18 kernel which have a very high disk and network load.
Since a few days i encounter regular the following error messages and pretty often completely hanging disk i/o:
[535040.439859] do_IRQ: 0.126 No irq handler for vector (irq -1)
Did this happen right after you updated to 3.18.18?
Which kernel version were you using before that?
Have you observed such error messages before the update?
Quoted 1 line
All systems are Single E5 Xeons and I'm running irqbalance on them.
Does it stop if you disable irqbalance ?
Thanks,
tglx
Am 20.07.2015 um 12:53 schrieb Thomas Gleixner:
Quoted 6 lines · Stefan Priebe - Profihost AG
Quoted 5 lines
Hello list,
i've 36 servers all running vanilla 3.18.18 kernel which have a very high disk and network load.
Quoted 7 lines
Quoted 3 lines
Since a few days i encounter regular the following error messages and pretty often completely hanging disk i/o:
[535040.439859] do_IRQ: 0.126 No irq handler for vector (irq -1)Did this happen right after you updated to 3.18.18? Which kernel version were you using before that? Have you observed such error messages before the update?
No it was always working before or at least i've never noticed a device hang and such messages. It was vanilla 3.10.78.
Quoted 3 lines
Quoted 1 line
All systems are Single E5 Xeons and I'm running irqbalance on them.
Does it stop if you disable irqbalance ?
Will try that today.
Stefan
Am 20.07.2015 um 12:53 schrieb Thomas Gleixner:
Quoted 11 lines · Stefan Priebe - Profihost AG
Quoted 8 lines
Hello list,
i've 36 servers all running vanilla 3.18.18 kernel which have a very high disk and network load.
Since a few days i encounter regular the following error messages and pretty often completely hanging disk i/o:
[535040.439859] do_IRQ: 0.126 No irq handler for vector (irq -1)Did this happen right after you updated to 3.18.18?
Seems so.
Quoted 1 line
Which kernel version were you using before that?
3.10.78
Quoted 1 line
Have you observed such error messages before the update?
No.
Quoted 3 lines
Quoted 1 line
All systems are Single E5 Xeons and I'm running irqbalance on them.
Does it stop if you disable irqbalance ?
No. The machines still crash.
Stefan
Quoted 5 lines
Thanks,
tglx
Quoted 11 lines · Stefan Priebe
Am 20.07.2015 um 12:53 schrieb Thomas Gleixner:
Quoted 10 lines · Stefan Priebe - Profihost AG
Quoted 9 lines
Hello list,
i've 36 servers all running vanilla 3.18.18 kernel which have a very high disk and network load.
Since a few days i encounter regular the following error messages and pretty often completely hanging disk i/o:
[535040.439859] do_IRQ: 0.126 No irq handler for vector (irq -1)
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
All systems are Single E5 Xeons and I'm running irqbalance on them.
Does it stop if you disable irqbalance ?
No. The machines still crash.
crash as in running into a BUG? Or is it just that disk I/O is stalled?
Can you please provide the full dmesg output of such a machine?
I'll cook up a debug patch for that against 3.18.18.
Thanks,
tglx
Am 21.07.2015 um 23:15 schrieb Thomas Gleixner:
Quoted 12 lines · Stefan Priebe
Quoted 11 lines
Am 20.07.2015 um 12:53 schrieb Thomas Gleixner:
Quoted 10 lines · Stefan Priebe - Profihost AG
Quoted 9 lines
Hello list,
i've 36 servers all running vanilla 3.18.18 kernel which have a very high disk and network load.
Since a few days i encounter regular the following error messages and pretty often completely hanging disk i/o:
[535040.439859] do_IRQ: 0.126 No irq handler for vector (irq -1)
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
All systems are Single E5 Xeons and I'm running irqbalance on them.
Does it stop if you disable irqbalance ?
No. The machines still crash.
crash as in running into a BUG? Or is it just that disk I/O is stalled?
Sorry i meant I/O is stalled. It crashes to me as i can't login anymore due to hanging I/O.
Quoted 1 line
Can you please provide the full dmesg output of such a machine?
Yes (this time from a machine using 3.18.14) =>
http://pastebin.com/raw.php?i=S6kAk0iS
Quoted 1 line
I'll cook up a debug patch for that against 3.18.18.
That would be great!
Stefan
Quoted 4 lines
Thanks,
tglx
Am 22.07.2015 um 09:23 schrieb Stefan Priebe - Profihost AG:
Quoted 22 lines
Am 21.07.2015 um 23:15 schrieb Thomas Gleixner:
Quoted 12 lines · Stefan Priebe
Quoted 11 lines
Am 20.07.2015 um 12:53 schrieb Thomas Gleixner:
Quoted 10 lines · Stefan Priebe - Profihost AG
Quoted 9 lines
Hello list,
i've 36 servers all running vanilla 3.18.18 kernel which have a very high disk and network load.
Since a few days i encounter regular the following error messages and pretty often completely hanging disk i/o:
[535040.439859] do_IRQ: 0.126 No irq handler for vector (irq -1)Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
All systems are Single E5 Xeons and I'm running irqbalance on them.
Does it stop if you disable irqbalance ?
No. The machines still crash.
crash as in running into a BUG? Or is it just that disk I/O is stalled?
Quoted 9 lines
Sorry i meant I/O is stalled. It crashes to me as i can't login anymore due to hanging I/O.
Quoted 1 line
Can you please provide the full dmesg output of such a machine?
Yes (this time from a machine using 3.18.14) =>
http://pastebin.com/raw.php?i=S6kAk0iSQuoted 1 line
I'll cook up a debug patch for that against 3.18.18.
Do you have any special upstream commits in mind?
Stefan
Quoted 9 lines
That would be great!
Stefan
Quoted 4 lines
Thanks,
tglx
Quoted 1 line · Stefan Priebe
Do you have any special upstream commits in mind?
Not yet. Could you run a test with 3.16 and 3.17 please? That would narrow down the problem space quite a bit.
Thanks,
tglx