Skip to contentSkip to conversation

do_IRQ: 0.126 No irq handler for vector (irq -1)

DISCUSSION Stefan Priebe - Profihost AG 2 participants 8 messages Jul 20, 2015 – Jul 26, 2015

lore
Share & tools
lore
#0 lore raw

Hello list,

i've 36 servers all running vanilla 3.18.18 kernel which have a very high disk and network load.

Since a few days i encounter regular the following error messages and pretty often completely hanging disk i/o:

[535040.439859] do_IRQ: 0.126 No irq handler for vector (irq -1)
[548400.353679] do_IRQ: 2.109 No irq handler for vector (irq -1)
[551624.894507] do_IRQ: 4.84 No irq handler for vector (irq -1)
[557524.288691] do_IRQ: 1.158 No irq handler for vector (irq -1)
[559786.928441] do_IRQ: 3.172 No irq handler for vector (irq -1)
[572906.281394] do_IRQ: 3.72 No irq handler for vector (irq -1)
[576611.808128] do_IRQ: 3.118 No irq handler for vector (irq -1)
[577242.682643] do_IRQ: 2.45 No irq handler for vector (irq -1)
[578524.584545] do_IRQ: 5.190 No irq handler for vector (irq -1)
[602109.548268] do_IRQ: 3.101 No irq handler for vector (irq -1)

All systems are Single E5 Xeons and I'm running irqbalance on them.

Chipset:
Intel C602J chipset

Is there anything i can do to fix this? Is there may be a kernel patch available?

Thanks!

Greets,
Stefan

#1 lore raw
Quoted 8 lines · Stefan Priebe - Profihost AG

Hello list,

i've 36 servers all running vanilla 3.18.18 kernel which have a very high disk and network load.

Since a few days i encounter regular the following error messages and pretty often completely hanging disk i/o:

[535040.439859] do_IRQ: 0.126 No irq handler for vector (irq -1)

Did this happen right after you updated to 3.18.18?

Which kernel version were you using before that?

Have you observed such error messages before the update?

Quoted 1 line

All systems are Single E5 Xeons and I'm running irqbalance on them.

Does it stop if you disable irqbalance ?

Thanks,

tglx

#2 lore raw

Am 20.07.2015 um 12:53 schrieb Thomas Gleixner:

Quoted 6 lines · Stefan Priebe - Profihost AG
Quoted 5 lines

Hello list,

i've 36 servers all running vanilla 3.18.18 kernel which have a very high disk and network load.

Quoted 7 lines
Quoted 3 lines

Since a few days i encounter regular the following error messages and pretty often completely hanging disk i/o:

[535040.439859] do_IRQ: 0.126 No irq handler for vector (irq -1)

Did this happen right after you updated to 3.18.18? Which kernel version were you using before that? Have you observed such error messages before the update?

No it was always working before or at least i've never noticed a device hang and such messages. It was vanilla 3.10.78.

Quoted 3 lines
Quoted 1 line

All systems are Single E5 Xeons and I'm running irqbalance on them.

Does it stop if you disable irqbalance ?

Will try that today.

Stefan

#3 lore raw
In reply to Thomas Gleixner

Am 20.07.2015 um 12:53 schrieb Thomas Gleixner:

Quoted 11 lines · Stefan Priebe - Profihost AG
Quoted 8 lines

Hello list,

i've 36 servers all running vanilla 3.18.18 kernel which have a very high disk and network load.

Since a few days i encounter regular the following error messages and pretty often completely hanging disk i/o:

[535040.439859] do_IRQ: 0.126 No irq handler for vector (irq -1)

Did this happen right after you updated to 3.18.18?

Seems so.

Quoted 1 line

Which kernel version were you using before that?

3.10.78

Quoted 1 line

Have you observed such error messages before the update?

No.

Quoted 3 lines
Quoted 1 line

All systems are Single E5 Xeons and I'm running irqbalance on them.

Does it stop if you disable irqbalance ?

No. The machines still crash.

Stefan

Quoted 5 lines

Thanks,

tglx

#4 lore raw
Quoted 11 lines · Stefan Priebe

Am 20.07.2015 um 12:53 schrieb Thomas Gleixner:

Quoted 10 lines · Stefan Priebe - Profihost AG
Quoted 9 lines

Hello list,

i've 36 servers all running vanilla 3.18.18 kernel which have a very high disk and network load.

Since a few days i encounter regular the following error messages and pretty often completely hanging disk i/o:

[535040.439859] do_IRQ: 0.126 No irq handler for vector (irq -1)
Quoted 5 lines
Quoted 3 lines
Quoted 1 line

All systems are Single E5 Xeons and I'm running irqbalance on them.

Does it stop if you disable irqbalance ?

No. The machines still crash.

crash as in running into a BUG? Or is it just that disk I/O is stalled?

Can you please provide the full dmesg output of such a machine?

I'll cook up a debug patch for that against 3.18.18.

Thanks,

tglx

#5 lore raw

Am 21.07.2015 um 23:15 schrieb Thomas Gleixner:

Quoted 12 lines · Stefan Priebe
Quoted 11 lines

Am 20.07.2015 um 12:53 schrieb Thomas Gleixner:

Quoted 10 lines · Stefan Priebe - Profihost AG
Quoted 9 lines

Hello list,

i've 36 servers all running vanilla 3.18.18 kernel which have a very high disk and network load.

Since a few days i encounter regular the following error messages and pretty often completely hanging disk i/o:

[535040.439859] do_IRQ: 0.126 No irq handler for vector (irq -1)
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
Quoted 1 line

All systems are Single E5 Xeons and I'm running irqbalance on them.

Does it stop if you disable irqbalance ?

No. The machines still crash.

crash as in running into a BUG? Or is it just that disk I/O is stalled?

Sorry i meant I/O is stalled. It crashes to me as i can't login anymore due to hanging I/O.

Quoted 1 line

Can you please provide the full dmesg output of such a machine?

Yes (this time from a machine using 3.18.14) =>
http://pastebin.com/raw.php?i=S6kAk0iS

Quoted 1 line

I'll cook up a debug patch for that against 3.18.18.

That would be great!

Stefan

Quoted 4 lines

Thanks,

tglx

#6 lore raw

Am 22.07.2015 um 09:23 schrieb Stefan Priebe - Profihost AG:

Quoted 22 lines

Am 21.07.2015 um 23:15 schrieb Thomas Gleixner:

Quoted 12 lines · Stefan Priebe
Quoted 11 lines

Am 20.07.2015 um 12:53 schrieb Thomas Gleixner:

Quoted 10 lines · Stefan Priebe - Profihost AG
Quoted 9 lines

Hello list,

i've 36 servers all running vanilla 3.18.18 kernel which have a very high disk and network load.

Since a few days i encounter regular the following error messages and pretty often completely hanging disk i/o:

[535040.439859] do_IRQ: 0.126 No irq handler for vector (irq -1)
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
Quoted 1 line

All systems are Single E5 Xeons and I'm running irqbalance on them.

Does it stop if you disable irqbalance ?

No. The machines still crash.

crash as in running into a BUG? Or is it just that disk I/O is stalled?

Quoted 9 lines

Sorry i meant I/O is stalled. It crashes to me as i can't login anymore due to hanging I/O.

Quoted 1 line

Can you please provide the full dmesg output of such a machine?

Yes (this time from a machine using 3.18.14) =>
http://pastebin.com/raw.php?i=S6kAk0iS

Quoted 1 line

I'll cook up a debug patch for that against 3.18.18.

Do you have any special upstream commits in mind?

Stefan

Quoted 9 lines

That would be great!

Stefan

Quoted 4 lines

Thanks,

tglx

#7 lore raw
Quoted 1 line · Stefan Priebe

Do you have any special upstream commits in mind?

Not yet. Could you run a test with 3.16 and 3.17 please? That would narrow down the problem space quite a bit.

Thanks,

tglx