Skip to contentSkip to conversation

irq0 stops working

DISCUSSION Vasily Averin 3 participants 6 messages Oct 9, 2007 – Nov 6, 2007

lore
Share & tools
lore
#0 lore raw

On one of our servers timer interrupts (i.e irq0) are stops working. As result any kernel timers do not triggers and tasks waiting some signals from timers hangs forever.

Most noticeable effect of this situation is that any write operations to disk are stalled, and nobody can log in on the node.

At the same time node all existing shells works away. I'm able to read interrupts statistic from /proc/interrupts file and it shows that all other interrupts are changed when these devices are accessed: disk on sata controller, network, cdrom on ide controller, keyboard, serial console, LOC interrupts.

Also I've found that disable of irqbalance service on the node helps to workaround this issue, however of course it fixes nothing.

All details about hardware/logs could be found in
http://bugzilla.kernel.org/show_bug.cgi?id=8650

I'm able to reproduce this situation, however now I have no ideas how to continue the investigation of this problem.

Could please anybody advise me any new ways for investigation of this issue?

Thank you,
Vasily Averin

OpenVZ Linux Kernel Team

#1 lore raw
Quoted 4 lines · Vasily Averin

On one of our servers timer interrupts (i.e irq0) are stops working. As result any kernel timers do not triggers and tasks waiting some signals from timers hangs forever.

What kernel.. and tried CONFIG_NO_HZ=n?

#4 lore raw
In reply to Vasily Averin
Quoted 3 lines · Vasily Averin

On one of our servers timer interrupts (i.e irq0) are stops working. As result any kernel timers do not triggers and tasks waiting some signals from timers hangs forever.

Which kernel version ?

Quoted 10 lines

Most noticeable effect of this situation is that any write operations to disk are stalled, and nobody can log in on the node.

At the same time node all existing shells works away. I'm able to read interrupts statistic from /proc/interrupts file and it shows that all other interrupts are changed when these devices are accessed: disk on sata controller, network, cdrom on ide controller, keyboard, serial console, LOC interrupts.

Also I've found that disable of irqbalance service on the node helps to workaround this issue, however of course it fixes nothing.

Well, it's at least a hint. Can you try the patch below please ?

tglx

arch/x86_64/kernel/time.c +2 -1
index 6d48a4e..248987a 100644
@@ -360,7 +360,7 @@ void stop_timer_interrupt(void)
 
 static struct irqaction irq0 = {
 	.handler	= timer_interrupt,
-	.flags		= IRQF_DISABLED | IRQF_IRQPOLL,
+	.flags		= IRQF_DISABLED | IRQF_IRQPOLL | IRQF_NOBALANCING,
 	.mask		= CPU_MASK_NONE,
 	.name		= "timer"
 };
@@ -403,6 +403,7 @@ void __init time_init(void)
 		cpu_khz / 1000, cpu_khz % 1000);
 	init_tsc_clocksource();
 
+	irq0.mask = cpumask_of_cpu(0);
 	setup_irq(0, &irq0);
 }
 
#2 lore raw
In reply to Jan Engelhardt
Quoted 6 lines · Jan Engelhardt
Quoted 3 lines · Vasily Averin

On one of our servers timer interrupts (i.e irq0) are stops working. As result any kernel timers do not triggers and tasks waiting some signals from timers hangs forever.

What kernel.. and tried CONFIG_NO_HZ=n?

Originally I've noticed this issue on RHEL5 kernels, but then I've reproduced it on latest mainstream kernels, in my last attempt it was 2.6.23-rc7.

Thank you for for your tips about CONFIG_NO_HZ=n, will try to to it.

thank you,
Vasily Averin

OpenVZ Linux Kernel Team

#3 lore raw
Quoted 6 lines · Vasily Averin
Quoted 5 lines · Jan Engelhardt
Quoted 3 lines · Vasily Averin

On one of our servers timer interrupts (i.e irq0) are stops working. As result any kernel timers do not triggers and tasks waiting some signals from timers hangs forever.

Quoted 6 lines
Quoted 1 line

What kernel.. and tried CONFIG_NO_HZ=n?

Originally I've noticed this issue on RHEL5 kernels, but then I've reproduced it on latest mainstream kernels, in my last attempt it was 2.6.23-rc7.

Thank you for for your tips about CONFIG_NO_HZ=n, will try to to it.

You run a 64 bit kernel, where this option is not available yet.

tglx

#5 lore raw
In reply to Thomas Gleixner
Quoted 9 lines · Thomas Gleixner
Quoted 6 lines · Vasily Averin

On one of our servers timer interrupts (i.e irq0) are stops working. As result any kernel timers do not triggers and tasks waiting some signals from timers hangs forever.

Also I've found that disable of irqbalance service on the node helps to workaround this issue, however of course it fixes nothing.

Well, it's at least a hint. Can you try the patch below please ?

Patch helps, after 1-month-long testing I could not reproduce this issue again, what are we going to do now?

Unfortunately my test node is still unstable, now it reproduces sata timeout issue, details are here:
http://bugzilla.kernel.org/show_bug.cgi?id=8650

Thank you,
Vasily Averin