Skip to contentSkip to conversation

x86 apic: Ack all pending irqs when crashed/on kexec

PATCH In review Thomas Renninger 3 participants 5 messages Feb 23, 2010 – Mar 8, 2010 x86

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 x86 apic: Ack all pending irqs when crashed/on kexec

From: Kerstin Jonsson <[email protected]>

When the SMP kernel decides to crash_kexec() the local APICs may have pending interrupts in their vector tables.
The setup routine for the local APIC has a deficient mechanism for clearing these interrupts, it only handles interrupts that has already been dispatched to the local core for servicing (the ISR register) safely, it doesn't consider lower prioritized queued interrupts stored in the IRR register.

If you have more than one pending interrupt within the same 32 bit word in the LAPIC vector table registers you may find yourself entering the IO APIC setup with pending interrupts left in the LAPIC. This is a situation for wich the IO APIC setup is not prepared. Depending of what/which interrupt vector/vectors are stuck in the APIC tables your system may show various degrees of malfunctioning.
That was the reason why the check_timer() failed in our system, the timer interrupts was blocked by pending interrupts from the old kernel when routed trough the IO APIC.

Additional comment from Jiri Bohac:
==============
If this should go into stable release,
I'd add some kind of limit on the number of iterations, just to be safe from hard to debug lock-ups:

+if (loops++ > MAX_LOOPS) {
+ printk("LAPIC pending clean-up")
+ break;
+}
while (queued);

with MAX_LOOPS something like 1E9 this would leave plenty of time for the pending IRQs to be cleared and would and still cause at most a second of delay if the loop were to lock-up for whatever reason.
==============

Quoted 1 line

From [email protected]:

Merged Jiri suggestion into the patch.
Also made the max_loops depend on cpu_khz. Not sure how long an apic_read takes, as it is on the CPU it may only be one cycle and we now wait 1 sec in WARN_ON(..) case?

---
 arch/x86/kernel/apic/apic.c |   34 +++++++++++++++++++++++++---------
 1 files changed, 25 insertions(+), 9 deletions(-)
arch/x86/kernel/apic/apic.c +25 -9
index 3987e44..912dd59 100644
@@ -51,6 +51,7 @@
 #include <asm/smp.h>
 #include <asm/mce.h>
 #include <asm/kvm_para.h>
+#include <asm/tsc.h>
 
 unsigned int num_processors;
 
@@ -1151,8 +1152,8 @@ static void __cpuinit lapic_setup_esr(void)
  */
 void __cpuinit setup_local_APIC(void)
 {
-	unsigned int value;
-	int i, j;
+	unsigned int value, queued;
+	int i, j, acked = 0, max_loops = cpu_khz * 1000;
 
 	if (disable_apic) {
 		arch_disable_smp_support();
@@ -1204,13 +1205,28 @@ void __cpuinit setup_local_APIC(void)
 	 * the interrupt. Hence a vector might get locked. It was noticed
 	 * for timer irq (vector 0x31). Issue an extra EOI to clear ISR.
 	 */
-	for (i = APIC_ISR_NR - 1; i >= 0; i--) {
-		value = apic_read(APIC_ISR + i*0x10);
-		for (j = 31; j >= 0; j--) {
-			if (value & (1<<j))
-				ack_APIC_irq();
-		}
-	}
+        do {
+            queued = 0;
+            for (i = APIC_ISR_NR - 1; i >= 0; i--)
+                queued |= apic_read(APIC_IRR + i*0x10);
+
+            for (i = APIC_ISR_NR - 1; i >= 0; i--) {
+                value = apic_read(APIC_ISR + i*0x10);
+                for (j = 31; j >= 0; j--) {
+                    if (value & (1<<j)) {
+                        ack_APIC_irq();
+                        acked++;
+                    }
+                }
+            }
+            if (acked > 256) {
+                printk(KERN_ERR "LAPIC pending interrupts after %d EOI\n",
+		       acked);
+                break;
+            }
+	    max_loops--;
+        } while (queued && max_loops > 0);
+	WARN_ON(!max_loops);
 
 	/*
 	 * Now that we are all set up, enable the APIC
Signature · 1 line
1.6.3
#1 lore raw
Quoted 1 line · On Tuesday 23 February 2010 12:51:25 Thomas Renninger wrote:

From: Kerstin Jonsson <[email protected]>

...

arch/x86/kernel/apic/apic.c static void __cpuinit lapic_setup_esr(void) · line 1156

Quoted 1 line
arch/x86/kernel/apic/apic.c +1 -0
+	int i, j, acked = 0, max_loops = cpu_khz * 1000;

Grmpfl, an unsigned long for max_loops, probably is a better idea... What do you think, could this one get picked up then, anything else to improve?

Thanks,

Thomas

#2 lore raw
In reply to Thomas Renninger
Quoted 37 lines · Thomas Renninger

From: Kerstin Jonsson<[email protected]>

When the SMP kernel decides to crash_kexec() the local APICs may have pending interrupts in their vector tables.
The setup routine for the local APIC has a deficient mechanism for clearing these interrupts, it only handles interrupts that has already been dispatched to the local core for servicing (the ISR register) safely, it doesn't consider lower prioritized queued interrupts stored in the IRR register.

If you have more than one pending interrupt within the same 32 bit word in the LAPIC vector table registers you may find yourself entering the IO APIC setup with pending interrupts left in the LAPIC. This is a situation for wich the IO APIC setup is not prepared. Depending of what/which interrupt vector/vectors are stuck in the APIC tables your system may show various degrees of malfunctioning.
That was the reason why the check_timer() failed in our system, the timer interrupts was blocked by pending interrupts from the old kernel when routed trough the IO APIC.

Additional comment from Jiri Bohac:
==============
If this should go into stable release,
I'd add some kind of limit on the number of iterations, just to be safe from hard to debug lock-ups:

(quoted diff) +4 -0
+if (loops++>  MAX_LOOPS) {
+        printk("LAPIC pending clean-up")
+        break;
+}
  while (queued);

with MAX_LOOPS something like 1E9 this would leave plenty of time for the pending IRQs to be cleared and would and still cause at most a second of delay if the loop were to lock-up for whatever reason.
==============

Quoted 7 lines

From [email protected]:
Merged Jiri suggestion into the patch.
Also made the max_loops depend on cpu_khz. Not sure how long an apic_read takes, as it is on the CPU it may only be one cycle and we now wait 1 sec in WARN_ON(..) case?

An apic_read() can take a couple of microseconds when running virtualized, so this loop may run for hours. On the other hand, virtualized hardware is unlikely to misbehave.

Still I recommend using a clocksource (tsc would do) and not a loop count.

Signature · 1 line
error compiling committee.c: too many arguments to function
#3 lore raw
Quoted 59 lines

________________________________________
From: Avi Kivity [[email protected]]
Sent: Tuesday, February 23, 2010 1:03 PM
To: Thomas Renninger
Cc: [email protected]; Kerstin Jonsson; [email protected]; Yinghai Lu; [email protected]; [email protected]
Subject: Re: [PATCH] x86 apic: Ack all pending irqs when crashed/on kexec

Quoted 37 lines · Thomas Renninger

From: Kerstin Jonsson<[email protected]>

When the SMP kernel decides to crash_kexec() the local APICs may have pending interrupts in their vector tables.
The setup routine for the local APIC has a deficient mechanism for clearing these interrupts, it only handles interrupts that has already been dispatched to the local core for servicing (the ISR register) safely, it doesn't consider lower prioritized queued interrupts stored in the IRR register.

If you have more than one pending interrupt within the same 32 bit word in the LAPIC vector table registers you may find yourself entering the IO APIC setup with pending interrupts left in the LAPIC. This is a situation for wich the IO APIC setup is not prepared. Depending of what/which interrupt vector/vectors are stuck in the APIC tables your system may show various degrees of malfunctioning.
That was the reason why the check_timer() failed in our system, the timer interrupts was blocked by pending interrupts from the old kernel when routed trough the IO APIC.

Additional comment from Jiri Bohac:
==============
If this should go into stable release,
I'd add some kind of limit on the number of iterations, just to be safe from hard to debug lock-ups:

(quoted diff) +4 -0
+if (loops++>  MAX_LOOPS) {
+        printk("LAPIC pending clean-up")
+        break;
+}
  while (queued);

with MAX_LOOPS something like 1E9 this would leave plenty of time for the pending IRQs to be cleared and would and still cause at most a second of delay if the loop were to lock-up for whatever reason.
==============

Quoted 8 lines

From [email protected]:
Merged Jiri suggestion into the patch.
Also made the max_loops depend on cpu_khz. Not sure how long an apic_read takes, as it is on the CPU it may only be one cycle and we now wait 1 sec in WARN_ON(..) case?

An apic_read() can take a couple of microseconds when running virtualized, so this loop may run for hours. On the other hand, virtualized hardware is unlikely to misbehave.

Quoted 7 lines

Still I recommend using a clocksource (tsc would do) and not a loop count.

--
error compiling committee.c: too many arguments to function

Is it possible/thinkable to distinguish between real and virtual targets? I.e. to somehow detect that the target is a virtual machine and adapt accordingly. There may be other cases as well, in which one would benefit from taking target type into consideration when e.g. estimating the reasonable number of cycles for a specific operation.

#4 lore raw
Quoted 60 lines · Kerstin Jonsson
Quoted 52 lines
Quoted 39 lines

From: Kerstin Jonsson<[email protected]>

When the SMP kernel decides to crash_kexec() the local APICs may have pending interrupts in their vector tables.
The setup routine for the local APIC has a deficient mechanism for clearing these interrupts, it only handles interrupts that has already been dispatched to the local core for servicing (the ISR register) safely, it doesn't consider lower prioritized queued interrupts stored in the IRR register.

If you have more than one pending interrupt within the same 32 bit word in the LAPIC vector table registers you may find yourself entering the IO APIC setup with pending interrupts left in the LAPIC. This is a situation for wich the IO APIC setup is not prepared. Depending of what/which interrupt vector/vectors are stuck in the APIC tables your system may show various degrees of malfunctioning.
That was the reason why the check_timer() failed in our system, the timer interrupts was blocked by pending interrupts from the old kernel when routed trough the IO APIC.

Additional comment from Jiri Bohac:
==============
If this should go into stable release,
I'd add some kind of limit on the number of iterations, just to be safe from hard to debug lock-ups:

(quoted diff) +4 -0
+if (loops++>   MAX_LOOPS) {
+        printk("LAPIC pending clean-up")
+        break;
+}
   while (queued);

with MAX_LOOPS something like 1E9 this would leave plenty of time for the pending IRQs to be cleared and would and still cause at most a second of delay if the loop were to lock-up for whatever reason.
==============

Quoted 9 lines

From [email protected]:
Merged Jiri suggestion into the patch.
Also made the max_loops depend on cpu_khz. Not sure how long an apic_read takes, as it is on the CPU it may only be one cycle and we now wait 1 sec in WARN_ON(..) case?

An apic_read() can take a couple of microseconds when running virtualized, so this loop may run for hours. On the other hand, virtualized hardware is unlikely to misbehave.

Quoted 8 lines

Still I recommend using a clocksource (tsc would do) and not a loop count.

--
error compiling committee.c: too many arguments to function

Quoted 5 lines

Is it possible/thinkable to distinguish between real and virtual targets? I.e. to somehow detect that the target is a virtual machine and adapt accordingly. There may be other cases as well, in which one would benefit from taking target type into consideration when e.g. estimating the reasonable number of cycles for a specific operation

It's possible (cpuid hypervisor bit), but I don't think it's a good idea. Splitting up code paths doubles the chance of bugs. Much better to find something that works both ways.

Signature · 1 line
error compiling committee.c: too many arguments to function