2.4.3: still experiencing APIC-related hangs

DISCUSSION Frank de Lange 4 participants 5 messages Mar 30, 2001 – Apr 9, 2001 IRQ

Share & tools

Frank de Lange

author

2001-03-30 12:49 UTC #0 lore raw

Hi'all,

Subject says it all: 2.4.3 (unpatchaed) is still causing the dreaded APIC-related hangs on SMP BX systems (Abit BP-6, maybe Gigabyte). I still need to apply one of Maciej's patches to get rid of these hangs. The source comments in arc/i386/kernel/apic.c ("If focus CPU is disabled then the hang goes away") are incorrect, as the hang does not go away by simply disabling focus CPU. The only way for me to get rid of the hangs is to apply patch-2.4.1-io_apic-46 (which does the LEVEL->EDGE->LEVEL triggered trick to 'free' the IO_APIC). I've been running with this patch for quite some time now, and have not experienced any problems with it. Maybe it it time to include it in the main kernel, perhaps as a configurable option ("BROKEN_IO_APIC")?

Maciej, did you submit the patch to Linus? It really seems to solve the (occurence of the) problems with these boards...

Cheers//Frank

Signature · 9 lines

  WWWWW      _______________________
 ## o o\    /     Frank de Lange     \
 }#   \|   /                          \
  ##---# _/     <Hacker for Hire>      \
   ####   \      +31-320-252965        /
           \    [email protected]    /
            -------------------------
 [ "Omnis enim res, quae dando non deficit, dum habetur
    et non datur, nondum habetur, quomodo habenda est."  ]

Maciej W. Rozycki

2001-03-30 16:12 UTC #1 lore raw

Quoted 2 lines · Frank de Lange

Maciej, did you submit the patch to Linus? It really seems to solve the (occurence of the) problems with these boards...

I suppose Alan is going to pass the patch to Linus eventually. I think
there is actually a number of people interested in the fix, so I may pass it to Linus independently, but I'm really time-constrained these days, so it might not happen before 2.4.3 (I don't feel safe about submitting a patch without actually run-time testing it against whatever test kernel is current at the moment).

Signature · 3 lines

+  Maciej W. Rozycki, Technical University of Gdansk, Poland   +
+--------------------------------------------------------------+
+        e-mail: [email protected], PGP key available        +

[email protected]

2001-03-30 16:34 UTC #2 lore raw

In reply to Frank de Lange

Quoted 13 lines · Frank de Lange

Hi'all,
Subject says it all: 2.4.3 (unpatchaed) is still causing the dreaded APIC-related hangs on SMP BX systems (Abit BP-6, maybe Gigabyte). I still need to apply one of Maciej's patches to get rid of these hangs. The source comments in arc/i386/kernel/apic.c ("If focus CPU is disabled then the hang goes away") are incorrect, as the hang does not go away by simply disabling focus CPU. The only way for me to get rid of the hangs is to apply patch-2.4.1-io_apic-46 (which does the LEVEL->EDGE->LEVEL triggered trick to 'free' the IO_APIC). I've been running with this patch for quite some time now, and have not experienced any problems with it. Maybe it it time to include it in the main kernel, perhaps as a configurable option ("BROKEN_IO_APIC")?

Quoted 2 lines

Maciej, did you submit the patch to Linus? It really seems to solve the (occurence of the) problems with these boards...

Where is this patch found? I am not seeing it so far on kernel.org.

-- Ferret

Frank de Lange

author

2001-03-30 16:37 UTC #3 lore raw

In reply to Frank de Lange

Quoted 6 lines · On Fri, Mar 30, 2001 at 08:32:39AM -0800, [email protected] wrote:

Quoted 3 lines · Frank de Lange
Maciej, did you submit the patch to Linus? It really seems to solve the (occurence of the) problems with these boards...
Where is this patch found? I am not seeing it so far on kernel.org.

It is allmost ancient history, from days long gone when men were men, women were women and Linux had only reached 2.4.1...

I can send you a copy, if you need it...

Cheers//Frank

Signature · 9 lines

  WWWWW      _______________________
 ## o o\    /     Frank de Lange     \
 }#   \|   /                          \
  ##---# _/     <Hacker for Hire>      \
   ####   \      +31-320-252965        /
           \    [email protected]    /
            -------------------------
 [ "Omnis enim res, quae dando non deficit, dum habetur
    et non datur, nondum habetur, quomodo habenda est."  ]

Kurt Garloff

2001-04-09 21:24 UTC #4 lore raw

In reply to Frank de Lange

Quoted 11 lines · On Fri, Mar 30, 2001 at 08:32:39AM -0800, [email protected] wrote:

Quoted 10 lines · Frank de Lange
Subject says it all: 2.4.3 (unpatchaed) is still causing the dreaded APIC-related hangs on SMP BX systems (Abit BP-6, maybe Gigabyte). I still need to apply one of Maciej's patches to get rid of these hangs. The source comments in arc/i386/kernel/apic.c ("If focus CPU is disabled then the hang goes away") are incorrect, as the hang does not go away by simply disabling focus CPU. The only way for me to get rid of the hangs is to apply patch-2.4.1-io_apic-46 (which does the LEVEL->EDGE->LEVEL triggered trick to 'free' the IO_APIC). I've been running with this patch for quite some time now, and have not experienced any problems with it. Maybe it it time to include it in the main kernel, perhaps as a configurable option ("BROKEN_IO_APIC")?

Same for me.

Quoted 4 lines

Quoted 2 lines
Maciej, did you submit the patch to Linus? It really seems to solve the (occurence of the) problems with these boards...
Where is this patch found? I am not seeing it so far on kernel.org.

Attached, as I assume more people are interested in it ...

Regards,

Signature · 3 lines

Kurt Garloff  <[email protected]>                          Eindhoven, NL
GPG key: See mail header, key servers         Linux kernel development
SuSE GmbH, Nuernberg, FRG                               SCSI, Security

patch-2.4.1-io_apic-46

arch/i386/kernel/apic.c +9 -2

diff -up --recursive --new-file linux-2.4.1.macro/arch/i386/kernel/apic.c linux-2.4.1/arch/i386/kernel/apic.c
@@ -23,6 +23,7 @@
 #include <linux/mc146818rtc.h>
 #include <linux/kernel_stat.h>
 
+#include <asm/atomic.h>
 #include <asm/smp.h>
 #include <asm/mtrr.h>
 #include <asm/mpspec.h>

@@ -270,7 +271,13 @@ void __init setup_local_APIC (void)
 	 *   PCI Ne2000 networking cards and PII/PIII processors, dual
 	 *   BX chipset. ]
 	 */
-#if 0
+	/*
+	 * Actually disabling the focus CPU check just makes the hang less
+	 * frequent as it makes the interrupt distributon model be more
+	 * like LRU than MRU (the short-term load is more even across CPUs).
+	 * See also the comment in end_level_ioapic_irq().  --macro
+	 */
+#if 1
 	/* Enable focus processor (bit==0) */
 	value &= ~(1<<9);
 #else

@@ -764,7 +771,7 @@ asmlinkage void smp_error_interrupt(void
 	apic_write(APIC_ESR, 0);
 	v1 = apic_read(APIC_ESR);
 	ack_APIC_irq();
-	irq_err_count++;
+	atomic_inc(&irq_err_count);
 
 	/* Here is what the APIC error bits mean:
 	   0: Send CS error

arch/i386/kernel/i8259.c +2 -1

diff -up --recursive --new-file linux-2.4.1.macro/arch/i386/kernel/i8259.c linux-2.4.1/arch/i386/kernel/i8259.c
@@ -12,6 +12,7 @@
 #include <linux/init.h>
 #include <linux/kernel_stat.h>
 
+#include <asm/atomic.h>
 #include <asm/system.h>
 #include <asm/io.h>
 #include <asm/irq.h>

@@ -321,7 +322,7 @@ spurious_8259A_irq:
 			printk("spurious 8259A interrupt: IRQ%d.\n", irq);
 			spurious_irq_mask |= irqmask;
 		}
-		irq_err_count++;
+		atomic_inc(&irq_err_count);
 		/*
 		 * Theoretically we do not have to handle this IRQ,
 		 * but in Linux this does not cause problems and is

arch/i386/kernel/io_apic.c +63 -4

95 lines not on this page · raw message

arch/i386/kernel/irq.c +13 -2

diff -up --recursive --new-file linux-2.4.1.macro/arch/i386/kernel/irq.c linux-2.4.1/arch/i386/kernel/irq.c
@@ -33,6 +33,7 @@
 #include <linux/irq.h>
 #include <linux/proc_fs.h>
 
+#include <asm/atomic.h>
 #include <asm/io.h>
 #include <asm/smp.h>
 #include <asm/system.h>

@@ -119,7 +120,12 @@ struct hw_interrupt_type no_irq_type = {
 	end_none
 };
 
-volatile unsigned long irq_err_count;
+atomic_t irq_err_count;
+#ifdef CONFIG_X86_IO_APIC
+#ifdef APIC_MISMATCH_DEBUG
+atomic_t irq_mis_count;
+#endif
+#endif
 
 /*
  * Generic, controller-independent functions:

@@ -167,7 +173,12 @@ int get_irq_list(char *buf)
 			apic_timer_irqs[cpu_logical_map(j)]);
 	p += sprintf(p, "\n");
 #endif
-	p += sprintf(p, "ERR: %10lu\n", irq_err_count);
+	p += sprintf(p, "ERR: %10u\n", atomic_read(&irq_err_count));
+#ifdef CONFIG_X86_IO_APIC
+#ifdef APIC_MISMATCH_DEBUG
+	p += sprintf(p, "MIS: %10u\n", atomic_read(&irq_mis_count));
+#endif
+#endif
 	return p - buf;
 }

include/asm-i386/hw_irq.h +4 -1

diff -up --recursive --new-file linux-2.4.1.macro/include/asm-i386/hw_irq.h linux-2.4.1/include/asm-i386/hw_irq.h
@@ -13,6 +13,7 @@
  */
 
 #include <linux/config.h>
+#include <asm/atomic.h>
 #include <asm/irq.h>
 
 /*

@@ -83,7 +84,9 @@ extern int IO_APIC_get_PCI_irq_vector(in
 extern void send_IPI(int dest, int vector);
 
 extern unsigned long io_apic_irqs;
-extern volatile unsigned long irq_err_count;
+
+extern atomic_t irq_err_count;
+extern atomic_t irq_mis_count;
 
 extern char _stext, _etext;

include/asm-i386/io_apic.h +2 -0

diff -up --recursive --new-file linux-2.4.1.macro/include/asm-i386/io_apic.h linux-2.4.1/include/asm-i386/io_apic.h
@@ -12,6 +12,8 @@
 
 #ifdef CONFIG_X86_IO_APIC
 
+#define APIC_MISMATCH_DEBUG
+
 #define IO_APIC_BASE(idx) \
 		((volatile int *)__fix_to_virt(FIX_IO_APIC_BASE_0 + idx))

include/linux/irq.h +0 -3

diff -up --recursive --new-file linux-2.4.1.macro/include/linux/irq.h linux-2.4.1/include/linux/irq.h
@@ -62,7 +62,4 @@ extern int setup_irq(unsigned int , stru
 extern hw_irq_controller no_irq_type;  /* needed in every arch ? */
 extern void no_action(int cpl, void *dev_id, struct pt_regs *regs);
 
-extern volatile unsigned long irq_err_count;
-
 #endif /* __asm_h */
-