Skip to contentSkip to conversation

irq balancing: performance disaster

DISCUSSION Jos Hulzink 5 participants 12 messages May 10, 2003 – May 11, 2003

lore
Share & tools
lore

Jos Hulzink

author
#0 lore raw

Hi,

While tackling bug 699, it became clear to me that irq balancing is the cause of the performance problems I, and all people using the SMP kernel Mandrake 9.1 ships, are dealing with. I got the problems with 2.5.69 too. After disabling irq balancing, the system is remarkably faster, and much more responsive.

For those interested in the issue, please look at bug 699.

Jos

#1 lore raw
Quoted 5 lines · Jos Hulzink

While tackling bug 699, it became clear to me that irq balancing is the cause of the performance problems I, and all people using the SMP kernel Mandrake 9.1 ships, are dealing with. I got the problems with 2.5.69 too. After disabling irq balancing, the system is remarkably faster, and much more responsive.

Alan fixed this in his tree a little while back, Mandrake didn't manage to squeeze that fix in. The 2.5 and 2.4 situation is different however, for 2.5 it's intentional, for 2.4 something broke.

Zwane
--
function.linuxpower.ca

#2 lore raw
In reply to Jos Hulzink
Quoted 7 lines

While tackling bug 699, it became clear to me that irq balancing is the cause of the performance problems I, and all people using the SMP kernel Mandrake 9.1 ships, are dealing with. I got the problems with 2.5.69 too. After disabling irq balancing, the system is remarkably faster, and much more responsive.

For those interested in the issue, please look at bug 699.

Could you test out this patch from Keith Mannthey if you're having trouble? It makes irq balance a config option, which makes it easier to disable. Various people have requested it, but I don't have a box to test it on ;-( Pulled out of -mjb tree, but should go onto mainline easily.

M.

arch/i386/Kconfig +8 -0
diff -urpN -X /home/fletch/.diff.exclude 699-oom_locking/arch/i386/Kconfig 700-config_irqbal/arch/i386/Kconfig
@@ -834,6 +834,14 @@ config	1000HZ
 	bool "1000 Hz"
 endchoice
 
+config IRQBALANCE
+ 	bool "Enable kernel irq balancing"
+	depends on SMP
+	default y
+	help
+ 	  The defalut yes will allow the kernel to do irq load balancing.  
+	  Saying no will keep the kernel from doing irq load balancing. 	
+
 config HAVE_DEC_LOCK
 	bool
 	depends on (SMP || PREEMPT) && X86_CMPXCHG
arch/i386/kernel/io_apic.c +3 -5
diff -urpN -X /home/fletch/.diff.exclude 699-oom_locking/arch/i386/kernel/io_apic.c 700-config_irqbal/arch/i386/kernel/io_apic.c
@@ -263,7 +263,7 @@ static void set_ioapic_affinity (unsigne
 	spin_unlock_irqrestore(&ioapic_lock, flags);
 }
 
-#if defined(CONFIG_SMP)
+#if defined(CONFIG_IRQBALANCE) 
 # include <asm/processor.h>	/* kernel_thread() */
 # include <linux/kernel_stat.h>	/* kstat */
 # include <linux/slab.h>		/* kmalloc() */

@@ -654,8 +654,6 @@ static int __init irqbalance_disable(cha
 
 __setup("noirqbalance", irqbalance_disable);
 
-static void set_ioapic_affinity (unsigned int irq, unsigned long mask);
-
 static inline void move_irq(int irq)
 {
 	/* note - we hold the desc->lock */

@@ -667,9 +665,9 @@ static inline void move_irq(int irq)
 
 __initcall(balanced_irq_init);
 
-#else /* !SMP */
+#else /* !IRQBALANCE */
 static inline void move_irq(int irq) { }
-#endif /* defined(CONFIG_SMP) */
+#endif /* defined(IRQBALANCE) */
 
 
 /*
#3 lore raw
Quoted 6 lines · Martin J. Bligh
Quoted 6 lines

While tackling bug 699, it became clear to me that irq balancing is the cause of the performance problems I, and all people using the SMP kernel Mandrake 9.1 ships, are dealing with. I got the problems with 2.5.69 too. After disabling irq balancing, the system is remarkably faster, and much more responsive.

Quoted 6 lines
Quoted 1 line

For those interested in the issue, please look at bug 699.

Could you test out this patch from Keith Mannthey if you're having trouble? It makes irq balance a config option, which makes it easier to disable. Various people have requested it, but I don't have a box to test it on ;-( Pulled out of -mjb tree, but should go onto mainline easily.

What's wrong with the noirqbalance parameter?

Signature · 1 line
function.linuxpower.ca
#4 lore raw
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

While tackling bug 699, it became clear to me that irq balancing is the cause of the performance problems I, and all people using the SMP kernel Mandrake 9.1 ships, are dealing with. I got the problems with 2.5.69 too. After disabling irq balancing, the system is remarkably faster, and much more responsive.

Quoted 8 lines
Quoted 6 lines
Quoted 1 line

For those interested in the issue, please look at bug 699.

Could you test out this patch from Keith Mannthey if you're having trouble? It makes irq balance a config option, which makes it easier to disable. Various people have requested it, but I don't have a box to test it on ;-( Pulled out of -mjb tree, but should go onto mainline easily.

What's wrong with the noirqbalance parameter?

You have to add it ;-) Functionally equivalanet, just an easier method for people who want to do such things in userspace for whatever reason.

M.

author
#5 lore raw
In reply to Martin J. Bligh
Quoted 8 lines · Martin J. Bligh

Could you test out this patch from Keith Mannthey if you're having trouble? It makes irq balance a config option, which makes it easier to disable. Various people have requested it, but I don't have a box to test it on ;-( Pulled out of -mjb tree, but should go onto mainline easily.

M.

For 2.5.68, this patch isn't needed, the noirqbalance command line option works fine. I'll test it though, for IMHO irq balancing really should be an option, if not deleted at all. (anyone who actually gained performance with this ? Maybe it only works on cpus that are fast enough to handle all interrutps without performance going down)

For the Mandrake 2.4.21-0.13mdk kernel, there is no noirqbalance option in the kernel. I tried to contact the Mandrake guys about this, but unfortunately their response is 0. This patch also fails badly, and I haven't decided yet wether I'm willing to help a company which doesn't seem to care at all and uses pre-kernels in their distribution.

Jos

#6 lore raw
Quoted 5 lines · Jos Hulzink

For the Mandrake 2.4.21-0.13mdk kernel, there is no noirqbalance option in the kernel. I tried to contact the Mandrake guys about this, but unfortunately their response is 0. This patch also fails badly, and I haven't decided yet wether I'm willing to help a company which doesn't seem to care at all and uses pre-kernels in their distribution.

It was a bug in 2.4, fixed in Alan's tree by setting target_cpus to 0xff (previously cpu_online_map). There is no noirqbalance option in 2.4 because there is no in kernel irq balancer.

Zwane

Index: linux-2.4.21-pre1/include/asm-i386/smpboot.h
===================================================================
RCS file: /build/cvsroot/linux-2.4.19/include/asm-i386/smpboot.h,v retrieving revision 1.2

include/asm-i386/smpboot.h +1 -1
diff -u -p -B -r1.2 smpboot.h
@@ -116,6 +116,6 @@ static inline int target_cpus(void)
 	return cpu_online_map;
 }
 #else
-#define target_cpus() (cpu_online_map)
+#define target_cpus() (0xFF)
 #endif
 #endif
Signature · 1 line
function.linuxpower.ca
#7 lore raw
Quoted 3 lines · Zwane Mwaikambo

It was a bug in 2.4, fixed in Alan's tree by setting target_cpus to 0xff (previously cpu_online_map). There is no noirqbalance option in 2.4 because there is no in kernel irq balancer.

I vaguely like this notion because it removes a #ifdef and cleans up a tiny bit of its surroundings. But it's not quite a one-liner.

-- wli

 smpboot.h |   21 ++++++++++-----------
 1 files changed, 10 insertions(+), 11 deletions(-)
include/asm-i386/smpboot.h +10 -11
diff -prauN linux-2.4.21-pre7-1/include/asm-i386/smpboot.h zwane-2.4.21-pre7-1/include/asm-i386/smpboot.h
@@ -99,23 +99,22 @@
 #define cpu_to_boot_apicid(cpu) cpu_2_physical_apicid[cpu]
 #endif /* CONFIG_MULTIQUAD */
 
-#ifdef CONFIG_X86_CLUSTERED_APIC
 static inline int target_cpus(void)
 {
-	static int cpu;
-	switch(clustered_apic_mode){
+	switch (clustered_apic_mode) {
+		/* physical broadcast, routed only to local APIC bus */
 		case CLUSTERED_APIC_NUMAQ:
-			/* Broadcast intrs to local quad only. */
 			return APIC_BROADCAST_ID_APIC;
-		case CLUSTERED_APIC_XAPIC:
-			/*round robin the interrupts*/
-			cpu = (cpu+1)%smp_num_cpus;
+		/* round robin the interrupts (physical unicast) */
+		case CLUSTERED_APIC_XAPIC: {
+			static int cpu;
+			cpu = (cpu + 1) % smp_num_cpus;
 			return cpu_to_physical_apicid(cpu);
+		}
+		/* flat logical broadcast */
+		case CLUSTERED_APIC_NONE:
 		default:
+			return 0xFF;
 	}
-	return cpu_online_map;
 }
-#else
-#define target_cpus() (cpu_online_map)
-#endif
 #endif
#8 lore raw
Quoted 7 lines · William Lee Irwin III
Quoted 3 lines · Zwane Mwaikambo

It was a bug in 2.4, fixed in Alan's tree by setting target_cpus to 0xff (previously cpu_online_map). There is no noirqbalance option in 2.4 because there is no in kernel irq balancer.

I vaguely like this notion because it removes a #ifdef and cleans up a tiny bit of its surroundings. But it's not quite a one-liner.

Nice, it's during init too so there really is no need for any sort of optimisation, the inline can also go and make it __init.

Zwane

Signature · 1 line
function.linuxpower.ca
#9 lore raw
Quoted 2 lines · William Lee Irwin III
Quoted 2 lines

I vaguely like this notion because it removes a #ifdef and cleans up a tiny bit of its surroundings. But it's not quite a one-liner.

Quoted 2 lines · Zwane Mwaikambo

Nice, it's during init too so there really is no need for any sort of optimisation, the inline can also go and make it __init.

Very good point. The static variable in an inline function is very fishy also. Here it is:

-- wli

Un-#idef target_cpus() and also shove it into arch/i386/kernel/io_apic.c as both static and __init.

 arch/i386/kernel/io_apic.c |   19 +++++++++++++++++++
 include/asm-i386/smpboot.h |   20 --------------------
 2 files changed, 19 insertions(+), 20 deletions(-)
include/asm-i386/smpboot.h +0 -20
diff -prauN linux-2.4.21-pre7-1/include/asm-i386/smpboot.h zwane-2.4.21-pre7-1/include/asm-i386/smpboot.h
@@ -98,24 +98,4 @@
 #define boot_apicid_to_cpu(apicid) physical_apicid_2_cpu[apicid]
 #define cpu_to_boot_apicid(cpu) cpu_2_physical_apicid[cpu]
 #endif /* CONFIG_MULTIQUAD */
-
-#ifdef CONFIG_X86_CLUSTERED_APIC
-static inline int target_cpus(void)
-{
-	static int cpu;
-	switch(clustered_apic_mode){
-		case CLUSTERED_APIC_NUMAQ:
-			/* Broadcast intrs to local quad only. */
-			return APIC_BROADCAST_ID_APIC;
-		case CLUSTERED_APIC_XAPIC:
-			/*round robin the interrupts*/
-			cpu = (cpu+1)%smp_num_cpus;
-			return cpu_to_physical_apicid(cpu);
-		default:
-	}
-	return cpu_online_map;
-}
-#else
-#define target_cpus() (cpu_online_map)
-#endif
 #endif
arch/i386/kernel/io_apic.c +19 -0
diff -prauN linux-2.4.21-pre7-1/arch/i386/kernel/io_apic.c zwane-2.4.21-pre7-1/arch/i386/kernel/io_apic.c
@@ -614,6 +614,25 @@
 static struct hw_interrupt_type ioapic_level_irq_type;
 static struct hw_interrupt_type ioapic_edge_irq_type;
 
+static int __init target_cpus(void)
+{
+	switch (clustered_apic_mode) {
+		/* physical broadcast, routed only to local APIC bus */
+		case CLUSTERED_APIC_NUMAQ:
+			return APIC_BROADCAST_ID_APIC;
+		/* round robin the interrupts (physical unicast) */
+		case CLUSTERED_APIC_XAPIC: {
+			static int cpu;
+			cpu = (cpu + 1) % smp_num_cpus;
+			return cpu_to_physical_apicid(cpu);
+		}
+		/* flat logical broadcast */
+		case CLUSTERED_APIC_NONE:
+		default:
+			return 0xFF;
+	}
+}
+
 void __init setup_IO_APIC_irqs(void)
 {
 	struct IO_APIC_route_entry entry;
#10 lore raw
Quoted 10 lines · William Lee Irwin III
Quoted 2 lines · William Lee Irwin III
Quoted 2 lines

I vaguely like this notion because it removes a #ifdef and cleans up a tiny bit of its surroundings. But it's not quite a one-liner.

Quoted 2 lines · Zwane Mwaikambo

Nice, it's during init too so there really is no need for any sort of optimisation, the inline can also go and make it __init.

Very good point. The static variable in an inline function is very fishy also. Here it is:

Cool, could you send it to Marcelo?

include/asm-i386/smpboot.h line 113

Quoted 4 lines
include/asm-i386/smpboot.h +0 -4
-		case CLUSTERED_APIC_XAPIC:
-			/*round robin the interrupts*/
-			cpu = (cpu+1)%smp_num_cpus;
-			return cpu_to_physical_apicid(cpu);

Side note, any idea what the largest x440 this has code has run on?

Signature · 1 line
function.linuxpower.ca
#11 lore raw
In reply to Jos Hulzink
Quoted 9 lines · Jos Hulzink

Hi,

While tackling bug 699, it became clear to me that irq balancing is the cause of the performance problems I, and all people using the SMP kernel Mandrake 9.1 ships, are dealing with. I got the problems with 2.5.69 too. After disabling irq balancing, the system is remarkably faster, and much more responsive.

For those interested in the issue, please look at bug 699.

please try the following app instead:

http://people.redhat.com/arjanv/irqbalance/

the in kernel irqbalancer in 2.4 kernels has some worst case behaviors at least, and the userspace implementation avoids those.