2.6.1 and irq balancing
Greetings all,
I upgraded my server to 2.6.1, and I'm finding I'm saddled with only interrupting on CPU0 again. 2.6.0 does this as well. This is the Supermicro X5DPL-iGM-O (E7501 chipset), 2 [email protected] HT enabled. /proc/cpuinfo is normal as per HT, displaying 4 cpus. 2.4.2(3|4) exhibited this behaviour as well, until I applied patches from here: http://www.hardrock.org/kernel/2.4.23/irqbalance-2.4.23-jb.patch, et al.
CPU0 CPU1 CPU2 CPU3
0: 1572323 0 0 0 IO-APIC-edge timer 2: 0 0 0 0 XT-PIC cascade 3: 23520 0 0 0 IO-APIC-edge serial 8: 2 0 0 0 IO-APIC-edge rtc 9: 0 0 0 0 IO-APIC-level acpi 14: 10 0 0 0 IO-APIC-edge ide0 16: 30 0 0 0 IO-APIC-level sym53c8xx 22: 4162 0 0 0 IO-APIC-level eth0 48: 7798 0 0 0 IO-APIC-level aic79xx 49: 3385 0 0 0 IO-APIC-level aic79xx 54: 17062 0 0 0 IO-APIC-level eth1 NMI: 0 0 0 0 LOC: 1572002 1572251 1572250 1572243 ERR: 0 MIS: 0
THey keyboard isn't working either, but we see the i8042..
serio: i8042 KBD port at 0x60,0x64 irq 1
-Ethan
Hi,
What is the load on the box when this is happening? If its low think this is optimal (for cache reasons).
Ed Tomlinson
Quoted 41 lines · Ethan Weinstein
Greetings all,
I upgraded my server to 2.6.1, and I'm finding I'm saddled with only interrupting on CPU0 again. 2.6.0 does this as well. This is the Supermicro X5DPL-iGM-O (E7501 chipset), 2 [email protected] HT enabled.
/proc/cpuinfois normal as per HT, displaying 4 cpus. 2.4.2(3|4) exhibited this behaviour as well, until I applied patches from here:
http://www.hardrock.org/kernel/2.4.23/irqbalance-2.4.23-jb.patch, et al.CPU0 CPU1 CPU2 CPU3
0: 1572323 0 0 0 IO-APIC-edge timer 2: 0 0 0 0 XT-PIC cascade 3: 23520 0 0 0 IO-APIC-edge serial 8: 2 0 0 0 IO-APIC-edge rtc 9: 0 0 0 0 IO-APIC-level acpi 14: 10 0 0 0 IO-APIC-edge ide0 16: 30 0 0 0 IO-APIC-level sym53c8xx 22: 4162 0 0 0 IO-APIC-level eth0 48: 7798 0 0 0 IO-APIC-level aic79xx 49: 3385 0 0 0 IO-APIC-level aic79xx 54: 17062 0 0 0 IO-APIC-level eth1 NMI: 0 0 0 0 LOC: 1572002 1572251 1572250 1572243 ERR: 0 MIS: 0THey keyboard isn't working either, but we see the i8042..
serio: i8042 KBD port at 0x60,0x64 irq 1
(quoted diff) +0
-2-Ethan -To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Quoted 6 lines · Ed Tomlinson
Hi,
What is the load on the box when this is happening? If its low think this is optimal (for cache reasons).
I'd rather see different interrupt sources run on different CPUs initially, which would help fairness a little bit, and should be more optimal with big interrupt loads.
0: xxx1 0 0 0 1: 0 xxx2 0 0 2: 0 0 xxx3 0 3: 0 0 0 xxx4
This would delay the need for interrupt balancing in the case where 2 or more interrupts are heavily used.
Quoted 5 lines · Ed Tomlinson
Hi,
What is the load on the box when this is happening? If its low think this is optimal (for cache reasons).
Admittedly, the machine's load was not high when I took this sample. However, creating a great deal of load does not change these statistics at all. Being that there are patches available for 2.4.x kernels to fix this, I don't think this at all by design, but what do I know? =)
2.6.0 running on a non-HT SMP machine I have (old Compaq proliant 2xPentium2) does interrupt on all CPU's with "noirqbalance" bootparam.
Regarding the keyboard, I noticed something interesting
2.6.1-rc1 shows the i8042 in /proc/interrupts:
1: 1871 0 0 0 IO-APIC-edge i8042
(keyboard still does not work, though..)
2.6.1 final does not show this at all, and [kseriod] eats a constant 5%
CPU. Something's awry =)
-Ethan
Quoted 6 lines · Ethan Weinstein
Greetings all,
I upgraded my server to 2.6.1, and I'm finding I'm saddled with only interrupting on CPU0 again. 2.6.0 does this as well. This is the Supermicro X5DPL-iGM-O (E7501 chipset), 2 [email protected] HT enabled.
/proc/cpuinfois normal as per HT, displaying 4 cpus.
you should run the userspace irq balance daemon:
http://people.redhat.com/arjanv/irqbalance/
(or part of the kernel-utils package if you run a RH based distro; afaik SuSE ships it too but I don't know in what package)
Quoted 14 lines · Nick Piggin
Quoted 6 lines · Ed Tomlinson
Hi,
What is the load on the box when this is happening? If its low think this is optimal (for cache reasons).
I'd rather see different interrupt sources run on different CPUs initially, which would help fairness a little bit, and should be more optimal with big interrupt loads.
Quoted 7 lines
0: xxx1 0 0 0 1: 0 xxx2 0 0 2: 0 0 xxx3 0 3: 0 0 0 xxx4This would delay the need for interrupt balancing in the case where 2 or more interrupts are heavily used.
this is what irqbalanced will do (but more inteligent than just using the irq number as round robin seed).
Quoted 29 lines · Ethan Weinstein
Greetings all,
I upgraded my server to 2.6.1, and I'm finding I'm saddled with only interrupting on CPU0 again. 2.6.0 does this as well. This is the Supermicro X5DPL-iGM-O (E7501 chipset), 2 [email protected] HT enabled.
/proc/cpuinfois normal as per HT, displaying 4 cpus. 2.4.2(3|4) exhibited this behaviour as well, until I applied patches from here:
http://www.hardrock.org/kernel/2.4.23/irqbalance-2.4.23-jb.patch, et al.CPU0 CPU1 CPU2 CPU3
0: 1572323 0 0 0 IO-APIC-edge timer 2: 0 0 0 0 XT-PIC cascade 3: 23520 0 0 0 IO-APIC-edge serial 8: 2 0 0 0 IO-APIC-edge rtc 9: 0 0 0 0 IO-APIC-level acpi 14: 10 0 0 0 IO-APIC-edge ide0 16: 30 0 0 0 IO-APIC-level sym53c8xx 22: 4162 0 0 0 IO-APIC-level eth0 48: 7798 0 0 0 IO-APIC-level aic79xx 49: 3385 0 0 0 IO-APIC-level aic79xx 54: 17062 0 0 0 IO-APIC-level eth1 NMI: 0 0 0 0 LOC: 1572002 1572251 1572250 1572243 ERR: 0 MIS: 0
Quoted 4 lines
THey keyboard isn't working either, but we see the i8042..
serio: i8042 KBD port at 0x60,0x64 irq 1
Are you running irqbalance ?
Signature · 1 line
Martin Schlemmer
Quoted 10 lines · Arjan van de Ven
Quoted 6 lines · Ethan Weinstein
Greetings all,
I upgraded my server to 2.6.1, and I'm finding I'm saddled with only interrupting on CPU0 again. 2.6.0 does this as well. This is the Supermicro X5DPL-iGM-O (E7501 chipset), 2 [email protected] HT enabled.
/proc/cpuinfois normal as per HT, displaying 4 cpus.you should run the userspace irq balance daemon:
http://people.redhat.com/arjanv/irqbalance/
I have long wondered what is so evil about most interrupts going to CPU 0 that we felt we had to have a pair of irqdaemons in 2.6. From my (admittedly imperfect) experience, the APIC will route an interrupt to CPU 1 if CPU 0 is busy with another interrupt, to CPU 2 if 0 and 1 are so occupied, and so on. I see no harm in this other than the strangely lopsided /proc/interrupt displays, which I can live with.
Earlier APICs had a variation where the search for where each new interrupt was to go started with first cpu after the one that got the last interrupt. If we call this 'round-robin' allocation, then today's technique could be described as 'first fit'.
If I am wrong about this, I would dearly love to be corrected :) Joe
Quoted 9 lines · Joe Korty
Quoted 8 lines · Arjan van de Ven
Quoted 6 lines · Ethan Weinstein
Greetings all,
I upgraded my server to 2.6.1, and I'm finding I'm saddled with only interrupting on CPU0 again. 2.6.0 does this as well. This is the Supermicro X5DPL-iGM-O (E7501 chipset), 2 [email protected] HT enabled.
/proc/cpuinfois normal as per HT, displaying 4 cpus.
Quoted 5 lines
Quoted 2 lines
you should run the userspace irq balance daemon:
http://people.redhat.com/arjanv/irqbalance/I have long wondered what is so evil about most interrupts going to CPU 0 that we felt we had to have a pair of irqdaemons in 2.6.
well irqbalanced is a userspace balancer
Quoted 4 lines
Earlier APICs had a variation where the search for where each new interrupt was to go started with first cpu after the one that got the last interrupt. If we call this 'round-robin' allocation, then today's technique could be described as 'first fit'.
if it's really busy it starves cpu0 ....
Hi!
Quoted 10 lines
Quoted 9 lines
Quoted 6 lines
Greetings all,
I upgraded my server to 2.6.1, and I'm finding I'm saddled with only interrupting on CPU0 again. 2.6.0 does this as well. This is the Supermicro X5DPL-iGM-O (E7501 chipset), 2 [email protected] HT enabled.
/proc/cpuinfois normal as per HT, displaying 4 cpus.you should run the userspace irq balance daemon:
http://people.redhat.com/arjanv/irqbalance/
Quoted 6 lines
I have long wondered what is so evil about most interrupts going to CPU 0 that we felt we had to have a pair of irqdaemons in 2.6. From my (admittedly imperfect) experience, the APIC will route an interrupt to CPU 1 if CPU 0 is busy with another interrupt, to CPU 2 if 0 and 1 are so occupied, and so on. I see no harm in this other than the strangely lopsided
/proc/interruptdisplays, which I can live with.
Well, imagine 8 CPU machine with high interrupt load. Poor process that gets scheduled on CPU#0 does little progress, but is shown as eating one whole CPU.
Pavel
Signature · 2 lines
When do you have a heart between your knees? [Johanka's followup: and *two* hearts?]