Although CONFIG_IRQBALANCE is enabled IRQ's don't seem to be balanced very well
I enabled CONFIG_IRQBALANCE with 2.6.15 and 2.6.15-mm2 (which the numbers below are from), and had expected that to evenly (or at least close to evenly) balance IRQ's across the two CPU cores of my Athlon X2 4400+. But as you can see below, CPU0 seems to be heavily favoured
- what's the reason for that and is it something I can improve upon? Is it due to this being a Dual Core CPU and not two physically sepperate CPU's? Do I need any userspace tools in addition to CONFIG_IRQBALANCE?
juhl@dragon:~$ date && cat /proc/interrupts
Tue Jan 10 11:20:33 CET 2006
CPU0 CPU1
0: 3818901 18920 IO-APIC-edge timer
1: 8398 7 IO-APIC-edge i8042
8: 0 1 IO-APIC-edge rtc
9: 0 0 IO-APIC-level acpi
12: 210259 11 IO-APIC-edge i8042
18: 12883 1 IO-APIC-level eth0
19: 15906 291 IO-APIC-level aic7xxx
20: 1284 1 IO-APIC-level EMU10K1
NMI: 0 0
LOC: 3838143 3837641
ERR: 0
MIS: 0juhl@dragon:~$ date && cat /proc/interrupts
Tue Jan 10 12:12:38 CET 2006
CPU0 CPU1
0: 6944246 18920 IO-APIC-edge timer
1: 11291 7 IO-APIC-edge i8042
8: 0 1 IO-APIC-edge rtc
9: 0 0 IO-APIC-level acpi
12: 393217 11 IO-APIC-edge i8042
18: 38690 1 IO-APIC-level eth0
19: 204122 291 IO-APIC-level aic7xxx
20: 10877 1 IO-APIC-level EMU10K1
NMI: 0 0
LOC: 6963608 6963106
ERR: 0
MIS: 0--
Jesper Juhl <[email protected]>
Don't top-post http://www.catb.org/~esr/jargon/html/T/top-post.html
Plain text mails only, please http://www.expita.com/nomime.html
Quoted 1 line · Jesper Juhl
Do I need any userspace tools in addition to
CONFIG_IRQBALANCE?
Last I checked, yes you do need "irqbalance" (at least that's what the package is called in debian.
Jeff.
Quoted 7 lines · Josef Sipek
Quoted 1 line · Jesper Juhl
Do I need any userspace tools in addition to
CONFIG_IRQBALANCE?Last I checked, yes you do need "irqbalance" (at least that's what the package is called in debian.
Nope - you need the kernel option turned on OR the userspace daemon, not both.
If you're not generating interrupts at a high enough rate, it won't rotate. That's deliberate.
M.
Quoted 12 lines · Martin Bligh
Quoted 7 lines · Josef Sipek
Quoted 1 line · Jesper Juhl
Do I need any userspace tools in addition to
CONFIG_IRQBALANCE?Last I checked, yes you do need "irqbalance" (at least that's what the package is called in debian.
Nope - you need the kernel option turned on OR the userspace daemon, not both.
Ok, good to know.
Quoted 3 lines
If you're not generating interrupts at a high enough rate, it won't rotate. That's deliberate.
Hmm, and what would count as "a high enough rate"?
I just did a small test with thousands of ping -f's through my NIC while at the same time giving the disk a good workout with tons of find's, sync's & updatedb's - that sure did drive up the number of interrupts and my load average went sky high (amazingly the box was still fairly responsive):
root@dragon:/home/juhl# uptime 22:59:58 up 12:43, 1 user, load average: 1015.48, 715.93, 429.07
but, not a single interrupt was handled by CPU1, they all went to CPU0.
Do you have a good way to drive up the nr of interrupts above the treshhold for balancing?
--
Jesper Juhl <[email protected]>
Don't top-post http://www.catb.org/~esr/jargon/html/T/top-post.html
Plain text mails only, please http://www.expita.com/nomime.html
Quoted 33 lines · Jesper Juhl
Quoted 5 lines · Martin Bligh
Quoted 3 lines · Josef Sipek
On Tue, Jan 10, 2006 at 12:14:42PM +0100, Jesper Juhl wrote:
Quoted 9 lines
Quoted 5 lines
Quoted 1 line
Do I need any userspace tools in addition to
CONFIG_IRQBALANCE?Last I checked, yes you do need "irqbalance" (at least that's what the package is called in debian.
Nope - you need the kernel option turned on OR the userspace daemon, not both.
Ok, good to know.
Quoted 3 lines
If you're not generating interrupts at a high enough rate, it won't rotate. That's deliberate.
Hmm, and what would count as "a high enough rate"?
I just did a small test with thousands of ping -f's through my NIC while at the same time giving the disk a good workout with tons of find's, sync's & updatedb's - that sure did drive up the number of interrupts and my load average went sky high (amazingly the box was still fairly responsive):
Quoted 7 lines
root@dragon:/home/juhl# uptime 22:59:58 up 12:43, 1 user, load average: 1015.48, 715.93, 429.07but, not a single interrupt was handled by CPU1, they all went to CPU0.
Do you have a good way to drive up the nr of interrupts above the treshhold for balancing?
Is it HT? ISTR it was intelligent enough to ignore that. But you'd have to look at the code to be sure.
M.
Quoted 37 lines · Martin Bligh
Quoted 36 lines · Jesper Juhl
Quoted 5 lines · Martin Bligh
Quoted 3 lines · Josef Sipek
On Tue, Jan 10, 2006 at 12:14:42PM +0100, Jesper Juhl wrote:
Quoted 9 lines
Quoted 5 lines
Quoted 1 line
Do I need any userspace tools in addition to
CONFIG_IRQBALANCE?Last I checked, yes you do need "irqbalance" (at least that's what the package is called in debian.
Nope - you need the kernel option turned on OR the userspace daemon, not both.
Ok, good to know.
Quoted 3 lines
If you're not generating interrupts at a high enough rate, it won't rotate. That's deliberate.
Hmm, and what would count as "a high enough rate"?
I just did a small test with thousands of ping -f's through my NIC while at the same time giving the disk a good workout with tons of find's, sync's & updatedb's - that sure did drive up the number of interrupts and my load average went sky high (amazingly the box was still fairly responsive):
root@dragon:/home/juhl# uptime 22:59:58 up 12:43, 1 user, load average: 1015.48, 715.93, 429.07
Quoted 8 lines
Quoted 4 lines
but, not a single interrupt was handled by CPU1, they all went to CPU0.
Do you have a good way to drive up the nr of interrupts above the treshhold for balancing?
Is it HT? ISTR it was intelligent enough to ignore that. But you'd have to look at the code to be sure.
Dual Core Athlon 64 X2 4400+
--
Jesper Juhl <[email protected]>
Don't top-post http://www.catb.org/~esr/jargon/html/T/top-post.html
Plain text mails only, please http://www.expita.com/nomime.html
Quoted 19 lines · Jesper Juhl
Quoted 18 lines · Martin Bligh
Quoted 17 lines · Jesper Juhl
Quoted 14 lines · Martin Bligh
Quoted 8 lines · Josef Sipek
Quoted 1 line · Jesper Juhl
Do I need any userspace tools in addition to
CONFIG_IRQBALANCE?Last I checked, yes you do need "irqbalance" (at least that's what the package is called in debian.
Nope - you need the kernel option turned on OR the userspace daemon, not both.
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Ok, good to know.
Quoted 3 lines
If you're not generating interrupts at a high enough rate, it won't rotate. That's deliberate.
What I have read is that first CPU is used more for interrupts to use the concept of maximizing cache locality. Probably kernel is optimizing this even with CONFIG option enabled.
Nauman
Quoted 33 lines
Quoted 13 lines
Quoted 13 lines
Hmm, and what would count as "a high enough rate"?
I just did a small test with thousands of ping -f's through my NIC while at the same time giving the disk a good workout with tons of find's, sync's & updatedb's - that sure did drive up the number of interrupts and my load average went sky high (amazingly the box was still fairly responsive):
root@dragon:/home/juhl# uptime 22:59:58 up 12:43, 1 user, load average: 1015.48, 715.93, 429.07Quoted 8 lines
Quoted 4 lines
but, not a single interrupt was handled by CPU1, they all went to CPU0.
Do you have a good way to drive up the nr of interrupts above the treshhold for balancing?
Is it HT? ISTR it was intelligent enough to ignore that. But you'd have to look at the code to be sure.
Dual Core Athlon 64 X2 4400+
--
Jesper Juhl <[email protected]>
Don't top-post http://www.catb.org/~esr/jargon/html/T/top-post.html
Plain text mails only, please http://www.expita.com/nomime.html
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Quoted 15 lines · Nauman Tahir
Quoted 14 lines · Jesper Juhl
Quoted 13 lines · Martin Bligh
Quoted 12 lines · Jesper Juhl
Quoted 10 lines · Martin Bligh
Quoted 6 lines · Josef Sipek
Quoted 1 line · Jesper Juhl
Do I need any userspace tools in addition to
CONFIG_IRQBALANCE?Last I checked, yes you do need "irqbalance" (at least that's what the package is called in debian.
Nope - you need the kernel option turned on OR the userspace daemon, not both.
Quoted 11 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Ok, good to know.
Quoted 2 lines
If you're not generating interrupts at a high enough rate, it won't rotate. That's deliberate.
What I have read is that first CPU is used more for interrupts to use the concept of maximizing cache locality. Probably kernel is optimizing this even with CONFIG option enabled.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Hmm, and what would count as "a high enough rate"?
This is what I tested a few months ago:
Test system: 2 dual Pentium3 systems
- with 2.6.11 kernel and kernel IRQ balancing;
- each with an Intel dual port E1000 NIC (e1000 driver 6.0.54);
- both systems connected back-to-back to each other with 2 links.
Test 1:
- I started 1 UDP flow (< 23 Mbps) on the first link with the Iperf network performance measurement tool. For a UDP bandwidth lower than 23 Mbps the interrupt rate at the receiver interface was lower than 2000 interrupts per second. In this case all interrupts were distributed to CPU 0. 2000 interrupts per second seemed to be the threshold for the interrupts to be distributed to 1 CPU.
Test 2:
- Then I started 1 UDP flow of 600 Mbps on the first link. 8000 interrupts per second were generated by the receiver interface. Approximately half of the interrupts were distributed to CPU 0, the other half to CPU 1.
Test 3:
- Then I did a test with 2 UDP flows of 600 Mbps, each over their own link. 8000 interrupts per second were generated by both receiver interfaces. All interrupts generated by the 1st interface were distributed to CPU 0, all interrupts generated by the 2nd interface were distributed to CPU 1.
Quoted 20 lines
Quoted 20 lines
Quoted 11 lines
Quoted 11 lines
I just did a small test with thousands of ping -f's through my NIC while at the same time giving the disk a good workout with tons of find's, sync's & updatedb's - that sure did drive up the number of interrupts and my load average went sky high (amazingly the box was still fairly responsive):
root@dragon:/home/juhl# uptime 22:59:58 up 12:43, 1 user, load average: 1015.48, 715.93, 429.07 but, not a single interrupt was handled by CPU1, they all went to CPU0.Quoted 6 lines
Quoted 2 lines
Do you have a good way to drive up the nr of interrupts above the treshhold for balancing?
Is it HT? ISTR it was intelligent enough to ignore that. But you'd have to look at the code to be sure.
Dual Core Athlon 64 X2 4400+