irq balancing question
hi
i have a SMP with 2x duo-core Opteron processors, and i have an ethernet card (eth0). i want to balance the interrupts generated by the card to all 4 processors. i can do this by writing to smp_affinity, but on another machine it get's done by "default", without doing anything.
is this irq balancing a feature in the chipset, or driver, or kernel ? i got the idea of the smp_affinity stuff, and i've set it to "f", but nothing happens, i mean i see only one CPU getting the interrupts.
i'm using kernel 2.6.18, but tried with 2.6.19, too.
thanks.
Quoted 5 lines · Imre Gergely
hi
i have a SMP with 2x duo-core Opteron processors, and i have an ethernet card (eth0). i want to balance the interrupts generated by the card to all 4 processors.
you have only one card? then balancing like you say will only really hurt you; neither linux nor the chipset will do this.
Quoted 3 lines
is this irq balancing a feature in the chipset, or driver, or kernel ? i got the idea of the
smp_affinitystuff, and i've set it to "f", but nothing happens, i mean i see only one CPU getting the interrupts.
if you have multiple real interrupt sources you can balance between them using irqbalance, see http://www.irqbalance.org
But note that this program will NOT move one irq across 4 cpus all the time, since that gives *really* bad performance.
Signature · 2 lines
if you want to mail me at work (you don't), use arjan (at) linux.intel.com Test the interaction between Linux and your BIOS via http://www.linuxfirmwarekit.org
Quoted 9 lines · Arjan van de Ven
Quoted 5 lines · Imre Gergely
hi
i have a SMP with 2x duo-core Opteron processors, and i have an ethernet card (eth0). i want to balance the interrupts generated by the card to all 4 processors.
you have only one card? then balancing like you say will only really hurt you; neither linux nor the chipset will do this.
in fact i have two cards, and 4 CPUs, but i was interested in then answer Robert gave, that only _some_ machines distribute interrupts in hardware. software distribution is obviously not good. consider this scenario:
you have one machine with 4 cpus, and two ethernet cards with a lot of traffic on them. if you bind every card to one cpu, two of them are not used, so you really use only half the power. not let's say you have so much traffic (with limiting enabled, htb or something), that the two CPUs are on 100% all the time, but the other two are doing nothing.
now if you could balance that to all 4 cpus, you could use all the power AND no cpu would be used 100%.
but i guess the machine has to do it in hardware, and that was the real question. irqbalance has no use here, i think, only if the hardware supports it.
is there a way to find out (some easy way) for sure, if one particular hardware supports this interrupt distribution?
Quoted 5 lines
in fact i have two cards, and 4 CPUs, but i was interested in then answer Robert gave, that only _some_ machines distribute interrupts in hardware. software distribution is obviously not good. consider this scenario:
Quoted 8 lines
you have one machine with 4 cpus, and two ethernet cards with a lot of traffic on them. if you bind every card to one cpu, two of them are not used, so you really use only half the power. not let's say you have so much traffic (with limiting enabled, htb or something), that the two CPUs are on 100% all the time, but the other two are doing nothing.
now if you could balance that to all 4 cpus, you could use all the power AND no cpu would be used 100%.
actually this will give you worse performance than only using 2 cores. The reason for this is twofold
1) If you rotate the irqs, TCP and IP packet fragments will arrive at different CPUs. This in turn means that a VERY expensive reassembly path gets taken, compared to local-cpu-only reassembly
2) If you rotate the irqs, you bounce cachelines between the caches ALL THE TIME, which is also very expensive.
Both make it more likely that you'll be slower than just using only 2 cores...
Signature · 2 lines
if you want to mail me at work (you don't), use arjan (at) linux.intel.com Test the interaction between Linux and your BIOS via http://www.linuxfirmwarekit.org
Quoted 21 lines · Arjan van de Ven
Quoted 4 lines
in fact i have two cards, and 4 CPUs, but i was interested in then answer Robert gave, that only _some_ machines distribute interrupts in hardware. software distribution is obviously not good. consider this scenario:
Quoted 8 lines
you have one machine with 4 cpus, and two ethernet cards with a lot of traffic on them. if you bind every card to one cpu, two of them are not used, so you really use only half the power. not let's say you have so much traffic (with limiting enabled, htb or something), that the two CPUs are on 100% all the time, but the other two are doing nothing.
now if you could balance that to all 4 cpus, you could use all the power AND no cpu would be used 100%.
actually this will give you worse performance than only using 2 cores. The reason for this is twofold
1) If you rotate the irqs, TCP and IP packet fragments will arrive at different CPUs. This in turn means that a VERY expensive reassembly path gets taken, compared to local-cpu-only reassembly
2) If you rotate the irqs, you bounce cachelines between the caches ALL THE TIME, which is also very expensive.
Quoted 2 lines
Both make it more likely that you'll be slower than just using only 2 cores...
and i guess it doesn't matter if the distribution is being done by the hardware, from the point of view of the kernel, i would still get the performance penalty.
and what if CPU0 and CPU1 is actually the same CPU, only duo core, and i'm distributing interrupts to them, and with the other card to CPU2 and 3 (which are part of the other physical CPU) ?
i'm just trying to figure it out, i have no real knowledge of the inner kernel workings, so i dont know. but i really would like to use all 4 cores. just how expensive is that reassembly path ?
Quoted 5 lines
and i guess it doesn't matter if the distribution is being done by the hardware, from the point of view of the kernel, i would still get the performance penalty.
for the cache bouncing you save assuming you have an Intel cpu (eg one where the cache is shared). You don't save on the cross-cpu reassembly, that is an entire different algorithm path you hit there..
Quoted 3 lines
i'm just trying to figure it out, i have no real knowledge of the inner kernel workings, so i dont know. but i really would like to use all 4 cores. just how expensive is that reassembly path ?
depends on your traffic to be honest, probably a question more suited for net-dev list.
Signature · 2 lines
if you want to mail me at work (you don't), use arjan (at) linux.intel.com Test the interaction between Linux and your BIOS via http://www.linuxfirmwarekit.org
Quoted 10 lines · Arjan van de Ven
Quoted 3 lines
and i guess it doesn't matter if the distribution is being done by the hardware, from the point of view of the kernel, i would still get the performance penalty.
for the cache bouncing you save assuming you have an Intel cpu (eg one where the cache is shared). You don't save on the cross-cpu reassembly, that is an entire different algorithm path you hit there..
Quoted 7 lines
Quoted 3 lines
i'm just trying to figure it out, i have no real knowledge of the inner kernel workings, so i dont know. but i really would like to use all 4 cores. just how expensive is that reassembly path ?
depends on your traffic to be honest, probably a question more suited for net-dev list.
i'll look into it, thanks a lot for the comments.