IRQ balancing on a router
Hello,
I have a dual-CPU router/firewall with five gigabit NICs. Recently I have found that irqbalance (0.55 from Fedora 9/x86_64) gives a suboptimal IRQ to CPU mapping on this box:
During traffic spikes, it assings two NICs to one CPU, and the
other three to the second CPU. However, this does not account for the fact that packets coming from the uplink interface are way more expensive to handle than the rest of the traffic: most iptables rules apply to the packets received from the uplink interface. The result is that the CPU which receives IRQs for the uplink interface is 100 % busy (softirq mostly), while the other one is 90% idle.
Setting the IRQ mapping by hand (uplink to one CPU, all the other
NICs to the other CPU) makes a well balanced system (both CPUs 30-60 % busy). I am not sure whether my configuration is too special, but it might be worth trying to make irqbalance daemon cope also with this usage pattern.
Another problem is that with one CPU 100 % busy in the kernel
the system latency of user-space programs is _way_ too high. For example, MRTG graphs from my router used to have blank stripes (i.e. snmpd has failed to respond in time). Also the shell response time was bad, even though I was logged in using SSH over the interface, which at that time had its IRQ routed to the other CPU.
With the same network load and manual IRQ to CPU assignment,
MRTG works well and both snmpd and shell response time is good.
-Yenya
Signature · 3 lines
| Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> |
| GPG: ID 1024/D3498839 Fingerprint 0D99A7FB206605D7 8B35FCDE05B18A5E |
| http://www.fi.muni.cz/~kas/ Journal: http://www.fi.muni.cz/~kas/blog/ |Quoted 2 lines
Quoted 2 lines
If you find yourself arguing with Alan Cox, you’re _probably_ wrong. <<
--James Morris in "How and Why You Should Become a Kernel Hacker" <<
Quoted 14 lines · On Fri, 3 Oct 2008 15:21:17 +0200 Jan Kasprzak <[email protected]> wrote:
Hello,
I have a dual-CPU router/firewall with five gigabit NICs. Recently I have found that irqbalance (0.55 from Fedora 9/x86_64) gives a suboptimal IRQ to CPU mapping on this box:
During traffic spikes, it assings two NICs to one CPU, and the
other three to the second CPU. However, this does not account for the fact that packets coming from the uplink interface are way more expensive to handle than the rest of the traffic: most iptables rules apply to the packets received from the uplink interface. The result is that the CPU which receives IRQs for the uplink interface is 100 % busy (softirq mostly), while the other one is 90% idle.
Quoted 6 lines
Setting the IRQ mapping by hand (uplink to one CPU, all the
other NICs to the other CPU) makes a well balanced system (both CPUs 30-60 % busy). I am not sure whether my configuration is too special, but it might be worth trying to make irqbalance daemon cope also with this usage pattern.
one of the hard cases for irqbalance is that irqbalance doesn't have a way to find out the actual cpu time spend in the handlers. For networking it makes an estimate just based on the number of packets (which is better than nothing)... but that breaks down if you have an non-symmetry in CPU costs per packet like you have.
The good news is that irqthreads at least have the potential to solve this "lack of information"; if not, we could consider doing a form of microaccounting for irq handlers....
Signature · 3 lines
Arjan van de Ven Intel Open Source Technology Centre For development, discussion and tips for power savings, visit http://www.lesswatts.org
Quoted 27 lines · Arjan van de Ven
Quoted 14 lines
Hello,
I have a dual-CPU router/firewall with five gigabit NICs. Recently I have found that irqbalance (0.55 from Fedora 9/x86_64) gives a suboptimal IRQ to CPU mapping on this box:
During traffic spikes, it assings two NICs to one CPU, and the
other three to the second CPU. However, this does not account for the fact that packets coming from the uplink interface are way more expensive to handle than the rest of the traffic: most iptables rules apply to the packets received from the uplink interface. The result is that the CPU which receives IRQs for the uplink interface is 100 % busy (softirq mostly), while the other one is 90% idle.Quoted 6 lines
Setting the IRQ mapping by hand (uplink to one CPU, all the
other NICs to the other CPU) makes a well balanced system (both CPUs 30-60 % busy). I am not sure whether my configuration is too special, but it might be worth trying to make irqbalance daemon cope also with this usage pattern.one of the hard cases for irqbalance is that irqbalance doesn't have a way to find out the actual cpu time spend in the handlers. For networking it makes an estimate just based on the number of packets (which is better than nothing)... but that breaks down if you have an non-symmetry in CPU costs per packet like you have.
Quoted 5 lines
The good news is that irqthreads at least have the potential to solve this "lack of information"; if not, we could consider doing a form of microaccounting for irq handlers....
perhaps, this could be addressed using tracepoints. The currently proposed ones are at the beginning and end of 'handle_IRQ_event()'. See:
http://marc.info/?l=linux-kernel&m=121616099830280&w=2
thanks,
-Jason
Quoted 12 lines · Jason Baron
Quoted 11 lines
one of the hard cases for irqbalance is that irqbalance doesn't have a way to find out the actual cpu time spend in the handlers. For networking it makes an estimate just based on the number of packets (which is better than nothing)... but that breaks down if you have an non-symmetry in CPU costs per packet like you have.
The good news is that irqthreads at least have the potential to solve this "lack of information"; if not, we could consider doing a form of microaccounting for irq handlers....
Quoted 4 lines
perhaps, this could be addressed using tracepoints. The currently proposed ones are at the beginning and end of '
handle_IRQ_event()'. See:
http://marc.info/?l=linux-kernel&m=121616099830280&w=2
something that you always need should not be a tracepoint.
Arjan van de Ven wrote:
: Jan Kasprzak <[email protected]> wrote:
: > The result is
: > that the CPU which receives IRQs for the uplink interface : > is 100 % busy (softirq mostly), while the other one is 90% idle. :
: one of the hard cases for irqbalance is that irqbalance doesn't have a : way to find out the actual cpu time spend in the handlers. For : networking it makes an estimate just based on the number of packets : (which is better than nothing)... but that breaks down if you have an : non-symmetry in CPU costs per packet like you have.
:
: The good news is that irqthreads at least have the potential to solve : this "lack of information"; if not, we could consider doing a form of : microaccounting for irq handlers....
I am not sure whether this would help. In my case, the most of the
in-kernel CPU time is not spend in the irq handler per se, but in softirq (i.e. checking the packet against iptables rules).
-Yenya
Signature · 3 lines
| Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> |
| GPG: ID 1024/D3498839 Fingerprint 0D99A7FB206605D7 8B35FCDE05B18A5E |
| http://www.fi.muni.cz/~kas/ Journal: http://www.fi.muni.cz/~kas/blog/ |Quoted 2 lines
Quoted 2 lines
If you find yourself arguing with Alan Cox, you’re _probably_ wrong. <<
--James Morris in "How and Why You Should Become a Kernel Hacker" <<
Quoted 14 lines · Jan Kasprzak
Arjan van de Ven wrote:
: Jan Kasprzak <[email protected]> wrote:
: > The result is
: > that the CPU which receives IRQs for the uplink interface : > is 100 % busy (softirq mostly), while the other one is 90% idle. : : one of the hard cases for irqbalance is that irqbalance doesn't have a : way to find out the actual cpu time spend in the handlers. For : networking it makes an estimate just based on the number of packets : (which is better than nothing)... but that breaks down if you have an : non-symmetry in CPU costs per packet like you have.
: : The good news is that irqthreads at least have the potential to solve : this "lack of information"; if not, we could consider doing a form of : microaccounting for irq handlers....
Quoted 3 lines
I am not sure whether this would help. In my case, the most of the
in-kernel CPU time is not spend in the irq handler per se, but in softirq (i.e. checking the packet against iptables rules).
there is some consideration of making softirqs that are raised run as part of the irq thread. or at least thoughts in that direction.
Quoted 3 lines · Arjan van de Ven
The good news is that irqthreads at least have the potential to solve this "lack of information"; if not, we could consider doing a form of microaccounting for irq handlers....
I have some patches floating about that account for nmi/irq/softirq time in a fine grained scale. The trouble is that i've so far not found a way to handle the case of a jiffie based sched_clock().
The trouble with that is that time always increases in IRQ context, so nmi=0, softirq=0, regular=0 but irq=100%, which is obviuosly sub-optimal :-)
Quoted 8 lines · Peter Zijlstra
Quoted 3 lines · Arjan van de Ven
The good news is that irqthreads at least have the potential to solve this "lack of information"; if not, we could consider doing a form of microaccounting for irq handlers....
I have some patches floating about that account for nmi/irq/softirq time in a fine grained scale. The trouble is that i've so far not found a way to handle the case of a jiffie based
sched_clock().
Would be nice to have.
Quoted 3 lines
The trouble with that is that time always increases in IRQ context, so nmi=0, softirq=0, regular=0 but irq=100%, which is obviuosly sub-optimal :-)
Can't you exempt the timer interrupt from counting itself as irq in the case of timer interrupt based sched_clock()?