Skip to contentSkip to conversation

IRQ balancing on a router

DISCUSSION Jan Kasprzak 5 participants 8 messages Oct 3, 2008 – Oct 7, 2008

lore
Share & tools
lore
#0 lore raw

Hello,

I have a dual-CPU router/firewall with five gigabit NICs. Recently I have found that irqbalance (0.55 from Fedora 9/x86_64) gives a suboptimal IRQ to CPU mapping on this box:

During traffic spikes, it assings two NICs to one CPU, and the
other three to the second CPU. However, this does not account for the fact that packets coming from the uplink interface are way more expensive to handle than the rest of the traffic: most iptables rules apply to the packets received from the uplink interface. The result is that the CPU which receives IRQs for the uplink interface is 100 % busy (softirq mostly), while the other one is 90% idle.

Setting the IRQ mapping by hand (uplink to one CPU, all the other
NICs to the other CPU) makes a well balanced system (both CPUs 30-60 % busy). I am not sure whether my configuration is too special, but it might be worth trying to make irqbalance daemon cope also with this usage pattern.

Another problem is that with one CPU 100 % busy in the kernel
the system latency of user-space programs is _way_ too high. For example, MRTG graphs from my router used to have blank stripes (i.e. snmpd has failed to respond in time). Also the shell response time was bad, even though I was logged in using SSH over the interface, which at that time had its IRQ routed to the other CPU.

With the same network load and manual IRQ to CPU assignment,
MRTG works well and both snmpd and shell response time is good.

-Yenya

Signature · 3 lines
| Jan "Yenya" Kasprzak  <kas at {fi.muni.cz - work | yenya.net - private}> |
| GPG: ID 1024/D3498839      Fingerprint 0D99A7FB206605D7 8B35FCDE05B18A5E |
| http://www.fi.muni.cz/~kas/    Journal: http://www.fi.muni.cz/~kas/blog/ |
Quoted 2 lines
Quoted 2 lines

If you find yourself arguing with Alan Cox, you’re _probably_ wrong. <<
--James Morris in "How and Why You Should Become a Kernel Hacker" <<

#1 lore raw
Quoted 14 lines · On Fri, 3 Oct 2008 15:21:17 +0200 Jan Kasprzak <[email protected]> wrote:

Hello,

I have a dual-CPU router/firewall with five gigabit NICs. Recently I have found that irqbalance (0.55 from Fedora 9/x86_64) gives a suboptimal IRQ to CPU mapping on this box:

During traffic spikes, it assings two NICs to one CPU, and the
other three to the second CPU. However, this does not account for the fact that packets coming from the uplink interface are way more expensive to handle than the rest of the traffic: most iptables rules apply to the packets received from the uplink interface. The result is that the CPU which receives IRQs for the uplink interface is 100 % busy (softirq mostly), while the other one is 90% idle.

Quoted 6 lines

Setting the IRQ mapping by hand (uplink to one CPU, all the
other NICs to the other CPU) makes a well balanced system (both CPUs 30-60 % busy). I am not sure whether my configuration is too special, but it might be worth trying to make irqbalance daemon cope also with this usage pattern.

one of the hard cases for irqbalance is that irqbalance doesn't have a way to find out the actual cpu time spend in the handlers. For networking it makes an estimate just based on the number of packets (which is better than nothing)... but that breaks down if you have an non-symmetry in CPU costs per packet like you have.

The good news is that irqthreads at least have the potential to solve this "lack of information"; if not, we could consider doing a form of microaccounting for irq handlers....

Signature · 3 lines
Arjan van de Ven 	Intel Open Source Technology Centre
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org
#2 lore raw
Quoted 27 lines · Arjan van de Ven
Quoted 14 lines

Hello,

I have a dual-CPU router/firewall with five gigabit NICs. Recently I have found that irqbalance (0.55 from Fedora 9/x86_64) gives a suboptimal IRQ to CPU mapping on this box:

During traffic spikes, it assings two NICs to one CPU, and the
other three to the second CPU. However, this does not account for the fact that packets coming from the uplink interface are way more expensive to handle than the rest of the traffic: most iptables rules apply to the packets received from the uplink interface. The result is that the CPU which receives IRQs for the uplink interface is 100 % busy (softirq mostly), while the other one is 90% idle.

Quoted 6 lines

Setting the IRQ mapping by hand (uplink to one CPU, all the
other NICs to the other CPU) makes a well balanced system (both CPUs 30-60 % busy). I am not sure whether my configuration is too special, but it might be worth trying to make irqbalance daemon cope also with this usage pattern.

one of the hard cases for irqbalance is that irqbalance doesn't have a way to find out the actual cpu time spend in the handlers. For networking it makes an estimate just based on the number of packets (which is better than nothing)... but that breaks down if you have an non-symmetry in CPU costs per packet like you have.

Quoted 5 lines

The good news is that irqthreads at least have the potential to solve this "lack of information"; if not, we could consider doing a form of microaccounting for irq handlers....

perhaps, this could be addressed using tracepoints. The currently proposed ones are at the beginning and end of 'handle_IRQ_event()'. See:
http://marc.info/?l=linux-kernel&m=121616099830280&w=2

thanks,

-Jason

#3 lore raw
Quoted 12 lines · Jason Baron
Quoted 11 lines

one of the hard cases for irqbalance is that irqbalance doesn't have a way to find out the actual cpu time spend in the handlers. For networking it makes an estimate just based on the number of packets (which is better than nothing)... but that breaks down if you have an non-symmetry in CPU costs per packet like you have.

The good news is that irqthreads at least have the potential to solve this "lack of information"; if not, we could consider doing a form of microaccounting for irq handlers....

Quoted 4 lines

perhaps, this could be addressed using tracepoints. The currently proposed ones are at the beginning and end of 'handle_IRQ_event()'. See:
http://marc.info/?l=linux-kernel&m=121616099830280&w=2

something that you always need should not be a tracepoint.

#4 lore raw
In reply to Arjan van de Ven

Arjan van de Ven wrote:
: Jan Kasprzak <[email protected]> wrote:
: > The result is
: > that the CPU which receives IRQs for the uplink interface : > is 100 % busy (softirq mostly), while the other one is 90% idle. :
: one of the hard cases for irqbalance is that irqbalance doesn't have a : way to find out the actual cpu time spend in the handlers. For : networking it makes an estimate just based on the number of packets : (which is better than nothing)... but that breaks down if you have an : non-symmetry in CPU costs per packet like you have.
:
: The good news is that irqthreads at least have the potential to solve : this "lack of information"; if not, we could consider doing a form of : microaccounting for irq handlers....

I am not sure whether this would help. In my case, the most of the
in-kernel CPU time is not spend in the irq handler per se, but in softirq (i.e. checking the packet against iptables rules).

-Yenya

Signature · 3 lines
| Jan "Yenya" Kasprzak  <kas at {fi.muni.cz - work | yenya.net - private}> |
| GPG: ID 1024/D3498839      Fingerprint 0D99A7FB206605D7 8B35FCDE05B18A5E |
| http://www.fi.muni.cz/~kas/    Journal: http://www.fi.muni.cz/~kas/blog/ |
Quoted 2 lines
Quoted 2 lines

If you find yourself arguing with Alan Cox, you’re _probably_ wrong. <<
--James Morris in "How and Why You Should Become a Kernel Hacker" <<

#5 lore raw
Quoted 14 lines · Jan Kasprzak

Arjan van de Ven wrote:
: Jan Kasprzak <[email protected]> wrote:
: > The result is
: > that the CPU which receives IRQs for the uplink interface : > is 100 % busy (softirq mostly), while the other one is 90% idle. : : one of the hard cases for irqbalance is that irqbalance doesn't have a : way to find out the actual cpu time spend in the handlers. For : networking it makes an estimate just based on the number of packets : (which is better than nothing)... but that breaks down if you have an : non-symmetry in CPU costs per packet like you have.
: : The good news is that irqthreads at least have the potential to solve : this "lack of information"; if not, we could consider doing a form of : microaccounting for irq handlers....

Quoted 3 lines

I am not sure whether this would help. In my case, the most of the
in-kernel CPU time is not spend in the irq handler per se, but in softirq (i.e. checking the packet against iptables rules).

there is some consideration of making softirqs that are raised run as part of the irq thread. or at least thoughts in that direction.

#6 lore raw
In reply to Arjan van de Ven
Quoted 3 lines · Arjan van de Ven

The good news is that irqthreads at least have the potential to solve this "lack of information"; if not, we could consider doing a form of microaccounting for irq handlers....

I have some patches floating about that account for nmi/irq/softirq time in a fine grained scale. The trouble is that i've so far not found a way to handle the case of a jiffie based sched_clock().

The trouble with that is that time always increases in IRQ context, so nmi=0, softirq=0, regular=0 but irq=100%, which is obviuosly sub-optimal :-)

#7 lore raw
Quoted 8 lines · Peter Zijlstra
Quoted 3 lines · Arjan van de Ven

The good news is that irqthreads at least have the potential to solve this "lack of information"; if not, we could consider doing a form of microaccounting for irq handlers....

I have some patches floating about that account for nmi/irq/softirq time in a fine grained scale. The trouble is that i've so far not found a way to handle the case of a jiffie based sched_clock().

Would be nice to have.

Quoted 3 lines

The trouble with that is that time always increases in IRQ context, so nmi=0, softirq=0, regular=0 but irq=100%, which is obviuosly sub-optimal :-)

Can't you exempt the timer interrupt from counting itself as irq in the case of timer interrupt based sched_clock()?