tg3 driver and interrupt coalescence questions

I've been digging around trying to get some information on the current status of interrupt mitigation features for a Braodcom 5704 interface.

Specifically I'm sending and receiving lots of VoIP packets (50 pps per stream, many streams).

What I can't seem to determine is this:

What version of the linux kernel & tg3 drivers are required to support both rx and tx mitigation?
What do the ethtool coalescence settings actually do (I've not been able to find actual descriptions of the different parameters in the -C section) Is there anything special that needs to be done when compiling a kernel to enable this feature for both the kernel and the tg3 driver.

Just a warning, I'm not a C coder, so I've not had much luck digging around the code and looking for answers.

I've currently got a blade server with 10 blades I'm using 9 blades to generate this small packet high rate traffic to the 10th blade and trying to improve the ability of a blade to handle VoIP traffic. I made some guesses at settings for the -C options in ethtool on both the test blade and the traffic generators. Interestingly it seems to have had a very good effect on the test blade (%cpu for interrupt down from 99.9% to ~20%), but the same settings on the traffic generation servers seems to have had no effect.

Hardware is identical, kernel is identical.

Any help is GREATLY appreciated.

-Chris

05:01.1 Ethernet controller: Broadcom Corporation NetXtreme BCM5704S Gigabit Ethernet (rev 10)
Subsystem: IBM: Unknown device 02e8
Flags: bus master, 66Mhz, medium devsel, latency 64, IRQ 201 Memory at dcfe0000 (64-bit, non-prefetchable) [size=64K]
Capabilities: [40] PCI-X non-bridge device.
Capabilities: [48] Power Management version 2
Capabilities: [50] Vital Product Data
Capabilities: [58] Message Signalled Interrupts: 64bit+
Queue=0/3 Enable-

Linux version 2.6.9-34.ELsmp ([email protected]) (gcc version 3.4.5 20051201 (Red Hat 3.4.5-2)) #1 SMP Thu Mar 9 06:23:23 GMT 2006

[root@db2 ~]# ethtool -c eth1
Coalesce parameters for eth1:
Adaptive RX: off TX: off
stats-block-usecs: 1000000
sample-interval: 0
pkt-rate-low: 0
pkt-rate-high: 0

rx-usecs: 500
rx-frames: 30
rx-usecs-irq: 500
rx-frames-irq: 20

tx-usecs: 400
tx-frames: 53
tx-usecs-irq: 490
tx-frames-irq: 5

rx-usecs-low: 0
rx-frame-low: 0
tx-usecs-low: 0
tx-frame-low: 0

rx-usecs-high: 0
rx-frame-high: 0
tx-usecs-high: 0
tx-frame-high: 0

[root@db2 ~]# ethtool -i eth1
driver: tg3
version: 3.43-rh
firmware-version:
bus-info: 0000:05:01.1

[root@db2 ~]# ethtool eth1
Settings for eth1:
Supported ports: [ FIBRE ]
Supported link modes: 1000baseT/Half 1000baseT/Full
Supports auto-negotiation: Yes
Advertised link modes: 1000baseT/Half 1000baseT/Full Advertised auto-negotiation: Yes
Speed: 1000Mb/s
Duplex: Full
Port: Twisted Pair
PHYAD: 1
Transceiver: internal
Auto-negotiation: on
Supports Wake-on: g
Wake-on: d
Current message level: 0x000000ff (255) Link detected: yes

ethtool -C for tg3 was added around July of 2005. The version with this change added was 3.33.

They set the delay between the tx and rx events and the generation of interrupts for those events.

These are the only parameters that are relevant for tg3:

rx-frames[-irq]
rx-usecs[-irq]
tx-frames[-irq]
tx-usecs[-irq]

The frames parameters specify how many packets are received/transmitted before generating an interrupt. The usecs parameters specify how many microseconds after at least 1 packet is received/transmitted before generating an interrupt. The [-irq] parameters are the corresponding delays in updating the status when the interrupt is disabled.

No.

This means that the first interrupt will be generated after 30 packets are received or 500 microseconds after the nth packet is received (1 <= n < 30). When irq is disabled, 20 packets instead of 30 before updating status.

The first tx interrupt will be generated after 53 packets are transmitted or 400 microseconds after the nth packet is transmitted (1 <= n < 53). When irq is disabled, 5 packets or 490 micosecs before updating status.

If the condition for generating a tx or rx interrupt is met, you get all the accumulated tx and rx status during the interrupt.

Hope this helps.

Delay interrupts and increase individual packet latency with the intention being decreasing CPU utilization and allowing a higher aggregate packet per second limit. IE bandwidth vs latency tradeoffs.

Are you looking to increase or decrease the settings? I would think (initially at least) that for VOIP one might not want to increase them.

rick jones

I'm looking to decrease the interrupt load on the system. During the test I mentioned above I had some interesting and confusing results. The changes from the default settings to the settings I posted resulted in a 100% performance increase (counted by the number of VoIP audio streams the tested server could support). With default settings one of the two CPUs in the system maxed out at 99% cpu usage handling interrupts, while the second CPU was not maxed out, but we started to drop packets and the VoIP call setups started showing retransmits (which is the measurement for failure in this test) at about 300 streams. With the new settings we were able to hit 600 streams.

So I definately recognized a significant improvement. However I'd still like to get more improvement. At 600 streams and 20ms packets we are looking at 30,000 pps. The % of cpu (1 CPU as apparently the interrupts can't be shared across multiple CPUs) used for interrupt handling at this 600 stream limit was 88.0%.

Now what was interesting was on the test generation side (same hardware exactly) of things, I was using the SIPP software to generate the VoIP streams, and each blade in the blade server was only able to generate ~200 streams, with default settings in ethtool, one of the CPUs would hit max usage for interrupt handling at that point. So I modified the ethtool settings to match those I listed above and there was no discernable difference. It was identical performance to the default settings.

Michael's response clarified for me what the actual parameters in the -C section of ethtool do, thanks Michael. However I';; be greatly appreciative of any recommedations anyone might have for interrupt mitigation settings for 100% UDP RTP traffic of 20ms packets (50 pps per stream).

-Chris

Hi Cris,

interrupts can be balances across multiple CPUs or not.
It depends on 4 areas:
1. enabling/disabling such option in kernel upon compilation;
2. enabling/disabling of a user-space service for interrupt balancing, "irqbalance" on redhat, nothing such on debian;
3. enabling of disabling cpu affinity for an irq;

Normally, irq-affinity for a nic interrupt is considered good, but if a CPU is overloaded you may try irq balancing.

RTP streams generation can burn your CPU cycles as well as output of them to network, thus balancing of the
load among the CPUs, irqbalancing may improve something.

Sincerely,
------------------------------------------------------------------
Robert Iakobashvili, coroberti at gmail dot com
Navigare necesse est, vivere non est necesse.
------------------------------------------------------------------