s2io: packet reordering with 2.6.25.4
Subsystems (2)
We've been encountering some packet reordering with our Neterion 10 Gigabit Ethernet-SR cards using the s2io driver in linux 2.6.25.4.
We have a WAN testbed with the Neterion cards in two dual-processor AMD Opteron machines; when the machines are connected back-to-back we start to encounter reordering when sending UDP traffic at over 500 Mbit/s. (This is synthetic traffic generated with iperf, which reports how many datagrams are received out-of-order.)
We also see the problem when sending iperf TCP traffic from a machine with an e1000 gigabit card to the Neterion card via a Cisco 7609 router. It seems to be related to the burstiness of the traffic, as inserting an intermediate 2.5Gbit/s POS link seems to reduce the reordering, and inserting a gigabit ethernet link eliminates it.
We see reordering with 64-bit and 32-bit kernels, but possibly more with the 32-bit kernel.
Is this a known issue? Is there anything that I can do to debug it further? (I'm fairly familiar with the Linux TCP stack, but pretty much a newbie when it comes to debugging network device drivers.)
Thanks, and let me know if there's anything I can do to help debug, -Tom
Some possibly relevant information:
NAPI is enabled; LRO is disabled. The s2io module is loaded with default settings.
lspci, .config, and some other info are available here:
http://wil-ns.cs.caltech.edu/~quetchen/s2io-reordering/
After loading the s2io module, dmesg output is:
PM: Writing back config space on device 0000:03:01.0 at offset 1 (was 2300142, writing 2300146)
eth2: Enabling MSIX failed
eth2: MSI-X requested but failed to enable
Copyright(c) 2002-2007 Neterion Inc.
eth2: Neterion 10 Gigabit Ethernet-SR PCI-X 2.0 DDR Adapter (rev 2)
eth2: Driver version 2.0.26.22
eth2: MAC ADDR: 00:0c:fc:00:0d:22
SERIAL NUMBER: SXT0541048
eth2: Device is on 64 bit 133MHz PCIX(M1) bus
eth2: 1-Buffer receive mode enabled
eth2: Interrupt type INTA
eth2: Link down
Here's a sample of the output we see from iperf:
[quetchen@fast-2 ~]$ iperf -usi1 -w10M
------------------------------------------------------------
Server listening on UDP port 5001
Receiving 1470 byte datagrams
UDP buffer size: 9.54 MByte (WARNING: requested 10.0 MByte)
------------------------------------------------------------
[ 3] local 10.4.72.2 port 5001 connected with 10.4.72.3 port 42532 [ 3] 0.0- 1.0 sec 279 MBytes 2.34 Gbits/sec 0.004 ms 65/198856 [ 3] 0.0- 1.0 sec 65 datagrams received out-of-order [ 3] 1.0- 2.0 sec 278 MBytes 2.33 Gbits/sec 0.004 ms 60/198413 [ 3] 1.0- 2.0 sec 60 datagrams received out-of-order [ 3] 2.0- 3.0 sec 279 MBytes 2.34 Gbits/sec 0.004 ms 87/199063 [ 3] 2.0- 3.0 sec 87 datagrams received out-of-order [ 3] 3.0- 4.0 sec 279 MBytes 2.34 Gbits/sec 0.004 ms 50/198882 [ 3] 3.0- 4.0 sec 50 datagrams received out-of-order [ 3] 4.0- 5.0 sec 276 MBytes 2.32 Gbits/sec 0.004 ms 0/197179 [ 3] 5.0- 6.0 sec 270 MBytes 2.27 Gbits/sec 0.001 ms 127/192705 [ 3] 5.0- 6.0 sec 127 datagrams received out-of-order [ 3] 6.0- 7.0 sec 279 MBytes 2.34 Gbits/sec 0.001 ms 71/198909 [ 3] 6.0- 7.0 sec 71 datagrams received out-of-order [ 3] 7.0- 8.0 sec 277 MBytes 2.32 Gbits/sec 0.003 ms 18/197584 [ 3] 7.0- 8.0 sec 18 datagrams received out-of-order [ 3] 8.0- 9.0 sec 280 MBytes 2.35 Gbits/sec 0.015 ms 41/199648 [ 3] 8.0- 9.0 sec 41 datagrams received out-of-order [ 3] 0.0-10.0 sec 2.71 GBytes 2.33 Gbits/sec 0.002 ms 519/1979857 [ 3] 0.0-10.0 sec 520 datagrams received out-of-order
Tom,
We have tested udp traffic on other kernel versions and did not observe this problem. We will reproduce this problem in our lab and let you know.
Alternatively, you are welcome to download the driver on our website and try.
Ram
Quoted 8 lines
-----Original Message-----
From: Tom Quetchenbach [mailto:[email protected]]
Sent: Wednesday, June 25, 2008 5:11 PM
To: [email protected]
Cc: [email protected]; Rastapur Santosh; Sivakumar Subramani; Sreenivasa Honnur
Subject: s2io: packet reordering with 2.6.25.4
Quoted 4 lines
We've been encountering some packet reordering with our Neterion 10 Gigabit Ethernet-SR cards using the s2io driver in linux 2.6.25.4.
We have a WAN testbed with the Neterion cards in two dual-processor
AMD
Quoted 1 line
Opteron machines; when the machines are connected back-to-back we
start
Quoted 2 lines
to encounter reordering when sending UDP traffic at over 500 Mbit/s. (This is synthetic traffic generated with iperf, which reports how
many
Quoted 4 lines
datagrams are received out-of-order.)
We also see the problem when sending iperf TCP traffic from a machine with an e1000 gigabit card to the Neterion card via a Cisco 7609
router.
Quoted 1 line
It seems to be related to the burstiness of the traffic, as inserting
an
Quoted 4 lines
intermediate 2.5Gbit/s POS link seems to reduce the reordering, and inserting a gigabit ethernet link eliminates it.
We see reordering with 64-bit and 32-bit kernels, but possibly more
with
Quoted 4 lines
the 32-bit kernel.
Is this a known issue? Is there anything that I can do to debug it further? (I'm fairly familiar with the Linux TCP stack, but pretty
much
Quoted 8 lines
a newbie when it comes to debugging network device drivers.)
Thanks, and let me know if there's anything I can do to help debug, -Tom
Some possibly relevant information:
NAPI is enabled; LRO is disabled. The s2io module is loaded with
default
Quoted 20 lines
settings.
lspci, .config, and some other info are available here:
http://wil-ns.cs.caltech.edu/~quetchen/s2io-reordering/After loading the s2io module, dmesg output is:
PM: Writing back config space on device 0000:03:01.0 at offset 1 (was 2300142, writing 2300146)
eth2: Enabling MSIX failed
eth2: MSI-X requested but failed to enable
Copyright(c) 2002-2007 Neterion Inc.
eth2: Neterion 10 Gigabit Ethernet-SR PCI-X 2.0 DDR Adapter (rev 2)
eth2: Driver version 2.0.26.22
eth2: MAC ADDR: 00:0c:fc:00:0d:22
SERIAL NUMBER: SXT0541048
eth2: Device is on 64 bit 133MHz PCIX(M1) bus
eth2: 1-Buffer receive mode enabled
eth2: Interrupt type INTA
eth2: Link down
Quoted 9 lines
Here's a sample of the output we see from iperf:
[quetchen@fast-2 ~]$ iperf -usi1 -w10M
------------------------------------------------------------
Server listening on UDP port 5001
Receiving 1470 byte datagrams
UDP buffer size: 9.54 MByte (WARNING: requested 10.0 MByte)
------------------------------------------------------------
[ 3] local 10.4.72.2 port 5001 connected with 10.4.72.3 port 42532
[ 3] 0.0- 1.0 sec 279 MBytes 2.34 Gbits/sec 0.004 ms
65/198856
Quoted 2 lines
[ 3] 0.0- 1.0 sec 65 datagrams received out-of-order [ 3] 1.0- 2.0 sec 278 MBytes 2.33 Gbits/sec 0.004 ms
60/198413
Quoted 2 lines
[ 3] 1.0- 2.0 sec 60 datagrams received out-of-order [ 3] 2.0- 3.0 sec 279 MBytes 2.34 Gbits/sec 0.004 ms
87/199063
Quoted 2 lines
[ 3] 2.0- 3.0 sec 87 datagrams received out-of-order [ 3] 3.0- 4.0 sec 279 MBytes 2.34 Gbits/sec 0.004 ms
50/198882
Quoted 2 lines
[ 3] 3.0- 4.0 sec 50 datagrams received out-of-order [ 3] 4.0- 5.0 sec 276 MBytes 2.32 Gbits/sec 0.004 ms
0/197179
Quoted 1 line
[ 3] 5.0- 6.0 sec 270 MBytes 2.27 Gbits/sec 0.001 ms
127/192705
Quoted 2 lines
[ 3] 5.0- 6.0 sec 127 datagrams received out-of-order [ 3] 6.0- 7.0 sec 279 MBytes 2.34 Gbits/sec 0.001 ms
71/198909
Quoted 2 lines
[ 3] 6.0- 7.0 sec 71 datagrams received out-of-order [ 3] 7.0- 8.0 sec 277 MBytes 2.32 Gbits/sec 0.003 ms
18/197584
Quoted 2 lines
[ 3] 7.0- 8.0 sec 18 datagrams received out-of-order [ 3] 8.0- 9.0 sec 280 MBytes 2.35 Gbits/sec 0.015 ms
41/199648
Quoted 2 lines
[ 3] 8.0- 9.0 sec 41 datagrams received out-of-order [ 3] 0.0-10.0 sec 2.71 GBytes 2.33 Gbits/sec 0.002 ms
519/1979857
Quoted 1 line
[ 3] 0.0-10.0 sec 520 datagrams received out-of-order
David Miller
maintainerFrom: Tom Quetchenbach <[email protected]>
Date: Wed, 25 Jun 2008 17:11:16 -0700
Quoted 2 lines
Is this a known issue? Is there anything that I can do to debug it further?
Reordering can and will happen quite naturally on an SMP system under Linux.
Our TCP stack has heurstics which are able to successfully detect reordering and cope with it just fine.
For UDP based applications, they need to be able to properly handle reordering. Even in a completely controlled network configuration, reordering by external aspects of the network is still possible, so even if Linux never reordered packets, the UDP applications would still need to cope with this for correct operation.
Tom Quetchenbach wrote, On 06/26/2008 02:11 AM:
Quoted 2 lines
We've been encountering some packet reordering with our Neterion 10 Gigabit Ethernet-SR cards using the s2io driver in linux 2.6.25.4.
...
Quoted 1 line
Thanks, and let me know if there's anything I can do to help debug,
Out of curiosity: is there affinity set for eths' irqs or they are balanced between cpus? (/proc/interrupts?)
Regards,
Jarek P.
Jarek Poplawski wrote, On 06/26/2008 10:28 AM:
...
Quoted 2 lines
Out of curiosity: is there affinity set for eths' irqs or they are balanced between cpus? (
/proc/interrupts?)
...and maybe ifconfig on TX side?
Jarek P.
Quoted 3 lines · Jarek Poplawski
Quoted 3 lines
Out of curiosity: is there affinity set for eths' irqs or they are balanced between cpus? (
/proc/interrupts?)
No affinity set, but they're not exactly balanced either:
CPU0 CPU1
0: 73 0 IO-APIC-edge timer 1: 0 2 IO-APIC-edge i8042 2: 0 0 XT-PIC-XT cascade 8: 0 1 IO-APIC-edge rtc 12: 0 5 IO-APIC-edge i8042 25: 47 53074 IO-APIC-fasteoi eth0 26: 21 334 IO-APIC-fasteoi eth1 28: 358 1280440 IO-APIC-fasteoi eth2 Neterion 10 Gigabit Ethernet-SR PCI-X 2.0 DDR Adapter NMI: 0 0 Non-maskable interrupts LOC: 227793 227721 Local timer interrupts RES: 6483 3213 Rescheduling interrupts CAL: 40 31 function call interrupts TLB: 1119 105 TLB shootdowns TRM: 0 0 Thermal event interrupts THR: 0 0 Threshold APIC interrupts SPU: 0 0 Spurious interrupts ERR: 0
Anyhow, it looks like setting CPU affinity for the card on the receiver (echo 1 > /proc/irq/28/smp_affinity) does eliminate the reordering, at least with UDP at up to 2.5 Gbit/s (which is as fast as iperf can spit out packets on this machine). So, it looks like the "mystery" is solved :-)
Quoted 2 lines
...and maybe ifconfig on TX side?
eth2 Link encap:Ethernet HWaddr 00:0C:FC:00:0D:3F
inet addr:10.4.72.3 Bcast:10.4.72.15 Mask:255.255.255.240 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:7 errors:0 dropped:0 overruns:0 frame:0 TX packets:10 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000
RX bytes:496 (496.0 b) TX bytes:532 (532.0 b) Interrupt:28
Thanks
-Tom