How fast can your 10G go?
Subsystems (2)
I've been running some tests on a new Nehalem based system with a 2 port pci-e x8 10G NIC (ixgbe driver).
When using pktgen, max I can get is about 5.6Gbps tx + rx on both ports. This is about 22Gbps across the backplane, so I don't mean to complain :)
However, I'm curious if anyone has gotten any better performance on some other system? In particular, it seems that my system is bound by the bus and/or the NIC. Would I need to find something like a x16 slot to have a chance at 10Gbps bi-directional on 2 ports?
Thanks,
Ben
Signature · 2 lines
Ben Greear <[email protected]> Candela Technologies Inc http://www.candelatech.com
Quoted 10 lines · Ben Greear
I've been running some tests on a new Nehalem based system with a 2 port pci-e x8 10G NIC (ixgbe driver).
When using pktgen, max I can get is about 5.6Gbps tx + rx on both ports. This is about 22Gbps across the backplane, so I don't mean to complain :)
However, I'm curious if anyone has gotten any better performance on some other system? In particular, it seems that my system is bound by the bus and/or the NIC. Would I need to find something like a x16 slot to have a chance at 10Gbps bi-directional on 2 ports?
We have achieved bidirectional 20-Gbps line rate traffic between a pair of Nehalem i7 quad-core servers, each with a dual-port 10-GigE Myricom PCI-E 2.0 x8 SFP+ NIC. Here's a unidirectional 10-second nuttcp TCP test (using 9000 byte jumbo frames):
[root@i7raid-1 ~]# ./nuttcp-6.2.6 -N2 -w10m /192.168.101.2/192.168.102.2 23627.0625 MB / 10.01 sec = 19792.4891 Mbps 53 %TX 57 %RX 0.11 msRTT
Your results sound about right for a PCI-E 1.0 x8 slot, which has 16 Gbps full-duplex usable bandwidth, and then subtracting about 25% PCI-E overhead leaves about 12 Gbps full-duplex or 24 Gbps total available bandwidth for TCP|UDP data transfers.
-Bill
Quoted 7 lines · Ben Greear
I've been running some tests on a new Nehalem based system with a 2 port pci-e x8 10G NIC (ixgbe driver).
When using pktgen, max I can get is about 5.6Gbps tx + rx on both ports. This is about 22Gbps across the backplane, so I don't mean to complain :)
However, I'm curious if anyone has gotten any better performance on some other system?
Robert Olsson (and Olof Hagsand and Bengt Gördén) got some better results using that exact hardware Intel 82598 chips (and Sun neptune NICs). They made an article called: "Open-source routing at 10Gb/s"
https://www.iis.se/docs/10G-OS-router_2_.pdf
Quoted 3 lines
In particular, it seems that my system is bound by
the bus and/or the NIC. Would I need to find something like a x16 slot to have a chance at 10Gbps bi-directional on 2 ports?
Are you doing a single flow bandwidth test?
These NICs designed for multiflow performance. They have hardware RX and TX queues to facilitate this, often call multiqueue NICs. This is also what helps us do real parallel processing across CPUs in the network stack (since DaveMs multiqueue changes, although not all drivers uses this correctly yet...).
Try doing a multi-flow network test, I bet you will see better results. (Normally you also need to adjust smp-affinity, but with large frames on my Core i7 system its fast enough for 9.5 Gbit/s without tuning, small frames is another case)
Cheers,
Jesper Brouer
--
-------------------------------------------------------------------
MSc. Master of Computer Science
Dept. of Computer Science, University of Copenhagen Author of http://www.adsl-optimizer.dk
-------------------------------------------------------------------