Intel 82599 ixgbe driver performance

Hi, everyone

I'm testing our network card which includes intel 82599 based on ixgbe driver. I wonder what is the Rx performance of i82599 without network stack only with 64Byte frames. Our driver reads the packet directly from DMA packet buffer and push to the application without passing through linux kernel stack.
It seems that the intel 82599 cannot push 64B frames to DMA area in 10G.
Is it right?

If it is the case, what is the bottleneck of 82599?

Thanks in advance.

Best Regards,
J.Hwan KIm

My experience with 82599 is that it can RX 13.4 Mpps using 64byte frames and a single port.
When using both ports the rate drops to 10.7 Mpps per port.

Note that my experience does not involve the ixgbe driver, these numbers were obtained using a custom driver and OS, but should give some indication of what the hardware is capable of.

(If you want to see something "fun", try 65byte packets with 82599 and the Intel X58 IOH (or 5500/5520 server versions) :)

--
/Martin

Does your driver perform a copy of that 64B frame to user space?

Is this a single-threaded test running?

What does an lat_mem_rd -t (-t for random stride) test from lmbench give for your system's memory latency? (Perhaps using numactl to ensure local, or remote memory access, as you desire)

At line rate for minimum sized frames over 10 GbE, you have a frame arriving every 60-odd nanoseconds. At that speed, you cannot take even one cache miss per frame (*) in a single-threaded path and still achieve line-rate PPS.

As it happens, there was a presentation at HP Labs recently, given by Luigi Rizzo on his netmap work. The slides can be found at
http://info.iet.unipi.it/~luigi/netmap/talk-hp.html . As it happens, Luigi presented some performance figures using an Intel 82599.

happy benchmarking,

rick jones
(*) much of my time has been spent in a world where a cache miss is three digits worth of nanoseconds (to the left of the decimal), sometimes high two digits.

Unless the chip you are using has a rather tiny (by today's standards) data cache, you need to go much father there - I suspect that at 0.5 MB you have not yet gotten beyond the size of the last level of data cache on the chip.

I would suggest:

(from a system that is not otherwise idle...)

./lat_mem_rd -t 512 256"stride=256
0.00049 1.237
0.00098 1.239
0.00195 1.228
0.00293 1.238
0.00391 1.243
0.00586 1.238
0.00781 1.250
0.01172 1.249
0.01562 1.251
0.02344 1.247
0.03125 1.247
0.04688 3.125
0.06250 3.153
0.09375 3.158
0.12500 3.177
0.18750 6.636
0.25000 8.729
0.37500 16.167
0.50000 16.901
0.75000 16.953
1.00000 17.362
1.50000 18.781
2.00000 20.243
3.00000 23.434
4.00000 24.965
6.00000 35.951
8.00000 56.026
12.00000 76.169
16.00000 80.741
24.00000 83.237
32.00000 84.043
48.00000 84.132
64.00000 83.775
96.00000 83.298
128.00000 83.039
192.00000 82.659
256.00000 82.464
384.00000 82.280
512.00000 82.092

You can see the large jump starting at 8MB - that is where the last level cache runs-out on the chip I'm using - an Intel W3550.

Now, as run, that will include TLB miss overhead once the area of memory being accessed is larger than can be mapped by the chip's TLB at the page size being used. You can use libhugetlbfs to mitigate that through the use of hugepages.

rick jones