i40e: XL710 Rx filters out frames above 1510 bytes

Hi, I have the most strange issue on a setup of 2 pairs of connected back to back XL710 cards.
It seems that frames above 1510 bytes are being filtered out by an i40e PMD receiver.
The same setup works perfectly when I use Linux drivers on both sides but when I use a PMD on one side and a Linux driver on the other - the issue above occurs.

i40e PMD statistics show nothing: all counters stay zero.
The MFS field in PRTGL_SAH register has the expected (reset) value: 0x2600.

I even tried to disable all Tx offload capabilities on the Linux side - still no success.

Could u, guys clarify to be what is going on and what I may be doing wrong? Pls., let me know if u need any additional info about the setup.

thanks in advance,
vlad

How to reproduce the issue:

I used the in tree apps from examples/multi_process/client_server_mp:

On the DPDK box:

1. Bound both NICs to DPDK's UIO.
2. Configured 1024 huge pages on each NUMA Node (there are two NUMA Nodes in total).

$ cat /proc/meminfo  | grep Huge
AnonHugePages:         0 kB
HugePages_Total:    2048
HugePages_Free:      292
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB

3. Compiled mp_server and mp_client example applications.
4. Run: sudo ./build/mp_server -c 6 -n 4 -- -p 0x3 -n 1
5. Run: sudo ./build/mp_client -c 8 -n 4 --proc-type=auto -- -n 0

On the "other side" box (with the Linux device drivers):

1. configure a static arp:

$ arp 192.169.10.118
Address                  HWtype  HWaddress           Flags Mask            Iface
192.169.10.118           ether   68:05:ca:2d:39:88 CM                    ens3

2. Set the MTU on ens3 to 3000 (see below).
3. Use *ping* application for sending frames of different sizes:
1. "ping 192.169.10.118 -s 1400" makes both Rx and Tx counters on the DPDK side increment.
2. "ping 192.169.10.118 -s 1500" - Rx counter on the DPDK side doesn't increment.

The DPDK PMD driven setup description:

* Two XL710 single port cards connected to XL710 cards in a different server back to back.

$ lspci | grep Ether
---snip---
05:00.0 Ethernet controller: Intel Corporation Ethernet Controller XL710 for 40GbE QSFP+ (rev 01)
83:00.0 Ethernet controller: Intel Corporation Ethernet Controller XL710 for 40GbE QSFP+ (rev 01)
---snip---

$ ethtool -i ens288
driver: i40e
version: 1.3.2-k
firmware-version: f4.22.26225 a1.1 n4.24 e13fd
bus-info: 0000:83:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

$ ethtool -i ens785
driver: i40e
version: 1.3.2-k
firmware-version: f4.22.26225 a1.1 n4.24 e13fd
bus-info: 0000:05:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

$ ifconfig
---snip---
ens288: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether 68:05:ca:2d:3a:00  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ens785: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether 68:05:ca:2d:39:88 txqueuelen 1000 (Ethernet) RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
---snip---

$ cat /etc/redhat-release
Fedora release 21 (Twenty One)

* DPDK git HEAD 7173acefc7cfdfbbb9b91fcba1c9a67adb4c07c9

The Linux drivers driven setup description:

* Two XL710 single port cards connected to XL710 cards in a different server back to back.

$ lspci | grep Ether
---snip---
04:00.0 Ethernet controller: Intel Corporation Ethernet Controller XL710 for 40GbE QSFP+ (rev 01)
83:00.0 Ethernet controller: Intel Corporation Ethernet Controller XL710 for 40GbE QSFP+ (rev 01)
---snip---

$ ethtool -i ens3
driver: i40e
version: 1.2.6-k
firmware-version: f4.1 a1.1 n04.10 e80001121
bus-info: 0000:83:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: no

$ ethtool -i ens5
driver: i40e
version: 1.2.6-k
firmware-version: f4.1 a1.1 n04.10 e80001121
bus-info: 0000:04:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: no

$ ifconfig
---snip---
ens3: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 3000
        inet 192.169.10.199  netmask 255.255.255.0  broadcast 
192.169.10.255
        ether 68:05:ca:27:d3:72  txqueuelen 1000  (Ethernet)
        RX packets 69  bytes 30155 (29.4 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 746  bytes 820489 (801.2 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ens5: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500

        inet 1.1.1.2  netmask 255.255.255.0  broadcast 1.1.1.255
        inet6 fe80::6a05:caff:fe27:d3ea  prefixlen 64  scopeid 0x20<link>
        ether 68:05:ca:27:d3:ea  txqueuelen 1000  (Ethernet)
        RX packets 523572634  bytes 34869102046 (32.4 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 523254659  bytes 153342981238 (142.8 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

---snip---

$ cat /etc/redhat-release
Fedora release 21 (Twenty One)_

_$ modinfo i40e

filename: /lib/modules/4.0.6-200.fc21.x86_64/kernel/drivers/net/ethernet/intel/i40e/i40e.ko.xz
version:        1.2.6-k
license:        GPL
description:    Intel(R) Ethernet Connection XL710 Network Driver
author:         Intel Corporation, <[email protected]>
srcversion:     14800421FE3354248473729
alias:          pci:v00008086d00001586sv*sd*bc*sc*i*
alias:          pci:v00008086d00001585sv*sd*bc*sc*i*
alias:          pci:v00008086d00001584sv*sd*bc*sc*i*
alias:          pci:v00008086d00001583sv*sd*bc*sc*i*
alias:          pci:v00008086d00001581sv*sd*bc*sc*i*
alias:          pci:v00008086d00001580sv*sd*bc*sc*i*
alias:          pci:v00008086d0000157Fsv*sd*bc*sc*i*
alias:          pci:v00008086d00001574sv*sd*bc*sc*i*
alias:          pci:v00008086d00001572sv*sd*bc*sc*i*
depends:        ptp,vxlan
intree:         Y
vermagic:       4.0.6-200.fc21.x86_64 SMP mod_unload
signer:         Fedora kernel signing key
sig_key: F9:FE:41:30:BC:4F:80:F6:C4:1E:A1:E1:5C:73:37:9B:B8:BA:C8:4C
sig_hashalgo:   sha256
parm:           debug:Debug level (0=none,...,16=all) (int)

_
_

We've found the problem. There is additional configuration per Rx queue context that limits the maximum
allowed frame size - RXMAX. And it is configured to be 1518 by default. Which is supposed to be MTU + L2 HDR + CRC = 1500 + 14 + 4. However,
when the MTU sized frames are being sent they are still being filtered out. This is probably due to some HW "feature" that requires RXMAX include VLAN header size even if there isn't any VLAN header. This "feature" has been addressed for instance in the Jesse's 61a46a4c0 commit in the net-next tree.

In addition there is some mysterious "GLV_REPC/GLVREPC counter of the VSI" that is supposed to count this and a few other Rx drops events but this counter is nowhere to be found - neither in the data sheet (its offset on the bar I mean), nor in the DPDK i40e PMD registers description, nor in the net-next Linux driver. DPDK has a "/* GLV_REPC not supported */" in the i40e_ethdev.c implying that there must be some problem with this register too and it's a pity.

I'll send a patch to the dpdk-dev that fixes the RXMAX issue shortly.

thanks,
vlad