bnx2 rxhash
Subsystems (2)
Hi list,
I'm doing some experiments with rxhash and packet capturing, and I'm seeing some unexpected results when receiving data (which happens in promiscuous mode). This behaviour seems to be similar to that seen in the question asked previously by Christophe Ngo Van Duc:
http://kerneltrap.com/mailarchive/linux-netdev/2010/7/2/6280474
The data I'm processing consists of (almost 100% tcp) traffic between an ssl-offloader and a cluster of webservers, so the range of mac-addresses is rather limited. This ssl-offloader does preserve the IP-address and tcp ports of the clients, so if the rxhash is based on that data, it should be distributed evenly.
Is there anything I can do about this?
Thanks,
Jasper
The raw data:
spaans@dhcp-074:~$ uname -a Linux dhcp-074 3.0.4 #7 SMP Tue Sep 13 12:52:01 CEST 2011 x86_64 GNU/Linux
spaans@dhcp-074:~$ grep eth6 /proc/interrupts | cut -b -30,270- 142: 57070 0 PCI-MSI-edge eth6-0 143: 0 0 PCI-MSI-edge eth6-1 144: 0 0 PCI-MSI-edge eth6-2 145: 0 0 PCI-MSI-edge eth6-3 146: 0 0 PCI-MSI-edge eth6-4 147: 17127 0 PCI-MSI-edge eth6-5 148: 24 0 PCI-MSI-edge eth6-6 149: 0 0 PCI-MSI-edge eth6-7
This is rather different from the behaviour I get with an intel card:
spaans@dhcp-074:~$ grep eth2 /proc/interrupts | cut -b -30,270- 98: 3 0 PCI-MSI-edge eth2 99: 4492 0 PCI-MSI-edge eth2-TxRx-0 100: 4647 0 PCI-MSI-edge eth2-TxRx-1 101: 4369 0 PCI-MSI-edge eth2-TxRx-2 102: 4579 0 PCI-MSI-edge eth2-TxRx-3 103: 4184 0 PCI-MSI-edge eth2-TxRx-4 104: 51268 0 PCI-MSI-edge eth2-TxRx-5 105: 4610 0 PCI-MSI-edge eth2-TxRx-6 106: 4528 0 PCI-MSI-edge eth2-TxRx-7
In both cases, only Tx-5 was used for sending data.
Information about the nics:
from dmesg:
[ 5.857778] igb 0000:08:00.0: Intel(R) Gigabit Ethernet Network Connection [ 5.857780] igb 0000:08:00.0: eth6: (PCIe:2.5Gb/s:Width x4) 00:1b:21:b9:b8:dc [ 5.858101] igb 0000:08:00.0: eth6: PBA No: G18771-002 [ 5.858102] igb 0000:08:00.0: Using MSI-X interrupts. 8 rx queue(s),
8 tx queue(s)
note that this interface is renamed to eth2 after boot
spaans@dhcp-074:~$ ip link sh eth2
8: eth2: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc mq
state UP qlen 1000
link/ether 00:1b:21:b9:b8:dc brd ff:ff:ff:ff:ff:ffspaans@dhcp-074:~$ sudo ethtool -k eth2 Offload parameters for eth2: rx-checksumming: off tx-checksumming: off scatter-gather: off tcp-segmentation-offload: off udp-fragmentation-offload: off generic-segmentation-offload: on generic-receive-offload: off large-receive-offload: off ntuple-filters: off receive-hashing: off spaans@dhcp-074:~$ sudo ethtool -i eth2 driver: igb version: 3.0.6-k2 firmware-version: 1.5-1 bus-info: 0000:08:00.0
vs
from dmesg:
[ 5.090860] bnx2 0000:02:00.0: eth2: Broadcom NetXtreme II BCM5709
1000Base-T (C0) PCI Express found at mem d8000000, IRQ 32, node addr 18:03:73:fa:0f:0e
note that this interface is renamed to eth6 after boot
spaans@dhcp-074:~$ ip link sh eth6
4: eth6: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc mq
state UP qlen 1000
link/ether 18:03:73:fa:0f:0e brd ff:ff:ff:ff:ff:ffspaans@dhcp-074:~$ sudo ethtool -k eth6 Offload parameters for eth6: rx-checksumming: off tx-checksumming: off scatter-gather: off tcp-segmentation-offload: off udp-fragmentation-offload: off generic-segmentation-offload: off generic-receive-offload: off large-receive-offload: off ntuple-filters: off receive-hashing: on spaans@dhcp-074:~$ sudo ethtool -i eth6 driver: bnx2 version: 2.1.6 firmware-version: 6.2.12 bc 5.2.3 NCSI 2.0.11 bus-info: 0000:02:00.0
Signature · 5 lines
/\____/\ Ir. Jasper Spaans
\ (_)/ Fox-IT Experts in IT Security!
\ X T: +31-15-2847999
\ / \ M: +31-6-41588725
\/ KvK Haaglanden 27301624Quoted 8 lines · Jasper Spaans
The data I'm processing consists of (almost 100% tcp) traffic between an ssl-offloader and a cluster of webservers, so the range of mac-addresses is rather limited. This ssl-offloader does preserve the IP-address and tcp ports of the clients, so if the rxhash is based on that data, it should be distributed evenly.
Is there anything I can do about this?
Can you send me a small tcpdump/wireshark trace of the incoming packets? I just need to see the tuple values in the packet headers. Thanks.