Skip to contentSkip to conversation

bnx2 rxhash

DISCUSSION Jasper Spaans 2 participants 2 messages Oct 6, 2011

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Hi list,

I'm doing some experiments with rxhash and packet capturing, and I'm seeing some unexpected results when receiving data (which happens in promiscuous mode). This behaviour seems to be similar to that seen in the question asked previously by Christophe Ngo Van Duc:
http://kerneltrap.com/mailarchive/linux-netdev/2010/7/2/6280474

The data I'm processing consists of (almost 100% tcp) traffic between an ssl-offloader and a cluster of webservers, so the range of mac-addresses is rather limited. This ssl-offloader does preserve the IP-address and tcp ports of the clients, so if the rxhash is based on that data, it should be distributed evenly.

Is there anything I can do about this?

Thanks,
Jasper

The raw data:

spaans@dhcp-074:~$ uname -a
Linux dhcp-074 3.0.4 #7 SMP Tue Sep 13 12:52:01 CEST 2011 x86_64 GNU/Linux
spaans@dhcp-074:~$ grep eth6 /proc/interrupts  | cut -b -30,270-
 142:      57070          0      PCI-MSI-edge      eth6-0
 143:          0          0      PCI-MSI-edge      eth6-1
 144:          0          0      PCI-MSI-edge      eth6-2
 145:          0          0      PCI-MSI-edge      eth6-3
 146:          0          0      PCI-MSI-edge      eth6-4
 147:      17127          0      PCI-MSI-edge      eth6-5
 148:         24          0      PCI-MSI-edge      eth6-6
 149:          0          0      PCI-MSI-edge      eth6-7

This is rather different from the behaviour I get with an intel card:

spaans@dhcp-074:~$ grep eth2 /proc/interrupts  | cut -b -30,270-
  98:          3          0      PCI-MSI-edge      eth2
  99:       4492          0      PCI-MSI-edge      eth2-TxRx-0
 100:       4647          0      PCI-MSI-edge      eth2-TxRx-1
 101:       4369          0      PCI-MSI-edge      eth2-TxRx-2
 102:       4579          0      PCI-MSI-edge      eth2-TxRx-3
 103:       4184          0      PCI-MSI-edge      eth2-TxRx-4
 104:      51268          0      PCI-MSI-edge      eth2-TxRx-5
 105:       4610          0      PCI-MSI-edge      eth2-TxRx-6
 106:       4528          0      PCI-MSI-edge      eth2-TxRx-7

In both cases, only Tx-5 was used for sending data.

Information about the nics:

from dmesg:

[    5.857778] igb 0000:08:00.0: Intel(R) Gigabit Ethernet Network
Connection
[    5.857780] igb 0000:08:00.0: eth6: (PCIe:2.5Gb/s:Width x4)
00:1b:21:b9:b8:dc
[    5.858101] igb 0000:08:00.0: eth6: PBA No: G18771-002
[    5.858102] igb 0000:08:00.0: Using MSI-X interrupts. 8 rx queue(s),

8 tx queue(s)
note that this interface is renamed to eth2 after boot

spaans@dhcp-074:~$ ip link sh eth2
8: eth2: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc mq
state UP qlen 1000
    link/ether 00:1b:21:b9:b8:dc brd ff:ff:ff:ff:ff:ff
spaans@dhcp-074:~$ sudo ethtool -k eth2
Offload parameters for eth2:
rx-checksumming: off
tx-checksumming: off
scatter-gather: off
tcp-segmentation-offload: off
udp-fragmentation-offload: off
generic-segmentation-offload: on
generic-receive-offload: off
large-receive-offload: off
ntuple-filters: off
receive-hashing: off
spaans@dhcp-074:~$ sudo ethtool -i eth2
driver: igb
version: 3.0.6-k2
firmware-version: 1.5-1
bus-info: 0000:08:00.0

vs

from dmesg:

[    5.090860] bnx2 0000:02:00.0: eth2: Broadcom NetXtreme II BCM5709

1000Base-T (C0) PCI Express found at mem d8000000, IRQ 32, node addr 18:03:73:fa:0f:0e
note that this interface is renamed to eth6 after boot

spaans@dhcp-074:~$ ip link sh eth6
4: eth6: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc mq
state UP qlen 1000
    link/ether 18:03:73:fa:0f:0e brd ff:ff:ff:ff:ff:ff
spaans@dhcp-074:~$ sudo ethtool -k eth6
Offload parameters for eth6:
rx-checksumming: off
tx-checksumming: off
scatter-gather: off
tcp-segmentation-offload: off
udp-fragmentation-offload: off
generic-segmentation-offload: off
generic-receive-offload: off
large-receive-offload: off
ntuple-filters: off
receive-hashing: on
spaans@dhcp-074:~$ sudo ethtool -i eth6
driver: bnx2
version: 2.1.6
firmware-version: 6.2.12 bc 5.2.3 NCSI 2.0.11
bus-info: 0000:02:00.0
Signature · 5 lines
 /\____/\   Ir. Jasper Spaans      
 \   (_)/   Fox-IT Experts in IT Security!
  \    X    T: +31-15-2847999
   \  / \   M: +31-6-41588725   
    \/      KvK Haaglanden 27301624
#1 lore raw
Quoted 8 lines · Jasper Spaans

The data I'm processing consists of (almost 100% tcp) traffic between an ssl-offloader and a cluster of webservers, so the range of mac-addresses is rather limited. This ssl-offloader does preserve the IP-address and tcp ports of the clients, so if the rxhash is based on that data, it should be distributed evenly.

Is there anything I can do about this?

Can you send me a small tcpdump/wireshark trace of the incoming packets? I just need to see the tuple values in the packet headers. Thanks.