Skip to contentSkip to conversation

BUG: sky2: hw csum failure with dual-port copper NIC on SMP

DISCUSSION Tony Battersby 2 participants 5 messages Nov 13, 2007 – Nov 14, 2007

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

I am getting "hw csum failure" messages with sky2. I have seen this problem reported elsewhere with a fibre NIC, but I am using a copper NIC. It seems to be triggered by SMP. It is easy to reproduce in 2.6.23. 2.6.24-rc2-git3 still has the problem, but it happens less frequently.

To reproduce the problem, I am using a simple network benchmark program that I wrote that basically does send()/recv() as fast as possible using a memory buffer (null data, no disk I/O, no data integrity checking). The computer with the SysKonnect NIC acts as the server. I have two other computers with Intel PRO/1000 NICs that are directly cabled to the two ports on the SysKonnect NIC. Each of them runs the client program, which connects to the server, send()s 10 GB, and then recv()s 10 GB. Essentially, both ports on the Syskonnect NIC are receiving at the maximum rate for a few minutes, and then transmitting at the maximum rate for a few minutes. Sustained throughput is about 117 MB/s on both ports simultaneously.

The "hw csum failure" does not seem to affect the test. send()/recv() continue to work normally. Nothing locks up.

I get several "hw csum failure" messages per minute on 2.6.23-SMP. The error does not happen with 2.6.23 if I boot with "max_cpus=1". The message seems less frequent with 2.6.24-SMP, but it still happens once every minute or so.

The "hw csum failure" message does not happen when only one port is in use. You have to stress both ports simultaneously to reproduce the problem.

Another cosmetic issue is that "ifconfig" shows eth2 at IRQ 16 and eth3 at IRQ 218, when in fact both are at IRQ 218. IRQ 16 is the regular interrupt line and IRQ 218 is the MSI interrupt. I imagine that the driver is just reporting the IRQ incorrectly in this case. It is just a minor cosmetic issue which doesn't break anything.

Let me know if I can be of any further assistance in tracking down this problem.

NIC: Syskonnect SK-9E22 dual-port copper PCI-express
motherboard: SuperMicro PDSME
CPU: Pentium D 945 (dual-core 3.4 GHz) kernel versions: 2.6.23 and 2.6.24-rc2-git3

All information below is from 2.6.24-rc2-git3.

portion of dmesg showing error:
<unknown>: hw csum failure.

 [<c02c0910>] skb_copy_and_csum_datagram_iovec+0x120/0x130
 [<c0180913>] __set_page_dirty+0x83/0x140
 [<c02ef2c1>] tcp_rcv_established+0x981/0x9a0
 [<c02f6490>] tcp_v4_do_rcv+0xc0/0x370
 [<c02ba042>] release_sock+0x12/0xa0
 [<c02bb0f1>] sk_wait_data+0xa1/0xd0
 [<c02e3ef8>] tcp_prequeue_process+0x48/0x70
 [<c02e4ea1>] tcp_recvmsg+0x671/0xc50
 [<c0117bc3>] enqueue_task_fair+0x73/0xb0
 [<c02ba305>] sock_common_recvmsg+0x45/0x70
 [<c02b98d8>] sock_recvmsg+0xd8/0x130
 [<c012eef0>] autoremove_wake_function+0x0/0x50
 [<c0120d62>] __do_softirq+0x82/0x100
 [<c0120f12>] irq_exit+0x52/0x90
 [<c010f6b4>] smp_apic_timer_interrupt+0x54/0x80
 [<c02b9c6b>] sys_recvfrom+0xeb/0x180
 [<c0111cea>] read_hpet+0xa/0x10
 [<c01347f0>] getnstimeofday+0x40/0xf0
 [<c0118c20>] rebalance_domains+0x110/0x3e0
 [<c02b9d33>] sys_recv+0x33/0x40
 [<c02b9ea5>] sys_socketcall+0x165/0x280
 [<c0102a4e>] sysenter_past_esp+0x5f/0x85
 =======================

dmesg | grep sky2
sky2 0000:04:00.0: v1.20 addr 0xea300000 irq 16 Yukon-XL (0xb3) rev 1
sky2 0000:04:00.0: PCI Express Advanced Error Reporting not configured or MMCONFIG problem? sky2 eth2: addr 00:00:5a:72:b8:91
sky2 eth3: addr 00:00:5a:72:b8:92
sky2 eth2: enabling interface
sky2 eth3: enabling interface
sky2 eth2: Link is up at 1000 Mbps, full duplex, flow control both
sky2 eth3: Link is up at 1000 Mbps, full duplex, flow control both

ifconfig
eth2 Link encap:Ethernet HWaddr 00:00:5A:72:B8:91

          inet addr:192.168.1.10  Bcast:192.168.1.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:34910877 errors:0 dropped:0 overruns:0 frame:0
          TX packets:22659597 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:3207874526 (2.9 GiB)  TX bytes:2888042042 (2.6 GiB)
          Interrupt:16 

eth3 Link encap:Ethernet HWaddr 00:00:5A:72:B8:92

          inet addr:137.157.10.224  Bcast:137.157.255.255  Mask:255.255.0.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:34902414 errors:0 dropped:0 overruns:0 frame:0
          TX packets:22641940 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:3207442696 (2.9 GiB)  TX bytes:2886952355 (2.6 GiB)
          Interrupt:218 

ethtool -i eth2
driver: sky2
version: 1.20
firmware-version: N/A
bus-info: 0000:04:00.0

ethtool eth2
Settings for eth2:
Supported ports: [ TP ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full 1000baseT/Half 1000baseT/Full
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full 1000baseT/Half 1000baseT/Full
Advertised auto-negotiation: Yes
Speed: 1000Mb/s
Duplex: Full
Port: Twisted Pair
PHYAD: 0
Transceiver: internal
Auto-negotiation: on
Supports Wake-on: pg
Wake-on: d
Current message level: 0x000000ff (255) Link detected: yes

ethtool -k eth2
Offload parameters for eth2:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp segmentation offload: on

ethtool -S eth2
NIC statistics:

     tx_bytes: 33946810766
     rx_bytes: 33901041384
     tx_broadcast: 0
     rx_broadcast: 1
     tx_multicast: 0
     rx_multicast: 0
     tx_unicast: 35564726
     rx_unicast: 34910876
     tx_mac_pause: 0
     rx_mac_pause: 0
     collisions: 0
     late_collision: 0
     aborted: 0
     single_collisions: 0
     multi_collisions: 0
     rx_short: 0
     rx_runt: 0
     rx_64_byte_packets: 13
     rx_65_to_127_byte_packets: 13166182
     rx_128_to_255_byte_packets: 5
     rx_256_to_511_byte_packets: 6049
     rx_512_to_1023_byte_packets: 23940
     rx_1024_to_1518_byte_packets: 21714688
     rx_1518_to_max_byte_packets: 0
     rx_too_long: 0
     rx_fifo_overflow: 0
     rx_jabber: 0
     rx_fcs_error: 0
     tx_64_byte_packets: 13
     tx_65_to_127_byte_packets: 10811129
     tx_128_to_255_byte_packets: 873915
     tx_256_to_511_byte_packets: 955169
     tx_512_to_1023_byte_packets: 2245568
     tx_1024_to_1518_byte_packets: 20678932
     tx_1519_to_max_byte_packets: 0
     tx_fifo_underrun: 0

ethtool -i eth3
driver: sky2
version: 1.20
firmware-version: N/A
bus-info: 0000:04:00.0

ethtool eth3
Settings for eth3:
Supported ports: [ TP ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full 1000baseT/Half 1000baseT/Full
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full 1000baseT/Half 1000baseT/Full
Advertised auto-negotiation: Yes
Speed: 1000Mb/s
Duplex: Full
Port: Twisted Pair
PHYAD: 0
Transceiver: internal
Auto-negotiation: on
Supports Wake-on: pg
Wake-on: d
Current message level: 0x000000ff (255) Link detected: yes

ethtool -k eth3
Offload parameters for eth3:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp segmentation offload: on

ethtool -S eth3
NIC statistics:

     tx_bytes: 33948750825
     rx_bytes: 33900457220
     tx_broadcast: 0
     rx_broadcast: 137
     tx_multicast: 0
     rx_multicast: 0
     tx_unicast: 35591358
     rx_unicast: 34902277
     tx_mac_pause: 0
     rx_mac_pause: 0
     collisions: 31
     late_collision: 0
     aborted: 0
     single_collisions: 29
     multi_collisions: 1
     rx_short: 0
     rx_runt: 0
     rx_64_byte_packets: 64
     rx_65_to_127_byte_packets: 13151060
     rx_128_to_255_byte_packets: 23
     rx_256_to_511_byte_packets: 7867
     rx_512_to_1023_byte_packets: 36713
     rx_1024_to_1518_byte_packets: 21706687
     rx_1518_to_max_byte_packets: 0
     rx_too_long: 0
     rx_fifo_overflow: 0
     rx_jabber: 0
     rx_fcs_error: 0
     tx_64_byte_packets: 21
     tx_65_to_127_byte_packets: 10750614
     tx_128_to_255_byte_packets: 945463
     tx_256_to_511_byte_packets: 1004551
     tx_512_to_1023_byte_packets: 2153163
     tx_1024_to_1518_byte_packets: 20737546
     tx_1519_to_max_byte_packets: 0
     tx_fifo_underrun: 0

cat /proc/interrupts
CPU0 CPU1

  0:         89          0   IO-APIC-edge      timer
  1:        207          0   IO-APIC-edge      i8042
  7:          0          0   IO-APIC-edge      parport0
  8:          3          0   IO-APIC-edge      rtc
  9:          0          0   IO-APIC-fasteoi   acpi
 12:          5          0   IO-APIC-edge      i8042
 14:        784          0   IO-APIC-edge      ide0
 16:          0          0   IO-APIC-fasteoi   uhci_hcd:usb5
 18:          0          0   IO-APIC-fasteoi   uhci_hcd:usb4
 19:          0          0   IO-APIC-fasteoi   uhci_hcd:usb3
 20:          0          0   IO-APIC-fasteoi   ehci_hcd:usb1, uhci_hcd:usb2
218:    4482759    4446537   PCI-MSI-edge      eth2
219:          0          0   PCI-MSI-edge      ahci
NMI:          0          0   Non-maskable interrupts
LOC:      65542      48825   Local timer interrupts
RES:        226         59   Rescheduling interrupts
CAL:         80         60   function call interrupts
TLB:         22         52   TLB shootdowns
TRM:          0          0   Thermal event interrupts
SPU:          0          0   Spurious interrupts
ERR:          0
MIS:          0

lspci -vv
04:00.0 0200: 1148:9e00 (rev 14)
04:00.0 Ethernet controller: SysKonnect SK-9Exx 10/100/1000Base-T Adapter (rev 14)
Subsystem: SysKonnect SK-9E22 Server Adapter
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 0, Cache Line Size 08
Interrupt: pin A routed to IRQ 218
Region 0: Memory at ea300000 (64-bit, non-prefetchable) [size=16K]
Region 2: I/O ports at 8000 [size=256]
[virtual] Expansion ROM at ea320000 [disabled] [size=128K]
Capabilities: [48] Power Management version 2
Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=375mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
Status: D0 PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [50] Vital Product Data
Capabilities: [5c] Message Signalled Interrupts: 64bit+ Queue=0/1 Enable+
Address: 00000000fee0200c Data: 413a
Capabilities: [e0] Express Legacy Endpoint IRQ 0
Device: Supported: MaxPayload 128 bytes, PhantFunc 0, ExtTag-
Device: Latency L0s unlimited, L1 unlimited
Device: AtnBtn- AtnInd- PwrInd-
Device: Errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+
Device: RlxdOrd- ExtTag- PhantFunc- AuxPwr+ NoSnoop-
Device: MaxPayload 128 bytes, MaxReadReq 512 bytes
Link: Supported Speed 2.5Gb/s, Width x4, ASPM L0s, Port 0
Link: Latency L0s <256ns, L1 unlimited
Link: ASPM Disabled RCB 128 bytes CommClk- ExtSynch-
Link: Speed 2.5Gb/s, Width x4

#1 lore raw
Quoted 30 lines · On Tue, 13 Nov 2007 12:51:33 -0500 Tony Battersby <[email protected]> wrote:

I am getting "hw csum failure" messages with sky2. I have seen this problem reported elsewhere with a fibre NIC, but I am using a copper NIC. It seems to be triggered by SMP. It is easy to reproduce in 2.6.23. 2.6.24-rc2-git3 still has the problem, but it happens less frequently.

To reproduce the problem, I am using a simple network benchmark program that I wrote that basically does send()/recv() as fast as possible using a memory buffer (null data, no disk I/O, no data integrity checking). The computer with the SysKonnect NIC acts as the server. I have two other computers with Intel PRO/1000 NICs that are directly cabled to the two ports on the SysKonnect NIC. Each of them runs the client program, which connects to the server, send()s 10 GB, and then recv()s 10 GB. Essentially, both ports on the Syskonnect NIC are receiving at the maximum rate for a few minutes, and then transmitting at the maximum rate for a few minutes. Sustained throughput is about 117 MB/s on both ports simultaneously.

The "hw csum failure" does not seem to affect the test. send()/recv() continue to work normally. Nothing locks up.

I get several "hw csum failure" messages per minute on 2.6.23-SMP. The error does not happen with 2.6.23 if I boot with "max_cpus=1". The message seems less frequent with 2.6.24-SMP, but it still happens once every minute or so.

The "hw csum failure" message does not happen when only one port is in use. You have to stress both ports simultaneously to reproduce the problem.

Quoted 5 lines

Another cosmetic issue is that "ifconfig" shows eth2 at IRQ 16 and eth3 at IRQ 218, when in fact both are at IRQ 218. IRQ 16 is the regular interrupt line and IRQ 218 is the MSI interrupt. I imagine that the driver is just reporting the IRQ incorrectly in this case. It is just a minor cosmetic issue which doesn't break anything.

ifconfig reports value from dev->irq, which is a a legacy thing. When using MSI it gets it wrong.

I can reproduce the problem under load with only a single port on 2.6.23. I haven't been able to reproduce it on 2.6.24-rc2 (latest) but that maybe because of either insufficient stress or another bug fix correcting the problem. There is an issue with Yukon XL updating the receive status index before updating the receive status structure, that is now fixed in 2.6.24. The fix is:

commit ab5adecb2d02f3688719dfb5936a82833fcc3955

Author: Stephen Hemminger <[email protected]>
Date:   Mon Nov 5 15:52:09 2007 -0800

sky2: status ring race fix

The D-Link PCI-X board (and maybe others) can lie about status ring entries. It seems it will update the register for last status index before completing the DMA for the ring entry. To avoid reading stale data, zap the old entry and check.

Signature · 1 line
Stephen Hemminger <[email protected]>
#2 lore raw
Quoted 13 lines · Stephen Hemminger

I can reproduce the problem under load with only a single port on 2.6.23. I haven't been able to reproduce it on 2.6.24-rc2 (latest) but that maybe because of either insufficient stress or another bug fix correcting the problem. There is an issue with Yukon XL updating the receive status index before updating the receive status structure, that is now fixed in 2.6.24. The fix is:

commit ab5adecb2d02f3688719dfb5936a82833fcc3955

Author: Stephen Hemminger <[email protected]>
Date:   Mon Nov 5 15:52:09 2007 -0800

sky2: status ring race fix

Quoted 7 lines

The D-Link PCI-X board (and maybe others) can lie about status ring entries. It seems it will update the register for last status index before completing the DMA for the ring entry. To avoid reading stale data, zap the old entry and check.

The kernel I tested (2.6.24-rc2-git3) has this patch in it already. Perhaps that is why the problem happens less frequently with that kernel, but it didn't fix it entirely.

Do you want the test program I am using? It is a pretty basic send()/recv() program, ~650 lines of C.

Tony

#3 lore raw
Quoted 21 lines · On Tue, 13 Nov 2007 17:57:12 -0500 Tony Battersby <[email protected]> wrote:
Quoted 13 lines · Stephen Hemminger

I can reproduce the problem under load with only a single port on 2.6.23. I haven't been able to reproduce it on 2.6.24-rc2 (latest) but that maybe because of either insufficient stress or another bug fix correcting the problem. There is an issue with Yukon XL updating the receive status index before updating the receive status structure, that is now fixed in 2.6.24. The fix is:

commit ab5adecb2d02f3688719dfb5936a82833fcc3955

Author: Stephen Hemminger <[email protected]>
Date:   Mon Nov 5 15:52:09 2007 -0800

sky2: status ring race fix

Quoted 7 lines

The D-Link PCI-X board (and maybe others) can lie about status ring entries. It seems it will update the register for last status index before completing the DMA for the ring entry. To avoid reading stale data, zap the old entry and check.

Quoted 9 lines

The kernel I tested (2.6.24-rc2-git3) has this patch in it already. Perhaps that is why the problem happens less frequently with that kernel, but it didn't fix it entirely.

Do you want the test program I am using? It is a pretty basic send()/recv() program, ~650 lines of C.

Tony

Not really, iperf drives the problem fine.

Signature · 1 line
Stephen Hemminger <[email protected]>
#4 lore raw
Quoted 11 lines
Quoted 7 lines

Do you want the test program I am using? It is a pretty basic send()/recv() program, ~650 lines of C.

Tony

Not really, iperf drives the problem fine.

Thanks for the tip; I hadn't tried iperf before today. I can reproduce the problem with "iperf -s" on the system with the dual-port SysKonnect NIC and "iperf -c host -t 120" on the two Intel PRO/1000 (e1000 driver) client systems. The problem doesn't seem to happen the other way around though (running the server on the two Intel PRO/1000 systems and two iperf clients on the SysKonnect system). So the problem appears to be triggered by recv() on the SysKonnect side but not send().

Tony