Skip to contentSkip to conversation

forcedeth doesn't pass traffic (3.0.x kernel and earliers)

DISCUSSION Arkadiusz Miśkiewicz 2 participants 4 messages Dec 10, 2011 – Jan 13, 2012

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Hello,

I have few supermicro machines connected to edge-core switch using 1Gbps uplink. The problem is that after boot I have to restart net to get network card pass traffic. The problem is only with forcedeth network cards. It happens on different supermicro servers and it happens for as far as I can remember (so, 2.6.3x kernels, too). Currently I'm on 3.0.13.

Logs below show state after fresh boot where network traffic doesn't get through. Note that it shows that link is up but still - traffic doesn't get passed. tcpdump on eth0 interface sees only traffic initiated from this server and no traffic from outside.

Tried to set every offload (via ethtool -K) to off but that didn't help.

I have to make:
ip link set eth0 down; ip link set eth0 up
wait few second and then traffic starts to get passed. When I do that this shows in dmesg:

[ 1058.266480] forcedeth 0000:00:08.0: irq 40 for MSI/MSI-X
[ 1069.163238] eth0: no IPv6 routers present

(CC to few people shown by get_maintainer.pl)

LOGS/OUTPUTs from non working state after fresh boot. # dmesg|grep eth

[   34.528326] forcedeth: Reverse Engineered nForce ethernet driver. Version 0.64.
[   34.528542] forcedeth 0000:00:08.0: PCI INT A -> Link[LMAC] -> GSI 22 (level, low) -> IRQ 22
[   34.528547] forcedeth 0000:00:08.0: setting latency timer to 64
[   35.055319] forcedeth 0000:00:08.0: ifname eth0, PHY OUI 0x5043 @ 2, addr 00:30:48:7a:b4:88
[   35.055324] forcedeth 0000:00:08.0: highdma csum vlan pwrctl mgmt gbit lnktim msi desc-v3
[   35.056480] forcedeth 0000:00:09.0: PCI INT A -> Link[LMAD] -> GSI 21 (level, low) -> IRQ 21
[   35.056485] forcedeth 0000:00:09.0: setting latency timer to 64
[   35.581960] forcedeth 0000:00:09.0: ifname eth1, PHY OUI 0x5043 @ 3, addr 00:30:48:7a:b4:89
[   35.581965] forcedeth 0000:00:09.0: highdma csum vlan pwrctl mgmt gbit lnktim msi desc-v3
[   35.584211] Loading kernel module for a network device with CAP_SYS_MODULE (deprecated).  Use CAP_NET_ADMIN and alias netdev-eth0 instead
[   35.591778] forcedeth 0000:00:08.0: irq 40 for MSI/MSI-X
[   35.591970] forcedeth 0000:00:08.0: eth0: no link during initialization
[   35.593030] ADDRCONF(NETDEV_UP): eth0: link is not ready
[   37.666757] forcedeth 0000:00:08.0: eth0: link up
[   37.668624] ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[   48.150006] eth0: no IPv6 routers present

lspci -vv:
00:09.0 Bridge: nVidia Corporation MCP55 Ethernet (rev a3)

        Subsystem: Super Micro Computer Inc Device 1611
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0 (250ns min, 5000ns max)
        Interrupt: pin A routed to IRQ 21
        Region 0: Memory at feaf5000 (32-bit, non-prefetchable) [size=4K]
        Region 1: I/O ports at b080 [size=8]
        Region 2: Memory at feafa000 (32-bit, non-prefetchable) [size=256]
        Region 3: Memory at feaf4c00 (32-bit, non-prefetchable) [size=16]
        Capabilities: [44] Power Management version 2
                Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
                Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [70] MSI-X: Enable- Count=8 Masked-
                Vector table: BAR=2 offset=00000000
                PBA: BAR=3 offset=00000000
        Capabilities: [50] MSI: Enable- Count=1/8 Maskable+ 64bit+
                Address: 0000000000000000  Data: 0000
                Masking: 00000000  Pending: 00000000
        Capabilities: [6c] HyperTransport: MSI Mapping Enable- Fixed+
        Kernel driver in use: forcedeth
# ethtool eth0
Settings for eth0:
        Supported ports: [ MII ]
        Supported link modes:   10baseT/Half 10baseT/Full 
                                100baseT/Half 100baseT/Full 
                                1000baseT/Full 
        Supported pause frame use: No
        Supports auto-negotiation: Yes
        Advertised link modes:  10baseT/Half 10baseT/Full 
                                100baseT/Half 100baseT/Full 
                                1000baseT/Full 
        Advertised pause frame use: No
        Advertised auto-negotiation: Yes
        Speed: 1000Mb/s
        Duplex: Full
        Port: MII
        PHYAD: 2
        Transceiver: external
        Auto-negotiation: on
        Supports Wake-on: g
        Wake-on: d
        Link detected: yes
# ethtool -S eth0
NIC statistics:
     tx_bytes: 46352
     tx_zero_rexmt: 704
     tx_one_rexmt: 0
     tx_many_rexmt: 0
     tx_late_collision: 0
     tx_fifo_errors: 0
     tx_carrier_errors: 0
     tx_excess_deferral: 0
     tx_retry_error: 0
     rx_frame_error: 0
     rx_extra_byte: 0
     rx_late_collision: 0
     rx_runt: 0
     rx_frame_too_long: 0
     rx_over_errors: 1
     rx_crc_errors: 0
     rx_frame_align_error: 0
     rx_length_error: 0
     rx_unicast: 31
     rx_multicast: 0
     rx_broadcast: 436
     rx_packets: 467
     rx_errors_total: 1
     tx_errors_total: 0
     tx_deferral: 0
     tx_packets: 704
     rx_bytes: 35872
     tx_pause: 0
     rx_pause: 0
     rx_drop_frame: 170
# ethtool -k eth0
Offload parameters for eth0:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp-segmentation-offload: on
udp-fragmentation-offload: off
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off
receive-hashing: off

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc sfq state UP qlen 1000

    link/ether 00:30:48:7a:b4:88 brd ff:ff:ff:ff:ff:ff
Signature · 2 lines
Arkadiusz Miśkiewicz        PLD/Linux Team
arekm / maven.pl            http://ftp.pld-linux.org/
#1 lore raw
Quoted 13 lines · Arkadiusz Miśkiewicz

Hello,

I have few supermicro machines connected to edge-core switch using 1Gbps uplink. The problem is that after boot I have to restart net to get network card pass traffic. The problem is only with forcedeth network cards. It happens on different supermicro servers and it happens for as far as I can remember (so, 2.6.3x kernels, too). Currently I'm on 3.0.13.

Logs below show state after fresh boot where network traffic doesn't get through. Note that it shows that link is up but still - traffic doesn't get passed. tcpdump on eth0 interface sees only traffic initiated from this server and no traffic from outside.

Quoted 8 lines

Tried to set every offload (via ethtool -K) to off but that didn't help.

I have to make:
ip link set eth0 down; ip link set eth0 up
wait few second and then traffic starts to get passed. When I do that this shows in dmesg: [ 1058.266480] forcedeth 0000:00:08.0: irq 40 for MSI/MSI-X

[ 1069.163238] eth0: no IPv6 routers present

One more note. This doesn't always happens like that. Sometimes I boot and I get link working, traffic gets passed fine:

[   34.038476] forcedeth: Reverse Engineered nForce ethernet driver. Version 0.64.
[   34.038698] ACPI: PCI Interrupt Link [LMAC] enabled at IRQ 22
[   34.038705] forcedeth 0000:00:08.0: PCI INT A -> Link[LMAC] -> GSI 22 (level, low) -> IRQ 22
[   34.038710] forcedeth 0000:00:08.0: setting latency timer to 64
[   34.565309] forcedeth 0000:00:08.0: ifname eth0, PHY OUI 0x5043 @ 2, addr 00:30:48:7a:b4:88
[   34.565314] forcedeth 0000:00:08.0: highdma csum vlan pwrctl mgmt gbit lnktim msi desc-v3
[   34.566823] ACPI: PCI Interrupt Link [LMAD] enabled at IRQ 21
[   34.566831] forcedeth 0000:00:09.0: PCI INT A -> Link[LMAD] -> GSI 21 (level, low) -> IRQ 21
[   34.566836] forcedeth 0000:00:09.0: setting latency timer to 64
[   35.091341] forcedeth 0000:00:09.0: ifname eth1, PHY OUI 0x5043 @ 3, addr 00:30:48:7a:b4:89
[   35.091346] forcedeth 0000:00:09.0: highdma csum vlan pwrctl mgmt gbit lnktim msi desc-v3
[   35.093492] Loading kernel module for a network device with CAP_SYS_MODULE (deprecated).  Use CAP_NET_ADMIN and alias netdev-eth0 instead
[   35.101074] forcedeth 0000:00:08.0: irq 40 for MSI/MSI-X
[   35.101267] forcedeth 0000:00:08.0: eth0: no link during initialization
[   35.102322] ADDRCONF(NETDEV_UP): eth0: link is not ready
[   35.663785] ip_tables: (C) 2000-2006 Netfilter Core Team
[   35.756083] nf_conntrack version 0.5.0 (16384 buckets, 65536 max)
[   37.147869] forcedeth 0000:00:08.0: eth0: link up
[   37.149108] ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[   47.176677] eth0: no IPv6 routers present
Quoted 124 lines

(CC to few people shown by get_maintainer.pl)

LOGS/OUTPUTs from non working state after fresh boot. # dmesg|grep eth

[   34.528326] forcedeth: Reverse Engineered nForce ethernet driver.

Version 0.64. [ 34.528542] forcedeth 0000:00:08.0: PCI INT A ->
Link[LMAC] -> GSI 22 (level, low) -> IRQ 22 [ 34.528547] forcedeth
0000:00:08.0: setting latency timer to 64 [ 35.055319] forcedeth
0000:00:08.0: ifname eth0, PHY OUI 0x5043 @ 2, addr 00:30:48:7a:b4:88 [ 35.055324] forcedeth 0000:00:08.0: highdma csum vlan pwrctl mgmt gbit
lnktim msi desc-v3 [ 35.056480] forcedeth 0000:00:09.0: PCI INT A ->
Link[LMAD] -> GSI 21 (level, low) -> IRQ 21 [ 35.056485] forcedeth
0000:00:09.0: setting latency timer to 64 [ 35.581960] forcedeth
0000:00:09.0: ifname eth1, PHY OUI 0x5043 @ 3, addr 00:30:48:7a:b4:89 [ 35.581965] forcedeth 0000:00:09.0: highdma csum vlan pwrctl mgmt gbit
lnktim msi desc-v3 [ 35.584211] Loading kernel module for a network
device with CAP_SYS_MODULE (deprecated). Use CAP_NET_ADMIN and alias
netdev-eth0 instead [ 35.591778] forcedeth 0000:00:08.0: irq 40 for
MSI/MSI-X

[   35.591970] forcedeth 0000:00:08.0: eth0: no link during initialization
[   35.593030] ADDRCONF(NETDEV_UP): eth0: link is not ready
[   37.666757] forcedeth 0000:00:08.0: eth0: link up
[   37.668624] ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[   48.150006] eth0: no IPv6 routers present

lspci -vv:
00:09.0 Bridge: nVidia Corporation MCP55 Ethernet (rev a3)
Subsystem: Super Micro Computer Inc Device 1611
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0 (250ns min, 5000ns max)

        Interrupt: pin A routed to IRQ 21
        Region 0: Memory at feaf5000 (32-bit, non-prefetchable) [size=4K]
        Region 1: I/O ports at b080 [size=8]
        Region 2: Memory at feafa000 (32-bit, non-prefetchable) [size=256]
        Region 3: Memory at feaf4c00 (32-bit, non-prefetchable) [size=16]
        Capabilities: [44] Power Management version 2
                Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA
PME(D0+,D1+,D2+,D3hot+,D3cold+) Status: D0 NoSoftRst- PME-Enable- DSel=0
DScale=0 PME- Capabilities: [70] MSI-X: Enable- Count=8 Masked-
                Vector table: BAR=2 offset=00000000
                PBA: BAR=3 offset=00000000
        Capabilities: [50] MSI: Enable- Count=1/8 Maskable+ 64bit+
                Address: 0000000000000000  Data: 0000
                Masking: 00000000  Pending: 00000000
        Capabilities: [6c] HyperTransport: MSI Mapping Enable- Fixed+
        Kernel driver in use: forcedeth
# ethtool eth0
Settings for eth0:
        Supported ports: [ MII ]
        Supported link modes:   10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
        Supported pause frame use: No
        Supports auto-negotiation: Yes
        Advertised link modes:  10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
        Advertised pause frame use: No
        Advertised auto-negotiation: Yes
        Speed: 1000Mb/s
        Duplex: Full
        Port: MII
        PHYAD: 2
        Transceiver: external
        Auto-negotiation: on
        Supports Wake-on: g
        Wake-on: d
        Link detected: yes
# ethtool -S eth0
NIC statistics:
     tx_bytes: 46352
     tx_zero_rexmt: 704
     tx_one_rexmt: 0
     tx_many_rexmt: 0
     tx_late_collision: 0
     tx_fifo_errors: 0
     tx_carrier_errors: 0
     tx_excess_deferral: 0
     tx_retry_error: 0
     rx_frame_error: 0
     rx_extra_byte: 0
     rx_late_collision: 0
     rx_runt: 0
     rx_frame_too_long: 0
     rx_over_errors: 1
     rx_crc_errors: 0
     rx_frame_align_error: 0
     rx_length_error: 0
     rx_unicast: 31
     rx_multicast: 0
     rx_broadcast: 436
     rx_packets: 467
     rx_errors_total: 1
     tx_errors_total: 0
     tx_deferral: 0
     tx_packets: 704
     rx_bytes: 35872
     tx_pause: 0
     rx_pause: 0
     rx_drop_frame: 170
# ethtool -k eth0
Offload parameters for eth0:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp-segmentation-offload: on
udp-fragmentation-offload: off
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off
receive-hashing: off

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc sfq state UP qlen 1000 link/ether 00:30:48:7a:b4:88 brd ff:ff:ff:ff:ff:ff

Signature · 2 lines
Arkadiusz Miśkiewicz        PLD/Linux Team
arekm / maven.pl            http://ftp.pld-linux.org/
#2 lore raw
Quoted 23 lines · Arkadiusz Miśkiewicz
Quoted 13 lines · Arkadiusz Miśkiewicz

Hello,

I have few supermicro machines connected to edge-core switch using 1Gbps uplink. The problem is that after boot I have to restart net to get network card pass traffic. The problem is only with forcedeth network cards. It happens on different supermicro servers and it happens for as far as I can remember (so, 2.6.3x kernels, too). Currently I'm on 3.0.13.

Logs below show state after fresh boot where network traffic doesn't get through. Note that it shows that link is up but still - traffic doesn't get passed. tcpdump on eth0 interface sees only traffic initiated from this server and no traffic from outside.

Quoted 8 lines

Tried to set every offload (via ethtool -K) to off but that didn't help.

I have to make:
ip link set eth0 down; ip link set eth0 up
wait few second and then traffic starts to get passed. When I do that this shows in dmesg: [ 1058.266480] forcedeth 0000:00:08.0: irq 40 for MSI/MSI-X

[ 1069.163238] eth0: no IPv6 routers present
Quoted 2 lines

One more note. This doesn't always happens like that. Sometimes I boot and I get link working, traffic gets passed fine:

rmmod forcedeth; modprobe forcedeth; ifup doesn't help

Counters on switch side after forcedeth machine reboot, no traffic being passed, switch counters reset.

#show interfaces counters ethernet 1/19
Ethernet 1/19
Iftable Stats:
Octets Input: 0, Octets Output: 11116437
Unicast Input: 0, Unicast Output: 59
Discard Input: 0, Discard Output: 0
Error Input: 0, Error Output: 0
Unknown Protos Input: 0, QLen Output: 0
Extended Iftable Stats:
Multi-cast Input: 0, Multi-cast Output: 24742
Broadcast Input: 0, Broadcast Output: 790
Ether-like Stats:
Alignment Errors: 0, FCS Errors: 0
Single Collision Frames: 0, Multiple Collision Frames: 0
SQE Test Errors: 0, Deferred Transmissions: 0
Late Collisions: 0, Excessive Collisions: 0
Internal Mac Transmit Errors: 0, Internal Mac Receive Errors: 0 Frames Too Long: 0, Carrier Sense Errors: 0
Symbol Errors: 0
RMON Stats:
Drop Events: 0, Octets: 11075170, Packets: 25549
Broadcast PKTS: 790, Multi-cast PKTS: 24742
Undersize PKTS: 0, Oversize PKTS: 0
Fragments: 0, Jabbers: 0
CRC Align Errors: 0, Collisions: 0
Packet Size <= 64 Octets: 2060, Packet Size 65 to 127 Octets: 12764 Packet Size 128 to 255 Octets: 364, Packet Size 256 to 511 Octets: 292 Packet Size 512 to 1023 Octets: 8755, Packet Size 1024 to 1518 Octets: 1314

No errors here.

# dmesg|grep eth0

[   33.432266] forcedeth 0000:00:08.0: ifname eth0, PHY OUI 0x5043 @ 2, addr 
00:30:48:7a:57:a6
[   33.960287] Loading kernel module for a network device with CAP_SYS_MODULE 
(deprecated).  Use CAP_NET_ADMIN and alias netdev-eth0 instead
[   33.967925] forcedeth 0000:00:08.0: eth0: no link during initialization
[   33.970995] ADDRCONF(NETDEV_UP): eth0: link is not ready
[   36.380311] forcedeth 0000:00:08.0: eth0: link up
[   36.384016] ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[   47.150014] eth0: no IPv6 routers present

forcedeth side:

# ethtool -S eth0
NIC statistics:
     tx_bytes: 69926
     tx_zero_rexmt: 1011
     tx_one_rexmt: 0
     tx_many_rexmt: 0
     tx_late_collision: 0
     tx_fifo_errors: 0
     tx_carrier_errors: 0
     tx_excess_deferral: 0
     tx_retry_error: 0
     rx_frame_error: 0
     rx_extra_byte: 0
     rx_late_collision: 0
     rx_runt: 0
     rx_frame_too_long: 0
     rx_over_errors: 0
     rx_crc_errors: 0
     rx_frame_align_error: 0
     rx_length_error: 0
     rx_unicast: 3047
     rx_multicast: 0
     rx_broadcast: 2113
     rx_packets: 5160
     rx_errors_total: 0
     tx_errors_total: 0
     tx_deferral: 0
     tx_packets: 1011
     rx_bytes: 382259
     tx_pause: 0
     rx_pause: 0
     rx_drop_frame: 0
Quoted 167 lines
[   34.038476] forcedeth: Reverse Engineered nForce ethernet driver.
Version 0.64. [   34.038698] ACPI: PCI Interrupt Link [LMAC] enabled at
IRQ 22
[   34.038705] forcedeth 0000:00:08.0: PCI INT A -> Link[LMAC] -> GSI 22

(level, low) -> IRQ 22 [ 34.038710] forcedeth 0000:00:08.0: setting
latency timer to 64 [ 34.565309] forcedeth 0000:00:08.0: ifname eth0,
PHY OUI 0x5043 @ 2, addr 00:30:48:7a:b4:88 [ 34.565314] forcedeth
0000:00:08.0: highdma csum vlan pwrctl mgmt gbit lnktim msi desc-v3 [ 34.566823] ACPI: PCI Interrupt Link [LMAD] enabled at IRQ 21

[   34.566831] forcedeth 0000:00:09.0: PCI INT A -> Link[LMAD] -> GSI 21

(level, low) -> IRQ 21 [ 34.566836] forcedeth 0000:00:09.0: setting
latency timer to 64 [ 35.091341] forcedeth 0000:00:09.0: ifname eth1,
PHY OUI 0x5043 @ 3, addr 00:30:48:7a:b4:89 [ 35.091346] forcedeth
0000:00:09.0: highdma csum vlan pwrctl mgmt gbit lnktim msi desc-v3 [ 35.093492] Loading kernel module for a network device with CAP_SYS_MODULE (deprecated). Use CAP_NET_ADMIN and alias netdev-eth0 instead [ 35.101074] forcedeth 0000:00:08.0: irq 40 for MSI/MSI-X

[   35.101267] forcedeth 0000:00:08.0: eth0: no link during initialization
[   35.102322] ADDRCONF(NETDEV_UP): eth0: link is not ready
[   35.663785] ip_tables: (C) 2000-2006 Netfilter Core Team
[   35.756083] nf_conntrack version 0.5.0 (16384 buckets, 65536 max)
[   37.147869] forcedeth 0000:00:08.0: eth0: link up
[   37.149108] ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[   47.176677] eth0: no IPv6 routers present
Quoted 141 lines

(CC to few people shown by get_maintainer.pl)

LOGS/OUTPUTs from non working state after fresh boot. # dmesg|grep eth

[   34.528326] forcedeth: Reverse Engineered nForce ethernet driver.

Version 0.64. [ 34.528542] forcedeth 0000:00:08.0: PCI INT A ->
Link[LMAC] -> GSI 22 (level, low) -> IRQ 22 [ 34.528547] forcedeth
0000:00:08.0: setting latency timer to 64 [ 35.055319] forcedeth
0000:00:08.0: ifname eth0, PHY OUI 0x5043 @ 2, addr 00:30:48:7a:b4:88 [ 35.055324] forcedeth 0000:00:08.0: highdma csum vlan pwrctl mgmt gbit
lnktim msi desc-v3 [ 35.056480] forcedeth 0000:00:09.0: PCI INT A ->
Link[LMAD] -> GSI 21 (level, low) -> IRQ 21 [ 35.056485] forcedeth
0000:00:09.0: setting latency timer to 64 [ 35.581960] forcedeth
0000:00:09.0: ifname eth1, PHY OUI 0x5043 @ 3, addr 00:30:48:7a:b4:89 [ 35.581965] forcedeth 0000:00:09.0: highdma csum vlan pwrctl mgmt gbit
lnktim msi desc-v3 [ 35.584211] Loading kernel module for a network
device with CAP_SYS_MODULE (deprecated). Use CAP_NET_ADMIN and alias
netdev-eth0 instead [ 35.591778] forcedeth 0000:00:08.0: irq 40 for
MSI/MSI-X

[   35.591970] forcedeth 0000:00:08.0: eth0: no link during
initialization [   35.593030] ADDRCONF(NETDEV_UP): eth0: link is not
ready
[   37.666757] forcedeth 0000:00:08.0: eth0: link up
[   37.668624] ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[   48.150006] eth0: no IPv6 routers present

lspci -vv:
00:09.0 Bridge: nVidia Corporation MCP55 Ethernet (rev a3)

Subsystem: Super Micro Computer Inc Device 1611
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-

Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0 (250ns min, 5000ns max)

Interrupt: pin A routed to IRQ 21
Region 0: Memory at feaf5000 (32-bit, non-prefetchable) [size=4K] Region 1: I/O ports at b080 [size=8]
Region 2: Memory at feafa000 (32-bit, non-prefetchable) [size=256] Region 3: Memory at feaf4c00 (32-bit, non-prefetchable) [size=16] Capabilities: [44] Power Management version 2

Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA

PME(D0+,D1+,D2+,D3hot+,D3cold+) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- Capabilities: [70] MSI-X: Enable- Count=8 Masked-

                Vector table: BAR=2 offset=00000000
                PBA: BAR=3 offset=00000000
        
        Capabilities: [50] MSI: Enable- Count=1/8 Maskable+ 64bit+
        
                Address: 0000000000000000  Data: 0000
                Masking: 00000000  Pending: 00000000
        
        Capabilities: [6c] HyperTransport: MSI Mapping Enable- Fixed+
        Kernel driver in use: forcedeth
# ethtool eth0

Settings for eth0:
Supported ports: [ MII ]
Supported link modes: 10baseT/Half 10baseT/Full

100baseT/Half 100baseT/Full 1000baseT/Full

Supported pause frame use: No
Supports auto-negotiation: Yes Advertised link modes: 10baseT/Half 10baseT/Full

100baseT/Half 100baseT/Full 1000baseT/Full

Advertised pause frame use: No
Advertised auto-negotiation: Yes
Speed: 1000Mb/s
Duplex: Full
Port: MII
PHYAD: 2
Transceiver: external
Auto-negotiation: on
Supports Wake-on: g
Wake-on: d
Link detected: yes

# ethtool -S eth0

NIC statistics:

     tx_bytes: 46352
     tx_zero_rexmt: 704
     tx_one_rexmt: 0
     tx_many_rexmt: 0
     tx_late_collision: 0
     tx_fifo_errors: 0
     tx_carrier_errors: 0
     tx_excess_deferral: 0
     tx_retry_error: 0
     rx_frame_error: 0
     rx_extra_byte: 0
     rx_late_collision: 0
     rx_runt: 0
     rx_frame_too_long: 0
     rx_over_errors: 1
     rx_crc_errors: 0
     rx_frame_align_error: 0
     rx_length_error: 0
     rx_unicast: 31
     rx_multicast: 0
     rx_broadcast: 436
     rx_packets: 467
     rx_errors_total: 1
     tx_errors_total: 0
     tx_deferral: 0
     tx_packets: 704
     rx_bytes: 35872
     tx_pause: 0
     rx_pause: 0
     rx_drop_frame: 170
# ethtool -k eth0
Offload parameters for eth0:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp-segmentation-offload: on
udp-fragmentation-offload: off
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off
receive-hashing: off

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc sfq state UP qlen 1000 link/ether 00:30:48:7a:b4:88 brd ff:ff:ff:ff:ff:ff

Signature · 2 lines
Arkadiusz Miśkiewicz        PLD/Linux Team
arekm / maven.pl            http://ftp.pld-linux.org/
#3 lore raw
Quoted 24 lines · Arkadiusz Miśkiewicz
Quoted 23 lines · Arkadiusz Miśkiewicz
Quoted 21 lines · Arkadiusz Miśkiewicz

Hello,

I have few supermicro machines connected to edge-core switch using 1Gbps uplink. The problem is that after boot I have to restart net to get network card pass traffic. The problem is only with forcedeth network cards. It happens on different supermicro servers and it happens for as far as I can remember (so, 2.6.3x kernels, too). Currently I'm on 3.0.13.

Logs below show state after fresh boot where network traffic doesn't get through. Note that it shows that link is up but still - traffic doesn't get passed. tcpdump on eth0 interface sees only traffic initiated from this server and no traffic from outside.

Tried to set every offload (via ethtool -K) to off but that didn't help.

I have to make:
ip link set eth0 down; ip link set eth0 up
wait few second and then traffic starts to get passed. When I do that this shows in dmesg: [ 1058.266480] forcedeth 0000:00:08.0: irq 40 for MSI/MSI-X

[ 1069.163238] eth0: no IPv6 routers present
Quoted 3 lines
Quoted 2 lines

One more note. This doesn't always happens like that. Sometimes I boot and I get link working, traffic gets passed fine:

<snip>
We have noticed similar behavior on some of our SuperMicro systems. It was worse for us; the interfaces would work fine on boot but then inexplicably stop after weeks of running causing production outages. We simply stopped using them - John