forcedeth doesn't pass traffic (3.0.x kernel and earliers)
Subsystems (2)
Hello,
I have few supermicro machines connected to edge-core switch using 1Gbps uplink. The problem is that after boot I have to restart net to get network card pass traffic. The problem is only with forcedeth network cards. It happens on different supermicro servers and it happens for as far as I can remember (so, 2.6.3x kernels, too). Currently I'm on 3.0.13.
Logs below show state after fresh boot where network traffic doesn't get through. Note that it shows that link is up but still - traffic doesn't get passed. tcpdump on eth0 interface sees only traffic initiated from this server and no traffic from outside.
Tried to set every offload (via ethtool -K) to off but that didn't help.
I have to make:
ip link set eth0 down; ip link set eth0 up
wait few second and then traffic starts to get passed. When I do that this shows in dmesg:
[ 1058.266480] forcedeth 0000:00:08.0: irq 40 for MSI/MSI-X [ 1069.163238] eth0: no IPv6 routers present
(CC to few people shown by get_maintainer.pl)
LOGS/OUTPUTs from non working state after fresh boot. # dmesg|grep eth
[ 34.528326] forcedeth: Reverse Engineered nForce ethernet driver. Version 0.64. [ 34.528542] forcedeth 0000:00:08.0: PCI INT A -> Link[LMAC] -> GSI 22 (level, low) -> IRQ 22 [ 34.528547] forcedeth 0000:00:08.0: setting latency timer to 64 [ 35.055319] forcedeth 0000:00:08.0: ifname eth0, PHY OUI 0x5043 @ 2, addr 00:30:48:7a:b4:88 [ 35.055324] forcedeth 0000:00:08.0: highdma csum vlan pwrctl mgmt gbit lnktim msi desc-v3 [ 35.056480] forcedeth 0000:00:09.0: PCI INT A -> Link[LMAD] -> GSI 21 (level, low) -> IRQ 21 [ 35.056485] forcedeth 0000:00:09.0: setting latency timer to 64 [ 35.581960] forcedeth 0000:00:09.0: ifname eth1, PHY OUI 0x5043 @ 3, addr 00:30:48:7a:b4:89 [ 35.581965] forcedeth 0000:00:09.0: highdma csum vlan pwrctl mgmt gbit lnktim msi desc-v3 [ 35.584211] Loading kernel module for a network device with CAP_SYS_MODULE (deprecated). Use CAP_NET_ADMIN and alias netdev-eth0 instead [ 35.591778] forcedeth 0000:00:08.0: irq 40 for MSI/MSI-X [ 35.591970] forcedeth 0000:00:08.0: eth0: no link during initialization [ 35.593030] ADDRCONF(NETDEV_UP): eth0: link is not ready [ 37.666757] forcedeth 0000:00:08.0: eth0: link up [ 37.668624] ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready [ 48.150006] eth0: no IPv6 routers present
lspci -vv:
00:09.0 Bridge: nVidia Corporation MCP55 Ethernet (rev a3)
Subsystem: Super Micro Computer Inc Device 1611
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0 (250ns min, 5000ns max)
Interrupt: pin A routed to IRQ 21
Region 0: Memory at feaf5000 (32-bit, non-prefetchable) [size=4K]
Region 1: I/O ports at b080 [size=8]
Region 2: Memory at feafa000 (32-bit, non-prefetchable) [size=256]
Region 3: Memory at feaf4c00 (32-bit, non-prefetchable) [size=16]
Capabilities: [44] Power Management version 2
Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [70] MSI-X: Enable- Count=8 Masked-
Vector table: BAR=2 offset=00000000
PBA: BAR=3 offset=00000000
Capabilities: [50] MSI: Enable- Count=1/8 Maskable+ 64bit+
Address: 0000000000000000 Data: 0000
Masking: 00000000 Pending: 00000000
Capabilities: [6c] HyperTransport: MSI Mapping Enable- Fixed+
Kernel driver in use: forcedeth# ethtool eth0
Settings for eth0:
Supported ports: [ MII ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Supported pause frame use: No
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Advertised pause frame use: No
Advertised auto-negotiation: Yes
Speed: 1000Mb/s
Duplex: Full
Port: MII
PHYAD: 2
Transceiver: external
Auto-negotiation: on
Supports Wake-on: g
Wake-on: d
Link detected: yes# ethtool -S eth0
NIC statistics:
tx_bytes: 46352
tx_zero_rexmt: 704
tx_one_rexmt: 0
tx_many_rexmt: 0
tx_late_collision: 0
tx_fifo_errors: 0
tx_carrier_errors: 0
tx_excess_deferral: 0
tx_retry_error: 0
rx_frame_error: 0
rx_extra_byte: 0
rx_late_collision: 0
rx_runt: 0
rx_frame_too_long: 0
rx_over_errors: 1
rx_crc_errors: 0
rx_frame_align_error: 0
rx_length_error: 0
rx_unicast: 31
rx_multicast: 0
rx_broadcast: 436
rx_packets: 467
rx_errors_total: 1
tx_errors_total: 0
tx_deferral: 0
tx_packets: 704
rx_bytes: 35872
tx_pause: 0
rx_pause: 0
rx_drop_frame: 170# ethtool -k eth0 Offload parameters for eth0: rx-checksumming: on tx-checksumming: on scatter-gather: on tcp-segmentation-offload: on udp-fragmentation-offload: off generic-segmentation-offload: on generic-receive-offload: on large-receive-offload: off rx-vlan-offload: on tx-vlan-offload: on ntuple-filters: off receive-hashing: off
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc sfq state UP qlen 1000
link/ether 00:30:48:7a:b4:88 brd ff:ff:ff:ff:ff:ff
Signature · 2 lines
Arkadiusz Miśkiewicz PLD/Linux Team arekm / maven.pl http://ftp.pld-linux.org/
Quoted 13 lines · Arkadiusz Miśkiewicz
Hello,
I have few supermicro machines connected to edge-core switch using 1Gbps uplink. The problem is that after boot I have to restart net to get network card pass traffic. The problem is only with forcedeth network cards. It happens on different supermicro servers and it happens for as far as I can remember (so, 2.6.3x kernels, too). Currently I'm on 3.0.13.
Logs below show state after fresh boot where network traffic doesn't get through. Note that it shows that link is up but still - traffic doesn't get passed. tcpdump on eth0 interface sees only traffic initiated from this server and no traffic from outside.
Quoted 8 lines
Tried to set every offload (via ethtool -K) to off but that didn't help.
I have to make:
ip link set eth0 down; ip link set eth0 up
wait few second and then traffic starts to get passed. When I do that this shows in dmesg: [ 1058.266480] forcedeth 0000:00:08.0: irq 40 for MSI/MSI-X[ 1069.163238] eth0: no IPv6 routers present
One more note. This doesn't always happens like that. Sometimes I boot and I get link working, traffic gets passed fine:
[ 34.038476] forcedeth: Reverse Engineered nForce ethernet driver. Version 0.64. [ 34.038698] ACPI: PCI Interrupt Link [LMAC] enabled at IRQ 22 [ 34.038705] forcedeth 0000:00:08.0: PCI INT A -> Link[LMAC] -> GSI 22 (level, low) -> IRQ 22 [ 34.038710] forcedeth 0000:00:08.0: setting latency timer to 64 [ 34.565309] forcedeth 0000:00:08.0: ifname eth0, PHY OUI 0x5043 @ 2, addr 00:30:48:7a:b4:88 [ 34.565314] forcedeth 0000:00:08.0: highdma csum vlan pwrctl mgmt gbit lnktim msi desc-v3 [ 34.566823] ACPI: PCI Interrupt Link [LMAD] enabled at IRQ 21 [ 34.566831] forcedeth 0000:00:09.0: PCI INT A -> Link[LMAD] -> GSI 21 (level, low) -> IRQ 21 [ 34.566836] forcedeth 0000:00:09.0: setting latency timer to 64 [ 35.091341] forcedeth 0000:00:09.0: ifname eth1, PHY OUI 0x5043 @ 3, addr 00:30:48:7a:b4:89 [ 35.091346] forcedeth 0000:00:09.0: highdma csum vlan pwrctl mgmt gbit lnktim msi desc-v3 [ 35.093492] Loading kernel module for a network device with CAP_SYS_MODULE (deprecated). Use CAP_NET_ADMIN and alias netdev-eth0 instead [ 35.101074] forcedeth 0000:00:08.0: irq 40 for MSI/MSI-X [ 35.101267] forcedeth 0000:00:08.0: eth0: no link during initialization [ 35.102322] ADDRCONF(NETDEV_UP): eth0: link is not ready [ 35.663785] ip_tables: (C) 2000-2006 Netfilter Core Team [ 35.756083] nf_conntrack version 0.5.0 (16384 buckets, 65536 max) [ 37.147869] forcedeth 0000:00:08.0: eth0: link up [ 37.149108] ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready [ 47.176677] eth0: no IPv6 routers present
Quoted 124 lines
(CC to few people shown by get_maintainer.pl)
LOGS/OUTPUTs from non working state after fresh boot. # dmesg|grep eth
[ 34.528326] forcedeth: Reverse Engineered nForce ethernet driver.Version 0.64. [ 34.528542] forcedeth 0000:00:08.0: PCI INT A ->
Link[LMAC] -> GSI 22 (level, low) -> IRQ 22 [ 34.528547] forcedeth
0000:00:08.0: setting latency timer to 64 [ 35.055319] forcedeth
0000:00:08.0: ifname eth0, PHY OUI0x5043@ 2, addr 00:30:48:7a:b4:88 [ 35.055324] forcedeth 0000:00:08.0: highdma csum vlan pwrctl mgmt gbit
lnktim msi desc-v3 [ 35.056480] forcedeth 0000:00:09.0: PCI INT A ->
Link[LMAD] -> GSI 21 (level, low) -> IRQ 21 [ 35.056485] forcedeth
0000:00:09.0: setting latency timer to 64 [ 35.581960] forcedeth
0000:00:09.0: ifname eth1, PHY OUI0x5043@ 3, addr 00:30:48:7a:b4:89 [ 35.581965] forcedeth 0000:00:09.0: highdma csum vlan pwrctl mgmt gbit
lnktim msi desc-v3 [ 35.584211] Loading kernel module for a network
device withCAP_SYS_MODULE(deprecated). UseCAP_NET_ADMINand alias
netdev-eth0 instead [ 35.591778] forcedeth 0000:00:08.0: irq 40 for
MSI/MSI-X[ 35.591970] forcedeth 0000:00:08.0: eth0: no link during initialization [ 35.593030] ADDRCONF(NETDEV_UP): eth0: link is not ready [ 37.666757] forcedeth 0000:00:08.0: eth0: link up [ 37.668624] ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready [ 48.150006] eth0: no IPv6 routers presentlspci -vv:
00:09.0 Bridge: nVidia Corporation MCP55 Ethernet (rev a3)
Subsystem: Super Micro Computer Inc Device 1611
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0 (250ns min, 5000ns max)Interrupt: pin A routed to IRQ 21 Region 0: Memory at feaf5000 (32-bit, non-prefetchable) [size=4K] Region 1: I/O ports at b080 [size=8] Region 2: Memory at feafa000 (32-bit, non-prefetchable) [size=256] Region 3: Memory at feaf4c00 (32-bit, non-prefetchable) [size=16] Capabilities: [44] Power Management version 2 Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold+) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- Capabilities: [70] MSI-X: Enable- Count=8 Masked- Vector table: BAR=2 offset=00000000 PBA: BAR=3 offset=00000000 Capabilities: [50] MSI: Enable- Count=1/8 Maskable+ 64bit+ Address: 0000000000000000 Data: 0000 Masking: 00000000 Pending: 00000000 Capabilities: [6c] HyperTransport: MSI Mapping Enable- Fixed+ Kernel driver in use: forcedeth# ethtool eth0 Settings for eth0: Supported ports: [ MII ] Supported link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full 1000baseT/Full Supported pause frame use: No Supports auto-negotiation: Yes Advertised link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full 1000baseT/Full Advertised pause frame use: No Advertised auto-negotiation: Yes Speed: 1000Mb/s Duplex: Full Port: MII PHYAD: 2 Transceiver: external Auto-negotiation: on Supports Wake-on: g Wake-on: d Link detected: yes# ethtool -S eth0 NIC statistics: tx_bytes: 46352 tx_zero_rexmt: 704 tx_one_rexmt: 0 tx_many_rexmt: 0 tx_late_collision: 0 tx_fifo_errors: 0 tx_carrier_errors: 0 tx_excess_deferral: 0 tx_retry_error: 0 rx_frame_error: 0 rx_extra_byte: 0 rx_late_collision: 0 rx_runt: 0 rx_frame_too_long: 0 rx_over_errors: 1 rx_crc_errors: 0 rx_frame_align_error: 0 rx_length_error: 0 rx_unicast: 31 rx_multicast: 0 rx_broadcast: 436 rx_packets: 467 rx_errors_total: 1 tx_errors_total: 0 tx_deferral: 0 tx_packets: 704 rx_bytes: 35872 tx_pause: 0 rx_pause: 0 rx_drop_frame: 170# ethtool -k eth0 Offload parameters for eth0: rx-checksumming: on tx-checksumming: on scatter-gather: on tcp-segmentation-offload: on udp-fragmentation-offload: off generic-segmentation-offload: on generic-receive-offload: on large-receive-offload: off rx-vlan-offload: on tx-vlan-offload: on ntuple-filters: off receive-hashing: off2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc sfq state UP qlen 1000 link/ether 00:30:48:7a:b4:88 brd ff:ff:ff:ff:ff:ff
Signature · 2 lines
Arkadiusz Miśkiewicz PLD/Linux Team arekm / maven.pl http://ftp.pld-linux.org/
Quoted 23 lines · Arkadiusz Miśkiewicz
Quoted 13 lines · Arkadiusz Miśkiewicz
Hello,
I have few supermicro machines connected to edge-core switch using 1Gbps uplink. The problem is that after boot I have to restart net to get network card pass traffic. The problem is only with forcedeth network cards. It happens on different supermicro servers and it happens for as far as I can remember (so, 2.6.3x kernels, too). Currently I'm on 3.0.13.
Logs below show state after fresh boot where network traffic doesn't get through. Note that it shows that link is up but still - traffic doesn't get passed. tcpdump on eth0 interface sees only traffic initiated from this server and no traffic from outside.
Quoted 8 lines
Tried to set every offload (via ethtool -K) to off but that didn't help.
I have to make:
ip link set eth0 down; ip link set eth0 up
wait few second and then traffic starts to get passed. When I do that this shows in dmesg: [ 1058.266480] forcedeth 0000:00:08.0: irq 40 for MSI/MSI-X[ 1069.163238] eth0: no IPv6 routers present
Quoted 2 lines
One more note. This doesn't always happens like that. Sometimes I boot and I get link working, traffic gets passed fine:
rmmod forcedeth; modprobe forcedeth; ifup doesn't help
Counters on switch side after forcedeth machine reboot, no traffic being passed, switch counters reset.
#show interfaces counters ethernet 1/19
Ethernet 1/19
Iftable Stats:
Octets Input: 0, Octets Output: 11116437
Unicast Input: 0, Unicast Output: 59
Discard Input: 0, Discard Output: 0
Error Input: 0, Error Output: 0
Unknown Protos Input: 0, QLen Output: 0
Extended Iftable Stats:
Multi-cast Input: 0, Multi-cast Output: 24742
Broadcast Input: 0, Broadcast Output: 790
Ether-like Stats:
Alignment Errors: 0, FCS Errors: 0
Single Collision Frames: 0, Multiple Collision Frames: 0
SQE Test Errors: 0, Deferred Transmissions: 0
Late Collisions: 0, Excessive Collisions: 0
Internal Mac Transmit Errors: 0, Internal Mac Receive Errors: 0 Frames Too Long: 0, Carrier Sense Errors: 0
Symbol Errors: 0
RMON Stats:
Drop Events: 0, Octets: 11075170, Packets: 25549
Broadcast PKTS: 790, Multi-cast PKTS: 24742
Undersize PKTS: 0, Oversize PKTS: 0
Fragments: 0, Jabbers: 0
CRC Align Errors: 0, Collisions: 0
Packet Size <= 64 Octets: 2060, Packet Size 65 to 127 Octets: 12764 Packet Size 128 to 255 Octets: 364, Packet Size 256 to 511 Octets: 292 Packet Size 512 to 1023 Octets: 8755, Packet Size 1024 to 1518 Octets: 1314
No errors here.
# dmesg|grep eth0
[ 33.432266] forcedeth 0000:00:08.0: ifname eth0, PHY OUI 0x5043 @ 2, addr 00:30:48:7a:57:a6 [ 33.960287] Loading kernel module for a network device with CAP_SYS_MODULE (deprecated). Use CAP_NET_ADMIN and alias netdev-eth0 instead [ 33.967925] forcedeth 0000:00:08.0: eth0: no link during initialization [ 33.970995] ADDRCONF(NETDEV_UP): eth0: link is not ready [ 36.380311] forcedeth 0000:00:08.0: eth0: link up [ 36.384016] ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready [ 47.150014] eth0: no IPv6 routers present
forcedeth side:
# ethtool -S eth0
NIC statistics:
tx_bytes: 69926
tx_zero_rexmt: 1011
tx_one_rexmt: 0
tx_many_rexmt: 0
tx_late_collision: 0
tx_fifo_errors: 0
tx_carrier_errors: 0
tx_excess_deferral: 0
tx_retry_error: 0
rx_frame_error: 0
rx_extra_byte: 0
rx_late_collision: 0
rx_runt: 0
rx_frame_too_long: 0
rx_over_errors: 0
rx_crc_errors: 0
rx_frame_align_error: 0
rx_length_error: 0
rx_unicast: 3047
rx_multicast: 0
rx_broadcast: 2113
rx_packets: 5160
rx_errors_total: 0
tx_errors_total: 0
tx_deferral: 0
tx_packets: 1011
rx_bytes: 382259
tx_pause: 0
rx_pause: 0
rx_drop_frame: 0Quoted 167 lines
[ 34.038476] forcedeth: Reverse Engineered nForce ethernet driver. Version 0.64. [ 34.038698] ACPI: PCI Interrupt Link [LMAC] enabled at IRQ 22 [ 34.038705] forcedeth 0000:00:08.0: PCI INT A -> Link[LMAC] -> GSI 22(level, low) -> IRQ 22 [ 34.038710] forcedeth 0000:00:08.0: setting
latency timer to 64 [ 34.565309] forcedeth 0000:00:08.0: ifname eth0,
PHY OUI0x5043@ 2, addr 00:30:48:7a:b4:88 [ 34.565314] forcedeth
0000:00:08.0: highdma csum vlan pwrctl mgmt gbit lnktim msi desc-v3 [ 34.566823] ACPI: PCI Interrupt Link [LMAD] enabled at IRQ 21[ 34.566831] forcedeth 0000:00:09.0: PCI INT A -> Link[LMAD] -> GSI 21(level, low) -> IRQ 21 [ 34.566836] forcedeth 0000:00:09.0: setting
latency timer to 64 [ 35.091341] forcedeth 0000:00:09.0: ifname eth1,
PHY OUI0x5043@ 3, addr 00:30:48:7a:b4:89 [ 35.091346] forcedeth
0000:00:09.0: highdma csum vlan pwrctl mgmt gbit lnktim msi desc-v3 [ 35.093492] Loading kernel module for a network device withCAP_SYS_MODULE(deprecated). UseCAP_NET_ADMINand alias netdev-eth0 instead [ 35.101074] forcedeth 0000:00:08.0: irq 40 for MSI/MSI-X[ 35.101267] forcedeth 0000:00:08.0: eth0: no link during initialization [ 35.102322] ADDRCONF(NETDEV_UP): eth0: link is not ready [ 35.663785] ip_tables: (C) 2000-2006 Netfilter Core Team [ 35.756083] nf_conntrack version 0.5.0 (16384 buckets, 65536 max) [ 37.147869] forcedeth 0000:00:08.0: eth0: link up [ 37.149108] ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready [ 47.176677] eth0: no IPv6 routers presentQuoted 141 lines
(CC to few people shown by get_maintainer.pl)
LOGS/OUTPUTs from non working state after fresh boot. # dmesg|grep eth
[ 34.528326] forcedeth: Reverse Engineered nForce ethernet driver.Version 0.64. [ 34.528542] forcedeth 0000:00:08.0: PCI INT A ->
Link[LMAC] -> GSI 22 (level, low) -> IRQ 22 [ 34.528547] forcedeth
0000:00:08.0: setting latency timer to 64 [ 35.055319] forcedeth
0000:00:08.0: ifname eth0, PHY OUI0x5043@ 2, addr 00:30:48:7a:b4:88 [ 35.055324] forcedeth 0000:00:08.0: highdma csum vlan pwrctl mgmt gbit
lnktim msi desc-v3 [ 35.056480] forcedeth 0000:00:09.0: PCI INT A ->
Link[LMAD] -> GSI 21 (level, low) -> IRQ 21 [ 35.056485] forcedeth
0000:00:09.0: setting latency timer to 64 [ 35.581960] forcedeth
0000:00:09.0: ifname eth1, PHY OUI0x5043@ 3, addr 00:30:48:7a:b4:89 [ 35.581965] forcedeth 0000:00:09.0: highdma csum vlan pwrctl mgmt gbit
lnktim msi desc-v3 [ 35.584211] Loading kernel module for a network
device withCAP_SYS_MODULE(deprecated). UseCAP_NET_ADMINand alias
netdev-eth0 instead [ 35.591778] forcedeth 0000:00:08.0: irq 40 for
MSI/MSI-X[ 35.591970] forcedeth 0000:00:08.0: eth0: no link during initialization [ 35.593030] ADDRCONF(NETDEV_UP): eth0: link is not ready [ 37.666757] forcedeth 0000:00:08.0: eth0: link up [ 37.668624] ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready [ 48.150006] eth0: no IPv6 routers presentlspci -vv:
00:09.0 Bridge: nVidia Corporation MCP55 Ethernet (rev a3)Subsystem: Super Micro Computer Inc Device 1611
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0 (250ns min, 5000ns max)Interrupt: pin A routed to IRQ 21
Region 0: Memory atfeaf5000(32-bit, non-prefetchable) [size=4K] Region 1: I/O ports at b080 [size=8]
Region 2: Memory atfeafa000(32-bit, non-prefetchable) [size=256] Region 3: Memory atfeaf4c00(32-bit, non-prefetchable) [size=16] Capabilities: [44] Power Management version 2Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA
PME(D0+,D1+,D2+,D3hot+,D3cold+)Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- Capabilities: [70] MSI-X: Enable- Count=8 Masked-Vector table: BAR=2 offset=00000000 PBA: BAR=3 offset=00000000 Capabilities: [50] MSI: Enable- Count=1/8 Maskable+ 64bit+ Address: 0000000000000000 Data: 0000 Masking: 00000000 Pending: 00000000 Capabilities: [6c] HyperTransport: MSI Mapping Enable- Fixed+ Kernel driver in use: forcedeth# ethtool eth0Settings for eth0:
Supported ports: [ MII ]
Supported link modes: 10baseT/Half 10baseT/Full100baseT/Half 100baseT/Full 1000baseT/Full
Supported pause frame use: No
Supports auto-negotiation: Yes Advertised link modes: 10baseT/Half 10baseT/Full100baseT/Half 100baseT/Full 1000baseT/Full
Advertised pause frame use: No
Advertised auto-negotiation: Yes
Speed: 1000Mb/s
Duplex: Full
Port: MII
PHYAD: 2
Transceiver: external
Auto-negotiation: on
Supports Wake-on: g
Wake-on: d
Link detected: yes# ethtool -S eth0NIC statistics:
tx_bytes: 46352 tx_zero_rexmt: 704 tx_one_rexmt: 0 tx_many_rexmt: 0 tx_late_collision: 0 tx_fifo_errors: 0 tx_carrier_errors: 0 tx_excess_deferral: 0 tx_retry_error: 0 rx_frame_error: 0 rx_extra_byte: 0 rx_late_collision: 0 rx_runt: 0 rx_frame_too_long: 0 rx_over_errors: 1 rx_crc_errors: 0 rx_frame_align_error: 0 rx_length_error: 0 rx_unicast: 31 rx_multicast: 0 rx_broadcast: 436 rx_packets: 467 rx_errors_total: 1 tx_errors_total: 0 tx_deferral: 0 tx_packets: 704 rx_bytes: 35872 tx_pause: 0 rx_pause: 0 rx_drop_frame: 170# ethtool -k eth0 Offload parameters for eth0: rx-checksumming: on tx-checksumming: on scatter-gather: on tcp-segmentation-offload: on udp-fragmentation-offload: off generic-segmentation-offload: on generic-receive-offload: on large-receive-offload: off rx-vlan-offload: on tx-vlan-offload: on ntuple-filters: off receive-hashing: off2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc sfq state UP qlen 1000 link/ether 00:30:48:7a:b4:88 brd ff:ff:ff:ff:ff:ff
Signature · 2 lines
Arkadiusz Miśkiewicz PLD/Linux Team arekm / maven.pl http://ftp.pld-linux.org/
Quoted 24 lines · Arkadiusz Miśkiewicz
Quoted 23 lines · Arkadiusz Miśkiewicz
Quoted 21 lines · Arkadiusz Miśkiewicz
Hello,
I have few supermicro machines connected to edge-core switch using 1Gbps uplink. The problem is that after boot I have to restart net to get network card pass traffic. The problem is only with forcedeth network cards. It happens on different supermicro servers and it happens for as far as I can remember (so, 2.6.3x kernels, too). Currently I'm on 3.0.13.
Logs below show state after fresh boot where network traffic doesn't get through. Note that it shows that link is up but still - traffic doesn't get passed. tcpdump on eth0 interface sees only traffic initiated from this server and no traffic from outside.
Tried to set every offload (via ethtool -K) to off but that didn't help.
I have to make:
ip link set eth0 down; ip link set eth0 up
wait few second and then traffic starts to get passed. When I do that this shows in dmesg: [ 1058.266480] forcedeth 0000:00:08.0: irq 40 for MSI/MSI-X[ 1069.163238] eth0: no IPv6 routers present
Quoted 3 lines
Quoted 2 lines
One more note. This doesn't always happens like that. Sometimes I boot and I get link working, traffic gets passed fine:
<snip>
We have noticed similar behavior on some of our SuperMicro systems. It was worse for us; the interfaces would work fine on boot but then inexplicably stop after weeks of running causing production outages. We simply stopped using them - John