Skip to contentSkip to conversation

r8169 crash

DISCUSSION Denys 2 participants 6 messages Nov 17, 2007 – Nov 19, 2007

Subsystems (2)
lore
Share & tools
lore

Denys

author
#0 lore raw

Hi

Recently i start to use r8169 cards and faced some issues.

How to reproduce (kind of difficult, and i cannot that on my live server anymore):

1)Badly crimped cable.
2)PCI-Express R8169 card

Kernel 2.6.23.1

05:01.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL-8169 Gigabit Ethernet (rev 10)
Subsystem: Realtek Semiconductor Co., Ltd. RTL-8169 Gigabit Ethernet
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr-
Stepping- SERR- FastB2B-
Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort-
<TAbort- <MAbort- >SERR- <PERR-
Latency: 64 (8000ns min, 16000ns max), Cache Line Size: 32 bytes
Interrupt: pin A routed to IRQ 21
Region 0: I/O ports at 1000 [size=256]
Region 1: Memory at 55001000 (32-bit, non-prefetchable) [size=256] Expansion ROM at fffe0000 [disabled] [size=128K]
Capabilities: [dc] Power Management version 2
Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=375mA
PME(D0-,D1+,D2+,D3hot+,D3cold+)
Status: D0 PME-Enable- DSel=0 DScale=0 PME-

r8169 Gigabit Ethernet driver 2.2LK-NAPI loaded
eth1: RTL8169sb/8110sb at 0xf881c000, 00:19:e0:72:49:88, XID 10000000 IRQ 21

To reproduce issue i need to shake faulty cable. Card will report:

Nov 15 22:09:36 vzone r8169: eth1: link up

So it is going up without going down!

Normally it is like this:

Nov  1 00:50:33 vzone r8169: eth1: link down
Nov  1 00:50:35 vzone r8169: eth1: link up

After that i have.

Nov 15 22:11:37 vzone NETDEV WATCHDOG: eth1: transmit timed out
Nov 15 22:11:49 vzone NETDEV WATCHDOG: eth1: transmit timed out
Nov 15 22:12:01 vzone NETDEV WATCHDOG: eth1: transmit timed out
Nov 15 22:12:13 vzone NETDEV WATCHDOG: eth1: transmit timed out
Nov 15 22:12:25 vzone NETDEV WATCHDOG: eth1: transmit timed out
Nov 15 22:12:37 vzone NETDEV WATCHDOG: eth1: transmit timed out
Nov 15 22:12:49 vzone NETDEV WATCHDOG: eth1: transmit timed out
Nov 15 22:13:01 vzone NETDEV WATCHDOG: eth1: transmit timed out

And card completely non-functional (doesn't see traffic at all and doesn't send anything). Probably some functions (like NAPI enabling) is has to be disabled first, and if interface come twice up, without going down, card come crazy.

--
Denys Fedoryshchenko
Technical Manager
Virtual ISP S.A.L.

#1 lore raw

Denys <[email protected]> :
[...]

Quoted 9 lines

After that i have.

Nov 15 22:11:37 vzone NETDEV WATCHDOG: eth1: transmit timed out
Nov 15 22:11:49 vzone NETDEV WATCHDOG: eth1: transmit timed out
Nov 15 22:12:01 vzone NETDEV WATCHDOG: eth1: transmit timed out
Nov 15 22:12:13 vzone NETDEV WATCHDOG: eth1: transmit timed out
Nov 15 22:12:25 vzone NETDEV WATCHDOG: eth1: transmit timed out
Nov 15 22:12:37 vzone NETDEV WATCHDOG: eth1: transmit timed out
Nov 15 22:12:49 vzone NETDEV WATCHDOG: eth1: transmit timed out
Nov 15 22:13:01 vzone NETDEV WATCHDOG: eth1: transmit timed out

Is there a chance for you to verify that the network interface does not recover if a real cable is plugged at this point ?

Signature · 1 line
Ueimor
author
#2 lore raw

Before it happens on 2.6.22, i tried to attach good cable, plug-unplug, whatever, interface up/down - card still remains dead.

I try to plug cable to laptop with rtl8139 and pc with e100 - it is worked when i shake bad cable(interface was going up/down as well), and good cable also worked fine, and never crashed. On 2.6.22 i was able to reproduce it easily, but later i can't cause it is server in internet cafe, and customers going crazy.

On Sat, 17 Nov 2007 23:07:59 +0100, Francois Romieu wrote

Quoted 12 lines

Denys <[email protected]> :
[...]

Quoted 9 lines

After that i have.

Nov 15 22:11:37 vzone NETDEV WATCHDOG: eth1: transmit timed out
Nov 15 22:11:49 vzone NETDEV WATCHDOG: eth1: transmit timed out
Nov 15 22:12:01 vzone NETDEV WATCHDOG: eth1: transmit timed out
Nov 15 22:12:13 vzone NETDEV WATCHDOG: eth1: transmit timed out
Nov 15 22:12:25 vzone NETDEV WATCHDOG: eth1: transmit timed out
Nov 15 22:12:37 vzone NETDEV WATCHDOG: eth1: transmit timed out
Nov 15 22:12:49 vzone NETDEV WATCHDOG: eth1: transmit timed out
Nov 15 22:13:01 vzone NETDEV WATCHDOG: eth1: transmit timed out
Quoted 5 lines

Is there a chance for you to verify that the network interface does not recover if a real cable is plugged at this point ?

Signature · 1 line
Ueimor

--
Denys Fedoryshchenko
Technical Manager
Virtual ISP S.A.L.

#3 lore raw

Denys <[email protected]> :

Quoted 3 lines

Before it happens on 2.6.22, i tried to attach good cable, plug-unplug, whatever, interface up/down - card still remains dead.

A few things have changed since 2.6.22 but I'll take that it is a real bug and that 2.6.23 would not recover either. Can you fill a PR at http://bugzilla.kernel.org, add a lspci -vvvxxxx and a dmesg ?

A dmesg including the watchdog message would be nice but if you can not retrieve one, a dmesg from your current 2.6.23.x kernel should be enough.

I'll take care of it once I am done with the sis190 watchdog failure (nothing hard, simply ugly).

[...]

Quoted 5 lines

I try to plug cable to laptop with rtl8139 and pc with e100 - it is worked when i shake bad cable(interface was going up/down as well), and good cable also worked fine, and never crashed. On 2.6.22 i was able to reproduce it easily, but later i can't cause it is server in internet cafe, and customers going crazy.

:o/

Signature · 1 line
Ueimor
author
#4 lore raw

Maybe it is good idea to print some registers/etc in watchdog message?

I will try to reproduce problem with debug enabled, maybe it will be possible to catch problem with more debug.

On Sun, 18 Nov 2007 22:00:38 +0100, Francois Romieu wrote

Quoted 9 lines

Denys <[email protected]> :

Quoted 3 lines

Before it happens on 2.6.22, i tried to attach good cable, plug-unplug, whatever, interface up/down - card still remains dead.

A few things have changed since 2.6.22 but I'll take that it is a real bug and that 2.6.23 would not recover either. Can you fill a PR at http://bugzilla.kernel.org, add a lspci -vvvxxxx and a dmesg ?

Quoted 9 lines

A dmesg including the watchdog message would be nice but if you can not retrieve one, a dmesg from your current 2.6.23.x kernel should be enough.

I'll take care of it once I am done with the sis190 watchdog failure (nothing hard, simply ugly).

[...]

Quoted 1 line

I try to plug cable to laptop with rtl8139 and pc with e100 - it is

worked

Quoted 1 line
Quoted 1 line

when i shake bad cable(interface was going up/down as well), and good

cable

Quoted 2 lines
Quoted 2 lines

also worked fine, and never crashed. On 2.6.22 i was able to reproduce it easily, but later i can't cause it is server in internet cafe, and

customers

Quoted 10 lines
Quoted 1 line

going crazy.

:o/

Signature · 5 lines
Ueimor
-
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
Denys Fedoryshchenko
Technical Manager
Virtual ISP S.A.L.

#5 lore raw

Denys <[email protected]> :

Quoted 4 lines

Maybe it is good idea to print some registers/etc in watchdog message?

I will try to reproduce problem with debug enabled, maybe it will be possible to catch problem with more debug.

Your customers will not necessarily like it but it could be useful to trace the Rx/Tx descriptor ring content and the irq status word.

If your make it dependent on tp->msg_enable (settable via 'ethtool -s ethX msglvl 65535), you can enable it at will and catch some bit of normal traffic just before you experiment with the magic cable.

Quoted 2 lines

On Sun, 18 Nov 2007 22:00:38 +0100, Francois Romieu wrote

Quoted 1 line

Denys <[email protected]> :

[...]

Quoted 2 lines
Quoted 2 lines

I'll take care of it once I am done with the sis190 watchdog failure (nothing hard, simply ugly).

Actually I lack a reliable indication that the Tx process is stuck. My hack seems able to recover but it can be quite slow to trigger. A contact at SiS would be welcome.

Signature · 1 line
Ueimor