r8169 crash
Subsystems (2)
Hi
Recently i start to use r8169 cards and faced some issues.
How to reproduce (kind of difficult, and i cannot that on my live server anymore):
1)Badly crimped cable.
2)PCI-Express R8169 card
Kernel 2.6.23.1
05:01.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL-8169 Gigabit Ethernet (rev 10)
Subsystem: Realtek Semiconductor Co., Ltd. RTL-8169 Gigabit Ethernet
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr-
Stepping- SERR- FastB2B-
Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort-
<TAbort- <MAbort- >SERR- <PERR-
Latency: 64 (8000ns min, 16000ns max), Cache Line Size: 32 bytes
Interrupt: pin A routed to IRQ 21
Region 0: I/O ports at 1000 [size=256]
Region 1: Memory at 55001000 (32-bit, non-prefetchable) [size=256] Expansion ROM at fffe0000 [disabled] [size=128K]
Capabilities: [dc] Power Management version 2
Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=375mAPME(D0-,D1+,D2+,D3hot+,D3cold+)
Status: D0 PME-Enable- DSel=0 DScale=0 PME-
r8169 Gigabit Ethernet driver 2.2LK-NAPI loaded
eth1: RTL8169sb/8110sb at 0xf881c000, 00:19:e0:72:49:88, XID 10000000 IRQ 21
To reproduce issue i need to shake faulty cable. Card will report:
Nov 15 22:09:36 vzone r8169: eth1: link up
So it is going up without going down!
Normally it is like this:
Nov 1 00:50:33 vzone r8169: eth1: link down Nov 1 00:50:35 vzone r8169: eth1: link up
After that i have.
Nov 15 22:11:37 vzone NETDEV WATCHDOG: eth1: transmit timed out Nov 15 22:11:49 vzone NETDEV WATCHDOG: eth1: transmit timed out Nov 15 22:12:01 vzone NETDEV WATCHDOG: eth1: transmit timed out Nov 15 22:12:13 vzone NETDEV WATCHDOG: eth1: transmit timed out Nov 15 22:12:25 vzone NETDEV WATCHDOG: eth1: transmit timed out Nov 15 22:12:37 vzone NETDEV WATCHDOG: eth1: transmit timed out Nov 15 22:12:49 vzone NETDEV WATCHDOG: eth1: transmit timed out Nov 15 22:13:01 vzone NETDEV WATCHDOG: eth1: transmit timed out
And card completely non-functional (doesn't see traffic at all and doesn't send anything). Probably some functions (like NAPI enabling) is has to be disabled first, and if interface come twice up, without going down, card come crazy.
--
Denys Fedoryshchenko
Technical Manager
Virtual ISP S.A.L.
Denys <[email protected]> :
[...]
Quoted 9 lines
After that i have.
Nov 15 22:11:37 vzone NETDEV WATCHDOG: eth1: transmit timed out Nov 15 22:11:49 vzone NETDEV WATCHDOG: eth1: transmit timed out Nov 15 22:12:01 vzone NETDEV WATCHDOG: eth1: transmit timed out Nov 15 22:12:13 vzone NETDEV WATCHDOG: eth1: transmit timed out Nov 15 22:12:25 vzone NETDEV WATCHDOG: eth1: transmit timed out Nov 15 22:12:37 vzone NETDEV WATCHDOG: eth1: transmit timed out Nov 15 22:12:49 vzone NETDEV WATCHDOG: eth1: transmit timed out Nov 15 22:13:01 vzone NETDEV WATCHDOG: eth1: transmit timed out
Is there a chance for you to verify that the network interface does not recover if a real cable is plugged at this point ?
Signature · 1 line
Ueimor
Before it happens on 2.6.22, i tried to attach good cable, plug-unplug, whatever, interface up/down - card still remains dead.
I try to plug cable to laptop with rtl8139 and pc with e100 - it is worked when i shake bad cable(interface was going up/down as well), and good cable also worked fine, and never crashed. On 2.6.22 i was able to reproduce it easily, but later i can't cause it is server in internet cafe, and customers going crazy.
On Sat, 17 Nov 2007 23:07:59 +0100, Francois Romieu wrote
Quoted 12 lines
Denys <[email protected]> :
[...]Quoted 9 lines
After that i have.
Nov 15 22:11:37 vzone NETDEV WATCHDOG: eth1: transmit timed out Nov 15 22:11:49 vzone NETDEV WATCHDOG: eth1: transmit timed out Nov 15 22:12:01 vzone NETDEV WATCHDOG: eth1: transmit timed out Nov 15 22:12:13 vzone NETDEV WATCHDOG: eth1: transmit timed out Nov 15 22:12:25 vzone NETDEV WATCHDOG: eth1: transmit timed out Nov 15 22:12:37 vzone NETDEV WATCHDOG: eth1: transmit timed out Nov 15 22:12:49 vzone NETDEV WATCHDOG: eth1: transmit timed out Nov 15 22:13:01 vzone NETDEV WATCHDOG: eth1: transmit timed out
Quoted 5 lines
Is there a chance for you to verify that the network interface does not recover if a real cable is plugged at this point ?
Signature · 1 line
Ueimor
--
Denys Fedoryshchenko
Technical Manager
Virtual ISP S.A.L.
Denys <[email protected]> :
Quoted 3 lines
Before it happens on 2.6.22, i tried to attach good cable, plug-unplug, whatever, interface up/down - card still remains dead.
A few things have changed since 2.6.22 but I'll take that it is a real bug and that 2.6.23 would not recover either. Can you fill a PR at http://bugzilla.kernel.org, add a lspci -vvvxxxx and a dmesg ?
A dmesg including the watchdog message would be nice but if you can not retrieve one, a dmesg from your current 2.6.23.x kernel should be enough.
I'll take care of it once I am done with the sis190 watchdog failure (nothing hard, simply ugly).
[...]
Quoted 5 lines
I try to plug cable to laptop with rtl8139 and pc with e100 - it is worked when i shake bad cable(interface was going up/down as well), and good cable also worked fine, and never crashed. On 2.6.22 i was able to reproduce it easily, but later i can't cause it is server in internet cafe, and customers going crazy.
:o/
Signature · 1 line
Ueimor
Maybe it is good idea to print some registers/etc in watchdog message?
I will try to reproduce problem with debug enabled, maybe it will be possible to catch problem with more debug.
On Sun, 18 Nov 2007 22:00:38 +0100, Francois Romieu wrote
Quoted 9 lines
Denys <[email protected]> :
Quoted 3 lines
Before it happens on 2.6.22, i tried to attach good cable, plug-unplug, whatever, interface up/down - card still remains dead.
A few things have changed since 2.6.22 but I'll take that it is a real bug and that 2.6.23 would not recover either. Can you fill a PR at http://bugzilla.kernel.org, add a lspci -vvvxxxx and a dmesg ?
Quoted 9 lines
A dmesg including the watchdog message would be nice but if you can not retrieve one, a dmesg from your current 2.6.23.x kernel should be enough.
I'll take care of it once I am done with the sis190 watchdog failure (nothing hard, simply ugly).
[...]
Quoted 1 line
I try to plug cable to laptop with rtl8139 and pc with e100 - it is
worked
Quoted 1 line
Quoted 1 line
when i shake bad cable(interface was going up/down as well), and good
cable
Quoted 2 lines
Quoted 2 lines
also worked fine, and never crashed. On 2.6.22 i was able to reproduce it easily, but later i can't cause it is server in internet cafe, and
customers
Quoted 10 lines
Quoted 1 line
going crazy.
:o/
Signature · 5 lines
Ueimor - To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
--
Denys Fedoryshchenko
Technical Manager
Virtual ISP S.A.L.
Denys <[email protected]> :
Quoted 4 lines
Maybe it is good idea to print some registers/etc in watchdog message?
I will try to reproduce problem with debug enabled, maybe it will be possible to catch problem with more debug.
Your customers will not necessarily like it but it could be useful to trace the Rx/Tx descriptor ring content and the irq status word.
If your make it dependent on tp->msg_enable (settable via 'ethtool -s ethX msglvl 65535), you can enable it at will and catch some bit of normal traffic just before you experiment with the magic cable.
Quoted 2 lines
On Sun, 18 Nov 2007 22:00:38 +0100, Francois Romieu wrote
Quoted 1 line
Denys <[email protected]> :
[...]
Quoted 2 lines
Quoted 2 lines
I'll take care of it once I am done with the sis190 watchdog failure (nothing hard, simply ugly).
Actually I lack a reliable indication that the Tx process is stuck. My hack seems able to recover but it can be quite slow to trigger. A contact at SiS would be welcome.
Signature · 1 line
Ueimor