2.6.3 - 8139too timeout debug info
Firstly, I am starting a fresh thread here, as I have subscribed to the lkml now so I can reply in a timely manner, and also the other thread was broke as I was replying 'off-list' - sorry about that.
OK, to recap. With 2.6.3 I get timeouts with 8139too under network load (any load). I have never had any problems with these _same_ two cards under any other kernel version over the last 3 years.
If I use the 8139too.c from 2.6.2, and build 2.6.3 with it, all works fine (I am running this version right now).
Here is the debuggered output (using #define RTL8139_NDEBUG 1):
#> /var/log/messages
Feb 27 17:12:27 Linux233 kernel: NETDEV WATCHDOG: eth0: transmit timed out Feb 27 17:12:39 Linux233 kernel: NETDEV WATCHDOG: eth0: transmit timed out Feb 27 17:12:42 Linux233 kernel: nfs: server 486Linux not responding, still trying Feb 27 17:12:43 Linux233 kernel: nfs: server 486Linux not responding, still trying Feb 27 17:12:51 Linux233 kernel: NETDEV WATCHDOG: eth0: transmit timed out Feb 27 17:12:51 Linux233 kernel: nfs: server 486Linux OK
Feb 27 17:12:51 Linux233 kernel: nfs: server 486Linux OK Feb 27 17:13:03 Linux233 kernel: nfs: server 486Linux not responding, still trying Feb 27 17:13:03 Linux233 kernel: NETDEV WATCHDOG: eth0: transmit timed out Feb 27 17:13:03 Linux233 kernel: nfs: server 486Linux OK
#> dmesg
NETDEV WATCHDOG: eth0: transmit timed out eth0: Tx queue start entry 1314 dirty entry 1310. eth0: Tx descriptor 0 is 00002000. eth0: Tx descriptor 1 is 00002000. eth0: Tx descriptor 2 is 00002000. (queue head) eth0: Tx descriptor 3 is 00002000. NETDEV WATCHDOG: eth0: transmit timed out eth0: Tx queue start entry 1392 dirty entry 1388. eth0: Tx descriptor 0 is 00002000. (queue head) eth0: Tx descriptor 1 is 00002000. eth0: Tx descriptor 2 is 00002000. eth0: Tx descriptor 3 is 00002000. nfs: server 486Linux not responding, still trying nfs: server 486Linux not responding, still trying NETDEV WATCHDOG: eth0: transmit timed out eth0: Tx queue start entry 363 dirty entry 359. eth0: Tx descriptor 0 is 00002000. eth0: Tx descriptor 1 is 00002000. eth0: Tx descriptor 2 is 00002000. eth0: Tx descriptor 3 is 00002000. (queue head) nfs: server 486Linux OK nfs: server 486Linux OK nfs: server 486Linux not responding, still trying NETDEV WATCHDOG: eth0: transmit timed out eth0: Tx queue start entry 3441 dirty entry 3437. eth0: Tx descriptor 0 is 00002000. eth0: Tx descriptor 1 is 00002000. (queue head) eth0: Tx descriptor 2 is 00002000. eth0: Tx descriptor 3 is 00002000. nfs: server 486Linux OK
#> lspci
00:00.0 Host bridge: VIA Technologies, Inc. VT82C585VP [Apollo VP1/VPX] (rev 23)
00:07.0 ISA bridge: VIA Technologies, Inc. VT82C586/A/B PCI-to-ISA [Apollo VP] (rev 27)
00:07.1 IDE interface: VIA Technologies, Inc. Bus Master IDE (rev 06) 00:07.2 USB Controller: VIA Technologies, Inc. UHCI USB (rev 02) 00:09.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL-8139 (rev 10)
00:0a.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL-8139 (rev 10)
00:0b.0 VGA compatible controller: Matrox Graphics, Inc. MGA 2164W [Millennium II]
Built with:
# CONFIG_8139CP is not setCONFIG_8139TOO=y
# CONFIG_8139TOO_PIO is not set
# CONFIG_8139TOO_TUNE_TWISTER is not set
# CONFIG_8139TOO_8129 is not set
# CONFIG_8139_OLD_RX_RESET is not setCONFIG_8139_RXBUF_IDX=2
Any more info required?
TIA,
Nick
Signature · 3 lines
"When you're chewing on life's gristle, Don't grumble, Give a whistle And this'll help things turn out for the best."
If your using acpi , boot up with either
acpi=ht
or
acpi=nopci
Matt H.
Quoted 97 lines · Nick Warne
Firstly, I am starting a fresh thread here, as I have subscribed to the lkml now so I can reply in a timely manner, and also the other thread was broke as I was replying 'off-list' - sorry about that.
OK, to recap. With 2.6.3 I get timeouts with 8139too under network load (any load). I have never had any problems with these _same_ two cards under any other kernel version over the last 3 years.
If I use the
8139too.cfrom 2.6.2, and build 2.6.3 with it, all works fine (I am running this version right now).Here is the debuggered output (using #define
RTL8139_NDEBUG1):#>
/var/log/messagesFeb 27 17:12:27 Linux233 kernel: NETDEV WATCHDOG: eth0: transmit timed out Feb 27 17:12:39 Linux233 kernel: NETDEV WATCHDOG: eth0: transmit timed out Feb 27 17:12:42 Linux233 kernel: nfs: server 486Linux not responding, still trying Feb 27 17:12:43 Linux233 kernel: nfs: server 486Linux not responding, still trying Feb 27 17:12:51 Linux233 kernel: NETDEV WATCHDOG: eth0: transmit timed out Feb 27 17:12:51 Linux233 kernel: nfs: server 486Linux OKFeb 27 17:12:51 Linux233 kernel: nfs: server 486Linux OK Feb 27 17:13:03 Linux233 kernel: nfs: server 486Linux not responding, still trying Feb 27 17:13:03 Linux233 kernel: NETDEV WATCHDOG: eth0: transmit timed out Feb 27 17:13:03 Linux233 kernel: nfs: server 486Linux OK#> dmesg
NETDEV WATCHDOG: eth0: transmit timed out eth0: Tx queue start entry 1314 dirty entry 1310. eth0: Tx descriptor 0 is 00002000. eth0: Tx descriptor 1 is 00002000. eth0: Tx descriptor 2 is 00002000. (queue head) eth0: Tx descriptor 3 is 00002000. NETDEV WATCHDOG: eth0: transmit timed out eth0: Tx queue start entry 1392 dirty entry 1388. eth0: Tx descriptor 0 is 00002000. (queue head) eth0: Tx descriptor 1 is 00002000. eth0: Tx descriptor 2 is 00002000. eth0: Tx descriptor 3 is 00002000. nfs: server 486Linux not responding, still trying nfs: server 486Linux not responding, still trying NETDEV WATCHDOG: eth0: transmit timed out eth0: Tx queue start entry 363 dirty entry 359. eth0: Tx descriptor 0 is 00002000. eth0: Tx descriptor 1 is 00002000. eth0: Tx descriptor 2 is 00002000. eth0: Tx descriptor 3 is 00002000. (queue head) nfs: server 486Linux OK nfs: server 486Linux OK nfs: server 486Linux not responding, still trying NETDEV WATCHDOG: eth0: transmit timed out eth0: Tx queue start entry 3441 dirty entry 3437. eth0: Tx descriptor 0 is 00002000. eth0: Tx descriptor 1 is 00002000. (queue head) eth0: Tx descriptor 2 is 00002000. eth0: Tx descriptor 3 is 00002000. nfs: server 486Linux OK#> lspci
00:00.0 Host bridge: VIA Technologies, Inc. VT82C585VP [Apollo VP1/VPX] (rev 23)
00:07.0 ISA bridge: VIA Technologies, Inc. VT82C586/A/B PCI-to-ISA [Apollo VP] (rev 27)
00:07.1 IDE interface: VIA Technologies, Inc. Bus Master IDE (rev 06) 00:07.2 USB Controller: VIA Technologies, Inc. UHCI USB (rev 02) 00:09.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL-8139 (rev 10)
00:0a.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL-8139 (rev 10)
00:0b.0 VGA compatible controller: Matrox Graphics, Inc. MGA 2164W [Millennium II]Built with:
#
CONFIG_8139CPis not setCONFIG_8139TOO=y
#CONFIG_8139TOO_PIOis not set
#CONFIG_8139TOO_TUNE_TWISTERis not set
#CONFIG_8139TOO_8129is not set
#CONFIG_8139_OLD_RX_RESETis not setCONFIG_8139_RXBUF_IDX=2Any more info required?
TIA,
Nick
Quoted 7 lines
If your using acpi , boot up with either
acpi=ht
or
acpi=nopciMatt H.
I do not have apci compiled.
# Power management options (ACPI, APM)
#
# CONFIG_PM is not set
#
# ACPI (Advanced Configuration and Power Interface) Support #
# CONFIG_ACPI is not set
Also I forgot to mention, 'noapic' makes no difference either.
Nick
Signature · 4 lines
"When you're chewing on life's gristle, Don't grumble, Give a whistle And this'll help things turn out for the best."
Quoted 10 lines · Nick Warne
Firstly, I am starting a fresh thread here, as I have subscribed to the lkml now so I can reply in a timely manner, and also the other thread was broke as I was replying 'off-list' - sorry about that.
OK, to recap. With 2.6.3 I get timeouts with 8139too under network load (any load). I have never had any problems with these _same_ two cards under any other kernel version over the last 3 years.
If I use the
8139too.cfrom 2.6.2, and build 2.6.3 with it, all works fine (I am running this version right now).
Interesting.
Please try the attached patch for debugging. After this problem happen, send the all output of dmesg, all .config, and "cat /proc/interrupts".
Thanks.
Signature · 1 line
OGAWA Hirofumi <[email protected]>
---
drivers/net/8139too.c | 56 ++++++++++++++++++++++++++++++++++++++++++++++++++ 1 files changed, 56 insertions(+)
drivers/net/8139too.c renamed from drivers/net/8139too.c~8139too-debug +56 -0
@@ -718,6 +718,25 @@ static const unsigned int rtl8139_rx_constatic const unsigned int rtl8139_tx_config = (TX_DMA_BURST << TxDMAShift) | (TX_RETRY << TxRetryShift); +#define RTL8139_DUMP (dev) do { \ + printk("%s: %s: run %d, stop %d\n", dev->name, __FUNCTION__, \ + netif_running(dev), netif_queue_stopped(dev)); \ + rtl8139_dump(dev->priv); \ +} while(0) + +static void rtl8139_dump (struct rtl8139_private *tp) +{ +void *ioaddr = tp->mmio_addr; +int i; + +for (i =0 ; i < tp->regs_len; i++) { +if (i && (i %16 ) ==0 ) +printk ("\n" ); +printk ("%02x " ,RTL_R8 (i)); + } +printk ("\n" ); +} +static void __rtl8139_cleanup_dev (struct net_device *dev) {struct rtl8139_private *tp;
@@ -1355,6 +1374,10 @@ static int rtl8139_open (struct net_devirtl8139_start_thread (dev); +spin_lock_irq (&tp->lock); +RTL8139_DUMP (dev); +spin_unlock_irq (&tp->lock); +return 0 ; }
@@ -1673,6 +1696,11 @@ static void rtl8139_tx_timeout (struct nu8 tmp8;unsigned long flags; +printk ("%s: irq %d\n" , __FUNCTION__,irqs_disabled ()); +spin_lock_irq (&tp->lock); +RTL8139_DUMP (dev); +spin_unlock_irq (&tp->lock); + DPRINTK ("%s: Transmit timeout, status %2.2x %4.4x " "media %2.2x.\n", dev->name, RTL_R8 (ChipCmd),
@@ -1737,6 +1765,12 @@ static int rtl8139_start_xmit (struct sk }spin_lock_irq (&tp->lock); + { +u32 txstatus =RTL_R32 (TxStatus0 + (entry *sizeof (u32))); +if (!(txstatus & TxStatOK) && (txstatus & TxUnderrun)) +printk ("8139too: durning re-transmit (%08x)\n" , + txstatus); + }RTL_W32_F (TxStatus0 + (entry *sizeof (u32)), tp->tx_flag |max (len, (unsigned int )ETH_ZLEN ));
@@ -1780,6 +1814,7 @@ static void rtl8139_tx_interrupt (struct/* Note: TxCarrierLost is always asserted at 100mbps. */ if (txstatus & (TxOutOfWindow | TxAborted)) { +RTL8139_DUMP (dev);/* There was an major error, log it. */ if (netif_msg_tx_err (tp)) printk(KERN_DEBUG "%s: Transmit error, Tx status %8.8x.\n",
@@ -1836,6 +1871,9 @@ static void rtl8139_rx_err (u32 rx_statu#ifdef CONFIG_8139_OLD_RX_RESET int tmp_work;#endif +spin_lock_irq (&tp->lock); +RTL8139_DUMP (dev); +spin_unlock_irq (&tp->lock);if (netif_msg_rx_err (tp)) printk(KERN_DEBUG "%s: Ethernet frame had errors, status %8.8x.\n",
@@ -2061,6 +2099,8 @@ static void rtl8139_weird_interrupt (strassert (tp !=NULL );assert (ioaddr !=NULL ); +RTL8139_DUMP (dev); +/* Update the error count. */ tp->stats.rx_missed_errors +=RTL_R32 (RxMissed);RTL_W32 (RxMissed,0 );
@@ -2209,6 +2249,10 @@ static int rtl8139_close (struct net_devint ret =0 ;unsigned long flags; +spin_lock_irq (&tp->lock); +RTL8139_DUMP (dev); +spin_unlock_irq (&tp->lock); +netif_stop_queue (dev);if (tp->thr_pid >=0 ) {
@@ -2271,6 +2315,7 @@ static void rtl8139_get_wol(struct net_dvoid *ioaddr = np->mmio_addr;spin_lock_irq (&np->lock); +RTL8139_DUMP (dev);if (rtl_chip_info[np->chipset].flags & HasLWake) {u8 cfg3 =RTL_R8 (Config3);u8 cfg5 =RTL_R8 (Config5);
@@ -2314,6 +2359,7 @@ static int rtl8139_set_wol(struct net_dereturn -EINVAL ;spin_lock_irq (&np->lock); +RTL8139_DUMP (dev); cfg3 =RTL_R8 (Config3) & ~(Cfg3_LinkUp | Cfg3_Magic);if (wol->wolopts &WAKE_PHY ) cfg3 |= Cfg3_LinkUp;
@@ -2352,6 +2398,7 @@ static int rtl8139_get_settings(struct n {struct rtl8139_private *np = dev->priv;spin_lock_irq (&np->lock); +RTL8139_DUMP (dev);mii_ethtool_gset (&np->mii, cmd);spin_unlock_irq (&np->lock);return 0 ;
@@ -2362,6 +2409,7 @@ static int rtl8139_set_settings(struct nstruct rtl8139_private *np = dev->priv;int rc;spin_lock_irq (&np->lock); +RTL8139_DUMP (dev); rc =mii_ethtool_sset (&np->mii, cmd);spin_unlock_irq (&np->lock);return rc;
@@ -2370,12 +2418,14 @@ static int rtl8139_set_settings(struct nstatic int rtl8139_nway_reset (struct net_device *dev) {struct rtl8139_private *np = dev->priv; +RTL8139_DUMP (dev);return mii_nway_restart (&np->mii); }static u32 rtl8139_get_link (struct net_device *dev) {struct rtl8139_private *np = dev->priv; +RTL8139_DUMP (dev);return mii_link_ok (&np->mii); }
@@ -2409,6 +2459,7 @@ static void rtl8139_get_regs(struct net_ regs->version =RTL_REGS_VER ;spin_lock_irq (&np->lock); +RTL8139_DUMP (dev);memcpy_fromio (regbuf, np->mmio_addr, regs->len);spin_unlock_irq (&np->lock); }
@@ -2554,6 +2605,10 @@ static int rtl8139_suspend (struct pci_dvoid *ioaddr = tp->mmio_addr;unsigned long flags; +spin_lock_irqsave (&tp->lock, flags); +RTL8139_DUMP (dev); +spin_unlock_irqrestore (&tp->lock, flags); +if (!netif_running (dev))return 0 ;
@@ -2583,6 +2638,7 @@ static int rtl8139_resume (struct pci_destruct net_device *dev =pci_get_drvdata (pdev);struct rtl8139_private *tp = dev->priv; +RTL8139_DUMP (dev);if (!netif_running (dev))return 0 ;pci_restore_state (pdev, tp->pci_state);
_
Quoted 11 lines
Quoted 2 lines
If I use the
8139too.cfrom 2.6.2, and build 2.6.3 with it, all works fine (I am running this version right now).Interesting.
Please try the attached patch for debugging. After this problem happen, send the all output of dmesg, all .config, and "cat
/proc/interrupts".Thanks.
Signature · 1 line
OGAWA Hirofumi <[email protected]>
Thanks for your help. I have hell of a trouble doing this, as soon as any network load happens, the box becomes unresponsive during timeouts - but hopefully I have caught the info required.
http://www.linicks.net/8139too_debug/
Thanks,
Nick
Signature · 2 lines
"When you're chewing on life's gristle, Don't grumble, Give a whistle..."
Quoted 3 lines · Nick Warne
Thanks for your help. I have hell of a trouble doing this, as soon as any network load happens, the box becomes unresponsive during timeouts - but hopefully I have caught the info required.
Umm.. Looks like chip registers is normal, but TX/RX interrupt doesn't happen. (BTW, there isn't rtl8139_open on debuginfo.txt. Was it already scrolled?)
The following patch (incremental patch) is some part reverts to 2.6.2. Is behavior changed?
Please try 8139too-revert01.patch, and 8139too-revert02.patch.
Signature · 1 line
OGAWA Hirofumi <[email protected]>
---
drivers/net/8139too.c | 13 +++++-------- 1 files changed, 5 insertions(+), 8 deletions(-)
drivers/net/8139too.c renamed from drivers/net/8139too.c~8139too-revert01 +5 -8
@@ -2043,12 +2043,10 @@ static int rtl8139_rx(struct net_deviceskb_put (skb, pkt_size); skb->protocol =eth_type_trans (skb, dev); - +netif_rx (skb); dev->last_rx = jiffies; tp->stats.rx_bytes += pkt_size; tp->stats.rx_packets++; - -netif_receive_skb (skb); }else {if (net_ratelimit ()) printk (KERN_WARNING
@@ -2204,11 +2202,10 @@ static irqreturn_t rtl8139_interrupt (in/* Receive packets are processed by poll routine. If not running start it now. */ -if (status & RxAckBits){ -if (netif_rx_schedule_prep (dev)) { -RTL_W16_F (IntrMask, rtl8139_norx_intr_mask); -__netif_rx_schedule (dev); - } +if (status & RxAckBits) { +RTL_W16_F (IntrMask, rtl8139_norx_intr_mask); +rtl8139_rx (dev, tp, dev->weight); +RTL_W16_F (IntrMask, rtl8139_intr_mask); }/* Check uncommon events with one test. */
_
---
drivers/net/8139too.c | 6 ++---- 1 files changed, 2 insertions(+), 4 deletions(-)
drivers/net/8139too.c renamed from drivers/net/8139too.c~8139too-revert02 +2 -4
@@ -1374,6 +1374,7 @@ static int rtl8139_open (struct net_devirtl8139_start_thread (dev); +printk ("%s: revert02\n" , dev->name);spin_lock_irq (&tp->lock);RTL8139_DUMP (dev);spin_unlock_irq (&tp->lock);
@@ -2202,11 +2203,8 @@ static irqreturn_t rtl8139_interrupt (in/* Receive packets are processed by poll routine. If not running start it now. */ -if (status & RxAckBits) { -RTL_W16_F (IntrMask, rtl8139_norx_intr_mask); +if (status & RxAckBits)rtl8139_rx (dev, tp, dev->weight); -RTL_W16_F (IntrMask, rtl8139_intr_mask); - }/* Check uncommon events with one test. */ if (unlikely (status & (PCIErr | PCSTimeout | RxUnderrun | RxErr)))
_
Quoted 6 lines · OGAWA Hirofumi
Quoted 3 lines · Nick Warne
Thanks for your help. I have hell of a trouble doing this, as soon as any network load happens, the box becomes unresponsive during timeouts - but hopefully I have caught the info required.
Quoted 6 lines
Umm.. Looks like chip registers is normal, but TX/RX interrupt doesn't happen. (BTW, there isn't
rtl8139_openon debuginfo.txt. Was it already scrolled?)The following patch (incremental patch) is some part reverts to 2.6.2. Is behavior changed?
Oops, wrong patches. Please try these.
Signature · 1 line
OGAWA Hirofumi <[email protected]>
---
drivers/net/8139too.c | 14 ++++++-------- 1 files changed, 6 insertions(+), 8 deletions(-)
drivers/net/8139too.c renamed from drivers/net/8139too.c~8139too-revert01 +6 -8
@@ -1374,6 +1374,7 @@ static int rtl8139_open (struct net_devirtl8139_start_thread (dev); +printk ("%s: revert01\n" , dev->name);spin_lock_irq (&tp->lock);RTL8139_DUMP (dev);spin_unlock_irq (&tp->lock);
@@ -2043,12 +2044,10 @@ static int rtl8139_rx(struct net_deviceskb_put (skb, pkt_size); skb->protocol =eth_type_trans (skb, dev); - +netif_rx (skb); dev->last_rx = jiffies; tp->stats.rx_bytes += pkt_size; tp->stats.rx_packets++; - -netif_receive_skb (skb); }else {if (net_ratelimit ()) printk (KERN_WARNING
@@ -2204,11 +2203,10 @@ static irqreturn_t rtl8139_interrupt (in/* Receive packets are processed by poll routine. If not running start it now. */ -if (status & RxAckBits){ -if (netif_rx_schedule_prep (dev)) { -RTL_W16_F (IntrMask, rtl8139_norx_intr_mask); -__netif_rx_schedule (dev); - } +if (status & RxAckBits) { +RTL_W16_F (IntrMask, rtl8139_norx_intr_mask); +rtl8139_rx (dev, tp, dev->weight); +RTL_W16_F (IntrMask, rtl8139_intr_mask); }/* Check uncommon events with one test. */
_
---
drivers/net/8139too.c | 7 ++----- 1 files changed, 2 insertions(+), 5 deletions(-)
drivers/net/8139too.c renamed from drivers/net/8139too.c~8139too-revert02 +2 -5
@@ -1374,7 +1374,7 @@ static int rtl8139_open (struct net_devirtl8139_start_thread (dev); -printk ("%s: revert01\n" , dev->name); +printk ("%s: revert02\n" , dev->name);spin_lock_irq (&tp->lock);RTL8139_DUMP (dev);spin_unlock_irq (&tp->lock);
@@ -2203,11 +2203,8 @@ static irqreturn_t rtl8139_interrupt (in/* Receive packets are processed by poll routine. If not running start it now. */ -if (status & RxAckBits) { -RTL_W16_F (IntrMask, rtl8139_norx_intr_mask); +if (status & RxAckBits)rtl8139_rx (dev, tp, dev->weight); -RTL_W16_F (IntrMask, rtl8139_intr_mask); - }/* Check uncommon events with one test. */ if (unlikely (status & (PCIErr | PCSTimeout | RxUnderrun | RxErr)))
_
Quoted 8 lines
Quoted 6 lines
Umm.. Looks like chip registers is normal, but TX/RX interrupt doesn't happen. (BTW, there isn't
rtl8139_openon debuginfo.txt. Was it already scrolled?)The following patch (incremental patch) is some part reverts to 2.6.2. Is behavior changed?
Oops, wrong patches. Please try these.
OK, I applied patch01 first - same problems. Then I applied patch02 _over_ patch01. Still same results.
I am not sure if you meant me to apply over the first debug patch you sent? Here, I just applied to standard 2.6.3 > 8139too.c file with RTL8139_NDEBUG 1 set.
With these patched dmesg gets flooded - and I have real trouble trying to do anything on the box with the timeout issues.
http://www.linicks.net/8139too_debug/patch01_02_info.txt
Thanks,
Nick
Signature · 2 lines
"When you're chewing on life's gristle, Don't grumble, Give a whistle..."