Skip to contentSkip to conversation

ucc_geth: Fix hangs after switching from full to half duplex

PATCH Applied Anton Vorontsov 3 participants 7 messages Sep 10, 2009 – Sep 11, 2009

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 3/3 ucc_geth: Fix hangs after switching from full to half duplex

MPC8360 QE UCC ethernet controllers hang when changing link duplex under a load (a bit of NFS activity is enough).

PHY: mdio@e0102120:00 - Link is Up - 1000/Full sh-3.00# ethtool -s eth0 speed 100 duplex half autoneg off
PHY: mdio@e0102120:00 - Link is Down
PHY: mdio@e0102120:00 - Link is Up - 100/Half NETDEV WATCHDOG: eth0 (ucc_geth): transmit queue 0 timed out

  ------------[ cut here ]------------
  Badness at c01fcbd0 [verbose debug info unavailable]
  NIP: c01fcbd0 LR: c01fcbd0 CTR: c0194e44
  ...

The cure is to disable the controller before changing speed/duplex and enable it afterwards.

---
 drivers/net/ucc_geth.c |    4 ++++
 1 files changed, 4 insertions(+), 0 deletions(-)
drivers/net/ucc_geth.c +4 -0
index 2a2c973..9ad9015 100644
@@ -1631,9 +1631,13 @@ static void adjust_link(struct net_device *dev)
 			ugeth->oldspeed = phydev->speed;
 		}
 
+		ugeth_disable(ugeth, COMM_DIR_RX_AND_TX);
+
 		out_be32(&ug_regs->maccfg2, tempval);
 		out_be32(&uf_regs->upsmr, upsmr);
 
+		ugeth_enable(ugeth, COMM_DIR_RX_AND_TX);
+
 		if (!ugeth->oldlink) {
 			new_state = 1;
 			ugeth->oldlink = 1;
Signature · 1 line
1.6.3.3
#1 lore raw

PATCH v2 3/3 ucc_geth: Fix hangs after switching from full to half duplex

MPC8360 QE UCC ethernet controllers hang when changing link duplex under a load (a bit of NFS activity is enough).

PHY: mdio@e0102120:00 - Link is Up - 1000/Full sh-3.00# ethtool -s eth0 speed 100 duplex half autoneg off
PHY: mdio@e0102120:00 - Link is Down
PHY: mdio@e0102120:00 - Link is Up - 100/Half NETDEV WATCHDOG: eth0 (ucc_geth): transmit queue 0 timed out

  ------------[ cut here ]------------
  Badness at c01fcbd0 [verbose debug info unavailable]
  NIP: c01fcbd0 LR: c01fcbd0 CTR: c0194e44
  ...

The cure is to disable the controller before changing speed/duplex and enable it afterwards.

Since ugeth_graceful_stop_{tx,rx} now may be called from an atomic context, switch the two functions from msleep() to mdelay().

---

v2: Switch ugeth_graceful_stop_{tx,rx} to mdelay. Noticed with
CONFIG_DEBUG_SPINLOCK_SLEEP.

 drivers/net/ucc_geth.c |    8 ++++++--
 1 files changed, 6 insertions(+), 2 deletions(-)
drivers/net/ucc_geth.c +6 -2
index 2a2c973..5a0803f 100644
@@ -1432,7 +1432,7 @@ static int ugeth_graceful_stop_tx(struct ucc_geth_private *ugeth)
 
 	/* Wait for command to complete */
 	do {
-		msleep(10);
+		mdelay(10);
 		temp = in_be32(uccf->p_ucce);
 	} while (!(temp & UCC_GETH_UCCE_GRA) && --i);
 
@@ -1464,7 +1464,7 @@ static int ugeth_graceful_stop_rx(struct ucc_geth_private *ugeth)
 						ucc_num);
 		qe_issue_cmd(QE_GRACEFUL_STOP_RX, cecr_subblock,
 			     QE_CR_PROTOCOL_ETHERNET, 0);
-		msleep(10);
+		mdelay(10);
 		temp = in_8(&ugeth->p_rx_glbl_pram->rxgstpack);
 	} while (!(temp & GRACEFUL_STOP_ACKNOWLEDGE_RX) && --i);
 
@@ -1631,9 +1631,13 @@ static void adjust_link(struct net_device *dev)
 			ugeth->oldspeed = phydev->speed;
 		}
 
+		ugeth_disable(ugeth, COMM_DIR_RX_AND_TX);
+
 		out_be32(&ug_regs->maccfg2, tempval);
 		out_be32(&uf_regs->upsmr, upsmr);
 
+		ugeth_enable(ugeth, COMM_DIR_RX_AND_TX);
+
 		if (!ugeth->oldlink) {
 			new_state = 1;
 			ugeth->oldlink = 1;
Signature · 1 line
1.6.3.3
#2 lore raw
Quoted 13 lines · Anton Vorontsov

MPC8360 QE UCC ethernet controllers hang when changing link duplex under a load (a bit of NFS activity is enough).

PHY: mdio@e0102120:00 - Link is Up - 1000/Full sh-3.00# ethtool -s eth0 speed 100 duplex half autoneg off
PHY: mdio@e0102120:00 - Link is Down
PHY: mdio@e0102120:00 - Link is Up - 100/Half NETDEV WATCHDOG: eth0 (ucc_geth): transmit queue 0 timed out

  ------------[ cut here ]------------
  Badness at c01fcbd0 [verbose debug info unavailable]
  NIP: c01fcbd0 LR: c01fcbd0 CTR: c0194e44
  ...
Quoted 5 lines

The cure is to disable the controller before changing speed/duplex and enable it afterwards.

Since ugeth_graceful_stop_{tx,rx} now may be called from an atomic context, switch the two functions from msleep() to mdelay().

Ouch. Can we put this in a workqueue or something?

-Scott

#3 lore raw
Quoted 14 lines · Scott Wood
Quoted 13 lines · Anton Vorontsov

MPC8360 QE UCC ethernet controllers hang when changing link duplex under a load (a bit of NFS activity is enough).

PHY: mdio@e0102120:00 - Link is Up - 1000/Full sh-3.00# ethtool -s eth0 speed 100 duplex half autoneg off
PHY: mdio@e0102120:00 - Link is Down
PHY: mdio@e0102120:00 - Link is Up - 100/Half NETDEV WATCHDOG: eth0 (ucc_geth): transmit queue 0 timed out

 ------------[ cut here ]------------
 Badness at c01fcbd0 [verbose debug info unavailable]
 NIP: c01fcbd0 LR: c01fcbd0 CTR: c0194e44
 ...
Quoted 7 lines
Quoted 5 lines

The cure is to disable the controller before changing speed/duplex and enable it afterwards.

Since ugeth_graceful_stop_{tx,rx} now may be called from an atomic context, switch the two functions from msleep() to mdelay().

Ouch.

Yeah, right... delaying for 10ms with irqs off isn't good.

Quoted 1 line

Can we put this in a workqueue or something?

adjust_link() itself isn't called from an atomic context.

It's we are grabbing ugeth->lock, i.e. a spinlock. I don't see why the lock is needed in adjust_link() in its current form, but if we're going to disable the controller for some time, we'll have to make sure that no start_xmit() or NAPI is running, scheduled or will be scheduled until we say so.

I think that lock-less, and thus completely sleep-able variant of adjust_link is doable.

Thanks,

Signature · 3 lines
Anton Vorontsov
email: [email protected]
irc://irc.freenode.net/bd2
#4 lore raw
Quoted 21 lines · Anton Vorontsov
Quoted 20 lines · Scott Wood
Quoted 18 lines · Anton Vorontsov

MPC8360 QE UCC ethernet controllers hang when changing link duplex under a load (a bit of NFS activity is enough).

PHY: mdio@e0102120:00 - Link is Up - 1000/Full sh-3.00# ethtool -s eth0 speed 100 duplex half autoneg off
PHY: mdio@e0102120:00 - Link is Down
PHY: mdio@e0102120:00 - Link is Up - 100/Half NETDEV WATCHDOG: eth0 (ucc_geth): transmit queue 0 timed out

 ------------[ cut here ]------------
 Badness at c01fcbd0 [verbose debug info unavailable]
 NIP: c01fcbd0 LR: c01fcbd0 CTR: c0194e44
 ...

The cure is to disable the controller before changing speed/duplex and enable it afterwards.

Since ugeth_graceful_stop_{tx,rx} now may be called from an atomic context, switch the two functions from msleep() to mdelay().

Quoted 7 lines
Quoted 1 line

Ouch.

Yeah, right... delaying for 10ms with irqs off isn't good.

Quoted 1 line

Can we put this in a workqueue or something?

adjust_link() itself isn't called from an atomic context.

Oops. I though that phylib calls us from a workqueue, not a timer. Hm...

Will be a little bit more work..

Signature · 3 lines
Anton Vorontsov
email: [email protected]
irc://irc.freenode.net/bd2
#5 lore raw

PATCH v3 3/3 ucc_geth: Fix hangs after switching from full to half duplex

MPC8360 QE UCC ethernet controllers hang when changing link duplex under a load (a bit of NFS activity is enough).

PHY: mdio@e0102120:00 - Link is Up - 1000/Full sh-3.00# ethtool -s eth0 speed 100 duplex half autoneg off
PHY: mdio@e0102120:00 - Link is Down
PHY: mdio@e0102120:00 - Link is Up - 100/Half NETDEV WATCHDOG: eth0 (ucc_geth): transmit queue 0 timed out

  ------------[ cut here ]------------
  Badness at c01fcbd0 [verbose debug info unavailable]
  NIP: c01fcbd0 LR: c01fcbd0 CTR: c0194e44
  ...

The cure is to disable the controller before changing speed/duplex and enable it afterwards.

Though, disabling the controller might take quite a while, so we better not grab any spinlocks in adjust_link(). Instead, we quiesce the driver's activity, and only then disable the controller.

---

Quoted 26 lines · Anton Vorontsov
Quoted 25 lines · Anton Vorontsov
Quoted 14 lines · Scott Wood
Quoted 13 lines · Anton Vorontsov

MPC8360 QE UCC ethernet controllers hang when changing link duplex under a load (a bit of NFS activity is enough).

PHY: mdio@e0102120:00 - Link is Up - 1000/Full sh-3.00# ethtool -s eth0 speed 100 duplex half autoneg off
PHY: mdio@e0102120:00 - Link is Down
PHY: mdio@e0102120:00 - Link is Up - 100/Half NETDEV WATCHDOG: eth0 (ucc_geth): transmit queue 0 timed out

 ------------[ cut here ]------------
 Badness at c01fcbd0 [verbose debug info unavailable]
 NIP: c01fcbd0 LR: c01fcbd0 CTR: c0194e44
 ...
Quoted 7 lines
Quoted 5 lines

The cure is to disable the controller before changing speed/duplex and enable it afterwards.

Since ugeth_graceful_stop_{tx,rx} now may be called from an atomic context, switch the two functions from msleep() to mdelay().

Ouch.

Yeah, right... delaying for 10ms with irqs off isn't good.

Quoted 7 lines
Quoted 3 lines
Quoted 1 line

Can we put this in a workqueue or something?

adjust_link() itself isn't called from an atomic context.

Oops. I though that phylib calls us from a workqueue, not a timer. Hm...

Will be a little bit more work..

Ignore me. I'm working on two kernel versions in parallel (2.6.21 and mainline), and it's 2.6.21 where phylib uses a timer. Mainline is OK.

How about this patch?

 drivers/net/ucc_geth.c |   36 +++++++++++++++++++++++++++++++-----
 1 files changed, 31 insertions(+), 5 deletions(-)
drivers/net/ucc_geth.c +31 -5
index 2a2c973..232fef9 100644
@@ -1560,6 +1560,25 @@ static int ugeth_disable(struct ucc_geth_private *ugeth, enum comm_dir mode)
 	return 0;
 }
 
+static void ugeth_quiesce(struct ucc_geth_private *ugeth)
+{
+	/* Wait for and prevent any further xmits. */
+	netif_tx_disable(ugeth->ndev);
+
+	/* Disable the interrupt to avoid NAPI rescheduling. */
+	disable_irq(ugeth->ug_info->uf_info.irq);
+
+	/* Stop NAPI, and possibly wait for its completion. */
+	napi_disable(&ugeth->napi);
+}
+
+static void ugeth_activate(struct ucc_geth_private *ugeth)
+{
+	napi_enable(&ugeth->napi);
+	enable_irq(ugeth->ug_info->uf_info.irq);
+	netif_tx_wake_all_queues(ugeth->ndev);
+}
+
 /* Called every time the controller might need to be made
  * aware of new link state.  The PHY code conveys this
  * information through variables in the ugeth structure, and this
@@ -1573,14 +1592,11 @@ static void adjust_link(struct net_device *dev)
 	struct ucc_geth __iomem *ug_regs;
 	struct ucc_fast __iomem *uf_regs;
 	struct phy_device *phydev = ugeth->phydev;
-	unsigned long flags;
 	int new_state = 0;
 
 	ug_regs = ugeth->ug_regs;
 	uf_regs = ugeth->uccf->uf_regs;
 
-	spin_lock_irqsave(&ugeth->lock, flags);
-
 	if (phydev->link) {
 		u32 tempval = in_be32(&ug_regs->maccfg2);
 		u32 upsmr = in_be32(&uf_regs->upsmr);
@@ -1631,9 +1647,21 @@ static void adjust_link(struct net_device *dev)
 			ugeth->oldspeed = phydev->speed;
 		}
 
+		/*
+		 * To change the MAC configuration we need to disable the
+		 * controller. To do so, we have to either grab ugeth->lock,
+		 * which is a bad idea since 'graceful stop' commands might
+		 * take quite a while, or we can quiesce driver's activity.
+		 */
+		ugeth_quiesce(ugeth);
+		ugeth_disable(ugeth, COMM_DIR_RX_AND_TX);
+
 		out_be32(&ug_regs->maccfg2, tempval);
 		out_be32(&uf_regs->upsmr, upsmr);
 
+		ugeth_enable(ugeth, COMM_DIR_RX_AND_TX);
+		ugeth_activate(ugeth);
+
 		if (!ugeth->oldlink) {
 			new_state = 1;
 			ugeth->oldlink = 1;
@@ -1647,8 +1675,6 @@ static void adjust_link(struct net_device *dev)
 
 	if (new_state && netif_msg_link(ugeth))
 		phy_print_status(phydev);
-
-	spin_unlock_irqrestore(&ugeth->lock, flags);
 }
 
 /* Initialize TBI PHY interface for communicating with the
Signature · 1 line
1.6.3.3
maintainer Applied
#6 lore raw

From: Anton Vorontsov <[email protected]>
Date: Fri, 11 Sep 2009 01:48:12 +0400

Quoted 13 lines

MPC8360 QE UCC ethernet controllers hang when changing link duplex under a load (a bit of NFS activity is enough).

PHY: mdio@e0102120:00 - Link is Up - 1000/Full sh-3.00# ethtool -s eth0 speed 100 duplex half autoneg off
PHY: mdio@e0102120:00 - Link is Down
PHY: mdio@e0102120:00 - Link is Up - 100/Half NETDEV WATCHDOG: eth0 (ucc_geth): transmit queue 0 timed out

  ------------[ cut here ]------------
  Badness at c01fcbd0 [verbose debug info unavailable]
  NIP: c01fcbd0 LR: c01fcbd0 CTR: c0194e44
  ...
Quoted 8 lines

The cure is to disable the controller before changing speed/duplex and enable it afterwards.

Though, disabling the controller might take quite a while, so we better not grab any spinlocks in adjust_link(). Instead, we quiesce the driver's activity, and only then disable the controller.

Applied.