Skip to contentSkip to conversation

fix ftgmac100 issues on aspeed soc

PATCH In review Dylan Hung 4 participants 14 messages Oct 19, 2020 – Mar 12, 2021

Subsystems (2)
lore
Share & tools
lore

Dylan Hung

author
#0 lore raw

cover fix ftgmac100 issues on aspeed soc

This patch series fixes the ftgmac100 mac hw issues on aspeed soc.

Fixes 52c0cae ("ftgmac100: Remove tx descriptor accessors")
Fixes 39bfab8 ("net: ftgmac100: Add support for DT phy-handle property")
Fixes 10cbd64 ("ftgmac100: Rework NAPI & interrupts handling")

Dylan Hung (4):
ftgmac100: Fix race issue on TX descriptor[0]
ftgmac100: Fix missing-poll issue
ftgmac100: Add a dummy read to ensure running sequence
ftgmac100: Restart MAC HW once

 drivers/net/ethernet/faraday/ftgmac100.c | 53 ++++++++++++++----------
 1 file changed, 32 insertions(+), 21 deletions(-)
Signature · 1 line
2.17.1
author
#1 lore raw

PATCH 1/4 ftgmac100: Fix race issue on TX descriptor[0]

These rules must be followed when accessing the TX descriptor:

1. A TX descriptor is "cleanable" only when its value is non-zero and the owner bit is set to "software"

2. A TX descriptor is "writable" only when its value is zero regardless the edotr mask.

Fixes 52c0cae87465 ("ftgmac100: Remove tx descriptor accessors")
---
 drivers/net/ethernet/faraday/ftgmac100.c | 10 ++++++++++
 1 file changed, 10 insertions(+)
drivers/net/ethernet/faraday/ftgmac100.c +10 -0
index 00024dd41147..7cacbe4aecb7 100644
@@ -647,6 +647,9 @@ static bool ftgmac100_tx_complete_packet(struct ftgmac100 *priv)
 	if (ctl_stat & FTGMAC100_TXDES0_TXDMA_OWN)
 		return false;
 
+	if ((ctl_stat & ~(priv->txdes0_edotr_mask)) == 0)
+		return false;
+
 	skb = priv->tx_skbs[pointer];
 	netdev->stats.tx_packets++;
 	netdev->stats.tx_bytes += skb->len;
@@ -756,6 +759,9 @@ static netdev_tx_t ftgmac100_hard_start_xmit(struct sk_buff *skb,
 	pointer = priv->tx_pointer;
 	txdes = first = &priv->txdes[pointer];
 
+	if (le32_to_cpu(txdes->txdes0) & ~priv->txdes0_edotr_mask)
+		goto drop;
+
 	/* Setup it up with the packet head. Don't write the head to the
 	 * ring just yet
 	 */
@@ -787,6 +793,10 @@ static netdev_tx_t ftgmac100_hard_start_xmit(struct sk_buff *skb,
 		/* Setup descriptor */
 		priv->tx_skbs[pointer] = skb;
 		txdes = &priv->txdes[pointer];
+
+		if (le32_to_cpu(txdes->txdes0) & ~priv->txdes0_edotr_mask)
+			goto dma_err;
+
 		ctl_stat = ftgmac100_base_tx_ctlstat(priv, pointer);
 		ctl_stat |= FTGMAC100_TXDES0_TXDMA_OWN;
 		ctl_stat |= FTGMAC100_TXDES0_TXBUF_SIZE(len);
Signature · 1 line
2.17.1
author
#6 lore raw

PATCH 2/4 ftgmac100: Fix missing-poll issue

In reply to Dylan Hung

the tx-poll command may advance the tx descriptor due the HW design. By adding a pseudo read and proper memory barrier, we can ensure all the data are ready before TX poll command.

Fixes 52c0cae87465 ("ftgmac100: Remove tx descriptor accessors")
---
 drivers/net/ethernet/faraday/ftgmac100.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/net/ethernet/faraday/ftgmac100.c +2 -2
index 7cacbe4aecb7..810bda80f138 100644
@@ -814,8 +814,8 @@ static netdev_tx_t ftgmac100_hard_start_xmit(struct sk_buff *skb,
 	 * before setting the OWN bit on the first descriptor.
 	 */
 	dma_wmb();
-	first->txdes0 = cpu_to_le32(f_ctl_stat);
-
+	WRITE_ONCE(first->txdes0, cpu_to_le32(f_ctl_stat));
+	READ_ONCE(first->txdes0);
 	/* Update next TX pointer */
 	priv->tx_pointer = pointer;
 
Signature · 1 line
2.17.1
author
#7 lore raw

PATCH 3/4 ftgmac100: Add a dummy read to ensure running sequence

In reply to Dylan Hung

On the AST2600 care must be taken to ensure writes appear correctly when modifying the interrupt reglated registers.

Add a function to perform a read after all writes to the IER and ISR registers.

Fixes 39bfab8844a0 ("net: ftgmac100: Add support for DT phy-handle property")
---
 drivers/net/ethernet/faraday/ftgmac100.c | 38 ++++++++++++------------
 1 file changed, 19 insertions(+), 19 deletions(-)
drivers/net/ethernet/faraday/ftgmac100.c +19 -19
index 810bda80f138..0c67fc3e27df 100644
@@ -111,6 +111,14 @@ struct ftgmac100 {
 	bool is_aspeed;
 };
 
+/* Helper to ensure writes are observed with the correct ordering. Use only
+ * for IER and ISR accesses. */
+static void ftgmac100_write(u32 val, void __iomem *addr)
+{
+	iowrite32(val, addr);
+	ioread32(addr);
+}
+
 static int ftgmac100_reset_mac(struct ftgmac100 *priv, u32 maccr)
 {
 	struct net_device *netdev = priv->netdev;
@@ -1048,7 +1056,7 @@ static void ftgmac100_adjust_link(struct net_device *netdev)
 		return;
 
 	/* Disable all interrupts */
-	iowrite32(0, priv->base + FTGMAC100_OFFSET_IER);
+	ftgmac100_write(0, priv->base + FTGMAC100_OFFSET_IER);
 
 	/* Reset the adapter asynchronously */
 	schedule_work(&priv->reset_task);
@@ -1246,7 +1254,7 @@ static irqreturn_t ftgmac100_interrupt(int irq, void *dev_id)
 
 	/* Fetch and clear interrupt bits, process abnormal ones */
 	status = ioread32(priv->base + FTGMAC100_OFFSET_ISR);
-	iowrite32(status, priv->base + FTGMAC100_OFFSET_ISR);
+	ftgmac100_write(status, priv->base + FTGMAC100_OFFSET_ISR);
 	if (unlikely(status & FTGMAC100_INT_BAD)) {
 
 		/* RX buffer unavailable */
@@ -1266,7 +1274,7 @@ static irqreturn_t ftgmac100_interrupt(int irq, void *dev_id)
 			if (net_ratelimit())
 				netdev_warn(netdev,
 					   "AHB bus error ! Resetting chip.\n");
-			iowrite32(0, priv->base + FTGMAC100_OFFSET_IER);
+			ftgmac100_write(0, priv->base + FTGMAC100_OFFSET_IER);
 			schedule_work(&priv->reset_task);
 			return IRQ_HANDLED;
 		}
@@ -1281,7 +1289,7 @@ static irqreturn_t ftgmac100_interrupt(int irq, void *dev_id)
 	}
 
 	/* Only enable "bad" interrupts while NAPI is on */
-	iowrite32(new_mask, priv->base + FTGMAC100_OFFSET_IER);
+	ftgmac100_write(new_mask, priv->base + FTGMAC100_OFFSET_IER);
 
 	/* Schedule NAPI bh */
 	napi_schedule_irqoff(&priv->napi);
@@ -1320,8 +1328,7 @@ static int ftgmac100_poll(struct napi_struct *napi, int budget)
 		ftgmac100_start_hw(priv);
 
 		/* Re-enable "bad" interrupts */
-		iowrite32(FTGMAC100_INT_BAD,
-			  priv->base + FTGMAC100_OFFSET_IER);
+		ftgmac100_write(FTGMAC100_INT_BAD, priv->base + FTGMAC100_OFFSET_IER);
 	}
 
 	/* As long as we are waiting for transmit packets to be
@@ -1336,13 +1343,7 @@ static int ftgmac100_poll(struct napi_struct *napi, int budget)
 		 * they were masked. So we clear them first, then we need
 		 * to re-check if there's something to process
 		 */
-		iowrite32(FTGMAC100_INT_RXTX,
-			  priv->base + FTGMAC100_OFFSET_ISR);
-
-		/* Push the above (and provides a barrier vs. subsequent
-		 * reads of the descriptor).
-		 */
-		ioread32(priv->base + FTGMAC100_OFFSET_ISR);
+		ftgmac100_write(FTGMAC100_INT_RXTX, priv->base + FTGMAC100_OFFSET_ISR);
 
 		/* Check RX and TX descriptors for more work to do */
 		if (ftgmac100_check_rx(priv) ||
@@ -1353,8 +1354,7 @@ static int ftgmac100_poll(struct napi_struct *napi, int budget)
 		napi_complete(napi);
 
 		/* enable all interrupts */
-		iowrite32(FTGMAC100_INT_ALL,
-			  priv->base + FTGMAC100_OFFSET_IER);
+		ftgmac100_write(FTGMAC100_INT_ALL, priv->base + FTGMAC100_OFFSET_IER);
 	}
 
 	return work_done;
@@ -1382,7 +1382,7 @@ static int ftgmac100_init_all(struct ftgmac100 *priv, bool ignore_alloc_err)
 	netif_start_queue(priv->netdev);
 
 	/* Enable all interrupts */
-	iowrite32(FTGMAC100_INT_ALL, priv->base + FTGMAC100_OFFSET_IER);
+	ftgmac100_write(FTGMAC100_INT_ALL, priv->base + FTGMAC100_OFFSET_IER);
 
 	return err;
 }
@@ -1508,7 +1508,7 @@ static int ftgmac100_open(struct net_device *netdev)
  err_irq:
 	netif_napi_del(&priv->napi);
  err_hw:
-	iowrite32(0, priv->base + FTGMAC100_OFFSET_IER);
+	ftgmac100_write(0, priv->base + FTGMAC100_OFFSET_IER);
 	ftgmac100_free_rings(priv);
 	return err;
 }
@@ -1526,7 +1526,7 @@ static int ftgmac100_stop(struct net_device *netdev)
 	 */
 
 	/* disable all interrupts */
-	iowrite32(0, priv->base + FTGMAC100_OFFSET_IER);
+	ftgmac100_write(0, priv->base + FTGMAC100_OFFSET_IER);
 
 	netif_stop_queue(netdev);
 	napi_disable(&priv->napi);
@@ -1549,7 +1549,7 @@ static void ftgmac100_tx_timeout(struct net_device *netdev, unsigned int txqueue
 	struct ftgmac100 *priv = netdev_priv(netdev);
 
 	/* Disable all interrupts */
-	iowrite32(0, priv->base + FTGMAC100_OFFSET_IER);
+	ftgmac100_write(0, priv->base + FTGMAC100_OFFSET_IER);
 
 	/* Do the reset outside of interrupt context */
 	schedule_work(&priv->reset_task);
Signature · 1 line
2.17.1
author
#9 lore raw

PATCH 4/4 ftgmac100: Restart MAC HW once

In reply to Dylan Hung

The interrupt handler may set the flag to reset the mac in the future, but that flag is not cleared once the reset has occured.

Fixes 10cbd6407609 ("ftgmac100: Rework NAPI & interrupts handling")
---
 drivers/net/ethernet/faraday/ftgmac100.c | 1 +
 1 file changed, 1 insertion(+)
drivers/net/ethernet/faraday/ftgmac100.c +1 -0
index 0c67fc3e27df..57736b049de3 100644
@@ -1326,6 +1326,7 @@ static int ftgmac100_poll(struct napi_struct *napi, int budget)
 	 */
 	if (unlikely(priv->need_mac_restart)) {
 		ftgmac100_start_hw(priv);
+		priv->need_mac_restart = false;
 
 		/* Re-enable "bad" interrupts */
 		ftgmac100_write(FTGMAC100_INT_BAD, priv->base + FTGMAC100_OFFSET_IER);
Signature · 1 line
2.17.1
#2 lore raw

Re: [PATCH 1/4] ftgmac100: Fix race issue on TX descriptor[0]

In reply to Dylan Hung
Quoted 4 lines · Dylan Hung

These rules must be followed when accessing the TX descriptor:

1. A TX descriptor is "cleanable" only when its value is non-zero and the owner bit is set to "software"

Can you elaborate ? What is the point of that change ? The owner bit should be sufficient, why do we need to check other fields ?

Quoted 2 lines

2. A TX descriptor is "writable" only when its value is zero regardless the edotr mask.

Again, why is that ? Can you elaborate ? What race are you trying to address here ?

Cheers,
Ben.

Quoted 48 lines
Fixes 52c0cae87465 ("ftgmac100: Remove tx descriptor accessors")
---
 drivers/net/ethernet/faraday/ftgmac100.c | 10 ++++++++++
 1 file changed, 10 insertions(+)
drivers/net/ethernet/faraday/ftgmac100.c +3 -0
index 00024dd41147..7cacbe4aecb7 100644
@@ -647,6 +647,9 @@ static bool ftgmac100_tx_complete_packet(struct
 ftgmac100 *priv)
 	if (ctl_stat & FTGMAC100_TXDES0_TXDMA_OWN)
 		return false;
 
+	if ((ctl_stat & ~(priv->txdes0_edotr_mask)) == 0)
+		return false;
+
 	skb = priv->tx_skbs[pointer];
 	netdev->stats.tx_packets++;
drivers/net/ethernet/faraday/ftgmac100.c +3 -0
 	netdev->stats.tx_bytes += skb->len;
@@ -756,6 +759,9 @@ static netdev_tx_t
 ftgmac100_hard_start_xmit(struct sk_buff *skb,
 	pointer = priv->tx_pointer;
 	txdes = first = &priv->txdes[pointer];
 
+	if (le32_to_cpu(txdes->txdes0) & ~priv->txdes0_edotr_mask)
+		goto drop;
+
 	/* Setup it up with the packet head. Don't write the head to

the

drivers/net/ethernet/faraday/ftgmac100.c +4 -0
 	 * ring just yet
 	 */
@@ -787,6 +793,10 @@ static netdev_tx_t
 ftgmac100_hard_start_xmit(struct sk_buff *skb,
 		/* Setup descriptor */
 		priv->tx_skbs[pointer] = skb;
 		txdes = &priv->txdes[pointer];
+
+		if (le32_to_cpu(txdes->txdes0) & ~priv-
 >txdes0_edotr_mask)
+			goto dma_err;
+
 		ctl_stat = ftgmac100_base_tx_ctlstat(priv, pointer);
drivers/net/ethernet/faraday/ftgmac100.c +0 -0
 		ctl_stat |= FTGMAC100_TXDES0_TXDMA_OWN;
 		ctl_stat |= FTGMAC100_TXDES0_TXBUF_SIZE(len);
#8 lore raw

Re: [PATCH 3/4] ftgmac100: Add a dummy read to ensure running sequence

In reply to Dylan Hung
Quoted 4 lines · Dylan Hung

On the AST2600 care must be taken to ensure writes appear correctly when modifying the interrupt reglated registers.

Add a function to perform a read after all writes to the IER and ISR registers.

You need to elaborate here. MMIO writes shouldn't get out of order, though they can get posted. I thus object to that "blanket" ftgmac100_write(). Instead, specifically add reads to flush posted writes where they are necessary and document it with a comment. Unless there's a deeper problem in the HW, in which case you need a better explanation.

Cheers,
Ben.

Quoted 133 lines
Fixes 39bfab8844a0 ("net: ftgmac100: Add support for DT phy-handle property")
---
 drivers/net/ethernet/faraday/ftgmac100.c | 38 ++++++++++++------------
 1 file changed, 19 insertions(+), 19 deletions(-)
drivers/net/ethernet/faraday/ftgmac100.c +19 -19
index 810bda80f138..0c67fc3e27df 100644
@@ -111,6 +111,14 @@ struct ftgmac100 {
 	bool is_aspeed;
 };
 
+/* Helper to ensure writes are observed with the correct ordering. Use only
+ * for IER and ISR accesses. */
+static void ftgmac100_write(u32 val, void __iomem *addr)
+{
+	iowrite32(val, addr);
+	ioread32(addr);
+}
+
 static int ftgmac100_reset_mac(struct ftgmac100 *priv, u32 maccr)
 {
 	struct net_device *netdev = priv->netdev;
@@ -1048,7 +1056,7 @@ static void ftgmac100_adjust_link(struct net_device *netdev)
 		return;
 
 	/* Disable all interrupts */
-	iowrite32(0, priv->base + FTGMAC100_OFFSET_IER);
+	ftgmac100_write(0, priv->base + FTGMAC100_OFFSET_IER);
 
 	/* Reset the adapter asynchronously */
 	schedule_work(&priv->reset_task);
@@ -1246,7 +1254,7 @@ static irqreturn_t ftgmac100_interrupt(int irq, void *dev_id)
 
 	/* Fetch and clear interrupt bits, process abnormal ones */
 	status = ioread32(priv->base + FTGMAC100_OFFSET_ISR);
-	iowrite32(status, priv->base + FTGMAC100_OFFSET_ISR);
+	ftgmac100_write(status, priv->base + FTGMAC100_OFFSET_ISR);
 	if (unlikely(status & FTGMAC100_INT_BAD)) {
 
 		/* RX buffer unavailable */
@@ -1266,7 +1274,7 @@ static irqreturn_t ftgmac100_interrupt(int irq, void *dev_id)
 			if (net_ratelimit())
 				netdev_warn(netdev,
 					   "AHB bus error ! Resetting chip.\n");
-			iowrite32(0, priv->base + FTGMAC100_OFFSET_IER);
+			ftgmac100_write(0, priv->base + FTGMAC100_OFFSET_IER);
 			schedule_work(&priv->reset_task);
 			return IRQ_HANDLED;
 		}
@@ -1281,7 +1289,7 @@ static irqreturn_t ftgmac100_interrupt(int irq, void *dev_id)
 	}
 
 	/* Only enable "bad" interrupts while NAPI is on */
-	iowrite32(new_mask, priv->base + FTGMAC100_OFFSET_IER);
+	ftgmac100_write(new_mask, priv->base + FTGMAC100_OFFSET_IER);
 
 	/* Schedule NAPI bh */
 	napi_schedule_irqoff(&priv->napi);
@@ -1320,8 +1328,7 @@ static int ftgmac100_poll(struct napi_struct *napi, int budget)
 		ftgmac100_start_hw(priv);
 
 		/* Re-enable "bad" interrupts */
-		iowrite32(FTGMAC100_INT_BAD,
-			  priv->base + FTGMAC100_OFFSET_IER);
+		ftgmac100_write(FTGMAC100_INT_BAD, priv->base + FTGMAC100_OFFSET_IER);
 	}
 
 	/* As long as we are waiting for transmit packets to be
@@ -1336,13 +1343,7 @@ static int ftgmac100_poll(struct napi_struct *napi, int budget)
 		 * they were masked. So we clear them first, then we need
 		 * to re-check if there's something to process
 		 */
-		iowrite32(FTGMAC100_INT_RXTX,
-			  priv->base + FTGMAC100_OFFSET_ISR);
-
-		/* Push the above (and provides a barrier vs. subsequent
-		 * reads of the descriptor).
-		 */
-		ioread32(priv->base + FTGMAC100_OFFSET_ISR);
+		ftgmac100_write(FTGMAC100_INT_RXTX, priv->base + FTGMAC100_OFFSET_ISR);
 
 		/* Check RX and TX descriptors for more work to do */
 		if (ftgmac100_check_rx(priv) ||
@@ -1353,8 +1354,7 @@ static int ftgmac100_poll(struct napi_struct *napi, int budget)
 		napi_complete(napi);
 
 		/* enable all interrupts */
-		iowrite32(FTGMAC100_INT_ALL,
-			  priv->base + FTGMAC100_OFFSET_IER);
+		ftgmac100_write(FTGMAC100_INT_ALL, priv->base + FTGMAC100_OFFSET_IER);
 	}
 
 	return work_done;
@@ -1382,7 +1382,7 @@ static int ftgmac100_init_all(struct ftgmac100 *priv, bool ignore_alloc_err)
 	netif_start_queue(priv->netdev);
 
 	/* Enable all interrupts */
-	iowrite32(FTGMAC100_INT_ALL, priv->base + FTGMAC100_OFFSET_IER);
+	ftgmac100_write(FTGMAC100_INT_ALL, priv->base + FTGMAC100_OFFSET_IER);
 
 	return err;
 }
@@ -1508,7 +1508,7 @@ static int ftgmac100_open(struct net_device *netdev)
  err_irq:
 	netif_napi_del(&priv->napi);
  err_hw:
-	iowrite32(0, priv->base + FTGMAC100_OFFSET_IER);
+	ftgmac100_write(0, priv->base + FTGMAC100_OFFSET_IER);
 	ftgmac100_free_rings(priv);
 	return err;
 }
@@ -1526,7 +1526,7 @@ static int ftgmac100_stop(struct net_device *netdev)
 	 */
 
 	/* disable all interrupts */
-	iowrite32(0, priv->base + FTGMAC100_OFFSET_IER);
+	ftgmac100_write(0, priv->base + FTGMAC100_OFFSET_IER);
 
 	netif_stop_queue(netdev);
 	napi_disable(&priv->napi);
@@ -1549,7 +1549,7 @@ static void ftgmac100_tx_timeout(struct net_device *netdev, unsigned int txqueue
 	struct ftgmac100 *priv = netdev_priv(netdev);
 
 	/* Disable all interrupts */
-	iowrite32(0, priv->base + FTGMAC100_OFFSET_IER);
+	ftgmac100_write(0, priv->base + FTGMAC100_OFFSET_IER);
 
 	/* Do the reset outside of interrupt context */
 	schedule_work(&priv->reset_task);
Acked-by on v1 4/4
#10 lore raw

Re: [PATCH 4/4] ftgmac100: Restart MAC HW once

In reply to Dylan Hung
Quoted 7 lines · Dylan Hung

The interrupt handler may set the flag to reset the mac in the future,
but that flag is not cleared once the reset has occured.

Fixes 10cbd6407609 ("ftgmac100: Rework NAPI & interrupts handling")
Quoted 19 lines
---
 drivers/net/ethernet/faraday/ftgmac100.c | 1 +
 1 file changed, 1 insertion(+)
drivers/net/ethernet/faraday/ftgmac100.c +1 -0
index 0c67fc3e27df..57736b049de3 100644
@@ -1326,6 +1326,7 @@ static int ftgmac100_poll(struct napi_struct
 *napi, int budget)
 	 */
 	if (unlikely(priv->need_mac_restart)) {
 		ftgmac100_start_hw(priv);
+		priv->need_mac_restart = false;
 
 		/* Re-enable "bad" interrupts */

ftgmac100_write(FTGMAC100_INT_BAD, priv->base +
FTGMAC100_OFFSET_IER);

#3 lore raw

Re: [PATCH 1/4] ftgmac100: Fix race issue on TX descriptor[0]

In reply to Benjamin Herrenschmidt
Quoted 9 lines · Benjamin Herrenschmidt
Quoted 4 lines · Dylan Hung

These rules must be followed when accessing the TX descriptor:

1. A TX descriptor is "cleanable" only when its value is non-zero and the owner bit is set to "software"

Can you elaborate ? What is the point of that change ? The owner bit should be sufficient, why do we need to check other fields ?

I would like Dylan to clarify too. The datasheet has a footnote below the descriptor layout:

- TXDES#0: Bits 27 ~ 14 are valid only when FTS = 1
- TXDES#1: Bits 31 ~ 0 are valid only when FTS = 1

So the ownership bit (31) is not valid unless FTS is set. However, this isn't what his patch does. It adds checks for EDOTR.

Quoted 59 lines
Quoted 3 lines

2. A TX descriptor is "writable" only when its value is zero regardless the edotr mask.

Again, why is that ? Can you elaborate ? What race are you trying to address here ?

Cheers,
Ben.

Quoted 48 lines
Fixes 52c0cae87465 ("ftgmac100: Remove tx descriptor accessors")
---
 drivers/net/ethernet/faraday/ftgmac100.c | 10 ++++++++++
 1 file changed, 10 insertions(+)
drivers/net/ethernet/faraday/ftgmac100.c +3 -0
index 00024dd41147..7cacbe4aecb7 100644
@@ -647,6 +647,9 @@ static bool ftgmac100_tx_complete_packet(struct
 ftgmac100 *priv)
      if (ctl_stat & FTGMAC100_TXDES0_TXDMA_OWN)
              return false;
 
+     if ((ctl_stat & ~(priv->txdes0_edotr_mask)) == 0)
+             return false;
+
      skb = priv->tx_skbs[pointer];
      netdev->stats.tx_packets++;
      netdev->stats.tx_bytes += skb->len;
drivers/net/ethernet/faraday/ftgmac100.c +3 -0
@@ -756,6 +759,9 @@ static netdev_tx_t
 ftgmac100_hard_start_xmit(struct sk_buff *skb,
      pointer = priv->tx_pointer;
      txdes = first = &priv->txdes[pointer];
 
+     if (le32_to_cpu(txdes->txdes0) & ~priv->txdes0_edotr_mask)
+             goto drop;
+
      /* Setup it up with the packet head. Don't write the head to

the
* ring just yet
*/

drivers/net/ethernet/faraday/ftgmac100.c +4 -0
@@ -787,6 +793,10 @@ static netdev_tx_t
 ftgmac100_hard_start_xmit(struct sk_buff *skb,
              /* Setup descriptor */
              priv->tx_skbs[pointer] = skb;
              txdes = &priv->txdes[pointer];
+
+             if (le32_to_cpu(txdes->txdes0) & ~priv-
 >txdes0_edotr_mask)
+                     goto dma_err;
+
              ctl_stat = ftgmac100_base_tx_ctlstat(priv, pointer);
              ctl_stat |= FTGMAC100_TXDES0_TXDMA_OWN;
              ctl_stat |= FTGMAC100_TXDES0_TXBUF_SIZE(len);
Reviewed-by on v1 4/4
#11 lore raw

Re: [PATCH 4/4] ftgmac100: Restart MAC HW once

In reply to Dylan Hung
Quoted 7 lines · Dylan Hung

The interrupt handler may set the flag to reset the mac in the future, but that flag is not cleared once the reset has occured.

Fixes 10cbd6407609 ("ftgmac100: Rework NAPI & interrupts handling")
Quoted 19 lines
---
 drivers/net/ethernet/faraday/ftgmac100.c | 1 +
 1 file changed, 1 insertion(+)
drivers/net/ethernet/faraday/ftgmac100.c +1 -1
index 0c67fc3e27df..57736b049de3 100644
@@ -1326,6 +1326,7 @@ static int ftgmac100_poll(struct napi_struct *napi, int budget)
         */
        if (unlikely(priv->need_mac_restart)) {
                ftgmac100_start_hw(priv);
+               priv->need_mac_restart = false;
 
                /* Re-enable "bad" interrupts */
                ftgmac100_write(FTGMAC100_INT_BAD, priv->base + FTGMAC100_OFFSET_IER);
--

2.17.1

#4 lore raw

Re: [PATCH 1/4] ftgmac100: Fix race issue on TX descriptor[0]

In reply to Joel Stanley
Quoted 13 lines · Joel Stanley
Quoted 10 lines · Benjamin Herrenschmidt
Quoted 4 lines · Dylan Hung

These rules must be followed when accessing the TX descriptor:

1. A TX descriptor is "cleanable" only when its value is non-zero and the owner bit is set to "software"

Can you elaborate ? What is the point of that change ? The owner bit
should be sufficient, why do we need to check other fields ?

Quoted 8 lines

I would like Dylan to clarify too. The datasheet has a footnote below the descriptor layout:

- TXDES#0: Bits 27 ~ 14 are valid only when FTS = 1
- TXDES#1: Bits 31 ~ 0 are valid only when FTS = 1

So the ownership bit (31) is not valid unless FTS is set. However, this isn't what his patch does. It adds checks for EDOTR.

No I think it adds a check for everything except EDOTR which just marks the end of ring and needs to be ignored in the comparison.

That said, we do need a better explanation.

One potential bug I did find by looking at my code however is:

static bool ftgmac100_tx_complete_packet(struct ftgmac100 *priv)

{
	struct net_device *netdev = priv->netdev;
	struct ftgmac100_txdes *txdes;
	struct sk_buff *skb;
	unsigned int pointer;
	u32 ctl_stat;

	pointer = priv->tx_clean_pointer;
	txdes = &priv->txdes[pointer];

	ctl_stat = le32_to_cpu(txdes->txdes0);
	if (ctl_stat & FTGMAC100_TXDES0_TXDMA_OWN)
		return false;

	skb = priv->tx_skbs[pointer];
	netdev->stats.tx_packets++;
	netdev->stats.tx_bytes += skb->len;
	ftgmac100_free_tx_packet(priv, pointer, skb, txdes, ctl_stat);
	txdes->txdes0 = cpu_to_le32(ctl_stat & priv->txdes0_edotr_mask);

^^^^ There should probably be an smp_wmb() here to ensure that all the above
stores are visible before the tx clean pointer is updated.

	priv->tx_clean_pointer = ftgmac100_next_tx_pointer(priv, pointer);

	return true;
}

Similarly we probablu should have one before setting tx_pointer in start_xmit().

As for the read side of this, I'm not 100% sure, I'll have to think more about it, it *think* the existing barriers are sufficient at first sight.

Cheers,
Ben.

Quoted 63 lines
Quoted 63 lines
Quoted 3 lines

2. A TX descriptor is "writable" only when its value is zero regardless the edotr mask.

Again, why is that ? Can you elaborate ? What race are you trying to
address here ?

Cheers,
Ben.

Quoted 52 lines
Fixes 52c0cae87465 ("ftgmac100: Remove tx descriptor accessors")
---
 drivers/net/ethernet/faraday/ftgmac100.c | 10 ++++++++++
 1 file changed, 10 insertions(+)
drivers/net/ethernet/faraday/ftgmac100.c +0 -0
index 00024dd41147..7cacbe4aecb7 100644
@@ -647,6 +647,9 @@ static bool

ftgmac100_tx_complete_packet(struct
ftgmac100 *priv)

      if (ctl_stat & FTGMAC100_TXDES0_TXDMA_OWN)
              return false;
drivers/net/ethernet/faraday/ftgmac100.c +6 -0
+     if ((ctl_stat & ~(priv->txdes0_edotr_mask)) == 0)
+             return false;
+
      skb = priv->tx_skbs[pointer];
      netdev->stats.tx_packets++;
      netdev->stats.tx_bytes += skb->len;
@@ -756,6 +759,9 @@ static netdev_tx_t
 ftgmac100_hard_start_xmit(struct sk_buff *skb,
      pointer = priv->tx_pointer;
      txdes = first = &priv->txdes[pointer];
 
+     if (le32_to_cpu(txdes->txdes0) & ~priv->txdes0_edotr_mask)
+             goto drop;
+
      /* Setup it up with the packet head. Don't write the head

to
the
* ring just yet
*/

drivers/net/ethernet/faraday/ftgmac100.c +4 -0
@@ -787,6 +793,10 @@ static netdev_tx_t
 ftgmac100_hard_start_xmit(struct sk_buff *skb,
              /* Setup descriptor */
              priv->tx_skbs[pointer] = skb;
              txdes = &priv->txdes[pointer];
+
+             if (le32_to_cpu(txdes->txdes0) & ~priv-
 > txdes0_edotr_mask)
 
+                     goto dma_err;
+

ctl_stat = ftgmac100_base_tx_ctlstat(priv,

pointer);
              ctl_stat |= FTGMAC100_TXDES0_TXDMA_OWN;
              ctl_stat |= FTGMAC100_TXDES0_TXBUF_SIZE(len);
#5 lore raw

Re: [PATCH 1/4] ftgmac100: Fix race issue on TX descriptor[0]

Quoted 18 lines · Benjamin Herrenschmidt
Quoted 16 lines · Joel Stanley
Quoted 10 lines · Benjamin Herrenschmidt
Quoted 4 lines · Dylan Hung

These rules must be followed when accessing the TX descriptor:

1. A TX descriptor is "cleanable" only when its value is non-zero and the owner bit is set to "software"

Can you elaborate ? What is the point of that change ? The owner bit
should be sufficient, why do we need to check other fields ?

I would like Dylan to clarify too. The datasheet has a footnote below the descriptor layout:

Quoted 8 lines
Quoted 5 lines

- TXDES#0: Bits 27 ~ 14 are valid only when FTS = 1
- TXDES#1: Bits 31 ~ 0 are valid only when FTS = 1

So the ownership bit (31) is not valid unless FTS is set. However, this isn't what his patch does. It adds checks for EDOTR.

No I think it adds a check for everything except EDOTR which just marks the end of ring and needs to be ignored in the comparison.

Of course. I missed the invert.

I did some testing with just this patch (and "[4/4] ftgmac100: Restart MAC HW once") from Dylan. It seemed to resolve the hang, but there were occasional retries. Putting in some tracing I only hit the condition in ftgmac100_tx_complete_packet, never in ftgmac100_hard_start_xmit.

Quoted 28 lines

That said, we do need a better explanation.

One potential bug I did find by looking at my code however is:

static bool ftgmac100_tx_complete_packet(struct ftgmac100 *priv)

{
        struct net_device *netdev = priv->netdev;
        struct ftgmac100_txdes *txdes;
        struct sk_buff *skb;
        unsigned int pointer;
        u32 ctl_stat;

        pointer = priv->tx_clean_pointer;
        txdes = &priv->txdes[pointer];

        ctl_stat = le32_to_cpu(txdes->txdes0);
        if (ctl_stat & FTGMAC100_TXDES0_TXDMA_OWN)
                return false;

        skb = priv->tx_skbs[pointer];
        netdev->stats.tx_packets++;
        netdev->stats.tx_bytes += skb->len;
        ftgmac100_free_tx_packet(priv, pointer, skb, txdes, ctl_stat);
        txdes->txdes0 = cpu_to_le32(ctl_stat & priv->txdes0_edotr_mask);

^^^^ There should probably be an smp_wmb() here to ensure that all the above
stores are visible before the tx clean pointer is updated.

Quoted 6 lines
        priv->tx_clean_pointer = ftgmac100_next_tx_pointer(priv, pointer);

        return true;
}

Similarly we probablu should have one before setting tx_pointer in start_xmit().

I added the two smp_wmb you suggested (with only 4/4 applied). This did the trick; iperf on a gigabit link is running well with no retries.

(quoted diff) +0 -0

b/drivers/net/ethernet/faraday/ftgmac100.c
index 331d4bdd4a67..15cdfeb135b0 100644

drivers/net/ethernet/faraday/ftgmac100.c +0 -0
@@ -653,6 +653,11 @@ static bool ftgmac100_tx_complete_packet(struct

ftgmac100 *priv)

     ftgmac100_free_tx_packet(priv, pointer, skb, txdes, ctl_stat);
     txdes->txdes0 = cpu_to_le32(ctl_stat & priv->txdes0_edotr_mask);

+ /* Ensure the descriptor config is visible before setting the tx
+ * pointer.
+ */
+ smp_wmb();
+

     priv->tx_clean_pointer = ftgmac100_next_tx_pointer(priv, pointer);

     return true;
@@ -806,6 +811,11 @@ static netdev_tx_t
ftgmac100_hard_start_xmit(struct sk_buff *skb,
     dma_wmb();
     first->txdes0 = cpu_to_le32(f_ctl_stat);

+ /* Ensure the descriptor config is visible before setting the tx
+ * pointer.
+ */
+ smp_wmb();
+

     /* Update next TX pointer */
     priv->tx_pointer = pointer;

I left the test running while writing this email and I did start to see some retries. I'm not sure if that's because my laptop is one of the test machines, or if we have another issue.

I will do some further testing over night.

Cheers,

Joel

Quoted 71 lines

As for the read side of this, I'm not 100% sure, I'll have to think more about it, it *think* the existing barriers are sufficient at first sight.

Cheers,
Ben.

Quoted 63 lines
Quoted 63 lines
Quoted 3 lines

2. A TX descriptor is "writable" only when its value is zero regardless the edotr mask.

Again, why is that ? Can you elaborate ? What race are you trying to
address here ?

Cheers,
Ben.

Quoted 52 lines
Fixes 52c0cae87465 ("ftgmac100: Remove tx descriptor accessors")
---
 drivers/net/ethernet/faraday/ftgmac100.c | 10 ++++++++++
 1 file changed, 10 insertions(+)
drivers/net/ethernet/faraday/ftgmac100.c +0 -0
index 00024dd41147..7cacbe4aecb7 100644
@@ -647,6 +647,9 @@ static bool

ftgmac100_tx_complete_packet(struct
ftgmac100 *priv)

      if (ctl_stat & FTGMAC100_TXDES0_TXDMA_OWN)
              return false;
drivers/net/ethernet/faraday/ftgmac100.c +6 -0
+     if ((ctl_stat & ~(priv->txdes0_edotr_mask)) == 0)
+             return false;
+
      skb = priv->tx_skbs[pointer];
      netdev->stats.tx_packets++;
      netdev->stats.tx_bytes += skb->len;
@@ -756,6 +759,9 @@ static netdev_tx_t
 ftgmac100_hard_start_xmit(struct sk_buff *skb,
      pointer = priv->tx_pointer;
      txdes = first = &priv->txdes[pointer];
 
+     if (le32_to_cpu(txdes->txdes0) & ~priv->txdes0_edotr_mask)
+             goto drop;
+
      /* Setup it up with the packet head. Don't write the head

to
the
* ring just yet
*/

drivers/net/ethernet/faraday/ftgmac100.c +4 -0
@@ -787,6 +793,10 @@ static netdev_tx_t
 ftgmac100_hard_start_xmit(struct sk_buff *skb,
              /* Setup descriptor */
              priv->tx_skbs[pointer] = skb;
              txdes = &priv->txdes[pointer];
+
+             if (le32_to_cpu(txdes->txdes0) & ~priv-
 > txdes0_edotr_mask)
 
+                     goto dma_err;
+

ctl_stat = ftgmac100_base_tx_ctlstat(priv,

pointer);
              ctl_stat |= FTGMAC100_TXDES0_TXDMA_OWN;
              ctl_stat |= FTGMAC100_TXDES0_TXBUF_SIZE(len);
#12 lore raw

Re: [PATCH 4/4] ftgmac100: Restart MAC HW once

In reply to Joel Stanley
Quoted 11 lines · Joel Stanley
Quoted 7 lines · Dylan Hung

The interrupt handler may set the flag to reset the mac in the future, but that flag is not cleared once the reset has occured.

Fixes 10cbd6407609 ("ftgmac100: Rework NAPI & interrupts handling")

net maintainers, this one never made it into the tree. Do you need me to re-send it?

Cheers,

Joel

Quoted 20 lines
Quoted 20 lines
---
 drivers/net/ethernet/faraday/ftgmac100.c | 1 +
 1 file changed, 1 insertion(+)
drivers/net/ethernet/faraday/ftgmac100.c +1 -1
index 0c67fc3e27df..57736b049de3 100644
@@ -1326,6 +1326,7 @@ static int ftgmac100_poll(struct napi_struct *napi, int budget)
         */
        if (unlikely(priv->need_mac_restart)) {
                ftgmac100_start_hw(priv);
+               priv->need_mac_restart = false;
 
                /* Re-enable "bad" interrupts */
                ftgmac100_write(FTGMAC100_INT_BAD, priv->base + FTGMAC100_OFFSET_IER);
--

2.17.1

maintainer
#13 lore raw

Re: [PATCH 4/4] ftgmac100: Restart MAC HW once

From: Joel Stanley <[email protected]>
Date: Fri, 12 Mar 2021 00:26:43 +0000

Quoted 7 lines · Joel Stanley
Quoted 6 lines · Dylan Hung
Quoted 4 lines

The interrupt handler may set the flag to reset the mac in the future, but that flag is not cleared once the reset has occured.

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
Fixes 10cbd6407609 ("ftgmac100: Rework NAPI & interrupts handling")

net maintainers, this one never made it into the tree. Do you need me to re-send it?

If it has been this long, definitely you need to resend.