Skip to contentSkip to conversation

Fix Aspeed ast2600 MAC TX hang

PATCH Applied Dylan Hung 3 participants 8 messages Oct 14, 2020 – Oct 16, 2020

Subsystems (2)
lore
Share & tools
lore

Dylan Hung

author
#0 lore raw

cover Fix Aspeed ast2600 MAC TX hang

This patch fixes the TX hang issue on Aspeed AST2600.
Two HW arbitration features are added onto ast2600, but these features will cause MAC TX to hang when handling scatter-gather DMA. These two problematic features can be disabled by setting MAC register 0x58 bit28 and bit27.

Dylan Hung (1):
net: ftgmac100: Fix Aspeed ast2600 TX hang issue

 drivers/net/ethernet/faraday/ftgmac100.c | 5 +++++
 drivers/net/ethernet/faraday/ftgmac100.h | 8 ++++++++
 2 files changed, 13 insertions(+)
Signature · 1 line
2.17.1
author
#1 lore raw

PATCH 1/1 net: ftgmac100: Fix Aspeed ast2600 TX hang issue

The new HW arbitration feature on Aspeed ast2600 will cause MAC TX to hang when handling scatter-gather DMA. Disable the problematic feature by setting MAC register 0x58 bit28 and bit27.

---
 drivers/net/ethernet/faraday/ftgmac100.c | 5 +++++
 drivers/net/ethernet/faraday/ftgmac100.h | 8 ++++++++
 2 files changed, 13 insertions(+)
drivers/net/ethernet/faraday/ftgmac100.c +5 -0
index 87236206366f..00024dd41147 100644
@@ -1817,6 +1817,11 @@ static int ftgmac100_probe(struct platform_device *pdev)
 		priv->rxdes0_edorr_mask = BIT(30);
 		priv->txdes0_edotr_mask = BIT(30);
 		priv->is_aspeed = true;
+		/* Disable ast2600 problematic HW arbitration */
+		if (of_device_is_compatible(np, "aspeed,ast2600-mac")) {
+			iowrite32(FTGMAC100_TM_DEFAULT,
+				  priv->base + FTGMAC100_OFFSET_TM);
+		}
 	} else {
 		priv->rxdes0_edorr_mask = BIT(15);
 		priv->txdes0_edotr_mask = BIT(15);
drivers/net/ethernet/faraday/ftgmac100.h +8 -0
index e5876a3fda91..63b3e02fab16 100644
@@ -169,6 +169,14 @@
 #define FTGMAC100_MACCR_FAST_MODE	(1 << 19)
 #define FTGMAC100_MACCR_SW_RST		(1 << 31)
 
+/*
+ * test mode control register
+ */
+#define FTGMAC100_TM_RQ_TX_VALID_DIS (1 << 28)
+#define FTGMAC100_TM_RQ_RR_IDLE_PREV (1 << 27)
+#define FTGMAC100_TM_DEFAULT                                                   \
+	(FTGMAC100_TM_RQ_TX_VALID_DIS | FTGMAC100_TM_RQ_RR_IDLE_PREV)
+
 /*
  * PHY control register
  */
Signature · 1 line
2.17.1
Reviewed-by on v1 1/1
#2 lore raw

Re: [PATCH 1/1] net: ftgmac100: Fix Aspeed ast2600 TX hang issue

Quoted 4 lines · Dylan Hung

The new HW arbitration feature on Aspeed ast2600 will cause MAC TX to hang when handling scatter-gather DMA. Disable the problematic feature by setting MAC register 0x58 bit28 and bit27.

Hi Dylan,

What are the symptoms of this issue? We are seeing this on our systems:

[29376.090637] WARNING: CPU: 0 PID: 9 at net/sched/sch_generic.c:442
dev_watchdog+0x2f0/0x2f4
[29376.099898] NETDEV WATCHDOG: eth0 (ftgmac100): transmit queue 0 timed out
Quoted 1 line

This fixes support for the ast2600, so we can put:

Fixes 39bfab8844a0 ("net: ftgmac100: Add support for DT phy-handle property")
Quoted 26 lines
---
 drivers/net/ethernet/faraday/ftgmac100.c | 5 +++++
 drivers/net/ethernet/faraday/ftgmac100.h | 8 ++++++++
 2 files changed, 13 insertions(+)
drivers/net/ethernet/faraday/ftgmac100.c +5 -0
index 87236206366f..00024dd41147 100644
@@ -1817,6 +1817,11 @@ static int ftgmac100_probe(struct platform_device *pdev)
                priv->rxdes0_edorr_mask = BIT(30);
                priv->txdes0_edotr_mask = BIT(30);
                priv->is_aspeed = true;
+               /* Disable ast2600 problematic HW arbitration */
+               if (of_device_is_compatible(np, "aspeed,ast2600-mac")) {
+                       iowrite32(FTGMAC100_TM_DEFAULT,
+                                 priv->base + FTGMAC100_OFFSET_TM);
+               }
        } else {
                priv->rxdes0_edorr_mask = BIT(15);
                priv->txdes0_edotr_mask = BIT(15);
drivers/net/ethernet/faraday/ftgmac100.h +0 -0
index e5876a3fda91..63b3e02fab16 100644

drivers/net/ethernet/faraday/ftgmac100.h line 178

Quoted 10 lines
drivers/net/ethernet/faraday/ftgmac100.h +7 -0
@@ -169,6 +169,14 @@
 #define FTGMAC100_MACCR_FAST_MODE      (1 << 19)
 #define FTGMAC100_MACCR_SW_RST         (1 << 31)
 
+/*
+ * test mode control register
+ */
+#define FTGMAC100_TM_RQ_TX_VALID_DIS (1 << 28)
+#define FTGMAC100_TM_RQ_RR_IDLE_PREV (1 << 27)
+#define FTGMAC100_TM_DEFAULT                                                   \
+       (FTGMAC100_TM_RQ_TX_VALID_DIS | FTGMAC100_TM_RQ_RR_IDLE_PREV)

Will aspeed issue an updated datasheet with this register documented?

Quoted 7 lines
drivers/net/ethernet/faraday/ftgmac100.h +1 -1
+
 /*
  * PHY control register
  */
--

2.17.1

author
#3 lore raw

RE: [PATCH 1/1] net: ftgmac100: Fix Aspeed ast2600 TX hang issue

Hi Joel,

Quoted 18 lines

-----Original Message-----
From: Joel Stanley [mailto:joel at jms.id.au]
Sent: Wednesday, October 14, 2020 2:41 PM
To: Dylan Hung <[email protected]>
Cc: David S . Miller <[email protected]>; Jakub Kicinski <[email protected]>; netdev at vger.kernel.org; Linux Kernel Mailing List <[email protected]>; Po-Yu Chuang <[email protected]>; linux-aspeed <[email protected]>; OpenBMC Maillist <[email protected]>; BMC-SW <[email protected]>
Subject: Re: [PATCH 1/1] net: ftgmac100: Fix Aspeed ast2600 TX hang issue

Quoted 4 lines · Dylan Hung

The new HW arbitration feature on Aspeed ast2600 will cause MAC TX to hang when handling scatter-gather DMA. Disable the problematic feature by setting MAC register 0x58 bit28 and bit27.

Quoted 9 lines

Hi Dylan,

What are the symptoms of this issue? We are seeing this on our systems:

[29376.090637] WARNING: CPU: 0 PID: 9 at net/sched/sch_generic.c:442
dev_watchdog+0x2f0/0x2f4
[29376.099898] NETDEV WATCHDOG: eth0 (ftgmac100): transmit queue 0

timed out

May I know your soc version? This issue happens on ast2600 version A1. The registers to fix this issue are meaningless/reserved on A0 chip, so it is okay to set them on either A0 or A1. I was encountering this issue when I was running the iperf TX test. The symptom is the TX descriptors are consumed, but no complete packet is sent out.

Quoted 63 lines
Quoted 1 line

This fixes support for the ast2600, so we can put:

Fixes 39bfab8844a0 ("net: ftgmac100: Add support for DT phy-handle

property")

drivers/net/ethernet/faraday/ftgmac100.c static int ftgmac100_probe(struct

Quoted 11 lines
---
 drivers/net/ethernet/faraday/ftgmac100.c | 5 +++++
drivers/net/ethernet/faraday/ftgmac100.h | 8 ++++++++
 2 files changed, 13 insertions(+)
drivers/net/ethernet/faraday/ftgmac100.c +0 -0
index 87236206366f..00024dd41147 100644
@@ -1817,6 +1817,11 @@ static int ftgmac100_probe(struct

platform_device *pdev)

drivers/net/ethernet/faraday/ftgmac100.c static int ftgmac100_probe(struct platform_device *pdev)

Quoted 5 lines
drivers/net/ethernet/faraday/ftgmac100.c +2 -0
                priv->rxdes0_edorr_mask = BIT(30);
                priv->txdes0_edotr_mask = BIT(30);
                priv->is_aspeed = true;
+               /* Disable ast2600 problematic HW arbitration */
+               if (of_device_is_compatible(np, "aspeed,ast2600-mac"))

{

drivers/net/ethernet/faraday/ftgmac100.c static int ftgmac100_probe(struct · line 1823

Quoted 2 lines
drivers/net/ethernet/faraday/ftgmac100.c +2 -0
+                       iowrite32(FTGMAC100_TM_DEFAULT,
+                                 priv->base +

FTGMAC100_OFFSET_TM);

Quoted 10 lines
drivers/net/ethernet/faraday/ftgmac100.c +1 -0
+               }
        } else {
                priv->rxdes0_edorr_mask = BIT(15);
                priv->txdes0_edotr_mask = BIT(15); diff --git

a/drivers/net/ethernet/faraday/ftgmac100.h
b/drivers/net/ethernet/faraday/ftgmac100.h
index e5876a3fda91..63b3e02fab16 100644

drivers/net/ethernet/faraday/ftgmac100.h line 177

Quoted 9 lines
drivers/net/ethernet/faraday/ftgmac100.h +6 -0
@@ -169,6 +169,14 @@
 #define FTGMAC100_MACCR_FAST_MODE      (1 << 19)
 #define FTGMAC100_MACCR_SW_RST         (1 << 31)
 
+/*
+ * test mode control register
+ */
+#define FTGMAC100_TM_RQ_TX_VALID_DIS (1 << 28) #define
+FTGMAC100_TM_RQ_RR_IDLE_PREV (1 << 27)
+#define FTGMAC100_TM_DEFAULT

\

drivers/net/ethernet/faraday/ftgmac100.h line 178

Quoted 1 line
drivers/net/ethernet/faraday/ftgmac100.h +1 -0
+       (FTGMAC100_TM_RQ_TX_VALID_DIS |

FTGMAC100_TM_RQ_RR_IDLE_PREV)

Will aspeed issue an updated datasheet with this register documented?

Quoted 7 lines
drivers/net/ethernet/faraday/ftgmac100.h +1 -1
+
 /*
  * PHY control register
  */
--

2.17.1

#4 lore raw

Re: [PATCH 1/1] net: ftgmac100: Fix Aspeed ast2600 TX hang issue

Quoted 8 lines · Dylan Hung
Quoted 8 lines
Quoted 3 lines

The new HW arbitration feature on Aspeed ast2600 will cause MAC TX to hang when handling scatter-gather DMA. Disable the problematic feature by setting MAC register 0x58 bit28 and bit27.

Hi Dylan,

What are the symptoms of this issue? We are seeing this on our systems:

Quoted 7 lines
Quoted 5 lines
[29376.090637] WARNING: CPU: 0 PID: 9 at net/sched/sch_generic.c:442
dev_watchdog+0x2f0/0x2f4
[29376.099898] NETDEV WATCHDOG: eth0 (ftgmac100): transmit queue 0

timed out

May I know your soc version? This issue happens on ast2600 version A1. The registers to fix this issue are meaningless/reserved on A0 chip, so it is okay to set them on either A0 or A1.

We are running the A1. All of our A0 parts have been replaced with A1.

Quoted 1 line

I was encountering this issue when I was running the iperf TX test. The symptom is the TX descriptors are consumed, but no complete packet is sent out.

What parameters are you using for iperf? I did a lot of testing with iperf3 (and stress-ng running at the same time) and couldn't reproduce the error.

We could only reproduce it when performing other functions, such as debugging/booting the host processor.

Quoted 11 lines
Quoted 11 lines

drivers/net/ethernet/faraday/ftgmac100.h

Quoted 6 lines
drivers/net/ethernet/faraday/ftgmac100.h +6 -0
+/*
+ * test mode control register
+ */
+#define FTGMAC100_TM_RQ_TX_VALID_DIS (1 << 28) #define
+FTGMAC100_TM_RQ_RR_IDLE_PREV (1 << 27)
+#define FTGMAC100_TM_DEFAULT

\

Quoted 1 line

+ (FTGMAC100_TM_RQ_TX_VALID_DIS |

FTGMAC100_TM_RQ_RR_IDLE_PREV)

Will aspeed issue an updated datasheet with this register documented?

Did you see this question?

Cheers,

Joel

author
#5 lore raw

RE: [PATCH 1/1] net: ftgmac100: Fix Aspeed ast2600 TX hang issue

Quoted 33 lines

-----Original Message-----
From: Joel Stanley [mailto:joel at jms.id.au]
Sent: Thursday, October 15, 2020 6:31 AM
To: Dylan Hung <[email protected]>
Cc: David S . Miller <[email protected]>; Jakub Kicinski <[email protected]>; netdev at vger.kernel.org; Linux Kernel Mailing List <[email protected]>; Po-Yu Chuang <[email protected]>; linux-aspeed <[email protected]>; OpenBMC Maillist <[email protected]>; BMC-SW <[email protected]>
Subject: Re: [PATCH 1/1] net: ftgmac100: Fix Aspeed ast2600 TX hang issue

Quoted 8 lines · Dylan Hung
Quoted 8 lines
Quoted 3 lines

The new HW arbitration feature on Aspeed ast2600 will cause MAC TX to hang when handling scatter-gather DMA. Disable the problematic feature by setting MAC register 0x58 bit28 and bit27.

Hi Dylan,

What are the symptoms of this issue? We are seeing this on our systems:

Quoted 7 lines
Quoted 5 lines
[29376.090637] WARNING: CPU: 0 PID: 9 at net/sched/sch_generic.c:442
dev_watchdog+0x2f0/0x2f4
[29376.099898] NETDEV WATCHDOG: eth0 (ftgmac100): transmit queue 0

timed out

May I know your soc version? This issue happens on ast2600 version A1.

The registers to fix this issue are meaningless/reserved on A0 chip, so it is okay to set them on either A0 or A1.

We are running the A1. All of our A0 parts have been replaced with A1.

Quoted 8 lines
Quoted 1 line

I was encountering this issue when I was running the iperf TX test. The

symptom is the TX descriptors are consumed, but no complete packet is sent out.

What parameters are you using for iperf? I did a lot of testing with iperf3 (and stress-ng running at the same time) and couldn't reproduce the error.

I simply use "iperf -c <server ip>" on ast2600. It is very easy to reproduce. I append the log below:
Noticed that this issue only happens when HW scatter-gather (NETIF_F_SG) is on.

[AST /]$ iperf3 -c 192.168.100.89
Connecting to host 192.168.100.89, port 5201
[ 4] local 192.168.100.45 port 45346 connected to 192.168.100.89 port 5201

[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec  44.8 MBytes   375 Mbits/sec    2   1.43 KBytes
[  4]   1.00-2.00   sec  0.00 Bytes  0.00 bits/sec    2   1.43 KBytes
[  4]   2.00-3.00   sec  0.00 Bytes  0.00 bits/sec    0   1.43 KBytes
[  4]   3.00-4.00   sec  0.00 Bytes  0.00 bits/sec    1   1.43 KBytes
[  4]   4.00-5.00   sec  0.00 Bytes  0.00 bits/sec    0   1.43 KBytes
^C[  4]   5.00-5.88   sec  0.00 Bytes  0.00 bits/sec    0   1.43 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-5.88   sec  44.8 MBytes  64.0 Mbits/sec    5             sender
[  4]   0.00-5.88   sec  0.00 Bytes  0.00 bits/sec                  receiver

iperf3: interrupt - the client has terminated

Quoted 3 lines

We could only reproduce it when performing other functions, such as debugging/booting the host processor.

Could it be another issue?

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

drivers/net/ethernet/faraday/ftgmac100.h

Quoted 6 lines
drivers/net/ethernet/faraday/ftgmac100.h +6 -0
+/*
+ * test mode control register
+ */
+#define FTGMAC100_TM_RQ_TX_VALID_DIS (1 << 28) #define
+FTGMAC100_TM_RQ_RR_IDLE_PREV (1 << 27) #define
+FTGMAC100_TM_DEFAULT

\

Quoted 7 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line

+ (FTGMAC100_TM_RQ_TX_VALID_DIS |

FTGMAC100_TM_RQ_RR_IDLE_PREV)

Will aspeed issue an updated datasheet with this register documented?

Did you see this question?

Sorry, I missed this question. Aspeed will update the datasheet accordingly.

Quoted 3 lines

Cheers,

Joel

#6 lore raw

Re: [PATCH 1/1] net: ftgmac100: Fix Aspeed ast2600 TX hang issue

Quoted 11 lines · Dylan Hung
Quoted 8 lines
Quoted 1 line

I was encountering this issue when I was running the iperf TX test. The

symptom is the TX descriptors are consumed, but no complete packet is sent out.

What parameters are you using for iperf? I did a lot of testing with iperf3 (and stress-ng running at the same time) and couldn't reproduce the error.

I simply use "iperf -c <server ip>" on ast2600. It is very easy to reproduce. I append the log below:
Noticed that this issue only happens when HW scatter-gather (NETIF_F_SG) is on.

Ok. This appears to be on by default in the
drivers/net/ethernet/faraday/ftgmac100.c:

        netdev->hw_features = NETIF_F_RXCSUM | NETIF_F_HW_CSUM |
                NETIF_F_GRO | NETIF_F_SG | NETIF_F_HW_VLAN_CTAG_RX |
                NETIF_F_HW_VLAN_CTAG_TX;
Quoted 14 lines

[AST /]$ iperf3 -c 192.168.100.89
Connecting to host 192.168.100.89, port 5201
[ 4] local 192.168.100.45 port 45346 connected to 192.168.100.89 port 5201

[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec  44.8 MBytes   375 Mbits/sec    2   1.43 KBytes
[  4]   1.00-2.00   sec  0.00 Bytes  0.00 bits/sec    2   1.43 KBytes
[  4]   2.00-3.00   sec  0.00 Bytes  0.00 bits/sec    0   1.43 KBytes
[  4]   3.00-4.00   sec  0.00 Bytes  0.00 bits/sec    1   1.43 KBytes
[  4]   4.00-5.00   sec  0.00 Bytes  0.00 bits/sec    0   1.43 KBytes
^C[  4]   5.00-5.88   sec  0.00 Bytes  0.00 bits/sec    0   1.43 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-5.88   sec  44.8 MBytes  64.0 Mbits/sec    5             sender
[  4]   0.00-5.88   sec  0.00 Bytes  0.00 bits/sec                  receiver
Quoted 1 line

iperf3: interrupt - the client has terminated

I just realised my test machine must be on a 100Mbit network. I will try testing on a gigabit network.

Quoted 4 lines
Quoted 3 lines

We could only reproduce it when performing other functions, such as debugging/booting the host processor.

Could it be another issue?

I hope not! We have deployed your patch on our systems and I will let you know if we see the bug again.

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

drivers/net/ethernet/faraday/ftgmac100.h

Quoted 6 lines
drivers/net/ethernet/faraday/ftgmac100.h +6 -0
+/*
+ * test mode control register
+ */
+#define FTGMAC100_TM_RQ_TX_VALID_DIS (1 << 28) #define
+FTGMAC100_TM_RQ_RR_IDLE_PREV (1 << 27) #define
+FTGMAC100_TM_DEFAULT

\

Quoted 8 lines
Quoted 7 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line

+ (FTGMAC100_TM_RQ_TX_VALID_DIS |

FTGMAC100_TM_RQ_RR_IDLE_PREV)

Will aspeed issue an updated datasheet with this register documented?

Did you see this question?

Sorry, I missed this question. Aspeed will update the datasheet accordingly.

Thank you.

maintainer Applied
#7 lore raw

Re: [PATCH 1/1] net: ftgmac100: Fix Aspeed ast2600 TX hang issue

In reply to Dylan Hung
Quoted 5 lines · On Wed, 14 Oct 2020 14:06:32 +0800 Dylan Hung wrote:

The new HW arbitration feature on Aspeed ast2600 will cause MAC TX to hang when handling scatter-gather DMA. Disable the problematic feature by setting MAC register 0x58 bit28 and bit27.

Applied, thank you.