net: dsa: qca8k: Disable mgmt Ethernet for qca8327
PATCH 1/1 net: dsa: qca8k: Disable mgmt Ethernet for qca8327
While the qca8327 switch appears to support in-band mgmt Ethernet, prolonged use of that protocol (e.g. for polling link state) makes the device unstable: within minutes, ports randomly go down and no traffic is forwarded anymore. The same issues do not occur when MDIO is used exclusively, so ensure mgmt Ethernet is not used on the qca8327.
---
I came across this while migrating an AR9344-based router with a QCA8327 rev. 4 switch to a DSA-aware driver. This glitch is a pain in the neck to troubleshoot any further as it occurs randomly, anywhere between a minute to an hour after the switch is set up; traffic load exerted on the switch does not seem to matter as the problem can be triggered on a virtually idle device. Previously working links are reported as going down (one by one, not all at once), even though port LEDs still blink; no traffic is forwarded; reloading qca8k does not alleviate the problem, only power cycling seems to help. Nothing like this happens when only MDIO is used. However, qca8k currently only uses MDIO as a fallback. I figured that simpler is better and that mgmt Ethernet should simply be disabled for the qca8327, but I would be happy to work on some configurable solution if that would be preferable.
drivers/net/dsa/qca/qca8k-8xxx.c | 14 ++++++++++++++ 1 file changed, 14 insertions(+)
drivers/net/dsa/qca/qca8k-8xxx.c +14 -0
@@ -160,6 +160,11 @@ qca8k_set_page(struct qca8k_priv *priv, u16 page)return 0 ; } +static bool qca8k_mgmt_eth_disabled (const struct qca8k_priv *priv) +{ +return priv->switch_id ==QCA8K_ID_QCA8327 ; +} +static void qca8k_rw_reg_ack_handler (struct dsa_switch *ds,struct sk_buff *skb) {struct qca8k_mgmt_eth_data *mgmt_eth_data;
@@ -316,6 +321,9 @@ static int qca8k_read_eth(struct qca8k_priv *priv, u32 reg, u32 *val, int len)bool ack;int ret; +if (qca8k_mgmt_eth_disabled (priv)) +return -ENXIO ; + skb =qca8k_alloc_mdio_header (MDIO_READ , reg,NULL ,QCA8K_ETHERNET_MDIO_PRIORITY , len);if (!skb)
@@ -368,6 +376,9 @@ static int qca8k_write_eth(struct qca8k_priv *priv, u32 reg, u32 *val, int len)bool ack;int ret; +if (qca8k_mgmt_eth_disabled (priv)) +return -ENXIO ; + skb =qca8k_alloc_mdio_header (MDIO_WRITE , reg, val,QCA8K_ETHERNET_MDIO_PRIORITY , len);if (!skb)
@@ -630,6 +641,9 @@ qca8k_phy_eth_command(struct qca8k_priv *priv, bool read, int phy,int ret, ret1;bool ack; +if (qca8k_mgmt_eth_disabled (priv)) +return -ENXIO ; +if (regnum >=QCA8K_MDIO_MASTER_MAX_REG )return -EINVAL ;
Signature · 1 line
2.55.0
Andrew Lunn
maintainerQuoted 5 lines · Michał Kępień
While the qca8327 switch appears to support in-band mgmt Ethernet, prolonged use of that protocol (e.g. for polling link state) makes the device unstable: within minutes, ports randomly go down and no traffic is forwarded anymore. The same issues do not occur when MDIO is used exclusively, so ensure mgmt Ethernet is not used on the qca8327.
Do you have time to narrow down the cause?
I _think_ in band management is used for a few different things. e.g. statistics, as you said, PHY polling etc. Rather than turning everything off, could you try just doing PHY polling via MDIO, but statistics via ethernet, and do an ethtool -S every so often to see if you can trigger the problem.
Could you also check if the management packets are getting lost? Look at the results from wait_for_completion_timeout(), is it timing out? If so, is the retry mechanism working? When i added the Marvell equivalent for in-band signalling i got the retry mechanism wrong, but never noticed because i was not loosing packets.
Thanks
Andrew
Andrew,
First of all, after taking a closer look at this issue and the driver code, I realized that the approach I went with in the patch stemmed from my ignorance and that I submitted it too quickly - apologies about that.
Quoted 7 lines
Quoted 5 lines
While the qca8327 switch appears to support in-band mgmt Ethernet, prolonged use of that protocol (e.g. for polling link state) makes the device unstable: within minutes, ports randomly go down and no traffic is forwarded anymore. The same issues do not occur when MDIO is used exclusively, so ensure mgmt Ethernet is not used on the qca8327.
Do you have time to narrow down the cause?
Sure, I should be able to spare a few cycles on this in the upcoming weeks.
Quoted 5 lines
I _think_ in band management is used for a few different things. e.g. statistics, as you said, PHY polling etc. Rather than turning everything off, could you try just doing PHY polling via MDIO, but statistics via ethernet, and do an ethtool -S every so often to see if you can trigger the problem.
Ack, I'll try that.
Quoted 2 lines
Could you also check if the management packets are getting lost? Look at the results from
wait_for_completion_timeout(), is it timing out?
Ack.
Quoted 3 lines
If so, is the retry mechanism working? When i added the Marvell equivalent for in-band signalling i got the retry mechanism wrong, but never noticed because i was not loosing packets.
Could you please point me at the exact retry mechanism you had in mind, either the qca8k one or the Marvell equivalent? I cannot see any in drivers/net/dsa/qca/qca8k-8xxx.c. If I'm reading qca8k code correctly, when wait_for_completion_timeout() fails, its return value is just bubbled up the call chain.
Thanks,
Signature · 2 lines
Best regards, Michał Kępień
Andrew Lunn
maintainerQuoted 5 lines
Could you please point me at the exact retry mechanism you had in mind, either the qca8k one or the Marvell equivalent? I cannot see any in
drivers/net/dsa/qca/qca8k-8xxx.c. If I'm reading qca8k code correctly, whenwait_for_completion_timeout()fails, its return value is just bubbled up the call chain.
With the mv88e6xxx code, if ethernet times out, i repeated the request using MDIO. The code is not in mainline but a few users are using it.
At minimum, i would put a printk() there, and see if it happens. It could be unrelated to your problem...
Andrew
Quoted 8 lines
Quoted 5 lines
Could you please point me at the exact retry mechanism you had in mind, either the qca8k one or the Marvell equivalent? I cannot see any in
drivers/net/dsa/qca/qca8k-8xxx.c. If I'm reading qca8k code correctly, whenwait_for_completion_timeout()fails, its return value is just bubbled up the call chain.With the mv88e6xxx code, if ethernet times out, i repeated the request using MDIO. The code is not in mainline but a few users are using it.
Ah, that way, okay. It looks like every mgmt Ethernet function in qca8k has an MDIO fallback path, but I'll keep this in mind, thanks.
Quoted 2 lines
At minimum, i would put a
printk()there, and see if it happens. It could be unrelated to your problem...
Ack, thanks.
Signature · 2 lines
Best regards, Michał Kępień
Christian Marangi
maintainerQuoted 7 lines · Michał Kępień
While the qca8327 switch appears to support in-band mgmt Ethernet, prolonged use of that protocol (e.g. for polling link state) makes the device unstable: within minutes, ports randomly go down and no traffic is forwarded anymore. The same issues do not occur when MDIO is used exclusively, so ensure mgmt Ethernet is not used on the qca8327.
Signed-off-by Michał Kępień <[email protected]>
Quoted 15 lines
---
I came across this while migrating an AR9344-based router with a QCA8327 rev. 4 switch to a DSA-aware driver. This glitch is a pain in the neck to troubleshoot any further as it occurs randomly, anywhere between a minute to an hour after the switch is set up; traffic load exerted on the switch does not seem to matter as the problem can be triggered on a virtually idle device. Previously working links are reported as going down (one by one, not all at once), even though port LEDs still blink; no traffic is forwarded; reloading qca8k does not alleviate the problem, only power cycling seems to help. Nothing like this happens when only MDIO is used. However, qca8k currently only uses MDIO as a fallback. I figured that simpler is better and that mgmt Ethernet should simply be disabled for the qca8327, but I would be happy to work on some configurable solution if that would be preferable.
This is a long standing issue and it seems to me disabling mgmt is just a big workaround to a real problem.
Long time ago it was reported that there seems to be a problem with the mdio master register for external and internall access and how mgmt was actually sending mdio command... just done by the switch. Could the 2 issue related?
One idea might be to verify that stuff gets actually written... as Andrew said to verify if some packets doesn't get lost or just ignored.
Also as Andrew said I would still save this for the MIB part as the 2 thing should be unrelated.
Quoted 8 lines
drivers/net/dsa/qca/qca8k-8xxx.c | 14 ++++++++++++++ 1 file changed, 14 insertions(+)drivers/net/dsa/qca/qca8k-8xxx.c +0
-0
drivers/net/dsa/qca/qca8k-8xxx.c qca8k_set_page(struct qca8k_priv *priv, u16 page) · line 167
Quoted 8 lines
drivers/net/dsa/qca/qca8k-8xxx.c +5
-0@@ -160,6 +160,11 @@ qca8k_set_page(struct qca8k_priv *priv, u16 page)return 0 ; } +static bool qca8k_mgmt_eth_disabled (const struct qca8k_priv *priv) +{ +return priv->switch_id ==QCA8K_ID_QCA8327 ; +} +
Instead of this and return ENXIO I would just not install the relevant OPs for the tagger and use the mdio path directly...
Makes the code cleaner and less CPU cycle (the target is ath79 and powerpc stuff)
But as said above disabling the feature is the last solution after all the verification are done.
Quoted 36 lines
drivers/net/dsa/qca/qca8k-8xxx.c +9
-0static void qca8k_rw_reg_ack_handler (struct dsa_switch *ds,struct sk_buff *skb) {struct qca8k_mgmt_eth_data *mgmt_eth_data;@@ -316,6 +321,9 @@ static int qca8k_read_eth(struct qca8k_priv *priv, u32 reg, u32 *val, int len)bool ack;int ret; +if (qca8k_mgmt_eth_disabled (priv)) +return -ENXIO ; + skb =qca8k_alloc_mdio_header (MDIO_READ , reg,NULL ,QCA8K_ETHERNET_MDIO_PRIORITY , len);if (!skb)@@ -368,6 +376,9 @@ static int qca8k_write_eth(struct qca8k_priv *priv, u32 reg, u32 *val, int len)bool ack;int ret; +if (qca8k_mgmt_eth_disabled (priv)) +return -ENXIO ; + skb =qca8k_alloc_mdio_header (MDIO_WRITE , reg, val,QCA8K_ETHERNET_MDIO_PRIORITY , len);if (!skb)@@ -630,6 +641,9 @@ qca8k_phy_eth_command(struct qca8k_priv *priv, bool read, int phy,int ret, ret1;bool ack; +if (qca8k_mgmt_eth_disabled (priv)) +return -ENXIO ; +if (regnum >=QCA8K_MDIO_MASTER_MAX_REG )return -EINVAL ;Signature · 1 line
2.55.0
Signature · 1 line
Ansuel
Quoted 15 lines
Quoted 5 lines
I came across this while migrating an AR9344-based router with a QCA8327 rev. 4 switch to a DSA-aware driver. This glitch is a pain in the neck to troubleshoot any further as it occurs randomly, anywhere between a minute to an hour after the switch is set up; traffic load exerted on the switch does not seem to matter as the problem can be triggered on a
Quoted 9 lines
virtually idle device. Previously working links are reported as going down (one by one, not all at once), even though port LEDs still blink; no traffic is forwarded; reloading qca8k does not alleviate the problem, only power cycling seems to help. Nothing like this happens when only MDIO is used. However, qca8k currently only uses MDIO as a fallback. I figured that simpler is better and that mgmt Ethernet should simply be disabled for the qca8327, but I would be happy to work on some configurable solution if that would be preferable.
Quoted 2 lines
This is a long standing issue and it seems to me disabling mgmt is just a big workaround to a real problem.
Understood. Is it a long-standing issue for the QCA mgmt Ethernet code in general or for a specific subset of switches (or devices)?
Quoted 4 lines
Long time ago it was reported that there seems to be a problem with the mdio master register for external and internall access and how mgmt was actually sending mdio command... just done by the switch. Could the 2 issue related?
This thread?
https://lore.kernel.org/netdev/[email protected]/
If so, that was for QCA8337, on a board where the external MDIO bus has an extra PHY attached, so it did not look like a match for my case. Of course, it _might_ be related to the issue I'm running into, it just did not seem to be at first glance.
Quoted 2 lines
One idea might be to verify that stuff gets actually written... as Andrew said to verify if some packets doesn't get lost or just ignored.
Ack.
Quoted 2 lines
Also as Andrew said I would still save this for the MIB part as the 2 thing should be unrelated.
Got it, I'll play around with it and see what I can find out, thanks.
Signature · 2 lines
Best regards, Michał Kępień
Christian Marangi
maintainerQuoted 15 lines · Michał Kępień
Quoted 15 lines
Quoted 14 lines
I came across this while migrating an AR9344-based router with a QCA8327 rev. 4 switch to a DSA-aware driver. This glitch is a pain in the neck to troubleshoot any further as it occurs randomly, anywhere between a minute to an hour after the switch is set up; traffic load exerted on the switch does not seem to matter as the problem can be triggered on a virtually idle device. Previously working links are reported as going down (one by one, not all at once), even though port LEDs still blink; no traffic is forwarded; reloading qca8k does not alleviate the problem, only power cycling seems to help. Nothing like this happens when only MDIO is used. However, qca8k currently only uses MDIO as a fallback. I figured that simpler is better and that mgmt Ethernet should simply be disabled for the qca8327, but I would be happy to work on some configurable solution if that would be preferable.
Quoted 6 lines
Quoted 2 lines
This is a long standing issue and it seems to me disabling mgmt is just a big workaround to a real problem.
Understood. Is it a long-standing issue for the QCA mgmt Ethernet code in general or for a specific subset of switches (or devices)?
specific subset of switches. On ipq806x the 8337 is mounted and mgmt works correctly without issue.
One thing I notice on a different vendor (Airoha) is that sometimes using these indirect way to access the Switch register might introduce interesting HW bug.
One bug that was there was that when PBUS was used to access single port PHY register (instead of direct MDIO) the link up/down was broken. My theory was that the Switch chip had some latch logic that was only triggered with MDIO. Using PBUS didn't trigger such thing.
Could be that the QCA 8327 switch also have some kind of HW bug where specific register needs to go with MDIO or some refresh/latch logic are not correctly triggered.
An idea might be to limit the mgmt to vlan and fdb and see if the problem is still there. (after all those are the path where mgmt would benefit due to the multiple register access required)
Quoted 27 lines
Quoted 4 lines
Long time ago it was reported that there seems to be a problem with the mdio master register for external and internall access and how mgmt was actually sending mdio command... just done by the switch. Could the 2 issue related?
This thread?
https://lore.kernel.org/netdev/[email protected]/
If so, that was for QCA8337, on a board where the external MDIO bus has an extra PHY attached, so it did not look like a match for my case. Of course, it _might_ be related to the issue I'm running into, it just did not seem to be at first glance.
Quoted 2 lines
One idea might be to verify that stuff gets actually written... as Andrew said to verify if some packets doesn't get lost or just ignored.
Ack.
Quoted 2 lines
Also as Andrew said I would still save this for the MIB part as the 2 thing should be unrelated.
Got it, I'll play around with it and see what I can find out, thanks.
Signature · 2 lines
Best regards, Michał Kępień
Signature · 1 line
Ansuel
Christian, Andrew,
Here is what I managed to determine for the QCA8327 switch on the board I have access to.
Quoted 8 lines
One bug that was there was that when PBUS was used to access single port PHY register (instead of direct MDIO) the link up/down was broken. My theory was that the Switch chip had some latch logic that was only triggered with MDIO. Using PBUS didn't trigger such thing.
Could be that the QCA 8327 switch also have some kind of HW bug where specific register needs to go with MDIO or some refresh/latch logic are not correctly triggered.
My problem sounds similar. The exact symptom I am observing is that at a random point in time after switch setup, PHY register reads performed via mgmt Ethernet frames start returning all zeros. The mgmt Ethernet transactions themselves are completed correctly, they just return all zeros in the data part. When the issue is triggered, _all_ subsequent PHY register reads performed via mgmt Ethernet frames for the affected PHY return all zeros, i.e. it never recovers.
The problem is triggered on distinct PHYs one by one, in a seemingly random order; I did not identify any patterns. Given enough time, all PHYs eventually start returning all zeros for all of their register reads.
At one point, I tweaked qca8k_internal_mdio_read() so that when a PHY register read fails in this specific way, it is immediately retried over MDIO. Those fallback reads over MDIO also returned all zeros.
No timeouts are occurring for mgmt Ethernet frames. I instrumented all wait_for_completion_timeout() call sites in qca8k-8xxx.c and have yet to see any of them log a timeout (or any other error, for that matter). In other words, all requests sent via mgmt Ethernet frames are responded to by the switch in a timely manner, so I have no reason to believe that the retry logic is involved here in any way.
All of this reproducibly happens on a device that does not even have any Ethernet cables connected to it.
Manually probing registers 2 and 3 on the external MDIO bus reveals no other devices except the expected five switch PHYs.
I used tcpdump to capture the traffic going through the DSA conduit device, from the moment the switch is configured until the failure occurs. I found nothing suspicious: the last PHY register read before the failed one happens about a second earlier and the sequence of mgmt Ethernet frames sent and received for the last _successful_ read is a 1:1 match against the first _failing_ read, sans the sequence numbers. No other traffic is interleaved, which is expected given that both the external MDIO bus mutex and the mgmt Ethernet mutex are held throughout each distinct PHY management transaction.
Given the above, I am out of ideas for something to latch on to for further investigations. It seems that over time, using mgmt Ethernet for PHY access on this board makes the hardware transition into some broken state that it cannot recover from. As I am able to reliably trigger this problem, I am open to ideas for further experiments that could possibly shed some light on this.
Quoted 3 lines
An idea might be to limit the mgmt to vlan and fdb and see if the problem is still there. (after all those are the path where mgmt would benefit due to the multiple register access required)
Indeed, it seems that the simplest workaround for this glitch is to move the PHY nodes in the DTS file from the ethernet-switch node to its parent, the mdio node, and use phy-handle for ethernet-ports to configure qca8k in "external MDIO" mode. That way, the external MDIO bus is used for PHY management while mgmt Ethernet is still used for MIB access and switch configuration, which seems to be stable; with PHY management moved to MDIO, I ran "ethtool -S <iface>" in a loop for a few hours and everything still seemingly worked fine. This seems to be a cleaner workaround than tweaking the driver code, so I don't plan to submit any further patches in this spirit.
Thank you for your guidance. If you have any ideas for further experiments in this area, I am all ears.
Signature · 2 lines
Best regards, Michał Kępień
Hi everyone,
To add to this thread, there are a couple of other well-documented issues tied to management Ethernet in qca8k:
1) TP-Link Archer C7 v2 (dual CPU ports 0 and 6): One CPU port becomes muted/unusable during setup when state transitions alter the management conduit. (ref:
https://github.com/openwrt/openwrt/pull/24011)
2) CZ.NIC Turris 1.0 and 1.1: Internal PHY access via management frames leaks onto the external MDC/MDIO pins, corrupting communication between the kernel and the external WAN PHY (commit 526c8ee was insufficient to fully fix this). (ref:
https://github.com/openwrt/openwrt/pull/24755)
There are also several historical reports for other TP-Link Archer devices (single CPU port) where ports randomly drop after some uptime, exhibiting similar symptoms Michał described. At least for the two cases I cited before, bypassing Ethernet-based PHY management and sticking to MDIO works around the issue reliably.
As noted in Turris 1.0 PR patch, when the switch translates an Ethernet management frame into an internal MDIO transaction, that signal leaks onto the external MDC/MDIO lines. Because the switch acts as an MDIO master for that internal query while the host CPU remains the primary MDIO master, electrical bus conflict might break the switch's internal MDIO translator state machine. Commit 526c8ee attempted to solve this by acquiring bus->mdio_lock before transmitting management frames, but software-level locking cannot prevent hardware-level bus contention or state corruption if QCA8K_MDIO_MASTER_EN toggles or leaks while lines are shared.
This raises the question: does Ethernet-based PHY access actually yield any meaningful performance benefit (or is it worth it)? PHY polling occurs at a low frequency. Also, maintaining a dual code path (Ethernet management with silent fallback to direct MDIO) significantly complicates troubleshooting. By comparison, drivers like rtl8365mb enforce a single, deterministic code path for register/PHY accesses to keep behavior predictable across all boards, even when we could use an optimized path.
Given that Ethernet management frames for PHY access introduce multiple hard-to-diagnose regressions across different board topologies, should we consider dropping qca8k_phy_eth_command() entirely in favor of direct MDIO, or at least providing a mechanism to disable it? If MIB access becomes an issue, we could introduce a cache mechanism.
Best regards,
Luiz