net: lan743x: automatically recover from TX hang

Background
----------
On a custom HW platform that uses the LAN7430 NIC we have been able to reproduce a scenario where the SoC stops receiving TX completion interrupts, appearing as a TX hang. After dumping and decoding the LAN7430 registers we see that the TX channel reports that it is in started state with no error, but TX_HEAD stops advancing. TX MSI-X stops incrementing while RX interrupts continue to increment.

This scenario has most often been reproduced in large server cluster environments, but it's also been seen locally during repeated SCP and iperf testing.

The root cause of this issue is still unknown and we are continuing debug. It was found that bringing the interface down and up recovers the network. This issue has not been reproduced with the LAN7430 EVB connected to a standard Linux server.

What this patch series does
---------------------------
This patch series attempts to implement automatic recovery of the network interface after a TX hang is detected. It does not fix the underlying root cause of the TX hang.

At a high level, the patches implement the following:
1. ndo_tx_timeout support to reset and recover the interface when the netdev TX watchdog triggers.
2. BQL support to decrease the amount of time it takes for the watchdog to trigger.
3. Stability improvements and error handling around the open/close paths that now get executed more frequently as part of the TX hang recovery flow.

Potential issues and concerns
-----------------------------
* One of the patches adds new lan743x_dmac_tx_reset and lan743x_dmac_rx_reset helper functions. There is no cleanup handling introduced if these return an error, only a new error log. This matches pre-existing behavior: if a reset timeout occurs then it is possible to DMA freed memory. The datasheet states that these resets are performed immediately in HW so I'm not sure if these will ever time out, but I'm open to suggestions on how we should handle an error in these cases.
* The TX timeout reset flow does a full lan743x_netdev_close followed by lan743x_netdev_open instead of individual TX channel reset. If the stall is happening inside the NIC DMA engine then the theory is that a full reset may be needed to recover. For internal use we also wanted to reduce risk by putting the NIC into a clean state after recovery with full reset. It would be great to have Microchip/LAN743x experts confirm the best reset flow.
* LLM review tools flagged several possible bugs during development that resulted in the inclusion of three "Fixes:" tagged patches in this series. They are included in this series with the ndo_tx_timeout support because lan743x_netdev_open and lan743x_netdev_close are now called more frequently as a result of TX hang recovery.

Testing
-------
Testing was performed on a 7.0 based kernel using the same driver code. Please let me know if you would like me to run any other tests.

Tests:
* Verified the TX timeout watchdog triggers and recovers the network interface when TX hang is reproduced. This was done as part of repeated SCP loop testing and is also being tested in a server cluster environment. dmesg was inspected for "NETDEV WATCHDOG" to verify that the watchdog is being triggered. No errors were seen in dmesg after reset.
* Performance tests using iperf3 to measure RX and TX throughput show no noticeable difference with the patches applied.
* Repeated module load and unload cycles produce no errors in dmesg. Network appears to function as expected when module is loaded.
* Repeated ifconfig down and up cycles produce no errors in dmesg. Network appears to function as expected when interface is back up.

Chris Babroski (5):
net: lan743x: reset DMA channels that fail to stop
net: lan743x: start phylink last and stop it first
net: lan743x: quiesce TX before freeing the rings
net: lan743x: reset the device on TX timeout
net: lan743x: add byte queue limits support

 drivers/net/ethernet/microchip/lan743x_main.c | 177 +++++++++++++++---
 drivers/net/ethernet/microchip/lan743x_main.h |   2 +
 2 files changed, 155 insertions(+), 24 deletions(-)

base-commit: 05b5019b01fb9443b09ae4ed34af1135a80a84c8

2.34.1