Skip to contentSkip to conversation

introduce mt76 tx hang watchdog

PATCH In review Lorenzo Bianconi 2 participants 7 messages Dec 24, 2018 – Dec 27, 2018

Subsystems (2)
Arm MediaTek MediaTek MT76 wireless LAN driver
lore
Share & tools
lore

Lorenzo Bianconi

author
#0 lore raw

cover introduce mt76 tx hang watchdog

Port mt76x02_check_tx_hang watchdog from vendor driver in order to reset the card when tx mac/dma engine is stucked. Tx mac/dma stuck occurs when the device is heavily loaded or in a noisy environment. In that condition tx queues are not drained since the device does not generate tx interrupts.
This series is based on "add EDCCA support to mt76x{0,2}e drivers"
https://patchwork.kernel.org/cover/10727149/

Lorenzo Bianconi (2):
mt76: run mac work every 100ms
mt76: introduce mt76x02_check_tx_hang watchdog

 drivers/net/wireless/mediatek/mt76/mt76.h     |  1 +
 .../net/wireless/mediatek/mt76/mt76x0/pci.c   |  2 +-
 .../net/wireless/mediatek/mt76/mt76x0/usb.c   |  2 +-
 drivers/net/wireless/mediatek/mt76/mt76x02.h  |  7 ++
 .../wireless/mediatek/mt76/mt76x02_debugfs.c  |  2 +
 .../net/wireless/mediatek/mt76/mt76x02_mac.c  | 90 ++++++++++++++++++-
 .../net/wireless/mediatek/mt76/mt76x02_mmio.c | 23 +++++
 .../wireless/mediatek/mt76/mt76x2/pci_main.c  |  2 +-
 .../wireless/mediatek/mt76/mt76x2/usb_main.c  |  2 +-
 9 files changed, 125 insertions(+), 6 deletions(-)
Signature · 1 line
2.20.1
author
#1 lore raw

PATCH 1/2 mt76: run mac work every 100ms

Introduce MT_WATCHDOG_TIME macro in order to run mac work every 100ms. This is a preliminary patch to implement a proper tx hang watchdog

Signed-off-by Lorenzo Bianconi <[email protected]>
---
 drivers/net/wireless/mediatek/mt76/mt76x0/pci.c      | 2 +-
 drivers/net/wireless/mediatek/mt76/mt76x0/usb.c      | 2 +-
 drivers/net/wireless/mediatek/mt76/mt76x02.h         | 1 +
 drivers/net/wireless/mediatek/mt76/mt76x02_mac.c     | 2 +-
 drivers/net/wireless/mediatek/mt76/mt76x2/pci_main.c | 2 +-
 drivers/net/wireless/mediatek/mt76/mt76x2/usb_main.c | 2 +-
 6 files changed, 6 insertions(+), 5 deletions(-)
drivers/net/wireless/mediatek/mt76/mt76x0/pci.c +1 -1
index 720b05e325a5..d2005632101f 100644
@@ -30,7 +30,7 @@ static int mt76x0e_start(struct ieee80211_hw *hw)
 	mt76x02_mac_start(dev);
 	mt76x0_phy_calibrate(dev, true);
 	ieee80211_queue_delayed_work(dev->mt76.hw, &dev->mac_work,
-				     MT_CALIBRATE_INTERVAL);
+				     MT_WATCHDOG_TIME);
 	ieee80211_queue_delayed_work(dev->mt76.hw, &dev->cal_work,
 				     MT_CALIBRATE_INTERVAL);
 	set_bit(MT76_STATE_RUNNING, &dev->mt76.state);
drivers/net/wireless/mediatek/mt76/mt76x0/usb.c +1 -1
index 0e6b43bb4678..b7ace310ca72 100644
@@ -118,7 +118,7 @@ static int mt76x0u_start(struct ieee80211_hw *hw)
 
 	mt76x0_phy_calibrate(dev, true);
 	ieee80211_queue_delayed_work(dev->mt76.hw, &dev->mac_work,
-				     MT_CALIBRATE_INTERVAL);
+				     MT_WATCHDOG_TIME);
 	ieee80211_queue_delayed_work(dev->mt76.hw, &dev->cal_work,
 				     MT_CALIBRATE_INTERVAL);
 	set_bit(MT76_STATE_RUNNING, &dev->mt76.state);
drivers/net/wireless/mediatek/mt76/mt76x02.h +1 -0
index 5f5831d18af5..17e13e373f53 100644
@@ -26,6 +26,7 @@
 #include "mt76x02_dfs.h"
 #include "mt76x02_dma.h"
 
+#define MT_WATCHDOG_TIME	(HZ / 10)
 #define MT_CALIBRATE_INTERVAL	HZ
 
 #define MT_MAX_CHAINS		2
drivers/net/wireless/mediatek/mt76/mt76x02_mac.c +1 -1
index aed108e8761f..a4ff5a7a219f 100644
@@ -888,7 +888,7 @@ void mt76x02_mac_work(struct work_struct *work)
 	mt76_tx_status_check(&dev->mt76, NULL, false);
 
 	ieee80211_queue_delayed_work(mt76_hw(dev), &dev->mac_work,
-				     MT_CALIBRATE_INTERVAL);
+				     MT_WATCHDOG_TIME);
 }
 
 void mt76x02_mac_set_bssid(struct mt76x02_dev *dev, u8 idx, const u8 *addr)
drivers/net/wireless/mediatek/mt76/mt76x2/pci_main.c +1 -1
index b54a32397486..ec936002a286 100644
@@ -33,7 +33,7 @@ mt76x2_start(struct ieee80211_hw *hw)
 		goto out;
 
 	ieee80211_queue_delayed_work(mt76_hw(dev), &dev->mac_work,
-				     MT_CALIBRATE_INTERVAL);
+				     MT_WATCHDOG_TIME);
 
 	set_bit(MT76_STATE_RUNNING, &dev->mt76.state);
drivers/net/wireless/mediatek/mt76/mt76x2/usb_main.c +1 -1
index 2b48cc51a30d..77f95688a41b 100644
@@ -28,7 +28,7 @@ static int mt76x2u_start(struct ieee80211_hw *hw)
 		goto out;
 
 	ieee80211_queue_delayed_work(mt76_hw(dev), &dev->mac_work,
-				     MT_CALIBRATE_INTERVAL);
+				     MT_WATCHDOG_TIME);
 	set_bit(MT76_STATE_RUNNING, &dev->mt76.state);
 
 out:
Signature · 1 line
2.20.1
#2 lore raw

Re: [PATCH 1/2] mt76: run mac work every 100ms

Quoted 2 lines · Lorenzo Bianconi

Introduce MT_WATCHDOG_TIME macro in order to run mac work every 100ms. This is a preliminary patch to implement a proper tx hang watchdog

<snip>

drivers/net/wireless/mediatek/mt76/mt76x2/pci_main.c mt76x2_start(struct ieee80211_hw *hw) · line 36

Quoted 10 lines
drivers/net/wireless/mediatek/mt76/mt76x2/pci_main.c +1 -1
index b54a32397486..ec936002a286 100644
@@ -33,7 +33,7 @@ mt76x2_start(struct ieee80211_hw *hw)
 		goto out;
 
 	ieee80211_queue_delayed_work(mt76_hw(dev), &dev->mac_work,
-				     MT_CALIBRATE_INTERVAL);
+				     MT_WATCHDOG_TIME);

There are other procedures that are performed on mac_work , which most likely don't have to be done so frequently. I think watchdog should be placed it it's own work.

Regards
Stanislaw

author
#3 lore raw

Re: [PATCH 1/2] mt76: run mac work every 100ms

Quoted 12 lines · Lorenzo Bianconi
Quoted 2 lines

Introduce MT_WATCHDOG_TIME macro in order to run mac work every 100ms. This is a preliminary patch to implement a proper tx hang watchdog

<snip>

Quoted 7 lines
drivers/net/wireless/mediatek/mt76/mt76x2/pci_main.c +0 -0
index b54a32397486..ec936002a286 100644
@@ -33,7 +33,7 @@ mt76x2_start(struct ieee80211_hw *hw)
              goto out;
Quoted 7 lines

drivers/net/wireless/mediatek/mt76/mt76x2/pci_main.c mt76x2_start(struct ieee80211_hw *hw) · line 36

Quoted 3 lines
drivers/net/wireless/mediatek/mt76/mt76x2/pci_main.c +1 -1
      ieee80211_queue_delayed_work(mt76_hw(dev), &dev->mac_work,
-                                  MT_CALIBRATE_INTERVAL);
+                                  MT_WATCHDOG_TIME);

There are other procedures that are performed on mac_work , which most likely don't have to be done so frequently. I think watchdog should be placed it it's own work.

I guess it will be useful to have a dedicated work for it if we add new watchdogs (like mt7603 driver).
I will fix it in v2.

Regards,
Lorenzo

Quoted 4 lines

Regards
Stanislaw

author
#4 lore raw

PATCH 2/2 mt76: introduce mt76x02_check_tx_hang watchdog

In reply to Lorenzo Bianconi

Port mt76x02_check_tx_hang watchdog from vendor driver in order to perform a device reset when tx mac/dma logic hangs. Tx mac/dma stuck has been observed when the device is heavily loaded or in a noisy environment.

Signed-off-by Lorenzo Bianconi <[email protected]>
---
 drivers/net/wireless/mediatek/mt76/mt76.h     |  1 +
 drivers/net/wireless/mediatek/mt76/mt76x02.h  |  6 ++
 .../wireless/mediatek/mt76/mt76x02_debugfs.c  |  2 +
 .../net/wireless/mediatek/mt76/mt76x02_mac.c  | 88 ++++++++++++++++++-
 .../net/wireless/mediatek/mt76/mt76x02_mmio.c | 23 +++++
 5 files changed, 119 insertions(+), 1 deletion(-)
drivers/net/wireless/mediatek/mt76/mt76.h +1 -0
index 5cd508a68609..8ef430d8afc4 100644
@@ -421,6 +421,7 @@ struct mt76_dev {
 	struct mt76_queue q_tx[__MT_TXQ_MAX];
 	struct mt76_queue q_rx[__MT_RXQ_MAX];
 	const struct mt76_queue_ops *queue_ops;
+	int tx_dma_idx[4];
 
 	wait_queue_head_t tx_wait;
 	struct sk_buff_head status_list;
drivers/net/wireless/mediatek/mt76/mt76x02.h +6 -0
index 17e13e373f53..09ae4f246131 100644
@@ -29,6 +29,8 @@
 #define MT_WATCHDOG_TIME	(HZ / 10)
 #define MT_CALIBRATE_INTERVAL	HZ
 
+#define MT_TX_HANG_TH		10
+
 #define MT_MAX_CHAINS		2
 struct mt76x02_rx_freq_cal {
 	s8 high_gain[MT_MAX_CHAINS];

@@ -91,6 +93,9 @@ struct mt76x02_dev {
 	u8 tbtt_count;
 	u16 beacon_int;
 
+	u32 tx_hang_reset;
+	u8 tx_hang_check;
+
 	struct mt76x02_calibration cal;
 
 	s8 target_power;

@@ -144,6 +149,7 @@ s8 mt76x02_tx_get_max_txpwr_adj(struct mt76x02_dev *dev,
 				const struct ieee80211_tx_rate *rate);
 s8 mt76x02_tx_get_txpwr_adj(struct mt76x02_dev *dev, s8 txpwr,
 			    s8 max_txpwr_adj);
+bool mt76x02_tx_hang(struct mt76x02_dev *dev);
 void mt76x02_tx_set_txpwr_auto(struct mt76x02_dev *dev, s8 txpwr);
 void mt76x02_set_tx_ackto(struct mt76x02_dev *dev);
 void mt76x02_set_coverage_class(struct ieee80211_hw *hw,
drivers/net/wireless/mediatek/mt76/mt76x02_debugfs.c +2 -0
index a9d52ba1e270..7580c5c986ff 100644
@@ -133,5 +133,7 @@ void mt76x02_init_debugfs(struct mt76x02_dev *dev)
 				    read_txpower);
 
 	debugfs_create_devm_seqfile(dev->mt76.dev, "agc", dir, read_agc);
+
+	debugfs_create_u32("tx_hang_reset", 0400, dir, &dev->tx_hang_reset);
 }
 EXPORT_SYMBOL_GPL(mt76x02_init_debugfs);
drivers/net/wireless/mediatek/mt76/mt76x02_mac.c +87 -1

100 lines not on this page · raw message

drivers/net/wireless/mediatek/mt76/mt76x02_mmio.c +23 -0
index 6974acc75e2b..89d9cb1f45bf 100644
@@ -385,3 +385,26 @@ void mt76x02_mac_start(struct mt76x02_dev *dev)
 			   MT_INT_TX_STAT);
 }
 EXPORT_SYMBOL_GPL(mt76x02_mac_start);
+
+bool mt76x02_tx_hang(struct mt76x02_dev *dev)
+{
+	u32 dma_idx, prev_dma_idx;
+	struct mt76_queue *q;
+	int i;
+
+	for (i = 0; i < 4; i++) {
+		q = &dev->mt76.q_tx[i];
+
+		if (!q->queued)
+			continue;
+
+		prev_dma_idx = dev->mt76.tx_dma_idx[i];
+		dma_idx = ioread32(&q->regs->dma_idx);
+		dev->mt76.tx_dma_idx[i] = dma_idx;
+
+		if (prev_dma_idx == dma_idx)
+			break;
+	}
+
+	return i < 4;
+}
Signature · 1 line
2.20.1
#5 lore raw

Re: [PATCH 2/2] mt76: introduce mt76x02_check_tx_hang watchdog

Quoted 4 lines · Lorenzo Bianconi

Port mt76x02_check_tx_hang watchdog from vendor driver in order to perform a device reset when tx mac/dma logic hangs. Tx mac/dma stuck has been observed when the device is heavily loaded or in a noisy environment.

On what hardware this happen ? Is this only for AP mode?

drivers/net/wireless/mediatek/mt76/mt76x02_mac.c static void mt76x02_check_mac_err(struct mt76x02_dev *dev) · line 781

Quoted 9 lines
drivers/net/wireless/mediatek/mt76/mt76x02_mac.c +1 -0
index a4ff5a7a219f..402944e81e5e 100644
@@ -778,6 +778,90 @@ static void mt76x02_check_mac_err(struct mt76x02_dev *dev)
 		MT_MAC_SYS_CTRL_ENABLE_TX | MT_MAC_SYS_CTRL_ENABLE_RX);
 }
 
+static void mt76x02_mac_watchdog_reset(struct mt76x02_dev *dev)

This all is MMIO specific, it should be moved to mt76x02_mmio.c

Regards
Stanislaw

author
#6 lore raw

Re: [PATCH 2/2] mt76: introduce mt76x02_check_tx_hang watchdog

Quoted 8 lines · Lorenzo Bianconi
Quoted 4 lines

Port mt76x02_check_tx_hang watchdog from vendor driver in order to perform a device reset when tx mac/dma logic hangs. Tx mac/dma stuck has been observed when the device is heavily loaded or in a noisy environment.

On what hardware this happen ? Is this only for AP mode?

Hi Stanislaw,

I an able to trigger the issue using a mt7662 AP device till now, not running STA or other modes yet.
I will respin this patch adding tx watchdog just on mt76x2 devices in AP mode for the moment.

Regards,
Lorenzo

Quoted 15 lines

drivers/net/wireless/mediatek/mt76/mt76x02_mac.c static void mt76x02_check_mac_err(struct mt76x02_dev *dev) · line 781

Quoted 10 lines
drivers/net/wireless/mediatek/mt76/mt76x02_mac.c +1 -0
index a4ff5a7a219f..402944e81e5e 100644
@@ -778,6 +778,90 @@ static void mt76x02_check_mac_err(struct mt76x02_dev *dev)
              MT_MAC_SYS_CTRL_ENABLE_TX | MT_MAC_SYS_CTRL_ENABLE_RX);
 }
 
+static void mt76x02_mac_watchdog_reset(struct mt76x02_dev *dev)

This all is MMIO specific, it should be moved to mt76x02_mmio.c

Regards
Stanislaw