Skip to contentSkip to conversation

wifi: rtw88: usb: keep bmc traffic from exhausting the TX page pool

PATCH Superseded Mehmet Fide 2 participants 7 messages Sep 7 – Sep 10 Realtek wireless driver (rtw88)

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

v2 cover wifi: rtw88: usb: keep bmc traffic from exhausting the TX page pool

From: Mehmet Fide <[email protected]>

An rtw88 USB AP with one station in power save and ordinary multicast chatter (mDNS, SSDP) starves its own TX page pool: every broadcast and multicast frame goes to the after-DTIM high queue, which drains a few frames per DTIM inside a 2 TU ATIM window, and the driver hands over as many as the network produces. On an RTL8822BU the free page count at 0x240 goes from 1803 to 16 in ~100 s and stays there for as long as the traffic lasts; nothing else gets out, nobody can join, the reserved page download fails ("error beacon valid"). Only a reboot recovers.

The series closes this from three sides:

1/3 bounds what the driver feeds the high queue (a small budget below
the measured drain rate, the excess leaves on the AC queue),
2/3 admits only the frames a dozing station actually needs to the
after-DTIM path (ARP, EAPOL, DHCP, the vendor driver's default "allow special" filter), everything else goes out at line rate,
3/3 widens the ATIM window to 0xa while an AP interface is up, as the
vendor driver does, so the high queue drains faster to begin with.

With 1/3 + 2/3 the pool stays at 1803 through a 180 s storm on the RTL8822BU, a DHCP flood still takes the after-DTIM path and is held by the budget, join/ping cycling without power save is unchanged (10/10). 3/3 comes from the review of v1. The beacon-early C2H that Ping-Ke found usable in AP mode is the long-term route to IEEE80211_HW_HOST_BROADCAST_PS_BUFFERING; that is a separate series, this one does not depend on it.

v1: https://lore.kernel.org/linux-wireless/[email protected]/

v2:
- 1/3: macro for the refill interval, 'budget' naming, refill bookkeeping rewritten and commented, blank lines (Ping-Ke)
- 2/3: classification from skb->protocol / ip_hdr() / udp_hdr() and the control port flag as in rtw89, SNAP parser gone, include order (Ping-Ke)
- 3/3: new, REG_ATIMWND 0xa in AP mode, default restored on stop (Ping-Ke)

Mehmet Fide (3):
wifi: rtw88: usb: bound what the driver feeds the after-DTIM queue
wifi: rtw88: usb: only let the frames a dozing station needs use the
after-DTIM queue
wifi: rtw88: widen the ATIM window while an AP interface is up

 drivers/net/wireless/realtek/rtw88/mac80211.c |  2 +
 drivers/net/wireless/realtek/rtw88/reg.h      |  2 +
 drivers/net/wireless/realtek/rtw88/usb.c      | 65 +++++++++++++++++--
 drivers/net/wireless/realtek/rtw88/usb.h      |  5 ++
 4 files changed, 70 insertions(+), 4 deletions(-)

base-commit: 81510c3d6f2199889a4a936d8cf8b9e05911b10e

Signature · 1 line
2.55.0
#1 lore raw

PATCH v2 1/3 wifi: rtw88: usb: bound what the driver feeds the after-DTIM queue

From: Mehmet Fide <[email protected]>

Frames routed to the high queue are transmitted right after DTIM beacons only, inside the ATIM window, while they wait in the shared TX page pool. The driver puts no limit on how many it hands over, so as long as one station dozes, any sustained broadcast or multicast traffic outruns the drain and empties the pool: measured on an RTL8822BU AP with a single client in power save and ~40 frames/s of mDNS chatter, the free page count at 0x240 goes from 1803 to 16 in about 100 seconds and stays there for as long as the traffic lasts. From that point every other transmit queues behind the backlog, authentication responses arrive too late for anyone to join, and the reserved page download fails ("error beacon valid"). The AP keeps beaconing and only a reboot recovers.

Feed the high queue through a small budget that refills below the measured drain rate (about 3 frames per DTIM, ~15/s at dtim_period 2 on the default 2 TU ATIM window); whatever exceeds the budget leaves on its access category queue right away. The high queue backlog is now bounded by the burst size under any load, so the page pool cannot run dry, at the price that a dozing station may miss part of a broadcast storm.

---
v2:
- macro RTW_USB_HIQ_REFILL_INTERVAL for HZ / 10, RTW_USB_HIQ_BUDGET_MAX (Ping-Ke)
- 'budget' instead of 'token': hiq_budget, rtw_usb_hiq_take_budget() (Ping-Ke)
- refill bookkeeping rewritten as elapsed / interval and
hiq_refill = jiffies - elapsed % interval, with a comment (Ping-Ke)
- blank lines after spin_lock_irqsave() and before spin_unlock_irqrestore() (Ping-Ke)

 drivers/net/wireless/realtek/rtw88/usb.c | 41 ++++++++++++++++++++++--
 drivers/net/wireless/realtek/rtw88/usb.h |  5 +++
 2 files changed, 43 insertions(+), 3 deletions(-)
drivers/net/wireless/realtek/rtw88/usb.c +38 -3
index c90802919473..5482e44f4a88 100644
@@ -562,7 +562,37 @@ static int rtw_usb_write_data_h2c(struct rtw_dev *rtwdev, u8 *buf, u32 size)
 	return rtw_usb_write_data(rtwdev, &pkt_info, buf);
 }
 
-static u8 rtw_usb_tx_queue_mapping_to_qsel(struct sk_buff *skb)
+#define RTW_USB_HIQ_REFILL_INTERVAL	(HZ / 10)	/* jiffies per unit of budget */
+#define RTW_USB_HIQ_BUDGET_MAX		16
+
+static bool rtw_usb_hiq_take_budget(struct rtw_usb *rtwusb)
+{
+	unsigned long flags, elapsed, add;
+	bool ok;
+
+	spin_lock_irqsave(&rtwusb->hiq_lock, flags);
+
+	elapsed = jiffies - rtwusb->hiq_refill;
+	add = elapsed / RTW_USB_HIQ_REFILL_INTERVAL;
+	if (add) {
+		rtwusb->hiq_budget = min_t(unsigned long, rtwusb->hiq_budget + add,
+					   RTW_USB_HIQ_BUDGET_MAX);
+		/* The part of the current interval that has not completed yet
+		 * keeps counting toward the next unit
+		 */
+		rtwusb->hiq_refill = jiffies - elapsed % RTW_USB_HIQ_REFILL_INTERVAL;
+	}
+	ok = rtwusb->hiq_budget > 0;
+	if (ok)
+		rtwusb->hiq_budget--;
+
+	spin_unlock_irqrestore(&rtwusb->hiq_lock, flags);
+
+	return ok;
+}
+
+static u8 rtw_usb_tx_queue_mapping_to_qsel(struct rtw_usb *rtwusb,
+					   struct sk_buff *skb)
 {
 	struct ieee80211_hdr *hdr = (struct ieee80211_hdr *)skb->data;
 	struct ieee80211_tx_info *info = IEEE80211_SKB_CB(skb);
@@ -573,7 +603,8 @@ static u8 rtw_usb_tx_queue_mapping_to_qsel(struct sk_buff *skb)
 		qsel = TX_DESC_QSEL_MGMT;
 	else if (is_broadcast_ether_addr(hdr->addr1) ||
 		 is_multicast_ether_addr(hdr->addr1))
-		qsel = (info->flags & IEEE80211_TX_CTL_SEND_AFTER_DTIM) ?
+		qsel = (info->flags & IEEE80211_TX_CTL_SEND_AFTER_DTIM) &&
+		       rtw_usb_hiq_take_budget(rtwusb) ?
 		       TX_DESC_QSEL_HIGH : skb->priority;
 	else if (skb_get_queue_mapping(skb) <= IEEE80211_AC_BK)
 		qsel = skb->priority;
@@ -593,7 +624,7 @@ static int rtw_usb_tx_write(struct rtw_dev *rtwdev,
 	u8 *pkt_desc;
 	int ep;
 
-	pkt_info->qsel = rtw_usb_tx_queue_mapping_to_qsel(skb);
+	pkt_info->qsel = rtw_usb_tx_queue_mapping_to_qsel(rtwusb, skb);
 	pkt_desc = skb_push(skb, chip->tx_pkt_desc_sz);
 	memset(pkt_desc, 0, chip->tx_pkt_desc_sz);
 	ep = qsel_to_ep(rtwusb, pkt_info->qsel);
@@ -1034,6 +1065,10 @@ static int rtw_usb_init_tx(struct rtw_dev *rtwdev)
 	struct rtw_usb *rtwusb = rtw_get_usb_priv(rtwdev);
 	int i;
 
+	spin_lock_init(&rtwusb->hiq_lock);
+	rtwusb->hiq_budget = RTW_USB_HIQ_BUDGET_MAX;
+	rtwusb->hiq_refill = jiffies;
+
 	rtwusb->txwq = create_singlethread_workqueue("rtw88_usb: tx wq");
 	if (!rtwusb->txwq) {
 		rtw_err(rtwdev, "failed to create TX work queue\n");
drivers/net/wireless/realtek/rtw88/usb.h +5 -0
index 9b695b688b24..94ff21dd0846 100644
@@ -75,6 +75,11 @@ struct rtw_usb {
 	u8 out_ep[RTW_USB_EP_MAX];
 	int qsel_to_ep[TX_DESC_QSEL_MAX];
 
+	/* protects hiq_budget and hiq_refill */
+	spinlock_t hiq_lock;
+	u32 hiq_budget;
+	unsigned long hiq_refill;
+
 	struct workqueue_struct *txwq, *rxwq;
 
 	struct sk_buff_head tx_queue[RTW_USB_EP_MAX];
Signature · 1 line
2.55.0
maintainer Acked-by on v2 1/3
#2 lore raw

RE: [PATCH rtw-next v2 1/3] wifi: rtw88: usb: bound what the driver feeds the after-DTIM queue

Quoted 14 lines · Mehmet Fide

From: Mehmet Fide <[email protected]>

Frames routed to the high queue are transmitted right after DTIM beacons only, inside the ATIM window, while they wait in the shared TX page pool. The driver puts no limit on how many it hands over, so as long as one station dozes, any sustained broadcast or multicast traffic outruns the drain and empties the pool: measured on an RTL8822BU AP with a single client in power save and ~40 frames/s of mDNS chatter, the free page count at 0x240 goes from 1803 to 16 in about 100 seconds and stays there for as long as the traffic lasts. From that point every other transmit queues behind the backlog, authentication responses arrive too late for anyone to join, and the reserved page download fails ("error beacon valid"). The AP keeps beaconing and only a reboot recovers.

Quoted 8 lines

Feed the high queue through a small budget that refills below the measured drain rate (about 3 frames per DTIM, ~15/s at dtim_period 2 on the default 2 TU ATIM window); whatever exceeds the budget leaves on its access category queue right away. The high queue backlog is now bounded by the burst size under any load, so the page pool cannot run dry, at the price that a dozing station may miss part of a broadcast storm.

Some minor/nit questions. If you don't change others, just take my acked-by to v3.

Quoted 10 lines
 drivers/net/wireless/realtek/rtw88/usb.c | 41 ++++++++++++++++++++++--
 drivers/net/wireless/realtek/rtw88/usb.h |  5 +++
 2 files changed, 43 insertions(+), 3 deletions(-)
drivers/net/wireless/realtek/rtw88/usb.c +0 -0
index c90802919473..5482e44f4a88 100644

drivers/net/wireless/realtek/rtw88/usb.c static int rtw_usb_write_data_h2c(struct rtw_dev *rtwdev, u8 *buf, u32 size) · line 565

Quoted 5 lines
drivers/net/wireless/realtek/rtw88/usb.c +1 -1
@@ -562,7 +562,37 @@ static int rtw_usb_write_data_h2c(struct rtw_dev *rtwdev, u8 *buf, u32 size)
        return rtw_usb_write_data(rtwdev, &pkt_info, buf);
 }
 
-static u8 rtw_usb_tx_queue_mapping_to_qsel(struct sk_buff *skb)
+#define RTW_USB_HIQ_REFILL_INTERVAL    (HZ / 10)       /* jiffies per unit of budget */

Does it mean a budget per 100ms roughly?

Quoted 7 lines
drivers/net/wireless/realtek/rtw88/usb.c +7 -0
+#define RTW_USB_HIQ_BUDGET_MAX         16
+
+static bool rtw_usb_hiq_take_budget(struct rtw_usb *rtwusb)
+{
+       unsigned long flags, elapsed, add;
+       bool ok;
+

drivers/net/wireless/realtek/rtw88/usb.c static int rtw_usb_write_data_h2c(struct rtw_dev *rtwdev, u8 *buf, u32 size) · line 582

Quoted 10 lines
drivers/net/wireless/realtek/rtw88/usb.c +10 -0
+       spin_lock_irqsave(&rtwusb->hiq_lock, flags);
+
+       elapsed = jiffies - rtwusb->hiq_refill;
+       add = elapsed / RTW_USB_HIQ_REFILL_INTERVAL;
+       if (add) {
+               rtwusb->hiq_budget = min_t(unsigned long, rtwusb->hiq_budget + add,
+                                          RTW_USB_HIQ_BUDGET_MAX);
+               /* The part of the current interval that has not completed yet
+                * keeps counting toward the next unit
+                */

First line of comment block should be empty.

drivers/net/wireless/realtek/rtw88/usb.c static int rtw_usb_write_data_h2c(struct rtw_dev *rtwdev, u8 *buf, u32 size) · line 584

Quoted 2 lines
drivers/net/wireless/realtek/rtw88/usb.c +2 -0
+               rtwusb->hiq_refill = jiffies - elapsed % RTW_USB_HIQ_REFILL_INTERVAL;
+       }

nit: an empty line

Quoted 14 lines
drivers/net/wireless/realtek/rtw88/usb.c +11 -0
+       ok = rtwusb->hiq_budget > 0;
+       if (ok)
+               rtwusb->hiq_budget--;
+
+       spin_unlock_irqrestore(&rtwusb->hiq_lock, flags);
+
+       return ok;
+}
+
+static u8 rtw_usb_tx_queue_mapping_to_qsel(struct rtw_usb *rtwusb,
+                                          struct sk_buff *skb)
 {
        struct ieee80211_hdr *hdr = (struct ieee80211_hdr *)skb->data;
        struct ieee80211_tx_info *info = IEEE80211_SKB_CB(skb);
#3 lore raw

PATCH v2 2/3 wifi: rtw88: usb: only let the frames a dozing station needs use the after-DTIM queue

In reply to Mehmet Fide

From: Mehmet Fide <[email protected]>

With the budget in place the high queue can no longer take the chip down, but ordinary chatter still competes with the frames a sleeping station actually needs. mac80211 marks every broadcast and multicast frame with IEEE80211_TX_CTL_SEND_AFTER_DTIM while a station dozes, mDNS and SSDP included, so under normal traffic the budget is spent on frames nobody waits for.

Do what the vendor driver does with its default "allow special" high queue filter: admit only ARP, EAPOL and DHCP to the after-DTIM path, the frames a station coming out of power save has to see; everything else goes out on its access category queue at line rate. The frames are classified the way rtw89_core_tx_btc_spec_pkt_notify() does it, from skb->protocol and the network and transport header offsets that mac80211 keeps from the netdev path; EAPOL is the control port flag. Frames the AP relays between its own stations arrive as ETH_P_802_3 and are not inspected, they leave on the access category queue.

With the filter in place the page pool stays at 1803 through the same 180 second storm, a DHCP flood still takes the after-DTIM path (and is then held by the budget), and join/ping cycling without power save is unchanged (10/10).

---
v2:
- classify from skb->protocol, ip_hdr() and udp_hdr() like rtw89_core_tx_btc_spec_pkt_notify(), EAPOL from the control port flag; the SNAP/IV parser and linux/unaligned.h are gone (Ping-Ke)
- <linux/*> includes in alphabetical order (Ping-Ke)
- relayed (ETH_P_802_3) frames documented as not inspected

 drivers/net/wireless/realtek/rtw88/usb.c | 26 ++++++++++++++++++++++--
 1 file changed, 24 insertions(+), 2 deletions(-)
drivers/net/wireless/realtek/rtw88/usb.c +24 -2
index 5482e44f4a88..473b63d16658 100644
@@ -2,9 +2,11 @@
 /* Copyright(c) 2018-2019  Realtek Corporation
  */
 
+#include <linux/ip.h>
 #include <linux/module.h>
-#include <linux/usb.h>
 #include <linux/mutex.h>
+#include <linux/udp.h>
+#include <linux/usb.h>
 #include "main.h"
 #include "debug.h"
 #include "mac.h"
@@ -562,6 +564,26 @@ static int rtw_usb_write_data_h2c(struct rtw_dev *rtwdev, u8 *buf, u32 size)
 	return rtw_usb_write_data(rtwdev, &pkt_info, buf);
 }
 
+static bool rtw_usb_bmc_needs_dtim(struct sk_buff *skb)
+{
+	struct ieee80211_tx_info *info = IEEE80211_SKB_CB(skb);
+	struct udphdr *udphdr;
+
+	if (info->control.flags & IEEE80211_TX_CTRL_PORT_CTRL_PROTO)
+		return true;
+
+	if (skb->protocol == htons(ETH_P_ARP))
+		return true;
+
+	if (skb->protocol != htons(ETH_P_IP) ||
+	    ip_hdr(skb)->protocol != IPPROTO_UDP)
+		return false;
+
+	udphdr = udp_hdr(skb);
+
+	return udphdr->dest == htons(67) || udphdr->dest == htons(68);
+}
+
 #define RTW_USB_HIQ_REFILL_INTERVAL	(HZ / 10)	/* jiffies per unit of budget */
 #define RTW_USB_HIQ_BUDGET_MAX		16
 
@@ -604,7 +626,7 @@ static u8 rtw_usb_tx_queue_mapping_to_qsel(struct rtw_usb *rtwusb,
 	else if (is_broadcast_ether_addr(hdr->addr1) ||
 		 is_multicast_ether_addr(hdr->addr1))
 		qsel = (info->flags & IEEE80211_TX_CTL_SEND_AFTER_DTIM) &&
-		       rtw_usb_hiq_take_budget(rtwusb) ?
+		       rtw_usb_bmc_needs_dtim(skb) && rtw_usb_hiq_take_budget(rtwusb) ?
 		       TX_DESC_QSEL_HIGH : skb->priority;
 	else if (skb_get_queue_mapping(skb) <= IEEE80211_AC_BK)
 		qsel = skb->priority;
Signature · 1 line
2.55.0
maintainer
#4 lore raw

RE: [PATCH rtw-next v2 2/3] wifi: rtw88: usb: only let the frames a dozing station needs use the after-DTIM queue

drivers/net/wireless/realtek/rtw88/usb.c static int rtw_usb_write_data_h2c(struct rtw_dev *rtwdev, u8 *buf, u32 size) · line 579

Quoted 13 lines · Mehmet Fide
drivers/net/wireless/realtek/rtw88/usb.c +13 -0
+static bool rtw_usb_bmc_needs_dtim(struct sk_buff *skb)
+{
+       struct ieee80211_tx_info *info = IEEE80211_SKB_CB(skb);
+       struct udphdr *udphdr;
+
+       if (info->control.flags & IEEE80211_TX_CTRL_PORT_CTRL_PROTO)
+               return true;
+
+       if (skb->protocol == htons(ETH_P_ARP))
+               return true;
+
+       if (skb->protocol != htons(ETH_P_IP) ||
+           ip_hdr(skb)->protocol != IPPROTO_UDP)

Prefer to list them in positive way.

	if (skb->protocol == htons(ETH_P_IP) &&
	    ip_hdr(skb)->protocol == IPPROTO_UDP) {
		udphdr = udp_hdr(skb);

		return udphdr->dest == htons(67) || udphdr->dest == htons(68);
	}

	return false;	
Quoted 7 lines
drivers/net/wireless/realtek/rtw88/usb.c +7 -0
+               return false;
+
+       udphdr = udp_hdr(skb);
+
+       return udphdr->dest == htons(67) || udphdr->dest == htons(68);
+}
+
#5 lore raw

PATCH v2 3/3 wifi: rtw88: widen the ATIM window while an AP interface is up

In reply to Mehmet Fide

From: Mehmet Fide <[email protected]>

The frames held for dozing stations go out right after the DTIM beacon and only for the duration of the ATIM window. The chips come up with a 2 TU window (the two that program it set 0x2 at init), which drains about three frames per DTIM; the vendor driver sets 0xa or 0xc in AP mode instead. With the 2 TU window the measured page pool exhaustion takes ~100 s of ordinary multicast chatter; at 0x4 the same storm already drains, so 0xa leaves comfortable headroom.

Program 0xa when an AP interface starts and put the default back when it stops, since most chips never initialize the register themselves. The bound and the filter of the previous patches still apply: the window adds drain capacity, the budget protects the pool when that is not enough, the filter keeps chatter off the beacon-paced path.

---
v2:
- new patch, REG_ATIMWND 0xa in AP mode as in the vendor driver, back to the 0x2 default on stop_ap since most chips never program it (Ping-Ke)

 drivers/net/wireless/realtek/rtw88/mac80211.c | 2 ++
 drivers/net/wireless/realtek/rtw88/reg.h      | 2 ++
 2 files changed, 4 insertions(+)
drivers/net/wireless/realtek/rtw88/mac80211.c +2 -0
index 0b622d7663b6..2a9b09fa76e7 100644
@@ -474,6 +474,7 @@ static int rtw_ops_start_ap(struct ieee80211_hw *hw,
 
 	mutex_lock(&rtwdev->mutex);
 	rtw_write32_set(rtwdev, REG_TCR, BIT_TCR_UPDATE_HGQMD);
+	rtw_write16(rtwdev, REG_ATIMWND, ATIMWND_AP);
 	rtwdev->ap_active = true;
 	rtw_store_op_chan(rtwdev, true);
 	chip->ops->phy_calibration(rtwdev);
@@ -490,6 +491,7 @@ static void rtw_ops_stop_ap(struct ieee80211_hw *hw,
 
 	mutex_lock(&rtwdev->mutex);
 	rtw_write32_clr(rtwdev, REG_TCR, BIT_TCR_UPDATE_HGQMD);
+	rtw_write16(rtwdev, REG_ATIMWND, ATIMWND_DEFAULT);
 	rtwdev->ap_active = false;
 	if (!rtw_core_check_sta_active(rtwdev))
 		rtw_clear_op_chan(rtwdev);
drivers/net/wireless/realtek/rtw88/reg.h +2 -0
index 08e9494977e0..0b9ba6f37680 100644
@@ -483,6 +483,8 @@
 #define REG_DRVERLYINT		0x0558
 #define REG_BCNDMATIM		0x0559
 #define REG_ATIMWND		0x055A
+#define ATIMWND_DEFAULT		0x2
+#define ATIMWND_AP		0xa
 #define REG_USTIME_TSF		0x055C
 #define REG_BCN_MAX_ERR		0x055D
 #define REG_RXTSF_OFFSET_CCK	0x055E
Signature · 1 line
2.55.0
maintainer Acked-by on v2 3/3
#6 lore raw

RE: [PATCH rtw-next v2 3/3] wifi: rtw88: widen the ATIM window while an AP interface is up

Quoted 10 lines · Mehmet Fide

From: Mehmet Fide <[email protected]>

The frames held for dozing stations go out right after the DTIM beacon and only for the duration of the ATIM window. The chips come up with a 2 TU window (the two that program it set 0x2 at init), which drains about three frames per DTIM; the vendor driver sets 0xa or 0xc in AP mode instead. With the 2 TU window the measured page pool exhaustion takes ~100 s of ordinary multicast chatter; at 0x4 the same storm already drains, so 0xa leaves comfortable headroom.

Quoted 8 lines

Program 0xa when an AP interface starts and put the default back when it stops, since most chips never initialize the register themselves. The bound and the filter of the previous patches still apply: the window adds drain capacity, the budget protects the pool when that is not enough, the filter keeps chatter off the beacon-paced path.