wifi: rtw88: usb: keep bmc traffic from exhausting the TX page pool
v3 cover wifi: rtw88: usb: keep bmc traffic from exhausting the TX page pool
From: Mehmet Fide <[email protected]>
An rtw88 USB AP with one station in power save and ordinary multicast chatter (mDNS, SSDP) starves its own TX page pool: every broadcast and multicast frame goes to the after-DTIM high queue, which drains a few frames per DTIM inside a 2 TU ATIM window, and the driver hands over as many as the network produces. On an RTL8822BU the free page count at 0x240 goes from 1803 to 16 in ~100 s and stays there for as long as the traffic lasts; nothing else gets out, nobody can join, the reserved page download fails ("error beacon valid"). Only a reboot recovers.
The series closes this from three sides:
1/3 bounds what the driver feeds the high queue (a small budget below
the measured drain rate, the excess leaves on the AC queue),
2/3 admits only the frames a dozing station actually needs to the
after-DTIM path (ARP, EAPOL, DHCP, the vendor driver's default "allow special" filter), everything else goes out at line rate,
3/3 widens the ATIM window to 0xa while an AP interface is up, as the
vendor driver does, so the high queue drains faster to begin with.
With 1/3 + 2/3 the pool stays at 1803 through a 180 s storm on the RTL8822BU, a DHCP flood still takes the after-DTIM path and is held by the budget, join/ping cycling without power save is unchanged (10/10). The beacon-early C2H that Ping-Ke found usable in AP mode is the long-term route to IEEE80211_HW_HOST_BROADCAST_PS_BUFFERING; that is a separate series, this one does not depend on it.
v3 is the review of v2 only, no functional change, so the v2 bench results above still apply.
v1: https://lore.kernel.org/linux-wireless/[email protected]/
v2: https://lore.kernel.org/linux-wireless/[email protected]/
v3:
- rebased on rtw-next
- 1/3: the refill macro comment now says what one interval buys, the bookkeeping comment is a single line and the block got the blank lines (Ping-Ke); Acked-by added
- 2/3: the IP/UDP test written in positive form (Ping-Ke)
- 3/3: unchanged, Acked-by added
v2:
- 1/3: macro for the refill interval, 'budget' naming, refill bookkeeping rewritten and commented, blank lines (Ping-Ke)
- 2/3: classification from skb->protocol / ip_hdr() / udp_hdr() and the control port flag as in rtw89, SNAP parser gone, include order (Ping-Ke)
- 3/3: new, REG_ATIMWND 0xa in AP mode, default restored on stop (Ping-Ke)
Mehmet Fide (3):
wifi: rtw88: usb: bound what the driver feeds the after-DTIM queue
wifi: rtw88: usb: only let the frames a dozing station needs use the
after-DTIM queue
wifi: rtw88: widen the ATIM window while an AP interface is up
drivers/net/wireless/realtek/rtw88/mac80211.c | 2 + drivers/net/wireless/realtek/rtw88/reg.h | 2 + drivers/net/wireless/realtek/rtw88/usb.c | 66 +++++++++++++++++-- drivers/net/wireless/realtek/rtw88/usb.h | 5 ++ 4 files changed, 71 insertions(+), 4 deletions(-)
base-commit: f71dd599a98182d6bc34dce39977f928068ecd64
Signature · 1 line
2.55.0
PATCH v3 1/3 wifi: rtw88: usb: bound what the driver feeds the after-DTIM queue
From: Mehmet Fide <[email protected]>
Frames routed to the high queue are transmitted right after DTIM beacons only, inside the ATIM window, while they wait in the shared TX page pool. The driver puts no limit on how many it hands over, so as long as one station dozes, any sustained broadcast or multicast traffic outruns the drain and empties the pool: measured on an RTL8822BU AP with a single client in power save and ~40 frames/s of mDNS chatter, the free page count at 0x240 goes from 1803 to 16 in about 100 seconds and stays there for as long as the traffic lasts. From that point every other transmit queues behind the backlog, authentication responses arrive too late for anyone to join, and the reserved page download fails ("error beacon valid"). The AP keeps beaconing and only a reboot recovers.
Feed the high queue through a small budget that refills below the measured drain rate (about 3 frames per DTIM, ~15/s at dtim_period 2 on the default 2 TU ATIM window); whatever exceeds the budget leaves on its access category queue right away. The high queue backlog is now bounded by the burst size under any load, so the page pool cannot run dry, at the price that a dozing station may miss part of a broadcast storm.
---
v3:
- the refill macro comment says what one interval buys (Ping-Ke)
- the bookkeeping comment is a single line, blank lines around the refill block (Ping-Ke)
drivers/net/wireless/realtek/rtw88/usb.c | 41 ++++++++++++++++++++++-- drivers/net/wireless/realtek/rtw88/usb.h | 5 +++ 2 files changed, 43 insertions(+), 3 deletions(-)
drivers/net/wireless/realtek/rtw88/usb.c +38 -3
@@ -562,7 +562,37 @@ static int rtw_usb_write_data_h2c(struct rtw_dev *rtwdev, u8 *buf, u32 size)return rtw_usb_write_data (rtwdev, &pkt_info, buf); } -static u8 rtw_usb_tx_queue_mapping_to_qsel (struct sk_buff *skb) +#define RTW_USB_HIQ_REFILL_INTERVAL (HZ / 10)/* one unit of budget per 100 ms */ +#define RTW_USB_HIQ_BUDGET_MAX 16 + +static bool rtw_usb_hiq_take_budget (struct rtw_usb *rtwusb) +{ +unsigned long flags, elapsed, add; +bool ok; + +spin_lock_irqsave (&rtwusb->hiq_lock, flags); + + elapsed = jiffies - rtwusb->hiq_refill; + add = elapsed /RTW_USB_HIQ_REFILL_INTERVAL ; +if (add) { + rtwusb->hiq_budget = min_t(unsigned long, rtwusb->hiq_budget + add, +RTW_USB_HIQ_BUDGET_MAX ); + +/* keep the unfinished part of the interval for the next unit */ + rtwusb->hiq_refill = jiffies - elapsed %RTW_USB_HIQ_REFILL_INTERVAL ; + } + + ok = rtwusb->hiq_budget >0 ; +if (ok) + rtwusb->hiq_budget--; + +spin_unlock_irqrestore (&rtwusb->hiq_lock, flags); + +return ok; +} + +static u8 rtw_usb_tx_queue_mapping_to_qsel (struct rtw_usb *rtwusb, +struct sk_buff *skb) {struct ieee80211_hdr *hdr = (struct ieee80211_hdr *)skb->data;struct ieee80211_tx_info *info =IEEE80211_SKB_CB (skb);
@@ -573,7 +603,8 @@ static u8 rtw_usb_tx_queue_mapping_to_qsel(struct sk_buff *skb) qsel =TX_DESC_QSEL_MGMT ; else if (is_broadcast_ether_addr(hdr->addr1) || is_multicast_ether_addr(hdr->addr1)) - qsel = (info->flags &IEEE80211_TX_CTL_SEND_AFTER_DTIM ) ? + qsel = (info->flags &IEEE80211_TX_CTL_SEND_AFTER_DTIM ) && +rtw_usb_hiq_take_budget (rtwusb) ?TX_DESC_QSEL_HIGH : skb->priority; else if (skb_get_queue_mapping(skb) <= IEEE80211_AC_BK) qsel = skb->priority;
@@ -593,7 +624,7 @@ static int rtw_usb_tx_write(struct rtw_dev *rtwdev,u8 *pkt_desc;int ep; - pkt_info->qsel =rtw_usb_tx_queue_mapping_to_qsel (skb); + pkt_info->qsel =rtw_usb_tx_queue_mapping_to_qsel (rtwusb, skb); pkt_desc =skb_push (skb, chip->tx_pkt_desc_sz);memset (pkt_desc,0 , chip->tx_pkt_desc_sz); ep =qsel_to_ep (rtwusb, pkt_info->qsel);
@@ -1034,6 +1065,10 @@ static int rtw_usb_init_tx(struct rtw_dev *rtwdev)struct rtw_usb *rtwusb =rtw_get_usb_priv (rtwdev);int i; +spin_lock_init (&rtwusb->hiq_lock); + rtwusb->hiq_budget =RTW_USB_HIQ_BUDGET_MAX ; + rtwusb->hiq_refill = jiffies; + rtwusb->txwq =create_singlethread_workqueue ("rtw88_usb: tx wq" );if (!rtwusb->txwq) {rtw_err (rtwdev,"failed to create TX work queue\n" );
drivers/net/wireless/realtek/rtw88/usb.h +5 -0
@@ -75,6 +75,11 @@ struct rtw_usb {u8 out_ep[RTW_USB_EP_MAX ];int qsel_to_ep[TX_DESC_QSEL_MAX ]; +/* protects hiq_budget and hiq_refill */ +spinlock_t hiq_lock; +u32 hiq_budget; +unsigned long hiq_refill; +struct workqueue_struct *txwq, *rxwq;struct sk_buff_head tx_queue[RTW_USB_EP_MAX ];
Signature · 1 line
2.55.0
Ping-Ke Shih
maintainerRe: [PATCH rtw-next v3 1/3] wifi: rtw88: usb: bound what the driver feeds the after-DTIM queue
Quoted 21 lines · Mehmet Fide
From: Mehmet Fide <[email protected]>
Frames routed to the high queue are transmitted right after DTIM beacons only, inside the ATIM window, while they wait in the shared TX page pool. The driver puts no limit on how many it hands over, so as long as one station dozes, any sustained broadcast or multicast traffic outruns the drain and empties the pool: measured on an RTL8822BU AP with a single client in power save and ~40 frames/s of mDNS chatter, the free page count at
0x240goes from 1803 to 16 in about 100 seconds and stays there for as long as the traffic lasts. From that point every other transmit queues behind the backlog, authentication responses arrive too late for anyone to join, and the reserved page download fails ("error beacon valid"). The AP keeps beaconing and only a reboot recovers.Feed the high queue through a small budget that refills below the measured drain rate (about 3 frames per DTIM, ~15/s at
dtim_period2 on the default 2 TU ATIM window); whatever exceeds the budget leaves on its access category queue right away. The high queue backlog is now bounded by the burst size under any load, so the page pool cannot run dry, at the price that a dozing station may miss part of a broadcast storm.
Quoted 2 lines
Signed-off-by Mehmet Fide <[email protected]>Acked-by Ping-Ke Shih <[email protected]>
3 patch(es) applied to rtw-next branch of rtw.git, thanks.
04cc8c59a3d7 wifi: rtw88: usb: bound what the driver feeds the after-DTIM queue 185971f1b4eb wifi: rtw88: usb: only let the frames a dozing station needs use the after-DTIM queue cf8b516f6765 wifi: rtw88: widen the ATIM window while an AP interface is up
PATCH v3 2/3 wifi: rtw88: usb: only let the frames a dozing station needs use the after-DTIM queue
In reply to Mehmet FideFrom: Mehmet Fide <[email protected]>
With the budget in place the high queue can no longer take the chip down, but ordinary chatter still competes with the frames a sleeping station actually needs. mac80211 marks every broadcast and multicast frame with IEEE80211_TX_CTL_SEND_AFTER_DTIM while a station dozes, mDNS and SSDP included, so under normal traffic the budget is spent on frames nobody waits for.
Do what the vendor driver does with its default "allow special" high queue filter: admit only ARP, EAPOL and DHCP to the after-DTIM path, the frames a station coming out of power save has to see; everything else goes out on its access category queue at line rate. The frames are classified the way rtw89_core_tx_btc_spec_pkt_notify() does it, from skb->protocol and the network and transport header offsets that mac80211 keeps from the netdev path; EAPOL is the control port flag. Frames the AP relays between its own stations arrive as ETH_P_802_3 and are not inspected, they leave on the access category queue.
With the filter in place the page pool stays at 1803 through the same 180 second storm, a DHCP flood still takes the after-DTIM path (and is then held by the budget), and join/ping cycling without power save is unchanged (10/10).
---
v3:
- the IP/UDP test written in positive form (Ping-Ke)
drivers/net/wireless/realtek/rtw88/usb.c | 27 ++++++++++++++++++++++-- 1 file changed, 25 insertions(+), 2 deletions(-)
drivers/net/wireless/realtek/rtw88/usb.c +25 -2
@@ -2,9 +2,11 @@/* Copyright(c) 2018-2019 Realtek Corporation */ +#include <linux/ip.h> #include <linux/module.h> -#include <linux/usb.h> #include <linux/mutex.h> +#include <linux/udp.h> +#include <linux/usb.h> #include "main.h" #include "debug.h" #include "mac.h"
@@ -562,6 +564,27 @@ static int rtw_usb_write_data_h2c(struct rtw_dev *rtwdev, u8 *buf, u32 size)return rtw_usb_write_data (rtwdev, &pkt_info, buf); } +static bool rtw_usb_bmc_needs_dtim (struct sk_buff *skb) +{ +struct ieee80211_tx_info *info =IEEE80211_SKB_CB (skb); +struct udphdr *udphdr; + +if (info->control.flags &IEEE80211_TX_CTRL_PORT_CTRL_PROTO ) +return true; + +if (skb->protocol ==htons (ETH_P_ARP )) +return true; + +if (skb->protocol ==htons (ETH_P_IP ) && +ip_hdr (skb)->protocol ==IPPROTO_UDP ) { + udphdr =udp_hdr (skb); + +return udphdr->dest ==htons (67 ) || udphdr->dest ==htons (68 ); + } + +return false; +} +#define RTW_USB_HIQ_REFILL_INTERVAL (HZ / 10)/* one unit of budget per 100 ms */ #define RTW_USB_HIQ_BUDGET_MAX 16
@@ -604,7 +627,7 @@ static u8 rtw_usb_tx_queue_mapping_to_qsel(struct rtw_usb *rtwusb,else if (is_broadcast_ether_addr (hdr->addr1) ||is_multicast_ether_addr (hdr->addr1)) qsel = (info->flags &IEEE80211_TX_CTL_SEND_AFTER_DTIM ) && -rtw_usb_hiq_take_budget (rtwusb) ? +rtw_usb_bmc_needs_dtim (skb) &&rtw_usb_hiq_take_budget (rtwusb) ?TX_DESC_QSEL_HIGH : skb->priority;else if (skb_get_queue_mapping (skb) <=IEEE80211_AC_BK ) qsel = skb->priority;
Signature · 1 line
2.55.0
RE: [PATCH rtw-next v3 2/3] wifi: rtw88: usb: only let the frames a dozing station needs use the after-DTIM queue
Quoted 19 lines · Mehmet Fide
From: Mehmet Fide <[email protected]>
With the budget in place the high queue can no longer take the chip down, but ordinary chatter still competes with the frames a sleeping station actually needs. mac80211 marks every broadcast and multicast frame with
IEEE80211_TX_CTL_SEND_AFTER_DTIMwhile a station dozes, mDNS and SSDP included, so under normal traffic the budget is spent on frames nobody waits for.Do what the vendor driver does with its default "allow special" high queue filter: admit only ARP, EAPOL and DHCP to the after-DTIM path, the frames a station coming out of power save has to see; everything else goes out on its access category queue at line rate. The frames are classified the way
rtw89_core_tx_btc_spec_pkt_notify()does it, fromskb->protocoland the network and transport header offsets that mac80211 keeps from the netdev path; EAPOL is the control port flag. Frames the AP relays between its own stations arrive asETH_P_802_3and are not inspected, they leave on the access category queue.
Quoted 6 lines
With the filter in place the page pool stays at 1803 through the same 180 second storm, a DHCP flood still takes the after-DTIM path (and is then held by the budget), and join/ping cycling without power save is unchanged (10/10).
Signed-off-by Mehmet Fide <[email protected]>
PATCH v3 3/3 wifi: rtw88: widen the ATIM window while an AP interface is up
In reply to Mehmet FideFrom: Mehmet Fide <[email protected]>
The frames held for dozing stations go out right after the DTIM beacon and only for the duration of the ATIM window. The chips come up with a 2 TU window (the two that program it set 0x2 at init), which drains about three frames per DTIM; the vendor driver sets 0xa or 0xc in AP mode instead. With the 2 TU window the measured page pool exhaustion takes ~100 s of ordinary multicast chatter; at 0x4 the same storm already drains, so 0xa leaves comfortable headroom.
Program 0xa when an AP interface starts and put the default back when it stops, since most chips never initialize the register themselves. The bound and the filter of the previous patches still apply: the window adds drain capacity, the budget protects the pool when that is not enough, the filter keeps chatter off the beacon-paced path.
--- drivers/net/wireless/realtek/rtw88/mac80211.c | 2 ++ drivers/net/wireless/realtek/rtw88/reg.h | 2 ++ 2 files changed, 4 insertions(+)
drivers/net/wireless/realtek/rtw88/mac80211.c +2 -0
@@ -474,6 +474,7 @@ static int rtw_ops_start_ap(struct ieee80211_hw *hw,mutex_lock (&rtwdev->mutex);rtw_write32_set (rtwdev,REG_TCR ,BIT_TCR_UPDATE_HGQMD ); +rtw_write16 (rtwdev,REG_ATIMWND ,ATIMWND_AP ); rtwdev->ap_active = true;rtw_store_op_chan (rtwdev, true); chip->ops->phy_calibration (rtwdev);
@@ -490,6 +491,7 @@ static void rtw_ops_stop_ap(struct ieee80211_hw *hw,mutex_lock (&rtwdev->mutex);rtw_write32_clr (rtwdev,REG_TCR ,BIT_TCR_UPDATE_HGQMD ); +rtw_write16 (rtwdev,REG_ATIMWND ,ATIMWND_DEFAULT ); rtwdev->ap_active = false;if (!rtw_core_check_sta_active (rtwdev))rtw_clear_op_chan (rtwdev);
drivers/net/wireless/realtek/rtw88/reg.h +2 -0
@@ -483,6 +483,8 @@#define REG_DRVERLYINT 0x0558#define REG_BCNDMATIM 0x0559#define REG_ATIMWND 0x055A +#define ATIMWND_DEFAULT 0x2 +#define ATIMWND_AP 0xa#define REG_USTIME_TSF 0x055C#define REG_BCN_MAX_ERR 0x055D#define REG_RXTSF_OFFSET_CCK 0x055E
Signature · 1 line
2.55.0