ath11k/ath12k: implement TX flow control
v2 cover ath11k/ath12k: implement TX flow control
This series implements custom wake_tx_queue operations for ath11k and ath12k drivers to prevent hardware ring overflow issues under heavy traffic.
Without proper flow control, both drivers experience -ENOMEM errors ("failed to transmit frame -12") when the hardware TCL ring fills up. Additionally, ath12k can hang under sustained high throughput. These issues are more commonly observed in VMs with PCIe passthrough but also occur on bare metal systems.
The implementation follows the pattern used in the iwlwifi driver, checking hardware ring space before dequeuing packets from mac80211. Per-ring locking serializes ring access while allowing TX completions to run, preventing deadlocks.
Testing shows stable operation with eliminated -ENOMEM errors, no hangs, and improved throughput under heavy traffic conditions.
Jose Ignacio Tornos Martinez (3):
mac80211: add ieee80211_tx_peek API
wifi: ath11k: implement custom wake_tx_queue with flow control
wifi: ath12k: implement custom wake_tx_queue with flow control
v2: Address feedback from Zhi-Jun You:
- Add ieee80211_tx_peek() API in mac80211 to peek at the next frame without dequeuing, enabling accurate ring selection before dequeue
- Use get_ring_selector() on the peeked skb to determine the exact target ring matching dp_tx, fixing ring mismatch on multi-ring platforms (ipq8074, wcn6750, qcn9274, wcn7850)
Address feedback from Tamizh Chelvam Raja:
- Fix ath12k radio assignment: obtain ar from txq->vif through the arvif chain instead of ah->radio flexible array
Move CRASH_FLUSH check inside the TX loop
v1: https://lore.kernel.org/all/[email protected]/
--
2.49.0
PATCH v2 1/3 mac80211: add ieee80211_tx_peek API
Add ieee80211_tx_peek() to allow drivers to inspect the next frame in a TXQ without removing it.
Drivers implementing custom wake_tx_queue operations may need to determine which hardware TX ring to use before dequeuing a packet, using properties like the skb hash or queue mapping that are only available from the skb itself.
The function checks pending fragments first, then iterates through all fair-queue flows (new_flows and old_flows) to find the first queued frame, matching the iteration behavior of fq_tin_dequeue. It also pre-caches the skb hash via skb_get_hash() so that hash-based ring selectors see a stable value through the subsequent dequeue and TX path.
---
New in v2
include/net/mac80211.h | 20 ++++++++++++++++++++ net/mac80211/tx.c | 39 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 59 insertions(+)
include/net/mac80211.h +20 -0
@@ -7742,6 +7742,26 @@ void ieee80211_unreserve_tid(struct ieee80211_sta *sta, u8 tid);struct sk_buff *ieee80211_tx_dequeue (struct ieee80211_hw *hw,struct ieee80211_txq *txq); +/** +* ieee80211_tx_peek - peek at the next packet in a software tx queue +* +* @hw: pointer as obtained from ieee80211_alloc_hw() +* @txq: pointer obtained from station or virtual interface, or from +* ieee80211_next_txq() +* +* Return: the next skb without dequeuing it, or %NULL if the queue is empty. +* The returned pointer is const — the caller must not modify or free the skb. +* The skb remains queued and will be returned by a subsequent +* ieee80211_tx_dequeue() call. +* +* This is useful for drivers that need to inspect the next frame (e.g. to +* determine the target TX ring) before deciding whether to dequeue. +* +* Must be called in the same context as ieee80211_tx_dequeue(). +*/ +const struct sk_buff *ieee80211_tx_peek (struct ieee80211_hw *hw, +struct ieee80211_txq *txq); +/** * ieee80211_tx_dequeue_ni - dequeue a packet from a software tx queue * (in process context)
net/mac80211/tx.c +39 -0
@@ -3857,6 +3857,45 @@ static bool ieee80211_xmit_fast(struct ieee80211_sub_if_data *sdata,return true; } +const struct sk_buff *ieee80211_tx_peek (struct ieee80211_hw *hw, +struct ieee80211_txq *txq) +{ + struct txq_info *txqi = container_of(txq, struct txq_info, txq); +struct ieee80211_local *local =hw_to_local (hw); +struct fq_tin *tin = &txqi->tin; +struct fq *fq = &local->fq; +struct sk_buff *skb =NULL ; +struct fq_flow *flow; + +WARN_ON_ONCE (softirq_count () ==0 ); + +spin_lock_bh (&fq->lock); + + skb =skb_peek (&txqi->frags); +if (skb) + goto out; + +list_for_each_entry (flow, &tin->new_flows, flowchain) { + skb =skb_peek (&flow->queue); +if (skb) + goto out; + } + +list_for_each_entry (flow, &tin->old_flows, flowchain) { + skb =skb_peek (&flow->queue); +if (skb) + goto out; + } + +out: +if (skb) +skb_get_hash (skb); + +spin_unlock_bh (&fq->lock); +return skb; +} +EXPORT_SYMBOL_GPL (ieee80211_tx_peek); +struct sk_buff *ieee80211_tx_dequeue (struct ieee80211_hw *hw,struct ieee80211_txq *txq) {
Signature · 1 line
2.54.0
PATCH v2 2/3 wifi: ath11k: implement custom wake_tx_queue with flow control
In reply to Jose Ignacio Tornos MartinezUnder heavy traffic, ath11k experiences frequent -ENOMEM errors ("failed to transmit frame -12") when the hardware TCL ring fills up. This issue is more commonly observed in VMs with PCIe passthrough but also occurs on bare metal systems. It is particularly problematic on devices with a single shared TCL ring where all traffic classes compete for the same 512 descriptor slots.
Implement a custom wake_tx_queue operation that:
1. Checks hardware ring space before dequeuing packets from mac80211
2. Uses per-ring locking to serialize ring access and prevent races
3. Syncs with hardware state to get accurate free slot count
4. Uses ieee80211_tx_peek() to determine the exact target ring via get_ring_selector(), matching dp_tx on all platforms
5. Returns early during firmware crash in the same way as other tx paths
This approach follows the pattern used in the iwlwifi driver, adapted for ath11k's hardware ring architecture.
This eliminates -ENOMEM errors and improves throughput by optimizing resource usage and preventing unnecessary packet drops.
---
v2: Address feedback from Zhi-Jun You:
- Add ieee80211_tx_peek() API in mac80211 to peek at the next frame without dequeuing, enabling accurate ring selection before dequeue
- Use get_ring_selector() on the peeked skb to determine the exact target ring matching dp_tx, fixing ring mismatch on multi-ring platforms (ipq8074, wcn6750, qcn9274, wcn7850)
Move CRASH_FLUSH check inside the TX loop
v1: https://lore.kernel.org/all/[email protected]/
drivers/net/wireless/ath/ath11k/dp.c | 1 + drivers/net/wireless/ath/ath11k/dp.h | 2 + drivers/net/wireless/ath/ath11k/mac.c | 60 ++++++++++++++++++++++++++- 3 files changed, 62 insertions(+), 1 deletion(-)
drivers/net/wireless/ath/ath11k/dp.c +1 -0
@@ -1087,6 +1087,7 @@ int ath11k_dp_alloc(struct ath11k_base *ab)for (i =0 ; i < ab->hw_params.hal_params->num_tx_rings; i++) {idr_init (&dp->tx_ring[i].txbuf_idr);spin_lock_init (&dp->tx_ring[i].tx_idr_lock); +spin_lock_init (&dp->tx_ring[i].wake_tx_lock); dp->tx_ring[i].tcl_data_ring_id = i; dp->tx_ring[i].tx_status_head =0 ;
drivers/net/wireless/ath/ath11k/dp.h +2 -0
@@ -87,6 +87,8 @@ struct dp_tx_ring {struct idr txbuf_idr;/* Protects txbuf_idr and num_pending */ spinlock_t tx_idr_lock; +/* Serializes wake_tx_queue operations for this ring */ +spinlock_t wake_tx_lock;struct hal_wbm_release_ring *tx_status;int tx_status_head;int tx_status_tail;
drivers/net/wireless/ath/ath11k/mac.c +59 -1
68 lines not on this page · raw message
Signature · 1 line
2.54.0
Re: [PATCH v2 2/3] wifi: ath11k: implement custom wake_tx_queue with flow control
....
Quoted 53 lines
drivers/net/wireless/ath/ath11k/dp.c | 1 + drivers/net/wireless/ath/ath11k/dp.h | 2 + drivers/net/wireless/ath/ath11k/mac.c | 60 ++++++++++++++++++++++++++- 3 files changed, 62 insertions(+), 1 deletion(-)drivers/net/wireless/ath/ath11k/dp.c +1
-0@@ -1087,6 +1087,7 @@ int ath11k_dp_alloc(struct ath11k_base *ab)for (i =0 ; i < ab->hw_params.hal_params->num_tx_rings; i++) {idr_init (&dp->tx_ring[i].txbuf_idr);spin_lock_init (&dp->tx_ring[i].tx_idr_lock); +spin_lock_init (&dp->tx_ring[i].wake_tx_lock); dp->tx_ring[i].tcl_data_ring_id = i; dp->tx_ring[i].tx_status_head =0 ;drivers/net/wireless/ath/ath11k/dp.h +2
-0@@ -87,6 +87,8 @@ struct dp_tx_ring {struct idr txbuf_idr;/* Protects txbuf_idr and num_pending */ spinlock_t tx_idr_lock; +/* Serializes wake_tx_queue operations for this ring */ +spinlock_t wake_tx_lock;struct hal_wbm_release_ring *tx_status;int tx_status_head;int tx_status_tail;drivers/net/wireless/ath/ath11k/mac.c +14
-0@@ -10065,9 +10065,67 @@ static int ath11k_mac_op_sta_state(struct ieee80211_hw *hw,return ret; } +static void ath11k_mac_op_wake_tx_queue (struct ieee80211_hw *hw, +struct ieee80211_txq *txq) +{ +struct ieee80211_tx_control control = { + .sta = txq->sta, + }; +const struct sk_buff *peek_skb; +struct ath11k *ar = hw->priv; +struct dp_tx_ring *tx_ring; +struct hal_srng *tcl_ring; +struct sk_buff *skb; +u32 ring_selector; +int num_free; +u8 ring_id;
drivers/net/wireless/ath/ath11k/mac.c static int ath11k_mac_op_sta_state(struct ieee80211_hw *hw, · line 10091
Quoted 10 lines
drivers/net/wireless/ath/ath11k/mac.c +10
-0+ +if (!ar) +return ; + +while (1 ) { +if (unlikely (test_bit (ATH11K_FLAG_CRASH_FLUSH , + &ar->ab->dev_flags))) +break ; + + peek_skb =ieee80211_tx_peek (hw, txq);
in ieee80211_tx_peek() acquires fq->lock, peeks at the skb, releases the lock, then returns the pointer.
Between spin_unlock_bh(&fq->lock) and using the pointer for ring_selector. The same skb can also be returned for
the another CPU which calls this function. This may cause use after a free scenario.
Quoted 45 lines
drivers/net/wireless/ath/ath11k/mac.c +35
-2+if (!peek_skb) +break ; + + ring_selector = ar->ab->hw_params.hw_ops->get_ring_selector ( + (struct sk_buff *)peek_skb); + ring_id = ring_selector % + ar->ab->hw_params.hal_params->num_tx_rings; + + tx_ring = &ar->ab->dp.tx_ring[ring_id]; + tcl_ring = &ar->ab->hal.srng_list[tx_ring->tcl_data_ring.ring_id]; + +spin_lock_bh (&tx_ring->wake_tx_lock); + +spin_lock (&tcl_ring->lock); + num_free =ath11k_hal_srng_src_num_free (ar->ab, tcl_ring, true); +spin_unlock (&tcl_ring->lock); + +if (num_free ==0 ) { +spin_unlock_bh (&tx_ring->wake_tx_lock); +break ; + } + + skb =ieee80211_tx_dequeue (hw, txq); +if (!skb) { +spin_unlock_bh (&tx_ring->wake_tx_lock); +break ; + } + +ath11k_mac_op_tx (hw, &control, skb); + +spin_unlock_bh (&tx_ring->wake_tx_lock); + } +} +static const struct ieee80211_ops ath11k_ops = { .tx = ath11k_mac_op_tx, - .wake_tx_queue = ieee80211_handle_wake_tx_queue, + .wake_tx_queue = ath11k_mac_op_wake_tx_queue, .start = ath11k_mac_op_start, .stop = ath11k_mac_op_stop, .reconfig_complete = ath11k_mac_op_reconfig_complete, --2.54.0
Signature · 1 line
- Tamizh.
Re: [PATCH v2 2/3] wifi: ath11k: implement custom wake_tx_queue with flow control
Hi Tamizh,
Thank you for the review. You are right, I will fix this in v3.
Best regards,
Jose Ignacio
PATCH v2 3/3 wifi: ath12k: implement custom wake_tx_queue with flow control
In reply to Jose Ignacio Tornos MartinezUnder heavy traffic, ath12k can hang and experiences -ENOMEM errors ("failed to transmit frame -12") when the hardware TCL ring fills up. This issue is more commonly observed in VMs with PCIe passthrough but also occurs on bare metal systems.
Implement a custom wake_tx_queue operation that:
1. Checks hardware ring space before dequeuing packets from mac80211
2. Uses per-ring locking to serialize ring access and prevent races
3. Syncs with hardware state to get accurate free slot count
4. Uses ieee80211_tx_peek() to determine the exact target ring via get_ring_selector(), matching dp_tx on all platforms
5. Returns early during firmware crash in the same way as other tx paths
This approach follows the pattern used in the iwlwifi driver, adapted for ath12k's hardware ring architecture.
This prevents hangs, eliminates -ENOMEM errors, and improves throughput by optimizing resource usage and preventing unnecessary packet drops.
---
v2: Address feedback from Zhi-Jun You:
- Add ieee80211_tx_peek() API in mac80211 to peek at the next frame without dequeuing, enabling accurate ring selection before dequeue
- Use get_ring_selector() on the peeked skb to determine the exact target ring matching dp_tx, fixing ring mismatch on multi-ring platforms (ipq8074, wcn6750, qcn9274, wcn7850)
Address feedback from Tamizh Chelvam Raja:
- Fix ath12k radio assignment: obtain ar from txq->vif through the arvif chain instead of ah->radio flexible array
Move CRASH_FLUSH check inside the TX loop
v1: https://lore.kernel.org/all/[email protected]/
drivers/net/wireless/ath/ath12k/dp.c | 1 + drivers/net/wireless/ath/ath12k/dp.h | 2 + drivers/net/wireless/ath/ath12k/hal.c | 1 + drivers/net/wireless/ath/ath12k/wifi7/hw.c | 65 +++++++++++++++++++++- 4 files changed, 68 insertions(+), 1 deletion(-)
drivers/net/wireless/ath/ath12k/dp.c +1 -0
@@ -1539,6 +1539,7 @@ static int ath12k_dp_setup(struct ath12k_base *ab) }for (i =0 ; i < ab->hw_params->max_tx_ring; i++) { +spin_lock_init (&dp->tx_ring[i].wake_tx_lock); dp->tx_ring[i].tcl_data_ring_id = i; dp->tx_ring[i].tx_status_head =0 ;
drivers/net/wireless/ath/ath12k/dp.h +2 -0
@@ -58,6 +58,8 @@ struct dp_tx_ring {u8 tcl_data_ring_id;struct dp_srng tcl_data_ring;struct dp_srng tcl_comp_ring; +/* Serializes wake_tx_queue operations for this ring */ +spinlock_t wake_tx_lock;struct hal_wbm_completion_ring_tx *tx_status;int tx_status_head;int tx_status_tail;
drivers/net/wireless/ath/ath12k/hal.c +1 -0
@@ -390,6 +390,7 @@ int ath12k_hal_srng_src_num_free(struct ath12k_base *ab, struct hal_srng *srng,else return ((srng->ring_size - hp + tp) / srng->entry_size) -1 ; } +EXPORT_SYMBOL (ath12k_hal_srng_src_num_free);void *ath12k_hal_srng_src_next_peek (struct ath12k_base *ab,struct hal_srng *srng)
drivers/net/wireless/ath/ath12k/wifi7/hw.c +64 -1
73 lines not on this page · raw message
Signature · 1 line
2.54.0
Re: [PATCH v2 3/3] wifi: ath12k: implement custom wake_tx_queue with flow control
Quoted 23 lines · Jose Ignacio Tornos Martinez
Under heavy traffic, ath12k can hang and experiences
-ENOMEMerrors ("failed to transmit frame -12") when the hardware TCL ring fills up. This issue is more commonly observed in VMs with PCIe passthrough but also occurs on bare metal systems.Implement a custom
wake_tx_queueoperation that:1. Checks hardware ring space before dequeuing packets from mac80211
2. Uses per-ring locking to serialize ring access and prevent races
3. Syncs with hardware state to get accurate free slot count
4. Usesieee80211_tx_peek()to determine the exact target ring viaget_ring_selector(), matchingdp_txon all platforms
5. Returns early during firmware crash in the same way as other tx pathsThis approach follows the pattern used in the iwlwifi driver, adapted for ath12k's hardware ring architecture.
This prevents hangs, eliminates
-ENOMEMerrors, and improves throughput by optimizing resource usage and preventing unnecessary packet drops.Signed-off-by Jose Ignacio Tornos Martinez <[email protected]>
.....
Quoted 48 lines
drivers/net/wireless/ath/ath12k/dp.c | 1 + drivers/net/wireless/ath/ath12k/dp.h | 2 + drivers/net/wireless/ath/ath12k/hal.c | 1 + drivers/net/wireless/ath/ath12k/wifi7/hw.c | 65 +++++++++++++++++++++- 4 files changed, 68 insertions(+), 1 deletion(-)drivers/net/wireless/ath/ath12k/dp.c +1
-0@@ -1539,6 +1539,7 @@ static int ath12k_dp_setup(struct ath12k_base *ab) }for (i =0 ; i < ab->hw_params->max_tx_ring; i++) { +spin_lock_init (&dp->tx_ring[i].wake_tx_lock); dp->tx_ring[i].tcl_data_ring_id = i; dp->tx_ring[i].tx_status_head =0 ;drivers/net/wireless/ath/ath12k/dp.h +2
-0@@ -58,6 +58,8 @@ struct dp_tx_ring {u8 tcl_data_ring_id;struct dp_srng tcl_data_ring;struct dp_srng tcl_comp_ring; +/* Serializes wake_tx_queue operations for this ring */ +spinlock_t wake_tx_lock;struct hal_wbm_completion_ring_tx *tx_status;int tx_status_head;int tx_status_tail;drivers/net/wireless/ath/ath12k/hal.c +1
-0@@ -390,6 +390,7 @@ int ath12k_hal_srng_src_num_free(struct ath12k_base *ab, struct hal_srng *srng,else return ((srng->ring_size - hp + tp) / srng->entry_size) -1 ; } +EXPORT_SYMBOL (ath12k_hal_srng_src_num_free);void *ath12k_hal_srng_src_next_peek (struct ath12k_base *ab,struct hal_srng *srng)drivers/net/wireless/ath/ath12k/wifi7/hw.c +0
-0
drivers/net/wireless/ath/ath12k/wifi7/hw.c static void ath12k_wifi7_mac_op_tx(struct ieee80211_hw *hw, · line 1106
Quoted 7 lines
drivers/net/wireless/ath/ath12k/wifi7/hw.c +4
-0@@ -1100,9 +1100,72 @@ static void ath12k_wifi7_mac_op_tx(struct ieee80211_hw *hw, } } +static void ath12k_wifi7_mac_op_wake_tx_queue (struct ieee80211_hw *hw, +struct ieee80211_txq *txq) +{ +struct ath12k_vif *ahvif =ath12k_vif_to_ahvif (txq->vif);
Is txq->vif always valid? Do we need a NULL check here?
Quoted 5 lines
drivers/net/wireless/ath/ath12k/wifi7/hw.c +5
-0+struct ath12k_link_vif *arvif = &ahvif->deflink; +struct ieee80211_tx_control control = { + .sta = txq->sta, + }; +const struct sk_buff *peek_skb;
drivers/net/wireless/ath/ath12k/wifi7/hw.c static void ath12k_wifi7_mac_op_tx(struct ieee80211_hw *hw, · line 1121
Quoted 10 lines
drivers/net/wireless/ath/ath12k/wifi7/hw.c +10
-0+struct dp_tx_ring *tx_ring; +struct hal_srng *tcl_ring; +struct ath12k_dp *dp; +struct sk_buff *skb; +struct ath12k *ar; +u32 ring_selector; +int num_free; +u8 ring_id; + + ar = arvif->ar;
Shouldn't we check the link_id for fetching proper arvif? this code always chooses deflink ar and dp for packet transmission. This needs to use the same link selection logic as op_tx.
Quoted 55 lines
drivers/net/wireless/ath/ath12k/wifi7/hw.c +45
-2+if (!ar) +return ; + + dp = ar->ab->dp; + +while (1 ) { +if (unlikely (test_bit (ATH12K_FLAG_CRASH_FLUSH , + &ar->ab->dev_flags))) +break ; + + peek_skb =ieee80211_tx_peek (hw, txq); +if (!peek_skb) +break ; + + ring_selector = dp->hw_params->hw_ops->get_ring_selector ( + (struct sk_buff *)peek_skb); + ring_id = ring_selector % dp->hw_params->max_tx_ring; + + tx_ring = &dp->tx_ring[ring_id]; + tcl_ring = &dp->hal->srng_list[tx_ring->tcl_data_ring.ring_id]; + +spin_lock_bh (&tx_ring->wake_tx_lock); + +spin_lock (&tcl_ring->lock); + num_free =ath12k_hal_srng_src_num_free (ar->ab, tcl_ring, true); +spin_unlock (&tcl_ring->lock); + +if (num_free ==0 ) { +spin_unlock_bh (&tx_ring->wake_tx_lock); +break ; + } + + skb =ieee80211_tx_dequeue (hw, txq); +if (!skb) { +spin_unlock_bh (&tx_ring->wake_tx_lock); +break ; + } + +ath12k_wifi7_mac_op_tx (hw, &control, skb); + +spin_unlock_bh (&tx_ring->wake_tx_lock); + } +} +static const struct ieee80211_ops ath12k_ops_wifi7 = { .tx = ath12k_wifi7_mac_op_tx, - .wake_tx_queue = ieee80211_handle_wake_tx_queue, + .wake_tx_queue = ath12k_wifi7_mac_op_wake_tx_queue, .start = ath12k_mac_op_start, .stop = ath12k_mac_op_stop, .reconfig_complete = ath12k_mac_op_reconfig_complete, --2.54.0
Signature · 1 line
- Tamizh.
Re: [PATCH v2 3/3] wifi: ath12k: implement custom wake_tx_queue with flow control
Hi Tamizh,
Thank you for the review. You are right, I will fix this in v3.
Best regards,
Jose Ignacio