Skip to contentSkip to conversation

Add Airtime Queue Limits (AQL) to mac80211

PATCH In review Kan Yan 5 participants 25 messages Nov 19, 2019 – Feb 27, 2020

Subsystems (2)
lore
Share & tools
lore

Kan Yan

author
#0 lore raw

v11 cover Add Airtime Queue Limits (AQL) to mac80211

This series is a first attempt at porting the Airtime Queue Limits concept from the out-of-tree ath10k implementation[0] to mac80211. This version takes Kan's patch to do the throttling in mac80211, and replaces the driver API with the mechanism from the previous version of Toke's series, which instead calculated the expected airtime at dequeue time inside mac80211, storing it in the SKB cb field.

This series also imports Felix' airtime calculation code from mt76 into mac80211, adjusting the API so it can be used from TX dequeue, by extracting the latest TX rate from the tx_stats structure kept for each station.

This series has been tested on the QCA9984 platform. The test data can be found in:
https://drive.google.com/drive/folders/16Th2PljrKXVWDgM1pqDredMdSNcqJGOB

[0] https://chromium-review.googlesource.com/c/chromiumos/third_party/kernel/+/1703105/7

Changelog:

v11:
- Fix softlockup in ieee80211_next_txq()
- Increase the default queue limit

v10:
- Fix return value from ieee80211_info_set_tx_time_est()

v9:
- Use atomic_sub_return() instead of separate atomic_sub() and atomic_read()
- Add getter/setter for tx_time_est
- Use get_sta_info_by_addrs() to find the station in

    ieee80211_report_used_skb()

- Integrate everything back into one series

v8:
- Includes Toke's v7 version of "mac80211: Import airtime calculation code from mt76"
- Don't clobber sta's customized queue limit when configuring the default via debugfs
- Fix a racing condition when reset aql_tx_pending.

v7:
- Fix aql_total_pending_airtime underflow due to insufficient locking.

v6:
- Fix sta lookup in ieee80211_report_used_skb().
- Move call to ieee80211_sta_update_pending_airtime() to a bit later in

    __ieee80211_tx_status()
v5:
  - Add missing export of ieee80211_calc_rx_airtime() and make
    ieee80211_calc_tx_airtime_rate() static (kbuildbot).

- Use skb_get_queue_mapping() to get the AC from the skb.
- Take basic rate configuration for the BSS into account when calculating multicast rate.
v4:
- Fix calculation that clamps the maximum airtime to fit into 10 bits
- Incorporate Rich Brown's nits for the commit message in Kan's patch
- Add fewer local variables to ieee80211_tx_dequeue()
v3:
- Move the tx_time_est field so it's shared with ack_frame_id, and use units of 4us for the value stored in it.
- Move the addition of the Ethernet header size into ieee80211_calc_expected_tx_airtime()
v2:
- Integrate Kan's approach to airtime throttling.
- Hopefully fix the cb struct alignment on big-endian architectures.

Kan Yan (1):
mac80211: Implement Airtime-based Queue Limit (AQL)

Toke Høiland-Jørgensen (3):
mac80211: Add new sta_info getter by sta/vif addrs
mac80211: Import airtime calculation code from mt76
mac80211: Use Airtime-based Queue Limits (AQL) on packet dequeue

 include/net/cfg80211.h     |   7 +
 include/net/mac80211.h     |  57 ++++
 net/mac80211/Makefile      |   3 +-
 net/mac80211/airtime.c     | 597 +++++++++++++++++++++++++++++++++++++
 net/mac80211/debugfs.c     |  85 ++++++
 net/mac80211/debugfs_sta.c |  43 ++-
 net/mac80211/ieee80211_i.h |   8 +
 net/mac80211/main.c        |  10 +-
 net/mac80211/sta_info.c    |  58 ++++
 net/mac80211/sta_info.h    |  11 +
 net/mac80211/status.c      |  36 ++-
 net/mac80211/tx.c          |  69 ++++-
 12 files changed, 961 insertions(+), 23 deletions(-)
 create mode 100644 net/mac80211/airtime.c
Signature · 1 line
2.24.0.432.g9d3f5f5b63-goog
author
#1 lore raw

PATCH v11 1/4 mac80211: Add new sta_info getter by sta/vif addrs

From: Toke Høiland-Jørgensen <[email protected]>

In ieee80211_tx_status() we don't have an sdata struct when looking up the destination sta. Instead, we just do a lookup by the vif addr that is the source of the packet being completed. Factor this out into a new sta_info getter helper, since we need to use it for accounting AQL as well.

---
 net/mac80211/sta_info.c | 20 ++++++++++++++++++++
 net/mac80211/sta_info.h |  3 +++
 net/mac80211/status.c   | 10 ++--------
 3 files changed, 25 insertions(+), 8 deletions(-)
net/mac80211/sta_info.c +20 -0
index bd11fef2139f..465d83b13582 100644
@@ -210,6 +210,26 @@ struct sta_info *sta_info_get_bss(struct ieee80211_sub_if_data *sdata,
 	return NULL;
 }
 
+struct sta_info *sta_info_get_by_addrs(struct ieee80211_local *local,
+				       const u8 *sta_addr, const u8 *vif_addr)
+{
+	struct rhlist_head *tmp;
+	struct sta_info *sta;
+
+	rcu_read_lock();
+	for_each_sta_info(local, sta_addr, sta, tmp) {
+		if (ether_addr_equal(vif_addr, sta->sdata->vif.addr)) {
+			rcu_read_unlock();
+			/* this is safe as the caller must already hold
+			 * another rcu read section or the mutex
+			 */
+			return sta;
+		}
+	}
+	rcu_read_unlock();
+	return NULL;
+}
+
 struct sta_info *sta_info_get_by_idx(struct ieee80211_sub_if_data *sdata,
 				     int idx)
 {
net/mac80211/sta_info.h +3 -0
index 369c2dddce52..80e76569144e 100644
@@ -725,6 +725,9 @@ struct sta_info *sta_info_get(struct ieee80211_sub_if_data *sdata,
 struct sta_info *sta_info_get_bss(struct ieee80211_sub_if_data *sdata,
 				  const u8 *addr);
 
+struct sta_info *sta_info_get_by_addrs(struct ieee80211_local *local,
+				       const u8 *sta_addr, const u8 *vif_addr);
+
 #define for_each_sta_info(local, _addr, _sta, _tmp)			\
 	rhl_for_each_entry_rcu(_sta, _tmp,				\
 			       sta_info_hash_lookup(local, _addr), hash_node)
net/mac80211/status.c +2 -8
index ab8ba5835ca0..0e51def35b8a 100644
@@ -1073,19 +1073,13 @@ void ieee80211_tx_status(struct ieee80211_hw *hw, struct sk_buff *skb)
 		.skb = skb,
 		.info = IEEE80211_SKB_CB(skb),
 	};
-	struct rhlist_head *tmp;
 	struct sta_info *sta;
 
 	rcu_read_lock();
 
-	for_each_sta_info(local, hdr->addr1, sta, tmp) {
-		/* skip wrong virtual interface */
-		if (!ether_addr_equal(hdr->addr2, sta->sdata->vif.addr))
-			continue;
-
+	sta = sta_info_get_by_addrs(local, hdr->addr1, hdr->addr2);
+	if (sta)
 		status.sta = &sta->sta;
-		break;
-	}
 
 	__ieee80211_tx_status(hw, &status);
 	rcu_read_unlock();
Signature · 1 line
2.24.0.432.g9d3f5f5b63-goog
author
#2 lore raw

PATCH v11 2/4 mac80211: Import airtime calculation code from mt76

In reply to Kan Yan

From: Toke Høiland-Jørgensen <[email protected]>

Felix recently added code to calculate airtime of packets to the mt76 driver. Import this into mac80211 so we can use it for airtime queue limit calculations.

The airtime.c file is copied verbatim from the mt76 driver, and adjusted to be usable in mac80211. This involves:

- Switching to mac80211 data structures.
- Adding support for 160 MHz channels and HE mode.
- Moving the symbol and duration calculations around a bit to avoid rounding with the higher rates and longer symbol times used for HE rates.

The per-rate TX rate calculation is also split out to its own function so it can be used directly for the AQL calculations later.

---
 include/net/mac80211.h     |  29 ++
 net/mac80211/Makefile      |   3 +-
 net/mac80211/airtime.c     | 597 +++++++++++++++++++++++++++++++++++++
 net/mac80211/ieee80211_i.h |   4 +
 4 files changed, 632 insertions(+), 1 deletion(-)
 create mode 100644 net/mac80211/airtime.c
include/net/mac80211.h +29 -0
index c643a19dce96..6fc26a051ba0 100644
@@ -6424,4 +6424,33 @@ void ieee80211_nan_func_match(struct ieee80211_vif *vif,
 			      struct cfg80211_nan_match_params *match,
 			      gfp_t gfp);
 
+/**
+ * ieee80211_calc_rx_airtime - calculate estimated transmission airtime for RX.
+ *
+ * This function calculates the estimated airtime usage of a frame based on the
+ * rate information in the RX status struct and the frame length.
+ *
+ * @hw: pointer as obtained from ieee80211_alloc_hw()
+ * @status: &struct ieee80211_rx_status containing the transmission rate
+ *          information.
+ * @len: frame length in bytes
+ */
+u32 ieee80211_calc_rx_airtime(struct ieee80211_hw *hw,
+			      struct ieee80211_rx_status *status,
+			      int len);
+
+/**
+ * ieee80211_calc_tx_airtime - calculate estimated transmission airtime for TX.
+ *
+ * This function calculates the estimated airtime usage of a frame based on the
+ * rate information in the TX info struct and the frame length.
+ *
+ * @hw: pointer as obtained from ieee80211_alloc_hw()
+ * @info: &struct ieee80211_tx_info of the frame.
+ * @len: frame length in bytes
+ */
+u32 ieee80211_calc_tx_airtime(struct ieee80211_hw *hw,
+			      struct ieee80211_tx_info *info,
+			      int len);
+
 #endif /* MAC80211_H */
net/mac80211/Makefile +2 -1
index 4f03ebe732fa..6cbb1286d6c0 100644
@@ -32,7 +32,8 @@ mac80211-y := \
 	chan.o \
 	trace.o mlme.o \
 	tdls.o \
-	ocb.o
+	ocb.o \
+	airtime.o
 
 mac80211-$(CONFIG_MAC80211_LEDS) += led.o
 mac80211-$(CONFIG_MAC80211_DEBUGFS) += \
net/mac80211/airtime.c new file +597 -0

597 lines not on this page · raw message

net/mac80211/ieee80211_i.h +4 -0
index 05406e9c05b3..225ea4e3cd76 100644
@@ -2249,6 +2249,10 @@ const char *ieee80211_get_reason_code_string(u16 reason_code);
 
 extern const struct ethtool_ops ieee80211_ethtool_ops;
 
+u32 ieee80211_calc_expected_tx_airtime(struct ieee80211_hw *hw,
+				       struct ieee80211_vif *vif,
+				       struct ieee80211_sta *pubsta,
+				       int len);
 #ifdef CONFIG_MAC80211_NOINLINE
 #define debug_noinline noinline
 #else
Signature · 1 line
2.24.0.432.g9d3f5f5b63-goog
maintainer
#3 lore raw

Re: [PATCH v11 2/4] mac80211: Import airtime calculation code from mt76

Quoted 6 lines · Kan Yan

From: Toke Høiland-Jørgensen <[email protected]>

Felix recently added code to calculate airtime of packets to the mt76 driver. Import this into mac80211 so we can use it for airtime queue limit calculations.

Quoted 7 lines

The airtime.c file is copied verbatim from the mt76 driver, and adjusted to be usable in mac80211. This involves:

- Switching to mac80211 data structures.
- Adding support for 160 MHz channels and HE mode.
- Moving the symbol and duration calculations around a bit to avoid rounding with the higher rates and longer symbol times used for HE rates.

:)

I'll apply this, I guess, but I do wonder to what extent it overlaps with cfg80211_calculate_bitrate()?

johannes

#4 lore raw

Re: [PATCH v11 2/4] mac80211: Import airtime calculation code from mt76

Quoted 10 lines · Johannes Berg
Quoted 9 lines · Kan Yan

From: Toke Høiland-Jørgensen <[email protected]>

Felix recently added code to calculate airtime of packets to the mt76 driver. Import this into mac80211 so we can use it for airtime queue limit calculations.

The airtime.c file is copied verbatim from the mt76 driver, and adjusted to be usable in mac80211. This involves:

Quoted 8 lines
Quoted 4 lines

- Switching to mac80211 data structures.
- Adding support for 160 MHz channels and HE mode.
- Moving the symbol and duration calculations around a bit to avoid rounding with the higher rates and longer symbol times used for HE rates.

:)

I'll apply this, I guess,

Great!

Quoted 2 lines

but I do wonder to what extent it overlaps with
cfg80211_calculate_bitrate()?

Well, one calculates bitrates while the other calculates airtime? ;)

But yeah, I get what you mean. I think Felix went through quite some pains to structure this code to avoid divisions in the fast path. I guess that is the main blocker for cfg80211_calculate_bitrate() to be used instead (assuming we do want to consolidate them eventually). Not sure if that can be fixed easily though?

-Toke

maintainer
#5 lore raw

Re: [PATCH v11 2/4] mac80211: Import airtime calculation code from mt76

Quoted 2 lines

Well, one calculates bitrates while the other calculates airtime? ;)

:)

Quoted 5 lines

But yeah, I get what you mean. I think Felix went through quite some pains to structure this code to avoid divisions in the fast path. I guess that is the main blocker for cfg80211_calculate_bitrate() to be used instead (assuming we do want to consolidate them eventually). Not sure if that can be fixed easily though?

We could also do it the other way around?

Or maybe we should keep both and use them as a sanity check for each other :P

johannes

#6 lore raw

Re: [PATCH v11 2/4] mac80211: Import airtime calculation code from mt76

Quoted 11 lines · Johannes Berg
Quoted 1 line

Well, one calculates bitrates while the other calculates airtime? ;)

:)

Quoted 5 lines

But yeah, I get what you mean. I think Felix went through quite some pains to structure this code to avoid divisions in the fast path. I guess that is the main blocker for cfg80211_calculate_bitrate() to be used instead (assuming we do want to consolidate them eventually). Not sure if that can be fixed easily though?

We could also do it the other way around?

as in bitrate = 1/airtime? Dunno, maybe? How important is precision to the bitrate calculations?

Quoted 2 lines

Or maybe we should keep both and use them as a sanity check for each other :P

Hmm, yeah, it should be possible to write a selftest to iterate through all rates and compare the output of each calculation, shouldn't it? Where would be a good place to put that?

-Toke

maintainer
#7 lore raw

Re: [PATCH v11 2/4] mac80211: Import airtime calculation code from mt76

Quoted 3 lines · Toke Høiland-Jørgensen

as in bitrate = 1/airtime? Dunno, maybe? How important is precision to the bitrate calculations?

Not *that* important, it's just sort of advisory for userspace.

Though I think there's at least one place in mac80211 related to radiotap that actually does use it for airtime calculation, which should probably be changed to the new code now ...

Quoted 6 lines
Quoted 2 lines

Or maybe we should keep both and use them as a sanity check for each other :P

Hmm, yeah, it should be possible to write a selftest to iterate through all rates and compare the output of each calculation, shouldn't it? Where would be a good place to put that?

No idea ... make a new kselftest thing for wireless?

johannes

#8 lore raw

Re: [PATCH v11 2/4] mac80211: Import airtime calculation code from mt76

Quoted 10 lines · Johannes Berg
Quoted 3 lines · Toke Høiland-Jørgensen

as in bitrate = 1/airtime? Dunno, maybe? How important is precision to the bitrate calculations?

Not *that* important, it's just sort of advisory for userspace.

Though I think there's at least one place in mac80211 related to radiotap that actually does use it for airtime calculation, which should probably be changed to the new code now ...

Yeah, I guess so. The ath9k driver is also doing software calculations for RX, so that could also be converted; and mt76 should be, of course...

Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

Or maybe we should keep both and use them as a sanity check for each other :P

Hmm, yeah, it should be possible to write a selftest to iterate through all rates and compare the output of each calculation, shouldn't it? Where would be a good place to put that?

No idea ... make a new kselftest thing for wireless?

Right; was just wondering if there was something like that already which I didn't know about :)

-Toke

maintainer
#9 lore raw

Re: [PATCH v11 2/4] mac80211: Import airtime calculation code from mt76

In reply to Kan Yan

net/mac80211/airtime.c line 171

Quoted 6 lines · Kan Yan
net/mac80211/airtime.c new file +5 -0
+#define HE_GROUP_IDX(_streams, _gi, _bw)				\
+	(IEEE80211_HE_GROUP_0 +					\
+	 IEEE80211_HE_MAX_STREAMS * 2 * (_bw) +			\
+	 IEEE80211_HE_MAX_STREAMS * (_gi) +				\
+	 (_streams) - 1)

I'll also fix that to be " * 3" instead of 2, since there are 3 possible _gi values.

johannes

#10 lore raw

Re: [PATCH v11 2/4] mac80211: Import airtime calculation code from mt76

Quoted 10 lines · Johannes Berg

net/mac80211/airtime.c line 171

Quoted 6 lines · Kan Yan
net/mac80211/airtime.c new file +5 -0
+#define HE_GROUP_IDX(_streams, _gi, _bw)				\
+	(IEEE80211_HE_GROUP_0 +					\
+	 IEEE80211_HE_MAX_STREAMS * 2 * (_bw) +			\
+	 IEEE80211_HE_MAX_STREAMS * (_gi) +				\
+	 (_streams) - 1)

I'll also fix that to be " * 3" instead of 2, since there are 3 possible _gi values.

Ah yes, good catch, thanks!

-Toke

author
#11 lore raw

PATCH v11 3/4 mac80211: Implement Airtime-based Queue Limit (AQL)

In reply to Kan Yan

In order for the Fq_CoDel algorithm integrated in mac80211 layer to operate effectively to control excessive queueing latency, the CoDel algorithm requires an accurate measure of how long packets stays in the queue, AKA sojourn time. The sojourn time measured at the mac80211 layer doesn't include queueing latency in the lower layer (firmware/hardware) and CoDel expects lower layer to have a short queue. However, most 802.11ac chipsets offload tasks such TX aggregation to firmware or hardware, thus have a deep lower layer queue.

Without a mechanism to control the lower layer queue size, packets only stay in mac80211 layer transiently before being sent to firmware queue. As a result, the sojourn time measured by CoDel in the mac80211 layer is almost always lower than the CoDel latency target, hence CoDel does little to control the latency, even when the lower layer queue causes excessive latency.

The Byte Queue Limits (BQL) mechanism is commonly used to address the similar issue with wired network interface. However, this method cannot be applied directly to the wireless network interface. "Bytes" is not a suitable measure of queue depth in the wireless network, as the data rate can vary dramatically from station to station in the same network, from a few Mbps to over Gbps.

This patch implements an Airtime-based Queue Limit (AQL) to make CoDel work effectively with wireless drivers that utilized firmware/hardware offloading. AQL allows each txq to release just enough packets to the lower layer to form 1-2 large aggregations to keep hardware fully utilized and retains the rest of the frames in mac80211 layer to be controlled by the CoDel algorithm.

[ Toke: Keep API to set pending airtime internal, fix nits in commit msg ]
---
 include/net/cfg80211.h     |  7 ++++
 include/net/mac80211.h     | 12 ++++++
 net/mac80211/debugfs.c     | 85 ++++++++++++++++++++++++++++++++++++++
 net/mac80211/debugfs_sta.c | 43 ++++++++++++++-----
 net/mac80211/ieee80211_i.h |  4 ++
 net/mac80211/main.c        | 10 ++++-
 net/mac80211/sta_info.c    | 38 +++++++++++++++++
 net/mac80211/sta_info.h    |  8 ++++
 net/mac80211/tx.c          | 51 +++++++++++++++++++++--
 9 files changed, 244 insertions(+), 14 deletions(-)
include/net/cfg80211.h +7 -0
index e309cc826b40..b0f440857aeb 100644
@@ -2606,6 +2606,13 @@ enum wiphy_params_flags {
 
 #define IEEE80211_DEFAULT_AIRTIME_WEIGHT	256
 
+/* The per TXQ device queue limit in airtime */
+#define IEEE80211_DEFAULT_AQL_TXQ_LIMIT_L	5000
+#define IEEE80211_DEFAULT_AQL_TXQ_LIMIT_H	12000
+
+/* The per interface airtime threshold to switch to lower queue limit */
+#define IEEE80211_AQL_THRESHOLD			24000
+
 /**
  * struct cfg80211_pmksa - PMK Security Association
  *
include/net/mac80211.h +12 -0
index 6fc26a051ba0..ba3f33cc41ea 100644
@@ -5565,6 +5565,18 @@ void ieee80211_send_eosp_nullfunc(struct ieee80211_sta *pubsta, int tid);
 void ieee80211_sta_register_airtime(struct ieee80211_sta *pubsta, u8 tid,
 				    u32 tx_airtime, u32 rx_airtime);
 
+/**
+ * ieee80211_txq_airtime_check - check if a txq can send frame to device
+ *
+ * @hw: pointer obtained from ieee80211_alloc_hw()
+ * @txq: pointer obtained from station or virtual interface
+ *
+ * Return true if the AQL's airtime limit has not been reached and the txq can
+ * continue to send more packets to the device. Otherwise return false.
+ */
+bool
+ieee80211_txq_airtime_check(struct ieee80211_hw *hw, struct ieee80211_txq *txq);
+
 /**
  * ieee80211_iter_keys - iterate keys programmed into the device
  * @hw: pointer obtained from ieee80211_alloc_hw()
net/mac80211/debugfs.c +85 -0

97 lines not on this page · raw message

net/mac80211/debugfs_sta.c +33 -10

69 lines not on this page · raw message

net/mac80211/ieee80211_i.h +4 -0
index 225ea4e3cd76..ad15b3be8bb3 100644
@@ -1142,6 +1142,10 @@ struct ieee80211_local {
 	u16 schedule_round[IEEE80211_NUM_ACS];
 
 	u16 airtime_flags;
+	u32 aql_txq_limit_low[IEEE80211_NUM_ACS];
+	u32 aql_txq_limit_high[IEEE80211_NUM_ACS];
+	u32 aql_threshold;
+	atomic_t aql_total_pending_airtime;
 
 	const struct ieee80211_ops *ops;
 
net/mac80211/main.c +9 -1
index aba094b4ccfc..071ea92a3748 100644
@@ -667,8 +667,16 @@ struct ieee80211_hw *ieee80211_alloc_hw_nm(size_t priv_data_len,
 	for (i = 0; i < IEEE80211_NUM_ACS; i++) {
 		INIT_LIST_HEAD(&local->active_txqs[i]);
 		spin_lock_init(&local->active_txq_lock[i]);
+		local->aql_txq_limit_low[i] = IEEE80211_DEFAULT_AQL_TXQ_LIMIT_L;
+		local->aql_txq_limit_high[i] =
+			IEEE80211_DEFAULT_AQL_TXQ_LIMIT_H;
 	}
-	local->airtime_flags = AIRTIME_USE_TX | AIRTIME_USE_RX;
+
+	local->airtime_flags = AIRTIME_USE_TX |
+			       AIRTIME_USE_RX |
+			       AIRTIME_USE_AQL;
+	local->aql_threshold = IEEE80211_AQL_THRESHOLD;
+	atomic_set(&local->aql_total_pending_airtime, 0);
 
 	INIT_LIST_HEAD(&local->chanctx_list);
 	mutex_init(&local->chanctx_mtx);
net/mac80211/sta_info.c +38 -0
index 465d83b13582..1fe02022a31a 100644
@@ -416,6 +416,9 @@ struct sta_info *sta_info_alloc(struct ieee80211_sub_if_data *sdata,
 		skb_queue_head_init(&sta->ps_tx_buf[i]);
 		skb_queue_head_init(&sta->tx_filtered[i]);
 		sta->airtime[i].deficit = sta->airtime_weight;
+		atomic_set(&sta->airtime[i].aql_tx_pending, 0);
+		sta->airtime[i].aql_limit_low = local->aql_txq_limit_low[i];
+		sta->airtime[i].aql_limit_high = local->aql_txq_limit_high[i];
 	}
 
 	for (i = 0; i < IEEE80211_NUM_TIDS; i++)
@@ -1913,6 +1916,41 @@ void ieee80211_sta_register_airtime(struct ieee80211_sta *pubsta, u8 tid,
 }
 EXPORT_SYMBOL(ieee80211_sta_register_airtime);
 
+void ieee80211_sta_update_pending_airtime(struct ieee80211_local *local,
+					  struct sta_info *sta, u8 ac,
+					  u16 tx_airtime, bool tx_completed)
+{
+	int tx_pending;
+
+	if (!tx_completed) {
+		if (sta)
+			atomic_add(tx_airtime,
+				   &sta->airtime[ac].aql_tx_pending);
+
+		atomic_add(tx_airtime, &local->aql_total_pending_airtime);
+		return;
+	}
+
+	if (sta) {
+		tx_pending = atomic_sub_return(tx_airtime,
+					       &sta->airtime[ac].aql_tx_pending);
+		if (WARN_ONCE(tx_pending < 0,
+			      "STA %pM AC %d txq pending airtime underflow: %u, %u",
+			      sta->addr, ac, tx_pending, tx_airtime))
+			atomic_cmpxchg(&sta->airtime[ac].aql_tx_pending,
+				       tx_pending, 0);
+	}
+
+	tx_pending = atomic_sub_return(tx_airtime,
+				       &local->aql_total_pending_airtime);
+	if (WARN_ONCE(tx_pending < 0,
+		      "Device %s AC %d pending airtime underflow: %u, %u",
+		      wiphy_name(local->hw.wiphy), ac, tx_pending,
+		      tx_airtime))
+		atomic_cmpxchg(&local->aql_total_pending_airtime,
+			       tx_pending, 0);
+}
+
 int sta_info_move_state(struct sta_info *sta,
 			enum ieee80211_sta_state new_state)
 {
net/mac80211/sta_info.h +8 -0
index 80e76569144e..0608d49b3826 100644
@@ -127,13 +127,21 @@ enum ieee80211_agg_stop_reason {
 /* Debugfs flags to enable/disable use of RX/TX airtime in scheduler */
 #define AIRTIME_USE_TX		BIT(0)
 #define AIRTIME_USE_RX		BIT(1)
+#define AIRTIME_USE_AQL		BIT(2)
 
 struct airtime_info {
 	u64 rx_airtime;
 	u64 tx_airtime;
 	s64 deficit;
+	atomic_t aql_tx_pending; /* Estimated airtime for frames pending */
+	u32 aql_limit_low;
+	u32 aql_limit_high;
 };
 
+void ieee80211_sta_update_pending_airtime(struct ieee80211_local *local,
+					  struct sta_info *sta, u8 ac,
+					  u16 tx_airtime, bool tx_completed);
+
 struct sta_info;
 
 /**
net/mac80211/tx.c +48 -3

74 lines not on this page · raw message

Signature · 1 line
2.24.0.432.g9d3f5f5b63-goog
author
#12 lore raw

PATCH v11 4/4 mac80211: Use Airtime-based Queue Limits (AQL) on packet dequeue

In reply to Kan Yan

From: Toke Høiland-Jørgensen <[email protected]>

The previous commit added the ability to throttle stations when they queue too much airtime in the hardware. This commit enables the functionality by calculating the expected airtime usage of each packet that is dequeued from the TXQs in mac80211, and accounting that as pending airtime.

The estimated airtime for each skb is stored in the tx_info, so we can subtract the same amount from the running total when the skb is freed or recycled. The throttling mechanism relies on this accounting to be accurate (i.e., that we are not freeing skbs without subtracting any airtime they were accounted for), so we put the subtraction into ieee80211_report_used_skb(). As an optimisation, we also subtract the airtime on regular TX completion, zeroing out the value stored in the packet afterwards, to avoid having to do an expensive lookup of the station from the packet data on every packet.

This patch does *not* include any mechanism to wake a throttled TXQ again, on the assumption that this will happen anyway as a side effect of whatever freed the skb (most commonly a TX completion).

---
 include/net/mac80211.h | 16 ++++++++++++++++
 net/mac80211/status.c  | 26 ++++++++++++++++++++++++++
 net/mac80211/tx.c      | 18 ++++++++++++++++++
 3 files changed, 60 insertions(+)
include/net/mac80211.h +16 -0
index ba3f33cc41ea..aa145808e57a 100644
@@ -1060,6 +1060,22 @@ struct ieee80211_tx_info {
 	};
 };
 
+static inline u16
+ieee80211_info_set_tx_time_est(struct ieee80211_tx_info *info, u16 tx_time_est)
+{
+	/* We only have 10 bits in tx_time_est, so store airtime
+	 * in increments of 4us and clamp the maximum to 2**12-1
+	 */
+	info->tx_time_est = min_t(u16, tx_time_est, 4095) >> 2;
+	return info->tx_time_est << 2;
+}
+
+static inline u16
+ieee80211_info_get_tx_time_est(struct ieee80211_tx_info *info)
+{
+	return info->tx_time_est << 2;
+}
+
 /**
  * struct ieee80211_tx_status - extended tx status info for rate control
  *
net/mac80211/status.c +26 -0
index 0e51def35b8a..39da82b35be9 100644
@@ -670,12 +670,26 @@ static void ieee80211_report_used_skb(struct ieee80211_local *local,
 				      struct sk_buff *skb, bool dropped)
 {
 	struct ieee80211_tx_info *info = IEEE80211_SKB_CB(skb);
+	u16 tx_time_est = ieee80211_info_get_tx_time_est(info);
 	struct ieee80211_hdr *hdr = (void *)skb->data;
 	bool acked = info->flags & IEEE80211_TX_STAT_ACK;
 
 	if (dropped)
 		acked = false;
 
+	if (tx_time_est) {
+		struct sta_info *sta;
+
+		rcu_read_lock();
+
+		sta = sta_info_get_by_addrs(local, hdr->addr1, hdr->addr2);
+		ieee80211_sta_update_pending_airtime(local, sta,
+						     skb_get_queue_mapping(skb),
+						     tx_time_est,
+						     true);
+		rcu_read_unlock();
+	}
+
 	if (info->flags & IEEE80211_TX_INTFL_MLME_CONN_TX) {
 		struct ieee80211_sub_if_data *sdata;
 
@@ -877,6 +891,7 @@ static void __ieee80211_tx_status(struct ieee80211_hw *hw,
 	struct ieee80211_bar *bar;
 	int shift = 0;
 	int tid = IEEE80211_NUM_TIDS;
+	u16 tx_time_est;
 
 	rates_idx = ieee80211_tx_get_rates(hw, info, &retry_count);
 
@@ -986,6 +1001,17 @@ static void __ieee80211_tx_status(struct ieee80211_hw *hw,
 			ieee80211_sta_register_airtime(&sta->sta, tid,
 						       info->status.tx_time, 0);
 
+		if ((tx_time_est = ieee80211_info_get_tx_time_est(info)) > 0) {
+			/* Do this here to avoid the expensive lookup of the sta
+			 * in ieee80211_report_used_skb().
+			 */
+			ieee80211_sta_update_pending_airtime(local, sta,
+							     skb_get_queue_mapping(skb),
+							     tx_time_est,
+							     true);
+			ieee80211_info_set_tx_time_est(info, 0);
+		}
+
 		if (ieee80211_hw_check(&local->hw, REPORTS_TX_ACK_STATUS)) {
 			if (info->flags & IEEE80211_TX_STAT_ACK) {
 				if (sta->status_stats.lost_packets)
net/mac80211/tx.c +18 -0
index 59db921d1086..dfbaab92dbf2 100644
@@ -3551,6 +3551,9 @@ struct sk_buff *ieee80211_tx_dequeue(struct ieee80211_hw *hw,
 
 	WARN_ON_ONCE(softirq_count() == 0);
 
+	if (!ieee80211_txq_airtime_check(hw, txq))
+		return NULL;
+
 begin:
 	spin_lock_bh(&fq->lock);
 
@@ -3661,6 +3664,21 @@ struct sk_buff *ieee80211_tx_dequeue(struct ieee80211_hw *hw,
 	}
 
 	IEEE80211_SKB_CB(skb)->control.vif = vif;
+
+	if (local->airtime_flags & AIRTIME_USE_AQL) {
+		u32 airtime;
+
+		airtime = ieee80211_calc_expected_tx_airtime(hw, vif, txq->sta,
+							     skb->len);
+		if (airtime) {
+			airtime = ieee80211_info_set_tx_time_est(info, airtime);
+			ieee80211_sta_update_pending_airtime(local, tx.sta,
+							     txq->ac,
+							     airtime,
+							     false);
+		}
+	}
+
 	return skb;
 
 out:
Signature · 1 line
2.24.0.432.g9d3f5f5b63-goog
#13 lore raw

Re: [PATCH v11 4/4] mac80211: Use Airtime-based Queue Limits (AQL) on packet dequeue

Hi,

I've take a closer look at the AQL implementation, and I found some corner cases that need to be addressed soon:

- AQL estimated airtime does not take into account A-MPDU, so it is significantly overestimating airtime use for aggregated traffic, especially on high rates.
My proposed solution would be to check for a running aggregation session and set estimated tx time to something like:
expected_airtime(16 * skb->len) / 16.

- We need an API that allows the driver to change the pending airtime values, e.g. subtract estimated tx time for a packet.
mt76 an ath9k can queue packets inside the driver that are not currently in the hardware queues. Typically if the txqs have more data than what gets put into the hardware queue, both drivers will pull an extra frame and queue it in its private txq struct. This frame will get used on the next txq scheduling round for that particular station.
If you have lots of stations doing traffic (or having driver buffered frames in powersave mode), this could use up a sizable chunk of the AQL budget.
While removing the airtime of those packages would lead to AQL temporarily underestimating airtime, I think it would be better than overestimating it.

I will work on some patches. What do you think about these issues and my proposed fixes?

- Felix

#14 lore raw

Re: [PATCH v11 4/4] mac80211: Use Airtime-based Queue Limits (AQL) on packet dequeue

Quoted 11 lines · Felix Fietkau

Hi,

I've take a closer look at the AQL implementation, and I found some corner cases that need to be addressed soon:

- AQL estimated airtime does not take into account A-MPDU, so it is significantly overestimating airtime use for aggregated traffic, especially on high rates.
My proposed solution would be to check for a running aggregation session and set estimated tx time to something like:
expected_airtime(16 * skb->len) / 16.

This seems reasonable. Not sure if it'll do anything for ath10k (does mac80211 know the aggregation state for that?), but maybe this is not such a big issue on that hardware?

Quoted 10 lines

- We need an API that allows the driver to change the pending airtime values, e.g. subtract estimated tx time for a packet.
mt76 an ath9k can queue packets inside the driver that are not currently in the hardware queues. Typically if the txqs have more data than what gets put into the hardware queue, both drivers will pull an extra frame and queue it in its private txq struct. This frame will get used on the next txq scheduling round for that particular station.
If you have lots of stations doing traffic (or having driver buffered frames in powersave mode), this could use up a sizable chunk of the AQL budget.

I'm a bit more skeptical about this. If the driver buffers a bunch of packets that are not accounted that will hurt that station due to extra latency when it wakes up. For ath9k, this is the retry_q you're talking about, right? The number of packets queued on that is fairly limited, isn't it? What kind of powersave buffering is the driver doing, and why can't it leave the packets on the TXQ? That would allow them to be scheduled along with any new ones that might have arrived in the meantime, which would be a benefit for latency.

I can see how it can be a problem that many stations in powersave can block transmissions for *other* stations, though. Maybe an alternative to the driver subtracting airtime could be to have mac80211 do something like this when a station is put into powersave mode:

local->aql_total_pending_airtime -= sum(sta->airtime[ac].aql_tx_pending)

(where sum is the summation over all ACs)

and the reverse when the station wakes back up? That should keep the whole device from throttling but still prevent a big queue building up for that sta when it wakes back up. Would that work, do you think?

-Toke

author
#15 lore raw

Re: [PATCH v11 4/4] mac80211: Use Airtime-based Queue Limits (AQL) on packet dequeue

Quoted 6 lines

- AQL estimated airtime does not take into account A-MPDU, so it is significantly overestimating airtime use for aggregated traffic, especially on high rates.
My proposed solution would be to check for a running aggregation session and set estimated tx time to something like:
expected_airtime(16 * skb->len) / 16.

Yes, that's a known limitation that needs to be improved. I actually post a comment for this patch:
"[PATCH v10 2/4] mac80211: Import airtime calculation code from mt76"

Quoted 13 lines

When a txq is subjected to the queue limit,
there is usually a significant amount of frames being queued and those frames are likely being sent out in large aggregations. So, in most cases when AQL is active, the medium access overhead can be amortized over many frames and the per frame overhead could be considerably lower than 36, especially at higher data rate. As a result, the pending airtime calculated this way could be higher than the actual airtime. In my test, I have to compensate that by increasing "aql_txq_limit" via debugfs to get the same peak throughput as without AQL.

Quoted 3 lines

My proposed solution would be to check for a running aggregation session and set estimated tx time to something like:
expected_airtime(16 * skb->len) / 16.

I think that's a reasonable approximation, but doubts aggregation information is available in all architectures. In some architecture, firmware may only report aggregation information after the frame has been transmitted.

In my earlier version of AQL for the out-of-tree ChromeOS kernel, A-MPDU is dealt this way: The the medium access overhead is only counted once for each TXQ for all frames released to the hardware over a 4ms period, assuming those frames are likely to be aggregated together.

Instead of calculating the estimated airtime using the last known phy rate, then try to add some estimated overhead for medium access time, another potentially better approach is to use average data rate, which is byte_transmited/firmware_reported_actual_airtime. The average rate not only includes medium access overhead, but also takes retries into account.

Quoted 10 lines

- We need an API that allows the driver to change the pending airtime values, e.g. subtract estimated tx time for a packet.
mt76 an ath9k can queue packets inside the driver that are not currently in the hardware queues. Typically if the txqs have more data than what gets put into the hardware queue, both drivers will pull an extra frame and queue it in its private txq struct. This frame will get used on the next txq scheduling round for that particular station.
If you have lots of stations doing traffic (or having driver buffered frames in powersave mode), this could use up a sizable chunk of the AQL budget.

Not sure I fully understand your concerns here. The AQL budget is per STA, per TID, so frames queued in the driver's special queue for one station won't impact another station's budget. Those frames are counted towards the per interface pending airtime, which could trigger AQL to switch to use the lower queue limit. In this case, that could be the desirable behavior when there is heavy traffic.

Regards,
Kan

Quoted 54 lines · Toke Høiland-Jørgensen
Quoted 11 lines · Felix Fietkau

Hi,

I've take a closer look at the AQL implementation, and I found some corner cases that need to be addressed soon:

- AQL estimated airtime does not take into account A-MPDU, so it is significantly overestimating airtime use for aggregated traffic, especially on high rates.
My proposed solution would be to check for a running aggregation session and set estimated tx time to something like:
expected_airtime(16 * skb->len) / 16.

This seems reasonable. Not sure if it'll do anything for ath10k (does mac80211 know the aggregation state for that?), but maybe this is not such a big issue on that hardware?

Quoted 10 lines

- We need an API that allows the driver to change the pending airtime values, e.g. subtract estimated tx time for a packet.
mt76 an ath9k can queue packets inside the driver that are not currently in the hardware queues. Typically if the txqs have more data than what gets put into the hardware queue, both drivers will pull an extra frame and queue it in its private txq struct. This frame will get used on the next txq scheduling round for that particular station.
If you have lots of stations doing traffic (or having driver buffered frames in powersave mode), this could use up a sizable chunk of the AQL budget.

I'm a bit more skeptical about this. If the driver buffers a bunch of packets that are not accounted that will hurt that station due to extra latency when it wakes up. For ath9k, this is the retry_q you're talking about, right? The number of packets queued on that is fairly limited, isn't it? What kind of powersave buffering is the driver doing, and why can't it leave the packets on the TXQ? That would allow them to be scheduled along with any new ones that might have arrived in the meantime, which would be a benefit for latency.

I can see how it can be a problem that many stations in powersave can block transmissions for *other* stations, though. Maybe an alternative to the driver subtracting airtime could be to have mac80211 do something like this when a station is put into powersave mode:

local->aql_total_pending_airtime -= sum(sta->airtime[ac].aql_tx_pending)

(where sum is the summation over all ACs)

and the reverse when the station wakes back up? That should keep the whole device from throttling but still prevent a big queue building up for that sta when it wakes back up. Would that work, do you think?

-Toke

#16 lore raw

Re: [PATCH v11 4/4] mac80211: Use Airtime-based Queue Limits (AQL) on packet dequeue

Quoted 23 lines · Kan Yan
Quoted 6 lines

- AQL estimated airtime does not take into account A-MPDU, so it is significantly overestimating airtime use for aggregated traffic, especially on high rates.
My proposed solution would be to check for a running aggregation session and set estimated tx time to something like:
expected_airtime(16 * skb->len) / 16.

Yes, that's a known limitation that needs to be improved. I actually post a comment for this patch:
"[PATCH v10 2/4] mac80211: Import airtime calculation code from mt76"

Quoted 11 lines

When a txq is subjected to the queue limit,
there is usually a significant amount of frames being queued and those frames are likely being sent out in large aggregations. So, in most cases when AQL is active, the medium access overhead can be amortized over many frames and the per frame overhead could be considerably lower than 36, especially at higher data rate. As a result, the pending airtime calculated this way could be higher than the actual airtime. In my test, I have to compensate that by increasing "aql_txq_limit" via debugfs to get the same peak throughput as without AQL.

Quoted 8 lines
Quoted 3 lines

My proposed solution would be to check for a running aggregation session and set estimated tx time to something like:
expected_airtime(16 * skb->len) / 16.

I think that's a reasonable approximation, but doubts aggregation information is available in all architectures. In some architecture, firmware may only report aggregation information after the frame has been transmitted.

I'm not proposing using frame transmission aggregation. It would be a simple approximation where it would use a fixed assumed average aggregation size if an aggregation session is established.

Quoted 5 lines

In my earlier version of AQL for the out-of-tree ChromeOS kernel, A-MPDU is dealt this way: The the medium access overhead is only counted once for each TXQ for all frames released to the hardware over a 4ms period, assuming those frames are likely to be aggregated together.

I think that would be more accurate, but probably also more expensive to calculate.

Quoted 6 lines

Instead of calculating the estimated airtime using the last known phy rate, then try to add some estimated overhead for medium access time, another potentially better approach is to use average data rate, which is byte_transmited/firmware_reported_actual_airtime. The average rate not only includes medium access overhead, but also takes retries into account.

Also an interesting idea, though probably not available on all hardware.

Quoted 11 lines
Quoted 10 lines

- We need an API that allows the driver to change the pending airtime values, e.g. subtract estimated tx time for a packet.
mt76 an ath9k can queue packets inside the driver that are not currently in the hardware queues. Typically if the txqs have more data than what gets put into the hardware queue, both drivers will pull an extra frame and queue it in its private txq struct. This frame will get used on the next txq scheduling round for that particular station.
If you have lots of stations doing traffic (or having driver buffered frames in powersave mode), this could use up a sizable chunk of the AQL budget.

Quoted 6 lines

Not sure I fully understand your concerns here. The AQL budget is per STA, per TID, so frames queued in the driver's special queue for one station won't impact another station's budget. Those frames are counted towards the per interface pending airtime, which could trigger AQL to switch to use the lower queue limit. In this case, that could be the desirable behavior when there is heavy traffic.

Yes, the per interface limit is what I'm concerned about. I'm not sure if it will be an issue in practice, it's just something that I noticed while reviewing the code.

- Felix

#17 lore raw

Re: [PATCH v11 4/4] mac80211: Use Airtime-based Queue Limits (AQL) on packet dequeue

Quoted 9 lines · Felix Fietkau
Quoted 6 lines

Not sure I fully understand your concerns here. The AQL budget is per STA, per TID, so frames queued in the driver's special queue for one station won't impact another station's budget. Those frames are counted towards the per interface pending airtime, which could trigger AQL to switch to use the lower queue limit. In this case, that could be the desirable behavior when there is heavy traffic.

Yes, the per interface limit is what I'm concerned about. I'm not sure if it will be an issue in practice, it's just something that I noticed while reviewing the code.

Ah, right. Note that it's not a hard limit, though; it's just a threshold for when the per-station limit switches to a smaller value. The idea is that when there are many stations with outstanding data, each station's latency will be higher since they have to also wait for their turn to transmit. And so we compensate for this by lowering each station's queue limit, since the hardware is unlikely to become idle when there are many stations to send to.

The lower limit is still 5ms of data, though, so there should still be enough for a full aggregate queued for each station. Arguably the limit could actually be made *smaller*: On a very busy network with lots of stations transmitting at once, IMO we *don't* want to send full-sized aggregates as that will add up to way too much latency. Better to sacrifice a bit of network efficiency to raise responsiveness :)

-Toke

#18 lore raw

Re: [PATCH v11 4/4] mac80211: Use Airtime-based Queue Limits (AQL) on packet dequeue

Quoted 20 lines · Toke Høiland-Jørgensen
Quoted 9 lines · Felix Fietkau
Quoted 6 lines

Not sure I fully understand your concerns here. The AQL budget is per STA, per TID, so frames queued in the driver's special queue for one station won't impact another station's budget. Those frames are counted towards the per interface pending airtime, which could trigger AQL to switch to use the lower queue limit. In this case, that could be the desirable behavior when there is heavy traffic.

Yes, the per interface limit is what I'm concerned about. I'm not sure if it will be an issue in practice, it's just something that I noticed while reviewing the code.

Ah, right. Note that it's not a hard limit, though; it's just a threshold for when the per-station limit switches to a smaller value. The idea is that when there are many stations with outstanding data, each station's latency will be higher since they have to also wait for their turn to transmit. And so we compensate for this by lowering each station's queue limit, since the hardware is unlikely to become idle when there are many stations to send to.

Quoted 6 lines

The lower limit is still 5ms of data, though, so there should still be enough for a full aggregate queued for each station. Arguably the limit could actually be made *smaller*: On a very busy network with lots of stations transmitting at once, IMO we *don't* want to send full-sized aggregates as that will add up to way too much latency. Better to sacrifice a bit of network efficiency to raise responsiveness :)

Makes sense, thanks.

- Felix

#19 lore raw

Re: [PATCH v11 4/4] mac80211: Use Airtime-based Queue Limits (AQL) on packet dequeue

In reply to Toke Høiland-Jørgensen
Quoted 12 lines · Toke Høiland-Jørgensen
Quoted 10 lines · Felix Fietkau

- We need an API that allows the driver to change the pending airtime values, e.g. subtract estimated tx time for a packet.
mt76 an ath9k can queue packets inside the driver that are not currently in the hardware queues. Typically if the txqs have more data than what gets put into the hardware queue, both drivers will pull an extra frame and queue it in its private txq struct. This frame will get used on the next txq scheduling round for that particular station.
If you have lots of stations doing traffic (or having driver buffered frames in powersave mode), this could use up a sizable chunk of the AQL budget.

Quoted 8 lines

I'm a bit more skeptical about this. If the driver buffers a bunch of packets that are not accounted that will hurt that station due to extra latency when it wakes up. For ath9k, this is the retry_q you're talking about, right? The number of packets queued on that is fairly limited, isn't it? What kind of powersave buffering is the driver doing, and why can't it leave the packets on the TXQ? That would allow them to be scheduled along with any new ones that might have arrived in the meantime, which would be a benefit for latency.

For mt76 there should be max. 1 frame in the retry queue, it's just a frame that was pulled from the txq in a transmission attempt but that it couldn't put in the hw queue because it didn't fit in the current aggregate batch.
If such a frame is in the retry queue and the sta ends up switching to powersave mode, that frame stays buffered in the driver queue until the sta wakes up.

Quoted 5 lines

I can see how it can be a problem that many stations in powersave can block transmissions for *other* stations, though. Maybe an alternative to the driver subtracting airtime could be to have mac80211 do something like this when a station is put into powersave mode:

Quoted 7 lines

local->aql_total_pending_airtime -= sum(sta->airtime[ac].aql_tx_pending)

(where sum is the summation over all ACs)

and the reverse when the station wakes back up? That should keep the whole device from throttling but still prevent a big queue building up for that sta when it wakes back up. Would that work, do you think?

That solves most of the issue but is still incomplete. It also gets tricky when the driver starts pulling dequeueing packets in powersave mode (e.g. on PS-Poll).
The remaining corner case is when there are a large number of stations that each have a single frame in their retry queue (see above). Having those frames counted for pending airtime could reduce the aggregation size for each station, even though those frames are not in the hw queue.

- Felix

#20 lore raw

Re: [PATCH v11 4/4] mac80211: Use Airtime-based Queue Limits (AQL) on packet dequeue

Quoted 21 lines · Felix Fietkau
Quoted 12 lines · Toke Høiland-Jørgensen
Quoted 10 lines · Felix Fietkau

- We need an API that allows the driver to change the pending airtime values, e.g. subtract estimated tx time for a packet.
mt76 an ath9k can queue packets inside the driver that are not currently in the hardware queues. Typically if the txqs have more data than what gets put into the hardware queue, both drivers will pull an extra frame and queue it in its private txq struct. This frame will get used on the next txq scheduling round for that particular station.
If you have lots of stations doing traffic (or having driver buffered frames in powersave mode), this could use up a sizable chunk of the AQL budget.

Quoted 8 lines

I'm a bit more skeptical about this. If the driver buffers a bunch of packets that are not accounted that will hurt that station due to extra latency when it wakes up. For ath9k, this is the retry_q you're talking about, right? The number of packets queued on that is fairly limited, isn't it? What kind of powersave buffering is the driver doing, and why can't it leave the packets on the TXQ? That would allow them to be scheduled along with any new ones that might have arrived in the meantime, which would be a benefit for latency.

Quoted 4 lines

For mt76 there should be max. 1 frame in the retry queue, it's just a frame that was pulled from the txq in a transmission attempt but that it couldn't put in the hw queue because it didn't fit in the current aggregate batch.

Wait, if it's only a single frame that is queued in the driver, how is this causing problems? We deliberately set the limit so there was a bit of slack above the size of an aggregate for things like this. Could you please describe in a bit more detail what symptoms you are seeing of this problem? :)

-Toke

#21 lore raw

Re: [PATCH v11 4/4] mac80211: Use Airtime-based Queue Limits (AQL) on packet dequeue

Quoted 28 lines · Toke Høiland-Jørgensen
Quoted 21 lines · Felix Fietkau
Quoted 12 lines · Toke Høiland-Jørgensen
Quoted 10 lines · Felix Fietkau

- We need an API that allows the driver to change the pending airtime values, e.g. subtract estimated tx time for a packet.
mt76 an ath9k can queue packets inside the driver that are not currently in the hardware queues. Typically if the txqs have more data than what gets put into the hardware queue, both drivers will pull an extra frame and queue it in its private txq struct. This frame will get used on the next txq scheduling round for that particular station.
If you have lots of stations doing traffic (or having driver buffered frames in powersave mode), this could use up a sizable chunk of the AQL budget.

Quoted 8 lines

I'm a bit more skeptical about this. If the driver buffers a bunch of packets that are not accounted that will hurt that station due to extra latency when it wakes up. For ath9k, this is the retry_q you're talking about, right? The number of packets queued on that is fairly limited, isn't it? What kind of powersave buffering is the driver doing, and why can't it leave the packets on the TXQ? That would allow them to be scheduled along with any new ones that might have arrived in the meantime, which would be a benefit for latency.

Quoted 4 lines

For mt76 there should be max. 1 frame in the retry queue, it's just a frame that was pulled from the txq in a transmission attempt but that it couldn't put in the hw queue because it didn't fit in the current aggregate batch.

Quoted 5 lines

Wait, if it's only a single frame that is queued in the driver, how is this causing problems? We deliberately set the limit so there was a bit of slack above the size of an aggregate for things like this. Could you please describe in a bit more detail what symptoms you are seeing of this problem? :)

It would be a single frame per sta/txq. I don't know if it will cause problems in practice, it's just a potential corner case that I found during review. I'd imagine this would probably show up in some benchmarks at least.
I'm not seeing any symptoms myself, but I also haven't run any intricate tests yet.

- Felix

#22 lore raw

Re: [PATCH v11 4/4] mac80211: Use Airtime-based Queue Limits (AQL) on packet dequeue

Quoted 34 lines · Felix Fietkau
Quoted 28 lines · Toke Høiland-Jørgensen
Quoted 21 lines · Felix Fietkau
Quoted 12 lines · Toke Høiland-Jørgensen
Quoted 10 lines · Felix Fietkau

- We need an API that allows the driver to change the pending airtime values, e.g. subtract estimated tx time for a packet.
mt76 an ath9k can queue packets inside the driver that are not currently in the hardware queues. Typically if the txqs have more data than what gets put into the hardware queue, both drivers will pull an extra frame and queue it in its private txq struct. This frame will get used on the next txq scheduling round for that particular station.
If you have lots of stations doing traffic (or having driver buffered frames in powersave mode), this could use up a sizable chunk of the AQL budget.

Quoted 8 lines

I'm a bit more skeptical about this. If the driver buffers a bunch of packets that are not accounted that will hurt that station due to extra latency when it wakes up. For ath9k, this is the retry_q you're talking about, right? The number of packets queued on that is fairly limited, isn't it? What kind of powersave buffering is the driver doing, and why can't it leave the packets on the TXQ? That would allow them to be scheduled along with any new ones that might have arrived in the meantime, which would be a benefit for latency.

Quoted 4 lines

For mt76 there should be max. 1 frame in the retry queue, it's just a frame that was pulled from the txq in a transmission attempt but that it couldn't put in the hw queue because it didn't fit in the current aggregate batch.

Quoted 5 lines

Wait, if it's only a single frame that is queued in the driver, how is this causing problems? We deliberately set the limit so there was a bit of slack above the size of an aggregate for things like this. Could you please describe in a bit more detail what symptoms you are seeing of this problem? :)

Quoted 6 lines

It would be a single frame per sta/txq. I don't know if it will cause problems in practice, it's just a potential corner case that I found during review. I'd imagine this would probably show up in some benchmarks at least.
I'm not seeing any symptoms myself, but I also haven't run any intricate tests yet.

See my other reply; I'm not convinced this behaviour is wrong :)

-Toke

#23 lore raw

Re: [PATCH v11 4/4] mac80211: Use Airtime-based Queue Limits (AQL) on packet dequeue

In reply to Kan Yan
Quoted 8 lines

ieee80211_report_used_skb(). As an optimisation, we also subtract the airtime on regular TX completion, zeroing out the value stored in the packet afterwards, to avoid having to do an expensive lookup of the station from the packet data on every packet.

This patch does *not* include any mechanism to wake a throttled TXQ again, on the assumption that this will happen anyway as a side effect of whatever freed the skb (most commonly a TX completion).

I recall a recent patch for ath10k sdio that disabled tx acknowledgement for performance gains and am wondering if that will be problematic? Presumably not since it would be caught at the dequeue, but thought I'd ask-- wondering what the effect of failed tx's or block acknowledgement is on this stuff I'll need to study the code some more

https://lore.kernel.org/linux-wireless/0101016eb1903db0-ef7063b4-0f42-4a01-8886-327541e6c1a4-000000@us-west-2.amazonses.com/T/#t

#24 lore raw

Re: [PATCH v11 4/4] mac80211: Use Airtime-based Queue Limits (AQL) on packet dequeue

Quoted 9 lines · Justin Capella
Quoted 8 lines

ieee80211_report_used_skb(). As an optimisation, we also subtract the airtime on regular TX completion, zeroing out the value stored in the packet afterwards, to avoid having to do an expensive lookup of the station from the packet data on every packet.

This patch does *not* include any mechanism to wake a throttled TXQ again, on the assumption that this will happen anyway as a side effect of whatever freed the skb (most commonly a TX completion).

Quoted 8 lines

I recall a recent patch for ath10k sdio that disabled tx acknowledgement for performance gains and am wondering if that will be problematic? Presumably not since it would be caught at the dequeue, but thought I'd ask-- wondering what the effect of failed tx's or block acknowledgement is on this stuff I'll need to study the code some more

https://lore.kernel.org/linux-wireless/0101016eb1903db0-ef7063b4-0f42-4a01-8886-327541e6c1a4-000000@us-west-2.amazonses.com/T/#t

It looks like that patch will just end up disabling AQL (because packets will be immediately completed as far as mac80211 is concerned) and replace it with whatever that credit-based scheme does? No idea how that will impact latency; you should go ask the developers of that series! As usual, the patch description only mentions throughput numbers :/

-Toke