Implement Airtime-based Queue Limit (AQL)
v4 cover Implement Airtime-based Queue Limit (AQL)
This patch series implements Airtime-based Queue Limit (AQL) in the mac80211 and Ath10k driver. It is based on an earlier version from the ChromiumOS tree[0].
This version has been tested with QCA9884 platform with 4.14 kernel. Tests show AQL is able to reduce latency by an order of magnitude in a congested environment without negative impact on the throughput.
[0] https://chromium-review.googlesource.com/c/chromiumos/third_party/kernel/+/1703105/7
Kan Yan (2):
mac80211: Implement Airtime-based Queue Limit (AQL)
ath10k: Enable Airtime-based Queue Limit (AQL)
drivers/net/wireless/ath/ath10k/htt_rx.c | 1 + drivers/net/wireless/ath/ath10k/mac.c | 6 +- drivers/net/wireless/ath/ath10k/txrx.c | 11 +++- include/net/cfg80211.h | 7 +++ include/net/mac80211.h | 47 ++++++++++++++ net/mac80211/debugfs.c | 78 ++++++++++++++++++++++++ net/mac80211/debugfs_sta.c | 43 ++++++++++--- net/mac80211/ieee80211_i.h | 4 ++ net/mac80211/main.c | 9 ++- net/mac80211/sta_info.c | 33 ++++++++++ net/mac80211/sta_info.h | 4 ++ net/mac80211/tx.c | 46 +++++++++++++- 12 files changed, 270 insertions(+), 19 deletions(-)
Signature · 12 lines
Changes from v1: - Fix checkpatch error. - Keep iterate the list of active_txq until an eligible txq's deficit become non-negative in ieee80211_next_txq(), instead of break the loop after one iteration. - Enforce the AQL limit in ath10k's pull mode in ath10k_htt_rx_tx_fetch_ind() Changes from v2: - More coding style fixes. - Fixed a bug that txq is not removed from the active_txqs list when txq.sta is null. - Removed AQL queue limit check from ieee80211_txq_may_transmit(), so at least one frame will be sent to the firmware when called from ath10k_htt_rx_tx_fetch_ind() in ath10k's poll mode. Changes from v3: - Coding style fixes.
PATCH v4 1/2 mac80211: Implement Airtime-based Queue Limit (AQL)
In order for the Fq_CoDel integrated in mac80211 layer operates effectively to control excessive queueing latency, the CoDel algorithm requires an accurate measure of how long the packets stays in the queue, aka sojourn time. The sojourn time measured at mac80211 layer doesn't include queueing latency in lower layer (firmware/hardware) and CoDel expects lower layer to have a short queue. However, most 802.11ac chipsets offload tasks such TX aggregation to firmware or hardware, thus have a deep lower layer queue. Without a mechanism to control the lower layer queue size, packets only stays in mac80211 layer transiently before being sent to firmware queue. As a result, the sojourn time measured by CoDel in the mac80211 layer is almost always lower than the CoDel latency target, hence CoDel does little to control the latency, even when the lower layer queue causes excessive latency.
Byte Queue limits (BQL) is commonly used to address the similar issue with wired network interface. However, this method cannot be applied directly to the wireless network interface. Byte is not a suitable measure of queue depth in the wireless network, as the data rate can vary dramatically from station to station in the same network, from a few Mbps to over Gbps.
This patch implemented an Airtime-based Queue Limit (AQL) to make CoDel works effectively with wireless drivers that utilized firmware/hardware offloading. AQL only allows each txq to release just enough packets to the lower layer to form 1-2 large aggregations to keep hardware fully utilized and keep the rest of frames in mac80211 layer to be controlled by the CoDel algorithm.
--- include/net/cfg80211.h | 7 ++++ include/net/mac80211.h | 47 +++++++++++++++++++++++ net/mac80211/debugfs.c | 78 ++++++++++++++++++++++++++++++++++++++ net/mac80211/debugfs_sta.c | 43 ++++++++++++++++----- net/mac80211/ieee80211_i.h | 4 ++ net/mac80211/main.c | 9 ++++- net/mac80211/sta_info.c | 33 ++++++++++++++++ net/mac80211/sta_info.h | 4 ++ net/mac80211/tx.c | 46 ++++++++++++++++++++-- 9 files changed, 257 insertions(+), 14 deletions(-)
include/net/cfg80211.h +7 -0
@@ -2499,6 +2499,13 @@ enum wiphy_params_flags {#define IEEE80211_DEFAULT_AIRTIME_WEIGHT 256 +/* The per TXQ device queue limit in airtime */ +#define IEEE80211_DEFAULT_AQL_TXQ_LIMIT_L 4000 +#define IEEE80211_DEFAULT_AQL_TXQ_LIMIT_H 8000 + +/* The per interface airtime threshold to switch to lower queue limit */ +#define IEEE80211_AQL_THRESHOLD 24000 +/** * struct cfg80211_pmksa - PMK Security Association *
include/net/mac80211.h +47 -0
53 lines not on this page · raw message
net/mac80211/debugfs.c +78 -0
90 lines not on this page · raw message
net/mac80211/debugfs_sta.c +33 -10
69 lines not on this page · raw message
net/mac80211/ieee80211_i.h +4 -0
@@ -1142,6 +1142,10 @@ struct ieee80211_local {u16 schedule_round[IEEE80211_NUM_ACS ];u16 airtime_flags; +u32 aql_txq_limit_low[IEEE80211_NUM_ACS ]; +u32 aql_txq_limit_high[IEEE80211_NUM_ACS ]; +u32 aql_threshold; +u32 aql_total_pending_airtime;const struct ieee80211_ops *ops;
net/mac80211/main.c +8 -1
@@ -666,8 +666,15 @@ struct ieee80211_hw *ieee80211_alloc_hw_nm(size_t priv_data_len,for (i =0 ; i <IEEE80211_NUM_ACS ; i++) {INIT_LIST_HEAD (&local->active_txqs[i]);spin_lock_init (&local->active_txq_lock[i]); + local->aql_txq_limit_low[i] =IEEE80211_DEFAULT_AQL_TXQ_LIMIT_L ; + local->aql_txq_limit_high[i] = +IEEE80211_DEFAULT_AQL_TXQ_LIMIT_H ; } - local->airtime_flags =AIRTIME_USE_TX |AIRTIME_USE_RX ; + + local->airtime_flags =AIRTIME_USE_TX | +AIRTIME_USE_RX | +AIRTIME_USE_AQL ; + local->aql_threshold =IEEE80211_AQL_THRESHOLD ;INIT_LIST_HEAD (&local->chanctx_list);mutex_init (&local->chanctx_mtx);
net/mac80211/sta_info.c +33 -0
@@ -396,6 +396,9 @@ struct sta_info *sta_info_alloc(struct ieee80211_sub_if_data *sdata,skb_queue_head_init (&sta->ps_tx_buf[i]);skb_queue_head_init (&sta->tx_filtered[i]); sta->airtime[i].deficit = sta->airtime_weight; + sta->airtime[i].aql_tx_pending =0 ; + sta->airtime[i].aql_limit_low = local->aql_txq_limit_low[i]; + sta->airtime[i].aql_limit_high = local->aql_txq_limit_high[i]; }for (i =0 ; i <IEEE80211_NUM_TIDS ; i++)
@@ -1894,6 +1897,36 @@ void ieee80211_sta_register_airtime(struct ieee80211_sta *pubsta, u8 tid, }EXPORT_SYMBOL (ieee80211_sta_register_airtime); +void ieee80211_sta_update_pending_airtime (struct ieee80211_sta *pubsta,u8 tid, +u32 tx_airtime,bool tx_completed) +{ +u8 ac =ieee80211_ac_from_tid (tid); + struct sta_info *sta = container_of(pubsta, struct sta_info, sta); +struct ieee80211_local *local = sta->local; + +spin_lock_bh (&local->active_txq_lock[ac]); +if (tx_completed) { +if (WARN_ONCE (sta->airtime[ac].aql_tx_pending < tx_airtime, +"TXQ pending airtime underflow: %u, %u" , + sta->airtime[ac].aql_tx_pending, tx_airtime)) + sta->airtime[ac].aql_tx_pending =0 ; +else + sta->airtime[ac].aql_tx_pending -= tx_airtime; + +if (WARN_ONCE (local->aql_total_pending_airtime < tx_airtime, +"Device pending airtime underflow: %u, %u" , + local->aql_total_pending_airtime, tx_airtime)) + local->aql_total_pending_airtime =0 ; +else + local->aql_total_pending_airtime -= tx_airtime; + }else { + sta->airtime[ac].aql_tx_pending += tx_airtime; + local->aql_total_pending_airtime += tx_airtime; + } +spin_unlock_bh (&local->active_txq_lock[ac]); +} +EXPORT_SYMBOL (ieee80211_sta_update_pending_airtime); +int sta_info_move_state (struct sta_info *sta,enum ieee80211_sta_state new_state) {
net/mac80211/sta_info.h +4 -0
@@ -127,11 +127,15 @@ enum ieee80211_agg_stop_reason {/* Debugfs flags to enable/disable use of RX/TX airtime in scheduler */ #define AIRTIME_USE_TX BIT(0)#define AIRTIME_USE_RX BIT(1) +#define AIRTIME_USE_AQL BIT(2)struct airtime_info {u64 rx_airtime;u64 tx_airtime;s64 deficit; +u32 aql_tx_pending;/* Estimated airtime for frames pending in queue */ +u32 aql_limit_low; +u32 aql_limit_high; };struct sta_info ;
net/mac80211/tx.c +43 -3
69 lines not on this page · raw message
Signature · 1 line
2.23.0.700.g56cf767bdb-goog
PATCH v4 2/2 ath10k: Enable Airtime-based Queue Limit (AQL)
In reply to Kan YanCalculate the estimated airtime pending in the txqs and apply AQL to prevent excessive amounts of packets being queued in the firmware queue.
--- drivers/net/wireless/ath/ath10k/htt_rx.c | 1 + drivers/net/wireless/ath/ath10k/mac.c | 6 ++++-- drivers/net/wireless/ath/ath10k/txrx.c | 11 ++++++++--- 3 files changed, 13 insertions(+), 5 deletions(-)
drivers/net/wireless/ath/ath10k/htt_rx.c +1 -0
@@ -3053,6 +3053,7 @@ static void ath10k_htt_rx_tx_fetch_ind(struct ath10k *ar, struct sk_buff *skb) num_msdus++; num_bytes += ret; + may_tx =ieee80211_txq_airtime_check (hw, txq); }ieee80211_return_txq (hw, txq, false);ieee80211_txq_schedule_end (hw, txq->ac);
drivers/net/wireless/ath/ath10k/mac.c +4 -2
@@ -3983,6 +3983,9 @@ static bool ath10k_mac_tx_can_push(struct ieee80211_hw *hw,struct ath10k_txq *artxq = (void *)txq->drv_priv;/* No need to get locks */ +if (!ieee80211_txq_airtime_check (hw, txq)) +return false; +if (ar->htt.tx_q_state.mode ==HTT_TX_MODE_SWITCH_PUSH )return true;
@@ -4014,8 +4017,6 @@ static u16 ath10k_mac_update_airtime(struct ath10k *ar,if (!txq || !txq->sta)return airtime; -if (test_bit (WMI_SERVICE_REPORT_AIRTIME , ar->wmi.svc_map)) -return airtime;spin_lock_bh (&ar->data_lock); arsta = (struct ath10k_sta *)txq->sta->drv_priv;
@@ -4038,6 +4039,7 @@ static u16 ath10k_mac_update_airtime(struct ath10k *ar, }spin_unlock_bh (&ar->data_lock); +ieee80211_sta_register_pending_airtime (txq->sta, txq->tid, airtime);return airtime; }
drivers/net/wireless/ath/ath10k/txrx.c +8 -3
@@ -84,9 +84,14 @@ int ath10k_txrx_tx_unref(struct ath10k_htt *htt,wake_up (&htt->empty_tx_wq);spin_unlock_bh (&htt->tx_lock); -if (txq && txq->sta && skb_cb->airtime_est) -ieee80211_sta_register_airtime (txq->sta, txq->tid, - skb_cb->airtime_est,0 ); +if (txq && txq->sta && skb_cb->airtime_est) { +if (!test_bit (WMI_SERVICE_REPORT_AIRTIME , ar->wmi.svc_map)) +ieee80211_sta_register_airtime (txq->sta, txq->tid, + skb_cb->airtime_est,0 ); + +ieee80211_sta_release_pending_airtime (txq->sta, txq->tid, + skb_cb->airtime_est); + }if (ar->bus_param.dev_type !=ATH10K_DEV_TYPE_HL )dma_unmap_single (dev, skb_cb->paddr, msdu->len,DMA_TO_DEVICE );
Signature · 1 line
2.23.0.700.g56cf767bdb-goog
Re: [PATCH v4 2/2] ath10k: Enable Airtime-based Queue Limit (AQL)
Quoted 4 lines · Kan Yan
Calculate the estimated airtime pending in the txqs and apply AQL to prevent excessive amounts of packets being queued in the firmware queue.
Signed-off-by Kan Yan <[email protected]>
Forgot to mention earlier that please add tested hardware and firmware versions:
https://wireless.wiki.kernel.org/en/users/drivers/ath10k/submittingpatches#guidelines
I can also add that during commit, no need to resend because of this.
Signature · 1 line
https://wireless.wiki.kernel.org/en/developers/documentation/submittingpatches
Re: [PATCH v4 2/2] ath10k: Enable Airtime-based Queue Limit (AQL)
In reply to Kan YanQuoted 4 lines · Kan Yan
Calculate the estimated airtime pending in the txqs and apply AQL to prevent excessive amounts of packets being queued in the firmware queue.
Signed-off-by Kan Yan <[email protected]>
Dropped per Kan's and Toke's comment:
https://lkml.kernel.org/r/[email protected]
Patch set to Rejected.
Signature · 9 lines
https://patchwork.kernel.org/patch/11184783/ https://wireless.wiki.kernel.org/en/developers/documentation/submittingpatches _______________________________________________ ath10k mailing list [email protected] http://lists.infradead.org/mailman/listinfo/ath10k
Re: [PATCH v4 2/2] ath10k: Enable Airtime-based Queue Limit (AQL)
In reply to Kan YanQuoted 4 lines · Kan Yan
Calculate the estimated airtime pending in the txqs and apply AQL to prevent excessive amounts of packets being queued in the firmware queue.
Signed-off-by Kan Yan <[email protected]>
Dropped per Kan's and Toke's comment:
https://lkml.kernel.org/r/[email protected]
Patch set to Rejected.
Signature · 3 lines
https://patchwork.kernel.org/patch/11184783/ https://wireless.wiki.kernel.org/en/developers/documentation/submittingpatches
Quoted 9 lines · Kan Yan
This patch series implements Airtime-based Queue Limit (AQL) in the mac80211 and Ath10k driver. It is based on an earlier version from the ChromiumOS tree[0].
This version has been tested with QCA9884 platform with 4.14 kernel. Tests show AQL is able to reduce latency by an order of magnitude in a congested environment without negative impact on the throughput.
[0] https://chromium-review.googlesource.com/c/chromiumos/third_party/kernel/+/1703105/7
Kan Yan (2):
mac80211: Implement Airtime-based Queue Limit (AQL)
ath10k: Enable Airtime-based Queue Limit (AQL)
Please always include a change log so that people know what has changed since the previous version:
https://wireless.wiki.kernel.org/en/developers/documentation/submittingpatches#changelog_missing
Signature · 1 line
https://wireless.wiki.kernel.org/en/developers/documentation/submittingpatches
Quoted 13 lines · Kalle Valo
Quoted 11 lines · Kan Yan
This patch series implements Airtime-based Queue Limit (AQL) in the mac80211 and Ath10k driver. It is based on an earlier version from the ChromiumOS tree[0].
This version has been tested with QCA9884 platform with 4.14 kernel. Tests show AQL is able to reduce latency by an order of magnitude in a congested environment without negative impact on the throughput.
[0]
https://chromium-review.googlesource.com/c/chromiumos/third_party/kernel/+/1703105/7
Quoted 8 lines
Quoted 3 lines
Kan Yan (2):
mac80211: Implement Airtime-based Queue Limit (AQL)
ath10k: Enable Airtime-based Queue Limit (AQL)Please always include a change log so that people know what has changed since the previous version:
https://wireless.wiki.kernel.org/en/developers/documentation/submittingpatches#changelog_missing
Never mind, you actually had the changelog after the signature separator ("-- \n" line) and I automatically skip the signature :) So instead I recommend moving the change log up and above the signature separator.
Signature · 6 lines
https://wireless.wiki.kernel.org/en/developers/documentation/submittingpatches _______________________________________________ ath10k mailing list [email protected] http://lists.infradead.org/mailman/listinfo/ath10k
Hi Kalle,
Thanks for the help and tips. Will do that if I need to submit again.
I believe Toke will integrate this with his version and move the estimating pending airtime part to mac80211, so maybe in the next version, ath10k change is no longer required.
Thanks,
Kan
Quoted 31 lines · Kalle Valo
Quoted 13 lines · Kalle Valo
Quoted 11 lines · Kan Yan
This patch series implements Airtime-based Queue Limit (AQL) in the mac80211 and Ath10k driver. It is based on an earlier version from the ChromiumOS tree[0].
This version has been tested with QCA9884 platform with 4.14 kernel. Tests show AQL is able to reduce latency by an order of magnitude in a congested environment without negative impact on the throughput.
[0]
https://chromium-review.googlesource.com/c/chromiumos/third_party/kernel/+/1703105/7Quoted 8 lines
Quoted 3 lines
Kan Yan (2):
mac80211: Implement Airtime-based Queue Limit (AQL)
ath10k: Enable Airtime-based Queue Limit (AQL)Please always include a change log so that people know what has changed since the previous version:
https://wireless.wiki.kernel.org/en/developers/documentation/submittingpatches#changelog_missing
Never mind, you actually had the changelog after the signature separator ("-- \n" line) and I automatically skip the signature :) So instead I recommend moving the change log up and above the signature separator.
--
https://wireless.wiki.kernel.org/en/developers/documentation/submittingpatches
(please don't top post)
Quoted 3 lines · Kan Yan
I believe Toke will integrate this with his version and move the estimating pending airtime part to mac80211, so maybe in the next version, ath10k change is no longer required.
What do you mean? Are you saying that I can drop this patch:
[v4,2/2] ath10k: Enable Airtime-based Queue Limit (AQL)
https://patchwork.kernel.org/patch/11184783/
Signature · 6 lines
https://wireless.wiki.kernel.org/en/developers/documentation/submittingpatches _______________________________________________ ath10k mailing list [email protected] http://lists.infradead.org/mailman/listinfo/ath10k
Quoted 8 lines · Kalle Valo
(please don't top post)
Quoted 3 lines · Kan Yan
I believe Toke will integrate this with his version and move the estimating pending airtime part to mac80211, so maybe in the next version, ath10k change is no longer required.
Quoted 5 lines
What do you mean? Are you saying that I can drop this patch:
[v4,2/2] ath10k: Enable Airtime-based Queue Limit (AQL)
Yes, we're trying to do it all in mac80211. See
https://patchwork.kernel.org/project/linux-wireless/list/?series=190333
-Toke
Quoted 10 lines · Toke Høiland-Jørgensen
Quoted 8 lines · Kalle Valo
(please don't top post)
Quoted 3 lines · Kan Yan
I believe Toke will integrate this with his version and move the estimating pending airtime part to mac80211, so maybe in the next version, ath10k change is no longer required.
Quoted 8 lines
Quoted 5 lines
What do you mean? Are you saying that I can drop this patch:
[v4,2/2] ath10k: Enable Airtime-based Queue Limit (AQL)
Yes, we're trying to do it all in mac80211. See
https://patchwork.kernel.org/project/linux-wireless/list/?series=190333
Thanks, I dropped this patch now.
Signature · 6 lines
Kalle Valo _______________________________________________ ath10k mailing list [email protected] http://lists.infradead.org/mailman/listinfo/ath10k