Implement Airtime-based Queue Limit (AQL)
Subsystems (2)
v8 cover Implement Airtime-based Queue Limit (AQL)
This patch series port the Airtime Queue Limits concept from the out-of-tree ath10k implementation[0] to mac80211. This version takes my patch to do the throttling in mac80211, and replaces the driver API with the mechanism from Toke's series, which instead calculated the expected airtime at dequeue time inside mac80211, storing it in the SKB cb field.
This version has been tested on QCA9984 platform.
[0] https://chromium-review.googlesource.com/c/chromiumos/third_party/kernel/+/1703105/7
Changelog:
v8:
- Includes Toke's v7 version of "mac80211: Import airtime calculation code from mt76"
- Don't clobber sta's customized queue limit when configuring the default via debugfs
- Fix a racing condition when reset aql_tx_pending.
v7:
- Fix aql_total_pending_airtime underflow due to insufficient locking.
v6:
- Fix sta lookup in ieee80211_report_used_skb().
- Move call to ieee80211_sta_update_pending_airtime() to a bit later in
__ieee80211_tx_status()
v5:
- Add missing export of ieee80211_calc_rx_airtime() and make
ieee80211_calc_tx_airtime_rate() static (kbuildbot). - Use skb_get_queue_mapping() to get the AC from the skb.
- Take basic rate configuration for the BSS into account when calculating multicast rate.
v4:
- Fix calculation that clamps the maximum airtime to fit into 10 bits
- Incorporate Rich Brown's nits for the commit message in Kan's patch
- Add fewer local variables to ieee80211_tx_dequeue()
v3:
- Move the tx_time_est field so it's shared with ack_frame_id, and use units of 4us for the value stored in it.
- Move the addition of the Ethernet header size into ieee80211_calc_expected_tx_airtime()
v2:
- Integrate Kan's approach to airtime throttling.
- Hopefully fix the cb struct alignment on big-endian architectures.
Kan Yan (1):
mac80211: Implement Airtime-based Queue Limit (AQL)
Toke Høiland-Jørgensen (1):
mac80211: Import airtime calculation code from mt76
include/net/cfg80211.h | 7 + include/net/mac80211.h | 41 +++ net/mac80211/Makefile | 3 +- net/mac80211/airtime.c | 597 +++++++++++++++++++++++++++++++++++++ net/mac80211/debugfs.c | 85 ++++++ net/mac80211/debugfs_sta.c | 43 ++- net/mac80211/ieee80211_i.h | 8 + net/mac80211/main.c | 10 +- net/mac80211/sta_info.c | 38 +++ net/mac80211/sta_info.h | 8 + net/mac80211/tx.c | 47 ++- 11 files changed, 872 insertions(+), 15 deletions(-) create mode 100644 net/mac80211/airtime.c
Signature · 1 line
2.24.0.rc1.363.gb1bccd3e3d-goog
PATCH v8 1/2 mac80211: Import airtime calculation code from mt76
From: Toke Høiland-Jørgensen <[email protected]>
Felix recently added code to calculate airtime of packets to the mt76 driver. Import this into mac80211 so we can use it for airtime queue limit calculations.
The airtime.c file is copied verbatim from the mt76 driver, and adjusted to be usable in mac80211. This involves:
- Switching to mac80211 data structures.
- Adding support for 160 MHz channels and HE mode.
- Moving the symbol and duration calculations around a bit to avoid rounding with the higher rates and longer symbol times used for HE rates.
The per-rate TX rate calculation is also split out to its own function so it can be used directly for the AQL calculations later.
--- include/net/mac80211.h | 29 ++ net/mac80211/Makefile | 3 +- net/mac80211/airtime.c | 597 +++++++++++++++++++++++++++++++++++++ net/mac80211/ieee80211_i.h | 4 + 4 files changed, 632 insertions(+), 1 deletion(-) create mode 100644 net/mac80211/airtime.c
include/net/mac80211.h +29 -0
@@ -6417,4 +6417,33 @@ void ieee80211_nan_func_match(struct ieee80211_vif *vif,struct cfg80211_nan_match_params *match, gfp_t gfp); +/** +* ieee80211_calc_rx_airtime - calculate estimated transmission airtime for RX. +* +* This function calculates the estimated airtime usage of a frame based on the +* rate information in the RX status struct and the frame length. +* +* @hw: pointer as obtained from ieee80211_alloc_hw() +* @status: &struct ieee80211_rx_status containing the transmission rate +* information. +* @len: frame length in bytes +*/ +u32 ieee80211_calc_rx_airtime (struct ieee80211_hw *hw, +struct ieee80211_rx_status *status, +int len); + +/** +* ieee80211_calc_tx_airtime - calculate estimated transmission airtime for TX. +* +* This function calculates the estimated airtime usage of a frame based on the +* rate information in the TX info struct and the frame length. +* +* @hw: pointer as obtained from ieee80211_alloc_hw() +* @info: &struct ieee80211_tx_info of the frame. +* @len: frame length in bytes +*/ +u32 ieee80211_calc_tx_airtime (struct ieee80211_hw *hw, +struct ieee80211_tx_info *info, +int len); +#endif /* MAC80211_H */
net/mac80211/Makefile +2 -1
@@ -32,7 +32,8 @@ mac80211-y := \ chan.o \ trace.o mlme.o \ tdls.o \ - ocb.o + ocb.o \ + airtime.o mac80211-$(CONFIG_MAC80211_LEDS) += led.o mac80211-$(CONFIG_MAC80211_DEBUGFS) += \
net/mac80211/airtime.c new file +597 -0
597 lines not on this page · raw message
net/mac80211/ieee80211_i.h +4 -0
@@ -2249,6 +2249,10 @@ const char *ieee80211_get_reason_code_string(u16 reason_code);extern const struct ethtool_ops ieee80211_ethtool_ops; +u32 ieee80211_calc_expected_tx_airtime (struct ieee80211_hw *hw, +struct ieee80211_vif *vif, +struct ieee80211_sta *pubsta, +int len);#ifdef CONFIG_MAC80211_NOINLINE #define debug_noinline noinline#else
Signature · 1 line
2.24.0.rc1.363.gb1bccd3e3d-goog
PATCH v8 2/2 mac80211: Implement Airtime-based Queue Limit (AQL)
In reply to Kan YanIn order for the Fq_CoDel algorithm integrated in mac80211 layer to operate effectively to control excessive queueing latency, the CoDel algorithm requires an accurate measure of how long packets stays in the queue, AKA sojourn time. The sojourn time measured at the mac80211 layer doesn't include queueing latency in the lower layer (firmware/hardware) and CoDel expects lower layer to have a short queue. However, most 802.11ac chipsets offload tasks such TX aggregation to firmware or hardware, thus have a deep lower layer queue.
Without a mechanism to control the lower layer queue size, packets only stay in mac80211 layer transiently before being sent to firmware queue. As a result, the sojourn time measured by CoDel in the mac80211 layer is almost always lower than the CoDel latency target, hence CoDel does little to control the latency, even when the lower layer queue causes excessive latency.
The Byte Queue Limits (BQL) mechanism is commonly used to address the similar issue with wired network interface. However, this method cannot be applied directly to the wireless network interface. "Bytes" is not a suitable measure of queue depth in the wireless network, as the data rate can vary dramatically from station to station in the same network, from a few Mbps to over Gbps.
This patch implements an Airtime-based Queue Limit (AQL) to make CoDel work effectively with wireless drivers that utilized firmware/hardware offloading. AQL allows each txq to release just enough packets to the lower layer to form 1-2 large aggregations to keep hardware fully utilized and retains the rest of the frames in mac80211 layer to be controlled by the CoDel algorithm.
--- include/net/cfg80211.h | 7 ++++ include/net/mac80211.h | 12 ++++++ net/mac80211/debugfs.c | 85 ++++++++++++++++++++++++++++++++++++++ net/mac80211/debugfs_sta.c | 43 ++++++++++++++----- net/mac80211/ieee80211_i.h | 4 ++ net/mac80211/main.c | 10 ++++- net/mac80211/sta_info.c | 38 +++++++++++++++++ net/mac80211/sta_info.h | 8 ++++ net/mac80211/tx.c | 47 +++++++++++++++++++-- 9 files changed, 240 insertions(+), 14 deletions(-)
include/net/cfg80211.h +7 -0
@@ -2602,6 +2602,13 @@ enum wiphy_params_flags {#define IEEE80211_DEFAULT_AIRTIME_WEIGHT 256 +/* The per TXQ device queue limit in airtime */ +#define IEEE80211_DEFAULT_AQL_TXQ_LIMIT_L 4000 +#define IEEE80211_DEFAULT_AQL_TXQ_LIMIT_H 8000 + +/* The per interface airtime threshold to switch to lower queue limit */ +#define IEEE80211_AQL_THRESHOLD 24000 +/** * struct cfg80211_pmksa - PMK Security Association *
include/net/mac80211.h +12 -0
@@ -5558,6 +5558,18 @@ void ieee80211_send_eosp_nullfunc(struct ieee80211_sta *pubsta, int tid);void ieee80211_sta_register_airtime (struct ieee80211_sta *pubsta,u8 tid,u32 tx_airtime,u32 rx_airtime); +/** +* ieee80211_txq_airtime_check - check if a txq can send frame to device +* +* @hw: pointer obtained from ieee80211_alloc_hw() +* @txq: pointer obtained from station or virtual interface +* +* Return true if the AQL's airtime limit has not been reached and the txq can +* continue to send more packets to the device. Otherwise return false. +*/ +bool +ieee80211_txq_airtime_check (struct ieee80211_hw *hw,struct ieee80211_txq *txq); +/** * ieee80211_iter_keys - iterate keys programmed into the device * @hw: pointer obtained from ieee80211_alloc_hw()
net/mac80211/debugfs.c +85 -0
97 lines not on this page · raw message
net/mac80211/debugfs_sta.c +33 -10
69 lines not on this page · raw message
net/mac80211/ieee80211_i.h +4 -0
@@ -1142,6 +1142,10 @@ struct ieee80211_local {u16 schedule_round[IEEE80211_NUM_ACS ];u16 airtime_flags; +u32 aql_txq_limit_low[IEEE80211_NUM_ACS ]; +u32 aql_txq_limit_high[IEEE80211_NUM_ACS ]; +u32 aql_threshold; +atomic_t aql_total_pending_airtime;const struct ieee80211_ops *ops;
net/mac80211/main.c +9 -1
@@ -667,8 +667,16 @@ struct ieee80211_hw *ieee80211_alloc_hw_nm(size_t priv_data_len,for (i =0 ; i <IEEE80211_NUM_ACS ; i++) {INIT_LIST_HEAD (&local->active_txqs[i]);spin_lock_init (&local->active_txq_lock[i]); + local->aql_txq_limit_low[i] =IEEE80211_DEFAULT_AQL_TXQ_LIMIT_L ; + local->aql_txq_limit_high[i] = +IEEE80211_DEFAULT_AQL_TXQ_LIMIT_H ; } - local->airtime_flags =AIRTIME_USE_TX |AIRTIME_USE_RX ; + + local->airtime_flags =AIRTIME_USE_TX | +AIRTIME_USE_RX | +AIRTIME_USE_AQL ; + local->aql_threshold =IEEE80211_AQL_THRESHOLD ; +atomic_set (&local->aql_total_pending_airtime,0 );INIT_LIST_HEAD (&local->chanctx_list);mutex_init (&local->chanctx_mtx);
net/mac80211/sta_info.c +38 -0
@@ -396,6 +396,9 @@ struct sta_info *sta_info_alloc(struct ieee80211_sub_if_data *sdata,skb_queue_head_init (&sta->ps_tx_buf[i]);skb_queue_head_init (&sta->tx_filtered[i]); sta->airtime[i].deficit = sta->airtime_weight; +atomic_set (&sta->airtime[i].aql_tx_pending,0 ); + sta->airtime[i].aql_limit_low = local->aql_txq_limit_low[i]; + sta->airtime[i].aql_limit_high = local->aql_txq_limit_high[i]; }for (i =0 ; i <IEEE80211_NUM_TIDS ; i++)
@@ -1893,6 +1896,41 @@ void ieee80211_sta_register_airtime(struct ieee80211_sta *pubsta, u8 tid, }EXPORT_SYMBOL (ieee80211_sta_register_airtime); +void ieee80211_sta_update_pending_airtime (struct ieee80211_local *local, +struct sta_info *sta,u8 ac, +u16 tx_airtime,bool tx_completed) +{ +int tx_pending; + +if (!tx_completed) { +if (sta) +atomic_add (tx_airtime, + &sta->airtime[ac].aql_tx_pending); + +atomic_add (tx_airtime, &local->aql_total_pending_airtime); +return ; + } + +if (sta) { +atomic_sub (tx_airtime, &sta->airtime[ac].aql_tx_pending); + tx_pending =atomic_read (&sta->airtime[ac].aql_tx_pending); +if (WARN_ONCE (tx_pending <0 , +"STA %pM AC %d txq pending airtime underflow: %u, %u" , + sta->addr, ac, tx_pending, tx_airtime)) +atomic_cmpxchg (&sta->airtime[ac].aql_tx_pending, + tx_pending,0 ); + } + +atomic_sub (tx_airtime, &local->aql_total_pending_airtime); + tx_pending =atomic_read (&local->aql_total_pending_airtime); +if (WARN_ONCE (tx_pending <0 , +"Device %s AC %d pending airtime underflow: %u, %u" , +wiphy_name (local->hw.wiphy), ac, tx_pending, + tx_airtime)) +atomic_cmpxchg (&local->aql_total_pending_airtime, + tx_pending,0 ); +} +int sta_info_move_state (struct sta_info *sta,enum ieee80211_sta_state new_state) {
net/mac80211/sta_info.h +8 -0
@@ -127,13 +127,21 @@ enum ieee80211_agg_stop_reason {/* Debugfs flags to enable/disable use of RX/TX airtime in scheduler */ #define AIRTIME_USE_TX BIT(0)#define AIRTIME_USE_RX BIT(1) +#define AIRTIME_USE_AQL BIT(2)struct airtime_info {u64 rx_airtime;u64 tx_airtime;s64 deficit; +atomic_t aql_tx_pending;/* Estimated airtime for frames pending */ +u32 aql_limit_low; +u32 aql_limit_high; }; +void ieee80211_sta_update_pending_airtime (struct ieee80211_local *local, +struct sta_info *sta,u8 ac, +u16 tx_airtime,bool tx_completed); +struct sta_info ;/**
net/mac80211/tx.c +44 -3
70 lines not on this page · raw message
Signature · 1 line
2.24.0.rc1.363.gb1bccd3e3d-goog
Re: [v8 PATCH 2/2] mac80211: Implement Airtime-based Queue Limit (AQL)
Quoted 34 lines · Kan Yan
In order for the
Fq_CoDelalgorithm integrated in mac80211 layer to operate effectively to control excessive queueing latency, the CoDel algorithm requires an accurate measure of how long packets stays in the queue, AKA sojourn time. The sojourn time measured at the mac80211 layer doesn't include queueing latency in the lower layer (firmware/hardware) and CoDel expects lower layer to have a short queue. However, most 802.11ac chipsets offload tasks such TX aggregation to firmware or hardware, thus have a deep lower layer queue.Without a mechanism to control the lower layer queue size, packets only stay in mac80211 layer transiently before being sent to firmware queue. As a result, the sojourn time measured by CoDel in the mac80211 layer is almost always lower than the CoDel latency target, hence CoDel does little to control the latency, even when the lower layer queue causes excessive latency.
The Byte Queue Limits (BQL) mechanism is commonly used to address the similar issue with wired network interface. However, this method cannot be applied directly to the wireless network interface. "Bytes" is not a suitable measure of queue depth in the wireless network, as the data rate can vary dramatically from station to station in the same network, from a few Mbps to over Gbps.
This patch implements an Airtime-based Queue Limit (AQL) to make CoDel work effectively with wireless drivers that utilized
firmware/hardwareoffloading. AQL allows each txq to release just enough packets to the lower layer to form 1-2 large aggregations to keep hardware fully utilized and retains the rest of the frames in mac80211 layer to be controlled by the CoDel algorithm.Signed-off-by Kan Yan <[email protected]>[ Toke: Keep API to set pending airtime internal, fix nits in commit msg ]Signed-off-by Toke Høiland-Jørgensen <[email protected]>---
[...]
net/mac80211/sta_info.c void ieee80211_sta_register_airtime(struct ieee80211_sta *pubsta, u8 tid, · line 1916
Quoted 3 lines
net/mac80211/sta_info.c +3
-0+if (sta) { +atomic_sub (tx_airtime, &sta->airtime[ac].aql_tx_pending); + tx_pending =atomic_read (&sta->airtime[ac].aql_tx_pending);
This is still racy, since you're splitting it over two calls; you'll need to use atomic_sub_return() instead.
I figure we've converged now to the point where it actually makes sense to collect everything back into a single series; so I can just fix this and re-send the full series.
net/mac80211/sta_info.c void ieee80211_sta_register_airtime(struct ieee80211_sta *pubsta, u8 tid, · line 1921
Quoted 5 lines
net/mac80211/sta_info.c +5
-0+if (WARN_ONCE (tx_pending <0 , +"STA %pM AC %d txq pending airtime underflow: %u, %u" , + sta->addr, ac, tx_pending, tx_airtime)) +atomic_cmpxchg (&sta->airtime[ac].aql_tx_pending, + tx_pending,0 );
This could still fail if there's a concurrent modification (and you're not checking the return of the cmpxchg). But at least it won't clobber any updated value, so I guess that is acceptable since we're in "should never happen" territory here :)
-Toke
Re: [v8 PATCH 2/2] mac80211: Implement Airtime-based Queue Limit (AQL)
Quoted 8 lines
net/mac80211/sta_info.c void ieee80211_sta_register_airtime(struct ieee80211_sta *pubsta, u8 tid, · line 1916
Quoted 3 lines
net/mac80211/sta_info.c +3
-0+if (sta) { +atomic_sub (tx_airtime, &sta->airtime[ac].aql_tx_pending); + tx_pending =atomic_read (&sta->airtime[ac].aql_tx_pending);This is still racy, since you're splitting it over two calls; you'll need to use
atomic_sub_return()instead.
I figure we've converged now to the point where it actually makes sense to collect everything back into a single series; so I can just fix this and re-send the full series.
Thanks for help fixing this. Yes, atomic_sub_return() is better.
Quoted 11 lines
net/mac80211/sta_info.c void ieee80211_sta_register_airtime(struct ieee80211_sta *pubsta, u8 tid, · line 1921
Quoted 7 lines
net/mac80211/sta_info.c +5
-0+if (WARN_ONCE (tx_pending <0 , +"STA %pM AC %d txq pending airtime underflow: %u, %u" , + sta->addr, ac, tx_pending, tx_airtime)) +atomic_cmpxchg (&sta->airtime[ac].aql_tx_pending, + tx_pending,0 );This could still fail if there's a concurrent modification (and you're not checking the return of the cmpxchg). But at least it won't clobber any updated value, so I guess that is acceptable since we're in "should never happen" territory here :)
I did this on purpose since I really don't like adding a loop to retry here. If aql_tx_pending indeed goes negative (should never happens and we got WARN_ONCE() to catch it) and the subsequent atomic_cmpxchg() failed (rare racing occasions), it is still ok. In this case, aql_tx_pending carries a negative offset and will be reset in one of the calls to ieee80211_sta_update_pending_airtime() later. aql_tx_pending being negative doesn't have much side-effects, such as causing txq stuck.
Quoted 63 lines · Toke Høiland-Jørgensen
Quoted 34 lines · Kan Yan
In order for the
Fq_CoDelalgorithm integrated in mac80211 layer to operate effectively to control excessive queueing latency, the CoDel algorithm requires an accurate measure of how long packets stays in the queue, AKA sojourn time. The sojourn time measured at the mac80211 layer doesn't include queueing latency in the lower layer (firmware/hardware) and CoDel expects lower layer to have a short queue. However, most 802.11ac chipsets offload tasks such TX aggregation to firmware or hardware, thus have a deep lower layer queue.Without a mechanism to control the lower layer queue size, packets only stay in mac80211 layer transiently before being sent to firmware queue. As a result, the sojourn time measured by CoDel in the mac80211 layer is almost always lower than the CoDel latency target, hence CoDel does little to control the latency, even when the lower layer queue causes excessive latency.
The Byte Queue Limits (BQL) mechanism is commonly used to address the similar issue with wired network interface. However, this method cannot be applied directly to the wireless network interface. "Bytes" is not a suitable measure of queue depth in the wireless network, as the data rate can vary dramatically from station to station in the same network, from a few Mbps to over Gbps.
This patch implements an Airtime-based Queue Limit (AQL) to make CoDel work effectively with wireless drivers that utilized
firmware/hardwareoffloading. AQL allows each txq to release just enough packets to the lower layer to form 1-2 large aggregations to keep hardware fully utilized and retains the rest of the frames in mac80211 layer to be controlled by the CoDel algorithm.Signed-off-by Kan Yan <[email protected]>[ Toke: Keep API to set pending airtime internal, fix nits in commit msg ]Signed-off-by Toke Høiland-Jørgensen <[email protected]>---
[...]
net/mac80211/sta_info.c void ieee80211_sta_register_airtime(struct ieee80211_sta *pubsta, u8 tid, · line 1916
Quoted 3 lines
net/mac80211/sta_info.c +3
-0+if (sta) { +atomic_sub (tx_airtime, &sta->airtime[ac].aql_tx_pending); + tx_pending =atomic_read (&sta->airtime[ac].aql_tx_pending);This is still racy, since you're splitting it over two calls; you'll need to use
atomic_sub_return()instead.I figure we've converged now to the point where it actually makes sense to collect everything back into a single series; so I can just fix this and re-send the full series.
net/mac80211/sta_info.c void ieee80211_sta_register_airtime(struct ieee80211_sta *pubsta, u8 tid, · line 1921
Quoted 5 lines
net/mac80211/sta_info.c +5
-0+if (WARN_ONCE (tx_pending <0 , +"STA %pM AC %d txq pending airtime underflow: %u, %u" , + sta->addr, ac, tx_pending, tx_airtime)) +atomic_cmpxchg (&sta->airtime[ac].aql_tx_pending, + tx_pending,0 );This could still fail if there's a concurrent modification (and you're not checking the return of the cmpxchg). But at least it won't clobber any updated value, so I guess that is acceptable since we're in "should never happen" territory here :)
-Toke
I have tested it with Toke's patch "[PATCH v6 4/4] mac80211: Use Airtime-based Queue Limits (AQL) on packet dequeue", but didn't include it here, as it is self contained and Toke has plan to update it.
The platform (QCA9984) used in my test doesn't support 802.11ax, so I was not able to test the HE mode support added in v7 update of "Import airtime calculation code from mt76" from Toke.
Quoted 68 lines · Kan Yan
This patch series port the Airtime Queue Limits concept from the out-of-tree ath10k implementation[0] to mac80211. This version takes my patch to do the throttling in mac80211, and replaces the driver API with the mechanism from Toke's series, which instead calculated the expected airtime at dequeue time inside mac80211, storing it in the SKB cb field.
This version has been tested on QCA9984 platform.
[0] https://chromium-review.googlesource.com/c/chromiumos/third_party/kernel/+/1703105/7
Changelog:
v8:
- Includes Toke's v7 version of "mac80211: Import airtime calculation code from mt76"
- Don't clobber sta's customized queue limit when configuring the default via debugfs
- Fix a racing condition when resetaql_tx_pending.v7:
- Fixaql_total_pending_airtimeunderflow due to insufficient locking.v6:
- Fix sta lookup inieee80211_report_used_skb().
- Move call toieee80211_sta_update_pending_airtime()to a bit later in__ieee80211_tx_status() v5: - Add missing export of ieee80211_calc_rx_airtime() and make ieee80211_calc_tx_airtime_rate() static (kbuildbot).- Use
skb_get_queue_mapping()to get the AC from the skb.
- Take basic rate configuration for the BSS into account when calculating multicast rate.
v4:
- Fix calculation that clamps the maximum airtime to fit into 10 bits
- Incorporate Rich Brown's nits for the commit message in Kan's patch
- Add fewer local variables toieee80211_tx_dequeue()
v3:
- Move thetx_time_estfield so it's shared withack_frame_id, and use units of 4us for the value stored in it.
- Move the addition of the Ethernet header size intoieee80211_calc_expected_tx_airtime()
v2:
- Integrate Kan's approach to airtime throttling.
- Hopefully fix the cb struct alignment on big-endian architectures.Kan Yan (1):
mac80211: Implement Airtime-based Queue Limit (AQL)Toke Høiland-Jørgensen (1):
mac80211: Import airtime calculation code from mt76include/net/cfg80211.h | 7 + include/net/mac80211.h | 41 +++ net/mac80211/Makefile | 3 +- net/mac80211/airtime.c | 597 +++++++++++++++++++++++++++++++++++++ net/mac80211/debugfs.c | 85 ++++++ net/mac80211/debugfs_sta.c | 43 ++- net/mac80211/ieee80211_i.h | 8 + net/mac80211/main.c | 10 +- net/mac80211/sta_info.c | 38 +++ net/mac80211/sta_info.h | 8 + net/mac80211/tx.c | 47 ++- 11 files changed, 872 insertions(+), 15 deletions(-) create mode 100644 net/mac80211/airtime.c--
2.24.0.rc1.363.gb1bccd3e3d-goog
Quoted 7 lines · Kan Yan
I have tested it with Toke's patch "[PATCH v6 4/4] mac80211: Use Airtime-based Queue Limits (AQL) on packet dequeue", but didn't include it here, as it is self contained and Toke has plan to update it.
The platform (QCA9984) used in my test
I do keep hoping for pretty pictures. Got any? :-P
Quoted 3 lines
doesn't support 802.11ax, so I
was not able to test the HE mode support added in v7 update of "Import airtime calculation code from mt76" from Toke.
Is there an ax QCAXXXX platform, m.2 card, or mini-pci card worth testing at this point?
How are they handling mu-mimo?
I have a round of tests scheduled for intel's ax200 chips, soon. Not sure what, if any, of this new work might apply.
Quoted 73 lines · Kan Yan
Quoted 65 lines
This patch series port the Airtime Queue Limits concept from the out-of-tree ath10k implementation[0] to mac80211. This version takes my patch to do the throttling in mac80211, and replaces the driver API with the mechanism from Toke's series, which instead calculated the expected airtime at dequeue time inside mac80211, storing it in the SKB cb field.
This version has been tested on QCA9984 platform.
[0] https://chromium-review.googlesource.com/c/chromiumos/third_party/kernel/+/1703105/7
Changelog:
v8:
- Includes Toke's v7 version of "mac80211: Import airtime calculation code from mt76"
- Don't clobber sta's customized queue limit when configuring the default via debugfs
- Fix a racing condition when resetaql_tx_pending.v7:
- Fixaql_total_pending_airtimeunderflow due to insufficient locking.v6:
- Fix sta lookup inieee80211_report_used_skb().
- Move call toieee80211_sta_update_pending_airtime()to a bit later in__ieee80211_tx_status() v5: - Add missing export of ieee80211_calc_rx_airtime() and make ieee80211_calc_tx_airtime_rate() static (kbuildbot).- Use
skb_get_queue_mapping()to get the AC from the skb.
- Take basic rate configuration for the BSS into account when calculating multicast rate.
v4:
- Fix calculation that clamps the maximum airtime to fit into 10 bits
- Incorporate Rich Brown's nits for the commit message in Kan's patch
- Add fewer local variables toieee80211_tx_dequeue()
v3:
- Move thetx_time_estfield so it's shared withack_frame_id, and use units of 4us for the value stored in it.
- Move the addition of the Ethernet header size intoieee80211_calc_expected_tx_airtime()
v2:
- Integrate Kan's approach to airtime throttling.
- Hopefully fix the cb struct alignment on big-endian architectures.Kan Yan (1):
mac80211: Implement Airtime-based Queue Limit (AQL)Toke Høiland-Jørgensen (1):
mac80211: Import airtime calculation code from mt76include/net/cfg80211.h | 7 + include/net/mac80211.h | 41 +++ net/mac80211/Makefile | 3 +- net/mac80211/airtime.c | 597 +++++++++++++++++++++++++++++++++++++ net/mac80211/debugfs.c | 85 ++++++ net/mac80211/debugfs_sta.c | 43 ++- net/mac80211/ieee80211_i.h | 8 + net/mac80211/main.c | 10 +- net/mac80211/sta_info.c | 38 +++ net/mac80211/sta_info.h | 8 + net/mac80211/tx.c | 47 ++- 11 files changed, 872 insertions(+), 15 deletions(-) create mode 100644 net/mac80211/airtime.cQuoted 3 lines
--
2.24.0.rc1.363.gb1bccd3e3d-goog_______________________________________________
Make-wifi-fast mailing list
[email protected]
https://lists.bufferbloat.net/listinfo/make-wifi-fast
Signature · 4 lines
Dave Täht CTO, TekLibre, LLC http://www.teklibre.com Tel: 1-831-205-9740
A message that is not in the archive (sent off-list or not yet received): CA+iem5s4ZY239Q4=Gwy3WrmVhcdhesirXph6XQoOP5w-nuWcYw@mail.gmail.com
Quoted 5 lines · Kan Yan
Quoted 2 lines
I do keep hoping for pretty pictures. Got any? :-P
Certainly! I do have some :). Here is the link:
https://drive.google.com/corp/drive/folders/14OIuQEHOUiIoNrVnKprj6rBYFNZ0Coif
Those were lovely, thanks!!!! Big win. Since you are on patch v10 now.... Any chance you could turn ecn on and off and give it a go again in your next test run?
Also:
--step-size=.04 --socket-stats # the first is helpful to gain more detail, the second as to the behavior of the tcp stack. You might need to run as root (and It's only useful on the tcp_nup test) for the latter (and have the right ss utility)
Secondly - and AFTER this patchset stablizes, I'd like us to look into returning the codel default to 10ms or less
from it's currently 20ms or worse setting. Tis another easy test
And y'all know how much I love the rrul_be and rrul tests.....
Quoted 123 lines
Quoted 3 lines
Is there an ax QCAXXXX platform, m.2 card, or mini-pci card worth testing at this point?
It will be great if someone with 11.ax platform can help give it a try.
Quoted 1 line
How are they handling mu-mimo?
I think it should still work. The queue length in airtime for each individual queue is unchanged, even the multiple queues are allowed to transmit concurrently with mu-mimo.
Quoted 2 lines
I have a round of tests scheduled for intel's ax200 chips, soon. Not sure what, if any, of this new work might apply.
It will be very interesting to know how it performance on 802.11ax platforms. Supposedly 802.11ax already fixed the latency problem so the benefit of this patch should be less significant.
Quoted 106 lines · Dave Taht
Quoted 7 lines · Kan Yan
I have tested it with Toke's patch "[PATCH v6 4/4] mac80211: Use Airtime-based Queue Limits (AQL) on packet dequeue", but didn't include it here, as it is self contained and Toke has plan to update it.
The platform (QCA9984) used in my test
I do keep hoping for pretty pictures. Got any? :-P
Quoted 3 lines
doesn't support 802.11ax, so I
was not able to test the HE mode support added in v7 update of "Import airtime calculation code from mt76" from Toke.Is there an ax QCAXXXX platform, m.2 card, or mini-pci card worth testing at this point?
How are they handling mu-mimo?
I have a round of tests scheduled for intel's ax200 chips, soon. Not sure what, if any, of this new work might apply.
Quoted 66 lines · Kan Yan
Quoted 65 lines
This patch series port the Airtime Queue Limits concept from the out-of-tree ath10k implementation[0] to mac80211. This version takes my patch to do the throttling in mac80211, and replaces the driver API with the mechanism from Toke's series, which instead calculated the expected airtime at dequeue time inside mac80211, storing it in the SKB cb field.
This version has been tested on QCA9984 platform.
[0] https://chromium-review.googlesource.com/c/chromiumos/third_party/kernel/+/1703105/7
Changelog:
v8:
- Includes Toke's v7 version of "mac80211: Import airtime calculation code from mt76"
- Don't clobber sta's customized queue limit when configuring the default via debugfs
- Fix a racing condition when resetaql_tx_pending.v7:
- Fixaql_total_pending_airtimeunderflow due to insufficient locking.v6:
- Fix sta lookup inieee80211_report_used_skb().
- Move call toieee80211_sta_update_pending_airtime()to a bit later in__ieee80211_tx_status() v5: - Add missing export of ieee80211_calc_rx_airtime() and make ieee80211_calc_tx_airtime_rate() static (kbuildbot).- Use
skb_get_queue_mapping()to get the AC from the skb.
- Take basic rate configuration for the BSS into account when calculating multicast rate.
v4:
- Fix calculation that clamps the maximum airtime to fit into 10 bits
- Incorporate Rich Brown's nits for the commit message in Kan's patch
- Add fewer local variables toieee80211_tx_dequeue()
v3:
- Move thetx_time_estfield so it's shared withack_frame_id, and use units of 4us for the value stored in it.
- Move the addition of the Ethernet header size intoieee80211_calc_expected_tx_airtime()
v2:
- Integrate Kan's approach to airtime throttling.
- Hopefully fix the cb struct alignment on big-endian architectures.Kan Yan (1):
mac80211: Implement Airtime-based Queue Limit (AQL)Toke Høiland-Jørgensen (1):
mac80211: Import airtime calculation code from mt76include/net/cfg80211.h | 7 + include/net/mac80211.h | 41 +++ net/mac80211/Makefile | 3 +- net/mac80211/airtime.c | 597 +++++++++++++++++++++++++++++++++++++ net/mac80211/debugfs.c | 85 ++++++ net/mac80211/debugfs_sta.c | 43 ++- net/mac80211/ieee80211_i.h | 8 + net/mac80211/main.c | 10 +- net/mac80211/sta_info.c | 38 +++ net/mac80211/sta_info.h | 8 + net/mac80211/tx.c | 47 ++- 11 files changed, 872 insertions(+), 15 deletions(-) create mode 100644 net/mac80211/airtime.cQuoted 7 lines
Quoted 3 lines
--
2.24.0.rc1.363.gb1bccd3e3d-goog_______________________________________________
Make-wifi-fast mailing list
[email protected]
https://lists.bufferbloat.net/listinfo/make-wifi-fast--
Dave Täht
CTO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-831-205-9740
Signature · 4 lines
Dave Täht CTO, TekLibre, LLC http://www.teklibre.com Tel: 1-831-205-9740
Quoted 9 lines
Those were lovely, thanks!!!! Big win. Since you are on patch v10 now.... Any chance you could turn ecn on and off and give it a go again in your next test run?
Also:
--step-size=.04 --socket-stats # the first is helpful to gain more detail, the second as to the behavior of the tcp stack.
Thanks for the feedback! I will do more tests in a few days.
Quoted 3 lines
Secondly - and AFTER this patchset stablizes, I'd like us to look into returning the codel default to 10ms or less
from it's currently 20ms or worse setting. Tis another easy test
Smaller CoDel "target" doesn't work well with wireless because the dequeue behavior in wireless driver is very bursty. It is quite often dequeues dozens of packets in one burst after one large aggregation is completed, so smaller CoDel "target" can cause unnecessary packet drop.
Quoted 158 lines · Dave Taht
Quoted 5 lines · Kan Yan
Quoted 2 lines
I do keep hoping for pretty pictures. Got any? :-P
Certainly! I do have some :). Here is the link:
https://drive.google.com/corp/drive/folders/14OIuQEHOUiIoNrVnKprj6rBYFNZ0CoifThose were lovely, thanks!!!! Big win. Since you are on patch v10 now.... Any chance you could turn ecn on and off and give it a go again in your next test run?
Also:
--step-size=.04 --socket-stats # the first is helpful to gain more detail, the second as to the behavior of the tcp stack. You might need to run as root (and It's only useful on the
tcp_nuptest) for the latter (and have the right ss utility)Secondly - and AFTER this patchset stablizes, I'd like us to look into returning the codel default to 10ms or less
from it's currently 20ms or worse setting. Tis another easy testAnd y'all know how much I love the
rrul_beand rrul tests.....Quoted 10 lines
Quoted 3 lines
Is there an ax QCAXXXX platform, m.2 card, or mini-pci card worth testing at this point?
It will be great if someone with 11.ax platform can help give it a try.
Quoted 1 line
How are they handling mu-mimo?
I think it should still work. The queue length in airtime for each individual queue is unchanged, even the multiple queues are allowed to transmit concurrently with mu-mimo.
Quoted 113 lines
Quoted 2 lines
I have a round of tests scheduled for intel's ax200 chips, soon. Not sure what, if any, of this new work might apply.
It will be very interesting to know how it performance on 802.11ax platforms. Supposedly 802.11ax already fixed the latency problem so the benefit of this patch should be less significant.
Quoted 106 lines · Dave Taht
Quoted 7 lines · Kan Yan
I have tested it with Toke's patch "[PATCH v6 4/4] mac80211: Use Airtime-based Queue Limits (AQL) on packet dequeue", but didn't include it here, as it is self contained and Toke has plan to update it.
The platform (QCA9984) used in my test
I do keep hoping for pretty pictures. Got any? :-P
Quoted 3 lines
doesn't support 802.11ax, so I
was not able to test the HE mode support added in v7 update of "Import airtime calculation code from mt76" from Toke.Is there an ax QCAXXXX platform, m.2 card, or mini-pci card worth testing at this point?
How are they handling mu-mimo?
I have a round of tests scheduled for intel's ax200 chips, soon. Not sure what, if any, of this new work might apply.
Quoted 66 lines · Kan Yan
Quoted 65 lines
This patch series port the Airtime Queue Limits concept from the out-of-tree ath10k implementation[0] to mac80211. This version takes my patch to do the throttling in mac80211, and replaces the driver API with the mechanism from Toke's series, which instead calculated the expected airtime at dequeue time inside mac80211, storing it in the SKB cb field.
This version has been tested on QCA9984 platform.
[0] https://chromium-review.googlesource.com/c/chromiumos/third_party/kernel/+/1703105/7
Changelog:
v8:
- Includes Toke's v7 version of "mac80211: Import airtime calculation code from mt76"
- Don't clobber sta's customized queue limit when configuring the default via debugfs
- Fix a racing condition when resetaql_tx_pending.v7:
- Fixaql_total_pending_airtimeunderflow due to insufficient locking.v6:
- Fix sta lookup inieee80211_report_used_skb().
- Move call toieee80211_sta_update_pending_airtime()to a bit later in__ieee80211_tx_status() v5: - Add missing export of ieee80211_calc_rx_airtime() and make ieee80211_calc_tx_airtime_rate() static (kbuildbot).- Use
skb_get_queue_mapping()to get the AC from the skb.
- Take basic rate configuration for the BSS into account when calculating multicast rate.
v4:
- Fix calculation that clamps the maximum airtime to fit into 10 bits
- Incorporate Rich Brown's nits for the commit message in Kan's patch
- Add fewer local variables toieee80211_tx_dequeue()
v3:
- Move thetx_time_estfield so it's shared withack_frame_id, and use units of 4us for the value stored in it.
- Move the addition of the Ethernet header size intoieee80211_calc_expected_tx_airtime()
v2:
- Integrate Kan's approach to airtime throttling.
- Hopefully fix the cb struct alignment on big-endian architectures.Kan Yan (1):
mac80211: Implement Airtime-based Queue Limit (AQL)Toke Høiland-Jørgensen (1):
mac80211: Import airtime calculation code from mt76include/net/cfg80211.h | 7 + include/net/mac80211.h | 41 +++ net/mac80211/Makefile | 3 +- net/mac80211/airtime.c | 597 +++++++++++++++++++++++++++++++++++++ net/mac80211/debugfs.c | 85 ++++++ net/mac80211/debugfs_sta.c | 43 ++- net/mac80211/ieee80211_i.h | 8 + net/mac80211/main.c | 10 +- net/mac80211/sta_info.c | 38 +++ net/mac80211/sta_info.h | 8 + net/mac80211/tx.c | 47 ++- 11 files changed, 872 insertions(+), 15 deletions(-) create mode 100644 net/mac80211/airtime.cQuoted 7 lines
Quoted 3 lines
--
2.24.0.rc1.363.gb1bccd3e3d-goog_______________________________________________
Make-wifi-fast mailing list
[email protected]
https://lists.bufferbloat.net/listinfo/make-wifi-fast--
Dave Täht
CTO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-831-205-9740--
Dave Täht
CTO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-831-205-9740
Quoted 17 lines · Kan Yan
Quoted 9 lines
Those were lovely, thanks!!!! Big win. Since you are on patch v10 now.... Any chance you could turn ecn on and off and give it a go again in your next test run?
Also:
--step-size=.04 --socket-stats # the first is helpful to gain more detail, the second as to the behavior of the tcp stack.
Thanks for the feedback! I will do more tests in a few days.
Quoted 3 lines
Secondly - and AFTER this patchset stablizes, I'd like us to look into returning the codel default to 10ms or less
from it's currently 20ms or worse setting. Tis another easy test
Quoted 5 lines
Smaller CoDel "target" doesn't work well with wireless because the dequeue behavior in wireless driver is very bursty. It is quite often dequeues dozens of packets in one burst after one large aggregation is completed, so smaller CoDel "target" can cause unnecessary packet drop.
It would be interesting to get some samples of the actual sojourn time as seen by CoDel in mac80211. Might be doable with bpftrace...
-Toke
Quoted 2 lines
It would be interesting to get some samples of the actual sojourn time as seen by CoDel in mac80211. Might be doable with bpftrace...
I will try to add some trace event to get the sojourn time for the next round of tests.
Quoted 31 lines · Toke Høiland-Jørgensen
Quoted 17 lines · Kan Yan
Quoted 9 lines
Those were lovely, thanks!!!! Big win. Since you are on patch v10 now.... Any chance you could turn ecn on and off and give it a go again in your next test run?
Also:
--step-size=.04 --socket-stats # the first is helpful to gain more detail, the second as to the behavior of the tcp stack.
Thanks for the feedback! I will do more tests in a few days.
Quoted 3 lines
Secondly - and AFTER this patchset stablizes, I'd like us to look into returning the codel default to 10ms or less
from it's currently 20ms or worse setting. Tis another easy testQuoted 5 lines
Smaller CoDel "target" doesn't work well with wireless because the dequeue behavior in wireless driver is very bursty. It is quite often dequeues dozens of packets in one burst after one large aggregation is completed, so smaller CoDel "target" can cause unnecessary packet drop.
It would be interesting to get some samples of the actual sojourn time as seen by CoDel in mac80211. Might be doable with bpftrace...
-Toke
Quoted 5 lines · Kan Yan
Quoted 2 lines
It would be interesting to get some samples of the actual sojourn time as seen by CoDel in mac80211. Might be doable with bpftrace...
I will try to add some trace event to get the sojourn time for the next round of tests.
In theory, this ought to produce a histogram of sojourn times (in microseconds):
bpftrace -e 'kretprobe:codel_skb_time_func { @sojourn = lhist((nsecs - (retval << 10))/1000, 0, 100000, 1000); }'
Can't get the CoDel drop mechanism to trigger on my system at all, though (a laptop running on iwl). I guess because there's queue backpressure to userspace first?
It would be interesting to see if it works for you, assuming you can get bpftrace to work on your test system :)
-Toke
A message that is not in the archive (sent off-list or not yet received): CA+iem5tNz2jjEOVmbh3aPTXLLZfkRjZ60-+bon1vDEJ8D4hQJw@mail.gmail.com
Quoted 5 lines · Kan Yan
Quoted 3 lines
In theory, this ought to produce a histogram of sojourn times (in microseconds):
bpftrace -e 'kretprobe:codel_skb_time_func { @sojourn = lhist((nsecs -(retval << 10))/1000, 0, 100000, 1000); }'
Quoted 8 lines
Thanks for the tips!
Quoted 3 lines
Can't get the CoDel drop mechanism to trigger on my system at all, though (a laptop running on iwl). I guess because there's queue backpressure to userspace first?
What's the
tcp_congestion_controlin your system? Maybe it is BBR that prevents bufferbloat.
It's not BBR, just plain old CUBIC. I've seen the issue before that it's almost impossible to build a queue in the mac80211 layer when the TCP session is originated on the local machine, though...
Quoted 5 lines
Quoted 2 lines
It would be interesting to see if it works for you, assuming you can get bpftrace to work on your test system :)
I can enable required kernel configuration easily, but cross-compile bpftrace for an ARM64 platform may take some time and effort.
Yeah, bpftrace can be a bit of a pain to get running; but it may be worth the investment longer term as well. It really is quite useful! :)
Some links:
Install guide:
https://github.com/iovisor/bpftrace/blob/master/INSTALL.md
Tutorial by one-liners:
https://github.com/iovisor/bpftrace/blob/master/docs/tutorial_one_liners.md
Reference guide:
https://github.com/iovisor/bpftrace/blob/master/docs/reference_guide.md#5-tracepoint-static-tracing-kernel-level
-Toke
Quoted 2 lines
Yeah, bpftrace can be a bit of a pain to get running; but it may be worth the investment longer term as well. It really is quite useful! :)
My attempt to build bpftrace didn't work out, so I just got the sojourn time using old fashioned trace event.
The raw trace, parsed data in csv format and plots can be found here:
https://drive.google.com/open?id=1Mg_wHu7elYAdkXz4u--42qGCVE1nrILV
All tests are done with 2 TCP download sessions that oversubscribed the link bandwidth.
With AQL on, the mean sojourn time about ~20000us, matches the default codel "target".
With AQL off, the mean sojourn time is less than 4us even the latency is off the charts, just as we expected that fd_codel with mac80211 alone is not effective for drivers with deep firmware/hardware queues.
Quoted 2 lines
Any chance you could turn ecn on and off and give it a go again in your next test run?
ECN on shows very similar results as with ECN off. "aqm" stats from debugfs shows it is doing ECN marking instead of dropping packets as expected. Flent test data also is in the same link.
Quoted 43 lines · Toke Høiland-Jørgensen
Quoted 5 lines · Kan Yan
Quoted 3 lines
In theory, this ought to produce a histogram of sojourn times (in microseconds):
bpftrace -e 'kretprobe:codel_skb_time_func { @sojourn = lhist((nsecs -(retval << 10))/1000, 0, 100000, 1000); }'
Quoted 8 lines
Thanks for the tips!
Quoted 3 lines
Can't get the CoDel drop mechanism to trigger on my system at all, though (a laptop running on iwl). I guess because there's queue backpressure to userspace first?
What's the
tcp_congestion_controlin your system? Maybe it is BBR that prevents bufferbloat.It's not BBR, just plain old CUBIC. I've seen the issue before that it's almost impossible to build a queue in the mac80211 layer when the TCP session is originated on the local machine, though...
Quoted 5 lines
Quoted 2 lines
It would be interesting to see if it works for you, assuming you can get bpftrace to work on your test system :)
I can enable required kernel configuration easily, but cross-compile bpftrace for an ARM64 platform may take some time and effort.
Yeah, bpftrace can be a bit of a pain to get running; but it may be worth the investment longer term as well. It really is quite useful! :)
Some links:
Install guide:
https://github.com/iovisor/bpftrace/blob/master/INSTALL.mdTutorial by one-liners:
https://github.com/iovisor/bpftrace/blob/master/docs/tutorial_one_liners.mdReference guide:
https://github.com/iovisor/bpftrace/blob/master/docs/reference_guide.md#5-tracepoint-static-tracing-kernel-level-Toke
Quoted 8 lines · Kan Yan
Quoted 2 lines
Yeah, bpftrace can be a bit of a pain to get running; but it may be worth the investment longer term as well. It really is quite useful! :)
My attempt to build bpftrace didn't work out, so I just got the sojourn time using old fashioned trace event.
The raw trace, parsed data in csv format and plots can be found here:
https://drive.google.com/open?id=1Mg_wHu7elYAdkXz4u--42qGCVE1nrILV
Quoted 4 lines
All tests are done with 2 TCP download sessions that oversubscribed the link bandwidth.
With AQL on, the mean sojourn time about ~20000us, matches the default codel "target".
Yeah, since CoDel is trying to control the latency to 20ms, it makes sense that the value is clustered around that. That means that the algorithm is working as they're supposed to :)
While you're running tests, could you do one with the target changed to 10ms, just to see what it looks like? Both sojourn time values and throughput would be interesting here, of course.
Quoted 3 lines
With AQL off, the mean sojourn time is less than 4us even the latency is off the charts, just as we expected that
fd_codelwith mac80211 alone is not effective for drivers with deepfirmware/hardwarequeues.
Yup, also kinda expected; but another good way to visualise the impact. Nice!
-Toke
Quoted 15 lines · Toke Høiland-Jørgensen
Quoted 8 lines · Kan Yan
Quoted 2 lines
Yeah, bpftrace can be a bit of a pain to get running; but it may be worth the investment longer term as well. It really is quite useful! :)
My attempt to build bpftrace didn't work out, so I just got the sojourn time using old fashioned trace event.
The raw trace, parsed data in csv format and plots can be found here:
https://drive.google.com/open?id=1Mg_wHu7elYAdkXz4u--42qGCVE1nrILVQuoted 4 lines
All tests are done with 2 TCP download sessions that oversubscribed the link bandwidth.
With AQL on, the mean sojourn time about ~20000us, matches the default codel "target".
Quoted 11 lines
Yeah, since CoDel is trying to control the latency to 20ms, it makes sense that the value is clustered around that. That means that the algorithm is working as they're supposed to :)
While you're running tests, could you do one with the target changed to 10ms, just to see what it looks like? Both sojourn time values and throughput would be interesting here, of course.
Quoted 3 lines
With AQL off, the mean sojourn time is less than 4us even the latency is off the charts, just as we expected that
fd_codelwith mac80211 alone is not effective for drivers with deepfirmware/hardwarequeues
I hope to take a close look at the iwl ax200 chips soon. Unless someone beats me to it. Can we get these sort of stats out of it?
Has anyone looked at the marvell chips of late?
Quoted 10 lines
Yup, also kinda expected; but another good way to visualise the impact. Nice!
-Toke
_______________________________________________
Make-wifi-fast mailing list
[email protected]
https://lists.bufferbloat.net/listinfo/make-wifi-fast
Quoted 2 lines · Dave Taht
I hope to take a close look at the iwl ax200 chips soon. Unless someone beats me to it. Can we get these sort of stats out of it?
Here is a patch for the trace event I used to get the sojourn time:
https://drive.google.com/open?id=1Mq8BO_kcneXBqf3m5Rz5xhEMj9jNbcJv
Quoted 3 lines · Toke Høiland-Jørgensen
While you're running tests, could you do one with the target changed to 10ms, just to see what it looks like? Both sojourn time values and throughput would be interesting here, of course.
Apologize for the late reply. Here is the test results with target set to 10ms. The trace for the sojourn time:
https://drive.google.com/open?id=1MEy_wbKKdl22yF17hZaGzpv3uOz6orTi
Flent test for 20 ms target time vs 10 ms target time:
https://drive.google.com/open?id=1leIWe0-L0XE78eFvlmRJlNmYgbpoH8xZ
The sojourn time measured during throughput test with a relative good 5G connection has mean value around 11 ms, pretty close to the 10 ms target.
A smaller CoDel "target" time could help reduce latency, but it may drop packets too aggressively for stations with low data rate and hurts throughput, as shown in one of the tests with 2.4 GHz client.
Overall, I think AQL and fq_codel works well, at least with ath10k. The current target value of 20 ms is a reasonable default. It is relatively conservative that helps stations with weak signal to maintain stable throughput. Although, a debugfs entry that allows runtime adjustment of target value could be useful.
Quoted 45 lines · Dave Taht
Quoted 15 lines · Toke Høiland-Jørgensen
Quoted 8 lines · Kan Yan
Quoted 2 lines
Yeah, bpftrace can be a bit of a pain to get running; but it may be worth the investment longer term as well. It really is quite useful! :)
My attempt to build bpftrace didn't work out, so I just got the sojourn time using old fashioned trace event.
The raw trace, parsed data in csv format and plots can be found here:
https://drive.google.com/open?id=1Mg_wHu7elYAdkXz4u--42qGCVE1nrILVQuoted 4 lines
All tests are done with 2 TCP download sessions that oversubscribed the link bandwidth.
With AQL on, the mean sojourn time about ~20000us, matches the default codel "target".Quoted 11 lines
Yeah, since CoDel is trying to control the latency to 20ms, it makes sense that the value is clustered around that. That means that the algorithm is working as they're supposed to :)
While you're running tests, could you do one with the target changed to 10ms, just to see what it looks like? Both sojourn time values and throughput would be interesting here, of course.
Quoted 3 lines
With AQL off, the mean sojourn time is less than 4us even the latency is off the charts, just as we expected that
fd_codelwith mac80211 alone is not effective for drivers with deepfirmware/hardwarequeuesI hope to take a close look at the iwl ax200 chips soon. Unless someone beats me to it. Can we get these sort of stats out of it?
Has anyone looked at the marvell chips of late?
Quoted 10 lines
Yup, also kinda expected; but another good way to visualise the impact. Nice!
-Toke
_______________________________________________
Make-wifi-fast mailing list
[email protected]
https://lists.bufferbloat.net/listinfo/make-wifi-fast
Quoted 29 lines · Kan Yan
Quoted 2 lines · Dave Taht
I hope to take a close look at the iwl ax200 chips soon. Unless someone beats me to it. Can we get these sort of stats out of it?
Here is a patch for the trace event I used to get the sojourn time:
https://drive.google.com/open?id=1Mq8BO_kcneXBqf3m5Rz5xhEMj9jNbcJvQuoted 3 lines · Toke Høiland-Jørgensen
While you're running tests, could you do one with the target changed to 10ms, just to see what it looks like? Both sojourn time values and throughput would be interesting here, of course.
Apologize for the late reply. Here is the test results with target set to 10ms. The trace for the sojourn time:
https://drive.google.com/open?id=1MEy_wbKKdl22yF17hZaGzpv3uOz6orTiFlent test for 20 ms target time vs 10 ms target time:
https://drive.google.com/open?id=1leIWe0-L0XE78eFvlmRJlNmYgbpoH8xZThe sojourn time measured during throughput test with a relative good 5G connection has mean value around 11 ms, pretty close to the 10 ms target.
A smaller CoDel "target" time could help reduce latency, but it may drop packets too aggressively for stations with low data rate and hurts throughput, as shown in one of the tests with 2.4 GHz client.
Quoted 5 lines
Overall, I think AQL and
fq_codelworks well, at least with ath10k. The current target value of 20 ms is a reasonable default. It is relatively conservative that helps stations with weak signal to maintain stable throughput. Although, a debugfs entry that allows runtime adjustment of target value could be useful.
Why not make it configurable via nl80211? We should use debugfs only for testing and debugging, not in production builds, and to me the use case for this value sounds like more than just testing.
Signature · 1 line
https://wireless.wiki.kernel.org/en/developers/documentation/submittingpatches
A message that is not in the archive (sent off-list or not yet received): 0101016ecf3bc899-6e391bba-96ed-4495-a7be-1aa8dd8f1bf2-000000@us-west-2.amazonses.com
Johannes Berg
maintainerQuoted 10 lines · Kalle Valo
Quoted 6 lines
Overall, I think AQL and
fq_codelworks well, at least with ath10k. The current target value of 20 ms is a reasonable default. It is relatively conservative that helps stations with weak signal to maintain stable throughput. Although, a debugfs entry that allows runtime adjustment of target value could be useful.Why not make it configurable via nl80211? We should use debugfs only for testing and debugging, not in production builds, and to me the use case for this value sounds like more than just testing.
On the other hand, what application/tool or even user would be able to set this correctly?
johannes
Quoted 8 lines · Johannes Berg
Quoted 7 lines · Kalle Valo
Quoted 6 lines
Overall, I think AQL and
fq_codelworks well, at least with ath10k. The current target value of 20 ms is a reasonable default. It is relatively conservative that helps stations with weak signal to maintain stable throughput. Although, a debugfs entry that allows runtime adjustment of target value could be useful.
Quoted 6 lines
Quoted 3 lines
Why not make it configurable via nl80211? We should use debugfs only for testing and debugging, not in production builds, and to me the use case for this value sounds like more than just testing.
On the other hand, what application/tool or even user would be able to set this correctly?
Well, it's not inconceivable that someone might write a tool to dynamically tune this; we do allow it to be changed at the qdisc layer after all.
But until such a time as someone does that, I agree that it's not terribly likely such a knob is going to see much use. As Kan's results show, for inter-flow latency (like what the separate ping in his test is showing), the FQ part takes care of the latency, and what's left there is the AQL buffering. So I'm a little bit "meh" about this; wouldn't object to making it a knob, but don't think I'm going to spend the time writing that patch myself :)
-Toke
Quoted 8 lines · Johannes Berg
Quoted 7 lines · Kalle Valo
Quoted 7 lines
Overall, I think AQL and
fq_codelworks well, at least with ath10k. The current target value of 20 ms is a reasonable default.It is
relatively conservative that helps stations with weak signal to maintain stable throughput.
This statement is overbroad and largely incorrect.
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines
Although, a debugfs entry that allows runtime adjustment of target value could be useful.
Why not make it configurable via nl80211? We should use debugfs only for testing and debugging, not in production builds, and to me the use case for this value sounds like more than just testing.
I certainly lean towards making it configurable AND autotuning it better.
Quoted 2 lines
On the other hand, what application/tool or even user would be able to set this correctly?
The guideline from the theory ("Power") is the target should 5-10% of the interval, and the interval fairly close to the most commonly observed max RTT. I should try to stress (based on some statements made here) - that you have to *consistently* exceed the target for the interval, in order for codel to have any effect at all. Please try to internalize that - the smoothing comes from the interval... 100ms is quite a large interval....
Judging from kan's (rather noisy) data set 10ms is a good default on 5ghz. There is zero difference in throughput as near as I can tell.
It would be interesting to try 3ms (as there's up to 8ms of buffering in the driver) to add to this dataset, helpful also to be measuring the actual tcp rtt rather in addition to the fq behavior.
I see what looks like channel scan behavior in the data. (on the client?) Running tests for 5 minutes will show the impact and frequency of channel scans better.
The 20ms figure we used initially was due to a variety of factors:
* This was the first ever attempt at applying an AQM technology to wifi!!! ** FIXED: http://blog.cerowrt.org/post/real_results/
* We were debugging the FQ component, primarily.
** FIXED: http://blog.cerowrt.org/post/crypto_fq_bug/
* We were working on backports and on integrating a zillion other pieces all in motion.
** sorta FIXED. I know dang full well how many darn variables there
are, as well as how much the network stack has changed since the initial work.
* We were working on 2.4ghz which has a baseline rate of 1Mbit (13ms target) Our rule of thumb is that min target needs to MTU*1.5. There was also a a fudge factor to account for half duplex operation and the minimum size of a txop.
** FIXED: 5ghz has a baseline rate of 6mbits.
* We didn't have tools to look at tcp rtts at the time
** FIXED: flent --socket-stats tcp_nup
* We had issues with power save
** Everybody has issues with powersave...
** These are still extant on many platforms, notably ones that wake up
and dump all their accumulated mcast data into the link. Not our problem.
* channel scans: http://blog.cerowrt.org/post/disabling_channel_scans/
** Non background channel scans are very damaging. I am unsure from this
data if that's what we are seeing from the client? Or the ath10k? the ability to do these in the background or notmight be a factor in autotuning things better.
* We had MAJOR issues with TSQ
** FIXED: https://lwn.net/Articles/757643/
Honestly the TSQ interaction was the biggest barrier to figuring out what was going wrong at the time we upstreamed this, and a tcp_nup test, now, with TSQ closer to "right", AQL in place and the reduced target should be interesting. I think the data we have now on TSQ vs wifi on this chip, is now totally obsolete.
* We had issues with mcast
** I think we still have many issues with multicast but improving that
is a separate problem entirely.
* We ran out of time and money, and had hit it so far out of the park ( https://lwn.net/Articles/705884/ )
that it seemed like sleeping more and tweaking things less was a win.
Judging from the results we now get on 5ghz and on ac, it seems good to reduce the target to 10ms (or less!) on 5ghz ghz, especially on ac, which will result in a less path inflation and no loss in throughput.
I have been running with a 6ms target for several years now on my 802.11n 5ghz devices. (I advertise a 3ms rather than the default txop size also) These are, admittedly, mostly used as backhaul links (so I didn't have tsq, aql, rate changes, etc) , but seing a path inflation of no more than 30ms under full bidirectional load is nice. (and still 22ms worse than it could be in a more perfect world)
Another thing I keep trying to stress: TCP's ability to grab more bandwidth is quadratic relative the delay.
Quoted 2 lines
johannes
Quoted 5 lines · Dave Taht (taht.net)
Judging from kan's (rather noisy) data set 10ms is a good default on 5ghz. There is zero difference in throughput as near as I can tell. It would be interesting to try 3ms (as there's up to 8ms of buffering in the driver) to add to this dataset, helpful also to be measuring the actual tcp rtt rather in addition to the fq behavior.
One large aggregation in 11ac can last 4-5 ms, with bursting, firmware/hardware can complete as much 8 -10 ms worth of frames in one shot and then try to dequeue more frames, so the jitter for the sojourn time can be as high as 8-10 ms. Setting the default target to something less than 10ms can cause unnecessary packet drop in some occasions.
Quoted 104 lines · Dave Taht
Quoted 8 lines · Johannes Berg
Quoted 7 lines · Kalle Valo
Quoted 7 lines
Overall, I think AQL and
fq_codelworks well, at least with ath10k. The current target value of 20 ms is a reasonable default.It is
relatively conservative that helps stations with weak signal to maintain stable throughput.This statement is overbroad and largely incorrect.
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines
Although, a debugfs entry that allows runtime adjustment of target value could be useful.
Why not make it configurable via nl80211? We should use debugfs only for testing and debugging, not in production builds, and to me the use case for this value sounds like more than just testing.
I certainly lean towards making it configurable AND autotuning it better.
Quoted 2 lines
On the other hand, what application/tool or even user would be able to set this correctly?
The guideline from the theory ("Power") is the target should 5-10% of the interval, and the interval fairly close to the most commonly observed max RTT. I should try to stress (based on some statements made here) - that you have to *consistently* exceed the target for the interval, in order for codel to have any effect at all. Please try to internalize that - the smoothing comes from the interval... 100ms is quite a large interval....
Judging from kan's (rather noisy) data set 10ms is a good default on 5ghz. There is zero difference in throughput as near as I can tell.
It would be interesting to try 3ms (as there's up to 8ms of buffering in the driver) to add to this dataset, helpful also to be measuring the actual tcp rtt rather in addition to the fq behavior.
I see what looks like channel scan behavior in the data. (on the client?) Running tests for 5 minutes will show the impact and frequency of channel scans better.
The 20ms figure we used initially was due to a variety of factors:
* This was the first ever attempt at applying an AQM technology to wifi!!! ** FIXED: http://blog.cerowrt.org/post/real_results/
* We were debugging the FQ component, primarily.
** FIXED: http://blog.cerowrt.org/post/crypto_fq_bug/
* We were working on backports and on integrating a zillion other pieces all in motion.
** sorta FIXED. I know dang full well how many darn variables there
are, as well as how much the network stack has changed since the initial work.
* We were working on 2.4ghz which has a baseline rate of 1Mbit (13ms target) Our rule of thumb is that min target needs to MTU*1.5. There was also a a fudge factor to account for half duplex operation and the minimum size of a txop.
** FIXED: 5ghz has a baseline rate of 6mbits.
* We didn't have tools to look at tcp rtts at the time
** FIXED: flent --socket-statstcp_nup
* We had issues with power save
** Everybody has issues with powersave...
** These are still extant on many platforms, notably ones that wake up
and dump all their accumulated mcast data into the link. Not our problem.
* channel scans: http://blog.cerowrt.org/post/disabling_channel_scans/
** Non background channel scans are very damaging. I am unsure from this
data if that's what we are seeing from the client? Or the ath10k? the ability to do these in the background or notmight be a factor in autotuning things better.
* We had MAJOR issues with TSQ
** FIXED: https://lwn.net/Articles/757643/Honestly the TSQ interaction was the biggest barrier to figuring out what was going wrong at the time we upstreamed this, and a
tcp_nuptest, now, with TSQ closer to "right", AQL in place and the reduced target should be interesting. I think the data we have now on TSQ vs wifi on this chip, is now totally obsolete.* We had issues with mcast
** I think we still have many issues with multicast but improving that
is a separate problem entirely.
* We ran out of time and money, and had hit it so far out of the park ( https://lwn.net/Articles/705884/ )
that it seemed like sleeping more and tweaking things less was a win.Judging from the results we now get on 5ghz and on ac, it seems good to reduce the target to 10ms (or less!) on 5ghz ghz, especially on ac, which will result in a less path inflation and no loss in throughput.
I have been running with a 6ms target for several years now on my 802.11n 5ghz devices. (I advertise a 3ms rather than the default txop size also) These are, admittedly, mostly used as backhaul links (so I didn't have tsq, aql, rate changes, etc) , but seing a path inflation of no more than 30ms under full bidirectional load is nice. (and still 22ms worse than it could be in a more perfect world)
Another thing I keep trying to stress: TCP's ability to grab more bandwidth is quadratic relative the delay.
Quoted 2 lines
johannes