start recovery process when payload length overflow for sdio
v2 cover start recovery process when payload length overflow for sdio
when it happened payload length exceeds max htc length, start recovery process
v2: add "add refcount for ath10k_core_restart" and remove ar_sdio->can_recovery
Wen Gong (2):
ath10k: add refcount for ath10k_core_restart
ath10k: start recovery process when payload length exceeds max htc
length for sdio
drivers/net/wireless/ath/ath10k/core.c | 8 ++++++++ drivers/net/wireless/ath/ath10k/core.h | 2 ++ drivers/net/wireless/ath/ath10k/mac.c | 1 + drivers/net/wireless/ath/ath10k/sdio.c | 6 ++++++ 4 files changed, 17 insertions(+)
Signature · 1 line
2.23.0
PATCH v2 1/2 ath10k: add refcount for ath10k_core_restart
When it has more than one restart_work queued meanwhile, the 2nd restart_work is very esay to break the 1st restart work and lead recovery fail.
Add a ref count to allow only one restart work running untill device successfully recovered.
This patch only effect sdio chips.
Tested with QCA6174 SDIO with firmware WLAN.RMH.4.4.1-00029.
--- drivers/net/wireless/ath/ath10k/core.c | 8 ++++++++ drivers/net/wireless/ath/ath10k/core.h | 2 ++ drivers/net/wireless/ath/ath10k/mac.c | 1 + 3 files changed, 11 insertions(+)
drivers/net/wireless/ath/ath10k/core.c +8 -0
@@ -2199,6 +2199,14 @@ static void ath10k_core_restart(struct work_struct *work) { struct ath10k *ar = container_of(work, struct ath10k, restart_work);int ret; +int restart_count; + + restart_count =atomic_inc_and_test (&ar->restart_count); +if (restart_count >1 ) { +ath10k_warn (ar,"can not restart, count: %d\n" , restart_count); +atomic_dec (&ar->restart_count); +return ; + }set_bit (ATH10K_FLAG_CRASH_FLUSH , &ar->dev_flags);
drivers/net/wireless/ath/ath10k/core.h +2 -0
@@ -982,6 +982,8 @@ struct ath10k {/* protected by conf_mutex */ u8 ps_state_enable; +atomic_t restart_count; +bool nlo_enabled;bool p2p;
drivers/net/wireless/ath/ath10k/mac.c +1 -0
@@ -7208,6 +7208,7 @@ static void ath10k_reconfig_complete(struct ieee80211_hw *hw,ath10k_info (ar,"device successfully recovered\n" ); ar->state =ATH10K_STATE_ON ;ieee80211_wake_queues (ar->hw); +atomic_dec (&ar->restart_count); }mutex_unlock (&ar->conf_mutex);
Signature · 6 lines
2.23.0 _______________________________________________ ath10k mailing list [email protected] http://lists.infradead.org/mailman/listinfo/ath10k
Re: [PATCH v2 1/2] ath10k: add refcount for ath10k_core_restart
This does not only effect SDIO.
Why a semaphore / count? Could the conf_mutex be held earlier, or perhaps change the state to ATH10K_STATE_RESTARTING first? ath10k_reconfig_complete is also called in mac.c when channel is changed so
Quoted 64 lines · Wen Gong
When it has more than one
restart_workqueued meanwhile, the 2ndrestart_workis very esay to break the 1st restart work and lead recovery fail.Add a ref count to allow only one restart work running untill device successfully recovered.
This patch only effect sdio chips.
Tested with QCA6174 SDIO with firmware WLAN.RMH.4.4.1-00029.
Signed-off-by Wen Gong <[email protected]>--- drivers/net/wireless/ath/ath10k/core.c | 8 ++++++++ drivers/net/wireless/ath/ath10k/core.h | 2 ++ drivers/net/wireless/ath/ath10k/mac.c | 1 + 3 files changed, 11 insertions(+)drivers/net/wireless/ath/ath10k/core.c +8
-0@@ -2199,6 +2199,14 @@ static void ath10k_core_restart(struct work_struct *work) { struct ath10k *ar = container_of(work, struct ath10k, restart_work);int ret; +int restart_count; + + restart_count =atomic_inc_and_test (&ar->restart_count); +if (restart_count >1 ) { +ath10k_warn (ar,"can not restart, count: %d\n" , restart_count); +atomic_dec (&ar->restart_count); +return ; + }set_bit (ATH10K_FLAG_CRASH_FLUSH , &ar->dev_flags);drivers/net/wireless/ath/ath10k/core.h +2
-0@@ -982,6 +982,8 @@ struct ath10k {/* protected by conf_mutex */ u8 ps_state_enable; +atomic_t restart_count; +bool nlo_enabled;bool p2p;drivers/net/wireless/ath/ath10k/mac.c +1
-1@@ -7208,6 +7208,7 @@ static void ath10k_reconfig_complete(struct ieee80211_hw *hw,ath10k_info (ar,"device successfully recovered\n" ); ar->state =ATH10K_STATE_ON ;ieee80211_wake_queues (ar->hw); +atomic_dec (&ar->restart_count); }mutex_unlock (&ar->conf_mutex); --
Quoted 1 line
2.23.0
_______________________________________________
ath10k mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/ath10k
Re: [PATCH v2 1/2] ath10k: add refcount for ath10k_core_restart
Quoted 5 lines · Justin Capella
This does not only effect SDIO.
Why a semaphore / count? Could the
conf_mutexbe held earlier, or perhaps change the state toATH10K_STATE_RESTARTINGfirst?ath10k_reconfig_completeis also called inmac.cwhen channel is changed so
patch v2:
https://patchwork.kernel.org/patch/11313853/
https://patchwork.kernel.org/patch/11313859/
_______________________________________________
ath10k mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/ath10k
Re: [PATCH v2 1/2] ath10k: add refcount for ath10k_core_restart
Instead of the atomic restart count, can the state be updated to ATH10K_STATE_RESTARTING while holding
mutex_unlock(&ar->conf_mutex);
I don't understand the bundles, but I wonder about the case when there are multiple packets (n_rx_pkts) and if pkt_bundle_len might be the one to check. Also if there needs to be a check that the len > sizeof HTC HDR.
Quoted 11 lines · On Tue, Dec 31, 2019 at 1:37 AM <[email protected]> wrote:
Quoted 6 lines · Justin Capella
This does not only effect SDIO.
Why a semaphore / count? Could the
conf_mutexbe held earlier, or perhaps change the state toATH10K_STATE_RESTARTINGfirst?ath10k_reconfig_completeis also called inmac.cwhen channel is changed sopatch v2:
https://patchwork.kernel.org/patch/11313853/
https://patchwork.kernel.org/patch/11313859/
_______________________________________________
ath10k mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/ath10k
Re: [PATCH v2 1/2] ath10k: add refcount for ath10k_core_restart
Quoted 4 lines · Justin Capella
Instead of the atomic restart count, can the state be updated to
ATH10K_STATE_RESTARTINGwhile holdingmutex_unlock(&ar->conf_mutex);
the recovery process is begin with ath10k_core_restart, and end with ath10k_reconfig_complete. I already see it has mutex_lock(&ar->conf_mutex) and mutex_unlock(&ar->conf_mutex) in ath10k_core_restart, but it is not enough, for example:
1st recovery has finished ath10k_core_restart, but not arrive ath10k_reconfig_complete, then the 2nd recovery begin to enter ath10k_core_restart, it will destroy the 1st recovery and let 1st recovery fail. After apply this patch, after recovery about 18000+ times, and still can connect/scan/ping success.
Quoted 5 lines
I don't understand the bundles, but I wonder about the case when there are multiple packets (
n_rx_pkts) and ifpkt_bundle_lenmight be the one to check. Also if there needs to be a check that the len > sizeof HTC HDR.
the htc_hdr->len is len of payload, so it allow < sizeof HTC HDR, but not allow > ATH10K_HTC_MBOX_MAX_PAYLOAD_LENGTH. pkt_bundle is only used when it has many packet in rx side, otherwise it is not bundled in rx.
patch v3:
https://patchwork.kernel.org/patch/11313853/
https://patchwork.kernel.org/patch/11313859/
Quoted 5 lines · On Tue, Dec 31, 2019 at 1:37 AM <[email protected]> wrote:
Quoted 4 lines · Justin Capella
Quoted 2 lines
This does not only effect SDIO.
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines
Why a semaphore / count? Could the
conf_mutexbe held earlier, or perhaps change the state toATH10K_STATE_RESTARTINGfirst?ath10k_reconfig_completeis also called inmac.cwhen channel is changed sopatch v2:
https://patchwork.kernel.org/patch/11313853/
https://patchwork.kernel.org/patch/11313859/
_______________________________________________
ath10k mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/ath10k
PATCH v2 2/2 ath10k: start recovery process when payload length exceeds max htc length for sdio
In reply to Wen GongWhen simulate random transfer fail for sdio write and read, it happened "payload length exceeds max htc length" and recovery later sometimes.
Test steps:
1. Add config and update kernel:CONFIG_FAIL_MMC_REQUEST=yCONFIG_FAULT_INJECTION=yCONFIG_FAULT_INJECTION_DEBUG_FS=y
2. Run simulate fail:
cd /sys/kernel/debug/mmc1/fail_mmc_request
echo 10 > probability
echo 10 > times # repeat until hitting issues
3. It happened payload length exceeds max htc length.
[ 199.935506] ath10k_sdio mmc1:0001:1: payload length 57005 exceeds max htc length: 4088 .... [ 264.990191] ath10k_sdio mmc1:0001:1: payload length 57005 exceeds max htc length: 4088
4. after some time, such as 60 seconds, it start recovery which triggered by wmi command timeout for periodic scan.
[ 269.229232] ieee80211 phy0: Hardware restart was requested [ 269.734693] ath10k_sdio mmc1:0001:1: device successfully recovered
The simulate fail of sdio is not a real sdio transter fail, it only set an error status in mmc_should_fail_request after the transfer end, actually the transfer is success, then sdio_io_rw_ext_helper will return error status and stop transfer the left data. For example, the really RX len is 286 bytes, then it will split to 2 blocks in sdio_io_rw_ext_helper, one is 256 bytes, left is 30 bytes, if the first 256 bytes get an error status by mmc_should_fail_request,then the left 30 bytes will not read in this RX operation. Then when the next RX arrive, the left 30 bytes will be considered as the header of the read, the top 4 bytes of the 30 bytes will be considered as lookaheads, but actually the 4 bytes is not the lookaheads, so the len from this lookaheads is not correct, it exceeds max htc length 4088 sometimes. When happened exceeds, the buffer chain is not matched between firmware and ath10k, then it need to start recovery ASAP. Recently then recovery will be started by wmi command timeout, but it will be long time later, for example, it is 60+ seconds later from the periodic scan, if it does not have periodic scan, it will be longer.
Start recovery when it happened "payload length exceeds max htc length" will be reasonable.
This patch only effect sdio chips.
Tested with QCA6174 SDIO with firmware WLAN.RMH.4.4.1-00029.
--- drivers/net/wireless/ath/ath10k/sdio.c | 6 ++++++ 1 file changed, 6 insertions(+)
drivers/net/wireless/ath/ath10k/sdio.c +6 -0
@@ -557,6 +557,12 @@ static int ath10k_sdio_mbox_rx_alloc(struct ath10k *ar,le16_to_cpu (htc_hdr->len), ATH10K_HTC_MBOX_MAX_PAYLOAD_LENGTH); ret = -ENOMEM ; + +if (ar->state ==ATH10K_STATE_ON ) { +queue_work (ar->workqueue, &ar->restart_work); +ath10k_warn (ar,"exceeds length, start recovery\n" ); + } + goto err; }
Signature · 1 line
2.23.0
Re: [PATCH v2 2/2] ath10k: start recovery process when payload length exceeds max htc length for sdio
Does the SDIO bus require addresses to be word aligned like the PCI bus does? I'm thinking of how netdev alloc uses skb_push to ensure that the payload is aligned.
Quoted 1 line
Quoted 1 line
if (
ar->state==ATH10K_STATE_ON)
What about the other STATEs: RESTARTED/ING
The value you mentioned 57005, is 0xDEAD is that a special case? Perhaps a result of fw crash? Maybe a lookahead gone wong? I see its the WMI PEER ALIVE/DEAD indicator but I'm not sure why it would be trailer of the other
Quoted 74 lines · Wen Gong
When simulate random transfer fail for sdio write and read, it happened "payload length exceeds max htc length" and recovery later sometimes.
Test steps:
1. Add config and update kernel:CONFIG_FAIL_MMC_REQUEST=yCONFIG_FAULT_INJECTION=yCONFIG_FAULT_INJECTION_DEBUG_FS=y2. Run simulate fail:
cd/sys/kernel/debug/mmc1/fail_mmc_request
echo 10 > probability
echo 10 > times # repeat until hitting issues3. It happened payload length exceeds max htc length.
[ 199.935506] ath10k_sdio mmc1:0001:1: payload length 57005 exceeds max htc length: 4088 .... [ 264.990191] ath10k_sdio mmc1:0001:1: payload length 57005 exceeds max htc length: 40884. after some time, such as 60 seconds, it start recovery which triggered by wmi command timeout for periodic scan.
[ 269.229232] ieee80211 phy0: Hardware restart was requested [ 269.734693] ath10k_sdio mmc1:0001:1: device successfully recoveredThe simulate fail of sdio is not a real sdio transter fail, it only set an error status in
mmc_should_fail_requestafter the transfer end, actually the transfer is success, thensdio_io_rw_ext_helperwill return error status and stop transfer the left data. For example, the really RX len is 286 bytes, then it will split to 2 blocks insdio_io_rw_ext_helper, one is 256 bytes, left is 30 bytes, if the first 256 bytes get an error status by mmc_should_fail_request,then the left 30 bytes will not read in this RX operation. Then when the next RX arrive, the left 30 bytes will be considered as the header of the read, the top 4 bytes of the 30 bytes will be considered as lookaheads, but actually the 4 bytes is not the lookaheads, so the len from this lookaheads is not correct, it exceeds max htc length 4088 sometimes. When happened exceeds, the buffer chain is not matched between firmware and ath10k, then it need to start recovery ASAP. Recently then recovery will be started by wmi command timeout, but it will be long time later, for example, it is 60+ seconds later from the periodic scan, if it does not have periodic scan, it will be longer.Start recovery when it happened "payload length exceeds max htc length" will be reasonable.
This patch only effect sdio chips.
Tested with QCA6174 SDIO with firmware WLAN.RMH.4.4.1-00029.
Signed-off-by Wen Gong <[email protected]>--- drivers/net/wireless/ath/ath10k/sdio.c | 6 ++++++ 1 file changed, 6 insertions(+)drivers/net/wireless/ath/ath10k/sdio.c +6
-1@@ -557,6 +557,12 @@ static int ath10k_sdio_mbox_rx_alloc(struct ath10k *ar,le16_to_cpu (htc_hdr->len), ATH10K_HTC_MBOX_MAX_PAYLOAD_LENGTH); ret = -ENOMEM ; + +if (ar->state ==ATH10K_STATE_ON ) { +queue_work (ar->workqueue, &ar->restart_work); +ath10k_warn (ar,"exceeds length, start recovery\n" ); + } + goto err; } --2.23.0
Re: [PATCH v2 2/2] ath10k: start recovery process when payload length exceeds max htc length for sdio
Quoted 5 lines · Justin Capella
Does the SDIO bus require addresses to be word aligned like the PCI bus does? I'm thinking of how netdev alloc uses
skb_pushto ensure that the payload is aligned.
Quoted 9 lines
Quoted 1 line
Quoted 1 line
if (
ar->state==ATH10K_STATE_ON)What about the other STATEs: RESTARTED/ING
The value you mentioned 57005, is
0xDEADis that a special case? Perhaps a result of fw crash? Maybe a lookahead gone wong? I see its the WMI PEER ALIVE/DEAD indicator but I'm not sure why it would be trailer of the other
i have removed "if (ar->state == ATH10K_STATE_ON)" and v3 sent:
patch v3:
https://patchwork.kernel.org/patch/11313853/
https://patchwork.kernel.org/patch/11313859/
Quoted 1 line
Quoted 1 line
2.23.0