ath10k: Attempt to work around napi_synchronize hang.
RFC 1/1 ath10k: Attempt to work around napi_synchronize hang.
From: Ben Greear <[email protected]>
Calling napi_disable twice in a row (w/out starting it and/or without having NAPI active leads to deadlock because napi_disable sets NAPI_STATE_SCHED and NAPI_STATE_NPSVC when it returns, as far as I can tell. So, guard this call to napi_disable. I believe the failure case is something like this:
rmmod ath10k_pci ath10k_core
Firmware crashes before hif_stop is called by the rmmod path The crash handling logic calls hif_stop
Then rmmod gets around to calling hif_stop, but spins endlessly in napi_synchronize.
I think one way this could happen is that ath10k_stop checks for state != ATH10K_STATE_OFF, but STATE_RESTARTING is also a possibility. That might be how we can have hif_stop called twice without a hif_start in between. --Ben
--- drivers/net/wireless/ath/ath10k/core.h | 1 + drivers/net/wireless/ath/ath10k/pci.c | 25 +++++++++++++++++++++++-- 2 files changed, 24 insertions(+), 2 deletions(-)
drivers/net/wireless/ath/ath10k/core.h +1 -0
@@ -1205,6 +1205,7 @@ struct ath10k {/* NAPI */ struct net_device napi_dev;struct napi_struct napi; +bool napi_enabled;struct work_struct stop_scan_work;
drivers/net/wireless/ath/ath10k/pci.c +23 -2
@@ -1956,6 +1956,7 @@ static int ath10k_pci_hif_start(struct ath10k *ar)ath10k_dbg (ar,ATH10K_DBG_BOOT ,"boot hif start\n" );napi_enable (&ar->napi); + ar->napi_enabled = true;ath10k_pci_irq_enable (ar);ath10k_pci_rx_post (ar);
@@ -2086,8 +2087,28 @@ static void ath10k_pci_hif_stop(struct ath10k *ar)ath10k_pci_irq_disable (ar);ath10k_pci_irq_sync (ar);ath10k_pci_flush (ar); -napi_synchronize (&ar->napi); -napi_disable (&ar->napi); + +/* Calling napi_disable twice in a row (w/out starting it and/or without +* having NAPI active leads to deadlock because napi_disable sets +* NAPI_STATE_SCHED and NAPI_STATE_NPSVC when it returns, as far as I +* can tell. So, guard this call to napi_disable. I believe the +* failure case is something like this: +* rmmod ath10k_pci ath10k_core +* Firmware crashes before hif_stop is called by the rmmod path +* The crash handling logic calls hif_stop +* Then rmmod gets around to calling hif_stop, but spins endlessly +* in napi_synchronize. +* +* I think one way this could happen is that ath10k_stop checks +* for state != ATH10K_STATE_OFF, but STATE_RESTARTING is also +* a possibility. That might be how we can have hif_stop called twice +* without a hif_start in between. --Ben +*/ +if (ar->napi_enabled) { +napi_synchronize (&ar->napi); +napi_disable (&ar->napi); + ar->napi_enabled = false; + }spin_lock_irqsave (&ar_pci->ps_lock, flags);WARN_ON (ar_pci->ps_wake_refcount >0 );
Signature · 1 line
2.4.11
On 28 February 2018 at 02:22, <[email protected]> wrote:
[...]
drivers/net/wireless/ath/ath10k/pci.c static void ath10k_pci_hif_stop(struct ath10k *ar) · line 2091
Quoted 8 lines
drivers/net/wireless/ath/ath10k/pci.c +2
-2@@ -2086,8 +2087,28 @@ static void ath10k_pci_hif_stop(struct ath10k *ar)ath10k_pci_irq_disable (ar);ath10k_pci_irq_sync (ar);ath10k_pci_flush (ar); -napi_synchronize (&ar->napi); -napi_disable (&ar->napi); + +/* Calling napi_disable twice in a row (w/out starting it and/or =
without
Review static void ath10k_pci_hif_stop(struct ath10k *ar) · line 2093
Quoted 2 lines
(quoted diff) +2
-0+ * having NAPI active leads to deadlock because napi_disable sets + * NAPI_STATE_SCHED and NAPI_STATE_NPSVC when it returns, as far =
as I
drivers/net/wireless/ath/ath10k/pci.c static void ath10k_pci_hif_stop(struct ath10k *ar)
Quoted 6 lines
drivers/net/wireless/ath/ath10k/pci.c +6
-0+* can tell. So, guard this call to napi_disable. I believe the +* failure case is something like this: +* rmmod ath10k_pci ath10k_core +* Firmware crashes before hif_stop is called by the rmmod path +* The crash handling logic calls hif_stop +* Then rmmod gets around to calling hif_stop, but spins endle=
ssly
drivers/net/wireless/ath/ath10k/pci.c static void ath10k_pci_hif_stop(struct ath10k *ar)
Quoted 5 lines
drivers/net/wireless/ath/ath10k/pci.c +5
-0+* in napi_synchronize. +* +* I think one way this could happen is that ath10k_stop checks +* for state !=3D ATH10K_STATE_OFF, but STATE_RESTARTING is also +* a possibility. That might be how we can have hif_stop called=
twice
drivers/net/wireless/ath/ath10k/pci.c static void ath10k_pci_hif_stop(struct ath10k *ar) · line 2111
Quoted 7 lines
drivers/net/wireless/ath/ath10k/pci.c +7
-0+* without a hif_start in between. --Ben +*/ +if (ar->napi_enabled) { +napi_synchronize (&ar->napi); +napi_disable (&ar->napi); + ar->napi_enabled =3D false; + }
Looking at the code and your comment the described fail case seems legit. I would consider tuning ath10k_stop() so that it calls ath10k_halt() only if ar->state !=3D OFF & ar->state !=3D RESTARTING though. Or did you try already?
While your approach will probably work it won't prevent other non-NAPI bad things from happening. Even if there's nothing today it might creep up in the future. And you'd need to update ahb.c too.
Micha=C5=82
Quoted 27 lines · Michał Kazior
On 28 February 2018 at 02:22, <[email protected]> wrote:
[...]Quoted 25 lines
drivers/net/wireless/ath/ath10k/pci.c +19
-2@@ -2086,8 +2087,28 @@ static void ath10k_pci_hif_stop(struct ath10k *ar)ath10k_pci_irq_disable (ar);ath10k_pci_irq_sync (ar);ath10k_pci_flush (ar); -napi_synchronize (&ar->napi); -napi_disable (&ar->napi); + +/* Calling napi_disable twice in a row (w/out starting it and/or without +* having NAPI active leads to deadlock because napi_disable sets +* NAPI_STATE_SCHED and NAPI_STATE_NPSVC when it returns, as far as I +* can tell. So, guard this call to napi_disable. I believe the +* failure case is something like this: +* rmmod ath10k_pci ath10k_core +* Firmware crashes before hif_stop is called by the rmmod path +* The crash handling logic calls hif_stop +* Then rmmod gets around to calling hif_stop, but spins endlessly +* in napi_synchronize. +* +* I think one way this could happen is that ath10k_stop checks +* for state != ATH10K_STATE_OFF, but STATE_RESTARTING is also +* a possibility. That might be how we can have hif_stop called twice +* without a hif_start in between. --Ben +*/ +if (ar->napi_enabled) { +napi_synchronize (&ar->napi);
Quoted 8 lines
drivers/net/wireless/ath/ath10k/pci.c static void ath10k_pci_hif_stop(struct ath10k *ar) · line 2111
Quoted 3 lines
drivers/net/wireless/ath/ath10k/pci.c +3
-0+napi_disable (&ar->napi); + ar->napi_enabled = false; + }Looking at the code and your comment the described fail case seems legit. I would consider tuning
ath10k_stop()so that it callsath10k_halt()only ifar->state!= OFF &ar->state!= RESTARTING though. Or did you try already?
I did not try tuning ath10k_stop(). The code in this area is quite complex, and in my opinion trying to keep the start/stop calls exactly matched without individual 'has_started' flags seems ripe for bugs.
Quoted 3 lines
While your approach will probably work it won't prevent other non-NAPI bad things from happening. Even if there's nothing today it might creep up in the future. And you'd need to update
ahb.ctoo.
I'll update ahb.c to match.
Thanks,
Ben
Quoted 4 lines
Michał
Signature · 2 lines
Ben Greear <[email protected]> Candela Technologies Inc http://www.candelatech.com