Skip to contentSkip to conversation

wlcore: fix race for WL1271_FLAG_IRQ_RUNNING

PATCH In review Tony Lindgren 2 participants 4 messages Oct 7, 2019 – Oct 9, 2019 3 messages not in the archive TI wilink wireless drivers

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 wlcore: fix race for WL1271_FLAG_IRQ_RUNNING

We set WL1271_FLAG_IRQ_RUNNING in the beginning of wlcore_irq(), and test for it in wlcore_runtime_resume(). But WL1271_FLAG_IRQ_RUNNING currently gets cleared too early by wlcore_irq_locked() before wlcore_irq() is done calling it. And this will race against wlcore_runtime_resume() testing it.

Let's set and clear IRQ_RUNNING in wlcore_irq() so wlcore_runtime_resume() can rely on it. And let's remove old comments about hardirq, that's no longer the case as we're using request_threaded_irq().

This fixes occasional annoying wlcore firmware reboots stat start with "wlcore: WARNING ELP wakeup timeout!" followed by a multisecond latency when the wlcore firmware gets wrongly rebooted waiting for an ELP wake interrupt that won't be coming.

Note that I also suspect some form of this issue was the root cause why the wlcore GPIO interrupt has been often configured as a level interrupt instead of edge as an attempt to work around the ELP wake timeout errors.

Fixes fa2648a34e73 ("wlcore: Add support for runtime PM")
Cc Anders Roxell <[email protected]>
Cc Eyal Reizer <[email protected]>
Cc Guy Mishol <[email protected]>
Cc John Stultz <[email protected]>
Cc Ulf Hansson <[email protected]>

---

Changes since v1:

- Add locking around clear_bit like we do elsewhere in the driver

 drivers/net/wireless/ti/wlcore/main.c | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)
drivers/net/wireless/ti/wlcore/main.c +6 -6
@@ -544,11 +544,6 @@ static int wlcore_irq_locked(struct wl1271 *wl)
 	}
 
 	while (!done && loopcount--) {
-		/*
-		 * In order to avoid a race with the hardirq, clear the flag
-		 * before acknowledging the chip.
-		 */
-		clear_bit(WL1271_FLAG_IRQ_RUNNING, &wl->flags);
 		smp_mb__after_atomic();
 
 		ret = wlcore_fw_status(wl, wl->fw_status);
@@ -668,7 +663,7 @@ static irqreturn_t wlcore_irq(int irq, void *cookie)
 		disable_irq_nosync(wl->irq);
 		pm_wakeup_event(wl->dev, 0);
 		spin_unlock_irqrestore(&wl->wl_lock, flags);
-		return IRQ_HANDLED;
+		goto out_handled;
 	}
 	spin_unlock_irqrestore(&wl->wl_lock, flags);
 
@@ -692,6 +687,11 @@ static irqreturn_t wlcore_irq(int irq, void *cookie)
 
 	mutex_unlock(&wl->mutex);
 
+out_handled:
+	spin_lock_irqsave(&wl->wl_lock, flags);
+	clear_bit(WL1271_FLAG_IRQ_RUNNING, &wl->flags);
+	spin_unlock_irqrestore(&wl->wl_lock, flags);
+
 	return IRQ_HANDLED;
 }
 
Signature · 1 line
2.23.0

A message that is not in the archive (sent off-list or not yet received): 20191007172800.64249-1-tony-4v6yS6AI5VpBDgjK7y7TUQ@public.gmane.org

#2 lore raw
Quoted 9 lines · Tony Lindgren

We set WL1271_FLAG_IRQ_RUNNING in the beginning of wlcore_irq(), and test for it in wlcore_runtime_resume(). But WL1271_FLAG_IRQ_RUNNING currently gets cleared too early by wlcore_irq_locked() before wlcore_irq() is done calling it. And this will race against wlcore_runtime_resume() testing it.

Let's set and clear IRQ_RUNNING in wlcore_irq() so wlcore_runtime_resume() can rely on it. And let's remove old comments about hardirq, that's no longer the case as we're using request_threaded_irq().

Quoted 8 lines

This fixes occasional annoying wlcore firmware reboots stat start with "wlcore: WARNING ELP wakeup timeout!" followed by a multisecond latency when the wlcore firmware gets wrongly rebooted waiting for an ELP wake interrupt that won't be coming.

Note that I also suspect some form of this issue was the root cause why the wlcore GPIO interrupt has been often configured as a level interrupt instead of edge as an attempt to work around the ELP wake timeout errors.

So this fixed a reproducable test case where loading some webpages often produced ELP timeout errors. But looks like I'm still seeing ELP timeouts elsewhere. So best to wait on this one. Something is still wrong with the ELP timeout handling.

Regards,

Tony

Quoted 54 lines
Fixes fa2648a34e73 ("wlcore: Add support for runtime PM")
Cc Anders Roxell <[email protected]>
Cc Eyal Reizer <[email protected]>
Cc Guy Mishol <[email protected]>
Cc John Stultz <[email protected]>
Cc Ulf Hansson <[email protected]>

---

Changes since v1:

- Add locking around clear_bit like we do elsewhere in the driver

 drivers/net/wireless/ti/wlcore/main.c | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)
drivers/net/wireless/ti/wlcore/main.c +6 -6
@@ -544,11 +544,6 @@ static int wlcore_irq_locked(struct wl1271 *wl)
 	}
 
 	while (!done && loopcount--) {
-		/*
-		 * In order to avoid a race with the hardirq, clear the flag
-		 * before acknowledging the chip.
-		 */
-		clear_bit(WL1271_FLAG_IRQ_RUNNING, &wl->flags);
 		smp_mb__after_atomic();
 
 		ret = wlcore_fw_status(wl, wl->fw_status);
@@ -668,7 +663,7 @@ static irqreturn_t wlcore_irq(int irq, void *cookie)
 		disable_irq_nosync(wl->irq);
 		pm_wakeup_event(wl->dev, 0);
 		spin_unlock_irqrestore(&wl->wl_lock, flags);
-		return IRQ_HANDLED;
+		goto out_handled;
 	}
 	spin_unlock_irqrestore(&wl->wl_lock, flags);
 
@@ -692,6 +687,11 @@ static irqreturn_t wlcore_irq(int irq, void *cookie)
 
 	mutex_unlock(&wl->mutex);
 
+out_handled:
+	spin_lock_irqsave(&wl->wl_lock, flags);
+	clear_bit(WL1271_FLAG_IRQ_RUNNING, &wl->flags);
+	spin_unlock_irqrestore(&wl->wl_lock, flags);
+
 	return IRQ_HANDLED;
 }
 
Signature · 1 line
2.23.0

A message that is not in the archive (sent off-list or not yet received): [email protected]

#4 lore raw
Quoted 15 lines · Tony Lindgren
Quoted 14 lines · Tony Lindgren

We set WL1271_FLAG_IRQ_RUNNING in the beginning of wlcore_irq(), and test for it in wlcore_runtime_resume(). But WL1271_FLAG_IRQ_RUNNING currently gets cleared too early by wlcore_irq_locked() before wlcore_irq() is done calling it. And this will race against wlcore_runtime_resume() testing it.

Let's set and clear IRQ_RUNNING in wlcore_irq() so wlcore_runtime_resume() can rely on it. And let's remove old comments about hardirq, that's no longer the case as we're using request_threaded_irq().

This fixes occasional annoying wlcore firmware reboots stat start with "wlcore: WARNING ELP wakeup timeout!" followed by a multisecond latency when the wlcore firmware gets wrongly rebooted waiting for an ELP wake interrupt that won't be coming.

Quoted 8 lines
Quoted 3 lines

Note that I also suspect some form of this issue was the root cause why the wlcore GPIO interrupt has been often configured as a level interrupt instead of edge as an attempt to work around the ELP wake timeout errors.

So this fixed a reproducable test case where loading some webpages often produced ELP timeout errors. But looks like I'm still seeing ELP timeouts elsewhere. So best to wait on this one. Something is still wrong with the ELP timeout handling.

Ok, I'll drop this then. Please send v3 once you think the patch is ready to be applied.

Signature · 1 line
https://wireless.wiki.kernel.org/en/developers/documentation/submittingpatches

A message that is not in the archive (sent off-list or not yet received): [email protected]

#6 lore raw
Quoted 21 lines · Kalle Valo
Quoted 19 lines · Tony Lindgren
Quoted 17 lines · Tony Lindgren

We set WL1271_FLAG_IRQ_RUNNING in the beginning of wlcore_irq(), and test for it in wlcore_runtime_resume(). But WL1271_FLAG_IRQ_RUNNING currently gets cleared too early by wlcore_irq_locked() before wlcore_irq() is done calling it. And this will race against wlcore_runtime_resume() testing it.

Let's set and clear IRQ_RUNNING in wlcore_irq() so wlcore_runtime_resume() can rely on it. And let's remove old comments about hardirq, that's no longer the case as we're using request_threaded_irq().

This fixes occasional annoying wlcore firmware reboots stat start with "wlcore: WARNING ELP wakeup timeout!" followed by a multisecond latency when the wlcore firmware gets wrongly rebooted waiting for an ELP wake interrupt that won't be coming.

Note that I also suspect some form of this issue was the root cause why the wlcore GPIO interrupt has been often configured as a level interrupt instead of edge as an attempt to work around the ELP wake timeout errors.

Quoted 7 lines
Quoted 4 lines

So this fixed a reproducable test case where loading some webpages often produced ELP timeout errors. But looks like I'm still seeing ELP timeouts elsewhere. So best to wait on this one. Something is still wrong with the ELP timeout handling.

Ok, I'll drop this then. Please send v3 once you think the patch is ready to be applied.

Looks like the real fix is to use level instead of edge interrupt for omap4 and 5 to avoid the check for untriggered interrupts in omap_gpio_unidle(). Should not be needed for other SoCs as their l4per can't idle independent of the CPUs.

I'll send a separate patch for that. And I'll send an updated clean-up patch for $subject patch as the race described above should never happen.

The clearing of WL1271_FLAG_IRQ_RUNNING bit happens already within pm_runtime_get_sync() section of wlcore_irq_locked(). So this patch just happened to sligthly change the timings for my reproducable test case. We should not be able to hit the race described above even with super short autosuspend timeouts between wlcore_irq_locked() and the end of wlcore_irq() :)

Regards,

Tony

Quoted 2 lines
Signature · 1 line
https://wireless.wiki.kernel.org/en/developers/documentation/submittingpatches