Skip to contentSkip to conversation

brcmfmac: abort and release host after error

PATCH In review Guenter Roeck 6 participants 9 messages Jan 28, 2020 – Feb 12, 2020

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 brcmfmac: abort and release host after error

With commit 216b44000ada ("brcmfmac: Fix use after free in brcmf_sdio_readframes()") applied, we see locking timeouts in brcmf_sdio_watchdog_thread().

brcmfmac: brcmf_escan_timeout: timer expired

INFO: task brcmf_wdog/mmc1:621 blocked for more than 120 seconds.

Not tainted 4.19.94-07984-g24ff99a0f713 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
brcmf_wdog/mmc1 D 0 621 2 0x00000000 last_sleep: 2440793077. last_runnable: 2440766827
[<c0aa1e60>] (__schedule) from [<c0aa2100>] (schedule+0x98/0xc4)
[<c0aa2100>] (schedule) from [<c0853830>] (__mmc_claim_host+0x154/0x274)
[<c0853830>] (__mmc_claim_host) from [<bf10c5b8>] (brcmf_sdio_watchdog_thread+0x1b0/0x1f8 [brcmfmac]) [<bf10c5b8>] (brcmf_sdio_watchdog_thread [brcmfmac]) from [<c02570b8>] (kthread+0x178/0x180)

In addition to restarting or exiting the loop, it is also necessary to abort the command and to release the host.

Fixes 216b44000ada ("brcmfmac: Fix use after free in brcmf_sdio_readframes()")
Cc Dan Carpenter <[email protected]>
Cc Matthias Kaehlcke <[email protected]>
Cc Brian Norris <[email protected]>
Cc Douglas Anderson <[email protected]>
---
 drivers/net/wireless/broadcom/brcm80211/brcmfmac/sdio.c | 2 ++
 1 file changed, 2 insertions(+)
drivers/net/wireless/broadcom/brcm80211/brcmfmac/sdio.c +2 -0
index f9df95bc7fa1..2e1c23c7269d 100644
@@ -1938,6 +1938,8 @@ static uint brcmf_sdio_readframes(struct brcmf_sdio *bus, uint maxframes)
 			if (brcmf_sdio_hdparse(bus, bus->rxhdr, &rd_new,
 					       BRCMF_SDIO_FT_NORMAL)) {
 				rd->len = 0;
+				brcmf_sdio_rxfail(bus, true, true);
+				sdio_release_host(bus->sdiodev->func1);
 				brcmu_pkt_buf_free_skb(pkt);
 				continue;
 			}
Signature · 1 line
2.17.1
Reviewed-by on v1 1/1
#1 lore raw

Hi,

Quoted 33 lines · Guenter Roeck

With commit 216b44000ada ("brcmfmac: Fix use after free in brcmf_sdio_readframes()") applied, we see locking timeouts in brcmf_sdio_watchdog_thread().

brcmfmac: brcmf_escan_timeout: timer expired

INFO: task brcmf_wdog/mmc1:621 blocked for more than 120 seconds.

Not tainted 4.19.94-07984-g24ff99a0f713 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
brcmf_wdog/mmc1 D 0 621 2 0x00000000 last_sleep: 2440793077. last_runnable: 2440766827
[<c0aa1e60>] (__schedule) from [<c0aa2100>] (schedule+0x98/0xc4)
[<c0aa2100>] (schedule) from [<c0853830>] (__mmc_claim_host+0x154/0x274)
[<c0853830>] (__mmc_claim_host) from [<bf10c5b8>] (brcmf_sdio_watchdog_thread+0x1b0/0x1f8 [brcmfmac]) [<bf10c5b8>] (brcmf_sdio_watchdog_thread [brcmfmac]) from [<c02570b8>] (kthread+0x178/0x180)

In addition to restarting or exiting the loop, it is also necessary to abort the command and to release the host.

Fixes 216b44000ada ("brcmfmac: Fix use after free in brcmf_sdio_readframes()")
Cc Dan Carpenter <[email protected]>
Cc Matthias Kaehlcke <[email protected]>
Cc Brian Norris <[email protected]>
Cc Douglas Anderson <[email protected]>
---
 drivers/net/wireless/broadcom/brcm80211/brcmfmac/sdio.c | 2 ++
 1 file changed, 2 insertions(+)
drivers/net/wireless/broadcom/brcm80211/brcmfmac/sdio.c +0 -0
index f9df95bc7fa1..2e1c23c7269d 100644

drivers/net/wireless/broadcom/brcm80211/brcmfmac/sdio.c static uint brcmf_sdio_readframes(struct brcmf_sdio *bus, uint maxframes) · line 1942

Quoted 5 lines
drivers/net/wireless/broadcom/brcm80211/brcmfmac/sdio.c +2 -0
@@ -1938,6 +1938,8 @@ static uint brcmf_sdio_readframes(struct brcmf_sdio *bus, uint maxframes)
                        if (brcmf_sdio_hdparse(bus, bus->rxhdr, &rd_new,
                                               BRCMF_SDIO_FT_NORMAL)) {
                                rd->len = 0;
+                               brcmf_sdio_rxfail(bus, true, true);
+                               sdio_release_host(bus->sdiodev->func1);

I don't know much about this driver so I don't personally know if "true, true" is the correct thing to pass to brcmf_sdio_rxfail(), but it seems plausible. Definitely the fix to call sdio_release_host() is sane.

Thus, unless someone knows for sure that brcmf_sdio_rxfail()'s parameters should be different:

#2 lore raw
Quoted 47 lines · Doug Anderson

Hi,

Quoted 33 lines · Guenter Roeck

With commit 216b44000ada ("brcmfmac: Fix use after free in brcmf_sdio_readframes()") applied, we see locking timeouts in brcmf_sdio_watchdog_thread().

brcmfmac: brcmf_escan_timeout: timer expired

INFO: task brcmf_wdog/mmc1:621 blocked for more than 120 seconds.

Not tainted 4.19.94-07984-g24ff99a0f713 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
brcmf_wdog/mmc1 D 0 621 2 0x00000000 last_sleep: 2440793077. last_runnable: 2440766827
[<c0aa1e60>] (__schedule) from [<c0aa2100>] (schedule+0x98/0xc4)
[<c0aa2100>] (schedule) from [<c0853830>] (__mmc_claim_host+0x154/0x274)
[<c0853830>] (__mmc_claim_host) from [<bf10c5b8>] (brcmf_sdio_watchdog_thread+0x1b0/0x1f8 [brcmfmac]) [<bf10c5b8>] (brcmf_sdio_watchdog_thread [brcmfmac]) from [<c02570b8>] (kthread+0x178/0x180)

In addition to restarting or exiting the loop, it is also necessary to abort the command and to release the host.

Fixes 216b44000ada ("brcmfmac: Fix use after free in brcmf_sdio_readframes()")
Cc Dan Carpenter <[email protected]>
Cc Matthias Kaehlcke <[email protected]>
Cc Brian Norris <[email protected]>
Cc Douglas Anderson <[email protected]>
---
 drivers/net/wireless/broadcom/brcm80211/brcmfmac/sdio.c | 2 ++
 1 file changed, 2 insertions(+)
drivers/net/wireless/broadcom/brcm80211/brcmfmac/sdio.c +0 -0
index f9df95bc7fa1..2e1c23c7269d 100644

drivers/net/wireless/broadcom/brcm80211/brcmfmac/sdio.c static uint brcmf_sdio_readframes(struct brcmf_sdio *bus, uint maxframes) · line 1942

Quoted 5 lines
drivers/net/wireless/broadcom/brcm80211/brcmfmac/sdio.c +2 -0
@@ -1938,6 +1938,8 @@ static uint brcmf_sdio_readframes(struct brcmf_sdio *bus, uint maxframes)
                        if (brcmf_sdio_hdparse(bus, bus->rxhdr, &rd_new,
                                               BRCMF_SDIO_FT_NORMAL)) {
                                rd->len = 0;
+                               brcmf_sdio_rxfail(bus, true, true);
+                               sdio_release_host(bus->sdiodev->func1);

I don't know much about this driver so I don't personally know if "true, true" is the correct thing to pass to brcmf_sdio_rxfail(), but it seems plausible. Definitely the fix to call sdio_release_host() is sane.

Quoted 3 lines

Thus, unless someone knows for sure that brcmf_sdio_rxfail()'s parameters should be different:

Actually, looking at brcmf_sdio_hdparse() and its other callers, I think it may not be needed at all - other callers don't do it, and there already are some calls to brcmf_sdio_rxfail() in that function. It would be nice though to get a confirmation before I submit v2.

Guenter

Acked-by
#3 lore raw
Quoted 49 lines · Guenter Roeck
Quoted 47 lines · Doug Anderson

Hi,

Quoted 33 lines · Guenter Roeck

With commit 216b44000ada ("brcmfmac: Fix use after free in brcmf_sdio_readframes()") applied, we see locking timeouts in brcmf_sdio_watchdog_thread().

brcmfmac: brcmf_escan_timeout: timer expired

INFO: task brcmf_wdog/mmc1:621 blocked for more than 120 seconds.

Not tainted 4.19.94-07984-g24ff99a0f713 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
brcmf_wdog/mmc1 D 0 621 2 0x00000000 last_sleep: 2440793077. last_runnable: 2440766827
[<c0aa1e60>] (__schedule) from [<c0aa2100>] (schedule+0x98/0xc4)
[<c0aa2100>] (schedule) from [<c0853830>] (__mmc_claim_host+0x154/0x274)
[<c0853830>] (__mmc_claim_host) from [<bf10c5b8>] (brcmf_sdio_watchdog_thread+0x1b0/0x1f8 [brcmfmac]) [<bf10c5b8>] (brcmf_sdio_watchdog_thread [brcmfmac]) from [<c02570b8>] (kthread+0x178/0x180)

In addition to restarting or exiting the loop, it is also necessary to abort the command and to release the host.

Fixes 216b44000ada ("brcmfmac: Fix use after free in brcmf_sdio_readframes()")
Cc Dan Carpenter <[email protected]>
Cc Matthias Kaehlcke <[email protected]>
Cc Brian Norris <[email protected]>
Cc Douglas Anderson <[email protected]>
---
 drivers/net/wireless/broadcom/brcm80211/brcmfmac/sdio.c | 2 ++
 1 file changed, 2 insertions(+)
drivers/net/wireless/broadcom/brcm80211/brcmfmac/sdio.c +0 -0
index f9df95bc7fa1..2e1c23c7269d 100644

drivers/net/wireless/broadcom/brcm80211/brcmfmac/sdio.c static uint brcmf_sdio_readframes(struct brcmf_sdio *bus, uint maxframes) · line 1942

Quoted 5 lines
drivers/net/wireless/broadcom/brcm80211/brcmfmac/sdio.c +2 -0
@@ -1938,6 +1938,8 @@ static uint brcmf_sdio_readframes(struct brcmf_sdio *bus, uint maxframes)
                        if (brcmf_sdio_hdparse(bus, bus->rxhdr, &rd_new,
                                               BRCMF_SDIO_FT_NORMAL)) {
                                rd->len = 0;
+                               brcmf_sdio_rxfail(bus, true, true);
+                               sdio_release_host(bus->sdiodev->func1);

I don't know much about this driver so I don't personally know if "true, true" is the correct thing to pass to brcmf_sdio_rxfail(), but it seems plausible. Definitely the fix to call sdio_release_host() is sane.

Quoted 7 lines
Quoted 3 lines

Thus, unless someone knows for sure that brcmf_sdio_rxfail()'s parameters should be different:

Actually, looking at brcmf_sdio_hdparse() and its other callers, I think it may not be needed at all - other callers don't do it, and there already are some calls to brcmf_sdio_rxfail() in that function. It would be nice though to get a confirmation before I submit v2.

I think invoking rxfail with both abort and NACK set to true is the right thing to do here so that the pipeline can be properly purged.

Thanks!

#4 lore raw

Hi Franky,

[I'm very unfamiliar with this driver, but I had the same questions as Guenter, I think:]

Quoted 20 lines · Franky Lin
Quoted 19 lines · Guenter Roeck
Quoted 18 lines · Doug Anderson

drivers/net/wireless/broadcom/brcm80211/brcmfmac/sdio.c static uint brcmf_sdio_readframes(struct brcmf_sdio *bus, uint maxframes) · line 1942

Quoted 8 lines · Guenter Roeck
drivers/net/wireless/broadcom/brcm80211/brcmfmac/sdio.c +2 -0
@@ -1938,6 +1938,8 @@ static uint brcmf_sdio_readframes(struct brcmf_sdio *bus, uint maxframes)
                        if (brcmf_sdio_hdparse(bus, bus->rxhdr, &rd_new,
                                               BRCMF_SDIO_FT_NORMAL)) {
                                rd->len = 0;
+                               brcmf_sdio_rxfail(bus, true, true);
+                               sdio_release_host(bus->sdiodev->func1);

I don't know much about this driver so I don't personally know if "true, true" is the correct thing to pass to brcmf_sdio_rxfail(), but it seems plausible. Definitely the fix to call sdio_release_host() is sane.

Thus, unless someone knows for sure that brcmf_sdio_rxfail()'s parameters should be different:

Quoted 7 lines
Quoted 4 lines

Actually, looking at brcmf_sdio_hdparse() and its other callers, I think it may not be needed at all - other callers don't do it, and there already are some calls to brcmf_sdio_rxfail() in that function. It would be nice though to get a confirmation before I submit v2.

I think invoking rxfail with both abort and NACK set to true is the right thing to do here so that the pipeline can be properly purged.

Thanks for looking here. I'm not sure I totally understand your answer:
brcmf_sdio_hdparse() already calls brcmf_sdio_rxfail() in several error cases. Is it really OK to call it twice in a row?

Brian

#5 lore raw
Quoted 32 lines · Brian Norris

Hi Franky,

[I'm very unfamiliar with this driver, but I had the same questions as Guenter, I think:]

Quoted 25 lines · Franky Lin
Quoted 23 lines · Guenter Roeck
Quoted 15 lines · Doug Anderson

drivers/net/wireless/broadcom/brcm80211/brcmfmac/sdio.c static uint brcmf_sdio_readframes(struct brcmf_sdio *bus, uint maxframes) · line 1942

Quoted 8 lines · Guenter Roeck
drivers/net/wireless/broadcom/brcm80211/brcmfmac/sdio.c +2 -0
@@ -1938,6 +1938,8 @@ static uint brcmf_sdio_readframes(struct brcmf_sdio *bus, uint maxframes)
                        if (brcmf_sdio_hdparse(bus, bus->rxhdr, &rd_new,
                                               BRCMF_SDIO_FT_NORMAL)) {
                                rd->len = 0;
+                               brcmf_sdio_rxfail(bus, true, true);
+                               sdio_release_host(bus->sdiodev->func1);

I don't know much about this driver so I don't personally know if "true, true" is the correct thing to pass to brcmf_sdio_rxfail(), but it seems plausible. Definitely the fix to call sdio_release_host() is sane.

Quoted 3 lines

Thus, unless someone knows for sure that brcmf_sdio_rxfail()'s parameters should be different:

Actually, looking at brcmf_sdio_hdparse() and its other callers, I think it may not be needed at all - other callers don't do it, and there already are some calls to brcmf_sdio_rxfail() in that function. It would be nice though to get a confirmation before I submit v2.

Quoted 6 lines
Quoted 2 lines

I think invoking rxfail with both abort and NACK set to true is the right thing to do here so that the pipeline can be properly purged.

Thanks for looking here. I'm not sure I totally understand your answer:
brcmf_sdio_hdparse() already calls brcmf_sdio_rxfail() in several error cases. Is it really OK to call it twice in a row?

Yes. brcmf_sdio_rxglom does the same thing that calls brcmf_sdio_rxfail again in error handling. For this instance I think it's better using the same logic as the length mismatch block below ( calling brcmf_sdio_rxfail with true ture).

Thanks,
- Franky

Acked-by on v1 1/1
#6 lore raw
In reply to Guenter Roeck
Quoted 14 lines · Guenter Roeck

With commit 216b44000ada ("brcmfmac: Fix use after free in brcmf_sdio_readframes()") applied, we see locking timeouts in brcmf_sdio_watchdog_thread().

brcmfmac: brcmf_escan_timeout: timer expired

INFO: task brcmf_wdog/mmc1:621 blocked for more than 120 seconds.

Not tainted 4.19.94-07984-g24ff99a0f713 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
brcmf_wdog/mmc1 D 0 621 2 0x00000000 last_sleep: 2440793077. last_runnable: 2440766827
[<c0aa1e60>] (__schedule) from [<c0aa2100>] (schedule+0x98/0xc4)
[<c0aa2100>] (schedule) from [<c0853830>] (__mmc_claim_host+0x154/0x274)
[<c0853830>] (__mmc_claim_host) from [<bf10c5b8>] (brcmf_sdio_watchdog_thread+0x1b0/0x1f8 [brcmfmac]) [<bf10c5b8>] (brcmf_sdio_watchdog_thread [brcmfmac]) from [<c02570b8>] (kthread+0x178/0x180)

Quoted 4 lines

In addition to restarting or exiting the loop, it is also necessary to abort the command and to release the host.

Fixes 216b44000ada ("brcmfmac: Fix use after free in brcmf_sdio_readframes()")

Huh... Thanks for fixing the bug. That seems to indicate that we were triggering the use after free but no one noticed at runtime. With kfree(), a use after free can be harmless if you don't have poisoning enabled and no other thread has re-used the memory. I'm not sure about kfree_skb() but presumably it's the same.

regards,
dan carpenter

#7 lore raw
Quoted 15 lines · Dan Carpenter
Quoted 14 lines · Guenter Roeck

With commit 216b44000ada ("brcmfmac: Fix use after free in brcmf_sdio_readframes()") applied, we see locking timeouts in brcmf_sdio_watchdog_thread().

brcmfmac: brcmf_escan_timeout: timer expired

INFO: task brcmf_wdog/mmc1:621 blocked for more than 120 seconds.

Not tainted 4.19.94-07984-g24ff99a0f713 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
brcmf_wdog/mmc1 D 0 621 2 0x00000000 last_sleep: 2440793077. last_runnable: 2440766827
[<c0aa1e60>] (__schedule) from [<c0aa2100>] (schedule+0x98/0xc4)
[<c0aa2100>] (schedule) from [<c0853830>] (__mmc_claim_host+0x154/0x274)
[<c0853830>] (__mmc_claim_host) from [<bf10c5b8>] (brcmf_sdio_watchdog_thread+0x1b0/0x1f8 [brcmfmac]) [<bf10c5b8>] (brcmf_sdio_watchdog_thread [brcmfmac]) from [<c02570b8>] (kthread+0x178/0x180)

Quoted 7 lines
Quoted 4 lines

In addition to restarting or exiting the loop, it is also necessary to abort the command and to release the host.

Fixes 216b44000ada ("brcmfmac: Fix use after free in brcmf_sdio_readframes()")

Huh... Thanks for fixing the bug. That seems to indicate that we were triggering the use after free but no one noticed at runtime. With

Actually, we did see the problem. We just didn't realize it.

Quoted 4 lines

kfree(), a use after free can be harmless if you don't have poisoning enabled and no other thread has re-used the memory. I'm not sure about kfree_skb() but presumably it's the same.

Not really; it ultimately does result in a crash. We see that in ChromeOS R80 (and probably in all earlier releases, but I didn't check), which does not (yet) include 216b44000ada. The upcoming R81, which does include 216b44000ada, doesn't crash but there are lots of stalls like the one above. The combination of both (ie the difference in behavior) helped tracking down the problem.

Guenter

#8 lore raw
In reply to Guenter Roeck
Quoted 17 lines · Guenter Roeck

With commit 216b44000ada ("brcmfmac: Fix use after free in brcmf_sdio_readframes()") applied, we see locking timeouts in brcmf_sdio_watchdog_thread().

brcmfmac: brcmf_escan_timeout: timer expired

INFO: task brcmf_wdog/mmc1:621 blocked for more than 120 seconds.

Not tainted 4.19.94-07984-g24ff99a0f713 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
brcmf_wdog/mmc1 D 0 621 2 0x00000000 last_sleep: 2440793077. last_runnable: 2440766827
[<c0aa1e60>] (__schedule) from [<c0aa2100>] (schedule+0x98/0xc4)
[<c0aa2100>] (schedule) from [<c0853830>] (__mmc_claim_host+0x154/0x274)
[<c0853830>] (__mmc_claim_host) from [<bf10c5b8>] (brcmf_sdio_watchdog_thread+0x1b0/0x1f8 [brcmfmac]) [<bf10c5b8>] (brcmf_sdio_watchdog_thread [brcmfmac]) from [<c02570b8>] (kthread+0x178/0x180)

In addition to restarting or exiting the loop, it is also necessary to abort the command and to release the host.

Quoted 9 lines
Fixes 216b44000ada ("brcmfmac: Fix use after free in brcmf_sdio_readframes()")
Cc Dan Carpenter <[email protected]>
Cc Matthias Kaehlcke <[email protected]>
Cc Brian Norris <[email protected]>
Cc Douglas Anderson <[email protected]>

Patch applied to wireless-drivers-next.git, thanks.

863844ee3bd3 brcmfmac: abort and release host after error

Signature · 3 lines
https://patchwork.kernel.org/patch/11355277/

https://wireless.wiki.kernel.org/en/developers/documentation/submittingpatches