Skip to contentSkip to conversation

Reproducible issue in hacked 3.17 kernel, CT firmware

DISCUSSION Ben Greear 2 participants 4 messages Dec 30, 2014 – Jan 7, 2015 Qualcomm Atheros ATH10K wireless driver

lore
Share & tools
lore

Ben Greear

author
#0 lore raw

yeah, so maybe not reproducible upstream, but anyway...

My test case is to re-associate 4 stations over and over again, with a scan and a 5 second sleep between iterations. After
a short time, something goes weird and OS is mostly hung, probably because important locks are held while ath10k is timing out communication to firmware.

The last message I see from firmware is that it is deleting vdev 4.

I do not see any indication that firmware is crashed, but something is wrong, maybe mgt buffers are used up?

I'm going to debug this further, but I am curious why the logs appear to show that we continue sending cmds (cts_prot, for example) after the vdev is configured down?

[  339.958906] sta104: deauthenticating from 04:f0:21:03:38:99 by local choice (Reason: 2=PREV_AUTH_NOT_VALID)
[  339.958918] ath10k_pci 0000:04:00.0: mac ampdu vdev_id 4 sta 04:f0:21:03:38:99 tid 0 action 1
[  339.958968] ath10k_pci 0000:04:00.0: wmi vdev install key idx 0 cipher 0 len 16
[  339.959076] ath10k_pci 0000:04:00.0: mac vdev 4 peer delete 04:f0:21:03:38:99 (sta gone)
[  339.959080] ath10k_pci 0000:04:00.0: wmi peer delete vdev_id 4 peer_addr 04:f0:21:03:38:99
[  339.959287] ath10k_pci 0000:04:00.0: mac vdev 4 stop (disassociated
[  339.959290] ath10k_pci 0000:04:00.0: wmi vdev stop id 0x4
[  339.959387] ath10k_pci 0000:04:00.0: WMI_VDEV_STOPPED_EVENTID
[  339.959405] ath10k_pci 0000:04:00.0: mac vdev 4 down
[  339.959407] ath10k_pci 0000:04:00.0: wmi mgmt vdev down id 0x4
[  339.959491] ath10k_pci 0000:04:00.0: mac vdev 4 cts_prot 0
[  339.959495] ath10k_pci 0000:04:00.0: wmi vdev id 0x4 set param 43 value 0
[  339.959499] ath10k_pci 0000:04:00.0: mac vdev 4 slot_time 1
[  339.959501] ath10k_pci 0000:04:00.0: wmi vdev id 0x4 set param 7 value 1
[  340.104623] ath10k_pci 0000:04:00.0: wmi event debug mesg len 144
[  340.104645] ATH10K_DBG_BUFFER:
[  340.104654] ath10k: [0000]: 00059396 100C2403 00000001 00000001 0043AF90 00000000 00059436 04104C1C
[  340.104662] ath10k: [0008]: 00000000 00059437 14105C0A 009B83F4 0043874C 009B83F4 009B81D4 009B83F4
[  340.104670] ath10k: [0016]: 00059437 0C104C30 711000BB 00422368 0043874C 00059437 0C104C30 711000BB
[  340.104684] ath10k: [0024]: 004223A0 0043874C 00059437 0C104C30 711000BB 004223D8 0043874C 00059437
[  340.104687] ath10k: [0032]: 08104C08 00000001 00000008
[  340.104688] ATH10K_END
[  342.962494] ath10k_pci 0000:04:00.0: failed to set erp slot for vdev 4: -11
[  342.962509] ath10k_pci 0000:04:00.0: mac vdev 4 preamble 1n
[  342.962512] ath10k_pci 0000:04:00.0: wmi vdev id 0x4 set param 8 value 1
[  345.965900] ath10k_pci 0000:04:00.0: failed to set preamble for vdev 4: -11
[  345.965916] ath10k_pci 0000:04:00.0: wmi pdev set wmm params
[  348.969287] ath10k_pci 0000:04:00.0: failed to set wmm params: -11
[  348.969307] ath10k_pci 0000:04:00.0: wmi pdev set wmm params
[  351.972696] ath10k_pci 0000:04:00.0: failed to set wmm params: -11
[  351.972713] ath10k_pci 0000:04:00.0: wmi pdev set wmm params

Thanks,
Ben

Signature · 8 lines
Ben Greear <[email protected]>
Candela Technologies Inc  http://www.candelatech.com


_______________________________________________
ath10k mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/ath10k
#1 lore raw
Quoted 12 lines · Ben Greear

yeah, so maybe not reproducible upstream, but anyway...

My test case is to re-associate 4 stations over and over again, with a scan and a 5 second sleep between iterations. After
a short time, something goes weird and OS is mostly hung, probably because important locks are held while ath10k is timing out communication to firmware.

The last message I see from firmware is that it is deleting vdev 4.

I do not see any indication that firmware is crashed, but something is wrong, maybe mgt buffers are used up?

[...]

Quoted 1 line
[  342.962494] ath10k_pci 0000:04:00.0: failed to set erp slot for vdev 4: -11

-11 = -EAGAIN = out of wmi-htc tx credits. I wonder what the dbg buffer is trying to say.

Either host sent a corrupted message and clogged up firmware buffers, firmware is busy processing other commands (wmi mgmt tx, wmi bcn non-dma tx) or became confused/corrupted.

Quoted 3 lines

I'm going to debug this further, but I am curious why the logs appear to show that we continue sending cmds (cts_prot, for example) after the vdev is configured down?

This is implied by mac80211. See ieee80211_set_disassoc(): it calls sta_info_flush() then ieee80211_reset_erp_info() and later ieee80211_bss_info_change_notify(). These yield ath10k_bss_disassoc() and later ath10k_bss_info_changed() respectively.

Michał

_______________________________________________
ath10k mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/ath10k

author
#2 lore raw
Quoted 14 lines · Michal Kazior
Quoted 12 lines · Ben Greear

yeah, so maybe not reproducible upstream, but anyway...

My test case is to re-associate 4 stations over and over again, with a scan and a 5 second sleep between iterations. After
a short time, something goes weird and OS is mostly hung, probably because important locks are held while ath10k is timing out communication to firmware.

The last message I see from firmware is that it is deleting vdev 4.

I do not see any indication that firmware is crashed, but something is wrong, maybe mgt buffers are used up?

[...]

Quoted 8 lines
Quoted 1 line
[  342.962494] ath10k_pci 0000:04:00.0: failed to set erp slot for vdev 4: -11

-11 = -EAGAIN = out of wmi-htc tx credits. I wonder what the dbg buffer is trying to say.

Either host sent a corrupted message and clogged up firmware buffers, firmware is busy processing other commands (wmi mgmt tx, wmi bcn non-dma tx) or became confused/corrupted.

I finally got back to debugging this yesterday, and interestingly, when
I added dbglog calls in the firmware around the credit handling, the problem is 'fixed'.

Looks like it ran overnight, where as before it would fail within a few minutes.

So, maybe a race around pci memory flushing or something like that?

I'll slowly back out my debug today and see what I can see.

Thanks,
Ben

Signature · 7 lines
Ben Greear <[email protected]>
Candela Technologies Inc  http://www.candelatech.com

_______________________________________________
ath10k mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/ath10k
author
#3 lore raw
Quoted 26 lines · Ben Greear
Quoted 14 lines · Michal Kazior
Quoted 12 lines · Ben Greear

yeah, so maybe not reproducible upstream, but anyway...

My test case is to re-associate 4 stations over and over again, with a scan and a 5 second sleep between iterations. After
a short time, something goes weird and OS is mostly hung, probably because important locks are held while ath10k is timing out communication to firmware.

The last message I see from firmware is that it is deleting vdev 4.

I do not see any indication that firmware is crashed, but something is wrong, maybe mgt buffers are used up?

[...]

Quoted 8 lines
Quoted 1 line
[  342.962494] ath10k_pci 0000:04:00.0: failed to set erp slot for vdev 4: -11

-11 = -EAGAIN = out of wmi-htc tx credits. I wonder what the dbg buffer is trying to say.

Either host sent a corrupted message and clogged up firmware buffers, firmware is busy processing other commands (wmi mgmt tx, wmi bcn non-dma tx) or became confused/corrupted.

Quoted 8 lines

I finally got back to debugging this yesterday, and interestingly, when
I added dbglog calls in the firmware around the credit handling, the problem is 'fixed'.

Looks like it ran overnight, where as before it would fail within a few minutes.

So, maybe a race around pci memory flushing or something like that?

I'll slowly back out my debug today and see what I can see.

It finally locked up this morning...I see last credit consumed at 8:37:02, and then finally I get two credits from the firmware at 9:12:42.

I guess more instrumentation is required :P

Thanks,
Ben

Signature · 8 lines
Ben Greear <[email protected]>
Candela Technologies Inc  http://www.candelatech.com


_______________________________________________
ath10k mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/ath10k