Skip to contentSkip to conversation

Firmware crash

DISCUSSION Sven Schnelle 3 participants 7 messages Jun 26, 2014 3 messages not in the archive Qualcomm Atheros ATH10K wireless driver

lore
Share & tools
lore
#0 lore raw

Hi List,

i've seen a strange firmware crash:

Athos/Xt BSP version Oct 19 2012-16:00:59
Baud rate host set is 115200.
Mac address is : 00:03:07:12:34:56
alloc rem: 197524
**WMI Service Ready **
NUM_DEV=16
TGT_VER: 4100016C
RegDomain: 0, CfgCtl: 224 - 224
ar_wal_peer_attach: num_peer_entries 145, num_tid_entries 443 allocated 64 tids at 0x439a08 (each 96 bytes) allocated 379 tids in IRAM at 0x9baeb4 (each 96 bytes) allocated 64 stateless tids at 0x43b230 (each 40 bytes) ### turn on Tx completion index tracking ###
**WMI Ready **
TGT_VER: 4100016C
RegDomain: 3A, CfgCtl: 16 - 16
peer create command for ****0:3:7:12:34:56**** _wlan_vdev_set_param: unimplemented command 0x2 _wlan_vdev_set_param: unimplemented command 0x7 _wlan_vdev_set_param: unimplemented command 0x8
TODO: Add DBGLOG for WMM Params
TODO: Add DBGLOG for WMM Params
TODO: Add DBGLOG for WMM Params
TODO: Add DBGLOG for WMM Params
TSF id: 1, free_tsf_id_map: 1
assertion failed? pc=0x9a0e15, line=0, dump area=0x401930 Target ID: 0x4100016c (1090519404)
Debug Info:
0x4100016c 0x00000000 0x009a0e15 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x009a0e15 0x00000000 0x00401930 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000

This happened on two APs at the same time, which were running before without problems. As both APs are in short distance (about 10m) i guess it was caused by some packet on the Air.

I see a 'assertion failed' above, can anyone tell what that assertion is?

Firmware Version is:

[   27.120000] ath10k: hardware name qca988x hw2.0 version 0x4100016c
[   27.130000] ath10k: firmware version: 10.1.467.2-1

Thanks
Sven

_______________________________________________
ath10k mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/ath10k

#1 lore raw
Quoted 39 lines · Sven Schnelle

Hi List,

i've seen a strange firmware crash:

Athos/Xt BSP version Oct 19 2012-16:00:59
Baud rate host set is 115200.
Mac address is : 00:03:07:12:34:56
alloc rem: 197524
**WMI Service Ready **
NUM_DEV=16
TGT_VER: 4100016C
RegDomain: 0, CfgCtl: 224 - 224
ar_wal_peer_attach: num_peer_entries 145, num_tid_entries 443 allocated 64 tids at 0x439a08 (each 96 bytes) allocated 379 tids in IRAM at 0x9baeb4 (each 96 bytes) allocated 64 stateless tids at 0x43b230 (each 40 bytes) ### turn on Tx completion index tracking ###
**WMI Ready **
TGT_VER: 4100016C
RegDomain: 3A, CfgCtl: 16 - 16
peer create command for ****0:3:7:12:34:56**** _wlan_vdev_set_param: unimplemented command 0x2 _wlan_vdev_set_param: unimplemented command 0x7 _wlan_vdev_set_param: unimplemented command 0x8
TODO: Add DBGLOG for WMM Params
TODO: Add DBGLOG for WMM Params
TODO: Add DBGLOG for WMM Params
TODO: Add DBGLOG for WMM Params
TSF id: 1, free_tsf_id_map: 1
assertion failed? pc=0x9a0e15, line=0, dump area=0x401930 Target ID: 0x4100016c (1090519404)
Debug Info:
0x4100016c 0x00000000 0x009a0e15 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x009a0e15 0x00000000 0x00401930 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000

Quoted 8 lines

0x00000000 0x00000000 0x00000000 0x00000000
0x00000000 0x00000000 0x00000000 0x00000000
0x00000000 0x00000000 0x00000000 0x00000000
0x00000000 0x00000000 0x00000000 0x00000000
0x00000000 0x00000000 0x00000000 0x00000000
0x00000000 0x00000000 0x00000000 0x00000000
0x00000000 0x00000000 0x00000000 0x00000000
0x00000000 0x00000000 0x00000000 0x00000000

Apparently firmware crashes while processing a beacon frame with a NULL dereference.

Quoted 11 lines

This happened on two APs at the same time, which were running before without problems. As both APs are in short distance (about 10m) i guess it was caused
by some packet on the Air.

I see a 'assertion failed' above, can anyone tell what that assertion is?

Firmware Version is:

[   27.120000] ath10k: hardware name qca988x hw2.0 version 0x4100016c
[   27.130000] ath10k: firmware version: 10.1.467.2-1

Did you happen to have the ath10k interface in a bridge? If so monitor vdev is most likely the culprit here. A very similar issue was reported for 4addr station bridging and also involves monitor vdev.

Michał

_______________________________________________
ath10k mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/ath10k

#2 lore raw
Quoted 20 lines · Michal Kazior
Quoted 11 lines

TSF id: 1, free_tsf_id_map: 1
assertion failed? pc=0x9a0e15, line=0, dump area=0x401930 Target ID: 0x4100016c (1090519404)
Debug Info:
0x4100016c 0x00000000 0x009a0e15 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x009a0e15 0x00000000 0x00401930 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000

Quoted 8 lines

0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000

Quoted 2 lines

Apparently firmware crashes while processing a beacon frame with a NULL dereference.

I fixed some issues in this area in my CT firmware, so you could try it if you want. If you can crash my firmware, please send me the similar crash dump and I can probably fix it.

Note this is not official QCA firmware, so any problems specific to this firmware should be directed at me, and not QCA, and when reporting general bugs (say, with driver), if you are using CT firmware be sure to note that.

http://www.candelatech.com/ath10k.php

Thanks,
Ben

Signature · 7 lines
Ben Greear <[email protected]>
Candela Technologies Inc  http://www.candelatech.com

_______________________________________________
ath10k mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/ath10k

A message that is not in the archive (sent off-list or not yet received): CAANdOW6yrv-Z_+bXw_z3zrmz=Q4N=r0xh9YitwwCoUAQchChLg@mail.gmail.com

#4 lore raw
Quoted 4 lines · Emanuel Taube

Hi Ben,

I am able to reproduce it with your firmware-2-ct-no-commercial-7.bin here is the crash dump:

Thanks. I am going to have to update my decode program to parse this...it doesn't look like 'normal' dmesg output on my systems. Should have it done shortly...

Ben

Quoted 84 lines

assertion failed? pc=0x9a1a49, line=0, dump area=0x401930 Target ID: 0x4100016c (1090519404)
Debug Info:
0x4100016c 0x00000000 0x009a1a49 0x00000000 0x00000000 0x00060324 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x009b58cb 0x0094085d 0x00000000 0x009a1a49 0x809430b8 0x00401a40 0x00000001 0x00000002 0x80940975 0x00401a60 0x0000001f 0x00403bec 0x409406b9 0x00401a80 0x0000001f 0x00419154 0x00000000 0x00401aa0 0x00050024 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000

to reproduce it I run 2 ath10k APs, one with the WIPHY_FLAG_IBSS_RSN enabled and one without. Whichever AP starts first keeps running while the other one crashes when trying to start hostapd.

Thanks,
Emanuel

2014-06-26 15:06 GMT+02:00 Ben Greear <[email protected] <mailto:[email protected]>>:

    On 06/26/2014 04:47 AM, Michal Kazior wrote:

            TSF id: 1, free_tsf_id_map: 1
            assertion failed? pc=0x9a0e15, line=0, dump area=0x401930
            Target ID: 0x4100016c (1090519404)
            Debug Info:
            0x4100016c 0x00000000 0x009a0e15 0x00000000
            0x00000000 0x00000000 0x00000000 0x00000000
            0x00000000 0x00000000 0x00000000 0x00000000
            0x00000000 0x00000000 0x00000000 0x00000000
            0x00000000 0x00000000 0x00000000 0x009a0e15
            0x00000000 0x00401930 0x00000000 0x00000000
            0x00000000 0x00000000 0x00000000 0x00000000
            0x00000000 0x00000000 0x00000000 0x00000000
            0x00000000 0x00000000 0x00000000 0x00000000
            0x00000000 0x00000000 0x00000000 0x00000000
            0x00000000 0x00000000 0x00000000 0x00000000
            0x00000000 0x00000000 0x00000000 0x00000000
            0x00000000 0x00000000 0x00000000 0x00000000
            0x00000000 0x00000000 0x00000000 0x00000000
            0x00000000 0x00000000 0x00000000 0x00000000


        Apparently firmware crashes while processing a beacon frame with a
        NULL dereference.


    I fixed some issues in this area in my CT firmware, so you could
    try it if you want.  If you can crash my firmware, please
    send me the similar crash dump and I can probably fix it.

    Note this is not official QCA firmware, so any problems specific to
    this firmware should be directed at me, and not QCA, and when reporting
    general bugs (say, with driver), if you are using CT firmware be sure to
    note that.

    http://www.candelatech.com/__ath10k.php <http://www.candelatech.com/ath10k.php>

    Thanks,
    Ben

    -- 
    Ben Greear <[email protected] <mailto:[email protected]>>
    Candela Technologies Inc  http://www.candelatech.com


    _________________________________________________
    ath10k mailing list
    [email protected] <mailto:[email protected]>
    http://lists.infradead.org/__mailman/listinfo/ath10k <http://lists.infradead.org/mailman/listinfo/ath10k>
Signature · 8 lines
Ben Greear <[email protected]>
Candela Technologies Inc  http://www.candelatech.com


_______________________________________________
ath10k mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/ath10k
#5 lore raw
In reply to a message not in the archive
Quoted 4 lines · Emanuel Taube

Hi Ben,

I am able to reproduce it with your firmware-2-ct-no-commercial-7.bin here is the crash dump:

Can you reproduce with this firmware? I have been making local changes, and I will get a better decode easier if you can re-run the test with this binary...

http://www.candelatech.com/downloads/firmware-2-community.bin

I fixed my tool to decode the bare dump, but dmesg output will probably be more convenient for the future (looks like you missed a line of the dump in the first email??)

Thanks,
Ben

Quoted 84 lines

assertion failed? pc=0x9a1a49, line=0, dump area=0x401930 Target ID: 0x4100016c (1090519404)
Debug Info:
0x4100016c 0x00000000 0x009a1a49 0x00000000 0x00000000 0x00060324 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x009b58cb 0x0094085d 0x00000000 0x009a1a49 0x809430b8 0x00401a40 0x00000001 0x00000002 0x80940975 0x00401a60 0x0000001f 0x00403bec 0x409406b9 0x00401a80 0x0000001f 0x00419154 0x00000000 0x00401aa0 0x00050024 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000

to reproduce it I run 2 ath10k APs, one with the WIPHY_FLAG_IBSS_RSN enabled and one without. Whichever AP starts first keeps running while the other one crashes when trying to start hostapd.

Thanks,
Emanuel

2014-06-26 15:06 GMT+02:00 Ben Greear <[email protected] <mailto:[email protected]>>:

    On 06/26/2014 04:47 AM, Michal Kazior wrote:

            TSF id: 1, free_tsf_id_map: 1
            assertion failed? pc=0x9a0e15, line=0, dump area=0x401930
            Target ID: 0x4100016c (1090519404)
            Debug Info:
            0x4100016c 0x00000000 0x009a0e15 0x00000000
            0x00000000 0x00000000 0x00000000 0x00000000
            0x00000000 0x00000000 0x00000000 0x00000000
            0x00000000 0x00000000 0x00000000 0x00000000
            0x00000000 0x00000000 0x00000000 0x009a0e15
            0x00000000 0x00401930 0x00000000 0x00000000
            0x00000000 0x00000000 0x00000000 0x00000000
            0x00000000 0x00000000 0x00000000 0x00000000
            0x00000000 0x00000000 0x00000000 0x00000000
            0x00000000 0x00000000 0x00000000 0x00000000
            0x00000000 0x00000000 0x00000000 0x00000000
            0x00000000 0x00000000 0x00000000 0x00000000
            0x00000000 0x00000000 0x00000000 0x00000000
            0x00000000 0x00000000 0x00000000 0x00000000
            0x00000000 0x00000000 0x00000000 0x00000000


        Apparently firmware crashes while processing a beacon frame with a
        NULL dereference.


    I fixed some issues in this area in my CT firmware, so you could
    try it if you want.  If you can crash my firmware, please
    send me the similar crash dump and I can probably fix it.

    Note this is not official QCA firmware, so any problems specific to
    this firmware should be directed at me, and not QCA, and when reporting
    general bugs (say, with driver), if you are using CT firmware be sure to
    note that.

    http://www.candelatech.com/__ath10k.php <http://www.candelatech.com/ath10k.php>

    Thanks,
    Ben

    -- 
    Ben Greear <[email protected] <mailto:[email protected]>>
    Candela Technologies Inc  http://www.candelatech.com


    _________________________________________________
    ath10k mailing list
    [email protected] <mailto:[email protected]>
    http://lists.infradead.org/__mailman/listinfo/ath10k <http://lists.infradead.org/mailman/listinfo/ath10k>
Signature · 8 lines
Ben Greear <[email protected]>
Candela Technologies Inc  http://www.candelatech.com


_______________________________________________
ath10k mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/ath10k

A message that is not in the archive (sent off-list or not yet received): CAANdOW7QyeNxJcG+kbXiUQZOXkF0ofZFh_RLbD6U1MjTyED+1Q@mail.gmail.com

#7 lore raw
Quoted 3 lines · Emanuel Taube

Hi Ben,

here is the dmesg output with your new community firmware:

Looks like null-dereference (or possibly some other bad memory dereference), not actually an assert (at least in my code).

Can you re-download the firmware (same location) and see if that fixes the problem?

Thanks,
Ben

Signature · 8 lines
Ben Greear <[email protected]>
Candela Technologies Inc  http://www.candelatech.com


_______________________________________________
ath10k mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/ath10k

A message that is not in the archive (sent off-list or not yet received): CAANdOW5bHeVqwqLVW7Ngr81mNJ6sa0tf3ojNDDFFzssrZTGvfw@mail.gmail.com

#9 lore raw
Quoted 3 lines · Emanuel Taube

Hi Ben,

your latest firmware fixes the crashes I was able to reproduce before.

Excellent, thanks for confirming. Looks like it was simple null-pointer deref. I'll roll this into my firmware, so next 'official' release will have the fix as well.

I'll send QCA a note off-list on what fixes it in case they want to put the fix upstream some day.

Thanks,
Ben

Quoted 25 lines

Thanks and regards,
Emanuel

2014-06-26 17:44 GMT+02:00 Ben Greear <[email protected] <mailto:[email protected]>>:

On 06/26/2014 08:39 AM, Emanuel Taube wrote:
> Hi Ben,
>
> here is the dmesg output with your new community firmware:

Looks like null-dereference (or possibly some other bad memory dereference), not actually an assert (at least in my code).

Can you re-download the firmware (same location) and see if that fixes the problem?

Thanks,
Ben

--
Ben Greear <[email protected] <mailto:[email protected]>> Candela Technologies Inc http://www.candelatech.com

Signature · 8 lines
Ben Greear <[email protected]>
Candela Technologies Inc  http://www.candelatech.com


_______________________________________________
ath10k mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/ath10k