Skip to contentSkip to conversation

Firmware debug messages stopped sometime after 3.18

DISCUSSION Ben Greear 1 participant 4 messages Jan 2, 2015 – Jan 6, 2015 Qualcomm Atheros ATH10K wireless driver

lore
Share & tools
lore

Ben Greear

author
#0 lore raw

While trying to reproduce some bugs on ath10k in 3.19.0-rc2, I could not get any debuglog messages from the firmware. For instance, in stock kernel it should print out something like this:

wmi event debug mesg len 28

I got other wmi debug messages, so the debug level appeared correct.

The message shows up as expected in stock 3.18 kernel.

Was this done on purpose?

I've started a bisect...should track it down directly.

Thanks,
Ben

Signature · 8 lines
Ben Greear <[email protected]>
Candela Technologies Inc  http://www.candelatech.com


_______________________________________________
ath10k mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/ath10k
author
#1 lore raw

Re: Firmware debug messages stopped sometime after 3.18 (bisected)

Quoted 6 lines · Ben Greear

While trying to reproduce some bugs on ath10k in 3.19.0-rc2, I could not get any debuglog messages from the firmware. For instance, in stock kernel it should print out something like this:

wmi event debug mesg len 28

Quoted 7 lines

I got other wmi debug messages, so the debug level appeared correct.

The message shows up as expected in stock 3.18 kernel.

Was this done on purpose?

I've started a bisect...should track it down directly.

Took a while, but bisect points to this. If you are doing bisect yourself, or playing with changes near here, you must manually apply the patch below to allow scan to work. I'll try to figure out why the patch below disables firmware debuglog messages next. I assume that was not the intention?

90174455ae058a1dcf1f3d344a9752385803c1d8 is the first bad commit commit 90174455ae058a1dcf1f3d344a9752385803c1d8

Author: Rajkumar Manoharan <[email protected]>
Date:   Fri Oct 3 08:02:33 2014 +0300

ath10k: add support to configure pktlog filter

Add support to configure packet log filters (tx, rx, rate control) via debugfs. To disable htt pktlog events set the filters to 0.

ex:

To enable pktlog for all filters

echo 0x1f > /sys/kernel/debug/ieee80211/phy*/ath10k/pktlog_filter

To disable pktlog

echo 0 > /sys/kernel/debug/ieee80211/phy*/ath10k/pktlog_filter

:040000 040000 1ff3b609db06d718fb22a7d8444a70e38a60ebcd 6504c0828edb532f19826167eab9ba0634b3f044 M drivers

Patch to allow scan to work (this patch made it upstream a bit later):

[greearb@ben-dt2 linux-bisect]$ git diff

drivers/net/wireless/ath/ath10k/wmi.c +2 -2
index f65032f..d878fbe 100644
@@ -3506,10 +3506,10 @@ int ath10k_wmi_start_scan(struct ath10k *ar,
                return ret;
 
        if (test_bit(ATH10K_FW_FEATURE_WMI_10X, ar->fw_features))
-               len = sizeof(struct wmi_start_scan_cmd) +
+               len = sizeof(struct wmi_10x_start_scan_cmd) +
                      ath10k_wmi_start_scan_tlvs_len(arg);
        else
-               len = sizeof(struct wmi_10x_start_scan_cmd) +
+               len = sizeof(struct wmi_start_scan_cmd) +
                      ath10k_wmi_start_scan_tlvs_len(arg);
 
        skb = ath10k_wmi_alloc_skb(ar, len);

[greearb@ben-dt2 linux-bisect]$

Thanks,
Ben

Quoted 4 lines

Thanks,
Ben

Signature · 8 lines
Ben Greear <[email protected]>
Candela Technologies Inc  http://www.candelatech.com


_______________________________________________
ath10k mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/ath10k
author
#2 lore raw

Re: Firmware debug messages stopped sometime after 3.18 (bisected)

I think I found another problem with this patch.

When user enables ATH10K_PKTLOG_DBG_PRINT pktlog filter, then the firmware can send pkts to the CE1 pipe, which has only 512 byte message length. firmware debuglog messages can be larger than that, and so (10.1.467, at least) firmware will crash.

If anyone with QCA firmware source rights wants the nasty details, contact me offline.

I will try to hack around this in my firmware, but I think this patch needs some more thought and/or fixes.

Thanks,
Ben

Quoted 79 lines · Ben Greear
Quoted 6 lines · Ben Greear

While trying to reproduce some bugs on ath10k in 3.19.0-rc2, I could not get any debuglog messages from the firmware. For instance, in stock kernel it should print out something like this:

wmi event debug mesg len 28

Quoted 7 lines

I got other wmi debug messages, so the debug level appeared correct.

The message shows up as expected in stock 3.18 kernel.

Was this done on purpose?

I've started a bisect...should track it down directly.

Took a while, but bisect points to this. If you are doing bisect yourself, or playing with changes near here, you must manually apply the patch below to allow scan to work. I'll try to figure out why the patch below disables firmware debuglog messages next. I assume that was not the intention?

90174455ae058a1dcf1f3d344a9752385803c1d8 is the first bad commit commit 90174455ae058a1dcf1f3d344a9752385803c1d8

Author: Rajkumar Manoharan <[email protected]>
Date:   Fri Oct 3 08:02:33 2014 +0300

ath10k: add support to configure pktlog filter

Add support to configure packet log filters (tx, rx, rate control) via debugfs. To disable htt pktlog events set the filters to 0.

ex:

To enable pktlog for all filters

echo 0x1f > /sys/kernel/debug/ieee80211/phy*/ath10k/pktlog_filter

To disable pktlog

echo 0 > /sys/kernel/debug/ieee80211/phy*/ath10k/pktlog_filter

:040000 040000 1ff3b609db06d718fb22a7d8444a70e38a60ebcd 6504c0828edb532f19826167eab9ba0634b3f044 M drivers

Patch to allow scan to work (this patch made it upstream a bit later):

[greearb@ben-dt2 linux-bisect]$ git diff

drivers/net/wireless/ath/ath10k/wmi.c +2 -2
index f65032f..d878fbe 100644
@@ -3506,10 +3506,10 @@ int ath10k_wmi_start_scan(struct ath10k *ar,
                return ret;
 
        if (test_bit(ATH10K_FW_FEATURE_WMI_10X, ar->fw_features))
-               len = sizeof(struct wmi_start_scan_cmd) +
+               len = sizeof(struct wmi_10x_start_scan_cmd) +
                      ath10k_wmi_start_scan_tlvs_len(arg);
        else
-               len = sizeof(struct wmi_10x_start_scan_cmd) +
+               len = sizeof(struct wmi_start_scan_cmd) +
                      ath10k_wmi_start_scan_tlvs_len(arg);
 
        skb = ath10k_wmi_alloc_skb(ar, len);

[greearb@ben-dt2 linux-bisect]$

Thanks,
Ben

Quoted 4 lines

Thanks,
Ben

Signature · 8 lines
Ben Greear <[email protected]>
Candela Technologies Inc  http://www.candelatech.com


_______________________________________________
ath10k mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/ath10k
author
#3 lore raw

Re: Firmware debug messages stopped sometime after 3.18 (bisected)

Quoted 11 lines · Ben Greear

I think I found another problem with this patch.

When user enables ATH10K_PKTLOG_DBG_PRINT pktlog filter, then the firmware can send pkts to the CE1 pipe, which has only 512 byte message length. firmware debuglog messages can be larger than that, and so (10.1.467, at least) firmware will crash.

If anyone with QCA firmware source rights wants the nasty details, contact me offline.

I will try to hack around this in my firmware, but I think this patch needs some more thought and/or fixes.

So, root cause of not getting dbglog over wmi after this patch is bug in 10.1.467 (and maybe other) firmware in it's dbglog code.

I have fixed that in CT firmware, and disabled pktlog ability to send dbglog messages to work around that crash scenario, and now things are working better again.

Upstream firmware is still going to have issues with this patch, however.

Thanks,
Ben

Quoted 86 lines

Thanks,
Ben

Quoted 79 lines · Ben Greear
Quoted 6 lines · Ben Greear

While trying to reproduce some bugs on ath10k in 3.19.0-rc2, I could not get any debuglog messages from the firmware. For instance, in stock kernel it should print out something like this:

wmi event debug mesg len 28

Quoted 7 lines

I got other wmi debug messages, so the debug level appeared correct.

The message shows up as expected in stock 3.18 kernel.

Was this done on purpose?

I've started a bisect...should track it down directly.

Took a while, but bisect points to this. If you are doing bisect yourself, or playing with changes near here, you must manually apply the patch below to allow scan to work. I'll try to figure out why the patch below disables firmware debuglog messages next. I assume that was not the intention?

90174455ae058a1dcf1f3d344a9752385803c1d8 is the first bad commit commit 90174455ae058a1dcf1f3d344a9752385803c1d8

Author: Rajkumar Manoharan <[email protected]>
Date:   Fri Oct 3 08:02:33 2014 +0300

ath10k: add support to configure pktlog filter

Add support to configure packet log filters (tx, rx, rate control) via debugfs. To disable htt pktlog events set the filters to 0.

ex:

To enable pktlog for all filters

echo 0x1f > /sys/kernel/debug/ieee80211/phy*/ath10k/pktlog_filter

To disable pktlog

echo 0 > /sys/kernel/debug/ieee80211/phy*/ath10k/pktlog_filter

:040000 040000 1ff3b609db06d718fb22a7d8444a70e38a60ebcd 6504c0828edb532f19826167eab9ba0634b3f044 M drivers

Patch to allow scan to work (this patch made it upstream a bit later):

[greearb@ben-dt2 linux-bisect]$ git diff

drivers/net/wireless/ath/ath10k/wmi.c +2 -2
index f65032f..d878fbe 100644
@@ -3506,10 +3506,10 @@ int ath10k_wmi_start_scan(struct ath10k *ar,
                return ret;
 
        if (test_bit(ATH10K_FW_FEATURE_WMI_10X, ar->fw_features))
-               len = sizeof(struct wmi_start_scan_cmd) +
+               len = sizeof(struct wmi_10x_start_scan_cmd) +
                      ath10k_wmi_start_scan_tlvs_len(arg);
        else
-               len = sizeof(struct wmi_10x_start_scan_cmd) +
+               len = sizeof(struct wmi_start_scan_cmd) +
                      ath10k_wmi_start_scan_tlvs_len(arg);
 
        skb = ath10k_wmi_alloc_skb(ar, len);

[greearb@ben-dt2 linux-bisect]$

Thanks,
Ben

Quoted 4 lines

Thanks,
Ben

Signature · 8 lines
Ben Greear <[email protected]>
Candela Technologies Inc  http://www.candelatech.com


_______________________________________________
ath10k mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/ath10k