wifi: ath12k: start-up crash with WCN7850 hw2.0 on TI AM69-SK board
Hi All,
I am performing tests on the SX-PCEBE Wi-Fi module, which utilizes the ATH12k driver, on the Texas Instruments AM69-SK board.
The board is running the TI Linux Kernel from the ti-linux-6.6.y branch. During testing, I observed a kernel crash from the ATH12k driver as soon as the probe is called. The crash log is as follows:
[ 9.492631] Kernel panic - not syncing: Asynchronous SError Interrupt [ 9.492634] CPU: 7 PID: 222 Comm: (udev-worker) Not tainted 6.6.58- 01497-ga7758da17c28-dirty #1 [ 9.492638] Hardware name: Texas Instruments AM69 SK (DT) [ 9.492640] Call trace: [ 9.492642] dump_backtrace+0x94/0xec [ 9.492658] show_stack+0x18/0x24 [ 9.492662] dump_stack_lvl+0x48/0x60 [ 9.492669] dump_stack+0x18/0x24 [ 9.492672] panic+0x320/0x378 [ 9.492677] nmi_panic+0x8c/0x90 [ 9.492681] arm64_serror_panic+0x6c/0x78 [ 9.492686] do_serror+0x3c/0x78 [ 9.492692] el1h_64_error_handler+0x34/0x4c [ 9.492697] el1h_64_error+0x64/0x68 [ 9.492700] ath12k_pci_read32+0x1bc/0x1e8 [ath12k] [ 9.492725] ath12k_pci_power_up+0xdc/0x340 [ath12k] [ 9.492747] ath12k_core_init+0x2c/0xa8 [ath12k] [ 9.492769] ath12k_pci_probe+0x698/0x908 [ath12k] [ 9.492791] pci_device_probe+0xa8/0x16c [ 9.492800] really_probe+0x110/0x27c [ 9.492805] __driver_probe_device+0x78/0x12c [ 9.492808] driver_probe_device+0x3c/0x118 [ 9.492810] __driver_attach+0x74/0x124 [ 9.492813] bus_for_each_dev+0x78/0xd8 [ 9.492819] driver_attach+0x24/0x30 [ 9.492824] bus_add_driver+0xe4/0x208 [ 9.492828] driver_register+0x60/0x128 [ 9.492831] __pci_register_driver+0x44/0x50 [ 9.492835] ath12k_pci_init+0x2c/0x6c [ath12k] [ 9.492858] do_one_initcall+0x70/0x1b4 [ 9.492861] do_init_module+0x58/0x1e4 [ 9.492867] load_module+0x19bc/0x1a8c [ 9.492869] init_module_from_file+0x88/0xc4 [ 9.492873] __arm64_sys_finit_module+0x1c0/0x2ac [ 9.492877] invoke_syscall+0x44/0x108 [ 9.492882] el0_svc_common.constprop.0+0xc0/0xe0 [ 9.492885] do_el0_svc+0x1c/0x28 [ 9.492889] el0_svc+0x2c/0x84 [ 9.492892] el0t_64_sync_handler+0xc0/0xc4 [ 9.492895] el0t_64_sync+0x190/0x194 [ 9.492899] SMP: stopping secondary CPUs [ 9.492908] Kernel Offset: disabled [ 9.492909] CPU features: 0x0,80000200,28020000,1000420b [ 9.492913] Memory Limit: none
Upon searching online, I found the OpenWRT patch that appears to address a similar issue: OpenWRT Patch: Prevent LTSSM Startup Crash.
https://git.openwrt.org/?p=openwrt/openwrt.git;a=blob;f=package/kernel/mac80211/patches/ath12k/100-ath12k-prevent-ltssm-startup-crash.patch;h=cd85a0f6aa2652d62bfbea04e9bcca3bcf831b7f;hb=935b2b7dcef61b2893ed5dff307dd8f8a1156899
With the above patch applied, I do not see the crash anymore.
Could anyone confirm if this issue has been reported before/known bug or provide any insights?
Any additional information or suggestions would be greatly appreciated.
Details about the test setup,
TI-AM69-SK board:
https://www.ti.com/tool/SK-AM69?keyMatch=am69%20sk&tisearch=universal_search
Silex WiFi card SX-PCEBE:
https://www.silextechnology.com/connectivity-solutions/embedded-wireless/sx-pcebe
TI Linux Repo:
https://git.ti.com/cgit/ti-linux-kernel/ti-linux-kernel/?h=ti-linux-6.6.y
Thank you.
Regards,
Parth P
Quoted 5 lines · Parth Panchoil
Hi All,
I am performing tests on the SX-PCEBE Wi-Fi module, which utilizes the ATH12k driver, on the Texas Instruments AM69-SK board.
The board is running the TI Linux Kernel from the ti-linux-6.6.y
6.6 is too old, and besides we don;t support customer kernel.
Could you try latest ath tree [1] or the mainline tree [2]?
[1] https://git.kernel.org/pub/scm/linux/kernel/git/ath/ath.git/
[2] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/
If the issue is still seen, please enable verbose ath12k log using below command and help collect dmesg logs:
sudo modprobe ath12k debug_mask=0xffffffff
One more thing, the open-WRT patch is overkill, can you narrow down to find which line of code in ath12k_pci_enable_ltssm() is causing this issue?
Quoted 72 lines
branch. During testing, I observed a kernel crash from the ATH12k driver as soon as the probe is called. The crash log is as follows:
[ 9.492631] Kernel panic - not syncing: Asynchronous SError Interrupt [ 9.492634] CPU: 7 PID: 222 Comm: (udev-worker) Not tainted 6.6.58- 01497-ga7758da17c28-dirty #1 [ 9.492638] Hardware name: Texas Instruments AM69 SK (DT) [ 9.492640] Call trace: [ 9.492642] dump_backtrace+0x94/0xec [ 9.492658] show_stack+0x18/0x24 [ 9.492662] dump_stack_lvl+0x48/0x60 [ 9.492669] dump_stack+0x18/0x24 [ 9.492672] panic+0x320/0x378 [ 9.492677] nmi_panic+0x8c/0x90 [ 9.492681] arm64_serror_panic+0x6c/0x78 [ 9.492686] do_serror+0x3c/0x78 [ 9.492692] el1h_64_error_handler+0x34/0x4c [ 9.492697] el1h_64_error+0x64/0x68 [ 9.492700] ath12k_pci_read32+0x1bc/0x1e8 [ath12k] [ 9.492725] ath12k_pci_power_up+0xdc/0x340 [ath12k] [ 9.492747] ath12k_core_init+0x2c/0xa8 [ath12k] [ 9.492769] ath12k_pci_probe+0x698/0x908 [ath12k] [ 9.492791] pci_device_probe+0xa8/0x16c [ 9.492800] really_probe+0x110/0x27c [ 9.492805] __driver_probe_device+0x78/0x12c [ 9.492808] driver_probe_device+0x3c/0x118 [ 9.492810] __driver_attach+0x74/0x124 [ 9.492813] bus_for_each_dev+0x78/0xd8 [ 9.492819] driver_attach+0x24/0x30 [ 9.492824] bus_add_driver+0xe4/0x208 [ 9.492828] driver_register+0x60/0x128 [ 9.492831] __pci_register_driver+0x44/0x50 [ 9.492835] ath12k_pci_init+0x2c/0x6c [ath12k] [ 9.492858] do_one_initcall+0x70/0x1b4 [ 9.492861] do_init_module+0x58/0x1e4 [ 9.492867] load_module+0x19bc/0x1a8c [ 9.492869] init_module_from_file+0x88/0xc4 [ 9.492873] __arm64_sys_finit_module+0x1c0/0x2ac [ 9.492877] invoke_syscall+0x44/0x108 [ 9.492882] el0_svc_common.constprop.0+0xc0/0xe0 [ 9.492885] do_el0_svc+0x1c/0x28 [ 9.492889] el0_svc+0x2c/0x84 [ 9.492892] el0t_64_sync_handler+0xc0/0xc4 [ 9.492895] el0t_64_sync+0x190/0x194 [ 9.492899] SMP: stopping secondary CPUs [ 9.492908] Kernel Offset: disabled [ 9.492909] CPU features: 0x0,80000200,28020000,1000420b [ 9.492913] Memory Limit: noneUpon searching online, I found the OpenWRT patch that appears to address a similar issue: OpenWRT Patch: Prevent LTSSM Startup Crash.
https://git.openwrt.org/?p=openwrt/openwrt.git;a=blob;f=package/kernel/mac80211/patches/ath12k/100-ath12k-prevent-ltssm-startup-crash.patch;h=cd85a0f6aa2652d62bfbea04e9bcca3bcf831b7f;hb=935b2b7dcef61b2893ed5dff307dd8f8a1156899
With the above patch applied, I do not see the crash anymore.Could anyone confirm if this issue has been reported before/known bug or provide any insights?
Any additional information or suggestions would be greatly appreciated.Details about the test setup,
TI-AM69-SK board:
https://www.ti.com/tool/SK-AM69?keyMatch=am69%20sk&tisearch=universal_search
Silex WiFi card SX-PCEBE:
https://www.silextechnology.com/connectivity-solutions/embedded-wireless/sx-pcebe
TI Linux Repo:
https://git.ti.com/cgit/ti-linux-kernel/ti-linux-kernel/?h=ti-linux-6.6.yThank you.
Regards,
Parth P
I appreciate your response, Baochen.
I have been working on enabling mainline kernel support on my TI AM69- SK board to test the mainline ath12k driver on my system.
Using the mainline kernel repository for the ath drivers [1], I made the following observation:
While the exact crash observed earlier is no longer present, the system hangs upon loading the ath12k mainline driver, displaying the messages below.
root@am69-sk:~# modprobe ath12k debug_mask=0xffffffff [ 1121.996554] ath12k_pci 0000:01:00.0: BAR 0 [mem 0x4410200000- 0x44103fffff 64bit]: assigned [ 1122.004884] ath12k_pci 0000:01:00.0: enabling device (0000 -> 0002) [ 1122.011818] ath12k_pci 0000:01:00.0: MSI vectors: 16 [ 1122.016798] ath12k_pci 0000:01:00.0: Hardware name: wcn7850 hw2.0 [ 1122.040183] NET: Registered PF_QIPCRTR protocol family
root@am69-sk:~# uname -a Linux am69-sk 6.13.0-rc7-wt-ath-ge7ef944b3e2c-dirty #2 SMP PREEMPT Wed Jan 22 16:55:17 CET 2025 aarch64 GNU/Linux
root@am69-sk:~# lspci 0000:00:00.0 PCI bridge: Texas Instruments Device b012 0000:01:00.0 Network controller: Qualcomm Technologies, Inc WCN785x Wi- Fi 7(802.11be) 320MHz 2x2 [FastConnect 7800] (rev 01) 0001:00:00.0 PCI bridge: Texas Instruments Device b012 0002:00:00.0 PCI bridge: Texas Instruments Device b012
Do you have any insights into what might still be missing or incorrect in my setup?
Regards,
Parth P
Quoted 107 lines · Baochen Qiang
Quoted 6 lines · Parth Panchoil
Hi All,
I am performing tests on the SX-PCEBE Wi-Fi module, which utilizes the
ATH12k driver, on the Texas Instruments AM69-SK board. The board is running the TI Linux Kernel from the ti-linux-6.6.y6.6 is too old, and besides we don;t support customer kernel.
Could you try latest ath tree [1] or the mainline tree [2]?
[1] https://git.kernel.org/pub/scm/linux/kernel/git/ath/ath.git/
[2]
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/If the issue is still seen, please enable verbose ath12k log using below command and help
collect dmesg logs:sudo modprobe ath12k debug_mask=0xffffffff
One more thing, the open-WRT patch is overkill, can you narrow down to find which line of
code inath12k_pci_enable_ltssm()is causing this issue?Quoted 76 lines
branch. During testing, I observed a kernel crash from the ATH12k driver as soon as the probe is called. The crash log is as follows:
[ 9.492631] Kernel panic - not syncing: Asynchronous SError Interrupt
[ 9.492634] CPU: 7 PID: 222 Comm: (udev-worker) Not tainted 6.6.58-
01497-ga7758da17c28-dirty #1
[ 9.492638] Hardware name: Texas Instruments AM69 SK (DT) [ 9.492640] Call trace:
[ 9.492642]dump_backtrace+0x94/0xec
[ 9.492658]show_stack+0x18/0x24
[ 9.492662]dump_stack_lvl+0x48/0x60
[ 9.492669]dump_stack+0x18/0x24
[ 9.492672]panic+0x320/0x378
[ 9.492677]nmi_panic+0x8c/0x90
[ 9.492681]arm64_serror_panic+0x6c/0x78
[ 9.492686]do_serror+0x3c/0x78
[ 9.492692]el1h_64_error_handler+0x34/0x4c
[ 9.492697]el1h_64_error+0x64/0x68
[ 9.492700]ath12k_pci_read32+0x1bc/0x1e8[ath12k] [ 9.492725]ath12k_pci_power_up+0xdc/0x340[ath12k] [ 9.492747]ath12k_core_init+0x2c/0xa8[ath12k] [ 9.492769]ath12k_pci_probe+0x698/0x908[ath12k] [ 9.492791]pci_device_probe+0xa8/0x16c
[ 9.492800]really_probe+0x110/0x27c
[ 9.492805]__driver_probe_device+0x78/0x12c
[ 9.492808]driver_probe_device+0x3c/0x118
[ 9.492810]__driver_attach+0x74/0x124
[ 9.492813]bus_for_each_dev+0x78/0xd8
[ 9.492819]driver_attach+0x24/0x30
[ 9.492824]bus_add_driver+0xe4/0x208
[ 9.492828]driver_register+0x60/0x128
[ 9.492831]__pci_register_driver+0x44/0x50
[ 9.492835]ath12k_pci_init+0x2c/0x6c[ath12k]
[ 9.492858]do_one_initcall+0x70/0x1b4
[ 9.492861]do_init_module+0x58/0x1e4
[ 9.492867]load_module+0x19bc/0x1a8c
[ 9.492869]init_module_from_file+0x88/0xc4
[ 9.492873]__arm64_sys_finit_module+0x1c0/0x2ac[ 9.492877]invoke_syscall+0x44/0x108
[ 9.492882] el0_svc_common.constprop.0+0xc0/0xe0 [ 9.492885]do_el0_svc+0x1c/0x28
[ 9.492889]el0_svc+0x2c/0x84
[ 9.492892]el0t_64_sync_handler+0xc0/0xc4
[ 9.492895]el0t_64_sync+0x190/0x194
[ 9.492899] SMP: stopping secondary CPUs
[ 9.492908] Kernel Offset: disabled
[ 9.492909] CPU features: 0x0,80000200,28020000,1000420b [ 9.492913] Memory Limit: noneUpon searching online, I found the OpenWRT patch that appears to address a similar issue: OpenWRT Patch: Prevent LTSSM Startup Crash.
https://git.openwrt.org/?p=openwrt/openwrt.git;a=blob;f=package/kernel/mac80211/patches/ath12k/100-ath12k-prevent-ltssm-startup-crash.patch;h=cd85a0f6aa2652d62bfbea04e9bcca3bcf831b7f;hb=935b2b7dcef61b2893ed5dff307dd8f8a1156899
With the above patch applied, I do not see the crash anymore.Could anyone confirm if this issue has been reported before/known bug
or provide any insights?
Any additional information or suggestions would be greatly appreciated.Details about the test setup,
TI-AM69-SK board:
https://www.ti.com/tool/SK-AM69?keyMatch=am69%20sk&tisearch=universal_search
Silex WiFi card SX-PCEBE:
https://www.silextechnology.com/connectivity-solutions/embedded-wireless/sx-pcebe
TI Linux Repo:
https://git.ti.com/cgit/ti-linux-kernel/ti-linux-kernel/?h=ti-linux-6.6.yThank you.
Regards,
Parth P
Hi,
I am currently debugging the ath12k_pci_enable_ltssm start up crash/bug with the mainline kernel on my system and would like to share my observations so far:
The ath12k mainline driver gets stuck at this specific line:
https://github.com/torvalds/linux/blob/9c5968db9e625019a0ee5226c7eebef5519d366a/drivers/net/wireless/ath/ath12k/pci.c#L295
in the ath12k_pci_enable_ltssm which attempts to read GCC_GCC_PCIE_HOT_RST, particularly
https://github.com/torvalds/linux/blob/9c5968db9e625019a0ee5226c7eebef5519d366a/drivers/net/wireless/ath/ath12k/pci.c#L1209
Interestingly, within the same function, the line val = ath12k_pci_read32(ab, PCIE_PCIE_PARF_LTSSM) successfully reads the expected value 0x111 for PCIE_PCIE_PARF_LTSSM.
I am continuing to debug from my end, although my understanding of the ath12k driver is limited. Any leads, suggestions, or hints to help resolve this issue would be greatly appreciated.
Thank you.
Regards,
Parth P
Quoted 155 lines · Parth Pancholi
I appreciate your response, Baochen.
I have been working on enabling mainline kernel support on my TI AM69-
SK board to test the mainline ath12k driver on my system.Using the mainline kernel repository for the ath drivers [1], I made the following observation:
While the exact crash observed earlier is no longer present, the system
hangs upon loading the ath12k mainline driver, displaying the messages
below.root@am69-sk:~# modprobe ath12k debug_mask=0xffffffff [ 1121.996554] ath12k_pci 0000:01:00.0: BAR 0 [mem 0x4410200000- 0x44103fffff 64bit]: assigned [ 1122.004884] ath12k_pci 0000:01:00.0: enabling device (0000 -> 0002) [ 1122.011818] ath12k_pci 0000:01:00.0: MSI vectors: 16 [ 1122.016798] ath12k_pci 0000:01:00.0: Hardware name: wcn7850 hw2.0 [ 1122.040183] NET: Registered PF_QIPCRTR protocol familyroot@am69-sk:~# uname -a Linux am69-sk 6.13.0-rc7-wt-ath-ge7ef944b3e2c-dirty #2 SMP PREEMPT Wed Jan 22 16:55:17 CET 2025 aarch64 GNU/Linuxroot@am69-sk:~# lspci 0000:00:00.0 PCI bridge: Texas Instruments Device b012 0000:01:00.0 Network controller: Qualcomm Technologies, Inc WCN785x Wi- Fi 7(802.11be) 320MHz 2x2 [FastConnect 7800] (rev 01) 0001:00:00.0 PCI bridge: Texas Instruments Device b012 0002:00:00.0 PCI bridge: Texas Instruments Device b012Do you have any insights into what might still be missing or incorrect
in my setup?Regards,
Parth PQuoted 109 lines · Baochen Qiang
Quoted 7 lines · Parth Panchoil
Hi All,
I am performing tests on the SX-PCEBE Wi-Fi module, which utilizes
the
ATH12k driver, on the Texas Instruments AM69-SK board. The board is running the TI Linux Kernel from the ti-linux-6.6.y6.6 is too old, and besides we don;t support customer kernel.
Could you try latest ath tree [1] or the mainline tree [2]?
[1] https://git.kernel.org/pub/scm/linux/kernel/git/ath/ath.git/
[2]
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/If the issue is still seen, please enable verbose ath12k log using below command and help
collect dmesg logs:sudo modprobe ath12k debug_mask=0xffffffff
One more thing, the open-WRT patch is overkill, can you narrow down to find which line of
code inath12k_pci_enable_ltssm()is causing this issue?Quoted 77 lines
branch. During testing, I observed a kernel crash from the ATH12k driver as soon as the probe is called. The crash log is as follows:
[ 9.492631] Kernel panic - not syncing: Asynchronous SError Interrupt
[ 9.492634] CPU: 7 PID: 222 Comm: (udev-worker) Not tainted 6.6.58-
01497-ga7758da17c28-dirty #1
[ 9.492638] Hardware name: Texas Instruments AM69 SK (DT) [ 9.492640] Call trace:
[ 9.492642]dump_backtrace+0x94/0xec
[ 9.492658]show_stack+0x18/0x24
[ 9.492662]dump_stack_lvl+0x48/0x60
[ 9.492669]dump_stack+0x18/0x24
[ 9.492672]panic+0x320/0x378
[ 9.492677]nmi_panic+0x8c/0x90
[ 9.492681]arm64_serror_panic+0x6c/0x78
[ 9.492686]do_serror+0x3c/0x78
[ 9.492692]el1h_64_error_handler+0x34/0x4c
[ 9.492697]el1h_64_error+0x64/0x68
[ 9.492700]ath12k_pci_read32+0x1bc/0x1e8[ath12k] [ 9.492725]ath12k_pci_power_up+0xdc/0x340[ath12k] [ 9.492747]ath12k_core_init+0x2c/0xa8[ath12k] [ 9.492769]ath12k_pci_probe+0x698/0x908[ath12k] [ 9.492791]pci_device_probe+0xa8/0x16c
[ 9.492800]really_probe+0x110/0x27c
[ 9.492805]__driver_probe_device+0x78/0x12c
[ 9.492808]driver_probe_device+0x3c/0x118
[ 9.492810]__driver_attach+0x74/0x124
[ 9.492813]bus_for_each_dev+0x78/0xd8
[ 9.492819]driver_attach+0x24/0x30
[ 9.492824]bus_add_driver+0xe4/0x208
[ 9.492828]driver_register+0x60/0x128
[ 9.492831]__pci_register_driver+0x44/0x50
[ 9.492835]ath12k_pci_init+0x2c/0x6c[ath12k]
[ 9.492858]do_one_initcall+0x70/0x1b4
[ 9.492861]do_init_module+0x58/0x1e4
[ 9.492867]load_module+0x19bc/0x1a8c
[ 9.492869]init_module_from_file+0x88/0xc4
[ 9.492873]__arm64_sys_finit_module+0x1c0/0x2ac[ 9.492877]invoke_syscall+0x44/0x108
[ 9.492882] el0_svc_common.constprop.0+0xc0/0xe0 [ 9.492885]do_el0_svc+0x1c/0x28
[ 9.492889]el0_svc+0x2c/0x84
[ 9.492892]el0t_64_sync_handler+0xc0/0xc4
[ 9.492895]el0t_64_sync+0x190/0x194
[ 9.492899] SMP: stopping secondary CPUs
[ 9.492908] Kernel Offset: disabled
[ 9.492909] CPU features: 0x0,80000200,28020000,1000420b [ 9.492913] Memory Limit: noneUpon searching online, I found the OpenWRT patch that appears to address a similar issue: OpenWRT Patch: Prevent LTSSM Startup Crash.
https://git.openwrt.org/?p=openwrt/openwrt.git;a=blob;f=package/kernel/mac80211/patches/ath12k/100-ath12k-prevent-ltssm-startup-crash.patch;h=cd85a0f6aa2652d62bfbea04e9bcca3bcf831b7f;hb=935b2b7dcef61b2893ed5dff307dd8f8a1156899
With the above patch applied, I do not see the crash anymore.Could anyone confirm if this issue has been reported before/known bug
or provide any insights?
Any additional information or suggestions would be greatly appreciated.Details about the test setup,
TI-AM69-SK board:
https://www.ti.com/tool/SK-AM69?keyMatch=am69%20sk&tisearch=universal_search
Silex WiFi card SX-PCEBE:
https://www.silextechnology.com/connectivity-solutions/embedded-wireless/sx-pcebe
TI Linux Repo:
https://git.ti.com/cgit/ti-linux-kernel/ti-linux-kernel/?h=ti-linux-6.6.yThank you.
Regards,
Parth P
Hi,
I would like to share more details regarding this issue.
A similar bug was previously reported:
https://lore.kernel.org/all/CAFED-jma7ZB17Rcgx9H+eVZ=N6Pty4__gz3B+O2V34XtcEzOGg@mail.gmail.com/
Continuing my debugging on the mainline ath12k driver, I have been investigating a startup hang/crash on my system.
So far, I have identified potential issues with memory mappings, where the driver attempts to read an offset like GCC_GCC_PCIE_HOT_RST (0x1e38338), which is significantly farther than PCIE_PCIE_PARF_LTSSM (0x1e081b0).
I have tried several approaches, including:
- Adjusting the MMIO window dynamically based on the offset instead of using a fixed window.
- Increasing the MMIO window size beyond 512 KiB.
However, these changes did not resolve the issue.
I am sharing my debug logs below (The patch enabling these exact logs is pasted), which may be useful for someone working on this driver to identify the root cause and help resolve this bug.
Any insights or suggestions would be greatly appreciated.
root@am69-sk:~# uname -a Linux am69-sk 6.13.0-rc7-wt-ath-ge7ef944b3e2c-dirty #4 SMP PREEMPT Wed Jan 29 08:55:48 CET 2025 aarch64 GNU/Linux
root@am69-sk:~# insmod ./ath12k.ko debug_mask=0xffffffff [ 50.148099] ath12k_pci 0002:01:00.0: BAR 0 [mem 0x4410200000- 0x44103fffff 64bit]: assigned [ 50.156434] ath12k_pci 0002:01:00.0: enabling device (0000 -> 0002) [ 50.162775] ath12k_pci 0002:01:00.0: boot pci_mem 0x000000005f7b1667 [ 50.169126] ath12k_pci 0002:01:00.0: pci probe 17cb:1107 17cb:1107 [ 50.175294] ### DBG: ath12k_pci_read32 1199 offset =1b00000 [ 50.180856] ### DBG: ath12k_pci_read32 1217 offset = 1b00000 [ 50.186504] ### DBG: ath12k_pci_select_window 177 window = 36 ab_pci->register_window = 0 WINDOW_REG_ADDRESS = 310c [ 50.196920] ### DBG: ath12k_pci_select_window 178 (WINDOW_ENABLE_BIT | window) = 40000036 [ 50.205084] ### DBG: ath12k_pci_select_window 179 ab->mem + WINDOW_REG_ADDRESS = 8aa0310c [ 50.213248] ### DBG: ath12k_pci_read32 1220 ab_pci->register_window = 36 [ 50.220023] ### DBG: ath12k_pci_read32 1221 window_start = 80000 offset = 1b00000 WINDOW_RANGE_MASK = 7ffff [ 50.229831] ### DBG: ath12k_pci_read32 1229 window_start = 80000 offset = 1b00000 WINDOW_RANGE_MASK = 7ffff ab->mem = 8aa00000 [ 50.241200] ### DBG: ath12k_pci_read32 1230 ab->mem + window_start + (offset & WINDOW_RANGE_MASK) = 8aa80000 [ 50.251010] ### DBG: ath12k_pci_read32 1233 [ 50.255185] ### DBG: ath12k_pci_read32 1243 val = 40170200 [ 50.260658] ath12k_pci 0002:01:00.0: pci tcsr_soc_hw_version major 2 minor 0 [ 50.268230] ath12k_pci 0002:01:00.0: MSI vectors: 16 [ 50.273212] ath12k_pci 0002:01:00.0: msi base data is 0 [ 50.278430] ath12k_pci 0002:01:00.0: Hardware name: wcn7850 hw2.0 [ 50.290532] ath12k_pci 0002:01:00.0: failed to load firmware-2.bin: -2 [ 50.297063] ath12k_pci 0002:01:00.0: using fw api 1 [ 50.301959] ath12k_pci 0002:01:00.0: Assign MSI to user: MHI, num_vectors: 3, user_base_data: 0, base_vector: 0 [ 50.312038] ath12k_pci 0002:01:00.0: Number of assigned MSI for MHI is 3, base vector is 0 [ 50.322527] ath12k_pci 0002:01:00.0: Assign MSI to user: CE, num_vectors: 5, user_base_data: 3, base_vector: 3 [ 50.332673] ath12k_pci 0002:01:00.0: Assign MSI to user: DP, num_vectors: 8, user_base_data: 8, base_vector: 8 [ 50.342697] ath12k_pci 0002:01:00.0: irq:604 group:0 [ 50.347696] ath12k_pci 0002:01:00.0: irq:605 group:1 [ 50.352694] ath12k_pci 0002:01:00.0: irq:606 group:2 [ 50.357698] ath12k_pci 0002:01:00.0: irq:607 group:3 [ 50.362696] ath12k_pci 0002:01:00.0: irq:608 group:4 [ 50.367693] ath12k_pci 0002:01:00.0: irq:609 group:5 [ 50.372689] ath12k_pci 0002:01:00.0: irq:610 group:6 [ 50.377716] ath12k_pci 0002:01:00.0: pci after request_irq msi_ep_base_data 0 [ 50.384846] ath12k_pci 0002:01:00.0: unable to get wsi info from dt, grouping single device [ 50.393196] ath12k_pci 0002:01:00.0: wsi group-id 255 num-devices 1 index 0 [ 50.400148] ath12k_pci 0002:01:00.0: num devices 1 num probed 1 [ 50.423637] NET: Registered PF_QIPCRTR protocol family [ 50.429598] ### DBG: ath12k_pci_enable_ltssm 290 [ 50.434225] ### DBG: ath12k_pci_read32 1199 offset =1e081b0 [ 50.439790] ### DBG: ath12k_pci_read32 1217 offset = 1e081b0 [ 50.445437] ### DBG: ath12k_pci_select_window 177 window = 3c ab_pci->register_window = 0 WINDOW_REG_ADDRESS = 310c [ 50.455852] ### DBG: ath12k_pci_select_window 178 (WINDOW_ENABLE_BIT | window) = 4000003c [ 50.464014] ### DBG: ath12k_pci_select_window 179 ab->mem + WINDOW_REG_ADDRESS = 8aa0310c [ 50.472178] ### DBG: ath12k_pci_read32 1220 ab_pci->register_window = 3c [ 50.478952] ### DBG: ath12k_pci_read32 1221 window_start = 80000 offset = 1e081b0 WINDOW_RANGE_MASK = 7ffff [ 50.488759] ### DBG: ath12k_pci_read32 1229 window_start = 80000 offset = 1e081b0 WINDOW_RANGE_MASK = 7ffff ab->mem = 8aa00000 [ 50.500126] ### DBG: ath12k_pci_read32 1230 ab->mem + window_start + (offset & WINDOW_RANGE_MASK) = 8aa881b0 [ 50.509936] ### DBG: ath12k_pci_read32 1233 [ 50.514109] ### DBG: ath12k_pci_read32 1243 val = 111 [ 50.519150] ### DBG: ath12k_pci_enable_ltssm 293 val=111 [ 50.524452] ath12k_pci 0002:01:00.0: pci ltssm 0x111 [ 50.529407] ### DBG: ath12k_pci_enable_ltssm 305 [ 50.534014] ### DBG: ath12k_pci_read32 1199 offset =1e38338 [ 50.539577] ### DBG: ath12k_pci_read32 1217 offset = 1e38338 [ 50.545227] ### DBG: ath12k_pci_read32 1220 ab_pci->register_window = 3c [ 50.552001] ### DBG: ath12k_pci_read32 1221 window_start = 80000 offset = 1e38338 WINDOW_RANGE_MASK = 7ffff [ 50.561808] ### DBG: ath12k_pci_read32 1229 window_start = 80000 offset = 1e38338 WINDOW_RANGE_MASK = 7ffff ab->mem = 8aa00000 [ 50.573175] ### DBG: ath12k_pci_read32 1230 ab->mem + window_start + (offset & WINDOW_RANGE_MASK) = 8aab8338
Thank you.
Regards,
Parth P
=======================================================================
(quoted diff) +0 -0
b/drivers/net/wireless/ath/ath12k/pci.c
index 06cff3849ab8..2984064783f9 100644
drivers/net/wireless/ath/ath12k/pci.c +0 -0
@@ -174,6 +174,9 @@ static void ath12k_pci_select_window(struct
ath12k_pci *ab_pci, u32 offset)
window |= static_window;
if (window != ab_pci->register_window) {
+ printk("### DBG: %s %d window = %x ab_pci-
Quoted 1 line
register_window= %xWINDOW_REG_ADDRESS=
%x\n",__func__,__LINE__,window, ab_pci->register_window, WINDOW_REG_ADDRESS);
+ printk("### DBG: %s %d (WINDOW_ENABLE_BIT | window) =
%x\n",__func__,__LINE__,(WINDOW_ENABLE_BIT | window) );
+ printk("### DBG: %s %d ab->mem + WINDOW_REG_ADDRESS =
%x\n",__func__,__LINE__,(ab->mem + WINDOW_REG_ADDRESS) );
iowrite32(WINDOW_ENABLE_BIT | window,
ab->mem + WINDOW_REG_ADDRESS);
ioread32(ab->mem + WINDOW_REG_ADDRESS);
@@ -279,8 +282,15 @@ static void ath12k_pci_enable_ltssm(struct ath12k_base *ab)
u32 val;
int i;
+#if 0
+ /* Prevent startup crash on BPI-Rx */
+ return;
+#endif
+
+ printk("### DBG: %s %d\n",__func__,__LINE__);
val = ath12k_pci_read32(ab, PCIE_PCIE_PARF_LTSSM);
+ printk("### DBG: %s %d val=%x\n",__func__,__LINE__,val);
/* PCIE link seems very unstable after the Hot Reset*/ for (i = 0; val != PARM_LTSSM_VALUE && i < 5; i++) {
if (val == 0xffffffff)
@@ -292,11 +302,15 @@ static void ath12k_pci_enable_ltssm(struct ath12k_base *ab)
ath12k_dbg(ab, ATH12K_DBG_PCI, "pci ltssm 0x%x\n", val);
+ printk("### DBG: %s %d\n",__func__,__LINE__);
val = ath12k_pci_read32(ab, GCC_GCC_PCIE_HOT_RST);
+ printk("### DBG: %s %d val = %x\n",__func__,__LINE__,val);
val |= GCC_GCC_PCIE_HOT_RST_VAL;
+ printk("### DBG: %s %d val = %x\n",__func__,__LINE__,val);
ath12k_pci_write32(ab, GCC_GCC_PCIE_HOT_RST, val); val = ath12k_pci_read32(ab, GCC_GCC_PCIE_HOT_RST);
+ printk("### DBG: %s %d\n",__func__,__LINE__);
ath12k_dbg(ab, ATH12K_DBG_PCI, "pci pcie_hot_rst 0x%x\n",
val);
mdelay(5);
@@ -1182,6 +1196,7 @@ u32 ath12k_pci_read32(struct ath12k_base *ab, u32 offset)
u32 val, window_start;
int ret = 0;
+ printk("### DBG: %s %d offset
=%x\n",__func__,__LINE__,offset);
/* for offset beyond BAR + 4K - 32, may
* need to wakeup MHI to access.
*/
@@ -1196,26 +1211,36 @@ u32 ath12k_pci_read32(struct ath12k_base *ab, u32 offset)
window_start = ath12k_pci_get_window_start(ab,
offset);
else
window_start = WINDOW_START;
+ //window_start = (offset &
~WINDOW_RANGE_MASK);
if (window_start == WINDOW_START) {
+ printk("### DBG: %s %d offset =
%x\n",__func__,__LINE__,offset);
spin_lock_bh(&ab_pci->window_lock); ath12k_pci_select_window(ab_pci, offset);
+ printk("### DBG: %s %d ab_pci->register_window
= %x \n",__func__,__LINE__, ab_pci->register_window);
+ printk("### DBG: %s %d window_start = %x
offset = %x WINDOW_RANGE_MASK = %x \n",__func__,__LINE__, window_start,offset, WINDOW_RANGE_MASK);
if
(ath12k_pci_is_offset_within_mhi_region(offset)) {
+ printk("### DBG: %s
%d\n",__func__,__LINE__);
offset = offset - PCI_MHIREGLEN_REG; val = ioread32(ab->mem +
(offset &WINDOW_RANGE_MASK));
} else {
+ printk("### DBG: %s %d window_start =
%x offset = %x WINDOW_RANGE_MASK = %x ab->mem = %x\n",__func__,__LINE__,window_start,offset, WINDOW_RANGE_MASK,ab-
Quoted 1 line
mem);
+ printk("### DBG: %s %d ab->mem +window_start + (offset & WINDOW_RANGE_MASK) = %x\n",__func__,__LINE__,ab->mem + window_start + (offset & WINDOW_RANGE_MASK));
val = ioread32(ab->mem + window_start
+
(offset &WINDOW_RANGE_MASK));
+ printk("### DBG: %s
%d\n",__func__,__LINE__);
} spin_unlock_bh(&ab_pci->window_lock);
} else {
+ printk("### DBG: %s %d\n",__func__,__LINE__);
val = ioread32(ab->mem + window_start +
(offset & WINDOW_RANGE_MASK));
}
}
+ printk("### DBG: %s %d val = %x\n",__func__,__LINE__,val);
if (test_bit(ATH12K_PCI_FLAG_INIT_DONE, &ab_pci->flags) &&
offset >= ACCESS_ALWAYS_OFF && ab_pci->pci_ops->release && !ret)
=======================================================================
Quoted 193 lines · Parth Panchoil
Hi,
I am currently debugging the
ath12k_pci_enable_ltssmstart up crash/bug
with the mainline kernel on my system and would like to share my observations so far:The ath12k mainline driver gets stuck at this specific line:
https://github.com/torvalds/linux/blob/9c5968db9e625019a0ee5226c7eebef5519d366a/drivers/net/wireless/ath/ath12k/pci.c#L295
in the ath12k_pci_enable_ltssm which attempts to readGCC_GCC_PCIE_HOT_RST, particularly
https://github.com/torvalds/linux/blob/9c5968db9e625019a0ee5226c7eebef5519d366a/drivers/net/wireless/ath/ath12k/pci.c#L1209Interestingly, within the same function, the line val =
ath12k_pci_read32(ab, PCIE_PCIE_PARF_LTSSM)successfully reads the expected value0x111forPCIE_PCIE_PARF_LTSSM.I am continuing to debug from my end, although my understanding of the
ath12k driver is limited. Any leads, suggestions, or hints to help resolve this issue would be greatly appreciated.Thank you.
Regards,
Parth PQuoted 163 lines · Parth Pancholi
I appreciate your response, Baochen.
I have been working on enabling mainline kernel support on my TI AM69-
SK board to test the mainline ath12k driver on my system.Using the mainline kernel repository for the ath drivers [1], I made
the following observation:
While the exact crash observed earlier is no longer present, the system
hangs upon loading the ath12k mainline driver, displaying the messages
below.root@am69-sk:~# modprobe ath12k debug_mask=0xffffffff [ 1121.996554] ath12k_pci 0000:01:00.0: BAR 0 [mem 0x4410200000- 0x44103fffff 64bit]: assigned [ 1122.004884] ath12k_pci 0000:01:00.0: enabling device (0000 -> 0002) [ 1122.011818] ath12k_pci 0000:01:00.0: MSI vectors: 16 [ 1122.016798] ath12k_pci 0000:01:00.0: Hardware name: wcn7850 hw2.0 [ 1122.040183] NET: Registered PF_QIPCRTR protocol familyroot@am69-sk:~# uname -a Linux am69-sk 6.13.0-rc7-wt-ath-ge7ef944b3e2c-dirty #2 SMP PREEMPT Wed Jan 22 16:55:17 CET 2025 aarch64 GNU/Linuxroot@am69-sk:~# lspci 0000:00:00.0 PCI bridge: Texas Instruments Device b012 0000:01:00.0 Network controller: Qualcomm Technologies, Inc WCN785x Wi- Fi 7(802.11be) 320MHz 2x2 [FastConnect 7800] (rev 01) 0001:00:00.0 PCI bridge: Texas Instruments Device b012 0002:00:00.0 PCI bridge: Texas Instruments Device b012Do you have any insights into what might still be missing or incorrect
in my setup?Regards,
Parth PQuoted 115 lines · Baochen Qiang
Quoted 8 lines · Parth Panchoil
Hi All,
I am performing tests on the SX-PCEBE Wi-Fi module, which utilizes
the
ATH12k driver, on the Texas Instruments AM69-SK board. The board is running the TI Linux Kernel from the ti-linux- 6.6.y6.6 is too old, and besides we don;t support customer kernel.
Could you try latest ath tree [1] or the mainline tree [2]?
[1] https://git.kernel.org/pub/scm/linux/kernel/git/ath/ath.git/
[2]
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/If the issue is still seen, please enable verbose ath12k log using
below command and help
collect dmesg logs:sudo modprobe ath12k debug_mask=0xffffffff
One more thing, the open-WRT patch is overkill, can you narrow down
to find which line of
code inath12k_pci_enable_ltssm()is causing this issue?Quoted 80 lines
branch. During testing, I observed a kernel crash from the ATH12k
driver as soon as the probe is called. The crash log is as follows:[ 9.492631] Kernel panic - not syncing: Asynchronous SError Interrupt
[ 9.492634] CPU: 7 PID: 222 Comm: (udev-worker) Not tainted 6.6.58-
01497-ga7758da17c28-dirty #1
[ 9.492638] Hardware name: Texas Instruments AM69 SK (DT) [ 9.492640] Call trace:
[ 9.492642]dump_backtrace+0x94/0xec
[ 9.492658]show_stack+0x18/0x24
[ 9.492662]dump_stack_lvl+0x48/0x60
[ 9.492669]dump_stack+0x18/0x24
[ 9.492672]panic+0x320/0x378
[ 9.492677]nmi_panic+0x8c/0x90
[ 9.492681]arm64_serror_panic+0x6c/0x78
[ 9.492686]do_serror+0x3c/0x78
[ 9.492692]el1h_64_error_handler+0x34/0x4c
[ 9.492697]el1h_64_error+0x64/0x68
[ 9.492700]ath12k_pci_read32+0x1bc/0x1e8[ath12k] [ 9.492725]ath12k_pci_power_up+0xdc/0x340[ath12k] [ 9.492747]ath12k_core_init+0x2c/0xa8[ath12k] [ 9.492769]ath12k_pci_probe+0x698/0x908[ath12k] [ 9.492791]pci_device_probe+0xa8/0x16c
[ 9.492800]really_probe+0x110/0x27c
[ 9.492805]__driver_probe_device+0x78/0x12c
[ 9.492808]driver_probe_device+0x3c/0x118
[ 9.492810]__driver_attach+0x74/0x124
[ 9.492813]bus_for_each_dev+0x78/0xd8
[ 9.492819]driver_attach+0x24/0x30
[ 9.492824]bus_add_driver+0xe4/0x208
[ 9.492828]driver_register+0x60/0x128
[ 9.492831]__pci_register_driver+0x44/0x50
[ 9.492835]ath12k_pci_init+0x2c/0x6c[ath12k]
[ 9.492858]do_one_initcall+0x70/0x1b4
[ 9.492861]do_init_module+0x58/0x1e4
[ 9.492867]load_module+0x19bc/0x1a8c
[ 9.492869]init_module_from_file+0x88/0xc4
[ 9.492873]__arm64_sys_finit_module+0x1c0/0x2ac[ 9.492877]invoke_syscall+0x44/0x108
[ 9.492882] el0_svc_common.constprop.0+0xc0/0xe0 [ 9.492885]do_el0_svc+0x1c/0x28
[ 9.492889]el0_svc+0x2c/0x84
[ 9.492892]el0t_64_sync_handler+0xc0/0xc4
[ 9.492895]el0t_64_sync+0x190/0x194
[ 9.492899] SMP: stopping secondary CPUs
[ 9.492908] Kernel Offset: disabled
[ 9.492909] CPU features: 0x0,80000200,28020000,1000420b [ 9.492913] Memory Limit: noneUpon searching online, I found the OpenWRT patch that appears to
address a similar issue: OpenWRT Patch: Prevent LTSSM Startup Crash.
https://git.openwrt.org/?p=openwrt/openwrt.git;a=blob;f=package/kernel/mac80211/patches/ath12k/100-ath12k-prevent-ltssm-startup-crash.patch;h=cd85a0f6aa2652d62bfbea04e9bcca3bcf831b7f;hb=935b2b7dcef61b2893ed5dff307dd8f8a1156899
With the above patch applied, I do not see the crash anymore.Could anyone confirm if this issue has been reported before/known
bug
or provide any insights?
Any additional information or suggestions would be greatly appreciated.Details about the test setup,
TI-AM69-SK board:
https://www.ti.com/tool/SK-AM69?keyMatch=am69%20sk&tisearch=universal_search
Silex WiFi card SX-PCEBE:
https://www.silextechnology.com/connectivity-solutions/embedded-wireless/sx-pcebe
TI Linux Repo:
https://git.ti.com/cgit/ti-linux-kernel/ti-linux-kernel/?h=ti-linux-6.6.yThank you.
Regards,
Parth P
Quoted 11 lines · Parth Panchoil
Hi,
I am currently debugging the
ath12k_pci_enable_ltssmstart up crash/bug with the mainline kernel on my system and would like to share my observations so far:The ath12k mainline driver gets stuck at this specific line:
https://github.com/torvalds/linux/blob/9c5968db9e625019a0ee5226c7eebef5519d366a/drivers/net/wireless/ath/ath12k/pci.c#L295
in theath12k_pci_enable_ltssmwhich attempts to readGCC_GCC_PCIE_HOT_RST, particularly
https://github.com/torvalds/linux/blob/9c5968db9e625019a0ee5226c7eebef5519d366a/drivers/net/wireless/ath/ath12k/pci.c#L1209
thanks for the narrow down, really helpful.
We internally have observed this issue, although at a different line:
For now I am suspecting that GCC_GCC_PCIE_HOT_RST is not a valid register on WLAN target side, I will check internally and get back.
Quoted 172 lines
Interestingly, within the same function, the line val =
ath12k_pci_read32(ab, PCIE_PCIE_PARF_LTSSM)successfully reads the expected value0x111forPCIE_PCIE_PARF_LTSSM.I am continuing to debug from my end, although my understanding of the ath12k driver is limited. Any leads, suggestions, or hints to help resolve this issue would be greatly appreciated.
Thank you.
Regards,
Parth PQuoted 155 lines · Parth Pancholi
I appreciate your response, Baochen.
I have been working on enabling mainline kernel support on my TI AM69-
SK board to test the mainline ath12k driver on my system.Using the mainline kernel repository for the ath drivers [1], I made the following observation:
While the exact crash observed earlier is no longer present, the system
hangs upon loading the ath12k mainline driver, displaying the messages
below.root@am69-sk:~# modprobe ath12k debug_mask=0xffffffff [ 1121.996554] ath12k_pci 0000:01:00.0: BAR 0 [mem 0x4410200000- 0x44103fffff 64bit]: assigned [ 1122.004884] ath12k_pci 0000:01:00.0: enabling device (0000 -> 0002) [ 1122.011818] ath12k_pci 0000:01:00.0: MSI vectors: 16 [ 1122.016798] ath12k_pci 0000:01:00.0: Hardware name: wcn7850 hw2.0 [ 1122.040183] NET: Registered PF_QIPCRTR protocol familyroot@am69-sk:~# uname -a Linux am69-sk 6.13.0-rc7-wt-ath-ge7ef944b3e2c-dirty #2 SMP PREEMPT Wed Jan 22 16:55:17 CET 2025 aarch64 GNU/Linuxroot@am69-sk:~# lspci 0000:00:00.0 PCI bridge: Texas Instruments Device b012 0000:01:00.0 Network controller: Qualcomm Technologies, Inc WCN785x Wi- Fi 7(802.11be) 320MHz 2x2 [FastConnect 7800] (rev 01) 0001:00:00.0 PCI bridge: Texas Instruments Device b012 0002:00:00.0 PCI bridge: Texas Instruments Device b012Do you have any insights into what might still be missing or incorrect
in my setup?Regards,
Parth PQuoted 109 lines · Baochen Qiang
Quoted 7 lines · Parth Panchoil
Hi All,
I am performing tests on the SX-PCEBE Wi-Fi module, which utilizes
the
ATH12k driver, on the Texas Instruments AM69-SK board. The board is running the TI Linux Kernel from the ti-linux-6.6.y6.6 is too old, and besides we don;t support customer kernel.
Could you try latest ath tree [1] or the mainline tree [2]?
[1] https://git.kernel.org/pub/scm/linux/kernel/git/ath/ath.git/
[2]
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/If the issue is still seen, please enable verbose ath12k log using below command and help
collect dmesg logs:sudo modprobe ath12k debug_mask=0xffffffff
One more thing, the open-WRT patch is overkill, can you narrow down to find which line of
code inath12k_pci_enable_ltssm()is causing this issue?Quoted 77 lines
branch. During testing, I observed a kernel crash from the ATH12k driver as soon as the probe is called. The crash log is as follows:
[ 9.492631] Kernel panic - not syncing: Asynchronous SError Interrupt
[ 9.492634] CPU: 7 PID: 222 Comm: (udev-worker) Not tainted 6.6.58-
01497-ga7758da17c28-dirty #1
[ 9.492638] Hardware name: Texas Instruments AM69 SK (DT) [ 9.492640] Call trace:
[ 9.492642]dump_backtrace+0x94/0xec
[ 9.492658]show_stack+0x18/0x24
[ 9.492662]dump_stack_lvl+0x48/0x60
[ 9.492669]dump_stack+0x18/0x24
[ 9.492672]panic+0x320/0x378
[ 9.492677]nmi_panic+0x8c/0x90
[ 9.492681]arm64_serror_panic+0x6c/0x78
[ 9.492686]do_serror+0x3c/0x78
[ 9.492692]el1h_64_error_handler+0x34/0x4c
[ 9.492697]el1h_64_error+0x64/0x68
[ 9.492700]ath12k_pci_read32+0x1bc/0x1e8[ath12k] [ 9.492725]ath12k_pci_power_up+0xdc/0x340[ath12k] [ 9.492747]ath12k_core_init+0x2c/0xa8[ath12k] [ 9.492769]ath12k_pci_probe+0x698/0x908[ath12k] [ 9.492791]pci_device_probe+0xa8/0x16c
[ 9.492800]really_probe+0x110/0x27c
[ 9.492805]__driver_probe_device+0x78/0x12c
[ 9.492808]driver_probe_device+0x3c/0x118
[ 9.492810]__driver_attach+0x74/0x124
[ 9.492813]bus_for_each_dev+0x78/0xd8
[ 9.492819]driver_attach+0x24/0x30
[ 9.492824]bus_add_driver+0xe4/0x208
[ 9.492828]driver_register+0x60/0x128
[ 9.492831]__pci_register_driver+0x44/0x50
[ 9.492835]ath12k_pci_init+0x2c/0x6c[ath12k]
[ 9.492858]do_one_initcall+0x70/0x1b4
[ 9.492861]do_init_module+0x58/0x1e4
[ 9.492867]load_module+0x19bc/0x1a8c
[ 9.492869]init_module_from_file+0x88/0xc4
[ 9.492873]__arm64_sys_finit_module+0x1c0/0x2ac[ 9.492877]invoke_syscall+0x44/0x108
[ 9.492882] el0_svc_common.constprop.0+0xc0/0xe0 [ 9.492885]do_el0_svc+0x1c/0x28
[ 9.492889]el0_svc+0x2c/0x84
[ 9.492892]el0t_64_sync_handler+0xc0/0xc4
[ 9.492895]el0t_64_sync+0x190/0x194
[ 9.492899] SMP: stopping secondary CPUs
[ 9.492908] Kernel Offset: disabled
[ 9.492909] CPU features: 0x0,80000200,28020000,1000420b [ 9.492913] Memory Limit: noneUpon searching online, I found the OpenWRT patch that appears to address a similar issue: OpenWRT Patch: Prevent LTSSM Startup Crash.
https://git.openwrt.org/?p=openwrt/openwrt.git;a=blob;f=package/kernel/mac80211/patches/ath12k/100-ath12k-prevent-ltssm-startup-crash.patch;h=cd85a0f6aa2652d62bfbea04e9bcca3bcf831b7f;hb=935b2b7dcef61b2893ed5dff307dd8f8a1156899
With the above patch applied, I do not see the crash anymore.Could anyone confirm if this issue has been reported before/known bug
or provide any insights?
Any additional information or suggestions would be greatly appreciated.Details about the test setup,
TI-AM69-SK board:
https://www.ti.com/tool/SK-AM69?keyMatch=am69%20sk&tisearch=universal_search
Silex WiFi card SX-PCEBE:
https://www.silextechnology.com/connectivity-solutions/embedded-wireless/sx-pcebe
TI Linux Repo:
https://git.ti.com/cgit/ti-linux-kernel/ti-linux-kernel/?h=ti-linux-6.6.yThank you.
Regards,
Parth P
Quoted 15 lines · Baochen Qiang
Quoted 11 lines · Parth Panchoil
Hi,
I am currently debugging the
ath12k_pci_enable_ltssmstart up crash/bug with the mainline kernel on my system and would like to share my observations so far:The ath12k mainline driver gets stuck at this specific line:
https://github.com/torvalds/linux/blob/9c5968db9e625019a0ee5226c7eebef5519d366a/drivers/net/wireless/ath/ath12k/pci.c#L295
in theath12k_pci_enable_ltssmwhich attempts to readGCC_GCC_PCIE_HOT_RST, particularly
https://github.com/torvalds/linux/blob/9c5968db9e625019a0ee5226c7eebef5519d366a/drivers/net/wireless/ath/ath12k/pci.c#L1209
Quoted 8 lines
thanks for the narrow down, really helpful.
We internally have observed this issue, although at a different line:
For now I am suspecting that
GCC_GCC_PCIE_HOT_RSTis not a valid register on WLAN target side, I will check internally and get back.
Parth, could you do below change and try again?
-#define GCC_GCC_PCIE_HOT_RST 0x1e38338
+#define GCC_GCC_PCIE_HOT_RST 0x1e40304
Quoted 175 lines
Quoted 173 lines
Interestingly, within the same function, the line val =
ath12k_pci_read32(ab, PCIE_PCIE_PARF_LTSSM)successfully reads the expected value0x111forPCIE_PCIE_PARF_LTSSM.I am continuing to debug from my end, although my understanding of the ath12k driver is limited. Any leads, suggestions, or hints to help resolve this issue would be greatly appreciated.
Thank you.
Regards,
Parth PQuoted 155 lines · Parth Pancholi
I appreciate your response, Baochen.
I have been working on enabling mainline kernel support on my TI AM69-
SK board to test the mainline ath12k driver on my system.Using the mainline kernel repository for the ath drivers [1], I made the following observation:
While the exact crash observed earlier is no longer present, the system
hangs upon loading the ath12k mainline driver, displaying the messages
below.root@am69-sk:~# modprobe ath12k debug_mask=0xffffffff [ 1121.996554] ath12k_pci 0000:01:00.0: BAR 0 [mem 0x4410200000- 0x44103fffff 64bit]: assigned [ 1122.004884] ath12k_pci 0000:01:00.0: enabling device (0000 -> 0002) [ 1122.011818] ath12k_pci 0000:01:00.0: MSI vectors: 16 [ 1122.016798] ath12k_pci 0000:01:00.0: Hardware name: wcn7850 hw2.0 [ 1122.040183] NET: Registered PF_QIPCRTR protocol familyroot@am69-sk:~# uname -a Linux am69-sk 6.13.0-rc7-wt-ath-ge7ef944b3e2c-dirty #2 SMP PREEMPT Wed Jan 22 16:55:17 CET 2025 aarch64 GNU/Linuxroot@am69-sk:~# lspci 0000:00:00.0 PCI bridge: Texas Instruments Device b012 0000:01:00.0 Network controller: Qualcomm Technologies, Inc WCN785x Wi- Fi 7(802.11be) 320MHz 2x2 [FastConnect 7800] (rev 01) 0001:00:00.0 PCI bridge: Texas Instruments Device b012 0002:00:00.0 PCI bridge: Texas Instruments Device b012Do you have any insights into what might still be missing or incorrect
in my setup?Regards,
Parth PQuoted 109 lines · Baochen Qiang
Quoted 7 lines · Parth Panchoil
Hi All,
I am performing tests on the SX-PCEBE Wi-Fi module, which utilizes
the
ATH12k driver, on the Texas Instruments AM69-SK board. The board is running the TI Linux Kernel from the ti-linux-6.6.y6.6 is too old, and besides we don;t support customer kernel.
Could you try latest ath tree [1] or the mainline tree [2]?
[1] https://git.kernel.org/pub/scm/linux/kernel/git/ath/ath.git/
[2]
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/If the issue is still seen, please enable verbose ath12k log using below command and help
collect dmesg logs:sudo modprobe ath12k debug_mask=0xffffffff
One more thing, the open-WRT patch is overkill, can you narrow down to find which line of
code inath12k_pci_enable_ltssm()is causing this issue?Quoted 77 lines
branch. During testing, I observed a kernel crash from the ATH12k driver as soon as the probe is called. The crash log is as follows:
[ 9.492631] Kernel panic - not syncing: Asynchronous SError Interrupt
[ 9.492634] CPU: 7 PID: 222 Comm: (udev-worker) Not tainted 6.6.58-
01497-ga7758da17c28-dirty #1
[ 9.492638] Hardware name: Texas Instruments AM69 SK (DT) [ 9.492640] Call trace:
[ 9.492642]dump_backtrace+0x94/0xec
[ 9.492658]show_stack+0x18/0x24
[ 9.492662]dump_stack_lvl+0x48/0x60
[ 9.492669]dump_stack+0x18/0x24
[ 9.492672]panic+0x320/0x378
[ 9.492677]nmi_panic+0x8c/0x90
[ 9.492681]arm64_serror_panic+0x6c/0x78
[ 9.492686]do_serror+0x3c/0x78
[ 9.492692]el1h_64_error_handler+0x34/0x4c
[ 9.492697]el1h_64_error+0x64/0x68
[ 9.492700]ath12k_pci_read32+0x1bc/0x1e8[ath12k] [ 9.492725]ath12k_pci_power_up+0xdc/0x340[ath12k] [ 9.492747]ath12k_core_init+0x2c/0xa8[ath12k] [ 9.492769]ath12k_pci_probe+0x698/0x908[ath12k] [ 9.492791]pci_device_probe+0xa8/0x16c
[ 9.492800]really_probe+0x110/0x27c
[ 9.492805]__driver_probe_device+0x78/0x12c
[ 9.492808]driver_probe_device+0x3c/0x118
[ 9.492810]__driver_attach+0x74/0x124
[ 9.492813]bus_for_each_dev+0x78/0xd8
[ 9.492819]driver_attach+0x24/0x30
[ 9.492824]bus_add_driver+0xe4/0x208
[ 9.492828]driver_register+0x60/0x128
[ 9.492831]__pci_register_driver+0x44/0x50
[ 9.492835]ath12k_pci_init+0x2c/0x6c[ath12k]
[ 9.492858]do_one_initcall+0x70/0x1b4
[ 9.492861]do_init_module+0x58/0x1e4
[ 9.492867]load_module+0x19bc/0x1a8c
[ 9.492869]init_module_from_file+0x88/0xc4
[ 9.492873]__arm64_sys_finit_module+0x1c0/0x2ac[ 9.492877]invoke_syscall+0x44/0x108
[ 9.492882] el0_svc_common.constprop.0+0xc0/0xe0 [ 9.492885]do_el0_svc+0x1c/0x28
[ 9.492889]el0_svc+0x2c/0x84
[ 9.492892]el0t_64_sync_handler+0xc0/0xc4
[ 9.492895]el0t_64_sync+0x190/0x194
[ 9.492899] SMP: stopping secondary CPUs
[ 9.492908] Kernel Offset: disabled
[ 9.492909] CPU features: 0x0,80000200,28020000,1000420b [ 9.492913] Memory Limit: noneUpon searching online, I found the OpenWRT patch that appears to address a similar issue: OpenWRT Patch: Prevent LTSSM Startup Crash.
https://git.openwrt.org/?p=openwrt/openwrt.git;a=blob;f=package/kernel/mac80211/patches/ath12k/100-ath12k-prevent-ltssm-startup-crash.patch;h=cd85a0f6aa2652d62bfbea04e9bcca3bcf831b7f;hb=935b2b7dcef61b2893ed5dff307dd8f8a1156899
With the above patch applied, I do not see the crash anymore.Could anyone confirm if this issue has been reported before/known bug
or provide any insights?
Any additional information or suggestions would be greatly appreciated.Details about the test setup,
TI-AM69-SK board:
https://www.ti.com/tool/SK-AM69?keyMatch=am69%20sk&tisearch=universal_search
Silex WiFi card SX-PCEBE:
https://www.silextechnology.com/connectivity-solutions/embedded-wireless/sx-pcebe
TI Linux Repo:
https://git.ti.com/cgit/ti-linux-kernel/ti-linux-kernel/?h=ti-linux-6.6.yThank you.
Regards,
Parth P
Quoted 33 lines · Baochen Qiang
Quoted 18 lines · Baochen Qiang
Quoted 7 lines · Parth Panchoil
Hi,
I am currently debugging the
ath12k_pci_enable_ltssmstart up crash/bug
with the mainline kernel on my system and would like to share my observations so far:Quoted 5 lines
The ath12k mainline driver gets stuck at this specific line:
https://github.com/torvalds/linux/blob/9c5968db9e625019a0ee5226c7eebef5519d366a/drivers/net/wireless/ath/ath12k/pci.c#L295
in the ath12k_pci_enable_ltssm which attempts to readGCC_GCC_PCIE_HOT_RST, particularly
https://github.com/torvalds/linux/blob/9c5968db9e625019a0ee5226c7eebef5519d366a/drivers/net/wireless/ath/ath12k/pci.c#L1209thanks for the narrow down, really helpful.
Quoted 8 lines
We internally have observed this issue, although at a different line:
For now I am suspecting that
GCC_GCC_PCIE_HOT_RSTis not a valid register on WLAN target
side, I will check internally and get back.Parth, could you do below change and try again?
Review
Quoted 2 lines
(quoted diff) +1
-1-#define GCC_GCC_PCIE_HOT_RST 0x1e38338 +#define GCC_GCC_PCIE_HOT_RST 0x1e40304
Hi Baochen,
Thanks for the hint regarding the change.
I tested this change on top of the ath-202504172310 tag on the TI AM69 platform and can confirm that the startup crash no longer occurs. Interestingly, this issue was not observed on other platforms like the NXP iMX8MP.
If I understand correctly, the change is related to a WLAN target register.
Could you help clarify why this issue only affects certain platforms (hosts)?
Regards,
Parth P
Quoted 199 lines
Quoted 198 lines
Quoted 196 lines
Interestingly, within the same function, the line val =
ath12k_pci_read32(ab, PCIE_PCIE_PARF_LTSSM)successfully reads the
expected value0x111forPCIE_PCIE_PARF_LTSSM.I am continuing to debug from my end, although my understanding of the
ath12k driver is limited. Any leads, suggestions, or hints to help
resolve this issue would be greatly appreciated.Thank you.
Regards,
Parth PQuoted 175 lines · Parth Pancholi
I appreciate your response, Baochen.
I have been working on enabling mainline kernel support on my TI
AM69-
SK board to test the mainline ath12k driver on my system.Using the mainline kernel repository for the ath drivers [1], I made
the following observation:
While the exact crash observed earlier is no longer present, the
system
hangs upon loading the ath12k mainline driver, displaying the messages
below.root@am69-sk:~# modprobe ath12k debug_mask=0xffffffff [ 1121.996554] ath12k_pci 0000:01:00.0: BAR 0 [mem 0x4410200000- 0x44103fffff 64bit]: assigned [ 1122.004884] ath12k_pci 0000:01:00.0: enabling device (0000 -Quoted 1 line
0002)
[ 1122.011818] ath12k_pci 0000:01:00.0: MSI vectors: 16 [ 1122.016798] ath12k_pci 0000:01:00.0: Hardware name: wcn7850 hw2.0 [ 1122.040183] NET: Registered PF_QIPCRTR protocol familyroot@am69-sk:~# uname -a Linux am69-sk 6.13.0-rc7-wt-ath-ge7ef944b3e2c-dirty #2 SMP PREEMPT Wed Jan 22 16:55:17 CET 2025 aarch64 GNU/Linuxroot@am69-sk:~# lspci 0000:00:00.0 PCI bridge: Texas Instruments Device b012 0000:01:00.0 Network controller: Qualcomm Technologies, Inc WCN785x Wi- Fi 7(802.11be) 320MHz 2x2 [FastConnect 7800] (rev 01) 0001:00:00.0 PCI bridge: Texas Instruments Device b012 0002:00:00.0 PCI bridge: Texas Instruments Device b012Do you have any insights into what might still be missing or incorrect
in my setup?Regards,
Parth PQuoted 121 lines · Baochen Qiang
Quoted 8 lines · Parth Panchoil
Hi All,
I am performing tests on the SX-PCEBE Wi-Fi module, which utilizes
the
ATH12k driver, on the Texas Instruments AM69-SK board. The board is running the TI Linux Kernel from the ti-linux- 6.6.y6.6 is too old, and besides we don;t support customer kernel.
Could you try latest ath tree [1] or the mainline tree [2]?
[1]
https://git.kernel.org/pub/scm/linux/kernel/git/ath/ath.git/
[2]
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/If the issue is still seen, please enable verbose ath12k log using
below command and help
collect dmesg logs:sudo modprobe ath12k debug_mask=0xffffffff
One more thing, the open-WRT patch is overkill, can you narrow down
to find which line of
code inath12k_pci_enable_ltssm()is causing this issue?Quoted 85 lines
branch. During testing, I observed a kernel crash from the ATH12k
driver as soon as the probe is called. The crash log is as follows:[ 9.492631] Kernel panic - not syncing: Asynchronous SError
Interrupt
[ 9.492634] CPU: 7 PID: 222 Comm: (udev-worker) Not tainted
6.6.58-
01497-ga7758da17c28-dirty #1
[ 9.492638] Hardware name: Texas Instruments AM69 SK (DT)
[ 9.492640] Call trace:
[ 9.492642]dump_backtrace+0x94/0xec
[ 9.492658]show_stack+0x18/0x24
[ 9.492662]dump_stack_lvl+0x48/0x60
[ 9.492669]dump_stack+0x18/0x24
[ 9.492672]panic+0x320/0x378
[ 9.492677]nmi_panic+0x8c/0x90
[ 9.492681]arm64_serror_panic+0x6c/0x78
[ 9.492686]do_serror+0x3c/0x78
[ 9.492692]el1h_64_error_handler+0x34/0x4c
[ 9.492697]el1h_64_error+0x64/0x68
[ 9.492700]ath12k_pci_read32+0x1bc/0x1e8[ath12k] [ 9.492725]ath12k_pci_power_up+0xdc/0x340[ath12k] [ 9.492747]ath12k_core_init+0x2c/0xa8[ath12k] [ 9.492769]ath12k_pci_probe+0x698/0x908[ath12k] [ 9.492791]pci_device_probe+0xa8/0x16c
[ 9.492800]really_probe+0x110/0x27c
[ 9.492805]__driver_probe_device+0x78/0x12c
[ 9.492808]driver_probe_device+0x3c/0x118
[ 9.492810]__driver_attach+0x74/0x124
[ 9.492813]bus_for_each_dev+0x78/0xd8
[ 9.492819]driver_attach+0x24/0x30
[ 9.492824]bus_add_driver+0xe4/0x208
[ 9.492828]driver_register+0x60/0x128
[ 9.492831]__pci_register_driver+0x44/0x50
[ 9.492835]ath12k_pci_init+0x2c/0x6c[ath12k] [ 9.492858]do_one_initcall+0x70/0x1b4
[ 9.492861]do_init_module+0x58/0x1e4
[ 9.492867]load_module+0x19bc/0x1a8c
[ 9.492869]init_module_from_file+0x88/0xc4
[ 9.492873]__arm64_sys_finit_module+0x1c0/0x2ac[ 9.492877]invoke_syscall+0x44/0x108
[ 9.492882] el0_svc_common.constprop.0+0xc0/0xe0 [ 9.492885]do_el0_svc+0x1c/0x28
[ 9.492889]el0_svc+0x2c/0x84
[ 9.492892]el0t_64_sync_handler+0xc0/0xc4
[ 9.492895]el0t_64_sync+0x190/0x194
[ 9.492899] SMP: stopping secondary CPUs
[ 9.492908] Kernel Offset: disabled
[ 9.492909] CPU features: 0x0,80000200,28020000,1000420b [ 9.492913] Memory Limit: noneUpon searching online, I found the OpenWRT patch that appears to
address a similar issue: OpenWRT Patch: Prevent LTSSM Startup
Crash.
https://git.openwrt.org/?p=openwrt/openwrt.git;a=blob;f=package/kernel/mac80211/patches/ath12k/100-ath12k-prevent-ltssm-startup-crash.patch;h=cd85a0f6aa2652d62bfbea04e9bcca3bcf831b7f;hb=935b2b7dcef61b2893ed5dff307dd8f8a1156899
With the above patch applied, I do not see the crash anymore.Could anyone confirm if this issue has been reported before/known
bug
or provide any insights?
Any additional information or suggestions would be greatly appreciated.Details about the test setup,
TI-AM69-SK board:
https://www.ti.com/tool/SK-AM69?keyMatch=am69%20sk&tisearch=universal_search
Silex WiFi card SX-PCEBE:
https://www.silextechnology.com/connectivity-solutions/embedded-wireless/sx-pcebe
TI Linux Repo:
https://git.ti.com/cgit/ti-linux-kernel/ti-linux-kernel/?h=ti-linux-6.6.yThank you.
Regards,
Parth P
Quoted 33 lines · Parth Panchoil
Quoted 32 lines · Baochen Qiang
Quoted 18 lines · Baochen Qiang
Quoted 7 lines · Parth Panchoil
Hi,
I am currently debugging the
ath12k_pci_enable_ltssmstart up crash/bug
with the mainline kernel on my system and would like to share my observations so far:Quoted 5 lines
The ath12k mainline driver gets stuck at this specific line:
https://github.com/torvalds/linux/blob/9c5968db9e625019a0ee5226c7eebef5519d366a/drivers/net/wireless/ath/ath12k/pci.c#L295
in the ath12k_pci_enable_ltssm which attempts to readGCC_GCC_PCIE_HOT_RST, particularly
https://github.com/torvalds/linux/blob/9c5968db9e625019a0ee5226c7eebef5519d366a/drivers/net/wireless/ath/ath12k/pci.c#L1209thanks for the narrow down, really helpful.
Quoted 8 lines
We internally have observed this issue, although at a different line:
For now I am suspecting that
GCC_GCC_PCIE_HOT_RSTis not a valid register on WLAN target
side, I will check internally and get back.Parth, could you do below change and try again?
Quoted 8 lines
Review
Quoted 3 lines
(quoted diff) +1
-1-#define GCC_GCC_PCIE_HOT_RST 0x1e38338 +#define GCC_GCC_PCIE_HOT_RST 0x1e40304Hi Baochen,
Thanks for the hint regarding the change.
I tested this change on top of the ath-202504172310 tag on the TI AM69 platform and can confirm that the startup crash no longer occurs.
great it helps
Quoted 6 lines
Interestingly, this issue was not observed on other platforms like the NXP iMX8MP.
If I understand correctly, the change is related to a WLAN target register.
correct
Quoted 2 lines
Could you help clarify why this issue only affects certain platforms (hosts)?
GCC_GCC_PCIE_HOT_RST is wrongly defined, normally this should not cause any critical issue, because IMO the RC is expected to return 0xffffffff when accessing a non-exist register. However in your case kernel crashes, so seems RC does not behave well, maybe not following spec?
anyway I will submit a patch to fix it.
Quoted 204 lines
Regards,
Parth PQuoted 199 lines
Quoted 198 lines
Quoted 196 lines
Interestingly, within the same function, the line val =
ath12k_pci_read32(ab, PCIE_PCIE_PARF_LTSSM)successfully reads the
expected value0x111forPCIE_PCIE_PARF_LTSSM.I am continuing to debug from my end, although my understanding of the
ath12k driver is limited. Any leads, suggestions, or hints to help
resolve this issue would be greatly appreciated.Thank you.
Regards,
Parth PQuoted 175 lines · Parth Pancholi
I appreciate your response, Baochen.
I have been working on enabling mainline kernel support on my TI
AM69-
SK board to test the mainline ath12k driver on my system.Using the mainline kernel repository for the ath drivers [1], I made
the following observation:
While the exact crash observed earlier is no longer present, the
system
hangs upon loading the ath12k mainline driver, displaying the messages
below.root@am69-sk:~# modprobe ath12k debug_mask=0xffffffff [ 1121.996554] ath12k_pci 0000:01:00.0: BAR 0 [mem 0x4410200000- 0x44103fffff 64bit]: assigned [ 1122.004884] ath12k_pci 0000:01:00.0: enabling device (0000 -Quoted 1 line
0002)
[ 1122.011818] ath12k_pci 0000:01:00.0: MSI vectors: 16 [ 1122.016798] ath12k_pci 0000:01:00.0: Hardware name: wcn7850 hw2.0 [ 1122.040183] NET: Registered PF_QIPCRTR protocol familyroot@am69-sk:~# uname -a Linux am69-sk 6.13.0-rc7-wt-ath-ge7ef944b3e2c-dirty #2 SMP PREEMPT Wed Jan 22 16:55:17 CET 2025 aarch64 GNU/Linuxroot@am69-sk:~# lspci 0000:00:00.0 PCI bridge: Texas Instruments Device b012 0000:01:00.0 Network controller: Qualcomm Technologies, Inc WCN785x Wi- Fi 7(802.11be) 320MHz 2x2 [FastConnect 7800] (rev 01) 0001:00:00.0 PCI bridge: Texas Instruments Device b012 0002:00:00.0 PCI bridge: Texas Instruments Device b012Do you have any insights into what might still be missing or incorrect
in my setup?Regards,
Parth PQuoted 121 lines · Baochen Qiang
Quoted 8 lines · Parth Panchoil
Hi All,
I am performing tests on the SX-PCEBE Wi-Fi module, which utilizes
the
ATH12k driver, on the Texas Instruments AM69-SK board. The board is running the TI Linux Kernel from the ti-linux- 6.6.y6.6 is too old, and besides we don;t support customer kernel.
Could you try latest ath tree [1] or the mainline tree [2]?
[1]
https://git.kernel.org/pub/scm/linux/kernel/git/ath/ath.git/
[2]
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/If the issue is still seen, please enable verbose ath12k log using
below command and help
collect dmesg logs:sudo modprobe ath12k debug_mask=0xffffffff
One more thing, the open-WRT patch is overkill, can you narrow down
to find which line of
code inath12k_pci_enable_ltssm()is causing this issue?Quoted 85 lines
branch. During testing, I observed a kernel crash from the ATH12k
driver as soon as the probe is called. The crash log is as follows:[ 9.492631] Kernel panic - not syncing: Asynchronous SError
Interrupt
[ 9.492634] CPU: 7 PID: 222 Comm: (udev-worker) Not tainted
6.6.58-
01497-ga7758da17c28-dirty #1
[ 9.492638] Hardware name: Texas Instruments AM69 SK (DT)
[ 9.492640] Call trace:
[ 9.492642]dump_backtrace+0x94/0xec
[ 9.492658]show_stack+0x18/0x24
[ 9.492662]dump_stack_lvl+0x48/0x60
[ 9.492669]dump_stack+0x18/0x24
[ 9.492672]panic+0x320/0x378
[ 9.492677]nmi_panic+0x8c/0x90
[ 9.492681]arm64_serror_panic+0x6c/0x78
[ 9.492686]do_serror+0x3c/0x78
[ 9.492692]el1h_64_error_handler+0x34/0x4c
[ 9.492697]el1h_64_error+0x64/0x68
[ 9.492700]ath12k_pci_read32+0x1bc/0x1e8[ath12k] [ 9.492725]ath12k_pci_power_up+0xdc/0x340[ath12k] [ 9.492747]ath12k_core_init+0x2c/0xa8[ath12k] [ 9.492769]ath12k_pci_probe+0x698/0x908[ath12k] [ 9.492791]pci_device_probe+0xa8/0x16c
[ 9.492800]really_probe+0x110/0x27c
[ 9.492805]__driver_probe_device+0x78/0x12c
[ 9.492808]driver_probe_device+0x3c/0x118
[ 9.492810]__driver_attach+0x74/0x124
[ 9.492813]bus_for_each_dev+0x78/0xd8
[ 9.492819]driver_attach+0x24/0x30
[ 9.492824]bus_add_driver+0xe4/0x208
[ 9.492828]driver_register+0x60/0x128
[ 9.492831]__pci_register_driver+0x44/0x50
[ 9.492835]ath12k_pci_init+0x2c/0x6c[ath12k] [ 9.492858]do_one_initcall+0x70/0x1b4
[ 9.492861]do_init_module+0x58/0x1e4
[ 9.492867]load_module+0x19bc/0x1a8c
[ 9.492869]init_module_from_file+0x88/0xc4
[ 9.492873]__arm64_sys_finit_module+0x1c0/0x2ac[ 9.492877]invoke_syscall+0x44/0x108
[ 9.492882] el0_svc_common.constprop.0+0xc0/0xe0 [ 9.492885]do_el0_svc+0x1c/0x28
[ 9.492889]el0_svc+0x2c/0x84
[ 9.492892]el0t_64_sync_handler+0xc0/0xc4
[ 9.492895]el0t_64_sync+0x190/0x194
[ 9.492899] SMP: stopping secondary CPUs
[ 9.492908] Kernel Offset: disabled
[ 9.492909] CPU features: 0x0,80000200,28020000,1000420b [ 9.492913] Memory Limit: noneUpon searching online, I found the OpenWRT patch that appears to
address a similar issue: OpenWRT Patch: Prevent LTSSM Startup
Crash.
https://git.openwrt.org/?p=openwrt/openwrt.git;a=blob;f=package/kernel/mac80211/patches/ath12k/100-ath12k-prevent-ltssm-startup-crash.patch;h=cd85a0f6aa2652d62bfbea04e9bcca3bcf831b7f;hb=935b2b7dcef61b2893ed5dff307dd8f8a1156899
With the above patch applied, I do not see the crash anymore.Could anyone confirm if this issue has been reported before/known
bug
or provide any insights?
Any additional information or suggestions would be greatly appreciated.Details about the test setup,
TI-AM69-SK board:
https://www.ti.com/tool/SK-AM69?keyMatch=am69%20sk&tisearch=universal_search
Silex WiFi card SX-PCEBE:
https://www.silextechnology.com/connectivity-solutions/embedded-wireless/sx-pcebe
TI Linux Repo:
https://git.ti.com/cgit/ti-linux-kernel/ti-linux-kernel/?h=ti-linux-6.6.yThank you.
Regards,
Parth P
Hello,
Quoted 14 lines · Baochen Qiang
Quoted 13 lines · Parth Panchoil
Quoted 12 lines · Baochen Qiang
Quoted 11 lines · Baochen Qiang
Quoted 10 lines · Parth Panchoil
I am currently debugging the
ath12k_pci_enable_ltssmstart up crash/bug
with the mainline kernel on my system and would like to share my observations so far:The ath12k mainline driver gets stuck at this specific line:
https://github.com/torvalds/linux/blob/9c5968db9e625019a0ee5226c7eebef5519d366a/drivers/net/wireless/ath/ath12k/pci.c#L295
in the ath12k_pci_enable_ltssm which attempts to readGCC_GCC_PCIE_HOT_RST, particularly
https://github.com/torvalds/linux/blob/9c5968db9e625019a0ee5226c7eebef5519d366a/drivers/net/wireless/ath/ath12k/pci.c#L1209
...
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
(quoted diff) +1
-1-#define GCC_GCC_PCIE_HOT_RST 0x1e38338 +#define GCC_GCC_PCIE_HOT_RST 0x1e40304
...
Quoted 7 lines
GCC_GCC_PCIE_HOT_RSTis wrongly defined, normally this should not cause any critical issue, because IMO the RC is expected to return0xffffffffwhen accessing a non-exist register. However in your case kernel crashes, so seems RC does not behave well, maybe not following spec?anyway I will submit a patch to fix it.
Any update on this? I can easily submit a patch myself, just let me know. Because of this bug we have a crash that is preventing us to use the device.
Francesco
Quoted 32 lines · Francesco Dolcini
Hello,
Quoted 14 lines · Baochen Qiang
Quoted 13 lines · Parth Panchoil
Quoted 12 lines · Baochen Qiang
Quoted 11 lines · Baochen Qiang
Quoted 10 lines · Parth Panchoil
I am currently debugging the
ath12k_pci_enable_ltssmstart up crash/bug
with the mainline kernel on my system and would like to share my observations so far:The ath12k mainline driver gets stuck at this specific line:
https://github.com/torvalds/linux/blob/9c5968db9e625019a0ee5226c7eebef5519d366a/drivers/net/wireless/ath/ath12k/pci.c#L295
in the ath12k_pci_enable_ltssm which attempts to readGCC_GCC_PCIE_HOT_RST, particularly
https://github.com/torvalds/linux/blob/9c5968db9e625019a0ee5226c7eebef5519d366a/drivers/net/wireless/ath/ath12k/pci.c#L1209...
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
(quoted diff) +1
-1-#define GCC_GCC_PCIE_HOT_RST 0x1e38338 +#define GCC_GCC_PCIE_HOT_RST 0x1e40304...
Quoted 6 lines
GCC_GCC_PCIE_HOT_RSTis wrongly defined, normally this should not cause any critical issue, because IMO the RC is expected to return0xffffffffwhen accessing a non-exist register. However in your case kernel crashes, so seems RC does not behave well, maybe not following spec?
Quoted 6 lines
Quoted 1 line
anyway I will submit a patch to fix it.
Any update on this? I can easily submit a patch myself, just let me know. Because of this bug we have a crash that is preventing us to use the device.
The patch has been submitted for internal review. Will make it public available once approved internally.
Quoted 2 lines
Francesco