Skip to contentSkip to conversation

wifi: ath12k: start-up crash with WCN7850 hw2.0 on TI AM69-SK board

DISCUSSION Parth Panchoil 4 participants 11 messages Jan 21, 2025 – May 16, 2025 Qualcomm ATH12K wireless driver

lore
Share & tools
lore
#0 lore raw

Hi All,

I am performing tests on the SX-PCEBE Wi-Fi module, which utilizes the ATH12k driver, on the Texas Instruments AM69-SK board.
The board is running the TI Linux Kernel from the ti-linux-6.6.y branch. During testing, I observed a kernel crash from the ATH12k driver as soon as the probe is called. The crash log is as follows:

[    9.492631] Kernel panic - not syncing: Asynchronous SError
Interrupt
[    9.492634] CPU: 7 PID: 222 Comm: (udev-worker) Not tainted 6.6.58-
01497-ga7758da17c28-dirty #1
[    9.492638] Hardware name: Texas Instruments AM69 SK (DT)
[    9.492640] Call trace:
[    9.492642]  dump_backtrace+0x94/0xec
[    9.492658]  show_stack+0x18/0x24
[    9.492662]  dump_stack_lvl+0x48/0x60
[    9.492669]  dump_stack+0x18/0x24
[    9.492672]  panic+0x320/0x378
[    9.492677]  nmi_panic+0x8c/0x90
[    9.492681]  arm64_serror_panic+0x6c/0x78
[    9.492686]  do_serror+0x3c/0x78
[    9.492692]  el1h_64_error_handler+0x34/0x4c
[    9.492697]  el1h_64_error+0x64/0x68
[    9.492700]  ath12k_pci_read32+0x1bc/0x1e8 [ath12k]
[    9.492725]  ath12k_pci_power_up+0xdc/0x340 [ath12k]
[    9.492747]  ath12k_core_init+0x2c/0xa8 [ath12k]
[    9.492769]  ath12k_pci_probe+0x698/0x908 [ath12k]
[    9.492791]  pci_device_probe+0xa8/0x16c
[    9.492800]  really_probe+0x110/0x27c
[    9.492805]  __driver_probe_device+0x78/0x12c
[    9.492808]  driver_probe_device+0x3c/0x118
[    9.492810]  __driver_attach+0x74/0x124
[    9.492813]  bus_for_each_dev+0x78/0xd8
[    9.492819]  driver_attach+0x24/0x30
[    9.492824]  bus_add_driver+0xe4/0x208
[    9.492828]  driver_register+0x60/0x128
[    9.492831]  __pci_register_driver+0x44/0x50
[    9.492835]  ath12k_pci_init+0x2c/0x6c [ath12k]
[    9.492858]  do_one_initcall+0x70/0x1b4
[    9.492861]  do_init_module+0x58/0x1e4
[    9.492867]  load_module+0x19bc/0x1a8c
[    9.492869]  init_module_from_file+0x88/0xc4
[    9.492873]  __arm64_sys_finit_module+0x1c0/0x2ac
[    9.492877]  invoke_syscall+0x44/0x108
[    9.492882]  el0_svc_common.constprop.0+0xc0/0xe0
[    9.492885]  do_el0_svc+0x1c/0x28
[    9.492889]  el0_svc+0x2c/0x84
[    9.492892]  el0t_64_sync_handler+0xc0/0xc4
[    9.492895]  el0t_64_sync+0x190/0x194
[    9.492899] SMP: stopping secondary CPUs
[    9.492908] Kernel Offset: disabled
[    9.492909] CPU features: 0x0,80000200,28020000,1000420b
[    9.492913] Memory Limit: none

Upon searching online, I found the OpenWRT patch that appears to address a similar issue: OpenWRT Patch: Prevent LTSSM Startup Crash.
https://git.openwrt.org/?p=openwrt/openwrt.git;a=blob;f=package/kernel/mac80211/patches/ath12k/100-ath12k-prevent-ltssm-startup-crash.patch;h=cd85a0f6aa2652d62bfbea04e9bcca3bcf831b7f;hb=935b2b7dcef61b2893ed5dff307dd8f8a1156899
With the above patch applied, I do not see the crash anymore.

Could anyone confirm if this issue has been reported before/known bug or provide any insights?
Any additional information or suggestions would be greatly appreciated.

Details about the test setup,
TI-AM69-SK board:
https://www.ti.com/tool/SK-AM69?keyMatch=am69%20sk&tisearch=universal_search
Silex WiFi card SX-PCEBE:
https://www.silextechnology.com/connectivity-solutions/embedded-wireless/sx-pcebe
TI Linux Repo:
https://git.ti.com/cgit/ti-linux-kernel/ti-linux-kernel/?h=ti-linux-6.6.y

Thank you.

Regards,
Parth P

#1 lore raw
Quoted 5 lines · Parth Panchoil

Hi All,

I am performing tests on the SX-PCEBE Wi-Fi module, which utilizes the ATH12k driver, on the Texas Instruments AM69-SK board.
The board is running the TI Linux Kernel from the ti-linux-6.6.y

6.6 is too old, and besides we don;t support customer kernel.

Could you try latest ath tree [1] or the mainline tree [2]?

[1] https://git.kernel.org/pub/scm/linux/kernel/git/ath/ath.git/
[2] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/

If the issue is still seen, please enable verbose ath12k log using below command and help collect dmesg logs:

	sudo modprobe ath12k debug_mask=0xffffffff

One more thing, the open-WRT patch is overkill, can you narrow down to find which line of code in ath12k_pci_enable_ltssm() is causing this issue?

Quoted 72 lines

branch. During testing, I observed a kernel crash from the ATH12k driver as soon as the probe is called. The crash log is as follows:

[    9.492631] Kernel panic - not syncing: Asynchronous SError
Interrupt
[    9.492634] CPU: 7 PID: 222 Comm: (udev-worker) Not tainted 6.6.58-
01497-ga7758da17c28-dirty #1
[    9.492638] Hardware name: Texas Instruments AM69 SK (DT)
[    9.492640] Call trace:
[    9.492642]  dump_backtrace+0x94/0xec
[    9.492658]  show_stack+0x18/0x24
[    9.492662]  dump_stack_lvl+0x48/0x60
[    9.492669]  dump_stack+0x18/0x24
[    9.492672]  panic+0x320/0x378
[    9.492677]  nmi_panic+0x8c/0x90
[    9.492681]  arm64_serror_panic+0x6c/0x78
[    9.492686]  do_serror+0x3c/0x78
[    9.492692]  el1h_64_error_handler+0x34/0x4c
[    9.492697]  el1h_64_error+0x64/0x68
[    9.492700]  ath12k_pci_read32+0x1bc/0x1e8 [ath12k]
[    9.492725]  ath12k_pci_power_up+0xdc/0x340 [ath12k]
[    9.492747]  ath12k_core_init+0x2c/0xa8 [ath12k]
[    9.492769]  ath12k_pci_probe+0x698/0x908 [ath12k]
[    9.492791]  pci_device_probe+0xa8/0x16c
[    9.492800]  really_probe+0x110/0x27c
[    9.492805]  __driver_probe_device+0x78/0x12c
[    9.492808]  driver_probe_device+0x3c/0x118
[    9.492810]  __driver_attach+0x74/0x124
[    9.492813]  bus_for_each_dev+0x78/0xd8
[    9.492819]  driver_attach+0x24/0x30
[    9.492824]  bus_add_driver+0xe4/0x208
[    9.492828]  driver_register+0x60/0x128
[    9.492831]  __pci_register_driver+0x44/0x50
[    9.492835]  ath12k_pci_init+0x2c/0x6c [ath12k]
[    9.492858]  do_one_initcall+0x70/0x1b4
[    9.492861]  do_init_module+0x58/0x1e4
[    9.492867]  load_module+0x19bc/0x1a8c
[    9.492869]  init_module_from_file+0x88/0xc4
[    9.492873]  __arm64_sys_finit_module+0x1c0/0x2ac
[    9.492877]  invoke_syscall+0x44/0x108
[    9.492882]  el0_svc_common.constprop.0+0xc0/0xe0
[    9.492885]  do_el0_svc+0x1c/0x28
[    9.492889]  el0_svc+0x2c/0x84
[    9.492892]  el0t_64_sync_handler+0xc0/0xc4
[    9.492895]  el0t_64_sync+0x190/0x194
[    9.492899] SMP: stopping secondary CPUs
[    9.492908] Kernel Offset: disabled
[    9.492909] CPU features: 0x0,80000200,28020000,1000420b
[    9.492913] Memory Limit: none

Upon searching online, I found the OpenWRT patch that appears to address a similar issue: OpenWRT Patch: Prevent LTSSM Startup Crash.
https://git.openwrt.org/?p=openwrt/openwrt.git;a=blob;f=package/kernel/mac80211/patches/ath12k/100-ath12k-prevent-ltssm-startup-crash.patch;h=cd85a0f6aa2652d62bfbea04e9bcca3bcf831b7f;hb=935b2b7dcef61b2893ed5dff307dd8f8a1156899
With the above patch applied, I do not see the crash anymore.

Could anyone confirm if this issue has been reported before/known bug or provide any insights?
Any additional information or suggestions would be greatly appreciated.

Details about the test setup,
TI-AM69-SK board:
https://www.ti.com/tool/SK-AM69?keyMatch=am69%20sk&tisearch=universal_search
Silex WiFi card SX-PCEBE:
https://www.silextechnology.com/connectivity-solutions/embedded-wireless/sx-pcebe
TI Linux Repo:
https://git.ti.com/cgit/ti-linux-kernel/ti-linux-kernel/?h=ti-linux-6.6.y

Thank you.

Regards,
Parth P

#2 lore raw

I appreciate your response, Baochen.

I have been working on enabling mainline kernel support on my TI AM69- SK board to test the mainline ath12k driver on my system.

Using the mainline kernel repository for the ath drivers [1], I made the following observation:
While the exact crash observed earlier is no longer present, the system hangs upon loading the ath12k mainline driver, displaying the messages below.

root@am69-sk:~# modprobe ath12k debug_mask=0xffffffff
[ 1121.996554] ath12k_pci 0000:01:00.0: BAR 0 [mem 0x4410200000-
0x44103fffff 64bit]: assigned
[ 1122.004884] ath12k_pci 0000:01:00.0: enabling device (0000 -> 0002)
[ 1122.011818] ath12k_pci 0000:01:00.0: MSI vectors: 16
[ 1122.016798] ath12k_pci 0000:01:00.0: Hardware name: wcn7850 hw2.0
[ 1122.040183] NET: Registered PF_QIPCRTR protocol family
root@am69-sk:~# uname -a
Linux am69-sk 6.13.0-rc7-wt-ath-ge7ef944b3e2c-dirty #2 SMP PREEMPT Wed
Jan 22 16:55:17 CET 2025 aarch64 GNU/Linux
root@am69-sk:~# lspci
0000:00:00.0 PCI bridge: Texas Instruments Device b012
0000:01:00.0 Network controller: Qualcomm Technologies, Inc WCN785x Wi-
Fi 7(802.11be) 320MHz 2x2 [FastConnect 7800] (rev 01)
0001:00:00.0 PCI bridge: Texas Instruments Device b012
0002:00:00.0 PCI bridge: Texas Instruments Device b012

Do you have any insights into what might still be missing or incorrect in my setup?

Regards,
Parth P

Quoted 107 lines · Baochen Qiang
Quoted 6 lines · Parth Panchoil

Hi All,

I am performing tests on the SX-PCEBE Wi-Fi module, which utilizes the
ATH12k driver, on the Texas Instruments AM69-SK board. The board is running the TI Linux Kernel from the ti-linux-6.6.y

6.6 is too old, and besides we don;t support customer kernel.

Could you try latest ath tree [1] or the mainline tree [2]?

[1] https://git.kernel.org/pub/scm/linux/kernel/git/ath/ath.git/
[2]
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/

If the issue is still seen, please enable verbose ath12k log using below command and help
collect dmesg logs:

        sudo modprobe ath12k debug_mask=0xffffffff

One more thing, the open-WRT patch is overkill, can you narrow down to find which line of
code in ath12k_pci_enable_ltssm() is causing this issue?

Quoted 76 lines

branch. During testing, I observed a kernel crash from the ATH12k driver as soon as the probe is called. The crash log is as follows:

[    9.492631] Kernel panic - not syncing: Asynchronous SError Interrupt
[    9.492634] CPU: 7 PID: 222 Comm: (udev-worker) Not tainted 6.6.58-
01497-ga7758da17c28-dirty #1
[    9.492638] Hardware name: Texas Instruments AM69 SK (DT) [    9.492640] Call trace:
[    9.492642]  dump_backtrace+0x94/0xec
[    9.492658]  show_stack+0x18/0x24
[    9.492662]  dump_stack_lvl+0x48/0x60
[    9.492669]  dump_stack+0x18/0x24
[    9.492672]  panic+0x320/0x378
[    9.492677]  nmi_panic+0x8c/0x90
[    9.492681]  arm64_serror_panic+0x6c/0x78
[    9.492686]  do_serror+0x3c/0x78
[    9.492692]  el1h_64_error_handler+0x34/0x4c
[    9.492697]  el1h_64_error+0x64/0x68
[    9.492700]  ath12k_pci_read32+0x1bc/0x1e8 [ath12k] [    9.492725]  ath12k_pci_power_up+0xdc/0x340 [ath12k] [    9.492747]  ath12k_core_init+0x2c/0xa8 [ath12k] [    9.492769]  ath12k_pci_probe+0x698/0x908 [ath12k] [    9.492791]  pci_device_probe+0xa8/0x16c
[    9.492800]  really_probe+0x110/0x27c
[    9.492805]  __driver_probe_device+0x78/0x12c
[    9.492808]  driver_probe_device+0x3c/0x118
[    9.492810]  __driver_attach+0x74/0x124
[    9.492813]  bus_for_each_dev+0x78/0xd8
[    9.492819]  driver_attach+0x24/0x30
[    9.492824]  bus_add_driver+0xe4/0x208
[    9.492828]  driver_register+0x60/0x128
[    9.492831]  __pci_register_driver+0x44/0x50
[    9.492835]  ath12k_pci_init+0x2c/0x6c [ath12k]
[    9.492858]  do_one_initcall+0x70/0x1b4
[    9.492861]  do_init_module+0x58/0x1e4
[    9.492867]  load_module+0x19bc/0x1a8c
[    9.492869]  init_module_from_file+0x88/0xc4
[    9.492873]  __arm64_sys_finit_module+0x1c0/0x2ac [    9.492877]  invoke_syscall+0x44/0x108
[    9.492882]  el0_svc_common.constprop.0+0xc0/0xe0 [    9.492885]  do_el0_svc+0x1c/0x28
[    9.492889]  el0_svc+0x2c/0x84
[    9.492892]  el0t_64_sync_handler+0xc0/0xc4
[    9.492895]  el0t_64_sync+0x190/0x194
[    9.492899] SMP: stopping secondary CPUs
[    9.492908] Kernel Offset: disabled
[    9.492909] CPU features: 0x0,80000200,28020000,1000420b [    9.492913] Memory Limit: none

Upon searching online, I found the OpenWRT patch that appears to address a similar issue: OpenWRT Patch: Prevent LTSSM Startup Crash.
https://git.openwrt.org/?p=openwrt/openwrt.git;a=blob;f=package/kernel/mac80211/patches/ath12k/100-ath12k-prevent-ltssm-startup-crash.patch;h=cd85a0f6aa2652d62bfbea04e9bcca3bcf831b7f;hb=935b2b7dcef61b2893ed5dff307dd8f8a1156899
With the above patch applied, I do not see the crash anymore.

Could anyone confirm if this issue has been reported before/known bug
or provide any insights?
Any additional information or suggestions would be greatly appreciated.

Details about the test setup,
TI-AM69-SK board:
https://www.ti.com/tool/SK-AM69?keyMatch=am69%20sk&tisearch=universal_search
Silex WiFi card SX-PCEBE:
https://www.silextechnology.com/connectivity-solutions/embedded-wireless/sx-pcebe
TI Linux Repo:
https://git.ti.com/cgit/ti-linux-kernel/ti-linux-kernel/?h=ti-linux-6.6.y

Thank you.

Regards,
Parth P

#3 lore raw

Hi,

I am currently debugging the ath12k_pci_enable_ltssm start up crash/bug with the mainline kernel on my system and would like to share my observations so far:

The ath12k mainline driver gets stuck at this specific line:
https://github.com/torvalds/linux/blob/9c5968db9e625019a0ee5226c7eebef5519d366a/drivers/net/wireless/ath/ath12k/pci.c#L295
in the ath12k_pci_enable_ltssm which attempts to read GCC_GCC_PCIE_HOT_RST, particularly
https://github.com/torvalds/linux/blob/9c5968db9e625019a0ee5226c7eebef5519d366a/drivers/net/wireless/ath/ath12k/pci.c#L1209

Interestingly, within the same function, the line val = ath12k_pci_read32(ab, PCIE_PCIE_PARF_LTSSM) successfully reads the expected value 0x111 for PCIE_PCIE_PARF_LTSSM.

I am continuing to debug from my end, although my understanding of the ath12k driver is limited. Any leads, suggestions, or hints to help resolve this issue would be greatly appreciated.

Thank you.

Regards,
Parth P

Quoted 155 lines · Parth Pancholi

I appreciate your response, Baochen.

I have been working on enabling mainline kernel support on my TI AM69-
SK board to test the mainline ath12k driver on my system.

Using the mainline kernel repository for the ath drivers [1], I made the following observation:
While the exact crash observed earlier is no longer present, the system
hangs upon loading the ath12k mainline driver, displaying the messages
below.

root@am69-sk:~# modprobe ath12k debug_mask=0xffffffff
[ 1121.996554] ath12k_pci 0000:01:00.0: BAR 0 [mem 0x4410200000-
0x44103fffff 64bit]: assigned
[ 1122.004884] ath12k_pci 0000:01:00.0: enabling device (0000 ->
0002)
[ 1122.011818] ath12k_pci 0000:01:00.0: MSI vectors: 16
[ 1122.016798] ath12k_pci 0000:01:00.0: Hardware name: wcn7850 hw2.0
[ 1122.040183] NET: Registered PF_QIPCRTR protocol family
root@am69-sk:~# uname -a
Linux am69-sk 6.13.0-rc7-wt-ath-ge7ef944b3e2c-dirty #2 SMP PREEMPT
Wed
Jan 22 16:55:17 CET 2025 aarch64 GNU/Linux
root@am69-sk:~# lspci
0000:00:00.0 PCI bridge: Texas Instruments Device b012
0000:01:00.0 Network controller: Qualcomm Technologies, Inc WCN785x
Wi-
Fi 7(802.11be) 320MHz 2x2 [FastConnect 7800] (rev 01)
0001:00:00.0 PCI bridge: Texas Instruments Device b012
0002:00:00.0 PCI bridge: Texas Instruments Device b012

Do you have any insights into what might still be missing or incorrect
in my setup?

Regards,
Parth P

Quoted 109 lines · Baochen Qiang
Quoted 7 lines · Parth Panchoil

Hi All,

I am performing tests on the SX-PCEBE Wi-Fi module, which utilizes
the
ATH12k driver, on the Texas Instruments AM69-SK board. The board is running the TI Linux Kernel from the ti-linux-6.6.y

6.6 is too old, and besides we don;t support customer kernel.

Could you try latest ath tree [1] or the mainline tree [2]?

[1] https://git.kernel.org/pub/scm/linux/kernel/git/ath/ath.git/
[2]
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/

If the issue is still seen, please enable verbose ath12k log using below command and help
collect dmesg logs:

        sudo modprobe ath12k debug_mask=0xffffffff

One more thing, the open-WRT patch is overkill, can you narrow down to find which line of
code in ath12k_pci_enable_ltssm() is causing this issue?

Quoted 77 lines

branch. During testing, I observed a kernel crash from the ATH12k driver as soon as the probe is called. The crash log is as follows:

[    9.492631] Kernel panic - not syncing: Asynchronous SError Interrupt
[    9.492634] CPU: 7 PID: 222 Comm: (udev-worker) Not tainted 6.6.58-
01497-ga7758da17c28-dirty #1
[    9.492638] Hardware name: Texas Instruments AM69 SK (DT) [    9.492640] Call trace:
[    9.492642]  dump_backtrace+0x94/0xec
[    9.492658]  show_stack+0x18/0x24
[    9.492662]  dump_stack_lvl+0x48/0x60
[    9.492669]  dump_stack+0x18/0x24
[    9.492672]  panic+0x320/0x378
[    9.492677]  nmi_panic+0x8c/0x90
[    9.492681]  arm64_serror_panic+0x6c/0x78
[    9.492686]  do_serror+0x3c/0x78
[    9.492692]  el1h_64_error_handler+0x34/0x4c
[    9.492697]  el1h_64_error+0x64/0x68
[    9.492700]  ath12k_pci_read32+0x1bc/0x1e8 [ath12k] [    9.492725]  ath12k_pci_power_up+0xdc/0x340 [ath12k] [    9.492747]  ath12k_core_init+0x2c/0xa8 [ath12k] [    9.492769]  ath12k_pci_probe+0x698/0x908 [ath12k] [    9.492791]  pci_device_probe+0xa8/0x16c
[    9.492800]  really_probe+0x110/0x27c
[    9.492805]  __driver_probe_device+0x78/0x12c
[    9.492808]  driver_probe_device+0x3c/0x118
[    9.492810]  __driver_attach+0x74/0x124
[    9.492813]  bus_for_each_dev+0x78/0xd8
[    9.492819]  driver_attach+0x24/0x30
[    9.492824]  bus_add_driver+0xe4/0x208
[    9.492828]  driver_register+0x60/0x128
[    9.492831]  __pci_register_driver+0x44/0x50
[    9.492835]  ath12k_pci_init+0x2c/0x6c [ath12k]
[    9.492858]  do_one_initcall+0x70/0x1b4
[    9.492861]  do_init_module+0x58/0x1e4
[    9.492867]  load_module+0x19bc/0x1a8c
[    9.492869]  init_module_from_file+0x88/0xc4
[    9.492873]  __arm64_sys_finit_module+0x1c0/0x2ac [    9.492877]  invoke_syscall+0x44/0x108
[    9.492882]  el0_svc_common.constprop.0+0xc0/0xe0 [    9.492885]  do_el0_svc+0x1c/0x28
[    9.492889]  el0_svc+0x2c/0x84
[    9.492892]  el0t_64_sync_handler+0xc0/0xc4
[    9.492895]  el0t_64_sync+0x190/0x194
[    9.492899] SMP: stopping secondary CPUs
[    9.492908] Kernel Offset: disabled
[    9.492909] CPU features: 0x0,80000200,28020000,1000420b [    9.492913] Memory Limit: none

Upon searching online, I found the OpenWRT patch that appears to address a similar issue: OpenWRT Patch: Prevent LTSSM Startup Crash.
https://git.openwrt.org/?p=openwrt/openwrt.git;a=blob;f=package/kernel/mac80211/patches/ath12k/100-ath12k-prevent-ltssm-startup-crash.patch;h=cd85a0f6aa2652d62bfbea04e9bcca3bcf831b7f;hb=935b2b7dcef61b2893ed5dff307dd8f8a1156899
With the above patch applied, I do not see the crash anymore.

Could anyone confirm if this issue has been reported before/known bug
or provide any insights?
Any additional information or suggestions would be greatly appreciated.

Details about the test setup,
TI-AM69-SK board:
https://www.ti.com/tool/SK-AM69?keyMatch=am69%20sk&tisearch=universal_search
Silex WiFi card SX-PCEBE:
https://www.silextechnology.com/connectivity-solutions/embedded-wireless/sx-pcebe
TI Linux Repo:
https://git.ti.com/cgit/ti-linux-kernel/ti-linux-kernel/?h=ti-linux-6.6.y

Thank you.

Regards,
Parth P

#4 lore raw

Hi,

I would like to share more details regarding this issue.

A similar bug was previously reported:
https://lore.kernel.org/all/CAFED-jma7ZB17Rcgx9H+eVZ=N6Pty4__gz3B+O2V34XtcEzOGg@mail.gmail.com/

Continuing my debugging on the mainline ath12k driver, I have been investigating a startup hang/crash on my system.
So far, I have identified potential issues with memory mappings, where the driver attempts to read an offset like GCC_GCC_PCIE_HOT_RST (0x1e38338), which is significantly farther than PCIE_PCIE_PARF_LTSSM (0x1e081b0).

I have tried several approaches, including:
- Adjusting the MMIO window dynamically based on the offset instead of using a fixed window.
- Increasing the MMIO window size beyond 512 KiB.
However, these changes did not resolve the issue.

I am sharing my debug logs below (The patch enabling these exact logs is pasted), which may be useful for someone working on this driver to identify the root cause and help resolve this bug.

Any insights or suggestions would be greatly appreciated.

root@am69-sk:~# uname -a
Linux am69-sk 6.13.0-rc7-wt-ath-ge7ef944b3e2c-dirty #4 SMP PREEMPT Wed
Jan 29 08:55:48 CET 2025 aarch64 GNU/Linux
root@am69-sk:~# insmod ./ath12k.ko debug_mask=0xffffffff
[   50.148099] ath12k_pci 0002:01:00.0: BAR 0 [mem 0x4410200000-
0x44103fffff 64bit]: assigned
[   50.156434] ath12k_pci 0002:01:00.0: enabling device (0000 -> 0002)
[   50.162775] ath12k_pci 0002:01:00.0: boot pci_mem 0x000000005f7b1667
[   50.169126] ath12k_pci 0002:01:00.0: pci probe 17cb:1107 17cb:1107
[   50.175294] ### DBG: ath12k_pci_read32 1199 offset =1b00000
[   50.180856] ### DBG: ath12k_pci_read32 1217 offset = 1b00000
[   50.186504] ### DBG: ath12k_pci_select_window 177 window = 36
ab_pci->register_window = 0 WINDOW_REG_ADDRESS = 310c
[   50.196920] ### DBG: ath12k_pci_select_window 178 (WINDOW_ENABLE_BIT
| window) = 40000036
[   50.205084] ### DBG: ath12k_pci_select_window 179 ab->mem +
WINDOW_REG_ADDRESS = 8aa0310c
[   50.213248] ### DBG: ath12k_pci_read32 1220 ab_pci->register_window
= 36 
[   50.220023] ### DBG: ath12k_pci_read32 1221 window_start = 80000
offset = 1b00000 WINDOW_RANGE_MASK = 7ffff 
[   50.229831] ### DBG: ath12k_pci_read32 1229 window_start = 80000
offset = 1b00000 WINDOW_RANGE_MASK = 7ffff ab->mem = 8aa00000
[   50.241200] ### DBG: ath12k_pci_read32 1230 ab->mem + window_start +
(offset & WINDOW_RANGE_MASK) = 8aa80000
[   50.251010] ### DBG: ath12k_pci_read32 1233
[   50.255185] ### DBG: ath12k_pci_read32 1243 val = 40170200
[   50.260658] ath12k_pci 0002:01:00.0: pci tcsr_soc_hw_version major 2
minor 0
[   50.268230] ath12k_pci 0002:01:00.0: MSI vectors: 16
[   50.273212] ath12k_pci 0002:01:00.0: msi base data is 0
[   50.278430] ath12k_pci 0002:01:00.0: Hardware name: wcn7850 hw2.0
[   50.290532] ath12k_pci 0002:01:00.0: failed to load firmware-2.bin:
-2
[   50.297063] ath12k_pci 0002:01:00.0: using fw api 1
[   50.301959] ath12k_pci 0002:01:00.0: Assign MSI to user: MHI,
num_vectors: 3, user_base_data: 0, base_vector: 0
[   50.312038] ath12k_pci 0002:01:00.0: Number of assigned MSI for MHI
is 3, base vector is 0
[   50.322527] ath12k_pci 0002:01:00.0: Assign MSI to user: CE,
num_vectors: 5, user_base_data: 3, base_vector: 3
[   50.332673] ath12k_pci 0002:01:00.0: Assign MSI to user: DP,
num_vectors: 8, user_base_data: 8, base_vector: 8
[   50.342697] ath12k_pci 0002:01:00.0: irq:604 group:0
[   50.347696] ath12k_pci 0002:01:00.0: irq:605 group:1
[   50.352694] ath12k_pci 0002:01:00.0: irq:606 group:2
[   50.357698] ath12k_pci 0002:01:00.0: irq:607 group:3
[   50.362696] ath12k_pci 0002:01:00.0: irq:608 group:4
[   50.367693] ath12k_pci 0002:01:00.0: irq:609 group:5
[   50.372689] ath12k_pci 0002:01:00.0: irq:610 group:6
[   50.377716] ath12k_pci 0002:01:00.0: pci after request_irq
msi_ep_base_data 0
[   50.384846] ath12k_pci 0002:01:00.0: unable to get wsi info from dt,
grouping single device
[   50.393196] ath12k_pci 0002:01:00.0: wsi group-id 255 num-devices 1
index 0
[   50.400148] ath12k_pci 0002:01:00.0: num devices 1 num probed 1
[   50.423637] NET: Registered PF_QIPCRTR protocol family
[   50.429598] ### DBG: ath12k_pci_enable_ltssm 290
[   50.434225] ### DBG: ath12k_pci_read32 1199 offset =1e081b0
[   50.439790] ### DBG: ath12k_pci_read32 1217 offset = 1e081b0
[   50.445437] ### DBG: ath12k_pci_select_window 177 window = 3c
ab_pci->register_window = 0 WINDOW_REG_ADDRESS = 310c
[   50.455852] ### DBG: ath12k_pci_select_window 178 (WINDOW_ENABLE_BIT
| window) = 4000003c
[   50.464014] ### DBG: ath12k_pci_select_window 179 ab->mem +
WINDOW_REG_ADDRESS = 8aa0310c
[   50.472178] ### DBG: ath12k_pci_read32 1220 ab_pci->register_window
= 3c 
[   50.478952] ### DBG: ath12k_pci_read32 1221 window_start = 80000
offset = 1e081b0 WINDOW_RANGE_MASK = 7ffff 
[   50.488759] ### DBG: ath12k_pci_read32 1229 window_start = 80000
offset = 1e081b0 WINDOW_RANGE_MASK = 7ffff ab->mem = 8aa00000
[   50.500126] ### DBG: ath12k_pci_read32 1230 ab->mem + window_start +
(offset & WINDOW_RANGE_MASK) = 8aa881b0
[   50.509936] ### DBG: ath12k_pci_read32 1233
[   50.514109] ### DBG: ath12k_pci_read32 1243 val = 111
[   50.519150] ### DBG: ath12k_pci_enable_ltssm 293 val=111
[   50.524452] ath12k_pci 0002:01:00.0: pci ltssm 0x111
[   50.529407] ### DBG: ath12k_pci_enable_ltssm 305
[   50.534014] ### DBG: ath12k_pci_read32 1199 offset =1e38338
[   50.539577] ### DBG: ath12k_pci_read32 1217 offset = 1e38338
[   50.545227] ### DBG: ath12k_pci_read32 1220 ab_pci->register_window
= 3c 
[   50.552001] ### DBG: ath12k_pci_read32 1221 window_start = 80000
offset = 1e38338 WINDOW_RANGE_MASK = 7ffff 
[   50.561808] ### DBG: ath12k_pci_read32 1229 window_start = 80000
offset = 1e38338 WINDOW_RANGE_MASK = 7ffff ab->mem = 8aa00000
[   50.573175] ### DBG: ath12k_pci_read32 1230 ab->mem + window_start +
(offset & WINDOW_RANGE_MASK) = 8aab8338

Thank you.

Regards,
Parth P

=======================================================================

(quoted diff) +0 -0

b/drivers/net/wireless/ath/ath12k/pci.c
index 06cff3849ab8..2984064783f9 100644

drivers/net/wireless/ath/ath12k/pci.c +0 -0
@@ -174,6 +174,9 @@ static void ath12k_pci_select_window(struct

ath12k_pci *ab_pci, u32 offset)
window |= static_window;

if (window != ab_pci->register_window) {
+ printk("### DBG: %s %d window = %x ab_pci-

Quoted 1 line

register_window = %x WINDOW_REG_ADDRESS =

%x\n",__func__,__LINE__,window, ab_pci->register_window, WINDOW_REG_ADDRESS);
+ printk("### DBG: %s %d (WINDOW_ENABLE_BIT | window) =
%x\n",__func__,__LINE__,(WINDOW_ENABLE_BIT | window) );
+ printk("### DBG: %s %d ab->mem + WINDOW_REG_ADDRESS =
%x\n",__func__,__LINE__,(ab->mem + WINDOW_REG_ADDRESS) );
iowrite32(WINDOW_ENABLE_BIT | window,
ab->mem + WINDOW_REG_ADDRESS);
ioread32(ab->mem + WINDOW_REG_ADDRESS);
@@ -279,8 +282,15 @@ static void ath12k_pci_enable_ltssm(struct ath12k_base *ab)
u32 val;
int i;

+#if 0
+ /* Prevent startup crash on BPI-Rx */
+ return;
+#endif
+
+ printk("### DBG: %s %d\n",__func__,__LINE__);
val = ath12k_pci_read32(ab, PCIE_PCIE_PARF_LTSSM);

+ printk("### DBG: %s %d val=%x\n",__func__,__LINE__,val);
/* PCIE link seems very unstable after the Hot Reset*/ for (i = 0; val != PARM_LTSSM_VALUE && i < 5; i++) {
if (val == 0xffffffff)
@@ -292,11 +302,15 @@ static void ath12k_pci_enable_ltssm(struct ath12k_base *ab)

ath12k_dbg(ab, ATH12K_DBG_PCI, "pci ltssm 0x%x\n", val);

+ printk("### DBG: %s %d\n",__func__,__LINE__);
val = ath12k_pci_read32(ab, GCC_GCC_PCIE_HOT_RST);
+ printk("### DBG: %s %d val = %x\n",__func__,__LINE__,val);
val |= GCC_GCC_PCIE_HOT_RST_VAL;
+ printk("### DBG: %s %d val = %x\n",__func__,__LINE__,val);
ath12k_pci_write32(ab, GCC_GCC_PCIE_HOT_RST, val); val = ath12k_pci_read32(ab, GCC_GCC_PCIE_HOT_RST);

+ printk("### DBG: %s %d\n",__func__,__LINE__);
ath12k_dbg(ab, ATH12K_DBG_PCI, "pci pcie_hot_rst 0x%x\n",
val);

mdelay(5);
@@ -1182,6 +1196,7 @@ u32 ath12k_pci_read32(struct ath12k_base *ab, u32 offset)
u32 val, window_start;
int ret = 0;

+ printk("### DBG: %s %d offset
=%x\n",__func__,__LINE__,offset);
/* for offset beyond BAR + 4K - 32, may
* need to wakeup MHI to access.
*/
@@ -1196,26 +1211,36 @@ u32 ath12k_pci_read32(struct ath12k_base *ab, u32 offset)
window_start = ath12k_pci_get_window_start(ab,
offset);
else
window_start = WINDOW_START;
+ //window_start = (offset &
~WINDOW_RANGE_MASK);

if (window_start == WINDOW_START) {
+ printk("### DBG: %s %d offset =
%x\n",__func__,__LINE__,offset);
spin_lock_bh(&ab_pci->window_lock); ath12k_pci_select_window(ab_pci, offset);
+ printk("### DBG: %s %d ab_pci->register_window
= %x \n",__func__,__LINE__, ab_pci->register_window);
+ printk("### DBG: %s %d window_start = %x
offset = %x WINDOW_RANGE_MASK = %x \n",__func__,__LINE__, window_start,offset, WINDOW_RANGE_MASK);

if
(ath12k_pci_is_offset_within_mhi_region(offset)) {
+ printk("### DBG: %s
%d\n",__func__,__LINE__);
offset = offset - PCI_MHIREGLEN_REG; val = ioread32(ab->mem +
(offset &
WINDOW_RANGE_MASK));
} else {
+ printk("### DBG: %s %d window_start =
%x offset = %x WINDOW_RANGE_MASK = %x ab->mem = %x\n",__func__,__LINE__,window_start,offset, WINDOW_RANGE_MASK,ab-

Quoted 1 line

mem);

+ printk("### DBG: %s %d ab->mem +
window_start + (offset & WINDOW_RANGE_MASK) = %x\n",__func__,__LINE__,ab->mem + window_start + (offset & WINDOW_RANGE_MASK));
val = ioread32(ab->mem + window_start
+
(offset &
WINDOW_RANGE_MASK));
+ printk("### DBG: %s
%d\n",__func__,__LINE__);
} spin_unlock_bh(&ab_pci->window_lock);
} else {
+ printk("### DBG: %s %d\n",__func__,__LINE__);
val = ioread32(ab->mem + window_start +
(offset & WINDOW_RANGE_MASK));
}
}

+ printk("### DBG: %s %d val = %x\n",__func__,__LINE__,val);
if (test_bit(ATH12K_PCI_FLAG_INIT_DONE, &ab_pci->flags) &&
offset >= ACCESS_ALWAYS_OFF && ab_pci->pci_ops->release && !ret)
=======================================================================

Quoted 193 lines · Parth Panchoil

Hi,

I am currently debugging the ath12k_pci_enable_ltssm start up crash/bug
with the mainline kernel on my system and would like to share my observations so far:

The ath12k mainline driver gets stuck at this specific line:
https://github.com/torvalds/linux/blob/9c5968db9e625019a0ee5226c7eebef5519d366a/drivers/net/wireless/ath/ath12k/pci.c#L295
in the ath12k_pci_enable_ltssm  which attempts to read GCC_GCC_PCIE_HOT_RST, particularly
https://github.com/torvalds/linux/blob/9c5968db9e625019a0ee5226c7eebef5519d366a/drivers/net/wireless/ath/ath12k/pci.c#L1209

Interestingly, within the same function, the line val = ath12k_pci_read32(ab, PCIE_PCIE_PARF_LTSSM) successfully reads the expected value 0x111 for PCIE_PCIE_PARF_LTSSM.

I am continuing to debug from my end, although my understanding of the
ath12k driver is limited. Any leads, suggestions, or hints to help resolve this issue would be greatly appreciated.

Thank you.

Regards,
Parth P

Quoted 163 lines · Parth Pancholi

I appreciate your response, Baochen.

I have been working on enabling mainline kernel support on my TI AM69-
SK board to test the mainline ath12k driver on my system.

Using the mainline kernel repository for the ath drivers [1], I made
the following observation:
While the exact crash observed earlier is no longer present, the system
hangs upon loading the ath12k mainline driver, displaying the messages
below.

root@am69-sk:~# modprobe ath12k debug_mask=0xffffffff
[ 1121.996554] ath12k_pci 0000:01:00.0: BAR 0 [mem 0x4410200000-
0x44103fffff 64bit]: assigned
[ 1122.004884] ath12k_pci 0000:01:00.0: enabling device (0000 ->
0002)
[ 1122.011818] ath12k_pci 0000:01:00.0: MSI vectors: 16
[ 1122.016798] ath12k_pci 0000:01:00.0: Hardware name: wcn7850
hw2.0
[ 1122.040183] NET: Registered PF_QIPCRTR protocol family
root@am69-sk:~# uname -a
Linux am69-sk 6.13.0-rc7-wt-ath-ge7ef944b3e2c-dirty #2 SMP PREEMPT
Wed
Jan 22 16:55:17 CET 2025 aarch64 GNU/Linux
root@am69-sk:~# lspci
0000:00:00.0 PCI bridge: Texas Instruments Device b012
0000:01:00.0 Network controller: Qualcomm Technologies, Inc WCN785x
Wi-
Fi 7(802.11be) 320MHz 2x2 [FastConnect 7800] (rev 01)
0001:00:00.0 PCI bridge: Texas Instruments Device b012
0002:00:00.0 PCI bridge: Texas Instruments Device b012

Do you have any insights into what might still be missing or incorrect
in my setup?

Regards,
Parth P

Quoted 115 lines · Baochen Qiang
Quoted 8 lines · Parth Panchoil

Hi All,

I am performing tests on the SX-PCEBE Wi-Fi module, which utilizes
the
ATH12k driver, on the Texas Instruments AM69-SK board. The board is running the TI Linux Kernel from the ti-linux- 6.6.y

6.6 is too old, and besides we don;t support customer kernel.

Could you try latest ath tree [1] or the mainline tree [2]?

[1] https://git.kernel.org/pub/scm/linux/kernel/git/ath/ath.git/
[2]
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/

If the issue is still seen, please enable verbose ath12k log using
below command and help
collect dmesg logs:

        sudo modprobe ath12k debug_mask=0xffffffff

One more thing, the open-WRT patch is overkill, can you narrow down
to find which line of
code in ath12k_pci_enable_ltssm() is causing this issue?

Quoted 80 lines

branch. During testing, I observed a kernel crash from the ATH12k
driver as soon as the probe is called. The crash log is as follows:

[    9.492631] Kernel panic - not syncing: Asynchronous SError Interrupt
[    9.492634] CPU: 7 PID: 222 Comm: (udev-worker) Not tainted 6.6.58-
01497-ga7758da17c28-dirty #1
[    9.492638] Hardware name: Texas Instruments AM69 SK (DT) [    9.492640] Call trace:
[    9.492642]  dump_backtrace+0x94/0xec
[    9.492658]  show_stack+0x18/0x24
[    9.492662]  dump_stack_lvl+0x48/0x60
[    9.492669]  dump_stack+0x18/0x24
[    9.492672]  panic+0x320/0x378
[    9.492677]  nmi_panic+0x8c/0x90
[    9.492681]  arm64_serror_panic+0x6c/0x78
[    9.492686]  do_serror+0x3c/0x78
[    9.492692]  el1h_64_error_handler+0x34/0x4c
[    9.492697]  el1h_64_error+0x64/0x68
[    9.492700]  ath12k_pci_read32+0x1bc/0x1e8 [ath12k] [    9.492725]  ath12k_pci_power_up+0xdc/0x340 [ath12k] [    9.492747]  ath12k_core_init+0x2c/0xa8 [ath12k] [    9.492769]  ath12k_pci_probe+0x698/0x908 [ath12k] [    9.492791]  pci_device_probe+0xa8/0x16c
[    9.492800]  really_probe+0x110/0x27c
[    9.492805]  __driver_probe_device+0x78/0x12c
[    9.492808]  driver_probe_device+0x3c/0x118
[    9.492810]  __driver_attach+0x74/0x124
[    9.492813]  bus_for_each_dev+0x78/0xd8
[    9.492819]  driver_attach+0x24/0x30
[    9.492824]  bus_add_driver+0xe4/0x208
[    9.492828]  driver_register+0x60/0x128
[    9.492831]  __pci_register_driver+0x44/0x50
[    9.492835]  ath12k_pci_init+0x2c/0x6c [ath12k]
[    9.492858]  do_one_initcall+0x70/0x1b4
[    9.492861]  do_init_module+0x58/0x1e4
[    9.492867]  load_module+0x19bc/0x1a8c
[    9.492869]  init_module_from_file+0x88/0xc4
[    9.492873]  __arm64_sys_finit_module+0x1c0/0x2ac [    9.492877]  invoke_syscall+0x44/0x108
[    9.492882]  el0_svc_common.constprop.0+0xc0/0xe0 [    9.492885]  do_el0_svc+0x1c/0x28
[    9.492889]  el0_svc+0x2c/0x84
[    9.492892]  el0t_64_sync_handler+0xc0/0xc4
[    9.492895]  el0t_64_sync+0x190/0x194
[    9.492899] SMP: stopping secondary CPUs
[    9.492908] Kernel Offset: disabled
[    9.492909] CPU features: 0x0,80000200,28020000,1000420b [    9.492913] Memory Limit: none

Upon searching online, I found the OpenWRT patch that appears to
address a similar issue: OpenWRT Patch: Prevent LTSSM Startup Crash.
https://git.openwrt.org/?p=openwrt/openwrt.git;a=blob;f=package/kernel/mac80211/patches/ath12k/100-ath12k-prevent-ltssm-startup-crash.patch;h=cd85a0f6aa2652d62bfbea04e9bcca3bcf831b7f;hb=935b2b7dcef61b2893ed5dff307dd8f8a1156899
With the above patch applied, I do not see the crash anymore.

Could anyone confirm if this issue has been reported before/known
bug
or provide any insights?
Any additional information or suggestions would be greatly appreciated.

Details about the test setup,
TI-AM69-SK board:
https://www.ti.com/tool/SK-AM69?keyMatch=am69%20sk&tisearch=universal_search
Silex WiFi card SX-PCEBE:
https://www.silextechnology.com/connectivity-solutions/embedded-wireless/sx-pcebe
TI Linux Repo:
https://git.ti.com/cgit/ti-linux-kernel/ti-linux-kernel/?h=ti-linux-6.6.y

Thank you.

Regards,
Parth P

#5 lore raw
In reply to Parth Panchoil
Quoted 11 lines · Parth Panchoil

Hi,

I am currently debugging the ath12k_pci_enable_ltssm start up crash/bug with the mainline kernel on my system and would like to share my observations so far:

The ath12k mainline driver gets stuck at this specific line:
https://github.com/torvalds/linux/blob/9c5968db9e625019a0ee5226c7eebef5519d366a/drivers/net/wireless/ath/ath12k/pci.c#L295
in the ath12k_pci_enable_ltssm which attempts to read GCC_GCC_PCIE_HOT_RST, particularly
https://github.com/torvalds/linux/blob/9c5968db9e625019a0ee5226c7eebef5519d366a/drivers/net/wireless/ath/ath12k/pci.c#L1209

thanks for the narrow down, really helpful.

We internally have observed this issue, although at a different line:

https://github.com/torvalds/linux/blob/9c5968db9e625019a0ee5226c7eebef5519d366a/drivers/net/wireless/ath/ath12k/pci.c#L298

For now I am suspecting that GCC_GCC_PCIE_HOT_RST is not a valid register on WLAN target side, I will check internally and get back.

Quoted 172 lines

Interestingly, within the same function, the line val = ath12k_pci_read32(ab, PCIE_PCIE_PARF_LTSSM) successfully reads the expected value 0x111 for PCIE_PCIE_PARF_LTSSM.

I am continuing to debug from my end, although my understanding of the ath12k driver is limited. Any leads, suggestions, or hints to help resolve this issue would be greatly appreciated.

Thank you.

Regards,
Parth P

Quoted 155 lines · Parth Pancholi

I appreciate your response, Baochen.

I have been working on enabling mainline kernel support on my TI AM69-
SK board to test the mainline ath12k driver on my system.

Using the mainline kernel repository for the ath drivers [1], I made the following observation:
While the exact crash observed earlier is no longer present, the system
hangs upon loading the ath12k mainline driver, displaying the messages
below.

root@am69-sk:~# modprobe ath12k debug_mask=0xffffffff
[ 1121.996554] ath12k_pci 0000:01:00.0: BAR 0 [mem 0x4410200000-
0x44103fffff 64bit]: assigned
[ 1122.004884] ath12k_pci 0000:01:00.0: enabling device (0000 ->
0002)
[ 1122.011818] ath12k_pci 0000:01:00.0: MSI vectors: 16
[ 1122.016798] ath12k_pci 0000:01:00.0: Hardware name: wcn7850 hw2.0
[ 1122.040183] NET: Registered PF_QIPCRTR protocol family
root@am69-sk:~# uname -a
Linux am69-sk 6.13.0-rc7-wt-ath-ge7ef944b3e2c-dirty #2 SMP PREEMPT
Wed
Jan 22 16:55:17 CET 2025 aarch64 GNU/Linux
root@am69-sk:~# lspci
0000:00:00.0 PCI bridge: Texas Instruments Device b012
0000:01:00.0 Network controller: Qualcomm Technologies, Inc WCN785x
Wi-
Fi 7(802.11be) 320MHz 2x2 [FastConnect 7800] (rev 01)
0001:00:00.0 PCI bridge: Texas Instruments Device b012
0002:00:00.0 PCI bridge: Texas Instruments Device b012

Do you have any insights into what might still be missing or incorrect
in my setup?

Regards,
Parth P

Quoted 109 lines · Baochen Qiang
Quoted 7 lines · Parth Panchoil

Hi All,

I am performing tests on the SX-PCEBE Wi-Fi module, which utilizes
the
ATH12k driver, on the Texas Instruments AM69-SK board. The board is running the TI Linux Kernel from the ti-linux-6.6.y

6.6 is too old, and besides we don;t support customer kernel.

Could you try latest ath tree [1] or the mainline tree [2]?

[1] https://git.kernel.org/pub/scm/linux/kernel/git/ath/ath.git/
[2]
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/

If the issue is still seen, please enable verbose ath12k log using below command and help
collect dmesg logs:

        sudo modprobe ath12k debug_mask=0xffffffff

One more thing, the open-WRT patch is overkill, can you narrow down to find which line of
code in ath12k_pci_enable_ltssm() is causing this issue?

Quoted 77 lines

branch. During testing, I observed a kernel crash from the ATH12k driver as soon as the probe is called. The crash log is as follows:

[    9.492631] Kernel panic - not syncing: Asynchronous SError Interrupt
[    9.492634] CPU: 7 PID: 222 Comm: (udev-worker) Not tainted 6.6.58-
01497-ga7758da17c28-dirty #1
[    9.492638] Hardware name: Texas Instruments AM69 SK (DT) [    9.492640] Call trace:
[    9.492642]  dump_backtrace+0x94/0xec
[    9.492658]  show_stack+0x18/0x24
[    9.492662]  dump_stack_lvl+0x48/0x60
[    9.492669]  dump_stack+0x18/0x24
[    9.492672]  panic+0x320/0x378
[    9.492677]  nmi_panic+0x8c/0x90
[    9.492681]  arm64_serror_panic+0x6c/0x78
[    9.492686]  do_serror+0x3c/0x78
[    9.492692]  el1h_64_error_handler+0x34/0x4c
[    9.492697]  el1h_64_error+0x64/0x68
[    9.492700]  ath12k_pci_read32+0x1bc/0x1e8 [ath12k] [    9.492725]  ath12k_pci_power_up+0xdc/0x340 [ath12k] [    9.492747]  ath12k_core_init+0x2c/0xa8 [ath12k] [    9.492769]  ath12k_pci_probe+0x698/0x908 [ath12k] [    9.492791]  pci_device_probe+0xa8/0x16c
[    9.492800]  really_probe+0x110/0x27c
[    9.492805]  __driver_probe_device+0x78/0x12c
[    9.492808]  driver_probe_device+0x3c/0x118
[    9.492810]  __driver_attach+0x74/0x124
[    9.492813]  bus_for_each_dev+0x78/0xd8
[    9.492819]  driver_attach+0x24/0x30
[    9.492824]  bus_add_driver+0xe4/0x208
[    9.492828]  driver_register+0x60/0x128
[    9.492831]  __pci_register_driver+0x44/0x50
[    9.492835]  ath12k_pci_init+0x2c/0x6c [ath12k]
[    9.492858]  do_one_initcall+0x70/0x1b4
[    9.492861]  do_init_module+0x58/0x1e4
[    9.492867]  load_module+0x19bc/0x1a8c
[    9.492869]  init_module_from_file+0x88/0xc4
[    9.492873]  __arm64_sys_finit_module+0x1c0/0x2ac [    9.492877]  invoke_syscall+0x44/0x108
[    9.492882]  el0_svc_common.constprop.0+0xc0/0xe0 [    9.492885]  do_el0_svc+0x1c/0x28
[    9.492889]  el0_svc+0x2c/0x84
[    9.492892]  el0t_64_sync_handler+0xc0/0xc4
[    9.492895]  el0t_64_sync+0x190/0x194
[    9.492899] SMP: stopping secondary CPUs
[    9.492908] Kernel Offset: disabled
[    9.492909] CPU features: 0x0,80000200,28020000,1000420b [    9.492913] Memory Limit: none

Upon searching online, I found the OpenWRT patch that appears to address a similar issue: OpenWRT Patch: Prevent LTSSM Startup Crash.
https://git.openwrt.org/?p=openwrt/openwrt.git;a=blob;f=package/kernel/mac80211/patches/ath12k/100-ath12k-prevent-ltssm-startup-crash.patch;h=cd85a0f6aa2652d62bfbea04e9bcca3bcf831b7f;hb=935b2b7dcef61b2893ed5dff307dd8f8a1156899
With the above patch applied, I do not see the crash anymore.

Could anyone confirm if this issue has been reported before/known bug
or provide any insights?
Any additional information or suggestions would be greatly appreciated.

Details about the test setup,
TI-AM69-SK board:
https://www.ti.com/tool/SK-AM69?keyMatch=am69%20sk&tisearch=universal_search
Silex WiFi card SX-PCEBE:
https://www.silextechnology.com/connectivity-solutions/embedded-wireless/sx-pcebe
TI Linux Repo:
https://git.ti.com/cgit/ti-linux-kernel/ti-linux-kernel/?h=ti-linux-6.6.y

Thank you.

Regards,
Parth P

#6 lore raw
Quoted 15 lines · Baochen Qiang
Quoted 11 lines · Parth Panchoil

Hi,

I am currently debugging the ath12k_pci_enable_ltssm start up crash/bug with the mainline kernel on my system and would like to share my observations so far:

The ath12k mainline driver gets stuck at this specific line:
https://github.com/torvalds/linux/blob/9c5968db9e625019a0ee5226c7eebef5519d366a/drivers/net/wireless/ath/ath12k/pci.c#L295
in the ath12k_pci_enable_ltssm which attempts to read GCC_GCC_PCIE_HOT_RST, particularly
https://github.com/torvalds/linux/blob/9c5968db9e625019a0ee5226c7eebef5519d366a/drivers/net/wireless/ath/ath12k/pci.c#L1209

Quoted 8 lines

thanks for the narrow down, really helpful.

We internally have observed this issue, although at a different line:

https://github.com/torvalds/linux/blob/9c5968db9e625019a0ee5226c7eebef5519d366a/drivers/net/wireless/ath/ath12k/pci.c#L298

For now I am suspecting that GCC_GCC_PCIE_HOT_RST is not a valid register on WLAN target side, I will check internally and get back.

Parth, could you do below change and try again?

-#define GCC_GCC_PCIE_HOT_RST 0x1e38338
+#define GCC_GCC_PCIE_HOT_RST 0x1e40304

Quoted 175 lines
Quoted 173 lines

Interestingly, within the same function, the line val = ath12k_pci_read32(ab, PCIE_PCIE_PARF_LTSSM) successfully reads the expected value 0x111 for PCIE_PCIE_PARF_LTSSM.

I am continuing to debug from my end, although my understanding of the ath12k driver is limited. Any leads, suggestions, or hints to help resolve this issue would be greatly appreciated.

Thank you.

Regards,
Parth P

Quoted 155 lines · Parth Pancholi

I appreciate your response, Baochen.

I have been working on enabling mainline kernel support on my TI AM69-
SK board to test the mainline ath12k driver on my system.

Using the mainline kernel repository for the ath drivers [1], I made the following observation:
While the exact crash observed earlier is no longer present, the system
hangs upon loading the ath12k mainline driver, displaying the messages
below.

root@am69-sk:~# modprobe ath12k debug_mask=0xffffffff
[ 1121.996554] ath12k_pci 0000:01:00.0: BAR 0 [mem 0x4410200000-
0x44103fffff 64bit]: assigned
[ 1122.004884] ath12k_pci 0000:01:00.0: enabling device (0000 ->
0002)
[ 1122.011818] ath12k_pci 0000:01:00.0: MSI vectors: 16
[ 1122.016798] ath12k_pci 0000:01:00.0: Hardware name: wcn7850 hw2.0
[ 1122.040183] NET: Registered PF_QIPCRTR protocol family
root@am69-sk:~# uname -a
Linux am69-sk 6.13.0-rc7-wt-ath-ge7ef944b3e2c-dirty #2 SMP PREEMPT
Wed
Jan 22 16:55:17 CET 2025 aarch64 GNU/Linux
root@am69-sk:~# lspci
0000:00:00.0 PCI bridge: Texas Instruments Device b012
0000:01:00.0 Network controller: Qualcomm Technologies, Inc WCN785x
Wi-
Fi 7(802.11be) 320MHz 2x2 [FastConnect 7800] (rev 01)
0001:00:00.0 PCI bridge: Texas Instruments Device b012
0002:00:00.0 PCI bridge: Texas Instruments Device b012

Do you have any insights into what might still be missing or incorrect
in my setup?

Regards,
Parth P

Quoted 109 lines · Baochen Qiang
Quoted 7 lines · Parth Panchoil

Hi All,

I am performing tests on the SX-PCEBE Wi-Fi module, which utilizes
the
ATH12k driver, on the Texas Instruments AM69-SK board. The board is running the TI Linux Kernel from the ti-linux-6.6.y

6.6 is too old, and besides we don;t support customer kernel.

Could you try latest ath tree [1] or the mainline tree [2]?

[1] https://git.kernel.org/pub/scm/linux/kernel/git/ath/ath.git/
[2]
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/

If the issue is still seen, please enable verbose ath12k log using below command and help
collect dmesg logs:

        sudo modprobe ath12k debug_mask=0xffffffff

One more thing, the open-WRT patch is overkill, can you narrow down to find which line of
code in ath12k_pci_enable_ltssm() is causing this issue?

Quoted 77 lines

branch. During testing, I observed a kernel crash from the ATH12k driver as soon as the probe is called. The crash log is as follows:

[    9.492631] Kernel panic - not syncing: Asynchronous SError Interrupt
[    9.492634] CPU: 7 PID: 222 Comm: (udev-worker) Not tainted 6.6.58-
01497-ga7758da17c28-dirty #1
[    9.492638] Hardware name: Texas Instruments AM69 SK (DT) [    9.492640] Call trace:
[    9.492642]  dump_backtrace+0x94/0xec
[    9.492658]  show_stack+0x18/0x24
[    9.492662]  dump_stack_lvl+0x48/0x60
[    9.492669]  dump_stack+0x18/0x24
[    9.492672]  panic+0x320/0x378
[    9.492677]  nmi_panic+0x8c/0x90
[    9.492681]  arm64_serror_panic+0x6c/0x78
[    9.492686]  do_serror+0x3c/0x78
[    9.492692]  el1h_64_error_handler+0x34/0x4c
[    9.492697]  el1h_64_error+0x64/0x68
[    9.492700]  ath12k_pci_read32+0x1bc/0x1e8 [ath12k] [    9.492725]  ath12k_pci_power_up+0xdc/0x340 [ath12k] [    9.492747]  ath12k_core_init+0x2c/0xa8 [ath12k] [    9.492769]  ath12k_pci_probe+0x698/0x908 [ath12k] [    9.492791]  pci_device_probe+0xa8/0x16c
[    9.492800]  really_probe+0x110/0x27c
[    9.492805]  __driver_probe_device+0x78/0x12c
[    9.492808]  driver_probe_device+0x3c/0x118
[    9.492810]  __driver_attach+0x74/0x124
[    9.492813]  bus_for_each_dev+0x78/0xd8
[    9.492819]  driver_attach+0x24/0x30
[    9.492824]  bus_add_driver+0xe4/0x208
[    9.492828]  driver_register+0x60/0x128
[    9.492831]  __pci_register_driver+0x44/0x50
[    9.492835]  ath12k_pci_init+0x2c/0x6c [ath12k]
[    9.492858]  do_one_initcall+0x70/0x1b4
[    9.492861]  do_init_module+0x58/0x1e4
[    9.492867]  load_module+0x19bc/0x1a8c
[    9.492869]  init_module_from_file+0x88/0xc4
[    9.492873]  __arm64_sys_finit_module+0x1c0/0x2ac [    9.492877]  invoke_syscall+0x44/0x108
[    9.492882]  el0_svc_common.constprop.0+0xc0/0xe0 [    9.492885]  do_el0_svc+0x1c/0x28
[    9.492889]  el0_svc+0x2c/0x84
[    9.492892]  el0t_64_sync_handler+0xc0/0xc4
[    9.492895]  el0t_64_sync+0x190/0x194
[    9.492899] SMP: stopping secondary CPUs
[    9.492908] Kernel Offset: disabled
[    9.492909] CPU features: 0x0,80000200,28020000,1000420b [    9.492913] Memory Limit: none

Upon searching online, I found the OpenWRT patch that appears to address a similar issue: OpenWRT Patch: Prevent LTSSM Startup Crash.
https://git.openwrt.org/?p=openwrt/openwrt.git;a=blob;f=package/kernel/mac80211/patches/ath12k/100-ath12k-prevent-ltssm-startup-crash.patch;h=cd85a0f6aa2652d62bfbea04e9bcca3bcf831b7f;hb=935b2b7dcef61b2893ed5dff307dd8f8a1156899
With the above patch applied, I do not see the crash anymore.

Could anyone confirm if this issue has been reported before/known bug
or provide any insights?
Any additional information or suggestions would be greatly appreciated.

Details about the test setup,
TI-AM69-SK board:
https://www.ti.com/tool/SK-AM69?keyMatch=am69%20sk&tisearch=universal_search
Silex WiFi card SX-PCEBE:
https://www.silextechnology.com/connectivity-solutions/embedded-wireless/sx-pcebe
TI Linux Repo:
https://git.ti.com/cgit/ti-linux-kernel/ti-linux-kernel/?h=ti-linux-6.6.y

Thank you.

Regards,
Parth P

#7 lore raw
Quoted 33 lines · Baochen Qiang
Quoted 18 lines · Baochen Qiang
Quoted 7 lines · Parth Panchoil

Hi,

I am currently debugging the ath12k_pci_enable_ltssm start up crash/bug
with the mainline kernel on my system and would like to share my observations so far:

Quoted 5 lines

The ath12k mainline driver gets stuck at this specific line:
https://github.com/torvalds/linux/blob/9c5968db9e625019a0ee5226c7eebef5519d366a/drivers/net/wireless/ath/ath12k/pci.c#L295
in the ath12k_pci_enable_ltssm  which attempts to read GCC_GCC_PCIE_HOT_RST, particularly
https://github.com/torvalds/linux/blob/9c5968db9e625019a0ee5226c7eebef5519d366a/drivers/net/wireless/ath/ath12k/pci.c#L1209

thanks for the narrow down, really helpful.

Quoted 8 lines

We internally have observed this issue, although at a different line:

https://github.com/torvalds/linux/blob/9c5968db9e625019a0ee5226c7eebef5519d366a/drivers/net/wireless/ath/ath12k/pci.c#L298

For now I am suspecting that GCC_GCC_PCIE_HOT_RST is not a valid register on WLAN target
side, I will check internally and get back.

Parth, could you do below change and try again?

Review

Quoted 2 lines
(quoted diff) +1 -1
-#define GCC_GCC_PCIE_HOT_RST                   0x1e38338
+#define GCC_GCC_PCIE_HOT_RST                   0x1e40304

Hi Baochen,

Thanks for the hint regarding the change.
I tested this change on top of the ath-202504172310 tag on the TI AM69 platform and can confirm that the startup crash no longer occurs. Interestingly, this issue was not observed on other platforms like the NXP iMX8MP.

If I understand correctly, the change is related to a WLAN target register.

Could you help clarify why this issue only affects certain platforms (hosts)?

Regards,
Parth P

Quoted 199 lines
Quoted 198 lines
Quoted 196 lines

Interestingly, within the same function, the line val = ath12k_pci_read32(ab, PCIE_PCIE_PARF_LTSSM) successfully reads the
expected value 0x111 for PCIE_PCIE_PARF_LTSSM.

I am continuing to debug from my end, although my understanding of the
ath12k driver is limited. Any leads, suggestions, or hints to help
resolve this issue would be greatly appreciated.

Thank you.

Regards,
Parth P

Quoted 175 lines · Parth Pancholi

I appreciate your response, Baochen.

I have been working on enabling mainline kernel support on my TI
AM69-
SK board to test the mainline ath12k driver on my system.

Using the mainline kernel repository for the ath drivers [1], I made
the following observation:
While the exact crash observed earlier is no longer present, the
system
hangs upon loading the ath12k mainline driver, displaying the messages
below.

root@am69-sk:~# modprobe ath12k debug_mask=0xffffffff
[ 1121.996554] ath12k_pci 0000:01:00.0: BAR 0 [mem
0x4410200000-
0x44103fffff 64bit]: assigned
[ 1122.004884] ath12k_pci 0000:01:00.0: enabling device (0000 -
Quoted 1 line

0002)

[ 1122.011818] ath12k_pci 0000:01:00.0: MSI vectors: 16
[ 1122.016798] ath12k_pci 0000:01:00.0: Hardware name: wcn7850
hw2.0
[ 1122.040183] NET: Registered PF_QIPCRTR protocol family
root@am69-sk:~# uname -a
Linux am69-sk 6.13.0-rc7-wt-ath-ge7ef944b3e2c-dirty #2 SMP
PREEMPT
Wed
Jan 22 16:55:17 CET 2025 aarch64 GNU/Linux
root@am69-sk:~# lspci
0000:00:00.0 PCI bridge: Texas Instruments Device b012
0000:01:00.0 Network controller: Qualcomm Technologies, Inc
WCN785x
Wi-
Fi 7(802.11be) 320MHz 2x2 [FastConnect 7800] (rev 01)
0001:00:00.0 PCI bridge: Texas Instruments Device b012
0002:00:00.0 PCI bridge: Texas Instruments Device b012

Do you have any insights into what might still be missing or incorrect
in my setup?

Regards,
Parth P

Quoted 121 lines · Baochen Qiang
Quoted 8 lines · Parth Panchoil

Hi All,

I am performing tests on the SX-PCEBE Wi-Fi module, which utilizes
the
ATH12k driver, on the Texas Instruments AM69-SK board. The board is running the TI Linux Kernel from the ti-linux- 6.6.y

6.6 is too old, and besides we don;t support customer kernel.

Could you try latest ath tree [1] or the mainline tree [2]?

[1]
https://git.kernel.org/pub/scm/linux/kernel/git/ath/ath.git/
[2]
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/

If the issue is still seen, please enable verbose ath12k log using
below command and help
collect dmesg logs:

        sudo modprobe ath12k debug_mask=0xffffffff

One more thing, the open-WRT patch is overkill, can you narrow down
to find which line of
code in ath12k_pci_enable_ltssm() is causing this issue?

Quoted 85 lines

branch. During testing, I observed a kernel crash from the ATH12k
driver as soon as the probe is called. The crash log is as follows:

[    9.492631] Kernel panic - not syncing: Asynchronous SError
Interrupt
[    9.492634] CPU: 7 PID: 222 Comm: (udev-worker) Not tainted
6.6.58-
01497-ga7758da17c28-dirty #1
[    9.492638] Hardware name: Texas Instruments AM69 SK (DT)
[    9.492640] Call trace:
[    9.492642]  dump_backtrace+0x94/0xec
[    9.492658]  show_stack+0x18/0x24
[    9.492662]  dump_stack_lvl+0x48/0x60
[    9.492669]  dump_stack+0x18/0x24
[    9.492672]  panic+0x320/0x378
[    9.492677]  nmi_panic+0x8c/0x90
[    9.492681]  arm64_serror_panic+0x6c/0x78
[    9.492686]  do_serror+0x3c/0x78
[    9.492692]  el1h_64_error_handler+0x34/0x4c
[    9.492697]  el1h_64_error+0x64/0x68
[    9.492700]  ath12k_pci_read32+0x1bc/0x1e8 [ath12k] [    9.492725]  ath12k_pci_power_up+0xdc/0x340 [ath12k] [    9.492747]  ath12k_core_init+0x2c/0xa8 [ath12k] [    9.492769]  ath12k_pci_probe+0x698/0x908 [ath12k] [    9.492791]  pci_device_probe+0xa8/0x16c
[    9.492800]  really_probe+0x110/0x27c
[    9.492805]  __driver_probe_device+0x78/0x12c
[    9.492808]  driver_probe_device+0x3c/0x118
[    9.492810]  __driver_attach+0x74/0x124
[    9.492813]  bus_for_each_dev+0x78/0xd8
[    9.492819]  driver_attach+0x24/0x30
[    9.492824]  bus_add_driver+0xe4/0x208
[    9.492828]  driver_register+0x60/0x128
[    9.492831]  __pci_register_driver+0x44/0x50
[    9.492835]  ath12k_pci_init+0x2c/0x6c [ath12k] [    9.492858]  do_one_initcall+0x70/0x1b4
[    9.492861]  do_init_module+0x58/0x1e4
[    9.492867]  load_module+0x19bc/0x1a8c
[    9.492869]  init_module_from_file+0x88/0xc4
[    9.492873]  __arm64_sys_finit_module+0x1c0/0x2ac [    9.492877]  invoke_syscall+0x44/0x108
[    9.492882]  el0_svc_common.constprop.0+0xc0/0xe0 [    9.492885]  do_el0_svc+0x1c/0x28
[    9.492889]  el0_svc+0x2c/0x84
[    9.492892]  el0t_64_sync_handler+0xc0/0xc4
[    9.492895]  el0t_64_sync+0x190/0x194
[    9.492899] SMP: stopping secondary CPUs
[    9.492908] Kernel Offset: disabled
[    9.492909] CPU features: 0x0,80000200,28020000,1000420b [    9.492913] Memory Limit: none

Upon searching online, I found the OpenWRT patch that appears to
address a similar issue: OpenWRT Patch: Prevent LTSSM Startup
Crash.
https://git.openwrt.org/?p=openwrt/openwrt.git;a=blob;f=package/kernel/mac80211/patches/ath12k/100-ath12k-prevent-ltssm-startup-crash.patch;h=cd85a0f6aa2652d62bfbea04e9bcca3bcf831b7f;hb=935b2b7dcef61b2893ed5dff307dd8f8a1156899
With the above patch applied, I do not see the crash anymore.

Could anyone confirm if this issue has been reported before/known
bug
or provide any insights?
Any additional information or suggestions would be greatly appreciated.

Details about the test setup,
TI-AM69-SK board:
https://www.ti.com/tool/SK-AM69?keyMatch=am69%20sk&tisearch=universal_search
Silex WiFi card SX-PCEBE:
https://www.silextechnology.com/connectivity-solutions/embedded-wireless/sx-pcebe
TI Linux Repo:
https://git.ti.com/cgit/ti-linux-kernel/ti-linux-kernel/?h=ti-linux-6.6.y

Thank you.

Regards,
Parth P

#8 lore raw
Quoted 33 lines · Parth Panchoil
Quoted 32 lines · Baochen Qiang
Quoted 18 lines · Baochen Qiang
Quoted 7 lines · Parth Panchoil

Hi,

I am currently debugging the ath12k_pci_enable_ltssm start up crash/bug
with the mainline kernel on my system and would like to share my observations so far:

Quoted 5 lines

The ath12k mainline driver gets stuck at this specific line:
https://github.com/torvalds/linux/blob/9c5968db9e625019a0ee5226c7eebef5519d366a/drivers/net/wireless/ath/ath12k/pci.c#L295
in the ath12k_pci_enable_ltssm  which attempts to read GCC_GCC_PCIE_HOT_RST, particularly
https://github.com/torvalds/linux/blob/9c5968db9e625019a0ee5226c7eebef5519d366a/drivers/net/wireless/ath/ath12k/pci.c#L1209

thanks for the narrow down, really helpful.

Quoted 8 lines

We internally have observed this issue, although at a different line:

https://github.com/torvalds/linux/blob/9c5968db9e625019a0ee5226c7eebef5519d366a/drivers/net/wireless/ath/ath12k/pci.c#L298

For now I am suspecting that GCC_GCC_PCIE_HOT_RST is not a valid register on WLAN target
side, I will check internally and get back.

Parth, could you do below change and try again?

Quoted 8 lines

Review

Quoted 3 lines
(quoted diff) +1 -1
-#define GCC_GCC_PCIE_HOT_RST                   0x1e38338
+#define GCC_GCC_PCIE_HOT_RST                   0x1e40304

Hi Baochen,

Thanks for the hint regarding the change.
I tested this change on top of the ath-202504172310 tag on the TI AM69 platform and can confirm that the startup crash no longer occurs.

great it helps

Quoted 6 lines

Interestingly, this issue was not observed on other platforms like the NXP iMX8MP.

If I understand correctly, the change is related to a WLAN target register.

correct

Quoted 2 lines

Could you help clarify why this issue only affects certain platforms (hosts)?

GCC_GCC_PCIE_HOT_RST is wrongly defined, normally this should not cause any critical issue, because IMO the RC is expected to return 0xffffffff when accessing a non-exist register. However in your case kernel crashes, so seems RC does not behave well, maybe not following spec?

anyway I will submit a patch to fix it.

Quoted 204 lines

Regards,
Parth P

Quoted 199 lines
Quoted 198 lines
Quoted 196 lines

Interestingly, within the same function, the line val = ath12k_pci_read32(ab, PCIE_PCIE_PARF_LTSSM) successfully reads the
expected value 0x111 for PCIE_PCIE_PARF_LTSSM.

I am continuing to debug from my end, although my understanding of the
ath12k driver is limited. Any leads, suggestions, or hints to help
resolve this issue would be greatly appreciated.

Thank you.

Regards,
Parth P

Quoted 175 lines · Parth Pancholi

I appreciate your response, Baochen.

I have been working on enabling mainline kernel support on my TI
AM69-
SK board to test the mainline ath12k driver on my system.

Using the mainline kernel repository for the ath drivers [1], I made
the following observation:
While the exact crash observed earlier is no longer present, the
system
hangs upon loading the ath12k mainline driver, displaying the messages
below.

root@am69-sk:~# modprobe ath12k debug_mask=0xffffffff
[ 1121.996554] ath12k_pci 0000:01:00.0: BAR 0 [mem
0x4410200000-
0x44103fffff 64bit]: assigned
[ 1122.004884] ath12k_pci 0000:01:00.0: enabling device (0000 -
Quoted 1 line

0002)

[ 1122.011818] ath12k_pci 0000:01:00.0: MSI vectors: 16
[ 1122.016798] ath12k_pci 0000:01:00.0: Hardware name: wcn7850
hw2.0
[ 1122.040183] NET: Registered PF_QIPCRTR protocol family
root@am69-sk:~# uname -a
Linux am69-sk 6.13.0-rc7-wt-ath-ge7ef944b3e2c-dirty #2 SMP
PREEMPT
Wed
Jan 22 16:55:17 CET 2025 aarch64 GNU/Linux
root@am69-sk:~# lspci
0000:00:00.0 PCI bridge: Texas Instruments Device b012
0000:01:00.0 Network controller: Qualcomm Technologies, Inc
WCN785x
Wi-
Fi 7(802.11be) 320MHz 2x2 [FastConnect 7800] (rev 01)
0001:00:00.0 PCI bridge: Texas Instruments Device b012
0002:00:00.0 PCI bridge: Texas Instruments Device b012

Do you have any insights into what might still be missing or incorrect
in my setup?

Regards,
Parth P

Quoted 121 lines · Baochen Qiang
Quoted 8 lines · Parth Panchoil

Hi All,

I am performing tests on the SX-PCEBE Wi-Fi module, which utilizes
the
ATH12k driver, on the Texas Instruments AM69-SK board. The board is running the TI Linux Kernel from the ti-linux- 6.6.y

6.6 is too old, and besides we don;t support customer kernel.

Could you try latest ath tree [1] or the mainline tree [2]?

[1]
https://git.kernel.org/pub/scm/linux/kernel/git/ath/ath.git/
[2]
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/

If the issue is still seen, please enable verbose ath12k log using
below command and help
collect dmesg logs:

        sudo modprobe ath12k debug_mask=0xffffffff

One more thing, the open-WRT patch is overkill, can you narrow down
to find which line of
code in ath12k_pci_enable_ltssm() is causing this issue?

Quoted 85 lines

branch. During testing, I observed a kernel crash from the ATH12k
driver as soon as the probe is called. The crash log is as follows:

[    9.492631] Kernel panic - not syncing: Asynchronous SError
Interrupt
[    9.492634] CPU: 7 PID: 222 Comm: (udev-worker) Not tainted
6.6.58-
01497-ga7758da17c28-dirty #1
[    9.492638] Hardware name: Texas Instruments AM69 SK (DT)
[    9.492640] Call trace:
[    9.492642]  dump_backtrace+0x94/0xec
[    9.492658]  show_stack+0x18/0x24
[    9.492662]  dump_stack_lvl+0x48/0x60
[    9.492669]  dump_stack+0x18/0x24
[    9.492672]  panic+0x320/0x378
[    9.492677]  nmi_panic+0x8c/0x90
[    9.492681]  arm64_serror_panic+0x6c/0x78
[    9.492686]  do_serror+0x3c/0x78
[    9.492692]  el1h_64_error_handler+0x34/0x4c
[    9.492697]  el1h_64_error+0x64/0x68
[    9.492700]  ath12k_pci_read32+0x1bc/0x1e8 [ath12k] [    9.492725]  ath12k_pci_power_up+0xdc/0x340 [ath12k] [    9.492747]  ath12k_core_init+0x2c/0xa8 [ath12k] [    9.492769]  ath12k_pci_probe+0x698/0x908 [ath12k] [    9.492791]  pci_device_probe+0xa8/0x16c
[    9.492800]  really_probe+0x110/0x27c
[    9.492805]  __driver_probe_device+0x78/0x12c
[    9.492808]  driver_probe_device+0x3c/0x118
[    9.492810]  __driver_attach+0x74/0x124
[    9.492813]  bus_for_each_dev+0x78/0xd8
[    9.492819]  driver_attach+0x24/0x30
[    9.492824]  bus_add_driver+0xe4/0x208
[    9.492828]  driver_register+0x60/0x128
[    9.492831]  __pci_register_driver+0x44/0x50
[    9.492835]  ath12k_pci_init+0x2c/0x6c [ath12k] [    9.492858]  do_one_initcall+0x70/0x1b4
[    9.492861]  do_init_module+0x58/0x1e4
[    9.492867]  load_module+0x19bc/0x1a8c
[    9.492869]  init_module_from_file+0x88/0xc4
[    9.492873]  __arm64_sys_finit_module+0x1c0/0x2ac [    9.492877]  invoke_syscall+0x44/0x108
[    9.492882]  el0_svc_common.constprop.0+0xc0/0xe0 [    9.492885]  do_el0_svc+0x1c/0x28
[    9.492889]  el0_svc+0x2c/0x84
[    9.492892]  el0t_64_sync_handler+0xc0/0xc4
[    9.492895]  el0t_64_sync+0x190/0x194
[    9.492899] SMP: stopping secondary CPUs
[    9.492908] Kernel Offset: disabled
[    9.492909] CPU features: 0x0,80000200,28020000,1000420b [    9.492913] Memory Limit: none

Upon searching online, I found the OpenWRT patch that appears to
address a similar issue: OpenWRT Patch: Prevent LTSSM Startup
Crash.
https://git.openwrt.org/?p=openwrt/openwrt.git;a=blob;f=package/kernel/mac80211/patches/ath12k/100-ath12k-prevent-ltssm-startup-crash.patch;h=cd85a0f6aa2652d62bfbea04e9bcca3bcf831b7f;hb=935b2b7dcef61b2893ed5dff307dd8f8a1156899
With the above patch applied, I do not see the crash anymore.

Could anyone confirm if this issue has been reported before/known
bug
or provide any insights?
Any additional information or suggestions would be greatly appreciated.

Details about the test setup,
TI-AM69-SK board:
https://www.ti.com/tool/SK-AM69?keyMatch=am69%20sk&tisearch=universal_search
Silex WiFi card SX-PCEBE:
https://www.silextechnology.com/connectivity-solutions/embedded-wireless/sx-pcebe
TI Linux Repo:
https://git.ti.com/cgit/ti-linux-kernel/ti-linux-kernel/?h=ti-linux-6.6.y

Thank you.

Regards,
Parth P

#9 lore raw

Hello,

Quoted 14 lines · Baochen Qiang
Quoted 13 lines · Parth Panchoil
Quoted 12 lines · Baochen Qiang
Quoted 11 lines · Baochen Qiang
Quoted 10 lines · Parth Panchoil

I am currently debugging the ath12k_pci_enable_ltssm start up crash/bug
with the mainline kernel on my system and would like to share my observations so far:

The ath12k mainline driver gets stuck at this specific line:
https://github.com/torvalds/linux/blob/9c5968db9e625019a0ee5226c7eebef5519d366a/drivers/net/wireless/ath/ath12k/pci.c#L295
in the ath12k_pci_enable_ltssm  which attempts to read GCC_GCC_PCIE_HOT_RST, particularly
https://github.com/torvalds/linux/blob/9c5968db9e625019a0ee5226c7eebef5519d366a/drivers/net/wireless/ath/ath12k/pci.c#L1209

...

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
(quoted diff) +1 -1
-#define GCC_GCC_PCIE_HOT_RST                   0x1e38338
+#define GCC_GCC_PCIE_HOT_RST                   0x1e40304

...

Quoted 7 lines

GCC_GCC_PCIE_HOT_RST is wrongly defined, normally this should not cause any critical issue, because IMO the RC is expected to return 0xffffffff when accessing a non-exist register. However in your case kernel crashes, so seems RC does not behave well, maybe not following spec?

anyway I will submit a patch to fix it.

Any update on this? I can easily submit a patch myself, just let me know. Because of this bug we have a crash that is preventing us to use the device.

Francesco

#10 lore raw
Quoted 32 lines · Francesco Dolcini

Hello,

Quoted 14 lines · Baochen Qiang
Quoted 13 lines · Parth Panchoil
Quoted 12 lines · Baochen Qiang
Quoted 11 lines · Baochen Qiang
Quoted 10 lines · Parth Panchoil

I am currently debugging the ath12k_pci_enable_ltssm start up crash/bug
with the mainline kernel on my system and would like to share my observations so far:

The ath12k mainline driver gets stuck at this specific line:
https://github.com/torvalds/linux/blob/9c5968db9e625019a0ee5226c7eebef5519d366a/drivers/net/wireless/ath/ath12k/pci.c#L295
in the ath12k_pci_enable_ltssm  which attempts to read GCC_GCC_PCIE_HOT_RST, particularly
https://github.com/torvalds/linux/blob/9c5968db9e625019a0ee5226c7eebef5519d366a/drivers/net/wireless/ath/ath12k/pci.c#L1209

...

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
(quoted diff) +1 -1
-#define GCC_GCC_PCIE_HOT_RST                   0x1e38338
+#define GCC_GCC_PCIE_HOT_RST                   0x1e40304

...

Quoted 6 lines

GCC_GCC_PCIE_HOT_RST is wrongly defined, normally this should not cause any critical issue, because IMO the RC is expected to return 0xffffffff when accessing a non-exist register. However in your case kernel crashes, so seems RC does not behave well, maybe not following spec?

Quoted 6 lines
Quoted 1 line

anyway I will submit a patch to fix it.

Any update on this? I can easily submit a patch myself, just let me know. Because of this bug we have a crash that is preventing us to use the device.

The patch has been submitted for internal review. Will make it public available once approved internally.

Quoted 2 lines

Francesco