Freeze on resume from S3 (bisected)

I've been experiencing system freezes that require a hard reset with the power button, starting from v6.8-rc1. The warnings produced suggest a wireless-related issue, and the bisection seems to confirm this.

The freezes usually occur right after resuming from S3 (suspend). To reproduce the issue, I just suspend (sudo systemctl suspend) and resume (press any key on keyboard) a few times until the freeze happens. It takes 4-5 cycles of this most of the time, though it varies a lot.

The upshot of all of this is that S3 (and S4 as well, see below) are pretty much unusable for me, due to the risk of losing work due to a freeze.

Since a hard reset causes the systemd journal to be dropped, I set up [kdumpst](https://gitlab.freedesktop.org/gpiccoli/kdumpst) so that I could record dmesg output at the time of the freeze. I've linked to the crash logs produced by that tool for every bad commit in the bisection (except in cases where I couldn't get the kdump to happen - see the comments in the bisection log below), as well as a crash log triggered by SysRq (Alt+SysRq+c) on a good commit for comparison.

I've already submitted this as an Arch Linux bug report:
https://gitlab.archlinux.org/archlinux/packaging/packages/linux/-/issues/61

The freezes have also occurred (seemingly) at random times, although this is quite rare. They do seem to be more likely to occur after the system has been hibernated, although I don't really have any way to confirm this.

Git bisect log:

git bisect start
# status: waiting for both good and bad commits
# bad: [83a7eefedc9b56fe7bfeff13b6c7356688ffa670] Linux 6.10-rc3
git bisect bad 83a7eefedc9b56fe7bfeff13b6c7356688ffa670
# good: [e8f897f4afef0031fe618a8e94127a0934896aba] Linux 6.8
git bisect good e8f897f4afef0031fe618a8e94127a0934896aba
# bad: [445e60303883950161f67e18b9f048b18d7fb706] Merge branch '1GbE' of git://git.kernel.org/pub/scm/linux/kernel/git/tnguy/next-queue
git bisect bad 445e60303883950161f67e18b9f048b18d7fb706
# bad: [e5e038b7ae9da96b93974bf072ca1876899a01a3] Merge tag 'fs_for_v6.9-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/jack/linux-fs

# this one froze on the first suspend/resume
git bisect bad e5e038b7ae9da96b93974bf072ca1876899a01a3
# good: [1f440397665f4241346e4cc6d93f8b73880815d1] Merge tag 'docs-6.9' of git://git.lwn.net/linux
git bisect good 1f440397665f4241346e4cc6d93f8b73880815d1
# bad: [a2f24c8a955c8f941d6ac08dd7f401f54eef4627] Merge branch 'mptcp-some-clean-up-patches'
git bisect bad a2f24c8a955c8f941d6ac08dd7f401f54eef4627
# bad: [26f4dac11775a1ca24e2605cb30e828d4dbdea93] netfilter: x_tables: Use unsafe_memcpy() for 0-sized destination
# This one was very hard to reproduce - it must have taken 25-30 cycles total, and I hibernated the system twice
git bisect bad 26f4dac11775a1ca24e2605cb30e828d4dbdea93
# good: [4f5e5092fdbf5cec6bedc19fbe69cce4f5f08372] Merge tag 'net-6.8-rc5' of git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net
git bisect good 4f5e5092fdbf5cec6bedc19fbe69cce4f5f08372
# good: [b4e8ae5c8c41355791a99fdf2fcac16deace1e79] net: add napi_busy_loop_rcu()
git bisect good b4e8ae5c8c41355791a99fdf2fcac16deace1e79
# bad: [20ea9327c2fd545d6b96e998727bcd724290694d] net: dccp: Simplify the allocation of slab caches in dccp_ackvec_init
git bisect bad 20ea9327c2fd545d6b96e998727bcd724290694d
# bad: [92046e83c07b064ca65ac4ae7660a540016bdfc1] Merge tag 'for-netdev' of https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next
git bisect bad 92046e83c07b064ca65ac4ae7660a540016bdfc1
# bad: [b54846da45942bbe4e5ebc59d497e4a48525ba5a] Merge tag 'wireless-next-2024-01-25' of git://git.kernel.org/pub/scm/linux/kernel/git/wireless/wireless-next
git bisect bad b54846da45942bbe4e5ebc59d497e4a48525ba5a
# good: [3832a9c40b356500c5b85a6fdf9577c590fcd637] wifi: rtw89: fw: extend JOIN H2C command to support WiFi 7 chips
git bisect good 3832a9c40b356500c5b85a6fdf9577c590fcd637
# bad: [5ba45ba77616637e554d66a57ef0334e5cc2efe4] wifi: rtw89: fix disabling concurrent mode TX hang issue
# I see the same kernel warnings here as with the other bad commits.
# However, there is no crash after a few suspend/resume cycles - instead
# the system simply restarts when I trigger the nth suspend. I'm not
# able to trigger a crash with SysRq either (Alt+SysRq+c) - again, the
# system restarts. This is the case with all bad commits after this.
git bisect bad 5ba45ba77616637e554d66a57ef0334e5cc2efe4
# good: [85da8f71aaa7b83ea7ef0e89182e0cd47e16d465] wifi: brcmfmac: Demote vendor-specific attach/detach messages to info
git bisect good 85da8f71aaa7b83ea7ef0e89182e0cd47e16d465
# good: [295304040d9f6f350b68652acd99650c7e16d0a8] wifi: rtw89: 8922a: add TX power related ops
git bisect good 295304040d9f6f350b68652acd99650c7e16d0a8
# good: [7cf6b6764b2f665d317ba0f91c247437019a2f4c] wifi: rtw89: Set default CQM config if not present
git bisect good 7cf6b6764b2f665d317ba0f91c247437019a2f4c
# good: [7e11a2966f51695c0af0b1f976a32d64dee243b2] wifi: rtw89: fix null pointer access when abort scan
git bisect good 7e11a2966f51695c0af0b1f976a32d64dee243b2
# bad: [f59a98c82534e986b06615ba94e060aa3129b08b] wifi: rtw89: fix HW scan timeout due to TSF sync issue
git bisect bad f59a98c82534e986b06615ba94e060aa3129b08b
# bad: [bcbefbd032df6bfe925e6afeca82eb9d2cc0cb23] wifi: rtw89: add wait/completion for abort scan
git bisect bad bcbefbd032df6bfe925e6afeca82eb9d2cc0cb23
# first bad commit: [bcbefbd032df6bfe925e6afeca82eb9d2cc0cb23] wifi: rtw89: add wait/completion for abort scan

#regzbot introduced: bcbefbd032df6bfe925e6afeca82eb9d2cc0cb23

Logs produced by kdumpst:

[kdumpst-e5e038b7ae9da96b93974bf072ca1876899a01a3.zip](https://gitlab.archlinux.org/archlinux/packaging/packages/linux/uploads/b925cb5e9c6cfb6b224993c017ba61f5/kdumpst-e5e038b7ae9da96b93974bf072ca1876899a01a3.zip)

[kdumpst-b54846da45942bbe4e5ebc59d497e4a48525ba5a.zip](https://gitlab.archlinux.org/archlinux/packaging/packages/linux/uploads/e1610c3ae26971a890121d3a5eb8fa9e/kdumpst-b54846da45942bbe4e5ebc59d497e4a48525ba5a.zip)

[kdumpst-a2f24c8a955c8f941d6ac08dd7f401f54eef4627.zip](https://gitlab.archlinux.org/archlinux/packaging/packages/linux/uploads/34171a87da304987cc8ce5066fc35dd5/kdumpst-a2f24c8a955c8f941d6ac08dd7f401f54eef4627.zip)

[kdumpst-92046e83c07b064ca65ac4ae7660a540016bdfc1.zip](https://gitlab.archlinux.org/archlinux/packaging/packages/linux/uploads/67d756f963ffafd0b332c88701286a30/kdumpst-92046e83c07b064ca65ac4ae7660a540016bdfc1.zip)

[kdumpst-445e60303883950161f67e18b9f048b18d7fb706.zip](https://gitlab.archlinux.org/archlinux/packaging/packages/linux/uploads/f75f91d127050a327114bf904f335087/kdumpst-445e60303883950161f67e18b9f048b18d7fb706.zip)

[kdumpst-26f4dac11775a1ca24e2605cb30e828d4dbdea93.zip](https://gitlab.archlinux.org/archlinux/packaging/packages/linux/uploads/f3d26fc4187c8b8854386fed6f15c972/kdumpst-26f4dac11775a1ca24e2605cb30e828d4dbdea93.zip)

[kdumpst-20ea9327c2fd545d6b96e998727bcd724290694d.zip](https://gitlab.archlinux.org/archlinux/packaging/packages/linux/uploads/8618207aff0eb92413ce0c2994c7888f/kdumpst-20ea9327c2fd545d6b96e998727bcd724290694d.zip)

Good commit: [kdumpst-e8f897f4afef0031fe618a8e94127a0934896aba.zip](https://gitlab.archlinux.org/archlinux/packaging/packages/linux/uploads/b13b3914e50fc85548c77a4163dec383/kdumpst-e8f897f4afef0031fe618a8e94127a0934896aba.zip) --

Please try [1] that fixed "wifi: rtw89: add wait/completion for abort" for certain cases.

[1] https://lore.kernel.org/linux-wireless/[email protected]/

Doesn't fix the issue. I get a freeze on the first suspend+resume. I've attached the crash log.

--

I tried 4.10-rc4 + the patch on ubuntu. Never reproduce the symptom. Please share Arch Linux image you are using.

Could you please help to collect 2 or more crash log? So I can check if there are more than one crash cases.

I guess you mean v6.10-rc4?

https://patchwork.kernel.org/project/linux-wireless/list/

https://wireless.wiki.kernel.org/en/developers/documentation/submittingpatches

Kalle Valo <[email protected]> worte:

Right. Sorry for the typo.

Not sure what you mean by 'image'. As you can see in the crash log attached to my previous mail, I built the latest mainline kernel (445e60303883 at the time) with your patch applied. I used [this PKGBUILD](https://aur.archlinux.org/packages/linux-git) to build it; the file config contains the kernel configuration (I did not apply any other options), and there are no patches applied except yours.

When I am back at my system, I will reproduce the issue a few more times with this kernel, and attach the logs. In the meantime, you could have a look at the logs linked in my first mail. There are logs for most of the bad commits encountered in the bisection.

I feel this problem may be easier to reproduce on Arch Linux, so I would like to know Arch Linux iso file you installed.

I will do it as your side.

I have seen that, but no clear idea for now, so I will install Arch Linux as yours in my side.

I really appreciate your dedication here, but I'm not sure that this is related to my OS. I feel it might be a corner case that manifests only on certain hardware configurations, otherwise it would probably have been encountered by other people by now.
I can't say this with any confidence, since this is my first kernel bug, and I don't have any factual basis for this feeling; I just don't want you to burn yourself out with the Arch installation process when it might not help in diagnosing the issue. Ultimately we'll go by whatever you think is best, though; you're the expert here, not me.

I don't remember the iso version that I used (it was years ago), and I don't know of any way to check, but it shouldn't matter. AFAIK the Arch iso is only used to bootstrap the system, so its version should not be of any consequence to my current configuration. You might want to look at https://wiki.archlinux.org/title/Installation_guide
to get an idea of what the process is like; as you'll see it's very manual and takes a fair bit of effort.

Please provide output of 'cat /etc/lsb-release', which Arch Linux version should be there, to me.

I hope using the same version as yours makes the symptom reproducible.

I don't really know any of this here, but ...

+ ret = rtw89_hw_scan_offload(rtwdev, vif, false);
+ if (ret)
+ rtw89_hw_scan_complete(rtwdev, vif, true);

seems strange? You have to say that it was completed here, in the good case, so maybe that was meant to be !ret?

It _looks_ like the crash is a use-after-free (the wiphy pointer in a scan request cannot become NULL in normal flows), so maybe try with KASAN rather than waiting for the crash. According to the logs, it doesn't happen every time even for the reporter.

There possibly seems to be some issue between cfg80211 and mac80211 in this code, we see the WARN_ON() in cfg80211_netdev_notifier_call() in the NETDEV_DOWN case, which calls ___cfg80211_scan_done() which frees the scan request. But shortly after the HW crashes, and we have "ieee80211_restart_work called with hardware scan in progress", mac80211 wants to cancel the HW scan but the HW is dead ("wlo1: Failed check- sdata-in-driver check, flags: 0x0"), and we see again "phy0: resume with hardware scan still in progress" ... but this time once tasks are restarted it crashes ...

So I think KASAN, possibly rtw debugs, and perhaps something like
https://p.sipsolutions.net/602684f34abfcf7c.txt will help debug it (yes it adds a leak)

johannes

I don't have that file. But I do have /etc/os-release:

NAME="Arch Linux"
PRETTY_NAME="Arch Linux"
ID=arch
BUILD_ID=rolling
ANSI_COLOR="38;2;23;147;209" HOME_URL="https://archlinux.org/" DOCUMENTATION_URL="https://wiki.archlinux.org/" SUPPORT_URL="https://bbs.archlinux.org/" BUG_REPORT_URL="https://gitlab.archlinux.org/groups/archlinux/-/issues" PRIVACY_POLICY_URL="https://terms.archlinux.org/docs/privacy-policy/" LOGO=archlinux-logo

I've attached more logs.

Hi Mathew,

Thanks for the logs, which you met two kinds of problems. One is firmware gets wrong during system resumes, and the other is to close a disappear netdev. However I still can't dig why it gets wrong. I will focus on latter one first.

The behavior between no commit [1] and the latest tree + [2] is the time waiting for ACK from scan abort firmware command. The former one is longer, and latter one is shorter.

I also enable kernel debug KASAN suggested by Johannes to dig problem, but in my side I can't see any kernel warning and the crash.

Since I saw 'NetworkManager' and 'hostapd' in code trace, I would like to know if you have two virtual interfaces, which for STA and AP modes? (Please check this by 'iw dev') If so, is it possible to remove hostapd (AP mode) to see if this is a factor causing crash.

Attachment is a debug patch that add more messages and code trace, please help to reproduce problem with patches of [2] and attachment. If your kernel enables dynamic debug, need additional commands to have debug message:
sudo bash -c 'echo -n "module rtw89_core +p" > /sys/kernel/debug/dynamic_debug/control' sudo bash -c 'echo -n "module rtw89_pci +p" > /sys/kernel/debug/dynamic_debug/control'
Since there are more than one symptoms causing system freeze, please collect four logs as before. Also please give me two logs that system can normally suspend/resume, so I can compare their difference.

[1] bcbefbd032 wifi: rtw89: add wait/completion for abort scan
[2] https://lore.kernel.org/linux-wireless/[email protected]/

Thanks
Ping-ke

From 0dec70207ab62c87e2f9b3a232166e07c3260f36 Mon Sep 17 00:00:00 2001
From: Ping-Ke Shih <[email protected]>
Date: Fri, 28 Jun 2024 11:17:59 +0800
Subject: [PATCH] debug scan abort

---
 drivers/net/wireless/realtek/rtw89/debug.c    |  2 +-
 drivers/net/wireless/realtek/rtw89/fw.c       |  5 +++
 drivers/net/wireless/realtek/rtw89/mac.c      |  4 ++
 drivers/net/wireless/realtek/rtw89/mac80211.c | 42 ++++++++++++++++---
 drivers/net/wireless/realtek/rtw89/pci.c      |  4 ++
 5 files changed, 50 insertions(+), 7 deletions(-)

index 49bbbd049316..4709d50b8fa1 100644
@@ -14,7 +14,7 @@
 #include "sar.h"
 
 #ifdef CONFIG_RTW89_DEBUGMSG
-unsigned int rtw89_debug_mask;
+unsigned int rtw89_debug_mask = 0x80021000;
 EXPORT_SYMBOL(rtw89_debug_mask);
 module_param_named(debug_mask, rtw89_debug_mask, uint, 0644);
 MODULE_PARM_DESC(debug_mask, "Debugging mask");

index fbe08c162b93..eecb43ad6735 100644
@@ -4855,6 +4855,9 @@ int rtw89_fw_h2c_scan_offload(struct rtw89_dev *rtwdev,
 	else
 		cond = RTW89_SCANOFLD_WAIT_COND_STOP;
 
+	printk("pk> %s:%d start scan offload abort=%d\n", __func__, __LINE__,
+		!option->enable);
+
 	ret = rtw89_h2c_tx_and_wait(rtwdev, skb, wait, cond);
 	if (ret) {
 		rtw89_debug(rtwdev, RTW89_DBG_FW, "failed to scan ofld\n");

@@ -6267,6 +6270,8 @@ void rtw89_hw_scan_complete(struct rtw89_dev *rtwdev, struct ieee80211_vif *vif,
 	if (!vif)
 		return;
 
+	printk("pk> %s:%d abort=%d\n", __func__, __LINE__, scan_info->abort);
+
 	rtw89_write32_mask(rtwdev,
 			   rtw89_mac_reg_by_idx(rtwdev, mac->rx_fltr, RTW89_MAC_0),
 			   B_AX_RX_FLTR_CFG_MASK,

index 73462f3343e3..48ef521618e2 100644
@@ -4757,6 +4757,8 @@ rtw89_mac_c2h_scanofld_rsp(struct rtw89_dev *rtwdev, struct sk_buff *skb,
 		}
 		return;
 	case RTW89_SCAN_END_SCAN_NOTIFY:
+		printk("pk> %s:%d scan end handler abort=%d\n", __func__, __LINE__,
+			rtwdev->scan_info.abort);
 		if (rtwdev->scan_info.abort)
 			return;
 

@@ -4895,6 +4897,7 @@ rtw89_mac_c2h_done_ack(struct rtw89_dev *rtwdev, struct sk_buff *skb_c2h, u32 le
 			break;
 		case H2C_FUNC_SCANOFLD:
 			cond = RTW89_SCANOFLD_WAIT_COND_START;
+			printk("pk> %s:%d DACK for scan offload\n", __func__, __LINE__);
 			break;
 		case H2C_FUNC_SCANOFLD_BE:
 			cond = RTW89_SCANOFLD_BE_WAIT_COND_START;

@@ -5283,6 +5286,7 @@ static void rtw89_mac_c2h_scanofld_rsp_atomic(struct rtw89_dev *rtwdev,
 		else
 			cond = RTW89_SCANOFLD_WAIT_COND_STOP;
 
+		printk("pk> %s:%d scan end ISR\n", __func__, __LINE__);
 		rtw89_complete_cond(fw_ofld_wait, cond, &data);
 	}
 }

index 02afeb3acce4..c95fa9e66cc4 100644
@@ -4160,6 +4160,8 @@ static int __maybe_unused rtw89_pci_suspend(struct device *dev)
 	struct rtw89_dev *rtwdev = hw->priv;
 	enum rtw89_core_chip_id chip_id = rtwdev->chip->chip_id;
 
+	printk("pk> %s:%d\n", __func__, __LINE__);
+
 	rtw89_write32_set(rtwdev, R_AX_RSV_CTRL, B_AX_WLOCK_1C_BIT6);
 	rtw89_write32_set(rtwdev, R_AX_RSV_CTRL, B_AX_R_DIS_PRST);
 	rtw89_write32_clr(rtwdev, R_AX_RSV_CTRL, B_AX_WLOCK_1C_BIT6);

@@ -4194,6 +4196,8 @@ static int __maybe_unused rtw89_pci_resume(struct device *dev)
 	struct rtw89_dev *rtwdev = hw->priv;
 	enum rtw89_core_chip_id chip_id = rtwdev->chip->chip_id;
 
+	printk("pk> %s:%d\n", __func__, __LINE__);
+
 	rtw89_write32_set(rtwdev, R_AX_RSV_CTRL, B_AX_WLOCK_1C_BIT6);
 	rtw89_write32_clr(rtwdev, R_AX_RSV_CTRL, B_AX_R_DIS_PRST);
 	rtw89_write32_clr(rtwdev, R_AX_RSV_CTRL, B_AX_WLOCK_1C_BIT6);

2.25.1