Skip to contentSkip to conversation

PCI: Add device-specific reset for Qualcomm devices

PATCH Superseded Jose Ignacio Tornos Martinez 4 participants 10 messages Jul 21 – Sep 18

Subsystems (4)

v13 of 15 Superseded Latest is v15 →

lore
Share & tools
lore
#0 lore raw

PATCH v13 1/1 PCI: Add device-specific reset for Qualcomm devices

Some Qualcomm PCIe devices (WCN6855/WCN7850 WLAN cards, SDX62/SDX65 modems) lack working reset methods for VFIO passthrough scenarios. These devices have no FLR capability, advertise NoSoftRst+ (blocking PM reset), and have broken bus reset.

The problem manifests in VFIO passthrough scenarios:

- WCN6855 (17cb:1103) and WCN7850 (17cb:1107) WLAN devices:
Normal VM operation works fine, including clean shutdown/reboot. However, when the VM terminates uncleanly (crash, force-off), VFIO attempts to reset the device before it can be assigned to another VM. Without a working reset method, the device remains in an undefined state, preventing reuse.

- SDX62/SDX65 (17cb:0308) 5G modems: Never successfully initialize even on first VM assignment without proper reset capability.

Add device-specific reset methods using BAR-space hardware reset registers that exist in these devices:

- WCN6855/WCN7850 WLAN devices use SoC global reset via BAR0 (sequence from ath11k/ath12k driver: ath11k_pci_soc_global_reset(), ath11k_pci_sw_reset(), ath11k_mhi_set_mhictrl_reset()):
- Write/clear reset bit at offset 0x3008
- Wait for PCIe link recovery (up to 5 seconds)
- Clear MHI controller SYSERR status at offset 0x38

- SDX62/SDX65 modem devices use MHI SoC reset via BAR0 (sequence from MHI
driver: mhi_soc_reset(), mhi_pci_reset_prepare()):
- Write reset request to offset 0xb0
- Wait 2 seconds for reset completion

These are true hardware reset mechanisms (not power management or firmware error recovery), providing proper device reset for VFIO scenarios.

Testing was performed on desktop platforms with M.2 WLAN and modem cards using M.2-to-PCIe adapters, including extensive force-reset cycling to verify stability.

---
v13: Address Alex Williamson feedback:
- Validate initial ioread32() with PCI_POSSIBLE_ERROR() before read-modify-write to avoid writing error response back to device on platforms with APEI/GHES error escalation
- Replace time_before()/msleep() polling loop with read_poll_timeout() for robustness against scheduling delays and code simplification
v12: https://lore.kernel.org/all/[email protected]/

 drivers/pci/quirks.c | 116 +++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 116 insertions(+)
drivers/pci/quirks.c +116 -0
index 431c021d7414..bd1e0742052e 100644
@@ -22,6 +22,7 @@
 #include <linux/isa-dma.h> /* isa_dma_bridge_buggy */
 #include <linux/init.h>
 #include <linux/iommu.h>
+#include <linux/iopoll.h>
 #include <linux/delay.h>
 #include <linux/acpi.h>
 #include <linux/dmi.h>
@@ -4240,6 +4241,118 @@ static int reset_hinic_vf_dev(struct pci_dev *pdev, bool probe)
 	return 0;
 }
 
+#define QUALCOMM_WLAN_PCIE_SOC_GLOBAL_RESET	0x3008
+#define QUALCOMM_WLAN_PCIE_SOC_GLOBAL_RESET_V	BIT(0)
+#define QUALCOMM_WLAN_MHICTRL			0x38
+#define QUALCOMM_WLAN_MHICTRL_RESET_MASK	0x2
+
+/*
+ * Qualcomm WLAN device-specific reset using SoC global reset via BAR0
+ * registers.
+ */
+static int reset_qualcomm_wlan(struct pci_dev *pdev, bool probe)
+{
+	void __iomem *bar;
+	u32 val;
+	u16 cmd;
+	int ret;
+
+	if (probe)
+		return 0;
+
+	if (pdev->current_state != PCI_D0)
+		return -EINVAL;
+
+	pci_read_config_word(pdev, PCI_COMMAND, &cmd);
+	pci_write_config_word(pdev, PCI_COMMAND, cmd | PCI_COMMAND_MEMORY);
+
+	bar = pci_iomap(pdev, 0, 0);
+	if (!bar) {
+		pci_write_config_word(pdev, PCI_COMMAND, cmd);
+		return -ENODEV;
+	}
+
+	val = ioread32(bar + QUALCOMM_WLAN_PCIE_SOC_GLOBAL_RESET);
+	if (PCI_POSSIBLE_ERROR(val)) {
+		ret = -ENODEV;
+		goto out_restore;
+	}
+	val |= QUALCOMM_WLAN_PCIE_SOC_GLOBAL_RESET_V;
+	iowrite32(val, bar + QUALCOMM_WLAN_PCIE_SOC_GLOBAL_RESET);
+	ioread32(bar + QUALCOMM_WLAN_PCIE_SOC_GLOBAL_RESET);
+
+	msleep(10);
+
+	val &= ~QUALCOMM_WLAN_PCIE_SOC_GLOBAL_RESET_V;
+	iowrite32(val, bar + QUALCOMM_WLAN_PCIE_SOC_GLOBAL_RESET);
+	ioread32(bar + QUALCOMM_WLAN_PCIE_SOC_GLOBAL_RESET);
+
+	msleep(10);
+
+	ret = read_poll_timeout(ioread32, val,
+				!PCI_POSSIBLE_ERROR(val),
+				20 * USEC_PER_MSEC,
+				5 * USEC_PER_SEC, false,
+				bar + QUALCOMM_WLAN_PCIE_SOC_GLOBAL_RESET);
+	if (ret) {
+		pci_err(pdev, "PCIe link failed to recover after reset\n");
+		goto out_restore;
+	}
+
+	/* After SOC_GLOBAL_RESET, MHISTATUS may still have SYSERR bit set
+	 * and thus need to set MHICTRL_RESET to clear SYSERR.
+	 */
+	iowrite32(QUALCOMM_WLAN_MHICTRL_RESET_MASK, bar + QUALCOMM_WLAN_MHICTRL);
+	ioread32(bar + QUALCOMM_WLAN_MHICTRL);
+
+	msleep(10);
+
+out_restore:
+	pci_iounmap(pdev, bar);
+	pci_write_config_word(pdev, PCI_COMMAND, cmd);
+
+	return ret;
+}
+
+#define MHI_SOC_RESET_REQ_OFFSET		0xb0
+#define MHI_SOC_RESET_REQ			BIT(0)
+
+/*
+ * Qualcomm modem device-specific reset using MHI SoC reset via BAR0
+ * register.
+ */
+static int reset_qualcomm_modem(struct pci_dev *pdev, bool probe)
+{
+	void __iomem *bar;
+	u16 cmd;
+
+	if (probe)
+		return 0;
+
+	if (pdev->current_state != PCI_D0)
+		return -EINVAL;
+
+	pci_read_config_word(pdev, PCI_COMMAND, &cmd);
+	pci_write_config_word(pdev, PCI_COMMAND, cmd | PCI_COMMAND_MEMORY);
+
+	bar = pci_iomap(pdev, 0, 0);
+	if (!bar) {
+		pci_write_config_word(pdev, PCI_COMMAND, cmd);
+		return -ENODEV;
+	}
+
+	iowrite32(MHI_SOC_RESET_REQ, bar + MHI_SOC_RESET_REQ_OFFSET);
+	ioread32(bar + MHI_SOC_RESET_REQ_OFFSET);
+
+	/* Be sure device reset has been executed */
+	msleep(2000);
+
+	pci_iounmap(pdev, bar);
+	pci_write_config_word(pdev, PCI_COMMAND, cmd);
+
+	return 0;
+}
+
 static const struct pci_dev_reset_methods pci_dev_reset_methods[] = {
 	{ PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_INTEL_82599_SFP_VF,
 		 reset_intel_82599_sfp_virtfn },
@@ -4255,6 +4368,9 @@ static const struct pci_dev_reset_methods pci_dev_reset_methods[] = {
 		reset_chelsio_generic_dev },
 	{ PCI_VENDOR_ID_HUAWEI, PCI_DEVICE_ID_HINIC_VF,
 		reset_hinic_vf_dev },
+	{ PCI_VENDOR_ID_QCOM, 0x0308, reset_qualcomm_modem }, /* SDX62/SDX65 modems */
+	{ PCI_VENDOR_ID_QCOM, 0x1103, reset_qualcomm_wlan },  /* WCN6855 WLAN */
+	{ PCI_VENDOR_ID_QCOM, 0x1107, reset_qualcomm_wlan },  /* WCN7850 WLAN */
 	{ 0 }
 };
 
Signature · 1 line
2.54.0
#1 lore raw
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [High] Polling MMIO registers and issuing flush reads on a disconnected PCIe link will trigger Master…

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] Polling MMIO registers and issuing flush reads on a disconnected PCIe link will trigger Master Aborts, escalating to AER errors and kernel panics on strict architectures. --

--- Patch [1]: [PATCH v13] PCI: Add device-specific reset for Qualcomm devices --- commit 8de0dab811e2555e876d5163ff0092117c834bb3
Author: Jose Ignacio Tornos Martinez <[email protected]>

PCI: Add device-specific reset for Qualcomm devices

This commit adds device-specific reset methods for certain Qualcomm PCIe devices (WCN6855/WCN7850 WLAN cards and SDX62/SDX65 modems) that lack working reset methods for VFIO passthrough. It implements hardware reset sequences using BAR-space registers via SoC global reset and MHI SoC reset mechanisms.

Quoted 4 lines
drivers/pci/quirks.c +0 -0
index b09f27f7846fc..c3799e6c58584 100644

[ ... ]

Quoted 1 line
drivers/pci/quirks.c +0 -0
@@ -4227,6 +4228,118 @@ static int reset_hinic_vf_dev(struct pci_dev *pdev, bool probe)

[ ... ]

Quoted 2 lines
drivers/pci/quirks.c +2 -0
+static int reset_qualcomm_wlan(struct pci_dev *pdev, bool probe)
+{

[ ... ]

Quoted 7 lines
drivers/pci/quirks.c +7 -0
+	val |= QUALCOMM_WLAN_PCIE_SOC_GLOBAL_RESET_V;
+	iowrite32(val, bar + QUALCOMM_WLAN_PCIE_SOC_GLOBAL_RESET);
+	ioread32(bar + QUALCOMM_WLAN_PCIE_SOC_GLOBAL_RESET);
+
+	msleep(10);
+
+	val &= ~QUALCOMM_WLAN_PCIE_SOC_GLOBAL_RESET_V;

drivers/pci/quirks.c static int reset_hinic_vf_dev(struct pci_dev *pdev, bool probe) · line 4296

Quoted 10 lines
drivers/pci/quirks.c +10 -0
+	iowrite32(val, bar + QUALCOMM_WLAN_PCIE_SOC_GLOBAL_RESET);
+	ioread32(bar + QUALCOMM_WLAN_PCIE_SOC_GLOBAL_RESET);
+
+	msleep(10);
+
+	ret = read_poll_timeout(ioread32, val,
+				!PCI_POSSIBLE_ERROR(val),
+				20 * USEC_PER_MSEC,
+				5 * USEC_PER_SEC, false,
+				bar + QUALCOMM_WLAN_PCIE_SOC_GLOBAL_RESET);

[Severity: High]
Will this sequence of ioread32 flush reads and read_poll_timeout trigger Master Aborts on architectures with strict error handling?

The SoC global reset explicitly drops the device PCIe link. Performing ioread32 flush reads immediately after asserting the reset, and then actively polling the disconnected MMIO space with read_poll_timeout while the link recovers, can result in Unsupported Request errors.

On strict platforms like ARM64 or x86 servers with APEI/GHES Firmware First error handling, these can escalate to fatal Uncorrectable Errors and cause a host kernel panic when a guest VM exits.

Should this avoid MMIO flush reads during these reset sequences and instead use fixed delays to safely bridge the link-down period?

[ ... ]

Quoted 2 lines
drivers/pci/quirks.c +2 -0
+static int reset_qualcomm_modem(struct pci_dev *pdev, bool probe)
+{

[ ... ]

drivers/pci/quirks.c static int reset_hinic_vf_dev(struct pci_dev *pdev, bool probe) · line 4345

Quoted 2 lines
drivers/pci/quirks.c +2 -0
+	iowrite32(MHI_SOC_RESET_REQ, bar + MHI_SOC_RESET_REQ_OFFSET);
+	ioread32(bar + MHI_SOC_RESET_REQ_OFFSET);

[Severity: High]
Could this immediate ioread32 flush read also trigger a fatal Master Abort if the link drops instantly after writing the MHI SoC reset request?

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
#2 lore raw
Quoted 11 lines

[Severity: High]
Will this sequence of ioread32 flush reads and read_poll_timeout trigger Master Aborts on architectures with strict error handling?

The SoC global reset explicitly drops the device PCIe link. Performing ioread32 flush reads immediately after asserting the reset, and then actively polling the disconnected MMIO space with read_poll_timeout while the link recovers, can result in Unsupported Request errors.

Quoted 8 lines

On strict platforms like ARM64 or x86 servers with APEI/GHES Firmware First error handling, these can escalate to fatal Uncorrectable Errors and cause a host kernel panic when a guest VM exits.

Should this avoid MMIO flush reads during these reset sequences and instead use fixed delays to safely bridge the link-down period?

The SoC global reset at 0x3008 does not immediately drop the PCIe link. The flush reads execute while the device is still responsive - the actual reset takes effect during the subsequent msleep. The ath11k/ath12k drivers perform the same register accesses during this reset sequence, including on ARM64 Qualcomm platforms, without issues.

The read_poll_timeout polling for link recovery follows the same pattern used elsewhere in the PCI subsystem for waiting on device responses after reset.

Tested over 100+ VM crash/reset cycles on x86 with VFIO passthrough without errors.

Quoted 4 lines

[Severity: High]
Could this immediate ioread32 flush read also trigger a fatal Master Abort if the link drops instantly after writing the MHI SoC reset request?

Same as above - the MHI SoC reset does not instantly drop the PCIe link. The flush read completes before the reset takes effect, and the 2-second delay covers the actual reset period. This replicates the MHI driver behavior (mhi_pci_reset_prepare()).

#3 lore raw
Quoted 20 lines · Jose Ignacio Tornos Martinez
Quoted 11 lines

[Severity: High]
Will this sequence of ioread32 flush reads and read_poll_timeout trigger Master Aborts on architectures with strict error handling?

The SoC global reset explicitly drops the device PCIe link. Performing ioread32 flush reads immediately after asserting the reset, and then actively polling the disconnected MMIO space with read_poll_timeout while the link recovers, can result in Unsupported Request errors.

Quoted 8 lines

On strict platforms like ARM64 or x86 servers with APEI/GHES Firmware First error handling, these can escalate to fatal Uncorrectable Errors and cause a host kernel panic when a guest VM exits.

Should this avoid MMIO flush reads during these reset sequences and instead use fixed delays to safely bridge the link-down period?

Quoted 7 lines

The SoC global reset at 0x3008 does not immediately drop the PCIe link. The flush reads execute while the device is still responsive - the actual reset takes effect during the subsequent msleep. The ath11k/ath12k drivers perform the same register accesses during this reset sequence, including on ARM64 Qualcomm platforms, without issues.

Yes, Qcom devices supporting SoC reset have a mechanism to keep the PCIe link alive (in L0/L1) when the reset gets triggered in the device. So the register access is available throughout the reset cycle.

- Mani

Signature · 1 line
மணிவண்ணன் சதாசிவம்
maintainer Reviewed-by on v13 1/1
#4 lore raw
In reply to Jose Ignacio Tornos Martinez
Quoted 32 lines · Jose Ignacio Tornos Martinez

Some Qualcomm PCIe devices (WCN6855/WCN7850 WLAN cards, SDX62/SDX65 modems) lack working reset methods for VFIO passthrough scenarios. These devices have no FLR capability, advertise NoSoftRst+ (blocking PM reset), and have broken bus reset.

The problem manifests in VFIO passthrough scenarios:

- WCN6855 (17cb:1103) and WCN7850 (17cb:1107) WLAN devices:
Normal VM operation works fine, including clean shutdown/reboot. However, when the VM terminates uncleanly (crash, force-off), VFIO attempts to reset the device before it can be assigned to another VM. Without a working reset method, the device remains in an undefined state, preventing reuse.

- SDX62/SDX65 (17cb:0308) 5G modems: Never successfully initialize even on first VM assignment without proper reset capability.

Add device-specific reset methods using BAR-space hardware reset registers that exist in these devices:

- WCN6855/WCN7850 WLAN devices use SoC global reset via BAR0 (sequence from ath11k/ath12k driver: ath11k_pci_soc_global_reset(), ath11k_pci_sw_reset(), ath11k_mhi_set_mhictrl_reset()):
- Write/clear reset bit at offset 0x3008
- Wait for PCIe link recovery (up to 5 seconds)
- Clear MHI controller SYSERR status at offset 0x38

- SDX62/SDX65 modem devices use MHI SoC reset via BAR0 (sequence from MHI
driver: mhi_soc_reset(), mhi_pci_reset_prepare()):
- Write reset request to offset 0xb0
- Wait 2 seconds for reset completion

Quoted 8 lines

These are true hardware reset mechanisms (not power management or firmware error recovery), providing proper device reset for VFIO scenarios.

Testing was performed on desktop platforms with M.2 WLAN and modem cards using M.2-to-PCIe adapters, including extensive force-reset cycling to verify stability.

- Mani

Quoted 156 lines

---
v13: Address Alex Williamson feedback:
- Validate initial ioread32() with PCI_POSSIBLE_ERROR() before read-modify-write to avoid writing error response back to device on platforms with APEI/GHES error escalation
- Replace time_before()/msleep() polling loop with read_poll_timeout() for robustness against scheduling delays and code simplification
v12: https://lore.kernel.org/all/[email protected]/

 drivers/pci/quirks.c | 116 +++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 116 insertions(+)
drivers/pci/quirks.c +116 -0
index 431c021d7414..bd1e0742052e 100644
@@ -22,6 +22,7 @@
 #include <linux/isa-dma.h> /* isa_dma_bridge_buggy */
 #include <linux/init.h>
 #include <linux/iommu.h>
+#include <linux/iopoll.h>
 #include <linux/delay.h>
 #include <linux/acpi.h>
 #include <linux/dmi.h>
@@ -4240,6 +4241,118 @@ static int reset_hinic_vf_dev(struct pci_dev *pdev, bool probe)
 	return 0;
 }
 
+#define QUALCOMM_WLAN_PCIE_SOC_GLOBAL_RESET	0x3008
+#define QUALCOMM_WLAN_PCIE_SOC_GLOBAL_RESET_V	BIT(0)
+#define QUALCOMM_WLAN_MHICTRL			0x38
+#define QUALCOMM_WLAN_MHICTRL_RESET_MASK	0x2
+
+/*
+ * Qualcomm WLAN device-specific reset using SoC global reset via BAR0
+ * registers.
+ */
+static int reset_qualcomm_wlan(struct pci_dev *pdev, bool probe)
+{
+	void __iomem *bar;
+	u32 val;
+	u16 cmd;
+	int ret;
+
+	if (probe)
+		return 0;
+
+	if (pdev->current_state != PCI_D0)
+		return -EINVAL;
+
+	pci_read_config_word(pdev, PCI_COMMAND, &cmd);
+	pci_write_config_word(pdev, PCI_COMMAND, cmd | PCI_COMMAND_MEMORY);
+
+	bar = pci_iomap(pdev, 0, 0);
+	if (!bar) {
+		pci_write_config_word(pdev, PCI_COMMAND, cmd);
+		return -ENODEV;
+	}
+
+	val = ioread32(bar + QUALCOMM_WLAN_PCIE_SOC_GLOBAL_RESET);
+	if (PCI_POSSIBLE_ERROR(val)) {
+		ret = -ENODEV;
+		goto out_restore;
+	}
+	val |= QUALCOMM_WLAN_PCIE_SOC_GLOBAL_RESET_V;
+	iowrite32(val, bar + QUALCOMM_WLAN_PCIE_SOC_GLOBAL_RESET);
+	ioread32(bar + QUALCOMM_WLAN_PCIE_SOC_GLOBAL_RESET);
+
+	msleep(10);
+
+	val &= ~QUALCOMM_WLAN_PCIE_SOC_GLOBAL_RESET_V;
+	iowrite32(val, bar + QUALCOMM_WLAN_PCIE_SOC_GLOBAL_RESET);
+	ioread32(bar + QUALCOMM_WLAN_PCIE_SOC_GLOBAL_RESET);
+
+	msleep(10);
+
+	ret = read_poll_timeout(ioread32, val,
+				!PCI_POSSIBLE_ERROR(val),
+				20 * USEC_PER_MSEC,
+				5 * USEC_PER_SEC, false,
+				bar + QUALCOMM_WLAN_PCIE_SOC_GLOBAL_RESET);
+	if (ret) {
+		pci_err(pdev, "PCIe link failed to recover after reset\n");
+		goto out_restore;
+	}
+
+	/* After SOC_GLOBAL_RESET, MHISTATUS may still have SYSERR bit set
+	 * and thus need to set MHICTRL_RESET to clear SYSERR.
+	 */
+	iowrite32(QUALCOMM_WLAN_MHICTRL_RESET_MASK, bar + QUALCOMM_WLAN_MHICTRL);
+	ioread32(bar + QUALCOMM_WLAN_MHICTRL);
+
+	msleep(10);
+
+out_restore:
+	pci_iounmap(pdev, bar);
+	pci_write_config_word(pdev, PCI_COMMAND, cmd);
+
+	return ret;
+}
+
+#define MHI_SOC_RESET_REQ_OFFSET		0xb0
+#define MHI_SOC_RESET_REQ			BIT(0)
+
+/*
+ * Qualcomm modem device-specific reset using MHI SoC reset via BAR0
+ * register.
+ */
+static int reset_qualcomm_modem(struct pci_dev *pdev, bool probe)
+{
+	void __iomem *bar;
+	u16 cmd;
+
+	if (probe)
+		return 0;
+
+	if (pdev->current_state != PCI_D0)
+		return -EINVAL;
+
+	pci_read_config_word(pdev, PCI_COMMAND, &cmd);
+	pci_write_config_word(pdev, PCI_COMMAND, cmd | PCI_COMMAND_MEMORY);
+
+	bar = pci_iomap(pdev, 0, 0);
+	if (!bar) {
+		pci_write_config_word(pdev, PCI_COMMAND, cmd);
+		return -ENODEV;
+	}
+
+	iowrite32(MHI_SOC_RESET_REQ, bar + MHI_SOC_RESET_REQ_OFFSET);
+	ioread32(bar + MHI_SOC_RESET_REQ_OFFSET);
+
+	/* Be sure device reset has been executed */
+	msleep(2000);
+
+	pci_iounmap(pdev, bar);
+	pci_write_config_word(pdev, PCI_COMMAND, cmd);
+
+	return 0;
+}
+
 static const struct pci_dev_reset_methods pci_dev_reset_methods[] = {
 	{ PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_INTEL_82599_SFP_VF,
 		 reset_intel_82599_sfp_virtfn },
@@ -4255,6 +4368,9 @@ static const struct pci_dev_reset_methods pci_dev_reset_methods[] = {
 		reset_chelsio_generic_dev },
 	{ PCI_VENDOR_ID_HUAWEI, PCI_DEVICE_ID_HINIC_VF,
 		reset_hinic_vf_dev },
+	{ PCI_VENDOR_ID_QCOM, 0x0308, reset_qualcomm_modem }, /* SDX62/SDX65 modems */
+	{ PCI_VENDOR_ID_QCOM, 0x1103, reset_qualcomm_wlan },  /* WCN6855 WLAN */
+	{ PCI_VENDOR_ID_QCOM, 0x1107, reset_qualcomm_wlan },  /* WCN7850 WLAN */
 	{ 0 }
 };
 
Signature · 1 line
2.54.0
Signature · 1 line
மணிவண்ணன் சதாசிவம்
#5 lore raw

Hi Bjorn,

Friendly ping on this patch, now almost two months old.
It has Mani's Reviewed-by and has been tested over 100+ VM crash/reset cycles.

The approach was developed following Alex's and Mani's guidance through the series, using device-specific hardware reset registers.

Without this, Qualcomm WLAN and modem devices cannot be reused after unclean VM termination in VFIO passthrough.

Could you help me to know if there is anything blocking this in order to move forward?

Thanks
Jose Ignacio

#6 lore raw
In reply to Jose Ignacio Tornos Martinez
Quoted 5 lines · Jose Ignacio Tornos Martinez

Some Qualcomm PCIe devices (WCN6855/WCN7850 WLAN cards, SDX62/SDX65 modems) lack working reset methods for VFIO passthrough scenarios. These devices have no FLR capability, advertise NoSoftRst+ (blocking PM reset), and have broken bus reset.

Quoted 8 lines

The problem manifests in VFIO passthrough scenarios:

- WCN6855 (17cb:1103) and WCN7850 (17cb:1107) WLAN devices:
Normal VM operation works fine, including clean shutdown/reboot. However, when the VM terminates uncleanly (crash, force-off), VFIO attempts to reset the device before it can be assigned to another VM. Without a working reset method, the device remains in an undefined state, preventing reuse.

Just to clarify the commit log, I asked earlier about why this says "when the VM terminates *uncleanly*", since VFIO resets the device on every reassignment regardless of whether the VM termination was clean or unclean.

But I didn't understand the answer (here's the question/answer from
https://lore.kernel.org/all/[email protected]/t/#m853c96910f7c842dca597f5c0a0f2fcc90d98be1):

Quoted 8 lines
Quoted 3 lines
Quoted 3 lines

I don't know enough about VFIO, but I sort of expected that VFIO would reset devices between reassignment regardless of how a VM terminates. I guess that's not true?

VFIO does attempt reset on every reassignment. Without a working reset method, the attempt fails and the device remains in undefined state. With this quirk, D3hot successfully resets the device allowing reassignment.

I guess you are saying that some kind of reset *does* work fine in normal, clean VM termination? If reset works fine for normal VM termination but not for VM crash, do we have any idea why they are different?

Quoted 13 lines

- SDX62/SDX65 (17cb:0308) 5G modems: Never successfully initialize even on first VM assignment without proper reset capability.

Add device-specific reset methods using BAR-space hardware reset registers that exist in these devices:

- WCN6855/WCN7850 WLAN devices use SoC global reset via BAR0 (sequence from ath11k/ath12k driver: ath11k_pci_soc_global_reset(), ath11k_pci_sw_reset(), ath11k_mhi_set_mhictrl_reset()):
- Write/clear reset bit at offset 0x3008
- Wait for PCIe link recovery (up to 5 seconds)
- Clear MHI controller SYSERR status at offset 0x38

Quoted 4 lines

- SDX62/SDX65 modem devices use MHI SoC reset via BAR0 (sequence from MHI
driver: mhi_soc_reset(), mhi_pci_reset_prepare()):
- Write reset request to offset 0xb0
- Wait 2 seconds for reset completion

Since these basically copy code from other drivers, is there an opportunity to remove it from those drivers and use this instead? I see that the maintainers for drivers/net/wireless/ath/ath11k/pci.c and drivers/bus/mhi/host/main.c are already cc'd, so this is something that could be done later (if it's possible); doesn't need to be done before merging this patch.

This looks like two patches squashed together. If there's any other reason to touch this patch, I would split them apart.

Quoted 164 lines

These are true hardware reset mechanisms (not power management or firmware error recovery), providing proper device reset for VFIO scenarios.

Testing was performed on desktop platforms with M.2 WLAN and modem cards using M.2-to-PCIe adapters, including extensive force-reset cycling to verify stability.

---
v13: Address Alex Williamson feedback:
- Validate initial ioread32() with PCI_POSSIBLE_ERROR() before read-modify-write to avoid writing error response back to device on platforms with APEI/GHES error escalation
- Replace time_before()/msleep() polling loop with read_poll_timeout() for robustness against scheduling delays and code simplification
v12: https://lore.kernel.org/all/[email protected]/

 drivers/pci/quirks.c | 116 +++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 116 insertions(+)
drivers/pci/quirks.c +116 -0
index 431c021d7414..bd1e0742052e 100644
@@ -22,6 +22,7 @@
 #include <linux/isa-dma.h> /* isa_dma_bridge_buggy */
 #include <linux/init.h>
 #include <linux/iommu.h>
+#include <linux/iopoll.h>
 #include <linux/delay.h>
 #include <linux/acpi.h>
 #include <linux/dmi.h>
@@ -4240,6 +4241,118 @@ static int reset_hinic_vf_dev(struct pci_dev *pdev, bool probe)
 	return 0;
 }
 
+#define QUALCOMM_WLAN_PCIE_SOC_GLOBAL_RESET	0x3008
+#define QUALCOMM_WLAN_PCIE_SOC_GLOBAL_RESET_V	BIT(0)
+#define QUALCOMM_WLAN_MHICTRL			0x38
+#define QUALCOMM_WLAN_MHICTRL_RESET_MASK	0x2
+
+/*
+ * Qualcomm WLAN device-specific reset using SoC global reset via BAR0
+ * registers.
+ */
+static int reset_qualcomm_wlan(struct pci_dev *pdev, bool probe)
+{
+	void __iomem *bar;
+	u32 val;
+	u16 cmd;
+	int ret;
+
+	if (probe)
+		return 0;
+
+	if (pdev->current_state != PCI_D0)
+		return -EINVAL;
+
+	pci_read_config_word(pdev, PCI_COMMAND, &cmd);
+	pci_write_config_word(pdev, PCI_COMMAND, cmd | PCI_COMMAND_MEMORY);
+
+	bar = pci_iomap(pdev, 0, 0);
+	if (!bar) {
+		pci_write_config_word(pdev, PCI_COMMAND, cmd);
+		return -ENODEV;
+	}
+
+	val = ioread32(bar + QUALCOMM_WLAN_PCIE_SOC_GLOBAL_RESET);
+	if (PCI_POSSIBLE_ERROR(val)) {
+		ret = -ENODEV;
+		goto out_restore;
+	}
+	val |= QUALCOMM_WLAN_PCIE_SOC_GLOBAL_RESET_V;
+	iowrite32(val, bar + QUALCOMM_WLAN_PCIE_SOC_GLOBAL_RESET);
+	ioread32(bar + QUALCOMM_WLAN_PCIE_SOC_GLOBAL_RESET);
+
+	msleep(10);
+
+	val &= ~QUALCOMM_WLAN_PCIE_SOC_GLOBAL_RESET_V;
+	iowrite32(val, bar + QUALCOMM_WLAN_PCIE_SOC_GLOBAL_RESET);
+	ioread32(bar + QUALCOMM_WLAN_PCIE_SOC_GLOBAL_RESET);
+
+	msleep(10);
+
+	ret = read_poll_timeout(ioread32, val,
+				!PCI_POSSIBLE_ERROR(val),
+				20 * USEC_PER_MSEC,
+				5 * USEC_PER_SEC, false,
+				bar + QUALCOMM_WLAN_PCIE_SOC_GLOBAL_RESET);
+	if (ret) {
+		pci_err(pdev, "PCIe link failed to recover after reset\n");
+		goto out_restore;
+	}
+
+	/* After SOC_GLOBAL_RESET, MHISTATUS may still have SYSERR bit set
+	 * and thus need to set MHICTRL_RESET to clear SYSERR.
+	 */
+	iowrite32(QUALCOMM_WLAN_MHICTRL_RESET_MASK, bar + QUALCOMM_WLAN_MHICTRL);
+	ioread32(bar + QUALCOMM_WLAN_MHICTRL);
+
+	msleep(10);
+
+out_restore:
+	pci_iounmap(pdev, bar);
+	pci_write_config_word(pdev, PCI_COMMAND, cmd);
+
+	return ret;
+}
+
+#define MHI_SOC_RESET_REQ_OFFSET		0xb0
+#define MHI_SOC_RESET_REQ			BIT(0)
+
+/*
+ * Qualcomm modem device-specific reset using MHI SoC reset via BAR0
+ * register.
+ */
+static int reset_qualcomm_modem(struct pci_dev *pdev, bool probe)
+{
+	void __iomem *bar;
+	u16 cmd;
+
+	if (probe)
+		return 0;
+
+	if (pdev->current_state != PCI_D0)
+		return -EINVAL;
+
+	pci_read_config_word(pdev, PCI_COMMAND, &cmd);
+	pci_write_config_word(pdev, PCI_COMMAND, cmd | PCI_COMMAND_MEMORY);
+
+	bar = pci_iomap(pdev, 0, 0);
+	if (!bar) {
+		pci_write_config_word(pdev, PCI_COMMAND, cmd);
+		return -ENODEV;
+	}
+
+	iowrite32(MHI_SOC_RESET_REQ, bar + MHI_SOC_RESET_REQ_OFFSET);
+	ioread32(bar + MHI_SOC_RESET_REQ_OFFSET);
+
+	/* Be sure device reset has been executed */
+	msleep(2000);
+
+	pci_iounmap(pdev, bar);
+	pci_write_config_word(pdev, PCI_COMMAND, cmd);
+
+	return 0;
+}
+
 static const struct pci_dev_reset_methods pci_dev_reset_methods[] = {
 	{ PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_INTEL_82599_SFP_VF,
 		 reset_intel_82599_sfp_virtfn },
@@ -4255,6 +4368,9 @@ static const struct pci_dev_reset_methods pci_dev_reset_methods[] = {
 		reset_chelsio_generic_dev },
 	{ PCI_VENDOR_ID_HUAWEI, PCI_DEVICE_ID_HINIC_VF,
 		reset_hinic_vf_dev },
+	{ PCI_VENDOR_ID_QCOM, 0x0308, reset_qualcomm_modem }, /* SDX62/SDX65 modems */
+	{ PCI_VENDOR_ID_QCOM, 0x1103, reset_qualcomm_wlan },  /* WCN6855 WLAN */
+	{ PCI_VENDOR_ID_QCOM, 0x1107, reset_qualcomm_wlan },  /* WCN7850 WLAN */
 	{ 0 }
 };
 
Signature · 1 line
2.54.0
#7 lore raw

Hi Bjorn,

Thanks for the review.

Quoted 4 lines

I guess you are saying that some kind of reset *does* work fine in normal, clean VM termination? If reset works fine for normal VM termination but not for VM crash, do we have any idea why they are different?

It depends on the device:

- For the modems (SDX62/SDX65), without a proper reset capability, these devices never successfully initialize even on first VM assignment.

- For the WLAN devices (WCN6855/WCN7850), the reset fails in both cases, but on clean VM shutdown the guest driver's .shutdown/.remove callbacks properly deinitialize the device (stop DMA, disable interrupts, etc.), so it ends up in a known-good state even without a successful reset. On unclean termination (crash, force-off), the guest driver callbacks never run, the device stays in an undefined state, and without a working reset it cannot be reused.

I will try to clarify this better in the commit messages for v14.

Quoted 3 lines

Since these basically copy code from other drivers, is there an opportunity to remove it from those drivers and use this instead?

Yes, that could be a follow-up after this lands.

Quoted 3 lines

This looks like two patches squashed together. If there's any other reason to touch this patch, I would split them apart.

Ok, I will split the WLAN and modem resets into separate patches for v14.

Thanks

Best regards
Jose Ignacio

#8 lore raw
Quoted 15 lines · Jose Ignacio Tornos Martinez

Hi Bjorn,

Thanks for the review.

Quoted 4 lines

I guess you are saying that some kind of reset *does* work fine in normal, clean VM termination? If reset works fine for normal VM termination but not for VM crash, do we have any idea why they are different?

It depends on the device:

- For the modems (SDX62/SDX65), without a proper reset capability, these devices never successfully initialize even on first VM assignment.

Quoted 8 lines

- For the WLAN devices (WCN6855/WCN7850), the reset fails in both cases, but on clean VM shutdown the guest driver's .shutdown/.remove callbacks properly deinitialize the device (stop DMA, disable interrupts, etc.), so it ends up in a known-good state even without a successful reset. On unclean termination (crash, force-off), the guest driver callbacks never run, the device stays in an undefined state, and without a working reset it cannot be reused.

Ah, so this depends on the previous VM tenant to clean up after itself instead of the host kernel enforcing the isolation?

That seems important to mention and potentially unacceptable to tenants if an issue in the tenant's kernel may cause their data to be leaked to the next user of the device.

Quoted 7 lines

I will try to clarify this better in the commit messages for v14.

Quoted 3 lines

Since these basically copy code from other drivers, is there an opportunity to remove it from those drivers and use this instead?

Yes, that could be a follow-up after this lands.

Sounds good.

#9 lore raw

Hi Bjorn,

Quoted 3 lines

Ah, so this depends on the previous VM tenant to clean up after itself instead of the host kernel enforcing the isolation?

That is actually the current situation without this patch, there is no working reset method, and since it depends on the previous status, the host cannot enforce isolation at all.

This patch adds a hardware reset that the host uses on every device reassignment (device-specific reset is position #1 in the reset hierarchy), regardless of how the previous VM terminated. It improves isolation by giving the host a real reset mechanism where none exists today.

The clean/unclean distinction was only explaining why things happen to work today without a reset in some cases, not a limitation of the proposed reset.

Thanks

Best regards
Jose Ignacio