PCI/PM: Do not save or restore the config space of an inaccessible device

When a PCI device becomes inaccessible while the system is suspending, pci_save_state() happily stores 0xFFFFFFFF into all 16 dwords of the saved config space, and pci_restore_state() writes that back on resume -- to a device that by then *is* responding again.

On a MacBookPro14,3 (Intel Alpine Ridge Thunderbolt 3) this is not theoretical. The upstream bridge of the Thunderbolt switch stops responding during suspend, and on resume the restore leaves it with:

- the Secondary Bus Reset bit asserted (0xFF contains PCI_BRIDGE_CTL_BUS_RESET),
- primary/secondary/subordinate bus numbers set to ff/ff/ff,
- the link retrained down from 8 GT/s to 2.5 GT/s.

The visible effect is a 65-second resume while the kernel waits for devices that can no longer be reached, followed by the removal of both xHCI controllers: every USB-C port on the machine is gone until reboot.

The series makes the save path refuse to snapshot a device that is not there, the restore path refuse to write back a snapshot that is all ones, and teaches one more caller not to mistake an absent device for a working link.

Patch 1 makes the USB PCI HCD honour the pci_save_state() return value, which it currently ignores. It is a fix in its own right -- pci_save_state() can already fail today -- and it is placed first so that no patch in the series introduces an error condition before its caller knows how to handle it.

Measured on the affected machine, comparing the same suspend/resume cycle with and without the series:

- resume time for the affected bridge: 65 s -> 1 s;
- the bridge keeps its bus numbers (04/05/79) and its 8 GT/s link,

    with no Secondary Bus Reset asserted;

- the xHCI behind it survives the cycle: 1 of 2 controllers instead of 0 of 2, and 4 USB buses instead of 2.

v2 of this work fixes two defects found in review of v1: a pci_WARN_ONCE() that the first version could trigger in pci_pm_suspend_noirq(), and a partially written snapshot -- v1 could leave dev->saved_config_space half updated if the device disappeared while it was being read. v2 reads into a temporary buffer and only commits it after re-checking that the device is still there. Both were verified on hardware: the warning count went from 1 and 1 to 0 and 0, and the per-device resume timings are unchanged to within 0.01%.

Tested on 6.18.49 on the machine described above. I do not have other affected hardware, so wider testing of the PCI core changes would be welcome.

hcd_pci_suspend_noirq() calls pci_save_state() and ignores its return value, then goes on to call pci_prepare_to_sleep(). pci_save_state() can fail -- it already propagates failures from pci_save_pcie_state() and friends -- and when it does there is no saved state to restore later, so putting the device into a low-power state only makes matters
worse: the subsequent transition fails too, and the PCI core then warns that the callback returned without saving the state.

Check the return value. If the state could not be saved, leave the device alone and return success, so the rest of the system can still suspend. Leaving the power state untouched also means the PCI core has nothing to complain about.

---

@@ -541,7 +541,17 @@
 	if (retval)
 		return retval;
 
-	pci_save_state(pci_dev);
+	/*
+	 * If the controller is already inaccessible, there is no state to
+	 * save and nothing to put into a low-power state.  Leaving the power
+	 * state alone also keeps the PCI core from warning that this callback
+	 * returned without saving the state.
+	 */
+	retval = pci_save_state(pci_dev);
+	if (retval) {
+		dev_dbg(dev, "--> not suspending, device inaccessible\n");
+		return 0;
+	}
 
 	/* If the root hub is dead rather than suspended, disallow remote
 	 * wakeup.  usb_hc_died() should ensure that both hosts are marked as

pci_save_state() reads sixteen dwords in a bare loop and then marks the snapshot valid unconditionally. If the device is already inaccessible, every read returns all ones, and that garbage replaces a previously good snapshot.

Restoring it later does not merely fail to help. On a bridge that is still alive, writing all ones sets every writable bit of BRIDGE_CONTROL, which asserts Secondary Bus Reset, and clears the primary, secondary and subordinate bus numbers, which unmaps everything behind the bridge. On a MacBookPro14,3 this is what removes the Thunderbolt USB controllers after a suspend/resume cycle: the bridge keeps answering, but the kernel has just written 0xffffffff over its configuration.

Check whether the device answers before saving, and again afterwards, because it can disappear while the loop is running -- on this machine the window between a successful read and a failing one has been measured at a few microseconds. Read into a temporary buffer so the previous, known-good snapshot survives if either check fails.

---

@@ -1740,13 +1740,43 @@
  */
 int pci_save_state(struct pci_dev *dev)
 {
+	u32 buf[16];
 	int i;
+	u32 val;
+
+	/*
+	 * If the device is already inaccessible, every config read returns
+	 * all ones.  Saving that would replace a previously good snapshot
+	 * with garbage, and restoring the garbage later does not merely fail
+	 * to help, it actively damages the device: on a bridge it asserts
+	 * Secondary Bus Reset and clears the bus numbers, which unmaps
+	 * everything behind it.  Keep the old snapshot instead.
+	 */
+	pci_read_config_dword(dev, PCI_VENDOR_ID, &val);
+	if (PCI_POSSIBLE_ERROR(val)) {
+		pci_warn(dev, "not saving config space, device inaccessible\n");
+		return -EIO;
+	}
+
+	/*
+	 * Read into a temporary buffer: the device can become inaccessible
+	 * while we are reading, and then only part of the snapshot is all
+	 * ones.  The previous snapshot must stay intact until we know the
+	 * new one is good.
+	 */
 	/* XXX: 100% dword access ok here? */
 	for (i = 0; i < 16; i++) {
-		pci_read_config_dword(dev, i * 4, &dev->saved_config_space[i]);
-		pci_dbg(dev, "save config %#04x: %#010x\n",
-			i * 4, dev->saved_config_space[i]);
+		pci_read_config_dword(dev, i * 4, &buf[i]);
+		pci_dbg(dev, "save config %#04x: %#010x\n", i * 4, buf[i]);
 	}
+
+	pci_read_config_dword(dev, PCI_VENDOR_ID, &val);
+	if (PCI_POSSIBLE_ERROR(val)) {
+		pci_warn(dev, "not saving config space, device became inaccessible\n");
+		return -EIO;
+	}
+
+	memcpy(dev->saved_config_space, buf, sizeof(buf));
 	dev->state_saved = true;
 
 	i = pci_save_pcie_state(dev);

Belt and braces for the previous patch: even if a snapshot of all ones is somehow recorded, never write it back. The vendor ID of a device that was present can never be 0xffffffff, so a snapshot whose first dword is all ones is known to be garbage.

---

@@ -1833,6 +1833,17 @@
 
 static void pci_restore_config_space(struct pci_dev *pdev)
 {
+	/*
+	 * Vendor and Device ID are read-only, so a snapshot whose first
+	 * dword reads as all ones was taken from an inaccessible device and
+	 * is garbage.  Writing it back would corrupt a device that has since
+	 * become accessible again, so leave the hardware alone.
+	 */
+	if (PCI_POSSIBLE_ERROR(pdev->saved_config_space[0])) {
+		pci_warn(pdev, "not restoring config space, saved state is invalid\n");
+		return;
+	}
+
 	if (pdev->hdr_type == PCI_HEADER_TYPE_NORMAL) {
 		pci_restore_config_space_range(pdev, 10, 15, 0, false);
 		/* Restore BARs before the command register. */

pci_bridge_wait_for_secondary_bus() reads PCI_EXP_LNKSTA and treats PCI_EXP_LNKSTA_DLLLA as "the link is up". When the device is gone the read returns 0xffff, which has that bit set, so an absent device looks like an active link and the code waits PCIE_RESET_READY_POLL_MS -- 60 seconds -- before giving up.

Treat an all-ones read as "no link", which is what it means. On a MacBookPro14,3 whose Thunderbolt controller does not come back from S3 this takes the resume from 65 seconds down to one.

---

@@ -4773,8 +4773,16 @@
 		if (!dev->link_active_reporting)
 			return -ENOTTY;
 
+		/*
+		 * A device that is gone answers config reads with all ones,
+		 * and all ones has DLLLA set, so an unchecked read here is
+		 * indistinguishable from an active link.  That makes the
+		 * code below wait the full PCIE_RESET_READY_POLL_MS for a
+		 * device that is not there.  Treat it as absent instead.
+		 */
 		pcie_capability_read_word(dev, PCI_EXP_LNKSTA, &status);
-		if (!(status & PCI_EXP_LNKSTA_DLLLA))
+		if (PCI_POSSIBLE_ERROR(status) ||
+		    !(status & PCI_EXP_LNKSTA_DLLLA))
 			return -ENOTTY;
 
 		return pci_dev_wait(child, reset_type,

Please don't apply patches 2/4 and 3/4 as they stand: they break SR-IOV virtual functions.

Both treat an all-ones first dword (Vendor ID and Device ID) as "device inaccessible", but a VF always reads 0xffff there; that is why pci_device_is_present() checks the PF instead. With 2/4, pci_save_state() fails for every VF, from pci_bus_add_device() and from pci_dev_save_and_disable() before a reset, so no snapshot is ever taken and nothing valid is restored after the reset. With 3/4 alone, a VF's config space is never restored. The machine I tested on has no SR-IOV devices, so nothing there could show it.

I'll send a v2 that uses pci_device_is_present() in 2/4 and reworks 3/4 accordingly. Two more things it will fix:

- The cover letter speaks of "v2 of this work", but the earlier version was only reviewed privately and never posted. The next posting will be v2, with changes listed against this one.

- 1/4 says the device is left alone when pci_save_state() fails. That holds on its own, but with 2/4 applied the save fails without setting state_saved, so pci_pm_suspend_noirq() calls pci_save_state() and pci_prepare_to_sleep() itself afterwards. The code in 1/4 is not affected, but its description is incomplete. Alan, I'm pointing this out since you acked it with that text.

The v2 will also carry the Assisted-by tag this posting was missing.

Francisco