Skip to contentSkip to conversation

vfio/pci: Handle PCI error recovery and report state to userspace

RFC PATCH In review Shameer Kolothum 3 participants 42 messages Nov 12

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

v3 cover vfio/pci: Handle PCI error recovery and report state to userspace

Hi,

Currently, vfio-pci signals the error eventfd from error_detected() and returns PCI_ERS_RESULT_CAN_RECOVER, including for permanent failure. It has no slot_reset() or resume() callback, so userspace is not notified when host recovery completes or whether the device was reset.

This series adds opt-in host PCI error recovery for vfio-pci. It blocks device access during recovery, restores device state after a host reset, and reports recovery status to userspace. The kernel runs host recovery and the VMM decides how to recover the guest.

Major Changes from v2:
v2: https://lore.kernel.org/kvm/[email protected]/

- Rework device access serialization around live and blocked access ops, using SRCU to drain ongoing accesses before recovery.
- Address the D0 bus-lock cycle described in RFC v2. Add pci_try_set_power_state() for ordinary transitions and use the locked helper during recovery to avoid taking pci_bus_sem again.
- Fix INTx restoration ordering, recovery status reporting and PF access during VF information queries.
- Limit enablement to supported native PCIe recovery paths. Exclude functions with VPD redirected to function zero, and Intel Ethernet function zero devices with VPD.

Design
------
Userspace opts in through VFIO_DEVICE_FEATURE_PCI_ERROR_RECOVERY. It registers an eventfd and reads status flags and a sequence number. The existing error eventfd is still signaled.

The callbacks handle the host recovery sequence:

- error_detected() blocks new accesses and drains admitted ones, quiesces INTx, and revokes BAR mappings and exported DMA-BUFs. On a normal channel it also clears bus mastering. It votes CAN_RECOVER for a normal channel, NEED_RESET for a frozen channel, and DISCONNECT for permanent failure. Failure to save PCI_COMMAND or disable bus mastering leaves access blocked and returns NONE so other devices can continue recovery.
- slot_reset() restores the configuration saved at open after a host reset, then tears down interrupt configuration. Userspace must set up interrupts again.
- resume() completes restoration, unblocks access and completes INTx handling. A restoration failure keeps the device blocked.

Non-fatal errors also block device access because host recovery may require a reset. Userspace reads the recovery status to determine whether the device was reset or recovery failed.

Without opt-in, the existing error notification behavior is retained. Open also rejects attempts while a recorded host recovery is in progress, even when userspace has not enabled recovery.

Locking and open questions
-------------------------

PCI core holds device_lock during recovery callbacks and pci_bus_sem for reading during the supported recovery walk.

VFIO uses access_lock to serialize recovery and switch access ops. Recovery blocks new accesses and waits for existing SRCU readers to finish. Switching ops also takes irqlock to serialize with INTx callbacks, which do not use SRCU. Power transitions and mapping changes use memory_lock.

The D0 bus-lock cycle described in RFC v2 is addressed by the try-lock helper for ordinary transitions and the locked helper in recovery. This uses the locking provided by the supported native PCI bus walk. It does not impose a new locking requirement on every PCI recovery dispatcher.

Pre-existing(Sashiko reported):
--------------
- Open and close were not serialized with the whole host recovery transaction before this series. That remains the case. Recovery can start after the open check, and close can overlap the physical reset. Restoration of closed devices is also still missing.

LLM tools assisted with implementation. I also ran local Sashiko reviews.

Testing
-------
Basic sanity tests performed on a GB200 with an NVIDIA GPU assigned.

Kernel branch:
https://github.com/shamiali2008/linux/tree/vfio-aer-rfc-v3-ext

QEMU test branch is here(This registers the recovery eventfd and uses pcie_aer_inject_error() to report the error to Guest)
https://github.com/shamiali2008/qemu-master/commits/private-master-vfio-aer-test-v2/

Software AER injection was performed using a modified pcieaer_inject module.

./aer-inject nonfatal.conf

qemu-system-aarch64: info: vfio 0018:06:00.0: host PCI error recovery completed (seq 1)

Guest kernel:

[   59.036988] pcieport 0000:01:00.0: AER: Uncorrectable (Non-Fatal) error message received from 0000:02:00.0
[   59.038326] nvidia 0000:02:00.0: PCIe Bus Error: severity=Uncorrectable (Non-Fatal), type=Transaction Layer, (Completer ID)
[   59.038440] nvidia 0000:02:00.0:   device [10de:2941] error status/mask=00008000/02400000
[   59.038533] nvidia 0000:02:00.0:    [15] CmpltAbrt              (First)
[   59.038705] nvidia 0000:02:00.0: AER:   TLP Header: 0x00000000 0x00000000 0x00000000 0x00000000

...

./aer-inject fatal.conf
qemu-system-aarch64: info: vfio 0018:06:00.0: host PCI error recovery started (seq 2, channel frozen)
qemu-system-aarch64: info: vfio 0018:06:00.0: host PCI error recovery completed (seq 2, device was reset)

Guest kernel:

[  150.909306] pcieport 0000:01:00.0: AER: Uncorrectable (Fatal) error message received from 0000:02:00.0
[  150.909566] nvidia 0000:02:00.0: AER: PCIe Bus Error: severity=Uncorrectable (Fatal), type=Inaccessible, (Unregistered Agent ID)
[  150.909835] nvidia 0000:02:00.0: AER: can't recover (no error_detected callback)
[  150.910062] pcieport 0000:01:00.0: unlocked secondary bus reset via: pciehp_reset_slot+0x54/0x98
[  156.603467] pcieport 0000:01:00.0: AER: Root Port link has been reset (0)

...

Please take a look and let me know your feedback.

Thanks,
Shameer

Shameer Kolothum (20):
vfio/pci: Add SRCU-protected access ops
PCI/PM: Add a power-state helper that tries the bus lock vfio/pci: Avoid waiting for the bus lock when preparing D0 vfio/pci: Use access ops for config-space accesses vfio/pci: Buffer ROM reads before copying to userspace vfio/pci: Use access ops for BAR and ROM accesses vfio/pci: Use access ops when mapping BAR pages vfio/pci: Use access ops for hardware IRQ operations vfio/pci: Use access ops for function reset
vfio/pci: Use access ops for runtime-PM requests vfio/pci: Use access ops for hardware information queries vfio/pci: Use access ops for DMA-BUF export and restoration vfio/pci: Track PCI recovery state
vfio/pci: Add INTx recovery operations
vfio/pci: Use access ops for SR-IOV configuration and shared reset vfio/pci: Restore device state from slot_reset() vfio/pci: Complete recovery in resume()
vfio/pci: Block device access during host recovery vfio/pci: Add VFIO_DEVICE_FEATURE_PCI_ERROR_RECOVERY vfio/pci: Enable recovery for supported native PCIe endpoints

 drivers/vfio/pci/vfio_pci_priv.h   |  58 ++
 include/linux/pci.h                |   3 +
 include/linux/vfio_pci_core.h      |  36 +-
 include/uapi/linux/vfio.h          |  55 ++
 drivers/pci/pci.c                  |  37 ++
 drivers/vfio/pci/vfio_pci.c        |  88 ++-
 drivers/vfio/pci/vfio_pci_config.c | 122 +++-
 drivers/vfio/pci/vfio_pci_core.c   | 955 +++++++++++++++++++++++++++--
 drivers/vfio/pci/vfio_pci_dmabuf.c | 113 ++--
 drivers/vfio/pci/vfio_pci_intrs.c  | 183 +++++-
 drivers/vfio/pci/vfio_pci_rdwr.c   | 297 ++++++---
 11 files changed, 1705 insertions(+), 242 deletions(-)
Signature · 1 line
2.43.0
#1 lore raw

RFC PATCH v3 1/20 vfio/pci: Add SRCU-protected access ops

Before recovering a device, VFIO needs to stop new hardware accesses and wait for operations already in progress to finish.

Give each open device an SRCU instance to track hardware accesses. Enable SRCU access tracking only for devices with pci_recovery_supported set.
Select and call the access operation within the same SRCU read-side section. Keep copies to and from userspace outside that section so a page fault cannot hold up recovery.

Introduce live and blocked tables. Initialize the ops pointer to blocked when the device is created, and select live ops when open finishes. Close uses a blocking transition that installs blocked ops and drains SRCU readers before teardown. Open publishes live ops after setup. Serialize these transitions with access_lock. Internal setup and cleanup use their own helpers.

Provide an accessor for operations that use their existing synchronization instead of SRCU. The accessor only returns the selected table. It does not acquire a lock or keep the device's per-open resources alive.

Subsequent patches convert hardware access paths to these ops and add support for host PCI error recovery using the same blocked table.

---
 drivers/vfio/pci/vfio_pci_priv.h | 15 ++++++
 include/linux/vfio_pci_core.h    | 10 ++++
 drivers/vfio/pci/vfio_pci_core.c | 86 ++++++++++++++++++++++++++++++++
 3 files changed, 111 insertions(+)
drivers/vfio/pci/vfio_pci_priv.h +15 -0
index 4e7162234a2e..97b69e4bc8be 100644
@@ -73,6 +73,21 @@ u16 vfio_pci_memory_lock_and_enable(struct vfio_pci_core_device *vdev);
 void vfio_pci_memory_unlock_and_restore(struct vfio_pci_core_device *vdev,
 					u16 cmd);
 
+/* Operations receive kernel buffers only. User copies stay outside SRCU. */
+typedef long (*vfio_pci_access_fn)(struct vfio_pci_core_device *vdev,
+				 void *data);
+long vfio_pci_core_access(struct vfio_pci_core_device *vdev,
+			  vfio_pci_access_fn fn, void *data);
+
+struct vfio_pci_access_ops {
+	/* Select and call within SRCU. The callback receives kernel buffers. */
+	long (*execute)(struct vfio_pci_core_device *vdev,
+			vfio_pci_access_fn fn, void *data);
+};
+
+const struct vfio_pci_access_ops *
+vfio_pci_access_ops(struct vfio_pci_core_device *vdev);
+
 #ifdef CONFIG_VFIO_PCI_IGD
 bool vfio_pci_is_intel_display(struct pci_dev *pdev);
 int vfio_pci_igd_init(struct vfio_pci_core_device *vdev);
include/linux/vfio_pci_core.h +10 -0
index 2f3fa2787737..88aa7b752231 100644
@@ -14,6 +14,7 @@
 #include <linux/vfio.h>
 #include <linux/irqbypass.h>
 #include <linux/rcupdate.h>
+#include <linux/srcu.h>
 #include <linux/types.h>
 #include <linux/uuid.h>
 #include <linux/notifier.h>
@@ -97,6 +98,8 @@ static inline int vfio_pci_core_get_dmabuf_phys(
 }
 #endif
 
+struct vfio_pci_access_ops;
+
 struct vfio_pci_core_device {
 	struct vfio_device	vdev;
 	struct pci_dev		*pdev;
@@ -131,6 +134,7 @@ struct vfio_pci_core_device {
 	bool			disable_idle_d3:1;
 	bool			nointxmask:1;
 	bool			disable_vga:1;
+	bool			pci_recovery_supported:1;
 	/* Flags modified at runtime - dedicated storage unit */
 	bool			needs_reset;
 	bool			pm_intx_masked;
@@ -150,6 +154,12 @@ struct vfio_pci_core_device {
 	struct vfio_pci_core_device	*sriov_pf_core_dev;
 	struct notifier_block	nb;
 	struct rw_semaphore	memory_lock;
+	struct srcu_struct	access_srcu;
+	/* Updated under access_lock. */
+	const struct vfio_pci_access_ops __rcu *access_ops;
+	/* Set after open completes, cleared before close tears down state. */
+	bool			device_open;
+	struct mutex		access_lock;	/* protects changes to access ops and flags */
 	struct list_head	dmabufs;
 };
 
drivers/vfio/pci/vfio_pci_core.c +86 -0

132 lines not on this page · raw message

Signature · 1 line
2.43.0
#2 lore raw

Re: [RFC PATCH v3 01/20] vfio/pci: Add SRCU-protected access ops

Sashiko has reviewed this patch and found no issues.
Quoted 17 lines

Before recovering a device, VFIO needs to stop new hardware accesses and wait for operations already in progress to finish.

Give each open device an SRCU instance to track hardware accesses. Enable SRCU access tracking only for devices with pci_recovery_supported set.
Select and call the access operation within the same SRCU read-side section. Keep copies to and from userspace outside that section so a page fault cannot hold up recovery.

Introduce live and blocked tables. Initialize the ops pointer to blocked when the device is created, and select live ops when open finishes. Close uses a blocking transition that installs blocked ops and drains SRCU readers before teardown. Open publishes live ops after setup. Serialize these transitions with access_lock. Internal setup and cleanup use their own helpers.

Quoted 8 lines

Provide an accessor for operations that use their existing synchronization instead of SRCU. The accessor only returns the selected table. It does not acquire a lock or keep the device's per-open resources alive.

Subsequent patches convert hardware access paths to these ops and add support for host PCI error recovery using the same blocked table.

Sashiko has reviewed this patch and found no issues.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
#3 lore raw

RFC PATCH v3 2/20 PCI/PM: Add a power-state helper that tries the bus lock

In reply to Shameer Kolothum

PCI error recovery can hold pci_bus_sem for reading while waiting for a lock held by a driver. If that driver also needs the bus read lock to bring a device to D0, a queued writer can block it. Recovery, the driver and the writer then all wait for each other.

Add pci_try_set_power_state(), which returns -EAGAIN if it cannot take the bus lock immediately. The caller can then release its locks. Once the bus lock is held, use pci_set_power_state_locked() for the transition. Keep the existing cases that need no power-state change. The transition itself can still sleep.

Callers that already hold the bus read lock must use the locked helper instead.

---
 include/linux/pci.h |  3 +++
 drivers/pci/pci.c   | 37 +++++++++++++++++++++++++++++++++++++
 2 files changed, 40 insertions(+)
include/linux/pci.h +3 -0
index aa3765a20a96..542fa4af0a43 100644
@@ -1535,6 +1535,7 @@ int pci_load_and_free_saved_state(struct pci_dev *dev,
 int pci_platform_power_transition(struct pci_dev *dev, pci_power_t state);
 int pci_set_power_state(struct pci_dev *dev, pci_power_t state);
 int pci_set_power_state_locked(struct pci_dev *dev, pci_power_t state);
+int pci_try_set_power_state(struct pci_dev *dev, pci_power_t state);
 pci_power_t pci_choose_state(struct pci_dev *dev, pm_message_t state);
 bool pci_pme_capable(struct pci_dev *dev, pci_power_t state);
 void pci_pme_active(struct pci_dev *dev, bool enable);
@@ -2216,6 +2217,8 @@ static inline int pci_set_power_state(struct pci_dev *dev, pci_power_t state)
 { return 0; }
 static inline int pci_set_power_state_locked(struct pci_dev *dev, pci_power_t state)
 { return 0; }
+static inline int pci_try_set_power_state(struct pci_dev *dev, pci_power_t state)
+{ return 0; }
 static inline int pci_wake_from_d3(struct pci_dev *dev, bool enable)
 { return 0; }
 static inline pci_power_t pci_choose_state(struct pci_dev *dev,
drivers/pci/pci.c +37 -0
index 7c6f2aba8144..f2ebd19204aa 100644
@@ -1655,6 +1655,43 @@ int pci_set_power_state_locked(struct pci_dev *dev, pci_power_t state)
 }
 EXPORT_SYMBOL(pci_set_power_state_locked);
 
+/**
+ * pci_try_set_power_state - Set power state without waiting for the bus lock
+ * @dev: PCI device to handle
+ * @state: Target PCI power state
+ *
+ * Like pci_set_power_state(), but return -EAGAIN without changing the power
+ * state if the bus lock cannot be acquired immediately. The transition
+ * itself can sleep.
+ *
+ * Callers already holding the bus read lock must use
+ * pci_set_power_state_locked().
+ */
+int pci_try_set_power_state(struct pci_dev *dev, pci_power_t state)
+{
+	int ret;
+
+	if (state > PCI_D3cold)
+		state = PCI_D3cold;
+	else if (state < PCI_D0)
+		state = PCI_D0;
+	else if ((state == PCI_D1 || state == PCI_D2) && pci_no_d1d2(dev))
+		return 0;
+
+	if (dev->current_state == state)
+		return 0;
+	if (state >= PCI_D3hot && (dev->dev_flags & PCI_DEV_FLAGS_NO_D3))
+		return 0;
+
+	if (!down_read_trylock(&pci_bus_sem))
+		return -EAGAIN;
+	ret = pci_set_power_state_locked(dev, state);
+	up_read(&pci_bus_sem);
+
+	return ret;
+}
+EXPORT_SYMBOL(pci_try_set_power_state);
+
 #define PCI_EXP_SAVE_REGS	7
 
 static struct pci_cap_saved_state *_pci_find_saved_cap(struct pci_dev *pci_dev,
Signature · 1 line
2.43.0
#4 lore raw

Re: [RFC PATCH v3 02/20] PCI/PM: Add a power-state helper that tries the bus lock

Sashiko has reviewed this patch and found no issues.
Quoted 11 lines

PCI error recovery can hold pci_bus_sem for reading while waiting for a lock held by a driver. If that driver also needs the bus read lock to bring a device to D0, a queued writer can block it. Recovery, the driver and the writer then all wait for each other.

Add pci_try_set_power_state(), which returns -EAGAIN if it cannot take the bus lock immediately. The caller can then release its locks. Once the bus lock is held, use pci_set_power_state_locked() for the transition. Keep the existing cases that need no power-state change. The transition itself can still sleep.

Quoted 4 lines

Callers that already hold the bus read lock must use the locked helper instead.

Sashiko has reviewed this patch and found no issues.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
Acked-by on v3 2/20
#5 lore raw

Re: [RFC PATCH v3 02/20] PCI/PM: Add a power-state helper that tries the bus lock

In reply to Shameer Kolothum
Quoted 11 lines · Shameer Kolothum

PCI error recovery can hold pci_bus_sem for reading while waiting for a lock held by a driver. If that driver also needs the bus read lock to bring a device to D0, a queued writer can block it. Recovery, the driver and the writer then all wait for each other.

Add pci_try_set_power_state(), which returns -EAGAIN if it cannot take the bus lock immediately. The caller can then release its locks. Once the bus lock is held, use pci_set_power_state_locked() for the transition. Keep the existing cases that need no power-state change. The transition itself can still sleep.

Quoted 4 lines

Callers that already hold the bus read lock must use the locked helper instead.

Quoted 77 lines
---
 include/linux/pci.h |  3 +++
 drivers/pci/pci.c   | 37 +++++++++++++++++++++++++++++++++++++
 2 files changed, 40 insertions(+)
include/linux/pci.h +3 -0
index aa3765a20a96..542fa4af0a43 100644
@@ -1535,6 +1535,7 @@ int pci_load_and_free_saved_state(struct pci_dev *dev,
 int pci_platform_power_transition(struct pci_dev *dev, pci_power_t state);
 int pci_set_power_state(struct pci_dev *dev, pci_power_t state);
 int pci_set_power_state_locked(struct pci_dev *dev, pci_power_t state);
+int pci_try_set_power_state(struct pci_dev *dev, pci_power_t state);
 pci_power_t pci_choose_state(struct pci_dev *dev, pm_message_t state);
 bool pci_pme_capable(struct pci_dev *dev, pci_power_t state);
 void pci_pme_active(struct pci_dev *dev, bool enable);
@@ -2216,6 +2217,8 @@ static inline int pci_set_power_state(struct pci_dev *dev, pci_power_t state)
 { return 0; }
 static inline int pci_set_power_state_locked(struct pci_dev *dev, pci_power_t state)
 { return 0; }
+static inline int pci_try_set_power_state(struct pci_dev *dev, pci_power_t state)
+{ return 0; }
 static inline int pci_wake_from_d3(struct pci_dev *dev, bool enable)
 { return 0; }
 static inline pci_power_t pci_choose_state(struct pci_dev *dev,
drivers/pci/pci.c +37 -0
index 7c6f2aba8144..f2ebd19204aa 100644
@@ -1655,6 +1655,43 @@ int pci_set_power_state_locked(struct pci_dev *dev, pci_power_t state)
 }
 EXPORT_SYMBOL(pci_set_power_state_locked);
 
+/**
+ * pci_try_set_power_state - Set power state without waiting for the bus lock
+ * @dev: PCI device to handle
+ * @state: Target PCI power state
+ *
+ * Like pci_set_power_state(), but return -EAGAIN without changing the power
+ * state if the bus lock cannot be acquired immediately. The transition
+ * itself can sleep.
+ *
+ * Callers already holding the bus read lock must use
+ * pci_set_power_state_locked().
+ */
+int pci_try_set_power_state(struct pci_dev *dev, pci_power_t state)
+{
+	int ret;
+
+	if (state > PCI_D3cold)
+		state = PCI_D3cold;
+	else if (state < PCI_D0)
+		state = PCI_D0;
+	else if ((state == PCI_D1 || state == PCI_D2) && pci_no_d1d2(dev))
+		return 0;
+
+	if (dev->current_state == state)
+		return 0;
+	if (state >= PCI_D3hot && (dev->dev_flags & PCI_DEV_FLAGS_NO_D3))
+		return 0;
+
+	if (!down_read_trylock(&pci_bus_sem))
+		return -EAGAIN;
+	ret = pci_set_power_state_locked(dev, state);
+	up_read(&pci_bus_sem);
+
+	return ret;
+}
+EXPORT_SYMBOL(pci_try_set_power_state);
+
 #define PCI_EXP_SAVE_REGS	7
 
 static struct pci_cap_saved_state *_pci_find_saved_cap(struct pci_dev *pci_dev,
Signature · 1 line
2.43.0
#6 lore raw

RFC PATCH v3 3/20 vfio/pci: Avoid waiting for the bus lock when preparing D0

In reply to Shameer Kolothum

A D0 transition can wait behind a pci_bus_sem writer while holding locks needed by PCI error recovery. Recovery holds the bus read lock, leaving all three tasks waiting for each other.

Use pci_try_set_power_state() for D0 preparation during function reset, hot reset, runtime suspend and SR-IOV enable. Preserve VFIO's saved PM state handling and unwind reset and SR-IOV requests if D0 preparation fails.

Runtime suspend returns -EAGAIN on bus-lock contention. Keep its existing handling of other D0 errors to avoid leaving runtime PM in an error state.

Apply this even when recovery is disabled, since global IRQ cleanup can wait for another device's D0 transition. Low-power transitions keep the existing helper. Subsequent patch handles config-space power requests.

---
 drivers/vfio/pci/vfio_pci_priv.h |  2 ++
 drivers/vfio/pci/vfio_pci_core.c | 49 +++++++++++++++++++++++++++-----
 2 files changed, 44 insertions(+), 7 deletions(-)
drivers/vfio/pci/vfio_pci_priv.h +2 -0
index 97b69e4bc8be..2a15a8956205 100644
@@ -67,6 +67,8 @@ void vfio_config_free(struct vfio_pci_core_device *vdev);
 
 int vfio_pci_set_power_state(struct vfio_pci_core_device *vdev,
 			     pci_power_t state);
+int vfio_pci_try_set_power_state(struct vfio_pci_core_device *vdev,
+				 pci_power_t state);
 
 void vfio_pci_zap_and_down_write_memory_lock(struct vfio_pci_core_device *vdev);
 u16 vfio_pci_memory_lock_and_enable(struct vfio_pci_core_device *vdev);
drivers/vfio/pci/vfio_pci_core.c +42 -7

101 lines not on this page · raw message

Signature · 1 line
2.43.0
#7 lore raw

Re: [RFC PATCH v3 03/20] vfio/pci: Avoid waiting for the bus lock when preparing D0

Sashiko has reviewed this patch and found no issues.
Quoted 9 lines

A D0 transition can wait behind a pci_bus_sem writer while holding locks needed by PCI error recovery. Recovery holds the bus read lock, leaving all three tasks waiting for each other.

Use pci_try_set_power_state() for D0 preparation during function reset, hot reset, runtime suspend and SR-IOV enable. Preserve VFIO's saved PM state handling and unwind reset and SR-IOV requests if D0 preparation fails.

Quoted 8 lines

Runtime suspend returns -EAGAIN on bus-lock contention. Keep its existing handling of other D0 errors to avoid leaving runtime PM in an error state.

Apply this even when recovery is disabled, since global IRQ cleanup can wait for another device's D0 transition. Low-power transitions keep the existing helper. Subsequent patch handles config-space power requests.

Sashiko has reviewed this patch and found no issues.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
#8 lore raw

RFC PATCH v3 4/20 vfio/pci: Use access ops for config-space accesses

In reply to Shameer Kolothum

Config-space handlers can touch hardware during recovery. Run them through vfio_pci_core_access() and keep userspace copies outside SRCU so page faults cannot delay recovery.

Defer D0 requests until the config handler has left SRCU. Select the power-state operation under memory_lock and use the bus try-lock helper for D0. Return transition errors to userspace.

Blocked power operations reject requests. Open and close continue to use the internal power helper.

---
 drivers/vfio/pci/vfio_pci_priv.h   |   5 ++
 drivers/vfio/pci/vfio_pci_config.c | 122 +++++++++++++++++++++++------
 drivers/vfio/pci/vfio_pci_core.c   |   8 ++
 3 files changed, 109 insertions(+), 26 deletions(-)
drivers/vfio/pci/vfio_pci_priv.h +5 -0
index 2a15a8956205..0ea7c710dc38 100644
@@ -41,6 +41,9 @@ ssize_t vfio_pci_config_rw_single(struct vfio_pci_core_device *vdev,
 				  char __user *buf, size_t count, loff_t *ppos,
 				  bool iswrite);
 
+int vfio_lock_and_set_power_state(struct vfio_pci_core_device *vdev,
+				  pci_power_t state);
+
 ssize_t vfio_pci_bar_rw(struct vfio_pci_core_device *vdev, char __user *buf,
 			size_t count, loff_t *ppos, bool iswrite);
 
@@ -85,6 +88,8 @@ struct vfio_pci_access_ops {
 	/* Select and call within SRCU. The callback receives kernel buffers. */
 	long (*execute)(struct vfio_pci_core_device *vdev,
 			vfio_pci_access_fn fn, void *data);
+	/* Caller holds memory_lock. */
+	int (*set_power_state)(struct vfio_pci_core_device *vdev, pci_power_t state);
 };
 
 const struct vfio_pci_access_ops *
drivers/vfio/pci/vfio_pci_config.c +96 -26

260 lines not on this page · raw message

drivers/vfio/pci/vfio_pci_core.c +8 -0
index 6529eab7865b..21c0dda37dd1 100644
@@ -324,12 +324,20 @@ static long vfio_pci_access_blocked(struct vfio_pci_core_device *vdev,
 	return -EIO;
 }
 
+static int vfio_pci_power_blocked(struct vfio_pci_core_device *vdev,
+				  pci_power_t state)
+{
+	return -EIO;
+}
+
 static const struct vfio_pci_access_ops vfio_pci_live_ops = {
 	.execute = vfio_pci_access_live,
+	.set_power_state = vfio_pci_try_set_power_state,
 };
 
 static const struct vfio_pci_access_ops vfio_pci_blocked_ops = {
 	.execute = vfio_pci_access_blocked,
+	.set_power_state = vfio_pci_power_blocked,
 };
 
 /*
Signature · 1 line
2.43.0
#9 lore raw

Re: [RFC PATCH v3 04/20] vfio/pci: Use access ops for config-space accesses

Thank you for your contribution! Sashiko AI review found 1 potential issue to consider: - [High] vfio/pci: pci_bus_sem lock inversion in config-space FLR under access_srcu --

Thank you for your contribution! Sashiko AI review found 1 potential issue to consider:
- [High] vfio/pci: pci_bus_sem lock inversion in config-space FLR under access_srcu
--

commit 07dee5e58e5bbc760b7e991d4fc5ab36537fd5e0
Author: Shameer Kolothum <[email protected]>

vfio/pci: Use access ops for config-space accesses

This commit runs PCI config-space accesses through vfio_pci_core_access(), routes power-state transitions through vfio_pci_access_ops, and defers PCI_PM_CTRL D0 transitions until after leaving access_srcu.

Quoted 4 lines
drivers/vfio/pci/vfio_pci_config.c +0 -0
index 9914f3ac69aef..1e54e2e8b2ef8 100644

[ ... ]

Quoted 5 lines
drivers/vfio/pci/vfio_pci_config.c +0 -1
@@ -1957,11 +2017,21 @@ ssize_t vfio_pci_config_rw_single(struct vfio_pci_core_device *vdev,
 		if (copy_from_user(&val, buf, count))
 			return -EFAULT;
 
-		ret = perm->writefn(vdev, *ppos, count, perm, offset, val);

drivers/vfio/pci/vfio_pci_config.c ssize_t vfio_pci_config_rw_single(struct vfio_pci_core_device *vdev, · line 2029

Quoted 10 lines
drivers/vfio/pci/vfio_pci_config.c +10 -0
+		access.val = val;
+		ret = vfio_pci_core_access(vdev, vfio_pci_access_config, &access);
+		if (ret < 0)
+			return ret;
+		if (access.power_up) {
+			int err = vfio_lock_and_set_power_state(vdev, PCI_D0);
+
+			if (err)
+				return err;
+		}

[Severity: High]
Can a config-space FLR write in vfio_exp_config_write() or vfio_af_config_write() invert the lock order between access_srcu and pci_bus_sem when the device is in a low-power state?

vfio_pm_config_write() defers D0 transitions outside access_srcu so that pcie_aspm_pm_state_change() does not block on pci_bus_sem while AER waits for access_srcu readers. However, if userspace puts the device into D1-D3hot via PCI_PM_CTRL and then writes PCI_EXP_DEVCTL_BCR_FLR or PCI_AF_CTRL_FLR, vfio_pci_core_access() runs the handler inside access_srcu, where it takes vdev->memory_lock and calls pci_try_reset_function():

drivers/pci/pci.c:pci_dev_save_and_disable() {
    ...
	pci_set_power_state(dev, PCI_D0);
    ...
}

pci_set_power_state() calls pci_set_full_power_state(dev, false) -> pcie_aspm_pm_state_change() -> down_read(&pci_bus_sem) while holding both access_srcu and vdev->memory_lock:

CPU0 CPU1 (AER recovery)
---- -------------------

                                  down_read(&pci_bus_sem)
vfio_pci_config_rw_single()
  vfio_pci_core_access()
    srcu_read_lock(&access_srcu)
    ...
    vfio_exp_config_write()
      down_write(&memory_lock)
      pci_try_reset_function()
        ...
        down_read(&pci_bus_sem)
                                  synchronize_srcu(&access_srcu)

*** DEADLOCK ***

(if a writer is queued on pci_bus_sem, or against any path nesting pci_bus_sem -> access_srcu / memory_lock)

In addition, letting pci_dev_save_and_disable() perform the D3hot -> D0 transition directly bypasses vfio_pci_try_set_power_state() and skips restoring vdev->pm_save on NoSoftRst- devices.

Calling vfio_pci_try_set_power_state(vdev, PCI_D0) before pci_try_reset_function() in vfio_exp_config_write() and vfio_af_config_write() (as commit 358432f8bbe9 ("vfio/pci: Avoid waiting for the bus lock when preparing D0") did for vfio_pci_ioctl_reset()), or deferring the FLR outside access_srcu, would avoid this.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=4
#10 lore raw

RFC PATCH v3 5/20 vfio/pci: Buffer ROM reads before copying to userspace

In reply to Shameer Kolothum

ROM reads copy directly to userspace while ROM decoding is enabled. Putting the whole operation inside SRCU would let a userspace page fault delay recovery.

Read the requested ROM data into a kernel buffer, then unmap the ROM before copying to userspace. Hold memory_lock across mapping and reading to serialize ROM decoding with power changes and other ROM accesses.

Preserve byte, word and dword reads and 0xff padding beyond the ROM. Subsequent patch adds SRCU protection around the hardware access.

---
 drivers/vfio/pci/vfio_pci_rdwr.c | 147 ++++++++++++++++++++++---------
 1 file changed, 104 insertions(+), 43 deletions(-)
drivers/vfio/pci/vfio_pci_rdwr.c +104 -43
index df8848087681..10cd091031fe 100644
@@ -10,8 +10,10 @@
  * Author: Tom Lyon, [email protected]
  */
 
+#include <linux/align.h>
 #include <linux/fs.h>
 #include <linux/pci.h>
+#include <linux/slab.h>
 #include <linux/uaccess.h>
 #include <linux/io.h>
 #include <linux/vfio.h>
@@ -198,6 +200,96 @@ ssize_t vfio_pci_core_do_io_rw(struct vfio_pci_core_device *vdev, bool test_mem,
 }
 EXPORT_SYMBOL_GPL(vfio_pci_core_do_io_rw);
 
+static ssize_t vfio_pci_rom_read(struct vfio_pci_core_device *vdev,
+				 char __user *buf, size_t count, loff_t pos)
+{
+	struct pci_dev *pdev = vdev->pdev;
+	bool rom_bar = pci_resource_start(pdev, PCI_ROM_RESOURCE);
+	size_t size, length = 0, done = 0;
+	void *data = NULL;
+	void __iomem *io;
+	ssize_t ret;
+
+	/* Protect ROM decoding against concurrent power changes and ROM accesses. */
+	down_write(&vdev->memory_lock);
+	if (rom_bar) {
+		io = pci_map_rom(pdev, &size);
+	} else {
+		io = ioremap(pdev->rom, pdev->romlen);
+		size = pdev->romlen;
+	}
+	if (!io) {
+		ret = -ENOMEM;
+		goto out_unlock;
+	}
+
+	/* Buffer only ROM data, not unused space in a large ROM BAR. */
+	if (pos < size)
+		length = min(count, size - (size_t)pos);
+	if (length) {
+		if ((pci_resource_flags(pdev, PCI_ROM_RESOURCE) & IORESOURCE_MEM) &&
+		    !__vfio_pci_memory_enabled(vdev)) {
+			ret = -EIO;
+			goto out_unmap;
+		}
+		data = kvmalloc(length, GFP_KERNEL_ACCOUNT);
+		if (!data) {
+			ret = -ENOMEM;
+			goto out_unmap;
+		}
+	}
+
+	/*
+	 * Certain devices (e.g. Intel X710) don't support qword
+	 * access to the ROM bar. Otherwise PCI AER errors might be
+	 * triggered.
+	 *
+	 * Disable qword access to the ROM bar universally, which
+	 * worked reliably for years before qword access is enabled.
+	 */
+	while (done < length) {
+		if (length - done >= 4 && IS_ALIGNED(pos + done, 4)) {
+			u32 val = vfio_ioread32(io + pos + done);
+
+			memcpy(data + done, &val, sizeof(val));
+			done += sizeof(val);
+		} else if (length - done >= 2 && IS_ALIGNED(pos + done, 2)) {
+			u16 val = vfio_ioread16(io + pos + done);
+
+			memcpy(data + done, &val, sizeof(val));
+			done += sizeof(val);
+		} else {
+			((u8 *)data)[done] = vfio_ioread8(io + pos + done);
+			done++;
+		}
+	}
+	ret = count;
+
+out_unmap:
+	if (rom_bar)
+		pci_unmap_rom(pdev, io);
+	else
+		iounmap(io);
+out_unlock:
+	up_write(&vdev->memory_lock);
+	if (ret < 0)
+		goto out_free;
+
+	if (length && copy_to_user(buf, data, length)) {
+		ret = -EFAULT;
+		goto out_free;
+	}
+	for (done = length; done < count; done++) {
+		if (put_user((u8)0xff, buf + done)) {
+			ret = -EFAULT;
+			break;
+		}
+	}
+out_free:
+	kvfree(data);
+	return ret;
+}
+
 ssize_t vfio_pci_bar_rw(struct vfio_pci_core_device *vdev, char __user *buf,
 			size_t count, loff_t *ppos, bool iswrite)
 {
@@ -209,7 +301,6 @@ ssize_t vfio_pci_bar_rw(struct vfio_pci_core_device *vdev, char __user *buf,
 	void __iomem *io;
 	struct resource *res = &vdev->pdev->resource[bar];
 	ssize_t done;
-	enum vfio_pci_io_width max_width = VFIO_PCI_IO_WIDTH_8;
 
 	if (pci_resource_start(pdev, bar))
 		end = pci_resource_len(pdev, bar);
@@ -224,57 +315,27 @@ ssize_t vfio_pci_bar_rw(struct vfio_pci_core_device *vdev, char __user *buf,
 	count = min(count, (size_t)(end - pos));
 
 	if (bar == PCI_ROM_RESOURCE) {
-		/*
-		 * The ROM can fill less space than the BAR, so we start the
-		 * excluded range at the end of the actual ROM.  This makes
-		 * filling large ROM BARs much faster.
-		 */
-		if (pci_resource_start(pdev, bar)) {
-			io = pci_map_rom(pdev, &x_start);
-		} else {
-			io = ioremap(pdev->rom, pdev->romlen);
-			x_start = pdev->romlen;
-		}
-		if (!io)
-			return -ENOMEM;
-		x_end = end;
-
-		/*
-		 * Certain devices (e.g. Intel X710) don't support qword
-		 * access to the ROM bar. Otherwise PCI AER errors might be
-		 * triggered.
-		 *
-		 * Disable qword access to the ROM bar universally, which
-		 * worked reliably for years before qword access is enabled.
-		 */
-		max_width = VFIO_PCI_IO_WIDTH_4;
+		if (iswrite)
+			return -EINVAL;
+		done = vfio_pci_rom_read(vdev, buf, count, pos);
 	} else {
 		io = vfio_pci_core_get_iomap(vdev, bar);
-		if (IS_ERR(io)) {
-			done = PTR_ERR(io);
-			goto out;
+		if (IS_ERR(io))
+			return PTR_ERR(io);
+
+		if (bar == vdev->msix_bar) {
+			x_start = vdev->msix_offset;
+			x_end = vdev->msix_offset + vdev->msix_size;
 		}
-	}
 
-	if (bar == vdev->msix_bar) {
-		x_start = vdev->msix_offset;
-		x_end = vdev->msix_offset + vdev->msix_size;
+		done = vfio_pci_core_do_io_rw(vdev, res->flags & IORESOURCE_MEM,
+					      io, buf, pos, count, x_start, x_end,
+					      iswrite, VFIO_PCI_IO_WIDTH_8);
 	}
 
-	done = vfio_pci_core_do_io_rw(vdev, res->flags & IORESOURCE_MEM, io, buf, pos,
-				      count, x_start, x_end, iswrite, max_width);
-
 	if (done >= 0)
 		*ppos += done;
 
-	if (bar == PCI_ROM_RESOURCE) {
-		if (pci_resource_start(pdev, bar))
-			pci_unmap_rom(pdev, io);
-		else
-			iounmap(io);
-	}
-
-out:
 	return done;
 }
 
Signature · 1 line
2.43.0
#11 lore raw

Re: [RFC PATCH v3 05/20] vfio/pci: Buffer ROM reads before copying to userspace

Thank you for your contribution! Sashiko AI review found 1 potential issue to consider: - [Medium] vfio/pci: pci_map_rom() called before __vfio_pci_memory_enabled() in vfio_pci_rom_read() --

Thank you for your contribution! Sashiko AI review found 1 potential issue to consider:
- [Medium] vfio/pci: pci_map_rom() called before __vfio_pci_memory_enabled() in vfio_pci_rom_read()
--

commit 95981493fc3ccf127f13d3a0117382988fa550cd
Author: Shameer Kolothum <[email protected]>

vfio/pci: Buffer ROM reads before copying to userspace

This commit buffers PCI ROM reads into a kernel buffer under memory_lock and unmaps the ROM before copying the data to userspace.

Quoted 4 lines
drivers/vfio/pci/vfio_pci_rdwr.c +0 -0
index df88480876815..10cd091031fe4 100644

[ ... ]

Quoted 26 lines
drivers/vfio/pci/vfio_pci_rdwr.c +22 -0
@@ -198,6 +200,96 @@ ssize_t vfio_pci_core_do_io_rw(struct vfio_pci_core_device *vdev, bool test_mem,
 }
 EXPORT_SYMBOL_GPL(vfio_pci_core_do_io_rw);
 
+static ssize_t vfio_pci_rom_read(struct vfio_pci_core_device *vdev,
+				 char __user *buf, size_t count, loff_t pos)
+{
+	struct pci_dev *pdev = vdev->pdev;
+	bool rom_bar = pci_resource_start(pdev, PCI_ROM_RESOURCE);
+	size_t size, length = 0, done = 0;
+	void *data = NULL;
+	void __iomem *io;
+	ssize_t ret;
+
+	/* Protect ROM decoding against concurrent power changes and ROM accesses. */
+	down_write(&vdev->memory_lock);
+	if (rom_bar) {
+		io = pci_map_rom(pdev, &size);
+	} else {
+		io = ioremap(pdev->rom, pdev->romlen);
+		size = pdev->romlen;
+	}
+	if (!io) {
+		ret = -ENOMEM;
+		goto out_unlock;
+	}

drivers/vfio/pci/vfio_pci_rdwr.c ssize_t vfio_pci_core_do_io_rw(struct vfio_pci_core_device *vdev, bool test_mem, · line 234

Quoted 10 lines
drivers/vfio/pci/vfio_pci_rdwr.c +10 -0
+
+	/* Buffer only ROM data, not unused space in a large ROM BAR. */
+	if (pos < size)
+		length = min(count, size - (size_t)pos);
+	if (length) {
+		if ((pci_resource_flags(pdev, PCI_ROM_RESOURCE) & IORESOURCE_MEM) &&
+		    !__vfio_pci_memory_enabled(vdev)) {
+			ret = -EIO;
+			goto out_unmap;
+		}

[Severity: Medium]
Should __vfio_pci_memory_enabled(vdev) be checked before calling pci_map_rom()?

pci_map_rom() enables ROM decoding and calls pci_get_rom_size(), which performs MMIO reads on the ROM BAR:

drivers/pci/rom.c:pci_map_rom() {
	...
	if (pci_enable_rom(pdev))
		return NULL;

	rom = ioremap(start, *size);
	if (!rom)
		goto err_ioremap;

	*size = pci_get_rom_size(pdev, rom, *size);
	if (!*size)
		goto invalid_rom;
	...
}

If userspace reads the ROM region while PCI_COMMAND_MEMORY is cleared in vconfig or the device is in PCI_D3hot, pci_get_rom_size() still issues MMIO reads to the disabled or D3hot device, which can trigger Unsupported Request AER errors or platform faults. When those reads return 0xffff, pci_map_rom() also logs an invalid ROM signature error and returns NULL, so vfio_pci_rom_read() fails with -ENOMEM before reaching the __vfio_pci_memory_enabled() check.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=5
#12 lore raw

RFC PATCH v3 6/20 vfio/pci: Use access ops for BAR and ROM accesses

In reply to Shameer Kolothum

Recovery needs to wait for hardware reads and writes to finish before changing device state. Use access ops for individual MMIO and PIO reads and writes, covering BAR, VGA, port-I/O and ioeventfd accesses. Keep copies to and from userspace outside SRCU.

For ROM reads, keep mapping, reading into the buffer and unmapping within one SRCU read-side section. Copy the buffered data to userspace afterward.

---
 drivers/vfio/pci/vfio_pci_rdwr.c | 156 ++++++++++++++++++++-----------
 1 file changed, 102 insertions(+), 54 deletions(-)
drivers/vfio/pci/vfio_pci_rdwr.c +102 -54
index 10cd091031fe..8494e93c18fb 100644
@@ -40,57 +40,81 @@
 #define vfio_ioread8	ioread8
 #define vfio_iowrite8	iowrite8
 
-#define VFIO_IOWRITE(size) \
-int vfio_pci_core_iowrite##size(struct vfio_pci_core_device *vdev,	\
-			bool test_mem, u##size val, void __iomem *io)	\
-{									\
-	if (test_mem) {							\
-		down_read(&vdev->memory_lock);				\
-		if (!__vfio_pci_memory_enabled(vdev)) {			\
-			up_read(&vdev->memory_lock);			\
-			return -EIO;					\
-		}							\
-	}								\
-									\
-	vfio_iowrite##size(val, io);					\
-									\
-	if (test_mem)							\
-		up_read(&vdev->memory_lock);				\
-									\
-	return 0;							\
-}									\
-EXPORT_SYMBOL_GPL(vfio_pci_core_iowrite##size);
-
-VFIO_IOWRITE(8)
-VFIO_IOWRITE(16)
-VFIO_IOWRITE(32)
-VFIO_IOWRITE(64)
-
-#define VFIO_IOREAD(size) \
-int vfio_pci_core_ioread##size(struct vfio_pci_core_device *vdev,	\
-			bool test_mem, u##size *val, void __iomem *io)	\
-{									\
-	if (test_mem) {							\
-		down_read(&vdev->memory_lock);				\
-		if (!__vfio_pci_memory_enabled(vdev)) {			\
-			up_read(&vdev->memory_lock);			\
-			return -EIO;					\
-		}							\
-	}								\
-									\
-	*val = vfio_ioread##size(io);					\
-									\
-	if (test_mem)							\
-		up_read(&vdev->memory_lock);				\
-									\
-	return 0;							\
-}									\
-EXPORT_SYMBOL_GPL(vfio_pci_core_ioread##size);
-
-VFIO_IOREAD(8)
-VFIO_IOREAD(16)
-VFIO_IOREAD(32)
-VFIO_IOREAD(64)
+struct vfio_pci_io_access {
+	bool test_mem;
+	void __iomem *io;
+	void *val;
+};
+
+#define VFIO_IOWRITE(size, type)						\
+static long vfio_pci_access_iowrite##size(struct vfio_pci_core_device *vdev,	\
+					       void *data)			\
+{										\
+	struct vfio_pci_io_access *access = data;				\
+	type *val = access->val;						\
+										\
+	if (access->test_mem) {							\
+		down_read(&vdev->memory_lock);					\
+		if (!__vfio_pci_memory_enabled(vdev)) {				\
+			up_read(&vdev->memory_lock);				\
+			return -EIO;						\
+		}								\
+	}									\
+										\
+	vfio_iowrite##size(*val, access->io);					\
+	if (access->test_mem)							\
+		up_read(&vdev->memory_lock);					\
+	return 0;								\
+}										\
+										\
+int vfio_pci_core_iowrite##size(struct vfio_pci_core_device *vdev,		\
+			 bool test_mem, type val, void __iomem *io)		\
+{										\
+	struct vfio_pci_io_access access = { test_mem, io, &val };		\
+										\
+	return vfio_pci_core_access(vdev, vfio_pci_access_iowrite##size, &access);	\
+}										\
+EXPORT_SYMBOL_GPL(vfio_pci_core_iowrite##size)
+
+VFIO_IOWRITE(8, u8);
+VFIO_IOWRITE(16, u16);
+VFIO_IOWRITE(32, u32);
+VFIO_IOWRITE(64, u64);
+
+#define VFIO_IOREAD(size, type)							\
+static long vfio_pci_access_ioread##size(struct vfio_pci_core_device *vdev,	\
+					       void *data)			\
+{										\
+	struct vfio_pci_io_access *access = data;				\
+	type *val = access->val;						\
+										\
+	if (access->test_mem) {							\
+		down_read(&vdev->memory_lock);					\
+		if (!__vfio_pci_memory_enabled(vdev)) {				\
+			up_read(&vdev->memory_lock);				\
+			return -EIO;						\
+		}								\
+	}									\
+										\
+	*val = vfio_ioread##size(access->io);					\
+	if (access->test_mem)							\
+		up_read(&vdev->memory_lock);					\
+	return 0;								\
+}										\
+										\
+int vfio_pci_core_ioread##size(struct vfio_pci_core_device *vdev,		\
+			 bool test_mem, type *val, void __iomem *io)		\
+{										\
+	struct vfio_pci_io_access access = { test_mem, io, val };		\
+										\
+	return vfio_pci_core_access(vdev, vfio_pci_access_ioread##size, &access);	\
+}										\
+EXPORT_SYMBOL_GPL(vfio_pci_core_ioread##size)
+
+VFIO_IOREAD(8, u8);
+VFIO_IOREAD(16, u16);
+VFIO_IOREAD(32, u32);
+VFIO_IOREAD(64, u64);
 
 #define VFIO_IORDWR(size)						\
 static int vfio_pci_iordwr##size(struct vfio_pci_core_device *vdev,\
@@ -200,12 +224,20 @@ ssize_t vfio_pci_core_do_io_rw(struct vfio_pci_core_device *vdev, bool test_mem,
 }
 EXPORT_SYMBOL_GPL(vfio_pci_core_do_io_rw);
 
-static ssize_t vfio_pci_rom_read(struct vfio_pci_core_device *vdev,
-				 char __user *buf, size_t count, loff_t pos)
+struct vfio_pci_rom_access {
+	size_t count;
+	loff_t pos;
+	void *data;
+	size_t length;
+};
+
+static long vfio_pci_access_rom(struct vfio_pci_core_device *vdev, void *arg)
 {
+	struct vfio_pci_rom_access *access = arg;
 	struct pci_dev *pdev = vdev->pdev;
 	bool rom_bar = pci_resource_start(pdev, PCI_ROM_RESOURCE);
-	size_t size, length = 0, done = 0;
+	size_t count = access->count, size, length = 0, done = 0;
+	loff_t pos = access->pos;
 	void *data = NULL;
 	void __iomem *io;
 	ssize_t ret;
@@ -272,6 +304,22 @@ static ssize_t vfio_pci_rom_read(struct vfio_pci_core_device *vdev,
 		iounmap(io);
 out_unlock:
 	up_write(&vdev->memory_lock);
+	access->data = data;
+	access->length = length;
+	return ret;
+}
+
+static ssize_t vfio_pci_rom_read(struct vfio_pci_core_device *vdev,
+				 char __user *buf, size_t count, loff_t pos)
+{
+	struct vfio_pci_rom_access access = { .count = count, .pos = pos };
+	size_t length, done;
+	void *data;
+	ssize_t ret;
+
+	ret = vfio_pci_core_access(vdev, vfio_pci_access_rom, &access);
+	data = access.data;
+	length = access.length;
 	if (ret < 0)
 		goto out_free;
 
Signature · 1 line
2.43.0
#13 lore raw

Re: [RFC PATCH v3 06/20] vfio/pci: Use access ops for BAR and ROM accesses

Sashiko has reviewed this patch and found no issues.
Quoted 9 lines

Recovery needs to wait for hardware reads and writes to finish before changing device state. Use access ops for individual MMIO and PIO reads and writes, covering BAR, VGA, port-I/O and ioeventfd accesses. Keep copies to and from userspace outside SRCU.

For ROM reads, keep mapping, reading into the buffer and unmapping within one SRCU read-side section. Copy the buffered data to userspace afterward.

Sashiko has reviewed this patch and found no issues.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=6
#14 lore raw

RFC PATCH v3 7/20 vfio/pci: Use access ops when mapping BAR pages

In reply to Shameer Kolothum

A BAR page fault can race with recovery and install a mapping after recovery has removed the existing mappings.

Insert the mapping through the SRCU-protected access ops while holding memory_lock. Recovery can then wait for pending insertions before removing mappings. If access is blocked, return VM_FAULT_SIGBUS instead of waiting for recovery in the fault handler.

---
 drivers/vfio/pci/vfio_pci_core.c | 26 ++++++++++++++++++++++++--
 1 file changed, 24 insertions(+), 2 deletions(-)
drivers/vfio/pci/vfio_pci_core.c +24 -2
index 21c0dda37dd1..b19721617e5f 100644
@@ -1880,6 +1880,24 @@ vm_fault_t vfio_pci_vmf_insert_pfn(struct vfio_pci_core_device *vdev,
 }
 EXPORT_SYMBOL_GPL(vfio_pci_vmf_insert_pfn);
 
+struct vfio_pci_fault_access {
+	struct vm_fault *vmf;
+	unsigned long pfn;
+	unsigned int order;
+};
+
+static long vfio_pci_access_fault(struct vfio_pci_core_device *vdev, void *data)
+{
+	struct vfio_pci_fault_access *access = data;
+	vm_fault_t ret;
+
+	down_read(&vdev->memory_lock);
+	ret = vfio_pci_vmf_insert_pfn(vdev, access->vmf, access->pfn,
+				      access->order);
+	up_read(&vdev->memory_lock);
+	return (__force unsigned int)ret;
+}
+
 static vm_fault_t vfio_pci_mmap_huge_fault(struct vm_fault *vmf,
 					   unsigned int order)
 {
@@ -1891,8 +1909,12 @@ static vm_fault_t vfio_pci_mmap_huge_fault(struct vm_fault *vmf,
 	vm_fault_t ret = VM_FAULT_FALLBACK;
 
 	if (is_aligned_for_order(vma, addr, pfn, order)) {
-		scoped_guard(rwsem_read, &vdev->memory_lock)
-			ret = vfio_pci_vmf_insert_pfn(vdev, vmf, pfn, order);
+		struct vfio_pci_fault_access access = { vmf, pfn, order };
+		long result;
+
+		/* Recovery must wait for this insertion before removing mappings. */
+		result = vfio_pci_core_access(vdev, vfio_pci_access_fault, &access);
+		ret = result < 0 ? VM_FAULT_SIGBUS : (__force vm_fault_t)result;
 	}
 
 	dev_dbg_ratelimited(&vdev->pdev->dev,
Signature · 1 line
2.43.0
#15 lore raw

Re: [RFC PATCH v3 07/20] vfio/pci: Use access ops when mapping BAR pages

Sashiko has reviewed this patch and found no issues.
Quoted 9 lines

A BAR page fault can race with recovery and install a mapping after recovery has removed the existing mappings.

Insert the mapping through the SRCU-protected access ops while holding memory_lock. Recovery can then wait for pending insertions before removing mappings. If access is blocked, return VM_FAULT_SIGBUS instead of waiting for recovery in the fault handler.

Sashiko has reviewed this patch and found no issues.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=7
#16 lore raw

RFC PATCH v3 8/20 vfio/pci: Use access ops for hardware IRQ operations

In reply to Shameer Kolothum

IRQ queries and configuration can access PCI registers while recovery resets the device or restores its configuration.
Use access ops when reading hardware IRQ counts and applying IRQ configuration. Continue to hold igate for configuration changes.

Use separate SRCU read-side sections for the count query and the configuration change so copies to and from userspace stay outside SRCU. ERR and REQ operations do not touch hardware and remain available while access is blocked.

IRQ teardown flushes the global virqfd workqueue. Waiting for an IRQ operation to finish therefore also waits for that flush, including work for other devices.

---
 drivers/vfio/pci/vfio_pci_core.c | 78 ++++++++++++++++++++++++++++----
 1 file changed, 70 insertions(+), 8 deletions(-)
drivers/vfio/pci/vfio_pci_core.c +70 -8
index b19721617e5f..de3a1c8b6d9e 100644
@@ -1342,11 +1342,33 @@ int vfio_pci_ioctl_get_region_info(struct vfio_device *core_vdev,
 }
 EXPORT_SYMBOL_GPL(vfio_pci_ioctl_get_region_info);
 
+/* ERR and REQ do not access hardware. */
+static bool vfio_pci_irq_index_is_device(u32 index)
+{
+	switch (index) {
+	case VFIO_PCI_INTX_IRQ_INDEX:
+	case VFIO_PCI_MSI_IRQ_INDEX:
+	case VFIO_PCI_MSIX_IRQ_INDEX:
+		return true;
+	default:
+		return false;
+	}
+}
+
+static long vfio_pci_access_irq_count(struct vfio_pci_core_device *vdev,
+				      void *data)
+{
+	u32 *index = data;
+
+	return vfio_pci_get_irq_count(vdev, *index);
+}
+
 static int vfio_pci_ioctl_get_irq_info(struct vfio_pci_core_device *vdev,
 				       struct vfio_irq_info __user *arg)
 {
 	unsigned long minsz = offsetofend(struct vfio_irq_info, count);
 	struct vfio_irq_info info;
+	int count;
 
 	if (copy_from_user(&info, arg, minsz))
 		return -EFAULT;
@@ -1365,7 +1387,15 @@ static int vfio_pci_ioctl_get_irq_info(struct vfio_pci_core_device *vdev,
 
 	info.flags = VFIO_IRQ_INFO_EVENTFD;
 
-	info.count = vfio_pci_get_irq_count(vdev, info.index);
+	if (vfio_pci_irq_index_is_device(info.index)) {
+		count = vfio_pci_core_access(vdev, vfio_pci_access_irq_count,
+					     &info.index);
+		if (count < 0)
+			return count;
+		info.count = count;
+	} else {
+		info.count = vfio_pci_get_irq_count(vdev, info.index);
+	}
 
 	if (info.index == VFIO_PCI_INTX_IRQ_INDEX)
 		info.flags |=
@@ -1376,19 +1406,48 @@ static int vfio_pci_ioctl_get_irq_info(struct vfio_pci_core_device *vdev,
 	return copy_to_user(arg, &info, minsz) ? -EFAULT : 0;
 }
 
+struct vfio_pci_irq_access {
+	struct vfio_irq_set *hdr;
+	u8 *data;
+};
+
+static long vfio_pci_access_irqs(struct vfio_pci_core_device *vdev, void *data)
+{
+	struct vfio_pci_irq_access *access = data;
+	struct vfio_irq_set *hdr = access->hdr;
+	int ret;
+
+	mutex_lock(&vdev->igate);
+	ret = vfio_pci_set_irqs_ioctl(vdev, hdr->flags, hdr->index, hdr->start,
+				      hdr->count, access->data);
+	mutex_unlock(&vdev->igate);
+	return ret;
+}
+
 static int vfio_pci_ioctl_set_irqs(struct vfio_pci_core_device *vdev,
 				   struct vfio_irq_set __user *arg)
 {
 	unsigned long minsz = offsetofend(struct vfio_irq_set, count);
 	struct vfio_irq_set hdr;
+	struct vfio_pci_irq_access access = { .hdr = &hdr };
 	u8 *data = NULL;
+	bool device_irq;
 	int max, ret = 0;
 	size_t data_size = 0;
 
 	if (copy_from_user(&hdr, arg, minsz))
 		return -EFAULT;
 
-	max = vfio_pci_get_irq_count(vdev, hdr.index);
+	/* Keep the userspace copy between the two SRCU sections. */
+	device_irq = vfio_pci_irq_index_is_device(hdr.index);
+	if (device_irq) {
+		max = vfio_pci_core_access(vdev, vfio_pci_access_irq_count,
+					   &hdr.index);
+		if (max < 0)
+			return max;
+	} else {
+		max = vfio_pci_get_irq_count(vdev, hdr.index);
+	}
 
 	ret = vfio_set_irqs_validate_and_prepare(&hdr, max, VFIO_PCI_NUM_IRQS,
 						 &data_size);
@@ -1401,12 +1460,15 @@ static int vfio_pci_ioctl_set_irqs(struct vfio_pci_core_device *vdev,
 			return PTR_ERR(data);
 	}
 
-	mutex_lock(&vdev->igate);
-
-	ret = vfio_pci_set_irqs_ioctl(vdev, hdr.flags, hdr.index, hdr.start,
-				      hdr.count, data);
-
-	mutex_unlock(&vdev->igate);
+	/*
+	 * IRQ teardown flushes the global virqfd workqueue, so draining SRCU
+	 * can also wait for work on other devices.
+	 */
+	access.data = data;
+	if (device_irq)
+		ret = vfio_pci_core_access(vdev, vfio_pci_access_irqs, &access);
+	else
+		ret = vfio_pci_access_irqs(vdev, &access);
 	kfree(data);
 
 	return ret;
Signature · 1 line
2.43.0
#17 lore raw

Re: [RFC PATCH v3 08/20] vfio/pci: Use access ops for hardware IRQ operations

Sashiko has reviewed this patch and found no issues.
Quoted 10 lines

IRQ queries and configuration can access PCI registers while recovery resets the device or restores its configuration.
Use access ops when reading hardware IRQ counts and applying IRQ configuration. Continue to hold igate for configuration changes.

Use separate SRCU read-side sections for the count query and the configuration change so copies to and from userspace stay outside SRCU. ERR and REQ operations do not touch hardware and remain available while access is blocked.

Quoted 5 lines

IRQ teardown flushes the global virqfd workqueue. Waiting for an IRQ operation to finish therefore also waits for that flush, including work for other devices.

Sashiko has reviewed this patch and found no issues.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=8
#18 lore raw

RFC PATCH v3 9/20 vfio/pci: Use access ops for function reset

In reply to Shameer Kolothum

Userspace function reset and its D0 preparation must finish before host recovery changes device state.

Run both through the SRCU-protected access ops. Reject blocked requests and let recovery drain admitted operations. The D0 helper returns -EAGAIN on bus-lock contention.

Hold memory_lock across preparation and reset so concurrent PM writes cannot put the device back into D3. Keep the existing DMA-BUF mapping removal and restoration around reset.

---
 drivers/vfio/pci/vfio_pci_core.c | 29 +++++++++++++++++------------
 1 file changed, 17 insertions(+), 12 deletions(-)
drivers/vfio/pci/vfio_pci_core.c +17 -12
index de3a1c8b6d9e..fb7f46c2ca36 100644
@@ -1474,16 +1474,11 @@ static int vfio_pci_ioctl_set_irqs(struct vfio_pci_core_device *vdev,
 	return ret;
 }
 
-static int vfio_pci_ioctl_reset(struct vfio_pci_core_device *vdev,
-				void __user *arg)
+static long vfio_pci_access_reset(struct vfio_pci_core_device *vdev, void *data)
 {
 	int ret;
 
-	if (!vdev->reset_works)
-		return -EINVAL;
-
-	vfio_pci_zap_and_down_write_memory_lock(vdev);
-
+	lockdep_assert_held_write(&vdev->memory_lock);
 	/*
 	 * This function can be invoked while the power state is non-D0. If
 	 * pci_try_reset_function() has been called while the power state is
@@ -1494,17 +1489,27 @@ static int vfio_pci_ioctl_reset(struct vfio_pci_core_device *vdev,
 	 * 'vdev->pm_save').
 	 */
 	ret = vfio_pci_try_set_power_state(vdev, PCI_D0);
-	if (ret) {
-		up_write(&vdev->memory_lock);
+	if (ret)
 		return ret;
-	}
-
 	vfio_pci_dma_buf_move(vdev, true);
 	ret = pci_try_reset_function(vdev->pdev);
 	if (__vfio_pci_memory_enabled(vdev))
 		vfio_pci_dma_buf_move(vdev, false);
-	up_write(&vdev->memory_lock);
+	return ret;
+}
+
+static int vfio_pci_ioctl_reset(struct vfio_pci_core_device *vdev,
+				void __user *arg)
+{
+	int ret;
+
+	if (!vdev->reset_works)
+		return -EINVAL;
 
+	vfio_pci_zap_and_down_write_memory_lock(vdev);
+	/* Keep D0 preparation and reset together under memory_lock and SRCU. */
+	ret = vfio_pci_core_access(vdev, vfio_pci_access_reset, NULL);
+	up_write(&vdev->memory_lock);
 	return ret;
 }
 
Signature · 1 line
2.43.0
#19 lore raw

Re: [RFC PATCH v3 09/20] vfio/pci: Use access ops for function reset

Sashiko has reviewed this patch and found no issues.
Quoted 7 lines

Userspace function reset and its D0 preparation must finish before host recovery changes device state.

Run both through the SRCU-protected access ops. Reject blocked requests and let recovery drain admitted operations. The D0 helper returns -EAGAIN on bus-lock contention.

Quoted 5 lines

Hold memory_lock across preparation and reset so concurrent PM writes cannot put the device back into D3. Keep the existing DMA-BUF mapping removal and restoration around reset.

Sashiko has reviewed this patch and found no issues.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=9
#20 lore raw

RFC PATCH v3 10/20 vfio/pci: Use access ops for runtime-PM requests

In reply to Shameer Kolothum

Userspace low-power requests change PM references and device mappings. Run these requests through access ops so recovery can reject new requests and drain admitted operations.

Validate userspace arguments and obtain the eventfd reference before entering SRCU. Release the reference if entry fails. Keep the existing memory_lock protection and PM reference handling.

---
 drivers/vfio/pci/vfio_pci_core.c | 20 ++++++++++++++------
 1 file changed, 14 insertions(+), 6 deletions(-)
drivers/vfio/pci/vfio_pci_core.c +14 -6
index fb7f46c2ca36..fc6b2673be88 100644
@@ -454,9 +454,11 @@ int vfio_pci_try_set_power_state(struct vfio_pci_core_device *vdev,
 					pci_set_power_state);
 }
 
-static int vfio_pci_runtime_pm_entry(struct vfio_pci_core_device *vdev,
-				     struct eventfd_ctx *efdctx)
+static long vfio_pci_runtime_pm_entry(struct vfio_pci_core_device *vdev,
+				      void *data)
 {
+	struct eventfd_ctx *efdctx = data;
+
 	/*
 	 * The vdev power related flags are protected with 'memory_lock'
 	 * semaphore.
@@ -492,7 +494,7 @@ static int vfio_pci_core_pm_entry(struct vfio_pci_core_device *vdev, u32 flags,
 	 * while returning from the ioctl and then the device can go into
 	 * runtime suspended state.
 	 */
-	return vfio_pci_runtime_pm_entry(vdev, NULL);
+	return vfio_pci_core_access(vdev, vfio_pci_runtime_pm_entry, NULL);
 }
 
 static int vfio_pci_core_pm_entry_with_wakeup(
@@ -519,7 +521,7 @@ static int vfio_pci_core_pm_entry_with_wakeup(
 	if (IS_ERR(efdctx))
 		return PTR_ERR(efdctx);
 
-	ret = vfio_pci_runtime_pm_entry(vdev, efdctx);
+	ret = vfio_pci_core_access(vdev, vfio_pci_runtime_pm_entry, efdctx);
 	if (ret)
 		eventfd_ctx_put(efdctx);
 
@@ -552,6 +554,13 @@ static void vfio_pci_runtime_pm_exit(struct vfio_pci_core_device *vdev)
 	up_write(&vdev->memory_lock);
 }
 
+static long vfio_pci_access_pm_exit(struct vfio_pci_core_device *vdev,
+				    void *data)
+{
+	vfio_pci_runtime_pm_exit(vdev);
+	return 0;
+}
+
 static int vfio_pci_core_pm_exit(struct vfio_pci_core_device *vdev, u32 flags,
 				 void __user *arg, size_t argsz)
 {
@@ -568,8 +577,7 @@ static int vfio_pci_core_pm_exit(struct vfio_pci_core_device *vdev, u32 flags,
 	 * already signaled the eventfd and exited low power mode itself.
 	 * pm_runtime_engaged protects the redundant call here.
 	 */
-	vfio_pci_runtime_pm_exit(vdev);
-	return 0;
+	return vfio_pci_core_access(vdev, vfio_pci_access_pm_exit, NULL);
 }
 
 #ifdef CONFIG_PM
Signature · 1 line
2.43.0
#21 lore raw

Re: [RFC PATCH v3 10/20] vfio/pci: Use access ops for runtime-PM requests

Sashiko has reviewed this patch and found no issues.
Quoted 9 lines

Userspace low-power requests change PM references and device mappings. Run these requests through access ops so recovery can reject new requests and drain admitted operations.

Validate userspace arguments and obtain the eventfd reference before entering SRCU. Release the reference if entry fails. Keep the existing memory_lock protection and PM reference handling.

Sashiko has reviewed this patch and found no issues.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=10
#22 lore raw

RFC PATCH v3 11/20 vfio/pci: Use access ops for hardware information queries

In reply to Shameer Kolothum

AtomicOps and ROM information queries access PCI configuration and ROM contents. Run these accesses through access ops and keep userspace copies outside SRCU. Free the capability buffer if the query fails.

A VF's AtomicOps query accesses its PF. If the PF is managed by VFIO and supports recovery, check its access ops too. Hold the PF's access_lock and reject queries when it is closed, since its SRCU instance exists only while open. Other PF drivers retain their existing behavior.

---
 drivers/vfio/pci/vfio_pci_core.c | 82 ++++++++++++++++++++++++--------
 1 file changed, 61 insertions(+), 21 deletions(-)
drivers/vfio/pci/vfio_pci_core.c +61 -21
index fc6b2673be88..da7166669dc7 100644
@@ -1178,6 +1178,33 @@ static int vfio_pci_info_atomic_cap(struct vfio_pci_core_device *vdev,
 	return vfio_info_add_capability(caps, &cap.header, sizeof(cap));
 }
 
+static long vfio_pci_access_atomic_cap_target(struct vfio_pci_core_device *vdev,
+					      void *data)
+{
+	return vfio_pci_info_atomic_cap(vdev, data);
+}
+
+static long vfio_pci_access_atomic_cap(struct vfio_pci_core_device *vdev,
+				       void *data)
+{
+	struct vfio_pci_core_device *pf_vdev = vdev->sriov_pf_core_dev;
+
+	if (!pf_vdev || !pf_vdev->pci_recovery_supported)
+		return vfio_pci_info_atomic_cap(vdev, data);
+
+	/*
+	 * VF AtomicOps queries read the PF's configuration. The PF association
+	 * lasts until VF removal, but its SRCU instance exists only while
+	 * the PF is open.
+	 */
+	guard(mutex)(&pf_vdev->access_lock);
+	if (!pf_vdev->device_open)
+		return -EIO;
+
+	return vfio_pci_core_access(pf_vdev, vfio_pci_access_atomic_cap_target,
+				    data);
+}
+
 static int vfio_pci_ioctl_get_info(struct vfio_pci_core_device *vdev,
 				   struct vfio_device_info __user *arg)
 {
@@ -1209,10 +1236,12 @@ static int vfio_pci_ioctl_get_info(struct vfio_pci_core_device *vdev,
 		return ret;
 	}
 
-	ret = vfio_pci_info_atomic_cap(vdev, &caps);
+	ret = vfio_pci_core_access(vdev, vfio_pci_access_atomic_cap, &caps);
 	if (ret && ret != -ENODEV) {
-		pci_warn(vdev->pdev,
-			 "Failed to setup AtomicOps info capability\n");
+		if (ret != -EIO)
+			pci_warn(vdev->pdev,
+				 "Failed to setup AtomicOps info capability\n");
+		kfree(caps.buf);
 		return ret;
 	}
 
@@ -1235,6 +1264,31 @@ static int vfio_pci_ioctl_get_info(struct vfio_pci_core_device *vdev,
 	return copy_to_user(arg, &info, minsz) ? -EFAULT : 0;
 }
 
+static long vfio_pci_access_rom_info(struct vfio_pci_core_device *vdev,
+				     void *data)
+{
+	struct vfio_region_info *info = data;
+	struct pci_dev *pdev = vdev->pdev;
+	void __iomem *io;
+	size_t size;
+	u16 cmd;
+
+	/*
+	 * Check ROM content is valid. Need to enable memory
+	 * decode for ROM access in pci_map_rom().
+	 */
+	cmd = vfio_pci_memory_lock_and_enable(vdev);
+	io = pci_map_rom(pdev, &size);
+	if (io) {
+		info->flags = VFIO_REGION_INFO_FLAG_READ;
+		/* Report the BAR size, not the ROM size. */
+		info->size = pci_resource_len(pdev, PCI_ROM_RESOURCE);
+		pci_unmap_rom(pdev, io);
+	}
+	vfio_pci_memory_unlock_and_restore(vdev, cmd);
+	return 0;
+}
+
 int vfio_pci_ioctl_get_region_info(struct vfio_device *core_vdev,
 				   struct vfio_region_info *info,
 				   struct vfio_info_cap *caps)
@@ -1272,29 +1326,15 @@ int vfio_pci_ioctl_get_region_info(struct vfio_device *core_vdev,
 
 		break;
 	case VFIO_PCI_ROM_REGION_INDEX: {
-		void __iomem *io;
-		size_t size;
-		u16 cmd;
-
 		info->offset = VFIO_PCI_INDEX_TO_OFFSET(info->index);
 		info->flags = 0;
 		info->size = 0;
 
 		if (pci_resource_start(pdev, PCI_ROM_RESOURCE)) {
-			/*
-			 * Check ROM content is valid. Need to enable memory
-			 * decode for ROM access in pci_map_rom().
-			 */
-			cmd = vfio_pci_memory_lock_and_enable(vdev);
-			io = pci_map_rom(pdev, &size);
-			if (io) {
-				info->flags = VFIO_REGION_INFO_FLAG_READ;
-				/* Report the BAR size, not the ROM size. */
-				info->size = pci_resource_len(pdev,
-							      PCI_ROM_RESOURCE);
-				pci_unmap_rom(pdev, io);
-			}
-			vfio_pci_memory_unlock_and_restore(vdev, cmd);
+			ret = vfio_pci_core_access(vdev, vfio_pci_access_rom_info,
+						   info);
+			if (ret)
+				return ret;
 		} else if (pdev->rom && pdev->romlen) {
 			info->flags = VFIO_REGION_INFO_FLAG_READ;
 			/* Report BAR size as power of two. */
Signature · 1 line
2.43.0
#23 lore raw

Re: [RFC PATCH v3 11/20] vfio/pci: Use access ops for hardware information queries

Thank you for your contribution! Sashiko AI review found 2 potential issues to consider: - [High] vfio/pci: VFIO_DEVICE_GET_INFO fails with -EIO on VF when PF is closed - [Medium] vfio/pci: lockdep…

Thank you for your contribution! Sashiko AI review found 2 potential issues to consider:
- [High] vfio/pci: VFIO_DEVICE_GET_INFO fails with -EIO on VF when PF is closed
- [Medium] vfio/pci: lockdep circular locking dependency in vfio_pci_access_atomic_cap()
--

commit 8a20ca42b1e7365162ac6cecdd060d990d426a4f
Author: Shameer Kolothum <[email protected]>

vfio/pci: Use access ops for hardware information queries

This commit routes AtomicOps and ROM region hardware information queries through VFIO PCI access ops so they participate in device error recovery.

Quoted 18 lines
drivers/vfio/pci/vfio_pci_core.c +10 -0
index fc6b2673be88c..da7166669dc75 100644
@@ -1178,6 +1178,33 @@ static int vfio_pci_info_atomic_cap(struct vfio_pci_core_device *vdev,
 	return vfio_info_add_capability(caps, &cap.header, sizeof(cap));
 }
 
+static long vfio_pci_access_atomic_cap_target(struct vfio_pci_core_device *vdev,
+					      void *data)
+{
+	return vfio_pci_info_atomic_cap(vdev, data);
+}
+
+static long vfio_pci_access_atomic_cap(struct vfio_pci_core_device *vdev,
+				       void *data)
+{
+	struct vfio_pci_core_device *pf_vdev = vdev->sriov_pf_core_dev;

drivers/vfio/pci/vfio_pci_core.c static int vfio_pci_info_atomic_cap(struct vfio_pci_core_device *vdev, · line 1200

Quoted 10 lines
drivers/vfio/pci/vfio_pci_core.c +10 -0
+
+	if (!pf_vdev || !pf_vdev->pci_recovery_supported)
+		return vfio_pci_info_atomic_cap(vdev, data);
+
+	/*
+	 * VF AtomicOps queries read the PF's configuration. The PF association
+	 * lasts until VF removal, but its SRCU instance exists only while
+	 * the PF is open.
+	 */
+	guard(mutex)(&pf_vdev->access_lock);

[Severity: Medium]
Can locking pf_vdev->access_lock here trigger a lockdep circular locking warning?

vfio_pci_ioctl_get_info() calls vfio_pci_core_access() on the VF, which takes srcu_read_lock(&vdev->access_srcu) before entering vfio_pci_access_atomic_cap() and acquiring pf_vdev->access_lock. Because every struct vfio_pci_core_device initializes access_lock and access_srcu at the same mutex_init() and init_srcu_struct() call sites, the VF and PF share lockdep classes.

Meanwhile, vfio_pci_core_close_device() holds vdev->access_lock while vfio_pci_core_block_access() calls synchronize_srcu(&vdev->access_srcu), establishing the opposite access_lock -> access_srcu class order:

CPU0 CPU1
---- ----
lock(&vdev->access_lock)

                                  srcu_read_lock(&vdev->access_srcu)
                                  lock(&pf_vdev->access_lock)

synchronize_srcu(&vdev->access_srcu)

*** DEADLOCK ***

CPU0: vfio_pci_core_close_device() -> vfio_pci_core_block_access()
CPU1: vfio_pci_ioctl_get_info() -> vfio_pci_core_access() ->
vfio_pci_access_atomic_cap()

Quoted 15 lines
drivers/vfio/pci/vfio_pci_core.c +7 -1
+	if (!pf_vdev->device_open)
+		return -EIO;
+
+	return vfio_pci_core_access(pf_vdev, vfio_pci_access_atomic_cap_target,
+				    data);
+}
+
 static int vfio_pci_ioctl_get_info(struct vfio_pci_core_device *vdev,
 				   struct vfio_device_info __user *arg)
 {
@@ -1209,10 +1236,12 @@ static int vfio_pci_ioctl_get_info(struct vfio_pci_core_device *vdev,
 		return ret;
 	}
 
-	ret = vfio_pci_info_atomic_cap(vdev, &caps);

drivers/vfio/pci/vfio_pci_core.c static int vfio_pci_ioctl_get_info(struct vfio_pci_core_device *vdev, · line 1246

Quoted 10 lines
drivers/vfio/pci/vfio_pci_core.c +5 -2
+	ret = vfio_pci_core_access(vdev, vfio_pci_access_atomic_cap, &caps);
 	if (ret && ret != -ENODEV) {
-		pci_warn(vdev->pdev,
-			 "Failed to setup AtomicOps info capability\n");
+		if (ret != -EIO)
+			pci_warn(vdev->pdev,
+				 "Failed to setup AtomicOps info capability\n");
+		kfree(caps.buf);
 		return ret;
 	}

[Severity: High]
Does this cause VFIO_DEVICE_GET_INFO on a VF to fail with -EIO whenever its recovery-capable vfio-pci PF is not currently open in userspace?

A PF bound to vfio-pci does not need to stay open while its VFs are in use; for example, userspace can open the PF only long enough to configure vf_token and then close the PF fd before opening VFs, while vfio_pci_core_sriov_configure() keeps the PF in PCI_D0. When pf_vdev->device_open is false, vfio_pci_access_atomic_cap() returns -EIO and vfio_pci_ioctl_get_info() aborts the entire ioctl, breaking VFIO_DEVICE_GET_INFO on healthy VFs even if AtomicOps are not supported.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=11
#24 lore raw

RFC PATCH v3 12/20 vfio/pci: Use access ops for DMA-BUF export and restoration

In reply to Shameer Kolothum

DMA-BUF export can race with recovery revoking device mappings.

Keep the range lookup, export and list insertion inside SRCU so recovery waits for admitted exports before revoking mappings. Keep userspace copies and file descriptor installation outside SRCU.

Select ordinary mapping restoration through access ops under memory_lock. Skip restoration while access is blocked. Provide an internal helper so recovery can restore mappings before allowing access again. Revocation remains available while access is blocked.

---
 drivers/vfio/pci/vfio_pci_priv.h   |   8 ++
 drivers/vfio/pci/vfio_pci_core.c   |   6 ++
 drivers/vfio/pci/vfio_pci_dmabuf.c | 113 +++++++++++++++++++----------
 3 files changed, 89 insertions(+), 38 deletions(-)
drivers/vfio/pci/vfio_pci_priv.h +8 -0
index 0ea7c710dc38..b02fd1fbcd31 100644
@@ -90,6 +90,8 @@ struct vfio_pci_access_ops {
 			vfio_pci_access_fn fn, void *data);
 	/* Caller holds memory_lock. */
 	int (*set_power_state)(struct vfio_pci_core_device *vdev, pci_power_t state);
+	/* Caller holds memory_lock. */
+	void (*restore_dmabufs)(struct vfio_pci_core_device *vdev);
 };
 
 const struct vfio_pci_access_ops *
@@ -151,6 +153,7 @@ int vfio_pci_core_feature_dma_buf(struct vfio_pci_core_device *vdev, u32 flags,
 				  size_t argsz);
 void vfio_pci_dma_buf_cleanup(struct vfio_pci_core_device *vdev);
 void vfio_pci_dma_buf_move(struct vfio_pci_core_device *vdev, bool revoked);
+void vfio_pci_dma_buf_restore(struct vfio_pci_core_device *vdev);
 #else
 static inline int
 vfio_pci_core_feature_dma_buf(struct vfio_pci_core_device *vdev, u32 flags,
@@ -162,6 +165,11 @@ vfio_pci_core_feature_dma_buf(struct vfio_pci_core_device *vdev, u32 flags,
 static inline void vfio_pci_dma_buf_cleanup(struct vfio_pci_core_device *vdev)
 {
 }
+
+static inline void vfio_pci_dma_buf_restore(struct vfio_pci_core_device *vdev)
+{
+}
+
 static inline void vfio_pci_dma_buf_move(struct vfio_pci_core_device *vdev,
 					 bool revoked)
 {
drivers/vfio/pci/vfio_pci_core.c +6 -0
index da7166669dc7..0299f5afca91 100644
@@ -330,14 +330,20 @@ static int vfio_pci_power_blocked(struct vfio_pci_core_device *vdev,
 	return -EIO;
 }
 
+static void vfio_pci_restore_dmabufs_blocked(struct vfio_pci_core_device *vdev)
+{
+}
+
 static const struct vfio_pci_access_ops vfio_pci_live_ops = {
 	.execute = vfio_pci_access_live,
 	.set_power_state = vfio_pci_try_set_power_state,
+	.restore_dmabufs = vfio_pci_dma_buf_restore,
 };
 
 static const struct vfio_pci_access_ops vfio_pci_blocked_ops = {
 	.execute = vfio_pci_access_blocked,
 	.set_power_state = vfio_pci_power_blocked,
+	.restore_dmabufs = vfio_pci_restore_dmabufs_blocked,
 };
 
 /*
drivers/vfio/pci/vfio_pci_dmabuf.c +75 -38

150 lines not on this page · raw message

Signature · 1 line
2.43.0
#25 lore raw

Re: [RFC PATCH v3 12/20] vfio/pci: Use access ops for DMA-BUF export and restoration

Sashiko has reviewed this patch and found no issues.
Quoted 6 lines

DMA-BUF export can race with recovery revoking device mappings.

Keep the range lookup, export and list insertion inside SRCU so recovery waits for admitted exports before revoking mappings. Keep userspace copies and file descriptor installation outside SRCU.

Quoted 6 lines

Select ordinary mapping restoration through access ops under memory_lock. Skip restoration while access is blocked. Provide an internal helper so recovery can restore mappings before allowing access again. Revocation remains available while access is blocked.

Sashiko has reviewed this patch and found no issues.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=12
#26 lore raw

RFC PATCH v3 13/20 vfio/pci: Track PCI recovery state

In reply to Shameer Kolothum

PCI error recovery spans several callbacks. Add state to track its progress so later patches can keep access blocked until recovery succeeds and leave it blocked if recovery fails.

Add recovery flags, a sequence counter and saved PCI_COMMAND state. Initialize per-open state during open and clear it on close. Keep host recovery state separate so close does not lose an ongoing recovery that must prevent reopening.

---
 include/linux/vfio_pci_core.h    | 18 +++++++++++++++++-
 drivers/vfio/pci/vfio_pci_core.c |  7 +++++++
 2 files changed, 24 insertions(+), 1 deletion(-)
include/linux/vfio_pci_core.h +17 -1
index 88aa7b752231..f2b64420cb07 100644
@@ -98,6 +98,11 @@ static inline int vfio_pci_core_get_dmabuf_phys(
 }
 #endif
 
+#define VFIO_PCI_RECOVERY_IN_PROGRESS	BIT(0)
+#define VFIO_PCI_RECOVERY_FROZEN	BIT(1)
+#define VFIO_PCI_RECOVERY_RESET		BIT(2)
+#define VFIO_PCI_RECOVERY_FAILED	BIT(3)
+
 struct vfio_pci_access_ops;
 
 struct vfio_pci_core_device {
@@ -159,7 +164,18 @@ struct vfio_pci_core_device {
 	const struct vfio_pci_access_ops __rcu *access_ops;
 	/* Set after open completes, cleared before close tears down state. */
 	bool			device_open;
-	struct mutex		access_lock;	/* protects changes to access ops and flags */
+	struct mutex		access_lock;	/* protects access and recovery state */
+	u32			pci_recovery_flags;
+	u64			pci_recovery_sequence;
+	/* Saved PCI_COMMAND, valid when pci_recovery_command_valid is set. */
+	u16			pci_recovery_command;
+	bool			pci_recovery_enabled;
+	bool			pci_recovery_command_valid;
+	/*
+	 * Host recovery is still running. Protected by access_lock and kept across
+	 * close, unlike recovery status for the current open.
+	 */
+	bool			pci_recovery_host_active;
 	struct list_head	dmabufs;
 };
 
drivers/vfio/pci/vfio_pci_core.c +7 -0
index 0299f5afca91..cdc6bafe9861 100644
@@ -923,8 +923,11 @@ void vfio_pci_core_close_device(struct vfio_device *core_vdev)
 
 	if (vdev->pci_recovery_supported) {
 		scoped_guard(mutex, &vdev->access_lock) {
+			WRITE_ONCE(vdev->pci_recovery_enabled, false);
+			vdev->pci_recovery_command_valid = false;
 			WRITE_ONCE(vdev->device_open, false);
 			vfio_pci_core_block_access(vdev);
+			WRITE_ONCE(vdev->pci_recovery_flags, 0);
 		}
 	}
 
@@ -963,6 +966,10 @@ void vfio_pci_core_finish_enable(struct vfio_pci_core_device *vdev)
 
 	if (vdev->pci_recovery_supported) {
 		guard(mutex)(&vdev->access_lock);
+		WRITE_ONCE(vdev->pci_recovery_flags, 0);
+		vdev->pci_recovery_sequence = 0;
+		vdev->pci_recovery_command_valid = false;
+		WRITE_ONCE(vdev->pci_recovery_enabled, false);
 		WRITE_ONCE(vdev->device_open, true);
 		rcu_assign_pointer(vdev->access_ops, &vfio_pci_live_ops);
 	}
Signature · 1 line
2.43.0
#27 lore raw

Re: [RFC PATCH v3 13/20] vfio/pci: Track PCI recovery state

Sashiko has reviewed this patch and found no issues.
Quoted 10 lines

PCI error recovery spans several callbacks. Add state to track its progress so later patches can keep access blocked until recovery succeeds and leave it blocked if recovery fails.

Add recovery flags, a sequence counter and saved PCI_COMMAND state. Initialize per-open state during open and clear it on close. Keep host recovery state separate so close does not lose an ongoing recovery that must prevent reopening.

Sashiko has reviewed this patch and found no issues.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=13
#28 lore raw

RFC PATCH v3 14/20 vfio/pci: Add INTx recovery operations

In reply to Shameer Kolothum

Physical INTx interrupts and eventfd unmask requests can arrive while device access is blocked for recovery. Add blocked operations that remember unmask requests and keep masking pending interrupts on shared lines. An explicit mask cancels a pending unmask request.

Select and call INTx operations under irqlock, and hold the same lock when switching access ops. Process pending unmask requests and publish live ops under irqlock so requests cannot be lost between these steps. Keep closed and failed devices blocked.

Provide helpers to mask INTx at recovery entry and preserve its mask when restoring PCI_COMMAND. The completion caller holds igate and memory_lock through restoration, then sends any resulting INTx notification after releasing memory_lock and irqlock.

---
 drivers/vfio/pci/vfio_pci_priv.h  |  22 ++++
 include/linux/vfio_pci_core.h     |   2 +-
 drivers/vfio/pci/vfio_pci_core.c  |  23 ++++
 drivers/vfio/pci/vfio_pci_intrs.c | 183 ++++++++++++++++++++++++++----
 4 files changed, 209 insertions(+), 21 deletions(-)
drivers/vfio/pci/vfio_pci_priv.h +22 -0
index b02fd1fbcd31..c195ed3e4d9e 100644
@@ -3,6 +3,7 @@
 #define VFIO_PCI_PRIV_H
 
 #include <linux/vfio_pci_core.h>
+#include <linux/irqreturn.h>
 
 /* Special capability IDs predefined access */
 #define PCI_CAP_ID_INVALID		0xFF	/* default raw access */
@@ -25,6 +26,11 @@ struct vfio_pci_ioeventfd {
 
 bool vfio_pci_intx_mask(struct vfio_pci_core_device *vdev);
 void vfio_pci_intx_unmask(struct vfio_pci_core_device *vdev);
+void vfio_pci_intx_recovery_start(struct vfio_pci_core_device *vdev);
+bool vfio_pci_intx_recovery_finish(struct vfio_pci_core_device *vdev);
+void vfio_pci_intx_recovery_signal(struct vfio_pci_core_device *vdev);
+u16 vfio_pci_intx_recovery_update_command(struct vfio_pci_core_device *vdev,
+					  u16 command);
 
 int vfio_pci_eventfd_replace_locked(struct vfio_pci_core_device *vdev,
 				    struct vfio_pci_eventfd __rcu **peventfd,
@@ -84,6 +90,8 @@ typedef long (*vfio_pci_access_fn)(struct vfio_pci_core_device *vdev,
 long vfio_pci_core_access(struct vfio_pci_core_device *vdev,
 			  vfio_pci_access_fn fn, void *data);
 
+struct vfio_pci_irq_ctx;
+
 struct vfio_pci_access_ops {
 	/* Select and call within SRCU. The callback receives kernel buffers. */
 	long (*execute)(struct vfio_pci_core_device *vdev,
@@ -92,10 +100,24 @@ struct vfio_pci_access_ops {
 	int (*set_power_state)(struct vfio_pci_core_device *vdev, pci_power_t state);
 	/* Caller holds memory_lock. */
 	void (*restore_dmabufs)(struct vfio_pci_core_device *vdev);
+	/* Select and call both INTx methods while holding irqlock. */
+	irqreturn_t (*intx_handler)(struct vfio_pci_core_device *vdev,
+				    struct vfio_pci_irq_ctx *ctx);
+	int (*intx_unmask)(struct vfio_pci_core_device *vdev,
+			   struct vfio_pci_irq_ctx *ctx);
 };
 
 const struct vfio_pci_access_ops *
 vfio_pci_access_ops(struct vfio_pci_core_device *vdev);
+void vfio_pci_restore_access(struct vfio_pci_core_device *vdev);
+irqreturn_t vfio_pci_intx_handle(struct vfio_pci_core_device *vdev,
+				 struct vfio_pci_irq_ctx *ctx);
+irqreturn_t vfio_pci_intx_handle_blocked(struct vfio_pci_core_device *vdev,
+					 struct vfio_pci_irq_ctx *ctx);
+int vfio_pci_intx_unmask_live(struct vfio_pci_core_device *vdev,
+			      struct vfio_pci_irq_ctx *ctx);
+int vfio_pci_intx_unmask_blocked(struct vfio_pci_core_device *vdev,
+				 struct vfio_pci_irq_ctx *ctx);
 
 #ifdef CONFIG_VFIO_PCI_IGD
 bool vfio_pci_is_intel_display(struct pci_dev *pdev);
include/linux/vfio_pci_core.h +1 -1
index f2b64420cb07..fadbbaa58770 100644
@@ -160,7 +160,7 @@ struct vfio_pci_core_device {
 	struct notifier_block	nb;
 	struct rw_semaphore	memory_lock;
 	struct srcu_struct	access_srcu;
-	/* Updated under access_lock. */
+	/* Updated under access_lock and irqlock. */
 	const struct vfio_pci_access_ops __rcu *access_ops;
 	/* Set after open completes, cleared before close tears down state. */
 	bool			device_open;
drivers/vfio/pci/vfio_pci_core.c +23 -0
index cdc6bafe9861..89cebe2522ba 100644
@@ -338,12 +338,16 @@ static const struct vfio_pci_access_ops vfio_pci_live_ops = {
 	.execute = vfio_pci_access_live,
 	.set_power_state = vfio_pci_try_set_power_state,
 	.restore_dmabufs = vfio_pci_dma_buf_restore,
+	.intx_handler = vfio_pci_intx_handle,
+	.intx_unmask = vfio_pci_intx_unmask_live,
 };
 
 static const struct vfio_pci_access_ops vfio_pci_blocked_ops = {
 	.execute = vfio_pci_access_blocked,
 	.set_power_state = vfio_pci_power_blocked,
 	.restore_dmabufs = vfio_pci_restore_dmabufs_blocked,
+	.intx_handler = vfio_pci_intx_handle_blocked,
+	.intx_unmask = vfio_pci_intx_unmask_blocked,
 };
 
 /*
@@ -362,12 +366,27 @@ vfio_pci_access_ops(struct vfio_pci_core_device *vdev)
 /* Block new operations and drain ordinary accesses before teardown or recovery. */
 static void vfio_pci_core_block_access(struct vfio_pci_core_device *vdev)
 {
+	unsigned long flags;
+
 	lockdep_assert_held(&vdev->access_lock);
+	spin_lock_irqsave(&vdev->irqlock, flags);
 	rcu_assign_pointer(vdev->access_ops, &vfio_pci_blocked_ops);
+	spin_unlock_irqrestore(&vdev->irqlock, flags);
 
 	synchronize_srcu(&vdev->access_srcu);
 }
 
+/* Caller finishes internal restoration before allowing ordinary access. */
+void vfio_pci_restore_access(struct vfio_pci_core_device *vdev)
+{
+	lockdep_assert_held(&vdev->access_lock);
+	lockdep_assert_held(&vdev->irqlock);
+	if (vdev->device_open &&
+	    !(vdev->pci_recovery_flags &
+	      (VFIO_PCI_RECOVERY_IN_PROGRESS | VFIO_PCI_RECOVERY_FAILED)))
+		rcu_assign_pointer(vdev->access_ops, &vfio_pci_live_ops);
+}
+
 long vfio_pci_core_access(struct vfio_pci_core_device *vdev,
 			  vfio_pci_access_fn fn, void *data)
 {
@@ -965,13 +984,17 @@ void vfio_pci_core_finish_enable(struct vfio_pci_core_device *vdev)
 	}
 
 	if (vdev->pci_recovery_supported) {
+		unsigned long flags;
+
 		guard(mutex)(&vdev->access_lock);
 		WRITE_ONCE(vdev->pci_recovery_flags, 0);
 		vdev->pci_recovery_sequence = 0;
 		vdev->pci_recovery_command_valid = false;
 		WRITE_ONCE(vdev->pci_recovery_enabled, false);
 		WRITE_ONCE(vdev->device_open, true);
+		spin_lock_irqsave(&vdev->irqlock, flags);
 		rcu_assign_pointer(vdev->access_ops, &vfio_pci_live_ops);
+		spin_unlock_irqrestore(&vdev->irqlock, flags);
 	}
 }
 EXPORT_SYMBOL_GPL(vfio_pci_core_finish_enable);
drivers/vfio/pci/vfio_pci_intrs.c +163 -20

242 lines not on this page · raw message

Signature · 1 line
2.43.0
#29 lore raw

Re: [RFC PATCH v3 14/20] vfio/pci: Add INTx recovery operations

Thank you for your contribution! Sashiko AI review found 3 potential issues to consider: - [High] vfio/pci: permanent INTx masking in __vfio_pci_intx_mask() on runtime suspend - [High] vfio/pci…

Thank you for your contribution! Sashiko AI review found 3 potential issues to consider:
- [High] vfio/pci: permanent INTx masking in __vfio_pci_intx_mask() on runtime suspend
- [High] vfio/pci: unmasked shared INTx in vfio_pci_intx_mask_for_recovery()
- [Medium] vfio/pci: premature INTx unmask after blocked irq in vfio_intx_handler()
--

commit 832141c440dca237115d9dc7d2fca6d1c8bdf80a
Author: Shameer Kolothum <[email protected]>

vfio/pci: Add INTx recovery operations

This commit adds blocked INTx operations and recovery helpers to mask INTx, record pending unmask requests while access is blocked, and restore INTx state when recovery finishes.

Quoted 14 lines
drivers/vfio/pci/vfio_pci_intrs.c +2 -0
index 64f80f64ff57a..f78c7217017d2 100644
@@ -29,6 +29,8 @@ struct vfio_pci_irq_ctx {
 	struct virqfd			*mask;
 	char				*name;
 	bool				masked;
+	bool				recovery_masked;
+	bool				unmask_pending;
 	struct irq_bypass_producer	producer;
 };
 

drivers/vfio/pci/vfio_pci_intrs.c static bool __vfio_pci_intx_mask(struct vfio_pci_core_device *vdev) · line 137

Quoted 10 lines
drivers/vfio/pci/vfio_pci_intrs.c +4 -0
@@ -126,6 +128,10 @@ static bool __vfio_pci_intx_mask(struct vfio_pci_core_device *vdev)
 	if (WARN_ON_ONCE(!ctx))
 		goto out_unlock;
 
+	/* An explicit mask cancels any unmask request saved during recovery. */
+	ctx->recovery_masked = false;
+	ctx->unmask_pending = false;
+
 	if (!ctx->masked) {
 		/*
 		 * Can't use check_and_mask here because we always want to

[Severity: High]
Can __vfio_pci_intx_mask() leave INTx permanently masked if vfio_pci_core_runtime_suspend() runs during recovery?

Unlike userspace mask paths, vfio_pci_core_runtime_suspend() is not gated by vfio_pci_core_access() and calls:

drivers/vfio/pci/vfio_pci_core.c:vfio_pci_core_runtime_suspend() {
    ...
	vdev->pm_intx_masked = ((vdev->irq_type == VFIO_PCI_INTX_IRQ_INDEX) &&
				vfio_pci_intx_mask(vdev));
    ...
}

When INTx is masked for recovery (ctx->masked is true and ctx->recovery_masked or ctx->unmask_pending is true), a runtime suspend (for example when pm_runtime_put() runs after an ioctl on a device in VFIO_DEVICE_FEATURE_LOW_POWER_ENTRY) clears both ctx->recovery_masked and ctx->unmask_pending here, skips the !ctx->masked block, and returns false. vfio_pci_core_runtime_suspend() then records vdev->pm_intx_masked = false, so neither vfio_pci_core_runtime_resume() nor vfio_pci_intx_recovery_finish() will unmask INTx again.

Quoted 26 lines
drivers/vfio/pci/vfio_pci_intrs.c +13 -6
@@ -220,33 +245,151 @@ void vfio_pci_intx_unmask(struct vfio_pci_core_device *vdev)
 	mutex_unlock(&vdev->igate);
 }
 
-static irqreturn_t vfio_intx_handler(int irq, void *dev_id)
+/*
+ * Recovery always masks INTx and remembers whether to unmask it afterward.
+ * The interrupt handler masks INTx only if this device has a pending
+ * interrupt, since another device may share the line.
+ */
+static bool vfio_pci_intx_mask_for_recovery(struct vfio_pci_core_device *vdev,
+					    struct vfio_pci_irq_ctx *ctx,
+					    bool quiesce)
 {
-	struct vfio_pci_irq_ctx *ctx = dev_id;
-	struct vfio_pci_core_device *vdev = ctx->vdev;
-	unsigned long flags;
-	int ret = IRQ_NONE;
+	lockdep_assert_held(&vdev->irqlock);
 
-	spin_lock_irqsave(&vdev->irqlock, flags);
+	if (ctx->masked)
+		return false;
 
+	if (!vdev->pci_2_3)
+		disable_irq_nosync(vdev->pdev->irq);

drivers/vfio/pci/vfio_pci_intrs.c void vfio_pci_intx_unmask(struct vfio_pci_core_device *vdev) · line 273

Quoted 10 lines
drivers/vfio/pci/vfio_pci_intrs.c +9 -0
+	else if (quiesce)
+		pci_intx(vdev->pdev, 0);
+	else if (!pci_check_and_mask_intx(vdev->pdev))
+		return false;
 
+	ctx->masked = true;
+	if (quiesce)
+		ctx->recovery_masked = true;
+	return true;
+}

[Severity: High]
Does the early return when ctx->masked is true prevent vfio_pci_intx_handle_blocked() from masking shared PCI 2.3 interrupts during recovery?

vfio_pci_intx_recovery_start() calls vfio_pci_intx_mask_for_recovery() with quiesce = true, setting ctx->masked = true. However, on a PCI 2.3 device, PCI_COMMAND_INTX_DISABLE can still be clear in hardware if config space read 0xffff when error_detected ran, or after a host slot/bus reset clears PCI_COMMAND before state restoration. If the device then asserts a shared INTx line while access is blocked, vfio_pci_intx_handle_blocked() calls vfio_pci_intx_mask_for_recovery(vdev, ctx, false), which returns false immediately because ctx->masked is true instead of calling pci_check_and_mask_intx(), returning IRQ_NONE and leaving the shared level interrupt asserted.

Quoted 38 lines
drivers/vfio/pci/vfio_pci_intrs.c +29 -4
+
+irqreturn_t vfio_pci_intx_handle(struct vfio_pci_core_device *vdev,
+				 struct vfio_pci_irq_ctx *ctx)
+{
+	lockdep_assert_held(&vdev->irqlock);
 	if (!vdev->pci_2_3) {
 		disable_irq_nosync(vdev->pdev->irq);
 		ctx->masked = true;
-		ret = IRQ_HANDLED;
-	} else if (!ctx->masked &&  /* may be shared */
-		   pci_check_and_mask_intx(vdev->pdev)) {
+		return IRQ_HANDLED;
+	}
+	if (!ctx->masked && pci_check_and_mask_intx(vdev->pdev)) {
 		ctx->masked = true;
-		ret = IRQ_HANDLED;
+		return IRQ_HANDLED;
 	}
+	return IRQ_NONE;
+}
+
+irqreturn_t vfio_pci_intx_handle_blocked(struct vfio_pci_core_device *vdev,
+					 struct vfio_pci_irq_ctx *ctx)
+{
+	lockdep_assert_held(&vdev->irqlock);
+	/* A shared interrupt line still needs pending INTx to be masked. */
+	if (vfio_pci_intx_mask_for_recovery(vdev, ctx, false) ||
+	    (ctx->masked && !vdev->pci_2_3))
+		return IRQ_HANDLED;
+	return IRQ_NONE;
+}
+
+static irqreturn_t vfio_intx_handler(int irq, void *dev_id)
+{
+	struct vfio_pci_irq_ctx *ctx = dev_id;
+	struct vfio_pci_core_device *vdev = ctx->vdev;
+	unsigned long flags;
+	irqreturn_t ret;

drivers/vfio/pci/vfio_pci_intrs.c void vfio_pci_intx_unmask(struct vfio_pci_core_device *vdev) · line 316

Quoted 10 lines
drivers/vfio/pci/vfio_pci_intrs.c +2 -1
 
+	spin_lock_irqsave(&vdev->irqlock, flags);
+	ret = vfio_pci_access_ops(vdev)->intx_handler(vdev, ctx);
 	spin_unlock_irqrestore(&vdev->irqlock, flags);
 
 	if (ret == IRQ_HANDLED)
 		vfio_send_intx_eventfd(vdev, ctx);
-
 	return ret;
 }

[Severity: Medium]
When vfio_pci_intx_handle_blocked() returns IRQ_HANDLED, vfio_intx_handler() delivers the automasked interrupt to userspace via vfio_send_intx_eventfd(), but neither vfio_pci_intx_mask_for_recovery() nor vfio_pci_intx_handle_blocked() clears ctx->recovery_masked or ctx->unmask_pending. Can this cause vfio_pci_intx_recovery_finish() to unmask INTx prematurely before userspace has processed and unmasked the interrupt?

CPU0 CPU1 (<hardirq>)
vfio_pci_intx_recovery_start()
spin_lock_irqsave(&vdev->irqlock)
vfio_pci_intx_mask_for_recovery(..., true)

    disable_irq_nosync(pdev->irq)
    ctx->masked = true
    ctx->recovery_masked = true
  spin_unlock_irqrestore(&vdev->irqlock)
                                          vfio_intx_handler()
                                            spin_lock_irqsave(&vdev->irqlock)
                                            vfio_pci_intx_handle_blocked()
                                              ctx->masked && !vdev->pci_2_3
                                              --> returns IRQ_HANDLED
                                            spin_unlock_irqrestore(...)
                                            vfio_send_intx_eventfd()

...
vfio_pci_intx_recovery_finish()
if (ctx->recovery_masked || ...)
vfio_pci_intx_unmask_live() <-- unmasks before userspace

On !vdev->pci_2_3 devices, if ctx->masked was already true before recovery started, the (ctx->masked && !vdev->pci_2_3) check also returns IRQ_HANDLED and triggers a spurious eventfd signal while masked. The same premature unmask happens if vfio_pci_intx_unmask_blocked() sets ctx->unmask_pending = true while ctx->masked is still false (before vfio_pci_intx_recovery_start() runs) and an interrupt is then masked and signaled by vfio_pci_intx_handle_blocked() without clearing ctx->unmask_pending.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=14
#30 lore raw

RFC PATCH v3 15/20 vfio/pci: Use access ops for SR-IOV configuration and shared reset

In reply to Shameer Kolothum

SR-IOV configuration and shared hot reset can change device state during recovery. Both can involve unopened devices, whose per-open SRCU instance is unavailable.

Select SR-IOV configuration through access ops under the PCI device lock. Reject requests for open devices while access is blocked, but preserve configuration of unopened devices. Unregister uses the internal helper to remove VFs regardless of the selected ops.

Select hot-reset preparation under each device's memory_lock. Reject open recovering devices, but allow unopened and failed devices so they do not prevent resetting healthy siblings. Unwind the acquired locks if preparation fails.

---
 drivers/vfio/pci/vfio_pci_priv.h |  4 +++
 drivers/vfio/pci/vfio_pci_core.c | 55 +++++++++++++++++++++++++++++---
 2 files changed, 54 insertions(+), 5 deletions(-)
drivers/vfio/pci/vfio_pci_priv.h +4 -0
index c195ed3e4d9e..e4f3e4f51180 100644
@@ -100,6 +100,10 @@ struct vfio_pci_access_ops {
 	int (*set_power_state)(struct vfio_pci_core_device *vdev, pci_power_t state);
 	/* Caller holds memory_lock. */
 	void (*restore_dmabufs)(struct vfio_pci_core_device *vdev);
+	/* Caller holds the PCI device lock. The device need not be open. */
+	int (*sriov_configure)(struct vfio_pci_core_device *vdev, int nr_virtfn);
+	/* Caller holds memory_lock for this member of the reset set. */
+	int (*prepare_hot_reset)(struct vfio_pci_core_device *vdev);
 	/* Select and call both INTx methods while holding irqlock. */
 	irqreturn_t (*intx_handler)(struct vfio_pci_core_device *vdev,
 				    struct vfio_pci_irq_ctx *ctx);
drivers/vfio/pci/vfio_pci_core.c +50 -5

107 lines not on this page · raw message

Signature · 1 line
2.43.0
#31 lore raw

Re: [RFC PATCH v3 15/20] vfio/pci: Use access ops for SR-IOV configuration and shared reset

Thank you for your contribution! Sashiko AI review found 3 potential issues to consider: - [Medium] vfio/pci: vfio_pci_dev_set_hot_reset() calls D0 helper on failed devices - [Medium] vfio/pci…

Thank you for your contribution! Sashiko AI review found 3 potential issues to consider:
- [Medium] vfio/pci: vfio_pci_dev_set_hot_reset() calls D0 helper on failed devices
- [Medium] vfio/pci: vfio_pci_prepare_hot_reset_blocked() misses host recovery and caller checks
- [Medium] vfio/pci: unsynchronized open and recovery state in vfio_pci_core_sriov_configure()
--

commit abe74651d949fa43ae45614261ad8c3282ae7603
Author: Shameer Kolothum <[email protected]>

vfio/pci: Use access ops for SR-IOV configuration and shared reset

This commit routes SR-IOV configuration and shared hot-reset preparation through vfio_pci_access_ops so that recovering open devices are rejected while unopened or failed devices remain allowed.

Quoted 6 lines
drivers/vfio/pci/vfio_pci_core.c +0 -0
index 89cebe2522ba0..0ef51810254ea 100644
@@ -334,10 +339,21 @@ static void vfio_pci_restore_dmabufs_blocked(struct vfio_pci_core_device *vdev)
 {

drivers/vfio/pci/vfio_pci_core.c static void vfio_pci_restore_dmabufs_blocked(struct vfio_pci_core_device *vdev) · line 349

Quoted 10 lines
drivers/vfio/pci/vfio_pci_core.c +8 -0
 }
 
+static int vfio_pci_sriov_configure_blocked(struct vfio_pci_core_device *vdev,
+					    int nr_virtfn)
+{
+	/* SR-IOV can be configured without an open VFIO device. */
+	if (!READ_ONCE(vdev->device_open))
+		return vfio_pci_sriov_configure_live(vdev, nr_virtfn);
+	return -EIO;
+}

[Severity: Medium]
Can vfio_pci_core_sriov_configure() race with device open, close, or host PCI recovery because neither vdev->access_ops nor vdev->device_open is serialized by device_lock(&vdev->pdev->dev)?

vfio_pci_core_finish_enable() and vfio_pci_core_close_device() update vdev->device_open and vdev->access_ops under vdev->access_lock without holding device_lock(&vdev->pdev->dev):

CPU0 CPU1
vfio_pci_core_finish_enable()
WRITE_ONCE(vdev->device_open, true)

                                        vfio_pci_core_sriov_configure()
                                          vfio_pci_access_ops(vdev)
                                            /* sees blocked_ops */
                                          vfio_pci_sriov_configure_blocked()
                                            READ_ONCE(vdev->device_open)
                                            --> returns -EIO
  rcu_assign_pointer(vdev->access_ops,
                     &vfio_pci_live_ops)

In the reverse direction, vfio_pci_core_close_device() sets vdev->device_open = false before vfio_pci_core_disable() finishes tearing down the PF, and pcie_do_recovery() drops device_lock(&pdev->dev) between recovery callbacks while vdev->pci_recovery_host_active is true. Because vfio_pci_sriov_configure_blocked() only checks !READ_ONCE(vdev->device_open) without taking vdev->access_lock or checking vdev->pci_recovery_host_active, sysfs sriov_numvfs can run vfio_pci_sriov_configure_live() while close teardown or host recovery is in progress.

[ ... ]

Quoted 10 lines
drivers/vfio/pci/vfio_pci_core.c +6 -0
@@ -2776,6 +2801,23 @@ static int vfio_pci_dev_set_pm_runtime_get(struct vfio_device_set *dev_set)
 	return ret;
 }
 
+static int vfio_pci_prepare_hot_reset(struct vfio_pci_core_device *vdev)
+{
+	lockdep_assert_held_write(&vdev->memory_lock);
+	vfio_pci_dma_buf_move(vdev, true);
+	vfio_pci_zap_bars(vdev);
+	return 0;

drivers/vfio/pci/vfio_pci_core.c static int vfio_pci_dev_set_pm_runtime_get(struct vfio_device_set *dev_set) · line 2819

Quoted 10 lines
drivers/vfio/pci/vfio_pci_core.c +10 -0
+}
+
+static int vfio_pci_prepare_hot_reset_blocked(struct vfio_pci_core_device *vdev)
+{
+	/* Unopened and failed devices must not prevent a shared reset. */
+	if (READ_ONCE(vdev->device_open) &&
+	    !(READ_ONCE(vdev->pci_recovery_flags) & VFIO_PCI_RECOVERY_FAILED))
+		return -EBUSY;
+	return vfio_pci_prepare_hot_reset(vdev);
+}

[Severity: Medium]
Can vfio_pci_prepare_hot_reset_blocked() allow pci_reset_bus() to run while host PCI recovery is active or when VFIO_DEVICE_PCI_HOT_RESET is called directly on a failed device FD?

When host recovery is running on an unopened sibling, or on a sibling that was closed during recovery (where vfio_pci_core_close_device() clears device_open and pci_recovery_flags while keeping pci_recovery_host_active set), vfio_pci_prepare_hot_reset_blocked() sees !vdev->device_open and allows userspace to trigger pci_reset_bus() concurrently with host recovery. In addition, unlike vfio_pci_ioctl_reset(), vfio_pci_ioctl_pci_hot_reset() does not check whether the calling device FD itself has blocked access, so invoking VFIO_DEVICE_PCI_HOT_RESET on a device with VFIO_PCI_RECOVERY_FAILED succeeds and resets the bus.

Quoted 5 lines
drivers/vfio/pci/vfio_pci_core.c +0 -0
 static int vfio_pci_dev_set_hot_reset(struct vfio_device_set *dev_set,
 				      struct vfio_pci_group_info *groups,
 				      struct iommufd_ctx *iommufd_ctx)
@@ -2849,8 +2891,11 @@ static int vfio_pci_dev_set_hot_reset(struct vfio_device_set *dev_set,
 			break;

drivers/vfio/pci/vfio_pci_core.c static int vfio_pci_dev_set_hot_reset(struct vfio_device_set *dev_set, · line 2899

Quoted 10 lines
drivers/vfio/pci/vfio_pci_core.c +5 -2
 		}
 
-		vfio_pci_dma_buf_move(vdev, true);
-		vfio_pci_zap_bars(vdev);
+		ret = vfio_pci_access_ops(vdev)->prepare_hot_reset(vdev);
+		if (ret) {
+			up_write(&vdev->memory_lock);
+			break;
+		}
 	}

[Severity: Medium]
When a failed sibling passes vfio_pci_prepare_hot_reset_blocked(), does the subsequent loop in vfio_pci_dev_set_hot_reset() still call vfio_pci_try_set_power_state(vdev, PCI_D0) directly on that failed device?

drivers/vfio/pci/vfio_pci_core.c:vfio_pci_dev_set_hot_reset() {
    ...
	list_for_each_entry(vdev, &dev_set->device_list, vdev.dev_set_list) {
		ret = vfio_pci_try_set_power_state(vdev, PCI_D0);
		if (ret)
			goto out_unlock_memory;
	}

If an open sibling in VFIO_PCI_RECOVERY_FAILED is in a non-D0 state and inaccessible, vfio_pci_try_set_power_state() fails with -EIO and aborts the shared reset for the healthy siblings. If the failed sibling was in PCI_D3hot with vdev->needs_pm_restore set and responds to the transition, __vfio_pci_set_power_state() loads vdev->pm_save and calls pci_restore_state(), re-enabling the pre-failure PCI_COMMAND (including bus mastering and memory decoding) on the failed device before pci_reset_bus() saves and restores config space.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=15
#32 lore raw

RFC PATCH v3 16/20 vfio/pci: Restore device state from slot_reset()

In reply to Shameer Kolothum

A host reset loses device configuration and IRQ state. Add slot_reset() to restore the configuration saved at open, discard the old PM snapshot and tear down IRQ configuration for userspace to rebuild. Record RESET before restoration so failures also report that a reset occurred. Leave failed devices blocked.

Use the locked PCI power helper because native PCI recovery already holds pci_bus_sem for reading. Taking it again could deadlock behind a queued writer.

Preserve NoINTx and recovery masking when restoring saved configuration. The state saved at open predates the NoINTx correction. Route PM restoration through access ops while the device is open. Open and close continue to restore state directly.

Keep the original error_detected-only callback table for Xen PV and EEH- enabled builds. Xen treats a missing slot_reset callback as recovered but treats NONE from a callback as failure. EEH also uses callback presence to decide whether to keep a device. Disabling the new feature alone would not preserve these decisions.

---
 drivers/vfio/pci/vfio_pci_priv.h |   2 +
 include/linux/vfio_pci_core.h    |   1 +
 drivers/vfio/pci/vfio_pci.c      |  17 +++++
 drivers/vfio/pci/vfio_pci_core.c | 112 ++++++++++++++++++++++++++++++-
 4 files changed, 131 insertions(+), 1 deletion(-)
drivers/vfio/pci/vfio_pci_priv.h +2 -0
index e4f3e4f51180..bb22f0a15416 100644
@@ -98,6 +98,8 @@ struct vfio_pci_access_ops {
 			vfio_pci_access_fn fn, void *data);
 	/* Caller holds memory_lock. */
 	int (*set_power_state)(struct vfio_pci_core_device *vdev, pci_power_t state);
+	/* Used by PM restoration while the device is open. */
+	void (*restore_config)(struct vfio_pci_core_device *vdev);
 	/* Caller holds memory_lock. */
 	void (*restore_dmabufs)(struct vfio_pci_core_device *vdev);
 	/* Caller holds the PCI device lock. The device need not be open. */
include/linux/vfio_pci_core.h +1 -0
index fadbbaa58770..d36bca65d180 100644
@@ -288,6 +288,7 @@ int vfio_pci_dma_buf_iommufd_map(struct dma_buf_attachment *attachment,
 int vfio_pci_core_runtime_suspend(struct vfio_pci_core_device *vdev);
 int vfio_pci_core_runtime_resume(struct vfio_pci_core_device *vdev);
 pci_ers_result_t vfio_pci_core_aer_err_detected(struct vfio_pci_core_device *vdev);
+pci_ers_result_t vfio_pci_core_aer_slot_reset(struct vfio_pci_core_device *vdev);
 
 /*
  * Per-driver PM/AER trampolines. driver_data points to the driver's private
drivers/vfio/pci/vfio_pci.c +17 -0
index e7c012febe91..b4b3351cbd85 100644
@@ -24,6 +24,7 @@
 #include <linux/slab.h>
 #include <linux/types.h>
 #include <linux/uaccess.h>
+#include <xen/xen.h>
 
 #include "vfio_pci_priv.h"
 
@@ -249,8 +250,20 @@ static pci_ers_result_t vfio_pci_aer_err_detected(struct pci_dev *pdev,
 	return vfio_pci_core_aer_err_detected(vdev);
 }
 
+static pci_ers_result_t vfio_pci_aer_slot_reset(struct pci_dev *pdev)
+{
+	struct vfio_pci_core_device *vdev = pci_get_drvdata(pdev);
+
+	return vfio_pci_core_aer_slot_reset(vdev);
+}
+
+static const struct pci_error_handlers vfio_pci_legacy_err_handlers = {
+	.error_detected = vfio_pci_aer_err_detected,
+};
+
 static const struct pci_error_handlers vfio_pci_err_handlers = {
 	.error_detected = vfio_pci_aer_err_detected,
+	.slot_reset = vfio_pci_aer_slot_reset,
 };
 
 static struct pci_driver vfio_pci_driver = {
@@ -309,6 +322,10 @@ static int __init vfio_pci_init(void)
 {
 	int ret;
 
+	/* Xen PV and EEH check for callbacks. Keep the original table. */
+	if (xen_pv_domain() || IS_ENABLED(CONFIG_EEH))
+		vfio_pci_driver.err_handler = &vfio_pci_legacy_err_handlers;
+
 	/* Register and scan for devices */
 	ret = pci_register_driver(&vfio_pci_driver);
 	if (ret)
drivers/vfio/pci/vfio_pci_core.c +111 -1

154 lines not on this page · raw message

Signature · 1 line
2.43.0
#33 lore raw

Re: [RFC PATCH v3 16/20] vfio/pci: Restore device state from slot_reset()

Sashiko has reviewed this patch and found no issues.
Quoted 15 lines

A host reset loses device configuration and IRQ state. Add slot_reset() to restore the configuration saved at open, discard the old PM snapshot and tear down IRQ configuration for userspace to rebuild. Record RESET before restoration so failures also report that a reset occurred. Leave failed devices blocked.

Use the locked PCI power helper because native PCI recovery already holds pci_bus_sem for reading. Taking it again could deadlock behind a queued writer.

Preserve NoINTx and recovery masking when restoring saved configuration. The state saved at open predates the NoINTx correction. Route PM restoration through access ops while the device is open. Open and close continue to restore state directly.

Quoted 7 lines

Keep the original error_detected-only callback table for Xen PV and EEH- enabled builds. Xen treats a missing slot_reset callback as recovered but treats NONE from a callback as failure. EEH also uses callback presence to decide whether to keep a device. Disabling the new feature alone would not preserve these decisions.

Sashiko has reviewed this patch and found no issues.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=16
#34 lore raw

RFC PATCH v3 17/20 vfio/pci: Complete recovery in resume()

In reply to Shameer Kolothum

Add resume() to finish recovery and allow device access again. This also handles recovery that did not require a reset.

Complete deferred D0 requests and restore PCI_COMMAND, DMA-BUF mappings and INTx state before installing live ops. Take igate before memory_lock, matching IRQ configuration, and hold both through restoration and live-ops installation so no deferred D0 request is missed.

Use the locked PCI power helper because native PCI recovery already holds pci_bus_sem for reading. Leave failed devices blocked.

---
 include/linux/vfio_pci_core.h    |  3 ++
 drivers/vfio/pci/vfio_pci.c      |  8 +++
 drivers/vfio/pci/vfio_pci_core.c | 87 ++++++++++++++++++++++++++++++++
 3 files changed, 98 insertions(+)
include/linux/vfio_pci_core.h +3 -0
index d36bca65d180..a7a87c04ee60 100644
@@ -147,6 +147,8 @@ struct vfio_pci_core_device {
 	bool			sriov_active;
 	struct pci_saved_state	*pci_saved_state;
 	struct pci_saved_state	*pm_save;
+	/* Deferred D0 request, protected by memory_lock. */
+	bool			power_up_pending;
 	int			ioeventfds_nr;
 	struct vfio_pci_eventfd __rcu *err_trigger;
 	struct vfio_pci_eventfd __rcu *req_trigger;
@@ -289,6 +291,7 @@ int vfio_pci_core_runtime_suspend(struct vfio_pci_core_device *vdev);
 int vfio_pci_core_runtime_resume(struct vfio_pci_core_device *vdev);
 pci_ers_result_t vfio_pci_core_aer_err_detected(struct vfio_pci_core_device *vdev);
 pci_ers_result_t vfio_pci_core_aer_slot_reset(struct vfio_pci_core_device *vdev);
+void vfio_pci_core_aer_resume(struct vfio_pci_core_device *vdev);
 
 /*
  * Per-driver PM/AER trampolines. driver_data points to the driver's private
drivers/vfio/pci/vfio_pci.c +8 -0
index b4b3351cbd85..13f4f1faf90d 100644
@@ -257,6 +257,13 @@ static pci_ers_result_t vfio_pci_aer_slot_reset(struct pci_dev *pdev)
 	return vfio_pci_core_aer_slot_reset(vdev);
 }
 
+static void vfio_pci_aer_resume(struct pci_dev *pdev)
+{
+	struct vfio_pci_core_device *vdev = pci_get_drvdata(pdev);
+
+	vfio_pci_core_aer_resume(vdev);
+}
+
 static const struct pci_error_handlers vfio_pci_legacy_err_handlers = {
 	.error_detected = vfio_pci_aer_err_detected,
 };
@@ -264,6 +271,7 @@ static const struct pci_error_handlers vfio_pci_legacy_err_handlers = {
 static const struct pci_error_handlers vfio_pci_err_handlers = {
 	.error_detected = vfio_pci_aer_err_detected,
 	.slot_reset = vfio_pci_aer_slot_reset,
+	.resume = vfio_pci_aer_resume,
 };
 
 static struct pci_driver vfio_pci_driver = {
drivers/vfio/pci/vfio_pci_core.c +87 -0

99 lines not on this page · raw message

Signature · 1 line
2.43.0
#35 lore raw

Re: [RFC PATCH v3 17/20] vfio/pci: Complete recovery in resume()

Sashiko has reviewed this patch and found no issues.
Quoted 9 lines

Add resume() to finish recovery and allow device access again. This also handles recovery that did not require a reset.

Complete deferred D0 requests and restore PCI_COMMAND, DMA-BUF mappings and INTx state before installing live ops. Take igate before memory_lock, matching IRQ configuration, and hold both through restoration and live-ops installation so no deferred D0 request is missed.

Quoted 4 lines

Use the locked PCI power helper because native PCI recovery already holds pci_bus_sem for reading. Leave failed devices blocked.

Sashiko has reviewed this patch and found no issues.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=17
#36 lore raw

RFC PATCH v3 18/20 vfio/pci: Block device access during host recovery

In reply to Shameer Kolothum

An error notification does not stop userspace or asynchronous VFIO work from accessing a device while the host is recovering it.

For an open device with recovery enabled, install blocked ops in error_detected() and drain existing SRCU readers. Block INTx unmasking, remove BAR mappings and revoke DMA-BUF mappings. Disable bus mastering on a normal channel. Keep shared-line INTx masking available.

Hold memory_lock and irqlock when publishing blocked ops so an earlier PM restore cannot undo the blocked handler's INTx mask. Release both locks before draining SRCU, since readers may need memory_lock to finish. Use the same transition during close.

Defer blocked D0 requests while the device is open and reject them once close begins. Reject low-power requests while blocked. Check for ongoing host recovery before enabling a device.

Return CAN_RECOVER for a normal channel, NEED_RESET for a frozen channel and DISCONNECT for permanent failure. If saving PCI_COMMAND or disabling bus mastering fails, keep access blocked and return NONE so other devices can continue recovery. Preserve existing error eventfd notifications.

---
 include/linux/vfio_pci_core.h    |   5 +-
 drivers/vfio/pci/vfio_pci.c      |   2 +-
 drivers/vfio/pci/vfio_pci_core.c | 156 ++++++++++++++++++++++++++++++-
 3 files changed, 157 insertions(+), 6 deletions(-)
include/linux/vfio_pci_core.h +3 -2
index a7a87c04ee60..cd7d87788de0 100644
@@ -289,7 +289,8 @@ int vfio_pci_dma_buf_iommufd_map(struct dma_buf_attachment *attachment,
 
 int vfio_pci_core_runtime_suspend(struct vfio_pci_core_device *vdev);
 int vfio_pci_core_runtime_resume(struct vfio_pci_core_device *vdev);
-pci_ers_result_t vfio_pci_core_aer_err_detected(struct vfio_pci_core_device *vdev);
+pci_ers_result_t vfio_pci_core_aer_err_detected(struct vfio_pci_core_device *vdev,
+						pci_channel_state_t state);
 pci_ers_result_t vfio_pci_core_aer_slot_reset(struct vfio_pci_core_device *vdev);
 void vfio_pci_core_aer_resume(struct vfio_pci_core_device *vdev);
 
@@ -322,7 +323,7 @@ static pci_ers_result_t name##_aer_err_detected(struct pci_dev *pdev,         \
 {                                                                             \
 	drv_type *drvdata = pci_get_drvdata(pdev);                            \
 									      \
-	return vfio_pci_core_aer_err_detected(&drvdata->member);              \
+	return vfio_pci_core_aer_err_detected(&drvdata->member, state);       \
 }
 
 #endif /* VFIO_PCI_CORE_H */
drivers/vfio/pci/vfio_pci.c +1 -1
index 13f4f1faf90d..a5ba4bb11638 100644
@@ -247,7 +247,7 @@ static pci_ers_result_t vfio_pci_aer_err_detected(struct pci_dev *pdev,
 {
 	struct vfio_pci_core_device *vdev = pci_get_drvdata(pdev);
 
-	return vfio_pci_core_aer_err_detected(vdev);
+	return vfio_pci_core_aer_err_detected(vdev, state);
 }
 
 static pci_ers_result_t vfio_pci_aer_slot_reset(struct pci_dev *pdev)
drivers/vfio/pci/vfio_pci_core.c +153 -3

199 lines not on this page · raw message

Signature · 1 line
2.43.0
#37 lore raw

Re: [RFC PATCH v3 18/20] vfio/pci: Block device access during host recovery

Thank you for your contribution! Sashiko AI review found 5 potential issues to consider: - [High] vfio/pci: unrestored BARs after bus reset when recovery fails in err_detected() - [Medium] vfio/pci…

Thank you for your contribution! Sashiko AI review found 5 potential issues to consider:
- [High] vfio/pci: unrestored BARs after bus reset when recovery fails in err_detected()
- [Medium] vfio/pci: TOCTOU race on pci_recovery_host_active during device open
- [Medium] vfio/pci: close during host recovery races with bus reset and loses saved state
- [Medium] vfio/pci: close clears device_open before blocking PM config restore
- [Low] vfio/pci: dead !device_open check in vfio_pci_core_aer_err_detected()
--

commit f9997001c74d4f1e906c8a9ec79cb059867865ef
Author: Shameer Kolothum <[email protected]>

vfio/pci: Block device access during host recovery

This commit blocks userspace and asynchronous VFIO device access during host PCI error recovery in vfio_pci_core_aer_err_detected(), coordinates blocked power-state and INTx handling with memory_lock and irqlock, and rejects device open while host recovery is active.

Quoted 4 lines
drivers/vfio/pci/vfio_pci_core.c +0 -0
index 09923a48a5470..5e9451e082630 100644

[ ... ]

Quoted 7 lines
drivers/vfio/pci/vfio_pci_core.c +3 -0
@@ -410,10 +414,17 @@ static void vfio_pci_core_block_access(struct vfio_pci_core_device *vdev)
 	unsigned long flags;
 
 	lockdep_assert_held(&vdev->access_lock);
+	/*
+	 * Finish live PM restoration before blocked INTx handlers can mask the
+	 * device. Otherwise an old restore can undo their hardware mask.

drivers/vfio/pci/vfio_pci_core.c static void vfio_pci_core_block_access(struct vfio_pci_core_device *vdev) · line 429

Quoted 10 lines
drivers/vfio/pci/vfio_pci_core.c +4 -0
+	 */
+	down_write(&vdev->memory_lock);
 	spin_lock_irqsave(&vdev->irqlock, flags);
 	rcu_assign_pointer(vdev->access_ops, &vfio_pci_blocked_ops);
 	spin_unlock_irqrestore(&vdev->irqlock, flags);
+	up_write(&vdev->memory_lock);
 
+	/* SRCU readers may need memory_lock to finish. */
 	synchronize_srcu(&vdev->access_srcu);
 }

[Severity: Medium]
Does clearing vdev->device_open before calling vfio_pci_core_block_access() in vfio_pci_core_close_device() defeat this protection against PM config restoration unmasking INTx?

In __vfio_pci_set_power_state(), D3hot-to-D0 restoration bypasses access_ops->restore_config() as soon as vdev->device_open is false:

drivers/vfio/pci/vfio_pci_core.c:__vfio_pci_set_power_state() {
    ...
		} else if (needs_restore) {
			pci_load_and_free_saved_state(pdev, &vdev->pm_save);
			/* Open and close restore state without reopening access. */
			if (!READ_ONCE(vdev->device_open))
				pci_restore_state(pdev);
			else
				vfio_pci_access_ops(vdev)->restore_config(vdev);
		}
    ...
}

If a NoSoftRst- PCI 2.3 device entered D3hot while INTx was unmasked (saving unmasked PCI_COMMAND in vdev->pm_save) and then had INTx masked by recovery, both an in-flight runtime PM transition holding memory_lock when vfio_pci_core_close_device() clears vdev->device_open and the subsequent vfio_pci_set_power_state(vdev, PCI_D0) at the start of vfio_pci_core_disable() take the pci_restore_state(pdev) branch. That restores the unmasked PCI_COMMAND in hardware while ctx->masked is still true and the IRQ handler is still registered, so a level-triggered INTx assertion before vfio_pci_set_irqs_ioctl() runs sees ctx->masked == true in vfio_pci_intx_mask_for_recovery() and returns IRQ_NONE without masking the shared interrupt line.

Quoted 5 lines
drivers/vfio/pci/vfio_pci_core.c +1 -0
@@ -756,6 +767,16 @@ int vfio_pci_core_enable(struct vfio_pci_core_device *vdev)
 	u16 cmd;
 	u8 msix_pos;
 
+	if (vdev->pci_recovery_supported) {

drivers/vfio/pci/vfio_pci_core.c int vfio_pci_core_enable(struct vfio_pci_core_device *vdev) · line 780

Quoted 10 lines
drivers/vfio/pci/vfio_pci_core.c +9 -0
+		if (pci_dev_is_disconnected(pdev))
+			return -ENODEV;
+		mutex_lock(&vdev->access_lock);
+		ret = vdev->pci_recovery_host_active ? -EBUSY : 0;
+		mutex_unlock(&vdev->access_lock);
+		if (ret)
+			return ret;
+	}
+
 	if (!vdev->disable_idle_d3) {

[Severity: Medium]
Can host recovery start after vfio_pci_core_enable() drops vdev->access_lock and still allow vfio_pci_core_finish_enable() to publish live access?

CPU0 CPU1
vfio_pci_core_enable()
mutex_lock(&vdev->access_lock)
/* pci_recovery_host_active is false */
mutex_unlock(&vdev->access_lock)

                                        pcie_do_recovery()
                                          vfio_pci_core_aer_err_detected()
                                            mutex_lock(&vdev->access_lock)
                                            vdev->pci_recovery_host_active = true
                                            /* !pci_recovery_enabled */
                                            mutex_unlock(&vdev->access_lock)
                                          reset_subordinates()

pci_save_state(pdev)
vfio_config_init(vdev)
vfio_pci_core_finish_enable()
mutex_lock(&vdev->access_lock)
WRITE_ONCE(vdev->device_open, true) rcu_assign_pointer(vdev->access_ops,
&vfio_pci_live_ops) <-- live access while recovery runs

Because vfio_pci_core_enable() runs without device_lock(&pdev->dev) held across enablement and vfio_pci_core_finish_enable() does not re-check vdev->pci_recovery_host_active or pci_dev_is_disconnected(pdev), the open succeeds while host recovery is active, vdev->pci_saved_state and vdev->rbar[] can capture corrupted or post-reset BAR values, and vfio_pci_core_aer_slot_reset() skips state restoration because VFIO_PCI_RECOVERY_IN_PROGRESS was never set.

drivers/vfio/pci/vfio_pci_core.c void vfio_pci_core_close_device(struct vfio_device *core_vdev) · line 1028

Quoted 10 lines
drivers/vfio/pci/vfio_pci_core.c +4 -0
@@ -999,6 +1020,10 @@ void vfio_pci_core_close_device(struct vfio_device *core_vdev)
 			WRITE_ONCE(vdev->device_open, false);
 			vfio_pci_core_block_access(vdev);
 			WRITE_ONCE(vdev->pci_recovery_flags, 0);
+			/* Discard pending D0 before the next open. */
+			down_write(&vdev->memory_lock);
+			vdev->power_up_pending = false;
+			up_write(&vdev->memory_lock);
 		}
 	}

[Severity: Medium]
What happens if userspace closes the device after vfio_pci_core_aer_err_detected() signals err_trigger, while pcie_do_recovery() is resetting the bus?

CPU0 CPU1
pcie_do_recovery()
vfio_pci_core_aer_err_detected()

    /* returns NEED_RESET */
                                        vfio_pci_core_close_device()
                                          WRITE_ONCE(vdev->device_open, false)
                                          WRITE_ONCE(vdev->pci_recovery_flags, 0)
                                          vfio_pci_core_disable()
                                            pci_load_and_free_saved_state(
                                                pdev, &vdev->pci_saved_state)
                                            pci_restore_state(pdev)
  reset_subordinates()
    /* resets bus, wiping BARs */
  vfio_pci_core_aer_slot_reset()
    /* !device_open -> returns NONE */  <-- BARs left zeroed in hardware

pcie_do_recovery() drops device_lock(&pdev->dev) between error_detected() and slot_reset(). If vfio_pci_core_disable() consumes and frees vdev->pci_saved_state before reset_subordinates() finishes resetting the secondary bus without restoring config space, vfio_pci_core_aer_slot_reset() sees !vdev->device_open and returns early, leaving the hardware BARs zeroed for the next open. If close instead runs just after reset_subordinates() wipes the BARs, vfio_pci_core_disable() tears down MSI-X in vfio_pci_set_irqs_ioctl() before restoring vdev->pci_saved_state.

Quoted 1 line
drivers/vfio/pci/vfio_pci_core.c +0 -0
@@ -2660,17 +2685,142 @@ void vfio_pci_core_unregister_device(struct vfio_pci_core_device *vdev)

[ ... ]

Quoted 10 lines
drivers/vfio/pci/vfio_pci_core.c +10 -0
+	mutex_lock(&vdev->access_lock);
+	/*
+	 * Track host recovery even without userspace opt-in so open can reject
+	 * attempts while recovery is running.
+	 */
+	vdev->pci_recovery_host_active =
+		state != pci_channel_io_perm_failure;
+	if (!vdev->pci_recovery_enabled)
+		goto out_unlock;
+

drivers/vfio/pci/vfio_pci_core.c void vfio_pci_core_unregister_device(struct vfio_pci_core_device *vdev) · line 2746

Quoted 10 lines
drivers/vfio/pci/vfio_pci_core.c +10 -0
+	/* Keep failed devices blocked until close. */
+	if (vdev->pci_recovery_flags & VFIO_PCI_RECOVERY_FAILED) {
+		result = PCI_ERS_RESULT_NONE;
+		goto out_unlock;
+	}
+
+	if (!vdev->device_open) {
+		result = PCI_ERS_RESULT_NONE;
+		goto out_unlock;
+	}

[Severity: Low]
Can the !vdev->device_open check ever be reached here?

Under vdev->access_lock, vdev->pci_recovery_enabled is only set when vdev->device_open is true and is cleared alongside vdev->device_open in vfio_pci_core_close_device(), so !vdev->pci_recovery_enabled always branches to out_unlock first (returning PCI_ERS_RESULT_CAN_RECOVER rather than PCI_ERS_RESULT_NONE when the device is closed).

Quoted 2 lines
drivers/vfio/pci/vfio_pci_core.c +2 -0
+
+	vfio_pci_core_block_access(vdev);

[ ... ]

drivers/vfio/pci/vfio_pci_core.c void vfio_pci_core_unregister_device(struct vfio_pci_core_device *vdev) · line 2809

Quoted 11 lines
drivers/vfio/pci/vfio_pci_core.c +10 -0
+	if (!ret && !vdev->pci_2_3 && !nested)
+		ret = vfio_pci_recovery_save_and_clear_master(vdev, 0);
 
+	if (ret) {
+		flags |= VFIO_PCI_RECOVERY_FAILED;
+		flags &= ~VFIO_PCI_RECOVERY_IN_PROGRESS;
+		result = PCI_ERS_RESULT_NONE;
+	} else {
+		flags |= VFIO_PCI_RECOVERY_IN_PROGRESS;
+	}
+	WRITE_ONCE(vdev->pci_recovery_flags, flags);

[Severity: High]
If recovery fails in vfio_pci_core_aer_err_detected() (or VFIO_PCI_RECOVERY_FAILED is already set on a subsequent frozen-channel error) and pcie_do_recovery() still resets the subordinate bus, will the device's BARs be left zeroed when vfio_pci_core_disable() later tears down MSI-X?

Clearing VFIO_PCI_RECOVERY_IN_PROGRESS and returning PCI_ERS_RESULT_NONE allows pcie_do_recovery() to proceed with reset_subordinates() when a sibling device requests PCI_ERS_RESULT_NEED_RESET or when state is pci_channel_io_frozen, wiping this device's BARs in hardware. Because VFIO_PCI_RECOVERY_IN_PROGRESS is not set, vfio_pci_core_aer_slot_reset() returns early without calling vfio_pci_recovery_restore_state() or tearing down IRQs:

drivers/vfio/pci/vfio_pci_core.c:vfio_pci_core_aer_slot_reset() {
	mutex_lock(&vdev->access_lock);
	if (!(vdev->pci_recovery_flags & VFIO_PCI_RECOVERY_IN_PROGRESS) ||
	    !vdev->device_open) {
		mutex_unlock(&vdev->access_lock);
		return PCI_ERS_RESULT_NONE;
	}
    ...
}

When userspace later closes the device, vfio_pci_core_disable() calls vfio_pci_set_irqs_ioctl() -> vfio_msi_disable() before loading vdev->pci_saved_state, which enables PCI_COMMAND_MEMORY via vfio_pci_memory_lock_and_enable() and writes to the MSI-X table while the hardware BARs are still 0x00000000.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=18
#38 lore raw

RFC PATCH v3 19/20 vfio/pci: Add VFIO_DEVICE_FEATURE_PCI_ERROR_RECOVERY

In reply to Shameer Kolothum

Userspace needs a way to enable host recovery and find out whether it is still running, succeeded, reset the device or failed.

Add a device feature that registers a recovery eventfd and reports status flags and a sequence number. SET with eventfd -1 disables recovery. GET is available while hardware access is blocked. The sequence number lets userspace detect recovery that finished before it read the notification.

Reject SET if access is blocked, recovery has failed or a recorded host recovery is still running. This also applies if userspace had not enabled recovery when the event started, so a late subscription cannot report part of an older event as a new one. Updating the eventfd resets status for the current open. The existing error eventfd is unchanged.

Add the recovery eventfd and signal it when recovery starts, completes or fails. Release it during close. Signal under access_lock so SET cannot replace the eventfd and reset status before notification.

Userspace must probe each device and restore guest state after a reset.

---
 include/linux/vfio_pci_core.h    |   1 +
 include/uapi/linux/vfio.h        |  51 ++++++++++++
 drivers/vfio/pci/vfio_pci_core.c | 139 +++++++++++++++++++++++++++++++
 3 files changed, 191 insertions(+)
include/linux/vfio_pci_core.h +1 -0
index cd7d87788de0..896c176c28a0 100644
@@ -152,6 +152,7 @@ struct vfio_pci_core_device {
 	int			ioeventfds_nr;
 	struct vfio_pci_eventfd __rcu *err_trigger;
 	struct vfio_pci_eventfd __rcu *req_trigger;
+	struct vfio_pci_eventfd __rcu *pci_recovery_trigger;
 	struct eventfd_ctx	*pm_wake_eventfd_ctx;
 	struct list_head	dummy_resources_list;
 	struct mutex		ioeventfds_lock;
include/uapi/linux/vfio.h +51 -0

57 lines not on this page · raw message

drivers/vfio/pci/vfio_pci_core.c +139 -0

205 lines not on this page · raw message

Signature · 1 line
2.43.0
#39 lore raw

Re: [RFC PATCH v3 19/20] vfio/pci: Add VFIO_DEVICE_FEATURE_PCI_ERROR_RECOVERY

Sashiko has reviewed this patch and found no issues.
Quoted 15 lines

Userspace needs a way to enable host recovery and find out whether it is still running, succeeded, reset the device or failed.

Add a device feature that registers a recovery eventfd and reports status flags and a sequence number. SET with eventfd -1 disables recovery. GET is available while hardware access is blocked. The sequence number lets userspace detect recovery that finished before it read the notification.

Reject SET if access is blocked, recovery has failed or a recorded host recovery is still running. This also applies if userspace had not enabled recovery when the event started, so a late subscription cannot report part of an older event as a new one. Updating the eventfd resets status for the current open. The existing error eventfd is unchanged.

Quoted 7 lines

Add the recovery eventfd and signal it when recovery starts, completes or fails. Release it during close. Signal under access_lock so SET cannot replace the eventfd and reset status before notification.

Userspace must probe each device and restore guest state after a reset.

Sashiko has reviewed this patch and found no issues.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=19
#40 lore raw

RFC PATCH v3 20/20 vfio/pci: Enable recovery for supported native PCIe endpoints

In reply to Shameer Kolothum

Enable recovery for generic vfio-pci endpoints on the subordinate bus of a PCIe Root Port or Downstream Port.

The recovery power helpers require pci_bus_sem held for reading. Native PCI recovery provides this through its bus walk. Check the endpoint's bus as well as its upstream port, since synthetic VF buses are outside that walk. Exclude direct RCiEP and root-bus endpoints.

Exclude EEH-enabled builds and Xen PV because their callbacks do not provide the required locking. Exclude s390 and CXL memory devices because their recovery paths can omit completion callbacks. Variant drivers remain unsupported until their hardware accesses are protected.

Exclude functions with VPD redirected to function zero. The lookup takes pci_bus_sem inside SRCU and can deadlock with recovery when a bus writer is queued.

Also exclude Intel Ethernet function zero devices with VPD. Redirected accesses from sibling functions bypass the target's access checks. Exclude potential targets even before siblings are probed so support does not depend on probe order.

These restrictions affect only the new recovery feature. Ordinary assignment and existing error notifications remain available.

---
 include/uapi/linux/vfio.h   |  6 +++-
 drivers/vfio/pci/vfio_pci.c | 61 +++++++++++++++++++++++++++++++++++++
 2 files changed, 66 insertions(+), 1 deletion(-)
include/uapi/linux/vfio.h +5 -1
index 86a3952ee302..f855d064c4b0 100644
@@ -1558,7 +1558,11 @@ struct vfio_device_feature_zpci_err {
 /*
  * Report host PCI error recovery state for this device.
  *
- * Userspace must probe the feature on each device before enabling it.
+ * Support is limited to PCIe endpoints on a Root Port or Downstream Port
+ * subordinate bus, using native PCI error recovery. Direct RCiEP, EEH, Xen
+ * PV, s390 and CXL memory recovery are not supported. Functions with
+ * VPD redirected to function zero, and Intel Ethernet function zero
+ * devices with VPD are excluded. Userspace must probe each device.
  *
  * VFIO_DEVICE_FEATURE_SET with a valid eventfd enables recovery for this
  * open and requests notifications when each event starts and ends. SET
drivers/vfio/pci/vfio_pci.c +61 -0

77 lines not on this page · raw message

Signature · 1 line
2.43.0
#41 lore raw

Re: [RFC PATCH v3 20/20] vfio/pci: Enable recovery for supported native PCIe endpoints

Thank you for your contribution! Sashiko AI review found 3 potential issues to consider: - [High] vfio/pci: VFIO_DEVICE_GET_INFO fails with -EIO on VFs when PF is closed - [Medium] vfio/pci…

Thank you for your contribution! Sashiko AI review found 3 potential issues to consider:
- [High] vfio/pci: VFIO_DEVICE_GET_INFO fails with -EIO on VFs when PF is closed
- [Medium] vfio/pci: circular locking between access_srcu and access_lock on VF and PF
- [Medium] vfio/pci: false recovery in vfio_pci_supports_recovery() for SR-IOV VFs
--

commit 4182befeb4982c12ce357363cc414cc54cc9381b
Author: Shameer Kolothum <[email protected]>

vfio/pci: Enable recovery for supported native PCIe endpoints

This commit enables host PCI error recovery for supported generic vfio-pci endpoints on the subordinate bus of a PCIe Root Port or Downstream Port and updates the UAPI documentation.

Quoted 49 lines
drivers/vfio/pci/vfio_pci.c +41 -0
index a5ba4bb116382..fe9b8b8deccab 100644
@@ -126,10 +126,70 @@ static int vfio_pci_open_device(struct vfio_device *core_vdev)
 	return 0;
 }
 
+static bool vfio_pci_supports_recovery(struct pci_dev *pdev)
+{
+	struct pci_dev *bridge = pci_upstream_bridge(pdev);
+
+	/*
+	 * Looking up function zero for VPD takes pci_bus_sem inside access_srcu.
+	 * Recovery holds the bus read lock while waiting for these SRCU readers.
+	 * A queued bus writer can leave both paths waiting for each other.
+	 */
+	if (pdev->dev_flags & PCI_DEV_FLAGS_VPD_REF_F0)
+		return false;
+
+	/*
+	 * quirk_f0_vpd_link() can redirect sibling VPD accesses to these devices
+	 * without checking whether recovery has blocked function zero. Exclude
+	 * them even if no redirecting sibling is present at probe.
+	 */
+	if (!PCI_FUNC(pdev->devfn) && pdev->vpd.cap &&
+	    pdev->vendor == PCI_VENDOR_ID_INTEL &&
+	    (pdev->class >> 8) == PCI_CLASS_NETWORK_ETHERNET)
+		return false;
+
+	/*
+	 * EEH and Xen PV can call recovery without the required bus locking.
+	 * s390 recovery can omit completion callbacks.
+	 */
+	if (IS_ENABLED(CONFIG_S390) || IS_ENABLED(CONFIG_EEH) ||
+	    xen_pv_domain())
+		return false;
+
+	/*
+	 * Recovery power helpers require pci_bus_sem held for reading.
+	 * The native recovery walk holds it on a port's subordinate bus.
+	 * Direct RCiEP recovery does not use that walk.
+	 */
+	if (!pci_is_pcie(pdev))
+		return false;
+
+	if (pci_pcie_type(pdev) != PCI_EXP_TYPE_ENDPOINT &&
+	    pci_pcie_type(pdev) != PCI_EXP_TYPE_LEG_END)
+		return false;

drivers/vfio/pci/vfio_pci.c static int vfio_pci_open_device(struct vfio_device *core_vdev) · line 179

Quoted 10 lines
drivers/vfio/pci/vfio_pci.c +10 -0
+
+	if (!bridge || !pci_is_pcie(bridge))
+		return false;
+
+	if (pci_pcie_type(bridge) != PCI_EXP_TYPE_ROOT_PORT &&
+	    pci_pcie_type(bridge) != PCI_EXP_TYPE_DOWNSTREAM)
+		return false;
+
+	if (pdev->bus != bridge->subordinate)
+		return false;

[Severity: Medium]
Does this allow SR-IOV VFs placed on the PF's bus to report a false recovery when the PF itself does not recover?

When pci_iov_virtfn_bus() matches the PF's bus number, virtfn_add_bus() places VFs directly on pf->bus (bridge->subordinate) rather than a synthetic VF bus, and pci_upstream_bridge(pdev) resolves through the PF to bridge. vfio_pci_supports_recovery() then returns true for the VF even if the PF is excluded from recovery (such as an Intel Ethernet function 0 PF with VPD), closed, or has recovery disabled.

If pcie_do_recovery() performs a secondary bus reset on bridge, the PF is reset in hardware, clearing PCI_SRIOV_CTRL_VFE and disabling all VFs. When the PF is not participating in vfio-pci recovery, vfio_pci_core_aer_slot_reset() returns PCI_ERS_RESULT_NONE on the PF without calling pci_restore_state() / pci_restore_iov_state(). The subsequent vfio_pci_core_aer_slot_reset() on the VF still succeeds because pci_set_power_state_locked() sees current_state == PCI_D0 and pci_restore_state() returns void, so the VF reports PCI_ERS_RESULT_RECOVERED and re-enables live access while VFE remains cleared on the PF.

[Severity: Medium]
Enabling pci_recovery_supported on both a PF and an SR-IOV VF on the same bus also triggers a lockdep circular locking splat when VFIO_DEVICE_GET_INFO is called on the VF while the PF is open.

All vfio_pci_core_device instances share the same lockdep classes for access_lock and access_srcu. vfio_pci_core_block_access() runs synchronize_srcu(&vdev->access_srcu) under vdev->access_lock (establishing access_lock -> access_srcu), whereas vfio_pci_ioctl_get_info() on the VF holds srcu_read_lock(&vdev->access_srcu) across vfio_pci_access_atomic_cap(), which takes pf_vdev->access_lock and srcu_read_lock(&pf_vdev->access_srcu):

drivers/vfio/pci/vfio_pci_core.c:vfio_pci_access_atomic_cap() {
	...
	guard(mutex)(&pf_vdev->access_lock);
	if (!pf_vdev->device_open)
		return -EIO;

	return vfio_pci_core_access(pf_vdev, vfio_pci_access_atomic_cap_target,
				    data);
}
Quoted 17 lines
drivers/vfio/pci/vfio_pci.c +9 -0
+
+	/* CXL RCH recovery can omit completion callbacks. */
+	if ((pdev->class >> 8) == PCI_CLASS_MEMORY_CXL)
+		return false;
+
+	return true;
+}
+
 static int vfio_pci_init_dev(struct vfio_device *core_vdev)
 {
 	struct vfio_pci_core_device *vdev =
 		container_of(core_vdev, struct vfio_pci_core_device, vdev);
+	struct pci_dev *pdev = to_pci_dev(core_vdev->dev);
 
 	/*
 	 * These behaviors originated in vfio-pci and moved into

drivers/vfio/pci/vfio_pci.c static int vfio_pci_init_dev(struct vfio_device *core_vdev) · line 203

Quoted 4 lines
drivers/vfio/pci/vfio_pci.c +1 -0
@@ -140,6 +200,7 @@ static int vfio_pci_init_dev(struct vfio_device *core_vdev)
 	 */
 	vdev->nointxmask = nointxmask;
 	vdev->disable_idle_d3 = disable_idle_d3;
+	vdev->pci_recovery_supported = vfio_pci_supports_recovery(pdev);

[Severity: High]
Does setting pci_recovery_supported on an SR-IOV PF cause VFIO_DEVICE_GET_INFO on its VFs to fail with -EIO whenever the PF is not open?

SR-IOV can be enabled while a vfio-pci PF is unopened, or the PF fd can be closed after configuring a vf_token while VFs are used. Once pf_vdev->pci_recovery_supported is true, vfio_pci_ioctl_get_info() -> vfio_pci_access_atomic_cap() on a VF checks pf_vdev->device_open and returns -EIO whenever the PF is closed, even when no host recovery is active on the PF:

drivers/vfio/pci/vfio_pci_core.c:vfio_pci_access_atomic_cap() {
	struct vfio_pci_core_device *pf_vdev = vdev->sriov_pf_core_dev;

	if (!pf_vdev || !pf_vdev->pci_recovery_supported)
		return vfio_pci_info_atomic_cap(vdev, data);
	...
	guard(mutex)(&pf_vdev->access_lock);
	if (!pf_vdev->device_open)
		return -EIO;
}

vfio_pci_ioctl_get_info() treats any return value other than 0 or -ENODEV as fatal and fails the VF's VFIO_DEVICE_GET_INFO ioctl with -EIO.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=20