Skip to contentSkip to conversation

vfio/pci: Fix power state race with SR-IOV enablement

PATCH New David Matlack 1 participant 9 messages Nov 19

Subsystems (5)
lore
Share & tools
lore

David Matlack

author maintainer
#0 lore raw

v2 cover vfio/pci: Fix power state race with SR-IOV enablement

This series can be found on GitHub:

https://github.com/dmatlack/linux/tree/vfio/set_power_state/v2

This series fixes a race between closing a vfio-pci device and enabling SR-IOV on it, both of which can transition the device from D3hot to D0 without being serialized. On devices with NoSoftRst-, this can restore the PF's BARs after VFs have been enabled (triggering a WARN in pci_iov_update_resource()), or double-free vdev->pm_save.

This series also adds a selftest that exercises VFIO's handling of power state transitions requested by userspace through the PM capability, including a test case that reproduces the race.

This series applies on top of vfio/next (vfio-v7.4-rc1).

Series Overview
---------------

This series implements the following:

1. Hold memory_lock for all calls to vfio_pci_set_power_state() and assert that it is held, which fixes the race (patch 1).

2. Annotate the intentional lockless accesses to vdev->sriov_active with READ_ONCE() and WRITE_ONCE() (patch 2).

3. Skip vfio_pci_sriov_uapi_test instead of failing when SR-IOV cannot be configured on the PF, e.g. because vfio-pci's enable_sriov module parameter is not set (patches 3-4).

4. Add library helpers to find a PCI capability, read /sys/kernel/warn_count, and print kernel warnings from /dev/kmsg (patches 5-7).

5. Add vfio_pci_power_state_test, which covers closing a device in D3hot and reopening it, enabling SR-IOV while the PF is in D3hot, and racing a close in D3hot against enabling SR-IOV. The test fails if /sys/kernel/warn_count changes during a test case, and prints the warnings logged to /dev/kmsg to help debug the failure (patch 8).

Testing
-------

All patches were build-tested individually. The series was tested on x86_64 in a QEMU VM with an emulated Intel 82576 NIC (igb), which has NoSoftRst- and supports SR-IOV, bound to vfio-pci:

- Without patch 1 (KASAN kernel), close_in_d3hot_races_sriov_enable fails, hitting the pci_iov_update_resource() WARN on 9 or 10 of the test's 10 iterations.

- With the full series (KASAN and lockdep kernels), all VFIO selftests pass with no warnings, KASAN reports, or lockdep splats.

To run the new test (the SR-IOV test cases are skipped unless vfio_pci.enable_sriov=1):

  # tools/testing/selftests/vfio/scripts/setup.sh 0000:01:00.0
  # tools/testing/selftests/vfio/scripts/run.sh \
      tools/testing/selftests/vfio/vfio_pci_power_state_test

Changelog
---------

v2:
- Rename close_in_d3hot to reopen_after_close_in_d3hot and clarify that it does not verify the D0 transition during close, since that is not reliably observable from userspace (Alex)
- Print the kernel warnings logged to /dev/kmsg when the warn_count check fails, so the failure can be debugged from the test log (Alex)
- Drop the claim that memory_lock protects the sriov_active check from the comment in vfio_pci_set_power_state(), since it is cleared without memory_lock
- Skip the SR-IOV tests, including the existing vfio_pci_sriov_uapi_test, when SR-IOV cannot be configured on the PF (e.g. vfio_pci.enable_sriov=N) instead of failing
- Use kernel-doc for the new library helpers
- Clean up commit messages, comments, and the cover letter

v1: https://lore.kernel.org/kvm/[email protected]/

David Matlack (8):
vfio/pci: Hold memory_lock for all calls to vfio_pci_set_power_state() vfio/pci: Annotate lockless accesses to vdev->sriov_active
vfio: selftests: Add __sysfs_sriov_numvfs_set()
vfio: selftests: Skip vfio_pci_sriov_uapi_test if SR-IOV can't be
enabled
vfio: selftests: Add vfio_pci_find_capability()
vfio: selftests: Add sysfs_kernel_warn_count_get()
vfio: selftests: Add helpers to print kernel warnings from /dev/kmsg
vfio: selftests: Add test for device power state transitions

 drivers/vfio/pci/vfio_pci_core.c              |  16 +-
 tools/testing/selftests/vfio/Makefile         |   1 +
 .../selftests/vfio/lib/include/libvfio.h      |   1 +
 .../selftests/vfio/lib/include/libvfio/kmsg.h |   8 +
 .../vfio/lib/include/libvfio/sysfs.h          |   2 +
 .../lib/include/libvfio/vfio_pci_device.h     |   2 +
 tools/testing/selftests/vfio/lib/kmsg.c       | 134 +++++++++
 tools/testing/selftests/vfio/lib/libvfio.mk   |   1 +
 tools/testing/selftests/vfio/lib/sysfs.c      |  66 ++++-
 .../selftests/vfio/lib/vfio_pci_device.c      |  36 +++
 .../vfio/vfio_pci_power_state_test.c          | 256 ++++++++++++++++++
 .../selftests/vfio/vfio_pci_sriov_uapi_test.c |   8 +-
 12 files changed, 512 insertions(+), 19 deletions(-)
 create mode 100644 tools/testing/selftests/vfio/lib/include/libvfio/kmsg.h
 create mode 100644 tools/testing/selftests/vfio/lib/kmsg.c
 create mode 100644 tools/testing/selftests/vfio/vfio_pci_power_state_test.c

base-commit: 076d7e989cb69f6ded7d60f2a080eb909930061b

Signature · 1 line
2.56.1.708.g77570a6a1d-goog
author maintainer
#1 lore raw

PATCH v2 1/8 vfio/pci: Hold memory_lock for all calls to vfio_pci_set_power_state()

Hold vdev->memory_lock for write when calling vfio_pci_set_power_state() from vfio_pci_core_disable() and vfio_pci_core_register_device(), and assert that the lock is held for all power state transitions, not just transitions out of D0.

vfio_pci_core_disable() calls vfio_pci_set_power_state(vdev, PCI_D0) without holding memory_lock, which allows it to race with vfio_pci_core_sriov_configure() when userspace writes to sriov_numvfs while the device is being closed. If the user left the device in D3hot via a config space write, both threads can transition the device from D3hot to D0 concurrently. Since a config space write does not change the runtime PM status, the device stays RPM_ACTIVE, pm_runtime_resume() in the close path is a no-op, and vfio_pci_set_power_state() performs the D3hot->D0 transition.

For devices with vdev->needs_pm_restore set (NoSoftRst-), this can result in:

- vfio_pci_core_disable() restoring the PF's BARs after vfio_pci_core_sriov_configure() has enabled VFs, since the D3hot->D0 transition sleeps before restoring device state. This reproduces reliably with vfio_pci_power_state_test and triggers the following warning:

     vfio-pci 0000:01:00.0: can't update enabled VF BAR0 [mem 0xe0200000000-0xe020001ffff 64bit pref]
     WARNING: drivers/pci/iov.c:1062 at pci_iov_update_resource+0x170/0x3b0
     Call Trace:
      pci_update_resource+0x1a3/0x5b0
      __pci_set_power_state+0x597/0xb90
      vfio_pci_core_disable+0x13b/0x1180
      vfio_pci_core_close_device+0x144/0x260
      vfio_df_close+0xf2/0x210
      vfio_df_unbind_iommufd+0x89/0x140
      vfio_device_fops_release+0x85/0xf0
      __fput+0x2d6/0x720
      fput_close_sync+0xe8/0x180
      __x64_sys_close+0x7f/0xc0
      do_syscall_64+0xbc/0x320
      entry_SYSCALL_64_after_hwframe+0x76/0x7e

- Both threads observing pdev->current_state >= PCI_D3hot and calling pci_load_and_free_saved_state(pdev, &vdev->pm_save) concurrently, resulting in a double-free of vdev->pm_save.

vfio_pci_core_register_device() cannot race with other callers since the device is not yet registered and device_lock is held across probe, but take the lock there as well so that the lockdep assertion can be made unconditional and catch any future unlocked callers.

Taking memory_lock in these paths does not introduce any new lock dependencies. vfio_pci_core_disable() already acquires memory_lock under dev_set->lock via vfio_pci_runtime_pm_exit(), and vfio_pci_core_register_device(), like vfio_pci_core_sriov_configure(), runs under device_lock, which is already taken before memory_lock.

Fixes f4162eb1e2fc ("vfio/pci: Change the PF power state to D0 before enabling VFs")
---
 drivers/vfio/pci/vfio_pci_core.c | 14 +++++++++-----
 1 file changed, 9 insertions(+), 5 deletions(-)
drivers/vfio/pci/vfio_pci_core.c +9 -5
index fcbb2da860e2..4289ec9bc32f 100644
@@ -325,12 +325,12 @@ int vfio_pci_set_power_state(struct vfio_pci_core_device *vdev, pci_power_t stat
 	bool needs_restore = false, needs_save = false;
 	int ret;
 
+	/* Serialize power state changes and accesses to vdev->pm_save. */
+	lockdep_assert_held_write(&vdev->memory_lock);
+
 	/* Prevent changing power state for PFs with VFs enabled */
-	if (state > PCI_D0) {
-		lockdep_assert_held_write(&vdev->memory_lock);
-		if (vdev->sriov_active)
-			return -EBUSY;
-	}
+	if (state > PCI_D0 && vdev->sriov_active)
+		return -EBUSY;
 
 	if (vdev->needs_pm_restore) {
 		if (pdev->current_state < PCI_D3hot && state >= PCI_D3hot) {
@@ -699,7 +699,9 @@ void vfio_pci_core_disable(struct vfio_pci_core_device *vdev)
 	 * reset without restoring the original state (saved locally in
 	 * 'vdev->pm_save').
 	 */
+	down_write(&vdev->memory_lock);
 	vfio_pci_set_power_state(vdev, PCI_D0);
+	up_write(&vdev->memory_lock);
 
 	/* Stop the device from further DMA */
 	pci_clear_master(pdev);
@@ -2273,7 +2275,9 @@ int vfio_pci_core_register_device(struct vfio_pci_core_device *vdev)
 	 * be able to get to D3.  Therefore first do a D0 transition
 	 * before enabling runtime PM.
 	 */
+	down_write(&vdev->memory_lock);
 	vfio_pci_set_power_state(vdev, PCI_D0);
+	up_write(&vdev->memory_lock);
 
 	pm_runtime_allow(dev);
 	if (!vdev->disable_idle_d3)
Signature · 1 line
2.56.1.708.g77570a6a1d-goog
author maintainer
#2 lore raw

PATCH v2 2/8 vfio/pci: Annotate lockless accesses to vdev->sriov_active

In reply to David Matlack

Use READ_ONCE() and WRITE_ONCE() to annotate the read of vdev->sriov_active in vfio_pci_set_power_state() and the lockless write that clears it in vfio_pci_core_sriov_configure().

vfio_pci_core_sriov_configure() intentionally clears sriov_active without holding memory_lock, so the write can race with the read in vfio_pci_set_power_state() (which runs under memory_lock). The race is
benign: at worst it causes a spurious -EBUSY while the device stays in D0. Mark the accesses anyway to document the intentional data race and avoid KCSAN reports.

The write that sets sriov_active is done under memory_lock and is serialized against the lockless write by device_lock, so it does not need to be annotated.

Fixes 40ef3edf151e ("vfio/pci: Use a private flag to prevent power state change with VFs")
---
 drivers/vfio/pci/vfio_pci_core.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/vfio/pci/vfio_pci_core.c +2 -2
index 4289ec9bc32f..511dcb49d2f6 100644
@@ -329,7 +329,7 @@ int vfio_pci_set_power_state(struct vfio_pci_core_device *vdev, pci_power_t stat
 	lockdep_assert_held_write(&vdev->memory_lock);
 
 	/* Prevent changing power state for PFs with VFs enabled */
-	if (state > PCI_D0 && vdev->sriov_active)
+	if (state > PCI_D0 && READ_ONCE(vdev->sriov_active))
 		return -EBUSY;
 
 	if (vdev->needs_pm_restore) {
@@ -2393,7 +2393,7 @@ int vfio_pci_core_sriov_configure(struct vfio_pci_core_device *vdev,
 	 * state transition would at most result in an -EBUSY, leaving the
 	 * device in PCI_D0.
 	 */
-	vdev->sriov_active = false;
+	WRITE_ONCE(vdev->sriov_active, false);
 
 	mutex_lock(&vfio_pci_sriov_pfs_mutex);
 	list_del_init(&vdev->sriov_pfs_item);
Signature · 1 line
2.56.1.708.g77570a6a1d-goog
author maintainer
#3 lore raw

PATCH v2 3/8 vfio: selftests: Add __sysfs_sriov_numvfs_set()

In reply to David Matlack

Add __sysfs_sriov_numvfs_set(), which returns the error from writing to a PF's sriov_numvfs file instead of asserting that the write succeeds, and implement sysfs_sriov_numvfs_set() on top of it.

This will be used in subsequent commits to skip SR-IOV tests when the kernel returns -ENOENT, which means SR-IOV cannot be configured on the PF, e.g. because vfio-pci's enable_sriov module parameter is not set.

---
 .../vfio/lib/include/libvfio/sysfs.h          |  1 +
 tools/testing/selftests/vfio/lib/sysfs.c      | 38 +++++++++++++++----
 2 files changed, 31 insertions(+), 8 deletions(-)
tools/testing/selftests/vfio/lib/include/libvfio/sysfs.h +1 -0
index c9ab1ea8f5a9..ede114181cf8 100644
@@ -4,6 +4,7 @@
 
 int sysfs_sriov_totalvfs_get(const char *bdf);
 int sysfs_sriov_numvfs_get(const char *bdf);
+int __sysfs_sriov_numvfs_set(const char *bdf, int numvfs);
 void sysfs_sriov_numvfs_set(const char *bdf, int numvfs);
 char *sysfs_sriov_vf_bdf_get(const char *pf_bdf, int i);
 int sysfs_iommu_group_get(const char *bdf);
tools/testing/selftests/vfio/lib/sysfs.c +30 -8
index d88e348a1e0d..308da0526543 100644
@@ -56,17 +56,24 @@ static int sysfs_val_get_int(const char *component, const char *name,
 	return ret;
 }
 
-static void sysfs_val_set(const char *component, const char *name,
-			  const char *file, const char *val)
+static int sysfs_val_set(const char *component, const char *name,
+			 const char *file, const char *val)
 {
 	char path[PATH_MAX];
+	ssize_t ret;
 	int fd;
 
 	snprintf_assert(path, PATH_MAX, "/sys/bus/pci/%s/%s/%s", component, name, file);
 	VFIO_ASSERT_GT(fd = open(path, O_WRONLY), 0);
 
-	VFIO_ASSERT_EQ(write(fd, val, strlen(val)), strlen(val));
+	ret = write(fd, val, strlen(val));
+	if (ret < 0)
+		ret = -errno;
+	else
+		VFIO_ASSERT_EQ(ret, strlen(val));
+
 	VFIO_ASSERT_EQ(close(fd), 0);
+	return ret < 0 ? ret : 0;
 }
 
 static int sysfs_device_val_get(const char *bdf, const char *file)
@@ -74,17 +81,17 @@ static int sysfs_device_val_get(const char *bdf, const char *file)
 	return sysfs_val_get_int("devices", bdf, file);
 }
 
-static void sysfs_device_val_set(const char *bdf, const char *file, const char *val)
+static int sysfs_device_val_set(const char *bdf, const char *file, const char *val)
 {
-	sysfs_val_set("devices", bdf, file, val);
+	return sysfs_val_set("devices", bdf, file, val);
 }
 
-static void sysfs_device_val_set_int(const char *bdf, const char *file, int val)
+static int sysfs_device_val_set_int(const char *bdf, const char *file, int val)
 {
 	char val_str[32];
 
 	snprintf_assert(val_str, sizeof(val_str), "%d", val);
-	sysfs_device_val_set(bdf, file, val_str);
+	return sysfs_device_val_set(bdf, file, val_str);
 }
 
 int sysfs_sriov_totalvfs_get(const char *bdf)
@@ -97,9 +104,24 @@ int sysfs_sriov_numvfs_get(const char *bdf)
 	return sysfs_device_val_get(bdf, "sriov_numvfs");
 }
 
+/**
+ * __sysfs_sriov_numvfs_set() - Set the number of VFs enabled on a PF
+ * @bdf: BDF of the PF.
+ * @numvfs: Number of VFs to enable, or 0 to disable SR-IOV.
+ *
+ * Return: 0 on success, or a negative errno on failure. The kernel returns
+ * -ENOENT if SR-IOV cannot be configured on the PF, e.g. because its driver
+ * does not support configuring SR-IOV or has it disabled (vfio-pci unless its
+ * enable_sriov module parameter is set).
+ */
+int __sysfs_sriov_numvfs_set(const char *bdf, int numvfs)
+{
+	return sysfs_device_val_set_int(bdf, "sriov_numvfs", numvfs);
+}
+
 void sysfs_sriov_numvfs_set(const char *bdf, int numvfs)
 {
-	sysfs_device_val_set_int(bdf, "sriov_numvfs", numvfs);
+	VFIO_ASSERT_EQ(__sysfs_sriov_numvfs_set(bdf, numvfs), 0);
 }
 
 char *sysfs_sriov_vf_bdf_get(const char *pf_bdf, int i)
Signature · 1 line
2.56.1.708.g77570a6a1d-goog
author maintainer
#4 lore raw

PATCH v2 4/8 vfio: selftests: Skip vfio_pci_sriov_uapi_test if SR-IOV can't be enabled

In reply to David Matlack

Skip vfio_pci_sriov_uapi_test if writing to the PF's sriov_numvfs file fails with -ENOENT, which means SR-IOV cannot be configured on the PF.

vfio-pci returns -ENOENT unless its enable_sriov module parameter is set, which it is not by default, so the test currently fails during setup instead of skipping.

Fixes a6e4e726600d ("vfio: selftests: Add tests to validate SR-IOV UAPI")
---
 tools/testing/selftests/vfio/vfio_pci_sriov_uapi_test.c | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)
tools/testing/selftests/vfio/vfio_pci_sriov_uapi_test.c +7 -1
index f3a018b8699d..f27418fa5960 100644
@@ -210,6 +210,7 @@ static void vf_setup(void)
 {
 	char *vf_driver;
 	int nr_vfs;
+	int ret;
 
 	nr_vfs = sysfs_sriov_totalvfs_get(pf_bdf);
 	if (nr_vfs <= 0)
@@ -220,7 +221,12 @@ static void vf_setup(void)
 		ksft_exit_skip("SR-IOV already configured for the PF: %s\n", pf_bdf);
 
 	/* Create only one VF for testing */
-	sysfs_sriov_numvfs_set(pf_bdf, 1);
+	ret = __sysfs_sriov_numvfs_set(pf_bdf, 1);
+	if (ret == -ENOENT)
+		ksft_exit_skip("Cannot enable SR-IOV on %s (vfio-pci requires enable_sriov=1)\n",
+			       pf_bdf);
+
+	VFIO_ASSERT_EQ(ret, 0);
 
 	/*
 	 * Setup an exit handler to destroy the VF in case of failures
Signature · 1 line
2.56.1.708.g77570a6a1d-goog
author maintainer
#5 lore raw

PATCH v2 5/8 vfio: selftests: Add vfio_pci_find_capability()

In reply to David Matlack

Add vfio_pci_find_capability(), which walks a device's standard capability list like the kernel's pci_find_capability().

This will be used in a subsequent commit to locate the PM capability.

---
 .../lib/include/libvfio/vfio_pci_device.h     |  2 ++
 .../selftests/vfio/lib/vfio_pci_device.c      | 36 +++++++++++++++++++
 2 files changed, 38 insertions(+)
tools/testing/selftests/vfio/lib/include/libvfio/vfio_pci_device.h +2 -0
index 007adc322c1e..0fc03bb79448 100644
@@ -79,6 +79,8 @@ void vfio_pci_config_access(struct vfio_pci_device *device, bool write,
 #define vfio_pci_config_writew(_d, _o, _v) vfio_pci_config_write(_d, _o, _v, u16)
 #define vfio_pci_config_writel(_d, _o, _v) vfio_pci_config_write(_d, _o, _v, u32)
 
+u8 vfio_pci_find_capability(struct vfio_pci_device *device, u8 cap_id);
+
 static inline void vfio_pci_cmd_set(struct vfio_pci_device *device, u16 bits)
 {
 	u16 cmd = vfio_pci_config_readw(device, PCI_COMMAND);
tools/testing/selftests/vfio/lib/vfio_pci_device.c +36 -0
index 4063a0e2b3df..ee3f76e2b874 100644
@@ -260,6 +260,42 @@ void vfio_pci_config_access(struct vfio_pci_device *device, bool write,
 		       write ? "write to" : "read from", config);
 }
 
+/* Bound the capability list walk in case the list contains a loop. */
+#define PCI_FIND_CAP_TTL 48
+
+/**
+ * vfio_pci_find_capability() - Find a capability in a device's config space
+ * @device: Device to search.
+ * @cap_id: Capability ID to find (PCI_CAP_ID_*).
+ *
+ * Walk the device's standard capability list, like the kernel's
+ * pci_find_capability().
+ *
+ * Return: The config space offset of the first capability with ID @cap_id, or
+ * 0 if the device does not have the capability.
+ */
+u8 vfio_pci_find_capability(struct vfio_pci_device *device, u8 cap_id)
+{
+	int ttl = PCI_FIND_CAP_TTL;
+	u8 pos;
+
+	if (!(vfio_pci_config_readw(device, PCI_STATUS) & PCI_STATUS_CAP_LIST))
+		return 0;
+
+	pos = vfio_pci_config_readb(device, PCI_CAPABILITY_LIST);
+
+	while (pos >= PCI_STD_HEADER_SIZEOF && ttl--) {
+		pos &= ~3;
+
+		if (vfio_pci_config_readb(device, pos + PCI_CAP_LIST_ID) == cap_id)
+			return pos;
+
+		pos = vfio_pci_config_readb(device, pos + PCI_CAP_LIST_NEXT);
+	}
+
+	return 0;
+}
+
 int __vfio_pci_device_reset(struct vfio_pci_device *device)
 {
 	if (ioctl(device->fd, VFIO_DEVICE_RESET, NULL))
Signature · 1 line
2.56.1.708.g77570a6a1d-goog
author maintainer
#6 lore raw

PATCH v2 6/8 vfio: selftests: Add sysfs_kernel_warn_count_get()

In reply to David Matlack

Add a helper to read /sys/kernel/warn_count, which counts the number of warnings the kernel has encountered since boot. Return a negative value if the file does not exist, e.g. on kernels that predate it.

Factor the code to read an integer from a sysfs file out of sysfs_val_get_int() so that it can be shared by both helpers.

This will be used in a subsequent commit to detect kernel warnings triggered by tests.

---
 .../vfio/lib/include/libvfio/sysfs.h          |  1 +
 tools/testing/selftests/vfio/lib/sysfs.c      | 28 ++++++++++++++++---
 2 files changed, 25 insertions(+), 4 deletions(-)
tools/testing/selftests/vfio/lib/include/libvfio/sysfs.h +1 -0
index ede114181cf8..5ef515eda686 100644
@@ -9,5 +9,6 @@ void sysfs_sriov_numvfs_set(const char *bdf, int numvfs);
 char *sysfs_sriov_vf_bdf_get(const char *pf_bdf, int i);
 int sysfs_iommu_group_get(const char *bdf);
 char *sysfs_driver_get(const char *bdf);
+int sysfs_kernel_warn_count_get(void);
 
 #endif /* SELFTESTS_VFIO_LIB_INCLUDE_LIBVFIO_SYSFS_H */
tools/testing/selftests/vfio/lib/sysfs.c +24 -4
index 308da0526543..3845d81825e3 100644
@@ -33,15 +33,12 @@ static void readlink_base(const char *path, const char *data_fmt, void *out_data
 	VFIO_ASSERT_EQ(ret, 1);
 }
 
-static int sysfs_val_get_int(const char *component, const char *name,
-			     const char *file)
+static int sysfs_path_get_int(const char *path)
 {
-	char path[PATH_MAX];
 	char buf[32];
 	int ret;
 	int fd;
 
-	snprintf_assert(path, PATH_MAX, "/sys/bus/pci/%s/%s/%s", component, name, file);
 	fd = open(path, O_RDONLY);
 	if (fd < 0)
 		return fd;
@@ -56,6 +53,15 @@ static int sysfs_val_get_int(const char *component, const char *name,
 	return ret;
 }
 
+static int sysfs_val_get_int(const char *component, const char *name,
+			     const char *file)
+{
+	char path[PATH_MAX];
+
+	snprintf_assert(path, PATH_MAX, "/sys/bus/pci/%s/%s/%s", component, name, file);
+	return sysfs_path_get_int(path);
+}
+
 static int sysfs_val_set(const char *component, const char *name,
 			 const char *file, const char *val)
 {
@@ -170,3 +176,17 @@ char *sysfs_driver_get(const char *bdf)
 
 	return out_driver;
 }
+
+/**
+ * sysfs_kernel_warn_count_get() - Get the number of kernel warnings since boot
+ *
+ * Read /sys/kernel/warn_count, which counts the warnings (e.g. WARN(), KASAN,
+ * and UBSAN reports) the kernel has encountered since boot.
+ *
+ * Return: The number of warnings, or a negative value if the kernel does not
+ * support /sys/kernel/warn_count.
+ */
+int sysfs_kernel_warn_count_get(void)
+{
+	return sysfs_path_get_int("/sys/kernel/warn_count");
+}
Signature · 1 line
2.56.1.708.g77570a6a1d-goog
author maintainer
#7 lore raw

PATCH v2 7/8 vfio: selftests: Add helpers to print kernel warnings from /dev/kmsg

In reply to David Matlack

Add kmsg_open() and kmsg_print_warnings() to the library. kmsg_open() opens /dev/kmsg positioned after the last message in the kernel log, and kmsg_print_warnings() prints any kernel messages at KERN_WARNING or higher priority that have been logged since.

Bound how much kmsg_print_warnings() reads and prints, since other processes (e.g. tests running in parallel) may log a large number of messages, and note in the output when messages were lost because the kernel log wrapped. If /dev/kmsg cannot be opened, kmsg_open() prints a message saying that kernel warnings will not be printed, so that callers do not each need to.

This will be used in a subsequent commit to print kernel warnings to the test log when a test triggers one, so the failure can be debugged from the test log.

---
 .../selftests/vfio/lib/include/libvfio.h      |   1 +
 .../selftests/vfio/lib/include/libvfio/kmsg.h |   8 ++
 tools/testing/selftests/vfio/lib/kmsg.c       | 134 ++++++++++++++++++
 tools/testing/selftests/vfio/lib/libvfio.mk   |   1 +
 4 files changed, 144 insertions(+)
 create mode 100644 tools/testing/selftests/vfio/lib/include/libvfio/kmsg.h
 create mode 100644 tools/testing/selftests/vfio/lib/kmsg.c
tools/testing/selftests/vfio/lib/include/libvfio.h +1 -0
index 07862b470777..dca9a9af961e 100644
@@ -5,6 +5,7 @@
 #include <libvfio/assert.h>
 #include <libvfio/iommu.h>
 #include <libvfio/iova_allocator.h>
+#include <libvfio/kmsg.h>
 #include <libvfio/sysfs.h>
 #include <libvfio/vfio_pci_device.h>
 #include <libvfio/vfio_pci_driver.h>
tools/testing/selftests/vfio/lib/include/libvfio/kmsg.h new file +8 -0
new file mode 100644
index 000000000000..9b9e89129b8f
@@ -0,0 +1,8 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+#ifndef SELFTESTS_VFIO_LIB_INCLUDE_LIBVFIO_KMSG_H
+#define SELFTESTS_VFIO_LIB_INCLUDE_LIBVFIO_KMSG_H
+
+int kmsg_open(void);
+void kmsg_print_warnings(int fd);
+
+#endif /* SELFTESTS_VFIO_LIB_INCLUDE_LIBVFIO_KMSG_H */
tools/testing/selftests/vfio/lib/kmsg.c new file +134 -0

134 lines not on this page · raw message

tools/testing/selftests/vfio/lib/libvfio.mk +1 -0
index eff95828719e..f8af0c14c3d0 100644
@@ -5,6 +5,7 @@ LIBVFIO_SRCDIR := $(selfdir)/vfio/lib
 
 LIBVFIO_C := iommu.c
 LIBVFIO_C += iova_allocator.c
+LIBVFIO_C += kmsg.c
 LIBVFIO_C += libvfio.c
 LIBVFIO_C += sysfs.c
 LIBVFIO_C += vfio_pci_device.c
Signature · 1 line
2.56.1.708.g77570a6a1d-goog
author maintainer
#8 lore raw

PATCH v2 8/8 vfio: selftests: Add test for device power state transitions

In reply to David Matlack

Add vfio_pci_power_state_test to exercise VFIO's handling of device power state changes initiated by userspace through the PM capability in config space.

The test covers:

- Closing a device in D3hot and then reopening it, which must leave the device in D0.
- Enabling SR-IOV while the PF is in D3hot, which must return the PF to D0 and prevent it from leaving D0 while VFs are enabled.
- Racing a close of the device while it is in D3hot against enabling SR-IOV, both of which transition the device back to D0.

The SR-IOV tests are skipped if the device does not support SR-IOV, already has VFs enabled, or SR-IOV cannot be configured on it (e.g. vfio-pci with vfio_pci.enable_sriov=N). If an SR-IOV test case fails, any VFs it left enabled are disabled so they don't affect later test cases.

Fail the test if /sys/kernel/warn_count changes between fixture setup and teardown, since bugs such as the close vs. SR-IOV enable race show up only as kernel warnings. When that happens, print the warnings logged to /dev/kmsg during the test so that the failure can be debugged from the test log.

---
 tools/testing/selftests/vfio/Makefile         |   1 +
 .../vfio/vfio_pci_power_state_test.c          | 256 ++++++++++++++++++
 2 files changed, 257 insertions(+)
 create mode 100644 tools/testing/selftests/vfio/vfio_pci_power_state_test.c
tools/testing/selftests/vfio/Makefile +1 -0
index 2c32c48db509..e2b152528560 100644
@@ -12,6 +12,7 @@ TEST_GEN_PROGS += vfio_iommufd_setup_test
 TEST_GEN_PROGS += vfio_pci_device_test
 TEST_GEN_PROGS += vfio_pci_device_init_perf_test
 TEST_GEN_PROGS += vfio_pci_driver_test
+TEST_GEN_PROGS += vfio_pci_power_state_test
 TEST_GEN_PROGS += vfio_pci_sriov_uapi_test
 
 TEST_FILES += scripts/cleanup.sh
tools/testing/selftests/vfio/vfio_pci_power_state_test.c new file +256 -0

256 lines not on this page · raw message

Signature · 1 line
2.56.1.708.g77570a6a1d-goog