vfio/pci: Fix power state race with SR-IOV enablement
Subsystems (5)
David Matlack
maintainerv2 cover vfio/pci: Fix power state race with SR-IOV enablement
This series can be found on GitHub:
https://github.com/dmatlack/linux/tree/vfio/set_power_state/v2
This series fixes a race between closing a vfio-pci device and enabling SR-IOV on it, both of which can transition the device from D3hot to D0 without being serialized. On devices with NoSoftRst-, this can restore the PF's BARs after VFs have been enabled (triggering a WARN in pci_iov_update_resource()), or double-free vdev->pm_save.
This series also adds a selftest that exercises VFIO's handling of power state transitions requested by userspace through the PM capability, including a test case that reproduces the race.
This series applies on top of vfio/next (vfio-v7.4-rc1).
Series Overview
---------------
This series implements the following:
1. Hold memory_lock for all calls to vfio_pci_set_power_state() and assert that it is held, which fixes the race (patch 1).
2. Annotate the intentional lockless accesses to vdev->sriov_active with READ_ONCE() and WRITE_ONCE() (patch 2).
3. Skip vfio_pci_sriov_uapi_test instead of failing when SR-IOV cannot be configured on the PF, e.g. because vfio-pci's enable_sriov module parameter is not set (patches 3-4).
4. Add library helpers to find a PCI capability, read /sys/kernel/warn_count, and print kernel warnings from /dev/kmsg (patches 5-7).
5. Add vfio_pci_power_state_test, which covers closing a device in D3hot and reopening it, enabling SR-IOV while the PF is in D3hot, and racing a close in D3hot against enabling SR-IOV. The test fails if /sys/kernel/warn_count changes during a test case, and prints the warnings logged to /dev/kmsg to help debug the failure (patch 8).
Testing
-------
All patches were build-tested individually. The series was tested on x86_64 in a QEMU VM with an emulated Intel 82576 NIC (igb), which has NoSoftRst- and supports SR-IOV, bound to vfio-pci:
- Without patch 1 (KASAN kernel), close_in_d3hot_races_sriov_enable fails, hitting the pci_iov_update_resource() WARN on 9 or 10 of the test's 10 iterations.
- With the full series (KASAN and lockdep kernels), all VFIO selftests pass with no warnings, KASAN reports, or lockdep splats.
To run the new test (the SR-IOV test cases are skipped unless vfio_pci.enable_sriov=1):
# tools/testing/selftests/vfio/scripts/setup.sh 0000:01:00.0
# tools/testing/selftests/vfio/scripts/run.sh \
tools/testing/selftests/vfio/vfio_pci_power_state_testChangelog
---------
v2:
- Rename close_in_d3hot to reopen_after_close_in_d3hot and clarify that it does not verify the D0 transition during close, since that is not reliably observable from userspace (Alex)
- Print the kernel warnings logged to /dev/kmsg when the warn_count check fails, so the failure can be debugged from the test log (Alex)
- Drop the claim that memory_lock protects the sriov_active check from the comment in vfio_pci_set_power_state(), since it is cleared without memory_lock
- Skip the SR-IOV tests, including the existing vfio_pci_sriov_uapi_test, when SR-IOV cannot be configured on the PF (e.g. vfio_pci.enable_sriov=N) instead of failing
- Use kernel-doc for the new library helpers
- Clean up commit messages, comments, and the cover letter
v1: https://lore.kernel.org/kvm/[email protected]/
David Matlack (8):
vfio/pci: Hold memory_lock for all calls to vfio_pci_set_power_state() vfio/pci: Annotate lockless accesses to vdev->sriov_active
vfio: selftests: Add __sysfs_sriov_numvfs_set()
vfio: selftests: Skip vfio_pci_sriov_uapi_test if SR-IOV can't be
enabled
vfio: selftests: Add vfio_pci_find_capability()
vfio: selftests: Add sysfs_kernel_warn_count_get()
vfio: selftests: Add helpers to print kernel warnings from /dev/kmsg
vfio: selftests: Add test for device power state transitions
drivers/vfio/pci/vfio_pci_core.c | 16 +- tools/testing/selftests/vfio/Makefile | 1 + .../selftests/vfio/lib/include/libvfio.h | 1 + .../selftests/vfio/lib/include/libvfio/kmsg.h | 8 + .../vfio/lib/include/libvfio/sysfs.h | 2 + .../lib/include/libvfio/vfio_pci_device.h | 2 + tools/testing/selftests/vfio/lib/kmsg.c | 134 +++++++++ tools/testing/selftests/vfio/lib/libvfio.mk | 1 + tools/testing/selftests/vfio/lib/sysfs.c | 66 ++++- .../selftests/vfio/lib/vfio_pci_device.c | 36 +++ .../vfio/vfio_pci_power_state_test.c | 256 ++++++++++++++++++ .../selftests/vfio/vfio_pci_sriov_uapi_test.c | 8 +- 12 files changed, 512 insertions(+), 19 deletions(-) create mode 100644 tools/testing/selftests/vfio/lib/include/libvfio/kmsg.h create mode 100644 tools/testing/selftests/vfio/lib/kmsg.c create mode 100644 tools/testing/selftests/vfio/vfio_pci_power_state_test.c
base-commit: 076d7e989cb69f6ded7d60f2a080eb909930061b
Signature · 1 line
2.56.1.708.g77570a6a1d-goog
David Matlack
maintainerPATCH v2 1/8 vfio/pci: Hold memory_lock for all calls to vfio_pci_set_power_state()
Hold vdev->memory_lock for write when calling vfio_pci_set_power_state() from vfio_pci_core_disable() and vfio_pci_core_register_device(), and assert that the lock is held for all power state transitions, not just transitions out of D0.
vfio_pci_core_disable() calls vfio_pci_set_power_state(vdev, PCI_D0) without holding memory_lock, which allows it to race with vfio_pci_core_sriov_configure() when userspace writes to sriov_numvfs while the device is being closed. If the user left the device in D3hot via a config space write, both threads can transition the device from D3hot to D0 concurrently. Since a config space write does not change the runtime PM status, the device stays RPM_ACTIVE, pm_runtime_resume() in the close path is a no-op, and vfio_pci_set_power_state() performs the D3hot->D0 transition.
For devices with vdev->needs_pm_restore set (NoSoftRst-), this can result in:
- vfio_pci_core_disable() restoring the PF's BARs after vfio_pci_core_sriov_configure() has enabled VFs, since the D3hot->D0 transition sleeps before restoring device state. This reproduces reliably with vfio_pci_power_state_test and triggers the following warning:
vfio-pci 0000:01:00.0: can't update enabled VF BAR0 [mem 0xe0200000000-0xe020001ffff 64bit pref]
WARNING: drivers/pci/iov.c:1062 at pci_iov_update_resource+0x170/0x3b0
Call Trace:
pci_update_resource+0x1a3/0x5b0
__pci_set_power_state+0x597/0xb90
vfio_pci_core_disable+0x13b/0x1180
vfio_pci_core_close_device+0x144/0x260
vfio_df_close+0xf2/0x210
vfio_df_unbind_iommufd+0x89/0x140
vfio_device_fops_release+0x85/0xf0
__fput+0x2d6/0x720
fput_close_sync+0xe8/0x180
__x64_sys_close+0x7f/0xc0
do_syscall_64+0xbc/0x320
entry_SYSCALL_64_after_hwframe+0x76/0x7e - Both threads observing pdev->current_state >= PCI_D3hot and calling pci_load_and_free_saved_state(pdev, &vdev->pm_save) concurrently, resulting in a double-free of vdev->pm_save.
vfio_pci_core_register_device() cannot race with other callers since the device is not yet registered and device_lock is held across probe, but take the lock there as well so that the lockdep assertion can be made unconditional and catch any future unlocked callers.
Taking memory_lock in these paths does not introduce any new lock dependencies. vfio_pci_core_disable() already acquires memory_lock under dev_set->lock via vfio_pci_runtime_pm_exit(), and vfio_pci_core_register_device(), like vfio_pci_core_sriov_configure(), runs under device_lock, which is already taken before memory_lock.
--- drivers/vfio/pci/vfio_pci_core.c | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-)
drivers/vfio/pci/vfio_pci_core.c +9 -5
@@ -325,12 +325,12 @@ int vfio_pci_set_power_state(struct vfio_pci_core_device *vdev, pci_power_t statbool needs_restore = false, needs_save = false;int ret; +/* Serialize power state changes and accesses to vdev->pm_save. */ +lockdep_assert_held_write (&vdev->memory_lock); +/* Prevent changing power state for PFs with VFs enabled */ -if (state >PCI_D0 ) { -lockdep_assert_held_write (&vdev->memory_lock); -if (vdev->sriov_active) -return -EBUSY ; - } +if (state >PCI_D0 && vdev->sriov_active) +return -EBUSY ;if (vdev->needs_pm_restore) {if (pdev->current_state < PCI_D3hot && state >= PCI_D3hot) {
@@ -699,7 +699,9 @@ void vfio_pci_core_disable(struct vfio_pci_core_device *vdev)* reset without restoring the original state (saved locally in * 'vdev->pm_save'). */ +down_write (&vdev->memory_lock);vfio_pci_set_power_state (vdev,PCI_D0 ); +up_write (&vdev->memory_lock);/* Stop the device from further DMA */ pci_clear_master (pdev);
@@ -2273,7 +2275,9 @@ int vfio_pci_core_register_device(struct vfio_pci_core_device *vdev)* be able to get to D3. Therefore first do a D0 transition * before enabling runtime PM. */ +down_write (&vdev->memory_lock);vfio_pci_set_power_state (vdev,PCI_D0 ); +up_write (&vdev->memory_lock);pm_runtime_allow (dev);if (!vdev->disable_idle_d3)
Signature · 1 line
2.56.1.708.g77570a6a1d-goog
David Matlack
maintainerPATCH v2 2/8 vfio/pci: Annotate lockless accesses to vdev->sriov_active
In reply to David MatlackUse READ_ONCE() and WRITE_ONCE() to annotate the read of vdev->sriov_active in vfio_pci_set_power_state() and the lockless write that clears it in vfio_pci_core_sriov_configure().
vfio_pci_core_sriov_configure() intentionally clears sriov_active without holding memory_lock, so the write can race with the read in vfio_pci_set_power_state() (which runs under memory_lock). The race is
benign: at worst it causes a spurious -EBUSY while the device stays in D0. Mark the accesses anyway to document the intentional data race and avoid KCSAN reports.
The write that sets sriov_active is done under memory_lock and is serialized against the lockless write by device_lock, so it does not need to be annotated.
--- drivers/vfio/pci/vfio_pci_core.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/vfio/pci/vfio_pci_core.c +2 -2
@@ -329,7 +329,7 @@ int vfio_pci_set_power_state(struct vfio_pci_core_device *vdev, pci_power_t statlockdep_assert_held_write (&vdev->memory_lock);/* Prevent changing power state for PFs with VFs enabled */ -if (state >PCI_D0 && vdev->sriov_active) +if (state >PCI_D0 &&READ_ONCE (vdev->sriov_active))return -EBUSY ;if (vdev->needs_pm_restore) {
@@ -2393,7 +2393,7 @@ int vfio_pci_core_sriov_configure(struct vfio_pci_core_device *vdev,* state transition would at most result in an -EBUSY, leaving the * device in PCI_D0. */ - vdev->sriov_active = false; +WRITE_ONCE (vdev->sriov_active, false);mutex_lock (&vfio_pci_sriov_pfs_mutex);list_del_init (&vdev->sriov_pfs_item);
Signature · 1 line
2.56.1.708.g77570a6a1d-goog
David Matlack
maintainerPATCH v2 3/8 vfio: selftests: Add __sysfs_sriov_numvfs_set()
In reply to David MatlackAdd __sysfs_sriov_numvfs_set(), which returns the error from writing to a PF's sriov_numvfs file instead of asserting that the write succeeds, and implement sysfs_sriov_numvfs_set() on top of it.
This will be used in subsequent commits to skip SR-IOV tests when the kernel returns -ENOENT, which means SR-IOV cannot be configured on the PF, e.g. because vfio-pci's enable_sriov module parameter is not set.
--- .../vfio/lib/include/libvfio/sysfs.h | 1 + tools/testing/selftests/vfio/lib/sysfs.c | 38 +++++++++++++++---- 2 files changed, 31 insertions(+), 8 deletions(-)
tools/testing/selftests/vfio/lib/include/libvfio/sysfs.h +1 -0
@@ -4,6 +4,7 @@int sysfs_sriov_totalvfs_get (const char *bdf);int sysfs_sriov_numvfs_get (const char *bdf); +int __sysfs_sriov_numvfs_set (const char *bdf,int numvfs);void sysfs_sriov_numvfs_set (const char *bdf,int numvfs);char *sysfs_sriov_vf_bdf_get (const char *pf_bdf,int i);int sysfs_iommu_group_get (const char *bdf);
tools/testing/selftests/vfio/lib/sysfs.c +30 -8
@@ -56,17 +56,24 @@ static int sysfs_val_get_int(const char *component, const char *name,return ret; } -static void sysfs_val_set (const char *component,const char *name, -const char *file,const char *val) +static int sysfs_val_set (const char *component,const char *name, +const char *file,const char *val) {char path[PATH_MAX ]; +ssize_t ret;int fd;snprintf_assert (path,PATH_MAX ,"/sys/bus/pci/%s/%s/%s" , component, name, file);VFIO_ASSERT_GT (fd =open (path,O_WRONLY ),0 ); -VFIO_ASSERT_EQ (write (fd, val,strlen (val)),strlen (val)); + ret =write (fd, val,strlen (val)); +if (ret <0 ) + ret = -errno; +else +VFIO_ASSERT_EQ (ret,strlen (val)); +VFIO_ASSERT_EQ (close (fd),0 ); +return ret <0 ? ret :0 ; }static int sysfs_device_val_get (const char *bdf,const char *file)
@@ -74,17 +81,17 @@ static int sysfs_device_val_get(const char *bdf, const char *file)return sysfs_val_get_int ("devices" , bdf, file); } -static void sysfs_device_val_set (const char *bdf,const char *file,const char *val) +static int sysfs_device_val_set (const char *bdf,const char *file,const char *val) { -sysfs_val_set ("devices" , bdf, file, val); +return sysfs_val_set ("devices" , bdf, file, val); } -static void sysfs_device_val_set_int (const char *bdf,const char *file,int val) +static int sysfs_device_val_set_int (const char *bdf,const char *file,int val) {char val_str[32 ];snprintf_assert (val_str,sizeof (val_str),"%d" , val); -sysfs_device_val_set (bdf, file, val_str); +return sysfs_device_val_set (bdf, file, val_str); }int sysfs_sriov_totalvfs_get (const char *bdf)
@@ -97,9 +104,24 @@ int sysfs_sriov_numvfs_get(const char *bdf)return sysfs_device_val_get (bdf,"sriov_numvfs" ); } +/** +* __sysfs_sriov_numvfs_set() - Set the number of VFs enabled on a PF +* @bdf: BDF of the PF. +* @numvfs: Number of VFs to enable, or 0 to disable SR-IOV. +* +* Return: 0 on success, or a negative errno on failure. The kernel returns +* -ENOENT if SR-IOV cannot be configured on the PF, e.g. because its driver +* does not support configuring SR-IOV or has it disabled (vfio-pci unless its +* enable_sriov module parameter is set). +*/ +int __sysfs_sriov_numvfs_set (const char *bdf,int numvfs) +{ +return sysfs_device_val_set_int (bdf,"sriov_numvfs" , numvfs); +} +void sysfs_sriov_numvfs_set (const char *bdf,int numvfs) { -sysfs_device_val_set_int (bdf,"sriov_numvfs" , numvfs); +VFIO_ASSERT_EQ (__sysfs_sriov_numvfs_set (bdf, numvfs),0 ); }char *sysfs_sriov_vf_bdf_get (const char *pf_bdf,int i)
Signature · 1 line
2.56.1.708.g77570a6a1d-goog
David Matlack
maintainerPATCH v2 4/8 vfio: selftests: Skip vfio_pci_sriov_uapi_test if SR-IOV can't be enabled
In reply to David MatlackSkip vfio_pci_sriov_uapi_test if writing to the PF's sriov_numvfs file fails with -ENOENT, which means SR-IOV cannot be configured on the PF.
vfio-pci returns -ENOENT unless its enable_sriov module parameter is set, which it is not by default, so the test currently fails during setup instead of skipping.
--- tools/testing/selftests/vfio/vfio_pci_sriov_uapi_test.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-)
tools/testing/selftests/vfio/vfio_pci_sriov_uapi_test.c +7 -1
@@ -210,6 +210,7 @@ static void vf_setup(void) {char *vf_driver;int nr_vfs; +int ret; nr_vfs =sysfs_sriov_totalvfs_get (pf_bdf);if (nr_vfs <=0 )
@@ -220,7 +221,12 @@ static void vf_setup(void)ksft_exit_skip ("SR-IOV already configured for the PF: %s\n" , pf_bdf);/* Create only one VF for testing */ -sysfs_sriov_numvfs_set (pf_bdf,1 ); + ret =__sysfs_sriov_numvfs_set (pf_bdf,1 ); +if (ret == -ENOENT ) +ksft_exit_skip ("Cannot enable SR-IOV on %s (vfio-pci requires enable_sriov=1)\n" , + pf_bdf); + +VFIO_ASSERT_EQ (ret,0 );/* * Setup an exit handler to destroy the VF in case of failures
Signature · 1 line
2.56.1.708.g77570a6a1d-goog
David Matlack
maintainerPATCH v2 5/8 vfio: selftests: Add vfio_pci_find_capability()
In reply to David MatlackAdd vfio_pci_find_capability(), which walks a device's standard capability list like the kernel's pci_find_capability().
This will be used in a subsequent commit to locate the PM capability.
--- .../lib/include/libvfio/vfio_pci_device.h | 2 ++ .../selftests/vfio/lib/vfio_pci_device.c | 36 +++++++++++++++++++ 2 files changed, 38 insertions(+)
tools/testing/selftests/vfio/lib/include/libvfio/vfio_pci_device.h +2 -0
@@ -79,6 +79,8 @@ void vfio_pci_config_access(struct vfio_pci_device *device, bool write,#define vfio_pci_config_writew (_d, _o, _v) vfio_pci_config_write(_d, _o, _v, u16)#define vfio_pci_config_writel (_d, _o, _v) vfio_pci_config_write(_d, _o, _v, u32) +u8 vfio_pci_find_capability (struct vfio_pci_device *device,u8 cap_id); +static inline void vfio_pci_cmd_set (struct vfio_pci_device *device,u16 bits) {u16 cmd =vfio_pci_config_readw (device,PCI_COMMAND );
tools/testing/selftests/vfio/lib/vfio_pci_device.c +36 -0
@@ -260,6 +260,42 @@ void vfio_pci_config_access(struct vfio_pci_device *device, bool write, write ? "write to" : "read from", config); } +/* Bound the capability list walk in case the list contains a loop. */ +#define PCI_FIND_CAP_TTL 48 + +/** +* vfio_pci_find_capability() - Find a capability in a device's config space +* @device: Device to search. +* @cap_id: Capability ID to find (PCI_CAP_ID_*). +* +* Walk the device's standard capability list, like the kernel's +* pci_find_capability(). +* +* Return: The config space offset of the first capability with ID @cap_id, or +* 0 if the device does not have the capability. +*/ +u8 vfio_pci_find_capability (struct vfio_pci_device *device,u8 cap_id) +{ +int ttl =PCI_FIND_CAP_TTL ; +u8 pos; + +if (!(vfio_pci_config_readw (device,PCI_STATUS ) &PCI_STATUS_CAP_LIST )) +return 0 ; + + pos =vfio_pci_config_readb (device,PCI_CAPABILITY_LIST ); + +while (pos >=PCI_STD_HEADER_SIZEOF && ttl--) { + pos &= ~3 ; + +if (vfio_pci_config_readb (device, pos +PCI_CAP_LIST_ID ) == cap_id) +return pos; + + pos =vfio_pci_config_readb (device, pos +PCI_CAP_LIST_NEXT ); + } + +return 0 ; +} +int __vfio_pci_device_reset (struct vfio_pci_device *device) {if (ioctl (device->fd,VFIO_DEVICE_RESET ,NULL ))
Signature · 1 line
2.56.1.708.g77570a6a1d-goog
David Matlack
maintainerPATCH v2 6/8 vfio: selftests: Add sysfs_kernel_warn_count_get()
In reply to David MatlackAdd a helper to read /sys/kernel/warn_count, which counts the number of warnings the kernel has encountered since boot. Return a negative value if the file does not exist, e.g. on kernels that predate it.
Factor the code to read an integer from a sysfs file out of sysfs_val_get_int() so that it can be shared by both helpers.
This will be used in a subsequent commit to detect kernel warnings triggered by tests.
--- .../vfio/lib/include/libvfio/sysfs.h | 1 + tools/testing/selftests/vfio/lib/sysfs.c | 28 ++++++++++++++++--- 2 files changed, 25 insertions(+), 4 deletions(-)
tools/testing/selftests/vfio/lib/include/libvfio/sysfs.h +1 -0
@@ -9,5 +9,6 @@ void sysfs_sriov_numvfs_set(const char *bdf, int numvfs);char *sysfs_sriov_vf_bdf_get (const char *pf_bdf,int i);int sysfs_iommu_group_get (const char *bdf);char *sysfs_driver_get (const char *bdf); +int sysfs_kernel_warn_count_get (void );#endif /* SELFTESTS_VFIO_LIB_INCLUDE_LIBVFIO_SYSFS_H */
tools/testing/selftests/vfio/lib/sysfs.c +24 -4
@@ -33,15 +33,12 @@ static void readlink_base(const char *path, const char *data_fmt, void *out_dataVFIO_ASSERT_EQ (ret,1 ); } -static int sysfs_val_get_int (const char *component,const char *name, -const char *file) +static int sysfs_path_get_int (const char *path) { -char path[PATH_MAX ];char buf[32 ];int ret;int fd; -snprintf_assert (path,PATH_MAX ,"/sys/bus/pci/%s/%s/%s" , component, name, file); fd =open (path,O_RDONLY );if (fd <0 )return fd;
@@ -56,6 +53,15 @@ static int sysfs_val_get_int(const char *component, const char *name,return ret; } +static int sysfs_val_get_int (const char *component,const char *name, +const char *file) +{ +char path[PATH_MAX ]; + +snprintf_assert (path,PATH_MAX ,"/sys/bus/pci/%s/%s/%s" , component, name, file); +return sysfs_path_get_int (path); +} +static int sysfs_val_set (const char *component,const char *name,const char *file,const char *val) {
@@ -170,3 +176,17 @@ char *sysfs_driver_get(const char *bdf)return out_driver; } + +/** +* sysfs_kernel_warn_count_get() - Get the number of kernel warnings since boot +* +* Read /sys/kernel/warn_count, which counts the warnings (e.g. WARN(), KASAN, +* and UBSAN reports) the kernel has encountered since boot. +* +* Return: The number of warnings, or a negative value if the kernel does not +* support /sys/kernel/warn_count. +*/ +int sysfs_kernel_warn_count_get (void ) +{ +return sysfs_path_get_int ("/sys/kernel/warn_count" ); +}
Signature · 1 line
2.56.1.708.g77570a6a1d-goog
David Matlack
maintainerPATCH v2 7/8 vfio: selftests: Add helpers to print kernel warnings from /dev/kmsg
In reply to David MatlackAdd kmsg_open() and kmsg_print_warnings() to the library. kmsg_open() opens /dev/kmsg positioned after the last message in the kernel log, and kmsg_print_warnings() prints any kernel messages at KERN_WARNING or higher priority that have been logged since.
Bound how much kmsg_print_warnings() reads and prints, since other processes (e.g. tests running in parallel) may log a large number of messages, and note in the output when messages were lost because the kernel log wrapped. If /dev/kmsg cannot be opened, kmsg_open() prints a message saying that kernel warnings will not be printed, so that callers do not each need to.
This will be used in a subsequent commit to print kernel warnings to the test log when a test triggers one, so the failure can be debugged from the test log.
--- .../selftests/vfio/lib/include/libvfio.h | 1 + .../selftests/vfio/lib/include/libvfio/kmsg.h | 8 ++ tools/testing/selftests/vfio/lib/kmsg.c | 134 ++++++++++++++++++ tools/testing/selftests/vfio/lib/libvfio.mk | 1 + 4 files changed, 144 insertions(+) create mode 100644 tools/testing/selftests/vfio/lib/include/libvfio/kmsg.h create mode 100644 tools/testing/selftests/vfio/lib/kmsg.c
tools/testing/selftests/vfio/lib/include/libvfio.h +1 -0
@@ -5,6 +5,7 @@#include <libvfio/assert.h> #include <libvfio/iommu.h> #include <libvfio/iova_allocator.h> +#include <libvfio/kmsg.h> #include <libvfio/sysfs.h> #include <libvfio/vfio_pci_device.h> #include <libvfio/vfio_pci_driver.h>
tools/testing/selftests/vfio/lib/include/libvfio/kmsg.h new file +8 -0
@@ -0,0 +1,8 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#ifndef SELFTESTS_VFIO_LIB_INCLUDE_LIBVFIO_KMSG_H +#define SELFTESTS_VFIO_LIB_INCLUDE_LIBVFIO_KMSG_H + +int kmsg_open (void ); +void kmsg_print_warnings (int fd); + +#endif /* SELFTESTS_VFIO_LIB_INCLUDE_LIBVFIO_KMSG_H */
tools/testing/selftests/vfio/lib/kmsg.c new file +134 -0
134 lines not on this page · raw message
tools/testing/selftests/vfio/lib/libvfio.mk +1 -0
@@ -5,6 +5,7 @@ LIBVFIO_SRCDIR := $(selfdir)/vfio/lib LIBVFIO_C := iommu.c LIBVFIO_C += iova_allocator.c +LIBVFIO_C += kmsg.c LIBVFIO_C += libvfio.c LIBVFIO_C += sysfs.c LIBVFIO_C += vfio_pci_device.c
Signature · 1 line
2.56.1.708.g77570a6a1d-goog
David Matlack
maintainerPATCH v2 8/8 vfio: selftests: Add test for device power state transitions
In reply to David MatlackAdd vfio_pci_power_state_test to exercise VFIO's handling of device power state changes initiated by userspace through the PM capability in config space.
The test covers:
- Closing a device in D3hot and then reopening it, which must leave the device in D0.
- Enabling SR-IOV while the PF is in D3hot, which must return the PF to D0 and prevent it from leaving D0 while VFs are enabled.
- Racing a close of the device while it is in D3hot against enabling SR-IOV, both of which transition the device back to D0.
The SR-IOV tests are skipped if the device does not support SR-IOV, already has VFs enabled, or SR-IOV cannot be configured on it (e.g. vfio-pci with vfio_pci.enable_sriov=N). If an SR-IOV test case fails, any VFs it left enabled are disabled so they don't affect later test cases.
Fail the test if /sys/kernel/warn_count changes between fixture setup and teardown, since bugs such as the close vs. SR-IOV enable race show up only as kernel warnings. When that happens, print the warnings logged to /dev/kmsg during the test so that the failure can be debugged from the test log.
--- tools/testing/selftests/vfio/Makefile | 1 + .../vfio/vfio_pci_power_state_test.c | 256 ++++++++++++++++++ 2 files changed, 257 insertions(+) create mode 100644 tools/testing/selftests/vfio/vfio_pci_power_state_test.c
tools/testing/selftests/vfio/Makefile +1 -0
@@ -12,6 +12,7 @@ TEST_GEN_PROGS += vfio_iommufd_setup_test TEST_GEN_PROGS += vfio_pci_device_test TEST_GEN_PROGS += vfio_pci_device_init_perf_test TEST_GEN_PROGS += vfio_pci_driver_test +TEST_GEN_PROGS += vfio_pci_power_state_test TEST_GEN_PROGS += vfio_pci_sriov_uapi_test TEST_FILES += scripts/cleanup.sh
tools/testing/selftests/vfio/vfio_pci_power_state_test.c new file +256 -0
256 lines not on this page · raw message
Signature · 1 line
2.56.1.708.g77570a6a1d-goog