Skip to content

vfio/pci: Fix power state race with SR-IOV enablement

David Matlack kvm linux-kselftest lkml stable 2 versions – New

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v2 Latest8 patches 0 repliesNew
  2. v15 patches 4 repliesSuperseded

Changes in v2 from the cover letter

- Rename close_in_d3hot to reopen_after_close_in_d3hot and clarify that it does not verify the D0 transition during close, since that is not reliably observable from userspace (Alex)
- Print the kernel warnings logged to /dev/kmsg when the warn_count check fails, so the failure can be debugged from the test log (Alex)
- Drop the claim that memory_lock protects the sriov_active check from the comment in vfio_pci_set_power_state(), since it is cleared without memory_lock
- Skip the SR-IOV tests, including the existing vfio_pci_sriov_uapi_test, when SR-IOV cannot be configured on the PF (e.g. vfio_pci.enable_sriov=N) instead of failing
- Use kernel-doc for the new library helpers
- Clean up commit messages, comments, and the cover letter

Patches in v2 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/8vfio/pci: Hold memory_lock for all calls to vfio_pci_set_power_state()drivers/vfio/pci/vfio_pci_core.c+9 -50 replies
2/8vfio/pci: Annotate lockless accesses to vdev->sriov_activedrivers/vfio/pci/vfio_pci_core.c+2 -20 replies
3/8vfio: selftests: Add __sysfs_sriov_numvfs_set()tools/testing/selftests/vfio/lib/include/libvfio/sysfs.h tools/testing/selftests/vfio/lib/sysfs.c+31 -80 replies
4/8vfio: selftests: Skip vfio_pci_sriov_uapi_test if SR-IOV can't be enabledtools/testing/selftests/vfio/vfio_pci_sriov_uapi_test.c+7 -10 replies
5/8vfio: selftests: Add vfio_pci_find_capability()tools/testing/selftests/vfio/lib/include/libvfio/vfio_pci_device.h tools/testing/selftests/vfio/lib/vfio_pci_device.c+38 -00 replies
6/8vfio: selftests: Add sysfs_kernel_warn_count_get()tools/testing/selftests/vfio/lib/include/libvfio/sysfs.h tools/testing/selftests/vfio/lib/sysfs.c+25 -40 replies
7/8vfio: selftests: Add helpers to print kernel warnings from /dev/kmsgtools/testing/selftests/vfio/lib/include/libvfio.h tools/testing/selftests/vfio/lib/include/libvfio/kmsg.h tools/testing/selftests/vfio/lib/kmsg.c +1 more+144 -00 replies
8/8vfio: selftests: Add test for device power state transitionstools/testing/selftests/vfio/Makefile tools/testing/selftests/vfio/vfio_pci_power_state_test.c+257 -00 replies

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v2 yet.

Read the v2 thread

Cover letter [PATCH v2 0/8] vfio/pci: Fix power state race with SR-IOV enablement

This series can be found on GitHub:

https://github.com/dmatlack/linux/tree/vfio/set_power_state/v2

This series fixes a race between closing a vfio-pci device and enabling SR-IOV on it, both of which can transition the device from D3hot to D0 without being serialized. On devices with NoSoftRst-, this can restore the PF's BARs after VFs have been enabled (triggering a WARN in pci_iov_update_resource()), or double-free vdev->pm_save.

This series also adds a selftest that exercises VFIO's handling of power state transitions requested by userspace through the PM capability, including a test case that reproduces the race.

This series applies on top of vfio/next (vfio-v7.4-rc1).

Series Overview
---------------

This series implements the following:

1. Hold memory_lock for all calls to vfio_pci_set_power_state() and assert that it is held, which fixes the race (patch 1).

2. Annotate the intentional lockless accesses to vdev->sriov_active with READ_ONCE() and WRITE_ONCE() (patch 2).

3. Skip vfio_pci_sriov_uapi_test instead of failing when SR-IOV cannot be configured on the PF, e.g. because vfio-pci's enable_sriov module parameter is not set (patches 3-4).

4. Add library helpers to find a PCI capability, read /sys/kernel/warn_count, and print kernel warnings from /dev/kmsg (patches 5-7).

5. Add vfio_pci_power_state_test, which covers closing a device in D3hot and reopening it, enabling SR-IOV while the PF is in D3hot, and racing a close in D3hot against enabling SR-IOV. The test fails if /sys/kernel/warn_count changes during a test case, and prints the warnings logged to /dev/kmsg to help debug the failure (patch 8).

Testing
-------

All patches were build-tested individually. The series was tested on x86_64 in a QEMU VM with an emulated Intel 82576 NIC (igb), which has NoSoftRst- and supports SR-IOV, bound to vfio-pci:

- Without patch 1 (KASAN kernel), close_in_d3hot_races_sriov_enable fails, hitting the pci_iov_update_resource() WARN on 9 or 10 of the test's 10 iterations.

- With the full series (KASAN and lockdep kernels), all VFIO selftests pass with no warnings, KASAN reports, or lockdep splats.

To run the new test (the SR-IOV test cases are skipped unless vfio_pci.enable_sriov=1):

  # tools/testing/selftests/vfio/scripts/setup.sh 0000:01:00.0
  # tools/testing/selftests/vfio/scripts/run.sh \
      tools/testing/selftests/vfio/vfio_pci_power_state_test

Changelog
---------

v2:
- Rename close_in_d3hot to reopen_after_close_in_d3hot and clarify that it does not verify the D0 transition during close, since that is not reliably observable from userspace (Alex)
- Print the kernel warnings logged to /dev/kmsg when the warn_count check fails, so the failure can be debugged from the test log (Alex)
- Drop the claim that memory_lock protects the sriov_active check from the comment in vfio_pci_set_power_state(), since it is cleared without memory_lock
- Skip the SR-IOV tests, including the existing vfio_pci_sriov_uapi_test, when SR-IOV cannot be configured on the PF (e.g. vfio_pci.enable_sriov=N) instead of failing
- Use kernel-doc for the new library helpers
- Clean up commit messages, comments, and the cover letter

v1: https://lore.kernel.org/kvm/[email protected]/

David Matlack (8):
vfio/pci: Hold memory_lock for all calls to vfio_pci_set_power_state() vfio/pci: Annotate lockless accesses to vdev->sriov_active
vfio: selftests: Add __sysfs_sriov_numvfs_set()
vfio: selftests: Skip vfio_pci_sriov_uapi_test if SR-IOV can't be
enabled
vfio: selftests: Add vfio_pci_find_capability()
vfio: selftests: Add sysfs_kernel_warn_count_get()
vfio: selftests: Add helpers to print kernel warnings from /dev/kmsg
vfio: selftests: Add test for device power state transitions

 drivers/vfio/pci/vfio_pci_core.c              |  16 +-
 tools/testing/selftests/vfio/Makefile         |   1 +
 .../selftests/vfio/lib/include/libvfio.h      |   1 +
 .../selftests/vfio/lib/include/libvfio/kmsg.h |   8 +
 .../vfio/lib/include/libvfio/sysfs.h          |   2 +
 .../lib/include/libvfio/vfio_pci_device.h     |   2 +
 tools/testing/selftests/vfio/lib/kmsg.c       | 134 +++++++++
 tools/testing/selftests/vfio/lib/libvfio.mk   |   1 +
 tools/testing/selftests/vfio/lib/sysfs.c      |  66 ++++-
 .../selftests/vfio/lib/vfio_pci_device.c      |  36 +++
 .../vfio/vfio_pci_power_state_test.c          | 256 ++++++++++++++++++
 .../selftests/vfio/vfio_pci_sriov_uapi_test.c |   8 +-
 12 files changed, 512 insertions(+), 19 deletions(-)
 create mode 100644 tools/testing/selftests/vfio/lib/include/libvfio/kmsg.h
 create mode 100644 tools/testing/selftests/vfio/lib/kmsg.c
 create mode 100644 tools/testing/selftests/vfio/vfio_pci_power_state_test.c

base-commit: 076d7e989cb69f6ded7d60f2a080eb909930061b

Signature · 1 line
2.56.1.708.g77570a6a1d-goog