vfio/pci: Fix power state race with SR-IOV enablement
Get series with b4
b4 am [email protected]Versions newest first · linked by author and title
- v2 Latest8 patches 0 repliesNew
- v15 patches 4 repliesSuperseded
Changes in v2 from the cover letter
- Rename close_in_d3hot to reopen_after_close_in_d3hot and clarify that it does not verify the D0 transition during close, since that is not reliably observable from userspace (Alex)
- Print the kernel warnings logged to /dev/kmsg when the warn_count check fails, so the failure can be debugged from the test log (Alex)
- Drop the claim that memory_lock protects the sriov_active check from the comment in vfio_pci_set_power_state(), since it is cleared without memory_lock
- Skip the SR-IOV tests, including the existing vfio_pci_sriov_uapi_test, when SR-IOV cannot be configured on the PF (e.g. vfio_pci.enable_sriov=N) instead of failing
- Use kernel-doc for the new library helpers
- Clean up commit messages, comments, and the cover letter
Patches in v2 No applies recorded yet
| # | Subject and files | Size | Replies | Applied |
|---|---|---|---|---|
| 1/8 | vfio/pci: Hold memory_lock for all calls to vfio_pci_set_power_state()drivers/ | +9 | 0 replies | |
| 2/8 | vfio/pci: Annotate lockless accesses to vdev->sriov_activedrivers/ | +2 | 0 replies | |
| 3/8 | vfio: selftests: Add __sysfs_sriov_numvfs_set()tools/ tools/ | +31 | 0 replies | |
| 4/8 | vfio: selftests: Skip vfio_pci_sriov_uapi_test if SR-IOV can't be enabledtools/ | +7 | 0 replies | |
| 5/8 | vfio: selftests: Add vfio_pci_find_capability()tools/ tools/ | +38 | 0 replies | |
| 6/8 | vfio: selftests: Add sysfs_kernel_warn_count_get()tools/ tools/ | +25 | 0 replies | |
| 7/8 | vfio: selftests: Add helpers to print kernel warnings from /dev/kmsgtools/ tools/ tools/ +1 more | +144 | 0 replies | |
| 8/8 | vfio: selftests: Add test for device power state transitionstools/ tools/ | +257 | 0 replies |
Reviews trailers per patch
No Reviewed-by, Acked-by or Tested-by trailers for v2 yet.
Cover letter [PATCH v2 0/8] vfio/pci: Fix power state race with SR-IOV enablement
This series can be found on GitHub:
https://github.com/dmatlack/linux/tree/vfio/set_power_state/v2
This series fixes a race between closing a vfio-pci device and enabling SR-IOV on it, both of which can transition the device from D3hot to D0 without being serialized. On devices with NoSoftRst-, this can restore the PF's BARs after VFs have been enabled (triggering a WARN in pci_iov_update_resource()), or double-free vdev->pm_save.
This series also adds a selftest that exercises VFIO's handling of power state transitions requested by userspace through the PM capability, including a test case that reproduces the race.
This series applies on top of vfio/next (vfio-v7.4-rc1).
Series Overview
---------------
This series implements the following:
1. Hold memory_lock for all calls to vfio_pci_set_power_state() and assert that it is held, which fixes the race (patch 1).
2. Annotate the intentional lockless accesses to vdev->sriov_active with READ_ONCE() and WRITE_ONCE() (patch 2).
3. Skip vfio_pci_sriov_uapi_test instead of failing when SR-IOV cannot be configured on the PF, e.g. because vfio-pci's enable_sriov module parameter is not set (patches 3-4).
4. Add library helpers to find a PCI capability, read /sys/kernel/warn_count, and print kernel warnings from /dev/kmsg (patches 5-7).
5. Add vfio_pci_power_state_test, which covers closing a device in D3hot and reopening it, enabling SR-IOV while the PF is in D3hot, and racing a close in D3hot against enabling SR-IOV. The test fails if /sys/kernel/warn_count changes during a test case, and prints the warnings logged to /dev/kmsg to help debug the failure (patch 8).
Testing
-------
All patches were build-tested individually. The series was tested on x86_64 in a QEMU VM with an emulated Intel 82576 NIC (igb), which has NoSoftRst- and supports SR-IOV, bound to vfio-pci:
- Without patch 1 (KASAN kernel), close_in_d3hot_races_sriov_enable fails, hitting the pci_iov_update_resource() WARN on 9 or 10 of the test's 10 iterations.
- With the full series (KASAN and lockdep kernels), all VFIO selftests pass with no warnings, KASAN reports, or lockdep splats.
To run the new test (the SR-IOV test cases are skipped unless vfio_pci.enable_sriov=1):
# tools/testing/selftests/vfio/scripts/setup.sh 0000:01:00.0
# tools/testing/selftests/vfio/scripts/run.sh \
tools/testing/selftests/vfio/vfio_pci_power_state_testChangelog
---------
v2:
- Rename close_in_d3hot to reopen_after_close_in_d3hot and clarify that it does not verify the D0 transition during close, since that is not reliably observable from userspace (Alex)
- Print the kernel warnings logged to /dev/kmsg when the warn_count check fails, so the failure can be debugged from the test log (Alex)
- Drop the claim that memory_lock protects the sriov_active check from the comment in vfio_pci_set_power_state(), since it is cleared without memory_lock
- Skip the SR-IOV tests, including the existing vfio_pci_sriov_uapi_test, when SR-IOV cannot be configured on the PF (e.g. vfio_pci.enable_sriov=N) instead of failing
- Use kernel-doc for the new library helpers
- Clean up commit messages, comments, and the cover letter
v1: https://lore.kernel.org/kvm/[email protected]/
David Matlack (8):
vfio/pci: Hold memory_lock for all calls to vfio_pci_set_power_state() vfio/pci: Annotate lockless accesses to vdev->sriov_active
vfio: selftests: Add __sysfs_sriov_numvfs_set()
vfio: selftests: Skip vfio_pci_sriov_uapi_test if SR-IOV can't be
enabled
vfio: selftests: Add vfio_pci_find_capability()
vfio: selftests: Add sysfs_kernel_warn_count_get()
vfio: selftests: Add helpers to print kernel warnings from /dev/kmsg
vfio: selftests: Add test for device power state transitions
drivers/vfio/pci/vfio_pci_core.c | 16 +- tools/testing/selftests/vfio/Makefile | 1 + .../selftests/vfio/lib/include/libvfio.h | 1 + .../selftests/vfio/lib/include/libvfio/kmsg.h | 8 + .../vfio/lib/include/libvfio/sysfs.h | 2 + .../lib/include/libvfio/vfio_pci_device.h | 2 + tools/testing/selftests/vfio/lib/kmsg.c | 134 +++++++++ tools/testing/selftests/vfio/lib/libvfio.mk | 1 + tools/testing/selftests/vfio/lib/sysfs.c | 66 ++++- .../selftests/vfio/lib/vfio_pci_device.c | 36 +++ .../vfio/vfio_pci_power_state_test.c | 256 ++++++++++++++++++ .../selftests/vfio/vfio_pci_sriov_uapi_test.c | 8 +- 12 files changed, 512 insertions(+), 19 deletions(-) create mode 100644 tools/testing/selftests/vfio/lib/include/libvfio/kmsg.h create mode 100644 tools/testing/selftests/vfio/lib/kmsg.c create mode 100644 tools/testing/selftests/vfio/vfio_pci_power_state_test.c
base-commit: 076d7e989cb69f6ded7d60f2a080eb909930061b
Signature · 1 line
2.56.1.708.g77570a6a1d-goog