KVM: selftests: Link with VFIO selftests lib and test device interrupts
v2 cover KVM: selftests: Link with VFIO selftests lib and test device interrupts
I am taking over development of this series from David Matlack.
v1: https://lore.kernel.org/kvm/[email protected]/
This series is based on the kvm-upstream/queue branch.
This series introduces vfio_pci_irq_test, which exercises the delivery of VFIO device interrupts (MSI/MSI-X) to guest vCPUs, and links the VFIO selftests library into KVM selftests to enable testing of VFIO-KVM interactions.
The test requires a PCI device bound to vfio-pci. It can be invoked by passing the BDF the VFIO selftests setup script, and then running the test with the device BDF:
$ ./tools/testing/selftests/vfio/scripts/setup.sh 0000:6a:01.0 $ tools/testing/selftests/kvm/vfio_pci_irq_test 0000:6a:01.0
This test only supports x86. Testing physical device interrupts (-d argument) is only supported for Intel because there are no VFIO selftest drivers for AMD devices.
The test can be run with following command-line arguments allow for broad coverage of the interrupt delivery path:
-a: Random IRQ Affinity. Randomly affinitizes the device IRQ to different
host CPUs to verify stable delivery during interrupt steering changes.
-b: Block vCPUs. Causes vCPUs to HLT instead of spinning, verifying that
Posted-interrupt wakeup (PIW) correctly kicks blocked vCPUs.
-d: Device MSI Triggers. Uses the physical device to trigger MSIs instead
of eventfd emulation (requires a supported device driver).
-e: Clear GSI Routes. Periodically destroys/recreates KVM's GSI routing
table to verify handling of dynamic IRQ updates.
-i: IRQ Count. Sets the number of interrupts to generate (default 1000). -n: NMI Delivery. Routes interrupts as NMIs into the guest to verify the
VFIO-NMI delivery path.
-p: vCPU Pinning. Migrates vCPUs to random physical CPUs to verify that
posted interrupts follow the vCPU across host cores.
-t: Timeout. Configurable timeout in seconds to wait for an interrupt. -v: vCPU Count. Distributes interrupts across multiple vCPUs via
round-robin routing.
-x: xAPIC Mode. Forces legacy xAPIC mode to verify compatibility.
Changelog
---------
v2:
- Split the monolithic test into a series of logical patches (Sean)
- Move READ_FROM_GUEST and WRITE_TO_GUEST macros to kvm_util.h (Sean)
- Add gettid, sched_setaffinity, and sched_getaffinity to kvm_syscalls.h (Sean)
- Extract /proc/interrupts parsing logic into lib/irq_util.c (Sean)
- Add -t command-line option for configurable interrupt timeout (Sean)
- Improve assertion messages with specific vCPU failure context (Sean)
- Update guest loop to use cpu_relax() when not blocking (Sean)
- Add KVM_RANDOM_SEED support to kvm_selftest_init for reproducibility (Sean)
- Randomize the default IRQ vector and GSI (Sean)
- Ensure blocked vCPUs are cleaned up correctly after HLT (Sean)
- Use kvm_mmap() instead of mmap() (Sean)
v1: https://lore.kernel.org/kvm/[email protected]/
David Matlack (11):
KVM: selftests: Build and link sefltests/vfio/lib into KVM selftests
KVM: selftests: Add vfio_pci_irq_test
KVM: selftests: Add support for random host IRQ affinity
KVM: selftests: Allow blocking vCPUs via HLT
KVM: selftests: Add support for physical device MSI triggers
KVM: selftests: Add option to clear GSI routes
KVM: selftests: Make test IRQ count configurable
KVM: selftests: Add support for NMI delivery
KVM: selftests: Add support for vCPU pinning
KVM: selftests: Support testing with multiple vCPUs
KVM: selftests: Add xAPIC mode support
Josh Hilke (3):
KVM: selftests: Add helper functions for IRQ testing
KVM: selftests: Reproduce tests that rely on randomization
KVM: selftests: Make vfio_pci_irq_test timeout configurable
tools/testing/selftests/kvm/Makefile.kvm | 7 +- .../testing/selftests/kvm/include/irq_util.h | 11 + .../selftests/kvm/include/kvm_syscalls.h | 7 + .../testing/selftests/kvm/include/kvm_util.h | 11 + tools/testing/selftests/kvm/lib/irq_util.c | 84 ++++ tools/testing/selftests/kvm/lib/kvm_util.c | 11 +- .../testing/selftests/kvm/vfio_pci_irq_test.c | 439 ++++++++++++++++++ 7 files changed, 568 insertions(+), 2 deletions(-) create mode 100644 tools/testing/selftests/kvm/include/irq_util.h create mode 100644 tools/testing/selftests/kvm/lib/irq_util.c create mode 100644 tools/testing/selftests/kvm/vfio_pci_irq_test.c
Signature · 1 line
2.53.0.1118.gaef5881109-goog
PATCH v2 1/14 KVM: selftests: Build and link sefltests/vfio/lib into KVM selftests
From: David Matlack <[email protected]>
Include libvfio.mk into the KVM selftests Makefile and link it into all KVM selftests by adding it to LIBKVM_OBJS.
Note that KVM selftests build their own copy of sefltests/vfio/lib and the resulting object files are placed in $(OUTPUT)/lib. This allows the KVM and VFIO selftests to apply different CFLAGS when building without conflicting with each other.
--- tools/testing/selftests/kvm/Makefile.kvm | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
tools/testing/selftests/kvm/Makefile.kvm +4 -1
@@ -252,6 +252,7 @@ OVERRIDE_TARGETS = 1 # which causes the environment variable to override the makefile). include ../lib.mk include ../cgroup/lib/libcgroup.mk +include ../vfio/lib/libvfio.mk INSTALL_HDR_PATH = $(top_srcdir)/usr LINUX_HDR_PATH = $(INSTALL_HDR_PATH)/include/
@@ -306,7 +307,9 @@ LIBKVM_S := $(filter %.S,$(LIBKVM)) LIBKVM_C_OBJ := $(patsubst %.c, $(OUTPUT)/%.o, $(LIBKVM_C)) LIBKVM_S_OBJ := $(patsubst %.S, $(OUTPUT)/%.o, $(LIBKVM_S)) LIBKVM_STRING_OBJ := $(patsubst %.c, $(OUTPUT)/%.o, $(LIBKVM_STRING)) -LIBKVM_OBJS = $(LIBKVM_C_OBJ) $(LIBKVM_S_OBJ) $(LIBKVM_STRING_OBJ) $(LIBCGROUP_O) +LIBKVM_OBJS = $(LIBKVM_C_OBJ) $(LIBKVM_S_OBJ) $(LIBKVM_STRING_OBJ) +LIBKVM_OBJS += $(LIBCGROUP_O) +LIBKVM_OBJS += $(LIBVFIO_O) SPLIT_TEST_GEN_PROGS := $(patsubst %, $(OUTPUT)/%, $(SPLIT_TESTS)) SPLIT_TEST_GEN_OBJ := $(patsubst %, $(OUTPUT)/$(ARCH)/%.o, $(SPLIT_TESTS))
Signature · 1 line
2.53.0.1118.gaef5881109-goog
Sean Christopherson
reviewerRe: [PATCH v2 01/14] KVM: selftests: Build and link sefltests/vfio/lib into KVM selftests
Quoted 5 lines · Josh Hilke
From: David Matlack <[email protected]>
Include libvfio.mk into the KVM selftests
Makefileand link it into all KVM selftests by adding it toLIBKVM_OBJS.
Quoted 8 lines
Note that KVM selftests build their own copy of sefltests/vfio/lib and the resulting object files are placed in $(OUTPUT)/lib. This allows the KVM and VFIO selftests to apply different CFLAGS when building without conflicting with each other.
Cc Alex Williamson <[email protected]>Signed-off-by David Matlack <[email protected]>Change-Id Ib76374497945025fff60f77e6c72308e900b33f0
Please drop gerrit's garbage before posting.
Re: [PATCH v2 01/14] KVM: selftests: Build and link sefltests/vfio/lib into KVM selftests
Quoted 12 lines · Sean Christopherson
Quoted 10 lines · Josh Hilke
From: David Matlack <[email protected]>
Include libvfio.mk into the KVM selftests
Makefileand link it into all KVM selftests by adding it toLIBKVM_OBJS.Note that KVM selftests build their own copy of sefltests/vfio/lib and the resulting object files are placed in $(OUTPUT)/lib. This allows the KVM and VFIO selftests to apply different CFLAGS when building without conflicting with each other.
Quoted 5 lines
Quoted 3 lines
Cc Alex Williamson <[email protected]>Signed-off-by David Matlack <[email protected]>Change-Id Ib76374497945025fff60f77e6c72308e900b33f0Please drop gerrit's garbage before posting.
Ugh sorry, *face palm*.
PATCH v2 2/14 KVM: selftests: Add helper functions for IRQ testing
In reply to Josh HilkeIntroduce a test library to get IRQ numbers and counts. Add utility functions to synchronize global variables between the host and guest to track interrupt reception and readiness.
--- tools/testing/selftests/kvm/Makefile.kvm | 1 + .../testing/selftests/kvm/include/irq_util.h | 11 +++ .../testing/selftests/kvm/include/kvm_util.h | 11 +++ tools/testing/selftests/kvm/lib/irq_util.c | 84 +++++++++++++++++++ 4 files changed, 107 insertions(+) create mode 100644 tools/testing/selftests/kvm/include/irq_util.h create mode 100644 tools/testing/selftests/kvm/lib/irq_util.c
tools/testing/selftests/kvm/Makefile.kvm +1 -0
@@ -11,6 +11,7 @@ LIBKVM += lib/kvm_util.c LIBKVM += lib/lru_gen_util.c LIBKVM += lib/memstress.c LIBKVM += lib/guest_sprintf.c +LIBKVM += lib/irq_util.c LIBKVM += lib/rbtree.c LIBKVM += lib/sparsebit.c LIBKVM += lib/test_util.c
tools/testing/selftests/kvm/include/irq_util.h new file +11 -0
@@ -0,0 +1,11 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#ifndef SELFTEST_KVM_PROC_UTIL_H +#define SELFTEST_KVM_PROC_UTIL_H + +#include <stdint.h> + +int get_irq_number (const char *device_bdf,int msi); +uint64_t get_irq_count (int irq); +uint64_t get_irq_count_by_name (const char *name); + +#endif /* SELFTEST_KVM_PROC_UTIL_H */
tools/testing/selftests/kvm/include/kvm_util.h +11 -0
@@ -1144,6 +1144,17 @@ vm_adjust_num_guest_pages(enum vm_guest_mode mode, unsigned int num_guest_pages)memcpy (&(g), _p,sizeof (g)); \ }) +#define READ_FROM_GUEST (_vm, _variable) ({ \ + sync_global_from_guest(_vm, _variable); \ + READ_ONCE(_variable); \ +}) + +#define WRITE_TO_GUEST (_vm, _variable, _value) do { \ + WRITE_ONCE(_variable, _value); \ + sync_global_to_guest(_vm, _variable); \ +} while (0) + +/* * Write a global value, but only in the VM's (guest's) domain. Primarily used * for "globals" that hold per-VM values (VMs always duplicate code and global
tools/testing/selftests/kvm/lib/irq_util.c new file +84 -0
84 lines not on this page · raw message
Signature · 1 line
2.53.0.1118.gaef5881109-goog
Sean Christopherson
reviewerRe: [PATCH v2 02/14] KVM: selftests: Add helper functions for IRQ testing
Quoted 6 lines · Josh Hilke
tools/testing/selftests/kvm/include/irq_util.h new file +0
-0
tools/testing/selftests/kvm/include/irq_util.h line 9
Quoted 9 lines
tools/testing/selftests/kvm/include/irq_util.h new file +9
-0@@ -0,0 +1,11 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#ifndef SELFTEST_KVM_PROC_UTIL_H +#define SELFTEST_KVM_PROC_UTIL_H + +#include <stdint.h> + +int get_irq_number (const char *device_bdf,int msi); +uint64_t get_irq_count (int irq); +uint64_t get_irq_count_by_name (const char *name);
These are far too generic. I'm also not convinced these should exist at all. I'll add more context in the next patch.
Quoted 8 lines
tools/testing/selftests/kvm/include/irq_util.h new file +1
-0+#endif /* SELFTEST_KVM_PROC_UTIL_H */ tools/testing/selftests/kvm/include/kvm_util.h +0
-0@@ -1144,6 +1144,17 @@ vm_adjust_num_guest_pages(enum vm_guest_mode mode, unsigned int num_guest_pages)memcpy (&(g), _p,sizeof (g)); \ })
tools/testing/selftests/kvm/include/kvm_util.h vm_adjust_num_guest_pages(enum vm_guest_mode mode, unsigned int num_guest_pages) · line 1155
Quoted 10 lines
tools/testing/selftests/kvm/include/kvm_util.h +9
-0+#define READ_FROM_GUEST (_vm, _variable) ({ \ + sync_global_from_guest(_vm, _variable); \ + READ_ONCE(_variable); \ +}) + +#define WRITE_TO_GUEST (_vm, _variable, _value) do { \ + WRITE_ONCE(_variable, _value); \ + sync_global_to_guest(_vm, _variable); \ +} while (0)
These belong in a separate patch. I also think we should convert as many users as possible when they're introduced.
We might want a better name than WRITE_TO_GUEST(), because that suggests it _just_ writes to guest, but it also writes to the host value. Maybe SYNC_FROM_GUEST_AND_READ() and WRITE_AND_SYNC_TO_GUEST()? Stupidly verbose, but IMO it doesn't look too weird in the code.
tools/testing/selftests/kvm/include/kvm_util.h vm_adjust_num_guest_pages(enum vm_guest_mode mode, unsigned int num_guest_pages) · line 1157
Quoted 2 lines
tools/testing/selftests/kvm/include/kvm_util.h +2
-0+ +
Unnecessary newline.
Quoted 3 lines
tools/testing/selftests/kvm/include/kvm_util.h +0
-0/* * Write a global value, but only in the VM's (guest's) domain. Primarily used * for "globals" that hold per-VM values (VMs always duplicate code and global
Re: [PATCH v2 02/14] KVM: selftests: Add helper functions for IRQ testing
Quoted 12 lines · Sean Christopherson
Quoted 10 lines · Josh Hilke
tools/testing/selftests/kvm/include/irq_util.h new file +4
-0@@ -0,0 +1,11 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#ifndef SELFTEST_KVM_PROC_UTIL_H +#define SELFTEST_KVM_PROC_UTIL_H +
Quoted 8 lines
tools/testing/selftests/kvm/include/irq_util.h line 9
Quoted 5 lines
tools/testing/selftests/kvm/include/irq_util.h new file +5
-0+#include <stdint.h> + +int get_irq_number (const char *device_bdf,int msi); +uint64_t get_irq_count (int irq); +uint64_t get_irq_count_by_name (const char *name);These are far too generic. I'm also not convinced these should exist at all. I'll add more context in the next patch.
Ack. Responded to your comments on this in patch 3.
Quoted 23 lines
Quoted 9 lines
tools/testing/selftests/kvm/include/irq_util.h new file +1
-0+#endif /* SELFTEST_KVM_PROC_UTIL_H */ tools/testing/selftests/kvm/include/kvm_util.h +0
-0@@ -1144,6 +1144,17 @@ vm_adjust_num_guest_pages(enum vm_guest_mode mode, unsigned int num_guest_pages)memcpy (&(g), _p,sizeof (g)); \ })tools/testing/selftests/kvm/include/kvm_util.h vm_adjust_num_guest_pages(enum vm_guest_mode mode, unsigned int num_guest_pages) · line 1155
Quoted 10 lines
tools/testing/selftests/kvm/include/kvm_util.h +9
-0+#define READ_FROM_GUEST (_vm, _variable) ({ \ + sync_global_from_guest(_vm, _variable); \ + READ_ONCE(_variable); \ +}) + +#define WRITE_TO_GUEST (_vm, _variable, _value) do { \ + WRITE_ONCE(_variable, _value); \ + sync_global_to_guest(_vm, _variable); \ +} while (0)These belong in a separate patch. I also think we should convert as many users as possible when they're introduced.
Quoted 4 lines
We might want a better name than
WRITE_TO_GUEST(), because that suggests it _just_ writes to guest, but it also writes to the host value. MaybeSYNC_FROM_GUEST_AND_READ()andWRITE_AND_SYNC_TO_GUEST()? Stupidly verbose, but IMO it doesn't look too weird in the code.
Ok, I'll put this in a separate patch and update the macro names in v3.
Quoted 5 lines
tools/testing/selftests/kvm/include/kvm_util.h vm_adjust_num_guest_pages(enum vm_guest_mode mode, unsigned int num_guest_pages) · line 1157
Quoted 3 lines
tools/testing/selftests/kvm/include/kvm_util.h +2
-0+ +Unnecessary newline.
Will remove in v3.
Quoted 4 lines
Quoted 4 lines
tools/testing/selftests/kvm/include/kvm_util.h +0
-0/* * Write a global value, but only in the VM's (guest's) domain. Primarily used * for "globals" that hold per-VM values (VMs always duplicate code and global
PATCH v2 3/14 KVM: selftests: Add vfio_pci_irq_test
In reply to Josh HilkeFrom: David Matlack <[email protected]>
Add the initial implementation of vfio_pci_irq_test, which routes and delivers an MSI from a vfio-pci device into a guest. This establishes the core test infrastructure, including the vCPU thread loop and the basic IRQ triggering mechanism using irqfd.
--- tools/testing/selftests/kvm/Makefile.kvm | 1 + .../testing/selftests/kvm/vfio_pci_irq_test.c | 217 ++++++++++++++++++ 2 files changed, 218 insertions(+) create mode 100644 tools/testing/selftests/kvm/vfio_pci_irq_test.c
tools/testing/selftests/kvm/Makefile.kvm +1 -0
@@ -156,6 +156,7 @@ TEST_GEN_PROGS_x86 += rseq_test TEST_GEN_PROGS_x86 += steal_time TEST_GEN_PROGS_x86 += system_counter_offset_test TEST_GEN_PROGS_x86 += pre_fault_memory_test +TEST_GEN_PROGS_x86 += vfio_pci_irq_test # Compiled outputs used by test targets TEST_GEN_PROGS_EXTENDED_x86 += x86/nx_huge_pages_test
tools/testing/selftests/kvm/vfio_pci_irq_test.c new file +217 -0
217 lines not on this page · raw message
Signature · 1 line
2.53.0.1118.gaef5881109-goog
Sean Christopherson
reviewerRe: [PATCH v2 03/14] KVM: selftests: Add vfio_pci_irq_test
Quoted 6 lines · Josh Hilke
tools/testing/selftests/kvm/vfio_pci_irq_test.c new file +6
-0+int main (int argc,char **argv) +{ +/* +* Pick a random vector and a random GSI to use for device IRQ. +* +* Pick an IRQ vector in range [32, UINT8_MAX]. Min value is 32 because
tools/testing/selftests/kvm/vfio_pci_irq_test.c line 113
Quoted 10 lines
tools/testing/selftests/kvm/vfio_pci_irq_test.c new file +10
-0+* Linux/x86 reserves vectors 0-31 for exceptions and architecture +* defined NMIs and interrupts. +* +* Pick a GSI in range [24, KVM_MAX_IRQ_ROUTES - 1]. The min value is 24 +* because KVM reserves GSIs 0-15 for legacy ISA IRQs and 16-23 only go +* to the IOAPIC. The max is KVM_MAX_IRQ_ROUTES - 1, because +* KVM_MAX_IRQ_ROUTES is exclusive. +*/ +u32 gsi =24 +rand () % (KVM_MAX_IRQ_ROUTES -1 -24 ); +u8 vector =32 +rand () % (UINT8_MAX -32 );
Ahh, now I see why you included "Reproduce tests that rely on randomization" in this series.
I think I'd prefer to tweak guest_random_state() to drop the "guest" in favor of "kvm", and then use that framework in host code as well. Then you wouldn't need to open code the % fun.
Quoted 24 lines
tools/testing/selftests/kvm/vfio_pci_irq_test.c new file +24
-0+/* Test configuration (overridable by command line flags). */ +int nr_irqs =1000 ; +int nr_vcpus =1 ; + +struct kvm_vcpu *vcpus[KVM_MAX_VCPUS ]; +pthread_t vcpu_threads[KVM_MAX_VCPUS ]; +u64 irq_count, pin_count, piw_count; +struct vfio_pci_device *device; +struct iommu *iommu; +const char *device_bdf; +int i, j, c, msi, irq; +struct kvm_vm *vm; + + device_bdf =vfio_selftests_get_bdf (&argc, argv); + +while ((c =getopt (argc, argv,"h" )) !=-1 ) { +switch (c) { +case 'h' : +default : +help (argv[0 ]); + } + } + + vm =vm_create_with_vcpus (nr_vcpus, guest_code, vcpus);
tools/testing/selftests/kvm/vfio_pci_irq_test.c line 148
Quoted 10 lines
tools/testing/selftests/kvm/vfio_pci_irq_test.c new file +10
-0+vm_install_exception_handler (vm, vector, guest_irq_handler); + + iommu =iommu_init (default_iommu_mode); + device =vfio_pci_device_init (device_bdf, iommu); + msi =setup_msi (device); + irq =get_irq_number (device_bdf, msi); + + irq_count =get_irq_count (irq); + pin_count =get_irq_count_by_name ("PIN:" ); + piw_count =get_irq_count_by_name ("PIW:" );
I'm *very* skeptical of blindling printing PIN/PIW. They're Intel specific and are global to the entire system. Outside of debugging specific issues, I don't see them providing much value, while on the other hand, they could easily confuse readers. And unlike the IRQ counts, the data won't be lost when the test exits.
As for the get_irq_number() and get_irq_count(), those need to be much more clearly scoped to /proc/interrupts. E.g. "irq" in KVM context most often refers to the guest IRQ vector, not the host Linux make-believe IRQ number.
tools/testing/selftests/kvm/vfio_pci_irq_test.c line 151
Quoted 2 lines
tools/testing/selftests/kvm/vfio_pci_irq_test.c new file +2
-0+ printf("%s %s MSI-X[%d] (IRQ-%d) %d times\n", + "Notifying the eventfd for",
Where's the rest of this line? Oh, this is the first %s. LOL, that's ridiculous.
printf("Notifying the eventfd for BDF %s, MSI-X[%d] (IRQ-%d) %d times\n",
device_bdf, msi, irq, nr_irqs);Quoted 6 lines
tools/testing/selftests/kvm/vfio_pci_irq_test.c new file +6
-0+ device_bdf, msi, irq, nr_irqs); + +kvm_assign_irqfd (vm, gsi, device->msi_eventfds[msi]); + +for (i =0 ; i < nr_vcpus; i++) +pthread_create (&vcpu_threads[i],NULL , vcpu_thread_main, vcpus[i]);
tools/testing/selftests/kvm/vfio_pci_irq_test.c line 167
Quoted 10 lines
tools/testing/selftests/kvm/vfio_pci_irq_test.c new file +10
-0+ +for (i =0 ; i < nr_vcpus; i++) { +struct kvm_vcpu *vcpu = vcpus[i]; + +while (!READ_FROM_GUEST (vm, guest_ready_for_irqs[vcpu->id])) +continue ; + } + +/* Set a consistent seed so that test are repeatable. */ +srand (0 );
And using a pRNG instead of rand() should make this unnecessary.
Quoted 15 lines
tools/testing/selftests/kvm/vfio_pci_irq_test.c new file +15
-0+for (i =0 ; i < nr_irqs; i++) { +struct kvm_vcpu *vcpu = vcpus[i % nr_vcpus]; +struct timespec start; + +kvm_route_msi (vm, gsi, vcpu, vector); + +for (j =0 ; j < nr_vcpus; j++) { +TEST_ASSERT ( + !READ_FROM_GUEST (vm, guest_received_irq[vcpu->id]), +"IRQ flag for vCPU %d not clear prior to test" , + vcpu->id); + } + +send_msi (device, msi); +
tools/testing/selftests/kvm/vfio_pci_irq_test.c line 193
Quoted 10 lines
tools/testing/selftests/kvm/vfio_pci_irq_test.c new file +10
-0+clock_gettime (CLOCK_MONOTONIC , &start); +for (;;) { +if (READ_FROM_GUEST (vm, guest_received_irq[vcpu->id])) +break ; + +if (timespec_to_ns (timespec_elapsed (start)) >TIMEOUT_NS ) { +printf ("Timeout waiting for interrupt!\n" ); +printf (" vCPU: %d\n" , vcpu->id); + +TEST_FAIL ("vCPU never received IRQ!\n" );
Which vCPU? What BDF+MSI? Some of that is available in earlier messages, but failure messages should be standalone.
tools/testing/selftests/kvm/vfio_pci_irq_test.c line 202
Quoted 9 lines
tools/testing/selftests/kvm/vfio_pci_irq_test.c new file +9
-0+ } + } + +WRITE_TO_GUEST (vm, guest_received_irq[vcpu->id], false); + } + +WRITE_TO_GUEST (vm, done, true); + +for (i =0 ; i < nr_vcpus; i++) {
Unnecessary curly braces.
Quoted 18 lines
tools/testing/selftests/kvm/vfio_pci_irq_test.c new file +15
-0+pthread_join (vcpu_threads[i],NULL ); + } + +printf ("Host interrupts handled:\n" ); +printf (" IRQ-%d: %lu\n" , irq,get_irq_count (irq) - irq_count); +printf (" Posted-interrupt notification events: %lu\n" , +get_irq_count_by_name ("PIN:" ) - pin_count); +printf (" Posted-interrupt wakeup events: %lu\n" , +get_irq_count_by_name ("PIW:" ) - piw_count); + +vfio_pci_device_cleanup (device); +iommu_cleanup (iommu); + +return 0 ; +}Signature · 1 line
2.53.0.1118.gaef5881109-goog
Re: [PATCH v2 03/14] KVM: selftests: Add vfio_pci_irq_test
Quoted 21 lines · Sean Christopherson
Quoted 6 lines · Josh Hilke
tools/testing/selftests/kvm/vfio_pci_irq_test.c new file +6
-0+int main (int argc,char **argv) +{ +/* +* Pick a random vector and a random GSI to use for device IRQ. +* +* Pick an IRQ vector in range [32, UINT8_MAX]. Min value is 32 because tools/testing/selftests/kvm/vfio_pci_irq_test.c line 113
Quoted 10 lines
tools/testing/selftests/kvm/vfio_pci_irq_test.c new file +10
-0+* Linux/x86 reserves vectors 0-31 for exceptions and architecture +* defined NMIs and interrupts. +* +* Pick a GSI in range [24, KVM_MAX_IRQ_ROUTES - 1]. The min value is 24 +* because KVM reserves GSIs 0-15 for legacy ISA IRQs and 16-23 only go +* to the IOAPIC. The max is KVM_MAX_IRQ_ROUTES - 1, because +* KVM_MAX_IRQ_ROUTES is exclusive. +*/ +u32 gsi =24 +rand () % (KVM_MAX_IRQ_ROUTES -1 -24 ); +u8 vector =32 +rand () % (UINT8_MAX -32 );Ahh, now I see why you included "Reproduce tests that rely on randomization" in this series.
Quoted 4 lines
I think I'd prefer to tweak
guest_random_state()to drop the "guest" in favor of "kvm", and then use that framework in host code as well. Then you wouldn't need to open code the % fun.
How would using the guest_random_state() library remove the need to calculate the appropriate ranges using %? The library (tools/testing/selftests/kvm/include/test_util.h) only provides:
uint64_t guest_random_u64(struct guest_random_state *state); uint32_t guest_random_u32(struct guest_random_state *state);
I could avoid using % by adding a new library function that picks a random number within a range:
uint64_t kvm_random_u64_in_range(struct kvm_random_state *state, uint64_t min, uint64_t max);
Quoted 31 lines
Quoted 31 lines
tools/testing/selftests/kvm/vfio_pci_irq_test.c new file +31
-0+/* Test configuration (overridable by command line flags). */ +int nr_irqs =1000 ; +int nr_vcpus =1 ; + +struct kvm_vcpu *vcpus[KVM_MAX_VCPUS ]; +pthread_t vcpu_threads[KVM_MAX_VCPUS ]; +u64 irq_count, pin_count, piw_count; +struct vfio_pci_device *device; +struct iommu *iommu; +const char *device_bdf; +int i, j, c, msi, irq; +struct kvm_vm *vm; + + device_bdf =vfio_selftests_get_bdf (&argc, argv); + +while ((c =getopt (argc, argv,"h" )) !=-1 ) { +switch (c) { +case 'h' : +default : +help (argv[0 ]); + } + } + + vm =vm_create_with_vcpus (nr_vcpus, guest_code, vcpus); +vm_install_exception_handler (vm, vector, guest_irq_handler); + + iommu =iommu_init (default_iommu_mode); + device =vfio_pci_device_init (device_bdf, iommu); + msi =setup_msi (device); + irq =get_irq_number (device_bdf, msi); +
Quoted 9 lines
tools/testing/selftests/kvm/vfio_pci_irq_test.c line 148
Quoted 3 lines
tools/testing/selftests/kvm/vfio_pci_irq_test.c new file +3
-0+ irq_count =get_irq_count (irq); + pin_count =get_irq_count_by_name ("PIN:" ); + piw_count =get_irq_count_by_name ("PIW:" );I'm *very* skeptical of blindling printing PIN/PIW. They're Intel specific and are global to the entire system. Outside of debugging specific issues, I don't see them providing much value, while on the other hand, they could easily confuse readers. And unlike the IRQ counts, the data won't be lost when the test exits.
I'll remove PIN/PIW in v3.
Quoted 4 lines
As for the
get_irq_number()andget_irq_count(), those need to be much more clearly scoped to/proc/interrupts. E.g. "irq" in KVM context most often refers to the guest IRQ vector, not the host Linux make-believe IRQ number.
How about I rename these to get_proc_irq_num() and get_proc_irq_count()? And then rename the library irq_util.h/c to proc_util.h/c?
Quoted 8 lines
tools/testing/selftests/kvm/vfio_pci_irq_test.c line 151
Quoted 2 lines
tools/testing/selftests/kvm/vfio_pci_irq_test.c new file +2
-0+ printf("%s %s MSI-X[%d] (IRQ-%d) %d times\n", + "Notifying the eventfd for",Where's the rest of this line? Oh, this is the first %s. LOL, that's ridiculous.
printf("Notifying the eventfd for BDF %s, MSI-X[%d] (IRQ-%d) %d times\n", device_bdf, msi, irq, nr_irqs);
Will fix in v3.
Quoted 10 lines
Quoted 10 lines
tools/testing/selftests/kvm/vfio_pci_irq_test.c new file +10
-0+ device_bdf, msi, irq, nr_irqs); + +kvm_assign_irqfd (vm, gsi, device->msi_eventfds[msi]); + +for (i =0 ; i < nr_vcpus; i++) +pthread_create (&vcpu_threads[i],NULL , vcpu_thread_main, vcpus[i]); + +for (i =0 ; i < nr_vcpus; i++) { +struct kvm_vcpu *vcpu = vcpus[i]; +
Quoted 9 lines
tools/testing/selftests/kvm/vfio_pci_irq_test.c line 167
Quoted 6 lines
tools/testing/selftests/kvm/vfio_pci_irq_test.c new file +6
-0+while (!READ_FROM_GUEST (vm, guest_ready_for_irqs[vcpu->id])) +continue ; + } + +/* Set a consistent seed so that test are repeatable. */ +srand (0 );And using a pRNG instead of
rand()should make this unnecessary.
We shouldn't even need a pRNG if we follow your suggestion above and use the guest_random_state() framework for the host, as it would provide the guest_random_u64() function.
Quoted 20 lines
Quoted 20 lines
tools/testing/selftests/kvm/vfio_pci_irq_test.c new file +20
-0+for (i =0 ; i < nr_irqs; i++) { +struct kvm_vcpu *vcpu = vcpus[i % nr_vcpus]; +struct timespec start; + +kvm_route_msi (vm, gsi, vcpu, vector); + +for (j =0 ; j < nr_vcpus; j++) { +TEST_ASSERT ( + !READ_FROM_GUEST (vm, guest_received_irq[vcpu->id]), +"IRQ flag for vCPU %d not clear prior to test" , + vcpu->id); + } + +send_msi (device, msi); + +clock_gettime (CLOCK_MONOTONIC , &start); +for (;;) { +if (READ_FROM_GUEST (vm, guest_received_irq[vcpu->id])) +break ; +
Quoted 9 lines
tools/testing/selftests/kvm/vfio_pci_irq_test.c line 193
Quoted 5 lines
tools/testing/selftests/kvm/vfio_pci_irq_test.c new file +5
-0+if (timespec_to_ns (timespec_elapsed (start)) >TIMEOUT_NS ) { +printf ("Timeout waiting for interrupt!\n" ); +printf (" vCPU: %d\n" , vcpu->id); + +TEST_FAIL ("vCPU never received IRQ!\n" );Which vCPU? What BDF+MSI? Some of that is available in earlier messages, but failure messages should be standalone.
Got it, I'll add that info in v3.
Quoted 12 lines
tools/testing/selftests/kvm/vfio_pci_irq_test.c line 202
Quoted 9 lines
tools/testing/selftests/kvm/vfio_pci_irq_test.c new file +9
-0+ } + } + +WRITE_TO_GUEST (vm, guest_received_irq[vcpu->id], false); + } + +WRITE_TO_GUEST (vm, done, true); + +for (i =0 ; i < nr_vcpus; i++) {Unnecessary curly braces.
Will remove in v3.
Quoted 18 lines
Quoted 18 lines
tools/testing/selftests/kvm/vfio_pci_irq_test.c new file +15
-0+pthread_join (vcpu_threads[i],NULL ); + } + +printf ("Host interrupts handled:\n" ); +printf (" IRQ-%d: %lu\n" , irq,get_irq_count (irq) - irq_count); +printf (" Posted-interrupt notification events: %lu\n" , +get_irq_count_by_name ("PIN:" ) - pin_count); +printf (" Posted-interrupt wakeup events: %lu\n" , +get_irq_count_by_name ("PIW:" ) - piw_count); + +vfio_pci_device_cleanup (device); +iommu_cleanup (iommu); + +return 0 ; +}Signature · 1 line
2.53.0.1118.gaef5881109-goog
Sean Christopherson
reviewerRe: [PATCH v2 03/14] KVM: selftests: Add vfio_pci_irq_test
Quoted 30 lines · Josh Hilke
Quoted 21 lines · Sean Christopherson
Quoted 6 lines · Josh Hilke
tools/testing/selftests/kvm/vfio_pci_irq_test.c new file +6
-0+int main (int argc,char **argv) +{ +/* +* Pick a random vector and a random GSI to use for device IRQ. +* +* Pick an IRQ vector in range [32, UINT8_MAX]. Min value is 32 because tools/testing/selftests/kvm/vfio_pci_irq_test.c line 113
Quoted 10 lines
tools/testing/selftests/kvm/vfio_pci_irq_test.c new file +10
-0+* Linux/x86 reserves vectors 0-31 for exceptions and architecture +* defined NMIs and interrupts. +* +* Pick a GSI in range [24, KVM_MAX_IRQ_ROUTES - 1]. The min value is 24 +* because KVM reserves GSIs 0-15 for legacy ISA IRQs and 16-23 only go +* to the IOAPIC. The max is KVM_MAX_IRQ_ROUTES - 1, because +* KVM_MAX_IRQ_ROUTES is exclusive. +*/ +u32 gsi =24 +rand () % (KVM_MAX_IRQ_ROUTES -1 -24 ); +u8 vector =32 +rand () % (UINT8_MAX -32 );Ahh, now I see why you included "Reproduce tests that rely on randomization" in this series.
Quoted 4 lines
I think I'd prefer to tweak
guest_random_state()to drop the "guest" in favor of "kvm", and then use that framework in host code as well. Then you wouldn't need to open code the % fun.How would using the
guest_random_state()library remove the need to calculate the appropriate ranges using %? The library (tools/testing/selftests/kvm/include/test_util.h) only provides:
Quoted 7 lines
uint64_tguest_random_u64(struct guest_random_state *state);uint32_tguest_random_u32(struct guest_random_state *state);I could avoid using % by adding a new library function that picks a random number within a range:
uint64_tkvm_random_u64_in_range(struct kvm_random_state *state, uint64_t min, uint64_t max);
Yes, please. I was thinking that in my head, but never typed it out. The original plan with the pRNG code was to build out the set of APIs for doing random number things, it just never picked up steam due to lack of need.
Quoted 7 lines
Quoted 4 lines
As for the
get_irq_number()andget_irq_count(), those need to be much more clearly scoped to/proc/interrupts. E.g. "irq" in KVM context most often refers to the guest IRQ vector, not the host Linux make-believe IRQ number.How about I rename these to
get_proc_irq_num()andget_proc_irq_count()? And then rename the library irq_util.h/c to proc_util.h/c?
Works for me.
Quoted 9 lines
Quoted 5 lines
tools/testing/selftests/kvm/vfio_pci_irq_test.c line 167
Quoted 2 lines
tools/testing/selftests/kvm/vfio_pci_irq_test.c new file +2
-0+/* Set a consistent seed so that test are repeatable. */ +srand (0 );And using a pRNG instead of
rand()should make this unnecessary.We shouldn't even need a pRNG if we follow your suggestion above and use the
guest_random_state()framework for the host, as it would provide theguest_random_u64()function.
Heh, by pRNG I meant guest_random_state itself: "struct guest_random_state" is itself a pseudo-RNG implementation.
PATCH v2 4/14 KVM: selftests: Reproduce tests that rely on randomization
In reply to Josh HilkeSeed srand() using the KVM_RANDOM_SEED environment variable during selftest initialization. If KVM_RANDOM_SEED is not set, seed srand() using the current time. In order to reproduce a test that relies on values that are set using rand(), the test must not call srand(), as this will override the call to srand() during test initialization.
To test this, run a selftest:
$ ./selftest
The first line of test output should look like:
Guest random seed: <guest_seed> (srand: <seed>)
Then to reproduce, run:
$ KVM_RANDOM_SEED=<seed> ./selftest
The test output should show the same value for the srand seed.
--- tools/testing/selftests/kvm/lib/kvm_util.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-)
tools/testing/selftests/kvm/lib/kvm_util.c +10 -1
@@ -2349,6 +2349,8 @@ void __attribute((constructor)) kvm_selftest_init(void)struct sigaction sig_sa = { .sa_handler = report_unexpected_signal, }; +char *env_seed =getenv ("KVM_RANDOM_SEED" ); +int random_seed;/* Tell stdout not to buffer its content. */ setbuf (stdout,NULL );
@@ -2358,8 +2360,15 @@ void __attribute((constructor)) kvm_selftest_init(void)sigaction (SIGILL , &sig_sa,NULL );sigaction (SIGFPE , &sig_sa,NULL ); +if (env_seed) + random_seed =strtol (env_seed,NULL ,0 ); +else + random_seed =time (NULL ); + +srand (random_seed); guest_random_seed = last_guest_seed =random (); -pr_info ("Random seed: 0x%x\n" , guest_random_seed); +pr_info ("Guest random seed: 0x%x (srand: 0x%x)\n" , + guest_random_seed, random_seed);kvm_selftest_arch_init (); }
Signature · 1 line
2.53.0.1118.gaef5881109-goog
PATCH v2 5/14 KVM: selftests: Add support for random host IRQ affinity
In reply to Josh HilkeFrom: David Matlack <[email protected]>
Add the '-a' flag to vfio_pci_irq_test to randomly affinitize the device's host IRQ to different physical CPUs throughout the test. This stresses the kernel's ability to maintain correct interrupt routing and delivery even as the underlying hardware IRQ affinity is changed dynamically via /proc/irq/.
--- .../testing/selftests/kvm/vfio_pci_irq_test.c | 33 +++++++++++++++++-- 1 file changed, 31 insertions(+), 2 deletions(-)
tools/testing/selftests/kvm/vfio_pci_irq_test.c +31 -2
@@ -88,9 +88,11 @@ static void send_msi(struct vfio_pci_device *device, int msi)static void help (const char *name) { -printf ("Usage: %s [-h] segment:bus:device.function\n" , +printf ("Usage: %s [-a] [-h] segment:bus:device.function\n" , name);printf ("\n" ); +printf (" -a: Randomly affinitize the device IRQ to different CPUs\n" +" throughout the test.\n" );printf ("\n" );exit (KSFT_FAIL ); }
@@ -113,6 +115,7 @@ int main(int argc, char **argv)u8 vector =32 +rand () % (UINT8_MAX -32 );/* Test configuration (overridable by command line flags). */ +bool irq_affinity = false;int nr_irqs =1000 ;int nr_vcpus =1 ;
@@ -122,13 +125,19 @@ int main(int argc, char **argv)struct vfio_pci_device *device;struct iommu *iommu;const char *device_bdf; +FILE *irq_affinity_fp;int i, j, c, msi, irq;struct kvm_vm *vm; +int irq_cpu; +int ret; device_bdf =vfio_selftests_get_bdf (&argc, argv); -while ((c =getopt (argc, argv,"h" )) !=-1 ) { +while ((c =getopt (argc, argv,"ah" )) !=-1 ) {switch (c) { +case 'a' : + irq_affinity = true; +break ;case 'h' :default :help (argv[0 ]);
@@ -163,6 +172,14 @@ int main(int argc, char **argv)continue ; } +if (irq_affinity) { +char path[PATH_MAX ]; + +snprintf (path,sizeof (path),"/proc/irq/%d/smp_affinity_list" , irq); + irq_affinity_fp =fopen (path,"w" ); +TEST_ASSERT (irq_affinity_fp,"fopen(%s) failed" , path); + } +/* Set a consistent seed so that test are repeatable. */ srand (0 );
@@ -172,6 +189,13 @@ int main(int argc, char **argv)kvm_route_msi (vm, gsi, vcpu, vector); +if (irq_affinity && vcpu->id ==0 ) { + irq_cpu =rand () %get_nprocs (); + + ret =fprintf (irq_affinity_fp,"%d\n" , irq_cpu); +TEST_ASSERT (ret >0 ,"Failed to affinitize IRQ-%d to CPU %d" , irq, irq_cpu); + } +for (j =0 ; j < nr_vcpus; j++) { TEST_ASSERT( !READ_FROM_GUEST(vm, guest_received_irq[vcpu->id]),
@@ -189,6 +213,8 @@ int main(int argc, char **argv)if (timespec_to_ns (timespec_elapsed (start)) >TIMEOUT_NS ) {printf ("Timeout waiting for interrupt!\n" );printf (" vCPU: %d\n" , vcpu->id); +if (irq_affinity) +printf (" irq_cpu: %d\n" , irq_cpu);TEST_FAIL ("vCPU never received IRQ!\n" ); }
@@ -203,6 +229,9 @@ int main(int argc, char **argv)pthread_join (vcpu_threads[i],NULL ); } +if (irq_affinity) +fclose (irq_affinity_fp); +printf ("Host interrupts handled:\n" );printf (" IRQ-%d: %lu\n" , irq,get_irq_count (irq) - irq_count); printf(" Posted-interrupt notification events: %lu\n",
Signature · 1 line
2.53.0.1118.gaef5881109-goog
Sean Christopherson
reviewerRe: [PATCH v2 05/14] KVM: selftests: Add support for random host IRQ affinity
Quoted 11 lines · Josh Hilke
From: David Matlack <[email protected]>
Add the '-a' flag to
vfio_pci_irq_testto randomly affinitize the device's host IRQ to different physical CPUs throughout the test. This stresses the kernel's ability to maintain correct interrupt routing and delivery even as the underlying hardware IRQ affinity is changed dynamically via/proc/irq/.Signed-off-by David Matlack <[email protected]>Signed-off-by Josh Hilke <[email protected]>Co-developed-by Josh Hilke <[email protected]>
Please run checkpatch, the order here is wrong.
tools/testing/selftests/kvm/vfio_pci_irq_test.c int main(int argc, char **argv) · line 132
Quoted 9 lines
tools/testing/selftests/kvm/vfio_pci_irq_test.c +3
-0@@ -122,13 +125,19 @@ int main(int argc, char **argv)struct vfio_pci_device *device;struct iommu *iommu;const char *device_bdf; +FILE *irq_affinity_fp;int i, j, c, msi, irq;struct kvm_vm *vm; +int irq_cpu; +int ret;
Use the existing "int" declaration.
Quoted 12 lines
tools/testing/selftests/kvm/vfio_pci_irq_test.c +4
-1device_bdf =vfio_selftests_get_bdf (&argc, argv); -while ((c =getopt (argc, argv,"h" )) !=-1 ) { +while ((c =getopt (argc, argv,"ah" )) !=-1 ) {switch (c) { +case 'a' : + irq_affinity = true; +break ;case 'h' :default :help (argv[0 ]);
tools/testing/selftests/kvm/vfio_pci_irq_test.c int main(int argc, char **argv) · line 181
Quoted 10 lines
tools/testing/selftests/kvm/vfio_pci_irq_test.c +7
-0@@ -163,6 +172,14 @@ int main(int argc, char **argv)continue ; } +if (irq_affinity) { +char path[PATH_MAX ]; + +snprintf (path,sizeof (path),"/proc/irq/%d/smp_affinity_list" , irq); + irq_affinity_fp =fopen (path,"w" ); +TEST_ASSERT (irq_affinity_fp,"fopen(%s) failed" , path); + }
This plus setting the affinity is begging for a library function, probably provided by what's currently called get_irq_number() and friends.
Quoted 40 lines
tools/testing/selftests/kvm/vfio_pci_irq_test.c +13
-0+/* Set a consistent seed so that test are repeatable. */ srand (0 );@@ -172,6 +189,13 @@ int main(int argc, char **argv)kvm_route_msi (vm, gsi, vcpu, vector); +if (irq_affinity && vcpu->id ==0 ) { + irq_cpu =rand () %get_nprocs (); + + ret =fprintf (irq_affinity_fp,"%d\n" , irq_cpu); +TEST_ASSERT (ret >0 ,"Failed to affinitize IRQ-%d to CPU %d" , irq, irq_cpu); + } +for (j =0 ; j < nr_vcpus; j++) { TEST_ASSERT( !READ_FROM_GUEST(vm, guest_received_irq[vcpu->id]),@@ -189,6 +213,8 @@ int main(int argc, char **argv)if (timespec_to_ns (timespec_elapsed (start)) >TIMEOUT_NS ) {printf ("Timeout waiting for interrupt!\n" );printf (" vCPU: %d\n" , vcpu->id); +if (irq_affinity) +printf (" irq_cpu: %d\n" , irq_cpu);TEST_FAIL ("vCPU never received IRQ!\n" ); }@@ -203,6 +229,9 @@ int main(int argc, char **argv)pthread_join (vcpu_threads[i],NULL ); } +if (irq_affinity) +fclose (irq_affinity_fp); +printf ("Host interrupts handled:\n" );printf (" IRQ-%d: %lu\n" , irq,get_irq_count (irq) - irq_count); printf(" Posted-interrupt notification events: %lu\n",Signature · 1 line
2.53.0.1118.gaef5881109-goog
Re: [PATCH v2 05/14] KVM: selftests: Add support for random host IRQ affinity
Quoted 9 lines · Sean Christopherson
Quoted 8 lines · Josh Hilke
From: David Matlack <[email protected]>
Add the '-a' flag to
vfio_pci_irq_testto randomly affinitize the device's host IRQ to different physical CPUs throughout the test. This stresses the kernel's ability to maintain correct interrupt routing and delivery even as the underlying hardware IRQ affinity is changed dynamically via/proc/irq/.
Quoted 6 lines
Quoted 3 lines
Signed-off-by David Matlack <[email protected]>Signed-off-by Josh Hilke <[email protected]>Co-developed-by Josh Hilke <[email protected]>Please run checkpatch, the order here is wrong.
Ah, my bad didn't know there was a checkpatch script. Will do.
Quoted 12 lines
tools/testing/selftests/kvm/vfio_pci_irq_test.c int main(int argc, char **argv) · line 132
Quoted 9 lines
tools/testing/selftests/kvm/vfio_pci_irq_test.c +3
-0@@ -122,13 +125,19 @@ int main(int argc, char **argv)struct vfio_pci_device *device;struct iommu *iommu;const char *device_bdf; +FILE *irq_affinity_fp;int i, j, c, msi, irq;struct kvm_vm *vm; +int irq_cpu; +int ret;Use the existing "int" declaration.
Will do in v3.
Quoted 15 lines
Quoted 15 lines
tools/testing/selftests/kvm/vfio_pci_irq_test.c +4
-1device_bdf =vfio_selftests_get_bdf (&argc, argv); -while ((c =getopt (argc, argv,"h" )) !=-1 ) { +while ((c =getopt (argc, argv,"ah" )) !=-1 ) {switch (c) { +case 'a' : + irq_affinity = true; +break ;case 'h' :default :help (argv[0 ]);@@ -163,6 +172,14 @@ int main(int argc, char **argv)continue ; }
Quoted 11 lines
tools/testing/selftests/kvm/vfio_pci_irq_test.c int main(int argc, char **argv) · line 181
Quoted 7 lines
tools/testing/selftests/kvm/vfio_pci_irq_test.c +7
-0+if (irq_affinity) { +char path[PATH_MAX ]; + +snprintf (path,sizeof (path),"/proc/irq/%d/smp_affinity_list" , irq); + irq_affinity_fp =fopen (path,"w" ); +TEST_ASSERT (irq_affinity_fp,"fopen(%s) failed" , path); + }This plus setting the affinity is begging for a library function, probably provided by what's currently called
get_irq_number()and friends.
I'll create a helper and add it to that library in v3.
Quoted 40 lines
Quoted 40 lines
tools/testing/selftests/kvm/vfio_pci_irq_test.c +13
-0+/* Set a consistent seed so that test are repeatable. */ srand (0 );@@ -172,6 +189,13 @@ int main(int argc, char **argv)kvm_route_msi (vm, gsi, vcpu, vector); +if (irq_affinity && vcpu->id ==0 ) { + irq_cpu =rand () %get_nprocs (); + + ret =fprintf (irq_affinity_fp,"%d\n" , irq_cpu); +TEST_ASSERT (ret >0 ,"Failed to affinitize IRQ-%d to CPU %d" , irq, irq_cpu); + } +for (j =0 ; j < nr_vcpus; j++) { TEST_ASSERT( !READ_FROM_GUEST(vm, guest_received_irq[vcpu->id]),@@ -189,6 +213,8 @@ int main(int argc, char **argv)if (timespec_to_ns (timespec_elapsed (start)) >TIMEOUT_NS ) {printf ("Timeout waiting for interrupt!\n" );printf (" vCPU: %d\n" , vcpu->id); +if (irq_affinity) +printf (" irq_cpu: %d\n" , irq_cpu);TEST_FAIL ("vCPU never received IRQ!\n" ); }@@ -203,6 +229,9 @@ int main(int argc, char **argv)pthread_join (vcpu_threads[i],NULL ); } +if (irq_affinity) +fclose (irq_affinity_fp); +printf ("Host interrupts handled:\n" );printf (" IRQ-%d: %lu\n" , irq,get_irq_count (irq) - irq_count); printf(" Posted-interrupt notification events: %lu\n",Signature · 1 line
2.53.0.1118.gaef5881109-goog
PATCH v2 6/14 KVM: selftests: Allow blocking vCPUs via HLT
In reply to Josh HilkeFrom: David Matlack <[email protected]>
Add the '-b' flag to vfio_pci_irq_test to test IRQ delivery while vCPUs are in a halted state. This verifies that interrupts correctly wake vCPUs from HLT, which is a critical path for both power management and correct architectural behavior during asynchronous events.
--- .../testing/selftests/kvm/vfio_pci_irq_test.c | 21 ++++++++++++++++--- 1 file changed, 18 insertions(+), 3 deletions(-)
tools/testing/selftests/kvm/vfio_pci_irq_test.c +18 -3
@@ -16,6 +16,7 @@#include <libvfio.h> static bool done; +static bool block;static bool guest_ready_for_irqs[KVM_MAX_VCPUS ];static bool guest_received_irq[KVM_MAX_VCPUS ];
@@ -42,7 +43,10 @@ static void guest_code(void)WRITE_ONCE (guest_ready_for_irqs[guest_get_vcpu_id ()], true);while (!READ_ONCE (done)) { -cpu_relax (); +if (block) +hlt (); +else +cpu_relax (); }GUEST_DONE ();
@@ -88,11 +92,12 @@ static void send_msi(struct vfio_pci_device *device, int msi)static void help (const char *name) { -printf ("Usage: %s [-a] [-h] segment:bus:device.function\n" , +printf ("Usage: %s [-a] [-b] [-h] segment:bus:device.function\n" , name);printf ("\n" );printf (" -a: Randomly affinitize the device IRQ to different CPUs\n" " throughout the test.\n" ); +printf (" -b: Block vCPUs (e.g. HLT) instead of spinning in guest-mode\n" );printf ("\n" );exit (KSFT_FAIL ); }
@@ -133,11 +138,14 @@ int main(int argc, char **argv) device_bdf =vfio_selftests_get_bdf (&argc, argv); -while ((c =getopt (argc, argv,"ah" )) !=-1 ) { +while ((c =getopt (argc, argv,"abh" )) !=-1 ) {switch (c) {case 'a' : irq_affinity = true;break ; +case 'b' : + block = true; +break ;case 'h' :default :help (argv[0 ]);
@@ -162,6 +170,8 @@ int main(int argc, char **argv)kvm_assign_irqfd (vm, gsi, device->msi_eventfds[msi]); +sync_global_to_guest (vm, block); +for (i =0 ; i < nr_vcpus; i++)pthread_create (&vcpu_threads[i],NULL , vcpu_thread_main, vcpus[i]);
@@ -226,6 +236,11 @@ int main(int argc, char **argv)WRITE_TO_GUEST (vm, done, true);for (i =0 ; i < nr_vcpus; i++) { +if (block) { +kvm_route_msi (vm, gsi, vcpus[i], vector); +send_msi (device, msi); + } +pthread_join (vcpu_threads[i],NULL ); }
Signature · 1 line
2.53.0.1118.gaef5881109-goog
Sean Christopherson
reviewerRe: [PATCH v2 06/14] KVM: selftests: Allow blocking vCPUs via HLT
Quoted 3 lines · Josh Hilke
Signed-off-by David Matlack <[email protected]>Signed-off-by Josh Hilke <[email protected]>Co-developed-by Josh Hilke <[email protected]>
Checkpatch.
tools/testing/selftests/kvm/vfio_pci_irq_test.c int main(int argc, char **argv) · line 238
Quoted 13 lines
tools/testing/selftests/kvm/vfio_pci_irq_test.c +2
-0@@ -162,6 +170,8 @@ int main(int argc, char **argv)kvm_assign_irqfd (vm, gsi, device->msi_eventfds[msi]); +sync_global_to_guest (vm, block); +for (i =0 ; i < nr_vcpus; i++)pthread_create (&vcpu_threads[i],NULL , vcpu_thread_main, vcpus[i]);@@ -226,6 +236,11 @@ int main(int argc, char **argv)WRITE_TO_GUEST (vm, done, true);for (i =0 ; i < nr_vcpus; i++) {
Please add a comment explaining what this is doing.
Quoted 11 lines
tools/testing/selftests/kvm/vfio_pci_irq_test.c +5
-0+if (block) { +kvm_route_msi (vm, gsi, vcpus[i], vector); +send_msi (device, msi); + } +pthread_join (vcpu_threads[i],NULL ); }Signature · 1 line
2.53.0.1118.gaef5881109-goog
PATCH v2 7/14 KVM: selftests: Add support for physical device MSI triggers
In reply to Josh HilkeFrom: David Matlack <[email protected]>
Add the '-d' flag to vfio_pci_irq_test to trigger interrupts using the physical device hardware instead of emulating them via eventfd writes. This leverages the VFIO library's driver support to provide a more realistic validation of the hardware-to-guest interrupt path.
--- .../testing/selftests/kvm/vfio_pci_irq_test.c | 51 +++++++++++++++---- 1 file changed, 41 insertions(+), 10 deletions(-)
tools/testing/selftests/kvm/vfio_pci_irq_test.c +41 -10
@@ -78,26 +78,54 @@ static void kvm_route_msi(struct kvm_vm *vm, u32 gsi, struct kvm_vcpu *vcpu,vm_ioctl (vm,KVM_SET_GSI_ROUTING , routes); } -static int setup_msi (struct vfio_pci_device *device) +static int setup_msi (struct vfio_pci_device *device,bool use_device_msi) { +const int flags =MAP_SHARED |MAP_ANONYMOUS ; +const int prot =PROT_READ |PROT_WRITE ; +struct dma_region *region; + +if (use_device_msi) { +/* A driver is required to generate an MSI. */ +TEST_REQUIRE (device->driver.ops); + +/* Set up a DMA-able region for the driver to use. */ + region = &device->driver.region; + region->iova =0 ; + region->size =SZ_2M ; + region->vaddr =kvm_mmap (region->size, prot, flags,-1 ); +TEST_ASSERT (region->vaddr !=MAP_FAILED ,"mmap() failed\n" ); +iommu_map (device->iommu, region); + +vfio_pci_driver_init (device); + +return device->driver.msi; + } +TEST_REQUIRE (device->msix_info.count >0 );vfio_pci_msix_enable (device,0 ,1 );return 0 ; } -static void send_msi (struct vfio_pci_device *device,int msi) +static void send_msi (struct vfio_pci_device *device,bool use_device_msi,int msi) { -vfio_pci_irq_trigger (device,VFIO_PCI_MSIX_IRQ_INDEX , msi); +if (use_device_msi) { +TEST_ASSERT_EQ (msi, device->driver.msi); +vfio_pci_driver_send_msi (device); + }else { +vfio_pci_irq_trigger (device,VFIO_PCI_MSIX_IRQ_INDEX , msi); + } }static void help (const char *name) { -printf ("Usage: %s [-a] [-b] [-h] segment:bus:device.function\n" , +printf ("Usage: %s [-a] [-b] [-d] [-h] segment:bus:device.function\n" , name);printf ("\n" );printf (" -a: Randomly affinitize the device IRQ to different CPUs\n" " throughout the test.\n" );printf (" -b: Block vCPUs (e.g. HLT) instead of spinning in guest-mode\n" ); +printf (" -d: Use the device to trigger the IRQ instead of emulating\n" +" it with an eventfd write.\n" );printf ("\n" );exit (KSFT_FAIL ); }
@@ -120,7 +148,7 @@ int main(int argc, char **argv)u8 vector =32 +rand () % (UINT8_MAX -32 );/* Test configuration (overridable by command line flags). */ -bool irq_affinity = false; +bool use_device_msi = false, irq_affinity = false;int nr_irqs =1000 ;int nr_vcpus =1 ;
@@ -138,7 +166,7 @@ int main(int argc, char **argv) device_bdf =vfio_selftests_get_bdf (&argc, argv); -while ((c =getopt (argc, argv,"abh" )) !=-1 ) { +while ((c =getopt (argc, argv,"abdh" )) !=-1 ) {switch (c) {case 'a' : irq_affinity = true;
@@ -146,6 +174,9 @@ int main(int argc, char **argv)case 'b' : block = true;break ; +case 'd' : + use_device_msi = true; +break ;case 'h' :default :help (argv[0 ]);
@@ -157,7 +188,7 @@ int main(int argc, char **argv) iommu =iommu_init (default_iommu_mode); device =vfio_pci_device_init (device_bdf, iommu); - msi =setup_msi (device); + msi =setup_msi (device, use_device_msi); irq =get_irq_number (device_bdf, msi); irq_count =get_irq_count (irq);
@@ -165,7 +196,7 @@ int main(int argc, char **argv) piw_count =get_irq_count_by_name ("PIW:" );printf ("%s %s MSI-X[%d] (IRQ-%d) %d times\n" , -"Notifying the eventfd for" , + use_device_msi ?"Triggering" :"Notifying the eventfd for" , device_bdf, msi, irq, nr_irqs);kvm_assign_irqfd (vm, gsi, device->msi_eventfds[msi]);
@@ -213,7 +244,7 @@ int main(int argc, char **argv) vcpu->id); } -send_msi (device, msi); +send_msi (device, use_device_msi, msi);clock_gettime (CLOCK_MONOTONIC , &start);for (;;) {
@@ -238,7 +269,7 @@ int main(int argc, char **argv)for (i =0 ; i < nr_vcpus; i++) {if (block) {kvm_route_msi (vm, gsi, vcpus[i], vector); -send_msi (device, msi); +send_msi (device, false, msi); }pthread_join (vcpu_threads[i],NULL );
Signature · 1 line
2.53.0.1118.gaef5881109-goog
Sean Christopherson
reviewerRe: [PATCH v2 07/14] KVM: selftests: Add support for physical device MSI triggers
Quoted 6 lines · Josh Hilke
From: David Matlack <[email protected]>
Add the '-d' flag to
vfio_pci_irq_testto trigger interrupts using the physical device hardware instead of emulating them via eventfd writes. This leverages the VFIO library's driver support to provide a more realistic validation of the hardware-to-guest interrupt path.
I don't think this should be optional, I think the test should unconditionally validate both path. The "signal eventfd directly" is effectively a smoke test of the full functionality; if that doesn't work, validating that sending via the device itself is pointless.
To keep the runtime reasonable, maybe do only 10% of the iterations for the eventfd version, capped at a lowish hardcoded value?
Re: [PATCH v2 07/14] KVM: selftests: Add support for physical device MSI triggers
Quoted 8 lines · Sean Christopherson
Quoted 6 lines · Josh Hilke
From: David Matlack <[email protected]>
Add the '-d' flag to
vfio_pci_irq_testto trigger interrupts using the physical device hardware instead of emulating them via eventfd writes. This leverages the VFIO library's driver support to provide a more realistic validation of the hardware-to-guest interrupt path.
Quoted 7 lines
I don't think this should be optional, I think the test should unconditionally validate both path. The "signal eventfd directly" is effectively a smoke test of the full functionality; if that doesn't work, validating that sending via the device itself is pointless.
To keep the runtime reasonable, maybe do only 10% of the iterations for the eventfd version, capped at a lowish hardcoded value?
We can only use physical device interrupts if there is a VFIO selftest driver for the device. I can update the test in v3 to validate both paths if there's a driver, and fall back to using just eventfds otherwise.
PATCH v2 8/14 KVM: selftests: Add option to clear GSI routes
In reply to Josh HilkeFrom: David Matlack <[email protected]>
Add the '-e' flag to vfio_pci_irq_test to destroy and recreate KVM's GSI routing table between interrupts. This ensures that KVM correctly handles dynamic updates to the interrupt routing table while interrupts are actively being signaled by assigned devices.
--- .../testing/selftests/kvm/vfio_pci_irq_test.c | 22 +++++++++++++++++-- 1 file changed, 20 insertions(+), 2 deletions(-)
tools/testing/selftests/kvm/vfio_pci_irq_test.c +20 -2
@@ -63,6 +63,13 @@ static void *vcpu_thread_main(void *arg)return NULL ; } +static void kvm_clear_gsi_routes (struct kvm_vm *vm) +{ +struct kvm_irq_routing routes = {}; + +vm_ioctl (vm,KVM_SET_GSI_ROUTING , &routes); +} +static void kvm_route_msi (struct kvm_vm *vm,u32 gsi,struct kvm_vcpu *vcpu,u8 vector) {
@@ -118,7 +125,7 @@ static void send_msi(struct vfio_pci_device *device, bool use_device_msi, int msstatic void help (const char *name) { -printf ("Usage: %s [-a] [-b] [-d] [-h] segment:bus:device.function\n" , +printf ("Usage: %s [-a] [-b] [-d] [-e] [-h] segment:bus:device.function\n" , name);printf ("\n" ); printf(" -a: Randomly affinitize the device IRQ to different CPUs\n"
@@ -126,6 +133,8 @@ static void help(const char *name)printf (" -b: Block vCPUs (e.g. HLT) instead of spinning in guest-mode\n" );printf (" -d: Use the device to trigger the IRQ instead of emulating\n" " it with an eventfd write.\n" ); +printf (" -e: Destroy and recreate KVM's GSI routing table in between\n" +" some interrupts.\n" );printf ("\n" );exit (KSFT_FAIL ); }
@@ -149,6 +158,7 @@ int main(int argc, char **argv)/* Test configuration (overridable by command line flags). */ bool use_device_msi = false, irq_affinity = false; +bool empty = false;int nr_irqs =1000 ;int nr_vcpus =1 ;
@@ -166,7 +176,7 @@ int main(int argc, char **argv) device_bdf =vfio_selftests_get_bdf (&argc, argv); -while ((c =getopt (argc, argv,"abdh" )) !=-1 ) { +while ((c =getopt (argc, argv,"abdeh" )) !=-1 ) {switch (c) {case 'a' : irq_affinity = true;
@@ -177,6 +187,9 @@ int main(int argc, char **argv)case 'd' : use_device_msi = true;break ; +case 'e' : + empty = true; +break ;case 'h' :default :help (argv[0 ]);
@@ -226,8 +239,12 @@ int main(int argc, char **argv)for (i =0 ; i < nr_irqs; i++) {struct kvm_vcpu *vcpu = vcpus[i % nr_vcpus]; +const bool do_empty = empty && (i &BIT (3 ));struct timespec start; +if (do_empty) +kvm_clear_gsi_routes (vm); +kvm_route_msi (vm, gsi, vcpu, vector);if (irq_affinity && vcpu->id ==0 ) {
@@ -254,6 +271,7 @@ int main(int argc, char **argv)if (timespec_to_ns (timespec_elapsed (start)) >TIMEOUT_NS ) {printf ("Timeout waiting for interrupt!\n" );printf (" vCPU: %d\n" , vcpu->id); +printf (" do_empty: %d\n" , do_empty);if (irq_affinity)printf (" irq_cpu: %d\n" , irq_cpu);
Signature · 1 line
2.53.0.1118.gaef5881109-goog
PATCH v2 9/14 KVM: selftests: Make test IRQ count configurable
In reply to Josh HilkeFrom: David Matlack <[email protected]>
Add the '-i' flag to vfio_pci_irq_test to allow users to specify the number of IRQs generated during a test run. This provides flexibility for quick smoke tests or extended stress testing to uncover rare races in posted-interrupt logic.
--- tools/testing/selftests/kvm/vfio_pci_irq_test.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)
tools/testing/selftests/kvm/vfio_pci_irq_test.c +6 -2
@@ -125,7 +125,7 @@ static void send_msi(struct vfio_pci_device *device, bool use_device_msi, int msstatic void help (const char *name) { -printf ("Usage: %s [-a] [-b] [-d] [-e] [-h] segment:bus:device.function\n" , +printf ("Usage: %s [-a] [-b] [-d] [-e] [-h] [-i nr_irqs] segment:bus:device.function\n" , name);printf ("\n" ); printf(" -a: Randomly affinitize the device IRQ to different CPUs\n"
@@ -135,6 +135,7 @@ static void help(const char *name) " it with an eventfd write.\n");printf (" -e: Destroy and recreate KVM's GSI routing table in between\n" " some interrupts.\n" ); +printf (" -i: The number of IRQs to generate during the test.\n" );printf ("\n" );exit (KSFT_FAIL ); }
@@ -176,7 +177,7 @@ int main(int argc, char **argv) device_bdf =vfio_selftests_get_bdf (&argc, argv); -while ((c =getopt (argc, argv,"abdeh" )) !=-1 ) { +while ((c =getopt (argc, argv,"abdehi:" )) !=-1 ) {switch (c) {case 'a' : irq_affinity = true;
@@ -190,6 +191,9 @@ int main(int argc, char **argv)case 'e' : empty = true;break ; +case 'i' : + nr_irqs =atoi_positive ("Number of IRQs" , optarg); +break ;case 'h' :default :help (argv[0 ]);
Signature · 1 line
2.53.0.1118.gaef5881109-goog
PATCH v2 10/14 KVM: selftests: Add support for NMI delivery
In reply to Josh HilkeFrom: David Matlack <[email protected]>
Add the '-n' flag to vfio_pci_irq_test to route device interrupts to be delivered as NMIs (Non-Maskable Interrupts) into the guest. This expands test coverage to include KVM's NMI injection and delivery logic for assigned devices.
--- .../testing/selftests/kvm/vfio_pci_irq_test.c | 43 +++++++++++++++---- 1 file changed, 34 insertions(+), 9 deletions(-)
tools/testing/selftests/kvm/vfio_pci_irq_test.c +34 -9
@@ -20,6 +20,8 @@ static bool block;static bool guest_ready_for_irqs[KVM_MAX_VCPUS ];static bool guest_received_irq[KVM_MAX_VCPUS ]; +static bool guest_received_nmi[KVM_MAX_VCPUS ]; +#define TIMEOUT_NS (2ULL * 1000 * 1000 * 1000)static u32 guest_get_vcpu_id (void )
@@ -34,6 +36,11 @@ static void guest_irq_handler(struct ex_regs *regs)x2apic_write_reg (APIC_EOI ,0 ); } +static void guest_nmi_handler (struct ex_regs *regs) +{ +WRITE_ONCE (guest_received_nmi[guest_get_vcpu_id ()], true); +} +static void guest_code (void ) {x2apic_enable ();
@@ -71,7 +78,7 @@ static void kvm_clear_gsi_routes(struct kvm_vm *vm) }static void kvm_route_msi (struct kvm_vm *vm,u32 gsi,struct kvm_vcpu *vcpu, -u8 vector) +u8 vector,bool do_nmi) {u8 buf[sizeof (struct kvm_irq_routing ) +sizeof (struct kvm_irq_routing_entry )] = {};struct kvm_irq_routing *routes = (void *)&buf;
@@ -80,7 +87,7 @@ static void kvm_route_msi(struct kvm_vm *vm, u32 gsi, struct kvm_vcpu *vcpu, routes->entries[0 ].gsi = gsi; routes->entries[0 ].type =KVM_IRQ_ROUTING_MSI ; routes->entries[0 ].u.msi.address_lo =0xFEE00000 | (vcpu->id <<12 ); - routes->entries[0 ].u.msi.data = vector; + routes->entries[0 ].u.msi.data = do_nmi ?NMI_VECTOR | (4 <<8 ) : vector;vm_ioctl (vm,KVM_SET_GSI_ROUTING , routes); }
@@ -125,7 +132,7 @@ static void send_msi(struct vfio_pci_device *device, bool use_device_msi, int msstatic void help (const char *name) { -printf ("Usage: %s [-a] [-b] [-d] [-e] [-h] [-i nr_irqs] segment:bus:device.function\n" , +printf ("Usage: %s [-a] [-b] [-d] [-e] [-h] [-i nr_irqs] [-n] segment:bus:device.function\n" , name);printf ("\n" ); printf(" -a: Randomly affinitize the device IRQ to different CPUs\n"
@@ -136,6 +143,8 @@ static void help(const char *name)printf (" -e: Destroy and recreate KVM's GSI routing table in between\n" " some interrupts.\n" );printf (" -i: The number of IRQs to generate during the test.\n" ); +printf (" -n: Route some of the device interrupts to be delivered as\n" +" an NMI into the guest.\n" );printf ("\n" );exit (KSFT_FAIL ); }
@@ -159,7 +168,7 @@ int main(int argc, char **argv)/* Test configuration (overridable by command line flags). */ bool use_device_msi = false, irq_affinity = false; -bool empty = false; +bool empty = false, nmi = false;int nr_irqs =1000 ;int nr_vcpus =1 ;
@@ -177,7 +186,7 @@ int main(int argc, char **argv) device_bdf =vfio_selftests_get_bdf (&argc, argv); -while ((c =getopt (argc, argv,"abdehi:" )) !=-1 ) { +while ((c =getopt (argc, argv,"abdehi:n" )) !=-1 ) {switch (c) {case 'a' : irq_affinity = true;
@@ -194,6 +203,9 @@ int main(int argc, char **argv)case 'i' : nr_irqs =atoi_positive ("Number of IRQs" , optarg);break ; +case 'n' : + nmi = true; +break ;case 'h' :default :help (argv[0 ]);
@@ -202,6 +214,7 @@ int main(int argc, char **argv) vm =vm_create_with_vcpus (nr_vcpus, guest_code, vcpus);vm_install_exception_handler (vm, vector, guest_irq_handler); +vm_install_exception_handler (vm,NMI_VECTOR , guest_nmi_handler); iommu =iommu_init (default_iommu_mode); device =vfio_pci_device_init (device_bdf, iommu);
@@ -243,13 +256,14 @@ int main(int argc, char **argv)for (i =0 ; i < nr_irqs; i++) {struct kvm_vcpu *vcpu = vcpus[i % nr_vcpus]; +const bool do_nmi = nmi && (i &BIT (2 ));const bool do_empty = empty && (i &BIT (3 ));struct timespec start;if (do_empty)kvm_clear_gsi_routes (vm); -kvm_route_msi (vm, gsi, vcpu, vector); +kvm_route_msi (vm, gsi, vcpu, vector, do_nmi);if (irq_affinity && vcpu->id ==0 ) { irq_cpu =rand () %get_nprocs ();
@@ -263,18 +277,26 @@ int main(int argc, char **argv) !READ_FROM_GUEST (vm, guest_received_irq[vcpu->id]),"IRQ flag for vCPU %d not clear prior to test" , vcpu->id); +TEST_ASSERT ( + !READ_FROM_GUEST (vm, guest_received_nmi[vcpu->id]), +"NMI flag for vCPU %d not clear prior to test" , + vcpu->id); }send_msi (device, use_device_msi, msi);clock_gettime (CLOCK_MONOTONIC , &start);for (;;) { -if (READ_FROM_GUEST (vm, guest_received_irq[vcpu->id])) +if (do_nmi &&READ_FROM_GUEST (vm, guest_received_nmi[vcpu->id])) +break ; + +if (!do_nmi &&READ_FROM_GUEST (vm, guest_received_irq[vcpu->id]))break ;if (timespec_to_ns (timespec_elapsed (start)) >TIMEOUT_NS ) {printf ("Timeout waiting for interrupt!\n" );printf (" vCPU: %d\n" , vcpu->id); +printf (" do_nmi: %d\n" , do_nmi);printf (" do_empty: %d\n" , do_empty);if (irq_affinity)printf (" irq_cpu: %d\n" , irq_cpu);
@@ -283,14 +305,17 @@ int main(int argc, char **argv) } } -WRITE_TO_GUEST (vm, guest_received_irq[vcpu->id], false); +if (do_nmi) +WRITE_TO_GUEST (vm, guest_received_nmi[vcpu->id], false); +else +WRITE_TO_GUEST (vm, guest_received_irq[vcpu->id], false); }WRITE_TO_GUEST (vm, done, true);for (i =0 ; i < nr_vcpus; i++) {if (block) { -kvm_route_msi (vm, gsi, vcpus[i], vector); +kvm_route_msi (vm, gsi, vcpus[i], vector, false);send_msi (device, false, msi); }
Signature · 1 line
2.53.0.1118.gaef5881109-goog
Sean Christopherson
reviewerRe: [PATCH v2 10/14] KVM: selftests: Add support for NMI delivery
Quoted 6 lines · Josh Hilke
From: David Matlack <[email protected]>
Add the '-n' flag to
vfio_pci_irq_testto route device interrupts to be delivered as NMIs (Non-Maskable Interrupts) into the guest. This expands test coverage to include KVM's NMI injection and delivery logic for assigned devices.
The original motivation for sending NMIs wasn't to validate NMI delivery, it was to validate KVM's handling of delivery types that _can't_ be posted, and more specifically KVM's handling of transitions between posted and remapped mode.
NMI just so happens to be the easy choice (SMI support isn't guaranteed, INIT is too destructive). I.e. any NMI testing is largely a bonus.
Quoted 10 lines
tools/testing/selftests/kvm/vfio_pci_irq_test.c +1
-1@@ -125,7 +132,7 @@ static void send_msi(struct vfio_pci_device *device, bool use_device_msi, int msstatic void help (const char *name) { -printf ("Usage: %s [-a] [-b] [-d] [-e] [-h] [-i nr_irqs] segment:bus:device.function\n" , +printf ("Usage: %s [-a] [-b] [-d] [-e] [-h] [-i nr_irqs] [-n] segment:bus:device.function\n" , name);printf ("\n" ); printf(" -a: Randomly affinitize the device IRQ to different CPUs\n"
tools/testing/selftests/kvm/vfio_pci_irq_test.c static void help(const char *name) · line 146
Quoted 4 lines
tools/testing/selftests/kvm/vfio_pci_irq_test.c +1
-0@@ -136,6 +143,8 @@ static void help(const char *name)printf (" -e: Destroy and recreate KVM's GSI routing table in between\n" " some interrupts.\n" );printf (" -i: The number of IRQs to generate during the test.\n" ); + printf(" -n: Route some of the device interrupts to be delivered as\n"
How much is some?
Quoted 4 lines
tools/testing/selftests/kvm/vfio_pci_irq_test.c +1
-0+ " an NMI into the guest.\n");printf ("\n" );exit (KSFT_FAIL ); }
Re: [PATCH v2 10/14] KVM: selftests: Add support for NMI delivery
Quoted 8 lines · Sean Christopherson
Quoted 6 lines · Josh Hilke
From: David Matlack <[email protected]>
Add the '-n' flag to
vfio_pci_irq_testto route device interrupts to be delivered as NMIs (Non-Maskable Interrupts) into the guest. This expands test coverage to include KVM's NMI injection and delivery logic for assigned devices.
Quoted 7 lines
The original motivation for sending NMIs wasn't to validate NMI delivery, it was to validate KVM's handling of delivery types that _can't_ be posted, and more specifically KVM's handling of transitions between posted and remapped mode.
NMI just so happens to be the easy choice (SMI support isn't guaranteed, INIT is too destructive). I.e. any NMI testing is largely a bonus.
Got it. I'll update the commit body to reflect this.
Quoted 17 lines
Quoted 10 lines
tools/testing/selftests/kvm/vfio_pci_irq_test.c +1
-1@@ -125,7 +132,7 @@ static void send_msi(struct vfio_pci_device *device, bool use_device_msi, int msstatic void help (const char *name) { -printf ("Usage: %s [-a] [-b] [-d] [-e] [-h] [-i nr_irqs] segment:bus:device.function\n" , +printf ("Usage: %s [-a] [-b] [-d] [-e] [-h] [-i nr_irqs] [-n] segment:bus:device.function\n" , name);printf ("\n" ); printf(" -a: Randomly affinitize the device IRQ to different CPUs\n"tools/testing/selftests/kvm/vfio_pci_irq_test.c static void help(const char *name) · line 146
Quoted 4 lines
tools/testing/selftests/kvm/vfio_pci_irq_test.c +1
-0@@ -136,6 +143,8 @@ static void help(const char *name)printf (" -e: Destroy and recreate KVM's GSI routing table in between\n" " some interrupts.\n" );printf (" -i: The number of IRQs to generate during the test.\n" ); + printf(" -n: Route some of the device interrupts to be delivered as\n"How much is some?
Every 4th interrupt is an NMI. I'll update the help text in v3.
Quoted 4 lines
Quoted 4 lines
tools/testing/selftests/kvm/vfio_pci_irq_test.c +1
-0+ " an NMI into the guest.\n");printf ("\n" );exit (KSFT_FAIL ); }
PATCH v2 11/14 KVM: selftests: Add support for vCPU pinning
In reply to Josh HilkeFrom: David Matlack <[email protected]>
Add the '-p' flag to vfio_pci_irq_test to pin vCPU threads to random physical CPUs throughout the test. This stresses the interaction between vCPU migration and interrupt delivery, ensuring that interrupts find their target even as the vCPU moves across the host's pCPUs.
--- .../selftests/kvm/include/kvm_syscalls.h | 7 ++ .../testing/selftests/kvm/vfio_pci_irq_test.c | 79 ++++++++++++++++++- 2 files changed, 83 insertions(+), 3 deletions(-)
tools/testing/selftests/kvm/include/kvm_syscalls.h +7 -0
@@ -73,9 +73,16 @@ static inline int kvm_dup(int fd)return new_fd; } +static inline int gettid (void ) +{ +return syscall (__NR_gettid); +} + __KVM_SYSCALL_DEFINE(munmap, 2, void *, mem, size_t, size);__KVM_SYSCALL_DEFINE (close,1 , int, fd);__KVM_SYSCALL_DEFINE (fallocate,4 , int, fd, int, mode, loff_t, offset, loff_t, len); __KVM_SYSCALL_DEFINE(ftruncate, 2, unsigned int, fd, off_t, length); +__KVM_SYSCALL_DEFINE(sched_setaffinity, 3, pid_t, pid, size_t, cpusetsize, const cpu_set_t *, mask); +__KVM_SYSCALL_DEFINE(sched_getaffinity, 3, pid_t, pid, size_t, cpusetsize, cpu_set_t *, mask);#endif /* SELFTEST_KVM_SYSCALLS_H */
tools/testing/selftests/kvm/vfio_pci_irq_test.c +76 -3
151 lines not on this page · raw message
Signature · 1 line
2.53.0.1118.gaef5881109-goog
Sean Christopherson
reviewerRe: [PATCH v2 11/14] KVM: selftests: Add support for vCPU pinning
Quoted 6 lines · Josh Hilke
From: David Matlack <[email protected]>
Add the '-p' flag to
vfio_pci_irq_testto pin vCPU threads to random physical CPUs throughout the test. This stresses the interaction between vCPU migration and interrupt delivery, ensuring that interrupts find their target even as the vCPU moves across the host's pCPUs.
This isn't "pinning" in the normal sense of the word in KVM-land. Pinning in KVM usually means affining a vCPU to _one_ pCPU and never changing that affinity. What you're doing here is *very* different; you're delibrately mucking with affinity to force pCPU migrations. That isn't bad by any means, but calling it "pinning" is super confusing, and I would argue the use of '-p' is flat out wrong. I would suggest '-m' for migrate if it's available.
Quoted 17 lines
Signed-off-by David Matlack <[email protected]>Signed-off-by Josh Hilke <[email protected]>Co-developed-by Josh Hilke <[email protected]>--- .../selftests/kvm/include/kvm_syscalls.h | 7 ++ .../testing/selftests/kvm/vfio_pci_irq_test.c | 79 ++++++++++++++++++- 2 files changed, 83 insertions(+), 3 deletions(-)tools/testing/selftests/kvm/include/kvm_syscalls.h +1
-0@@ -73,9 +73,16 @@ static inline int kvm_dup(int fd)return new_fd; } +static inline int gettid (void )
tools/testing/selftests/kvm/include/kvm_syscalls.h static inline int kvm_dup(int fd) · line 86
Quoted 10 lines
tools/testing/selftests/kvm/include/kvm_syscalls.h +6
-0+{ +return syscall (__NR_gettid); +} + __KVM_SYSCALL_DEFINE(munmap, 2, void *, mem, size_t, size);__KVM_SYSCALL_DEFINE (close,1 , int, fd);__KVM_SYSCALL_DEFINE (fallocate,4 , int, fd, int, mode, loff_t, offset, loff_t, len); __KVM_SYSCALL_DEFINE(ftruncate, 2, unsigned int, fd, off_t, length); +__KVM_SYSCALL_DEFINE(sched_setaffinity, 3, pid_t, pid, size_t, cpusetsize, const cpu_set_t *, mask); +__KVM_SYSCALL_DEFINE(sched_getaffinity, 3, pid_t, pid, size_t, cpusetsize, cpu_set_t *, mask);
Please add these separately and use then throughout, e.g. in rseq_test.c (which doesn't very similar migration shenanigans).
Quoted 15 lines
tools/testing/selftests/kvm/include/kvm_syscalls.h +0
-0#endif /* SELFTEST_KVM_SYSCALLS_H */ tools/testing/selftests/kvm/vfio_pci_irq_test.c +2
-0@@ -22,6 +22,8 @@ static bool guest_ready_for_irqs[KVM_MAX_VCPUS];static bool guest_received_irq[KVM_MAX_VCPUS ];static bool guest_received_nmi[KVM_MAX_VCPUS ]; +static pid_t vcpu_tids[KVM_MAX_VCPUS ]; +#define TIMEOUT_NS (2ULL * 1000 * 1000 * 1000)static u32 guest_get_vcpu_id (void )
tools/testing/selftests/kvm/vfio_pci_irq_test.c static void *vcpu_thread_main(void *arg) · line 69
Quoted 4 lines
tools/testing/selftests/kvm/vfio_pci_irq_test.c +1
-0@@ -64,12 +66,64 @@ static void *vcpu_thread_main(void *arg)struct kvm_vcpu *vcpu = arg;struct ucall uc; +WRITE_ONCE (vcpu_tids[vcpu->id],syscall (__NR_gettid));
Heh, what's the point of adding gettid() if you don't use it?
Quoted 15 lines
tools/testing/selftests/kvm/vfio_pci_irq_test.c +9
-0+vcpu_run (vcpu);TEST_ASSERT_EQ (UCALL_DONE ,get_ucall (vcpu, &uc));return NULL ; } +static int get_cpu (struct kvm_vcpu *vcpu) +{ +pid_t tid = vcpu_tids[vcpu->id]; +cpu_set_t cpus; +int cpu =-1 ; +int i; + +kvm_sched_getaffinity (tid,sizeof (cpus), &cpus);
tools/testing/selftests/kvm/vfio_pci_irq_test.c static void *vcpu_thread_main(void *arg) · line 94
Quoted 10 lines
tools/testing/selftests/kvm/vfio_pci_irq_test.c +10
-0+ +for (i =0 ; i <get_nprocs (); i++) { +if (!CPU_ISSET (i, &cpus)) +continue ; + +if (cpu !=-1 ) { + cpu = i; + }else { +/* vCPU is pinned to multiple CPUs */ +return -1 ;
Given that this is for debugging failures, printing -1 isn't as helpful as it could be. Ideally the test would print something similar to the output of taskset -c -p <pid>.
Quoted 6 lines
tools/testing/selftests/kvm/vfio_pci_irq_test.c +6
-0+ } + } + +return cpu; +} +
tools/testing/selftests/kvm/vfio_pci_irq_test.c static void *vcpu_thread_main(void *arg) · line 110
Quoted 10 lines
tools/testing/selftests/kvm/vfio_pci_irq_test.c +10
-0+static void pin_vcpu_threads (int nr_vcpus,int start_cpu,cpu_set_t *available_cpus) +{ +const size_t size =sizeof (cpu_set_t); +int nr_cpus, cpu, vcpu_index =0 ; +cpu_set_t target_cpu; + + nr_cpus =get_nprocs (); +CPU_ZERO (&target_cpu); + +for (cpu = start_cpu;; cpu = (cpu +1 ) % nr_cpus) {
Why round-robin the pCPUs? If anything, that _reduces_ coverage because it creates a single pattern for how vCPUs are affined. And vCPUs will never be affined to the same pCPU.
Why not fully randomize the affinity? Again, this isn't pinning, it's forced task migration.
tools/testing/selftests/kvm/vfio_pci_irq_test.c static void *vcpu_thread_main(void *arg) · line 119
Quoted 9 lines
tools/testing/selftests/kvm/vfio_pci_irq_test.c +9
-0+if (vcpu_index == nr_vcpus) +break ; + +if (!CPU_ISSET (cpu, available_cpus)) +continue ; + +CPU_SET (cpu, &target_cpu); + +kvm_sched_setaffinity (vcpu_tids[vcpu_index], size, &target_cpu);
If you're affining to _one_ pCPU, just use pin_task_to_cpu(). Actually, all of this belongs in a helper, e.g. pin_task_to_random_cpu().
Quoted 5 lines
tools/testing/selftests/kvm/vfio_pci_irq_test.c +5
-0+CPU_CLR (cpu, &target_cpu); + + vcpu_index++; + } +}
PATCH v2 12/14 KVM: selftests: Support testing with multiple vCPUs
In reply to Josh HilkeFrom: David Matlack <[email protected]>
Add the '-v' flag to vfio_pci_irq_test to support testing with multiple vCPUs. The test will round-robin the device's interrupt across all active vCPUs, ensuring that multi-processor interrupt routing and load balancing are working correctly within the guest.
--- tools/testing/selftests/kvm/vfio_pci_irq_test.c | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-)
tools/testing/selftests/kvm/vfio_pci_irq_test.c +7 -2
@@ -186,7 +186,7 @@ static void send_msi(struct vfio_pci_device *device, bool use_device_msi, int msstatic void help (const char *name) { -printf ("Usage: %s [-a] [-b] [-d] [-e] [-h] [-i nr_irqs] [-n] [-p] segment:bus:device.function\n" , +printf ("Usage: %s [-a] [-b] [-d] [-e] [-h] [-i nr_irqs] [-n] [-p] [-v nr_vcpus] segment:bus:device.function\n" , name);printf ("\n" ); printf(" -a: Randomly affinitize the device IRQ to different CPUs\n"
@@ -200,6 +200,8 @@ static void help(const char *name)printf (" -n: Route some of the device interrupts to be delivered as\n" " an NMI into the guest.\n" );printf (" -p: Pin vCPU threads to random pCPUs throughout the test.\n" ); +printf (" -v: Set the number of vCPUs that the test should create.\n" +" Interrupts will be round-robined among vCPUs.\n" );printf ("\n" );exit (KSFT_FAIL ); }
@@ -242,7 +244,7 @@ int main(int argc, char **argv) device_bdf =vfio_selftests_get_bdf (&argc, argv); -while ((c =getopt (argc, argv,"abdehi:np" )) !=-1 ) { +while ((c =getopt (argc, argv,"abdehi:npv:" )) !=-1 ) {switch (c) {case 'a' : irq_affinity = true;
@@ -265,6 +267,9 @@ int main(int argc, char **argv)case 'p' : pin_vcpus = true;break ; +case 'v' : + nr_vcpus =atoi_positive ("nr_vcpus" , optarg); +break ;case 'h' :default :help (argv[0 ]);
Signature · 1 line
2.53.0.1118.gaef5881109-goog
PATCH v2 13/14 KVM: selftests: Add xAPIC mode support
In reply to Josh HilkeFrom: David Matlack <[email protected]>
Add the '-x' flag to vfio_pci_irq_test to allow the guest to use legacy xAPIC mode instead of the default x2APIC mode. This ensures the test verifies the interrupt delivery paths for both generations of APIC architecture in the KVM selftest environment.
--- .../testing/selftests/kvm/vfio_pci_irq_test.c | 28 +++++++++++++++---- 1 file changed, 23 insertions(+), 5 deletions(-)
tools/testing/selftests/kvm/vfio_pci_irq_test.c +23 -5
@@ -15,6 +15,7 @@#include <libvfio.h> +static bool x2apic = true;static bool done;static bool block;
@@ -28,14 +29,20 @@ static pid_t vcpu_tids[KVM_MAX_VCPUS];static u32 guest_get_vcpu_id (void ) { -return x2apic_read_reg (APIC_ID ); +if (x2apic) +return x2apic_read_reg (APIC_ID ); +else +return xapic_read_reg (APIC_ID ) >>24 ; }static void guest_irq_handler (struct ex_regs *regs) {WRITE_ONCE (guest_received_irq[guest_get_vcpu_id ()], true); -x2apic_write_reg (APIC_EOI ,0 ); +if (x2apic) +x2apic_write_reg (APIC_EOI ,0 ); +else +xapic_write_reg (APIC_EOI ,0 ); }static void guest_nmi_handler (struct ex_regs *regs)
@@ -45,7 +52,10 @@ static void guest_nmi_handler(struct ex_regs *regs)static void guest_code (void ) { -x2apic_enable (); +if (x2apic) +x2apic_enable (); +else +xapic_enable ();sti_nop ();
@@ -186,7 +196,7 @@ static void send_msi(struct vfio_pci_device *device, bool use_device_msi, int msstatic void help (const char *name) { -printf ("Usage: %s [-a] [-b] [-d] [-e] [-h] [-i nr_irqs] [-n] [-p] [-v nr_vcpus] segment:bus:device.function\n" , +printf ("Usage: %s [-a] [-b] [-d] [-e] [-h] [-i nr_irqs] [-n] [-p] [-v nr_vcpus] [-x] segment:bus:device.function\n" , name);printf ("\n" ); printf(" -a: Randomly affinitize the device IRQ to different CPUs\n"
@@ -202,6 +212,7 @@ static void help(const char *name)printf (" -p: Pin vCPU threads to random pCPUs throughout the test.\n" );printf (" -v: Set the number of vCPUs that the test should create.\n" " Interrupts will be round-robined among vCPUs.\n" ); +printf (" -x: Use xAPIC mode instead of x2APIC mode in the guest.\n" );printf ("\n" );exit (KSFT_FAIL ); }
@@ -244,7 +255,7 @@ int main(int argc, char **argv) device_bdf =vfio_selftests_get_bdf (&argc, argv); -while ((c =getopt (argc, argv,"abdehi:npv:" )) !=-1 ) { +while ((c =getopt (argc, argv,"abdehi:npv:x" )) !=-1 ) {switch (c) {case 'a' : irq_affinity = true;
@@ -270,6 +281,9 @@ int main(int argc, char **argv)case 'v' : nr_vcpus =atoi_positive ("nr_vcpus" , optarg);break ; +case 'x' : + x2apic = false; +break ;case 'h' :default :help (argv[0 ]);
@@ -280,6 +294,9 @@ int main(int argc, char **argv)vm_install_exception_handler (vm, vector, guest_irq_handler);vm_install_exception_handler (vm,NMI_VECTOR , guest_nmi_handler); +if (!x2apic) +virt_pg_map (vm,APIC_DEFAULT_GPA ,APIC_DEFAULT_GPA ); + iommu =iommu_init (default_iommu_mode); device =vfio_pci_device_init (device_bdf, iommu); msi =setup_msi (device, use_device_msi);
@@ -295,6 +312,7 @@ int main(int argc, char **argv)kvm_assign_irqfd (vm, gsi, device->msi_eventfds[msi]); +sync_global_to_guest (vm, x2apic);sync_global_to_guest (vm, block);for (i =0 ; i < nr_vcpus; i++)
Signature · 1 line
2.53.0.1118.gaef5881109-goog
PATCH v2 14/14 KVM: selftests: Make vfio_pci_irq_test timeout configurable
In reply to Josh HilkeAdd a '-t' flag to vfio_pci_irq_test to allow configuring the interrupt timeout. This provides flexibility for running the test in different environments where interrupt delivery times may vary.
--- tools/testing/selftests/kvm/vfio_pci_irq_test.c | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-)
tools/testing/selftests/kvm/vfio_pci_irq_test.c +8 -4
@@ -25,7 +25,7 @@ static bool guest_received_nmi[KVM_MAX_VCPUS];static pid_t vcpu_tids[KVM_MAX_VCPUS ]; -#define TIMEOUT_NS (2ULL * 1000 * 1000 * 1000) +static u64 timeout_ns =2ULL *1000 *1000 *1000 ;static u32 guest_get_vcpu_id (void ) {
@@ -196,7 +196,7 @@ static void send_msi(struct vfio_pci_device *device, bool use_device_msi, int msstatic void help (const char *name) { -printf ("Usage: %s [-a] [-b] [-d] [-e] [-h] [-i nr_irqs] [-n] [-p] [-v nr_vcpus] [-x] segment:bus:device.function\n" , +printf ("Usage: %s [-a] [-b] [-d] [-e] [-h] [-i nr_irqs] [-n] [-p] [-t timeout] [-v nr_vcpus] [-x] segment:bus:device.function\n" , name);printf ("\n" ); printf(" -a: Randomly affinitize the device IRQ to different CPUs\n"
@@ -210,6 +210,7 @@ static void help(const char *name)printf (" -n: Route some of the device interrupts to be delivered as\n" " an NMI into the guest.\n" );printf (" -p: Pin vCPU threads to random pCPUs throughout the test.\n" ); +printf (" -t: The timeout in seconds to wait for an interrupt.\n" );printf (" -v: Set the number of vCPUs that the test should create.\n" " Interrupts will be round-robined among vCPUs.\n" );printf (" -x: Use xAPIC mode instead of x2APIC mode in the guest.\n" );
@@ -255,7 +256,7 @@ int main(int argc, char **argv) device_bdf =vfio_selftests_get_bdf (&argc, argv); -while ((c =getopt (argc, argv,"abdehi:npv:x" )) !=-1 ) { +while ((c =getopt (argc, argv,"abdehi:npv:xt:" )) !=-1 ) {switch (c) {case 'a' : irq_affinity = true;
@@ -278,6 +279,9 @@ int main(int argc, char **argv)case 'p' : pin_vcpus = true;break ; +case 't' : + timeout_ns = (u64 )atoi_positive ("Timeout (seconds)" , optarg) *1000ULL *1000 *1000 ; +break ;case 'v' : nr_vcpus =atoi_positive ("nr_vcpus" , optarg);break ;
@@ -387,7 +391,7 @@ int main(int argc, char **argv)if (!do_nmi &&READ_FROM_GUEST (vm, guest_received_irq[vcpu->id]))break ; -if (timespec_to_ns (timespec_elapsed (start)) >TIMEOUT_NS ) { +if (timespec_to_ns (timespec_elapsed (start)) > timeout_ns) {printf ("Timeout waiting for interrupt!\n" );printf (" vCPU: %d\n" , vcpu->id);printf (" do_nmi: %d\n" , do_nmi);
Signature · 1 line
2.53.0.1118.gaef5881109-goog
Sean Christopherson
reviewerRe: [PATCH v2 14/14] KVM: selftests: Make vfio_pci_irq_test timeout configurable
Quoted 15 lines · Josh Hilke
Add a '-t' flag to
vfio_pci_irq_testto allow configuring the interrupt timeout. This provides flexibility for running the test in different environments where interrupt delivery times may vary.Signed-off-by David Matlack <[email protected]>Signed-off-by Josh Hilke <[email protected]>--- tools/testing/selftests/kvm/vfio_pci_irq_test.c | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-)tools/testing/selftests/kvm/vfio_pci_irq_test.c +0
-0
tools/testing/selftests/kvm/vfio_pci_irq_test.c static bool guest_received_nmi[KVM_MAX_VCPUS]; · line 28
Quoted 4 lines
tools/testing/selftests/kvm/vfio_pci_irq_test.c +0
-1@@ -25,7 +25,7 @@ static bool guest_received_nmi[KVM_MAX_VCPUS];static pid_t vcpu_tids[KVM_MAX_VCPUS ]; -#define TIMEOUT_NS (2ULL * 1000 * 1000 * 1000)
Aren't there defines for this sort of thing?
Quoted 14 lines
tools/testing/selftests/kvm/vfio_pci_irq_test.c +2
-1+static u64 timeout_ns =2ULL *1000 *1000 *1000 ;static u32 guest_get_vcpu_id (void ) {@@ -196,7 +196,7 @@ static void send_msi(struct vfio_pci_device *device, bool use_device_msi, int msstatic void help (const char *name) { -printf ("Usage: %s [-a] [-b] [-d] [-e] [-h] [-i nr_irqs] [-n] [-p] [-v nr_vcpus] [-x] segment:bus:device.function\n" , +printf ("Usage: %s [-a] [-b] [-d] [-e] [-h] [-i nr_irqs] [-n] [-p] [-t timeout] [-v nr_vcpus] [-x] segment:bus:device.function\n" , name);printf ("\n" ); printf(" -a: Randomly affinitize the device IRQ to different CPUs\n"
tools/testing/selftests/kvm/vfio_pci_irq_test.c static void help(const char *name) · line 213
Quoted 4 lines
tools/testing/selftests/kvm/vfio_pci_irq_test.c +1
-0@@ -210,6 +210,7 @@ static void help(const char *name)printf (" -n: Route some of the device interrupts to be delivered as\n" " an NMI into the guest.\n" );printf (" -p: Pin vCPU threads to random pCPUs throughout the test.\n" ); +printf (" -t: The timeout in seconds to wait for an interrupt.\n" );
Uh, for *each* interrupt? That's insane. Something is massively wrong if an IRQ isn't delivered in waaay under a second. If this is purely a "sometimes IRQs get delayed for a long time because the host is massively overloaded", set the timeout to something absurd, e.g. 10 seconds, and call it good. Then rely on the runner to deal with the total runtime.
Sean Christopherson
reviewerQuoted 5 lines · Josh Hilke
I am taking over development of this series from David Matlack.
v1: https://lore.kernel.org/kvm/[email protected]/This series is based on the kvm-upstream/queue branch.
Quoted 7 lines
This series introduces
vfio_pci_irq_test, which exercises the delivery of VFIO device interrupts (MSI/MSI-X) to guest vCPUs, and links the VFIO selftests library into KVM selftests to enable testing of VFIO-KVM interactions.The test requires a PCI device bound to vfio-pci. It can be invoked by passing the BDF the VFIO selftests setup script,
-ENOPARSE
Quoted 3 lines
and then running the test with the device BDF:
$ ./tools/testing/selftests/vfio/scripts/setup.sh 0000:6a:01.0
IIUC, what this is saying is that to run the test, the device needs to be bound to VFIO, and _one_ way to do that is to use ./tools/testing/selftests/vfio/scripts/setup.sh to do the heavy lifting.
Quoted 5 lines
$ tools/testing/selftests/kvm/vfio_pci_irq_test 0000:6a:01.0This test only supports x86. Testing physical device interrupts (-d argument) is only supported for Intel because there are no VFIO selftest drivers for AMD devices.
This needs to be expressed in code and human readable error messages to the user. I already knew the basic of what the VFIO selfests are doing, and it still took me 5+ minutes to find the DSA driver and probing logic.
Concretely, this code needs to be part of the standard vfio selftests APIs:
device = vfio_pci_device_init(device_bdf, iommu); has_driver = !!device->driver.ops;
E.g. have vfio_pci_device_init() explicitly return an error (or NULL) if it can't init the device driver.
The fixtures in vfio_pci_driver_test.c already blindly assume success, so you might not need to do much beyond changing main() in that file.
And then, also as a vfio_pci_device_xxx() API, add a helper to print out what devices are supported, e.g. so that KVM selftests can do something like:
struct vfio_pci_device *kvm_vfio_device_init(const char *bdf)
{
struct vfio_pci_device *device;
struct iommu *iommu;
iommu = iommu_init(default_iommu_mode);
device = vfio_pci_device_init(bdf, iommu);
if (!device) {
vfio_pci_device_print_drivers(stderr);
TEST_FAIL("No driver found for BDF '%s'", bdf);
}
return device;
}Because as is, this requires way too much a priori knowledge and magic for upstream. The user shouldn't have to dig through code just to understand what devices are supported.
Ideally, VFIO selftests would also provide a script, utility, and/or helper to identify BDFs for devices it has drivers for. In my experience, binding a device to VFIO is trivial. Finding the device in the first place is more annoying.
Quoted 3 lines
David Matlack (11):
KVM: selftests: Build and link sefltests/vfio/lib into KVM selftests
KVM: selftests: Addvfio_pci_irq_test
Make this one human friendly, e.g.
KVM: selftests: Add a test to verify IRQ delivery from assigned VFIO devices
Quoted 9 lines
KVM: selftests: Add support for random host IRQ affinity
KVM: selftests: Allow blocking vCPUs via HLT
KVM: selftests: Add support for physical device MSI triggers
KVM: selftests: Add option to clear GSI routes
KVM: selftests: Make test IRQ count configurable
KVM: selftests: Add support for NMI delivery
KVM: selftests: Add support for vCPU pinning
KVM: selftests: Support testing with multiple vCPUs
KVM: selftests: Add xAPIC mode support
And then for these shortlog, explicitly call out what test they are changing. The "Allow blocking vCPUs via HLT" shortlog in particular was super confusing. E.g.
KVM: selftests: Extend VFIO IRQ test to validate waking vCPU from HLT
Quoted 3 lines
Josh Hilke (3):
KVM: selftests: Add helper functions for IRQ testing
KVM: selftests: Reproduce tests that rely on randomization
Please post this as a separate, standalone patch. It's a fix that isn't dependent on this series in any way.
Quoted 1 line
KVM: selftests: Make
vfio_pci_irq_testtimeout configurable
Quoted 10 lines · Sean Christopherson
Quoted 3 lines · Josh Hilke
This test only supports x86. Testing physical device interrupts (-d argument) is only supported for Intel because there are no VFIO selftest drivers for AMD devices.
This needs to be expressed in code and human readable error messages to the user. I already knew the basic of what the VFIO selfests are doing, and it still took me 5+ minutes to find the DSA driver and probing logic.
Quoted 8 lines
Concretely, this code needs to be part of the standard vfio selftests APIs:
device = vfio_pci_device_init(device_bdf, iommu); has_driver = !!device->driver.ops;E.g. have
vfio_pci_device_init()explicitly return an error (or NULL) if it can't init the device driver.
No. Not every VFIO selftest or user of libvfio needs a driver. A driver is only needed in tests that want to use the driver, i.e. want to call one of the very obviously named functions:
- vfio_pci_driver_send_msi()
- vfio_pci_driver_memcpy()
- etc.
Requiring a driver for vfio_pci_device_init() to return non-NULL would significantly reduce our ability to test VFIO on different platforms and environments.
Quoted 2 lines
The fixtures in
vfio_pci_driver_test.calready blindly assume success, so you might not need to do much beyond changingmain()in that file.
I don't follow. What is the issue with vfio_pci_driver_test.?
Quoted 2 lines
And then, also as a
vfio_pci_device_xxx()API, add a helper to print out what devices are supported, e.g. so that KVM selftests can do something like:
Agree this would be nice to have. Is this a blocker for this series?
Quoted 15 lines
struct
vfio_pci_device*kvm_vfio_device_init(const char *bdf){ struct vfio_pci_device *device; struct iommu *iommu; iommu = iommu_init(default_iommu_mode); device = vfio_pci_device_init(bdf, iommu); if (!device) { vfio_pci_device_print_drivers(stderr); TEST_FAIL("No driver found for BDF '%s'", bdf); } return device; }
Quoted 7 lines
Because as is, this requires way too much a priori knowledge and magic for upstream. The user shouldn't have to dig through code just to understand what devices are supported.
Ideally, VFIO selftests would also provide a script, utility, and/or helper to identify BDFs for devices it has drivers for. In my experience, binding a device to VFIO is trivial. Finding the device in the first place is more annoying.
Agree this would be nice to have. Is this a blocker for this series?
Sean Christopherson
reviewerQuoted 24 lines · David Matlack
Quoted 10 lines · Sean Christopherson
Quoted 3 lines · Josh Hilke
This test only supports x86. Testing physical device interrupts (-d argument) is only supported for Intel because there are no VFIO selftest drivers for AMD devices.
This needs to be expressed in code and human readable error messages to the user. I already knew the basic of what the VFIO selfests are doing, and it still took me 5+ minutes to find the DSA driver and probing logic.
Quoted 8 lines
Concretely, this code needs to be part of the standard vfio selftests APIs:
device = vfio_pci_device_init(device_bdf, iommu); has_driver = !!device->driver.ops;E.g. have
vfio_pci_device_init()explicitly return an error (or NULL) if it can't init the device driver.No. Not every VFIO selftest or user of libvfio needs a driver. A driver is only needed in tests that want to use the driver, i.e. want to call one of the very obviously named functions:
Quoted 7 lines
-
vfio_pci_driver_send_msi()
-vfio_pci_driver_memcpy()
- etc.Requiring a driver for
vfio_pci_device_init()to return non-NULL would significantly reduce our ability to test VFIO on different platforms and environments.
Ah, sorry. I was speed reading and missed that some (most?) tests don't require a driver. Scratch the "make it unconditional", but there still needs to be an API to up-level binding to a driver. KVM selftests shouldn't have to manually check for that.
Quoted 4 lines
Quoted 2 lines
The fixtures in
vfio_pci_driver_test.calready blindly assume success, so you might not need to do much beyond changingmain()in that file.I don't follow. What is the issue with
vfio_pci_driver_test.?
What I was trying to say is that if vfio_pci_device_init() were to unconditionally require a driver, then vfio_pci_driver_test() should Just Work. But that doesn't make a whole lot of sense given the above. Though it could use the new API that checks for a driver.
Quoted 4 lines
Quoted 2 lines
And then, also as a
vfio_pci_device_xxx()API, add a helper to print out what devices are supported, e.g. so that KVM selftests can do something like:Agree this would be nice to have. Is this a blocker for this series?
Yes. KVM selftests need to assert there's a driver before trying to proceed.
Quoted 19 lines
Quoted 19 lines
struct
vfio_pci_device*kvm_vfio_device_init(const char *bdf){ struct vfio_pci_device *device; struct iommu *iommu; iommu = iommu_init(default_iommu_mode); device = vfio_pci_device_init(bdf, iommu); if (!device) { vfio_pci_device_print_drivers(stderr); TEST_FAIL("No driver found for BDF '%s'", bdf); } return device; }Because as is, this requires way too much a priori knowledge and magic for upstream. The user shouldn't have to dig through code just to understand what devices are supported.
Quoted 5 lines
Quoted 3 lines
Ideally, VFIO selftests would also provide a script, utility, and/or helper to identify BDFs for devices it has drivers for. In my experience, binding a device to VFIO is trivial. Finding the device in the first place is more annoying.
Agree this would be nice to have. Is this a blocker for this series?
Yes. As I mentioned to Josh off-list, I'm not willing to accept a "we pinky-swear we'll make this stuff user friendly". It's not that I don't trust you and Josh (and others), it's that for me, this level of user-friendly behavior isn't nice to have, it's mandatory for these tests to be usable upstream.
Internally we can squeak by with barebones, unhelpful tests, because the run commands are often hardcoded somewhere and there is piles of documentation elsewhere. But for upstream, none of that holds true.
If it were weeks of effort, I would be more open to treating this as nice to have, but AFAICT writing the code should be less than a days worth of work.
Quoted 7 lines · Sean Christopherson
Quoted 4 lines · David Matlack
Quoted 1 line · Sean Christopherson
On Tue, Mar 31, 2026, Josh Hilke wrote:
Agree this would be nice to have. Is this a blocker for this series?
Yes. KVM selftests need to assert there's a driver before trying to proceed.
That exists in the new test:
static int setup_msi(struct vfio_pci_device *device, bool use_device_msi)
{
const int flags = MAP_SHARED | MAP_ANONYMOUS;
const int prot = PROT_READ | PROT_WRITE;
struct dma_region *region;
if (use_device_msi) {
/* A driver is required to generate an MSI. */
TEST_REQUIRE(device->driver.ops);But I think you are also asking to have a helper that prints out the list of available drivers in this case. That should be trivial to add.
VFIO selftests could expose add a helper that handles checking for device->driver.ops and, if null, printing an error message and list of available drivers and exiting with KSFT_SKIP.
One wrinkle here is we are probably going to merge a driver soon that does not support send_msi():
https://lore.kernel.org/kvm/[email protected]/
So we should also have an API for that. e.g. Add 2 helpsers:
- vfio_pci_has_driver()
- vfio_pci_has_driver_send_msi()
This test would use the latter.
Quoted 30 lines
Quoted 19 lines
Quoted 19 lines
struct
vfio_pci_device*kvm_vfio_device_init(const char *bdf){ struct vfio_pci_device *device; struct iommu *iommu; iommu = iommu_init(default_iommu_mode); device = vfio_pci_device_init(bdf, iommu); if (!device) { vfio_pci_device_print_drivers(stderr); TEST_FAIL("No driver found for BDF '%s'", bdf); } return device; }Because as is, this requires way too much a priori knowledge and magic for upstream. The user shouldn't have to dig through code just to understand what devices are supported.
Quoted 5 lines
Quoted 3 lines
Ideally, VFIO selftests would also provide a script, utility, and/or helper to identify BDFs for devices it has drivers for. In my experience, binding a device to VFIO is trivial. Finding the device in the first place is more annoying.
Agree this would be nice to have. Is this a blocker for this series?
Yes. As I mentioned to Josh off-list, I'm not willing to accept a "we pinky-swear we'll make this stuff user friendly". It's not that I don't trust you and Josh (and others), it's that for me, this level of user-friendly behavior isn't nice to have, it's mandatory for these tests to be usable upstream.
Quoted 6 lines
Internally we can squeak by with barebones, unhelpful tests, because the run commands are often hardcoded somewhere and there is piles of documentation elsewhere. But for upstream, none of that holds true.
If it were weeks of effort, I would be more open to treating this as nice to have, but AFAICT writing the code should be less than a days worth of work.
Yeah that's fine, I don't think it adds too much extra work, and it's something I'd like to have anyway. I just wanted to confirm if it should be part (or merged ahead of this series) or can be separate. Thanks!
Quoted 32 lines · David Matlack
Quoted 7 lines · Sean Christopherson
Quoted 4 lines · David Matlack
Quoted 1 line · Sean Christopherson
On Tue, Mar 31, 2026, Josh Hilke wrote:
Agree this would be nice to have. Is this a blocker for this series?
Yes. KVM selftests need to assert there's a driver before trying to proceed.
That exists in the new test:
static int
setup_msi(struct vfio_pci_device *device, bool use_device_msi){ const int flags = MAP_SHARED | MAP_ANONYMOUS; const int prot = PROT_READ | PROT_WRITE; struct dma_region *region; if (use_device_msi) { /* A driver is required to generate an MSI. */ TEST_REQUIRE(device->driver.ops);But I think you are also asking to have a helper that prints out the list of available drivers in this case. That should be trivial to add.
VFIO selftests could expose add a helper that handles checking for
device->driver.opsand, if null, printing an error message and list of available drivers and exiting withKSFT_SKIP.One wrinkle here is we are probably going to merge a driver soon that does not support
send_msi():
Quoted 8 lines
https://lore.kernel.org/kvm/[email protected]/
So we should also have an API for that. e.g. Add 2 helpsers:
-
vfio_pci_has_driver()
-vfio_pci_has_driver_send_msi()This test would use the latter.
Ack. I'll add a patch in the series to add these functions to the vfio library in v3.
Quoted 35 lines
Quoted 35 lines
Quoted 20 lines
Quoted 20 lines
struct
vfio_pci_device*kvm_vfio_device_init(const char *bdf) {
structvfio_pci_device*device;
struct iommu *iommu;iommu = iommu_init(default_iommu_mode);device =
vfio_pci_device_init(bdf, iommu);if (!device) { vfio_pci_device_print_drivers(stderr); TEST_FAIL("No driver found for BDF '%s'", bdf);}
return device;
}Because as is, this requires way too much a priori knowledge and magic for upstream. The user shouldn't have to dig through code just to understand what devices are supported.
Quoted 5 lines
Quoted 3 lines
Ideally, VFIO selftests would also provide a script, utility, and/or helper to identify BDFs for devices it has drivers for. In my experience, binding a device to VFIO is trivial. Finding the device in the first place is more annoying.
Agree this would be nice to have. Is this a blocker for this series?
Yes. As I mentioned to Josh off-list, I'm not willing to accept a "we pinky-swear we'll make this stuff user friendly". It's not that I don't trust you and Josh (and others), it's that for me, this level of user-friendly behavior isn't nice to have, it's mandatory for these tests to be usable upstream.
Internally we can squeak by with barebones, unhelpful tests, because the run commands are often hardcoded somewhere and there is piles of documentation elsewhere. But for upstream, none of that holds true.
Quoted 6 lines
Quoted 2 lines
If it were weeks of effort, I would be more open to treating this as nice to have, but AFAICT writing the code should be less than a days worth of work.
Yeah that's fine, I don't think it adds too much extra work, and it's something I'd like to have anyway. I just wanted to confirm if it should be part (or merged ahead of this series) or can be separate. Thanks!
So I need to add kvm_vfio_device_init() and it will call vfio_pci_has_driver(). Then, kvm_vfio_device_init() will TEST_FAIL if vfio_pci_has_driver() returns false, correct?
Should kvm_vfio_device_init() live intools/testing/selftests/kvm/lib/kvm_util.c?
Quoted 88 lines · Josh Hilke
Quoted 32 lines · David Matlack
Quoted 7 lines · Sean Christopherson
Quoted 4 lines · David Matlack
Quoted 1 line · Sean Christopherson
On Tue, Mar 31, 2026, Josh Hilke wrote:
Agree this would be nice to have. Is this a blocker for this series?
Yes. KVM selftests need to assert there's a driver before trying to proceed.
That exists in the new test:
static int
setup_msi(struct vfio_pci_device *device, bool use_device_msi){ const int flags = MAP_SHARED | MAP_ANONYMOUS; const int prot = PROT_READ | PROT_WRITE; struct dma_region *region; if (use_device_msi) { /* A driver is required to generate an MSI. */ TEST_REQUIRE(device->driver.ops);But I think you are also asking to have a helper that prints out the list of available drivers in this case. That should be trivial to add.
VFIO selftests could expose add a helper that handles checking for
device->driver.opsand, if null, printing an error message and list of available drivers and exiting withKSFT_SKIP.One wrinkle here is we are probably going to merge a driver soon that does not support
send_msi():Quoted 8 lines
https://lore.kernel.org/kvm/[email protected]/
So we should also have an API for that. e.g. Add 2 helpsers:
-
vfio_pci_has_driver()
-vfio_pci_has_driver_send_msi()This test would use the latter.
Ack. I'll add a patch in the series to add these functions to the vfio library in v3.
Quoted 35 lines
Quoted 35 lines
Quoted 20 lines
Quoted 20 lines
struct
vfio_pci_device*kvm_vfio_device_init(const char *bdf) {
structvfio_pci_device*device;
struct iommu *iommu;iommu = iommu_init(default_iommu_mode);device =
vfio_pci_device_init(bdf, iommu);if (!device) { vfio_pci_device_print_drivers(stderr); TEST_FAIL("No driver found for BDF '%s'", bdf);}
return device;
}Because as is, this requires way too much a priori knowledge and magic for upstream. The user shouldn't have to dig through code just to understand what devices are supported.
Quoted 5 lines
Quoted 3 lines
Ideally, VFIO selftests would also provide a script, utility, and/or helper to identify BDFs for devices it has drivers for. In my experience, binding a device to VFIO is trivial. Finding the device in the first place is more annoying.
Agree this would be nice to have. Is this a blocker for this series?
Yes. As I mentioned to Josh off-list, I'm not willing to accept a "we pinky-swear we'll make this stuff user friendly". It's not that I don't trust you and Josh (and others), it's that for me, this level of user-friendly behavior isn't nice to have, it's mandatory for these tests to be usable upstream.
Internally we can squeak by with barebones, unhelpful tests, because the run commands are often hardcoded somewhere and there is piles of documentation elsewhere. But for upstream, none of that holds true.
Quoted 6 lines
Quoted 2 lines
If it were weeks of effort, I would be more open to treating this as nice to have, but AFAICT writing the code should be less than a days worth of work.
Yeah that's fine, I don't think it adds too much extra work, and it's something I'd like to have anyway. I just wanted to confirm if it should be part (or merged ahead of this series) or can be separate. Thanks!
Quoted 6 lines
So I need to add
kvm_vfio_device_init()and it will callvfio_pci_has_driver(). Then,kvm_vfio_device_init()willTEST_FAILifvfio_pci_has_driver()returns false, correct?Should
kvm_vfio_device_init()live intools/testing/selftests/kvm/lib/kvm_util.c?
I was thinking the new test would do:
TEST_REQUIRE(vfio_pci_has_driver_send_msi(device));
instead of:
TEST_REQUIRE(device->driver.ops);
vfio_pci_has_driver_send_msi() would call vfio_pci_has_driver() internally and also check that send_msi op is non-null. Both functions would also log error messages if driver/send_msi is missing to help the user, as Sean requested.
And you'll also need to a patch to add a helper script to print the list of BDFs on the current host that have a VFIO selftests driver. It might make sense to send that as a standalone patch if Sean is ok with that.
Quoted 109 lines · David Matlack
Quoted 88 lines · Josh Hilke
Quoted 32 lines · David Matlack
Quoted 7 lines · Sean Christopherson
Quoted 4 lines · David Matlack
Quoted 1 line · Sean Christopherson
On Tue, Mar 31, 2026, Josh Hilke wrote:
Agree this would be nice to have. Is this a blocker for this series?
Yes. KVM selftests need to assert there's a driver before trying to proceed.
That exists in the new test:
static int
setup_msi(struct vfio_pci_device *device, bool use_device_msi){ const int flags = MAP_SHARED | MAP_ANONYMOUS; const int prot = PROT_READ | PROT_WRITE; struct dma_region *region; if (use_device_msi) { /* A driver is required to generate an MSI. */ TEST_REQUIRE(device->driver.ops);But I think you are also asking to have a helper that prints out the list of available drivers in this case. That should be trivial to add.
VFIO selftests could expose add a helper that handles checking for
device->driver.opsand, if null, printing an error message and list of available drivers and exiting withKSFT_SKIP.One wrinkle here is we are probably going to merge a driver soon that does not support
send_msi():Quoted 8 lines
https://lore.kernel.org/kvm/[email protected]/
So we should also have an API for that. e.g. Add 2 helpsers:
-
vfio_pci_has_driver()
-vfio_pci_has_driver_send_msi()This test would use the latter.
Ack. I'll add a patch in the series to add these functions to the vfio library in v3.
Quoted 35 lines
Quoted 35 lines
Quoted 20 lines
Quoted 20 lines
struct
vfio_pci_device*kvm_vfio_device_init(const char *bdf) {
structvfio_pci_device*device;
struct iommu *iommu;iommu = iommu_init(default_iommu_mode);device =
vfio_pci_device_init(bdf, iommu);if (!device) { vfio_pci_device_print_drivers(stderr); TEST_FAIL("No driver found for BDF '%s'", bdf);}
return device;
}Because as is, this requires way too much a priori knowledge and magic for upstream. The user shouldn't have to dig through code just to understand what devices are supported.
Quoted 5 lines
Quoted 3 lines
Ideally, VFIO selftests would also provide a script, utility, and/or helper to identify BDFs for devices it has drivers for. In my experience, binding a device to VFIO is trivial. Finding the device in the first place is more annoying.
Agree this would be nice to have. Is this a blocker for this series?
Yes. As I mentioned to Josh off-list, I'm not willing to accept a "we pinky-swear we'll make this stuff user friendly". It's not that I don't trust you and Josh (and others), it's that for me, this level of user-friendly behavior isn't nice to have, it's mandatory for these tests to be usable upstream.
Internally we can squeak by with barebones, unhelpful tests, because the run commands are often hardcoded somewhere and there is piles of documentation elsewhere. But for upstream, none of that holds true.
Quoted 6 lines
Quoted 2 lines
If it were weeks of effort, I would be more open to treating this as nice to have, but AFAICT writing the code should be less than a days worth of work.
Yeah that's fine, I don't think it adds too much extra work, and it's something I'd like to have anyway. I just wanted to confirm if it should be part (or merged ahead of this series) or can be separate. Thanks!
Quoted 6 lines
So I need to add
kvm_vfio_device_init()and it will callvfio_pci_has_driver(). Then,kvm_vfio_device_init()willTEST_FAILifvfio_pci_has_driver()returns false, correct?Should
kvm_vfio_device_init()live intools/testing/selftests/kvm/lib/kvm_util.c?I was thinking the new test would do:
TEST_REQUIRE(vfio_pci_has_driver_send_msi(device));instead of:
TEST_REQUIRE(device->driver.ops);
vfio_pci_has_driver_send_msi()would callvfio_pci_has_driver()internally and also check thatsend_msiop is non-null. Both functions would also log error messages if driver/send_msi is missing to help the user, as Sean requested.
Quoted 4 lines
And you'll also need to a patch to add a helper script to print the list of BDFs on the current host that have a VFIO selftests driver. It might make sense to send that as a standalone patch if Sean is ok with that.
That makes sense for setup_msi(). But in Sean's implementation of kvm_vfio_device_init() above, it looks like he wants the test to fail if there's no VFIO selftest driver for the BDF passed into the test.
Quoted 111 lines · Josh Hilke
Quoted 109 lines · David Matlack
Quoted 88 lines · Josh Hilke
Quoted 32 lines · David Matlack
Quoted 7 lines · Sean Christopherson
Quoted 4 lines · David Matlack
Quoted 1 line · Sean Christopherson
On Tue, Mar 31, 2026, Josh Hilke wrote:
Agree this would be nice to have. Is this a blocker for this series?
Yes. KVM selftests need to assert there's a driver before trying to proceed.
That exists in the new test:
static int
setup_msi(struct vfio_pci_device *device, bool use_device_msi){ const int flags = MAP_SHARED | MAP_ANONYMOUS; const int prot = PROT_READ | PROT_WRITE; struct dma_region *region; if (use_device_msi) { /* A driver is required to generate an MSI. */ TEST_REQUIRE(device->driver.ops);But I think you are also asking to have a helper that prints out the list of available drivers in this case. That should be trivial to add.
VFIO selftests could expose add a helper that handles checking for
device->driver.opsand, if null, printing an error message and list of available drivers and exiting withKSFT_SKIP.One wrinkle here is we are probably going to merge a driver soon that does not support
send_msi():Quoted 8 lines
https://lore.kernel.org/kvm/[email protected]/
So we should also have an API for that. e.g. Add 2 helpsers:
-
vfio_pci_has_driver()
-vfio_pci_has_driver_send_msi()This test would use the latter.
Ack. I'll add a patch in the series to add these functions to the vfio library in v3.
Quoted 35 lines
Quoted 35 lines
Quoted 20 lines
Quoted 20 lines
struct
vfio_pci_device*kvm_vfio_device_init(const char *bdf) {
structvfio_pci_device*device;
struct iommu *iommu;iommu = iommu_init(default_iommu_mode);device =
vfio_pci_device_init(bdf, iommu);if (!device) { vfio_pci_device_print_drivers(stderr); TEST_FAIL("No driver found for BDF '%s'", bdf);}
return device;
}Because as is, this requires way too much a priori knowledge and magic for upstream. The user shouldn't have to dig through code just to understand what devices are supported.
Quoted 5 lines
Quoted 3 lines
Ideally, VFIO selftests would also provide a script, utility, and/or helper to identify BDFs for devices it has drivers for. In my experience, binding a device to VFIO is trivial. Finding the device in the first place is more annoying.
Agree this would be nice to have. Is this a blocker for this series?
Yes. As I mentioned to Josh off-list, I'm not willing to accept a "we pinky-swear we'll make this stuff user friendly". It's not that I don't trust you and Josh (and others), it's that for me, this level of user-friendly behavior isn't nice to have, it's mandatory for these tests to be usable upstream.
Internally we can squeak by with barebones, unhelpful tests, because the run commands are often hardcoded somewhere and there is piles of documentation elsewhere. But for upstream, none of that holds true.
Quoted 6 lines
Quoted 2 lines
If it were weeks of effort, I would be more open to treating this as nice to have, but AFAICT writing the code should be less than a days worth of work.
Yeah that's fine, I don't think it adds too much extra work, and it's something I'd like to have anyway. I just wanted to confirm if it should be part (or merged ahead of this series) or can be separate. Thanks!
Quoted 6 lines
So I need to add
kvm_vfio_device_init()and it will callvfio_pci_has_driver(). Then,kvm_vfio_device_init()willTEST_FAILifvfio_pci_has_driver()returns false, correct?Should
kvm_vfio_device_init()live intools/testing/selftests/kvm/lib/kvm_util.c?I was thinking the new test would do:
TEST_REQUIRE(vfio_pci_has_driver_send_msi(device));instead of:
TEST_REQUIRE(device->driver.ops);
vfio_pci_has_driver_send_msi()would callvfio_pci_has_driver()internally and also check thatsend_msiop is non-null. Both functions would also log error messages if driver/send_msi is missing to help the user, as Sean requested.
Quoted 8 lines
Quoted 4 lines
And you'll also need to a patch to add a helper script to print the list of BDFs on the current host that have a VFIO selftests driver. It might make sense to send that as a standalone patch if Sean is ok with that.
That makes sense for
setup_msi(). But in Sean's implementation ofkvm_vfio_device_init()above, it looks like he wants the test to fail if there's no VFIO selftest driver for the BDF passed into the test.
Whoops, never mind. I missed part of the discussion above. We're only requiring a VFIO selftest driver if we're using device MSIs, correct?
Sean Christopherson
reviewerQuoted 67 lines · Josh Hilke
Quoted 66 lines · Josh Hilke
Quoted 65 lines
Quoted 41 lines
Quoted 34 lines
Quoted 34 lines
Quoted 19 lines
Quoted 19 lines
struct
vfio_pci_device*kvm_vfio_device_init(const char *bdf) {
structvfio_pci_device*device;
struct iommu *iommu;iommu = iommu_init(default_iommu_mode);device =
vfio_pci_device_init(bdf, iommu);if (!device) { vfio_pci_device_print_drivers(stderr); TEST_FAIL("No driver found for BDF '%s'", bdf);}
return device;
}Because as is, this requires way too much a priori knowledge and magic for upstream. The user shouldn't have to dig through code just to understand what devices are supported.
Quoted 5 lines
Quoted 3 lines
Ideally, VFIO selftests would also provide a script, utility, and/or helper to identify BDFs for devices it has drivers for. In my experience, binding a device to VFIO is trivial. Finding the device in the first place is more annoying.
Agree this would be nice to have. Is this a blocker for this series?
Yes. As I mentioned to Josh off-list, I'm not willing to accept a "we pinky-swear we'll make this stuff user friendly". It's not that I don't trust you and Josh (and others), it's that for me, this level of user-friendly behavior isn't nice to have, it's mandatory for these tests to be usable upstream.
Internally we can squeak by with barebones, unhelpful tests, because the run commands are often hardcoded somewhere and there is piles of documentation elsewhere. But for upstream, none of that holds true.
Quoted 6 lines
Quoted 2 lines
If it were weeks of effort, I would be more open to treating this as nice to have, but AFAICT writing the code should be less than a days worth of work.
Yeah that's fine, I don't think it adds too much extra work, and it's something I'd like to have anyway. I just wanted to confirm if it should be part (or merged ahead of this series) or can be separate. Thanks!
Quoted 6 lines
So I need to add
kvm_vfio_device_init()and it will callvfio_pci_has_driver(). Then,kvm_vfio_device_init()willTEST_FAILifvfio_pci_has_driver()returns false, correct?Should
kvm_vfio_device_init()live intools/testing/selftests/kvm/lib/kvm_util.c?I was thinking the new test would do:
TEST_REQUIRE(vfio_pci_has_driver_send_msi(device));instead of:
TEST_REQUIRE(device->driver.ops);
vfio_pci_has_driver_send_msi()would callvfio_pci_has_driver()internally and also check thatsend_msiop is non-null. Both functions would also log error messages if driver/send_msi is missing to help the user, as Sean requested.And you'll also need to a patch to add a helper script to print the list of BDFs on the current host that have a VFIO selftests driver. It might make sense to send that as a standalone patch if Sean is ok with that.
Quoted 6 lines
Quoted 3 lines
That makes sense for
setup_msi(). But in Sean's implementation ofkvm_vfio_device_init()above, it looks like he wants the test to fail if there's no VFIO selftest driver for the BDF passed into the test.Whoops, never mind. I missed part of the discussion above. We're only requiring a VFIO selftest driver if we're using device MSIs, correct?
From a user-friendliness perspective, I don't see the point in trying to support devices for which there is no selftest driver. The *best* case scenario is that the user gets lucky and picks a device that has MSI-X capabilities, and thus is able to run the test in a degraded mode.
If the device doesn't have MSI-X capabilities, the test will skip thanks to this check:
TEST_REQUIRE(device->msix_info.count > 0); vfio_pci_msix_enable(device, 0, 1);
Which is "fine", but IMO, the extra complexity, subtlety and potential for confusion doesn't justify the coverage provided, because the coverage provided is basically zero relative to what coverage is provided by dedicated KVM_IRQFD and VFIO IRQ tests. And most importantly requiring VFIO makes it practically impossible to run the test by default.
Doing VFIO_DEVICE_SET_IRQS via vfio_pci_irq_trigger() is just a convoluted, comically slow way of doing eventfd_signal(). The VFIO => eventfd_signal() path can be fully tested by having userspace listen on the eventfd, and the eventfd_signal() => KVM IRQ injection path can be fully tested by having userspace signal the eventfd directly.
I'm a-ok doing a short pass of the test with vfio_pci_irq_trigger() to smoke test things before proceeding to the "real" test, but IMO making the vfio_pci_irq_trigger() path a standalone mode for the test is pointless.
However! We _do_ have a KVM testing gap that can be trivially filled as there is no existing eventfd_signal() => KVM IRQ injection selftest. And this test is pretty much exactly that. I just don't want to deal with jumping through VFIO hoops for effectively no coverage (for KVM).
If we strip out that fluff, then the test is runnable without _any_ VFIO involvement, which means it can run by default, e.g. in n environments without VFIO or VFIO-able devices, e.g. in automated CI testgrids.
Something like so. And then we should also rename the test to drop vfio_pci from the name, as the VFIO aspect becomes purely optional.
--- .../testing/selftests/kvm/vfio_pci_irq_test.c | 105 ++++++++---------- 1 file changed, 48 insertions(+), 57 deletions(-)
tools/testing/selftests/kvm/vfio_pci_irq_test.c +48 -57
@@ -156,42 +156,34 @@ static void kvm_route_msi(struct kvm_vm *vm, u32 gsi, struct kvm_vcpu *vcpu,vm_ioctl (vm,KVM_SET_GSI_ROUTING , routes); } -static int setup_msi (struct vfio_pci_device *device,bool use_device_msi) +static int vfio_setup_msi (struct vfio_pci_device *device) {const int flags =MAP_SHARED |MAP_ANONYMOUS ;const int prot =PROT_READ |PROT_WRITE ;struct dma_region *region; -if (use_device_msi) { -/* A driver is required to generate an MSI. */ -TEST_REQUIRE (device->driver.ops); +/* A driver is required to generate an MSI. */ +TEST_REQUIRE (device->driver.ops); -/* Set up a DMA-able region for the driver to use. */ - region = &device->driver.region; - region->iova =0 ; - region->size =SZ_2M ; - region->vaddr =kvm_mmap (region->size, prot, flags,-1 ); -TEST_ASSERT (region->vaddr !=MAP_FAILED ,"mmap() failed\n" ); -iommu_map (device->iommu, region); +/* Set up a DMA-able region for the driver to use. */ + region = &device->driver.region; + region->iova =0 ; + region->size =SZ_2M ; + region->vaddr =kvm_mmap (region->size, prot, flags,-1 ); +TEST_ASSERT (region->vaddr !=MAP_FAILED ,"mmap() failed\n" ); +iommu_map (device->iommu, region); -vfio_pci_driver_init (device); +vfio_pci_driver_init (device); -return device->driver.msi; - } - -TEST_REQUIRE (device->msix_info.count >0 ); -vfio_pci_msix_enable (device,0 ,1 ); -return 0 ; +return device->driver.msi; } -static void send_msi (struct vfio_pci_device *device,bool use_device_msi,int msi) +static void trigger_interrupt (struct vfio_pci_device *device,int eventfd) { -if (use_device_msi) { -TEST_ASSERT_EQ (msi, device->driver.msi); +if (device)vfio_pci_driver_send_msi (device); - }else { -vfio_pci_irq_trigger (device,VFIO_PCI_MSIX_IRQ_INDEX , msi); - } +else +eventfd_signal (eventfd); }static void help (const char *name)
@@ -236,7 +228,7 @@ int main(int argc, char **argv)u8 vector =32 +rand () % (UINT8_MAX -32 );/* Test configuration (overridable by command line flags). */ -bool use_device_msi = false, irq_affinity = false, pin_vcpus = false; +bool irq_affinity = false, pin_vcpus = false;bool empty = false, nmi = false;int nr_irqs =1000 ;int nr_vcpus =1 ;
@@ -244,19 +236,18 @@ int main(int argc, char **argv)struct kvm_vcpu *vcpus[KVM_MAX_VCPUS ];pthread_t vcpu_threads[KVM_MAX_VCPUS ];u64 irq_count, pin_count, piw_count; +const char *device_bdf =NULL ;struct vfio_pci_device *device;struct iommu *iommu;cpu_set_t available_cpus; -const char *device_bdf;FILE *irq_affinity_fp;int i, j, c, msi, irq;struct kvm_vm *vm;int irq_cpu;int ret; - device_bdf =vfio_selftests_get_bdf (&argc, argv); -while ((c =getopt (argc, argv,"abdehi:npv:xt:" )) !=-1 ) { +while ((c =getopt (argc, argv,"abd:ehi:npv:xt:" )) !=-1 ) {switch (c) {case 'a' : irq_affinity = true;
@@ -265,7 +256,7 @@ int main(int argc, char **argv) block = true;break ;case 'd' : - use_device_msi = true; + device_bdf = optarg;break ;case 'e' : empty = true;
@@ -301,20 +292,23 @@ int main(int argc, char **argv)if (!x2apic)virt_pg_map (vm,APIC_DEFAULT_GPA ,APIC_DEFAULT_GPA ); - iommu =iommu_init (default_iommu_mode); - device =vfio_pci_device_init (device_bdf, iommu); - msi =setup_msi (device, use_device_msi); - irq =get_irq_number (device_bdf, msi); +if (device_bdf) { + iommu =iommu_init (default_iommu_mode); + device =vfio_pci_device_init (device_bdf, iommu); + msi =vfio_setup_msi (device); + irq =get_irq_number (device_bdf, msi); + irq_count =get_irq_count (irq); - irq_count =get_irq_count (irq); - pin_count =get_irq_count_by_name ("PIN:" ); - piw_count =get_irq_count_by_name ("PIW:" ); + eventfd = device->msi_eventfds[msi]; -printf ("%s %s MSI-X[%d] (IRQ-%d) %d times\n" , - use_device_msi ?"Triggering" :"Notifying the eventfd for" , - device_bdf, msi, irq, nr_irqs); +printf ("Using device %s MSI-X[%d] (IRQ-%d)\n" , + device_bdf, msi, irq); + }else { + device =NULL ; + eventfd =kvm_new_eventfd (); + } -kvm_assign_irqfd (vm, gsi, device->msi_eventfds[msi]); +kvm_assign_irqfd (vm, gsi, eventfd);sync_global_to_guest (vm, x2apic);sync_global_to_guest (vm, block);
@@ -341,6 +335,8 @@ int main(int argc, char **argv)if (irq_affinity) {char path[PATH_MAX ]; +TEST_ASSERT (device,"Manipulating IRQ affinity requires a device" ); +snprintf (path,sizeof (path),"/proc/irq/%d/smp_affinity_list" , irq); irq_affinity_fp =fopen (path,"w" );TEST_ASSERT (irq_affinity_fp,"fopen(%s) failed" , path);
@@ -371,17 +367,14 @@ int main(int argc, char **argv)pin_vcpu_threads (nr_vcpus,rand () %get_nprocs (), &available_cpus);for (j =0 ; j < nr_vcpus; j++) { -TEST_ASSERT ( - !READ_FROM_GUEST (vm, guest_received_irq[vcpu->id]), -"IRQ flag for vCPU %d not clear prior to test" , - vcpu->id); -TEST_ASSERT ( - !READ_FROM_GUEST (vm, guest_received_nmi[vcpu->id]), -"NMI flag for vCPU %d not clear prior to test" , +TEST_ASSERT (!READ_FROM_GUEST (vm, guest_received_irq[vcpu->id]), +"IRQ flag for vCPU %d not clear prior to test" , vcpu->id); +TEST_ASSERT (!READ_FROM_GUEST (vm, guest_received_nmi[vcpu->id]), +"NMI flag for vCPU %d not clear prior to test" , vcpu->id); } -send_msi (device, use_device_msi, msi); +trigger_interrupt (device, eventfd);clock_gettime (CLOCK_MONOTONIC , &start);for (;;) {
@@ -416,7 +409,7 @@ int main(int argc, char **argv)for (i =0 ; i < nr_vcpus; i++) {if (block) {kvm_route_msi (vm, gsi, vcpus[i], vector, false); -send_msi (device, false, msi); +trigger_interrupt (device, eventfd); }pthread_join (vcpu_threads[i],NULL );
@@ -425,15 +418,13 @@ int main(int argc, char **argv)if (irq_affinity)fclose (irq_affinity_fp); -printf ("Host interrupts handled:\n" ); -printf (" IRQ-%d: %lu\n" , irq,get_irq_count (irq) - irq_count); -printf (" Posted-interrupt notification events: %lu\n" , -get_irq_count_by_name ("PIN:" ) - pin_count); -printf (" Posted-interrupt wakeup events: %lu\n" , -get_irq_count_by_name ("PIW:" ) - piw_count); +if (device) { +printf ("Host interrupts handled:\n" ); +printf (" IRQ-%d: %lu\n" , irq,get_irq_count (irq) - irq_count); -vfio_pci_device_cleanup (device); -iommu_cleanup (iommu); +vfio_pci_device_cleanup (device); +iommu_cleanup (iommu); + }return 0 ; }
base-commit: 97efbd75a6c9ef0946234df78d3b656e28035883 --
Quoted 11 lines · Sean Christopherson
However! We _do_ have a KVM testing gap that can be trivially filled as there is no existing
eventfd_signal()=> KVM IRQ injection selftest. And this test is pretty much exactly that. I just don't want to deal with jumping through VFIO hoops for effectively no coverage (for KVM).If we strip out that fluff, then the test is runnable without _any_ VFIO involvement, which means it can run by default, e.g. in n environments without VFIO or VFIO-able devices, e.g. in automated CI testgrids.
Something like so. And then we should also rename the test to drop
vfio_pcifrom the name, as the VFIO aspect becomes purely optional.
That plan sounds good to me!
Quoted 8 lines · David Matlack
Quoted 5 lines · Sean Christopherson
However! We _do_ have a KVM testing gap that can be trivially filled as there is no existing
eventfd_signal()=> KVM IRQ injection selftest. And this test is pretty much exactly that. I just don't want to deal with jumping through VFIO hoops for effectively no coverage (for KVM).
Quoted 8 lines
Quoted 6 lines
If we strip out that fluff, then the test is runnable without _any_ VFIO involvement, which means it can run by default, e.g. in n environments without VFIO or VFIO-able devices, e.g. in automated CI testgrids.
Something like so. And then we should also rename the test to drop
vfio_pcifrom the name, as the VFIO aspect becomes purely optional.That plan sounds good to me!
Awesome, thanks for the thorough example, Sean!
I'll go with this approach in v3 :)