KVM: selftests: Link with VFIO selftests lib and test device interrupts
cover KVM: selftests: Link with VFIO selftests lib and test device interrupts
This series can be found on GitHub:
https://github.com/dmatlack/linux/tree/kvm/selftests/vfio_pci_irq_test/v1
I'm sending this series out a little early. VFIO selftests has landed in vfio/next[1] and is slated for 6.18, but has not yet been merged by Linus. But I wanted to start the discussion of linking the VFIO selftests library into KVM selftests so we can test VFIO-KVM interactions.
To demostrate testing VFIO+KVM interactions, patch 2 contains a test to exercise delivering VFIO device interrupts to vCPUs. This test is heavily based on Sean's vfio_irq_test.c [2].
Running selftests with VFIO devices is slightly different than typical KVM selftests since it requires a PCI device bound to vfio-pci. The VFIO selftests have a helper script for setting up a device:
$ ./run.sh -s -d 0000:6a:01.0 + echo "vfio-pci" > /sys/bus/pci/devices/0000:6a:01.0/driver_override + echo "0000:6a:01.0" > /sys/bus/pci/drivers/vfio-pci/bind
Dropping into /bin/bash with VFIO_SELFTESTS_BDF=0000:6a:01.0
The test can then be run and it will detect the $VFIO_SELFTESTS_BDF environment variable.
$ tools/testing/selftests/kvm/vfio_pci_irq_test $ tools/testing/selftests/kvm/vfio_pci_irq_test -v64 -d -x
You can also pass the BDF directly to the test on the command-line (i.e. using run.sh and the environment variable is not a required).
$ tools/testing/selftests/kvm/vfio_pci_irq_test 0000:6a:01.0 $ tools/testing/selftests/kvm/vfio_pci_irq_test -v64 -d -x 0000:6a:01.0
In order to test with real MSIs generated by the device (-d option), the device must also have a supported driver in tools/testing/selftests/vfio/lib/drivers/. At the moment we only have drivers for Intel CBDMA and Intel DSA devices, so I haven't been able to test with -d on AMD yet.
Currently this test only supports x86_64, but it should be portable to at least ARM in the future, so I optimistically placed it in the top-level KVM selftests directory.
[1] https://github.com/awilliam/linux-vfio/tree/next
[2] https://lore.kernel.org/kvm/[email protected]/
David Matlack (2):
KVM: selftests: Build and link sefltests/vfio/lib into KVM selftests
KVM: selftests: Add a test for vfio-pci device IRQ delivery to vCPUs
tools/testing/selftests/kvm/Makefile.kvm | 6 +- .../testing/selftests/kvm/vfio_pci_irq_test.c | 507 ++++++++++++++++++ 2 files changed, 512 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/kvm/vfio_pci_irq_test.c
base-commit: 093458c58f830d0a713fab0de037df5f0ce24fef
prerequisite-patch-id: 72dce9cd586ac36ea378354735d9fabe2f3c445e
prerequisite-patch-id: a8c7ccfd91ce3208f328e8af7b25c83bff8d464d
Signature · 1 line
2.51.0.384.g4c02a37b29-goog
PATCH 1/2 KVM: selftests: Build and link sefltests/vfio/lib into KVM selftests
Include libvfio.mk into the KVM selftests Makefile and link it into all KVM selftests by adding it to LIBKVM_OBJS.
Note that KVM selftests build their own copy of sefltests/vfio/lib and the resulting object files are placed in $(OUTPUT)/lib. This allows the KVM and VFIO selftests to apply different CFLAGS when building without conflicting with each other.
--- tools/testing/selftests/kvm/Makefile.kvm | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
tools/testing/selftests/kvm/Makefile.kvm +4 -1
@@ -228,6 +228,7 @@ OVERRIDE_TARGETS = 1 # which causes the environment variable to override the makefile). include ../lib.mk include ../cgroup/lib/libcgroup.mk +include ../vfio/lib/libvfio.mk INSTALL_HDR_PATH = $(top_srcdir)/usr LINUX_HDR_PATH = $(INSTALL_HDR_PATH)/include/
@@ -281,7 +282,9 @@ LIBKVM_S := $(filter %.S,$(LIBKVM)) LIBKVM_C_OBJ := $(patsubst %.c, $(OUTPUT)/%.o, $(LIBKVM_C)) LIBKVM_S_OBJ := $(patsubst %.S, $(OUTPUT)/%.o, $(LIBKVM_S)) LIBKVM_STRING_OBJ := $(patsubst %.c, $(OUTPUT)/%.o, $(LIBKVM_STRING)) -LIBKVM_OBJS = $(LIBKVM_C_OBJ) $(LIBKVM_S_OBJ) $(LIBKVM_STRING_OBJ) $(LIBCGROUP_O) +LIBKVM_OBJS = $(LIBKVM_C_OBJ) $(LIBKVM_S_OBJ) $(LIBKVM_STRING_OBJ) +LIBKVM_OBJS += $(LIBCGROUP_O) +LIBKVM_OBJS += $(LIBVFIO_O) SPLIT_TEST_GEN_PROGS := $(patsubst %, $(OUTPUT)/%, $(SPLIT_TESTS)) SPLIT_TEST_GEN_OBJ := $(patsubst %, $(OUTPUT)/$(SRCARCH)/%.o, $(SPLIT_TESTS))
Signature · 1 line
2.51.0.384.g4c02a37b29-goog
PATCH 2/2 KVM: selftests: Add a test for vfio-pci device IRQ delivery to vCPUs
In reply to David MatlackAdd a new selftest called vfio_pci_irq_test that routes and delivers an MSI from a vfio-pci device into a guest. This test builds on top of the VFIO selftests library, which provides helpers for interacting with VFIO devices and drivers for generating interrupts with specific devices.
This test sets up a configurable number of vCPUs in separate threads that all spin in guest-mode or halt. Then the test round robin routes the device's interrupt to different CPUs, triggers it, and then verifies the guest received it. The test supports several options to enable affinitizing the host IRQ handler to different CPUs, pinning vCPU threads to different CPUs, and more.
This test also measure and reports the number of times the device IRQ was handled by the host. This can be used to confirm whether device-posted interrupts are working as expected.
Running this test requires a PCI device bound to the vfio-pci driver, and then passing the BDF of the device to the test, e.g.:
$ ./vfio_pci_irq_test 0000:6a:01.0
To run the test with real device-sent MSIs (-d option), the PCI device must also have a supported driver in tools/testing/selftests/vfio/lib/drivers/.
This test only supports x86_64 for now, but can be ported to other architectures in the future.
--- tools/testing/selftests/kvm/Makefile.kvm | 1 + .../testing/selftests/kvm/vfio_pci_irq_test.c | 507 ++++++++++++++++++ 2 files changed, 508 insertions(+) create mode 100644 tools/testing/selftests/kvm/vfio_pci_irq_test.c
tools/testing/selftests/kvm/Makefile.kvm +1 -0
@@ -148,6 +148,7 @@ TEST_GEN_PROGS_x86 += rseq_test TEST_GEN_PROGS_x86 += steal_time TEST_GEN_PROGS_x86 += system_counter_offset_test TEST_GEN_PROGS_x86 += pre_fault_memory_test +TEST_GEN_PROGS_x86 += vfio_pci_irq_test # Compiled outputs used by test targets TEST_GEN_PROGS_EXTENDED_x86 += x86/nx_huge_pages_test
tools/testing/selftests/kvm/vfio_pci_irq_test.c new file +507 -0
507 lines not on this page · raw message
Signature · 1 line
2.51.0.384.g4c02a37b29-goog
Sean Christopherson
reviewerRe: [PATCH 2/2] KVM: selftests: Add a test for vfio-pci device IRQ delivery to vCPUs
Quoted 19 lines · David Matlack
Add a new selftest called
vfio_pci_irq_testthat routes and delivers an MSI from a vfio-pci device into a guest. This test builds on top of the VFIO selftests library, which provides helpers for interacting with VFIO devices and drivers for generating interrupts with specific devices.This test sets up a configurable number of vCPUs in separate threads that all spin in guest-mode or halt. Then the test round robin routes the device's interrupt to different CPUs, triggers it, and then verifies the guest received it. The test supports several options to enable affinitizing the host IRQ handler to different CPUs, pinning vCPU threads to different CPUs, and more.
This test also measure and reports the number of times the device IRQ was handled by the host. This can be used to confirm whether device-posted interrupts are working as expected.
Running this test requires a PCI device bound to the vfio-pci driver, and then passing the BDF of the device to the test, e.g.:
Quoted 8 lines
$ ./vfio_pci_irq_test 0000:6a:01.0To run the test with real device-sent MSIs (-d option), the PCI device must also have a supported driver in
tools/testing/selftests/vfio/lib/drivers/.This test only supports
x86_64for now, but can be ported to other architectures in the future.
Can it though? There are bits and pieces that can be reused, but this test is x86 through and through.
Quoted 6 lines
Suggested-by Sean Christopherson <[email protected]>Signed-off-by David Matlack <[email protected]>--- tools/testing/selftests/kvm/Makefile.kvm | 1 + .../testing/selftests/kvm/vfio_pci_irq_test.c | 507 ++++++++++++++++++
Please break this into multiple patches, e.g. a "basic" patch and and then roughly one per "advanced" command line option.
There is a _lot_ going on here, with very little documentation, e.g. the changelog just says:
The test supports several options to enable affinitizing the host IRQ handler to different CPUs, pinning vCPU threads to different CPUs, and more.
which is rather useless for readers that aren't already familiar with much of what is being tested, and why.
Quoted 35 lines
2 files changed, 508 insertions(+) create mode 100644 tools/testing/selftests/kvm/vfio_pci_irq_test.ctools/testing/selftests/kvm/Makefile.kvm +1
-0@@ -148,6 +148,7 @@ TEST_GEN_PROGS_x86 += rseq_test TEST_GEN_PROGS_x86 += steal_time TEST_GEN_PROGS_x86 += system_counter_offset_test TEST_GEN_PROGS_x86 += pre_fault_memory_test +TEST_GEN_PROGS_x86 += vfio_pci_irq_test # Compiled outputs used by test targets TEST_GEN_PROGS_EXTENDED_x86 += x86/nx_huge_pages_testtools/testing/selftests/kvm/vfio_pci_irq_test.c new file +14
-0@@ -0,0 +1,507 @@ +// SPDX-License-Identifier: GPL-2.0 +#include "kvm_util.h" +#include "test_util.h" +#include "apic.h" +#include "processor.h" + +#include <pthread.h> +#include <ctype.h> +#include <time.h> +#include <linux/vfio.h> +#include <linux/sizes.h> +#include <sys/sysinfo.h> + +#include <vfio_util.h>
tools/testing/selftests/kvm/vfio_pci_irq_test.c line 24
Quoted 10 lines
tools/testing/selftests/kvm/vfio_pci_irq_test.c new file +10
-0+ +static bool x2apic = true; +static bool done; +static bool block; + +static bool guest_ready_for_irqs[KVM_MAX_VCPUS ]; +static bool guest_received_irq[KVM_MAX_VCPUS ]; +static bool guest_received_nmi[KVM_MAX_VCPUS ]; + +static pid_t vcpu_tids[KVM_MAX_VCPUS ];
s/tids/pids?
tools/testing/selftests/kvm/vfio_pci_irq_test.c line 25
Quoted 1 line
tools/testing/selftests/kvm/vfio_pci_irq_test.c new file +1
-0+#define TIMEOUT_NS (2ULL * 1000 * 1000 * 1000)
The timeout should be configurable via command line.
tools/testing/selftests/kvm/vfio_pci_irq_test.c line 34
Quoted 9 lines
tools/testing/selftests/kvm/vfio_pci_irq_test.c new file +9
-0+#define READ_FROM_GUEST (_vm, _variable) ({ \ + sync_global_from_guest(_vm, _variable); \ + READ_ONCE(_variable); \ +}) + +#define WRITE_TO_GUEST (_vm, _variable, _value) do { \ + WRITE_ONCE(_variable, _value); \ + sync_global_to_guest(_vm, _variable); \ +} while (0)
These belong in a separate patch, and in tools/testing/selftests/kvm/include/kvm_util.h.
tools/testing/selftests/kvm/vfio_pci_irq_test.c line 44
Quoted 10 lines
tools/testing/selftests/kvm/vfio_pci_irq_test.c new file +10
-0+ +static u32 guest_get_vcpu_id (void ) +{ +if (x2apic) +return x2apic_read_reg (APIC_ID ); +else +return xapic_read_reg (APIC_ID ) >>24 ; +} + +static void guest_enable_interrupts (void )
This is a misleading name, e.g. I would expect it to _just_ be sti_nop(). If the intent is to provide a hook for a potential non-x86 implementation, then it should probably be split into guest_arch_test_setup() and guest_arch_irq_enable() or so (to align with the kernel's local_irq_{dis,en}able()).
As is, I would just omit the helper.
tools/testing/selftests/kvm/vfio_pci_irq_test.c line 56
Quoted 12 lines
tools/testing/selftests/kvm/vfio_pci_irq_test.c new file +12
-0+{ +if (x2apic) +x2apic_enable (); +else +xapic_enable (); + +sti_nop (); +} + +static void guest_irq_handler (struct ex_regs *regs) +{ +WRITE_ONCE (guest_received_irq[guest_get_vcpu_id ()], true);
Hmm, using APID ID works, but I don't like the hidden dependency on the library using ascending IDs starting from '0'. This would also be a good opportunity to improve the core infrastructure.
Rather than use APIC IDs, vm_arch_vcpu_add() could set MSR_TSC_AUX if RDTSCP or RDPID is supported, and then make guest_get_vcpu_id() a common API that uses RDPID or RDTSCP (prefer RDPID when possible).
Then tests that want to use guest_get_vcpu_id() can do a TEST_REQUIRE(). RDTSCP exists at least as far back as Haswell, so I don't think it's unreasonable to start depending on MSR_TSC_AUX.
Hmm, or we could steal an idea from the kernel and use the LDT to stash info in a segment limit as a fallback.
Quoted 10 lines
tools/testing/selftests/kvm/vfio_pci_irq_test.c new file +10
-0+ +if (x2apic) +x2apic_write_reg (APIC_EOI ,0 ); +else +xapic_write_reg (APIC_EOI ,0 ); +} + +static void guest_nmi_handler (struct ex_regs *regs) +{ +WRITE_ONCE (guest_received_nmi[guest_get_vcpu_id ()], true);
tools/testing/selftests/kvm/vfio_pci_irq_test.c line 76
Quoted 10 lines
tools/testing/selftests/kvm/vfio_pci_irq_test.c new file +10
-0+} + +static void guest_code (void ) +{ +guest_enable_interrupts (); +WRITE_ONCE (guest_ready_for_irqs[guest_get_vcpu_id ()], true); + +while (!READ_ONCE (done)) { +if (block) +hlt ();
else cpu_relax();
tools/testing/selftests/kvm/vfio_pci_irq_test.c line 87
Quoted 11 lines
tools/testing/selftests/kvm/vfio_pci_irq_test.c new file +11
-0+ } + +GUEST_DONE (); +} + +static void *vcpu_thread_main (void *arg) +{ +struct kvm_vcpu *vcpu = arg; +struct ucall uc; + +WRITE_ONCE (vcpu_tids[vcpu->id],syscall (__NR_gettid));
Please add wrapper in tools/testing/selftests/kvm/include/kvm_syscalls.h.
Quoted 30 lines
tools/testing/selftests/kvm/vfio_pci_irq_test.c new file +30
-0+ +vcpu_run (vcpu); +TEST_ASSERT_EQ (UCALL_DONE ,get_ucall (vcpu, &uc)); + +return NULL ; +} + +static int get_cpu (struct kvm_vcpu *vcpu) +{ +pid_t tid = vcpu_tids[vcpu->id]; +cpu_set_t cpus; +int cpu =-1 ; +int i, ret; + + ret =sched_getaffinity (tid,sizeof (cpus), &cpus); +TEST_ASSERT (ret ==0 ,"sched_getaffinity() failed" ); + +for (i =0 ; i <get_nprocs (); i++) { +if (!CPU_ISSET (i, &cpus)) +continue ; + +if (cpu !=-1 ) { + cpu = i; + }else { +/* vCPU is pinned to multiple CPUs */ +return -1 ; + } + } + +return cpu;
tools/testing/selftests/kvm/vfio_pci_irq_test.c line 127
Quoted 10 lines
tools/testing/selftests/kvm/vfio_pci_irq_test.c new file +10
-0+} + +static void pin_vcpu_threads (int nr_vcpus,int start_cpu,cpu_set_t *available_cpus) +{ +const size_t size =sizeof (cpu_set_t); +int nr_cpus, cpu, vcpu_index =0 ; +cpu_set_t target_cpu; +int r; + + nr_cpus =get_nprocs ();
Generally speaking, KVM selftests try to avoid affining tasks to CPUs that are outside of the original affinity list. See various usage of sched_getaffinity().
Quoted 16 lines
tools/testing/selftests/kvm/vfio_pci_irq_test.c new file +16
-0+CPU_ZERO (&target_cpu); + +for (cpu = start_cpu;; cpu = (cpu +1 ) % nr_cpus) { +if (vcpu_index == nr_vcpus) +break ; + +if (!CPU_ISSET (cpu, available_cpus)) +continue ; + +CPU_SET (cpu, &target_cpu); + + r =sched_setaffinity (vcpu_tids[vcpu_index], size, &target_cpu); +TEST_ASSERT (r ==0 ,"sched_setaffinity() failed (cpu: %d)" , cpu); + +CPU_CLR (cpu, &target_cpu); +
tools/testing/selftests/kvm/vfio_pci_irq_test.c line 153
Quoted 10 lines
tools/testing/selftests/kvm/vfio_pci_irq_test.c new file +10
-0+ vcpu_index++; + } +} + +static FILE *open_proc_interrupts (void ) +{ +FILE *fp; + + fp =fopen ("/proc/interrupts" ,"r" ); +TEST_ASSERT (fp,"fopen(/proc/interrupts) failed" );
open_path_or_exit()?
tools/testing/selftests/kvm/vfio_pci_irq_test.c line 156
Quoted 3 lines
tools/testing/selftests/kvm/vfio_pci_irq_test.c new file +3
-0+ +return fp; +}
And all of these /proc helpers belong in library code.
Quoted 91 lines
tools/testing/selftests/kvm/vfio_pci_irq_test.c new file +91
-0+static int get_irq_number (const char *device_bdf,int msi) +{ +char search_string[64 ]; +char line[4096 ]; +int irq =-1 ; +FILE *fp; + + fp =open_proc_interrupts (); + +snprintf (search_string,sizeof (search_string),"vfio-msix[%d]" , msi); + +while (fgets (line,sizeof (line), fp)) { +if (strstr (line, device_bdf) &&strstr (line, search_string)) { +TEST_ASSERT_EQ (1 ,sscanf (line,"%d:" , &irq)); +break ; + } + } + +fclose (fp); + +TEST_ASSERT (irq !=-1 ,"Failed to locate IRQ for %s %s" , device_bdf, search_string); +return irq; +} + +static int parse_interrupt_count (char *token) +{ +char *c; + +for (c = token; *c; c++) { +if (!isdigit (*c)) +return 0 ; + } + +return atoi_non_negative ("interrupt count" , token); +} + +static u64 __get_irq_count (const char *search_string) +{ +u64 total_count =0 ; +char line[4096 ]; +FILE *fp; + + fp =open_proc_interrupts (); + +while (fgets (line,sizeof (line), fp)) { +char *token =strtok (line," " ); + +if (strcmp (token, search_string)) +continue ; + +while ((token =strtok (NULL ," " ))) + total_count +=parse_interrupt_count (token); + +break ; + } + +fclose (fp); +return total_count; +} + +static u64 get_irq_count (int irq) +{ +char search_string[32 ]; + +snprintf (search_string,sizeof (search_string),"%d:" , irq); +return __get_irq_count (search_string); +} + +static void kvm_clear_gsi_routes (struct kvm_vm *vm) +{ +struct kvm_irq_routing routes = {}; + +vm_ioctl (vm,KVM_SET_GSI_ROUTING , &routes); +} + +static void kvm_route_msi (struct kvm_vm *vm,u32 gsi,struct kvm_vcpu *vcpu, +u8 vector,bool do_nmi) +{ +u8 buf[sizeof (struct kvm_irq_routing ) +sizeof (struct kvm_irq_routing_entry )] = {}; +struct kvm_irq_routing *routes = (void *)&buf; + + routes->nr =1 ; + routes->entries[0 ].gsi = gsi; + routes->entries[0 ].type =KVM_IRQ_ROUTING_MSI ; + routes->entries[0 ].u.msi.address_lo =0xFEE00000 | (vcpu->id <<12 ); + routes->entries[0 ].u.msi.data = do_nmi ?NMI_VECTOR | (4 <<8 ) : vector; + +vm_ioctl (vm,KVM_SET_GSI_ROUTING , routes); +} + +static int setup_msi (struct vfio_pci_device *device,bool use_device_msi)
tools/testing/selftests/kvm/vfio_pci_irq_test.c line 257
Quoted 10 lines
tools/testing/selftests/kvm/vfio_pci_irq_test.c new file +10
-0+{ +const int flags =MAP_SHARED |MAP_ANONYMOUS ; +const int prot =PROT_READ |PROT_WRITE ; +struct vfio_dma_region *region; + +if (use_device_msi) { +/* A driver is required to generate an MSI. */ +TEST_REQUIRE (device->driver.ops); + +/* Set up a DMA-able region for the driver to use. */
Why?
tools/testing/selftests/kvm/vfio_pci_irq_test.c line 261
Quoted 4 lines
tools/testing/selftests/kvm/vfio_pci_irq_test.c new file +4
-0+ region = &device->driver.region; + region->iova =0 ; + region->size =SZ_2M ; + region->vaddr =mmap (NULL , region->size, prot, flags,-1 ,0 );
kvm_mmap()
Quoted 4 lines
tools/testing/selftests/kvm/vfio_pci_irq_test.c new file +4
-0+TEST_ASSERT (region->vaddr !=MAP_FAILED ,"mmap() failed\n" ); +vfio_pci_dma_map (device, region); + +vfio_pci_driver_init (device);
tools/testing/selftests/kvm/vfio_pci_irq_test.c line 275
Quoted 10 lines
tools/testing/selftests/kvm/vfio_pci_irq_test.c new file +10
-0+ +return device->driver.msi; + } + +TEST_REQUIRE (device->msix_info.count >0 ); +vfio_pci_msix_enable (device,0 ,1 ); +return 0 ; +} + +static void send_msi (struct vfio_pci_device *device,bool use_device_msi,int msi)
IMO, this helper does more harm than good. There is only one real user, the second call unconditionally passes %false for @use_device_msi.
If you drop the helper, than there should be no need to assert that the MSI is the device MSI on *every* send via device.
Quoted 26 lines
tools/testing/selftests/kvm/vfio_pci_irq_test.c new file +26
-0+{ +if (use_device_msi) { +TEST_ASSERT_EQ (msi, device->driver.msi); +vfio_pci_driver_send_msi (device); + }else { +vfio_pci_irq_trigger (device,VFIO_PCI_MSIX_IRQ_INDEX , msi); + } +} + +static void help (const char *name) +{ +printf ("Usage: %s [-a] [-b] [-d] [-e] [-h] [-i nr_irqs] [-n] [-p] [-v nr_vcpus] [-x] segment:bus:device.function\n" , + name); +printf ("\n" ); +printf (" -a: Randomly affinitize the device IRQ to different CPUs\n" +" throughout the test.\n" ); +printf (" -b: Block vCPUs (e.g. HLT) instead of spinning in guest-mode\n" ); +printf (" -d: Use the device to trigger the IRQ instead of emulating\n" +" it with an eventfd write.\n" ); +printf (" -e: Destroy and recreate KVM's GSI routing table in between\n" +" some interrupts.\n" ); +printf (" -i: The number of IRQs to generate during the test.\n" ); +printf (" -n: Route some of the device interrupts to be delivered as\n" +" an NMI into the guest.\n" ); +printf (" -p: Pin vCPU threads to random pCPUs throughout the test.\n" ); + printf(" -v: Set the number of vCPUs that the test should create.\n"
tools/testing/selftests/kvm/vfio_pci_irq_test.c line 311
Quoted 10 lines
tools/testing/selftests/kvm/vfio_pci_irq_test.c new file +10
-0+ " Interrupts will be round-robined among vCPUs.\n"); +printf (" -x: Use xAPIC mode instead of x2APIC mode in the guest.\n" ); +printf ("\n" ); +exit (KSFT_FAIL ); +} + +int main (int argc,char **argv) +{ +/* Random non-reserved vector and GSI to use for the device IRQ */ +const u8 vector =0xe0 ;
s/random/arbitrary
Why not make it truly random?
tools/testing/selftests/kvm/vfio_pci_irq_test.c line 312
Quoted 1 line
tools/testing/selftests/kvm/vfio_pci_irq_test.c new file +1
-0+const u32 gsi =32 ;
Ditto here.
Quoted 59 lines
tools/testing/selftests/kvm/vfio_pci_irq_test.c new file +59
-0+/* Test configuration (overridable by command line flags). */ +bool use_device_msi = false, irq_affinity = false, pin_vcpus = false; +bool empty = false, nmi = false; +int nr_irqs =1000 ; +int nr_vcpus =1 ; + +struct kvm_vcpu *vcpus[KVM_MAX_VCPUS ]; +pthread_t vcpu_threads[KVM_MAX_VCPUS ]; +u64 irq_count, pin_count, piw_count; +struct vfio_pci_device *device; +cpu_set_t available_cpus; +const char *device_bdf; +FILE *irq_affinity_fp; +int i, j, c, msi, irq; +struct kvm_vm *vm; +int irq_cpu; +int ret; + + device_bdf =vfio_selftests_get_bdf (&argc, argv); + +while ((c =getopt (argc, argv,"abdehi:npv:x" )) !=-1 ) { +switch (c) { +case 'a' : + irq_affinity = true; +break ; +case 'b' : + block = true; +break ; +case 'd' : + use_device_msi = true; +break ; +case 'e' : + empty = true; +break ; +case 'i' : + nr_irqs =atoi_positive ("Number of IRQs" , optarg); +break ; +case 'n' : + nmi = true; +break ; +case 'p' : + pin_vcpus = true; +break ; +case 'v' : + nr_vcpus =atoi_positive ("nr_vcpus" , optarg); +break ; +case 'x' : + x2apic = false; +break ; +case 'h' : +default : +help (argv[0 ]); + } + } + + vm =vm_create_with_vcpus (nr_vcpus, guest_code, vcpus); +vm_install_exception_handler (vm, vector, guest_irq_handler); +vm_install_exception_handler (vm,NMI_VECTOR , guest_nmi_handler); +
tools/testing/selftests/kvm/vfio_pci_irq_test.c line 381
Quoted 10 lines
tools/testing/selftests/kvm/vfio_pci_irq_test.c new file +10
-0+if (!x2apic) +virt_pg_map (vm,APIC_DEFAULT_GPA ,APIC_DEFAULT_GPA ); + + device =vfio_pci_device_init (device_bdf, default_iommu_mode); + msi =setup_msi (device, use_device_msi); + irq =get_irq_number (device_bdf, msi); + + irq_count =get_irq_count (irq); + pin_count =__get_irq_count ("PIN:" ); + piw_count =__get_irq_count ("PIW:" );
This is obviously very Intel specific information. If you're going to print the posted IRQ info, then the test should also print e.g. AMD GALogIntr events.
Quoted 12 lines
tools/testing/selftests/kvm/vfio_pci_irq_test.c new file +12
-0+printf ("%s %s MSI-X[%d] (IRQ-%d) %d times\n" , + use_device_msi ?"Triggering" :"Notifying the eventfd for" , + device_bdf, msi, irq, nr_irqs); + +kvm_assign_irqfd (vm, gsi, device->msi_eventfds[msi]); + +sync_global_to_guest (vm, x2apic); +sync_global_to_guest (vm, block); + +for (i =0 ; i < nr_vcpus; i++) +pthread_create (&vcpu_threads[i],NULL , vcpu_thread_main, vcpus[i]); +
tools/testing/selftests/kvm/vfio_pci_irq_test.c line 403
Quoted 10 lines
tools/testing/selftests/kvm/vfio_pci_irq_test.c new file +10
-0+for (i =0 ; i < nr_vcpus; i++) { +struct kvm_vcpu *vcpu = vcpus[i]; + +while (!READ_FROM_GUEST (vm, guest_ready_for_irqs[vcpu->id])) +continue ; + } + +if (pin_vcpus) { + ret =sched_getaffinity (vcpu_tids[0 ],sizeof (available_cpus), &available_cpus); +TEST_ASSERT (ret ==0 ,"sched_getaffinity() failed" );
!ret
Though this is another syscall that deserves a wrapper in kvm_syscalls.h.
tools/testing/selftests/kvm/vfio_pci_irq_test.c line 407
Quoted 4 lines
tools/testing/selftests/kvm/vfio_pci_irq_test.c new file +4
-0+ +if (nr_vcpus >CPU_COUNT (&available_cpus)) { +printf ("There are more vCPUs than pCPUs; refusing to pin.\n" ); + pin_vcpus = false;
Why is this not an assertion? Alternatively, why not double/triple/quadruple up as needed?
tools/testing/selftests/kvm/vfio_pci_irq_test.c line 416
Quoted 9 lines
tools/testing/selftests/kvm/vfio_pci_irq_test.c new file +9
-0+ } + } + +if (irq_affinity) { +char path[PATH_MAX ]; + +snprintf (path,sizeof (path),"/proc/irq/%d/smp_affinity_list" , irq); + irq_affinity_fp =fopen (path,"w" ); +TEST_ASSERT (irq_affinity_fp,"fopen(%s) failed" , path);
More code that belongs in the library.
tools/testing/selftests/kvm/vfio_pci_irq_test.c line 420
Quoted 4 lines
tools/testing/selftests/kvm/vfio_pci_irq_test.c new file +4
-0+ } + +/* Set a consistent seed so that test are repeatable. */ +srand (0 );
We should really figure out a solution for reproducible random numbers in the host. Ah, and kvm_selftest_init()'s handling of guest random seeds is flawed, because it does random() without srand() and so AFAICT, gets the same seed every time. E.g. seems like we want something like this, but with a way to override "random_seed" from a test.
tools/testing/selftests/kvm/lib/kvm_util.c +7 -1
@@ -2310,6 +2310,7 @@ void __attribute((constructor)) kvm_selftest_init(void)struct sigaction sig_sa = { .sa_handler = report_unexpected_signal, }; +int random_seed;/* Tell stdout not to buffer its content. */ setbuf (stdout,NULL );
@@ -2319,8 +2320,13 @@ void __attribute((constructor)) kvm_selftest_init(void)sigaction (SIGILL , &sig_sa,NULL );sigaction (SIGFPE , &sig_sa,NULL ); + random_seed =time (NULL ); +srand (random_seed); + guest_random_seed = last_guest_seed =random (); -pr_info ("Random seed: 0x%x\n" , guest_random_seed); + +pr_info ("Guest random seed: 0x%x (srand: 0x%x)\n" , + guest_random_seed, random_seed);kvm_selftest_arch_init (); }
tools/testing/selftests/kvm/vfio_pci_irq_test.c line 432
Quoted 12 lines
tools/testing/selftests/kvm/vfio_pci_irq_test.c new file +12
-0+for (i =0 ; i < nr_irqs; i++) { +struct kvm_vcpu *vcpu = vcpus[i % nr_vcpus]; +const bool do_nmi = nmi && (i &BIT (2 )); +const bool do_empty = empty && (i &BIT (3 )); +struct timespec start; + +if (do_empty) +kvm_clear_gsi_routes (vm); + +kvm_route_msi (vm, gsi, vcpu, vector, do_nmi); + +if (irq_affinity && vcpu->id ==0 ) {
Please add comments explaining why affinity related stuff is only applied to vCPU0. I could probably figure it out, but I really shouldn't have to.
Quoted 5 lines
tools/testing/selftests/kvm/vfio_pci_irq_test.c new file +4
-0+ irq_cpu =rand () %get_nprocs (); + + ret =fprintf (irq_affinity_fp,"%d\n" , irq_cpu); +TEST_ASSERT (ret >0 ,"Failed to affinitize IRQ-%d to CPU %d" , irq, irq_cpu);
tools/testing/selftests/kvm/vfio_pci_irq_test.c
Quoted 10 lines
tools/testing/selftests/kvm/vfio_pci_irq_test.c new file +8
-0+ } + +if (pin_vcpus && vcpu->id ==0 ) +pin_vcpu_threads (nr_vcpus,rand () %get_nprocs (), &available_cpus); + +for (j =0 ; j < nr_vcpus; j++) { +TEST_ASSERT_EQ (READ_FROM_GUEST (vm, guest_received_irq[vcpu->id]), false); +TEST_ASSERT_EQ (READ_FROM_GUEST (vm, guest_received_nmi[vcpu->id]), false);
These won't generate helpful assert messages.
tools/testing/selftests/kvm/vfio_pci_irq_test.c
Quoted 11 lines
tools/testing/selftests/kvm/vfio_pci_irq_test.c new file +11
-0+ } + +send_msi (device, use_device_msi, msi); + +clock_gettime (CLOCK_MONOTONIC , &start); +for (;;) { +if (do_nmi &&READ_FROM_GUEST (vm, guest_received_nmi[vcpu->id])) +break ; + +if (!do_nmi &&READ_FROM_GUEST (vm, guest_received_irq[vcpu->id])) +break ;
received_irq = do_nmi ? &guest_received_nmi[vcpu->id] :
&guest_received_irq[vcpu->id];
while (!READ_FROM_GUEST(vm, *received_irq))
if (timespec_to_ns(timespec_elapsed(start)) > TIMEOUT_NS) {
...
}
cpu_relax();
}tools/testing/selftests/kvm/vfio_pci_irq_test.c
Quoted 8 lines
tools/testing/selftests/kvm/vfio_pci_irq_test.c new file +8
-0+ +if (timespec_to_ns (timespec_elapsed (start)) >TIMEOUT_NS ) { +printf ("Timeout waiting for interrupt!\n" ); +printf (" vCPU: %d\n" , vcpu->id); +printf (" do_nmi: %d\n" , do_nmi); +printf (" do_empty: %d\n" , do_empty); +if (irq_affinity) +printf (" irq_cpu: %d\n" , irq_cpu);
vfio_pci_irq_test.c: In function ‘main’:
vfio_pci_irq_test.c:469:41: error: ‘irq_cpu’ may be used uninitialized [-Werror=maybe-uninitialized]
469 | printf(" irq_cpu: %d\n", irq_cpu);
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
vfio_pci_irq_test.c:332:13: note: ‘irq_cpu’ was declared here
332 | int irq_cpu;
| ^~~~~~~tools/testing/selftests/kvm/vfio_pci_irq_test.c
Quoted 6 lines
tools/testing/selftests/kvm/vfio_pci_irq_test.c new file +6
-0+if (pin_vcpus) +printf (" vcpu_cpu: %d\n" ,get_cpu (vcpu)); + +TEST_FAIL ("vCPU never received IRQ!\n" ); + } + }
TEST_ASSERT(guest_received_irq[vcpu->id] != guest_received_nmi[vcpu->id], "blah blah blah"); WRITE_TO_GUEST(vm, *received_irq, false);
Quoted 34 lines
tools/testing/selftests/kvm/vfio_pci_irq_test.c new file +31
-0+if (do_nmi) +WRITE_TO_GUEST (vm, guest_received_nmi[vcpu->id], false); +else +WRITE_TO_GUEST (vm, guest_received_irq[vcpu->id], false); + } + +WRITE_TO_GUEST (vm, done, true); + +for (i =0 ; i < nr_vcpus; i++) { +if (block) { +kvm_route_msi (vm, gsi, vcpus[i], vector, false); +send_msi (device, false, msi); + } + +pthread_join (vcpu_threads[i],NULL ); + } + +if (irq_affinity) +fclose (irq_affinity_fp); + +printf ("Host interrupts handled:\n" ); +printf (" IRQ-%d: %lu\n" , irq,get_irq_count (irq) - irq_count); +printf (" Posted-interrupt notification events: %lu\n" , +__get_irq_count ("PIN:" ) - pin_count); +printf (" Posted-interrupt wakeup events: %lu\n" , +__get_irq_count ("PIW:" ) - piw_count); + +vfio_pci_device_cleanup (device); + +return 0 ; +}Signature · 1 line
2.51.0.384.g4c02a37b29-goog
Re: [PATCH 2/2] KVM: selftests: Add a test for vfio-pci device IRQ delivery to vCPUs
On Mon, Oct 27, 2025 at 9:52 AM Sean Christopherson <[email protected]> wrote:
Thank you for the feedback!
Quoted 7 lines · David Matlack
Quoted 3 lines
This test only supports
x86_64for now, but can be ported to other architectures in the future.Can it though? There are bits and pieces that can be reused, but this test is x86 through and through.
Delivering MSIs from a vfio-pci device into a guest is not an x86-specific thing. But I think you could make the argument that it will be simpler for each architecture to have their own version of this test, with some shared code, rather than the other way around.
Quoted 5 lines
Quoted 2 lines
tools/testing/selftests/kvm/Makefile.kvm | 1 + .../testing/selftests/kvm/vfio_pci_irq_test.c | 507 ++++++++++++++++++Please break this into multiple patches, e.g. a "basic" patch and and then roughly one per "advanced" command line option.
Will do.
Quoted 3 lines
Quoted 1 line
+static
pid_tvcpu_tids[KVM_MAX_VCPUS];s/tids/pids?
This stores output from gettid(), so tids felt more appropriate.
Quoted 3 lines
Quoted 1 line
+#define
TIMEOUT_NS(2ULL * 1000 * 1000 * 1000)The timeout should be configurable via command line.
Will do.
Quoted 11 lines
Review
Quoted 9 lines
(quoted diff) +9
-0+#define READ_FROM_GUEST(_vm, _variable) ({ \ + sync_global_from_guest(_vm, _variable); \ + READ_ONCE(_variable); \ +}) + +#define WRITE_TO_GUEST(_vm, _variable, _value) do { \ + WRITE_ONCE(_variable, _value); \ + sync_global_to_guest(_vm, _variable); \ +} while (0)These belong in a separate patch, and in
tools/testing/selftests/kvm/include/kvm_util.h.
Will do.
Quoted 8 lines
Quoted 1 line
+static void
guest_enable_interrupts(void)This is a misleading name, e.g. I would expect it to _just_ be
sti_nop(). If the intent is to provide a hook for a potential non-x86 implementation, then it should probably be split intoguest_arch_test_setup()andguest_arch_irq_enable()or so (to align with the kernel's local_irq_{dis,en}able()).As is, I would just omit the helper.
Will do.
Quoted 7 lines
Review
Quoted 3 lines
(quoted diff) +3
-0+static void guest_irq_handler(struct ex_regs *regs) +{ + WRITE_ONCE(guest_received_irq[guest_get_vcpu_id()], true);Hmm, using APID ID works, but I don't like the hidden dependency on the library using ascending IDs starting from '0'. This would also be a good opportunity to improve the core infrastructure.
What dependency are you referring to? I think the only requirement is vcpu->id == APIC ID. Are you saying tests should not make that assumption?
Quoted 3 lines
Quoted 1 line
+
WRITE_ONCE(vcpu_tids[vcpu->id], syscall(__NR_gettid));Please add wrapper in
tools/testing/selftests/kvm/include/kvm_syscalls.h.
Will do.
Quoted 11 lines
Review
Quoted 8 lines
(quoted diff) +8
-0+static void pin_vcpu_threads(int nr_vcpus, int start_cpu, cpu_set_t *available_cpus) +{ + const size_t size = sizeof(cpu_set_t); + int nr_cpus, cpu, vcpu_index = 0; + cpu_set_t target_cpu; + int r; + + nr_cpus = get_nprocs();Generally speaking, KVM selftests try to avoid affining tasks to CPUs that are outside of the original affinity list. See various usage of
sched_getaffinity().
available_cpus is initialized by calling sched_getaffinity().
Quoted 8 lines
Review
Quoted 6 lines
(quoted diff) +6
-0+static FILE *open_proc_interrupts(void) +{ + FILE *fp; + + fp = fopen("/proc/interrupts", "r"); + TEST_ASSERT(fp, "fopen(/proc/interrupts) failed");
open_path_or_exit()?
I guess I'll have to rework this code to use fds instead of FILE?
Quoted 6 lines
Review
Quoted 4 lines
(quoted diff) +3
-0+ + return fp; +}And all of these /proc helpers belong in library code.
Will do.
Quoted 5 lines
Quoted 5 lines
(quoted diff) +5
-0+static int setup_msi(struct vfio_pci_device *device, bool use_device_msi) +{ + const int flags = MAP_SHARED | MAP_ANONYMOUS; + const int prot = PROT_READ | PROT_WRITE; + struct vfio_dma_region *region;
Quoted 8 lines
Review
Quoted 6 lines
(quoted diff) +6
-0+ + if (use_device_msi) { + /* A driver is required to generate an MSI. */ + TEST_REQUIRE(device->driver.ops); + + /* Set up a DMA-able region for the driver to use. */Why?
I will extend the comment to explain.
Each driver needs DMA-able memory so that it can post commands to the device to get the device to perform actions like sending an MSI and doing a memcpy.
You might wonder why tests have to worry about setting up this region and why the driver doesn't just do it automatically. That is because some tests will want to control how the memory is mapped in the host (e.g. Live Update tests want to use memfds so it can persist them across Live Update) and how it is mapped into the IOMMU.
That being said, I think it would be worth adding a helper to set up a default mapping for drivers for tests that don't care.
Quoted 7 lines
Review
Quoted 5 lines
(quoted diff) +4
-0+ region = &device->driver.region; + region->iova = 0; + region->size = SZ_2M; + region->vaddr = mmap(NULL, region->size, prot, flags, -1, 0);
kvm_mmap()
Will do.
Quoted 7 lines
Quoted 1 line
+static void
send_msi(struct vfio_pci_device *device, bool use_device_msi, int msi)IMO, this helper does more harm than good. There is only one real user, the second call unconditionally passes %false for @use_device_msi.
If you drop the helper, than there should be no need to assert that the MSI is the device MSI on *every* send via device.
Will do.
Quoted 8 lines
Review
Quoted 4 lines
(quoted diff) +4
-0+int main(int argc, char **argv) +{ + /* Random non-reserved vector and GSI to use for the device IRQ */ + const u8 vector = 0xe0;s/random/arbitrary
Why not make it truly random?
Only because there's already a lot going on in this test. Do you think it's worth randomizing these?
Quoted 6 lines
Review
Quoted 3 lines
(quoted diff) +3
-0+ irq_count = get_irq_count(irq); + pin_count = __get_irq_count("PIN:"); + piw_count = __get_irq_count("PIW:");This is obviously very Intel specific information. If you're going to print the posted IRQ info, then the test should also print e.g. AMD GALogIntr events.
I saw PIN and PIW in /proc/interrupts when I tested on AMD hosts, that's why I included them both by default.
I can look into adding GALogIntr if you want.
Quoted 7 lines
Review
Quoted 3 lines
(quoted diff) +3
-0+ if (pin_vcpus) { + ret = sched_getaffinity(vcpu_tids[0], sizeof(available_cpus), &available_cpus); + TEST_ASSERT(ret == 0, "sched_getaffinity() failed");!ret
Though this is another syscall that deserves a wrapper in
kvm_syscalls.h.
Will do.
Quoted 6 lines
Review
Quoted 3 lines
(quoted diff) +3
-0+ if (nr_vcpus > CPU_COUNT(&available_cpus)) { + printf("There are more vCPUs than pCPUs; refusing to pin.\n"); + pin_vcpus = false;Why is this not an assertion? Alternatively, why not double/triple/quadruple up as needed?
I don't recall why I added this... I can probably just drop it in the next version. I'll report back here if I remember why when working on v2.
Quoted 8 lines
Review
Quoted 6 lines
(quoted diff) +6
-0+ if (irq_affinity) { + char path[PATH_MAX]; + + snprintf(path, sizeof(path), "/proc/irq/%d/smp_affinity_list", irq); + irq_affinity_fp = fopen(path, "w"); + TEST_ASSERT(irq_affinity_fp, "fopen(%s) failed", path);More code that belongs in the library.
Will do.
Quoted 26 lines
Review
Quoted 2 lines
(quoted diff) +2
-0+ /* Set a consistent seed so that test are repeatable. */ + srand(0);We should really figure out a solution for reproducible random numbers in the host. Ah, and kvm_selftest_init()'s handling of guest random seeds is flawed, because it does
random()withoutsrand()and so AFAICT, gets the same seed every time. E.g. seems like we want something like this, but with a way to override "random_seed" from a test.tools/testing/selftests/kvm/lib/kvm_util.c +2
-0@@ -2310,6 +2310,7 @@ void __attribute((constructor)) kvm_selftest_init(void)struct sigaction sig_sa = { .sa_handler = report_unexpected_signal, }; +int random_seed;/* Tell stdout not to buffer its content. */ setbuf (stdout,NULL );@@ -2319,8 +2320,13 @@ void __attribute((constructor)) kvm_selftest_init(void)sigaction (SIGILL , &sig_sa,NULL );sigaction (SIGFPE , &sig_sa,NULL ); + random_seed =time (NULL );
tools/testing/selftests/kvm/lib/kvm_util.c void __attribute((constructor)) kvm_selftest_init(void) · line 2332
Quoted 10 lines
tools/testing/selftests/kvm/lib/kvm_util.c +5
-1+srand (random_seed); + guest_random_seed = last_guest_seed =random (); -pr_info ("Random seed: 0x%x\n" , guest_random_seed); + +pr_info ("Guest random seed: 0x%x (srand: 0x%x)\n" , + guest_random_seed, random_seed);kvm_selftest_arch_init (); }
Just to make sure I understand: You are proposing using the current time as the seed and printing it to the console. That way each run uses a different random seed and we get broader test coverage. Then if someone wants to reproduce a test result, there would be some way for them to override the seed via cmdline? That sounds reasonable to me, I can take a look at adding that in the next version.
Quoted 4 lines
Quoted 1 line
+ if (
irq_affinity&&vcpu->id== 0) {Please add comments explaining why affinity related stuff is only applied to vCPU0. I could probably figure it out, but I really shouldn't have to.
Will do.
Quoted 5 lines
Review
Quoted 3 lines
(quoted diff) +3
-0+ for (j = 0; j < nr_vcpus; j++) { + TEST_ASSERT_EQ(READ_FROM_GUEST(vm, guest_received_irq[vcpu->id]), false); + TEST_ASSERT_EQ(READ_FROM_GUEST(vm, guest_received_nmi[vcpu->id]), false);These won't generate helpful assert messages.
Good point. I'll improve these in the next version.
Quoted 7 lines
Review
Quoted 6 lines
(quoted diff) +6
-0+ for (;;) { + if (do_nmi && READ_FROM_GUEST(vm, guest_received_nmi[vcpu->id])) + break; + + if (!do_nmi && READ_FROM_GUEST(vm, guest_received_irq[vcpu->id])) + break;
Quoted 9 lines
received_irq = do_nmi ? &guest_received_nmi[vcpu->id] :
&guest_received_irq[vcpu->id];
while (!READ_FROM_GUEST(vm, *received_irq))
if (timespec_to_ns(timespec_elapsed(start)) > TIMEOUT_NS) {
...
}
cpu_relax();
}LGTM
Quoted 45 lines
Quoted 40 lines
(quoted diff) +39
-1+ + if (timespec_to_ns(timespec_elapsed(start)) > TIMEOUT_NS) { + printf("Timeout waiting for interrupt!\n"); + printf(" vCPU: %d\n", vcpu->id); + printf(" do_nmi: %d\n", do_nmi); + printf(" do_empty: %d\n", do_empty); + if (irq_affinity) + printf(" irq_cpu: %d\n", irq_cpu> + if (do_nmi) + WRITE_TO_GUEST(vm, guest_received_nmi[vcpu->id], false); + else + WRITE_TO_GUEST(vm, guest_received_irq[vcpu->id], false); + } + + WRITE_TO_GUEST(vm, done, true); + + for (i = 0; i < nr_vcpus; i++) { + if (block) { + kvm_route_msi(vm, gsi, vcpus[i], vector, false); + send_msi(device, false, msi); + } + + pthread_join(vcpu_threads[i], NULL); + } + + if (irq_affinity) + fclose(irq_affinity_fp); + + printf("Host interrupts handled:\n"); + printf(" IRQ-%d: %lu\n", irq, get_irq_count(irq) - irq_count); + printf(" Posted-interrupt notification events: %lu\n", + __get_irq_count("PIN:") - pin_count); + printf(" Posted-interrupt wakeup events: %lu\n", + __get_irq_count("PIW:") - piw_count); + + vfio_pci_device_cleanup(device); + + return 0; +} --Quoted 2 lines
2.51.0.384.g4c02a37b29-goog
);
vfio_pci_irq_test.c: In function ‘main’:
vfio_pci_irq_test.c:469:41: error: ‘irq_cpu’ may be used uninitialized [-Werror=maybe-uninitialized]
Quoted 5 lines
469 | printf(" irq_cpu: %d\n", irq_cpu);
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
vfio_pci_irq_test.c:332:13: note: ‘irq_cpu’ was declared here
332 | int irq_cpu;
| ^~~~~~~Ack, will fix.
Quoted 7 lines
Review
Quoted 6 lines
(quoted diff) +6
-0+ if (pin_vcpus) + printf(" vcpu_cpu: %d\n", get_cpu(vcpu)); + + TEST_FAIL("vCPU never received IRQ!\n"); + } + }
Quoted 5 lines
TEST_ASSERT(guest_received_irq[vcpu->id] !=
guest_received_nmi[vcpu->id],
"blah blah blah");
WRITE_TO_GUEST(vm, *received_irq, false);LGTM
Sean Christopherson
reviewerRe: [PATCH 2/2] KVM: selftests: Add a test for vfio-pci device IRQ delivery to vCPUs
Quoted 9 lines · David Matlack
On Mon, Oct 27, 2025 at 9:52 AM Sean Christopherson <[email protected]> wrote:
Thank you for the feedback!
Quoted 5 lines · David Matlack
Quoted 3 lines
This test only supports
x86_64for now, but can be ported to other architectures in the future.
Quoted 7 lines
Quoted 2 lines
Can it though? There are bits and pieces that can be reused, but this test is x86 through and through.
Delivering MSIs from a vfio-pci device into a guest is not an x86-specific thing. But I think you could make the argument that it will be simpler for each architecture to have their own version of this test, with some shared code, rather than the other way around.
Right, the concept is x86, but the code as written is very x86 centric.
Quoted 10 lines
Quoted 7 lines
Review
Quoted 3 lines
(quoted diff) +3
-0+static void guest_irq_handler(struct ex_regs *regs) +{ + WRITE_ONCE(guest_received_irq[guest_get_vcpu_id()], true);Hmm, using APID ID works, but I don't like the hidden dependency on the library using ascending IDs starting from '0'. This would also be a good opportunity to improve the core infrastructure.
What dependency are you referring to? I think the only requirement is
vcpu->id== APIC ID
Yep, this one.
Quoted 1 line
Are you saying tests should not make that assumption?
Ya, ideally, they would not.
Quoted 18 lines
Quoted 3 lines
Quoted 1 line
+
WRITE_ONCE(vcpu_tids[vcpu->id], syscall(__NR_gettid));Please add wrapper in
tools/testing/selftests/kvm/include/kvm_syscalls.h.Will do.
Quoted 11 lines
Review
Quoted 8 lines
(quoted diff) +8
-0+static void pin_vcpu_threads(int nr_vcpus, int start_cpu, cpu_set_t *available_cpus) +{ + const size_t size = sizeof(cpu_set_t); + int nr_cpus, cpu, vcpu_index = 0; + cpu_set_t target_cpu; + int r; + + nr_cpus = get_nprocs();Generally speaking, KVM selftests try to avoid affining tasks to CPUs that are outside of the original affinity list. See various usage of
sched_getaffinity().
Quoted 1 line
available_cpusis initialized by callingsched_getaffinity().
Ah, I missed that in the for-loop.
Quoted 10 lines
Quoted 8 lines
Review
Quoted 6 lines
(quoted diff) +6
-0+static FILE *open_proc_interrupts(void) +{ + FILE *fp; + + fp = fopen("/proc/interrupts", "r"); + TEST_ASSERT(fp, "fopen(/proc/interrupts) failed");
open_path_or_exit()?I guess I'll have to rework this code to use fds instead of FILE?
Hmm, not necessarily. E.g. maybe open_file_or_exit()? I don't have a strong preference (and have put zero thought into what would work well).
Quoted 11 lines
Quoted 8 lines
Review
Quoted 4 lines
(quoted diff) +4
-0+int main(int argc, char **argv) +{ + /* Random non-reserved vector and GSI to use for the device IRQ */ + const u8 vector = 0xe0;s/random/arbitrary
Why not make it truly random?
Only because there's already a lot going on in this test. Do you think it's worth randomizing these?
Probably? But without a command line option, to keep this somewhat less crazy?
Quoted 8 lines
Quoted 6 lines
Review
Quoted 3 lines
(quoted diff) +3
-0+ irq_count = get_irq_count(irq); + pin_count = __get_irq_count("PIN:"); + piw_count = __get_irq_count("PIW:");This is obviously very Intel specific information. If you're going to print the posted IRQ info, then the test should also print e.g. AMD GALogIntr events.
I saw PIN and PIW in
/proc/interruptswhen I tested on AMD hosts,
The kernel really should key off CONFIG_KVM_INTEL={m,y}, not CONFIG_KVM (though in practice it doesn't matter all that much).
Quoted 1 line
that's why I included them both by default.
I think it makes sense to only print PIN+PIW on Intel, otherwise it's pure noise.
Quoted 1 line
I can look into adding GALogIntr if you want.
PIN is midly interesting. PIW and GALogIntr are much more interesting from a coverage perspective. E.g. if the deltas for PIW and GALogIntr are '0', then the test likely isn't exercising the blocking path.
Quoted 34 lines
Quoted 34 lines
Review
Quoted 2 lines
(quoted diff) +2
-0+ /* Set a consistent seed so that test are repeatable. */ + srand(0);We should really figure out a solution for reproducible random numbers in the host. Ah, and kvm_selftest_init()'s handling of guest random seeds is flawed, because it does
random()withoutsrand()and so AFAICT, gets the same seed every time. E.g. seems like we want something like this, but with a way to override "random_seed" from a test.tools/testing/selftests/kvm/lib/kvm_util.c +7
-1@@ -2310,6 +2310,7 @@ void __attribute((constructor)) kvm_selftest_init(void)struct sigaction sig_sa = { .sa_handler = report_unexpected_signal, }; +int random_seed;/* Tell stdout not to buffer its content. */ setbuf (stdout,NULL );@@ -2319,8 +2320,13 @@ void __attribute((constructor)) kvm_selftest_init(void)sigaction (SIGILL , &sig_sa,NULL );sigaction (SIGFPE , &sig_sa,NULL ); + random_seed =time (NULL ); +srand (random_seed); + guest_random_seed = last_guest_seed =random (); -pr_info ("Random seed: 0x%x\n" , guest_random_seed); + +pr_info ("Guest random seed: 0x%x (srand: 0x%x)\n" , + guest_random_seed, random_seed);
Quoted 5 lines
tools/testing/selftests/kvm/lib/kvm_util.c void __attribute((constructor)) kvm_selftest_init(void) · line 2332
Quoted 2 lines
tools/testing/selftests/kvm/lib/kvm_util.c +0
-0kvm_selftest_arch_init (); }Just to make sure I understand: You are proposing using the current time as the seed
Or anything that's somewhat random. I don't know what glibc is doing under the hood, so it's entirely possible/likely there's a much better method.
Quoted 5 lines
and printing it to the console. That way each run uses a different random seed and we get broader test coverage. Then if someone wants to reproduce a test result, there would be some way for them to override the seed via cmdline? That sounds reasonable to me, I can take a look at adding that in the next version.
Yep. That was the intent with the guest_random_seed, I just didn't implement it very well :-)
Regarding reproducibility, one idea would be to have kvm_selftest_init() pull the seed from an environment variable, e.g. so that reproducing with a specific seed doesn't require hacking or test support. I'm not entirely sure I like the idea of using environment variables though...
Sean Christopherson
reviewerQuoted 3 lines · David Matlack
This series can be found on GitHub:
https://github.com/dmatlack/linux/tree/kvm/selftests/vfio_pci_irq_test/v1
...
Quoted 10 lines
David Matlack (2):
KVM: selftests: Build and link sefltests/vfio/lib into KVM selftests
KVM: selftests: Add a test for vfio-pci device IRQ delivery to vCPUstools/testing/selftests/kvm/Makefile.kvm | 6 +- .../testing/selftests/kvm/vfio_pci_irq_test.c | 507 ++++++++++++++++++ 2 files changed, 512 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/kvm/vfio_pci_irq_test.c
Quoted 3 lines
base-commit:
093458c58f830d0a713fab0de037df5f0ce24fef
prerequisite-patch-id:72dce9cd586ac36ea378354735d9fabe2f3c445e
prerequisite-patch-id:a8c7ccfd91ce3208f328e8af7b25c83bff8d464d
Please don't base series on prerequisite patches unless there is a hard dependency.
ffdc6a9d6d9eb20c855404e2c09b6b2ea25b4a04 KVM: selftests: Rename $(ARCH_DIR) to $(SRCARCH) 9dc0c1189dfa1f4eef3856445fa72c9fb1e14d1c Revert "KVM: selftests: Override ARCH for x86_64 instead of using ARCH_DIR"
By all means, do testing on top of such patches if that's what your environment effectively dictates, but rebase/drop such prereqs before posting, as they add noise and friction. E.g. these patches don't apply cleanly on kvm-x86/next due to the prereqs.
Quoted 24 lines · Sean Christopherson
Quoted 3 lines · David Matlack
This series can be found on GitHub:
https://github.com/dmatlack/linux/tree/kvm/selftests/vfio_pci_irq_test/v1
...
Quoted 10 lines
David Matlack (2):
KVM: selftests: Build and link sefltests/vfio/lib into KVM selftests
KVM: selftests: Add a test for vfio-pci device IRQ delivery to vCPUstools/testing/selftests/kvm/Makefile.kvm | 6 +- .../testing/selftests/kvm/vfio_pci_irq_test.c | 507 ++++++++++++++++++ 2 files changed, 512 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/kvm/vfio_pci_irq_test.cQuoted 3 lines
base-commit:
093458c58f830d0a713fab0de037df5f0ce24fef
prerequisite-patch-id:72dce9cd586ac36ea378354735d9fabe2f3c445e
prerequisite-patch-id:a8c7ccfd91ce3208f328e8af7b25c83bff8d464dPlease don't base series on prerequisite patches unless there is a hard dependency.
Quoted 7 lines
ffdc6a9d6d9eb20c855404e2c09b6b2ea25b4a04KVM: selftests: Rename $(ARCH_DIR) to $(SRCARCH)9dc0c1189dfa1f4eef3856445fa72c9fb1e14d1cRevert "KVM: selftests: Override ARCH forx86_64instead of usingARCH_DIR"By all means, do testing on top of such patches if that's what your environment effectively dictates, but rebase/drop such prereqs before posting, as they add noise and friction. E.g. these patches don't apply cleanly on kvm-x86/next due to the prereqs.
Makes sense. I will send patches applied without prereqs specific to my environment next time.