KVM: Send reschedule IPIs instead of SMP function calls for non-waiting requests
Sean Christopherson
reviewerPATCH 1/1 KVM: Send reschedule IPIs instead of SMP function calls for non-waiting requests
When kicking multiple vCPUs for a non-waiting request, send a reschedule IPI to each vCPU instead of batching IPIs via a single SMP function call so that KVM can make non-waiting requests with IRQs disabled. I.e. handle multi-vCPU kicks the same way KVM handles single-vCPU kicks, for the exact same reason.
Avoiding SMP function calls for non-waiting requests fixes a potential deadlock if KVM VMX triggers KVM_REQ_UNBLOCK to all vCPUs when starting device assignmennt:
WARNING: kernel/smp.c:943 at smp_call_function_many_cond
kvm_make_all_cpus_request [kvm]
kvm_arch_irq_bypass_add_producer [kvm]
__connect [irqbypass]
irq_bypass_register_producer [irqbypass]
vfio_msi_set_vector_signal [vfio_pci_core]Send reschedule IPIs immediately to avoid having to re-walk the set of CPUs, even though it makes the code slightly convoluted. It's also generally advantageous to kick vCPUs as soon as possible, e.g. so that the target vCPU(s) can react to pending events/actions, but that's purely nice-to-have as there are very few paths in KVM that kick multiple vCPUs for non-waiting requests.
--- virt/kvm/kvm_main.c | 17 ++++++++++++++--- 1 file changed, 14 insertions(+), 3 deletions(-)
virt/kvm/kvm_main.c +14 -3
@@ -218,7 +218,10 @@ static inline bool kvm_kick_many_cpus(struct cpumask *cpus, bool wait)if (cpumask_empty (cpus))return false; -smp_call_function_many (cpus, ack_kick,NULL , wait); +/* See kvm_vcpu_kick() and kvm_make_vcpu_request() for details. */ +if (wait) +smp_call_function_many (cpus, ack_kick,NULL , wait); +return true; }
@@ -241,12 +244,20 @@ static void kvm_make_vcpu_request(struct kvm_vcpu *vcpu, unsigned int req,* satisfied if the vCPU migrates. Entering READING_SHADOW_PAGE_TABLES * after this point is also OK, as the requirement is only that KVM wait * for vCPUs that were reading SPTEs _before_ any changes were -* finalized. See kvm_vcpu_kick() for more details on handling requests. +* finalized. +* +* See kvm_vcpu_kick() for more details on handling requests, including +* the use of a reschedule IPI for non-waiting requests. If waiting is +* required, batch the SMP function call until after all to-be-kicked +* CPUs are gathered, to avoid waiting multiple times. */ if (kvm_request_needs_ipi (vcpu, req)) { cpu =READ_ONCE (vcpu->cpu); -if (cpu !=-1 && cpu != current_cpu) +if (cpu !=-1 && cpu != current_cpu) { +if (!(req &KVM_REQUEST_WAIT )) +smp_send_reschedule (cpu);__cpumask_set_cpu (cpu, tmp); + } } }
base-commit: d21eef2084a9af87ac2a783bc7c2b175ba114e4e
Signature · 1 line
2.56.1.703.gf848ed807c-goog
[email protected]
AI reviewThank you for your contribution! Sashiko AI review found 2 potential issues to consider: - [High] KVM: per-CPU cpu_kick_mask corruption in kvm_make_all_cpus_request() - [High] KVM: s390 build…
Thank you for your contribution! Sashiko AI review found 2 potential issues to consider:
- [High] KVM: per-CPU cpu_kick_mask corruption in kvm_make_all_cpus_request()
- [High] KVM: s390 build failure in kvm_make_vcpu_request() with CONFIG_KVM=m
--
commit 38c2c2306d98411374ab3430ab319733689457e7
Author: Sean Christopherson <[email protected]>
KVM: Send reschedule IPIs instead of SMP function calls for non-waiting requests
This commit sends a reschedule IPI to each target vCPU instead of batching IPIs via smp_call_function_many() when making non-waiting multi-vCPU requests, allowing non-waiting requests to be made with interrupts disabled.
Quoted 4 lines
virt/kvm/kvm_main.c +0
-0
[ ... ]
Quoted 8 lines
virt/kvm/kvm_main.c +3
-1@@ -241,12 +244,20 @@ static void kvm_make_vcpu_request(struct kvm_vcpu *vcpu, unsigned int req,* satisfied if the vCPU migrates. Entering READING_SHADOW_PAGE_TABLES * after this point is also OK, as the requirement is only that KVM wait * for vCPUs that were reading SPTEs _before_ any changes were -* finalized. See kvm_vcpu_kick() for more details on handling requests. +* finalized. +* +* See kvm_vcpu_kick() for more details on handling requests, including
virt/kvm/kvm_main.c static void kvm_make_vcpu_request(struct kvm_vcpu *vcpu, unsigned int req, · line 258
Quoted 10 lines
virt/kvm/kvm_main.c +6
-1+* the use of a reschedule IPI for non-waiting requests. If waiting is +* required, batch the SMP function call until after all to-be-kicked +* CPUs are gathered, to avoid waiting multiple times. */ if (kvm_request_needs_ipi (vcpu, req)) { cpu =READ_ONCE (vcpu->cpu); -if (cpu !=-1 && cpu != current_cpu) +if (cpu !=-1 && cpu != current_cpu) { +if (!(req &KVM_REQUEST_WAIT )) +smp_send_reschedule (cpu);
[Severity: High]
Does calling smp_send_reschedule() here break the build on s390 when CONFIG_KVM=m?
On s390, smp_send_reschedule() expands to arch_smp_send_reschedule(), which is not exported to modules. __kvm_vcpu_kick() is guarded by #ifndef CONFIG_S390 for that reason (see 0266c894b51c ("KVM: put back #ifndef CONFIG_S390 around kvm_vcpu_kick")), whereas kvm_make_vcpu_request() is built into kvm.ko on s390.
virt/kvm/kvm_main.c static void kvm_make_vcpu_request(struct kvm_vcpu *vcpu, unsigned int req, · line 262
Quoted 4 lines
virt/kvm/kvm_main.c +1
-0__cpumask_set_cpu (cpu, tmp); + } } }
[Severity: High]
Can a non-waiting request in interrupt context clobber the per-CPU cpu_kick_mask of an in-progress waiting request on the same CPU?
kvm_make_all_cpus_request() and kvm_make_vcpus_request_mask() only disable preemption via get_cpu() while building cpu_kick_mask, leaving interrupts enabled for waiting requests, and still clear and populate cpu_kick_mask even when KVM_REQUEST_WAIT is not set:
virt/kvm/kvm_main.c:kvm_make_all_cpus_request() {
me = get_cpu();
cpus = this_cpu_cpumask_var_ptr(cpu_kick_mask);
cpumask_clear(cpus);
kvm_for_each_vcpu(i, vcpu, kvm)
kvm_make_vcpu_request(vcpu, req, cpus, me);
called = kvm_kick_many_cpus(cpus, !!(req & KVM_REQUEST_WAIT));
put_cpu();...
}
If an interrupt fires mid-loop and makes a non-waiting multi-vCPU request (for example on arm64 via irqfd_wakeup() -> kvm_arch_set_irq_inatomic() -> ... -> vgic_queue_irq_unlock()):
kvm_flush_remote_tlbs()
kvm_make_all_cpus_request(kvm, KVM_REQ_TLB_FLUSH)
cpumask_clear(cpus)
kvm_make_vcpu_request(vcpu0, ...)
kvm_vcpu_exiting_guest_mode(vcpu0)
cmpxchg(&vcpu0->mode, IN_GUEST_MODE, EXITING_GUEST_MODE)
__cpumask_set_cpu(vcpu0->cpu, cpus)
<hardirq>
irqfd_wakeup()
...
kvm_make_all_cpus_request(kvm, KVM_REQ_IRQ_PENDING)
cpumask_clear(cpus) <-- wipes vcpu0->cpu from cpu_kick_mask
kvm_make_vcpu_request(vcpu0, ...)
kvm_request_needs_ipi(vcpu0, ...) <-- false (EXITING_GUEST_MODE)
kvm_kick_many_cpus(cpus, true) <-- misses vcpu0->cpuBecause the outer request already transitioned vcpu0->mode from IN_GUEST_MODE to EXITING_GUEST_MODE, the nested non-waiting request sees mode != IN_GUEST_MODE and neither sends a reschedule IPI nor re-adds vcpu0->cpu to cpu_kick_mask. When the interrupt returns, kvm_kick_many_cpus() skips kicking and waiting for vcpu0, so the waiting request (such as a remote TLB flush) can complete while vcpu0 is still in guest mode or reading shadow page tables.
Only touching cpu_kick_mask when KVM_REQUEST_WAIT is set (and tracking whether any CPU was kicked with a local boolean for the return value) would avoid this.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
Sean Christopherson
reviewer+Oliver and Marc,
FYI on what I think is a pre-existing TLB flush bug (that can affect other requests as well).
Quoted 5 lines · On Tue, Nov 17, 2026, [email protected] wrote:
KVM: Send reschedule IPIs instead of SMP function calls for non-waiting requests
This commit sends a reschedule IPI to each target vCPU instead of batching IPIs via
smp_call_function_many()when making non-waiting multi-vCPU requests, allowing non-waiting requests to be made with interrupts disabled.
...
Quoted 51 lines
[Severity: High]
Can a non-waiting request in interrupt context clobber the per-CPUcpu_kick_maskof an in-progress waiting request on the same CPU?
kvm_make_all_cpus_request()andkvm_make_vcpus_request_mask()only disable preemption viaget_cpu()while buildingcpu_kick_mask, leaving interrupts enabled for waiting requests, and still clear and populatecpu_kick_maskeven whenKVM_REQUEST_WAITis not set:virt/kvm/kvm_main.c:kvm_make_all_cpus_request() { me = get_cpu(); cpus = this_cpu_cpumask_var_ptr(cpu_kick_mask); cpumask_clear(cpus); kvm_for_each_vcpu(i, vcpu, kvm) kvm_make_vcpu_request(vcpu, req, cpus, me); called = kvm_kick_many_cpus(cpus, !!(req & KVM_REQUEST_WAIT)); put_cpu();...
}If an interrupt fires mid-loop and makes a non-waiting multi-vCPU request (for example on arm64 via
irqfd_wakeup()->kvm_arch_set_irq_inatomic()-> ... ->vgic_queue_irq_unlock()):
kvm_flush_remote_tlbs()
kvm_make_all_cpus_request(kvm, KVM_REQ_TLB_FLUSH)cpumask_clear(cpus) kvm_make_vcpu_request(vcpu0, ...) kvm_vcpu_exiting_guest_mode(vcpu0) cmpxchg(&vcpu0->mode, IN_GUEST_MODE, EXITING_GUEST_MODE) __cpumask_set_cpu(vcpu0->cpu, cpus) <hardirq> irqfd_wakeup() ... kvm_make_all_cpus_request(kvm, KVM_REQ_IRQ_PENDING) cpumask_clear(cpus) <-- wipes vcpu0->cpu from cpu_kick_mask kvm_make_vcpu_request(vcpu0, ...) kvm_request_needs_ipi(vcpu0, ...) <-- false (EXITING_GUEST_MODE) kvm_kick_many_cpus(cpus, true) <-- misses vcpu0->cpuBecause the outer request already transitioned
vcpu0->modefromIN_GUEST_MODEtoEXITING_GUEST_MODE, the nested non-waiting request sees mode !=IN_GUEST_MODEand neither sends a reschedule IPI nor re-addsvcpu0->cputocpu_kick_mask. When the interrupt returns,kvm_kick_many_cpus()skips kicking and waiting for vcpu0, so the waiting request (such as a remote TLB flush) can complete while vcpu0 is still in guest mode or reading shadow page tables.
Quoted 3 lines
Only touching
cpu_kick_maskwhenKVM_REQUEST_WAITis set (and tracking whether any CPU was kicked with a local boolean for the return value) would avoid this.
Unless I'm misreading this, clobbering cpu_kick_mask and dropping the kick for a waiting request is a pre-existing bug. I'll address this in v2, and make sure the series is tagged for stable@.
Quoted 4 lines · Sean Christopherson
+Oliver and Marc,
FYI on what I think is a pre-existing TLB flush bug (that can affect other requests as well).
Thanks!
Quoted 28 lines
Quoted 28 lines
If an interrupt fires mid-loop and makes a non-waiting multi-vCPU request (for example on arm64 via
irqfd_wakeup()->kvm_arch_set_irq_inatomic()-> ... ->vgic_queue_irq_unlock()):
kvm_flush_remote_tlbs()
kvm_make_all_cpus_request(kvm, KVM_REQ_TLB_FLUSH)cpumask_clear(cpus) kvm_make_vcpu_request(vcpu0, ...) kvm_vcpu_exiting_guest_mode(vcpu0) cmpxchg(&vcpu0->mode, IN_GUEST_MODE, EXITING_GUEST_MODE) __cpumask_set_cpu(vcpu0->cpu, cpus) <hardirq> irqfd_wakeup() ... kvm_make_all_cpus_request(kvm, KVM_REQ_IRQ_PENDING) cpumask_clear(cpus) <-- wipes vcpu0->cpu from cpu_kick_mask kvm_make_vcpu_request(vcpu0, ...) kvm_request_needs_ipi(vcpu0, ...) <-- false (EXITING_GUEST_MODE) kvm_kick_many_cpus(cpus, true) <-- misses vcpu0->cpuBecause the outer request already transitioned
vcpu0->modefromIN_GUEST_MODEtoEXITING_GUEST_MODE, the nested non-waiting request sees mode !=IN_GUEST_MODEand neither sends a reschedule IPI nor re-addsvcpu0->cputocpu_kick_mask. When the interrupt returns,kvm_kick_many_cpus()skips kicking and waiting for vcpu0, so the waiting request (such as a remote TLB flush) can complete while vcpu0 is still in guest mode or reading shadow page tables.
Quoted 7 lines
Quoted 3 lines
Only touching
cpu_kick_maskwhenKVM_REQUEST_WAITis set (and tracking whether any CPU was kicked with a local boolean for the return value) would avoid this.Unless I'm misreading this, clobbering
cpu_kick_maskand dropping the kick for a waiting request is a pre-existing bug. I'll address this in v2, and make sure the series is tagged for stable@.
Yeah, this looks like bad things could happen on arm64 when we pause the VM for global VGIC state changes among other things. CC us on the v2 if you don't mind.
Thanks,
Oliver
Hi Sean,
I tested this in place of my IRQ-bypass patch. The latter was reverted in the test source, and I loaded a kvm.ko built with your change against the running test kernel.
On an Intel UCS C220 M7, a four-vCPU guest used an ENIC VF through VFIO. With the other three vCPUs busy, reloading the guest VF driver traced vfio_msi_set_vector_signal() through vt_pi_start_bypass() to kvm_make_all_cpus_request() with local IRQs disabled. The trace then showed three reschedule IPIs from kvm_make_vcpu_request(), with no SMP function call in that callback and no warning. Posted-interrupt updates occurred. Three more VF reload cycles passed bidirectional traffic after VF traffic became ready.
On an Intel UCS C240 M6 with the same test kernel and patched kvm.ko, irqfd_test, irq_test, vmx_posted_intr_test and smptest passed. I restored the original KVM modules on both hosts after testing.
One source-review question: should the direct IPI path check cpu_online(cpu), as __kvm_vcpu_kick() does? The old batched helper filters offline CPUs, while x86 native_smp_send_reschedule() warns for one. A target vCPU could leave guest mode and its CPU could go offline between the request check and the IPI. I did not test CPU hotplug, and this would not imply a lost request.
Sean Christopherson
reviewerQuoted 19 lines · Satish Kharat
Hi Sean,
I tested this in place of my IRQ-bypass patch. The latter was reverted in the test source, and I loaded a kvm.ko built with your change against the running test kernel.
On an Intel UCS C220 M7, a four-vCPU guest used an ENIC VF through VFIO. With the other three vCPUs busy, reloading the guest VF driver traced
vfio_msi_set_vector_signal()throughvt_pi_start_bypass()tokvm_make_all_cpus_request()with local IRQs disabled. The trace then showed three reschedule IPIs fromkvm_make_vcpu_request(), with no SMP function call in that callback and no warning. Posted-interrupt updates occurred. Three more VF reload cycles passed bidirectional traffic after VF traffic became ready.On an Intel UCS C240 M6 with the same test kernel and patched kvm.ko,
irqfd_test,irq_test,vmx_posted_intr_testand smptest passed. I restored the original KVM modules on both hosts after testing.
Quoted 6 lines
One source-review question: should the direct IPI path check
cpu_online(cpu), as__kvm_vcpu_kick()does? The old batched helper filters offline CPUs, while x86native_smp_send_reschedule()warns for one. A target vCPU could leave guest mode and its CPU could go offline between the request check and the IPI. I did not test CPU hotplug, and this would not imply a lost request.
Ugh, yeah, that WARN could be problematic. Thanks much!
Quoted 2 lines
Tested-by Satish Kharat <[email protected]>
Hi Sean,
Quoted 1 line
Ugh, yeah, that WARN could be problematic. Thanks much!
I tried a guard on top of your patch. The non-waiting path checks cpu_active() before sending a reschedule IPI. Both callers hold get_cpu() through this helper and the send. CPU-down clears cpu_active() and then waits for preempt-disabled readers before taking the CPU offline. That appears to protect the interval between the check and the IPI.
This is the incremental diff I built and tested:
virt/kvm/kvm_main.c +5 -2
@@ -253,9 +253,12 @@*/ if (kvm_request_needs_ipi (vcpu, req)) { cpu =READ_ONCE (vcpu->cpu); -if (cpu !=-1 && cpu != current_cpu) { -if (!(req &KVM_REQUEST_WAIT )) +if (cpu != current_cpu && (unsigned int )cpu < nr_cpu_ids) { +if (!(req &KVM_REQUEST_WAIT )) { +if (!cpu_active (cpu)) +return ;smp_send_reschedule (cpu); + }__cpumask_set_cpu (cpu, tmp); } }
With a four-vCPU VFIO guest, three ENIC VF driver reloads passed bidirectional traffic. During the same run, 20 CPU offline/online cycles completed. The trace recorded 16 non-waiting KVM requests and 11 reschedule IPIs from kvm_make_vcpu_request(), all on the VFIO IRQ-bypass registration path. The traced IPIs did not target the CPU being offlined. There was no new host warning. On a second Intel system, five irqfd_test/irq_test cycles and ten CPU offline/online cycles passed, although its trace only saw waiting requests.
I also tried to force two CPU-down orderings with a separate test-only kvm.ko. It started offlining the target CPU from kernel work while the non-waiting request was held (1) before cpu_active() and (2) after an active result but before smp_send_reschedule(). In both runs the target remained active throughout a 1.2-second hold; CPU-down completed after the request returned. There was no new warning. These runs did not exercise the inactive-CPU branch or reproduce an offline-CPU IPI.
Source inspection suggests a reason: CPU-down's hotplug write lock uses RCU synchronization, which can wait for a preempt-disabled KVM request before reaching sched_cpu_deactivate(). I did not trace that wait. CPU-down could instead reach that point before a new KVM request starts. I have not proved that such a request still sees a vCPU requiring an IPI. Would you consider the guard worthwhile for that ordering, or does the vCPU mode/CPU ordering exclude the target?
Thanks,
Satish