VMX: configure posted interrupt descriptor when assigning device (v5)
Subsystems (6)
PATCH 3/3 KVM: VMX: update vcpu posted-interrupt descriptor when assigning device
In reply to Marcelo TosattiFor VMX, when a vcpu enters HLT emulation, pi_post_block will:
1) Add vcpu to per-cpu list of blocked vcpus.
2) Program the posted-interrupt descriptor "notification vector" to POSTED_INTR_WAKEUP_VECTOR
With interrupt remapping, an interrupt will set the PIR bit for the vector programmed for the device on the CPU, test-and-set the ON bit on the posted interrupt descriptor, and if the ON bit is clear generate an interrupt for the notification vector.
This way, the target CPU wakes upon a device interrupt and wakes up the target vcpu.
Problem is that pi_post_block only programs the notification vector if kvm_arch_has_assigned_device() is true. Its possible for the following to happen:
1) vcpu V HLTs on pcpu P, kvm_arch_has_assigned_device is false, notification vector is not programmed
2) device is assigned to VM
3) device interrupts vcpu V, sets ON bit
(notification vector not programmed, so pcpu P remains in idle)
4) vcpu 0 IPIs vcpu V (in guest), but since pi descriptor ON bit is set, kvm_vcpu_kick is skipped
5) vcpu 0 busy spins on vcpu V's response for several seconds, until RCU watchdog NMIs all vCPUs.
To fix this, use the start_assignment kvm_x86_ops callback to kick vcpus out of the halt loop, so the notification vector is properly reprogrammed to the wakeup vector.
arch/x86/kvm/vmx/posted_intr.c +7 -0
@@ -236,6 +236,13 @@ bool pi_has_pending_interrupt(struct kvm (pi_test_sn (pi_desc) && !pi_is_pir_empty (pi_desc)); } +void vmx_pi_start_assignment (struct kvm *kvm) +{ +if (!irq_remapping_cap (IRQ_POSTING_CAP )) +return ; + +kvm_make_all_cpus_request (kvm,KVM_REQ_UNBLOCK ); +}/* * pi_update_irte - set IRTE for Posted-Interrupts
arch/x86/kvm/vmx/posted_intr.h +1 -0
@@ -95,5 +95,6 @@ void __init pi_init_cpu(int cpu);bool pi_has_pending_interrupt (struct kvm_vcpu *vcpu);int pi_update_irte (struct kvm *kvm,unsigned int host_irq,uint32_t guest_irq,bool set); +void vmx_pi_start_assignment (struct kvm *kvm);#endif /* __KVM_X86_VMX_POSTED_INTR_H */
arch/x86/kvm/vmx/vmx.c +1 -0
@@ -7732,6 +7732,7 @@ static struct kvm_x86_ops vmx_x86_ops __ .nested_ops = &vmx_nested_ops, .update_pi_irte = pi_update_irte, + .start_assignment = vmx_pi_start_assignment, #ifdef CONFIG_X86_64 .set_hv_timer = vmx_set_hv_timer,
PATCH 1/3 KVM: x86: add start_assignment hook to kvm_x86_ops
In reply to Marcelo TosattiAdd a start_assignment hook to kvm_x86_ops, which is called when kvm_arch_start_assignment is done.
The hook is required to update the wakeup vector of a sleeping vCPU when a device is assigned to the guest.
arch/x86/include/asm/kvm_host.h +1 -0
@@ -1322,6 +1322,7 @@ struct kvm_x86_ops {int (*update_pi_irte)(struct kvm *kvm,unsigned int host_irq,uint32_t guest_irq,bool set); +void (*start_assignment)(struct kvm *kvm);void (*apicv_post_state_restore)(struct kvm_vcpu *vcpu);bool (*dy_apicv_has_pending_interrupt)(struct kvm_vcpu *vcpu);
arch/x86/kvm/x86.c +2 -1
@@ -11295,7 +11295,8 @@ bool kvm_arch_can_dequeue_async_page_prevoid kvm_arch_start_assignment (struct kvm *kvm) { -atomic_inc (&kvm->arch.assigned_device_count); +if (atomic_inc_return (&kvm->arch.assigned_device_count) ==1 ) +static_call_cond (kvm_x86_start_assignment)(kvm); }EXPORT_SYMBOL_GPL (kvm_arch_start_assignment);
arch/x86/include/asm/kvm-x86-ops.h +1 -0
@@ -99,6 +99,7 @@ KVM_X86_OP_NULL(post_block) KVM_X86_OP_NULL(vcpu_blocking) KVM_X86_OP_NULL(vcpu_unblocking) KVM_X86_OP_NULL(update_pi_irte) +KVM_X86_OP_NULL(start_assignment) KVM_X86_OP_NULL(apicv_post_state_restore) KVM_X86_OP_NULL(dy_apicv_has_pending_interrupt) KVM_X86_OP_NULL(set_hv_timer)
cover VMX: configure posted interrupt descriptor when assigning device (v5)
Configuration of the posted interrupt descriptor is incorrect when devices are hotplugged to the guest (and vcpus are halted).
See patch 3 for details.
---
v5: use KVM_REQ_UNBLOCK instead of kvm_check_block callback (Paolo / Peter Xu)
v4: remove NULL assignments from kvm_x86_ops (Peter Xu)
check for return value of ->start_assignment directly (Peter Xu)
v3: improved comments (Sean)
use kvm_vcpu_wake_up (Sean)
drop device_count from start_assignment function (Peter Xu)
v2: rather than using a potentially racy IPI (vs vcpu->cpu switches),
kick the vcpus when assigning a device and let the blocked per-CPU list manipulation happen locally at ->pre_block and ->post_block (Sean Christopherson).
PATCH 2/3 KVM: rename KVM_REQ_PENDING_TIMER to KVM_REQ_UNBLOCK
KVM_REQ_UNBLOCK will be used to exit a vcpu from
its inner vcpu halt emulation loop.
Rename KVM_REQ_PENDING_TIMER to KVM_REQ_UNBLOCK, switch PowerPC to arch specific request bit.
include/linux/kvm_host.h +1 -1
@@ -146,7 +146,7 @@ static inline bool is_error_page(struct*/ #define KVM_REQ_TLB_FLUSH (0 | KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP)#define KVM_REQ_MMU_RELOAD (1 | KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP) -#define KVM_REQ_PENDING_TIMER 2 +#define KVM_REQ_UNBLOCK 2#define KVM_REQ_UNHALT 3#define KVM_REQUEST_ARCH_BASE 8
virt/kvm/kvm_main.c +2 -0
@@ -2794,6 +2794,8 @@ static int kvm_vcpu_check_block(struct k goto out;if (signal_pending (current)) goto out; +if (kvm_check_request (KVM_REQ_UNBLOCK , vcpu)) + goto out; ret =0 ; out:
Documentation/virt/kvm/vcpu-requests.rst +3 -2
@@ -118,10 +118,11 @@ KVM_REQ_MMU_RELOAD necessary to inform each VCPU to completely refresh the tables. This request is used for that. -KVM_REQ_PENDING_TIMER +KVM_REQ_UNBLOCK This request may be made from a timer handler run on the host on behalf - of a VCPU. It informs the VCPU thread to inject a timer interrupt. + of a VCPU, or when device assignment is performed. It informs the VCPU to + exit the vcpu halt inner loop. KVM_REQ_UNHALT
arch/powerpc/include/asm/kvm_host.h +1 -0
@@ -51,6 +51,7 @@/* PPC-specific vcpu->requests bit members */ #define KVM_REQ_WATCHDOG KVM_ARCH_REQ(0)#define KVM_REQ_EPR_EXIT KVM_ARCH_REQ(1) +#define KVM_REQ_PENDING_TIMER KVM_ARCH_REQ(2)#include <linux/mmu_notifier.h>
arch/x86/kvm/lapic.c +1 -1
@@ -1657,7 +1657,7 @@ static void apic_timer_expired(struct kv }atomic_inc (&apic->lapic_timer.pending); -kvm_make_request (KVM_REQ_PENDING_TIMER , vcpu); +kvm_make_request (KVM_REQ_UNBLOCK , vcpu);if (from_timer_fn)kvm_vcpu_kick (vcpu); }
arch/x86/kvm/x86.c +1 -1
@@ -9300,7 +9300,7 @@ static int vcpu_run(struct kvm_vcpu *vcpif (r <=0 )break ; -kvm_clear_request (KVM_REQ_PENDING_TIMER , vcpu); +kvm_clear_request (KVM_REQ_UNBLOCK , vcpu);if (kvm_cpu_has_pending_timer (vcpu))kvm_inject_pending_timer_irqs (vcpu);
Re: [patch 2/3] KVM: rename KVM_REQ_PENDING_TIMER to KVM_REQ_UNBLOCK
Quoted 69 lines · Marcelo Tosatti
KVM_REQ_UNBLOCKwill be used to exit a vcpu from
its inner vcpu halt emulation loop.Rename
KVM_REQ_PENDING_TIMERtoKVM_REQ_UNBLOCK, switch PowerPC to arch specific request bit.Signed-off-by Marcelo Tosatti <[email protected]>Index:
kvm/include/linux/kvm_host.h
===================================================================include/linux/kvm_host.h +1
-1@@ -146,7 +146,7 @@ static inline bool is_error_page(struct*/ #define KVM_REQ_TLB_FLUSH (0 | KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP)#define KVM_REQ_MMU_RELOAD (1 | KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP) -#define KVM_REQ_PENDING_TIMER 2 +#define KVM_REQ_UNBLOCK 2#define KVM_REQ_UNHALT 3#define KVM_REQUEST_ARCH_BASE 8Index:
kvm/virt/kvm/kvm_main.c
===================================================================virt/kvm/kvm_main.c +2
-0@@ -2794,6 +2794,8 @@ static int kvm_vcpu_check_block(struct k goto out;if (signal_pending (current)) goto out; +if (kvm_check_request (KVM_REQ_UNBLOCK , vcpu)) + goto out; ret =0 ; out:Index:
kvm/Documentation/virt/kvm/vcpu-requests.rst
===================================================================Documentation/virt/kvm/vcpu-requests.rst +3
-2@@ -118,10 +118,11 @@ KVM_REQ_MMU_RELOAD necessary to inform each VCPU to completely refresh the tables. This request is used for that. -KVM_REQ_PENDING_TIMER +KVM_REQ_UNBLOCK This request may be made from a timer handler run on the host on behalf - of a VCPU. It informs the VCPU thread to inject a timer interrupt. + of a VCPU, or when device assignment is performed. It informs the VCPU to + exit the vcpu halt inner loop. KVM_REQ_UNHALTIndex:
kvm/arch/powerpc/include/asm/kvm_host.h
===================================================================arch/powerpc/include/asm/kvm_host.h +1
-0@@ -51,6 +51,7 @@/* PPC-specific vcpu->requests bit members */ #define KVM_REQ_WATCHDOG KVM_ARCH_REQ(0)#define KVM_REQ_EPR_EXIT KVM_ARCH_REQ(1) +#define KVM_REQ_PENDING_TIMER KVM_ARCH_REQ(2)#include <linux/mmu_notifier.h> Index:
kvm/arch/x86/kvm/lapic.c
===================================================================
arch/x86/kvm/lapic.c static void apic_timer_expired(struct kv · line 1663
Quoted 8 lines
arch/x86/kvm/lapic.c +1
-1@@ -1657,7 +1657,7 @@ static void apic_timer_expired(struct kv }atomic_inc (&apic->lapic_timer.pending); -kvm_make_request (KVM_REQ_PENDING_TIMER , vcpu); +kvm_make_request (KVM_REQ_UNBLOCK , vcpu);if (from_timer_fn)kvm_vcpu_kick (vcpu); }
Pure question on the existing code: why do we need kvm_make_request() for timer? As I see kvm_vcpu_check_block() already checks explicitly for timers:
static int kvm_vcpu_check_block(struct kvm_vcpu *vcpu)
{
...
if (kvm_cpu_has_pending_timer(vcpu))
goto out;
...
}for x86:
int kvm_cpu_has_pending_timer(struct kvm_vcpu *vcpu)
{
if (lapic_in_kernel(vcpu))
return apic_has_pending_timer(vcpu);
return 0;
}So wondering why we can drop the two references to KVM_REQ_PENDING_TIMER in x86 directly..
Quoted 15 lines
Index:
kvm/arch/x86/kvm/x86.c
===================================================================arch/x86/kvm/x86.c +1
-1@@ -9300,7 +9300,7 @@ static int vcpu_run(struct kvm_vcpu *vcpif (r <=0 )break ; -kvm_clear_request (KVM_REQ_PENDING_TIMER , vcpu); +kvm_clear_request (KVM_REQ_UNBLOCK , vcpu);if (kvm_cpu_has_pending_timer (vcpu))kvm_inject_pending_timer_irqs (vcpu);
Signature · 1 line
Peter Xu
Re: [patch 2/3] KVM: rename KVM_REQ_PENDING_TIMER to KVM_REQ_UNBLOCK
Quoted 100 lines · Peter Xu
Quoted 69 lines · Marcelo Tosatti
KVM_REQ_UNBLOCKwill be used to exit a vcpu from
its inner vcpu halt emulation loop.Rename
KVM_REQ_PENDING_TIMERtoKVM_REQ_UNBLOCK, switch PowerPC to arch specific request bit.Signed-off-by Marcelo Tosatti <[email protected]>Index:
kvm/include/linux/kvm_host.h
===================================================================include/linux/kvm_host.h +1
-1@@ -146,7 +146,7 @@ static inline bool is_error_page(struct*/ #define KVM_REQ_TLB_FLUSH (0 | KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP)#define KVM_REQ_MMU_RELOAD (1 | KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP) -#define KVM_REQ_PENDING_TIMER 2 +#define KVM_REQ_UNBLOCK 2#define KVM_REQ_UNHALT 3#define KVM_REQUEST_ARCH_BASE 8Index:
kvm/virt/kvm/kvm_main.c
===================================================================virt/kvm/kvm_main.c +2
-0@@ -2794,6 +2794,8 @@ static int kvm_vcpu_check_block(struct k goto out;if (signal_pending (current)) goto out; +if (kvm_check_request (KVM_REQ_UNBLOCK , vcpu)) + goto out; ret =0 ; out:Index:
kvm/Documentation/virt/kvm/vcpu-requests.rst
===================================================================Documentation/virt/kvm/vcpu-requests.rst +3
-2@@ -118,10 +118,11 @@ KVM_REQ_MMU_RELOAD necessary to inform each VCPU to completely refresh the tables. This request is used for that. -KVM_REQ_PENDING_TIMER +KVM_REQ_UNBLOCK This request may be made from a timer handler run on the host on behalf - of a VCPU. It informs the VCPU thread to inject a timer interrupt. + of a VCPU, or when device assignment is performed. It informs the VCPU to + exit the vcpu halt inner loop. KVM_REQ_UNHALTIndex:
kvm/arch/powerpc/include/asm/kvm_host.h
===================================================================arch/powerpc/include/asm/kvm_host.h +1
-0@@ -51,6 +51,7 @@/* PPC-specific vcpu->requests bit members */ #define KVM_REQ_WATCHDOG KVM_ARCH_REQ(0)#define KVM_REQ_EPR_EXIT KVM_ARCH_REQ(1) +#define KVM_REQ_PENDING_TIMER KVM_ARCH_REQ(2)#include <linux/mmu_notifier.h> Index:
kvm/arch/x86/kvm/lapic.c
===================================================================arch/x86/kvm/lapic.c static void apic_timer_expired(struct kv · line 1663
Quoted 8 lines
arch/x86/kvm/lapic.c +1
-1@@ -1657,7 +1657,7 @@ static void apic_timer_expired(struct kv }atomic_inc (&apic->lapic_timer.pending); -kvm_make_request (KVM_REQ_PENDING_TIMER , vcpu); +kvm_make_request (KVM_REQ_UNBLOCK , vcpu);if (from_timer_fn)kvm_vcpu_kick (vcpu); }Pure question on the existing code: why do we need
kvm_make_request()for timer? As I seekvm_vcpu_check_block()already checks explicitly for timers:static int
kvm_vcpu_check_block(struct kvm_vcpu *vcpu){ ... if (kvm_cpu_has_pending_timer(vcpu)) goto out; ... }for x86:
int
kvm_cpu_has_pending_timer(struct kvm_vcpu *vcpu){ if (lapic_in_kernel(vcpu)) return apic_has_pending_timer(vcpu); return 0; }
Quoted 2 lines
So wondering why we can drop the two references to
KVM_REQ_PENDING_TIMERin x86 directly..
See commit 06e05645661211b9eaadaf6344c335d2e80f0ba2
Peter Xu
Reviewed-byRe: [patch 2/3] KVM: rename KVM_REQ_PENDING_TIMER to KVM_REQ_UNBLOCK
Quoted 98 lines · Marcelo Tosatti
Quoted 97 lines · Peter Xu
Quoted 69 lines · Marcelo Tosatti
KVM_REQ_UNBLOCKwill be used to exit a vcpu from
its inner vcpu halt emulation loop.Rename
KVM_REQ_PENDING_TIMERtoKVM_REQ_UNBLOCK, switch PowerPC to arch specific request bit.Signed-off-by Marcelo Tosatti <[email protected]>Index:
kvm/include/linux/kvm_host.h
===================================================================include/linux/kvm_host.h +1
-1@@ -146,7 +146,7 @@ static inline bool is_error_page(struct*/ #define KVM_REQ_TLB_FLUSH (0 | KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP)#define KVM_REQ_MMU_RELOAD (1 | KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP) -#define KVM_REQ_PENDING_TIMER 2 +#define KVM_REQ_UNBLOCK 2#define KVM_REQ_UNHALT 3#define KVM_REQUEST_ARCH_BASE 8Index:
kvm/virt/kvm/kvm_main.c
===================================================================virt/kvm/kvm_main.c +2
-0@@ -2794,6 +2794,8 @@ static int kvm_vcpu_check_block(struct k goto out;if (signal_pending (current)) goto out; +if (kvm_check_request (KVM_REQ_UNBLOCK , vcpu)) + goto out; ret =0 ; out:Index:
kvm/Documentation/virt/kvm/vcpu-requests.rst
===================================================================Documentation/virt/kvm/vcpu-requests.rst +3
-2@@ -118,10 +118,11 @@ KVM_REQ_MMU_RELOAD necessary to inform each VCPU to completely refresh the tables. This request is used for that. -KVM_REQ_PENDING_TIMER +KVM_REQ_UNBLOCK This request may be made from a timer handler run on the host on behalf - of a VCPU. It informs the VCPU thread to inject a timer interrupt. + of a VCPU, or when device assignment is performed. It informs the VCPU to + exit the vcpu halt inner loop. KVM_REQ_UNHALTIndex:
kvm/arch/powerpc/include/asm/kvm_host.h
===================================================================arch/powerpc/include/asm/kvm_host.h +1
-0@@ -51,6 +51,7 @@/* PPC-specific vcpu->requests bit members */ #define KVM_REQ_WATCHDOG KVM_ARCH_REQ(0)#define KVM_REQ_EPR_EXIT KVM_ARCH_REQ(1) +#define KVM_REQ_PENDING_TIMER KVM_ARCH_REQ(2)#include <linux/mmu_notifier.h> Index:
kvm/arch/x86/kvm/lapic.c
===================================================================arch/x86/kvm/lapic.c static void apic_timer_expired(struct kv · line 1663
Quoted 8 lines
arch/x86/kvm/lapic.c +1
-1@@ -1657,7 +1657,7 @@ static void apic_timer_expired(struct kv }atomic_inc (&apic->lapic_timer.pending); -kvm_make_request (KVM_REQ_PENDING_TIMER , vcpu); +kvm_make_request (KVM_REQ_UNBLOCK , vcpu);if (from_timer_fn)kvm_vcpu_kick (vcpu); }Pure question on the existing code: why do we need
kvm_make_request()for timer? As I seekvm_vcpu_check_block()already checks explicitly for timers:static int
kvm_vcpu_check_block(struct kvm_vcpu *vcpu){ ... if (kvm_cpu_has_pending_timer(vcpu)) goto out; ... }for x86:
int
kvm_cpu_has_pending_timer(struct kvm_vcpu *vcpu){ if (lapic_in_kernel(vcpu)) return apic_has_pending_timer(vcpu);
Quoted 7 lines
Quoted 5 lines
return 0; }So wondering why we can drop the two references to
KVM_REQ_PENDING_TIMERin x86 directly..See commit
06e05645661211b9eaadaf6344c335d2e80f0ba2
I see, thanks Marcelo.
Then we might have checked twice on timer pending in kvm_vcpu_check_block() for x86, as we also checks KVM_REQ_UNBLOCK now. Didn't think further on how to make it better, e.g. simply dropping kvm_cpu_has_pending_timer() won't work since it seems to still be useful for non-x86..
Then this patch looks good to me:
Signature · 1 line
Peter Xu
Re: [patch 1/3] KVM: x86: add start_assignment hook to kvm_x86_ops
In reply to Marcelo TosattiQuoted 7 lines · Marcelo Tosatti
Add a
start_assignmenthook tokvm_x86_ops, which is called whenkvm_arch_start_assignmentis done.The hook is required to update the wakeup vector of a sleeping vCPU when a device is assigned to the guest.
Signed-off-by Marcelo Tosatti <[email protected]>
Signature · 1 line
Peter Xu
Re: [patch 3/3] KVM: VMX: update vcpu posted-interrupt descriptor when assigning device
In reply to Marcelo TosattiQuoted 42 lines · Marcelo Tosatti
For VMX, when a vcpu enters HLT emulation,
pi_post_blockwill:1) Add vcpu to per-cpu list of blocked vcpus.
2) Program the posted-interrupt descriptor "notification vector" to
POSTED_INTR_WAKEUP_VECTORWith interrupt remapping, an interrupt will set the PIR bit for the vector programmed for the device on the CPU, test-and-set the ON bit on the posted interrupt descriptor, and if the ON bit is clear generate an interrupt for the notification vector.
This way, the target CPU wakes upon a device interrupt and wakes up the target vcpu.
Problem is that
pi_post_blockonly programs the notification vector ifkvm_arch_has_assigned_device()is true. Its possible for the following to happen:1) vcpu V HLTs on pcpu P,
kvm_arch_has_assigned_deviceis false, notification vector is not programmed
2) device is assigned to VM
3) device interrupts vcpu V, sets ON bit
(notification vector not programmed, so pcpu P remains in idle)
4) vcpu 0 IPIs vcpu V (in guest), but since pi descriptor ON bit is set,kvm_vcpu_kickis skipped
5) vcpu 0 busy spins on vcpu V's response for several seconds, until RCU watchdog NMIs all vCPUs.To fix this, use the
start_assignmentkvm_x86_opscallback to kick vcpus out of the halt loop, so the notification vector is properly reprogrammed to the wakeup vector.Reported-by Pei Zhang <[email protected]>Signed-off-by Marcelo Tosatti <[email protected]>Index:
kvm/arch/x86/kvm/vmx/posted_intr.c
===================================================================
arch/x86/kvm/vmx/posted_intr.c bool pi_has_pending_interrupt(struct kvm · line 244
Quoted 9 lines
arch/x86/kvm/vmx/posted_intr.c +6
-0@@ -236,6 +236,13 @@ bool pi_has_pending_interrupt(struct kvm (pi_test_sn (pi_desc) && !pi_is_pir_empty (pi_desc)); } +void vmx_pi_start_assignment (struct kvm *kvm) +{ +if (!irq_remapping_cap (IRQ_POSTING_CAP )) +return ; + +kvm_make_all_cpus_request (kvm,KVM_REQ_UNBLOCK );
Shall we add a simple comment block explaining why we need this?
arch/x86/kvm/vmx/posted_intr.c bool pi_has_pending_interrupt(struct kvm · line 245
Quoted 1 line
arch/x86/kvm/vmx/posted_intr.c +1
-0+}
The patch itself looks right to me.
Thanks,
Signature · 1 line
Peter Xu
kernel test robot
build botRe: [patch 3/3] KVM: VMX: update vcpu posted-interrupt descriptor when assigning device
In reply to Marcelo TosattiThank you for the patch! Yet something to improve:
Hi Marcelo,
Thank you for the patch! Yet something to improve:
[auto build test ERROR on kvm/queue]
[also build test ERROR on vhost/linux-next v5.13-rc3 next-20210525] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]
url: https://github.com/0day-ci/linux/commits/Marcelo-Tosatti/VMX-configure-posted-interrupt-descriptor-when-assigning-device-v5/20210525-215604 base: https://git.kernel.org/pub/scm/virt/kvm/kvm.git queue config: x86_64-rhel-8.3 (attached as .config) compiler: gcc-9 (Debian 9.3.0-22) 9.3.0 reproduce (this is a W=1 build): # https://github.com/0day-ci/linux/commit/e515e68f330d3787af0952dcfb3e0fbf2d9b9f06 git remote add linux-review https://github.com/0day-ci/linux git fetch --no-tags linux-review Marcelo-Tosatti/VMX-configure-posted-interrupt-descriptor-when-assigning-device-v5/20210525-215604 git checkout e515e68f330d3787af0952dcfb3e0fbf2d9b9f06 # save the attached .config to linux build tree make W=1 ARCH=x86_64
If you fix the issue, kindly add following tag as appropriate
All errors (new ones prefixed by >>, old ones prefixed by <<):
>> ERROR: modpost: "kvm_make_all_cpus_request" [arch/x86/kvm/kvm-intel.ko] undefined!
---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org
PATCH v2 3/3 KVM: VMX: update vcpu posted-interrupt descriptor when assigning device
In reply to Peter XuFor VMX, when a vcpu enters HLT emulation, pi_post_block will:
1) Add vcpu to per-cpu list of blocked vcpus.
2) Program the posted-interrupt descriptor "notification vector" to POSTED_INTR_WAKEUP_VECTOR
With interrupt remapping, an interrupt will set the PIR bit for the vector programmed for the device on the CPU, test-and-set the ON bit on the posted interrupt descriptor, and if the ON bit is clear generate an interrupt for the notification vector.
This way, the target CPU wakes upon a device interrupt and wakes up the target vcpu.
Problem is that pi_post_block only programs the notification vector if kvm_arch_has_assigned_device() is true. Its possible for the following to happen:
1) vcpu V HLTs on pcpu P, kvm_arch_has_assigned_device is false, notification vector is not programmed
2) device is assigned to VM
3) device interrupts vcpu V, sets ON bit
(notification vector not programmed, so pcpu P remains in idle)
4) vcpu 0 IPIs vcpu V (in guest), but since pi descriptor ON bit is set, kvm_vcpu_kick is skipped
5) vcpu 0 busy spins on vcpu V's response for several seconds, until RCU watchdog NMIs all vCPUs.
To fix this, use the start_assignment kvm_x86_ops callback to kick vcpus out of the halt loop, so the notification vector is properly reprogrammed to the wakeup vector.
For build error:
---
v2: Add brief comment to vmx_pi_start_assignment (Peter Xu).
Export kvm_make_all_cpus_request (kernel test robot).
arch/x86/kvm/vmx/posted_intr.c +14 -0
@@ -238,6 +238,20 @@ bool pi_has_pending_interrupt(struct kvm/* +* Bail out of the block loop if the VM has an assigned +* device, but the blocking vCPU didn't reconfigure the +* PI.NV to the wakeup vector, i.e. the assigned device +* came along after the initial check in pi_pre_block(). +*/ +void vmx_pi_start_assignment (struct kvm *kvm) +{ +if (!irq_remapping_cap (IRQ_POSTING_CAP )) +return ; + +kvm_make_all_cpus_request (kvm,KVM_REQ_UNBLOCK ); +} + +/* * pi_update_irte - set IRTE for Posted-Interrupts * * @kvm: kvm
arch/x86/kvm/vmx/posted_intr.h +1 -0
@@ -95,5 +95,6 @@ void __init pi_init_cpu(int cpu);bool pi_has_pending_interrupt (struct kvm_vcpu *vcpu);int pi_update_irte (struct kvm *kvm,unsigned int host_irq,uint32_t guest_irq,bool set); +void vmx_pi_start_assignment (struct kvm *kvm);#endif /* __KVM_X86_VMX_POSTED_INTR_H */
arch/x86/kvm/vmx/vmx.c +1 -0
@@ -7732,6 +7732,7 @@ static struct kvm_x86_ops vmx_x86_ops __ .nested_ops = &vmx_nested_ops, .update_pi_irte = pi_update_irte, + .start_assignment = vmx_pi_start_assignment, #ifdef CONFIG_X86_64 .set_hv_timer = vmx_set_hv_timer,
virt/kvm/kvm_main.c +1 -0
@@ -307,6 +307,7 @@ bool kvm_make_all_cpus_request(struct kv {return kvm_make_all_cpus_request_except (kvm, req,NULL ); } +EXPORT_SYMBOL_GPL (kvm_make_all_cpus_request);#ifndef CONFIG_HAVE_KVM_ARCH_TLB_FLUSH_ALL void kvm_flush_remote_tlbs (struct kvm *kvm)
Paolo Bonzini
maintainerRe: [patch 2/3] KVM: rename KVM_REQ_PENDING_TIMER to KVM_REQ_UNBLOCK
In reply to Peter XuQuoted 3 lines · Peter Xu
Didn't think further on how to make it better, e.g. simply dropping
kvm_cpu_has_pending_timer()won't work since it seems to still be useful for non-x86..
It should be possible to use KVM_REQ_UNBLOCK in other architectures, and drop kvm_cpu_has_pending_timer().
Paolo
Paolo Bonzini
maintainerRe: [patch 2/3] KVM: rename KVM_REQ_PENDING_TIMER to KVM_REQ_UNBLOCK
In reply to Marcelo TosattiDocumentation/virt/kvm/vcpu-requests.rst KVM_REQ_MMU_RELOAD · line 125
Quoted 8 lines · Marcelo Tosatti
Documentation/virt/kvm/vcpu-requests.rst +3
-2-KVM_REQ_PENDING_TIMER +KVM_REQ_UNBLOCK This request may be made from a timer handler run on the host on behalf - of a VCPU. It informs the VCPU thread to inject a timer interrupt. + of a VCPU, or when device assignment is performed. It informs the VCPU to + exit the vcpu halt inner loop.
A small change:
This request informs the vCPU to exit kvm_vcpu_block. It is used for example from timer handlers that run on the host on behalf of a vCPU, or in order to update the interrupt routing and ensure that assigned devices will wake up the vCPU.
Paolo
Quoted 12 lines · Marcelo Tosatti
Configuration of the posted interrupt descriptor is incorrect when devices are hotplugged to the guest (and vcpus are halted).
See patch 3 for details.
---
v5: use
KVM_REQ_UNBLOCKinstead ofkvm_check_blockcallback (Paolo / Peter Xu)v4: remove NULL assignments from
kvm_x86_ops(Peter Xu)check for return value of ->start_assignment directly (Peter Xu)
Quoted 11 lines
v3: improved comments (Sean)
usekvm_vcpu_wake_up(Sean)
dropdevice_countfromstart_assignmentfunction (Peter Xu)v2: rather than using a potentially racy IPI (vs
vcpu->cpuswitches),
kick the vcpus when assigning a device and let the blocked per-CPU list manipulation happen locally at ->pre_block and ->post_block (Sean Christopherson).
Queued, thanks.
Paolo