Skip to contentSkip to conversation

KVM: x86: Sync the pending Posted-Interrupts

PATCH Superseded Luwei Kang 2 participants 4 messages Jan 31, 2019 – Feb 4, 2019

Subsystems (3)

v3 of 4 Superseded Latest is v4 →

lore
Share & tools
lore

Luwei Kang

author
#0 lore raw

PATCH v3 1/1 KVM: x86: Sync the pending Posted-Interrupts

Some Posted-Interrupts from passthrough devices may be lost or overwritten when the vCPU is in runnable state.

The SN (Suppress Notification) of PID (Posted Interrupt Descriptor) will be set when the vCPU is preempted (vCPU in KVM_MP_STATE_RUNNABLE state but not running on physical CPU). If a posted interrupt coming at this time, the irq remmaping facility will set the bit of PIR (Posted Interrupt Requests) without ON (Outstanding Notification). So this interrupt can't be sync to APIC virtualization register and will not be handled by Guest because ON is zero.

---
 arch/x86/kvm/vmx/vmx.c | 5 +++++
 arch/x86/kvm/x86.c     | 2 +-
 2 files changed, 6 insertions(+), 1 deletion(-)
arch/x86/kvm/vmx/vmx.c +5 -0
index 4341175..8ed9634 100644
@@ -1221,6 +1221,11 @@ static void vmx_vcpu_pi_load(struct kvm_vcpu *vcpu, int cpu)
 		new.sn = 0;
 	} while (cmpxchg64(&pi_desc->control, old.control,
 			   new.control) != old.control);
+
+	smp_mb__after_atomic();
+
+	if (!bitmap_empty((unsigned long *)pi_desc->pir, NR_VECTORS))
+		pi_test_and_set_on(pi_desc);
 }
 
 /*
arch/x86/kvm/x86.c +1 -1
index 3d27206..5bcf2c4 100644
@@ -7794,7 +7794,7 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)
 	 * 1) We should set ->mode before checking ->requests.  Please see
 	 * the comment in kvm_vcpu_exiting_guest_mode().
 	 *
-	 * 2) For APICv, we should set ->mode before checking PIR.ON.  This
+	 * 2) For APICv, we should set ->mode before checking PID.PIR. This
 	 * pairs with the memory barrier implicit in pi_test_and_set_on
 	 * (see vmx_deliver_posted_interrupt).
 	 *
Signature · 1 line
1.8.3.1
maintainer
#1 lore raw
Quoted 22 lines · Luwei Kang

Some Posted-Interrupts from passthrough devices may be lost or overwritten when the vCPU is in runnable state.

The SN (Suppress Notification) of PID (Posted Interrupt Descriptor) will be set when the vCPU is preempted (vCPU in KVM_MP_STATE_RUNNABLE state but not running on physical CPU). If a posted interrupt coming at this time, the irq remmaping facility will set the bit of PIR (Posted Interrupt Requests) without ON (Outstanding Notification). So this interrupt can't be sync to APIC virtualization register and will not be handled by Guest because ON is zero.

---
 arch/x86/kvm/vmx/vmx.c | 5 +++++
 arch/x86/kvm/x86.c     | 2 +-
 2 files changed, 6 insertions(+), 1 deletion(-)
arch/x86/kvm/vmx/vmx.c +0 -0
index 4341175..8ed9634 100644

arch/x86/kvm/vmx/vmx.c static void vmx_vcpu_pi_load(struct kvm_vcpu *vcpu, int cpu) · line 1224

Quoted 4 lines
arch/x86/kvm/vmx/vmx.c +1 -0
@@ -1221,6 +1221,11 @@ static void vmx_vcpu_pi_load(struct kvm_vcpu *vcpu, int cpu)
 		new.sn = 0;
 	} while (cmpxchg64(&pi_desc->control, old.control,
 			   new.control) != old.control);
+

/*
* Clear SN before reading the bitmap. The VT-d firmware
* writes the bitmap and reads SN atomically (5.2.3 in the
* spec), so it doesn't really have a memory barrier that
* pairs with this, but we cannot do that and we need one. */

arch/x86/kvm/vmx/vmx.c static void vmx_vcpu_pi_load(struct kvm_vcpu *vcpu, int cpu) · line 1228

Quoted 4 lines
arch/x86/kvm/vmx/vmx.c +4 -0
+	smp_mb__after_atomic();
+
+	if (!bitmap_empty((unsigned long *)pi_desc->pir, NR_VECTORS))
+		pi_test_and_set_on(pi_desc);

You can add pi_set_on for use here. The fast path with pi_clear_sn should also be removed.

arch/x86/kvm/x86.c static int vcpu_enter_guest(struct kvm_vcpu *vcpu) · line 7797

Quoted 13 lines
arch/x86/kvm/vmx/vmx.c +0 -0
 }
 
 /*
arch/x86/kvm/x86.c +1 -1
index 3d27206..5bcf2c4 100644
@@ -7794,7 +7794,7 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)
 	 * 1) We should set ->mode before checking ->requests.  Please see
 	 * the comment in kvm_vcpu_exiting_guest_mode().
 	 *
-	 * 2) For APICv, we should set ->mode before checking PIR.ON.  This
+	 * 2) For APICv, we should set ->mode before checking PID.PIR. This

This should be PID.ON.

Paolo

Quoted 4 lines
arch/x86/kvm/x86.c +0 -0
 	 * pairs with the memory barrier implicit in pi_test_and_set_on
 	 * (see vmx_deliver_posted_interrupt).
 	 *
#2 lore raw
Quoted 36 lines
Quoted 5 lines

Some Posted-Interrupts from passthrough devices may be lost or overwritten when the vCPU is in runnable state.

The SN (Suppress Notification) of PID (Posted Interrupt Descriptor) will be set when the vCPU is preempted (vCPU in

KVM_MP_STATE_RUNNABLE

arch/x86/kvm/vmx/vmx.c static void vmx_vcpu_pi_load(struct kvm_vcpu

Quoted 17 lines

state but not running on physical CPU). If a posted interrupt coming at this time, the irq remmaping facility will set the bit of PIR (Posted Interrupt Requests) without ON (Outstanding Notification). So this interrupt can't be sync to APIC virtualization register and will not be handled by Guest because ON is zero.

---
 arch/x86/kvm/vmx/vmx.c | 5 +++++
 arch/x86/kvm/x86.c     | 2 +-
 2 files changed, 6 insertions(+), 1 deletion(-)
arch/x86/kvm/vmx/vmx.c index renamed from arch/x86/kvm/vmx/vmx.c +0 -0

4341175..8ed9634 100644

arch/x86/kvm/vmx/vmx.c +0 -0
@@ -1221,6 +1221,11 @@ static void vmx_vcpu_pi_load(struct kvm_vcpu

*vcpu, int cpu)

arch/x86/kvm/vmx/vmx.c static void vmx_vcpu_pi_load(struct kvm_vcpu · line 1224

Quoted 4 lines
arch/x86/kvm/vmx/vmx.c +1 -0
 		new.sn = 0;
 	} while (cmpxchg64(&pi_desc->control, old.control,
 			   new.control) != old.control);
+

/*
* Clear SN before reading the bitmap. The VT-d firmware
* writes the bitmap and reads SN atomically (5.2.3 in the
* spec), so it doesn't really have a memory barrier that
* pairs with this, but we cannot do that and we need one. */

Quoted 7 lines

arch/x86/kvm/vmx/vmx.c static void vmx_vcpu_pi_load(struct kvm_vcpu · line 1228

Quoted 4 lines
arch/x86/kvm/vmx/vmx.c +4 -0
+	smp_mb__after_atomic();
+
+	if (!bitmap_empty((unsigned long *)pi_desc->pir, NR_VECTORS))
+		pi_test_and_set_on(pi_desc);

You can add pi_set_on for use here. The fast path with pi_clear_sn should also be removed.

Do you mean remove the blow code in vmx_vcpu_pi_load() function to make the ON can be set if PIR is not zero?

arch/x86/kvm/vmx/vmx.c +0 -14
@@ -1192,21 +1192,6 @@ static void vmx_vcpu_pi_load(struct kvm_vcpu *vcpu, int cpu)
        if (!pi_test_sn(pi_desc) && vcpu->cpu == cpu)
                return;
 
-       /*
-        * First handle the simple case where no cmpxchg is necessary; just
-        * allow posting non-urgent interrupts.
-        *
-        * If the 'nv' field is POSTED_INTR_WAKEUP_VECTOR, do not change
-        * PI.NDST: pi_post_block will do it for us and the wakeup_handler
-        * expects the VCPU to be on the blocked_vcpu_list that matches
-        * PI.NDST.
-        */
-       if (pi_desc->nv == POSTED_INTR_WAKEUP_VECTOR ||
-           vcpu->cpu == cpu) {
-               pi_clear_sn(pi_desc);
-               return;
-       }
 

Thanks,
Luwei Kang

maintainer
#3 lore raw
Quoted 13 lines · Luwei

Do you mean remove the blow code in vmx_vcpu_pi_load() function to make the ON can be set if PIR is not zero?

arch/x86/kvm/vmx/vmx.c +0 -4
@@ -1192,21 +1192,6 @@ static void vmx_vcpu_pi_load(struct kvm_vcpu *vcpu, int cpu)
        if (!pi_test_sn(pi_desc) && vcpu->cpu == cpu)
                return;
 
-       /*
-        * First handle the simple case where no cmpxchg is necessary; just
-        * allow posting non-urgent interrupts.
-        *

arch/x86/kvm/vmx/vmx.c static void vmx_vcpu_pi_load(struct kvm_vcpu *vcpu, int cpu) · line 1208

Quoted 10 lines
arch/x86/kvm/vmx/vmx.c +0 -10
-        * If the 'nv' field is POSTED_INTR_WAKEUP_VECTOR, do not change
-        * PI.NDST: pi_post_block will do it for us and the wakeup_handler
-        * expects the VCPU to be on the blocked_vcpu_list that matches
-        * PI.NDST.
-        */
-       if (pi_desc->nv == POSTED_INTR_WAKEUP_VECTOR ||
-           vcpu->cpu == cpu) {
-               pi_clear_sn(pi_desc);
-               return;
-       }

Yes, exactly.

Paolo