Skip to contentSkip to conversation

KVM: x86: ioapic: Update remote_irr only after successful delivery

PATCH Superseded Hao Zhang 2 participants 8 messages Aug 11

Subsystems (4)

v4 of 5 Superseded Latest is v5 →

lore
Share & tools
lore

Hao Zhang

author
#0 lore raw

PATCH v4 1/4 KVM: x86: ioapic: Update remote_irr only after successful delivery

From: Hao Zhang <[email protected]>

The I/O APIC sets remote_irr for level-triggered interrupts that have been accepted by a local APIC and are waiting for an EOI.

ioapic_service() currently treats any non-zero return from kvm_irq_delivery_to_apic() as successful delivery. But kvm_irq_delivery_to_apic() returns -1 when no destination is found. Setting remote_irr for that failed delivery leaves the pin blocked forever waiting for an EOI that will never be generated.

Update remote_irr only when the delivery result is positive, i.e. when at least one local APIC accepted the interrupt.

Fixes 4925663a079c ("KVM: Report IRQ injection status to userspace.")
---
 arch/x86/kvm/ioapic.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/kvm/ioapic.c +1 -1
index 757667fb2bfa..24a7cc3b8b7e 100644
@@ -491,7 +491,7 @@ static int ioapic_service(struct kvm_ioapic *ioapic, int irq, bool line_status)
 	} else
 		ret = kvm_irq_delivery_to_apic(ioapic->kvm, NULL, &irqe);
 
-	if (ret && irqe.trig_mode == IOAPIC_LEVEL_TRIG)
+	if (ret > 0 && irqe.trig_mode == IOAPIC_LEVEL_TRIG)
 		entry->fields.remote_irr = 1;
 
 	return ret;

base-commit: c21bb4193868a8de71fc4693fa741e195fdf5d86

Signature · 1 line
2.15.0
author
#1 lore raw

PATCH v4 2/4 KVM: selftests: Verify failed IOAPIC delivery preserves state

From: Hao Zhang <[email protected]>

Add an x86 selftest for the in-kernel I/O APIC state that is exposed through KVM_GET_IRQCHIP.

Create a VM with an in-kernel irqchip and no vCPUs, route an unmasked level-triggered I/O APIC pin to a non-existent APIC ID, and verify that failed delivery does not set remote_irr.

---
 tools/testing/selftests/kvm/Makefile.kvm           |  1 +
 .../testing/selftests/kvm/x86/ioapic_state_test.c  | 92 ++++++++++++++++++++++
 2 files changed, 93 insertions(+)
 create mode 100644 tools/testing/selftests/kvm/x86/ioapic_state_test.c
tools/testing/selftests/kvm/Makefile.kvm +1 -0
index 6fc34e9bf8e1..89e3f82e75b0 100644
@@ -86,6 +86,7 @@ TEST_GEN_PROGS_x86 += x86/hyperv_features
 TEST_GEN_PROGS_x86 += x86/hyperv_ipi
 TEST_GEN_PROGS_x86 += x86/hyperv_svm_test
 TEST_GEN_PROGS_x86 += x86/hyperv_tlb_flush
+TEST_GEN_PROGS_x86 += x86/ioapic_state_test
 TEST_GEN_PROGS_x86 += x86/kvm_clock_test
 TEST_GEN_PROGS_x86 += x86/kvm_pv_test
 TEST_GEN_PROGS_x86 += x86/kvm_buslock_test
tools/testing/selftests/kvm/x86/ioapic_state_test.c new file +92 -0

92 lines not on this page · raw message

Signature · 1 line
2.15.0
author
#2 lore raw

PATCH v4 3/4 KVM: x86: ioapic: Preserve irr_delivered for duplicate edge interrupts

In reply to Hao Zhang

From: Hao Zhang <[email protected]>

For edge-triggered interrupts, ioapic_set_irq() clears irr_delivered before checking whether the new edge is a duplicate. A duplicate edge is then coalesced without being serviced, leaving the interrupt visible in KVM_GET_IRQCHIP state even though the original interrupt was already delivered.

Move the state update after the duplicate-edge check so duplicate edges do not make a delivered interrupt appear pending during VM migration.

Add a selftest to verify that a duplicate edge interrupt does not make an already-delivered interrupt visible in the saved IRR state.

Fixes 5bda6eed2e36 ("KVM: ioapic: Record edge-triggered interrupts delivery status")
---
 arch/x86/kvm/ioapic.c                              |  2 +-
 .../testing/selftests/kvm/x86/ioapic_state_test.c  | 47 ++++++++++++++++++++++
 2 files changed, 48 insertions(+), 1 deletion(-)
arch/x86/kvm/ioapic.c +1 -1
index 24a7cc3b8b7e..676effd674f9 100644
@@ -230,11 +230,11 @@ static int ioapic_set_irq(struct kvm_ioapic *ioapic, unsigned int irq,
 	old_irr = ioapic->irr;
 	ioapic->irr |= mask;
 	if (edge) {
-		ioapic->irr_delivered &= ~mask;
 		if (old_irr == ioapic->irr) {
 			ret = 0;
 			goto out;
 		}
+		ioapic->irr_delivered &= ~mask;
 	}
 
 	ret = ioapic_service(ioapic, irq, line_status);
tools/testing/selftests/kvm/x86/ioapic_state_test.c +47 -0

69 lines not on this page · raw message

Signature · 1 line
2.15.0
#3 lore raw

Re: [PATCH v4 3/4] KVM: x86: ioapic: Preserve irr_delivered for duplicate edge interrupts

Quoted 10 lines · Hao Zhang

From: Hao Zhang <[email protected]>

For edge-triggered interrupts, ioapic_set_irq() clears irr_delivered before checking whether the new edge is a duplicate. A duplicate edge is then coalesced without being serviced, leaving the interrupt visible in KVM_GET_IRQCHIP state even though the original interrupt was already delivered.

Move the state update after the duplicate-edge check so duplicate edges do not make a delivered interrupt appear pending during VM migration.

ioapic_set_irq() returns 0 for coalesced edge triggered IRQ, meaning it is considered not delivered. I don't see a problem if it is marked as pending in ioapic->irr during KVM_GET_IRQCHIP for migration? It seems the result is the coalesced IRQ will get re-injected, but what's wrong of this?

But I am not sure on this, so will leave to maintainers.

Quoted 3 lines

Add a selftest to verify that a duplicate edge interrupt does not make an already-delivered interrupt visible in the saved IRR state.

IMO it's better to split kselftest changes to a separate patch out of KVM changes.

Quoted 2 lines
Fixes 5bda6eed2e36 ("KVM: ioapic: Record edge-triggered interrupts delivery status")

When this commit was merged, AFAICT the then ioapic_set_irq() didn't support coalesced edge triggered IRQ at all, so technically this one shouldn't be the one to blame.

author
#4 lore raw

PATCH v4 4/4 KVM: x86: ioapic: Clear irr_delivered on level-triggered RTE

In reply to Hao Zhang

From: Hao Zhang <[email protected]>

irr_delivered hides delivered edge interrupts from KVM_GET_IRQCHIP. The interrupt remains in IRR until the guest deasserts the line.

The guest can reprogram an IOAPIC RTE from edge to level via MMIO. If an edge interrupt was delivered before the trigger mode change, a stale irr_delivered bit can incorrectly mask a subsequent pending level interrupt from KVM_GET_IRQCHIP.

That can cause userspace to migrate the VM without a pending level interrupt that is still present in KVM's in-kernel IOAPIC.

Clear irr_delivered when the RTE is configured for level-triggered interrupts, and add a regression test that reprograms the IOAPIC via guest MMIO to verify that the pending IRR remains visible.

Fixes 5bda6eed2e36 ("KVM: ioapic: Record edge-triggered interrupts delivery status")
---
 arch/x86/kvm/ioapic.c                              | 11 +++-
 .../testing/selftests/kvm/x86/ioapic_state_test.c  | 67 +++++++++++++++++++---
 2 files changed, 69 insertions(+), 9 deletions(-)
arch/x86/kvm/ioapic.c +10 -1
index 676effd674f9..81db025d2ac9 100644
@@ -377,8 +377,17 @@ static void ioapic_write_indirect(struct kvm_ioapic *ioapic, u32 val)
 		 * as edge-triggered. This behavior is used to simulate an
 		 * explicit EOI on IOAPICs that don't have the EOI register.
 		 */
-		if (e->fields.trig_mode == IOAPIC_EDGE_TRIG)
+		if (e->fields.trig_mode == IOAPIC_EDGE_TRIG) {
 			e->fields.remote_irr = 0;
+		} else {
+			/*
+			 * irr_delivered tracks edge-triggered interrupts that
+			 * have already been delivered.  Clear stale edge-triggered
+			 * delivery state when the entry is configured for
+			 * level-triggered interrupts.
+			 */
+			ioapic->irr_delivered &= ~(1 << index);
+		}
 
 		mask_after = e->fields.mask;
 		if (mask_before != mask_after)
tools/testing/selftests/kvm/x86/ioapic_state_test.c +59 -8

106 lines not on this page · raw message

Signature · 1 line
2.15.0
#5 lore raw

Re: [PATCH v4 4/4] KVM: x86: ioapic: Clear irr_delivered on level-triggered RTE

Quoted 13 lines · Hao Zhang

From: Hao Zhang <[email protected]>

irr_delivered hides delivered edge interrupts from KVM_GET_IRQCHIP. The interrupt remains in IRR until the guest deasserts the line.

The guest can reprogram an IOAPIC RTE from edge to level via MMIO. If an edge interrupt was delivered before the trigger mode change, a stale irr_delivered bit can incorrectly mask a subsequent pending level interrupt from KVM_GET_IRQCHIP.

That can cause userspace to migrate the VM without a pending level interrupt that is still present in KVM's in-kernel IOAPIC.

[...]

Quoted 8 lines
arch/x86/kvm/ioapic.c +1 -1
@@ -377,8 +377,17 @@ static void ioapic_write_indirect(struct kvm_ioapic *ioapic, u32 val)
 		 * as edge-triggered. This behavior is used to simulate an
 		 * explicit EOI on IOAPICs that don't have the EOI register.
 		 */
-		if (e->fields.trig_mode == IOAPIC_EDGE_TRIG)
+		if (e->fields.trig_mode == IOAPIC_EDGE_TRIG) {

arch/x86/kvm/ioapic.c static void ioapic_write_indirect(struct kvm_ioapic *ioapic, u32 val) · line 390

Quoted 10 lines
arch/x86/kvm/ioapic.c +9 -0
 			e->fields.remote_irr = 0;
+		} else {
+			/*
+			 * irr_delivered tracks edge-triggered interrupts that
+			 * have already been delivered.  Clear stale edge-triggered
+			 * delivery state when the entry is configured for
+			 * level-triggered interrupts.
+			 */
+			ioapic->irr_delivered &= ~(1 << index);
+		}

To be honest I am not sure whether we should consider this case, but IIUC this code change doesn't seem to have any harm, so will leave to others.

tools/testing/selftests/kvm/x86/ioapic_state_test.c

Quoted 7 lines
arch/x86/kvm/ioapic.c +0 -0
 		mask_after = e->fields.mask;
 		if (mask_before != mask_after)
tools/testing/selftests/kvm/x86/ioapic_state_test.c +0 -0
index da0276a1542f..f024f4f01571 100644

Again, I think you should split selftest code change as a separate patch.

And it's better to have a cover letter for such series with multiple patches.

#6 lore raw
In reply to Hao Zhang
Quoted 11 lines · Hao Zhang

From: Hao Zhang <[email protected]>

The I/O APIC sets remote_irr for level-triggered interrupts that have been accepted by a local APIC and are waiting for an EOI.

ioapic_service() currently treats any non-zero return from kvm_irq_delivery_to_apic() as successful delivery. But kvm_irq_delivery_to_apic() returns -1 when no destination is found. Setting remote_irr for that failed delivery leaves the pin blocked forever waiting for an EOI that will never be generated.

Quoted 4 lines

Update remote_irr only when the delivery result is positive, i.e. when at least one local APIC accepted the interrupt.

Fixes 4925663a079c ("KVM: Report IRQ injection status to userspace.")

The code LGTM, but I am not sure this is the right commit for the Fixes tag.

Looking at the relevant diff:

+static int ioapic_service(struct kvm_ioapic *ioapic, unsigned int idx)

 {
        union ioapic_redir_entry *pent;

+ int injected = -1;

        pent = &ioapic->redirtbl[idx];
 
        if (!pent->fields.mask) {
-               int injected = ioapic_deliver(ioapic, idx);
+               injected = ioapic_deliver(ioapic, idx);
                if (injected && pent->fields.trig_mode == IOAPIC_LEVEL_TRIG)
                        pent->fields.remote_irr = 1;
        }

This commit didn't change any logic related to remote_irr, since it didn't touch ioapic_deliver() at all.

The one that is close seems to be

ff4b9df877b30 ("KVM: IOAPIC: only set remote_irr if interrupt was injected")

since it actually touched the logic of when to set remote_irr:

@@ -89,8 +89,8 @@ static void ioapic_service(struct kvm_ioapic *ioapic, unsigned int idx)

        pent = &ioapic->redirtbl[idx];
 
        if (!pent->fields.mask) {
-               ioapic_deliver(ioapic, idx);
-               if (pent->fields.trig_mode == IOAPIC_LEVEL_TRIG)
+               int injected = ioapic_deliver(ioapic, idx);
+               if (injected && pent->fields.trig_mode == IOAPIC_LEVEL_TRIG)
                        pent->fields.remote_irr = 1;
        }

But looking closer, at that time the ioapic_deliver() could only return 0 or 1 (or >1 I didn't check), i.e., it couldn't return -1 at all.

In fact, even when commit 4925663a079c was merged the then ioapic_deliver() couldn't return -1.

So AFAICT both of the above two commits are not the right one.

I haven't found which one to blame, but it seems at some point the function kvm_irq_delivery_to_apic() was introduced and at some other point it was changed to be able to return -1. But I didn't dig further.

Reviewed-by
#7 lore raw
Quoted 35 lines · Kai
Quoted 11 lines · Hao Zhang

From: Hao Zhang <[email protected]>

The I/O APIC sets remote_irr for level-triggered interrupts that have been accepted by a local APIC and are waiting for an EOI.

ioapic_service() currently treats any non-zero return from kvm_irq_delivery_to_apic() as successful delivery. But kvm_irq_delivery_to_apic() returns -1 when no destination is found. Setting remote_irr for that failed delivery leaves the pin blocked forever waiting for an EOI that will never be generated.

Quoted 4 lines

Update remote_irr only when the delivery result is positive, i.e. when at least one local APIC accepted the interrupt.

Fixes 4925663a079c ("KVM: Report IRQ injection status to userspace.")

The code LGTM, but I am not sure this is the right commit for the Fixes tag.

Looking at the relevant diff:

(quoted diff) +3 -1
+static int ioapic_service(struct kvm_ioapic *ioapic, unsigned int idx)
 {
        union ioapic_redir_entry *pent;
+       int injected = -1;
 
        pent = &ioapic->redirtbl[idx];
 
        if (!pent->fields.mask) {
-               int injected = ioapic_deliver(ioapic, idx);
+               injected = ioapic_deliver(ioapic, idx);
                if (injected && pent->fields.trig_mode == IOAPIC_LEVEL_TRIG)
                        pent->fields.remote_irr = 1;
        }
Quoted 3 lines

This commit didn't change any logic related to remote_irr, since it didn't touch ioapic_deliver() at all.

I asked the AI to double confirm what I was reading last night, and indeed I got mixed up a bit among multiple commits in history.

Firstly, the above "it didn't touch ioapic_deliver() at all" isn't true. It indeed introduced 'r = -1' initializer at the beginning of the function. While I was right for "no target on destination" it returns 0 (but not -1), AI also pointed out there are other cases such as invalid delivery_mode which can cause ioapic_deliver() to return -1.

FWIW, AI also found it was commit

a53c17d21c46 ("KVM: ioapic/msi interrupt delivery consolidation")

which made ioapic_deliver() return -1 when "no target on destination":

        if (find_first_bit(deliver_bitmask, KVM_MAX_VCPUS) >= KVM_MAX_VCPUS) {
                ioapic_debug("no target on destination\n");
  -             return 0;
  +             return r;
        }

So I guess you are right, 4925663a079c is the right one to blame, but not due to "no target on destination" case.

Anyway, IMHO we might also want to put such history digging into the changelog to justify the Fixes tag, so people can easily see the reason behind it.

That being said: