KVM: x86: ioapic: Update remote_irr only after successful delivery
PATCH v4 1/4 KVM: x86: ioapic: Update remote_irr only after successful delivery
From: Hao Zhang <[email protected]>
The I/O APIC sets remote_irr for level-triggered interrupts that have been accepted by a local APIC and are waiting for an EOI.
ioapic_service() currently treats any non-zero return from kvm_irq_delivery_to_apic() as successful delivery. But kvm_irq_delivery_to_apic() returns -1 when no destination is found. Setting remote_irr for that failed delivery leaves the pin blocked forever waiting for an EOI that will never be generated.
Update remote_irr only when the delivery result is positive, i.e. when at least one local APIC accepted the interrupt.
--- arch/x86/kvm/ioapic.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/kvm/ioapic.c +1 -1
@@ -491,7 +491,7 @@ static int ioapic_service(struct kvm_ioapic *ioapic, int irq, bool line_status) }else ret =kvm_irq_delivery_to_apic (ioapic->kvm,NULL , &irqe); -if (ret && irqe.trig_mode ==IOAPIC_LEVEL_TRIG ) +if (ret >0 && irqe.trig_mode ==IOAPIC_LEVEL_TRIG ) entry->fields.remote_irr =1 ;return ret;
base-commit: c21bb4193868a8de71fc4693fa741e195fdf5d86
Signature · 1 line
2.15.0
PATCH v4 2/4 KVM: selftests: Verify failed IOAPIC delivery preserves state
From: Hao Zhang <[email protected]>
Add an x86 selftest for the in-kernel I/O APIC state that is exposed through KVM_GET_IRQCHIP.
Create a VM with an in-kernel irqchip and no vCPUs, route an unmasked level-triggered I/O APIC pin to a non-existent APIC ID, and verify that failed delivery does not set remote_irr.
--- tools/testing/selftests/kvm/Makefile.kvm | 1 + .../testing/selftests/kvm/x86/ioapic_state_test.c | 92 ++++++++++++++++++++++ 2 files changed, 93 insertions(+) create mode 100644 tools/testing/selftests/kvm/x86/ioapic_state_test.c
tools/testing/selftests/kvm/Makefile.kvm +1 -0
@@ -86,6 +86,7 @@ TEST_GEN_PROGS_x86 += x86/hyperv_features TEST_GEN_PROGS_x86 += x86/hyperv_ipi TEST_GEN_PROGS_x86 += x86/hyperv_svm_test TEST_GEN_PROGS_x86 += x86/hyperv_tlb_flush +TEST_GEN_PROGS_x86 += x86/ioapic_state_test TEST_GEN_PROGS_x86 += x86/kvm_clock_test TEST_GEN_PROGS_x86 += x86/kvm_pv_test TEST_GEN_PROGS_x86 += x86/kvm_buslock_test
tools/testing/selftests/kvm/x86/ioapic_state_test.c new file +92 -0
92 lines not on this page · raw message
Signature · 1 line
2.15.0
PATCH v4 3/4 KVM: x86: ioapic: Preserve irr_delivered for duplicate edge interrupts
In reply to Hao ZhangFrom: Hao Zhang <[email protected]>
For edge-triggered interrupts, ioapic_set_irq() clears irr_delivered before checking whether the new edge is a duplicate. A duplicate edge is then coalesced without being serviced, leaving the interrupt visible in KVM_GET_IRQCHIP state even though the original interrupt was already delivered.
Move the state update after the duplicate-edge check so duplicate edges do not make a delivered interrupt appear pending during VM migration.
Add a selftest to verify that a duplicate edge interrupt does not make an already-delivered interrupt visible in the saved IRR state.
--- arch/x86/kvm/ioapic.c | 2 +- .../testing/selftests/kvm/x86/ioapic_state_test.c | 47 ++++++++++++++++++++++ 2 files changed, 48 insertions(+), 1 deletion(-)
arch/x86/kvm/ioapic.c +1 -1
@@ -230,11 +230,11 @@ static int ioapic_set_irq(struct kvm_ioapic *ioapic, unsigned int irq, old_irr = ioapic->irr; ioapic->irr |= mask;if (edge) { - ioapic->irr_delivered &= ~mask;if (old_irr == ioapic->irr) { ret =0 ; goto out; } + ioapic->irr_delivered &= ~mask; } ret =ioapic_service (ioapic, irq, line_status);
tools/testing/selftests/kvm/x86/ioapic_state_test.c +47 -0
69 lines not on this page · raw message
Signature · 1 line
2.15.0
Re: [PATCH v4 3/4] KVM: x86: ioapic: Preserve irr_delivered for duplicate edge interrupts
Quoted 10 lines · Hao Zhang
From: Hao Zhang <[email protected]>
For edge-triggered interrupts,
ioapic_set_irq()clearsirr_deliveredbefore checking whether the new edge is a duplicate. A duplicate edge is then coalesced without being serviced, leaving the interrupt visible inKVM_GET_IRQCHIPstate even though the original interrupt was already delivered.Move the state update after the duplicate-edge check so duplicate edges do not make a delivered interrupt appear pending during VM migration.
ioapic_set_irq() returns 0 for coalesced edge triggered IRQ, meaning it is considered not delivered. I don't see a problem if it is marked as pending in ioapic->irr during KVM_GET_IRQCHIP for migration? It seems the result is the coalesced IRQ will get re-injected, but what's wrong of this?
But I am not sure on this, so will leave to maintainers.
Quoted 3 lines
Add a selftest to verify that a duplicate edge interrupt does not make an already-delivered interrupt visible in the saved IRR state.
IMO it's better to split kselftest changes to a separate patch out of KVM changes.
Quoted 2 lines
Fixes 5bda6eed2e36 ("KVM: ioapic: Record edge-triggered interrupts delivery status")
When this commit was merged, AFAICT the then ioapic_set_irq() didn't support coalesced edge triggered IRQ at all, so technically this one shouldn't be the one to blame.
PATCH v4 4/4 KVM: x86: ioapic: Clear irr_delivered on level-triggered RTE
In reply to Hao ZhangFrom: Hao Zhang <[email protected]>
irr_delivered hides delivered edge interrupts from KVM_GET_IRQCHIP. The interrupt remains in IRR until the guest deasserts the line.
The guest can reprogram an IOAPIC RTE from edge to level via MMIO. If an edge interrupt was delivered before the trigger mode change, a stale irr_delivered bit can incorrectly mask a subsequent pending level interrupt from KVM_GET_IRQCHIP.
That can cause userspace to migrate the VM without a pending level interrupt that is still present in KVM's in-kernel IOAPIC.
Clear irr_delivered when the RTE is configured for level-triggered interrupts, and add a regression test that reprograms the IOAPIC via guest MMIO to verify that the pending IRR remains visible.
--- arch/x86/kvm/ioapic.c | 11 +++- .../testing/selftests/kvm/x86/ioapic_state_test.c | 67 +++++++++++++++++++--- 2 files changed, 69 insertions(+), 9 deletions(-)
arch/x86/kvm/ioapic.c +10 -1
@@ -377,8 +377,17 @@ static void ioapic_write_indirect(struct kvm_ioapic *ioapic, u32 val)* as edge-triggered. This behavior is used to simulate an * explicit EOI on IOAPICs that don't have the EOI register. */ -if (e->fields.trig_mode ==IOAPIC_EDGE_TRIG ) +if (e->fields.trig_mode ==IOAPIC_EDGE_TRIG ) { e->fields.remote_irr =0 ; + }else { +/* +* irr_delivered tracks edge-triggered interrupts that +* have already been delivered. Clear stale edge-triggered +* delivery state when the entry is configured for +* level-triggered interrupts. +*/ + ioapic->irr_delivered &= ~(1 << index); + } mask_after = e->fields.mask;if (mask_before != mask_after)
tools/testing/selftests/kvm/x86/ioapic_state_test.c +59 -8
106 lines not on this page · raw message
Signature · 1 line
2.15.0
Re: [PATCH v4 4/4] KVM: x86: ioapic: Clear irr_delivered on level-triggered RTE
Quoted 13 lines · Hao Zhang
From: Hao Zhang <[email protected]>
irr_deliveredhides delivered edge interrupts fromKVM_GET_IRQCHIP. The interrupt remains in IRR until the guest deasserts the line.The guest can reprogram an IOAPIC RTE from edge to level via MMIO. If an edge interrupt was delivered before the trigger mode change, a stale
irr_deliveredbit can incorrectly mask a subsequent pending level interrupt fromKVM_GET_IRQCHIP.That can cause userspace to migrate the VM without a pending level interrupt that is still present in KVM's in-kernel IOAPIC.
[...]
Quoted 8 lines
arch/x86/kvm/ioapic.c +1
-1@@ -377,8 +377,17 @@ static void ioapic_write_indirect(struct kvm_ioapic *ioapic, u32 val)* as edge-triggered. This behavior is used to simulate an * explicit EOI on IOAPICs that don't have the EOI register. */ -if (e->fields.trig_mode ==IOAPIC_EDGE_TRIG ) +if (e->fields.trig_mode ==IOAPIC_EDGE_TRIG ) {
arch/x86/kvm/ioapic.c static void ioapic_write_indirect(struct kvm_ioapic *ioapic, u32 val) · line 390
Quoted 10 lines
arch/x86/kvm/ioapic.c +9
-0e->fields.remote_irr =0 ; + }else { +/* +* irr_delivered tracks edge-triggered interrupts that +* have already been delivered. Clear stale edge-triggered +* delivery state when the entry is configured for +* level-triggered interrupts. +*/ + ioapic->irr_delivered &= ~(1 << index); + }
To be honest I am not sure whether we should consider this case, but IIUC this code change doesn't seem to have any harm, so will leave to others.
tools/testing/selftests/kvm/x86/ioapic_state_test.c
Quoted 7 lines
arch/x86/kvm/ioapic.c +0
-0mask_after = e->fields.mask;if (mask_before != mask_after)tools/testing/selftests/kvm/x86/ioapic_state_test.c +0
-0
Again, I think you should split selftest code change as a separate patch.
And it's better to have a cover letter for such series with multiple patches.
Quoted 11 lines · Hao Zhang
From: Hao Zhang <[email protected]>
The I/O APIC sets
remote_irrfor level-triggered interrupts that have been accepted by a local APIC and are waiting for an EOI.
ioapic_service()currently treats any non-zero return fromkvm_irq_delivery_to_apic()as successful delivery. Butkvm_irq_delivery_to_apic()returns -1 when no destination is found. Settingremote_irrfor that failed delivery leaves the pin blocked forever waiting for an EOI that will never be generated.
Quoted 4 lines
Update
remote_irronly when the delivery result is positive, i.e. when at least one local APIC accepted the interrupt.Fixes 4925663a079c ("KVM: Report IRQ injection status to userspace.")
The code LGTM, but I am not sure this is the right commit for the Fixes tag.
Looking at the relevant diff:
+static int ioapic_service(struct kvm_ioapic *ioapic, unsigned int idx)
{
union ioapic_redir_entry *pent;+ int injected = -1;
pent = &ioapic->redirtbl[idx];
if (!pent->fields.mask) {
- int injected = ioapic_deliver(ioapic, idx);
+ injected = ioapic_deliver(ioapic, idx);
if (injected && pent->fields.trig_mode == IOAPIC_LEVEL_TRIG)
pent->fields.remote_irr = 1;
}This commit didn't change any logic related to remote_irr, since it didn't touch ioapic_deliver() at all.
The one that is close seems to be
ff4b9df877b30 ("KVM: IOAPIC: only set remote_irr if interrupt was injected")
since it actually touched the logic of when to set remote_irr:
@@ -89,8 +89,8 @@ static void ioapic_service(struct kvm_ioapic *ioapic, unsigned int idx)
pent = &ioapic->redirtbl[idx];
if (!pent->fields.mask) {
- ioapic_deliver(ioapic, idx);
- if (pent->fields.trig_mode == IOAPIC_LEVEL_TRIG)
+ int injected = ioapic_deliver(ioapic, idx);
+ if (injected && pent->fields.trig_mode == IOAPIC_LEVEL_TRIG)
pent->fields.remote_irr = 1;
}But looking closer, at that time the ioapic_deliver() could only return 0 or 1 (or >1 I didn't check), i.e., it couldn't return -1 at all.
In fact, even when commit 4925663a079c was merged the then ioapic_deliver() couldn't return -1.
So AFAICT both of the above two commits are not the right one.
I haven't found which one to blame, but it seems at some point the function kvm_irq_delivery_to_apic() was introduced and at some other point it was changed to be able to return -1. But I didn't dig further.
Huang, Kai
Reviewed-byQuoted 35 lines · Kai
Quoted 11 lines · Hao Zhang
From: Hao Zhang <[email protected]>
The I/O APIC sets
remote_irrfor level-triggered interrupts that have been accepted by a local APIC and are waiting for an EOI.
ioapic_service()currently treats any non-zero return fromkvm_irq_delivery_to_apic()as successful delivery. Butkvm_irq_delivery_to_apic()returns -1 when no destination is found. Settingremote_irrfor that failed delivery leaves the pin blocked forever waiting for an EOI that will never be generated.Quoted 4 lines
Update
remote_irronly when the delivery result is positive, i.e. when at least one local APIC accepted the interrupt.Fixes 4925663a079c ("KVM: Report IRQ injection status to userspace.")The code LGTM, but I am not sure this is the right commit for the Fixes tag.
Looking at the relevant diff:
(quoted diff) +3
-1+static int ioapic_service(struct kvm_ioapic *ioapic, unsigned int idx) { union ioapic_redir_entry *pent; + int injected = -1; pent = &ioapic->redirtbl[idx]; if (!pent->fields.mask) { - int injected = ioapic_deliver(ioapic, idx); + injected = ioapic_deliver(ioapic, idx); if (injected && pent->fields.trig_mode == IOAPIC_LEVEL_TRIG) pent->fields.remote_irr = 1; }
Quoted 3 lines
This commit didn't change any logic related to
remote_irr, since it didn't touchioapic_deliver()at all.
I asked the AI to double confirm what I was reading last night, and indeed I got mixed up a bit among multiple commits in history.
Firstly, the above "it didn't touch ioapic_deliver() at all" isn't true. It indeed introduced 'r = -1' initializer at the beginning of the function. While I was right for "no target on destination" it returns 0 (but not -1), AI also pointed out there are other cases such as invalid delivery_mode which can cause ioapic_deliver() to return -1.
FWIW, AI also found it was commit
a53c17d21c46 ("KVM: ioapic/msi interrupt delivery consolidation")
which made ioapic_deliver() return -1 when "no target on destination":
if (find_first_bit(deliver_bitmask, KVM_MAX_VCPUS) >= KVM_MAX_VCPUS) {
ioapic_debug("no target on destination\n");
- return 0;
+ return r;
}So I guess you are right, 4925663a079c is the right one to blame, but not due to "no target on destination" case.
Anyway, IMHO we might also want to put such history digging into the changelog to justify the Fixes tag, so people can easily see the reason behind it.
That being said: