KVM: x86: ioapic: Set remote_irr only after successful delivery
PATCH 1/2 KVM: x86: ioapic: Set remote_irr only after successful delivery
From: Hao Zhang <[email protected]>
The I/O APIC sets remote_irr for level-triggered interrupts to track that the interrupt has been accepted by a local APIC and that the I/O APIC must wait for the corresponding EOI before delivering the interrupt again.
But ioapic_service() currently sets remote_irr for any non-zero return from kvm_irq_delivery_to_apic(). The delivery helper can return -1 when no destination is found. Treating -1 as success causes KVM to set remote_irr even though no interrupt was delivered and no EOI will ever be generated, leaving the pin blocked.
Set remote_irr only when the delivery result is positive, i.e. when at least one local APIC accepted the interrupt.
--- arch/x86/kvm/ioapic.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/kvm/ioapic.c +1 -1
@@ -491,7 +491,7 @@ static int ioapic_service(struct kvm_ioapic *ioapic, int irq, bool line_status) }else ret =kvm_irq_delivery_to_apic (ioapic->kvm,NULL , &irqe); -if (ret && irqe.trig_mode ==IOAPIC_LEVEL_TRIG ) +if (ret >0 && irqe.trig_mode ==IOAPIC_LEVEL_TRIG ) entry->fields.remote_irr =1 ;return ret;
base-commit: c21bb4193868a8de71fc4693fa741e195fdf5d86
Signature · 1 line
2.15.0
PATCH 2/2 KVM: selftests: Verify IOAPIC doesn't set remote_irr on failed delivery
From: Hao Zhang <[email protected]>
Add a regression test for level-triggered I/O APIC interrupt delivery when no local APIC can accept the interrupt.
remote_irr tracks that a level-triggered interrupt has been accepted by a local APIC and that the I/O APIC must wait for the corresponding EOI before delivering the interrupt again. If no local APIC accepts the interrupt, remote_irr must remain clear, as no EOI will ever be generated.
Configure an I/O APIC redirection entry and inject an I/O APIC-only GSI before creating any vCPUs. Verify that KVM_IRQ_LINE_STATUS reports failed delivery and that KVM_GET_IRQCHIP reports remote_irr clear.
This covers the case where failed delivery returns a negative value and must not be treated as a successful delivery for remote_irr tracking.
--- tools/testing/selftests/kvm/Makefile.kvm | 1 + .../testing/selftests/kvm/x86/ioapic_state_test.c | 70 ++++++++++++++++++++++ 2 files changed, 71 insertions(+) create mode 100644 tools/testing/selftests/kvm/x86/ioapic_state_test.c
tools/testing/selftests/kvm/Makefile.kvm +1 -0
@@ -86,6 +86,7 @@ TEST_GEN_PROGS_x86 += x86/hyperv_features TEST_GEN_PROGS_x86 += x86/hyperv_ipi TEST_GEN_PROGS_x86 += x86/hyperv_svm_test TEST_GEN_PROGS_x86 += x86/hyperv_tlb_flush +TEST_GEN_PROGS_x86 += x86/ioapic_state_test TEST_GEN_PROGS_x86 += x86/kvm_clock_test TEST_GEN_PROGS_x86 += x86/kvm_pv_test TEST_GEN_PROGS_x86 += x86/kvm_buslock_test
tools/testing/selftests/kvm/x86/ioapic_state_test.c new file +70 -0
70 lines not on this page · raw message
Signature · 1 line
2.15.0
Quoted 35 lines · Hao Zhang
From: Hao Zhang <[email protected]>
The I/O APIC sets
remote_irrfor level-triggered interrupts to track that the interrupt has been accepted by a local APIC and that the I/O APIC must wait for the corresponding EOI before delivering the interrupt again.But
ioapic_service()currently setsremote_irrfor any non-zero return fromkvm_irq_delivery_to_apic(). The delivery helper can return -1 when no destination is found. Treating -1 as success causes KVM to setremote_irreven though no interrupt was delivered and no EOI will ever be generated, leaving the pin blocked.Set
remote_irronly when the delivery result is positive, i.e. when at least one local APIC accepted the interrupt.Fixes 4925663a079c ("KVM: Report IRQ injection status to userspace.")Signed-off-by Hao Zhang <[email protected]>--- arch/x86/kvm/ioapic.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)arch/x86/kvm/ioapic.c +1
-1@@ -491,7 +491,7 @@ static int ioapic_service(struct kvm_ioapic *ioapic, int irq, bool line_status) }else ret =kvm_irq_delivery_to_apic (ioapic->kvm,NULL , &irqe); -if (ret && irqe.trig_mode ==IOAPIC_LEVEL_TRIG ) +if (ret >0 && irqe.trig_mode ==IOAPIC_LEVEL_TRIG ) entry->fields.remote_irr =1 ;return ret;
Quoted 1 line
base-commit:
c21bb4193868a8de71fc4693fa741e195fdf5d86
This seems reasonable to me. Just wondering did you meet any real bug?
Btw, currently the IRQ is set to irr_delivered for level-triggered IRQ regardless of the return value of kvm_irq_delivery_to_apic():
if (irqe.trig_mode == IOAPIC_EDGE_TRIG)
ioapic->irr_delivered |= 1 << irq;
if (irq == RTC_GSI && line_status) {
...
} else
ret = kvm_irq_delivery_to_apic(ioapic->kvm, NULL, &irqe);
...Similarly, should this only be done when kvm_irq_delivery_to_apic() returns positive?
E.g., kvm_get_ioapic() clears the bits in irr_delivered, so if one IRQ is set in it but actually not accepted by LAPIC then kvm_get_ioapic() will not return such IRQ in irr, therefore it can potentially be lost?
Quoted 34 lines · Kai
Quoted 33 lines · Hao Zhang
From: Hao Zhang <[email protected]>
The I/O APIC sets
remote_irrfor level-triggered interrupts to track that the interrupt has been accepted by a local APIC and that the I/O APIC must wait for the corresponding EOI before delivering the interrupt again.But
ioapic_service()currently setsremote_irrfor any non-zero return fromkvm_irq_delivery_to_apic(). The delivery helper can return -1 when no destination is found. Treating -1 as success causes KVM to setremote_irreven though no interrupt was delivered and no EOI will ever be generated, leaving the pin blocked.Set
remote_irronly when the delivery result is positive, i.e. when at least one local APIC accepted the interrupt.Fixes 4925663a079c ("KVM: Report IRQ injection status to userspace.")Signed-off-by Hao Zhang <[email protected]>--- arch/x86/kvm/ioapic.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)arch/x86/kvm/ioapic.c +1
-1@@ -491,7 +491,7 @@ static int ioapic_service(struct kvm_ioapic *ioapic, int irq, bool line_status) }else ret =kvm_irq_delivery_to_apic (ioapic->kvm,NULL , &irqe); -if (ret && irqe.trig_mode ==IOAPIC_LEVEL_TRIG ) +if (ret >0 && irqe.trig_mode ==IOAPIC_LEVEL_TRIG ) entry->fields.remote_irr =1 ;
Quoted 6 lines
Quoted 3 lines
arch/x86/kvm/ioapic.c +0
-0return ret;base-commit:
c21bb4193868a8de71fc4693fa741e195fdf5d86This seems reasonable to me. Just wondering did you meet any real bug?
I didn't hit this from a production workload; I found it while testing the failed-delivery path. But the state corruption is guest/userspace visible, especially across KVM_GET_IRQCHIP/KVM_SET_IRQCHIP.
Quoted 13 lines
Btw, currently the IRQ is set to
irr_deliveredfor level-triggered IRQ regardless of the return value ofkvm_irq_delivery_to_apic():if (irqe.trig_mode == IOAPIC_EDGE_TRIG) ioapic->irr_delivered |= 1 << irq; if (irq == RTC_GSI && line_status) { ... } else ret = kvm_irq_delivery_to_apic(ioapic->kvm, NULL, &irqe); ...
Quoted 8 lines
Similarly, should this only be done when
kvm_irq_delivery_to_apic()returns positive?E.g.,
kvm_get_ioapic()clears the bits inirr_delivered, so if one IRQ is set in it but actually not accepted by LAPIC thenkvm_get_ioapic()will not return such IRQ in irr, therefore it can potentially be lost?
Thanks, yes, I think you're right. irr_delivered has the same problem for edge-triggered interrupts: it is used to hide delivered edge IRQs from KVM_GET_IRQCHIP, but currently it is set before knowing whether delivery actually succeeded. If delivery fails, KVM_GET_IRQCHIP can incorrectly drop the pending IRR bit.
I'll send a v2 that gates both remote_irr and irr_delivered on ret > 0, and extend the selftest to cover the edge-triggered irr_delivered case too.
Thanks,
Hao
Quoted 6 lines
Quoted 6 lines
Btw, currently the IRQ is set to
irr_deliveredfor level-triggered IRQ regardless of the return value ofkvm_irq_delivery_to_apic():if (irqe.trig_mode == IOAPIC_EDGE_TRIG) ioapic->irr_delivered |= 1 << irq;
[...]
Quoted 5 lines
Thanks, yes, I think you're right.
irr_deliveredhas the same problem for edge-triggered interrupts: it is used to hide delivered edge IRQs fromKVM_GET_IRQCHIP, but currently it is set before knowing whether delivery actually succeeded. If delivery fails,KVM_GET_IRQCHIPcan incorrectly drop the pending IRR bit.
Sorry as I replied in v2, I misread the code that I thought irr_delivered is for level-triggered IRQ too. For edge triggered I think the hardware behaviour is once it is triggered the IRQ is considered delivered, since it's just an "edge".
Quoted 3 lines
I'll send a v2 that gates both
remote_irrandirr_deliveredon ret > 0, and extend the selftest to cover the edge-triggeredirr_deliveredcase too.
I wish I saw this reply earlier. Again sorry for the noise.