Skip to contentSkip to conversation

KVM: x86: ioapic: Set remote_irr only after successful delivery

PATCH In review Hao Zhang 2 participants 5 messages Aug 6 – Aug 10

Subsystems (4)
lore
Share & tools
lore

Hao Zhang

author
#0 lore raw

PATCH 1/2 KVM: x86: ioapic: Set remote_irr only after successful delivery

From: Hao Zhang <[email protected]>

The I/O APIC sets remote_irr for level-triggered interrupts to track that the interrupt has been accepted by a local APIC and that the I/O APIC must wait for the corresponding EOI before delivering the interrupt again.

But ioapic_service() currently sets remote_irr for any non-zero return from kvm_irq_delivery_to_apic(). The delivery helper can return -1 when no destination is found. Treating -1 as success causes KVM to set remote_irr even though no interrupt was delivered and no EOI will ever be generated, leaving the pin blocked.

Set remote_irr only when the delivery result is positive, i.e. when at least one local APIC accepted the interrupt.

Fixes 4925663a079c ("KVM: Report IRQ injection status to userspace.")
---
 arch/x86/kvm/ioapic.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/kvm/ioapic.c +1 -1
index 757667fb2bfa..24a7cc3b8b7e 100644
@@ -491,7 +491,7 @@ static int ioapic_service(struct kvm_ioapic *ioapic, int irq, bool line_status)
 	} else
 		ret = kvm_irq_delivery_to_apic(ioapic->kvm, NULL, &irqe);
 
-	if (ret && irqe.trig_mode == IOAPIC_LEVEL_TRIG)
+	if (ret > 0 && irqe.trig_mode == IOAPIC_LEVEL_TRIG)
 		entry->fields.remote_irr = 1;
 
 	return ret;

base-commit: c21bb4193868a8de71fc4693fa741e195fdf5d86

Signature · 1 line
2.15.0
author
#1 lore raw

PATCH 2/2 KVM: selftests: Verify IOAPIC doesn't set remote_irr on failed delivery

From: Hao Zhang <[email protected]>

Add a regression test for level-triggered I/O APIC interrupt delivery when no local APIC can accept the interrupt.

remote_irr tracks that a level-triggered interrupt has been accepted by a local APIC and that the I/O APIC must wait for the corresponding EOI before delivering the interrupt again. If no local APIC accepts the interrupt, remote_irr must remain clear, as no EOI will ever be generated.

Configure an I/O APIC redirection entry and inject an I/O APIC-only GSI before creating any vCPUs. Verify that KVM_IRQ_LINE_STATUS reports failed delivery and that KVM_GET_IRQCHIP reports remote_irr clear.

This covers the case where failed delivery returns a negative value and must not be treated as a successful delivery for remote_irr tracking.

---
 tools/testing/selftests/kvm/Makefile.kvm           |  1 +
 .../testing/selftests/kvm/x86/ioapic_state_test.c  | 70 ++++++++++++++++++++++
 2 files changed, 71 insertions(+)
 create mode 100644 tools/testing/selftests/kvm/x86/ioapic_state_test.c
tools/testing/selftests/kvm/Makefile.kvm +1 -0
index 6fc34e9bf8e1..89e3f82e75b0 100644
@@ -86,6 +86,7 @@ TEST_GEN_PROGS_x86 += x86/hyperv_features
 TEST_GEN_PROGS_x86 += x86/hyperv_ipi
 TEST_GEN_PROGS_x86 += x86/hyperv_svm_test
 TEST_GEN_PROGS_x86 += x86/hyperv_tlb_flush
+TEST_GEN_PROGS_x86 += x86/ioapic_state_test
 TEST_GEN_PROGS_x86 += x86/kvm_clock_test
 TEST_GEN_PROGS_x86 += x86/kvm_pv_test
 TEST_GEN_PROGS_x86 += x86/kvm_buslock_test
tools/testing/selftests/kvm/x86/ioapic_state_test.c new file +70 -0

70 lines not on this page · raw message

Signature · 1 line
2.15.0
#2 lore raw
In reply to Hao Zhang
Quoted 35 lines · Hao Zhang

From: Hao Zhang <[email protected]>

The I/O APIC sets remote_irr for level-triggered interrupts to track that the interrupt has been accepted by a local APIC and that the I/O APIC must wait for the corresponding EOI before delivering the interrupt again.

But ioapic_service() currently sets remote_irr for any non-zero return from kvm_irq_delivery_to_apic(). The delivery helper can return -1 when no destination is found. Treating -1 as success causes KVM to set remote_irr even though no interrupt was delivered and no EOI will ever be generated, leaving the pin blocked.

Set remote_irr only when the delivery result is positive, i.e. when at least one local APIC accepted the interrupt.

Fixes 4925663a079c ("KVM: Report IRQ injection status to userspace.")
---
 arch/x86/kvm/ioapic.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/kvm/ioapic.c +1 -1
index 757667fb2bfa..24a7cc3b8b7e 100644
@@ -491,7 +491,7 @@ static int ioapic_service(struct kvm_ioapic *ioapic, int irq, bool line_status)
 	} else
 		ret = kvm_irq_delivery_to_apic(ioapic->kvm, NULL, &irqe);
 
-	if (ret && irqe.trig_mode == IOAPIC_LEVEL_TRIG)
+	if (ret > 0 && irqe.trig_mode == IOAPIC_LEVEL_TRIG)
 		entry->fields.remote_irr = 1;
 
 	return ret;
Quoted 1 line

base-commit: c21bb4193868a8de71fc4693fa741e195fdf5d86

This seems reasonable to me. Just wondering did you meet any real bug?

Btw, currently the IRQ is set to irr_delivered for level-triggered IRQ regardless of the return value of kvm_irq_delivery_to_apic():

        if (irqe.trig_mode == IOAPIC_EDGE_TRIG)                                
                ioapic->irr_delivered |= 1 << irq;           

        if (irq == RTC_GSI && line_status) {                                   
		...                                                                          
        } else                                                                 
                ret = kvm_irq_delivery_to_apic(ioapic->kvm, NULL, &irqe);

	...

Similarly, should this only be done when kvm_irq_delivery_to_apic() returns positive?

E.g., kvm_get_ioapic() clears the bits in irr_delivered, so if one IRQ is set in it but actually not accepted by LAPIC then kvm_get_ioapic() will not return such IRQ in irr, therefore it can potentially be lost?

#3 lore raw
Quoted 34 lines · Kai
Quoted 33 lines · Hao Zhang

From: Hao Zhang <[email protected]>

The I/O APIC sets remote_irr for level-triggered interrupts to track that the interrupt has been accepted by a local APIC and that the I/O APIC must wait for the corresponding EOI before delivering the interrupt again.

But ioapic_service() currently sets remote_irr for any non-zero return from kvm_irq_delivery_to_apic(). The delivery helper can return -1 when no destination is found. Treating -1 as success causes KVM to set remote_irr even though no interrupt was delivered and no EOI will ever be generated, leaving the pin blocked.

Set remote_irr only when the delivery result is positive, i.e. when at least one local APIC accepted the interrupt.

Fixes 4925663a079c ("KVM: Report IRQ injection status to userspace.")
---
 arch/x86/kvm/ioapic.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/kvm/ioapic.c +1 -1
index 757667fb2bfa..24a7cc3b8b7e 100644
@@ -491,7 +491,7 @@ static int ioapic_service(struct kvm_ioapic *ioapic, int irq, bool line_status)
 	} else
 		ret = kvm_irq_delivery_to_apic(ioapic->kvm, NULL, &irqe);
 
-	if (ret && irqe.trig_mode == IOAPIC_LEVEL_TRIG)
+	if (ret > 0 && irqe.trig_mode == IOAPIC_LEVEL_TRIG)
 		entry->fields.remote_irr = 1;
 
Quoted 6 lines
Quoted 3 lines
arch/x86/kvm/ioapic.c +0 -0
 	return ret;

base-commit: c21bb4193868a8de71fc4693fa741e195fdf5d86

This seems reasonable to me. Just wondering did you meet any real bug?

I didn't hit this from a production workload; I found it while testing the failed-delivery path. But the state corruption is guest/userspace visible, especially across KVM_GET_IRQCHIP/KVM_SET_IRQCHIP.

Quoted 13 lines

Btw, currently the IRQ is set to irr_delivered for level-triggered IRQ regardless of the return value of kvm_irq_delivery_to_apic():

        if (irqe.trig_mode == IOAPIC_EDGE_TRIG)                                
                ioapic->irr_delivered |= 1 << irq;           

        if (irq == RTC_GSI && line_status) {                                   
		...                                                                          
        } else                                                                 
                ret = kvm_irq_delivery_to_apic(ioapic->kvm, NULL, &irqe);

	...
Quoted 8 lines

Similarly, should this only be done when kvm_irq_delivery_to_apic() returns positive?

E.g., kvm_get_ioapic() clears the bits in irr_delivered, so if one IRQ is set in it but actually not accepted by LAPIC then kvm_get_ioapic() will not return such IRQ in irr, therefore it can potentially be lost?

Thanks, yes, I think you're right. irr_delivered has the same problem for edge-triggered interrupts: it is used to hide delivered edge IRQs from KVM_GET_IRQCHIP, but currently it is set before knowing whether delivery actually succeeded. If delivery fails, KVM_GET_IRQCHIP can incorrectly drop the pending IRR bit.

I'll send a v2 that gates both remote_irr and irr_delivered on ret > 0, and extend the selftest to cover the edge-triggered irr_delivered case too.

Thanks,
Hao

#4 lore raw
Quoted 6 lines
Quoted 6 lines

Btw, currently the IRQ is set to irr_delivered for level-triggered IRQ regardless of the return value of kvm_irq_delivery_to_apic():

        if (irqe.trig_mode == IOAPIC_EDGE_TRIG)                                
                ioapic->irr_delivered |= 1 << irq;           

[...]

Quoted 5 lines

Thanks, yes, I think you're right. irr_delivered has the same problem for edge-triggered interrupts: it is used to hide delivered edge IRQs from KVM_GET_IRQCHIP, but currently it is set before knowing whether delivery actually succeeded. If delivery fails, KVM_GET_IRQCHIP can incorrectly drop the pending IRR bit.

Sorry as I replied in v2, I misread the code that I thought irr_delivered is for level-triggered IRQ too. For edge triggered I think the hardware behaviour is once it is triggered the IRQ is considered delivered, since it's just an "edge".

Quoted 3 lines

I'll send a v2 that gates both remote_irr and irr_delivered on ret > 0, and extend the selftest to cover the edge-triggered irr_delivered case too.

I wish I saw this reply earlier. Again sorry for the noise.