KVM: x86: ioapic: Update state only after successful delivery
PATCH 1/2 KVM: x86: ioapic: Update state only after successful delivery
From: Hao Zhang <[email protected]>
The I/O APIC tracks delivered interrupts in state that is later used to decide whether an interrupt is still pending or blocked waiting for an EOI.
For level-triggered interrupts, remote_irr means that a local APIC accepted the interrupt and that the I/O APIC must wait for the corresponding EOI before delivering the interrupt again. For edge-triggered interrupts, irr_delivered is used to hide delivered interrupts from KVM_GET_IRQCHIP so that userspace does not reinject an interrupt that has already left the I/O APIC.
But ioapic_service() currently updates that state before or without checking that interrupt delivery actually succeeded. kvm_irq_delivery_to_apic() can return -1 when no destination is found. Treating failed delivery as success can either leave a level-triggered pin blocked forever waiting for an EOI that will never be generated, or cause KVM_GET_IRQCHIP to drop an undelivered edge-triggered interrupt from the saved IRR state.
Update I/O APIC delivery state only when the delivery result is positive, i.e. when at least one local APIC accepted the interrupt.
---
Changes in v2:
- Address Kai Huang's review by deferring both remote_irr and irr_delivered updates until interrupt delivery succeeds.
- Extend the selftest to cover failed edge-triggered delivery and verify that the interrupt remains pending in IRR.
Link to v1: https://lore.kernel.org/all/[email protected]/
arch/x86/kvm/ioapic.c | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-)
arch/x86/kvm/ioapic.c +6 -5
@@ -474,9 +474,6 @@ static int ioapic_service(struct kvm_ioapic *ioapic, int irq, bool line_status) irqe.shorthand =APIC_DEST_NOSHORT ; irqe.msi_redir_hint = false; -if (irqe.trig_mode ==IOAPIC_EDGE_TRIG ) - ioapic->irr_delivered |=1 << irq; -if (irq ==RTC_GSI && line_status) {/* * pending_eoi cannot ever become negative (see
@@ -491,8 +488,12 @@ static int ioapic_service(struct kvm_ioapic *ioapic, int irq, bool line_status) }else ret =kvm_irq_delivery_to_apic (ioapic->kvm,NULL , &irqe); -if (ret && irqe.trig_mode ==IOAPIC_LEVEL_TRIG ) - entry->fields.remote_irr =1 ; +if (ret >0 ) { +if (irqe.trig_mode ==IOAPIC_EDGE_TRIG ) + ioapic->irr_delivered |=1 << irq; +else if (irqe.trig_mode ==IOAPIC_LEVEL_TRIG ) + entry->fields.remote_irr =1 ; + }return ret; }
base-commit: c21bb4193868a8de71fc4693fa741e195fdf5d86
Signature · 1 line
2.15.0
PATCH v2 2/2 KVM: selftests: Verify failed IOAPIC delivery preserves state
From: Hao Zhang <[email protected]>
Add regression coverage for I/O APIC interrupt delivery when no local APIC can accept the interrupt.
For level-triggered interrupts, verify that failed delivery does not set remote_irr, because no local APIC accepted the interrupt and no EOI will ever be generated.
For edge-triggered interrupts, verify that failed delivery does not cause KVM_GET_IRQCHIP to drop the pending IRR bit. Edge interrupts are hidden from the saved IRR state only after they have actually been delivered.
Both cases use an I/O APIC-only GSI and inject the interrupt before creating any vCPUs, forcing KVM_IRQ_LINE_STATUS to report failed delivery.
--- tools/testing/selftests/kvm/Makefile.kvm | 1 + .../testing/selftests/kvm/x86/ioapic_state_test.c | 108 +++++++++++++++++++++ 2 files changed, 109 insertions(+) create mode 100644 tools/testing/selftests/kvm/x86/ioapic_state_test.c
tools/testing/selftests/kvm/Makefile.kvm +1 -0
@@ -86,6 +86,7 @@ TEST_GEN_PROGS_x86 += x86/hyperv_features TEST_GEN_PROGS_x86 += x86/hyperv_ipi TEST_GEN_PROGS_x86 += x86/hyperv_svm_test TEST_GEN_PROGS_x86 += x86/hyperv_tlb_flush +TEST_GEN_PROGS_x86 += x86/ioapic_state_test TEST_GEN_PROGS_x86 += x86/kvm_clock_test TEST_GEN_PROGS_x86 += x86/kvm_pv_test TEST_GEN_PROGS_x86 += x86/kvm_buslock_test
tools/testing/selftests/kvm/x86/ioapic_state_test.c new file +108 -0
108 lines not on this page · raw message
Signature · 1 line
2.15.0
[email protected]
AI reviewRe: [PATCH v2 2/2] KVM: selftests: Verify failed IOAPIC delivery preserves state
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] Uninitialized `struct kvm_irqchip` passed to `KVM_GET_IRQCHIP` ioctl leaks stack memory to the…
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] Uninitialized struct kvm_irqchip passed to KVM_GET_IRQCHIP ioctl leaks stack memory to the kernel, triggering memory sanitizer errors. --
--- Patch [2]: [PATCH v2 2/2] KVM: selftests: Verify failed IOAPIC delivery preserves state --- commit bdb3cd0c37699df03cd34fd3d221e67589076c39
Author: Hao Zhang <[email protected]>
KVM: selftests: Verify failed IOAPIC delivery preserves state
This commit adds regression coverage for I/O APIC interrupt delivery when no local APIC can accept the interrupt. It verifies that failed delivery behaves correctly for both level-triggered and edge-triggered interrupts.
Quoted 5 lines
tools/testing/selftests/kvm/x86/ioapic_state_test.c new file +0
-0
[ ... ]
tools/testing/selftests/kvm/x86/ioapic_state_test.c line 38
Quoted 6 lines
tools/testing/selftests/kvm/x86/ioapic_state_test.c new file +6
-0+static void set_undeliverable_ioapic_entry (struct kvm_vm *vm, +bool level_triggered) +{ +struct kvm_irqchip irqchip; + +get_ioapic (vm, &irqchip);
[Severity: Medium]
Will this leak uninitialized stack memory to the kernel?
Since KVM_GET_IRQCHIP is defined as _IOWR, the kernel uses memdup_user() to copy the entire struct kvm_irqchip from userspace. Because the struct contains a 512-byte union and is declared uninitialized on the stack, passing it to the kernel will trigger memory sanitizer or Valgrind errors during testing.
Could this be zero-initialized?
[ ... ]
Quoted 9 lines
tools/testing/selftests/kvm/x86/ioapic_state_test.c new file +9
-0+static void test_no_remote_irr_for_undelivered_interrupt (void ) +{ +struct kvm_irq_level irq = { + .irq =TEST_IOAPIC_PIN , + .level =1 , + }; +struct kvm_irqchip irqchip; +struct kvm_vm *vm; +
tools/testing/selftests/kvm/x86/ioapic_state_test.c line 70
Quoted 10 lines
tools/testing/selftests/kvm/x86/ioapic_state_test.c new file +10
-0+ vm =vm_create_barebones (); +vm_create_irqchip (vm); + +set_undeliverable_ioapic_entry (vm, true); + +vm_ioctl (vm,KVM_IRQ_LINE_STATUS , &irq); +TEST_ASSERT (irq.status ==-1 , +"Expected failed interrupt delivery, got %d" , irq.status); + +get_ioapic (vm, &irqchip);
[Severity: Medium]
Does this struct also need to be zero-initialized to prevent memory sanitizer errors when calling the KVM_GET_IRQCHIP ioctl?
[ ... ]
Quoted 9 lines
tools/testing/selftests/kvm/x86/ioapic_state_test.c new file +9
-0+static void test_undelivered_edge_interrupt_stays_pending (void ) +{ +struct kvm_irq_level irq = { + .irq =TEST_IOAPIC_PIN , + .level =1 , + }; +struct kvm_irqchip irqchip; +struct kvm_vm *vm; +
tools/testing/selftests/kvm/x86/ioapic_state_test.c line 95
Quoted 10 lines
tools/testing/selftests/kvm/x86/ioapic_state_test.c new file +10
-0+ vm =vm_create_barebones (); +vm_create_irqchip (vm); + +set_undeliverable_ioapic_entry (vm, false); + +vm_ioctl (vm,KVM_IRQ_LINE_STATUS , &irq); +TEST_ASSERT (irq.status ==-1 , +"Expected failed interrupt delivery, got %d" , irq.status); + +get_ioapic (vm, &irqchip);
[Severity: Medium]
Does this uninitialized struct also need to be zero-initialized for the same reason?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
[email protected]
AI reviewThank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Pre-existing issues:
- [High] Clearing irr_delivered for duplicate edge interrupts causes spurious injections upon VM migration. --
--- Patch [1]: [PATCH 1/2] KVM: x86: ioapic: Update state only after successful delivery --- commit 709ff05e7e6d1df68c0523d442968791b55f37d1
Author: Hao Zhang <[email protected]>
KVM: x86: ioapic: Update state only after successful delivery
This commit defers updating I/O APIC delivery state until interrupt delivery actually succeeds. It ensures level-triggered pins aren't blocked forever on a failed delivery and edge-triggered interrupts aren't incorrectly dropped from the saved IRR state.
Quoted 4 lines
arch/x86/kvm/ioapic.c +0
-0
[ ... ]
arch/x86/kvm/ioapic.c static int ioapic_service(struct kvm_ioapic *ioapic, int irq, bool line_status) · line 496
Quoted 12 lines
arch/x86/kvm/ioapic.c +6
-2@@ -491,8 +488,12 @@ static int ioapic_service(struct kvm_ioapic *ioapic, int irq, bool line_status) }else ret =kvm_irq_delivery_to_apic (ioapic->kvm,NULL , &irqe); -if (ret && irqe.trig_mode ==IOAPIC_LEVEL_TRIG ) - entry->fields.remote_irr =1 ; +if (ret >0 ) { +if (irqe.trig_mode ==IOAPIC_EDGE_TRIG ) + ioapic->irr_delivered |=1 << irq; +else if (irqe.trig_mode ==IOAPIC_LEVEL_TRIG ) + entry->fields.remote_irr =1 ; + }
[Severity: High]
This isn't a bug introduced by this patch, but while looking at irr_delivered tracking, does ioapic_set_irq() have an issue with duplicate edge interrupts?
If a device emulator calls KVM_IRQ_LINE multiple times for an edge-triggered interrupt without an intervening level=0, the first call delivers the interrupt and sets irr_delivered here in ioapic_service().
However, on subsequent calls, ioapic_set_irq() does this:
arch/x86/kvm/ioapic.c:ioapic_set_irq() {
...
old_irr = ioapic->irr;
ioapic->irr |= mask;
if (edge) {
ioapic->irr_delivered &= ~mask;
if (old_irr == ioapic->irr) {
ret = 0;
goto out;
}
}
...
}It unsets irr_delivered but then detects the duplicate edge and returns early.
Could this leave irr_delivered incorrectly cleared?
If a VM migration occurs after this, KVM_GET_IRQCHIP might read the interrupt as pending instead of delivered.
Would this cause a spurious injection on the destination?
Quoted 3 lines
arch/x86/kvm/ioapic.c +0
-0return ret; }
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
Quoted 56 lines · Hao Zhang
From: Hao Zhang <[email protected]>
The I/O APIC tracks delivered interrupts in state that is later used to decide whether an interrupt is still pending or blocked waiting for an EOI.
For level-triggered interrupts,
remote_irrmeans that a local APIC accepted the interrupt and that the I/O APIC must wait for the corresponding EOI before delivering the interrupt again. For edge-triggered interrupts,irr_deliveredis used to hide delivered interrupts fromKVM_GET_IRQCHIPso that userspace does not reinject an interrupt that has already left the I/O APIC.But
ioapic_service()currently updates that state before or without checking that interrupt delivery actually succeeded.kvm_irq_delivery_to_apic()can return -1 when no destination is found. Treating failed delivery as success can either leave a level-triggered pin blocked forever waiting for an EOI that will never be generated, or causeKVM_GET_IRQCHIPto drop an undelivered edge-triggered interrupt from the saved IRR state.Update I/O APIC delivery state only when the delivery result is positive, i.e. when at least one local APIC accepted the interrupt.
Fixes 4925663a079c ("KVM: Report IRQ injection status to userspace.")Fixes 5bda6eed2e36 ("KVM: ioapic: Record edge-triggered interrupts delivery status")Signed-off-by Hao Zhang <[email protected]>---
Changes in v2:
- Address Kai Huang's review by deferring bothremote_irrandirr_deliveredupdates until interrupt delivery succeeds.
- Extend the selftest to cover failed edge-triggered delivery and verify that the interrupt remains pending in IRR.Link to v1: https://lore.kernel.org/all/[email protected]/
arch/x86/kvm/ioapic.c | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-)arch/x86/kvm/ioapic.c +0
-3@@ -474,9 +474,6 @@ static int ioapic_service(struct kvm_ioapic *ioapic, int irq, bool line_status) irqe.shorthand =APIC_DEST_NOSHORT ; irqe.msi_redir_hint = false; -if (irqe.trig_mode ==IOAPIC_EDGE_TRIG ) - ioapic->irr_delivered |=1 << irq; -if (irq ==RTC_GSI && line_status) {/* * pending_eoi cannot ever become negative (see @@ -491,8 +488,12 @@ static int ioapic_service(struct kvm_ioapic *ioapic, int irq, bool line_status) }else ret =kvm_irq_delivery_to_apic (ioapic->kvm,NULL , &irqe);
arch/x86/kvm/ioapic.c static int ioapic_service(struct kvm_ioapic *ioapic, int irq, bool line_status) · line 497
Quoted 10 lines
arch/x86/kvm/ioapic.c +6
-2-if (ret && irqe.trig_mode ==IOAPIC_LEVEL_TRIG ) - entry->fields.remote_irr =1 ; +if (ret >0 ) { +if (irqe.trig_mode ==IOAPIC_EDGE_TRIG ) + ioapic->irr_delivered |=1 << irq; +else if (irqe.trig_mode ==IOAPIC_LEVEL_TRIG ) + entry->fields.remote_irr =1 ; + }
Ah looking at this I immediately realized I misread the code, that I thought the irr_delivered was for level triggered as well, but actually it is for edge triggered. I guess we both got it wrong :-)
For edge triggered IRQ the existing code is correct I believe, since once the function is called the IRQ is considered delivered no matter whether it is actually accepted by LAPIC. I think this exactly reflects the hardware behaviour. (In fact, in ioapic_set_irq() you can see the IRQ is removed from irr_delivered for edge triggered right before ioapic_service() is called.)
For level triggered, I don't see ioapic->irr is cleared by KVM, but only cleared when ioapic_set_irq() is called with irq_level == 0. I now (AFAICT) realize it is the right behaviour since it should be the driver which dessert the level to stop the level-triggered IRQ.
So in short, I think your v1 is correct, and sorry about my noise. :-(
Quoted 78 lines · Kai
Quoted 56 lines · Hao Zhang
From: Hao Zhang <[email protected]>
The I/O APIC tracks delivered interrupts in state that is later used to decide whether an interrupt is still pending or blocked waiting for an EOI.
For level-triggered interrupts,
remote_irrmeans that a local APIC accepted the interrupt and that the I/O APIC must wait for the corresponding EOI before delivering the interrupt again. For edge-triggered interrupts,irr_deliveredis used to hide delivered interrupts fromKVM_GET_IRQCHIPso that userspace does not reinject an interrupt that has already left the I/O APIC.But
ioapic_service()currently updates that state before or without checking that interrupt delivery actually succeeded.kvm_irq_delivery_to_apic()can return -1 when no destination is found. Treating failed delivery as success can either leave a level-triggered pin blocked forever waiting for an EOI that will never be generated, or causeKVM_GET_IRQCHIPto drop an undelivered edge-triggered interrupt from the saved IRR state.Update I/O APIC delivery state only when the delivery result is positive, i.e. when at least one local APIC accepted the interrupt.
Fixes 4925663a079c ("KVM: Report IRQ injection status to userspace.")Fixes 5bda6eed2e36 ("KVM: ioapic: Record edge-triggered interrupts delivery status")Signed-off-by Hao Zhang <[email protected]>---
Changes in v2:
- Address Kai Huang's review by deferring bothremote_irrandirr_deliveredupdates until interrupt delivery succeeds.
- Extend the selftest to cover failed edge-triggered delivery and verify that the interrupt remains pending in IRR.Link to v1: https://lore.kernel.org/all/[email protected]/
arch/x86/kvm/ioapic.c | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-)arch/x86/kvm/ioapic.c +0
-3@@ -474,9 +474,6 @@ static int ioapic_service(struct kvm_ioapic *ioapic, int irq, bool line_status) irqe.shorthand =APIC_DEST_NOSHORT ; irqe.msi_redir_hint = false; -if (irqe.trig_mode ==IOAPIC_EDGE_TRIG ) - ioapic->irr_delivered |=1 << irq; -if (irq ==RTC_GSI && line_status) {/* * pending_eoi cannot ever become negative (see @@ -491,8 +488,12 @@ static int ioapic_service(struct kvm_ioapic *ioapic, int irq, bool line_status) }else ret =kvm_irq_delivery_to_apic (ioapic->kvm,NULL , &irqe);arch/x86/kvm/ioapic.c static int ioapic_service(struct kvm_ioapic *ioapic, int irq, bool line_status) · line 497
Quoted 10 lines
arch/x86/kvm/ioapic.c +6
-2-if (ret && irqe.trig_mode ==IOAPIC_LEVEL_TRIG ) - entry->fields.remote_irr =1 ; +if (ret >0 ) { +if (irqe.trig_mode ==IOAPIC_EDGE_TRIG ) + ioapic->irr_delivered |=1 << irq; +else if (irqe.trig_mode ==IOAPIC_LEVEL_TRIG ) + entry->fields.remote_irr =1 ; + }Ah looking at this I immediately realized I misread the code, that I thought the
irr_deliveredwas for level triggered as well, but actually it is for edge triggered. I guess we both got it wrong :-)For edge triggered IRQ the existing code is correct I believe, since once the function is called the IRQ is considered delivered no matter whether it is actually accepted by LAPIC. I think this exactly reflects the hardware behaviour. (In fact, in
ioapic_set_irq()you can see the IRQ is removed fromirr_deliveredfor edge triggered right beforeioapic_service()is called.)
Quoted 6 lines
For level triggered, I don't see
ioapic->irris cleared by KVM, but only cleared whenioapic_set_irq()is called withirq_level== 0. I now (AFAICT) realize it is the right behaviour since it should be the driver which dessert the level to stop the level-triggered IRQ.So in short, I think your v1 is correct, and sorry about my noise. :-(
Thanks Kai,
I checked the Intel I/O APIC redirection table documentation(https://edc.intel.com/content/www/it/it/design/products-and-solutions/processors-and-chipsets/comet-lake-u/intel-400-series-chipset-on-package-platform-controller-hub-register-database/1.2/redirection-table-entry-0-rte0-offset-10/). The Remote IRR bit is defined for level-triggered interrupts; for edge-triggered
interrupts its meaning is undefined. So yes, treating irr_delivered likeremote_irr was wrong.
For edge-triggered interrupts, once the I/O APIC starts delivering the interrupt, the interrupt should no longer be considered pending in the I/O APIC state that is saved through KVM_GET_IRQCHIP, regardless of whether a local APIC eventually accepts it.
So I agree that v2 went too far by moving the irr_delivered update behind the delivery result check. I will drop that part and restore the patch to only fix the level-triggered remote_irr case.
Thanks,
Hao