genirq: Expand doc for PENDING and REPLAY flags
PATCH 1/2 genirq: Expand doc for PENDING and REPLAY flags
Adding a bit more info about what the flags are used for may help future code readers.
--- kernel/irq/internals.h | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)
kernel/irq/internals.h +5 -2
@@ -47,9 +47,12 @@ enum {* detection * IRQS_POLL_INPROGRESS - polling in progress * IRQS_ONESHOT - irq is not unmasked in primary handler -* IRQS_REPLAY - irq is replayed +* IRQS_REPLAY - irq has been resent and will not be resent +* again until the handler has run and cleared +* this flag. * IRQS_WAITING - irq is waiting -* IRQS_PENDING - irq is pending and replayed later +* IRQS_PENDING - irq needs to be resent and should be resent +* at the next available opportunity. * IRQS_SUSPENDED - irq is suspended * IRQS_NMI - irq line is used to deliver NMIs * IRQS_SYSFS - descriptor has been added to sysfs
Signature · 1 line
2.25.1
PATCH 2/2 genirq: fasteoi resends interrupt on concurrent invoke
Update the generic handle_fasteoi_irq to cater for the case when the next interrupt comes in while the previous handler is still running. Currently when that happens the irq_may_run() early out causes the next IRQ to be lost. Change the behaviour to mark the interrupt as pending and re-send the interrupt when handle_fasteoi_irq sees that the pending flag has been set. This is largely inspired by handle_edge_irq.
Generally it should not be possible for the next interrupt to arrive while the previous handler is still running: the next interrupt should only arrive after the EOI message has been sent and the previous handler has returned. However, there is a race where if the interrupt affinity is changed while the previous handler is running, then the next interrupt can arrive at a different CPU while the previous handler is still running. In that case there will be a concurrent invoke and the early out will be taken.
For example:
CPU 0 | CPU 1
-----------------------------|-----------------------------
interrupt start |
handle_fasteoi_irq | set_affinity(CPU 1)
handler |
... | interrupt start
... | handle_fasteoi_irq -> early out
handle_fasteoi_irq return | interrupt end
interrupt end |This issue was observed specifically on an arm64 system with a GIC-v3 handling MSIs; GIC-v3 uses the handle_fasteoi_irq handler. The issue is that the global ITS is responsible for affinity but does not know whether interrupts are pending/running, only the CPU-local redistributor handles the EOI. Hence when the affinity is changed in the ITS, the new CPU's redistributor does not know that the original CPU is still running the handler.
Implementation notes:
It is believed that it's NOT necessary to mask the interrupt in handle_fasteoi_irq() the way that handle_edge_irq() does. This is because handle_edge_irq() caters for controllers which are too simple to gate interrupts from the same source, so the kernel explicitly masks the interrupt if it re-occurs [0].
[0] https://lore.kernel.org/all/[email protected]/
--- kernel/irq/chip.c | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-)
kernel/irq/chip.c +14 -1
@@ -692,8 +692,15 @@ void handle_fasteoi_irq(struct irq_desc *desc)raw_spin_lock (&desc->lock); -if (!irq_may_run (desc)) +/* +* When an affinity change races with IRQ delivery, the next interrupt +* can arrive on the new CPU before the original CPU has completed +* handling the previous one. Mark it as pending and return EOI. +*/ +if (!irq_may_run (desc)) { + desc->istate |=IRQS_PENDING ; goto out; + } desc->istate &= ~(IRQS_REPLAY |IRQS_WAITING );
@@ -715,6 +722,12 @@ void handle_fasteoi_irq(struct irq_desc *desc)cond_unmask_eoi_irq (desc, chip); +/* +* When the race descibed above happens, this will resend the interrupt. +*/ +if (unlikely (desc->istate &IRQS_PENDING )) +check_irq_resend (desc, false); +raw_spin_unlock (&desc->lock);return ; out:
Signature · 1 line
2.25.1
Re: [PATCH 2/2] genirq: fasteoi resends interrupt on concurrent invoke
Hi--
Quoted 26 lines · James Gowans
--- kernel/irq/chip.c | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-)kernel/irq/chip.c +8
-1@@ -692,8 +692,15 @@ void handle_fasteoi_irq(struct irq_desc *desc)raw_spin_lock (&desc->lock); -if (!irq_may_run (desc)) +/* +* When an affinity change races with IRQ delivery, the next interrupt +* can arrive on the new CPU before the original CPU has completed +* handling the previous one. Mark it as pending and return EOI. +*/ +if (!irq_may_run (desc)) { + desc->istate |=IRQS_PENDING ; goto out; + } desc->istate &= ~(IRQS_REPLAY |IRQS_WAITING );
kernel/irq/chip.c void handle_fasteoi_irq(struct irq_desc *desc) · line 726
Quoted 5 lines
kernel/irq/chip.c +2
-0@@ -715,6 +722,12 @@ void handle_fasteoi_irq(struct irq_desc *desc)cond_unmask_eoi_irq (desc, chip); +/* +* When the race descibed above happens, this will resend the interrupt.
described
Quoted 7 lines
kernel/irq/chip.c +4
-0+*/ +if (unlikely (desc->istate &IRQS_PENDING )) +check_irq_resend (desc, false); +raw_spin_unlock (&desc->lock);return ; out:
Signature · 1 line
~Randy
Re: [PATCH 2/2] genirq: fasteoi resends interrupt on concurrent invoke
In reply to James GowansQuoted 8 lines · James Gowans
Update the generic
handle_fasteoi_irqto cater for the case when the next interrupt comes in while the previous handler is still running. Currently when that happens theirq_may_run()early out causes the next IRQ to be lost. Change the behaviour to mark the interrupt as pending and re-send the interrupt whenhandle_fasteoi_irqsees that the pending flag has been set. This is largely inspired byhandle_edge_irq.
Quoted 4 lines
Generally it should not be possible for the next interrupt to arrive while the previous handler is still running: the next interrupt should only arrive after the EOI message has been sent and the previous handler has returned.
There is no such message with LPIs. I pointed that out previously.
Quoted 18 lines
However, there is a race where if the interrupt affinity is changed while the previous handler is running, then the next interrupt can arrive at a different CPU while the previous handler is still running. In that case there will be a concurrent invoke and the early out will be taken.
For example:
CPU 0 | CPU 1 -----------------------------|----------------------------- interrupt start | handle_fasteoi_irq | set_affinity(CPU 1) handler | ... | interrupt start ... | handle_fasteoi_irq -> early out handle_fasteoi_irq return | interrupt end interrupt end |
Quoted 7 lines
This issue was observed specifically on an arm64 system with a GIC-v3 handling MSIs; GIC-v3 uses the
handle_fasteoi_irqhandler. The issue is that the global ITS is responsible for affinity but does not know whether interrupts are pending/running, only the CPU-local redistributor handles the EOI. Hence when the affinity is changed in the ITS, the new CPU's redistributor does not know that the original CPU is still running the handler.
Similar to your previous patch, you don't explain *why* the interrupt gets delivered when it is an LPI, and not for any of the other GICv3 interrupt types. That's an important point.
Quoted 46 lines
Implementation notes:
It is believed that it's NOT necessary to mask the interrupt in
handle_fasteoi_irq()the way thathandle_edge_irq()does. This is becausehandle_edge_irq()caters for controllers which are too simple to gate interrupts from the same source, so the kernel explicitly masks the interrupt if it re-occurs [0].[0] https://lore.kernel.org/all/[email protected]/
Suggested-by Liao Chang <[email protected]>Signed-off-by James Gowans <[email protected]>Cc Thomas Gleixner <[email protected]>Cc Marc Zyngier <[email protected]>Cc KarimAllah Raslan <[email protected]>Cc Yipeng Zou <[email protected]>Cc Zhang Jianhua <[email protected]>--- kernel/irq/chip.c | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-)kernel/irq/chip.c +8
-1@@ -692,8 +692,15 @@ void handle_fasteoi_irq(struct irq_desc *desc)raw_spin_lock (&desc->lock); -if (!irq_may_run (desc)) +/* +* When an affinity change races with IRQ delivery, the next interrupt +* can arrive on the new CPU before the original CPU has completed +* handling the previous one. Mark it as pending and return EOI. +*/ +if (!irq_may_run (desc)) { + desc->istate |=IRQS_PENDING ; goto out; + } desc->istate &= ~(IRQS_REPLAY |IRQS_WAITING );@@ -715,6 +722,12 @@ void handle_fasteoi_irq(struct irq_desc *desc)cond_unmask_eoi_irq (desc, chip);
kernel/irq/chip.c void handle_fasteoi_irq(struct irq_desc *desc) · line 733
Quoted 10 lines
kernel/irq/chip.c +6
-0+/* +* When the race descibed above happens, this will resend the interrupt. +*/ +if (unlikely (desc->istate &IRQS_PENDING )) +check_irq_resend (desc, false); +raw_spin_unlock (&desc->lock);return ; out:
While I'm glad that you eventually decided to use the resend mechanism instead of spinning on the "old" CPU, I still think imposing this behaviour on all users without any discrimination is wrong.
Look at what it does if an interrupt is a wake-up source. You'd pointlessly requeue the interrupt (bonus points if the irqchip doesn't provide a HW-based retrigger mechanism).
I still maintain that this change should only be applied for the particular interrupts that *require* it, and not as a blanket change affecting everything under the sun. I have proposed such a change in the past, feel free to use it or roll your own.
In the meantime, I strongly oppose this change as proposed.
Thanks,
M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
Re: [PATCH 2/2] genirq: fasteoi resends interrupt on concurrent invoke
Quoted 6 lines · Marc Zyngier
Quoted 4 lines
Generally it should not be possible for the next interrupt to arrive while the previous handler is still running: the next interrupt should only arrive after the EOI message has been sent and the previous handler has returned.
There is no such message with LPIs. I pointed that out previously.
Arg, thanks, I'll re-word this to:
"Generally it should not be possible for the next interrupt to arrive while the previous handler is still running: the CPU will not preempt an interrupt with another from the same source or same priority."
I hope that's more accurate?
Quoted 11 lines
Quoted 7 lines
This issue was observed specifically on an arm64 system with a GIC-v3 handling MSIs; GIC-v3 uses the
handle_fasteoi_irqhandler. The issue is that the global ITS is responsible for affinity but does not know whether interrupts are pending/running, only the CPU-local redistributor handles the EOI. Hence when the affinity is changed in the ITS, the new CPU's redistributor does not know that the original CPU is still running the handler.Similar to your previous patch, you don't explain *why* the interrupt gets delivered when it is an LPI, and not for any of the other GICv3 interrupt types. That's an important point.
Right, you pointed out the issue with this sentence too and I missed updating it. :-/ How about:
"This issue was observed specifically on an arm64 system with a GIC-v3 handling MSIs; GIC-v3 uses the handle_fasteoi_irq handler. The issue is that the GIC-v3's physical LPIs do not have a global active state. If LPIs had an active state, then it would not be be able to be retriggered until the first CPU had issued a deactivation"
Quoted 16 lines
kernel/irq/chip.c void handle_fasteoi_irq(struct irq_desc *desc) · line 733
Quoted 11 lines
kernel/irq/chip.c +6
-0+/* +* When the race descibed above happens, this will resend the interrupt. +*/ +if (unlikely (desc->istate &IRQS_PENDING )) +check_irq_resend (desc, false); +raw_spin_unlock (&desc->lock);return ; out:While I'm glad that you eventually decided to use the resend mechanism instead of spinning on the "old" CPU, I still think imposing this behaviour on all users without any discrimination is wrong.
Quoted 8 lines
Look at what it does if an interrupt is a wake-up source. You'd pointlessly requeue the interrupt (bonus points if the irqchip doesn't provide a HW-based retrigger mechanism).
I still maintain that this change should only be applied for the particular interrupts that *require* it, and not as a blanket change affecting everything under the sun. I have proposed such a change in the past, feel free to use it or roll your own.
Thanks for the example of where this blanket functionality wouldn't be desired - I'll re-work this to introduce and use
the IRQD_RESEND_WHEN_IN_PROGRESS flag as you originally suggested.
Just one more thing before I post V3: are you okay with doing the resend here *after* the handler finished running, and using the IRQ_PENDING flag to know to resend it? Or would you like it to be resent in the !irq_may_run(desc) block as you suggested?
I have a slight preference to do it after, only when we know it's ready to be run again, and hence not needed to modify check_irq_resend() to cater for multiple retries.
JG
Re: [PATCH 2/2] genirq: fasteoi resends interrupt on concurrent invoke
Hi Marc,
Quoted 17 lines · James Gowans
While I'm glad that you eventually decided to use the resend mechanism
Quoted 11 lines
instead of spinning on the "old" CPU, I still think imposing this behaviour on all users without any discrimination is wrong.
Look at what it does if an interrupt is a wake-up source. You'd pointlessly requeue the interrupt (bonus points if the irqchip doesn't provide a HW-based retrigger mechanism).
I still maintain that this change should only be applied for the particular interrupts that *require* it, and not as a blanket change affecting everything under the sun. I have proposed such a change in the past, feel free to use it or roll your own.
Thanks for the example of where this blanket functionality wouldn't be desired - I'll re-work this to introduce and use
theIRQD_RESEND_WHEN_IN_PROGRESSflag as you originally suggested.
Quoted 8 lines
Just one more thing before I post V3: are you okay with doing the resend here *after* the handler finished running, and using the
IRQ_PENDINGflag to know to resend it? Or would you like it to be resent in the !irq_may_run(desc) block as you suggested?I have a slight preference to do it after, only when we know it's ready to be run again, and hence not needed to modify
check_irq_resend()to cater for multiple retries.
Hoping/assuming that you're okay with the keeping the resend at the end of the function, here's V3:
https://lore.kernel.org/lkml/[email protected]/
It's now very close to what to suggested originally. :-)
One thing: I'm not totally sure that it's necessary or correct to set this flag on its_vpe_irq_domain_alloc() too - on my system (r6g.metal Graviton 2 host) that doesn't seem to be run. My guess is that it's for a GIC-v4 system which supports posted interrupt which generally get delivery directly to the vCPU if running, but sometimes need to be delivered to the hypervisor (for example if vCPU not running). I can try test this on r7g.metal which might have GIC-v4 to be sure...
JG