irqchip/gic: Generalize use of HW-based retriggering
Subsystems (3)
Marc Zyngier
maintainerv2 cover irqchip/gic: Generalize use of HW-based retriggering
Valentin recently pointed out that that relying on SW-based retrigger with any of the GIC interrupt controllers is both inefficient and slightly broken, as it messes the GIC's own state machine.
For this to work with the hierarchical irqchip model that the GIC uses, we need check_irq_resend grow an understanding of hierarchies, something it has been lacking for some time.
The 3 other patches simply implement the HW resend callbacks, and finally prevent any SW resend for the GICs.
Unless anyone objects, I plan to take this into 5.10.
* From v1:
- Dropped most of the patches in favour of a teaching the core kernel to use irq_chip_retriger_hierarchy()
Marc Zyngier (2):
genirq: Walk the irq_data hierarchy when resending an interrupt irqchip/git-v3-its: Implement irq_retrigger callback for
device-triggered LPIs
Valentin Schneider (2):
irqchip/gic-v2, v3: Implement irq_chip->irq_retrigger() irqchip/gic-v2, v3: Prevent SW resends entirely
drivers/irqchip/irq-gic-v3-its.c | 6 ++++++ drivers/irqchip/irq-gic-v3.c | 12 +++++++++++- drivers/irqchip/irq-gic.c | 12 +++++++++++- kernel/irq/resend.c | 15 +++++++++++++-- 4 files changed, 41 insertions(+), 4 deletions(-)
Signature · 1 line
2.28.0
Marc Zyngier
maintainerPATCH v2 3/4 irqchip/git-v3-its: Implement irq_retrigger callback for device-triggered LPIs
It is pretty easy to provide a retrigger callback for the ITS, as it we already have the required support in terms of irq_set_irqchip_state().
Note that this only works for device-generated LPIs, and not the GICv4 doorbells, which should never have to be retriggered anyway.
--- drivers/irqchip/irq-gic-v3-its.c | 6 ++++++ 1 file changed, 6 insertions(+)
drivers/irqchip/irq-gic-v3-its.c +6 -0
@@ -1720,6 +1720,11 @@ static int its_irq_set_irqchip_state(struct irq_data *d,return 0 ; } +static int its_irq_retrigger (struct irq_data *d) +{ +return !its_irq_set_irqchip_state (d,IRQCHIP_STATE_PENDING , true); +} +/* * Two favourable cases: *
@@ -1971,6 +1976,7 @@ static struct irq_chip its_irq_chip = { .irq_set_affinity = its_set_affinity, .irq_compose_msi_msg = its_irq_compose_msi_msg, .irq_set_irqchip_state = its_irq_set_irqchip_state, + .irq_retrigger = its_irq_retrigger, .irq_set_vcpu_affinity = its_irq_set_vcpu_affinity, };
Signature · 1 line
2.28.0
Marc Zyngier
maintainerPATCH v2 1/4 genirq: Walk the irq_data hierarchy when resending an interrupt
In reply to Marc ZyngierOn resending an interrupt, we only check the topmost irqchip for a irq_retrigger callback. However, this callback could be implemented at a lower level. Use irq_chip_retrigger_hierarchy() in this case.
--- kernel/irq/resend.c | 15 +++++++++++++-- 1 file changed, 13 insertions(+), 2 deletions(-)
kernel/irq/resend.c +13 -2
@@ -86,6 +86,18 @@ static int irq_sw_resend(struct irq_desc *desc) }#endif +static int try_retrigger (struct irq_desc *desc) +{ +if (desc->irq_data.chip->irq_retrigger) +return desc->irq_data.chip->irq_retrigger (&desc->irq_data); + +#ifdef CONFIG_IRQ_DOMAIN_HIERARCHY +return irq_chip_retrigger_hierarchy (&desc->irq_data); +#else +return 0 ; +#endif +} +/* * IRQ resend *
@@ -113,8 +125,7 @@ int check_irq_resend(struct irq_desc *desc, bool inject) desc->istate &= ~IRQS_PENDING ; -if (!desc->irq_data.chip->irq_retrigger || - !desc->irq_data.chip->irq_retrigger (&desc->irq_data)) +if (!try_retrigger (desc)) err =irq_sw_resend (desc);/* If the retrigger was successfull, mark it with the REPLAY bit */
Signature · 1 line
2.28.0
Marc Zyngier
maintainerPATCH v2 4/4 irqchip/gic-v2, v3: Prevent SW resends entirely
In reply to Marc ZyngierFrom: Valentin Schneider <[email protected]>
The GIC irqchips can now use a HW resend when a retrigger is invoked by check_irq_resend(). However, should the HW resend fail, check_irq_resend() will still attempt to trigger a SW resend, which is still a bad idea for the GICs.
Prevent this from happening by setting IRQD_HANDLE_ENFORCE_IRQCTX on all GIC IRQs. Technically per-cpu IRQs do not need this, as their flow handlers never set IRQS_PENDING, but this aligns all IRQs wrt context enforcement:
this also forces all GIC IRQ handling to happen in IRQ context (as defined by in_irq()).
--- drivers/irqchip/irq-gic-v3.c | 5 ++++- drivers/irqchip/irq-gic.c | 6 +++++- 2 files changed, 9 insertions(+), 2 deletions(-)
drivers/irqchip/irq-gic-v3.c +4 -1
@@ -1279,6 +1279,7 @@ static int gic_irq_domain_map(struct irq_domain *d, unsigned int irq, irq_hw_number_t hw) {struct irq_chip *chip = &gic_chip; +struct irq_data *irqd =irq_desc_get_irq_data (irq_to_desc (irq));if (static_branch_likely (&supports_deactivate_key)) chip = &gic_eoimode1_chip;
@@ -1296,7 +1297,7 @@ static int gic_irq_domain_map(struct irq_domain *d, unsigned int irq,irq_domain_set_info (d, irq, hw, chip, d->host_data, handle_fasteoi_irq,NULL ,NULL );irq_set_probe (irq); -irqd_set_single_target (irq_desc_get_irq_data (irq_to_desc (irq))); +irqd_set_single_target (irqd);break ;case LPI_RANGE :
@@ -1310,6 +1311,8 @@ static int gic_irq_domain_map(struct irq_domain *d, unsigned int irq,return -EPERM ; } +/* Prevents SW retriggers which mess up the ACK/EOI ordering */ +irqd_set_handle_enforce_irqctx (irqd);return 0 ; }
drivers/irqchip/irq-gic.c +5 -1
@@ -975,6 +975,7 @@ static int gic_irq_domain_map(struct irq_domain *d, unsigned int irq, irq_hw_number_t hw) {struct gic_chip_data *gic = d->host_data; +struct irq_data *irqd =irq_desc_get_irq_data (irq_to_desc (irq));if (hw <32 ) {irq_set_percpu_devid (irq);
@@ -984,8 +985,11 @@ static int gic_irq_domain_map(struct irq_domain *d, unsigned int irq,irq_domain_set_info (d, irq, hw, &gic->chip, d->host_data, handle_fasteoi_irq,NULL ,NULL );irq_set_probe (irq); -irqd_set_single_target (irq_desc_get_irq_data (irq_to_desc (irq))); +irqd_set_single_target (irqd); } + +/* Prevents SW retriggers which mess up the ACK/EOI ordering */ +irqd_set_handle_enforce_irqctx (irqd);return 0 ; }
Signature · 1 line
2.28.0
Marc Zyngier
maintainerPATCH v2 2/4 irqchip/gic-v2, v3: Implement irq_chip->irq_retrigger()
In reply to Marc ZyngierFrom: Valentin Schneider <[email protected]>
While digging around IRQCHIP_EOI_IF_HANDLED and irq/resend.c, it has come to my attention that the IRQ resend situation seems a bit precarious for the GIC(s).
When marking an IRQ with IRQS_PENDING, handle_fasteoi_irq() will bail out and issue an irq_eoi(). Should the IRQ in question be re-enabled, check_irq_resend() will trigger a SW resend, which will go through the flow handler again and issue *another* irq_eoi() on the *same* IRQ activation. This is something the GIC spec clearly describes as a bad idea:
any EOI must match a previous ACK.
Implement irq_chip.irq_retrigger() for the GIC chips by setting the GIC pending bit of the relevant IRQ. After being called by check_irq_resend(), this will eventually trigger a *new* interrupt which we will handle as usual.
--- drivers/irqchip/irq-gic-v3.c | 7 +++++++ drivers/irqchip/irq-gic.c | 6 ++++++ 2 files changed, 13 insertions(+)
drivers/irqchip/irq-gic-v3.c +7 -0
@@ -1207,6 +1207,11 @@ static int gic_set_affinity(struct irq_data *d, const struct cpumask *mask_val,#define gic_smp_init () do { } while(0)#endif +static int gic_retrigger (struct irq_data *data) +{ +return !gic_irq_set_irqchip_state (data,IRQCHIP_STATE_PENDING , true); +} +#ifdef CONFIG_CPU_PM static int gic_cpu_pm_notifier (struct notifier_block *self,unsigned long cmd,void *v)
@@ -1242,6 +1247,7 @@ static struct irq_chip gic_chip = { .irq_eoi = gic_eoi_irq, .irq_set_type = gic_set_type, .irq_set_affinity = gic_set_affinity, + .irq_retrigger = gic_retrigger, .irq_get_irqchip_state = gic_irq_get_irqchip_state, .irq_set_irqchip_state = gic_irq_set_irqchip_state, .irq_nmi_setup = gic_irq_nmi_setup,
@@ -1258,6 +1264,7 @@ static struct irq_chip gic_eoimode1_chip = { .irq_eoi = gic_eoimode1_eoi_irq, .irq_set_type = gic_set_type, .irq_set_affinity = gic_set_affinity, + .irq_retrigger = gic_retrigger, .irq_get_irqchip_state = gic_irq_get_irqchip_state, .irq_set_irqchip_state = gic_irq_set_irqchip_state, .irq_set_vcpu_affinity = gic_irq_set_vcpu_affinity,
drivers/irqchip/irq-gic.c +6 -0
@@ -347,6 +347,11 @@ static int gic_set_affinity(struct irq_data *d, const struct cpumask *mask_val, }#endif +static int gic_retrigger (struct irq_data *data) +{ +return !gic_irq_set_irqchip_state (data,IRQCHIP_STATE_PENDING , true); +} + static void __exception_irq_entry gic_handle_irq(struct pt_regs *regs) {u32 irqstat, irqnr;
@@ -417,6 +422,7 @@ static const struct irq_chip gic_chip = { .irq_unmask = gic_unmask_irq, .irq_eoi = gic_eoi_irq, .irq_set_type = gic_set_type, + .irq_retrigger = gic_retrigger, .irq_get_irqchip_state = gic_irq_get_irqchip_state, .irq_set_irqchip_state = gic_irq_set_irqchip_state, .flags = IRQCHIP_SET_TYPE_MASKED |
Signature · 1 line
2.28.0
Re: [PATCH v2 1/4] genirq: Walk the irq_data hierarchy when resending an interrupt
In reply to Marc ZyngierHi Marc,
Quoted 4 lines · Marc Zyngier
On resending an interrupt, we only check the topmost irqchip for a
irq_retriggercallback. However, this callback could be implemented at a lower level. Useirq_chip_retrigger_hierarchy()in this case.
Rookie wording question here; re-reading this I'm questioning which way is up.
From an irq_data hierarchy PoV, the topmost chip (i.e. last ->parent) should be the root irqchip. However, the irq_desc we get from irq_to_desc() ought to hold the irq_data for the lowermost irqchip in that irq_data hierarchy.
Is it that here by "topmost" you instead mean topmost of the irqchip stack on top of the root (IOW furthest away from the root)?
Quoted 38 lines
Signed-off-by Marc Zyngier <[email protected]>--- kernel/irq/resend.c | 15 +++++++++++++-- 1 file changed, 13 insertions(+), 2 deletions(-)kernel/irq/resend.c +13
-2@@ -86,6 +86,18 @@ static int irq_sw_resend(struct irq_desc *desc) }#endif +static int try_retrigger (struct irq_desc *desc) +{ +if (desc->irq_data.chip->irq_retrigger) +return desc->irq_data.chip->irq_retrigger (&desc->irq_data); + +#ifdef CONFIG_IRQ_DOMAIN_HIERARCHY +return irq_chip_retrigger_hierarchy (&desc->irq_data); +#else +return 0 ; +#endif +} +/* * IRQ resend * @@ -113,8 +125,7 @@ int check_irq_resend(struct irq_desc *desc, bool inject) desc->istate &= ~IRQS_PENDING ; -if (!desc->irq_data.chip->irq_retrigger || - !desc->irq_data.chip->irq_retrigger (&desc->irq_data)) +if (!try_retrigger (desc)) err =irq_sw_resend (desc);/* If the retrigger was successfull, mark it with the REPLAY bit */
Marc Zyngier
maintainerRe: [PATCH v2 1/4] genirq: Walk the irq_data hierarchy when resending an interrupt
Hi Valentin,
Quoted 13 lines · Valentin Schneider
Hi Marc,
Quoted 4 lines · Marc Zyngier
On resending an interrupt, we only check the topmost irqchip for a
irq_retriggercallback. However, this callback could be implemented at a lower level. Useirq_chip_retrigger_hierarchy()in this case.Rookie wording question here; re-reading this I'm questioning which way is up.
Quoted 7 lines
From an
irq_datahierarchy PoV, the topmost chip (i.e. last ->parent) should be the root irqchip. However, theirq_descwe get fromirq_to_desc()ought to hold theirq_datafor the lowermost irqchip in thatirq_datahierarchy.Is it that here by "topmost" you instead mean topmost of the irqchip stack on top of the root (IOW furthest away from the root)?
That's indeed what I mean, but I agree that the terminology is confusing, and often used inconsistently (by me included).
<random>
Maybe considering the irqchip stack along a vertical axis is the wrong thing to do, and that looking at it as a volume would be marginally better?
How about innermost (close to the CPU) vs outermost (close to the device)?
</random>
Thanks,
M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
Valentin Schneider
Reviewed-byRe: [PATCH v2 1/4] genirq: Walk the irq_data hierarchy when resending an interrupt
Quoted 9 lines · Marc Zyngier
<random>
Maybe considering the irqchip stack along a vertical axis is the wrong thing to do, and that looking at it as a volume would be marginally better?How about innermost (close to the CPU) vs outermost (close to the device)?
</random>
I guess this is fairly subjective, but the inner/outer thing does click with me.
I think the "issue" with the top / bottom wording is that existing data structures (domain / irq_data hierarchy) bias my interpretation of it, but it's upside down from the irq_chip stack representation. That's not the case of inner / outer where all I can think of is the actual chip layout (i.e. as an image of the distance from the CPUs).
Anyway, that's enough psychoanalysis from me, the patches look fine - I also reran my quick rtcwake test on GICv2 (tests the WAKEUP_ARMED path). Feel free to add
to 1/4 & 3/4.
Quoted 3 lines
Thanks,
M.