irqchip/gic: Allow the use of SGI interrupts
cover irqchip/gic: Allow the use of SGI interrupts
Hi all,
Sending this as RFC so as to gather comments on the approach chosen here. The Broadcom STB mailbox driver and its firmware in EL3 use a combination of "smc" for inbound (Linux to monitor) and SGI for outbound (monitor to Linux) signaling. This mailbox driver can be seen here:
https://github.com/ffainelli/linux/commit/17cc97919f4cd2583d67e624273da8b54b44a4a7
(we may switch to the recently proposed standard arm-smc mailbox driver proposed by Peng Fang, but we would need interrupt notification anyway).
In our downstream kernel, we have hacked the arch/*/kernel/smp.c code to permit the installation of custom "IPI" handlers, this is obviously wrong and absolutely not suitable for usptream.
Here, we allow the GIC to recognize SGI interrupt specified in Device Tree with a new specifier in the first cell (2) and then we let the mapping and translation occur provided that we are above the NR_IPI range.
Immediate problems that I am aware of:
- on ARM (32-bit) NR_IPI does not include IPI_CPU_BACKTRACE, so we could (are) be off by one in our check against NR_IPI
Florian Fainelli (3):
dt-bindings: Define interrupt type for SGI interrupts irqchip/gic: Allow the use of SGI interrupts
.../interrupt-controller/arm,gic.yaml | 2 +- drivers/irqchip/irq-gic.c | 41 ++- .../interrupt-controller/arm-gic.h | 1 + 7 files changed, 313 insertions(+), 16 deletions(-) create mode 100644 drivers/mailbox/brcmstb-mailbox.c
Signature · 1 line
2.17.1
RFC PATCH 1/2 dt-bindings: Define interrupt type for SGI interrupts
In preparation for allowing drivers to use SGIs, define a new value for the first ARM GIC interrupt controller cell to take a new value to specifically designate a SGI interrupt.
--- .../devicetree/bindings/interrupt-controller/arm,gic.yaml | 2 +- include/dt-bindings/interrupt-controller/arm-gic.h | 1 + 2 files changed, 2 insertions(+), 1 deletion(-)
Documentation/devicetree/bindings/interrupt-controller/arm,gic.yaml +1 -1
@@ -58,7 +58,7 @@ properties: const: 3 description: | The 1st cell is the interrupt type; 0 for SPI interrupts, 1 for PPI - interrupts. + interrupts and 2 for SGI interrupts. The 2nd cell contains the interrupt number for the interrupt type. SPI interrupts are in the range [0-987]. PPI interrupts are in the
include/dt-bindings/interrupt-controller/arm-gic.h +1 -0
@@ -12,6 +12,7 @@#define GIC_SPI 0#define GIC_PPI 1 +#define GIC_SGI 2/* * Interrupt specifier cell 2.
Signature · 1 line
2.17.1
RFC PATCH 2/2 irqchip/gic: Allow the use of SGI interrupts
In reply to Florian FainelliSGI interrupts are a convenient way for trusted firmware to target a specific set of CPUs. Update the ARM GIC code to allow the translation and mapping of SGI interrupts.
Since the kernel already uses SGIs for various inter-processor interrupt activities, we specifically make sure that we do not let users of the IRQ API to even try to map those.
Internal IPIs remain dispatched through handle_IPI() while public SGIs get promoted to a normal interrupt flow management.
--- drivers/irqchip/irq-gic.c | 41 +++++++++++++++++++++++++++------------ 1 file changed, 29 insertions(+), 12 deletions(-)
drivers/irqchip/irq-gic.c +29 -12
@@ -385,7 +385,10 @@ static void __exception_irq_entry gic_handle_irq(struct pt_regs *regs)* Pairs with the write barrier in gic_raise_softirq */ smp_rmb (); -handle_IPI (irqnr, regs); +if (irqnr <NR_IPI ) +handle_IPI (irqnr, regs); +else +handle_domain_irq (gic->domain, irqnr, regs);#endif continue ; }
@@ -1005,20 +1008,34 @@ static int gic_irq_domain_translate(struct irq_domain *d,if (fwspec->param_count <3 )return -EINVAL ; -/* Get the interrupt number and add 16 to skip over SGIs */ - *hwirq = fwspec->param[1 ] +16 ; - -/* -* For SPIs, we need to add 16 more to get the GIC irq -* ID number -*/ -if (!fwspec->param[0 ]) + *hwirq = fwspec->param[1 ]; +switch (fwspec->param[0 ]) { +case 0 : +/* +* For SPIs, we need to add 16 more to get the GIC irq +* ID number +*/ + *hwirq +=16 ; +/* fall through */ +case 1 : +/* Add 16 to skip over SGIs */ *hwirq +=16 ; + *type = fwspec->param[2 ] &IRQ_TYPE_SENSE_MASK ; - *type = fwspec->param[2 ] &IRQ_TYPE_SENSE_MASK ; +/* Make it clear that broken DTs are... broken */ +WARN_ON (*type ==IRQ_TYPE_NONE ); +break ; +case 2 : +/* Refuse to map internal IPIs */ +if (*hwirq <NR_IPI ) +return -EPERM ; + + *type =IRQ_TYPE_NONE ; +break ; +default : +break ; + } -/* Make it clear that broken DTs are... broken */ -WARN_ON (*type ==IRQ_TYPE_NONE );return 0 ; }
Signature · 1 line
2.17.1
Marc Zyngier
maintainerHi Florian,
Needless to say, I mostly have questions...
Quoted 22 lines · Florian Fainelli
SGI interrupts are a convenient way for trusted firmware to target a specific set of CPUs. Update the ARM GIC code to allow the translation and mapping of SGI interrupts.
Since the kernel already uses SGIs for various inter-processor interrupt activities, we specifically make sure that we do not let users of the IRQ API to even try to map those.
Internal IPIs remain dispatched through
handle_IPI()while public SGIs get promoted to a normal interrupt flow management.Signed-off-by Florian Fainelli <[email protected]>--- drivers/irqchip/irq-gic.c | 41 +++++++++++++++++++++++++++------------ 1 file changed, 29 insertions(+), 12 deletions(-)drivers/irqchip/irq-gic.c +0
-0
drivers/irqchip/irq-gic.c static void __exception_irq_entry · line 392
Quoted 9 lines
drivers/irqchip/irq-gic.c +4
-1@@ -385,7 +385,10 @@ static void __exception_irq_entrygic_handle_irq(struct pt_regs *regs) * Pairs with the write barrier in gic_raise_softirq */ smp_rmb (); -handle_IPI (irqnr, regs); +if (irqnr <NR_IPI ) +handle_IPI (irqnr, regs); +else +handle_domain_irq (gic->domain, irqnr, regs);
Double EOI, UNPREDICTABLE territory, your state machine is now dead.
Quoted 27 lines
drivers/irqchip/irq-gic.c +0
-0#endif continue ;drivers/irqchip/irq-gic.c +11
-8}@@ -1005,20 +1008,34 @@ static int gic_irq_domain_translate(struct irq_domain *d, if (fwspec->param_count < 3) return -EINVAL; -/* Get the interrupt number and add 16 to skip over SGIs */ - *hwirq = fwspec->param[1 ] +16 ; - -/* -* For SPIs, we need to add 16 more to get the GIC irq -* ID number -*/ -if (!fwspec->param[0 ]) + *hwirq = fwspec->param[1 ]; +switch (fwspec->param[0 ]) { +case 0 : +/* +* For SPIs, we need to add 16 more to get the GIC irq +* ID number +*/ + *hwirq +=16 ; +/* fall through */ +case 1 : +/* Add 16 to skip over SGIs */
drivers/irqchip/irq-gic.c static int gic_irq_domain_translate(struct · line 1031
Quoted 10 lines
drivers/irqchip/irq-gic.c +7
-1*hwirq +=16 ; + *type = fwspec->param[2 ] &IRQ_TYPE_SENSE_MASK ; - *type = fwspec->param[2 ] &IRQ_TYPE_SENSE_MASK ; +/* Make it clear that broken DTs are... broken */ +WARN_ON (*type ==IRQ_TYPE_NONE ); +break ; +case 2 : +/* Refuse to map internal IPIs */ +if (*hwirq <NR_IPI )
So depending on how the kernel uses SGIs, you can or cannot use these SGIs. That looks like a good way to corner ourselves into not being to change much.
Also, do you expect this to work for both Group-0 and Group-1 interrupts (since you imply that this works as a communication medium with the secure side)? Given that the kernel running in NS has no way to enable/disable Group-0 interrupts, this looks terminally flawed. Or is that Group-1 only?
How do we describe which SGIs are guaranteed to be available to Linux?
drivers/irqchip/irq-gic.c static int gic_irq_domain_translate(struct irq_domain *d, · line 1033
Quoted 3 lines
drivers/irqchip/irq-gic.c +3
-0+return -EPERM ; + + *type =IRQ_TYPE_NONE ;
Or not. SGI are edge triggered, by definition.
drivers/irqchip/irq-gic.c static int gic_irq_domain_translate(struct irq_domain *d, · line 1021
Quoted 7 lines
drivers/irqchip/irq-gic.c +4
-2+break ; +default : +break ; + } -/* Make it clear that broken DTs are... broken */ -WARN_ON (*type ==IRQ_TYPE_NONE );
Really?
M.
Signature · 1 line
Jazz is not dead. It just smells funny...
Hello marc,
Quoted 21 lines · Marc Zyngier
Hi Florian,
Needless to say, I mostly have questions...
Quoted 16 lines · Florian Fainelli
SGI interrupts are a convenient way for trusted firmware to target a specific set of CPUs. Update the ARM GIC code to allow the translation and mapping of SGI interrupts.
Since the kernel already uses SGIs for various inter-processor interrupt activities, we specifically make sure that we do not let users of the IRQ API to even try to map those.
Internal IPIs remain dispatched through
handle_IPI()while public SGIs get promoted to a normal interrupt flow management.Signed-off-by Florian Fainelli <[email protected]>---
drivers/irqchip/irq-gic.c | 41 +++++++++++++++++++++++++++------------1 file changed, 29 insertions(+), 12 deletions(-)
Quoted 16 lines
Quoted 5 lines
drivers/irqchip/irq-gic.c +0
-0drivers/irqchip/irq-gic.c static void __exception_irq_entry · line 392
Quoted 9 lines
drivers/irqchip/irq-gic.c +4
-1@@ -385,7 +385,10 @@ static void __exception_irq_entrygic_handle_irq(struct pt_regs *regs) * Pairs with the write barrier in gic_raise_softirq */ smp_rmb(); - handle_IPI(irqnr, regs); + if (irqnr < NR_IPI) + handle_IPI(irqnr, regs); + else + handle_domain_irq(gic->domain, irqnr, regs);Double EOI, UNPREDICTABLE territory, your state machine is now dead.
Oh yes, the interrupt flow now also goes through ->irq_eoi (that's the whole point), meh.
Quoted 31 lines
Quoted 31 lines
drivers/irqchip/irq-gic.c +0
-0#endif continue;}
drivers/irqchip/irq-gic.c +12
-8@@ -1005,20 +1008,34 @@ static int gic_irq_domain_translate(struct irq_domain *d, if (fwspec->param_count < 3) return -EINVAL; - /* Get the interrupt number and add 16 to skip over SGIs */ - *hwirq = fwspec->param[1] + 16; - - /* - * For SPIs, we need to add 16 more to get the GIC irq - * ID number - */ - if (!fwspec->param[0]) + *hwirq = fwspec->param[1]; + switch (fwspec->param[0]) { + case 0: + /* + * For SPIs, we need to add 16 more to get the GIC irq + * ID number + */ + *hwirq += 16; + /* fall through */ + case 1: + /* Add 16 to skip over SGIs */ *hwirq += 16; + *type = fwspec->param[2] & IRQ_TYPE_SENSE_MASK;
Quoted 11 lines
drivers/irqchip/irq-gic.c static int gic_irq_domain_translate(struct irq_domain *d, · line 1030
Quoted 7 lines
drivers/irqchip/irq-gic.c +6
-1- *type = fwspec->param[2] & IRQ_TYPE_SENSE_MASK; + /* Make it clear that broken DTs are... broken */ + WARN_ON(*type == IRQ_TYPE_NONE); + break; + case 2: + /* Refuse to map internal IPIs */ + if (*hwirq < NR_IPI)So depending on how the kernel uses SGIs, you can or cannot use these SGIs. That looks like a good way to corner ourselves into not being to change much.
arch/arm/kernel/smp.c has a forward looking statement about SGI numbering:
/*
* SGI8-15 can be reserved by secure firmware, and thus may
* not be usable by the kernel. Please keep the above limited
* to at most 8 entries.
*/
is this something that can be used as an universal and unbreakable rule for the ARM64 kernel as well in order to ensure SGIs 8-15 can be usable through the IRQ API or is this simply not a guarantee at all?
Quoted 5 lines
Also, do you expect this to work for both Group-0 and Group-1 interrupts (since you imply that this works as a communication medium with the secure side)? Given that the kernel running in NS has no way to enable/disable Group-0 interrupts, this looks terminally flawed. Or is that Group-1 only?
That would be Group-1 interrupts only, are you suggesting there is an additional check being done that such SGIs are actually part of Group-1?
Quoted 2 lines
How do we describe which SGIs are guaranteed to be available to Linux?
In our case, the Device Tree mailbox node gets populated its interrupts property with the SGI number(s), and that same number is also passed as a configuration parameter to the trusted firmware. Or are you echoing back to your earlier comment about the fact that if the kernel changes its own definition of NR_IPI then we suddenly start breaking IRQ API uses of SGIs in a certain range?
Quoted 12 lines
drivers/irqchip/irq-gic.c static int gic_irq_domain_translate(struct irq_domain *d, · line 1033
Quoted 4 lines
drivers/irqchip/irq-gic.c +3
-0+ return -EPERM; + + *type = IRQ_TYPE_NONE;Or not. SGI are edge triggered, by definition.
Quoted 5 lines
drivers/irqchip/irq-gic.c +4
-0+ break; + default: + break; + }
Quoted 4 lines
drivers/irqchip/irq-gic.c static int gic_irq_domain_translate(struct irq_domain *d, · line 1021
Quoted 2 lines
drivers/irqchip/irq-gic.c +0
-2- /* Make it clear that broken DTs are... broken */ - WARN_ON(*type == IRQ_TYPE_NONE);Really?
Given the comment in gic_set_type() about SGIs, the WARN_ON() was moved above to continue checking for GIC_SPI and GIC_PPI, but we should extract the type from the Devic eTree and only permit an edge mask.
Signature · 1 line
Florian
Marc Zyngier
maintainerQuoted 42 lines · Florian Fainelli
Hello marc,
Quoted 22 lines · Marc Zyngier
Hi Florian,
Needless to say, I mostly have questions...
Quoted 17 lines · Florian Fainelli
SGI interrupts are a convenient way for trusted firmware to target a specific set of CPUs. Update the ARM GIC code to allow the translation and mapping of SGI interrupts.
Since the kernel already uses SGIs for various inter-processor interrupt activities, we specifically make sure that we do not let users of the IRQ API to even try to map those.
Internal IPIs remain dispatched through
handle_IPI()while public SGIs get promoted to a normal interrupt flow management.Signed-off-by Florian Fainelli <[email protected]>---
drivers/irqchip/irq-gic.c | 41
+++++++++++++++++++++++++++------------1 file changed, 29 insertions(+), 12 deletions(-)Quoted 16 lines
Quoted 5 lines
drivers/irqchip/irq-gic.c +0
-0drivers/irqchip/irq-gic.c static void __exception_irq_entry · line 392
Quoted 9 lines
drivers/irqchip/irq-gic.c +4
-1@@ -385,7 +385,10 @@ static void __exception_irq_entrygic_handle_irq(struct pt_regs *regs) * Pairs with the write barrier in gic_raise_softirq */ smp_rmb(); - handle_IPI(irqnr, regs); + if (irqnr < NR_IPI) + handle_IPI(irqnr, regs); + else + handle_domain_irq(gic->domain, irqnr, regs);Double EOI, UNPREDICTABLE territory, your state machine is now dead.
Quoted 2 lines
Oh yes, the interrupt flow now also goes through ->irq_eoi (that's the whole point), meh.
Indeed. But to be honest, we should probably consider moving all the SGI handling to normal interrupts. There's hardly any reason why we should keep SGIs out of the normal interrupt model, other than maybe performance (and that's pretty dubious).
Quoted 53 lines
Quoted 33 lines
Quoted 33 lines
drivers/irqchip/irq-gic.c +0
-0#endif continue;}
drivers/irqchip/irq-gic.c +12
-8@@ -1005,20 +1008,34 @@ static int gic_irq_domain_translate(structirq_domain *d, if (fwspec->param_count < 3) return -EINVAL; - /* Get the interrupt number and add 16 to skip over SGIs*/ - *hwirq = fwspec->param[1] + 16; - - /* - * For SPIs, we need to add 16 more to get the GIC irq - * ID number - */ - if (!fwspec->param[0]) + *hwirq = fwspec->param[1]; + switch (fwspec->param[0]) { + case 0: + /* + * For SPIs, we need to add 16 more to get the GIC irq + * ID number + */ + *hwirq += 16; + /* fall through */ + case 1: + /* Add 16 to skip over SGIs */ *hwirq += 16; + *type = fwspec->param[2] & IRQ_TYPE_SENSE_MASK;Quoted 11 lines
drivers/irqchip/irq-gic.c static int gic_irq_domain_translate(struct irq_domain *d, · line 1030
Quoted 7 lines
drivers/irqchip/irq-gic.c +6
-1- *type = fwspec->param[2] & IRQ_TYPE_SENSE_MASK; + /* Make it clear that broken DTs are... broken */ + WARN_ON(*type == IRQ_TYPE_NONE); + break; + case 2: + /* Refuse to map internal IPIs */ + if (*hwirq < NR_IPI)So depending on how the kernel uses SGIs, you can or cannot use these SGIs. That looks like a good way to corner ourselves into not being to change much.
arch/arm/kernel/smp.chas a forward looking statement about SGI numbering:/*
* SGI8-15 can be reserved by secure firmware, and thus may
* not be usable by the kernel. Please keep the above limited
* to at most 8 entries.
*/
Quoted 3 lines
is this something that can be used as an universal and unbreakable rule for the ARM64 kernel as well in order to ensure SGIs 8-15 can be usable through the IRQ API or is this simply not a guarantee at all?
There is no guarantee whatsoever. There's an ARM recommendation about the above split, but that's it. Hardly something that can be enforced.
Now, your firmware is the one that gives you the DT, so if it is inconsistent in configuring the interrupt and presenting it to the kernel, tough luck.
Quoted 7 lines
Quoted 4 lines
Also, do you expect this to work for both Group-0 and Group-1 interrupts (since you imply that this works as a communication medium with the secure side)? Given that the kernel running in NS has no way to enable/disable Group-0 interrupts, this looks terminally flawed. Or is that Group-1 only?
That would be Group-1 interrupts only, are you suggesting there is an additional check being done that such SGIs are actually part of Group-1?
You can try and change the configuration of that interrupt (priority, for example), and see if that sticks. If it doesn't, you're in trouble (and nothing you can do about it).
Quoted 9 lines
Quoted 2 lines
How do we describe which SGIs are guaranteed to be available to Linux?
In our case, the Device Tree mailbox node gets populated its interrupts property with the SGI number(s), and that same number is also passed as a configuration parameter to the trusted firmware. Or are you echoing back to your earlier comment about the fact that if the kernel changes its own definition of
NR_IPIthen we suddenly start breaking IRQ API uses of SGIs in a certain range?
That's indeed my worry. There is also the fact that the kernel itself will never expose such reservation in DT, so we'd have to tread carefully here. We probably need to specify that *only* SGI8-15 are allowed to be described as such.
Another thing: why don't you use a PPI? If you use a GICv2, you're also using ancient cores, and they have PPIs to spare. Or do you rely on being able to inject interrupts from one core to another?
Thanks,
M.
Signature · 1 line
Jazz is not dead. It just smells funny...