powerpc/xics: Fix migrate_irqs_away - set CPPR to lowest priority
PATCH 1/1 powerpc/xics: Fix migrate_irqs_away - set CPPR to lowest priority
With XICS emulation, setting the CPPR to DEFAULT_PRIORITY masks all interrupts including IPI's which map to a single underlying priority. The fix does two things
1. It moves the setting of CPPR to after all IRQ migration is complete
2. It sets the CPPR to LOWEST_PRIORITY, so that interrupts and IPI's are effectively enabled. The expectation is that we'll see just IPI's after migration
The fix is quite generic in that it will work when we move DEFAULT and IPI priorities to the same value in the kernel later.
--- arch/powerpc/sysdev/xics/xics-common.c | 17 ++++++++++++++--- 1 file changed, 14 insertions(+), 3 deletions(-)
arch/powerpc/sysdev/xics/xics-common.c +14 -3
@@ -20,6 +20,7 @@#include <linux/of.h> #include <linux/slab.h> #include <linux/spinlock.h> +#include <linux/delay.h> #include <asm/prom.h> #include <asm/io.h>
@@ -198,9 +199,6 @@ void xics_migrate_irqs_away(void)/* Remove ourselves from the global interrupt queue */ xics_set_cpu_giq (xics_default_distrib_server,0 ); -/* Allow IPIs again... */ - icp_ops->set_priority (LOWEST_PRIORITY ); -for_each_irq_desc (virq, desc) {struct irq_chip *chip;long server;
@@ -255,6 +253,19 @@ void xics_migrate_irqs_away(void) unlock: raw_spin_unlock_irqrestore(&desc->lock, flags); } + +/* +* Allow sufficient time to drop any inflight IRQ's +*/ +mdelay (1 ); + +/* +* Allow IPIs again. This is done at the very end, after +* migrating all interrupts, the expectation is that we'll +* only get woken up by an IPI interrupt beyond this point +*/ + icp_ops->set_priority (LOWEST_PRIORITY ); + }#endif /* CONFIG_HOTPLUG_CPU */
Signature · 1 line
2.9.3
Michael Ellerman
reviewerQuoted 1 line · Balbir Singh
With XICS emulation, setting the CPPR to
DEFAULT_PRIORITY
^
(Current Processor Priority Register)
Quoted 2 lines
masks all interrupts including IPI's which map to a single underlying priority.
It took me a while to parse that.
So because of the way the OPAL XICS emulation is implemented, setting the CPPR to DEFAULT_PRIORITY has the effect of masking all interrupts.
That is because the OPAL code internally maps all priorities that are > 0 and < 0xff to a single priority. It happens to use 7, but I don't think that matters does it? It's just that there's no differentiation between DEFAULT and IPI.
I realise we need to work around it anyway, but are we calling this a bug in the XICS emulation? Or just an alternate feature? :)
Have we thought about doing the fix in icp_opal_set_cpu_priority() instead?
Quoted 11 lines
The fix does two things
1. It moves the setting of CPPR to after all IRQ migration is complete
2. It sets the CPPR toLOWEST_PRIORITY, so that interrupts and IPI's are effectively enabled. The expectation is that we'll see just IPI's after migrationThe fix is quite generic in that it will work when we move DEFAULT and IPI priorities to the same value in the kernel later.
I didn't know we were doing that :)
What I'm most interested in is some soothing words to tell me that this definitely won't break on existing machines, using either a real native XICS or the HV backend.
Quoted 2 lines
I really dislike those Cc lines in the change log, they record nothing, other than the fact that Ben & Mikey get lots of email that they don't have time to read. In fact I'm not sure you even Cc'ed them?
Quoted 23 lines
Fixes d74361881f0d ("powerpc/xics: Add ICP OPAL backend")Cc [email protected] # v4.8+Reported-by Vaidyanathan Srinivasan <[email protected]>Signed-off-by Balbir Singh <[email protected]>--- arch/powerpc/sysdev/xics/xics-common.c | 17 ++++++++++++++--- 1 file changed, 14 insertions(+), 3 deletions(-)arch/powerpc/sysdev/xics/xics-common.c +1
-0@@ -20,6 +20,7 @@#include <linux/of.h> #include <linux/slab.h> #include <linux/spinlock.h> +#include <linux/delay.h> #include <asm/prom.h> #include <asm/io.h>
arch/powerpc/sysdev/xics/xics-common.c void xics_migrate_irqs_away(void) · line 203
Quoted 6 lines
arch/powerpc/sysdev/xics/xics-common.c +0
-3@@ -198,9 +199,6 @@ void xics_migrate_irqs_away(void)/* Remove ourselves from the global interrupt queue */ xics_set_cpu_giq (xics_default_distrib_server,0 ); -/* Allow IPIs again... */ - icp_ops->set_priority (LOWEST_PRIORITY ); -
That's DEFAULT_PRIORITY in my tree?
arch/powerpc/sysdev/xics/xics-common.c void xics_migrate_irqs_away(void) · line 260
Quoted 9 lines
arch/powerpc/sysdev/xics/xics-common.c +5
-0@@ -255,6 +253,19 @@ void xics_migrate_irqs_away(void) unlock:raw_spin_unlock_irqrestore (&desc->lock, flags); } + +/* +* Allow sufficient time to drop any inflight IRQ's +*/ +mdelay (1 );
That looks suspiciously like the code in migrate_irqs(), except it lacks the irq enable/disable.
So the comment should make it clear that we're waiting for the hardware to drop any inflight IRQs, not that we're waiting for them to be handled by Linux. And the dropping is caused by the set_priority(0) we did previously (not visible in the diff).
Do we have *any* basis for 1ms, other than it's "a while"?
arch/powerpc/sysdev/xics/xics-common.c void xics_migrate_irqs_away(void) · line 263
Quoted 3 lines
arch/powerpc/sysdev/xics/xics-common.c +3
-0+ +/* +* Allow IPIs again. This is done at the very end, after
^
and all other interrupts
arch/powerpc/sysdev/xics/xics-common.c void xics_migrate_irqs_away(void) · line 265
Quoted 2 lines
arch/powerpc/sysdev/xics/xics-common.c +2
-0+* migrating all interrupts, the expectation is that we'll +* only get woken up by an IPI interrupt beyond this point
**because we have migrated all other irqs away**
?
arch/powerpc/sysdev/xics/xics-common.c void xics_migrate_irqs_away(void) · line 270
Quoted 5 lines
arch/powerpc/sysdev/xics/xics-common.c +3
-0+*/ + icp_ops->set_priority(LOWEST_PRIORITY); + }#endif /* CONFIG_HOTPLUG_CPU */
cheers
Quoted 12 lines · Michael Ellerman
It took me a while to parse that.
So because of the way the OPAL XICS emulation is implemented, setting the CPPR to
DEFAULT_PRIORITYhas the effect of masking all interrupts.That is because the OPAL code internally maps all priorities that are
Quoted 1 line
0 and <
0xffto a single priority. It happens to use 7, but I don't think that matters does it? It's just that there's no differentiation between DEFAULT and IPI.
Quoted 5 lines
I realise we need to work around it anyway, but are we calling this a bug in the XICS emulation? Or just an alternate feature? :)
Have we thought about doing the fix in
icp_opal_set_cpu_priority()instead?
The fix in OPAL is doable but requires churn. I would have to do IPIs differently (the way I do them in KVM XICS emulation actually which is noticeably better).
I will eventually do it I think but for now, I favor this patch instead because the existing code is also broken for Balbir WIP patch that removes the separate priority for IPIs to do lazy masking in the XICS.
Cheers,
Ben.