Skip to contentSkip to conversation

powerpc/xics: Fix migrate_irqs_away - set CPPR to lowest priority

PATCH In review Balbir Singh 3 participants 3 messages Feb 27, 2017 – Mar 5, 2017 PowerPC

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 powerpc/xics: Fix migrate_irqs_away - set CPPR to lowest priority

With XICS emulation, setting the CPPR to DEFAULT_PRIORITY masks all interrupts including IPI's which map to a single underlying priority. The fix does two things

1. It moves the setting of CPPR to after all IRQ migration is complete
2. It sets the CPPR to LOWEST_PRIORITY, so that interrupts and IPI's are effectively enabled. The expectation is that we'll see just IPI's after migration

The fix is quite generic in that it will work when we move DEFAULT and IPI priorities to the same value in the kernel later.

Fixes d74361881f0d ("powerpc/xics: Add ICP OPAL backend")
---
 arch/powerpc/sysdev/xics/xics-common.c | 17 ++++++++++++++---
 1 file changed, 14 insertions(+), 3 deletions(-)
arch/powerpc/sysdev/xics/xics-common.c +14 -3
index c674a9d..31774e0 100644
@@ -20,6 +20,7 @@
 #include <linux/of.h>
 #include <linux/slab.h>
 #include <linux/spinlock.h>
+#include <linux/delay.h>
 
 #include <asm/prom.h>
 #include <asm/io.h>
@@ -198,9 +199,6 @@ void xics_migrate_irqs_away(void)
 	/* Remove ourselves from the global interrupt queue */
 	xics_set_cpu_giq(xics_default_distrib_server, 0);
 
-	/* Allow IPIs again... */
-	icp_ops->set_priority(LOWEST_PRIORITY);
-
 	for_each_irq_desc(virq, desc) {
 		struct irq_chip *chip;
 		long server;
@@ -255,6 +253,19 @@ void xics_migrate_irqs_away(void)
 unlock:
 		raw_spin_unlock_irqrestore(&desc->lock, flags);
 	}
+
+	/*
+	 * Allow sufficient time to drop any inflight IRQ's
+	 */
+	mdelay(1);
+
+	/*
+	 * Allow IPIs again. This is done at the very end, after
+	 * migrating all interrupts, the expectation is that we'll
+	 * only get woken up by an IPI interrupt beyond this point
+	 */
+	icp_ops->set_priority(LOWEST_PRIORITY);
+
 }
 #endif /* CONFIG_HOTPLUG_CPU */
 
Signature · 1 line
2.9.3
#1 lore raw
Quoted 1 line · Balbir Singh

With XICS emulation, setting the CPPR to DEFAULT_PRIORITY

^
(Current Processor Priority Register)

Quoted 2 lines

masks all interrupts including IPI's which map to a single underlying priority.

It took me a while to parse that.

So because of the way the OPAL XICS emulation is implemented, setting the CPPR to DEFAULT_PRIORITY has the effect of masking all interrupts.

That is because the OPAL code internally maps all priorities that are > 0 and < 0xff to a single priority. It happens to use 7, but I don't think that matters does it? It's just that there's no differentiation between DEFAULT and IPI.

I realise we need to work around it anyway, but are we calling this a bug in the XICS emulation? Or just an alternate feature? :)

Have we thought about doing the fix in icp_opal_set_cpu_priority() instead?

Quoted 11 lines

The fix does two things

1. It moves the setting of CPPR to after all IRQ migration is complete
2. It sets the CPPR to LOWEST_PRIORITY, so that interrupts and IPI's are effectively enabled. The expectation is that we'll see just IPI's after migration

The fix is quite generic in that it will work when we move DEFAULT and IPI priorities to the same value in the kernel later.

I didn't know we were doing that :)

What I'm most interested in is some soothing words to tell me that this definitely won't break on existing machines, using either a real native XICS or the HV backend.

Quoted 2 lines

I really dislike those Cc lines in the change log, they record nothing, other than the fact that Ben & Mikey get lots of email that they don't have time to read. In fact I'm not sure you even Cc'ed them?

Quoted 23 lines
Fixes d74361881f0d ("powerpc/xics: Add ICP OPAL backend")
---
 arch/powerpc/sysdev/xics/xics-common.c | 17 ++++++++++++++---
 1 file changed, 14 insertions(+), 3 deletions(-)
arch/powerpc/sysdev/xics/xics-common.c +1 -0
index c674a9d..31774e0 100644
@@ -20,6 +20,7 @@
 #include <linux/of.h>
 #include <linux/slab.h>
 #include <linux/spinlock.h>
+#include <linux/delay.h>
 
 #include <asm/prom.h>
 #include <asm/io.h>

arch/powerpc/sysdev/xics/xics-common.c void xics_migrate_irqs_away(void) · line 203

Quoted 6 lines
arch/powerpc/sysdev/xics/xics-common.c +0 -3
@@ -198,9 +199,6 @@ void xics_migrate_irqs_away(void)
 	/* Remove ourselves from the global interrupt queue */
 	xics_set_cpu_giq(xics_default_distrib_server, 0);
 
-	/* Allow IPIs again... */
-	icp_ops->set_priority(LOWEST_PRIORITY);
-

That's DEFAULT_PRIORITY in my tree?

arch/powerpc/sysdev/xics/xics-common.c void xics_migrate_irqs_away(void) · line 260

Quoted 9 lines
arch/powerpc/sysdev/xics/xics-common.c +5 -0
@@ -255,6 +253,19 @@ void xics_migrate_irqs_away(void)
 unlock:
 		raw_spin_unlock_irqrestore(&desc->lock, flags);
 	}
+
+	/*
+	 * Allow sufficient time to drop any inflight IRQ's
+	 */
+	mdelay(1);

That looks suspiciously like the code in migrate_irqs(), except it lacks the irq enable/disable.

So the comment should make it clear that we're waiting for the hardware to drop any inflight IRQs, not that we're waiting for them to be handled by Linux. And the dropping is caused by the set_priority(0) we did previously (not visible in the diff).

Do we have *any* basis for 1ms, other than it's "a while"?

arch/powerpc/sysdev/xics/xics-common.c void xics_migrate_irqs_away(void) · line 263

Quoted 3 lines
arch/powerpc/sysdev/xics/xics-common.c +3 -0
+
+	/*
+	 * Allow IPIs again. This is done at the very end, after

^
and all other interrupts

arch/powerpc/sysdev/xics/xics-common.c void xics_migrate_irqs_away(void) · line 265

Quoted 2 lines
arch/powerpc/sysdev/xics/xics-common.c +2 -0
+	 * migrating all interrupts, the expectation is that we'll
+	 * only get woken up by an IPI interrupt beyond this point

**because we have migrated all other irqs away**

?

arch/powerpc/sysdev/xics/xics-common.c void xics_migrate_irqs_away(void) · line 270

Quoted 5 lines
arch/powerpc/sysdev/xics/xics-common.c +3 -0
+	 */
+	icp_ops->set_priority(LOWEST_PRIORITY);
+
 }
 #endif /* CONFIG_HOTPLUG_CPU */

cheers

#2 lore raw
Quoted 12 lines · Michael Ellerman

It took me a while to parse that.

So because of the way the OPAL XICS emulation is implemented, setting the CPPR to DEFAULT_PRIORITY has the effect of masking all interrupts.

That is because the OPAL code internally maps all priorities that are

Quoted 1 line

0 and < 0xff to a single priority. It happens to use 7, but I don't think that matters does it? It's just that there's no differentiation between DEFAULT and IPI.

Quoted 5 lines

I realise we need to work around it anyway, but are we calling this a bug in the XICS emulation? Or just an alternate feature? :)

Have we thought about doing the fix in icp_opal_set_cpu_priority() instead?

The fix in OPAL is doable but requires churn. I would have to do IPIs differently (the way I do them in KVM XICS emulation actually which is noticeably better).

I will eventually do it I think but for now, I favor this patch instead because the existing code is also broken for Balbir WIP patch that removes the separate priority for IPIs to do lazy masking in the XICS.

Cheers,
Ben.