Skip to contentSkip to conversation

arm64: dts: fix PMU IRQ ordering for Juno

PATCH New Will Deacon 4 participants 15 messages Jan 26, 2015 – Feb 5, 2015

Subsystems (6)
lore
Share & tools
lore
#0 lore raw

PATCH 1/4 arm64: dts: fix PMU IRQ ordering for Juno

For better or worse, perf expects the per-cpu SPI PMU interrupts to be listed in order of logical CPU. This patch fixes the Juno .dts to satisfy that requirement.

Without this patch, I see unhandled IRQs in mainline:

irq 9: nobody cared (try booting with the "irqpoll" option)

  CPU: 3 PID: 2830 Comm: cc1 Not tainted 3.19.0-rc6+ #1
  Hardware name: ARM Juno development board (r0) (DT)

[...]

handlers:
[<ffffffc00009447c>] armv8pmu_handle_irq
Disabling IRQ #9

---

This is an immediate fix for mainline, with the remaining patches in the series solving this by extending the binding.

 arch/arm64/boot/dts/arm/juno.dts | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
arch/arm64/boot/dts/arm/juno.dts +3 -3
index cb3073e4e7a8..4ed9287aaef1 100644
@@ -107,11 +107,11 @@
 	pmu {
 		compatible = "arm,armv8-pmuv3";
 		interrupts = <GIC_SPI 18 IRQ_TYPE_LEVEL_HIGH>,
+			     <GIC_SPI 02 IRQ_TYPE_LEVEL_HIGH>,
+			     <GIC_SPI 06 IRQ_TYPE_LEVEL_HIGH>,
 			     <GIC_SPI 22 IRQ_TYPE_LEVEL_HIGH>,
 			     <GIC_SPI 26 IRQ_TYPE_LEVEL_HIGH>,
-			     <GIC_SPI 30 IRQ_TYPE_LEVEL_HIGH>,
-			     <GIC_SPI 02 IRQ_TYPE_LEVEL_HIGH>,
-			     <GIC_SPI 06 IRQ_TYPE_LEVEL_HIGH>;
+			     <GIC_SPI 30 IRQ_TYPE_LEVEL_HIGH>;
 	};
 
 	/include/ "juno-clocks.dtsi"
Signature · 1 line
2.1.4
#1 lore raw

PATCH 2/4 arm64: pmu: add support for interrupt-affinity property

Historically, the PMU devicetree bindings have expected SPIs to be listed in order of *logical* CPU number. This is problematic for bootloaders, especially when the boot CPU (logical ID 0) isn't listed first in the devicetree.

This patch adds a new optional property, interrupt-affinity, to the PMU node which allows the interrupt affinity to be described using a list of phandled to CPU nodes, with each entry in the list corresponding to the SPI at the same index in the interrupts property.

---
 Documentation/devicetree/bindings/arm/pmu.txt |  6 +++
 arch/arm64/include/asm/pmu.h                  |  1 +
 arch/arm64/kernel/perf_event.c                | 57 +++++++++++++++++++++++++--
 3 files changed, 60 insertions(+), 4 deletions(-)
Documentation/devicetree/bindings/arm/pmu.txt +6 -0
index 75ef91d08f3b..a9281fc48743 100644
@@ -24,6 +24,12 @@ Required properties:
 
 Optional properties:
 
+- interrupt-affinity : Valid only when using SPIs, specifies a list of phandles
+                       to CPU nodes corresponding directly to the affinity of
+		       the SPIs listed in the interrupts property. If absent,
+		       the interrupts are assumed to be listed in logical CPU
+		       order.
+
 - qcom,no-pc-write : Indicates that this PMU doesn't support the 0xc and 0xd
                      events.
 
arch/arm64/include/asm/pmu.h +1 -0
index e6f087806aaf..b7710a59672c 100644
@@ -44,6 +44,7 @@ struct pmu_hw_events {
 struct arm_pmu {
 	struct pmu		pmu;
 	cpumask_t		active_irqs;
+	int			*irq_affinity;
 	const char		*name;
 	irqreturn_t		(*handle_irq)(int irq_num, void *dev);
 	void			(*enable)(struct hw_perf_event *evt, int idx);
arch/arm64/kernel/perf_event.c +53 -4

103 lines not on this page · raw message

Signature · 1 line
2.1.4
maintainer
#2 lore raw

PATCH 2/4 arm64: pmu: add support for interrupt-affinity property

Quoted 23 lines · Will Deacon

Historically, the PMU devicetree bindings have expected SPIs to be listed in order of *logical* CPU number. This is problematic for bootloaders, especially when the boot CPU (logical ID 0) isn't listed first in the devicetree.

This patch adds a new optional property, interrupt-affinity, to the PMU node which allows the interrupt affinity to be described using a list of phandled to CPU nodes, with each entry in the list corresponding to the SPI at the same index in the interrupts property.

---
 Documentation/devicetree/bindings/arm/pmu.txt |  6 +++
 arch/arm64/include/asm/pmu.h                  |  1 +
 arch/arm64/kernel/perf_event.c                | 57 +++++++++++++++++++++++++--
 3 files changed, 60 insertions(+), 4 deletions(-)
Documentation/devicetree/bindings/arm/pmu.txt +0 -0
index 75ef91d08f3b..a9281fc48743 100644

Documentation/devicetree/bindings/arm/pmu.txt Required properties: · line 31

Quoted 8 lines
Documentation/devicetree/bindings/arm/pmu.txt +5 -0
@@ -24,6 +24,12 @@ Required properties:
 
 Optional properties:
 
+- interrupt-affinity : Valid only when using SPIs, specifies a list of phandles
+                       to CPU nodes corresponding directly to the affinity of
+		       the SPIs listed in the interrupts property. If absent,
+		       the interrupts are assumed to be listed in logical CPU
+		       order.

This covers the case we care about today, but it's problematic in cases where the number of interrupts is not equal to the number of CPUs affine to that interrupt. For example:

* PPIs in big.LITTLE systems, where we may need a node per cluster, and will need a way of associating a PMU node with a subset of all CPUs, despite having only one interrupt.

* Muxed SPIs per-cluster (is this likely to happen?)

The former can be covered by allowing multiple entries in interrupt-affintiy for PPIs. I'm not sure if the latter is something we need to cater for. If we do, then perhaps we need an interruptN-affinity property per interrupt (though that's ugly and painful to deal with).

It would be nice to have a solution now that's not radically different to what needs to come next.

Thanks,
Mark.

#3 lore raw

PATCH 2/4 arm64: pmu: add support for interrupt-affinity property

Quoted 37 lines · Mark Rutland
Quoted 23 lines · Will Deacon

Historically, the PMU devicetree bindings have expected SPIs to be listed in order of *logical* CPU number. This is problematic for bootloaders, especially when the boot CPU (logical ID 0) isn't listed first in the devicetree.

This patch adds a new optional property, interrupt-affinity, to the PMU node which allows the interrupt affinity to be described using a list of phandled to CPU nodes, with each entry in the list corresponding to the SPI at the same index in the interrupts property.

---
 Documentation/devicetree/bindings/arm/pmu.txt |  6 +++
 arch/arm64/include/asm/pmu.h                  |  1 +
 arch/arm64/kernel/perf_event.c                | 57 +++++++++++++++++++++++++--
 3 files changed, 60 insertions(+), 4 deletions(-)
Documentation/devicetree/bindings/arm/pmu.txt +0 -0
index 75ef91d08f3b..a9281fc48743 100644

Documentation/devicetree/bindings/arm/pmu.txt Required properties: · line 31

Quoted 8 lines
Documentation/devicetree/bindings/arm/pmu.txt +5 -0
@@ -24,6 +24,12 @@ Required properties:
 
 Optional properties:
 
+- interrupt-affinity : Valid only when using SPIs, specifies a list of phandles
+                       to CPU nodes corresponding directly to the affinity of
+		       the SPIs listed in the interrupts property. If absent,
+		       the interrupts are assumed to be listed in logical CPU
+		       order.

This covers the case we care about today, but it's problematic in cases where the number of interrupts is not equal to the number of CPUs affine to that interrupt. For example:

Quoted 8 lines

* PPIs in big.LITTLE systems, where we may need a node per cluster, and will need a way of associating a PMU node with a subset of all CPUs, despite having only one interrupt.

* Muxed SPIs per-cluster (is this likely to happen?)

The former can be covered by allowing multiple entries in interrupt-affintiy for PPIs.

Yes, that sounds like a sensible extension in the future if we have to support such a platform.

Quoted 3 lines

I'm not sure if the latter is something we need to cater for. If we do, then perhaps we need an interruptN-affinity property per interrupt (though that's ugly and painful to deal with).

I'm not keen to handle this, so I'd rather defer it to whoever ends up building it. Trying to design for every possibility is usually impossible in my experience and you just end up carrying something that isn't useful.

Will

maintainer
#4 lore raw

PATCH 2/4 arm64: pmu: add support for interrupt-affinity property

Quoted 50 lines · Will Deacon
Quoted 37 lines · Mark Rutland
Quoted 23 lines · Will Deacon

Historically, the PMU devicetree bindings have expected SPIs to be listed in order of *logical* CPU number. This is problematic for bootloaders, especially when the boot CPU (logical ID 0) isn't listed first in the devicetree.

This patch adds a new optional property, interrupt-affinity, to the PMU node which allows the interrupt affinity to be described using a list of phandled to CPU nodes, with each entry in the list corresponding to the SPI at the same index in the interrupts property.

---
 Documentation/devicetree/bindings/arm/pmu.txt |  6 +++
 arch/arm64/include/asm/pmu.h                  |  1 +
 arch/arm64/kernel/perf_event.c                | 57 +++++++++++++++++++++++++--
 3 files changed, 60 insertions(+), 4 deletions(-)
Documentation/devicetree/bindings/arm/pmu.txt +0 -0
index 75ef91d08f3b..a9281fc48743 100644

Documentation/devicetree/bindings/arm/pmu.txt Required properties: · line 31

Quoted 8 lines
Documentation/devicetree/bindings/arm/pmu.txt +5 -0
@@ -24,6 +24,12 @@ Required properties:
 
 Optional properties:
 
+- interrupt-affinity : Valid only when using SPIs, specifies a list of phandles
+                       to CPU nodes corresponding directly to the affinity of
+		       the SPIs listed in the interrupts property. If absent,
+		       the interrupts are assumed to be listed in logical CPU
+		       order.

This covers the case we care about today, but it's problematic in cases where the number of interrupts is not equal to the number of CPUs affine to that interrupt. For example:

Quoted 8 lines

* PPIs in big.LITTLE systems, where we may need a node per cluster, and will need a way of associating a PMU node with a subset of all CPUs, despite having only one interrupt.

* Muxed SPIs per-cluster (is this likely to happen?)

The former can be covered by allowing multiple entries in interrupt-affintiy for PPIs.

Yes, that sounds like a sensible extension in the future if we have to support such a platform.

Quoted 7 lines
Quoted 3 lines

I'm not sure if the latter is something we need to cater for. If we do, then perhaps we need an interruptN-affinity property per interrupt (though that's ugly and painful to deal with).

I'm not keen to handle this, so I'd rather defer it to whoever ends up building it. Trying to design for every possibility is usually impossible in my experience and you just end up carrying something that isn't useful.

I suspected that would be the case. Just thought I should raise it as a potential problem.

Thanks,
Mark.

#5 lore raw

PATCH 3/4 ARM: pmu: add support for interrupt-affinity property

In reply to Will Deacon

Historically, the PMU devicetree bindings have expected SPIs to be listed in order of *logical* CPU number. This is problematic for bootloaders, especially when the boot CPU (logical ID 0) isn't listed first in the devicetree.

This patch adds a new optional property, interrupt-affinity, to the PMU node which allows the interrupt affinity to be described using a list of phandled to CPU nodes, with each entry in the list corresponding to the SPI at the same index in the interrupts property.

---
 arch/arm/include/asm/pmu.h       |  1 +
 arch/arm/kernel/perf_event_cpu.c | 69 ++++++++++++++++++++++++++++++++++++----
 2 files changed, 63 insertions(+), 7 deletions(-)
arch/arm/include/asm/pmu.h +1 -0
index b1596bd59129..675e4ab79f68 100644
@@ -92,6 +92,7 @@ struct pmu_hw_events {
 struct arm_pmu {
 	struct pmu	pmu;
 	cpumask_t	active_irqs;
+	int		*irq_affinity;
 	char		*name;
 	irqreturn_t	(*handle_irq)(int irq_num, void *dev);
 	void		(*enable)(struct perf_event *event);
arch/arm/kernel/perf_event_cpu.c +62 -7

118 lines not on this page · raw message

Signature · 1 line
2.1.4
#6 lore raw

PATCH 4/4 arm64: dts: add interrupt-affinity property to pmu node for juno

In reply to Will Deacon

Make the Juno .dts robust against potential reordering of the CPU nodes by adding an explicit interrupt-affinity property to the PMU node.

---
 arch/arm64/boot/dts/arm/juno.dts | 6 ++++++
 1 file changed, 6 insertions(+)
arch/arm64/boot/dts/arm/juno.dts +6 -0
index 4ed9287aaef1..3542500137f0 100644
@@ -112,6 +112,12 @@
 			     <GIC_SPI 22 IRQ_TYPE_LEVEL_HIGH>,
 			     <GIC_SPI 26 IRQ_TYPE_LEVEL_HIGH>,
 			     <GIC_SPI 30 IRQ_TYPE_LEVEL_HIGH>;
+		interrupt-affinity = <&A53_0>,
+				     <&A57_0>,
+				     <&A57_1>,
+				     <&A53_1>,
+				     <&A53_2>,
+				     <&A53_3>;
 	};
 
 	/include/ "juno-clocks.dtsi"
Signature · 1 line
2.1.4
maintainer
#7 lore raw

PATCH 1/4 arm64: dts: fix PMU IRQ ordering for Juno

In reply to Will Deacon

Hi Will,

Quoted 33 lines · Will Deacon

For better or worse, perf expects the per-cpu SPI PMU interrupts to be listed in order of logical CPU. This patch fixes the Juno .dts to satisfy that requirement.

Without this patch, I see unhandled IRQs in mainline:

irq 9: nobody cared (try booting with the "irqpoll" option)

  CPU: 3 PID: 2830 Comm: cc1 Not tainted 3.19.0-rc6+ #1
  Hardware name: ARM Juno development board (r0) (DT)

[...]

handlers:
[<ffffffc00009447c>] armv8pmu_handle_irq
Disabling IRQ #9

---

This is an immediate fix for mainline, with the remaining patches in the series solving this by extending the binding.

 arch/arm64/boot/dts/arm/juno.dts | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
arch/arm64/boot/dts/arm/juno.dts +0 -0
index cb3073e4e7a8..4ed9287aaef1 100644
@@ -107,11 +107,11 @@
 	pmu {
 		compatible = "arm,armv8-pmuv3";

arch/arm64/boot/dts/arm/juno.dts line 115

Quoted 10 lines
arch/arm64/boot/dts/arm/juno.dts +3 -3
 		interrupts = <GIC_SPI 18 IRQ_TYPE_LEVEL_HIGH>,
+			     <GIC_SPI 02 IRQ_TYPE_LEVEL_HIGH>,
+			     <GIC_SPI 06 IRQ_TYPE_LEVEL_HIGH>,
 			     <GIC_SPI 22 IRQ_TYPE_LEVEL_HIGH>,
 			     <GIC_SPI 26 IRQ_TYPE_LEVEL_HIGH>,
-			     <GIC_SPI 30 IRQ_TYPE_LEVEL_HIGH>,
-			     <GIC_SPI 02 IRQ_TYPE_LEVEL_HIGH>,
-			     <GIC_SPI 06 IRQ_TYPE_LEVEL_HIGH>;
+			     <GIC_SPI 30 IRQ_TYPE_LEVEL_HIGH>;
 	};

I am very much not keen on this. While this may get things working today, it completely relies on Linux-internal details (the order of CPU bringup, which in this case is different from the order of entries in /cpus).

In all other dts that I am aware of, the order of entries in /cpus aligns with the order of interrupts in the PMU node, and the first entry is the boot CPU.

I think that we should ensure that the ordering of CPU nodes matches the order of interrupts here. That way we can fall back to that ordering (if not explicitly overridden), and even after an arbitrary logical renumbering (e.g. after a kexec) the relationship should stay intact.

This DT has clearly never worked (nor been tested), and I think having this as an intermediary step only adds to the long term support burden by having the juno dts arbitrarily different to all other dts files (by relying on a logical order that's different to the /cpus order).

Longer term we must ensure we have a more explicit ordering, as with your later patches.

Thanks,
Mark.

#8 lore raw

PATCH 1/4 arm64: dts: fix PMU IRQ ordering for Juno

Quoted 24 lines · Mark Rutland
Quoted 7 lines · Will Deacon
arch/arm64/boot/dts/arm/juno.dts +0 -0
index cb3073e4e7a8..4ed9287aaef1 100644
@@ -107,11 +107,11 @@
 	pmu {
 		compatible = "arm,armv8-pmuv3";

arch/arm64/boot/dts/arm/juno.dts line 115

Quoted 10 lines
arch/arm64/boot/dts/arm/juno.dts +3 -3
 		interrupts = <GIC_SPI 18 IRQ_TYPE_LEVEL_HIGH>,
+			     <GIC_SPI 02 IRQ_TYPE_LEVEL_HIGH>,
+			     <GIC_SPI 06 IRQ_TYPE_LEVEL_HIGH>,
 			     <GIC_SPI 22 IRQ_TYPE_LEVEL_HIGH>,
 			     <GIC_SPI 26 IRQ_TYPE_LEVEL_HIGH>,
-			     <GIC_SPI 30 IRQ_TYPE_LEVEL_HIGH>,
-			     <GIC_SPI 02 IRQ_TYPE_LEVEL_HIGH>,
-			     <GIC_SPI 06 IRQ_TYPE_LEVEL_HIGH>;
+			     <GIC_SPI 30 IRQ_TYPE_LEVEL_HIGH>;
 	};

I am very much not keen on this. While this may get things working today, it completely relies on Linux-internal details (the order of CPU bringup, which in this case is different from the order of entries in /cpus).

Quoted 8 lines

In all other dts that I am aware of, the order of entries in /cpus aligns with the order of interrupts in the PMU node, and the first entry is the boot CPU.

I think that we should ensure that the ordering of CPU nodes matches the order of interrupts here. That way we can fall back to that ordering (if not explicitly overridden), and even after an arbitrary logical renumbering (e.g. after a kexec) the relationship should stay intact.

There are a few problems with reordering the CPU nodes:

(1) It breaks any existing users of taskset to pin on big/little clusters.

(2) It's not generally possible if, for example, the bootloader decides to boot Linux on a different CPU then we have no choice but to change the PMU interrupt order.

(3) I didn't think that the ordering of CPU nodes was guaranteed to be preserved by dtc, whereas the order of the interrupts will be.

Quoted 7 lines

This DT has clearly never worked (nor been tested), and I think having this as an intermediary step only adds to the long term support burden by having the juno dts arbitrarily different to all other dts files (by relying on a logical order that's different to the /cpus order).

Longer term we must ensure we have a more explicit ordering, as with your later patches.

Agreed. This is intended as something simpler for -stable.

Will

maintainer
#9 lore raw

PATCH 1/4 arm64: dts: fix PMU IRQ ordering for Juno

Quoted 34 lines · Will Deacon
Quoted 24 lines · Mark Rutland
Quoted 7 lines · Will Deacon
arch/arm64/boot/dts/arm/juno.dts +0 -0
index cb3073e4e7a8..4ed9287aaef1 100644
@@ -107,11 +107,11 @@
 	pmu {
 		compatible = "arm,armv8-pmuv3";

arch/arm64/boot/dts/arm/juno.dts line 115

Quoted 10 lines
arch/arm64/boot/dts/arm/juno.dts +3 -3
 		interrupts = <GIC_SPI 18 IRQ_TYPE_LEVEL_HIGH>,
+			     <GIC_SPI 02 IRQ_TYPE_LEVEL_HIGH>,
+			     <GIC_SPI 06 IRQ_TYPE_LEVEL_HIGH>,
 			     <GIC_SPI 22 IRQ_TYPE_LEVEL_HIGH>,
 			     <GIC_SPI 26 IRQ_TYPE_LEVEL_HIGH>,
-			     <GIC_SPI 30 IRQ_TYPE_LEVEL_HIGH>,
-			     <GIC_SPI 02 IRQ_TYPE_LEVEL_HIGH>,
-			     <GIC_SPI 06 IRQ_TYPE_LEVEL_HIGH>;
+			     <GIC_SPI 30 IRQ_TYPE_LEVEL_HIGH>;
 	};

I am very much not keen on this. While this may get things working today, it completely relies on Linux-internal details (the order of CPU bringup, which in this case is different from the order of entries in /cpus).

Quoted 8 lines

In all other dts that I am aware of, the order of entries in /cpus aligns with the order of interrupts in the PMU node, and the first entry is the boot CPU.

I think that we should ensure that the ordering of CPU nodes matches the order of interrupts here. That way we can fall back to that ordering (if not explicitly overridden), and even after an arbitrary logical renumbering (e.g. after a kexec) the relationship should stay intact.

Quoted 4 lines

There are a few problems with reordering the CPU nodes:

(1) It breaks any existing users of taskset to pin on big/little clusters.

This is unfortunate, but this is also the case if the boot CPU is different.

Quoted 3 lines

(2) It's not generally possible if, for example, the bootloader decides to boot Linux on a different CPU then we have no choice but to change the PMU interrupt order.

In that case _this_ patch is broken.

If we associate the interrupt with a CPU by node order, the relationship is preserved regardless of which CPU is the boot CPU (whether it was the bootloader's choice, kexec, or whatever).

Quoted 2 lines

(3) I didn't think that the ordering of CPU nodes was guaranteed to be preserved by dtc, whereas the order of the interrupts will be.

The order of nodes is presently preserved.

Thanks,
Mark.

#10 lore raw

PATCH 1/4 arm64: dts: fix PMU IRQ ordering for Juno

Quoted 35 lines · Mark Rutland
Quoted 34 lines · Will Deacon
Quoted 32 lines · Mark Rutland
Quoted 7 lines · Will Deacon
arch/arm64/boot/dts/arm/juno.dts +0 -0
index cb3073e4e7a8..4ed9287aaef1 100644
@@ -107,11 +107,11 @@
 	pmu {
 		compatible = "arm,armv8-pmuv3";

arch/arm64/boot/dts/arm/juno.dts line 115

Quoted 10 lines
arch/arm64/boot/dts/arm/juno.dts +3 -3
 		interrupts = <GIC_SPI 18 IRQ_TYPE_LEVEL_HIGH>,
+			     <GIC_SPI 02 IRQ_TYPE_LEVEL_HIGH>,
+			     <GIC_SPI 06 IRQ_TYPE_LEVEL_HIGH>,
 			     <GIC_SPI 22 IRQ_TYPE_LEVEL_HIGH>,
 			     <GIC_SPI 26 IRQ_TYPE_LEVEL_HIGH>,
-			     <GIC_SPI 30 IRQ_TYPE_LEVEL_HIGH>,
-			     <GIC_SPI 02 IRQ_TYPE_LEVEL_HIGH>,
-			     <GIC_SPI 06 IRQ_TYPE_LEVEL_HIGH>;
+			     <GIC_SPI 30 IRQ_TYPE_LEVEL_HIGH>;
 	};

I am very much not keen on this. While this may get things working today, it completely relies on Linux-internal details (the order of CPU bringup, which in this case is different from the order of entries in /cpus).

In all other dts that I am aware of, the order of entries in /cpus aligns with the order of interrupts in the PMU node, and the first entry is the boot CPU.

I think that we should ensure that the ordering of CPU nodes matches the order of interrupts here. That way we can fall back to that ordering (if not explicitly overridden), and even after an arbitrary logical renumbering (e.g. after a kexec) the relationship should stay intact.

Quoted 7 lines
Quoted 4 lines

There are a few problems with reordering the CPU nodes:

(1) It breaks any existing users of taskset to pin on big/little clusters.

This is unfortunate, but this is also the case if the boot CPU is different.

Right, so don't change the boot CPU. In that vain, we also shouldn't change the CPU order in the .dts -- the current .dts is working for taskset and we shouldn't break people's scripts just because they want to use the PMU.

Quoted 5 lines
Quoted 3 lines

(2) It's not generally possible if, for example, the bootloader decides to boot Linux on a different CPU then we have no choice but to change the PMU interrupt order.

In that case _this_ patch is broken.

Why? I'm not denying that changing the boot CPU causes problems, I'm saying that you *can't* fix that by changing the CPU node order. You still have to change the interrupt order in that case, so why not just localise the changes there in the first place?

Quoted 3 lines

If we associate the interrupt with a CPU by node order, the relationship is preserved regardless of which CPU is the boot CPU (whether it was the bootloader's choice, kexec, or whatever).

Sure, and that requires code changes. If we're going to change the code, then I'd much rather we make the binding explicit, like I did in the follow-up patches to this one. As I mentioned before, this is a .dts fix to get things working with the current code. It's really too late to argue about the existing binding, even if it sucks.

Quoted 4 lines
Quoted 2 lines

(3) I didn't think that the ordering of CPU nodes was guaranteed to be preserved by dtc, whereas the order of the interrupts will be.

The order of nodes is presently preserved.

It's not about the present behaviour; I need a _guarantee_ that dtc/libfdt will *never* reorder CPU nodes. Today's working .dts file needs to continue to work with future tools.

Will

maintainer
#11 lore raw

PATCH 1/4 arm64: dts: fix PMU IRQ ordering for Juno

[Adding dtc folk]

Quoted 41 lines · Will Deacon
Quoted 40 lines · Mark Rutland
Quoted 38 lines · Will Deacon
Quoted 24 lines · Mark Rutland
Quoted 7 lines · Will Deacon
arch/arm64/boot/dts/arm/juno.dts +0 -0
index cb3073e4e7a8..4ed9287aaef1 100644
@@ -107,11 +107,11 @@
 	pmu {
 		compatible = "arm,armv8-pmuv3";

arch/arm64/boot/dts/arm/juno.dts line 115

Quoted 10 lines
arch/arm64/boot/dts/arm/juno.dts +3 -3
 		interrupts = <GIC_SPI 18 IRQ_TYPE_LEVEL_HIGH>,
+			     <GIC_SPI 02 IRQ_TYPE_LEVEL_HIGH>,
+			     <GIC_SPI 06 IRQ_TYPE_LEVEL_HIGH>,
 			     <GIC_SPI 22 IRQ_TYPE_LEVEL_HIGH>,
 			     <GIC_SPI 26 IRQ_TYPE_LEVEL_HIGH>,
-			     <GIC_SPI 30 IRQ_TYPE_LEVEL_HIGH>,
-			     <GIC_SPI 02 IRQ_TYPE_LEVEL_HIGH>,
-			     <GIC_SPI 06 IRQ_TYPE_LEVEL_HIGH>;
+			     <GIC_SPI 30 IRQ_TYPE_LEVEL_HIGH>;
 	};

I am very much not keen on this. While this may get things working today, it completely relies on Linux-internal details (the order of CPU bringup, which in this case is different from the order of entries in /cpus).

Quoted 8 lines

In all other dts that I am aware of, the order of entries in /cpus aligns with the order of interrupts in the PMU node, and the first entry is the boot CPU.

I think that we should ensure that the ordering of CPU nodes matches the order of interrupts here. That way we can fall back to that ordering (if not explicitly overridden), and even after an arbitrary logical renumbering (e.g. after a kexec) the relationship should stay intact.

There are a few problems with reordering the CPU nodes:

(1) It breaks any existing users of taskset to pin on big/little clusters.

Quoted 6 lines
Quoted 2 lines

This is unfortunate, but this is also the case if the boot CPU is different.

Right, so don't change the boot CPU. In that vain, we also shouldn't change the CPU order in the .dts -- the current .dts is working for taskset and we shouldn't break people's scripts just because they want to use the PMU.

I think this is an orthogonal discussion. If Linux is booted on a different CPU, it's not the fault of Linux that CPU0 is different.

Quoted 10 lines
Quoted 5 lines
Quoted 3 lines

(2) It's not generally possible if, for example, the bootloader decides to boot Linux on a different CPU then we have no choice but to change the PMU interrupt order.

In that case _this_ patch is broken.

Why? I'm not denying that changing the boot CPU causes problems, I'm saying that you *can't* fix that by changing the CPU node order. You still have to change the interrupt order in that case, so why not just localise the changes there in the first place?

If we're going to try to maintain support for these DTs long-term (with kexec and whatever logical renumbering can occur there), then we need a consistent invariant that we can rely on to associate interrupts and CPUs correctly.

The Linux logical ordering is not invariant, so we know that this _will_ break.

As far as I am aware, every other DT lists the boot CPU first, and the order of entries in /cpus mathes the logical order. Using the order of entries in /cpus will remain consistent in the face of arbitrary renumbering, and is (currently) consistent with logical numbering.

So keeping the CPU nodes and interrupt entries in the same order provides us with a long-term consistent order, regardless of which CPU is the boot CPU.

This DT is currently broken. If we're going to make it work we should do so in a manner that will continue to work. Anything else is a broken bodge that hurts us in the long-term as we'll have to hack around it.

Quoted 9 lines
Quoted 3 lines

If we associate the interrupt with a CPU by node order, the relationship is preserved regardless of which CPU is the boot CPU (whether it was the bootloader's choice, kexec, or whatever).

Sure, and that requires code changes. If we're going to change the code, then I'd much rather we make the binding explicit, like I did in the follow-up patches to this one. As I mentioned before, this is a .dts fix to get things working with the current code. It's really too late to argue about the existing binding, even if it sucks.

Sure, the binding sucks.

This DT has also _never_ worked.

If we're going to fix things, let's not introduce a middle step that's broken in a different way.

Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

(3) I didn't think that the ordering of CPU nodes was guaranteed to be preserved by dtc, whereas the order of the interrupts will be.

The order of nodes is presently preserved.

It's not about the present behaviour; I need a _guarantee_ that dtc/libfdt will *never* reorder CPU nodes. Today's working .dts file needs to continue to work with future tools.

Jon, David, Grant, thoughts?

Mark.

#12 lore raw

PATCH 1/4 arm64: dts: fix PMU IRQ ordering for Juno

Quoted 106 lines · Mark Rutland

[Adding dtc folk]

Quoted 41 lines · Will Deacon
Quoted 40 lines · Mark Rutland
Quoted 38 lines · Will Deacon
Quoted 24 lines · Mark Rutland
Quoted 7 lines · Will Deacon
arch/arm64/boot/dts/arm/juno.dts +0 -0
index cb3073e4e7a8..4ed9287aaef1 100644
@@ -107,11 +107,11 @@
 	pmu {
 		compatible = "arm,armv8-pmuv3";

arch/arm64/boot/dts/arm/juno.dts line 115

Quoted 10 lines
arch/arm64/boot/dts/arm/juno.dts +3 -3
 		interrupts = <GIC_SPI 18 IRQ_TYPE_LEVEL_HIGH>,
+			     <GIC_SPI 02 IRQ_TYPE_LEVEL_HIGH>,
+			     <GIC_SPI 06 IRQ_TYPE_LEVEL_HIGH>,
 			     <GIC_SPI 22 IRQ_TYPE_LEVEL_HIGH>,
 			     <GIC_SPI 26 IRQ_TYPE_LEVEL_HIGH>,
-			     <GIC_SPI 30 IRQ_TYPE_LEVEL_HIGH>,
-			     <GIC_SPI 02 IRQ_TYPE_LEVEL_HIGH>,
-			     <GIC_SPI 06 IRQ_TYPE_LEVEL_HIGH>;
+			     <GIC_SPI 30 IRQ_TYPE_LEVEL_HIGH>;
 	};

I am very much not keen on this. While this may get things working today, it completely relies on Linux-internal details (the order of CPU bringup, which in this case is different from the order of entries in /cpus).

Quoted 8 lines

In all other dts that I am aware of, the order of entries in /cpus aligns with the order of interrupts in the PMU node, and the first entry is the boot CPU.

I think that we should ensure that the ordering of CPU nodes matches the order of interrupts here. That way we can fall back to that ordering (if not explicitly overridden), and even after an arbitrary logical renumbering (e.g. after a kexec) the relationship should stay intact.

There are a few problems with reordering the CPU nodes:

(1) It breaks any existing users of taskset to pin on big/little clusters.

Quoted 6 lines
Quoted 2 lines

This is unfortunate, but this is also the case if the boot CPU is different.

Right, so don't change the boot CPU. In that vain, we also shouldn't change the CPU order in the .dts -- the current .dts is working for taskset and we shouldn't break people's scripts just because they want to use the PMU.

I think this is an orthogonal discussion. If Linux is booted on a different CPU, it's not the fault of Linux that CPU0 is different.

Quoted 10 lines
Quoted 5 lines
Quoted 3 lines

(2) It's not generally possible if, for example, the bootloader decides to boot Linux on a different CPU then we have no choice but to change the PMU interrupt order.

In that case _this_ patch is broken.

Why? I'm not denying that changing the boot CPU causes problems, I'm saying that you *can't* fix that by changing the CPU node order. You still have to change the interrupt order in that case, so why not just localise the changes there in the first place?

If we're going to try to maintain support for these DTs long-term (with kexec and whatever logical renumbering can occur there), then we need a consistent invariant that we can rely on to associate interrupts and CPUs correctly.

The Linux logical ordering is not invariant, so we know that this _will_ break.

As far as I am aware, every other DT lists the boot CPU first, and the order of entries in /cpus mathes the logical order. Using the order of entries in /cpus will remain consistent in the face of arbitrary renumbering, and is (currently) consistent with logical numbering.

So keeping the CPU nodes and interrupt entries in the same order provides us with a long-term consistent order, regardless of which CPU is the boot CPU.

This DT is currently broken. If we're going to make it work we should do so in a manner that will continue to work. Anything else is a broken bodge that hurts us in the long-term as we'll have to hack around it.

Quoted 9 lines
Quoted 3 lines

If we associate the interrupt with a CPU by node order, the relationship is preserved regardless of which CPU is the boot CPU (whether it was the bootloader's choice, kexec, or whatever).

Sure, and that requires code changes. If we're going to change the code, then I'd much rather we make the binding explicit, like I did in the follow-up patches to this one. As I mentioned before, this is a .dts fix to get things working with the current code. It's really too late to argue about the existing binding, even if it sucks.

Sure, the binding sucks.

This DT has also _never_ worked.

If we're going to fix things, let's not introduce a middle step that's broken in a different way.

Quoted 3 lines
Quoted 3 lines
Quoted 2 lines

(3) I didn't think that the ordering of CPU nodes was guaranteed to be preserved by dtc, whereas the order of the interrupts will be.

Quoted 7 lines
Quoted 5 lines
Quoted 1 line

The order of nodes is presently preserved.

It's not about the present behaviour; I need a _guarantee_ that dtc/libfdt will *never* reorder CPU nodes. Today's working .dts file needs to continue to work with future tools.

Jon, David, Grant, thoughts?

As a general rule, neither dtc nor libfdt will re-order any nodes unless you explicitly ask them to (e.g. dtc's "-s" option). That said, you should try not to rely on dt order.

Signature · 11 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 819 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20150205/4bcc5625/attachment.sig>
#13 lore raw

PATCH 1/4 arm64: dts: fix PMU IRQ ordering for Juno

Quoted 114 lines · David Gibson
Quoted 112 lines · Mark Rutland

[Adding dtc folk]

Quoted 41 lines · Will Deacon
Quoted 40 lines · Mark Rutland
Quoted 38 lines · Will Deacon
Quoted 24 lines · Mark Rutland
Quoted 7 lines · Will Deacon
arch/arm64/boot/dts/arm/juno.dts +0 -0
index cb3073e4e7a8..4ed9287aaef1 100644
@@ -107,11 +107,11 @@
 	pmu {
 		compatible = "arm,armv8-pmuv3";

arch/arm64/boot/dts/arm/juno.dts line 115

Quoted 10 lines
arch/arm64/boot/dts/arm/juno.dts +3 -3
 		interrupts = <GIC_SPI 18 IRQ_TYPE_LEVEL_HIGH>,
+			     <GIC_SPI 02 IRQ_TYPE_LEVEL_HIGH>,
+			     <GIC_SPI 06 IRQ_TYPE_LEVEL_HIGH>,
 			     <GIC_SPI 22 IRQ_TYPE_LEVEL_HIGH>,
 			     <GIC_SPI 26 IRQ_TYPE_LEVEL_HIGH>,
-			     <GIC_SPI 30 IRQ_TYPE_LEVEL_HIGH>,
-			     <GIC_SPI 02 IRQ_TYPE_LEVEL_HIGH>,
-			     <GIC_SPI 06 IRQ_TYPE_LEVEL_HIGH>;
+			     <GIC_SPI 30 IRQ_TYPE_LEVEL_HIGH>;
 	};

I am very much not keen on this. While this may get things working today, it completely relies on Linux-internal details (the order of CPU bringup, which in this case is different from the order of entries in /cpus).

Quoted 8 lines

In all other dts that I am aware of, the order of entries in /cpus aligns with the order of interrupts in the PMU node, and the first entry is the boot CPU.

I think that we should ensure that the ordering of CPU nodes matches the order of interrupts here. That way we can fall back to that ordering (if not explicitly overridden), and even after an arbitrary logical renumbering (e.g. after a kexec) the relationship should stay intact.

There are a few problems with reordering the CPU nodes:

(1) It breaks any existing users of taskset to pin on big/little clusters.

Quoted 6 lines
Quoted 2 lines

This is unfortunate, but this is also the case if the boot CPU is different.

Right, so don't change the boot CPU. In that vain, we also shouldn't change the CPU order in the .dts -- the current .dts is working for taskset and we shouldn't break people's scripts just because they want to use the PMU.

I think this is an orthogonal discussion. If Linux is booted on a different CPU, it's not the fault of Linux that CPU0 is different.

Quoted 10 lines
Quoted 5 lines
Quoted 3 lines

(2) It's not generally possible if, for example, the bootloader decides to boot Linux on a different CPU then we have no choice but to change the PMU interrupt order.

In that case _this_ patch is broken.

Why? I'm not denying that changing the boot CPU causes problems, I'm saying that you *can't* fix that by changing the CPU node order. You still have to change the interrupt order in that case, so why not just localise the changes there in the first place?

If we're going to try to maintain support for these DTs long-term (with kexec and whatever logical renumbering can occur there), then we need a consistent invariant that we can rely on to associate interrupts and CPUs correctly.

The Linux logical ordering is not invariant, so we know that this _will_ break.

As far as I am aware, every other DT lists the boot CPU first, and the order of entries in /cpus mathes the logical order. Using the order of entries in /cpus will remain consistent in the face of arbitrary renumbering, and is (currently) consistent with logical numbering.

So keeping the CPU nodes and interrupt entries in the same order provides us with a long-term consistent order, regardless of which CPU is the boot CPU.

This DT is currently broken. If we're going to make it work we should do so in a manner that will continue to work. Anything else is a broken bodge that hurts us in the long-term as we'll have to hack around it.

Quoted 9 lines
Quoted 3 lines

If we associate the interrupt with a CPU by node order, the relationship is preserved regardless of which CPU is the boot CPU (whether it was the bootloader's choice, kexec, or whatever).

Sure, and that requires code changes. If we're going to change the code, then I'd much rather we make the binding explicit, like I did in the follow-up patches to this one. As I mentioned before, this is a .dts fix to get things working with the current code. It's really too late to argue about the existing binding, even if it sucks.

Sure, the binding sucks.

This DT has also _never_ worked.

If we're going to fix things, let's not introduce a middle step that's broken in a different way.

Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

(3) I didn't think that the ordering of CPU nodes was guaranteed to be preserved by dtc, whereas the order of the interrupts will be.

The order of nodes is presently preserved.

It's not about the present behaviour; I need a _guarantee_ that dtc/libfdt will *never* reorder CPU nodes. Today's working .dts file needs to continue to work with future tools.

Quoted 5 lines
Quoted 1 line

Jon, David, Grant, thoughts?

As a general rule, neither dtc nor libfdt will re-order any nodes unless you explicitly ask them to (e.g. dtc's "-s" option). That said, you should try not to rely on dt order.

Hi guys,

As you explicitly solicited my opinion, I will tell you what I think.

First, I agree with David: DTC and libfdt are not gratuitously re-ordering the nodes within the tree. But nothing should rely on that behaviour either. It is a tree, and the children of a particular node are unordered. There *are* manipulation primitives that can cause a restructuring of the tree, and DTS consumers should be prepared to accomodate that.

Second, that Linux assumes an ordering on nodes from the tree is really unfortunate. I think we should try to remoce any such dependency or accomodate a more relaxed DTB read.

Finally, remember that the DTS is supposed describe the hardware. The hardware doesn't (*usually*) require an ordering on its components. In a true SMP, it is S -- symmetric -- and shouldn't rely on one Core being more first than another. That one is treated as a special entity is entirely a SW description. As such, it really shouldn't be represented as some tacit or hidden fact within the DTS.

HTH,
jdl

maintainer
#14 lore raw

PATCH 1/4 arm64: dts: fix PMU IRQ ordering for Juno

Quoted 133 lines · Jon Loeliger
Quoted 114 lines · David Gibson
Quoted 112 lines · Mark Rutland

[Adding dtc folk]

Quoted 41 lines · Will Deacon
Quoted 40 lines · Mark Rutland
Quoted 38 lines · Will Deacon
Quoted 24 lines · Mark Rutland
Quoted 7 lines · Will Deacon
arch/arm64/boot/dts/arm/juno.dts +0 -0
index cb3073e4e7a8..4ed9287aaef1 100644
@@ -107,11 +107,11 @@
 	pmu {
 		compatible = "arm,armv8-pmuv3";

arch/arm64/boot/dts/arm/juno.dts line 115

Quoted 10 lines
arch/arm64/boot/dts/arm/juno.dts +3 -3
 		interrupts = <GIC_SPI 18 IRQ_TYPE_LEVEL_HIGH>,
+			     <GIC_SPI 02 IRQ_TYPE_LEVEL_HIGH>,
+			     <GIC_SPI 06 IRQ_TYPE_LEVEL_HIGH>,
 			     <GIC_SPI 22 IRQ_TYPE_LEVEL_HIGH>,
 			     <GIC_SPI 26 IRQ_TYPE_LEVEL_HIGH>,
-			     <GIC_SPI 30 IRQ_TYPE_LEVEL_HIGH>,
-			     <GIC_SPI 02 IRQ_TYPE_LEVEL_HIGH>,
-			     <GIC_SPI 06 IRQ_TYPE_LEVEL_HIGH>;
+			     <GIC_SPI 30 IRQ_TYPE_LEVEL_HIGH>;
 	};

I am very much not keen on this. While this may get things working today, it completely relies on Linux-internal details (the order of CPU bringup, which in this case is different from the order of entries in /cpus).

Quoted 8 lines

In all other dts that I am aware of, the order of entries in /cpus aligns with the order of interrupts in the PMU node, and the first entry is the boot CPU.

I think that we should ensure that the ordering of CPU nodes matches the order of interrupts here. That way we can fall back to that ordering (if not explicitly overridden), and even after an arbitrary logical renumbering (e.g. after a kexec) the relationship should stay intact.

There are a few problems with reordering the CPU nodes:

(1) It breaks any existing users of taskset to pin on big/little clusters.

Quoted 6 lines
Quoted 2 lines

This is unfortunate, but this is also the case if the boot CPU is different.

Right, so don't change the boot CPU. In that vain, we also shouldn't change the CPU order in the .dts -- the current .dts is working for taskset and we shouldn't break people's scripts just because they want to use the PMU.

I think this is an orthogonal discussion. If Linux is booted on a different CPU, it's not the fault of Linux that CPU0 is different.

Quoted 10 lines
Quoted 5 lines
Quoted 3 lines

(2) It's not generally possible if, for example, the bootloader decides to boot Linux on a different CPU then we have no choice but to change the PMU interrupt order.

In that case _this_ patch is broken.

Why? I'm not denying that changing the boot CPU causes problems, I'm saying that you *can't* fix that by changing the CPU node order. You still have to change the interrupt order in that case, so why not just localise the changes there in the first place?

If we're going to try to maintain support for these DTs long-term (with kexec and whatever logical renumbering can occur there), then we need a consistent invariant that we can rely on to associate interrupts and CPUs correctly.

The Linux logical ordering is not invariant, so we know that this _will_ break.

As far as I am aware, every other DT lists the boot CPU first, and the order of entries in /cpus mathes the logical order. Using the order of entries in /cpus will remain consistent in the face of arbitrary renumbering, and is (currently) consistent with logical numbering.

So keeping the CPU nodes and interrupt entries in the same order provides us with a long-term consistent order, regardless of which CPU is the boot CPU.

This DT is currently broken. If we're going to make it work we should do so in a manner that will continue to work. Anything else is a broken bodge that hurts us in the long-term as we'll have to hack around it.

Quoted 9 lines
Quoted 3 lines

If we associate the interrupt with a CPU by node order, the relationship is preserved regardless of which CPU is the boot CPU (whether it was the bootloader's choice, kexec, or whatever).

Sure, and that requires code changes. If we're going to change the code, then I'd much rather we make the binding explicit, like I did in the follow-up patches to this one. As I mentioned before, this is a .dts fix to get things working with the current code. It's really too late to argue about the existing binding, even if it sucks.

Sure, the binding sucks.

This DT has also _never_ worked.

If we're going to fix things, let's not introduce a middle step that's broken in a different way.

Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

(3) I didn't think that the ordering of CPU nodes was guaranteed to be preserved by dtc, whereas the order of the interrupts will be.

The order of nodes is presently preserved.

It's not about the present behaviour; I need a _guarantee_ that dtc/libfdt will *never* reorder CPU nodes. Today's working .dts file needs to continue to work with future tools.

Quoted 5 lines
Quoted 1 line

Jon, David, Grant, thoughts?

As a general rule, neither dtc nor libfdt will re-order any nodes unless you explicitly ask them to (e.g. dtc's "-s" option). That said, you should try not to rely on dt order.

Hi guys,

As you explicitly solicited my opinion, I will tell you what I think.

First, I agree with David: DTC and libfdt are not gratuitously re-ordering the nodes within the tree. But nothing should rely on that behaviour either. It is a tree, and the children of a particular node are unordered. There *are* manipulation primitives that can cause a restructuring of the tree, and DTS consumers should be prepared to accomodate that.

Quoted 3 lines

Second, that Linux assumes an ordering on nodes from the tree is really unfortunate. I think we should try to remoce any such dependency or accomodate a more relaxed DTB read.

I agree. Will's later patch will do this by describing the relationship between interrupts and CPUs explciitly:

http://lists.infradead.org/pipermail/linux-arm-kernel/2015-January/319899.html

Quoted 7 lines

Finally, remember that the DTS is supposed describe the hardware. The hardware doesn't (*usually*) require an ordering on its components. In a true SMP, it is S -- symmetric -- and shouldn't rely on one Core being more first than another. That one is treated as a special entity is entirely a SW description. As such, it really shouldn't be represented as some tacit or hidden fact within the DTS.

I completely agree with this, which is why I'm opposed to relyiong on the order Linux probes CPUs (as this has no guaranteed relationship with the DTB).

While relying on the order of CPU nodes is not something I want to do, it's at least contained within the DTB and has some chance of functioning when Linux aribtrarily renumbers CPUs.

Does that make sense?

Thanks,
Mark.