Skip to contentSkip to conversation

Topology bindings / Perf for big.LITTLE systems

RFC PATCH New Mark Rutland 4 participants 22 messages Apr 11, 2013 – Apr 12, 2013

Subsystems (5)
lore
Share & tools
lore

Mark Rutland

author maintainer
#0 lore raw

cover Topology bindings / Perf for big.LITTLE systems

The following patches implement perf support for big.LITTLE systems (e.g. the A15x2 A7x3 coretile). The current implementation of one virtual PMU across all CPUs is modified to have one virtual PMU per set of compatible CPUs (e.g. one for the A15s, one for the A7s). Multiple compatible clusters may be combined in one virtual PMU.

The affinity of these virtual PMUs is determined by the affinity of interrupts for constituent physical PMUs. An interrupts-affinity property is added to the PMU devicetree binding to handle this, which refers to nodes in a topology description. When affinity information is not provided, affinity information is determined in the same way as the current implementation - interrupts are in order of (logical) cpu id, and each is affine to a single CPU.

The topology description is necessary due to the lax nature of MPIDR assignments in the wild, making them unsuitable in general for determining the hierarchical structure of CPUs and clusters.

I've had a couple of previous attempts at representing this information in devicetree [1,2], this series adds an example consumer (perf) and adds more general infrastructure rather than limiting the use of the binding to interrupts only.

Patches are based on v3.9-rc6.

Thanks,
Mark.

[1] http://lists.infradead.org/pipermail/linux-arm-kernel/2012-October/128205.html
[2] http://lists.infradead.org/pipermail/linux-arm-kernel/2012-December/137290.html

Lorenzo Pieralisi (1):
Documentation: DT: arm: define CPU topology bindings

Mark Rutland (10):
arm: add functions to parse cpu affinity from dt
arm: perf: clean up PMU names
arm: perf: use IDR types for CPU PMUs
arm: perf: make get_hw_events take arm_pmu
arm: perf: dynamically allocate cpu hardware data
arm: perf: treat PMUs as CPU affine
arm: perf: probe number of counters on affine CPUs
arm: perf: parse cpu affinity from dt
arm: perf: allow multiple CPU PMUs to be registered
arm: dts: add all PMUs for A15x2 A7x3 coretile

 Documentation/devicetree/bindings/arm/pmu.txt      |   7 +
 Documentation/devicetree/bindings/arm/topology.txt | 524 +++++++++++++++++++++
 arch/arm/boot/dts/vexpress-v2p-ca15_a7.dts         |  44 +-
 arch/arm/include/asm/dt_affinity.h                 |  12 +
 arch/arm/include/asm/pmu.h                         |   3 +-
 arch/arm/kernel/devtree.c                          | 139 ++++++
 arch/arm/kernel/perf_event.c                       |  32 +-
 arch/arm/kernel/perf_event_cpu.c                   | 270 ++++++++---
 arch/arm/kernel/perf_event_v6.c                    |  16 +-
 arch/arm/kernel/perf_event_v7.c                    |  55 ++-
 arch/arm/kernel/perf_event_xscale.c                |  20 +-
 arch/arm/oprofile/common.c                         |   8 +-
 12 files changed, 1004 insertions(+), 126 deletions(-)
 create mode 100644 Documentation/devicetree/bindings/arm/topology.txt
 create mode 100644 arch/arm/include/asm/dt_affinity.h
Signature · 1 line
1.8.1.1
author maintainer
#1 lore raw

RFC PATCH 1/11 Documentation: DT: arm: define CPU topology bindings

From: Lorenzo Pieralisi <[email protected]>

The advent of multi-cluster ARM systems requires a mechanism to describe how in hierarchical terms CPUs are connected in ARM SoCs so that the kernel can initialize and map resources like IRQs and memory space to specific group(s) of CPUs.

The CPU topology is made up of multiple hierarchy levels whose bottom layers (aka leaf nodes in device tree syntax) contain links to the HW CPUs in the system.

The topology bindings are generic for both 32-bit and 64-bit systems and lay the groundwork on top of which affinity schemes can be built.

This patch provides the documentation in the kernel required to define the device tree bindings describing the CPU topology for ARM 32-bit and 64-bit systems.

---
 Documentation/devicetree/bindings/arm/topology.txt | 524 +++++++++++++++++++++
 1 file changed, 524 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/arm/topology.txt
Documentation/devicetree/bindings/arm/topology.txt new file +524 -0
new file mode 100644
index 0000000..07c4961
@@ -0,0 +1,524 @@
+===========================================
+ARM topology binding description
+===========================================
+
+===========================================
+1 - Introduction
+===========================================
+
+In an ARM system, the hierarchy of CPUs is defined through three entities that
+are used to describe the layout of physical CPUs in the system:
+
+- cluster
+- core
+- thread
+
+The cpu nodes (bindings defined in [1]) represent the devices that
+correspond to physical CPUs and are to be mapped to the hierarchy levels.
+
+The bottom hierarchy level sits at core or thread level depending on whether
+symmetric multi-threading (SMT) is supported or not.
+
+For instance in a system where CPUs support SMT, "cpu" nodes represent all
+threads existing in the system and map to the hierarchy level "thread" above.
+In systems where SMT is not supported "cpu" nodes represent all cores present
+in the system and map to the hierarchy level "core" above.
+
+ARM topology bindings allow one to associate cpu nodes with hierarchical groups
+corresponding to the system hierarchy; syntactically they are defined as device
+tree nodes.
+
+The remainder of this document provides the topology bindings for ARM, based
+on the ePAPR standard, available from:
+
+http://devicetree.org
+
+If not stated otherwise, whenever a reference to a cpu node phandle is made its
+value must point to a cpu node compliant with the cpu node bindings as
+documented in [1].
+A topology description containing phandles to cpu nodes that are not compliant
+with bindings standardized in [1] is therefore considered invalid.
+
+===========================================
+2 - cpu-map node
+===========================================
+
+The ARM CPU topology is defined within a container node, sitting at the top
+level of the device tree (/), the cpu-map node.
+
+- cpu-map node
+
+	Usage: Required to define ARM CPU topology
+
+	Description: The cpu-map node is just a container node where its
+		     subnodes describe the CPU topology
+
+	Node name must be "cpu-map".
+
+	A cpu-map node's child nodes can be:
+
+	- one or more cluster nodes
+
+	Any other configuration is considered invalid.
+
+The cpu-map node can only contain three types of child nodes:
+
+- cluster node
+- core node
+- thread node
+
+whose bindings are described in paragraph 3.
+
+The nodes describing the CPU topology (cluster/core/thread) can only be
+defined within the cpu-map node.
+Any other configuration is consider invalid and therefore must be ignored.
+
+===========================================
+2.1 - cpu-map child nodes naming convention
+===========================================
+
+cpu-map child nodes must follow a naming convention where the node name
+must be "clusterN", "coreN", "threadN" depending on the node type (ie
+cluster/core/thread) (where N = {0, 1, ...} is the node number; nodes which
+are siblings within a single common parent node must be given a unique and
+sequential N value, starting from 0).
+cpu-map child nodes which do not share a common parent node can have the same
+name (ie same number N as other cpu-map child nodes at different device tree
+levels) since name uniqueness will be guaranteed by the device tree hierarchy.
+
+===========================================
+3 - cluster/core/thread node bindings
+===========================================
+
+Bindings for cluster/cpu/thread nodes are defined as follows:
+
+- cluster node
+
+	 Description: must be declared within a cpu-map node, one node
+		      per cluster. A system can contain several layers of
+		      clustering and cluster nodes can be contained in parent
+		      cluster nodes.
+
+	The cluster node name must be "clusterN" as described in 2.1 above.
+	A cluster node can not be a leaf node.
+
+	A cluster node's child nodes must be:
+
+	- one or more cluster nodes; or
+	- one or more core nodes
+
+	Any other configuration is considered invalid.
+
+- core node
+
+	Description: must be declared in a cluster node, one node per core in
+		     the cluster. If the system does not support SMT, core
+		     nodes are leaf nodes, otherwise they become containers of
+		     thread nodes.
+
+	The core node name must be "coreN" as described in 2.1 above.
+
+	A core node must be a leaf node if SMT is not supported.
+
+	Properties for core nodes that are leaf nodes:
+
+	- cpu
+		Usage: required
+		Value type: <phandle>
+		Definition: a phandle to the cpu node that corresponds to the
+			    core node.
+
+	If a core node is not a leaf node (CPUs supporting SMT) a core node's
+	child nodes can be:
+
+	- one or more thread nodes
+
+	Any other configuration is considered invalid.
+
+- thread node
+
+	Description: must be declared in a core node, one node per thread
+		     in the core if the system supports SMT. Thread nodes are
+		     always leaf nodes in the device tree.
+
+	The thread node name must be "threadN" as described in 2.1 above.
+
+	A thread node must be a leaf node.
+
+	A thread node must contain the following property:
+
+	- cpu
+		Usage: required
+		Value type: <phandle>
+		Definition: a phandle to the cpu node that corresponds to
+			    the thread node.
+
+===========================================
+4 - Example dts
+===========================================
+
+Example 1 (ARM 64-bit, 16-cpu system, two clusters of clusters):
+
+cpu-map {
+
+	cluster0 {
+
+		cluster0 {
+
+			core0 {
+
+				thread0 {
+					cpu = <&CPU0>;
+				};
+
+				thread1 {
+					cpu = <&CPU1>;
+				};
+			};
+
+			core1 {
+
+				thread0 {
+					cpu = <&CPU2>;
+				};
+
+				thread1 {
+					cpu = <&CPU3>;
+				};
+			};
+		};
+
+		cluster1 {
+
+			core0 {
+
+				thread0 {
+					cpu = <&CPU4>;
+				};
+
+				thread1 {
+					cpu = <&CPU5>;
+				};
+			};
+
+			core1 {
+
+				thread0 {
+					cpu = <&CPU6>;
+				};
+
+				thread1 {
+					cpu = <&CPU7>;
+				};
+			};
+		};
+	};
+
+	cluster1 {
+
+		cluster0 {
+
+			core0 {
+
+				thread0 {
+					cpu = <&CPU8>;
+				};
+
+				thread1 {
+					cpu = <&CPU9>;
+				};
+			};
+
+			core1 {
+
+				thread0 {
+					cpu = <&CPU10>;
+				};
+
+				thread1 {
+					cpu = <&CPU11>;
+				};
+			};
+		};
+
+		cluster1 {
+
+			core0 {
+
+				thread0 {
+					cpu = <&CPU12>;
+				};
+
+				thread1 {
+					cpu = <&CPU13>;
+				};
+			};
+
+			core1 {
+
+				thread0 {
+					cpu = <&CPU14>;
+				};
+
+				thread1 {
+					cpu = <&CPU15>;
+				};
+			};
+		};
+	};
+};
+
+cpus {
+	#size-cells = <0>;
+	#address-cells = <2>;
+
+	CPU0: cpu at 0 {
+		device_type = "cpu";
+		compatible = "arm,cortex-a57";
+		reg = <0x0 0x0>;
+		enable-method = "spin-table";
+		cpu-release-addr = <0 0x20000000>;
+	};
+
+	CPU1: cpu at 1 {
+		device_type = "cpu";
+		compatible = "arm,cortex-a57";
+		reg = <0x0 0x1>;
+		enable-method = "spin-table";
+		cpu-release-addr = <0 0x20000000>;
+	};
+
+	CPU2: cpu at 100 {
+		device_type = "cpu";
+		compatible = "arm,cortex-a57";
+		reg = <0x0 0x100>;
+		enable-method = "spin-table";
+		cpu-release-addr = <0 0x20000000>;
+	};
+
+	CPU3: cpu at 101 {
+		device_type = "cpu";
+		compatible = "arm,cortex-a57";
+		reg = <0x0 0x101>;
+		enable-method = "spin-table";
+		cpu-release-addr = <0 0x20000000>;
+	};
+
+	CPU4: cpu at 10000 {
+		device_type = "cpu";
+		compatible = "arm,cortex-a57";
+		reg = <0x0 0x10000>;
+		enable-method = "spin-table";
+		cpu-release-addr = <0 0x20000000>;
+	};
+
+	CPU5: cpu at 10001 {
+		device_type = "cpu";
+		compatible = "arm,cortex-a57";
+		reg = <0x0 0x10001>;
+		enable-method = "spin-table";
+		cpu-release-addr = <0 0x20000000>;
+	};
+
+	CPU6: cpu at 10100 {
+		device_type = "cpu";
+		compatible = "arm,cortex-a57";
+		reg = <0x0 0x10100>;
+		enable-method = "spin-table";
+		cpu-release-addr = <0 0x20000000>;
+	};
+
+	CPU7: cpu at 10101 {
+		device_type = "cpu";
+		compatible = "arm,cortex-a57";
+		reg = <0x0 0x10101>;
+		enable-method = "spin-table";
+		cpu-release-addr = <0 0x20000000>;
+	};
+
+	CPU8: cpu at 100000000 {
+		device_type = "cpu";
+		compatible = "arm,cortex-a57";
+		reg = <0x1 0x0>;
+		enable-method = "spin-table";
+		cpu-release-addr = <0 0x20000000>;
+	};
+
+	CPU9: cpu at 100000001 {
+		device_type = "cpu";
+		compatible = "arm,cortex-a57";
+		reg = <0x1 0x1>;
+		enable-method = "spin-table";
+		cpu-release-addr = <0 0x20000000>;
+	};
+
+	CPU10: cpu at 100000100 {
+		device_type = "cpu";
+		compatible = "arm,cortex-a57";
+		reg = <0x1 0x100>;
+		enable-method = "spin-table";
+		cpu-release-addr = <0 0x20000000>;
+	};
+
+	CPU11: cpu at 100000101 {
+		device_type = "cpu";
+		compatible = "arm,cortex-a57";
+		reg = <0x1 0x101>;
+		enable-method = "spin-table";
+		cpu-release-addr = <0 0x20000000>;
+	};
+
+	CPU12: cpu at 100010000 {
+		device_type = "cpu";
+		compatible = "arm,cortex-a57";
+		reg = <0x1 0x10000>;
+		enable-method = "spin-table";
+		cpu-release-addr = <0 0x20000000>;
+	};
+
+	CPU13: cpu at 100010001 {
+		device_type = "cpu";
+		compatible = "arm,cortex-a57";
+		reg = <0x1 0x10001>;
+		enable-method = "spin-table";
+		cpu-release-addr = <0 0x20000000>;
+	};
+
+	CPU14: cpu at 100010100 {
+		device_type = "cpu";
+		compatible = "arm,cortex-a57";
+		reg = <0x1 0x10100>;
+		enable-method = "spin-table";
+		cpu-release-addr = <0 0x20000000>;
+	};
+
+	CPU15: cpu at 100010101 {
+		device_type = "cpu";
+		compatible = "arm,cortex-a57";
+		reg = <0x1 0x10101>;
+		enable-method = "spin-table";
+		cpu-release-addr = <0 0x20000000>;
+	};
+};
+
+Example 2 (ARM 32-bit, dual-cluster, 8-cpu system, no SMT):
+
+cpu-map {
+
+	cluster0 {
+
+		core0 {
+			cpu = <&CPU0>;
+		};
+
+		core1 {
+			cpu = <&CPU1>;
+		};
+
+		core2 {
+			cpu = <&CPU2>;
+		};
+
+		core3 {
+			cpu = <&CPU3>;
+		};
+
+	};
+
+	cluster1 {
+
+		core0 {
+			cpu = <&CPU4>;
+		};
+
+		core1 {
+			cpu = <&CPU5>;
+		};
+
+		core2 {
+			cpu = <&CPU6>;
+		};
+
+		core3 {
+			cpu = <&CPU7>;
+		};
+
+	};
+
+cpus {
+	#size-cells = <0>;
+	#address-cells = <1>;
+
+	CPU0: cpu at 0 {
+		device_type = "cpu";
+		compatible = "arm,cortex-a15";
+		reg = <0x0>;
+	};
+
+	CPU1: cpu at 1 {
+		device_type = "cpu";
+		compatible = "arm,cortex-a15";
+		reg = <0x1>;
+	};
+
+	CPU2: cpu at 2 {
+		device_type = "cpu";
+		compatible = "arm,cortex-a15";
+		reg = <0x2>;
+	};
+
+	CPU3: cpu at 3 {
+		device_type = "cpu";
+		compatible = "arm,cortex-a15";
+		reg = <0x3>;
+	};
+
+	CPU4: cpu at 100 {
+		device_type = "cpu";
+		compatible = "arm,cortex-a7";
+		reg = <0x100>;
+	};
+
+	CPU5: cpu at 101 {
+		device_type = "cpu";
+		compatible = "arm,cortex-a7";
+		reg = <0x101>;
+	};
+
+	CPU6: cpu at 102 {
+		device_type = "cpu";
+		compatible = "arm,cortex-a7";
+		reg = <0x102>;
+	};
+
+	CPU7: cpu at 103 {
+		device_type = "cpu";
+		compatible = "arm,cortex-a7";
+		reg = <0x103>;
+	};
+};
+
+Example 3 (ARM 32-bit, cortex-a8 single core):
+
+cpu-map {
+	cluster0 {
+		core0 {
+			cpu = <&CPU0>;
+		};
+	};
+};
+
+cpus {
+	#size-cells = <0>;
+	#address-cells = <1>;
+
+	CPU0: cpu at 0 {
+		device_type = "cpu";
+		compatible = "arm,cortex-a8";
+		reg = <0x0>;
+	};
+};
+
+===============================================================================
+[1] ARM Linux kernel documentation
+    Documentation/devicetree/bindings/arm/cpus.txt
Signature · 1 line
1.8.1.1
#2 lore raw

RFC PATCH 1/11 Documentation: DT: arm: define CPU topology bindings

Quoted 11 lines · Mark Rutland

From: Lorenzo Pieralisi <[email protected]>

The advent of multi-cluster ARM systems requires a mechanism to describe how in hierarchical terms CPUs are connected in ARM SoCs so that the kernel can initialize and map resources like IRQs and memory space to specific group(s) of CPUs.

The CPU topology is made up of multiple hierarchy levels whose bottom layers (aka leaf nodes in device tree syntax) contain links to the HW CPUs in the system.

Quoted 6 lines

The topology bindings are generic for both 32-bit and 64-bit systems and lay the groundwork on top of which affinity schemes can be built.

This patch provides the documentation in the kernel required to define the device tree bindings describing the CPU topology for ARM 32-bit and 64-bit systems.

I'm now very weary of continued /cpu changes after the pain of making the reg property reflect the mpidr value in 3.8.

Quoted 72 lines
---
 Documentation/devicetree/bindings/arm/topology.txt | 524 +++++++++++++++++++++
 1 file changed, 524 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/arm/topology.txt
Documentation/devicetree/bindings/arm/topology.txt new file +59 -0
new file mode 100644
index 0000000..07c4961
@@ -0,0 +1,524 @@
+===========================================
+ARM topology binding description
+===========================================
+
+===========================================
+1 - Introduction
+===========================================
+
+In an ARM system, the hierarchy of CPUs is defined through three entities that
+are used to describe the layout of physical CPUs in the system:
+
+- cluster
+- core
+- thread
+
+The cpu nodes (bindings defined in [1]) represent the devices that
+correspond to physical CPUs and are to be mapped to the hierarchy levels.
+
+The bottom hierarchy level sits at core or thread level depending on whether
+symmetric multi-threading (SMT) is supported or not.
+
+For instance in a system where CPUs support SMT, "cpu" nodes represent all
+threads existing in the system and map to the hierarchy level "thread" above.
+In systems where SMT is not supported "cpu" nodes represent all cores present
+in the system and map to the hierarchy level "core" above.
+
+ARM topology bindings allow one to associate cpu nodes with hierarchical groups
+corresponding to the system hierarchy; syntactically they are defined as device
+tree nodes.
+
+The remainder of this document provides the topology bindings for ARM, based
+on the ePAPR standard, available from:
+
+http://devicetree.org
+
+If not stated otherwise, whenever a reference to a cpu node phandle is made its
+value must point to a cpu node compliant with the cpu node bindings as
+documented in [1].
+A topology description containing phandles to cpu nodes that are not compliant
+with bindings standardized in [1] is therefore considered invalid.
+
+===========================================
+2 - cpu-map node
+===========================================
+
+The ARM CPU topology is defined within a container node, sitting at the top
+level of the device tree (/), the cpu-map node.
+
+- cpu-map node
+
+	Usage: Required to define ARM CPU topology
+
+	Description: The cpu-map node is just a container node where its
+		     subnodes describe the CPU topology
+
+	Node name must be "cpu-map".
+
+	A cpu-map node's child nodes can be:
+

Documentation/devicetree/bindings/arm/topology.txt line 69

Quoted 10 lines
Documentation/devicetree/bindings/arm/topology.txt new file +10 -0
+	- one or more cluster nodes
+
+	Any other configuration is considered invalid.
+
+The cpu-map node can only contain three types of child nodes:
+
+- cluster node
+- core node
+- thread node
+

Why not put the topology in the /cpus nodes? I don't really see the point of having a flat list of cpus and separate topology info. There is some compatibility issue, but adding optional levels for clusters can be handled.

Quoted 24 lines
Documentation/devicetree/bindings/arm/topology.txt new file +24 -0
+whose bindings are described in paragraph 3.
+
+The nodes describing the CPU topology (cluster/core/thread) can only be
+defined within the cpu-map node.
+Any other configuration is consider invalid and therefore must be ignored.
+
+===========================================
+2.1 - cpu-map child nodes naming convention
+===========================================
+
+cpu-map child nodes must follow a naming convention where the node name
+must be "clusterN", "coreN", "threadN" depending on the node type (ie
+cluster/core/thread) (where N = {0, 1, ...} is the node number; nodes which
+are siblings within a single common parent node must be given a unique and
+sequential N value, starting from 0).
+cpu-map child nodes which do not share a common parent node can have the same
+name (ie same number N as other cpu-map child nodes at different device tree
+levels) since name uniqueness will be guaranteed by the device tree hierarchy.
+
+===========================================
+3 - cluster/core/thread node bindings
+===========================================
+
+Bindings for cluster/cpu/thread nodes are defined as follows:

Documentation/devicetree/bindings/arm/topology.txt line 103

Quoted 10 lines
Documentation/devicetree/bindings/arm/topology.txt new file +10 -0
+
+- cluster node
+
+	 Description: must be declared within a cpu-map node, one node
+		      per cluster. A system can contain several layers of
+		      clustering and cluster nodes can be contained in parent
+		      cluster nodes.
+
+	The cluster node name must be "clusterN" as described in 2.1 above.
+	A cluster node can not be a leaf node.

Follow standard conventions with "cluster at N" and a reg property with the number.

Quoted 41 lines
Documentation/devicetree/bindings/arm/topology.txt new file +41 -0
+
+	A cluster node's child nodes must be:
+
+	- one or more cluster nodes; or
+	- one or more core nodes
+
+	Any other configuration is considered invalid.
+
+- core node
+
+	Description: must be declared in a cluster node, one node per core in
+		     the cluster. If the system does not support SMT, core
+		     nodes are leaf nodes, otherwise they become containers of
+		     thread nodes.
+
+	The core node name must be "coreN" as described in 2.1 above.
+
+	A core node must be a leaf node if SMT is not supported.
+
+	Properties for core nodes that are leaf nodes:
+
+	- cpu
+		Usage: required
+		Value type: <phandle>
+		Definition: a phandle to the cpu node that corresponds to the
+			    core node.
+
+	If a core node is not a leaf node (CPUs supporting SMT) a core node's
+	child nodes can be:
+
+	- one or more thread nodes
+
+	Any other configuration is considered invalid.
+
+- thread node
+
+	Description: must be declared in a core node, one node per thread
+		     in the core if the system supports SMT. Thread nodes are
+		     always leaf nodes in the device tree.
+
+	The thread node name must be "threadN" as described in 2.1 above.

Documentation/devicetree/bindings/arm/topology.txt line 154

Quoted 10 lines
Documentation/devicetree/bindings/arm/topology.txt new file +10 -0
+
+	A thread node must be a leaf node.
+
+	A thread node must contain the following property:
+
+	- cpu
+		Usage: required
+		Value type: <phandle>
+		Definition: a phandle to the cpu node that corresponds to
+			    the thread node.

According to the ePAPR, threads are represented by an array of ids for reg property, not another cpu node. Why the deviation.

Rob

#3 lore raw

RFC PATCH 1/11 Documentation: DT: arm: define CPU topology bindings

Quoted 15 lines · Rob Herring
Quoted 14 lines · Mark Rutland

From: Lorenzo Pieralisi <[email protected]>

The advent of multi-cluster ARM systems requires a mechanism to describe how in hierarchical terms CPUs are connected in ARM SoCs so that the kernel can initialize and map resources like IRQs and memory space to specific group(s) of CPUs.

The CPU topology is made up of multiple hierarchy levels whose bottom layers (aka leaf nodes in device tree syntax) contain links to the HW CPUs in the system.

The topology bindings are generic for both 32-bit and 64-bit systems and lay the groundwork on top of which affinity schemes can be built.

Quoted 6 lines
Quoted 3 lines

This patch provides the documentation in the kernel required to define the device tree bindings describing the CPU topology for ARM 32-bit and 64-bit systems.

I'm now very weary of continued /cpu changes after the pain of making the reg property reflect the mpidr value in 3.8.

We won't change the reg property value, the code I am about to post provides stricter bindings, stricter semantics and extends bindings to cater for arm 64-bit systems.

Quoted 79 lines
Quoted 79 lines
---
 Documentation/devicetree/bindings/arm/topology.txt | 524 +++++++++++++++++++++
 1 file changed, 524 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/arm/topology.txt
Documentation/devicetree/bindings/arm/topology.txt new file +66 -0
new file mode 100644
index 0000000..07c4961
@@ -0,0 +1,524 @@
+===========================================
+ARM topology binding description
+===========================================
+
+===========================================
+1 - Introduction
+===========================================
+
+In an ARM system, the hierarchy of CPUs is defined through three entities that
+are used to describe the layout of physical CPUs in the system:
+
+- cluster
+- core
+- thread
+
+The cpu nodes (bindings defined in [1]) represent the devices that
+correspond to physical CPUs and are to be mapped to the hierarchy levels.
+
+The bottom hierarchy level sits at core or thread level depending on whether
+symmetric multi-threading (SMT) is supported or not.
+
+For instance in a system where CPUs support SMT, "cpu" nodes represent all
+threads existing in the system and map to the hierarchy level "thread" above.
+In systems where SMT is not supported "cpu" nodes represent all cores present
+in the system and map to the hierarchy level "core" above.
+
+ARM topology bindings allow one to associate cpu nodes with hierarchical groups
+corresponding to the system hierarchy; syntactically they are defined as device
+tree nodes.
+
+The remainder of this document provides the topology bindings for ARM, based
+on the ePAPR standard, available from:
+
+http://devicetree.org
+
+If not stated otherwise, whenever a reference to a cpu node phandle is made its
+value must point to a cpu node compliant with the cpu node bindings as
+documented in [1].
+A topology description containing phandles to cpu nodes that are not compliant
+with bindings standardized in [1] is therefore considered invalid.
+
+===========================================
+2 - cpu-map node
+===========================================
+
+The ARM CPU topology is defined within a container node, sitting at the top
+level of the device tree (/), the cpu-map node.
+
+- cpu-map node
+
+	Usage: Required to define ARM CPU topology
+
+	Description: The cpu-map node is just a container node where its
+		     subnodes describe the CPU topology
+
+	Node name must be "cpu-map".
+
+	A cpu-map node's child nodes can be:
+
+	- one or more cluster nodes
+
+	Any other configuration is considered invalid.
+
+The cpu-map node can only contain three types of child nodes:
+
+- cluster node
Quoted 8 lines

Documentation/devicetree/bindings/arm/topology.txt line 69

Quoted 3 lines
Documentation/devicetree/bindings/arm/topology.txt new file +3 -0
+- core node
+- thread node
+

Why not put the topology in the /cpus nodes? I don't really see the point of having a flat list of cpus and separate topology info. There is some compatibility issue, but adding optional levels for clusters can be handled.

I thought this would break all code relying on /cpu nodes being /cpus node's children. Furthermore, I was told that the /cpus node can only have /cpu nodes as children.

If you wish so, we can define the topology in the /cpus node, fine by me.

Quoted 29 lines
Quoted 29 lines
Documentation/devicetree/bindings/arm/topology.txt new file +29 -0
+whose bindings are described in paragraph 3.
+
+The nodes describing the CPU topology (cluster/core/thread) can only be
+defined within the cpu-map node.
+Any other configuration is consider invalid and therefore must be ignored.
+
+===========================================
+2.1 - cpu-map child nodes naming convention
+===========================================
+
+cpu-map child nodes must follow a naming convention where the node name
+must be "clusterN", "coreN", "threadN" depending on the node type (ie
+cluster/core/thread) (where N = {0, 1, ...} is the node number; nodes which
+are siblings within a single common parent node must be given a unique and
+sequential N value, starting from 0).
+cpu-map child nodes which do not share a common parent node can have the same
+name (ie same number N as other cpu-map child nodes at different device tree
+levels) since name uniqueness will be guaranteed by the device tree hierarchy.
+
+===========================================
+3 - cluster/core/thread node bindings
+===========================================
+
+Bindings for cluster/cpu/thread nodes are defined as follows:
+
+- cluster node
+
+	 Description: must be declared within a cpu-map node, one node
+		      per cluster. A system can contain several layers of
Quoted 8 lines

Documentation/devicetree/bindings/arm/topology.txt line 103

Quoted 5 lines
Documentation/devicetree/bindings/arm/topology.txt new file +5 -0
+		      clustering and cluster nodes can be contained in parent
+		      cluster nodes.
+
+	The cluster node name must be "clusterN" as described in 2.1 above.
+	A cluster node can not be a leaf node.

Follow standard conventions with "cluster at N" and a reg property with the number.

We are defining the topology to decouple the cluster/core/thread concept from the MPIDR. Having a reg property in the cluster (and core) nodes would complicate things if that reg property must correspond to an MPIDR bitfield. If it is meant to be just an enumeration at a given device tree level, I am ok with changing that.

Quoted 47 lines
Quoted 47 lines
Documentation/devicetree/bindings/arm/topology.txt new file +47 -0
+
+	A cluster node's child nodes must be:
+
+	- one or more cluster nodes; or
+	- one or more core nodes
+
+	Any other configuration is considered invalid.
+
+- core node
+
+	Description: must be declared in a cluster node, one node per core in
+		     the cluster. If the system does not support SMT, core
+		     nodes are leaf nodes, otherwise they become containers of
+		     thread nodes.
+
+	The core node name must be "coreN" as described in 2.1 above.
+
+	A core node must be a leaf node if SMT is not supported.
+
+	Properties for core nodes that are leaf nodes:
+
+	- cpu
+		Usage: required
+		Value type: <phandle>
+		Definition: a phandle to the cpu node that corresponds to the
+			    core node.
+
+	If a core node is not a leaf node (CPUs supporting SMT) a core node's
+	child nodes can be:
+
+	- one or more thread nodes
+
+	Any other configuration is considered invalid.
+
+- thread node
+
+	Description: must be declared in a core node, one node per thread
+		     in the core if the system supports SMT. Thread nodes are
+		     always leaf nodes in the device tree.
+
+	The thread node name must be "threadN" as described in 2.1 above.
+
+	A thread node must be a leaf node.
+
+	A thread node must contain the following property:
+
+	- cpu
Quoted 8 lines

Documentation/devicetree/bindings/arm/topology.txt line 154

Quoted 4 lines
Documentation/devicetree/bindings/arm/topology.txt new file +4 -0
+		Usage: required
+		Value type: <phandle>
+		Definition: a phandle to the cpu node that corresponds to
+			    the thread node.

According to the ePAPR, threads are represented by an array of ids for reg property, not another cpu node. Why the deviation.

It is not a cpu node, it is a phandle property named cpu. Can you point me to the ePAPR section where threads bindings are described please ? I have not managed to find these details, I am reading version 1.0.

Thanks a lot for having a look !
Lorenzo

#4 lore raw

RFC PATCH 1/11 Documentation: DT: arm: define CPU topology bindings

Quoted 115 lines · Lorenzo Pieralisi
Quoted 15 lines · Rob Herring
Quoted 14 lines · Mark Rutland

From: Lorenzo Pieralisi <[email protected]>

The advent of multi-cluster ARM systems requires a mechanism to describe how in hierarchical terms CPUs are connected in ARM SoCs so that the kernel can initialize and map resources like IRQs and memory space to specific group(s) of CPUs.

The CPU topology is made up of multiple hierarchy levels whose bottom layers (aka leaf nodes in device tree syntax) contain links to the HW CPUs in the system.

The topology bindings are generic for both 32-bit and 64-bit systems and lay the groundwork on top of which affinity schemes can be built.

Quoted 6 lines
Quoted 3 lines

This patch provides the documentation in the kernel required to define the device tree bindings describing the CPU topology for ARM 32-bit and 64-bit systems.

I'm now very weary of continued /cpu changes after the pain of making the reg property reflect the mpidr value in 3.8.

We won't change the reg property value, the code I am about to post provides stricter bindings, stricter semantics and extends bindings to cater for arm 64-bit systems.

Quoted 79 lines
Quoted 79 lines
---
 Documentation/devicetree/bindings/arm/topology.txt | 524 +++++++++++++++++++++
 1 file changed, 524 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/arm/topology.txt
Documentation/devicetree/bindings/arm/topology.txt new file +66 -0
new file mode 100644
index 0000000..07c4961
@@ -0,0 +1,524 @@
+===========================================
+ARM topology binding description
+===========================================
+
+===========================================
+1 - Introduction
+===========================================
+
+In an ARM system, the hierarchy of CPUs is defined through three entities that
+are used to describe the layout of physical CPUs in the system:
+
+- cluster
+- core
+- thread
+
+The cpu nodes (bindings defined in [1]) represent the devices that
+correspond to physical CPUs and are to be mapped to the hierarchy levels.
+
+The bottom hierarchy level sits at core or thread level depending on whether
+symmetric multi-threading (SMT) is supported or not.
+
+For instance in a system where CPUs support SMT, "cpu" nodes represent all
+threads existing in the system and map to the hierarchy level "thread" above.
+In systems where SMT is not supported "cpu" nodes represent all cores present
+in the system and map to the hierarchy level "core" above.
+
+ARM topology bindings allow one to associate cpu nodes with hierarchical groups
+corresponding to the system hierarchy; syntactically they are defined as device
+tree nodes.
+
+The remainder of this document provides the topology bindings for ARM, based
+on the ePAPR standard, available from:
+
+http://devicetree.org
+
+If not stated otherwise, whenever a reference to a cpu node phandle is made its
+value must point to a cpu node compliant with the cpu node bindings as
+documented in [1].
+A topology description containing phandles to cpu nodes that are not compliant
+with bindings standardized in [1] is therefore considered invalid.
+
+===========================================
+2 - cpu-map node
+===========================================
+
+The ARM CPU topology is defined within a container node, sitting at the top
+level of the device tree (/), the cpu-map node.
+
+- cpu-map node
+
+	Usage: Required to define ARM CPU topology
+
+	Description: The cpu-map node is just a container node where its
+		     subnodes describe the CPU topology
+
+	Node name must be "cpu-map".
+
+	A cpu-map node's child nodes can be:
+
+	- one or more cluster nodes
+
+	Any other configuration is considered invalid.
+
+The cpu-map node can only contain three types of child nodes:
+
+- cluster node
Quoted 8 lines

Documentation/devicetree/bindings/arm/topology.txt line 69

Quoted 3 lines
Documentation/devicetree/bindings/arm/topology.txt new file +3 -0
+- core node
+- thread node
+

Why not put the topology in the /cpus nodes? I don't really see the point of having a flat list of cpus and separate topology info. There is some compatibility issue, but adding optional levels for clusters can be handled.

Quoted 3 lines

I thought this would break all code relying on /cpu nodes being /cpus node's children. Furthermore, I was told that the /cpus node can only have /cpu nodes as children.

IIRC the context, that was in regards to putting things like the PMUs under the /cpus node. Or are you referring to something else? I think this situation is a bit different.

You will have to support existing single cluster systems without the hierarchy.

Quoted 41 lines

If you wish so, we can define the topology in the /cpus node, fine by me.

Quoted 29 lines
Quoted 29 lines
Documentation/devicetree/bindings/arm/topology.txt new file +29 -0
+whose bindings are described in paragraph 3.
+
+The nodes describing the CPU topology (cluster/core/thread) can only be
+defined within the cpu-map node.
+Any other configuration is consider invalid and therefore must be ignored.
+
+===========================================
+2.1 - cpu-map child nodes naming convention
+===========================================
+
+cpu-map child nodes must follow a naming convention where the node name
+must be "clusterN", "coreN", "threadN" depending on the node type (ie
+cluster/core/thread) (where N = {0, 1, ...} is the node number; nodes which
+are siblings within a single common parent node must be given a unique and
+sequential N value, starting from 0).
+cpu-map child nodes which do not share a common parent node can have the same
+name (ie same number N as other cpu-map child nodes at different device tree
+levels) since name uniqueness will be guaranteed by the device tree hierarchy.
+
+===========================================
+3 - cluster/core/thread node bindings
+===========================================
+
+Bindings for cluster/cpu/thread nodes are defined as follows:
+
+- cluster node
+
+	 Description: must be declared within a cpu-map node, one node
+		      per cluster. A system can contain several layers of
Quoted 8 lines

Documentation/devicetree/bindings/arm/topology.txt line 103

Quoted 5 lines
Documentation/devicetree/bindings/arm/topology.txt new file +5 -0
+		      clustering and cluster nodes can be contained in parent
+		      cluster nodes.
+
+	The cluster node name must be "clusterN" as described in 2.1 above.
+	A cluster node can not be a leaf node.

Follow standard conventions with "cluster at N" and a reg property with the number.

Quoted 5 lines

We are defining the topology to decouple the cluster/core/thread concept from the MPIDR. Having a reg property in the cluster (and core) nodes would complicate things if that reg property must correspond to an MPIDR bitfield. If it is meant to be just an enumeration at a given device tree level, I am ok with changing that.

Because the cluster itself doesn't really have an id, I'm fine if its not linked to the mpidr. Just don't change that later.

Quoted 56 lines
Quoted 47 lines
Quoted 47 lines
Documentation/devicetree/bindings/arm/topology.txt new file +47 -0
+
+	A cluster node's child nodes must be:
+
+	- one or more cluster nodes; or
+	- one or more core nodes
+
+	Any other configuration is considered invalid.
+
+- core node
+
+	Description: must be declared in a cluster node, one node per core in
+		     the cluster. If the system does not support SMT, core
+		     nodes are leaf nodes, otherwise they become containers of
+		     thread nodes.
+
+	The core node name must be "coreN" as described in 2.1 above.
+
+	A core node must be a leaf node if SMT is not supported.
+
+	Properties for core nodes that are leaf nodes:
+
+	- cpu
+		Usage: required
+		Value type: <phandle>
+		Definition: a phandle to the cpu node that corresponds to the
+			    core node.
+
+	If a core node is not a leaf node (CPUs supporting SMT) a core node's
+	child nodes can be:
+
+	- one or more thread nodes
+
+	Any other configuration is considered invalid.
+
+- thread node
+
+	Description: must be declared in a core node, one node per thread
+		     in the core if the system supports SMT. Thread nodes are
+		     always leaf nodes in the device tree.
+
+	The thread node name must be "threadN" as described in 2.1 above.
+
+	A thread node must be a leaf node.
+
+	A thread node must contain the following property:
+
+	- cpu
Quoted 8 lines

Documentation/devicetree/bindings/arm/topology.txt line 154

Quoted 4 lines
Documentation/devicetree/bindings/arm/topology.txt new file +4 -0
+		Usage: required
+		Value type: <phandle>
+		Definition: a phandle to the cpu node that corresponds to
+			    the thread node.

According to the ePAPR, threads are represented by an array of ids for reg property, not another cpu node. Why the deviation.

Quoted 4 lines

It is not a cpu node, it is a phandle property named cpu. Can you point me to the ePAPR section where threads bindings are described please ? I have not managed to find these details, I am reading version 1.0.

You should get 1.1.

Quoted 1 line

From the reg prop description:

If a CPU supports more than one thread (i.e. multiple streams of execution) the reg property is an array with 1 element per thread. The #address-cells on the /cpus node specifies how many cells each element of the array takes. Software can determine the number of threads by dividing the size of reg by the parent node's #address-cells. If a CPU/thread can be the target of an external interrupt the "reg" property value must be a unique CPU/thread id that is addressable by the interrupt controller.
If a CPU/thread cannot be the target of an external interrupt, then "reg" must be unique and out of bounds of the range addressed by the interrupt controller
If a CPU/thread's PIR is modifiable, a client program should modify PIR to match the "reg" property value. If PIR cannot be modified and the PIR value is distinct from the interrupt controller numberspace, the CPUs binding may define a binding-specific representation of PIR values if desired.

There is also this statement:

Hardware threads that share an MMU would generally be represented under one cpu node. If other
more complex CPU topographies are designed, the binding for the CPU must describe the topography
(e.g. threads that don't share an MMU).

Rob

#5 lore raw

RFC PATCH 1/11 Documentation: DT: arm: define CPU topology bindings

Quoted 4 lines · Rob Herring
Quoted 3 lines · Lorenzo Pieralisi
Quoted 2 lines · Rob Herring
Quoted 1 line · Mark Rutland

From: Lorenzo Pieralisi <[email protected]>

[...]

Quoted 19 lines
Quoted 19 lines
Quoted 18 lines
Quoted 5 lines

+===========================================
+3 - cluster/core/thread node bindings
+===========================================

Documentation/devicetree/bindings/arm/topology.txt new file +2 -0
+
+Bindings for cluster/cpu/thread nodes are defined as follows:

Documentation/devicetree/bindings/arm/topology.txt line 103

Quoted 10 lines
Documentation/devicetree/bindings/arm/topology.txt new file +10 -0
+
+- cluster node
+
+	 Description: must be declared within a cpu-map node, one node
+		      per cluster. A system can contain several layers of
+		      clustering and cluster nodes can be contained in parent
+		      cluster nodes.
+
+	The cluster node name must be "clusterN" as described in 2.1 above.
+	A cluster node can not be a leaf node.

Follow standard conventions with "cluster at N" and a reg property with the number.

Quoted 8 lines
Quoted 5 lines

We are defining the topology to decouple the cluster/core/thread concept from the MPIDR. Having a reg property in the cluster (and core) nodes would complicate things if that reg property must correspond to an MPIDR bitfield. If it is meant to be just an enumeration at a given device tree level, I am ok with changing that.

Because the cluster itself doesn't really have an id, I'm fine if its not linked to the mpidr. Just don't change that later.

These enumerations can follow the MPIDR for convenience if the MPIDR allocations are sane, but otherwise it would be abstract. The Architecture doesn't specify any kind of an ID for a cluster (indeed, it doesn't really specify what a cluster is at all).

However, we could require a platform or SoC binding to specify the enumeration and it how it maps to the actual hardware -- i.e., cluster at 1 on Tuesday should be the same physical cluster as cluster at 1 on Monday, to the extent that cluster at 1 is present in the DT on both days and the hardware hasn't been physically modified (i.e., cluster at 1 might refer to a fixed physical socket on a hypothetical board with physically pluggable CPU modules).

Correspondingly, on TC2, cluster at 0 would always be the A15 cluster; cluster at 1 would always be the A7 cluster. But if the firmware disables one of them, that cluster and the relevant CPU nodes should be removed from the DT before passing it to the kernel.

Does this make any sense, or is it overkill?

Cheers
---Dave

#6 lore raw

RFC PATCH 1/11 Documentation: DT: arm: define CPU topology bindings

Quoted 50 lines · Dave Martin
Quoted 4 lines · Rob Herring
Quoted 3 lines · Lorenzo Pieralisi
Quoted 2 lines · Rob Herring
Quoted 1 line · Mark Rutland

From: Lorenzo Pieralisi <[email protected]>

[...]

Quoted 19 lines
Quoted 19 lines
Quoted 18 lines
Quoted 5 lines

+===========================================
+3 - cluster/core/thread node bindings
+===========================================

Documentation/devicetree/bindings/arm/topology.txt new file +2 -0
+
+Bindings for cluster/cpu/thread nodes are defined as follows:

Documentation/devicetree/bindings/arm/topology.txt line 103

Quoted 10 lines
Documentation/devicetree/bindings/arm/topology.txt new file +10 -0
+
+- cluster node
+
+	 Description: must be declared within a cpu-map node, one node
+		      per cluster. A system can contain several layers of
+		      clustering and cluster nodes can be contained in parent
+		      cluster nodes.
+
+	The cluster node name must be "clusterN" as described in 2.1 above.
+	A cluster node can not be a leaf node.

Follow standard conventions with "cluster at N" and a reg property with the number.

Quoted 8 lines
Quoted 5 lines

We are defining the topology to decouple the cluster/core/thread concept from the MPIDR. Having a reg property in the cluster (and core) nodes would complicate things if that reg property must correspond to an MPIDR bitfield. If it is meant to be just an enumeration at a given device tree level, I am ok with changing that.

Because the cluster itself doesn't really have an id, I'm fine if its not linked to the mpidr. Just don't change that later.

These enumerations can follow the MPIDR for convenience if the MPIDR allocations are sane, but otherwise it would be abstract. The Architecture doesn't specify any kind of an ID for a cluster (indeed, it doesn't really specify what a cluster is at all).

However, we could require a platform or SoC binding to specify the enumeration and it how it maps to the actual hardware -- i.e., cluster at 1 on Tuesday should be the same physical cluster as cluster at 1 on Monday, to the extent that cluster at 1 is present in the DT on both days and the hardware hasn't been physically modified (i.e., cluster at 1 might refer to a fixed physical socket on a hypothetical board with physically pluggable CPU modules).

Quoted 6 lines

Correspondingly, on TC2, cluster at 0 would always be the A15 cluster; cluster at 1 would always be the A7 cluster. But if the firmware disables one of them, that cluster and the relevant CPU nodes should be removed from the DT before passing it to the kernel.

Does this make any sense, or is it overkill?

I think we can say that the reg property must correspond to a cluster id, irrespective of the MPIDR. Whatever it is, as you said, a platform has to have a cluster numbering scheme. How it is mapped to the MPIDR that's platform specific. So I am happy to add a reg property but linking it to the MPIDR goes against what we are doing here hence I would not do it, simply because we can't say "cluster reg property must be MPIDR[15:8]", that's wrong and there is precious little we can do about that.

Lorenzo

#7 lore raw

RFC PATCH 1/11 Documentation: DT: arm: define CPU topology bindings

In reply to Rob Herring

On Thu, Apr 11, 2013 at 06:55:20PM +0100, Rob Herring wrote:

[...]

Quoted 12 lines
Quoted 12 lines
Quoted 11 lines

Documentation/devicetree/bindings/arm/topology.txt line 69

Quoted 6 lines
Documentation/devicetree/bindings/arm/topology.txt new file +6 -0
+The cpu-map node can only contain three types of child nodes:
+
+- cluster node
+- core node
+- thread node
+

Why not put the topology in the /cpus nodes? I don't really see the point of having a flat list of cpus and separate topology info. There is some compatibility issue, but adding optional levels for clusters can be handled.

Quoted 7 lines
Quoted 3 lines

I thought this would break all code relying on /cpu nodes being /cpus node's children. Furthermore, I was told that the /cpus node can only have /cpu nodes as children.

IIRC the context, that was in regards to putting things like the PMUs under the /cpus node. Or are you referring to something else? I think this situation is a bit different.

No I am referring to:

(a) PowerPC code relying on cpu nodes to be cpus node's children
(b) code that parses the DT cpus/cpu node for ARM that does require cpu nodes to be cpus node's children (but that has nothing to do with the bindings, code can be changed)

I think that our definition of cpu nodes does not comply with ePAPR 3.7, since an MPIDR corresponding to an SMT thread cannot be represented with one CPU node "A cpu node represents a hardware execution block that is sufficiently independent that it is capable of running an operating system without interfering with other CPUs possibly running other operating systems".

This also has consequences on the cpu node bindings I am about to post since if we go by the definition above a cpu node representing an SMT core should contain a list of MPIDRs.

Quoted 2 lines

You will have to support existing single cluster systems without the hierarchy.

Well yes, that's why we used another (optional) node to define the topology.

Quoted 40 lines
Quoted 40 lines

If you wish so, we can define the topology in the /cpus node, fine by me.

Quoted 37 lines
Quoted 24 lines
Documentation/devicetree/bindings/arm/topology.txt new file +24 -0
+whose bindings are described in paragraph 3.
+
+The nodes describing the CPU topology (cluster/core/thread) can only be
+defined within the cpu-map node.
+Any other configuration is consider invalid and therefore must be ignored.
+
+===========================================
+2.1 - cpu-map child nodes naming convention
+===========================================
+
+cpu-map child nodes must follow a naming convention where the node name
+must be "clusterN", "coreN", "threadN" depending on the node type (ie
+cluster/core/thread) (where N = {0, 1, ...} is the node number; nodes which
+are siblings within a single common parent node must be given a unique and
+sequential N value, starting from 0).
+cpu-map child nodes which do not share a common parent node can have the same
+name (ie same number N as other cpu-map child nodes at different device tree
+levels) since name uniqueness will be guaranteed by the device tree hierarchy.
+
+===========================================
+3 - cluster/core/thread node bindings
+===========================================
+
+Bindings for cluster/cpu/thread nodes are defined as follows:

Documentation/devicetree/bindings/arm/topology.txt line 103

Quoted 10 lines
Documentation/devicetree/bindings/arm/topology.txt new file +10 -0
+
+- cluster node
+
+    Description: must be declared within a cpu-map node, one node
+                 per cluster. A system can contain several layers of
+                 clustering and cluster nodes can be contained in parent
+                 cluster nodes.
+
+   The cluster node name must be "clusterN" as described in 2.1 above.
+   A cluster node can not be a leaf node.

Follow standard conventions with "cluster at N" and a reg property with the number.

Quoted 8 lines
Quoted 5 lines

We are defining the topology to decouple the cluster/core/thread concept from the MPIDR. Having a reg property in the cluster (and core) nodes would complicate things if that reg property must correspond to an MPIDR bitfield. If it is meant to be just an enumeration at a given device tree level, I am ok with changing that.

Because the cluster itself doesn't really have an id, I'm fine if its not linked to the mpidr. Just don't change that later.

Eh, the cluster does have an id, that depends on the MPIDR du jour. I will give it more thought.

Quoted 62 lines
Quoted 56 lines
Quoted 47 lines
Quoted 47 lines
Documentation/devicetree/bindings/arm/topology.txt new file +47 -0
+
+   A cluster node's child nodes must be:
+
+   - one or more cluster nodes; or
+   - one or more core nodes
+
+   Any other configuration is considered invalid.
+
+- core node
+
+   Description: must be declared in a cluster node, one node per core in
+                the cluster. If the system does not support SMT, core
+                nodes are leaf nodes, otherwise they become containers of
+                thread nodes.
+
+   The core node name must be "coreN" as described in 2.1 above.
+
+   A core node must be a leaf node if SMT is not supported.
+
+   Properties for core nodes that are leaf nodes:
+
+   - cpu
+           Usage: required
+           Value type: <phandle>
+           Definition: a phandle to the cpu node that corresponds to the
+                       core node.
+
+   If a core node is not a leaf node (CPUs supporting SMT) a core node's
+   child nodes can be:
+
+   - one or more thread nodes
+
+   Any other configuration is considered invalid.
+
+- thread node
+
+   Description: must be declared in a core node, one node per thread
+                in the core if the system supports SMT. Thread nodes are
+                always leaf nodes in the device tree.
+
+   The thread node name must be "threadN" as described in 2.1 above.
+
+   A thread node must be a leaf node.
+
+   A thread node must contain the following property:
+
+   - cpu
Quoted 8 lines

Documentation/devicetree/bindings/arm/topology.txt line 154

Quoted 4 lines
Documentation/devicetree/bindings/arm/topology.txt new file +4 -0
+           Usage: required
+           Value type: <phandle>
+           Definition: a phandle to the cpu node that corresponds to
+                       the thread node.

According to the ePAPR, threads are represented by an array of ids for reg property, not another cpu node. Why the deviation.

Quoted 4 lines

It is not a cpu node, it is a phandle property named cpu. Can you point me to the ePAPR section where threads bindings are described please ? I have not managed to find these details, I am reading version 1.0.

You should get 1.1.

Quoted 7 lines

From the reg prop description:

If a CPU supports more than one thread (i.e. multiple streams of execution) the reg property is an array with 1 element per thread. The #address-cells on the /cpus node specifies how many cells each element of the array takes. Software can determine the number of threads by dividing the size of reg by the parent node's #address-cells.

See above, I need to investigate what this implies.

Quoted 3 lines

If a CPU/thread can be the target of an external interrupt the "reg" property value must be a unique CPU/thread id that is addressable by the interrupt controller.

I am not sure what "external" means here, but basically it looks like our DT cpu bindings are non-compliant, the interrupt controller cannot address a cpu by using an MPIDR (yet).

Quoted 10 lines

If a CPU/thread cannot be the target of an external interrupt, then "reg" must be unique and out of bounds of the range addressed by the interrupt controller
If a CPU/thread's PIR is modifiable, a client program should modify PIR to match the "reg" property value. If PIR cannot be modified and the PIR value is distinct from the interrupt controller numberspace, the CPUs binding may define a binding-specific representation of PIR values if desired.

Quoted 7 lines

There is also this statement:

Hardware threads that share an MMU would generally be represented under one cpu node. If other
more complex CPU topographies are designed, the binding for the CPU must describe the topography
(e.g. threads that don't share an MMU).

Again, see above, HW threads do share an MMU so we have to make a decision here, either we make ARM comply with ePAPR or change the bindings for ARM.

Lorenzo

#8 lore raw

RFC PATCH 1/11 Documentation: DT: arm: define CPU topology bindings

In reply to Lorenzo Pieralisi
Quoted 115 lines · Lorenzo Pieralisi
Quoted 15 lines · Rob Herring
Quoted 14 lines · Mark Rutland

From: Lorenzo Pieralisi <[email protected]>

The advent of multi-cluster ARM systems requires a mechanism to describe how in hierarchical terms CPUs are connected in ARM SoCs so that the kernel can initialize and map resources like IRQs and memory space to specific group(s) of CPUs.

The CPU topology is made up of multiple hierarchy levels whose bottom layers (aka leaf nodes in device tree syntax) contain links to the HW CPUs in the system.

The topology bindings are generic for both 32-bit and 64-bit systems and lay the groundwork on top of which affinity schemes can be built.

Quoted 6 lines
Quoted 3 lines

This patch provides the documentation in the kernel required to define the device tree bindings describing the CPU topology for ARM 32-bit and 64-bit systems.

I'm now very weary of continued /cpu changes after the pain of making the reg property reflect the mpidr value in 3.8.

We won't change the reg property value, the code I am about to post provides stricter bindings, stricter semantics and extends bindings to cater for arm 64-bit systems.

Quoted 79 lines
Quoted 79 lines
---
 Documentation/devicetree/bindings/arm/topology.txt | 524 +++++++++++++++++++++
 1 file changed, 524 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/arm/topology.txt
Documentation/devicetree/bindings/arm/topology.txt new file +66 -0
new file mode 100644
index 0000000..07c4961
@@ -0,0 +1,524 @@
+===========================================
+ARM topology binding description
+===========================================
+
+===========================================
+1 - Introduction
+===========================================
+
+In an ARM system, the hierarchy of CPUs is defined through three entities that
+are used to describe the layout of physical CPUs in the system:
+
+- cluster
+- core
+- thread
+
+The cpu nodes (bindings defined in [1]) represent the devices that
+correspond to physical CPUs and are to be mapped to the hierarchy levels.
+
+The bottom hierarchy level sits at core or thread level depending on whether
+symmetric multi-threading (SMT) is supported or not.
+
+For instance in a system where CPUs support SMT, "cpu" nodes represent all
+threads existing in the system and map to the hierarchy level "thread" above.
+In systems where SMT is not supported "cpu" nodes represent all cores present
+in the system and map to the hierarchy level "core" above.
+
+ARM topology bindings allow one to associate cpu nodes with hierarchical groups
+corresponding to the system hierarchy; syntactically they are defined as device
+tree nodes.
+
+The remainder of this document provides the topology bindings for ARM, based
+on the ePAPR standard, available from:
+
+http://devicetree.org
+
+If not stated otherwise, whenever a reference to a cpu node phandle is made its
+value must point to a cpu node compliant with the cpu node bindings as
+documented in [1].
+A topology description containing phandles to cpu nodes that are not compliant
+with bindings standardized in [1] is therefore considered invalid.
+
+===========================================
+2 - cpu-map node
+===========================================
+
+The ARM CPU topology is defined within a container node, sitting at the top
+level of the device tree (/), the cpu-map node.
+
+- cpu-map node
+
+	Usage: Required to define ARM CPU topology
+
+	Description: The cpu-map node is just a container node where its
+		     subnodes describe the CPU topology
+
+	Node name must be "cpu-map".
+
+	A cpu-map node's child nodes can be:
+
+	- one or more cluster nodes
+
+	Any other configuration is considered invalid.
+
+The cpu-map node can only contain three types of child nodes:
+
+- cluster node
Quoted 8 lines

Documentation/devicetree/bindings/arm/topology.txt line 69

Quoted 3 lines
Documentation/devicetree/bindings/arm/topology.txt new file +3 -0
+- core node
+- thread node
+

Why not put the topology in the /cpus nodes? I don't really see the point of having a flat list of cpus and separate topology info. There is some compatibility issue, but adding optional levels for clusters can be handled.

Quoted 5 lines

I thought this would break all code relying on /cpu nodes being /cpus node's children. Furthermore, I was told that the /cpus node can only have /cpu nodes as children.

If you wish so, we can define the topology in the /cpus node, fine by me.

Can we make such extensive changes to the cpus node without violating the ePAPR specification?

If we can, great, but I'm a but unclear on how this would be achieved.

Quoted 38 lines
Quoted 29 lines
Quoted 29 lines
Documentation/devicetree/bindings/arm/topology.txt new file +29 -0
+whose bindings are described in paragraph 3.
+
+The nodes describing the CPU topology (cluster/core/thread) can only be
+defined within the cpu-map node.
+Any other configuration is consider invalid and therefore must be ignored.
+
+===========================================
+2.1 - cpu-map child nodes naming convention
+===========================================
+
+cpu-map child nodes must follow a naming convention where the node name
+must be "clusterN", "coreN", "threadN" depending on the node type (ie
+cluster/core/thread) (where N = {0, 1, ...} is the node number; nodes which
+are siblings within a single common parent node must be given a unique and
+sequential N value, starting from 0).
+cpu-map child nodes which do not share a common parent node can have the same
+name (ie same number N as other cpu-map child nodes at different device tree
+levels) since name uniqueness will be guaranteed by the device tree hierarchy.
+
+===========================================
+3 - cluster/core/thread node bindings
+===========================================
+
+Bindings for cluster/cpu/thread nodes are defined as follows:
+
+- cluster node
+
+	 Description: must be declared within a cpu-map node, one node
+		      per cluster. A system can contain several layers of
Quoted 8 lines

Documentation/devicetree/bindings/arm/topology.txt line 103

Quoted 5 lines
Documentation/devicetree/bindings/arm/topology.txt new file +5 -0
+		      clustering and cluster nodes can be contained in parent
+		      cluster nodes.
+
+	The cluster node name must be "clusterN" as described in 2.1 above.
+	A cluster node can not be a leaf node.

Follow standard conventions with "cluster at N" and a reg property with the number.

Quoted 5 lines

We are defining the topology to decouple the cluster/core/thread concept from the MPIDR. Having a reg property in the cluster (and core) nodes would complicate things if that reg property must correspond to an MPIDR bitfield. If it is meant to be just an enumeration at a given device tree level, I am ok with changing that.

As a pure enumeration, I think that works fine. It's more verbose but also more conformant to DT conventions. I'm not sure there's another difference.

The proposed support for C preprocessing of dts files might provide a way to help debloat this to some extent in dts source, while still following the DT convention of using unit addresses and reg properties. This will significantly increase the size of the FDT blob if the number of CPUs is large. I don't remember offhand if we have a limit on the size of FDT we can cope with. Finding ways to relax the limit is a better solution than dodging round standards, though. We can cross that bridge when/if we come to it.

Quoted 57 lines
Quoted 48 lines
Quoted 48 lines
Documentation/devicetree/bindings/arm/topology.txt new file +47 -0
+
+	A cluster node's child nodes must be:
+
+	- one or more cluster nodes; or
+	- one or more core nodes
+
+	Any other configuration is considered invalid.
+
+- core node
+
+	Description: must be declared in a cluster node, one node per core in
+		     the cluster. If the system does not support SMT, core
+		     nodes are leaf nodes, otherwise they become containers of
+		     thread nodes.
+
+	The core node name must be "coreN" as described in 2.1 above.
+
+	A core node must be a leaf node if SMT is not supported.
+
+	Properties for core nodes that are leaf nodes:
+
+	- cpu
+		Usage: required
+		Value type: <phandle>
+		Definition: a phandle to the cpu node that corresponds to the
+			    core node.
+
+	If a core node is not a leaf node (CPUs supporting SMT) a core node's
+	child nodes can be:
+
+	- one or more thread nodes
+
+	Any other configuration is considered invalid.
+
+- thread node
+
+	Description: must be declared in a core node, one node per thread
+		     in the core if the system supports SMT. Thread nodes are
+		     always leaf nodes in the device tree.
+
+	The thread node name must be "threadN" as described in 2.1 above.
+
+	A thread node must be a leaf node.
+
+	A thread node must contain the following property:
+
+	- cpu
Quoted 8 lines

Documentation/devicetree/bindings/arm/topology.txt line 154

Quoted 4 lines
Documentation/devicetree/bindings/arm/topology.txt new file +4 -0
+		Usage: required
+		Value type: <phandle>
+		Definition: a phandle to the cpu node that corresponds to
+			    the thread node.

According to the ePAPR, threads are represented by an array of ids for reg property, not another cpu node. Why the deviation.

Quoted 3 lines

It is not a cpu node, it is a phandle property named cpu. Can you point me to the ePAPR section where threads bindings are described please ? I have not managed to find these details, I am reading version 1.0.

For cpu/reg:

[1] If a CPU supports more than one thread (i.e. multiple streams of
execution) the reg property is an array with 1 element per thread. The #address-cells on the /cpus node specifies how many cells each element of the array takes. Software can determine the number of threads by dividing the size of reg by the parent node's #address-cells.

I had not previously been aware of this, but I see no reason not to follow this convention.

Also:
[2] If other more complex CPU topographies are designed, the binding
for the CPU must describe the topography

That's rather less helpful, but the suggestion is clear enough in that such information should be in the cpu node and specific to that CPU's binding. For ARM, we can have some global extensions to the CPU node.

The problems start when you want to refer to clusters and groups of CPUs from other nodes. Only individual cpu nodes can be places in the cpus node, so there is no node for a phandle to point at.

If you want to describe how other things like power, clock and coherency domains map to clusters and larger entities, things could get pretty awkward.

Keeping the topology description separate allows all topological entities to appear as addressable entities in the DT; otherwise, a cryptic convention is needed.

Hybrid approaches might be possible, putting cpu nodes into /cpus, and giving them a "parent" property where appropriate pointing at the relevant cluster node, which we put elsewhere in the DT.

I'm not sure whether any of these approaches is an awful lot less ugly or more easy to handle than what it currently proposed though.

The global binding for all ARM CPUs would specify that the topology is described by /cpu-map and its associated binding. For my interpretation of [2], this is a compliant approach. ePAPR does not specify _how_ the cpu node binding achieves a description of the topography, just that it must achieve it. There's no statement to say that it must not involve other nodes or bindings.

Cheers
---Dave

#9 lore raw

RFC PATCH 1/11 Documentation: DT: arm: define CPU topology bindings

On Thu, Apr 11, 2013 at 07:01:25PM +0100, Dave Martin wrote:

[...]

Quoted 38 lines
Quoted 38 lines
Quoted 25 lines
Quoted 25 lines

+===========================================
+2 - cpu-map node
+===========================================

Documentation/devicetree/bindings/arm/topology.txt new file +22 -0
+
+The ARM CPU topology is defined within a container node, sitting at the top
+level of the device tree (/), the cpu-map node.
+
+- cpu-map node
+
+ Usage: Required to define ARM CPU topology
+
+ Description: The cpu-map node is just a container node where its
+              subnodes describe the CPU topology
+
+ Node name must be "cpu-map".
+
+ A cpu-map node's child nodes can be:
+
+ - one or more cluster nodes
+
+ Any other configuration is considered invalid.
+
+The cpu-map node can only contain three types of child nodes:
+
+- cluster node
Quoted 8 lines
Quoted 3 lines
+- core node
+- thread node

+

Why not put the topology in the /cpus nodes? I don't really see the point of having a flat list of cpus and separate topology info. There is some compatibility issue, but adding optional levels for clusters can be handled.

I thought this would break all code relying on /cpu nodes being /cpus node's children. Furthermore, I was told that the /cpus node can only have /cpu nodes as children.

Quoted 6 lines
Quoted 1 line

If you wish so, we can define the topology in the /cpus node, fine by me.

Can we make such extensive changes to the cpus node without violating the ePAPR specification?

If we can, great, but I'm a but unclear on how this would be achieved.

+1

Quoted 48 lines
Quoted 38 lines
Quoted 29 lines
Quoted 29 lines
Documentation/devicetree/bindings/arm/topology.txt new file +29 -0
+whose bindings are described in paragraph 3.
+
+The nodes describing the CPU topology (cluster/core/thread) can only be
+defined within the cpu-map node.
+Any other configuration is consider invalid and therefore must be ignored.
+
+===========================================
+2.1 - cpu-map child nodes naming convention
+===========================================
+
+cpu-map child nodes must follow a naming convention where the node name
+must be "clusterN", "coreN", "threadN" depending on the node type (ie
+cluster/core/thread) (where N = {0, 1, ...} is the node number; nodes which
+are siblings within a single common parent node must be given a unique and
+sequential N value, starting from 0).
+cpu-map child nodes which do not share a common parent node can have the same
+name (ie same number N as other cpu-map child nodes at different device tree
+levels) since name uniqueness will be guaranteed by the device tree hierarchy.
+
+===========================================
+3 - cluster/core/thread node bindings
+===========================================
+
+Bindings for cluster/cpu/thread nodes are defined as follows:
+
+- cluster node
+
+  Description: must be declared within a cpu-map node, one node
+               per cluster. A system can contain several layers of
Quoted 8 lines

Documentation/devicetree/bindings/arm/topology.txt line 103

Quoted 5 lines
Documentation/devicetree/bindings/arm/topology.txt new file +5 -0
+               clustering and cluster nodes can be contained in parent
+               cluster nodes.
+
+ The cluster node name must be "clusterN" as described in 2.1 above.
+ A cluster node can not be a leaf node.

Follow standard conventions with "cluster at N" and a reg property with the number.

Quoted 5 lines

We are defining the topology to decouple the cluster/core/thread concept from the MPIDR. Having a reg property in the cluster (and core) nodes would complicate things if that reg property must correspond to an MPIDR bitfield. If it is meant to be just an enumeration at a given device tree level, I am ok with changing that.

As a pure enumeration, I think that works fine. It's more verbose but also more conformant to DT conventions. I'm not sure there's another difference.

Quoted 8 lines

The proposed support for C preprocessing of dts files might provide a way to help debloat this to some extent in dts source, while still following the DT convention of using unit addresses and reg properties. This will significantly increase the size of the FDT blob if the number of CPUs is large. I don't remember offhand if we have a limit on the size of FDT we can cope with. Finding ways to relax the limit is a better solution than dodging round standards, though. We can cross that bridge when/if we come to it.

Well, that's a problem by itself, certainly adding a reg property to the cluster nodes will make it worse, but as you said still better that than dodging standards.

Quoted 70 lines
Quoted 57 lines
Quoted 48 lines
Quoted 48 lines
Documentation/devicetree/bindings/arm/topology.txt new file +47 -0
+
+ A cluster node's child nodes must be:
+
+ - one or more cluster nodes; or
+ - one or more core nodes
+
+ Any other configuration is considered invalid.
+
+- core node
+
+ Description: must be declared in a cluster node, one node per core in
+              the cluster. If the system does not support SMT, core
+              nodes are leaf nodes, otherwise they become containers of
+              thread nodes.
+
+ The core node name must be "coreN" as described in 2.1 above.
+
+ A core node must be a leaf node if SMT is not supported.
+
+ Properties for core nodes that are leaf nodes:
+
+ - cpu
+         Usage: required
+         Value type: <phandle>
+         Definition: a phandle to the cpu node that corresponds to the
+                     core node.
+
+ If a core node is not a leaf node (CPUs supporting SMT) a core node's
+ child nodes can be:
+
+ - one or more thread nodes
+
+ Any other configuration is considered invalid.
+
+- thread node
+
+ Description: must be declared in a core node, one node per thread
+              in the core if the system supports SMT. Thread nodes are
+              always leaf nodes in the device tree.
+
+ The thread node name must be "threadN" as described in 2.1 above.
+
+ A thread node must be a leaf node.
+
+ A thread node must contain the following property:
+
+ - cpu
Quoted 8 lines

Documentation/devicetree/bindings/arm/topology.txt line 154

Quoted 4 lines
Documentation/devicetree/bindings/arm/topology.txt new file +4 -0
+         Usage: required
+         Value type: <phandle>
+         Definition: a phandle to the cpu node that corresponds to
+                     the thread node.

According to the ePAPR, threads are represented by an array of ids for reg property, not another cpu node. Why the deviation.

Quoted 3 lines

It is not a cpu node, it is a phandle property named cpu. Can you point me to the ePAPR section where threads bindings are described please ? I have not managed to find these details, I am reading version 1.0.

For cpu/reg:

[1] If a CPU supports more than one thread (i.e. multiple streams of
execution) the reg property is an array with 1 element per thread. The #address-cells on the /cpus node specifies how many cells each element of the array takes. Software can determine the number of threads by dividing the size of reg by the parent node's #address-cells.

Quoted 2 lines

I had not previously been aware of this, but I see no reason not to follow this convention.

I don't see a reason either, but this changes the current cpu node bindings for ARM. On the upside there are no SMT ARM platforms out there, so no backward compatibility to worry about.

This would reduce the topology problem to where cluster nodes should be defined, either in the cpus node or a separate node (ie cpu-map :-)).

Quoted 17 lines

Also:
[2] If other more complex CPU topographies are designed, the binding
for the CPU must describe the topography

That's rather less helpful, but the suggestion is clear enough in that such information should be in the cpu node and specific to that CPU's binding. For ARM, we can have some global extensions to the CPU node.

The problems start when you want to refer to clusters and groups of CPUs from other nodes. Only individual cpu nodes can be places in the cpus node, so there is no node for a phandle to point at.

If you want to describe how other things like power, clock and coherency domains map to clusters and larger entities, things could get pretty awkward.

Quoted 8 lines

Keeping the topology description separate allows all topological entities to appear as addressable entities in the DT; otherwise, a cryptic convention is needed.

Hybrid approaches might be possible, putting cpu nodes into /cpus, and giving them a "parent" property where appropriate pointing at the relevant cluster node, which we put elsewhere in the DT.

That's what I did, with a couple of twists:

http://lists.infradead.org/pipermail/linux-arm-kernel/2012-January/080873.html

I have no preference, time to make a decision though.

Quoted 2 lines

I'm not sure whether any of these approaches is an awful lot less ugly or more easy to handle than what it currently proposed though.

+1

Quoted 6 lines

The global binding for all ARM CPUs would specify that the topology is described by /cpu-map and its associated binding. For my interpretation of [2], this is a compliant approach. ePAPR does not specify _how_ the cpu node binding achieves a description of the topography, just that it must achieve it. There's no statement to say that it must not involve other nodes or bindings.

Again, I think it all boils down to deciding where cluster nodes should live.

More opinions ?

Thx,
Lorenzo

#10 lore raw

RFC PATCH 1/11 Documentation: DT: arm: define CPU topology bindings

Quoted 182 lines · Lorenzo Pieralisi

On Thu, Apr 11, 2013 at 07:01:25PM +0100, Dave Martin wrote:

[...]

Quoted 38 lines
Quoted 38 lines
Quoted 25 lines
Quoted 25 lines

+===========================================
+2 - cpu-map node
+===========================================

Documentation/devicetree/bindings/arm/topology.txt new file +22 -0
+
+The ARM CPU topology is defined within a container node, sitting at the top
+level of the device tree (/), the cpu-map node.
+
+- cpu-map node
+
+ Usage: Required to define ARM CPU topology
+
+ Description: The cpu-map node is just a container node where its
+              subnodes describe the CPU topology
+
+ Node name must be "cpu-map".
+
+ A cpu-map node's child nodes can be:
+
+ - one or more cluster nodes
+
+ Any other configuration is considered invalid.
+
+The cpu-map node can only contain three types of child nodes:
+
+- cluster node
Quoted 8 lines
Quoted 3 lines
+- core node
+- thread node

+

Why not put the topology in the /cpus nodes? I don't really see the point of having a flat list of cpus and separate topology info. There is some compatibility issue, but adding optional levels for clusters can be handled.

I thought this would break all code relying on /cpu nodes being /cpus node's children. Furthermore, I was told that the /cpus node can only have /cpu nodes as children.

Quoted 6 lines
Quoted 1 line

If you wish so, we can define the topology in the /cpus node, fine by me.

Can we make such extensive changes to the cpus node without violating the ePAPR specification?

If we can, great, but I'm a but unclear on how this would be achieved.

+1

Quoted 48 lines
Quoted 38 lines
Quoted 29 lines
Quoted 29 lines
Documentation/devicetree/bindings/arm/topology.txt new file +29 -0
+whose bindings are described in paragraph 3.
+
+The nodes describing the CPU topology (cluster/core/thread) can only be
+defined within the cpu-map node.
+Any other configuration is consider invalid and therefore must be ignored.
+
+===========================================
+2.1 - cpu-map child nodes naming convention
+===========================================
+
+cpu-map child nodes must follow a naming convention where the node name
+must be "clusterN", "coreN", "threadN" depending on the node type (ie
+cluster/core/thread) (where N = {0, 1, ...} is the node number; nodes which
+are siblings within a single common parent node must be given a unique and
+sequential N value, starting from 0).
+cpu-map child nodes which do not share a common parent node can have the same
+name (ie same number N as other cpu-map child nodes at different device tree
+levels) since name uniqueness will be guaranteed by the device tree hierarchy.
+
+===========================================
+3 - cluster/core/thread node bindings
+===========================================
+
+Bindings for cluster/cpu/thread nodes are defined as follows:
+
+- cluster node
+
+  Description: must be declared within a cpu-map node, one node
+               per cluster. A system can contain several layers of
Quoted 8 lines

Documentation/devicetree/bindings/arm/topology.txt line 103

Quoted 5 lines
Documentation/devicetree/bindings/arm/topology.txt new file +5 -0
+               clustering and cluster nodes can be contained in parent
+               cluster nodes.
+
+ The cluster node name must be "clusterN" as described in 2.1 above.
+ A cluster node can not be a leaf node.

Follow standard conventions with "cluster at N" and a reg property with the number.

Quoted 5 lines

We are defining the topology to decouple the cluster/core/thread concept from the MPIDR. Having a reg property in the cluster (and core) nodes would complicate things if that reg property must correspond to an MPIDR bitfield. If it is meant to be just an enumeration at a given device tree level, I am ok with changing that.

As a pure enumeration, I think that works fine. It's more verbose but also more conformant to DT conventions. I'm not sure there's another difference.

Quoted 8 lines

The proposed support for C preprocessing of dts files might provide a way to help debloat this to some extent in dts source, while still following the DT convention of using unit addresses and reg properties. This will significantly increase the size of the FDT blob if the number of CPUs is large. I don't remember offhand if we have a limit on the size of FDT we can cope with. Finding ways to relax the limit is a better solution than dodging round standards, though. We can cross that bridge when/if we come to it.

Well, that's a problem by itself, certainly adding a reg property to the cluster nodes will make it worse, but as you said still better that than dodging standards.

Quoted 70 lines
Quoted 57 lines
Quoted 48 lines
Quoted 48 lines
Documentation/devicetree/bindings/arm/topology.txt new file +47 -0
+
+ A cluster node's child nodes must be:
+
+ - one or more cluster nodes; or
+ - one or more core nodes
+
+ Any other configuration is considered invalid.
+
+- core node
+
+ Description: must be declared in a cluster node, one node per core in
+              the cluster. If the system does not support SMT, core
+              nodes are leaf nodes, otherwise they become containers of
+              thread nodes.
+
+ The core node name must be "coreN" as described in 2.1 above.
+
+ A core node must be a leaf node if SMT is not supported.
+
+ Properties for core nodes that are leaf nodes:
+
+ - cpu
+         Usage: required
+         Value type: <phandle>
+         Definition: a phandle to the cpu node that corresponds to the
+                     core node.
+
+ If a core node is not a leaf node (CPUs supporting SMT) a core node's
+ child nodes can be:
+
+ - one or more thread nodes
+
+ Any other configuration is considered invalid.
+
+- thread node
+
+ Description: must be declared in a core node, one node per thread
+              in the core if the system supports SMT. Thread nodes are
+              always leaf nodes in the device tree.
+
+ The thread node name must be "threadN" as described in 2.1 above.
+
+ A thread node must be a leaf node.
+
+ A thread node must contain the following property:
+
+ - cpu
Quoted 8 lines

Documentation/devicetree/bindings/arm/topology.txt line 154

Quoted 4 lines
Documentation/devicetree/bindings/arm/topology.txt new file +4 -0
+         Usage: required
+         Value type: <phandle>
+         Definition: a phandle to the cpu node that corresponds to
+                     the thread node.

According to the ePAPR, threads are represented by an array of ids for reg property, not another cpu node. Why the deviation.

Quoted 3 lines

It is not a cpu node, it is a phandle property named cpu. Can you point me to the ePAPR section where threads bindings are described please ? I have not managed to find these details, I am reading version 1.0.

For cpu/reg:

[1] If a CPU supports more than one thread (i.e. multiple streams of
execution) the reg property is an array with 1 element per thread. The #address-cells on the /cpus node specifies how many cells each element of the array takes. Software can determine the number of threads by dividing the size of reg by the parent node's #address-cells.

Quoted 6 lines
Quoted 2 lines

I had not previously been aware of this, but I see no reason not to follow this convention.

I don't see a reason either, but this changes the current cpu node bindings for ARM. On the upside there are no SMT ARM platforms out there, so no backward compatibility to worry about.

Actually, I've had second thoughts about this, from discussion with Mark et al.

The extent to which threads share stuff is not really architecturally visible on ARM. It will be visible in performance terms (i.e., scheduling two threads of the same process on threads of on CPU will give better performance than scheduling threads of different processes), but in architectural terms still look like fully-fledged, independent CPUs.

I don't know enough about how SMT scheduling currently works in the kernel to know how best to describe this situation to the kernel...

Anyway, for the ARM case, there is not much architectural difference between threads within a CPU, and CPUs in a cluster. At both topological levels the siblings are independent. At both levels, there is an advantage in scheduling related threads topologically close to each other -- though probably more so for threads in a CPU than CPUs in a cluster.

Also, threads are independent interrupt destinations. If we want to put flat lists of SMT threads inside CPU nodes, then we need an extra means of describing interrupt affinities, different from the way this is described for CPUs and clusters. This is definitely complexity. I'm not sure if there is a related benefit.

Quoted 29 lines

This would reduce the topology problem to where cluster nodes should be defined, either in the cpus node or a separate node (ie cpu-map :-)).

Quoted 17 lines

Also:
[2] If other more complex CPU topographies are designed, the binding
for the CPU must describe the topography

That's rather less helpful, but the suggestion is clear enough in that such information should be in the cpu node and specific to that CPU's binding. For ARM, we can have some global extensions to the CPU node.

The problems start when you want to refer to clusters and groups of CPUs from other nodes. Only individual cpu nodes can be places in the cpus node, so there is no node for a phandle to point at.

If you want to describe how other things like power, clock and coherency domains map to clusters and larger entities, things could get pretty awkward.

Quoted 8 lines

Keeping the topology description separate allows all topological entities to appear as addressable entities in the DT; otherwise, a cryptic convention is needed.

Hybrid approaches might be possible, putting cpu nodes into /cpus, and giving them a "parent" property where appropriate pointing at the relevant cluster node, which we put elsewhere in the DT.

Quoted 5 lines

That's what I did, with a couple of twists:

http://lists.infradead.org/pipermail/linux-arm-kernel/2012-January/080873.html

I have no preference, time to make a decision though.

With the arguments above, I'm not sure this is really better than the current proposal...

Quoted 13 lines
Quoted 3 lines

I'm not sure whether any of these approaches is an awful lot less ugly or more easy to handle than what it currently proposed though.

+1

Quoted 6 lines

The global binding for all ARM CPUs would specify that the topology is described by /cpu-map and its associated binding. For my interpretation of [2], this is a compliant approach. ePAPR does not specify _how_ the cpu node binding achieves a description of the topography, just that it must achieve it. There's no statement to say that it must not involve other nodes or bindings.

Quoted 2 lines

Again, I think it all boils down to deciding where cluster nodes should live.

If we want to be able to describe affinities and other hardware linkages, describing the real hardware units as nodes still feels "right". ePAPR doesn't insist upon how this is done, so we do have choice.

The older/hybrid proposals seem to require different means of describing linkage depending on whether the target is a topological leaf or not.

I guess the question should be "what advantage is gained from describing this stuff in the cpus node?"

Cheers
---Dave

#11 lore raw

RFC PATCH 1/11 Documentation: DT: arm: define CPU topology bindings

On Fri, Apr 12, 2013 at 03:36:44PM +0100, Dave Martin wrote:

[...]

Quoted 42 lines
Quoted 16 lines
Quoted 16 lines
Quoted 6 lines
Quoted 2 lines

According to the ePAPR, threads are represented by an array of ids for reg property, not another cpu node. Why the deviation.

It is not a cpu node, it is a phandle property named cpu. Can you point me to the ePAPR section where threads bindings are described please ? I have not managed to find these details, I am reading version 1.0.

For cpu/reg:

[1] If a CPU supports more than one thread (i.e. multiple streams of
execution) the reg property is an array with 1 element per thread. The #address-cells on the /cpus node specifies how many cells each element of the array takes. Software can determine the number of threads by dividing the size of reg by the parent node's #address-cells.

Quoted 6 lines
Quoted 2 lines

I had not previously been aware of this, but I see no reason not to follow this convention.

I don't see a reason either, but this changes the current cpu node bindings for ARM. On the upside there are no SMT ARM platforms out there, so no backward compatibility to worry about.

Actually, I've had second thoughts about this, from discussion with Mark et al.

The extent to which threads share stuff is not really architecturally visible on ARM. It will be visible in performance terms (i.e., scheduling two threads of the same process on threads of on CPU will give better performance than scheduling threads of different processes), but in architectural terms still look like fully-fledged, independent CPUs.

I don't know enough about how SMT scheduling currently works in the kernel to know how best to describe this situation to the kernel...

Anyway, for the ARM case, there is not much architectural difference between threads within a CPU, and CPUs in a cluster. At both topological levels the siblings are independent. At both levels, there is an advantage in scheduling related threads topologically close to each other -- though probably more so for threads in a CPU than CPUs in a cluster.

Quoted 5 lines

Also, threads are independent interrupt destinations. If we want to put flat lists of SMT threads inside CPU nodes, then we need an extra means of describing interrupt affinities, different from the way this is described for CPUs and clusters. This is definitely complexity. I'm not sure if there is a related benefit.

Yes, I agree, I think the bindings we came up with are neater than having threads as multiple reg properties in cpu nodes and adding cluster nodes (within cpus node or elsewhere). On the interrupt affinity side I think that it should still be feasible since cpu nodes would become containers of threads, and cluster nodes containers (or pointed at by) of cpu nodes, but it is true that the description won't be uniform anymore.

I prefer our solution :-)

Quoted 29 lines
Quoted 29 lines

This would reduce the topology problem to where cluster nodes should be defined, either in the cpus node or a separate node (ie cpu-map :-)).

Quoted 25 lines

Also:
[2] If other more complex CPU topographies are designed, the binding
for the CPU must describe the topography

That's rather less helpful, but the suggestion is clear enough in that such information should be in the cpu node and specific to that CPU's binding. For ARM, we can have some global extensions to the CPU node.

The problems start when you want to refer to clusters and groups of CPUs from other nodes. Only individual cpu nodes can be places in the cpus node, so there is no node for a phandle to point at.

If you want to describe how other things like power, clock and coherency domains map to clusters and larger entities, things could get pretty awkward.

Keeping the topology description separate allows all topological entities to appear as addressable entities in the DT; otherwise, a cryptic convention is needed.

Hybrid approaches might be possible, putting cpu nodes into /cpus, and giving them a "parent" property where appropriate pointing at the relevant cluster node, which we put elsewhere in the DT.

Quoted 9 lines
Quoted 5 lines

That's what I did, with a couple of twists:

http://lists.infradead.org/pipermail/linux-arm-kernel/2012-January/080873.html

I have no preference, time to make a decision though.

With the arguments above, I'm not sure this is really better than the current proposal...

I am not sure either, that's why I would like to hear other opinions as well.

Quoted 11 lines
Quoted 11 lines
Quoted 2 lines

I'm not sure whether any of these approaches is an awful lot less ugly or more easy to handle than what it currently proposed though.

+1

Quoted 6 lines

The global binding for all ARM CPUs would specify that the topology is described by /cpu-map and its associated binding. For my interpretation of [2], this is a compliant approach. ePAPR does not specify _how_ the cpu node binding achieves a description of the topography, just that it must achieve it. There's no statement to say that it must not involve other nodes or bindings.

I agree with you, the SMT thread specification in the ePAPR would become a bit confusing though for ARM, we must make sure that what we are doing and the ePAPR spec evolve in concert otherwise this will become unmanageable in the long run.

Quoted 7 lines
Quoted 2 lines

Again, I think it all boils down to deciding where cluster nodes should live.

If we want to be able to describe affinities and other hardware linkages, describing the real hardware units as nodes still feels "right". ePAPR doesn't insist upon how this is done, so we do have choice.

Quoted 5 lines

The older/hybrid proposals seem to require different means of describing linkage depending on whether the target is a topological leaf or not.

I guess the question should be "what advantage is gained from describing this stuff in the cpus node?"

We do not need phandles (pointers to cpu nodes) to describe the topology, I guess, is the only answer.
Backward compatibility is still my main worry as far as the ePAPR is concerned, so I am still looking forward to getting feedback from powerPC and DT people on our proposal.

Thanks a lot,
Lorenzo

author maintainer
#12 lore raw

RFC PATCH 2/11 arm: add functions to parse cpu affinity from dt

In reply to Mark Rutland

Depending on hardware configuration, some devices may only be accessible from certain CPUs, may have interrupts wired up to a subset of CPUs, or may have operations which affect subsets of CPUs. To handle these devices it is necessary to describe this affinity information in devicetree.

This patch adds functions to handle parsing the CPU affinity of properties from devicetree, based on Lorenzo's topology binding, allowing subsets of CPUs to be associated with interrupts, hardware ports, etc. The functions can be used to build cpumasks and also to test whether an affinity property only targets one CPU independent of the current configuration (e.g. when the kernel supports fewer CPUs than are physically present). This is useful for dealing with mixed SPI/PPI devices.

A device may have an arbitrary number of affinity properties, the meaning of which is device-specific and should be specified in a given device's binding document.

For example, an affinity property describing interrupt routing may consist of a phandle pointing to a subtree of the topology nodes, indicating the set of CPUs an interrupt originates from or may be taken on. Bindings may have restrictions on the topology nodes referenced - for describing coherency controls an affinity property may indicate a whole cluster (uncluding any non-CPU logic it contains) is affected by some configuration.

---
 arch/arm/include/asm/dt_affinity.h |  12 ++++
 arch/arm/kernel/devtree.c          | 139 +++++++++++++++++++++++++++++++++++++
 2 files changed, 151 insertions(+)
 create mode 100644 arch/arm/include/asm/dt_affinity.h
arch/arm/include/asm/dt_affinity.h new file +12 -0
new file mode 100644
index 0000000..3acc7ce
@@ -0,0 +1,12 @@
+#ifndef ARM_ASM_DT_IRQ_H
+#define ARM_ASM_DT_IRQ_H
+
+#include <linux/cpumask.h>
+#include <linux/of.h>
+#include <linux/types.h>
+
+int arm_dt_affine_get_mask(struct device_node *node, char *prop,
+			   int idx, cpumask_t *mask);
+bool arm_dt_affine_is_single(struct device_node *node, char *prop,
+			     int idx);
+#endif /* ARM_ASM_DT_IRQ_H */
arch/arm/kernel/devtree.c +139 -0

151 lines not on this page · raw message

Signature · 1 line
1.8.1.1
author maintainer
#13 lore raw

RFC PATCH 3/11 arm: perf: clean up PMU names

In reply to Mark Rutland

The perf userspace tools can't handle dashes or spaces in PMU names, which conflicts with the current naming scheme in the arm perf backend. This prevents these PMUs from being accessed by name from the perf tools. Additionally the ARMv6 pmus are named "v6", which does not fully distinguish them in the sys/bus/event_source namespace.

This patch replaces the spaces and dashes in pmu names with underscores, and renames "v6" => "ARMv6" to better describe the PMU to userspace. The oprofile name conversion code is updated to handle this.

---
 arch/arm/kernel/perf_event_v6.c |  4 ++--
 arch/arm/kernel/perf_event_v7.c | 10 +++++-----
 arch/arm/oprofile/common.c      |  8 ++++----
 3 files changed, 11 insertions(+), 11 deletions(-)
arch/arm/kernel/perf_event_v6.c +2 -2
index 03664b0..e2e5795 100644
@@ -655,7 +655,7 @@ static int armv6_map_event(struct perf_event *event)
 
 static int armv6pmu_init(struct arm_pmu *cpu_pmu)
 {
-	cpu_pmu->name		= "v6";
+	cpu_pmu->name		= "ARMv6";
 	cpu_pmu->handle_irq	= armv6pmu_handle_irq;
 	cpu_pmu->enable		= armv6pmu_enable_event;
 	cpu_pmu->disable	= armv6pmu_disable_event;
@@ -687,7 +687,7 @@ static int armv6mpcore_map_event(struct perf_event *event)
 
 static int armv6mpcore_pmu_init(struct arm_pmu *cpu_pmu)
 {
-	cpu_pmu->name		= "v6mpcore";
+	cpu_pmu->name		= "ARMv6mpcore";
 	cpu_pmu->handle_irq	= armv6pmu_handle_irq;
 	cpu_pmu->enable		= armv6pmu_enable_event;
 	cpu_pmu->disable	= armv6mpcore_pmu_disable_event;
arch/arm/kernel/perf_event_v7.c +5 -5
index 039cffb..16a0455 100644
@@ -1240,7 +1240,7 @@ static u32 armv7_read_num_pmnc_events(void)
 static int armv7_a8_pmu_init(struct arm_pmu *cpu_pmu)
 {
 	armv7pmu_init(cpu_pmu);
-	cpu_pmu->name		= "ARMv7 Cortex-A8";
+	cpu_pmu->name		= "ARMv7_Cortex_A8";
 	cpu_pmu->map_event	= armv7_a8_map_event;
 	cpu_pmu->num_events	= armv7_read_num_pmnc_events();
 	return 0;
@@ -1249,7 +1249,7 @@ static int armv7_a8_pmu_init(struct arm_pmu *cpu_pmu)
 static int armv7_a9_pmu_init(struct arm_pmu *cpu_pmu)
 {
 	armv7pmu_init(cpu_pmu);
-	cpu_pmu->name		= "ARMv7 Cortex-A9";
+	cpu_pmu->name		= "ARMv7_Cortex_A9";
 	cpu_pmu->map_event	= armv7_a9_map_event;
 	cpu_pmu->num_events	= armv7_read_num_pmnc_events();
 	return 0;
@@ -1258,7 +1258,7 @@ static int armv7_a9_pmu_init(struct arm_pmu *cpu_pmu)
 static int armv7_a5_pmu_init(struct arm_pmu *cpu_pmu)
 {
 	armv7pmu_init(cpu_pmu);
-	cpu_pmu->name		= "ARMv7 Cortex-A5";
+	cpu_pmu->name		= "ARMv7_Cortex_A5";
 	cpu_pmu->map_event	= armv7_a5_map_event;
 	cpu_pmu->num_events	= armv7_read_num_pmnc_events();
 	return 0;
@@ -1267,7 +1267,7 @@ static int armv7_a5_pmu_init(struct arm_pmu *cpu_pmu)
 static int armv7_a15_pmu_init(struct arm_pmu *cpu_pmu)
 {
 	armv7pmu_init(cpu_pmu);
-	cpu_pmu->name		= "ARMv7 Cortex-A15";
+	cpu_pmu->name		= "ARMv7_Cortex_A15";
 	cpu_pmu->map_event	= armv7_a15_map_event;
 	cpu_pmu->num_events	= armv7_read_num_pmnc_events();
 	cpu_pmu->set_event_filter = armv7pmu_set_event_filter;
@@ -1277,7 +1277,7 @@ static int armv7_a15_pmu_init(struct arm_pmu *cpu_pmu)
 static int armv7_a7_pmu_init(struct arm_pmu *cpu_pmu)
 {
 	armv7pmu_init(cpu_pmu);
-	cpu_pmu->name		= "ARMv7 Cortex-A7";
+	cpu_pmu->name		= "ARMv7_Cortex_A7";
 	cpu_pmu->map_event	= armv7_a7_map_event;
 	cpu_pmu->num_events	= armv7_read_num_pmnc_events();
 	cpu_pmu->set_event_filter = armv7pmu_set_event_filter;
arch/arm/oprofile/common.c +4 -4
index 99c63d4b..31be981 100644
@@ -35,10 +35,10 @@ static struct op_perf_name {
 } op_perf_name_map[] = {
 	{ "xscale1",		"arm/xscale1"	},
 	{ "xscale1",		"arm/xscale2"	},
-	{ "v6",			"arm/armv6"	},
-	{ "v6mpcore",		"arm/mpcore"	},
-	{ "ARMv7 Cortex-A8",	"arm/armv7"	},
-	{ "ARMv7 Cortex-A9",	"arm/armv7-ca9"	},
+	{ "ARMv6",		"arm/armv6"	},
+	{ "ARMv6mpcore",	"arm/mpcore"	},
+	{ "ARMv7_Cortex_A8",	"arm/armv7"	},
+	{ "ARMv7_Cortex_A9",	"arm/armv7-ca9"	},
 };
 
 char *op_name_from_perf_id(void)
Signature · 1 line
1.8.1.1
author maintainer
#14 lore raw

RFC PATCH 4/11 arm: perf: use IDR types for CPU PMUs

In reply to Mark Rutland

For systems with heterogeneous CPUs (e.g. big.LITTLE systems) the PMUs can be different in each cluster, and not all events can be migrated between clusters. To allow userspace to deal with this, it must be possible to address each PMU independently.

This patch changes PMUs to be registered with dynamic (IDR) types, allowing them to be targeted individually. Each PMU's type can be found in ${SYSFS_ROOT}/bus/event_source/devices/${PMU_NAME}/type.

Quoted 1 line

From userspace, raw events can be targeted at a specific PMU:

$ perf stat -e ${PMU_NAME}/config=V,config1=V1,.../

Doing this does not break existing tools which use existing perf types:
when perf core can't find a PMU of matching type (in perf_init_event) it'll iterate over the set of all PMUs. If a compatible PMU exists, it'll be found eventually. If more than one compatible PMU exists, the event will be handled by whichever PMU happens to be earlier in the pmus list (which currently will be the last compatible PMU registered).

---
 arch/arm/kernel/perf_event.c     | 6 +++++-
 arch/arm/kernel/perf_event_cpu.c | 2 +-
 2 files changed, 6 insertions(+), 2 deletions(-)
arch/arm/kernel/perf_event.c +5 -1
index 146157d..3e20fa5 100644
@@ -73,8 +73,12 @@ armpmu_map_event(struct perf_event *event,
 		 u32 raw_event_mask)
 {
 	u64 config = event->attr.config;
+	int type = event->attr.type;
 
-	switch (event->attr.type) {
+	if (type >= PERF_TYPE_MAX && type == event->pmu->type)
+		return armpmu_map_raw_event(raw_event_mask, config);
+
+	switch (type) {
 	case PERF_TYPE_HARDWARE:
 		return armpmu_map_hw_event(event_map, config);
 	case PERF_TYPE_HW_CACHE:
arch/arm/kernel/perf_event_cpu.c +1 -1
index 1f2740e..31e9c41 100644
@@ -284,7 +284,7 @@ static int cpu_pmu_device_probe(struct platform_device *pdev)
 	cpu_pmu = pmu;
 	cpu_pmu->plat_device = pdev;
 	cpu_pmu_init(cpu_pmu);
-	ret = armpmu_register(cpu_pmu, PERF_TYPE_RAW);
+	ret = armpmu_register(cpu_pmu, -1);
 
 	if (!ret)
 		return 0;
Signature · 1 line
1.8.1.1
author maintainer
#15 lore raw

RFC PATCH 5/11 arm: perf: make get_hw_events take arm_pmu

In reply to Mark Rutland

Currently get_hw_events can only handle one PMU of a particular type, as it doesn't receive any input to differentiate instances. To support multiple instances of similar PMUs, each with their own set of pmu_hw_events structures, get_hw_events must be passed some value that can be used to find the correct set of pmu_hw_events structures.

This patch changes the interface of get_hw_events to take a struct arm_pmu pointer, to enable future support for multiple PMUs.

---
 arch/arm/include/asm/pmu.h          |  2 +-
 arch/arm/kernel/perf_event.c        |  6 +++---
 arch/arm/kernel/perf_event_cpu.c    |  2 +-
 arch/arm/kernel/perf_event_v6.c     | 12 ++++++------
 arch/arm/kernel/perf_event_v7.c     | 10 +++++-----
 arch/arm/kernel/perf_event_xscale.c | 20 ++++++++++----------
 6 files changed, 26 insertions(+), 26 deletions(-)
arch/arm/include/asm/pmu.h +1 -1
index f24edad..56724b2 100644
@@ -86,7 +86,7 @@ struct arm_pmu {
 	struct mutex	reserve_mutex;
 	u64		max_period;
 	struct platform_device	*plat_device;
-	struct pmu_hw_events	*(*get_hw_events)(void);
+	struct pmu_hw_events	*(*get_hw_events)(struct arm_pmu *pmu);
 };
 
 #define to_arm_pmu(p) (container_of(p, struct arm_pmu, pmu))
arch/arm/kernel/perf_event.c +3 -3
index 3e20fa5..5bcd8ac 100644
@@ -201,7 +201,7 @@ static void
 armpmu_del(struct perf_event *event, int flags)
 {
 	struct arm_pmu *armpmu = to_arm_pmu(event->pmu);
-	struct pmu_hw_events *hw_events = armpmu->get_hw_events();
+	struct pmu_hw_events *hw_events = armpmu->get_hw_events(armpmu);
 	struct hw_perf_event *hwc = &event->hw;
 	int idx = hwc->idx;
 
@@ -216,7 +216,7 @@ static int
 armpmu_add(struct perf_event *event, int flags)
 {
 	struct arm_pmu *armpmu = to_arm_pmu(event->pmu);
-	struct pmu_hw_events *hw_events = armpmu->get_hw_events();
+	struct pmu_hw_events *hw_events = armpmu->get_hw_events(armpmu);
 	struct hw_perf_event *hwc = &event->hw;
 	int idx;
 	int err = 0;
@@ -449,7 +449,7 @@ static int armpmu_event_init(struct perf_event *event)
 static void armpmu_enable(struct pmu *pmu)
 {
 	struct arm_pmu *armpmu = to_arm_pmu(pmu);
-	struct pmu_hw_events *hw_events = armpmu->get_hw_events();
+	struct pmu_hw_events *hw_events = armpmu->get_hw_events(armpmu);
 	int enabled = bitmap_weight(hw_events->used_mask, armpmu->num_events);
 
 	if (enabled)
arch/arm/kernel/perf_event_cpu.c +1 -1
index 31e9c41..80f1a26 100644
@@ -66,7 +66,7 @@ EXPORT_SYMBOL_GPL(perf_num_counters);
 #include "perf_event_v6.c"
 #include "perf_event_v7.c"
 
-static struct pmu_hw_events *cpu_pmu_get_cpu_events(void)
+static struct pmu_hw_events *cpu_pmu_get_cpu_events(struct arm_pmu *pmu)
 {
 	return &__get_cpu_var(cpu_hw_events);
 }
arch/arm/kernel/perf_event_v6.c +6 -6
index e2e5795..1647716 100644
@@ -439,7 +439,7 @@ static void armv6pmu_enable_event(struct perf_event *event)
 	unsigned long val, mask, evt, flags;
 	struct arm_pmu *cpu_pmu = to_arm_pmu(event->pmu);
 	struct hw_perf_event *hwc = &event->hw;
-	struct pmu_hw_events *events = cpu_pmu->get_hw_events();
+	struct pmu_hw_events *events = cpu_pmu->get_hw_events(cpu_pmu);
 	int idx = hwc->idx;
 
 	if (ARMV6_CYCLE_COUNTER == idx) {
@@ -477,7 +477,7 @@ armv6pmu_handle_irq(int irq_num,
 	unsigned long pmcr = armv6_pmcr_read();
 	struct perf_sample_data data;
 	struct arm_pmu *cpu_pmu = (struct arm_pmu *)dev;
-	struct pmu_hw_events *cpuc = cpu_pmu->get_hw_events();
+	struct pmu_hw_events *cpuc = cpu_pmu->get_hw_events(cpu_pmu);
 	struct pt_regs *regs;
 	int idx;
 
@@ -533,7 +533,7 @@ armv6pmu_handle_irq(int irq_num,
 static void armv6pmu_start(struct arm_pmu *cpu_pmu)
 {
 	unsigned long flags, val;
-	struct pmu_hw_events *events = cpu_pmu->get_hw_events();
+	struct pmu_hw_events *events = cpu_pmu->get_hw_events(cpu_pmu);
 
 	raw_spin_lock_irqsave(&events->pmu_lock, flags);
 	val = armv6_pmcr_read();
@@ -545,7 +545,7 @@ static void armv6pmu_start(struct arm_pmu *cpu_pmu)
 static void armv6pmu_stop(struct arm_pmu *cpu_pmu)
 {
 	unsigned long flags, val;
-	struct pmu_hw_events *events = cpu_pmu->get_hw_events();
+	struct pmu_hw_events *events = cpu_pmu->get_hw_events(cpu_pmu);
 
 	raw_spin_lock_irqsave(&events->pmu_lock, flags);
 	val = armv6_pmcr_read();
@@ -586,7 +586,7 @@ static void armv6pmu_disable_event(struct perf_event *event)
 	unsigned long val, mask, evt, flags;
 	struct arm_pmu *cpu_pmu = to_arm_pmu(event->pmu);
 	struct hw_perf_event *hwc = &event->hw;
-	struct pmu_hw_events *events = cpu_pmu->get_hw_events();
+	struct pmu_hw_events *events = cpu_pmu->get_hw_events(cpu_pmu);
 	int idx = hwc->idx;
 
 	if (ARMV6_CYCLE_COUNTER == idx) {
@@ -621,7 +621,7 @@ static void armv6mpcore_pmu_disable_event(struct perf_event *event)
 	unsigned long val, mask, flags, evt = 0;
 	struct arm_pmu *cpu_pmu = to_arm_pmu(event->pmu);
 	struct hw_perf_event *hwc = &event->hw;
-	struct pmu_hw_events *events = cpu_pmu->get_hw_events();
+	struct pmu_hw_events *events = cpu_pmu->get_hw_events(cpu_pmu);
 	int idx = hwc->idx;
 
 	if (ARMV6_CYCLE_COUNTER == idx) {
arch/arm/kernel/perf_event_v7.c +5 -5
index 16a0455..f4d185e 100644
@@ -955,7 +955,7 @@ static void armv7pmu_enable_event(struct perf_event *event)
 	unsigned long flags;
 	struct hw_perf_event *hwc = &event->hw;
 	struct arm_pmu *cpu_pmu = to_arm_pmu(event->pmu);
-	struct pmu_hw_events *events = cpu_pmu->get_hw_events();
+	struct pmu_hw_events *events = cpu_pmu->get_hw_events(cpu_pmu);
 	int idx = hwc->idx;
 
 	if (!armv7_pmnc_counter_valid(cpu_pmu, idx)) {
@@ -1001,7 +1001,7 @@ static void armv7pmu_disable_event(struct perf_event *event)
 	unsigned long flags;
 	struct hw_perf_event *hwc = &event->hw;
 	struct arm_pmu *cpu_pmu = to_arm_pmu(event->pmu);
-	struct pmu_hw_events *events = cpu_pmu->get_hw_events();
+	struct pmu_hw_events *events = cpu_pmu->get_hw_events(cpu_pmu);
 	int idx = hwc->idx;
 
 	if (!armv7_pmnc_counter_valid(cpu_pmu, idx)) {
@@ -1033,7 +1033,7 @@ static irqreturn_t armv7pmu_handle_irq(int irq_num, void *dev)
 	u32 pmnc;
 	struct perf_sample_data data;
 	struct arm_pmu *cpu_pmu = (struct arm_pmu *)dev;
-	struct pmu_hw_events *cpuc = cpu_pmu->get_hw_events();
+	struct pmu_hw_events *cpuc = cpu_pmu->get_hw_events(cpu_pmu);
 	struct pt_regs *regs;
 	int idx;
 
@@ -1093,7 +1093,7 @@ static irqreturn_t armv7pmu_handle_irq(int irq_num, void *dev)
 static void armv7pmu_start(struct arm_pmu *cpu_pmu)
 {
 	unsigned long flags;
-	struct pmu_hw_events *events = cpu_pmu->get_hw_events();
+	struct pmu_hw_events *events = cpu_pmu->get_hw_events(cpu_pmu);
 
 	raw_spin_lock_irqsave(&events->pmu_lock, flags);
 	/* Enable all counters */
@@ -1104,7 +1104,7 @@ static void armv7pmu_start(struct arm_pmu *cpu_pmu)
 static void armv7pmu_stop(struct arm_pmu *cpu_pmu)
 {
 	unsigned long flags;
-	struct pmu_hw_events *events = cpu_pmu->get_hw_events();
+	struct pmu_hw_events *events = cpu_pmu->get_hw_events(cpu_pmu);
 
 	raw_spin_lock_irqsave(&events->pmu_lock, flags);
 	/* Disable all counters */
arch/arm/kernel/perf_event_xscale.c +10 -10
index 63990c4..cd670ea 100644
@@ -225,7 +225,7 @@ xscale1pmu_handle_irq(int irq_num, void *dev)
 	unsigned long pmnc;
 	struct perf_sample_data data;
 	struct arm_pmu *cpu_pmu = (struct arm_pmu *)dev;
-	struct pmu_hw_events *cpuc = cpu_pmu->get_hw_events();
+	struct pmu_hw_events *cpuc = cpu_pmu->get_hw_events(cpu_pmu);
 	struct pt_regs *regs;
 	int idx;
 
@@ -285,7 +285,7 @@ static void xscale1pmu_enable_event(struct perf_event *event)
 	unsigned long val, mask, evt, flags;
 	struct arm_pmu *cpu_pmu = to_arm_pmu(event->pmu);
 	struct hw_perf_event *hwc = &event->hw;
-	struct pmu_hw_events *events = cpu_pmu->get_hw_events();
+	struct pmu_hw_events *events = cpu_pmu->get_hw_events(cpu_pmu);
 	int idx = hwc->idx;
 
 	switch (idx) {
@@ -321,7 +321,7 @@ static void xscale1pmu_disable_event(struct perf_event *event)
 	unsigned long val, mask, evt, flags;
 	struct arm_pmu *cpu_pmu = to_arm_pmu(event->pmu);
 	struct hw_perf_event *hwc = &event->hw;
-	struct pmu_hw_events *events = cpu_pmu->get_hw_events();
+	struct pmu_hw_events *events = cpu_pmu->get_hw_events(cpu_pmu);
 	int idx = hwc->idx;
 
 	switch (idx) {
@@ -374,7 +374,7 @@ xscale1pmu_get_event_idx(struct pmu_hw_events *cpuc,
 static void xscale1pmu_start(struct arm_pmu *cpu_pmu)
 {
 	unsigned long flags, val;
-	struct pmu_hw_events *events = cpu_pmu->get_hw_events();
+	struct pmu_hw_events *events = cpu_pmu->get_hw_events(cpu_pmu);
 
 	raw_spin_lock_irqsave(&events->pmu_lock, flags);
 	val = xscale1pmu_read_pmnc();
@@ -386,7 +386,7 @@ static void xscale1pmu_start(struct arm_pmu *cpu_pmu)
 static void xscale1pmu_stop(struct arm_pmu *cpu_pmu)
 {
 	unsigned long flags, val;
-	struct pmu_hw_events *events = cpu_pmu->get_hw_events();
+	struct pmu_hw_events *events = cpu_pmu->get_hw_events(cpu_pmu);
 
 	raw_spin_lock_irqsave(&events->pmu_lock, flags);
 	val = xscale1pmu_read_pmnc();
@@ -572,7 +572,7 @@ xscale2pmu_handle_irq(int irq_num, void *dev)
 	unsigned long pmnc, of_flags;
 	struct perf_sample_data data;
 	struct arm_pmu *cpu_pmu = (struct arm_pmu *)dev;
-	struct pmu_hw_events *cpuc = cpu_pmu->get_hw_events();
+	struct pmu_hw_events *cpuc = cpu_pmu->get_hw_events(cpu_pmu);
 	struct pt_regs *regs;
 	int idx;
 
@@ -626,7 +626,7 @@ static void xscale2pmu_enable_event(struct perf_event *event)
 	unsigned long flags, ien, evtsel;
 	struct arm_pmu *cpu_pmu = to_arm_pmu(event->pmu);
 	struct hw_perf_event *hwc = &event->hw;
-	struct pmu_hw_events *events = cpu_pmu->get_hw_events();
+	struct pmu_hw_events *events = cpu_pmu->get_hw_events(cpu_pmu);
 	int idx = hwc->idx;
 
 	ien = xscale2pmu_read_int_enable();
@@ -672,7 +672,7 @@ static void xscale2pmu_disable_event(struct perf_event *event)
 	unsigned long flags, ien, evtsel, of_flags;
 	struct arm_pmu *cpu_pmu = to_arm_pmu(event->pmu);
 	struct hw_perf_event *hwc = &event->hw;
-	struct pmu_hw_events *events = cpu_pmu->get_hw_events();
+	struct pmu_hw_events *events = cpu_pmu->get_hw_events(cpu_pmu);
 	int idx = hwc->idx;
 
 	ien = xscale2pmu_read_int_enable();
@@ -738,7 +738,7 @@ out:
 static void xscale2pmu_start(struct arm_pmu *cpu_pmu)
 {
 	unsigned long flags, val;
-	struct pmu_hw_events *events = cpu_pmu->get_hw_events();
+	struct pmu_hw_events *events = cpu_pmu->get_hw_events(cpu_pmu);
 
 	raw_spin_lock_irqsave(&events->pmu_lock, flags);
 	val = xscale2pmu_read_pmnc() & ~XSCALE_PMU_CNT64;
@@ -750,7 +750,7 @@ static void xscale2pmu_start(struct arm_pmu *cpu_pmu)
 static void xscale2pmu_stop(struct arm_pmu *cpu_pmu)
 {
 	unsigned long flags, val;
-	struct pmu_hw_events *events = cpu_pmu->get_hw_events();
+	struct pmu_hw_events *events = cpu_pmu->get_hw_events(cpu_pmu);
 
 	raw_spin_lock_irqsave(&events->pmu_lock, flags);
 	val = xscale2pmu_read_pmnc();
Signature · 1 line
1.8.1.1
author maintainer
#16 lore raw

RFC PATCH 6/11 arm: perf: dynamically allocate cpu hardware data

In reply to Mark Rutland

To support multiple PMUs, each PMU will need its own accounting data. As we don't know how (in general) many PMUs we'll have to support at compile-time, we must allocate the data at runtime dynamically

---
 arch/arm/kernel/perf_event_cpu.c | 73 ++++++++++++++++++++++++++++------------
 1 file changed, 51 insertions(+), 22 deletions(-)
arch/arm/kernel/perf_event_cpu.c +51 -22
index 80f1a26..ae5be75 100644
@@ -33,9 +33,25 @@
 /* Set at runtime when we know what CPU type we are. */
 static struct arm_pmu *cpu_pmu;
 
-static DEFINE_PER_CPU(struct perf_event * [ARMPMU_MAX_HWEVENTS], hw_events);
-static DEFINE_PER_CPU(unsigned long [BITS_TO_LONGS(ARMPMU_MAX_HWEVENTS)], used_mask);
-static DEFINE_PER_CPU(struct pmu_hw_events, cpu_hw_events);
+/*
+ * All of the dynamically sized pmu_hw data for the number of events supported
+ * by CPU PMUs, aggregated together for easier allocation / freeing.
+ */
+struct cpu_pmu_hw {
+	struct pmu_hw_events cpu_hw_events;
+	struct perf_event *hw_events[ARMPMU_MAX_HWEVENTS];
+	unsigned long used_mask[BITS_TO_LONGS(ARMPMU_MAX_HWEVENTS)];
+};
+
+/*
+ * For mapping between an arm_pmu for a CPU and its CPU-affine data.
+ */
+struct cpu_pmu {
+	struct arm_pmu armpmu;
+	struct cpu_pmu_hw __percpu *cpu_hw;
+};
+
+#define to_cpu_pmu(p) (container_of(p, struct cpu_pmu, armpmu))
 
 /*
  * Despite the names, these two functions are CPU-specific and are used
@@ -68,7 +84,9 @@ EXPORT_SYMBOL_GPL(perf_num_counters);
 
 static struct pmu_hw_events *cpu_pmu_get_cpu_events(struct arm_pmu *pmu)
 {
-	return &__get_cpu_var(cpu_hw_events);
+	struct cpu_pmu *cpu_pmu = to_cpu_pmu(pmu);
+	struct cpu_pmu_hw *hw = this_cpu_ptr(cpu_pmu->cpu_hw);
+	return &hw->cpu_hw_events;
 }
 
 static void cpu_pmu_free_irq(struct arm_pmu *cpu_pmu)
@@ -132,23 +150,25 @@ static int cpu_pmu_request_irq(struct arm_pmu *cpu_pmu, irq_handler_t handler)
 	return 0;
 }
 
-static void cpu_pmu_init(struct arm_pmu *cpu_pmu)
+static void cpu_pmu_init(struct cpu_pmu *cpu_pmu)
 {
 	int cpu;
+	struct arm_pmu *arm_pmu = &cpu_pmu->armpmu;
+
 	for_each_possible_cpu(cpu) {
-		struct pmu_hw_events *events = &per_cpu(cpu_hw_events, cpu);
-		events->events = per_cpu(hw_events, cpu);
-		events->used_mask = per_cpu(used_mask, cpu);
-		raw_spin_lock_init(&events->pmu_lock);
+		struct cpu_pmu_hw *cpu_hw = per_cpu_ptr(cpu_pmu->cpu_hw, cpu);
+		cpu_hw->cpu_hw_events.events = cpu_hw->hw_events;
+		cpu_hw->cpu_hw_events.used_mask = cpu_hw->used_mask;
+		raw_spin_lock_init(&cpu_hw->cpu_hw_events.pmu_lock);
 	}
 
-	cpu_pmu->get_hw_events	= cpu_pmu_get_cpu_events;
-	cpu_pmu->request_irq	= cpu_pmu_request_irq;
-	cpu_pmu->free_irq	= cpu_pmu_free_irq;
+	arm_pmu->get_hw_events	= cpu_pmu_get_cpu_events;
+	arm_pmu->request_irq	= cpu_pmu_request_irq;
+	arm_pmu->free_irq	= cpu_pmu_free_irq;
 
 	/* Ensure the PMU has sane values out of reset. */
-	if (cpu_pmu->reset)
-		on_each_cpu(cpu_pmu->reset, cpu_pmu, 1);
+	if (arm_pmu->reset)
+		on_each_cpu(arm_pmu->reset, arm_pmu, 1);
 }
 
 /*
@@ -255,7 +275,7 @@ static int cpu_pmu_device_probe(struct platform_device *pdev)
 	const struct of_device_id *of_id;
 	int (*init_fn)(struct arm_pmu *);
 	struct device_node *node = pdev->dev.of_node;
-	struct arm_pmu *pmu;
+	struct cpu_pmu *pmu;
 	int ret = -ENODEV;
 
 	if (cpu_pmu) {
@@ -263,34 +283,43 @@ static int cpu_pmu_device_probe(struct platform_device *pdev)
 		return -ENOSPC;
 	}
 
-	pmu = kzalloc(sizeof(struct arm_pmu), GFP_KERNEL);
+	pmu = kzalloc(sizeof(*pmu), GFP_KERNEL);
 	if (!pmu) {
 		pr_info("failed to allocate PMU device!");
 		return -ENOMEM;
 	}
 
+	pmu->cpu_hw = alloc_percpu(struct cpu_pmu_hw);
+	if (!pmu->cpu_hw) {
+		pr_info("failed to allocate PMU hw data!\n");
+		ret = -ENOMEM;
+		goto out_pmu;
+	}
+
 	if (node && (of_id = of_match_node(cpu_pmu_of_device_ids, pdev->dev.of_node))) {
 		init_fn = of_id->data;
-		ret = init_fn(pmu);
+		ret = init_fn(&pmu->armpmu);
 	} else {
-		ret = probe_current_pmu(pmu);
+		ret = probe_current_pmu(&pmu->armpmu);
 	}
 
 	if (ret) {
 		pr_info("failed to probe PMU!");
-		goto out_free;
+		goto out_hw;
 	}
 
-	cpu_pmu = pmu;
+	cpu_pmu = &pmu->armpmu;
 	cpu_pmu->plat_device = pdev;
-	cpu_pmu_init(cpu_pmu);
+	cpu_pmu_init(pmu);
 	ret = armpmu_register(cpu_pmu, -1);
 
 	if (!ret)
 		return 0;
 
-out_free:
+out_hw:
+	free_percpu(pmu->cpu_hw);
 	pr_info("failed to register PMU devices!");
+out_pmu:
 	kfree(pmu);
 	return ret;
 }
Signature · 1 line
1.8.1.1
author maintainer
#17 lore raw

RFC PATCH 7/11 arm: perf: treat PMUs as CPU affine

In reply to Mark Rutland

In multi-cluster systems, the PMUs can be different across clusters, and so our logical PMU may not be able to schedule events on all CPUs.

This patch adds a cpumask to encode which CPUs a PMU driver supports controlling events for, and limits the driver to scheduling events on those CPUs. Currently the cpumask is set to match all CPUs.

---
 arch/arm/include/asm/pmu.h       |  1 +
 arch/arm/kernel/perf_event.c     | 20 ++++++++++++++++++++
 arch/arm/kernel/perf_event_cpu.c | 10 +++++++++-
 3 files changed, 30 insertions(+), 1 deletion(-)
arch/arm/include/asm/pmu.h +1 -0
index 56724b2..4688692 100644
@@ -65,6 +65,7 @@ struct pmu_hw_events {
 struct arm_pmu {
 	struct pmu	pmu;
 	cpumask_t	active_irqs;
+	cpumask_t	supported_cpus;
 	char		*name;
 	irqreturn_t	(*handle_irq)(int irq_num, void *dev);
 	void		(*enable)(struct perf_event *event);
arch/arm/kernel/perf_event.c +20 -0
index 5bcd8ac..ebcd123 100644
@@ -12,6 +12,7 @@
  */
 #define pr_fmt(fmt) "hw perfevents: " fmt
 
+#include <linux/cpumask.h>
 #include <linux/kernel.h>
 #include <linux/platform_device.h>
 #include <linux/pm_runtime.h>
@@ -21,6 +22,11 @@
 #include <asm/pmu.h>
 #include <asm/stacktrace.h>
 
+static bool cpu_supported(struct arm_pmu *armpmu, int cpu)
+{
+	return cpumask_test_cpu(cpu, &armpmu->supported_cpus);
+}
+
 static int
 armpmu_map_cache_event(const unsigned (*cache_map)
 				      [PERF_COUNT_HW_CACHE_MAX]
@@ -221,6 +227,10 @@ armpmu_add(struct perf_event *event, int flags)
 	int idx;
 	int err = 0;
 
+	/* An event following a process won't be stopped earlier */
+	if (!cpu_supported(armpmu, smp_processor_id()))
+		return -ENOENT;
+
 	perf_pmu_disable(event->pmu);
 
 	/* If we don't have a space for the counter then finish early. */
@@ -417,6 +427,16 @@ static int armpmu_event_init(struct perf_event *event)
 	int err = 0;
 	atomic_t *active_events = &armpmu->active_events;
 
+	/*
+	 * Reject CPU-affine events for CPUs that are of a different class to
+	 * that which this PMU handles. Process-following events (where
+	 * event->cpu == -1) can be migrated between CPUs, and thus we have to
+	 * reject them later (in armpmu_add) if they're scheduled on a
+	 * different class of CPU.
+	 */
+	if (event->cpu != -1 && !cpu_supported(armpmu, event->cpu))
+		return -ENOENT;
+
 	/* does not support taken branch sampling */
 	if (has_branch_stack(event))
 		return -EOPNOTSUPP;
arch/arm/kernel/perf_event_cpu.c +9 -1
index ae5be75..c1a9880c 100644
@@ -168,7 +168,8 @@ static void cpu_pmu_init(struct cpu_pmu *cpu_pmu)
 
 	/* Ensure the PMU has sane values out of reset. */
 	if (arm_pmu->reset)
-		on_each_cpu(arm_pmu->reset, arm_pmu, 1);
+		on_each_cpu_mask(&arm_pmu->supported_cpus, arm_pmu->reset,
+				 arm_pmu, 1);
 }
 
 /*
@@ -180,9 +181,13 @@ static void cpu_pmu_init(struct cpu_pmu *cpu_pmu)
 static int __cpuinit cpu_pmu_notify(struct notifier_block *b,
 				    unsigned long action, void *hcpu)
 {
+	int cpu = (long)hcpu;
 	if ((action & ~CPU_TASKS_FROZEN) != CPU_STARTING)
 		return NOTIFY_DONE;
 
+	if (!cpumask_test_cpu(cpu, &cpu_pmu->supported_cpus))
+		return NOTIFY_DONE;
+
 	if (cpu_pmu && cpu_pmu->reset)
 		cpu_pmu->reset(cpu_pmu);
 	else
@@ -296,6 +301,9 @@ static int cpu_pmu_device_probe(struct platform_device *pdev)
 		goto out_pmu;
 	}
 
+	/* Assume by default that we're on a homogeneous system */
+	cpumask_setall(&pmu->armpmu.supported_cpus);
+
 	if (node && (of_id = of_match_node(cpu_pmu_of_device_ids, pdev->dev.of_node))) {
 		init_fn = of_id->data;
 		ret = init_fn(&pmu->armpmu);
Signature · 1 line
1.8.1.1
author maintainer
#18 lore raw

RFC PATCH 8/11 arm: perf: probe number of counters on affine CPUs

In reply to Mark Rutland

In heterogeneous systems, the number of counters may differ across clusters. To find the number of counters for a cluster, we must probe the PMU from a CPU in that cluster.

---
 arch/arm/kernel/perf_event_v7.c | 35 ++++++++++++++++++++---------------
 1 file changed, 20 insertions(+), 15 deletions(-)
arch/arm/kernel/perf_event_v7.c +20 -15
index f4d185e..519948f 100644
@@ -1226,15 +1226,25 @@ static void armv7pmu_init(struct arm_pmu *cpu_pmu)
 	cpu_pmu->max_period	= (1LLU << 32) - 1;
 };
 
-static u32 armv7_read_num_pmnc_events(void)
+static void armv7_read_num_pmnc_events(void *info)
 {
-	u32 nb_cnt;
+	int *nb_cnt = info;
 
 	/* Read the nb of CNTx counters supported from PMNC */
-	nb_cnt = (armv7_pmnc_read() >> ARMV7_PMNC_N_SHIFT) & ARMV7_PMNC_N_MASK;
+	*nb_cnt = (armv7_pmnc_read() >> ARMV7_PMNC_N_SHIFT) & ARMV7_PMNC_N_MASK;
 
-	/* Add the CPU cycles counter and return */
-	return nb_cnt + 1;
+	/* Add the CPU cycles counter */
+	*nb_cnt += 1;
+}
+
+static int armv7_probe_num_events(struct arm_pmu *arm_pmu)
+{
+	int ret;
+	ret = smp_call_function_any(&arm_pmu->supported_cpus,
+				    armv7_read_num_pmnc_events,
+				    &arm_pmu->num_events, 1);
+
+	return ret;
 }
 
 static int armv7_a8_pmu_init(struct arm_pmu *cpu_pmu)
@@ -1242,8 +1252,7 @@ static int armv7_a8_pmu_init(struct arm_pmu *cpu_pmu)
 	armv7pmu_init(cpu_pmu);
 	cpu_pmu->name		= "ARMv7_Cortex_A8";
 	cpu_pmu->map_event	= armv7_a8_map_event;
-	cpu_pmu->num_events	= armv7_read_num_pmnc_events();
-	return 0;
+	return armv7_probe_num_events(cpu_pmu);
 }
 
 static int armv7_a9_pmu_init(struct arm_pmu *cpu_pmu)
@@ -1251,8 +1260,7 @@ static int armv7_a9_pmu_init(struct arm_pmu *cpu_pmu)
 	armv7pmu_init(cpu_pmu);
 	cpu_pmu->name		= "ARMv7_Cortex_A9";
 	cpu_pmu->map_event	= armv7_a9_map_event;
-	cpu_pmu->num_events	= armv7_read_num_pmnc_events();
-	return 0;
+	return armv7_probe_num_events(cpu_pmu);
 }
 
 static int armv7_a5_pmu_init(struct arm_pmu *cpu_pmu)
@@ -1260,8 +1268,7 @@ static int armv7_a5_pmu_init(struct arm_pmu *cpu_pmu)
 	armv7pmu_init(cpu_pmu);
 	cpu_pmu->name		= "ARMv7_Cortex_A5";
 	cpu_pmu->map_event	= armv7_a5_map_event;
-	cpu_pmu->num_events	= armv7_read_num_pmnc_events();
-	return 0;
+	return armv7_probe_num_events(cpu_pmu);
 }
 
 static int armv7_a15_pmu_init(struct arm_pmu *cpu_pmu)
@@ -1269,9 +1276,8 @@ static int armv7_a15_pmu_init(struct arm_pmu *cpu_pmu)
 	armv7pmu_init(cpu_pmu);
 	cpu_pmu->name		= "ARMv7_Cortex_A15";
 	cpu_pmu->map_event	= armv7_a15_map_event;
-	cpu_pmu->num_events	= armv7_read_num_pmnc_events();
 	cpu_pmu->set_event_filter = armv7pmu_set_event_filter;
-	return 0;
+	return armv7_probe_num_events(cpu_pmu);
 }
 
 static int armv7_a7_pmu_init(struct arm_pmu *cpu_pmu)
@@ -1279,9 +1285,8 @@ static int armv7_a7_pmu_init(struct arm_pmu *cpu_pmu)
 	armv7pmu_init(cpu_pmu);
 	cpu_pmu->name		= "ARMv7_Cortex_A7";
 	cpu_pmu->map_event	= armv7_a7_map_event;
-	cpu_pmu->num_events	= armv7_read_num_pmnc_events();
 	cpu_pmu->set_event_filter = armv7pmu_set_event_filter;
-	return 0;
+	return armv7_probe_num_events(cpu_pmu);
 }
 #else
 static inline int armv7_a8_pmu_init(struct arm_pmu *cpu_pmu)
Signature · 1 line
1.8.1.1
author maintainer
#19 lore raw

RFC PATCH 9/11 arm: perf: parse cpu affinity from dt

In reply to Mark Rutland

The current way we read interrupts form devicetree assumes that interrupts are in increasing order of logical cpu id (MPIDR.Aff{2,1,0}), and that these logical ids are in a contiguous block. This may not be the case in general - after a kexec cpu ids may be arbitrarily assigned, and multi-cluster systems do not have a contiguous range of cpu ids.

This patch parses cpu affinity information for interrupts from an optional "interrupts-affinity" devicetree property described in the devicetree binding document. Currently only SPIs are supported.

---
 Documentation/devicetree/bindings/arm/pmu.txt |   7 ++
 arch/arm/kernel/perf_event_cpu.c              | 136 ++++++++++++++++++++------
 2 files changed, 115 insertions(+), 28 deletions(-)
Documentation/devicetree/bindings/arm/pmu.txt +7 -0
index 343781b..0caf968 100644
@@ -17,6 +17,13 @@ Required properties:
 	"arm,arm1136-pmu"
 - interrupts : 1 combined interrupt or 1 per core.
 
+Optional properties:
+
+- interrupts-affinity : a list of phandles to topology nodes as described in
+                        Documentation/devicetree/bindings/arm/topology.txt.
+			Each phandle describes the affinity of the element in
+			the interrupts property at the same index.
+
 Example:
 
 pmu {
arch/arm/kernel/perf_event_cpu.c +108 -28

199 lines not on this page · raw message

Signature · 1 line
1.8.1.1
author maintainer
#20 lore raw

RFC PATCH 10/11 arm: perf: allow multiple CPU PMUs to be registered

In reply to Mark Rutland

In systems with multiple clusters, the CPUs may be different in each cluster, and their PMUs may not be entirely compatible. As they cannot be managed as one logical homogeneous unit, we will have to support each set of compatible PMUs separately. To do so, we need to be able to register and use multiple PMUs simultaneously.

This patch replaces the global cpu_pmu pointer with a list of cpu_pmus, enabling multiple PMUs to be registered and used simultaneously.

---
 arch/arm/kernel/perf_event_cpu.c | 61 +++++++++++++++++++++++++---------------
 1 file changed, 38 insertions(+), 23 deletions(-)
arch/arm/kernel/perf_event_cpu.c +38 -23
index 6d8cbb1..4d6bfbb 100644
@@ -37,7 +37,7 @@ struct cpu_pmu_irq {
 };
 
 /* Set at runtime when we know what CPU type we are. */
-static struct arm_pmu *cpu_pmu;
+static LIST_HEAD(pmus_list);
 
 /*
  * All of the dynamically sized pmu_hw data for the number of events supported
@@ -57,20 +57,28 @@ struct cpu_pmu {
 	struct cpu_pmu_hw __percpu *cpu_hw;
 	struct cpu_pmu_irq *interrupts;
 	int nr_irqs;
+	struct list_head list;
 };
 
 #define to_cpu_pmu(p) (container_of(p, struct cpu_pmu, armpmu))
 
+static void cpu_pmu_add(struct cpu_pmu *pmu)
+{
+	list_add_tail(&pmu->list, &pmus_list);
+}
+
 /*
  * Despite the names, these two functions are CPU-specific and are used
  * by the OProfile/perf code.
  */
 const char *perf_pmu_name(void)
 {
-	if (!cpu_pmu)
+	struct cpu_pmu *pmu = list_first_entry(&pmus_list,
+					       struct cpu_pmu, list);
+	if (!pmu)
 		return NULL;
 
-	return cpu_pmu->name;
+	return pmu->armpmu.name;
 }
 EXPORT_SYMBOL_GPL(perf_pmu_name);
 
@@ -78,8 +86,10 @@ int perf_num_counters(void)
 {
 	int max_events = 0;
 
-	if (cpu_pmu != NULL)
-		max_events = cpu_pmu->num_events;
+	struct cpu_pmu *pmu = list_first_entry(&pmus_list,
+					       struct cpu_pmu, list);
+	if (pmu != NULL)
+		max_events = pmu->armpmu.num_events;
 
 	return max_events;
 }
@@ -186,18 +196,26 @@ static int __cpuinit cpu_pmu_notify(struct notifier_block *b,
 				    unsigned long action, void *hcpu)
 {
 	int cpu = (long)hcpu;
+	int ret = NOTIFY_DONE;
+	struct cpu_pmu *cpu_pmu;
+
 	if ((action & ~CPU_TASKS_FROZEN) != CPU_STARTING)
-		return NOTIFY_DONE;
+		return ret;
 
-	if (!cpumask_test_cpu(cpu, &cpu_pmu->supported_cpus))
-		return NOTIFY_DONE;
+	list_for_each_entry(cpu_pmu, &pmus_list, list) {
+		struct arm_pmu *arm_pmu = &cpu_pmu->armpmu;
 
-	if (cpu_pmu && cpu_pmu->reset)
-		cpu_pmu->reset(cpu_pmu);
-	else
-		return NOTIFY_DONE;
+		if (!arm_pmu->reset)
+			continue;
+
+		if (!cpumask_test_cpu(cpu, &arm_pmu->supported_cpus))
+			continue;
 
-	return NOTIFY_OK;
+		arm_pmu->reset(arm_pmu);
+		ret = NOTIFY_OK;
+	}
+
+	return ret;
 }
 
 static struct notifier_block __cpuinitdata cpu_pmu_hotplug_notifier = {
@@ -360,11 +378,6 @@ static int cpu_pmu_device_probe(struct platform_device *pdev)
 	struct cpu_pmu *pmu;
 	int ret = -ENODEV;
 
-	if (cpu_pmu) {
-		pr_info("attempt to register multiple PMU devices!");
-		return -ENOSPC;
-	}
-
 	pmu = kzalloc(sizeof(*pmu), GFP_KERNEL);
 	if (!pmu) {
 		pr_info("failed to allocate PMU device!");
@@ -396,13 +409,15 @@ static int cpu_pmu_device_probe(struct platform_device *pdev)
 		goto out_hw;
 	}
 
-	cpu_pmu = &pmu->armpmu;
-	cpu_pmu->plat_device = pdev;
+	pmu->armpmu.plat_device = pdev;
 	cpu_pmu_init(pmu);
-	ret = armpmu_register(cpu_pmu, -1);
 
-	if (!ret)
-		return 0;
+	ret = armpmu_register(&pmu->armpmu, -1);
+	if (ret)
+		goto out_hw;
+
+	cpu_pmu_add(pmu);
+	return 0;
 
 out_hw:
 	free_percpu(pmu->cpu_hw);
Signature · 1 line
1.8.1.1
author maintainer
#21 lore raw

RFC PATCH 11/11 arm: dts: add all PMUs for A15x2 A7x3 coretile

In reply to Mark Rutland

Currently we only describe the A15 PMUs, as previously it was not possible to describe the A7 PMUs.

This patch adds a cpu-map node and interrupts-affinity information for the PMUs to the dts, enabling the use of both PMUs. It also corrects the unit addresses of the cpu nodes, which should match their reg properties. As the A15 PMU has different events than the A9 PMU, this patch also removes the "arm,cortex-a9-pmu" compatible string from the dts.

---
 arch/arm/boot/dts/vexpress-v2p-ca15_a7.dts | 44 ++++++++++++++++++++++++++----
 1 file changed, 39 insertions(+), 5 deletions(-)
arch/arm/boot/dts/vexpress-v2p-ca15_a7.dts +39 -5
index dfe371e..4117b1b 100644
@@ -45,25 +45,47 @@
 			reg = <1>;
 		};
 
-		cpu2: cpu at 2 {
+		cpu2: cpu at 100 {
 			device_type = "cpu";
 			compatible = "arm,cortex-a7";
 			reg = <0x100>;
 		};
 
-		cpu3: cpu at 3 {
+		cpu3: cpu at 101 {
 			device_type = "cpu";
 			compatible = "arm,cortex-a7";
 			reg = <0x101>;
 		};
 
-		cpu4: cpu at 4 {
+		cpu4: cpu at 102 {
 			device_type = "cpu";
 			compatible = "arm,cortex-a7";
 			reg = <0x102>;
 		};
 	};
 
+	cpu-map {
+		cluster0 {
+			core0_0: core0 {
+				cpu = <&cpu0>;
+			};
+			core0_1: core1 {
+				cpu = <&cpu1>;
+			};
+		};
+		cluster1 {
+			core1_0: core0 {
+				cpu = <&cpu2>;
+			};
+			core1_1: core1 {
+				cpu = <&cpu3>;
+			};
+			core1_2: core2 {
+				cpu = <&cpu4>;
+			};
+		};
+	};
+
 	memory at 80000000 {
 		device_type = "memory";
 		reg = <0 0x80000000 0 0x40000000>;
@@ -133,10 +155,22 @@
 			     <1 10 0xf08>;
 	};
 
-	pmu {
-		compatible = "arm,cortex-a15-pmu", "arm,cortex-a9-pmu";
+	pmu_a15s {
+		compatible = "arm,cortex-a15-pmu";
 		interrupts = <0 68 4>,
 			     <0 69 4>;
+		interrupts-affinity = <&core0_0>,
+				      <&core0_1>;
+	};
+
+	pmu_a7s {
+		compatible = "arm,cortex-a7-pmu";
+		interrupts = <0 128 4>,
+			     <0 129 4>,
+			     <0 130 4>;
+		interrupts-affinity = <&core1_0>,
+				      <&core1_1>,
+				      <&core1_2>;
 	};
 
 	oscclk6a: oscclk6a {
Signature · 1 line
1.8.1.1