arm64:numa: Add numa support for arm64 platforms.

Raising the maximum limit to 4096.
This is to accomadate up-coming higher multi-core platforms.

Signed-off-by Ganapatrao Kulkarni <ganapatrao.kulkarni-M3mlKVOIwJVv6pq1l3V1OdBPR1lH4CV8@public.gmane.org>

Raising the maximum limit to 4096.
This is to accomadate up-coming higher multi-core platforms.

Signed-off-by Ganapatrao Kulkarni <[email protected]>

DT bindings for numa map for memory, cores and IOs using arm,associativity device node property.

Signed-off-by Ganapatrao Kulkarni <[email protected]>

---
 Documentation/devicetree/bindings/arm/numa.txt | 198 +++++++++++++++++++++++++
 1 file changed, 198 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/arm/numa.txt

Documentation/devicetree/bindings/arm/numa.txt new file +23 -0

new file mode 100644
index 0000000..4f51e25
@@ -0,0 +1,198 @@
+==============================================================================
+NUMA binding description.
+==============================================================================
+
+==============================================================================
+1 - Introduction
+==============================================================================
+
+Systems employing a Non Uniform Memory Access (NUMA) architecture contain
+collections of hardware resources including processors, memory, and I/O buses,
+that comprise what is commonly known as a “NUMA node�.
+Processor accesses to memory within the local NUMA node is generally faster
+than processor accesses to memory outside of the local NUMA node.
+DT defines interfaces that allow the platform to convey NUMA node
+topology information to OS.
+
+==============================================================================
+2 - arm,associativity
+==============================================================================
+
+The mapping is done using arm,associativity device property.
+this property needs to be present in every device node which needs to to be
+mapped to numa nodes.

Documentation/devicetree/bindings/arm/numa.txt new file +19 -0

+==============================================================================
+3 - arm,associativity-reference-points
+==============================================================================
+This property is a set of 32-bit integers, each representing an index into
+the arm,associativity nodes. The first integer is the most significant
+NUMA boundary and the following are progressively less significant boundaries.
+There can be more than one level of NUMA.
+
+Ex:
+	arm,associativity-reference-points = <0 1>;
+	The board Id(index 0) used first to calculate the associativity (node
+	distance), then follows the  socket id(index 1).
+
+	arm,associativity-reference-points = <1 0>;
+	The socket Id(index 1) used first to calculate the associativity,
+	then follows the board id(index 0).
+
+	arm,associativity-reference-points = <0>;
+	Only the board Id(index 0) used to calculate the associativity.

DT bindings for numa map for memory, cores and IOs using arm,associativity device node property.

Signed-off-by Ganapatrao Kulkarni <[email protected]>

---
 Documentation/devicetree/bindings/arm/numa.txt | 198 +++++++++++++++++++++++++
 1 file changed, 198 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/arm/numa.txt

Documentation/devicetree/bindings/arm/numa.txt new file +23 -0

new file mode 100644
index 0000000..4f51e25
@@ -0,0 +1,198 @@
+==============================================================================
+NUMA binding description.
+==============================================================================
+
+==============================================================================
+1 - Introduction
+==============================================================================
+
+Systems employing a Non Uniform Memory Access (NUMA) architecture contain
+collections of hardware resources including processors, memory, and I/O buses,
+that comprise what is commonly known as a ???NUMA node???.
+Processor accesses to memory within the local NUMA node is generally faster
+than processor accesses to memory outside of the local NUMA node.
+DT defines interfaces that allow the platform to convey NUMA node
+topology information to OS.
+
+==============================================================================
+2 - arm,associativity
+==============================================================================
+
+The mapping is done using arm,associativity device property.
+this property needs to be present in every device node which needs to to be
+mapped to numa nodes.

Documentation/devicetree/bindings/arm/numa.txt new file +19 -0

+==============================================================================
+3 - arm,associativity-reference-points
+==============================================================================
+This property is a set of 32-bit integers, each representing an index into
+the arm,associativity nodes. The first integer is the most significant
+NUMA boundary and the following are progressively less significant boundaries.
+There can be more than one level of NUMA.
+
+Ex:
+	arm,associativity-reference-points = <0 1>;
+	The board Id(index 0) used first to calculate the associativity (node
+	distance), then follows the  socket id(index 1).
+
+	arm,associativity-reference-points = <1 0>;
+	The socket Id(index 1) used first to calculate the associativity,
+	then follows the board id(index 0).
+
+	arm,associativity-reference-points = <0>;
+	Only the board Id(index 0) used to calculate the associativity.

Quoted 38 lines · On Wednesday 31 December 2014 13:03:26 Ganapatrao Kulkarni wrote:

DT bindings for numa map for memory, cores and IOs using arm,associativity device node property.

Signed-off-by Ganapatrao Kulkarni <[email protected]>

---
 Documentation/devicetree/bindings/arm/numa.txt | 198 +++++++++++++++++++++++++
 1 file changed, 198 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/arm/numa.txt

Documentation/devicetree/bindings/arm/numa.txt new file +23 -0

new file mode 100644
index 0000000..4f51e25
@@ -0,0 +1,198 @@
+==============================================================================
+NUMA binding description.
+==============================================================================
+
+==============================================================================
+1 - Introduction
+==============================================================================
+
+Systems employing a Non Uniform Memory Access (NUMA) architecture contain
+collections of hardware resources including processors, memory, and I/O buses,
+that comprise what is commonly known as a ???NUMA node?? .
+Processor accesses to memory within the local NUMA node is generally faster
+than processor accesses to memory outside of the local NUMA node.
+DT defines interfaces that allow the platform to convey NUMA node
+topology information to OS.
+
+==============================================================================
+2 - arm,associativity
+==============================================================================
+
+The mapping is done using arm,associativity device property.
+this property needs to be present in every device node which needs to to be
+mapped to numa nodes.

Documentation/devicetree/bindings/arm/numa.txt line 33

Quoted 10 lines

Documentation/devicetree/bindings/arm/numa.txt new file +10 -0

+
+arm,associativity property is set of 32-bit integers. representing the
+board id, socket id and core id.
+
+ex:
+     /* board 0, socket 0, core 0 */
+     arm,associativity = <0 0 0x000>;
+
+     /* board 1, socket 0, core 8 */
+     arm,associativity = <1 0 0x08>;

This is way too specific to Cavium machines. Most other vendors will not (at first) have multiple boards or multiple sockets, but need to represent multiple clusters and/or SMT threads instead. Also the wording suggests that this is only relevant for NUMA, which I don't think is helpful because we will also want to describe the topology within one NUMA node for locality.

I think we should stick to the powerpc definition here and not define what the levels mean at the binding level. Something like:

"Each level of topology defines a boundary in the system at which a significant difference in performance can be measured between cross-device accesses within a single location and those spanning multiple locations. The first cell always contains the broadest subdivision within the system, while the last cell enumerates the individual devices, such as an SMT thread of a CPU, or a bus bridge within an SoC".

Quoted 24 lines

Documentation/devicetree/bindings/arm/numa.txt new file +23 -0

+==============================================================================
+3 - arm,associativity-reference-points
+==============================================================================
+This property is a set of 32-bit integers, each representing an index into
+the arm,associativity nodes. The first integer is the most significant
+NUMA boundary and the following are progressively less significant boundaries.
+There can be more than one level of NUMA.
+
+Ex:
+     arm,associativity-reference-points = <0 1>;
+     The board Id(index 0) used first to calculate the associativity (node
+     distance), then follows the  socket id(index 1).
+
+     arm,associativity-reference-points = <1 0>;
+     The socket Id(index 1) used first to calculate the associativity,
+     then follows the board id(index 0).
+
+     arm,associativity-reference-points = <0>;
+     Only the board Id(index 0) used to calculate the associativity.
+
+     arm,associativity-reference-points = <1>;
+     Only socket Id(index 1) used to calculate the associativity.
+

Documentation/devicetree/bindings/arm/numa.txt line 63

Quoted 6 lines

Documentation/devicetree/bindings/arm/numa.txt new file +6 -0

+==============================================================================
+4 - Example dts
+==============================================================================
+
+Example: 2 Node system consists of 2 boards and each board having one socket
+and 8 core in each socket.

I think the example should also include a PCI controller.

Documentation/devicetree/bindings/arm/numa.txt line 64

Quoted 3 lines

Documentation/devicetree/bindings/arm/numa.txt new file +2 -0

+
+     arm,associativity-reference-points = <0 1>;

This doesn't really match the associativity properties, because the second level in the cpus nodes is completely meaningless and should not be listed as a secondary reference point.

Arnd

arch/arm64/boot/dts/thunder-88xx-2n.dts new file +5 -0

+
+	memory@00000000 {
+		device_type = "memory";
+		reg = <0x0 0x00000000 0x0 0x80000000>;
+		/* board 0, socket 0, no specific core */

arch/arm64/boot/dts/thunder-88xx-2n.dtsi new file +4 -0

+		cpu@00f {
+			device_type = "cpu";
+			compatible = "cavium,thunder", "arm,armv8";
+			reg = <0x0 0x00f>;

arch/arm64/boot/dts/thunder-88xx-2n.dts new file +5 -0

+
+	memory at 00000000 {
+		device_type = "memory";
+		reg = <0x0 0x00000000 0x0 0x80000000>;
+		/* board 0, socket 0, no specific core */

arch/arm64/boot/dts/thunder-88xx-2n.dtsi new file +4 -0

+		cpu at 00f {
+			device_type = "cpu";
+			compatible = "cavium,thunder", "arm,armv8";
+			reg = <0x0 0x00f>;

Quoted 5 lines · On Wednesday 31 December 2014 13:03:27 Ganapatrao Kulkarni wrote:

arch/arm64/boot/dts/thunder-88xx-2n.dts new file +5 -0

+
+     memory at 00000000 {
+             device_type = "memory";
+             reg = <0x0 0x00000000 0x0 0x80000000>;
+             /* board 0, socket 0, no specific core */

arch/arm64/boot/dts/thunder-88xx-2n.dts line 78

Quoted 10 lines

arch/arm64/boot/dts/thunder-88xx-2n.dts new file +10 -0

+             arm,associativity = <0 0 0xffff>;
+     };
+
+     memory at 10000000000 {
+             device_type = "memory";
+             reg = <0x100 0x00000000 0x0 0x80000000>;
+             /* board 1, socket 0, no specific core */
+             arm,associativity = <1 0 0xffff>;
+     };
+};

So no memory in any other socket?

Quoted 4 lines

arch/arm64/boot/dts/thunder-88xx-2n.dtsi new file +4 -0

+             cpu at 00f {
+                     device_type = "cpu";
+                     compatible = "cavium,thunder", "arm,armv8";
+                     reg = <0x0 0x00f>;

arch/arm64/boot/dts/thunder-88xx-2n.dtsi line 184

Quoted 10 lines

arch/arm64/boot/dts/thunder-88xx-2n.dtsi new file +10 -0

+                     enable-method = "psci";
+                     arm,associativity = <0 0 0x00f>;
+             };
+             cpu at 100 {
+                     device_type = "cpu";
+                     compatible = "cavium,thunder", "arm,armv8";
+                     reg = <0x0 0x100>;
+                     enable-method = "psci";
+                     arm,associativity = <0 0 0x100>;
+             };

What is the 0x100 offset in the last-level topology field? Does this have no significance to topology at all? I would expect that to be something like cluster number that is relevant to caching and should be represented as a separate level.

In contrast, the level-two topology information seems to always be zero for all CPUs, so you could probably leave that one out.

arch/arm64/boot/dts/thunder-88xx-2n.dtsi line 753

Quoted 5 lines

arch/arm64/boot/dts/thunder-88xx-2n.dtsi new file +5 -0

+     soc {
+             compatible = "simple-bus";
+             #address-cells = <2>;
+             #size-cells = <2>;
+             ranges;

The soc node is missing a topology information, please add one.

Arnd

Quoted 16 lines · Arnd Bergmann

Quoted 4 lines · On Wednesday 31 December 2014 13:03:27 Ganapatrao Kulkarni wrote:

arch/arm64/boot/dts/thunder-88xx-2n.dtsi new file +4 -0

+             cpu at 00f {
+                     device_type = "cpu";
+                     compatible = "cavium,thunder", "arm,armv8";
+                     reg = <0x0 0x00f>;

arch/arm64/boot/dts/thunder-88xx-2n.dtsi line 184

Quoted 10 lines

arch/arm64/boot/dts/thunder-88xx-2n.dtsi new file +10 -0

+                     enable-method = "psci";
+                     arm,associativity = <0 0 0x00f>;
+             };
+             cpu at 100 {
+                     device_type = "cpu";
+                     compatible = "cavium,thunder", "arm,armv8";
+                     reg = <0x0 0x100>;
+                     enable-method = "psci";
+                     arm,associativity = <0 0 0x100>;
+             };

Quoted 4 lines

What is the 0x100 offset in the last-level topology field? Does this have no significance to topology at all? I would expect that to be something like cluster number that is relevant to caching and should be represented as a separate level.

i did not understand, can you please explain little more about " should be represented as a separate level."
at present, i have put the hwid of a cpu.

From what I undertand, the hwid of the CPU contains the "cluster" number in

Quoted 17 lines · On Tuesday 06 January 2015 15:04:26 Ganapatrao Kulkarni wrote:

Quoted 16 lines · Arnd Bergmann

Quoted 14 lines · On Wednesday 31 December 2014 13:03:27 Ganapatrao Kulkarni wrote:

arch/arm64/boot/dts/thunder-88xx-2n.dtsi new file +14 -0

+             cpu at 00f {
+                     device_type = "cpu";
+                     compatible = "cavium,thunder", "arm,armv8";
+                     reg = <0x0 0x00f>;
+                     enable-method = "psci";
+                     arm,associativity = <0 0 0x00f>;
+             };
+             cpu at 100 {
+                     device_type = "cpu";
+                     compatible = "cavium,thunder", "arm,armv8";
+                     reg = <0x0 0x100>;
+                     enable-method = "psci";
+                     arm,associativity = <0 0 0x100>;
+             };

Quoted 8 lines

Quoted 4 lines

What is the 0x100 offset in the last-level topology field? Does this have no significance to topology at all? I would expect that to be something like cluster number that is relevant to caching and should be represented as a separate level.

i did not understand, can you please explain little more about " should be represented as a separate level."
at present, i have put the hwid of a cpu.

From what I undertand, the hwid of the CPU contains the "cluster" number in this bit position, so you typically have a shared L2 or L3 cache between all cores within a cluster, but separate caches in other clusters.

If this is the case, there will be a measurable difference in performance between two processes sharing memory when running on the same cluster, or when running on different clusters on the same socket. If the performance difference is relevant, it should be described as a separate level in the associativity property.

Arnd

Hi Arnd,

Quoted 31 lines · Arnd Bergmann

Quoted 17 lines · On Tuesday 06 January 2015 15:04:26 Ganapatrao Kulkarni wrote:

Quoted 16 lines · Arnd Bergmann

Quoted 14 lines · On Wednesday 31 December 2014 13:03:27 Ganapatrao Kulkarni wrote:

arch/arm64/boot/dts/thunder-88xx-2n.dtsi new file +14 -0

+             cpu at 00f {
+                     device_type = "cpu";
+                     compatible = "cavium,thunder", "arm,armv8";
+                     reg = <0x0 0x00f>;
+                     enable-method = "psci";
+                     arm,associativity = <0 0 0x00f>;
+             };
+             cpu at 100 {
+                     device_type = "cpu";
+                     compatible = "cavium,thunder", "arm,armv8";
+                     reg = <0x0 0x100>;
+                     enable-method = "psci";
+                     arm,associativity = <0 0 0x100>;
+             };

Quoted 8 lines

Quoted 4 lines

What is the 0x100 offset in the last-level topology field? Does this have no significance to topology at all? I would expect that to be something like cluster number that is relevant to caching and should be represented as a separate level.

i did not understand, can you please explain little more about " should be represented as a separate level."
at present, i have put the hwid of a cpu.

From what I undertand, the hwid of the CPU contains the "cluster" number in this bit position, so you typically have a shared L2 or L3 cache between all cores within a cluster, but separate caches in other clusters.

Quoted 5 lines

If this is the case, there will be a measurable difference in performance between two processes sharing memory when running on the same cluster, or when running on different clusters on the same socket. If the performance difference is relevant, it should be described as a separate level in the associativity property.

you mean, the associativity as array of <board> <socket> <cluster>

Quoted 34 lines · On Wednesday 07 January 2015 12:37:51 Ganapatrao Kulkarni wrote:

Hi Arnd,

Quoted 31 lines · Arnd Bergmann

Quoted 17 lines · On Tuesday 06 January 2015 15:04:26 Ganapatrao Kulkarni wrote:

Quoted 16 lines · Arnd Bergmann

Quoted 14 lines · On Wednesday 31 December 2014 13:03:27 Ganapatrao Kulkarni wrote:

arch/arm64/boot/dts/thunder-88xx-2n.dtsi new file +14 -0

+             cpu at 00f {
+                     device_type = "cpu";
+                     compatible = "cavium,thunder", "arm,armv8";
+                     reg = <0x0 0x00f>;
+                     enable-method = "psci";
+                     arm,associativity = <0 0 0x00f>;
+             };
+             cpu at 100 {
+                     device_type = "cpu";
+                     compatible = "cavium,thunder", "arm,armv8";
+                     reg = <0x0 0x100>;
+                     enable-method = "psci";
+                     arm,associativity = <0 0 0x100>;
+             };

Quoted 8 lines

Quoted 4 lines

What is the 0x100 offset in the last-level topology field? Does this have no significance to topology at all? I would expect that to be something like cluster number that is relevant to caching and should be represented as a separate level.

i did not understand, can you please explain little more about " should be represented as a separate level."
at present, i have put the hwid of a cpu.

From what I undertand, the hwid of the CPU contains the "cluster" number in this bit position, so you typically have a shared L2 or L3 cache between all cores within a cluster, but separate caches in other clusters.

Quoted 6 lines

Quoted 5 lines

If this is the case, there will be a measurable difference in performance between two processes sharing memory when running on the same cluster, or when running on different clusters on the same socket. If the performance difference is relevant, it should be described as a separate level in the associativity property.

you mean, the associativity as array of <board> <socket> <cluster>

No, that would leave out the core number, which is required to identify the individual thread. I meant adding an extra level such as

<board> <socket> <cluster> <core>

A lot of machines will leave out the <board> number because they are built with SoCs that don't have a long-distance coherency protocol.

Quoted 45 lines · Arnd Bergmann

Quoted 34 lines · On Wednesday 07 January 2015 12:37:51 Ganapatrao Kulkarni wrote:

Hi Arnd,

Quoted 31 lines · Arnd Bergmann

Quoted 17 lines · On Tuesday 06 January 2015 15:04:26 Ganapatrao Kulkarni wrote:

Quoted 16 lines · Arnd Bergmann

Quoted 14 lines · On Wednesday 31 December 2014 13:03:27 Ganapatrao Kulkarni wrote:

(quoted diff) +14 -0

+             cpu at 00f {
+                     device_type = "cpu";
+                     compatible = "cavium,thunder", "arm,armv8";
+                     reg = <0x0 0x00f>;
+                     enable-method = "psci";
+                     arm,associativity = <0 0 0x00f>;
+             };
+             cpu at 100 {
+                     device_type = "cpu";
+                     compatible = "cavium,thunder", "arm,armv8";
+                     reg = <0x0 0x100>;
+                     enable-method = "psci";
+                     arm,associativity = <0 0 0x100>;
+             };

Quoted 8 lines

Quoted 4 lines

What is the 0x100 offset in the last-level topology field? Does this have no significance to topology at all? I would expect that to be something like cluster number that is relevant to caching and should be represented as a separate level.

i did not understand, can you please explain little more about " should be represented as a separate level."
at present, i have put the hwid of a cpu.

From what I undertand, the hwid of the CPU contains the "cluster" number in this bit position, so you typically have a shared L2 or L3 cache between all cores within a cluster, but separate caches in other clusters.

Quoted 6 lines

Quoted 5 lines

If this is the case, there will be a measurable difference in performance between two processes sharing memory when running on the same cluster, or when running on different clusters on the same socket. If the performance difference is relevant, it should be described as a separate level in the associativity property.

you mean, the associativity as array of <board> <socket> <cluster>

No, that would leave out the core number, which is required to identify the individual thread. I meant adding an extra level such as

Quoted 4 lines

<board> <socket> <cluster> <core>

A lot of machines will leave out the <board> number because they are built with SoCs that don't have a long-distance coherency protocol.

Can't we use phandles to cpu-map nodes instead of a list of numbers (and yet another topology binding description) ?

Is arm,associativity used solely to map "devices" (inclusive of caches) to a set of cpus ?

cpu-map misses a notion of distance between hierarchy layers, but we can add to that.

Lorenzo

Hi Lorenzo,

Quoted 51 lines · Lorenzo Pieralisi

Quoted 49 lines · Arnd Bergmann

Quoted 34 lines · On Wednesday 07 January 2015 12:37:51 Ganapatrao Kulkarni wrote:

Hi Arnd,

Quoted 31 lines · Arnd Bergmann

Quoted 17 lines · On Tuesday 06 January 2015 15:04:26 Ganapatrao Kulkarni wrote:

Quoted 16 lines · Arnd Bergmann

Quoted 14 lines · On Wednesday 31 December 2014 13:03:27 Ganapatrao Kulkarni wrote:

(quoted diff) +14 -0

+             cpu at 00f {
+                     device_type = "cpu";
+                     compatible = "cavium,thunder", "arm,armv8";
+                     reg = <0x0 0x00f>;
+                     enable-method = "psci";
+                     arm,associativity = <0 0 0x00f>;
+             };
+             cpu at 100 {
+                     device_type = "cpu";
+                     compatible = "cavium,thunder", "arm,armv8";
+                     reg = <0x0 0x100>;
+                     enable-method = "psci";
+                     arm,associativity = <0 0 0x100>;
+             };

Quoted 8 lines

Quoted 4 lines

What is the 0x100 offset in the last-level topology field? Does this have no significance to topology at all? I would expect that to be something like cluster number that is relevant to caching and should be represented as a separate level.

i did not understand, can you please explain little more about " should be represented as a separate level."
at present, i have put the hwid of a cpu.

From what I undertand, the hwid of the CPU contains the "cluster" number in this bit position, so you typically have a shared L2 or L3 cache between all cores within a cluster, but separate caches in other clusters.

Quoted 6 lines

Quoted 5 lines

If this is the case, there will be a measurable difference in performance between two processes sharing memory when running on the same cluster, or when running on different clusters on the same socket. If the performance difference is relevant, it should be described as a separate level in the associativity property.

you mean, the associativity as array of <board> <socket> <cluster>

No, that would leave out the core number, which is required to identify the individual thread. I meant adding an extra level such as

<board> <socket> <cluster> <core>

A lot of machines will leave out the <board> number because they are built with SoCs that don't have a long-distance coherency protocol.

Quoted 2 lines

Can't we use phandles to cpu-map nodes instead of a list of numbers (and yet another topology binding description) ?

cpu-map describes only a cpu topology.
infact, i have tried initially(in v1 patch set) to use topology for the numa mapping.
However, for numa, we need to define association of cpu, memory and IOs. arm,associativity is a generic node property and can be used in any dt nodes.

Quoted 11 lines · Ganapatrao Kulkarni

Quoted 9 lines · Lorenzo Pieralisi

Quoted 7 lines · Arnd Bergmann

No, that would leave out the core number, which is required to identify the individual thread. I meant adding an extra level such as

<board> <socket> <cluster> <core>

A lot of machines will leave out the <board> number because they are built with SoCs that don't have a long-distance coherency protocol.

Quoted 7 lines

Quoted 2 lines

Can't we use phandles to cpu-map nodes instead of a list of numbers (and yet another topology binding description) ?

cpu-map describes only a cpu topology.
infact, i have tried initially(in v1 patch set) to use topology for the numa mapping.
However, for numa, we need to define association of cpu, memory and IOs. arm,associativity is a generic node property and can be used in any dt nodes.

I understand that, I was advising to define "arm,associativity" as a phandle in cpu nodes AND all devices.

Why can't you make it point at a phandle in the cpu-map instead of adding a t-uple doing the same thing. Am I missing something here ?

cpu-map allows you to describe the system hierarchy and can expand beyond clusters (several layers of clusterings, above core it is just a way to define the system hierarchy, leaves node will always be cores or threads).

On a side note, one of the reasons cpu-map was devised for was exactly that, to allow mappings of resources (ie IRQs but it is valid for caches and other devices too) to groups of CPUs.

Is there anything that you can't do by using cpu-map phandles to describe devices associativity ?

We have to add bindings that allow to compute the distance as you do by using the reference points (I am reading the code to figure out how it is used), but that's feasible as a binding update.

Adding numa support for arm64 based platforms.
Adding dt node pasring for numa topology using property arm,associativity.

Signed-off-by Ganapatrao Kulkarni <ganapatrao.kulkarni-M3mlKVOIwJVv6pq1l3V1OdBPR1lH4CV8@public.gmane.org>

+#define MAX_DISTANCE_REF_POINTS 4

arch/arm64/kernel/dt_numa.c new file +16 -0

+int dt_get_cpu_node_id(int cpu)
+{
+	struct device_node *dn = NULL;
+
+	while ((dn = of_find_node_by_type(dn, "cpu"))) {
+		const u32 *cell;
+		u64 hwid;
+
+		/*
+		 * A cpu node with missing "reg" property is
+		 * considered invalid to build a cpu_logical_map
+		 * entry.
+		 */
+		cell = of_get_property(dn, "reg", NULL);
+		if (!cell) {
+			pr_err("%s: missing reg property\n", dn->full_name);

arch/arm64/kernel/dt_numa.c new file +11 -0

+/* DT node mapping is done already early_init_dt_scan_memory */
+int __init arm64_dt_numa_init(void)
+{
+	int i;
+	u32 nodea, nodeb, distance, node_count = 0;
+
+	of_scan_flat_dt(early_init_dt_scan_numa_map, NULL);
+
+	for_each_node_mask(i, numa_nodes_parsed)
+		node_count = i;
+	node_count++;

arch/arm64/mm/numa.c new file +9 -0

+
+/**
+ * arm64_numa_init - Initialize NUMA
+ *
+ * Try each configured NUMA initialization method until one succeeds.  The
+ * last fallback is dummy single node config encomapssing whole memory and
+ * never fails.
+ */
+void __init arm64_numa_init(void)

Adding numa support for arm64 based platforms.
Adding dt node pasring for numa topology using property arm,associativity.

Signed-off-by Ganapatrao Kulkarni <[email protected]>

+#define MAX_DISTANCE_REF_POINTS 4

arch/arm64/kernel/dt_numa.c new file +16 -0

+int dt_get_cpu_node_id(int cpu)
+{
+	struct device_node *dn = NULL;
+
+	while ((dn = of_find_node_by_type(dn, "cpu"))) {
+		const u32 *cell;
+		u64 hwid;
+
+		/*
+		 * A cpu node with missing "reg" property is
+		 * considered invalid to build a cpu_logical_map
+		 * entry.
+		 */
+		cell = of_get_property(dn, "reg", NULL);
+		if (!cell) {
+			pr_err("%s: missing reg property\n", dn->full_name);

arch/arm64/kernel/dt_numa.c new file +11 -0

+/* DT node mapping is done already early_init_dt_scan_memory */
+int __init arm64_dt_numa_init(void)
+{
+	int i;
+	u32 nodea, nodeb, distance, node_count = 0;
+
+	of_scan_flat_dt(early_init_dt_scan_numa_map, NULL);
+
+	for_each_node_mask(i, numa_nodes_parsed)
+		node_count = i;
+	node_count++;

arch/arm64/mm/numa.c new file +9 -0

+
+/**
+ * arm64_numa_init - Initialize NUMA
+ *
+ * Try each configured NUMA initialization method until one succeeds.  The
+ * last fallback is dummy single node config encomapssing whole memory and
+ * never fails.
+ */
+void __init arm64_numa_init(void)

[re-sent with correct mailing list address]

Quoted 4 lines · On Wednesday 31 December 2014 13:03:28 Ganapatrao Kulkarni wrote:

Adding numa support for arm64 based platforms.
Adding dt node pasring for numa topology using property arm,associativity.

Signed-off-by Ganapatrao Kulkarni <[email protected]>

Maybe the parts that are common with powerpc can be moved to drivers/of/numa.c? We can always look for both arm,associativity and ibm,associativity, I don't think we should be worried about any conflicts that way.

Quoted 2 lines

+#define MAX_DISTANCE_REF_POINTS 4

I think we should use 8 here like powerpc, four levels might get exceeded on complex SoCs.

Quoted 22 lines

arch/arm64/kernel/dt_numa.c new file +21 -0

+int dt_get_cpu_node_id(int cpu)
+{
+     struct device_node *dn = NULL;
+
+     while ((dn = of_find_node_by_type(dn, "cpu"))) {
+             const u32 *cell;
+             u64 hwid;
+
+             /*
+              * A cpu node with missing "reg" property is
+              * considered invalid to build a cpu_logical_map
+              * entry.
+              */
+             cell = of_get_property(dn, "reg", NULL);
+             if (!cell) {
+                     pr_err("%s: missing reg property\n", dn->full_name);
+                     return default_nid;
+             }
+             hwid = of_read_number(cell, of_n_addr_cells(dn));
+
+             if (cpu_logical_map(cpu) == hwid)

arch/arm64/kernel/dt_numa.c line 201

Quoted 5 lines

arch/arm64/kernel/dt_numa.c new file +5 -0

+             return of_node_to_nid_single(dn);
+     }
+     return NUMA_NO_NODE;
+}
+EXPORT_SYMBOL(dt_get_cpu_node_id);

Maybe just expose a function to the device node for a CPU ID here, and expect callers to use of_node_to_nid?

arch/arm64/kernel/dt_numa.c line 245

Quoted 11 lines

arch/arm64/kernel/dt_numa.c new file +10 -0

+
+/**
+ * early_init_dt_scan_numa_map - parse memory node and map nid to memory range.
+ */
+int __init early_init_dt_scan_numa_map(unsigned long node, const char *uname,
+                                  int depth, void *data)
+{
+     const char *type = of_get_flat_dt_prop(node, "device_type", NULL);
+
+     /* We are scanning "numa-map" nodes only */

a stale comment?

Quoted 18 lines

arch/arm64/kernel/dt_numa.c new file +17 -0

+/* DT node mapping is done already early_init_dt_scan_memory */
+int __init arm64_dt_numa_init(void)
+{
+     int i;
+     u32 nodea, nodeb, distance, node_count = 0;
+
+     of_scan_flat_dt(early_init_dt_scan_numa_map, NULL);
+
+     for_each_node_mask(i, numa_nodes_parsed)
+             node_count = i;
+     node_count++;
+
+     for (nodea =  0; nodea < node_count; nodea++) {
+             for (nodeb = 0; nodeb < node_count; nodeb++) {
+                     distance = dt_get_node_distance(nodea, nodeb);
+                     numa_set_distance(nodea, nodeb, distance);
+             }

arch/arm64/kernel/dt_numa.c line 296

Quoted 4 lines

arch/arm64/kernel/dt_numa.c new file +4 -0

+     }
+     return 0;
+}
+EXPORT_SYMBOL(arm64_dt_numa_init);

No need to export functions that are called only be architecture code. Since this works on the flattened device tree format, you can never have loadable modules calling it.

arch/arm64/kernel/setup.c static int c_show(struct seq_file *m, void *v) · line 474

Quoted 13 lines

arch/arm64/kernel/setup.c +5 -0

@@ -461,7 +464,12 @@ static int c_show(struct seq_file *m, void *v)
               * "processor".  Give glibc what it expects.
               */
 #ifdef CONFIG_SMP
+     if (IS_ENABLED(CONFIG_NUMA)) {
+             seq_printf(m, "processor\t: %d", i);
+             seq_printf(m, " [nid: %d]\n", cpu_to_node(i));
+     } else {
              seq_printf(m, "processor\t: %d\n", i);
+     }
 #endif
      }

Do we need to make this conditional? I think we can just always print the node number, even if it's going to be zero for systems without the associativity properties.

arch/arm64/mm/numa.c line 83

Quoted 4 lines

arch/arm64/mm/numa.c new file +3 -0

+
+int cpu_to_node_map[NR_CPUS];
+EXPORT_SYMBOL(cpu_to_node_map);

This seems to be x86 specific, do we need it?

arch/arm64/mm/numa.c line 120

Quoted 8 lines

arch/arm64/mm/numa.c new file +8 -0

+/*
+ *  Set the cpu to node and mem mapping
+ */
+void numa_store_cpu_info(int cpu)
+{
+#ifdef CONFIG_ARM64_DT_NUMA
+     node_cpu_hwid[cpu].node_id  =  dt_get_cpu_node_id(cpu);
+#endif

I would try to avoid the #ifdef here, by providing a stub function of dt_get_cpu_node_id or whichever function we end up calling here when NUMA is disabled.

Quoted 16 lines

arch/arm64/mm/numa.c new file +16 -0

+
+/**
+ * arm64_numa_init - Initialize NUMA
+ *
+ * Try each configured NUMA initialization method until one succeeds.  The
+ * last fallback is dummy single node config encomapssing whole memory and
+ * never fails.
+ */
+void __init arm64_numa_init(void)
+{
+     if (!numa_off) {
+#ifdef CONFIG_ARM64_DT_NUMA
+             if (!numa_init(arm64_dt_numa_init))
+                     return;
+#endif
+     }

arch/arm64/mm/numa.c line 520

Quoted 3 lines

arch/arm64/mm/numa.c new file +3 -0

+
+     numa_init(dummy_numa_init);
+}

I don't think we need the CONFIG_ARM64_DT_NUMA=n case here, it should just not be conditional, and the arm64_dt_numa_init should fall back to doing something reasonable when numa is turned off or there are no associativity properties.

Arnd

Quoted 21 lines · Arnd Bergmann

Quoted 21 lines

arch/arm64/kernel/dt_numa.c new file +21 -0

+int dt_get_cpu_node_id(int cpu)
+{
+     struct device_node *dn = NULL;
+
+     while ((dn = of_find_node_by_type(dn, "cpu"))) {
+             const u32 *cell;
+             u64 hwid;
+
+             /*
+              * A cpu node with missing "reg" property is
+              * considered invalid to build a cpu_logical_map
+              * entry.
+              */
+             cell = of_get_property(dn, "reg", NULL);
+             if (!cell) {
+                     pr_err("%s: missing reg property\n", dn->full_name);
+                     return default_nid;
+             }
+             hwid = of_read_number(cell, of_n_addr_cells(dn));
+
+             if (cpu_logical_map(cpu) == hwid)

Quoted 8 lines

arch/arm64/kernel/dt_numa.c line 201

Quoted 5 lines

arch/arm64/kernel/dt_numa.c new file +5 -0

+             return of_node_to_nid_single(dn);
+     }
+     return NUMA_NO_NODE;
+}
+EXPORT_SYMBOL(dt_get_cpu_node_id);

Maybe just expose a function to the device node for a CPU ID here, and expect callers to use of_node_to_nid?

shall i make this wrapper function in dt_numa.c, which will use functions _of_node_to_nid and _of_cpu_to_node(cpu)

And, this function can be a weak function in numa.c which returns 0.

Quoted 30 lines · On Tuesday 06 January 2015 14:55:53 Ganapatrao Kulkarni wrote:

Quoted 21 lines · Arnd Bergmann

Quoted 21 lines

arch/arm64/kernel/dt_numa.c new file +21 -0

+int dt_get_cpu_node_id(int cpu)
+{
+     struct device_node *dn = NULL;
+
+     while ((dn = of_find_node_by_type(dn, "cpu"))) {
+             const u32 *cell;
+             u64 hwid;
+
+             /*
+              * A cpu node with missing "reg" property is
+              * considered invalid to build a cpu_logical_map
+              * entry.
+              */
+             cell = of_get_property(dn, "reg", NULL);
+             if (!cell) {
+                     pr_err("%s: missing reg property\n", dn->full_name);
+                     return default_nid;
+             }
+             hwid = of_read_number(cell, of_n_addr_cells(dn));
+
+             if (cpu_logical_map(cpu) == hwid)

Quoted 8 lines

arch/arm64/kernel/dt_numa.c line 201

Quoted 5 lines

arch/arm64/kernel/dt_numa.c new file +5 -0

+             return of_node_to_nid_single(dn);
+     }
+     return NUMA_NO_NODE;
+}
+EXPORT_SYMBOL(dt_get_cpu_node_id);

Maybe just expose a function to the device node for a CPU ID here, and expect callers to use of_node_to_nid?

Quoted 2 lines

shall i make this wrapper function in dt_numa.c, which will use functions _of_node_to_nid and _of_cpu_to_node(cpu)

Yes, I guess that would work.

Quoted 1 line

And, this function can be a weak function in numa.c which returns 0.

No, please don't use weak functions. You can either use IS_ENABLED() tricks to remove function calls at compile-time, or in the header file provide an inline function as an alternative to the extern declaration, based on a configuration symbol.

Arnd