arm64:numa: Add numa support for arm64 platforms.

Raising the maximum limit to 128. This is needed for Cavium's Thunder system that will have 96 cores on Multi-node system.

Signed-off-by Ganapatrao Kulkarni <[email protected]>

Quoted 3 lines · On Saturday 22 November 2014 02:53:27 Ganapatrao Kulkarni wrote:

Raising the maximum limit to 128. This is needed for Cavium's Thunder system that will have 96 cores on Multi-node system.

Quoted 2 lines

Signed-off-by Ganapatrao Kulkarni <[email protected]>

Could we please raise the compile-time limit to the highest number that you are able to boot successfully on some existing machine?

There isn't much point in doubling this every few months.

Arnd

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel at lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Hi Arnd,

Quoted 11 lines · Arnd Bergmann

Quoted 5 lines · On Saturday 22 November 2014 02:53:27 Ganapatrao Kulkarni wrote:

Raising the maximum limit to 128. This is needed for Cavium's Thunder system that will have 96 cores on Multi-node system.

Signed-off-by Ganapatrao Kulkarni <[email protected]>

Could we please raise the compile-time limit to the highest number that you are able to boot successfully on some existing machine?

Quoted 1 line

There isn't much point in doubling this every few months.

Agreed. If we look back at [1], Mark Rutland has actually compiled and boot-tested NR_CPUS=4096 on Juno.

[1] https://lkml.org/lkml/2014/9/8/537

Quoted 14 lines · On Monday 08 December 2014 17:57:03 Zi Shen Lim wrote:

Hi Arnd,

Quoted 11 lines · Arnd Bergmann

Quoted 5 lines · On Saturday 22 November 2014 02:53:27 Ganapatrao Kulkarni wrote:

Raising the maximum limit to 128. This is needed for Cavium's Thunder system that will have 96 cores on Multi-node system.

Signed-off-by Ganapatrao Kulkarni <[email protected]>

Could we please raise the compile-time limit to the highest number that you are able to boot successfully on some existing machine?

Quoted 6 lines

Quoted 1 line

There isn't much point in doubling this every few months.

Agreed. If we look back at [1], Mark Rutland has actually compiled and boot-tested NR_CPUS=4096 on Juno.

[1] https://lkml.org/lkml/2014/9/8/537

Ok, 4096 sounds like a good NR_CPUS limit then, it should last for a while.

For the defconfig, we probably want a much smaller value, either one that covers all known machines (96 at this time), or something that covers 95% of all users (maybe 32?) and does not have an serious impact on memory consumption or performance on small machines.

Arnd

DT bindings for numa map for memory, cores to node and proximity distance matrix of nodes to each other.

Signed-off-by Ganapatrao Kulkarni <[email protected]>

---
 Documentation/devicetree/bindings/arm/numa.txt | 103 +++++++++++++++++++++++++
 1 file changed, 103 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/arm/numa.txt

Documentation/devicetree/bindings/arm/numa.txt new file +53 -0

new file mode 100644
index 0000000..ec6bf2d
@@ -0,0 +1,103 @@
+==============================================================================
+NUMA binding description.
+==============================================================================
+
+==============================================================================
+1 - Introduction
+==============================================================================
+
+Systems employing a Non Uniform Memory Access (NUMA) architecture contain
+collections of hardware resources including processors, memory, and I/O buses,
+that comprise what is commonly known as a ??UMA node?? Processor
+accesses to memory within the local NUMA node is
+generally faster than processor accesses to memory outside of the local
+NUMA node. DT defines interfaces that allow the platform to convey NUMA node
+topology information to OS.
+
+==============================================================================
+2 - numa-map node
+==============================================================================
+
+DT Binding for NUMA can be defined for memory and CPUs to map them to
+respective NUMA nodes.
+
+The DT binding can defined using numa-map node.
+The numa-map will have following properties to define NUMA topology.
+
+- mem-map:	This property defines the association between a range of
+		memory and the proximity domain/numa node to which it belongs.
+
+note: memory range address is passed using either memory node of
+DT or UEFI system table and should match to the address defined in mem-map.
+
+- cpu-map:	This property defines the association of range of processors
+		(range of cpu ids) and the proximity domain to which
+		the processor belongs.
+
+- node-matrix:	This table provides a matrix that describes the relative
+		distance (memory latency) between all System Localities.
+		The value of each Entry[i j distance] in node-matrix table,
+		where i represents a row of a matrix and j represents a
+		column of a matrix, indicates the relative distances
+		from Proximity Domain/Numa node i to every other
+		node j in the system (including itself).
+
+The numa-map node must contain the appropriate #address-cells,
+#size-cells and #node-count properties.
+
+
+==============================================================================
+4 - Example dts
+==============================================================================
+
+Example 1: 2 Node system each having 8 CPUs and a Memory.

Documentation/devicetree/bindings/arm/numa.txt new file +40 -0

+
+		node-matrix = <0 0 10>,
+			      <0 1 20>,
+			      <1 0 20>,
+			      <1 1 10>;
+	};
+
+Example 2: 4 Node system each having 4 CPUs and a Memory.
+
+	numa-map {
+		#address-cells = <2>;
+		#size-cells = <1>;
+		#node-count = <2>;
+		mem-map =  <0x0 0x00000000 0>,
+		           <0x100 0x00000000 1>,
+		           <0x200 0x00000000 2>,
+		           <0x300 0x00000000 3>;
+
+		cpu-map = <0 7 0>,
+			  <8 15 1>,
+			  <16 23 2>,
+			  <24 31 3>;
+
+		node-matrix = <0 0 10>,
+			      <0 1 20>,
+			      <0 2 20>,
+			      <0 3 20>,
+			      <1 0 20>,
+			      <1 1 10>,
+			      <1 2 20>,
+			      <1 3 20>,
+			      <2 0 20>,
+			      <2 1 20>,
+			      <2 2 10>,
+			      <2 3 20>,
+			      <3 0 20>,
+			      <3 1 20>,
+			      <3 2 20>,
+			      <3 3 10>;
+	};

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel at lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Hi,

On 2014/11/22 5:23, Ganapatrao Kulkarni wrote:

Quoted 5 lines

+==============================================================================
+4 - Example dts
+==============================================================================

Documentation/devicetree/bindings/arm/numa.txt new file +2 -0

+
+Example 1: 2 Node system each having 8 CPUs and a Memory.

Documentation/devicetree/bindings/arm/numa.txt line 63

Quoted 10 lines

Documentation/devicetree/bindings/arm/numa.txt new file +10 -0

+
+	numa-map {
+		#address-cells = <2>;
+		#size-cells = <1>;
+		#node-count = <2>;
+		mem-map =  <0x0 0x00000000 0>,
+		           <0x100 0x00000000 1>;
+
+		cpu-map = <0 7 0>,
+			  <8 15 1>;

The cpu range is continuous here. But if there is a situation like below:

0 2 4 6 belong to node 0
1 3 5 7 belong to node 1

This case is very common on X86. I don't know the real situation of arm as I don't have a hardware with 2 nodes.

How can we generate a DTS about this situation? like below? Can be parsed?

		cpu-map = <0 2 4 6 0>,
			  <1 3 5 7 1>;

Quoted 3 lines · Shannon Zhao

Hi,

On 2014/11/22 5:23, Ganapatrao Kulkarni wrote:

[...]

Quoted 21 lines

Quoted 5 lines

+==============================================================================
+4 - Example dts
+==============================================================================

Documentation/devicetree/bindings/arm/numa.txt new file +2 -0

+
+Example 1: 2 Node system each having 8 CPUs and a Memory.

Documentation/devicetree/bindings/arm/numa.txt line 63

Quoted 10 lines

Documentation/devicetree/bindings/arm/numa.txt new file +10 -0

+
+    numa-map {
+            #address-cells = <2>;
+            #size-cells = <1>;
+            #node-count = <2>;
+            mem-map =  <0x0 0x00000000 0>,
+                       <0x100 0x00000000 1>;
+
+            cpu-map = <0 7 0>,
+                      <8 15 1>;

The cpu range is continuous here. But if there is a situation like below:

0 2 4 6 belong to node 0
1 3 5 7 belong to node 1

Quoted 7 lines

This case is very common on X86. I don't know the real situation of arm as I don't have a hardware with 2 nodes.

How can we generate a DTS about this situation? like below? Can be parsed?

              cpu-map = <0 2 4 6 0>,
                        <1 3 5 7 1>;

I think the binding proposed here can not cover your needs, and I think this binding is not suitable, there are some reasons.

- CPU logical ID is allocated by OS, and it depends on the order of CPU node in the device tree, so it may be in a clean order like this patch proposed, or it will like the order Shannon pointed out.

- Since CPU logical ID is allocated by OS, DTS file will not know these numbers.

So the problem behind this is the mappings between CPUs and NUMA nodes, there is already mapping for CPU hardware ID (MPIDR) and CPU logical ID, and MPIDR will be not changed, why not using MPIDR for the mapping of NUMA node and CPU? then the mappings will be:

CPU logical ID <------> CPU MPIDR <-----> NUMA node ID <-----> proximity domain
(allocated by OS) (constant) (allocated by OS)

Quoted 37 lines · On Tuesday 25 November 2014 17:42:44 Hanjun Guo wrote:

Quoted 3 lines · Shannon Zhao

Hi,

On 2014/11/22 5:23, Ganapatrao Kulkarni wrote:

[...]

Quoted 21 lines

Quoted 5 lines

+==============================================================================
+4 - Example dts
+==============================================================================

Documentation/devicetree/bindings/arm/numa.txt new file +2 -0

+
+Example 1: 2 Node system each having 8 CPUs and a Memory.

Documentation/devicetree/bindings/arm/numa.txt line 63

Quoted 10 lines

Documentation/devicetree/bindings/arm/numa.txt new file +10 -0

+
+    numa-map {
+            #address-cells = <2>;
+            #size-cells = <1>;
+            #node-count = <2>;
+            mem-map =  <0x0 0x00000000 0>,
+                       <0x100 0x00000000 1>;
+
+            cpu-map = <0 7 0>,
+                      <8 15 1>;

The cpu range is continuous here. But if there is a situation like below:

0 2 4 6 belong to node 0
1 3 5 7 belong to node 1

Quoted 7 lines

This case is very common on X86. I don't know the real situation of arm as I don't have a hardware with 2 nodes.

How can we generate a DTS about this situation? like below? Can be parsed?

              cpu-map = <0 2 4 6 0>,
                        <1 3 5 7 1>;

I think the binding proposed here can not cover your needs, and I think this binding is not suitable, there are some reasons.

Quoted 6 lines

- CPU logical ID is allocated by OS, and it depends on the order of CPU node in the device tree, so it may be in a clean order like this patch proposed, or it will like the order Shannon pointed out.

- Since CPU logical ID is allocated by OS, DTS file will not know these numbers.

Also:

- you cannot support hierarchical NUMA topology

- you cannot have CPU-less or memory-less nodes

- you cannot associate I/O devices with NUMA nodes, only memory and CPU

Quoted 5 lines

So the problem behind this is the mappings between CPUs and NUMA nodes, there is already mapping for CPU hardware ID (MPIDR) and CPU logical ID, and MPIDR will be not changed, why not using MPIDR for the mapping of NUMA node and CPU? then the mappings will be:

Quoted 2 lines

CPU logical ID <------> CPU MPIDR <-----> NUMA node ID <-----> proximity domain
(allocated by OS) (constant) (allocated by OS)

No, don't hardcode ARM specifics into a common binding either. I've looked at the ibm,associativity properties again, and I think we should just use those, they can cover all cases and are completely independent of the architecture. We should probably discuss about the property name though, as using the "ibm," prefix might not be the best idea.

Arnd

Quoted 5 lines

No, don't hardcode ARM specifics into a common binding either. I've looked at the ibm,associativity properties again, and I think we should just use those, they can cover all cases and are completely independent of the architecture. We should probably discuss about the property name though, as using the "ibm," prefix might not be the best idea.

We have started with new proposal, since not got enough details how ibm/ppc is managing the numa using dt.
there is no documentation and there is no power/PAPR spec for numa in public domain and there are no single dt file in arch/powerpc which describes the numa. if we get any one of these details, we can align to powerpc implementation.

Quoted 6 lines · On Tuesday 25 November 2014 08:15:47 Ganapatrao Kulkarni wrote:

Quoted 5 lines

No, don't hardcode ARM specifics into a common binding either. I've looked at the ibm,associativity properties again, and I think we should just use those, they can cover all cases and are completely independent of the architecture. We should probably discuss about the property name though, as using the "ibm," prefix might not be the best idea.

Quoted 6 lines

We have started with new proposal, since not got enough details how ibm/ppc is managing the numa using dt.
there is no documentation and there is no power/PAPR spec for numa in public domain and there are no single dt file in arch/powerpc which describes the numa. if we get any one of these details, we can align to powerpc implementation.

Basically the idea is to have an "ibm,associativity" property in each bus or device that is node specific, and this includes all CPUs and memory nodes. ...

Quoted 6 lines · On Tuesday 25 November 2014 08:15:47 Ganapatrao Kulkarni wrote:

Quoted 5 lines

No, don't hardcode ARM specifics into a common binding either. I've looked at the ibm,associativity properties again, and I think we should just use those, they can cover all cases and are completely independent of the architecture. We should probably discuss about the property name though, as using the "ibm," prefix might not be the best idea.

Quoted 6 lines

We have started with new proposal, since not got enough details how ibm/ppc is managing the numa using dt.
there is no documentation and there is no power/PAPR spec for numa in public domain and there are no single dt file in arch/powerpc which describes the numa. if we get any one of these details, we can align to powerpc implementation.

Basically the idea is to have an "ibm,associativity" property in each bus or device that is node specific, and this includes all CPUs and memory nodes. The property contains an array of 32-bit integers that count the resources. Take an example of a NUMA cluster of two machines with four sockets and four cores each (32 cores total), a memory channel on each socket and one PCI host per board that is connected at equal speed to each socket on the board.

The ibm,associativity property in each PCI host, CPU or memory device node consequently has an array of three (board, socket, core) integers:

	memory at 0,0 {
		device_type = "memory";
		reg = <0x0 0x0  0x4 0x0;
		/* board 0, socket 0, no specific core */
		ibm,asssociativity = <0 0 0xffff>;
	};

	memory at 4,0 {
		device_type = "memory";
		reg = <0x4 0x0  0x4 0x0>;
		/* board 0, socket 1, no specific core */
		ibm,asssociativity = <0 1 0xffff>; 
	};

	...

	memory at 1c,0 {
		device_type = "memory";
		reg = <0x1c 0x0  0x4 0x0>;
		/* board 0, socket 7, no specific core */
		ibm,asssociativity = <1 7 0xffff>; 
	};

	cpus {
		#address-cells = <2>;
		#size-cells = <0>;
		cpu at 0 {
			device_type = "cpu";
			reg = <0 0>;
			/* board 0, socket 0, core 0*/
			ibm,asssociativity = <0 0 0>; 
		};

		cpu at 1 {
			device_type = "cpu";
			reg = <0 0>;
			/* board 0, socket 0, core 0*/
			ibm,asssociativity = <0 0 0>; 
		};

		...

		cpu at 31 {
			device_type = "cpu";
			reg = <0 32>;
			/* board 1, socket 7, core 31*/
			ibm,asssociativity = <1 7 31>; 
		};
	};

	pci at 100,0 {
		device_type = "pci";
		/* board 0 */
		ibm,associativity = <0 0xffff 0xffff>;
		...
	};

	pci at 200,0 {
		device_type = "pci";
		/* board 1 */
		ibm,associativity = <1 0xffff 0xffff>;
		...
	};

	ibm,associativity-reference-points = <0 1>;

The "ibm,associativity-reference-points" property here indicates that index 2 of each array is the most important NUMA boundary for the particular system, because the performance impact of allocating memory on the remote board is more significant than the impact of using memory on a remote socket of the same board. Linux will consequently use the first field in the array as the NUMA node ID. If the link between the boards however is relatively fast, so you care mostly about allocating memory on the same socket, but going to another board isn't much worse than going to another socket on the same board, this would be

	ibm,associativity-reference-points = <1 0>;

so Linux would ignore the board ID and use the socket ID as the NUMA node number. The same would apply if you have only one (otherwise identical board, then you would get

	ibm,associativity-reference-points = <1>;

which means that index 0 is completely irrelevant for NUMA considerations and you just care about the socket ID. In this case, devices on the PCI bus would also not care about NUMA policy and just allocate buffers from anywhere, while in original example Linux would allocate DMA buffers only from the local board.

Thanks for the detail information. I have the concerns about the distance for NUMA nodes, does the "ibm,associativity-reference-points" property can represent the distance between NUMA nodes?

For example, a system with 4 sockets connected like below:

Socket 0 <----> Socket 1 <----> Socket 2 <----> Socket 3

So from socket 0 to socket 1 (maybe on the same board), it just need 1 jump to access the memory, but from socket 0 to socket 2/3, it needs 2/3 jumps and the *distance* relative longer. Can "ibm,associativity-reference-points" property cover this?

Quoted 5 lines · On Wednesday 26 November 2014 17:12:49 Hanjun Guo wrote:

Thanks for the detail information. I have the concerns about the distance for NUMA nodes, does the "ibm,associativity-reference-points" property can represent the distance between NUMA nodes?

Quoted 9 lines

For example, a system with 4 sockets connected like below:

Socket 0 <----> Socket 1 <----> Socket 2 <----> Socket 3

So from socket 0 to socket 1 (maybe on the same board), it just need 1 jump to access the memory, but from socket 0 to socket 2/3, it needs 2/3 jumps and the *distance* relative longer. Can "ibm,associativity-reference-points" property cover this?

Hi Hanjun,

I only today found your replies in my spam folder, I need to put you on a whitelist so that doesn't happen again.

The above topology is not easy to represent, but I think it would work like this (ignoring the threads/cores/clusters on the socket, which would also need to be described in a full DT), using multiple logical paths between the nodes:

socket 0
ibm,associativity = <0 0 0 0>, <1 1 1 0>, <2 2 0 0>, <3 0 0 0>;

socket 1
ibm,associativity = <1 1 1 1>, <0 0 0 1>, <2 2 2 1>, <3 3 1 1>;

socket 2
ibm,associativity = <2 2 2 2>, <0 0 2 2>, <1 1 1 2>, <3 3 3 2>;

socket 3
ibm,associativity = <3 3 3 3>, <0 3 3 3>, <1 1 3 3>, <2 2 2 3>;

This describes four levels or hierarchy, with the lowest level being a single CPU core on one socket, and four paths between the sockets. To compute the associativity between two sockets, you need to look at each combination of paths to find the best match.

Comparing sockets 0 and 1, the best matches are <1 1 1 0> with <1 1 1 1>, and <0 0 0 0> with <0 0 0 1>. In each case, the associativity is "3", meaning the first three entries match.

Comparing sockets 0 and 3, we have four equally bad matches that each only match in the highest-level domain, e.g. <0 0 0 0> with <0 3 3 3>, so the associativity is only "1", and that means the two nodes are less closely associated than two neighboring ones.

With the algorithm that powerpc uses to turn associativity into distance, 2**(numlevels - associativity), this would put the distance of neighboring nodes at "2", and the longest distance at "8".

Quoted 34 lines · Arnd Bergmann

On Wednesday 26 November 2014 17:12:49 Hanjun Guo wrote:
The above topology is not easy to represent, but I think it would work like this (ignoring the threads/cores/clusters on the socket, which would also need to be described in a full DT), using multiple logical paths between the nodes:

socket 0
ibm,associativity = <0 0 0 0>, <1 1 1 0>, <2 2 0 0>, 0 0 0>;

socket 1
ibm,associativity = <1 1 1 1>, <0 0 0 1>, <2 2 2 1>, 3 1 1>;

socket 2
ibm,associativity = <2 2 2 2>, <0 0 2 2>, <1 1 1 2>, 3 3 2>;

socket 3
ibm,associativity = 3 3 3>, <0 3 3 3>, <1 1 3 3>, <2 2 2 3>;

This describes four levels or hierarchy, with the lowest level being a single CPU core on one socket, and four paths between the sockets. To compute the associativity between two sockets, you need to look at each combination of paths to find the best match.

Comparing sockets 0 and 1, the best matches are <1 1 1 0> with <1 1 1 1>, and <0 0 0 0> with <0 0 0 1>. In each case, the associativity is "3", meaning the first three entries match.

Comparing sockets 0 and 3, we have four equally bad matches that each only match in the highest-level domain, e.g. <0 0 0 0> with <0 3 3 3>, so the associativity is only "1", and that means the two nodes are less closely associated than two neighboring ones.

Quoted 4 lines

With the algorithm that powerpc uses to turn associativity into distance, 2**(numlevels - associativity), this would put the distance of neighboring nodes at "2", and the longest distance at "8".

Thanks for the explain, I can understand how it works now, a bit complicated for me and I think the distance property "node-matrix" in Ganapatrao's patch is straight forward, what do you think?

Quoted 40 lines · On Thursday 11 December 2014 17:16:35 Hanjun Guo wrote:

Quoted 38 lines · Arnd Bergmann

On Wednesday 26 November 2014 17:12:49 Hanjun Guo wrote:
The above topology is not easy to represent, but I think it would work like this (ignoring the threads/cores/clusters on the socket, which would also need to be described in a full DT), using multiple logical paths between the nodes:

socket 0
ibm,associativity = <0 0 0 0>, <1 1 1 0>, <2 2 0 0>, 0 0 0>;

socket 1
ibm,associativity = <1 1 1 1>, <0 0 0 1>, <2 2 2 1>, 3 1 1>;

socket 2
ibm,associativity = <2 2 2 2>, <0 0 2 2>, <1 1 1 2>, 3 3 2>;

socket 3
ibm,associativity = 3 3 3>, <0 3 3 3>, <1 1 3 3>, <2 2 2 3>;

This describes four levels or hierarchy, with the lowest level being a single CPU core on one socket, and four paths between the sockets. To compute the associativity between two sockets, you need to look at each combination of paths to find the best match.

Comparing sockets 0 and 1, the best matches are <1 1 1 0> with <1 1 1 1>, and <0 0 0 0> with <0 0 0 1>. In each case, the associativity is "3", meaning the first three entries match.

Comparing sockets 0 and 3, we have four equally bad matches that each only match in the highest-level domain, e.g. <0 0 0 0> with <0 3 3 3>, so the associativity is only "1", and that means the two nodes are less closely associated than two neighboring ones.

With the algorithm that powerpc uses to turn associativity into distance, 2**(numlevels - associativity), this would put the distance of neighboring nodes at "2", and the longest distance at "8".

Quoted 4 lines

Thanks for the explain, I can understand how it works now, a bit complicated for me and I think the distance property "node-matrix" in Ganapatrao's patch is straight forward, what do you think?

I still think we should go the whole way of having something compatible with the existing bindings, possibly using different property names if there are objections to using the "ibm," prefix.

Quoted 6 lines · On Tuesday 25 November 2014 08:15:47 Ganapatrao Kulkarni wrote:

Quoted 5 lines

No, don't hardcode ARM specifics into a common binding either. I've looked at the ibm,associativity properties again, and I think we should just use those, they can cover all cases and are completely independent of the architecture. We should probably discuss about the property name though, as using the "ibm," prefix might not be the best idea.

Quoted 6 lines

We have started with new proposal, since not got enough details how ibm/ppc is managing the numa using dt.
there is no documentation and there is no power/PAPR spec for numa in public domain and there are no single dt file in arch/powerpc which describes the numa. if we get any one of these details, we can align to powerpc implementation.

Basically the idea is to have an "ibm,associativity" property in each bus or device that is node specific, and this includes all CPUs and memory nodes. The property contains an array of 32-bit integers that count the resources. Take an example of a NUMA cluster of two machines with four sockets and four cores each (32 cores total), a memory channel on each socket and one PCI host per board that is connected at equal speed to each socket on the board.

The ibm,associativity property in each PCI host, CPU or memory device node consequently has an array of three (board, socket, core) integers:

        memory at 0,0 {
                device_type = "memory";
                reg = <0x0 0x0  0x4 0x0;
                /* board 0, socket 0, no specific core */
                ibm,asssociativity = <0 0 0xffff>;
        };

        memory at 4,0 {
                device_type = "memory";
                reg = <0x4 0x0  0x4 0x0>;
                /* board 0, socket 1, no specific core */
                ibm,asssociativity = <0 1 0xffff>;
        };

        ...

        memory at 1c,0 {
                device_type = "memory";
                reg = <0x1c 0x0  0x4 0x0>;
                /* board 0, socket 7, no specific core */
                ibm,asssociativity = <1 7 0xffff>;
        };

        cpus {
                #address-cells = <2>;
                #size-cells = <0>;
                cpu at 0 {
                        device_type = "cpu";
                        reg = <0 0>;
                        /* board 0, socket 0, core 0*/
                        ibm,asssociativity = <0 0 0>;
                };

                cpu at 1 {
                        device_type = "cpu";
                        reg = <0 0>;
                        /* board 0, socket 0, core 0*/
                        ibm,asssociativity = <0 0 0>;
                };

                ...

                cpu at 31 {
                        device_type = "cpu";
                        reg = <0 32>;
                        /* board 1, socket 7, core 31*/
                        ibm,asssociativity = <1 7 31>;
                };
        };

        pci at 100,0 {
                device_type = "pci";
                /* board 0 */
                ibm,associativity = <0 0xffff 0xffff>;
                ...
        };

        pci at 200,0 {
                device_type = "pci";
                /* board 1 */
                ibm,associativity = <1 0xffff 0xffff>;
                ...
        };

        ibm,associativity-reference-points = <0 1>;

The "ibm,associativity-reference-points" property here indicates that index 2 of each array is the most important NUMA boundary for the particular system, because the performance impact of allocating memory on the remote board is more significant than the impact of using memory on a remote socket of the same board. Linux will consequently use the first field in the array as the NUMA node ID. If the link between the boards however is relatively fast, so you care mostly about allocating memory on the same socket, but going to another board isn't much worse than going to another socket on the same board, this would be

        ibm,associativity-reference-points = <1 0>;

so Linux would ignore the board ID and use the socket ID as the NUMA node number. The same would apply if you have only one (otherwise identical board, then you would get

        ibm,associativity-reference-points = <1>;

which means that index 0 is completely irrelevant for NUMA considerations and you just care about the socket ID. In this case, devices on the PCI bus would also not care about NUMA policy and just allocate buffers from anywhere, while in original example Linux would allocate DMA buffers only from the local board.

Arnd

Quoted 14 lines · Arnd Bergmann

Quoted 6 lines · On Tuesday 25 November 2014 08:15:47 Ganapatrao Kulkarni wrote:

Quoted 5 lines

No, don't hardcode ARM specifics into a common binding either. I've looked at the ibm,associativity properties again, and I think we should just use those, they can cover all cases and are completely independent of the architecture. We should probably discuss about the property name though, as using the "ibm," prefix might not be the best idea.

Quoted 6 lines

We have started with new proposal, since not got enough details how ibm/ppc is managing the numa using dt.
there is no documentation and there is no power/PAPR spec for numa in public domain and there are no single dt file in arch/powerpc which describes the numa. if we get any one of these details, we can align to powerpc implementation.

Quoted 7 lines

Basically the idea is to have an "ibm,associativity" property in each bus or device that is node specific, and this includes all CPUs and memory nodes. The property contains an array of 32-bit integers that count the resources. Take an example of a NUMA cluster of two machines with four sockets and four cores each (32 cores total), a memory channel on each socket and one PCI host per board that is connected at equal speed to each socket on the board.

thanks for the detailed information.
IMHO, linux-numa code does not care about how the hardware design is, like how many boards and how many sockets it has. It only needs to know how many numa nodes system has, how resources are mapped to nodes and node-distance to define inter node memory access latency. i think it will be simple, if we merge board and socket to single entry say node.

also we are assuming here that numa h/w design will have multiple boards and sockets, what if it has something different/more.

Quoted 11 lines

The "ibm,associativity-reference-points" property here indicates that index 2 of each array is the most important NUMA boundary for the particular system, because the performance impact of allocating memory on the remote board is more significant than the impact of using memory on a remote socket of the same board. Linux will consequently use the first field in the array as the NUMA node ID. If the link between the boards however is relatively fast, so you care mostly about allocating memory on the same socket, but going to another board isn't much worse than going to another socket on the same board, this would be

        ibm,associativity-reference-points = <1 0>;

i am not able to understand fully, it will be grate help, if you explain, how we capture the node distance matrix using "ibm,associativity-reference-points "
for example, how DT looks like for the system with 4 nodes, with below inter-node distance matrix.
node 0 1 distance 20
node 0 2 distance 20
node 0 3 distance 20
node 1 2 distance 20
node 1 3 distance 20
node 2 3 distance 20

Quoted 37 lines · On Tuesday 25 November 2014 17:42:44 Hanjun Guo wrote:

Quoted 3 lines · Shannon Zhao

Hi,

On 2014/11/22 5:23, Ganapatrao Kulkarni wrote:

[...]

Quoted 21 lines

Quoted 5 lines

+==============================================================================
+4 - Example dts
+==============================================================================

Documentation/devicetree/bindings/arm/numa.txt new file +2 -0

+
+Example 1: 2 Node system each having 8 CPUs and a Memory.

Documentation/devicetree/bindings/arm/numa.txt line 63

Quoted 10 lines

Documentation/devicetree/bindings/arm/numa.txt new file +10 -0

+
+    numa-map {
+            #address-cells = <2>;
+            #size-cells = <1>;
+            #node-count = <2>;
+            mem-map =  <0x0 0x00000000 0>,
+                       <0x100 0x00000000 1>;
+
+            cpu-map = <0 7 0>,
+                      <8 15 1>;

The cpu range is continuous here. But if there is a situation like below:

0 2 4 6 belong to node 0
1 3 5 7 belong to node 1

Quoted 7 lines

This case is very common on X86. I don't know the real situation of arm as I don't have a hardware with 2 nodes.

How can we generate a DTS about this situation? like below? Can be parsed?

               cpu-map = <0 2 4 6 0>,
                         <1 3 5 7 1>;

I think the binding proposed here can not cover your needs, and I think this binding is not suitable, there are some reasons.

Quoted 6 lines

- CPU logical ID is allocated by OS, and it depends on the order of CPU node in the device tree, so it may be in a clean order like this patch proposed, or it will like the order Shannon pointed out.

- Since CPU logical ID is allocated by OS, DTS file will not know these numbers.

Also:

- you cannot support hierarchical NUMA topology

- you cannot have CPU-less or memory-less nodes

- you cannot associate I/O devices with NUMA nodes, only memory and CPU

Quoted 6 lines

So the problem behind this is the mappings between CPUs and NUMA nodes, there is already mapping for CPU hardware ID (MPIDR) and CPU logical ID, and MPIDR will be not changed, why not using MPIDR for the mapping of NUMA node and CPU? then the mappings will be:

Quoted 2 lines

CPU logical ID <------> CPU MPIDR <-----> NUMA node ID <-----> proximity domain
(allocated by OS) (constant) (allocated by OS)

No, don't hardcode ARM specifics into a common binding either. I've looked at the ibm,associativity properties again, and I think we should just use those, they can cover all cases and are completely independent of the architecture. We should probably discuss about the property name though, as using the "ibm," prefix might not be the best idea.

Quoted 37 lines · On Tuesday 25 November 2014 17:42:44 Hanjun Guo wrote:

Quoted 3 lines · Shannon Zhao

Hi,

On 2014/11/22 5:23, Ganapatrao Kulkarni wrote:

[...]

Quoted 21 lines

Quoted 5 lines

+==============================================================================
+4 - Example dts
+==============================================================================

Documentation/devicetree/bindings/arm/numa.txt new file +2 -0

+
+Example 1: 2 Node system each having 8 CPUs and a Memory.

Documentation/devicetree/bindings/arm/numa.txt line 63

Quoted 10 lines

Documentation/devicetree/bindings/arm/numa.txt new file +10 -0

+
+    numa-map {
+            #address-cells = <2>;
+            #size-cells = <1>;
+            #node-count = <2>;
+            mem-map =  <0x0 0x00000000 0>,
+                       <0x100 0x00000000 1>;
+
+            cpu-map = <0 7 0>,
+                      <8 15 1>;

The cpu range is continuous here. But if there is a situation like below:

0 2 4 6 belong to node 0
1 3 5 7 belong to node 1

Quoted 7 lines

This case is very common on X86. I don't know the real situation of arm as I don't have a hardware with 2 nodes.

How can we generate a DTS about this situation? like below? Can be parsed?

              cpu-map = <0 2 4 6 0>,
                        <1 3 5 7 1>;

I think the binding proposed here can not cover your needs, and I think this binding is not suitable, there are some reasons.

Quoted 6 lines

- CPU logical ID is allocated by OS, and it depends on the order of CPU node in the device tree, so it may be in a clean order like this patch proposed, or it will like the order Shannon pointed out.

- Since CPU logical ID is allocated by OS, DTS file will not know these numbers.

Also:

- you cannot support hierarchical NUMA topology

- you cannot have CPU-less or memory-less nodes

- you cannot associate I/O devices with NUMA nodes, only memory and CPU

Quoted 5 lines

So the problem behind this is the mappings between CPUs and NUMA nodes, there is already mapping for CPU hardware ID (MPIDR) and CPU logical ID, and MPIDR will be not changed, why not using MPIDR for the mapping of NUMA node and CPU? then the mappings will be:

Quoted 2 lines

CPU logical ID <------> CPU MPIDR <-----> NUMA node ID <-----> proximity domain
(allocated by OS) (constant) (allocated by OS)

No, don't hardcode ARM specifics into a common binding either. I've looked at the ibm,associativity properties again, and I think we should just use those, they can cover all cases and are completely independent of the architecture. We should probably discuss about the property name though, as using the "ibm," prefix might not be the best idea.

Quoted 6 lines · Arnd Bergmann

No, don't hardcode ARM specifics into a common binding either. I've looked at the ibm,associativity properties again, and I think we should just use those, they can cover all cases and are completely independent of the architecture. We should probably discuss about the property name though, as using the "ibm," prefix might not be the best idea.

Yeah, I have read the relevant codes in qemu. I think the "ibm,associativity" is more scalable:-)

About the prefix, my opinion is that as this is relevant with NUMA, maybe we can use "numa" as the prefix.

arch/arm64/boot/dts/thunder-88xx-2n.dts new file +19 -0

+	aliases {
+		serial0 = &uaa0;
+		serial1 = &uaa1;
+	};
+
+	memory at 00c00000 {
+		device_type = "memory";
+		reg = <0x0 0x00000000 0x0 0x80000000>;
+	};
+
+	memory at 10000000000 {
+		device_type = "memory";
+		reg = <0x100 0x00000000 0x0 0x80000000>;
+	};
+
+	numa-map {
+		#address-cells = <2>;
+		#size-cells = <1>;
+		#node-count = <2>;

arch/arm64/boot/dts/thunder-88xx-2n.dtsi new file +11 -0

+
+	timer {
+		compatible = "arm,armv8-timer";
+		interrupts = <1 13 0xff01>,
+		             <1 14 0xff01>,
+		             <1 11 0xff01>,
+		             <1 10 0xff01>;
+	};
+
+	soc {
+		compatible = "simple-bus";

arch/arm64/boot/dts/thunder-88xx-2n.dts line 56

Quoted 3 lines · On Saturday 22 November 2014 02:53:29 Ganapatrao Kulkarni wrote:

arch/arm64/boot/dts/thunder-88xx-2n.dts new file +3 -0

+/ {
+     model = "Cavium ThunderX CN88XX board";
+     compatible = "cavium,thunder-88xx";

No wildcards in compatible strings or model names please. List the exact chip that you are using.

Quoted 19 lines

arch/arm64/boot/dts/thunder-88xx-2n.dts new file +19 -0

+     aliases {
+             serial0 = &uaa0;
+             serial1 = &uaa1;
+     };
+
+     memory at 00c00000 {
+             device_type = "memory";
+             reg = <0x0 0x00000000 0x0 0x80000000>;
+     };
+
+     memory at 10000000000 {
+             device_type = "memory";
+             reg = <0x100 0x00000000 0x0 0x80000000>;
+     };
+
+     numa-map {
+             #address-cells = <2>;
+             #size-cells = <1>;
+             #node-count = <2>;

arch/arm64/boot/dts/thunder-88xx-2n.dts line 86

Quoted 10 lines

arch/arm64/boot/dts/thunder-88xx-2n.dts new file +10 -0

+             mem-map = <0x0 0x00000000 0>,
+                        <0x100 0x00000000 1>;
+
+               cpu-map = <0 47 0>,
+                     <48 95 1>;
+
+             node-matrix=    <0 0 10>,
+                             <0 1 20>,
+                             <1 0 20>,
+                             <1 1 10>;

I don't know how much history is behind this binding. Have you looked at the sPAPR way of doing this? I don't remember exactly how that is done, but we'd need a good reason to discard that and implement something else for arm64.

If we create a new binding, I don't think the 'numa-map' node you have here is the best solution. We already have device nodes for each memory segment and each CPU in the system. Why not work with those nodes directly?

Quoted 11 lines

arch/arm64/boot/dts/thunder-88xx-2n.dtsi new file +11 -0

+
+     timer {
+             compatible = "arm,armv8-timer";
+             interrupts = <1 13 0xff01>,
+                          <1 14 0xff01>,
+                          <1 11 0xff01>,
+                          <1 10 0xff01>;
+     };
+
+     soc {
+             compatible = "simple-bus";

arch/arm64/boot/dts/thunder-88xx-2n.dtsi line 662

Quoted 10 lines

arch/arm64/boot/dts/thunder-88xx-2n.dtsi new file +10 -0

+             #address-cells = <2>;
+             #size-cells = <2>;
+             ranges;
+
+             refclk50mhz: refclk50mhz {
+                     compatible = "fixed-clock";
+                     #clock-cells = <0>;
+                     clock-frequency = <50000000>;
+                     clock-output-names = "refclk50mhz";
+             };

Why is the timer outside of the soc and the refclk is inside? I would have expected the opposite.

Arnd

Quoted 10 lines

I don't know how much history is behind this binding. Have you looked at the sPAPR way of doing this? I don't remember exactly how that is done, but we'd need a good reason to discard that and implement something else for arm64.

If we create a new binding, I don't think the 'numa-map' node you have here is the best solution. We already have device nodes for each memory segment and each CPU in the system. Why not work with those nodes directly?

The DT memory nodes don't exist in an EFI system, as the EFI memory map is used instead.
Using EFI as the boot firmware doesn't require the use of ACPI for hardware description,
so the EFI/DT case is one that we should support.

Quoted 11 lines · On Monday 24 November 2014 11:32:46 Roy Franz wrote:

Quoted 10 lines

I don't know how much history is behind this binding. Have you looked at the sPAPR way of doing this? I don't remember exactly how that is done, but we'd need a good reason to discard that and implement something else for arm64.

If we create a new binding, I don't think the 'numa-map' node you have here is the best solution. We already have device nodes for each memory segment and each CPU in the system. Why not work with those nodes directly?

Quoted 5 lines

The DT memory nodes don't exist in an EFI system, as the EFI memory map is used instead.
Using EFI as the boot firmware doesn't require the use of ACPI for hardware description,
so the EFI/DT case is one that we should support.

But they /could/ exist, right? Can we just require them to be present in order to use NUMA features?

Quoted 12 lines · Arnd Bergmann

Quoted 11 lines · On Monday 24 November 2014 11:32:46 Roy Franz wrote:

Quoted 10 lines

I don't know how much history is behind this binding. Have you looked at the sPAPR way of doing this? I don't remember exactly how that is done, but we'd need a good reason to discard that and implement something else for arm64.

If we create a new binding, I don't think the 'numa-map' node you have here is the best solution. We already have device nodes for each memory segment and each CPU in the system. Why not work with those nodes directly?

Quoted 9 lines

Quoted 5 lines

The DT memory nodes don't exist in an EFI system, as the EFI memory map is used instead.
Using EFI as the boot firmware doesn't require the use of ACPI for hardware description,
so the EFI/DT case is one that we should support.

But they /could/ exist, right? Can we just require them to be present in order to use NUMA features?

Actually, currently the memory nodes are stripped from the device tree by the EFI stub, so the kernel will never get to see them. This is done more or less as a fixup, under the assumption that EFI systems should not have DT memory nodes in the first place.

We could revisit this, of course, but it needs to be taken into account in this discussion.

adding devicetree definition for Cavium's Thunder SoC in 2 Node topology.

Signed-off-by Ganapatrao Kulkarni <[email protected]>

---
 arch/arm64/boot/dts/thunder-88xx-2n.dts  |  88 ++++
 arch/arm64/boot/dts/thunder-88xx-2n.dtsi | 694 +++++++++++++++++++++++++++++++
 2 files changed, 782 insertions(+)
 create mode 100644 arch/arm64/boot/dts/thunder-88xx-2n.dts
 create mode 100644 arch/arm64/boot/dts/thunder-88xx-2n.dtsi

arch/arm64/boot/dts/thunder-88xx-2n.dtsi new file +0 -0

new file mode 100644
index 0000000..3f217b4

arch/arm64/boot/dts/thunder-88xx-2n.dtsi new file +17 -0

+
+       soc {
+               compatible = "simple-bus";
+               #address-cells = <2>;
+               #size-cells = <2>;
+               ranges;
+
+               refclk50mhz: refclk50mhz {
+                       compatible = "fixed-clock";
+                       #clock-cells = <0>;
+                       clock-frequency = <50000000>;
+                       clock-output-names = "refclk50mhz";
+               };
+
+               gic0: interrupt-controller at 8010,00000000 {
+                       compatible = "arm,gic-v3";
+                       #interrupt-cells = <3>;

Adding numa support for arm64 based platforms. creating numa mapping by parsing the dt node numa-map.

Signed-off-by Ganapatrao Kulkarni <[email protected]>

Adding numa support for arm64 based platforms. creating numa mapping by parsing the dt node numa-map.

Signed-off-by Ganapatrao Kulkarni <[email protected]>

Adding numa support for arm64 based platforms. creating numa mapping by parsing the dt node numa-map.

Signed-off-by Ganapatrao Kulkarni <[email protected]>

Adding numa support for arm64 based platforms. creating numa mapping by parsing the dt node numa-map.

Signed-off-by Ganapatrao Kulkarni <[email protected]>

Quoted 4 lines · Ganapatrao Kulkarni

Adding numa support for arm64 based platforms. creating numa mapping by parsing the dt node numa-map.

Signed-off-by Ganapatrao Kulkarni <[email protected]>

Ganapat,

Can we get a simple version of this patchset with just the associativity parameter for the cpu and memory nodes upstream? The initial patchset can be with the assumption of a simple full mesh topology for the distance. The hierarchical complex topologies and the DT nodes and properties for that can be added in a later patch, once the standard for that is agreed upon.
We are planning to support ACPI based NUMA for Broadcom Vulcan processors(http://www.broadcom.com/press/release.php?id=s797235).
I am working on a patch to add support for the SRAT and SLIT ACPI tables based on your patch and I will post that patchset soon, so it will be helpful if the core changes are upstream.

Thanks,
Ashok

Quoted 7 lines · Ashok Kumar

Quoted 4 lines · Ganapatrao Kulkarni

Adding numa support for arm64 based platforms. creating numa mapping by parsing the dt node numa-map.

Signed-off-by Ganapatrao Kulkarni <[email protected]>

Ganapat,

Quoted 8 lines

Can we get a simple version of this patchset with just the associativity parameter for the cpu and memory nodes upstream? The initial patchset can be with the assumption of a simple full mesh topology for the distance. The hierarchical complex topologies and the DT nodes and properties for that can be added in a later patch, once the standard for that is agreed upon.
We are planning to support ACPI based NUMA for Broadcom Vulcan processors(http://www.broadcom.com/press/release.php?id=s797235). I am working on a patch to add support for the SRAT and SLIT ACPI tables based on your patch and I will post that patchset soon, so it will be helpful if the core changes are upstream.

to implement, ibm/ppc like implementation, we need efi-stub patch to not remove the memory node from the dt.
current efi-stub parses dt file and removes memory nodes. can i get the efi-stub patch to start with please?

From 27b4aecf09707e3d5bd4ff7bf765cd609772476f Mon Sep 17 00:00:00 2001

Quoted 7 lines · Ashok Kumar

Quoted 4 lines · Ganapatrao Kulkarni

Adding numa support for arm64 based platforms. creating numa mapping by parsing the dt node numa-map.

Signed-off-by Ganapatrao Kulkarni <[email protected]>

Ganapat,

Quoted 8 lines

Can we get a simple version of this patchset with just the associativity parameter for the cpu and memory nodes upstream? The initial patchset can be with the assumption of a simple full mesh topology for the distance. The hierarchical complex topologies and the DT nodes and properties for that can be added in a later patch, once the standard for that is agreed upon.
We are planning to support ACPI based NUMA for Broadcom Vulcan processors(http://www.broadcom.com/press/release.php?id=s797235). I am working on a patch to add support for the SRAT and SLIT ACPI tables based on your patch and I will post that patchset soon, so it will be helpful if the core changes are upstream.

to implement, ibm/ppc like implementation, we need efi-stub patch to not remove the memory node from the dt.
current efi-stub parses dt file and removes memory nodes. can i get the efi-stub patch to start with please?

From 27b4aecf09707e3d5bd4ff7bf765cd609772476f Mon Sep 17 00:00:00 2001

Quoted 7 lines · Ashok Kumar

Quoted 4 lines · Ganapatrao Kulkarni

Adding numa support for arm64 based platforms. creating numa mapping by parsing the dt node numa-map.

Signed-off-by Ganapatrao Kulkarni <[email protected]>

Ganapat,

Quoted 8 lines

Can we get a simple version of this patchset with just the associativity parameter for the cpu and memory nodes upstream? The initial patchset can be with the assumption of a simple full mesh topology for the distance. The hierarchical complex topologies and the DT nodes and properties for that can be added in a later patch, once the standard for that is agreed upon.
We are planning to support ACPI based NUMA for Broadcom Vulcan processors(http://www.broadcom.com/press/release.php?id=s797235). I am working on a patch to add support for the SRAT and SLIT ACPI tables based on your patch and I will post that patchset soon, so it will be helpful if the core changes are upstream.

to implement, ibm/ppc like implementation, we need efi-stub patch to not remove the memory node from the dt.
current efi-stub parses dt file and removes memory nodes. can i get the efi-stub patch to start with please?

From 27b4aecf09707e3d5bd4ff7bf765cd609772476f Mon Sep 17 00:00:00 2001

Quoted 7 lines · Ashok Kumar

Quoted 4 lines · Ganapatrao Kulkarni

Adding numa support for arm64 based platforms. creating numa mapping by parsing the dt node numa-map.

Signed-off-by Ganapatrao Kulkarni <[email protected]>

Ganapat,

Quoted 8 lines

Can we get a simple version of this patchset with just the associativity parameter for the cpu and memory nodes upstream? The initial patchset can be with the assumption of a simple full mesh topology for the distance. The hierarchical complex topologies and the DT nodes and properties for that can be added in a later patch, once the standard for that is agreed upon.
We are planning to support ACPI based NUMA for Broadcom Vulcan processors(http://www.broadcom.com/press/release.php?id=s797235). I am working on a patch to add support for the SRAT and SLIT ACPI tables based on your patch and I will post that patchset soon, so it will be helpful if the core changes are upstream.

to implement, ibm/ppc like implementation, we need efi-stub patch to not remove the memory node from the dt.
current efi-stub parses dt file and removes memory nodes. can i get the efi-stub patch to start with please?

From 27b4aecf09707e3d5bd4ff7bf765cd609772476f Mon Sep 17 00:00:00 2001

Quoted 16 lines · Ganapatrao Kulkarni

Quoted 7 lines · Ashok Kumar

Quoted 4 lines · Ganapatrao Kulkarni

Adding numa support for arm64 based platforms. creating numa mapping by parsing the dt node numa-map.

Signed-off-by Ganapatrao Kulkarni <[email protected]>

Ganapat,

Quoted 8 lines

Can we get a simple version of this patchset with just the associativity parameter for the cpu and memory nodes upstream? The initial patchset can be with the assumption of a simple full mesh topology for the distance. The hierarchical complex topologies and the DT nodes and properties for that can be added in a later patch, once the standard for that is agreed upon.
We are planning to support ACPI based NUMA for Broadcom Vulcan processors(http://www.broadcom.com/press/release.php?id=s797235). I am working on a patch to add support for the SRAT and SLIT ACPI tables based on your patch and I will post that patchset soon, so it will be helpful if the core changes are upstream.

Quoted 4 lines

to implement, ibm/ppc like implementation, we need efi-stub patch to not remove the memory node from the dt.
current efi-stub parses dt file and removes memory nodes. can i get the efi-stub patch to start with please?

Ganapat,
I tried the below patch in qemu and it works. Would this be of help to you?

Roy/Ard,
Is the below patch fine?

From 27b4aecf09707e3d5bd4ff7bf765cd609772476f Mon Sep 17 00:00:00 2001
From: Ashok Kumar <[email protected]>
Date: Tue, 9 Dec 2014 17:23:11 +0530
Subject: [PATCH] efi/arm64: Remove deleting memory nodes in efi-stub

Dont delete memory nodes from DT as it will be
used by NUMA configuration.

Signed-off-by Ashok Kumar <[email protected]>

---
 arch/arm64/kernel/setup.c          | 12 +++++++++++-
 drivers/firmware/efi/libstub/fdt.c | 24 +-----------------------
 2 files changed, 12 insertions(+), 24 deletions(-)

arch/arm64/kernel/setup.c +11 -1

index 27f65b0..ac6d34f 100644
@@ -302,7 +302,7 @@ static void __init setup_processor(void)
 
 static void __init setup_machine_fdt(phys_addr_t dt_phys)
 {
-       if (!dt_phys || !early_init_dt_scan(phys_to_virt(dt_phys))) {
+       if (!dt_phys || !early_init_dt_verify(phys_to_virt(dt_phys))) {
                early_print("\n"
                        "Error: invalid device tree blob at physical address 0x%p (virtual address 0x%p)\n"
                        "The dtb must be 8-byte aligned and passed in the first 512MB of memory\n"

@@ -313,6 +313,16 @@ static void __init setup_machine_fdt(phys_addr_t dt_phys)
                        cpu_relax();
        }
 
+       /* Retrieve various information from the /chosen node */
+       of_scan_flat_dt(early_init_dt_scan_chosen, boot_command_line);
+
+       /* Initialize {size,address}-cells info */
+       of_scan_flat_dt(early_init_dt_scan_root, NULL);
+
+       /* Setup memory, calling early_init_dt_add_memory_arch */
+       if (!IS_ENABLED(CONFIG_EFI))
+               of_scan_flat_dt(early_init_dt_scan_memory, NULL);
+
        machine_name = of_flat_dt_get_machine_name();
 }
 

drivers/firmware/efi/libstub/fdt.c +1 -24

index c846a96..a02e56e 100644
@@ -22,7 +22,7 @@ efi_status_t update_fdt(efi_system_table_t *sys_table, void *orig_fdt,
                        unsigned long map_size, unsigned long desc_size,
                        u32 desc_ver)
 {
-       int node, prev, num_rsv;
+       int node, num_rsv;
        int status;
        u32 fdt_val32;
        u64 fdt_val64;

@@ -52,28 +52,6 @@ efi_status_t update_fdt(efi_system_table_t *sys_table, void *orig_fdt,
                goto fdt_set_fail;
 
        /*
-        * Delete any memory nodes present. We must delete nodes which
-        * early_init_dt_scan_memory may try to use.
-        */
-       prev = 0;
-       for (;;) {
-               const char *type;
-               int len;
-
-               node = fdt_next_node(fdt, prev, NULL);
-               if (node < 0)
-                       break;
-
-               type = fdt_getprop(fdt, node, "device_type", &len);
-               if (type && strncmp(type, "memory", len) == 0) {
-                       fdt_del_node(fdt, node);
-                       continue;
-               }
-
-               prev = node;
-       }
-
-       /*
         * Delete all memory reserve map entries. When booting via UEFI,
         * kernel will use the UEFI memory map to find reserved regions.
         */
--

1.9.1

Thanks,
Ashok