arm64:numa: Add numa support for arm64 platforms.

Raising the maximum limit to 128. This is needed for Cavium's Thunder systems that will have 96 cores on Multi-node system.

Signed-off-by Ganapatrao Kulkarni <[email protected]>

---
 arch/arm64/Kconfig | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

arch/arm64/Kconfig +2 -2

index 4d42453..a409105 100644
@@ -296,8 +296,8 @@ config SCHED_SMT
 	  places. If unsure say N here.
 
 config NR_CPUS
-	int "Maximum number of CPUs (2-64)"
-	range 2 64
+	int "Maximum number of CPUs (2-128)"
+	range 2 128
 	depends on SMP
 	# These have to remain sorted largest to smallest
 	default "64"

Signature · 6 lines

1.8.1.4

--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo at vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Quoted 2 lines · Ganapatrao Kulkarni

Raising the maximum limit to 128. This is needed for Cavium's Thunder systems that will have 96 cores on Multi-node system.

Has this been tested on any such systems?

Mark.

Quoted 24 lines

Signed-off-by Ganapatrao Kulkarni <[email protected]>

---
 arch/arm64/Kconfig | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

arch/arm64/Kconfig +2 -3

index 4d42453..a409105 100644
@@ -296,8 +296,8 @@ config SCHED_SMT
        places. If unsure say N here.
 
 config NR_CPUS
-     int "Maximum number of CPUs (2-64)"
-     range 2 64
+     int "Maximum number of CPUs (2-128)"
+     range 2 128
      depends on SMP
      # These have to remain sorted largest to smallest
      default "64"
--

1.8.1.4

Quoted 5 lines

--
To unsubscribe from this list: send the line "unsubscribe devicetree" in the body of a message to majordomo at vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Adding Documentation for dt binding for memory to numa node mapping.

Signed-off-by Ganapatrao Kulkarni <[email protected]>

---
 Documentation/devicetree/bindings/arm/numa.txt | 60 ++++++++++++++++++++++++++
 1 file changed, 60 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/arm/numa.txt

Documentation/devicetree/bindings/arm/numa.txt new file +60 -0

new file mode 100644
index 0000000..1cdc6d3
@@ -0,0 +1,60 @@
+========================================================
+ARM numa id binding description
+========================================================
+
+========================================================
+1 - Introduction
+========================================================
+
+The device node  property nid (numa node id) can be added
+to memory device node to map the range of memory addresses
+as defined in property reg. The property nid maps the memory
+range to the numa node id, which is used to find the local
+and remote pages on numa aware systems.
+
+========================================================
+2 - nid property
+========================================================
+nid is required property of memory device node for
+numa enabled platforms.
+
+|------------------------------------------------------|
+|Property Type  | Usage | Value Type | Definition      |
+|------------------------------------------------------|
+|  nid          |  R    |    <u32>   | Numa Node id    |
+|               |       |            | for this memory |
+|------------------------------------------------------|
+
+========================================================
+4 - Example memory nodes with numa information
+========================================================
+
+Example 1 (2 memory nodes, each mapped to a numa node.):
+
+	memory at 00000000 {
+		device_type = "memory";
+		reg = <0x0 0x00000000 0x0 0x80000000>;
+		nid = <0x0>;
+	};
+
+	memory at 10000000000 {
+		device_type = "memory";
+		reg = <0x100 0x00000000 0x0 0x80000000>;
+		nid = <0x1>;
+	};
+
+Example 2 (multiple memory ranges in each memory node and mapped to numa node):
+
+	memory at 00000000 {
+		device_type = "memory";
+		reg = <0x0 0x00000000 0x0 0x80000000>,
+		      <0x1 0x00000000 0x0 0x80000000>;
+		nid = <0x0>;
+	};
+
+	memory at 10000000000 {
+		device_type = "memory";
+		reg = <0x100 0x00000000 0x0 0x80000000>;
+		reg = <0x100 0x80000000 0x0 0x80000000>;
+		nid = <0x1>;
+	};

Signature · 6 lines

1.8.1.4

--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo at vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Quoted 1 line · Ganapatrao Kulkarni

Adding Documentation for dt binding for memory to numa node mapping.

As I previously commented [1], this binding doesn't specify what a nid maps to in terms of the CPU hierarchy, and is thus unusable. The binding absolutely must be explicit about this, and NAK until it is.

Given we're seeing systems with increasing numbers of CPUs and increasingly complex interconnect hierarchies, I would expect at minimum that we would refer to elements in the cpu-map to define the relationship between memory banks and CPUs.

What does the interconnect/memory hierarchy look like in your system?

Mark.

[1] http://lists.infradead.org/pipermail/linux-arm-kernel/2014-September/288263.html

Quoted 76 lines

Signed-off-by Ganapatrao Kulkarni <[email protected]>

---
 Documentation/devicetree/bindings/arm/numa.txt | 60 ++++++++++++++++++++++++++
 1 file changed, 60 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/arm/numa.txt

Documentation/devicetree/bindings/arm/numa.txt new file +60 -1

new file mode 100644
index 0000000..1cdc6d3
@@ -0,0 +1,60 @@
+========================================================
+ARM numa id binding description
+========================================================
+
+========================================================
+1 - Introduction
+========================================================
+
+The device node  property nid (numa node id) can be added
+to memory device node to map the range of memory addresses
+as defined in property reg. The property nid maps the memory
+range to the numa node id, which is used to find the local
+and remote pages on numa aware systems.
+
+========================================================
+2 - nid property
+========================================================
+nid is required property of memory device node for
+numa enabled platforms.
+
+|------------------------------------------------------|
+|Property Type  | Usage | Value Type | Definition      |
+|------------------------------------------------------|
+|  nid          |  R    |    <u32>   | Numa Node id    |
+|               |       |            | for this memory |
+|------------------------------------------------------|
+
+========================================================
+4 - Example memory nodes with numa information
+========================================================
+
+Example 1 (2 memory nodes, each mapped to a numa node.):
+
+     memory at 00000000 {
+             device_type = "memory";
+             reg = <0x0 0x00000000 0x0 0x80000000>;
+             nid = <0x0>;
+     };
+
+     memory at 10000000000 {
+             device_type = "memory";
+             reg = <0x100 0x00000000 0x0 0x80000000>;
+             nid = <0x1>;
+     };
+
+Example 2 (multiple memory ranges in each memory node and mapped to numa node):
+
+     memory at 00000000 {
+             device_type = "memory";
+             reg = <0x0 0x00000000 0x0 0x80000000>,
+                   <0x1 0x00000000 0x0 0x80000000>;
+             nid = <0x0>;
+     };
+
+     memory at 10000000000 {
+             device_type = "memory";
+             reg = <0x100 0x00000000 0x0 0x80000000>;
+             reg = <0x100 0x80000000 0x0 0x80000000>;
+             nid = <0x1>;
+     };
--

1.8.1.4

Quoted 5 lines

--
To unsubscribe from this list: send the line "unsubscribe devicetree" in the body of a message to majordomo at vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Hi Mark,

Quoted 6 lines · Mark Rutland

Quoted 1 line · Ganapatrao Kulkarni

Adding Documentation for dt binding for memory to numa node mapping.

As I previously commented [1], this binding doesn't specify what a nid maps to in terms of the CPU hierarchy, and is thus unusable. The binding absolutely must be explicit about this, and NAK until it is.

The nid/numa node id is to map the each memory range/bank to numa node.

IIUC, the numa manages the resources based on which node they are tide to. with nid, i am trying to map the memory range to a node.
Same follows for the all IO peripherals and for CPUs.
for cpus, i am using cluster-id as a node id to map all cpus to node.

thunder has 2 nodes, in this patch, i have grouped all cpus which belongs to each node under cluster-id(cluster0, cluster1).

Quoted 6 lines

Given we're seeing systems with increasing numbers of CPUs and increasingly complex interconnect hierarchies, I would expect at minimum that we would refer to elements in the cpu-map to define the relationship between memory banks and CPUs.

What does the interconnect/memory hierarchy look like in your system?

In tunder, 2 SoCs (each has 48 cores and ram controllers and IOs) can be connected to form 2 node NUMA system.
in a SoC(within node) there is no hierarchy with respect to memory or IO access. However w.r.t GICv3,
48 cores are in each SoC/node are split in to 3 clusters each of 16 cores.

the MPIDR mapping for this topology is,
Aff0 is mapped to 16 cores within a cluster. Valid range is 0 to 0xf Aff1 is mapped to cluster number, valid values are 0,1 and 2. Aff2 is mapped to Socket-id/node id/SoC number. Valid values are 0 and 1.

Quoted 10 lines · Ganapatrao Kulkarni

Hi Mark,

Quoted 6 lines · Mark Rutland

Quoted 1 line · Ganapatrao Kulkarni

Adding Documentation for dt binding for memory to numa node mapping.

As I previously commented [1], this binding doesn't specify what a nid maps to in terms of the CPU hierarchy, and is thus unusable. The binding absolutely must be explicit about this, and NAK until it is.

The nid/numa node id is to map the each memory range/bank to numa node.

The issue is what constitutes a "numa node" is not defined. Hence the mapping a memory banks to a "nid" is just a mapping to an arbitrary number -- the mapping of this number to CPUs isn't defined.

Quoted 4 lines

IIUC, the numa manages the resources based on which node they are tide to. with nid, i am trying to map the memory range to a node.
Same follows for the all IO peripherals and for CPUs.
for cpus, i am using cluster-id as a node id to map all cpus to node.

I strongly suspect that this is not going to work for very long. I don't think relying on a mapping of nid to a top-level cluster-id is a good idea, especially given we have the facility to be more explicit through use of the cpu-map.

We don't need to handle all the possible cases from the start, but I'd rather we consistently used the cou-map to explicitly define the relationship between CPUs and memory.

Quoted 17 lines

thunder has 2 nodes, in this patch, i have grouped all cpus which belongs to each node under cluster-id(cluster0, cluster1).

Quoted 6 lines

Given we're seeing systems with increasing numbers of CPUs and increasingly complex interconnect hierarchies, I would expect at minimum that we would refer to elements in the cpu-map to define the relationship between memory banks and CPUs.

What does the interconnect/memory hierarchy look like in your system?

In tunder, 2 SoCs (each has 48 cores and ram controllers and IOs) can be connected to form 2 node NUMA system.
in a SoC(within node) there is no hierarchy with respect to memory or IO access. However w.r.t GICv3,
48 cores are in each SoC/node are split in to 3 clusters each of 16 cores.

Quoted 4 lines

the MPIDR mapping for this topology is,
Aff0 is mapped to 16 cores within a cluster. Valid range is 0 to 0xf Aff1 is mapped to cluster number, valid values are 0,1 and 2. Aff2 is mapped to Socket-id/node id/SoC number. Valid values are 0 and 1.

Thanks for the information.

Mark.

adding devicetree definition for thunder's 2 node topology. Defined cpu-map for all 96 cores of 2 node system.

Adding numa support for arm64 based platforms.
This version creates numa mapping by parsing the dt table. cpu to node id mapping is derived from cluster_id as defined in cpu-map. memory to node id mapping is derived from nid property of memory node.

arch/arm64/mm/numa.c new file +17 -0

+       pr_info("Faking a node at [mem %#018Lx-%#018Lx]\n",
+              0LLU, PFN_PHYS(max_pfn) - 1);
+
+       node_set(0, numa_nodes_parsed);
+       numa_add_memblk(0, 0, PFN_PHYS(max_pfn));
+
+       return 0;
+}
+
+/**
+ * early_init_dt_scan_numa_map - parse memory node and map nid to memory range.
+ */
+int __init early_init_dt_scan_numa_map(unsigned long node, const char *uname,
+                                    int depth, void *data)
+{
+       const char *type = of_get_flat_dt_prop(node, "device_type", NULL);
+       const __be32 *reg, *endp, *nid_prop;

Quoted 4 lines · Ganapatrao Kulkarni

Adding numa support for arm64 based platforms.
This version creates numa mapping by parsing the dt table. cpu to node id mapping is derived from cluster_id as defined in cpu-map. memory to node id mapping is derived from nid property of memory node.

[...]

arch/arm64/include/asm/numa.h line 17

Quoted 6 lines

arch/arm64/include/asm/numa.h new file +6 -0

+/*
+ * Too small node sizes may confuse the VM badly. Usually they
+ * result from BIOS bugs. So dont recognize nodes as standalone
+ * NUMA entities that have less than this amount of RAM listed:
+ */
+#define NODE_MIN_SIZE (4*1024*1024)

Why do these confuse the VM? what does BIOS have to do with arm64?

Review

Quoted 3 lines

(quoted diff) +2 -0

+
+#define parent_node(node)      (node)

Huh?

[...]

Quoted 4 lines

arch/arm64/mm/init.c +0 -0

@@ -168,6 +191,11 @@ void __init bootmem_init(void)
        min = PFN_UP(memblock_start_of_DRAM());
        max = PFN_DOWN(memblock_end_of_DRAM());
 

arch/arm64/mm/init.c void __init bootmem_init(void) · line 198

Quoted 5 lines

arch/arm64/mm/init.c +5 -0

+       high_memory = __va((max << PAGE_SHIFT) - 1) + 1;
+       max_pfn = max_low_pfn = max;
+
+       if (IS_ENABLED(CONFIG_NUMA))
+               arm64_numa_init();

Is this function defined if !CONFIG_NUMA? Surely it must do nothing in that case anyway?

[...]

arch/arm64/mm/numa.c line 117

Quoted 10 lines

arch/arm64/mm/numa.c new file +10 -0

+/*
+ *  Set the cpu to node and mem mapping
+ */
+void numa_store_cpu_info(cpu)
+{
+       cpu_to_node_map[cpu] = cpu_topology[cpu].cluster_id;
+       cpumask_set_cpu(cpu, node_to_cpumask_map[cpu_to_node_map[cpu]]);
+       set_numa_node(cpu_to_node_map[cpu]);
+       set_numa_mem(local_memory_node(cpu_to_node_map[cpu]));
+}

I don't like this. I think we need to be more explicit in the DT w.r.t. the relationship between memory and the CPU hierarchy.

I can imagine that we might end up with systems with multiple levels of NUMA hierarchy (using MPIDR_EL1.Aff{3,2}), and I'd rather that we were explcit as possible from the start w.r.t. the relationship between memory and groups of CPUs such that we don't end up with multiple ways of specifying said relationship, and horrible edge cases around implicit definitions (e.g. the nid to cluster mapping).

Quoted 4 lines

arch/arm64/mm/numa.c new file +3 -0

+/**
+ * dummy_numa_init - Fallback dummy NUMA init
+ *

arch/arm64/mm/numa.c line 386

Quoted 10 lines

arch/arm64/mm/numa.c new file +10 -0

+ * Used if there's no underlying NUMA architecture, NUMA initialization
+ * fails, or NUMA is disabled on the command line.
+ *
+ * Must online at least one node and add memory blocks that cover all
+ * allowed memory.  This function must not fail.
+ */
+static int __init dummy_numa_init(void)
+{
+       pr_info("%s\n",
+              numa_off ? "NUMA turned off" : "No NUMA configuration found");

Why not print "NUMA turned off" in numa_setup?

Quoted 22 lines

arch/arm64/mm/numa.c new file +21 -0

+       pr_info("Faking a node at [mem %#018Lx-%#018Lx]\n",
+              0LLU, PFN_PHYS(max_pfn) - 1);
+
+       node_set(0, numa_nodes_parsed);
+       numa_add_memblk(0, 0, PFN_PHYS(max_pfn));
+
+       return 0;
+}
+
+/**
+ * early_init_dt_scan_numa_map - parse memory node and map nid to memory range.
+ */
+int __init early_init_dt_scan_numa_map(unsigned long node, const char *uname,
+                                    int depth, void *data)
+{
+       const char *type = of_get_flat_dt_prop(node, "device_type", NULL);
+       const __be32 *reg, *endp, *nid_prop;
+       int l, nid;
+
+       /* We are scanning "memory" nodes only */
+       if (type == NULL) {

arch/arm64/mm/numa.c line 413

Quoted 6 lines

arch/arm64/mm/numa.c new file +6 -0

+               /*
+                * The longtrail doesn't have a device_type on the
+                * /memory node, so look for the node called /memory at 0.
+                */
+               if (depth != 1 || strcmp(uname, "memory at 0") != 0)
+                       return 0;

This has no place on arm64.

We limited to longtrail workaround in the core memory parsing to PPC32 only in commit b44aa25d20e2ef6b (of: Handle memory at 0 node on PPC32 only). This code doesn't need it enabled ever.

Are you booting using UEFI? This isn't going to work when the memory map

comes from UEFI and we have no memory nodes in the DTB.

Mark.

Hi Mark,

Quoted 8 lines · Mark Rutland

Quoted 4 lines · Ganapatrao Kulkarni

Adding numa support for arm64 based platforms.
This version creates numa mapping by parsing the dt table. cpu to node id mapping is derived from cluster_id as defined in cpu-map. memory to node id mapping is derived from nid property of memory node.

[...]

Quoted 8 lines

arch/arm64/include/asm/numa.h line 17

Quoted 6 lines

arch/arm64/include/asm/numa.h new file +6 -0

+/*
+ * Too small node sizes may confuse the VM badly. Usually they
+ * result from BIOS bugs. So dont recognize nodes as standalone
+ * NUMA entities that have less than this amount of RAM listed:
+ */
+#define NODE_MIN_SIZE (4*1024*1024)

Why do these confuse the VM? what does BIOS have to do with arm64?

sneaked in from x86, will remove this.

Quoted 5 lines

Review

Quoted 3 lines

(quoted diff) +2 -0

+
+#define parent_node(node)      (node)

Huh?

for thunder, no hierarchy at numa nodes. shall i put under ifdef or separate header file?

Quoted 7 lines

[...]

Quoted 4 lines

arch/arm64/mm/init.c +0 -0

@@ -168,6 +191,11 @@ void __init bootmem_init(void)
        min = PFN_UP(memblock_start_of_DRAM());
        max = PFN_DOWN(memblock_end_of_DRAM());
 

Quoted 8 lines

arch/arm64/mm/init.c void __init bootmem_init(void) · line 198

Quoted 5 lines

arch/arm64/mm/init.c +5 -0

+       high_memory = __va((max << PAGE_SHIFT) - 1) + 1;
+       max_pfn = max_low_pfn = max;
+
+       if (IS_ENABLED(CONFIG_NUMA))
+               arm64_numa_init();

Is this function defined if !CONFIG_NUMA? Surely it must do nothing in that case anyway?

yes, if is not required, will remove it.

Quoted 17 lines

[...]

arch/arm64/mm/numa.c line 117

Quoted 10 lines

arch/arm64/mm/numa.c new file +10 -0

+/*
+ *  Set the cpu to node and mem mapping
+ */
+void numa_store_cpu_info(cpu)
+{
+       cpu_to_node_map[cpu] = cpu_topology[cpu].cluster_id;
+       cpumask_set_cpu(cpu, node_to_cpumask_map[cpu_to_node_map[cpu]]);
+       set_numa_node(cpu_to_node_map[cpu]);
+       set_numa_mem(local_memory_node(cpu_to_node_map[cpu]));
+}

I don't like this. I think we need to be more explicit in the DT w.r.t. the relationship between memory and the CPU hierarchy.

Quoted 6 lines

I can imagine that we might end up with systems with multiple levels of NUMA hierarchy (using MPIDR_EL1.Aff{3,2}), and I'd rather that we were explcit as possible from the start w.r.t. the relationship between memory and groups of CPUs such that we don't end up with multiple ways of specifying said relationship, and horrible edge cases around implicit definitions (e.g. the nid to cluster mapping).

are you recomending to have explicit nid attribute to each cpu device node?

Quoted 7 lines

Quoted 7 lines

+/**
+ * dummy_numa_init - Fallback dummy NUMA init
+ *
+ * Used if there's no underlying NUMA architecture, NUMA initialization
+ * fails, or NUMA is disabled on the command line.
+ *
+ * Must online at least one node and add memory blocks that cover all

Quoted 8 lines

arch/arm64/mm/numa.c line 386

Quoted 6 lines

+ * allowed memory. This function must not fail.
+ */

arch/arm64/mm/numa.c new file +4 -0

+static int __init dummy_numa_init(void)
+{
+       pr_info("%s\n",
+              numa_off ? "NUMA turned off" : "No NUMA configuration found");

Why not print "NUMA turned off" in numa_setup?

enters this function only when, numa is turned off or the DT/ACPI based numa init fails.

Quoted 22 lines

Quoted 22 lines

arch/arm64/mm/numa.c new file +21 -0

+       pr_info("Faking a node at [mem %#018Lx-%#018Lx]\n",
+              0LLU, PFN_PHYS(max_pfn) - 1);
+
+       node_set(0, numa_nodes_parsed);
+       numa_add_memblk(0, 0, PFN_PHYS(max_pfn));
+
+       return 0;
+}
+
+/**
+ * early_init_dt_scan_numa_map - parse memory node and map nid to memory range.
+ */
+int __init early_init_dt_scan_numa_map(unsigned long node, const char *uname,
+                                    int depth, void *data)
+{
+       const char *type = of_get_flat_dt_prop(node, "device_type", NULL);
+       const __be32 *reg, *endp, *nid_prop;
+       int l, nid;
+
+       /* We are scanning "memory" nodes only */
+       if (type == NULL) {

Quoted 8 lines

arch/arm64/mm/numa.c line 413

Quoted 6 lines

arch/arm64/mm/numa.c new file +6 -0

+               /*
+                * The longtrail doesn't have a device_type on the
+                * /memory node, so look for the node called /memory at 0.
+                */
+               if (depth != 1 || strcmp(uname, "memory at 0") != 0)
+                       return 0;

This has no place on arm64.

i am not sure that we can move to driver/of, at this moment this is arm64 specific binding.

Quoted 5 lines

We limited to longtrail workaround in the core memory parsing to PPC32 only in commit b44aa25d20e2ef6b (of: Handle memory at 0 node on PPC32 only). This code doesn't need it enabled ever.

Are you booting using UEFI? This isn't going to work when the memory map

tried with bootwrapper, working on to boot from UEFI.

Quoted 1 line

comes from UEFI and we have no memory nodes in the DTB.

yes, there is issue with UEFI boot, since memory node is removed. i request UEFI stub dev-team to suggest the possible ways to address this.

Quoted 25 lines · Ganapatrao Kulkarni

Hi Mark,

Quoted 8 lines · Mark Rutland

Quoted 4 lines · Ganapatrao Kulkarni

Adding numa support for arm64 based platforms.
This version creates numa mapping by parsing the dt table. cpu to node id mapping is derived from cluster_id as defined in cpu-map. memory to node id mapping is derived from nid property of memory node.

[...]

Quoted 8 lines

arch/arm64/include/asm/numa.h line 17

Quoted 6 lines

arch/arm64/include/asm/numa.h new file +6 -0

+/*
+ * Too small node sizes may confuse the VM badly. Usually they
+ * result from BIOS bugs. So dont recognize nodes as standalone
+ * NUMA entities that have less than this amount of RAM listed:
+ */
+#define NODE_MIN_SIZE (4*1024*1024)

Why do these confuse the VM? what does BIOS have to do with arm64?

sneaked in from x86, will remove this.

Quoted 5 lines

Review

Quoted 3 lines

(quoted diff) +2 -0

+
+#define parent_node(node)      (node)

Huh?

Quoted 2 lines

for thunder, no hierarchy at numa nodes. shall i put under ifdef or separate header file?

I was confused by a node being its own parent, but that seems to be the case elsewhere for parent_node() implementations. Please at least have a comment that we're assuming a flat hierarchy (for now).

Quoted 40 lines

Quoted 8 lines

[...]

Quoted 4 lines

arch/arm64/mm/init.c +0 -0

@@ -168,6 +191,11 @@ void __init bootmem_init(void)
        min = PFN_UP(memblock_start_of_DRAM());
        max = PFN_DOWN(memblock_end_of_DRAM());
 

Quoted 8 lines

arch/arm64/mm/init.c void __init bootmem_init(void) · line 198

Quoted 5 lines

arch/arm64/mm/init.c +5 -0

+       high_memory = __va((max << PAGE_SHIFT) - 1) + 1;
+       max_pfn = max_low_pfn = max;
+
+       if (IS_ENABLED(CONFIG_NUMA))
+               arm64_numa_init();

Is this function defined if !CONFIG_NUMA? Surely it must do nothing in that case anyway?

yes, if is not required, will remove it.

Quoted 23 lines

[...]

arch/arm64/mm/numa.c line 117

Quoted 10 lines

arch/arm64/mm/numa.c new file +10 -0

+/*
+ *  Set the cpu to node and mem mapping
+ */
+void numa_store_cpu_info(cpu)
+{
+       cpu_to_node_map[cpu] = cpu_topology[cpu].cluster_id;
+       cpumask_set_cpu(cpu, node_to_cpumask_map[cpu_to_node_map[cpu]]);
+       set_numa_node(cpu_to_node_map[cpu]);
+       set_numa_mem(local_memory_node(cpu_to_node_map[cpu]));
+}

I don't like this. I think we need to be more explicit in the DT w.r.t. the relationship between memory and the CPU hierarchy.

I can imagine that we might end up with systems with multiple levels of NUMA hierarchy (using MPIDR_EL1.Aff{3,2}), and I'd rather that we were explcit as possible from the start w.r.t. the relationship between memory and groups of CPUs such that we don't end up with multiple ways of specifying said relationship, and horrible edge cases around implicit definitions (e.g. the nid to cluster mapping).

Quoted 1 line

are you recomending to have explicit nid attribute to each cpu device node?

I am recommending that we make the relationship explicit. If anything, using the cpu-map (with phandles) seems like a better approach.

Quoted 16 lines

Quoted 16 lines

Quoted 4 lines

arch/arm64/mm/numa.c new file +3 -0

+/**
+ * dummy_numa_init - Fallback dummy NUMA init
+ *

arch/arm64/mm/numa.c line 386

Quoted 10 lines

arch/arm64/mm/numa.c new file +10 -0

+ * Used if there's no underlying NUMA architecture, NUMA initialization
+ * fails, or NUMA is disabled on the command line.
+ *
+ * Must online at least one node and add memory blocks that cover all
+ * allowed memory.  This function must not fail.
+ */
+static int __init dummy_numa_init(void)
+{
+       pr_info("%s\n",
+              numa_off ? "NUMA turned off" : "No NUMA configuration found");

Why not print "NUMA turned off" in numa_setup?

Quoted 2 lines

enters this function only when, numa is turned off or the DT/ACPI based numa init fails.

Sure. Moving the "NUMA turned off" print into numa_setup would mean you could just print "Using dummy NUMA layout" or something to that effect here -- the function has no need to care about the value of numa_off.

Quoted 31 lines

Quoted 31 lines

Quoted 19 lines

arch/arm64/mm/numa.c new file +17 -0

+       pr_info("Faking a node at [mem %#018Lx-%#018Lx]\n",
+              0LLU, PFN_PHYS(max_pfn) - 1);
+
+       node_set(0, numa_nodes_parsed);
+       numa_add_memblk(0, 0, PFN_PHYS(max_pfn));
+
+       return 0;
+}
+
+/**
+ * early_init_dt_scan_numa_map - parse memory node and map nid to memory range.
+ */
+int __init early_init_dt_scan_numa_map(unsigned long node, const char *uname,
+                                    int depth, void *data)
+{
+       const char *type = of_get_flat_dt_prop(node, "device_type", NULL);
+       const __be32 *reg, *endp, *nid_prop;

arch/arm64/mm/numa.c line 413

Quoted 10 lines

arch/arm64/mm/numa.c new file +10 -0

+       int l, nid;
+
+       /* We are scanning "memory" nodes only */
+       if (type == NULL) {
+               /*
+                * The longtrail doesn't have a device_type on the
+                * /memory node, so look for the node called /memory at 0.
+                */
+               if (depth != 1 || strcmp(uname, "memory at 0") != 0)
+                       return 0;

This has no place on arm64.

Quoted 2 lines

i am not sure that we can move to driver/of, at this moment this is arm64 specific binding.

I meant the longtrail workaround, hence my comments on that below.

Quoted 5 lines

Quoted 5 lines

We limited to longtrail workaround in the core memory parsing to PPC32 only in commit b44aa25d20e2ef6b (of: Handle memory at 0 node on PPC32 only). This code doesn't need it enabled ever.

Quoted 5 lines

Quoted 1 line

Are you booting using UEFI? This isn't going to work when the memory map

tried with bootwrapper, working on to boot from UEFI.

Quoted 1 line

comes from UEFI and we have no memory nodes in the DTB.

yes, there is issue with UEFI boot, since memory node is removed. i request UEFI stub dev-team to suggest the possible ways to address this.

I've Cc'd a few people who have worked on the stub and/or EFI memory map stuff. It would be worth keeping them on Cc so as to keep them informed.

I believe that the EFI stub is doing the right thing by ensuring that the EFI memory map is used, so this is just another configuration that your binding has to consider.

Mark.

Quoted 26 lines · Mark Rutland

Quoted 25 lines · Ganapatrao Kulkarni

Hi Mark,

Quoted 8 lines · Mark Rutland

Quoted 4 lines · Ganapatrao Kulkarni

Adding numa support for arm64 based platforms.
This version creates numa mapping by parsing the dt table. cpu to node id mapping is derived from cluster_id as defined in cpu-map. memory to node id mapping is derived from nid property of memory node.

[...]

Quoted 8 lines

arch/arm64/include/asm/numa.h line 17

Quoted 6 lines

arch/arm64/include/asm/numa.h new file +6 -0

+/*
+ * Too small node sizes may confuse the VM badly. Usually they
+ * result from BIOS bugs. So dont recognize nodes as standalone
+ * NUMA entities that have less than this amount of RAM listed:
+ */
+#define NODE_MIN_SIZE (4*1024*1024)

Why do these confuse the VM? what does BIOS have to do with arm64?

sneaked in from x86, will remove this.

Quoted 5 lines

Review

Quoted 3 lines

(quoted diff) +2 -0

+
+#define parent_node(node)      (node)

Huh?

Quoted 6 lines

Quoted 2 lines

for thunder, no hierarchy at numa nodes. shall i put under ifdef or separate header file?

I was confused by a node being its own parent, but that seems to be the case elsewhere for parent_node() implementations. Please at least have a comment that we're assuming a flat hierarchy (for now).

sure, will add the required comments.

Quoted 40 lines

Quoted 40 lines

Quoted 8 lines

[...]

Quoted 4 lines

arch/arm64/mm/init.c +0 -0

@@ -168,6 +191,11 @@ void __init bootmem_init(void)
        min = PFN_UP(memblock_start_of_DRAM());
        max = PFN_DOWN(memblock_end_of_DRAM());
 

Quoted 8 lines

arch/arm64/mm/init.c void __init bootmem_init(void) · line 198

Quoted 5 lines

arch/arm64/mm/init.c +5 -0

+       high_memory = __va((max << PAGE_SHIFT) - 1) + 1;
+       max_pfn = max_low_pfn = max;
+
+       if (IS_ENABLED(CONFIG_NUMA))
+               arm64_numa_init();

Is this function defined if !CONFIG_NUMA? Surely it must do nothing in that case anyway?

yes, if is not required, will remove it.

Quoted 23 lines

[...]

arch/arm64/mm/numa.c line 117

Quoted 10 lines

arch/arm64/mm/numa.c new file +10 -0

+/*
+ *  Set the cpu to node and mem mapping
+ */
+void numa_store_cpu_info(cpu)
+{
+       cpu_to_node_map[cpu] = cpu_topology[cpu].cluster_id;
+       cpumask_set_cpu(cpu, node_to_cpumask_map[cpu_to_node_map[cpu]]);
+       set_numa_node(cpu_to_node_map[cpu]);
+       set_numa_mem(local_memory_node(cpu_to_node_map[cpu]));
+}

I don't like this. I think we need to be more explicit in the DT w.r.t. the relationship between memory and the CPU hierarchy.

I can imagine that we might end up with systems with multiple levels of NUMA hierarchy (using MPIDR_EL1.Aff{3,2}), and I'd rather that we were explcit as possible from the start w.r.t. the relationship between memory and groups of CPUs such that we don't end up with multiple ways of specifying said relationship, and horrible edge cases around implicit definitions (e.g. the nid to cluster mapping).

Quoted 4 lines

Quoted 1 line

are you recomending to have explicit nid attribute to each cpu device node?

I am recommending that we make the relationship explicit. If anything, using the cpu-map (with phandles) seems like a better approach.

yes, will add the mapping.

Quoted 16 lines

Quoted 16 lines

Quoted 16 lines

Quoted 4 lines

arch/arm64/mm/numa.c new file +3 -0

+/**
+ * dummy_numa_init - Fallback dummy NUMA init
+ *

arch/arm64/mm/numa.c line 386

Quoted 10 lines

arch/arm64/mm/numa.c new file +10 -0

+ * Used if there's no underlying NUMA architecture, NUMA initialization
+ * fails, or NUMA is disabled on the command line.
+ *
+ * Must online at least one node and add memory blocks that cover all
+ * allowed memory.  This function must not fail.
+ */
+static int __init dummy_numa_init(void)
+{
+       pr_info("%s\n",
+              numa_off ? "NUMA turned off" : "No NUMA configuration found");

Why not print "NUMA turned off" in numa_setup?

Quoted 6 lines

Quoted 2 lines

enters this function only when, numa is turned off or the DT/ACPI based numa init fails.

Sure. Moving the "NUMA turned off" print into numa_setup would mean you could just print "Using dummy NUMA layout" or something to that effect here -- the function has no need to care about the value of numa_off.

agreed.

Quoted 31 lines

Quoted 31 lines

Quoted 31 lines

Quoted 19 lines

arch/arm64/mm/numa.c new file +17 -0

+       pr_info("Faking a node at [mem %#018Lx-%#018Lx]\n",
+              0LLU, PFN_PHYS(max_pfn) - 1);
+
+       node_set(0, numa_nodes_parsed);
+       numa_add_memblk(0, 0, PFN_PHYS(max_pfn));
+
+       return 0;
+}
+
+/**
+ * early_init_dt_scan_numa_map - parse memory node and map nid to memory range.
+ */
+int __init early_init_dt_scan_numa_map(unsigned long node, const char *uname,
+                                    int depth, void *data)
+{
+       const char *type = of_get_flat_dt_prop(node, "device_type", NULL);
+       const __be32 *reg, *endp, *nid_prop;

arch/arm64/mm/numa.c line 413

Quoted 10 lines

arch/arm64/mm/numa.c new file +10 -0

+       int l, nid;
+
+       /* We are scanning "memory" nodes only */
+       if (type == NULL) {
+               /*
+                * The longtrail doesn't have a device_type on the
+                * /memory node, so look for the node called /memory at 0.
+                */
+               if (depth != 1 || strcmp(uname, "memory at 0") != 0)
+                       return 0;

This has no place on arm64.

Quoted 4 lines

Quoted 2 lines

i am not sure that we can move to driver/of, at this moment this is arm64 specific binding.

I meant the longtrail workaround, hence my comments on that below.

oh! thanks, will remove this.

Quoted 5 lines

Quoted 5 lines

Quoted 5 lines

We limited to longtrail workaround in the core memory parsing to PPC32 only in commit b44aa25d20e2ef6b (of: Handle memory at 0 node on PPC32 only). This code doesn't need it enabled ever.

Quoted 8 lines

Quoted 5 lines

Quoted 1 line

Are you booting using UEFI? This isn't going to work when the memory map

tried with bootwrapper, working on to boot from UEFI.

Quoted 1 line

comes from UEFI and we have no memory nodes in the DTB.

yes, there is issue with UEFI boot, since memory node is removed. i request UEFI stub dev-team to suggest the possible ways to address this.

I've Cc'd a few people who have worked on the stub and/or EFI memory map stuff. It would be worth keeping them on Cc so as to keep them informed.

thanks.

Quoted 4 lines

I believe that the EFI stub is doing the right thing by ensuring that the EFI memory map is used, so this is just another configuration that your binding has to consider.

going through EFI stub, next is to boot numa kernel using UEFI, will include UEFI boot support in v2 patch.

Quoted 2 lines

Mark.

Hi All,

Quoted 151 lines · Ganapatrao Kulkarni

Quoted 26 lines · Mark Rutland

Quoted 25 lines · Ganapatrao Kulkarni

Hi Mark,

Quoted 8 lines · Mark Rutland

Quoted 4 lines · Ganapatrao Kulkarni

Adding numa support for arm64 based platforms.
This version creates numa mapping by parsing the dt table. cpu to node id mapping is derived from cluster_id as defined in cpu-map. memory to node id mapping is derived from nid property of memory node.

[...]

Quoted 8 lines

arch/arm64/include/asm/numa.h line 17

Quoted 6 lines

arch/arm64/include/asm/numa.h new file +6 -0

+/*
+ * Too small node sizes may confuse the VM badly. Usually they
+ * result from BIOS bugs. So dont recognize nodes as standalone
+ * NUMA entities that have less than this amount of RAM listed:
+ */
+#define NODE_MIN_SIZE (4*1024*1024)

Why do these confuse the VM? what does BIOS have to do with arm64?

sneaked in from x86, will remove this.

Quoted 5 lines

Review

Quoted 3 lines

(quoted diff) +2 -0

+
+#define parent_node(node)      (node)

Huh?

Quoted 6 lines

Quoted 2 lines

for thunder, no hierarchy at numa nodes. shall i put under ifdef or separate header file?

I was confused by a node being its own parent, but that seems to be the case elsewhere for parent_node() implementations. Please at least have a comment that we're assuming a flat hierarchy (for now).

sure, will add the required comments.

Quoted 40 lines

Quoted 40 lines

Quoted 8 lines

[...]

Quoted 4 lines

arch/arm64/mm/init.c +0 -0

@@ -168,6 +191,11 @@ void __init bootmem_init(void)
        min = PFN_UP(memblock_start_of_DRAM());
        max = PFN_DOWN(memblock_end_of_DRAM());
 

Quoted 8 lines

arch/arm64/mm/init.c void __init bootmem_init(void) · line 198

Quoted 5 lines

arch/arm64/mm/init.c +5 -0

+       high_memory = __va((max << PAGE_SHIFT) - 1) + 1;
+       max_pfn = max_low_pfn = max;
+
+       if (IS_ENABLED(CONFIG_NUMA))
+               arm64_numa_init();

Is this function defined if !CONFIG_NUMA? Surely it must do nothing in that case anyway?

yes, if is not required, will remove it.

Quoted 23 lines

[...]

arch/arm64/mm/numa.c line 117

Quoted 10 lines

arch/arm64/mm/numa.c new file +10 -0

+/*
+ *  Set the cpu to node and mem mapping
+ */
+void numa_store_cpu_info(cpu)
+{
+       cpu_to_node_map[cpu] = cpu_topology[cpu].cluster_id;
+       cpumask_set_cpu(cpu, node_to_cpumask_map[cpu_to_node_map[cpu]]);
+       set_numa_node(cpu_to_node_map[cpu]);
+       set_numa_mem(local_memory_node(cpu_to_node_map[cpu]));
+}

I don't like this. I think we need to be more explicit in the DT w.r.t. the relationship between memory and the CPU hierarchy.

I can imagine that we might end up with systems with multiple levels of NUMA hierarchy (using MPIDR_EL1.Aff{3,2}), and I'd rather that we were explcit as possible from the start w.r.t. the relationship between memory and groups of CPUs such that we don't end up with multiple ways of specifying said relationship, and horrible edge cases around implicit definitions (e.g. the nid to cluster mapping).

Quoted 4 lines

Quoted 1 line

are you recomending to have explicit nid attribute to each cpu device node?

I am recommending that we make the relationship explicit. If anything, using the cpu-map (with phandles) seems like a better approach.

yes, will add the mapping.

Quoted 16 lines

Quoted 16 lines

Quoted 16 lines

Quoted 4 lines

arch/arm64/mm/numa.c new file +3 -0

+/**
+ * dummy_numa_init - Fallback dummy NUMA init
+ *

arch/arm64/mm/numa.c line 386

Quoted 10 lines

arch/arm64/mm/numa.c new file +10 -0

+ * Used if there's no underlying NUMA architecture, NUMA initialization
+ * fails, or NUMA is disabled on the command line.
+ *
+ * Must online at least one node and add memory blocks that cover all
+ * allowed memory.  This function must not fail.
+ */
+static int __init dummy_numa_init(void)
+{
+       pr_info("%s\n",
+              numa_off ? "NUMA turned off" : "No NUMA configuration found");

Why not print "NUMA turned off" in numa_setup?

Quoted 6 lines

Quoted 2 lines

enters this function only when, numa is turned off or the DT/ACPI based numa init fails.

Sure. Moving the "NUMA turned off" print into numa_setup would mean you could just print "Using dummy NUMA layout" or something to that effect here -- the function has no need to care about the value of numa_off.

agreed.

Quoted 31 lines

Quoted 31 lines

Quoted 31 lines

Quoted 19 lines

arch/arm64/mm/numa.c new file +17 -0

+       pr_info("Faking a node at [mem %#018Lx-%#018Lx]\n",
+              0LLU, PFN_PHYS(max_pfn) - 1);
+
+       node_set(0, numa_nodes_parsed);
+       numa_add_memblk(0, 0, PFN_PHYS(max_pfn));
+
+       return 0;
+}
+
+/**
+ * early_init_dt_scan_numa_map - parse memory node and map nid to memory range.
+ */
+int __init early_init_dt_scan_numa_map(unsigned long node, const char *uname,
+                                    int depth, void *data)
+{
+       const char *type = of_get_flat_dt_prop(node, "device_type", NULL);
+       const __be32 *reg, *endp, *nid_prop;

arch/arm64/mm/numa.c line 413

Quoted 10 lines

arch/arm64/mm/numa.c new file +10 -0

+       int l, nid;
+
+       /* We are scanning "memory" nodes only */
+       if (type == NULL) {
+               /*
+                * The longtrail doesn't have a device_type on the
+                * /memory node, so look for the node called /memory at 0.
+                */
+               if (depth != 1 || strcmp(uname, "memory at 0") != 0)
+                       return 0;

This has no place on arm64.

Quoted 4 lines

Quoted 2 lines

i am not sure that we can move to driver/of, at this moment this is arm64 specific binding.

I meant the longtrail workaround, hence my comments on that below.

oh! thanks, will remove this.

Quoted 5 lines

Quoted 5 lines

Quoted 5 lines

We limited to longtrail workaround in the core memory parsing to PPC32 only in commit b44aa25d20e2ef6b (of: Handle memory at 0 node on PPC32 only). This code doesn't need it enabled ever.

Quoted 8 lines

Quoted 5 lines

Quoted 1 line

Are you booting using UEFI? This isn't going to work when the memory map

tried with bootwrapper, working on to boot from UEFI.

Quoted 1 line

comes from UEFI and we have no memory nodes in the DTB.

yes, there is issue with UEFI boot, since memory node is removed. i request UEFI stub dev-team to suggest the possible ways to address this.

I've Cc'd a few people who have worked on the stub and/or EFI memory map stuff. It would be worth keeping them on Cc so as to keep them informed.

Quoted 9 lines

thanks.

Quoted 4 lines

I believe that the EFI stub is doing the right thing by ensuring that the EFI memory map is used, so this is just another configuration that your binding has to consider.

going through EFI stub, next is to boot numa kernel using UEFI, will include UEFI boot support in v2 patch.

Quoted 2 lines

Mark.

Below is the example for the proposal of numa bindings in DT. This covers cpu to node mapping, memory ranges to node mapping. Also defines proximity distance matrix of nodes to each other. please let me know your comments to go ahead with the implementation.

numa-map{
         /*  Address cells used for memory range base address in mem-map.
             For all others, size-cells is used.
             Node-count tells the number of numa nodes in the system.
         */
         #address-cells = <2>;
         #size-cells = <1>;
         #node-count = <4>;

         /* Memmap for memory ranges on each node>

         mem-map = <0x0   0x00c00000 0>,
           <0x1   0x00000000 1>,
           <0x100 0x00000000 2>,
           <0x200 0x00000000 3>;

         /* CPU to node map for 4 NODE and 16 CPUs system
        < first-cpu last-cpu  node belongs>
         */
         cpu-map = <0 3 0>,
                         <4 7 1>,
                        <8 11 2>,
                        <12 16 3>;

         /*Proximity Distance matrix for 4Node system
                       <from-node to-node distance>
        */
         node-matrix=    <0 0 10>,
                               <0 1 20>,
                               <0 2 30>,
                               <0 3 10>,
                               <1 0 20>,
                                <1 1 10>,
                               <1 2 30>,
                              <1 3 10>,
                              <2 0 30>,
                              <2 1 20>,
                              <2 2 10>,
                             <2 3 10>,
                            <3 0 10>,
                            <3 1 20>,
                            <3 2 30>,
                            <3 3 10>,

}

On Fri, Oct 17, 2014 at 10:49:56PM +0530, Ganapatrao Kulkarni wrote:

         /*Proximity Distance matrix for 4Node system

Quoted 18 lines

                       <from-node to-node distance>
        */
         node-matrix=    <0 0 10>,
                               <0 1 20>,
                               <0 2 30>,
                               <0 3 10>,
                               <1 0 20>,
                                <1 1 10>,
                               <1 2 30>,
                              <1 3 10>,
                              <2 0 30>,
                              <2 1 20>,
                              <2 2 10>,
                             <2 3 10>,
                            <3 0 10>,
                            <3 1 20>,
                            <3 2 30>,
                            <3 3 10>,

Quoted 1 line

}

Hi Ganapat,
The above caught my attention.

For a 4-node system do we not need 16 distances; the implication of that would be that the distance between node A-B could be different from the distance between B-A? Also the distance from a node to itself could be safely assumed to be zero?

I think we should have a symmetric matrix with zero-diagonals so strictly only seven values would need specifying for a 4-node system.

Cheers,

Signature · 1 line

Steve

Quoted 1 line · Steve Capper

On Fri, Oct 17, 2014 at 10:49:56PM +0530, Ganapatrao Kulkarni wrote:

[...]

Quoted 24 lines

         /*Proximity Distance matrix for 4Node system

Quoted 18 lines

                       <from-node to-node distance>
        */
         node-matrix=    <0 0 10>,
                               <0 1 20>,
                               <0 2 30>,
                               <0 3 10>,
                               <1 0 20>,
                                <1 1 10>,
                               <1 2 30>,
                              <1 3 10>,
                              <2 0 30>,
                              <2 1 20>,
                              <2 2 10>,
                             <2 3 10>,
                            <3 0 10>,
                            <3 1 20>,
                            <3 2 30>,
                            <3 3 10>,

Quoted 1 line

}

Hi Ganapat,
The above caught my attention.

Quoted 7 lines

For a 4-node system do we not need 16 distances; the implication of that would be that the distance between node A-B could be different from the distance between B-A? Also the distance from a node to itself could be safely assumed to be zero?

I think we should have a symmetric matrix with zero-diagonals so strictly only seven values would need specifying for a 4-node system.

s/seven/six/

I really need to learn how to count.... :-/

Quoted 4 lines

Cheers,
--
Steve

Below is the example for the proposal of numa bindings in DT. This covers cpu to node mapping, memory ranges to node mapping. Also defines proximity distance matrix of nodes to each other. please let me know your comments to go ahead with the implementation.

numa-map{
         /*  Address cells used for memory range base address in mem-map.
             For all others, size-cells is used.
             Node-count tells the number of numa nodes in the system.
         */
         #address-cells = <2>;
         #size-cells = <1>;
         #node-count = <4>;

         /* Memmap for memory ranges on each node>

         mem-map = <0x0   0x00c00000 0>,
           <0x1   0x00000000 1>,
           <0x100 0x00000000 2>,
           <0x200 0x00000000 3>;

         /* CPU to node map for 4 NODE and 16 CPUs system
        < first-cpu last-cpu  node belongs>

         */
         cpu-map = <0 3 0>,
                         <4 7 1>,
                        <8 11 2>,
                        <12 16 3>;

Hi Ganapatrao,

On 2014-10-18 1:19, Ganapatrao Kulkarni wrote:
[...]

Quoted 21 lines

Below is the example for the proposal of numa bindings in DT. This covers cpu to node mapping, memory ranges to node mapping. Also defines proximity distance matrix of nodes to each other. please let me know your comments to go ahead with the implementation.

numa-map{
         /*  Address cells used for memory range base address in mem-map.
             For all others, size-cells is used.
             Node-count tells the number of numa nodes in the system.
         */
         #address-cells = <2>;
         #size-cells = <1>;
         #node-count = <4>;

         /* Memmap for memory ranges on each node>

         mem-map = <0x0   0x00c00000 0>,
           <0x1   0x00000000 1>,
           <0x100 0x00000000 2>,
           <0x200 0x00000000 3>;

Quoted 2 lines

/* CPU to node map for 4 NODE and 16 CPUs system
< first-cpu last-cpu node belongs>

What's the property for the cpu? MPIDR of this CPU?

Quoted 7 lines

         */
         cpu-map = <0 3 0>,
                         <4 7 1>,
                        <8 11 2>,
                        <12 16 3>;

Thanks
Hanjun

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel at lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel