Skip to contentSkip to conversation

Offline memoryless cpuless node 0

PATCH Superseded Srikar Dronamraju 3 participants 9 messages May 1, 2020 – May 11, 2020

Subsystems (3)

v3 of 6 Superseded Latest is v6 →

lore
Share & tools
lore
#0 lore raw

v3 cover Offline memoryless cpuless node 0

Changelog v2:->v3:
- Resolved comments from Gautham.
Link v2: https://lore.kernel.org/linuxppc-dev/[email protected]/t/#u

Changelog v1:->v2:
- Rebased to v5.7-rc3
- Updated the changelog.
Link v1: https://lore.kernel.org/linuxppc-dev/[email protected]/t/#u

Linux kernel configured with CONFIG_NUMA on a system with multiple possible nodes, marks node 0 as online at boot. However in practice, there are systems which have node 0 as memoryless and cpuless.

This can cause
1. numa_balancing to be enabled on systems with only one online node.
2. Existence of dummy (cpuless and memoryless) node which can confuse users/scripts looking at output of lscpu / numactl.

This patchset wants to correct this anomaly.

This should only affect systems that have CONFIG_MEMORYLESS_NODES. Currently there are only 2 architectures ia64 and powerpc that have this config.

Note: Patch 3 in this patch series depends on patches 1 and 2. Without patches 1 and 2, patch 3 might crash powerpc.

v5.7-rc3
available: 2 nodes (0,2)
node 0 cpus:
node 0 size: 0 MB
node 0 free: 0 MB
node 2 cpus: 0 1 2 3 4 5 6 7
node 2 size: 32625 MB
node 2 free: 31490 MB
node distances:
node 0 2

   0:  10  20
   2:  20  10

proc and sys files
------------------

 /sys/devices/system/node/online:            0,2
 /proc/sys/kernel/numa_balancing:            1
 /sys/devices/system/node/has_cpu:           2
 /sys/devices/system/node/has_memory:        2

/sys/devices/system/node/has_normal_memory: 2
/sys/devices/system/node/possible: 0-31

v5.7-rc3 + patches
------------------
available: 1 nodes (2)
node 2 cpus: 0 1 2 3 4 5 6 7
node 2 size: 32625 MB
node 2 free: 31487 MB
node distances:
node 2
2: 10

proc and sys files
------------------

/sys/devices/system/node/online:            2
/proc/sys/kernel/numa_balancing:            0
/sys/devices/system/node/has_cpu:           2
/sys/devices/system/node/has_memory:        2

/sys/devices/system/node/has_normal_memory: 2
/sys/devices/system/node/possible: 0-31

Cc Michal Hocko <[email protected]>
Cc Mel Gorman <[email protected]>
Cc Vlastimil Babka <[email protected]>
Cc "Kirill A. Shutemov" <[email protected]>
Cc Christopher Lameter <[email protected]>
Cc Michael Ellerman <[email protected]>
Cc Andrew Morton <[email protected]>
Cc Linus Torvalds <[email protected]>
Cc Gautham R Shenoy <[email protected]>

Srikar Dronamraju (3):
powerpc/numa: Set numa_node for all possible cpus powerpc/numa: Prefer node id queried from vphn mm/page_alloc: Keep memoryless cpuless node 0 offline

 arch/powerpc/mm/numa.c | 35 ++++++++++++++++++++++++-----------
 mm/page_alloc.c        |  4 +++-
 2 files changed, 27 insertions(+), 12 deletions(-)
Signature · 1 line
2.20.1
#1 lore raw

PATCH v3 1/3 powerpc/numa: Set numa_node for all possible cpus

A Powerpc system with multiple possible nodes and with CONFIG_NUMA enabled always used to have a node 0, even if node 0 does not any cpus or memory attached to it. As per PAPR, node affinity of a cpu is only available once its present / online. For all cpus that are possible but not present, cpu_to_node() would point to node 0.

To ensure a cpuless, memoryless dummy node is not online, powerpc need to make sure all possible but not present cpu_to_node are set to a proper node.

Cc Michal Hocko <[email protected]>
Cc Mel Gorman <[email protected]>
Cc Vlastimil Babka <[email protected]>
Cc "Kirill A. Shutemov" <[email protected]>
Cc Christopher Lameter <[email protected]>
Cc Michael Ellerman <[email protected]>
Cc Andrew Morton <[email protected]>
Cc Linus Torvalds <[email protected]>
Cc Gautham R Shenoy <[email protected]>

---
Changelog v1:->v2:
- Rebased to v5.7-rc3

 arch/powerpc/mm/numa.c | 16 ++++++++++++++--
 1 file changed, 14 insertions(+), 2 deletions(-)
arch/powerpc/mm/numa.c +14 -2
index 9fcf2d195830..b3615b7fdbdf 100644
@@ -931,8 +931,20 @@ void __init mem_topology_setup(void)
 
 	reset_numa_cpu_lookup_table();
 
-	for_each_present_cpu(cpu)
-		numa_setup_cpu(cpu);
+	for_each_possible_cpu(cpu) {
+		/*
+		 * Powerpc with CONFIG_NUMA always used to have a node 0,
+		 * even if it was memoryless or cpuless. For all cpus that
+		 * are possible but not present, cpu_to_node() would point
+		 * to node 0. To remove a cpuless, memoryless dummy node,
+		 * powerpc need to make sure all possible but not present
+		 * cpu_to_node are set to a proper node.
+		 */
+		if (cpu_present(cpu))
+			numa_setup_cpu(cpu);
+		else
+			set_cpu_numa_node(cpu, first_online_node);
+	}
 }
 
 void __init initmem_init(void)
Signature · 1 line
2.20.1
#2 lore raw

Re: [PATCH v3 1/3] powerpc/numa: Set numa_node for all possible cpus

Quoted 7 lines · Srikar Dronamraju
arch/powerpc/mm/numa.c +5 -2
-	for_each_present_cpu(cpu)
-		numa_setup_cpu(cpu);
+	for_each_possible_cpu(cpu) {
+		/*
+		 * Powerpc with CONFIG_NUMA always used to have a node 0,
+		 * even if it was memoryless or cpuless. For all cpus that
+		 * are possible but not present, cpu_to_node() would point

arch/powerpc/mm/numa.c void __init mem_topology_setup(void) · line 948

Quoted 10 lines
arch/powerpc/mm/numa.c +9 -0
+		 * to node 0. To remove a cpuless, memoryless dummy node,
+		 * powerpc need to make sure all possible but not present
+		 * cpu_to_node are set to a proper node.
+		 */
+		if (cpu_present(cpu))
+			numa_setup_cpu(cpu);
+		else
+			set_cpu_numa_node(cpu, first_online_node);
+	}
 }

Can this be folded into numa_setup_cpu?

This looks more like numa_setup_cpu needs to change?

#3 lore raw

Re: [PATCH v3 1/3] powerpc/numa: Set numa_node for all possible cpus

Quoted 23 lines · Christopher Lameter
Quoted 7 lines · Srikar Dronamraju
arch/powerpc/mm/numa.c +5 -2
-	for_each_present_cpu(cpu)
-		numa_setup_cpu(cpu);
+	for_each_possible_cpu(cpu) {
+		/*
+		 * Powerpc with CONFIG_NUMA always used to have a node 0,
+		 * even if it was memoryless or cpuless. For all cpus that
+		 * are possible but not present, cpu_to_node() would point

arch/powerpc/mm/numa.c void __init mem_topology_setup(void) · line 948

Quoted 10 lines
arch/powerpc/mm/numa.c +9 -0
+		 * to node 0. To remove a cpuless, memoryless dummy node,
+		 * powerpc need to make sure all possible but not present
+		 * cpu_to_node are set to a proper node.
+		 */
+		if (cpu_present(cpu))
+			numa_setup_cpu(cpu);
+		else
+			set_cpu_numa_node(cpu, first_online_node);
+	}
 }

Can this be folded into numa_setup_cpu?

Quoted 2 lines

This looks more like numa_setup_cpu needs to change?

We can fold this into numa_setup_cpu().

However till now we were sure that numa_setup_cpu() would be called only for a present cpu. That assumption will change.
+ (non-consequential) an additional check everytime cpu is hotplugged in.

If Michael Ellerman is okay with the change, I can fold it in.

Signature · 2 lines
Thanks and Regards
Srikar Dronamraju
#4 lore raw

Re: [PATCH v3 1/3] powerpc/numa: Set numa_node for all possible cpus

Quoted 26 lines · Srikar Dronamraju
Quoted 22 lines · Christopher Lameter
Quoted 7 lines · Srikar Dronamraju
arch/powerpc/mm/numa.c +5 -2
-	for_each_present_cpu(cpu)
-		numa_setup_cpu(cpu);
+	for_each_possible_cpu(cpu) {
+		/*
+		 * Powerpc with CONFIG_NUMA always used to have a node 0,
+		 * even if it was memoryless or cpuless. For all cpus that
+		 * are possible but not present, cpu_to_node() would point

arch/powerpc/mm/numa.c void __init mem_topology_setup(void) · line 948

Quoted 10 lines
arch/powerpc/mm/numa.c +9 -0
+		 * to node 0. To remove a cpuless, memoryless dummy node,
+		 * powerpc need to make sure all possible but not present
+		 * cpu_to_node are set to a proper node.
+		 */
+		if (cpu_present(cpu))
+			numa_setup_cpu(cpu);
+		else
+			set_cpu_numa_node(cpu, first_online_node);
+	}
 }

Can this be folded into numa_setup_cpu?

Quoted 1 line

This looks more like numa_setup_cpu needs to change?

Quoted 7 lines

We can fold this into numa_setup_cpu().

However till now we were sure that numa_setup_cpu() would be called only for a present cpu. That assumption will change.
+ (non-consequential) an additional check everytime cpu is hotplugged in.

If Michael Ellerman is okay with the change, I can fold it in.

Yes I agree it would be better in numa_setup_cpu().

cheers

#5 lore raw

PATCH v3 2/3 powerpc/numa: Prefer node id queried from vphn

In reply to Srikar Dronamraju

Node id queried from the static device tree may not
be correct. For example: it may always show 0 on a shared processor. Hence prefer the node id queried from vphn and fallback on the device tree based node id if vphn query fails.

Cc Michal Hocko <[email protected]>
Cc Mel Gorman <[email protected]>
Cc Vlastimil Babka <[email protected]>
Cc "Kirill A. Shutemov" <[email protected]>
Cc Christopher Lameter <[email protected]>
Cc Michael Ellerman <[email protected]>
Cc Andrew Morton <[email protected]>
Cc Linus Torvalds <[email protected]>
Cc Gautham R Shenoy <[email protected]>

---
Changelog v2:->v3:
- Resolved comments from Gautham.
Link v2: https://lore.kernel.org/linuxppc-dev/[email protected]/t/#u

Changelog v1:->v2:
- Rebased to v5.7-rc3

 arch/powerpc/mm/numa.c | 16 ++++++++--------
 1 file changed, 8 insertions(+), 8 deletions(-)
arch/powerpc/mm/numa.c +10 -9
index b3615b7fdbdf..79c74a2753c1 100644
@@ -719,21 +719,22 @@ static int __init parse_numa_properties(void)
 	 */
 	for_each_present_cpu(i) {
 		struct device_node *cpu;
-		int nid;
-
-		cpu = of_get_cpu_node(i, NULL);
-		BUG_ON(!cpu);
-		nid = of_node_to_nid_single(cpu);
-		of_node_put(cpu);
+		int nid = vphn_get_nid(i);
 
 		/*
 		 * Don't fall back to default_nid yet -- we will plug
 		 * cpus into nodes once the memory scan has discovered
 		 * the topology.
 		 */
-		if (nid < 0)
-			continue;
-		node_set_online(nid);
+		if (nid == NUMA_NO_NODE) {
+			cpu = of_get_cpu_node(i, NULL);
+			BUG_ON(!cpu);
+			nid = of_node_to_nid_single(cpu);
+			of_node_put(cpu);
+		}
+
+		if (likely(nid > 0))
+			node_set_online(nid);
 	}
 
 	get_n_mem_cells(&n_mem_addr_cells, &n_mem_size_cells);
Signature · 1 line
2.20.1
#6 lore raw

PATCH v3 3/3 mm/page_alloc: Keep memoryless cpuless node 0 offline

In reply to Srikar Dronamraju

Currently Linux kernel with CONFIG_NUMA on a system with multiple possible nodes, marks node 0 as online at boot. However in practice, there are systems which have node 0 as memoryless and cpuless.

This can cause numa_balancing to be enabled on systems with only one node with memory and CPUs. The existence of this dummy node which is cpuless and memoryless node can confuse users/scripts looking at output of lscpu / numactl.

By marking, N_ONLINE as NODE_MASK_NONE, lets stop assuming that Node 0 is always online.

v5.7-rc3
available: 2 nodes (0,2)
node 0 cpus:
node 0 size: 0 MB
node 0 free: 0 MB
node 2 cpus: 0 1 2 3 4 5 6 7
node 2 size: 32625 MB
node 2 free: 31490 MB
node distances:
node 0 2

   0:  10  20
   2:  20  10

proc and sys files
------------------

 /sys/devices/system/node/online:            0,2
 /proc/sys/kernel/numa_balancing:            1
 /sys/devices/system/node/has_cpu:           2
 /sys/devices/system/node/has_memory:        2

/sys/devices/system/node/has_normal_memory: 2
/sys/devices/system/node/possible: 0-31

v5.7-rc3 + patch
------------------
available: 1 nodes (2)
node 2 cpus: 0 1 2 3 4 5 6 7
node 2 size: 32625 MB
node 2 free: 31487 MB
node distances:
node 2
2: 10

proc and sys files
------------------

/sys/devices/system/node/online:            2
/proc/sys/kernel/numa_balancing:            0
/sys/devices/system/node/has_cpu:           2
/sys/devices/system/node/has_memory:        2

/sys/devices/system/node/has_normal_memory: 2
/sys/devices/system/node/possible: 0-31

Note: On Powerpc, cpu_to_node of possible but not present cpus would previously return 0. Hence this commit depends on commit ("powerpc/numa: Set numa_node for all possible cpus") and commit ("powerpc/numa: Prefer node id queried from vphn"). Without the 2 commits, Powerpc system might crash.

Cc Michal Hocko <[email protected]>
Cc Mel Gorman <[email protected]>
Cc Vlastimil Babka <[email protected]>
Cc "Kirill A. Shutemov" <[email protected]>
Cc Christopher Lameter <[email protected]>
Cc Michael Ellerman <[email protected]>
Cc Andrew Morton <[email protected]>
Cc Linus Torvalds <[email protected]>
Cc Gautham R Shenoy <[email protected]>

---
Changelog v1:->v2:
- Rebased to v5.7-rc3
Link v2: https://lore.kernel.org/linuxppc-dev/[email protected]/t/#u

 mm/page_alloc.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
mm/page_alloc.c +3 -1
index 69827d4fa052..03b89592af04 100644
@@ -116,8 +116,10 @@ EXPORT_SYMBOL(latent_entropy);
  */
 nodemask_t node_states[NR_NODE_STATES] __read_mostly = {
 	[N_POSSIBLE] = NODE_MASK_ALL,
+#ifdef CONFIG_NUMA
+	[N_ONLINE] = NODE_MASK_NONE,
+#else
 	[N_ONLINE] = { { [0] = 1UL } },
-#ifndef CONFIG_NUMA
 	[N_NORMAL_MEMORY] = { { [0] = 1UL } },
 #ifdef CONFIG_HIGHMEM
 	[N_HIGH_MEMORY] = { { [0] = 1UL } },
Signature · 1 line
2.20.1
#7 lore raw

Re: [PATCH v3 3/3] mm/page_alloc: Keep memoryless cpuless node 0 offline

mm/page_alloc.c EXPORT_SYMBOL(latent_entropy); · line 120

Quoted 8 lines · Srikar Dronamraju
mm/page_alloc.c +2 -0
@@ -116,8 +116,10 @@ EXPORT_SYMBOL(latent_entropy);
  */
 nodemask_t node_states[NR_NODE_STATES] __read_mostly = {
 	[N_POSSIBLE] = NODE_MASK_ALL,
+#ifdef CONFIG_NUMA
+	[N_ONLINE] = NODE_MASK_NONE,

Hmmm.... I would have expected that you would have added something early in boot that would mark the current node (whatever is is) online instead?

#8 lore raw

Re: [PATCH v3 3/3] mm/page_alloc: Keep memoryless cpuless node 0 offline

Quoted 13 lines · Christopher Lameter

mm/page_alloc.c EXPORT_SYMBOL(latent_entropy); · line 120

Quoted 8 lines · Srikar Dronamraju
mm/page_alloc.c +2 -0
@@ -116,8 +116,10 @@ EXPORT_SYMBOL(latent_entropy);
  */
 nodemask_t node_states[NR_NODE_STATES] __read_mostly = {
 	[N_POSSIBLE] = NODE_MASK_ALL,
+#ifdef CONFIG_NUMA
+	[N_ONLINE] = NODE_MASK_NONE,

Hmmm.... I would have expected that you would have added something early in boot that would mark the current node (whatever is is) online instead?

Do correct me, but these are structure initialization in page_alloc.c Wouldn't these happen much before the numa initialization happens? I think we are already marking nodes as online as soon as we detect the nodes.

Signature · 2 lines
Thanks and Regards
Srikar Dronamraju