powerpc: numa: enable USE_PERCPU_NUMA_NODE_ID

PATCH New Nishanth Aravamudan 3 participants 8 messages May 16, 2014 – May 28, 2014

Subsystems (2)

Memory management PowerPC

lore

Share & tools

lore

Nishanth Aravamudan

author

2014-05-16 23:40 UTC #0 lore raw

PATCH 1/1 powerpc: numa: enable USE_PERCPU_NUMA_NODE_ID

Based off 3bccd996 for ia64, convert powerpc to use the generic per-CPU topology tracking, specifically:

initialize per cpu numa_node entry in start_secondary remove the powerpc cpu_to_node()
define CONFIG_USE_PERCPU_NUMA_NODE_ID if NUMA

Signed-off-by Nishanth Aravamudan <[email protected]>

arch/powerpc/Kconfig +4 -0

index e099899..9125964 100644
@@ -453,6 +453,10 @@ config NODES_SHIFT
 	default "4"
 	depends on NEED_MULTIPLE_NODES
 
+config USE_PERCPU_NUMA_NODE_ID
+	def_bool y
+	depends on NUMA
+
 config ARCH_SELECT_MEMORY_MODEL
 	def_bool y
 	depends on PPC64

arch/powerpc/include/asm/topology.h +0 ~~-13~~

index c920215..5ecf7ea 100644
@@ -20,19 +20,6 @@ struct device_node;
 
 #include <asm/mmzone.h>
 
-static inline int cpu_to_node(int cpu)
-{
-	int nid;
-
-	nid = numa_cpu_lookup_table[cpu];
-
-	/*
-	 * During early boot, the numa-cpu lookup table might not have been
-	 * setup for all CPUs yet. In such cases, default to node 0.
-	 */
-	return (nid < 0) ? 0 : nid;
-}
-
 #define parent_node(node)	(node)
 
 #define cpumask_of_node(node) ((node) == -1 ?				\

arch/powerpc/kernel/smp.c +5 -0

index e2a4232..b95be24 100644
@@ -750,6 +750,11 @@ void start_secondary(void *unused)
 	}
 	traverse_core_siblings(cpu, true);
 
+	/*
+	 * numa_node_id() works after this.
+	 */
+	set_numa_node(numa_cpu_lookup_table[cpu]);
+
 	smp_wmb();
 	notify_cpu_starting(cpu);
 	set_cpu_online(cpu, true);

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Nishanth Aravamudan

author

2014-05-16 23:42 UTC #1 lore raw

PATCH 2/2 powerpc: numa: enable CONFIG_HAVE_MEMORYLESS_NODES

Based off fd1197f1 for ia64, enable CONFIG_HAVE_MEMORYLESS_NODES if NUMA. Initialize the local memory node in start_secondary.

With this commit and the preceding to enable CONFIG_USER_PERCPU_NUMA_NODE_ID, which is a prerequisite, in a PowerKVM guest with the following topology:

numactl --hardware
available: 3 nodes (0-2)
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
node 0 size: 1998 MB
node 0 free: 521 MB
node 1 cpus: 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199
node 1 size: 0 MB
node 1 free: 0 MB
node 2 cpus:
node 2 size: 2039 MB
node 2 free: 1739 MB
node distances:
node 0 1 2

  0:  10  40  40
  1:  40  10  40
  2:  40  40  10

the unreclaimable slab is reduced by close to 130M:

Before:

        Slab:             418176 kB
        SReclaimable:      26624 kB
        SUnreclaim:       391552 kB

After:

        Slab:             298944 kB
        SReclaimable:      31744 kB
        SUnreclaim:       267200 kB

Signed-off-by Nishanth Aravamudan <[email protected]>

arch/powerpc/Kconfig +4 -0

index 9125964..bd6dd6e 100644
@@ -457,6 +457,10 @@ config USE_PERCPU_NUMA_NODE_ID
 	def_bool y
 	depends on NUMA
 
+config HAVE_MEMORYLESS_NODES
+	def_bool y
+	depends on NUMA
+
 config ARCH_SELECT_MEMORY_MODEL
 	def_bool y
 	depends on PPC64

arch/powerpc/kernel/smp.c +1 -0

index b95be24..ebd7b9d 100644
@@ -754,6 +754,7 @@ void start_secondary(void *unused)
 	 * numa_node_id() works after this.
 	 */
 	set_numa_node(numa_cpu_lookup_table[cpu]);
+	set_numa_mem(local_memory_node(numa_cpu_lookup_table[cpu]));
 
 	smp_wmb();
 	notify_cpu_starting(cpu);

Nishanth Aravamudan

author

2014-05-19 18:14 UTC #2 lore raw

RFC PATCH v2 1/2 powerpc: numa: enable USE_PERCPU_NUMA_NODE_ID

In reply to Nishanth Aravamudan

Hi Andrew,

I found one issue with my patch, fixed below...

Quoted 8 lines · Nishanth Aravamudan

Based off 3bccd996 for ia64, convert powerpc to use the generic per-CPU topology tracking, specifically:
initialize per cpu numa_node entry in start_secondary remove the powerpc cpu_to_node()
define CONFIG_USE_PERCPU_NUMA_NODE_ID if NUMA
Signed-off-by Nishanth Aravamudan <[email protected]>

<snip>

arch/powerpc/kernel/smp.c void start_secondary(void *unused) · line 757

Quoted 13 lines

arch/powerpc/kernel/smp.c +5 -0

index e2a4232..b95be24 100644
@@ -750,6 +750,11 @@ void start_secondary(void *unused)
 	}
 	traverse_core_siblings(cpu, true);
 
+	/*
+	 * numa_node_id() works after this.
+	 */
+	set_numa_node(numa_cpu_lookup_table[cpu]);
+

Similar change is needed for the boot CPU. Update patch:

powerpc: numa: enable USE_PERCPU_NUMA_NODE_ID

Based off 3bccd996 for ia64, convert powerpc to use the generic per-CPU topology tracking, specifically:

initialize per cpu numa_node entry in start_secondary remove the powerpc cpu_to_node()
define CONFIG_USE_PERCPU_NUMA_NODE_ID if NUMA

Signed-off-by Nishanth Aravamudan <[email protected]>

arch/powerpc/Kconfig +4 -0

index e099899..9125964 100644
@@ -453,6 +453,10 @@ config NODES_SHIFT
 	default "4"
 	depends on NEED_MULTIPLE_NODES
 
+config USE_PERCPU_NUMA_NODE_ID
+	def_bool y
+	depends on NUMA
+
 config ARCH_SELECT_MEMORY_MODEL
 	def_bool y
 	depends on PPC64

arch/powerpc/include/asm/topology.h +0 ~~-13~~

index c920215..5ecf7ea 100644
@@ -20,19 +20,6 @@ struct device_node;
 
 #include <asm/mmzone.h>
 
-static inline int cpu_to_node(int cpu)
-{
-	int nid;
-
-	nid = numa_cpu_lookup_table[cpu];
-
-	/*
-	 * During early boot, the numa-cpu lookup table might not have been
-	 * setup for all CPUs yet. In such cases, default to node 0.
-	 */
-	return (nid < 0) ? 0 : nid;
-}
-
 #define parent_node(node)	(node)
 
 #define cpumask_of_node(node) ((node) == -1 ?				\

arch/powerpc/kernel/smp.c +6 -0

index e2a4232..d7252ad 100644
@@ -390,6 +390,7 @@ void smp_prepare_boot_cpu(void)
 #ifdef CONFIG_PPC64
 	paca[boot_cpuid].__current = current;
 #endif
+	set_numa_node(numa_cpu_lookup_table[boot_cpuid]);
 	current_set[boot_cpuid] = task_thread_info(current);
 }

@@ -750,6 +751,11 @@ void start_secondary(void *unused)
 	}
 	traverse_core_siblings(cpu, true);
 
+	/*
+	 * numa_node_id() works after this.
+	 */
+	set_numa_node(numa_cpu_lookup_table[cpu]);
+
 	smp_wmb();
 	notify_cpu_starting(cpu);
 	set_cpu_online(cpu, true);

Nishanth Aravamudan

author

2014-05-27 23:44 UTC #3 lore raw

Quoted 35 lines · Nishanth Aravamudan

Hi Andrew,
I found one issue with my patch, fixed below...
Quoted 8 lines · Nishanth Aravamudan
Based off 3bccd996 for ia64, convert powerpc to use the generic per-CPU topology tracking, specifically:
initialize per cpu numa_node entry in start_secondary remove the powerpc cpu_to_node()
define CONFIG_USE_PERCPU_NUMA_NODE_ID if NUMA
Signed-off-by Nishanth Aravamudan <[email protected]>
<snip>
arch/powerpc/kernel/smp.c void start_secondary(void *unused) · line 757
Quoted 13 lines
arch/powerpc/kernel/smp.c +5 -0
index e2a4232..b95be24 100644
@@ -750,6 +750,11 @@ void start_secondary(void *unused)
 	}
 	traverse_core_siblings(cpu, true);
 
+	/*
+	 * numa_node_id() works after this.
+	 */
+	set_numa_node(numa_cpu_lookup_table[cpu]);
+
Similar change is needed for the boot CPU. Update patch:
powerpc: numa: enable USE_PERCPU_NUMA_NODE_ID

Quoted 8 lines

Based off 3bccd996 for ia64, convert powerpc to use the generic per-CPU topology tracking, specifically:
initialize per cpu numa_node entry in start_secondary remove the powerpc cpu_to_node()
define CONFIG_USE_PERCPU_NUMA_NODE_ID if NUMA
Signed-off-by Nishanth Aravamudan <[email protected]>

Ping on this and patch 2/2. Ben, would you be willing to pull these into your -next branch so they'd get some testing?

http://patchwork.ozlabs.org/patch/350368/
http://patchwork.ozlabs.org/patch/349838/

Without any further changes, these two help quite a bit with the slab consumption on CONFIG_SLUB kernels when memoryless nodes are present.

Thanks,
Nish

Benjamin Herrenschmidt

2014-05-27 23:56 UTC #4 lore raw

Quoted 11 lines · Nishanth Aravamudan

Quoted 1 line
Signed-off-by Nishanth Aravamudan <[email protected]>
Ping on this and patch 2/2. Ben, would you be willing to pull these into
your -next branch so they'd get some testing?
http://patchwork.ozlabs.org/patch/350368/
http://patchwork.ozlabs.org/patch/349838/
Without any further changes, these two help quite a bit with the slab consumption on CONFIG_SLUB kernels when memoryless nodes are present.

I don't mind at all :-) I haven't really been following that story so I was waiting for the dust to settle and maybe acks from MM people but if you tell me they are good I'm prepared to trust you.

Cheers,
Ben.

Nishanth Aravamudan

author

2014-05-28 00:10 UTC #5 lore raw

Quoted 10 lines · Benjamin Herrenschmidt

Quoted 9 lines · Nishanth Aravamudan
Quoted 1 line
Signed-off-by Nishanth Aravamudan <[email protected]>
Ping on this and patch 2/2. Ben, would you be willing to pull these into
your -next branch so they'd get some testing?
http://patchwork.ozlabs.org/patch/350368/
http://patchwork.ozlabs.org/patch/349838/

Quoted 6 lines

Quoted 2 lines
Without any further changes, these two help quite a bit with the slab consumption on CONFIG_SLUB kernels when memoryless nodes are present.
I don't mind at all :-) I haven't really been following that story so I was waiting for the dust to settle and maybe acks from MM people but if you tell me they are good I'm prepared to trust you.

The patches themselves are pretty minimal and similar to the ia64 changes (and the affected code seems like it hasn't changed in some time, beyond in the common code). I'd mostly like to get some broad-range build & boot testing.

Also, is NUMA a sufficient symbol to depend, you think? I figure most of the PPC NUMA systems are the pSeries/IBM variety, which is where I've run into memoryless nodes in the first place.

Thanks,
Nish

Andrew Morton

maintainer

2014-05-28 23:36 UTC #6 lore raw

Quoted 18 lines · On Tue, 27 May 2014 17:09:58 -0700 Nishanth Aravamudan…

Quoted 10 lines · Benjamin Herrenschmidt
Quoted 9 lines · Nishanth Aravamudan
Quoted 1 line
Signed-off-by Nishanth Aravamudan <[email protected]>
Ping on this and patch 2/2. Ben, would you be willing to pull these into
your -next branch so they'd get some testing?
http://patchwork.ozlabs.org/patch/350368/
http://patchwork.ozlabs.org/patch/349838/
Quoted 6 lines
Quoted 2 lines
Without any further changes, these two help quite a bit with the slab consumption on CONFIG_SLUB kernels when memoryless nodes are present.
I don't mind at all :-) I haven't really been following that story so I was waiting for the dust to settle and maybe acks from MM people but if you tell me they are good I'm prepared to trust you.

Quoted 8 lines

The patches themselves are pretty minimal and similar to the ia64 changes (and the affected code seems like it hasn't changed in some time, beyond in the common code). I'd mostly like to get some broad-range build & boot testing.
Also, is NUMA a sufficient symbol to depend, you think? I figure most of the PPC NUMA systems are the pSeries/IBM variety, which is where I've run into memoryless nodes in the first place.

It's a powerpc-only patchset so I didn't do anything.

Nits:

- When referring to git commits, use 12 digits of hash and include the name of the patch as well (because patches can have different hashes in different trees). So

Based on 3bccd996276b ("numa: ia64: use generic percpu var numa_node_id() implementation") for ia64.

- It's nice to include a diffstat. If there's a [0/n] patch then that's a great place for it, so people can see the overall impact at a glance.

Nishanth Aravamudan

author

2014-05-28 23:43 UTC #7 lore raw

Quoted 37 lines · Andrew Morton

Quoted 18 lines · On Tue, 27 May 2014 17:09:58 -0700 Nishanth Aravamudan…
Quoted 10 lines · Benjamin Herrenschmidt
Quoted 9 lines · Nishanth Aravamudan
Quoted 1 line
Signed-off-by Nishanth Aravamudan <[email protected]>
Ping on this and patch 2/2. Ben, would you be willing to pull these into
your -next branch so they'd get some testing?
http://patchwork.ozlabs.org/patch/350368/
http://patchwork.ozlabs.org/patch/349838/
Quoted 6 lines
Quoted 2 lines
Without any further changes, these two help quite a bit with the slab consumption on CONFIG_SLUB kernels when memoryless nodes are present.
I don't mind at all :-) I haven't really been following that story so I was waiting for the dust to settle and maybe acks from MM people but if you tell me they are good I'm prepared to trust you.
Quoted 8 lines
The patches themselves are pretty minimal and similar to the ia64 changes (and the affected code seems like it hasn't changed in some time, beyond in the common code). I'd mostly like to get some broad-range build & boot testing.
Also, is NUMA a sufficient symbol to depend, you think? I figure most of the PPC NUMA systems are the pSeries/IBM variety, which is where I've run into memoryless nodes in the first place.
It's a powerpc-only patchset so I didn't do anything.
Nits:
- When referring to git commits, use 12 digits of hash and include the name of the patch as well (because patches can have different hashes in different trees). So

Quoted 6 lines

Based on 3bccd996276b ("numa: ia64: use generic percpu var numa_node_id() implementation") for ia64.
- It's nice to include a diffstat. If there's a [0/n] patch then that's a great place for it, so people can see the overall impact at a glance.

Thanks for the notes, I'll include those in any updated patches.

-Nish