Numa: Use Generic Per-cpu Variables for numa_*_id()

Rework the generic version of the numa_node_id() function to use the new generic percpu variable infrastructure.

Guard the new implementation with a new config option:

CONFIG_USE_PERCPU_NUMA_NODE_ID.

Archs which support this new implemention will default this option to 'y' when NUMA is configured. This config option could be removed if/when all archs switch over to the generic percpu implementation of numa_node_id(). Arch support involves:

1) converting any existing per cpu variable implementations to use this implementation. x86_64 is an instance of such an arch.
2) archs that don't use a per cpu variable for numa_node_id() will need to initialize the new per cpu variable "numa_node" as cpus are brought on-line. ia64 is an example.
3) Defining USE_PERCPU_NUMA_NODE_ID in arch dependent Kconfig--e.g., when NUMA is configured. This is required because I have retained the old implementation by default to allow archs to be modified incrementally, as desired.

Subsequent patches will convert x86_64 and ia64 to use this implemenation.

Quoted 22 lines · On Thu, 15 Apr 2010 13:29:56 -0400 Lee Schermerhorn <[email protected]>…

Rework the generic version of the numa_node_id() function to use the new generic percpu variable infrastructure.

Guard the new implementation with a new config option:

CONFIG_USE_PERCPU_NUMA_NODE_ID.

Archs which support this new implemention will default this option to 'y' when NUMA is configured. This config option could be removed if/when all archs switch over to the generic percpu implementation of numa_node_id(). Arch support involves:

1) converting any existing per cpu variable implementations to use this implementation. x86_64 is an instance of such an arch.
2) archs that don't use a per cpu variable for numa_node_id() will need to initialize the new per cpu variable "numa_node" as cpus are brought on-line. ia64 is an example.
3) Defining USE_PERCPU_NUMA_NODE_ID in arch dependent Kconfig--e.g., when NUMA is configured. This is required because I have retained the old implementation by default to allow archs to be modified incrementally, as desired.

Quoted 2 lines

Subsequent patches will convert x86_64 and ia64 to use this implemenation.

So which arches _aren't_ converted? powerpc, sparc and alpha?

Is there sufficient info here for the maintainers to be able to perform the conversion with minimal head-scratching?

Against: 2.6.34-rc3-mmotm-100405-1609

Rework the generic version of the numa_node_id() function to use the new generic percpu variable infrastructure.

Guard the new implementation with a new config option:

CONFIG_USE_PERCPU_NUMA_NODE_ID.

Archs which support this new implemention will default this option to 'y' when NUMA is configured. This config option could be removed if/when all archs switch over to the generic percpu implementation of numa_node_id(). Arch support involves:

1) converting any existing per cpu variable implementations to use this implementation. x86_64 is an instance of such an arch.
2) archs that don't use a per cpu variable for numa_node_id() will need to initialize the new per cpu variable "numa_node" as cpus are brought on-line. ia64 is an example.
3) Defining USE_PERCPU_NUMA_NODE_ID in arch dependent Kconfig--e.g., when NUMA is configured. This is required because I have retained the old implementation by default to allow archs to be modified incrementally, as desired.

Subsequent patches will convert x86_64 and ia64 to use this implemenation.

Signed-off-by Lee Schermerhorn <[email protected]>

---

V0:

#  From [email protected] Wed Nov  4 10:36:12 2009
#  Date: Wed, 4 Nov 2009 12:35:14 -0500 (EST)
#  From: Christoph Lameter <[email protected]>
#  To: Lee Schermerhorn <[email protected]>
#  Subject: Re: [PATCH/RFC] slab:  handle memoryless nodes efficiently

#
# I have a very early form of a draft of a patch here that genericizes # numa_node_id(). Uses the new generic this_cpu_xxx stuff. #
# Not complete.

V1:
+ split out x86 specific changes to subsequent patch
+ split out "numa_mem_id()" and related changes to separate patch
+ moved generic definitions of __this_cpu_xxx from linux/percpu.h to asm-generic/percpu.h where asm/percpu.h and other asm hdrs can use them.
+ export new percpu symbol 'numa_node' in mm/percpu.h
+ include <asm/percpu.h> in <linux/topology.h> for use by new

    numa_node_id().

V2:
+ add back the #ifndef/#endif guard around numa_node_id() so that archs can override generic definition
+ add generic stub for set_numa_node()
+ use generic percpu numa_node_id() only if enabled by CONFIG_USE_PERCPU_NUMA_NODE_ID
to allow incremental per arch support. This option could be removed when/if all archs that support NUMA support this option.

V3:
+ separated the rework of linux/percpu.h into another [preceding] patch.
+ moved definition of the numa_node percpu variable from mm/percpu.c to mm/page-alloc.c
+ moved premature definition of cpu_to_mem() to later patch.

V4:
+ topology.h: include <linux/percpu.h> rather than <linux/percpu-defs.h> Requires Tejun Heo's percpu.h/slab.h cleanup series

 include/linux/topology.h |   33 ++++++++++++++++++++++++++++-----
 mm/page_alloc.c          |    5 +++++
 2 files changed, 33 insertions(+), 5 deletions(-)

Index: linux-2.6.34-rc3-mmotm-100405-1609/mm/page_alloc.c
===================================================================

mm/page_alloc.c +5 -0

@@ -56,6 +56,11 @@
 #include <asm/div64.h>
 #include "internal.h"
 
+#ifdef CONFIG_USE_PERCPU_NUMA_NODE_ID
+DEFINE_PER_CPU(int, numa_node);
+EXPORT_PER_CPU_SYMBOL(numa_node);
+#endif
+
 /*
  * Array of node states.
  */

Index: linux-2.6.34-rc3-mmotm-100405-1609/include/linux/topology.h
===================================================================

include/linux/topology.h +28 -6

@@ -31,6 +31,7 @@
 #include <linux/bitops.h>
 #include <linux/mmzone.h>
 #include <linux/smp.h>
+#include <linux/percpu.h>
 #include <asm/topology.h>
 
 #ifndef node_has_online_mem

@@ -203,8 +204,35 @@ int arch_update_cpu_topology(void);
 #ifndef SD_NODE_INIT
 #error Please define an appropriate SD_NODE_INIT in include/asm/topology.h!!!
 #endif
+
 #endif /* CONFIG_NUMA */
 
+#ifdef CONFIG_USE_PERCPU_NUMA_NODE_ID
+DECLARE_PER_CPU(int, numa_node);
+
+#ifndef numa_node_id
+/* Returns the number of the current Node. */
+#define numa_node_id()		__this_cpu_read(numa_node)
+#endif
+
+#ifndef cpu_to_node
+#define cpu_to_node(__cpu)	per_cpu(numa_node, (__cpu))
+#endif
+
+#ifndef set_numa_node
+#define set_numa_node(__node) percpu_write(numa_node, __node)
+#endif
+
+#else	/* !CONFIG_USE_PERCPU_NUMA_NODE_ID */
+
+/* Returns the number of the current Node. */
+#ifndef numa_node_id
+#define numa_node_id()		(cpu_to_node(raw_smp_processor_id()))
+
+#endif
+
+#endif	/* [!]CONFIG_USE_PERCPU_NUMA_NODE_ID */
+
 #ifndef topology_physical_package_id
 #define topology_physical_package_id(cpu)	((void)(cpu), -1)
 #endif

@@ -218,9 +246,4 @@ int arch_update_cpu_topology(void);
 #define topology_core_cpumask(cpu)		cpumask_of(cpu)
 #endif
 
-/* Returns the number of the current Node. */
-#ifndef numa_node_id
-#define numa_node_id()		(cpu_to_node(raw_smp_processor_id()))
-#endif
-
 #endif /* _LINUX_TOPOLOGY_H */
 
--

To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

x86 arch specific changes to use generic numa_node_id() based on generic percpu variable infrastructure. Back out x86's custom version of numa_node_id()

Signed-off-by Lee Schermerhorn <[email protected]>

[Christoph's signoff here?]

Maybe provide a generic function to set the node for cpu X?

Quoted 1 line · Christoph Lameter

Maybe provide a generic function to set the node for cpu X?

Yeap, seconded. Also, why not use numa_node_id() in common.c::cpu_init()?

Tejun: do you mean:

#ifdef CONFIG_NUMA
        if (cpu != 0 && percpu_read(numa_node) == 0 &&
........................^ here?
            early_cpu_to_node(cpu) != NUMA_NO_NODE)
                set_numa_node(early_cpu_to_node(cpu));

#endif

Looks like 'numa_node_id()' would work there.

But, I wonder what the "cpu != 0 && percpu_read(numa_node) == 0" is trying to do?

Just trying to grok the intent. Maybe someone will chime in.

Hello,

Quoted 10 lines · Lee Schermerhorn

Tejun: do you mean:

#ifdef CONFIG_NUMA
        if (cpu != 0 && percpu_read(numa_node) == 0 &&
........................^ here?
            early_cpu_to_node(cpu) != NUMA_NO_NODE)
                set_numa_node(early_cpu_to_node(cpu));

#endif

Looks like 'numa_node_id()' would work there.

Yeah, it just looked weird to use raw variable when an access wrapper is there.

Quoted 2 lines

But, I wonder what the "cpu != 0 && percpu_read(numa_node) == 0" is trying to do?

That I have don't have any clue about. :-)

Against: 2.6.34-rc3-mmotm-100405-1609

ia64: Use generic percpu implementation of numa_node_id()
+ intialize per cpu 'numa_node'
+ remove ia64 cpu_to_node() macro; use generic
+ define CONFIG_USE_PERCPU_NUMA_NODE_ID when NUMA configured

Signed-off-by Lee Schermerhorn <[email protected]>

Reviewed-by Christoph Lameter <[email protected]>

---

New in V2

V3, V4: no change

 arch/ia64/Kconfig                |    4 ++++
 arch/ia64/include/asm/topology.h |    5 -----
 arch/ia64/kernel/smpboot.c       |    6 ++++++
 3 files changed, 10 insertions(+), 5 deletions(-)

Index: linux-2.6.34-rc3-mmotm-100405-1609/arch/ia64/kernel/smpboot.c
===================================================================

arch/ia64/kernel/smpboot.c +6 -0

@@ -390,6 +390,11 @@ smp_callin (void)
 
 	fix_b0_for_bsp();
 
+	/*
+	 * numa_node_id() works after this.
+	 */
+	set_numa_node(cpu_to_node_map[cpuid]);
+
 	ipi_call_lock_irq();
 	spin_lock(&vector_lock);
 	/* Setup the per cpu irq handling data structures */

@@ -632,6 +637,7 @@ void __devinit smp_prepare_boot_cpu(void
 {
 	cpu_set(smp_processor_id(), cpu_online_map);
 	cpu_set(smp_processor_id(), cpu_callin_map);
+	set_numa_node(cpu_to_node_map[smp_processor_id()]);
 	per_cpu(cpu_state, smp_processor_id()) = CPU_ONLINE;
 	paravirt_post_smp_prepare_boot_cpu();
 }

Index: linux-2.6.34-rc3-mmotm-100405-1609/arch/ia64/include/asm/topology.h
===================================================================

arch/ia64/include/asm/topology.h +0 -5

@@ -26,11 +26,6 @@
 #define RECLAIM_DISTANCE 15
 
 /*
- * Returns the number of the node containing CPU 'cpu'
- */
-#define cpu_to_node(cpu) (int)(cpu_to_node_map[cpu])
-
-/*
  * Returns a bitmask of CPUs on Node 'node'.
  */
 #define cpumask_of_node(node) ((node) == -1 ?				\

Index: linux-2.6.34-rc3-mmotm-100405-1609/arch/ia64/Kconfig
===================================================================

arch/ia64/Kconfig +4 -1

@@ -497,6 +497,10 @@ config HAVE_ARCH_NODEDATA_EXTENSION
 	def_bool y
 	depends on NUMA
 
+config USE_PERCPU_NUMA_NODE_ID
+	def_bool y
+	depends on NUMA
+
 config ARCH_PROC_KCORE_TEXT
 	def_bool y
 	depends on PROC_KCORE
 
--

To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

+static inline void set_numa_mem(int node) {}

Against: 2.6.34-rc3-mmotm-100405-1609

IA64: Support memoryless nodes

Enable 'HAVE_MEMORYLESS_NODES' by default when NUMA configured

on ia64. Initialize percpu 'numa_mem' variable when starting secondary cpus. Generic initialization will handle the boot cpu.

Nothing uses 'numa_mem_id()' yet. Subsequent patch with modify

slab to use this.

I have a note here that this patch "breaks slab.c". But I don't recall what the problem was and I don't see a fix against this patch in your recently-sent fixup series?

Quoted 3 lines · Andrew Morton

I have a note here that this patch "breaks slab.c". But I don't recall what the problem was and I don't see a fix against this patch in your recently-sent fixup series?

Is that Valdis Kletnieks' issue? That was an i386 build. Happened because the earlier patches didn't properly default numa_mem_id() to numa_node_id() for the i386 build. The rework to those patches has
fixed that. I have successfully built mmotm with the rework patches
for i386+!NUMA. Valdis tested the series and confirmed that it fixed the problem.

Quoted 11 lines · Lee Schermerhorn

Quoted 3 lines · Andrew Morton

I have a note here that this patch "breaks slab.c". But I don't recall what the problem was and I don't see a fix against this patch in your recently-sent fixup series?

Is that Valdis Kletnieks' issue? That was an i386 build. Happened because the earlier patches didn't properly default numa_mem_id() to numa_node_id() for the i386 build. The rework to those patches has
fixed that. I have successfully built mmotm with the rework patches
for i386+!NUMA. Valdis tested the series and confirmed that it fixed the problem.

I thought the problem was common to both i386 and X86_64 non-NUMA (which is where I hit the problem). In any case, builds OK for me now.

Against: 2.6.34-rc3-mmotm-100405-1609

Kamezawa Hiroyuki requested documentation for the numa_mem_id() and slab related changes. He suggested Documentation/vm/numa for this documentation. Looking at this file, it seems to me to be hopelessly out of date relative to current Linux NUMA support. At the risk of going down a rathole, I have made an attempt to rewrite the doc at a slightly higher level [I think] and provide pointers to other in-tree documents and out-of-tree man pages that cover the details.

Let the games begin.

Signed-off-by Lee Schermerhorn <[email protected]>

---

New in V4.

 Documentation/vm/numa |  184 +++++++++++++++++++++++++++++++++++++++-----------
 1 files changed, 146 insertions(+), 38 deletions(-)

Index: linux-2.6.34-rc3-mmotm-100405-1609/Documentation/vm/numa
===================================================================

Documentation/vm/numa +1 -2

@@ -1,41 +1,149 @@
 Started Nov 1999 by Kanoj Sarcar <[email protected]>
 
-The intent of this file is to have an uptodate, running commentary 
-from different people about NUMA specific code in the Linux vm.
+What is NUMA?
 

Documentation/vm/numa +13 -0

+This question can be answered from a couple of perspectives:  the
+hardware view and the Linux software view.
+
+From the hardware perspective, a NUMA system is a computer platform that
+comprises multiple components or assemblies each of which may contain 0
+or more cpus, local memory, and/or IO buses.  For brevity and to
+disambiguate the hardware view of these physical components/assemblies
+from the software abstraction thereof, we'll call the components/assemblies
+'cells' in this document.
+
+Each of the 'cells' may be viewed as an SMP [symmetric multi-processor] subset
+of the system--although some components necessary for a stand-alone SMP system
+may not be populated on any given cell.   The cells of the NUMA system are

Documentation/vm/numa +26 -0

+Memory access time and effective memory bandwidth varies depending on how far
+away the cell containing the cpu or io bus making the memory access is from the
+cell containing the target memory.  For example, access to memory by cpus
+attached to the same cell will experience faster access times and higher
+bandwidths than accesses to memory on other, remote cells.  NUMA platforms
+can have cells at multiple remote distances from any given cell.
+
+Platform vendors don't build NUMA systems just to make software developers'
+lives interesting.  Rather, this architecture is a means to provide scalable
+memory bandwidth.  However, to achieve scalable memory bandwidth, system and
+application software must arrange for a large majority of the memory references
+[cache misses] to be to "local" memory--memory on the same cell, if any--or
+to the closest cell with memory.
+
+This leads to the Linux software view of a NUMA system:
+
+Linux divides the system's hardware resources into multiple software
+abstractions called "nodes".  Linux maps the nodes onto the physical cells
+of the hardware platform, abstracting away some of the details for some
+architectures.  As with physical cells, software nodes may contain 0 or more
+cpus, memory and/or IO buses.  And, again, memory access times to memory on
+"closer" nodes [nodes that map to closer cells] will generally experience
+faster access times and higher effective bandwidth than accesses to more
+remote cells.
+
+For some architectures, such as x86, Linux will "hide" any node representing a

+'overflow" or "fallback".

Documentation/vm/numa +16 -0

+
+Because some nodes contain multiple zones containing different types of
+memory, Linux must decide whether to order the zonelists such that allocations
+fall back to the same zone type on a different node, or to a different zone
+type on the same node.  This is an important consideration because some zones,
+such as DMA or DMA32, represent relatively scarce resources.  Linux chooses
+a default zonelist order based on the sizes of the various zone types relative
+to the total memory of the node and the total memory of the system.  The
+default zonelist order may be overridden using the numa_zonelist_order kernel
+boot parameter or sysctl.  [See Documentation/kernel-parameters.txt and
+Documentation/sysctl/vm.txt]
+
+By default, Linux will attempt to satisfy memory allocation requests from the
+node to which the cpu that executes the request is assigned.  Specifically,
+Linux will attempt to allocate from the first node in the appropriate zonelist
+for the node where the request originates.  This is called "local allocation."

Documentation/vm/numa +21 -0

+to improve NUMA locality using various cpu affinity command line interfaces,
+such as taskset(1) and numactl(1), and program interfaces such as
+sched_setaffinity(2).  Further, one can modify the kernel's default local
+allocation behavior using Linux NUMA memory policy.
+[See Documentation/vm/numa_memory_policy.]
+
+System administrators can restrict the cpus and nodes' memories that a non-
+privileged user can specify in the scheduling or NUMA commands and functions
+using control groups and cpusets.  [See Documentation/cgroups/cpusets.txt]
+
+On architectures that do not hide memoryless nodes, Linux will include only
+zones [nodes] with memory in the zonelists.  This means that for a memoryless
+node the "local memory node"--the node of the first zone in cpu's node's
+zonelist--will not be the node itself.  Rather, it will be the node that the
+kernel selected as the nearest node with memory when it built the zonelists.
+So, default, local allocations will succeed with the kernel supplying the
+closest available memory.  This is a consequence of the same mechanism that
+allows such allocations to fallback to other nearby nodes when a node that
+does contain memory overflows.
+
+Some kernel allocations do not want or cannot tolerate this allocation fallback

Against: 2.6.34-rc3-mmotm-100405-1609

Kamezawa Hiroyuki requested documentation for the numa_mem_id() and slab related changes. He suggested Documentation/vm/numa for this documentation. Looking at this file, it seems to me to be hopelessly out of date relative to current Linux NUMA support. At the risk of going down a rathole, I have made an attempt to rewrite the doc at a slightly higher level [I think] and provide pointers to other in-tree documents and out-of-tree man pages that cover the details.

Let the games begin.

Signed-off-by Lee Schermerhorn <[email protected]>

Use Generic Per cpu infrastructure for numa_*_id() V4

Series Against: 2.6.34-rc3-mmotm-100405-1609

Quoted 3 lines · Lee Schermerhorn

Use Generic Per cpu infrastructure for numa_*_id() V4

Series Against: 2.6.34-rc3-mmotm-100405-1609

Other than the minor nitpicks, the patchset looks great to me. Through which tree should this be routed? If no one else is gonna take it, I can route it through percpu after patchset refresh.