topology for ia64

Hi David,

please find attached a first attempt to implement the topology.h macros/routines for IA64. We need this for the NUMA scheduler setup.

Thanks!
Best regards,
Erich

diff -urNp linux-2.5.39-ia64/arch/ia64/kernel/acpi.c linux-2.5.39-ia64-top/arch/ia64/kernel/acpi.c
@@ -631,6 +631,7 @@ acpi_boot_init (char *cmdline)
 	smp_boot_data.cpu_count = total_cpus;
 
 	smp_build_cpu_map();
+	build_cpu_to_node_map();
 #endif
 	/* Make boot-up look pretty */
 	printk("%d CPUs available, %d CPUs total\n", available_cpus, total_cpus);

diff -urNp linux-2.5.39-ia64/arch/ia64/kernel/smpboot.c linux-2.5.39-ia64-top/arch/ia64/kernel/smpboot.c
@@ -16,6 +16,7 @@
 
 #include <linux/config.h>
 
+#include <linux/acpi.h>
 #include <linux/bootmem.h>
 #include <linux/delay.h>
 #include <linux/init.h>

@@ -427,6 +428,32 @@ smp_build_cpu_map (void)
 	}
 }
 
+char cpu_to_node_map[NR_CPUS] __cacheline_aligned;
+/*
+ * Build cpu to node mapping.
+ */
+void __init
+build_cpu_to_node_map(void)
+{
+	int cpu, i;
+
+	for(cpu=0; cpu<NR_CPUS; cpu++) {
+		if (!(phys_cpu_present_map & (1UL << cpu))) {
+			cpu_to_node_map[cpu] = -1;
+			continue;
+		}
+#ifdef CONFIG_ACPI_NUMA
+		for(i=0; i<NR_CPUS; i++)
+			if (cpu_physical_id(cpu) == node_cpuid[i].phys_id) {
+				cpu_to_node_map[cpu]=node_cpuid[i].nid;
+				break;
+			}
+#else
+		cpu_to_node_map[cpu] = 0;
+#endif
+	}
+}
+
 /*
  * Cycle through the APs sending Wakeup IPIs to boot each.
  */

diff -urNp linux-2.5.39-ia64/include/asm-ia64/smp.h linux-2.5.39-ia64-top/include/asm-ia64/smp.h
@@ -13,6 +13,7 @@
 
 #ifdef CONFIG_SMP
 
+#include <linux/cache.h>
 #include <linux/init.h>
 #include <linux/threads.h>
 #include <linux/kernel.h>

@@ -44,6 +45,7 @@ extern unsigned char smp_int_redirect;
 
 extern volatile int ia64_cpu_to_sapicid[];
 #define cpu_physical_id(i)	ia64_cpu_to_sapicid[i]
+extern char cpu_to_node_map[NR_CPUS] __cacheline_aligned;
 
 extern unsigned long ap_wakeup_vector;
 

Erich> Hi David, please find attached a first attempt to implement Erich> the topology.h macros/routines for IA64. We need this for the Erich> NUMA scheduler setup.

Why does the cpu_to_node_map[] exist for non-NUMA configurations? It seems to me that it would be better to make cpu_to_node_map a macro that uses an array-check for NUMA configurations and a simple test against phys_cpu_present_map() for non-NUMA.

	--david

Attached is a modified patch for implementing the topology stuff on ia64. It's on top of your 2.5.39 tree including acpi_numa and the acpi_numa fix which I've sent you separately.

I dropped the cpu_to_node_map array for the non-NUMA case. The macro __cpu_to_node() returns 0 in this case. In the places where it is used (e.g. in the NUMA scheduler) we either run on a valid CPU or have cpu_online() checks before using it, therefore I also removed the phys_cpu_present_map check when building the cpu to node map.

Hope this can be included now...

Regards,
Erich

diff -urNp 2.5.39-ia64-acpi-dmf/arch/ia64/kernel/acpi.c 2.5.39-ia64-acpi-dmf-top/arch/ia64/kernel/acpi.c
@@ -784,6 +784,9 @@ acpi_boot_init (char *cmdline)
 	smp_boot_data.cpu_count = total_cpus;
 
 	smp_build_cpu_map();
+#ifdef CONFIG_NUMA
+	build_cpu_to_node_map();
+#endif
 #endif
 	/* Make boot-up look pretty */
 	printk("%d CPUs available, %d CPUs total\n", available_cpus, total_cpus);

diff -urNp 2.5.39-ia64-acpi-dmf/arch/ia64/kernel/smpboot.c 2.5.39-ia64-acpi-dmf-top/arch/ia64/kernel/smpboot.c
@@ -16,6 +16,7 @@
 
 #include <linux/config.h>
 
+#include <linux/acpi.h>
 #include <linux/bootmem.h>
 #include <linux/delay.h>
 #include <linux/init.h>

@@ -427,6 +428,32 @@ smp_build_cpu_map (void)
 	}
 }
 
+#ifdef CONFIG_NUMA
+char cpu_to_node_map[NR_CPUS] __cacheline_aligned;
+/*
+ * Build cpu to node mapping.
+ */
+void __init
+build_cpu_to_node_map(void)
+{
+	int cpu, i;
+
+	for(cpu=0; cpu<NR_CPUS; cpu++) {
+		/*
+		 * All Itanium NUMA platforms I know use ACPI, so maybe we
+		 * can drop this ifdef completely.                    [EF] 
+		 */
+#ifdef CONFIG_ACPI_NUMA
+		for(i=0; i<NR_CPUS; i++)
+			if (cpu_physical_id(cpu) == node_cpuid[i].phys_id) {
+				cpu_to_node_map[cpu]=node_cpuid[i].nid;
+				break;
+			}
+#endif
+	}
+}
+#endif
+
 /*
  * Cycle through the APs sending Wakeup IPIs to boot each.
  */

diff -urNp 2.5.39-ia64-acpi-dmf/include/asm-ia64/numa.h 2.5.39-ia64-acpi-dmf-top/include/asm-ia64/numa.h
@@ -22,6 +22,8 @@
 # define NR_MEMBLKS   (NR_NODES * 8)
 #endif
 
+extern char cpu_to_node_map[NR_CPUS] __cacheline_aligned;
+
 /* Stuff below this line could be architecture independent */
 
 extern int num_memblks;		/* total number of memory chunks */

Hi Erich! Apologies for the long response delay... I think our mail server must be a bit lagged. ;)

It looks good to me. As far as this comment:
+/*
+ * Returns the number of the first CPU on Node 'node'.
+ * Slow in the current implementation.
+ * Who needs this?
+ */
+/* #define __node_to_first_cpu(node) pool_cpus[pool_ptr[node]] */ +static inline int __node_to_first_cpu(int node)

No one is using it now. I think that I will probably deprecate this function in the near future as it is pretty useless. Anyone looking for that functionality can just do an __ffs(__node_to_cpu_mask(node)) instead, and hope that there is a reasonably quick implementation of __node_to_cpu_mask.

I agree! Linus or another maintainer, please pick this up. These macros should be implemented intelligently on as many architectures as possible, now that they're beginning to be used in more and more places.

Cheers!

-Matt

I'm using this in the simple NUMA scheduler. This is quite useful for iterating through a specific node's CPUs. Yes, the functionality can be obtained in a different manner, but is less obvious.

Michael Hohnbaum                      503-578-5486
[email protected]                   T/L 775-5486

Hmmm... This is true, but I'd like to keep the topology functions as minimal as possible. Since the functionality is so easily duplicated, and in almost every arch, is implemented almost identically, it seems a waste. In most architectures, they itterate across cpus, either returning the first one on the node, or adding each to a mask and returning that.

Besides, it's not gone yet... ;)

Cheers!

-Matt

On Tue, Oct 29, 2002 at 02:19:25PM -0800, Matthew Dobson wrote:
+/*
+ * Returns the number of the first CPU on Node 'node'.
+ * Slow in the current implementation.
+ * Who needs this?
+ */
+/* #define __node_to_first_cpu(node) pool_cpus[pool_ptr[node]] */ +static inline int __node_to_first_cpu(int node)

So far so safe... though no obvious use of it.

This assumes the value returned by __node_to_cpu_mask() is a single word.

Bill

Yep...

Which is the case right now. When (not if) that changes, we'll come up with more flexible ffs macros, or a better way to count variable length bitmasks... especially as there will be a TON of them.

Cheers!

-Matt

Should I use another email address?

Yes, I know of one usage meanwhile. The problem I see is: as far as I understand the CPUs are not sorted by the node numbers. The NUMA API doesn't require that, I think. So finding the first CPU in a node is not really useful for looping over the CPUs of only one node.

When you think of further developments of the NUMA API, I'd suggest two:

1: Add a sorted list of the nodes and a pointer array into that list pointing to the first CPU in the node. Like

int node_cpus[NR_CPUS];
int node_first_ptr[MAX_NUMNODES+1];

(or macros, doesn't matter).

Example: 2 nodes:

         node_cpus : 0 1 4 5 2 3 6 7
              node : 0 0 0 0 1 1 1 1
           pointer : ^       ^       ^

=> node_first_ptr[]: 0 4 8

One can initialize this easilly by using the __cpu_to_node() macro. And with this you can loop over the cpus of one node by doing:

       for (i=node_first_ptr[node]; i<node_first_ptr[node+1]; i++) {
           cpu = node_cpus[i];
           ... do stuff with cpu ...
       }

2: In ACPI there is a table describing the latency ratios between the nodes. It is called SLIT (System Locality Information Table). On an 8 node system (NEC TX7) this is a matrix of dimension numnodes*numnodes:

int __node_distance[ 8 * 8] = { 10, 15, 15, 15, 20, 20, 20, 20,

                                   15, 10, 15, 15, 20, 20, 20, 20,
                                   15, 15, 10, 15, 20, 20, 20, 20,
                                   15, 15, 15, 10, 20, 20, 20, 20,
                                   20, 20, 20, 20, 10, 15, 15, 15,
                                   20, 20, 20, 20, 15, 10, 15, 15,
                                   20, 20, 20, 20, 15, 15, 10, 15,
                                   20, 20, 20, 20, 15, 15, 15, 10 };

#define node_distance(i,j) __node_distance[i*8+j]

This means:

   node_distance(i,i) = 10   (i==j: same node, lowest latency)
   node_distance(i,j) = 15   (i!=j: different node, same supernode)
   node_distance(i,j) = 20   (i!=j: different node, different

supernode)

This macro or function describes the NUMA topology of a multi-level system very well. As the table comes for free with NUMA systems with ACPI (sooner or later they'll all have this, especially if they want to run Windows, too), it would be easy to take it into the topology.h.

Regards,
Erich