Skip to contentSkip to conversation

discontigmem patch for 2.4.20

Discontig-devel Kimi Suganuma 2 participants 3 messages Mar 7, 2003 – Mar 12, 2003

Subsystems (5)
lore
Share & tools
lore
#0 lore raw

Hi Bjorn,

Thank you for your consideration. I'll fix all warnings.

And about problems on HP machines, I guess the reason is that the current CONFIG_NUMA kernel doesn't work right on a system which needs the VIRTUAL_MEM_MAP function.
I thought that the CONFIG_NUMA kernel doesn't have to work on such the systems, I mean you can make a kernel with just turning off CONFIG_NUMA at config.
However, it would better if CONFIG_NUMA kernel works on all types of machines. So I'll try to find out a solution for this issue.

Best Regards,
Kimi

Quoted 27 lines · On Thu, 6 Mar 2003 11:31:25 -0700 Bjorn Helgaas <[email protected]> wrote:
Quoted 6 lines

I back ported the IA64 discontigmem function in 2.5 to 2.4.20. I tested the patch on 8 way Itanium2 NUMA server with a NUMA kernel and an SMP kernel.

David, Bjorn, please let me know is there any possibility that you take this patch into the ia64-patch for 2.4.

I'm in the process of merging this patch, but when I build a "generic" kernel (with CONFIG_NUMA=y and CONFIG_DISCONTIGMEM=y), I get many new warnings:

/home/helgaas/bk/testing/include/asm/mmzone.h:62:21: warning: "virt_to_page" redefined /home/helgaas/bk/testing/include/asm/page.h:57:1: warning: this is the location of the previous definition /home/helgaas/bk/testing/include/asm/mmzone.h:71:21: warning: "page_to_phys" redefined /home/helgaas/bk/testing/include/asm/page.h:58:1: warning: this is the location of the previous definition /home/helgaas/bk/testing/include/linux/mmzone.h:229:21: warning: "numa_node_id" redefined /home/helgaas/bk/testing/include/asm/processor.h:206:1: warning: this is the location of the previous definition

Could you look into these and send a new patch to correct them? Also, the resulting kernel doesn't boot (it MCAs) on HP rx2600 and zx2000. I'll look into it in my spare time, but you can probably do so more efficiently.

A small patch that applies on top of the previous patch would be easiest.

Bjorn

Signature · 1 line
suganuma <[email protected]>
#1 lore raw
Quoted 3 lines

And about problems on HP machines, I guess the reason is that the current CONFIG_NUMA kernel doesn't work right on a system which needs the VIRTUAL_MEM_MAP function.

Ah, it looks like you enforce this in the 2.5 config files:

config VIRTUAL_MEM_MAP
bool "Enable Virtual Mem Map"
depends on !NUMA

Unfortunately, 2.4 doesn't have VIRTUAL_MEM_MAP as a config option (it's enabled automatically if needed), so we can't do that directly.

Quoted 2 lines

... it would better if CONFIG_NUMA kernel works on all types of machines. So I'll try to find out a solution for this issue.

That would be ideal. Seems like there ought to be opportunities for unification of DISCONTIGMEM, VIRTUAL_MEM_MAP, and parts of NUMA. The current scheme feels a little clunky, especially for configuring a "generic" kernel. (It sounds like it isn't even possible to configure a kernel generic enough for both SGI SNx and HP zx1 boxes.)

Bjorn

#2 lore raw
In reply to Kimi Suganuma

Hi Bjorn, Erich,

I fixed a bug about cpu_to_node_map[] and eliminated warnings as much as I can. Now, I'll start to think of merging VIRTUAL_MEM_MAP and NUMA. :-)

Regards,
Kimi

Quoted 26 lines · On Fri, 7 Mar 2003 12:36:01 -0700 Bjorn Helgaas <[email protected]> wrote:
Quoted 3 lines

And about problems on HP machines, I guess the reason is that the current CONFIG_NUMA kernel doesn't work right on a system which needs the VIRTUAL_MEM_MAP function.

Ah, it looks like you enforce this in the 2.5 config files:

config VIRTUAL_MEM_MAP
bool "Enable Virtual Mem Map"
depends on !NUMA

Unfortunately, 2.4 doesn't have VIRTUAL_MEM_MAP as a config option (it's enabled automatically if needed), so we can't do that directly.

Quoted 2 lines

... it would better if CONFIG_NUMA kernel works on all types of machines. So I'll try to find out a solution for this issue.

That would be ideal. Seems like there ought to be opportunities for unification of DISCONTIGMEM, VIRTUAL_MEM_MAP, and parts of NUMA. The current scheme feels a little clunky, especially for configuring a "generic" kernel. (It sounds like it isn't even possible to configure a kernel generic enough for both SGI SNx and HP zx1 boxes.)

Bjorn

Signature · 1 line
suganuma <[email protected]>
arch/ia64/config.in +8 -0
diff -Nur linux-2.4.20-base/arch/ia64/config.in linux-2.4.20-dcm/arch/ia64/config.in
@@ -66,6 +66,14 @@
 fi
 
 if [ "$CONFIG_IA64_GENERIC" = "y" -o "$CONFIG_IA64_DIG" = "y" -o "$CONFIG_IA64_HP_ZX1" = "y" ]; then
+	bool '  Enable NUMA support' CONFIG_NUMA
+	if [ "$CONFIG_NUMA" = "y" ]; then
+		define_bool CONFIG_DISCONTIGMEM y
+  		choice 'Maximum Memory per NUMA Node'			\
+		"16GB		CONFIG_IA64_NODESIZE_16GB		\
+		 64GB		CONFIG_IA64_NODESIZE_64GB		\
+		 256GB		CONFIG_IA64_NODESIZE_256GB" 16GB
+	fi
 	bool '  Enable IA-64 Machine Check Abort' CONFIG_IA64_MCA
 	define_bool CONFIG_PM y
 fi
arch/ia64/kernel/acpi.c +195 -7

243 lines not on this page · raw message

arch/ia64/kernel/setup.c +91 -50

230 lines not on this page · raw message

arch/ia64/kernel/smpboot.c +63 -0

66 lines not on this page · raw message

arch/ia64/mm/Makefile +2 -0
diff -Nur linux-2.4.20-base/arch/ia64/mm/Makefile linux-2.4.20-dcm/arch/ia64/mm/Makefile
@@ -12,5 +12,7 @@
 export-objs := init.o
 
 obj-y	 := init.o fault.o tlb.o extable.o
+obj-$(CONFIG_NUMA) += numa.o
+obj-$(CONFIG_DISCONTIGMEM) += discontig.o
 
 include $(TOPDIR)/Rules.make
arch/ia64/mm/discontig.c +315 -0

315 lines not on this page · raw message

arch/ia64/mm/init.c +63 -41

215 lines not on this page · raw message

arch/ia64/mm/numa.c +46 -0

46 lines not on this page · raw message

drivers/acpi/Config.in +6 -0
diff -Nur linux-2.4.20-base/drivers/acpi/Config.in linux-2.4.20-dcm/drivers/acpi/Config.in
@@ -36,6 +36,9 @@
     tristate     '  Fan'		CONFIG_ACPI_FAN
     tristate     '  Processor'		CONFIG_ACPI_PROCESSOR
     dep_tristate '  Thermal Zone' CONFIG_ACPI_THERMAL $CONFIG_ACPI_PROCESSOR
+    if [ "$CONFIG_NUMA" = "y" ]; then
+      bool	 '  NUMA support' 	CONFIG_ACPI_NUMA
+    fi
     bool         '  Debug Statements' 	CONFIG_ACPI_DEBUG
   fi
 
@@ -119,6 +122,9 @@
     tristate     '  Fan'		CONFIG_ACPI_FAN
     tristate     '  Processor'		CONFIG_ACPI_PROCESSOR
     dep_tristate '  Thermal Zone' CONFIG_ACPI_THERMAL $CONFIG_ACPI_PROCESSOR
+    if [ "$CONFIG_NUMA" = "y" ]; then
+      bool	 '  NUMA support' 	CONFIG_ACPI_NUMA
+    fi
     bool         '  Debug Statements' 	CONFIG_ACPI_DEBUG
     endmenu
   fi
drivers/acpi/Makefile +1 -0
diff -Nur linux-2.4.20-base/drivers/acpi/Makefile linux-2.4.20-dcm/drivers/acpi/Makefile
@@ -50,6 +50,7 @@
   obj-$(CONFIG_ACPI_PROCESSOR)	+= processor.o
   obj-$(CONFIG_ACPI_THERMAL)	+= thermal.o
   obj-$(CONFIG_ACPI_SYSTEM)	+= system.o
+obj-$(CONFIG_ACPI_NUMA)         += numa.o
 endif
 
 include $(TOPDIR)/Rules.make
drivers/acpi/numa.c +186 -0

186 lines not on this page · raw message

drivers/acpi/tables.c +23 -9
diff -Nur linux-2.4.20-base/drivers/acpi/tables.c linux-2.4.20-dcm/drivers/acpi/tables.c
@@ -224,11 +224,13 @@
 
 
 int __init
-acpi_table_parse_madt (
+acpi_table_parse_madt_family (
 	enum acpi_table_id	id,
+	unsigned long		madt_size,
+	int			entry_id,
 	acpi_madt_entry_handler	handler)
 {
-	struct acpi_table_madt	*madt = NULL;
+	void			*madt = NULL;
 	acpi_table_entry_header	*entry = NULL;
 	unsigned long		count = 0;
 	unsigned long		madt_end = 0;
@@ -240,19 +242,21 @@
 	/* Locate the MADT (if exists). There should only be one. */
 
 	for (i = 0; i < sdt.count; i++) {
-		if (sdt.entry[i].id != ACPI_APIC)
+		if (sdt.entry[i].id != id)
 			continue;
-		madt = (struct acpi_table_madt *)
+		madt = (void *)
 			__acpi_map_table(sdt.entry[i].pa, sdt.entry[i].size);
 		if (!madt) {
-			printk(KERN_WARNING PREFIX "Unable to map MADT\n");
+			printk(KERN_WARNING PREFIX "Unable to map %s\n",
+			       acpi_table_signatures[id]);
 			return -ENODEV;
 		}
 		break;
 	}
 
 	if (!madt) {
-		printk(KERN_WARNING PREFIX "MADT not present\n");
+		printk(KERN_WARNING PREFIX "%s not present\n",
+		       acpi_table_signatures[id]);
 		return -ENODEV;
 	}
 
@@ -261,18 +265,28 @@
 	/* Parse all entries looking for a match. */
 
 	entry = (acpi_table_entry_header *)
-		((unsigned long) madt + sizeof(struct acpi_table_madt));
+		((unsigned long) madt + madt_size);
 
 	while (((unsigned long) entry) < madt_end) {
-		if (entry->type == id) {
+		if (entry->type == entry_id) {
 			count++;
 			handler(entry);
 		}
 		entry = (acpi_table_entry_header *)
-			((unsigned long) entry += entry->length);
+			((unsigned long) entry + entry->length);
 	}
 
 	return count;
+}
+
+
+int __init
+acpi_table_parse_madt (
+	enum acpi_madt_entry_id	id,
+	acpi_madt_entry_handler	handler)
+{
+	return acpi_table_parse_madt_family(ACPI_APIC, sizeof(struct acpi_table_madt),
+					    id, handler);
 }
 
 
include/asm-ia64/acpi.h +7 -6
diff -Nur linux-2.4.20-base/include/asm-ia64/acpi.h linux-2.4.20-dcm/include/asm-ia64/acpi.h
@@ -97,17 +97,18 @@
 	} while (0)
 
 const char *acpi_get_sysname (void);
-int acpi_boot_init (char *cdline);
 int acpi_request_vector (u32 int_type);
 int acpi_get_prt (struct pci_vector_struct **vectors, int *count);
 int acpi_get_interrupt_model (int *type);
 int acpi_irq_to_vector (u32 irq);
 
-#ifdef CONFIG_DISCONTIGMEM
-#define NODE_ARRAY_INDEX(x)	((x) / 8)	/* 8 bits/char */
-#define NODE_ARRAY_OFFSET(x)	((x) % 8)	/* 8 bits/char */
-#define MAX_PXM_DOMAINS		(256)
-#endif /* CONFIG_DISCONTIGMEM */
+#ifdef CONFIG_ACPI_NUMA
+#include <asm/numa.h>
+/* Proximity bitmap length; _PXM is at most 255 (8 bit)*/
+#define MAX_PXM_DOMAINS (256)
+extern int __initdata pxm_to_nid_map[MAX_PXM_DOMAINS];
+extern int __initdata nid_to_pxm_map[NR_NODES];
+#endif
 
 #endif /*__KERNEL__*/
 
include/asm-ia64/mmzone.h +143 -0

143 lines not on this page · raw message

include/asm-ia64/nodedata.h +75 -0

75 lines not on this page · raw message

include/asm-ia64/numa.h +70 -0

70 lines not on this page · raw message

include/asm-ia64/numnodes.h +7 -0
diff -Nur linux-2.4.20-base/include/asm-ia64/numnodes.h linux-2.4.20-dcm/include/asm-ia64/numnodes.h
@@ -0,0 +1,7 @@
+#ifndef _ASM_MAX_NUMNODES_H
+#define _ASM_MAX_NUMNODES_H
+
+#include <asm/mmzone.h>
+#define MAX_NUMNODES	NR_NODES
+
+#endif /* _ASM_MAX_NUMNODES_H */
include/asm-ia64/page.h +2 -6
diff -Nur linux-2.4.20-base/include/asm-ia64/page.h linux-2.4.20-dcm/include/asm-ia64/page.h
@@ -52,20 +52,16 @@
  */
 #define MAP_NR_DENSE(addr)	(((unsigned long) (addr) - PAGE_OFFSET) >> PAGE_SHIFT)
 
+#ifndef CONFIG_DISCONTIGMEM
 #ifdef CONFIG_IA64_GENERIC
 # include <asm/machvec.h>
 # define virt_to_page(kaddr)	(mem_map + platform_map_nr(kaddr))
 # define page_to_phys(page)	((page - mem_map) << PAGE_SHIFT)
-#elif defined (CONFIG_IA64_SGI_SN1)
-# ifndef CONFIG_DISCONTIGMEM
-#  define virt_to_page(kaddr)	(mem_map + MAP_NR_DENSE(kaddr))
-#  define page_to_phys(page)	XXX fix me
-# endif
 #else
 # define virt_to_page(kaddr)	(mem_map + MAP_NR_DENSE(kaddr))
 # define page_to_phys(page)	((page - mem_map) << PAGE_SHIFT)
 #endif
-
+#endif
 struct page;
 extern int ia64_page_valid (struct page *);
 #define VALID_PAGE(page)	(((page - mem_map) < max_mapnr) && ia64_page_valid(page))
include/asm-ia64/pgtable.h +12 -1
diff -Nur linux-2.4.20-base/include/asm-ia64/pgtable.h linux-2.4.20-dcm/include/asm-ia64/pgtable.h
@@ -206,6 +206,15 @@
  * Conversion functions: convert a page and protection to a page entry,
  * and a page entry and page directory to the page they refer to.
  */
+#ifdef CONFIG_DISCONTIGMEM
+#define mk_pte(page,pgprot)							\
+({										\
+	pte_t __pte;								\
+										\
+	pte_val(__pte) = (unsigned long)page_address(page) - PAGE_OFFSET + pgprot_val(pgprot);	\
+	__pte;									\
+})
+#else
 #define mk_pte(page,pgprot)							\
 ({										\
 	pte_t __pte;								\
@@ -213,6 +222,7 @@
 	pte_val(__pte) = ((page - mem_map) << PAGE_SHIFT) | pgprot_val(pgprot);	\
 	__pte;									\
 })
+#endif
 
 /* This takes a physical page address that is used by the remapping functions */
 #define mk_pte_phys(physpage, pgprot) \
@@ -440,6 +450,7 @@
  */
 #define pgtable_cache_init()	do { } while (0)
 
+#ifndef CONFIG_DISCONTIGMEM
 /* arch mem_map init routines are needed due to holes in a virtual mem_map */
 #define HAVE_ARCH_MEMMAP_INIT
 
@@ -449,7 +460,7 @@
 extern unsigned long arch_memmap_init (memmap_init_callback_t *callback,
 	struct page *start, struct page *end, int zone,
 	unsigned long start_paddr, int highmem);
-
+#endif /* CONFIG_DISCONTIGMEM */
 # endif /* !__ASSEMBLY__ */
 
 /*
include/asm-ia64/processor.h +7 -20
diff -Nur linux-2.4.20-base/include/asm-ia64/processor.h linux-2.4.20-dcm/include/asm-ia64/processor.h
@@ -87,6 +87,9 @@
 #include <asm/rse.h>
 #include <asm/unwind.h>
 #include <asm/atomic.h>
+#ifdef CONFIG_NUMA
+#include <asm/nodedata.h>
+#endif
 
 /* like above but expressed as bitfields for more efficient access: */
 struct ia64_psr {
@@ -187,9 +190,8 @@
 	} ipi;
 #endif
 #ifdef CONFIG_NUMA
-	void *node_directory;
-	int numa_node_id;
-	struct cpuinfo_ia64 *cpu_data[NR_CPUS];
+	struct ia64_node_data *node_data;
+	int nodeid;
 #endif
 	/* Platform specific word.  MUST BE LAST IN STRUCT */
 	__u64 platform_specific;
@@ -201,23 +203,8 @@
  */
 #define local_cpu_data		((struct cpuinfo_ia64 *) PERCPU_ADDR)
 
-/*
- * On NUMA systems, cpu_data for each cpu is allocated during cpu_init() & is allocated on
- * the node that contains the cpu. This minimizes off-node memory references.  cpu_data
- * for each cpu contains an array of pointers to the cpu_data structures of each of the
- * other cpus.
- *
- * On non-NUMA systems, cpu_data is a static array allocated at compile time.  References
- * to the cpu_data of another cpu is done by direct references to the appropriate entry of
- * the array.
- */
-#ifdef CONFIG_NUMA
-# define cpu_data(cpu)		local_cpu_data->cpu_data[cpu]
-# define numa_node_id()		(local_cpu_data->numa_node_id)
-#else
-  extern struct cpuinfo_ia64 _cpu_data[NR_CPUS];
-# define cpu_data(cpu)		(&_cpu_data[cpu])
-#endif
+extern struct cpuinfo_ia64	_cpu_data[NR_CPUS];
+#define cpu_data(cpu)		(&_cpu_data[cpu])
 
 extern void identify_cpu (struct cpuinfo_ia64 *);
 extern void print_cpu_info (struct cpuinfo_ia64 *);
include/asm-ia64/smp.h +2 -0
diff -Nur linux-2.4.20-base/include/asm-ia64/smp.h linux-2.4.20-dcm/include/asm-ia64/smp.h
@@ -122,6 +122,8 @@
 extern int smp_call_function_single (int cpuid, void (*func) (void *info), void *info,
 				     int retry, int wait);
 
+extern void smp_build_cpu_map(void);
+
 
 #endif /* CONFIG_SMP */
 #endif /* _ASM_IA64_SMP_H */
include/asm-ia64/topology.h +63 -0

63 lines not on this page · raw message

include/linux/acpi.h +8 -0
diff -Nur linux-2.4.20-base/include/linux/acpi.h linux-2.4.20-dcm/include/linux/acpi.h
@@ -344,6 +344,14 @@
 void acpi_table_print (struct acpi_table_header *, unsigned long);
 void acpi_table_print_madt_entry (acpi_table_entry_header *);
 
+#ifdef CONFIG_ACPI_NUMA
+int __init acpi_numa_init(void);
+void __init acpi_numa_slit_init (struct acpi_table_slit *);
+void __init acpi_numa_processor_affinity_init (struct acpi_table_processor_affinity *);
+void __init acpi_numa_memory_affinity_init (struct acpi_table_memory_affinity *);
+void __init acpi_numa_arch_fixup(void);
+#endif
+
 #endif /*CONFIG_ACPI_BOOT*/
 
 
include/linux/mmzone.h +15 -0
diff -Nur linux-2.4.20-base/include/linux/mmzone.h linux-2.4.20-dcm/include/linux/mmzone.h
@@ -8,6 +8,12 @@
 #include <linux/spinlock.h>
 #include <linux/list.h>
 #include <linux/wait.h>
+#ifdef CONFIG_DISCONTIGMEM
+#include <asm/numnodes.h>
+#endif
+#ifndef MAX_NUMNODES
+#define MAX_NUMNODES 1
+#endif
 
 /*
  * Free memory management - zoned buddy allocator.
@@ -212,6 +218,15 @@
 #define for_each_zone(zone) \
 	for(zone = pgdat_list->node_zones; zone; zone = next_zone(zone))
 
+#ifdef CONFIG_NUMA
+#define MAX_NR_MEMBLKS  BITS_PER_LONG /* Max number of Memory Blocks */
+#else /* !CONFIG_NUMA */
+#define MAX_NR_MEMBLKS  1
+#endif /* CONFIG_NUMA */
+
+#include <asm/topology.h>
+/* Returns the number of the current Node. */
+#define numa_node_id()          (__cpu_to_node(smp_processor_id()))
 
 #ifndef CONFIG_DISCONTIGMEM
 
init/main.c +2 -0
diff -Nur linux-2.4.20-base/init/main.c linux-2.4.20-dcm/init/main.c
@@ -290,6 +290,7 @@
 
 
 extern void setup_arch(char **);
+extern void __init build_all_zonelists(void);
 extern void cpu_idle(void);
 
 unsigned long wait_init_idle;
@@ -360,6 +361,7 @@
 	lock_kernel();
 	printk(linux_banner);
 	setup_arch(&command_line);
+	build_all_zonelists();
 	printk("Kernel command line: %s\n", saved_command_line);
 	parse_options(command_line);
 	trap_init();
mm/page_alloc.c +55 -28

112 lines not on this page · raw message