Skip to contentSkip to conversation

you have how many nodes??

PATCH In review Jesse Barnes 5 participants 17 messages Sep 10, 2003 – Oct 21, 2003

Subsystems (6)
Memory management Arm (32-bit) MM core SuperH +2 more subsystems
lore
Share & tools
lore

Jesse Barnes

author
#0 lore raw

PATCH 1/1 you have how many nodes??

Needed this for booting on a 128 node system.

Thanks,
Jesse

include/linux/mm.h +1 -1
diff -Nru a/include/linux/mm.h b/include/linux/mm.h
@@ -324,7 +324,7 @@
  * sets it, so none of the operations on it need to be atomic.
  */
 #define NODE_SHIFT 4
-#define ZONE_SHIFT (BITS_PER_LONG - 8)
+#define ZONE_SHIFT (BITS_PER_LONG - 10)
 
 struct zone;
 extern struct zone *zone_table[];
include/linux/mmzone.h +2 -2
diff -Nru a/include/linux/mmzone.h b/include/linux/mmzone.h
@@ -311,8 +311,8 @@
 
 #include <asm/mmzone.h>
 
-/* page->zone is currently 8 bits ... */
-#define MAX_NR_NODES		(255 / MAX_NR_ZONES)
+/* page->zone is currently 10 bits ... */
+#define MAX_NR_NODES		NR_NODES
 
 #endif /* !CONFIG_DISCONTIGMEM */
#1 lore raw

include/linux/mm.h line 327

Quoted 5 lines · [email protected] (Jesse Barnes) wrote:

Needed this for booting on a 128 node system.

include/linux/mm.h +1 -1
-#define ZONE_SHIFT (BITS_PER_LONG - 8)
+#define ZONE_SHIFT (BITS_PER_LONG - 10)

eeek, ia32 just lost another two page flags.

This stuff needs to be controlled by per-arch and per-subarch header files.

Instead of going backwards like this we'd like to actually free up _more_ bits in page->flags. The worst (and controlling) case is on 32-bit NUMA:
eight nodes, three zones per node. That's five bits, leaving us 27 page flags.

So we'd need

	include/asm-foo/zonestuff.h:
	
	#define ARCH_MAX_NODES_SHIFT	3	/* Up to 8 nodes */
	#define ARCH_MAX_ZONES_SHIFT	2	/* Up to 4 zones per node */

and all the mm.h/mmzone.h constants use those two.

I think. We could just say "dang numaq needs five bits", so:

	#if BITS_PER_LONG == 32
	#define ZONE_SHIFT 5
	#else
	#define ZONE_SHIFT 10
	#endif

Bit sleazy, but I think that would suffice.

author
#2 lore raw
Quoted 8 lines · Andrew Morton

I think. We could just say "dang numaq needs five bits", so:

	#if BITS_PER_LONG == 32
	#define ZONE_SHIFT 5
	#else
	#define ZONE_SHIFT 10
	#endif

That's fine with me, do you want me to rediff and send a new patch?

Thanks,
Jesse

#3 lore raw
Quoted 5 lines · [email protected] (Jesse Barnes) wrote:
Quoted 3 lines · Andrew Morton

I think. We could just say "dang numaq needs five bits", so:

Quoted 8 lines
Quoted 5 lines
	#if BITS_PER_LONG == 32
	#define ZONE_SHIFT 5
	#else
	#define ZONE_SHIFT 10
	#endif

That's fine with me, do you want me to rediff and send a new patch?

Well your patch as it stands would appear to break NUMAQ builds, due to NUMAQ setting MAX_NUMNODES directly in the arch code. ia64 is using another layer of macroification via NR_NODES instead.

MAX_NUMNODES, NR_NODES and MAX_NR_NODES appear to be a bit of a mess, and they should all be replaced with shift distances anyway.

Could you please get together with Martin Bligh, come up with something which works on NUMAQ and your 128 CPU PDA and also cast an eye across the other architectures (sparc64, sh, ...)? It all needs a bit of thought and a spring clean.

Thanks.

#4 lore raw
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines

I think. We could just say "dang numaq needs five bits", so:

	# if BITS_PER_LONG == 32
	# define ZONE_SHIFT 5
	# else
	# define ZONE_SHIFT 10
	# endif
Quoted 8 lines
Quoted 1 line

That's fine with me, do you want me to rediff and send a new patch?

Well your patch as it stands would appear to break NUMAQ builds, due to NUMAQ setting MAX_NUMNODES directly in the arch code. ia64 is using another layer of macroification via NR_NODES instead.

MAX_NUMNODES, NR_NODES and MAX_NR_NODES appear to be a bit of a mess, and they should all be replaced with shift distances anyway.

;-)

Yes, it's a turgid mess.

I'd prefer to define things in terms of MAX_NUMNODES, and derive the shifts from that if possible - much more intuitive to maintain.
But other than that I agree completely with you.

Quoted 4 lines

Could you please get together with Martin Bligh, come up with something which works on NUMAQ and your 128 CPU PDA and also cast an eye across the other architectures (sparc64, sh, ...)? It all needs a bit of thought and a spring clean.

I'll have a look, I'm sure we can come up with something between us.

M.

author
#5 lore raw
Quoted 5 lines · Martin J. Bligh

Yes, it's a turgid mess.

I'd prefer to define things in terms of MAX_NUMNODES, and derive the shifts from that if possible - much more intuitive to maintain.
But other than that I agree completely with you.

Yeah, I don't mind switching, should just be a search and replace.

Quoted 6 lines
Quoted 4 lines

Could you please get together with Martin Bligh, come up with something which works on NUMAQ and your 128 CPU PDA and also cast an eye across the other architectures (sparc64, sh, ...)? It all needs a bit of thought and a spring clean.

I'll have a look, I'm sure we can come up with something between us.

Cool, thanks.

Jesse

#16 lore raw
In reply to Andrew Morton
Quoted 4 lines · Andrew Morton

Instead of going backwards like this we'd like to actually free up _more_ bits in page->flags. The worst (and controlling) case is on 32-bit NUMA:
eight nodes, three zones per node. That's five bits, leaving us 27 page flags.

The worst case for i386 NUMA is actually 16 nodes.

-- wli

#6 lore raw
In reply to Jesse Barnes

Ok, I made an attempt to clean up this mess quite a while ago (2.5.47), but that patch is utterly useless now. At Martin's urging I've created a new series of patches to resolve this.

01 - Make sure MAX_NUMNODES is defined in one and only one place. Remove superfluous definitions. Instead of defining MAX_NUMNODES in asm/numnodes.h, we define NODES_SHIFT there. Then in linux/mmzone.h we turn that NODES_SHIFT value into MAX_NUMNODES.

02 - Remove MAX_NR_NODES. This value is only used in a couple of places, and it's incorrectly used in all those places as far as I can tell. Replace with MAX_NUMNODES. Create MAX_NODES_SHIFT and use this value to check NODES_SHIFT is appropriate. A possible future patch should make MAX_NODES_SHIFT vary based on 32 vs. 64 bit archs.

03 - Fix up the sh arch. sh defined NR_NODES, change sh to use standard MAX_NUMNODES instead.

04 - Fix up the arm arch. This needs to be reviewed. Relatively straightforward replacement of NR_NODES with standard MAX_NUMNODES.

05 - Fix up the ia64 arch. This *definitely* needs to be reviewed. This code made my head hurt. I think I may have gotten it right. Totally untested.

Cheers!

-Matt

Quoted 29 lines · Jesse Barnes
Quoted 5 lines · Martin J. Bligh

Yes, it's a turgid mess.

I'd prefer to define things in terms of MAX_NUMNODES, and derive the shifts from that if possible - much more intuitive to maintain.
But other than that I agree completely with you.

Yeah, I don't mind switching, should just be a search and replace.

Quoted 6 lines
Quoted 4 lines

Could you please get together with Martin Bligh, come up with something which works on NUMAQ and your 128 CPU PDA and also cast an eye across the other architectures (sparc64, sh, ...)? It all needs a bit of thought and a spring clean.

I'll have a look, I'm sure we can come up with something between us.

Cool, thanks.

Jesse
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

#7 lore raw

PATCH 1/1 Clean up MAX_NR_NODES/NUMNODES/etc. [1/5]

Quoted 11 lines · Matthew Dobson

Ok, I made an attempt to clean up this mess quite a while ago (2.5.47), but that patch is utterly useless now. At Martin's urging I've created a new series of patches to resolve this.

01 - Make sure MAX_NUMNODES is defined in one and only one place. Remove superfluous definitions. Instead of defining MAX_NUMNODES in asm/numnodes.h, we define NODES_SHIFT there. Then in linux/mmzone.h we turn that NODES_SHIFT value into MAX_NUMNODES.

02 - Remove MAX_NR_NODES. This value is only used in a couple of places, and it's incorrectly used in all those places as far as I can tell. Replace with MAX_NUMNODES. Create MAX_NODES_SHIFT and use this value to check NODES_SHIFT is appropriate. A possible future patch should make MAX_NODES_SHIFT vary based on 32 vs. 64 bit archs.

03 - Fix up the sh arch. sh defined NR_NODES, change sh to use standard MAX_NUMNODES instead.

04 - Fix up the arm arch. This needs to be reviewed. Relatively straightforward replacement of NR_NODES with standard MAX_NUMNODES.

05 - Fix up the ia64 arch. This *definitely* needs to be reviewed. This code made my head hurt. I think I may have gotten it right. Totally untested.

Cheers!

-Matt

include/asm-alpha/numnodes.h +2 -1
diff -Nurp --exclude-from=/home/mcd/.dontdiff linux-2.6.0-test5/include/asm-alpha/numnodes.h linux-2.6.0-test5-max_numnodes2nodes_shift/include/asm-alpha/numnodes.h
@@ -1,6 +1,7 @@
 #ifndef _ASM_MAX_NUMNODES_H
 #define _ASM_MAX_NUMNODES_H
 
-#define MAX_NUMNODES		128 /* Marvel */
+/* Max 128 Nodes - Marvel */
+#define NODES_SHIFT	7
 
 #endif /* _ASM_MAX_NUMNODES_H */
include/asm-i386/numaq.h +0 -1
diff -Nurp --exclude-from=/home/mcd/.dontdiff linux-2.6.0-test5/include/asm-i386/numaq.h linux-2.6.0-test5-max_numnodes2nodes_shift/include/asm-i386/numaq.h
@@ -28,7 +28,6 @@
 
 #ifdef CONFIG_X86_NUMAQ
 
-#define MAX_NUMNODES		16
 extern void get_memcfg_numaq(void);
 #define get_memcfg_numa() get_memcfg_numaq()
include/asm-i386/numnodes.h +8 -4
diff -Nurp --exclude-from=/home/mcd/.dontdiff linux-2.6.0-test5/include/asm-i386/numnodes.h linux-2.6.0-test5-max_numnodes2nodes_shift/include/asm-i386/numnodes.h
@@ -4,11 +4,15 @@
 #include <linux/config.h>
 
 #ifdef CONFIG_X86_NUMAQ
-#include <asm/numaq.h>
+
+/* Max 16 Nodes */
+#define NODES_SHIFT	4
+
 #elif CONFIG_NUMA
-#include <asm/srat.h>
-#else
-#define MAX_NUMNODES	1
+
+/* Max 8 Nodes */
+#define NODES_SHIFT	3
+
 #endif /* CONFIG_X86_NUMAQ */
 
 #endif /* _ASM_MAX_NUMNODES_H */
include/asm-i386/srat.h +0 -1
diff -Nurp --exclude-from=/home/mcd/.dontdiff linux-2.6.0-test5/include/asm-i386/srat.h linux-2.6.0-test5-max_numnodes2nodes_shift/include/asm-i386/srat.h
@@ -27,7 +27,6 @@
 #ifndef _ASM_SRAT_H_
 #define _ASM_SRAT_H_
 
-#define MAX_NUMNODES		8
 extern void get_memcfg_from_srat(void);
 extern unsigned long *get_zholes_size(int);
 #define get_memcfg_numa() get_memcfg_from_srat()
include/asm-ppc64/numnodes.h +2 -1
diff -Nurp --exclude-from=/home/mcd/.dontdiff linux-2.6.0-test5/include/asm-ppc64/numnodes.h linux-2.6.0-test5-max_numnodes2nodes_shift/include/asm-ppc64/numnodes.h
@@ -1,6 +1,7 @@
 #ifndef _ASM_MAX_NUMNODES_H
 #define _ASM_MAX_NUMNODES_H
 
-#define MAX_NUMNODES 16
+/* Max 16 Nodes */
+#define NODES_SHIFT	4
 
 #endif /* _ASM_MAX_NUMNODES_H */
include/asm-x86_64/mmzone.h +0 -1
diff -Nurp --exclude-from=/home/mcd/.dontdiff linux-2.6.0-test5/include/asm-x86_64/mmzone.h linux-2.6.0-test5-max_numnodes2nodes_shift/include/asm-x86_64/mmzone.h
@@ -10,7 +10,6 @@
 
 #define VIRTUAL_BUG_ON(x) 
 
-#include <asm/numnodes.h>
 #include <asm/smp.h>
 
 #define MAXNODE 8
include/asm-x86_64/numnodes.h +2 -5
diff -Nurp --exclude-from=/home/mcd/.dontdiff linux-2.6.0-test5/include/asm-x86_64/numnodes.h linux-2.6.0-test5-max_numnodes2nodes_shift/include/asm-x86_64/numnodes.h
@@ -3,10 +3,7 @@
 
 #include <linux/config.h>
 
-#ifdef CONFIG_DISCONTIGMEM
-#define MAX_NUMNODES 8	/* APIC limit currently */
-#else
-#define MAX_NUMNODES 1
-#endif
+/* Max 8 Nodes - APIC limit currently */
+#define NODES_SHIFT	3
 
 #endif
include/linux/mmzone.h +3 -2
diff -Nurp --exclude-from=/home/mcd/.dontdiff linux-2.6.0-test5/include/linux/mmzone.h linux-2.6.0-test5-max_numnodes2nodes_shift/include/linux/mmzone.h
@@ -14,9 +14,10 @@
 #ifdef CONFIG_DISCONTIGMEM
 #include <asm/numnodes.h>
 #endif
-#ifndef MAX_NUMNODES
-#define MAX_NUMNODES 1
+#ifndef NODES_SHIFT
+#define NODES_SHIFT	0
 #endif
+#define MAX_NUMNODES	(1 << NODES_SHIFT)
 
 /* Free memory management - zoned buddy allocator.  */
 #ifndef CONFIG_FORCE_MAX_ZONEORDER
mm/slab.c +1 -1
diff -Nurp --exclude-from=/home/mcd/.dontdiff linux-2.6.0-test5/mm/slab.c linux-2.6.0-test5-nr_nodes/mm/slab.c
@@ -249,7 +249,7 @@ struct kmem_cache_s {
 	unsigned int		limit;
 /* 2) touched by every alloc & free from the backend */
 	struct kmem_list3	lists;
-	/* NUMA: kmem_3list_t	*nodelists[NR_NODES] */
+	/* NUMA: kmem_3list_t	*nodelists[MAX_NUMNODES] */
 	unsigned int		objsize;
 	unsigned int	 	flags;	/* constant flags */
 	unsigned int		num;	/* # of objs per slab */
#9 lore raw

PATCH 2/5 [PATCH[ Clean up MAX_NR_NODES/NUMNODES/etc. [2/5]

In reply to Matthew Dobson
Quoted 11 lines · Matthew Dobson

Ok, I made an attempt to clean up this mess quite a while ago (2.5.47), but that patch is utterly useless now. At Martin's urging I've created a new series of patches to resolve this.

01 - Make sure MAX_NUMNODES is defined in one and only one place. Remove superfluous definitions. Instead of defining MAX_NUMNODES in asm/numnodes.h, we define NODES_SHIFT there. Then in linux/mmzone.h we turn that NODES_SHIFT value into MAX_NUMNODES.

02 - Remove MAX_NR_NODES. This value is only used in a couple of places, and it's incorrectly used in all those places as far as I can tell. Replace with MAX_NUMNODES. Create MAX_NODES_SHIFT and use this value to check NODES_SHIFT is appropriate. A possible future patch should make MAX_NODES_SHIFT vary based on 32 vs. 64 bit archs.

03 - Fix up the sh arch. sh defined NR_NODES, change sh to use standard MAX_NUMNODES instead.

04 - Fix up the arm arch. This needs to be reviewed. Relatively straightforward replacement of NR_NODES with standard MAX_NUMNODES.

05 - Fix up the ia64 arch. This *definitely* needs to be reviewed. This code made my head hurt. I think I may have gotten it right. Totally untested.

Cheers!

-Matt

arch/i386/kernel/smpboot.c +3 -3
diff -Nurp --exclude-from=/home/mcd/.dontdiff linux-2.6.0-test5-max_numnodes2nodes_shift/arch/i386/kernel/smpboot.c linux-2.6.0-test5-remove-max_nr_nodes/arch/i386/kernel/smpboot.c
@@ -499,8 +499,8 @@ static struct task_struct * __init fork_
 #ifdef CONFIG_NUMA
 
 /* which logical CPUs are on which nodes */
-cpumask_t node_2_cpu_mask[MAX_NR_NODES] =
-				{ [0 ... MAX_NR_NODES-1] = CPU_MASK_NONE };
+cpumask_t node_2_cpu_mask[MAX_NUMNODES] =
+				{ [0 ... MAX_NUMNODES-1] = CPU_MASK_NONE };
 /* which node each logical CPU is on */
 int cpu_2_node[NR_CPUS] = { [0 ... NR_CPUS-1] = 0 };
 

@@ -518,7 +518,7 @@ static inline void unmap_cpu_to_node(int
 	int node;
 
 	printk("Unmapping cpu %d from all nodes\n", cpu);
-	for (node = 0; node < MAX_NR_NODES; node ++)
+	for (node = 0; node < MAX_NUMNODES; node ++)
 		cpu_clear(cpu, node_2_cpu_mask[node]);
 	cpu_2_node[cpu] = -1;
 }
include/linux/mmzone.h +12 -4
diff -Nurp --exclude-from=/home/mcd/.dontdiff linux-2.6.0-test5-max_numnodes2nodes_shift/include/linux/mmzone.h linux-2.6.0-test5-remove-max_nr_nodes/include/linux/mmzone.h
@@ -304,19 +304,27 @@ extern void setup_per_zone_pages_min(voi
 #define numa_node_id()		(cpu_to_node(smp_processor_id()))
 
 #ifndef CONFIG_DISCONTIGMEM
+
 extern struct pglist_data contig_page_data;
 #define NODE_DATA(nid)		(&contig_page_data)
 #define NODE_MEM_MAP(nid)	mem_map
-#define MAX_NR_NODES		1
+#define MAX_NODES_SHIFT		0
+
 #else /* CONFIG_DISCONTIGMEM */
 
 #include <asm/mmzone.h>
-
-/* page->zone is currently 8 bits ... */
-#define MAX_NR_NODES		(255 / MAX_NR_ZONES)
+/*
+ * page->zone is currently 8 bits
+ * there are 3 zones (2 bits)
+ * this leaves 8-2=6 bits for nodes
+ */
+#define MAX_NODES_SHIFT		6
 
 #endif /* !CONFIG_DISCONTIGMEM */
 
+#if NODES_SHIFT > MAX_NODES_SHIFT
+#error NODES_SHIFT > MAX_NODES_SHIFT
+#endif
 
 extern DECLARE_BITMAP(node_online_map, MAX_NUMNODES);
 extern DECLARE_BITMAP(memblk_online_map, MAX_NR_MEMBLKS);
mm/page_alloc.c +1 -1
diff -Nurp --exclude-from=/home/mcd/.dontdiff linux-2.6.0-test5-max_numnodes2nodes_shift/mm/page_alloc.c linux-2.6.0-test5-remove-max_nr_nodes/mm/page_alloc.c
@@ -50,7 +50,7 @@ EXPORT_SYMBOL(nr_swap_pages);
  * Used by page_zone() to look up the address of the struct zone whose
  * id is encoded in the upper bits of page->flags
  */
-struct zone *zone_table[MAX_NR_ZONES*MAX_NR_NODES];
+struct zone *zone_table[MAX_NR_ZONES*MAX_NUMNODES];
 EXPORT_SYMBOL(zone_table);
 
 static char *zone_names[MAX_NR_ZONES] = { "DMA", "Normal", "HighMem" };
#10 lore raw

PATCH 1/1 Clean up MAX_NR_NODES/NUMNODES/etc. [3/5]

In reply to Matthew Dobson
Quoted 11 lines · Matthew Dobson

Ok, I made an attempt to clean up this mess quite a while ago (2.5.47), but that patch is utterly useless now. At Martin's urging I've created a new series of patches to resolve this.

01 - Make sure MAX_NUMNODES is defined in one and only one place. Remove superfluous definitions. Instead of defining MAX_NUMNODES in asm/numnodes.h, we define NODES_SHIFT there. Then in linux/mmzone.h we turn that NODES_SHIFT value into MAX_NUMNODES.

02 - Remove MAX_NR_NODES. This value is only used in a couple of places, and it's incorrectly used in all those places as far as I can tell. Replace with MAX_NUMNODES. Create MAX_NODES_SHIFT and use this value to check NODES_SHIFT is appropriate. A possible future patch should make MAX_NODES_SHIFT vary based on 32 vs. 64 bit archs.

03 - Fix up the sh arch. sh defined NR_NODES, change sh to use standard MAX_NUMNODES instead.

04 - Fix up the arm arch. This needs to be reviewed. Relatively straightforward replacement of NR_NODES with standard MAX_NUMNODES.

05 - Fix up the ia64 arch. This *definitely* needs to be reviewed. This code made my head hurt. I think I may have gotten it right. Totally untested.

Cheers!

-Matt

include/asm-sh/numnodes.h +7 -0
diff -Nurp --exclude-from=/home/mcd/.dontdiff linux-2.6.0-test5/include/asm-sh/numnodes.h linux-2.6.0-test5-max_numnodes2nodes_shift/include/asm-sh/numnodes.h
@@ -0,0 +1,7 @@
+#ifndef _ASM_MAX_NUMNODES_H
+#define _ASM_MAX_NUMNODES_H
+
+/* Max 2 Nodes */
+#define NODES_SHIFT	1
+
+#endif /* _ASM_MAX_NUMNODES_H */
arch/sh/mm/init.c +2 -2
diff -Nurp --exclude-from=/home/mcd/.dontdiff linux-2.6.0-test5/arch/sh/mm/init.c linux-2.6.0-test5-nr_nodes/arch/sh/mm/init.c
@@ -51,8 +51,8 @@ unsigned long mmu_context_cache = NO_CON
 #endif
 
 #ifdef CONFIG_DISCONTIGMEM
-pg_data_t discontig_page_data[NR_NODES];
-bootmem_data_t discontig_node_bdata[NR_NODES];
+pg_data_t discontig_page_data[MAX_NUMNODES];
+bootmem_data_t discontig_node_bdata[MAX_NUMNODES];
 #endif
 
 void show_mem(void)
include/asm-sh/mmzone.h +5 -5
diff -Nurp --exclude-from=/home/mcd/.dontdiff linux-2.6.0-test5/include/asm-sh/mmzone.h linux-2.6.0-test5-nr_nodes/include/asm-sh/mmzone.h
@@ -10,14 +10,14 @@
 
 #include <linux/config.h>
 
+#ifdef CONFIG_DISCONTIGMEM
+
 /* Currently, just for HP690 */
 #define PHYSADDR_TO_NID(phys)	((((phys) - __MEMORY_START) >= 0x01000000)?1:0)
-#define NR_NODES 2
 
-extern pg_data_t discontig_page_data[NR_NODES];
-extern bootmem_data_t discontig_node_bdata[NR_NODES];
+extern pg_data_t discontig_page_data[MAX_NUMNODES];
+extern bootmem_data_t discontig_node_bdata[MAX_NUMNODES];
 
-#ifdef CONFIG_DISCONTIGMEM
 /*
  * Following are macros that each numa implmentation must define.
  */

@@ -46,7 +46,7 @@ static inline int is_valid_page(struct p
 {
 	unsigned int i;
 
-	for (i = 0; i < NR_NODES; i++) {
+	for (i = 0; i < MAX_NUMNODES; i++) {
 		if (page >= NODE_MEM_MAP(i) &&
 		    page < NODE_MEM_MAP(i) + NODE_DATA(i)->node_size)
 			return 1;
#11 lore raw

PATCH 1/1 Clean up MAX_NR_NODES/NUMNODES/etc. [4/5]

In reply to Matthew Dobson
Quoted 11 lines · Matthew Dobson

Ok, I made an attempt to clean up this mess quite a while ago (2.5.47), but that patch is utterly useless now. At Martin's urging I've created a new series of patches to resolve this.

01 - Make sure MAX_NUMNODES is defined in one and only one place. Remove superfluous definitions. Instead of defining MAX_NUMNODES in asm/numnodes.h, we define NODES_SHIFT there. Then in linux/mmzone.h we turn that NODES_SHIFT value into MAX_NUMNODES.

02 - Remove MAX_NR_NODES. This value is only used in a couple of places, and it's incorrectly used in all those places as far as I can tell. Replace with MAX_NUMNODES. Create MAX_NODES_SHIFT and use this value to check NODES_SHIFT is appropriate. A possible future patch should make MAX_NODES_SHIFT vary based on 32 vs. 64 bit archs.

03 - Fix up the sh arch. sh defined NR_NODES, change sh to use standard MAX_NUMNODES instead.

04 - Fix up the arm arch. This needs to be reviewed. Relatively straightforward replacement of NR_NODES with standard MAX_NUMNODES.

05 - Fix up the ia64 arch. This *definitely* needs to be reviewed. This code made my head hurt. I think I may have gotten it right. Totally untested.

Cheers!

-Matt

arch/arm/mm/discontig.c +3 -3
diff -Nurp --exclude-from=/home/mcd/.dontdiff linux-2.6.0-test5/arch/arm/mm/discontig.c linux-2.6.0-test5-nr_nodes/arch/arm/mm/discontig.c
@@ -15,7 +15,7 @@
 #include <linux/init.h>
 #include <linux/bootmem.h>
 
-#if NR_NODES != 4
+#if MAX_NUMNODES != 4
 #error Fix Me Please
 #endif
 

@@ -23,9 +23,9 @@
  * Our node_data structure for discontiguous memory.
  */
 
-static bootmem_data_t node_bootmem_data[NR_NODES];
+static bootmem_data_t node_bootmem_data[MAX_NUMNODES];
 
-pg_data_t discontig_node_data[NR_NODES] = {
+pg_data_t discontig_node_data[MAX_NUMNODES] = {
   { .bdata = &node_bootmem_data[0] },
   { .bdata = &node_bootmem_data[1] },
   { .bdata = &node_bootmem_data[2] },
arch/arm/mm/init.c +3 -9
diff -Nurp --exclude-from=/home/mcd/.dontdiff linux-2.6.0-test5/arch/arm/mm/init.c linux-2.6.0-test5-nr_nodes/arch/arm/mm/init.c
@@ -33,12 +33,6 @@
 #include <asm/mach/arch.h>
 #include <asm/mach/map.h>
 
-#ifndef CONFIG_DISCONTIGMEM
-#define NR_NODES	1
-#else
-#define NR_NODES	4
-#endif
-
 #ifdef CONFIG_CPU_32
 #define TABLE_OFFSET	(PTRS_PER_PTE)
 #else

@@ -178,7 +172,7 @@ find_memend_and_nodes(struct meminfo *mi
 {
 	unsigned int i, bootmem_pages = 0, memend_pfn = 0;
 
-	for (i = 0; i < NR_NODES; i++) {
+	for (i = 0; i < MAX_NUMNODES; i++) {
 		np[i].start = -1U;
 		np[i].end = 0;
 		np[i].bootmap_pages = 0;

@@ -207,7 +201,7 @@ find_memend_and_nodes(struct meminfo *mi
 			 * we have, we're in trouble.  (maybe we ought to
 			 * limit, instead of bugging?)
 			 */
-			if (numnodes > NR_NODES)
+			if (numnodes > MAX_NUMNODES)
 				BUG();
 		}
 

@@ -365,7 +359,7 @@ static inline void free_bootmem_node_ban
  */
 void __init bootmem_init(struct meminfo *mi)
 {
-	struct node_info node_info[NR_NODES], *np = node_info;
+	struct node_info node_info[MAX_NUMNODES], *np = node_info;
 	unsigned int bootmap_pages, bootmap_pfn, map_pg;
 	int node, initrd_node;
include/asm-arm/arch-clps711x/memory.h +0 -2
diff -Nurp --exclude-from=/home/mcd/.dontdiff linux-2.6.0-test5/include/asm-arm/arch-clps711x/memory.h linux-2.6.0-test5-nr_nodes/include/asm-arm/arch-clps711x/memory.h
@@ -109,8 +109,6 @@
  * 	node 3:  0xd8000000 - 0xdfffffff
  */
 
-#define NR_NODES	4
-
 /*
  * Given a kernel address, find the home node of the underlying memory.
  */
include/asm-arm/arch-sa1100/memory.h +0 -2
diff -Nurp --exclude-from=/home/mcd/.dontdiff linux-2.6.0-test5/include/asm-arm/arch-sa1100/memory.h linux-2.6.0-test5-nr_nodes/include/asm-arm/arch-sa1100/memory.h
@@ -74,8 +74,6 @@
  * 	node 3:  0xd8000000 - 0xdfffffff
  */
 
-#define NR_NODES	4
-
 /*
  * Given a kernel address, find the home node of the underlying memory.
  */
include/asm-arm/memory.h +7 -2
diff -Nurp --exclude-from=/home/mcd/.dontdiff linux-2.6.0-test5/include/asm-arm/memory.h linux-2.6.0-test5-nr_nodes/include/asm-arm/memory.h
@@ -89,6 +89,9 @@ static inline void *phys_to_virt(unsigne
  * This is more complex.  We have a set of mem_map arrays spread
  * around in memory.
  */
+#include <asm/numnodes.h>
+#define NUM_NODES	(1 << NODES_SHIFT)
+
 #define page_to_pfn(page)					\
 	(( (page) - page_zone(page)->zone_mem_map)		\
 	  + page_zone(page)->zone_start_pfn)

@@ -96,12 +99,12 @@ static inline void *phys_to_virt(unsigne
 #define pfn_to_page(pfn)					\
 	(PFN_TO_MAPBASE(pfn) + LOCAL_MAP_NR((pfn) << PAGE_SHIFT))
 
-#define pfn_valid(pfn)		(PFN_TO_NID(pfn) < NR_NODES)
+#define pfn_valid(pfn)		(PFN_TO_NID(pfn) < NUM_NODES)
 
 #define virt_to_page(kaddr)					\
 	(ADDR_TO_MAPBASE(kaddr) + LOCAL_MAP_NR(kaddr))
 
-#define virt_addr_valid(kaddr)	(KVADDR_TO_NID(kaddr) < NR_NODES)
+#define virt_addr_valid(kaddr)	(KVADDR_TO_NID(kaddr) < NUM_NODES)
 
 /*
  * Common discontigmem stuff.

@@ -109,6 +112,8 @@ static inline void *phys_to_virt(unsigne
  */
 #define PHYS_TO_NID(addr)	PFN_TO_NID((addr) >> PAGE_SHIFT)
 
+#undef NUM_NODES
+
 #endif
 
 /*
include/asm-arm/numnodes.h +2 -3
diff -Nurp --exclude-from=/home/mcd/.dontdiff linux-2.6.0-test5/include/asm-arm/numnodes.h linux-2.6.0-test5-nr_nodes/include/asm-arm/numnodes.h
@@ -10,8 +10,7 @@
 #ifndef __ASM_ARM_NUMNODES_H
 #define __ASM_ARM_NUMNODES_H
 
-#include <asm/memory.h>
-
-#define MAX_NUMNODES	NR_NODES
+/* Max 4 Nodes */
+#define NODES_SHIFT	2
 
 #endif
#12 lore raw

PATCH 1/1 Clean up MAX_NR_NODES/NUMNODES/etc. [5/5]

In reply to Matthew Dobson
Quoted 11 lines · Matthew Dobson

Ok, I made an attempt to clean up this mess quite a while ago (2.5.47), but that patch is utterly useless now. At Martin's urging I've created a new series of patches to resolve this.

01 - Make sure MAX_NUMNODES is defined in one and only one place. Remove superfluous definitions. Instead of defining MAX_NUMNODES in asm/numnodes.h, we define NODES_SHIFT there. Then in linux/mmzone.h we turn that NODES_SHIFT value into MAX_NUMNODES.

02 - Remove MAX_NR_NODES. This value is only used in a couple of places, and it's incorrectly used in all those places as far as I can tell. Replace with MAX_NUMNODES. Create MAX_NODES_SHIFT and use this value to check NODES_SHIFT is appropriate. A possible future patch should make MAX_NODES_SHIFT vary based on 32 vs. 64 bit archs.

03 - Fix up the sh arch. sh defined NR_NODES, change sh to use standard MAX_NUMNODES instead.

04 - Fix up the arm arch. This needs to be reviewed. Relatively straightforward replacement of NR_NODES with standard MAX_NUMNODES.

05 - Fix up the ia64 arch. This *definitely* needs to be reviewed. This code made my head hurt. I think I may have gotten it right. Totally untested.

Cheers!

-Matt

arch/ia64/kernel/acpi.c +2 -1
diff -Nurp --exclude-from=/home/mcd/.dontdiff linux-2.6.0-test5-arm/arch/ia64/kernel/acpi.c linux-2.6.0-test5-ia64/arch/ia64/kernel/acpi.c
@@ -41,6 +41,7 @@
 #include <linux/irq.h>
 #include <linux/acpi.h>
 #include <linux/efi.h>
+#include <linux/mmzone.h>
 #include <asm/io.h>
 #include <asm/iosapic.h>
 #include <asm/machvec.h>

@@ -341,7 +342,7 @@ static u32 __initdata pxm_flag[PXM_FLAG_
 #define pxm_bit_test(bit)	(test_bit(bit,(void *)pxm_flag))
 /* maps to convert between proximity domain and logical node ID */
 int __initdata pxm_to_nid_map[MAX_PXM_DOMAINS];
-int __initdata nid_to_pxm_map[NR_NODES];
+int __initdata nid_to_pxm_map[MAX_NUMNODES];
 static struct acpi_table_slit __initdata *slit_table;
 
 /*
arch/ia64/mm/discontig.c +4 -5
diff -Nurp --exclude-from=/home/mcd/.dontdiff linux-2.6.0-test5-arm/arch/ia64/mm/discontig.c linux-2.6.0-test5-ia64/arch/ia64/mm/discontig.c
@@ -13,7 +13,6 @@
 #include <linux/kernel.h>
 #include <linux/mm.h>
 #include <linux/bootmem.h>
-#include <linux/mmzone.h>
 #include <linux/acpi.h>
 #include <linux/efi.h>
 

@@ -23,10 +22,10 @@
  */
 #define GRANULEROUNDUP(n) (((n)+IA64_GRANULE_SIZE-1) & ~(IA64_GRANULE_SIZE-1))
 
-static struct ia64_node_data	*node_data[NR_NODES];
-static long			boot_pg_data[8*NR_NODES+sizeof(pg_data_t)]  __initdata;
-static pg_data_t		*pg_data_ptr[NR_NODES] __initdata;
-static bootmem_data_t		bdata[NR_NODES][NR_BANKS_PER_NODE+1] __initdata;
+static struct ia64_node_data	*node_data[MAX_NUMNODES];
+static long			boot_pg_data[8*MAX_NUMNODES+sizeof(pg_data_t)]  __initdata;
+static pg_data_t		*pg_data_ptr[MAX_NUMNODES] __initdata;
+static bootmem_data_t		bdata[MAX_NUMNODES][NR_BANKS_PER_NODE+1] __initdata;
 
 extern int  filter_rsvd_memory (unsigned long start, unsigned long end, void *arg);
arch/ia64/mm/numa.c +1 -2
diff -Nurp --exclude-from=/home/mcd/.dontdiff linux-2.6.0-test5-arm/arch/ia64/mm/numa.c linux-2.6.0-test5-ia64/arch/ia64/mm/numa.c
@@ -15,7 +15,6 @@
 #include <linux/mm.h>
 #include <linux/init.h>
 #include <linux/bootmem.h>
-#include <linux/mmzone.h>
 #include <asm/numa.h>
 
 /*

@@ -29,7 +28,7 @@ struct node_cpuid_s node_cpuid[NR_CPUS];
  * This is a matrix with "distances" between nodes, they should be
  * proportional to the memory access latency ratios.
  */
-u8 numa_slit[NR_NODES * NR_NODES];
+u8 numa_slit[MAX_NUMNODES * MAX_NUMNODES];
 
 /* Identify which cnode a physical address resides on */
 int
include/asm-ia64/acpi.h +1 -1
diff -Nurp --exclude-from=/home/mcd/.dontdiff linux-2.6.0-test5-arm/include/asm-ia64/acpi.h linux-2.6.0-test5-ia64/include/asm-ia64/acpi.h
@@ -109,7 +109,7 @@ int acpi_get_addr_space (void *obj, u8 t
 /* Proximity bitmap length; _PXM is at most 255 (8 bit)*/
 #define MAX_PXM_DOMAINS (256)
 extern int __initdata pxm_to_nid_map[MAX_PXM_DOMAINS];
-extern int __initdata nid_to_pxm_map[NR_NODES];
+extern int __initdata nid_to_pxm_map[MAX_NUMNODES];
 #endif
 
 #endif /*__KERNEL__*/
include/asm-ia64/mmzone.h +20 -24

76 lines not on this page · raw message

include/asm-ia64/nodedata.h +3 -3
diff -Nurp --exclude-from=/home/mcd/.dontdiff linux-2.6.0-test5-arm/include/asm-ia64/nodedata.h linux-2.6.0-test5-ia64/include/asm-ia64/nodedata.h
@@ -14,7 +14,7 @@
 #define _ASM_IA64_NODEDATA_H
 
 
-#include <asm/mmzone.h>
+#include <linux/mmzone.h>
 
 /*
  * Node Data. One of these structures is located on each node of a NUMA system.

@@ -24,9 +24,9 @@ struct pglist_data;
 struct ia64_node_data {
 	short			active_cpu_count;
 	short			node;
-        struct pglist_data	*pg_data_ptrs[NR_NODES];
+        struct pglist_data	*pg_data_ptrs[MAX_NUMNODES];
 	struct page		*bank_mem_map_base[NR_BANKS];
-	struct ia64_node_data	*node_data_ptrs[NR_NODES];
+	struct ia64_node_data	*node_data_ptrs[MAX_NUMNODES];
 	short			node_id_map[NR_BANKS];
 };
include/asm-ia64/numa.h +3 -9
diff -Nurp --exclude-from=/home/mcd/.dontdiff linux-2.6.0-test5-arm/include/asm-ia64/numa.h linux-2.6.0-test5-ia64/include/asm-ia64/numa.h
@@ -13,17 +13,11 @@
 
 #ifdef CONFIG_NUMA
 
-#ifdef CONFIG_DISCONTIGMEM
-# include <asm/mmzone.h>
-# define NR_MEMBLKS   (NR_BANKS)
-#else
-# define NR_NODES     (8)
-# define NR_MEMBLKS   (NR_NODES * 8)
-#endif
+#include <linux/mmzone.h>
 
 #include <linux/cache.h>
 extern volatile char cpu_to_node_map[NR_CPUS] __cacheline_aligned;
-extern volatile unsigned long node_to_cpu_mask[NR_NODES] __cacheline_aligned;
+extern volatile unsigned long node_to_cpu_mask[MAX_NUMNODES] __cacheline_aligned;
 
 /* Stuff below this line could be architecture independent */
 

@@ -57,7 +51,7 @@ extern struct node_cpuid_s node_cpuid[NR
  * proportional to the memory access latency ratios.
  */
 
-extern u8 numa_slit[NR_NODES * NR_NODES];
+extern u8 numa_slit[MAX_NUMNODES * MAX_NUMNODES];
 #define node_distance(from,to) (numa_slit[from * numnodes + to])
 
 extern int paddr_to_nid(unsigned long paddr);
include/asm-ia64/numnodes.h +7 -2
diff -Nurp --exclude-from=/home/mcd/.dontdiff linux-2.6.0-test5-arm/include/asm-ia64/numnodes.h linux-2.6.0-test5-ia64/include/asm-ia64/numnodes.h
@@ -1,7 +1,12 @@
 #ifndef _ASM_MAX_NUMNODES_H
 #define _ASM_MAX_NUMNODES_H
 
-#include <asm/mmzone.h>
-#define MAX_NUMNODES	NR_NODES
+#ifdef CONFIG_IA64_DIG
+/* Max 8 Nodes */
+#define NODES_SHIFT	3
+#elif defined(CONFIG_IA64_SGI_SN2)
+/* Max 128 Nodes */
+#define NODES_SHIFT	7
+#endif
 
 #endif /* _ASM_MAX_NUMNODES_H */
include/asm-ia64/sn/pda.h +2 -1
diff -Nurp --exclude-from=/home/mcd/.dontdiff linux-2.6.0-test5-arm/include/asm-ia64/sn/pda.h linux-2.6.0-test5-ia64/include/asm-ia64/sn/pda.h
@@ -10,6 +10,7 @@
 
 #include <linux/config.h>
 #include <linux/cache.h>
+#include <linux/mmzone.h>
 #include <asm/percpu.h>
 #include <asm/system.h>
 #include <asm/processor.h>

@@ -56,7 +57,7 @@ typedef struct pda_s {
 
 	unsigned long	sn_soft_irr[4];
 	unsigned long	sn_in_service_ivecs[4];
-	short		cnodeid_to_nasid_table[NR_NODES];	
+	short		cnodeid_to_nasid_table[MAX_NUMNODES];	
 	int		sn_lb_int_war_ticks;
 	int		sn_last_irq;
 	int		sn_first_irq;
#8 lore raw

Re: [PATCH] Clean up MAX_NR_NODES/NUMNODES/etc. [1/5]

In reply to Matthew Dobson

Here's a version rediffed against test5-mm2. The other patches, except for a little fuzz on 5/5, apply fine to either test5 or -mm2.

Cheers!

-Matt

include/asm-alpha/numnodes.h +2 -1
diff -Nurp --exclude-from=/home/mcd/.dontdiff linux-2.6.0-test5-mm2/include/asm-alpha/numnodes.h linux-2.6.0-test5-01/include/asm-alpha/numnodes.h
@@ -1,6 +1,7 @@
 #ifndef _ASM_MAX_NUMNODES_H
 #define _ASM_MAX_NUMNODES_H
 
-#define MAX_NUMNODES		128 /* Marvel */
+/* Max 128 Nodes - Marvel */
+#define NODES_SHIFT	7
 
 #endif /* _ASM_MAX_NUMNODES_H */
include/asm-i386/numaq.h +0 -1
diff -Nurp --exclude-from=/home/mcd/.dontdiff linux-2.6.0-test5-mm2/include/asm-i386/numaq.h linux-2.6.0-test5-01/include/asm-i386/numaq.h
@@ -28,7 +28,6 @@
 
 #ifdef CONFIG_X86_NUMAQ
 
-#define MAX_NUMNODES		16
 extern int get_memcfg_numaq(void);
 
 /*
include/asm-i386/numnodes.h +8 -4
diff -Nurp --exclude-from=/home/mcd/.dontdiff linux-2.6.0-test5-mm2/include/asm-i386/numnodes.h linux-2.6.0-test5-01/include/asm-i386/numnodes.h
@@ -4,11 +4,15 @@
 #include <linux/config.h>
 
 #ifdef CONFIG_X86_NUMAQ
-#include <asm/numaq.h>
+
+/* Max 16 Nodes */
+#define NODES_SHIFT	4
+
 #elif CONFIG_ACPI_SRAT
-#include <asm/srat.h>
-#else
-#define MAX_NUMNODES	1
+
+/* Max 8 Nodes */
+#define NODES_SHIFT	3
+
 #endif /* CONFIG_X86_NUMAQ */
 
 #endif /* _ASM_MAX_NUMNODES_H */
include/asm-i386/srat.h +0 -1
diff -Nurp --exclude-from=/home/mcd/.dontdiff linux-2.6.0-test5-mm2/include/asm-i386/srat.h linux-2.6.0-test5-01/include/asm-i386/srat.h
@@ -31,7 +31,6 @@
 #error CONFIG_ACPI_SRAT not defined, and srat.h header has been included
 #endif
 
-#define MAX_NUMNODES		8
 extern int get_memcfg_from_srat(void);
 extern unsigned long *get_zholes_size(int);
include/asm-ppc64/numnodes.h +2 -1
diff -Nurp --exclude-from=/home/mcd/.dontdiff linux-2.6.0-test5-mm2/include/asm-ppc64/numnodes.h linux-2.6.0-test5-01/include/asm-ppc64/numnodes.h
@@ -1,6 +1,7 @@
 #ifndef _ASM_MAX_NUMNODES_H
 #define _ASM_MAX_NUMNODES_H
 
-#define MAX_NUMNODES 16
+/* Max 16 Nodes */
+#define NODES_SHIFT	4
 
 #endif /* _ASM_MAX_NUMNODES_H */
include/asm-x86_64/mmzone.h +0 -1
diff -Nurp --exclude-from=/home/mcd/.dontdiff linux-2.6.0-test5-mm2/include/asm-x86_64/mmzone.h linux-2.6.0-test5-01/include/asm-x86_64/mmzone.h
@@ -10,7 +10,6 @@
 
 #define VIRTUAL_BUG_ON(x) 
 
-#include <asm/numnodes.h>
 #include <asm/smp.h>
 
 #define MAXNODE 8
include/asm-x86_64/numnodes.h +2 -5
diff -Nurp --exclude-from=/home/mcd/.dontdiff linux-2.6.0-test5-mm2/include/asm-x86_64/numnodes.h linux-2.6.0-test5-01/include/asm-x86_64/numnodes.h
@@ -3,10 +3,7 @@
 
 #include <linux/config.h>
 
-#ifdef CONFIG_DISCONTIGMEM
-#define MAX_NUMNODES 8	/* APIC limit currently */
-#else
-#define MAX_NUMNODES 1
-#endif
+/* Max 8 Nodes - APIC limit currently */
+#define NODES_SHIFT	3
 
 #endif
include/linux/mmzone.h +3 -2
diff -Nurp --exclude-from=/home/mcd/.dontdiff linux-2.6.0-test5-mm2/include/linux/mmzone.h linux-2.6.0-test5-01/include/linux/mmzone.h
@@ -14,9 +14,10 @@
 #ifdef CONFIG_DISCONTIGMEM
 #include <asm/numnodes.h>
 #endif
-#ifndef MAX_NUMNODES
-#define MAX_NUMNODES 1
+#ifndef NODES_SHIFT
+#define NODES_SHIFT	0
 #endif
+#define MAX_NUMNODES	(1 << NODES_SHIFT)
 
 /* Free memory management - zoned buddy allocator.  */
 #ifndef CONFIG_FORCE_MAX_ZONEORDER
mm/slab.c +1 -1
diff -Nurp --exclude-from=/home/mcd/.dontdiff linux-2.6.0-test5-mm2/mm/slab.c linux-2.6.0-test5-01/mm/slab.c
@@ -250,7 +250,7 @@ struct kmem_cache_s {
 	unsigned int		limit;
 /* 2) touched by every alloc & free from the backend */
 	struct kmem_list3	lists;
-	/* NUMA: kmem_3list_t	*nodelists[NR_NODES] */
+	/* NUMA: kmem_3list_t	*nodelists[MAX_NUMNODES] */
 	unsigned int		objsize;
 	unsigned int	 	flags;	/* constant flags */
 	unsigned int		num;	/* # of objs per slab */
author
#13 lore raw

Re: [PATCH] Clean up MAX_NR_NODES/NUMNODES/etc. [5/5]

In reply to Matthew Dobson
Quoted 19 lines · Matthew Dobson
Quoted 18 lines · Matthew Dobson

Ok, I made an attempt to clean up this mess quite a while ago (2.5.47), but that patch is utterly useless now. At Martin's urging I've created a new series of patches to resolve this.

01 - Make sure MAX_NUMNODES is defined in one and only one place. Remove superfluous definitions. Instead of defining MAX_NUMNODES in asm/numnodes.h, we define NODES_SHIFT there. Then in linux/mmzone.h we turn that NODES_SHIFT value into MAX_NUMNODES.

02 - Remove MAX_NR_NODES. This value is only used in a couple of places, and it's incorrectly used in all those places as far as I can tell. Replace with MAX_NUMNODES. Create MAX_NODES_SHIFT and use this value to check NODES_SHIFT is appropriate. A possible future patch should make MAX_NODES_SHIFT vary based on 32 vs. 64 bit archs.

03 - Fix up the sh arch. sh defined NR_NODES, change sh to use standard MAX_NUMNODES instead.

Quoted 6 lines
Quoted 6 lines

04 - Fix up the arm arch. This needs to be reviewed. Relatively straightforward replacement of NR_NODES with standard MAX_NUMNODES.

05 - Fix up the ia64 arch. This *definitely* needs to be reviewed. This code made my head hurt. I think I may have gotten it right. Totally untested.

Can you send me a patch that contains everything (or just the generic code plus the ia64 stuff)? The stuff you posted looks good, and I'd like to test ia64, but I have to merge your patch into the latest discontig stuff I've been working on to do so.

Thanks,
Jesse

#14 lore raw
In reply to Matthew Dobson

Andrew,
Here's a small update to the numnodes fix that went into -mm3. The
biggest changes are:
1) move the actual NODES_SHIFT and MAX_NUMNODES definitions into
linux/numa.h and include this in linux/mmzone.h, instead of being directly in linux/mmzone.h. This allows other files to include *just* the NUMNODES stuff w/out grabbing all of mmzone.h.
2) pull NODE_SHIFT out of linux/mm.h. This isn't used anywhere in the
kernel, and it will only get confused with NODES_SHIFT.
3) Fix the IA64 patch. The original patch I had sent out hadn't been
tested on IA64. It was mostly right, but there were circular dependencies. All better now, and acked by Jesse.
4) In linux/mmzone.h, insert code to define MAX_NODES_SHIFT based on
the size of unsigned long. For 64-bit arches, we can have a much larger value. This allows IA64 to have 100's or 1000's of nodes. MAX_NODES_SHIFT is defined as 10 (ie: 1024 nodes) for 64-bit for now, although it could likely be much larger. For 32-bit it is 6 (ie: 64 nodes).
5) Small cleanup in include/asm-arm/memory.h. Mostly the result of the
new linux/numa.h file. Much cleaner and more readable now.

Russell, if you get a chance, I'd really appreciate a sanity check on the arm code. It really hasn't been tested, but the changes are pretty small.

Cheers!

-Matt

include/asm-arm/memory.h +6 -11
diff -Nurp --exclude-from=/home/mcd/.dontdiff linux-2.6.0-test5-mm3/include/asm-arm/memory.h linux-2.6.0-test5-numnodes_update/include/asm-arm/memory.h
@@ -84,27 +84,24 @@ static inline void *phys_to_virt(unsigne
 
 #define PHYS_TO_NID(addr)	(0)
 
-#else
+#else /* CONFIG_DISCONTIGMEM */
+
 /*
  * This is more complex.  We have a set of mem_map arrays spread
  * around in memory.
  */
-#include <asm/numnodes.h>
-#define NUM_NODES	(1 << NODES_SHIFT)
+#include <linux/numa.h>
 
 #define page_to_pfn(page)					\
 	(( (page) - page_zone(page)->zone_mem_map)		\
 	  + page_zone(page)->zone_start_pfn)
-
 #define pfn_to_page(pfn)					\
 	(PFN_TO_MAPBASE(pfn) + LOCAL_MAP_NR((pfn) << PAGE_SHIFT))
-
-#define pfn_valid(pfn)		(PFN_TO_NID(pfn) < NUM_NODES)
+#define pfn_valid(pfn)		(PFN_TO_NID(pfn) < MAX_NUMNODES)
 
 #define virt_to_page(kaddr)					\
 	(ADDR_TO_MAPBASE(kaddr) + LOCAL_MAP_NR(kaddr))
-
-#define virt_addr_valid(kaddr)	(KVADDR_TO_NID(kaddr) < NUM_NODES)
+#define virt_addr_valid(kaddr)	(KVADDR_TO_NID(kaddr) < MAX_NUMNODES)
 
 /*
  * Common discontigmem stuff.

@@ -112,9 +109,7 @@ static inline void *phys_to_virt(unsigne
  */
 #define PHYS_TO_NID(addr)	PFN_TO_NID((addr) >> PAGE_SHIFT)
 
-#undef NUM_NODES
-
-#endif
+#endif /* !CONFIG_DISCONTIGMEM */
 
 /*
  * For BIO.  "will die".  Kill me when bio_to_phys() and bvec_to_phys() die.
include/asm-ia64/nodedata.h +3 -5
diff -Nurp --exclude-from=/home/mcd/.dontdiff linux-2.6.0-test5-mm3/include/asm-ia64/nodedata.h linux-2.6.0-test5-numnodes_update/include/asm-ia64/nodedata.h
@@ -8,13 +8,11 @@
  * Copyright (c) 2002 Erich Focht <[email protected]>
  * Copyright (c) 2002 Kimio Suganuma <[email protected]>
  */
-
-
 #ifndef _ASM_IA64_NODEDATA_H
 #define _ASM_IA64_NODEDATA_H
 
-
-#include <linux/mmzone.h>
+#include <linux/numa.h>
+#include <asm/mmzone.h>
 
 /*
  * Node Data. One of these structures is located on each node of a NUMA system.

@@ -24,7 +22,7 @@ struct pglist_data;
 struct ia64_node_data {
 	short			active_cpu_count;
 	short			node;
-        struct pglist_data	*pg_data_ptrs[MAX_NUMNODES];
+	struct pglist_data	*pg_data_ptrs[MAX_NUMNODES];
 	struct page		*bank_mem_map_base[NR_BANKS];
 	struct ia64_node_data	*node_data_ptrs[MAX_NUMNODES];
 	short			node_id_map[NR_BANKS];
include/asm-ia64/numa.h +2 -2
diff -Nurp --exclude-from=/home/mcd/.dontdiff linux-2.6.0-test5-mm3/include/asm-ia64/numa.h linux-2.6.0-test5-numnodes_update/include/asm-ia64/numa.h
@@ -13,9 +13,9 @@
 
 #ifdef CONFIG_NUMA
 
-#include <linux/mmzone.h>
-
+#include <linux/numa.h>
 #include <linux/cache.h>
+
 extern volatile char cpu_to_node_map[NR_CPUS] __cacheline_aligned;
 extern volatile unsigned long node_to_cpu_mask[MAX_NUMNODES] __cacheline_aligned;
include/asm-ia64/sn/pda.h +1 -1
diff -Nurp --exclude-from=/home/mcd/.dontdiff linux-2.6.0-test5-mm3/include/asm-ia64/sn/pda.h linux-2.6.0-test5-numnodes_update/include/asm-ia64/sn/pda.h
@@ -10,7 +10,7 @@
 
 #include <linux/config.h>
 #include <linux/cache.h>
-#include <linux/mmzone.h>
+#include <linux/numa.h>
 #include <asm/percpu.h>
 #include <asm/system.h>
 #include <asm/processor.h>
include/linux/mm.h +0 -1
diff -Nurp --exclude-from=/home/mcd/.dontdiff linux-2.6.0-test5-mm3/include/linux/mm.h linux-2.6.0-test5-numnodes_update/include/linux/mm.h
@@ -323,7 +323,6 @@ static inline void put_page(struct page 
  * The zone field is never updated after free_area_init_core()
  * sets it, so none of the operations on it need to be atomic.
  */
-#define NODE_SHIFT 4
 #define ZONE_SHIFT (BITS_PER_LONG - 8)
 
 struct zone;
include/linux/mmzone.h +11 -10
diff -Nurp --exclude-from=/home/mcd/.dontdiff linux-2.6.0-test5-mm3/include/linux/mmzone.h linux-2.6.0-test5-numnodes_update/include/linux/mmzone.h
@@ -10,14 +10,8 @@
 #include <linux/wait.h>
 #include <linux/cache.h>
 #include <linux/threads.h>
+#include <linux/numa.h>
 #include <asm/atomic.h>
-#ifdef CONFIG_DISCONTIGMEM
-#include <asm/numnodes.h>
-#endif
-#ifndef NODES_SHIFT
-#define NODES_SHIFT	0
-#endif
-#define MAX_NUMNODES	(1 << NODES_SHIFT)
 
 /* Free memory management - zoned buddy allocator.  */
 #ifndef CONFIG_FORCE_MAX_ZONEORDER

@@ -313,12 +307,19 @@ extern struct pglist_data contig_page_da
 #else /* CONFIG_DISCONTIGMEM */
 
 #include <asm/mmzone.h>
+
+#if BITS_PER_LONG == 32
 /*
- * page->zone is currently 8 bits
- * there are 3 zones (2 bits)
- * this leaves 8-2=6 bits for nodes
+ * with 32 bit flags field, page->zone is currently 8 bits.
+ * there are 3 zones (2 bits) and this leaves 8-2=6 bits for nodes.
  */
 #define MAX_NODES_SHIFT		6
+#elif BITS_PER_LONG == 64
+/*
+ * with 64 bit flags field, there's plenty of room.
+ */
+#define MAX_NODES_SHIFT		10
+#endif
 
 #endif /* !CONFIG_DISCONTIGMEM */
include/linux/numa.h +16 -0
diff -Nurp --exclude-from=/home/mcd/.dontdiff linux-2.6.0-test5-mm3/include/linux/numa.h linux-2.6.0-test5-numnodes_update/include/linux/numa.h
@@ -0,0 +1,16 @@
+#ifndef _LINUX_NUMA_H
+#define _LINUX_NUMA_H
+
+#include <linux/config.h>
+
+#ifdef CONFIG_DISCONTIGMEM
+#include <asm/numnodes.h>
+#endif
+
+#ifndef NODES_SHIFT
+#define NODES_SHIFT     0
+#endif
+
+#define MAX_NUMNODES    (1 << NODES_SHIFT)
+
+#endif /* _LINUX_NUMA_H */
author
#15 lore raw

I just tried -test8-mm1 on a 256p/128 node machine and these changes almost work. I still needed to change ZONE_SHIFT from BITS_PER_LONG - 8 to BITS_PER_LONG - 10 because of the way set_page_zone() is called from memmap_init_zone() (set_page_zone(page, nid * MAX_NR_ZONES + zone);). So shouldn't ZONE_SHIFT be BITS_PER_LONG - NODES_SHIFT or something?

Jesse

Quoted 23 lines · Matthew Dobson

Andrew,
Here's a small update to the numnodes fix that went into -mm3. The
biggest changes are:
1) move the actual NODES_SHIFT and MAX_NUMNODES definitions into
linux/numa.h and include this in linux/mmzone.h, instead of being directly in linux/mmzone.h. This allows other files to include *just* the NUMNODES stuff w/out grabbing all of mmzone.h.
2) pull NODE_SHIFT out of linux/mm.h. This isn't used anywhere in the kernel, and it will only get confused with NODES_SHIFT.
3) Fix the IA64 patch. The original patch I had sent out hadn't been tested on IA64. It was mostly right, but there were circular
dependencies. All better now, and acked by Jesse.
4) In linux/mmzone.h, insert code to define MAX_NODES_SHIFT based on
the size of unsigned long. For 64-bit arches, we can have a much larger value. This allows IA64 to have 100's or 1000's of nodes. MAX_NODES_SHIFT is defined as 10 (ie: 1024 nodes) for 64-bit for now, although it could likely be much larger. For 32-bit it is 6 (ie: 64 nodes).
5) Small cleanup in include/asm-arm/memory.h. Mostly the result of the new linux/numa.h file. Much cleaner and more readable now.

Russell, if you get a chance, I'd really appreciate a sanity check on the arm code. It really hasn't been tested, but the changes are pretty small.