Sizing zones and holes in an architecture independent manner
Subsystems (5)
cover Sizing zones and holes in an architecture independent manner
(The TO: list are the architecture maintainers according to the MAINTAINERS. Apologies in advance if I got the list wrong)
At a basic level, architectures define structures to record where active ranges of page frames are located. Once located, the code to calculate zone sizes and holes in each architecture is very similar. Some of this zone and hole sizing code is difficult to read for no good reason. This set of patches eliminates the similar-looking architecture-specific code.
The patches introduce a mechanism where architectures register where the active ranges of page frames are with add_active_range(). When all areas have been discovered, free_area_init_nodes() is called to initialise the pgdat and zones. The zone sizes and holes are then calculated in an architecture independent manner.
Patch 1 introduces the mechanism for registering and initialising PFN ranges Patch 2 changes ppc to use the mechanism - 136 arch-specific LOC removed Patch 3 changes x86 to use the mechanism - 150 arch-specific LOC removed Patch 4 changes x86_64 to use the mechanism - 35 arch-specific LOC removed Patch 5 changes ia64 to use the mechanism - 59 arch-specific LOC removed
At this point, there is a net reduction of 27 lines of code and the arch-independent code is a lot easier to read in comparison to some of the arch-specific stuff, particularly in arch/i386/ .
For Patch 6, it was also noted that page_alloc.c has a *lot* of initialisation code which makes the file harder to read than it needs to be. Patch 6 creates a new file mem_init.c and moves a lot of initialisation code from page_alloc.c to it. After the patch is applied, there is still a net reduction of 3 lines of code.
The patches have been successfully boot tested on
o x86, flatmem
o x86, NUMAQ
o PPC64, NUMA
o PPC64, CONFIG_NUMA=n
o x86_64, NUMA with SRAT
The patches have only been *compile tested* for ia64 with a flatmem configuration. At attempt was made to boot test on an ancient RS/6000 but the vanilla kernel does not boot so I have to investigate there.
The net reduction seems small but the big benefit of this set of patches is the reduction of 380 lines of architecture-specific code, some of which is very hairy. There should be a greater net reduction when other architectures use the same mechanisms for zone and hole sizing but I lack the hardware to test on.
Comments?
Additional credit;
Dave Hansen for the initial suggestion and comments on early patches Andy Whitcroft for reviewing early versions and catching numerous errors
arch/i386/Kconfig | 8 arch/i386/kernel/setup.c | 19 arch/i386/kernel/srat.c | 98 ---- arch/i386/mm/discontig.c | 59 -- arch/ia64/Kconfig | 3 arch/ia64/mm/contig.c | 62 -- arch/ia64/mm/discontig.c | 43 - arch/ia64/mm/init.c | 10 arch/powerpc/Kconfig | 13 arch/powerpc/mm/mem.c | 50 -- arch/powerpc/mm/numa.c | 157 ------ arch/ppc/Kconfig | 3 arch/ppc/mm/init.c | 21 arch/x86_64/Kconfig | 3 arch/x86_64/kernel/e820.c | 18 arch/x86_64/mm/init.c | 60 -- arch/x86_64/mm/numa.c | 15 include/asm-ia64/meminit.h | 1 include/asm-x86_64/e820.h | 1 include/asm-x86_64/proto.h | 2 include/linux/mm.h | 14 include/linux/mmzone.h | 15 mm/Makefile | 2 mm/mem_init.c | 1028 +++++++++++++++++++++++++++++++++++++++++++++ mm/page_alloc.c | 678 ----------------------------- 25 files changed, 1190 insertions(+), 1193 deletions(-)
Signature · 4 lines
-- Mel Gorman Part-time Phd Student Linux Technology Center University of Limerick IBM Dublin Software Lab
PATCH 1/6 Introduce mechanism for registering active regions of memory
This patch defines the structure to represent an active range of page frames within a node in an architecture independent manner. Architectures are expected to register active ranges of PFNs using add_active_range(nid, start_pfn, end_pfn) and call free_area_init_nodes() passing the PFNs of the end of each zone.
include/linux/mm.h | 14 + include/linux/mmzone.h | 15 + mm/page_alloc.c | 374 +++++++++++++++++++++++++++++++++++++++++--- 3 files changed, 378 insertions(+), 25 deletions(-)
include/linux/mm.h +14 -0
@@ -867,6 +867,20 @@ extern void free_area_init(unsigned longextern void free_area_init_node (int nid,pg_data_t *pgdat,unsigned long * zones_size,unsigned long zone_start_pfn,unsigned long *zholes_size); +#ifdef CONFIG_ARCH_POPULATES_NODE_MAP +extern void free_area_init_nodes (unsigned long max_dma_pfn, +unsigned long max_dma32_pfn, +unsigned long max_low_pfn, +unsigned long max_high_pfn); +extern void add_active_range (unsigned int nid,unsigned long start_pfn, +unsigned long end_pfn); +extern void get_pfn_range_for_nid (unsigned int nid, +unsigned long *start_pfn,unsigned long *end_pfn); +extern int early_pfn_to_nid (unsigned long pfn); +extern void free_bootmem_with_active_regions (int nid, +unsigned long max_low_pfn); +extern void memory_present_with_active_regions (int nid); +#endif extern void memmap_init_zone (unsigned long ,int ,unsigned long ,unsigned long );extern void setup_per_zone_pages_min (void );extern void mem_init (void );
include/linux/mmzone.h +14 -1
@@ -271,6 +271,18 @@ struct zonelist {struct zone *zones[MAX_NUMNODES *MAX_NR_ZONES +1 ];// NULL delimited }; +#ifdef CONFIG_ARCH_POPULATES_NODE_MAP +/* +* This represents an active range of physical memory. Architectures register +* a pfn range using add_active_range() and later initialise the nodes and +* free list with free_area_init_nodes() +*/ +struct node_active_region { +unsigned long start_pfn; +unsigned long end_pfn; +int nid; +}; +#endif /* CONFIG_ARCH_POPULATES_NODE_MAP */ /* * The pg_data_t structure is used in machines with CONFIG_DISCONTIGMEM
@@ -465,7 +477,8 @@ extern struct zone *next_zone(struct zon#endif -#ifndef CONFIG_HAVE_ARCH_EARLY_PFN_TO_NID +#if !defined(CONFIG_HAVE_ARCH_EARLY_PFN_TO_NID ) && \ + !defined(CONFIG_ARCH_POPULATES_NODE_MAP )#define early_pfn_to_nid (nid) (0UL)#endif
mm/page_alloc.c +350 -24
416 lines not on this page · raw message
PATCH 2/6 Have Power use add_active_range() and free_area_init_nodes()
In reply to Mel GormanSize zones and holes in an architecture independent manner for Power.
This has been boot tested on PPC64 with NUMA both enabled and disabled. It has been compile tested for an older CHRP-based machine.
powerpc/Kconfig | 13 ++-- powerpc/mm/mem.c | 50 +++++---------- powerpc/mm/numa.c | 157 ++++--------------------------------------------- ppc/Kconfig | 3 ppc/mm/init.c | 21 +++--- 5 files changed, 54 insertions(+), 190 deletions(-)
arch/powerpc/Kconfig +9 -4
@@ -665,11 +665,16 @@ config ARCH_SPARSEMEM_DEFAULT def_bool y depends on SMP && PPC_PSERIES -source "mm/Kconfig" - -config HAVE_ARCH_EARLY_PFN_TO_NID +config ARCH_POPULATES_NODE_MAP def_bool y - depends on NEED_MULTIPLE_NODES + +# Value of 256 is MAX_LMB_REGIONS * 2 +config MAX_ACTIVE_REGIONS + int + default 256 + depends on ARCH_POPULATES_NODE_MAP + +source "mm/Kconfig" config ARCH_MEMORY_PROBE def_bool y
arch/powerpc/mm/mem.c +18 -32
82 lines not on this page · raw message
arch/powerpc/mm/numa.c +14 -143
207 lines not on this page · raw message
arch/ppc/Kconfig +3 -0
@@ -949,6 +949,9 @@ config NR_CPUS config HIGHMEM bool "High memory support" +config ARCH_POPULATES_NODE_MAP + def_bool y + source kernel/Kconfig.hz source kernel/Kconfig.preempt source "mm/Kconfig"
arch/ppc/mm/init.c +10 -11
@@ -359,8 +359,7 @@ void __init do_init_bootmem(void)*/ void __init paging_init (void ) { -unsigned long zones_size[MAX_NR_ZONES ], i; - +unsigned long start_pfn, end_pfn;#ifdef CONFIG_HIGHMEM map_page (PKMAP_BASE ,0 ,0 );/* XXX gross */ pkmap_page_table = pte_offset_kernel(pmd_offset(pgd_offset_k
@@ -371,18 +370,18 @@ void __init paging_init(void) kmap_prot =PAGE_KERNEL ;#endif /* CONFIG_HIGHMEM */ -/* -* All pages are DMA-able so we put them all in the DMA zone. -*/ - zones_size[ZONE_DMA ] = total_lowmem >>PAGE_SHIFT ; -for (i =1 ; i <MAX_NR_ZONES ; i++) - zones_size[i] =0 ; +/* All pages are DMA-able so we put them all in the DMA zone. */ + start_pfn =__pa (PAGE_OFFSET ) >>PAGE_SHIFT ; + end_pfn = start_pfn + (total_memory >>PAGE_SHIFT ); +add_active_range (0 , start_pfn, end_pfn);#ifdef CONFIG_HIGHMEM - zones_size[ZONE_HIGHMEM ] = (total_memory - total_lowmem) >>PAGE_SHIFT ; +free_area_init_nodes (total_lowmem, total_lowmem, + total_lowmem, total_memory); +#else +free_area_init_nodes (total_memory, total_memory, + total_memory, total_memory);#endif /* CONFIG_HIGHMEM */ - -free_area_init (zones_size); }void __init mem_init (void )
PATCH 3/6 Have x86 use add_active_range() and free_area_init_nodes
In reply to Mel GormanSize zones and holes in an architecture independent manner for x86.
This has been boot tested on;
x86 with 4 CPUs, flatmem
x86 on NUMAQ
It needs to be boot tested on an x86 machine that uses SRAT.
Kconfig | 8 +--- kernel/setup.c | 19 +++------- kernel/srat.c | 98 ---------------------------------------------------- mm/discontig.c | 59 ++++--------------------------- 4 files changed, 17 insertions(+), 167 deletions(-)
arch/i386/Kconfig +3 -5
@@ -563,12 +563,10 @@ config ARCH_SELECT_MEMORY_MODEL def_bool y depends on ARCH_SPARSEMEM_ENABLE -source "mm/Kconfig" +config ARCH_POPULATES_NODE_MAP + def_bool y -config HAVE_ARCH_EARLY_PFN_TO_NID - bool - default y - depends on NUMA +source "mm/Kconfig" config HIGHPTE bool "Allocate 3rd-level pagetables from highmem"
arch/i386/kernel/setup.c +6 -13
@@ -1156,22 +1156,15 @@ static unsigned long __init setup_memoryvoid __init zone_sizes_init (void ) { -unsigned long zones_size[MAX_NR_ZONES ] = {0 ,0 ,0 }; -unsigned int max_dma, low; +unsigned int max_dma; +#ifndef CONFIG_HIGHMEM +unsigned long highend_pfn = max_low_pfn; +#endif max_dma =virt_to_phys ((char *)MAX_DMA_ADDRESS ) >>PAGE_SHIFT ; - low = max_low_pfn; -if (low < max_dma) - zones_size[ZONE_DMA ] = low; -else { - zones_size[ZONE_DMA ] = max_dma; - zones_size[ZONE_NORMAL ] = low - max_dma; -#ifdef CONFIG_HIGHMEM - zones_size[ZONE_HIGHMEM ] = highend_pfn - low; -#endif - } -free_area_init (zones_size); +add_active_range (0 ,0 , highend_pfn); +free_area_init_nodes (max_dma, max_dma, max_low_pfn, highend_pfn); }#else extern unsigned long __init setup_memory (void );
arch/i386/kernel/srat.c +0 -98
119 lines not on this page · raw message
arch/i386/mm/discontig.c +8 -51
74 lines not on this page · raw message
PATCH 4/6 Have x86_64 use add_active_range() and free_area_init_nodes
In reply to Mel GormanSize zones and holes in an architecture independent manner for x86_64.
This has only been boot tested on an x86_64 with NUMA and SRAT.
arch/x86_64/Kconfig | 3 ++ arch/x86_64/kernel/e820.c | 18 ++++++++++++ arch/x86_64/mm/init.c | 60 +--------------------------------------- arch/x86_64/mm/numa.c | 15 +++++----- include/asm-x86_64/e820.h | 1 include/asm-x86_64/proto.h | 2 - 6 files changed, 32 insertions(+), 67 deletions(-)
arch/x86_64/Kconfig +3 -0
@@ -73,6 +73,9 @@ config ARCH_MAY_HAVE_PC_FDC bool default y +config ARCH_POPULATES_NODE_MAP + def_bool y + config DMI bool default y
arch/x86_64/kernel/e820.c +18 -0
@@ -220,6 +220,24 @@ e820_hole_size(unsigned long start_pfn,return ((end - start) - ram) >>PAGE_SHIFT ; } +/* Walk the e820 map and register active regions */ +unsigned long __init +e820_register_active_regions (void ) +{ +int i; +unsigned long start_pfn, end_pfn; +for (i =0 ; i < e820.nr_map; i++) { +struct e820entry *ei = &e820.map[i]; +if (ei->type !=E820_RAM ) +continue ; + + start_pfn =round_up (ei->addr,PAGE_SIZE ); + end_pfn =round_down (ei->addr + ei->size,PAGE_SIZE ); + +add_active_range (0 , start_pfn, end_pfn); + } +} +/* * Mark e820 reserved areas as busy for the resource manager. */
arch/x86_64/mm/init.c +2 -58
71 lines not on this page · raw message
arch/x86_64/mm/numa.c +8 -7
@@ -149,13 +149,12 @@ void __init setup_node_bootmem(int nodeivoid __init setup_node_zones (int nodeid) {unsigned long start_pfn, end_pfn, memmapsize, limit; -unsigned long zones[MAX_NR_ZONES ]; -unsigned long holes[MAX_NR_ZONES ]; start_pfn =node_start_pfn (nodeid); end_pfn =node_end_pfn (nodeid); +add_active_range (nodeid, start_pfn, end_pfn); -Dprintk (KERN_INFO "Setting up node %d %lx-%lx\n" , +Dprintk (KERN_INFO "Setting up memmap for node %d %lx-%lx\n" , nodeid, start_pfn, end_pfn);/* Try to allocate mem_map at end to not fill up precious <4GB
@@ -167,10 +166,6 @@ void __init setup_node_zones(int nodeid) memmapsize,SMP_CACHE_BYTES ,round_down (limit - memmapsize,PAGE_SIZE ), limit); - -size_zones (zones, holes, start_pfn, end_pfn); -free_area_init_node (nodeid,NODE_DATA (nodeid), zones, - start_pfn, holes); }void __init numa_init_array (void )
@@ -312,12 +307,18 @@ static void __init arch_sparse_init(voidvoid __init paging_init (void ) {int i; +unsigned long max_normal_pfn =0 ;arch_sparse_init ();for_each_online_node (i) {setup_node_zones (i); +if (max_normal_pfn <node_end_pfn (i)) + max_normal_pfn =node_end_pfn (i); } + +free_area_init_nodes (MAX_DMA_PFN ,MAX_DMA32_PFN , max_normal_pfn, + max_normal_pfn); }/* [numa=off] */
include/asm-x86_64/e820.h +1 -0
@@ -53,6 +53,7 @@ extern void e820_bootmem_free(pg_data_textern void e820_setup_gap (void );extern unsigned long e820_hole_size (unsigned long start_pfn,unsigned long end_pfn); +extern unsigned long e820_register_active_regions (void );extern void __init parse_memopt (char *p,char **end);extern void __init parse_memmapopt (char *p,char **end);
include/asm-x86_64/proto.h +0 -2
@@ -24,8 +24,6 @@ extern void mtrr_bp_init(void);#define mtrr_bp_init () do {} while (0)#endif extern void init_memory_mapping (unsigned long start,unsigned long end); -extern void size_zones (unsigned long *z,unsigned long *h, -unsigned long start_pfn,unsigned long end_pfn);extern void system_call (void );extern int kernel_syscall (void );
PATCH 5/6 Have ia64 use add_active_range() and free_area_init_nodes
In reply to Mel GormanSize zones and holes in an architecture independent manner for ia64.
This has only been compile-tested due to lack of a suitable test machine.
arch/ia64/Kconfig | 3 + arch/ia64/mm/contig.c | 62 +++++----------------------------------- arch/ia64/mm/discontig.c | 43 +++++---------------------- arch/ia64/mm/init.c | 10 ++++++ include/asm-ia64/meminit.h | 1 5 files changed, 30 insertions(+), 89 deletions(-)
arch/ia64/Kconfig +3 -0
@@ -352,6 +352,9 @@ config NUMA Access). This option is for configuring high-end multiprocessor server systems. If in doubt, say N. +config ARCH_POPULATES_NODE_MAP + def_bool y + # VIRTUAL_MEM_MAP and FLAT_NODE_MEM_MAP are functionally equivalent. # VIRTUAL_MEM_MAP has been retained for historical reasons. config VIRTUAL_MEM_MAP
arch/ia64/mm/contig.c +8 -54
107 lines not on this page · raw message
arch/ia64/mm/discontig.c +8 -35
68 lines not on this page · raw message
arch/ia64/mm/init.c +10 -0
@@ -526,6 +526,16 @@ ia64_pfn_valid (unsigned long pfn)EXPORT_SYMBOL (ia64_pfn_valid);int __init +register_active_ranges (u64 start,u64 end,void *nid) +{ +BUG_ON (nid ==NULL ); +BUG_ON (*(unsigned long *)nid >=MAX_NUMNODES ); + +add_active_range (*(unsigned long *)nid, start, end); +return 0 ; +} + +int __init find_largest_hole (u64 start,u64 end,void *arg) {u64 *max_gap = arg;
include/asm-ia64/meminit.h +1 -0
@@ -56,6 +56,7 @@ extern void efi_memmap_init(unsigned lonextern unsigned long vmalloc_end;extern struct page *vmem_map;extern int find_largest_hole (u64 start,u64 end,void *arg); +extern int register_active_ranges (u64 start,u64 end,void *arg);extern int create_mem_map_page_table (u64 start,u64 end,void *arg);#endif
PATCH 6/6 Break out memory initialisation code from page_alloc.c to mem_init.c
In reply to Mel Gormanpage_alloc.c contains a large amount of memory initialisation code. This patch breaks out the initialisation code to a separate file to make page_alloc.c a bit easier to read.
Makefile | 2 mem_init.c | 1028 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ page_alloc.c | 1004 ---------------------------------------------------- 3 files changed, 1029 insertions(+), 1005 deletions(-)
mm/Makefile +1 -1
@@ -8,7 +8,7 @@ mmu-$(CONFIG_MMU) := fremap.o highmem.o vmalloc.o obj-y := bootmem.o filemap.o mempool.o oom_kill.o fadvise.o \ - page_alloc.o page-writeback.o pdflush.o \ + page_alloc.o mem_init.o page-writeback.o pdflush.o \ readahead.o swap.o truncate.o vmscan.o \ prio_tree.o util.o mmzone.o $(mmu-y)
mm/mem_init.c +1028 -0
1,028 lines not on this page · raw message
mm/page_alloc.c +0 -1004
1,039 lines not on this page · raw message
Re: [PATCH 6/6] Break out memory initialisation code from page_alloc.c to mem_init.c
Quoted 4 lines · Mel Gorman
page_alloc.ccontains a large amount of memory initialisation code. This patch breaks out the initialisation code to a separate file to makepage_alloc.ca bit easier to read.
Seems like a very good idea to me.
Quoted 11 lines
+/*
+ *mm/mem_init.c
+ * Initialises the architecture independant view of memory. pgdats, zones, etc
+ *+ * Copyright (C) 1991, 1992, 1993, 1994 Linus Torvalds + * Swap reorganised 29.12.95, Stephen Tweedie + * Support of BIGMEM added by Gerhard Wichert, Siemens AG, July 1999 + * Reshaped it to be a zoned allocator, Ingo Molnar, Red Hat, 1999 + * Discontiguous memory support, Kanoj Sarcar, SGI, Nov 1999 + * Zone balancing, Kanoj Sarcar, SGI, Jan 2000 + * Per cpu hot/cold page lists, bulk allocation, Martin J. Bligh, Sept 2002
Quoted 4 lines
+ * (lots of bits borrowed from Ingo Molnar & Andrew Morton)
+ * Arch-independant zone size and hole calculation, Mel Gorman, IBM, Apr 2006
+ * (lots of bits taken from architecture code)
+ */
Maybe drop the duplicated changelog? (just retain copyrights I guess)
Signature · 2 lines
SUSE Labs, Novell Inc. Send instant messages to your online friends http://au.messenger.yahoo.com
Re: [PATCH 6/6] Break out memory initialisation code from page_alloc.c to mem_init.c
Quoted 8 lines · Nick Piggin
Quoted 4 lines · Mel Gorman
page_alloc.ccontains a large amount of memory initialisation code. This patch breaks out the initialisation code to a separate file to makepage_alloc.ca bit easier to read.Seems like a very good idea to me.
If there is interest in treating this separetly, it can be broken out as a standalone patch. In this form, it depends on the first patch from the set.
Quoted 19 lines
Quoted 12 lines
+/*
+ *mm/mem_init.c
+ * Initialises the architecture independant view of memory. pgdats, zones, etc
+ *+ * Copyright (C) 1991, 1992, 1993, 1994 Linus Torvalds + * Swap reorganised 29.12.95, Stephen Tweedie + * Support of BIGMEM added by Gerhard Wichert, Siemens AG, July 1999 + * Reshaped it to be a zoned allocator, Ingo Molnar, Red Hat, 1999 + * Discontiguous memory support, Kanoj Sarcar, SGI, Nov 1999 + * Zone balancing, Kanoj Sarcar, SGI, Jan 2000 + * Per cpu hot/cold page lists, bulk allocation, Martin J. Bligh, SeptQuoted 6 lines
2002
+ * (lots of bits borrowed from Ingo Molnar & Andrew Morton)
+ * Arch-independant zone size and hole calculation, Mel Gorman, IBM, Apr 2006
+ * (lots of bits taken from architecture code)
+ */
Quoted 2 lines
Maybe drop the duplicated changelog? (just retain copyrights I guess)
Makes sense.
+ * Copyright (C) 1991, 1992, 1993, 1994 Linus Torvalds + * Copyright (C) 1995, Stephen Tweedie + * Copyright (C) July 1999, Gerhard Wichert, Siemens AG + * Copyright (C) 1999, Ingo Molnar, Red Hat + * Copyright (C) 1999, 2000, Kanoj Sarcar, SGI + * Copyright (C) Sept 2000, Martin J. Bligh
+ * (lots of bits borrowed from Ingo Molnar & Andrew Morton)
+ * Copyright (C) Apr 2006, Mel Gorman, IBM
+ * (lots of bits taken from architecture-specific code)
Signature · 3 lines
Mel Gorman Part-time Phd Student Linux Technology Center University of Limerick IBM Dublin Software Lab
Quoted 3 lines · Mel Gorman
The patches have only been *compile tested* for ia64 with a flatmem configuration. At attempt was made to boot test on an ancient RS/6000 but the vanilla kernel does not boot so I have to investigate there.
The good news: Compilation is clean on the ia64 config variants that I usually build (all 10 of them).
The bad (or at least consistent) news: It doesn't boot on an Intel Tiger either (oops at kmem_cache_alloc+0x41).
-Tony
Quoted 9 lines · Tony
Quoted 3 lines · Mel Gorman
The patches have only been *compile tested* for ia64 with a flatmem configuration. At attempt was made to boot test on an ancient RS/6000 but the vanilla kernel does not boot so I have to investigate there.
The good news: Compilation is clean on the ia64 config variants that I usually build (all 10 of them).
One plus at least.
Quoted 3 lines
The bad (or at least consistent) news: It doesn't boot on an Intel Tiger either (oops at
kmem_cache_alloc+0x41).
Darn.
o Did it boot on other IA64 machines or was the Tiger the first boot failure?
o Possibly a stupid question but does the Tiger configuration use the flatmem memory model, sparsemem or discontig?
If it's flatmem, I noticed I made a stupid mistake where vmem_map is not getting set to (void *)0 for machines with small memory holes. Nothing else really obvious jumped out at me.
I've attached a patch called "105-ia64_use_init_nodes.patch". Can you reverse Patch 5/6 and apply this one instead please? I've also attached 107-debug.diff that applies on top of patch 6/6. It just prints out debugging information during startup that may tell me where I went wrong in arch/ia64. I'd really appreciate it if you could use both patches, let me know if it still fails to boot and send me the console log of the machine starting up if it fails so I can make guesses as to what is going wrong.
Thanks a lot for trying the patches out on ia64. It was the one arch of the set I had no chance to test with at all :/
Signature · 3 lines
Mel Gorman Part-time Phd Student Linux Technology Center University of Limerick IBM Dublin Software Lab
arch/ia64/Kconfig +3 -0
@@ -352,6 +352,9 @@ config NUMA Access). This option is for configuring high-end multiprocessor server systems. If in doubt, say N. +config ARCH_POPULATES_NODE_MAP + def_bool y + # VIRTUAL_MEM_MAP and FLAT_NODE_MEM_MAP are functionally equivalent. # VIRTUAL_MEM_MAP has been retained for historical reasons. config VIRTUAL_MEM_MAP
arch/ia64/mm/contig.c +8 -52
102 lines not on this page · raw message
arch/ia64/mm/discontig.c +6 -35
67 lines not on this page · raw message
arch/ia64/mm/init.c +10 -0
@@ -539,6 +539,16 @@ find_largest_hole (u64 start, u64 end, v last_end = end;return 0 ; } + +int __init +register_active_ranges (u64 start,u64 end,void *nid) +{ +BUG_ON (nid ==NULL ); +BUG_ON (*(unsigned long *)nid >=MAX_NUMNODES ); + +add_active_range (*(unsigned long *)nid, start, end); +return 0 ; +}#endif /* CONFIG_VIRTUAL_MEM_MAP */ static int __init
include/asm-ia64/meminit.h +1 -0
@@ -56,6 +56,7 @@ extern void efi_memmap_init(unsigned lonextern unsigned long vmalloc_end;extern struct page *vmem_map;extern int find_largest_hole (u64 start,u64 end,void *arg); +extern int register_active_ranges (u64 start,u64 end,void *arg);extern int create_mem_map_page_table (u64 start,u64 end,void *arg);#endif
mm/mem_init.c +40 -4
100 lines not on this page · raw message
Quoted 3 lines · Mel Gorman
Darn.
o Did it boot on other IA64 machines or was the Tiger the first boot failure?
I only tried to boot on the Tiger.
Quoted 2 lines
o Possibly a stupid question but does the Tiger configuration use the flatmem memory model, sparsemem or discontig?
I built using arch/ia64/configs/tiger_defconfig - a FLATMEM config with VIRT_MEM_MAP=y. The machine has 4G of memory, 2G at 0-2G, and 2G at 6G-8G (so it is somewhat sparse ... but this is pretty normal for an ia64 with
Quoted 15 lines
2G).
If it's flatmem, I noticed I made a stupid mistake where
vmem_mapis not getting set to (void *)0 for machines with small memory holes. Nothing else really obvious jumped out at me.I've attached a patch called "105-ia64_use_init_nodes.patch". Can you reverse Patch 5/6 and apply this one instead please? I've also attached 107-debug.diff that applies on top of patch 6/6. It just prints out debugging information during startup that may tell me where I went wrong in
arch/ia64. I'd really appreciate it if you could use both patches, let me know if it still fails to boot and send me the console log of the machine starting up if it fails so I can make guesses as to what is going wrong.
Quoted 2 lines
Thanks a lot for trying the patches out on ia64. It was the one arch of the set I had no chance to test with at all :/
Ok, I cloned a branch from patch4, applied the new patch 5, git-cherry-picked patch 6, and then applied the debug patch7.
Here's the console log:
Linux version 2.6.17-rc1-tiger-smpxx (aegl@linux-t10) (gcc version 3.4.3 20050227 (Red Hat 3.4.3-22.1)) #2 SMP Tue Apr 11 16:45:31 PDT 2006 EFI v1.10 by INTEL: SALsystab=0x7fe54980 ACPI=0x7ff84000 ACPI 2.0=0x7ff83000 MPS=0x7ff82000 SMBIOS=0xf0000
Early serial console at I/O port 0x2f8 (options '115200')
Initial ramdisk at: 0xe0000001fedf5000 (1303568 bytes)
SAL 3.20: Intel Corp SR870BN4 version 3.0
SAL Platform features: BusLock IRQ_Redirection
SAL: AP wakeup using external interrupt vector 0xf0
No logical to physical processor mapping availableiosapic_system_init: Disabling PC-AT compatible 8259 interrupts
ACPI: Local APIC address c0000000fee00000
PLATFORM int CPEI (0x3): GSI 22 (level, low) -> CPU 0 (0xc618) vector 30register_intr: changing vector 39 from IO-SAPIC-edge to IO-SAPIC-level
4 CPUs available, 4 CPUs total
MCA related initialization doneadd_active_range(0, 16140901064512634880, 16140901066637049856): Newadd_active_range(0, 16140901066641899520, 16140901066642489344): Newadd_active_range(0, 16140901070938308608, 16140901073083760640): Newadd_active_range(0, 16140901073084219392, 16140901073085480960): New
Dumping sorted node map
entry 0: 0 16140901064512634880 -> 16140901066637049856
entry 1: 0 16140901066641899520 -> 16140901066642489344
entry 2: 0 16140901070938308608 -> 16140901073083760640
entry 3: 0 16140901073084219392 -> 16140901073085480960
Virtual mem_map starts at 0x0000000000000000
SMP: Allowing 4 CPUs, 0 hotplug CPUs
Built 1 zonelists
Kernel command line: BOOT_IMAGE=scsi0:EFI\redhat\l-tiger-smpxx.gz root=LABEL=/ console=uart,io,0x2f8 ro
PID hash table entries: 16 (order: 4, 128 bytes)
Console: colour VGA+ 80x25
Dentry cache hash table entries: 1 (order: -11, 8 bytes)
Inode-cache hash table entries: 1 (order: -11, 8 bytes)
Placing software IO TLB between 0x4a84000 - 0x8a84000
kernel BUG at arch/ia64/mm/init.c:609!
swapper[0]: bugcheck! 0 [1]
Modules linked in:
Pid: 0, CPU 0, comm: swapper
psr : 00001010084a6010 ifs : 800000000000040f ip : [<a0000001007dd620>] Not tainted
ip is at mem_init+0x80/0x580
unat: 0000000000000000 pfs : 000000000000040f rsc : 0000000000000003
rnat: a00000010095fd80 bsps: 00000000000002f9 pr : 80000000afb5666b
ldrs: 0000000000000000 ccv : 0000000000000000 fpsr: 0009804c8a70433f csd : 0930ffff00090000 ssd : 0930ffff00090000
b0 : a0000001007dd620 b6 : a0000001007f6ba0 b7 : a0000001003c27e0 f6 : 0fffbccccccccc8c00000 f7 : 0ffdbf300000000000000 f8 : 10001c000000000000000 f9 : 10002a000000000000000 f10 : 0fffe9999999996900000 f11 : 1003e0000000000000000 r1 : a000000100b460d0 r2 : 0000000000000000 r3 : a00000010095cba0 r8 : 000000000000002a r9 : a00000010095cb90 r10 : 00000000000002f9 r11 : 00000000000be000 r12 : a00000010080fe10 r13 : a000000100808000 r14 : 0000000000004000 r15 : a00000010095cba8 r16 : 0000000000000001 r17 : a00000010095cb98 r18 : ffffffffffffffff r19 : a00000010095fd88 r20 : 00000000000000be r21 : a00000010095bd50 r22 : 0000000000000000 r23 : a00000010095cbb8 r24 : a00000010087f7e8 r25 : a00000010087f7e0 r26 : a000000100946308 r27 : 00000010084a6010 r28 : 00000000000002f9 r29 : 00000000000002f8 r30 : 0000000000000000 r31 : a00000010095cb68
Unable to handle kernel NULL pointer dereference (address 0000000000000000) swapper[0]: Oops 11012296146944 [2] Modules linked in:
Pid: 0, CPU 0, comm: swapper
psr : 0000121008022018 ifs : 8000000000000287 ip : [<a000000100116b81>] Not tainted
ip is at kmem_cache_alloc+0x41/0x100
unat: 0000000000000000 pfs : 0000000000000793 rsc : 0000000000000003
rnat: 0000000000000000 bsps: 0000000000000000 pr : 80000000afb56967
ldrs: 0000000000000000 ccv : 0000000000000000 fpsr: 0009804c8a70033f csd : 0930ffff00090000 ssd : 0930ffff00090000
b0 : a00000010003d820 b6 : a00000010003e600 b7 : a00000010000c9d0 f6 : 1003ea08f5c3b783104ea f7 : 1003e9e3779b97f4a7c16 f8 : 1003e0a0000001000117f f9 : 1003e000000000000007f f10 : 1003e0000000000000379 f11 : 1003e6db6db6db6db6db7 r1 : a000000100b460d0 r2 : 0000000000000000 r3 : a000000100949240 r8 : 0000000000000000 r9 : 0000000000000000 r10 : 0000000000000000 r11 : a000000100808f14 r12 : a00000010080f260 r13 : a000000100808000 r14 : 0000000000000000 r15 : 000000000000000f r16 : a00000010080f2f0 r17 : 0000000000000000 r18 : a000000100876bd8 r19 : a00000010080f2ec r20 : a00000010080f2e8 r21 : 000000007fffffff r22 : 0000000000000000 r23 : 0000000000000050 r24 : a0000001000117f0 r25 : a0000001000117a0 r26 : a000000100885480 r27 : a000000100946fb0 r28 : a00000010087df40 r29 : 0000000000000002 r30 : 0000000000000002 r31 : 00000000000000c0
Quoted 7 lines · Tony
Quoted 3 lines · Mel Gorman
Darn.
o Did it boot on other IA64 machines or was the Tiger the first boot failure?
I only tried to boot on the Tiger.
ok, based on your console log, I'm pretty sure it would have broken on almost any IA64.
Quoted 8 lines
Quoted 2 lines
o Possibly a stupid question but does the Tiger configuration use the flatmem memory model, sparsemem or discontig?
I built using
arch/ia64/configs/tiger_defconfig- a FLATMEM config with VIRT_MEM_MAP=y. The machine has 4G of memory, 2G at 0-2G, and 2G at 6G-8G (so it is somewhat sparse ... but this is pretty normal for an ia64 withQuoted 1 line
2G).
That's useful to know. It means I know what pfn ranges I expect to see being passed to add_active_range().
Quoted 19 lines
Quoted 13 lines
If it's flatmem, I noticed I made a stupid mistake where
vmem_mapis not getting set to (void *)0 for machines with small memory holes. Nothing else really obvious jumped out at me.I've attached a patch called "105-ia64_use_init_nodes.patch". Can you reverse Patch 5/6 and apply this one instead please? I've also attached 107-debug.diff that applies on top of patch 6/6. It just prints out debugging information during startup that may tell me where I went wrong in
arch/ia64. I'd really appreciate it if you could use both patches, let me know if it still fails to boot and send me the console log of the machine starting up if it fails so I can make guesses as to what is going wrong.Quoted 2 lines
Thanks a lot for trying the patches out on ia64. It was the one arch of the set I had no chance to test with at all :/
Ok, I cloned a branch from patch4, applied the new patch 5, git-cherry-picked patch 6, and then applied the debug patch7.
Quoted 8 lines
Here's the console log:
<snip snip>
add_active_range(0, 16140901064512634880, 16140901066637049856): Newadd_active_range(0, 16140901066641899520, 16140901066642489344): Newadd_active_range(0, 16140901070938308608, 16140901073083760640): Newadd_active_range(0, 16140901073084219392, 16140901073085480960): New <snip snip>
Good man Mel! The callback register_active_ranges() callback is getting *virtual addresses*, not PFNs (which is brutally obvious now!). For discontig, there is a similar story. count_node_pages() is getting a *physical address*, not a pfn (also called start which is a bit confusing but a different problem).
So some thinking out loud to see if you spot problems;
o PAGE_OFFSET seems to be 16140901064495857664 from the header file
o Instead of using add_active_range(node, start, end), assume I had used add_active_range(node,
(start - PAGE_OFFSET) >> PAGE_SHIFT, (end - PAGE_OFFSET) >> PAGE_SHIFT);
That would have made the console log look something like;
add_active_range(0, 4096, 522752): Newadd_active_range(0, 523936, 524080): Newadd_active_range(0, 1572864, 2096656): Newadd_active_range(0, 2096768, 2097076): New
That seems to register memory about the 0-2G mark and 6-8G with some small holes here and there. Sounds like what you expected to happen. In case the 1:1 virt->phys mapping is not always true on IA64, I decided to use __pa() instead of PAGE_OFFSET like;
add_active_range(node, __pa(start) >> PAGE_SHIFT, __pa(end) >> PAGE_SHIFT);
Is this the correct thing to do or is "start - PAGE_OFFSET" safer? Optimistically assuming __pa() is ok, the following patch (which replaces Patch 5/6 again) should boot (passed compile testing here). If it doesn't, can you send the console log again please?
Thanks again.
arch/ia64/Kconfig +3 -0
@@ -352,6 +352,9 @@ config NUMA Access). This option is for configuring high-end multiprocessor server systems. If in doubt, say N. +config ARCH_POPULATES_NODE_MAP + def_bool y + # VIRTUAL_MEM_MAP and FLAT_NODE_MEM_MAP are functionally equivalent. # VIRTUAL_MEM_MAP has been retained for historical reasons. config VIRTUAL_MEM_MAP
arch/ia64/mm/contig.c +8 -52
102 lines not on this page · raw message
arch/ia64/mm/discontig.c +6 -35
67 lines not on this page · raw message
arch/ia64/mm/init.c +12 -0
@@ -539,6 +539,18 @@ find_largest_hole (u64 start, u64 end, v last_end = end;return 0 ; } + +int __init +register_active_ranges (u64 start,u64 end,void *nid) +{ +BUG_ON (nid ==NULL ); +BUG_ON (*(unsigned long *)nid >=MAX_NUMNODES ); + +add_active_range (*(unsigned long *)nid, +__pa (start) >>PAGE_SHIFT , +__pa (end) >>PAGE_SHIFT ); +return 0 ; +}#endif /* CONFIG_VIRTUAL_MEM_MAP */ static int __init
include/asm-ia64/meminit.h +1 -0
@@ -56,6 +56,7 @@ extern void efi_memmap_init(unsigned lonextern unsigned long vmalloc_end;extern struct page *vmem_map;extern int find_largest_hole (u64 start,u64 end,void *arg); +extern int register_active_ranges (u64 start,u64 end,void *arg);extern int create_mem_map_page_table (u64 start,u64 end,void *arg);#endif
On Wed, Apr 12, 2006 at 11:50:31AM +0100, Mel Gorman wrote:
Patch got corrupted in transit and won't apply (looks like something stripped trailing spaces from empty lines). E.g.
Quoted 13 lines
diff -rup -X
/usr/src/patchset-0.5/bin//dontdifflinux-2.6.17-rc1-104-x86_64_use_init_nodes/arch/ia64/Kconfig linux-2.6.17-rc1-105-ia64_use_init_nodes/arch/ia64/Kconfigarch/ia64/Kconfig +3
-0@@ -352,6 +352,9 @@ config NUMA Access). This option is for configuring high-end multiprocessor server systems. If in doubt, say N. +config ARCH_POPULATES_NODE_MAP + def_bool y + # VIRTUAL_MEM_MAP and FLAT_NODE_MEM_MAP are functionally equivalent. # VIRTUAL_MEM_MAP has been retained for historical reasons. config VIRTUAL_MEM_MAP
Quoted 5 lines · Tony
On Wed, Apr 12, 2006 at 11:50:31AM +0100, Mel Gorman wrote:
Patch got corrupted in transit and won't apply (looks like something stripped trailing spaces from empty lines). E.g.
*swears at his mailer*
Patch is attached as 105-ia64_use_init_nodes.patch until I beat sense into my mail setup. I've added Bob Picco to the cc list as he will hit the same issue with whitespace corruption.
Sorry for the inconvenience.
Quoted 14 lines
Quoted 13 lines
diff -rup -X
/usr/src/patchset-0.5/bin//dontdifflinux-2.6.17-rc1-104-x86_64_use_init_nodes/arch/ia64/Kconfig linux-2.6.17-rc1-105-ia64_use_init_nodes/arch/ia64/Kconfigarch/ia64/Kconfig +3
-0@@ -352,6 +352,9 @@ config NUMA Access). This option is for configuring high-end multiprocessor server systems. If in doubt, say N. +config ARCH_POPULATES_NODE_MAP + def_bool y + # VIRTUAL_MEM_MAP and FLAT_NODE_MEM_MAP are functionally equivalent. # VIRTUAL_MEM_MAP has been retained for historical reasons. config VIRTUAL_MEM_MAP
Signature · 3 lines
Mel Gorman Part-time Phd Student Linux Technology Center University of Limerick IBM Dublin Software Lab
arch/ia64/Kconfig +3 -0
@@ -352,6 +352,9 @@ config NUMA Access). This option is for configuring high-end multiprocessor server systems. If in doubt, say N. +config ARCH_POPULATES_NODE_MAP + def_bool y + # VIRTUAL_MEM_MAP and FLAT_NODE_MEM_MAP are functionally equivalent. # VIRTUAL_MEM_MAP has been retained for historical reasons. config VIRTUAL_MEM_MAP
arch/ia64/mm/contig.c +8 -52
102 lines not on this page · raw message
arch/ia64/mm/discontig.c +6 -35
67 lines not on this page · raw message
arch/ia64/mm/init.c +12 -0
@@ -539,6 +539,18 @@ find_largest_hole (u64 start, u64 end, v last_end = end;return 0 ; } + +int __init +register_active_ranges (u64 start,u64 end,void *nid) +{ +BUG_ON (nid ==NULL ); +BUG_ON (*(unsigned long *)nid >=MAX_NUMNODES ); + +add_active_range (*(unsigned long *)nid, +__pa (start) >>PAGE_SHIFT , +__pa (end) >>PAGE_SHIFT ); +return 0 ; +}#endif /* CONFIG_VIRTUAL_MEM_MAP */ static int __init
include/asm-ia64/meminit.h +1 -0
@@ -56,6 +56,7 @@ extern void efi_memmap_init(unsigned lonextern unsigned long vmalloc_end;extern struct page *vmem_map;extern int find_largest_hole (u64 start,u64 end,void *arg); +extern int register_active_ranges (u64 start,u64 end,void *arg);extern int create_mem_map_page_table (u64 start,u64 end,void *arg);#endif
Quoted 3 lines · Mel Gorman
Patch is attached as 105-ia64_use_init_nodes.patch until I beat sense into my mail setup. I've added Bob Picco to the cc list as he will hit the same issue with whitespace corruption.
Ok! That boots on the tiger_defconfig.
Some stuff is weird in the dmesg output though. You report about twice as many pages in each zone, but then the total memory is about right. Here's the diff of my regular kernel (got a bunch of patches post-2.6.17-rc1) against a 2.6.17-rc1 with your patches applied. Note also that the Dentry and Inode caches allocated twice as much space (presumably based on the belief that there is more memory). My guess is that you are counting the holes.
-Tony
19,21c20,37
< On node 0 totalpages: 260725
< DMA zone: 129700 pages, LIFO batch:7
< Normal zone: 131025 pages, LIFO batch:7
---
Quoted 10 lines
add_active_range(0, 1024, 130688): Newadd_active_range(0, 130984, 131020): Newadd_active_range(0, 393216, 524164): Newadd_active_range(0, 524192, 524269): New
Dumping sorted node map
entry 0: 0 1024 -> 130688
entry 1: 0 130984 -> 131020
entry 2: 0 393216 -> 524164
entry 3: 0 524192 -> 524269
Hole found index 0: 1024 -> 1024
Quoted 8 lines
Hole found index 1: 130688 -> 130984
Hole found index 3: 524164 -> 524192
On node 0 totalpages: 522921
Hole found index 0: 1024 -> 1024
Hole found index 1: 130688 -> 130984
DMA zone: 260824 pages, LIFO batch:7
Hole found index 3: 524164 -> 524192
Normal zone: 262097 pages, LIFO batch:7
25c41
< Kernel command line: BOOT_IMAGE=scsi0:EFI\redhat\l-tiger-smp.gz root=LABEL=/ console=tty1 console=ttyS1,115200 ro
---
Quoted 1 line
Kernel command line: BOOT_IMAGE=scsi0:EFI\redhat\l-tiger-smpxx.gz root=LABEL=/ console=uart,io,0x2f8 ro
29,30c45,46
< Dentry cache hash table entries: 524288 (order: 8, 4194304 bytes) < Inode-cache hash table entries: 262144 (order: 7, 2097152 bytes)
---
Quoted 2 lines
Dentry cache hash table entries: 1048576 (order: 9, 8388608 bytes) Inode-cache hash table entries: 524288 (order: 8, 4194304 bytes)
32c48
< Memory: 4070560k/4171600k available (6836k code, 99792k reserved, 2749k data, 256k init)
---
Quoted 1 line
Memory: 4064416k/4171600k available (6832k code, 105936k reserved, 2753k data, 256k init)
Quoted 8 lines · Tony
Quoted 3 lines · Mel Gorman
Patch is attached as 105-ia64_use_init_nodes.patch until I beat sense into my mail setup. I've added Bob Picco to the cc list as he will hit the same issue with whitespace corruption.
Ok! That boots on the
tiger_defconfig.Some stuff is weird in the dmesg output though.
Ok, I see the problem. It happened because the zone boundary between DMA and NORMAL was in a hole.
When I am working out the size of a hole, I check the zone for the end_pfn of one active range is the same zone as the start_pfn in the next range. In this case, the end of area 1 is 131020 in DMA and the start of area 2 is 393216 in NORMAL so the hole does not get accounted for.
Quoted 48 lines
You report about
twice as many pages in each zone, but then the total memory is about right. Here's the diff of my regular kernel (got a bunch of patches post-2.6.17-rc1) against a 2.6.17-rc1 with your patches applied. Note also that the Dentry and Inode caches allocated twice as much space (presumably based on the belief that there is more memory). My guess is that you are counting the holes.-Tony
19,21c20,37
< On node 0 totalpages: 260725
< DMA zone: 129700 pages, LIFO batch:7
< Normal zone: 131025 pages, LIFO batch:7
---Quoted 10 lines
add_active_range(0, 1024, 130688): Newadd_active_range(0, 130984, 131020): Newadd_active_range(0, 393216, 524164): Newadd_active_range(0, 524192, 524269): New
Dumping sorted node map
entry 0: 0 1024 -> 130688
entry 1: 0 130984 -> 131020
entry 2: 0 393216 -> 524164
entry 3: 0 524192 -> 524269
Hole found index 0: 1024 -> 1024Quoted 8 lines
Hole found index 1: 130688 -> 130984
Hole found index 3: 524164 -> 524192
On node 0 totalpages: 522921
Hole found index 0: 1024 -> 1024
Hole found index 1: 130688 -> 130984
DMA zone: 260824 pages, LIFO batch:7
Hole found index 3: 524164 -> 524192
Normal zone: 262097 pages, LIFO batch:725c41
< Kernel command line: BOOT_IMAGE=scsi0:EFI\redhat\l-tiger-smp.gz root=LABEL=/ console=tty1 console=ttyS1,115200 ro
---Quoted 1 line
Kernel command line: BOOT_IMAGE=scsi0:EFI\redhat\l-tiger-smpxx.gz root=LABEL=/ console=uart,io,0x2f8 ro
29,30c45,46
< Dentry cache hash table entries: 524288 (order: 8, 4194304 bytes) < Inode-cache hash table entries: 262144 (order: 7, 2097152 bytes)
---Quoted 2 lines
Dentry cache hash table entries: 1048576 (order: 9, 8388608 bytes) Inode-cache hash table entries: 524288 (order: 8, 4194304 bytes)
32c48
< Memory: 4070560k/4171600k available (6836k code, 99792k reserved, 2749k data, 256k init)
---Quoted 1 line
Memory: 4064416k/4171600k available (6832k code, 105936k reserved, 2753k data, 256k init)
Signature · 3 lines
Mel Gorman Part-time Phd Student Linux Technology Center University of Limerick IBM Dublin Software Lab
Quoted 3 lines · Mel Gorman
Patch is attached as 105-ia64_use_init_nodes.patch until I beat sense into my mail setup. I've added Bob Picco to the cc list as he will hit the same issue with whitespace corruption.
Next I tried building a "generic" kernel (using arch/ia64/defconfig). This has NUMA=y and DISCONTIG=y). This crashes with the following console log.
-Tony
Linux version 2.6.17-rc1-tiger-smpxx (aegl@linux-t10) (gcc version 3.4.3 20050227 (Red Hat 3.4.3-22.1)) #2 SMP Wed Apr 12 09:41:12 PDT 2006 EFI v1.10 by INTEL: SALsystab=0x7fe54980 ACPI=0x7ff84000 ACPI 2.0=0x7ff83000 MPS=0x7ff82000 SMBIOS=0xf0000
booting generic kernel on platform dig
Early serial console at I/O port 0x2f8 (options '115200')
Initial ramdisk at: 0xe0000001fedf5000 (1303564 bytes)
SAL 3.20: Intel Corp SR870BN4 version 3.0
SAL Platform features: BusLock IRQ_Redirection
SAL: AP wakeup using external interrupt vector 0xf0
No logical to physical processor mapping availableiosapic_system_init: Disabling PC-AT compatible 8259 interrupts
ACPI: Local APIC address c0000000fee00000
PLATFORM int CPEI (0x3): GSI 22 (level, low) -> CPU 0 (0xc618) vector 30register_intr: changing vector 39 from IO-SAPIC-edge to IO-SAPIC-level
4 CPUs available, 4 CPUs total
MCA related initialization doneadd_active_range(0, 0, 4096): Newadd_active_range(0, 0, 131072): Newadd_active_range(0, 0, 131072): Newadd_active_range(0, 393216, 523264): Newadd_active_range(0, 393216, 523264): Newadd_active_range(0, 393216, 524288): Newadd_active_range(0, 393216, 524288): New
Virtual mem_map starts at 0xa0007ffffe400000
Dumping sorted node map
entry 0: 0 0 -> 131072
entry 1: 0 0 -> 4096
entry 2: 0 0 -> 131072
entry 3: 0 393216 -> 523264
entry 4: 0 393216 -> 524288
entry 5: 0 393216 -> 524288
entry 6: 0 393216 -> 523264
Hole found index 0: 0 -> 0prev_end > start_pfn : 131072 > 0
kernel BUG at mm/mem_init.c:775!
swapper[0]: bugcheck! 0 [1]
Modules linked in:
Pid: 0, CPU 0, comm: swapper
psr : 00001010084a2010 ifs : 800000000000048d ip : [<a000000100803b40>] Not tainted
ip is at zone_absent_pages_in_node+0x1c0/0x260
unat: 0000000000000000 pfs : 000000000000048d rsc : 0000000000000003
rnat: 0000000000000090 bsps: 000000000001003e pr : 80000000afb566ab
ldrs: 0000000000000000 ccv : 0000000000000000 fpsr: 0009804c8a70433f csd : 0930ffff00090000 ssd : 0930ffff00090000
b0 : a000000100803b40 b6 : a0000001003bfe00 b7 : a0000001003bfbc0 f6 : 0fffbccccccccc8c00000 f7 : 0ffdc8dc0000000000000 f8 : 10001e000000000000000 f9 : 10002a000000000000000 f10 : 0fffeb33333332fa80000 f11 : 1003e0000000000000000 r1 : a000000100bfb890 r2 : 0000000000000000 r3 : a000000100a125d8 r8 : 0000000000000024 r9 : a000000100a125c8 r10 : 0000000000000fff r11 : 0000000000ffffff r12 : a00000010084fd60 r13 : a000000100848000 r14 : 0000000000004000 r15 : a000000100a125e0 r16 : 00000000000002f9 r17 : a000000100a125d0 r18 : ffffffffffffffff r19 : 0000000000000000 r20 : a000000100a37a20 r21 : a000000100a117c0 r22 : 0000000000000000 r23 : a000000100a125f0 r24 : a0000001008cfaa8 r25 : a0000001008cfaa0 r26 : a0000001009fbae0 r27 : 00000010084a2010 r28 : a000000100ac9221 r29 : 0000000000000809 r30 : 0000000000000000 r31 : a000000100a125a0
Unable to handle kernel NULL pointer dereference (address 0000000000000000) swapper[0]: Oops 8813272891392 [2] Modules linked in:
Pid: 0, CPU 0, comm: swapper
psr : 0000101008022018 ifs : 8000000000000308 ip : [<a000000100124940>] Not tainted
ip is at kmem_cache_alloc+0xa0/0x160
unat: 0000000000000000 pfs : 0000000000000793 rsc : 0000000000000003
rnat: 0000000000000000 bsps: 0000000000000000 pr : 80000000afb56967
ldrs: 0000000000000000 ccv : 0000000000000000 fpsr: 0009804c8a70033f csd : 0930ffff00090000 ssd : 0930ffff00090000
b0 : a00000010003e6c0 b6 : a00000010003f4a0 b7 : a00000010000d090 f6 : 1003ea321ff35f9fb4c36 f7 : 1003e9e3779b97f4a7c16 f8 : 1003e0a00000010001231 f9 : 1003e000000000000007f f10 : 1003e0000000000000379 f11 : 1003e6db6db6db6db6db7 r1 : a000000100bfb890 r2 : 0000000000000000 r3 : a000000100848018 r8 : 0000000000000000 r9 : 0000000000000000 r10 : 0000000000000000 r11 : 0000000000000000 r12 : a00000010084f1b0 r13 : a000000100848000 r14 : 0000000000000000 r15 : 0000000018000000 r16 : a00000010084f240 r17 : a000000100848f64 r18 : a0000001008c6cb8 r19 : a00000010084f23c r20 : a00000010084f238 r21 : 000000007fffffff r22 : 0000000000000000 r23 : 0000000000000050 r24 : a000000100012310 r25 : a0000001000122c0 r26 : a0000001008d7870 r27 : a0000001009fc818 r28 : a0000001008ce020 r29 : 0000000000000002 r30 : 0000000000000002 r31 : 00000000000000c8
luck wrote: [Wed Apr 12 2006, 01:07:26PM EDT]
Quoted 10 lines · Mel Gorman
Quoted 3 lines
Patch is attached as 105-ia64_use_init_nodes.patch until I beat sense into my mail setup. I've added Bob Picco to the cc list as he will hit the same issue with whitespace corruption.
Next I tried building a "generic" kernel (using
arch/ia64/defconfig). This has NUMA=y and DISCONTIG=y). This crashes with the following console log.-Tony
[snip]
Yes. I see the same. It's because with granules we have intersecting regions which add_active_range doesn't handle. At least that's what appears to be the reason. I modified add_active_range to combine an added intersecting region and it boots. However, present pages in zones is enormous which probably means hole calculation is wrong. That's what I'm pursuing now.
bob
Quoted 9 lines · Tony
Quoted 3 lines · Mel Gorman
Patch is attached as 105-ia64_use_init_nodes.patch until I beat sense into my mail setup. I've added Bob Picco to the cc list as he will hit the same issue with whitespace corruption.
Next I tried building a "generic" kernel (using
arch/ia64/defconfig). This has NUMA=y and DISCONTIG=y). This crashes with the following console log.
Quoted 8 lines
<snipped>
add_active_range(0, 0, 4096): Newadd_active_range(0, 0, 131072): Newadd_active_range(0, 0, 131072): Newadd_active_range(0, 393216, 523264): Newadd_active_range(0, 393216, 523264): Newadd_active_range(0, 393216, 524288): Newadd_active_range(0, 393216, 524288): New
This is where it started going wrong. I did not expect add_active_range() to be called with overlapping PFNs so they were not getting merged. If they were getting merged correctly, I'd expect the output to be
add_active_range(0, 0, 4096): New add_active_range(0, 0, 131072): Merging forward add_active_range(0, 0, 131072): Merging forward add_active_range(0, 393216, 523264): New add_active_range(0, 393216, 523264): Merging forward add_active_range(0, 393216, 524288): Merging forward add_active_range(0, 393216, 524288): Merging forward
Quoted 11 lines
Virtual
mem_mapstarts at0xa0007ffffe400000
Dumping sorted node map
entry 0: 0 0 -> 131072
entry 1: 0 0 -> 4096
entry 2: 0 0 -> 131072
entry 3: 0 393216 -> 523264
entry 4: 0 393216 -> 524288
entry 5: 0 393216 -> 524288
entry 6: 0 393216 -> 523264
Hole found index 0: 0 -> 0prev_end>start_pfn: 131072 > 0
And here is where it goes BLAM. Without the debugging patch, the check is just;
BUG_ON(prev_end_pfn > start_pfn);
The error I was *expecting* to catch was an unsorted node map. It's just nice it caught this situation as well. It'll take a while to fix this up properly.
Thanks
Signature · 3 lines
Mel Gorman Part-time Phd Student Linux Technology Center University of Limerick IBM Dublin Software Lab
Quoted 11 lines
That seems to register memory about the 0-2G mark and 6-8G with some small holes here and there. Sounds like what you expected to happen. In case the 1:1
virt->physmapping is not always true on IA64, I decided to use__pa()instead ofPAGE_OFFSETlike;
add_active_range(node, __pa(start) >> PAGE_SHIFT, __pa(end) >> PAGE_SHIFT);Is this the correct thing to do or is "start -
PAGE_OFFSET" safer? Optimistically assuming__pa()is ok, the following patch (which replaces Patch 5/6 again) should boot (passed compile testing here). If it doesn't, can you send the console log again please?
Almost all of "region 7" (0xE000000000000000-0xFFFFFFFFFFFFFFFF) of the kernel address space is defined to have a 1:1 mapping with physical memory (the exception being the top 64K (0xFFFFFFFFFFFF0000-0xFFFFFFFFFFFFFFFF) which is mapped as a per-cpu area). So __pa(x) is simply defined as ((x) - PAGE_OFFSET). Using __pa(start) is effectively identical to (start - PAGE_OFFSET), but __pa() is a bit cleaner and easier to read.
-Tony
luck wrote: [Tue Apr 11 2006, 06:20:29PM EDT]
Quoted 6 lines · Mel Gorman
Quoted 4 lines
The patches have only been *compile tested* for ia64 with a flatmem configuration. At attempt was made to boot test on an ancient RS/6000 but the vanilla kernel does not boot so I have to investigate there.
Quoted 7 lines
The good news: Compilation is clean on the ia64 config variants that I usually build (all 10 of them).
The bad (or at least consistent) news: It doesn't boot on an Intel Tiger either (oops at
kmem_cache_alloc+0x41).-Tony
I had a reply queued to report the same failure with DISCONTIG+NUMA+VIRTUAL_MEM_MAP. This was 2 CPU HP rx2600. I'll take a closer look at the code tomorrow.
bob
Quoted 10 lines · Bob Picco
luck wrote: [Tue Apr 11 2006, 06:20:29PM EDT]
Quoted 9 lines · Mel Gorman
Quoted 4 lines
The patches have only been *compile tested* for ia64 with a flatmem configuration. At attempt was made to boot test on an ancient RS/6000 but the vanilla kernel does not boot so I have to investigate there.
The good news: Compilation is clean on the ia64 config variants that I usually build (all 10 of them).
Quoted 8 lines
Quoted 4 lines
The bad (or at least consistent) news: It doesn't boot on an Intel Tiger either (oops at
kmem_cache_alloc+0x41).-Tony
I had a reply queued to report the same failure with DISCONTIG+NUMA+VIRTUAL_MEM_MAP. This was 2 CPU HP rx2600. I'll take a closer look at the code tomorrow.
hmm, ok, so discontig.c is in use which narrows things down. When build_node_maps() is called, I assumed that the start and end pfn passed in was for a valid page range. Was this a valid assumption? When I re-read the comment, it implies that memory holes could be within this range which would cause boot failures. If that is the case, the correct thing to do was to call add_active_range() in count_node_pages() instead of build_node_maps().
Signature · 3 lines
Mel Gorman Part-time Phd Student Linux Technology Center University of Limerick IBM Dublin Software Lab
Mel Gorman wrote: [Tue Apr 11 2006, 08:02:10PM EDT]
Quoted 22 lines · Bob Picco
Quoted 11 lines
luck wrote: [Tue Apr 11 2006, 06:20:29PM EDT]
Quoted 9 lines · Mel Gorman
Quoted 4 lines
The patches have only been *compile tested* for ia64 with a flatmem configuration. At attempt was made to boot test on an ancient RS/6000 but the vanilla kernel does not boot so I have to investigate there.
The good news: Compilation is clean on the ia64 config variants that I usually build (all 10 of them).
Quoted 9 lines
Quoted 4 lines
The bad (or at least consistent) news: It doesn't boot on an Intel Tiger either (oops at
kmem_cache_alloc+0x41).-Tony
I had a reply queued to report the same failure with DISCONTIG+NUMA+VIRTUAL_MEM_MAP. This was 2 CPU HP rx2600. I'll take a closer
look at the code tomorrow.
Quoted 3 lines
hmm, ok, so
discontig.cis in use which narrows things down. Whenbuild_node_maps()is called, I assumed that the start and end pfn passed in was for a valid page range. Was this a valid assumption? When I re-read
The addresses are a valid physical range. The caution should be that filter_rsvd_memory converts the addresses from identity mapped to physical. efi_memmap_walk calls back to function with identity mapped addresses. What you've done seems okay.
BTW - I like want you are attempting to achieve.
Quoted 4 lines
the comment, it implies that memory holes could be within this range which would cause boot failures. If that is the case, the correct thing to do was to call
add_active_range()incount_node_pages()instead ofbuild_node_maps().
Yes that helps because of granules and it boots. The patch below is applied on top of your original post. But..
arch/ia64/mm/discontig.c +2 -3
@@ -88,9 +88,6 @@ static int __init build_node_maps(unsign min_low_pfn =min (min_low_pfn, bdp->node_boot_start>>PAGE_SHIFT ); max_low_pfn =max (max_low_pfn, bdp->node_low_pfn); -/* Add a known active range */ -add_active_range (node, start, end); -return 0 ; }
@@ -651,6 +648,8 @@ static __init int count_node_pages(unsig mem_data[node].min_pfn =min (mem_data[node].min_pfn, start >>PAGE_SHIFT ); +add_active_range (node, start, end); +return 0 ; }
Page free/avail accounting is off and I'm done for tonight. I believe it's how you treat holes but haven't looked closely yet.
Let me wrap my head around this code again. It's been some time.
Quoted 1 line
bob
Quoted 23 lines · Bob Picco
Mel Gorman wrote: [Tue Apr 11 2006, 08:02:10PM EDT]
Quoted 22 lines · Bob Picco
Quoted 20 lines
luck wrote: [Tue Apr 11 2006, 06:20:29PM EDT]
Quoted 6 lines · Mel Gorman
Quoted 4 lines
The patches have only been *compile tested* for ia64 with a flatmem configuration. At attempt was made to boot test on an ancient RS/6000 but the vanilla kernel does not boot so I have to investigate there.
Quoted 7 lines
The good news: Compilation is clean on the ia64 config variants that I usually build (all 10 of them).
The bad (or at least consistent) news: It doesn't boot on an Intel Tiger either (oops at
kmem_cache_alloc+0x41).-Tony
I had a reply queued to report the same failure with DISCONTIG+NUMA+VIRTUAL_MEM_MAP. This was 2 CPU HP rx2600. I'll take a closer
look at the code tomorrow.
Quoted 7 lines
Quoted 3 lines
hmm, ok, so
discontig.cis in use which narrows things down. Whenbuild_node_maps()is called, I assumed that the start and end pfn passed in was for a valid page range. Was this a valid assumption? When I re-readThe addresses are a valid physical range. The caution should be that
filter_rsvd_memoryconverts the addresses from identity mapped to physical.efi_memmap_walkcalls back to function with identity mapped addresses. What you've done seems okay.
It would have been ok if I spotted it was physical addresses being passed into count_node_pages(). add_active_range() expects pfns so a >> PAGE_SHIFT was missing there.
Quoted 1 line
BTW - I like want you are attempting to achieve.
Thanks
Quoted 4 lines
Quoted 4 lines
the comment, it implies that memory holes could be within this range which would cause boot failures. If that is the case, the correct thing to do was to call
add_active_range()incount_node_pages()instead ofbuild_node_maps().
Quoted 8 lines
Yes that helps because of granules and it boots. The patch below is applied on top of your original post. But..
<Patch Snipped>
Page free/avail accounting is off and I'm done for tonight. I believe it's how you treat holes but haven't looked closely yet.
Thanks for trying it out so late in the evening. The accounting is off because I was passing in physical addresses instead of pfns. The fact it booted at all means we probably registered the memory near address 0 by accident and it would eventually oops.
Quoted 2 lines
Let me wrap my head around this code again. It's been some time.
This is the same patch I posted to Tony that hopefully fix the problems on flatmem. The important changes for your discontig machine is;
o Registering in count_node_pages() as your patch fixed
o Converting the physical address passed to count_node_pages() to a PFN
Can you try it out when you're next looking at this? Thanks
arch/ia64/Kconfig +3 -0
@@ -352,6 +352,9 @@ config NUMA Access). This option is for configuring high-end multiprocessor server systems. If in doubt, say N. +config ARCH_POPULATES_NODE_MAP + def_bool y + # VIRTUAL_MEM_MAP and FLAT_NODE_MEM_MAP are functionally equivalent. # VIRTUAL_MEM_MAP has been retained for historical reasons. config VIRTUAL_MEM_MAP
arch/ia64/mm/contig.c +8 -52
102 lines not on this page · raw message
arch/ia64/mm/discontig.c +6 -35
67 lines not on this page · raw message
arch/ia64/mm/init.c +12 -0
@@ -539,6 +539,18 @@ find_largest_hole (u64 start, u64 end, v last_end = end;return 0 ; } + +int __init +register_active_ranges (u64 start,u64 end,void *nid) +{ +BUG_ON (nid ==NULL ); +BUG_ON (*(unsigned long *)nid >=MAX_NUMNODES ); + +add_active_range (*(unsigned long *)nid, +__pa (start) >>PAGE_SHIFT , +__pa (end) >>PAGE_SHIFT ); +return 0 ; +}#endif /* CONFIG_VIRTUAL_MEM_MAP */ static int __init
include/asm-ia64/meminit.h +1 -0
@@ -56,6 +56,7 @@ extern void efi_memmap_init(unsigned lonextern unsigned long vmalloc_end;extern struct page *vmem_map;extern int find_largest_hole (u64 start,u64 end,void *arg); +extern int register_active_ranges (u64 start,u64 end,void *arg);extern int create_mem_map_page_table (u64 start,u64 end,void *arg);#endif