New: nodes_clear cause hugepage unusable on non-NUMA machine
(switched to email. Please respond via emailed reply-to-all, not via the bugzilla web interface).
Quoted 26 lines · On Thu, 2 Jul 2009 01:22:24 GMT [email protected] wrote:
http://bugzilla.kernel.org/show_bug.cgi?id=13690
Summary: nodes_clear cause hugepage unusable on non-NUMA machine Product: Platform Specific/Hardware Version: 2.5 Kernel Version: 2.6.31-rc1 Platform: All OS/Version: Linux Tree: Mainline Status: NEW Severity: high Priority: P1 Component: i386 AssignedTo: [email protected] ReportedBy: [email protected] CC: [email protected] Regression: Yes
73d60b7f747176dbdff826c4127d22e1fd3f9f74commit introduced anodes_clearfunction for NUMA machine. But seems the commit omits non-NUMA machine.
If find_zone_movable_pfns_for_nodes/early_calculate_totalpages has no chance to run.nodes_clearwill block HUPEPAGE using in my specjbb2005 testing on myStoakely(i386/x86_64), waybridge(i386), IBM T61(i386)
Review
Quoted 5 lines
(quoted diff) +5
-0+ /* + * find_zone_movable_pfns_for_nodes/early_calculate_totalpages init + * that node_mask, clear it at first + */ + nodes_clear(node_states[N_HIGH_MEMORY]);
Thanks.
fyi, with recently-occurring bugs and regressions of this nature, it is (I think) best to deal with them via email rather than bugzilla. Bugzilla is better-suited to longer-lived bugs where we have a need to track them, generate statistics, etc.
that looks strange...
config is 32bit.
the second patch only do save and restore. and should be right right.
please check following patch on today's linus tree. and send out /proc/iomem
Thanks
Yinghai
[PATCH] x86: add boundary check for 32bit res before expand e820 resource to alignment
fix hang with HIGHMEM_64G and 32bit resource.
according to hpa and Linus, use (resource_size_t)-1 to fend off big ranges.
analyized by hpa
--- arch/x86/include/asm/proto.h | 3 --- arch/x86/kernel/e820.c | 20 ++++++++++++-------- include/linux/kernel.h | 5 +++++ 3 files changed, 17 insertions(+), 11 deletions(-)
arch/x86/kernel/e820.c +12 -8
@@ -1367,9 +1367,9 @@ void __init e820_reserve_resources(void) }/* How much should we pad RAM ending depending on where it is? */ -static unsigned long ram_alignment (resource_size_t pos) +static u64 ram_alignment (u64 pos) { -unsigned long mb = pos >>20 ; +u64 mb = pos >>20 ;/* To 64kB in the first megabyte */ if (!mb)
@@ -1383,6 +1383,8 @@ static unsigned long ram_alignment(resoureturn 32 *1024 *1024 ; } +#define MAX_RESOURCE_SIZE ((resource_size_t)-1) +void __init e820_reserve_resources_late (void ) {int i;
@@ -1400,17 +1402,19 @@ void __init e820_reserve_resources_late(* avoid stolen RAM: */ for (i =0 ; i < e820.nr_map; i++) { -struct e820entry *entry = &e820_saved.map[i]; -resource_size_t start, end; +struct e820entry *entry = &e820.map[i]; +u64 start, end;if (entry->type !=E820_RAM )continue ; start = entry->addr + entry->size; - end =round_up (start,ram_alignment (start)); -if (start == end) + end =round_up (start,ram_alignment (start)) -1 ; +if (end >MAX_RESOURCE_SIZE ) + end =MAX_RESOURCE_SIZE ; +if (start > end)continue ; -reserve_region_with_split (&iomem_resource, start, - end -1 ,"RAM buffer" ); +reserve_region_with_split (&iomem_resource, start, end, +"RAM buffer" ); } }
The new patch works for my stoakley i386 machine. But for x86_64 machine the specjbb2005 still can not run with hugepage. The specjbb2005 use the same java setting as i386 system. After apply your patch, the iomem of x86_64 is:
00000000-0000ffff : reserved
00010000-0009cbff : System RAM
0009cc00-0009ffff : reserved
000cc000-000cffff : reserved
000e0000-000fffff : reserved
00100000-cfefffff : System RAM
01000000-014eb53e : Kernel code
014eb53f-0177390f : Kernel data
01830000-018f583f : Kernel bss
cff00000-cff0afff : ACPI Tables
cff0b000-cff0bfff : ACPI Non-volatile Storage
cff0c000-cfffffff : reserved
d0000000-d7ffffff : PCI Bus 0000:08
d0000000-d7ffffff : 0000:08:01.0
d8000000-d81fffff : PCI Bus 0000:03
d8000000-d81fffff : PCI Bus 0000:06
d8000000-d80fffff : 0000:06:02.0
d8100000-d810ffff : 0000:06:01.0
d8200000-d84fffff : PCI Bus 0000:03
d8200000-d83fffff : PCI Bus 0000:06
d8200000-d82fffff : 0000:06:02.0
d8200000-d82fffff : e100
d8300000-d831ffff : 0000:06:01.0
d8300000-d831ffff : e1000
d8320000-d832ffff : 0000:06:01.0
d8320000-d832ffff : e1000
d8330000-d8330fff : 0000:06:02.0
d8330000-d8330fff : e100d8500000-d87fffff : PCI Bus 0000:07
d8500000-d8503fff : 0000:07:00.0
d8504000-d8507fff : 0000:07:00.1
d8520000-d853ffff : 0000:07:00.0
d8540000-d855ffff : 0000:07:00.1
d8600000-d86fffff : 0000:07:00.0
d8700000-d87fffff : 0000:07:00.1
d8800000-d88fffff : PCI Bus 0000:08
d8800000-d880ffff : 0000:08:01.0
d8810000-d8813fff : 0000:08:08.0
d8814000-d88147ff : 0000:08:08.0
d8820000-d883ffff : 0000:08:01.0
d8904000-d8907fff : 0000:00:1b.0
d8908000-d89083ff : 0000:00:1d.7
d8908000-d89083ff : ehci_hcd
d8908400-d89087ff : 0000:00:1f.2
d8908400-d89087ff : ahci
d8a00000-d8bfffff : PCI Bus 0000:07
d8a00000-d8afffff : 0000:07:00.0
d8b00000-d8bfffff : 0000:07:00.1
e0000000-efffffff : reserved
e0000000-efffffff : pnp 00:01
e0000000-e07fffff : PCI MMCONFIG 0 [00-07]
fe000000-fe01ffff : pnp 00:01
fe000000-fe01ffff : i5k_amb
fe600000-fe6fffff : pnp 00:01
fe700000-fe703fff : 0000:00:0f.0
fec00000-fec0ffff : reserved
fec00000-fec00fff : IOAPIC 0
fec88000-fec88fff : IOAPIC 1
fec88000-fec88fff : pnp 00:01
fec89000-fec89fff : IOAPIC 2
fec89000-fec89fff : pnp 00:01
fed00000-fed003ff : HPET 0
fed1c000-fed1ffff : pnp 00:01
fed20000-fed44fff : pnp 00:01
fed45000-fed8ffff : pnp 00:01
fee00000-fee00fff : Local APIC
fee00000-fee00fff : reserved
ff000000-ffffffff : reserved
100000000-12fffffff : System RAM
====================
The iomem of i386 stoakley is:
stoakley.iomem.i386 2009-07-02 14:19:59.000000000 +0800 renamed from stoakley.iomem.x86_64 2009-07-02 13:53:35.000000000 +0800 +6 -5
@@ -1,12 +1,15 @@ 00000000-0000ffff : reserved 00010000-0009cbff : System RAM 0009cc00-0009ffff : reserved +000a0000-000bffff : Video RAM area +000c0000-000cafff : Video ROM 000cc000-000cffff : reserved 000e0000-000fffff : reserved + 000f0000-000fffff : System ROM 00100000-cfefffff : System RAM - 01000000-014eb53e : Kernel code - 014eb53f-0177390f : Kernel data - 01830000-018f583f : Kernel bss + 00100000-00602876 : Kernel code + 00602877-008e49db : Kernel data + 00954000-009fe433 : Kernel bss cff00000-cff0afff : ACPI Tables cff0b000-cff0bfff : ACPI Non-volatile Storage cff0c000-cfffffff : reserved
@@ -50,7 +53,6 @@ e0000000-efffffff : pnp 00:01 e0000000-e07fffff : PCI MMCONFIG 0 [00-07] fe000000-fe01ffff : pnp 00:01 - fe000000-fe01ffff : i5k_amb fe600000-fe6fffff : pnp 00:01 fe700000-fe703fff : 0000:00:0f.0 fec00000-fec0ffff : reserved
@@ -66,4 +68,3 @@ fee00000-fee00fff : Local APIC fee00000-fee00fff : reserved ff000000-ffffffff : reserved -100000000-12fffffff : System RAM
Alex
Quoted 79 lines · Yinghai Lu
that looks strange...
config is 32bit.
the second patch only do save and restore. and should be right right.
please check following patch on today's linus tree. and send out
/proc/iomemThanks
Yinghai
[PATCH] x86: add boundary check for 32bit res before expand e820 resource to alignment
fix hang with
HIGHMEM_64Gand 32bit resource.
according to hpa and Linus, use (resource_size_t)-1 to fend off big ranges.analyized by hpa
Reported-and-tested-by Mikael Pettersson <[email protected]>Signed-off-by Yinghai Lu <[email protected]>--- arch/x86/include/asm/proto.h | 3 --- arch/x86/kernel/e820.c | 20 ++++++++++++-------- include/linux/kernel.h | 5 +++++ 3 files changed, 17 insertions(+), 11 deletions(-)Index:
linux-2.6/arch/x86/kernel/e820.c
===================================================================arch/x86/kernel/e820.c +12
-8@@ -1367,9 +1367,9 @@ void __init e820_reserve_resources(void) }/* How much should we pad RAM ending depending on where it is? */ -static unsigned long ram_alignment (resource_size_t pos) +static u64 ram_alignment (u64 pos) { -unsigned long mb = pos >>20 ; +u64 mb = pos >>20 ;/* To 64kB in the first megabyte */ if (!mb)@@ -1383,6 +1383,8 @@ static unsigned long ram_alignment(resoureturn 32 *1024 *1024 ; } +#define MAX_RESOURCE_SIZE ((resource_size_t)-1) +void __init e820_reserve_resources_late (void ) {int i;@@ -1400,17 +1402,19 @@ void __init e820_reserve_resources_late(* avoid stolen RAM: */ for (i =0 ; i < e820.nr_map; i++) { -struct e820entry *entry = &e820_saved.map[i]; -resource_size_t start, end; +struct e820entry *entry = &e820.map[i]; +u64 start, end;if (entry->type !=E820_RAM )continue ; start = entry->addr + entry->size; - end =round_up (start,ram_alignment (start)); -if (start == end) + end =round_up (start,ram_alignment (start)) -1 ; +if (end >MAX_RESOURCE_SIZE ) + end =MAX_RESOURCE_SIZE ; +if (start > end)continue ; -reserve_region_with_split (&iomem_resource, start, - end -1 ,"RAM buffer" ); +reserve_region_with_split (&iomem_resource, start, end, +"RAM buffer" ); } }
Quoted 4 lines · Alex Shi
The new patch works for my stoakley i386 machine. But for
x86_64machine the specjbb2005 still can not run with hugepage. The specjbb2005 use the same java setting as i386 system. After apply your patch, the iomem ofx86_64is:
please check
[PATCH] x86: don't clear nodes_states[N_NORMAL_MEMORY] when numa is not compiled in
Alex found:
for x86_64 machine the specjbb2005 still can not run with hugepage
only happens when numa is not compiled in
the root cause: node_set_state will not set it back for us in that case
so don't clear that when numa is not select in config
--- arch/x86/mm/init_64.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-)
arch/x86/mm/init_64.c +7 -1
@@ -598,8 +598,14 @@ void __init paging_init(void)sparse_memory_present_with_active_regions (MAX_NUMNODES );sparse_init (); -/* clear the default setting with node 0 */ +#if MAX_NUMNODES >1 +/* +* clear the default setting with node 0 +* note: don't clear it, node_set_state will do nothing +* (aka set it back) when numa support is not compiled in +*/ nodes_clear (node_states[N_NORMAL_MEMORY ]); +#endif free_area_init_nodes (max_zone_pfns); }
Quoted 6 lines · Yinghai Lu
Index:
linux-2.6/arch/x86/mm/init_64.c
===================================================================arch/x86/mm/init_64.c +0
-0@@ -598,8 +598,14 @@ void __init paging_init(void)
arch/x86/mm/init_64.c void __init paging_init(void) · line 607
Quoted 10 lines
arch/x86/mm/init_64.c +6
-1sparse_memory_present_with_active_regions (MAX_NUMNODES );sparse_init (); -/* clear the default setting with node 0 */ +#if MAX_NUMNODES >1 +/* +* clear the default setting with node 0 +* note: don't clear it, node_set_state will do nothing +* (aka set it back) when numa support is not compiled in +*/ nodes_clear (node_states[N_NORMAL_MEMORY ]);
The problem was that nodes_clear() does not fall back to a noop on !NUMA. The node_set/clear_states() operations do become noops.
Could we make it more consistent by using only operations of the same type? F.e. Add a node_clearall_states() in include/linux/nodemask.h that falls back to a noop on !NUMA like the node_*_states operation?
Another options is to restore node_states[N_NORMAL_MEMORY] to its initial condition. See the definition of node_states in page_alloc.c.
Yes, the patch fixes this bug!
Alex
Quoted 47 lines
-----Original Message-----
From: Yinghai Lu [mailto:[email protected]]
Sent: 2009年7月2日 16:51
To: Shi, Alex; Andrew Morton; Ingo Molnar
Cc: [email protected]; [email protected]; Christoph Lameter; Mel Gorman; [email protected]; Zhang, Yanmin; Chen, Tim C
Subject: Re: [Bugme-new] [Bug 13690] New:nodes_clearcause hugepage unusable on non-NUMA machineQuoted 4 lines · Alex Shi
The new patch works for my stoakley i386 machine. But for
x86_64machine the specjbb2005 still can not run with hugepage. The specjbb2005 use the same java setting as i386 system. After apply your patch, the iomem ofx86_64is:please check
[PATCH] x86: don't clear nodes_states[N_NORMAL_MEMORY] when numa is not compiled in
Alex found:
forx86_64machine the specjbb2005 still can not run with hugepageonly happens when numa is not compiled in
the root cause:
node_set_statewill not set it back for us in that caseso don't clear that when numa is not select in config
Reported-by Alex Shi <[email protected]>Signed-off-by Yinghai Lu <[email protected]>---
arch/x86/mm/init_64.c| 8 +++++++-1 file changed, 7 insertions(+), 1 deletion(-)Index:
linux-2.6/arch/x86/mm/init_64.c
===================================================================arch/x86/mm/init_64.c +1
-1@@ -598,8 +598,14 @@ void __init paging_init(void)sparse_memory_present_with_active_regions (MAX_NUMNODES );sparse_init (); -/* clear the default setting with node 0 */ +#if MAX_NUMNODES >1
arch/x86/mm/init_64.c void __init paging_init(void) · line 611
Quoted 10 lines
arch/x86/mm/init_64.c +6
-0+/* +* clear the default setting with node 0 +* note: don't clear it, node_set_state will do nothing +* (aka set it back) when numa support is not compiled in +*/ nodes_clear (node_states[N_NORMAL_MEMORY ]); +#endif free_area_init_nodes (max_zone_pfns); }
�翳簕.n�+壏煯壄�+%娝�遍荻w�簕.n�+壏{炳G珴�妠ay�蕠跈�,jf"穐殢飦�戧鐉_璁(殠娸"濟�m���G珴�⒏?櫒璀�&x忈秈O曟瑉窔v豝�m����鹅�⒏?朓�
PATCH 1/1 x86: don't clear nodes_states[N_NORMAL_MEMORY] when numa is not compiled in -v2
In reply to Christoph LameterAlex found:
for x86_64 machine the specjbb2005 still can not run with hugepage
only happens when numa is not compiled in
the root cause: node_set_state will not set it back for us in that case
so don't clear that when numa is not select in config
v2: use node_clear_state instead
--- arch/x86/mm/init_64.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-)
arch/x86/mm/init_64.c +9 -2
@@ -598,8 +598,15 @@ void __init paging_init(void)sparse_memory_present_with_active_regions (MAX_NUMNODES );sparse_init (); -/* clear the default setting with node 0 */ -nodes_clear (node_states[N_NORMAL_MEMORY ]); + +/* +* clear the default setting with node 0 +* note: don't use nodes_clear here, that is really clearing when +* numa support is not compiled in, and later node_set_state +* will not set it back. +*/ +node_clear_state (0 ,N_NORMAL_MEMORY ); +free_area_init_nodes (max_zone_pfns); }
Quoted 22 lines · Christoph Lameter
Quoted 6 lines · Yinghai Lu
Index:
linux-2.6/arch/x86/mm/init_64.c
===================================================================arch/x86/mm/init_64.c +0
-0@@ -598,8 +598,14 @@ void __init paging_init(void)arch/x86/mm/init_64.c void __init paging_init(void) · line 607
Quoted 10 lines
arch/x86/mm/init_64.c +6
-1sparse_memory_present_with_active_regions (MAX_NUMNODES );sparse_init (); -/* clear the default setting with node 0 */ +#if MAX_NUMNODES >1 +/* +* clear the default setting with node 0 +* note: don't clear it, node_set_state will do nothing +* (aka set it back) when numa support is not compiled in +*/ nodes_clear (node_states[N_NORMAL_MEMORY ]);The problem was that
nodes_clear()does not fall back to a noop on !NUMA. The node_set/clear_states() operations do become noops.
Quoted 6 lines
Could we make it more consistent by using only operations of the same type? F.e. Add a
node_clearall_states()ininclude/linux/nodemask.hthat falls back to a noop on !NUMA like the node_*_states operation?Another options is to restore node_states[N_NORMAL_MEMORY] to its initial condition. See the definition of
node_statesinpage_alloc.c.
could use node_clear_state(0, N_NORMAL_MEMORY) instead. because default one only have node 0 set in that mask.
YH
Yinghai:
The 31-rc2 kernel still can not use hugepage on non-NUMA machine. And this patch did not appear on rc2 kernel. Are there some concern about this?
BRG
Alex
Quoted 46 lines · Yinghai Lu
Quoted 4 lines · Alex Shi
The new patch works for my stoakley i386 machine. But for
x86_64machine the specjbb2005 still can not run with hugepage. The specjbb2005 use the same java setting as i386 system. After apply your patch, the iomem ofx86_64is:please check
[PATCH] x86: don't clear nodes_states[N_NORMAL_MEMORY] when numa is not compiled in
Alex found:
forx86_64machine the specjbb2005 still can not run with hugepageonly happens when numa is not compiled in
the root cause:
node_set_statewill not set it back for us in that caseso don't clear that when numa is not select in config
Reported-by Alex Shi <[email protected]>Signed-off-by Yinghai Lu <[email protected]>--- arch/x86/mm/init_64.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-)Index:
linux-2.6/arch/x86/mm/init_64.c
===================================================================arch/x86/mm/init_64.c +7
-1@@ -598,8 +598,14 @@ void __init paging_init(void)sparse_memory_present_with_active_regions (MAX_NUMNODES );sparse_init (); -/* clear the default setting with node 0 */ +#if MAX_NUMNODES >1 +/* +* clear the default setting with node 0 +* note: don't clear it, node_set_state will do nothing +* (aka set it back) when numa support is not compiled in +*/ nodes_clear (node_states[N_NORMAL_MEMORY ]); +#endif free_area_init_nodes (max_zone_pfns); }
Quoted 6 lines · Alex Shi
Yinghai:
The 31-rc2 kernel still can not use hugepage on non-NUMA machine. And this patch did not appear on rc2 kernel. Are there some concern about this?
can you check
http://lkml.org/lkml/2009/7/2/326
YH
Quoted 8 lines · Yinghai Lu
Quoted 6 lines · Alex Shi
Yinghai:
The 31-rc2 kernel still can not use hugepage on non-NUMA machine. And this patch did not appear on rc2 kernel. Are there some concern about this?
Quoted 4 lines
can you check
http://lkml.org/lkml/2009/7/2/326YH
It works on my Stoakley i386 and x86_64 with latest Linus' kernel tree.
Alex
PATCH 1/1 x86: don't clear nodes_states[N_NORMAL_MEMORY] when numa is not compiled in -v2
In reply to Christoph LameterAlex found:
for x86_64 machine the specjbb2005 still can not run with hugepage
only happens when numa is not compiled in
the root cause: node_set_state will not set it back for us in that case
so don't clear that when numa is not select in config
v2: use node_clear_state instead
--- arch/x86/mm/init_64.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-)
arch/x86/mm/init_64.c +9 -2
@@ -598,8 +598,15 @@ void __init paging_init(void)sparse_memory_present_with_active_regions (MAX_NUMNODES );sparse_init (); -/* clear the default setting with node 0 */ -nodes_clear (node_states[N_NORMAL_MEMORY ]); + +/* +* clear the default setting with node 0 +* note: don't use nodes_clear here, that is really clearing when +* numa support is not compiled in, and later node_set_state +* will not set it back. +*/ +node_clear_state (0 ,N_NORMAL_MEMORY ); +free_area_init_nodes (max_zone_pfns); }
Re: [PATCH] x86: don't clear nodes_states[N_NORMAL_MEMORY] when numa is not compiled in -v2