2.6.23 boot failures on x86-64.
Subsystems (4)
Dave Jones
Acked-by Suresh SiddhaWe've had a number of people reporting that their x86-64s stopped booting when they moved to 2.6.23. It rebooted just after discovering the AGP bridge as a result of the IOMMU init.
Martin tracked this down to the following commit.
commit 2e1c49db4c640b35df13889b86b9d62215ade4b6
Author: Zou Nan hai <[email protected]> Date: Fri Jun 1 00:46:28 2007 -0700
x86_64: allocate sparsemem memmap above 4G
On systems with huge amount of physical memory, VFS cache and memory memmap may eat all available system memory under 4G, then the system may fail to allocate swiotlb bounce buffer.
There was a fix for this issue in arch/x86_64/mm/numa.c, but that fix dose not cover sparsemem model.
This patch add fix to sparsemem model by first try to allocate memmap above 4G.
This should be probably be reverted for 2.6.23-stable, and either fixed properly in .24, or reverted there too.
More info at https://bugzilla.redhat.com/show_bug.cgi?id=249174
Dave
Signature · 1 line
http://www.codemonkey.org.uk
Quoted 31 lines · Dave Jones
We've had a number of people reporting that their x86-64s stopped booting when they moved to 2.6.23. It rebooted just after discovering the AGP bridge as a result of the IOMMU init.
Martin tracked this down to the following commit.
commit
2e1c49db4c640b35df13889b86b9d62215ade4b6Author: Zou Nan hai <[email protected]> Date: Fri Jun 1 00:46:28 2007 -0700
x86_64: allocate sparsemem memmap above 4GOn systems with huge amount of physical memory, VFS cache and memory memmap may eat all available system memory under 4G, then the system may fail to allocate swiotlb bounce buffer.
There was a fix for this issue in
arch/x86_64/mm/numa.c, but that fix dose not cover sparsemem model.This patch add fix to sparsemem model by first try to allocate memmap above 4G.
Signed-off-by Zou Nan hai <[email protected]>Acked-by Suresh Siddha <[email protected]>Cc Andi Kleen <[email protected]>Cc <[email protected]>Signed-off-by Andrew Morton <[email protected]>Signed-off-by Linus Torvalds <[email protected]>
Quoted 2 lines
This should be probably be reverted for 2.6.23-stable, and either fixed properly in .24, or reverted there too.
I'll be glad to revert it in -stable, if it's also reverted in Linus's tree first :)
thanks,
greg k-h
Quoted 3 lines · Greg KH
I'll be glad to revert it in -stable, if it's also reverted in Linus's tree first :)
We've had some changes since 2.6.23, and afaik, the "alloc_bootmem_high_node()" code is alreadt effectively dead there. It's only called if CONFIG_SPARSEMEM_VMEMMAP is *not* enabled, and I *think* we enable it by force on x86-64 these days.
More people added to Cc, just to clarify whether I'm just confused.
Andy, Christoph, Mel: commit 2e1c49db4c640b35df13889b86b9d62215ade4b6 aka "x86_64: allocate sparsemem memmap above 4G" is the one that causes the failures, just fyi.
Martin - it would be great if you could try out your failing machine with 2.6.24-rc1 (or a nightly snapshot or current git.. the more recent the better).
But if I'm right, that commit should be reverted from 2.6.24 just because it's pointless (even if the bug itself is gone). And if I'm wrong, it should be reverted. So something like the appended would make sense regardless.
Can I get a "tested-by"? And/or ack/nack's on my half-arsed theory above?
Linus
--
From: Linus Torvalds <[email protected]>
Revert "x86_64: allocate sparsemem memmap above 4G"
This reverts commit 2e1c49db4c640b35df13889b86b9d62215ade4b6, since testing in Fedora has shown it to cause boot failures, as per Dave Jones. Bisected down by Martin Ebourne.
arch/x86/mm/init_64.c +0 -6
@@ -728,12 +728,6 @@ int in_gate_area_no_task(unsigned long addr)return (addr >=VSYSCALL_START ) && (addr <VSYSCALL_END ); } -void *__init alloc_bootmem_high_node (pg_data_t *pgdat,unsigned long size) -{ -return __alloc_bootmem_core (pgdat->bdata, size, -SMP_CACHE_BYTES , (4UL *1024 *1024 *1024 ),0 ); -} -const char *arch_vma_name (struct vm_area_struct *vma) {if (vma->vm_mm && vma->vm_start == (long )vma->vm_mm->context.vdso)
include/linux/bootmem.h +0 -1
@@ -59,7 +59,6 @@ extern void *__alloc_bootmem_core(struct bootmem_data *bdata,unsigned long align, unsigned long goal, unsigned long limit); -extern void *alloc_bootmem_high_node (pg_data_t *pgdat,unsigned long size);#ifndef CONFIG_HAVE_ARCH_BOOTMEM_NODE extern void reserve_bootmem (unsigned long addr,unsigned long size);
mm/sparse.c +0 -11
@@ -220,12 +220,6 @@ static int __meminit sparse_init_one_section(struct mem_section *ms,return 1 ; } -__attribute__((weak))__init -void *alloc_bootmem_high_node (pg_data_t *pgdat,unsigned long size) -{ -return NULL ; -} -static unsigned long usemap_size (void ) {unsigned long size_bytes;
@@ -267,11 +261,6 @@ struct page __init *sparse_mem_map_populate(unsigned long pnum, int nid)if (map)return map; - map =alloc_bootmem_high_node (NODE_DATA (nid), -sizeof (struct page ) *PAGES_PER_SECTION ); -if (map) -return map; - map =alloc_bootmem_node (NODE_DATA (nid),sizeof (struct page ) *PAGES_PER_SECTION );return map;
Quoted 4 lines · Linus Torvalds
We've had some changes since 2.6.23, and afaik, the "
alloc_bootmem_high_node()" code is alreadt effectively dead there. It's only called ifCONFIG_SPARSEMEM_VMEMMAPis *not* enabled, and I *think* we enable it by force on x86-64 these days.
CONFIG_SPARSEMEM_VMEMMAP was introduced in 2.6.24-rc1.
If I read this Kconfig.x86_64 correctly then it seems that DISCONTIG is still the default. Andy?
config ARCH_DISCONTIGMEM_ENABLE
bool
depends on NUMA
default y
config ARCH_DISCONTIGMEM_DEFAULT
def_bool y
depends on NUMA
config ARCH_SPARSEMEM_ENABLE
def_bool y
depends on (NUMA || EXPERIMENTAL)
select SPARSEMEM_VMEMMAP_ENABLEconfig ARCH_MEMORY_PROBE
def_bool y
depends on MEMORY_HOTPLUG
config ARCH_FLATMEM_ENABLE
def_bool y
depends on !NUMA
Quoted 11 lines · Linus Torvalds
Quoted 3 lines · Greg KH
I'll be glad to revert it in -stable, if it's also reverted in Linus's tree first :)
We've had some changes since 2.6.23, and afaik, the "
alloc_bootmem_high_node()" code is alreadt effectively dead there. It's only called ifCONFIG_SPARSEMEM_VMEMMAPis *not* enabled, and I *think* we enable it by force on x86-64 these days.
If so, we(Nanhai and myself) will take a look at VMEMMAP changes and see if the bug that the commit 2e1c49db4c640b35df13889b86b9d62215ade4b6 tries to fix is still open in the latest git.
But I can't explain how 2e1c49db4c640b35df13889b86b9d62215ade4b6 can be the root cause of Dave's issue in 2.6.23.
thanks,
suresh
Quoted 2 lines · Suresh B
But I can't explain how
2e1c49db4c640b35df13889b86b9d62215ade4b6can be the root cause of Dave's issue in 2.6.23.
2.6.23 has no VMEMMAP support for x86_64.
Andy Whitcroft
Acked-byQuoted 11 lines · Linus Torvalds
Quoted 3 lines · Greg KH
I'll be glad to revert it in -stable, if it's also reverted in Linus's tree first :)
We've had some changes since 2.6.23, and afaik, the "
alloc_bootmem_high_node()" code is alreadt effectively dead there. It's only called ifCONFIG_SPARSEMEM_VMEMMAPis *not* enabled, and I *think* we enable it by force on x86-64 these days.
CONFIG_SPARSEMEM_VMEMMAP is the default when SPARSEMEM is enabled
on x86_64. The overall default remains DISCONTIGMEM, mainly as a
safety measure while the i386/x86_64 => x86 merge stablises. But yes this code is only used when SPARSEMEM is enabled but VMEMMAP is not. So it is effectivly redundant.
Quoted 5 lines
More people added to Cc, just to clarify whether I'm just confused.
Andy, Christoph, Mel: commit
2e1c49db4c640b35df13889b86b9d62215ade4b6aka "x86_64: allocate sparsemem memmap above 4G" is the one that causes the failures, just fyi.
That patch seems to have a laudable goal of trying to push the memory which backs the sparsemem memmap out to non-dma memory. I would have expected that call to actually succeed as the bootmem allocator seems to try at the goal which would likely be outside the node on a small machine, and then retry without a goal. Which is what the code without the goal does. Most illogical.
Quoted 10 lines
Martin - it would be great if you could try out your failing machine with 2.6.24-rc1 (or a nightly snapshot or current git.. the more recent the better).
But if I'm right, that commit should be reverted from 2.6.24 just because it's pointless (even if the bug itself is gone). And if I'm wrong, it should be reverted. So something like the appended would make sense regardless.
Can I get a "tested-by"? And/or ack/nack's on my half-arsed theory above?
This code is definatly only used when SPARSEMEM is enabled, and VMEMMAP is not which is not a combination we see on x86_64.
Quoted 65 lines
Linus
--
From: Linus Torvalds <[email protected]>Revert "
x86_64: allocate sparsemem memmap above 4G"This reverts commit
2e1c49db4c640b35df13889b86b9d62215ade4b6, since testing in Fedora has shown it to cause boot failures, as per Dave Jones. Bisected down by Martin Ebourne.Cc Dave Jones <[email protected]>Cc Martin Ebourne <[email protected]>Cc Zou Nan hai <[email protected]>Cc Suresh Siddha <[email protected]>Cc Andrew Morton <[email protected]>Signed-off-by Linus Torvalds <[email protected]>arch/x86/mm/init_64.c +0
-6@@ -728,12 +728,6 @@ int in_gate_area_no_task(unsigned long addr)return (addr >=VSYSCALL_START ) && (addr <VSYSCALL_END ); } -void *__init alloc_bootmem_high_node (pg_data_t *pgdat,unsigned long size) -{ -return __alloc_bootmem_core (pgdat->bdata, size, -SMP_CACHE_BYTES , (4UL *1024 *1024 *1024 ),0 ); -} -const char *arch_vma_name (struct vm_area_struct *vma) {if (vma->vm_mm && vma->vm_start == (long )vma->vm_mm->context.vdso)include/linux/bootmem.h +0
-1@@ -59,7 +59,6 @@ extern void *__alloc_bootmem_core(struct bootmem_data *bdata,unsigned long align, unsigned long goal, unsigned long limit); -extern void *alloc_bootmem_high_node (pg_data_t *pgdat,unsigned long size);#ifndef CONFIG_HAVE_ARCH_BOOTMEM_NODE extern void reserve_bootmem (unsigned long addr,unsigned long size);mm/sparse.c +0
-6@@ -220,12 +220,6 @@ static int __meminit sparse_init_one_section(struct mem_section *ms,return 1 ; } -__attribute__((weak))__init -void *alloc_bootmem_high_node (pg_data_t *pgdat,unsigned long size) -{ -return NULL ; -} -static unsigned long usemap_size (void ) {unsigned long size_bytes;@@ -267,11 +261,6 @@ struct page __init *sparse_mem_map_populate(unsigned long pnum, int nid)if (map)
mm/sparse.c struct page __init *sparse_mem_map_populate(unsigned long pnum, int nid) · line 266
Quoted 10 lines
mm/sparse.c +0
-5return map; - map =alloc_bootmem_high_node (NODE_DATA (nid), -sizeof (struct page ) *PAGES_PER_SECTION ); -if (map) -return map; - map =alloc_bootmem_node (NODE_DATA (nid),sizeof (struct page ) *PAGES_PER_SECTION );return map;
-apw
Quoted 10 lines · Linus Torvalds
Martin - it would be great if you could try out your failing machine with 2.6.24-rc1 (or a nightly snapshot or current git.. the more recent the better).
But if I'm right, that commit should be reverted from 2.6.24 just because it's pointless (even if the bug itself is gone). And if I'm wrong, it should be reverted. So something like the appended would make sense regardless.
Can I get a "tested-by"? And/or ack/nack's on my half-arsed theory above?
Current git boots ok as is.
I used the config from the fedora 2.6.23 kernel and accepted defaults for all the new options.
Cheers,
Martin.
Quoted 3 lines · On Monday 29 October 2007 18:50:14 Dave Jones wrote:
We've had a number of people reporting that their x86-64s stopped booting when they moved to 2.6.23. It rebooted just after discovering the AGP bridge as a result of the IOMMU init.
It's probably the usual "nobody tests sparsemem at all" issue.
But if allocating bootmem >4G doesn't work on these systems most likely they have more problems anyways. It might be better to find out what goes wrong exactly.
-Andi
Quoted 6 lines · Andi Kleen
Quoted 3 lines · On Monday 29 October 2007 18:50:14 Dave Jones wrote:
We've had a number of people reporting that their x86-64s stopped booting when they moved to 2.6.23. It rebooted just after discovering the AGP bridge as a result of the IOMMU init.
It's probably the usual "nobody tests sparsemem at all" issue.
We've been using SPARSEMEM in Fedora for a *long* time.
So long in fact, I forget why we moved away from DISCONTIGMEM, so there's a significant number of users using that configuration for some time.
Quoted 3 lines
But if allocating bootmem >4G doesn't work on these systems most likely they have more problems anyways. It might be better to find out what goes wrong exactly.
Any ideas on what to instrument ?
Dave
Signature · 1 line
http://www.codemonkey.org.uk
Quoted 11 lines · On Monday 29 October 2007 19:47:47 Dave Jones wrote:
Quoted 6 lines · Andi Kleen
Quoted 3 lines · On Monday 29 October 2007 18:50:14 Dave Jones wrote:
We've had a number of people reporting that their x86-64s stopped booting when they moved to 2.6.23. It rebooted just after discovering the AGP bridge as a result of the IOMMU init.
It's probably the usual "nobody tests sparsemem at all" issue.
We've been using SPARSEMEM in Fedora for a *long* time.
So long in fact, I forget why we moved away from DISCONTIGMEM, so there's a significant number of users using that configuration for some time.
Supposedly you wanted a slower kernel that needs more memory?
Ok I wasn't aware of that. I tended to get sparsemem reports usually at least 1-2 releases after the fact, so it looked like it was undertested.
Quoted 6 lines
Quoted 4 lines
But if allocating bootmem >4G doesn't work on these systems most likely they have more problems anyways. It might be better to find out what goes wrong exactly.
Any ideas on what to instrument ?
See what address the bootmem_alloc_high returns; check if it overlaps with something etc.
Fill the memory on the system and see if it can access all of its memory.
-Andi
Quoted 10 lines · Andi Kleen
Quoted 5 lines
Quoted 1 line
It's probably the usual "nobody tests sparsemem at all" issue.
We've been using SPARSEMEM in Fedora for a *long* time.
So long in fact, I forget why we moved away from DISCONTIGMEM, so there's a significant number of users using that configuration for some time.Supposedly you wanted a slower kernel that needs more memory?
Ok I wasn't aware of that. I tended to get sparsemem reports usually at least 1-2 releases after the fact, so it looked like it was undertested.
Looking at cvs history, I can't figure out what the reasoning was, but every Fedora (and RHEL5) kernel since 2006/07/05 has been that way.
Curious how no-one noticed either of the side-effects you mention.
Quoted 9 lines
Quoted 4 lines
Quoted 3 lines
But if allocating bootmem >4G doesn't work on these systems most likely they have more problems anyways. It might be better to find out what goes wrong exactly.
Any ideas on what to instrument ?
See what address the
bootmem_alloc_highreturns; check if it overlaps with something etc.Fill the memory on the system and see if it can access all of its memory.
Martin, as you have one of the affected systems, do you feel up to this?
Dave
Signature · 1 line
http://www.codemonkey.org.uk
Quoted 10 lines · On Monday 29 October 2007 20:43:11 Dave Jones wrote:
Quoted 8 lines · Andi Kleen
Quoted 5 lines
Quoted 1 line
It's probably the usual "nobody tests sparsemem at all" issue.
We've been using SPARSEMEM in Fedora for a *long* time.
So long in fact, I forget why we moved away from DISCONTIGMEM, so there's a significant number of users using that configuration for some time.Supposedly you wanted a slower kernel that needs more memory?
Quoted 7 lines
Quoted 2 lines
Ok I wasn't aware of that. I tended to get sparsemem reports usually at least 1-2 releases after the fact, so it looked like it was undertested.
Looking at cvs history, I can't figure out what the reasoning was, but every Fedora (and RHEL5) kernel since 2006/07/05 has been that way.
Curious how no-one noticed either of the side-effects you mention.
It's a few percent on a few benchmarks iirc. vmemmap (now in .24) was supposed to address that.
-Andi
Quoted 6 lines · Dave Jones
Quoted 5 lines · Andi Kleen
Quoted 4 lines
Quoted 3 lines
But if allocating bootmem >4G doesn't work on these systems most likely they have more problems anyways. It might be better to find out what goes wrong exactly.
Any ideas on what to instrument ?
Quoted 6 lines
Quoted 4 lines
See what address the
bootmem_alloc_highreturns; check if it overlaps with something etc.Fill the memory on the system and see if it can access all of its memory.
Martin, as you have one of the affected systems, do you feel up to this?
Faking a node at 0000000000000000-000000001fff0000 Bootmem setup node 0 0000000000000000-000000001fff0000 sparse_early_mem_map_alloc: returned address ffff81000070b000
My box has 512MB of RAM.
Cheers,
Martin.
Quoted 18 lines · Martin Ebourne
Quoted 6 lines · Dave Jones
Quoted 5 lines · Andi Kleen
Quoted 4 lines
Quoted 3 lines
But if allocating bootmem >4G doesn't work on these systems most likely they have more problems anyways. It might be better to find out what goes wrong exactly.
Any ideas on what to instrument ?
Quoted 6 lines
Quoted 4 lines
See what address the
bootmem_alloc_highreturns; check if it overlaps with something etc.Fill the memory on the system and see if it can access all of its memory.
Martin, as you have one of the affected systems, do you feel up to this?
Faking a node at 0000000000000000-000000001fff0000 Bootmem setup node 0 0000000000000000-000000001fff0000
sparse_early_mem_map_alloc: returned addressffff81000070b000
Quoted 5 lines
My box has 512MB of RAM.
Cheers,
Martin.
Oops, sorry,
seem to be a mistake of me.
I forget to exclude the DMA range.
Does the following patch fix the issue?
Thanks
Zou Nan hai
arch/x86/mm/init_64.c +1 -1
@@ -731,7 +731,7 @@ int in_gate_area_no_task(unsigned long avoid *__init alloc_bootmem_high_node (pg_data_t *pgdat,unsigned long size) {return __alloc_bootmem_core (pgdat->bdata, size, -SMP_CACHE_BYTES , (4UL *1024 *1024 *1024 ),0 ); +SMP_CACHE_BYTES , (4UL *1024 *1024 *1024 ),__pa (MAX_DMA_ADDRESS )); }const char *arch_vma_name (struct vm_area_struct *vma)
Quoted 41 lines · Zou Nan hai
Quoted 18 lines · Martin Ebourne
Quoted 6 lines · Dave Jones
Quoted 5 lines · Andi Kleen
Quoted 4 lines
Quoted 3 lines
But if allocating bootmem >4G doesn't work on these systems most likely they have more problems anyways. It might be better to find out what goes wrong exactly.
Any ideas on what to instrument ?
Quoted 6 lines
Quoted 4 lines
See what address the
bootmem_alloc_highreturns; check if it overlaps with something etc.Fill the memory on the system and see if it can access all of its memory.
Martin, as you have one of the affected systems, do you feel up to this?
Faking a node at 0000000000000000-000000001fff0000 Bootmem setup node 0 0000000000000000-000000001fff0000
sparse_early_mem_map_alloc: returned addressffff81000070b000Quoted 5 lines
My box has 512MB of RAM.
Cheers,
Martin.
Oops, sorry,
seem to be a mistake of me.
I forget to exclude the DMA range.Does the following patch fix the issue?
Thanks
Zou Nan hai
arch/x86/mm/init_64.c int in_gate_area_no_task(unsigned long a · line 737
Quoted 8 lines
arch/x86/mm/init_64.c +1
-1@@ -731,7 +731,7 @@ int in_gate_area_no_task(unsigned long avoid *__init alloc_bootmem_high_node (pg_data_t *pgdat,unsigned long size) {return __alloc_bootmem_core (pgdat->bdata, size, -SMP_CACHE_BYTES , (4UL *1024 *1024 *1024 ),0 ); +SMP_CACHE_BYTES , (4UL *1024 *1024 *1024 ),__pa (MAX_DMA_ADDRESS )); }const char *arch_vma_name (struct vm_area_struct *vma)
Please ignore the patch, the patch is wrong.
However I think the root cause is when __alloc_bootmem_core fail to allocate a memory above 4G it will fall back to allocate from the lowest page.
Then happens to be allocated in DMA region sometimes...
Since this code path is dead, I am OK to revert the patch.
Suresh and I will check the CONFIG_SPARSE_VMEMMAP path. Thanks
Zou Nan hai
Quoted 7 lines · Andi Kleen
Quoted 6 lines · On Monday 29 October 2007 19:47:47 Dave Jones wrote:
Quoted 5 lines · Andi Kleen
Quoted 3 lines · On Monday 29 October 2007 18:50:14 Dave Jones wrote:
We've had a number of people reporting that their x86-64s stopped booting when they moved to 2.6.23. It rebooted just after discovering the AGP bridge as a result of the IOMMU init.
Quoted 7 lines
Quoted 5 lines
Quoted 1 line
It's probably the usual "nobody tests sparsemem at all" issue.
We've been using SPARSEMEM in Fedora for a *long* time.
So long in fact, I forget why we moved away from DISCONTIGMEM, so there's a significant number of users using that configuration for some time.Supposedly you wanted a slower kernel that needs more memory?
Actually if what you say is true, the Kconfig entry for sparsemem could use changing as it suggests the opposite...
This option provides some potential
performance benefits, along with decreased code complexity, but it is newer, and more experimental.
I'm still unclear why exactly we enabled it. The other comment in the Kconfig..
This will be the only option for some systems, including
memory hotplug systems. This is normal.
Sounds unlikely to be the reason, but maybe.
Maybe benchmarking at some point in history showed sparsemem actually beat out discontigmem. I'm at a loss to explain it thanks to a particularly unhelpful changelog entry I wrote at the time.
Dave
Signature · 1 line
http://www.codemonkey.org.uk