Skip to contentSkip to conversation

early_node_mem()'s memory allocation policy

PATCH In review Jeremy Fitzhardinge 3 participants 6 messages Oct 26, 2010 – Oct 28, 2010

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

We're seeing problems under Xen where large portions of the memory
could be reserved (because they're not yet physically present, even though the appear in E820), and the 'start' and 'end' early_node_mem() is choosing is entirely within that reserved range.

Also, the code seems dubious because it adjusts start and end without regarding how much space it is trying to allocate:

	/* extend the search scope */
	end = max_pfn_mapped << PAGE_SHIFT;
	if (end > (MAX_DMA32_PFN<<PAGE_SHIFT))
		start = MAX_DMA32_PFN<<PAGE_SHIFT;
	else
		start = MAX_DMA_PFN<<PAGE_SHIFT;

what if max_pfn_mapped is only a few pages larger than MAX_DMA32_PFN, and that is smaller than the size it is trying to allocate?

I tried just removing the start and end adjustments in early_node_mem() and the kernel booted fine under Xen, but it seemed to allocate at a very low address. Should the for_each_active_range_index_in_nid() loop in find_memory_core_early() be iterating from high to low addresses? If the allocation could be relied on to be top-down, then you wouldn't need to adjust start at all, and it would return the highest available memory in a natural way.

Thanks,
J

#1 lore raw
Quoted 18 lines · Jeremy Fitzhardinge

We're seeing problems under Xen where large portions of the memory
could be reserved (because they're not yet physically present, even though the appear in E820), and the 'start' and 'end' early_node_mem() is choosing is entirely within that reserved range.

Also, the code seems dubious because it adjusts start and end without regarding how much space it is trying to allocate:

	/* extend the search scope */
	end = max_pfn_mapped << PAGE_SHIFT;
	if (end > (MAX_DMA32_PFN<<PAGE_SHIFT))
		start = MAX_DMA32_PFN<<PAGE_SHIFT;
	else
		start = MAX_DMA_PFN<<PAGE_SHIFT;

what if max_pfn_mapped is only a few pages larger than MAX_DMA32_PFN, and that is smaller than the size it is trying to allocate?

Quoted 7 lines

I tried just removing the start and end adjustments in early_node_mem() and the kernel booted fine under Xen, but it seemed to allocate at a very low address. Should the for_each_active_range_index_in_nid() loop in find_memory_core_early() be iterating from high to low addresses? If the allocation could be relied on to be top-down, then you wouldn't need to adjust start at all, and it would return the highest available memory in a natural way.

please check

[PATCH] x86, memblock: Fix early_node_mem with big reserved region.

Jeremy said Xen could reserve huge mem but still show as ram in e820.

early_node_mem could not find range because of start/end adjusting.

Let's use memblock_find_in_range instead ***_node. So get real top down in fallback path.

arch/x86/mm/numa_64.c +2 -5
index 60f4985..7ffc9b7 100644
@@ -178,11 +178,8 @@ static void * __init early_node_mem(int nodeid, unsigned long start,
 
 	/* extend the search scope */
 	end = max_pfn_mapped << PAGE_SHIFT;
-	if (end > (MAX_DMA32_PFN<<PAGE_SHIFT))
-		start = MAX_DMA32_PFN<<PAGE_SHIFT;
-	else
-		start = MAX_DMA_PFN<<PAGE_SHIFT;
-	mem = memblock_x86_find_in_range_node(nodeid, start, end, size, align);
+	start = MAX_DMA_PFN << PAGE_SHIFT;
+	mem = memblock_find_in_range(start, end, size, align);
 	if (mem != MEMBLOCK_ERROR)
 		return __va(mem);
 
#2 lore raw
Quoted 27 lines · Yinghai Lu
Quoted 18 lines · Jeremy Fitzhardinge

We're seeing problems under Xen where large portions of the memory
could be reserved (because they're not yet physically present, even though the appear in E820), and the 'start' and 'end' early_node_mem() is choosing is entirely within that reserved range.

Also, the code seems dubious because it adjusts start and end without regarding how much space it is trying to allocate:

	/* extend the search scope */
	end = max_pfn_mapped << PAGE_SHIFT;
	if (end > (MAX_DMA32_PFN<<PAGE_SHIFT))
		start = MAX_DMA32_PFN<<PAGE_SHIFT;
	else
		start = MAX_DMA_PFN<<PAGE_SHIFT;

what if max_pfn_mapped is only a few pages larger than MAX_DMA32_PFN, and that is smaller than the size it is trying to allocate?

Quoted 7 lines

I tried just removing the start and end adjustments in early_node_mem() and the kernel booted fine under Xen, but it seemed to allocate at a very low address. Should the for_each_active_range_index_in_nid() loop in find_memory_core_early() be iterating from high to low addresses? If the allocation could be relied on to be top-down, then you wouldn't need to adjust start at all, and it would return the highest available memory in a natural way.

Quoted 1 line

please check

It definitly gets us across that hump. Thanks.

Quoted 10 lines

[PATCH] x86, memblock: Fix early_node_mem with big reserved region.

Jeremy said Xen could reserve huge mem but still show as ram in e820.

early_node_mem could not find range because of start/end adjusting.

Let's use memblock_find_in_range instead ***_node. So get real top down in fallback path.

Quoted 19 lines
arch/x86/mm/numa_64.c +2 -5
index 60f4985..7ffc9b7 100644
@@ -178,11 +178,8 @@ static void * __init early_node_mem(int nodeid, unsigned long start,
 
 	/* extend the search scope */
 	end = max_pfn_mapped << PAGE_SHIFT;
-	if (end > (MAX_DMA32_PFN<<PAGE_SHIFT))
-		start = MAX_DMA32_PFN<<PAGE_SHIFT;
-	else
-		start = MAX_DMA_PFN<<PAGE_SHIFT;
-	mem = memblock_x86_find_in_range_node(nodeid, start, end, size, align);
+	start = MAX_DMA_PFN << PAGE_SHIFT;
+	mem = memblock_find_in_range(start, end, size, align);
 	if (mem != MEMBLOCK_ERROR)
 		return __va(mem);
 
#3 lore raw
In reply to Yinghai Lu
Quoted 4 lines · Yinghai Lu

please check

[PATCH] x86, memblock: Fix early_node_mem with big reserved region.

Quoted 8 lines

Jeremy said Xen could reserve huge mem but still show as ram in e820.

early_node_mem could not find range because of start/end adjusting.

Let's use memblock_find_in_range instead ***_node. So get real top down in fallback path.

Yes, that works. Could you queue it for upstream? Without this we see Xen crashes with certain memory configurations.

Thanks,
J

Quoted 19 lines
arch/x86/mm/numa_64.c +2 -5
index 60f4985..7ffc9b7 100644
@@ -178,11 +178,8 @@ static void * __init early_node_mem(int nodeid, unsigned long start,
 
 	/* extend the search scope */
 	end = max_pfn_mapped << PAGE_SHIFT;
-	if (end > (MAX_DMA32_PFN<<PAGE_SHIFT))
-		start = MAX_DMA32_PFN<<PAGE_SHIFT;
-	else
-		start = MAX_DMA_PFN<<PAGE_SHIFT;
-	mem = memblock_x86_find_in_range_node(nodeid, start, end, size, align);
+	start = MAX_DMA_PFN << PAGE_SHIFT;
+	mem = memblock_find_in_range(start, end, size, align);
 	if (mem != MEMBLOCK_ERROR)
 		return __va(mem);
 
#4 lore raw

PATCH 1/1 x86, memblock: Fix early_node_mem with big reserved region.

Jeremy said Xen could reserve huge mem but still show as ram in e820. early_node_mem could not find range because of start/end adjusting. and will go through fallback path. but fallback is still using memblock_x86_find_range_node(), and it is partially top-down because it go through active_range entries from low to high.

Let's use memblock_find_in_range instead memblock_x86_find_range_node. So get real top down in fallback path.

This is for 2.6.37.

And We may still need to make memblock_x86_find_range_node to do overall top_down work.

---
 arch/x86/mm/numa_64.c |    7 ++-----
 1 file changed, 2 insertions(+), 5 deletions(-)
arch/x86/mm/numa_64.c +2 -5
Index: linux-2.6/arch/x86/mm/numa_64.c
===================================================================
@@ -178,11 +178,8 @@ static void * __init early_node_mem(int
 
 	/* extend the search scope */
 	end = max_pfn_mapped << PAGE_SHIFT;
-	if (end > (MAX_DMA32_PFN<<PAGE_SHIFT))
-		start = MAX_DMA32_PFN<<PAGE_SHIFT;
-	else
-		start = MAX_DMA_PFN<<PAGE_SHIFT;
-	mem = memblock_x86_find_in_range_node(nodeid, start, end, size, align);
+	start = MAX_DMA_PFN << PAGE_SHIFT;
+	mem = memblock_find_in_range(start, end, size, align);
 	if (mem != MEMBLOCK_ERROR)
 		return __va(mem);
 
#5 lore raw

[tip:x86/urgent] x86, memblock: Fix early_node_mem with big reserved region.

Commit-ID:  419db274bed4269f475a8e78cbe9c917192cfe8b
Gitweb:     http://git.kernel.org/tip/419db274bed4269f475a8e78cbe9c917192cfe8b
Author:     Yinghai Lu <[email protected]>
AuthorDate: Thu, 28 Oct 2010 09:50:17 -0700
Committer:  H. Peter Anvin <[email protected]>
CommitDate: Thu, 28 Oct 2010 15:52:36 -0700

x86, memblock: Fix early_node_mem with big reserved region.

Xen can reserve huge amounts of memory for pre-ballooning, but that still shows as RAM in the e820 memory map. early_node_mem could not find range because of start/end adjusting, and will go through the fallback path. However, the fallback patch is still using memblock_x86_find_range_node(), and it is partially top-down because it go through active_range entries from low to high.

Let's use memblock_find_in_range instead memblock_x86_find_range_node. So get real top down in fallback path.

We may still need to make memblock_x86_find_range_node to do overall top_down work.

LKML-Reference: <[email protected]>

---
 arch/x86/mm/numa_64.c |    7 ++-----
 1 files changed, 2 insertions(+), 5 deletions(-)
arch/x86/mm/numa_64.c +2 -5
index 60f4985..7ffc9b7 100644
@@ -178,11 +178,8 @@ static void * __init early_node_mem(int nodeid, unsigned long start,
 
 	/* extend the search scope */
 	end = max_pfn_mapped << PAGE_SHIFT;
-	if (end > (MAX_DMA32_PFN<<PAGE_SHIFT))
-		start = MAX_DMA32_PFN<<PAGE_SHIFT;
-	else
-		start = MAX_DMA_PFN<<PAGE_SHIFT;
-	mem = memblock_x86_find_in_range_node(nodeid, start, end, size, align);
+	start = MAX_DMA_PFN << PAGE_SHIFT;
+	mem = memblock_find_in_range(start, end, size, align);
 	if (mem != MEMBLOCK_ERROR)
 		return __va(mem);