mem-hotplug: let memblock skip the hotpluggable memory regions in __next_mem_range()

PATCH Superseded Xishi Qiu 3 participants 8 messages Aug 11, 2014 – Aug 18, 2014

Subsystems (2)

Memory management memblock and memory management initialization

lore

Share & tools

lore

Xishi Qiu

author

2014-08-11 13:32 UTC #0 lore raw

PATCH 1/1 mem-hotplug: let memblock skip the hotpluggable memory regions in __next_mem_range()

Let memblock skip the hotpluggable memory regions in __next_mem_range(), it is used to to prevent memblock from allocating hotpluggable memory for the kernel at early time. The code is the same as __next_mem_range_rev().

Clear hotpluggable flag before releasing free pages to the buddy allocator.

Signed-off-by Xishi Qiu <[email protected]>

---
 mm/memblock.c  |    4 ++++
 mm/nobootmem.c |    2 ++
 2 files changed, 6 insertions(+), 0 deletions(-)

mm/memblock.c +4 -0

index 6d2f219..5090050 100644
@@ -817,6 +817,10 @@ void __init_memblock __next_mem_range(u64 *idx, int nid,
 		if (nid != NUMA_NO_NODE && nid != m_nid)
 			continue;
 
+		/* skip hotpluggable memory regions if needed */
+		if (movable_node_is_enabled() && memblock_is_hotpluggable(m))
+			continue;
+
 		if (!type_b) {
 			if (out_start)
 				*out_start = m_start;

mm/nobootmem.c +2 -0

index 7ed5860..03de286 100644
@@ -119,6 +119,8 @@ static unsigned long __init free_low_memory_core_early(void)
 	phys_addr_t start, end;
 	u64 i;
 
+	memblock_clear_hotplug(0, ULLONG_MAX);
+
 	for_each_free_mem_range(i, NUMA_NO_NODE, &start, &end, NULL)
 		count += __free_memory_core(start, end);

Signature · 1 line

1.7.1

Tejun Heo

2014-08-16 13:05 UTC #1 lore raw

Quoted 5 lines · Xishi Qiu

Let memblock skip the hotpluggable memory regions in __next_mem_range(), it is used to to prevent memblock from allocating hotpluggable memory for the kernel at early time. The code is the same as __next_mem_range_rev().
Clear hotpluggable flag before releasing free pages to the buddy allocator.

Please try to explain "why" in addition to "what". Why do we need to clear hotpluggable flag in free_low_memory_core_early() in addition to numa_clear_node_hotplug() in x86 numa.c? Does this make x86 code redundant? If not, why?

Signature · 7 lines

tejun

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Xishi Qiu

author

2014-08-16 14:39 UTC #2 lore raw

Quoted 5 lines · Tejun Heo

Quoted 4 lines · Xishi Qiu
Let memblock skip the hotpluggable memory regions in __next_mem_range(), it is used to to prevent memblock from allocating hotpluggable memory for the kernel at early time. The code is the same as __next_mem_range_rev().

Quoted 7 lines

Quoted 1 line
Clear hotpluggable flag before releasing free pages to the buddy allocator.
Please try to explain "why" in addition to "what". Why do we need to clear hotpluggable flag in free_low_memory_core_early() in addition to numa_clear_node_hotplug() in x86 numa.c? Does this make x86 code redundant? If not, why?

Hi Tejun,

numa_clear_node_hotplug()? There is only numa_clear_kernel_node_hotplug().

If we don't clear hotpluggable flag in free_low_memory_core_early(), the memory which marked hotpluggable flag will not free to buddy allocator. Because __next_mem_range() will skip them.

free_low_memory_core_early
for_each_free_mem_range
for_each_mem_range
__next_mem_range

Thanks,
Xishi Qiu

Tejun Heo

2014-08-17 11:08 UTC #3 lore raw

Hello,

Quoted 1 line · Xishi Qiu

numa_clear_node_hotplug()? There is only numa_clear_kernel_node_hotplug().

Yeah, that one.

Quoted 8 lines

If we don't clear hotpluggable flag in free_low_memory_core_early(), the memory which marked hotpluggable flag will not free to buddy allocator. Because __next_mem_range() will skip them.
free_low_memory_core_early
for_each_free_mem_range
for_each_mem_range
__next_mem_range

Ah, okay, so the patch fixes __next_mem_range() and thus makes free_low_memory_core_early() to skip hotpluggable regions unlike before. Please explain things like that in the changelog. Also, what's its relationship with numa_clear_kernel_node_hotplug()? Do we still need them? If so, what are the different roles that these two separate places serve?

Thanks.

Signature · 7 lines

tejun

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

tangchen

2014-08-18 01:12 UTC #4 lore raw

Hi tj,

Quoted 14 lines · Tejun Heo

Hello,
Quoted 1 line · Xishi Qiu
numa_clear_node_hotplug()? There is only numa_clear_kernel_node_hotplug().
Yeah, that one.
Quoted 8 lines
If we don't clear hotpluggable flag in free_low_memory_core_early(), the memory which marked hotpluggable flag will not free to buddy allocator. Because __next_mem_range() will skip them.
free_low_memory_core_early
for_each_free_mem_range
for_each_mem_range
__next_mem_range

Quoted 6 lines

Ah, okay, so the patch fixes __next_mem_range() and thus makes free_low_memory_core_early() to skip hotpluggable regions unlike before. Please explain things like that in the changelog. Also, what's its relationship with numa_clear_kernel_node_hotplug()? Do we still need them? If so, what are the different roles that these two separate places serve?

numa_clear_kernel_node_hotplug() only clears hotplug flags for the nodes the kernel resides in, not for hotpluggable nodes. The reason why we did this is to enable the kernel to allocate memory in case all the nodes are hotpluggable.

And we clear hotplug flags for all the nodes in free_low_memory_core_early() is because if we do not, all hotpluggable memory won't be able to be freed to buddy after Qiu's patch.

Thanks.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Xishi Qiu

author

2014-08-18 03:21 UTC #5 lore raw

Quoted 24 lines · tangchen

Hi tj,
Quoted 14 lines · Tejun Heo
Hello,
Quoted 1 line · Xishi Qiu
numa_clear_node_hotplug()? There is only numa_clear_kernel_node_hotplug().
Yeah, that one.
Quoted 8 lines
If we don't clear hotpluggable flag in free_low_memory_core_early(), the memory which marked hotpluggable flag will not free to buddy allocator. Because __next_mem_range() will skip them.
free_low_memory_core_early
for_each_free_mem_range
for_each_mem_range
__next_mem_range
Quoted 6 lines
Ah, okay, so the patch fixes __next_mem_range() and thus makes free_low_memory_core_early() to skip hotpluggable regions unlike before. Please explain things like that in the changelog. Also, what's its relationship with numa_clear_kernel_node_hotplug()? Do we still need them? If so, what are the different roles that these two separate places serve?

Quoted 5 lines

numa_clear_kernel_node_hotplug() only clears hotplug flags for the nodes the kernel resides in, not for hotpluggable nodes. The reason why we did this is to enable the kernel to allocate memory in case all the nodes are hotpluggable.

Hi TangChen,

I find a problem in numa_init() (arch/x86/mm/numa.c) numa_init()

	...
	ret = init_func();  // this will mark hotpluggable flag from SRAT
	...
	memblock_set_bottom_up(false);
	...
	ret = numa_register_memblks(&numa_meminfo);  // this will alloc node data(pglist_data) 
	...
	numa_clear_kernel_node_hotplug();  // in case all the nodes are hotpluggable
	...

If all the nodes are marked hotpluggable flag, alloc node data will fail. Because __next_mem_range_rev() will skip the hotpluggable memory regions. numa_register_memblks()

	setup_node_data()
		memblock_find_in_range_node()
			__memblock_find_range_top_down()
				for_each_mem_range_rev()
					__next_mem_range_rev()

What do you think?
How about move numa_clear_kernel_node_hotplug() into numa_register_memblks(), like this:

numa_register_memblks()

...

                memblock_set_node(mb->start, mb->end - mb->start,
                                  &memblock.reserved, mb->nid);
        }

+ numa_clear_kernel_node_hotplug();

/*
* If sections array is gonna be used for pfn -> nid mapping, check
...

Thanks,
Xishi Qiu

Quoted 9 lines

And we clear hotplug flags for all the nodes in free_low_memory_core_early() is because if we do not, all hotpluggable memory won't be able to be freed to buddy after Qiu's patch.
Thanks.
.

Tejun Heo

2014-08-18 13:13 UTC #6 lore raw

Hello, Xishi, Tang.

Quoted 8 lines · Xishi Qiu

If all the nodes are marked hotpluggable flag, alloc node data will fail. Because __next_mem_range_rev() will skip the hotpluggable memory regions. numa_register_memblks()
	setup_node_data()
		memblock_find_in_range_node()
			__memblock_find_range_top_down()
				for_each_mem_range_rev()
					__next_mem_range_rev()

I'm not sure clearing hotplug flag for all memory is the best approach here. The problem is that there are places where we want to be selectively ignoring the hotplug status and apparently we may want it back later. Why not add an agument to memblock allocation / iteration functions so that hotplug area can be skipped selectively?

Thanks.

Signature · 7 lines

tejun

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Xishi Qiu

author

2014-08-18 02:02 UTC #7 lore raw

In reply to Tejun Heo

Quoted 11 lines · Tejun Heo

Hello,
Quoted 1 line · Xishi Qiu
numa_clear_node_hotplug()? There is only numa_clear_kernel_node_hotplug().
Yeah, that one.
Quoted 4 lines
If we don't clear hotpluggable flag in free_low_memory_core_early(), the memory which marked hotpluggable flag will not free to buddy allocator. Because __next_mem_range() will skip them.

Quoted 8 lines

Quoted 4 lines
free_low_memory_core_early
for_each_free_mem_range
for_each_mem_range
__next_mem_range
Ah, okay, so the patch fixes __next_mem_range() and thus makes free_low_memory_core_early() to skip hotpluggable regions unlike before. Please explain things like that in the changelog. Also,

OK, I will send V2.

Thanks,
Xishi Qiu

Quoted 6 lines

what's its relationship with numa_clear_kernel_node_hotplug()? Do we still need them? If so, what are the different roles that these two separate places serve?
Thanks.