x86, memblock: Allocate memory near kernel image before SRAT parsed

+/**
+ * memory_map_top_down - Map [map_start, map_end) top down
+ * @map_start: start address of the target memory range
+ * @map_end: end address of the target memory range
+ *
+ * This function will setup direct mapping for memory range
+ * [map_start, map_end) in top-down.

+/**
+ * memory_map_bottom_up - Map [map_start, map_end) bottom up
+ * @map_start: start address of the target memory range
+ * @map_end: end address of the target memory range
+ *
+ * This function will setup direct mapping for memory range
+ * [map_start, map_end) in bottom-up.

+ */

arch/x86/mm/init.c +23 -0

+static void __init memory_map_bottom_up(unsigned long map_start,
+					unsigned long map_end)
+{
+	unsigned long next, new_mapped_ram_size, start;
+	unsigned long mapped_ram_size = 0;
+	/* step_size need to be small so pgt_buf from BRK could cover it */
+	unsigned long step_size = PMD_SIZE;
+
+	start = map_start;
+	min_pfn_mapped = start >> PAGE_SHIFT;
+
+	/*
+	 * We start from the bottom (@map_start) and go to the top (@map_end).
+	 * The memblock_find_in_range() gets us a block of RAM from the
+	 * end of RAM in [min_pfn_mapped, max_pfn_mapped) used as new pages
+	 * for page table.
+	 */
+	while (start < map_end) {
+		if (map_end - start > step_size) {
+			next = round_up(start + 1, step_size);
+			if (next > map_end)
+				next = map_end;
+		} else

From: Tang Chen <[email protected]>

Memory reserved for crashkernel could be large. So we should not allocate this memory bottom up from the end of kernel image.

When SRAT is parsed, we will be able to know whihc memory is hotpluggable, and we can avoid allocating this memory for the kernel. So reorder reserve_crashkernel() after SRAT is parsed.

Acked-by Tejun Heo <[email protected]>

From: Tang Chen <[email protected]>

The hot-Pluggable field in SRAT specifies which memory is hotpluggable. As we mentioned before, if hotpluggable memory is used by the kernel, it cannot be hot-removed. So memory hotplug users may want to set all hotpluggable memory in ZONE_MOVABLE so that the kernel won't use it.

Memory hotplug users may also set a node as movable node, which has ZONE_MOVABLE only, so that the whole node can be hot-removed.

But the kernel cannot use memory in ZONE_MOVABLE. By doing this, the kernel cannot use memory in movable nodes. This will cause NUMA performance down. And other users may be unhappy.

So we need a way to allow users to enable and disable this functionality. In this patch, we introduce movablenode boot option to allow users to choose to not to consume hotpluggable memory at early boot time and later we can set it as ZONE_MOVABLE.

To achieve this, the movablenode boot option will control the memblock allocation direction. That said, after memblock is ready, before SRAT is parsed, we should allocate memory near the kernel image as we explained in the previous patches. So if movablenode boot option is set, the kernel does the following:

1. After memblock is ready, make memblock allocate memory bottom up.
2. After SRAT is parsed, make memblock behave as default, allocate memory top down.

Users can specify "movablenode" in kernel commandline to enable this functionality. For those who don't use memory hotplug or who don't want to lose their NUMA performance, just don't specify anything. The kernel will work as before.

Suggested-by Kamezawa Hiroyuki <[email protected]>

Signed-off-by Tang Chen <[email protected]>

Signed-off-by Zhang Yanfei <[email protected]>

Hello,

include/linux/memblock.h phys_addr_t memblock_alloc_try_nid(phys_addr_t size, phys_addr_t align, int nid) · line 161

Quoted 10 lines · Zhang Yanfei

include/linux/memblock.h +10 -0

+#ifdef CONFIG_MOVABLE_NODE
+static inline void memblock_set_bottom_up(bool enable)
+{
+	memblock.bottom_up = enable;
+}
+
+static inline bool memblock_bottom_up(void)
+{
+	return memblock.bottom_up;
+}

Can you please explain what this is for here?

Quoted 10 lines

mm/memblock.c +9 -0

+		/*
+		 * we always limit bottom-up allocation above the kernel,
+		 * but top-down allocation doesn't have the limit, so
+		 * retrying top-down allocation may succeed when bottom-up
+		 * allocation failed.
+		 *
+		 * bottom-up allocation is expected to be fail very rarely,
+		 * so we use WARN_ONCE() here to see the stack trace if
+		 * fail happens.

mm/memblock.c phys_addr_t __init_memblock memblock_find_in_range_node(phys_addr_t start, · line 220

Quoted 4 lines

mm/memblock.c +4 -0

+		 */
+		WARN_ONCE(1, "memblock: Failed to allocate memory in bottom up "
+			"direction. Now try top down direction.\n");
+	}

You and I would know what was going on and what the consequence of the failure may be but the above warning message is kinda useless to a user / admin, right? It doesn't really say anything meaningful.

Quoted 7 lines · Zhang Yanfei

+/**
+ * memory_map_top_down - Map [map_start, map_end) top down
+ * @map_start: start address of the target memory range
+ * @map_end: end address of the target memory range
+ *
+ * This function will setup direct mapping for memory range
+ * [map_start, map_end) in top-down.

Can you please put a bit more effort into the function description?

Other than that,

Acked-by Tejun Heo <[email protected]>

Hello,

Quoted 7 lines · Zhang Yanfei

+/**
+ * memory_map_bottom_up - Map [map_start, map_end) bottom up
+ * @map_start: start address of the target memory range
+ * @map_end: end address of the target memory range
+ *
+ * This function will setup direct mapping for memory range
+ * [map_start, map_end) in bottom-up.

Ditto about the comment.

Quoted 30 lines

+ */

arch/x86/mm/init.c +28 -0

+static void __init memory_map_bottom_up(unsigned long map_start,
+					unsigned long map_end)
+{
+	unsigned long next, new_mapped_ram_size, start;
+	unsigned long mapped_ram_size = 0;
+	/* step_size need to be small so pgt_buf from BRK could cover it */
+	unsigned long step_size = PMD_SIZE;
+
+	start = map_start;
+	min_pfn_mapped = start >> PAGE_SHIFT;
+
+	/*
+	 * We start from the bottom (@map_start) and go to the top (@map_end).
+	 * The memblock_find_in_range() gets us a block of RAM from the
+	 * end of RAM in [min_pfn_mapped, max_pfn_mapped) used as new pages
+	 * for page table.
+	 */
+	while (start < map_end) {
+		if (map_end - start > step_size) {
+			next = round_up(start + 1, step_size);
+			if (next > map_end)
+				next = map_end;
+		} else
+			next = map_end;
+
+		new_mapped_ram_size = init_range_memory_mapping(start, next);
+		start = next;
+

arch/x86/mm/init.c static void __init memory_map_top_down(unsigned long map_start, · line 499

Quoted 5 lines

arch/x86/mm/init.c +5 -0

+		if (new_mapped_ram_size > mapped_ram_size)
+			step_size <<= STEP_SIZE_SHIFT;
+		mapped_ram_size += new_mapped_ram_size;
+	}
+}

As Yinghai pointed out in another thread, do we need to worry about falling back to top-down?

Quoted 5 lines · Zhang Yanfei

From: Tang Chen <[email protected]>

Memory reserved for crashkernel could be large. So we should not allocate this memory bottom up from the end of kernel image.

Quoted 5 lines

When SRAT is parsed, we will be able to know whihc memory is hotpluggable, and we can avoid allocating this memory for the kernel. So reorder reserve_crashkernel() after SRAT is parsed.

Acked-by Tejun Heo <[email protected]>

So, I was hoping to hear from you on how you tested it when I wrote the previous comment - the "provided..." part.

Quoted 2 lines

As Yinghai pointed out in another thread, do we need to worry about falling back to top-down?

I've explained to him. Nop, we don't need to worry about that. Because even the min_pfn_mapped becomes ISA_END_ADDRESS in the second call below, we won't allocate memory below the kernel because we have limited the allocation above the kernel.

Quoted 8 lines

mm/memblock.c phys_addr_t __init_memblock memblock_find_in_range_node(phys_addr_t start, · line 220

Quoted 3 lines

mm/memblock.c +3 -0

+		WARN_ONCE(1, "memblock: Failed to allocate memory in bottom up "
+			"direction. Now try top down direction.\n");
+	}

You and I would know what was going on and what the consequence of the failure may be but the above warning message is kinda useless to a user / admin, right? It doesn't really say anything meaningful.

Hmmmm.. May be something like this:

WARN_ONCE(1, "Failed to allocated memory above the kernel in bottom-up,"

          "so try to allocate memory below the kernel.");

Quoted 9 lines · Zhang Yanfei

Quoted 8 lines

mm/memblock.c phys_addr_t __init_memblock memblock_find_in_range_node(phys_addr_t start, · line 220

Quoted 3 lines

mm/memblock.c +3 -0

+		WARN_ONCE(1, "memblock: Failed to allocate memory in bottom up "
+			"direction. Now try top down direction.\n");
+	}

You and I would know what was going on and what the consequence of the failure may be but the above warning message is kinda useless to a user / admin, right? It doesn't really say anything meaningful.

Quoted 4 lines

Hmmmm.. May be something like this:

WARN_ONCE(1, "Failed to allocated memory above the kernel in bottom-up,"

          "so try to allocate memory below the kernel.");

How about something like "memblock: bottom-up allocation failed, memory hotunplug may be affected\n".

Quoted 3 lines · Zhang Yanfei

Quoted 2 lines

As Yinghai pointed out in another thread, do we need to worry about falling back to top-down?

Quoted 4 lines

I've explained to him. Nop, we don't need to worry about that. Because even the min_pfn_mapped becomes ISA_END_ADDRESS in the second call below, we won't allocate memory below the kernel because we have limited the allocation above the kernel.

Maybe I misunderstood but wasn't he worrying about there not being enough space above kernel? In that case, it'd automatically fall back to top-down allocation anyway, right?

Ah, I see. You are saying another issue. He is worrying that if we use kexec to load the kernel high, say we have 16GB, we put the kernel in 15.99GB (just an example), so we only have less than 100MB above the kernel.

But as I've explained to him, in almost all the cases, if we want our memory hotplug work, we don't do that. And yeah, assume we have this problem, it'd fall back to top down and that return backs to patch 2, we will trigger the WARN_ONCE, and the admin will know what has happened.

Quoted 34 lines · Zhang Yanfei

From: Tang Chen <[email protected]>

The hot-Pluggable field in SRAT specifies which memory is hotpluggable. As we mentioned before, if hotpluggable memory is used by the kernel, it cannot be hot-removed. So memory hotplug users may want to set all hotpluggable memory in ZONE_MOVABLE so that the kernel won't use it.

Memory hotplug users may also set a node as movable node, which has ZONE_MOVABLE only, so that the whole node can be hot-removed.

But the kernel cannot use memory in ZONE_MOVABLE. By doing this, the kernel cannot use memory in movable nodes. This will cause NUMA performance down. And other users may be unhappy.

So we need a way to allow users to enable and disable this functionality. In this patch, we introduce movablenode boot option to allow users to choose to not to consume hotpluggable memory at early boot time and later we can set it as ZONE_MOVABLE.

To achieve this, the movablenode boot option will control the memblock allocation direction. That said, after memblock is ready, before SRAT is parsed, we should allocate memory near the kernel image as we explained in the previous patches. So if movablenode boot option is set, the kernel does the following:

1. After memblock is ready, make memblock allocate memory bottom up.
2. After SRAT is parsed, make memblock behave as default, allocate memory top down.

Users can specify "movablenode" in kernel commandline to enable this functionality. For those who don't use memory hotplug or who don't want to lose their NUMA performance, just don't specify anything. The kernel will work as before.

Quoted 3 lines

Suggested-by Kamezawa Hiroyuki <[email protected]>

Signed-off-by Tang Chen <[email protected]>

Signed-off-by Zhang Yanfei <[email protected]>

I hope the param description and comment were better. Not necessarily longer, but clearer, so it'd be great if you can polish them a bit

more. Other than that,

Acked-by Tejun Heo <[email protected]>

Thanks.

Hello tejun,

Thanks for your quick comments first:)

Quoted 15 lines · Tejun Heo

Hello,

include/linux/memblock.h phys_addr_t memblock_alloc_try_nid(phys_addr_t size, phys_addr_t align, int nid) · line 161

Quoted 10 lines · Zhang Yanfei

include/linux/memblock.h +10 -0

+#ifdef CONFIG_MOVABLE_NODE
+static inline void memblock_set_bottom_up(bool enable)
+{
+	memblock.bottom_up = enable;
+}
+
+static inline bool memblock_bottom_up(void)
+{
+	return memblock.bottom_up;
+}

Can you please explain what this is for here?

OK, will do.

Quoted 9 lines · Tejun Heo

Quoted 7 lines · Zhang Yanfei

+/**
+ * memory_map_top_down - Map [map_start, map_end) top down
+ * @map_start: start address of the target memory range
+ * @map_end: end address of the target memory range
+ *
+ * This function will setup direct mapping for memory range
+ * [map_start, map_end) in top-down.

Quoted 1 line

Can you please put a bit more effort into the function description?

Sorry.... I will try to make a more detailed description.

Quoted 5 lines

Other than that,

Acked-by Tejun Heo <[email protected]>

Thanks.

Hello,

Quoted 7 lines · Zhang Yanfei

+/**
+ * memory_map_bottom_up - Map [map_start, map_end) bottom up
+ * @map_start: start address of the target memory range
+ * @map_end: end address of the target memory range
+ *
+ * This function will setup direct mapping for memory range
+ * [map_start, map_end) in bottom-up.

Ditto about the comment.

From: Tang Chen <[email protected]>

The Linux kernel cannot migrate pages used by the kernel. As a result, kernel pages cannot be hot-removed. So we cannot allocate hotpluggable memory for the kernel.

In a memory hotplug system, any numa node the kernel resides in should be unhotpluggable. And for a modern server, each node could have at least 16GB memory. So memory around the kernel image is highly likely unhotpluggable.

ACPI SRAT (System Resource Affinity Table) contains the memory hotplug info. But before SRAT is parsed, memblock has already started to allocate memory for the kernel. So we need to prevent memblock from doing this.

So direct memory mapping page tables setup is the case. init_mem_mapping() is called before SRAT is parsed. To prevent page tables being allocated within hotpluggable memory, we will use bottom-up direction to allocate page tables from the end of kernel image to the higher memory.

...

Quoted 11 lines · On Wed, 25 Sep 2013 02:30:51 +0800 Zhang Yanfei <[email protected]>…

From: Tang Chen <[email protected]>

The Linux kernel cannot migrate pages used by the kernel. As a result, kernel pages cannot be hot-removed. So we cannot allocate hotpluggable memory for the kernel.

In a memory hotplug system, any numa node the kernel resides in should be unhotpluggable. And for a modern server, each node could have at least 16GB memory. So memory around the kernel image is highly likely unhotpluggable.

Quoted 6 lines

ACPI SRAT (System Resource Affinity Table) contains the memory hotplug info. But before SRAT is parsed, memblock has already started to allocate memory for the kernel. So we need to prevent memblock from doing this.

So direct memory mapping page tables setup is the case.

init_mem_mapping()

Quoted 1 line

is called before SRAT is parsed. To prevent page tables being

allocated

Quoted 1 line

within hotpluggable memory, we will use bottom-up direction to

allocate

Review

Quoted 5 lines

page tables from the end of kernel image to the higher memory.

...

(quoted diff) +1 -0

+		kernel_end = __pa_symbol(_end);

__pa_symbol() is implemented only on mips and x86.

I stole the mips implementation like this:

mm/memblock.c renamed from mm/memblock.c~a +4 -1

@@ -187,8 +187,11 @@ phys_addr_t __init_memblock memblock_fin
 	/* avoid allocating the first page */
 	start = max_t(phys_addr_t, start, PAGE_SIZE);
 	end = max(start, end);
+#ifdef CONFIG_X86
 	kernel_end = __pa_symbol(_end);
-
+#else
+	kernel_end = __pa(RELOC_HIDE((unsigned long)(_end), 0));
+#endif
 	/*
 	 * try bottom-up allocation only when bottom-up mode
 	 * is set and @end is above the kernel image.

just so I can get a -mm release out the door.

OK. Trying below:

movablenode [KNL,X86] This option enables the kernel to arrange
hotpluggable memory into ZONE_MOVABLE zone. If memory in a node is all hotpluggable, the option may make the whole node has only one ZONE_MOVABLE zone, so that the whole node can be hot-removed after system is up. Note that this option may cause NUMA performance down.

Quoted 2 lines · Zhang Yanfei

OK. Trying below:

Quoted 6 lines

movablenode [KNL,X86] This option enables the kernel to arrange
hotpluggable memory into ZONE_MOVABLE zone. If memory in a node is all hotpluggable, the option may make the whole node has only one ZONE_MOVABLE zone, so that the whole node can be hot-removed after system is up. Note that this option may cause NUMA performance down.

That paragraph doesn't really parse in several places ...

Also, more importantly, please explain why this needs to be a boot option. In terms of user friendliness boot options are at the bottom of the list, and boot options also don't really help feature tests.

Presumably the feature is safe and has no costs, and hence could be added as a regular .config option, with a boot option only as an additional configurability option?

Thanks,

Ingo

Quoted 4 lines

Also, more importantly, please explain why this needs to be a boot option. In terms of user friendliness boot options are at the bottom of the list, and boot options also don't really help feature tests.

Quoted 3 lines

Presumably the feature is safe and has no costs, and hence could be added as a regular .config option, with a boot option only as an additional configurability option?

Yeah, the kernel already has config MOVABLE_NODE, which is the config enabing this feature, and we introduce this boot option to expand the configurability.

Documentation/kernel-parameters.txt +6 -0

+     movablenode             [KNL,X86] This parameter enables/disables the
+                     kernel to arrange hotpluggable memory ranges recorded
+                     in ACPI SRAT(System Resource Affinity Table) as
+                     ZONE_MOVABLE. And these memory can be hot-removed when
+                     the system is up.
+                     By specifying this option, all the hotpluggable memory

From: Tang Chen <[email protected]>

This patch creates a new function __memblock_find_range_rev to factor out of top-down allocation from memblock_find_in_range_node. This is a preparation because we will introduce a new bottom-up allocation mode in the following patch.

Acked-by Tejun Heo <[email protected]>

Signed-off-by Tang Chen <[email protected]>

Signed-off-by Zhang Yanfei <[email protected]>

---
 mm/memblock.c |   47 ++++++++++++++++++++++++++++++++++-------------
 1 files changed, 34 insertions(+), 13 deletions(-)

mm/memblock.c +1 -2

index 0ac412a..3d80c74 100644
@@ -83,33 +83,25 @@ static long __init_memblock memblock_overlaps_region(struct memblock_type *type,
 }
 
 /**
- * memblock_find_in_range_node - find free area in given range and node
+ * __memblock_find_range_rev - find free area utility, in reverse order
  * @start: start of candidate range
  * @end: end of candidate range, can be %MEMBLOCK_ALLOC_{ANYWHERE|ACCESSIBLE}
  * @size: size of free area to find
  * @align: alignment of free area to find
  * @nid: nid of the free area to find, %MAX_NUMNODES for any node
  *
- * Find @size free area aligned to @align in the specified range and node.

From: Tang Chen <[email protected]>

The Linux kernel cannot migrate pages used by the kernel. As a result, kernel pages cannot be hot-removed. So we cannot allocate hotpluggable memory for the kernel.

ACPI SRAT (System Resource Affinity Table) contains the memory hotplug info. But before SRAT is parsed, memblock has already started to allocate memory for the kernel. So we need to prevent memblock from doing this.

In a memory hotplug system, any numa node the kernel resides in should be unhotpluggable. And for a modern server, each node could have at least 16GB memory. So memory around the kernel image is highly likely unhotpluggable.

So the basic idea is: Allocate memory from the end of the kernel image and to the higher memory. Since memory allocation before SRAT is parsed won't be too much, it could highly likely be in the same node with kernel image.

The current memblock can only allocate memory top-down. So this patch introduces a new bottom-up allocation mode to allocate memory bottom-up. And later when we use this allocation direction to allocate memory, we will limit the start address above the kernel.

Signed-off-by Tang Chen <[email protected]>

Signed-off-by Zhang Yanfei <[email protected]>

/**
+ * __memblock_find_range - find free area utility
+ * @start: start of candidate range
+ * @end: end of candidate range, can be %MEMBLOCK_ALLOC_{ANYWHERE|ACCESSIBLE}
+ * @size: size of free area to find
+ * @align: alignment of free area to find
+ * @nid: nid of the free area to find, %MAX_NUMNODES for any node
+ *
+ * Utility called from memblock_find_in_range_node(), find free area bottom-up.
+ *
+ * RETURNS:
+ * Found address on success, 0 on failure.
+ */

mm/memblock.c +31 -2

+		      phys_addr_t align, int nid)
+{
+	phys_addr_t this_start, this_end, cand;
+	u64 i;
+
+	for_each_free_mem_range(i, nid, &this_start, &this_end, NULL) {
+		this_start = clamp(this_start, start, end);
+		this_end = clamp(this_end, start, end);
+
+		cand = round_up(this_start, align);
+		if (cand < this_end && this_end - cand >= size)
+			return cand;
+	}
+
+	return 0;
+}
+
+/**
  * __memblock_find_range_rev - find free area utility, in reverse order
  * @start: start of candidate range
  * @end: end of candidate range, can be %MEMBLOCK_ALLOC_{ANYWHERE|ACCESSIBLE}

@@ -93,7 +128,7 @@ static long __init_memblock memblock_overlaps_region(struct memblock_type *type,
  * Utility called from memblock_find_in_range_node(), find free area top-down.
  *
  * RETURNS:
- * Found address on success, %0 on failure.
+ * Found address on success, 0 on failure.
  */
 static phys_addr_t __init_memblock
 __memblock_find_range_rev(phys_addr_t start, phys_addr_t end,

@@ -127,13 +162,24 @@ __memblock_find_range_rev(phys_addr_t start, phys_addr_t end,
  *
  * Find @size free area aligned to @align in the specified range and node.
  *
+ * When allocation direction is bottom-up, the @start should be greater
+ * than the end of the kernel image. Otherwise, it will be trimmed. The
+ * reason is that we want the bottom-up allocation just near the kernel
+ * image so it is highly likely that the allocated memory and the kernel
+ * will reside in the same node.
+ *
+ * If bottom-up allocation failed, will try to allocate memory top-down.
+ *
  * RETURNS:
- * Found address on success, %0 on failure.
+ * Found address on success, 0 on failure.
  */
 phys_addr_t __init_memblock memblock_find_in_range_node(phys_addr_t start,
 					phys_addr_t end, phys_addr_t size,
 					phys_addr_t align, int nid)
 {
+	int ret;
+	phys_addr_t kernel_end;
+
 	/* pump up @end */
 	if (end == MEMBLOCK_ALLOC_ACCESSIBLE)
 		end = memblock.current_limit;

From: Tang Chen <[email protected]>

This patch creates a new function memory_map_top_down to factor out of the top-down direct memory mapping pagetable setup. This is also a preparation for the following patch, which will introduce the bottom-up memory mapping. That said, we will put the two ways of pagetable setup into separate functions, and choose to use which way in init_mem_mapping, which makes the code more clear.

Signed-off-by Tang Chen <[email protected]>

Signed-off-by Zhang Yanfei <[email protected]>

From: Tang Chen <[email protected]>

Memory reserved for crashkernel could be large. So we should not allocate this memory bottom up from the end of kernel image.

When SRAT is parsed, we will be able to know whihc memory is hotpluggable, and we can avoid allocating this memory for the kernel. So reorder reserve_crashkernel() after SRAT is parsed.

Acked-by Tejun Heo <[email protected]>

Signed-off-by Tang Chen <[email protected]>

Reviewed-by Zhang Yanfei <[email protected]>

---
 arch/x86/kernel/setup.c |    8 ++++++--
 1 files changed, 6 insertions(+), 2 deletions(-)

arch/x86/kernel/setup.c +0 -2

index f0de629..36cfce3 100644
@@ -1120,8 +1120,6 @@ void __init setup_arch(char **cmdline_p)
 	acpi_initrd_override((void *)initrd_start, initrd_end - initrd_start);
 #endif
 
-	reserve_crashkernel();
-
 	vsmp_init();
 
 	io_delay_init();