memory-hotplug : notification of memoty block's state

PATCH In review Yasuaki Ishimatsu 3 participants 9 messages Oct 2, 2012 – Oct 3, 2012

Subsystems (4)

Memory management ACPI Driver core Memory hot(un)plug

lore

Share & tools

lore

Yasuaki Ishimatsu

author

2012-10-02 08:25 UTC #0 lore raw

cover memory-hotplug : notification of memoty block's state

We are trying to implement a physical memory hot removing function as following thread.

https://lkml.org/lkml/2012/9/5/201

But there is not enough review to merge into linux kernel.

I think there are following blockades.
1. no physical memory hot removable system
2. huge patch-set

If you have a KVM system, we can get rid of 1st blockade. Because applying following patch, we can create memory hot removable system on KVM guest.

http://lists.gnu.org/archive/html/qemu-devel/2012-07/msg01389.html

2nd blockade is own problem. So we try to divide huge patch into a small patch in each function as follows:

- bug fix
- acpi framework
- kernel core

We had already sent bug fix patches.

https://lkml.org/lkml/2012/9/27/39

And the patch fixes following bug.

remove_memory() offlines memory. And it is called by following two cases:

1. echo offline >/sys/devices/system/memory/memoryXX/state
2. hot remove a memory device

In the 1st case, the memory block's state is changed and the notification that memory block's state changed is sent to userland after calling offline_memory(). So user can notice memory block is changed.

But in the 2nd case, the memory block's state is not changed and the notification is not also sent to userspcae even if calling offline_memory(). So user cannot notice memory block is changed.

We should also notify to userspace at 2nd case.

Yasuaki Ishimatsu

author

2012-10-02 08:27 UTC #1 lore raw

PATCH 1/2 memory-hotplug : Preparation to notify memory block's state at memory hot remove

From: Wen Congyang <[email protected]>

remove_memory() is called in two cases:
1. echo offline >/sys/devices/system/memory/memoryXX/state
2. hot remove a memory device

In the 1st case, the memory block's state is changed and the notification that memory block's state changed is sent to userland after calling remove_memory(). So user can notice memory block is changed.

But in the 2nd case, the memory block's state is not changed and the notification is not also sent to userspcae even if calling remove_memory(). So user cannot notice memory block is changed.

For adding the notification at memory hot remove, the patch just prepare as follows:
1st case uses offline_pages() for offlining memory.
2nd case uses remove_memory() for offlining memory and changing memory block's state and notifing the information.

The patch does not implement notification to remove_memory().

CC David Rientjes <[email protected]>

CC Jiang Liu <[email protected]>

CC Len Brown <[email protected]>

CC Christoph Lameter <[email protected]>

Cc Minchan Kim <[email protected]>

CC Andrew Morton <[email protected]>

CC KOSAKI Motohiro <[email protected]>

Signed-off-by Wen Congyang <[email protected]>

Signed-off-by Yasuaki Ishimatsu <[email protected]>

---
 drivers/base/memory.c          |    9 +++------
 include/linux/memory_hotplug.h |    1 +
 mm/memory_hotplug.c            |   13 +++++++++++--
 3 files changed, 15 insertions(+), 8 deletions(-)

drivers/base/memory.c +3 -6

Index: linux-3.6/drivers/base/memory.c
===================================================================
@@ -248,26 +248,23 @@ static bool pages_correctly_reserved(uns
 static int
 memory_block_action(unsigned long phys_index, unsigned long action)
 {
-	unsigned long start_pfn, start_paddr;
+	unsigned long start_pfn;
 	unsigned long nr_pages = PAGES_PER_SECTION * sections_per_block;
 	struct page *first_page;
 	int ret;
 
 	first_page = pfn_to_page(phys_index << PFN_SECTION_SHIFT);
+	start_pfn = page_to_pfn(first_page);
 
 	switch (action) {
 		case MEM_ONLINE:
-			start_pfn = page_to_pfn(first_page);
-
 			if (!pages_correctly_reserved(start_pfn, nr_pages))
 				return -EBUSY;
 
 			ret = online_pages(start_pfn, nr_pages);
 			break;
 		case MEM_OFFLINE:
-			start_paddr = page_to_pfn(first_page) << PAGE_SHIFT;
-			ret = remove_memory(start_paddr,
-					    nr_pages << PAGE_SHIFT);
+			ret = offline_pages(start_pfn, nr_pages);
 			break;
 		default:
 			WARN(1, KERN_WARNING "%s(%ld, %ld) unknown action: "

include/linux/memory_hotplug.h +1 -0

Index: linux-3.6/include/linux/memory_hotplug.h
===================================================================
@@ -233,6 +233,7 @@ static inline int is_mem_section_removab
 extern int mem_online_node(int nid);
 extern int add_memory(int nid, u64 start, u64 size);
 extern int arch_add_memory(int nid, u64 start, u64 size);
+extern int offline_pages(unsigned long start_pfn, unsigned long nr_pages);
 extern int remove_memory(u64 start, u64 size);
 extern int sparse_add_one_section(struct zone *zone, unsigned long start_pfn,
 								int nr_pages);

mm/memory_hotplug.c +11 -2

Index: linux-3.6/mm/memory_hotplug.c
===================================================================
@@ -870,7 +870,7 @@ check_pages_isolated(unsigned long start
 	return offlined;
 }
 
-static int __ref offline_pages(unsigned long start_pfn,
+static int __ref __offline_pages(unsigned long start_pfn,
 		  unsigned long end_pfn, unsigned long timeout)
 {
 	unsigned long pfn, nr_pages, expire;

@@ -998,15 +998,24 @@ out:
 	return ret;
 }
 
+int offline_pages(unsigned long start_pfn, unsigned long nr_pages)
+{
+	return __offline_pages(start_pfn, start_pfn + nr_pages, 120 * HZ);
+}
+
 int remove_memory(u64 start, u64 size)
 {
 	unsigned long start_pfn, end_pfn;
 
 	start_pfn = PFN_DOWN(start);
 	end_pfn = start_pfn + PFN_DOWN(size);
-	return offline_pages(start_pfn, end_pfn, 120 * HZ);
+	return __offline_pages(start_pfn, end_pfn, 120 * HZ);
 }
 #else
+int offline_pages(unsigned long start_pfn, unsigned long nr_pages)
+{
+	return -EINVAL;
+}
 int remove_memory(u64 start, u64 size)
 {
 	return -EINVAL;

Yasuaki Ishimatsu

author

2012-10-02 08:28 UTC #2 lore raw

PATCH 2/2 memory-hotplug : update memory block's state and notfy theinformation to userspace

In reply to Yasuaki Ishimatsu

From: Wen Congyang <[email protected]>

The function remove_memory() will be called when hot removing a memory device. But even if offlining memory, we cannot notice it. So the patch update memory block's state and notify to userspace.

Additionally, the memory device may contain more than one memory block. If the memory block has been offlined, __offline_pages() will fail. So we should try to offline one memory block at a time.

Thus the function remove_memory() also check each memory block's state. So there is no need to check the memory block's state before calling remove_memory().

CC David Rientjes <[email protected]>

CC Jiang Liu <[email protected]>

CC Len Brown <[email protected]>

CC Christoph Lameter <[email protected]>

Cc Minchan Kim <[email protected]>

CC Andrew Morton <[email protected]>

CC KOSAKI Motohiro <[email protected]>

Signed-off-by Wen Congyang <[email protected]>

Signed-off-by Yasuaki Ishimatsu <[email protected]>

---
 drivers/base/memory.c          |   31 +++++++++++++++++++++++++++----
 include/linux/memory_hotplug.h |    2 ++
 mm/memory_hotplug.c            |   33 ++++++++++++++++++++++++++++++++-
 3 files changed, 61 insertions(+), 5 deletions(-)

drivers/base/memory.c +27 -4

Index: linux-3.6/drivers/base/memory.c
===================================================================
@@ -275,13 +275,11 @@ memory_block_action(unsigned long phys_i
 	return ret;
 }
 
-static int memory_block_change_state(struct memory_block *mem,
+static int __memory_block_change_state(struct memory_block *mem,
 		unsigned long to_state, unsigned long from_state_req)
 {
 	int ret = 0;
 
-	mutex_lock(&mem->state_mutex);
-
 	if (mem->state != from_state_req) {
 		ret = -EINVAL;
 		goto out;

@@ -309,10 +307,20 @@ static int memory_block_change_state(str
 		break;
 	}
 out:
-	mutex_unlock(&mem->state_mutex);
 	return ret;
 }
 
+static int memory_block_change_state(struct memory_block *mem,
+		unsigned long to_state, unsigned long from_state_req)
+{
+	int ret;
+
+	mutex_lock(&mem->state_mutex);
+	ret = __memory_block_change_state(mem, to_state, from_state_req);
+	mutex_unlock(&mem->state_mutex);
+
+	return ret;
+}
 static ssize_t
 store_mem_state(struct device *dev,
 		struct device_attribute *attr, const char *buf, size_t count)

@@ -653,6 +661,21 @@ int unregister_memory_section(struct mem
 }
 
 /*
+ * offline one memory block. If the memory block has been offlined, do nothing.
+ */
+int offline_memory_block(struct memory_block *mem)
+{
+	int ret = 0;
+
+	mutex_lock(&mem->state_mutex);
+	if (mem->state != MEM_OFFLINE)
+		ret = __memory_block_change_state(mem, MEM_OFFLINE, MEM_ONLINE);
+	mutex_unlock(&mem->state_mutex);
+
+	return ret;
+}
+
+/*
  * Initialize the sysfs support for memory devices...
  */
 int __init memory_dev_init(void)

include/linux/memory_hotplug.h +2 -0

Index: linux-3.6/include/linux/memory_hotplug.h
===================================================================
@@ -10,6 +10,7 @@ struct page;
 struct zone;
 struct pglist_data;
 struct mem_section;
+struct memory_block;
 
 #ifdef CONFIG_MEMORY_HOTPLUG

@@ -234,6 +235,7 @@ extern int mem_online_node(int nid);
 extern int add_memory(int nid, u64 start, u64 size);
 extern int arch_add_memory(int nid, u64 start, u64 size);
 extern int offline_pages(unsigned long start_pfn, unsigned long nr_pages);
+extern int offline_memory_block(struct memory_block *mem);
 extern int remove_memory(u64 start, u64 size);
 extern int sparse_add_one_section(struct zone *zone, unsigned long start_pfn,
 								int nr_pages);

mm/memory_hotplug.c +32 -1

Index: linux-3.6/mm/memory_hotplug.c
===================================================================
@@ -1005,11 +1005,42 @@ int offline_pages(unsigned long start_pf
 
 int remove_memory(u64 start, u64 size)
 {
+	struct memory_block *mem = NULL;
+	struct mem_section *section;
 	unsigned long start_pfn, end_pfn;
+	unsigned long pfn, section_nr;
+	int ret;
 
 	start_pfn = PFN_DOWN(start);
 	end_pfn = start_pfn + PFN_DOWN(size);
-	return __offline_pages(start_pfn, end_pfn, 120 * HZ);
+
+	for (pfn = start_pfn; pfn < end_pfn; pfn += PAGES_PER_SECTION) {
+		section_nr = pfn_to_section_nr(pfn);
+		if (!present_section_nr(section_nr))
+			continue;
+
+		section = __nr_to_section(section_nr);
+		/* same memblock? */
+		if (mem)
+			if ((section_nr >= mem->start_section_nr) &&
+			    (section_nr <= mem->end_section_nr))
+				continue;
+
+		mem = find_memory_block_hinted(section, mem);
+		if (!mem)
+			continue;
+
+		ret = offline_memory_block(mem);
+		if (ret) {
+			kobject_put(&mem->dev.kobj);
+			return ret;
+		}
+	}
+
+	if (mem)
+		kobject_put(&mem->dev.kobj);
+
+	return 0;
 }
 #else
 int offline_pages(unsigned long start_pfn, unsigned long nr_pages)

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Ni zhan Chen

2012-10-02 09:42 UTC #3 lore raw

In reply to Yasuaki Ishimatsu

Quoted 30 lines · Yasuaki Ishimatsu

We are trying to implement a physical memory hot removing function as following thread.
https://lkml.org/lkml/2012/9/5/201
But there is not enough review to merge into linux kernel.
I think there are following blockades.
1. no physical memory hot removable system
2. huge patch-set
If you have a KVM system, we can get rid of 1st blockade. Because applying following patch, we can create memory hot removable system on KVM guest.
http://lists.gnu.org/archive/html/qemu-devel/2012-07/msg01389.html
2nd blockade is own problem. So we try to divide huge patch into a small patch in each function as follows:
- bug fix
- acpi framework
- kernel core
We had already sent bug fix patches.
https://lkml.org/lkml/2012/9/27/39
And the patch fixes following bug.

Quoted 8 lines

remove_memory() offlines memory. And it is called by following two cases:
1. echo offline >/sys/devices/system/memory/memoryXX/state
2. hot remove a memory device
In the 1st case, the memory block's state is changed and the notification that memory block's state changed is sent to userland after calling offline_memory(). So user can notice memory block is changed.,

Hi Yasuaki,

Thanks for splitting the patchset, it's more easier to review this time. One question:

How can notify userspace? you mean function node_memory_callback or ...., but
this function basically do nothing.

Quoted 5 lines

But in the 2nd case, the memory block's state is not changed and the notification is not also sent to userspcae even if calling offline_memory(). So user cannot notice memory block is changed.

Quoted 8 lines

We should also notify to userspace at 2nd case.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Yasuaki Ishimatsu

author

2012-10-02 10:00 UTC #4 lore raw

Quoted 40 lines · Ni zhan Chen

Quoted 30 lines · Yasuaki Ishimatsu
We are trying to implement a physical memory hot removing function as following thread.
https://lkml.org/lkml/2012/9/5/201
But there is not enough review to merge into linux kernel.
I think there are following blockades.
1. no physical memory hot removable system
2. huge patch-set
If you have a KVM system, we can get rid of 1st blockade. Because applying following patch, we can create memory hot removable system on KVM guest.
http://lists.gnu.org/archive/html/qemu-devel/2012-07/msg01389.html
2nd blockade is own problem. So we try to divide huge patch into a small patch in each function as follows:
- bug fix
- acpi framework
- kernel core
We had already sent bug fix patches.
https://lkml.org/lkml/2012/9/27/39
And the patch fixes following bug.
Quoted 8 lines
remove_memory() offlines memory. And it is called by following two cases:
1. echo offline >/sys/devices/system/memory/memoryXX/state
2. hot remove a memory device
In the 1st case, the memory block's state is changed and the notification that memory block's state changed is sent to userland after calling offline_memory(). So user can notice memory block is changed.,

Quoted 7 lines

Hi Yasuaki,
Thanks for splitting the patchset, it's more easier to review this time. One question:
How can notify userspace? you mean function node_memory_callback or ...., but

When calling memory_block_change_state(), it calls kobject_uevent(). This function notifies userspace of the online/offline notification.

Thanks,
Yasuaki Ishimatsu

Quoted 21 lines

this function basically do nothing.
Quoted 5 lines
But in the 2nd case, the memory block's state is not changed and the notification is not also sent to userspcae even if calling offline_memory(). So user cannot notice memory block is changed.
Quoted 8 lines
We should also notify to userspace at 2nd case.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
--
To unsubscribe from this list: send the line "unsubscribe linux-acpi" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Andrew Morton

maintainer

2012-10-02 21:42 UTC #5 lore raw

In reply to Yasuaki Ishimatsu

Quoted 9 lines · On Tue, 2 Oct 2012 17:25:06 +0900 Yasuaki Ishimatsu…

remove_memory() offlines memory. And it is called by following two cases:
1. echo offline >/sys/devices/system/memory/memoryXX/state
2. hot remove a memory device
In the 1st case, the memory block's state is changed and the notification that memory block's state changed is sent to userland after calling offline_memory(). So user can notice memory block is changed.

Quoted 5 lines

But in the 2nd case, the memory block's state is not changed and the notification is not also sent to userspcae even if calling offline_memory(). So user cannot notice memory block is changed.
We should also notify to userspace at 2nd case.

These two little patches look reasonable to me.

There's a lot of recent activity with memory hotplug! We're in the 3.7 merge window now so it is not a good time to be merging new material. Also there appear to be two teams working on it and it's unclear to me how well coordinated this work is?

However these two patches are pretty simple and do fix a problem, so I added them to the 3.7 MM queue.

Yasuaki Ishimatsu

author

2012-10-03 01:21 UTC #6 lore raw

Hi Andrew,

Quoted 18 lines · Andrew Morton

Quoted 9 lines · On Tue, 2 Oct 2012 17:25:06 +0900 Yasuaki Ishimatsu…
remove_memory() offlines memory. And it is called by following two cases:
1. echo offline >/sys/devices/system/memory/memoryXX/state
2. hot remove a memory device
In the 1st case, the memory block's state is changed and the notification that memory block's state changed is sent to userland after calling offline_memory(). So user can notice memory block is changed.
Quoted 5 lines
But in the 2nd case, the memory block's state is not changed and the notification is not also sent to userspcae even if calling offline_memory(). So user cannot notice memory block is changed.
We should also notify to userspace at 2nd case.

Quoted 7 lines

These two little patches look reasonable to me.
There's a lot of recent activity with memory hotplug! We're in the 3.7 merge window now so it is not a good time to be merging new material.
Also there appear to be two teams working on it and it's unclear to me how well coordinated this work is?

As you know, there are two teams for developing the memory hotplug.
- Wen's patch-set

    https://lkml.org/lkml/2012/9/5/201

- Lai's patch-set

    https://lkml.org/lkml/2012/9/10/180

Wen's patch-set is for removing physical memory. Now, I'm splitting the patch-set for reviewing more easy. If the patch-set is merged into linux kernel, I believe that linux on x86 can hot remove a physical memory device.

But it is not enough since we cannot remove a memory which has kernel memory. If we guarantee the memory hot remove, the memory must belong to ZONE_MOVABLE.

So Lai's patch-set tries to create a movable node that the all memory belongs to ZONE_MOVABLE.

I think there are two chances for creating the movable node.
- boot time
- after hot add memory

- boot time

For creating a movable memory, linux has two kernel parameters (kernelcore and movablecore). But it is not enough, since even if we set the kernel paramter, the movable memory is distributed evenly in each node. So we introduce the kernelcore_max_addr boot parameter. The parameter limits the range of the memory used as a kernel memory.

For example, the system has following nodes.

    node0 : 0x40000000 - 0x80000000
    node1 : 0x80000000 - 0xc0000000

And when I want to hot remove a node1, we set "kernelcore_max_addr=0x80000000". In doing so, kernel memory is limited within 0x80000000 and node1's memory belongs to ZONE_MOEVALBE. As a result, we can guarantee that node1 is a movable node and we always hot remove node1.

- after hot add memory

When hot adding memory, the memory belongs to ZONE_NORMAL and is offline. If we online the memory, the memory may have kernel memory. In this case, we cannot hot remove the memory. So we introduce the online_movable function. If we use the function as follow, the memory belongs to ZONE_MOVABLE.

echo online_movable > /sys/devices/system/node/nodeX/memoryX/state

So when new node is hot added and I echo "online_movale" to all hot added memory, the node's memory belongs to ZONE_MOVABLE. As a result, we can guarantee that the node is a movable node and we always hot remove node.

# I hope to help your understanding about our works by the information.

Thanks,
Yasuaki Ishimatsu

Quoted 4 lines

However these two patches are pretty simple and do fix a problem, so I added them to the 3.7 MM queue.

Ni zhan Chen

2012-10-03 02:12 UTC #7 lore raw

Quoted 78 lines · Yasuaki Ishimatsu

Hi Andrew,
Quoted 18 lines · Andrew Morton
Quoted 9 lines · On Tue, 2 Oct 2012 17:25:06 +0900 Yasuaki Ishimatsu…
remove_memory() offlines memory. And it is called by following two cases:
1. echo offline >/sys/devices/system/memory/memoryXX/state
2. hot remove a memory device
In the 1st case, the memory block's state is changed and the notification that memory block's state changed is sent to userland after calling offline_memory(). So user can notice memory block is changed.
Quoted 5 lines
But in the 2nd case, the memory block's state is not changed and the notification is not also sent to userspcae even if calling offline_memory(). So user cannot notice memory block is changed.
We should also notify to userspace at 2nd case.
Quoted 7 lines
These two little patches look reasonable to me.
There's a lot of recent activity with memory hotplug! We're in the 3.7 merge window now so it is not a good time to be merging new material.
Also there appear to be two teams working on it and it's unclear to me how well coordinated this work is?
As you know, there are two teams for developing the memory hotplug.
- Wen's patch-set
    https://lkml.org/lkml/2012/9/5/201
- Lai's patch-set
    https://lkml.org/lkml/2012/9/10/180
Wen's patch-set is for removing physical memory. Now, I'm splitting the patch-set for reviewing more easy. If the patch-set is merged into linux kernel, I believe that linux on x86 can hot remove a physical memory device.
But it is not enough since we cannot remove a memory which has kernel memory. If we guarantee the memory hot remove, the memory must belong to ZONE_MOVABLE.
So Lai's patch-set tries to create a movable node that the all memory belongs to ZONE_MOVABLE.
I think there are two chances for creating the movable node.
- boot time
- after hot add memory
- boot time
For creating a movable memory, linux has two kernel parameters (kernelcore and movablecore). But it is not enough, since even if we set the kernel paramter, the movable memory is distributed evenly in each node. So we introduce the kernelcore_max_addr boot parameter. The parameter limits the range of the memory used as a kernel memory.
For example, the system has following nodes.
    node0 : 0x40000000 - 0x80000000
    node1 : 0x80000000 - 0xc0000000
And when I want to hot remove a node1, we set "kernelcore_max_addr=0x80000000". In doing so, kernel memory is limited within 0x80000000 and node1's memory belongs to ZONE_MOEVALBE. As a result, we can guarantee that node1 is a movable node and we always hot remove node1.
- after hot add memory
When hot adding memory, the memory belongs to ZONE_NORMAL and is offline. If we online the memory, the memory may have kernel memory. In this case, we cannot hot remove the memory. So we introduce the online_movable function. If we use the function as follow, the memory belongs to ZONE_MOVABLE.

Quoted 5 lines

echo online_movable > /sys/devices/system/node/nodeX/memoryX/state
So when new node is hot added and I echo "online_movale" to all hot added memory, the node's memory belongs to ZONE_MOVABLE. As a result, we can Y guarantee that the node is a movable node and we always hot remove node.

Hi Yasuaki,

This time can kernel memory allocated from ZONE_MOVABLE ?

Quoted 18 lines

# I hope to help your understanding about our works by the information.
Thanks,
Yasuaki Ishimatsu
Quoted 4 lines
However these two patches are pretty simple and do fix a problem, so I added them to the 3.7 MM queue.
Signature · 4 lines
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Yasuaki Ishimatsu

author

2012-10-03 03:58 UTC #8 lore raw

Hi Chen,

Quoted 81 lines · Ni zhan Chen

Quoted 80 lines · Yasuaki Ishimatsu
Hi Andrew,
Quoted 18 lines · Andrew Morton
Quoted 9 lines · On Tue, 2 Oct 2012 17:25:06 +0900 Yasuaki Ishimatsu…
remove_memory() offlines memory. And it is called by following two cases:
1. echo offline >/sys/devices/system/memory/memoryXX/state
2. hot remove a memory device
In the 1st case, the memory block's state is changed and the notification that memory block's state changed is sent to userland after calling offline_memory(). So user can notice memory block is changed.
Quoted 5 lines
But in the 2nd case, the memory block's state is not changed and the notification is not also sent to userspcae even if calling offline_memory(). So user cannot notice memory block is changed.
We should also notify to userspace at 2nd case.
Quoted 7 lines
These two little patches look reasonable to me.
There's a lot of recent activity with memory hotplug! We're in the 3.7 merge window now so it is not a good time to be merging new material.
Also there appear to be two teams working on it and it's unclear to me how well coordinated this work is?
As you know, there are two teams for developing the memory hotplug.
- Wen's patch-set
    https://lkml.org/lkml/2012/9/5/201
- Lai's patch-set
    https://lkml.org/lkml/2012/9/10/180
Wen's patch-set is for removing physical memory. Now, I'm splitting the patch-set for reviewing more easy. If the patch-set is merged into linux kernel, I believe that linux on x86 can hot remove a physical memory device.
But it is not enough since we cannot remove a memory which has kernel memory. If we guarantee the memory hot remove, the memory must belong to ZONE_MOVABLE.
So Lai's patch-set tries to create a movable node that the all memory belongs to ZONE_MOVABLE.
I think there are two chances for creating the movable node.
- boot time
- after hot add memory
- boot time
For creating a movable memory, linux has two kernel parameters (kernelcore and movablecore). But it is not enough, since even if we set the kernel paramter, the movable memory is distributed evenly in each node. So we introduce the kernelcore_max_addr boot parameter. The parameter limits the range of the memory used as a kernel memory.
For example, the system has following nodes.
    node0 : 0x40000000 - 0x80000000
    node1 : 0x80000000 - 0xc0000000
And when I want to hot remove a node1, we set "kernelcore_max_addr=0x80000000". In doing so, kernel memory is limited within 0x80000000 and node1's memory belongs to ZONE_MOEVALBE. As a result, we can guarantee that node1 is a movable node and we always hot remove node1.
- after hot add memory
When hot adding memory, the memory belongs to ZONE_NORMAL and is offline. If we online the memory, the memory may have kernel memory. In this case, we cannot hot remove the memory. So we introduce the online_movable function. If we use the function as follow, the memory belongs to ZONE_MOVABLE.
echo online_movable > /sys/devices/system/node/nodeX/memoryX/state

Quoted 7 lines

Quoted 3 lines
So when new node is hot added and I echo "online_movale" to all hot added memory, the node's memory belongs to ZONE_MOVABLE. As a result, we can Y guarantee that the node is a movable node and we always hot remove node.
Hi Yasuaki,
This time can kernel memory allocated from ZONE_MOVABLE ?

No. In this case, the memory cannot be used as kernel memory.

Thanks,
Yasuaki Ishimatsu

Quoted 13 lines

Quoted 13 lines
# I hope to help your understanding about our works by the information.
Thanks,
Yasuaki Ishimatsu
Quoted 4 lines
However these two patches are pretty simple and do fix a problem, so I added them to the 3.7 MM queue.

Quoted 7 lines

Quoted 6 lines
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>