Skip to contentSkip to conversation

mm/memory_hotplug: Adds option to hot-add memory in ZONE_MOVABLE

PATCH In review Leonardo Bras 5 participants 11 messages Jul 18, 2019

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 mm/memory_hotplug: Adds option to hot-add memory in ZONE_MOVABLE

Adds an option on kernel config to make hot-added memory online in ZONE_MOVABLE by default.

This would be great in systems with MEMORY_HOTPLUG_DEFAULT_ONLINE=y by allowing to choose which zone it will be auto-onlined

---
 drivers/base/memory.c |  3 +++
 mm/Kconfig            | 14 ++++++++++++++
 2 files changed, 17 insertions(+)
drivers/base/memory.c +3 -0
index f180427e48f4..378b585785c1 100644
@@ -670,6 +670,9 @@ static int init_memory_block(struct memory_block **memory,
 	mem->state = state;
 	start_pfn = section_nr_to_pfn(mem->start_section_nr);
 	mem->phys_device = arch_get_memory_phys_device(start_pfn);
+#ifdef CONFIG_MEMORY_HOTPLUG_MOVABLE
+	mem->online_type = MMOP_ONLINE_MOVABLE;
+#endif
 
 	ret = register_memory(mem);
 
mm/Kconfig +14 -0
index f0c76ba47695..74e793720f43 100644
@@ -180,6 +180,20 @@ config MEMORY_HOTREMOVE
 	depends on MEMORY_HOTPLUG && ARCH_ENABLE_MEMORY_HOTREMOVE
 	depends on MIGRATION
 
+config MEMORY_HOTPLUG_MOVABLE
+	bool "Enhance the likelihood of hot-remove"
+	depends on MEMORY_HOTREMOVE
+	help
+	  This option sets the hot-added memory zone to MOVABLE which
+	  drastically reduces the chance of a hot-remove to fail due to
+	  unmovable memory segments. Kernel memory can't be allocated in
+	  this zone.
+
+	  Say Y here if you want to have better chance to hot-remove memory
+	  that have been previously hot-added.
+	  Say N here if you want to make all hot-added memory available to
+	  kernel space.
+
 # Heavily threaded applications may benefit from splitting the mm-wide
 # page_table_lock, so that faults on different parts of the user address
 # space can be handled with less contention: split it at this NR_CPUS.
Signature · 1 line
2.20.1
maintainer
#1 lore raw
Quoted 6 lines · Leonardo Bras

Adds an option on kernel config to make hot-added memory online in ZONE_MOVABLE by default.

This would be great in systems with MEMORY_HOTPLUG_DEFAULT_ONLINE=y by
allowing to choose which zone it will be auto-onlined

We do already have "movable_node" boot option, which exactly has that effect.
Any hotplugged range will be placed in ZONE_MOVABLE.

Why do we need yet another option to achieve the same? Was not that enough for your case?

Quoted 55 lines
---
 drivers/base/memory.c |  3 +++
 mm/Kconfig            | 14 ++++++++++++++
 2 files changed, 17 insertions(+)
drivers/base/memory.c +3 -0
index f180427e48f4..378b585785c1 100644
@@ -670,6 +670,9 @@ static int init_memory_block(struct memory_block
 **memory,
 	mem->state = state;
 	start_pfn = section_nr_to_pfn(mem->start_section_nr);
 	mem->phys_device = arch_get_memory_phys_device(start_pfn);
+#ifdef CONFIG_MEMORY_HOTPLUG_MOVABLE
+	mem->online_type = MMOP_ONLINE_MOVABLE;
+#endif
 
 	ret = register_memory(mem);
mm/Kconfig +5 -0
index f0c76ba47695..74e793720f43 100644
@@ -180,6 +180,20 @@ config MEMORY_HOTREMOVE
 	depends on MEMORY_HOTPLUG && ARCH_ENABLE_MEMORY_HOTREMOVE
 	depends on MIGRATION
 
+config MEMORY_HOTPLUG_MOVABLE
+	bool "Enhance the likelihood of hot-remove"
+	depends on MEMORY_HOTREMOVE
+	help
+	  This option sets the hot-added memory zone to MOVABLE

which

mm/Kconfig +1 -0
+	  drastically reduces the chance of a hot-remove to fail due

to

mm/Kconfig +1 -0
+	  unmovable memory segments. Kernel memory can't be

allocated in

mm/Kconfig +3 -0
+	  this zone.
+
+	  Say Y here if you want to have better chance to hot-remove 

memory

mm/Kconfig +2 -0
+	  that have been previously hot-added.
+	  Say N here if you want to make all hot-added memory

available to

mm/Kconfig +2 -0
+	  kernel space.
+
 # Heavily threaded applications may benefit from splitting the mm-

wide
# page_table_lock, so that faults on different parts of the user
address

 # space can be handled with less contention: split it at this
NR_CPUS.
Signature · 2 lines
Oscar Salvador
SUSE L3
#2 lore raw
Quoted 3 lines · Oscar Salvador

We do already have "movable_node" boot option, which exactly has that effect.
Any hotplugged range will be placed in ZONE_MOVABLE.

Oh, I was not aware of it.

Quoted 2 lines

Why do we need yet another option to achieve the same? Was not that enough for your case?

Well, another use of this config could be doing this boot option a default on any given kernel.
But in the above case I agree it would be wiser to add the code on movable_node_is_enabled() directly, and not where I did put.

What do you think about it?

Thanks for the feedback,

Leonardo Brás

#3 lore raw
Quoted 6 lines · Leonardo Bras
Quoted 3 lines · Oscar Salvador

We do already have "movable_node" boot option, which exactly has that effect.
Any hotplugged range will be placed in ZONE_MOVABLE.

Oh, I was not aware of it.

Quoted 8 lines
Quoted 2 lines

Why do we need yet another option to achieve the same? Was not that enough for your case?

Well, another use of this config could be doing this boot option a default on any given kernel.
But in the above case I agree it would be wiser to add the code on movable_node_is_enabled() directly, and not where I did put.

What do you think about it?

No further config options please. We do have means a more flexible way to achieve movable node onlining so let's use it. Or could you be more specific about cases which cannot use the command line option and really need a config option to workaround that?

Signature · 2 lines
Michal Hocko
SUSE Labs
#4 lore raw
Quoted 15 lines · Michal Hocko
Quoted 13 lines · Leonardo Bras
Quoted 3 lines · Oscar Salvador

We do already have "movable_node" boot option, which exactly has that effect.
Any hotplugged range will be placed in ZONE_MOVABLE.

Oh, I was not aware of it.

Quoted 2 lines

Why do we need yet another option to achieve the same? Was not that enough for your case?

Well, another use of this config could be doing this boot option a default on any given kernel.
But in the above case I agree it would be wiser to add the code on movable_node_is_enabled() directly, and not where I did put.

Quoted 6 lines
Quoted 1 line

What do you think about it?

No further config options please. We do have means a more flexible way to achieve movable node onlining so let's use it. Or could you be more specific about cases which cannot use the command line option and really need a config option to workaround that?

Hi Michal,

Just trying to understand, if kernel parameters is the preferable method, why do we even have

MEMORY_HOTPLUG_DEFAULT_ONLINE

It is just strange that we have a config to online memory by default without kernel parameter, but no way to specify how to online it. It just looks as incomplete interface to me. Perhaps this config should be removed as well?

Pasha

#5 lore raw
Quoted 23 lines · Pavel Tatashin
Quoted 15 lines · Michal Hocko
Quoted 13 lines · Leonardo Bras
Quoted 3 lines · Oscar Salvador

We do already have "movable_node" boot option, which exactly has that effect.
Any hotplugged range will be placed in ZONE_MOVABLE.

Oh, I was not aware of it.

Quoted 2 lines

Why do we need yet another option to achieve the same? Was not that enough for your case?

Well, another use of this config could be doing this boot option a default on any given kernel.
But in the above case I agree it would be wiser to add the code on movable_node_is_enabled() directly, and not where I did put.

Quoted 6 lines
Quoted 1 line

What do you think about it?

No further config options please. We do have means a more flexible way to achieve movable node onlining so let's use it. Or could you be more specific about cases which cannot use the command line option and really need a config option to workaround that?

Quoted 6 lines

Hi Michal,

Just trying to understand, if kernel parameters is the preferable method, why do we even have

MEMORY_HOTPLUG_DEFAULT_ONLINE

I have some opinion on this one TBH. I have even tried to remove it. The config option has been added to workaround hotplug issues for some memory balloning usecases where it was believed that the memory consumed for the memory hotadd (struct pages) could get machine to OOM before userspace manages to online it. So I would be more than happy to remove it but there were some objections in the past. Maybe the work by Oscar to allocate memmaps from the hotplugged memory can finally put an end to this gross hack.

In any case, I do not think we want to repeat that pattern again.

Signature · 2 lines
Michal Hocko
SUSE Labs
#6 lore raw
Quoted 5 lines
Quoted 4 lines

Just trying to understand, if kernel parameters is the preferable method, why do we even have

MEMORY_HOTPLUG_DEFAULT_ONLINE

Quoted 8 lines

I have some opinion on this one TBH. I have even tried to remove it. The config option has been added to workaround hotplug issues for some memory balloning usecases where it was believed that the memory consumed for the memory hotadd (struct pages) could get machine to OOM before userspace manages to online it. So I would be more than happy to remove it but there were some objections in the past. Maybe the work by Oscar to allocate memmaps from the hotplugged memory can finally put an end to this gross hack.

Makes sense, thank you for the background info.

Pasha

#7 lore raw
In reply to Leonardo Bras
Quoted 5 lines · Leonardo Bras

Adds an option on kernel config to make hot-added memory online in ZONE_MOVABLE by default.

This would be great in systems with MEMORY_HOTPLUG_DEFAULT_ONLINE=y by allowing to choose which zone it will be auto-onlined

Please add more elaborate description of the problem you are solving and the solution outline.

Quoted 11 lines
---
 drivers/base/memory.c |  3 +++
 mm/Kconfig            | 14 ++++++++++++++
 2 files changed, 17 insertions(+)
drivers/base/memory.c +0 -0
index f180427e48f4..378b585785c1 100644

drivers/base/memory.c static int init_memory_block(struct memory_block **memory, · line 675

Quoted 6 lines
drivers/base/memory.c +3 -0
@@ -670,6 +670,9 @@ static int init_memory_block(struct memory_block **memory,
 	mem->state = state;
 	start_pfn = section_nr_to_pfn(mem->start_section_nr);
 	mem->phys_device = arch_get_memory_phys_device(start_pfn);
+#ifdef CONFIG_MEMORY_HOTPLUG_MOVABLE
+	mem->online_type = MMOP_ONLINE_MOVABLE;
+#endif

Does it has to be a compile time option?
Seems like this can be changed at run time or at least at boot.

Quoted 30 lines
drivers/base/memory.c +0 -0
 	ret = register_memory(mem);
 
mm/Kconfig +14 -0
index f0c76ba47695..74e793720f43 100644
@@ -180,6 +180,20 @@ config MEMORY_HOTREMOVE
 	depends on MEMORY_HOTPLUG && ARCH_ENABLE_MEMORY_HOTREMOVE
 	depends on MIGRATION
 
+config MEMORY_HOTPLUG_MOVABLE
+	bool "Enhance the likelihood of hot-remove"
+	depends on MEMORY_HOTREMOVE
+	help
+	  This option sets the hot-added memory zone to MOVABLE which
+	  drastically reduces the chance of a hot-remove to fail due to
+	  unmovable memory segments. Kernel memory can't be allocated in
+	  this zone.
+
+	  Say Y here if you want to have better chance to hot-remove memory
+	  that have been previously hot-added.
+	  Say N here if you want to make all hot-added memory available to
+	  kernel space.
+
 # Heavily threaded applications may benefit from splitting the mm-wide
 # page_table_lock, so that faults on different parts of the user address
 # space can be handled with less contention: split it at this NR_CPUS.
Signature · 1 line
2.20.1
Signature · 2 lines
Sincerely yours,
Mike.
#8 lore raw
In reply to Leonardo Bras
Quoted 6 lines · Leonardo Bras

Adds an option on kernel config to make hot-added memory online in ZONE_MOVABLE by default.

This would be great in systems with MEMORY_HOTPLUG_DEFAULT_ONLINE=y by allowing to choose which zone it will be auto-onlined

This is a desired feature. From reading the code it looks to me that auto-selection of online method type should be done in memory_subsys_online().

When it is called from device online, mem->online_type should be -1:

if (mem->online_type < 0)

     mem->online_type = MMOP_ONLINE_KEEP;

Change it to:
if (mem->online_type < 0)

     mem->online_type = MMOP_DEFAULT_ONLINE_TYPE;

And in "linux/memory_hotplug.h"
#ifdef CONFIG_MEMORY_HOTPLUG_MOVABLE
#define MMOP_DEFAULT_ONLINE_TYPE MMOP_ONLINE_MOVABLE #else
#define MMOP_DEFAULT_ONLINE_TYPE MMOP_ONLINE_KEEP #endif

Could be expanded to support MMOP_ONLINE_KERNEL as well.

Pasha

Quoted 49 lines
---
 drivers/base/memory.c |  3 +++
 mm/Kconfig            | 14 ++++++++++++++
 2 files changed, 17 insertions(+)
drivers/base/memory.c +3 -0
index f180427e48f4..378b585785c1 100644
@@ -670,6 +670,9 @@ static int init_memory_block(struct memory_block **memory,
        mem->state = state;
        start_pfn = section_nr_to_pfn(mem->start_section_nr);
        mem->phys_device = arch_get_memory_phys_device(start_pfn);
+#ifdef CONFIG_MEMORY_HOTPLUG_MOVABLE
+       mem->online_type = MMOP_ONLINE_MOVABLE;
+#endif
 
        ret = register_memory(mem);
 
mm/Kconfig +14 -1
index f0c76ba47695..74e793720f43 100644
@@ -180,6 +180,20 @@ config MEMORY_HOTREMOVE
        depends on MEMORY_HOTPLUG && ARCH_ENABLE_MEMORY_HOTREMOVE
        depends on MIGRATION
 
+config MEMORY_HOTPLUG_MOVABLE
+       bool "Enhance the likelihood of hot-remove"
+       depends on MEMORY_HOTREMOVE
+       help
+         This option sets the hot-added memory zone to MOVABLE which
+         drastically reduces the chance of a hot-remove to fail due to
+         unmovable memory segments. Kernel memory can't be allocated in
+         this zone.
+
+         Say Y here if you want to have better chance to hot-remove memory
+         that have been previously hot-added.
+         Say N here if you want to make all hot-added memory available to
+         kernel space.
+
 # Heavily threaded applications may benefit from splitting the mm-wide
 # page_table_lock, so that faults on different parts of the user address
 # space can be handled with less contention: split it at this NR_CPUS.
--

2.20.1

Quoted 55 lines · Leonardo Bras

Adds an option on kernel config to make hot-added memory online in ZONE_MOVABLE by default.

This would be great in systems with MEMORY_HOTPLUG_DEFAULT_ONLINE=y by allowing to choose which zone it will be auto-onlined

---
 drivers/base/memory.c |  3 +++
 mm/Kconfig            | 14 ++++++++++++++
 2 files changed, 17 insertions(+)
drivers/base/memory.c +3 -0
index f180427e48f4..378b585785c1 100644
@@ -670,6 +670,9 @@ static int init_memory_block(struct memory_block **memory,
        mem->state = state;
        start_pfn = section_nr_to_pfn(mem->start_section_nr);
        mem->phys_device = arch_get_memory_phys_device(start_pfn);
+#ifdef CONFIG_MEMORY_HOTPLUG_MOVABLE
+       mem->online_type = MMOP_ONLINE_MOVABLE;
+#endif
 
        ret = register_memory(mem);
 
mm/Kconfig +14 -1
index f0c76ba47695..74e793720f43 100644
@@ -180,6 +180,20 @@ config MEMORY_HOTREMOVE
        depends on MEMORY_HOTPLUG && ARCH_ENABLE_MEMORY_HOTREMOVE
        depends on MIGRATION
 
+config MEMORY_HOTPLUG_MOVABLE
+       bool "Enhance the likelihood of hot-remove"
+       depends on MEMORY_HOTREMOVE
+       help
+         This option sets the hot-added memory zone to MOVABLE which
+         drastically reduces the chance of a hot-remove to fail due to
+         unmovable memory segments. Kernel memory can't be allocated in
+         this zone.
+
+         Say Y here if you want to have better chance to hot-remove memory
+         that have been previously hot-added.
+         Say N here if you want to make all hot-added memory available to
+         kernel space.
+
 # Heavily threaded applications may benefit from splitting the mm-wide
 # page_table_lock, so that faults on different parts of the user address
 # space can be handled with less contention: split it at this NR_CPUS.
--

2.20.1

#9 lore raw
Quoted 27 lines · Pavel Tatashin
Quoted 5 lines · Leonardo Bras

Adds an option on kernel config to make hot-added memory online in ZONE_MOVABLE by default.

This would be great in systems with MEMORY_HOTPLUG_DEFAULT_ONLINE=y by allowing to choose which zone it will be auto-onlined

This is a desired feature. From reading the code it looks to me that auto-selection of online method type should be done in memory_subsys_online().

When it is called from device online, mem->online_type should be -1:

if (mem->online_type < 0)

     mem->online_type = MMOP_ONLINE_KEEP;

Change it to:
if (mem->online_type < 0)

     mem->online_type = MMOP_DEFAULT_ONLINE_TYPE;

And in "linux/memory_hotplug.h"
#ifdef CONFIG_MEMORY_HOTPLUG_MOVABLE
#define MMOP_DEFAULT_ONLINE_TYPE MMOP_ONLINE_MOVABLE #else
#define MMOP_DEFAULT_ONLINE_TYPE MMOP_ONLINE_KEEP #endif

Quoted 3 lines

Could be expanded to support MMOP_ONLINE_KERNEL as well.

Pasha

Thanks for the suggestions Pasha,

I was made aware there is a kernel boot option "movable_node" that already creates the behavior I was trying to reproduce.

I was thinking of changing my patch in order to add a config option that makes this behavior default (i.e. not need to pass it as a boot parameter.

Do you think that it would still be a desired feature?

Regards,

Leonardo Brás

#10 lore raw
Quoted 29 lines · Leonardo Bras
Quoted 27 lines · Pavel Tatashin
Quoted 5 lines · Leonardo Bras

Adds an option on kernel config to make hot-added memory online in ZONE_MOVABLE by default.

This would be great in systems with MEMORY_HOTPLUG_DEFAULT_ONLINE=y by allowing to choose which zone it will be auto-onlined

This is a desired feature. From reading the code it looks to me that auto-selection of online method type should be done in memory_subsys_online().

When it is called from device online, mem->online_type should be -1:

if (mem->online_type < 0)

     mem->online_type = MMOP_ONLINE_KEEP;

Change it to:
if (mem->online_type < 0)

     mem->online_type = MMOP_DEFAULT_ONLINE_TYPE;

And in "linux/memory_hotplug.h"
#ifdef CONFIG_MEMORY_HOTPLUG_MOVABLE
#define MMOP_DEFAULT_ONLINE_TYPE MMOP_ONLINE_MOVABLE #else
#define MMOP_DEFAULT_ONLINE_TYPE MMOP_ONLINE_KEEP #endif

Quoted 8 lines
Quoted 3 lines

Could be expanded to support MMOP_ONLINE_KERNEL as well.

Pasha

Thanks for the suggestions Pasha,

I was made aware there is a kernel boot option "movable_node" that already creates the behavior I was trying to reproduce.

I agree with others, no need to duplicate this functionality in a config, and Michal in a separate e-mail explained the reasons why we have MEMORY_HOTPLUG_DEFAULT_ONLINE.

Quoted 10 lines

I was thinking of changing my patch in order to add a config option that makes this behavior default (i.e. not need to pass it as a boot parameter.

Do you think that it would still be a desired feature?

Regards,

Leonardo Brás