mm/memory_hotplug: Adds option to hot-add memory in ZONE_MOVABLE
Subsystems (4)
PATCH 1/1 mm/memory_hotplug: Adds option to hot-add memory in ZONE_MOVABLE
Adds an option on kernel config to make hot-added memory online in ZONE_MOVABLE by default.
This would be great in systems with MEMORY_HOTPLUG_DEFAULT_ONLINE=y by allowing to choose which zone it will be auto-onlined
--- drivers/base/memory.c | 3 +++ mm/Kconfig | 14 ++++++++++++++ 2 files changed, 17 insertions(+)
drivers/base/memory.c +3 -0
@@ -670,6 +670,9 @@ static int init_memory_block(struct memory_block **memory, mem->state = state; start_pfn =section_nr_to_pfn (mem->start_section_nr); mem->phys_device =arch_get_memory_phys_device (start_pfn); +#ifdef CONFIG_MEMORY_HOTPLUG_MOVABLE + mem->online_type =MMOP_ONLINE_MOVABLE ; +#endif ret =register_memory (mem);
mm/Kconfig +14 -0
@@ -180,6 +180,20 @@ config MEMORY_HOTREMOVE depends on MEMORY_HOTPLUG && ARCH_ENABLE_MEMORY_HOTREMOVE depends on MIGRATION +config MEMORY_HOTPLUG_MOVABLE + bool "Enhance the likelihood of hot-remove" + depends on MEMORY_HOTREMOVE + help + This option sets the hot-added memory zone to MOVABLE which + drastically reduces the chance of a hot-remove to fail due to + unmovable memory segments. Kernel memory can't be allocated in + this zone. + + Say Y here if you want to have better chance to hot-remove memory + that have been previously hot-added. + Say N here if you want to make all hot-added memory available to + kernel space. + # Heavily threaded applications may benefit from splitting the mm-wide # page_table_lock, so that faults on different parts of the user address # space can be handled with less contention: split it at this NR_CPUS.
Signature · 1 line
2.20.1
Oscar Salvador
maintainerQuoted 6 lines · Leonardo Bras
Adds an option on kernel config to make hot-added memory online in
ZONE_MOVABLEby default.This would be great in systems with MEMORY_HOTPLUG_DEFAULT_ONLINE=y by
allowing to choose which zone it will be auto-onlined
We do already have "movable_node" boot option, which exactly has that effect.
Any hotplugged range will be placed in ZONE_MOVABLE.
Why do we need yet another option to achieve the same? Was not that enough for your case?
Quoted 55 lines
Signed-off-by Leonardo Bras <[email protected]>--- drivers/base/memory.c | 3 +++ mm/Kconfig | 14 ++++++++++++++ 2 files changed, 17 insertions(+)drivers/base/memory.c +3
-0@@ -670,6 +670,9 @@ static int init_memory_block(struct memory_block **memory, mem->state = state; start_pfn =section_nr_to_pfn (mem->start_section_nr); mem->phys_device =arch_get_memory_phys_device (start_pfn); +#ifdef CONFIG_MEMORY_HOTPLUG_MOVABLE + mem->online_type =MMOP_ONLINE_MOVABLE ; +#endif ret =register_memory (mem);mm/Kconfig +5
-0@@ -180,6 +180,20 @@ config MEMORY_HOTREMOVE depends on MEMORY_HOTPLUG && ARCH_ENABLE_MEMORY_HOTREMOVE depends on MIGRATION +config MEMORY_HOTPLUG_MOVABLE + bool "Enhance the likelihood of hot-remove" + depends on MEMORY_HOTREMOVE + help + This option sets the hot-added memory zone to MOVABLEwhich
mm/Kconfig +1
-0+ drastically reduces the chance of a hot-remove to fail dueto
mm/Kconfig +1
-0+ unmovable memory segments. Kernel memory can't beallocated in
mm/Kconfig +3
-0+ this zone. + + Say Y here if you want to have better chance to hot-removememory
mm/Kconfig +2
-0+ that have been previously hot-added. + Say N here if you want to make all hot-added memoryavailable to
mm/Kconfig +2
-0+ kernel space. + # Heavily threaded applications may benefit from splitting the mm-wide
#page_table_lock, so that faults on different parts of the user
address# space can be handled with less contention: split it at this NR_CPUS.
Signature · 2 lines
Oscar Salvador SUSE L3
Quoted 3 lines · Oscar Salvador
We do already have "
movable_node" boot option, which exactly has that effect.
Any hotplugged range will be placed inZONE_MOVABLE.
Oh, I was not aware of it.
Quoted 2 lines
Why do we need yet another option to achieve the same? Was not that enough for your case?
Well, another use of this config could be doing this boot option a default on any given kernel.
But in the above case I agree it would be wiser to add the code on movable_node_is_enabled() directly, and not where I did put.
What do you think about it?
Thanks for the feedback,
Leonardo Brás
Quoted 6 lines · Leonardo Bras
Quoted 3 lines · Oscar Salvador
We do already have "
movable_node" boot option, which exactly has that effect.
Any hotplugged range will be placed inZONE_MOVABLE.Oh, I was not aware of it.
Quoted 8 lines
Quoted 2 lines
Why do we need yet another option to achieve the same? Was not that enough for your case?
Well, another use of this config could be doing this boot option a default on any given kernel.
But in the above case I agree it would be wiser to add the code onmovable_node_is_enabled()directly, and not where I did put.What do you think about it?
No further config options please. We do have means a more flexible way to achieve movable node onlining so let's use it. Or could you be more specific about cases which cannot use the command line option and really need a config option to workaround that?
Signature · 2 lines
Michal Hocko SUSE Labs
Quoted 15 lines · Michal Hocko
Quoted 13 lines · Leonardo Bras
Quoted 3 lines · Oscar Salvador
We do already have "
movable_node" boot option, which exactly has that effect.
Any hotplugged range will be placed inZONE_MOVABLE.Oh, I was not aware of it.
Quoted 2 lines
Why do we need yet another option to achieve the same? Was not that enough for your case?
Well, another use of this config could be doing this boot option a default on any given kernel.
But in the above case I agree it would be wiser to add the code onmovable_node_is_enabled()directly, and not where I did put.
Quoted 6 lines
Quoted 1 line
What do you think about it?
No further config options please. We do have means a more flexible way to achieve movable node onlining so let's use it. Or could you be more specific about cases which cannot use the command line option and really need a config option to workaround that?
Hi Michal,
Just trying to understand, if kernel parameters is the preferable method, why do we even have
MEMORY_HOTPLUG_DEFAULT_ONLINE
It is just strange that we have a config to online memory by default without kernel parameter, but no way to specify how to online it. It just looks as incomplete interface to me. Perhaps this config should be removed as well?
Pasha
Quoted 23 lines · Pavel Tatashin
Quoted 15 lines · Michal Hocko
Quoted 13 lines · Leonardo Bras
Quoted 3 lines · Oscar Salvador
We do already have "
movable_node" boot option, which exactly has that effect.
Any hotplugged range will be placed inZONE_MOVABLE.Oh, I was not aware of it.
Quoted 2 lines
Why do we need yet another option to achieve the same? Was not that enough for your case?
Well, another use of this config could be doing this boot option a default on any given kernel.
But in the above case I agree it would be wiser to add the code onmovable_node_is_enabled()directly, and not where I did put.Quoted 6 lines
Quoted 1 line
What do you think about it?
No further config options please. We do have means a more flexible way to achieve movable node onlining so let's use it. Or could you be more specific about cases which cannot use the command line option and really need a config option to workaround that?
Quoted 6 lines
Hi Michal,
Just trying to understand, if kernel parameters is the preferable method, why do we even have
MEMORY_HOTPLUG_DEFAULT_ONLINE
I have some opinion on this one TBH. I have even tried to remove it. The config option has been added to workaround hotplug issues for some memory balloning usecases where it was believed that the memory consumed for the memory hotadd (struct pages) could get machine to OOM before userspace manages to online it. So I would be more than happy to remove it but there were some objections in the past. Maybe the work by Oscar to allocate memmaps from the hotplugged memory can finally put an end to this gross hack.
In any case, I do not think we want to repeat that pattern again.
Signature · 2 lines
Michal Hocko SUSE Labs
Quoted 5 lines
Quoted 4 lines
Just trying to understand, if kernel parameters is the preferable method, why do we even have
MEMORY_HOTPLUG_DEFAULT_ONLINE
Quoted 8 lines
I have some opinion on this one TBH. I have even tried to remove it. The config option has been added to workaround hotplug issues for some memory balloning usecases where it was believed that the memory consumed for the memory hotadd (struct pages) could get machine to OOM before userspace manages to online it. So I would be more than happy to remove it but there were some objections in the past. Maybe the work by Oscar to allocate memmaps from the hotplugged memory can finally put an end to this gross hack.
Makes sense, thank you for the background info.
Pasha
Quoted 5 lines · Leonardo Bras
Adds an option on kernel config to make hot-added memory online in
ZONE_MOVABLEby default.This would be great in systems with MEMORY_HOTPLUG_DEFAULT_ONLINE=y by allowing to choose which zone it will be auto-onlined
Please add more elaborate description of the problem you are solving and the solution outline.
Quoted 11 lines
Signed-off-by Leonardo Bras <[email protected]>--- drivers/base/memory.c | 3 +++ mm/Kconfig | 14 ++++++++++++++ 2 files changed, 17 insertions(+)drivers/base/memory.c +0
-0
drivers/base/memory.c static int init_memory_block(struct memory_block **memory, · line 675
Quoted 6 lines
drivers/base/memory.c +3
-0@@ -670,6 +670,9 @@ static int init_memory_block(struct memory_block **memory, mem->state = state; start_pfn =section_nr_to_pfn (mem->start_section_nr); mem->phys_device =arch_get_memory_phys_device (start_pfn); +#ifdef CONFIG_MEMORY_HOTPLUG_MOVABLE + mem->online_type =MMOP_ONLINE_MOVABLE ; +#endif
Does it has to be a compile time option?
Seems like this can be changed at run time or at least at boot.
Quoted 30 lines
drivers/base/memory.c +0
-0ret =register_memory (mem);mm/Kconfig +14
-0@@ -180,6 +180,20 @@ config MEMORY_HOTREMOVE depends on MEMORY_HOTPLUG && ARCH_ENABLE_MEMORY_HOTREMOVE depends on MIGRATION +config MEMORY_HOTPLUG_MOVABLE + bool "Enhance the likelihood of hot-remove" + depends on MEMORY_HOTREMOVE + help + This option sets the hot-added memory zone to MOVABLE which + drastically reduces the chance of a hot-remove to fail due to + unmovable memory segments. Kernel memory can't be allocated in + this zone. + + Say Y here if you want to have better chance to hot-remove memory + that have been previously hot-added. + Say N here if you want to make all hot-added memory available to + kernel space. + # Heavily threaded applications may benefit from splitting the mm-wide # page_table_lock, so that faults on different parts of the user address # space can be handled with less contention: split it at this NR_CPUS.Signature · 1 line
2.20.1
Signature · 2 lines
Sincerely yours, Mike.
Quoted 6 lines · Leonardo Bras
Adds an option on kernel config to make hot-added memory online in
ZONE_MOVABLEby default.This would be great in systems with MEMORY_HOTPLUG_DEFAULT_ONLINE=y by allowing to choose which zone it will be auto-onlined
This is a desired feature. From reading the code it looks to me that auto-selection of online method type should be done in memory_subsys_online().
When it is called from device online, mem->online_type should be -1:
if (mem->online_type < 0)
mem->online_type = MMOP_ONLINE_KEEP;
Change it to:
if (mem->online_type < 0)
mem->online_type = MMOP_DEFAULT_ONLINE_TYPE;
And in "linux/memory_hotplug.h"
#ifdef CONFIG_MEMORY_HOTPLUG_MOVABLE
#define MMOP_DEFAULT_ONLINE_TYPE MMOP_ONLINE_MOVABLE #else
#define MMOP_DEFAULT_ONLINE_TYPE MMOP_ONLINE_KEEP #endif
Could be expanded to support MMOP_ONLINE_KERNEL as well.
Pasha
Quoted 49 lines
Signed-off-by Leonardo Bras <[email protected]>--- drivers/base/memory.c | 3 +++ mm/Kconfig | 14 ++++++++++++++ 2 files changed, 17 insertions(+)drivers/base/memory.c +3
-0@@ -670,6 +670,9 @@ static int init_memory_block(struct memory_block **memory, mem->state = state; start_pfn =section_nr_to_pfn (mem->start_section_nr); mem->phys_device =arch_get_memory_phys_device (start_pfn); +#ifdef CONFIG_MEMORY_HOTPLUG_MOVABLE + mem->online_type =MMOP_ONLINE_MOVABLE ; +#endif ret =register_memory (mem);mm/Kconfig +14
-1@@ -180,6 +180,20 @@ config MEMORY_HOTREMOVE depends on MEMORY_HOTPLUG && ARCH_ENABLE_MEMORY_HOTREMOVE depends on MIGRATION +config MEMORY_HOTPLUG_MOVABLE + bool "Enhance the likelihood of hot-remove" + depends on MEMORY_HOTREMOVE + help + This option sets the hot-added memory zone to MOVABLE which + drastically reduces the chance of a hot-remove to fail due to + unmovable memory segments. Kernel memory can't be allocated in + this zone. + + Say Y here if you want to have better chance to hot-remove memory + that have been previously hot-added. + Say N here if you want to make all hot-added memory available to + kernel space. + # Heavily threaded applications may benefit from splitting the mm-wide # page_table_lock, so that faults on different parts of the user address # space can be handled with less contention: split it at this NR_CPUS. --2.20.1
Quoted 55 lines · Leonardo Bras
Adds an option on kernel config to make hot-added memory online in
ZONE_MOVABLEby default.This would be great in systems with MEMORY_HOTPLUG_DEFAULT_ONLINE=y by allowing to choose which zone it will be auto-onlined
Signed-off-by Leonardo Bras <[email protected]>--- drivers/base/memory.c | 3 +++ mm/Kconfig | 14 ++++++++++++++ 2 files changed, 17 insertions(+)drivers/base/memory.c +3
-0@@ -670,6 +670,9 @@ static int init_memory_block(struct memory_block **memory, mem->state = state; start_pfn =section_nr_to_pfn (mem->start_section_nr); mem->phys_device =arch_get_memory_phys_device (start_pfn); +#ifdef CONFIG_MEMORY_HOTPLUG_MOVABLE + mem->online_type =MMOP_ONLINE_MOVABLE ; +#endif ret =register_memory (mem);mm/Kconfig +14
-1@@ -180,6 +180,20 @@ config MEMORY_HOTREMOVE depends on MEMORY_HOTPLUG && ARCH_ENABLE_MEMORY_HOTREMOVE depends on MIGRATION +config MEMORY_HOTPLUG_MOVABLE + bool "Enhance the likelihood of hot-remove" + depends on MEMORY_HOTREMOVE + help + This option sets the hot-added memory zone to MOVABLE which + drastically reduces the chance of a hot-remove to fail due to + unmovable memory segments. Kernel memory can't be allocated in + this zone. + + Say Y here if you want to have better chance to hot-remove memory + that have been previously hot-added. + Say N here if you want to make all hot-added memory available to + kernel space. + # Heavily threaded applications may benefit from splitting the mm-wide # page_table_lock, so that faults on different parts of the user address # space can be handled with less contention: split it at this NR_CPUS. --2.20.1
Quoted 27 lines · Pavel Tatashin
Quoted 5 lines · Leonardo Bras
Adds an option on kernel config to make hot-added memory online in
ZONE_MOVABLEby default.This would be great in systems with MEMORY_HOTPLUG_DEFAULT_ONLINE=y by allowing to choose which zone it will be auto-onlined
This is a desired feature. From reading the code it looks to me that auto-selection of online method type should be done in
memory_subsys_online().When it is called from device online,
mem->online_typeshould be -1:if (
mem->online_type< 0)mem->online_type = MMOP_ONLINE_KEEP;Change it to:
if (mem->online_type< 0)mem->online_type = MMOP_DEFAULT_ONLINE_TYPE;And in "
linux/memory_hotplug.h"
#ifdefCONFIG_MEMORY_HOTPLUG_MOVABLE
#defineMMOP_DEFAULT_ONLINE_TYPEMMOP_ONLINE_MOVABLE#else
#defineMMOP_DEFAULT_ONLINE_TYPEMMOP_ONLINE_KEEP#endif
Quoted 3 lines
Could be expanded to support
MMOP_ONLINE_KERNELas well.Pasha
Thanks for the suggestions Pasha,
I was made aware there is a kernel boot option "movable_node" that already creates the behavior I was trying to reproduce.
I was thinking of changing my patch in order to add a config option that makes this behavior default (i.e. not need to pass it as a boot parameter.
Do you think that it would still be a desired feature?
Regards,
Leonardo Brás
Quoted 29 lines · Leonardo Bras
Quoted 27 lines · Pavel Tatashin
Quoted 5 lines · Leonardo Bras
Adds an option on kernel config to make hot-added memory online in
ZONE_MOVABLEby default.This would be great in systems with MEMORY_HOTPLUG_DEFAULT_ONLINE=y by allowing to choose which zone it will be auto-onlined
This is a desired feature. From reading the code it looks to me that auto-selection of online method type should be done in
memory_subsys_online().When it is called from device online,
mem->online_typeshould be -1:if (
mem->online_type< 0)mem->online_type = MMOP_ONLINE_KEEP;Change it to:
if (mem->online_type< 0)mem->online_type = MMOP_DEFAULT_ONLINE_TYPE;And in "
linux/memory_hotplug.h"
#ifdefCONFIG_MEMORY_HOTPLUG_MOVABLE
#defineMMOP_DEFAULT_ONLINE_TYPEMMOP_ONLINE_MOVABLE#else
#defineMMOP_DEFAULT_ONLINE_TYPEMMOP_ONLINE_KEEP#endif
Quoted 8 lines
Quoted 3 lines
Could be expanded to support
MMOP_ONLINE_KERNELas well.Pasha
Thanks for the suggestions Pasha,
I was made aware there is a kernel boot option "
movable_node" that already creates the behavior I was trying to reproduce.
I agree with others, no need to duplicate this functionality in a config, and Michal in a separate e-mail explained the reasons why we have MEMORY_HOTPLUG_DEFAULT_ONLINE.
Quoted 10 lines
I was thinking of changing my patch in order to add a config option that makes this behavior default (i.e. not need to pass it as a boot parameter.
Do you think that it would still be a desired feature?
Regards,
Leonardo Brás