Skip to contentSkip to conversation

mm/hotplug: Only respect mem= parameter during boot stage

PATCH Superseded Baoquan He 5 participants 11 messages Dec 10, 2019

Subsystems (2)

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Baoquan He

author
#0 lore raw

PATCH v2 1/1 mm/hotplug: Only respect mem= parameter during boot stage

In commit 357b4da50a62 ("x86: respect memory size limiting via mem= parameter") a global varialbe global max_mem_size is added to store the value parsed from 'mem= ', then checked when memory region is added. This truly stops those DIMM from being added into system memory during boot-time.

However, it also limits the later memory hotplug functionality. Any memory board can't be hot added any more if its region is beyond the max_mem_size. System will print error like below:

[  216.387164] acpi PNP0C80:02: add_memory failed
[  216.389301] acpi PNP0C80:02: acpi_memory_enable_device() error
[  216.392187] acpi PNP0C80:02: Enumeration failure

From document of 'mem= ' parameter, it should be a restriction during boot, but not impact the system memory adding/removing after booting.

mem=nn[KMG] [KNL,BOOT] Force usage of a specific amount of memory

	          ...

So fix it by also checking if it's during boot-time when restrict memory adding. Otherwise, skip the restriction.

Fixes 357b4da50a62 ("x86: respect memory size limiting via mem= parameter")
---
 mm/memory_hotplug.c | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
mm/memory_hotplug.c +5 -1
index 55ac23ef11c1..989707295d15 100644
@@ -105,7 +105,11 @@ static struct resource *register_memory_resource(u64 start, u64 size)
 	unsigned long flags =  IORESOURCE_SYSTEM_RAM | IORESOURCE_BUSY;
 	char *resource_name = "System RAM";
 
-	if (start + size > max_mem_size)
+	/*
+	 * Make sure value parsed from 'mem=' only restricts memory adding
+	 * during boot-time, so that memory hotplug won't be impacted.
+	 */
+	if (start + size > max_mem_size && system_state < SYSTEM_RUNNING)
 		return ERR_PTR(-E2BIG);
 
 	/*
Signature · 1 line
2.17.2
Reviewed-by on v2 1/1
#1 lore raw
Quoted 17 lines · Baoquan He

In commit 357b4da50a62 ("x86: respect memory size limiting via mem= parameter") a global varialbe global max_mem_size is added to store the value parsed from 'mem= ', then checked when memory region is added. This truly stops those DIMM from being added into system memory during boot-time.

However, it also limits the later memory hotplug functionality. Any memory board can't be hot added any more if its region is beyond the max_mem_size. System will print error like below:

[  216.387164] acpi PNP0C80:02: add_memory failed
[  216.389301] acpi PNP0C80:02: acpi_memory_enable_device() error
[  216.392187] acpi PNP0C80:02: Enumeration failure

From document of 'mem= ' parameter, it should be a restriction during
boot, but not impact the system memory adding/removing after booting.

Quoted 8 lines

mem=nn[KMG] [KNL,BOOT] Force usage of a specific amount of memory

	          ...

So fix it by also checking if it's during boot-time when restrict memory adding. Otherwise, skip the restriction.

Fixes 357b4da50a62 ("x86: respect memory size limiting via mem= parameter")

Juergen

#2 lore raw
In reply to Baoquan He
Quoted 2 lines · Baoquan He

In commit 357b4da50a62 ("x86: respect memory size limiting via mem= parameter") a global varialbe global max_mem_size is added to store

                  typo ^^^
Quoted 12 lines

the value parsed from 'mem= ', then checked when memory region is added. This truly stops those DIMM from being added into system memory during boot-time.

However, it also limits the later memory hotplug functionality. Any memory board can't be hot added any more if its region is beyond the max_mem_size. System will print error like below:

[  216.387164] acpi PNP0C80:02: add_memory failed
[  216.389301] acpi PNP0C80:02: acpi_memory_enable_device() error
[  216.392187] acpi PNP0C80:02: Enumeration failure
Quoted 9 lines

From document of 'mem= ' parameter, it should be a restriction during boot, but not impact the system memory adding/removing after booting.

mem=nn[KMG] [KNL,BOOT] Force usage of a specific amount of memory

	          ...

So fix it by also checking if it's during boot-time when restrict memory adding. Otherwise, skip the restriction.

The fix looks reasonable, but I don't get the use case. Booting with mem= is generally a debug option, is this for debugging memory hotplug + limited memory?

Balbir

author
#3 lore raw
Quoted 24 lines · Balbir Singh
Quoted 2 lines · Baoquan He

In commit 357b4da50a62 ("x86: respect memory size limiting via mem= parameter") a global varialbe global max_mem_size is added to store

                  typo ^^^
Quoted 18 lines

the value parsed from 'mem= ', then checked when memory region is added. This truly stops those DIMM from being added into system memory during boot-time.

However, it also limits the later memory hotplug functionality. Any memory board can't be hot added any more if its region is beyond the max_mem_size. System will print error like below:

[  216.387164] acpi PNP0C80:02: add_memory failed
[  216.389301] acpi PNP0C80:02: acpi_memory_enable_device() error
[  216.392187] acpi PNP0C80:02: Enumeration failure

From document of 'mem= ' parameter, it should be a restriction during boot, but not impact the system memory adding/removing after booting.

mem=nn[KMG] [KNL,BOOT] Force usage of a specific amount of memory

	          ...
Quoted 6 lines
Quoted 3 lines

So fix it by also checking if it's during boot-time when restrict memory adding. Otherwise, skip the restriction.

The fix looks reasonable, but I don't get the use case. Booting with mem= is generally a debug option, is this for debugging memory hotplug + limited memory?

Not really. My understanding is whether 'mem= ' is for debugging or not, it should only take effect during boot-time. Even though people add 'mem= ' for debugging only, they will have to reboot if they want to do futher testing, e.g memory hotplug, or any other feature which makes use of the memory hotplug mechanism to implement their functionality, like ballon, or pmem.

Thanks
Baoquan

#4 lore raw
In reply to Balbir Singh
Quoted 24 lines · Balbir Singh
Quoted 2 lines · Baoquan He

In commit 357b4da50a62 ("x86: respect memory size limiting via mem= parameter") a global varialbe global max_mem_size is added to store

                  typo ^^^
Quoted 18 lines

the value parsed from 'mem= ', then checked when memory region is added. This truly stops those DIMM from being added into system memory during boot-time.

However, it also limits the later memory hotplug functionality. Any memory board can't be hot added any more if its region is beyond the max_mem_size. System will print error like below:

[  216.387164] acpi PNP0C80:02: add_memory failed
[  216.389301] acpi PNP0C80:02: acpi_memory_enable_device() error
[  216.392187] acpi PNP0C80:02: Enumeration failure

From document of 'mem= ' parameter, it should be a restriction during boot, but not impact the system memory adding/removing after booting.

mem=nn[KMG] [KNL,BOOT] Force usage of a specific amount of memory

	          ...
Quoted 6 lines
Quoted 3 lines

So fix it by also checking if it's during boot-time when restrict memory adding. Otherwise, skip the restriction.

The fix looks reasonable, but I don't get the use case. Booting with mem= is generally a debug option, is this for debugging memory hotplug + limited memory?

Some people/companies use "mem=" along with KVM e.g., to avoid allocating memmaps for guest backing memory and to not expose it to the buddy across kexec's. The excluded physical memory is then memmap into the hypervisor process and KVM can deal with that. I can imagine that hotplug might be desirable as well for such use cases.

Quoted 3 lines

Balbir

Signature · 3 lines
Thanks,

David / dhildenb
#5 lore raw
Quoted 29 lines · David Hildenbrand
Quoted 28 lines · Balbir Singh
Quoted 2 lines · Baoquan He

In commit 357b4da50a62 ("x86: respect memory size limiting via mem= parameter") a global varialbe global max_mem_size is added to store

                  typo ^^^
Quoted 21 lines

the value parsed from 'mem= ', then checked when memory region is added. This truly stops those DIMM from being added into system memory during boot-time.

However, it also limits the later memory hotplug functionality. Any memory board can't be hot added any more if its region is beyond the max_mem_size. System will print error like below:

[  216.387164] acpi PNP0C80:02: add_memory failed
[  216.389301] acpi PNP0C80:02: acpi_memory_enable_device() error
[  216.392187] acpi PNP0C80:02: Enumeration failure

From document of 'mem= ' parameter, it should be a restriction during boot, but not impact the system memory adding/removing after booting.

mem=nn[KMG] [KNL,BOOT] Force usage of a specific amount of memory

	          ...

So fix it by also checking if it's during boot-time when restrict memory adding. Otherwise, skip the restriction.

Quoted 6 lines
Quoted 2 lines

The fix looks reasonable, but I don't get the use case. Booting with mem= is generally a debug option, is this for debugging memory hotplug + limited memory?

Some people/companies use "mem=" along with KVM e.g., to avoid allocating memmaps for guest backing memory and to not expose it to the buddy across kexec's. The excluded physical memory is then memmap into

s/memmap/mmaped/

Quoted 8 lines

the hypervisor process and KVM can deal with that. I can imagine that hotplug might be desirable as well for such use cases.

Quoted 3 lines

Balbir

Signature · 3 lines
Thanks,

David / dhildenb
Acked-by
#6 lore raw
Quoted 30 lines · David Hildenbrand
Quoted 29 lines · David Hildenbrand
Quoted 28 lines · Balbir Singh
Quoted 2 lines · Baoquan He

In commit 357b4da50a62 ("x86: respect memory size limiting via mem= parameter") a global varialbe global max_mem_size is added to store

                  typo ^^^
Quoted 21 lines

the value parsed from 'mem= ', then checked when memory region is added. This truly stops those DIMM from being added into system memory during boot-time.

However, it also limits the later memory hotplug functionality. Any memory board can't be hot added any more if its region is beyond the max_mem_size. System will print error like below:

[  216.387164] acpi PNP0C80:02: add_memory failed
[  216.389301] acpi PNP0C80:02: acpi_memory_enable_device() error
[  216.392187] acpi PNP0C80:02: Enumeration failure

From document of 'mem= ' parameter, it should be a restriction during boot, but not impact the system memory adding/removing after booting.

mem=nn[KMG] [KNL,BOOT] Force usage of a specific amount of memory

	          ...

So fix it by also checking if it's during boot-time when restrict memory adding. Otherwise, skip the restriction.

Quoted 11 lines
Quoted 6 lines
Quoted 2 lines

The fix looks reasonable, but I don't get the use case. Booting with mem= is generally a debug option, is this for debugging memory hotplug + limited memory?

Some people/companies use "mem=" along with KVM e.g., to avoid allocating memmaps for guest backing memory and to not expose it to the buddy across kexec's. The excluded physical memory is then memmap into

s/memmap/mmaped/

Quoted 2 lines

the hypervisor process and KVM can deal with that. I can imagine that hotplug might be desirable as well for such use cases.

Makes sense, sounds hacky, but it seems like mem= is no longer a debug option

Quoted 8 lines
Quoted 6 lines
Quoted 4 lines

Balbir

#7 lore raw
In reply to David Hildenbrand
Quoted 29 lines · David Hildenbrand
Quoted 28 lines · Balbir Singh
Quoted 2 lines · Baoquan He

In commit 357b4da50a62 ("x86: respect memory size limiting via mem= parameter") a global varialbe global max_mem_size is added to store

                  typo ^^^
Quoted 21 lines

the value parsed from 'mem= ', then checked when memory region is added. This truly stops those DIMM from being added into system memory during boot-time.

However, it also limits the later memory hotplug functionality. Any memory board can't be hot added any more if its region is beyond the max_mem_size. System will print error like below:

[  216.387164] acpi PNP0C80:02: add_memory failed
[  216.389301] acpi PNP0C80:02: acpi_memory_enable_device() error
[  216.392187] acpi PNP0C80:02: Enumeration failure

From document of 'mem= ' parameter, it should be a restriction during boot, but not impact the system memory adding/removing after booting.

mem=nn[KMG] [KNL,BOOT] Force usage of a specific amount of memory

	          ...

So fix it by also checking if it's during boot-time when restrict memory adding. Otherwise, skip the restriction.

Quoted 8 lines
Quoted 2 lines

The fix looks reasonable, but I don't get the use case. Booting with mem= is generally a debug option, is this for debugging memory hotplug + limited memory?

Some people/companies use "mem=" along with KVM e.g., to avoid allocating memmaps for guest backing memory and to not expose it to the buddy across kexec's. The excluded physical memory is then memmap into the hypervisor process and KVM can deal with that. I can imagine that hotplug might be desirable as well for such use cases.

If this is really the usecase (it makes some sense to me) then it should be folded into the changelog. Because the real semantic is not really clear as I've pointed out in the previous version of this patch [1]. The restriction to BOOT is documented since ever long before the memory hotplug was a thing.

Thanks!

[1] Btw. it would have been much better if you posted the version 2 only after all the feedback got discussed properly.

Signature · 2 lines
Michal Hocko
SUSE Labs
author
#8 lore raw
Quoted 39 lines · Michal Hocko
Quoted 29 lines · David Hildenbrand
Quoted 28 lines · Balbir Singh
Quoted 2 lines · Baoquan He

In commit 357b4da50a62 ("x86: respect memory size limiting via mem= parameter") a global varialbe global max_mem_size is added to store

                  typo ^^^
Quoted 21 lines

the value parsed from 'mem= ', then checked when memory region is added. This truly stops those DIMM from being added into system memory during boot-time.

However, it also limits the later memory hotplug functionality. Any memory board can't be hot added any more if its region is beyond the max_mem_size. System will print error like below:

[  216.387164] acpi PNP0C80:02: add_memory failed
[  216.389301] acpi PNP0C80:02: acpi_memory_enable_device() error
[  216.392187] acpi PNP0C80:02: Enumeration failure

From document of 'mem= ' parameter, it should be a restriction during boot, but not impact the system memory adding/removing after booting.

mem=nn[KMG] [KNL,BOOT] Force usage of a specific amount of memory

	          ...

So fix it by also checking if it's during boot-time when restrict memory adding. Otherwise, skip the restriction.

Quoted 8 lines
Quoted 2 lines

The fix looks reasonable, but I don't get the use case. Booting with mem= is generally a debug option, is this for debugging memory hotplug + limited memory?

Some people/companies use "mem=" along with KVM e.g., to avoid allocating memmaps for guest backing memory and to not expose it to the buddy across kexec's. The excluded physical memory is then memmap into the hypervisor process and KVM can deal with that. I can imagine that hotplug might be desirable as well for such use cases.

Quoted 5 lines

If this is really the usecase (it makes some sense to me) then it should be folded into the changelog. Because the real semantic is not really clear as I've pointed out in the previous version of this patch [1]. The restriction to BOOT is documented since ever long before the memory hotplug was a thing.

I will hold a while and post v3 to take this into log.

Quoted 3 lines

[1] Btw. it would have been much better if you posted the version 2 only after all the feedback got discussed properly.

You could have replied to wrong person.

I don't know the use case David told. My motivation was to make memory hotplug not impacted after boot. I thought I have got your question and answered it. I will wait a little longer when post next time.

Thanks
Baoquan

#9 lore raw
Quoted 1 line · Baoquan He

On 12/10/19 at 10:50am, Michal Hocko wrote:

[...]

Quoted 8 lines
Quoted 2 lines

[1] Btw. it would have been much better if you posted the version 2 only after all the feedback got discussed properly.

You could have replied to wrong person.

I don't know the use case David told. My motivation was to make memory hotplug not impacted after boot. I thought I have got your question and answered it. I will wait a little longer when post next time.

let me reply to your original email.

Signature · 2 lines
Michal Hocko
SUSE Labs
Reviewed-by on v2 1/1
#10 lore raw
In reply to Baoquan He
Quoted 2 lines · Baoquan He

In commit 357b4da50a62 ("x86: respect memory size limiting via mem= parameter") a global varialbe global max_mem_size is added to store

Only some nits:

s/varialbe global/variable/

Quoted 2 lines

the value parsed from 'mem= ', then checked when memory region is added. This truly stops those DIMM from being added into system memory

s/DIMM/DIMMs/

Quoted 4 lines

during boot-time.

However, it also limits the later memory hotplug functionality. Any memory board can't be hot added any more if its region is beyond the

s/memory board/DIMM/ ?

s/hot added/hotplugged/ ?

Quoted 1 line

max_mem_size. System will print error like below:

"We will get errors like:"

Quoted 6 lines
[  216.387164] acpi PNP0C80:02: add_memory failed
[  216.389301] acpi PNP0C80:02: acpi_memory_enable_device() error
[  216.392187] acpi PNP0C80:02: Enumeration failure

From document of 'mem= ' parameter, it should be a restriction during

"the documentation of the"

Quoted 1 line

boot, but not impact the system memory adding/removing after booting.

"but not impact memory hotplug once booted."

Quoted 5 lines

mem=nn[KMG] [KNL,BOOT] Force usage of a specific amount of memory

	          ...

So fix it by also checking if it's during boot-time when restrict memory

s/when restrict memory adding/restricting to add memory/

Quoted 13 lines

adding. Otherwise, skip the restriction.

Fixes 357b4da50a62 ("x86: respect memory size limiting via mem= parameter")
---
 mm/memory_hotplug.c | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
mm/memory_hotplug.c +0 -0
index 55ac23ef11c1..989707295d15 100644

mm/memory_hotplug.c static struct resource *register_memory_resource(u64 start, u64 size) · line 110

Quoted 7 lines
mm/memory_hotplug.c +3 -1
@@ -105,7 +105,11 @@ static struct resource *register_memory_resource(u64 start, u64 size)
 	unsigned long flags =  IORESOURCE_SYSTEM_RAM | IORESOURCE_BUSY;
 	char *resource_name = "System RAM";
 
-	if (start + size > max_mem_size)
+	/*
+	 * Make sure value parsed from 'mem=' only restricts memory adding
+	 * during boot-time, so that memory hotplug won't be impacted.

s/during boot-time/while booting/

Only some nits, thanks!

Signature · 3 lines
Thanks,

David / dhildenb