Skip to contentSkip to conversation

x86: respect memory size limits

PATCH In review Juergen Gross 3 participants 11 messages Jan 22, 2019 – Jan 23, 2019

Subsystems (4)
lore
Share & tools
lore

Juergen Gross

author maintainer
#0 lore raw

cover x86: respect memory size limits

On a customer system running Xen a boot problem was observed due to the kernel not respecting the memory size limit imposed by the Xen hypervisor.

During analysis I found the same problem should be able to occur on bare metal in case the memory would be limited via the "mem=" boot parameter.

The system this problem has been observed on has tons of memory added via PCI. So while in the E820 map the not to be used memory has been wiped out the additional PCI memory is detected during ACPI scan and it is added via __add_memory().

This small series tries to repair the issue by testing the imposed memory limit during the memory hotplug process and refusing to add it in case the limit is being violated.

I've chosen to refuse adding the complete memory chunk in case the limit is reached instead of adding only some of the memory, as I thought this would result in less problems (e.g. avoiding to add only parts of a 128MB memory bar which might be difficult to remove later).

Juergen Gross (2):
x86: respect memory size limiting via mem= parameter x86/xen: dont add memory above max allowed allocation

 arch/x86/kernel/e820.c         | 5 +++++
 arch/x86/xen/setup.c           | 5 +++++
 include/linux/memory_hotplug.h | 2 ++
 mm/memory_hotplug.c            | 6 ++++++
 4 files changed, 18 insertions(+)
Signature · 1 line
2.16.4
author maintainer
#1 lore raw

PATCH 1/2 x86: respect memory size limiting via mem= parameter

When limiting memory size via kernel parameter "mem=" this should be respected even in case of memory made accessible via a PCI card.

Today this kind of memory won't be made usable in initial memory setup as the memory won't be visible in E820 map, but it might be added when adding PCI devices due to corresponding ACPI table entries.

Not respecting "mem=" can be corrected by adding a global max_mem_size variable set by parse_memopt() which will result in rejecting adding memory areas resulting in a memory size above the allowed limit.

---
 arch/x86/kernel/e820.c         | 5 +++++
 include/linux/memory_hotplug.h | 2 ++
 mm/memory_hotplug.c            | 6 ++++++
 3 files changed, 13 insertions(+)
arch/x86/kernel/e820.c +5 -0
index 50895c2f937d..e67513e2cbbb 100644
@@ -14,6 +14,7 @@
 #include <linux/acpi.h>
 #include <linux/firmware-map.h>
 #include <linux/sort.h>
+#include <linux/memory_hotplug.h>
 
 #include <asm/e820/api.h>
 #include <asm/setup.h>
@@ -881,6 +882,10 @@ static int __init parse_memopt(char *p)
 
 	e820__range_remove(mem_size, ULLONG_MAX - mem_size, E820_TYPE_RAM, 1);
 
+#ifdef CONFIG_MEMORY_HOTPLUG
+	max_mem_size = mem_size;
+#endif
+
 	return 0;
 }
 early_param("mem", parse_memopt);
include/linux/memory_hotplug.h +2 -0
index 07da5c6c5ba0..fb6bd0022d41 100644
@@ -98,6 +98,8 @@ extern void __online_page_free(struct page *page);
 
 extern int try_online_node(int nid);
 
+extern u64 max_mem_size;
+
 extern bool memhp_auto_online;
 /* If movable_node boot option specified */
 extern bool movable_node_enabled;
mm/memory_hotplug.c +6 -0
index b9a667d36c55..7fc2a87110a3 100644
@@ -96,10 +96,16 @@ void mem_hotplug_done(void)
 	cpus_read_unlock();
 }
 
+u64 max_mem_size = -1;
+
 /* add this memory to iomem resource */
 static struct resource *register_memory_resource(u64 start, u64 size)
 {
 	struct resource *res, *conflict;
+
+	if (start + size > max_mem_size)
+		return ERR_PTR(-E2BIG);
+
 	res = kzalloc(sizeof(struct resource), GFP_KERNEL);
 	if (!res)
 		return ERR_PTR(-ENOMEM);
Signature · 1 line
2.16.4
#2 lore raw

Re: [PATCH 1/2] x86: respect memory size limiting via mem= parameter

mm/memory_hotplug.c void mem_hotplug_done(void) · line 99

Quoted 13 lines · Juergen Gross
mm/memory_hotplug.c +1 -0
index b9a667d36c55..7fc2a87110a3 100644
@@ -96,10 +96,16 @@ void mem_hotplug_done(void)
 	cpus_read_unlock();
 }
 
+u64 max_mem_size = -1;

This may be pedantic, but I'd rather see U64_MAX used here.

author maintainer
#3 lore raw

Re: [PATCH 1/2] x86: respect memory size limiting via mem= parameter

Quoted 12 lines · William Kucharski
Quoted 12 lines · Juergen Gross
mm/memory_hotplug.c +0 -0
index b9a667d36c55..7fc2a87110a3 100644
@@ -96,10 +96,16 @@ void mem_hotplug_done(void)
 	cpus_read_unlock();
 }
 
Quoted 3 lines

mm/memory_hotplug.c void mem_hotplug_done(void) · line 99

Quoted 1 line
mm/memory_hotplug.c +1 -0
+u64 max_mem_size = -1;

This may be pedantic, but I'd rather see U64_MAX used here.

Fine with me. Will change.

Juergen

author maintainer
#4 lore raw

Re: [PATCH 1/2] x86: respect memory size limiting via mem= parameter

In reply to William Kucharski
Quoted 12 lines · William Kucharski
Quoted 12 lines · Juergen Gross
mm/memory_hotplug.c +0 -0
index b9a667d36c55..7fc2a87110a3 100644
@@ -96,10 +96,16 @@ void mem_hotplug_done(void)
 	cpus_read_unlock();
 }
 
Quoted 3 lines

mm/memory_hotplug.c void mem_hotplug_done(void) · line 99

Quoted 1 line
mm/memory_hotplug.c +1 -0
+u64 max_mem_size = -1;

This may be pedantic, but I'd rather see U64_MAX used here.

Fine with me. Will change.

Juergen

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#5 lore raw

Re: [PATCH 1/2] x86: respect memory size limiting via mem= parameter

In reply to Juergen Gross

mm/memory_hotplug.c void mem_hotplug_done(void) · line 99

Quoted 13 lines · Juergen Gross
mm/memory_hotplug.c +1 -0
index b9a667d36c55..7fc2a87110a3 100644
@@ -96,10 +96,16 @@ void mem_hotplug_done(void)
 	cpus_read_unlock();
 }
 
+u64 max_mem_size = -1;

This may be pedantic, but I'd rather see U64_MAX used here.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

author maintainer
#6 lore raw

PATCH 1/2 x86: respect memory size limiting via mem= parameter

In reply to Juergen Gross

When limiting memory size via kernel parameter "mem=" this should be respected even in case of memory made accessible via a PCI card.

Today this kind of memory won't be made usable in initial memory setup as the memory won't be visible in E820 map, but it might be added when adding PCI devices due to corresponding ACPI table entries.

Not respecting "mem=" can be corrected by adding a global max_mem_size variable set by parse_memopt() which will result in rejecting adding memory areas resulting in a memory size above the allowed limit.

---
 arch/x86/kernel/e820.c         | 5 +++++
 include/linux/memory_hotplug.h | 2 ++
 mm/memory_hotplug.c            | 6 ++++++
 3 files changed, 13 insertions(+)
arch/x86/kernel/e820.c +5 -0
index 50895c2f937d..e67513e2cbbb 100644
@@ -14,6 +14,7 @@
 #include <linux/acpi.h>
 #include <linux/firmware-map.h>
 #include <linux/sort.h>
+#include <linux/memory_hotplug.h>
 
 #include <asm/e820/api.h>
 #include <asm/setup.h>
@@ -881,6 +882,10 @@ static int __init parse_memopt(char *p)
 
 	e820__range_remove(mem_size, ULLONG_MAX - mem_size, E820_TYPE_RAM, 1);
 
+#ifdef CONFIG_MEMORY_HOTPLUG
+	max_mem_size = mem_size;
+#endif
+
 	return 0;
 }
 early_param("mem", parse_memopt);
include/linux/memory_hotplug.h +2 -0
index 07da5c6c5ba0..fb6bd0022d41 100644
@@ -98,6 +98,8 @@ extern void __online_page_free(struct page *page);
 
 extern int try_online_node(int nid);
 
+extern u64 max_mem_size;
+
 extern bool memhp_auto_online;
 /* If movable_node boot option specified */
 extern bool movable_node_enabled;
mm/memory_hotplug.c +6 -0
index b9a667d36c55..7fc2a87110a3 100644
@@ -96,10 +96,16 @@ void mem_hotplug_done(void)
 	cpus_read_unlock();
 }
 
+u64 max_mem_size = -1;
+
 /* add this memory to iomem resource */
 static struct resource *register_memory_resource(u64 start, u64 size)
 {
 	struct resource *res, *conflict;
+
+	if (start + size > max_mem_size)
+		return ERR_PTR(-E2BIG);
+
 	res = kzalloc(sizeof(struct resource), GFP_KERNEL);
 	if (!res)
 		return ERR_PTR(-ENOMEM);
Signature · 7 lines
2.16.4


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
author maintainer
#7 lore raw

PATCH 2/2 x86/xen: dont add memory above max allowed allocation

In reply to Juergen Gross

Don't allow memory to be added above the allowed maximum allocation limit set by Xen.

Trying to do so would result in cases like the following:

[  584.559652] ------------[ cut here ]------------
[  584.564897] WARNING: CPU: 2 PID: 1 at ../arch/x86/xen/multicalls.c:129 xen_alloc_pte+0x1c7/0x390()
[  584.575151] Modules linked in:
[  584.578643] Supported: Yes
[  584.581750] CPU: 2 PID: 1 Comm: swapper/0 Not tainted 4.4.120-92.70-default #1
[  584.590000] Hardware name: Cisco Systems Inc UCSC-C460-M4/UCSC-C460-M4, BIOS C460M4.4.0.1b.0.0629181419 06/29/2018
[  584.601862]  0000000000000000 ffffffff813175a0 0000000000000000 ffffffff8184777c
[  584.610200]  ffffffff8107f4e1 ffff880487eb7000 ffff8801862b79c0 ffff88048608d290
[  584.618537]  0000000000487eb7 ffffea0000000201 ffffffff81009de7 ffffffff81068561
[  584.626876] Call Trace:
[  584.629699]  [<ffffffff81019ad9>] dump_trace+0x59/0x340
[  584.635645]  [<ffffffff81019eaa>] show_stack_log_lvl+0xea/0x170
[  584.642391]  [<ffffffff8101ac51>] show_stack+0x21/0x40
[  584.648238]  [<ffffffff813175a0>] dump_stack+0x5c/0x7c
[  584.654085]  [<ffffffff8107f4e1>] warn_slowpath_common+0x81/0xb0
[  584.660932]  [<ffffffff81009de7>] xen_alloc_pte+0x1c7/0x390
[  584.667289]  [<ffffffff810647f0>] pmd_populate_kernel.constprop.6+0x40/0x80
[  584.675241]  [<ffffffff815ecfe8>] phys_pmd_init+0x210/0x255
[  584.681587]  [<ffffffff815ed207>] phys_pud_init+0x1da/0x247
[  584.687931]  [<ffffffff815edb3b>] kernel_physical_mapping_init+0xf5/0x1d4
[  584.695682]  [<ffffffff815e9bdd>] init_memory_mapping+0x18d/0x380
[  584.702631]  [<ffffffff81064699>] arch_add_memory+0x59/0xf0
---
 arch/x86/xen/setup.c | 5 +++++
 1 file changed, 5 insertions(+)
arch/x86/xen/setup.c +5 -0
index d5f303c0e656..5929a6ba5c25 100644
@@ -12,6 +12,7 @@
 #include <linux/memblock.h>
 #include <linux/cpuidle.h>
 #include <linux/cpufreq.h>
+#include <linux/memory_hotplug.h>
 
 #include <asm/elf.h>
 #include <asm/vdso.h>
@@ -785,6 +786,10 @@ char * __init xen_memory_setup(void)
 	/* How many extra pages do we need due to remapping? */
 	max_pages += xen_foreach_remap_area(max_pfn, xen_count_remap_pages);
 
+#ifdef CONFIG_MEMORY_HOTPLUG
+	max_mem_size = PFN_PHYS(max_pages);
+#endif
+
 	if (max_pages > max_pfn)
 		extra_pages += max_pages - max_pfn;
 
Signature · 1 line
2.16.4
#8 lore raw

Re: [Xen-devel] [PATCH 2/2] x86/xen: dont add memory above max allowed allocation

Quoted 3 lines
Quoted 1 line
Quoted 1 line · On 22.01.19 at 09:06, <[email protected]> wrote:

Don't allow memory to be added above the allowed maximum allocation limit set by Xen.

This reads as if the hypervisor was imposing a limit here, but looking at xen_get_max_pages(), xen_foreach_remap_area(), and xen_count_remap_pages() I take it that it's a restriction enforced by the Xen subsystem in Linux. Furthermore from the cover letter I imply that the observed issue was on a Dom0, yet xen_get_max_pages()'s use of XENMEM_maximum_reservation wouldn't impose any limit there at all (without use of the hypervisor option "dom0_mem=max:..."), would it?

Jan

#9 lore raw

Re: [PATCH 2/2] x86/xen: dont add memory above max allowed allocation

In reply to Juergen Gross
Quoted 3 lines
Quoted 1 line
Quoted 1 line · On 22.01.19 at 09:06, <[email protected]> wrote:

Don't allow memory to be added above the allowed maximum allocation limit set by Xen.

This reads as if the hypervisor was imposing a limit here, but looking at xen_get_max_pages(), xen_foreach_remap_area(), and xen_count_remap_pages() I take it that it's a restriction enforced by the Xen subsystem in Linux. Furthermore from the cover letter I imply that the observed issue was on a Dom0, yet xen_get_max_pages()'s use of XENMEM_maximum_reservation wouldn't impose any limit there at all (without use of the hypervisor option "dom0_mem=max:..."), would it?

Jan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

author maintainer
#10 lore raw

PATCH 2/2 x86/xen: dont add memory above max allowed allocation

In reply to Juergen Gross

Don't allow memory to be added above the allowed maximum allocation limit set by Xen.

Trying to do so would result in cases like the following:

[  584.559652] ------------[ cut here ]------------
[  584.564897] WARNING: CPU: 2 PID: 1 at ../arch/x86/xen/multicalls.c:129 xen_alloc_pte+0x1c7/0x390()
[  584.575151] Modules linked in:
[  584.578643] Supported: Yes
[  584.581750] CPU: 2 PID: 1 Comm: swapper/0 Not tainted 4.4.120-92.70-default #1
[  584.590000] Hardware name: Cisco Systems Inc UCSC-C460-M4/UCSC-C460-M4, BIOS C460M4.4.0.1b.0.0629181419 06/29/2018
[  584.601862]  0000000000000000 ffffffff813175a0 0000000000000000 ffffffff8184777c
[  584.610200]  ffffffff8107f4e1 ffff880487eb7000 ffff8801862b79c0 ffff88048608d290
[  584.618537]  0000000000487eb7 ffffea0000000201 ffffffff81009de7 ffffffff81068561
[  584.626876] Call Trace:
[  584.629699]  [<ffffffff81019ad9>] dump_trace+0x59/0x340
[  584.635645]  [<ffffffff81019eaa>] show_stack_log_lvl+0xea/0x170
[  584.642391]  [<ffffffff8101ac51>] show_stack+0x21/0x40
[  584.648238]  [<ffffffff813175a0>] dump_stack+0x5c/0x7c
[  584.654085]  [<ffffffff8107f4e1>] warn_slowpath_common+0x81/0xb0
[  584.660932]  [<ffffffff81009de7>] xen_alloc_pte+0x1c7/0x390
[  584.667289]  [<ffffffff810647f0>] pmd_populate_kernel.constprop.6+0x40/0x80
[  584.675241]  [<ffffffff815ecfe8>] phys_pmd_init+0x210/0x255
[  584.681587]  [<ffffffff815ed207>] phys_pud_init+0x1da/0x247
[  584.687931]  [<ffffffff815edb3b>] kernel_physical_mapping_init+0xf5/0x1d4
[  584.695682]  [<ffffffff815e9bdd>] init_memory_mapping+0x18d/0x380
[  584.702631]  [<ffffffff81064699>] arch_add_memory+0x59/0xf0
---
 arch/x86/xen/setup.c | 5 +++++
 1 file changed, 5 insertions(+)
arch/x86/xen/setup.c +5 -0
index d5f303c0e656..5929a6ba5c25 100644
@@ -12,6 +12,7 @@
 #include <linux/memblock.h>
 #include <linux/cpuidle.h>
 #include <linux/cpufreq.h>
+#include <linux/memory_hotplug.h>
 
 #include <asm/elf.h>
 #include <asm/vdso.h>
@@ -785,6 +786,10 @@ char * __init xen_memory_setup(void)
 	/* How many extra pages do we need due to remapping? */
 	max_pages += xen_foreach_remap_area(max_pfn, xen_count_remap_pages);
 
+#ifdef CONFIG_MEMORY_HOTPLUG
+	max_mem_size = PFN_PHYS(max_pages);
+#endif
+
 	if (max_pages > max_pfn)
 		extra_pages += max_pages - max_pfn;
 
Signature · 7 lines
2.16.4


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel