Skip to contentSkip to conversation

mm: hugetlb: optionally allocate gigantic hugepages using cma 65;5803;1c Commit 944d9fec8d7a ("hugetlb: add support for gigantic page allocation at runtime") has added the run-time allocation of gigantic pages. However it actually works only at early stages of the system loading, when the majority of memory is free. After some time the memory gets fragmented by non-movable pages, so the chances to find a contiguous 1 GB block are getting close to zero. Even dropping caches manually doesn't help a lot.

PATCH In review Roman Gushchin 5 participants 8 messages Mar 11, 2020 – Apr 3, 2020

Subsystems (5)
lore
Share & tools
lore
#0 lore raw

PATCH v3 1/1 mm: hugetlb: optionally allocate gigantic hugepages using cma 65;5803;1c Commit 944d9fec8d7a ("hugetlb: add support for gigantic page allocation at runtime") has added the run-time allocation of gigantic pages. However it actually works only at early stages of the system loading, when the majority of memory is free. After some time the memory gets fragmented by non-movable pages, so the chances to find a contiguous 1 GB block are getting close to zero. Even dropping caches manually doesn't help a lot.

At large scale rebooting servers in order to allocate gigantic hugepages is quite expensive and complex. At the same time keeping some constant percentage of memory in reserved hugepages even if the workload isn't using it is a big waste: not all workloads can benefit from using 1 GB pages.

The following solution can solve the problem:
1) On boot time a dedicated cma area* is reserved. The size is passed as a kernel argument.
2) Run-time allocations of gigantic hugepages are performed using the cma allocator and the dedicated cma area

In this case gigantic hugepages can be allocated successfully with a high probability, however the memory isn't completely wasted if nobody is using 1GB hugepages: it can be used for pagecache, anon memory, THPs, etc.

* On a multi-node machine a per-node cma area is allocated on each node. Following gigantic hugetlb allocation are using the first available numa node if the mask isn't specified by a user.

Usage:
1) configure the kernel to allocate a cma area for hugetlb allocations:
pass hugetlb_cma=10G as a kernel argument

2) allocate hugetlb pages as usual, e.g.
echo 10 > /sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages

If the option isn't enabled or the allocation of the cma area failed, the current behavior of the system is preserved.

x86 and arm-64 are covered by this patch, other architectures can be trivially added later.

v3:
- added fallback to the existing allocation mechanism
- added min/max checks
- switched to MiB in debug output
- removed percentage option
- added arch-specific order argument to determine an alignment
- added arm support
- fixed the !CONFIG_HUGETLBFS build

Thanks to Michal, Mike, Andreas and Rik for ideas and suggestions!

v2:
-fixed !CONFIG_CMA build, suggested by Andrew Morton

---
 .../admin-guide/kernel-parameters.txt         |   7 ++
 arch/arm64/mm/init.c                          |   6 +
 arch/x86/kernel/setup.c                       |   4 +
 include/linux/hugetlb.h                       |   8 ++
 mm/hugetlb.c                                  | 116 ++++++++++++++++++
 5 files changed, 141 insertions(+)
Documentation/admin-guide/kernel-parameters.txt +7 -0
index 0c9894247015..9eb0df40643d 100644
@@ -1452,6 +1452,13 @@
 	hpet_mmap=	[X86, HPET_MMAP] Allow userspace to mmap HPET
 			registers.  Default set by CONFIG_HPET_MMAP_DEFAULT.
 
+	hugetlb_cma=	[x86-64] The size of a cma area used for allocation
+			of gigantic hugepages.
+			Format: nn[KMGTPE]
+
+			If enabled, boot-time allocation of gigantic hugepages
+			is skipped.
+
 	hugepages=	[HW,X86-32,IA-64] HugeTLB pages to allocate at boot.
 	hugepagesz=	[HW,IA-64,PPC,X86-64] The size of the HugeTLB pages.
 			On x86-64 and powerpc, this option can be specified
arch/arm64/mm/init.c +6 -0
index b65dffdfb201..e42727e3568e 100644
@@ -29,6 +29,7 @@
 #include <linux/mm.h>
 #include <linux/kexec.h>
 #include <linux/crash_dump.h>
+#include <linux/hugetlb.h>
 
 #include <asm/boot.h>
 #include <asm/fixmap.h>
@@ -457,6 +458,11 @@ void __init arm64_memblock_init(void)
 	high_memory = __va(memblock_end_of_DRAM() - 1) + 1;
 
 	dma_contiguous_reserve(arm64_dma32_phys_limit);
+
+#ifdef CONFIG_ARM64_4K_PAGES
+	hugetlb_cma_reserve(PUD_SHIFT - PAGE_SHIFT);
+#endif
+
 }
 
 void __init bootmem_init(void)
arch/x86/kernel/setup.c +4 -0
index a74262c71484..fc3e326a62b9 100644
@@ -16,6 +16,7 @@
 #include <linux/pci.h>
 #include <linux/root_dev.h>
 #include <linux/sfi.h>
+#include <linux/hugetlb.h>
 #include <linux/tboot.h>
 #include <linux/usb/xhci-dbgp.h>
 
@@ -1158,6 +1159,9 @@ void __init setup_arch(char **cmdline_p)
 	initmem_init();
 	dma_contiguous_reserve(max_pfn_mapped << PAGE_SHIFT);
 
+	if (boot_cpu_has(X86_FEATURE_GBPAGES))
+		hugetlb_cma_reserve(PUD_SHIFT - PAGE_SHIFT);
+
 	/*
 	 * Reserve memory for crash kernel after SRAT is parsed so that it
 	 * won't consume hotpluggable memory.
include/linux/hugetlb.h +8 -0
index 50480d16bd33..b831e9fa1a26 100644
@@ -890,4 +890,12 @@ static inline spinlock_t *huge_pte_lock(struct hstate *h,
 	return ptl;
 }
 
+#if defined(CONFIG_HUGETLB_PAGE) && defined(CONFIG_CMA)
+extern void __init hugetlb_cma_reserve(int order);
+#else
+static inline __init void hugetlb_cma_reserve(int order)
+{
+}
+#endif
+
 #endif /* _LINUX_HUGETLB_H */
mm/hugetlb.c +116 -0

149 lines not on this page · raw message

Signature · 1 line
2.24.1
#1 lore raw

Re: [PATCH v3] mm: hugetlb: optionally allocate gigantic hugepages using cma

Quoted 6 lines · Roman Gushchin
mm/hugetlb.c +6 -0
+#ifdef CONFIG_CMA
+static unsigned long hugetlb_cma_size __initdata;
+
+static int __init cmdline_parse_hugetlb_cma(char *p)
+{
+	unsigned long long val;

mm/hugetlb.c void move_hugetlb_state(struct page *oldpage, struct page *newpage, int reason) · line 5422

Quoted 10 lines
mm/hugetlb.c +10 -0
+	char *endptr;
+
+	if (!p)
+		return -EINVAL;
+
+	val = simple_strtoull(p, &endptr, 0);
+	hugetlb_cma_size = memparse(p, &p);
+	return 0;
+}
+

Here will generate a compilation warning,

mm/hugetlb.c: In function 'cmdline_parse_hugetlb_cma':
mm/hugetlb.c:5548:21: warning: variable 'val' set but not used [-Wunused-but- set-variable]
  unsigned long long val;
                     ^~~
Also, the comments for simple_strtoull() in lib/vsprintf.c said,

"This function is obsolete. Please use kstrtoull instead."

#2 lore raw

Re: [PATCH v3] mm: hugetlb: optionally allocate gigantic hugepages using cma

Quoted 20 lines · Qian Cai
Quoted 6 lines · Roman Gushchin
(quoted diff) +6 -0
+#ifdef CONFIG_CMA
+static unsigned long hugetlb_cma_size __initdata;
+
+static int __init cmdline_parse_hugetlb_cma(char *p)
+{
+	unsigned long long val;

Review

Quoted 10 lines
(quoted diff) +10 -0
+	char *endptr;
+
+	if (!p)
+		return -EINVAL;
+
+	val = simple_strtoull(p, &endptr, 0);
+	hugetlb_cma_size = memparse(p, &p);
+	return 0;
+}
+

Here will generate a compilation warning,

Quoted 9 lines

mm/hugetlb.c: In function 'cmdline_parse_hugetlb_cma':
mm/hugetlb.c:5548:21: warning: variable 'val' set but not used [-Wunused-but- set-variable]
  unsigned long long val;
                     ^~~
Also, the comments for simple_strtoull() in lib/vsprintf.c said,

"This function is obsolete. Please use kstrtoull instead."

Hello Qian!

Thank you, you're absolutely right.
The following patch should solve both problems:

--

From d45741afdb9a760e48915c2d19e750f53019e19c Mon Sep 17 00:00:00 2001
From: Roman Gushchin <[email protected]>
Date: Thu, 12 Mar 2020 17:40:04 -0700
Subject: [PATCH] mm: cleanup cmdline_parse_hugetlb_cma()

Remove unused code.

---
 mm/hugetlb.c | 7 -------
 1 file changed, 7 deletions(-)
mm/hugetlb.c +0 -7
index 66bfc2bdc203..7a20cae7c77a 100644
@@ -5409,13 +5409,6 @@ static unsigned long hugetlb_cma_size __initdata;
 
 static int __init cmdline_parse_hugetlb_cma(char *p)
 {
-       unsigned long long val;
-       char *endptr;
-
-       if (!p)
-               return -EINVAL;
-
-       val = simple_strtoull(p, &endptr, 0);
        hugetlb_cma_size = memparse(p, &p);
        return 0;
 }
Signature · 1 line
2.24.1
maintainer
#3 lore raw
In reply to Roman Gushchin
Quoted 25 lines · On Wed, 11 Mar 2020 15:09:20 -0700 Roman Gushchin <[email protected]> wrote:

At large scale rebooting servers in order to allocate gigantic hugepages is quite expensive and complex. At the same time keeping some constant percentage of memory in reserved hugepages even if the workload isn't using it is a big waste: not all workloads can benefit from using 1 GB pages.

The following solution can solve the problem:
1) On boot time a dedicated cma area* is reserved. The size is passed as a kernel argument.
2) Run-time allocations of gigantic hugepages are performed using the cma allocator and the dedicated cma area

In this case gigantic hugepages can be allocated successfully with a high probability, however the memory isn't completely wasted if nobody is using 1GB hugepages: it can be used for pagecache, anon memory, THPs, etc.

* On a multi-node machine a per-node cma area is allocated on each node. Following gigantic hugetlb allocation are using the first available numa node if the mask isn't specified by a user.

Usage:
1) configure the kernel to allocate a cma area for hugetlb allocations:
pass hugetlb_cma=10G as a kernel argument

Quoted 8 lines

2) allocate hugetlb pages as usual, e.g.
echo 10 > /sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages

If the option isn't enabled or the allocation of the cma area failed, the current behavior of the system is preserved.

x86 and arm-64 are covered by this patch, other architectures can be trivially added later.

Lots of review input on v2, but then everyone went quiet ;)

Has everything been addressed?

#4 lore raw
Quoted 33 lines · Andrew Morton
Quoted 31 lines · On Wed, 11 Mar 2020 15:09:20 -0700 Roman Gushchin <[email protected]> wrote:

At large scale rebooting servers in order to allocate gigantic hugepages is quite expensive and complex. At the same time keeping some constant percentage of memory in reserved hugepages even if the workload isn't using it is a big waste: not all workloads can benefit from using 1 GB pages.

The following solution can solve the problem:
1) On boot time a dedicated cma area* is reserved. The size is passed as a kernel argument.
2) Run-time allocations of gigantic hugepages are performed using the cma allocator and the dedicated cma area

In this case gigantic hugepages can be allocated successfully with a high probability, however the memory isn't completely wasted if nobody is using 1GB hugepages: it can be used for pagecache, anon memory, THPs, etc.

* On a multi-node machine a per-node cma area is allocated on each node. Following gigantic hugetlb allocation are using the first available numa node if the mask isn't specified by a user.

Usage:
1) configure the kernel to allocate a cma area for hugetlb allocations:
pass hugetlb_cma=10G as a kernel argument

2) allocate hugetlb pages as usual, e.g.
echo 10 > /sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages

If the option isn't enabled or the allocation of the cma area failed, the current behavior of the system is preserved.

Quoted 6 lines
Quoted 2 lines

x86 and arm-64 are covered by this patch, other architectures can be trivially added later.

Lots of review input on v2, but then everyone went quiet ;)

Has everything been addressed?

I hope so. There is a nice cleanup from Aslan, which can be merged in or treated as a separate patch.

If someone else has any concerns, I'm happy to address them too.

Thanks!

#5 lore raw

Re: [PATCH v3] mm: hugetlb: optionally allocate gigantic hugepages using cma

Quoted 37 lines · Roman Gushchin
Quoted 36 lines · Andrew Morton
Quoted 33 lines · On Wed, 11 Mar 2020 15:09:20 -0700 Roman Gushchin <[email protected]> wrote:

At large scale rebooting servers in order to allocate gigantic hugepages is quite expensive and complex. At the same time keeping some constant percentage of memory in reserved hugepages even if the workload isn't using it is a big waste: not all workloads can benefit from using 1 GB pages.

The following solution can solve the problem:
1) On boot time a dedicated cma area* is reserved. The size is passed as a kernel argument.
2) Run-time allocations of gigantic hugepages are performed using the cma allocator and the dedicated cma area

In this case gigantic hugepages can be allocated successfully with a high probability, however the memory isn't completely wasted if nobody is using 1GB hugepages: it can be used for pagecache, anon memory, THPs, etc.

* On a multi-node machine a per-node cma area is allocated on each node. Following gigantic hugetlb allocation are using the first available numa node if the mask isn't specified by a user.

Usage:
1) configure the kernel to allocate a cma area for hugetlb allocations:
pass hugetlb_cma=10G as a kernel argument

2) allocate hugetlb pages as usual, e.g.
echo 10 > /sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages

If the option isn't enabled or the allocation of the cma area failed, the current behavior of the system is preserved.

x86 and arm-64 are covered by this patch, other architectures can be trivially added later.

Quoted 6 lines
Quoted 3 lines

Lots of review input on v2, but then everyone went quiet ;)

Has everything been addressed?

I hope so. There is a nice cleanup from Aslan, which can be merged in or treated as a separate patch.

With the follow up patche I didn't have any objections. I would prefer having hugetlb parts folded into the original patch to make the review easier though. Then I can have a look again. If those patches are going to be as they are now then no problem with me.

Signature · 2 lines
Michal Hocko
SUSE Labs
Acked-by on v3 1/1
#6 lore raw
In reply to Roman Gushchin
Quoted 43 lines · Roman Gushchin

At large scale rebooting servers in order to allocate gigantic hugepages is quite expensive and complex. At the same time keeping some constant percentage of memory in reserved hugepages even if the workload isn't using it is a big waste: not all workloads can benefit from using 1 GB pages.

The following solution can solve the problem:
1) On boot time a dedicated cma area* is reserved. The size is passed as a kernel argument.
2) Run-time allocations of gigantic hugepages are performed using the cma allocator and the dedicated cma area

In this case gigantic hugepages can be allocated successfully with a high probability, however the memory isn't completely wasted if nobody is using 1GB hugepages: it can be used for pagecache, anon memory, THPs, etc.

* On a multi-node machine a per-node cma area is allocated on each node. Following gigantic hugetlb allocation are using the first available numa node if the mask isn't specified by a user.

Usage:
1) configure the kernel to allocate a cma area for hugetlb allocations:
pass hugetlb_cma=10G as a kernel argument

2) allocate hugetlb pages as usual, e.g.
echo 10 > /sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages

If the option isn't enabled or the allocation of the cma area failed, the current behavior of the system is preserved.

x86 and arm-64 are covered by this patch, other architectures can be trivially added later.

v3:
- added fallback to the existing allocation mechanism
- added min/max checks
- switched to MiB in debug output
- removed percentage option
- added arch-specific order argument to determine an alignment
- added arm support
- fixed the !CONFIG_HUGETLBFS build

Quoted 6 lines

Thanks to Michal, Mike, Andreas and Rik for ideas and suggestions!

v2:
-fixed !CONFIG_CMA build, suggested by Andrew Morton

It is a bit difficult to keep track of all the followup patches. One small issue below.

Quoted 39 lines
---
 .../admin-guide/kernel-parameters.txt         |   7 ++
 arch/arm64/mm/init.c                          |   6 +
 arch/x86/kernel/setup.c                       |   4 +
 include/linux/hugetlb.h                       |   8 ++
 mm/hugetlb.c                                  | 116 ++++++++++++++++++
 5 files changed, 141 insertions(+)
Documentation/admin-guide/kernel-parameters.txt +7 -0
index 0c9894247015..9eb0df40643d 100644
@@ -1452,6 +1452,13 @@
 	hpet_mmap=	[X86, HPET_MMAP] Allow userspace to mmap HPET
 			registers.  Default set by CONFIG_HPET_MMAP_DEFAULT.
 
+	hugetlb_cma=	[x86-64] The size of a cma area used for allocation
+			of gigantic hugepages.
+			Format: nn[KMGTPE]
+
+			If enabled, boot-time allocation of gigantic hugepages
+			is skipped.
+
 	hugepages=	[HW,X86-32,IA-64] HugeTLB pages to allocate at boot.
 	hugepagesz=	[HW,IA-64,PPC,X86-64] The size of the HugeTLB pages.
 			On x86-64 and powerpc, this option can be specified
arch/arm64/mm/init.c +1 -0
index b65dffdfb201..e42727e3568e 100644
@@ -29,6 +29,7 @@
 #include <linux/mm.h>
 #include <linux/kexec.h>
 #include <linux/crash_dump.h>
+#include <linux/hugetlb.h>
 
 #include <asm/boot.h>
 #include <asm/fixmap.h>

arch/arm64/mm/init.c void __init arm64_memblock_init(void) · line 466

Quoted 9 lines
arch/arm64/mm/init.c +5 -0
@@ -457,6 +458,11 @@ void __init arm64_memblock_init(void)
 	high_memory = __va(memblock_end_of_DRAM() - 1) + 1;
 
 	dma_contiguous_reserve(arm64_dma32_phys_limit);
+
+#ifdef CONFIG_ARM64_4K_PAGES
+	hugetlb_cma_reserve(PUD_SHIFT - PAGE_SHIFT);
+#endif
+
 }

The documentation is already 'out of date' as you added support for arm64. Not a huge deal as documentation rarely keeps up with code, but we should at least be correct here.

I have a patch series in progress which cleans up existing hugetlb command line processing.
https://lore.kernel.org/linux-mm/[email protected]/

No need to make any changes here, but assuming this support goes in first I would make the following changes as part of my series:
- Don't list architectures in Documentation. Just say support is arch dependent.
- Introduce some mechanism to print an error if hugetlb_cma is specified on the command line, but not supported by architecture. IIUC, no message is printed today. IMO, this only becomes important if the documentation does not list supported architectures.

Not insisting that documentation be updated to include arm64.

Signature · 1 line
Mike Kravetz
#7 lore raw
Quoted 46 lines · Mike Kravetz
Quoted 45 lines · Roman Gushchin

At large scale rebooting servers in order to allocate gigantic hugepages is quite expensive and complex. At the same time keeping some constant percentage of memory in reserved hugepages even if the workload isn't using it is a big waste: not all workloads can benefit from using 1 GB pages.

The following solution can solve the problem:
1) On boot time a dedicated cma area* is reserved. The size is passed as a kernel argument.
2) Run-time allocations of gigantic hugepages are performed using the cma allocator and the dedicated cma area

In this case gigantic hugepages can be allocated successfully with a high probability, however the memory isn't completely wasted if nobody is using 1GB hugepages: it can be used for pagecache, anon memory, THPs, etc.

* On a multi-node machine a per-node cma area is allocated on each node. Following gigantic hugetlb allocation are using the first available numa node if the mask isn't specified by a user.

Usage:
1) configure the kernel to allocate a cma area for hugetlb allocations:
pass hugetlb_cma=10G as a kernel argument

2) allocate hugetlb pages as usual, e.g.
echo 10 > /sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages

If the option isn't enabled or the allocation of the cma area failed, the current behavior of the system is preserved.

x86 and arm-64 are covered by this patch, other architectures can be trivially added later.

v3:
- added fallback to the existing allocation mechanism
- added min/max checks
- switched to MiB in debug output
- removed percentage option
- added arch-specific order argument to determine an alignment
- added arm support
- fixed the !CONFIG_HUGETLBFS build

Thanks to Michal, Mike, Andreas and Rik for ideas and suggestions!

Quoted 7 lines
Quoted 4 lines

v2:
-fixed !CONFIG_CMA build, suggested by Andrew Morton

It is a bit difficult to keep track of all the followup patches. One small issue below.

I agree. There was a dozen of cleanups and fixes from several other people, so it's a bit messy now.
I'll merge it all together (including documentation fixes proposed by you) and resend, as soon as I'll figure out the hugetlb/cma locking issue.

Quoted 67 lines
Quoted 40 lines
---
 .../admin-guide/kernel-parameters.txt         |   7 ++
 arch/arm64/mm/init.c                          |   6 +
 arch/x86/kernel/setup.c                       |   4 +
 include/linux/hugetlb.h                       |   8 ++
 mm/hugetlb.c                                  | 116 ++++++++++++++++++
 5 files changed, 141 insertions(+)
Documentation/admin-guide/kernel-parameters.txt +7 -0
index 0c9894247015..9eb0df40643d 100644
@@ -1452,6 +1452,13 @@
 	hpet_mmap=	[X86, HPET_MMAP] Allow userspace to mmap HPET
 			registers.  Default set by CONFIG_HPET_MMAP_DEFAULT.
 
+	hugetlb_cma=	[x86-64] The size of a cma area used for allocation
+			of gigantic hugepages.
+			Format: nn[KMGTPE]
+
+			If enabled, boot-time allocation of gigantic hugepages
+			is skipped.
+
 	hugepages=	[HW,X86-32,IA-64] HugeTLB pages to allocate at boot.
 	hugepagesz=	[HW,IA-64,PPC,X86-64] The size of the HugeTLB pages.
 			On x86-64 and powerpc, this option can be specified
arch/arm64/mm/init.c +1 -0
index b65dffdfb201..e42727e3568e 100644
@@ -29,6 +29,7 @@
 #include <linux/mm.h>
 #include <linux/kexec.h>
 #include <linux/crash_dump.h>
+#include <linux/hugetlb.h>
 
 #include <asm/boot.h>
 #include <asm/fixmap.h>

arch/arm64/mm/init.c void __init arm64_memblock_init(void) · line 466

Quoted 9 lines
arch/arm64/mm/init.c +5 -0
@@ -457,6 +458,11 @@ void __init arm64_memblock_init(void)
 	high_memory = __va(memblock_end_of_DRAM() - 1) + 1;
 
 	dma_contiguous_reserve(arm64_dma32_phys_limit);
+
+#ifdef CONFIG_ARM64_4K_PAGES
+	hugetlb_cma_reserve(PUD_SHIFT - PAGE_SHIFT);
+#endif
+
 }

The documentation is already 'out of date' as you added support for arm64. Not a huge deal as documentation rarely keeps up with code, but we should at least be correct here.

I have a patch series in progress which cleans up existing hugetlb command line processing.
https://lore.kernel.org/linux-mm/[email protected]/

No need to make any changes here, but assuming this support goes in first I would make the following changes as part of my series:
- Don't list architectures in Documentation. Just say support is arch dependent.
- Introduce some mechanism to print an error if hugetlb_cma is specified on the command line, but not supported by architecture. IIUC, no message is printed today. IMO, this only becomes important if the documentation does not list supported architectures.

Quoted 2 lines

Not insisting that documentation be updated to include arm64.

Thank you!

Roman