mm: hugetlb: optionally allocate gigantic hugepages using cma 65;5803;1c Commit 944d9fec8d7a ("hugetlb: add support for gigantic page allocation at runtime") has added the run-time allocation of gigantic pages. However it actually works only at early stages of the system loading, when the majority of memory is free. After some time the memory gets fragmented by non-movable pages, so the chances to find a contiguous 1 GB block are getting close to zero. Even dropping caches manually doesn't help a lot.
Subsystems (5)
PATCH v3 1/1 mm: hugetlb: optionally allocate gigantic hugepages using cma 65;5803;1c Commit 944d9fec8d7a ("hugetlb: add support for gigantic page allocation at runtime") has added the run-time allocation of gigantic pages. However it actually works only at early stages of the system loading, when the majority of memory is free. After some time the memory gets fragmented by non-movable pages, so the chances to find a contiguous 1 GB block are getting close to zero. Even dropping caches manually doesn't help a lot.
At large scale rebooting servers in order to allocate gigantic hugepages is quite expensive and complex. At the same time keeping some constant percentage of memory in reserved hugepages even if the workload isn't using it is a big waste: not all workloads can benefit from using 1 GB pages.
The following solution can solve the problem:
1) On boot time a dedicated cma area* is reserved. The size is passed as a kernel argument.
2) Run-time allocations of gigantic hugepages are performed using the cma allocator and the dedicated cma area
In this case gigantic hugepages can be allocated successfully with a high probability, however the memory isn't completely wasted if nobody is using 1GB hugepages: it can be used for pagecache, anon memory, THPs, etc.
* On a multi-node machine a per-node cma area is allocated on each node. Following gigantic hugetlb allocation are using the first available numa node if the mask isn't specified by a user.
Usage:
1) configure the kernel to allocate a cma area for hugetlb allocations:
pass hugetlb_cma=10G as a kernel argument
2) allocate hugetlb pages as usual, e.g.
echo 10 > /sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages
If the option isn't enabled or the allocation of the cma area failed, the current behavior of the system is preserved.
x86 and arm-64 are covered by this patch, other architectures can be trivially added later.
v3:
- added fallback to the existing allocation mechanism
- added min/max checks
- switched to MiB in debug output
- removed percentage option
- added arch-specific order argument to determine an alignment
- added arm support
- fixed the !CONFIG_HUGETLBFS build
Thanks to Michal, Mike, Andreas and Rik for ideas and suggestions!
v2:
-fixed !CONFIG_CMA build, suggested by Andrew Morton
--- .../admin-guide/kernel-parameters.txt | 7 ++ arch/arm64/mm/init.c | 6 + arch/x86/kernel/setup.c | 4 + include/linux/hugetlb.h | 8 ++ mm/hugetlb.c | 116 ++++++++++++++++++ 5 files changed, 141 insertions(+)
Documentation/admin-guide/kernel-parameters.txt +7 -0
@@ -1452,6 +1452,13 @@ hpet_mmap= [X86, HPET_MMAP] Allow userspace to mmap HPET registers. Default set by CONFIG_HPET_MMAP_DEFAULT. + hugetlb_cma= [x86-64] The size of a cma area used for allocation + of gigantic hugepages. + Format: nn[KMGTPE] + + If enabled, boot-time allocation of gigantic hugepages + is skipped. + hugepages= [HW,X86-32,IA-64] HugeTLB pages to allocate at boot. hugepagesz= [HW,IA-64,PPC,X86-64] The size of the HugeTLB pages. On x86-64 and powerpc, this option can be specified
arch/arm64/mm/init.c +6 -0
@@ -29,6 +29,7 @@#include <linux/mm.h> #include <linux/kexec.h> #include <linux/crash_dump.h> +#include <linux/hugetlb.h> #include <asm/boot.h> #include <asm/fixmap.h>
@@ -457,6 +458,11 @@ void __init arm64_memblock_init(void) high_memory =__va (memblock_end_of_DRAM () -1 ) +1 ;dma_contiguous_reserve (arm64_dma32_phys_limit); + +#ifdef CONFIG_ARM64_4K_PAGES +hugetlb_cma_reserve (PUD_SHIFT -PAGE_SHIFT ); +#endif + }void __init bootmem_init (void )
arch/x86/kernel/setup.c +4 -0
@@ -16,6 +16,7 @@#include <linux/pci.h> #include <linux/root_dev.h> #include <linux/sfi.h> +#include <linux/hugetlb.h> #include <linux/tboot.h> #include <linux/usb/xhci-dbgp.h>
@@ -1158,6 +1159,9 @@ void __init setup_arch(char **cmdline_p)initmem_init ();dma_contiguous_reserve (max_pfn_mapped <<PAGE_SHIFT ); +if (boot_cpu_has (X86_FEATURE_GBPAGES )) +hugetlb_cma_reserve (PUD_SHIFT -PAGE_SHIFT ); +/* * Reserve memory for crash kernel after SRAT is parsed so that it * won't consume hotpluggable memory.
include/linux/hugetlb.h +8 -0
@@ -890,4 +890,12 @@ static inline spinlock_t *huge_pte_lock(struct hstate *h, return ptl; } +#if defined(CONFIG_HUGETLB_PAGE ) && defined(CONFIG_CMA ) +extern void __init hugetlb_cma_reserve (int order); +#else +static inline __init void hugetlb_cma_reserve (int order) +{ +} +#endif +#endif /* _LINUX_HUGETLB_H */
mm/hugetlb.c +116 -0
149 lines not on this page · raw message
Signature · 1 line
2.24.1
Re: [PATCH v3] mm: hugetlb: optionally allocate gigantic hugepages using cma
Quoted 6 lines · Roman Gushchin
mm/hugetlb.c +6
-0+#ifdef CONFIG_CMA +static unsigned long hugetlb_cma_size__initdata ; + +static int __init cmdline_parse_hugetlb_cma (char *p) +{ +unsigned long long val;
mm/hugetlb.c void move_hugetlb_state(struct page *oldpage, struct page *newpage, int reason) · line 5422
Quoted 10 lines
mm/hugetlb.c +10
-0+char *endptr; + +if (!p) +return -EINVAL ; + + val =simple_strtoull (p, &endptr,0 ); + hugetlb_cma_size =memparse (p, &p); +return 0 ; +} +
Here will generate a compilation warning,
mm/hugetlb.c: In function 'cmdline_parse_hugetlb_cma':mm/hugetlb.c:5548:21: warning: variable 'val' set but not used [-Wunused-but- set-variable]
unsigned long long val;
^~~
Also, the comments for simple_strtoull() in lib/vsprintf.c said,
"This function is obsolete. Please use kstrtoull instead."
Re: [PATCH v3] mm: hugetlb: optionally allocate gigantic hugepages using cma
Quoted 20 lines · Qian Cai
Quoted 6 lines · Roman Gushchin
(quoted diff) +6
-0+#ifdef CONFIG_CMA +static unsigned long hugetlb_cma_size __initdata; + +static int __init cmdline_parse_hugetlb_cma(char *p) +{ + unsigned long long val;Review
Quoted 10 lines
(quoted diff) +10
-0+ char *endptr; + + if (!p) + return -EINVAL; + + val = simple_strtoull(p, &endptr, 0); + hugetlb_cma_size = memparse(p, &p); + return 0; +} +Here will generate a compilation warning,
Quoted 9 lines
mm/hugetlb.c: In function 'cmdline_parse_hugetlb_cma':mm/hugetlb.c:5548:21: warning: variable 'val' set but not used [-Wunused-but- set-variable]
unsigned long long val;
^~~
Also, the comments forsimple_strtoull()inlib/vsprintf.csaid,"This function is obsolete. Please use kstrtoull instead."
Hello Qian!
Thank you, you're absolutely right.
The following patch should solve both problems:
--
From d45741afdb9a760e48915c2d19e750f53019e19c Mon Sep 17 00:00:00 2001
From: Roman Gushchin <[email protected]>
Date: Thu, 12 Mar 2020 17:40:04 -0700
Subject: [PATCH] mm: cleanup cmdline_parse_hugetlb_cma()
Remove unused code.
--- mm/hugetlb.c | 7 ------- 1 file changed, 7 deletions(-)
mm/hugetlb.c +0 -7
@@ -5409,13 +5409,6 @@ static unsigned long hugetlb_cma_size __initdata;static int __init cmdline_parse_hugetlb_cma (char *p) { -unsigned long long val; -char *endptr; - -if (!p) -return -EINVAL ; - - val =simple_strtoull (p, &endptr,0 ); hugetlb_cma_size =memparse (p, &p);return 0 ; }
Signature · 1 line
2.24.1
Andrew Morton
maintainerQuoted 25 lines · On Wed, 11 Mar 2020 15:09:20 -0700 Roman Gushchin <[email protected]> wrote:
At large scale rebooting servers in order to allocate gigantic hugepages is quite expensive and complex. At the same time keeping some constant percentage of memory in reserved hugepages even if the workload isn't using it is a big waste: not all workloads can benefit from using 1 GB pages.
The following solution can solve the problem:
1) On boot time a dedicated cma area* is reserved. The size is passed as a kernel argument.
2) Run-time allocations of gigantic hugepages are performed using the cma allocator and the dedicated cma areaIn this case gigantic hugepages can be allocated successfully with a high probability, however the memory isn't completely wasted if nobody is using 1GB hugepages: it can be used for pagecache, anon memory, THPs, etc.
* On a multi-node machine a per-node cma area is allocated on each node. Following gigantic hugetlb allocation are using the first available numa node if the mask isn't specified by a user.
Usage:
1) configure the kernel to allocate a cma area for hugetlb allocations:
pass hugetlb_cma=10G as a kernel argument
Quoted 8 lines
2) allocate hugetlb pages as usual, e.g.
echo 10 >/sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepagesIf the option isn't enabled or the allocation of the cma area failed, the current behavior of the system is preserved.
x86 and arm-64 are covered by this patch, other architectures can be trivially added later.
Lots of review input on v2, but then everyone went quiet ;)
Has everything been addressed?
Quoted 33 lines · Andrew Morton
Quoted 31 lines · On Wed, 11 Mar 2020 15:09:20 -0700 Roman Gushchin <[email protected]> wrote:
At large scale rebooting servers in order to allocate gigantic hugepages is quite expensive and complex. At the same time keeping some constant percentage of memory in reserved hugepages even if the workload isn't using it is a big waste: not all workloads can benefit from using 1 GB pages.
The following solution can solve the problem:
1) On boot time a dedicated cma area* is reserved. The size is passed as a kernel argument.
2) Run-time allocations of gigantic hugepages are performed using the cma allocator and the dedicated cma areaIn this case gigantic hugepages can be allocated successfully with a high probability, however the memory isn't completely wasted if nobody is using 1GB hugepages: it can be used for pagecache, anon memory, THPs, etc.
* On a multi-node machine a per-node cma area is allocated on each node. Following gigantic hugetlb allocation are using the first available numa node if the mask isn't specified by a user.
Usage:
1) configure the kernel to allocate a cma area for hugetlb allocations:
pass hugetlb_cma=10G as a kernel argument2) allocate hugetlb pages as usual, e.g.
echo 10 >/sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepagesIf the option isn't enabled or the allocation of the cma area failed, the current behavior of the system is preserved.
Quoted 6 lines
Quoted 2 lines
x86 and arm-64 are covered by this patch, other architectures can be trivially added later.
Lots of review input on v2, but then everyone went quiet ;)
Has everything been addressed?
I hope so. There is a nice cleanup from Aslan, which can be merged in or treated as a separate patch.
If someone else has any concerns, I'm happy to address them too.
Thanks!
Re: [PATCH v3] mm: hugetlb: optionally allocate gigantic hugepages using cma
Quoted 37 lines · Roman Gushchin
Quoted 36 lines · Andrew Morton
Quoted 33 lines · On Wed, 11 Mar 2020 15:09:20 -0700 Roman Gushchin <[email protected]> wrote:
At large scale rebooting servers in order to allocate gigantic hugepages is quite expensive and complex. At the same time keeping some constant percentage of memory in reserved hugepages even if the workload isn't using it is a big waste: not all workloads can benefit from using 1 GB pages.
The following solution can solve the problem:
1) On boot time a dedicated cma area* is reserved. The size is passed as a kernel argument.
2) Run-time allocations of gigantic hugepages are performed using the cma allocator and the dedicated cma areaIn this case gigantic hugepages can be allocated successfully with a high probability, however the memory isn't completely wasted if nobody is using 1GB hugepages: it can be used for pagecache, anon memory, THPs, etc.
* On a multi-node machine a per-node cma area is allocated on each node. Following gigantic hugetlb allocation are using the first available numa node if the mask isn't specified by a user.
Usage:
1) configure the kernel to allocate a cma area for hugetlb allocations:
pass hugetlb_cma=10G as a kernel argument2) allocate hugetlb pages as usual, e.g.
echo 10 >/sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepagesIf the option isn't enabled or the allocation of the cma area failed, the current behavior of the system is preserved.
x86 and arm-64 are covered by this patch, other architectures can be trivially added later.
Quoted 6 lines
Quoted 3 lines
Lots of review input on v2, but then everyone went quiet ;)
Has everything been addressed?
I hope so. There is a nice cleanup from Aslan, which can be merged in or treated as a separate patch.
With the follow up patche I didn't have any objections. I would prefer having hugetlb parts folded into the original patch to make the review easier though. Then I can have a look again. If those patches are going to be as they are now then no problem with me.
Signature · 2 lines
Michal Hocko SUSE Labs
Quoted 43 lines · Roman Gushchin
At large scale rebooting servers in order to allocate gigantic hugepages is quite expensive and complex. At the same time keeping some constant percentage of memory in reserved hugepages even if the workload isn't using it is a big waste: not all workloads can benefit from using 1 GB pages.
The following solution can solve the problem:
1) On boot time a dedicated cma area* is reserved. The size is passed as a kernel argument.
2) Run-time allocations of gigantic hugepages are performed using the cma allocator and the dedicated cma areaIn this case gigantic hugepages can be allocated successfully with a high probability, however the memory isn't completely wasted if nobody is using 1GB hugepages: it can be used for pagecache, anon memory, THPs, etc.
* On a multi-node machine a per-node cma area is allocated on each node. Following gigantic hugetlb allocation are using the first available numa node if the mask isn't specified by a user.
Usage:
1) configure the kernel to allocate a cma area for hugetlb allocations:
pass hugetlb_cma=10G as a kernel argument2) allocate hugetlb pages as usual, e.g.
echo 10 >/sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepagesIf the option isn't enabled or the allocation of the cma area failed, the current behavior of the system is preserved.
x86 and arm-64 are covered by this patch, other architectures can be trivially added later.
v3:
- added fallback to the existing allocation mechanism
- added min/max checks
- switched to MiB in debug output
- removed percentage option
- added arch-specific order argument to determine an alignment
- added arm support
- fixed the !CONFIG_HUGETLBFS build
Quoted 6 lines
Thanks to Michal, Mike, Andreas and Rik for ideas and suggestions!
v2:
-fixed !CONFIG_CMA build, suggested by Andrew MortonSigned-off-by Roman Gushchin <[email protected]>
It is a bit difficult to keep track of all the followup patches. One small issue below.
Quoted 39 lines
--- .../admin-guide/kernel-parameters.txt | 7 ++ arch/arm64/mm/init.c | 6 + arch/x86/kernel/setup.c | 4 + include/linux/hugetlb.h | 8 ++ mm/hugetlb.c | 116 ++++++++++++++++++ 5 files changed, 141 insertions(+)Documentation/admin-guide/kernel-parameters.txt +7
-0@@ -1452,6 +1452,13 @@ hpet_mmap= [X86, HPET_MMAP] Allow userspace to mmap HPET registers. Default set by CONFIG_HPET_MMAP_DEFAULT. + hugetlb_cma= [x86-64] The size of a cma area used for allocation + of gigantic hugepages. + Format: nn[KMGTPE] + + If enabled, boot-time allocation of gigantic hugepages + is skipped. + hugepages= [HW,X86-32,IA-64] HugeTLB pages to allocate at boot. hugepagesz= [HW,IA-64,PPC,X86-64] The size of the HugeTLB pages. On x86-64 and powerpc, this option can be specifiedarch/arm64/mm/init.c +1
-0@@ -29,6 +29,7 @@#include <linux/mm.h> #include <linux/kexec.h> #include <linux/crash_dump.h> +#include <linux/hugetlb.h> #include <asm/boot.h> #include <asm/fixmap.h>
arch/arm64/mm/init.c void __init arm64_memblock_init(void) · line 466
Quoted 9 lines
arch/arm64/mm/init.c +5
-0@@ -457,6 +458,11 @@ void __init arm64_memblock_init(void) high_memory =__va (memblock_end_of_DRAM () -1 ) +1 ;dma_contiguous_reserve (arm64_dma32_phys_limit); + +#ifdef CONFIG_ARM64_4K_PAGES +hugetlb_cma_reserve (PUD_SHIFT -PAGE_SHIFT ); +#endif + }
The documentation is already 'out of date' as you added support for arm64. Not a huge deal as documentation rarely keeps up with code, but we should at least be correct here.
I have a patch series in progress which cleans up existing hugetlb command line processing.
https://lore.kernel.org/linux-mm/[email protected]/
No need to make any changes here, but assuming this support goes in first I would make the following changes as part of my series:
- Don't list architectures in Documentation. Just say support is arch dependent.
- Introduce some mechanism to print an error if hugetlb_cma is specified on the command line, but not supported by architecture. IIUC, no message is printed today. IMO, this only becomes important if the documentation does not list supported architectures.
Not insisting that documentation be updated to include arm64.
Signature · 1 line
Mike Kravetz
Quoted 46 lines · Mike Kravetz
Quoted 45 lines · Roman Gushchin
At large scale rebooting servers in order to allocate gigantic hugepages is quite expensive and complex. At the same time keeping some constant percentage of memory in reserved hugepages even if the workload isn't using it is a big waste: not all workloads can benefit from using 1 GB pages.
The following solution can solve the problem:
1) On boot time a dedicated cma area* is reserved. The size is passed as a kernel argument.
2) Run-time allocations of gigantic hugepages are performed using the cma allocator and the dedicated cma areaIn this case gigantic hugepages can be allocated successfully with a high probability, however the memory isn't completely wasted if nobody is using 1GB hugepages: it can be used for pagecache, anon memory, THPs, etc.
* On a multi-node machine a per-node cma area is allocated on each node. Following gigantic hugetlb allocation are using the first available numa node if the mask isn't specified by a user.
Usage:
1) configure the kernel to allocate a cma area for hugetlb allocations:
pass hugetlb_cma=10G as a kernel argument2) allocate hugetlb pages as usual, e.g.
echo 10 >/sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepagesIf the option isn't enabled or the allocation of the cma area failed, the current behavior of the system is preserved.
x86 and arm-64 are covered by this patch, other architectures can be trivially added later.
v3:
- added fallback to the existing allocation mechanism
- added min/max checks
- switched to MiB in debug output
- removed percentage option
- added arch-specific order argument to determine an alignment
- added arm support
- fixed the !CONFIG_HUGETLBFS buildThanks to Michal, Mike, Andreas and Rik for ideas and suggestions!
Quoted 7 lines
Quoted 4 lines
v2:
-fixed !CONFIG_CMA build, suggested by Andrew MortonSigned-off-by Roman Gushchin <[email protected]>It is a bit difficult to keep track of all the followup patches. One small issue below.
I agree. There was a dozen of cleanups and fixes from several other people, so it's a bit messy now.
I'll merge it all together (including documentation fixes proposed by you) and resend, as soon as I'll figure out the hugetlb/cma locking issue.
Quoted 67 lines
Quoted 40 lines
--- .../admin-guide/kernel-parameters.txt | 7 ++ arch/arm64/mm/init.c | 6 + arch/x86/kernel/setup.c | 4 + include/linux/hugetlb.h | 8 ++ mm/hugetlb.c | 116 ++++++++++++++++++ 5 files changed, 141 insertions(+)Documentation/admin-guide/kernel-parameters.txt +7
-0@@ -1452,6 +1452,13 @@ hpet_mmap= [X86, HPET_MMAP] Allow userspace to mmap HPET registers. Default set by CONFIG_HPET_MMAP_DEFAULT. + hugetlb_cma= [x86-64] The size of a cma area used for allocation + of gigantic hugepages. + Format: nn[KMGTPE] + + If enabled, boot-time allocation of gigantic hugepages + is skipped. + hugepages= [HW,X86-32,IA-64] HugeTLB pages to allocate at boot. hugepagesz= [HW,IA-64,PPC,X86-64] The size of the HugeTLB pages. On x86-64 and powerpc, this option can be specifiedarch/arm64/mm/init.c +1
-0@@ -29,6 +29,7 @@#include <linux/mm.h> #include <linux/kexec.h> #include <linux/crash_dump.h> +#include <linux/hugetlb.h> #include <asm/boot.h> #include <asm/fixmap.h> arch/arm64/mm/init.c void __init arm64_memblock_init(void) · line 466
Quoted 9 lines
arch/arm64/mm/init.c +5
-0@@ -457,6 +458,11 @@ void __init arm64_memblock_init(void) high_memory =__va (memblock_end_of_DRAM () -1 ) +1 ;dma_contiguous_reserve (arm64_dma32_phys_limit); + +#ifdef CONFIG_ARM64_4K_PAGES +hugetlb_cma_reserve (PUD_SHIFT -PAGE_SHIFT ); +#endif + }The documentation is already 'out of date' as you added support for arm64. Not a huge deal as documentation rarely keeps up with code, but we should at least be correct here.
I have a patch series in progress which cleans up existing hugetlb command line processing.
https://lore.kernel.org/linux-mm/[email protected]/No need to make any changes here, but assuming this support goes in first I would make the following changes as part of my series:
- Don't list architectures in Documentation. Just say support is arch dependent.
- Introduce some mechanism to print an error ifhugetlb_cmais specified on the command line, but not supported by architecture. IIUC, no message is printed today. IMO, this only becomes important if the documentation does not list supported architectures.
Quoted 2 lines
Not insisting that documentation be updated to include arm64.
Acked-by Mike Kravetz <[email protected]>
Thank you!
Roman