mm: mmap: no need to call khugepaged_enter_vma() for stack
Subsystems (2)
PATCH 1/2 mm: mmap: no need to call khugepaged_enter_vma() for stack
From: Yang Shi <[email protected]>
We avoid allocating THP for temporary stack, even tnough khugepaged_enter_vma() is called for stack VMAs, it actualy returns false. So no need to call it in the first place at all.
--- mm/mmap.c | 2 -- 1 file changed, 2 deletions(-)
mm/mmap.c +0 -2
@@ -2046,7 +2046,6 @@ static int expand_upwards(struct vm_area_struct *vma, unsigned long address) } }anon_vma_unlock_write (vma->anon_vma); -khugepaged_enter_vma (vma, vma->vm_flags);mas_destroy (&mas);validate_mm (mm);return error;
@@ -2140,7 +2139,6 @@ int expand_downwards(struct vm_area_struct *vma, unsigned long address) } }anon_vma_unlock_write (vma->anon_vma); -khugepaged_enter_vma (vma, vma->vm_flags);mas_destroy (&mas);validate_mm (mm);return error;
Signature · 1 line
2.41.0
PATCH 2/2 mm: mmap: map MAP_STACK to VM_NOHUGEPAGE
From: Yang Shi <[email protected]>
The commit efa7df3e3bb5 ("mm: align larger anonymous mappings on THP boundaries") incured regression for stress-ng pthread benchmark [1]. It is because THP get allocated to pthread's stack area much more possible than before. Pthread's stack area is allocated by mmap without VM_GROWSDOWN or VM_GROWSUP flag, so kernel can't tell whether it is a stack area or not.
The MAP_STACK flag is used to mark the stack area, but it is a no-op on Linux. Mapping MAP_STACK to VM_NOHUGEPAGE to prevent from allocating THP for such stack area.
With this change the stack area looks like:
fffd18e10000-fffd19610000 rw-p 00000000 00:00 0
Size: 8192 kB KernelPageSize: 4 kB MMUPageSize: 4 kB Rss: 12 kB Pss: 12 kB Pss_Dirty: 12 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 12 kB Referenced: 12 kB Anonymous: 12 kB KSM: 0 kB LazyFree: 0 kB AnonHugePages: 0 kB ShmemPmdMapped: 0 kB FilePmdMapped: 0 kB Shared_Hugetlb: 0 kB Private_Hugetlb: 0 kB Swap: 0 kB SwapPss: 0 kB Locked: 0 kB THPeligible: 0 VmFlags: rd wr mr mw me ac nh
The "nh" flag is set.
[1] https://lore.kernel.org/linux-mm/[email protected]/
--- include/linux/mman.h | 1 + 1 file changed, 1 insertion(+)
include/linux/mman.h +1 -0
@@ -156,6 +156,7 @@ calc_vm_flag_bits(unsigned long flags)return _calc_vm_trans (flags,MAP_GROWSDOWN ,VM_GROWSDOWN ) |_calc_vm_trans (flags,MAP_LOCKED ,VM_LOCKED ) |_calc_vm_trans (flags,MAP_SYNC ,VM_SYNC ) | +_calc_vm_trans (flags,MAP_STACK ,VM_NOHUGEPAGE ) |arch_calc_vm_flag_bits (flags); }
Signature · 1 line
2.41.0
Re: [PATCH 2/2] mm: mmap: map MAP_STACK to VM_NOHUGEPAGE
Quoted 44 lines · Yang Shi
From: Yang Shi <[email protected]>
The commit
efa7df3e3bb5("mm: align larger anonymous mappings on THP boundaries") incured regression for stress-ng pthread benchmark [1]. It is because THP get allocated to pthread's stack area much more possible than before. Pthread's stack area is allocated by mmap withoutVM_GROWSDOWNorVM_GROWSUPflag, so kernel can't tell whether it is a stack area or not.The
MAP_STACKflag is used to mark the stack area, but it is a no-op on Linux. MappingMAP_STACKtoVM_NOHUGEPAGEto prevent from allocating THP for such stack area.With this change the stack area looks like:
fffd18e10000-fffd19610000 rw-p 00000000 00:00 0
Size: 8192 kB KernelPageSize: 4 kB MMUPageSize: 4 kB Rss: 12 kB Pss: 12 kB Pss_Dirty: 12 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 12 kB Referenced: 12 kB Anonymous: 12 kB KSM: 0 kB LazyFree: 0 kB AnonHugePages: 0 kB ShmemPmdMapped: 0 kB FilePmdMapped: 0 kB Shared_Hugetlb: 0 kB Private_Hugetlb: 0 kB Swap: 0 kB SwapPss: 0 kB Locked: 0 kB THPeligible: 0 VmFlags: rd wr mr mw me ac nhThe "nh" flag is set.
Quoted 8 lines
Reported-by kernel test robot <[email protected]>Tested-by Oliver Sang <[email protected]>Cc Yin Fengwei <[email protected]>Cc Rik van Riel <[email protected]>Cc Matthew Wilcox <[email protected]>Cc Christopher Lameter <[email protected]>Cc Huang, Ying <[email protected]>Signed-off-by Yang Shi <[email protected]>
Quoted 16 lines
--- include/linux/mman.h | 1 + 1 file changed, 1 insertion(+)include/linux/mman.h +1
-0@@ -156,6 +156,7 @@ calc_vm_flag_bits(unsigned long flags)return _calc_vm_trans (flags,MAP_GROWSDOWN ,VM_GROWSDOWN ) |_calc_vm_trans (flags,MAP_LOCKED ,VM_LOCKED ) |_calc_vm_trans (flags,MAP_SYNC ,VM_SYNC ) | +_calc_vm_trans (flags,MAP_STACK ,VM_NOHUGEPAGE ) |arch_calc_vm_flag_bits (flags); }
Re: [PATCH 2/2] mm: mmap: map MAP_STACK to VM_NOHUGEPAGE
Thanks Yang,
Should this be marked for stable? Given how easily it is for pthreads to allocate hugepages w/o this change, it can easily cause memory bloat on larger systems and/or users with high thread counts. I don't think that will be welcomed, and seems odd that just 6.7 should suffer this.
Thanks,
Zach
Quoted 76 lines · Yin Fengwei
Quoted 44 lines · Yang Shi
From: Yang Shi <[email protected]>
The commit
efa7df3e3bb5("mm: align larger anonymous mappings on THP boundaries") incured regression for stress-ng pthread benchmark [1]. It is because THP get allocated to pthread's stack area much more possible than before. Pthread's stack area is allocated by mmap withoutVM_GROWSDOWNorVM_GROWSUPflag, so kernel can't tell whether it is a stack area or not.The
MAP_STACKflag is used to mark the stack area, but it is a no-op on Linux. MappingMAP_STACKtoVM_NOHUGEPAGEto prevent from allocating THP for such stack area.With this change the stack area looks like:
fffd18e10000-fffd19610000 rw-p 00000000 00:00 0
Size: 8192 kB KernelPageSize: 4 kB MMUPageSize: 4 kB Rss: 12 kB Pss: 12 kB Pss_Dirty: 12 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 12 kB Referenced: 12 kB Anonymous: 12 kB KSM: 0 kB LazyFree: 0 kB AnonHugePages: 0 kB ShmemPmdMapped: 0 kB FilePmdMapped: 0 kB Shared_Hugetlb: 0 kB Private_Hugetlb: 0 kB Swap: 0 kB SwapPss: 0 kB Locked: 0 kB THPeligible: 0 VmFlags: rd wr mr mw me ac nhThe "nh" flag is set.
Quoted 8 lines
Reported-by kernel test robot <[email protected]>Tested-by Oliver Sang <[email protected]>Cc Yin Fengwei <[email protected]>Cc Rik van Riel <[email protected]>Cc Matthew Wilcox <[email protected]>Cc Christopher Lameter <[email protected]>Cc Huang, Ying <[email protected]>Signed-off-by Yang Shi <[email protected]>Reviewed-by Yin Fengwei <[email protected]>Quoted 16 lines
--- include/linux/mman.h | 1 + 1 file changed, 1 insertion(+)include/linux/mman.h +1
-0@@ -156,6 +156,7 @@ calc_vm_flag_bits(unsigned long flags)return _calc_vm_trans (flags,MAP_GROWSDOWN ,VM_GROWSDOWN ) |_calc_vm_trans (flags,MAP_LOCKED ,VM_LOCKED ) |_calc_vm_trans (flags,MAP_SYNC ,VM_SYNC ) | +_calc_vm_trans (flags,MAP_STACK ,VM_NOHUGEPAGE ) |arch_calc_vm_flag_bits (flags); }
Re: [PATCH 2/2] mm: mmap: map MAP_STACK to VM_NOHUGEPAGE
Quoted 8 lines · Zach O'Keefe
Thanks Yang,
Should this be marked for stable? Given how easily it is for pthreads to allocate hugepages w/o this change, it can easily cause memory bloat on larger systems and/or users with high thread counts. I don't think that will be welcomed, and seems odd that just 6.7 should suffer this.
Thanks for the suggestion, fine to me.
Quoted 81 lines
Thanks,
ZachQuoted 76 lines · Yin Fengwei
Quoted 44 lines · Yang Shi
From: Yang Shi <[email protected]>
The commit
efa7df3e3bb5("mm: align larger anonymous mappings on THP boundaries") incured regression for stress-ng pthread benchmark [1]. It is because THP get allocated to pthread's stack area much more possible than before. Pthread's stack area is allocated by mmap withoutVM_GROWSDOWNorVM_GROWSUPflag, so kernel can't tell whether it is a stack area or not.The
MAP_STACKflag is used to mark the stack area, but it is a no-op on Linux. MappingMAP_STACKtoVM_NOHUGEPAGEto prevent from allocating THP for such stack area.With this change the stack area looks like:
fffd18e10000-fffd19610000 rw-p 00000000 00:00 0
Size: 8192 kB KernelPageSize: 4 kB MMUPageSize: 4 kB Rss: 12 kB Pss: 12 kB Pss_Dirty: 12 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 12 kB Referenced: 12 kB Anonymous: 12 kB KSM: 0 kB LazyFree: 0 kB AnonHugePages: 0 kB ShmemPmdMapped: 0 kB FilePmdMapped: 0 kB Shared_Hugetlb: 0 kB Private_Hugetlb: 0 kB Swap: 0 kB SwapPss: 0 kB Locked: 0 kB THPeligible: 0 VmFlags: rd wr mr mw me ac nhThe "nh" flag is set.
Quoted 8 lines
Reported-by kernel test robot <[email protected]>Tested-by Oliver Sang <[email protected]>Cc Yin Fengwei <[email protected]>Cc Rik van Riel <[email protected]>Cc Matthew Wilcox <[email protected]>Cc Christopher Lameter <[email protected]>Cc Huang, Ying <[email protected]>Signed-off-by Yang Shi <[email protected]>Reviewed-by Yin Fengwei <[email protected]>Quoted 16 lines
--- include/linux/mman.h | 1 + 1 file changed, 1 insertion(+)include/linux/mman.h +1
-0@@ -156,6 +156,7 @@ calc_vm_flag_bits(unsigned long flags)return _calc_vm_trans (flags,MAP_GROWSDOWN ,VM_GROWSDOWN ) |_calc_vm_trans (flags,MAP_LOCKED ,VM_LOCKED ) |_calc_vm_trans (flags,MAP_SYNC ,VM_SYNC ) | +_calc_vm_trans (flags,MAP_STACK ,VM_NOHUGEPAGE ) |arch_calc_vm_flag_bits (flags); }
Andrew Morton
maintainerRe: [PATCH 2/2] mm: mmap: map MAP_STACK to VM_NOHUGEPAGE
Quoted 8 lines · On Tue, 16 Jan 2024 12:57:41 -0800 Yang Shi <[email protected]> wrote:
Quoted 5 lines
Should this be marked for stable? Given how easily it is for pthreads to allocate hugepages w/o this change, it can easily cause memory bloat on larger systems and/or users with high thread counts. I don't think that will be welcomed, and seems odd that just 6.7 should suffer this.
Thanks for the suggestion, fine to me.
Thanks, added, along with
Re: [PATCH 2/2] mm: mmap: map MAP_STACK to VM_NOHUGEPAGE
In reply to Yang Shi* Yang Shi:
Quoted 11 lines
From: Yang Shi <[email protected]>
The commit
efa7df3e3bb5("mm: align larger anonymous mappings on THP boundaries") incured regression for stress-ng pthread benchmark [1]. It is because THP get allocated to pthread's stack area much more possible than before. Pthread's stack area is allocated by mmap withoutVM_GROWSDOWNorVM_GROWSUPflag, so kernel can't tell whether it is a stack area or not.The
MAP_STACKflag is used to mark the stack area, but it is a no-op on Linux. MappingMAP_STACKtoVM_NOHUGEPAGEto prevent from allocating THP for such stack area.
Doesn't this introduce a regression in the other direction, where workloads expect to use a hugepage TLB entry for the stack?
It's seems an odd approach to fixing the stress-ng regression. Isn't it very much coding to the benchmark?
Thanks,
Florian
Re: [PATCH 2/2] mm: mmap: map MAP_STACK to VM_NOHUGEPAGE
Quoted 11 lines · Florian Weimer
* Yang Shi:
Quoted 8 lines
From: Yang Shi <[email protected]>
The commit
efa7df3e3bb5("mm: align larger anonymous mappings on THP boundaries") incured regression for stress-ng pthread benchmark [1]. It is because THP get allocated to pthread's stack area much more possible than before. Pthread's stack area is allocated by mmap withoutVM_GROWSDOWNorVM_GROWSUPflag, so kernel can't tell whether it is a stack area or not.
Quoted 6 lines
Quoted 3 lines
The
MAP_STACKflag is used to mark the stack area, but it is a no-op on Linux. MappingMAP_STACKtoVM_NOHUGEPAGEto prevent from allocating THP for such stack area.Doesn't this introduce a regression in the other direction, where workloads expect to use a hugepage TLB entry for the stack?
Maybe, it is theoretically possible. But AFAICT, the real life workloads performance usually gets hurt if THP is used for stack. Willy has an example:
https://lore.kernel.org/linux-mm/[email protected]/#t
And avoiding THP on stack is not new, VM_GROWSDOWN | VM_GROWSUP areas have been applied before, this patch just extends this to MAP_STACK.
Quoted 7 lines
It's seems an odd approach to fixing the stress-ng regression. Isn't it very much coding to the benchmark?
Thanks,
Florian
Re: [PATCH 2/2] mm: mmap: map MAP_STACK to VM_NOHUGEPAGE
* Yang Shi:
Quoted 19 lines · Florian Weimer
Quoted 11 lines
* Yang Shi:
Quoted 8 lines
From: Yang Shi <[email protected]>
The commit
efa7df3e3bb5("mm: align larger anonymous mappings on THP boundaries") incured regression for stress-ng pthread benchmark [1]. It is because THP get allocated to pthread's stack area much more possible than before. Pthread's stack area is allocated by mmap withoutVM_GROWSDOWNorVM_GROWSUPflag, so kernel can't tell whether it is a stack area or not.Quoted 6 lines
Quoted 3 lines
The
MAP_STACKflag is used to mark the stack area, but it is a no-op on Linux. MappingMAP_STACKtoVM_NOHUGEPAGEto prevent from allocating THP for such stack area.Doesn't this introduce a regression in the other direction, where workloads expect to use a hugepage TLB entry for the stack?
Quoted 8 lines
Maybe, it is theoretically possible. But AFAICT, the real life workloads performance usually gets hurt if THP is used for stack. Willy has an example:
https://lore.kernel.org/linux-mm/[email protected]/#t
And avoiding THP on stack is not new,
VM_GROWSDOWN|VM_GROWSUPareas have been applied before, this patch just extends this toMAP_STACK.
If it's *always* beneficial then we should help it along in glibc as well. We've started to offer a tunable in response to this observation (also paper over in OpenJDK):
Make thread stacks not use huge pages
<https://bugs.openjdk.org/browse/JDK-8303215>
But this is specifically about RSS usage, and not directly about reducing TLB misses etc.
Thanks,
Florian
Re: [PATCH 2/2] mm: mmap: map MAP_STACK to VM_NOHUGEPAGE
Quoted 35 lines · Florian Weimer
* Yang Shi:
Quoted 19 lines · Florian Weimer
Quoted 11 lines
* Yang Shi:
Quoted 8 lines
From: Yang Shi <[email protected]>
The commit
efa7df3e3bb5("mm: align larger anonymous mappings on THP boundaries") incured regression for stress-ng pthread benchmark [1]. It is because THP get allocated to pthread's stack area much more possible than before. Pthread's stack area is allocated by mmap withoutVM_GROWSDOWNorVM_GROWSUPflag, so kernel can't tell whether it is a stack area or not.Quoted 6 lines
Quoted 3 lines
The
MAP_STACKflag is used to mark the stack area, but it is a no-op on Linux. MappingMAP_STACKtoVM_NOHUGEPAGEto prevent from allocating THP for such stack area.Doesn't this introduce a regression in the other direction, where workloads expect to use a hugepage TLB entry for the stack?
Quoted 8 lines
Maybe, it is theoretically possible. But AFAICT, the real life workloads performance usually gets hurt if THP is used for stack. Willy has an example:
https://lore.kernel.org/linux-mm/[email protected]/#t
And avoiding THP on stack is not new,
VM_GROWSDOWN|VM_GROWSUPareas have been applied before, this patch just extends this toMAP_STACK.If it's *always* beneficial then we should help it along in glibc as well. We've started to offer a tunable in response to this observation (also paper over in OpenJDK):
Quoted 5 lines
Make thread stacks not use huge pages
<https://bugs.openjdk.org/browse/JDK-8303215>But this is specifically about RSS usage, and not directly about reducing TLB misses etc.
Thanks for the data point. Out of curiosity, what mmap flags are used by JVM to indicate a stack? MAP_STACK? If so it should get VM_NOHUGEPAGE due to this patch (of course, on older kernel MADV_NOHUGEPAGE must be called by JVM).
Letting others, for example, glibc, call MADV_NOHUGEPAGE explicitly on stack area is fine too, but it may take some time to get there...
Quoted 4 lines
Thanks,
Florian
Quoted 7 lines · Yang Shi
From: Yang Shi <[email protected]>
We avoid allocating THP for temporary stack, even tnough
khugepaged_enter_vma()is called for stack VMAs, it actualy returns false. So no need to call it in the first place at all.Signed-off-by Yang Shi <[email protected]>
Quoted 24 lines
--- mm/mmap.c | 2 -- 1 file changed, 2 deletions(-)mm/mmap.c +0
-2@@ -2046,7 +2046,6 @@ static int expand_upwards(struct vm_area_struct *vma, unsigned long address) } }anon_vma_unlock_write (vma->anon_vma); -khugepaged_enter_vma (vma, vma->vm_flags);mas_destroy (&mas);validate_mm (mm);return error;@@ -2140,7 +2139,6 @@ int expand_downwards(struct vm_area_struct *vma, unsigned long address) } }anon_vma_unlock_write (vma->anon_vma); -khugepaged_enter_vma (vma, vma->vm_flags);mas_destroy (&mas);validate_mm (mm);return error;
Quoted 3 lines · Yang Shi
From: Yang Shi <[email protected]>
We avoid allocating THP for temporary stack, even tnough
~~~~~~
though?--
Best Regards,
Huang, Ying
Quoted 28 lines
khugepaged_enter_vma()is called for stack VMAs, it actualy returns false. So no need to call it in the first place at all.Signed-off-by Yang Shi <[email protected]>--- mm/mmap.c | 2 -- 1 file changed, 2 deletions(-)mm/mmap.c +0
-2@@ -2046,7 +2046,6 @@ static int expand_upwards(struct vm_area_struct *vma, unsigned long address) } }anon_vma_unlock_write (vma->anon_vma); -khugepaged_enter_vma (vma, vma->vm_flags);mas_destroy (&mas);validate_mm (mm);return error;@@ -2140,7 +2139,6 @@ int expand_downwards(struct vm_area_struct *vma, unsigned long address) } }anon_vma_unlock_write (vma->anon_vma); -khugepaged_enter_vma (vma, vma->vm_flags);mas_destroy (&mas);validate_mm (mm);return error;
Quoted 8 lines · Ying
Quoted 3 lines · Yang Shi
From: Yang Shi <[email protected]>
We avoid allocating THP for temporary stack, even tnough
~~~~~~ though?
Yeah, it is a typo. Thanks for noticing this.
Quoted 33 lines
--
Best Regards,
Huang, YingQuoted 28 lines
khugepaged_enter_vma()is called for stack VMAs, it actualy returns false. So no need to call it in the first place at all.Signed-off-by Yang Shi <[email protected]>--- mm/mmap.c | 2 -- 1 file changed, 2 deletions(-)mm/mmap.c +0
-2@@ -2046,7 +2046,6 @@ static int expand_upwards(struct vm_area_struct *vma, unsigned long address) } }anon_vma_unlock_write (vma->anon_vma); -khugepaged_enter_vma (vma, vma->vm_flags);mas_destroy (&mas);validate_mm (mm);return error;@@ -2140,7 +2139,6 @@ int expand_downwards(struct vm_area_struct *vma, unsigned long address) } }anon_vma_unlock_write (vma->anon_vma); -khugepaged_enter_vma (vma, vma->vm_flags);mas_destroy (&mas);validate_mm (mm);return error;