Skip to contentSkip to conversation

mm: mmap: no need to call khugepaged_enter_vma() for stack

PATCH Applied Yang Shi 6 participants 13 messages Jan 9, 2024 – Feb 1, 2024

Subsystems (2)
lore
Share & tools
lore

Yang Shi

author
#0 lore raw

PATCH 1/2 mm: mmap: no need to call khugepaged_enter_vma() for stack

From: Yang Shi <[email protected]>

We avoid allocating THP for temporary stack, even tnough khugepaged_enter_vma() is called for stack VMAs, it actualy returns false. So no need to call it in the first place at all.

---
 mm/mmap.c | 2 --
 1 file changed, 2 deletions(-)
mm/mmap.c +0 -2
index b78e83d351d2..2ff79b1d1564 100644
@@ -2046,7 +2046,6 @@ static int expand_upwards(struct vm_area_struct *vma, unsigned long address)
 		}
 	}
 	anon_vma_unlock_write(vma->anon_vma);
-	khugepaged_enter_vma(vma, vma->vm_flags);
 	mas_destroy(&mas);
 	validate_mm(mm);
 	return error;
@@ -2140,7 +2139,6 @@ int expand_downwards(struct vm_area_struct *vma, unsigned long address)
 		}
 	}
 	anon_vma_unlock_write(vma->anon_vma);
-	khugepaged_enter_vma(vma, vma->vm_flags);
 	mas_destroy(&mas);
 	validate_mm(mm);
 	return error;
Signature · 1 line
2.41.0
author
#1 lore raw

PATCH 2/2 mm: mmap: map MAP_STACK to VM_NOHUGEPAGE

From: Yang Shi <[email protected]>

The commit efa7df3e3bb5 ("mm: align larger anonymous mappings on THP boundaries") incured regression for stress-ng pthread benchmark [1]. It is because THP get allocated to pthread's stack area much more possible than before. Pthread's stack area is allocated by mmap without VM_GROWSDOWN or VM_GROWSUP flag, so kernel can't tell whether it is a stack area or not.

The MAP_STACK flag is used to mark the stack area, but it is a no-op on Linux. Mapping MAP_STACK to VM_NOHUGEPAGE to prevent from allocating THP for such stack area.

With this change the stack area looks like:

fffd18e10000-fffd19610000 rw-p 00000000 00:00 0

Size:               8192 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Rss:                  12 kB
Pss:                  12 kB
Pss_Dirty:            12 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:        12 kB
Referenced:           12 kB
Anonymous:            12 kB
KSM:                   0 kB
LazyFree:              0 kB
AnonHugePages:         0 kB
ShmemPmdMapped:        0 kB
FilePmdMapped:         0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
Locked:                0 kB
THPeligible:           0
VmFlags: rd wr mr mw me ac nh

The "nh" flag is set.

[1] https://lore.kernel.org/linux-mm/[email protected]/

Cc Yin Fengwei <[email protected]>
Cc Rik van Riel <[email protected]>
Cc Matthew Wilcox <[email protected]>
Cc Christopher Lameter <[email protected]>
Cc Huang, Ying <[email protected]>
---
 include/linux/mman.h | 1 +
 1 file changed, 1 insertion(+)
include/linux/mman.h +1 -0
index 40d94411d492..dc7048824be8 100644
@@ -156,6 +156,7 @@ calc_vm_flag_bits(unsigned long flags)
 	return _calc_vm_trans(flags, MAP_GROWSDOWN,  VM_GROWSDOWN ) |
 	       _calc_vm_trans(flags, MAP_LOCKED,     VM_LOCKED    ) |
 	       _calc_vm_trans(flags, MAP_SYNC,	     VM_SYNC      ) |
+	       _calc_vm_trans(flags, MAP_STACK,	     VM_NOHUGEPAGE) |
 	       arch_calc_vm_flag_bits(flags);
 }
 
Signature · 1 line
2.41.0
Reviewed-by on v1 2/2
#2 lore raw

Re: [PATCH 2/2] mm: mmap: map MAP_STACK to VM_NOHUGEPAGE

Quoted 44 lines · Yang Shi

From: Yang Shi <[email protected]>

The commit efa7df3e3bb5 ("mm: align larger anonymous mappings on THP boundaries") incured regression for stress-ng pthread benchmark [1]. It is because THP get allocated to pthread's stack area much more possible than before. Pthread's stack area is allocated by mmap without VM_GROWSDOWN or VM_GROWSUP flag, so kernel can't tell whether it is a stack area or not.

The MAP_STACK flag is used to mark the stack area, but it is a no-op on Linux. Mapping MAP_STACK to VM_NOHUGEPAGE to prevent from allocating THP for such stack area.

With this change the stack area looks like:

fffd18e10000-fffd19610000 rw-p 00000000 00:00 0

Size:               8192 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Rss:                  12 kB
Pss:                  12 kB
Pss_Dirty:            12 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:        12 kB
Referenced:           12 kB
Anonymous:            12 kB
KSM:                   0 kB
LazyFree:              0 kB
AnonHugePages:         0 kB
ShmemPmdMapped:        0 kB
FilePmdMapped:         0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
Locked:                0 kB
THPeligible:           0
VmFlags: rd wr mr mw me ac nh

The "nh" flag is set.

[1] https://lore.kernel.org/linux-mm/[email protected]/

Quoted 8 lines
Cc Yin Fengwei <[email protected]>
Cc Rik van Riel <[email protected]>
Cc Matthew Wilcox <[email protected]>
Cc Christopher Lameter <[email protected]>
Cc Huang, Ying <[email protected]>
Quoted 16 lines
---
  include/linux/mman.h | 1 +
  1 file changed, 1 insertion(+)
include/linux/mman.h +1 -0
index 40d94411d492..dc7048824be8 100644
@@ -156,6 +156,7 @@ calc_vm_flag_bits(unsigned long flags)
  	return _calc_vm_trans(flags, MAP_GROWSDOWN,  VM_GROWSDOWN ) |
  	       _calc_vm_trans(flags, MAP_LOCKED,     VM_LOCKED    ) |
  	       _calc_vm_trans(flags, MAP_SYNC,	     VM_SYNC      ) |
+	       _calc_vm_trans(flags, MAP_STACK,	     VM_NOHUGEPAGE) |
  	       arch_calc_vm_flag_bits(flags);
  }
 
#3 lore raw

Re: [PATCH 2/2] mm: mmap: map MAP_STACK to VM_NOHUGEPAGE

Thanks Yang,

Should this be marked for stable? Given how easily it is for pthreads to allocate hugepages w/o this change, it can easily cause memory bloat on larger systems and/or users with high thread counts. I don't think that will be welcomed, and seems odd that just 6.7 should suffer this.

Thanks,
Zach

Quoted 76 lines · Yin Fengwei
Quoted 44 lines · Yang Shi

From: Yang Shi <[email protected]>

The commit efa7df3e3bb5 ("mm: align larger anonymous mappings on THP boundaries") incured regression for stress-ng pthread benchmark [1]. It is because THP get allocated to pthread's stack area much more possible than before. Pthread's stack area is allocated by mmap without VM_GROWSDOWN or VM_GROWSUP flag, so kernel can't tell whether it is a stack area or not.

The MAP_STACK flag is used to mark the stack area, but it is a no-op on Linux. Mapping MAP_STACK to VM_NOHUGEPAGE to prevent from allocating THP for such stack area.

With this change the stack area looks like:

fffd18e10000-fffd19610000 rw-p 00000000 00:00 0

Size:               8192 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Rss:                  12 kB
Pss:                  12 kB
Pss_Dirty:            12 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:        12 kB
Referenced:           12 kB
Anonymous:            12 kB
KSM:                   0 kB
LazyFree:              0 kB
AnonHugePages:         0 kB
ShmemPmdMapped:        0 kB
FilePmdMapped:         0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
Locked:                0 kB
THPeligible:           0
VmFlags: rd wr mr mw me ac nh

The "nh" flag is set.

[1] https://lore.kernel.org/linux-mm/[email protected]/

Quoted 8 lines
Cc Yin Fengwei <[email protected]>
Cc Rik van Riel <[email protected]>
Cc Matthew Wilcox <[email protected]>
Cc Christopher Lameter <[email protected]>
Cc Huang, Ying <[email protected]>
Quoted 16 lines
---
  include/linux/mman.h | 1 +
  1 file changed, 1 insertion(+)
include/linux/mman.h +1 -0
index 40d94411d492..dc7048824be8 100644
@@ -156,6 +156,7 @@ calc_vm_flag_bits(unsigned long flags)
      return _calc_vm_trans(flags, MAP_GROWSDOWN,  VM_GROWSDOWN ) |
             _calc_vm_trans(flags, MAP_LOCKED,     VM_LOCKED    ) |
             _calc_vm_trans(flags, MAP_SYNC,       VM_SYNC      ) |
+            _calc_vm_trans(flags, MAP_STACK,      VM_NOHUGEPAGE) |
             arch_calc_vm_flag_bits(flags);
  }
 
author
#4 lore raw

Re: [PATCH 2/2] mm: mmap: map MAP_STACK to VM_NOHUGEPAGE

Quoted 8 lines · Zach O'Keefe

Thanks Yang,

Should this be marked for stable? Given how easily it is for pthreads to allocate hugepages w/o this change, it can easily cause memory bloat on larger systems and/or users with high thread counts. I don't think that will be welcomed, and seems odd that just 6.7 should suffer this.

Thanks for the suggestion, fine to me.

Quoted 81 lines

Thanks,
Zach

Quoted 76 lines · Yin Fengwei
Quoted 44 lines · Yang Shi

From: Yang Shi <[email protected]>

The commit efa7df3e3bb5 ("mm: align larger anonymous mappings on THP boundaries") incured regression for stress-ng pthread benchmark [1]. It is because THP get allocated to pthread's stack area much more possible than before. Pthread's stack area is allocated by mmap without VM_GROWSDOWN or VM_GROWSUP flag, so kernel can't tell whether it is a stack area or not.

The MAP_STACK flag is used to mark the stack area, but it is a no-op on Linux. Mapping MAP_STACK to VM_NOHUGEPAGE to prevent from allocating THP for such stack area.

With this change the stack area looks like:

fffd18e10000-fffd19610000 rw-p 00000000 00:00 0

Size:               8192 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Rss:                  12 kB
Pss:                  12 kB
Pss_Dirty:            12 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:        12 kB
Referenced:           12 kB
Anonymous:            12 kB
KSM:                   0 kB
LazyFree:              0 kB
AnonHugePages:         0 kB
ShmemPmdMapped:        0 kB
FilePmdMapped:         0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
Locked:                0 kB
THPeligible:           0
VmFlags: rd wr mr mw me ac nh

The "nh" flag is set.

[1] https://lore.kernel.org/linux-mm/[email protected]/

Quoted 8 lines
Cc Yin Fengwei <[email protected]>
Cc Rik van Riel <[email protected]>
Cc Matthew Wilcox <[email protected]>
Cc Christopher Lameter <[email protected]>
Cc Huang, Ying <[email protected]>
Quoted 16 lines
---
  include/linux/mman.h | 1 +
  1 file changed, 1 insertion(+)
include/linux/mman.h +1 -0
index 40d94411d492..dc7048824be8 100644
@@ -156,6 +156,7 @@ calc_vm_flag_bits(unsigned long flags)
      return _calc_vm_trans(flags, MAP_GROWSDOWN,  VM_GROWSDOWN ) |
             _calc_vm_trans(flags, MAP_LOCKED,     VM_LOCKED    ) |
             _calc_vm_trans(flags, MAP_SYNC,       VM_SYNC      ) |
+            _calc_vm_trans(flags, MAP_STACK,      VM_NOHUGEPAGE) |
             arch_calc_vm_flag_bits(flags);
  }
 
maintainer
#5 lore raw

Re: [PATCH 2/2] mm: mmap: map MAP_STACK to VM_NOHUGEPAGE

Quoted 8 lines · On Tue, 16 Jan 2024 12:57:41 -0800 Yang Shi <[email protected]> wrote:
Quoted 5 lines

Should this be marked for stable? Given how easily it is for pthreads to allocate hugepages w/o this change, it can easily cause memory bloat on larger systems and/or users with high thread counts. I don't think that will be welcomed, and seems odd that just 6.7 should suffer this.

Thanks for the suggestion, fine to me.

Thanks, added, along with

Fixes efa7df3e3bb5 ("mm: align larger anonymous mappings on THP boundaries")
#6 lore raw

Re: [PATCH 2/2] mm: mmap: map MAP_STACK to VM_NOHUGEPAGE

In reply to Yang Shi

* Yang Shi:

Quoted 11 lines

From: Yang Shi <[email protected]>

The commit efa7df3e3bb5 ("mm: align larger anonymous mappings on THP boundaries") incured regression for stress-ng pthread benchmark [1]. It is because THP get allocated to pthread's stack area much more possible than before. Pthread's stack area is allocated by mmap without VM_GROWSDOWN or VM_GROWSUP flag, so kernel can't tell whether it is a stack area or not.

The MAP_STACK flag is used to mark the stack area, but it is a no-op on Linux. Mapping MAP_STACK to VM_NOHUGEPAGE to prevent from allocating THP for such stack area.

Doesn't this introduce a regression in the other direction, where workloads expect to use a hugepage TLB entry for the stack?

It's seems an odd approach to fixing the stress-ng regression. Isn't it very much coding to the benchmark?

Thanks,
Florian

author
#7 lore raw

Re: [PATCH 2/2] mm: mmap: map MAP_STACK to VM_NOHUGEPAGE

Quoted 11 lines · Florian Weimer

* Yang Shi:

Quoted 8 lines

From: Yang Shi <[email protected]>

The commit efa7df3e3bb5 ("mm: align larger anonymous mappings on THP boundaries") incured regression for stress-ng pthread benchmark [1]. It is because THP get allocated to pthread's stack area much more possible than before. Pthread's stack area is allocated by mmap without VM_GROWSDOWN or VM_GROWSUP flag, so kernel can't tell whether it is a stack area or not.

Quoted 6 lines
Quoted 3 lines

The MAP_STACK flag is used to mark the stack area, but it is a no-op on Linux. Mapping MAP_STACK to VM_NOHUGEPAGE to prevent from allocating THP for such stack area.

Doesn't this introduce a regression in the other direction, where workloads expect to use a hugepage TLB entry for the stack?

Maybe, it is theoretically possible. But AFAICT, the real life workloads performance usually gets hurt if THP is used for stack. Willy has an example:

https://lore.kernel.org/linux-mm/[email protected]/#t

And avoiding THP on stack is not new, VM_GROWSDOWN | VM_GROWSUP areas have been applied before, this patch just extends this to MAP_STACK.

Quoted 7 lines

It's seems an odd approach to fixing the stress-ng regression. Isn't it very much coding to the benchmark?

Thanks,
Florian

#8 lore raw

Re: [PATCH 2/2] mm: mmap: map MAP_STACK to VM_NOHUGEPAGE

* Yang Shi:

Quoted 19 lines · Florian Weimer
Quoted 11 lines

* Yang Shi:

Quoted 8 lines

From: Yang Shi <[email protected]>

The commit efa7df3e3bb5 ("mm: align larger anonymous mappings on THP boundaries") incured regression for stress-ng pthread benchmark [1]. It is because THP get allocated to pthread's stack area much more possible than before. Pthread's stack area is allocated by mmap without VM_GROWSDOWN or VM_GROWSUP flag, so kernel can't tell whether it is a stack area or not.

Quoted 6 lines
Quoted 3 lines

The MAP_STACK flag is used to mark the stack area, but it is a no-op on Linux. Mapping MAP_STACK to VM_NOHUGEPAGE to prevent from allocating THP for such stack area.

Doesn't this introduce a regression in the other direction, where workloads expect to use a hugepage TLB entry for the stack?

Quoted 8 lines

Maybe, it is theoretically possible. But AFAICT, the real life workloads performance usually gets hurt if THP is used for stack. Willy has an example:

https://lore.kernel.org/linux-mm/[email protected]/#t

And avoiding THP on stack is not new, VM_GROWSDOWN | VM_GROWSUP areas have been applied before, this patch just extends this to MAP_STACK.

If it's *always* beneficial then we should help it along in glibc as well. We've started to offer a tunable in response to this observation (also paper over in OpenJDK):

Make thread stacks not use huge pages
<https://bugs.openjdk.org/browse/JDK-8303215>

But this is specifically about RSS usage, and not directly about reducing TLB misses etc.

Thanks,
Florian

author
#9 lore raw

Re: [PATCH 2/2] mm: mmap: map MAP_STACK to VM_NOHUGEPAGE

Quoted 35 lines · Florian Weimer

* Yang Shi:

Quoted 19 lines · Florian Weimer
Quoted 11 lines

* Yang Shi:

Quoted 8 lines

From: Yang Shi <[email protected]>

The commit efa7df3e3bb5 ("mm: align larger anonymous mappings on THP boundaries") incured regression for stress-ng pthread benchmark [1]. It is because THP get allocated to pthread's stack area much more possible than before. Pthread's stack area is allocated by mmap without VM_GROWSDOWN or VM_GROWSUP flag, so kernel can't tell whether it is a stack area or not.

Quoted 6 lines
Quoted 3 lines

The MAP_STACK flag is used to mark the stack area, but it is a no-op on Linux. Mapping MAP_STACK to VM_NOHUGEPAGE to prevent from allocating THP for such stack area.

Doesn't this introduce a regression in the other direction, where workloads expect to use a hugepage TLB entry for the stack?

Quoted 8 lines

Maybe, it is theoretically possible. But AFAICT, the real life workloads performance usually gets hurt if THP is used for stack. Willy has an example:

https://lore.kernel.org/linux-mm/[email protected]/#t

And avoiding THP on stack is not new, VM_GROWSDOWN | VM_GROWSUP areas have been applied before, this patch just extends this to MAP_STACK.

If it's *always* beneficial then we should help it along in glibc as well. We've started to offer a tunable in response to this observation (also paper over in OpenJDK):

Quoted 5 lines

Make thread stacks not use huge pages
<https://bugs.openjdk.org/browse/JDK-8303215>

But this is specifically about RSS usage, and not directly about reducing TLB misses etc.

Thanks for the data point. Out of curiosity, what mmap flags are used by JVM to indicate a stack? MAP_STACK? If so it should get VM_NOHUGEPAGE due to this patch (of course, on older kernel MADV_NOHUGEPAGE must be called by JVM).

Letting others, for example, glibc, call MADV_NOHUGEPAGE explicitly on stack area is fine too, but it may take some time to get there...

Quoted 4 lines

Thanks,
Florian

Reviewed-by on v1 1/2
#10 lore raw
In reply to Yang Shi
Quoted 7 lines · Yang Shi

From: Yang Shi <[email protected]>

We avoid allocating THP for temporary stack, even tnough khugepaged_enter_vma() is called for stack VMAs, it actualy returns false. So no need to call it in the first place at all.

Quoted 24 lines
---
  mm/mmap.c | 2 --
  1 file changed, 2 deletions(-)
mm/mmap.c +0 -2
index b78e83d351d2..2ff79b1d1564 100644
@@ -2046,7 +2046,6 @@ static int expand_upwards(struct vm_area_struct *vma, unsigned long address)
  		}
  	}
  	anon_vma_unlock_write(vma->anon_vma);
-	khugepaged_enter_vma(vma, vma->vm_flags);
  	mas_destroy(&mas);
  	validate_mm(mm);
  	return error;
@@ -2140,7 +2139,6 @@ int expand_downwards(struct vm_area_struct *vma, unsigned long address)
  		}
  	}
  	anon_vma_unlock_write(vma->anon_vma);
-	khugepaged_enter_vma(vma, vma->vm_flags);
  	mas_destroy(&mas);
  	validate_mm(mm);
  	return error;
#11 lore raw
In reply to Yang Shi
Quoted 3 lines · Yang Shi

From: Yang Shi <[email protected]>

We avoid allocating THP for temporary stack, even tnough

                                                    ~~~~~~
                                                    though?

--
Best Regards,
Huang, Ying

Quoted 28 lines

khugepaged_enter_vma() is called for stack VMAs, it actualy returns false. So no need to call it in the first place at all.

---
 mm/mmap.c | 2 --
 1 file changed, 2 deletions(-)
mm/mmap.c +0 -2
index b78e83d351d2..2ff79b1d1564 100644
@@ -2046,7 +2046,6 @@ static int expand_upwards(struct vm_area_struct *vma, unsigned long address)
 		}
 	}
 	anon_vma_unlock_write(vma->anon_vma);
-	khugepaged_enter_vma(vma, vma->vm_flags);
 	mas_destroy(&mas);
 	validate_mm(mm);
 	return error;
@@ -2140,7 +2139,6 @@ int expand_downwards(struct vm_area_struct *vma, unsigned long address)
 		}
 	}
 	anon_vma_unlock_write(vma->anon_vma);
-	khugepaged_enter_vma(vma, vma->vm_flags);
 	mas_destroy(&mas);
 	validate_mm(mm);
 	return error;
author
#12 lore raw
Quoted 8 lines · Ying
Quoted 3 lines · Yang Shi

From: Yang Shi <[email protected]>

We avoid allocating THP for temporary stack, even tnough

                                                    ~~~~~~
                                                    though?

Yeah, it is a typo. Thanks for noticing this.

Quoted 33 lines

--
Best Regards,
Huang, Ying

Quoted 28 lines

khugepaged_enter_vma() is called for stack VMAs, it actualy returns false. So no need to call it in the first place at all.

---
 mm/mmap.c | 2 --
 1 file changed, 2 deletions(-)
mm/mmap.c +0 -2
index b78e83d351d2..2ff79b1d1564 100644
@@ -2046,7 +2046,6 @@ static int expand_upwards(struct vm_area_struct *vma, unsigned long address)
              }
      }
      anon_vma_unlock_write(vma->anon_vma);
-     khugepaged_enter_vma(vma, vma->vm_flags);
      mas_destroy(&mas);
      validate_mm(mm);
      return error;
@@ -2140,7 +2139,6 @@ int expand_downwards(struct vm_area_struct *vma, unsigned long address)
              }
      }
      anon_vma_unlock_write(vma->anon_vma);
-     khugepaged_enter_vma(vma, vma->vm_flags);
      mas_destroy(&mas);
      validate_mm(mm);
      return error;