variable-order, large folios for anonymous memory
Subsystems (6)
v4 cover variable-order, large folios for anonymous memory
Hi All,
This is v4 of a series to implement variable order, large folios for anonymous memory. (currently called "LARGE_ANON_FOLIO", previously called "FLEXIBLE_THP"). The objective of this is to improve performance by allocating larger chunks of memory during anonymous page faults:
1) Since SW (the kernel) is dealing with larger chunks of memory than base pages, there are efficiency savings to be had; fewer page faults, batched PTE and RMAP manipulation, reduced lru list, etc. In short, we reduce kernel overhead. This should benefit all architectures.
2) Since we are now mapping physically contiguous chunks of memory, we can take advantage of HW TLB compression techniques. A reduction in TLB pressure speeds up kernel and user space. arm64 systems have 2 mechanisms to coalesce TLB entries; "the contiguous bit" (architectural) and HPA (uarch).
This patch set deals with the SW side of things (1). (2) is being tackled in a separate series. The new behaviour is hidden behind a new Kconfig switch, LARGE_ANON_FOLIO, which is disabled by default. Although the eventual aim is to enable it by default.
The "arm64: mm: Override arch_wants_pte_order()" patch is for arm64 to explicitly override the default arch_wants_pte_order() and is intended as an example. If this series is accepted I suggest ignoring this patch which I will take through the arm64 tree separately. Neither has any build dependency on the other.
My hope is that we are pretty much there with the changes at this point. Although they should not be merged until the prerequisites are complete. These are in progress and tracked at [4].
The patches are based on top of v6.5-rc3. I have a branch at [5].
Testing
-------
This version adds patches to mm selftests so that the cow tests explicitly test large anon folios, in the same way that thp is tested. When enabled you should see something similar at the start of the test suite:
# [INFO] detected large anon folio size: 64 KiB
Then the following results are expected. The fails and skips are due to existing issues:
# Totals: pass:263 fail:16 xfail:0 xpass:0 skip:29 error:0
Existing mm selftests reveal 1 regression in khugepaged tests when LARGE_ANON_FOLIO is enabled:
Run test: collapse_max_ptes_none (khugepaged:anon) Maybe collapse with max_ptes_none exceeded.... Fail Unexpected huge page
I believe this is because khugepaged currently skips non-order-0 pages when looking for collapse opportunities and should get fixed with the help of DavidH's work to create a mechanism to precisely determine shared vs exclusive pages.
Changes since v3 [3]
--------------------
- Renamed feature from FLEXIBLE_THP to LARGE_ANON_FOLIO.
- Removed flexthp_unhinted_max boot parameter. Discussion concluded that a sysctl is preferable but we will wait until real workload needs it.
- Fixed uninitialized addr on read fault path in do_anonymous_page().
- Added mm selftests for large anon folios in cow test suite.
Changes since v2 [2]
--------------------
- Dropped commit "Allow deferred splitting of arbitrary large anon folios"
- Huang, Ying suggested the "batch zap" work (which I dropped from this series after v1) is a prerequisite for merging FLXEIBLE_THP, so I've moved the deferred split patch to a separate series along with the batch zap changes. I plan to submit this series early next week.
- Changed folio order fallback policy
- We no longer iterate from preferred to 0 looking for acceptable policy
- Instead we iterate through preferred, PAGE_ALLOC_COSTLY_ORDER and 0 only
- Removed vma parameter from arch_wants_pte_order()
- Added command line parameter flexthp_unhinted_max
- clamps preferred order when vma hasn't explicitly opted-in to THP
- Never allocate large folio for MADV_NOHUGEPAGE vma (or when THP is disabled for process or system).
- Simplified implementation and integration with do_anonymous_page()
- Removed dependency on set_ptes()
Changes since v1 [1]
--------------------
- removed changes to arch-dependent vma_alloc_zeroed_movable_folio()
- replaced with arch-independent alloc_anon_folio()
- follows THP allocation approach
- no longer retry with intermediate orders if allocation fails
- fallback directly to order-0
- remove folio_add_new_anon_rmap_range() patch
- instead add its new functionality to folio_add_new_anon_rmap()
- remove batch-zap pte mappings optimization patch
- remove enabler folio_remove_rmap_range() patch too
- These offer real perf improvement so will submit separately
- simplify Kconfig
- single FLEXIBLE_THP option, which is independent of arch
- depends on TRANSPARENT_HUGEPAGE
- when enabled default to max anon folio size of 64K unless arch explicitly overrides
- simplify changes to do_anonymous_page():
- no more retry loop
Performance
-----------
Kernel compilation with 8 jobs:
| kernel | real-time | kern-time | user-time | |:---------------------------|------------:|------------:|------------:| | baseline-v6.5-rc3 | 0.0% | 0.0% | 0.0% | | anonfolio-v4 | -4.9% | -38.2% | -0.7% |
Kernel compilation with 80 jobs:
| kernel | real-time | kern-time | user-time | |:---------------------------|------------:|------------:|------------:| | baseline-v6.5-rc3 | 0.0% | 0.0% | 0.0% | | anonfolio-v4 | -5.1% | -42.3% | -0.4% |
Speedometer:
| kernel | runs_per_min | |:---------------------------|---------------:| | baseline-v6.5-rc3 | 0.0% | | anonfolio-v4 | 0.6% |
[1] https://lore.kernel.org/linux-mm/[email protected]/
[2] https://lore.kernel.org/linux-mm/[email protected]/
[3] https://lore.kernel.org/linux-mm/[email protected]/
[4] https://lore.kernel.org/linux-mm/[email protected]/
[5] https://gitlab.arm.com/linux-arm/linux-rr/-/tree/features/granule_perf/anonfolio-for_lkml_v4
Thanks,
Ryan
Ryan Roberts (5):
mm: Non-pmd-mappable, large folios for folio_add_new_anon_rmap()
mm: LARGE_ANON_FOLIO for improved performance
arm64: mm: Override arch_wants_pte_order() selftests/mm/cow: Generalize do_run_with_thp() helper selftests/mm/cow: Add large anon folio tests
arch/arm64/include/asm/pgtable.h | 6 + include/linux/pgtable.h | 13 ++ mm/Kconfig | 10 ++ mm/memory.c | 166 +++++++++++++++++++--- mm/rmap.c | 27 +++- tools/testing/selftests/mm/cow.c | 229 ++++++++++++++++++++++--------- 6 files changed, 365 insertions(+), 86 deletions(-)
--
2.25.1
PATCH v4 1/5 mm: Non-pmd-mappable, large folios for folio_add_new_anon_rmap()
In preparation for LARGE_ANON_FOLIO support, improve folio_add_new_anon_rmap() to allow a non-pmd-mappable, large folio to be passed to it. In this case, all contained pages are accounted using the order-0 folio (or base page) scheme.
--- mm/rmap.c | 27 ++++++++++++++++++++------- 1 file changed, 20 insertions(+), 7 deletions(-)
mm/rmap.c +20 -7
@@ -1278,31 +1278,44 @@ void page_add_anon_rmap(struct page *page, struct vm_area_struct *vma,* This means the inc-and-test can be bypassed. * The folio does not have to be locked. * -* If the folio is large, it is accounted as a THP. As the folio +* If the folio is pmd-mappable, it is accounted as a THP. As the folio * is new, it's assumed to be mapped exclusively by a single process. */ void folio_add_new_anon_rmap (struct folio *folio,struct vm_area_struct *vma,unsigned long address) { -int nr; +int nr =folio_nr_pages (folio); -VM_BUG_ON_VMA (address < vma->vm_start || address >= vma->vm_end, vma); +VM_BUG_ON_VMA (address < vma->vm_start || + address + (nr <<PAGE_SHIFT ) > vma->vm_end, vma);__folio_set_swapbacked (folio); -if (likely (!folio_test_pmd_mappable (folio))) { +if (likely (!folio_test_large (folio))) {/* increment count (starts at -1) */ atomic_set (&folio->_mapcount,0 ); - nr =1 ; +__page_set_anon_rmap (folio, &folio->page, vma, address,1 ); + }else if (!folio_test_pmd_mappable (folio)) { +int i; + +for (i =0 ; i < nr; i++) { +struct page *page =folio_page (folio, i); + +/* increment count (starts at -1) */ +atomic_set (&page->_mapcount,0 ); +__page_set_anon_rmap (folio, page, vma, + address + (i <<PAGE_SHIFT ),1 ); + } + +atomic_set (&folio->_nr_pages_mapped, nr); }else {/* increment count (starts at -1) */ atomic_set (&folio->_entire_mapcount,0 );atomic_set (&folio->_nr_pages_mapped,COMPOUND_MAPPED ); - nr =folio_nr_pages (folio); +__page_set_anon_rmap (folio, &folio->page, vma, address,1 );__lruvec_stat_mod_folio (folio,NR_ANON_THPS , nr); }__lruvec_stat_mod_folio (folio,NR_ANON_MAPPED , nr); -__page_set_anon_rmap (folio, &folio->page, vma, address,1 ); }/**
Signature · 1 line
2.25.1
PATCH v4 2/5 mm: LARGE_ANON_FOLIO for improved performance
In reply to Ryan RobertsIntroduce LARGE_ANON_FOLIO feature, which allows anonymous memory to be allocated in large folios of a determined order. All pages of the large folio are pte-mapped during the same page fault, significantly reducing the number of page faults. The number of per-page operations (e.g. ref counting, rmap management lru list management) are also significantly reduced since those ops now become per-folio.
The new behaviour is hidden behind the new LARGE_ANON_FOLIO Kconfig, which defaults to disabled for now; The long term aim is for this to defaut to enabled, but there are some risks around internal fragmentation that need to be better understood first.
When enabled, the folio order is determined as such: For a vma, process or system that has explicitly disabled THP, we continue to allocate order-0. THP is most likely disabled to avoid any possible internal fragmentation so we honour that request.
Otherwise, the return value of arch_wants_pte_order() is used. For vmas that have not explicitly opted-in to use transparent hugepages (e.g. where thp=madvise and the vma does not have MADV_HUGEPAGE), then arch_wants_pte_order() is limited to 64K (or PAGE_SIZE, whichever is bigger). This allows for a performance boost without requiring any explicit opt-in from the workload while limitting internal fragmentation.
If the preferred order can't be used (e.g. because the folio would breach the bounds of the vma, or because ptes in the region are already mapped) then we fall back to a suitable lower order; first PAGE_ALLOC_COSTLY_ORDER, then order-0.
arch_wants_pte_order() can be overridden by the architecture if desired. Some architectures (e.g. arm64) can coalsece TLB entries if a contiguous set of ptes map physically contigious, naturally aligned memory, so this mechanism allows the architecture to optimize as required.
Here we add the default implementation of arch_wants_pte_order(), used when the architecture does not define it, which returns -1, implying that the HW has no preference. In this case, mm will choose it's own default order.
--- include/linux/pgtable.h | 13 ++++ mm/Kconfig | 10 +++ mm/memory.c | 166 ++++++++++++++++++++++++++++++++++++---- 3 files changed, 172 insertions(+), 17 deletions(-)
include/linux/pgtable.h +13 -0
@@ -313,6 +313,19 @@ static inline bool arch_has_hw_pte_young(void) }#endif +#ifndef arch_wants_pte_order +/* +* Returns preferred folio order for pte-mapped memory. Must be in range [0, +* PMD_SHIFT-PAGE_SHIFT) and must not be order-1 since THP requires large folios +* to be at least order-2. Negative value implies that the HW has no preference +* and mm will choose it's own default order. +*/ +static inline int arch_wants_pte_order (void ) +{ +return -1 ; +} +#endif +#ifndef __HAVE_ARCH_PTEP_GET_AND_CLEAR static inline pte_t ptep_get_and_clear(struct mm_struct *mm, unsigned long address,
mm/Kconfig +10 -0
@@ -1238,4 +1238,14 @@ config LOCK_MM_AND_FIND_VMA source "mm/damon/Kconfig" +config LARGE_ANON_FOLIO + bool "Allocate large folios for anonymous memory" + depends on TRANSPARENT_HUGEPAGE + default n + help + Use large (bigger than order-0) folios to back anonymous memory where + possible, even for pte-mapped memory. This reduces the number of page + faults, as well as other per-page overheads to improve performance for + many workloads. + endmenu
mm/memory.c +149 -17
204 lines not on this page · raw message
Signature · 1 line
2.25.1
Re: [PATCH v4 2/5] mm: LARGE_ANON_FOLIO for improved performance
Quoted 129 lines · Ryan Roberts
Introduce
LARGE_ANON_FOLIOfeature, which allows anonymous memory to be allocated in large folios of a determined order. All pages of the large folio are pte-mapped during the same page fault, significantly reducing the number of page faults. The number of per-page operations (e.g. ref counting, rmap management lru list management) are also significantly reduced since those ops now become per-folio.The new behaviour is hidden behind the new
LARGE_ANON_FOLIOKconfig, which defaults to disabled for now; The long term aim is for this to defaut to enabled, but there are some risks around internal fragmentation that need to be better understood first.When enabled, the folio order is determined as such: For a vma, process or system that has explicitly disabled THP, we continue to allocate order-0. THP is most likely disabled to avoid any possible internal fragmentation so we honour that request.
Otherwise, the return value of
arch_wants_pte_order()is used. For vmas that have not explicitly opted-in to use transparent hugepages (e.g. where thp=madvise and the vma does not haveMADV_HUGEPAGE), thenarch_wants_pte_order()is limited to 64K (orPAGE_SIZE, whichever is bigger). This allows for a performance boost without requiring any explicit opt-in from the workload while limitting internal fragmentation.If the preferred order can't be used (e.g. because the folio would breach the bounds of the vma, or because ptes in the region are already mapped) then we fall back to a suitable lower order; first
PAGE_ALLOC_COSTLY_ORDER, then order-0.
arch_wants_pte_order()can be overridden by the architecture if desired. Some architectures (e.g. arm64) can coalsece TLB entries if a contiguous set of ptes map physically contigious, naturally aligned memory, so this mechanism allows the architecture to optimize as required.Here we add the default implementation of
arch_wants_pte_order(), used when the architecture does not define it, which returns -1, implying that the HW has no preference. In this case, mm will choose it's own default order.Signed-off-by Ryan Roberts <[email protected]>--- include/linux/pgtable.h | 13 ++++ mm/Kconfig | 10 +++ mm/memory.c | 166 ++++++++++++++++++++++++++++++++++++---- 3 files changed, 172 insertions(+), 17 deletions(-)include/linux/pgtable.h +13
-0@@ -313,6 +313,19 @@ static inline bool arch_has_hw_pte_young(void) }#endif +#ifndef arch_wants_pte_order +/* +* Returns preferred folio order for pte-mapped memory. Must be in range [0, +* PMD_SHIFT-PAGE_SHIFT) and must not be order-1 since THP requires large folios +* to be at least order-2. Negative value implies that the HW has no preference +* and mm will choose it's own default order. +*/ +static inline int arch_wants_pte_order (void ) +{ +return -1 ; +} +#endif +#ifndef __HAVE_ARCH_PTEP_GET_AND_CLEAR static inline pte_t ptep_get_and_clear(struct mm_struct *mm, unsigned long address,mm/Kconfig +10
-0@@ -1238,4 +1238,14 @@ config LOCK_MM_AND_FIND_VMA source "mm/damon/Kconfig" +config LARGE_ANON_FOLIO + bool "Allocate large folios for anonymous memory" + depends on TRANSPARENT_HUGEPAGE + default n + help + Use large (bigger than order-0) folios to back anonymous memory where + possible, even for pte-mapped memory. This reduces the number of page + faults, as well as other per-page overheads to improve performance for + many workloads. + endmenumm/memory.c +30
-0@@ -4050,6 +4050,127 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)return ret; } +static bool vmf_pte_range_changed (struct vm_fault *vmf,int nr_pages) +{ +int i; + +if (nr_pages ==1 ) +return vmf_pte_changed (vmf); + +for (i =0 ; i < nr_pages; i++) { +if (!pte_none (ptep_get_lockless (vmf->pte + i))) +return true; + } + +return false; +} + +#ifdef CONFIG_LARGE_ANON_FOLIO +#define ANON_FOLIO_MAX_ORDER_UNHINTED \ + (ilog2(max_t(unsigned long, SZ_64K, PAGE_SIZE)) - PAGE_SHIFT) + +static int anon_folio_order (struct vm_area_struct *vma) +{ +int order; + +/* +* If THP is explicitly disabled for either the vma, the process or the +* system, then this is very likely intended to limit internal +* fragmentation; in this case, don't attempt to allocate a large +* anonymous folio. +* +* Else, if the vma is eligible for thp, allocate a large folio of the
mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 4092
Quoted 10 lines
mm/memory.c +10
-0+* size preferred by the arch. Or if the arch requested a very small +* size or didn't request a size, then use PAGE_ALLOC_COSTLY_ORDER, +* which still meets the arch's requirements but means we still take +* advantage of SW optimizations (e.g. fewer page faults). +* +* Finally if thp is enabled but the vma isn't eligible, take the +* arch-preferred size and limit it to ANON_FOLIO_MAX_ORDER_UNHINTED. +* This ensures workloads that have not explicitly opted-in take benefit +* while capping the potential for internal fragmentation. +*/
What empirical evidence is SZ_64K based on?
What workloads would benefit from it?
How much would they benefit from it?
Would they benefit more or less from different values? How much internal fragmentation would it cause?
What cost function was used to arrive at the conclusion that its benefits outweigh its costs?
Quoted 5 lines
mm/memory.c +5
-0+if ((vma->vm_flags &VM_NOHUGEPAGE ) || +test_bit (MMF_DISABLE_THP , &vma->vm_mm->flags) || + !hugepage_flags_enabled ()) + order =0 ; +else {
mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 4108
Quoted 10 lines
mm/memory.c +10
-0+ order =max (arch_wants_pte_order (),PAGE_ALLOC_COSTLY_ORDER ); + +if (!hugepage_vma_check (vma, vma->vm_flags, false, true, true)) + order =min (order,ANON_FOLIO_MAX_ORDER_UNHINTED ); + } + +return order; +} + +static int alloc_anon_folio (struct vm_fault *vmf,struct folio **folio)
static struct folio *alloc_anon_folio(struct vm_fault *vmf)
and use ERR_PTR() and its friends.
Quoted 139 lines
mm/memory.c +89
-15+{ +int i; +gfp_t gfp; +pte_t *pte; +unsigned long addr; +struct vm_area_struct *vma = vmf->vma; +int prefer =anon_folio_order (vma); +int orders[] = { + prefer, + prefer >PAGE_ALLOC_COSTLY_ORDER ?PAGE_ALLOC_COSTLY_ORDER :0 , +0 , + }; + + *folio =NULL ; + +if (vmf_orig_pte_uffd_wp (vmf)) + goto fallback; + +for (i =0 ; orders[i]; i++) { + addr =ALIGN_DOWN (vmf->address,PAGE_SIZE << orders[i]); +if (addr >= vma->vm_start && + addr + (PAGE_SIZE << orders[i]) <= vma->vm_end) +break ; + } + +if (!orders[i]) + goto fallback; + + pte =pte_offset_map (vmf->pmd, vmf->address &PMD_MASK ); +if (!pte) +return -EAGAIN ; + +for (; orders[i]; i++) { + addr =ALIGN_DOWN (vmf->address,PAGE_SIZE << orders[i]); + vmf->pte = pte +pte_index (addr); +if (!vmf_pte_range_changed (vmf,1 << orders[i])) +break ; + } + + vmf->pte =NULL ; +pte_unmap (pte); + + gfp =vma_thp_gfp_mask (vma); + +for (; orders[i]; i++) { + addr =ALIGN_DOWN (vmf->address,PAGE_SIZE << orders[i]); + *folio =vma_alloc_folio (gfp, orders[i], vma, addr, true); +if (*folio) { +clear_huge_page (&(*folio)->page, addr,1 << orders[i]); +return 0 ; + } + } + +fallback: + *folio =vma_alloc_zeroed_movable_folio (vma, vmf->address); +return *folio ?0 : -ENOMEM ; +} +#else +static inline int alloc_anon_folio (struct vm_fault *vmf,struct folio **folio) +{ + *folio =vma_alloc_zeroed_movable_folio (vmf->vma, vmf->address); +return *folio ?0 : -ENOMEM ; +} +#endif +/* * We enter with non-exclusive mmap_lock (to exclude vma changes, * but allow concurrent faults), and pte mapped but not yet locked. @@ -4057,6 +4178,9 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)*/ static vm_fault_t do_anonymous_page (struct vm_fault *vmf) { +int i =0 ; +int nr_pages =1 ; +unsigned long addr = vmf->address;bool uffd_wp =vmf_orig_pte_uffd_wp (vmf);struct vm_area_struct *vma = vmf->vma;struct folio *folio;@@ -4101,10 +4225,15 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)/* Allocate our own private page. */ if (unlikely (anon_vma_prepare (vma))) goto oom; - folio =vma_alloc_zeroed_movable_folio (vma, vmf->address); + ret =alloc_anon_folio (vmf, &folio); +if (unlikely (ret == -EAGAIN )) +return 0 ;if (!folio) goto oom; + nr_pages =folio_nr_pages (folio); + addr =ALIGN_DOWN (vmf->address, nr_pages *PAGE_SIZE ); +if (mem_cgroup_charge (folio, vma->vm_mm,GFP_KERNEL )) goto oom_free_page;folio_throttle_swaprate (folio,GFP_KERNEL );@@ -4116,17 +4245,12 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)*/ __folio_mark_uptodate (folio); - entry =mk_pte (&folio->page, vma->vm_page_prot); - entry =pte_sw_mkyoung (entry); -if (vma->vm_flags &VM_WRITE ) - entry =pte_mkwrite (pte_mkdirty (entry)); - - vmf->pte =pte_offset_map_lock (vma->vm_mm, vmf->pmd, vmf->address, - &vmf->ptl); + vmf->pte =pte_offset_map_lock (vma->vm_mm, vmf->pmd, addr, &vmf->ptl);if (!vmf->pte) goto release; -if (vmf_pte_changed (vmf)) { -update_mmu_tlb (vma, vmf->address, vmf->pte); +if (vmf_pte_range_changed (vmf, nr_pages)) { +for (i =0 ; i < nr_pages; i++) +update_mmu_tlb (vma, addr +PAGE_SIZE * i, vmf->pte + i); goto release; }@@ -4141,16 +4265,24 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)return handle_userfault (vmf,VM_UFFD_MISSING ); } -inc_mm_counter (vma->vm_mm,MM_ANONPAGES ); -folio_add_new_anon_rmap (folio, vma, vmf->address); +folio_ref_add (folio, nr_pages -1 ); +add_mm_counter (vma->vm_mm,MM_ANONPAGES , nr_pages); +folio_add_new_anon_rmap (folio, vma, addr);folio_add_lru_vma (folio, vma); + +for (i =0 ; i < nr_pages; i++) { + entry =mk_pte (folio_page (folio, i), vma->vm_page_prot); + entry =pte_sw_mkyoung (entry); +if (vma->vm_flags &VM_WRITE ) + entry =pte_mkwrite (pte_mkdirty (entry)); setpte: -if (uffd_wp) - entry =pte_mkuffd_wp (entry); -set_pte_at (vma->vm_mm, vmf->address, vmf->pte, entry); +if (uffd_wp) + entry =pte_mkuffd_wp (entry);
mm/memory.c static vm_fault_t do_anonymous_page(struct vm_fault *vmf) · line 4288
Quoted 10 lines
mm/memory.c +4
-2+set_pte_at (vma->vm_mm, addr +PAGE_SIZE * i, vmf->pte + i, entry); -/* No need to invalidate - it was non-present before */ -update_mmu_cache (vma, vmf->address, vmf->pte); +/* No need to invalidate - it was non-present before */ +update_mmu_cache (vma, addr +PAGE_SIZE * i, vmf->pte + i); + } unlock:if (vmf->pte)pte_unmap_unlock (vmf->pte, vmf->ptl);
The rest looks good to me.
Re: [PATCH v4 2/5] mm: LARGE_ANON_FOLIO for improved performance
Quoted 142 lines · Yu Zhao
Quoted 129 lines · Ryan Roberts
Introduce
LARGE_ANON_FOLIOfeature, which allows anonymous memory to be allocated in large folios of a determined order. All pages of the large folio are pte-mapped during the same page fault, significantly reducing the number of page faults. The number of per-page operations (e.g. ref counting, rmap management lru list management) are also significantly reduced since those ops now become per-folio.The new behaviour is hidden behind the new
LARGE_ANON_FOLIOKconfig, which defaults to disabled for now; The long term aim is for this to defaut to enabled, but there are some risks around internal fragmentation that need to be better understood first.When enabled, the folio order is determined as such: For a vma, process or system that has explicitly disabled THP, we continue to allocate order-0. THP is most likely disabled to avoid any possible internal fragmentation so we honour that request.
Otherwise, the return value of
arch_wants_pte_order()is used. For vmas that have not explicitly opted-in to use transparent hugepages (e.g. where thp=madvise and the vma does not haveMADV_HUGEPAGE), thenarch_wants_pte_order()is limited to 64K (orPAGE_SIZE, whichever is bigger). This allows for a performance boost without requiring any explicit opt-in from the workload while limitting internal fragmentation.If the preferred order can't be used (e.g. because the folio would breach the bounds of the vma, or because ptes in the region are already mapped) then we fall back to a suitable lower order; first
PAGE_ALLOC_COSTLY_ORDER, then order-0.
arch_wants_pte_order()can be overridden by the architecture if desired. Some architectures (e.g. arm64) can coalsece TLB entries if a contiguous set of ptes map physically contigious, naturally aligned memory, so this mechanism allows the architecture to optimize as required.Here we add the default implementation of
arch_wants_pte_order(), used when the architecture does not define it, which returns -1, implying that the HW has no preference. In this case, mm will choose it's own default order.Signed-off-by Ryan Roberts <[email protected]>--- include/linux/pgtable.h | 13 ++++ mm/Kconfig | 10 +++ mm/memory.c | 166 ++++++++++++++++++++++++++++++++++++---- 3 files changed, 172 insertions(+), 17 deletions(-)include/linux/pgtable.h +13
-0@@ -313,6 +313,19 @@ static inline bool arch_has_hw_pte_young(void) }#endif +#ifndef arch_wants_pte_order +/* +* Returns preferred folio order for pte-mapped memory. Must be in range [0, +* PMD_SHIFT-PAGE_SHIFT) and must not be order-1 since THP requires large folios +* to be at least order-2. Negative value implies that the HW has no preference +* and mm will choose it's own default order. +*/ +static inline int arch_wants_pte_order (void ) +{ +return -1 ; +} +#endif +#ifndef __HAVE_ARCH_PTEP_GET_AND_CLEAR static inline pte_t ptep_get_and_clear(struct mm_struct *mm, unsigned long address,mm/Kconfig +10
-0@@ -1238,4 +1238,14 @@ config LOCK_MM_AND_FIND_VMA source "mm/damon/Kconfig" +config LARGE_ANON_FOLIO + bool "Allocate large folios for anonymous memory" + depends on TRANSPARENT_HUGEPAGE + default n + help + Use large (bigger than order-0) folios to back anonymous memory where + possible, even for pte-mapped memory. This reduces the number of page + faults, as well as other per-page overheads to improve performance for + many workloads. + endmenumm/memory.c +30
-0@@ -4050,6 +4050,127 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)return ret; } +static bool vmf_pte_range_changed (struct vm_fault *vmf,int nr_pages) +{ +int i; + +if (nr_pages ==1 ) +return vmf_pte_changed (vmf); + +for (i =0 ; i < nr_pages; i++) { +if (!pte_none (ptep_get_lockless (vmf->pte + i))) +return true; + } + +return false; +} + +#ifdef CONFIG_LARGE_ANON_FOLIO +#define ANON_FOLIO_MAX_ORDER_UNHINTED \ + (ilog2(max_t(unsigned long, SZ_64K, PAGE_SIZE)) - PAGE_SHIFT) + +static int anon_folio_order (struct vm_area_struct *vma) +{ +int order; + +/* +* If THP is explicitly disabled for either the vma, the process or the +* system, then this is very likely intended to limit internal +* fragmentation; in this case, don't attempt to allocate a large +* anonymous folio. +* +* Else, if the vma is eligible for thp, allocate a large folio of the mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 4092
Quoted 10 lines
mm/memory.c +10
-0+* size preferred by the arch. Or if the arch requested a very small +* size or didn't request a size, then use PAGE_ALLOC_COSTLY_ORDER, +* which still meets the arch's requirements but means we still take +* advantage of SW optimizations (e.g. fewer page faults). +* +* Finally if thp is enabled but the vma isn't eligible, take the +* arch-preferred size and limit it to ANON_FOLIO_MAX_ORDER_UNHINTED. +* This ensures workloads that have not explicitly opted-in take benefit +* while capping the potential for internal fragmentation. +*/
Quoted 18 lines
What empirical evidence is
SZ_64Kbased on?
What workloads would benefit from it?
How much would they benefit from it?
Would they benefit more or less from different values? How much internal fragmentation would it cause?
What cost function was used to arrive at the conclusion that its benefits outweigh its costs?Quoted 10 lines
mm/memory.c +10
-0+if ((vma->vm_flags &VM_NOHUGEPAGE ) || +test_bit (MMF_DISABLE_THP , &vma->vm_mm->flags) || + !hugepage_flags_enabled ()) + order =0 ; +else { + order =max (arch_wants_pte_order (),PAGE_ALLOC_COSTLY_ORDER ); + +if (!hugepage_vma_check (vma, vma->vm_flags, false, true, true)) + order =min (order,ANON_FOLIO_MAX_ORDER_UNHINTED ); + }
I'm a bit surprised to see the above: why can we overload existing ABIs? I don't think we can. Assuming we could, you would have to update Documentation/admin-guide/mm/transhuge.rst in the same patchset, and the man page for madvise() in a separate patch.
Most importantly, existing userspace programs that don't work well with THPs won't be able to use (try) large folios either -- this is a big no no.
Quoted 161 lines
mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 4108
Quoted 5 lines
mm/memory.c +5
-0+ +return order; +} + +static int alloc_anon_folio (struct vm_fault *vmf,struct folio **folio)static struct folio *alloc_anon_folio(struct
vm_fault*vmf)and use
ERR_PTR()and its friends.Quoted 139 lines
mm/memory.c +89
-15+{ +int i; +gfp_t gfp; +pte_t *pte; +unsigned long addr; +struct vm_area_struct *vma = vmf->vma; +int prefer =anon_folio_order (vma); +int orders[] = { + prefer, + prefer >PAGE_ALLOC_COSTLY_ORDER ?PAGE_ALLOC_COSTLY_ORDER :0 , +0 , + }; + + *folio =NULL ; + +if (vmf_orig_pte_uffd_wp (vmf)) + goto fallback; + +for (i =0 ; orders[i]; i++) { + addr =ALIGN_DOWN (vmf->address,PAGE_SIZE << orders[i]); +if (addr >= vma->vm_start && + addr + (PAGE_SIZE << orders[i]) <= vma->vm_end) +break ; + } + +if (!orders[i]) + goto fallback; + + pte =pte_offset_map (vmf->pmd, vmf->address &PMD_MASK ); +if (!pte) +return -EAGAIN ; + +for (; orders[i]; i++) { + addr =ALIGN_DOWN (vmf->address,PAGE_SIZE << orders[i]); + vmf->pte = pte +pte_index (addr); +if (!vmf_pte_range_changed (vmf,1 << orders[i])) +break ; + } + + vmf->pte =NULL ; +pte_unmap (pte); + + gfp =vma_thp_gfp_mask (vma); + +for (; orders[i]; i++) { + addr =ALIGN_DOWN (vmf->address,PAGE_SIZE << orders[i]); + *folio =vma_alloc_folio (gfp, orders[i], vma, addr, true); +if (*folio) { +clear_huge_page (&(*folio)->page, addr,1 << orders[i]); +return 0 ; + } + } + +fallback: + *folio =vma_alloc_zeroed_movable_folio (vma, vmf->address); +return *folio ?0 : -ENOMEM ; +} +#else +static inline int alloc_anon_folio (struct vm_fault *vmf,struct folio **folio) +{ + *folio =vma_alloc_zeroed_movable_folio (vmf->vma, vmf->address); +return *folio ?0 : -ENOMEM ; +} +#endif +/* * We enter with non-exclusive mmap_lock (to exclude vma changes, * but allow concurrent faults), and pte mapped but not yet locked. @@ -4057,6 +4178,9 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)*/ static vm_fault_t do_anonymous_page (struct vm_fault *vmf) { +int i =0 ; +int nr_pages =1 ; +unsigned long addr = vmf->address;bool uffd_wp =vmf_orig_pte_uffd_wp (vmf);struct vm_area_struct *vma = vmf->vma;struct folio *folio;@@ -4101,10 +4225,15 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)/* Allocate our own private page. */ if (unlikely (anon_vma_prepare (vma))) goto oom; - folio =vma_alloc_zeroed_movable_folio (vma, vmf->address); + ret =alloc_anon_folio (vmf, &folio); +if (unlikely (ret == -EAGAIN )) +return 0 ;if (!folio) goto oom; + nr_pages =folio_nr_pages (folio); + addr =ALIGN_DOWN (vmf->address, nr_pages *PAGE_SIZE ); +if (mem_cgroup_charge (folio, vma->vm_mm,GFP_KERNEL )) goto oom_free_page;folio_throttle_swaprate (folio,GFP_KERNEL );@@ -4116,17 +4245,12 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)*/ __folio_mark_uptodate (folio); - entry =mk_pte (&folio->page, vma->vm_page_prot); - entry =pte_sw_mkyoung (entry); -if (vma->vm_flags &VM_WRITE ) - entry =pte_mkwrite (pte_mkdirty (entry)); - - vmf->pte =pte_offset_map_lock (vma->vm_mm, vmf->pmd, vmf->address, - &vmf->ptl); + vmf->pte =pte_offset_map_lock (vma->vm_mm, vmf->pmd, addr, &vmf->ptl);if (!vmf->pte) goto release; -if (vmf_pte_changed (vmf)) { -update_mmu_tlb (vma, vmf->address, vmf->pte); +if (vmf_pte_range_changed (vmf, nr_pages)) { +for (i =0 ; i < nr_pages; i++) +update_mmu_tlb (vma, addr +PAGE_SIZE * i, vmf->pte + i); goto release; }@@ -4141,16 +4265,24 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)return handle_userfault (vmf,VM_UFFD_MISSING ); } -inc_mm_counter (vma->vm_mm,MM_ANONPAGES ); -folio_add_new_anon_rmap (folio, vma, vmf->address); +folio_ref_add (folio, nr_pages -1 ); +add_mm_counter (vma->vm_mm,MM_ANONPAGES , nr_pages); +folio_add_new_anon_rmap (folio, vma, addr);folio_add_lru_vma (folio, vma); + +for (i =0 ; i < nr_pages; i++) { + entry =mk_pte (folio_page (folio, i), vma->vm_page_prot); + entry =pte_sw_mkyoung (entry); +if (vma->vm_flags &VM_WRITE ) + entry =pte_mkwrite (pte_mkdirty (entry)); setpte: -if (uffd_wp) - entry =pte_mkuffd_wp (entry); -set_pte_at (vma->vm_mm, vmf->address, vmf->pte, entry); +if (uffd_wp) + entry =pte_mkuffd_wp (entry);mm/memory.c static vm_fault_t do_anonymous_page(struct vm_fault *vmf) · line 4288
Quoted 10 lines
mm/memory.c +4
-2+set_pte_at (vma->vm_mm, addr +PAGE_SIZE * i, vmf->pte + i, entry); -/* No need to invalidate - it was non-present before */ -update_mmu_cache (vma, vmf->address, vmf->pte); +/* No need to invalidate - it was non-present before */ +update_mmu_cache (vma, addr +PAGE_SIZE * i, vmf->pte + i); + } unlock:if (vmf->pte)pte_unmap_unlock (vmf->pte, vmf->ptl);The rest looks good to me.
Re: [PATCH v4 2/5] mm: LARGE_ANON_FOLIO for improved performance
Quoted 142 lines · Yu Zhao
Quoted 141 lines · Yu Zhao
Quoted 139 lines · Ryan Roberts
Introduce
LARGE_ANON_FOLIOfeature, which allows anonymous memory to be allocated in large folios of a determined order. All pages of the large folio are pte-mapped during the same page fault, significantly reducing the number of page faults. The number of per-page operations (e.g. ref counting, rmap management lru list management) are also significantly reduced since those ops now become per-folio.The new behaviour is hidden behind the new
LARGE_ANON_FOLIOKconfig, which defaults to disabled for now; The long term aim is for this to defaut to enabled, but there are some risks around internal fragmentation that need to be better understood first.When enabled, the folio order is determined as such: For a vma, process or system that has explicitly disabled THP, we continue to allocate order-0. THP is most likely disabled to avoid any possible internal fragmentation so we honour that request.
Otherwise, the return value of
arch_wants_pte_order()is used. For vmas that have not explicitly opted-in to use transparent hugepages (e.g. where thp=madvise and the vma does not haveMADV_HUGEPAGE), thenarch_wants_pte_order()is limited to 64K (orPAGE_SIZE, whichever is bigger). This allows for a performance boost without requiring any explicit opt-in from the workload while limitting internal fragmentation.If the preferred order can't be used (e.g. because the folio would breach the bounds of the vma, or because ptes in the region are already mapped) then we fall back to a suitable lower order; first
PAGE_ALLOC_COSTLY_ORDER, then order-0.
arch_wants_pte_order()can be overridden by the architecture if desired. Some architectures (e.g. arm64) can coalsece TLB entries if a contiguous set of ptes map physically contigious, naturally aligned memory, so this mechanism allows the architecture to optimize as required.Here we add the default implementation of
arch_wants_pte_order(), used when the architecture does not define it, which returns -1, implying that the HW has no preference. In this case, mm will choose it's own default order.Signed-off-by Ryan Roberts <[email protected]>--- include/linux/pgtable.h | 13 ++++ mm/Kconfig | 10 +++ mm/memory.c | 166 ++++++++++++++++++++++++++++++++++++---- 3 files changed, 172 insertions(+), 17 deletions(-)include/linux/pgtable.h +13
-0@@ -313,6 +313,19 @@ static inline bool arch_has_hw_pte_young(void) }#endif +#ifndef arch_wants_pte_order +/* +* Returns preferred folio order for pte-mapped memory. Must be in range [0, +* PMD_SHIFT-PAGE_SHIFT) and must not be order-1 since THP requires large folios +* to be at least order-2. Negative value implies that the HW has no preference +* and mm will choose it's own default order. +*/ +static inline int arch_wants_pte_order (void ) +{ +return -1 ; +} +#endif +#ifndef __HAVE_ARCH_PTEP_GET_AND_CLEAR static inline pte_t ptep_get_and_clear(struct mm_struct *mm, unsigned long address,mm/Kconfig +10
-0@@ -1238,4 +1238,14 @@ config LOCK_MM_AND_FIND_VMA source "mm/damon/Kconfig" +config LARGE_ANON_FOLIO + bool "Allocate large folios for anonymous memory" + depends on TRANSPARENT_HUGEPAGE + default n + help + Use large (bigger than order-0) folios to back anonymous memory where + possible, even for pte-mapped memory. This reduces the number of page + faults, as well as other per-page overheads to improve performance for + many workloads. + endmenumm/memory.c +40
-0@@ -4050,6 +4050,127 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)return ret; } +static bool vmf_pte_range_changed (struct vm_fault *vmf,int nr_pages) +{ +int i; + +if (nr_pages ==1 ) +return vmf_pte_changed (vmf); + +for (i =0 ; i < nr_pages; i++) { +if (!pte_none (ptep_get_lockless (vmf->pte + i))) +return true; + } + +return false; +} + +#ifdef CONFIG_LARGE_ANON_FOLIO +#define ANON_FOLIO_MAX_ORDER_UNHINTED \ + (ilog2(max_t(unsigned long, SZ_64K, PAGE_SIZE)) - PAGE_SHIFT) + +static int anon_folio_order (struct vm_area_struct *vma) +{ +int order; + +/* +* If THP is explicitly disabled for either the vma, the process or the +* system, then this is very likely intended to limit internal +* fragmentation; in this case, don't attempt to allocate a large +* anonymous folio. +* +* Else, if the vma is eligible for thp, allocate a large folio of the +* size preferred by the arch. Or if the arch requested a very small +* size or didn't request a size, then use PAGE_ALLOC_COSTLY_ORDER, +* which still meets the arch's requirements but means we still take +* advantage of SW optimizations (e.g. fewer page faults). +* +* Finally if thp is enabled but the vma isn't eligible, take the +* arch-preferred size and limit it to ANON_FOLIO_MAX_ORDER_UNHINTED. +* This ensures workloads that have not explicitly opted-in take benefit +* while capping the potential for internal fragmentation. +*/
Quoted 8 lines
Quoted 8 lines
What empirical evidence is
SZ_64Kbased on?
What workloads would benefit from it?
How much would they benefit from it?
Would they benefit more or less from different values? How much internal fragmentation would it cause?
What cost function was used to arrive at the conclusion that its benefits outweigh its costs?
Sorry this has taken a little while to reply to; I've been re-running my perf tests with the modern patches to recomfirm old data.
In terms of empirical evidence, I've run the kernel compilation benchmark (yes I know its a narrow use case, but I figure some data is better than no data), for all values of ANON_FOLIO_MAX_ORDER_UNHINTED {4k, 16k, 32k, 64k, 128k, 256k}.
I've run each test 15 times across 5 system reboots on Ampere Altra (arm64), with the kernel configured for 4K base pages - I could rerun for other base page sizes if we want to go further down this route.
I've captured run time and peak memory usage, and taken the mean. The stdev for the peak memory usage is big-ish, but I'm confident this still captures the central tendancy well:
| MAX_ORDER_UNHINTED | real-time | kern-time | user-time | peak memory | |:-------------------|------------:|------------:|------------:|:------------| | 4k | 0.0% | 0.0% | 0.0% | 0.0% | | 16k | -3.6% | -26.5% | -0.5% | -0.1% | | 32k | -4.8% | -37.4% | -0.6% | -0.1% | | 64k | -5.7% | -42.0% | -0.6% | -1.1% | | 128k | -5.6% | -42.1% | -0.7% | 1.4% | | 256k | -4.9% | -41.9% | -0.4% | 1.9% |
64K looks like the clear sweet spot to me.
I know you have argued for using a page order in the past, rather than a size in bytes. But my argument is that user space is mostly doing mmaps based on sizes independent of the base page size (an assumption!) and a system's memory is obviously a fixed quantity that doesn't it doesn't change with base page size. So it feels more natural to limit internal fragmentation based on an absolute size rather than a quantity of pages. Kyril have also suggested using absolute sizes in the past [1].
It's also worth mentioning that the file-backed memory "fault_around" mechanism chooses 64K.
If this approach really looks unacceptable, I have a couple of other ideas. But I personally favour the approach that is already in the patch.
1) Add a large/small flag to arch_wants_pte_order(). arm64, at least, actually has 2 mechanisms, HPA and contpte. Currently arm64 is always returning the contpte order, but with a flag, it could return contpte order for large, and HPA order for small. (I know we previously passed the vma and we didn't like that, and this is pretty similar). I still think the SW (core-mm) needs a way to sensibly limit internal fragmentation though, so personally I still think having an upper limit in this case is useful.
2) More radical: move to a per-vma auto-tuning solution, which looks at the fault pattern and maintains an allocation order in the VMA, which is modified based on fault pattern. e.g. When we get faults that occur immediately adjacent to the allocated range, we increase; when we get faults not connected to previously allocated pages we decrease. I think it's an interesting thing to look at, but certainly prefer that it's not part of an MVP implementation.
[1]
https://lore.kernel.org/linux-mm/[email protected]/
Quoted 12 lines
Quoted 11 lines
mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 4103
Quoted 11 lines
mm/memory.c +10
-0+if ((vma->vm_flags &VM_NOHUGEPAGE ) || +test_bit (MMF_DISABLE_THP , &vma->vm_mm->flags) || + !hugepage_flags_enabled ()) + order =0 ; +else { + order =max (arch_wants_pte_order (),PAGE_ALLOC_COSTLY_ORDER ); + +if (!hugepage_vma_check (vma, vma->vm_flags, false, true, true)) + order =min (order,ANON_FOLIO_MAX_ORDER_UNHINTED ); + }
Quoted 2 lines
I'm a bit surprised to see the above: why can we overload existing ABIs? I don't think we can.
I think this is all covered by the conversation with David against v2; see [2] and proceeding replies. Argument is that VM_NOHUGEPAGE (and friends) is really a request from user space to optimize for the least memory wastage possible and avoid populating ptes that have not been expressly requested.
[2]
https://lore.kernel.org/linux-mm/[email protected]/
Assuming we could, you would have to
Quoted 2 lines
update
Documentation/admin-guide/mm/transhuge.rstin the same patchset, and the man page formadvise()in a separate patch.
Yes, that's a fair point. Although transhuge.rst doesn't even mention MADV_NOHUGEPAGE today.
Quoted 4 lines
Most importantly, existing userspace programs that don't work well with THPs won't be able to use (try) large folios either -- this is a big no no.
I think we need some comments from David here. As mentioned I've added this tie-in based on his (strong) recommendation.
Quoted 11 lines
Quoted 11 lines
mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 4108
Quoted 8 lines
mm/memory.c +5
-0+ +return order; +} + +static int alloc_anon_folio (struct vm_fault *vmf,struct folio **folio)static struct folio *alloc_anon_folio(struct
vm_fault*vmf)
Quoted 1 line
Quoted 1 line
and use
ERR_PTR()and its friends.
Yes, agreed. I'll change this for the next version.
Quoted 142 lines
Quoted 142 lines
Quoted 142 lines
mm/memory.c +90
-15+{ +int i; +gfp_t gfp; +pte_t *pte; +unsigned long addr; +struct vm_area_struct *vma = vmf->vma; +int prefer =anon_folio_order (vma); +int orders[] = { + prefer, + prefer >PAGE_ALLOC_COSTLY_ORDER ?PAGE_ALLOC_COSTLY_ORDER :0 , +0 , + }; + + *folio =NULL ; + +if (vmf_orig_pte_uffd_wp (vmf)) + goto fallback; + +for (i =0 ; orders[i]; i++) { + addr =ALIGN_DOWN (vmf->address,PAGE_SIZE << orders[i]); +if (addr >= vma->vm_start && + addr + (PAGE_SIZE << orders[i]) <= vma->vm_end) +break ; + } + +if (!orders[i]) + goto fallback; + + pte =pte_offset_map (vmf->pmd, vmf->address &PMD_MASK ); +if (!pte) +return -EAGAIN ; + +for (; orders[i]; i++) { + addr =ALIGN_DOWN (vmf->address,PAGE_SIZE << orders[i]); + vmf->pte = pte +pte_index (addr); +if (!vmf_pte_range_changed (vmf,1 << orders[i])) +break ; + } + + vmf->pte =NULL ; +pte_unmap (pte); + + gfp =vma_thp_gfp_mask (vma); + +for (; orders[i]; i++) { + addr =ALIGN_DOWN (vmf->address,PAGE_SIZE << orders[i]); + *folio =vma_alloc_folio (gfp, orders[i], vma, addr, true); +if (*folio) { +clear_huge_page (&(*folio)->page, addr,1 << orders[i]); +return 0 ; + } + } + +fallback: + *folio =vma_alloc_zeroed_movable_folio (vma, vmf->address); +return *folio ?0 : -ENOMEM ; +} +#else +static inline int alloc_anon_folio (struct vm_fault *vmf,struct folio **folio) +{ + *folio =vma_alloc_zeroed_movable_folio (vmf->vma, vmf->address); +return *folio ?0 : -ENOMEM ; +} +#endif +/* * We enter with non-exclusive mmap_lock (to exclude vma changes, * but allow concurrent faults), and pte mapped but not yet locked. @@ -4057,6 +4178,9 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)*/ static vm_fault_t do_anonymous_page (struct vm_fault *vmf) { +int i =0 ; +int nr_pages =1 ; +unsigned long addr = vmf->address;bool uffd_wp =vmf_orig_pte_uffd_wp (vmf);struct vm_area_struct *vma = vmf->vma;struct folio *folio;@@ -4101,10 +4225,15 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)/* Allocate our own private page. */ if (unlikely (anon_vma_prepare (vma))) goto oom; - folio =vma_alloc_zeroed_movable_folio (vma, vmf->address); + ret =alloc_anon_folio (vmf, &folio); +if (unlikely (ret == -EAGAIN )) +return 0 ;if (!folio) goto oom; + nr_pages =folio_nr_pages (folio); + addr =ALIGN_DOWN (vmf->address, nr_pages *PAGE_SIZE ); +if (mem_cgroup_charge (folio, vma->vm_mm,GFP_KERNEL )) goto oom_free_page;folio_throttle_swaprate (folio,GFP_KERNEL );@@ -4116,17 +4245,12 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)*/ __folio_mark_uptodate (folio); - entry =mk_pte (&folio->page, vma->vm_page_prot); - entry =pte_sw_mkyoung (entry); -if (vma->vm_flags &VM_WRITE ) - entry =pte_mkwrite (pte_mkdirty (entry)); - - vmf->pte =pte_offset_map_lock (vma->vm_mm, vmf->pmd, vmf->address, - &vmf->ptl); + vmf->pte =pte_offset_map_lock (vma->vm_mm, vmf->pmd, addr, &vmf->ptl);if (!vmf->pte) goto release; -if (vmf_pte_changed (vmf)) { -update_mmu_tlb (vma, vmf->address, vmf->pte); +if (vmf_pte_range_changed (vmf, nr_pages)) { +for (i =0 ; i < nr_pages; i++) +update_mmu_tlb (vma, addr +PAGE_SIZE * i, vmf->pte + i); goto release; }@@ -4141,16 +4265,24 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)return handle_userfault (vmf,VM_UFFD_MISSING ); } -inc_mm_counter (vma->vm_mm,MM_ANONPAGES ); -folio_add_new_anon_rmap (folio, vma, vmf->address); +folio_ref_add (folio, nr_pages -1 ); +add_mm_counter (vma->vm_mm,MM_ANONPAGES , nr_pages); +folio_add_new_anon_rmap (folio, vma, addr);folio_add_lru_vma (folio, vma); + +for (i =0 ; i < nr_pages; i++) { + entry =mk_pte (folio_page (folio, i), vma->vm_page_prot); + entry =pte_sw_mkyoung (entry); +if (vma->vm_flags &VM_WRITE ) + entry =pte_mkwrite (pte_mkdirty (entry)); setpte: -if (uffd_wp) - entry =pte_mkuffd_wp (entry); -set_pte_at (vma->vm_mm, vmf->address, vmf->pte, entry); +if (uffd_wp) + entry =pte_mkuffd_wp (entry); +set_pte_at (vma->vm_mm, addr +PAGE_SIZE * i, vmf->pte + i, entry);
Quoted 10 lines
Quoted 10 lines
mm/memory.c static vm_fault_t do_anonymous_page(struct vm_fault *vmf) · line 4288
Quoted 8 lines
mm/memory.c +3
-2-/* No need to invalidate - it was non-present before */ -update_mmu_cache (vma, vmf->address, vmf->pte); +/* No need to invalidate - it was non-present before */ +update_mmu_cache (vma, addr +PAGE_SIZE * i, vmf->pte + i); + } unlock:if (vmf->pte)pte_unmap_unlock (vmf->pte, vmf->ptl);The rest looks good to me.
Thanks, as always, for the detailed review and feedback!
Thanks,
Ryan
Re: [PATCH v4 2/5] mm: LARGE_ANON_FOLIO for improved performance
Quoted 153 lines · Ryan Roberts
Quoted 142 lines · Yu Zhao
Quoted 141 lines · Yu Zhao
Quoted 139 lines · Ryan Roberts
Introduce
LARGE_ANON_FOLIOfeature, which allows anonymous memory to be allocated in large folios of a determined order. All pages of the large folio are pte-mapped during the same page fault, significantly reducing the number of page faults. The number of per-page operations (e.g. ref counting, rmap management lru list management) are also significantly reduced since those ops now become per-folio.The new behaviour is hidden behind the new
LARGE_ANON_FOLIOKconfig, which defaults to disabled for now; The long term aim is for this to defaut to enabled, but there are some risks around internal fragmentation that need to be better understood first.When enabled, the folio order is determined as such: For a vma, process or system that has explicitly disabled THP, we continue to allocate order-0. THP is most likely disabled to avoid any possible internal fragmentation so we honour that request.
Otherwise, the return value of
arch_wants_pte_order()is used. For vmas that have not explicitly opted-in to use transparent hugepages (e.g. where thp=madvise and the vma does not haveMADV_HUGEPAGE), thenarch_wants_pte_order()is limited to 64K (orPAGE_SIZE, whichever is bigger). This allows for a performance boost without requiring any explicit opt-in from the workload while limitting internal fragmentation.If the preferred order can't be used (e.g. because the folio would breach the bounds of the vma, or because ptes in the region are already mapped) then we fall back to a suitable lower order; first
PAGE_ALLOC_COSTLY_ORDER, then order-0.
arch_wants_pte_order()can be overridden by the architecture if desired. Some architectures (e.g. arm64) can coalsece TLB entries if a contiguous set of ptes map physically contigious, naturally aligned memory, so this mechanism allows the architecture to optimize as required.Here we add the default implementation of
arch_wants_pte_order(), used when the architecture does not define it, which returns -1, implying that the HW has no preference. In this case, mm will choose it's own default order.Signed-off-by Ryan Roberts <[email protected]>--- include/linux/pgtable.h | 13 ++++ mm/Kconfig | 10 +++ mm/memory.c | 166 ++++++++++++++++++++++++++++++++++++---- 3 files changed, 172 insertions(+), 17 deletions(-)include/linux/pgtable.h +13
-0@@ -313,6 +313,19 @@ static inline bool arch_has_hw_pte_young(void) }#endif +#ifndef arch_wants_pte_order +/* +* Returns preferred folio order for pte-mapped memory. Must be in range [0, +* PMD_SHIFT-PAGE_SHIFT) and must not be order-1 since THP requires large folios +* to be at least order-2. Negative value implies that the HW has no preference +* and mm will choose it's own default order. +*/ +static inline int arch_wants_pte_order (void ) +{ +return -1 ; +} +#endif +#ifndef __HAVE_ARCH_PTEP_GET_AND_CLEAR static inline pte_t ptep_get_and_clear(struct mm_struct *mm, unsigned long address,mm/Kconfig +10
-0@@ -1238,4 +1238,14 @@ config LOCK_MM_AND_FIND_VMA source "mm/damon/Kconfig" +config LARGE_ANON_FOLIO + bool "Allocate large folios for anonymous memory" + depends on TRANSPARENT_HUGEPAGE + default n + help + Use large (bigger than order-0) folios to back anonymous memory where + possible, even for pte-mapped memory. This reduces the number of page + faults, as well as other per-page overheads to improve performance for + many workloads. + endmenumm/memory.c +40
-0@@ -4050,6 +4050,127 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)return ret; } +static bool vmf_pte_range_changed (struct vm_fault *vmf,int nr_pages) +{ +int i; + +if (nr_pages ==1 ) +return vmf_pte_changed (vmf); + +for (i =0 ; i < nr_pages; i++) { +if (!pte_none (ptep_get_lockless (vmf->pte + i))) +return true; + } + +return false; +} + +#ifdef CONFIG_LARGE_ANON_FOLIO +#define ANON_FOLIO_MAX_ORDER_UNHINTED \ + (ilog2(max_t(unsigned long, SZ_64K, PAGE_SIZE)) - PAGE_SHIFT) + +static int anon_folio_order (struct vm_area_struct *vma) +{ +int order; + +/* +* If THP is explicitly disabled for either the vma, the process or the +* system, then this is very likely intended to limit internal +* fragmentation; in this case, don't attempt to allocate a large +* anonymous folio. +* +* Else, if the vma is eligible for thp, allocate a large folio of the +* size preferred by the arch. Or if the arch requested a very small +* size or didn't request a size, then use PAGE_ALLOC_COSTLY_ORDER, +* which still meets the arch's requirements but means we still take +* advantage of SW optimizations (e.g. fewer page faults). +* +* Finally if thp is enabled but the vma isn't eligible, take the +* arch-preferred size and limit it to ANON_FOLIO_MAX_ORDER_UNHINTED. +* This ensures workloads that have not explicitly opted-in take benefit +* while capping the potential for internal fragmentation. +*/ Quoted 8 lines
Quoted 8 lines
What empirical evidence is
SZ_64Kbased on?
What workloads would benefit from it?
How much would they benefit from it?
Would they benefit more or less from different values? How much internal fragmentation would it cause?
What cost function was used to arrive at the conclusion that its benefits outweigh its costs?
Quoted 2 lines
Sorry this has taken a little while to reply to; I've been re-running my perf tests with the modern patches to recomfirm old data.
Thanks for the data!
Quoted 5 lines
In terms of empirical evidence, I've run the kernel compilation benchmark (yes I know its a narrow use case, but I figure some data is better than no data), for all values of
ANON_FOLIO_MAX_ORDER_UNHINTED{4k, 16k, 32k, 64k, 128k, 256k}.I've run each test 15 times across 5 system reboots on Ampere Altra (arm64),
What about x86 and ppc? Do we expect they might perform similarly wrt different page sizes?
Quoted 16 lines
with the kernel configured for 4K base pages - I could rerun for other base page sizes if we want to go further down this route.
I've captured run time and peak memory usage, and taken the mean. The stdev for the peak memory usage is big-ish, but I'm confident this still captures the central tendancy well:
| MAX_ORDER_UNHINTED | real-time | kern-time | user-time | peak memory | |:-------------------|------------:|------------:|------------:|:------------| | 4k | 0.0% | 0.0% | 0.0% | 0.0% | | 16k | -3.6% | -26.5% | -0.5% | -0.1% | | 32k | -4.8% | -37.4% | -0.6% | -0.1% | | 64k | -5.7% | -42.0% | -0.6% | -1.1% | | 128k | -5.6% | -42.1% | -0.7% | 1.4% | | 256k | -4.9% | -41.9% | -0.4% | 1.9% |
Quoted 1 line
64K looks like the clear sweet spot to me.
Were the tests done under memory pressure? I agree 64KB might be a reasonable value, but I don't think we can or need to make a conclusion at this point: there are still pending questions from my list.
Just to double check: we only need ANON_FOLIO_MAX_ORDER_UNHINTED because of hugepage_vma_check(), is it correct?
Quoted 10 lines
I know you have argued for using a page order in the past, rather than a size in bytes. But my argument is that user space is mostly doing mmaps based on sizes independent of the base page size (an assumption!) and a system's memory is obviously a fixed quantity that doesn't it doesn't change with base page size. So it feels more natural to limit internal fragmentation based on an absolute size rather than a quantity of pages. Kyril have also suggested using absolute sizes in the past [1].
It's also worth mentioning that the file-backed memory "
fault_around" mechanism chooses 64K.
This example actually is against your argument:
1. There have been multiple reports that fault around hurt performances and had to be disabled for some workloads over the years -- ANON_FOLIO_MAX_ORDER_UNHINTED is likely to cause regressions too.
2. Not only can fault around be disabled, its default value can be changed too -- this series can't do either.
3. Most importantly, fault around does not do high-order allocations -- this series does, and high-order allocations can be very difficult under memory pressure.
Quoted 2 lines
If this approach really looks unacceptable, I have a couple of other ideas. But I personally favour the approach that is already in the patch.
I understand. If the answer to my question above is yes, then let's take a step back and figure out whether overloading existing ABIs is acceptable or not. Does this sound good to you?
Quoted 34 lines
1) Add a large/small flag to
arch_wants_pte_order(). arm64, at least, actually has 2 mechanisms, HPA and contpte. Currently arm64 is always returning the contpte order, but with a flag, it could return contpte order for large, and HPA order for small. (I know we previously passed the vma and we didn't like that, and this is pretty similar). I still think the SW (core-mm) needs a way to sensibly limit internal fragmentation though, so personally I still think having an upper limit in this case is useful.2) More radical: move to a per-vma auto-tuning solution, which looks at the fault pattern and maintains an allocation order in the VMA, which is modified based on fault pattern. e.g. When we get faults that occur immediately adjacent to the allocated range, we increase; when we get faults not connected to previously allocated pages we decrease. I think it's an interesting thing to look at, but certainly prefer that it's not part of an MVP implementation.
[1]
https://lore.kernel.org/linux-mm/[email protected]/Quoted 12 lines
Quoted 11 lines
mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 4103
Quoted 11 lines
mm/memory.c +10
-0+if ((vma->vm_flags &VM_NOHUGEPAGE ) || +test_bit (MMF_DISABLE_THP , &vma->vm_mm->flags) || + !hugepage_flags_enabled ()) + order =0 ; +else { + order =max (arch_wants_pte_order (),PAGE_ALLOC_COSTLY_ORDER ); + +if (!hugepage_vma_check (vma, vma->vm_flags, false, true, true)) + order =min (order,ANON_FOLIO_MAX_ORDER_UNHINTED ); + }Quoted 2 lines
I'm a bit surprised to see the above: why can we overload existing ABIs? I don't think we can.
Quoted 7 lines
I think this is all covered by the conversation with David against v2; see [2] and proceeding replies. Argument is that
VM_NOHUGEPAGE(and friends) is really a request from user space to optimize for the least memory wastage possible and avoid populating ptes that have not been expressly requested.
Thanks for the info.
I think there might be a misunderstanding here.
David, can you please clarify whether you suggested we overland (change the semantics) of existing ABIs?
This sounds like a big red flag to me. If that's really what you suggest, can you shed some light on why this is acceptable to existing userspace at all?
Thanks.
Re: [PATCH v4 2/5] mm: LARGE_ANON_FOLIO for improved performance
Quoted 159 lines · Yu Zhao
Quoted 153 lines · Ryan Roberts
Quoted 142 lines · Yu Zhao
Quoted 141 lines · Yu Zhao
Quoted 139 lines · Ryan Roberts
Introduce
LARGE_ANON_FOLIOfeature, which allows anonymous memory to be allocated in large folios of a determined order. All pages of the large folio are pte-mapped during the same page fault, significantly reducing the number of page faults. The number of per-page operations (e.g. ref counting, rmap management lru list management) are also significantly reduced since those ops now become per-folio.The new behaviour is hidden behind the new
LARGE_ANON_FOLIOKconfig, which defaults to disabled for now; The long term aim is for this to defaut to enabled, but there are some risks around internal fragmentation that need to be better understood first.When enabled, the folio order is determined as such: For a vma, process or system that has explicitly disabled THP, we continue to allocate order-0. THP is most likely disabled to avoid any possible internal fragmentation so we honour that request.
Otherwise, the return value of
arch_wants_pte_order()is used. For vmas that have not explicitly opted-in to use transparent hugepages (e.g. where thp=madvise and the vma does not haveMADV_HUGEPAGE), thenarch_wants_pte_order()is limited to 64K (orPAGE_SIZE, whichever is bigger). This allows for a performance boost without requiring any explicit opt-in from the workload while limitting internal fragmentation.If the preferred order can't be used (e.g. because the folio would breach the bounds of the vma, or because ptes in the region are already mapped) then we fall back to a suitable lower order; first
PAGE_ALLOC_COSTLY_ORDER, then order-0.
arch_wants_pte_order()can be overridden by the architecture if desired. Some architectures (e.g. arm64) can coalsece TLB entries if a contiguous set of ptes map physically contigious, naturally aligned memory, so this mechanism allows the architecture to optimize as required.Here we add the default implementation of
arch_wants_pte_order(), used when the architecture does not define it, which returns -1, implying that the HW has no preference. In this case, mm will choose it's own default order.Signed-off-by Ryan Roberts <[email protected]>--- include/linux/pgtable.h | 13 ++++ mm/Kconfig | 10 +++ mm/memory.c | 166 ++++++++++++++++++++++++++++++++++++---- 3 files changed, 172 insertions(+), 17 deletions(-)include/linux/pgtable.h +13
-0@@ -313,6 +313,19 @@ static inline bool arch_has_hw_pte_young(void) }#endif +#ifndef arch_wants_pte_order +/* +* Returns preferred folio order for pte-mapped memory. Must be in range [0, +* PMD_SHIFT-PAGE_SHIFT) and must not be order-1 since THP requires large folios +* to be at least order-2. Negative value implies that the HW has no preference +* and mm will choose it's own default order. +*/ +static inline int arch_wants_pte_order (void ) +{ +return -1 ; +} +#endif +#ifndef __HAVE_ARCH_PTEP_GET_AND_CLEAR static inline pte_t ptep_get_and_clear(struct mm_struct *mm, unsigned long address,mm/Kconfig +10
-0@@ -1238,4 +1238,14 @@ config LOCK_MM_AND_FIND_VMA source "mm/damon/Kconfig" +config LARGE_ANON_FOLIO + bool "Allocate large folios for anonymous memory" + depends on TRANSPARENT_HUGEPAGE + default n + help + Use large (bigger than order-0) folios to back anonymous memory where + possible, even for pte-mapped memory. This reduces the number of page + faults, as well as other per-page overheads to improve performance for + many workloads. + endmenumm/memory.c +40
-0@@ -4050,6 +4050,127 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)return ret; } +static bool vmf_pte_range_changed (struct vm_fault *vmf,int nr_pages) +{ +int i; + +if (nr_pages ==1 ) +return vmf_pte_changed (vmf); + +for (i =0 ; i < nr_pages; i++) { +if (!pte_none (ptep_get_lockless (vmf->pte + i))) +return true; + } + +return false; +} + +#ifdef CONFIG_LARGE_ANON_FOLIO +#define ANON_FOLIO_MAX_ORDER_UNHINTED \ + (ilog2(max_t(unsigned long, SZ_64K, PAGE_SIZE)) - PAGE_SHIFT) + +static int anon_folio_order (struct vm_area_struct *vma) +{ +int order; + +/* +* If THP is explicitly disabled for either the vma, the process or the +* system, then this is very likely intended to limit internal +* fragmentation; in this case, don't attempt to allocate a large +* anonymous folio. +* +* Else, if the vma is eligible for thp, allocate a large folio of the +* size preferred by the arch. Or if the arch requested a very small +* size or didn't request a size, then use PAGE_ALLOC_COSTLY_ORDER, +* which still meets the arch's requirements but means we still take +* advantage of SW optimizations (e.g. fewer page faults). +* +* Finally if thp is enabled but the vma isn't eligible, take the +* arch-preferred size and limit it to ANON_FOLIO_MAX_ORDER_UNHINTED. +* This ensures workloads that have not explicitly opted-in take benefit +* while capping the potential for internal fragmentation. +*/ Quoted 8 lines
Quoted 8 lines
What empirical evidence is
SZ_64Kbased on?
What workloads would benefit from it?
How much would they benefit from it?
Would they benefit more or less from different values? How much internal fragmentation would it cause?
What cost function was used to arrive at the conclusion that its benefits outweigh its costs?Quoted 2 lines
Sorry this has taken a little while to reply to; I've been re-running my perf tests with the modern patches to recomfirm old data.
Thanks for the data!
Quoted 8 lines
Quoted 5 lines
In terms of empirical evidence, I've run the kernel compilation benchmark (yes I know its a narrow use case, but I figure some data is better than no data), for all values of
ANON_FOLIO_MAX_ORDER_UNHINTED{4k, 16k, 32k, 64k, 128k, 256k}.I've run each test 15 times across 5 system reboots on Ampere Altra (arm64),
What about x86 and ppc? Do we expect they might perform similarly wrt different page sizes?
I will run the same text on Intel x86 platform.
Regards
Yin, Fengwei
Quoted 109 lines
Quoted 17 lines
with the kernel configured for 4K base pages - I could rerun for other base page sizes if we want to go further down this route.
I've captured run time and peak memory usage, and taken the mean. The stdev for the peak memory usage is big-ish, but I'm confident this still captures the central tendancy well:
| MAX_ORDER_UNHINTED | real-time | kern-time | user-time | peak memory | |:-------------------|------------:|------------:|------------:|:------------| | 4k | 0.0% | 0.0% | 0.0% | 0.0% | | 16k | -3.6% | -26.5% | -0.5% | -0.1% | | 32k | -4.8% | -37.4% | -0.6% | -0.1% | | 64k | -5.7% | -42.0% | -0.6% | -1.1% | | 128k | -5.6% | -42.1% | -0.7% | 1.4% | | 256k | -4.9% | -41.9% | -0.4% | 1.9% |Quoted 1 line
64K looks like the clear sweet spot to me.
Were the tests done under memory pressure? I agree 64KB might be a reasonable value, but I don't think we can or need to make a conclusion at this point: there are still pending questions from my list.
Just to double check: we only need
ANON_FOLIO_MAX_ORDER_UNHINTEDbecause ofhugepage_vma_check(), is it correct?Quoted 10 lines
I know you have argued for using a page order in the past, rather than a size in bytes. But my argument is that user space is mostly doing mmaps based on sizes independent of the base page size (an assumption!) and a system's memory is obviously a fixed quantity that doesn't it doesn't change with base page size. So it feels more natural to limit internal fragmentation based on an absolute size rather than a quantity of pages. Kyril have also suggested using absolute sizes in the past [1].
It's also worth mentioning that the file-backed memory "
fault_around" mechanism chooses 64K.This example actually is against your argument:
1. There have been multiple reports that fault around hurt performances and had to be disabled for some workloads over the years --ANON_FOLIO_MAX_ORDER_UNHINTEDis likely to cause regressions too.
2. Not only can fault around be disabled, its default value can be changed too -- this series can't do either.
3. Most importantly, fault around does not do high-order allocations -- this series does, and high-order allocations can be very difficult under memory pressure.Quoted 2 lines
If this approach really looks unacceptable, I have a couple of other ideas. But I personally favour the approach that is already in the patch.
I understand. If the answer to my question above is yes, then let's take a step back and figure out whether overloading existing ABIs is acceptable or not. Does this sound good to you?
Quoted 34 lines
1) Add a large/small flag to
arch_wants_pte_order(). arm64, at least, actually has 2 mechanisms, HPA and contpte. Currently arm64 is always returning the contpte order, but with a flag, it could return contpte order for large, and HPA order for small. (I know we previously passed the vma and we didn't like that, and this is pretty similar). I still think the SW (core-mm) needs a way to sensibly limit internal fragmentation though, so personally I still think having an upper limit in this case is useful.2) More radical: move to a per-vma auto-tuning solution, which looks at the fault pattern and maintains an allocation order in the VMA, which is modified based on fault pattern. e.g. When we get faults that occur immediately adjacent to the allocated range, we increase; when we get faults not connected to previously allocated pages we decrease. I think it's an interesting thing to look at, but certainly prefer that it's not part of an MVP implementation.
[1]
https://lore.kernel.org/linux-mm/[email protected]/Quoted 12 lines
Quoted 11 lines
mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 4103
Quoted 11 lines
mm/memory.c +10
-0+if ((vma->vm_flags &VM_NOHUGEPAGE ) || +test_bit (MMF_DISABLE_THP , &vma->vm_mm->flags) || + !hugepage_flags_enabled ()) + order =0 ; +else { + order =max (arch_wants_pte_order (),PAGE_ALLOC_COSTLY_ORDER ); + +if (!hugepage_vma_check (vma, vma->vm_flags, false, true, true)) + order =min (order,ANON_FOLIO_MAX_ORDER_UNHINTED ); + }Quoted 2 lines
I'm a bit surprised to see the above: why can we overload existing ABIs? I don't think we can.
Quoted 7 lines
I think this is all covered by the conversation with David against v2; see [2] and proceeding replies. Argument is that
VM_NOHUGEPAGE(and friends) is really a request from user space to optimize for the least memory wastage possible and avoid populating ptes that have not been expressly requested.Thanks for the info.
I think there might be a misunderstanding here.
David, can you please clarify whether you suggested we overland (change the semantics) of existing ABIs?
This sounds like a big red flag to me. If that's really what you suggest, can you shed some light on why this is acceptable to existing userspace at all?
Thanks.
Re: [PATCH v4 2/5] mm: LARGE_ANON_FOLIO for improved performance
In reply to Yu ZhaoQuoted 159 lines · Yu Zhao
Quoted 153 lines · Ryan Roberts
Quoted 142 lines · Yu Zhao
Quoted 141 lines · Yu Zhao
Quoted 139 lines · Ryan Roberts
Introduce
LARGE_ANON_FOLIOfeature, which allows anonymous memory to be allocated in large folios of a determined order. All pages of the large folio are pte-mapped during the same page fault, significantly reducing the number of page faults. The number of per-page operations (e.g. ref counting, rmap management lru list management) are also significantly reduced since those ops now become per-folio.The new behaviour is hidden behind the new
LARGE_ANON_FOLIOKconfig, which defaults to disabled for now; The long term aim is for this to defaut to enabled, but there are some risks around internal fragmentation that need to be better understood first.When enabled, the folio order is determined as such: For a vma, process or system that has explicitly disabled THP, we continue to allocate order-0. THP is most likely disabled to avoid any possible internal fragmentation so we honour that request.
Otherwise, the return value of
arch_wants_pte_order()is used. For vmas that have not explicitly opted-in to use transparent hugepages (e.g. where thp=madvise and the vma does not haveMADV_HUGEPAGE), thenarch_wants_pte_order()is limited to 64K (orPAGE_SIZE, whichever is bigger). This allows for a performance boost without requiring any explicit opt-in from the workload while limitting internal fragmentation.If the preferred order can't be used (e.g. because the folio would breach the bounds of the vma, or because ptes in the region are already mapped) then we fall back to a suitable lower order; first
PAGE_ALLOC_COSTLY_ORDER, then order-0.
arch_wants_pte_order()can be overridden by the architecture if desired. Some architectures (e.g. arm64) can coalsece TLB entries if a contiguous set of ptes map physically contigious, naturally aligned memory, so this mechanism allows the architecture to optimize as required.Here we add the default implementation of
arch_wants_pte_order(), used when the architecture does not define it, which returns -1, implying that the HW has no preference. In this case, mm will choose it's own default order.Signed-off-by Ryan Roberts <[email protected]>--- include/linux/pgtable.h | 13 ++++ mm/Kconfig | 10 +++ mm/memory.c | 166 ++++++++++++++++++++++++++++++++++++---- 3 files changed, 172 insertions(+), 17 deletions(-)include/linux/pgtable.h +13
-0@@ -313,6 +313,19 @@ static inline bool arch_has_hw_pte_young(void) }#endif +#ifndef arch_wants_pte_order +/* +* Returns preferred folio order for pte-mapped memory. Must be in range [0, +* PMD_SHIFT-PAGE_SHIFT) and must not be order-1 since THP requires large folios +* to be at least order-2. Negative value implies that the HW has no preference +* and mm will choose it's own default order. +*/ +static inline int arch_wants_pte_order (void ) +{ +return -1 ; +} +#endif +#ifndef __HAVE_ARCH_PTEP_GET_AND_CLEAR static inline pte_t ptep_get_and_clear(struct mm_struct *mm, unsigned long address,mm/Kconfig +10
-0@@ -1238,4 +1238,14 @@ config LOCK_MM_AND_FIND_VMA source "mm/damon/Kconfig" +config LARGE_ANON_FOLIO + bool "Allocate large folios for anonymous memory" + depends on TRANSPARENT_HUGEPAGE + default n + help + Use large (bigger than order-0) folios to back anonymous memory where + possible, even for pte-mapped memory. This reduces the number of page + faults, as well as other per-page overheads to improve performance for + many workloads. + endmenumm/memory.c +40
-0@@ -4050,6 +4050,127 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)return ret; } +static bool vmf_pte_range_changed (struct vm_fault *vmf,int nr_pages) +{ +int i; + +if (nr_pages ==1 ) +return vmf_pte_changed (vmf); + +for (i =0 ; i < nr_pages; i++) { +if (!pte_none (ptep_get_lockless (vmf->pte + i))) +return true; + } + +return false; +} + +#ifdef CONFIG_LARGE_ANON_FOLIO +#define ANON_FOLIO_MAX_ORDER_UNHINTED \ + (ilog2(max_t(unsigned long, SZ_64K, PAGE_SIZE)) - PAGE_SHIFT) + +static int anon_folio_order (struct vm_area_struct *vma) +{ +int order; + +/* +* If THP is explicitly disabled for either the vma, the process or the +* system, then this is very likely intended to limit internal +* fragmentation; in this case, don't attempt to allocate a large +* anonymous folio. +* +* Else, if the vma is eligible for thp, allocate a large folio of the +* size preferred by the arch. Or if the arch requested a very small +* size or didn't request a size, then use PAGE_ALLOC_COSTLY_ORDER, +* which still meets the arch's requirements but means we still take +* advantage of SW optimizations (e.g. fewer page faults). +* +* Finally if thp is enabled but the vma isn't eligible, take the +* arch-preferred size and limit it to ANON_FOLIO_MAX_ORDER_UNHINTED. +* This ensures workloads that have not explicitly opted-in take benefit +* while capping the potential for internal fragmentation. +*/ Quoted 8 lines
Quoted 8 lines
What empirical evidence is
SZ_64Kbased on?
What workloads would benefit from it?
How much would they benefit from it?
Would they benefit more or less from different values? How much internal fragmentation would it cause?
What cost function was used to arrive at the conclusion that its benefits outweigh its costs?Quoted 2 lines
Sorry this has taken a little while to reply to; I've been re-running my perf tests with the modern patches to recomfirm old data.
Thanks for the data!
Quoted 8 lines
Quoted 5 lines
In terms of empirical evidence, I've run the kernel compilation benchmark (yes I know its a narrow use case, but I figure some data is better than no data), for all values of
ANON_FOLIO_MAX_ORDER_UNHINTED{4k, 16k, 32k, 64k, 128k, 256k}.I've run each test 15 times across 5 system reboots on Ampere Altra (arm64),
What about x86 and ppc? Do we expect they might perform similarly wrt different page sizes?
It's my assumption that they should behave similarly, but I haven't actually tested it. Thanks to Yin Fengwei for the kind offer to run these tests on x86!
Yin: I have a test tool that will automate running this and gather up the results. Not sure if this is useful to you? I can share if you want? I also slightly modified the code to add a boot param to set the ANON_FOLIO_MAX_ORDER_UNHINTED value, and the test tool automatically injected the values.
Quoted 17 lines
Quoted 17 lines
with the kernel configured for 4K base pages - I could rerun for other base page sizes if we want to go further down this route.
I've captured run time and peak memory usage, and taken the mean. The stdev for the peak memory usage is big-ish, but I'm confident this still captures the central tendancy well:
| MAX_ORDER_UNHINTED | real-time | kern-time | user-time | peak memory | |:-------------------|------------:|------------:|------------:|:------------| | 4k | 0.0% | 0.0% | 0.0% | 0.0% | | 16k | -3.6% | -26.5% | -0.5% | -0.1% | | 32k | -4.8% | -37.4% | -0.6% | -0.1% | | 64k | -5.7% | -42.0% | -0.6% | -1.1% | | 128k | -5.6% | -42.1% | -0.7% | 1.4% | | 256k | -4.9% | -41.9% | -0.4% | 1.9% |
Quoted 3 lines
Quoted 1 line
64K looks like the clear sweet spot to me.
Were the tests done under memory pressure?
No. I have the capability to run these tests in a memcg with limited max memory though to force swap. I planned to do some sweeps increasing memory pressure, all for ANON_FOLIO_MAX_ORDER_UNHINTED=64k. Doing this for all values above will take too much time, but I could do them at a single value of max memory if that helps? I'm not sure how I would choose that single value though? (probably do the sweep for 64k then choose a sensible point on that graph?).
I agree 64KB might be a
Quoted 3 lines
reasonable value, but I don't think we can or need to make a conclusion at this point: there are still pending questions from my list.
You mean there are still pending questions from your list above, or you have others that you havent asked yet? If the former, I've answered all of the above to the best of my ability. My view is that this value is always going to be tuned empirically so its difficult to answer with absolutes. What can I do to improve confidence? If the latter, then please let me know your other questions.
Quoted 3 lines
Just to double check: we only need
ANON_FOLIO_MAX_ORDER_UNHINTEDbecause ofhugepage_vma_check(), is it correct?
tldr; yes, correct.
My problem is that for arm64 16k and 64k base page configs, the contpte size is 2M. It's my view that this is too big to allocate when it has not been explicitly asked for. And I think experience with THP (which is 2M for 4K systems today) demonstrates that. But I would still like to benefit from reduced SW overhead where possible (i.e. reduced page faults), and I would still like to use the contpte 2M mappings when the user has signalled that they can tolerate the potential internal fragmentation (MADV_HUGEPAGE).
In practical terms, ANON_FOLIO_MAX_ORDER_UNHINTED does not affect arm64/4K at all (because the contpte size is 64K) and it does not impact the other 4K base page arches, which don't currently implement arch_wants_pte_order() meaning they get the default PAGE_ALLOC_COSTLY_ORDER=3=32k.
Quoted 9 lines
Quoted 9 lines
I know you have argued for using a page order in the past, rather than a size in bytes. But my argument is that user space is mostly doing mmaps based on sizes independent of the base page size (an assumption!) and a system's memory is obviously a fixed quantity that doesn't it doesn't change with base page size. So it feels more natural to limit internal fragmentation based on an absolute size rather than a quantity of pages. Kyril have also suggested using absolute sizes in the past [1].
Quoted 6 lines
Quoted 2 lines
It's also worth mentioning that the file-backed memory "
fault_around" mechanism chooses 64K.This example actually is against your argument:
1. There have been multiple reports that fault around hurt
performances and had to be disabled for some workloads over the years> --ANON_FOLIO_MAX_ORDER_UNHINTEDis likely to cause regressions too.
I don't see how ANON_FOLIO_MAX_ORDER_UNHINTED can cause regressions; it's adding a limit making the behaviour of Large Anon Folios more similar to the old behaviour than it otherwise would be. Without it, we will be allocating 2M folios in some cases which would be much more likely to cause regression in unprepared apps IMHO.
Quoted 2 lines
2. Not only can fault around be disabled, its default value can be changed too -- this series can't do either.
I had a mechanism for that in the previous version, but discussion concluded that we should avoid adding the control for now and add it only if/when we have identified a workload that will benefit.
Quoted 3 lines
3. Most importantly, fault around does not do high-order allocations -- this series does, and high-order allocations can be very difficult under memory pressure.
But ANON_FOLIO_MAX_ORDER_UNHINTED *reduces* the order from what it otherwise would be. So I don't see how its making things worse?
Quoted 7 lines
Quoted 3 lines
If this approach really looks unacceptable, I have a couple of other ideas. But I personally favour the approach that is already in the patch.
I understand. If the answer to my question above is yes, then let's take a step back and figure out whether overloading existing ABIs is acceptable or not. Does this sound good to you?
Yes, good idea. Hopefully my explanation above (and all the previous threads) gives you a good idea for the problem as I see it, and how I think hooking the THP hints is helpful to the solution. If I've understood David's previuous remarks correctly, then this also aligns with David's opinions (David you could confirm/deny this please?). Could you explain your position that hooking these ABIs is a bad approach?
Quoted 55 lines
Quoted 35 lines
1) Add a large/small flag to
arch_wants_pte_order(). arm64, at least, actually has 2 mechanisms, HPA and contpte. Currently arm64 is always returning the contpte order, but with a flag, it could return contpte order for large, and HPA order for small. (I know we previously passed the vma and we didn't like that, and this is pretty similar). I still think the SW (core-mm) needs a way to sensibly limit internal fragmentation though, so personally I still think having an upper limit in this case is useful.2) More radical: move to a per-vma auto-tuning solution, which looks at the fault pattern and maintains an allocation order in the VMA, which is modified based on fault pattern. e.g. When we get faults that occur immediately adjacent to the allocated range, we increase; when we get faults not connected to previously allocated pages we decrease. I think it's an interesting thing to look at, but certainly prefer that it's not part of an MVP implementation.
[1]
https://lore.kernel.org/linux-mm/[email protected]/Quoted 12 lines
Quoted 11 lines
mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 4103
Quoted 11 lines
mm/memory.c +10
-0+if ((vma->vm_flags &VM_NOHUGEPAGE ) || +test_bit (MMF_DISABLE_THP , &vma->vm_mm->flags) || + !hugepage_flags_enabled ()) + order =0 ; +else { + order =max (arch_wants_pte_order (),PAGE_ALLOC_COSTLY_ORDER ); + +if (!hugepage_vma_check (vma, vma->vm_flags, false, true, true)) + order =min (order,ANON_FOLIO_MAX_ORDER_UNHINTED ); + }Quoted 2 lines
I'm a bit surprised to see the above: why can we overload existing ABIs? I don't think we can.
Quoted 7 lines
I think this is all covered by the conversation with David against v2; see [2] and proceeding replies. Argument is that
VM_NOHUGEPAGE(and friends) is really a request from user space to optimize for the least memory wastage possible and avoid populating ptes that have not been expressly requested.Thanks for the info.
I think there might be a misunderstanding here.
David, can you please clarify whether you suggested we overland (change the semantics) of existing ABIs?
This sounds like a big red flag to me. If that's really what you suggest, can you shed some light on why this is acceptable to existing userspace at all?
Thanks.
Re: [PATCH v4 2/5] mm: LARGE_ANON_FOLIO for improved performance
On 02/08/2023 09:02, Ryan Roberts wrote:
...
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
I've captured run time and peak memory usage, and taken the mean. The stdev for the peak memory usage is big-ish, but I'm confident this still captures the central tendancy well:
| MAX_ORDER_UNHINTED | real-time | kern-time | user-time | peak memory | |:-------------------|------------:|------------:|------------:|:------------| | 4k | 0.0% | 0.0% | 0.0% | 0.0% | | 16k | -3.6% | -26.5% | -0.5% | -0.1% | | 32k | -4.8% | -37.4% | -0.6% | -0.1% | | 64k | -5.7% | -42.0% | -0.6% | -1.1% | | 128k | -5.6% | -42.1% | -0.7% | 1.4% | | 256k | -4.9% | -41.9% | -0.4% | 1.9% |
Quoted 1 line
Quoted 1 line
Quoted 1 line
64K looks like the clear sweet spot to me.
I'm sorry about this; I've concluded that these tests are flawed. While I'm correctly setting the MAX_ORDER_UNHINTED value in each case, this is run against a 4K base page kernel, which means that it's arch_wants_pte_order() return value is order-4. So for MAX_ORDER_UNHINTED = {64k, 128k, 256k}, the actual order used is order-4 (=64K):
order = max(arch_wants_pte_order(), PAGE_ALLOC_COSTLY_ORDER); if (!hugepage_vma_check(vma, vma->vm_flags, false, true, true)) order = min(order, ANON_FOLIO_MAX_ORDER_UNHINTED);
So while I think we can conclude that the performance improves from 4k -> 64k, and the peak memory is about the same, we can't conclude that 64k is definely where performance gains peak or that peak memory increases after this.
The error bars on the memory consumption are fairly big.
I'll rework the tests so that I'm actually measuring what I was intending to measure and repost in due course.
Re: [PATCH v4 2/5] mm: LARGE_ANON_FOLIO for improved performance
In reply to Ryan RobertsQuoted 169 lines · Ryan Roberts
Quoted 159 lines · Yu Zhao
Quoted 153 lines · Ryan Roberts
Quoted 142 lines · Yu Zhao
Quoted 141 lines · Yu Zhao
Quoted 139 lines · Ryan Roberts
Introduce
LARGE_ANON_FOLIOfeature, which allows anonymous memory to be allocated in large folios of a determined order. All pages of the large folio are pte-mapped during the same page fault, significantly reducing the number of page faults. The number of per-page operations (e.g. ref counting, rmap management lru list management) are also significantly reduced since those ops now become per-folio.The new behaviour is hidden behind the new
LARGE_ANON_FOLIOKconfig, which defaults to disabled for now; The long term aim is for this to defaut to enabled, but there are some risks around internal fragmentation that need to be better understood first.When enabled, the folio order is determined as such: For a vma, process or system that has explicitly disabled THP, we continue to allocate order-0. THP is most likely disabled to avoid any possible internal fragmentation so we honour that request.
Otherwise, the return value of
arch_wants_pte_order()is used. For vmas that have not explicitly opted-in to use transparent hugepages (e.g. where thp=madvise and the vma does not haveMADV_HUGEPAGE), thenarch_wants_pte_order()is limited to 64K (orPAGE_SIZE, whichever is bigger). This allows for a performance boost without requiring any explicit opt-in from the workload while limitting internal fragmentation.If the preferred order can't be used (e.g. because the folio would breach the bounds of the vma, or because ptes in the region are already mapped) then we fall back to a suitable lower order; first
PAGE_ALLOC_COSTLY_ORDER, then order-0.
arch_wants_pte_order()can be overridden by the architecture if desired. Some architectures (e.g. arm64) can coalsece TLB entries if a contiguous set of ptes map physically contigious, naturally aligned memory, so this mechanism allows the architecture to optimize as required.Here we add the default implementation of
arch_wants_pte_order(), used when the architecture does not define it, which returns -1, implying that the HW has no preference. In this case, mm will choose it's own default order.Signed-off-by Ryan Roberts <[email protected]>--- include/linux/pgtable.h | 13 ++++ mm/Kconfig | 10 +++ mm/memory.c | 166 ++++++++++++++++++++++++++++++++++++---- 3 files changed, 172 insertions(+), 17 deletions(-)include/linux/pgtable.h +13
-0@@ -313,6 +313,19 @@ static inline bool arch_has_hw_pte_young(void) }#endif +#ifndef arch_wants_pte_order +/* +* Returns preferred folio order for pte-mapped memory. Must be in range [0, +* PMD_SHIFT-PAGE_SHIFT) and must not be order-1 since THP requires large folios +* to be at least order-2. Negative value implies that the HW has no preference +* and mm will choose it's own default order. +*/ +static inline int arch_wants_pte_order (void ) +{ +return -1 ; +} +#endif +#ifndef __HAVE_ARCH_PTEP_GET_AND_CLEAR static inline pte_t ptep_get_and_clear(struct mm_struct *mm, unsigned long address,mm/Kconfig +10
-0@@ -1238,4 +1238,14 @@ config LOCK_MM_AND_FIND_VMA source "mm/damon/Kconfig" +config LARGE_ANON_FOLIO + bool "Allocate large folios for anonymous memory" + depends on TRANSPARENT_HUGEPAGE + default n + help + Use large (bigger than order-0) folios to back anonymous memory where + possible, even for pte-mapped memory. This reduces the number of page + faults, as well as other per-page overheads to improve performance for + many workloads. + endmenumm/memory.c +40
-0@@ -4050,6 +4050,127 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)return ret; } +static bool vmf_pte_range_changed (struct vm_fault *vmf,int nr_pages) +{ +int i; + +if (nr_pages ==1 ) +return vmf_pte_changed (vmf); + +for (i =0 ; i < nr_pages; i++) { +if (!pte_none (ptep_get_lockless (vmf->pte + i))) +return true; + } + +return false; +} + +#ifdef CONFIG_LARGE_ANON_FOLIO +#define ANON_FOLIO_MAX_ORDER_UNHINTED \ + (ilog2(max_t(unsigned long, SZ_64K, PAGE_SIZE)) - PAGE_SHIFT) + +static int anon_folio_order (struct vm_area_struct *vma) +{ +int order; + +/* +* If THP is explicitly disabled for either the vma, the process or the +* system, then this is very likely intended to limit internal +* fragmentation; in this case, don't attempt to allocate a large +* anonymous folio. +* +* Else, if the vma is eligible for thp, allocate a large folio of the +* size preferred by the arch. Or if the arch requested a very small +* size or didn't request a size, then use PAGE_ALLOC_COSTLY_ORDER, +* which still meets the arch's requirements but means we still take +* advantage of SW optimizations (e.g. fewer page faults). +* +* Finally if thp is enabled but the vma isn't eligible, take the +* arch-preferred size and limit it to ANON_FOLIO_MAX_ORDER_UNHINTED. +* This ensures workloads that have not explicitly opted-in take benefit +* while capping the potential for internal fragmentation. +*/ Quoted 8 lines
Quoted 8 lines
What empirical evidence is
SZ_64Kbased on?
What workloads would benefit from it?
How much would they benefit from it?
Would they benefit more or less from different values? How much internal fragmentation would it cause?
What cost function was used to arrive at the conclusion that its benefits outweigh its costs?Quoted 2 lines
Sorry this has taken a little while to reply to; I've been re-running my perf tests with the modern patches to recomfirm old data.
Thanks for the data!
Quoted 8 lines
Quoted 5 lines
In terms of empirical evidence, I've run the kernel compilation benchmark (yes I know its a narrow use case, but I figure some data is better than no data), for all values of
ANON_FOLIO_MAX_ORDER_UNHINTED{4k, 16k, 32k, 64k, 128k, 256k}.I've run each test 15 times across 5 system reboots on Ampere Altra (arm64),
What about x86 and ppc? Do we expect they might perform similarly wrt different page sizes?
Quoted 8 lines
It's my assumption that they should behave similarly, but I haven't actually tested it. Thanks to Yin Fengwei for the kind offer to run these tests on x86!
Yin: I have a test tool that will automate running this and gather up the results. Not sure if this is useful to you? I can share if you want? I also slightly modified the code to add a boot param to set the
ANON_FOLIO_MAX_ORDER_UNHINTEDvalue, and the test tool automatically injected the values.
Not necessary. I started to run the test. I suppose I could share the test result tomorrow.
Regards
Yin, Fengwei
Quoted 166 lines
Quoted 18 lines
Quoted 18 lines
with the kernel configured for 4K base pages - I could rerun for other base page sizes if we want to go further down this route.
I've captured run time and peak memory usage, and taken the mean. The stdev for the peak memory usage is big-ish, but I'm confident this still captures the central tendancy well:
| MAX_ORDER_UNHINTED | real-time | kern-time | user-time | peak memory | |:-------------------|------------:|------------:|------------:|:------------| | 4k | 0.0% | 0.0% | 0.0% | 0.0% | | 16k | -3.6% | -26.5% | -0.5% | -0.1% | | 32k | -4.8% | -37.4% | -0.6% | -0.1% | | 64k | -5.7% | -42.0% | -0.6% | -1.1% | | 128k | -5.6% | -42.1% | -0.7% | 1.4% | | 256k | -4.9% | -41.9% | -0.4% | 1.9% |Quoted 3 lines
Quoted 1 line
64K looks like the clear sweet spot to me.
Were the tests done under memory pressure?
No. I have the capability to run these tests in a memcg with limited max memory though to force swap. I planned to do some sweeps increasing memory pressure, all for ANON_FOLIO_MAX_ORDER_UNHINTED=64k. Doing this for all values above will take too much time, but I could do them at a single value of max memory if that helps? I'm not sure how I would choose that single value though? (probably do the sweep for 64k then choose a sensible point on that graph?).
I agree 64KB might be a
Quoted 3 lines
reasonable value, but I don't think we can or need to make a conclusion at this point: there are still pending questions from my list.
You mean there are still pending questions from your list above, or you have others that you havent asked yet? If the former, I've answered all of the above to the best of my ability. My view is that this value is always going to be tuned empirically so its difficult to answer with absolutes. What can I do to improve confidence? If the latter, then please let me know your other questions.
Quoted 3 lines
Just to double check: we only need
ANON_FOLIO_MAX_ORDER_UNHINTEDbecause ofhugepage_vma_check(), is it correct?tldr; yes, correct.
My problem is that for arm64 16k and 64k base page configs, the contpte size is 2M. It's my view that this is too big to allocate when it has not been explicitly asked for. And I think experience with THP (which is 2M for 4K systems today) demonstrates that. But I would still like to benefit from reduced SW overhead where possible (i.e. reduced page faults), and I would still like to use the contpte 2M mappings when the user has signalled that they can tolerate the potential internal fragmentation (
MADV_HUGEPAGE).In practical terms,
ANON_FOLIO_MAX_ORDER_UNHINTEDdoes not affect arm64/4K at all (because the contpte size is 64K) and it does not impact the other 4K base page arches, which don't currently implementarch_wants_pte_order()meaning they get the default PAGE_ALLOC_COSTLY_ORDER=3=32k.Quoted 9 lines
Quoted 9 lines
I know you have argued for using a page order in the past, rather than a size in bytes. But my argument is that user space is mostly doing mmaps based on sizes independent of the base page size (an assumption!) and a system's memory is obviously a fixed quantity that doesn't it doesn't change with base page size. So it feels more natural to limit internal fragmentation based on an absolute size rather than a quantity of pages. Kyril have also suggested using absolute sizes in the past [1].
Quoted 6 lines
Quoted 2 lines
It's also worth mentioning that the file-backed memory "
fault_around" mechanism chooses 64K.This example actually is against your argument:
1. There have been multiple reports that fault around hurt
performances and had to be disabled for some workloads over the years> --ANON_FOLIO_MAX_ORDER_UNHINTEDis likely to cause regressions too.I don't see how
ANON_FOLIO_MAX_ORDER_UNHINTEDcan cause regressions; it's adding a limit making the behaviour of Large Anon Folios more similar to the old behaviour than it otherwise would be. Without it, we will be allocating 2M folios in some cases which would be much more likely to cause regression in unprepared apps IMHO.Quoted 2 lines
2. Not only can fault around be disabled, its default value can be changed too -- this series can't do either.
I had a mechanism for that in the previous version, but discussion concluded that we should avoid adding the control for now and add it only if/when we have identified a workload that will benefit.
Quoted 3 lines
3. Most importantly, fault around does not do high-order allocations -- this series does, and high-order allocations can be very difficult under memory pressure.
But
ANON_FOLIO_MAX_ORDER_UNHINTED*reduces* the order from what it otherwise would be. So I don't see how its making things worse?Quoted 7 lines
Quoted 3 lines
If this approach really looks unacceptable, I have a couple of other ideas. But I personally favour the approach that is already in the patch.
I understand. If the answer to my question above is yes, then let's take a step back and figure out whether overloading existing ABIs is acceptable or not. Does this sound good to you?
Yes, good idea. Hopefully my explanation above (and all the previous threads) gives you a good idea for the problem as I see it, and how I think hooking the THP hints is helpful to the solution. If I've understood David's previuous remarks correctly, then this also aligns with David's opinions (David you could confirm/deny this please?). Could you explain your position that hooking these ABIs is a bad approach?
Quoted 55 lines
Quoted 35 lines
1) Add a large/small flag to
arch_wants_pte_order(). arm64, at least, actually has 2 mechanisms, HPA and contpte. Currently arm64 is always returning the contpte order, but with a flag, it could return contpte order for large, and HPA order for small. (I know we previously passed the vma and we didn't like that, and this is pretty similar). I still think the SW (core-mm) needs a way to sensibly limit internal fragmentation though, so personally I still think having an upper limit in this case is useful.2) More radical: move to a per-vma auto-tuning solution, which looks at the fault pattern and maintains an allocation order in the VMA, which is modified based on fault pattern. e.g. When we get faults that occur immediately adjacent to the allocated range, we increase; when we get faults not connected to previously allocated pages we decrease. I think it's an interesting thing to look at, but certainly prefer that it's not part of an MVP implementation.
[1]
https://lore.kernel.org/linux-mm/[email protected]/Quoted 12 lines
Quoted 11 lines
mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 4103
Quoted 11 lines
mm/memory.c +10
-0+if ((vma->vm_flags &VM_NOHUGEPAGE ) || +test_bit (MMF_DISABLE_THP , &vma->vm_mm->flags) || + !hugepage_flags_enabled ()) + order =0 ; +else { + order =max (arch_wants_pte_order (),PAGE_ALLOC_COSTLY_ORDER ); + +if (!hugepage_vma_check (vma, vma->vm_flags, false, true, true)) + order =min (order,ANON_FOLIO_MAX_ORDER_UNHINTED ); + }Quoted 2 lines
I'm a bit surprised to see the above: why can we overload existing ABIs? I don't think we can.
Quoted 7 lines
I think this is all covered by the conversation with David against v2; see [2] and proceeding replies. Argument is that
VM_NOHUGEPAGE(and friends) is really a request from user space to optimize for the least memory wastage possible and avoid populating ptes that have not been expressly requested.Thanks for the info.
I think there might be a misunderstanding here.
David, can you please clarify whether you suggested we overland (change the semantics) of existing ABIs?
This sounds like a big red flag to me. If that's really what you suggest, can you shed some light on why this is acceptable to existing userspace at all?
Thanks.
Re: [PATCH v4 2/5] mm: LARGE_ANON_FOLIO for improved performance
In reply to Ryan RobertsQuoted 167 lines · Ryan Roberts
Quoted 142 lines · Yu Zhao
Quoted 141 lines · Yu Zhao
Quoted 139 lines · Ryan Roberts
Introduce
LARGE_ANON_FOLIOfeature, which allows anonymous memory to be allocated in large folios of a determined order. All pages of the large folio are pte-mapped during the same page fault, significantly reducing the number of page faults. The number of per-page operations (e.g. ref counting, rmap management lru list management) are also significantly reduced since those ops now become per-folio.The new behaviour is hidden behind the new
LARGE_ANON_FOLIOKconfig, which defaults to disabled for now; The long term aim is for this to defaut to enabled, but there are some risks around internal fragmentation that need to be better understood first.When enabled, the folio order is determined as such: For a vma, process or system that has explicitly disabled THP, we continue to allocate order-0. THP is most likely disabled to avoid any possible internal fragmentation so we honour that request.
Otherwise, the return value of
arch_wants_pte_order()is used. For vmas that have not explicitly opted-in to use transparent hugepages (e.g. where thp=madvise and the vma does not haveMADV_HUGEPAGE), thenarch_wants_pte_order()is limited to 64K (orPAGE_SIZE, whichever is bigger). This allows for a performance boost without requiring any explicit opt-in from the workload while limitting internal fragmentation.If the preferred order can't be used (e.g. because the folio would breach the bounds of the vma, or because ptes in the region are already mapped) then we fall back to a suitable lower order; first
PAGE_ALLOC_COSTLY_ORDER, then order-0.
arch_wants_pte_order()can be overridden by the architecture if desired. Some architectures (e.g. arm64) can coalsece TLB entries if a contiguous set of ptes map physically contigious, naturally aligned memory, so this mechanism allows the architecture to optimize as required.Here we add the default implementation of
arch_wants_pte_order(), used when the architecture does not define it, which returns -1, implying that the HW has no preference. In this case, mm will choose it's own default order.Signed-off-by Ryan Roberts <[email protected]>--- include/linux/pgtable.h | 13 ++++ mm/Kconfig | 10 +++ mm/memory.c | 166 ++++++++++++++++++++++++++++++++++++---- 3 files changed, 172 insertions(+), 17 deletions(-)include/linux/pgtable.h +13
-0@@ -313,6 +313,19 @@ static inline bool arch_has_hw_pte_young(void) }#endif +#ifndef arch_wants_pte_order +/* +* Returns preferred folio order for pte-mapped memory. Must be in range [0, +* PMD_SHIFT-PAGE_SHIFT) and must not be order-1 since THP requires large folios +* to be at least order-2. Negative value implies that the HW has no preference +* and mm will choose it's own default order. +*/ +static inline int arch_wants_pte_order (void ) +{ +return -1 ; +} +#endif +#ifndef __HAVE_ARCH_PTEP_GET_AND_CLEAR static inline pte_t ptep_get_and_clear(struct mm_struct *mm, unsigned long address,mm/Kconfig +10
-0@@ -1238,4 +1238,14 @@ config LOCK_MM_AND_FIND_VMA source "mm/damon/Kconfig" +config LARGE_ANON_FOLIO + bool "Allocate large folios for anonymous memory" + depends on TRANSPARENT_HUGEPAGE + default n + help + Use large (bigger than order-0) folios to back anonymous memory where + possible, even for pte-mapped memory. This reduces the number of page + faults, as well as other per-page overheads to improve performance for + many workloads. + endmenumm/memory.c +40
-0@@ -4050,6 +4050,127 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)return ret; } +static bool vmf_pte_range_changed (struct vm_fault *vmf,int nr_pages) +{ +int i; + +if (nr_pages ==1 ) +return vmf_pte_changed (vmf); + +for (i =0 ; i < nr_pages; i++) { +if (!pte_none (ptep_get_lockless (vmf->pte + i))) +return true; + } + +return false; +} + +#ifdef CONFIG_LARGE_ANON_FOLIO +#define ANON_FOLIO_MAX_ORDER_UNHINTED \ + (ilog2(max_t(unsigned long, SZ_64K, PAGE_SIZE)) - PAGE_SHIFT) + +static int anon_folio_order (struct vm_area_struct *vma) +{ +int order; + +/* +* If THP is explicitly disabled for either the vma, the process or the +* system, then this is very likely intended to limit internal +* fragmentation; in this case, don't attempt to allocate a large +* anonymous folio. +* +* Else, if the vma is eligible for thp, allocate a large folio of the +* size preferred by the arch. Or if the arch requested a very small +* size or didn't request a size, then use PAGE_ALLOC_COSTLY_ORDER, +* which still meets the arch's requirements but means we still take +* advantage of SW optimizations (e.g. fewer page faults). +* +* Finally if thp is enabled but the vma isn't eligible, take the +* arch-preferred size and limit it to ANON_FOLIO_MAX_ORDER_UNHINTED. +* This ensures workloads that have not explicitly opted-in take benefit +* while capping the potential for internal fragmentation. +*/ Quoted 8 lines
Quoted 8 lines
What empirical evidence is
SZ_64Kbased on?
What workloads would benefit from it?
How much would they benefit from it?
Would they benefit more or less from different values? How much internal fragmentation would it cause?
What cost function was used to arrive at the conclusion that its benefits outweigh its costs?Sorry this has taken a little while to reply to; I've been re-running my perf tests with the modern patches to recomfirm old data.
In terms of empirical evidence, I've run the kernel compilation benchmark (yes I know its a narrow use case, but I figure some data is better than no data), for all values of
ANON_FOLIO_MAX_ORDER_UNHINTED{4k, 16k, 32k, 64k, 128k, 256k}.I've run each test 15 times across 5 system reboots on Ampere Altra (arm64), with the kernel configured for 4K base pages - I could rerun for other base page sizes if we want to go further down this route.
I've captured run time and peak memory usage, and taken the mean. The stdev for the peak memory usage is big-ish, but I'm confident this still captures the central tendancy well:
Quoted 8 lines
| MAX_ORDER_UNHINTED | real-time | kern-time | user-time | peak memory | |:-------------------|------------:|------------:|------------:|:------------| | 4k | 0.0% | 0.0% | 0.0% | 0.0% | | 16k | -3.6% | -26.5% | -0.5% | -0.1% | | 32k | -4.8% | -37.4% | -0.6% | -0.1% | | 64k | -5.7% | -42.0% | -0.6% | -1.1% | | 128k | -5.6% | -42.1% | -0.7% | 1.4% | | 256k | -4.9% | -41.9% | -0.4% | 1.9% |
Here is my test result:
real user sys
hink-4k: 0% 0% 0%
hink-16K: -3% 0.1% -18.3%
hink-32K: -4% 0.2% -27.2%
hink-64K: -4% 0.5% -31.0%
hink-128K: -4% 0.9% -33.7%
hink-256K: -5% 1% -34.6%
I used command:/usr/bin/time -f "\t%E real,\t%U user,\t%S sys" make -skj96 allmodconfig all to build kernel and collect the real time/user time/kernel time. /sys/kernel/mm/transparent_hugepage/enabled is "madvise".
Let me know if you have any question about the test.
I also find one strange behavior with this version. It's related with why I need to set the /sys/kernel/mm/transparent_hugepage/enabled to "madvise". If it's "never", the large folio is disabled either.
If it's "always", the THP will be active before large folio. So the system is in the mixed mode. it's not suitable for this test.
So if it's "never", large folio is disabled. But why "madvise" enables large folio unconditionly? Suppose it's only enabled for the VMA range which user madvise large folio (or THP)?
Specific for the hink setting, my understand is that we can't choose it only by this testing. Other workloads may have different behavior with differnt hink setting.
Regards
Yin, Fengwei
Quoted 249 lines
64K looks like the clear sweet spot to me.
I know you have argued for using a page order in the past, rather than a size in bytes. But my argument is that user space is mostly doing mmaps based on sizes independent of the base page size (an assumption!) and a system's memory is obviously a fixed quantity that doesn't it doesn't change with base page size. So it feels more natural to limit internal fragmentation based on an absolute size rather than a quantity of pages. Kyril have also suggested using absolute sizes in the past [1].
It's also worth mentioning that the file-backed memory "
fault_around" mechanism chooses 64K.If this approach really looks unacceptable, I have a couple of other ideas. But I personally favour the approach that is already in the patch.
1) Add a large/small flag to
arch_wants_pte_order(). arm64, at least, actually has 2 mechanisms, HPA and contpte. Currently arm64 is always returning the contpte order, but with a flag, it could return contpte order for large, and HPA order for small. (I know we previously passed the vma and we didn't like that, and this is pretty similar). I still think the SW (core-mm) needs a way to sensibly limit internal fragmentation though, so personally I still think having an upper limit in this case is useful.2) More radical: move to a per-vma auto-tuning solution, which looks at the fault pattern and maintains an allocation order in the VMA, which is modified based on fault pattern. e.g. When we get faults that occur immediately adjacent to the allocated range, we increase; when we get faults not connected to previously allocated pages we decrease. I think it's an interesting thing to look at, but certainly prefer that it's not part of an MVP implementation.
[1]
https://lore.kernel.org/linux-mm/[email protected]/Quoted 12 lines
Quoted 11 lines
mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 4103
Quoted 11 lines
mm/memory.c +10
-0+if ((vma->vm_flags &VM_NOHUGEPAGE ) || +test_bit (MMF_DISABLE_THP , &vma->vm_mm->flags) || + !hugepage_flags_enabled ()) + order =0 ; +else { + order =max (arch_wants_pte_order (),PAGE_ALLOC_COSTLY_ORDER ); + +if (!hugepage_vma_check (vma, vma->vm_flags, false, true, true)) + order =min (order,ANON_FOLIO_MAX_ORDER_UNHINTED ); + }Quoted 2 lines
I'm a bit surprised to see the above: why can we overload existing ABIs? I don't think we can.
I think this is all covered by the conversation with David against v2; see [2] and proceeding replies. Argument is that
VM_NOHUGEPAGE(and friends) is really a request from user space to optimize for the least memory wastage possible and avoid populating ptes that have not been expressly requested.[2]
https://lore.kernel.org/linux-mm/[email protected]/Assuming we could, you would have to
Quoted 2 lines
update
Documentation/admin-guide/mm/transhuge.rstin the same patchset, and the man page formadvise()in a separate patch.Yes, that's a fair point. Although
transhuge.rstdoesn't even mentionMADV_NOHUGEPAGEtoday.Quoted 4 lines
Most importantly, existing userspace programs that don't work well with THPs won't be able to use (try) large folios either -- this is a big no no.
I think we need some comments from David here. As mentioned I've added this tie-in based on his (strong) recommendation.
Quoted 11 lines
Quoted 11 lines
mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 4108
Quoted 8 lines
mm/memory.c +5
-0+ +return order; +} + +static int alloc_anon_folio (struct vm_fault *vmf,struct folio **folio)static struct folio *alloc_anon_folio(struct
vm_fault*vmf)Quoted 1 line
Quoted 1 line
and use
ERR_PTR()and its friends.Yes, agreed. I'll change this for the next version.
Quoted 142 lines
Quoted 142 lines
Quoted 142 lines
mm/memory.c +90
-15+{ +int i; +gfp_t gfp; +pte_t *pte; +unsigned long addr; +struct vm_area_struct *vma = vmf->vma; +int prefer =anon_folio_order (vma); +int orders[] = { + prefer, + prefer >PAGE_ALLOC_COSTLY_ORDER ?PAGE_ALLOC_COSTLY_ORDER :0 , +0 , + }; + + *folio =NULL ; + +if (vmf_orig_pte_uffd_wp (vmf)) + goto fallback; + +for (i =0 ; orders[i]; i++) { + addr =ALIGN_DOWN (vmf->address,PAGE_SIZE << orders[i]); +if (addr >= vma->vm_start && + addr + (PAGE_SIZE << orders[i]) <= vma->vm_end) +break ; + } + +if (!orders[i]) + goto fallback; + + pte =pte_offset_map (vmf->pmd, vmf->address &PMD_MASK ); +if (!pte) +return -EAGAIN ; + +for (; orders[i]; i++) { + addr =ALIGN_DOWN (vmf->address,PAGE_SIZE << orders[i]); + vmf->pte = pte +pte_index (addr); +if (!vmf_pte_range_changed (vmf,1 << orders[i])) +break ; + } + + vmf->pte =NULL ; +pte_unmap (pte); + + gfp =vma_thp_gfp_mask (vma); + +for (; orders[i]; i++) { + addr =ALIGN_DOWN (vmf->address,PAGE_SIZE << orders[i]); + *folio =vma_alloc_folio (gfp, orders[i], vma, addr, true); +if (*folio) { +clear_huge_page (&(*folio)->page, addr,1 << orders[i]); +return 0 ; + } + } + +fallback: + *folio =vma_alloc_zeroed_movable_folio (vma, vmf->address); +return *folio ?0 : -ENOMEM ; +} +#else +static inline int alloc_anon_folio (struct vm_fault *vmf,struct folio **folio) +{ + *folio =vma_alloc_zeroed_movable_folio (vmf->vma, vmf->address); +return *folio ?0 : -ENOMEM ; +} +#endif +/* * We enter with non-exclusive mmap_lock (to exclude vma changes, * but allow concurrent faults), and pte mapped but not yet locked. @@ -4057,6 +4178,9 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)*/ static vm_fault_t do_anonymous_page (struct vm_fault *vmf) { +int i =0 ; +int nr_pages =1 ; +unsigned long addr = vmf->address;bool uffd_wp =vmf_orig_pte_uffd_wp (vmf);struct vm_area_struct *vma = vmf->vma;struct folio *folio;@@ -4101,10 +4225,15 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)/* Allocate our own private page. */ if (unlikely (anon_vma_prepare (vma))) goto oom; - folio =vma_alloc_zeroed_movable_folio (vma, vmf->address); + ret =alloc_anon_folio (vmf, &folio); +if (unlikely (ret == -EAGAIN )) +return 0 ;if (!folio) goto oom; + nr_pages =folio_nr_pages (folio); + addr =ALIGN_DOWN (vmf->address, nr_pages *PAGE_SIZE ); +if (mem_cgroup_charge (folio, vma->vm_mm,GFP_KERNEL )) goto oom_free_page;folio_throttle_swaprate (folio,GFP_KERNEL );@@ -4116,17 +4245,12 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)*/ __folio_mark_uptodate (folio); - entry =mk_pte (&folio->page, vma->vm_page_prot); - entry =pte_sw_mkyoung (entry); -if (vma->vm_flags &VM_WRITE ) - entry =pte_mkwrite (pte_mkdirty (entry)); - - vmf->pte =pte_offset_map_lock (vma->vm_mm, vmf->pmd, vmf->address, - &vmf->ptl); + vmf->pte =pte_offset_map_lock (vma->vm_mm, vmf->pmd, addr, &vmf->ptl);if (!vmf->pte) goto release; -if (vmf_pte_changed (vmf)) { -update_mmu_tlb (vma, vmf->address, vmf->pte); +if (vmf_pte_range_changed (vmf, nr_pages)) { +for (i =0 ; i < nr_pages; i++) +update_mmu_tlb (vma, addr +PAGE_SIZE * i, vmf->pte + i); goto release; }@@ -4141,16 +4265,24 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)return handle_userfault (vmf,VM_UFFD_MISSING ); } -inc_mm_counter (vma->vm_mm,MM_ANONPAGES ); -folio_add_new_anon_rmap (folio, vma, vmf->address); +folio_ref_add (folio, nr_pages -1 ); +add_mm_counter (vma->vm_mm,MM_ANONPAGES , nr_pages); +folio_add_new_anon_rmap (folio, vma, addr);folio_add_lru_vma (folio, vma); + +for (i =0 ; i < nr_pages; i++) { + entry =mk_pte (folio_page (folio, i), vma->vm_page_prot); + entry =pte_sw_mkyoung (entry); +if (vma->vm_flags &VM_WRITE ) + entry =pte_mkwrite (pte_mkdirty (entry)); setpte: -if (uffd_wp) - entry =pte_mkuffd_wp (entry); -set_pte_at (vma->vm_mm, vmf->address, vmf->pte, entry); +if (uffd_wp) + entry =pte_mkuffd_wp (entry); +set_pte_at (vma->vm_mm, addr +PAGE_SIZE * i, vmf->pte + i, entry);Quoted 10 lines
Quoted 10 lines
mm/memory.c static vm_fault_t do_anonymous_page(struct vm_fault *vmf) · line 4288
Quoted 8 lines
mm/memory.c +3
-2-/* No need to invalidate - it was non-present before */ -update_mmu_cache (vma, vmf->address, vmf->pte); +/* No need to invalidate - it was non-present before */ +update_mmu_cache (vma, addr +PAGE_SIZE * i, vmf->pte + i); + } unlock:if (vmf->pte)pte_unmap_unlock (vmf->pte, vmf->ptl);The rest looks good to me.
Thanks, as always, for the detailed review and feedback!
Thanks,
Ryan
Re: [PATCH v4 2/5] mm: LARGE_ANON_FOLIO for improved performance
On 03/08/2023 09:05, Yin Fengwei wrote:
...
Quoted 24 lines
Quoted 4 lines
I've captured run time and peak memory usage, and taken the mean. The stdev for the peak memory usage is big-ish, but I'm confident this still captures the central tendancy well:
Quoted 8 lines
| MAX_ORDER_UNHINTED | real-time | kern-time | user-time | peak memory | |:-------------------|------------:|------------:|------------:|:------------| | 4k | 0.0% | 0.0% | 0.0% | 0.0% | | 16k | -3.6% | -26.5% | -0.5% | -0.1% | | 32k | -4.8% | -37.4% | -0.6% | -0.1% | | 64k | -5.7% | -42.0% | -0.6% | -1.1% | | 128k | -5.6% | -42.1% | -0.7% | 1.4% | | 256k | -4.9% | -41.9% | -0.4% | 1.9% |Here is my test result:
real user syshink-4k: 0% 0% 0%
hink-16K: -3% 0.1% -18.3%
hink-32K: -4% 0.2% -27.2%
hink-64K: -4% 0.5% -31.0%
hink-128K: -4% 0.9% -33.7%
hink-256K: -5% 1% -34.6%
Quoted 5 lines
I used command:
/usr/bin/time-f "\t%E real,\t%U user,\t%S sys" make -skj96 allmodconfig all to build kernel and collect the real time/user time/kernel time./sys/kernel/mm/transparent_hugepage/enabledis "madvise".
Let me know if you have any question about the test.
Thanks for doing this! I have a couple of questions:
- how many times did you run each test?
- how did you configure the large page size? (I sent an email out yesterday saying that I was doing it wrong from my tests, so the 128k and 256k results for my test set are not valid.
- what does "hink" mean??
Quoted 6 lines
I also find one strange behavior with this version. It's related with why I need to set the
/sys/kernel/mm/transparent_hugepage/enabledto "madvise". If it's "never", the large folio is disabled either.
If it's "always", the THP will be active before large folio. So the system is in the mixed mode. it's not suitable for this test.
We had a discussion around this in the THP meeting yesterday. I'm going to write this up propoerly so we can have proper systematic discussion. The tentative conclusion is that MADV_NOHUGEPAGE must continue to mean "do not fault in more than is absolutely necessary". I would assume we need to extend that thinking to the process-wide and system-wide knobs (as is done in the patch), but we didn't explicitly say so in the meeting.
My intention is that if you have requested THP and your vma is big enough for PMD-size then you get that, else you fallback to large anon folios. And if you have neither opted in nor out, then you get large anon folios.
We talked about the idea of adding a new knob that let's you set the max order, but that needs a lot more thought.
Anyway, as I said, I'll write it up so we can all systematically discuss.
Quoted 13 lines
So if it's "never", large folio is disabled. But why "madvise" enables large folio unconditionly? Suppose it's only enabled for the VMA range which user madvise large folio (or THP)?
Specific for the hink setting, my understand is that we can't choose it only by this testing. Other workloads may have different behavior with differnt hink setting.
Regards
Yin, Fengwei
Re: [PATCH v4 2/5] mm: LARGE_ANON_FOLIO for improved performance
Quoted 34 lines · Ryan Roberts
On 03/08/2023 09:05, Yin Fengwei wrote:
...
Quoted 24 lines
Quoted 4 lines
I've captured run time and peak memory usage, and taken the mean. The stdev for the peak memory usage is big-ish, but I'm confident this still captures the central tendancy well:
Quoted 8 lines
| MAX_ORDER_UNHINTED | real-time | kern-time | user-time | peak memory | |:-------------------|------------:|------------:|------------:|:------------| | 4k | 0.0% | 0.0% | 0.0% | 0.0% | | 16k | -3.6% | -26.5% | -0.5% | -0.1% | | 32k | -4.8% | -37.4% | -0.6% | -0.1% | | 64k | -5.7% | -42.0% | -0.6% | -1.1% | | 128k | -5.6% | -42.1% | -0.7% | 1.4% | | 256k | -4.9% | -41.9% | -0.4% | 1.9% |Here is my test result:
real user syshink-4k: 0% 0% 0%
hink-16K: -3% 0.1% -18.3%
hink-32K: -4% 0.2% -27.2%
hink-64K: -4% 0.5% -31.0%
hink-128K: -4% 0.9% -33.7%
hink-256K: -5% 1% -34.6%Quoted 5 lines
I used command:
/usr/bin/time-f "\t%E real,\t%U user,\t%S sys" make -skj96 allmodconfig all to build kernel and collect the real time/user time/kernel time./sys/kernel/mm/transparent_hugepage/enabledis "madvise".
Let me know if you have any question about the test.
Quoted 3 lines
Thanks for doing this! I have a couple of questions:
- how many times did you run each test?
Three times for each ANON_FOLIO_MAX_ORDER_UNHINTED. The stddev is quite small like less than %1.
Quoted 4 lines
- how did you configure the large page size? (I sent an email out yesterday saying that I was doing it wrong from my tests, so the 128k and 256k results for my test set are not valid.
I changed the ANON_FOLIO_MAX_ORDER_UNHINTED definition manually every time.
Quoted 2 lines
- what does "hink" mean??
Sorry for the typo. It should be ANON_FOLIO_MAX_ORDER_UNHINTED.
Quoted 8 lines
Quoted 7 lines
I also find one strange behavior with this version. It's related with why I need to set the
/sys/kernel/mm/transparent_hugepage/enabledto "madvise". If it's "never", the large folio is disabled either.
If it's "always", the THP will be active before large folio. So the system is in the mixed mode. it's not suitable for this test.
Quoted 6 lines
We had a discussion around this in the THP meeting yesterday. I'm going to write this up propoerly so we can have proper systematic discussion. The tentative conclusion is that
MADV_NOHUGEPAGEmust continue to mean "do not fault in more than is absolutely necessary". I would assume we need to extend that thinking to the process-wide and system-wide knobs (as is done in the patch), but we didn't explicitly say so in the meeting.
There are cases that THP is not appreciated because of the latency or memory consumption. For these cases, large folio may fill the gap as less latency and memory consumption.
So if disabling THP means large folio can't be used, we loose the chance to benefit those cases with large folio.
Regards
Yin, Fengwei
Quoted 24 lines
My intention is that if you have requested THP and your vma is big enough for PMD-size then you get that, else you fallback to large anon folios. And if you have neither opted in nor out, then you get large anon folios.
We talked about the idea of adding a new knob that let's you set the max order, but that needs a lot more thought.
Anyway, as I said, I'll write it up so we can all systematically discuss.
Quoted 13 lines
So if it's "never", large folio is disabled. But why "madvise" enables large folio unconditionly? Suppose it's only enabled for the VMA range which user madvise large folio (or THP)?
Specific for the hink setting, my understand is that we can't choose it only by this testing. Other workloads may have different behavior with differnt hink setting.
Regards
Yin, Fengwei
Re: [PATCH v4 2/5] mm: LARGE_ANON_FOLIO for improved performance
Quoted 37 lines · Yin Fengwei
Quoted 34 lines · Ryan Roberts
On 03/08/2023 09:05, Yin Fengwei wrote:
...
Quoted 29 lines
Quoted 4 lines
I've captured run time and peak memory usage, and taken the mean. The stdev for the peak memory usage is big-ish, but I'm confident this still captures the central tendancy well:
Quoted 8 lines
| MAX_ORDER_UNHINTED | real-time | kern-time | user-time | peak memory | |:-------------------|------------:|------------:|------------:|:------------| | 4k | 0.0% | 0.0% | 0.0% | 0.0% | | 16k | -3.6% | -26.5% | -0.5% | -0.1% | | 32k | -4.8% | -37.4% | -0.6% | -0.1% | | 64k | -5.7% | -42.0% | -0.6% | -1.1% | | 128k | -5.6% | -42.1% | -0.7% | 1.4% | | 256k | -4.9% | -41.9% | -0.4% | 1.9% |Here is my test result:
real user syshink-4k: 0% 0% 0%
hink-16K: -3% 0.1% -18.3%
hink-32K: -4% 0.2% -27.2%
hink-64K: -4% 0.5% -31.0%
hink-128K: -4% 0.9% -33.7%
hink-256K: -5% 1% -34.6%I used command:
/usr/bin/time-f "\t%E real,\t%U user,\t%S sys" make -skj96 allmodconfig all to build kernel and collect the real time/user time/kernel time./sys/kernel/mm/transparent_hugepage/enabledis "madvise".
Let me know if you have any question about the test.
Quoted 5 lines
Quoted 3 lines
Thanks for doing this! I have a couple of questions:
- how many times did you run each test?
Three times for each
ANON_FOLIO_MAX_ORDER_UNHINTED. The stddev is quite small like less than %1.
And out of interest, were you running on bare metal or in VM? And did you reboot between each run?
Quoted 5 lines
Quoted 4 lines
- how did you configure the large page size? (I sent an email out yesterday saying that I was doing it wrong from my tests, so the 128k and 256k results for my test set are not valid.
I changed the
ANON_FOLIO_MAX_ORDER_UNHINTEDdefinition manually every time.
In that case, I think your results are broken in a similar way to mine. This code means that order will never be higher than 3 (32K) on x86:
+ order = max(arch_wants_pte_order(), PAGE_ALLOC_COSTLY_ORDER);
+
+ if (!hugepage_vma_check(vma, vma->vm_flags, false, true, true))
+ order = min(order, ANON_FOLIO_MAX_ORDER_UNHINTED);
On x86, arch_wants_pte_order() is not implemented and the default returns -1, so you end up with:
order = min(PAGE_ALLOC_COSTLY_ORDER, ANON_FOLIO_MAX_ORDER_UNHINTED)
So your 4k, 16k and 32k results should be valid, but 64k, 128k and 256k results are actually using 32k, I think? Which is odd because you are getting more stddev than the < 1% you quoted above? So perhaps this is down to rebooting (kaslr, or something...?)
(on arm64, arch_wants_pte_order() returns 4, so my 64k result is also valid).
As a quick hack to work around this, would you be able to change the code to this:
+ if (!hugepage_vma_check(vma, vma->vm_flags, false, true, true))
+ order = ANON_FOLIO_MAX_ORDER_UNHINTED;
Quoted 19 lines
Quoted 3 lines
- what does "hink" mean??
Sorry for the typo. It should be
ANON_FOLIO_MAX_ORDER_UNHINTED.Quoted 8 lines
Quoted 7 lines
I also find one strange behavior with this version. It's related with why I need to set the
/sys/kernel/mm/transparent_hugepage/enabledto "madvise". If it's "never", the large folio is disabled either.
If it's "always", the THP will be active before large folio. So the system is in the mixed mode. it's not suitable for this test.Quoted 6 lines
We had a discussion around this in the THP meeting yesterday. I'm going to write this up propoerly so we can have proper systematic discussion. The tentative conclusion is that
MADV_NOHUGEPAGEmust continue to mean "do not fault in more than is absolutely necessary". I would assume we need to extend that thinking to the process-wide and system-wide knobs (as is done in the patch), but we didn't explicitly say so in the meeting.
Quoted 7 lines
There are cases that THP is not appreciated because of the latency or memory consumption. For these cases, large folio may fill the gap as less latency and memory consumption.
So if disabling THP means large folio can't be used, we loose the chance to benefit those cases with large folio.
Yes, I appreciate that. But there are also real use cases that expect MADV_NOHUGEPAGE means "do not fault more than is absolutely necessary" and the use cases break if that's not obeyed (e.g. live migration w/ qemu). So I think we need to be conservitive to start. These apps that are explicitly forbidding THP today, should be updated in the long run to opt-in to large anon folios using some as-yet undefined control.
Quoted 29 lines
Regards
Yin, FengweiQuoted 24 lines
My intention is that if you have requested THP and your vma is big enough for PMD-size then you get that, else you fallback to large anon folios. And if you have neither opted in nor out, then you get large anon folios.
We talked about the idea of adding a new knob that let's you set the max order, but that needs a lot more thought.
Anyway, as I said, I'll write it up so we can all systematically discuss.
Quoted 13 lines
So if it's "never", large folio is disabled. But why "madvise" enables large folio unconditionly? Suppose it's only enabled for the VMA range which user madvise large folio (or THP)?
Specific for the hink setting, my understand is that we can't choose it only by this testing. Other workloads may have different behavior with differnt hink setting.
Regards
Yin, Fengwei
Re: [PATCH v4 2/5] mm: LARGE_ANON_FOLIO for improved performance
Quoted 38 lines · Ryan Roberts
Quoted 37 lines · Yin Fengwei
Quoted 34 lines · Ryan Roberts
On 03/08/2023 09:05, Yin Fengwei wrote:
...
Quoted 29 lines
Quoted 4 lines
I've captured run time and peak memory usage, and taken the mean. The stdev for the peak memory usage is big-ish, but I'm confident this still captures the central tendancy well:
Quoted 8 lines
| MAX_ORDER_UNHINTED | real-time | kern-time | user-time | peak memory | |:-------------------|------------:|------------:|------------:|:------------| | 4k | 0.0% | 0.0% | 0.0% | 0.0% | | 16k | -3.6% | -26.5% | -0.5% | -0.1% | | 32k | -4.8% | -37.4% | -0.6% | -0.1% | | 64k | -5.7% | -42.0% | -0.6% | -1.1% | | 128k | -5.6% | -42.1% | -0.7% | 1.4% | | 256k | -4.9% | -41.9% | -0.4% | 1.9% |Here is my test result:
real user syshink-4k: 0% 0% 0%
hink-16K: -3% 0.1% -18.3%
hink-32K: -4% 0.2% -27.2%
hink-64K: -4% 0.5% -31.0%
hink-128K: -4% 0.9% -33.7%
hink-256K: -5% 1% -34.6%I used command:
/usr/bin/time-f "\t%E real,\t%U user,\t%S sys" make -skj96 allmodconfig all to build kernel and collect the real time/user time/kernel time./sys/kernel/mm/transparent_hugepage/enabledis "madvise".
Let me know if you have any question about the test.
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
Thanks for doing this! I have a couple of questions:
- how many times did you run each test?
Three times for each
ANON_FOLIO_MAX_ORDER_UNHINTED. The stddev is quite small like less than %1.And out of interest, were you running on bare metal or in VM? And did you reboot between each run?
I run the test on bare metal env. I didn't reboot for every run. But have to reboot for different ANON_FOLIO_MAX_ORDER_UNHINTED size. I do
echo 3 > /proc/sys/vm/drop_caches
for everything run after "make mrproper" even after a fresh boot.
Quoted 15 lines
Quoted 6 lines
Quoted 5 lines
- how did you configure the large page size? (I sent an email out yesterday saying that I was doing it wrong from my tests, so the 128k and 256k results for my test set are not valid.
I changed the
ANON_FOLIO_MAX_ORDER_UNHINTEDdefinition manually every time.In that case, I think your results are broken in a similar way to mine. This code means that order will never be higher than 3 (32K) on x86:
(quoted diff) +4
-0+ order = max(arch_wants_pte_order(), PAGE_ALLOC_COSTLY_ORDER); + + if (!hugepage_vma_check(vma, vma->vm_flags, false, true, true)) + order = min(order, ANON_FOLIO_MAX_ORDER_UNHINTED);
Quoted 2 lines
On x86,
arch_wants_pte_order()is not implemented and the default returns -1, so you end up with:
I added arch_waits_pte_order() for x86 and gave it a very large number. So the order is decided by ANON_FOLIO_MAX_ORDER_UNHINTED. I suppose my data is valid.
Quoted 42 lines
order =
min(PAGE_ALLOC_COSTLY_ORDER, ANON_FOLIO_MAX_ORDER_UNHINTED)So your 4k, 16k and 32k results should be valid, but 64k, 128k and 256k results are actually using 32k, I think? Which is odd because you are getting more stddev than the < 1% you quoted above? So perhaps this is down to rebooting (kaslr, or something...?)
(on arm64,
arch_wants_pte_order()returns 4, so my 64k result is also valid).As a quick hack to work around this, would you be able to change the code to this:
(quoted diff) +2
-0+ if (!hugepage_vma_check(vma, vma->vm_flags, false, true, true)) + order = ANON_FOLIO_MAX_ORDER_UNHINTED;Quoted 20 lines
Quoted 4 lines
- what does "hink" mean??
Sorry for the typo. It should be
ANON_FOLIO_MAX_ORDER_UNHINTED.Quoted 8 lines
Quoted 7 lines
I also find one strange behavior with this version. It's related with why I need to set the
/sys/kernel/mm/transparent_hugepage/enabledto "madvise". If it's "never", the large folio is disabled either.
If it's "always", the THP will be active before large folio. So the system is in the mixed mode. it's not suitable for this test.Quoted 6 lines
We had a discussion around this in the THP meeting yesterday. I'm going to write this up propoerly so we can have proper systematic discussion. The tentative conclusion is that
MADV_NOHUGEPAGEmust continue to mean "do not fault in more than is absolutely necessary". I would assume we need to extend that thinking to the process-wide and system-wide knobs (as is done in the patch), but we didn't explicitly say so in the meeting.Quoted 7 lines
There are cases that THP is not appreciated because of the latency or memory consumption. For these cases, large folio may fill the gap as less latency and memory consumption.
So if disabling THP means large folio can't be used, we loose the chance to benefit those cases with large folio.
Quoted 6 lines
Yes, I appreciate that. But there are also real use cases that expect
MADV_NOHUGEPAGEmeans "do not fault more than is absolutely necessary" and the use cases break if that's not obeyed (e.g. live migration w/ qemu). So I think we need to be conservitive to start. These apps that are explicitly forbidding THP today, should be updated in the long run to opt-in to large anon folios using some as-yet undefined control.
Fair enough.
Regards
Yin, Fengwei
Quoted 31 lines
Quoted 30 lines
Regards
Yin, FengweiQuoted 24 lines
My intention is that if you have requested THP and your vma is big enough for PMD-size then you get that, else you fallback to large anon folios. And if you have neither opted in nor out, then you get large anon folios.
We talked about the idea of adding a new knob that let's you set the max order, but that needs a lot more thought.
Anyway, as I said, I'll write it up so we can all systematically discuss.
Quoted 13 lines
So if it's "never", large folio is disabled. But why "madvise" enables large folio unconditionly? Suppose it's only enabled for the VMA range which user madvise large folio (or THP)?
Specific for the hink setting, my understand is that we can't choose it only by this testing. Other workloads may have different behavior with differnt hink setting.
Regards
Yin, Fengwei
Re: [PATCH v4 2/5] mm: LARGE_ANON_FOLIO for improved performance
Quoted 70 lines · Yin Fengwei
Quoted 38 lines · Ryan Roberts
Quoted 37 lines · Yin Fengwei
Quoted 34 lines · Ryan Roberts
On 03/08/2023 09:05, Yin Fengwei wrote:
...
Quoted 29 lines
Quoted 4 lines
I've captured run time and peak memory usage, and taken the mean. The stdev for the peak memory usage is big-ish, but I'm confident this still captures the central tendancy well:
Quoted 8 lines
| MAX_ORDER_UNHINTED | real-time | kern-time | user-time | peak memory | |:-------------------|------------:|------------:|------------:|:------------| | 4k | 0.0% | 0.0% | 0.0% | 0.0% | | 16k | -3.6% | -26.5% | -0.5% | -0.1% | | 32k | -4.8% | -37.4% | -0.6% | -0.1% | | 64k | -5.7% | -42.0% | -0.6% | -1.1% | | 128k | -5.6% | -42.1% | -0.7% | 1.4% | | 256k | -4.9% | -41.9% | -0.4% | 1.9% |Here is my test result:
real user syshink-4k: 0% 0% 0%
hink-16K: -3% 0.1% -18.3%
hink-32K: -4% 0.2% -27.2%
hink-64K: -4% 0.5% -31.0%
hink-128K: -4% 0.9% -33.7%
hink-256K: -5% 1% -34.6%I used command:
/usr/bin/time-f "\t%E real,\t%U user,\t%S sys" make -skj96 allmodconfig all to build kernel and collect the real time/user time/kernel time./sys/kernel/mm/transparent_hugepage/enabledis "madvise".
Let me know if you have any question about the test.Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
Thanks for doing this! I have a couple of questions:
- how many times did you run each test?
Three times for each
ANON_FOLIO_MAX_ORDER_UNHINTED. The stddev is quite small like less than %1.And out of interest, were you running on bare metal or in VM? And did you reboot between each run?
I run the test on bare metal env. I didn't reboot for every run. But have to reboot for different
ANON_FOLIO_MAX_ORDER_UNHINTEDsize. I do
echo 3 >/proc/sys/vm/drop_caches
for everything run after "make mrproper" even after a fresh boot.Quoted 15 lines
Quoted 6 lines
Quoted 5 lines
- how did you configure the large page size? (I sent an email out yesterday saying that I was doing it wrong from my tests, so the 128k and 256k results for my test set are not valid.
I changed the
ANON_FOLIO_MAX_ORDER_UNHINTEDdefinition manually every time.In that case, I think your results are broken in a similar way to mine. This code means that order will never be higher than 3 (32K) on x86:
(quoted diff) +4
-0+ order = max(arch_wants_pte_order(), PAGE_ALLOC_COSTLY_ORDER); + + if (!hugepage_vma_check(vma, vma->vm_flags, false, true, true)) + order = min(order, ANON_FOLIO_MAX_ORDER_UNHINTED);
Quoted 4 lines
Quoted 2 lines
On x86,
arch_wants_pte_order()is not implemented and the default returns -1, so you end up with:I added
arch_waits_pte_order()for x86 and gave it a very large number. So the order is decided byANON_FOLIO_MAX_ORDER_UNHINTED. I suppose my data is valid.
Ahh great! ok sorry for the noise.
Given part of the rationale for the experiment was to plot perf against memory usage, did you collect any memory numbers?
Quoted 86 lines
Quoted 43 lines
order =
min(PAGE_ALLOC_COSTLY_ORDER, ANON_FOLIO_MAX_ORDER_UNHINTED)So your 4k, 16k and 32k results should be valid, but 64k, 128k and 256k results are actually using 32k, I think? Which is odd because you are getting more stddev than the < 1% you quoted above? So perhaps this is down to rebooting (kaslr, or something...?)
(on arm64,
arch_wants_pte_order()returns 4, so my 64k result is also valid).As a quick hack to work around this, would you be able to change the code to this:
(quoted diff) +2
-0+ if (!hugepage_vma_check(vma, vma->vm_flags, false, true, true)) + order = ANON_FOLIO_MAX_ORDER_UNHINTED;Quoted 20 lines
Quoted 4 lines
- what does "hink" mean??
Sorry for the typo. It should be
ANON_FOLIO_MAX_ORDER_UNHINTED.Quoted 8 lines
Quoted 7 lines
I also find one strange behavior with this version. It's related with why I need to set the
/sys/kernel/mm/transparent_hugepage/enabledto "madvise". If it's "never", the large folio is disabled either.
If it's "always", the THP will be active before large folio. So the system is in the mixed mode. it's not suitable for this test.Quoted 6 lines
We had a discussion around this in the THP meeting yesterday. I'm going to write this up propoerly so we can have proper systematic discussion. The tentative conclusion is that
MADV_NOHUGEPAGEmust continue to mean "do not fault in more than is absolutely necessary". I would assume we need to extend that thinking to the process-wide and system-wide knobs (as is done in the patch), but we didn't explicitly say so in the meeting.Quoted 7 lines
There are cases that THP is not appreciated because of the latency or memory consumption. For these cases, large folio may fill the gap as less latency and memory consumption.
So if disabling THP means large folio can't be used, we loose the chance to benefit those cases with large folio.
Quoted 6 lines
Yes, I appreciate that. But there are also real use cases that expect
MADV_NOHUGEPAGEmeans "do not fault more than is absolutely necessary" and the use cases break if that's not obeyed (e.g. live migration w/ qemu). So I think we need to be conservitive to start. These apps that are explicitly forbidding THP today, should be updated in the long run to opt-in to large anon folios using some as-yet undefined control.Fair enough.
Regards
Yin, FengweiQuoted 31 lines
Quoted 30 lines
Regards
Yin, FengweiQuoted 24 lines
My intention is that if you have requested THP and your vma is big enough for PMD-size then you get that, else you fallback to large anon folios. And if you have neither opted in nor out, then you get large anon folios.
We talked about the idea of adding a new knob that let's you set the max order, but that needs a lot more thought.
Anyway, as I said, I'll write it up so we can all systematically discuss.
Quoted 13 lines
So if it's "never", large folio is disabled. But why "madvise" enables large folio unconditionly? Suppose it's only enabled for the VMA range which user madvise large folio (or THP)?
Specific for the hink setting, my understand is that we can't choose it only by this testing. Other workloads may have different behavior with differnt hink setting.
Regards
Yin, Fengwei
Re: [PATCH v4 2/5] mm: LARGE_ANON_FOLIO for improved performance
Quoted 76 lines · Ryan Roberts
Quoted 70 lines · Yin Fengwei
Quoted 38 lines · Ryan Roberts
Quoted 37 lines · Yin Fengwei
Quoted 34 lines · Ryan Roberts
On 03/08/2023 09:05, Yin Fengwei wrote:
...
Quoted 29 lines
Quoted 4 lines
I've captured run time and peak memory usage, and taken the mean. The stdev for the peak memory usage is big-ish, but I'm confident this still captures the central tendancy well:
Quoted 8 lines
| MAX_ORDER_UNHINTED | real-time | kern-time | user-time | peak memory | |:-------------------|------------:|------------:|------------:|:------------| | 4k | 0.0% | 0.0% | 0.0% | 0.0% | | 16k | -3.6% | -26.5% | -0.5% | -0.1% | | 32k | -4.8% | -37.4% | -0.6% | -0.1% | | 64k | -5.7% | -42.0% | -0.6% | -1.1% | | 128k | -5.6% | -42.1% | -0.7% | 1.4% | | 256k | -4.9% | -41.9% | -0.4% | 1.9% |Here is my test result:
real user syshink-4k: 0% 0% 0%
hink-16K: -3% 0.1% -18.3%
hink-32K: -4% 0.2% -27.2%
hink-64K: -4% 0.5% -31.0%
hink-128K: -4% 0.9% -33.7%
hink-256K: -5% 1% -34.6%I used command:
/usr/bin/time-f "\t%E real,\t%U user,\t%S sys" make -skj96 allmodconfig all to build kernel and collect the real time/user time/kernel time./sys/kernel/mm/transparent_hugepage/enabledis "madvise".
Let me know if you have any question about the test.Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
Thanks for doing this! I have a couple of questions:
- how many times did you run each test?
Three times for each
ANON_FOLIO_MAX_ORDER_UNHINTED. The stddev is quite small like less than %1.And out of interest, were you running on bare metal or in VM? And did you reboot between each run?
I run the test on bare metal env. I didn't reboot for every run. But have to reboot for different
ANON_FOLIO_MAX_ORDER_UNHINTEDsize. I do
echo 3 >/proc/sys/vm/drop_caches
for everything run after "make mrproper" even after a fresh boot.Quoted 15 lines
Quoted 6 lines
Quoted 5 lines
- how did you configure the large page size? (I sent an email out yesterday saying that I was doing it wrong from my tests, so the 128k and 256k results for my test set are not valid.
I changed the
ANON_FOLIO_MAX_ORDER_UNHINTEDdefinition manually every time.In that case, I think your results are broken in a similar way to mine. This code means that order will never be higher than 3 (32K) on x86:
(quoted diff) +4
-0+ order = max(arch_wants_pte_order(), PAGE_ALLOC_COSTLY_ORDER); + + if (!hugepage_vma_check(vma, vma->vm_flags, false, true, true)) + order = min(order, ANON_FOLIO_MAX_ORDER_UNHINTED);Quoted 4 lines
Quoted 2 lines
On x86,
arch_wants_pte_order()is not implemented and the default returns -1, so you end up with:I added
arch_waits_pte_order()for x86 and gave it a very large number. So the order is decided byANON_FOLIO_MAX_ORDER_UNHINTED. I suppose my data is valid.
Quoted 4 lines
Ahh great! ok sorry for the noise.
Given part of the rationale for the experiment was to plot perf against memory usage, did you collect any memory numbers?
No. I didn't collect the memory consumption.
Regards
Yin, Fengwei
Quoted 88 lines
Quoted 87 lines
Quoted 44 lines
order =
min(PAGE_ALLOC_COSTLY_ORDER, ANON_FOLIO_MAX_ORDER_UNHINTED)So your 4k, 16k and 32k results should be valid, but 64k, 128k and 256k results are actually using 32k, I think? Which is odd because you are getting more stddev than the < 1% you quoted above? So perhaps this is down to rebooting (kaslr, or something...?)
(on arm64,
arch_wants_pte_order()returns 4, so my 64k result is also valid).As a quick hack to work around this, would you be able to change the code to this:
(quoted diff) +2
-0+ if (!hugepage_vma_check(vma, vma->vm_flags, false, true, true)) + order = ANON_FOLIO_MAX_ORDER_UNHINTED;Quoted 20 lines
Quoted 4 lines
- what does "hink" mean??
Sorry for the typo. It should be
ANON_FOLIO_MAX_ORDER_UNHINTED.Quoted 8 lines
Quoted 7 lines
I also find one strange behavior with this version. It's related with why I need to set the
/sys/kernel/mm/transparent_hugepage/enabledto "madvise". If it's "never", the large folio is disabled either.
If it's "always", the THP will be active before large folio. So the system is in the mixed mode. it's not suitable for this test.Quoted 6 lines
We had a discussion around this in the THP meeting yesterday. I'm going to write this up propoerly so we can have proper systematic discussion. The tentative conclusion is that
MADV_NOHUGEPAGEmust continue to mean "do not fault in more than is absolutely necessary". I would assume we need to extend that thinking to the process-wide and system-wide knobs (as is done in the patch), but we didn't explicitly say so in the meeting.Quoted 7 lines
There are cases that THP is not appreciated because of the latency or memory consumption. For these cases, large folio may fill the gap as less latency and memory consumption.
So if disabling THP means large folio can't be used, we loose the chance to benefit those cases with large folio.
Quoted 6 lines
Yes, I appreciate that. But there are also real use cases that expect
MADV_NOHUGEPAGEmeans "do not fault more than is absolutely necessary" and the use cases break if that's not obeyed (e.g. live migration w/ qemu). So I think we need to be conservitive to start. These apps that are explicitly forbidding THP today, should be updated in the long run to opt-in to large anon folios using some as-yet undefined control.Fair enough.
Regards
Yin, FengweiQuoted 31 lines
Quoted 30 lines
Regards
Yin, FengweiQuoted 24 lines
My intention is that if you have requested THP and your vma is big enough for PMD-size then you get that, else you fallback to large anon folios. And if you have neither opted in nor out, then you get large anon folios.
We talked about the idea of adding a new knob that let's you set the max order, but that needs a lot more thought.
Anyway, as I said, I'll write it up so we can all systematically discuss.
Quoted 13 lines
So if it's "never", large folio is disabled. But why "madvise" enables large folio unconditionly? Suppose it's only enabled for the VMA range which user madvise large folio (or THP)?
Specific for the hink setting, my understand is that we can't choose it only by this testing. Other workloads may have different behavior with differnt hink setting.
Regards
Yin, Fengwei
Re: [PATCH v4 2/5] mm: LARGE_ANON_FOLIO for improved performance
In reply to Yin FengweiQuoted 201 lines · Yin Fengwei
Quoted 167 lines · Ryan Roberts
Quoted 142 lines · Yu Zhao
Quoted 141 lines · Yu Zhao
Quoted 139 lines · Ryan Roberts
Introduce
LARGE_ANON_FOLIOfeature, which allows anonymous memory to be allocated in large folios of a determined order. All pages of the large folio are pte-mapped during the same page fault, significantly reducing the number of page faults. The number of per-page operations (e.g. ref counting, rmap management lru list management) are also significantly reduced since those ops now become per-folio.The new behaviour is hidden behind the new
LARGE_ANON_FOLIOKconfig, which defaults to disabled for now; The long term aim is for this to defaut to enabled, but there are some risks around internal fragmentation that need to be better understood first.When enabled, the folio order is determined as such: For a vma, process or system that has explicitly disabled THP, we continue to allocate order-0. THP is most likely disabled to avoid any possible internal fragmentation so we honour that request.
Otherwise, the return value of
arch_wants_pte_order()is used. For vmas that have not explicitly opted-in to use transparent hugepages (e.g. where thp=madvise and the vma does not haveMADV_HUGEPAGE), thenarch_wants_pte_order()is limited to 64K (orPAGE_SIZE, whichever is bigger). This allows for a performance boost without requiring any explicit opt-in from the workload while limitting internal fragmentation.If the preferred order can't be used (e.g. because the folio would breach the bounds of the vma, or because ptes in the region are already mapped) then we fall back to a suitable lower order; first
PAGE_ALLOC_COSTLY_ORDER, then order-0.
arch_wants_pte_order()can be overridden by the architecture if desired. Some architectures (e.g. arm64) can coalsece TLB entries if a contiguous set of ptes map physically contigious, naturally aligned memory, so this mechanism allows the architecture to optimize as required.Here we add the default implementation of
arch_wants_pte_order(), used when the architecture does not define it, which returns -1, implying that the HW has no preference. In this case, mm will choose it's own default order.Signed-off-by Ryan Roberts <[email protected]>--- include/linux/pgtable.h | 13 ++++ mm/Kconfig | 10 +++ mm/memory.c | 166 ++++++++++++++++++++++++++++++++++++---- 3 files changed, 172 insertions(+), 17 deletions(-)include/linux/pgtable.h +13
-0@@ -313,6 +313,19 @@ static inline bool arch_has_hw_pte_young(void) }#endif +#ifndef arch_wants_pte_order +/* +* Returns preferred folio order for pte-mapped memory. Must be in range [0, +* PMD_SHIFT-PAGE_SHIFT) and must not be order-1 since THP requires large folios +* to be at least order-2. Negative value implies that the HW has no preference +* and mm will choose it's own default order. +*/ +static inline int arch_wants_pte_order (void ) +{ +return -1 ; +} +#endif +#ifndef __HAVE_ARCH_PTEP_GET_AND_CLEAR static inline pte_t ptep_get_and_clear(struct mm_struct *mm, unsigned long address,mm/Kconfig +10
-0@@ -1238,4 +1238,14 @@ config LOCK_MM_AND_FIND_VMA source "mm/damon/Kconfig" +config LARGE_ANON_FOLIO + bool "Allocate large folios for anonymous memory" + depends on TRANSPARENT_HUGEPAGE + default n + help + Use large (bigger than order-0) folios to back anonymous memory where + possible, even for pte-mapped memory. This reduces the number of page + faults, as well as other per-page overheads to improve performance for + many workloads. + endmenumm/memory.c +40
-0@@ -4050,6 +4050,127 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)return ret; } +static bool vmf_pte_range_changed (struct vm_fault *vmf,int nr_pages) +{ +int i; + +if (nr_pages ==1 ) +return vmf_pte_changed (vmf); + +for (i =0 ; i < nr_pages; i++) { +if (!pte_none (ptep_get_lockless (vmf->pte + i))) +return true; + } + +return false; +} + +#ifdef CONFIG_LARGE_ANON_FOLIO +#define ANON_FOLIO_MAX_ORDER_UNHINTED \ + (ilog2(max_t(unsigned long, SZ_64K, PAGE_SIZE)) - PAGE_SHIFT) + +static int anon_folio_order (struct vm_area_struct *vma) +{ +int order; + +/* +* If THP is explicitly disabled for either the vma, the process or the +* system, then this is very likely intended to limit internal +* fragmentation; in this case, don't attempt to allocate a large +* anonymous folio. +* +* Else, if the vma is eligible for thp, allocate a large folio of the +* size preferred by the arch. Or if the arch requested a very small +* size or didn't request a size, then use PAGE_ALLOC_COSTLY_ORDER, +* which still meets the arch's requirements but means we still take +* advantage of SW optimizations (e.g. fewer page faults). +* +* Finally if thp is enabled but the vma isn't eligible, take the +* arch-preferred size and limit it to ANON_FOLIO_MAX_ORDER_UNHINTED. +* This ensures workloads that have not explicitly opted-in take benefit +* while capping the potential for internal fragmentation. +*/ Quoted 8 lines
Quoted 8 lines
What empirical evidence is
SZ_64Kbased on?
What workloads would benefit from it?
How much would they benefit from it?
Would they benefit more or less from different values? How much internal fragmentation would it cause?
What cost function was used to arrive at the conclusion that its benefits outweigh its costs?Sorry this has taken a little while to reply to; I've been re-running my perf tests with the modern patches to recomfirm old data.
In terms of empirical evidence, I've run the kernel compilation benchmark (yes I know its a narrow use case, but I figure some data is better than no data), for all values of
ANON_FOLIO_MAX_ORDER_UNHINTED{4k, 16k, 32k, 64k, 128k, 256k}.I've run each test 15 times across 5 system reboots on Ampere Altra (arm64), with the kernel configured for 4K base pages - I could rerun for other base page sizes if we want to go further down this route.
I've captured run time and peak memory usage, and taken the mean. The stdev for the peak memory usage is big-ish, but I'm confident this still captures the central tendancy well:
Quoted 8 lines
| MAX_ORDER_UNHINTED | real-time | kern-time | user-time | peak memory | |:-------------------|------------:|------------:|------------:|:------------| | 4k | 0.0% | 0.0% | 0.0% | 0.0% | | 16k | -3.6% | -26.5% | -0.5% | -0.1% | | 32k | -4.8% | -37.4% | -0.6% | -0.1% | | 64k | -5.7% | -42.0% | -0.6% | -1.1% | | 128k | -5.6% | -42.1% | -0.7% | 1.4% | | 256k | -4.9% | -41.9% | -0.4% | 1.9% |Here is my test result:
real user sys hink-4k: 0% 0% 0% hink-16K: -3% 0.1% -18.3% hink-32K: -4% 0.2% -27.2% hink-64K: -4% 0.5% -31.0% hink-128K: -4% 0.9% -33.7% hink-256K: -5% 1% -34.6%I used command:
/usr/bin/time-f "\t%E real,\t%U user,\t%S sys" make -skj96 allmodconfig all to build kernel and collect the real time/user time/kernel time./sys/kernel/mm/transparent_hugepage/enabledis "madvise".
Let me know if you have any question about the test.I also find one strange behavior with this version. It's related with why I need to set the
/sys/kernel/mm/transparent_hugepage/enabledto "madvise". If it's "never", the large folio is disabled either.
If it's "always", the THP will be active before large folio. So the system is in the mixed mode. it's not suitable for this test.
Quoted 3 lines
So if it's "never", large folio is disabled. But why "madvise" enables large folio unconditionly? Suppose it's only enabled for the VMA range which user madvise large folio (or THP)?
Indeed. It's a very peculiar behavior, as I called out in another email
Re: [PATCH v4 2/5] mm: LARGE_ANON_FOLIO for improved performance
In reply to Ryan RobertsQuoted 162 lines · Ryan Roberts
Introduce
LARGE_ANON_FOLIOfeature, which allows anonymous memory to be allocated in large folios of a determined order. All pages of the large folio are pte-mapped during the same page fault, significantly reducing the number of page faults. The number of per-page operations (e.g. ref counting, rmap management lru list management) are also significantly reduced since those ops now become per-folio.The new behaviour is hidden behind the new
LARGE_ANON_FOLIOKconfig, which defaults to disabled for now; The long term aim is for this to defaut to enabled, but there are some risks around internal fragmentation that need to be better understood first.When enabled, the folio order is determined as such: For a vma, process or system that has explicitly disabled THP, we continue to allocate order-0. THP is most likely disabled to avoid any possible internal fragmentation so we honour that request.
Otherwise, the return value of
arch_wants_pte_order()is used. For vmas that have not explicitly opted-in to use transparent hugepages (e.g. where thp=madvise and the vma does not haveMADV_HUGEPAGE), thenarch_wants_pte_order()is limited to 64K (orPAGE_SIZE, whichever is bigger). This allows for a performance boost without requiring any explicit opt-in from the workload while limitting internal fragmentation.If the preferred order can't be used (e.g. because the folio would breach the bounds of the vma, or because ptes in the region are already mapped) then we fall back to a suitable lower order; first
PAGE_ALLOC_COSTLY_ORDER, then order-0.
arch_wants_pte_order()can be overridden by the architecture if desired. Some architectures (e.g. arm64) can coalsece TLB entries if a contiguous set of ptes map physically contigious, naturally aligned memory, so this mechanism allows the architecture to optimize as required.Here we add the default implementation of
arch_wants_pte_order(), used when the architecture does not define it, which returns -1, implying that the HW has no preference. In this case, mm will choose it's own default order.Signed-off-by Ryan Roberts <[email protected]>--- include/linux/pgtable.h | 13 ++++ mm/Kconfig | 10 +++ mm/memory.c | 166 ++++++++++++++++++++++++++++++++++++---- 3 files changed, 172 insertions(+), 17 deletions(-)include/linux/pgtable.h +13
-0@@ -313,6 +313,19 @@ static inline bool arch_has_hw_pte_young(void) }#endif +#ifndef arch_wants_pte_order +/* +* Returns preferred folio order for pte-mapped memory. Must be in range [0, +* PMD_SHIFT-PAGE_SHIFT) and must not be order-1 since THP requires large folios +* to be at least order-2. Negative value implies that the HW has no preference +* and mm will choose it's own default order. +*/ +static inline int arch_wants_pte_order (void ) +{ +return -1 ; +} +#endif +#ifndef __HAVE_ARCH_PTEP_GET_AND_CLEAR static inline pte_t ptep_get_and_clear(struct mm_struct *mm, unsigned long address,mm/Kconfig +10
-0@@ -1238,4 +1238,14 @@ config LOCK_MM_AND_FIND_VMA source "mm/damon/Kconfig" +config LARGE_ANON_FOLIO + bool "Allocate large folios for anonymous memory" + depends on TRANSPARENT_HUGEPAGE + default n + help + Use large (bigger than order-0) folios to back anonymous memory where + possible, even for pte-mapped memory. This reduces the number of page + faults, as well as other per-page overheads to improve performance for + many workloads. + endmenumm/memory.c +63
-0@@ -4050,6 +4050,127 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)return ret; } +static bool vmf_pte_range_changed (struct vm_fault *vmf,int nr_pages) +{ +int i; + +if (nr_pages ==1 ) +return vmf_pte_changed (vmf); + +for (i =0 ; i < nr_pages; i++) { +if (!pte_none (ptep_get_lockless (vmf->pte + i))) +return true; + } + +return false; +} + +#ifdef CONFIG_LARGE_ANON_FOLIO +#define ANON_FOLIO_MAX_ORDER_UNHINTED \ + (ilog2(max_t(unsigned long, SZ_64K, PAGE_SIZE)) - PAGE_SHIFT) + +static int anon_folio_order (struct vm_area_struct *vma) +{ +int order; + +/* +* If THP is explicitly disabled for either the vma, the process or the +* system, then this is very likely intended to limit internal +* fragmentation; in this case, don't attempt to allocate a large +* anonymous folio. +* +* Else, if the vma is eligible for thp, allocate a large folio of the +* size preferred by the arch. Or if the arch requested a very small +* size or didn't request a size, then use PAGE_ALLOC_COSTLY_ORDER, +* which still meets the arch's requirements but means we still take +* advantage of SW optimizations (e.g. fewer page faults). +* +* Finally if thp is enabled but the vma isn't eligible, take the +* arch-preferred size and limit it to ANON_FOLIO_MAX_ORDER_UNHINTED. +* This ensures workloads that have not explicitly opted-in take benefit +* while capping the potential for internal fragmentation. +*/ + +if ((vma->vm_flags &VM_NOHUGEPAGE ) || +test_bit (MMF_DISABLE_THP , &vma->vm_mm->flags) || + !hugepage_flags_enabled ()) + order =0 ; +else { + order =max (arch_wants_pte_order (),PAGE_ALLOC_COSTLY_ORDER ); + +if (!hugepage_vma_check (vma, vma->vm_flags, false, true, true)) + order =min (order,ANON_FOLIO_MAX_ORDER_UNHINTED ); + } + +return order; +} + +static int alloc_anon_folio (struct vm_fault *vmf,struct folio **folio) +{ +int i; +gfp_t gfp; +pte_t *pte; +unsigned long addr; +struct vm_area_struct *vma = vmf->vma; +int prefer =anon_folio_order (vma);
mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 4125
Quoted 10 lines
mm/memory.c +10
-0+int orders[] = { + prefer, + prefer >PAGE_ALLOC_COSTLY_ORDER ?PAGE_ALLOC_COSTLY_ORDER :0 , +0 , + }; + + *folio =NULL ; + +if (vmf_orig_pte_uffd_wp (vmf)) + goto fallback;
I think we need to s/vmf_orig_pte_uffd_wp/userfaultfd_armed/ here; otherwise UFFD would miss VM_UFFD_MISSING/MINOR.
Re: [PATCH v4 2/5] mm: LARGE_ANON_FOLIO for improved performance
Quoted 168 lines · Yu Zhao
Quoted 167 lines · Ryan Roberts
Introduce
LARGE_ANON_FOLIOfeature, which allows anonymous memory to be allocated in large folios of a determined order. All pages of the large folio are pte-mapped during the same page fault, significantly reducing the number of page faults. The number of per-page operations (e.g. ref counting, rmap management lru list management) are also significantly reduced since those ops now become per-folio.The new behaviour is hidden behind the new
LARGE_ANON_FOLIOKconfig, which defaults to disabled for now; The long term aim is for this to defaut to enabled, but there are some risks around internal fragmentation that need to be better understood first.When enabled, the folio order is determined as such: For a vma, process or system that has explicitly disabled THP, we continue to allocate order-0. THP is most likely disabled to avoid any possible internal fragmentation so we honour that request.
Otherwise, the return value of
arch_wants_pte_order()is used. For vmas that have not explicitly opted-in to use transparent hugepages (e.g. where thp=madvise and the vma does not haveMADV_HUGEPAGE), thenarch_wants_pte_order()is limited to 64K (orPAGE_SIZE, whichever is bigger). This allows for a performance boost without requiring any explicit opt-in from the workload while limitting internal fragmentation.If the preferred order can't be used (e.g. because the folio would breach the bounds of the vma, or because ptes in the region are already mapped) then we fall back to a suitable lower order; first
PAGE_ALLOC_COSTLY_ORDER, then order-0.
arch_wants_pte_order()can be overridden by the architecture if desired. Some architectures (e.g. arm64) can coalsece TLB entries if a contiguous set of ptes map physically contigious, naturally aligned memory, so this mechanism allows the architecture to optimize as required.Here we add the default implementation of
arch_wants_pte_order(), used when the architecture does not define it, which returns -1, implying that the HW has no preference. In this case, mm will choose it's own default order.Signed-off-by Ryan Roberts <[email protected]>--- include/linux/pgtable.h | 13 ++++ mm/Kconfig | 10 +++ mm/memory.c | 166 ++++++++++++++++++++++++++++++++++++---- 3 files changed, 172 insertions(+), 17 deletions(-)include/linux/pgtable.h +13
-0@@ -313,6 +313,19 @@ static inline bool arch_has_hw_pte_young(void) }#endif +#ifndef arch_wants_pte_order +/* +* Returns preferred folio order for pte-mapped memory. Must be in range [0, +* PMD_SHIFT-PAGE_SHIFT) and must not be order-1 since THP requires large folios +* to be at least order-2. Negative value implies that the HW has no preference +* and mm will choose it's own default order. +*/ +static inline int arch_wants_pte_order (void ) +{ +return -1 ; +} +#endif +#ifndef __HAVE_ARCH_PTEP_GET_AND_CLEAR static inline pte_t ptep_get_and_clear(struct mm_struct *mm, unsigned long address,mm/Kconfig +10
-0@@ -1238,4 +1238,14 @@ config LOCK_MM_AND_FIND_VMA source "mm/damon/Kconfig" +config LARGE_ANON_FOLIO + bool "Allocate large folios for anonymous memory" + depends on TRANSPARENT_HUGEPAGE + default n + help + Use large (bigger than order-0) folios to back anonymous memory where + possible, even for pte-mapped memory. This reduces the number of page + faults, as well as other per-page overheads to improve performance for + many workloads. + endmenumm/memory.c +68
-0@@ -4050,6 +4050,127 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)return ret; } +static bool vmf_pte_range_changed (struct vm_fault *vmf,int nr_pages) +{ +int i; + +if (nr_pages ==1 ) +return vmf_pte_changed (vmf); + +for (i =0 ; i < nr_pages; i++) { +if (!pte_none (ptep_get_lockless (vmf->pte + i))) +return true; + } + +return false; +} + +#ifdef CONFIG_LARGE_ANON_FOLIO +#define ANON_FOLIO_MAX_ORDER_UNHINTED \ + (ilog2(max_t(unsigned long, SZ_64K, PAGE_SIZE)) - PAGE_SHIFT) + +static int anon_folio_order (struct vm_area_struct *vma) +{ +int order; + +/* +* If THP is explicitly disabled for either the vma, the process or the +* system, then this is very likely intended to limit internal +* fragmentation; in this case, don't attempt to allocate a large +* anonymous folio. +* +* Else, if the vma is eligible for thp, allocate a large folio of the +* size preferred by the arch. Or if the arch requested a very small +* size or didn't request a size, then use PAGE_ALLOC_COSTLY_ORDER, +* which still meets the arch's requirements but means we still take +* advantage of SW optimizations (e.g. fewer page faults). +* +* Finally if thp is enabled but the vma isn't eligible, take the +* arch-preferred size and limit it to ANON_FOLIO_MAX_ORDER_UNHINTED. +* This ensures workloads that have not explicitly opted-in take benefit +* while capping the potential for internal fragmentation. +*/ + +if ((vma->vm_flags &VM_NOHUGEPAGE ) || +test_bit (MMF_DISABLE_THP , &vma->vm_mm->flags) || + !hugepage_flags_enabled ()) + order =0 ; +else { + order =max (arch_wants_pte_order (),PAGE_ALLOC_COSTLY_ORDER ); + +if (!hugepage_vma_check (vma, vma->vm_flags, false, true, true)) + order =min (order,ANON_FOLIO_MAX_ORDER_UNHINTED ); + } + +return order; +} + +static int alloc_anon_folio (struct vm_fault *vmf,struct folio **folio) +{ +int i; +gfp_t gfp; +pte_t *pte; +unsigned long addr; +struct vm_area_struct *vma = vmf->vma; +int prefer =anon_folio_order (vma); +int orders[] = { + prefer, + prefer >PAGE_ALLOC_COSTLY_ORDER ?PAGE_ALLOC_COSTLY_ORDER :0 , +0 , + };
Quoted 8 lines
mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 4125
Quoted 5 lines
mm/memory.c +5
-0+ + *folio =NULL ; + +if (vmf_orig_pte_uffd_wp (vmf)) + goto fallback;I think we need to s/vmf_orig_pte_uffd_wp/userfaultfd_armed/ here; otherwise UFFD would miss VM_UFFD_MISSING/MINOR.
I don't think this is the case. As far as I can see, VM_UFFD_MINOR only applies to shmem and hugetlb. VM_UFFD_MISSING is checked under the PTL and if set on the VMA, then it is handled without mapping the folio that was just allocated:
/* Deliver the page fault to userland, check inside PT lock */
if (userfaultfd_missing(vma)) {
pte_unmap_unlock(vmf->pte, vmf->ptl);
folio_put(folio);
return handle_userfault(vmf, VM_UFFD_MISSING);
}So we are racing to allocate a large folio; if the vma later turns out to have MISSING handling registered, we drop the folio and handle it, else we map the large folio.
The vmf_orig_pte_uffd_wp() *is* required because we need to individually check each PTE for the uffd_wp bit and fix it up.
So I think the code is correct, but perhaps it is safer/simpler to always avoid allocating a large folio if the vma is registered for uffd in the way you suggest? I don't know enough about uffd to form a strong opinion either way.
Re: [PATCH v4 2/5] mm: LARGE_ANON_FOLIO for improved performance
Quoted 179 lines · Ryan Roberts
Quoted 168 lines · Yu Zhao
Quoted 167 lines · Ryan Roberts
Introduce
LARGE_ANON_FOLIOfeature, which allows anonymous memory to be allocated in large folios of a determined order. All pages of the large folio are pte-mapped during the same page fault, significantly reducing the number of page faults. The number of per-page operations (e.g. ref counting, rmap management lru list management) are also significantly reduced since those ops now become per-folio.The new behaviour is hidden behind the new
LARGE_ANON_FOLIOKconfig, which defaults to disabled for now; The long term aim is for this to defaut to enabled, but there are some risks around internal fragmentation that need to be better understood first.When enabled, the folio order is determined as such: For a vma, process or system that has explicitly disabled THP, we continue to allocate order-0. THP is most likely disabled to avoid any possible internal fragmentation so we honour that request.
Otherwise, the return value of
arch_wants_pte_order()is used. For vmas that have not explicitly opted-in to use transparent hugepages (e.g. where thp=madvise and the vma does not haveMADV_HUGEPAGE), thenarch_wants_pte_order()is limited to 64K (orPAGE_SIZE, whichever is bigger). This allows for a performance boost without requiring any explicit opt-in from the workload while limitting internal fragmentation.If the preferred order can't be used (e.g. because the folio would breach the bounds of the vma, or because ptes in the region are already mapped) then we fall back to a suitable lower order; first
PAGE_ALLOC_COSTLY_ORDER, then order-0.
arch_wants_pte_order()can be overridden by the architecture if desired. Some architectures (e.g. arm64) can coalsece TLB entries if a contiguous set of ptes map physically contigious, naturally aligned memory, so this mechanism allows the architecture to optimize as required.Here we add the default implementation of
arch_wants_pte_order(), used when the architecture does not define it, which returns -1, implying that the HW has no preference. In this case, mm will choose it's own default order.Signed-off-by Ryan Roberts <[email protected]>--- include/linux/pgtable.h | 13 ++++ mm/Kconfig | 10 +++ mm/memory.c | 166 ++++++++++++++++++++++++++++++++++++---- 3 files changed, 172 insertions(+), 17 deletions(-)include/linux/pgtable.h +13
-0@@ -313,6 +313,19 @@ static inline bool arch_has_hw_pte_young(void) }#endif +#ifndef arch_wants_pte_order +/* +* Returns preferred folio order for pte-mapped memory. Must be in range [0, +* PMD_SHIFT-PAGE_SHIFT) and must not be order-1 since THP requires large folios +* to be at least order-2. Negative value implies that the HW has no preference +* and mm will choose it's own default order. +*/ +static inline int arch_wants_pte_order (void ) +{ +return -1 ; +} +#endif +#ifndef __HAVE_ARCH_PTEP_GET_AND_CLEAR static inline pte_t ptep_get_and_clear(struct mm_struct *mm, unsigned long address,mm/Kconfig +10
-0@@ -1238,4 +1238,14 @@ config LOCK_MM_AND_FIND_VMA source "mm/damon/Kconfig" +config LARGE_ANON_FOLIO + bool "Allocate large folios for anonymous memory" + depends on TRANSPARENT_HUGEPAGE + default n + help + Use large (bigger than order-0) folios to back anonymous memory where + possible, even for pte-mapped memory. This reduces the number of page + faults, as well as other per-page overheads to improve performance for + many workloads. + endmenumm/memory.c +68
-0@@ -4050,6 +4050,127 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)return ret; } +static bool vmf_pte_range_changed (struct vm_fault *vmf,int nr_pages) +{ +int i; + +if (nr_pages ==1 ) +return vmf_pte_changed (vmf); + +for (i =0 ; i < nr_pages; i++) { +if (!pte_none (ptep_get_lockless (vmf->pte + i))) +return true; + } + +return false; +} + +#ifdef CONFIG_LARGE_ANON_FOLIO +#define ANON_FOLIO_MAX_ORDER_UNHINTED \ + (ilog2(max_t(unsigned long, SZ_64K, PAGE_SIZE)) - PAGE_SHIFT) + +static int anon_folio_order (struct vm_area_struct *vma) +{ +int order; + +/* +* If THP is explicitly disabled for either the vma, the process or the +* system, then this is very likely intended to limit internal +* fragmentation; in this case, don't attempt to allocate a large +* anonymous folio. +* +* Else, if the vma is eligible for thp, allocate a large folio of the +* size preferred by the arch. Or if the arch requested a very small +* size or didn't request a size, then use PAGE_ALLOC_COSTLY_ORDER, +* which still meets the arch's requirements but means we still take +* advantage of SW optimizations (e.g. fewer page faults). +* +* Finally if thp is enabled but the vma isn't eligible, take the +* arch-preferred size and limit it to ANON_FOLIO_MAX_ORDER_UNHINTED. +* This ensures workloads that have not explicitly opted-in take benefit +* while capping the potential for internal fragmentation. +*/ + +if ((vma->vm_flags &VM_NOHUGEPAGE ) || +test_bit (MMF_DISABLE_THP , &vma->vm_mm->flags) || + !hugepage_flags_enabled ()) + order =0 ; +else { + order =max (arch_wants_pte_order (),PAGE_ALLOC_COSTLY_ORDER ); + +if (!hugepage_vma_check (vma, vma->vm_flags, false, true, true)) + order =min (order,ANON_FOLIO_MAX_ORDER_UNHINTED ); + } + +return order; +} + +static int alloc_anon_folio (struct vm_fault *vmf,struct folio **folio) +{ +int i; +gfp_t gfp; +pte_t *pte; +unsigned long addr; +struct vm_area_struct *vma = vmf->vma; +int prefer =anon_folio_order (vma); +int orders[] = { + prefer, + prefer >PAGE_ALLOC_COSTLY_ORDER ?PAGE_ALLOC_COSTLY_ORDER :0 , +0 , + };Quoted 8 lines
mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 4125
Quoted 5 lines
mm/memory.c +5
-0+ + *folio =NULL ; + +if (vmf_orig_pte_uffd_wp (vmf)) + goto fallback;I think we need to s/vmf_orig_pte_uffd_wp/userfaultfd_armed/ here; otherwise UFFD would miss VM_UFFD_MISSING/MINOR.
Quoted 2 lines
I don't think this is the case. As far as I can see,
VM_UFFD_MINORonly applies to shmem and hugetlb.
Correct, but we don't have a helper to check against (VM_UFFD_WP | VM_UFFD_MISSING). Reusing userfaultfd_armed() seems cleaner to me or even future proof.
Quoted 10 lines
VM_UFFD_MISSINGis checked under the PTL and if set on the VMA, then it is handled without mapping the folio that was just allocated:/* Deliver the page fault to userland, check inside PT lock */ if (userfaultfd_missing(vma)) { pte_unmap_unlock(vmf->pte, vmf->ptl); folio_put(folio); return handle_userfault(vmf, VM_UFFD_MISSING); }
Quoted 3 lines
So we are racing to allocate a large folio; if the vma later turns out to have MISSING handling registered, we drop the folio and handle it, else we map the large folio.
Yes, then we have allocated a large folio (with great effort if under heavy memory pressure) for nothing.
Quoted 2 lines
The
vmf_orig_pte_uffd_wp()*is* required because we need to individually check each PTE for theuffd_wpbit and fix it up.
This is not correct: we cannot see a WP PTE before you see VM_UFFD_WP. So checking VM_UFFD_WP is perfectly safe.
The reason we might want to check individual PTEs is because WP can be done to a subrange of a VMA that has VM_UFFD_WP, which I don't think is the common case and worth considering here. But if you want to keep it, that's fine with me. Without some comments, the next person might find these two checks confusing though, if you plan to add both.
Quoted 3 lines
So I think the code is correct, but perhaps it is safer/simpler to always avoid allocating a large folio if the vma is registered for uffd in the way you suggest? I don't know enough about uffd to form a strong opinion either way.
Yes, it's not about correctness. Just a second thought about not allocating large folios unnecessarily when possible.
Re: [PATCH v4 2/5] mm: LARGE_ANON_FOLIO for improved performance
Quoted 201 lines · Yu Zhao
Quoted 179 lines · Ryan Roberts
Quoted 168 lines · Yu Zhao
Quoted 167 lines · Ryan Roberts
Introduce
LARGE_ANON_FOLIOfeature, which allows anonymous memory to be allocated in large folios of a determined order. All pages of the large folio are pte-mapped during the same page fault, significantly reducing the number of page faults. The number of per-page operations (e.g. ref counting, rmap management lru list management) are also significantly reduced since those ops now become per-folio.The new behaviour is hidden behind the new
LARGE_ANON_FOLIOKconfig, which defaults to disabled for now; The long term aim is for this to defaut to enabled, but there are some risks around internal fragmentation that need to be better understood first.When enabled, the folio order is determined as such: For a vma, process or system that has explicitly disabled THP, we continue to allocate order-0. THP is most likely disabled to avoid any possible internal fragmentation so we honour that request.
Otherwise, the return value of
arch_wants_pte_order()is used. For vmas that have not explicitly opted-in to use transparent hugepages (e.g. where thp=madvise and the vma does not haveMADV_HUGEPAGE), thenarch_wants_pte_order()is limited to 64K (orPAGE_SIZE, whichever is bigger). This allows for a performance boost without requiring any explicit opt-in from the workload while limitting internal fragmentation.If the preferred order can't be used (e.g. because the folio would breach the bounds of the vma, or because ptes in the region are already mapped) then we fall back to a suitable lower order; first
PAGE_ALLOC_COSTLY_ORDER, then order-0.
arch_wants_pte_order()can be overridden by the architecture if desired. Some architectures (e.g. arm64) can coalsece TLB entries if a contiguous set of ptes map physically contigious, naturally aligned memory, so this mechanism allows the architecture to optimize as required.Here we add the default implementation of
arch_wants_pte_order(), used when the architecture does not define it, which returns -1, implying that the HW has no preference. In this case, mm will choose it's own default order.Signed-off-by Ryan Roberts <[email protected]>--- include/linux/pgtable.h | 13 ++++ mm/Kconfig | 10 +++ mm/memory.c | 166 ++++++++++++++++++++++++++++++++++++---- 3 files changed, 172 insertions(+), 17 deletions(-)include/linux/pgtable.h +13
-0@@ -313,6 +313,19 @@ static inline bool arch_has_hw_pte_young(void) }#endif +#ifndef arch_wants_pte_order +/* +* Returns preferred folio order for pte-mapped memory. Must be in range [0, +* PMD_SHIFT-PAGE_SHIFT) and must not be order-1 since THP requires large folios +* to be at least order-2. Negative value implies that the HW has no preference +* and mm will choose it's own default order. +*/ +static inline int arch_wants_pte_order (void ) +{ +return -1 ; +} +#endif +#ifndef __HAVE_ARCH_PTEP_GET_AND_CLEAR static inline pte_t ptep_get_and_clear(struct mm_struct *mm, unsigned long address,mm/Kconfig +10
-0@@ -1238,4 +1238,14 @@ config LOCK_MM_AND_FIND_VMA source "mm/damon/Kconfig" +config LARGE_ANON_FOLIO + bool "Allocate large folios for anonymous memory" + depends on TRANSPARENT_HUGEPAGE + default n + help + Use large (bigger than order-0) folios to back anonymous memory where + possible, even for pte-mapped memory. This reduces the number of page + faults, as well as other per-page overheads to improve performance for + many workloads. + endmenumm/memory.c +68
-0@@ -4050,6 +4050,127 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)return ret; } +static bool vmf_pte_range_changed (struct vm_fault *vmf,int nr_pages) +{ +int i; + +if (nr_pages ==1 ) +return vmf_pte_changed (vmf); + +for (i =0 ; i < nr_pages; i++) { +if (!pte_none (ptep_get_lockless (vmf->pte + i))) +return true; + } + +return false; +} + +#ifdef CONFIG_LARGE_ANON_FOLIO +#define ANON_FOLIO_MAX_ORDER_UNHINTED \ + (ilog2(max_t(unsigned long, SZ_64K, PAGE_SIZE)) - PAGE_SHIFT) + +static int anon_folio_order (struct vm_area_struct *vma) +{ +int order; + +/* +* If THP is explicitly disabled for either the vma, the process or the +* system, then this is very likely intended to limit internal +* fragmentation; in this case, don't attempt to allocate a large +* anonymous folio. +* +* Else, if the vma is eligible for thp, allocate a large folio of the +* size preferred by the arch. Or if the arch requested a very small +* size or didn't request a size, then use PAGE_ALLOC_COSTLY_ORDER, +* which still meets the arch's requirements but means we still take +* advantage of SW optimizations (e.g. fewer page faults). +* +* Finally if thp is enabled but the vma isn't eligible, take the +* arch-preferred size and limit it to ANON_FOLIO_MAX_ORDER_UNHINTED. +* This ensures workloads that have not explicitly opted-in take benefit +* while capping the potential for internal fragmentation. +*/ + +if ((vma->vm_flags &VM_NOHUGEPAGE ) || +test_bit (MMF_DISABLE_THP , &vma->vm_mm->flags) || + !hugepage_flags_enabled ()) + order =0 ; +else { + order =max (arch_wants_pte_order (),PAGE_ALLOC_COSTLY_ORDER ); + +if (!hugepage_vma_check (vma, vma->vm_flags, false, true, true)) + order =min (order,ANON_FOLIO_MAX_ORDER_UNHINTED ); + } + +return order; +} + +static int alloc_anon_folio (struct vm_fault *vmf,struct folio **folio) +{ +int i; +gfp_t gfp; +pte_t *pte; +unsigned long addr; +struct vm_area_struct *vma = vmf->vma; +int prefer =anon_folio_order (vma); +int orders[] = { + prefer, + prefer >PAGE_ALLOC_COSTLY_ORDER ?PAGE_ALLOC_COSTLY_ORDER :0 , +0 , + };Quoted 8 lines
mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 4125
Quoted 5 lines
mm/memory.c +5
-0+ + *folio =NULL ; + +if (vmf_orig_pte_uffd_wp (vmf)) + goto fallback;I think we need to s/vmf_orig_pte_uffd_wp/userfaultfd_armed/ here; otherwise UFFD would miss VM_UFFD_MISSING/MINOR.
Quoted 2 lines
I don't think this is the case. As far as I can see,
VM_UFFD_MINORonly applies to shmem and hugetlb.Correct, but we don't have a helper to check against (
VM_UFFD_WP|VM_UFFD_MISSING). Reusinguserfaultfd_armed()seems cleaner to me or even future proof.Quoted 10 lines
VM_UFFD_MISSINGis checked under the PTL and if set on the VMA, then it is handled without mapping the folio that was just allocated:/* Deliver the page fault to userland, check inside PT lock */ if (userfaultfd_missing(vma)) { pte_unmap_unlock(vmf->pte, vmf->ptl); folio_put(folio); return handle_userfault(vmf, VM_UFFD_MISSING); }Quoted 3 lines
So we are racing to allocate a large folio; if the vma later turns out to have MISSING handling registered, we drop the folio and handle it, else we map the large folio.
Quoted 8 lines
Yes, then we have allocated a large folio (with great effort if under heavy memory pressure) for nothing.
Quoted 2 lines
The
vmf_orig_pte_uffd_wp()*is* required because we need to individually check each PTE for theuffd_wpbit and fix it up.This is not correct: we cannot see a WP PTE before you see
VM_UFFD_WP. So checkingVM_UFFD_WPis perfectly safe.
I think you misunderstood me; I was trying to say that assuming we don't check userfaultfd_armed() then we need the vmf_orig_pte_uffd_wp() check because we need to ensure that the marker gets preserved for that specific pte and we can only do that if we are operating on a single pte.
Quoted 6 lines
The reason we might want to check individual PTEs is because WP can be done to a subrange of a VMA that has
VM_UFFD_WP, which I don't think is the common case and worth considering here. But if you want to keep it, that's fine with me. Without some comments, the next person might find these two checks confusing though, if you plan to add both.
I'm not proposing we need both checks.
Quoted 7 lines
Quoted 4 lines
So I think the code is correct, but perhaps it is safer/simpler to always avoid allocating a large folio if the vma is registered for uffd in the way you suggest? I don't know enough about uffd to form a strong opinion either way.
Yes, it's not about correctness. Just a second thought about not allocating large folios unnecessarily when possible.
OK, I misunderstood you; I thought your original point is about correctness.
Anyway, you have convinced me that we should s/vmf_orig_pte_uffd_wp/userfaultfd_armed/ on the grounds that trying hard to allocate a high order folio is almost always going to be a waste of effort. I'll change this in the next version.
Thanks,
Ryan
Re: [PATCH v4 2/5] mm: LARGE_ANON_FOLIO for improved performance
In reply to Ryan Roberts+ Kirill
Quoted 30 lines · Ryan Roberts
Introduce
LARGE_ANON_FOLIOfeature, which allows anonymous memory to be allocated in large folios of a determined order. All pages of the large folio are pte-mapped during the same page fault, significantly reducing the number of page faults. The number of per-page operations (e.g. ref counting, rmap management lru list management) are also significantly reduced since those ops now become per-folio.The new behaviour is hidden behind the new
LARGE_ANON_FOLIOKconfig, which defaults to disabled for now; The long term aim is for this to defaut to enabled, but there are some risks around internal fragmentation that need to be better understood first.When enabled, the folio order is determined as such: For a vma, process or system that has explicitly disabled THP, we continue to allocate order-0. THP is most likely disabled to avoid any possible internal fragmentation so we honour that request.
Otherwise, the return value of
arch_wants_pte_order()is used. For vmas that have not explicitly opted-in to use transparent hugepages (e.g. where thp=madvise and the vma does not haveMADV_HUGEPAGE), thenarch_wants_pte_order()is limited to 64K (orPAGE_SIZE, whichever is bigger). This allows for a performance boost without requiring any explicit opt-in from the workload while limitting internal fragmentation.If the preferred order can't be used (e.g. because the folio would breach the bounds of the vma, or because ptes in the region are already mapped) then we fall back to a suitable lower order; first
PAGE_ALLOC_COSTLY_ORDER, then order-0.
...
Quoted 28 lines
mm/memory.c +28
-0+#define ANON_FOLIO_MAX_ORDER_UNHINTED \ + (ilog2(max_t(unsigned long, SZ_64K, PAGE_SIZE)) - PAGE_SHIFT) + +static int anon_folio_order (struct vm_area_struct *vma) +{ +int order; + +/* +* If THP is explicitly disabled for either the vma, the process or the +* system, then this is very likely intended to limit internal +* fragmentation; in this case, don't attempt to allocate a large +* anonymous folio. +* +* Else, if the vma is eligible for thp, allocate a large folio of the +* size preferred by the arch. Or if the arch requested a very small +* size or didn't request a size, then use PAGE_ALLOC_COSTLY_ORDER, +* which still meets the arch's requirements but means we still take +* advantage of SW optimizations (e.g. fewer page faults). +* +* Finally if thp is enabled but the vma isn't eligible, take the +* arch-preferred size and limit it to ANON_FOLIO_MAX_ORDER_UNHINTED. +* This ensures workloads that have not explicitly opted-in take benefit +* while capping the potential for internal fragmentation. +*/ + +if ((vma->vm_flags &VM_NOHUGEPAGE ) || +test_bit (MMF_DISABLE_THP , &vma->vm_mm->flags) || + !hugepage_flags_enabled ())
mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 4106
Quoted 10 lines
mm/memory.c +10
-0+ order =0 ; + else { + order =max (arch_wants_pte_order (),PAGE_ALLOC_COSTLY_ORDER ); + +if (!hugepage_vma_check (vma, vma->vm_flags, false, true, true)) + order =min (order,ANON_FOLIO_MAX_ORDER_UNHINTED ); + } + +return order; +}
Hi All,
I'm writing up the conclusions that we arrived at during discussion in the THP meeting yesterday, regarding linkage with exiting THP ABIs. It would be great if I can get explicit "agree" or disagree + rationale from at least David, Yu and Kirill.
In summary; I think we are converging on the approach that is already coded, but I'd like confirmation.
The THP situation today
-----------------------
- At system level: THP can be set to "never", "madvise" or "always"
- At process level: THP can be "never" or "defer to system setting"
- At VMA level: no-hint, MADV_HUGEPAGE, MADV_NOHUGEPAGE
That gives us this table to describe how a page fault is handled, according to process state (columns) and vma flags (rows):
| never | madvise | always ----------------|-----------|-----------|----------- no hint | S | S | THP>S MADV_HUGEPAGE | S | THP>S | THP>S MADV_NOHUGEPAGE | S | S | S
Legend:
S allocate single page (PTE-mapped)
LAF allocate lage anon folio (PTE-mapped)
THP allocate THP-sized folio (PMD-mapped)
Quoted 1 line
fallback (usually because vma size/alignment insufficient for folio)
Principles for Large Anon Folios (LAF)
--------------------------------------
David tells us there are use cases today (e.g. qemu live migration) which use MADV_NOHUGEPAGE to mean "don't fill any PTEs that are not explicitly faulted" and these use cases will break (i.e. functionally incorrect) if this request is not honoured.
So LAF must at least honour MADV_NOHUGEPAGE to prevent breaking existing use cases. And once we do this, then I think the least confusing thing is for it to also honor the "never" system/process state; so if either the system, process or vma has explicitly opted-out of THP, then LAF should also be bypassed.
Similarly, any case that would previously cause the allocation of PMD-sized THP must continue to be honoured, else we risk performance regression.
That leaves the "madvise/no-hint" case, and all THP fallback paths due to the VMA not being correctly aligned or sized to hold a PMD-sized mapping. In these cases, we will attempt to use LAF first, and fallback to single page if the vma size/alignment doesn't permit it.
| never | madvise | always ----------------|-----------|-----------|----------- no hint | S | LAF>S | THP>LAF>S MADV_HUGEPAGE | S | THP>LAF>S | THP>LAF>S MADV_NOHUGEPAGE | S | S | S
I think this (perhaps conservative) approach will be the least surprising to users. And is the policy that is already implemented in this patch.
Downsides of this policy
------------------------
As Yu and Yin have pointed out, there are some workloads which do not perform well with THP, due to large fault latency or memory wastage, etc. But which _may_ still benefit from LAF. By taking the conservative approach, we exclude these workloads from benefiting automatically.
But given they have explicitly opted out of THP, it doesn't seem unreasonable that those workloads should be explicitly modified to opt-in to LAF. The question is what should a control for this look like? And do we need to implement the control for an MVP implementation of LAF? For the latter question, I would suggest this can come later - its a tool to further optimize, but its absence does not regress today's performance.
What should a control look like?
One suggestion was to expose a "maximum order" tunable, which would limit the size of THP that could be allocated. setting it to 1M would cause traditional THP to be bypassed (assuming for now PMD-sized THP is 2M) but would permit LAF. But Kirill suggested that this type of control might turn out to be restrictive in the long run.
Another suggestion was to provide a more abstracted hint to the kernel, which the kernel could then derive a policy from, and that policy would be easier to change over time.
Large Anon Folio Size
---------------------
Once we have decided to use LAF (vs THP vs S), we need to decide how big the folio should be. If/when we get a control as described above, that will obviously place an upper bound on the size. HW may also have a preferred size due to tricks it can do in the TLB (arch_wants_pte_order() in this patch) but you may still want to allocate a bigger folio than the HW wants (since bigger folios will reduce page faults) or you may want to allocate a smaller folio than the HW wants (due to concerns about latency or memory wastage).
I've had a stab at addressing this in the patch too, using the same decision as for THP (ignoring the vma size/alignment requirement) to decide if we use the HW preferred order or if we cap it (currently set at 64K).
Thoughts, comments?
Thanks,
Ryan
Re: [PATCH v4 2/5] mm: LARGE_ANON_FOLIO for improved performance
Quoted 133 lines · Ryan Roberts
+ Kirill
Quoted 30 lines · Ryan Roberts
Introduce
LARGE_ANON_FOLIOfeature, which allows anonymous memory to be allocated in large folios of a determined order. All pages of the large folio are pte-mapped during the same page fault, significantly reducing the number of page faults. The number of per-page operations (e.g. ref counting, rmap management lru list management) are also significantly reduced since those ops now become per-folio.The new behaviour is hidden behind the new
LARGE_ANON_FOLIOKconfig, which defaults to disabled for now; The long term aim is for this to defaut to enabled, but there are some risks around internal fragmentation that need to be better understood first.When enabled, the folio order is determined as such: For a vma, process or system that has explicitly disabled THP, we continue to allocate order-0. THP is most likely disabled to avoid any possible internal fragmentation so we honour that request.
Otherwise, the return value of
arch_wants_pte_order()is used. For vmas that have not explicitly opted-in to use transparent hugepages (e.g. where thp=madvise and the vma does not haveMADV_HUGEPAGE), thenarch_wants_pte_order()is limited to 64K (orPAGE_SIZE, whichever is bigger). This allows for a performance boost without requiring any explicit opt-in from the workload while limitting internal fragmentation.If the preferred order can't be used (e.g. because the folio would breach the bounds of the vma, or because ptes in the region are already mapped) then we fall back to a suitable lower order; first
PAGE_ALLOC_COSTLY_ORDER, then order-0....
Quoted 28 lines
mm/memory.c +28
-0+#define ANON_FOLIO_MAX_ORDER_UNHINTED \ + (ilog2(max_t(unsigned long, SZ_64K, PAGE_SIZE)) - PAGE_SHIFT) + +static int anon_folio_order (struct vm_area_struct *vma) +{ +int order; + +/* +* If THP is explicitly disabled for either the vma, the process or the +* system, then this is very likely intended to limit internal +* fragmentation; in this case, don't attempt to allocate a large +* anonymous folio. +* +* Else, if the vma is eligible for thp, allocate a large folio of the +* size preferred by the arch. Or if the arch requested a very small +* size or didn't request a size, then use PAGE_ALLOC_COSTLY_ORDER, +* which still meets the arch's requirements but means we still take +* advantage of SW optimizations (e.g. fewer page faults). +* +* Finally if thp is enabled but the vma isn't eligible, take the +* arch-preferred size and limit it to ANON_FOLIO_MAX_ORDER_UNHINTED. +* This ensures workloads that have not explicitly opted-in take benefit +* while capping the potential for internal fragmentation. +*/ + +if ((vma->vm_flags &VM_NOHUGEPAGE ) || +test_bit (MMF_DISABLE_THP , &vma->vm_mm->flags) || + !hugepage_flags_enabled ())mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 4106
Quoted 10 lines
mm/memory.c +10
-0+ order =0 ; + else { + order =max (arch_wants_pte_order (),PAGE_ALLOC_COSTLY_ORDER ); + +if (!hugepage_vma_check (vma, vma->vm_flags, false, true, true)) + order =min (order,ANON_FOLIO_MAX_ORDER_UNHINTED ); + } + +return order; +}Hi All,
I'm writing up the conclusions that we arrived at during discussion in the THP meeting yesterday, regarding linkage with exiting THP ABIs. It would be great if I can get explicit "agree" or disagree + rationale from at least David, Yu and Kirill.
In summary; I think we are converging on the approach that is already coded, but I'd like confirmation.
The THP situation today
------------------------ At system level: THP can be set to "never", "madvise" or "always"
- At process level: THP can be "never" or "defer to system setting"
- At VMA level: no-hint,MADV_HUGEPAGE,MADV_NOHUGEPAGEThat gives us this table to describe how a page fault is handled, according to process state (columns) and vma flags (rows):
| never | madvise | always ----------------|-----------|-----------|----------- no hint | S | S | THP>S MADV_HUGEPAGE | S | THP>S | THP>S MADV_NOHUGEPAGE | S | S | SLegend:
S allocate single page (PTE-mapped)
LAF allocate lage anon folio (PTE-mapped)
THP allocate THP-sized folio (PMD-mapped)Quoted 1 line
fallback (usually because vma size/alignment insufficient for folio)
Principles for Large Anon Folios (LAF)
--------------------------------------David tells us there are use cases today (e.g. qemu live migration) which use
MADV_NOHUGEPAGEto mean "don't fill any PTEs that are not explicitly faulted" and these use cases will break (i.e. functionally incorrect) if this request is not honoured.So LAF must at least honour
MADV_NOHUGEPAGEto prevent breaking existing use cases. And once we do this, then I think the least confusing thing is for it to also honor the "never" system/process state; so if either the system, process or vma has explicitly opted-out of THP, then LAF should also be bypassed.Similarly, any case that would previously cause the allocation of PMD-sized THP must continue to be honoured, else we risk performance regression.
That leaves the "madvise/no-hint" case, and all THP fallback paths due to the VMA not being correctly aligned or sized to hold a PMD-sized mapping. In these cases, we will attempt to use LAF first, and fallback to single page if the vma size/alignment doesn't permit it.
Quoted 8 lines
| never | madvise | always ----------------|-----------|-----------|----------- no hint | S | LAF>S | THP>LAF>S MADV_HUGEPAGE | S | THP>LAF>S | THP>LAF>S MADV_NOHUGEPAGE | S | S | SI think this (perhaps conservative) approach will be the least surprising to users. And is the policy that is already implemented in this patch.
This looks very reasonable.
The only questionable field is no-hint/madvise. I can argue for both LAF>S and S here. I think LAF>S is fine as long as we are not too aggressive with allocation order.
I think we need to work on eliminating reasons for users to set 'never'. If something behaves better with 'never' kernel has failed user.
Quoted 7 lines
Downsides of this policy
------------------------As Yu and Yin have pointed out, there are some workloads which do not perform well with THP, due to large fault latency or memory wastage, etc. But which _may_ still benefit from LAF. By taking the conservative approach, we exclude these workloads from benefiting automatically.
Hm. I don't buy it. Why THP with order-9 is too much, but order-8 LAF is fine?
If allocation latency is a problem, it has to be fixed. Maybe with introducing an API to page allocator where user can request a range of acceptable orders and page allocator returns largest readily available possibly starting background compaction.
Quoted 2 lines
But given they have explicitly opted out of THP, it doesn't seem unreasonable that those workloads should be explicitly modified to opt-in to LAF.
No, we should address the reason the why THP is off. I think there shouldn't be hard wall between THP and LAF, but smooth gradient.
Quoted 7 lines
The
question is what should a control for this look like? And do we need to implement the control for an MVP implementation of LAF? For the latter question, I would suggest this can come later - its a tool to further optimize, but its absence does not regress today's performance.What should a control look like?
I would start with zero-API. Let's see if we can live with it.
If something is required for debug or benchmarking, we can add it to debugfs.
Quoted 36 lines
One suggestion was to expose a "maximum order" tunable, which would limit the size of THP that could be allocated. setting it to 1M would cause traditional THP to be bypassed (assuming for now PMD-sized THP is 2M) but would permit LAF. But Kirill suggested that this type of control might turn out to be restrictive in the long run.
Another suggestion was to provide a more abstracted hint to the kernel, which the kernel could then derive a policy from, and that policy would be easier to change over time.
Large Anon Folio Size
---------------------Once we have decided to use LAF (vs THP vs S), we need to decide how big the folio should be. If/when we get a control as described above, that will obviously place an upper bound on the size. HW may also have a preferred size due to tricks it can do in the TLB (
arch_wants_pte_order()in this patch) but you may still want to allocate a bigger folio than the HW wants (since bigger folios will reduce page faults) or you may want to allocate a smaller folio than the HW wants (due to concerns about latency or memory wastage).I've had a stab at addressing this in the patch too, using the same decision as for THP (ignoring the vma size/alignment requirement) to decide if we use the HW preferred order or if we cap it (currently set at 64K).
Thoughts, comments?
Thanks,
Ryan
Signature · 1 line
Kiryl Shutsemau / Kirill A. Shutemov
Re: [PATCH v4 2/5] mm: LARGE_ANON_FOLIO for improved performance
Quoted 156 lines · Kirill A. Shutemov
Quoted 133 lines · Ryan Roberts
+ Kirill
Quoted 30 lines · Ryan Roberts
Introduce
LARGE_ANON_FOLIOfeature, which allows anonymous memory to be allocated in large folios of a determined order. All pages of the large folio are pte-mapped during the same page fault, significantly reducing the number of page faults. The number of per-page operations (e.g. ref counting, rmap management lru list management) are also significantly reduced since those ops now become per-folio.The new behaviour is hidden behind the new
LARGE_ANON_FOLIOKconfig, which defaults to disabled for now; The long term aim is for this to defaut to enabled, but there are some risks around internal fragmentation that need to be better understood first.When enabled, the folio order is determined as such: For a vma, process or system that has explicitly disabled THP, we continue to allocate order-0. THP is most likely disabled to avoid any possible internal fragmentation so we honour that request.
Otherwise, the return value of
arch_wants_pte_order()is used. For vmas that have not explicitly opted-in to use transparent hugepages (e.g. where thp=madvise and the vma does not haveMADV_HUGEPAGE), thenarch_wants_pte_order()is limited to 64K (orPAGE_SIZE, whichever is bigger). This allows for a performance boost without requiring any explicit opt-in from the workload while limitting internal fragmentation.If the preferred order can't be used (e.g. because the folio would breach the bounds of the vma, or because ptes in the region are already mapped) then we fall back to a suitable lower order; first
PAGE_ALLOC_COSTLY_ORDER, then order-0....
Quoted 28 lines
mm/memory.c +28
-0+#define ANON_FOLIO_MAX_ORDER_UNHINTED \ + (ilog2(max_t(unsigned long, SZ_64K, PAGE_SIZE)) - PAGE_SHIFT) + +static int anon_folio_order (struct vm_area_struct *vma) +{ +int order; + +/* +* If THP is explicitly disabled for either the vma, the process or the +* system, then this is very likely intended to limit internal +* fragmentation; in this case, don't attempt to allocate a large +* anonymous folio. +* +* Else, if the vma is eligible for thp, allocate a large folio of the +* size preferred by the arch. Or if the arch requested a very small +* size or didn't request a size, then use PAGE_ALLOC_COSTLY_ORDER, +* which still meets the arch's requirements but means we still take +* advantage of SW optimizations (e.g. fewer page faults). +* +* Finally if thp is enabled but the vma isn't eligible, take the +* arch-preferred size and limit it to ANON_FOLIO_MAX_ORDER_UNHINTED. +* This ensures workloads that have not explicitly opted-in take benefit +* while capping the potential for internal fragmentation. +*/ + +if ((vma->vm_flags &VM_NOHUGEPAGE ) || +test_bit (MMF_DISABLE_THP , &vma->vm_mm->flags) || + !hugepage_flags_enabled ())mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 4106
Quoted 10 lines
mm/memory.c +10
-0+ order =0 ; + else { + order =max (arch_wants_pte_order (),PAGE_ALLOC_COSTLY_ORDER ); + +if (!hugepage_vma_check (vma, vma->vm_flags, false, true, true)) + order =min (order,ANON_FOLIO_MAX_ORDER_UNHINTED ); + } + +return order; +}Hi All,
I'm writing up the conclusions that we arrived at during discussion in the THP meeting yesterday, regarding linkage with exiting THP ABIs. It would be great if I can get explicit "agree" or disagree + rationale from at least David, Yu and Kirill.
In summary; I think we are converging on the approach that is already coded, but I'd like confirmation.
The THP situation today
------------------------ At system level: THP can be set to "never", "madvise" or "always"
- At process level: THP can be "never" or "defer to system setting"
- At VMA level: no-hint,MADV_HUGEPAGE,MADV_NOHUGEPAGEThat gives us this table to describe how a page fault is handled, according to process state (columns) and vma flags (rows):
| never | madvise | always ----------------|-----------|-----------|----------- no hint | S | S | THP>S MADV_HUGEPAGE | S | THP>S | THP>S MADV_NOHUGEPAGE | S | S | SLegend:
S allocate single page (PTE-mapped) LAF allocate lage anon folio (PTE-mapped) THP allocate THP-sized folio (PMD-mapped) > fallback (usually because vma size/alignment insufficient for folio)Principles for Large Anon Folios (LAF)
--------------------------------------David tells us there are use cases today (e.g. qemu live migration) which use
MADV_NOHUGEPAGEto mean "don't fill any PTEs that are not explicitly faulted" and these use cases will break (i.e. functionally incorrect) if this request is not honoured.So LAF must at least honour
MADV_NOHUGEPAGEto prevent breaking existing use cases. And once we do this, then I think the least confusing thing is for it to also honor the "never" system/process state; so if either the system, process or vma has explicitly opted-out of THP, then LAF should also be bypassed.Similarly, any case that would previously cause the allocation of PMD-sized THP must continue to be honoured, else we risk performance regression.
That leaves the "madvise/no-hint" case, and all THP fallback paths due to the VMA not being correctly aligned or sized to hold a PMD-sized mapping. In these cases, we will attempt to use LAF first, and fallback to single page if the vma size/alignment doesn't permit it.
Quoted 8 lines
| never | madvise | always ----------------|-----------|-----------|----------- no hint | S | LAF>S | THP>LAF>S MADV_HUGEPAGE | S | THP>LAF>S | THP>LAF>S MADV_NOHUGEPAGE | S | S | SI think this (perhaps conservative) approach will be the least surprising to users. And is the policy that is already implemented in this patch.
This looks very reasonable.
The only questionable field is no-hint/madvise. I can argue for both LAF>S and S here. I think LAF>S is fine as long as we are not too aggressive with allocation order.
I think we need to work on eliminating reasons for users to set 'never'. If something behaves better with 'never' kernel has failed user.
Quoted 3 lines
Downsides of this policy
------------------------
Quoted 7 lines
Quoted 4 lines
As Yu and Yin have pointed out, there are some workloads which do not perform well with THP, due to large fault latency or memory wastage, etc. But which _may_ still benefit from LAF. By taking the conservative approach, we exclude these workloads from benefiting automatically.
Hm. I don't buy it. Why THP with order-9 is too much, but order-8 LAF is fine?
No, it's not. And no one said order-8 LAF is fine :) The starting order for LAF that we have been discussing is at most 64KB (vs 2MB THP). For my taste, it's still too large. I'd go with 32KB/16KB.
However, the same argument can be used to argue against the policy Ryan listed above: why order-10 LAF is ok for madvise but not order-11 (which becomes "always")?
I'm strongly against this policy for two practical reasons I learned from tuning THPs in our data centers:
1. By doing the above, we are blurring the lines between those values and making real-world performance tuning extremely hard if not impractice.
2. As I previously pointed out: if we mix LAFs with THPs, we actually risk causing performance regressions because giving smaller VMAs LAFs can deprive large VMAs of THPs.
Re: [PATCH v4 2/5] mm: LARGE_ANON_FOLIO for improved performance
Quoted 163 lines · Yu Zhao
Quoted 161 lines · Kirill A. Shutemov
Quoted 133 lines · Ryan Roberts
+ Kirill
Quoted 30 lines · Ryan Roberts
Introduce
LARGE_ANON_FOLIOfeature, which allows anonymous memory to be allocated in large folios of a determined order. All pages of the large folio are pte-mapped during the same page fault, significantly reducing the number of page faults. The number of per-page operations (e.g. ref counting, rmap management lru list management) are also significantly reduced since those ops now become per-folio.The new behaviour is hidden behind the new
LARGE_ANON_FOLIOKconfig, which defaults to disabled for now; The long term aim is for this to defaut to enabled, but there are some risks around internal fragmentation that need to be better understood first.When enabled, the folio order is determined as such: For a vma, process or system that has explicitly disabled THP, we continue to allocate order-0. THP is most likely disabled to avoid any possible internal fragmentation so we honour that request.
Otherwise, the return value of
arch_wants_pte_order()is used. For vmas that have not explicitly opted-in to use transparent hugepages (e.g. where thp=madvise and the vma does not haveMADV_HUGEPAGE), thenarch_wants_pte_order()is limited to 64K (orPAGE_SIZE, whichever is bigger). This allows for a performance boost without requiring any explicit opt-in from the workload while limitting internal fragmentation.If the preferred order can't be used (e.g. because the folio would breach the bounds of the vma, or because ptes in the region are already mapped) then we fall back to a suitable lower order; first
PAGE_ALLOC_COSTLY_ORDER, then order-0....
Quoted 28 lines
mm/memory.c +28
-0+#define ANON_FOLIO_MAX_ORDER_UNHINTED \ + (ilog2(max_t(unsigned long, SZ_64K, PAGE_SIZE)) - PAGE_SHIFT) + +static int anon_folio_order (struct vm_area_struct *vma) +{ +int order; + +/* +* If THP is explicitly disabled for either the vma, the process or the +* system, then this is very likely intended to limit internal +* fragmentation; in this case, don't attempt to allocate a large +* anonymous folio. +* +* Else, if the vma is eligible for thp, allocate a large folio of the +* size preferred by the arch. Or if the arch requested a very small +* size or didn't request a size, then use PAGE_ALLOC_COSTLY_ORDER, +* which still meets the arch's requirements but means we still take +* advantage of SW optimizations (e.g. fewer page faults). +* +* Finally if thp is enabled but the vma isn't eligible, take the +* arch-preferred size and limit it to ANON_FOLIO_MAX_ORDER_UNHINTED. +* This ensures workloads that have not explicitly opted-in take benefit +* while capping the potential for internal fragmentation. +*/ + +if ((vma->vm_flags &VM_NOHUGEPAGE ) || +test_bit (MMF_DISABLE_THP , &vma->vm_mm->flags) || + !hugepage_flags_enabled ())mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 4106
Quoted 10 lines
mm/memory.c +10
-0+ order =0 ; + else { + order =max (arch_wants_pte_order (),PAGE_ALLOC_COSTLY_ORDER ); + +if (!hugepage_vma_check (vma, vma->vm_flags, false, true, true)) + order =min (order,ANON_FOLIO_MAX_ORDER_UNHINTED ); + } + +return order; +}Hi All,
I'm writing up the conclusions that we arrived at during discussion in the THP meeting yesterday, regarding linkage with exiting THP ABIs. It would be great if I can get explicit "agree" or disagree + rationale from at least David, Yu and Kirill.
In summary; I think we are converging on the approach that is already coded, but I'd like confirmation.
The THP situation today
------------------------ At system level: THP can be set to "never", "madvise" or "always"
- At process level: THP can be "never" or "defer to system setting"
- At VMA level: no-hint,MADV_HUGEPAGE,MADV_NOHUGEPAGEThat gives us this table to describe how a page fault is handled, according to process state (columns) and vma flags (rows):
| never | madvise | always ----------------|-----------|-----------|----------- no hint | S | S | THP>S MADV_HUGEPAGE | S | THP>S | THP>S MADV_NOHUGEPAGE | S | S | SLegend:
S allocate single page (PTE-mapped) LAF allocate lage anon folio (PTE-mapped) THP allocate THP-sized folio (PMD-mapped) > fallback (usually because vma size/alignment insufficient for folio)Principles for Large Anon Folios (LAF)
--------------------------------------David tells us there are use cases today (e.g. qemu live migration) which use
MADV_NOHUGEPAGEto mean "don't fill any PTEs that are not explicitly faulted" and these use cases will break (i.e. functionally incorrect) if this request is not honoured.So LAF must at least honour
MADV_NOHUGEPAGEto prevent breaking existing use cases. And once we do this, then I think the least confusing thing is for it to also honor the "never" system/process state; so if either the system, process or vma has explicitly opted-out of THP, then LAF should also be bypassed.Similarly, any case that would previously cause the allocation of PMD-sized THP must continue to be honoured, else we risk performance regression.
That leaves the "madvise/no-hint" case, and all THP fallback paths due to the VMA not being correctly aligned or sized to hold a PMD-sized mapping. In these cases, we will attempt to use LAF first, and fallback to single page if the vma size/alignment doesn't permit it.
Quoted 8 lines
| never | madvise | always ----------------|-----------|-----------|----------- no hint | S | LAF>S | THP>LAF>S MADV_HUGEPAGE | S | THP>LAF>S | THP>LAF>S MADV_NOHUGEPAGE | S | S | SI think this (perhaps conservative) approach will be the least surprising to users. And is the policy that is already implemented in this patch.
This looks very reasonable.
The only questionable field is no-hint/madvise. I can argue for both LAF>S and S here. I think LAF>S is fine as long as we are not too aggressive with allocation order.
I think we need to work on eliminating reasons for users to set 'never'. If something behaves better with 'never' kernel has failed user.
Quoted 7 lines
Downsides of this policy
------------------------As Yu and Yin have pointed out, there are some workloads which do not perform well with THP, due to large fault latency or memory wastage, etc. But which _may_ still benefit from LAF. By taking the conservative approach, we exclude these workloads from benefiting automatically.
Quoted 6 lines
Quoted 2 lines
Hm. I don't buy it. Why THP with order-9 is too much, but order-8 LAF is fine?
No, it's not. And no one said order-8 LAF is fine :) The starting order for LAF that we have been discussing is at most 64KB (vs 2MB THP). For my taste, it's still too large. I'd go with 32KB/16KB.
I guess it is because ARM64 supports contig PTE at 64KB, so getting large anon folio at 64KB on ARM64 would have an extra perf boost when set contig PTE bits patch is also in.
On x86_64, 32KB might be better on AMD CPUs that support PTE clustering, which would use a single TLB entry for 8 contiguous 4KB pages and is done at microarchitecture level without additional software changes.
Quoted 7 lines
However, the same argument can be used to argue against the policy Ryan listed above: why order-10 LAF is ok for madvise but not order-11 (which becomes "always")?
I'm strongly against this policy for two practical reasons I learned from tuning THPs in our data centers:
Do you mind writing down your policy? That would help us see and discuss the difference.
Quoted 6 lines
1. By doing the above, we are blurring the lines between those values and making real-world performance tuning extremely hard if not impractice.
2. As I previously pointed out: if we mix LAFs with THPs, we actually risk causing performance regressions because giving smaller VMAs LAFs can deprive large VMAs of THPs.
I think these two reasons are based on that we do not have a reasonable LAF+THP allocation and management policy and we do not fully understand the pros and cons of using LAF and mixing LAF with THP. It would be safe to separate LAF and THP. By doing so,
1. for workloads do not benefit from THP, we can turn on LAF alone to see if there is a performance boost and further understand if LAF hurts, has no impactor , or improves the performance of these workloads.
2. for workloads benefit from THP, we can also turn on LAF separately to understand the performance impact of LAF (hurt, no change, or improve).
Ultimately, after we understand the performance impact of LAF, THP, and mix of them and come up a reasonable kernel policy, a unified knob would make sense. But we are not there yet.
--
Best Regards,
Yan, Zi
Re: [PATCH v4 2/5] mm: LARGE_ANON_FOLIO for improved performance
Quoted 204 lines · Zi Yan
Quoted 163 lines · Yu Zhao
Quoted 161 lines · Kirill A. Shutemov
Quoted 133 lines · Ryan Roberts
+ Kirill
Quoted 30 lines · Ryan Roberts
Introduce
LARGE_ANON_FOLIOfeature, which allows anonymous memory to be allocated in large folios of a determined order. All pages of the large folio are pte-mapped during the same page fault, significantly reducing the number of page faults. The number of per-page operations (e.g. ref counting, rmap management lru list management) are also significantly reduced since those ops now become per-folio.The new behaviour is hidden behind the new
LARGE_ANON_FOLIOKconfig, which defaults to disabled for now; The long term aim is for this to defaut to enabled, but there are some risks around internal fragmentation that need to be better understood first.When enabled, the folio order is determined as such: For a vma, process or system that has explicitly disabled THP, we continue to allocate order-0. THP is most likely disabled to avoid any possible internal fragmentation so we honour that request.
Otherwise, the return value of
arch_wants_pte_order()is used. For vmas that have not explicitly opted-in to use transparent hugepages (e.g. where thp=madvise and the vma does not haveMADV_HUGEPAGE), thenarch_wants_pte_order()is limited to 64K (orPAGE_SIZE, whichever is bigger). This allows for a performance boost without requiring any explicit opt-in from the workload while limitting internal fragmentation.If the preferred order can't be used (e.g. because the folio would breach the bounds of the vma, or because ptes in the region are already mapped) then we fall back to a suitable lower order; first
PAGE_ALLOC_COSTLY_ORDER, then order-0....
Quoted 28 lines
mm/memory.c +28
-0+#define ANON_FOLIO_MAX_ORDER_UNHINTED \ + (ilog2(max_t(unsigned long, SZ_64K, PAGE_SIZE)) - PAGE_SHIFT) + +static int anon_folio_order (struct vm_area_struct *vma) +{ +int order; + +/* +* If THP is explicitly disabled for either the vma, the process or the +* system, then this is very likely intended to limit internal +* fragmentation; in this case, don't attempt to allocate a large +* anonymous folio. +* +* Else, if the vma is eligible for thp, allocate a large folio of the +* size preferred by the arch. Or if the arch requested a very small +* size or didn't request a size, then use PAGE_ALLOC_COSTLY_ORDER, +* which still meets the arch's requirements but means we still take +* advantage of SW optimizations (e.g. fewer page faults). +* +* Finally if thp is enabled but the vma isn't eligible, take the +* arch-preferred size and limit it to ANON_FOLIO_MAX_ORDER_UNHINTED. +* This ensures workloads that have not explicitly opted-in take benefit +* while capping the potential for internal fragmentation. +*/ + +if ((vma->vm_flags &VM_NOHUGEPAGE ) || +test_bit (MMF_DISABLE_THP , &vma->vm_mm->flags) || + !hugepage_flags_enabled ())mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 4106
Quoted 10 lines
mm/memory.c +10
-0+ order =0 ; + else { + order =max (arch_wants_pte_order (),PAGE_ALLOC_COSTLY_ORDER ); + +if (!hugepage_vma_check (vma, vma->vm_flags, false, true, true)) + order =min (order,ANON_FOLIO_MAX_ORDER_UNHINTED ); + } + +return order; +}Hi All,
I'm writing up the conclusions that we arrived at during discussion in the THP meeting yesterday, regarding linkage with exiting THP ABIs. It would be great if I can get explicit "agree" or disagree + rationale from at least David, Yu and Kirill.
In summary; I think we are converging on the approach that is already coded, but I'd like confirmation.
The THP situation today
------------------------ At system level: THP can be set to "never", "madvise" or "always"
- At process level: THP can be "never" or "defer to system setting"
- At VMA level: no-hint,MADV_HUGEPAGE,MADV_NOHUGEPAGEThat gives us this table to describe how a page fault is handled, according to process state (columns) and vma flags (rows):
| never | madvise | always ----------------|-----------|-----------|----------- no hint | S | S | THP>S MADV_HUGEPAGE | S | THP>S | THP>S MADV_NOHUGEPAGE | S | S | SLegend:
S allocate single page (PTE-mapped) LAF allocate lage anon folio (PTE-mapped) THP allocate THP-sized folio (PMD-mapped) > fallback (usually because vma size/alignment insufficient for folio)Principles for Large Anon Folios (LAF)
--------------------------------------David tells us there are use cases today (e.g. qemu live migration) which use
MADV_NOHUGEPAGEto mean "don't fill any PTEs that are not explicitly faulted" and these use cases will break (i.e. functionally incorrect) if this request is not honoured.So LAF must at least honour
MADV_NOHUGEPAGEto prevent breaking existing use cases. And once we do this, then I think the least confusing thing is for it to also honor the "never" system/process state; so if either the system, process or vma has explicitly opted-out of THP, then LAF should also be bypassed.Similarly, any case that would previously cause the allocation of PMD-sized THP must continue to be honoured, else we risk performance regression.
That leaves the "madvise/no-hint" case, and all THP fallback paths due to the VMA not being correctly aligned or sized to hold a PMD-sized mapping. In these cases, we will attempt to use LAF first, and fallback to single page if the vma size/alignment doesn't permit it.
Quoted 8 lines
| never | madvise | always ----------------|-----------|-----------|----------- no hint | S | LAF>S | THP>LAF>S MADV_HUGEPAGE | S | THP>LAF>S | THP>LAF>S MADV_NOHUGEPAGE | S | S | SI think this (perhaps conservative) approach will be the least surprising to users. And is the policy that is already implemented in this patch.
This looks very reasonable.
The only questionable field is no-hint/madvise. I can argue for both LAF>S and S here. I think LAF>S is fine as long as we are not too aggressive with allocation order.
I think we need to work on eliminating reasons for users to set 'never'. If something behaves better with 'never' kernel has failed user.
Quoted 7 lines
Downsides of this policy
------------------------As Yu and Yin have pointed out, there are some workloads which do not perform well with THP, due to large fault latency or memory wastage, etc. But which _may_ still benefit from LAF. By taking the conservative approach, we exclude these workloads from benefiting automatically.
Quoted 6 lines
Quoted 2 lines
Hm. I don't buy it. Why THP with order-9 is too much, but order-8 LAF is fine?
No, it's not. And no one said order-8 LAF is fine :) The starting order for LAF that we have been discussing is at most 64KB (vs 2MB THP). For my taste, it's still too large. I'd go with 32KB/16KB.
I guess it is because ARM64 supports contig PTE at 64KB, so getting large anon folio at 64KB on ARM64 would have an extra perf boost when set contig PTE bits patch is also in.
On
x86_64, 32KB might be better on AMD CPUs that support PTE clustering, which would use a single TLB entry for 8 contiguous 4KB pages and is done at microarchitecture level without additional software changes.Quoted 7 lines
However, the same argument can be used to argue against the policy Ryan listed above: why order-10 LAF is ok for madvise but not order-11 (which becomes "always")?
I'm strongly against this policy for two practical reasons I learned from tuning THPs in our data centers:
Do you mind writing down your policy? That would help us see and discuss the difference.
Quoted 6 lines
1. By doing the above, we are blurring the lines between those values and making real-world performance tuning extremely hard if not impractice.
2. As I previously pointed out: if we mix LAFs with THPs, we actually risk causing performance regressions because giving smaller VMAs LAFs can deprive large VMAs of THPs.I think these two reasons are based on that we do not have a reasonable LAF+THP allocation and management policy and we do not fully understand the pros and cons of using LAF and mixing LAF with THP. It would be safe to separate LAF and THP. By doing so,
Quoted 6 lines
1. for workloads do not benefit from THP, we can turn on LAF alone to see if there is a performance boost and further understand if LAF hurts, has no impactor , or improves the performance of these workloads.
2. for workloads benefit from THP, we can also turn on LAF separately to understand the performance impact of LAF (hurt, no change, or improve).
This is basically what I've been suggesting. We should have a separate knob, not overload the existing ones. And this separate knob should be able to take a list of fallback orders. After we have a wider deployment, we might gain a better understanding of the "cost function". Then we can try to build some in-kernel heuristics that automatically decides the best orders to fallback. If/when we get there, we can simply extend the knob by adding a new "magic word", e.g., "auto".
Quoted 3 lines
Ultimately, after we understand the performance impact of LAF, THP, and mix of them and come up a reasonable kernel policy, a unified knob would make sense. But we are not there yet.
Exactly.
Re: [PATCH v4 2/5] mm: LARGE_ANON_FOLIO for improved performance
In reply to Yu ZhaoQuoted 163 lines · Yu Zhao
Quoted 161 lines · Kirill A. Shutemov
Quoted 133 lines · Ryan Roberts
+ Kirill
Quoted 30 lines · Ryan Roberts
Introduce
LARGE_ANON_FOLIOfeature, which allows anonymous memory to be allocated in large folios of a determined order. All pages of the large folio are pte-mapped during the same page fault, significantly reducing the number of page faults. The number of per-page operations (e.g. ref counting, rmap management lru list management) are also significantly reduced since those ops now become per-folio.The new behaviour is hidden behind the new
LARGE_ANON_FOLIOKconfig, which defaults to disabled for now; The long term aim is for this to defaut to enabled, but there are some risks around internal fragmentation that need to be better understood first.When enabled, the folio order is determined as such: For a vma, process or system that has explicitly disabled THP, we continue to allocate order-0. THP is most likely disabled to avoid any possible internal fragmentation so we honour that request.
Otherwise, the return value of
arch_wants_pte_order()is used. For vmas that have not explicitly opted-in to use transparent hugepages (e.g. where thp=madvise and the vma does not haveMADV_HUGEPAGE), thenarch_wants_pte_order()is limited to 64K (orPAGE_SIZE, whichever is bigger). This allows for a performance boost without requiring any explicit opt-in from the workload while limitting internal fragmentation.If the preferred order can't be used (e.g. because the folio would breach the bounds of the vma, or because ptes in the region are already mapped) then we fall back to a suitable lower order; first
PAGE_ALLOC_COSTLY_ORDER, then order-0....
Quoted 28 lines
mm/memory.c +28
-0+#define ANON_FOLIO_MAX_ORDER_UNHINTED \ + (ilog2(max_t(unsigned long, SZ_64K, PAGE_SIZE)) - PAGE_SHIFT) + +static int anon_folio_order (struct vm_area_struct *vma) +{ +int order; + +/* +* If THP is explicitly disabled for either the vma, the process or the +* system, then this is very likely intended to limit internal +* fragmentation; in this case, don't attempt to allocate a large +* anonymous folio. +* +* Else, if the vma is eligible for thp, allocate a large folio of the +* size preferred by the arch. Or if the arch requested a very small +* size or didn't request a size, then use PAGE_ALLOC_COSTLY_ORDER, +* which still meets the arch's requirements but means we still take +* advantage of SW optimizations (e.g. fewer page faults). +* +* Finally if thp is enabled but the vma isn't eligible, take the +* arch-preferred size and limit it to ANON_FOLIO_MAX_ORDER_UNHINTED. +* This ensures workloads that have not explicitly opted-in take benefit +* while capping the potential for internal fragmentation. +*/ + +if ((vma->vm_flags &VM_NOHUGEPAGE ) || +test_bit (MMF_DISABLE_THP , &vma->vm_mm->flags) || + !hugepage_flags_enabled ())mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 4106
Quoted 10 lines
mm/memory.c +10
-0+ order =0 ; + else { + order =max (arch_wants_pte_order (),PAGE_ALLOC_COSTLY_ORDER ); + +if (!hugepage_vma_check (vma, vma->vm_flags, false, true, true)) + order =min (order,ANON_FOLIO_MAX_ORDER_UNHINTED ); + } + +return order; +}Hi All,
I'm writing up the conclusions that we arrived at during discussion in the THP meeting yesterday, regarding linkage with exiting THP ABIs. It would be great if I can get explicit "agree" or disagree + rationale from at least David, Yu and Kirill.
In summary; I think we are converging on the approach that is already coded, but I'd like confirmation.
The THP situation today
------------------------ At system level: THP can be set to "never", "madvise" or "always"
- At process level: THP can be "never" or "defer to system setting"
- At VMA level: no-hint,MADV_HUGEPAGE,MADV_NOHUGEPAGEThat gives us this table to describe how a page fault is handled, according to process state (columns) and vma flags (rows):
| never | madvise | always ----------------|-----------|-----------|----------- no hint | S | S | THP>S MADV_HUGEPAGE | S | THP>S | THP>S MADV_NOHUGEPAGE | S | S | SLegend:
S allocate single page (PTE-mapped) LAF allocate lage anon folio (PTE-mapped) THP allocate THP-sized folio (PMD-mapped) > fallback (usually because vma size/alignment insufficient for folio)Principles for Large Anon Folios (LAF)
--------------------------------------David tells us there are use cases today (e.g. qemu live migration) which use
MADV_NOHUGEPAGEto mean "don't fill any PTEs that are not explicitly faulted" and these use cases will break (i.e. functionally incorrect) if this request is not honoured.So LAF must at least honour
MADV_NOHUGEPAGEto prevent breaking existing use cases. And once we do this, then I think the least confusing thing is for it to also honor the "never" system/process state; so if either the system, process or vma has explicitly opted-out of THP, then LAF should also be bypassed.Similarly, any case that would previously cause the allocation of PMD-sized THP must continue to be honoured, else we risk performance regression.
That leaves the "madvise/no-hint" case, and all THP fallback paths due to the VMA not being correctly aligned or sized to hold a PMD-sized mapping. In these cases, we will attempt to use LAF first, and fallback to single page if the vma size/alignment doesn't permit it.
Quoted 8 lines
| never | madvise | always ----------------|-----------|-----------|----------- no hint | S | LAF>S | THP>LAF>S MADV_HUGEPAGE | S | THP>LAF>S | THP>LAF>S MADV_NOHUGEPAGE | S | S | SI think this (perhaps conservative) approach will be the least surprising to users. And is the policy that is already implemented in this patch.
This looks very reasonable.
The only questionable field is no-hint/madvise. I can argue for both LAF>S and S here. I think LAF>S is fine as long as we are not too aggressive with allocation order.
I think we need to work on eliminating reasons for users to set 'never'. If something behaves better with 'never' kernel has failed user.
Quoted 7 lines
Downsides of this policy
------------------------As Yu and Yin have pointed out, there are some workloads which do not perform well with THP, due to large fault latency or memory wastage, etc. But which _may_ still benefit from LAF. By taking the conservative approach, we exclude these workloads from benefiting automatically.
Quoted 6 lines
Quoted 2 lines
Hm. I don't buy it. Why THP with order-9 is too much, but order-8 LAF is fine?
No, it's not. And no one said order-8 LAF is fine :) The starting order for LAF that we have been discussing is at most 64KB (vs 2MB THP). For my taste, it's still too large. I'd go with 32KB/16KB.
Its currently influenced by the arch. If the arch doesn't have an opinion then its currently 32K in the code. The 64K size is my aspiration for arm64 if/when I land the contpte mapping work.
Quoted 4 lines
However, the same argument can be used to argue against the policy Ryan listed above: why order-10 LAF is ok for madvise but not order-11 (which becomes "always")?
Sorry I don't understand what you are saying here. Where has order-10 LAF come from?
Quoted 2 lines
I'm strongly against this policy
Ugh, I thought we came to an agreement (or at least "disagree and commit") on the THP call. Obviously I was wrong.
David is telling us that we will break user space if we don't consider MADV_NOHUGEPAGE to mean "never allocate memory to unfaulted addresses". So tying to at least this must be cast in stone, no? Could you lay out any policy proposal you have as an alternative that still follows this requirement?
Quoted 8 lines
for two practical reasons I learned
from tuning THPs in our data centers:
1. By doing the above, we are blurring the lines between those values and making real-world performance tuning extremely hard if not impractice.
2. As I previously pointed out: if we mix LAFs with THPs, we actually risk causing performance regressions because giving smaller VMAs LAFs can deprive large VMAs of THPs.
Re: [PATCH v4 2/5] mm: LARGE_ANON_FOLIO for improved performance
Quoted 168 lines · Ryan Roberts
Quoted 166 lines · Yu Zhao
Quoted 163 lines · Kirill A. Shutemov
Quoted 133 lines · Ryan Roberts
+ Kirill
Quoted 30 lines · Ryan Roberts
Introduce
LARGE_ANON_FOLIOfeature, which allows anonymous memory to be allocated in large folios of a determined order. All pages of the large folio are pte-mapped during the same page fault, significantly reducing the number of page faults. The number of per-page operations (e.g. ref counting, rmap management lru list management) are also significantly reduced since those ops now become per-folio.The new behaviour is hidden behind the new
LARGE_ANON_FOLIOKconfig, which defaults to disabled for now; The long term aim is for this to defaut to enabled, but there are some risks around internal fragmentation that need to be better understood first.When enabled, the folio order is determined as such: For a vma, process or system that has explicitly disabled THP, we continue to allocate order-0. THP is most likely disabled to avoid any possible internal fragmentation so we honour that request.
Otherwise, the return value of
arch_wants_pte_order()is used. For vmas that have not explicitly opted-in to use transparent hugepages (e.g. where thp=madvise and the vma does not haveMADV_HUGEPAGE), thenarch_wants_pte_order()is limited to 64K (orPAGE_SIZE, whichever is bigger). This allows for a performance boost without requiring any explicit opt-in from the workload while limitting internal fragmentation.If the preferred order can't be used (e.g. because the folio would breach the bounds of the vma, or because ptes in the region are already mapped) then we fall back to a suitable lower order; first
PAGE_ALLOC_COSTLY_ORDER, then order-0....
Quoted 28 lines
mm/memory.c +28
-0+#define ANON_FOLIO_MAX_ORDER_UNHINTED \ + (ilog2(max_t(unsigned long, SZ_64K, PAGE_SIZE)) - PAGE_SHIFT) + +static int anon_folio_order (struct vm_area_struct *vma) +{ +int order; + +/* +* If THP is explicitly disabled for either the vma, the process or the +* system, then this is very likely intended to limit internal +* fragmentation; in this case, don't attempt to allocate a large +* anonymous folio. +* +* Else, if the vma is eligible for thp, allocate a large folio of the +* size preferred by the arch. Or if the arch requested a very small +* size or didn't request a size, then use PAGE_ALLOC_COSTLY_ORDER, +* which still meets the arch's requirements but means we still take +* advantage of SW optimizations (e.g. fewer page faults). +* +* Finally if thp is enabled but the vma isn't eligible, take the +* arch-preferred size and limit it to ANON_FOLIO_MAX_ORDER_UNHINTED. +* This ensures workloads that have not explicitly opted-in take benefit +* while capping the potential for internal fragmentation. +*/ + +if ((vma->vm_flags &VM_NOHUGEPAGE ) || +test_bit (MMF_DISABLE_THP , &vma->vm_mm->flags) || + !hugepage_flags_enabled ())mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 4106
Quoted 10 lines
mm/memory.c +10
-0+ order =0 ; + else { + order =max (arch_wants_pte_order (),PAGE_ALLOC_COSTLY_ORDER ); + +if (!hugepage_vma_check (vma, vma->vm_flags, false, true, true)) + order =min (order,ANON_FOLIO_MAX_ORDER_UNHINTED ); + } + +return order; +}Hi All,
I'm writing up the conclusions that we arrived at during discussion in the THP meeting yesterday, regarding linkage with exiting THP ABIs. It would be great if I can get explicit "agree" or disagree + rationale from at least David, Yu and Kirill.
In summary; I think we are converging on the approach that is already coded, but I'd like confirmation.
The THP situation today
------------------------ At system level: THP can be set to "never", "madvise" or "always"
- At process level: THP can be "never" or "defer to system setting"
- At VMA level: no-hint,MADV_HUGEPAGE,MADV_NOHUGEPAGEThat gives us this table to describe how a page fault is handled, according to process state (columns) and vma flags (rows):
| never | madvise | always ----------------|-----------|-----------|----------- no hint | S | S | THP>S MADV_HUGEPAGE | S | THP>S | THP>S MADV_NOHUGEPAGE | S | S | SLegend:
S allocate single page (PTE-mapped) LAF allocate lage anon folio (PTE-mapped) THP allocate THP-sized folio (PMD-mapped) > fallback (usually because vma size/alignment insufficient for folio)Principles for Large Anon Folios (LAF)
--------------------------------------David tells us there are use cases today (e.g. qemu live migration) which use
MADV_NOHUGEPAGEto mean "don't fill any PTEs that are not explicitly faulted" and these use cases will break (i.e. functionally incorrect) if this request is not honoured.So LAF must at least honour
MADV_NOHUGEPAGEto prevent breaking existing use cases. And once we do this, then I think the least confusing thing is for it to also honor the "never" system/process state; so if either the system, process or vma has explicitly opted-out of THP, then LAF should also be bypassed.Similarly, any case that would previously cause the allocation of PMD-sized THP must continue to be honoured, else we risk performance regression.
That leaves the "madvise/no-hint" case, and all THP fallback paths due to the VMA not being correctly aligned or sized to hold a PMD-sized mapping. In these cases, we will attempt to use LAF first, and fallback to single page if the vma size/alignment doesn't permit it.
Quoted 8 lines
| never | madvise | always ----------------|-----------|-----------|----------- no hint | S | LAF>S | THP>LAF>S MADV_HUGEPAGE | S | THP>LAF>S | THP>LAF>S MADV_NOHUGEPAGE | S | S | SI think this (perhaps conservative) approach will be the least surprising to users. And is the policy that is already implemented in this patch.
This looks very reasonable.
The only questionable field is no-hint/madvise. I can argue for both LAF>S and S here. I think LAF>S is fine as long as we are not too aggressive with allocation order.
I think we need to work on eliminating reasons for users to set 'never'. If something behaves better with 'never' kernel has failed user.
Quoted 7 lines
Downsides of this policy
------------------------As Yu and Yin have pointed out, there are some workloads which do not perform well with THP, due to large fault latency or memory wastage, etc. But which _may_ still benefit from LAF. By taking the conservative approach, we exclude these workloads from benefiting automatically.
Hm. I don't buy it. Why THP with order-9 is too much, but order-8 LAF is fine?
Quoted 7 lines
Quoted 3 lines
No, it's not. And no one said order-8 LAF is fine :) The starting order for LAF that we have been discussing is at most 64KB (vs 2MB THP). For my taste, it's still too large. I'd go with 32KB/16KB.
Its currently influenced by the arch. If the arch doesn't have an opinion then its currently 32K in the code. The 64K size is my aspiration for arm64 if/when I land the contpte mapping work.
Just to double check: this discussion covers the long term/permanente solution/roadmap, correct? That's what Kirill and I were arguing about. Otherwise, the order-8/9 concern above is totally irrelevant, since we don't have them in this series.
For the short term (this series), what you described above looks good to me: we may regress but will not break any existing use cases, and we are behind a Kconfig option.
Quoted 5 lines
Quoted 3 lines
However, the same argument can be used to argue against the policy Ryan listed above: why order-10 LAF is ok for madvise but not order-11 (which becomes "always")?
Sorry I don't understand what you are saying here. Where has order-10 LAF come from?
I pushed that rhetoric a bit further: order-11 is the THP size (32MB) with 16KB base page size on ARM. Confusing, isn't it? And there is another complaint from Fengwei here [1].
[1] https://lore.kernel.org/linux-mm/CAOUHufasZ6w32sHO+Lq33+tGy3+GiO0_dd6mNYwfS_5gqhzYbw@mail.gmail.com/
Quoted 1 line
Quoted 1 line
I'm strongly against this policy
Again, just to be clear: I'm strongly against this policy to be exposed to userspace in any way and become a long-term/permanent thing we have to maintain/change in the future, since I'm assuming that's the context.
Quoted 2 lines
Ugh, I thought we came to an agreement (or at least "disagree and commit") on the THP call. Obviously I was wrong.
My impression is we only agreed on one thing: at the current stage, we should respect things we absolutely have to. We didn't agree on what "never" means ("never 2MB" or "never >4KB"), and we didn't touch on how "always" should behave at all.
Quoted 4 lines
David is telling us that we will break user space if we don't consider
MADV_NOHUGEPAGEto mean "never allocate memory to unfaulted addresses". So tying to at least this must be cast in stone, no? Could you lay out any policy proposal you have as an alternative that still follows this requirement?
If MADV_NOHUGEPAGE falls into the category of things we have to absolutely respect, then we will. But I don't think it does, because the UFFD check we have in this series already guarantees the KVM use case. I can explain how it works in detail if it's still not clear to
you: long story short, the UFFD check precedes the MADV_NOHUGEPAGE check in alloc_anon_folio().
Here is what I recommend for the medium and long terms:
https://lore.kernel.org/linux-mm/CAOUHufYm6Lkm4tLRbyKOc3-NYU-8d6ZDMNDWHo=e=E16oasN8A@mail.gmail.com/
For the short term, hard-coding two orders (hw/sw preferred), putting them behind a Kconfig and not exposing this info to the userspace are good enough for me.
Re: [PATCH v4 2/5] mm: LARGE_ANON_FOLIO for improved performance
Quoted 177 lines · Yu Zhao
Quoted 168 lines · Ryan Roberts
Quoted 166 lines · Yu Zhao
Quoted 163 lines · Kirill A. Shutemov
Quoted 133 lines · Ryan Roberts
+ Kirill
Quoted 30 lines · Ryan Roberts
Introduce
LARGE_ANON_FOLIOfeature, which allows anonymous memory to be allocated in large folios of a determined order. All pages of the large folio are pte-mapped during the same page fault, significantly reducing the number of page faults. The number of per-page operations (e.g. ref counting, rmap management lru list management) are also significantly reduced since those ops now become per-folio.The new behaviour is hidden behind the new
LARGE_ANON_FOLIOKconfig, which defaults to disabled for now; The long term aim is for this to defaut to enabled, but there are some risks around internal fragmentation that need to be better understood first.When enabled, the folio order is determined as such: For a vma, process or system that has explicitly disabled THP, we continue to allocate order-0. THP is most likely disabled to avoid any possible internal fragmentation so we honour that request.
Otherwise, the return value of
arch_wants_pte_order()is used. For vmas that have not explicitly opted-in to use transparent hugepages (e.g. where thp=madvise and the vma does not haveMADV_HUGEPAGE), thenarch_wants_pte_order()is limited to 64K (orPAGE_SIZE, whichever is bigger). This allows for a performance boost without requiring any explicit opt-in from the workload while limitting internal fragmentation.If the preferred order can't be used (e.g. because the folio would breach the bounds of the vma, or because ptes in the region are already mapped) then we fall back to a suitable lower order; first
PAGE_ALLOC_COSTLY_ORDER, then order-0....
Quoted 28 lines
mm/memory.c +28
-0+#define ANON_FOLIO_MAX_ORDER_UNHINTED \ + (ilog2(max_t(unsigned long, SZ_64K, PAGE_SIZE)) - PAGE_SHIFT) + +static int anon_folio_order (struct vm_area_struct *vma) +{ +int order; + +/* +* If THP is explicitly disabled for either the vma, the process or the +* system, then this is very likely intended to limit internal +* fragmentation; in this case, don't attempt to allocate a large +* anonymous folio. +* +* Else, if the vma is eligible for thp, allocate a large folio of the +* size preferred by the arch. Or if the arch requested a very small +* size or didn't request a size, then use PAGE_ALLOC_COSTLY_ORDER, +* which still meets the arch's requirements but means we still take +* advantage of SW optimizations (e.g. fewer page faults). +* +* Finally if thp is enabled but the vma isn't eligible, take the +* arch-preferred size and limit it to ANON_FOLIO_MAX_ORDER_UNHINTED. +* This ensures workloads that have not explicitly opted-in take benefit +* while capping the potential for internal fragmentation. +*/ + +if ((vma->vm_flags &VM_NOHUGEPAGE ) || +test_bit (MMF_DISABLE_THP , &vma->vm_mm->flags) || + !hugepage_flags_enabled ())mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 4106
Quoted 10 lines
mm/memory.c +10
-0+ order =0 ; + else { + order =max (arch_wants_pte_order (),PAGE_ALLOC_COSTLY_ORDER ); + +if (!hugepage_vma_check (vma, vma->vm_flags, false, true, true)) + order =min (order,ANON_FOLIO_MAX_ORDER_UNHINTED ); + } + +return order; +}Hi All,
I'm writing up the conclusions that we arrived at during discussion in the THP meeting yesterday, regarding linkage with exiting THP ABIs. It would be great if I can get explicit "agree" or disagree + rationale from at least David, Yu and Kirill.
In summary; I think we are converging on the approach that is already coded, but I'd like confirmation.
The THP situation today
------------------------ At system level: THP can be set to "never", "madvise" or "always"
- At process level: THP can be "never" or "defer to system setting"
- At VMA level: no-hint,MADV_HUGEPAGE,MADV_NOHUGEPAGEThat gives us this table to describe how a page fault is handled, according to process state (columns) and vma flags (rows):
| never | madvise | always ----------------|-----------|-----------|----------- no hint | S | S | THP>S MADV_HUGEPAGE | S | THP>S | THP>S MADV_NOHUGEPAGE | S | S | SLegend:
S allocate single page (PTE-mapped) LAF allocate lage anon folio (PTE-mapped) THP allocate THP-sized folio (PMD-mapped) > fallback (usually because vma size/alignment insufficient for folio)Principles for Large Anon Folios (LAF)
--------------------------------------David tells us there are use cases today (e.g. qemu live migration) which use
MADV_NOHUGEPAGEto mean "don't fill any PTEs that are not explicitly faulted" and these use cases will break (i.e. functionally incorrect) if this request is not honoured.So LAF must at least honour
MADV_NOHUGEPAGEto prevent breaking existing use cases. And once we do this, then I think the least confusing thing is for it to also honor the "never" system/process state; so if either the system, process or vma has explicitly opted-out of THP, then LAF should also be bypassed.Similarly, any case that would previously cause the allocation of PMD-sized THP must continue to be honoured, else we risk performance regression.
That leaves the "madvise/no-hint" case, and all THP fallback paths due to the VMA not being correctly aligned or sized to hold a PMD-sized mapping. In these cases, we will attempt to use LAF first, and fallback to single page if the vma size/alignment doesn't permit it.
Quoted 8 lines
| never | madvise | always ----------------|-----------|-----------|----------- no hint | S | LAF>S | THP>LAF>S MADV_HUGEPAGE | S | THP>LAF>S | THP>LAF>S MADV_NOHUGEPAGE | S | S | SI think this (perhaps conservative) approach will be the least surprising to users. And is the policy that is already implemented in this patch.
This looks very reasonable.
The only questionable field is no-hint/madvise. I can argue for both LAF>S and S here. I think LAF>S is fine as long as we are not too aggressive with allocation order.
I think we need to work on eliminating reasons for users to set 'never'. If something behaves better with 'never' kernel has failed user.
Quoted 7 lines
Downsides of this policy
------------------------As Yu and Yin have pointed out, there are some workloads which do not perform well with THP, due to large fault latency or memory wastage, etc. But which _may_ still benefit from LAF. By taking the conservative approach, we exclude these workloads from benefiting automatically.
Hm. I don't buy it. Why THP with order-9 is too much, but order-8 LAF is fine?
Quoted 7 lines
Quoted 3 lines
No, it's not. And no one said order-8 LAF is fine :) The starting order for LAF that we have been discussing is at most 64KB (vs 2MB THP). For my taste, it's still too large. I'd go with 32KB/16KB.
Its currently influenced by the arch. If the arch doesn't have an opinion then its currently 32K in the code. The 64K size is my aspiration for arm64 if/when I land the contpte mapping work.
Quoted 8 lines
Just to double check: this discussion covers the long term/permanente solution/roadmap, correct? That's what Kirill and I were arguing about. Otherwise, the order-8/9 concern above is totally irrelevant, since we don't have them in this series.
For the short term (this series), what you described above looks good to me: we may regress but will not break any existing use cases, and we are behind a
Kconfigoption.
OK that's good to hear.
Quoted 11 lines
Quoted 6 lines
Quoted 4 lines
However, the same argument can be used to argue against the policy Ryan listed above: why order-10 LAF is ok for madvise but not order-11 (which becomes "always")?
Sorry I don't understand what you are saying here. Where has order-10 LAF come from?
I pushed that rhetoric a bit further: order-11 is the THP size (32MB) with 16KB base page size on ARM. Confusing, isn't it? And there is another complaint from Fengwei here [1].
Quoted 8 lines
[1] https://lore.kernel.org/linux-mm/CAOUHufasZ6w32sHO+Lq33+tGy3+GiO0_dd6mNYwfS_5gqhzYbw@mail.gmail.com/
Quoted 1 line
Quoted 1 line
I'm strongly against this policy
Again, just to be clear: I'm strongly against this policy to be exposed to userspace in any way and become a long-term/permanent thing we have to maintain/change in the future, since I'm assuming that's the context.
I'm still confused. The policy I described (and which I thought we were discussing) does not expose any new tunables to user space. And you said above that what I described "looks good to me". So is "this policy" which you are strongly against referring to the policy I wrote down or something else?
Quoted 8 lines
Quoted 3 lines
Ugh, I thought we came to an agreement (or at least "disagree and commit") on the THP call. Obviously I was wrong.
My impression is we only agreed on one thing: at the current stage, we should respect things we absolutely have to. We didn't agree on what "never" means ("never 2MB" or "never >4KB"), and we didn't touch on how "always" should behave at all.
I _think_ we have now agreed some of this in other threads, but please re-raise in the context of the other email thread I just sent out - its probably cleaner to continue discussion there.
Quoted 6 lines
Quoted 5 lines
David is telling us that we will break user space if we don't consider
MADV_NOHUGEPAGEto mean "never allocate memory to unfaulted addresses". So tying to at least this must be cast in stone, no? Could you lay out any policy proposal you have as an alternative that still follows this requirement?
Quoted 6 lines
If
MADV_NOHUGEPAGEfalls into the category of things we have to absolutely respect, then we will. But I don't think it does, because the UFFD check we have in this series already guarantees the KVM use case. I can explain how it works in detail if it's still not clear to
you: long story short, the UFFD check precedes theMADV_NOHUGEPAGEcheck inalloc_anon_folio().
I think we have now concluded that its not this simple; MADV_NOHUGEPAGE is applied to the region and pages are faulted in before UFFD is registered, so checking for UFFD is not sufficient.
Quoted 7 lines
Here is what I recommend for the medium and long terms:
https://lore.kernel.org/linux-mm/CAOUHufYm6Lkm4tLRbyKOc3-NYU-8d6ZDMNDWHo=e=E16oasN8A@mail.gmail.com/For the short term, hard-coding two orders (hw/sw preferred), putting them behind a
Kconfigand not exposing this info to the userspace are good enough for me.
I think that's pretty much what I have now, so perhaps I'm laboring the point a bit too much here. Let's just get the prerequisites ticked off then get this patch set merged??
Re: [PATCH v4 2/5] mm: LARGE_ANON_FOLIO for improved performance
In reply to Ryan RobertsQuoted 113 lines · Ryan Roberts
+ Kirill
Quoted 30 lines · Ryan Roberts
Introduce
LARGE_ANON_FOLIOfeature, which allows anonymous memory to be allocated in large folios of a determined order. All pages of the large folio are pte-mapped during the same page fault, significantly reducing the number of page faults. The number of per-page operations (e.g. ref counting, rmap management lru list management) are also significantly reduced since those ops now become per-folio.The new behaviour is hidden behind the new
LARGE_ANON_FOLIOKconfig, which defaults to disabled for now; The long term aim is for this to defaut to enabled, but there are some risks around internal fragmentation that need to be better understood first.When enabled, the folio order is determined as such: For a vma, process or system that has explicitly disabled THP, we continue to allocate order-0. THP is most likely disabled to avoid any possible internal fragmentation so we honour that request.
Otherwise, the return value of
arch_wants_pte_order()is used. For vmas that have not explicitly opted-in to use transparent hugepages (e.g. where thp=madvise and the vma does not haveMADV_HUGEPAGE), thenarch_wants_pte_order()is limited to 64K (orPAGE_SIZE, whichever is bigger). This allows for a performance boost without requiring any explicit opt-in from the workload while limitting internal fragmentation.If the preferred order can't be used (e.g. because the folio would breach the bounds of the vma, or because ptes in the region are already mapped) then we fall back to a suitable lower order; first
PAGE_ALLOC_COSTLY_ORDER, then order-0....
Quoted 28 lines
mm/memory.c +28
-0+#define ANON_FOLIO_MAX_ORDER_UNHINTED \ + (ilog2(max_t(unsigned long, SZ_64K, PAGE_SIZE)) - PAGE_SHIFT) + +static int anon_folio_order (struct vm_area_struct *vma) +{ +int order; + +/* +* If THP is explicitly disabled for either the vma, the process or the +* system, then this is very likely intended to limit internal +* fragmentation; in this case, don't attempt to allocate a large +* anonymous folio. +* +* Else, if the vma is eligible for thp, allocate a large folio of the +* size preferred by the arch. Or if the arch requested a very small +* size or didn't request a size, then use PAGE_ALLOC_COSTLY_ORDER, +* which still meets the arch's requirements but means we still take +* advantage of SW optimizations (e.g. fewer page faults). +* +* Finally if thp is enabled but the vma isn't eligible, take the +* arch-preferred size and limit it to ANON_FOLIO_MAX_ORDER_UNHINTED. +* This ensures workloads that have not explicitly opted-in take benefit +* while capping the potential for internal fragmentation. +*/ + +if ((vma->vm_flags &VM_NOHUGEPAGE ) || +test_bit (MMF_DISABLE_THP , &vma->vm_mm->flags) || + !hugepage_flags_enabled ())mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 4106
Quoted 10 lines
mm/memory.c +10
-0+ order =0 ; + else { + order =max (arch_wants_pte_order (),PAGE_ALLOC_COSTLY_ORDER ); + +if (!hugepage_vma_check (vma, vma->vm_flags, false, true, true)) + order =min (order,ANON_FOLIO_MAX_ORDER_UNHINTED ); + } + +return order; +}Hi All,
I'm writing up the conclusions that we arrived at during discussion in the THP meeting yesterday, regarding linkage with exiting THP ABIs. It would be great if I can get explicit "agree" or disagree + rationale from at least David, Yu and Kirill.
In summary; I think we are converging on the approach that is already coded, but I'd like confirmation.
The THP situation today
------------------------ At system level: THP can be set to "never", "madvise" or "always"
- At process level: THP can be "never" or "defer to system setting"
- At VMA level: no-hint,MADV_HUGEPAGE,MADV_NOHUGEPAGEThat gives us this table to describe how a page fault is handled, according to process state (columns) and vma flags (rows):
| never | madvise | always ----------------|-----------|-----------|----------- no hint | S | S | THP>S MADV_HUGEPAGE | S | THP>S | THP>S MADV_NOHUGEPAGE | S | S | SLegend:
S allocate single page (PTE-mapped) LAF allocate lage anon folio (PTE-mapped) THP allocate THP-sized folio (PMD-mapped) > fallback (usually because vma size/alignment insufficient for folio)
Quoted 7 lines
Principles for Large Anon Folios (LAF)
--------------------------------------David tells us there are use cases today (e.g. qemu live migration) which use
MADV_NOHUGEPAGEto mean "don't fill any PTEs that are not explicitly faulted" and these use cases will break (i.e. functionally incorrect) if this request is not honoured.
I don't remember David saying this. I think he was referring to UFFD, not MADV_NOHUGEPAGE, when discussing what we need to absolutely respect.
Re: [PATCH v4 2/5] mm: LARGE_ANON_FOLIO for improved performance
Quoted 117 lines · Yu Zhao
Quoted 116 lines · Ryan Roberts
+ Kirill
Quoted 30 lines · Ryan Roberts
Introduce
LARGE_ANON_FOLIOfeature, which allows anonymous memory to be allocated in large folios of a determined order. All pages of the large folio are pte-mapped during the same page fault, significantly reducing the number of page faults. The number of per-page operations (e.g. ref counting, rmap management lru list management) are also significantly reduced since those ops now become per-folio.The new behaviour is hidden behind the new
LARGE_ANON_FOLIOKconfig, which defaults to disabled for now; The long term aim is for this to defaut to enabled, but there are some risks around internal fragmentation that need to be better understood first.When enabled, the folio order is determined as such: For a vma, process or system that has explicitly disabled THP, we continue to allocate order-0. THP is most likely disabled to avoid any possible internal fragmentation so we honour that request.
Otherwise, the return value of
arch_wants_pte_order()is used. For vmas that have not explicitly opted-in to use transparent hugepages (e.g. where thp=madvise and the vma does not haveMADV_HUGEPAGE), thenarch_wants_pte_order()is limited to 64K (orPAGE_SIZE, whichever is bigger). This allows for a performance boost without requiring any explicit opt-in from the workload while limitting internal fragmentation.If the preferred order can't be used (e.g. because the folio would breach the bounds of the vma, or because ptes in the region are already mapped) then we fall back to a suitable lower order; first
PAGE_ALLOC_COSTLY_ORDER, then order-0....
Quoted 28 lines
mm/memory.c +28
-0+#define ANON_FOLIO_MAX_ORDER_UNHINTED \ + (ilog2(max_t(unsigned long, SZ_64K, PAGE_SIZE)) - PAGE_SHIFT) + +static int anon_folio_order (struct vm_area_struct *vma) +{ +int order; + +/* +* If THP is explicitly disabled for either the vma, the process or the +* system, then this is very likely intended to limit internal +* fragmentation; in this case, don't attempt to allocate a large +* anonymous folio. +* +* Else, if the vma is eligible for thp, allocate a large folio of the +* size preferred by the arch. Or if the arch requested a very small +* size or didn't request a size, then use PAGE_ALLOC_COSTLY_ORDER, +* which still meets the arch's requirements but means we still take +* advantage of SW optimizations (e.g. fewer page faults). +* +* Finally if thp is enabled but the vma isn't eligible, take the +* arch-preferred size and limit it to ANON_FOLIO_MAX_ORDER_UNHINTED. +* This ensures workloads that have not explicitly opted-in take benefit +* while capping the potential for internal fragmentation. +*/ + +if ((vma->vm_flags &VM_NOHUGEPAGE ) || +test_bit (MMF_DISABLE_THP , &vma->vm_mm->flags) || + !hugepage_flags_enabled ())mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 4106
Quoted 10 lines
mm/memory.c +10
-0+ order =0 ; + else { + order =max (arch_wants_pte_order (),PAGE_ALLOC_COSTLY_ORDER ); + +if (!hugepage_vma_check (vma, vma->vm_flags, false, true, true)) + order =min (order,ANON_FOLIO_MAX_ORDER_UNHINTED ); + } + +return order; +}Hi All,
I'm writing up the conclusions that we arrived at during discussion in the THP meeting yesterday, regarding linkage with exiting THP ABIs. It would be great if I can get explicit "agree" or disagree + rationale from at least David, Yu and Kirill.
In summary; I think we are converging on the approach that is already coded, but I'd like confirmation.
The THP situation today
------------------------ At system level: THP can be set to "never", "madvise" or "always"
- At process level: THP can be "never" or "defer to system setting"
- At VMA level: no-hint,MADV_HUGEPAGE,MADV_NOHUGEPAGEThat gives us this table to describe how a page fault is handled, according to process state (columns) and vma flags (rows):
| never | madvise | always ----------------|-----------|-----------|----------- no hint | S | S | THP>S MADV_HUGEPAGE | S | THP>S | THP>S MADV_NOHUGEPAGE | S | S | SLegend:
S allocate single page (PTE-mapped) LAF allocate lage anon folio (PTE-mapped) THP allocate THP-sized folio (PMD-mapped) > fallback (usually because vma size/alignment insufficient for folio)Principles for Large Anon Folios (LAF)
--------------------------------------
Quoted 8 lines
Quoted 4 lines
David tells us there are use cases today (e.g. qemu live migration) which use
MADV_NOHUGEPAGEto mean "don't fill any PTEs that are not explicitly faulted" and these use cases will break (i.e. functionally incorrect) if this request is not honoured.I don't remember David saying this. I think he was referring to UFFD, not
MADV_NOHUGEPAGE, when discussing what we need to absolutely respect.
My understanding was that MADV_NOHUGEPAGE was being applied to regions *before* UFFD was being registered, and the app relied on MADV_NOHUGEPAGE to not back any unfaulted pages. It's not completely clear to me how not honouring MADV_NOHUGEPAGE would break things though. David?
Re: [PATCH v4 2/5] mm: LARGE_ANON_FOLIO for improved performance
Quoted 123 lines · Ryan Roberts
Quoted 122 lines · Yu Zhao
Quoted 120 lines · Ryan Roberts
+ Kirill
Quoted 30 lines · Ryan Roberts
Introduce
LARGE_ANON_FOLIOfeature, which allows anonymous memory to be allocated in large folios of a determined order. All pages of the large folio are pte-mapped during the same page fault, significantly reducing the number of page faults. The number of per-page operations (e.g. ref counting, rmap management lru list management) are also significantly reduced since those ops now become per-folio.The new behaviour is hidden behind the new
LARGE_ANON_FOLIOKconfig, which defaults to disabled for now; The long term aim is for this to defaut to enabled, but there are some risks around internal fragmentation that need to be better understood first.When enabled, the folio order is determined as such: For a vma, process or system that has explicitly disabled THP, we continue to allocate order-0. THP is most likely disabled to avoid any possible internal fragmentation so we honour that request.
Otherwise, the return value of
arch_wants_pte_order()is used. For vmas that have not explicitly opted-in to use transparent hugepages (e.g. where thp=madvise and the vma does not haveMADV_HUGEPAGE), thenarch_wants_pte_order()is limited to 64K (orPAGE_SIZE, whichever is bigger). This allows for a performance boost without requiring any explicit opt-in from the workload while limitting internal fragmentation.If the preferred order can't be used (e.g. because the folio would breach the bounds of the vma, or because ptes in the region are already mapped) then we fall back to a suitable lower order; first
PAGE_ALLOC_COSTLY_ORDER, then order-0....
Quoted 28 lines
mm/memory.c +28
-0+#define ANON_FOLIO_MAX_ORDER_UNHINTED \ + (ilog2(max_t(unsigned long, SZ_64K, PAGE_SIZE)) - PAGE_SHIFT) + +static int anon_folio_order (struct vm_area_struct *vma) +{ +int order; + +/* +* If THP is explicitly disabled for either the vma, the process or the +* system, then this is very likely intended to limit internal +* fragmentation; in this case, don't attempt to allocate a large +* anonymous folio. +* +* Else, if the vma is eligible for thp, allocate a large folio of the +* size preferred by the arch. Or if the arch requested a very small +* size or didn't request a size, then use PAGE_ALLOC_COSTLY_ORDER, +* which still meets the arch's requirements but means we still take +* advantage of SW optimizations (e.g. fewer page faults). +* +* Finally if thp is enabled but the vma isn't eligible, take the +* arch-preferred size and limit it to ANON_FOLIO_MAX_ORDER_UNHINTED. +* This ensures workloads that have not explicitly opted-in take benefit +* while capping the potential for internal fragmentation. +*/ + +if ((vma->vm_flags &VM_NOHUGEPAGE ) || +test_bit (MMF_DISABLE_THP , &vma->vm_mm->flags) || + !hugepage_flags_enabled ())mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 4106
Quoted 10 lines
mm/memory.c +10
-0+ order =0 ; + else { + order =max (arch_wants_pte_order (),PAGE_ALLOC_COSTLY_ORDER ); + +if (!hugepage_vma_check (vma, vma->vm_flags, false, true, true)) + order =min (order,ANON_FOLIO_MAX_ORDER_UNHINTED ); + } + +return order; +}Hi All,
I'm writing up the conclusions that we arrived at during discussion in the THP meeting yesterday, regarding linkage with exiting THP ABIs. It would be great if I can get explicit "agree" or disagree + rationale from at least David, Yu and Kirill.
In summary; I think we are converging on the approach that is already coded, but I'd like confirmation.
The THP situation today
------------------------ At system level: THP can be set to "never", "madvise" or "always"
- At process level: THP can be "never" or "defer to system setting"
- At VMA level: no-hint,MADV_HUGEPAGE,MADV_NOHUGEPAGEThat gives us this table to describe how a page fault is handled, according to process state (columns) and vma flags (rows):
| never | madvise | always ----------------|-----------|-----------|----------- no hint | S | S | THP>S MADV_HUGEPAGE | S | THP>S | THP>S MADV_NOHUGEPAGE | S | S | SLegend:
S allocate single page (PTE-mapped) LAF allocate lage anon folio (PTE-mapped) THP allocate THP-sized folio (PMD-mapped)Quoted 1 line
fallback (usually because vma size/alignment insufficient for folio)
Principles for Large Anon Folios (LAF)
--------------------------------------David tells us there are use cases today (e.g. qemu live migration) which use
MADV_NOHUGEPAGEto mean "don't fill any PTEs that are not explicitly faulted" and these use cases will break (i.e. functionally incorrect) if this request is not honoured.
Quoted 8 lines
Quoted 3 lines
I don't remember David saying this. I think he was referring to UFFD, not
MADV_NOHUGEPAGE, when discussing what we need to absolutely respect.My understanding was that
MADV_NOHUGEPAGEwas being applied to regions *before* UFFD was being registered, and the app relied onMADV_NOHUGEPAGEto not back any unfaulted pages. It's not completely clear to me how not honouringMADV_NOHUGEPAGEwould break things though. David?
Sorry, I'm still lagging behind on some threads.
Imagine the following for VM postcopy live migration:
(1) Set MADV_NOHUGEPAGE on guest memory and discard all memory (e.g., MADV_DONTNEED), to start with a clean slate.
(2) Migrates some pages during precopy from the source and stores them into guest memory on the destination. Some of the memory locations will have pages populated.
(3) At some point, decide to enable postcopy: enable userfaultfd on guest memory.
(4) Discard *selected* pages again that have been dirtied in the meantime on the source. These are pages that have been migrated previously.
(5) Start running the VM on the destination.
(6) Anything that's not populated will trigger userfaultfd missing faults. Then, you can request them from the source and place them.
Assume you would populate more than required during 2), you can end up not getting userfaultfd faults during 4) and corrupt your guest state. It works if during (2) you migrated all guest memory, or if during 4) you zap everything that still needs migration.
According to the man page:
MADV_NOHUGEPAGE (since Linux 2.6.38): Ensures that memory in the address range specified by addr and length will not be backed by transparent hugepages.
To me, that includes any other page size that is different to the base page size (getpagesize()) and, therefore, the traditional system behavior.
Even if we end up calling these "transparent huge pages of different size" differently and eventually handle them slightly differently.
But I can see why people want to try finding ways around why "never" should not mean "never" when we come up with a new shiny name for "transparent huge pages of different size".
Not that it makes anything clearer or easier if we call 2 MiB pages on x86 THP and 1 MiB pages TLP (Transparent Large Pages ?), whereby 1 MiB pages on s390x are THP ... for sure we can come up with new names for new sizes and cause more confusion.
Most probably we want to clarify in the docs what a transparent huge page is and what these toggles do.
Signature · 3 lines
Cheers, David / dhildenb
Re: [PATCH v4 2/5] mm: LARGE_ANON_FOLIO for improved performance
Quoted 152 lines · David Hildenbrand
Quoted 123 lines · Ryan Roberts
Quoted 122 lines · Yu Zhao
Quoted 120 lines · Ryan Roberts
+ Kirill
Quoted 30 lines · Ryan Roberts
Introduce
LARGE_ANON_FOLIOfeature, which allows anonymous memory to be allocated in large folios of a determined order. All pages of the large folio are pte-mapped during the same page fault, significantly reducing the number of page faults. The number of per-page operations (e.g. ref counting, rmap management lru list management) are also significantly reduced since those ops now become per-folio.The new behaviour is hidden behind the new
LARGE_ANON_FOLIOKconfig, which defaults to disabled for now; The long term aim is for this to defaut to enabled, but there are some risks around internal fragmentation that need to be better understood first.When enabled, the folio order is determined as such: For a vma, process or system that has explicitly disabled THP, we continue to allocate order-0. THP is most likely disabled to avoid any possible internal fragmentation so we honour that request.
Otherwise, the return value of
arch_wants_pte_order()is used. For vmas that have not explicitly opted-in to use transparent hugepages (e.g. where thp=madvise and the vma does not haveMADV_HUGEPAGE), thenarch_wants_pte_order()is limited to 64K (orPAGE_SIZE, whichever is bigger). This allows for a performance boost without requiring any explicit opt-in from the workload while limitting internal fragmentation.If the preferred order can't be used (e.g. because the folio would breach the bounds of the vma, or because ptes in the region are already mapped) then we fall back to a suitable lower order; first
PAGE_ALLOC_COSTLY_ORDER, then order-0....
Quoted 28 lines
mm/memory.c +28
-0+#define ANON_FOLIO_MAX_ORDER_UNHINTED \ + (ilog2(max_t(unsigned long, SZ_64K, PAGE_SIZE)) - PAGE_SHIFT) + +static int anon_folio_order (struct vm_area_struct *vma) +{ +int order; + +/* +* If THP is explicitly disabled for either the vma, the process or the +* system, then this is very likely intended to limit internal +* fragmentation; in this case, don't attempt to allocate a large +* anonymous folio. +* +* Else, if the vma is eligible for thp, allocate a large folio of the +* size preferred by the arch. Or if the arch requested a very small +* size or didn't request a size, then use PAGE_ALLOC_COSTLY_ORDER, +* which still meets the arch's requirements but means we still take +* advantage of SW optimizations (e.g. fewer page faults). +* +* Finally if thp is enabled but the vma isn't eligible, take the +* arch-preferred size and limit it to ANON_FOLIO_MAX_ORDER_UNHINTED. +* This ensures workloads that have not explicitly opted-in take benefit +* while capping the potential for internal fragmentation. +*/ + +if ((vma->vm_flags &VM_NOHUGEPAGE ) || +test_bit (MMF_DISABLE_THP , &vma->vm_mm->flags) || + !hugepage_flags_enabled ())mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 4106
Quoted 10 lines
mm/memory.c +10
-0+ order =0 ; + else { + order =max (arch_wants_pte_order (),PAGE_ALLOC_COSTLY_ORDER ); + +if (!hugepage_vma_check (vma, vma->vm_flags, false, true, true)) + order =min (order,ANON_FOLIO_MAX_ORDER_UNHINTED ); + } + +return order; +}Hi All,
I'm writing up the conclusions that we arrived at during discussion in the THP meeting yesterday, regarding linkage with exiting THP ABIs. It would be great if I can get explicit "agree" or disagree + rationale from at least David, Yu and Kirill.
In summary; I think we are converging on the approach that is already coded, but I'd like confirmation.
The THP situation today
------------------------ At system level: THP can be set to "never", "madvise" or "always"
- At process level: THP can be "never" or "defer to system setting"
- At VMA level: no-hint,MADV_HUGEPAGE,MADV_NOHUGEPAGEThat gives us this table to describe how a page fault is handled, according to process state (columns) and vma flags (rows):
| never | madvise | always ----------------|-----------|-----------|----------- no hint | S | S | THP>S MADV_HUGEPAGE | S | THP>S | THP>S MADV_NOHUGEPAGE | S | S | SLegend:
S allocate single page (PTE-mapped) LAF allocate lage anon folio (PTE-mapped) THP allocate THP-sized folio (PMD-mapped)Quoted 1 line
fallback (usually because vma size/alignment insufficient for folio)
Principles for Large Anon Folios (LAF)
--------------------------------------David tells us there are use cases today (e.g. qemu live migration) which use
MADV_NOHUGEPAGEto mean "don't fill any PTEs that are not explicitly faulted" and these use cases will break (i.e. functionally incorrect) if this request is not honoured.Quoted 8 lines
Quoted 3 lines
I don't remember David saying this. I think he was referring to UFFD, not
MADV_NOHUGEPAGE, when discussing what we need to absolutely respect.My understanding was that
MADV_NOHUGEPAGEwas being applied to regions *before* UFFD was being registered, and the app relied onMADV_NOHUGEPAGEto not back any unfaulted pages. It's not completely clear to me how not honouringMADV_NOHUGEPAGEwould break things though. David?Sorry, I'm still lagging behind on some threads.
Imagine the following for VM postcopy live migration:
(1) Set
MADV_NOHUGEPAGEon guest memory and discard all memory (e.g.,MADV_DONTNEED), to start with a clean slate.
(2) Migrates some pages during precopy from the source and stores them into guest memory on the destination. Some of the memory locations will have pages populated.
(3) At some point, decide to enable postcopy: enable userfaultfd on guest memory.
(4) Discard *selected* pages again that have been dirtied in the meantime on the source. These are pages that have been migrated previously.
(5) Start running the VM on the destination.
(6) Anything that's not populated will trigger userfaultfd missing faults. Then, you can request them from the source and place them.
Quoted 4 lines
Assume you would populate more than required during 2), you can end up not getting userfaultfd faults during 4) and corrupt your guest state. It works if during (2) you migrated all guest memory, or if during 4) you zap everything that still needs migr
I see what you mean now. Thanks.
Yes, in this case we have to interpret MADV_NOHUGEPAGE as nothing >4KB.
Re: [PATCH v4 2/5] mm: LARGE_ANON_FOLIO for improved performance
Quoted 153 lines · Yu Zhao
Quoted 152 lines · David Hildenbrand
Quoted 123 lines · Ryan Roberts
Quoted 122 lines · Yu Zhao
Quoted 120 lines · Ryan Roberts
+ Kirill
Quoted 30 lines · Ryan Roberts
Introduce
LARGE_ANON_FOLIOfeature, which allows anonymous memory to be allocated in large folios of a determined order. All pages of the large folio are pte-mapped during the same page fault, significantly reducing the number of page faults. The number of per-page operations (e.g. ref counting, rmap management lru list management) are also significantly reduced since those ops now become per-folio.The new behaviour is hidden behind the new
LARGE_ANON_FOLIOKconfig, which defaults to disabled for now; The long term aim is for this to defaut to enabled, but there are some risks around internal fragmentation that need to be better understood first.When enabled, the folio order is determined as such: For a vma, process or system that has explicitly disabled THP, we continue to allocate order-0. THP is most likely disabled to avoid any possible internal fragmentation so we honour that request.
Otherwise, the return value of
arch_wants_pte_order()is used. For vmas that have not explicitly opted-in to use transparent hugepages (e.g. where thp=madvise and the vma does not haveMADV_HUGEPAGE), thenarch_wants_pte_order()is limited to 64K (orPAGE_SIZE, whichever is bigger). This allows for a performance boost without requiring any explicit opt-in from the workload while limitting internal fragmentation.If the preferred order can't be used (e.g. because the folio would breach the bounds of the vma, or because ptes in the region are already mapped) then we fall back to a suitable lower order; first
PAGE_ALLOC_COSTLY_ORDER, then order-0....
Quoted 28 lines
mm/memory.c +28
-0+#define ANON_FOLIO_MAX_ORDER_UNHINTED \ + (ilog2(max_t(unsigned long, SZ_64K, PAGE_SIZE)) - PAGE_SHIFT) + +static int anon_folio_order (struct vm_area_struct *vma) +{ +int order; + +/* +* If THP is explicitly disabled for either the vma, the process or the +* system, then this is very likely intended to limit internal +* fragmentation; in this case, don't attempt to allocate a large +* anonymous folio. +* +* Else, if the vma is eligible for thp, allocate a large folio of the +* size preferred by the arch. Or if the arch requested a very small +* size or didn't request a size, then use PAGE_ALLOC_COSTLY_ORDER, +* which still meets the arch's requirements but means we still take +* advantage of SW optimizations (e.g. fewer page faults). +* +* Finally if thp is enabled but the vma isn't eligible, take the +* arch-preferred size and limit it to ANON_FOLIO_MAX_ORDER_UNHINTED. +* This ensures workloads that have not explicitly opted-in take benefit +* while capping the potential for internal fragmentation. +*/ + +if ((vma->vm_flags &VM_NOHUGEPAGE ) || +test_bit (MMF_DISABLE_THP , &vma->vm_mm->flags) || + !hugepage_flags_enabled ())mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 4106
Quoted 10 lines
mm/memory.c +10
-0+ order =0 ; + else { + order =max (arch_wants_pte_order (),PAGE_ALLOC_COSTLY_ORDER ); + +if (!hugepage_vma_check (vma, vma->vm_flags, false, true, true)) + order =min (order,ANON_FOLIO_MAX_ORDER_UNHINTED ); + } + +return order; +}Hi All,
I'm writing up the conclusions that we arrived at during discussion in the THP meeting yesterday, regarding linkage with exiting THP ABIs. It would be great if I can get explicit "agree" or disagree + rationale from at least David, Yu and Kirill.
In summary; I think we are converging on the approach that is already coded, but I'd like confirmation.
The THP situation today
------------------------ At system level: THP can be set to "never", "madvise" or "always"
- At process level: THP can be "never" or "defer to system setting"
- At VMA level: no-hint,MADV_HUGEPAGE,MADV_NOHUGEPAGEThat gives us this table to describe how a page fault is handled, according to process state (columns) and vma flags (rows):
| never | madvise | always ----------------|-----------|-----------|----------- no hint | S | S | THP>S MADV_HUGEPAGE | S | THP>S | THP>S MADV_NOHUGEPAGE | S | S | SLegend:
S allocate single page (PTE-mapped) LAF allocate lage anon folio (PTE-mapped) THP allocate THP-sized folio (PMD-mapped)Quoted 1 line
fallback (usually because vma size/alignment insufficient for folio)
Principles for Large Anon Folios (LAF)
--------------------------------------David tells us there are use cases today (e.g. qemu live migration) which use
MADV_NOHUGEPAGEto mean "don't fill any PTEs that are not explicitly faulted" and these use cases will break (i.e. functionally incorrect) if this request is not honoured.Quoted 8 lines
Quoted 3 lines
I don't remember David saying this. I think he was referring to UFFD, not
MADV_NOHUGEPAGE, when discussing what we need to absolutely respect.My understanding was that
MADV_NOHUGEPAGEwas being applied to regions *before* UFFD was being registered, and the app relied onMADV_NOHUGEPAGEto not back any unfaulted pages. It's not completely clear to me how not honouringMADV_NOHUGEPAGEwould break things though. David?Sorry, I'm still lagging behind on some threads.
Imagine the following for VM postcopy live migration:
(1) Set
MADV_NOHUGEPAGEon guest memory and discard all memory (e.g.,MADV_DONTNEED), to start with a clean slate.
(2) Migrates some pages during precopy from the source and stores them into guest memory on the destination. Some of the memory locations will have pages populated.
(3) At some point, decide to enable postcopy: enable userfaultfd on guest memory.
(4) Discard *selected* pages again that have been dirtied in the meantime on the source. These are pages that have been migrated previously.
(5) Start running the VM on the destination.
(6) Anything that's not populated will trigger userfaultfd missing faults. Then, you can request them from the source and place them.
Quoted 8 lines
Quoted 4 lines
Assume you would populate more than required during 2), you can end up not getting userfaultfd faults during 4) and corrupt your guest state. It works if during (2) you migrated all guest memory, or if during 4) you zap everything that still needs migr
I see what you mean now. Thanks.
Yes, in this case we have to interpret
MADV_NOHUGEPAGEas nothing >4KB.
Note that it's still even unclear to me why we want to *not* call these things THP. It would certainly make everything less confusing if we call them THP, but with additional attributes.
I think that is one of the first things we should figure out because it also indirectly tells us what all these toggles mean and how/if we should redefine them (and if they even apply).
Currently THP == PMD size
In 2016, Hugh already envisioned PUD/PGD THP (see 49920d28781d ("mm: make transparent hugepage size public")) when he explicitly exposed "hpage_pmd_size". Not "hpage_size".
For hugetlb on arm64 we already support various sizes that are < PMD size and *not* call them differently. It's a huge(tlb) page. Sometimes we refer to them as cont-PTE hugetlb pages.
So, nowadays we do have "PMD-sized THP", someday we might have "PUD-sized THP". Can't we come up with a name to describe sub-PMD THP?
Is it really of value if we invent a new term for them? Yes, I was not enjoying "Flexible THP".
Once we figured that out, we should figure out if MADV_HUGEPAGE meant "only PMD-sized THP" or anything else?
Also, we can then figure out if MADV_NOHUGEPAGE meant "only PMD-sized THP" or anything else?
The simplest approach to me would be "they imply any THP, and once we need more tunables we might add some", similar to what Kirill also raised.
Again, it's all unclear to me at this point and I'm happy to hear opinions, because I really don't know.
Signature · 3 lines
Cheers, David / dhildenb
Re: [PATCH v4 2/5] mm: LARGE_ANON_FOLIO for improved performance
Quoted 197 lines · David Hildenbrand
Quoted 153 lines · Yu Zhao
Quoted 152 lines · David Hildenbrand
Quoted 123 lines · Ryan Roberts
Quoted 122 lines · Yu Zhao
Quoted 120 lines · Ryan Roberts
+ Kirill
Quoted 30 lines · Ryan Roberts
Introduce
LARGE_ANON_FOLIOfeature, which allows anonymous memory to be allocated in large folios of a determined order. All pages of the large folio are pte-mapped during the same page fault, significantly reducing the number of page faults. The number of per-page operations (e.g. ref counting, rmap management lru list management) are also significantly reduced since those ops now become per-folio.The new behaviour is hidden behind the new
LARGE_ANON_FOLIOKconfig, which defaults to disabled for now; The long term aim is for this to defaut to enabled, but there are some risks around internal fragmentation that need to be better understood first.When enabled, the folio order is determined as such: For a vma, process or system that has explicitly disabled THP, we continue to allocate order-0. THP is most likely disabled to avoid any possible internal fragmentation so we honour that request.
Otherwise, the return value of
arch_wants_pte_order()is used. For vmas that have not explicitly opted-in to use transparent hugepages (e.g. where thp=madvise and the vma does not haveMADV_HUGEPAGE), thenarch_wants_pte_order()is limited to 64K (orPAGE_SIZE, whichever is bigger). This allows for a performance boost without requiring any explicit opt-in from the workload while limitting internal fragmentation.If the preferred order can't be used (e.g. because the folio would breach the bounds of the vma, or because ptes in the region are already mapped) then we fall back to a suitable lower order; first
PAGE_ALLOC_COSTLY_ORDER, then order-0....
Quoted 28 lines
(quoted diff) +28
-0+#define ANON_FOLIO_MAX_ORDER_UNHINTED \ + (ilog2(max_t(unsigned long, SZ_64K, PAGE_SIZE)) - PAGE_SHIFT) + +static int anon_folio_order(struct vm_area_struct *vma) +{ + int order; + + /* + * If THP is explicitly disabled for either the vma, the process or the + * system, then this is very likely intended to limit internal + * fragmentation; in this case, don't attempt to allocate a large + * anonymous folio. + * + * Else, if the vma is eligible for thp, allocate a large folio of the + * size preferred by the arch. Or if the arch requested a very small + * size or didn't request a size, then use PAGE_ALLOC_COSTLY_ORDER, + * which still meets the arch's requirements but means we still take + * advantage of SW optimizations (e.g. fewer page faults). + * + * Finally if thp is enabled but the vma isn't eligible, take the + * arch-preferred size and limit it to ANON_FOLIO_MAX_ORDER_UNHINTED. + * This ensures workloads that have not explicitly opted-in take benefit + * while capping the potential for internal fragmentation. + */ + + if ((vma->vm_flags & VM_NOHUGEPAGE) || + test_bit(MMF_DISABLE_THP, &vma->vm_mm->flags) || + !hugepage_flags_enabled())Review
Quoted 10 lines
(quoted diff) +10
-0+ order = 0; + else { + order = max(arch_wants_pte_order(), PAGE_ALLOC_COSTLY_ORDER); + + if (!hugepage_vma_check(vma, vma->vm_flags, false, true, true)) + order = min(order, ANON_FOLIO_MAX_ORDER_UNHINTED); + } + + return order; +}Hi All,
I'm writing up the conclusions that we arrived at during discussion in the THP meeting yesterday, regarding linkage with exiting THP ABIs. It would be great if I can get explicit "agree" or disagree + rationale from at least David, Yu and Kirill.
In summary; I think we are converging on the approach that is already coded, but I'd like confirmation.
The THP situation today
------------------------ At system level: THP can be set to "never", "madvise" or "always"
- At process level: THP can be "never" or "defer to system setting"
- At VMA level: no-hint,MADV_HUGEPAGE,MADV_NOHUGEPAGEThat gives us this table to describe how a page fault is handled, according to process state (columns) and vma flags (rows):
| never | madvise | always ----------------|-----------|-----------|----------- no hint | S | S | THP>S MADV_HUGEPAGE | S | THP>S | THP>S MADV_NOHUGEPAGE | S | S | SLegend:
S allocate single page (PTE-mapped) LAF allocate lage anon folio (PTE-mapped) THP allocate THP-sized folio (PMD-mapped)Quoted 1 line
fallback (usually because vma size/alignment insufficient for folio)
Principles for Large Anon Folios (LAF)
--------------------------------------David tells us there are use cases today (e.g. qemu live migration) which use
MADV_NOHUGEPAGEto mean "don't fill any PTEs that are not explicitly faulted" and these use cases will break (i.e. functionally incorrect) if this request is not honoured.Quoted 8 lines
Quoted 3 lines
I don't remember David saying this. I think he was referring to UFFD, not
MADV_NOHUGEPAGE, when discussing what we need to absolutely respect.My understanding was that
MADV_NOHUGEPAGEwas being applied to regions *before* UFFD was being registered, and the app relied onMADV_NOHUGEPAGEto not back any unfaulted pages. It's not completely clear to me how not honouringMADV_NOHUGEPAGEwould break things though. David?Sorry, I'm still lagging behind on some threads.
Imagine the following for VM postcopy live migration:
(1) Set
MADV_NOHUGEPAGEon guest memory and discard all memory (e.g.,MADV_DONTNEED), to start with a clean slate.
(2) Migrates some pages during precopy from the source and stores them into guest memory on the destination. Some of the memory locations will have pages populated.
(3) At some point, decide to enable postcopy: enable userfaultfd on guest memory.
(4) Discard *selected* pages again that have been dirtied in the meantime on the source. These are pages that have been migrated previously.
(5) Start running the VM on the destination.
(6) Anything that's not populated will trigger userfaultfd missing faults. Then, you can request them from the source and place them.Quoted 8 lines
Quoted 4 lines
Assume you would populate more than required during 2), you can end up not getting userfaultfd faults during 4) and corrupt your guest state. It works if during (2) you migrated all guest memory, or if during 4) you zap everything that still needs migr
I see what you mean now. Thanks.
Yes, in this case we have to interpret
MADV_NOHUGEPAGEas nothing >4KB.Note that it's still even unclear to me why we want to *not* call these things THP. It would certainly make everything less confusing if we call them THP, but with additional attributes.
I think that is one of the first things we should figure out because it also indirectly tells us what all these toggles mean and how/if we should redefine them (and if they even apply).
Currently THP == PMD size
In 2016, Hugh already envisioned PUD/PGD THP (see
49920d28781d("mm:
make transparent hugepage size public")) when he explicitly exposed "hpage_pmd_size". Not "hpage_size".For hugetlb on arm64 we already support various sizes that are < PMD size and *not* call them differently. It's a huge(tlb) page. Sometimes we refer to them as cont-PTE hugetlb pages.
So, nowadays we do have "PMD-sized THP", someday we might have "PUD-sized THP". Can't we come up with a name to describe sub-PMD THP?
Is it really of value if we invent a new term for them? Yes, I was not enjoying "Flexible THP".
Once we figured that out, we should figure out if
MADV_HUGEPAGEmeant "only PMD-sized THP" or anything else?Also, we can then figure out if
MADV_NOHUGEPAGEmeant "only PMD-sized THP" or anything else?
Quoted 6 lines
The simplest approach to me would be "they imply any THP, and once we need more tunables we might add some", similar to what Kirill also raised.
Again, it's all unclear to me at this point and I'm happy to hear opinions, because I really don't know.
I agree these points require more discussion. But I don't think we need to conclude them now, unless they cause correctness issues like ignoring MADV_NOHUGEPAGE would. My concern is that if we decide to go with "they imply any THP" and *expose this to userspace now*, we might regret later.
Also that "Flexible THP" Kconfig is just a placeholder, from my POV. It should be removed after we nail down the runtime ABI, which again IMO, isn't now.
Re: [PATCH v4 2/5] mm: LARGE_ANON_FOLIO for improved performance
Quoted 202 lines · Yu Zhao
Quoted 201 lines · David Hildenbrand
Quoted 153 lines · Yu Zhao
Quoted 152 lines · David Hildenbrand
Quoted 123 lines · Ryan Roberts
Quoted 122 lines · Yu Zhao
Quoted 120 lines · Ryan Roberts
+ Kirill
Quoted 30 lines · Ryan Roberts
Introduce
LARGE_ANON_FOLIOfeature, which allows anonymous memory to be allocated in large folios of a determined order. All pages of the large folio are pte-mapped during the same page fault, significantly reducing the number of page faults. The number of per-page operations (e.g. ref counting, rmap management lru list management) are also significantly reduced since those ops now become per-folio.The new behaviour is hidden behind the new
LARGE_ANON_FOLIOKconfig, which defaults to disabled for now; The long term aim is for this to defaut to enabled, but there are some risks around internal fragmentation that need to be better understood first.When enabled, the folio order is determined as such: For a vma, process or system that has explicitly disabled THP, we continue to allocate order-0. THP is most likely disabled to avoid any possible internal fragmentation so we honour that request.
Otherwise, the return value of
arch_wants_pte_order()is used. For vmas that have not explicitly opted-in to use transparent hugepages (e.g. where thp=madvise and the vma does not haveMADV_HUGEPAGE), thenarch_wants_pte_order()is limited to 64K (orPAGE_SIZE, whichever is bigger). This allows for a performance boost without requiring any explicit opt-in from the workload while limitting internal fragmentation.If the preferred order can't be used (e.g. because the folio would breach the bounds of the vma, or because ptes in the region are already mapped) then we fall back to a suitable lower order; first
PAGE_ALLOC_COSTLY_ORDER, then order-0....
Quoted 28 lines
(quoted diff) +28
-0+#define ANON_FOLIO_MAX_ORDER_UNHINTED \ + (ilog2(max_t(unsigned long, SZ_64K, PAGE_SIZE)) - PAGE_SHIFT) + +static int anon_folio_order(struct vm_area_struct *vma) +{ + int order; + + /* + * If THP is explicitly disabled for either the vma, the process or the + * system, then this is very likely intended to limit internal + * fragmentation; in this case, don't attempt to allocate a large + * anonymous folio. + * + * Else, if the vma is eligible for thp, allocate a large folio of the + * size preferred by the arch. Or if the arch requested a very small + * size or didn't request a size, then use PAGE_ALLOC_COSTLY_ORDER, + * which still meets the arch's requirements but means we still take + * advantage of SW optimizations (e.g. fewer page faults). + * + * Finally if thp is enabled but the vma isn't eligible, take the + * arch-preferred size and limit it to ANON_FOLIO_MAX_ORDER_UNHINTED. + * This ensures workloads that have not explicitly opted-in take benefit + * while capping the potential for internal fragmentation. + */ + + if ((vma->vm_flags & VM_NOHUGEPAGE) || + test_bit(MMF_DISABLE_THP, &vma->vm_mm->flags) || + !hugepage_flags_enabled())Review
Quoted 10 lines
(quoted diff) +10
-0+ order = 0; + else { + order = max(arch_wants_pte_order(), PAGE_ALLOC_COSTLY_ORDER); + + if (!hugepage_vma_check(vma, vma->vm_flags, false, true, true)) + order = min(order, ANON_FOLIO_MAX_ORDER_UNHINTED); + } + + return order; +}Hi All,
I'm writing up the conclusions that we arrived at during discussion in the THP meeting yesterday, regarding linkage with exiting THP ABIs. It would be great if I can get explicit "agree" or disagree + rationale from at least David, Yu and Kirill.
In summary; I think we are converging on the approach that is already coded, but I'd like confirmation.
The THP situation today
------------------------ At system level: THP can be set to "never", "madvise" or "always"
- At process level: THP can be "never" or "defer to system setting"
- At VMA level: no-hint,MADV_HUGEPAGE,MADV_NOHUGEPAGEThat gives us this table to describe how a page fault is handled, according to process state (columns) and vma flags (rows):
| never | madvise | always ----------------|-----------|-----------|----------- no hint | S | S | THP>S MADV_HUGEPAGE | S | THP>S | THP>S MADV_NOHUGEPAGE | S | S | SLegend:
S allocate single page (PTE-mapped) LAF allocate lage anon folio (PTE-mapped) THP allocate THP-sized folio (PMD-mapped)Quoted 1 line
fallback (usually because vma size/alignment insufficient for folio)
Principles for Large Anon Folios (LAF)
--------------------------------------David tells us there are use cases today (e.g. qemu live migration) which use
MADV_NOHUGEPAGEto mean "don't fill any PTEs that are not explicitly faulted" and these use cases will break (i.e. functionally incorrect) if this request is not honoured.Quoted 8 lines
Quoted 3 lines
I don't remember David saying this. I think he was referring to UFFD, not
MADV_NOHUGEPAGE, when discussing what we need to absolutely respect.My understanding was that
MADV_NOHUGEPAGEwas being applied to regions *before* UFFD was being registered, and the app relied onMADV_NOHUGEPAGEto not back any unfaulted pages. It's not completely clear to me how not honouringMADV_NOHUGEPAGEwould break things though. David?Sorry, I'm still lagging behind on some threads.
Imagine the following for VM postcopy live migration:
(1) Set
MADV_NOHUGEPAGEon guest memory and discard all memory (e.g.,MADV_DONTNEED), to start with a clean slate.
(2) Migrates some pages during precopy from the source and stores them into guest memory on the destination. Some of the memory locations will have pages populated.
(3) At some point, decide to enable postcopy: enable userfaultfd on guest memory.
(4) Discard *selected* pages again that have been dirtied in the meantime on the source. These are pages that have been migrated previously.
(5) Start running the VM on the destination.
(6) Anything that's not populated will trigger userfaultfd missing faults. Then, you can request them from the source and place them.Quoted 8 lines
Quoted 4 lines
Assume you would populate more than required during 2), you can end up not getting userfaultfd faults during 4) and corrupt your guest state. It works if during (2) you migrated all guest memory, or if during 4) you zap everything that still needs migr
I see what you mean now. Thanks.
Yes, in this case we have to interpret
MADV_NOHUGEPAGEas nothing >4KB.Note that it's still even unclear to me why we want to *not* call these things THP. It would certainly make everything less confusing if we call them THP, but with additional attributes.
I think that is one of the first things we should figure out because it also indirectly tells us what all these toggles mean and how/if we should redefine them (and if they even apply).
Currently THP == PMD size
In 2016, Hugh already envisioned PUD/PGD THP (see
49920d28781d("mm:
make transparent hugepage size public")) when he explicitly exposed "hpage_pmd_size". Not "hpage_size".For hugetlb on arm64 we already support various sizes that are < PMD size and *not* call them differently. It's a huge(tlb) page. Sometimes we refer to them as cont-PTE hugetlb pages.
So, nowadays we do have "PMD-sized THP", someday we might have "PUD-sized THP". Can't we come up with a name to describe sub-PMD THP?
Is it really of value if we invent a new term for them? Yes, I was not enjoying "Flexible THP".
Once we figured that out, we should figure out if
MADV_HUGEPAGEmeant "only PMD-sized THP" or anything else?Also, we can then figure out if
MADV_NOHUGEPAGEmeant "only PMD-sized THP" or anything else?The simplest approach to me would be "they imply any THP, and once we need more tunables we might add some", similar to what Kirill also raised.
Quoted 8 lines
Quoted 2 lines
Again, it's all unclear to me at this point and I'm happy to hear opinions, because I really don't know.
I agree these points require more discussion. But I don't think we need to conclude them now, unless they cause correctness issues like ignoring
MADV_NOHUGEPAGEwould. My concern is that if we decide to go with "they imply any THP" and *expose this to userspace now*, we might regret later.
If we don't think they are THP, probably MADV_NOHUGEPAGE should not apply and we should be ready to find other ways to deal with the mess we eventually create. If we want to go down that path, sure.
If they are THP, to me there is not really a question if MADV_NOHUGEPAGE applies to them or not. Unless we want to build a confusing piece of software ;)
Signature · 9 lines
Cheers, David / dhildenb _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Re: [PATCH v4 2/5] mm: LARGE_ANON_FOLIO for improved performance
Quoted 212 lines · David Hildenbrand
Quoted 202 lines · Yu Zhao
Quoted 201 lines · David Hildenbrand
Quoted 153 lines · Yu Zhao
Quoted 152 lines · David Hildenbrand
Quoted 123 lines · Ryan Roberts
Quoted 122 lines · Yu Zhao
Quoted 120 lines · Ryan Roberts
+ Kirill
Quoted 30 lines · Ryan Roberts
Introduce
LARGE_ANON_FOLIOfeature, which allows anonymous memory to be allocated in large folios of a determined order. All pages of the large folio are pte-mapped during the same page fault, significantly reducing the number of page faults. The number of per-page operations (e.g. ref counting, rmap management lru list management) are also significantly reduced since those ops now become per-folio.The new behaviour is hidden behind the new
LARGE_ANON_FOLIOKconfig, which defaults to disabled for now; The long term aim is for this to defaut to enabled, but there are some risks around internal fragmentation that need to be better understood first.When enabled, the folio order is determined as such: For a vma, process or system that has explicitly disabled THP, we continue to allocate order-0. THP is most likely disabled to avoid any possible internal fragmentation so we honour that request.
Otherwise, the return value of
arch_wants_pte_order()is used. For vmas that have not explicitly opted-in to use transparent hugepages (e.g. where thp=madvise and the vma does not haveMADV_HUGEPAGE), thenarch_wants_pte_order()is limited to 64K (orPAGE_SIZE, whichever is bigger). This allows for a performance boost without requiring any explicit opt-in from the workload while limitting internal fragmentation.If the preferred order can't be used (e.g. because the folio would breach the bounds of the vma, or because ptes in the region are already mapped) then we fall back to a suitable lower order; first
PAGE_ALLOC_COSTLY_ORDER, then order-0....
Quoted 28 lines
(quoted diff) +28
-0+#define ANON_FOLIO_MAX_ORDER_UNHINTED \ + (ilog2(max_t(unsigned long, SZ_64K, PAGE_SIZE)) - PAGE_SHIFT) + +static int anon_folio_order(struct vm_area_struct *vma) +{ + int order; + + /* + * If THP is explicitly disabled for either the vma, the process or the + * system, then this is very likely intended to limit internal + * fragmentation; in this case, don't attempt to allocate a large + * anonymous folio. + * + * Else, if the vma is eligible for thp, allocate a large folio of the + * size preferred by the arch. Or if the arch requested a very small + * size or didn't request a size, then use PAGE_ALLOC_COSTLY_ORDER, + * which still meets the arch's requirements but means we still take + * advantage of SW optimizations (e.g. fewer page faults). + * + * Finally if thp is enabled but the vma isn't eligible, take the + * arch-preferred size and limit it to ANON_FOLIO_MAX_ORDER_UNHINTED. + * This ensures workloads that have not explicitly opted-in take benefit + * while capping the potential for internal fragmentation. + */ + + if ((vma->vm_flags & VM_NOHUGEPAGE) || + test_bit(MMF_DISABLE_THP, &vma->vm_mm->flags) || + !hugepage_flags_enabled())Review
Quoted 10 lines
(quoted diff) +10
-0+ order = 0; + else { + order = max(arch_wants_pte_order(), PAGE_ALLOC_COSTLY_ORDER); + + if (!hugepage_vma_check(vma, vma->vm_flags, false, true, true)) + order = min(order, ANON_FOLIO_MAX_ORDER_UNHINTED); + } + + return order; +}Hi All,
I'm writing up the conclusions that we arrived at during discussion in the THP meeting yesterday, regarding linkage with exiting THP ABIs. It would be great if I can get explicit "agree" or disagree + rationale from at least David, Yu and Kirill.
In summary; I think we are converging on the approach that is already coded, but I'd like confirmation.
The THP situation today
------------------------ At system level: THP can be set to "never", "madvise" or "always"
- At process level: THP can be "never" or "defer to system setting"
- At VMA level: no-hint,MADV_HUGEPAGE,MADV_NOHUGEPAGEThat gives us this table to describe how a page fault is handled, according to process state (columns) and vma flags (rows):
| never | madvise | always ----------------|-----------|-----------|----------- no hint | S | S | THP>S MADV_HUGEPAGE | S | THP>S | THP>S MADV_NOHUGEPAGE | S | S | SLegend:
S allocate single page (PTE-mapped) LAF allocate lage anon folio (PTE-mapped) THP allocate THP-sized folio (PMD-mapped)Quoted 1 line
fallback (usually because vma size/alignment insufficient for folio)
Principles for Large Anon Folios (LAF)
--------------------------------------David tells us there are use cases today (e.g. qemu live migration) which use
MADV_NOHUGEPAGEto mean "don't fill any PTEs that are not explicitly faulted" and these use cases will break (i.e. functionally incorrect) if this request is not honoured.Quoted 8 lines
Quoted 3 lines
I don't remember David saying this. I think he was referring to UFFD, not
MADV_NOHUGEPAGE, when discussing what we need to absolutely respect.My understanding was that
MADV_NOHUGEPAGEwas being applied to regions *before* UFFD was being registered, and the app relied onMADV_NOHUGEPAGEto not back any unfaulted pages. It's not completely clear to me how not honouringMADV_NOHUGEPAGEwould break things though. David?Sorry, I'm still lagging behind on some threads.
Imagine the following for VM postcopy live migration:
(1) Set
MADV_NOHUGEPAGEon guest memory and discard all memory (e.g.,MADV_DONTNEED), to start with a clean slate.
(2) Migrates some pages during precopy from the source and stores them into guest memory on the destination. Some of the memory locations will have pages populated.
(3) At some point, decide to enable postcopy: enable userfaultfd on guest memory.
(4) Discard *selected* pages again that have been dirtied in the meantime on the source. These are pages that have been migrated previously.
(5) Start running the VM on the destination.
(6) Anything that's not populated will trigger userfaultfd missing faults. Then, you can request them from the source and place them.Quoted 8 lines
Quoted 4 lines
Assume you would populate more than required during 2), you can end up not getting userfaultfd faults during 4) and corrupt your guest state. It works if during (2) you migrated all guest memory, or if during 4) you zap everything that still needs migr
I see what you mean now. Thanks.
Yes, in this case we have to interpret
MADV_NOHUGEPAGEas nothing >4KB.Note that it's still even unclear to me why we want to *not* call these things THP. It would certainly make everything less confusing if we call them THP, but with additional attributes.
I think that is one of the first things we should figure out because it also indirectly tells us what all these toggles mean and how/if we should redefine them (and if they even apply).
Currently THP == PMD size
In 2016, Hugh already envisioned PUD/PGD THP (see
49920d28781d("mm:
make transparent hugepage size public")) when he explicitly exposed "hpage_pmd_size". Not "hpage_size".For hugetlb on arm64 we already support various sizes that are < PMD size and *not* call them differently. It's a huge(tlb) page. Sometimes we refer to them as cont-PTE hugetlb pages.
So, nowadays we do have "PMD-sized THP", someday we might have "PUD-sized THP". Can't we come up with a name to describe sub-PMD THP?
Is it really of value if we invent a new term for them? Yes, I was not enjoying "Flexible THP".
Once we figured that out, we should figure out if
MADV_HUGEPAGEmeant "only PMD-sized THP" or anything else?Also, we can then figure out if
MADV_NOHUGEPAGEmeant "only PMD-sized THP" or anything else?The simplest approach to me would be "they imply any THP, and once we need more tunables we might add some", similar to what Kirill also raised.
Quoted 8 lines
Quoted 2 lines
Again, it's all unclear to me at this point and I'm happy to hear opinions, because I really don't know.
I agree these points require more discussion. But I don't think we need to conclude them now, unless they cause correctness issues like ignoring
MADV_NOHUGEPAGEwould. My concern is that if we decide to go with "they imply any THP" and *expose this to userspace now*, we might regret later.
Quoted 3 lines
If we don't think they are THP, probably
MADV_NOHUGEPAGEshould not apply and we should be ready to find other ways to deal with the mess we eventually create. If we want to go down that path, sure.If they are THP, to me there is not really a question if
MADV_NOHUGEPAGEapplies to them or not. Unless we want to build a confusing piece of software ;)
I think it is good to call them THP, since they are transparent huge (>order-0) pages. But the concern is that before we have a reasonable management policy for order>0 && order<9 THPs, mixing them with existing order-9 THP might give user unexpected performance outcome. Unless we are sure they will always performance improvement, we might repeat the old THP path, namely users begin to disable THP by default to avoid unexpected performance hiccup. That is the reason Yu wants to separate LAF from THP at the moment.
Maybe call it THP (experimental) for now and merge it to THP when we have a stable policy. For knobs, we might add "any-order" to the existing "never", "madvise" and another interface to specify max hinted order (enforcing <9) for "any-order". Later, we can allow users to specify any max hinted order, including 9. Just an idea.
--
Best Regards,
Yan, Zi
Re: [PATCH v4 2/5] mm: LARGE_ANON_FOLIO for improved performance
Quoted 226 lines · Zi Yan
Quoted 212 lines · David Hildenbrand
Quoted 202 lines · Yu Zhao
Quoted 201 lines · David Hildenbrand
Quoted 153 lines · Yu Zhao
Quoted 152 lines · David Hildenbrand
Quoted 123 lines · Ryan Roberts
Quoted 122 lines · Yu Zhao
Quoted 120 lines · Ryan Roberts
+ Kirill
Quoted 30 lines · Ryan Roberts
Introduce
LARGE_ANON_FOLIOfeature, which allows anonymous memory to be allocated in large folios of a determined order. All pages of the large folio are pte-mapped during the same page fault, significantly reducing the number of page faults. The number of per-page operations (e.g. ref counting, rmap management lru list management) are also significantly reduced since those ops now become per-folio.The new behaviour is hidden behind the new
LARGE_ANON_FOLIOKconfig, which defaults to disabled for now; The long term aim is for this to defaut to enabled, but there are some risks around internal fragmentation that need to be better understood first.When enabled, the folio order is determined as such: For a vma, process or system that has explicitly disabled THP, we continue to allocate order-0. THP is most likely disabled to avoid any possible internal fragmentation so we honour that request.
Otherwise, the return value of
arch_wants_pte_order()is used. For vmas that have not explicitly opted-in to use transparent hugepages (e.g. where thp=madvise and the vma does not haveMADV_HUGEPAGE), thenarch_wants_pte_order()is limited to 64K (orPAGE_SIZE, whichever is bigger). This allows for a performance boost without requiring any explicit opt-in from the workload while limitting internal fragmentation.If the preferred order can't be used (e.g. because the folio would breach the bounds of the vma, or because ptes in the region are already mapped) then we fall back to a suitable lower order; first
PAGE_ALLOC_COSTLY_ORDER, then order-0....
Quoted 28 lines
(quoted diff) +28
-0+#define ANON_FOLIO_MAX_ORDER_UNHINTED \ + (ilog2(max_t(unsigned long, SZ_64K, PAGE_SIZE)) - PAGE_SHIFT) + +static int anon_folio_order(struct vm_area_struct *vma) +{ + int order; + + /* + * If THP is explicitly disabled for either the vma, the process or the + * system, then this is very likely intended to limit internal + * fragmentation; in this case, don't attempt to allocate a large + * anonymous folio. + * + * Else, if the vma is eligible for thp, allocate a large folio of the + * size preferred by the arch. Or if the arch requested a very small + * size or didn't request a size, then use PAGE_ALLOC_COSTLY_ORDER, + * which still meets the arch's requirements but means we still take + * advantage of SW optimizations (e.g. fewer page faults). + * + * Finally if thp is enabled but the vma isn't eligible, take the + * arch-preferred size and limit it to ANON_FOLIO_MAX_ORDER_UNHINTED. + * This ensures workloads that have not explicitly opted-in take benefit + * while capping the potential for internal fragmentation. + */ + + if ((vma->vm_flags & VM_NOHUGEPAGE) || + test_bit(MMF_DISABLE_THP, &vma->vm_mm->flags) || + !hugepage_flags_enabled())Review
Quoted 10 lines
(quoted diff) +10
-0+ order = 0; + else { + order = max(arch_wants_pte_order(), PAGE_ALLOC_COSTLY_ORDER); + + if (!hugepage_vma_check(vma, vma->vm_flags, false, true, true)) + order = min(order, ANON_FOLIO_MAX_ORDER_UNHINTED); + } + + return order; +}Hi All,
I'm writing up the conclusions that we arrived at during discussion in the THP meeting yesterday, regarding linkage with exiting THP ABIs. It would be great if I can get explicit "agree" or disagree + rationale from at least David, Yu and Kirill.
In summary; I think we are converging on the approach that is already coded, but I'd like confirmation.
The THP situation today
------------------------ At system level: THP can be set to "never", "madvise" or "always"
- At process level: THP can be "never" or "defer to system setting"
- At VMA level: no-hint,MADV_HUGEPAGE,MADV_NOHUGEPAGEThat gives us this table to describe how a page fault is handled, according to process state (columns) and vma flags (rows):
| never | madvise | always ----------------|-----------|-----------|----------- no hint | S | S | THP>S MADV_HUGEPAGE | S | THP>S | THP>S MADV_NOHUGEPAGE | S | S | SLegend:
S allocate single page (PTE-mapped) LAF allocate lage anon folio (PTE-mapped) THP allocate THP-sized folio (PMD-mapped)Quoted 1 line
fallback (usually because vma size/alignment insufficient for folio)
Principles for Large Anon Folios (LAF)
--------------------------------------David tells us there are use cases today (e.g. qemu live migration) which use
MADV_NOHUGEPAGEto mean "don't fill any PTEs that are not explicitly faulted" and these use cases will break (i.e. functionally incorrect) if this request is not honoured.Quoted 8 lines
Quoted 3 lines
I don't remember David saying this. I think he was referring to UFFD, not
MADV_NOHUGEPAGE, when discussing what we need to absolutely respect.My understanding was that
MADV_NOHUGEPAGEwas being applied to regions *before* UFFD was being registered, and the app relied onMADV_NOHUGEPAGEto not back any unfaulted pages. It's not completely clear to me how not honouringMADV_NOHUGEPAGEwould break things though. David?Sorry, I'm still lagging behind on some threads.
Imagine the following for VM postcopy live migration:
(1) Set
MADV_NOHUGEPAGEon guest memory and discard all memory (e.g.,MADV_DONTNEED), to start with a clean slate.
(2) Migrates some pages during precopy from the source and stores them into guest memory on the destination. Some of the memory locations will have pages populated.
(3) At some point, decide to enable postcopy: enable userfaultfd on guest memory.
(4) Discard *selected* pages again that have been dirtied in the meantime on the source. These are pages that have been migrated previously.
(5) Start running the VM on the destination.
(6) Anything that's not populated will trigger userfaultfd missing faults. Then, you can request them from the source and place them.Quoted 8 lines
Quoted 4 lines
Assume you would populate more than required during 2), you can end up not getting userfaultfd faults during 4) and corrupt your guest state. It works if during (2) you migrated all guest memory, or if during 4) you zap everything that still needs migr
I see what you mean now. Thanks.
Yes, in this case we have to interpret
MADV_NOHUGEPAGEas nothing >4KB.Note that it's still even unclear to me why we want to *not* call these things THP. It would certainly make everything less confusing if we call them THP, but with additional attributes.
I think that is one of the first things we should figure out because it also indirectly tells us what all these toggles mean and how/if we should redefine them (and if they even apply).
Currently THP == PMD size
In 2016, Hugh already envisioned PUD/PGD THP (see
49920d28781d("mm:
make transparent hugepage size public")) when he explicitly exposed "hpage_pmd_size". Not "hpage_size".For hugetlb on arm64 we already support various sizes that are < PMD size and *not* call them differently. It's a huge(tlb) page. Sometimes we refer to them as cont-PTE hugetlb pages.
So, nowadays we do have "PMD-sized THP", someday we might have "PUD-sized THP". Can't we come up with a name to describe sub-PMD THP?
Is it really of value if we invent a new term for them? Yes, I was not enjoying "Flexible THP".
Once we figured that out, we should figure out if
MADV_HUGEPAGEmeant "only PMD-sized THP" or anything else?Also, we can then figure out if
MADV_NOHUGEPAGEmeant "only PMD-sized THP" or anything else?The simplest approach to me would be "they imply any THP, and once we need more tunables we might add some", similar to what Kirill also raised.
Quoted 8 lines
Quoted 2 lines
Again, it's all unclear to me at this point and I'm happy to hear opinions, because I really don't know.
I agree these points require more discussion. But I don't think we need to conclude them now, unless they cause correctness issues like ignoring
MADV_NOHUGEPAGEwould. My concern is that if we decide to go with "they imply any THP" and *expose this to userspace now*, we might regret later.Quoted 3 lines
If we don't think they are THP, probably
MADV_NOHUGEPAGEshould not apply and we should be ready to find other ways to deal with the mess we eventually create. If we want to go down that path, sure.If they are THP, to me there is not really a question if
MADV_NOHUGEPAGEapplies to them or not. Unless we want to build a confusing piece of software ;)I think it is good to call them THP, since they are transparent huge (>order-0) pages. But the concern is that before we have a reasonable management policy for order>0 && order<9 THPs, mixing them with existing order-9 THP might give user unexpected performance outcome. Unless we are sure they will always performance improvement, we might repeat the old THP path, namely users begin to disable THP by default to avoid unexpected performance hiccup. That is the reason Yu wants to separate LAF from THP at the moment.
Quoted 5 lines
Maybe call it THP (experimental) for now and merge it to THP when we have a stable policy. For knobs, we might add "any-order" to the existing "never", "madvise" and another interface to specify max hinted order (enforcing <9) for "any-order". Later, we can allow users to specify any max hinted order, including 9. Just an idea.
I suspect that all the config knobs (enable/disable mixing mode, define "any-order" or "specific-order") will be exist long term. Because there are always new workloads need be tuned against these configs.
Regards
Yin, Fengwei
Quoted 5 lines
--
Best Regards,
Yan, Zi
Re: [PATCH v4 2/5] mm: LARGE_ANON_FOLIO for improved performance
Quoted 162 lines · Fengwei
Quoted 159 lines · Zi Yan
Quoted 157 lines · David Hildenbrand
Quoted 156 lines · Yu Zhao
Quoted 155 lines · David Hildenbrand
Quoted 153 lines · Yu Zhao
Quoted 152 lines · David Hildenbrand
Quoted 123 lines · Ryan Roberts
Quoted 122 lines · Yu Zhao
Quoted 120 lines · Ryan Roberts
+ Kirill
Quoted 30 lines · Ryan Roberts
Introduce
LARGE_ANON_FOLIOfeature, which allows anonymous memory to be allocated in large folios of a determined order. All pages of the large folio are pte-mapped during the same page fault, significantly reducing the number of page faults. The number of per-page operations (e.g. ref counting, rmap management lru list management) are also significantly reduced since those ops now become per-folio.The new behaviour is hidden behind the new
LARGE_ANON_FOLIOKconfig, which defaults to disabled for now; The long term aim is for this to defaut to enabled, but there are some risks around internal fragmentation that need to be better understood first.When enabled, the folio order is determined as such: For a vma, process or system that has explicitly disabled THP, we continue to allocate order-0. THP is most likely disabled to avoid any possible internal fragmentation so we honour that request.
Otherwise, the return value of
arch_wants_pte_order()is used. For vmas that have not explicitly opted-in to use transparent hugepages (e.g. where thp=madvise and the vma does not haveMADV_HUGEPAGE), thenarch_wants_pte_order()is limited to 64K (orPAGE_SIZE, whichever is bigger). This allows for a performance boost without requiring any explicit opt-in from the workload while limitting internal fragmentation.If the preferred order can't be used (e.g. because the folio would breach the bounds of the vma, or because ptes in the region are already mapped) then we fall back to a suitable lower order; first
PAGE_ALLOC_COSTLY_ORDER, then order-0....
Quoted 28 lines
(quoted diff) +28
-0+#define ANON_FOLIO_MAX_ORDER_UNHINTED \ + (ilog2(max_t(unsigned long, SZ_64K, PAGE_SIZE)) - PAGE_SHIFT) + +static int anon_folio_order(struct vm_area_struct *vma) +{ + int order; + + /* + * If THP is explicitly disabled for either the vma, the process or the + * system, then this is very likely intended to limit internal + * fragmentation; in this case, don't attempt to allocate a large + * anonymous folio. + * + * Else, if the vma is eligible for thp, allocate a large folio of the + * size preferred by the arch. Or if the arch requested a very small + * size or didn't request a size, then use PAGE_ALLOC_COSTLY_ORDER, + * which still meets the arch's requirements but means we still take + * advantage of SW optimizations (e.g. fewer page faults). + * + * Finally if thp is enabled but the vma isn't eligible, take the + * arch-preferred size and limit it to ANON_FOLIO_MAX_ORDER_UNHINTED. + * This ensures workloads that have not explicitly opted-in take benefit + * while capping the potential for internal fragmentation. + */ + + if ((vma->vm_flags & VM_NOHUGEPAGE) || + test_bit(MMF_DISABLE_THP, &vma->vm_mm->flags) || + !hugepage_flags_enabled())Review
Quoted 10 lines
(quoted diff) +10
-0+ order = 0; + else { + order = max(arch_wants_pte_order(), PAGE_ALLOC_COSTLY_ORDER); + + if (!hugepage_vma_check(vma, vma->vm_flags, false, true, true)) + order = min(order, ANON_FOLIO_MAX_ORDER_UNHINTED); + } + + return order; +}Hi All,
I'm writing up the conclusions that we arrived at during discussion in the THP meeting yesterday, regarding linkage with exiting THP ABIs. It would be great if I can get explicit "agree" or disagree + rationale from at least David, Yu and Kirill.
In summary; I think we are converging on the approach that is already coded, but I'd like confirmation.
The THP situation today
------------------------ At system level: THP can be set to "never", "madvise" or "always"
- At process level: THP can be "never" or "defer to system setting"
- At VMA level: no-hint,MADV_HUGEPAGE,MADV_NOHUGEPAGEThat gives us this table to describe how a page fault is handled, according to process state (columns) and vma flags (rows):
| never | madvise | always ----------------|-----------|-----------|----------- no hint | S | S | THP>S MADV_HUGEPAGE | S | THP>S | THP>S MADV_NOHUGEPAGE | S | S | SLegend:
S allocate single page (PTE-mapped) LAF allocate lage anon folio (PTE-mapped) THP allocate THP-sized folio (PMD-mapped)Quoted 1 line
fallback (usually because vma size/alignment insufficient for folio)
Principles for Large Anon Folios (LAF)
--------------------------------------David tells us there are use cases today (e.g. qemu live migration) which use
MADV_NOHUGEPAGEto mean "don't fill any PTEs that are not explicitly faulted" and these use cases will break (i.e. functionally incorrect) if this request is not honoured.Quoted 8 lines
Quoted 3 lines
I don't remember David saying this. I think he was referring to UFFD, not
MADV_NOHUGEPAGE, when discussing what we need to absolutely respect.My understanding was that
MADV_NOHUGEPAGEwas being applied to regions *before* UFFD was being registered, and the app relied onMADV_NOHUGEPAGEto not back any unfaulted pages. It's not completely clear to me how not honouringMADV_NOHUGEPAGEwould break things though. David?Sorry, I'm still lagging behind on some threads.
Imagine the following for VM postcopy live migration:
(1) Set
MADV_NOHUGEPAGEon guest memory and discard all memory (e.g.,MADV_DONTNEED), to start with a clean slate.
(2) Migrates some pages during precopy from the source and stores them into guest memory on the destination. Some of the memory locations will have pages populated.
(3) At some point, decide to enable postcopy: enable userfaultfd on guest memory.
(4) Discard *selected* pages again that have been dirtied in the meantime on the source. These are pages that have been migrated previously.
(5) Start running the VM on the destination.
(6) Anything that's not populated will trigger userfaultfd missing faults. Then, you can request them from the source and place them.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 4 lines
Assume you would populate more than required during 2), you can end up not getting userfaultfd faults during 4) and corrupt your guest state. It works if during (2) you migrated all guest memory, or if during 4) you zap everything that still needs migr
I see what you mean now. Thanks.
Yes, in this case we have to interpret
MADV_NOHUGEPAGEas nothing >4KB.
I'm glad we have agreement on this.
In some threads Yu has been talking about this series in the short term, vs long term roadmap; so to be clear, I interpret this as meaning we must consider that MADV_NOHUGEPAGE means nothing bigger than order-0 both in the context of this series and for the long term - that's behavior that user space depends upon.
I think we should also apply the same logic to system/process THP mode = "never", even if the vma does not have MADV_NOHUGEPAGE. If the user has explicitly set "never" on the system or process, that means "nothing bigger than order-0". Shout if you disagree.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Note that it's still even unclear to me why we want to *not* call these things THP. It would certainly make everything less confusing if we call them THP, but with additional attributes.
I think I've stated in the past that I don't have a strong opinion on what we call them. But I do think you make a convincing argument for calling them after THP. Regardless, I'd rather agree on a name up front, before this initial series goes in - it's always better to be consistent across all the commit messages and comments to make things more grepable.
The only concrete objection I remember hearing to a name with "THP" in the title was that there are stats (meminfo, vmstats, etc) that count THPs and this becomes confusing if those counters now only mean a subset of THPs. But that feels like a small issue in the scheme of things.
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
I think that is one of the first things we should figure out because it also indirectly tells us what all these toggles mean and how/if we should redefine them (and if they even apply).
Currently THP == PMD size
In 2016, Hugh already envisioned PUD/PGD THP (see
49920d28781d("mm:
make transparent hugepage size public")) when he explicitly exposed "hpage_pmd_size". Not "hpage_size".
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
For hugetlb on arm64 we already support various sizes that are < PMD size and *not* call them differently. It's a huge(tlb) page. Sometimes we refer to them as cont-PTE hugetlb pages.
So, nowadays we do have "PMD-sized THP", someday we might have "PUD-sized THP". Can't we come up with a name to describe sub-PMD THP?
I think one subtle difference is that these sub-PMD THPs, likely won't always have a single size.
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Is it really of value if we invent a new term for them? Yes, I was not enjoying "Flexible THP".
How about "variable-order THP"? Or "SW THP" vs "HW THP"?
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Once we figured that out, we should figure out if
MADV_HUGEPAGEmeant "only PMD-sized THP" or anything else?Also, we can then figure out if
MADV_NOHUGEPAGEmeant "only PMD-sized THP" or anything else?
Based on the existing user space expectation that MADV_NOHUGEPAGE means "nothing bigger than order-0" I'm not sure how we could ever decide MADV_NOHUGEPAGE means anything different? This feels set in stone to me.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
The simplest approach to me would be "they imply any THP, and once we need more tunables we might add some", similar to what Kirill also raised.
Agreed.
Quoted 15 lines
Quoted 15 lines
Quoted 14 lines
Quoted 10 lines
Quoted 4 lines
Again, it's all unclear to me at this point and I'm happy to hear opinions, because I really don't know.
I agree these points require more discussion. But I don't think we need to conclude them now, unless they cause correctness issues like ignoring
MADV_NOHUGEPAGEwould. My concern is that if we decide to go with "they imply any THP" and *expose this to userspace now*, we might regret later.If we don't think they are THP, probably
MADV_NOHUGEPAGEshould not apply and we should be ready to find other ways to deal with the mess we eventually create. If we want to go down that path, sure.If they are THP, to me there is not really a question if
MADV_NOHUGEPAGEapplies to them or not. Unless we want to build a confusing piece of software ;)
Quoted 7 lines
Quoted 7 lines
I think it is good to call them THP, since they are transparent huge (>order-0) pages. But the concern is that before we have a reasonable management policy for order>0 && order<9 THPs, mixing them with existing order-9 THP might give user unexpected performance outcome. Unless we are sure they will always performance improvement, we might repeat the old THP path, namely users begin to disable THP by default to avoid unexpected performance hiccup. That is the reason Yu wants to separate LAF from THP at the moment.
(for the purposes of this; LAF="sub-PMD THP", THP="PMD-size THP", we treat them both as forms of THP)...
How about this for a strawman:
When introducing LAF we can either use an opt-in or an opt-out model. The opt-in model would require a new ABI from day 1 (something I think there is concensus that we do not want to do) and would prevent apps from automatically getting benefit. So I don't like that model.
If going with the opt-out model, we already have an opt-out mechanism (thp="never" and MADV_NOHUGEPAGE) that we can piggyback. But that mechanism doesn't give us all the control we would like for benchmarking/characterizing the interactions between LAF/THP for different workloads. Ideally we need a way to enable THP while keeping LAF disabled and enable LAF while keeping THP disabled.
Can we do this with debugfs? I think controls in there can come and go without too much concern about back-compat?
Perhaps 2 controls:
laf_enable=0|1
enable/disable LAF independently of THP
default=1
laf_max_order=N
applies to both LAF and THP
when max_order < PMD-order, THP acts like thp="never" puts a ceiling on folio order allocated by LAF default=PMD-order
This gives:
laf_enable=1, laf_max_order=PMD-order (LAF+THP):
| never | madvise | always ----------------|-----------|-----------|----------- no hint | S | LAF>S | THP>LAF>S MADV_HUGEPAGE | S | THP>LAF>S | THP>LAF>S MADV_NOHUGEPAGE | S | S | S
laf_enable=0, laf_max_order=PMD-order (THP only):
| never | madvise | always ----------------|-----------|-----------|----------- no hint | S | S | THP>S MADV_HUGEPAGE | S | THP>S | THP>S MADV_NOHUGEPAGE | S | S | S
laf_enable=1, laf_max_order=(PMD-order - 1) (LAF only):
| never | madvise | always ----------------|-----------|-----------|----------- no hint | S | LAF>S | LAF>S MADV_HUGEPAGE | S | LAF>S | LAF>S MADV_NOHUGEPAGE | S | S | S
This would allow us to get something into the kernel that would allow people to more broadly characterize different workloads under THP, LAF, THP+LAF, which would give us a better understanding of if/how we want to design ABIs for the long term.
Quoted 19 lines
Quoted 6 lines
Maybe call it THP (experimental) for now and merge it to THP when we have a stable policy. For knobs, we might add "any-order" to the existing "never", "madvise" and another interface to specify max hinted order (enforcing <9) for "any-order". Later, we can allow users to specify any max hinted order, including 9. Just an idea.
I suspect that all the config knobs (enable/disable mixing mode, define "any-order" or "specific-order") will be exist long term. Because there are always new workloads need be tuned against these configs.
Regards
Yin, FengweiQuoted 5 lines
--
Best Regards,
Yan, Zi
Re: [PATCH v4 2/5] mm: LARGE_ANON_FOLIO for improved performance
Quoted 219 lines · Ryan Roberts
Quoted 162 lines · Fengwei
Quoted 159 lines · Zi Yan
Quoted 157 lines · David Hildenbrand
Quoted 156 lines · Yu Zhao
Quoted 155 lines · David Hildenbrand
Quoted 153 lines · Yu Zhao
Quoted 152 lines · David Hildenbrand
Quoted 123 lines · Ryan Roberts
Quoted 122 lines · Yu Zhao
Quoted 120 lines · Ryan Roberts
+ Kirill
Quoted 30 lines · Ryan Roberts
Introduce
LARGE_ANON_FOLIOfeature, which allows anonymous memory to be allocated in large folios of a determined order. All pages of the large folio are pte-mapped during the same page fault, significantly reducing the number of page faults. The number of per-page operations (e.g. ref counting, rmap management lru list management) are also significantly reduced since those ops now become per-folio.The new behaviour is hidden behind the new
LARGE_ANON_FOLIOKconfig, which defaults to disabled for now; The long term aim is for this to defaut to enabled, but there are some risks around internal fragmentation that need to be better understood first.When enabled, the folio order is determined as such: For a vma, process or system that has explicitly disabled THP, we continue to allocate order-0. THP is most likely disabled to avoid any possible internal fragmentation so we honour that request.
Otherwise, the return value of
arch_wants_pte_order()is used. For vmas that have not explicitly opted-in to use transparent hugepages (e.g. where thp=madvise and the vma does not haveMADV_HUGEPAGE), thenarch_wants_pte_order()is limited to 64K (orPAGE_SIZE, whichever is bigger). This allows for a performance boost without requiring any explicit opt-in from the workload while limitting internal fragmentation.If the preferred order can't be used (e.g. because the folio would breach the bounds of the vma, or because ptes in the region are already mapped) then we fall back to a suitable lower order; first
PAGE_ALLOC_COSTLY_ORDER, then order-0....
Quoted 28 lines
(quoted diff) +28
-0+#define ANON_FOLIO_MAX_ORDER_UNHINTED \ + (ilog2(max_t(unsigned long, SZ_64K, PAGE_SIZE)) - PAGE_SHIFT) + +static int anon_folio_order(struct vm_area_struct *vma) +{ + int order; + + /* + * If THP is explicitly disabled for either the vma, the process or the + * system, then this is very likely intended to limit internal + * fragmentation; in this case, don't attempt to allocate a large + * anonymous folio. + * + * Else, if the vma is eligible for thp, allocate a large folio of the + * size preferred by the arch. Or if the arch requested a very small + * size or didn't request a size, then use PAGE_ALLOC_COSTLY_ORDER, + * which still meets the arch's requirements but means we still take + * advantage of SW optimizations (e.g. fewer page faults). + * + * Finally if thp is enabled but the vma isn't eligible, take the + * arch-preferred size and limit it to ANON_FOLIO_MAX_ORDER_UNHINTED. + * This ensures workloads that have not explicitly opted-in take benefit + * while capping the potential for internal fragmentation. + */ + + if ((vma->vm_flags & VM_NOHUGEPAGE) || + test_bit(MMF_DISABLE_THP, &vma->vm_mm->flags) || + !hugepage_flags_enabled())Review
Quoted 10 lines
(quoted diff) +10
-0+ order = 0; + else { + order = max(arch_wants_pte_order(), PAGE_ALLOC_COSTLY_ORDER); + + if (!hugepage_vma_check(vma, vma->vm_flags, false, true, true)) + order = min(order, ANON_FOLIO_MAX_ORDER_UNHINTED); + } + + return order; +}Hi All,
I'm writing up the conclusions that we arrived at during discussion in the THP meeting yesterday, regarding linkage with exiting THP ABIs. It would be great if I can get explicit "agree" or disagree + rationale from at least David, Yu and Kirill.
In summary; I think we are converging on the approach that is already coded, but I'd like confirmation.
The THP situation today
------------------------ At system level: THP can be set to "never", "madvise" or "always"
- At process level: THP can be "never" or "defer to system setting"
- At VMA level: no-hint,MADV_HUGEPAGE,MADV_NOHUGEPAGEThat gives us this table to describe how a page fault is handled, according to process state (columns) and vma flags (rows):
| never | madvise | always ----------------|-----------|-----------|----------- no hint | S | S | THP>S MADV_HUGEPAGE | S | THP>S | THP>S MADV_NOHUGEPAGE | S | S | SLegend:
S allocate single page (PTE-mapped) LAF allocate lage anon folio (PTE-mapped) THP allocate THP-sized folio (PMD-mapped)Quoted 1 line
fallback (usually because vma size/alignment insufficient for folio)
Principles for Large Anon Folios (LAF)
--------------------------------------David tells us there are use cases today (e.g. qemu live migration) which use
MADV_NOHUGEPAGEto mean "don't fill any PTEs that are not explicitly faulted" and these use cases will break (i.e. functionally incorrect) if this request is not honoured.Quoted 8 lines
Quoted 3 lines
I don't remember David saying this. I think he was referring to UFFD, not
MADV_NOHUGEPAGE, when discussing what we need to absolutely respect.My understanding was that
MADV_NOHUGEPAGEwas being applied to regions *before* UFFD was being registered, and the app relied onMADV_NOHUGEPAGEto not back any unfaulted pages. It's not completely clear to me how not honouringMADV_NOHUGEPAGEwould break things though. David?Sorry, I'm still lagging behind on some threads.
Imagine the following for VM postcopy live migration:
(1) Set
MADV_NOHUGEPAGEon guest memory and discard all memory (e.g.,MADV_DONTNEED), to start with a clean slate.
(2) Migrates some pages during precopy from the source and stores them into guest memory on the destination. Some of the memory locations will have pages populated.
(3) At some point, decide to enable postcopy: enable userfaultfd on guest memory.
(4) Discard *selected* pages again that have been dirtied in the meantime on the source. These are pages that have been migrated previously.
(5) Start running the VM on the destination.
(6) Anything that's not populated will trigger userfaultfd missing faults. Then, you can request them from the source and place them.Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 4 lines
Assume you would populate more than required during 2), you can end up not getting userfaultfd faults during 4) and corrupt your guest state. It works if during (2) you migrated all guest memory, or if during 4) you zap everything that still needs migr
I see what you mean now. Thanks.
Yes, in this case we have to interpret
MADV_NOHUGEPAGEas nothing >4KB.I'm glad we have agreement on this.
In some threads Yu has been talking about this series in the short term, vs long term roadmap; so to be clear, I interpret this as meaning we must consider that
MADV_NOHUGEPAGEmeans nothing bigger than order-0 both in the context of this series and for the long term - that's behavior that user space depends upon.I think we should also apply the same logic to system/process THP mode = "never", even if the vma does not have
MADV_NOHUGEPAGE. If the user has explicitly set "never" on the system or process, that means "nothing bigger than order-0". Shout if you disagree.Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Note that it's still even unclear to me why we want to *not* call these things THP. It would certainly make everything less confusing if we call them THP, but with additional attributes.
I think I've stated in the past that I don't have a strong opinion on what we call them. But I do think you make a convincing argument for calling them after THP. Regardless, I'd rather agree on a name up front, before this initial series goes in - it's always better to be consistent across all the commit messages and comments to make things more grepable.
The only concrete objection I remember hearing to a name with "THP" in the title was that there are stats (meminfo, vmstats, etc) that count THPs and this becomes confusing if those counters now only mean a subset of THPs. But that feels like a small issue in the scheme of things.
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
I think that is one of the first things we should figure out because it also indirectly tells us what all these toggles mean and how/if we should redefine them (and if they even apply).
Currently THP == PMD size
In 2016, Hugh already envisioned PUD/PGD THP (see
49920d28781d("mm:
make transparent hugepage size public")) when he explicitly exposed "hpage_pmd_size". Not "hpage_size".Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
For hugetlb on arm64 we already support various sizes that are < PMD size and *not* call them differently. It's a huge(tlb) page. Sometimes we refer to them as cont-PTE hugetlb pages.
So, nowadays we do have "PMD-sized THP", someday we might have "PUD-sized THP". Can't we come up with a name to describe sub-PMD THP?
Quoted 8 lines
I think one subtle difference is that these sub-PMD THPs, likely won't always have a single size.
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Is it really of value if we invent a new term for them? Yes, I was not enjoying "Flexible THP".
How about "variable-order THP"? Or "SW THP" vs "HW THP"?
variable-order THP sounds good to me.
One question I have is that although Ryan is only working on sub-PMD THPs, do we want to plan for sub-PUD THPs now? Like are sub-PUD THPs variable-order THPs? And leave TODOs and comments like "variable-order THPs can be bigger than PMD and smaller than PUD in the future"? Maybe sub-PUD THPs are still too far to consider it for now. Just think out loud.
Quoted 53 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Once we figured that out, we should figure out if
MADV_HUGEPAGEmeant "only PMD-sized THP" or anything else?Also, we can then figure out if
MADV_NOHUGEPAGEmeant "only PMD-sized THP" or anything else?Based on the existing user space expectation that
MADV_NOHUGEPAGEmeans "nothing bigger than order-0" I'm not sure how we could ever decideMADV_NOHUGEPAGEmeans anything different? This feels set in stone to me.Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
The simplest approach to me would be "they imply any THP, and once we need more tunables we might add some", similar to what Kirill also raised.
Agreed.
Quoted 15 lines
Quoted 15 lines
Quoted 14 lines
Quoted 10 lines
Quoted 4 lines
Again, it's all unclear to me at this point and I'm happy to hear opinions, because I really don't know.
I agree these points require more discussion. But I don't think we need to conclude them now, unless they cause correctness issues like ignoring
MADV_NOHUGEPAGEwould. My concern is that if we decide to go with "they imply any THP" and *expose this to userspace now*, we might regret later.If we don't think they are THP, probably
MADV_NOHUGEPAGEshould not apply and we should be ready to find other ways to deal with the mess we eventually create. If we want to go down that path, sure.If they are THP, to me there is not really a question if
MADV_NOHUGEPAGEapplies to them or not. Unless we want to build a confusing piece of software ;)Quoted 7 lines
Quoted 7 lines
I think it is good to call them THP, since they are transparent huge (>order-0) pages. But the concern is that before we have a reasonable management policy for order>0 && order<9 THPs, mixing them with existing order-9 THP might give user unexpected performance outcome. Unless we are sure they will always performance improvement, we might repeat the old THP path, namely users begin to disable THP by default to avoid unexpected performance hiccup. That is the reason Yu wants to separate LAF from THP at the moment.
(for the purposes of this; LAF="sub-PMD THP", THP="PMD-size THP", we treat them both as forms of THP)...
How about this for a strawman:
When introducing LAF we can either use an opt-in or an opt-out model. The opt-in model would require a new ABI from day 1 (something I think there is concensus that we do not want to do) and would prevent apps from automatically getting benefit. So I don't like that model.
Quoted 8 lines
If going with the opt-out model, we already have an opt-out mechanism (thp="never" and
MADV_NOHUGEPAGE) that we can piggyback. But that mechanism doesn't give us all the control we would like for benchmarking/characterizing the interactions between LAF/THP for different workloads. Ideally we need a way to enable THP while keeping LAF disabled and enable LAF while keeping THP disabled.Can we do this with debugfs? I think controls in there can come and go without too much concern about back-compat?
Is debugfs always available on all distros? For system without debugfs, user is going to lose control of LAF. IMHO, the two knobs below can live in /sys/kernel/mm/transparent_hugepage/ and could be in sync with "enabled" once we think LAF is well studied and goes along well with existing PMD THPs, namely when setting "always", "madvise", or "never" to "enabled", "laf_enabled" is set to the same value.
Quoted 7 lines
Perhaps 2 controls:
laf_enable=0|1
enable/disable LAF independently of THP
default=1
Quoted 5 lines
laf_max_order=N
applies to both LAF and THP
whenmax_order< PMD-order, THP acts like thp="never" puts a ceiling on folio order allocated by LAF default=PMD-order
I think it is better to keep it independent of PMD THP. Just make laf_max_order can never be bigger than PMD-order. Later, when we understand the performance impact of mixing LAF with PMD THP, we can lift this limit to allow laf_max_order to be any possible page order.
Quoted 31 lines
This gives:
laf_enable=1, laf_max_order=PMD-order (LAF+THP):
| never | madvise | always ----------------|-----------|-----------|----------- no hint | S | LAF>S | THP>LAF>S MADV_HUGEPAGE | S | THP>LAF>S | THP>LAF>S MADV_NOHUGEPAGE | S | S | Slaf_enable=0, laf_max_order=PMD-order (THP only):
| never | madvise | always ----------------|-----------|-----------|----------- no hint | S | S | THP>S MADV_HUGEPAGE | S | THP>S | THP>S MADV_NOHUGEPAGE | S | S | Slaf_enable=1, laf_max_order=(PMD-order - 1) (LAF only):
| never | madvise | always ----------------|-----------|-----------|----------- no hint | S | LAF>S | LAF>S MADV_HUGEPAGE | S | LAF>S | LAF>S MADV_NOHUGEPAGE | S | S | S
Quoted 25 lines
This would allow us to get something into the kernel that would allow people to more broadly characterize different workloads under THP, LAF, THP+LAF, which would give us a better understanding of if/how we want to design ABIs for the long term.
Quoted 19 lines
Quoted 6 lines
Maybe call it THP (experimental) for now and merge it to THP when we have a stable policy. For knobs, we might add "any-order" to the existing "never", "madvise" and another interface to specify max hinted order (enforcing <9) for "any-order". Later, we can allow users to specify any max hinted order, including 9. Just an idea.
I suspect that all the config knobs (enable/disable mixing mode, define "any-order" or "specific-order") will be exist long term. Because there are always new workloads need be tuned against these configs.
Regards
Yin, FengweiQuoted 5 lines
--
Best Regards,
Yan, Zi
--
Best Regards,
Yan, Zi
Re: [PATCH v4 2/5] mm: LARGE_ANON_FOLIO for improved performance
Quoted 230 lines · Zi Yan
Quoted 219 lines · Ryan Roberts
Quoted 162 lines · Fengwei
Quoted 159 lines · Zi Yan
Quoted 157 lines · David Hildenbrand
Quoted 156 lines · Yu Zhao
Quoted 155 lines · David Hildenbrand
Quoted 153 lines · Yu Zhao
Quoted 152 lines · David Hildenbrand
Quoted 123 lines · Ryan Roberts
Quoted 122 lines · Yu Zhao
Quoted 120 lines · Ryan Roberts
+ Kirill
Quoted 30 lines · Ryan Roberts
Introduce LARGE_ANON_FOLIO feature, which allows anonymous memory to be
allocated in large folios of a determined order. All pages of the large
folio are pte-mapped during the same page fault, significantly reducing
the number of page faults. The number of per-page operations (e.g. ref
counting, rmap management lru list management) are also significantly
reduced since those ops now become per-folio.The new behaviour is hidden behind the new LARGE_ANON_FOLIO Kconfig,
which defaults to disabled for now; The long term aim is for this to
defaut to enabled, but there are some risks around internal
fragmentation that need to be better understood first.When enabled, the folio order is determined as such: For a vma, process
or system that has explicitly disabled THP, we continue to allocate
order-0. THP is most likely disabled to avoid any possible internal
fragmentation so we honour that request.Otherwise, the return value of arch_wants_pte_order() is used. For vmas
that have not explicitly opted-in to use transparent hugepages (e.g.
where thp=madvise and the vma does not have MADV_HUGEPAGE), then
arch_wants_pte_order() is limited to 64K (or PAGE_SIZE, whichever is
bigger). This allows for a performance boost without requiring any
explicit opt-in from the workload while limitting internal
fragmentation.If the preferred order can't be used (e.g. because the folio would
breach the bounds of the vma, or because ptes in the region are already
mapped) then we fall back to a suitable lower order; first
PAGE_ALLOC_COSTLY_ORDER, then order-0....
Quoted 38 lines
+#define ANON_FOLIO_MAX_ORDER_UNHINTED \
+ (ilog2(max_t(unsigned long, SZ_64K, PAGE_SIZE)) - PAGE_SHIFT)
+
+static int anon_folio_order(struct vm_area_struct *vma)
+{
+ int order;
+
+ /*
+ * If THP is explicitly disabled for either the vma, the process or the
+ * system, then this is very likely intended to limit internal
+ * fragmentation; in this case, don't attempt to allocate a large
+ * anonymous folio.
+ *
+ * Else, if the vma is eligible for thp, allocate a large folio of the
+ * size preferred by the arch. Or if the arch requested a very small
+ * size or didn't request a size, then use PAGE_ALLOC_COSTLY_ORDER,
+ * which still meets the arch's requirements but means we still take
+ * advantage of SW optimizations (e.g. fewer page faults).
+ *
+ * Finally if thp is enabled but the vma isn't eligible, take the
+ * arch-preferred size and limit it to ANON_FOLIO_MAX_ORDER_UNHINTED.
+ * This ensures workloads that have not explicitly opted-in take benefit
+ * while capping the potential for internal fragmentation.
+ */
+
+ if ((vma->vm_flags & VM_NOHUGEPAGE) ||
+ test_bit(MMF_DISABLE_THP, &vma->vm_mm->flags) ||
+ !hugepage_flags_enabled())
+ order = 0;
+ else {
+ order = max(arch_wants_pte_order(), PAGE_ALLOC_COSTLY_ORDER);
+
+ if (!hugepage_vma_check(vma, vma->vm_flags, false, true, true))
+ order = min(order, ANON_FOLIO_MAX_ORDER_UNHINTED);
+ }
+
+ return order;
+}Hi All,
I'm writing up the conclusions that we arrived at during discussion in the THP meeting yesterday, regarding linkage with exiting THP ABIs. It would be great if I can get explicit "agree" or disagree + rationale from at least David, Yu and Kirill.
In summary; I think we are converging on the approach that is already coded, but I'd like confirmation.
The THP situation today
------------------------ At system level: THP can be set to "never", "madvise" or "always"
- At process level: THP can be "never" or "defer to system setting"
- At VMA level: no-hint,MADV_HUGEPAGE,MADV_NOHUGEPAGEThat gives us this table to describe how a page fault is handled, according to process state (columns) and vma flags (rows):
| never | madvise | always ----------------|-----------|-----------|----------- no hint | S | S | THP>S MADV_HUGEPAGE | S | THP>S | THP>S MADV_NOHUGEPAGE | S | S | SLegend:
S allocate single page (PTE-mapped) LAF allocate lage anon folio (PTE-mapped) THP allocate THP-sized folio (PMD-mapped)Quoted 1 line
fallback (usually because vma size/alignment insufficient for folio)
Principles for Large Anon Folios (LAF)
--------------------------------------David tells us there are use cases today (e.g. qemu live migration) which use
MADV_NOHUGEPAGEto mean "don't fill any PTEs that are not explicitly faulted" and these use cases will break (i.e. functionally incorrect) if this request is not honoured.Quoted 8 lines
Quoted 3 lines
I don't remember David saying this. I think he was referring to UFFD, not
MADV_NOHUGEPAGE, when discussing what we need to absolutely respect.My understanding was that
MADV_NOHUGEPAGEwas being applied to regions *before* UFFD was being registered, and the app relied onMADV_NOHUGEPAGEto not back any unfaulted pages. It's not completely clear to me how not honouringMADV_NOHUGEPAGEwould break things though. David?Sorry, I'm still lagging behind on some threads.
Imagine the following for VM postcopy live migration:
(1) Set
MADV_NOHUGEPAGEon guest memory and discard all memory (e.g.,MADV_DONTNEED), to start with a clean slate.
(2) Migrates some pages during precopy from the source and stores them into guest memory on the destination. Some of the memory locations will have pages populated.
(3) At some point, decide to enable postcopy: enable userfaultfd on guest memory.
(4) Discard *selected* pages again that have been dirtied in the meantime on the source. These are pages that have been migrated previously.
(5) Start running the VM on the destination.
(6) Anything that's not populated will trigger userfaultfd missing faults. Then, you can request them from the source and place them.Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 4 lines
Assume you would populate more than required during 2), you can end up not getting userfaultfd faults during 4) and corrupt your guest state. It works if during (2) you migrated all guest memory, or if during 4) you zap everything that still needs migr
I see what you mean now. Thanks.
Yes, in this case we have to interpret
MADV_NOHUGEPAGEas nothing >4KB.I'm glad we have agreement on this.
In some threads Yu has been talking about this series in the short term, vs long term roadmap; so to be clear, I interpret this as meaning we must consider that
MADV_NOHUGEPAGEmeans nothing bigger than order-0 both in the context of this series and for the long term - that's behavior that user space depends upon.I think we should also apply the same logic to system/process THP mode = "never", even if the vma does not have
MADV_NOHUGEPAGE. If the user has explicitly set "never" on the system or process, that means "nothing bigger than order-0". Shout if you disagree.Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Note that it's still even unclear to me why we want to *not* call these things THP. It would certainly make everything less confusing if we call them THP, but with additional attributes.
I think I've stated in the past that I don't have a strong opinion on what we call them. But I do think you make a convincing argument for calling them after THP. Regardless, I'd rather agree on a name up front, before this initial series goes in - it's always better to be consistent across all the commit messages and comments to make things more grepable.
The only concrete objection I remember hearing to a name with "THP" in the title was that there are stats (meminfo, vmstats, etc) that count THPs and this becomes confusing if those counters now only mean a subset of THPs. But that feels like a small issue in the scheme of things.
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
I think that is one of the first things we should figure out because it also indirectly tells us what all these toggles mean and how/if we should redefine them (and if they even apply).
Currently THP == PMD size
In 2016, Hugh already envisioned PUD/PGD THP (see
49920d28781d("mm:
make transparent hugepage size public")) when he explicitly exposed "hpage_pmd_size". Not "hpage_size".Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
For hugetlb on arm64 we already support various sizes that are < PMD size and *not* call them differently. It's a huge(tlb) page. Sometimes we refer to them as cont-PTE hugetlb pages.
So, nowadays we do have "PMD-sized THP", someday we might have "PUD-sized THP". Can't we come up with a name to describe sub-PMD THP?
Quoted 8 lines
I think one subtle difference is that these sub-PMD THPs, likely won't always have a single size.
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Is it really of value if we invent a new term for them? Yes, I was not enjoying "Flexible THP".
How about "variable-order THP"? Or "SW THP" vs "HW THP"?
Quoted 7 lines
variable-order THP sounds good to me.
One question I have is that although Ryan is only working on sub-PMD THPs, do we want to plan for sub-PUD THPs now? Like are sub-PUD THPs variable-order THPs? And leave TODOs and comments like "variable-order THPs can be bigger than PMD and smaller than PUD in the future"? Maybe sub-PUD THPs are still too far to consider it for now. Just think out loud.
I'm not personally planning to do any work here. Such a thing would need similar but separate implementation IMHO, since you would be working at the PMD level not the PTE level.
Then there is the question of what the benefit would be. My working assumption would be that you will not be getting any further HW benefits until its big enough for a PUD. And the SW costs of allocating such a large contig block would very likely outweigh the benefits of a few less page faults; you're surely better off allocating multiple PMD-sized THPs?
Quoted 64 lines
Quoted 55 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Once we figured that out, we should figure out if
MADV_HUGEPAGEmeant "only PMD-sized THP" or anything else?Also, we can then figure out if
MADV_NOHUGEPAGEmeant "only PMD-sized THP" or anything else?Based on the existing user space expectation that
MADV_NOHUGEPAGEmeans "nothing bigger than order-0" I'm not sure how we could ever decideMADV_NOHUGEPAGEmeans anything different? This feels set in stone to me.Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
The simplest approach to me would be "they imply any THP, and once we need more tunables we might add some", similar to what Kirill also raised.
Agreed.
Quoted 15 lines
Quoted 15 lines
Quoted 14 lines
Quoted 10 lines
Quoted 4 lines
Again, it's all unclear to me at this point and I'm happy to hear opinions, because I really don't know.
I agree these points require more discussion. But I don't think we need to conclude them now, unless they cause correctness issues like ignoring
MADV_NOHUGEPAGEwould. My concern is that if we decide to go with "they imply any THP" and *expose this to userspace now*, we might regret later.If we don't think they are THP, probably
MADV_NOHUGEPAGEshould not apply and we should be ready to find other ways to deal with the mess we eventually create. If we want to go down that path, sure.If they are THP, to me there is not really a question if
MADV_NOHUGEPAGEapplies to them or not. Unless we want to build a confusing piece of software ;)Quoted 7 lines
Quoted 7 lines
I think it is good to call them THP, since they are transparent huge (>order-0) pages. But the concern is that before we have a reasonable management policy for order>0 && order<9 THPs, mixing them with existing order-9 THP might give user unexpected performance outcome. Unless we are sure they will always performance improvement, we might repeat the old THP path, namely users begin to disable THP by default to avoid unexpected performance hiccup. That is the reason Yu wants to separate LAF from THP at the moment.
(for the purposes of this; LAF="sub-PMD THP", THP="PMD-size THP", we treat them both as forms of THP)...
How about this for a strawman:
When introducing LAF we can either use an opt-in or an opt-out model. The opt-in model would require a new ABI from day 1 (something I think there is concensus that we do not want to do) and would prevent apps from automatically getting benefit. So I don't like that model.
Quoted 8 lines
If going with the opt-out model, we already have an opt-out mechanism (thp="never" and
MADV_NOHUGEPAGE) that we can piggyback. But that mechanism doesn't give us all the control we would like for benchmarking/characterizing the interactions between LAF/THP for different workloads. Ideally we need a way to enable THP while keeping LAF disabled and enable LAF while keeping THP disabled.Can we do this with debugfs? I think controls in there can come and go without too much concern about back-compat?
Quoted 6 lines
Is debugfs always available on all distros? For system without debugfs, user is going to lose control of LAF. IMHO, the two knobs below can live in
/sys/kernel/mm/transparent_hugepage/and could be in sync with "enabled" once we think LAF is well studied and goes along well with existing PMD THPs, namely when setting "always", "madvise", or "never" to "enabled", "laf_enabled" is set to the same value.
I really don't want to add any sysfs knobs until we properly understand what we need/want. I couldn't tell you what the availability of debugfs is like across all distros though; certainly its available on Ubuntu and userdebug builds of Android.
Yu suggested another policy [1], which would allow us to disable THP while keeping LAF enabled (but not the other way around) without having to add any knobs, so perhaps that's the way to go. It does assume its safe to use LAF when thp=never though. (Copying here for completeness):
| prctl/fw | sysfs | sysfs | sysfs
| disable | never | madvise | always
----------------|-----------|-----------|-----------|-----------
no hint | S | LAF>S | LAF>S | THP>LAF>S
MADV_HUGEPAGE | S | LAF>S | THP>LAF>S | THP>LAF>S
MADV_NOHUGEPAGE | S | S | S | SWhere "prctl/fw disable" trumps the sysfs setting.
[1] https://lore.kernel.org/linux-mm/[email protected]/
Quoted 80 lines
Quoted 8 lines
Perhaps 2 controls:
laf_enable=0|1
enable/disable LAF independently of THP
default=1Quoted 5 lines
laf_max_order=N
applies to both LAF and THP
whenmax_order< PMD-order, THP acts like thp="never" puts a ceiling on folio order allocated by LAF default=PMD-orderI think it is better to keep it independent of PMD THP. Just make
laf_max_ordercan never be bigger than PMD-order. Later, when we understand the performance impact of mixing LAF with PMD THP, we can lift this limit to allowlaf_max_orderto be any possible page order.Quoted 31 lines
This gives:
laf_enable=1, laf_max_order=PMD-order (LAF+THP):
| never | madvise | always ----------------|-----------|-----------|----------- no hint | S | LAF>S | THP>LAF>S MADV_HUGEPAGE | S | THP>LAF>S | THP>LAF>S MADV_NOHUGEPAGE | S | S | Slaf_enable=0, laf_max_order=PMD-order (THP only):
| never | madvise | always ----------------|-----------|-----------|----------- no hint | S | S | THP>S MADV_HUGEPAGE | S | THP>S | THP>S MADV_NOHUGEPAGE | S | S | Slaf_enable=1, laf_max_order=(PMD-order - 1) (LAF only):
| never | madvise | always ----------------|-----------|-----------|----------- no hint | S | LAF>S | LAF>S MADV_HUGEPAGE | S | LAF>S | LAF>S MADV_NOHUGEPAGE | S | S | SQuoted 25 lines
This would allow us to get something into the kernel that would allow people to more broadly characterize different workloads under THP, LAF, THP+LAF, which would give us a better understanding of if/how we want to design ABIs for the long term.
Quoted 19 lines
Quoted 6 lines
Maybe call it THP (experimental) for now and merge it to THP when we have a stable policy. For knobs, we might add "any-order" to the existing "never", "madvise" and another interface to specify max hinted order (enforcing <9) for "any-order". Later, we can allow users to specify any max hinted order, including 9. Just an idea.
I suspect that all the config knobs (enable/disable mixing mode, define "any-order" or "specific-order") will be exist long term. Because there are always new workloads need be tuned against these configs.
Regards
Yin, FengweiQuoted 5 lines
--
Best Regards,
Yan, Zi--
Best Regards,
Yan, Zi
Re: [PATCH v4 2/5] mm: LARGE_ANON_FOLIO for improved performance
In reply to Ryan RobertsQuoted 162 lines · Ryan Roberts
Introduce
LARGE_ANON_FOLIOfeature, which allows anonymous memory to be allocated in large folios of a determined order. All pages of the large folio are pte-mapped during the same page fault, significantly reducing the number of page faults. The number of per-page operations (e.g. ref counting, rmap management lru list management) are also significantly reduced since those ops now become per-folio.The new behaviour is hidden behind the new
LARGE_ANON_FOLIOKconfig, which defaults to disabled for now; The long term aim is for this to defaut to enabled, but there are some risks around internal fragmentation that need to be better understood first.When enabled, the folio order is determined as such: For a vma, process or system that has explicitly disabled THP, we continue to allocate order-0. THP is most likely disabled to avoid any possible internal fragmentation so we honour that request.
Otherwise, the return value of
arch_wants_pte_order()is used. For vmas that have not explicitly opted-in to use transparent hugepages (e.g. where thp=madvise and the vma does not haveMADV_HUGEPAGE), thenarch_wants_pte_order()is limited to 64K (orPAGE_SIZE, whichever is bigger). This allows for a performance boost without requiring any explicit opt-in from the workload while limitting internal fragmentation.If the preferred order can't be used (e.g. because the folio would breach the bounds of the vma, or because ptes in the region are already mapped) then we fall back to a suitable lower order; first
PAGE_ALLOC_COSTLY_ORDER, then order-0.
arch_wants_pte_order()can be overridden by the architecture if desired. Some architectures (e.g. arm64) can coalsece TLB entries if a contiguous set of ptes map physically contigious, naturally aligned memory, so this mechanism allows the architecture to optimize as required.Here we add the default implementation of
arch_wants_pte_order(), used when the architecture does not define it, which returns -1, implying that the HW has no preference. In this case, mm will choose it's own default order.Signed-off-by Ryan Roberts <[email protected]>--- include/linux/pgtable.h | 13 ++++ mm/Kconfig | 10 +++ mm/memory.c | 166 ++++++++++++++++++++++++++++++++++++---- 3 files changed, 172 insertions(+), 17 deletions(-)include/linux/pgtable.h +13
-0@@ -313,6 +313,19 @@ static inline bool arch_has_hw_pte_young(void) }#endif +#ifndef arch_wants_pte_order +/* +* Returns preferred folio order for pte-mapped memory. Must be in range [0, +* PMD_SHIFT-PAGE_SHIFT) and must not be order-1 since THP requires large folios +* to be at least order-2. Negative value implies that the HW has no preference +* and mm will choose it's own default order. +*/ +static inline int arch_wants_pte_order (void ) +{ +return -1 ; +} +#endif +#ifndef __HAVE_ARCH_PTEP_GET_AND_CLEAR static inline pte_t ptep_get_and_clear(struct mm_struct *mm, unsigned long address,mm/Kconfig +10
-0@@ -1238,4 +1238,14 @@ config LOCK_MM_AND_FIND_VMA source "mm/damon/Kconfig" +config LARGE_ANON_FOLIO + bool "Allocate large folios for anonymous memory" + depends on TRANSPARENT_HUGEPAGE + default n + help + Use large (bigger than order-0) folios to back anonymous memory where + possible, even for pte-mapped memory. This reduces the number of page + faults, as well as other per-page overheads to improve performance for + many workloads. + endmenumm/memory.c +63
-0@@ -4050,6 +4050,127 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)return ret; } +static bool vmf_pte_range_changed (struct vm_fault *vmf,int nr_pages) +{ +int i; + +if (nr_pages ==1 ) +return vmf_pte_changed (vmf); + +for (i =0 ; i < nr_pages; i++) { +if (!pte_none (ptep_get_lockless (vmf->pte + i))) +return true; + } + +return false; +} + +#ifdef CONFIG_LARGE_ANON_FOLIO +#define ANON_FOLIO_MAX_ORDER_UNHINTED \ + (ilog2(max_t(unsigned long, SZ_64K, PAGE_SIZE)) - PAGE_SHIFT) + +static int anon_folio_order (struct vm_area_struct *vma) +{ +int order; + +/* +* If THP is explicitly disabled for either the vma, the process or the +* system, then this is very likely intended to limit internal +* fragmentation; in this case, don't attempt to allocate a large +* anonymous folio. +* +* Else, if the vma is eligible for thp, allocate a large folio of the +* size preferred by the arch. Or if the arch requested a very small +* size or didn't request a size, then use PAGE_ALLOC_COSTLY_ORDER, +* which still meets the arch's requirements but means we still take +* advantage of SW optimizations (e.g. fewer page faults). +* +* Finally if thp is enabled but the vma isn't eligible, take the +* arch-preferred size and limit it to ANON_FOLIO_MAX_ORDER_UNHINTED. +* This ensures workloads that have not explicitly opted-in take benefit +* while capping the potential for internal fragmentation. +*/ + +if ((vma->vm_flags &VM_NOHUGEPAGE ) || +test_bit (MMF_DISABLE_THP , &vma->vm_mm->flags) || + !hugepage_flags_enabled ()) + order =0 ; +else { + order =max (arch_wants_pte_order (),PAGE_ALLOC_COSTLY_ORDER ); + +if (!hugepage_vma_check (vma, vma->vm_flags, false, true, true)) + order =min (order,ANON_FOLIO_MAX_ORDER_UNHINTED ); + } + +return order; +} + +static int alloc_anon_folio (struct vm_fault *vmf,struct folio **folio) +{ +int i; +gfp_t gfp; +pte_t *pte; +unsigned long addr; +struct vm_area_struct *vma = vmf->vma; +int prefer =anon_folio_order (vma);
mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 4125
Quoted 10 lines
mm/memory.c +10
-0+int orders[] = { + prefer, + prefer >PAGE_ALLOC_COSTLY_ORDER ?PAGE_ALLOC_COSTLY_ORDER :0 , +0 , + }; + + *folio =NULL ; + +if (vmf_orig_pte_uffd_wp (vmf)) + goto fallback;
Per the discussion, we need to check hugepage_vma_check() for correctness of VM LM. I'd just check it here and fall back to order 0 if that helper returns false.
Re: [PATCH v4 2/5] mm: LARGE_ANON_FOLIO for improved performance
Quoted 169 lines · Yu Zhao
Quoted 168 lines · Ryan Roberts
Introduce
LARGE_ANON_FOLIOfeature, which allows anonymous memory to be allocated in large folios of a determined order. All pages of the large folio are pte-mapped during the same page fault, significantly reducing the number of page faults. The number of per-page operations (e.g. ref counting, rmap management lru list management) are also significantly reduced since those ops now become per-folio.The new behaviour is hidden behind the new
LARGE_ANON_FOLIOKconfig, which defaults to disabled for now; The long term aim is for this to defaut to enabled, but there are some risks around internal fragmentation that need to be better understood first.When enabled, the folio order is determined as such: For a vma, process or system that has explicitly disabled THP, we continue to allocate order-0. THP is most likely disabled to avoid any possible internal fragmentation so we honour that request.
Otherwise, the return value of
arch_wants_pte_order()is used. For vmas that have not explicitly opted-in to use transparent hugepages (e.g. where thp=madvise and the vma does not haveMADV_HUGEPAGE), thenarch_wants_pte_order()is limited to 64K (orPAGE_SIZE, whichever is bigger). This allows for a performance boost without requiring any explicit opt-in from the workload while limitting internal fragmentation.If the preferred order can't be used (e.g. because the folio would breach the bounds of the vma, or because ptes in the region are already mapped) then we fall back to a suitable lower order; first
PAGE_ALLOC_COSTLY_ORDER, then order-0.
arch_wants_pte_order()can be overridden by the architecture if desired. Some architectures (e.g. arm64) can coalsece TLB entries if a contiguous set of ptes map physically contigious, naturally aligned memory, so this mechanism allows the architecture to optimize as required.Here we add the default implementation of
arch_wants_pte_order(), used when the architecture does not define it, which returns -1, implying that the HW has no preference. In this case, mm will choose it's own default order.Signed-off-by Ryan Roberts <[email protected]>--- include/linux/pgtable.h | 13 ++++ mm/Kconfig | 10 +++ mm/memory.c | 166 ++++++++++++++++++++++++++++++++++++---- 3 files changed, 172 insertions(+), 17 deletions(-)include/linux/pgtable.h +13
-0@@ -313,6 +313,19 @@ static inline bool arch_has_hw_pte_young(void) }#endif +#ifndef arch_wants_pte_order +/* +* Returns preferred folio order for pte-mapped memory. Must be in range [0, +* PMD_SHIFT-PAGE_SHIFT) and must not be order-1 since THP requires large folios +* to be at least order-2. Negative value implies that the HW has no preference +* and mm will choose it's own default order. +*/ +static inline int arch_wants_pte_order (void ) +{ +return -1 ; +} +#endif +#ifndef __HAVE_ARCH_PTEP_GET_AND_CLEAR static inline pte_t ptep_get_and_clear(struct mm_struct *mm, unsigned long address,mm/Kconfig +10
-0@@ -1238,4 +1238,14 @@ config LOCK_MM_AND_FIND_VMA source "mm/damon/Kconfig" +config LARGE_ANON_FOLIO + bool "Allocate large folios for anonymous memory" + depends on TRANSPARENT_HUGEPAGE + default n + help + Use large (bigger than order-0) folios to back anonymous memory where + possible, even for pte-mapped memory. This reduces the number of page + faults, as well as other per-page overheads to improve performance for + many workloads. + endmenumm/memory.c +69
-0@@ -4050,6 +4050,127 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)return ret; } +static bool vmf_pte_range_changed (struct vm_fault *vmf,int nr_pages) +{ +int i; + +if (nr_pages ==1 ) +return vmf_pte_changed (vmf); + +for (i =0 ; i < nr_pages; i++) { +if (!pte_none (ptep_get_lockless (vmf->pte + i))) +return true; + } + +return false; +} + +#ifdef CONFIG_LARGE_ANON_FOLIO +#define ANON_FOLIO_MAX_ORDER_UNHINTED \ + (ilog2(max_t(unsigned long, SZ_64K, PAGE_SIZE)) - PAGE_SHIFT) + +static int anon_folio_order (struct vm_area_struct *vma) +{ +int order; + +/* +* If THP is explicitly disabled for either the vma, the process or the +* system, then this is very likely intended to limit internal +* fragmentation; in this case, don't attempt to allocate a large +* anonymous folio. +* +* Else, if the vma is eligible for thp, allocate a large folio of the +* size preferred by the arch. Or if the arch requested a very small +* size or didn't request a size, then use PAGE_ALLOC_COSTLY_ORDER, +* which still meets the arch's requirements but means we still take +* advantage of SW optimizations (e.g. fewer page faults). +* +* Finally if thp is enabled but the vma isn't eligible, take the +* arch-preferred size and limit it to ANON_FOLIO_MAX_ORDER_UNHINTED. +* This ensures workloads that have not explicitly opted-in take benefit +* while capping the potential for internal fragmentation. +*/ + +if ((vma->vm_flags &VM_NOHUGEPAGE ) || +test_bit (MMF_DISABLE_THP , &vma->vm_mm->flags) || + !hugepage_flags_enabled ()) + order =0 ; +else { + order =max (arch_wants_pte_order (),PAGE_ALLOC_COSTLY_ORDER ); + +if (!hugepage_vma_check (vma, vma->vm_flags, false, true, true)) + order =min (order,ANON_FOLIO_MAX_ORDER_UNHINTED ); + } + +return order; +} + +static int alloc_anon_folio (struct vm_fault *vmf,struct folio **folio) +{ +int i; +gfp_t gfp; +pte_t *pte; +unsigned long addr; +struct vm_area_struct *vma = vmf->vma; +int prefer =anon_folio_order (vma); +int orders[] = { + prefer, + prefer >PAGE_ALLOC_COSTLY_ORDER ?PAGE_ALLOC_COSTLY_ORDER :0 , +0 , + }; +
Quoted 8 lines
mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 4125
Quoted 4 lines
mm/memory.c +4
-0+ *folio =NULL ; + +if (vmf_orig_pte_uffd_wp (vmf)) + goto fallback;Per the discussion, we need to check
hugepage_vma_check()for correctness of VM LM. I'd just check it here and fall back to order 0 if that helper returns false.
I'm not sure if either you haven't noticed the logic in anon_folio_order() above, or whether you are making this suggestion because you disagree with the subtle difference in my logic?
My logic is deliberately not calling hugepage_vma_check() because that would return false for the thp=madvise,mmap=unhinted case, whereas the policy I'm implementing wants to apply LAF in that case.
My intended policy:
| never | madvise | always ----------------|-----------|-----------|----------- no hint | S | LAF>S | THP>LAF>S MADV_HUGEPAGE | S | THP>LAF>S | THP>LAF>S MADV_NOHUGEPAGE | S | S | S
What your suggestion would give:
| never | madvise | always ----------------|-----------|-----------|----------- no hint | S | S | THP>LAF>S MADV_HUGEPAGE | S | THP>LAF>S | THP>LAF>S MADV_NOHUGEPAGE | S | S | S
Thanks,
Ryan
Re: [PATCH v4 2/5] mm: LARGE_ANON_FOLIO for improved performance
Quoted 180 lines · Ryan Roberts
Quoted 169 lines · Yu Zhao
Quoted 168 lines · Ryan Roberts
Introduce
LARGE_ANON_FOLIOfeature, which allows anonymous memory to be allocated in large folios of a determined order. All pages of the large folio are pte-mapped during the same page fault, significantly reducing the number of page faults. The number of per-page operations (e.g. ref counting, rmap management lru list management) are also significantly reduced since those ops now become per-folio.The new behaviour is hidden behind the new
LARGE_ANON_FOLIOKconfig, which defaults to disabled for now; The long term aim is for this to defaut to enabled, but there are some risks around internal fragmentation that need to be better understood first.When enabled, the folio order is determined as such: For a vma, process or system that has explicitly disabled THP, we continue to allocate order-0. THP is most likely disabled to avoid any possible internal fragmentation so we honour that request.
Otherwise, the return value of
arch_wants_pte_order()is used. For vmas that have not explicitly opted-in to use transparent hugepages (e.g. where thp=madvise and the vma does not haveMADV_HUGEPAGE), thenarch_wants_pte_order()is limited to 64K (orPAGE_SIZE, whichever is bigger). This allows for a performance boost without requiring any explicit opt-in from the workload while limitting internal fragmentation.If the preferred order can't be used (e.g. because the folio would breach the bounds of the vma, or because ptes in the region are already mapped) then we fall back to a suitable lower order; first
PAGE_ALLOC_COSTLY_ORDER, then order-0.
arch_wants_pte_order()can be overridden by the architecture if desired. Some architectures (e.g. arm64) can coalsece TLB entries if a contiguous set of ptes map physically contigious, naturally aligned memory, so this mechanism allows the architecture to optimize as required.Here we add the default implementation of
arch_wants_pte_order(), used when the architecture does not define it, which returns -1, implying that the HW has no preference. In this case, mm will choose it's own default order.Signed-off-by Ryan Roberts <[email protected]>--- include/linux/pgtable.h | 13 ++++ mm/Kconfig | 10 +++ mm/memory.c | 166 ++++++++++++++++++++++++++++++++++++---- 3 files changed, 172 insertions(+), 17 deletions(-)include/linux/pgtable.h +13
-0@@ -313,6 +313,19 @@ static inline bool arch_has_hw_pte_young(void) }#endif +#ifndef arch_wants_pte_order +/* +* Returns preferred folio order for pte-mapped memory. Must be in range [0, +* PMD_SHIFT-PAGE_SHIFT) and must not be order-1 since THP requires large folios +* to be at least order-2. Negative value implies that the HW has no preference +* and mm will choose it's own default order. +*/ +static inline int arch_wants_pte_order (void ) +{ +return -1 ; +} +#endif +#ifndef __HAVE_ARCH_PTEP_GET_AND_CLEAR static inline pte_t ptep_get_and_clear(struct mm_struct *mm, unsigned long address,mm/Kconfig +10
-0@@ -1238,4 +1238,14 @@ config LOCK_MM_AND_FIND_VMA source "mm/damon/Kconfig" +config LARGE_ANON_FOLIO + bool "Allocate large folios for anonymous memory" + depends on TRANSPARENT_HUGEPAGE + default n + help + Use large (bigger than order-0) folios to back anonymous memory where + possible, even for pte-mapped memory. This reduces the number of page + faults, as well as other per-page overheads to improve performance for + many workloads. + endmenumm/memory.c +69
-0@@ -4050,6 +4050,127 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)return ret; } +static bool vmf_pte_range_changed (struct vm_fault *vmf,int nr_pages) +{ +int i; + +if (nr_pages ==1 ) +return vmf_pte_changed (vmf); + +for (i =0 ; i < nr_pages; i++) { +if (!pte_none (ptep_get_lockless (vmf->pte + i))) +return true; + } + +return false; +} + +#ifdef CONFIG_LARGE_ANON_FOLIO +#define ANON_FOLIO_MAX_ORDER_UNHINTED \ + (ilog2(max_t(unsigned long, SZ_64K, PAGE_SIZE)) - PAGE_SHIFT) + +static int anon_folio_order (struct vm_area_struct *vma) +{ +int order; + +/* +* If THP is explicitly disabled for either the vma, the process or the +* system, then this is very likely intended to limit internal +* fragmentation; in this case, don't attempt to allocate a large +* anonymous folio. +* +* Else, if the vma is eligible for thp, allocate a large folio of the +* size preferred by the arch. Or if the arch requested a very small +* size or didn't request a size, then use PAGE_ALLOC_COSTLY_ORDER, +* which still meets the arch's requirements but means we still take +* advantage of SW optimizations (e.g. fewer page faults). +* +* Finally if thp is enabled but the vma isn't eligible, take the +* arch-preferred size and limit it to ANON_FOLIO_MAX_ORDER_UNHINTED. +* This ensures workloads that have not explicitly opted-in take benefit +* while capping the potential for internal fragmentation. +*/ + +if ((vma->vm_flags &VM_NOHUGEPAGE ) || +test_bit (MMF_DISABLE_THP , &vma->vm_mm->flags) || + !hugepage_flags_enabled ()) + order =0 ; +else { + order =max (arch_wants_pte_order (),PAGE_ALLOC_COSTLY_ORDER ); + +if (!hugepage_vma_check (vma, vma->vm_flags, false, true, true)) + order =min (order,ANON_FOLIO_MAX_ORDER_UNHINTED ); + } + +return order; +} + +static int alloc_anon_folio (struct vm_fault *vmf,struct folio **folio) +{ +int i; +gfp_t gfp; +pte_t *pte; +unsigned long addr; +struct vm_area_struct *vma = vmf->vma; +int prefer =anon_folio_order (vma); +int orders[] = { + prefer, + prefer >PAGE_ALLOC_COSTLY_ORDER ?PAGE_ALLOC_COSTLY_ORDER :0 , +0 , + }; +Quoted 8 lines
mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 4125
Quoted 4 lines
mm/memory.c +4
-0+ *folio =NULL ; + +if (vmf_orig_pte_uffd_wp (vmf)) + goto fallback;Per the discussion, we need to check
hugepage_vma_check()for correctness of VM LM. I'd just check it here and fall back to order 0 if that helper returns false.
Quoted 3 lines
I'm not sure if either you haven't noticed the logic in
anon_folio_order()above, or whether you are making this suggestion because you disagree with the subtle difference in my logic?
The latter, or more generally the policy you described earlier.
Quoted 14 lines
My logic is deliberately not calling
hugepage_vma_check()because that would return false for the thp=madvise,mmap=unhinted case, whereas the policy I'm implementing wants to apply LAF in that case.My intended policy:
| never | madvise | always ----------------|-----------|-----------|----------- no hint | S | LAF>S | THP>LAF>S MADV_HUGEPAGE | S | THP>LAF>S | THP>LAF>S MADV_NOHUGEPAGE | S | S | S
Quoted 7 lines
What your suggestion would give:
| never | madvise | always ----------------|-----------|-----------|----------- no hint | S | S | THP>LAF>S MADV_HUGEPAGE | S | THP>LAF>S | THP>LAF>S MADV_NOHUGEPAGE | S | S | S
This is not what I'm suggesting.
Let me reiterate [1]:
My impression is we only agreed on one thing: at the current stage, we should respect things we absolutely have to. We didn't agree on what "never" means ("never 2MB" or "never >4KB"), and we didn't touch on how "always" should behave at all.
And [2]:
(Thanks to David, now I agree that) we have to interpret MADV_NOHUGEPAGE as nothing >4KB.
My final take [3]:
I agree these points require more discussion. But I don't think we need to conclude them now, unless they cause correctness issues like ignoring MADV_NOHUGEPAGE would.
But I should have been clear about the parameters to hugepage_vma_check(): enforce_sysfs=false.
[1] https://lore.kernel.org/linux-mm/CAOUHufYQTcOdKU=1mPq-fdLV7a66sHx1=EJpPpMVogciCNKO9A@mail.gmail.com/
[2] https://lore.kernel.org/linux-mm/CAOUHufafd4GNna2GKdSyQdW6CLVh0gxhNgeOc6t+ZOphwgw7tw@mail.gmail.com/
[3] https://lore.kernel.org/linux-mm/CAOUHufYQTcOdKU=1mPq-fdLV7a66sHx1=EJpPpMVogciCNKO9A@mail.gmail.com/
Re: [PATCH v4 2/5] mm: LARGE_ANON_FOLIO for improved performance
Quoted 217 lines · Yu Zhao
Quoted 180 lines · Ryan Roberts
Quoted 169 lines · Yu Zhao
Quoted 168 lines · Ryan Roberts
Introduce
LARGE_ANON_FOLIOfeature, which allows anonymous memory to be allocated in large folios of a determined order. All pages of the large folio are pte-mapped during the same page fault, significantly reducing the number of page faults. The number of per-page operations (e.g. ref counting, rmap management lru list management) are also significantly reduced since those ops now become per-folio.The new behaviour is hidden behind the new
LARGE_ANON_FOLIOKconfig, which defaults to disabled for now; The long term aim is for this to defaut to enabled, but there are some risks around internal fragmentation that need to be better understood first.When enabled, the folio order is determined as such: For a vma, process or system that has explicitly disabled THP, we continue to allocate order-0. THP is most likely disabled to avoid any possible internal fragmentation so we honour that request.
Otherwise, the return value of
arch_wants_pte_order()is used. For vmas that have not explicitly opted-in to use transparent hugepages (e.g. where thp=madvise and the vma does not haveMADV_HUGEPAGE), thenarch_wants_pte_order()is limited to 64K (orPAGE_SIZE, whichever is bigger). This allows for a performance boost without requiring any explicit opt-in from the workload while limitting internal fragmentation.If the preferred order can't be used (e.g. because the folio would breach the bounds of the vma, or because ptes in the region are already mapped) then we fall back to a suitable lower order; first
PAGE_ALLOC_COSTLY_ORDER, then order-0.
arch_wants_pte_order()can be overridden by the architecture if desired. Some architectures (e.g. arm64) can coalsece TLB entries if a contiguous set of ptes map physically contigious, naturally aligned memory, so this mechanism allows the architecture to optimize as required.Here we add the default implementation of
arch_wants_pte_order(), used when the architecture does not define it, which returns -1, implying that the HW has no preference. In this case, mm will choose it's own default order.Signed-off-by Ryan Roberts <[email protected]>--- include/linux/pgtable.h | 13 ++++ mm/Kconfig | 10 +++ mm/memory.c | 166 ++++++++++++++++++++++++++++++++++++---- 3 files changed, 172 insertions(+), 17 deletions(-)include/linux/pgtable.h +13
-0@@ -313,6 +313,19 @@ static inline bool arch_has_hw_pte_young(void) }#endif +#ifndef arch_wants_pte_order +/* +* Returns preferred folio order for pte-mapped memory. Must be in range [0, +* PMD_SHIFT-PAGE_SHIFT) and must not be order-1 since THP requires large folios +* to be at least order-2. Negative value implies that the HW has no preference +* and mm will choose it's own default order. +*/ +static inline int arch_wants_pte_order (void ) +{ +return -1 ; +} +#endif +#ifndef __HAVE_ARCH_PTEP_GET_AND_CLEAR static inline pte_t ptep_get_and_clear(struct mm_struct *mm, unsigned long address,mm/Kconfig +10
-0@@ -1238,4 +1238,14 @@ config LOCK_MM_AND_FIND_VMA source "mm/damon/Kconfig" +config LARGE_ANON_FOLIO + bool "Allocate large folios for anonymous memory" + depends on TRANSPARENT_HUGEPAGE + default n + help + Use large (bigger than order-0) folios to back anonymous memory where + possible, even for pte-mapped memory. This reduces the number of page + faults, as well as other per-page overheads to improve performance for + many workloads. + endmenumm/memory.c +69
-0@@ -4050,6 +4050,127 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)return ret; } +static bool vmf_pte_range_changed (struct vm_fault *vmf,int nr_pages) +{ +int i; + +if (nr_pages ==1 ) +return vmf_pte_changed (vmf); + +for (i =0 ; i < nr_pages; i++) { +if (!pte_none (ptep_get_lockless (vmf->pte + i))) +return true; + } + +return false; +} + +#ifdef CONFIG_LARGE_ANON_FOLIO +#define ANON_FOLIO_MAX_ORDER_UNHINTED \ + (ilog2(max_t(unsigned long, SZ_64K, PAGE_SIZE)) - PAGE_SHIFT) + +static int anon_folio_order (struct vm_area_struct *vma) +{ +int order; + +/* +* If THP is explicitly disabled for either the vma, the process or the +* system, then this is very likely intended to limit internal +* fragmentation; in this case, don't attempt to allocate a large +* anonymous folio. +* +* Else, if the vma is eligible for thp, allocate a large folio of the +* size preferred by the arch. Or if the arch requested a very small +* size or didn't request a size, then use PAGE_ALLOC_COSTLY_ORDER, +* which still meets the arch's requirements but means we still take +* advantage of SW optimizations (e.g. fewer page faults). +* +* Finally if thp is enabled but the vma isn't eligible, take the +* arch-preferred size and limit it to ANON_FOLIO_MAX_ORDER_UNHINTED. +* This ensures workloads that have not explicitly opted-in take benefit +* while capping the potential for internal fragmentation. +*/ + +if ((vma->vm_flags &VM_NOHUGEPAGE ) || +test_bit (MMF_DISABLE_THP , &vma->vm_mm->flags) || + !hugepage_flags_enabled ()) + order =0 ; +else { + order =max (arch_wants_pte_order (),PAGE_ALLOC_COSTLY_ORDER ); + +if (!hugepage_vma_check (vma, vma->vm_flags, false, true, true)) + order =min (order,ANON_FOLIO_MAX_ORDER_UNHINTED ); + } + +return order; +} + +static int alloc_anon_folio (struct vm_fault *vmf,struct folio **folio) +{ +int i; +gfp_t gfp; +pte_t *pte; +unsigned long addr; +struct vm_area_struct *vma = vmf->vma; +int prefer =anon_folio_order (vma); +int orders[] = { + prefer, + prefer >PAGE_ALLOC_COSTLY_ORDER ?PAGE_ALLOC_COSTLY_ORDER :0 , +0 , + }; +Quoted 8 lines
mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 4125
Quoted 4 lines
mm/memory.c +4
-0+ *folio =NULL ; + +if (vmf_orig_pte_uffd_wp (vmf)) + goto fallback;Per the discussion, we need to check
hugepage_vma_check()for correctness of VM LM. I'd just check it here and fall back to order 0 if that helper returns false.Quoted 3 lines
I'm not sure if either you haven't noticed the logic in
anon_folio_order()above, or whether you are making this suggestion because you disagree with the subtle difference in my logic?The latter, or more generally the policy you described earlier.
Quoted 14 lines
My logic is deliberately not calling
hugepage_vma_check()because that would return false for the thp=madvise,mmap=unhinted case, whereas the policy I'm implementing wants to apply LAF in that case.My intended policy:
| never | madvise | always ----------------|-----------|-----------|----------- no hint | S | LAF>S | THP>LAF>S MADV_HUGEPAGE | S | THP>LAF>S | THP>LAF>S MADV_NOHUGEPAGE | S | S | SQuoted 7 lines
What your suggestion would give:
| never | madvise | always ----------------|-----------|-----------|----------- no hint | S | S | THP>LAF>S MADV_HUGEPAGE | S | THP>LAF>S | THP>LAF>S MADV_NOHUGEPAGE | S | S | SThis is not what I'm suggesting.
Let me reiterate [1]:
My impression is we only agreed on one thing: at the current stage, we should respect things we absolutely have to. We didn't agree on what "never" means ("never 2MB" or "never >4KB"), and we didn't touch on how "always" should behave at all.
Quoted 8 lines
And [2]:
(Thanks to David, now I agree that) we have to interpretMADV_NOHUGEPAGEas nothing >4KB.My final take [3]:
I agree these points require more discussion. But I don't think we need to conclude them now, unless they cause correctness issues like ignoringMADV_NOHUGEPAGEwould.
Thanks, I've read all of these comments previously, and appreciate the time you have put into the feedback. I'm not sure I fully agree with your point that we don't need to conclude on a policy now; I certainly don't think we need the whole thing in place on day 1, but I do think that whatever we put in should strive to be a strict subset of where we think we are going. For example, if we put something in with one policy (i.e. "never" only means "never 2MB") then find a problem and have to change that to be more conservative, are we risking perf regressions for any LAF users that started using it on day 1?
Quoted 3 lines
But I should have been clear about the parameters to
hugepage_vma_check(): enforce_sysfs=false.
So hugepage_vma_check(..., smaps=false, in_pf=true, enforce_sysfs=false) would give us:
| prctl/fw | sysfs | sysfs | sysfs
| disable | never | madvise | always
----------------|-----------|-----------|-----------|-----------
no hint | S | LAF>S | LAF>S | THP>LAF>S
MADV_HUGEPAGE | S | LAF>S | THP>LAF>S | THP>LAF>S
MADV_NOHUGEPAGE | S | S | S | SWhere "prctl/fw disable" trumps the sysfs setting.
I can certainly see the benefit of this approach; it gives us a way to enable LAF while disabling THP (thp=never). It doesn't give us a way to enable THP without enabling LAF though (unless you recompile with LAF disabled). Does anyone see a problem with this?
Quoted 4 lines
[1] https://lore.kernel.org/linux-mm/CAOUHufYQTcOdKU=1mPq-fdLV7a66sHx1=EJpPpMVogciCNKO9A@mail.gmail.com/
[2] https://lore.kernel.org/linux-mm/CAOUHufafd4GNna2GKdSyQdW6CLVh0gxhNgeOc6t+ZOphwgw7tw@mail.gmail.com/
[3] https://lore.kernel.org/linux-mm/CAOUHufYQTcOdKU=1mPq-fdLV7a66sHx1=EJpPpMVogciCNKO9A@mail.gmail.com/
Re: [PATCH v4 2/5] mm: LARGE_ANON_FOLIO for improved performance
Quoted 228 lines · Ryan Roberts
Quoted 217 lines · Yu Zhao
Quoted 180 lines · Ryan Roberts
Quoted 169 lines · Yu Zhao
Quoted 168 lines · Ryan Roberts
Introduce
LARGE_ANON_FOLIOfeature, which allows anonymous memory to be allocated in large folios of a determined order. All pages of the large folio are pte-mapped during the same page fault, significantly reducing the number of page faults. The number of per-page operations (e.g. ref counting, rmap management lru list management) are also significantly reduced since those ops now become per-folio.The new behaviour is hidden behind the new
LARGE_ANON_FOLIOKconfig, which defaults to disabled for now; The long term aim is for this to defaut to enabled, but there are some risks around internal fragmentation that need to be better understood first.When enabled, the folio order is determined as such: For a vma, process or system that has explicitly disabled THP, we continue to allocate order-0. THP is most likely disabled to avoid any possible internal fragmentation so we honour that request.
Otherwise, the return value of
arch_wants_pte_order()is used. For vmas that have not explicitly opted-in to use transparent hugepages (e.g. where thp=madvise and the vma does not haveMADV_HUGEPAGE), thenarch_wants_pte_order()is limited to 64K (orPAGE_SIZE, whichever is bigger). This allows for a performance boost without requiring any explicit opt-in from the workload while limitting internal fragmentation.If the preferred order can't be used (e.g. because the folio would breach the bounds of the vma, or because ptes in the region are already mapped) then we fall back to a suitable lower order; first
PAGE_ALLOC_COSTLY_ORDER, then order-0.
arch_wants_pte_order()can be overridden by the architecture if desired. Some architectures (e.g. arm64) can coalsece TLB entries if a contiguous set of ptes map physically contigious, naturally aligned memory, so this mechanism allows the architecture to optimize as required.Here we add the default implementation of
arch_wants_pte_order(), used when the architecture does not define it, which returns -1, implying that the HW has no preference. In this case, mm will choose it's own default order.Signed-off-by Ryan Roberts <[email protected]>--- include/linux/pgtable.h | 13 ++++ mm/Kconfig | 10 +++ mm/memory.c | 166 ++++++++++++++++++++++++++++++++++++---- 3 files changed, 172 insertions(+), 17 deletions(-)include/linux/pgtable.h +13
-0@@ -313,6 +313,19 @@ static inline bool arch_has_hw_pte_young(void) }#endif +#ifndef arch_wants_pte_order +/* +* Returns preferred folio order for pte-mapped memory. Must be in range [0, +* PMD_SHIFT-PAGE_SHIFT) and must not be order-1 since THP requires large folios +* to be at least order-2. Negative value implies that the HW has no preference +* and mm will choose it's own default order. +*/ +static inline int arch_wants_pte_order (void ) +{ +return -1 ; +} +#endif +#ifndef __HAVE_ARCH_PTEP_GET_AND_CLEAR static inline pte_t ptep_get_and_clear(struct mm_struct *mm, unsigned long address,mm/Kconfig +10
-0@@ -1238,4 +1238,14 @@ config LOCK_MM_AND_FIND_VMA source "mm/damon/Kconfig" +config LARGE_ANON_FOLIO + bool "Allocate large folios for anonymous memory" + depends on TRANSPARENT_HUGEPAGE + default n + help + Use large (bigger than order-0) folios to back anonymous memory where + possible, even for pte-mapped memory. This reduces the number of page + faults, as well as other per-page overheads to improve performance for + many workloads. + endmenumm/memory.c +69
-0@@ -4050,6 +4050,127 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)return ret; } +static bool vmf_pte_range_changed (struct vm_fault *vmf,int nr_pages) +{ +int i; + +if (nr_pages ==1 ) +return vmf_pte_changed (vmf); + +for (i =0 ; i < nr_pages; i++) { +if (!pte_none (ptep_get_lockless (vmf->pte + i))) +return true; + } + +return false; +} + +#ifdef CONFIG_LARGE_ANON_FOLIO +#define ANON_FOLIO_MAX_ORDER_UNHINTED \ + (ilog2(max_t(unsigned long, SZ_64K, PAGE_SIZE)) - PAGE_SHIFT) + +static int anon_folio_order (struct vm_area_struct *vma) +{ +int order; + +/* +* If THP is explicitly disabled for either the vma, the process or the +* system, then this is very likely intended to limit internal +* fragmentation; in this case, don't attempt to allocate a large +* anonymous folio. +* +* Else, if the vma is eligible for thp, allocate a large folio of the +* size preferred by the arch. Or if the arch requested a very small +* size or didn't request a size, then use PAGE_ALLOC_COSTLY_ORDER, +* which still meets the arch's requirements but means we still take +* advantage of SW optimizations (e.g. fewer page faults). +* +* Finally if thp is enabled but the vma isn't eligible, take the +* arch-preferred size and limit it to ANON_FOLIO_MAX_ORDER_UNHINTED. +* This ensures workloads that have not explicitly opted-in take benefit +* while capping the potential for internal fragmentation. +*/ + +if ((vma->vm_flags &VM_NOHUGEPAGE ) || +test_bit (MMF_DISABLE_THP , &vma->vm_mm->flags) || + !hugepage_flags_enabled ()) + order =0 ; +else { + order =max (arch_wants_pte_order (),PAGE_ALLOC_COSTLY_ORDER ); + +if (!hugepage_vma_check (vma, vma->vm_flags, false, true, true)) + order =min (order,ANON_FOLIO_MAX_ORDER_UNHINTED ); + } + +return order; +} + +static int alloc_anon_folio (struct vm_fault *vmf,struct folio **folio) +{ +int i; +gfp_t gfp; +pte_t *pte; +unsigned long addr; +struct vm_area_struct *vma = vmf->vma; +int prefer =anon_folio_order (vma); +int orders[] = { + prefer, + prefer >PAGE_ALLOC_COSTLY_ORDER ?PAGE_ALLOC_COSTLY_ORDER :0 , +0 , + }; +Quoted 8 lines
mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 4125
Quoted 4 lines
mm/memory.c +4
-0+ *folio =NULL ; + +if (vmf_orig_pte_uffd_wp (vmf)) + goto fallback;Per the discussion, we need to check
hugepage_vma_check()for correctness of VM LM. I'd just check it here and fall back to order 0 if that helper returns false.Quoted 3 lines
I'm not sure if either you haven't noticed the logic in
anon_folio_order()above, or whether you are making this suggestion because you disagree with the subtle difference in my logic?The latter, or more generally the policy you described earlier.
Quoted 14 lines
My logic is deliberately not calling
hugepage_vma_check()because that would return false for the thp=madvise,mmap=unhinted case, whereas the policy I'm implementing wants to apply LAF in that case.My intended policy:
| never | madvise | always ----------------|-----------|-----------|----------- no hint | S | LAF>S | THP>LAF>S MADV_HUGEPAGE | S | THP>LAF>S | THP>LAF>S MADV_NOHUGEPAGE | S | S | SQuoted 7 lines
What your suggestion would give:
| never | madvise | always ----------------|-----------|-----------|----------- no hint | S | S | THP>LAF>S MADV_HUGEPAGE | S | THP>LAF>S | THP>LAF>S MADV_NOHUGEPAGE | S | S | SThis is not what I'm suggesting.
Let me reiterate [1]:
My impression is we only agreed on one thing: at the current stage, we should respect things we absolutely have to. We didn't agree on what "never" means ("never 2MB" or "never >4KB"), and we didn't touch on how "always" should behave at all.Quoted 8 lines
And [2]:
(Thanks to David, now I agree that) we have to interpretMADV_NOHUGEPAGEas nothing >4KB.My final take [3]:
I agree these points require more discussion. But I don't think we need to conclude them now, unless they cause correctness issues like ignoringMADV_NOHUGEPAGEwould.
Quoted 8 lines
Thanks, I've read all of these comments previously, and appreciate the time you have put into the feedback. I'm not sure I fully agree with your point that we don't need to conclude on a policy now; I certainly don't think we need the whole thing in place on day 1, but I do think that whatever we put in should strive to be a strict subset of where we think we are going. For example, if we put something in with one policy (i.e. "never" only means "never 2MB") then find a problem and have to change that to be more conservative, are we risking perf regressions for any LAF users that started using it on day 1?
It's not that I don't want to -- I just don't think we have enough information before we have a wider deployment [1] and gain a better understanding of real-world scenarios.
Of course we could force a conclusion, a mostly opinion-based one. But it would still involve prolonged discussions and delay this series, or rush into decisions we might regret later.
[1] Our fleets (servers, laptops and phones) support large-scale experiments and I plan to run them on both client and server devices.
Quoted 13 lines
Quoted 2 lines
But I should have been clear about the parameters to
hugepage_vma_check(): enforce_sysfs=false.So
hugepage_vma_check(..., smaps=false, in_pf=true, enforce_sysfs=false)would give us:| prctl/fw | sysfs | sysfs | sysfs | disable | never | madvise | always ----------------|-----------|-----------|-----------|----------- no hint | S | LAF>S | LAF>S | THP>LAF>S MADV_HUGEPAGE | S | LAF>S | THP>LAF>S | THP>LAF>S MADV_NOHUGEPAGE | S | S | S | S
Quoted 6 lines
Where "prctl/fw disable" trumps the sysfs setting.
I can certainly see the benefit of this approach; it gives us a way to enable LAF while disabling THP (thp=never). It doesn't give us a way to enable THP without enabling LAF though (unless you recompile with LAF disabled). Does anyone see a problem with this?
I do myself :)
This is just something temporary to get this series landed. We are hiding behind a Kconfig, not making any ABI changes, and not exposing this policy to userspace (i.e., not updating Documentation/, man pages, etc.)
Meanwhile, we can keep discussing all the open questions in parallel.
Re: [PATCH v4 2/5] mm: LARGE_ANON_FOLIO for improved performance
Quoted 270 lines · Yu Zhao
Quoted 228 lines · Ryan Roberts
Quoted 217 lines · Yu Zhao
Quoted 180 lines · Ryan Roberts
Quoted 169 lines · Yu Zhao
Quoted 168 lines · Ryan Roberts
Introduce
LARGE_ANON_FOLIOfeature, which allows anonymous memory to be allocated in large folios of a determined order. All pages of the large folio are pte-mapped during the same page fault, significantly reducing the number of page faults. The number of per-page operations (e.g. ref counting, rmap management lru list management) are also significantly reduced since those ops now become per-folio.The new behaviour is hidden behind the new
LARGE_ANON_FOLIOKconfig, which defaults to disabled for now; The long term aim is for this to defaut to enabled, but there are some risks around internal fragmentation that need to be better understood first.When enabled, the folio order is determined as such: For a vma, process or system that has explicitly disabled THP, we continue to allocate order-0. THP is most likely disabled to avoid any possible internal fragmentation so we honour that request.
Otherwise, the return value of
arch_wants_pte_order()is used. For vmas that have not explicitly opted-in to use transparent hugepages (e.g. where thp=madvise and the vma does not haveMADV_HUGEPAGE), thenarch_wants_pte_order()is limited to 64K (orPAGE_SIZE, whichever is bigger). This allows for a performance boost without requiring any explicit opt-in from the workload while limitting internal fragmentation.If the preferred order can't be used (e.g. because the folio would breach the bounds of the vma, or because ptes in the region are already mapped) then we fall back to a suitable lower order; first
PAGE_ALLOC_COSTLY_ORDER, then order-0.
arch_wants_pte_order()can be overridden by the architecture if desired. Some architectures (e.g. arm64) can coalsece TLB entries if a contiguous set of ptes map physically contigious, naturally aligned memory, so this mechanism allows the architecture to optimize as required.Here we add the default implementation of
arch_wants_pte_order(), used when the architecture does not define it, which returns -1, implying that the HW has no preference. In this case, mm will choose it's own default order.Signed-off-by Ryan Roberts <[email protected]>--- include/linux/pgtable.h | 13 ++++ mm/Kconfig | 10 +++ mm/memory.c | 166 ++++++++++++++++++++++++++++++++++++---- 3 files changed, 172 insertions(+), 17 deletions(-)include/linux/pgtable.h +13
-0@@ -313,6 +313,19 @@ static inline bool arch_has_hw_pte_young(void) }#endif +#ifndef arch_wants_pte_order +/* +* Returns preferred folio order for pte-mapped memory. Must be in range [0, +* PMD_SHIFT-PAGE_SHIFT) and must not be order-1 since THP requires large folios +* to be at least order-2. Negative value implies that the HW has no preference +* and mm will choose it's own default order. +*/ +static inline int arch_wants_pte_order (void ) +{ +return -1 ; +} +#endif +#ifndef __HAVE_ARCH_PTEP_GET_AND_CLEAR static inline pte_t ptep_get_and_clear(struct mm_struct *mm, unsigned long address,mm/Kconfig +10
-0@@ -1238,4 +1238,14 @@ config LOCK_MM_AND_FIND_VMA source "mm/damon/Kconfig" +config LARGE_ANON_FOLIO + bool "Allocate large folios for anonymous memory" + depends on TRANSPARENT_HUGEPAGE + default n + help + Use large (bigger than order-0) folios to back anonymous memory where + possible, even for pte-mapped memory. This reduces the number of page + faults, as well as other per-page overheads to improve performance for + many workloads. + endmenumm/memory.c +69
-0@@ -4050,6 +4050,127 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)return ret; } +static bool vmf_pte_range_changed (struct vm_fault *vmf,int nr_pages) +{ +int i; + +if (nr_pages ==1 ) +return vmf_pte_changed (vmf); + +for (i =0 ; i < nr_pages; i++) { +if (!pte_none (ptep_get_lockless (vmf->pte + i))) +return true; + } + +return false; +} + +#ifdef CONFIG_LARGE_ANON_FOLIO +#define ANON_FOLIO_MAX_ORDER_UNHINTED \ + (ilog2(max_t(unsigned long, SZ_64K, PAGE_SIZE)) - PAGE_SHIFT) + +static int anon_folio_order (struct vm_area_struct *vma) +{ +int order; + +/* +* If THP is explicitly disabled for either the vma, the process or the +* system, then this is very likely intended to limit internal +* fragmentation; in this case, don't attempt to allocate a large +* anonymous folio. +* +* Else, if the vma is eligible for thp, allocate a large folio of the +* size preferred by the arch. Or if the arch requested a very small +* size or didn't request a size, then use PAGE_ALLOC_COSTLY_ORDER, +* which still meets the arch's requirements but means we still take +* advantage of SW optimizations (e.g. fewer page faults). +* +* Finally if thp is enabled but the vma isn't eligible, take the +* arch-preferred size and limit it to ANON_FOLIO_MAX_ORDER_UNHINTED. +* This ensures workloads that have not explicitly opted-in take benefit +* while capping the potential for internal fragmentation. +*/ + +if ((vma->vm_flags &VM_NOHUGEPAGE ) || +test_bit (MMF_DISABLE_THP , &vma->vm_mm->flags) || + !hugepage_flags_enabled ()) + order =0 ; +else { + order =max (arch_wants_pte_order (),PAGE_ALLOC_COSTLY_ORDER ); + +if (!hugepage_vma_check (vma, vma->vm_flags, false, true, true)) + order =min (order,ANON_FOLIO_MAX_ORDER_UNHINTED ); + } + +return order; +} + +static int alloc_anon_folio (struct vm_fault *vmf,struct folio **folio) +{ +int i; +gfp_t gfp; +pte_t *pte; +unsigned long addr; +struct vm_area_struct *vma = vmf->vma; +int prefer =anon_folio_order (vma); +int orders[] = { + prefer, + prefer >PAGE_ALLOC_COSTLY_ORDER ?PAGE_ALLOC_COSTLY_ORDER :0 , +0 , + }; +Quoted 8 lines
mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 4125
Quoted 4 lines
mm/memory.c +4
-0+ *folio =NULL ; + +if (vmf_orig_pte_uffd_wp (vmf)) + goto fallback;Per the discussion, we need to check
hugepage_vma_check()for correctness of VM LM. I'd just check it here and fall back to order 0 if that helper returns false.Quoted 3 lines
I'm not sure if either you haven't noticed the logic in
anon_folio_order()above, or whether you are making this suggestion because you disagree with the subtle difference in my logic?The latter, or more generally the policy you described earlier.
Quoted 14 lines
My logic is deliberately not calling
hugepage_vma_check()because that would return false for the thp=madvise,mmap=unhinted case, whereas the policy I'm implementing wants to apply LAF in that case.My intended policy:
| never | madvise | always ----------------|-----------|-----------|----------- no hint | S | LAF>S | THP>LAF>S MADV_HUGEPAGE | S | THP>LAF>S | THP>LAF>S MADV_NOHUGEPAGE | S | S | SQuoted 7 lines
What your suggestion would give:
| never | madvise | always ----------------|-----------|-----------|----------- no hint | S | S | THP>LAF>S MADV_HUGEPAGE | S | THP>LAF>S | THP>LAF>S MADV_NOHUGEPAGE | S | S | SThis is not what I'm suggesting.
Let me reiterate [1]:
My impression is we only agreed on one thing: at the current stage, we should respect things we absolutely have to. We didn't agree on what "never" means ("never 2MB" or "never >4KB"), and we didn't touch on how "always" should behave at all.Quoted 8 lines
And [2]:
(Thanks to David, now I agree that) we have to interpretMADV_NOHUGEPAGEas nothing >4KB.My final take [3]:
I agree these points require more discussion. But I don't think we need to conclude them now, unless they cause correctness issues like ignoringMADV_NOHUGEPAGEwould.Quoted 8 lines
Thanks, I've read all of these comments previously, and appreciate the time you have put into the feedback. I'm not sure I fully agree with your point that we don't need to conclude on a policy now; I certainly don't think we need the whole thing in place on day 1, but I do think that whatever we put in should strive to be a strict subset of where we think we are going. For example, if we put something in with one policy (i.e. "never" only means "never 2MB") then find a problem and have to change that to be more conservative, are we risking perf regressions for any LAF users that started using it on day 1?
It's not that I don't want to -- I just don't think we have enough information before we have a wider deployment [1] and gain a better understanding of real-world scenarios.
Of course we could force a conclusion, a mostly opinion-based one. But it would still involve prolonged discussions and delay this series, or rush into decisions we might regret later.
[1] Our fleets (servers, laptops and phones) support large-scale experiments and I plan to run them on both client and server devices.
Quoted 13 lines
Quoted 2 lines
But I should have been clear about the parameters to
hugepage_vma_check(): enforce_sysfs=false.So
hugepage_vma_check(..., smaps=false, in_pf=true, enforce_sysfs=false)would give us:| prctl/fw | sysfs | sysfs | sysfs | disable | never | madvise | always ----------------|-----------|-----------|-----------|----------- no hint | S | LAF>S | LAF>S | THP>LAF>S MADV_HUGEPAGE | S | LAF>S | THP>LAF>S | THP>LAF>S MADV_NOHUGEPAGE | S | S | S | SQuoted 6 lines
Where "prctl/fw disable" trumps the sysfs setting.
I can certainly see the benefit of this approach; it gives us a way to enable LAF while disabling THP (thp=never). It doesn't give us a way to enable THP without enabling LAF though (unless you recompile with LAF disabled). Does anyone see a problem with this?
Quoted 8 lines
I do myself :)
This is just something temporary to get this series landed. We are hiding behind a
Kconfig, not making any ABI changes, and not exposing this policy to userspace (i.e., not updatingDocumentation/, man pages, etc.)Meanwhile, we can keep discussing all the open questions in parallel.
And the stat ABI changes should be discussed before or at the same time. If we came up with a policy but there was *zero* observability of how well that policy works...
Re: [PATCH v4 2/5] mm: LARGE_ANON_FOLIO for improved performance
[...]
Quoted 28 lines
Quoted 28 lines
Quoted 15 lines
Quoted 6 lines
Let me reiterate [1]:
My impression is we only agreed on one thing: at the current stage, we should respect things we absolutely have to. We didn't agree on what "never" means ("never 2MB" or "never >4KB"), and we didn't touch on how "always" should behave at all.Quoted 8 lines
And [2]:
(Thanks to David, now I agree that) we have to interpretMADV_NOHUGEPAGEas nothing >4KB.My final take [3]:
I agree these points require more discussion. But I don't think we need to conclude them now, unless they cause correctness issues like ignoringMADV_NOHUGEPAGEwould.Quoted 8 lines
Thanks, I've read all of these comments previously, and appreciate the time you have put into the feedback. I'm not sure I fully agree with your point that we don't need to conclude on a policy now; I certainly don't think we need the whole thing in place on day 1, but I do think that whatever we put in should strive to be a strict subset of where we think we are going. For example, if we put something in with one policy (i.e. "never" only means "never 2MB") then find a problem and have to change that to be more conservative, are we risking perf regressions for any LAF users that started using it on day 1?
It's not that I don't want to -- I just don't think we have enough information before we have a wider deployment [1] and gain a better understanding of real-world scenarios.
Quoted 6 lines
Quoted 6 lines
Of course we could force a conclusion, a mostly opinion-based one. But it would still involve prolonged discussions and delay this series, or rush into decisions we might regret later.
[1] Our fleets (servers, laptops and phones) support large-scale experiments and I plan to run them on both client and server devices.
This all sounds great and I'm looking forward to seeing results! But I guess I had been assuming that this sort of testing would be preferable to do before we merge; that allows us to get confidence in the approach and reduces the changes of having to change it later. I guess you have policies that prevent you from testing this series at the scale you want until it is merged?
I'm not convinced this testing will help us answer the "what does never mean?" question; if nothing breaks in your testing, it doesn't mean there aren't systems out there that would break - it's hard to prove a negative. I think its mostly embedded systems that use thp=never to reduce memory footprint to the absolute minimum?
Quoted 21 lines
Quoted 21 lines
Quoted 20 lines
Quoted 3 lines
But I should have been clear about the parameters to
hugepage_vma_check(): enforce_sysfs=false.So
hugepage_vma_check(..., smaps=false, in_pf=true, enforce_sysfs=false)would give us:| prctl/fw | sysfs | sysfs | sysfs | disable | never | madvise | always ----------------|-----------|-----------|-----------|----------- no hint | S | LAF>S | LAF>S | THP>LAF>S MADV_HUGEPAGE | S | LAF>S | THP>LAF>S | THP>LAF>S MADV_NOHUGEPAGE | S | S | S | SWhere "prctl/fw disable" trumps the sysfs setting.
I can certainly see the benefit of this approach; it gives us a way to enable LAF while disabling THP (thp=never). It doesn't give us a way to enable THP without enabling LAF though (unless you recompile with LAF disabled). Does anyone see a problem with this?
Quoted 8 lines
Quoted 8 lines
I do myself :)
This is just something temporary to get this series landed. We are hiding behind a
Kconfig, not making any ABI changes, and not exposing this policy to userspace (i.e., not updatingDocumentation/, man pages, etc.)Meanwhile, we can keep discussing all the open questions in parallel.
You're right - don't want to slow down the testing, so I'm going to post a v5 tomorrow with the policy in the table above. We're still waiting for the prerequisites to land before we can kick off testing in anger though.
Quoted 4 lines
And the stat ABI changes should be discussed before or at the same time. If we came up with a policy but there was *zero* observability of how well that policy works...
Yep agreed. I have a series at [1] which I hoped would kickstart that discussion.
[1] https://lore.kernel.org/linux-mm/[email protected]/
Thanks,
Ryan
PATCH v4 3/5 arm64: mm: Override arch_wants_pte_order()
In reply to Ryan RobertsDefine an arch-specific override of arch_wants_pte_order() so that when LARGE_ANON_FOLIO is enabled, large folios will be allocated for anonymous memory with an order that is compatible with arm64's contpte mappings.
--- arch/arm64/include/asm/pgtable.h | 6 ++++++ 1 file changed, 6 insertions(+)
arch/arm64/include/asm/pgtable.h +6 -0
@@ -1106,6 +1106,12 @@ extern pte_t ptep_modify_prot_start(struct vm_area_struct *vma,extern void ptep_modify_prot_commit (struct vm_area_struct *vma,unsigned long addr,pte_t *ptep,pte_t old_pte,pte_t new_pte); + +#define arch_wants_pte_order arch_wants_pte_order +static inline int arch_wants_pte_order (void ) +{ +return CONT_PTE_SHIFT -PAGE_SHIFT ; +}#endif /* !__ASSEMBLY__ */ #endif /* __ASM_PGTABLE_H */
Signature · 1 line
2.25.1
PATCH v4 4/5 selftests/mm/cow: Generalize do_run_with_thp() helper
In reply to Ryan Robertsdo_run_with_thp() prepares THP memory into different states before running tests. We would like to reuse this logic to also test large anon folios. So let's add a size parameter which tells the function what size of memory it should operate on.
Remove references to THP and replace with LARGE, and fix up all existing call sites to pass thpsize as the required size.
No functional change intended here, but a separate commit will add new large anon folio tests that use this new capability.
--- tools/testing/selftests/mm/cow.c | 118 ++++++++++++++++--------------- 1 file changed, 61 insertions(+), 57 deletions(-)
tools/testing/selftests/mm/cow.c +61 -57
@@ -723,25 +723,25 @@ static void run_with_base_page_swap(test_fn fn, const char *desc)do_run_with_base_page (fn, true); } -enum thp_run { -THP_RUN_PMD , -THP_RUN_PMD_SWAPOUT , -THP_RUN_PTE , -THP_RUN_PTE_SWAPOUT , -THP_RUN_SINGLE_PTE , -THP_RUN_SINGLE_PTE_SWAPOUT , -THP_RUN_PARTIAL_MREMAP , -THP_RUN_PARTIAL_SHARED , +enum large_run { +LARGE_RUN_PMD , +LARGE_RUN_PMD_SWAPOUT , +LARGE_RUN_PTE , +LARGE_RUN_PTE_SWAPOUT , +LARGE_RUN_SINGLE_PTE , +LARGE_RUN_SINGLE_PTE_SWAPOUT , +LARGE_RUN_PARTIAL_MREMAP , +LARGE_RUN_PARTIAL_SHARED , }; -static void do_run_with_thp (test_fn fn,enum thp_run thp_run) +static void do_run_with_large (test_fn fn,enum large_run large_run,size_t size) {char *mem, *mmap_mem, *tmp, *mremap_mem =MAP_FAILED ; -size_t size, mmap_size, mremap_size; +size_t mmap_size, mremap_size;int ret; -/* For alignment purposes, we need twice the thp size. */ - mmap_size =2 * thpsize; +/* For alignment purposes, we need twice the requested size. */ + mmap_size =2 * size; mmap_mem =mmap (NULL , mmap_size,PROT_READ |PROT_WRITE ,MAP_PRIVATE |MAP_ANONYMOUS ,-1 ,0 );if (mmap_mem ==MAP_FAILED ) {
@@ -749,36 +749,40 @@ static void do_run_with_thp(test_fn fn, enum thp_run thp_run)return ; } -/* We need a THP-aligned memory area. */ - mem = (char *)(((uintptr_t )mmap_mem + thpsize) & ~(thpsize -1 )); +/* We need to naturally align the memory area. */ + mem = (char *)(((uintptr_t )mmap_mem + size) & ~(size -1 )); - ret =madvise (mem, thpsize,MADV_HUGEPAGE ); + ret =madvise (mem, size,MADV_HUGEPAGE );if (ret) {ksft_test_result_fail ("MADV_HUGEPAGE failed\n" ); goto munmap; }/* -* Try to populate a THP. Touch the first sub-page and test if we get -* another sub-page populated automatically. +* Try to populate a large folio. Touch the first sub-page and test if +* we get the last sub-page populated automatically. */ mem[0 ] =0 ; -if (!pagemap_is_populated (pagemap_fd, mem + pagesize)) { -ksft_test_result_skip ("Did not get a THP populated\n" ); +if (!pagemap_is_populated (pagemap_fd, mem + size - pagesize)) { +ksft_test_result_skip ("Did not get fully populated\n" ); goto munmap; } -memset (mem,0 , thpsize); +memset (mem,0 , size); - size = thpsize; -switch (thp_run) { -case THP_RUN_PMD : -case THP_RUN_PMD_SWAPOUT : +switch (large_run) { +case LARGE_RUN_PMD : +case LARGE_RUN_PMD_SWAPOUT : +if (size != thpsize) { +ksft_test_result_fail ("test bug: can't PMD-map size\n" ); + goto munmap; + }break ; -case THP_RUN_PTE : -case THP_RUN_PTE_SWAPOUT : +case LARGE_RUN_PTE : +case LARGE_RUN_PTE_SWAPOUT :/* -* Trigger PTE-mapping the THP by temporarily mapping a single -* subpage R/O. +* Trigger PTE-mapping the large folio by temporarily mapping a +* single subpage R/O. This is a noop if the large-folio is not +* thpsize (and therefore already PTE-mapped). */ ret =mprotect (mem + pagesize, pagesize,PROT_READ );if (ret) {
@@ -791,25 +795,25 @@ static void do_run_with_thp(test_fn fn, enum thp_run thp_run) goto munmap; }break ; -case THP_RUN_SINGLE_PTE : -case THP_RUN_SINGLE_PTE_SWAPOUT : +case LARGE_RUN_SINGLE_PTE : +case LARGE_RUN_SINGLE_PTE_SWAPOUT :/* -* Discard all but a single subpage of that PTE-mapped THP. What -* remains is a single PTE mapping a single subpage. +* Discard all but a single subpage of that PTE-mapped large +* folio. What remains is a single PTE mapping a single subpage. */ - ret =madvise (mem + pagesize, thpsize - pagesize,MADV_DONTNEED ); + ret =madvise (mem + pagesize, size - pagesize,MADV_DONTNEED );if (ret) {ksft_test_result_fail ("MADV_DONTNEED failed\n" ); goto munmap; } size = pagesize;break ; -case THP_RUN_PARTIAL_MREMAP : +case LARGE_RUN_PARTIAL_MREMAP :/* -* Remap half of the THP. We need some new memory location -* for that. +* Remap half of the lareg folio. We need some new memory +* location for that. */ - mremap_size = thpsize /2 ; + mremap_size = size /2 ; mremap_mem =mmap (NULL , mremap_size,PROT_NONE ,MAP_PRIVATE |MAP_ANONYMOUS ,-1 ,0 );if (mem ==MAP_FAILED ) {
@@ -824,13 +828,13 @@ static void do_run_with_thp(test_fn fn, enum thp_run thp_run) } size = mremap_size;break ; -case THP_RUN_PARTIAL_SHARED : +case LARGE_RUN_PARTIAL_SHARED :/* -* Share the first page of the THP with a child and quit the -* child. This will result in some parts of the THP never -* have been shared. +* Share the first page of the large folio with a child and quit +* the child. This will result in some parts of the large folio +* never have been shared. */ - ret =madvise (mem + pagesize, thpsize - pagesize,MADV_DONTFORK ); + ret =madvise (mem + pagesize, size - pagesize,MADV_DONTFORK );if (ret) {ksft_test_result_fail ("MADV_DONTFORK failed\n" ); goto munmap;
@@ -844,7 +848,7 @@ static void do_run_with_thp(test_fn fn, enum thp_run thp_run) }wait (&ret);/* Allow for sharing all pages again. */ - ret =madvise (mem + pagesize, thpsize - pagesize,MADV_DOFORK ); + ret =madvise (mem + pagesize, size - pagesize,MADV_DOFORK );if (ret) {ksft_test_result_fail ("MADV_DOFORK failed\n" ); goto munmap;
@@ -854,10 +858,10 @@ static void do_run_with_thp(test_fn fn, enum thp_run thp_run)assert (false); } -switch (thp_run) { -case THP_RUN_PMD_SWAPOUT : -case THP_RUN_PTE_SWAPOUT : -case THP_RUN_SINGLE_PTE_SWAPOUT : +switch (large_run) { +case LARGE_RUN_PMD_SWAPOUT : +case LARGE_RUN_PTE_SWAPOUT : +case LARGE_RUN_SINGLE_PTE_SWAPOUT :madvise (mem, size,MADV_PAGEOUT );if (!range_is_swapped (mem, size)) {ksft_test_result_skip ("MADV_PAGEOUT did not work, is swap enabled?\n" );
@@ -878,49 +882,49 @@ static void do_run_with_thp(test_fn fn, enum thp_run thp_run)static void run_with_thp (test_fn fn,const char *desc) {ksft_print_msg ("[RUN] %s ... with THP\n" , desc); -do_run_with_thp (fn,THP_RUN_PMD ); +do_run_with_large (fn,LARGE_RUN_PMD , thpsize); }static void run_with_thp_swap (test_fn fn,const char *desc) {ksft_print_msg ("[RUN] %s ... with swapped-out THP\n" , desc); -do_run_with_thp (fn,THP_RUN_PMD_SWAPOUT ); +do_run_with_large (fn,LARGE_RUN_PMD_SWAPOUT , thpsize); }static void run_with_pte_mapped_thp (test_fn fn,const char *desc) {ksft_print_msg ("[RUN] %s ... with PTE-mapped THP\n" , desc); -do_run_with_thp (fn,THP_RUN_PTE ); +do_run_with_large (fn,LARGE_RUN_PTE , thpsize); }static void run_with_pte_mapped_thp_swap (test_fn fn,const char *desc) {ksft_print_msg ("[RUN] %s ... with swapped-out, PTE-mapped THP\n" , desc); -do_run_with_thp (fn,THP_RUN_PTE_SWAPOUT ); +do_run_with_large (fn,LARGE_RUN_PTE_SWAPOUT , thpsize); }static void run_with_single_pte_of_thp (test_fn fn,const char *desc) {ksft_print_msg ("[RUN] %s ... with single PTE of THP\n" , desc); -do_run_with_thp (fn,THP_RUN_SINGLE_PTE ); +do_run_with_large (fn,LARGE_RUN_SINGLE_PTE , thpsize); }static void run_with_single_pte_of_thp_swap (test_fn fn,const char *desc) {ksft_print_msg ("[RUN] %s ... with single PTE of swapped-out THP\n" , desc); -do_run_with_thp (fn,THP_RUN_SINGLE_PTE_SWAPOUT ); +do_run_with_large (fn,LARGE_RUN_SINGLE_PTE_SWAPOUT , thpsize); }static void run_with_partial_mremap_thp (test_fn fn,const char *desc) {ksft_print_msg ("[RUN] %s ... with partially mremap()'ed THP\n" , desc); -do_run_with_thp (fn,THP_RUN_PARTIAL_MREMAP ); +do_run_with_large (fn,LARGE_RUN_PARTIAL_MREMAP , thpsize); }static void run_with_partial_shared_thp (test_fn fn,const char *desc) {ksft_print_msg ("[RUN] %s ... with partially shared THP\n" , desc); -do_run_with_thp (fn,THP_RUN_PARTIAL_SHARED ); +do_run_with_large (fn,LARGE_RUN_PARTIAL_SHARED , thpsize); }static void run_with_hugetlb (test_fn fn,const char *desc,size_t hugetlbsize)
@@ -1338,7 +1342,7 @@ static void run_anon_thp_test_cases(void)struct test_case const *test_case = &anon_thp_test_cases[i];ksft_print_msg ("[RUN] %s\n" , test_case->desc); -do_run_with_thp (test_case->fn,THP_RUN_PMD ); +do_run_with_large (test_case->fn,LARGE_RUN_PMD , thpsize); } }
Signature · 1 line
2.25.1
PATCH v4 5/5 selftests/mm/cow: Add large anon folio tests
In reply to Ryan RobertsAdd tests similar to the existing THP tests, but which operate on memory backed by large anonymous folios, which are smaller than THP.
This reuses all the existing infrastructure. If the test suite detects that large anonyomous folios are not supported by the kernel, the new tests are skipped.
--- tools/testing/selftests/mm/cow.c | 111 +++++++++++++++++++++++++++++-- 1 file changed, 106 insertions(+), 5 deletions(-)
tools/testing/selftests/mm/cow.c +106 -5
@@ -33,6 +33,7 @@static size_t pagesize;static int pagemap_fd;static size_t thpsize; +static size_t lafsize;static int nr_hugetlbsizes;static size_t hugetlbsizes[10 ];static int gup_fd;
@@ -927,6 +928,42 @@ static void run_with_partial_shared_thp(test_fn fn, const char *desc)do_run_with_large (fn,LARGE_RUN_PARTIAL_SHARED , thpsize); } +static void run_with_laf (test_fn fn,const char *desc) +{ +ksft_print_msg ("[RUN] %s ... with large anon folio\n" , desc); +do_run_with_large (fn,LARGE_RUN_PTE , lafsize); +} + +static void run_with_laf_swap (test_fn fn,const char *desc) +{ +ksft_print_msg ("[RUN] %s ... with swapped-out large anon folio\n" , desc); +do_run_with_large (fn,LARGE_RUN_PTE_SWAPOUT , lafsize); +} + +static void run_with_single_pte_of_laf (test_fn fn,const char *desc) +{ +ksft_print_msg ("[RUN] %s ... with single PTE of large anon folio\n" , desc); +do_run_with_large (fn,LARGE_RUN_SINGLE_PTE , lafsize); +} + +static void run_with_single_pte_of_laf_swap (test_fn fn,const char *desc) +{ +ksft_print_msg ("[RUN] %s ... with single PTE of swapped-out large anon folio\n" , desc); +do_run_with_large (fn,LARGE_RUN_SINGLE_PTE_SWAPOUT , lafsize); +} + +static void run_with_partial_mremap_laf (test_fn fn,const char *desc) +{ +ksft_print_msg ("[RUN] %s ... with partially mremap()'ed large anon folio\n" , desc); +do_run_with_large (fn,LARGE_RUN_PARTIAL_MREMAP , lafsize); +} + +static void run_with_partial_shared_laf (test_fn fn,const char *desc) +{ +ksft_print_msg ("[RUN] %s ... with partially shared large anon folio\n" , desc); +do_run_with_large (fn,LARGE_RUN_PARTIAL_SHARED , lafsize); +} +static void run_with_hugetlb (test_fn fn,const char *desc,size_t hugetlbsize) {int flags =MAP_PRIVATE |MAP_ANONYMOUS |MAP_HUGETLB ;
@@ -1105,6 +1142,14 @@ static void run_anon_test_case(struct test_case const *test_case)run_with_partial_mremap_thp (test_case->fn, test_case->desc);run_with_partial_shared_thp (test_case->fn, test_case->desc); } +if (lafsize) { +run_with_laf (test_case->fn, test_case->desc); +run_with_laf_swap (test_case->fn, test_case->desc); +run_with_single_pte_of_laf (test_case->fn, test_case->desc); +run_with_single_pte_of_laf_swap (test_case->fn, test_case->desc); +run_with_partial_mremap_laf (test_case->fn, test_case->desc); +run_with_partial_shared_laf (test_case->fn, test_case->desc); + }for (i =0 ; i < nr_hugetlbsizes; i++)run_with_hugetlb (test_case->fn, test_case->desc, hugetlbsizes[i]);
@@ -1126,6 +1171,8 @@ static int tests_per_anon_test_case(void)if (thpsize) tests +=8 ; +if (lafsize) + tests +=6 ;return tests; }
@@ -1680,15 +1727,74 @@ static int tests_per_non_anon_test_case(void)return tests; } +static size_t large_anon_folio_size (void ) +{ +/* +* There is no interface to query this. But we know that it must be less +* than thpsize. So we map a thpsize area, aligned to thpsize offset by +* thpsize/2 (to avoid a hugepage being allocated), then touch the first +* page and see how many pages get faulted in. +*/ + +int max_order =__builtin_ctz (thpsize); +size_t mmap_size = thpsize *3 ; +char *mmap_mem =NULL ; +int order =0 ; +char *mem; +size_t offset; +int ret; + +/* For alignment purposes, we need 2.5x the requested size. */ + mmap_mem =mmap (NULL , mmap_size,PROT_READ |PROT_WRITE , +MAP_PRIVATE |MAP_ANONYMOUS ,-1 ,0 ); +if (mmap_mem ==MAP_FAILED ) + goto out; + +/* Align the memory area to thpsize then offset it by thpsize/2. */ + mem = (char *)(((uintptr_t )mmap_mem + thpsize) & ~(thpsize -1 )); + mem += thpsize /2 ; + +/* We might get a bigger large anon folio when MADV_HUGEPAGE is set. */ + ret =madvise (mem, thpsize,MADV_HUGEPAGE ); +if (ret) + goto out; + +/* Probe the memory to see how much is populated. */ + mem[0 ] =0 ; +for (order =0 ; order < max_order; order++) { + offset = (1 << order) * pagesize; +if (!pagemap_is_populated (pagemap_fd, mem + offset)) +break ; + } + +out: +if (mmap_mem) +munmap (mmap_mem, mmap_size); + +if (order ==0 ) +return 0 ; + +return offset; +} +int main (int argc,char **argv) {int err; + gup_fd =open ("/sys/kernel/debug/gup_test" ,O_RDWR ); + pagemap_fd =open ("/proc/self/pagemap" ,O_RDONLY ); +if (pagemap_fd <0 ) +ksft_exit_fail_msg ("opening pagemap failed\n" ); + pagesize =getpagesize (); thpsize =read_pmd_pagesize ();if (thpsize)ksft_print_msg ("[INFO] detected THP size: %zu KiB\n" , thpsize /1024 ); + lafsize =large_anon_folio_size (); +if (lafsize) +ksft_print_msg ("[INFO] detected large anon folio size: %zu KiB\n" , + lafsize /1024 ); nr_hugetlbsizes =detect_hugetlb_page_sizes (hugetlbsizes,ARRAY_SIZE (hugetlbsizes));detect_huge_zeropage ();
@@ -1698,11 +1804,6 @@ int main(int argc, char **argv)ARRAY_SIZE (anon_thp_test_cases) *tests_per_anon_thp_test_case () + ARRAY_SIZE(non_anon_test_cases) * tests_per_non_anon_test_case()); - gup_fd =open ("/sys/kernel/debug/gup_test" ,O_RDWR ); - pagemap_fd =open ("/proc/self/pagemap" ,O_RDONLY ); -if (pagemap_fd <0 ) -ksft_exit_fail_msg ("opening pagemap failed\n" ); -run_anon_test_cases ();run_anon_thp_test_cases ();run_non_anon_test_cases ();
Signature · 1 line
2.25.1