variable-order, large folios for anonymous memory
Subsystems (4)
v2 cover variable-order, large folios for anonymous memory
Hi All,
This is v2 of a series to implement variable order, large folios for anonymous memory. The objective of this is to improve performance by allocating larger chunks of memory during anonymous page faults. See [1] for background.
I've significantly reworked and simplified the patch set based on comments from Yu Zhao (thanks for all your feedback!). I've also renamed the feature to VARIABLE_THP, on Yu's advice.
The last patch is for arm64 to explicitly override the default arch_wants_pte_order() and is intended as an example. If this series is accepted I suggest taking the first 4 patches through the mm tree and the arm64 change could be handled through the arm64 tree separately. Neither has any build dependency on the other.
The one area where I haven't followed Yu's advice is in the determination of the size of folio to use. It was suggested that I have a single preferred large order, and if it doesn't fit in the VMA (due to exceeding VMA bounds, or there being existing overlapping populated PTEs, etc) then fallback immediately to order-0. It turned out that this approach caused a performance regression in the Speedometer benchmark. With my v1 patch, there were significant quantities of memory which could not be placed in the 64K bucket and were instead being allocated for the 32K and 16K buckets. With the proposed simplification, that memory ended up using the 4K bucket, so page faults increased by 2.75x compared to the v1 patch (although due to the 64K bucket, this number is still a bit lower than the baseline). So instead, I continue to calculate a folio order that is somewhere between the preferred order and 0. (See below for more details).
The patches are based on top of v6.4 plus Matthew Wilcox's set_ptes() series [2], which is a hard dependency. I have a branch at [3].
Changes since v1 [1]
--------------------
- removed changes to arch-dependent vma_alloc_zeroed_movable_folio()
- replaced with arch-independent alloc_anon_folio()
- follows THP allocation approach
- no longer retry with intermediate orders if allocation fails
- fallback directly to order-0
- remove folio_add_new_anon_rmap_range() patch
- instead add its new functionality to folio_add_new_anon_rmap()
- remove batch-zap pte mappings optimization patch
- remove enabler folio_remove_rmap_range() patch too
- These offer real perf improvement so will submit separately
- simplify Kconfig
- single FLEXIBLE_THP option, which is independent of arch
- depends on TRANSPARENT_HUGEPAGE
- when enabled default to max anon folio size of 64K unless arch explicitly overrides
- simplify changes to do_anonymous_page():
- no more retry loop
Performance
-----------
Below results show 3 benchmarks; kernel compilation with 8 jobs, kernel compilation with 80 jobs, and speedometer 2.0 (a javascript benchmark running in Chromium). All cases are running on Ampere Altra with 1 NUMA node enabled, Ubuntu 22.04 and XFS filesystem. Each benchmark is repeated 15 times over 5 reboots and averaged.
'anonfolio-lkml-v1' is the v1 patchset at [1]. 'anonfolio-lkml-v2' is this v2 patchset. 'anonfolio-lkml-v2-simple-order' is anonfolio-lkml-v2 but with the order selection simplification that Yu Zhao suggested - I'm trying to justify here why I did not follow the advice.
Kernel compilation with 8 jobs:
| kernel | real-time | kern-time | user-time | |:-------------------------------|------------:|------------:|------------:| | baseline-4k | 0.0% | 0.0% | 0.0% | | anonfolio-lkml-v1 | -5.3% | -42.9% | -0.6% | | anonfolio-lkml-v2-simple-order | -4.4% | -36.5% | -0.4% | | anonfolio-lkml-v2 | -4.8% | -38.6% | -0.6% |
We can see that the simple-order approach is responsible for a regression of 0.4%.
Kernel compilation with 80 jobs:
| kernel | real-time | kern-time | user-time | |:-------------------------------|------------:|------------:|------------:| | baseline-4k | 0.0% | 0.0% | 0.0% | | anonfolio-lkml-v1 | -4.6% | -45.7% | 1.4% | | anonfolio-lkml-v2-simple-order | -4.7% | -40.2% | -0.1% | | anonfolio-lkml-v2 | -5.0% | -42.6% | -0.3% |
simple-order costs 0.3 % here. v2 is actually performing higher than v1 due to fixing the v1 regression on user-time.
Speedometer 2.0:
| kernel | runs_per_min | |:-------------------------------|---------------:| | baseline-4k | 0.0% | | anonfolio-lkml-v1 | 0.7% | | anonfolio-lkml-v2-simple-order | -0.9% | | anonfolio-lkml-v2 | 0.5% |
simple-order regresses performance by 0.9% vs the baseline, for a total negative swing of 1.6% vs v1. This is fixed by keeping the more complex order selection mechanism from v1.
The remaining (kernel time) performance gap between v1 and v2 for the above benchmarks is due to the removal of the "batch zap" patch in v2. Adding that back in gives us the performance back. I intend to submit that as a separate series once this series is accepted.
[1] https://lore.kernel.org/linux-mm/[email protected]/
[2] https://lore.kernel.org/linux-mm/[email protected]/
[3] https://gitlab.arm.com/linux-arm/linux-rr/-/tree/features/granule_perf/anonfolio-lkml_v2
Thanks,
Ryan
Ryan Roberts (5):
mm: Non-pmd-mappable, large folios for folio_add_new_anon_rmap()
mm: Allow deferred splitting of arbitrary large anon folios
mm: Default implementation of arch_wants_pte_order()
mm: FLEXIBLE_THP for improved performance
arm64: mm: Override arch_wants_pte_order()
arch/arm64/Kconfig | 12 +++ arch/arm64/include/asm/pgtable.h | 4 + arch/arm64/mm/mmu.c | 8 ++ include/linux/pgtable.h | 13 +++ mm/Kconfig | 10 ++ mm/memory.c | 168 ++++++++++++++++++++++++++++--- mm/rmap.c | 28 ++++-- 7 files changed, 222 insertions(+), 21 deletions(-)
--
2.25.1
PATCH v2 1/5 mm: Non-pmd-mappable, large folios for folio_add_new_anon_rmap()
In preparation for FLEXIBLE_THP support, improve folio_add_new_anon_rmap() to allow a non-pmd-mappable, large folio to be passed to it. In this case, all contained pages are accounted using the "small" pages scheme.
--- mm/rmap.c | 26 +++++++++++++++++++------- 1 file changed, 19 insertions(+), 7 deletions(-)
mm/rmap.c +19 -7
@@ -1278,31 +1278,43 @@ void page_add_anon_rmap(struct page *page, struct vm_area_struct *vma,* This means the inc-and-test can be bypassed. * The folio does not have to be locked. * -* If the folio is large, it is accounted as a THP. As the folio +* If the folio is pmd-mappable, it is accounted as a THP. As the folio * is new, it's assumed to be mapped exclusively by a single process. */ void folio_add_new_anon_rmap (struct folio *folio,struct vm_area_struct *vma,unsigned long address) { -int nr; +int nr =folio_nr_pages (folio); +int i; +struct page *page; -VM_BUG_ON_VMA (address < vma->vm_start || address >= vma->vm_end, vma); +VM_BUG_ON_VMA (address < vma->vm_start || + address + (nr <<PAGE_SHIFT ) > vma->vm_end, vma);__folio_set_swapbacked (folio); -if (likely (!folio_test_pmd_mappable (folio))) { +if (!folio_test_large (folio)) {/* increment count (starts at -1) */ atomic_set (&folio->_mapcount,0 ); - nr =1 ; +__page_set_anon_rmap (folio, &folio->page, vma, address,1 ); + }else if (!folio_test_pmd_mappable (folio)) { +/* increment count (starts at 0) */ +atomic_set (&folio->_nr_pages_mapped, nr); + + page = &folio->page; +for (i =0 ; i < nr; i++, page++, address +=PAGE_SIZE ) { +/* increment count (starts at -1) */ +atomic_set (&page->_mapcount,0 ); +__page_set_anon_rmap (folio, page, vma, address,1 ); + } }else {/* increment count (starts at -1) */ atomic_set (&folio->_entire_mapcount,0 );atomic_set (&folio->_nr_pages_mapped,COMPOUND_MAPPED ); - nr =folio_nr_pages (folio);__lruvec_stat_mod_folio (folio,NR_ANON_THPS , nr); +__page_set_anon_rmap (folio, &folio->page, vma, address,1 ); }__lruvec_stat_mod_folio (folio,NR_ANON_MAPPED , nr); -__page_set_anon_rmap (folio, &folio->page, vma, address,1 ); }/**
Signature · 1 line
2.25.1
Re: [PATCH v2 1/5] mm: Non-pmd-mappable, large folios for folio_add_new_anon_rmap()
Quoted 5 lines · Ryan Roberts
In preparation for
FLEXIBLE_THPsupport, improvefolio_add_new_anon_rmap()to allow a non-pmd-mappable, large folio to be passed to it. In this case, all contained pages are accounted using the "small" pages scheme.
Nit: In this case, all *subpages* are accounted using the *order-0 folio* (or base page) scheme.
Quoted 1 line
Signed-off-by Ryan Roberts <[email protected]>
Quoted 34 lines
mm/rmap.c | 26 +++++++++++++++++++------- 1 file changed, 19 insertions(+), 7 deletions(-)mm/rmap.c +8
-5@@ -1278,31 +1278,43 @@ void page_add_anon_rmap(struct page *page, struct vm_area_struct *vma,* This means the inc-and-test can be bypassed. * The folio does not have to be locked. * -* If the folio is large, it is accounted as a THP. As the folio +* If the folio is pmd-mappable, it is accounted as a THP. As the folio * is new, it's assumed to be mapped exclusively by a single process. */ void folio_add_new_anon_rmap (struct folio *folio,struct vm_area_struct *vma,unsigned long address) { -int nr; +int nr =folio_nr_pages (folio); +int i; +struct page *page; -VM_BUG_ON_VMA (address < vma->vm_start || address >= vma->vm_end, vma); +VM_BUG_ON_VMA (address < vma->vm_start || + address + (nr <<PAGE_SHIFT ) > vma->vm_end, vma);__folio_set_swapbacked (folio); -if (likely (!folio_test_pmd_mappable (folio))) { +if (!folio_test_large (folio)) {/* increment count (starts at -1) */ atomic_set (&folio->_mapcount,0 ); - nr =1 ; +__page_set_anon_rmap (folio, &folio->page, vma, address,1 );
mm/rmap.c void page_add_anon_rmap(struct page *page, struct vm_area_struct *vma, · line 1308
Quoted 10 lines
mm/rmap.c +10
-0+ }else if (!folio_test_pmd_mappable (folio)) { +/* increment count (starts at 0) */ +atomic_set (&folio->_nr_pages_mapped, nr); + + page = &folio->page; +for (i =0 ; i < nr; i++, page++, address +=PAGE_SIZE ) { +/* increment count (starts at -1) */ +atomic_set (&page->_mapcount,0 ); +__page_set_anon_rmap (folio, page, vma, address,1 ); + }
Nit: use folio_page(), e.g.,
} else if (!folio_test_pmd_mappable(folio)) {
int i; for (i = 0; i < nr; i++) {
struct page *page = folio_page(folio, i);
/* increment count (starts at -1) */
atomic_set(&page->_mapcount, 0);
__page_set_anon_rmap(folio, page, vma, address + PAGE_SIZE * i, 1);
}
/* increment count (starts at 0) */
atomic_set(&folio->_nr_pages_mapped, nr);} else {
Quoted 12 lines
mm/rmap.c +1
-2}else {/* increment count (starts at -1) */ atomic_set (&folio->_entire_mapcount,0 );atomic_set (&folio->_nr_pages_mapped,COMPOUND_MAPPED ); - nr =folio_nr_pages (folio);__lruvec_stat_mod_folio (folio,NR_ANON_THPS , nr); +__page_set_anon_rmap (folio, &folio->page, vma, address,1 ); }__lruvec_stat_mod_folio (folio,NR_ANON_MAPPED , nr); -__page_set_anon_rmap (folio, &folio->page, vma, address,1 ); }
Re: [PATCH v2 1/5] mm: Non-pmd-mappable, large folios for folio_add_new_anon_rmap()
Quoted 9 lines · Yu Zhao
Quoted 5 lines · Ryan Roberts
In preparation for
FLEXIBLE_THPsupport, improvefolio_add_new_anon_rmap()to allow a non-pmd-mappable, large folio to be passed to it. In this case, all contained pages are accounted using the "small" pages scheme.Nit: In this case, all *subpages* are accounted using the *order-0 folio* (or base page) scheme.
Matthew suggested not to use subpage with folio. Using page with folio:
https://lore.kernel.org/linux-mm/Y9qiS%[email protected]/
Quoted 78 lines
Quoted 2 lines
Signed-off-by Ryan Roberts <[email protected]>Reviewed-by Yu Zhao <[email protected]>Quoted 34 lines
mm/rmap.c | 26 +++++++++++++++++++------- 1 file changed, 19 insertions(+), 7 deletions(-)mm/rmap.c +8
-5@@ -1278,31 +1278,43 @@ void page_add_anon_rmap(struct page *page, struct vm_area_struct *vma,* This means the inc-and-test can be bypassed. * The folio does not have to be locked. * -* If the folio is large, it is accounted as a THP. As the folio +* If the folio is pmd-mappable, it is accounted as a THP. As the folio * is new, it's assumed to be mapped exclusively by a single process. */ void folio_add_new_anon_rmap (struct folio *folio,struct vm_area_struct *vma,unsigned long address) { -int nr; +int nr =folio_nr_pages (folio); +int i; +struct page *page; -VM_BUG_ON_VMA (address < vma->vm_start || address >= vma->vm_end, vma); +VM_BUG_ON_VMA (address < vma->vm_start || + address + (nr <<PAGE_SHIFT ) > vma->vm_end, vma);__folio_set_swapbacked (folio); -if (likely (!folio_test_pmd_mappable (folio))) { +if (!folio_test_large (folio)) {/* increment count (starts at -1) */ atomic_set (&folio->_mapcount,0 ); - nr =1 ; +__page_set_anon_rmap (folio, &folio->page, vma, address,1 );mm/rmap.c void page_add_anon_rmap(struct page *page, struct vm_area_struct *vma, · line 1308
Quoted 10 lines
mm/rmap.c +10
-0+ }else if (!folio_test_pmd_mappable (folio)) { +/* increment count (starts at 0) */ +atomic_set (&folio->_nr_pages_mapped, nr); + + page = &folio->page; +for (i =0 ; i < nr; i++, page++, address +=PAGE_SIZE ) { +/* increment count (starts at -1) */ +atomic_set (&page->_mapcount,0 ); +__page_set_anon_rmap (folio, page, vma, address,1 ); + }Nit: use
folio_page(), e.g.,} else if (!folio_test_pmd_mappable(folio)) { int i;for (i = 0; i < nr; i++) { struct page *page = folio_page(folio, i); /* increment count (starts at -1) */ atomic_set(&page->_mapcount, 0); __page_set_anon_rmap(folio, page, vma, address + PAGE_SIZE * i, 1); } /* increment count (starts at 0) */ atomic_set(&folio->_nr_pages_mapped, nr);} else {
Quoted 12 lines
mm/rmap.c +1
-2}else {/* increment count (starts at -1) */ atomic_set (&folio->_entire_mapcount,0 );atomic_set (&folio->_nr_pages_mapped,COMPOUND_MAPPED ); - nr =folio_nr_pages (folio);__lruvec_stat_mod_folio (folio,NR_ANON_THPS , nr); +__page_set_anon_rmap (folio, &folio->page, vma, address,1 ); }__lruvec_stat_mod_folio (folio,NR_ANON_MAPPED , nr); -__page_set_anon_rmap (folio, &folio->page, vma, address,1 ); }
Re: [PATCH v2 1/5] mm: Non-pmd-mappable, large folios for folio_add_new_anon_rmap()
Quoted 10 lines · Fengwei
Quoted 7 lines · Yu Zhao
Quoted 5 lines · Ryan Roberts
In preparation for
FLEXIBLE_THPsupport, improvefolio_add_new_anon_rmap()to allow a non-pmd-mappable, large folio to be passed to it. In this case, all contained pages are accounted using the "small" pages scheme.
Quoted 4 lines
Quoted 2 lines
Nit: In this case, all *subpages* are accounted using the *order-0 folio* (or base page) scheme.
Matthew suggested not to use subpage with folio. Using page with folio:
https://lore.kernel.org/linux-mm/Y9qiS%[email protected]/
OK, I'll change this to "In this case, all contained pages are accounted using the *order-0 folio* (or base page) scheme."
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
Signed-off-by Ryan Roberts <[email protected]>Reviewed-by Yu Zhao <[email protected]>
Thanks!
Quoted 39 lines
Quoted 39 lines
Quoted 39 lines
mm/rmap.c | 26 +++++++++++++++++++------- 1 file changed, 19 insertions(+), 7 deletions(-)mm/rmap.c +12
-5@@ -1278,31 +1278,43 @@ void page_add_anon_rmap(struct page *page, struct vm_area_struct *vma,* This means the inc-and-test can be bypassed. * The folio does not have to be locked. * -* If the folio is large, it is accounted as a THP. As the folio +* If the folio is pmd-mappable, it is accounted as a THP. As the folio * is new, it's assumed to be mapped exclusively by a single process. */ void folio_add_new_anon_rmap (struct folio *folio,struct vm_area_struct *vma,unsigned long address) { -int nr; +int nr =folio_nr_pages (folio); +int i; +struct page *page; -VM_BUG_ON_VMA (address < vma->vm_start || address >= vma->vm_end, vma); +VM_BUG_ON_VMA (address < vma->vm_start || + address + (nr <<PAGE_SHIFT ) > vma->vm_end, vma);__folio_set_swapbacked (folio); -if (likely (!folio_test_pmd_mappable (folio))) { +if (!folio_test_large (folio)) {/* increment count (starts at -1) */ atomic_set (&folio->_mapcount,0 ); - nr =1 ; +__page_set_anon_rmap (folio, &folio->page, vma, address,1 ); + }else if (!folio_test_pmd_mappable (folio)) { +/* increment count (starts at 0) */ +atomic_set (&folio->_nr_pages_mapped, nr); +
Quoted 8 lines
Quoted 8 lines
mm/rmap.c void page_add_anon_rmap(struct page *page, struct vm_area_struct *vma, · line 1308
Quoted 6 lines
mm/rmap.c +6
-0+ page = &folio->page; +for (i =0 ; i < nr; i++, page++, address +=PAGE_SIZE ) { +/* increment count (starts at -1) */ +atomic_set (&page->_mapcount,0 ); +__page_set_anon_rmap (folio, page, vma, address,1 ); + }Nit: use
folio_page(), e.g.,
Yep, will change for v3.
Quoted 27 lines
Quoted 27 lines
} else if (!folio_test_pmd_mappable(folio)) { int i;for (i = 0; i < nr; i++) { struct page *page = folio_page(folio, i); /* increment count (starts at -1) */ atomic_set(&page->_mapcount, 0); __page_set_anon_rmap(folio, page, vma, address + PAGE_SIZE * i, 1); } /* increment count (starts at 0) */ atomic_set(&folio->_nr_pages_mapped, nr);} else {
Quoted 12 lines
mm/rmap.c +1
-2}else {/* increment count (starts at -1) */ atomic_set (&folio->_entire_mapcount,0 );atomic_set (&folio->_nr_pages_mapped,COMPOUND_MAPPED ); - nr =folio_nr_pages (folio);__lruvec_stat_mod_folio (folio,NR_ANON_THPS , nr); +__page_set_anon_rmap (folio, &folio->page, vma, address,1 ); }__lruvec_stat_mod_folio (folio,NR_ANON_MAPPED , nr); -__page_set_anon_rmap (folio, &folio->page, vma, address,1 ); }
Re: [PATCH v2 1/5] mm: Non-pmd-mappable, large folios for folio_add_new_anon_rmap()
In reply to Ryan RobertsQuoted 6 lines · Ryan Roberts
In preparation for
FLEXIBLE_THPsupport, improvefolio_add_new_anon_rmap()to allow a non-pmd-mappable, large folio to be passed to it. In this case, all contained pages are accounted using the "small" pages scheme.Signed-off-by Ryan Roberts <[email protected]>
Quoted 59 lines
--- mm/rmap.c | 26 +++++++++++++++++++------- 1 file changed, 19 insertions(+), 7 deletions(-)mm/rmap.c +19
-7@@ -1278,31 +1278,43 @@ void page_add_anon_rmap(struct page *page, struct vm_area_struct *vma,* This means the inc-and-test can be bypassed. * The folio does not have to be locked. * -* If the folio is large, it is accounted as a THP. As the folio +* If the folio is pmd-mappable, it is accounted as a THP. As the folio * is new, it's assumed to be mapped exclusively by a single process. */ void folio_add_new_anon_rmap (struct folio *folio,struct vm_area_struct *vma,unsigned long address) { -int nr; +int nr =folio_nr_pages (folio); +int i; +struct page *page; -VM_BUG_ON_VMA (address < vma->vm_start || address >= vma->vm_end, vma); +VM_BUG_ON_VMA (address < vma->vm_start || + address + (nr <<PAGE_SHIFT ) > vma->vm_end, vma);__folio_set_swapbacked (folio); -if (likely (!folio_test_pmd_mappable (folio))) { +if (!folio_test_large (folio)) {/* increment count (starts at -1) */ atomic_set (&folio->_mapcount,0 ); - nr =1 ; +__page_set_anon_rmap (folio, &folio->page, vma, address,1 ); + }else if (!folio_test_pmd_mappable (folio)) { +/* increment count (starts at 0) */ +atomic_set (&folio->_nr_pages_mapped, nr); + + page = &folio->page; +for (i =0 ; i < nr; i++, page++, address +=PAGE_SIZE ) { +/* increment count (starts at -1) */ +atomic_set (&page->_mapcount,0 ); +__page_set_anon_rmap (folio, page, vma, address,1 ); + } }else {/* increment count (starts at -1) */ atomic_set (&folio->_entire_mapcount,0 );atomic_set (&folio->_nr_pages_mapped,COMPOUND_MAPPED ); - nr =folio_nr_pages (folio);__lruvec_stat_mod_folio (folio,NR_ANON_THPS , nr); +__page_set_anon_rmap (folio, &folio->page, vma, address,1 ); }__lruvec_stat_mod_folio (folio,NR_ANON_MAPPED , nr); -__page_set_anon_rmap (folio, &folio->page, vma, address,1 ); }/**
PATCH v2 2/5 mm: Allow deferred splitting of arbitrary large anon folios
In reply to Ryan RobertsWith the introduction of large folios for anonymous memory, we would like to be able to split them when they have unmapped subpages, in order to free those unused pages under memory pressure. So remove the artificial requirement that the large folio needed to be at least PMD-sized.
--- mm/rmap.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
mm/rmap.c +1 -1
@@ -1474,7 +1474,7 @@ void page_remove_rmap(struct page *page, struct vm_area_struct *vma,* page of the folio is unmapped and at least one page * is still mapped. */ -if (folio_test_pmd_mappable (folio) &&folio_test_anon (folio)) +if (folio_test_large (folio) &&folio_test_anon (folio))if (!compound || nr < nr_pmdmapped)deferred_split_folio (folio); }
Signature · 1 line
2.25.1
Re: [PATCH v2 2/5] mm: Allow deferred splitting of arbitrary large anon folios
Quoted 18 lines · Ryan Roberts
With the introduction of large folios for anonymous memory, we would like to be able to split them when they have unmapped subpages, in order to free those unused pages under memory pressure. So remove the artificial requirement that the large folio needed to be at least PMD-sized.
Signed-off-by Ryan Roberts <[email protected]>Reviewed-by Yu Zhao <[email protected]>Reviewed-by Yin Fengwei <[email protected]>--- mm/rmap.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)mm/rmap.c +0
-0
mm/rmap.c void page_remove_rmap(struct page *page, struct vm_area_struct *vma, · line 1480
Quoted 8 lines
mm/rmap.c +1
-1@@ -1474,7 +1474,7 @@ void page_remove_rmap(struct page *page, struct vm_area_struct *vma,* page of the folio is unmapped and at least one page * is still mapped. */ -if (folio_test_pmd_mappable (folio) &&folio_test_anon (folio)) +if (folio_test_large (folio) &&folio_test_anon (folio))if (!compound || nr < nr_pmdmapped)deferred_split_folio (folio); }
One possible issue is that even for large folios mapped only in one process, in zap_pte_range(), we will always call deferred_split_folio() unnecessarily before freeing a large folio.
Best Regards,
Huang, Ying
Re: [PATCH v2 2/5] mm: Allow deferred splitting of arbitrary large anon folios
Quoted 24 lines · Ying
Quoted 22 lines · Ryan Roberts
With the introduction of large folios for anonymous memory, we would like to be able to split them when they have unmapped subpages, in order to free those unused pages under memory pressure. So remove the artificial requirement that the large folio needed to be at least PMD-sized.
Signed-off-by Ryan Roberts <[email protected]>Reviewed-by Yu Zhao <[email protected]>Reviewed-by Yin Fengwei <[email protected]>--- mm/rmap.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)mm/rmap.c +0
-1@@ -1474,7 +1474,7 @@ void page_remove_rmap(struct page *page, struct vm_area_struct *vma,* page of the folio is unmapped and at least one page * is still mapped. */ -if (folio_test_pmd_mappable (folio) &&folio_test_anon (folio))
Quoted 8 lines
mm/rmap.c void page_remove_rmap(struct page *page, struct vm_area_struct *vma, · line 1480
Quoted 4 lines
mm/rmap.c +1
-0+if (folio_test_large (folio) &&folio_test_anon (folio))if (!compound || nr < nr_pmdmapped)deferred_split_folio (folio); }One possible issue is that even for large folios mapped only in one process, in
zap_pte_range(), we will always calldeferred_split_folio()unnecessarily before freeing a large folio.
Hi Huang, thanks for reviewing!
I have a patch that solves this problem by determining a range of ptes covered by a single folio and doing a "batch zap". This prevents the need to add the folio to the deferred split queue, only to remove it again shortly afterwards. This reduces lock contention and I can measure a performance improvement for the kernel compilation benchmark. See [1].
However, I decided to remove it from this patch set on Yu Zhao's advice. We are aiming for the minimal patch set to start with and wanted to focus people on that. I intend to submit it separately later on.
[1] https://lore.kernel.org/linux-mm/[email protected]/
Thanks,
Ryan
Quoted 8 lines
Best Regards,
Huang, Ying_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Re: [PATCH v2 2/5] mm: Allow deferred splitting of arbitrary large anon folios
In reply to Huang, YingSomehow I managed to reply only to the linux-arm-kernel list on first attempt so resending:
Quoted 24 lines · Ying
Quoted 22 lines · Ryan Roberts
With the introduction of large folios for anonymous memory, we would like to be able to split them when they have unmapped subpages, in order to free those unused pages under memory pressure. So remove the artificial requirement that the large folio needed to be at least PMD-sized.
Signed-off-by Ryan Roberts <[email protected]>Reviewed-by Yu Zhao <[email protected]>Reviewed-by Yin Fengwei <[email protected]>--- mm/rmap.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)mm/rmap.c +0
-1@@ -1474,7 +1474,7 @@ void page_remove_rmap(struct page *page, struct vm_area_struct *vma,* page of the folio is unmapped and at least one page * is still mapped. */ -if (folio_test_pmd_mappable (folio) &&folio_test_anon (folio))
Quoted 8 lines
mm/rmap.c void page_remove_rmap(struct page *page, struct vm_area_struct *vma, · line 1480
Quoted 4 lines
mm/rmap.c +1
-0+if (folio_test_large (folio) &&folio_test_anon (folio))if (!compound || nr < nr_pmdmapped)deferred_split_folio (folio); }One possible issue is that even for large folios mapped only in one process, in
zap_pte_range(), we will always calldeferred_split_folio()unnecessarily before freeing a large folio.
Hi Huang, thanks for reviewing!
I have a patch that solves this problem by determining a range of ptes covered by a single folio and doing a "batch zap". This prevents the need to add the folio to the deferred split queue, only to remove it again shortly afterwards. This reduces lock contention and I can measure a performance improvement for the kernel compilation benchmark. See [1].
However, I decided to remove it from this patch set on Yu Zhao's advice. We are aiming for the minimal patch set to start with and wanted to focus people on that. I intend to submit it separately later on.
[1] https://lore.kernel.org/linux-mm/[email protected]/
Thanks,
Ryan
Quoted 8 lines
Best Regards,
Huang, Ying_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Re: [PATCH v2 2/5] mm: Allow deferred splitting of arbitrary large anon folios
Quoted 45 lines · Ryan Roberts
Somehow I managed to reply only to the linux-arm-kernel list on first attempt so resending:
Quoted 24 lines · Ying
Quoted 22 lines · Ryan Roberts
With the introduction of large folios for anonymous memory, we would like to be able to split them when they have unmapped subpages, in order to free those unused pages under memory pressure. So remove the artificial requirement that the large folio needed to be at least PMD-sized.
Signed-off-by Ryan Roberts <[email protected]>Reviewed-by Yu Zhao <[email protected]>Reviewed-by Yin Fengwei <[email protected]>--- mm/rmap.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)mm/rmap.c +0
-1@@ -1474,7 +1474,7 @@ void page_remove_rmap(struct page *page, struct vm_area_struct *vma,* page of the folio is unmapped and at least one page * is still mapped. */ -if (folio_test_pmd_mappable (folio) &&folio_test_anon (folio))Quoted 8 lines
mm/rmap.c void page_remove_rmap(struct page *page, struct vm_area_struct *vma, · line 1480
Quoted 4 lines
mm/rmap.c +1
-0+if (folio_test_large (folio) &&folio_test_anon (folio))if (!compound || nr < nr_pmdmapped)deferred_split_folio (folio); }One possible issue is that even for large folios mapped only in one process, in
zap_pte_range(), we will always calldeferred_split_folio()unnecessarily before freeing a large folio.Hi Huang, thanks for reviewing!
I have a patch that solves this problem by determining a range of ptes covered by a single folio and doing a "batch zap". This prevents the need to add the folio to the deferred split queue, only to remove it again shortly afterwards. This reduces lock contention and I can measure a performance improvement for the kernel compilation benchmark. See [1].
Quoted 5 lines
However, I decided to remove it from this patch set on Yu Zhao's advice. We are aiming for the minimal patch set to start with and wanted to focus people on that. I intend to submit it separately later on.
Thanks for your information! "batch zap" can solve the problem.
And, I agree with Matthew's comments to fix the large folios interaction issues before merging the patches to allocate large folios as in the following email.
https://lore.kernel.org/linux-mm/[email protected]/
If so, we don't need to introduce the above problem or a large patchset.
Best Regards,
Huang, Ying
Re: [PATCH v2 2/5] mm: Allow deferred splitting of arbitrary large anon folios
Quoted 55 lines · Ying
Quoted 45 lines · Ryan Roberts
Somehow I managed to reply only to the linux-arm-kernel list on first attempt so resending:
Quoted 24 lines · Ying
Quoted 22 lines · Ryan Roberts
With the introduction of large folios for anonymous memory, we would like to be able to split them when they have unmapped subpages, in order to free those unused pages under memory pressure. So remove the artificial requirement that the large folio needed to be at least PMD-sized.
Signed-off-by Ryan Roberts <[email protected]>Reviewed-by Yu Zhao <[email protected]>Reviewed-by Yin Fengwei <[email protected]>--- mm/rmap.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)mm/rmap.c +0
-1@@ -1474,7 +1474,7 @@ void page_remove_rmap(struct page *page, struct vm_area_struct *vma,* page of the folio is unmapped and at least one page * is still mapped. */ -if (folio_test_pmd_mappable (folio) &&folio_test_anon (folio))Quoted 8 lines
mm/rmap.c void page_remove_rmap(struct page *page, struct vm_area_struct *vma, · line 1480
Quoted 4 lines
mm/rmap.c +1
-0+if (folio_test_large (folio) &&folio_test_anon (folio))if (!compound || nr < nr_pmdmapped)deferred_split_folio (folio); }One possible issue is that even for large folios mapped only in one process, in
zap_pte_range(), we will always calldeferred_split_folio()unnecessarily before freeing a large folio.Hi Huang, thanks for reviewing!
I have a patch that solves this problem by determining a range of ptes covered by a single folio and doing a "batch zap". This prevents the need to add the folio to the deferred split queue, only to remove it again shortly afterwards. This reduces lock contention and I can measure a performance improvement for the kernel compilation benchmark. See [1].
Quoted 5 lines
However, I decided to remove it from this patch set on Yu Zhao's advice. We are aiming for the minimal patch set to start with and wanted to focus people on that. I intend to submit it separately later on.
Thanks for your information! "batch zap" can solve the problem.
Quoted 7 lines
And, I agree with Matthew's comments to fix the large folios interaction issues before merging the patches to allocate large folios as in the following email.
https://lore.kernel.org/linux-mm/[email protected]/
If so, we don't need to introduce the above problem or a large patchset.
I appreciate Matthew's and others position about not wanting to merge a minimal implementation while there are some fundamental features (e.g. compaction) it doesn't play well with - I'm working to create a definitive list so these items can be tracked and tackled.
That said, I don't see this "batch zap" patch as an example of this. It's just a performance enhancement that improves things even further than large anon folios on their own. I'd rather concentrate on the core changes first then deal with this type of thing later. Does that work for you?
Quoted 3 lines
Best Regards,
Huang, Ying
Re: [PATCH v2 2/5] mm: Allow deferred splitting of arbitrary large anon folios
Quoted 60 lines · Ryan Roberts
Quoted 59 lines · Ying
Quoted 45 lines · Ryan Roberts
Somehow I managed to reply only to the linux-arm-kernel list on first attempt so resending:
Quoted 24 lines · Ying
Quoted 22 lines · Ryan Roberts
With the introduction of large folios for anonymous memory, we would like to be able to split them when they have unmapped subpages, in order to free those unused pages under memory pressure. So remove the artificial requirement that the large folio needed to be at least PMD-sized.
Signed-off-by Ryan Roberts <[email protected]>Reviewed-by Yu Zhao <[email protected]>Reviewed-by Yin Fengwei <[email protected]>--- mm/rmap.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)mm/rmap.c +0
-1@@ -1474,7 +1474,7 @@ void page_remove_rmap(struct page *page, struct vm_area_struct *vma,* page of the folio is unmapped and at least one page * is still mapped. */ -if (folio_test_pmd_mappable (folio) &&folio_test_anon (folio))Quoted 8 lines
mm/rmap.c void page_remove_rmap(struct page *page, struct vm_area_struct *vma, · line 1480
Quoted 4 lines
mm/rmap.c +1
-0+if (folio_test_large (folio) &&folio_test_anon (folio))if (!compound || nr < nr_pmdmapped)deferred_split_folio (folio); }One possible issue is that even for large folios mapped only in one process, in
zap_pte_range(), we will always calldeferred_split_folio()unnecessarily before freeing a large folio.Hi Huang, thanks for reviewing!
I have a patch that solves this problem by determining a range of ptes covered by a single folio and doing a "batch zap". This prevents the need to add the folio to the deferred split queue, only to remove it again shortly afterwards. This reduces lock contention and I can measure a performance improvement for the kernel compilation benchmark. See [1].
Quoted 5 lines
However, I decided to remove it from this patch set on Yu Zhao's advice. We are aiming for the minimal patch set to start with and wanted to focus people on that. I intend to submit it separately later on.
Thanks for your information! "batch zap" can solve the problem.
And, I agree with Matthew's comments to fix the large folios interaction issues before merging the patches to allocate large folios as in the following email.
Quoted 8 lines
Quoted 3 lines
https://lore.kernel.org/linux-mm/[email protected]/
If so, we don't need to introduce the above problem or a large patchset.
I appreciate Matthew's and others position about not wanting to merge a minimal implementation while there are some fundamental features (e.g. compaction) it doesn't play well with - I'm working to create a definitive list so these items can be tracked and tackled.
Good to know this, Thanks!
Quoted 4 lines
That said, I don't see this "batch zap" patch as an example of this. It's just a performance enhancement that improves things even further than large anon folios on their own. I'd rather concentrate on the core changes first then deal with this type of thing later. Does that work for you?
IIUC, allocating large folios upon page fault depends on splitting large folios in page_remove_rmap() to avoid memory wastage. Splitting large folios in page_remove_rmap() depends on "batch zap" to avoid performance regression in zap_pte_range(). So we need them to be done earlier. Or I miss something?
Best Regards,
Huang, Ying
Re: [PATCH v2 2/5] mm: Allow deferred splitting of arbitrary large anon folios
Quoted 78 lines · Ying
Quoted 60 lines · Ryan Roberts
Quoted 59 lines · Ying
Quoted 45 lines · Ryan Roberts
Somehow I managed to reply only to the linux-arm-kernel list on first attempt so resending:
Quoted 24 lines · Ying
Quoted 22 lines · Ryan Roberts
With the introduction of large folios for anonymous memory, we would like to be able to split them when they have unmapped subpages, in order to free those unused pages under memory pressure. So remove the artificial requirement that the large folio needed to be at least PMD-sized.
Signed-off-by Ryan Roberts <[email protected]>Reviewed-by Yu Zhao <[email protected]>Reviewed-by Yin Fengwei <[email protected]>--- mm/rmap.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)mm/rmap.c +0
-1@@ -1474,7 +1474,7 @@ void page_remove_rmap(struct page *page, struct vm_area_struct *vma,* page of the folio is unmapped and at least one page * is still mapped. */ -if (folio_test_pmd_mappable (folio) &&folio_test_anon (folio))Quoted 8 lines
mm/rmap.c void page_remove_rmap(struct page *page, struct vm_area_struct *vma, · line 1480
Quoted 4 lines
mm/rmap.c +1
-0+if (folio_test_large (folio) &&folio_test_anon (folio))if (!compound || nr < nr_pmdmapped)deferred_split_folio (folio); }One possible issue is that even for large folios mapped only in one process, in
zap_pte_range(), we will always calldeferred_split_folio()unnecessarily before freeing a large folio.Hi Huang, thanks for reviewing!
I have a patch that solves this problem by determining a range of ptes covered by a single folio and doing a "batch zap". This prevents the need to add the folio to the deferred split queue, only to remove it again shortly afterwards. This reduces lock contention and I can measure a performance improvement for the kernel compilation benchmark. See [1].
Quoted 5 lines
However, I decided to remove it from this patch set on Yu Zhao's advice. We are aiming for the minimal patch set to start with and wanted to focus people on that. I intend to submit it separately later on.
Thanks for your information! "batch zap" can solve the problem.
And, I agree with Matthew's comments to fix the large folios interaction issues before merging the patches to allocate large folios as in the following email.
Quoted 8 lines
Quoted 3 lines
https://lore.kernel.org/linux-mm/[email protected]/
If so, we don't need to introduce the above problem or a large patchset.
I appreciate Matthew's and others position about not wanting to merge a minimal implementation while there are some fundamental features (e.g. compaction) it doesn't play well with - I'm working to create a definitive list so these items can be tracked and tackled.
Good to know this, Thanks!
Quoted 4 lines
That said, I don't see this "batch zap" patch as an example of this. It's just a performance enhancement that improves things even further than large anon folios on their own. I'd rather concentrate on the core changes first then deal with this type of thing later. Does that work for you?
Quoted 5 lines
IIUC, allocating large folios upon page fault depends on splitting large folios in
page_remove_rmap()to avoid memory wastage. Splitting large folios inpage_remove_rmap()depends on "batch zap" to avoid performance regression inzap_pte_range(). So we need them to be done earlier. Or I miss something?
My point was just that large anon folios improves performance significantly overall, despite a small perf regression in zap_pte_range(). That regression is reduced further by a patch from Yin Fengwei to reduce the lock contention [1]. So it doesn't seem urgent to me to get the "batch zap" change in.
I'll add it to my list, then prioritize it against the other stuff.
[1] https://lore.kernel.org/linux-mm/[email protected]/
Quoted 3 lines
Best Regards,
Huang, Ying
Re: [PATCH v2 2/5] mm: Allow deferred splitting of arbitrary large anon folios
Quoted 85 lines · Ryan Roberts
Quoted 78 lines · Ying
Quoted 60 lines · Ryan Roberts
Quoted 59 lines · Ying
Quoted 45 lines · Ryan Roberts
Somehow I managed to reply only to the linux-arm-kernel list on first attempt so resending:
Quoted 24 lines · Ying
Quoted 22 lines · Ryan Roberts
With the introduction of large folios for anonymous memory, we would like to be able to split them when they have unmapped subpages, in order to free those unused pages under memory pressure. So remove the artificial requirement that the large folio needed to be at least PMD-sized.
Signed-off-by Ryan Roberts <[email protected]>Reviewed-by Yu Zhao <[email protected]>Reviewed-by Yin Fengwei <[email protected]>--- mm/rmap.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)mm/rmap.c +0
-1@@ -1474,7 +1474,7 @@ void page_remove_rmap(struct page *page, struct vm_area_struct *vma,* page of the folio is unmapped and at least one page * is still mapped. */ -if (folio_test_pmd_mappable (folio) &&folio_test_anon (folio))Quoted 8 lines
mm/rmap.c void page_remove_rmap(struct page *page, struct vm_area_struct *vma, · line 1480
Quoted 4 lines
mm/rmap.c +1
-0+if (folio_test_large (folio) &&folio_test_anon (folio))if (!compound || nr < nr_pmdmapped)deferred_split_folio (folio); }One possible issue is that even for large folios mapped only in one process, in
zap_pte_range(), we will always calldeferred_split_folio()unnecessarily before freeing a large folio.Hi Huang, thanks for reviewing!
I have a patch that solves this problem by determining a range of ptes covered by a single folio and doing a "batch zap". This prevents the need to add the folio to the deferred split queue, only to remove it again shortly afterwards. This reduces lock contention and I can measure a performance improvement for the kernel compilation benchmark. See [1].
Quoted 5 lines
However, I decided to remove it from this patch set on Yu Zhao's advice. We are aiming for the minimal patch set to start with and wanted to focus people on that. I intend to submit it separately later on.
Thanks for your information! "batch zap" can solve the problem.
And, I agree with Matthew's comments to fix the large folios interaction issues before merging the patches to allocate large folios as in the following email.
Quoted 8 lines
Quoted 3 lines
https://lore.kernel.org/linux-mm/[email protected]/
If so, we don't need to introduce the above problem or a large patchset.
I appreciate Matthew's and others position about not wanting to merge a minimal implementation while there are some fundamental features (e.g. compaction) it doesn't play well with - I'm working to create a definitive list so these items can be tracked and tackled.
Good to know this, Thanks!
Quoted 4 lines
That said, I don't see this "batch zap" patch as an example of this. It's just a performance enhancement that improves things even further than large anon folios on their own. I'd rather concentrate on the core changes first then deal with this type of thing later. Does that work for you?
Quoted 5 lines
IIUC, allocating large folios upon page fault depends on splitting large folios in
page_remove_rmap()to avoid memory wastage. Splitting large folios inpage_remove_rmap()depends on "batch zap" to avoid performance regression inzap_pte_range(). So we need them to be done earlier. Or I miss something?
Quoted 4 lines
My point was just that large anon folios improves performance significantly overall, despite a small perf regression in
zap_pte_range(). That regression is reduced further by a patch from Yin Fengwei to reduce the lock contention [1]. So it doesn't seem urgent to me to get the "batch zap" change in.
I don't think Fengwei's patch will help much here. Because that patch is to optimize if the folio isn't in deferred split queue, but now the folio will be put in deferred split queue.
And I don't think allocating large folios upon page fault is more urgent. We should avoid regression if possible.
Quoted 4 lines
I'll add it to my list, then prioritize it against the other stuff.
Best Regards,
Huang, Ying
PATCH v2 3/5 mm: Default implementation of arch_wants_pte_order()
In reply to Ryan Robertsarch_wants_pte_order() can be overridden by the arch to return the preferred folio order for pte-mapped memory. This is useful as some architectures (e.g. arm64) can coalesce TLB entries when the physical memory is suitably contiguous.
The first user for this hint will be FLEXIBLE_THP, which aims to allocate large folios for anonymous memory to reduce page faults and other per-page operation costs.
Here we add the default implementation of the function, used when the architecture does not define it, which returns the order corresponding to 64K.
--- include/linux/pgtable.h | 13 +++++++++++++ 1 file changed, 13 insertions(+)
include/linux/pgtable.h +13 -0
@@ -13,6 +13,7 @@#include <linux/errno.h> #include <asm-generic/pgtable_uffd.h> #include <linux/page_table_check.h> +#include <linux/sizes.h> #if 5 - defined(__PAGETABLE_P4D_FOLDED) - defined(__PAGETABLE_PUD_FOLDED) - \ defined(__PAGETABLE_PMD_FOLDED) !=CONFIG_PGTABLE_LEVELS
@@ -336,6 +337,18 @@ static inline bool arch_has_hw_pte_young(void) }#endif +#ifndef arch_wants_pte_order +/* +* Returns preferred folio order for pte-mapped memory. Must be in range [0, +* PMD_SHIFT-PAGE_SHIFT) and must not be order-1 since THP requires large folios +* to be at least order-2. +*/ +static inline int arch_wants_pte_order (struct vm_area_struct *vma) +{ +return ilog2 (SZ_64K >>PAGE_SHIFT ); +} +#endif +#ifndef __HAVE_ARCH_PTEP_GET_AND_CLEAR static inline pte_t ptep_get_and_clear(struct mm_struct *mm, unsigned long address,
Signature · 1 line
2.25.1
Re: [PATCH v2 3/5] mm: Default implementation of arch_wants_pte_order()
Quoted 6 lines · Ryan Roberts
arch_wants_pte_order()can be overridden by the arch to return the preferred folio order for pte-mapped memory. This is useful as some architectures (e.g. arm64) can coalesce TLB entries when the physical memory is suitably contiguous.
Quoted 7 lines
The first user for this hint will be
FLEXIBLE_THP, which aims to allocate large folios for anonymous memory to reduce page faults and other per-page operation costs.Here we add the default implementation of the function, used when the architecture does not define it, which returns the order corresponding to 64K.
I don't really mind a non-zero default value. But people would ask why non-zero and why 64KB. Probably you could argue this is the large size all known archs support if they have TLB coalescing. For x86, AMD CPUs would want to override this. I'll leave it to Fengwei to decide whether Intel wants a different default value.
Also I don't like the vma parameter because it makes arch_wants_pte_order() a mix of hw preference and vma policy. From my POV, the function should be only about the former; the latter should be decided by arch-independent MM code. However, I can live with it if ARM MM people think this is really what you want. ATM, I'm skeptical they do.
Quoted 1 line
Signed-off-by Ryan Roberts <[email protected]>
After another CPU vendor, e.g., Fengwei, and an ARM MM person, e.g., Will give the green light:
Quoted 17 lines
--- include/linux/pgtable.h | 13 +++++++++++++ 1 file changed, 13 insertions(+)include/linux/pgtable.h +1
-0@@ -13,6 +13,7 @@#include <linux/errno.h> #include <asm-generic/pgtable_uffd.h> #include <linux/page_table_check.h> +#include <linux/sizes.h> #if 5 - defined(__PAGETABLE_P4D_FOLDED) - defined(__PAGETABLE_PUD_FOLDED) - \ defined(__PAGETABLE_PMD_FOLDED) !=CONFIG_PGTABLE_LEVELS
include/linux/pgtable.h static inline bool arch_has_hw_pte_young(void) · line 343
Quoted 7 lines
include/linux/pgtable.h +4
-0@@ -336,6 +337,18 @@ static inline bool arch_has_hw_pte_young(void) }#endif +#ifndef arch_wants_pte_order +/* +* Returns preferred folio order for pte-mapped memory. Must be in range [0, +* PMD_SHIFT-PAGE_SHIFT) and must not be order-1 since THP requires large folios
The warning is helpful.
Quoted 11 lines
include/linux/pgtable.h +8
-0+* to be at least order-2. +*/ +static inline int arch_wants_pte_order (struct vm_area_struct *vma) +{ +return ilog2 (SZ_64K >>PAGE_SHIFT ); +} +#endif +#ifndef __HAVE_ARCH_PTEP_GET_AND_CLEAR static inline pte_t ptep_get_and_clear(struct mm_struct *mm, unsigned long address,
Re: [PATCH v2 3/5] mm: Default implementation of arch_wants_pte_order()
Quoted 15 lines · Yu Zhao
Quoted 6 lines · Ryan Roberts
arch_wants_pte_order()can be overridden by the arch to return the preferred folio order for pte-mapped memory. This is useful as some architectures (e.g. arm64) can coalesce TLB entries when the physical memory is suitably contiguous.Quoted 7 lines
The first user for this hint will be
FLEXIBLE_THP, which aims to allocate large folios for anonymous memory to reduce page faults and other per-page operation costs.Here we add the default implementation of the function, used when the architecture does not define it, which returns the order corresponding to 64K.
Quoted 11 lines
I don't really mind a non-zero default value. But people would ask why non-zero and why 64KB. Probably you could argue this is the large size all known archs support if they have TLB coalescing. For x86, AMD CPUs would want to override this. I'll leave it to Fengwei to decide whether Intel wants a different default value.>
Also I don't like the vma parameter because it makesarch_wants_pte_order()a mix of hw preference and vma policy. From my POV, the function should be only about the former; the latter should be decided by arch-independent MM code. However, I can live with it if ARM MM people think this is really what you want. ATM, I'm skeptical they do.
Here's the big picture for what I'm tryng to achieve:
- In the common case, I'd like all programs to get a performance bump by
automatically and transparently using large anon folios - so no explicit requirement on the process to opt-in.
- On arm64, in the above case, I'd like the preferred folio size to be 64K;
from the (admittedly limitted) testing I've done that's about where the performance knee is and it doesn't appear to increase the memory wastage very much. It also has the benefits that for 4K base pages this is the contpte size (order-4) so I can take full benefit of contpte mappings transparently to the process. And for 16K this is the HPA size (order-2).
- On arm64 when the process has marked the VMA for THP (or when
transparent_hugepage=always) but the VMA does not meet the requirements for a PMD-sized mapping (or we failed to allocate, ...) then I'd like to map using contpte. For 4K base pages this is 64K (order-4), for 16K this is 2M (order-7) and for 64K this is 2M (order-5). The 64K base page case is very important since the PMD size for that base page is 512MB which is almost impossible to allocate in practice.
So one approach would be to define arch_wants_pte_order() as always returning the contpte size (remove the vma parameter). Then max_anon_folio_order() in memory.c could so this:
#define MAX_ANON_FOLIO_ORDER_NOTHP ilog2(SZ_64K >> PAGE_SHIFT);
static inline int max_anon_folio_order(struct vm_area_struct *vma)
{
int order = arch_wants_pte_order();
// Fix up default case which returns 0 because PAGE_ALLOC_COSTLY_ORDER
// can't be used directly in pgtable.h
order = order ? order : PAGE_ALLOC_COSTLY_ORDER;
if (hugepage_vma_check(vma, vma->vm_flags, false, true, true))
return order;
else
return min(order, MAX_ANON_FOLIO_ORDER_NOTHP);
}This moves the SW policy into memory.c and gives you PAGE_ALLOC_COSTLY_ORDER (or whatever default we decide on) as the default for arches with no override, and also meets all my goals above.
Quoted 45 lines
Quoted 2 lines
Signed-off-by Ryan Roberts <[email protected]>After another CPU vendor, e.g., Fengwei, and an ARM MM person, e.g., Will give the green light:
Reviewed-by Yu Zhao <[email protected]>Quoted 17 lines
--- include/linux/pgtable.h | 13 +++++++++++++ 1 file changed, 13 insertions(+)include/linux/pgtable.h +1
-0@@ -13,6 +13,7 @@#include <linux/errno.h> #include <asm-generic/pgtable_uffd.h> #include <linux/page_table_check.h> +#include <linux/sizes.h> #if 5 - defined(__PAGETABLE_P4D_FOLDED) - defined(__PAGETABLE_PUD_FOLDED) - \ defined(__PAGETABLE_PMD_FOLDED) !=CONFIG_PGTABLE_LEVELS include/linux/pgtable.h static inline bool arch_has_hw_pte_young(void) · line 343
Quoted 7 lines
include/linux/pgtable.h +4
-0@@ -336,6 +337,18 @@ static inline bool arch_has_hw_pte_young(void) }#endif +#ifndef arch_wants_pte_order +/* +* Returns preferred folio order for pte-mapped memory. Must be in range [0, +* PMD_SHIFT-PAGE_SHIFT) and must not be order-1 since THP requires large folios The warning is helpful.
Quoted 11 lines
include/linux/pgtable.h +8
-0+* to be at least order-2. +*/ +static inline int arch_wants_pte_order (struct vm_area_struct *vma) +{ +return ilog2 (SZ_64K >>PAGE_SHIFT ); +} +#endif +#ifndef __HAVE_ARCH_PTEP_GET_AND_CLEAR static inline pte_t ptep_get_and_clear(struct mm_struct *mm, unsigned long address,
Re: [PATCH v2 3/5] mm: Default implementation of arch_wants_pte_order()
Quoted 29 lines · Ryan Roberts
Quoted 15 lines · Yu Zhao
Quoted 6 lines · Ryan Roberts
arch_wants_pte_order()can be overridden by the arch to return the preferred folio order for pte-mapped memory. This is useful as some architectures (e.g. arm64) can coalesce TLB entries when the physical memory is suitably contiguous.Quoted 7 lines
The first user for this hint will be
FLEXIBLE_THP, which aims to allocate large folios for anonymous memory to reduce page faults and other per-page operation costs.Here we add the default implementation of the function, used when the architecture does not define it, which returns the order corresponding to 64K.
Quoted 11 lines
I don't really mind a non-zero default value. But people would ask why non-zero and why 64KB. Probably you could argue this is the large size all known archs support if they have TLB coalescing. For x86, AMD CPUs would want to override this. I'll leave it to Fengwei to decide whether Intel wants a different default value.>
Also I don't like the vma parameter because it makesarch_wants_pte_order()a mix of hw preference and vma policy. From my POV, the function should be only about the former; the latter should be decided by arch-independent MM code. However, I can live with it if ARM MM people think this is really what you want. ATM, I'm skeptical they do.
Quoted 5 lines
Here's the big picture for what I'm tryng to achieve:
- In the common case, I'd like all programs to get a performance bump by
automatically and transparently using large anon folios - so no explicit requirement on the process to opt-in.
We all agree on this :)
Quoted 6 lines
- On arm64, in the above case, I'd like the preferred folio size to be 64K;
from the (admittedly limitted) testing I've done that's about where the performance knee is and it doesn't appear to increase the memory wastage very much. It also has the benefits that for 4K base pages this is the contpte size (order-4) so I can take full benefit of contpte mappings transparently to the process. And for 16K this is the HPA size (order-2).
My highest priority is to get 16KB proven first because it would benefit both client and server devices. So it may be different from yours but I don't see any conflict.
Quoted 7 lines
- On arm64 when the process has marked the VMA for THP (or when
transparent_hugepage=always) but the VMA does not meet the requirements for a PMD-sized mapping (or we failed to allocate, ...) then I'd like to map using contpte. For 4K base pages this is 64K (order-4), for 16K this is 2M (order-7) and for 64K this is 2M (order-5). The 64K base page case is very important since the PMD size for that base page is 512MB which is almost impossible to allocate in practice.
Which case (server or client) are you focusing on here? For our client devices, I can confidently say that 64KB has to be after 16KB, if it happens at all. For servers in general, I don't know of any major memory-intensive workloads that are not THP-aware, i.e., I don't think "VMA does not meet the requirements" is a concern.
Re: [PATCH v2 3/5] mm: Default implementation of arch_wants_pte_order()
Quoted 45 lines · Yu Zhao
Quoted 29 lines · Ryan Roberts
Quoted 15 lines · Yu Zhao
Quoted 6 lines · Ryan Roberts
arch_wants_pte_order()can be overridden by the arch to return the preferred folio order for pte-mapped memory. This is useful as some architectures (e.g. arm64) can coalesce TLB entries when the physical memory is suitably contiguous.Quoted 7 lines
The first user for this hint will be
FLEXIBLE_THP, which aims to allocate large folios for anonymous memory to reduce page faults and other per-page operation costs.Here we add the default implementation of the function, used when the architecture does not define it, which returns the order corresponding to 64K.
Quoted 11 lines
I don't really mind a non-zero default value. But people would ask why non-zero and why 64KB. Probably you could argue this is the large size all known archs support if they have TLB coalescing. For x86, AMD CPUs would want to override this. I'll leave it to Fengwei to decide whether Intel wants a different default value.>
Also I don't like the vma parameter because it makesarch_wants_pte_order()a mix of hw preference and vma policy. From my POV, the function should be only about the former; the latter should be decided by arch-independent MM code. However, I can live with it if ARM MM people think this is really what you want. ATM, I'm skeptical they do.Quoted 5 lines
Here's the big picture for what I'm tryng to achieve:
- In the common case, I'd like all programs to get a performance bump by
automatically and transparently using large anon folios - so no explicit requirement on the process to opt-in.We all agree on this :)
Quoted 6 lines
- On arm64, in the above case, I'd like the preferred folio size to be 64K;
from the (admittedly limitted) testing I've done that's about where the performance knee is and it doesn't appear to increase the memory wastage very much. It also has the benefits that for 4K base pages this is the contpte size (order-4) so I can take full benefit of contpte mappings transparently to the process. And for 16K this is the HPA size (order-2).
Quoted 3 lines
My highest priority is to get 16KB proven first because it would benefit both client and server devices. So it may be different from yours but I don't see any conflict.
Do you mean 16K folios on a 4K base page system, or large folios on a 16K base page system? I thought your focus was on speeding up 4K base page client systems but this statement has got me wondering?
Quoted 9 lines
Quoted 8 lines
- On arm64 when the process has marked the VMA for THP (or when
transparent_hugepage=always) but the VMA does not meet the requirements for a PMD-sized mapping (or we failed to allocate, ...) then I'd like to map using contpte. For 4K base pages this is 64K (order-4), for 16K this is 2M (order-7) and for 64K this is 2M (order-5). The 64K base page case is very important since the PMD size for that base page is 512MB which is almost impossible to allocate in practice.
Quoted 5 lines
Which case (server or client) are you focusing on here? For our client devices, I can confidently say that 64KB has to be after 16KB, if it happens at all. For servers in general, I don't know of any major memory-intensive workloads that are not THP-aware, i.e., I don't think "VMA does not meet the requirements" is a concern.
For the 64K base page case, the focus is server. The problem reported by our partner is that the 512M huge page size is too big to reliably allocate and so the fauls always fall back to 64K base pages in practice. I would also speculate (happy to be proved wrong) that there are many THP-aware workloads that assume the THP size is 2M. In this case, their VMAs may well be too small to fit a 512M huge page when running on 64K base page system.
But the TL;DR is that Arm has a partner for which enabling 2M THP on a 64K base page system is a very real requirement. Our intent is that this will be the mechanism we use to enable it.
Re: [PATCH v2 3/5] mm: Default implementation of arch_wants_pte_order()
Quoted 47 lines · Ryan Roberts
Quoted 45 lines · Yu Zhao
Quoted 29 lines · Ryan Roberts
Quoted 15 lines · Yu Zhao
Quoted 6 lines · Ryan Roberts
arch_wants_pte_order()can be overridden by the arch to return the preferred folio order for pte-mapped memory. This is useful as some architectures (e.g. arm64) can coalesce TLB entries when the physical memory is suitably contiguous.Quoted 7 lines
The first user for this hint will be
FLEXIBLE_THP, which aims to allocate large folios for anonymous memory to reduce page faults and other per-page operation costs.Here we add the default implementation of the function, used when the architecture does not define it, which returns the order corresponding to 64K.
Quoted 11 lines
I don't really mind a non-zero default value. But people would ask why non-zero and why 64KB. Probably you could argue this is the large size all known archs support if they have TLB coalescing. For x86, AMD CPUs would want to override this. I'll leave it to Fengwei to decide whether Intel wants a different default value.>
Also I don't like the vma parameter because it makesarch_wants_pte_order()a mix of hw preference and vma policy. From my POV, the function should be only about the former; the latter should be decided by arch-independent MM code. However, I can live with it if ARM MM people think this is really what you want. ATM, I'm skeptical they do.Quoted 5 lines
Here's the big picture for what I'm tryng to achieve:
- In the common case, I'd like all programs to get a performance bump by
automatically and transparently using large anon folios - so no explicit requirement on the process to opt-in.We all agree on this :)
Quoted 6 lines
- On arm64, in the above case, I'd like the preferred folio size to be 64K;
from the (admittedly limitted) testing I've done that's about where the performance knee is and it doesn't appear to increase the memory wastage very much. It also has the benefits that for 4K base pages this is the contpte size (order-4) so I can take full benefit of contpte mappings transparently to the process. And for 16K this is the HPA size (order-2).
Quoted 5 lines
Quoted 3 lines
My highest priority is to get 16KB proven first because it would benefit both client and server devices. So it may be different from yours but I don't see any conflict.
Do you mean 16K folios on a 4K base page system
Yes.
Quoted 3 lines
or large folios on a 16K base
page system? I thought your focus was on speeding up 4K base page client systems but this statement has got me wondering?
Sorry, I should have said 4x4KB.
Quoted 14 lines
Quoted 8 lines
Quoted 7 lines
- On arm64 when the process has marked the VMA for THP (or when
transparent_hugepage=always) but the VMA does not meet the requirements for a PMD-sized mapping (or we failed to allocate, ...) then I'd like to map using contpte. For 4K base pages this is 64K (order-4), for 16K this is 2M (order-7) and for 64K this is 2M (order-5). The 64K base page case is very important since the PMD size for that base page is 512MB which is almost impossible to allocate in practice.Quoted 5 lines
Which case (server or client) are you focusing on here? For our client devices, I can confidently say that 64KB has to be after 16KB, if it happens at all. For servers in general, I don't know of any major memory-intensive workloads that are not THP-aware, i.e., I don't think "VMA does not meet the requirements" is a concern.
Quoted 6 lines
For the 64K base page case, the focus is server. The problem reported by our partner is that the 512M huge page size is too big to reliably allocate and so the fauls always fall back to 64K base pages in practice. I would also speculate (happy to be proved wrong) that there are many THP-aware workloads that assume the THP size is 2M. In this case, their VMAs may well be too small to fit a 512M huge page when running on 64K base page system.
Interesting. When you have something ready to share, I might be able to try it on our ARM servers as well.
Quoted 3 lines
But the TL;DR is that Arm has a partner for which enabling 2M THP on a 64K base page system is a very real requirement. Our intent is that this will be the mechanism we use to enable it.
Yes, contpte makes more sense for what you described. It'd fit in a lot better in the hugetlb case, but I guess your partner uses anon.
Re: [PATCH v2 3/5] mm: Default implementation of arch_wants_pte_order()
Quoted 54 lines · Yu Zhao
Quoted 47 lines · Ryan Roberts
Quoted 45 lines · Yu Zhao
Quoted 29 lines · Ryan Roberts
Quoted 15 lines · Yu Zhao
Quoted 6 lines · Ryan Roberts
arch_wants_pte_order()can be overridden by the arch to return the preferred folio order for pte-mapped memory. This is useful as some architectures (e.g. arm64) can coalesce TLB entries when the physical memory is suitably contiguous.Quoted 7 lines
The first user for this hint will be
FLEXIBLE_THP, which aims to allocate large folios for anonymous memory to reduce page faults and other per-page operation costs.Here we add the default implementation of the function, used when the architecture does not define it, which returns the order corresponding to 64K.
Quoted 11 lines
I don't really mind a non-zero default value. But people would ask why non-zero and why 64KB. Probably you could argue this is the large size all known archs support if they have TLB coalescing. For x86, AMD CPUs would want to override this. I'll leave it to Fengwei to decide whether Intel wants a different default value.>
Also I don't like the vma parameter because it makesarch_wants_pte_order()a mix of hw preference and vma policy. From my POV, the function should be only about the former; the latter should be decided by arch-independent MM code. However, I can live with it if ARM MM people think this is really what you want. ATM, I'm skeptical they do.Quoted 5 lines
Here's the big picture for what I'm tryng to achieve:
- In the common case, I'd like all programs to get a performance bump by
automatically and transparently using large anon folios - so no explicit requirement on the process to opt-in.We all agree on this :)
Quoted 6 lines
- On arm64, in the above case, I'd like the preferred folio size to be 64K;
from the (admittedly limitted) testing I've done that's about where the performance knee is and it doesn't appear to increase the memory wastage very much. It also has the benefits that for 4K base pages this is the contpte size (order-4) so I can take full benefit of contpte mappings transparently to the process. And for 16K this is the HPA size (order-2).Quoted 5 lines
Quoted 3 lines
My highest priority is to get 16KB proven first because it would benefit both client and server devices. So it may be different from yours but I don't see any conflict.
Do you mean 16K folios on a 4K base page system
Quoted 7 lines
Yes.
Quoted 3 lines
or large folios on a 16K base
page system? I thought your focus was on speeding up 4K base page client systems but this statement has got me wondering?Sorry, I should have said 4x4KB.
OK. Be aware that a number of Arm CPUs that support HPA don't have it enabled by default (or at least don't have it enabled in the mode that you would want it to see best performance with large anon folios). You would need EL3 access to reconfigure it.
Quoted 22 lines
Quoted 15 lines
Quoted 9 lines
Quoted 8 lines
- On arm64 when the process has marked the VMA for THP (or when
transparent_hugepage=always) but the VMA does not meet the requirements for a PMD-sized mapping (or we failed to allocate, ...) then I'd like to map using contpte. For 4K base pages this is 64K (order-4), for 16K this is 2M (order-7) and for 64K this is 2M (order-5). The 64K base page case is very important since the PMD size for that base page is 512MB which is almost impossible to allocate in practice.Quoted 5 lines
Which case (server or client) are you focusing on here? For our client devices, I can confidently say that 64KB has to be after 16KB, if it happens at all. For servers in general, I don't know of any major memory-intensive workloads that are not THP-aware, i.e., I don't think "VMA does not meet the requirements" is a concern.
Quoted 6 lines
For the 64K base page case, the focus is server. The problem reported by our partner is that the 512M huge page size is too big to reliably allocate and so the fauls always fall back to 64K base pages in practice. I would also speculate (happy to be proved wrong) that there are many THP-aware workloads that assume the THP size is 2M. In this case, their VMAs may well be too small to fit a 512M huge page when running on 64K base page system.
Quoted 2 lines
Interesting. When you have something ready to share, I might be able to try it on our ARM servers as well.
That would be really helpful. I'm currently updating my branch that collates everything to reflect the review comments in this patch set and the contpte patch set. I'll share it in a couple of weeks.
Quoted 7 lines
Quoted 4 lines
But the TL;DR is that Arm has a partner for which enabling 2M THP on a 64K base page system is a very real requirement. Our intent is that this will be the mechanism we use to enable it.
Yes, contpte makes more sense for what you described. It'd fit in a lot better in the hugetlb case, but I guess your partner uses anon.
arm64 already supports contpte for hugetlb, but they need it to work with anon memory using THP.
Re: [PATCH v2 3/5] mm: Default implementation of arch_wants_pte_order()
In reply to Yu ZhaoQuoted 8 lines · Yu Zhao
Quoted 7 lines
- On arm64 when the process has marked the VMA for THP (or when
transparent_hugepage=always) but the VMA does not meet the requirements for a PMD-sized mapping (or we failed to allocate, ...) then I'd like to map using contpte. For 4K base pages this is 64K (order-4), for 16K this is 2M (order-7) and for 64K this is 2M (order-5). The 64K base page case is very important since the PMD size for that base page is 512MB which is almost impossible to allocate in practice.
Quoted 5 lines
Which case (server or client) are you focusing on here? For our client devices, I can confidently say that 64KB has to be after 16KB, if it happens at all. For servers in general, I don't know of any major memory-intensive workloads that are not THP-aware, i.e., I don't think "VMA does not meet the requirements" is a concern.
It sounds like you've done some measurements, and I'd like to understand those a bit better. There are a number of factors involved:
- A larger page size shrinks the length of the LRU list, so systems which see heavy LRU lock contention benefit more
- A larger page size has more internal fragmentation, so we run out of memory and have to do reclaim more often (and maybe workload which used to fit in DRAM now do not)
(probably others; i'm not at 100% right now)
I think concerns about "allocating lots of order-2 folios makes it harder to allocate order-4 folios" are _probably_ not warranted (without data to prove otherwise). All anonymous memory is movable, so our compaction code should be able to create larger order folios.
Re: [PATCH v2 3/5] mm: Default implementation of arch_wants_pte_order()
Quoted 9 lines · Matthew Wilcox
Quoted 8 lines · Yu Zhao
Quoted 7 lines
- On arm64 when the process has marked the VMA for THP (or when
transparent_hugepage=always) but the VMA does not meet the requirements for a PMD-sized mapping (or we failed to allocate, ...) then I'd like to map using contpte. For 4K base pages this is 64K (order-4), for 16K this is 2M (order-7) and for 64K this is 2M (order-5). The 64K base page case is very important since the PMD size for that base page is 512MB which is almost impossible to allocate in practice.
Quoted 8 lines
Quoted 5 lines
Which case (server or client) are you focusing on here? For our client devices, I can confidently say that 64KB has to be after 16KB, if it happens at all. For servers in general, I don't know of any major memory-intensive workloads that are not THP-aware, i.e., I don't think "VMA does not meet the requirements" is a concern.
It sounds like you've done some measurements, and I'd like to understand those a bit better. There are a number of factors involved:
I'm not sure if that's a question to me or Yu? I haven't personally done any measurements for the 64K base page case. But Arm has a partner that is pushing for this. I'm hoping to see some test results from them posted publicly in the coming weeks. See [1] for more explanation on the rationale.
[1]
https://lore.kernel.org/linux-mm/[email protected]/T/#m8a7c4b71f94224ec3fe6d0a407f48d74c789ba4f
Quoted 13 lines
- A larger page size shrinks the length of the LRU list, so systems which see heavy LRU lock contention benefit more
- A larger page size has more internal fragmentation, so we run out of memory and have to do reclaim more often (and maybe workload which used to fit in DRAM now do not)
(probably others; i'm not at 100% right now)I think concerns about "allocating lots of order-2 folios makes it harder to allocate order-4 folios" are _probably_ not warranted (without data to prove otherwise). All anonymous memory is movable, so our compaction code should be able to create larger order folios.
Re: [PATCH v2 3/5] mm: Default implementation of arch_wants_pte_order()
In reply to Ryan RobertsQuoted 33 lines · Ryan Roberts
arch_wants_pte_order()can be overridden by the arch to return the preferred folio order for pte-mapped memory. This is useful as some architectures (e.g. arm64) can coalesce TLB entries when the physical memory is suitably contiguous.The first user for this hint will be
FLEXIBLE_THP, which aims to allocate large folios for anonymous memory to reduce page faults and other per-page operation costs.Here we add the default implementation of the function, used when the architecture does not define it, which returns the order corresponding to 64K.
Signed-off-by Ryan Roberts <[email protected]>--- include/linux/pgtable.h | 13 +++++++++++++ 1 file changed, 13 insertions(+)include/linux/pgtable.h +1
-0@@ -13,6 +13,7 @@#include <linux/errno.h> #include <asm-generic/pgtable_uffd.h> #include <linux/page_table_check.h> +#include <linux/sizes.h> #if 5 - defined(__PAGETABLE_P4D_FOLDED) - defined(__PAGETABLE_PUD_FOLDED) - \ defined(__PAGETABLE_PMD_FOLDED) !=CONFIG_PGTABLE_LEVELS @@ -336,6 +337,18 @@ static inline bool arch_has_hw_pte_young(void) }#endif
include/linux/pgtable.h static inline bool arch_has_hw_pte_young(void) · line 348
Quoted 10 lines
include/linux/pgtable.h +9
-0+#ifndef arch_wants_pte_order +/* +* Returns preferred folio order for pte-mapped memory. Must be in range [0, +* PMD_SHIFT-PAGE_SHIFT) and must not be order-1 since THP requires large folios +* to be at least order-2. +*/ +static inline int arch_wants_pte_order (struct vm_area_struct *vma) +{ +return ilog2 (SZ_64K >>PAGE_SHIFT );
Default value which is not related with any silicon may be: PAGE_ALLOC_COSTLY_ORDER?
Also, current pcp list support cache page with order 0...PAGE_ALLOC_COSTLY_ORDER, 9. If the pcp could cover the page, the pressure to zone lock will be reduced by pcp.
Regards
Yin, Fengwei
Quoted 6 lines
include/linux/pgtable.h +3
-0+} +#endif +#ifndef __HAVE_ARCH_PTEP_GET_AND_CLEAR static inline pte_t ptep_get_and_clear(struct mm_struct *mm, unsigned long address,
Re: [PATCH v2 3/5] mm: Default implementation of arch_wants_pte_order()
Quoted 49 lines · Fengwei
Quoted 33 lines · Ryan Roberts
arch_wants_pte_order()can be overridden by the arch to return the preferred folio order for pte-mapped memory. This is useful as some architectures (e.g. arm64) can coalesce TLB entries when the physical memory is suitably contiguous.The first user for this hint will be
FLEXIBLE_THP, which aims to allocate large folios for anonymous memory to reduce page faults and other per-page operation costs.Here we add the default implementation of the function, used when the architecture does not define it, which returns the order corresponding to 64K.
Signed-off-by Ryan Roberts <[email protected]>--- include/linux/pgtable.h | 13 +++++++++++++ 1 file changed, 13 insertions(+)include/linux/pgtable.h +1
-0@@ -13,6 +13,7 @@#include <linux/errno.h> #include <asm-generic/pgtable_uffd.h> #include <linux/page_table_check.h> +#include <linux/sizes.h> #if 5 - defined(__PAGETABLE_P4D_FOLDED) - defined(__PAGETABLE_PUD_FOLDED) - \ defined(__PAGETABLE_PMD_FOLDED) !=CONFIG_PGTABLE_LEVELS @@ -336,6 +337,18 @@ static inline bool arch_has_hw_pte_young(void) }#endif include/linux/pgtable.h static inline bool arch_has_hw_pte_young(void) · line 348
Quoted 10 lines
include/linux/pgtable.h +9
-0+#ifndef arch_wants_pte_order +/* +* Returns preferred folio order for pte-mapped memory. Must be in range [0, +* PMD_SHIFT-PAGE_SHIFT) and must not be order-1 since THP requires large folios +* to be at least order-2. +*/ +static inline int arch_wants_pte_order (struct vm_area_struct *vma) +{ +return ilog2 (SZ_64K >>PAGE_SHIFT );Default value which is not related with any silicon may be:
PAGE_ALLOC_COSTLY_ORDER?
Quoted 2 lines
Also, current pcp list support cache page with order 0...PAGE_ALLOC_COSTLY_ORDER, 9. If the pcp could cover the page, the pressure to zone lock will be reduced by pcp.
The value of PAGE_ALLOC_COSTLY_ORDER is reasonable but again it's a s/w policy not a h/w preference. Besides, I don't think we can include mmzone.h in pgtable.h.
Re: [PATCH v2 3/5] mm: Default implementation of arch_wants_pte_order()
Quoted 51 lines · Yu Zhao
Quoted 49 lines · Fengwei
Quoted 33 lines · Ryan Roberts
arch_wants_pte_order()can be overridden by the arch to return the preferred folio order for pte-mapped memory. This is useful as some architectures (e.g. arm64) can coalesce TLB entries when the physical memory is suitably contiguous.The first user for this hint will be
FLEXIBLE_THP, which aims to allocate large folios for anonymous memory to reduce page faults and other per-page operation costs.Here we add the default implementation of the function, used when the architecture does not define it, which returns the order corresponding to 64K.
Signed-off-by Ryan Roberts <[email protected]>--- include/linux/pgtable.h | 13 +++++++++++++ 1 file changed, 13 insertions(+)include/linux/pgtable.h +1
-0@@ -13,6 +13,7 @@#include <linux/errno.h> #include <asm-generic/pgtable_uffd.h> #include <linux/page_table_check.h> +#include <linux/sizes.h> #if 5 - defined(__PAGETABLE_P4D_FOLDED) - defined(__PAGETABLE_PUD_FOLDED) - \ defined(__PAGETABLE_PMD_FOLDED) !=CONFIG_PGTABLE_LEVELS @@ -336,6 +337,18 @@ static inline bool arch_has_hw_pte_young(void) }#endif include/linux/pgtable.h static inline bool arch_has_hw_pte_young(void) · line 348
Quoted 10 lines
include/linux/pgtable.h +9
-0+#ifndef arch_wants_pte_order +/* +* Returns preferred folio order for pte-mapped memory. Must be in range [0, +* PMD_SHIFT-PAGE_SHIFT) and must not be order-1 since THP requires large folios +* to be at least order-2. +*/ +static inline int arch_wants_pte_order (struct vm_area_struct *vma) +{ +return ilog2 (SZ_64K >>PAGE_SHIFT );Default value which is not related with any silicon may be:
PAGE_ALLOC_COSTLY_ORDER?
Quoted 6 lines
Quoted 2 lines
Also, current pcp list support cache page with order 0...PAGE_ALLOC_COSTLY_ORDER, 9. If the pcp could cover the page, the pressure to zone lock will be reduced by pcp.
The value of
PAGE_ALLOC_COSTLY_ORDERis reasonable but again it's a s/w policy not a h/w preference. Besides, I don't think we can includemmzone.hinpgtable.h.
I think we can make a compromise:
1. change the default implementation of arch_has_hw_pte_young() to return 0, and
2. in memory.c, we can try PAGE_ALLOC_COSTLY_ORDER for archs that don't override arch_has_hw_pte_young(), or if its return value is too large to fit.
This should also take care of the regression, right?
Re: [PATCH v2 3/5] mm: Default implementation of arch_wants_pte_order()
Quoted 59 lines · Yu Zhao
Quoted 51 lines · Yu Zhao
Quoted 49 lines · Fengwei
Quoted 33 lines · Ryan Roberts
arch_wants_pte_order()can be overridden by the arch to return the preferred folio order for pte-mapped memory. This is useful as some architectures (e.g. arm64) can coalesce TLB entries when the physical memory is suitably contiguous.The first user for this hint will be
FLEXIBLE_THP, which aims to allocate large folios for anonymous memory to reduce page faults and other per-page operation costs.Here we add the default implementation of the function, used when the architecture does not define it, which returns the order corresponding to 64K.
Signed-off-by Ryan Roberts <[email protected]>--- include/linux/pgtable.h | 13 +++++++++++++ 1 file changed, 13 insertions(+)include/linux/pgtable.h +1
-0@@ -13,6 +13,7 @@#include <linux/errno.h> #include <asm-generic/pgtable_uffd.h> #include <linux/page_table_check.h> +#include <linux/sizes.h> #if 5 - defined(__PAGETABLE_P4D_FOLDED) - defined(__PAGETABLE_PUD_FOLDED) - \ defined(__PAGETABLE_PMD_FOLDED) !=CONFIG_PGTABLE_LEVELS @@ -336,6 +337,18 @@ static inline bool arch_has_hw_pte_young(void) }#endif include/linux/pgtable.h static inline bool arch_has_hw_pte_young(void) · line 348
Quoted 10 lines
include/linux/pgtable.h +9
-0+#ifndef arch_wants_pte_order +/* +* Returns preferred folio order for pte-mapped memory. Must be in range [0, +* PMD_SHIFT-PAGE_SHIFT) and must not be order-1 since THP requires large folios +* to be at least order-2. +*/ +static inline int arch_wants_pte_order (struct vm_area_struct *vma) +{ +return ilog2 (SZ_64K >>PAGE_SHIFT );Default value which is not related with any silicon may be:
PAGE_ALLOC_COSTLY_ORDER?Quoted 6 lines
Quoted 2 lines
Also, current pcp list support cache page with order 0...PAGE_ALLOC_COSTLY_ORDER, 9. If the pcp could cover the page, the pressure to zone lock will be reduced by pcp.
The value of
PAGE_ALLOC_COSTLY_ORDERis reasonable but again it's a s/w policy not a h/w preference. Besides, I don't think we can includemmzone.hinpgtable.h.
Quoted 5 lines
I think we can make a compromise:
1. change the default implementation ofarch_has_hw_pte_young()to return 0, and
2. inmemory.c, we can tryPAGE_ALLOC_COSTLY_ORDERfor archs that don't overridearch_has_hw_pte_young(), or if its return value is too large to fit.
Do you mean arch_wants_pte_order()? Yes. This looks good to me. Thanks.
Regards
Yin, Fengwei
Quoted 1 line
This should also take care of the regression, right?
Re: [PATCH v2 3/5] mm: Default implementation of arch_wants_pte_order()
Quoted 63 lines · Fengwei
Quoted 59 lines · Yu Zhao
Quoted 57 lines · Yu Zhao
Quoted 49 lines · Fengwei
Quoted 33 lines · Ryan Roberts
arch_wants_pte_order()can be overridden by the arch to return the preferred folio order for pte-mapped memory. This is useful as some architectures (e.g. arm64) can coalesce TLB entries when the physical memory is suitably contiguous.The first user for this hint will be
FLEXIBLE_THP, which aims to allocate large folios for anonymous memory to reduce page faults and other per-page operation costs.Here we add the default implementation of the function, used when the architecture does not define it, which returns the order corresponding to 64K.
Signed-off-by Ryan Roberts <[email protected]>--- include/linux/pgtable.h | 13 +++++++++++++ 1 file changed, 13 insertions(+)include/linux/pgtable.h +1
-0@@ -13,6 +13,7 @@#include <linux/errno.h> #include <asm-generic/pgtable_uffd.h> #include <linux/page_table_check.h> +#include <linux/sizes.h> #if 5 - defined(__PAGETABLE_P4D_FOLDED) - defined(__PAGETABLE_PUD_FOLDED) - \ defined(__PAGETABLE_PMD_FOLDED) !=CONFIG_PGTABLE_LEVELS @@ -336,6 +337,18 @@ static inline bool arch_has_hw_pte_young(void) }#endif include/linux/pgtable.h static inline bool arch_has_hw_pte_young(void) · line 348
Quoted 10 lines
include/linux/pgtable.h +9
-0+#ifndef arch_wants_pte_order +/* +* Returns preferred folio order for pte-mapped memory. Must be in range [0, +* PMD_SHIFT-PAGE_SHIFT) and must not be order-1 since THP requires large folios +* to be at least order-2. +*/ +static inline int arch_wants_pte_order (struct vm_area_struct *vma) +{ +return ilog2 (SZ_64K >>PAGE_SHIFT );Default value which is not related with any silicon may be:
PAGE_ALLOC_COSTLY_ORDER?Quoted 2 lines
Also, current pcp list support cache page with order 0...PAGE_ALLOC_COSTLY_ORDER, 9. If the pcp could cover the page, the pressure to zone lock will be reduced by pcp.
The value of
PAGE_ALLOC_COSTLY_ORDERis reasonable but again it's a s/w policy not a h/w preference. Besides, I don't think we can includemmzone.hinpgtable.h.
Quoted 6 lines
Quoted 5 lines
I think we can make a compromise:
1. change the default implementation ofarch_has_hw_pte_young()to return 0, and
2. inmemory.c, we can tryPAGE_ALLOC_COSTLY_ORDERfor archs that don't overridearch_has_hw_pte_young(), or if its return value is too large to fit.Do you mean
arch_wants_pte_order()? Yes. This looks good to me. Thanks.
Sorry, copied the wrong function from above and pasted without looking...
Re: [PATCH v2 3/5] mm: Default implementation of arch_wants_pte_order()
In reply to Yu ZhaoQuoted 59 lines · Yu Zhao
Quoted 51 lines · Yu Zhao
Quoted 49 lines · Fengwei
Quoted 33 lines · Ryan Roberts
arch_wants_pte_order()can be overridden by the arch to return the preferred folio order for pte-mapped memory. This is useful as some architectures (e.g. arm64) can coalesce TLB entries when the physical memory is suitably contiguous.The first user for this hint will be
FLEXIBLE_THP, which aims to allocate large folios for anonymous memory to reduce page faults and other per-page operation costs.Here we add the default implementation of the function, used when the architecture does not define it, which returns the order corresponding to 64K.
Signed-off-by Ryan Roberts <[email protected]>--- include/linux/pgtable.h | 13 +++++++++++++ 1 file changed, 13 insertions(+)include/linux/pgtable.h +1
-0@@ -13,6 +13,7 @@#include <linux/errno.h> #include <asm-generic/pgtable_uffd.h> #include <linux/page_table_check.h> +#include <linux/sizes.h> #if 5 - defined(__PAGETABLE_P4D_FOLDED) - defined(__PAGETABLE_PUD_FOLDED) - \ defined(__PAGETABLE_PMD_FOLDED) !=CONFIG_PGTABLE_LEVELS @@ -336,6 +337,18 @@ static inline bool arch_has_hw_pte_young(void) }#endif include/linux/pgtable.h static inline bool arch_has_hw_pte_young(void) · line 348
Quoted 10 lines
include/linux/pgtable.h +9
-0+#ifndef arch_wants_pte_order +/* +* Returns preferred folio order for pte-mapped memory. Must be in range [0, +* PMD_SHIFT-PAGE_SHIFT) and must not be order-1 since THP requires large folios +* to be at least order-2. +*/ +static inline int arch_wants_pte_order (struct vm_area_struct *vma) +{ +return ilog2 (SZ_64K >>PAGE_SHIFT );Default value which is not related with any silicon may be:
PAGE_ALLOC_COSTLY_ORDER?Quoted 6 lines
Quoted 2 lines
Also, current pcp list support cache page with order 0...PAGE_ALLOC_COSTLY_ORDER, 9. If the pcp could cover the page, the pressure to zone lock will be reduced by pcp.
The value of
PAGE_ALLOC_COSTLY_ORDERis reasonable but again it's a s/w policy not a h/w preference. Besides, I don't think we can includemmzone.hinpgtable.h.
Quoted 6 lines
I think we can make a compromise:
1. change the default implementation ofarch_has_hw_pte_young()to return 0, and
2. inmemory.c, we can tryPAGE_ALLOC_COSTLY_ORDERfor archs that don't overridearch_has_hw_pte_young(), or if its return value is too large to fit.
This should also take care of the regression, right?
I think you are suggesting that we use 0 as a sentinel which we then translate to PAGE_ALLOC_COSTLY_ORDER? I already have a max_anon_folio_order() function in memory.c (actually it is currently a macro defined as arch_wants_pte_order()).
So it would become (I'll talk about the vma concern separately in the thread where you raised it):
static inline int max_anon_folio_order(struct vm_area_struct *vma)
{
int order = arch_wants_pte_order(vma);
return order ? order : PAGE_ALLOC_COSTLY_ORDER;
}Correct?
I don't see how it fixes the regression (assume you're talking about Speedometer) though? On arm64 arch_wants_pte_order() will still be returning order-4.
Re: [PATCH v2 3/5] mm: Default implementation of arch_wants_pte_order()
Quoted 74 lines · Ryan Roberts
Quoted 59 lines · Yu Zhao
Quoted 51 lines · Yu Zhao
Quoted 49 lines · Fengwei
Quoted 33 lines · Ryan Roberts
arch_wants_pte_order()can be overridden by the arch to return the preferred folio order for pte-mapped memory. This is useful as some architectures (e.g. arm64) can coalesce TLB entries when the physical memory is suitably contiguous.The first user for this hint will be
FLEXIBLE_THP, which aims to allocate large folios for anonymous memory to reduce page faults and other per-page operation costs.Here we add the default implementation of the function, used when the architecture does not define it, which returns the order corresponding to 64K.
Signed-off-by Ryan Roberts <[email protected]>--- include/linux/pgtable.h | 13 +++++++++++++ 1 file changed, 13 insertions(+)include/linux/pgtable.h +1
-0@@ -13,6 +13,7 @@#include <linux/errno.h> #include <asm-generic/pgtable_uffd.h> #include <linux/page_table_check.h> +#include <linux/sizes.h> #if 5 - defined(__PAGETABLE_P4D_FOLDED) - defined(__PAGETABLE_PUD_FOLDED) - \ defined(__PAGETABLE_PMD_FOLDED) !=CONFIG_PGTABLE_LEVELS @@ -336,6 +337,18 @@ static inline bool arch_has_hw_pte_young(void) }#endif include/linux/pgtable.h static inline bool arch_has_hw_pte_young(void) · line 348
Quoted 10 lines
include/linux/pgtable.h +9
-0+#ifndef arch_wants_pte_order +/* +* Returns preferred folio order for pte-mapped memory. Must be in range [0, +* PMD_SHIFT-PAGE_SHIFT) and must not be order-1 since THP requires large folios +* to be at least order-2. +*/ +static inline int arch_wants_pte_order (struct vm_area_struct *vma) +{ +return ilog2 (SZ_64K >>PAGE_SHIFT );Default value which is not related with any silicon may be:
PAGE_ALLOC_COSTLY_ORDER?Quoted 6 lines
Quoted 2 lines
Also, current pcp list support cache page with order 0...PAGE_ALLOC_COSTLY_ORDER, 9. If the pcp could cover the page, the pressure to zone lock will be reduced by pcp.
The value of
PAGE_ALLOC_COSTLY_ORDERis reasonable but again it's a s/w policy not a h/w preference. Besides, I don't think we can includemmzone.hinpgtable.h.Quoted 6 lines
I think we can make a compromise:
1. change the default implementation ofarch_has_hw_pte_young()to return 0, and
2. inmemory.c, we can tryPAGE_ALLOC_COSTLY_ORDERfor archs that don't overridearch_has_hw_pte_young(), or if its return value is too large to fit.
This should also take care of the regression, right?I think you are suggesting that we use 0 as a sentinel which we then translate to
PAGE_ALLOC_COSTLY_ORDER? I already have amax_anon_folio_order()function inmemory.c(actually it is currently a macro defined asarch_wants_pte_order()).So it would become (I'll talk about the vma concern separately in the thread where you raised it):
Quoted 8 lines
static inline int
max_anon_folio_order(struct vm_area_struct *vma){ int order = arch_wants_pte_order(vma); return order ? order : PAGE_ALLOC_COSTLY_ORDER; }Correct?
Actually, I'm not sure its a good idea to default to a fixed order. If running on an arch with big base pages (e.g. powerpc with 64K pages?), that will soon add up to a big chunk of memory, which could be wasteful?
PAGE_ALLOC_COSTLY_ORDER = 3 so with 64K base page, that 512K. Is that a concern? Wouldn't it be better to define this as an absolute size? Or even the min of PAGE_ALLOC_COSTLY_ORDER and an absolute size?
Quoted 5 lines
I don't see how it fixes the regression (assume you're talking about Speedometer) though? On arm64
arch_wants_pte_order()will still be returning order-4.
Re: [PATCH v2 3/5] mm: Default implementation of arch_wants_pte_order()
Quoted 85 lines · Ryan Roberts
Quoted 74 lines · Ryan Roberts
Quoted 59 lines · Yu Zhao
Quoted 51 lines · Yu Zhao
Quoted 49 lines · Fengwei
Quoted 33 lines · Ryan Roberts
arch_wants_pte_order()can be overridden by the arch to return the preferred folio order for pte-mapped memory. This is useful as some architectures (e.g. arm64) can coalesce TLB entries when the physical memory is suitably contiguous.The first user for this hint will be
FLEXIBLE_THP, which aims to allocate large folios for anonymous memory to reduce page faults and other per-page operation costs.Here we add the default implementation of the function, used when the architecture does not define it, which returns the order corresponding to 64K.
Signed-off-by Ryan Roberts <[email protected]>--- include/linux/pgtable.h | 13 +++++++++++++ 1 file changed, 13 insertions(+)include/linux/pgtable.h +1
-0@@ -13,6 +13,7 @@#include <linux/errno.h> #include <asm-generic/pgtable_uffd.h> #include <linux/page_table_check.h> +#include <linux/sizes.h> #if 5 - defined(__PAGETABLE_P4D_FOLDED) - defined(__PAGETABLE_PUD_FOLDED) - \ defined(__PAGETABLE_PMD_FOLDED) !=CONFIG_PGTABLE_LEVELS @@ -336,6 +337,18 @@ static inline bool arch_has_hw_pte_young(void) }#endif include/linux/pgtable.h static inline bool arch_has_hw_pte_young(void) · line 348
Quoted 10 lines
include/linux/pgtable.h +9
-0+#ifndef arch_wants_pte_order +/* +* Returns preferred folio order for pte-mapped memory. Must be in range [0, +* PMD_SHIFT-PAGE_SHIFT) and must not be order-1 since THP requires large folios +* to be at least order-2. +*/ +static inline int arch_wants_pte_order (struct vm_area_struct *vma) +{ +return ilog2 (SZ_64K >>PAGE_SHIFT );Default value which is not related with any silicon may be:
PAGE_ALLOC_COSTLY_ORDER?Quoted 6 lines
Quoted 2 lines
Also, current pcp list support cache page with order 0...PAGE_ALLOC_COSTLY_ORDER, 9. If the pcp could cover the page, the pressure to zone lock will be reduced by pcp.
The value of
PAGE_ALLOC_COSTLY_ORDERis reasonable but again it's a s/w policy not a h/w preference. Besides, I don't think we can includemmzone.hinpgtable.h.Quoted 6 lines
I think we can make a compromise:
1. change the default implementation ofarch_has_hw_pte_young()to return 0, and
2. inmemory.c, we can tryPAGE_ALLOC_COSTLY_ORDERfor archs that don't overridearch_has_hw_pte_young(), or if its return value is too large to fit.
This should also take care of the regression, right?I think you are suggesting that we use 0 as a sentinel which we then translate to
PAGE_ALLOC_COSTLY_ORDER? I already have amax_anon_folio_order()function inmemory.c(actually it is currently a macro defined asarch_wants_pte_order()).So it would become (I'll talk about the vma concern separately in the thread where you raised it):
Quoted 8 lines
static inline int
max_anon_folio_order(struct vm_area_struct *vma){ int order = arch_wants_pte_order(vma); return order ? order : PAGE_ALLOC_COSTLY_ORDER; }Correct?
Quoted 7 lines
Actually, I'm not sure its a good idea to default to a fixed order. If running on an arch with big base pages (e.g. powerpc with 64K pages?), that will soon add up to a big chunk of memory, which could be wasteful?
PAGE_ALLOC_COSTLY_ORDER= 3 so with 64K base page, that 512K. Is that a concern? Wouldn't it be better to define this as an absolute size? Or even the min ofPAGE_ALLOC_COSTLY_ORDERand an absolute size?
For my POV, not at all. POWER can use smaller page sizes if they wanted to -- I don't think they do: at least the distros I use on my POWER9 all have THP=always by default (2MB).
Re: [PATCH v2 3/5] mm: Default implementation of arch_wants_pte_order()
In reply to Ryan RobertsQuoted 82 lines · Ryan Roberts
Quoted 59 lines · Yu Zhao
Quoted 51 lines · Yu Zhao
Quoted 49 lines · Fengwei
Quoted 33 lines · Ryan Roberts
arch_wants_pte_order()can be overridden by the arch to return the preferred folio order for pte-mapped memory. This is useful as some architectures (e.g. arm64) can coalesce TLB entries when the physical memory is suitably contiguous.The first user for this hint will be
FLEXIBLE_THP, which aims to allocate large folios for anonymous memory to reduce page faults and other per-page operation costs.Here we add the default implementation of the function, used when the architecture does not define it, which returns the order corresponding to 64K.
Signed-off-by Ryan Roberts <[email protected]>--- include/linux/pgtable.h | 13 +++++++++++++ 1 file changed, 13 insertions(+)include/linux/pgtable.h +1
-0@@ -13,6 +13,7 @@#include <linux/errno.h> #include <asm-generic/pgtable_uffd.h> #include <linux/page_table_check.h> +#include <linux/sizes.h> #if 5 - defined(__PAGETABLE_P4D_FOLDED) - defined(__PAGETABLE_PUD_FOLDED) - \ defined(__PAGETABLE_PMD_FOLDED) !=CONFIG_PGTABLE_LEVELS @@ -336,6 +337,18 @@ static inline bool arch_has_hw_pte_young(void) }#endif include/linux/pgtable.h static inline bool arch_has_hw_pte_young(void) · line 348
Quoted 10 lines
include/linux/pgtable.h +9
-0+#ifndef arch_wants_pte_order +/* +* Returns preferred folio order for pte-mapped memory. Must be in range [0, +* PMD_SHIFT-PAGE_SHIFT) and must not be order-1 since THP requires large folios +* to be at least order-2. +*/ +static inline int arch_wants_pte_order (struct vm_area_struct *vma) +{ +return ilog2 (SZ_64K >>PAGE_SHIFT );Default value which is not related with any silicon may be:
PAGE_ALLOC_COSTLY_ORDER?Quoted 6 lines
Quoted 2 lines
Also, current pcp list support cache page with order 0...PAGE_ALLOC_COSTLY_ORDER, 9. If the pcp could cover the page, the pressure to zone lock will be reduced by pcp.
The value of
PAGE_ALLOC_COSTLY_ORDERis reasonable but again it's a s/w policy not a h/w preference. Besides, I don't think we can includemmzone.hinpgtable.h.Quoted 6 lines
I think we can make a compromise:
1. change the default implementation ofarch_has_hw_pte_young()to return 0, and
2. inmemory.c, we can tryPAGE_ALLOC_COSTLY_ORDERfor archs that don't overridearch_has_hw_pte_young(), or if its return value is too large to fit.
This should also take care of the regression, right?I think you are suggesting that we use 0 as a sentinel which we then translate to
PAGE_ALLOC_COSTLY_ORDER? I already have amax_anon_folio_order()function inmemory.c(actually it is currently a macro defined asarch_wants_pte_order()).So it would become (I'll talk about the vma concern separately in the thread where you raised it):
static inline int
max_anon_folio_order(struct vm_area_struct *vma){ int order = arch_wants_pte_order(vma); return order ? order : PAGE_ALLOC_COSTLY_ORDER; }
Quoted 5 lines
Correct?
I don't see how it fixes the regression (assume you're talking about Speedometer) though? On arm64
arch_wants_pte_order()will still be returning order-4.
Here is what I was actually suggesting -- I think the problem was because contpte is a bit too large for that benchmark and for the page allocator too, unfortunately. The following allows one retry (32KB) before fallback to order 0 when using contpte (64KB). There is no retry for HPA (16KB) and other archs.
+ int preferred = arch_wants_pte_order(vma) ? : PAGE_ALLOC_COSTLY_ORDER;
+ int orders[] = {
+ preferred,
+ preferred > PAGE_ALLOC_COSTLY_ORDER ?PAGE_ALLOC_COSTLY_ORDER : 0,
+ 0,
+ };
I'm attaching a patch which fills in the two helpers I left empty here [1].
Would the above work for Intel, Fengwei?
(AMD wouldn't need to override arch_wants_pte_order() since PTE coalescing on Zen is also PAGE_ALLOC_COSTLY_ORDER.)
mm/memory.c +69 -0
@@ -4023,6 +4023,75 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)return ret; } +static bool vmf_pte_range_changed (struct vm_fault *vmf,int nr_pages) +{ +int i; + +if (nr_pages ==1 ) +return vmf_pte_changed (vmf); + +for (i =0 ; i < nr_pages; i++) { +if (!pte_none (ptep_get_lockless (vmf->pte + i))) +return true; + } + +return false; +} + +#ifdef CONFIG_FLEXIBLE_THP +static struct folio *alloc_anon_folio (struct vm_fault *vmf) +{ +int i; +unsigned long addr; +struct vm_area_struct *vma = vmf->vma; +int preferred =arch_wants_pte_order (vma) ? :PAGE_ALLOC_COSTLY_ORDER ; +int orders[] = { + preferred, + preferred >PAGE_ALLOC_COSTLY_ORDER ?PAGE_ALLOC_COSTLY_ORDER :0 , +0 , + }; + +if (vmf_orig_pte_uffd_wp (vmf)) + goto fallback; + +for (i =0 ; orders[i]; i++) { + addr =ALIGN_DOWN (vmf->address,PAGE_SIZE << orders[i]); +if (addr >= vma->vm_start && addr + (PAGE_SIZE << orders[i]) <= vma->vm_end) +break ; + } + +if (!orders[i]) + goto fallback; + + vmf->pte =pte_offset_map (vmf->pmd, addr); + +for (; orders[i]; i++) { +if (!vmf_pte_range_changed (vmf,1 << orders[i])) +break ; + } + +pte_unmap (vmf->pte); + vmf->pte =NULL ; + +for (; orders[i]; i++) { + struct folio *folio +gfp_t gfp =vma_thp_gfp_mask (vma); + + addr =ALIGN_DOWN (vmf->address,PAGE_SIZE << orders[i]); + folio =vma_alloc_folio (gfp, orders[i], vma, addr, true); +if (folio) { +clear_huge_page (&folio->page, addr,1 << orders[i]); + vmf->address = addr; +return folio; + } + } +fallback: +return vma_alloc_zeroed_movable_folio (vma, vmf->address); +} +#else +#define alloc_anon_folio (vmf) vma_alloc_zeroed_movable_folio((vmf)->vma, (vmf)->address) +#endif +/* * We enter with non-exclusive mmap_lock (to exclude vma changes, * but allow concurrent faults), and pte mapped but not yet locked.
Re: [PATCH v2 3/5] mm: Default implementation of arch_wants_pte_order()
Quoted 103 lines · Yu Zhao
Quoted 82 lines · Ryan Roberts
Quoted 59 lines · Yu Zhao
Quoted 51 lines · Yu Zhao
Quoted 49 lines · Fengwei
Quoted 33 lines · Ryan Roberts
arch_wants_pte_order()can be overridden by the arch to return the preferred folio order for pte-mapped memory. This is useful as some architectures (e.g. arm64) can coalesce TLB entries when the physical memory is suitably contiguous.The first user for this hint will be
FLEXIBLE_THP, which aims to allocate large folios for anonymous memory to reduce page faults and other per-page operation costs.Here we add the default implementation of the function, used when the architecture does not define it, which returns the order corresponding to 64K.
Signed-off-by Ryan Roberts <[email protected]>--- include/linux/pgtable.h | 13 +++++++++++++ 1 file changed, 13 insertions(+)include/linux/pgtable.h +1
-0@@ -13,6 +13,7 @@#include <linux/errno.h> #include <asm-generic/pgtable_uffd.h> #include <linux/page_table_check.h> +#include <linux/sizes.h> #if 5 - defined(__PAGETABLE_P4D_FOLDED) - defined(__PAGETABLE_PUD_FOLDED) - \ defined(__PAGETABLE_PMD_FOLDED) !=CONFIG_PGTABLE_LEVELS @@ -336,6 +337,18 @@ static inline bool arch_has_hw_pte_young(void) }#endif include/linux/pgtable.h static inline bool arch_has_hw_pte_young(void) · line 348
Quoted 10 lines
include/linux/pgtable.h +9
-0+#ifndef arch_wants_pte_order +/* +* Returns preferred folio order for pte-mapped memory. Must be in range [0, +* PMD_SHIFT-PAGE_SHIFT) and must not be order-1 since THP requires large folios +* to be at least order-2. +*/ +static inline int arch_wants_pte_order (struct vm_area_struct *vma) +{ +return ilog2 (SZ_64K >>PAGE_SHIFT );Default value which is not related with any silicon may be:
PAGE_ALLOC_COSTLY_ORDER?Quoted 6 lines
Quoted 2 lines
Also, current pcp list support cache page with order 0...PAGE_ALLOC_COSTLY_ORDER, 9. If the pcp could cover the page, the pressure to zone lock will be reduced by pcp.
The value of
PAGE_ALLOC_COSTLY_ORDERis reasonable but again it's a s/w policy not a h/w preference. Besides, I don't think we can includemmzone.hinpgtable.h.Quoted 6 lines
I think we can make a compromise:
1. change the default implementation ofarch_has_hw_pte_young()to return 0, and
2. inmemory.c, we can tryPAGE_ALLOC_COSTLY_ORDERfor archs that don't overridearch_has_hw_pte_young(), or if its return value is too large to fit.
This should also take care of the regression, right?I think you are suggesting that we use 0 as a sentinel which we then translate to
PAGE_ALLOC_COSTLY_ORDER? I already have amax_anon_folio_order()function inmemory.c(actually it is currently a macro defined asarch_wants_pte_order()).So it would become (I'll talk about the vma concern separately in the thread where you raised it):
static inline int
max_anon_folio_order(struct vm_area_struct *vma){ int order = arch_wants_pte_order(vma); return order ? order : PAGE_ALLOC_COSTLY_ORDER; }Quoted 5 lines
Correct?
I don't see how it fixes the regression (assume you're talking about Speedometer) though? On arm64
arch_wants_pte_order()will still be returning order-4.Here is what I was actually suggesting -- I think the problem was because contpte is a bit too large for that benchmark and for the page allocator too, unfortunately. The following allows one retry (32KB) before fallback to order 0 when using contpte (64KB). There is no retry for HPA (16KB) and other archs.
mm/memory.c +4
-0+int preferred =arch_wants_pte_order (vma) ? :PAGE_ALLOC_COSTLY_ORDER ; +int orders[] = { + preferred, + preferred > PAGE_ALLOC_COSTLY_ORDER ?
PAGE_ALLOC_COSTLY_ORDER: 0,mm/memory.c +2
-0+ 0, + };
Quoted 3 lines
I'm attaching a patch which fills in the two helpers I left empty here [1].
Would the above work for Intel, Fengwei?
PAGE_ALLOC_COSTLY_ORDER is Intel preferred because it fits the most common Intel system. So yes. This works for Intel.
Regards
Yin, Fengwei
Quoted 5 lines
(AMD wouldn't need to override
arch_wants_pte_order()since PTE coalescing on Zen is alsoPAGE_ALLOC_COSTLY_ORDER.)
PATCH v2 4/5 mm: FLEXIBLE_THP for improved performance
In reply to Ryan RobertsIntroduce FLEXIBLE_THP feature, which allows anonymous memory to be allocated in large folios of a specified order. All pages of the large folio are pte-mapped during the same page fault, significantly reducing the number of page faults. The number of per-page operations (e.g. ref counting, rmap management lru list management) are also significantly reduced since those ops now become per-folio.
The new behaviour is hidden behind the new FLEXIBLE_THP Kconfig, which defaults to disabled for now; there is a long list of todos to make FLEXIBLE_THP robust with existing features (e.g. compaction, mlock, some madvise ops, etc). These items will be tackled in subsequent patches.
When enabled, the preferred folio order is as returned by arch_wants_pte_order(), which may be overridden by the arch as it sees fit. Some architectures (e.g. arm64) can coalsece TLB entries if a contiguous set of ptes map physically contigious, naturally aligned memory, so this mechanism allows the architecture to optimize as required.
If the preferred order can't be used (e.g. because the folio would breach the bounds of the vma, or because ptes in the region are already mapped) then we fall back to a suitable lower order.
--- mm/Kconfig | 10 ++++ mm/memory.c | 168 ++++++++++++++++++++++++++++++++++++++++++++++++---- 2 files changed, 165 insertions(+), 13 deletions(-)
mm/Kconfig +10 -0
@@ -822,6 +822,16 @@ config READ_ONLY_THP_FOR_FS support of file THPs will be developed in the next few release cycles. +config FLEXIBLE_THP + bool "Flexible order THP" + depends on TRANSPARENT_HUGEPAGE + default n + help + Use large (bigger than order-0) folios to back anonymous memory where + possible, even if the order of the folio is smaller than the PMD + order. This reduces the number of page faults, as well as other + per-page overheads to improve performance for many workloads. + endif # TRANSPARENT_HUGEPAGE #
mm/memory.c +155 -13
225 lines not on this page · raw message
Signature · 1 line
2.25.1
kernel test robot
build botRe: [PATCH v2 4/5] mm: FLEXIBLE_THP for improved performance
kernel test robot noticed the following build errors:
Hi Ryan,
kernel test robot noticed the following build errors:
[auto build test ERROR on arm64/for-next/core]
[also build test ERROR on v6.4]
[cannot apply to akpm-mm/mm-everything linus/master next-20230703] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Ryan-Roberts/mm-Non-pmd-mappable-large-folios-for-folio_add_new_anon_rmap/20230703-215627 base: https://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-next/core patch link: https://lore.kernel.org/r/20230703135330.1865927-5-ryan.roberts%40arm.com patch subject: [PATCH v2 4/5] mm: FLEXIBLE_THP for improved performance config: um-allyesconfig (https://download.01.org/0day-ci/archive/20230703/[email protected]/config) compiler: clang version 14.0.6 (https://github.com/llvm/llvm-project.git f28c006a5895fc0e329fe15fead81e37457cb1d1) reproduce: (https://download.01.org/0day-ci/archive/20230703/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All errors (new ones prefixed by >>):
In file included from mm/memory.c:42:
In file included from include/linux/kernel_stat.h:9:
In file included from include/linux/interrupt.h:11:
In file included from include/linux/hardirq.h:11:
In file included from arch/um/include/asm/hardirq.h:5:
In file included from include/asm-generic/hardirq.h:17:
In file included from include/linux/irq.h:20:
In file included from include/linux/io.h:13:
In file included from arch/um/include/asm/io.h:24:
include/asm-generic/io.h:547:31: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]
val = __raw_readb(PCI_IOBASE + addr);
~~~~~~~~~~ ^
include/asm-generic/io.h:560:61: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]
val = __le16_to_cpu((__le16 __force)__raw_readw(PCI_IOBASE + addr));
~~~~~~~~~~ ^
include/uapi/linux/byteorder/little_endian.h:37:51: note: expanded from macro '__le16_to_cpu'
#define __le16_to_cpu(x) ((__force __u16)(__le16)(x))
^
In file included from mm/memory.c:42:
In file included from include/linux/kernel_stat.h:9:
In file included from include/linux/interrupt.h:11:
In file included from include/linux/hardirq.h:11:
In file included from arch/um/include/asm/hardirq.h:5:
In file included from include/asm-generic/hardirq.h:17:
In file included from include/linux/irq.h:20:
In file included from include/linux/io.h:13:
In file included from arch/um/include/asm/io.h:24:
include/asm-generic/io.h:573:61: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]
val = __le32_to_cpu((__le32 __force)__raw_readl(PCI_IOBASE + addr));
~~~~~~~~~~ ^
include/uapi/linux/byteorder/little_endian.h:35:51: note: expanded from macro '__le32_to_cpu'
#define __le32_to_cpu(x) ((__force __u32)(__le32)(x))
^
In file included from mm/memory.c:42:
In file included from include/linux/kernel_stat.h:9:
In file included from include/linux/interrupt.h:11:
In file included from include/linux/hardirq.h:11:
In file included from arch/um/include/asm/hardirq.h:5:
In file included from include/asm-generic/hardirq.h:17:
In file included from include/linux/irq.h:20:
In file included from include/linux/io.h:13:
In file included from arch/um/include/asm/io.h:24:
include/asm-generic/io.h:584:33: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]
__raw_writeb(value, PCI_IOBASE + addr);
~~~~~~~~~~ ^
include/asm-generic/io.h:594:59: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]
__raw_writew((u16 __force)cpu_to_le16(value), PCI_IOBASE + addr);
~~~~~~~~~~ ^
include/asm-generic/io.h:604:59: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]
__raw_writel((u32 __force)cpu_to_le32(value), PCI_IOBASE + addr);
~~~~~~~~~~ ^
include/asm-generic/io.h:692:20: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]
readsb(PCI_IOBASE + addr, buffer, count);
~~~~~~~~~~ ^
include/asm-generic/io.h:700:20: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]
readsw(PCI_IOBASE + addr, buffer, count);
~~~~~~~~~~ ^
include/asm-generic/io.h:708:20: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]
readsl(PCI_IOBASE + addr, buffer, count);
~~~~~~~~~~ ^
include/asm-generic/io.h:717:21: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]
writesb(PCI_IOBASE + addr, buffer, count);
~~~~~~~~~~ ^
include/asm-generic/io.h:726:21: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]
writesw(PCI_IOBASE + addr, buffer, count);
~~~~~~~~~~ ^
include/asm-generic/io.h:735:21: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]
writesl(PCI_IOBASE + addr, buffer, count);
~~~~~~~~~~ ^
>> mm/memory.c:4271:2: error: implicit declaration of function 'set_ptes' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
set_ptes(vma->vm_mm, addr, vmf->pte, entry, pgcount);
^
mm/memory.c:4271:2: note: did you mean 'set_pte'?
arch/um/include/asm/pgtable.h:232:20: note: 'set_pte' declared here
static inline void set_pte(pte_t *pteptr, pte_t pteval)
^
>> mm/memory.c:4274:2: error: implicit declaration of function 'update_mmu_cache_range' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
update_mmu_cache_range(vma, addr, vmf->pte, pgcount);
^12 warnings and 2 errors generated.
vim +/set_ptes +4271 mm/memory.c
4135
4136 /*
4137 * We enter with non-exclusive mmap_lock (to exclude vma changes,
4138 * but allow concurrent faults), and pte mapped but not yet locked.
4139 * We return with mmap_lock still held, but pte unmapped and unlocked.
4140 */
4141 static vm_fault_t do_anonymous_page(struct vm_fault *vmf)
4142 {
4143 bool uffd_wp = vmf_orig_pte_uffd_wp(vmf);
4144 struct vm_area_struct *vma = vmf->vma;
4145 struct folio *folio;
4146 vm_fault_t ret = 0;
4147 pte_t entry;
4148 int order;
4149 int pgcount;
4150 unsigned long addr;
4151
4152 /* File mapping without ->vm_ops ? */
4153 if (vma->vm_flags & VM_SHARED)
4154 return VM_FAULT_SIGBUS;
4155
4156 /*
4157 * Use pte_alloc() instead of pte_alloc_map(). We can't run
4158 * pte_offset_map() on pmds where a huge pmd might be created
4159 * from a different thread.
4160 *
4161 * pte_alloc_map() is safe to use under mmap_write_lock(mm) or when
4162 * parallel threads are excluded by other means.
4163 *
4164 * Here we only have mmap_read_lock(mm).
4165 */
4166 if (pte_alloc(vma->vm_mm, vmf->pmd))
4167 return VM_FAULT_OOM;
4168
4169 /* See comment in handle_pte_fault() */
4170 if (unlikely(pmd_trans_unstable(vmf->pmd)))
4171 return 0;
4172
4173 /* Use the zero-page for reads */
4174 if (!(vmf->flags & FAULT_FLAG_WRITE) &&
4175 !mm_forbids_zeropage(vma->vm_mm)) {
4176 entry = pte_mkspecial(pfn_pte(my_zero_pfn(vmf->address),
4177 vma->vm_page_prot));
4178 vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd,
4179 vmf->address, &vmf->ptl);
4180 if (vmf_pte_changed(vmf)) {
4181 update_mmu_tlb(vma, vmf->address, vmf->pte);
4182 goto unlock;
4183 }
4184 ret = check_stable_address_space(vma->vm_mm);
4185 if (ret)
4186 goto unlock;
4187 /* Deliver the page fault to userland, check inside PT lock */
4188 if (userfaultfd_missing(vma)) {
4189 pte_unmap_unlock(vmf->pte, vmf->ptl);
4190 return handle_userfault(vmf, VM_UFFD_MISSING);
4191 }
4192 if (uffd_wp)
4193 entry = pte_mkuffd_wp(entry);
4194 set_pte_at(vma->vm_mm, vmf->address, vmf->pte, entry);
4195
4196 /* No need to invalidate - it was non-present before */
4197 update_mmu_cache(vma, vmf->address, vmf->pte);
4198 goto unlock;
4199 }
4200
4201 /*
4202 * If allocating a large folio, determine the biggest suitable order for
4203 * the VMA (e.g. it must not exceed the VMA's bounds, it must not
4204 * overlap with any populated PTEs, etc). We are not under the ptl here
4205 * so we will need to re-check that we are not overlapping any populated
4206 * PTEs once we have the lock.
4207 */
4208 order = uffd_wp ? 0 : max_anon_folio_order(vma);
4209 if (order > 0) {
4210 vmf->pte = pte_offset_map(vmf->pmd, vmf->address);
4211 order = calc_anon_folio_order_alloc(vmf, order);
4212 pte_unmap(vmf->pte);
4213 }
4214
4215 /* Allocate our own private folio. */
4216 if (unlikely(anon_vma_prepare(vma)))
4217 goto oom;
4218 folio = alloc_anon_folio(vma, vmf->address, order);
4219 if (!folio && order > 0) {
4220 order = 0;
4221 folio = alloc_anon_folio(vma, vmf->address, order);
4222 }
4223 if (!folio)
4224 goto oom;
4225
4226 pgcount = 1 << order;
4227 addr = ALIGN_DOWN(vmf->address, pgcount << PAGE_SHIFT);
4228
4229 if (mem_cgroup_charge(folio, vma->vm_mm, GFP_KERNEL))
4230 goto oom_free_page;
4231 folio_throttle_swaprate(folio, GFP_KERNEL);
4232
4233 /*
4234 * The memory barrier inside __folio_mark_uptodate makes sure that
4235 * preceding stores to the folio contents become visible before
4236 * the set_ptes() write.
4237 */
4238 __folio_mark_uptodate(folio);
4239
4240 entry = mk_pte(&folio->page, vma->vm_page_prot);
4241 entry = pte_sw_mkyoung(entry);
4242 if (vma->vm_flags & VM_WRITE)
4243 entry = pte_mkwrite(pte_mkdirty(entry));
4244
4245 vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, addr, &vmf->ptl);
4246 if (vmf_pte_changed(vmf)) {
4247 update_mmu_tlb(vma, vmf->address, vmf->pte);
4248 goto release;
4249 } else if (order > 0 && check_ptes_none(vmf->pte, pgcount) != pgcount) {
4250 goto release;
4251 }
4252
4253 ret = check_stable_address_space(vma->vm_mm);
4254 if (ret)
4255 goto release;
4256
4257 /* Deliver the page fault to userland, check inside PT lock */
4258 if (userfaultfd_missing(vma)) {
4259 pte_unmap_unlock(vmf->pte, vmf->ptl);
4260 folio_put(folio);
4261 return handle_userfault(vmf, VM_UFFD_MISSING);
4262 }
4263
4264 folio_ref_add(folio, pgcount - 1);
4265 add_mm_counter(vma->vm_mm, MM_ANONPAGES, pgcount);
4266 folio_add_new_anon_rmap(folio, vma, addr);
4267 folio_add_lru_vma(folio, vma);
4268
4269 if (uffd_wp)
4270 entry = pte_mkuffd_wp(entry);
> 4271 set_ptes(vma->vm_mm, addr, vmf->pte, entry, pgcount); 4272 4273 /* No need to invalidate - it was non-present before */ > 4274 update_mmu_cache_range(vma, addr, vmf->pte, pgcount);
4275 unlock:
4276 pte_unmap_unlock(vmf->pte, vmf->ptl);
4277 return ret;
4278 release:
4279 folio_put(folio);
4280 goto unlock;
4281 oom_free_page:
4282 folio_put(folio);
4283 oom:
4284 return VM_FAULT_OOM;
4285 }
4286
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
kernel test robot
build botRe: [PATCH v2 4/5] mm: FLEXIBLE_THP for improved performance
In reply to Ryan Robertskernel test robot noticed the following build errors:
Hi Ryan,
kernel test robot noticed the following build errors:
[auto build test ERROR on arm64/for-next/core]
[also build test ERROR on v6.4]
[cannot apply to akpm-mm/mm-everything linus/master next-20230703] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Ryan-Roberts/mm-Non-pmd-mappable-large-folios-for-folio_add_new_anon_rmap/20230703-215627 base: https://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-next/core patch link: https://lore.kernel.org/r/20230703135330.1865927-5-ryan.roberts%40arm.com patch subject: [PATCH v2 4/5] mm: FLEXIBLE_THP for improved performance config: um-allnoconfig (https://download.01.org/0day-ci/archive/20230703/[email protected]/config) compiler: clang version 17.0.0 (https://github.com/llvm/llvm-project.git 4a5ac14ee968ff0ad5d2cc1ffa0299048db4c88a) reproduce: (https://download.01.org/0day-ci/archive/20230703/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All errors (new ones prefixed by >>):
In file included from mm/memory.c:42:
In file included from include/linux/kernel_stat.h:9:
In file included from include/linux/interrupt.h:11:
In file included from include/linux/hardirq.h:11:
In file included from arch/um/include/asm/hardirq.h:5:
In file included from include/asm-generic/hardirq.h:17:
In file included from include/linux/irq.h:20:
In file included from include/linux/io.h:13:
In file included from arch/um/include/asm/io.h:24:
include/asm-generic/io.h:547:31: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]
547 | val = __raw_readb(PCI_IOBASE + addr);
| ~~~~~~~~~~ ^
include/asm-generic/io.h:560:61: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]
560 | val = __le16_to_cpu((__le16 __force)__raw_readw(PCI_IOBASE + addr));
| ~~~~~~~~~~ ^
include/uapi/linux/byteorder/little_endian.h:37:51: note: expanded from macro '__le16_to_cpu'
37 | #define __le16_to_cpu(x) ((__force __u16)(__le16)(x))
| ^
In file included from mm/memory.c:42:
In file included from include/linux/kernel_stat.h:9:
In file included from include/linux/interrupt.h:11:
In file included from include/linux/hardirq.h:11:
In file included from arch/um/include/asm/hardirq.h:5:
In file included from include/asm-generic/hardirq.h:17:
In file included from include/linux/irq.h:20:
In file included from include/linux/io.h:13:
In file included from arch/um/include/asm/io.h:24:
include/asm-generic/io.h:573:61: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]
573 | val = __le32_to_cpu((__le32 __force)__raw_readl(PCI_IOBASE + addr));
| ~~~~~~~~~~ ^
include/uapi/linux/byteorder/little_endian.h:35:51: note: expanded from macro '__le32_to_cpu'
35 | #define __le32_to_cpu(x) ((__force __u32)(__le32)(x))
| ^
In file included from mm/memory.c:42:
In file included from include/linux/kernel_stat.h:9:
In file included from include/linux/interrupt.h:11:
In file included from include/linux/hardirq.h:11:
In file included from arch/um/include/asm/hardirq.h:5:
In file included from include/asm-generic/hardirq.h:17:
In file included from include/linux/irq.h:20:
In file included from include/linux/io.h:13:
In file included from arch/um/include/asm/io.h:24:
include/asm-generic/io.h:584:33: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]
584 | __raw_writeb(value, PCI_IOBASE + addr);
| ~~~~~~~~~~ ^
include/asm-generic/io.h:594:59: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]
594 | __raw_writew((u16 __force)cpu_to_le16(value), PCI_IOBASE + addr);
| ~~~~~~~~~~ ^
include/asm-generic/io.h:604:59: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]
604 | __raw_writel((u32 __force)cpu_to_le32(value), PCI_IOBASE + addr);
| ~~~~~~~~~~ ^
include/asm-generic/io.h:692:20: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]
692 | readsb(PCI_IOBASE + addr, buffer, count);
| ~~~~~~~~~~ ^
include/asm-generic/io.h:700:20: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]
700 | readsw(PCI_IOBASE + addr, buffer, count);
| ~~~~~~~~~~ ^
include/asm-generic/io.h:708:20: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]
708 | readsl(PCI_IOBASE + addr, buffer, count);
| ~~~~~~~~~~ ^
include/asm-generic/io.h:717:21: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]
717 | writesb(PCI_IOBASE + addr, buffer, count);
| ~~~~~~~~~~ ^
include/asm-generic/io.h:726:21: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]
726 | writesw(PCI_IOBASE + addr, buffer, count);
| ~~~~~~~~~~ ^
include/asm-generic/io.h:735:21: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]
735 | writesl(PCI_IOBASE + addr, buffer, count);
| ~~~~~~~~~~ ^
>> mm/memory.c:4271:2: error: call to undeclared function 'set_ptes'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
4271 | set_ptes(vma->vm_mm, addr, vmf->pte, entry, pgcount);
| ^
mm/memory.c:4271:2: note: did you mean 'set_pte'?
arch/um/include/asm/pgtable.h:232:20: note: 'set_pte' declared here
232 | static inline void set_pte(pte_t *pteptr, pte_t pteval)
| ^
>> mm/memory.c:4274:2: error: call to undeclared function 'update_mmu_cache_range'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
4274 | update_mmu_cache_range(vma, addr, vmf->pte, pgcount);
| ^12 warnings and 2 errors generated.
vim +/set_ptes +4271 mm/memory.c
4135
4136 /*
4137 * We enter with non-exclusive mmap_lock (to exclude vma changes,
4138 * but allow concurrent faults), and pte mapped but not yet locked.
4139 * We return with mmap_lock still held, but pte unmapped and unlocked.
4140 */
4141 static vm_fault_t do_anonymous_page(struct vm_fault *vmf)
4142 {
4143 bool uffd_wp = vmf_orig_pte_uffd_wp(vmf);
4144 struct vm_area_struct *vma = vmf->vma;
4145 struct folio *folio;
4146 vm_fault_t ret = 0;
4147 pte_t entry;
4148 int order;
4149 int pgcount;
4150 unsigned long addr;
4151
4152 /* File mapping without ->vm_ops ? */
4153 if (vma->vm_flags & VM_SHARED)
4154 return VM_FAULT_SIGBUS;
4155
4156 /*
4157 * Use pte_alloc() instead of pte_alloc_map(). We can't run
4158 * pte_offset_map() on pmds where a huge pmd might be created
4159 * from a different thread.
4160 *
4161 * pte_alloc_map() is safe to use under mmap_write_lock(mm) or when
4162 * parallel threads are excluded by other means.
4163 *
4164 * Here we only have mmap_read_lock(mm).
4165 */
4166 if (pte_alloc(vma->vm_mm, vmf->pmd))
4167 return VM_FAULT_OOM;
4168
4169 /* See comment in handle_pte_fault() */
4170 if (unlikely(pmd_trans_unstable(vmf->pmd)))
4171 return 0;
4172
4173 /* Use the zero-page for reads */
4174 if (!(vmf->flags & FAULT_FLAG_WRITE) &&
4175 !mm_forbids_zeropage(vma->vm_mm)) {
4176 entry = pte_mkspecial(pfn_pte(my_zero_pfn(vmf->address),
4177 vma->vm_page_prot));
4178 vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd,
4179 vmf->address, &vmf->ptl);
4180 if (vmf_pte_changed(vmf)) {
4181 update_mmu_tlb(vma, vmf->address, vmf->pte);
4182 goto unlock;
4183 }
4184 ret = check_stable_address_space(vma->vm_mm);
4185 if (ret)
4186 goto unlock;
4187 /* Deliver the page fault to userland, check inside PT lock */
4188 if (userfaultfd_missing(vma)) {
4189 pte_unmap_unlock(vmf->pte, vmf->ptl);
4190 return handle_userfault(vmf, VM_UFFD_MISSING);
4191 }
4192 if (uffd_wp)
4193 entry = pte_mkuffd_wp(entry);
4194 set_pte_at(vma->vm_mm, vmf->address, vmf->pte, entry);
4195
4196 /* No need to invalidate - it was non-present before */
4197 update_mmu_cache(vma, vmf->address, vmf->pte);
4198 goto unlock;
4199 }
4200
4201 /*
4202 * If allocating a large folio, determine the biggest suitable order for
4203 * the VMA (e.g. it must not exceed the VMA's bounds, it must not
4204 * overlap with any populated PTEs, etc). We are not under the ptl here
4205 * so we will need to re-check that we are not overlapping any populated
4206 * PTEs once we have the lock.
4207 */
4208 order = uffd_wp ? 0 : max_anon_folio_order(vma);
4209 if (order > 0) {
4210 vmf->pte = pte_offset_map(vmf->pmd, vmf->address);
4211 order = calc_anon_folio_order_alloc(vmf, order);
4212 pte_unmap(vmf->pte);
4213 }
4214
4215 /* Allocate our own private folio. */
4216 if (unlikely(anon_vma_prepare(vma)))
4217 goto oom;
4218 folio = alloc_anon_folio(vma, vmf->address, order);
4219 if (!folio && order > 0) {
4220 order = 0;
4221 folio = alloc_anon_folio(vma, vmf->address, order);
4222 }
4223 if (!folio)
4224 goto oom;
4225
4226 pgcount = 1 << order;
4227 addr = ALIGN_DOWN(vmf->address, pgcount << PAGE_SHIFT);
4228
4229 if (mem_cgroup_charge(folio, vma->vm_mm, GFP_KERNEL))
4230 goto oom_free_page;
4231 folio_throttle_swaprate(folio, GFP_KERNEL);
4232
4233 /*
4234 * The memory barrier inside __folio_mark_uptodate makes sure that
4235 * preceding stores to the folio contents become visible before
4236 * the set_ptes() write.
4237 */
4238 __folio_mark_uptodate(folio);
4239
4240 entry = mk_pte(&folio->page, vma->vm_page_prot);
4241 entry = pte_sw_mkyoung(entry);
4242 if (vma->vm_flags & VM_WRITE)
4243 entry = pte_mkwrite(pte_mkdirty(entry));
4244
4245 vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, addr, &vmf->ptl);
4246 if (vmf_pte_changed(vmf)) {
4247 update_mmu_tlb(vma, vmf->address, vmf->pte);
4248 goto release;
4249 } else if (order > 0 && check_ptes_none(vmf->pte, pgcount) != pgcount) {
4250 goto release;
4251 }
4252
4253 ret = check_stable_address_space(vma->vm_mm);
4254 if (ret)
4255 goto release;
4256
4257 /* Deliver the page fault to userland, check inside PT lock */
4258 if (userfaultfd_missing(vma)) {
4259 pte_unmap_unlock(vmf->pte, vmf->ptl);
4260 folio_put(folio);
4261 return handle_userfault(vmf, VM_UFFD_MISSING);
4262 }
4263
4264 folio_ref_add(folio, pgcount - 1);
4265 add_mm_counter(vma->vm_mm, MM_ANONPAGES, pgcount);
4266 folio_add_new_anon_rmap(folio, vma, addr);
4267 folio_add_lru_vma(folio, vma);
4268
4269 if (uffd_wp)
4270 entry = pte_mkuffd_wp(entry);
> 4271 set_ptes(vma->vm_mm, addr, vmf->pte, entry, pgcount); 4272 4273 /* No need to invalidate - it was non-present before */ > 4274 update_mmu_cache_range(vma, addr, vmf->pte, pgcount);
4275 unlock:
4276 pte_unmap_unlock(vmf->pte, vmf->ptl);
4277 return ret;
4278 release:
4279 folio_put(folio);
4280 goto unlock;
4281 oom_free_page:
4282 folio_put(folio);
4283 oom:
4284 return VM_FAULT_OOM;
4285 }
4286
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
Re: [PATCH v2 4/5] mm: FLEXIBLE_THP for improved performance
In reply to Ryan RobertsQuoted 8 lines · Ryan Roberts
Introduce
FLEXIBLE_THPfeature, which allows anonymous memory to be allocated in large folios of a specified order. All pages of the large folio are pte-mapped during the same page fault, significantly reducing the number of page faults. The number of per-page operations (e.g. ref counting, rmap management lru list management) are also significantly reduced since those ops now become per-folio.
Quoted 8 lines
The new behaviour is hidden behind the new
FLEXIBLE_THPKconfig, which defaults to disabled for now; there is a long list of todos to makeFLEXIBLE_THProbust with existing features (e.g. compaction, mlock, some madvise ops, etc). These items will be tackled in subsequent patches.When enabled, the preferred folio order is as returned by
arch_wants_pte_order(), which may be overridden by the arch as it sees fit. Some architectures (e.g. arm64) can coalsece TLB entries if a
coalesce
Quoted 1 line
contiguous set of ptes map physically contigious, naturally aligned
contiguous
Quoted 18 lines
memory, so this mechanism allows the architecture to optimize as required.
If the preferred order can't be used (e.g. because the folio would breach the bounds of the vma, or because ptes in the region are already mapped) then we fall back to a suitable lower order.
Signed-off-by Ryan Roberts <[email protected]>--- mm/Kconfig | 10 ++++ mm/memory.c | 168 ++++++++++++++++++++++++++++++++++++++++++++++++---- 2 files changed, 165 insertions(+), 13 deletions(-)mm/Kconfig +0
-0
mm/Kconfig config READ_ONLY_THP_FOR_FS · line 828
Quoted 7 lines
mm/Kconfig +4
-0@@ -822,6 +822,16 @@ config READ_ONLY_THP_FOR_FS support of file THPs will be developed in the next few release cycles. +config FLEXIBLE_THP + bool "Flexible order THP" + depends on TRANSPARENT_HUGEPAGE + default n
The default value is already N.
Quoted 86 lines
mm/Kconfig +6
-0+ help + Use large (bigger than order-0) folios to back anonymous memory where + possible, even if the order of the folio is smaller than the PMD + order. This reduces the number of page faults, as well as other + per-page overheads to improve performance for many workloads. + endif # TRANSPARENT_HUGEPAGE #mm/memory.c +69
-0@@ -3001,6 +3001,116 @@ static vm_fault_t fault_dirty_shared_page(struct vm_fault *vmf)return 0 ; } +#ifdef CONFIG_FLEXIBLE_THP +/* +* Allocates, zeros and returns a folio of the requested order for use as +* anonymous memory. +*/ +static struct folio *alloc_anon_folio (struct vm_area_struct *vma, +unsigned long addr,int order) +{ +gfp_t gfp; +struct folio *folio; + +if (order ==0 ) +return vma_alloc_zeroed_movable_folio (vma, addr); + + gfp =vma_thp_gfp_mask (vma); + folio =vma_alloc_folio (gfp, order, vma, addr, true); +if (folio) +clear_huge_page (&folio->page, addr,folio_nr_pages (folio)); + +return folio; +} + +/* +* Preferred folio order to allocate for anonymous memory. +*/ +#define max_anon_folio_order (vma) arch_wants_pte_order(vma) +#else +#define alloc_anon_folio (vma, addr, order) \ + vma_alloc_zeroed_movable_folio(vma, addr) +#define max_anon_folio_order (vma) 0 +#endif + +/* +* Returns index of first pte that is not none, or nr if all are none. +*/ +static inline int check_ptes_none (pte_t *pte,int nr) +{ +int i; + +for (i =0 ; i < nr; i++) { +if (!pte_none (ptep_get (pte++))) +return i; + } + +return nr; +} + +static int calc_anon_folio_order_alloc (struct vm_fault *vmf,int order) +{ +/* +* The aim here is to determine what size of folio we should allocate +* for this fault. Factors include: +* - Order must not be higher than `order` upon entry +* - Folio must be naturally aligned within VA space +* - Folio must be fully contained inside one pmd entry +* - Folio must not breach boundaries of vma +* - Folio must not overlap any non-none ptes +* +* Additionally, we do not allow order-1 since this breaks assumptions +* elsewhere in the mm; THP pages must be at least order-2 (since they +* store state up to the 3rd struct page subpage), and these pages must +* be THP in order to correctly use pre-existing THP infrastructure such +* as folio_split(). +* +* Note that the caller may or may not choose to lock the pte. If +* unlocked, the result is racy and the user must re-check any overlap +* with non-none ptes under the lock. +*/ +
mm/memory.c static vm_fault_t fault_dirty_shared_page(struct vm_fault *vmf) · line 3082
Quoted 10 lines
mm/memory.c +10
-0+struct vm_area_struct *vma = vmf->vma; +int nr; +unsigned long addr; +pte_t *pte; +pte_t *first_set =NULL ; +int ret; + + order =min (order,PMD_SHIFT -PAGE_SHIFT ); + +for (; order >1 ; order--) {
I'm not sure how we can justify this policy. As an initial step, it'd be a lot easier to sell if we only considered the order of arch_wants_pte_order() and the order 0.
Quoted 20 lines
mm/memory.c +20
-0+ nr =1 << order; + addr =ALIGN_DOWN (vmf->address, nr <<PAGE_SHIFT ); + pte = vmf->pte - ((vmf->address - addr) >>PAGE_SHIFT ); + +/* Check vma bounds. */ +if (addr < vma->vm_start || + addr + (nr <<PAGE_SHIFT ) > vma->vm_end) +continue ; + +/* Ptes covered by order already known to be none. */ +if (pte + nr <= first_set) +break ; + +/* Already found set pte in range covered by order. */ +if (pte <= first_set) +continue ; + +/* Need to check if all the ptes are none. */ + ret =check_ptes_none (pte, nr); +if (ret == nr)
mm/memory.c static vm_fault_t fault_dirty_shared_page(struct vm_fault *vmf) · line 3112
Quoted 10 lines
mm/memory.c +10
-0+break ; + + first_set = pte + ret; + } + +if (order ==1 ) + order =0 ; + +return order; +}
Everything above can be simplified into two helpers:vmf_pte_range_changed() and alloc_anon_folio() (or whatever names you prefer). Details below.
mm/memory.c static vm_fault_t wp_page_copy(struct vm_fault *vmf) · line 3186
Quoted 9 lines
mm/memory.c +1
-1/* * Handle write page faults for pages that can be reused in the current vma * @@ -3073,7 +3183,7 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) goto oom;if (is_zero_pfn (pte_pfn (vmf->orig_pte))) { - new_folio =vma_alloc_zeroed_movable_folio (vma, vmf->address); + new_folio =alloc_anon_folio (vma, vmf->address,0 );
This seems unnecessary for now. Later on, we could fill in an aligned area with multiple write-protected zero pages during a read fault and then replace them with a large folio here.
Quoted 16 lines
mm/memory.c +3
-0if (!new_folio) goto oom; }else {@@ -4040,6 +4150,9 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)struct folio *folio;vm_fault_t ret =0 ;pte_t entry; +int order; +int pgcount; +unsigned long addr;/* File mapping without ->vm_ops ? */ if (vma->vm_flags &VM_SHARED )@@ -4081,24 +4194,51 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)pte_unmap_unlock (vmf->pte, vmf->ptl);return handle_userfault (vmf,VM_UFFD_MISSING );
mm/memory.c static vm_fault_t do_anonymous_page(struct vm_fault *vmf) · line 4204
Quoted 10 lines
mm/memory.c +8
-1} - goto setpte; +if (uffd_wp) + entry =pte_mkuffd_wp (entry); +set_pte_at (vma->vm_mm, vmf->address, vmf->pte, entry); + +/* No need to invalidate - it was non-present before */ +update_mmu_cache (vma, vmf->address, vmf->pte); + goto unlock; + }
Nor really needed IMO. Details below.
===
mm/memory.c static vm_fault_t do_anonymous_page(struct vm_fault *vmf) · line 4217
Quoted 13 lines
mm/memory.c +12
-0+/* +* If allocating a large folio, determine the biggest suitable order for +* the VMA (e.g. it must not exceed the VMA's bounds, it must not +* overlap with any populated PTEs, etc). We are not under the ptl here +* so we will need to re-check that we are not overlapping any populated +* PTEs once we have the lock. +*/ + order = uffd_wp ?0 :max_anon_folio_order (vma); +if (order >0 ) { + vmf->pte =pte_offset_map (vmf->pmd, vmf->address); + order =calc_anon_folio_order_alloc (vmf, order); +pte_unmap (vmf->pte); }
===
The section above together with the section below should be wrapped in a helper.
mm/memory.c static vm_fault_t do_anonymous_page(struct vm_fault *vmf) · line 4220
Quoted 4 lines
mm/memory.c +1
-1-/* Allocate our own private page. */ +/* Allocate our own private folio. */ if (unlikely (anon_vma_prepare (vma))) goto oom;
===
mm/memory.c static vm_fault_t do_anonymous_page(struct vm_fault *vmf) · line 4225
Quoted 6 lines
mm/memory.c +5
-1- folio =vma_alloc_zeroed_movable_folio (vma, vmf->address); + folio =alloc_anon_folio (vma, vmf->address, order); +if (!folio && order >0 ) { + order =0 ; + folio =alloc_anon_folio (vma, vmf->address, order); + }
===
One helper returns a folio of order arch_wants_pte_order(), or order 0 if it fails to allocate that order, e.g.,
folio = alloc_anon_folio(vmf);
And if vmf_orig_pte_uffd_wp(vmf) is true, the helper allocates order 0 regardless of arch_wants_pte_order(). Upon success, it can update vmf->address, since if we run into a race with another PF, we exit the fault handler and retry anyway.
mm/memory.c static vm_fault_t do_anonymous_page(struct vm_fault *vmf) · line 4230
Quoted 5 lines
mm/memory.c +2
-0if (!folio) goto oom; + pgcount =1 << order; + addr =ALIGN_DOWN (vmf->address, pgcount <<PAGE_SHIFT );
As shown above, the helper already updates vmf->address. And mm/ never used pgcount before -- the convention is nr_pages = folio_nr_pages().
mm/memory.c static vm_fault_t do_anonymous_page(struct vm_fault *vmf) · line 4238
Quoted 10 lines
mm/memory.c +2
-2if (mem_cgroup_charge (folio, vma->vm_mm,GFP_KERNEL )) goto oom_free_page;folio_throttle_swaprate (folio,GFP_KERNEL );/* * The memory barrier inside __folio_mark_uptodate makes sure that -* preceding stores to the page contents become visible before -* the set_pte_at() write. +* preceding stores to the folio contents become visible before +* the set_ptes() write.
We don't have set_ptes() yet.
Quoted 6 lines
mm/memory.c +0
-0*/ __folio_mark_uptodate (folio);@@ -4107,11 +4247,12 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)if (vma->vm_flags &VM_WRITE ) entry =pte_mkwrite (pte_mkdirty (entry));
mm/memory.c static vm_fault_t do_anonymous_page(struct vm_fault *vmf) · line 4256
Quoted 10 lines
mm/memory.c +3
-2- vmf->pte =pte_offset_map_lock (vma->vm_mm, vmf->pmd, vmf->address, - &vmf->ptl); + vmf->pte =pte_offset_map_lock (vma->vm_mm, vmf->pmd, addr, &vmf->ptl);if (vmf_pte_changed (vmf)) {update_mmu_tlb (vma, vmf->address, vmf->pte); goto release; + }else if (order >0 &&check_ptes_none (vmf->pte, pgcount) != pgcount) { + goto release; }
Need new helper:
if (vmf_pte_range_changed(vmf, nr_pages)) {
for (i = 0; i < nr_pages; i++)
update_mmu_tlb(vma, vmf->address + PAGE_SIZE * i, vmf->pte + i);
goto release;
}(It should be fine to call update_mmu_tlb() even if it's not really necessary.)
Quoted 7 lines
mm/memory.c +0
-2ret =check_stable_address_space (vma->vm_mm);@@ -4125,16 +4266,17 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)return handle_userfault (vmf,VM_UFFD_MISSING ); } -inc_mm_counter (vma->vm_mm,MM_ANONPAGES ); -folio_add_new_anon_rmap (folio, vma, vmf->address);
mm/memory.c static vm_fault_t do_anonymous_page(struct vm_fault *vmf) · line 4276
Quoted 10 lines
mm/memory.c +5
-2+folio_ref_add (folio, pgcount -1 ); +add_mm_counter (vma->vm_mm,MM_ANONPAGES , pgcount); +folio_add_new_anon_rmap (folio, vma, addr);folio_add_lru_vma (folio, vma); -setpte: +if (uffd_wp) entry =pte_mkuffd_wp (entry); -set_pte_at (vma->vm_mm, vmf->address, vmf->pte, entry); +set_ptes (vma->vm_mm, addr, vmf->pte, entry, pgcount);
We would have to do it one by one for now.
mm/memory.c static vm_fault_t do_anonymous_page(struct vm_fault *vmf) · line 4278
Quoted 3 lines
mm/memory.c +1
-1/* No need to invalidate - it was non-present before */ -update_mmu_cache (vma, vmf->address, vmf->pte); +update_mmu_cache_range (vma, addr, vmf->pte, pgcount);
Ditto.
How about this (by moving mk_pte() and its friends here):
...
folio_add_lru_vma(folio, vma);
for (i = 0; i < nr_pages; i++) {
entry = mk_pte(folio_page(folio, i), vma->vm_page_prot);
entry = pte_sw_mkyoung(entry);
if (vma->vm_flags & VM_WRITE)
entry = pte_mkwrite(pte_mkdirty(entry));
setpte:
if (uffd_wp)
entry = pte_mkuffd_wp(entry);
set_pte_at(vma->vm_mm, vmf->address + PAGE_SIZE * i,vmf->pte + i, entry);
/* No need to invalidate - it was non-present before */
update_mmu_cache(vma, vmf->address + PAGE_SIZE * i,
vmf->pte + i);
}mm/memory.c static vm_fault_t do_anonymous_page(struct vm_fault *vmf) · line 4281
Quoted 3 lines
mm/memory.c +0
-0unlock:pte_unmap_unlock (vmf->pte, vmf->ptl);return ret;
Attaching a small patch in case anything above is not clear. Please take a look. Thanks.
mm/memory.c +22 -14
@@ -4063,6 +4063,8 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)*/ static vm_fault_t do_anonymous_page (struct vm_fault *vmf) { +int i =0 ; +int nr_pages =1 ;bool uffd_wp =vmf_orig_pte_uffd_wp (vmf);struct vm_area_struct *vma = vmf->vma;struct folio *folio;
@@ -4107,10 +4109,12 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)/* Allocate our own private page. */ if (unlikely (anon_vma_prepare (vma))) goto oom; - folio =vma_alloc_zeroed_movable_folio (vma, vmf->address); + folio =alloc_anon_folio (vmf);// updates vmf->address accordingly if (!folio) goto oom; + nr_pages =folio_nr_pages (folio); +if (mem_cgroup_charge (folio, vma->vm_mm,GFP_KERNEL )) goto oom_free_page;folio_throttle_swaprate (folio,GFP_KERNEL );
@@ -4122,17 +4126,13 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)*/ __folio_mark_uptodate (folio); - entry =mk_pte (&folio->page, vma->vm_page_prot); - entry =pte_sw_mkyoung (entry); -if (vma->vm_flags &VM_WRITE ) - entry =pte_mkwrite (pte_mkdirty (entry)); - vmf->pte =pte_offset_map_lock (vma->vm_mm, vmf->pmd, vmf->address, &vmf->ptl);if (!vmf->pte) goto release; -if (vmf_pte_changed (vmf)) { -update_mmu_tlb (vma, vmf->address, vmf->pte); +if (vmf_pte_range_changed (vmf, nr_pages)) { +for (i =0 ; i < nr_pages; i++) +update_mmu_tlb (vma, vmf->address +PAGE_SIZE * i, vmf->pte + i); goto release; }
@@ -4147,16 +4147,24 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)return handle_userfault (vmf,VM_UFFD_MISSING ); } -inc_mm_counter (vma->vm_mm,MM_ANONPAGES ); +folio_ref_add (folio, nr_pages -1 ); +add_mm_counter (vma->vm_mm,MM_ANONPAGES , nr_pages);folio_add_new_anon_rmap (folio, vma, vmf->address);folio_add_lru_vma (folio, vma); + +for (i =0 ; i < nr_pages; i++) { + entry =mk_pte (folio_page (folio, i), vma->vm_page_prot); + entry =pte_sw_mkyoung (entry); +if (vma->vm_flags &VM_WRITE ) + entry =pte_mkwrite (pte_mkdirty (entry)); setpte: -if (uffd_wp) - entry =pte_mkuffd_wp (entry); -set_pte_at (vma->vm_mm, vmf->address, vmf->pte, entry); +if (uffd_wp) + entry =pte_mkuffd_wp (entry); +set_pte_at (vma->vm_mm, vmf->address +PAGE_SIZE * i, vmf->pte + i, entry); -/* No need to invalidate - it was non-present before */ -update_mmu_cache (vma, vmf->address, vmf->pte); +/* No need to invalidate - it was non-present before */ +update_mmu_cache (vma, vmf->address +PAGE_SIZE * i, vmf->pte + i); + } unlock:if (vmf->pte)pte_unmap_unlock (vmf->pte, vmf->ptl);
Re: [PATCH v2 4/5] mm: FLEXIBLE_THP for improved performance
Quoted 14 lines · Yu Zhao
Quoted 13 lines · Ryan Roberts
Introduce
FLEXIBLE_THPfeature, which allows anonymous memory to be allocated in large folios of a specified order. All pages of the large folio are pte-mapped during the same page fault, significantly reducing the number of page faults. The number of per-page operations (e.g. ref counting, rmap management lru list management) are also significantly reduced since those ops now become per-folio.The new behaviour is hidden behind the new
FLEXIBLE_THPKconfig, which defaults to disabled for now; there is a long list of todos to makeFLEXIBLE_THProbust with existing features (e.g. compaction, mlock, some madvise ops, etc). These items will be tackled in subsequent patches.
Quoted 5 lines
Quoted 3 lines
When enabled, the preferred folio order is as returned by
arch_wants_pte_order(), which may be overridden by the arch as it sees fit. Some architectures (e.g. arm64) can coalsece TLB entries if acoalesce
ACK
Quoted 4 lines
Quoted 2 lines
contiguous set of ptes map physically contigious, naturally aligned
contiguous
ACK
Quoted 22 lines
Quoted 22 lines
memory, so this mechanism allows the architecture to optimize as required.
If the preferred order can't be used (e.g. because the folio would breach the bounds of the vma, or because ptes in the region are already mapped) then we fall back to a suitable lower order.
Signed-off-by Ryan Roberts <[email protected]>--- mm/Kconfig | 10 ++++ mm/memory.c | 168 ++++++++++++++++++++++++++++++++++++++++++++++++---- 2 files changed, 165 insertions(+), 13 deletions(-)mm/Kconfig +0
-0@@ -822,6 +822,16 @@ config READ_ONLY_THP_FOR_FS support of file THPs will be developed in the next few release cycles.
Quoted 6 lines
mm/Kconfig config READ_ONLY_THP_FOR_FS · line 828
Quoted 4 lines
mm/Kconfig +4
-0+config FLEXIBLE_THP + bool "Flexible order THP" + depends on TRANSPARENT_HUGEPAGE + default nThe default value is already N.
Is there a coding standard for this? Personally I prefer to make it explicit.
Quoted 93 lines
Quoted 93 lines
mm/Kconfig +6
-0+ help + Use large (bigger than order-0) folios to back anonymous memory where + possible, even if the order of the folio is smaller than the PMD + order. This reduces the number of page faults, as well as other + per-page overheads to improve performance for many workloads. + endif # TRANSPARENT_HUGEPAGE #mm/memory.c +75
-0@@ -3001,6 +3001,116 @@ static vm_fault_t fault_dirty_shared_page(struct vm_fault *vmf)return 0 ; } +#ifdef CONFIG_FLEXIBLE_THP +/* +* Allocates, zeros and returns a folio of the requested order for use as +* anonymous memory. +*/ +static struct folio *alloc_anon_folio (struct vm_area_struct *vma, +unsigned long addr,int order) +{ +gfp_t gfp; +struct folio *folio; + +if (order ==0 ) +return vma_alloc_zeroed_movable_folio (vma, addr); + + gfp =vma_thp_gfp_mask (vma); + folio =vma_alloc_folio (gfp, order, vma, addr, true); +if (folio) +clear_huge_page (&folio->page, addr,folio_nr_pages (folio)); + +return folio; +} + +/* +* Preferred folio order to allocate for anonymous memory. +*/ +#define max_anon_folio_order (vma) arch_wants_pte_order(vma) +#else +#define alloc_anon_folio (vma, addr, order) \ + vma_alloc_zeroed_movable_folio(vma, addr) +#define max_anon_folio_order (vma) 0 +#endif + +/* +* Returns index of first pte that is not none, or nr if all are none. +*/ +static inline int check_ptes_none (pte_t *pte,int nr) +{ +int i; + +for (i =0 ; i < nr; i++) { +if (!pte_none (ptep_get (pte++))) +return i; + } + +return nr; +} + +static int calc_anon_folio_order_alloc (struct vm_fault *vmf,int order) +{ +/* +* The aim here is to determine what size of folio we should allocate +* for this fault. Factors include: +* - Order must not be higher than `order` upon entry +* - Folio must be naturally aligned within VA space +* - Folio must be fully contained inside one pmd entry +* - Folio must not breach boundaries of vma +* - Folio must not overlap any non-none ptes +* +* Additionally, we do not allow order-1 since this breaks assumptions +* elsewhere in the mm; THP pages must be at least order-2 (since they +* store state up to the 3rd struct page subpage), and these pages must +* be THP in order to correctly use pre-existing THP infrastructure such +* as folio_split(). +* +* Note that the caller may or may not choose to lock the pte. If +* unlocked, the result is racy and the user must re-check any overlap +* with non-none ptes under the lock. +*/ + +struct vm_area_struct *vma = vmf->vma; +int nr; +unsigned long addr; +pte_t *pte; +pte_t *first_set =NULL ; +int ret;
Quoted 8 lines
mm/memory.c static vm_fault_t fault_dirty_shared_page(struct vm_fault *vmf) · line 3082
Quoted 4 lines
mm/memory.c +4
-0+ + order =min (order,PMD_SHIFT -PAGE_SHIFT ); + +for (; order >1 ; order--) {I'm not sure how we can justify this policy. As an initial step, it'd be a lot easier to sell if we only considered the order of
arch_wants_pte_order()and the order 0.
My justification is in the cover letter; I see performance regression (vs the unpatched kernel) when using the policy you suggest. This policy performs much better in my tests. (I'll reply directly to your follow up questions in the cover letter shortly).
What are your technical concerns about this approach? It is pretty light weight (I only touch each PTE once, regardless of the number of loops). If we have strong technical reasons for reverting to the less performant approach then fair enough, but I'd like to hear the rational first.
Quoted 42 lines
Quoted 21 lines
mm/memory.c +20
-0+ nr =1 << order; + addr =ALIGN_DOWN (vmf->address, nr <<PAGE_SHIFT ); + pte = vmf->pte - ((vmf->address - addr) >>PAGE_SHIFT ); + +/* Check vma bounds. */ +if (addr < vma->vm_start || + addr + (nr <<PAGE_SHIFT ) > vma->vm_end) +continue ; + +/* Ptes covered by order already known to be none. */ +if (pte + nr <= first_set) +break ; + +/* Already found set pte in range covered by order. */ +if (pte <= first_set) +continue ; + +/* Need to check if all the ptes are none. */ + ret =check_ptes_none (pte, nr); +if (ret == nr)mm/memory.c static vm_fault_t fault_dirty_shared_page(struct vm_fault *vmf) · line 3112
Quoted 10 lines
mm/memory.c +10
-0+break ; + + first_set = pte + ret; + } + +if (order ==1 ) + order =0 ; + +return order; +}Everything above can be simplified into two helpers:
vmf_pte_range_changed()andalloc_anon_folio()(or whatever names you prefer). Details below.Quoted 6 lines
mm/memory.c +0
-0/* * Handle write page faults for pages that can be reused in the current vma * @@ -3073,7 +3183,7 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) goto oom;
Quoted 7 lines
mm/memory.c static vm_fault_t wp_page_copy(struct vm_fault *vmf) · line 3186
Quoted 3 lines
mm/memory.c +1
-1if (is_zero_pfn (pte_pfn (vmf->orig_pte))) { - new_folio =vma_alloc_zeroed_movable_folio (vma, vmf->address); + new_folio =alloc_anon_folio (vma, vmf->address,0 );This seems unnecessary for now. Later on, we could fill in an aligned area with multiple write-protected zero pages during a read fault and then replace them with a large folio here.
I don't have a strong opinion. I thought that it would be neater to use the same API everywhere, but happy to revert.
Quoted 76 lines
Quoted 17 lines
mm/memory.c +3
-0if (!new_folio) goto oom; }else {@@ -4040,6 +4150,9 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)struct folio *folio;vm_fault_t ret =0 ;pte_t entry; +int order; +int pgcount; +unsigned long addr;/* File mapping without ->vm_ops ? */ if (vma->vm_flags &VM_SHARED )@@ -4081,24 +4194,51 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)pte_unmap_unlock (vmf->pte, vmf->ptl);return handle_userfault (vmf,VM_UFFD_MISSING );mm/memory.c static vm_fault_t do_anonymous_page(struct vm_fault *vmf) · line 4204
Quoted 10 lines
mm/memory.c +8
-1} - goto setpte; +if (uffd_wp) + entry =pte_mkuffd_wp (entry); +set_pte_at (vma->vm_mm, vmf->address, vmf->pte, entry); + +/* No need to invalidate - it was non-present before */ +update_mmu_cache (vma, vmf->address, vmf->pte); + goto unlock; + }Nor really needed IMO. Details below.
===
mm/memory.c static vm_fault_t do_anonymous_page(struct vm_fault *vmf) · line 4217
Quoted 13 lines
mm/memory.c +12
-0+/* +* If allocating a large folio, determine the biggest suitable order for +* the VMA (e.g. it must not exceed the VMA's bounds, it must not +* overlap with any populated PTEs, etc). We are not under the ptl here +* so we will need to re-check that we are not overlapping any populated +* PTEs once we have the lock. +*/ + order = uffd_wp ?0 :max_anon_folio_order (vma); +if (order >0 ) { + vmf->pte =pte_offset_map (vmf->pmd, vmf->address); + order =calc_anon_folio_order_alloc (vmf, order); +pte_unmap (vmf->pte); }===
The section above together with the section below should be wrapped in a helper.
mm/memory.c static vm_fault_t do_anonymous_page(struct vm_fault *vmf) · line 4111
Quoted 4 lines
mm/memory.c +1
-1-/* Allocate our own private page. */ +/* Allocate our own private folio. */ if (unlikely (anon_vma_prepare (vma))) goto oom;===
mm/memory.c static vm_fault_t do_anonymous_page(struct vm_fault *vmf) · line 4225
Quoted 6 lines
mm/memory.c +5
-1- folio =vma_alloc_zeroed_movable_folio (vma, vmf->address); + folio =alloc_anon_folio (vma, vmf->address, order); +if (!folio && order >0 ) { + order =0 ; + folio =alloc_anon_folio (vma, vmf->address, order); + }===
One helper returns a folio of order
arch_wants_pte_order(), or order 0 if it fails to allocate that order, e.g.,folio =
alloc_anon_folio(vmf);And if
vmf_orig_pte_uffd_wp(vmf)is true, the helper allocates order 0 regardless ofarch_wants_pte_order(). Upon success, it can updatevmf->address, since if we run into a race with another PF, we exit the fault handler and retry anyway.
Quoted 8 lines
mm/memory.c static vm_fault_t do_anonymous_page(struct vm_fault *vmf)
Quoted 5 lines
mm/memory.c +2
-0if (!folio) goto oom; + pgcount =1 << order; + addr =ALIGN_DOWN (vmf->address, pgcount <<PAGE_SHIFT );As shown above, the helper already updates
vmf->address. Andmm/never used pgcount before -- the convention isnr_pages=folio_nr_pages().
ACK
Quoted 5 lines
Quoted 5 lines
mm/memory.c +0
-0if (mem_cgroup_charge (folio, vma->vm_mm,GFP_KERNEL )) goto oom_free_page;folio_throttle_swaprate (folio,GFP_KERNEL );
Quoted 8 lines
mm/memory.c static vm_fault_t do_anonymous_page(struct vm_fault *vmf) · line 4238
Quoted 6 lines
mm/memory.c +2
-2/* * The memory barrier inside __folio_mark_uptodate makes sure that -* preceding stores to the page contents become visible before -* the set_pte_at() write. +* preceding stores to the folio contents become visible before +* the set_ptes() write. We don't have
set_ptes()yet.
Indeed, that's why I listed the set_ptes() patch set as a hard dependency ;-)
Quoted 74 lines
Quoted 7 lines
mm/memory.c +0
-0*/ __folio_mark_uptodate (folio);@@ -4107,11 +4247,12 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)if (vma->vm_flags &VM_WRITE ) entry =pte_mkwrite (pte_mkdirty (entry));mm/memory.c static vm_fault_t do_anonymous_page(struct vm_fault *vmf) · line 4256
Quoted 10 lines
mm/memory.c +3
-2- vmf->pte =pte_offset_map_lock (vma->vm_mm, vmf->pmd, vmf->address, - &vmf->ptl); + vmf->pte =pte_offset_map_lock (vma->vm_mm, vmf->pmd, addr, &vmf->ptl);if (vmf_pte_changed (vmf)) {update_mmu_tlb (vma, vmf->address, vmf->pte); goto release; + }else if (order >0 &&check_ptes_none (vmf->pte, pgcount) != pgcount) { + goto release; }Need new helper:
if (vmf_pte_range_changed(vmf, nr_pages)) { for (i = 0; i < nr_pages; i++) update_mmu_tlb(vma, vmf->address + PAGE_SIZE * i, vmf->pte + i); goto release; }(It should be fine to call
update_mmu_tlb()even if it's not really necessary.)Quoted 7 lines
mm/memory.c +0
-2ret =check_stable_address_space (vma->vm_mm);@@ -4125,16 +4266,17 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)return handle_userfault (vmf,VM_UFFD_MISSING ); } -inc_mm_counter (vma->vm_mm,MM_ANONPAGES ); -folio_add_new_anon_rmap (folio, vma, vmf->address);mm/memory.c static vm_fault_t do_anonymous_page(struct vm_fault *vmf) · line 4276
Quoted 10 lines
mm/memory.c +5
-2+folio_ref_add (folio, pgcount -1 ); +add_mm_counter (vma->vm_mm,MM_ANONPAGES , pgcount); +folio_add_new_anon_rmap (folio, vma, addr);folio_add_lru_vma (folio, vma); -setpte: +if (uffd_wp) entry =pte_mkuffd_wp (entry); -set_pte_at (vma->vm_mm, vmf->address, vmf->pte, entry); +set_ptes (vma->vm_mm, addr, vmf->pte, entry, pgcount);We would have to do it one by one for now.
mm/memory.c static vm_fault_t do_anonymous_page(struct vm_fault *vmf) · line 4278
Quoted 3 lines
mm/memory.c +1
-1/* No need to invalidate - it was non-present before */ -update_mmu_cache (vma, vmf->address, vmf->pte); +update_mmu_cache_range (vma, addr, vmf->pte, pgcount);Ditto.
How about this (by moving
mk_pte()and its friends here):
...folio_add_lru_vma(folio, vma); for (i = 0; i < nr_pages; i++) { entry = mk_pte(folio_page(folio, i), vma->vm_page_prot); entry = pte_sw_mkyoung(entry); if (vma->vm_flags & VM_WRITE) entry = pte_mkwrite(pte_mkdirty(entry)); setpte: if (uffd_wp) entry = pte_mkuffd_wp(entry); set_pte_at(vma->vm_mm, vmf->address + PAGE_SIZE * i,
vmf->pte+ i, entry);/* No need to invalidate - it was non-present before */ update_mmu_cache(vma, vmf->address + PAGE_SIZE * i, vmf->pte + i); }
Quoted 6 lines
mm/memory.c static vm_fault_t do_anonymous_page(struct vm_fault *vmf)
Quoted 3 lines
mm/memory.c +0
-0unlock:pte_unmap_unlock (vmf->pte, vmf->ptl);return ret;Attaching a small patch in case anything above is not clear. Please take a look. Thanks.
OK, I'll take a look and rework for v3.
Re: [PATCH v2 4/5] mm: FLEXIBLE_THP for improved performance
Quoted 169 lines · Ryan Roberts
Quoted 14 lines · Yu Zhao
Quoted 13 lines · Ryan Roberts
Introduce
FLEXIBLE_THPfeature, which allows anonymous memory to be allocated in large folios of a specified order. All pages of the large folio are pte-mapped during the same page fault, significantly reducing the number of page faults. The number of per-page operations (e.g. ref counting, rmap management lru list management) are also significantly reduced since those ops now become per-folio.The new behaviour is hidden behind the new
FLEXIBLE_THPKconfig, which defaults to disabled for now; there is a long list of todos to makeFLEXIBLE_THProbust with existing features (e.g. compaction, mlock, some madvise ops, etc). These items will be tackled in subsequent patches.Quoted 5 lines
Quoted 3 lines
When enabled, the preferred folio order is as returned by
arch_wants_pte_order(), which may be overridden by the arch as it sees fit. Some architectures (e.g. arm64) can coalsece TLB entries if acoalesce
ACK
Quoted 4 lines
Quoted 2 lines
contiguous set of ptes map physically contigious, naturally aligned
contiguous
ACK
Quoted 22 lines
Quoted 22 lines
memory, so this mechanism allows the architecture to optimize as required.
If the preferred order can't be used (e.g. because the folio would breach the bounds of the vma, or because ptes in the region are already mapped) then we fall back to a suitable lower order.
Signed-off-by Ryan Roberts <[email protected]>--- mm/Kconfig | 10 ++++ mm/memory.c | 168 ++++++++++++++++++++++++++++++++++++++++++++++++---- 2 files changed, 165 insertions(+), 13 deletions(-)mm/Kconfig +0
-0@@ -822,6 +822,16 @@ config READ_ONLY_THP_FOR_FS support of file THPs will be developed in the next few release cycles.Quoted 6 lines
mm/Kconfig config READ_ONLY_THP_FOR_FS · line 828
Quoted 4 lines
mm/Kconfig +4
-0+config FLEXIBLE_THP + bool "Flexible order THP" + depends on TRANSPARENT_HUGEPAGE + default nThe default value is already N.
Is there a coding standard for this? Personally I prefer to make it explicit.
Quoted 93 lines
Quoted 93 lines
mm/Kconfig +6
-0+ help + Use large (bigger than order-0) folios to back anonymous memory where + possible, even if the order of the folio is smaller than the PMD + order. This reduces the number of page faults, as well as other + per-page overheads to improve performance for many workloads. + endif # TRANSPARENT_HUGEPAGE #mm/memory.c +75
-0@@ -3001,6 +3001,116 @@ static vm_fault_t fault_dirty_shared_page(struct vm_fault *vmf)return 0 ; } +#ifdef CONFIG_FLEXIBLE_THP +/* +* Allocates, zeros and returns a folio of the requested order for use as +* anonymous memory. +*/ +static struct folio *alloc_anon_folio (struct vm_area_struct *vma, +unsigned long addr,int order) +{ +gfp_t gfp; +struct folio *folio; + +if (order ==0 ) +return vma_alloc_zeroed_movable_folio (vma, addr); + + gfp =vma_thp_gfp_mask (vma); + folio =vma_alloc_folio (gfp, order, vma, addr, true); +if (folio) +clear_huge_page (&folio->page, addr,folio_nr_pages (folio)); + +return folio; +} + +/* +* Preferred folio order to allocate for anonymous memory. +*/ +#define max_anon_folio_order (vma) arch_wants_pte_order(vma) +#else +#define alloc_anon_folio (vma, addr, order) \ + vma_alloc_zeroed_movable_folio(vma, addr) +#define max_anon_folio_order (vma) 0 +#endif + +/* +* Returns index of first pte that is not none, or nr if all are none. +*/ +static inline int check_ptes_none (pte_t *pte,int nr) +{ +int i; + +for (i =0 ; i < nr; i++) { +if (!pte_none (ptep_get (pte++))) +return i; + } + +return nr; +} + +static int calc_anon_folio_order_alloc (struct vm_fault *vmf,int order) +{ +/* +* The aim here is to determine what size of folio we should allocate +* for this fault. Factors include: +* - Order must not be higher than `order` upon entry +* - Folio must be naturally aligned within VA space +* - Folio must be fully contained inside one pmd entry +* - Folio must not breach boundaries of vma +* - Folio must not overlap any non-none ptes +* +* Additionally, we do not allow order-1 since this breaks assumptions +* elsewhere in the mm; THP pages must be at least order-2 (since they +* store state up to the 3rd struct page subpage), and these pages must +* be THP in order to correctly use pre-existing THP infrastructure such +* as folio_split(). +* +* Note that the caller may or may not choose to lock the pte. If +* unlocked, the result is racy and the user must re-check any overlap +* with non-none ptes under the lock. +*/ + +struct vm_area_struct *vma = vmf->vma; +int nr; +unsigned long addr; +pte_t *pte; +pte_t *first_set =NULL ; +int ret;Quoted 8 lines
mm/memory.c static vm_fault_t fault_dirty_shared_page(struct vm_fault *vmf) · line 3082
Quoted 4 lines
mm/memory.c +4
-0+ + order =min (order,PMD_SHIFT -PAGE_SHIFT ); + +for (; order >1 ; order--) {I'm not sure how we can justify this policy. As an initial step, it'd be a lot easier to sell if we only considered the order of
arch_wants_pte_order()and the order 0.My justification is in the cover letter; I see performance regression (vs the unpatched kernel) when using the policy you suggest. This policy performs much better in my tests. (I'll reply directly to your follow up questions in the cover letter shortly).
Quoted 4 lines
What are your technical concerns about this approach? It is pretty light weight (I only touch each PTE once, regardless of the number of loops). If we have strong technical reasons for reverting to the less performant approach then fair enough, but I'd like to hear the rational first.
Yes, mainly from three different angles:
1. The engineering principle: we'd want to separate the mechanical part and the policy part when attacking something large. This way it'd be easier to root cause any regressions if they happen. In our case, assuming the regression is real, it might actually prove my point
here: I really don't think the two checks (if a vma range fits and if it does, which is unlikely according to your description, if all 64 PTEs are none) caused the regression. My theory is that 64KB itself caused the regression, but smaller sizes made an improvement. If this is really the case, I'd say the fallback policy masked the real problem, which is that 64KB is too large to begin with.
2. The benchmark methodology: I appreciate your effort in doing it, but we also need to consider that the setup is an uncommon scenario. The common scenarios are devices that have been running for weeks without reboots, generally having higher external fragmentation. In addition, for client devices, they are often under memory pressure, which makes fragmentation worse. So we should take the result with a grain of salt, and for that matter, results from after refresh reboots.
3. The technical concern: an ideal policy would consider all three major factors: the h/w features, userspace behaviors and the page allocator behavior. So far we only have the first one handy. The second one is too challenging, so let's forget about it for now. The third one is why I really don't like this best-fit policy. By falling back to smaller orders, we can waste a limited number of physically contiguous pages on wrong vmas (small vmas only), leading to failures to serve large vmas which otherwise would have a higher overall ROI. This can only be addressed within the page allocator: we need to enlighten it to return the highest order available, i.e., not breaking up any higher orders.
I'm not really saying we should never try this fallback policy. I'm just thinking we can leave it for later, probably after we've addressed all the concerns with basic functionality.
Re: [PATCH v2 4/5] mm: FLEXIBLE_THP for improved performance
In reply to Ryan RobertsQuoted 1 line · Ryan Roberts
Introduce
FLEXIBLE_THPfeature, which allows anonymous memory to be
THP is for huge page which is 2M size. We are not huge page here. But I don't have good name either.
Quoted 156 lines
allocated in large folios of a specified order. All pages of the large folio are pte-mapped during the same page fault, significantly reducing the number of page faults. The number of per-page operations (e.g. ref counting, rmap management lru list management) are also significantly reduced since those ops now become per-folio.
The new behaviour is hidden behind the new
FLEXIBLE_THPKconfig, which defaults to disabled for now; there is a long list of todos to makeFLEXIBLE_THProbust with existing features (e.g. compaction, mlock, some madvise ops, etc). These items will be tackled in subsequent patches.When enabled, the preferred folio order is as returned by
arch_wants_pte_order(), which may be overridden by the arch as it sees fit. Some architectures (e.g. arm64) can coalsece TLB entries if a contiguous set of ptes map physically contigious, naturally aligned memory, so this mechanism allows the architecture to optimize as required.If the preferred order can't be used (e.g. because the folio would breach the bounds of the vma, or because ptes in the region are already mapped) then we fall back to a suitable lower order.
Signed-off-by Ryan Roberts <[email protected]>--- mm/Kconfig | 10 ++++ mm/memory.c | 168 ++++++++++++++++++++++++++++++++++++++++++++++++---- 2 files changed, 165 insertions(+), 13 deletions(-)mm/Kconfig +10
-0@@ -822,6 +822,16 @@ config READ_ONLY_THP_FOR_FS support of file THPs will be developed in the next few release cycles. +config FLEXIBLE_THP + bool "Flexible order THP" + depends on TRANSPARENT_HUGEPAGE + default n + help + Use large (bigger than order-0) folios to back anonymous memory where + possible, even if the order of the folio is smaller than the PMD + order. This reduces the number of page faults, as well as other + per-page overheads to improve performance for many workloads. + endif # TRANSPARENT_HUGEPAGE #mm/memory.c +99
-0@@ -3001,6 +3001,116 @@ static vm_fault_t fault_dirty_shared_page(struct vm_fault *vmf)return 0 ; } +#ifdef CONFIG_FLEXIBLE_THP +/* +* Allocates, zeros and returns a folio of the requested order for use as +* anonymous memory. +*/ +static struct folio *alloc_anon_folio (struct vm_area_struct *vma, +unsigned long addr,int order) +{ +gfp_t gfp; +struct folio *folio; + +if (order ==0 ) +return vma_alloc_zeroed_movable_folio (vma, addr); + + gfp =vma_thp_gfp_mask (vma); + folio =vma_alloc_folio (gfp, order, vma, addr, true); +if (folio) +clear_huge_page (&folio->page, addr,folio_nr_pages (folio)); + +return folio; +} + +/* +* Preferred folio order to allocate for anonymous memory. +*/ +#define max_anon_folio_order (vma) arch_wants_pte_order(vma) +#else +#define alloc_anon_folio (vma, addr, order) \ + vma_alloc_zeroed_movable_folio(vma, addr) +#define max_anon_folio_order (vma) 0 +#endif + +/* +* Returns index of first pte that is not none, or nr if all are none. +*/ +static inline int check_ptes_none (pte_t *pte,int nr) +{ +int i; + +for (i =0 ; i < nr; i++) { +if (!pte_none (ptep_get (pte++))) +return i; + } + +return nr; +} + +static int calc_anon_folio_order_alloc (struct vm_fault *vmf,int order) +{ +/* +* The aim here is to determine what size of folio we should allocate +* for this fault. Factors include: +* - Order must not be higher than `order` upon entry +* - Folio must be naturally aligned within VA space +* - Folio must be fully contained inside one pmd entry +* - Folio must not breach boundaries of vma +* - Folio must not overlap any non-none ptes +* +* Additionally, we do not allow order-1 since this breaks assumptions +* elsewhere in the mm; THP pages must be at least order-2 (since they +* store state up to the 3rd struct page subpage), and these pages must +* be THP in order to correctly use pre-existing THP infrastructure such +* as folio_split(). +* +* Note that the caller may or may not choose to lock the pte. If +* unlocked, the result is racy and the user must re-check any overlap +* with non-none ptes under the lock. +*/ + +struct vm_area_struct *vma = vmf->vma; +int nr; +unsigned long addr; +pte_t *pte; +pte_t *first_set =NULL ; +int ret; + + order =min (order,PMD_SHIFT -PAGE_SHIFT ); + +for (; order >1 ; order--) { + nr =1 << order; + addr =ALIGN_DOWN (vmf->address, nr <<PAGE_SHIFT ); + pte = vmf->pte - ((vmf->address - addr) >>PAGE_SHIFT ); + +/* Check vma bounds. */ +if (addr < vma->vm_start || + addr + (nr <<PAGE_SHIFT ) > vma->vm_end) +continue ; + +/* Ptes covered by order already known to be none. */ +if (pte + nr <= first_set) +break ; + +/* Already found set pte in range covered by order. */ +if (pte <= first_set) +continue ; + +/* Need to check if all the ptes are none. */ + ret =check_ptes_none (pte, nr); +if (ret == nr)
mm/memory.c static vm_fault_t fault_dirty_shared_page(struct vm_fault *vmf) · line 3112
Quoted 10 lines
mm/memory.c +10
-0+break ; + + first_set = pte + ret; + } + +if (order ==1 ) + order =0 ; + +return order; +}
The logic in above function should be kept is whether the order fit in vma range.
check_ptes_none() is not accurate here because no page table lock hold and concurrent fault could happen. So may just drop the check here? Check_ptes_none() is done after take the page table lock.
We pick the arch prefered order or order 0 now.
Quoted 81 lines
mm/memory.c +37
-6+/* * Handle write page faults for pages that can be reused in the current vma * @@ -3073,7 +3183,7 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) goto oom;if (is_zero_pfn (pte_pfn (vmf->orig_pte))) { - new_folio =vma_alloc_zeroed_movable_folio (vma, vmf->address); + new_folio =alloc_anon_folio (vma, vmf->address,0 );if (!new_folio) goto oom; }else {@@ -4040,6 +4150,9 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)struct folio *folio;vm_fault_t ret =0 ;pte_t entry; +int order; +int pgcount; +unsigned long addr;/* File mapping without ->vm_ops ? */ if (vma->vm_flags &VM_SHARED )@@ -4081,24 +4194,51 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)pte_unmap_unlock (vmf->pte, vmf->ptl);return handle_userfault (vmf,VM_UFFD_MISSING ); } - goto setpte; +if (uffd_wp) + entry =pte_mkuffd_wp (entry); +set_pte_at (vma->vm_mm, vmf->address, vmf->pte, entry); + +/* No need to invalidate - it was non-present before */ +update_mmu_cache (vma, vmf->address, vmf->pte); + goto unlock; + } + +/* +* If allocating a large folio, determine the biggest suitable order for +* the VMA (e.g. it must not exceed the VMA's bounds, it must not +* overlap with any populated PTEs, etc). We are not under the ptl here +* so we will need to re-check that we are not overlapping any populated +* PTEs once we have the lock. +*/ + order = uffd_wp ?0 :max_anon_folio_order (vma); +if (order >0 ) { + vmf->pte =pte_offset_map (vmf->pmd, vmf->address); + order =calc_anon_folio_order_alloc (vmf, order); +pte_unmap (vmf->pte); } -/* Allocate our own private page. */ +/* Allocate our own private folio. */ if (unlikely (anon_vma_prepare (vma))) goto oom; - folio =vma_alloc_zeroed_movable_folio (vma, vmf->address); + folio =alloc_anon_folio (vma, vmf->address, order); +if (!folio && order >0 ) { + order =0 ; + folio =alloc_anon_folio (vma, vmf->address, order); + }if (!folio) goto oom; + pgcount =1 << order; + addr =ALIGN_DOWN (vmf->address, pgcount <<PAGE_SHIFT ); +if (mem_cgroup_charge (folio, vma->vm_mm,GFP_KERNEL )) goto oom_free_page;folio_throttle_swaprate (folio,GFP_KERNEL );/* * The memory barrier inside __folio_mark_uptodate makes sure that -* preceding stores to the page contents become visible before -* the set_pte_at() write. +* preceding stores to the folio contents become visible before +* the set_ptes() write. */ __folio_mark_uptodate (folio);
mm/memory.c static vm_fault_t do_anonymous_page(struct vm_fault *vmf) · line 4254
Quoted 10 lines
mm/memory.c +2
-2@@ -4107,11 +4247,12 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)if (vma->vm_flags &VM_WRITE ) entry =pte_mkwrite (pte_mkdirty (entry)); - vmf->pte =pte_offset_map_lock (vma->vm_mm, vmf->pmd, vmf->address, - &vmf->ptl); + vmf->pte =pte_offset_map_lock (vma->vm_mm, vmf->pmd, addr, &vmf->ptl);if (vmf_pte_changed (vmf)) {update_mmu_tlb (vma, vmf->address, vmf->pte); goto release; + }else if (order >0 &&check_ptes_none (vmf->pte, pgcount) != pgcount) {
This could be the case that we allocated order 4 page and find a neighbor PTE is filled by concurrent fault. Should we put current folio and fallback to order 0 and try again immedately (goto order 0 allocation instead of return from this function which will go through some page fault path again)?
Regards
Yin, Fengwei
Quoted 27 lines
mm/memory.c +7
-5+ goto release; } ret =check_stable_address_space (vma->vm_mm);@@ -4125,16 +4266,17 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)return handle_userfault (vmf,VM_UFFD_MISSING ); } -inc_mm_counter (vma->vm_mm,MM_ANONPAGES ); -folio_add_new_anon_rmap (folio, vma, vmf->address); +folio_ref_add (folio, pgcount -1 ); +add_mm_counter (vma->vm_mm,MM_ANONPAGES , pgcount); +folio_add_new_anon_rmap (folio, vma, addr);folio_add_lru_vma (folio, vma); -setpte: +if (uffd_wp) entry =pte_mkuffd_wp (entry); -set_pte_at (vma->vm_mm, vmf->address, vmf->pte, entry); +set_ptes (vma->vm_mm, addr, vmf->pte, entry, pgcount);/* No need to invalidate - it was non-present before */ -update_mmu_cache (vma, vmf->address, vmf->pte); +update_mmu_cache_range (vma, addr, vmf->pte, pgcount); unlock:pte_unmap_unlock (vmf->pte, vmf->ptl);return ret;
Re: [PATCH v2 4/5] mm: FLEXIBLE_THP for improved performance
Quoted 5 lines · Fengwei
Quoted 1 line · Ryan Roberts
Introduce
FLEXIBLE_THPfeature, which allows anonymous memory to beTHP is for huge page which is 2M size. We are not huge page here. But I don't have good name either.
Is that really true? On arm64 with 16K pages, huge pages are 32M and with 64K base page, they are 512M. So huge pages already have a variable size. And they sometimes get PTE-mapped. So can't we just think of this as an extension of the THP feature?
Quoted 169 lines
Quoted 157 lines
allocated in large folios of a specified order. All pages of the large folio are pte-mapped during the same page fault, significantly reducing the number of page faults. The number of per-page operations (e.g. ref counting, rmap management lru list management) are also significantly reduced since those ops now become per-folio.
The new behaviour is hidden behind the new
FLEXIBLE_THPKconfig, which defaults to disabled for now; there is a long list of todos to makeFLEXIBLE_THProbust with existing features (e.g. compaction, mlock, some madvise ops, etc). These items will be tackled in subsequent patches.When enabled, the preferred folio order is as returned by
arch_wants_pte_order(), which may be overridden by the arch as it sees fit. Some architectures (e.g. arm64) can coalsece TLB entries if a contiguous set of ptes map physically contigious, naturally aligned memory, so this mechanism allows the architecture to optimize as required.If the preferred order can't be used (e.g. because the folio would breach the bounds of the vma, or because ptes in the region are already mapped) then we fall back to a suitable lower order.
Signed-off-by Ryan Roberts <[email protected]>--- mm/Kconfig | 10 ++++ mm/memory.c | 168 ++++++++++++++++++++++++++++++++++++++++++++++++---- 2 files changed, 165 insertions(+), 13 deletions(-)mm/Kconfig +10
-0@@ -822,6 +822,16 @@ config READ_ONLY_THP_FOR_FS support of file THPs will be developed in the next few release cycles. +config FLEXIBLE_THP + bool "Flexible order THP" + depends on TRANSPARENT_HUGEPAGE + default n + help + Use large (bigger than order-0) folios to back anonymous memory where + possible, even if the order of the folio is smaller than the PMD + order. This reduces the number of page faults, as well as other + per-page overheads to improve performance for many workloads. + endif # TRANSPARENT_HUGEPAGE #mm/memory.c +99
-0@@ -3001,6 +3001,116 @@ static vm_fault_t fault_dirty_shared_page(struct vm_fault *vmf)return 0 ; } +#ifdef CONFIG_FLEXIBLE_THP +/* +* Allocates, zeros and returns a folio of the requested order for use as +* anonymous memory. +*/ +static struct folio *alloc_anon_folio (struct vm_area_struct *vma, +unsigned long addr,int order) +{ +gfp_t gfp; +struct folio *folio; + +if (order ==0 ) +return vma_alloc_zeroed_movable_folio (vma, addr); + + gfp =vma_thp_gfp_mask (vma); + folio =vma_alloc_folio (gfp, order, vma, addr, true); +if (folio) +clear_huge_page (&folio->page, addr,folio_nr_pages (folio)); + +return folio; +} + +/* +* Preferred folio order to allocate for anonymous memory. +*/ +#define max_anon_folio_order (vma) arch_wants_pte_order(vma) +#else +#define alloc_anon_folio (vma, addr, order) \ + vma_alloc_zeroed_movable_folio(vma, addr) +#define max_anon_folio_order (vma) 0 +#endif + +/* +* Returns index of first pte that is not none, or nr if all are none. +*/ +static inline int check_ptes_none (pte_t *pte,int nr) +{ +int i; + +for (i =0 ; i < nr; i++) { +if (!pte_none (ptep_get (pte++))) +return i; + } + +return nr; +} + +static int calc_anon_folio_order_alloc (struct vm_fault *vmf,int order) +{ +/* +* The aim here is to determine what size of folio we should allocate +* for this fault. Factors include: +* - Order must not be higher than `order` upon entry +* - Folio must be naturally aligned within VA space +* - Folio must be fully contained inside one pmd entry +* - Folio must not breach boundaries of vma +* - Folio must not overlap any non-none ptes +* +* Additionally, we do not allow order-1 since this breaks assumptions +* elsewhere in the mm; THP pages must be at least order-2 (since they +* store state up to the 3rd struct page subpage), and these pages must +* be THP in order to correctly use pre-existing THP infrastructure such +* as folio_split(). +* +* Note that the caller may or may not choose to lock the pte. If +* unlocked, the result is racy and the user must re-check any overlap +* with non-none ptes under the lock. +*/ + +struct vm_area_struct *vma = vmf->vma; +int nr; +unsigned long addr; +pte_t *pte; +pte_t *first_set =NULL ; +int ret; + + order =min (order,PMD_SHIFT -PAGE_SHIFT ); + +for (; order >1 ; order--) { + nr =1 << order; + addr =ALIGN_DOWN (vmf->address, nr <<PAGE_SHIFT ); + pte = vmf->pte - ((vmf->address - addr) >>PAGE_SHIFT ); + +/* Check vma bounds. */ +if (addr < vma->vm_start || + addr + (nr <<PAGE_SHIFT ) > vma->vm_end) +continue ; + +/* Ptes covered by order already known to be none. */ +if (pte + nr <= first_set) +break ; + +/* Already found set pte in range covered by order. */ +if (pte <= first_set) +continue ; + +/* Need to check if all the ptes are none. */ + ret =check_ptes_none (pte, nr); +if (ret == nr)mm/memory.c static vm_fault_t fault_dirty_shared_page(struct vm_fault *vmf) · line 3112
Quoted 10 lines
mm/memory.c +10
-0+break ; + + first_set = pte + ret; + } + +if (order ==1 ) + order =0 ; + +return order; +}The logic in above function should be kept is whether the order fit in vma range.
Quoted 3 lines
check_ptes_none()is not accurate here because no page table lock hold and concurrent fault could happen. So may just drop the check here?Check_ptes_none()is done after take the page table lock.
I agree it is just an estimate given the lock is not held; the comment at the top says the same. But I don't think we can wait until after the lock is taken to measure this. We can't hold the lock while allocating the folio and we need a guess at what to allocate. If we don't guess here, we will allocate the biggest, then take the lock, see that it doesn't fit, and exit. Then the system will re-fault and we will follow the exact same path - ending up in live lock.
Quoted 87 lines
We pick the arch prefered order or order 0 now.
Quoted 84 lines
mm/memory.c +37
-6+/* * Handle write page faults for pages that can be reused in the current vma * @@ -3073,7 +3183,7 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) goto oom;if (is_zero_pfn (pte_pfn (vmf->orig_pte))) { - new_folio =vma_alloc_zeroed_movable_folio (vma, vmf->address); + new_folio =alloc_anon_folio (vma, vmf->address,0 );if (!new_folio) goto oom; }else {@@ -4040,6 +4150,9 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)struct folio *folio;vm_fault_t ret =0 ;pte_t entry; +int order; +int pgcount; +unsigned long addr;/* File mapping without ->vm_ops ? */ if (vma->vm_flags &VM_SHARED )@@ -4081,24 +4194,51 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)pte_unmap_unlock (vmf->pte, vmf->ptl);return handle_userfault (vmf,VM_UFFD_MISSING ); } - goto setpte; +if (uffd_wp) + entry =pte_mkuffd_wp (entry); +set_pte_at (vma->vm_mm, vmf->address, vmf->pte, entry); + +/* No need to invalidate - it was non-present before */ +update_mmu_cache (vma, vmf->address, vmf->pte); + goto unlock; + } + +/* +* If allocating a large folio, determine the biggest suitable order for +* the VMA (e.g. it must not exceed the VMA's bounds, it must not +* overlap with any populated PTEs, etc). We are not under the ptl here +* so we will need to re-check that we are not overlapping any populated +* PTEs once we have the lock. +*/ + order = uffd_wp ?0 :max_anon_folio_order (vma); +if (order >0 ) { + vmf->pte =pte_offset_map (vmf->pmd, vmf->address); + order =calc_anon_folio_order_alloc (vmf, order); +pte_unmap (vmf->pte); } -/* Allocate our own private page. */ +/* Allocate our own private folio. */ if (unlikely (anon_vma_prepare (vma))) goto oom; - folio =vma_alloc_zeroed_movable_folio (vma, vmf->address); + folio =alloc_anon_folio (vma, vmf->address, order); +if (!folio && order >0 ) { + order =0 ; + folio =alloc_anon_folio (vma, vmf->address, order); + }if (!folio) goto oom; + pgcount =1 << order; + addr =ALIGN_DOWN (vmf->address, pgcount <<PAGE_SHIFT ); +if (mem_cgroup_charge (folio, vma->vm_mm,GFP_KERNEL )) goto oom_free_page;folio_throttle_swaprate (folio,GFP_KERNEL );/* * The memory barrier inside __folio_mark_uptodate makes sure that -* preceding stores to the page contents become visible before -* the set_pte_at() write. +* preceding stores to the folio contents become visible before +* the set_ptes() write. */ __folio_mark_uptodate (folio);@@ -4107,11 +4247,12 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)if (vma->vm_flags &VM_WRITE ) entry =pte_mkwrite (pte_mkdirty (entry));
Quoted 11 lines
mm/memory.c static vm_fault_t do_anonymous_page(struct vm_fault *vmf) · line 4254
Quoted 7 lines
mm/memory.c +2
-2- vmf->pte =pte_offset_map_lock (vma->vm_mm, vmf->pmd, vmf->address, - &vmf->ptl); + vmf->pte =pte_offset_map_lock (vma->vm_mm, vmf->pmd, addr, &vmf->ptl);if (vmf_pte_changed (vmf)) {update_mmu_tlb (vma, vmf->address, vmf->pte); goto release; + }else if (order >0 &&check_ptes_none (vmf->pte, pgcount) != pgcount) {This could be the case that we allocated order 4 page and find a neighbor PTE is filled by concurrent fault. Should we put current folio and fallback to order 0 and try again immedately (goto order 0 allocation instead of return from this function which will go through some page fault path again)?
That's how it worked in v1, but I had review comments from Yang Shi asking me to re-fault instead. This approach is certainly cleaner from a code point of view. And I expect races of that nature will be rare.
Quoted 32 lines
Regards
Yin, FengweiQuoted 27 lines
mm/memory.c +7
-5+ goto release; } ret =check_stable_address_space (vma->vm_mm);@@ -4125,16 +4266,17 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)return handle_userfault (vmf,VM_UFFD_MISSING ); } -inc_mm_counter (vma->vm_mm,MM_ANONPAGES ); -folio_add_new_anon_rmap (folio, vma, vmf->address); +folio_ref_add (folio, pgcount -1 ); +add_mm_counter (vma->vm_mm,MM_ANONPAGES , pgcount); +folio_add_new_anon_rmap (folio, vma, addr);folio_add_lru_vma (folio, vma); -setpte: +if (uffd_wp) entry =pte_mkuffd_wp (entry); -set_pte_at (vma->vm_mm, vmf->address, vmf->pte, entry); +set_ptes (vma->vm_mm, addr, vmf->pte, entry, pgcount);/* No need to invalidate - it was non-present before */ -update_mmu_cache (vma, vmf->address, vmf->pte); +update_mmu_cache_range (vma, addr, vmf->pte, pgcount); unlock:pte_unmap_unlock (vmf->pte, vmf->ptl);return ret;
Re: [PATCH v2 4/5] mm: FLEXIBLE_THP for improved performance
Quoted 11 lines · Ryan Roberts
Quoted 5 lines · Fengwei
Quoted 1 line · Ryan Roberts
Introduce
FLEXIBLE_THPfeature, which allows anonymous memory to beTHP is for huge page which is 2M size. We are not huge page here. But I don't have good name either.
Is that really true? On arm64 with 16K pages, huge pages are 32M and with 64K base page, they are 512M. So huge pages already have a variable size. And they sometimes get PTE-mapped. So can't we just think of this as an extension of the THP feature?
My understanding is the THP has several fixed size on different arch. The 32K or 16K which could be picked here are not THP size.
Quoted 174 lines
Quoted 170 lines
Quoted 158 lines
allocated in large folios of a specified order. All pages of the large folio are pte-mapped during the same page fault, significantly reducing the number of page faults. The number of per-page operations (e.g. ref counting, rmap management lru list management) are also significantly reduced since those ops now become per-folio.
The new behaviour is hidden behind the new
FLEXIBLE_THPKconfig, which defaults to disabled for now; there is a long list of todos to makeFLEXIBLE_THProbust with existing features (e.g. compaction, mlock, some madvise ops, etc). These items will be tackled in subsequent patches.When enabled, the preferred folio order is as returned by
arch_wants_pte_order(), which may be overridden by the arch as it sees fit. Some architectures (e.g. arm64) can coalsece TLB entries if a contiguous set of ptes map physically contigious, naturally aligned memory, so this mechanism allows the architecture to optimize as required.If the preferred order can't be used (e.g. because the folio would breach the bounds of the vma, or because ptes in the region are already mapped) then we fall back to a suitable lower order.
Signed-off-by Ryan Roberts <[email protected]>--- mm/Kconfig | 10 ++++ mm/memory.c | 168 ++++++++++++++++++++++++++++++++++++++++++++++++---- 2 files changed, 165 insertions(+), 13 deletions(-)mm/Kconfig +10
-0@@ -822,6 +822,16 @@ config READ_ONLY_THP_FOR_FS support of file THPs will be developed in the next few release cycles. +config FLEXIBLE_THP + bool "Flexible order THP" + depends on TRANSPARENT_HUGEPAGE + default n + help + Use large (bigger than order-0) folios to back anonymous memory where + possible, even if the order of the folio is smaller than the PMD + order. This reduces the number of page faults, as well as other + per-page overheads to improve performance for many workloads. + endif # TRANSPARENT_HUGEPAGE #mm/memory.c +99
-0@@ -3001,6 +3001,116 @@ static vm_fault_t fault_dirty_shared_page(struct vm_fault *vmf)return 0 ; } +#ifdef CONFIG_FLEXIBLE_THP +/* +* Allocates, zeros and returns a folio of the requested order for use as +* anonymous memory. +*/ +static struct folio *alloc_anon_folio (struct vm_area_struct *vma, +unsigned long addr,int order) +{ +gfp_t gfp; +struct folio *folio; + +if (order ==0 ) +return vma_alloc_zeroed_movable_folio (vma, addr); + + gfp =vma_thp_gfp_mask (vma); + folio =vma_alloc_folio (gfp, order, vma, addr, true); +if (folio) +clear_huge_page (&folio->page, addr,folio_nr_pages (folio)); + +return folio; +} + +/* +* Preferred folio order to allocate for anonymous memory. +*/ +#define max_anon_folio_order (vma) arch_wants_pte_order(vma) +#else +#define alloc_anon_folio (vma, addr, order) \ + vma_alloc_zeroed_movable_folio(vma, addr) +#define max_anon_folio_order (vma) 0 +#endif + +/* +* Returns index of first pte that is not none, or nr if all are none. +*/ +static inline int check_ptes_none (pte_t *pte,int nr) +{ +int i; + +for (i =0 ; i < nr; i++) { +if (!pte_none (ptep_get (pte++))) +return i; + } + +return nr; +} + +static int calc_anon_folio_order_alloc (struct vm_fault *vmf,int order) +{ +/* +* The aim here is to determine what size of folio we should allocate +* for this fault. Factors include: +* - Order must not be higher than `order` upon entry +* - Folio must be naturally aligned within VA space +* - Folio must be fully contained inside one pmd entry +* - Folio must not breach boundaries of vma +* - Folio must not overlap any non-none ptes +* +* Additionally, we do not allow order-1 since this breaks assumptions +* elsewhere in the mm; THP pages must be at least order-2 (since they +* store state up to the 3rd struct page subpage), and these pages must +* be THP in order to correctly use pre-existing THP infrastructure such +* as folio_split(). +* +* Note that the caller may or may not choose to lock the pte. If +* unlocked, the result is racy and the user must re-check any overlap +* with non-none ptes under the lock. +*/ + +struct vm_area_struct *vma = vmf->vma; +int nr; +unsigned long addr; +pte_t *pte; +pte_t *first_set =NULL ; +int ret; + + order =min (order,PMD_SHIFT -PAGE_SHIFT ); + +for (; order >1 ; order--) { + nr =1 << order; + addr =ALIGN_DOWN (vmf->address, nr <<PAGE_SHIFT ); + pte = vmf->pte - ((vmf->address - addr) >>PAGE_SHIFT ); + +/* Check vma bounds. */ +if (addr < vma->vm_start || + addr + (nr <<PAGE_SHIFT ) > vma->vm_end) +continue ; + +/* Ptes covered by order already known to be none. */ +if (pte + nr <= first_set) +break ; + +/* Already found set pte in range covered by order. */ +if (pte <= first_set) +continue ; + +/* Need to check if all the ptes are none. */ + ret =check_ptes_none (pte, nr); +if (ret == nr)mm/memory.c static vm_fault_t fault_dirty_shared_page(struct vm_fault *vmf) · line 3112
Quoted 10 lines
mm/memory.c +10
-0+break ; + + first_set = pte + ret; + } + +if (order ==1 ) + order =0 ; + +return order; +}The logic in above function should be kept is whether the order fit in vma range.
Quoted 3 lines
check_ptes_none()is not accurate here because no page table lock hold and concurrent fault could happen. So may just drop the check here?Check_ptes_none()is done after take the page table lock.
Quoted 6 lines
I agree it is just an estimate given the lock is not held; the comment at the top says the same. But I don't think we can wait until after the lock is taken to measure this. We can't hold the lock while allocating the folio and we need a guess at what to allocate. If we don't guess here, we will allocate the biggest, then take the lock, see that it doesn't fit, and exit. Then the system will re-fault and we will follow the exact same path - ending up in live lock.
It will not if we try order0 immediately. But see my comments to the refault.
Quoted 100 lines
Quoted 88 lines
We pick the arch prefered order or order 0 now.
Quoted 84 lines
mm/memory.c +37
-6+/* * Handle write page faults for pages that can be reused in the current vma * @@ -3073,7 +3183,7 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) goto oom;if (is_zero_pfn (pte_pfn (vmf->orig_pte))) { - new_folio =vma_alloc_zeroed_movable_folio (vma, vmf->address); + new_folio =alloc_anon_folio (vma, vmf->address,0 );if (!new_folio) goto oom; }else {@@ -4040,6 +4150,9 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)struct folio *folio;vm_fault_t ret =0 ;pte_t entry; +int order; +int pgcount; +unsigned long addr;/* File mapping without ->vm_ops ? */ if (vma->vm_flags &VM_SHARED )@@ -4081,24 +4194,51 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)pte_unmap_unlock (vmf->pte, vmf->ptl);return handle_userfault (vmf,VM_UFFD_MISSING ); } - goto setpte; +if (uffd_wp) + entry =pte_mkuffd_wp (entry); +set_pte_at (vma->vm_mm, vmf->address, vmf->pte, entry); + +/* No need to invalidate - it was non-present before */ +update_mmu_cache (vma, vmf->address, vmf->pte); + goto unlock; + } + +/* +* If allocating a large folio, determine the biggest suitable order for +* the VMA (e.g. it must not exceed the VMA's bounds, it must not +* overlap with any populated PTEs, etc). We are not under the ptl here +* so we will need to re-check that we are not overlapping any populated +* PTEs once we have the lock. +*/ + order = uffd_wp ?0 :max_anon_folio_order (vma); +if (order >0 ) { + vmf->pte =pte_offset_map (vmf->pmd, vmf->address); + order =calc_anon_folio_order_alloc (vmf, order); +pte_unmap (vmf->pte); } -/* Allocate our own private page. */ +/* Allocate our own private folio. */ if (unlikely (anon_vma_prepare (vma))) goto oom; - folio =vma_alloc_zeroed_movable_folio (vma, vmf->address); + folio =alloc_anon_folio (vma, vmf->address, order); +if (!folio && order >0 ) { + order =0 ; + folio =alloc_anon_folio (vma, vmf->address, order); + }if (!folio) goto oom; + pgcount =1 << order; + addr =ALIGN_DOWN (vmf->address, pgcount <<PAGE_SHIFT ); +if (mem_cgroup_charge (folio, vma->vm_mm,GFP_KERNEL )) goto oom_free_page;folio_throttle_swaprate (folio,GFP_KERNEL );/* * The memory barrier inside __folio_mark_uptodate makes sure that -* preceding stores to the page contents become visible before -* the set_pte_at() write. +* preceding stores to the folio contents become visible before +* the set_ptes() write. */ __folio_mark_uptodate (folio);@@ -4107,11 +4247,12 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)if (vma->vm_flags &VM_WRITE ) entry =pte_mkwrite (pte_mkdirty (entry));Quoted 11 lines
mm/memory.c static vm_fault_t do_anonymous_page(struct vm_fault *vmf) · line 4254
Quoted 7 lines
mm/memory.c +2
-2- vmf->pte =pte_offset_map_lock (vma->vm_mm, vmf->pmd, vmf->address, - &vmf->ptl); + vmf->pte =pte_offset_map_lock (vma->vm_mm, vmf->pmd, addr, &vmf->ptl);if (vmf_pte_changed (vmf)) {update_mmu_tlb (vma, vmf->address, vmf->pte); goto release; + }else if (order >0 &&check_ptes_none (vmf->pte, pgcount) != pgcount) {This could be the case that we allocated order 4 page and find a neighbor PTE is filled by concurrent fault. Should we put current folio and fallback to order 0 and try again immedately (goto order 0 allocation instead of return from this function which will go through some page fault path again)?
Quoted 3 lines
That's how it worked in v1, but I had review comments from Yang Shi asking me to re-fault instead. This approach is certainly cleaner from a code point of view. And I expect races of that nature will be rare.
I must miss that discussion in v1. My bad. I should jump in that discussion. So I will drop my comment here even I still think we should avoid refault. I don't want the comment back and forth.
Regards
Yin, Fengwei
Quoted 34 lines
Quoted 33 lines
Regards
Yin, FengweiQuoted 27 lines
mm/memory.c +7
-5+ goto release; } ret =check_stable_address_space (vma->vm_mm);@@ -4125,16 +4266,17 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)return handle_userfault (vmf,VM_UFFD_MISSING ); } -inc_mm_counter (vma->vm_mm,MM_ANONPAGES ); -folio_add_new_anon_rmap (folio, vma, vmf->address); +folio_ref_add (folio, pgcount -1 ); +add_mm_counter (vma->vm_mm,MM_ANONPAGES , pgcount); +folio_add_new_anon_rmap (folio, vma, addr);folio_add_lru_vma (folio, vma); -setpte: +if (uffd_wp) entry =pte_mkuffd_wp (entry); -set_pte_at (vma->vm_mm, vmf->address, vmf->pte, entry); +set_ptes (vma->vm_mm, addr, vmf->pte, entry, pgcount);/* No need to invalidate - it was non-present before */ -update_mmu_cache (vma, vmf->address, vmf->pte); +update_mmu_cache_range (vma, addr, vmf->pte, pgcount); unlock:pte_unmap_unlock (vmf->pte, vmf->ptl);return ret;
Re: [PATCH v2 4/5] mm: FLEXIBLE_THP for improved performance
In reply to Ryan RobertsQuoted 11 lines · Ryan Roberts
Quoted 5 lines · Fengwei
Quoted 1 line · Ryan Roberts
Introduce
FLEXIBLE_THPfeature, which allows anonymous memory to beTHP is for huge page which is 2M size. We are not huge page here. But I don't have good name either.
Is that really true? On arm64 with 16K pages, huge pages are 32M and with 64K base page, they are 512M. So huge pages already have a variable size. And they sometimes get PTE-mapped. So can't we just think of this as an extension of the THP feature?
The confusing thing is that we have counters for the number of THP allocated (and number of THP mapped), and for those we always use PMD-size folios.
If we must have a config option, then this is ANON_LARGE_FOLIOS.
But why do we need a config option? We don't have one for the page cache, and we're better off for it. Yes, it depends on CONFIG_TRANSPARENT_HUGEPAGE today, but that's more of an accidental heritage, and it'd be great to do away with that dependency eventually.
Hardware support isn't needed. Large folios benefit us from a software point of view. if we need a chicken bit, we can edit the source code to not create anon folios larger than order 0.
Re: [PATCH v2 4/5] mm: FLEXIBLE_THP for improved performance
Quoted 8 lines · Matthew Wilcox
Quoted 7 lines · Ryan Roberts
Quoted 5 lines · Fengwei
Quoted 1 line · Ryan Roberts
Introduce
FLEXIBLE_THPfeature, which allows anonymous memory to beTHP is for huge page which is 2M size. We are not huge page here. But I don't have good name either.
Quoted 8 lines
Quoted 4 lines
Is that really true? On arm64 with 16K pages, huge pages are 32M and with 64K base page, they are 512M. So huge pages already have a variable size. And they sometimes get PTE-mapped. So can't we just think of this as an extension of the THP feature?
The confusing thing is that we have counters for the number of THP allocated (and number of THP mapped), and for those we always use PMD-size folios.
OK fair point. I really don't have a strong opinion on the name - I changed it from LARGE_ANON_FOLIO because Yu was suggesting it should be tied to THP. So I'm happy to change it back to LARGE_ANON_FOLIO (or something else) if that's the concensus. But I expect I'll end up in a game of ping-pong. So I'm going to keep this name for now and focus on converging the actual implementation to something that is agreeable. Once we are there, we can argue about the name.
Quoted 11 lines
If we must have a config option, then this is
ANON_LARGE_FOLIOS.But why do we need a config option? We don't have one for the page cache, and we're better off for it. Yes, it depends on
CONFIG_TRANSPARENT_HUGEPAGEtoday, but that's more of an accidental heritage, and it'd be great to do away with that dependency eventually.Hardware support isn't needed. Large folios benefit us from a software point of view. if we need a chicken bit, we can edit the source code to not create anon folios larger than order 0.
From my PoV it's about managing risk; there are currently parts of the mm that will interact poorly with large pte-mapped folios (madvise, compaction, ...). We want to incrementally fix that stuff, but until it's all fixed, we can't deploy this as always-on. Further down the line when things are more complete and there is more test coverage, we could remove the Kconfig or default it to enabled.
Re: [PATCH v2 4/5] mm: FLEXIBLE_THP for improved performance
Quoted 11 lines · Ryan Roberts
Quoted 3 lines · Matthew Wilcox
The confusing thing is that we have counters for the number of THP allocated (and number of THP mapped), and for those we always use PMD-size folios.
OK fair point. I really don't have a strong opinion on the name - I changed it from
LARGE_ANON_FOLIObecause Yu was suggesting it should be tied to THP. So I'm happy to change it back toLARGE_ANON_FOLIO(or something else) if that's the concensus. But I expect I'll end up in a game of ping-pong. So I'm going to keep this name for now and focus on converging the actual implementation to something that is agreeable. Once we are there, we can argue about the name.
I didn't see Yu arguing for changing the name of the config options, just having far fewer of them.
Quoted 11 lines
Quoted 10 lines
If we must have a config option, then this is
ANON_LARGE_FOLIOS.But why do we need a config option? We don't have one for the page cache, and we're better off for it. Yes, it depends on
CONFIG_TRANSPARENT_HUGEPAGEtoday, but that's more of an accidental heritage, and it'd be great to do away with that dependency eventually.Hardware support isn't needed. Large folios benefit us from a software point of view. if we need a chicken bit, we can edit the source code to not create anon folios larger than order 0.
Quoted 5 lines
Quoted 1 line
From my PoV it's about managing risk; there are currently parts of the mm that
will interact poorly with large pte-mapped folios (madvise, compaction, ...). We want to incrementally fix that stuff, but until it's all fixed, we can't deploy this as always-on. Further down the line when things are more complete and there is more test coverage, we could remove the
Kconfigor default it to enabled.
We have to fix those places with the bad interactions, not merge a Kconfig option that lets you turn it on to experiment. That's how you get a bad reputation and advice to disable a config option. We had that for years with CONFIG_TRANSPARENT_HUGEPAGE; people tried it out early on, found the performance problems, and all these years later we still have articles being published that say to turn it off.
By all means, we can have a golden patchset that we all agree is the one to use for finding problems, and we can merge the pre-enabling work "We don't have large anonymous folios yet, but when we do, this will need to iterate over each page in the folio".
Re: [PATCH v2 4/5] mm: FLEXIBLE_THP for improved performance
In reply to Ryan RobertsQuoted 6 lines · Ryan Roberts
Introduce
FLEXIBLE_THPfeature, which allows anonymous memory to be allocated in large folios of a specified order. All pages of the large folio are pte-mapped during the same page fault, significantly reducing the number of page faults. The number of per-page operations (e.g. ref counting, rmap management lru list management) are also significantly reduced since those ops now become per-folio.
I likes the idea to share as much code as possible between large (anonymous) folio and THP. Finally, THP becomes just a special kind of large folio.
Although we can use smaller page order for FLEXIBLE_THP, it's hard to avoid internal fragmentation completely. So, I think that finally we will need to provide a mechanism for the users to opt out, e.g., something like "always madvise never" via /sys/kernel/mm/transparent_hugepage/enabled. I'm not sure whether it's a good idea to reuse the existing interface of THP.
Best Regards,
Huang, Ying
Re: [PATCH v2 4/5] mm: FLEXIBLE_THP for improved performance
Quoted 13 lines · Ying
Quoted 6 lines · Ryan Roberts
Introduce
FLEXIBLE_THPfeature, which allows anonymous memory to be allocated in large folios of a specified order. All pages of the large folio are pte-mapped during the same page fault, significantly reducing the number of page faults. The number of per-page operations (e.g. ref counting, rmap management lru list management) are also significantly reduced since those ops now become per-folio.I likes the idea to share as much code as possible between large (anonymous) folio and THP. Finally, THP becomes just a special kind of large folio.
Quoted 6 lines
Although we can use smaller page order for
FLEXIBLE_THP, it's hard to avoid internal fragmentation completely. So, I think that finally we will need to provide a mechanism for the users to opt out, e.g., something like "always madvise never" via/sys/kernel/mm/transparent_hugepage/enabled. I'm not sure whether it's a good idea to reuse the existing interface of THP.
I wouldn't want to tie this to the existing interface, simply because that implies that we would want to follow the "always" and "madvise" advice too; That means that on a thp=madvise system (which is certainly the case for android and other client systems) we would have to disable large anon folios for VMAs that haven't explicitly opted in. That breaks the intention that this should be an invisible performance boost. I think it's important to set the policy for use of THP separately to use of large anon folios.
I could be persuaded on the merrits of a new runtime enable/disable interface if there is concensus.
Quoted 3 lines
Best Regards,
Huang, Ying
Re: [PATCH v2 4/5] mm: FLEXIBLE_THP for improved performance
Quoted 21 lines · Ryan Roberts
Quoted 13 lines · Ying
Quoted 6 lines · Ryan Roberts
Introduce
FLEXIBLE_THPfeature, which allows anonymous memory to be allocated in large folios of a specified order. All pages of the large folio are pte-mapped during the same page fault, significantly reducing the number of page faults. The number of per-page operations (e.g. ref counting, rmap management lru list management) are also significantly reduced since those ops now become per-folio.I likes the idea to share as much code as possible between large (anonymous) folio and THP. Finally, THP becomes just a special kind of large folio.
Quoted 6 lines
Although we can use smaller page order for
FLEXIBLE_THP, it's hard to avoid internal fragmentation completely. So, I think that finally we will need to provide a mechanism for the users to opt out, e.g., something like "always madvise never" via/sys/kernel/mm/transparent_hugepage/enabled. I'm not sure whether it's a good idea to reuse the existing interface of THP.
Quoted 6 lines
I wouldn't want to tie this to the existing interface, simply because that implies that we would want to follow the "always" and "madvise" advice too; That means that on a thp=madvise system (which is certainly the case for android and other client systems) we would have to disable large anon folios for VMAs that haven't explicitly opted in. That breaks the intention that this should be an invisible performance boost. I think it's important to set the policy for use of
It will never ever be a completely invisible performance boost, just like ordinary THP.
Using the exact same existing toggle is the right thing to do. If someone specify "never" or "madvise", then do exactly that.
It might make sense to have more modes or additional toggles, but "madvise=never" means no memory waste.
I remember I raised it already in the past, but you *absolutely* have to respect the MADV_NOHUGEPAGE flag. There is user space out there (for example, userfaultfd) that doesn't want the kernel to populate any additional page tables. So if you have to respect that already, then also respect MADV_HUGEPAGE, simple.
Quoted 4 lines
THP separately to use of large anon folios.
I could be persuaded on the merrits of a new runtime enable/disable interface if there is concensus.
There would have to be very good reason for a completely separate control. Bypassing MADV_NOHUGEPAGE or "madvise=never" simply because we add a "flexible" before the THP sounds broken.
Signature · 3 lines
Cheers, David / dhildenb
Re: [PATCH v2 4/5] mm: FLEXIBLE_THP for improved performance
Quoted 16 lines · David Hildenbrand
Quoted 8 lines · Ryan Roberts
Quoted 6 lines · Ying
Although we can use smaller page order for
FLEXIBLE_THP, it's hard to avoid internal fragmentation completely. So, I think that finally we will need to provide a mechanism for the users to opt out, e.g., something like "always madvise never" via/sys/kernel/mm/transparent_hugepage/enabled. I'm not sure whether it's a good idea to reuse the existing interface of THP.Quoted 6 lines
I wouldn't want to tie this to the existing interface, simply because that implies that we would want to follow the "always" and "madvise" advice too; That means that on a thp=madvise system (which is certainly the case for android and other client systems) we would have to disable large anon folios for VMAs that haven't explicitly opted in. That breaks the intention that this should be an invisible performance boost. I think it's important to set the policy for use of
Quoted 8 lines
It will never ever be a completely invisible performance boost, just like ordinary THP.
Using the exact same existing toggle is the right thing to do. If someone specify "never" or "madvise", then do exactly that.
It might make sense to have more modes or additional toggles, but "madvise=never" means no memory waste.
I hate the existing mechanisms. They are an abdication of our responsibility, and an attempt to blame the user (be it the sysadmin or the programmer) of our code for using it wrongly. We should not replicate this mistake.
Our code should be auto-tuning. I posted a long, detailed outline here:
https://lore.kernel.org/linux-mm/Y%[email protected]/
Quoted 5 lines
I remember I raised it already in the past, but you *absolutely* have to respect the
MADV_NOHUGEPAGEflag. There is user space out there (for example, userfaultfd) that doesn't want the kernel to populate any additional page tables. So if you have to respect that already, then also respectMADV_HUGEPAGE, simple.
Possibly having uffd enabled on a VMA should disable using large folios, I can get behind that. But the notion that userspace knows what it's doing ... hahaha. Just ignore the madvise flags. Userspace doesn't know what it's doing.
Re: [PATCH v2 4/5] mm: FLEXIBLE_THP for improved performance
Quoted 23 lines · Matthew Wilcox
Quoted 22 lines · David Hildenbrand
Quoted 8 lines · Ryan Roberts
Quoted 6 lines · Ying
Although we can use smaller page order for
FLEXIBLE_THP, it's hard to avoid internal fragmentation completely. So, I think that finally we will need to provide a mechanism for the users to opt out, e.g., something like "always madvise never" via/sys/kernel/mm/transparent_hugepage/enabled. I'm not sure whether it's a good idea to reuse the existing interface of THP.Quoted 6 lines
I wouldn't want to tie this to the existing interface, simply because that implies that we would want to follow the "always" and "madvise" advice too; That means that on a thp=madvise system (which is certainly the case for android and other client systems) we would have to disable large anon folios for VMAs that haven't explicitly opted in. That breaks the intention that this should be an invisible performance boost. I think it's important to set the policy for use of
It will never ever be a completely invisible performance boost, just like ordinary THP.
Using the exact same existing toggle is the right thing to do. If someone specify "never" or "madvise", then do exactly that.
Quoted 7 lines
Quoted 2 lines
It might make sense to have more modes or additional toggles, but "madvise=never" means no memory waste.
I hate the existing mechanisms. They are an abdication of our responsibility, and an attempt to blame the user (be it the sysadmin or the programmer) of our code for using it wrongly. We should not replicate this mistake.
I don't agree regarding the programmer responsibility. In some cases the programmer really doesn't want to get more memory populated than requested -- and knows exactly why setting MADV_NOHUGEPAGE is the right thing to do.
Regarding the madvise=never/madvise/always (sys admin decision), memory waste (and nailing down bugs or working around them in customer setups) have been very good reasons to let the admin have a word.
Quoted 4 lines
Our code should be auto-tuning. I posted a long, detailed outline here:
https://lore.kernel.org/linux-mm/Y%[email protected]/
Well, "auto-tuning" also should be perfect for everybody, but once reality strikes you know it isn't.
If people don't feel like using THP, let them have a word. The "madvise" config option is probably more controversial. But the "always vs. never" absolutely makes sense to me.
Quoted 7 lines
Quoted 5 lines
I remember I raised it already in the past, but you *absolutely* have to respect the
MADV_NOHUGEPAGEflag. There is user space out there (for example, userfaultfd) that doesn't want the kernel to populate any additional page tables. So if you have to respect that already, then also respectMADV_HUGEPAGE, simple.Possibly having uffd enabled on a VMA should disable using large folios,
There are cases where we enable uffd *after* already touching memory (postcopy live migration in QEMU being the famous example). That doesn't fly.
Quoted 3 lines
I can get behind that. But the notion that userspace knows what it's doing ... hahaha. Just ignore the madvise flags. Userspace doesn't know what it's doing.
If user space sets MADV_NOHUGEPAGE, it exactly knows what it is doing ... in some cases. And these include cases I care about messing with sparse VM memory :)
I have strong opinions against populating more than required when user space set MADV_NOHUGEPAGE.
Signature · 9 lines
Cheers, David / dhildenb _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Re: [PATCH v2 4/5] mm: FLEXIBLE_THP for improved performance
Quoted 69 lines · David Hildenbrand
Quoted 25 lines · Matthew Wilcox
Quoted 24 lines · David Hildenbrand
Quoted 8 lines · Ryan Roberts
Quoted 6 lines · Ying
Although we can use smaller page order for
FLEXIBLE_THP, it's hard to avoid internal fragmentation completely. So, I think that finally we will need to provide a mechanism for the users to opt out, e.g., something like "always madvise never" via /sys/kernel/mm/transparent_hugepage/enabled. I'm not sure whether it's a good idea to reuse the existing interface of THP.Quoted 8 lines
I wouldn't want to tie this to the existing interface, simply because that implies that we would want to follow the "always" and "madvise" advice too; That
means that on a thp=madvise system (which is certainly the case for android and other client systems) we would have to disable large anon folios for VMAs that haven't explicitly opted in. That breaks the intention that this should be an invisible performance boost. I think it's important to set the policy for use ofIt will never ever be a completely invisible performance boost, just like ordinary THP.
Using the exact same existing toggle is the right thing to do. If someone specify "never" or "madvise", then do exactly that.
Quoted 7 lines
Quoted 2 lines
It might make sense to have more modes or additional toggles, but "madvise=never" means no memory waste.
I hate the existing mechanisms. They are an abdication of our responsibility, and an attempt to blame the user (be it the sysadmin or the programmer) of our code for using it wrongly. We should not replicate this mistake.
I don't agree regarding the programmer responsibility. In some cases the programmer really doesn't want to get more memory populated than requested -- and knows exactly why setting
MADV_NOHUGEPAGEis the right thing to do.Regarding the madvise=never/madvise/always (sys admin decision), memory waste (and nailing down bugs or working around them in customer setups) have been very good reasons to let the admin have a word.
Quoted 4 lines
Our code should be auto-tuning. I posted a long, detailed outline here:
https://lore.kernel.org/linux-mm/Y%[email protected]/Well, "auto-tuning" also should be perfect for everybody, but once reality strikes you know it isn't.
If people don't feel like using THP, let them have a word. The "madvise" config option is probably more controversial. But the "always vs. never" absolutely makes sense to me.
Quoted 7 lines
Quoted 5 lines
I remember I raised it already in the past, but you *absolutely* have to respect the
MADV_NOHUGEPAGEflag. There is user space out there (for example, userfaultfd) that doesn't want the kernel to populate any additional page tables. So if you have to respect that already, then also respectMADV_HUGEPAGE, simple.Possibly having uffd enabled on a VMA should disable using large folios,
There are cases where we enable uffd *after* already touching memory (postcopy live migration in QEMU being the famous example). That doesn't fly.
Quoted 3 lines
I can get behind that. But the notion that userspace knows what it's doing ... hahaha. Just ignore the madvise flags. Userspace doesn't know what it's doing.
Quoted 5 lines
If user space sets
MADV_NOHUGEPAGE, it exactly knows what it is doing ... in some cases. And these include cases I care about messing with sparse VM memory :)I have strong opinions against populating more than required when user space set
MADV_NOHUGEPAGE.
I can see your point about honouring MADV_NOHUGEPAGE, so think that it is reasonable to fallback to allocating an order-0 page in a VMA that has it set. The app has gone out of its way to explicitly set it, after all.
I think the correct behaviour for the global thp controls (cmdline and sysfs) are less obvious though. I could get on board with disabling large anon folios globally when thp="never". But for other situations, I would prefer to keep large anon folios enabled (treat "madvise" as "always"), with the argument that their order is much smaller than traditional THP and therefore the internal fragmentation is significantly reduced. I really don't want to end up with user space ever having to opt-in (with MADV_HUGEPAGE) to see the benefits of large anon folios.
I still feel that it would be better for the thp and large anon folio controls to be independent though - what's the argument for tying them together?
Quoted 1 line
Re: [PATCH v2 4/5] mm: FLEXIBLE_THP for improved performance
Quoted 80 lines · Ryan Roberts
Quoted 69 lines · David Hildenbrand
Quoted 25 lines · Matthew Wilcox
Quoted 24 lines · David Hildenbrand
Quoted 8 lines · Ryan Roberts
Quoted 6 lines · Ying
Although we can use smaller page order for
FLEXIBLE_THP, it's hard to avoid internal fragmentation completely. So, I think that finally we will need to provide a mechanism for the users to opt out, e.g., something like "always madvise never" via /sys/kernel/mm/transparent_hugepage/enabled. I'm not sure whether it's a good idea to reuse the existing interface of THP.Quoted 8 lines
I wouldn't want to tie this to the existing interface, simply because that implies that we would want to follow the "always" and "madvise" advice too; That
means that on a thp=madvise system (which is certainly the case for android and other client systems) we would have to disable large anon folios for VMAs that haven't explicitly opted in. That breaks the intention that this should be an invisible performance boost. I think it's important to set the policy for use ofIt will never ever be a completely invisible performance boost, just like ordinary THP.
Using the exact same existing toggle is the right thing to do. If someone specify "never" or "madvise", then do exactly that.
Quoted 7 lines
Quoted 2 lines
It might make sense to have more modes or additional toggles, but "madvise=never" means no memory waste.
I hate the existing mechanisms. They are an abdication of our responsibility, and an attempt to blame the user (be it the sysadmin or the programmer) of our code for using it wrongly. We should not replicate this mistake.
I don't agree regarding the programmer responsibility. In some cases the programmer really doesn't want to get more memory populated than requested -- and knows exactly why setting
MADV_NOHUGEPAGEis the right thing to do.Regarding the madvise=never/madvise/always (sys admin decision), memory waste (and nailing down bugs or working around them in customer setups) have been very good reasons to let the admin have a word.
Quoted 4 lines
Our code should be auto-tuning. I posted a long, detailed outline here:
https://lore.kernel.org/linux-mm/Y%[email protected]/Well, "auto-tuning" also should be perfect for everybody, but once reality strikes you know it isn't.
If people don't feel like using THP, let them have a word. The "madvise" config option is probably more controversial. But the "always vs. never" absolutely makes sense to me.
Quoted 7 lines
Quoted 5 lines
I remember I raised it already in the past, but you *absolutely* have to respect the
MADV_NOHUGEPAGEflag. There is user space out there (for example, userfaultfd) that doesn't want the kernel to populate any additional page tables. So if you have to respect that already, then also respectMADV_HUGEPAGE, simple.Possibly having uffd enabled on a VMA should disable using large folios,
There are cases where we enable uffd *after* already touching memory (postcopy live migration in QEMU being the famous example). That doesn't fly.
Quoted 3 lines
I can get behind that. But the notion that userspace knows what it's doing ... hahaha. Just ignore the madvise flags. Userspace doesn't know what it's doing.
Quoted 5 lines
If user space sets
MADV_NOHUGEPAGE, it exactly knows what it is doing ... in some cases. And these include cases I care about messing with sparse VM memory :)I have strong opinions against populating more than required when user space set
MADV_NOHUGEPAGE.I can see your point about honouring
MADV_NOHUGEPAGE, so think that it is reasonable to fallback to allocating an order-0 page in a VMA that has it set. The app has gone out of its way to explicitly set it, after all.
Quoted 8 lines
I think the correct behaviour for the global thp controls (cmdline and sysfs) are less obvious though. I could get on board with disabling large anon folios globally when thp="never". But for other situations, I would prefer to keep large anon folios enabled (treat "madvise" as "always"), with the argument that their order is much smaller than traditional THP and therefore the internal fragmentation is significantly reduced. I really don't want to end up with user space ever having to opt-in (with
MADV_HUGEPAGE) to see the benefits of large anon folios.
I was briefly playing with a nasty idea of an additional "madvise-pmd" option (that could be the new default), that would use PMD THP only in madvise'd regions, and ordinary everywhere else. But let's disregard that for now. I think there is a bigger issue (below).
Quoted 3 lines
I still feel that it would be better for the thp and large anon folio controls to be independent though - what's the argument for tying them together?
Thinking about desired 2 MiB flexible THP on aarch64 (64k kernel) vs, 2 MiB PMD THP on aarch64 (4k kernel), how are they any different? Just the way they are mapped ...
It's easy to say "64k vs. 2 MiB" is a difference and we want separate controls, but how is "2MiB vs. 2 MiB" different?
Having that said, I think we have to make up our mind how much control we want to give user space. Again, the "2MiB vs. 2 MiB" case nicely shows that it's not trivial: memory waste is a real issue on some systems where we limit THP to madvise().
Just throwing it out for discussing:
What about keeping the "all / madvise / never" semantics (and MADV_NOHUGEPAGE ...) but having an additional config knob that specifies in which cases we *still* allow flexible THP even though the system was configured for "madvise".
I can't come up with a good name for that, but something like "max_auto_size=64k" could be something reasonable to set. We could have an arch+hw specific default.
(we all hate config options, I know, but I think there are good reasons to have such bare-minimum ones)
Signature · 9 lines
Cheers, David / dhildenb _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Re: [PATCH v2 4/5] mm: FLEXIBLE_THP for improved performance
Quoted 99 lines · David Hildenbrand
Quoted 83 lines · Ryan Roberts
Quoted 71 lines · David Hildenbrand
Quoted 27 lines · Matthew Wilcox
Quoted 26 lines · David Hildenbrand
Quoted 8 lines · Ryan Roberts
Quoted 6 lines · Ying
Although we can use smaller page order for
FLEXIBLE_THP, it's hard to avoid internal fragmentation completely. So, I think that finally we will need to provide a mechanism for the users to opt out, e.g., something like "always madvise never" via /sys/kernel/mm/transparent_hugepage/enabled. I'm not sure whether it's a good idea to reuse the existing interface of THP.Quoted 10 lines
I wouldn't want to tie this to the existing interface, simply because that implies that we would want to follow the "always" and "madvise" advice too; That
means that on a thp=madvise system (which is certainly the case for android and
other client systems) we would have to disable large anon folios for VMAs that
haven't explicitly opted in. That breaks the intention that this should be an invisible performance boost. I think it's important to set the policy for use ofIt will never ever be a completely invisible performance boost, just like ordinary THP.
Using the exact same existing toggle is the right thing to do. If someone specify "never" or "madvise", then do exactly that.
Quoted 7 lines
Quoted 2 lines
It might make sense to have more modes or additional toggles, but "madvise=never" means no memory waste.
I hate the existing mechanisms. They are an abdication of our responsibility, and an attempt to blame the user (be it the sysadmin or the programmer) of our code for using it wrongly. We should not replicate this mistake.
I don't agree regarding the programmer responsibility. In some cases the programmer really doesn't want to get more memory populated than requested -- and knows exactly why setting
MADV_NOHUGEPAGEis the right thing to do.Regarding the madvise=never/madvise/always (sys admin decision), memory waste (and nailing down bugs or working around them in customer setups) have been very good reasons to let the admin have a word.
Quoted 4 lines
Our code should be auto-tuning. I posted a long, detailed outline here:
https://lore.kernel.org/linux-mm/Y%[email protected]/Well, "auto-tuning" also should be perfect for everybody, but once reality strikes you know it isn't.
If people don't feel like using THP, let them have a word. The "madvise" config option is probably more controversial. But the "always vs. never" absolutely makes sense to me.
Quoted 7 lines
Quoted 5 lines
I remember I raised it already in the past, but you *absolutely* have to respect the
MADV_NOHUGEPAGEflag. There is user space out there (for example, userfaultfd) that doesn't want the kernel to populate any additional page tables. So if you have to respect that already, then also respectMADV_HUGEPAGE, simple.Possibly having uffd enabled on a VMA should disable using large folios,
There are cases where we enable uffd *after* already touching memory (postcopy live migration in QEMU being the famous example). That doesn't fly.
Quoted 3 lines
I can get behind that. But the notion that userspace knows what it's doing ... hahaha. Just ignore the madvise flags. Userspace doesn't know what it's doing.
Quoted 6 lines
If user space sets
MADV_NOHUGEPAGE, it exactly knows what it is doing ... in some cases. And these include cases I care about messing with sparse VM memory :)I have strong opinions against populating more than required when user space set
MADV_NOHUGEPAGE.I can see your point about honouring
MADV_NOHUGEPAGE, so think that it is reasonable to fallback to allocating an order-0 page in a VMA that has it set. The app has gone out of its way to explicitly set it, after all.Quoted 8 lines
I think the correct behaviour for the global thp controls (cmdline and sysfs) are less obvious though. I could get on board with disabling large anon folios globally when thp="never". But for other situations, I would prefer to keep large anon folios enabled (treat "madvise" as "always"), with the argument that their order is much smaller than traditional THP and therefore the internal fragmentation is significantly reduced. I really don't want to end up with user space ever having to opt-in (with
MADV_HUGEPAGE) to see the benefits of large anon folios.I was briefly playing with a nasty idea of an additional "madvise-pmd" option (that could be the new default), that would use PMD THP only in madvise'd regions, and ordinary everywhere else. But let's disregard that for now. I think there is a bigger issue (below).
Quoted 1 line
Quoted 6 lines
Quoted 2 lines
I still feel that it would be better for the thp and large anon folio controls to be independent though - what's the argument for tying them together?
Thinking about desired 2 MiB flexible THP on aarch64 (64k kernel) vs, 2 MiB PMD THP on aarch64 (4k kernel), how are they any different? Just the way they are mapped ...
The last patch in the series shows my current approach to that:
int arch_wants_pte_order(struct vm_area_struct *vma)
{
if (hugepage_vma_check(vma, vma->vm_flags, false, true, true))
return CONFIG_ARM64_PTE_ORDER_THP; <<< always the contpte size
else
return CONFIG_ARM64_PTE_ORDER_NOTHP; <<< limited to 64K
}
But Yu has raised concerns that this type of policy needs to be in the core mm. So we could have the arch blindly return the preferred order from HW perspective (which would be contpte size for arm64). Then for !hugepage_vma_check(), mm could take the min of that value and some determined "acceptable" limit (which in my mind is 64K ;-).
Quoted 12 lines
It's easy to say "64k vs. 2 MiB" is a difference and we want separate controls, but how is "2MiB vs. 2 MiB" different?
Having that said, I think we have to make up our mind how much control we want to give user space. Again, the "2MiB vs. 2 MiB" case nicely shows that it's not
trivial: memory waste is a real issue on some systems where we limit THP tomadvise().Just throwing it out for discussing:
Quoted 7 lines
What about keeping the "all / madvise / never" semantics (and
MADV_NOHUGEPAGE...) but having an additional config knob that specifies in which cases we *still* allow flexible THP even though the system was configured for "madvise".I can't come up with a good name for that, but something like "max_auto_size=64k" could be something reasonable to set. We could have an arch+hw specific default.
Ahha, yes, that's essentially what I have above. I personally also like the idea of the limit being an absolute value rather than an order. Although I know Yu feels differently (see [1]).
[1]
https://lore.kernel.org/linux-mm/[email protected]/T/#m2aff6eebd7f14d0d0620b48497d26eacecf970e6
Quoted 4 lines
(we all hate config options, I know, but I think there are good reasons to have such bare-minimum ones)
Re: [PATCH v2 4/5] mm: FLEXIBLE_THP for improved performance
Quoted 17 lines
Quoted 6 lines
Quoted 2 lines
I still feel that it would be better for the thp and large anon folio controls to be independent though - what's the argument for tying them together?
Thinking about desired 2 MiB flexible THP on aarch64 (64k kernel) vs, 2 MiB PMD THP on aarch64 (4k kernel), how are they any different? Just the way they are mapped ...
The last patch in the series shows my current approach to that:
int
arch_wants_pte_order(struct vm_area_struct *vma)
{
if (hugepage_vma_check(vma, vma->vm_flags, false, true, true))
returnCONFIG_ARM64_PTE_ORDER_THP; <<< always the contpte size
else
returnCONFIG_ARM64_PTE_ORDER_NOTHP; <<< limited to 64K
}
Quoted 5 lines
But Yu has raised concerns that this type of policy needs to be in the core mm. So we could have the arch blindly return the preferred order from HW perspective (which would be contpte size for arm64). Then for !hugepage_vma_check(), mm could take the min of that value and some determined "acceptable" limit (which in my mind is 64K ;-).
Yeah, it's really tricky. Because why should arm64 with 64k base pages *not* return 2MiB (which is one possible cont-pte size IIRC) ?
I share the idea that 64k might *currently* on *some platforms* be a reasonable choice. But that's where the "fun" begins.
Quoted 17 lines
Quoted 17 lines
It's easy to say "64k vs. 2 MiB" is a difference and we want separate controls, but how is "2MiB vs. 2 MiB" different?
Having that said, I think we have to make up our mind how much control we want to give user space. Again, the "2MiB vs. 2 MiB" case nicely shows that it's not
trivial: memory waste is a real issue on some systems where we limit THP tomadvise().Just throwing it out for discussing:
What about keeping the "all / madvise / never" semantics (and
MADV_NOHUGEPAGE...) but having an additional config knob that specifies in which cases we *still* allow flexible THP even though the system was configured for "madvise".
Quoted 7 lines
Quoted 3 lines
I can't come up with a good name for that, but something like "max_auto_size=64k" could be something reasonable to set. We could have an arch+hw specific default.
Ahha, yes, that's essentially what I have above. I personally also like the idea of the limit being an absolute value rather than an order. Although I know Yu feels differently (see [1]).
Exposed to user space I think it should be a human-readable value. Inside the kernel, I don't particularly care.
(Having databases/VMs on arch64 with 64k in mind) I think it might be interesting to have something like the following:
thp=madvise
max_auto_size=64k/128k/256k
So in MADV_HUGEPAGE VMAs (such as under QEMU), we'd happily take any flexible THP, especially ones < PMD THP (512 MiB) as well. 2 MiB or 4 MiB THP? sure, give them to my VM. You're barely going to find 512 MiB THP either way in practice ....
But for the remainder of my system, just do something reasonable and don't go crazy on the memory waste.
I'll try reading all the previous discussions next week.
Signature · 3 lines
Cheers, David / dhildenb
Re: [PATCH v2 4/5] mm: FLEXIBLE_THP for improved performance
Quoted 50 lines · David Hildenbrand
Quoted 17 lines
Quoted 6 lines
Quoted 2 lines
I still feel that it would be better for the thp and large anon folio controls to be independent though - what's the argument for tying them together?
Thinking about desired 2 MiB flexible THP on aarch64 (64k kernel) vs, 2 MiB PMD THP on aarch64 (4k kernel), how are they any different? Just the way they are mapped ...
The last patch in the series shows my current approach to that:
int
arch_wants_pte_order(struct vm_area_struct *vma)
{
if (hugepage_vma_check(vma, vma->vm_flags, false, true, true)) returnCONFIG_ARM64_PTE_ORDER_THP; <<< always the contpte size else
returnCONFIG_ARM64_PTE_ORDER_NOTHP; <<< limited to 64K }Quoted 5 lines
But Yu has raised concerns that this type of policy needs to be in the core mm. So we could have the arch blindly return the preferred order from HW perspective (which would be contpte size for arm64). Then for !hugepage_vma_check(), mm could take the min of that value and some determined "acceptable" limit (which in my mind is 64K ;-).
Yeah, it's really tricky. Because why should arm64 with 64k base pages *not* return 2MiB (which is one possible cont-pte size IIRC) ?
I share the idea that 64k might *currently* on *some platforms* be a reasonable choice. But that's where the "fun" begins.
Quoted 21 lines
Quoted 20 lines
It's easy to say "64k vs. 2 MiB" is a difference and we want separate controls, but how is "2MiB vs. 2 MiB" different?
Having that said, I think we have to make up our mind how much control we want to give user space. Again, the "2MiB vs. 2 MiB" case nicely shows that it's not
trivial: memory waste is a real issue on some systems where we limit THP tomadvise().Just throwing it out for discussing:
What about keeping the "all / madvise / never" semantics (and
MADV_NOHUGEPAGE...) but having an additional config knob that specifies in which cases we *still* allow flexible THP even though the system was configured for "madvise".I can't come up with a good name for that, but something like "max_auto_size=64k" could be something reasonable to set. We could have an arch+hw specific default.
Quoted 6 lines
Quoted 3 lines
Ahha, yes, that's essentially what I have above. I personally also like the idea of the limit being an absolute value rather than an order. Although I know Yu feels differently (see [1]).
Exposed to user space I think it should be a human-readable value. Inside the kernel, I don't particularly care.
My point was less about human-readable vs not. It was about expressing a value that is relative to the base page size vs expressing a value that is independent of base page size. If the concern is about limiting internal fragmentation, I think its the absolute size that matters.
Quoted 8 lines
(Having databases/VMs on arch64 with 64k in mind) I think it might be interesting to have something like the following:
thp=madvise
max_auto_size=64k/128k/256k
Quoted 6 lines
So in
MADV_HUGEPAGEVMAs (such as under QEMU), we'd happily take any flexible THP, especially ones < PMD THP (512 MiB) as well. 2 MiB or 4 MiB THP? sure, give them to my VM. You're barely going to find 512 MiB THP either way in practice ....But for the remainder of my system, just do something reasonable and don't go crazy on the memory waste.
Yep, we're on the same page. I've got a v3 that's almost ready to go, based on Yu's prevuous round of review. I'm going to encorporate this mechanism into it then post hopefully later in the week. Now I just need to figure out a decent name for the max_auto_size control...
Quoted 4 lines
I'll try reading all the previous discussions next week.
Re: [PATCH v2 4/5] mm: FLEXIBLE_THP for improved performance
In reply to Ryan RobertsQuoted 76 lines · Ryan Roberts
Quoted 69 lines · David Hildenbrand
Quoted 25 lines · Matthew Wilcox
Quoted 24 lines · David Hildenbrand
Quoted 8 lines · Ryan Roberts
Quoted 6 lines · Ying
Although we can use smaller page order for
FLEXIBLE_THP, it's hard to avoid internal fragmentation completely. So, I think that finally we will need to provide a mechanism for the users to opt out, e.g., something like "always madvise never" via /sys/kernel/mm/transparent_hugepage/enabled. I'm not sure whether it's a good idea to reuse the existing interface of THP.Quoted 8 lines
I wouldn't want to tie this to the existing interface, simply because that implies that we would want to follow the "always" and "madvise" advice too; That
means that on a thp=madvise system (which is certainly the case for android and other client systems) we would have to disable large anon folios for VMAs that haven't explicitly opted in. That breaks the intention that this should be an invisible performance boost. I think it's important to set the policy for use ofIt will never ever be a completely invisible performance boost, just like ordinary THP.
Using the exact same existing toggle is the right thing to do. If someone specify "never" or "madvise", then do exactly that.
Quoted 7 lines
Quoted 2 lines
It might make sense to have more modes or additional toggles, but "madvise=never" means no memory waste.
I hate the existing mechanisms. They are an abdication of our responsibility, and an attempt to blame the user (be it the sysadmin or the programmer) of our code for using it wrongly. We should not replicate this mistake.
I don't agree regarding the programmer responsibility. In some cases the programmer really doesn't want to get more memory populated than requested -- and knows exactly why setting
MADV_NOHUGEPAGEis the right thing to do.Regarding the madvise=never/madvise/always (sys admin decision), memory waste (and nailing down bugs or working around them in customer setups) have been very good reasons to let the admin have a word.
Quoted 4 lines
Our code should be auto-tuning. I posted a long, detailed outline here:
https://lore.kernel.org/linux-mm/Y%[email protected]/Well, "auto-tuning" also should be perfect for everybody, but once reality strikes you know it isn't.
If people don't feel like using THP, let them have a word. The "madvise" config option is probably more controversial. But the "always vs. never" absolutely makes sense to me.
Quoted 7 lines
Quoted 5 lines
I remember I raised it already in the past, but you *absolutely* have to respect the
MADV_NOHUGEPAGEflag. There is user space out there (for example, userfaultfd) that doesn't want the kernel to populate any additional page tables. So if you have to respect that already, then also respectMADV_HUGEPAGE, simple.Possibly having uffd enabled on a VMA should disable using large folios,
There are cases where we enable uffd *after* already touching memory (postcopy live migration in QEMU being the famous example). That doesn't fly.
Quoted 3 lines
I can get behind that. But the notion that userspace knows what it's doing ... hahaha. Just ignore the madvise flags. Userspace doesn't know what it's doing.
Quoted 5 lines
If user space sets
MADV_NOHUGEPAGE, it exactly knows what it is doing ... in some cases. And these include cases I care about messing with sparse VM memory :)I have strong opinions against populating more than required when user space set
MADV_NOHUGEPAGE.
Quoted 8 lines
I can see your point about honouring
MADV_NOHUGEPAGE, so think that it is reasonable to fallback to allocating an order-0 page in a VMA that has it set. The app has gone out of its way to explicitly set it, after all.I think the correct behaviour for the global thp controls (cmdline and sysfs) are less obvious though. I could get on board with disabling large anon folios globally when thp="never". But for other situations, I would prefer to keep large anon folios enabled (treat "madvise" as "always"),
If we have some mechanism to auto-tune the large folios usage, for example, detect the internal fragmentation and split the large folio, then we can use thp="always" as default configuration. If my memory were correct, this is what Johannes and Alexander is working on.
Quoted 3 lines
with the argument that
their order is much smaller than traditional THP and therefore the internal fragmentation is significantly reduced.
Do you have any data for this?
Quoted 7 lines
I really don't want to end up with user
space ever having to opt-in (withMADV_HUGEPAGE) to see the benefits of large anon folios.I still feel that it would be better for the thp and large anon folio controls to be independent though - what's the argument for tying them together?
Best Regards,
Huang, Ying
Re: [PATCH v2 4/5] mm: FLEXIBLE_THP for improved performance
Quoted 87 lines · Ying
Quoted 76 lines · Ryan Roberts
Quoted 69 lines · David Hildenbrand
Quoted 25 lines · Matthew Wilcox
Quoted 24 lines · David Hildenbrand
Quoted 8 lines · Ryan Roberts
Quoted 6 lines · Ying
Although we can use smaller page order for
FLEXIBLE_THP, it's hard to avoid internal fragmentation completely. So, I think that finally we will need to provide a mechanism for the users to opt out, e.g., something like "always madvise never" via /sys/kernel/mm/transparent_hugepage/enabled. I'm not sure whether it's a good idea to reuse the existing interface of THP.Quoted 8 lines
I wouldn't want to tie this to the existing interface, simply because that implies that we would want to follow the "always" and "madvise" advice too; That
means that on a thp=madvise system (which is certainly the case for android and other client systems) we would have to disable large anon folios for VMAs that haven't explicitly opted in. That breaks the intention that this should be an invisible performance boost. I think it's important to set the policy for use ofIt will never ever be a completely invisible performance boost, just like ordinary THP.
Using the exact same existing toggle is the right thing to do. If someone specify "never" or "madvise", then do exactly that.
Quoted 7 lines
Quoted 2 lines
It might make sense to have more modes or additional toggles, but "madvise=never" means no memory waste.
I hate the existing mechanisms. They are an abdication of our responsibility, and an attempt to blame the user (be it the sysadmin or the programmer) of our code for using it wrongly. We should not replicate this mistake.
I don't agree regarding the programmer responsibility. In some cases the programmer really doesn't want to get more memory populated than requested -- and knows exactly why setting
MADV_NOHUGEPAGEis the right thing to do.Regarding the madvise=never/madvise/always (sys admin decision), memory waste (and nailing down bugs or working around them in customer setups) have been very good reasons to let the admin have a word.
Quoted 4 lines
Our code should be auto-tuning. I posted a long, detailed outline here:
https://lore.kernel.org/linux-mm/Y%[email protected]/Well, "auto-tuning" also should be perfect for everybody, but once reality strikes you know it isn't.
If people don't feel like using THP, let them have a word. The "madvise" config option is probably more controversial. But the "always vs. never" absolutely makes sense to me.
Quoted 7 lines
Quoted 5 lines
I remember I raised it already in the past, but you *absolutely* have to respect the
MADV_NOHUGEPAGEflag. There is user space out there (for example, userfaultfd) that doesn't want the kernel to populate any additional page tables. So if you have to respect that already, then also respectMADV_HUGEPAGE, simple.Possibly having uffd enabled on a VMA should disable using large folios,
There are cases where we enable uffd *after* already touching memory (postcopy live migration in QEMU being the famous example). That doesn't fly.
Quoted 3 lines
I can get behind that. But the notion that userspace knows what it's doing ... hahaha. Just ignore the madvise flags. Userspace doesn't know what it's doing.
Quoted 5 lines
If user space sets
MADV_NOHUGEPAGE, it exactly knows what it is doing ... in some cases. And these include cases I care about messing with sparse VM memory :)I have strong opinions against populating more than required when user space set
MADV_NOHUGEPAGE.Quoted 8 lines
I can see your point about honouring
MADV_NOHUGEPAGE, so think that it is reasonable to fallback to allocating an order-0 page in a VMA that has it set. The app has gone out of its way to explicitly set it, after all.I think the correct behaviour for the global thp controls (cmdline and sysfs) are less obvious though. I could get on board with disabling large anon folios globally when thp="never". But for other situations, I would prefer to keep large anon folios enabled (treat "madvise" as "always"),
Quoted 4 lines
If we have some mechanism to auto-tune the large folios usage, for example, detect the internal fragmentation and split the large folio, then we can use thp="always" as default configuration. If my memory were correct, this is what Johannes and Alexander is working on.
Could you point me to that work? I'd like to understand what the mechanism is. The other half of my work aims to use arm64's pte "contiguous bit" to tell the HW that a span of PTEs share the same mapping and is therefore coalesced into a single TLB entry. The side effect of this, however, is that we only have a single access and dirty bit for the whole contpte extent. So I'd like to avoid any mechanism that relies on getting access/dirty at the base page granularity for a large folio.
Quoted 6 lines
Quoted 4 lines
with the argument that
their order is much smaller than traditional THP and therefore the internal fragmentation is significantly reduced.Do you have any data for this?
Some; its partly based on intuition that the smaller the allocation unit, the smaller the internal fragmentation. And partly on peak memory usage data I've collected for the benchmarks I'm running, comparing baseline-4k kernel with baseline-16k and baseline-64 kernels along with a 4k kernel that supports large anon folios (I appreciate that's not exactly what we are talking about here, and it's not exactly an extensive set of results!):
Kernel Compliation with 8 Jobs:
| kernel | peak | |:--------------|-------:| | baseline-4k | 0.0% | | anonfolio | 0.1% | | baseline-16k | 6.3% | | baseline-64k | 28.1% |
Kernel Compliation with 80 Jobs:
| kernel | peak | |:--------------|-------:| | baseline-4k | 0.0% | | anonfolio | 1.7% | | baseline-16k | 2.6% | | baseline-64k | 12.3% |
Quoted 12 lines
Quoted 8 lines
I really don't want to end up with user
space ever having to opt-in (withMADV_HUGEPAGE) to see the benefits of large anon folios.I still feel that it would be better for the thp and large anon folio controls to be independent though - what's the argument for tying them together?
Best Regards,
Huang, Ying
Re: [PATCH v2 4/5] mm: FLEXIBLE_THP for improved performance
Quoted 93 lines · Ryan Roberts
Quoted 87 lines · Ying
Quoted 76 lines · Ryan Roberts
Quoted 69 lines · David Hildenbrand
Quoted 25 lines · Matthew Wilcox
Quoted 24 lines · David Hildenbrand
Quoted 8 lines · Ryan Roberts
Quoted 6 lines · Ying
Although we can use smaller page order for
FLEXIBLE_THP, it's hard to avoid internal fragmentation completely. So, I think that finally we will need to provide a mechanism for the users to opt out, e.g., something like "always madvise never" via /sys/kernel/mm/transparent_hugepage/enabled. I'm not sure whether it's a good idea to reuse the existing interface of THP.Quoted 8 lines
I wouldn't want to tie this to the existing interface, simply because that implies that we would want to follow the "always" and "madvise" advice too; That
means that on a thp=madvise system (which is certainly the case for android and other client systems) we would have to disable large anon folios for VMAs that haven't explicitly opted in. That breaks the intention that this should be an invisible performance boost. I think it's important to set the policy for use ofIt will never ever be a completely invisible performance boost, just like ordinary THP.
Using the exact same existing toggle is the right thing to do. If someone specify "never" or "madvise", then do exactly that.
Quoted 7 lines
Quoted 2 lines
It might make sense to have more modes or additional toggles, but "madvise=never" means no memory waste.
I hate the existing mechanisms. They are an abdication of our responsibility, and an attempt to blame the user (be it the sysadmin or the programmer) of our code for using it wrongly. We should not replicate this mistake.
I don't agree regarding the programmer responsibility. In some cases the programmer really doesn't want to get more memory populated than requested -- and knows exactly why setting
MADV_NOHUGEPAGEis the right thing to do.Regarding the madvise=never/madvise/always (sys admin decision), memory waste (and nailing down bugs or working around them in customer setups) have been very good reasons to let the admin have a word.
Quoted 4 lines
Our code should be auto-tuning. I posted a long, detailed outline here:
https://lore.kernel.org/linux-mm/Y%[email protected]/Well, "auto-tuning" also should be perfect for everybody, but once reality strikes you know it isn't.
If people don't feel like using THP, let them have a word. The "madvise" config option is probably more controversial. But the "always vs. never" absolutely makes sense to me.
Quoted 7 lines
Quoted 5 lines
I remember I raised it already in the past, but you *absolutely* have to respect the
MADV_NOHUGEPAGEflag. There is user space out there (for example, userfaultfd) that doesn't want the kernel to populate any additional page tables. So if you have to respect that already, then also respectMADV_HUGEPAGE, simple.Possibly having uffd enabled on a VMA should disable using large folios,
There are cases where we enable uffd *after* already touching memory (postcopy live migration in QEMU being the famous example). That doesn't fly.
Quoted 3 lines
I can get behind that. But the notion that userspace knows what it's doing ... hahaha. Just ignore the madvise flags. Userspace doesn't know what it's doing.
Quoted 5 lines
If user space sets
MADV_NOHUGEPAGE, it exactly knows what it is doing ... in some cases. And these include cases I care about messing with sparse VM memory :)I have strong opinions against populating more than required when user space set
MADV_NOHUGEPAGE.Quoted 8 lines
I can see your point about honouring
MADV_NOHUGEPAGE, so think that it is reasonable to fallback to allocating an order-0 page in a VMA that has it set. The app has gone out of its way to explicitly set it, after all.I think the correct behaviour for the global thp controls (cmdline and sysfs) are less obvious though. I could get on board with disabling large anon folios globally when thp="never". But for other situations, I would prefer to keep large anon folios enabled (treat "madvise" as "always"),
Quoted 4 lines
If we have some mechanism to auto-tune the large folios usage, for example, detect the internal fragmentation and split the large folio, then we can use thp="always" as default configuration. If my memory were correct, this is what Johannes and Alexander is working on.
Quoted 7 lines
Could you point me to that work? I'd like to understand what the mechanism is. The other half of my work aims to use arm64's pte "contiguous bit" to tell the HW that a span of PTEs share the same mapping and is therefore coalesced into a single TLB entry. The side effect of this, however, is that we only have a single access and dirty bit for the whole contpte extent. So I'd like to avoid any mechanism that relies on getting access/dirty at the base page granularity for a large folio.
Please take a look at the THP shrinker patchset,
https://lore.kernel.org/linux-mm/[email protected]/
Quoted 24 lines
Quoted 6 lines
Quoted 4 lines
with the argument that
their order is much smaller than traditional THP and therefore the internal fragmentation is significantly reduced.Do you have any data for this?
Some; its partly based on intuition that the smaller the allocation unit, the smaller the internal fragmentation. And partly on peak memory usage data I've collected for the benchmarks I'm running, comparing baseline-4k kernel with baseline-16k and baseline-64 kernels along with a 4k kernel that supports large anon folios (I appreciate that's not exactly what we are talking about here, and it's not exactly an extensive set of results!):
Kernel Compliation with 8 Jobs:
| kernel | peak | |:--------------|-------:| | baseline-4k | 0.0% | | anonfolio | 0.1% | | baseline-16k | 6.3% | | baseline-64k | 28.1% |
Quoted 8 lines
Kernel Compliation with 80 Jobs:
| kernel | peak | |:--------------|-------:| | baseline-4k | 0.0% | | anonfolio | 1.7% | | baseline-16k | 2.6% | | baseline-64k | 12.3% |
Why is anonfolio better than baseline-64k if you always allocate 64k anonymous folio? Because page cache uses 64k in baseline-64k?
We may need to test some workloads with sparse access patterns too.
Best Regards,
Huang, Ying
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
I really don't want to end up with user
space ever having to opt-in (withMADV_HUGEPAGE) to see the benefits of large anon folios.I still feel that it would be better for the thp and large anon folio controls to be independent though - what's the argument for tying them together?
Re: [PATCH v2 4/5] mm: FLEXIBLE_THP for improved performance
Quoted 103 lines · Ying
Quoted 93 lines · Ryan Roberts
Quoted 87 lines · Ying
Quoted 76 lines · Ryan Roberts
Quoted 69 lines · David Hildenbrand
Quoted 25 lines · Matthew Wilcox
Quoted 24 lines · David Hildenbrand
Quoted 8 lines · Ryan Roberts
Quoted 6 lines · Ying
Although we can use smaller page order for
FLEXIBLE_THP, it's hard to avoid internal fragmentation completely. So, I think that finally we will need to provide a mechanism for the users to opt out, e.g., something like "always madvise never" via /sys/kernel/mm/transparent_hugepage/enabled. I'm not sure whether it's a good idea to reuse the existing interface of THP.Quoted 8 lines
I wouldn't want to tie this to the existing interface, simply because that implies that we would want to follow the "always" and "madvise" advice too; That
means that on a thp=madvise system (which is certainly the case for android and other client systems) we would have to disable large anon folios for VMAs that haven't explicitly opted in. That breaks the intention that this should be an invisible performance boost. I think it's important to set the policy for use ofIt will never ever be a completely invisible performance boost, just like ordinary THP.
Using the exact same existing toggle is the right thing to do. If someone specify "never" or "madvise", then do exactly that.
Quoted 7 lines
Quoted 2 lines
It might make sense to have more modes or additional toggles, but "madvise=never" means no memory waste.
I hate the existing mechanisms. They are an abdication of our responsibility, and an attempt to blame the user (be it the sysadmin or the programmer) of our code for using it wrongly. We should not replicate this mistake.
I don't agree regarding the programmer responsibility. In some cases the programmer really doesn't want to get more memory populated than requested -- and knows exactly why setting
MADV_NOHUGEPAGEis the right thing to do.Regarding the madvise=never/madvise/always (sys admin decision), memory waste (and nailing down bugs or working around them in customer setups) have been very good reasons to let the admin have a word.
Quoted 4 lines
Our code should be auto-tuning. I posted a long, detailed outline here:
https://lore.kernel.org/linux-mm/Y%[email protected]/Well, "auto-tuning" also should be perfect for everybody, but once reality strikes you know it isn't.
If people don't feel like using THP, let them have a word. The "madvise" config option is probably more controversial. But the "always vs. never" absolutely makes sense to me.
Quoted 7 lines
Quoted 5 lines
I remember I raised it already in the past, but you *absolutely* have to respect the
MADV_NOHUGEPAGEflag. There is user space out there (for example, userfaultfd) that doesn't want the kernel to populate any additional page tables. So if you have to respect that already, then also respectMADV_HUGEPAGE, simple.Possibly having uffd enabled on a VMA should disable using large folios,
There are cases where we enable uffd *after* already touching memory (postcopy live migration in QEMU being the famous example). That doesn't fly.
Quoted 3 lines
I can get behind that. But the notion that userspace knows what it's doing ... hahaha. Just ignore the madvise flags. Userspace doesn't know what it's doing.
Quoted 5 lines
If user space sets
MADV_NOHUGEPAGE, it exactly knows what it is doing ... in some cases. And these include cases I care about messing with sparse VM memory :)I have strong opinions against populating more than required when user space set
MADV_NOHUGEPAGE.Quoted 8 lines
I can see your point about honouring
MADV_NOHUGEPAGE, so think that it is reasonable to fallback to allocating an order-0 page in a VMA that has it set. The app has gone out of its way to explicitly set it, after all.I think the correct behaviour for the global thp controls (cmdline and sysfs) are less obvious though. I could get on board with disabling large anon folios globally when thp="never". But for other situations, I would prefer to keep large anon folios enabled (treat "madvise" as "always"),
Quoted 4 lines
If we have some mechanism to auto-tune the large folios usage, for example, detect the internal fragmentation and split the large folio, then we can use thp="always" as default configuration. If my memory were correct, this is what Johannes and Alexander is working on.
Quoted 7 lines
Could you point me to that work? I'd like to understand what the mechanism is. The other half of my work aims to use arm64's pte "contiguous bit" to tell the HW that a span of PTEs share the same mapping and is therefore coalesced into a single TLB entry. The side effect of this, however, is that we only have a single access and dirty bit for the whole contpte extent. So I'd like to avoid any mechanism that relies on getting access/dirty at the base page granularity for a large folio.
Quoted 3 lines
Please take a look at the THP shrinker patchset,
Thanks!
Quoted 34 lines
Quoted 25 lines
Quoted 7 lines
Quoted 5 lines
with the argument that
their order is much smaller than traditional THP and therefore the internal fragmentation is significantly reduced.Do you have any data for this?
Some; its partly based on intuition that the smaller the allocation unit, the smaller the internal fragmentation. And partly on peak memory usage data I've collected for the benchmarks I'm running, comparing baseline-4k kernel with baseline-16k and baseline-64 kernels along with a 4k kernel that supports large anon folios (I appreciate that's not exactly what we are talking about here, and it's not exactly an extensive set of results!):
Kernel Compliation with 8 Jobs:
| kernel | peak | |:--------------|-------:| | baseline-4k | 0.0% | | anonfolio | 0.1% | | baseline-16k | 6.3% | | baseline-64k | 28.1% |Quoted 8 lines
Kernel Compliation with 80 Jobs:
| kernel | peak | |:--------------|-------:| | baseline-4k | 0.0% | | anonfolio | 1.7% | | baseline-16k | 2.6% | | baseline-64k | 12.3% |
Quoted 2 lines
Why is anonfolio better than baseline-64k if you always allocate 64k anonymous folio? Because page cache uses 64k in baseline-64k?
No, because the VMA boundaries are aligned to 4K and not 64K. Large Anon Folios only allocates a 64K folio if it does not breach the bounds of the VMA (and if it doesn't overlap other allocated PTEs).
Quoted 2 lines
We may need to test some workloads with sparse access patterns too.
Yes, I agree if you have a workload with a pathalogical memory access pattern where it writes to addresses with a stride of 64K, all contained in a single VMA, then you will end up allocating 16x the memory. This is obviously an unrealistic extreme though.
Quoted 13 lines
Best Regards,
Huang, YingQuoted 8 lines
Quoted 8 lines
Quoted 8 lines
I really don't want to end up with user
space ever having to opt-in (withMADV_HUGEPAGE) to see the benefits of large anon folios.I still feel that it would be better for the thp and large anon folio controls to be independent though - what's the argument for tying them together?
Re: [PATCH v2 4/5] mm: FLEXIBLE_THP for improved performance
Quoted 144 lines · Ryan Roberts
Quoted 103 lines · Ying
Quoted 93 lines · Ryan Roberts
Quoted 87 lines · Ying
Quoted 76 lines · Ryan Roberts
Quoted 69 lines · David Hildenbrand
Quoted 25 lines · Matthew Wilcox
Quoted 24 lines · David Hildenbrand
Quoted 8 lines · Ryan Roberts
Quoted 6 lines · Ying
Although we can use smaller page order for
FLEXIBLE_THP, it's hard to avoid internal fragmentation completely. So, I think that finally we will need to provide a mechanism for the users to opt out, e.g., something like "always madvise never" via /sys/kernel/mm/transparent_hugepage/enabled. I'm not sure whether it's a good idea to reuse the existing interface of THP.Quoted 8 lines
I wouldn't want to tie this to the existing interface, simply because that implies that we would want to follow the "always" and "madvise" advice too; That
means that on a thp=madvise system (which is certainly the case for android and other client systems) we would have to disable large anon folios for VMAs that haven't explicitly opted in. That breaks the intention that this should be an invisible performance boost. I think it's important to set the policy for use ofIt will never ever be a completely invisible performance boost, just like ordinary THP.
Using the exact same existing toggle is the right thing to do. If someone specify "never" or "madvise", then do exactly that.
Quoted 7 lines
Quoted 2 lines
It might make sense to have more modes or additional toggles, but "madvise=never" means no memory waste.
I hate the existing mechanisms. They are an abdication of our responsibility, and an attempt to blame the user (be it the sysadmin or the programmer) of our code for using it wrongly. We should not replicate this mistake.
I don't agree regarding the programmer responsibility. In some cases the programmer really doesn't want to get more memory populated than requested -- and knows exactly why setting
MADV_NOHUGEPAGEis the right thing to do.Regarding the madvise=never/madvise/always (sys admin decision), memory waste (and nailing down bugs or working around them in customer setups) have been very good reasons to let the admin have a word.
Quoted 4 lines
Our code should be auto-tuning. I posted a long, detailed outline here:
https://lore.kernel.org/linux-mm/Y%[email protected]/Well, "auto-tuning" also should be perfect for everybody, but once reality strikes you know it isn't.
If people don't feel like using THP, let them have a word. The "madvise" config option is probably more controversial. But the "always vs. never" absolutely makes sense to me.
Quoted 7 lines
Quoted 5 lines
I remember I raised it already in the past, but you *absolutely* have to respect the
MADV_NOHUGEPAGEflag. There is user space out there (for example, userfaultfd) that doesn't want the kernel to populate any additional page tables. So if you have to respect that already, then also respectMADV_HUGEPAGE, simple.Possibly having uffd enabled on a VMA should disable using large folios,
There are cases where we enable uffd *after* already touching memory (postcopy live migration in QEMU being the famous example). That doesn't fly.
Quoted 3 lines
I can get behind that. But the notion that userspace knows what it's doing ... hahaha. Just ignore the madvise flags. Userspace doesn't know what it's doing.
Quoted 5 lines
If user space sets
MADV_NOHUGEPAGE, it exactly knows what it is doing ... in some cases. And these include cases I care about messing with sparse VM memory :)I have strong opinions against populating more than required when user space set
MADV_NOHUGEPAGE.Quoted 8 lines
I can see your point about honouring
MADV_NOHUGEPAGE, so think that it is reasonable to fallback to allocating an order-0 page in a VMA that has it set. The app has gone out of its way to explicitly set it, after all.I think the correct behaviour for the global thp controls (cmdline and sysfs) are less obvious though. I could get on board with disabling large anon folios globally when thp="never". But for other situations, I would prefer to keep large anon folios enabled (treat "madvise" as "always"),
Quoted 4 lines
If we have some mechanism to auto-tune the large folios usage, for example, detect the internal fragmentation and split the large folio, then we can use thp="always" as default configuration. If my memory were correct, this is what Johannes and Alexander is working on.
Quoted 7 lines
Could you point me to that work? I'd like to understand what the mechanism is. The other half of my work aims to use arm64's pte "contiguous bit" to tell the HW that a span of PTEs share the same mapping and is therefore coalesced into a single TLB entry. The side effect of this, however, is that we only have a single access and dirty bit for the whole contpte extent. So I'd like to avoid any mechanism that relies on getting access/dirty at the base page granularity for a large folio.
Quoted 3 lines
Please take a look at the THP shrinker patchset,
Thanks!
Quoted 34 lines
Quoted 33 lines
Quoted 7 lines
Quoted 5 lines
with the argument that
their order is much smaller than traditional THP and therefore the internal fragmentation is significantly reduced.Do you have any data for this?
Some; its partly based on intuition that the smaller the allocation unit, the smaller the internal fragmentation. And partly on peak memory usage data I've collected for the benchmarks I'm running, comparing baseline-4k kernel with baseline-16k and baseline-64 kernels along with a 4k kernel that supports large anon folios (I appreciate that's not exactly what we are talking about here, and it's not exactly an extensive set of results!):
Kernel Compliation with 8 Jobs:
| kernel | peak | |:--------------|-------:| | baseline-4k | 0.0% | | anonfolio | 0.1% | | baseline-16k | 6.3% | | baseline-64k | 28.1% |Kernel Compliation with 80 Jobs:
| kernel | peak | |:--------------|-------:| | baseline-4k | 0.0% | | anonfolio | 1.7% | | baseline-16k | 2.6% | | baseline-64k | 12.3% |
Quoted 6 lines
Quoted 2 lines
Why is anonfolio better than baseline-64k if you always allocate 64k anonymous folio? Because page cache uses 64k in baseline-64k?
No, because the VMA boundaries are aligned to 4K and not 64K. Large Anon Folios only allocates a 64K folio if it does not breach the bounds of the VMA (and if it doesn't overlap other allocated PTEs).
Thanks for explanation!
We will use more memory for file cache too for baseline-64k, right? So, you observed much more anonymous pages, but not so for file cache pages?
Quoted 7 lines
Quoted 2 lines
We may need to test some workloads with sparse access patterns too.
Yes, I agree if you have a workload with a pathalogical memory access pattern where it writes to addresses with a stride of 64K, all contained in a single VMA, then you will end up allocating 16x the memory. This is obviously an unrealistic extreme though.
I think that there should be some realistic workload which has sparse access patterns.
Best Regards,
Huang, Ying
Quoted 10 lines
Quoted 10 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
I really don't want to end up with user
space ever having to opt-in (withMADV_HUGEPAGE) to see the benefits of large anon folios.I still feel that it would be better for the thp and large anon folio controls to be independent though - what's the argument for tying them together?
Re: [PATCH v2 4/5] mm: FLEXIBLE_THP for improved performance
In reply to Matthew WilcoxQuoted 26 lines · Matthew Wilcox
Quoted 16 lines · David Hildenbrand
Quoted 8 lines · Ryan Roberts
Quoted 6 lines · Ying
Although we can use smaller page order for
FLEXIBLE_THP, it's hard to avoid internal fragmentation completely. So, I think that finally we will need to provide a mechanism for the users to opt out, e.g., something like "always madvise never" via/sys/kernel/mm/transparent_hugepage/enabled. I'm not sure whether it's a good idea to reuse the existing interface of THP.Quoted 6 lines
I wouldn't want to tie this to the existing interface, simply because that implies that we would want to follow the "always" and "madvise" advice too; That means that on a thp=madvise system (which is certainly the case for android and other client systems) we would have to disable large anon folios for VMAs that haven't explicitly opted in. That breaks the intention that this should be an invisible performance boost. I think it's important to set the policy for use of
Quoted 8 lines
It will never ever be a completely invisible performance boost, just like ordinary THP.
Using the exact same existing toggle is the right thing to do. If someone specify "never" or "madvise", then do exactly that.
It might make sense to have more modes or additional toggles, but "madvise=never" means no memory waste.
Quoted 7 lines
I hate the existing mechanisms. They are an abdication of our responsibility, and an attempt to blame the user (be it the sysadmin or the programmer) of our code for using it wrongly. We should not replicate this mistake.
Our code should be auto-tuning. I posted a long, detailed outline here:
https://lore.kernel.org/linux-mm/Y%[email protected]/
Yes. Auto-tuning should be more preferable than any configuration mechanisms.
Something like THP shrinker could be another way of auto-tuning.
https://lore.kernel.org/linux-mm/[email protected]/
That is, allocating the large folios on page fault, then try to detect internal fragmentation.
Quoted 10 lines
Quoted 5 lines
I remember I raised it already in the past, but you *absolutely* have to respect the
MADV_NOHUGEPAGEflag. There is user space out there (for example, userfaultfd) that doesn't want the kernel to populate any additional page tables. So if you have to respect that already, then also respectMADV_HUGEPAGE, simple.Possibly having uffd enabled on a VMA should disable using large folios, I can get behind that. But the notion that userspace knows what it's doing ... hahaha. Just ignore the madvise flags. Userspace doesn't know what it's doing.
Best Regards,
Huang, Ying
PATCH v2 5/5 arm64: mm: Override arch_wants_pte_order()
In reply to Ryan RobertsDefine an arch-specific override of arch_wants_pte_order() so that when FLEXIBLE_THP is enabled, large folios will be allocated for anonymous memory with an order that is compatible with arm64's contpte mappings.
arch_wants_pte_order() returns an order according to the following
policy: For the unhinted case, when THP is not requested for the vma, don't allow anything bigger than 64K. This means we don't waste too much memory. Additionally, for 4K pages this is the contpte size, and for 16K, this is (usually) the HPA size when the uarch feature is implemented. For the hinted case, when THP is requested for the vma, allow the contpte size for all page size configurations; 64K for 4K, 2M for 16K and 2M for 64K.
Additionally, the THP and NOTHP order constants are defined using Kconfig so it is possible to override them at build time.
--- arch/arm64/Kconfig | 12 ++++++++++++ arch/arm64/include/asm/pgtable.h | 4 ++++ arch/arm64/mm/mmu.c | 8 ++++++++ 3 files changed, 24 insertions(+)
arch/arm64/Kconfig +12 -0
@@ -281,6 +281,18 @@ config ARM64_CONT_PMD_SHIFT default 5 if ARM64_16K_PAGES default 4 +config ARM64_PTE_ORDER_NOTHP + int + default 0 if ARM64_64K_PAGES # 64K (1 page) + default 2 if ARM64_16K_PAGES # 64K (4 pages; benefits from HPA where HW supports it) + default 4 if ARM64_4K_PAGES # 64K (16 pages; eligible for contpte-mapping) + +config ARM64_PTE_ORDER_THP + int + default 5 if ARM64_64K_PAGES # 2M (32 pages; eligible for contpte-mapping) + default 7 if ARM64_16K_PAGES # 2M (128 pages; eligible for contpte-mapping) + default 4 if ARM64_4K_PAGES # 64K (16 pages; eligible for contpte-mapping) + config ARCH_MMAP_RND_BITS_MIN default 14 if ARM64_64K_PAGES default 16 if ARM64_16K_PAGES
arch/arm64/include/asm/pgtable.h +4 -0
@@ -1117,6 +1117,10 @@ extern pte_t ptep_modify_prot_start(struct vm_area_struct *vma,extern void ptep_modify_prot_commit (struct vm_area_struct *vma,unsigned long addr,pte_t *ptep,pte_t old_pte,pte_t new_pte); + +#define arch_wants_pte_order arch_wants_pte_order +extern int arch_wants_pte_order (struct vm_area_struct *vma); +#endif /* !__ASSEMBLY__ */ #endif /* __ASM_PGTABLE_H */
arch/arm64/mm/mmu.c +8 -0
@@ -1481,3 +1481,11 @@ void ptep_modify_prot_commit(struct vm_area_struct *vma, unsigned long addr, pte {set_pte_at (vma->vm_mm, addr, ptep, pte); } + +int arch_wants_pte_order (struct vm_area_struct *vma) +{ +if (hugepage_vma_check (vma, vma->vm_flags, false, true, true)) +return CONFIG_ARM64_PTE_ORDER_THP ; +else +return CONFIG_ARM64_PTE_ORDER_NOTHP ; +}
Signature · 1 line
2.25.1
Re: [PATCH v2 5/5] arm64: mm: Override arch_wants_pte_order()
Quoted 68 lines · Ryan Roberts
Define an arch-specific override of
arch_wants_pte_order()so that whenFLEXIBLE_THPis enabled, large folios will be allocated for anonymous memory with an order that is compatible with arm64's contpte mappings.
arch_wants_pte_order()returns an order according to the following
policy: For the unhinted case, when THP is not requested for the vma, don't allow anything bigger than 64K. This means we don't waste too much memory. Additionally, for 4K pages this is the contpte size, and for 16K, this is (usually) the HPA size when the uarch feature is implemented. For the hinted case, when THP is requested for the vma, allow the contpte size for all page size configurations; 64K for 4K, 2M for 16K and 2M for 64K.Additionally, the THP and NOTHP order constants are defined using
Kconfigso it is possible to override them at build time.Signed-off-by Ryan Roberts <[email protected]>--- arch/arm64/Kconfig | 12 ++++++++++++ arch/arm64/include/asm/pgtable.h | 4 ++++ arch/arm64/mm/mmu.c | 8 ++++++++ 3 files changed, 24 insertions(+)arch/arm64/Kconfig +12
-0@@ -281,6 +281,18 @@ config ARM64_CONT_PMD_SHIFT default 5 if ARM64_16K_PAGES default 4 +config ARM64_PTE_ORDER_NOTHP + int + default 0 if ARM64_64K_PAGES # 64K (1 page) + default 2 if ARM64_16K_PAGES # 64K (4 pages; benefits from HPA where HW supports it) + default 4 if ARM64_4K_PAGES # 64K (16 pages; eligible for contpte-mapping) + +config ARM64_PTE_ORDER_THP + int + default 5 if ARM64_64K_PAGES # 2M (32 pages; eligible for contpte-mapping) + default 7 if ARM64_16K_PAGES # 2M (128 pages; eligible for contpte-mapping) + default 4 if ARM64_4K_PAGES # 64K (16 pages; eligible for contpte-mapping) + config ARCH_MMAP_RND_BITS_MIN default 14 if ARM64_64K_PAGES default 16 if ARM64_16K_PAGESarch/arm64/include/asm/pgtable.h +4
-0@@ -1117,6 +1117,10 @@ extern pte_t ptep_modify_prot_start(struct vm_area_struct *vma,extern void ptep_modify_prot_commit (struct vm_area_struct *vma,unsigned long addr,pte_t *ptep,pte_t old_pte,pte_t new_pte); + +#define arch_wants_pte_order arch_wants_pte_order +extern int arch_wants_pte_order (struct vm_area_struct *vma); +#endif /* !__ASSEMBLY__ */ #endif /* __ASM_PGTABLE_H */ arch/arm64/mm/mmu.c +0
-0@@ -1481,3 +1481,11 @@ void ptep_modify_prot_commit(struct vm_area_struct *vma, unsigned long addr, pte {
arch/arm64/mm/mmu.c void ptep_modify_prot_commit(struct vm_area_struct *vma, unsigned long addr, pte · line 1491
Quoted 10 lines
arch/arm64/mm/mmu.c +8
-0set_pte_at (vma->vm_mm, addr, ptep, pte); } + +int arch_wants_pte_order (struct vm_area_struct *vma) +{ +if (hugepage_vma_check (vma, vma->vm_flags, false, true, true)) +return CONFIG_ARM64_PTE_ORDER_THP ; +else +return CONFIG_ARM64_PTE_ORDER_NOTHP ; +}
I don't really like this because it's a mix of h/w preference and s/w policy -- from my POV, it's supposed to be the former only. The policy part should be left to core MM (arch-independent).
That being said, no objection if ARM MM people think this is really what they want.
Quoted 6 lines · Ryan Roberts
Hi All,
This is v2 of a series to implement variable order, large folios for anonymous memory. The objective of this is to improve performance by allocating larger chunks of memory during anonymous page faults. See [1] for background.
Thanks for the quick response!
Quoted 10 lines
I've significantly reworked and simplified the patch set based on comments from Yu Zhao (thanks for all your feedback!). I've also renamed the feature to
VARIABLE_THP, on Yu's advice.The last patch is for arm64 to explicitly override the default
arch_wants_pte_order()and is intended as an example. If this series is accepted I suggest taking the first 4 patches through the mm tree and the arm64 change could be handled through the arm64 tree separately. Neither has any build dependency on the other.
Quoted 6 lines
The one area where I haven't followed Yu's advice is in the determination of the size of folio to use. It was suggested that I have a single preferred large order, and if it doesn't fit in the VMA (due to exceeding VMA bounds, or there being existing overlapping populated PTEs, etc) then fallback immediately to order-0. It turned out that this approach caused a performance regression in the Speedometer benchmark.
I suppose it's regression against the v1, not the unpatched kernel.
Quoted 7 lines
With my v1 patch, there were significant quantities of
memory which could not be placed in the 64K bucket and were instead being allocated for the 32K and 16K buckets. With the proposed simplification, that memory ended up using the 4K bucket, so page faults increased by 2.75x compared to the v1 patch (although due to the 64K bucket, this number is still a bit lower than the baseline). So instead, I continue to calculate a folio order that is somewhere between the preferred order and 0. (See below for more details).
I suppose the benchmark wasn't running under memory pressure, which is uncommon for client devices. It could be easier the other way around:
using 32/16KB shows regression whereas order-0 shows better performance under memory pressure.
I'm not sure we should use v1 as the baseline. Unpatched kernel sounds more reasonable at this point. If 32/16KB is proven to be better in most scenarios including under memory pressure, we can reintroduce that policy. I highly doubt this is the case: we tried 16KB base page size on client devices, and overall, the regressions outweighs the benefits.
Quoted 2 lines
The patches are based on top of v6.4 plus Matthew Wilcox's
set_ptes()series [2], which is a hard dependency. I have a branch at [3].
It's not clear to me why [2] is a hard dependency.
It seems to me we are getting close and I was hoping we could get into mm-unstable soon without depending on other series...
Quoted 20 lines · Yu Zhao
Quoted 6 lines · Ryan Roberts
Hi All,
This is v2 of a series to implement variable order, large folios for anonymous memory. The objective of this is to improve performance by allocating larger chunks of memory during anonymous page faults. See [1] for background.
Thanks for the quick response!
Quoted 10 lines
I've significantly reworked and simplified the patch set based on comments from Yu Zhao (thanks for all your feedback!). I've also renamed the feature to
VARIABLE_THP, on Yu's advice.The last patch is for arm64 to explicitly override the default
arch_wants_pte_order()and is intended as an example. If this series is accepted I suggest taking the first 4 patches through the mm tree and the arm64 change could be handled through the arm64 tree separately. Neither has any build dependency on the other.
Quoted 8 lines
Quoted 6 lines
The one area where I haven't followed Yu's advice is in the determination of the size of folio to use. It was suggested that I have a single preferred large order, and if it doesn't fit in the VMA (due to exceeding VMA bounds, or there being existing overlapping populated PTEs, etc) then fallback immediately to order-0. It turned out that this approach caused a performance regression in the Speedometer benchmark.
I suppose it's regression against the v1, not the unpatched kernel.
From the performance data Ryan shared, it's against unpatched kernel:
Speedometer 2.0:
| kernel | runs_per_min | |:-------------------------------|---------------:| | baseline-4k | 0.0% | | anonfolio-lkml-v1 | 0.7% | | anonfolio-lkml-v2-simple-order | -0.9% | | anonfolio-lkml-v2 | 0.5% |
What if we use 32K or 16K instead of 64K as default anonymous folio size? I suspect this app may have 32K or 16K anon folio as sweet spot.
Regards
Yin, Fengwei
Quoted 29 lines
Quoted 8 lines
With my v1 patch, there were significant quantities of
memory which could not be placed in the 64K bucket and were instead being allocated for the 32K and 16K buckets. With the proposed simplification, that memory ended up using the 4K bucket, so page faults increased by 2.75x compared to the v1 patch (although due to the 64K bucket, this number is still a bit lower than the baseline). So instead, I continue to calculate a folio order that is somewhere between the preferred order and 0. (See below for more details).I suppose the benchmark wasn't running under memory pressure, which is uncommon for client devices. It could be easier the other way around:
using 32/16KB shows regression whereas order-0 shows better performance under memory pressure.I'm not sure we should use v1 as the baseline. Unpatched kernel sounds more reasonable at this point. If 32/16KB is proven to be better in most scenarios including under memory pressure, we can reintroduce that policy. I highly doubt this is the case: we tried 16KB base page size on client devices, and overall, the regressions outweighs the benefits.
Quoted 2 lines
The patches are based on top of v6.4 plus Matthew Wilcox's
set_ptes()series [2], which is a hard dependency. I have a branch at [3].It's not clear to me why [2] is a hard dependency.
It seems to me we are getting close and I was hoping we could get into mm-unstable soon without depending on other series...
Quoted 32 lines · Fengwei
Quoted 20 lines · Yu Zhao
Quoted 6 lines · Ryan Roberts
Hi All,
This is v2 of a series to implement variable order, large folios for anonymous memory. The objective of this is to improve performance by allocating larger chunks of memory during anonymous page faults. See [1] for background.
Thanks for the quick response!
Quoted 10 lines
I've significantly reworked and simplified the patch set based on comments from Yu Zhao (thanks for all your feedback!). I've also renamed the feature to
VARIABLE_THP, on Yu's advice.The last patch is for arm64 to explicitly override the default
arch_wants_pte_order()and is intended as an example. If this series is accepted I suggest taking the first 4 patches through the mm tree and the arm64 change could be handled through the arm64 tree separately. Neither has any build dependency on the other.Quoted 8 lines
Quoted 6 lines
The one area where I haven't followed Yu's advice is in the determination of the size of folio to use. It was suggested that I have a single preferred large order, and if it doesn't fit in the VMA (due to exceeding VMA bounds, or there being existing overlapping populated PTEs, etc) then fallback immediately to order-0. It turned out that this approach caused a performance regression in the Speedometer benchmark.
I suppose it's regression against the v1, not the unpatched kernel.
From the performance data Ryan shared, it's against unpatched kernel:
Quoted 8 lines
Speedometer 2.0:
| kernel | runs_per_min | |:-------------------------------|---------------:| | baseline-4k | 0.0% | | anonfolio-lkml-v1 | 0.7% | | anonfolio-lkml-v2-simple-order | -0.9% | | anonfolio-lkml-v2 | 0.5% |
I see. Thanks.
A couple of questions:
1. Do we have a stddev?
2. Do we have a theory why it regressed?
Assuming no bugs, I don't see how a real regression could happen -- falling back to order-0 isn't different from the original behavior. Ryan, could you perf record and cat /proc/vmstat and share them?
Quoted 42 lines · Yu Zhao
Quoted 32 lines · Fengwei
Quoted 20 lines · Yu Zhao
Quoted 6 lines · Ryan Roberts
Hi All,
This is v2 of a series to implement variable order, large folios for anonymous memory. The objective of this is to improve performance by allocating larger chunks of memory during anonymous page faults. See [1] for background.
Thanks for the quick response!
Quoted 10 lines
I've significantly reworked and simplified the patch set based on comments from Yu Zhao (thanks for all your feedback!). I've also renamed the feature to
VARIABLE_THP, on Yu's advice.The last patch is for arm64 to explicitly override the default
arch_wants_pte_order()and is intended as an example. If this series is accepted I suggest taking the first 4 patches through the mm tree and the arm64 change could be handled through the arm64 tree separately. Neither has any build dependency on the other.Quoted 8 lines
Quoted 6 lines
The one area where I haven't followed Yu's advice is in the determination of the size of folio to use. It was suggested that I have a single preferred large order, and if it doesn't fit in the VMA (due to exceeding VMA bounds, or there being existing overlapping populated PTEs, etc) then fallback immediately to order-0. It turned out that this approach caused a performance regression in the Speedometer benchmark.
I suppose it's regression against the v1, not the unpatched kernel.
From the performance data Ryan shared, it's against unpatched kernel:
Quoted 8 lines
Speedometer 2.0:
| kernel | runs_per_min | |:-------------------------------|---------------:| | baseline-4k | 0.0% | | anonfolio-lkml-v1 | 0.7% | | anonfolio-lkml-v2-simple-order | -0.9% | | anonfolio-lkml-v2 | 0.5% |
Quoted 4 lines
I see. Thanks.
A couple of questions:
1. Do we have a stddev?
| kernel | mean_abs | std_abs | mean_rel | std_rel | |:------------------------- |-----------:|----------:|-----------:|----------:| | baseline-4k | 117.4 | 0.8 | 0.0% | 0.7% | | anonfolio-v1 | 118.2 | 1 | 0.7% | 0.9% | | anonfolio-v2-simple-order | 116.4 | 1.1 | -0.9% | 0.9% | | anonfolio-v2 | 118 | 1.2 | 0.5% | 1.0% |
This is with 3 runs per reboot across 5 reboots, with first run after reboot trimmed (it's always a bit slower, I assume due to cold page cache). So 10 data points per kernel in total.
I've rerun the test multiple times and see similar results each time.
I've also run anonfolio-v2 with Kconfig FLEXIBLE_THP=disabled and in this case I see the same performance as baseline-4k.
Quoted 1 line
2. Do we have a theory why it regressed?
I have a woolly hypothesis; I think Chromium is doing mmap/munmap in ways that mean when we fault, order-4 is often too big to fit in the VMA. So we fallback to order-0. I guess this is happening so often for this workload that the cost of doing the checks and fallback is outweighing the benefit of the memory that does end up with order-4 folios.
I've sampled the memory in each bucket (once per second) while running and its roughly:
64K: 25%
32K: 15%
16K: 15%
4K: 45%
32K and 16K obviously fold into the 4K bucket with anonfolio-v2-simple-order. But potentially, I suspect there is lots of mmap/unmap for the smaller sizes and the 64K contents is more static - that's just a guess though.
Quoted 3 lines
Assuming no bugs, I don't see how a real regression could happen -- falling back to order-0 isn't different from the original behavior. Ryan, could you
perf recordandcat /proc/vmstatand share them?
I can, but it will have to be a bit later in the week. I'll do some more test runs overnight so we have a larger number of runs - hopefully that might tell us that this is noise to a certain extent.
I'd still like to hear a clear technical argument for why the bin-packing approach is not the correct one!
Thanks,
Ryan
Quoted 76 lines · Ryan Roberts
Quoted 42 lines · Yu Zhao
Quoted 32 lines · Fengwei
Quoted 20 lines · Yu Zhao
Quoted 6 lines · Ryan Roberts
Hi All,
This is v2 of a series to implement variable order, large folios for anonymous memory. The objective of this is to improve performance by allocating larger chunks of memory during anonymous page faults. See [1] for background.
Thanks for the quick response!
Quoted 10 lines
I've significantly reworked and simplified the patch set based on comments from Yu Zhao (thanks for all your feedback!). I've also renamed the feature to
VARIABLE_THP, on Yu's advice.The last patch is for arm64 to explicitly override the default
arch_wants_pte_order()and is intended as an example. If this series is accepted I suggest taking the first 4 patches through the mm tree and the arm64 change could be handled through the arm64 tree separately. Neither has any build dependency on the other.Quoted 8 lines
Quoted 6 lines
The one area where I haven't followed Yu's advice is in the determination of the size of folio to use. It was suggested that I have a single preferred large order, and if it doesn't fit in the VMA (due to exceeding VMA bounds, or there being existing overlapping populated PTEs, etc) then fallback immediately to order-0. It turned out that this approach caused a performance regression in the Speedometer benchmark.
I suppose it's regression against the v1, not the unpatched kernel.
From the performance data Ryan shared, it's against unpatched kernel:
Quoted 8 lines
Speedometer 2.0:
| kernel | runs_per_min | |:-------------------------------|---------------:| | baseline-4k | 0.0% | | anonfolio-lkml-v1 | 0.7% | | anonfolio-lkml-v2-simple-order | -0.9% | | anonfolio-lkml-v2 | 0.5% |Quoted 4 lines
I see. Thanks.
A couple of questions:
1. Do we have a stddev?| kernel | mean_abs | std_abs | mean_rel | std_rel | |:------------------------- |-----------:|----------:|-----------:|----------:| | baseline-4k | 117.4 | 0.8 | 0.0% | 0.7% | | anonfolio-v1 | 118.2 | 1 | 0.7% | 0.9% | | anonfolio-v2-simple-order | 116.4 | 1.1 | -0.9% | 0.9% | | anonfolio-v2 | 118 | 1.2 | 0.5% | 1.0% |This is with 3 runs per reboot across 5 reboots, with first run after reboot trimmed (it's always a bit slower, I assume due to cold page cache). So 10 data points per kernel in total.
I've rerun the test multiple times and see similar results each time.
I've also run anonfolio-v2 with
KconfigFLEXIBLE_THP=disabled and in this case I see the same performance as baseline-4k.Quoted 1 line
2. Do we have a theory why it regressed?
I have a woolly hypothesis; I think Chromium is doing mmap/munmap in ways that mean when we fault, order-4 is often too big to fit in the VMA. So we fallback to order-0. I guess this is happening so often for this workload that the cost of doing the checks and fallback is outweighing the benefit of the memory that does end up with order-4 folios.
I've sampled the memory in each bucket (once per second) while running and its roughly:
Quoted 8 lines
64K: 25%
32K: 15%
16K: 15%
4K: 45%32K and 16K obviously fold into the 4K bucket with anonfolio-v2-simple-order. But potentially, I suspect there is lots of mmap/unmap for the smaller sizes and the 64K contents is more static - that's just a guess though.
So this is like out of vma range thing.
Quoted 11 lines
Quoted 4 lines
Assuming no bugs, I don't see how a real regression could happen -- falling back to order-0 isn't different from the original behavior. Ryan, could you
perf recordandcat /proc/vmstatand share them?I can, but it will have to be a bit later in the week. I'll do some more test runs overnight so we have a larger number of runs - hopefully that might tell us that this is noise to a certain extent.
I'd still like to hear a clear technical argument for why the bin-packing approach is not the correct one!
My understanding to Yu's (Yu, correct me if I am wrong) comments is that we postpone this part of change and make basic anon large folio support in. Then discuss which approach we should take. Maybe people will agree retry is the choice, maybe other approach will be taken...
For example, for this out of VMA range case, per VMA order should be considered. We don't need make decision that the retry should be taken now.
Regards
Yin, Fengwei
Quoted 6 lines
Thanks,
Ryan
Quoted 101 lines · Yin Fengwei
Quoted 76 lines · Ryan Roberts
Quoted 42 lines · Yu Zhao
Quoted 32 lines · Fengwei
Quoted 20 lines · Yu Zhao
Quoted 6 lines · Ryan Roberts
Hi All,
This is v2 of a series to implement variable order, large folios for anonymous memory. The objective of this is to improve performance by allocating larger chunks of memory during anonymous page faults. See [1] for background.
Thanks for the quick response!
Quoted 10 lines
I've significantly reworked and simplified the patch set based on comments from Yu Zhao (thanks for all your feedback!). I've also renamed the feature to
VARIABLE_THP, on Yu's advice.The last patch is for arm64 to explicitly override the default
arch_wants_pte_order()and is intended as an example. If this series is accepted I suggest taking the first 4 patches through the mm tree and the arm64 change could be handled through the arm64 tree separately. Neither has any build dependency on the other.Quoted 8 lines
Quoted 6 lines
The one area where I haven't followed Yu's advice is in the determination of the size of folio to use. It was suggested that I have a single preferred large order, and if it doesn't fit in the VMA (due to exceeding VMA bounds, or there being existing overlapping populated PTEs, etc) then fallback immediately to order-0. It turned out that this approach caused a performance regression in the Speedometer benchmark.
I suppose it's regression against the v1, not the unpatched kernel.
From the performance data Ryan shared, it's against unpatched kernel:
Quoted 8 lines
Speedometer 2.0:
| kernel | runs_per_min | |:-------------------------------|---------------:| | baseline-4k | 0.0% | | anonfolio-lkml-v1 | 0.7% | | anonfolio-lkml-v2-simple-order | -0.9% | | anonfolio-lkml-v2 | 0.5% |Quoted 4 lines
I see. Thanks.
A couple of questions:
1. Do we have a stddev?| kernel | mean_abs | std_abs | mean_rel | std_rel | |:------------------------- |-----------:|----------:|-----------:|----------:| | baseline-4k | 117.4 | 0.8 | 0.0% | 0.7% | | anonfolio-v1 | 118.2 | 1 | 0.7% | 0.9% | | anonfolio-v2-simple-order | 116.4 | 1.1 | -0.9% | 0.9% | | anonfolio-v2 | 118 | 1.2 | 0.5% | 1.0% |This is with 3 runs per reboot across 5 reboots, with first run after reboot trimmed (it's always a bit slower, I assume due to cold page cache). So 10 data points per kernel in total.
I've rerun the test multiple times and see similar results each time.
I've also run anonfolio-v2 with
KconfigFLEXIBLE_THP=disabled and in this case I see the same performance as baseline-4k.Quoted 1 line
2. Do we have a theory why it regressed?
I have a woolly hypothesis; I think Chromium is doing mmap/munmap in ways that mean when we fault, order-4 is often too big to fit in the VMA. So we fallback to order-0. I guess this is happening so often for this workload that the cost of doing the checks and fallback is outweighing the benefit of the memory that does end up with order-4 folios.
I've sampled the memory in each bucket (once per second) while running and its roughly:
Quoted 8 lines
64K: 25%
32K: 15%
16K: 15%
4K: 45%32K and 16K obviously fold into the 4K bucket with anonfolio-v2-simple-order. But potentially, I suspect there is lots of mmap/unmap for the smaller sizes and the 64K contents is more static - that's just a guess though.
So this is like out of vma range thing.
Quoted 11 lines
Quoted 4 lines
Assuming no bugs, I don't see how a real regression could happen -- falling back to order-0 isn't different from the original behavior. Ryan, could you
perf recordandcat /proc/vmstatand share them?I can, but it will have to be a bit later in the week. I'll do some more test runs overnight so we have a larger number of runs - hopefully that might tell us that this is noise to a certain extent.
I'd still like to hear a clear technical argument for why the bin-packing approach is not the correct one!
Quoted 7 lines
My understanding to Yu's (Yu, correct me if I am wrong) comments is that we postpone this part of change and make basic anon large folio support in. Then discuss which approach we should take. Maybe people will agree retry is the choice, maybe other approach will be taken...
For example, for this out of VMA range case, per VMA order should be considered. We don't need make decision that the retry should be taken now.
I've articulated the reasons in another email. Just summarize the most important point here:
using more fallback orders makes a system reach equilibrium faster, at which point it can't allocate the order of arch_wants_pte_order() anymore. IOW, this best-fit policy can reduce the number of folios of the h/w prefered order for a system running long enough.
Quoted 110 lines · Yu Zhao
Quoted 101 lines · Yin Fengwei
Quoted 76 lines · Ryan Roberts
Quoted 42 lines · Yu Zhao
Quoted 32 lines · Fengwei
Quoted 20 lines · Yu Zhao
Quoted 6 lines · Ryan Roberts
Hi All,
This is v2 of a series to implement variable order, large folios for anonymous memory. The objective of this is to improve performance by allocating larger chunks of memory during anonymous page faults. See [1] for background.
Thanks for the quick response!
Quoted 10 lines
I've significantly reworked and simplified the patch set based on comments from Yu Zhao (thanks for all your feedback!). I've also renamed the feature to
VARIABLE_THP, on Yu's advice.The last patch is for arm64 to explicitly override the default
arch_wants_pte_order()and is intended as an example. If this series is accepted I suggest taking the first 4 patches through the mm tree and the arm64 change could be handled through the arm64 tree separately. Neither has any build dependency on the other.Quoted 8 lines
Quoted 6 lines
The one area where I haven't followed Yu's advice is in the determination of the size of folio to use. It was suggested that I have a single preferred large order, and if it doesn't fit in the VMA (due to exceeding VMA bounds, or there being existing overlapping populated PTEs, etc) then fallback immediately to order-0. It turned out that this approach caused a performance regression in the Speedometer benchmark.
I suppose it's regression against the v1, not the unpatched kernel.
From the performance data Ryan shared, it's against unpatched kernel:
Quoted 8 lines
Speedometer 2.0:
| kernel | runs_per_min | |:-------------------------------|---------------:| | baseline-4k | 0.0% | | anonfolio-lkml-v1 | 0.7% | | anonfolio-lkml-v2-simple-order | -0.9% | | anonfolio-lkml-v2 | 0.5% |Quoted 4 lines
I see. Thanks.
A couple of questions:
1. Do we have a stddev?| kernel | mean_abs | std_abs | mean_rel | std_rel | |:------------------------- |-----------:|----------:|-----------:|----------:| | baseline-4k | 117.4 | 0.8 | 0.0% | 0.7% | | anonfolio-v1 | 118.2 | 1 | 0.7% | 0.9% | | anonfolio-v2-simple-order | 116.4 | 1.1 | -0.9% | 0.9% | | anonfolio-v2 | 118 | 1.2 | 0.5% | 1.0% |This is with 3 runs per reboot across 5 reboots, with first run after reboot trimmed (it's always a bit slower, I assume due to cold page cache). So 10 data points per kernel in total.
I've rerun the test multiple times and see similar results each time.
I've also run anonfolio-v2 with
KconfigFLEXIBLE_THP=disabled and in this case I see the same performance as baseline-4k.Quoted 1 line
2. Do we have a theory why it regressed?
I have a woolly hypothesis; I think Chromium is doing mmap/munmap in ways that mean when we fault, order-4 is often too big to fit in the VMA. So we fallback to order-0. I guess this is happening so often for this workload that the cost of doing the checks and fallback is outweighing the benefit of the memory that does end up with order-4 folios.
I've sampled the memory in each bucket (once per second) while running and its roughly:
Quoted 8 lines
64K: 25%
32K: 15%
16K: 15%
4K: 45%32K and 16K obviously fold into the 4K bucket with anonfolio-v2-simple-order. But potentially, I suspect there is lots of mmap/unmap for the smaller sizes and the 64K contents is more static - that's just a guess though.
So this is like out of vma range thing.
Quoted 11 lines
Quoted 4 lines
Assuming no bugs, I don't see how a real regression could happen -- falling back to order-0 isn't different from the original behavior. Ryan, could you
perf recordandcat /proc/vmstatand share them?I can, but it will have to be a bit later in the week. I'll do some more test runs overnight so we have a larger number of runs - hopefully that might tell us that this is noise to a certain extent.
I'd still like to hear a clear technical argument for why the bin-packing approach is not the correct one!
Quoted 7 lines
My understanding to Yu's (Yu, correct me if I am wrong) comments is that we postpone this part of change and make basic anon large folio support in. Then discuss which approach we should take. Maybe people will agree retry is the choice, maybe other approach will be taken...
For example, for this out of VMA range case, per VMA order should be considered. We don't need make decision that the retry should be taken now.
Quoted 6 lines
I've articulated the reasons in another email. Just summarize the most important point here:
using more fallback orders makes a system reach equilibrium faster, at which point it can't allocate the order ofarch_wants_pte_order()anymore. IOW, this best-fit policy can reduce the number of folios of the h/w prefered order for a system running long enough.
Thanks for taking the time to write all the arguments down. I understand what you are saying. If we are considering the whole system, then we also need to think about the page cache though, and that will allocate multiple orders, so you are still going to suffer fragmentation from that user.
That said, I like the proposal patch posted where we have up to 3 orders that we try in order of preference; hw-preferred, PAGE_ALLOC_COSTLY_ORDER and 0. That feels like a good compromise that allows me to fulfil my objectives. I'm going to pull this together into a v3 patch set and aim to post towards the end of the week.
Are you ok for me to add a Suggested-by: for you? (submitting-patches.rst says I need your explicit permission).
On the regression front, I've done a much bigger test run and see the regression is still present (although the mean has shifted a little bit). I've also built a kernel based on anonfolio-lkml-v2 but where arch_wants_pte_order() returns order-3. The aim was to test your hypothesis that 64K allocation is slow. This kernel is performing even better, so I think that confirms your hypothesis:
| kernel | runs_per_min | runs | sessions | |:-------------------------------|---------------:|-------:|-----------:| | baseline-4k | 0.0% | 75 | 15 | | anonfolio-lkml-v1 | 1.0% | 75 | 15 | | anonfolio-lkml-v2-simple-order | -0.4% | 75 | 15 | | anonfolio-lkml-v2 | 0.9% | 75 | 15 | | anonfolio-lkml-v2-32k | 1.4% | 10 | 5 |
Thanks,
Ryan
Quoted 119 lines · Ryan Roberts
Quoted 110 lines · Yu Zhao
Quoted 101 lines · Yin Fengwei
Quoted 76 lines · Ryan Roberts
Quoted 42 lines · Yu Zhao
Quoted 32 lines · Fengwei
Quoted 20 lines · Yu Zhao
Quoted 6 lines · Ryan Roberts
Hi All,
This is v2 of a series to implement variable order, large folios for anonymous memory. The objective of this is to improve performance by allocating larger chunks of memory during anonymous page faults. See [1] for background.
Thanks for the quick response!
Quoted 10 lines
I've significantly reworked and simplified the patch set based on comments from Yu Zhao (thanks for all your feedback!). I've also renamed the feature to
VARIABLE_THP, on Yu's advice.The last patch is for arm64 to explicitly override the default
arch_wants_pte_order()and is intended as an example. If this series is accepted I suggest taking the first 4 patches through the mm tree and the arm64 change could be handled through the arm64 tree separately. Neither has any build dependency on the other.Quoted 8 lines
Quoted 6 lines
The one area where I haven't followed Yu's advice is in the determination of the size of folio to use. It was suggested that I have a single preferred large order, and if it doesn't fit in the VMA (due to exceeding VMA bounds, or there being existing overlapping populated PTEs, etc) then fallback immediately to order-0. It turned out that this approach caused a performance regression in the Speedometer benchmark.
I suppose it's regression against the v1, not the unpatched kernel.
From the performance data Ryan shared, it's against unpatched kernel:
Quoted 8 lines
Speedometer 2.0:
| kernel | runs_per_min | |:-------------------------------|---------------:| | baseline-4k | 0.0% | | anonfolio-lkml-v1 | 0.7% | | anonfolio-lkml-v2-simple-order | -0.9% | | anonfolio-lkml-v2 | 0.5% |Quoted 4 lines
I see. Thanks.
A couple of questions:
1. Do we have a stddev?| kernel | mean_abs | std_abs | mean_rel | std_rel | |:------------------------- |-----------:|----------:|-----------:|----------:| | baseline-4k | 117.4 | 0.8 | 0.0% | 0.7% | | anonfolio-v1 | 118.2 | 1 | 0.7% | 0.9% | | anonfolio-v2-simple-order | 116.4 | 1.1 | -0.9% | 0.9% | | anonfolio-v2 | 118 | 1.2 | 0.5% | 1.0% |This is with 3 runs per reboot across 5 reboots, with first run after reboot trimmed (it's always a bit slower, I assume due to cold page cache). So 10 data points per kernel in total.
I've rerun the test multiple times and see similar results each time.
I've also run anonfolio-v2 with
KconfigFLEXIBLE_THP=disabled and in this case I see the same performance as baseline-4k.Quoted 1 line
2. Do we have a theory why it regressed?
I have a woolly hypothesis; I think Chromium is doing mmap/munmap in ways that mean when we fault, order-4 is often too big to fit in the VMA. So we fallback to order-0. I guess this is happening so often for this workload that the cost of doing the checks and fallback is outweighing the benefit of the memory that does end up with order-4 folios.
I've sampled the memory in each bucket (once per second) while running and its roughly:
Quoted 8 lines
64K: 25%
32K: 15%
16K: 15%
4K: 45%32K and 16K obviously fold into the 4K bucket with anonfolio-v2-simple-order. But potentially, I suspect there is lots of mmap/unmap for the smaller sizes and the 64K contents is more static - that's just a guess though.
So this is like out of vma range thing.
Quoted 11 lines
Quoted 4 lines
Assuming no bugs, I don't see how a real regression could happen -- falling back to order-0 isn't different from the original behavior. Ryan, could you
perf recordandcat /proc/vmstatand share them?I can, but it will have to be a bit later in the week. I'll do some more test runs overnight so we have a larger number of runs - hopefully that might tell us that this is noise to a certain extent.
I'd still like to hear a clear technical argument for why the bin-packing approach is not the correct one!
Quoted 7 lines
My understanding to Yu's (Yu, correct me if I am wrong) comments is that we postpone this part of change and make basic anon large folio support in. Then discuss which approach we should take. Maybe people will agree retry is the choice, maybe other approach will be taken...
For example, for this out of VMA range case, per VMA order should be considered. We don't need make decision that the retry should be taken now.
Quoted 6 lines
I've articulated the reasons in another email. Just summarize the most important point here:
using more fallback orders makes a system reach equilibrium faster, at which point it can't allocate the order ofarch_wants_pte_order()anymore. IOW, this best-fit policy can reduce the number of folios of the h/w prefered order for a system running long enough.
Quoted 4 lines
Thanks for taking the time to write all the arguments down. I understand what you are saying. If we are considering the whole system, then we also need to think about the page cache though, and that will allocate multiple orders, so you are still going to suffer fragmentation from that user.
1. page cache doesn't use the best-fit policy -- it has the advantage of having RA hit/miss numbers -- IOW, it doesn't try all orders without an estimated ROI.
2. page cache causes far less fragmentation in my experience: clean page cache gets reclaimed first under memory; unmapped page cache is less costly to migrate. Neither is true for anon, and what makes it worse is that heavy anon users usually enable zram/zswap: allocating memory (to store compressed data) under memory pressure makes reclaim/compaction even harder.
Quoted 8 lines
That said, I like the proposal patch posted where we have up to 3 orders that we try in order of preference; hw-preferred,
PAGE_ALLOC_COSTLY_ORDERand 0. That feels like a good compromise that allows me to fulfil my objectives. I'm going to pull this together into a v3 patch set and aim to post towards the end of the week.Are you ok for me to add a Suggested-by: for you? (
submitting-patches.rstsays I need your explicit permission).
Thanks for asking. No need to worry about it -- it's been a great team work with you, Fengwei, Yang et al.
I'm attaching a single patch containing all pieces I spelled out/implied/forgot to mention. It doesn't depend on other series -- I just stress-tested it on top of the latest mm-unstable. Please feel free to reuse any bits you see fit. Again no need to worry about Suggested-by.
Quoted 5 lines
On the regression front, I've done a much bigger test run and see the regression is still present (although the mean has shifted a little bit). I've also built a kernel based on anonfolio-lkml-v2 but where
arch_wants_pte_order()returns order-3. The aim was to test your hypothesis that 64K allocation is slow. This kernel is performing even better, so I think that confirms your hypothesis:
Great, thanks for confirming.
Quoted 7 lines
| kernel | runs_per_min | runs | sessions | |:-------------------------------|---------------:|-------:|-----------:| | baseline-4k | 0.0% | 75 | 15 | | anonfolio-lkml-v1 | 1.0% | 75 | 15 | | anonfolio-lkml-v2-simple-order | -0.4% | 75 | 15 | | anonfolio-lkml-v2 | 0.9% | 75 | 15 | | anonfolio-lkml-v2-32k | 1.4% | 10 | 5 |
Since we are all committed to the effort long term, the last number is good enough for the initial step to conclude. Hopefully v3 can address all pending comments and get into mm-unstable.
include/linux/pgtable.h +7 -0
@@ -313,6 +313,13 @@ static inline bool arch_has_hw_pte_young(void) }#endif +#ifndef arch_wants_pte_order +static inline int arch_wants_pte_order (void ) +{ +return 0 ; +} +#endif +#ifndef __HAVE_ARCH_PTEP_GET_AND_CLEAR static inline pte_t ptep_get_and_clear(struct mm_struct *mm, unsigned long address,
mm/memory.c +97 -14
152 lines not on this page · raw message
mm/rmap.c +18 -7
@@ -1284,25 +1284,36 @@ void page_add_anon_rmap(struct page *page, struct vm_area_struct *vma,void folio_add_new_anon_rmap (struct folio *folio,struct vm_area_struct *vma,unsigned long address) { -int nr; +int nr =folio_nr_pages (folio); -VM_BUG_ON_VMA (address < vma->vm_start || address >= vma->vm_end, vma); +VM_BUG_ON_VMA (address < vma->vm_start || address +PAGE_SIZE * nr > vma->vm_end, vma);__folio_set_swapbacked (folio); -if (likely (!folio_test_pmd_mappable (folio))) { +if (!folio_test_large (folio)) {/* increment count (starts at -1) */ atomic_set (&folio->_mapcount,0 ); - nr =1 ; +__page_set_anon_rmap (folio, &folio->page, vma, address,1 ); + }else if (!folio_test_pmd_mappable (folio)) { +int i; + +for (i =0 ; i < nr; i++) { +struct page *page =folio_page (folio, i); + +/* increment count (starts at -1) */ +atomic_set (&page->_mapcount,0 ); +__page_set_anon_rmap (folio, page, vma, address +PAGE_SIZE * i,1 ); + } +/* increment count (starts at 0) */ +atomic_set (&folio->_nr_pages_mapped, nr); }else {/* increment count (starts at -1) */ atomic_set (&folio->_entire_mapcount,0 );atomic_set (&folio->_nr_pages_mapped,COMPOUND_MAPPED ); - nr =folio_nr_pages (folio); +__page_set_anon_rmap (folio, &folio->page, vma, address,1 );__lruvec_stat_mod_folio (folio,NR_ANON_THPS , nr); }__lruvec_stat_mod_folio (folio,NR_ANON_MAPPED , nr); -__page_set_anon_rmap (folio, &folio->page, vma, address,1 ); }/**
@@ -1430,7 +1441,7 @@ void page_remove_rmap(struct page *page, struct vm_area_struct *vma,* page of the folio is unmapped and at least one page * is still mapped. */ -if (folio_test_pmd_mappable (folio) &&folio_test_anon (folio)) +if (folio_test_large (folio) &&folio_test_anon (folio))if (!compound || nr < nr_pmdmapped)deferred_split_folio (folio); }
Quoted 116 lines · Ryan Roberts
Hi All,
This is v2 of a series to implement variable order, large folios for anonymous memory. The objective of this is to improve performance by allocating larger chunks of memory during anonymous page faults. See [1] for background.
I've significantly reworked and simplified the patch set based on comments from Yu Zhao (thanks for all your feedback!). I've also renamed the feature to
VARIABLE_THP, on Yu's advice.The last patch is for arm64 to explicitly override the default
arch_wants_pte_order()and is intended as an example. If this series is accepted I suggest taking the first 4 patches through the mm tree and the arm64 change could be handled through the arm64 tree separately. Neither has any build dependency on the other.The one area where I haven't followed Yu's advice is in the determination of the size of folio to use. It was suggested that I have a single preferred large order, and if it doesn't fit in the VMA (due to exceeding VMA bounds, or there being existing overlapping populated PTEs, etc) then fallback immediately to order-0. It turned out that this approach caused a performance regression in the Speedometer benchmark. With my v1 patch, there were significant quantities of memory which could not be placed in the 64K bucket and were instead being allocated for the 32K and 16K buckets. With the proposed simplification, that memory ended up using the 4K bucket, so page faults increased by 2.75x compared to the v1 patch (although due to the 64K bucket, this number is still a bit lower than the baseline). So instead, I continue to calculate a folio order that is somewhere between the preferred order and 0. (See below for more details).
The patches are based on top of v6.4 plus Matthew Wilcox's
set_ptes()series [2], which is a hard dependency. I have a branch at [3].Changes since v1 [1]
--------------------- removed changes to arch-dependent
vma_alloc_zeroed_movable_folio()
- replaced with arch-independentalloc_anon_folio()
- follows THP allocation approach
- no longer retry with intermediate orders if allocation fails
- fallback directly to order-0
- removefolio_add_new_anon_rmap_range()patch
- instead add its new functionality tofolio_add_new_anon_rmap()
- remove batch-zap pte mappings optimization patch
- remove enablerfolio_remove_rmap_range()patch too
- These offer real perf improvement so will submit separately
- simplifyKconfig
- singleFLEXIBLE_THPoption, which is independent of arch
- depends onTRANSPARENT_HUGEPAGE
- when enabled default to max anon folio size of 64K unless arch explicitly overrides
- simplify changes todo_anonymous_page():
- no more retry loopPerformance
-----------Below results show 3 benchmarks; kernel compilation with 8 jobs, kernel compilation with 80 jobs, and speedometer 2.0 (a javascript benchmark running in Chromium). All cases are running on Ampere Altra with 1 NUMA node enabled, Ubuntu 22.04 and XFS filesystem. Each benchmark is repeated 15 times over 5 reboots and averaged.
'anonfolio-lkml-v1' is the v1 patchset at [1]. 'anonfolio-lkml-v2' is this v2 patchset. 'anonfolio-lkml-v2-simple-order' is anonfolio-lkml-v2 but with the order selection simplification that Yu Zhao suggested - I'm trying to justify here why I did not follow the advice.
Kernel compilation with 8 jobs:
| kernel | real-time | kern-time | user-time | |:-------------------------------|------------:|------------:|------------:| | baseline-4k | 0.0% | 0.0% | 0.0% | | anonfolio-lkml-v1 | -5.3% | -42.9% | -0.6% | | anonfolio-lkml-v2-simple-order | -4.4% | -36.5% | -0.4% | | anonfolio-lkml-v2 | -4.8% | -38.6% | -0.6% |We can see that the simple-order approach is responsible for a regression of 0.4%.
Kernel compilation with 80 jobs:
| kernel | real-time | kern-time | user-time | |:-------------------------------|------------:|------------:|------------:| | baseline-4k | 0.0% | 0.0% | 0.0% | | anonfolio-lkml-v1 | -4.6% | -45.7% | 1.4% | | anonfolio-lkml-v2-simple-order | -4.7% | -40.2% | -0.1% | | anonfolio-lkml-v2 | -5.0% | -42.6% | -0.3% |simple-order costs 0.3 % here. v2 is actually performing higher than v1 due to fixing the v1 regression on user-time.
Speedometer 2.0:
| kernel | runs_per_min | |:-------------------------------|---------------:| | baseline-4k | 0.0% | | anonfolio-lkml-v1 | 0.7% | | anonfolio-lkml-v2-simple-order | -0.9% | | anonfolio-lkml-v2 | 0.5% |simple-order regresses performance by 0.9% vs the baseline, for a total negative swing of 1.6% vs v1. This is fixed by keeping the more complex order selection mechanism from v1.
The remaining (kernel time) performance gap between v1 and v2 for the above benchmarks is due to the removal of the "batch zap" patch in v2. Adding that back in gives us the performance back. I intend to submit that as a separate series once this series is accepted.
Quoted 6 lines
[1] https://lore.kernel.org/linux-mm/[email protected]/
[2] https://lore.kernel.org/linux-mm/[email protected]/
[3] https://gitlab.arm.com/linux-arm/linux-rr/-/tree/features/granule_perf/anonfolio-lkml_v2Thanks,
Ryan
Hi Ryan,
is page migration already working as expected (what about page compaction?), and do we handle migration -ENOMEM when allocating a target page: do we split an fallback to 4k page migration?
Signature · 3 lines
Cheers, David / dhildenb
Quoted 7 lines · David Hildenbrand
Quoted 6 lines · Ryan Roberts
Hi All,
This is v2 of a series to implement variable order, large folios for anonymous memory. The objective of this is to improve performance by allocating larger chunks of memory during anonymous page faults. See [1] for background.
[...]
Quoted 9 lines
Quoted 2 lines
Thanks,
RyanHi Ryan,
is page migration already working as expected (what about page compaction?), and do we handle migration
-ENOMEMwhen allocating a target page: do we split an fallback to 4k page migration?
Hi David, All,
This series aims to be the bare minimum to demonstrate allocation of large anon folios. As such, there is a laundry list of things that need to be done for this feature to play nicely with other features. My preferred route is to merge this with it's Kconfig defaulted to disabled, and its Kconfig description clearly shouting that it's EXPERIMENTAL with an explanation of why (similar to READ_ONLY_THP_FOR_FS).
That said, I've put together a table of the items that I'm aware of that need attention. It would be great if people can review and add any missing items. Then we can hopefully parallelize the implementation work. David, I don't think the items you raised are covered - would you mind providing a bit more detail so I can add them to the list? (or just add them to the list yourself, if you prefer).
---
- item:
mlock
description: >-
Large, pte-mapped folios are ignored when mlock is requested. Code comment for mlock_vma_folio() says "...filter out pte mappings of THPs, which cannot be consistently counted: a pte mapping of the THP head cannot be distinguished by the page alone."
location:
- mlock_pte_range()
- mlock_vma_folio()
assignee:
Yin, Fengwei
- item:
numa balancing
description: >-
Large, pte-mapped folios are ignored by numa-balancing code. Commit comment (e81c480): "We're going to have THP mapped with PTEs. It will confuse numabalancing. Let's skip them for now."
location:
- do_numa_page()
assignee:
<none>
- item:
madvise
description: >-
MADV_COLD, MADV_PAGEOUT, MADV_FREE: For large folios, code assumes exclusive only if mapcount==1, else skips remainder of operation. For large, pte-mapped folios, exclusive folios can have mapcount upto nr_pages and still be exclusive. Even better; don't split the folio if it fits entirely within the range? Discussion at
https://lore.kernel.org/linux-mm/[email protected]/
talks about changing folio mapcounting - may help determine if exclusive without pgtable scan?
location:
- madvise_cold_or_pageout_pte_range()
- madvise_free_pte_range()
assignee:
<none>
- item:
shrink_folio_list
description: >-
Raised by Yu Zhao; I can't see the problem in the code - need clarification
location:
- shrink_folio_list()
assignee:
<none>
- item:
compaction
description: >-
Raised at LSFMM: Compaction skips non-order-0 pages. Already problem for page-cache pages today. Is my understand correct?
location:
- <where?>
assignee:
<none>
---
Thanks,
Ryan
Quoted 14 lines · Ryan Roberts
Quoted 7 lines · David Hildenbrand
Quoted 6 lines · Ryan Roberts
Hi All,
This is v2 of a series to implement variable order, large folios for anonymous memory. The objective of this is to improve performance by allocating larger chunks of memory during anonymous page faults. See [1] for background.
[...]
Quoted 3 lines
Quoted 2 lines
Thanks,
Ryan
Quoted 8 lines
Quoted 6 lines
Hi Ryan,
is page migration already working as expected (what about page compaction?), and do we handle migration
-ENOMEMwhen allocating a target page: do we split an fallback to 4k page migration?Hi David, All,
Hi Ryan,
thanks a lot for the list.
But can you comment on the page migration part (IOW did you try it already)?
For example, memory hotunplug, CMA, MCE handling, compaction all rely on page migration of something that was allocated using GFP_MOVABLE to actually work.
Compaction seems to skip any higher-order folios, but the question is if the udnerlying migration itself works.
If it already works: great! If not, this really has to be tackled early, because otherwise we'll be breaking the GFP_MOVABLE semantics.
Quoted 7 lines
This series aims to be the bare minimum to demonstrate allocation of large anon folios. As such, there is a laundry list of things that need to be done for this feature to play nicely with other features. My preferred route is to merge this with it's
Kconfigdefaulted to disabled, and itsKconfigdescription clearly shouting that it's EXPERIMENTAL with an explanation of why (similar toREAD_ONLY_THP_FOR_FS).
As long as we are not sure about the user space control and as long as basic functionality is not working (example, page migration), I would tend to not merge this early just for the sake of it.
But yes, something like mlock can eventually be tackled later: as long as there is a runtime interface to disable it ;)
Quoted 84 lines
That said, I've put together a table of the items that I'm aware of that need attention. It would be great if people can review and add any missing items. Then we can hopefully parallelize the implementation work. David, I don't think the items you raised are covered - would you mind providing a bit more detail so I can add them to the list? (or just add them to the list yourself, if you prefer).
---
- item:
mlockdescription: >-
Large, pte-mapped folios are ignored when mlock is requested. Code comment formlock_vma_folio()says "...filter out pte mappings of THPs, which cannot be consistently counted: a pte mapping of the THP head cannot be distinguished by the page alone."location:
-mlock_pte_range()
-mlock_vma_folio()assignee:
Yin, Fengwei- item:
numa balancingdescription: >-
Large, pte-mapped folios are ignored by numa-balancing code. Commit comment (e81c480): "We're going to have THP mapped with PTEs. It will confuse numabalancing. Let's skip them for now."location:
-do_numa_page()assignee: <none>- item:
madvisedescription: >-
MADV_COLD,MADV_PAGEOUT,MADV_FREE: For large folios, code assumes exclusive only if mapcount==1, else skips remainder of operation. For large, pte-mapped folios, exclusive folios can have mapcount uptonr_pagesand still be exclusive. Even better; don't split the folio if it fits entirely within the range? Discussion athttps://lore.kernel.org/linux-mm/[email protected]/
talks about changing folio mapcounting - may help determine if exclusive without pgtable scan?location:
-madvise_cold_or_pageout_pte_range()
-madvise_free_pte_range()assignee: <none>- item:
shrink_folio_listdescription: >-
Raised by Yu Zhao; I can't see the problem in the code - need clarificationlocation:
-shrink_folio_list()assignee: <none>- item:
compactiondescription: >-
Raised at LSFMM: Compaction skips non-order-0 pages. Already problem for page-cache pages today. Is my understand correct?
Quoted 5 lines
location:
- <where?>assignee: <none>
I'm still thinking about the whole mapcount thingy (and I burned way too much time on that yesterday), which is a big item for such a list and affects some of these items.
A pagetable scan is pretty much irrelevant for order-2 pages. But once we're talking about higher orders we really don't want to do that.
I'm preparing a writeup with users and challenges.
Is swapping working as expected? zswap?
Signature · 3 lines
Cheers, David / dhildenb
Quoted 7 lines · David Hildenbrand
On 06.07.23 10:02, Ryan Roberts wrote:
But can you comment on the page migration part (IOW did you try it already)?For example, memory hotunplug, CMA, MCE handling, compaction all rely on page migration of something that was allocated using
GFP_MOVABLEto actually work.
Quoted 5 lines
Compaction seems to skip any higher-order folios, but the question is if the udnerlying migration itself works.
If it already works: great! If not, this really has to be tackled early, because otherwise we'll be breaking the
GFP_MOVABLEsemantics.
I have looked at this a bit. _Migration_ should be fine. _Compaction_ is not.
If you look at a function like folio_migrate_mapping(), it all seems appropriately folio-ised. There might be something in there that is slightly wrong, but that would just be a bug to fix, not a huge architectural problem.
The problem comes in the callers of migrate_pages(). They pass a new_folio_t callback. alloc_migration_target() is the usual one passed and as far as I can tell is fine. I've seen no problems reported with it.
compaction_alloc() is a disaster, and I don't know how to fix it. The compaction code has its own allocator which is populated with order-0 folios. How it populates that freelist is awful ... see split_map_pages()
Quoted 1 line
Is swapping working as expected? zswap?
Suboptimally. Swap will split folios in order to swap them. Somebody needs to fix that, but it should work.
Quoted 8 lines · Matthew Wilcox
Quoted 7 lines · David Hildenbrand
On 06.07.23 10:02, Ryan Roberts wrote:
But can you comment on the page migration part (IOW did you try it already)?For example, memory hotunplug, CMA, MCE handling, compaction all rely on page migration of something that was allocated using
GFP_MOVABLEto actually work.
Quoted 8 lines
Quoted 5 lines
Compaction seems to skip any higher-order folios, but the question is if the udnerlying migration itself works.
If it already works: great! If not, this really has to be tackled early, because otherwise we'll be breaking the
GFP_MOVABLEsemantics.I have looked at this a bit. _Migration_ should be fine. _Compaction_ is not.
Thanks! Very nice if at least ordinary migration works.
Quoted 6 lines
If you look at a function like
folio_migrate_mapping(), it all seems appropriately folio-ised. There might be something in there that is slightly wrong, but that would just be a bug to fix, not a huge architectural problem.
Quoted 7 lines
The problem comes in the callers of
migrate_pages(). They pass anew_folio_tcallback.alloc_migration_target()is the usual one passed and as far as I can tell is fine. I've seen no problems reported with it.
compaction_alloc()is a disaster, and I don't know how to fix it. The compaction code has its own allocator which is populated with order-0 folios. How it populates that freelist is awful ... seesplit_map_pages()
Yeah, all that code was written under the assumption that we're moving order-0 pages (which is what the anon+pagecache pages part).
From what I recall, we're allocating order-0 pages from the high memory addresses, so we can migrate from low memory addresses, effectively freeing up low memory addresses and filling high memory addresses.
Adjusting that will be ... interesting. Instead of allocating order-0 pages from high addresses, we might want to allocate "as large as possible" ("grab what we can") from high addresses and then have our own kind of buddy for allocating from that pool a compaction destination page, depending on our source page. Nasty.
What should always work is the split->migrate. But that's definitely not what we want in many cases.
Quoted 5 lines
Quoted 2 lines
Is swapping working as expected? zswap?
Suboptimally. Swap will split folios in order to swap them. Somebody needs to fix that, but it should work.
Good!
It would be great to have some kind of a feature matrix that tells us what works perfectly, sub-optimally, barely, not at all (and what has not been tested). Maybe (likely!) we'll also find things that are sub-optimal for ordinary THP (like swapping, not even sure about).
I suspect that KSM should work mostly fine with flexible-thp. When deduplciating, we'll simply split the compound page and proceed as expected. But might be worth testing as well.
Signature · 3 lines
Cheers, David / dhildenb
Quoted 12 lines · David Hildenbrand
Quoted 11 lines · Matthew Wilcox
Quoted 10 lines · David Hildenbrand
On 06.07.23 10:02, Ryan Roberts wrote:
But can you comment on the page migration part (IOW did you try it already)?For example, memory hotunplug, CMA, MCE handling, compaction all rely on page migration of something that was allocated using
GFP_MOVABLEto actually work.Compaction seems to skip any higher-order folios, but the question is if the udnerlying migration itself works.
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
If it already works: great! If not, this really has to be tackled early, because otherwise we'll be breaking the
GFP_MOVABLEsemantics.I have looked at this a bit. _Migration_ should be fine. _Compaction_ is not.
Thanks! Very nice if at least ordinary migration works.
That's good to hear - I hadn't personally investigated.
Quoted 7 lines
Quoted 7 lines
If you look at a function like
folio_migrate_mapping(), it all seems appropriately folio-ised. There might be something in there that is slightly wrong, but that would just be a bug to fix, not a huge architectural problem.
Quoted 7 lines
Quoted 7 lines
The problem comes in the callers of migrate_pages(). They pass a
new_folio_tcallback.alloc_migration_target()is the usual one passed and as far as I can tell is fine. I've seen no problems reported with it.
compaction_alloc()is a disaster, and I don't know how to fix it. The compaction code has its own allocator which is populated with order-0 folios. How it populates that freelist is awful ... seesplit_map_pages()
I think this compaction issue also affects large folios in the page cache? So really it is a pre-existing bug in the code base that needs to be fixed independently of large anon folios? Should I assume you are tackling this, Matthew?
Quoted 22 lines
Yeah, all that code was written under the assumption that we're moving order-0 pages (which is what the anon+pagecache pages part).
From what I recall, we're allocating order-0 pages from the high memory addresses, so we can migrate from low memory addresses, effectively freeing up low memory addresses and filling high memory addresses.
Adjusting that will be ... interesting. Instead of allocating order-0 pages from high addresses, we might want to allocate "as large as possible" ("grab what we can") from high addresses and then have our own kind of buddy for allocating from that pool a compaction destination page, depending on our source page. Nasty.
What should always work is the
split->migrate. But that's definitely not what we want in many cases.Quoted 5 lines
Quoted 2 lines
Is swapping working as expected? zswap?
Suboptimally. Swap will split folios in order to swap them. Somebody needs to fix that, but it should work.
Quoted 6 lines
Good!
It would be great to have some kind of a feature matrix that tells us what works perfectly, sub-optimally, barely, not at all (and what has not been tested). Maybe (likely!) we'll also find things that are sub-optimal for ordinary THP (like swapping, not even sure about).
I'm building a list of known issues, but so far it has been based on code I've found during review and things raised by people in these threads. Are there test suites that explicitly test these features? If so I'll happily run them against large anon folios, but at the moment I'm ignorant I'm afraid. I have been trying to get mm selftests up and running, but I currently have a bunch of failures on arm64, even without any of my patches - somthing I'm working through.
Quoted 5 lines
I suspect that KSM should work mostly fine with flexible-thp. When deduplciating, we'll simply split the compound page and proceed as expected. But might be worth testing as well.
Quoted 3 lines · Ryan Roberts
I think this compaction issue also affects large folios in the page cache? So really it is a pre-existing bug in the code base that needs to be fixed independently of large anon folios? Should I assume you are tackling this, Matthew?
It does need to be fixed independently of large anon folios. Said fix should probably be backported to 6.1 once it's suitably stable. However, I'm not working on it. I have a lot of projects and this one's a missed-opportunity, not a show-stopper. Sounds like Zi Yan might be interested in tackling it though!
Quoted 34 lines · David Hildenbrand
Quoted 8 lines · Matthew Wilcox
Quoted 7 lines · David Hildenbrand
On 06.07.23 10:02, Ryan Roberts wrote:
But can you comment on the page migration part (IOW did you try it already)?For example, memory hotunplug, CMA, MCE handling, compaction all rely on page migration of something that was allocated using
GFP_MOVABLEto actually work.Quoted 8 lines
Quoted 5 lines
Compaction seems to skip any higher-order folios, but the question is if the udnerlying migration itself works.
If it already works: great! If not, this really has to be tackled early, because otherwise we'll be breaking the
GFP_MOVABLEsemantics.I have looked at this a bit. _Migration_ should be fine. _Compaction_ is not.
Thanks! Very nice if at least ordinary migration works.
Quoted 6 lines
If you look at a function like
folio_migrate_mapping(), it all seems appropriately folio-ised. There might be something in there that is slightly wrong, but that would just be a bug to fix, not a huge architectural problem.Quoted 7 lines
The problem comes in the callers of
migrate_pages(). They pass anew_folio_tcallback.alloc_migration_target()is the usual one passed and as far as I can tell is fine. I've seen no problems reported with it.
compaction_alloc()is a disaster, and I don't know how to fix it. The compaction code has its own allocator which is populated with order-0 folios. How it populates that freelist is awful ... seesplit_map_pages()
Quoted 5 lines
Yeah, all that code was written under the assumption that we're moving order-0 pages (which is what the anon+pagecache pages part).
From what I recall, we're allocating order-0 pages from the high memory addresses, so we can migrate from low memory addresses, effectively freeing up low memory addresses and filling high memory addresses.
Adjusting that will be ... interesting. Instead of allocating order-0 pages from high addresses, we might want to allocate "as large as possible" ("grab what we can") from high addresses and then have our own kind of buddy for allocating from that pool a compaction destination page, depending on our source page. Nasty.
We probably do not need a pool, since before migration, we have isolated folios to be migrated and can come up with a stats on how many folios there are at each order. Then, we can isolate free pages based on the stats and do not split free pages all the way down to order-0. We can sort the source folios based on their orders and isolate free pages from largest order to smallest order. That could avoid a free page pool.
--
Best Regards,
Yan, Zi
Quoted 40 lines · Zi Yan
Quoted 38 lines · David Hildenbrand
Quoted 8 lines · Matthew Wilcox
Quoted 7 lines · David Hildenbrand
On 06.07.23 10:02, Ryan Roberts wrote:
But can you comment on the page migration part (IOW did you try it already)?For example, memory hotunplug, CMA, MCE handling, compaction all rely on page migration of something that was allocated using
GFP_MOVABLEto actually work.Quoted 8 lines
Quoted 5 lines
Compaction seems to skip any higher-order folios, but the question is if the udnerlying migration itself works.
If it already works: great! If not, this really has to be tackled early, because otherwise we'll be breaking the
GFP_MOVABLEsemantics.I have looked at this a bit. _Migration_ should be fine. _Compaction_ is not.
Thanks! Very nice if at least ordinary migration works.
Quoted 6 lines
If you look at a function like
folio_migrate_mapping(), it all seems appropriately folio-ised. There might be something in there that is slightly wrong, but that would just be a bug to fix, not a huge architectural problem.Quoted 7 lines
The problem comes in the callers of
migrate_pages(). They pass anew_folio_tcallback.alloc_migration_target()is the usual one passed and as far as I can tell is fine. I've seen no problems reported with it.
compaction_alloc()is a disaster, and I don't know how to fix it. The compaction code has its own allocator which is populated with order-0 folios. How it populates that freelist is awful ... seesplit_map_pages()Yeah, all that code was written under the assumption that we're moving order-0 pages (which is what the anon+pagecache pages part).
From what I recall, we're allocating order-0 pages from the high memory addresses, so we can migrate from low memory addresses, effectively freeing up low memory addresses and filling high memory addresses.
Quoted 8 lines
Quoted 1 line
Adjusting that will be ... interesting. Instead of allocating order-0 pages from high addresses, we might want to allocate "as large as possible" ("grab what we can") from high addresses and then have our own kind of buddy for allocating from that pool a compaction destination page, depending on our source page. Nasty.
We probably do not need a pool, since before migration, we have isolated folios to be migrated and can come up with a stats on how many folios there are at each order. Then, we can isolate free pages based on the stats and do not split free pages all the way down to order-0. We can sort the source folios based on their orders and isolate free pages from largest order to smallest order. That could avoid a free page pool.
Hi Zi, I just wanted to check; is this something you are working on or planning to work on? I'm trying to maintain a list of all the items that need to get sorted for large anon folios. It would be great to put your name against it! ;-)
Quoted 4 lines
--
Best Regards,
Yan, Zi
Quoted 50 lines · Ryan Roberts
Quoted 40 lines · Zi Yan
Quoted 38 lines · David Hildenbrand
Quoted 8 lines · Matthew Wilcox
Quoted 7 lines · David Hildenbrand
On 06.07.23 10:02, Ryan Roberts wrote:
But can you comment on the page migration part (IOW did you try it already)?For example, memory hotunplug, CMA, MCE handling, compaction all rely on page migration of something that was allocated using
GFP_MOVABLEto actually work.Quoted 8 lines
Quoted 5 lines
Compaction seems to skip any higher-order folios, but the question is if the udnerlying migration itself works.
If it already works: great! If not, this really has to be tackled early, because otherwise we'll be breaking the
GFP_MOVABLEsemantics.I have looked at this a bit. _Migration_ should be fine. _Compaction_ is not.
Thanks! Very nice if at least ordinary migration works.
Quoted 6 lines
If you look at a function like
folio_migrate_mapping(), it all seems appropriately folio-ised. There might be something in there that is slightly wrong, but that would just be a bug to fix, not a huge architectural problem.Quoted 7 lines
The problem comes in the callers of
migrate_pages(). They pass anew_folio_tcallback.alloc_migration_target()is the usual one passed and as far as I can tell is fine. I've seen no problems reported with it.
compaction_alloc()is a disaster, and I don't know how to fix it. The compaction code has its own allocator which is populated with order-0 folios. How it populates that freelist is awful ... seesplit_map_pages()Yeah, all that code was written under the assumption that we're moving order-0 pages (which is what the anon+pagecache pages part).
From what I recall, we're allocating order-0 pages from the high memory addresses, so we can migrate from low memory addresses, effectively freeing up low memory addresses and filling high memory addresses.
Quoted 8 lines
Quoted 1 line
Adjusting that will be ... interesting. Instead of allocating order-0 pages from high addresses, we might want to allocate "as large as possible" ("grab what we can") from high addresses and then have our own kind of buddy for allocating from that pool a compaction destination page, depending on our source page. Nasty.
We probably do not need a pool, since before migration, we have isolated folios to be migrated and can come up with a stats on how many folios there are at each order. Then, we can isolate free pages based on the stats and do not split free pages all the way down to order-0. We can sort the source folios based on their orders and isolate free pages from largest order to smallest order. That could avoid a free page pool.
Quoted 3 lines
Hi Zi, I just wanted to check; is this something you are working on or planning to work on? I'm trying to maintain a list of all the items that need to get sorted for large anon folios. It would be great to put your name against it! ;-)
Sure. I can work on this one.
--
Best Regards,
Yan, Zi
Quoted 52 lines · Zi Yan
Quoted 50 lines · Ryan Roberts
Quoted 48 lines · Zi Yan
Quoted 34 lines · David Hildenbrand
Quoted 8 lines · Matthew Wilcox
Quoted 7 lines · David Hildenbrand
On 06.07.23 10:02, Ryan Roberts wrote:
But can you comment on the page migration part (IOW did you try it already)?For example, memory hotunplug, CMA, MCE handling, compaction all rely on page migration of something that was allocated using
GFP_MOVABLEto actually work.Quoted 8 lines
Quoted 5 lines
Compaction seems to skip any higher-order folios, but the question is if the udnerlying migration itself works.
If it already works: great! If not, this really has to be tackled early, because otherwise we'll be breaking the
GFP_MOVABLEsemantics.I have looked at this a bit. _Migration_ should be fine. _Compaction_ is not.
Thanks! Very nice if at least ordinary migration works.
Quoted 6 lines
If you look at a function like
folio_migrate_mapping(), it all seems appropriately folio-ised. There might be something in there that is slightly wrong, but that would just be a bug to fix, not a huge architectural problem.Quoted 7 lines
The problem comes in the callers of
migrate_pages(). They pass anew_folio_tcallback.alloc_migration_target()is the usual one passed and as far as I can tell is fine. I've seen no problems reported with it.
compaction_alloc()is a disaster, and I don't know how to fix it. The compaction code has its own allocator which is populated with order-0 folios. How it populates that freelist is awful ... seesplit_map_pages()Quoted 5 lines
Yeah, all that code was written under the assumption that we're moving order-0 pages (which is what the anon+pagecache pages part).
From what I recall, we're allocating order-0 pages from the high memory addresses, so we can migrate from low memory addresses, effectively freeing up low memory addresses and filling high memory addresses.
Adjusting that will be ... interesting. Instead of allocating order-0 pages from high addresses, we might want to allocate "as large as possible" ("grab what we can") from high addresses and then have our own kind of buddy for allocating from that pool a compaction destination page, depending on our source page. Nasty.
We probably do not need a pool, since before migration, we have isolated folios to be migrated and can come up with a stats on how many folios there are at each order. Then, we can isolate free pages based on the stats and do not split free pages all the way down to order-0. We can sort the source folios based on their orders and isolate free pages from largest order to smallest order. That could avoid a free page pool.
Quoted 5 lines
Quoted 3 lines
Hi Zi, I just wanted to check; is this something you are working on or planning to work on? I'm trying to maintain a list of all the items that need to get sorted for large anon folios. It would be great to put your name against it! ;-)
Sure. I can work on this one.
Awesome - thanks!
Quoted 4 lines
--
Best Regards,
Yan, Zi
Quoted 5 lines · Matthew Wilcox
Quoted 1 line · David Hildenbrand
Is swapping working as expected? zswap?
Suboptimally. Swap will split folios in order to swap them.
Wouldn't that mean if high order folios are used a lot but swap is also used, until this is fixed you wouldn't get the expected reclaim gains for high order folios and we'd need compaction more then?
Quoted 1 line
Somebody needs to fix that, but it should work.
As we look at shmem stuff it was on the path so something we have considered doing. Ie, it's on our team's list of items to help with but currently on a backburner.
Luis
Quoted 10 lines · Luis Chamberlain
Quoted 5 lines · Matthew Wilcox
Quoted 1 line · David Hildenbrand
Is swapping working as expected? zswap?
Suboptimally. Swap will split folios in order to swap them.
Wouldn't that mean if high order folios are used a lot but swap is also used, until this is fixed you wouldn't get the expected reclaim gains for high order folios and we'd need compaction more then?
They're split in shrink_folio_list(), so they stay intact until that point?
Quoted 5 lines
Quoted 1 line
Somebody needs to fix that, but it should work.
As we look at shmem stuff it was on the path so something we have considered doing. Ie, it's on our team's list of items to help with but currently on a backburner.
Something I was thinking about is that you'll need to prohibit swap devices or swap files being created on large block devices. Until we rewrite the entire swap subsystem ...
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel