mm: rmap: merge HugeTLB mapcount logic with THPs
Subsystems (5)
cover mm: rmap: merge HugeTLB mapcount logic with THPs
HugeTLB pages may soon support being mapped with PTEs. To allow for this case, merge HugeTLB's mapcount scheme with THP's.
The first patch of this series comes from the HugeTLB high-granularity mapping series[1], though with some updates, as the original version was buggy[2] and incomplete.
I am sending this change as part of this smaller series in hopes that it can be more thoroughly scrutinized.
I haven't run any THP performance tests with this series applied. HugeTLB pages don't currently support being mapped with compound=false, but this mapcount scheme will make collapsing compound=false mappings in HugeTLB pages quite slow. This can be optimized with future patches (likely by taking advantage of HugeTLB's alignment guarantees).
Matthew Wilcox is working on a mapcounting scheme[3] that will avoid the use of each subpage's mapcount. If this series is applied, Matthew's new scheme will automatically apply to HugeTLB pages.
[1]: https://lore.kernel.org/linux-mm/[email protected]/
[2]: https://lore.kernel.org/linux-mm/CACw3F538H+bYcvSY-qG4-gmrgGPRBgTScDzrX9suLyp_q+v_bQ@mail.gmail.com/
[3]: https://lore.kernel.org/linux-mm/Y9Afwds%[email protected]/
James Houghton (2):
mm: rmap: make hugetlb pages participate in _nr_pages_mapped
mm: rmap: increase COMPOUND_MAPPED to support 512G HugeTLB pages
include/linux/mm.h | 7 +------ mm/hugetlb.c | 4 ++-- mm/internal.h | 9 ++++----- mm/migrate.c | 2 +- mm/rmap.c | 35 ++++++++++++++++++++--------------- 5 files changed, 28 insertions(+), 29 deletions(-)
base-commit: 9caa15b8a49949342bdf495bd47660267a3bd371
Signature · 1 line
2.40.0.rc0.216.gc4246ad0f0-goog
PATCH 1/2 mm: rmap: make hugetlb pages participate in _nr_pages_mapped
For compound mappings (compound=true), _nr_pages_mapped will now be incremented by COMPOUND_MAPPED when the first compound mapping is created.
For small mappings, _nr_pages_mapped is incremented by 1 when the particular small page is mapped for the first time. This is incompatible with HPageVmemmapOptimize()ed folios, as most of the tail page structs will be mapped read-only.
Currently HugeTLB always passes compound=true, but in the future, HugeTLB pages may be mapped with small mappings.
To implement this change:
1. Replace most of HugeTLB's calls to page_dup_file_rmap() with page_add_file_rmap(). The call in copy_hugetlb_page_range() is kept.
2. Update page_add_file_rmap() and page_remove_rmap() to support HugeTLB folios.
3. Update hugepage_add_anon_rmap() and hugepage_add_new_anon_rmap() to also increment _nr_pages_mapped properly.
With these changes, folio_large_is_mapped() no longer needs to check _entire_mapcount.
HugeTLB doesn't use LRU or mlock, so page_add_file_rmap() and page_remove_rmap() excludes those pieces. It is also important that the folio_test_pmd_mappable() check is removed (or changed), as it's possible to have a HugeTLB page whose order is not >= HPAGE_PMD_ORDER, like arm64's CONT_PTE_SIZE HugeTLB pages.
This patch limits HugeTLB pages to 16G in size. That limit can be increased if COMPOUND_MAPPED is raised.
include/linux/mm.h +1 -6
@@ -1143,12 +1143,7 @@ static inline int total_mapcount(struct page *page)static inline bool folio_large_is_mapped (struct folio *folio) { -/* -* Reading _entire_mapcount below could be omitted if hugetlb -* participated in incrementing nr_pages_mapped when compound mapped. -*/ -return atomic_read (&folio->_nr_pages_mapped) >0 || -atomic_read (&folio->_entire_mapcount) >=0 ; +return atomic_read (&folio->_nr_pages_mapped) >0 ; }/**
mm/hugetlb.c +2 -2
@@ -5911,7 +5911,7 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm,if (anon_rmap)hugepage_add_new_anon_rmap (folio, vma, haddr);else -page_dup_file_rmap (&folio->page, true); +page_add_file_rmap (&folio->page, vma, true); new_pte =make_huge_pte (vma, &folio->page, ((vma->vm_flags &VM_WRITE ) && (vma->vm_flags &VM_SHARED )));/*
@@ -6293,7 +6293,7 @@ int hugetlb_mcopy_atomic_pte(struct mm_struct *dst_mm, goto out_release_unlock;if (folio_in_pagecache) -page_dup_file_rmap (&folio->page, true); +page_add_file_rmap (&folio->page, dst_vma, true);else hugepage_add_new_anon_rmap (folio, dst_vma, dst_addr);
mm/internal.h +1 -2
@@ -55,8 +55,7 @@ void page_writeback_init(void);/* * If a 16GB hugetlb folio were mapped by PTEs of all of its 4kB pages, * its nr_pages_mapped would be 0x400000: choose the COMPOUND_MAPPED bit -* above that range, instead of 2*(PMD_SIZE/PAGE_SIZE). Hugetlb currently -* leaves nr_pages_mapped at 0, but avoid surprise if it participates later. +* above that range, instead of 2*(PMD_SIZE/PAGE_SIZE). */ #define COMPOUND_MAPPED 0x800000#define FOLIO_PAGES_MAPPED (COMPOUND_MAPPED - 1)
mm/migrate.c +1 -1
@@ -254,7 +254,7 @@ static bool remove_migration_pte(struct folio *folio,hugepage_add_anon_rmap (new, vma, pvmw.address, rmap_flags); else -page_dup_file_rmap (new ,true ); +page_add_file_rmap (new ,vma ,true );set_huge_pte_at (vma->vm_mm, pvmw.address, pvmw.pte, pte); } else#endif
mm/rmap.c +20 -15
@@ -1316,19 +1316,21 @@ void page_add_file_rmap(struct page *page, struct vm_area_struct *vma,int nr =0 , nr_pmdmapped =0 ;bool first; -VM_BUG_ON_PAGE (compound && !PageTransHuge (page), page); +VM_BUG_ON_PAGE (compound && !PageTransHuge (page) + && !folio_test_hugetlb (folio), page);/* Is page being mapped by PTE? Is this its first map to be added? */ if (likely (!compound)) { +if (unlikely (folio_test_hugetlb (folio))) +VM_BUG_ON_PAGE (HPageVmemmapOptimized (&folio->page), + page); first =atomic_inc_and_test (&page->_mapcount); nr = first;if (first &&folio_test_large (folio)) { nr =atomic_inc_return_relaxed (mapped); nr = (nr <COMPOUND_MAPPED ); } - }else if (folio_test_pmd_mappable (folio)) { -/* That test is redundant: it's for safety or to optimize out */ - + }else { first =atomic_inc_and_test (&folio->_entire_mapcount);if (first) { nr =atomic_add_return_relaxed (COMPOUND_MAPPED , mapped);
@@ -1345,6 +1347,9 @@ void page_add_file_rmap(struct page *page, struct vm_area_struct *vma, } } +if (folio_test_hugetlb (folio)) +return ; +if (nr_pmdmapped)__lruvec_stat_mod_folio (folio,folio_test_swapbacked (folio) ?NR_SHMEM_PMDMAPPED :NR_FILE_PMDMAPPED , nr_pmdmapped);
@@ -1373,24 +1378,18 @@ void page_remove_rmap(struct page *page, struct vm_area_struct *vma,VM_BUG_ON_PAGE (compound && !PageHead (page), page); -/* Hugetlb pages are not counted in NR_*MAPPED */ -if (unlikely (folio_test_hugetlb (folio))) { -/* hugetlb pages are always mapped with pmds */ -atomic_dec (&folio->_entire_mapcount); -return ; - } -/* Is page being unmapped by PTE? Is this its last map to be removed? */ if (likely (!compound)) { +if (unlikely (folio_test_hugetlb (folio))) +VM_BUG_ON_PAGE (HPageVmemmapOptimized (&folio->page), + page); last =atomic_add_negative (-1 , &page->_mapcount); nr = last;if (last &&folio_test_large (folio)) { nr =atomic_dec_return_relaxed (mapped); nr = (nr <COMPOUND_MAPPED ); } - }else if (folio_test_pmd_mappable (folio)) { -/* That test is redundant: it's for safety or to optimize out */ - + }else { last =atomic_add_negative (-1 , &folio->_entire_mapcount);if (last) { nr =atomic_sub_return_relaxed (COMPOUND_MAPPED , mapped);
@@ -1407,6 +1406,9 @@ void page_remove_rmap(struct page *page, struct vm_area_struct *vma, } } +if (folio_test_hugetlb (folio)) +return ; +if (nr_pmdmapped) {if (folio_test_anon (folio)) idx =NR_ANON_THPS ;
@@ -2541,9 +2543,11 @@ void hugepage_add_anon_rmap(struct page *page, struct vm_area_struct *vma, first =atomic_inc_and_test (&folio->_entire_mapcount);VM_BUG_ON_PAGE (!first && (flags &RMAP_EXCLUSIVE ), page);VM_BUG_ON_PAGE (!first &&PageAnonExclusive (page), page); -if (first) +if (first) { +atomic_add (COMPOUND_MAPPED , &folio->_nr_pages_mapped);__page_set_anon_rmap (folio, page, vma, address, !!(flags &RMAP_EXCLUSIVE )); + } } void hugepage_add_new_anon_rmap(struct folio *folio,
@@ -2552,6 +2556,7 @@ void hugepage_add_new_anon_rmap(struct folio *folio,BUG_ON (address < vma->vm_start || address >= vma->vm_end);/* increment count (starts at -1) */ atomic_set (&folio->_entire_mapcount,0 ); +atomic_set (&folio->_nr_pages_mapped,COMPOUND_MAPPED );folio_clear_hugetlb_restore_reserve (folio);__page_set_anon_rmap (folio, &folio->page, vma, address,1 ); }
Signature · 1 line
2.40.0.rc0.216.gc4246ad0f0-goog
Re: [PATCH 1/2] mm: rmap: make hugetlb pages participate in _nr_pages_mapped
Quoted 3 lines · James Houghton
For compound mappings (compound=true),
_nr_pages_mappedwill now be incremented byCOMPOUND_MAPPEDwhen the first compound mapping is created.
This sentence makes it sound like incrementing by COMPOUND_MAPPED for compound pages is introduced by this patch. Rather, it is just for
hugetlb (now always) compound mappings. Perhaps change that to read:
For hugetlb mappings ...
Quoted 25 lines
For small mappings,
_nr_pages_mappedis incremented by 1 when the particular small page is mapped for the first time. This is incompatible with HPageVmemmapOptimize()ed folios, as most of the tail page structs will be mapped read-only.Currently HugeTLB always passes compound=true, but in the future, HugeTLB pages may be mapped with small mappings.
To implement this change:
1. Replace most of HugeTLB's calls topage_dup_file_rmap()withpage_add_file_rmap(). The call incopy_hugetlb_page_range()is kept.
2. Updatepage_add_file_rmap()andpage_remove_rmap()to support HugeTLB folios.
3. Updatehugepage_add_anon_rmap()andhugepage_add_new_anon_rmap()to also increment_nr_pages_mappedproperly.With these changes,
folio_large_is_mapped()no longer needs to check_entire_mapcount.HugeTLB doesn't use LRU or mlock, so
page_add_file_rmap()andpage_remove_rmap()excludes those pieces. It is also important that thefolio_test_pmd_mappable()check is removed (or changed), as it's possible to have a HugeTLB page whose order is not >=HPAGE_PMD_ORDER, like arm64'sCONT_PTE_SIZEHugeTLB pages.
Quoted 5 lines
This patch limits HugeTLB pages to 16G in size. That limit can be increased if
COMPOUND_MAPPEDis raised.Signed-off-by James Houghton <[email protected]>
Thanks!
This is a step in the direction of having hugetlb use the same mapcount scheme as elsewhere. As you mention, with this in place future mapcount changes should mostly 'just work' for hugetlb.
Because of this,
I have a few nits below, and I'm sure others will chime in later.
Quoted 15 lines
mm/rmap.c +3
-1@@ -1316,19 +1316,21 @@ void page_add_file_rmap(struct page *page, struct vm_area_struct *vma,int nr =0 , nr_pmdmapped =0 ;bool first; -VM_BUG_ON_PAGE (compound && !PageTransHuge (page), page); +VM_BUG_ON_PAGE (compound && !PageTransHuge (page) + && !folio_test_hugetlb (folio), page);/* Is page being mapped by PTE? Is this its first map to be added? */ if (likely (!compound)) { +if (unlikely (folio_test_hugetlb (folio)))
mm/rmap.c void page_add_file_rmap(struct page *page, struct vm_area_struct *vma, · line 1330
Quoted 10 lines
mm/rmap.c +2
-2+VM_BUG_ON_PAGE (HPageVmemmapOptimized (&folio->page), + page); first =atomic_inc_and_test (&page->_mapcount); nr = first;if (first &&folio_test_large (folio)) { nr =atomic_inc_return_relaxed (mapped); nr = (nr <COMPOUND_MAPPED ); } - }else if (folio_test_pmd_mappable (folio)) { -/* That test is redundant: it's for safety or to optimize out */
I 'think' removing this check is OK. It would seem that the caller knows if the folio is mappable. If we want a similar test, we might be able to use something like:
arch_hugetlb_valid_size(folio_size(folio))
mm/rmap.c void page_add_file_rmap(struct page *page, struct vm_area_struct *vma, · line 1351
Quoted 11 lines
mm/rmap.c +3
-1- + }else { first =atomic_inc_and_test (&folio->_entire_mapcount);if (first) { nr =atomic_add_return_relaxed (COMPOUND_MAPPED , mapped);@@ -1345,6 +1347,9 @@ void page_add_file_rmap(struct page *page, struct vm_area_struct *vma, } } +if (folio_test_hugetlb (folio)) +return ;
IMO, a comment saying hugetlb is special and does not participate in lru would be appropriate here.
Quoted 34 lines
mm/rmap.c +5
-10+if (nr_pmdmapped)__lruvec_stat_mod_folio (folio,folio_test_swapbacked (folio) ?NR_SHMEM_PMDMAPPED :NR_FILE_PMDMAPPED , nr_pmdmapped);@@ -1373,24 +1378,18 @@ void page_remove_rmap(struct page *page, struct vm_area_struct *vma,VM_BUG_ON_PAGE (compound && !PageHead (page), page); -/* Hugetlb pages are not counted in NR_*MAPPED */ -if (unlikely (folio_test_hugetlb (folio))) { -/* hugetlb pages are always mapped with pmds */ -atomic_dec (&folio->_entire_mapcount); -return ; - } -/* Is page being unmapped by PTE? Is this its last map to be removed? */ if (likely (!compound)) { +if (unlikely (folio_test_hugetlb (folio))) +VM_BUG_ON_PAGE (HPageVmemmapOptimized (&folio->page), + page); last =atomic_add_negative (-1 , &page->_mapcount); nr = last;if (last &&folio_test_large (folio)) { nr =atomic_dec_return_relaxed (mapped); nr = (nr <COMPOUND_MAPPED ); } - }else if (folio_test_pmd_mappable (folio)) { -/* That test is redundant: it's for safety or to optimize out */ - + }else { last =atomic_add_negative (-1 , &folio->_entire_mapcount);if (last) { nr =atomic_sub_return_relaxed (COMPOUND_MAPPED , mapped);
mm/rmap.c void page_remove_rmap(struct page *page, struct vm_area_struct *vma, · line 1410
Quoted 5 lines
mm/rmap.c +2
-0@@ -1407,6 +1406,9 @@ void page_remove_rmap(struct page *page, struct vm_area_struct *vma, } } +if (folio_test_hugetlb (folio)) +return ;
Same as above in page_add_file_rmap.
Quoted 18 lines
mm/rmap.c +4
-1+if (nr_pmdmapped) {if (folio_test_anon (folio)) idx =NR_ANON_THPS ;@@ -2541,9 +2543,11 @@ void hugepage_add_anon_rmap(struct page *page, struct vm_area_struct *vma, first =atomic_inc_and_test (&folio->_entire_mapcount);VM_BUG_ON_PAGE (!first && (flags &RMAP_EXCLUSIVE ), page);VM_BUG_ON_PAGE (!first &&PageAnonExclusive (page), page); -if (first) +if (first) { +atomic_add (COMPOUND_MAPPED , &folio->_nr_pages_mapped);__page_set_anon_rmap (folio, page, vma, address, !!(flags &RMAP_EXCLUSIVE )); + } } void hugepage_add_new_anon_rmap(struct folio *folio,
mm/rmap.c void hugepage_add_new_anon_rmap(struct folio *folio, · line 2562
Quoted 7 lines
mm/rmap.c +1
-0@@ -2552,6 +2556,7 @@ void hugepage_add_new_anon_rmap(struct folio *folio,BUG_ON (address < vma->vm_start || address >= vma->vm_end);/* increment count (starts at -1) */ atomic_set (&folio->_entire_mapcount,0 ); +atomic_set (&folio->_nr_pages_mapped,COMPOUND_MAPPED );folio_clear_hugetlb_restore_reserve (folio);__page_set_anon_rmap (folio, &folio->page, vma, address,1 ); }
Should we look at perhaps modifying page_add_anon_rmap and folio_add_new_anon_rmap as well?
Signature · 1 line
Mike Kravetz
Re: [PATCH 1/2] mm: rmap: make hugetlb pages participate in _nr_pages_mapped
Quoted 10 lines · Mike Kravetz
Quoted 3 lines · James Houghton
For compound mappings (compound=true),
_nr_pages_mappedwill now be incremented byCOMPOUND_MAPPEDwhen the first compound mapping is created.This sentence makes it sound like incrementing by
COMPOUND_MAPPEDfor compound pages is introduced by this patch. Rather, it is just for
hugetlb (now always) compound mappings. Perhaps change that to read:
For hugetlb mappings ...
Yes this is kind of confusing. I'll fix it like you suggest.
Quoted 32 lines
Quoted 26 lines
For small mappings,
_nr_pages_mappedis incremented by 1 when the particular small page is mapped for the first time. This is incompatible with HPageVmemmapOptimize()ed folios, as most of the tail page structs will be mapped read-only.Currently HugeTLB always passes compound=true, but in the future, HugeTLB pages may be mapped with small mappings.
To implement this change:
1. Replace most of HugeTLB's calls topage_dup_file_rmap()withpage_add_file_rmap(). The call incopy_hugetlb_page_range()is kept.
2. Updatepage_add_file_rmap()andpage_remove_rmap()to support HugeTLB folios.
3. Updatehugepage_add_anon_rmap()andhugepage_add_new_anon_rmap()to also increment_nr_pages_mappedproperly.With these changes,
folio_large_is_mapped()no longer needs to check_entire_mapcount.HugeTLB doesn't use LRU or mlock, so
page_add_file_rmap()andpage_remove_rmap()excludes those pieces. It is also important that thefolio_test_pmd_mappable()check is removed (or changed), as it's possible to have a HugeTLB page whose order is not >=HPAGE_PMD_ORDER, like arm64'sCONT_PTE_SIZEHugeTLB pages.Quoted 5 lines
This patch limits HugeTLB pages to 16G in size. That limit can be increased if
COMPOUND_MAPPEDis raised.Signed-off-by James Houghton <[email protected]>
Quoted 8 lines
Thanks!
This is a step in the direction of having hugetlb use the same mapcount scheme as elsewhere. As you mention, with this in place future mapcount changes should mostly 'just work' for hugetlb.
Because of this,
Acked-by Mike Kravetz <[email protected]>
Thanks!
Quoted 33 lines
I have a few nits below, and I'm sure others will chime in later.
Quoted 15 lines
mm/rmap.c +3
-1@@ -1316,19 +1316,21 @@ void page_add_file_rmap(struct page *page, struct vm_area_struct *vma,int nr =0 , nr_pmdmapped =0 ;bool first; -VM_BUG_ON_PAGE (compound && !PageTransHuge (page), page); +VM_BUG_ON_PAGE (compound && !PageTransHuge (page) + && !folio_test_hugetlb (folio), page);/* Is page being mapped by PTE? Is this its first map to be added? */ if (likely (!compound)) { +if (unlikely (folio_test_hugetlb (folio)))mm/rmap.c void page_add_file_rmap(struct page *page, struct vm_area_struct *vma, · line 1330
Quoted 10 lines
mm/rmap.c +2
-2+VM_BUG_ON_PAGE (HPageVmemmapOptimized (&folio->page), + page); first =atomic_inc_and_test (&page->_mapcount); nr = first;if (first &&folio_test_large (folio)) { nr =atomic_inc_return_relaxed (mapped); nr = (nr <COMPOUND_MAPPED ); } - }else if (folio_test_pmd_mappable (folio)) { -/* That test is redundant: it's for safety or to optimize out */ I 'think' removing this check is OK. It would seem that the caller knows if the folio is mappable. If we want a similar test, we might be able to use something like:
Quoted 2 lines
arch_hugetlb_valid_size(folio_size(folio))
Ack. I think leaving the check(s) removed is fine.
Quoted 9 lines
Quoted 9 lines
mm/rmap.c +1
-1- + }else { first =atomic_inc_and_test (&folio->_entire_mapcount);if (first) { nr =atomic_add_return_relaxed (COMPOUND_MAPPED , mapped);@@ -1345,6 +1347,9 @@ void page_add_file_rmap(struct page *page, struct vm_area_struct *vma, } }
Quoted 5 lines
mm/rmap.c void page_add_file_rmap(struct page *page, struct vm_area_struct *vma, · line 1351
Quoted 2 lines
mm/rmap.c +2
-0+if (folio_test_hugetlb (folio)) +return ;IMO, a comment saying hugetlb is special and does not participate in lru would be appropriate here.
Will do.
Quoted 59 lines
Quoted 35 lines
mm/rmap.c +5
-10+if (nr_pmdmapped)__lruvec_stat_mod_folio (folio,folio_test_swapbacked (folio) ?NR_SHMEM_PMDMAPPED :NR_FILE_PMDMAPPED , nr_pmdmapped);@@ -1373,24 +1378,18 @@ void page_remove_rmap(struct page *page, struct vm_area_struct *vma,VM_BUG_ON_PAGE (compound && !PageHead (page), page); -/* Hugetlb pages are not counted in NR_*MAPPED */ -if (unlikely (folio_test_hugetlb (folio))) { -/* hugetlb pages are always mapped with pmds */ -atomic_dec (&folio->_entire_mapcount); -return ; - } -/* Is page being unmapped by PTE? Is this its last map to be removed? */ if (likely (!compound)) { +if (unlikely (folio_test_hugetlb (folio))) +VM_BUG_ON_PAGE (HPageVmemmapOptimized (&folio->page), + page); last =atomic_add_negative (-1 , &page->_mapcount); nr = last;if (last &&folio_test_large (folio)) { nr =atomic_dec_return_relaxed (mapped); nr = (nr <COMPOUND_MAPPED ); } - }else if (folio_test_pmd_mappable (folio)) { -/* That test is redundant: it's for safety or to optimize out */ - + }else { last =atomic_add_negative (-1 , &folio->_entire_mapcount);if (last) { nr =atomic_sub_return_relaxed (COMPOUND_MAPPED , mapped);mm/rmap.c void page_remove_rmap(struct page *page, struct vm_area_struct *vma, · line 1410
Quoted 5 lines
mm/rmap.c +2
-0@@ -1407,6 +1406,9 @@ void page_remove_rmap(struct page *page, struct vm_area_struct *vma, } } +if (folio_test_hugetlb (folio)) +return ;Same as above in
page_add_file_rmap.Quoted 16 lines
mm/rmap.c +4
-1+if (nr_pmdmapped) {if (folio_test_anon (folio)) idx =NR_ANON_THPS ;@@ -2541,9 +2543,11 @@ void hugepage_add_anon_rmap(struct page *page, struct vm_area_struct *vma, first =atomic_inc_and_test (&folio->_entire_mapcount);VM_BUG_ON_PAGE (!first && (flags &RMAP_EXCLUSIVE ), page);VM_BUG_ON_PAGE (!first &&PageAnonExclusive (page), page); -if (first) +if (first) { +atomic_add (COMPOUND_MAPPED , &folio->_nr_pages_mapped);__page_set_anon_rmap (folio, page, vma, address, !!(flags &RMAP_EXCLUSIVE )); + } }
Quoted 12 lines
mm/rmap.c void hugepage_add_new_anon_rmap(struct folio *folio, · line 2562
Quoted 9 lines
mm/rmap.c +1
-0void hugepage_add_new_anon_rmap(struct folio *folio,@@ -2552,6 +2556,7 @@ void hugepage_add_new_anon_rmap(struct folio *folio,BUG_ON (address < vma->vm_start || address >= vma->vm_end);/* increment count (starts at -1) */ atomic_set (&folio->_entire_mapcount,0 ); +atomic_set (&folio->_nr_pages_mapped,COMPOUND_MAPPED );folio_clear_hugetlb_restore_reserve (folio);__page_set_anon_rmap (folio, &folio->page, vma, address,1 ); }Should we look at perhaps modifying
page_add_anon_rmapandfolio_add_new_anon_rmapas well?
I think I can merge hugepage_add_anon_rmap with page_add_anon_rmap and hugepage_add_new_anon_rmap with folio_add_new_anon_rmap. With them merged, it's pretty easy to see what HugeTLB does differently from generic mm, which is nice. :)
Re: [PATCH 1/2] mm: rmap: make hugetlb pages participate in _nr_pages_mapped
Quoted 4 lines · James Houghton
Quoted 4 lines
mm/rmap.c void page_add_file_rmap(struct page *page, struct vm_area_struct *vma, · line 1326
Quoted 4 lines
if (likely(!compound)) {mm/rmap.c +3
-0+if (unlikely (folio_test_hugetlb (folio))) +VM_BUG_ON_PAGE (HPageVmemmapOptimized (&folio->page), + page);
How about moving folio_test_hugetlb() into the BUG_ON()?
VM_BUG_ON_PAGE(folio_test_hugetlb(folio) &&
HPageVmemmapOptimized(&folio->page),
page);Note that BUG_ON() already contains an "unlikely".
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
first = atomic_inc_and_test(&page->_mapcount); nr = first; if (first && folio_test_large(folio)) { nr = atomic_inc_return_relaxed(mapped); nr = (nr < COMPOUND_MAPPED); }(quoted diff) +0
-2- } else if (folio_test_pmd_mappable(folio)) { - /* That test is redundant: it's for safety or to optimize out */
Quoted 8 lines
Quoted 6 lines
I 'think' removing this check is OK. It would seem that the caller knows if the folio is mappable. If we want a similar test, we might be able to use something like:
arch_hugetlb_valid_size(folio_size(folio))
Ack. I think leaving the check(s) removed is fine.
Would it still be good to keep that as another BUG_ON()?
Signature · 1 line
Peter Xu
Re: [PATCH 1/2] mm: rmap: make hugetlb pages participate in _nr_pages_mapped
Quoted 7 lines · Peter Xu
Quoted 4 lines · James Houghton
Quoted 4 lines
mm/rmap.c void page_add_file_rmap(struct page *page, struct vm_area_struct *vma, · line 1326
Quoted 4 lines
if (likely(!compound)) {mm/rmap.c +3
-0+if (unlikely (folio_test_hugetlb (folio))) +VM_BUG_ON_PAGE (HPageVmemmapOptimized (&folio->page), + page);
Quoted 7 lines
How about moving
folio_test_hugetlb()into theBUG_ON()?VM_BUG_ON_PAGE(folio_test_hugetlb(folio) && HPageVmemmapOptimized(&folio->page), page);Note that
BUG_ON()already contains an "unlikely".
Ok I can do that. It's a little cleaner.
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Review
Quoted 8 lines
first = atomic_inc_and_test(&page->_mapcount); nr = first; if (first && folio_test_large(folio)) { nr = atomic_inc_return_relaxed(mapped); nr = (nr < COMPOUND_MAPPED); }(quoted diff) +0
-2- } else if (folio_test_pmd_mappable(folio)) { - /* That test is redundant: it's for safety or to optimize out */I 'think' removing this check is OK. It would seem that the caller knows if the folio is mappable. If we want a similar test, we might be able to use something like:
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
arch_hugetlb_valid_size(folio_size(folio))
Ack. I think leaving the check(s) removed is fine.
Would it still be good to keep that as another
BUG_ON()?
Sure, that sounds reasonable to me. I'll add it unless someone disagrees.
As you suggested in your other email, I'll also add a BUG_ON() if we attempt to do a non-compound mapping of a folio that is larger than COMPOUND_MAPPED / 2. (Maybe a BUG_ON() in alloc_hugetlb_folio() to check that the size of the folio we're allocating is less than COMPOUND_MAPPED / 2 makes sense instead. Just an idea.)
- James
PATCH 2/2 mm: rmap: increase COMPOUND_MAPPED to support 512G HugeTLB pages
In reply to James HoughtonSome architectures may want to support 512G HugeTLB pages in the future. Adjust COMPOUND_MAPPED to support this page size.
mm/internal.h +4 -4
@@ -53,11 +53,11 @@ struct folio_batch;void page_writeback_init (void );/* -* If a 16GB hugetlb folio were mapped by PTEs of all of its 4kB pages, -* its nr_pages_mapped would be 0x400000: choose the COMPOUND_MAPPED bit -* above that range, instead of 2*(PMD_SIZE/PAGE_SIZE). +* If a 512G hugetlb folio were mapped by PTEs of all of its 4kB pages, +* its nr_pages_mapped would be 2^27 (0x8000000): choose the COMPOUND_MAPPED bit +* above that range. */ -#define COMPOUND_MAPPED 0x800000 +#define COMPOUND_MAPPED (1 << 28)#define FOLIO_PAGES_MAPPED (COMPOUND_MAPPED - 1)/*
Signature · 1 line
2.40.0.rc0.216.gc4246ad0f0-goog
Quoted 17 lines · James Houghton
HugeTLB pages may soon support being mapped with PTEs. To allow for this case, merge HugeTLB's mapcount scheme with THP's.
The first patch of this series comes from the HugeTLB high-granularity mapping series[1], though with some updates, as the original version was buggy[2] and incomplete.
I am sending this change as part of this smaller series in hopes that it can be more thoroughly scrutinized.
I haven't run any THP performance tests with this series applied. HugeTLB pages don't currently support being mapped with
compound=false, but this mapcount scheme will make collapsing compound=false mappings in HugeTLB pages quite slow. This can be optimized with future patches (likely by taking advantage of HugeTLB's alignment guarantees).
Quoted 3 lines
Matthew Wilcox is working on a mapcounting scheme[3] that will avoid the use of each subpage's mapcount. If this series is applied, Matthew's new scheme will automatically apply to HugeTLB pages.
Is this the plan?
I may have not followed closely on the latest development of Matthew's idea. The thing is if the design requires ptes being installed / removed at the same time for the whole folio, then it may not work directly for HGM if HGM wants to support at least postcopy, iiuc, because if we install the whole folio ptes at the same time it seems to beat the whole purpose of having HGM..
The patch (especially patch 1) looks good. So it's a pure question just to make sure we're on the same page. IIUC your other mapcount proposal may work, but it still needs to be able to take care of ptes in less-than-folio sizes whatever it'll look like at last.
A trivial comment on patch 2 since we're at it: does "a future plan on some arch to support 512GB huge page" justify itself? It would be better justified, IMHO, when that support is added (and decided to use HGM)?
What I feel like is missing (rather than patch 2 itself) is some guard to make sure thp mapcountings will not be abused with new hugetlb sizes coming.
How about another BUG_ON() squashed into patch 1 (probably somewhere in page_add_file|anon_rmap()) to make sure folio_size() is always smaller than COMPOUND_MAPPED / 2)?
Signature · 1 line
Peter Xu
Quoted 23 lines · Peter Xu
Quoted 17 lines · James Houghton
HugeTLB pages may soon support being mapped with PTEs. To allow for this case, merge HugeTLB's mapcount scheme with THP's.
The first patch of this series comes from the HugeTLB high-granularity mapping series[1], though with some updates, as the original version was buggy[2] and incomplete.
I am sending this change as part of this smaller series in hopes that it can be more thoroughly scrutinized.
I haven't run any THP performance tests with this series applied. HugeTLB pages don't currently support being mapped with
compound=false, but this mapcount scheme will make collapsing compound=false mappings in HugeTLB pages quite slow. This can be optimized with future patches (likely by taking advantage of HugeTLB's alignment guarantees).Quoted 3 lines
Matthew Wilcox is working on a mapcounting scheme[3] that will avoid the use of each subpage's mapcount. If this series is applied, Matthew's new scheme will automatically apply to HugeTLB pages.
Quoted 8 lines
Is this the plan?
I may have not followed closely on the latest development of Matthew's idea. The thing is if the design requires ptes being installed / removed at the same time for the whole folio, then it may not work directly for HGM if HGM wants to support at least postcopy, iiuc, because if we install the whole folio ptes at the same time it seems to beat the whole purpose of having HGM..
My understanding is that it doesn't *require* all the PTEs in a folio to be mapped at the same time. I don't see how it possibly could, given that UFFDIO_CONTINUE exists (which can already create PTE-mapped THPs today). It would be faster to populate all the PTEs at the same time (you would only need to traverse the page table once for the entire group to see if you should be incrementing mapcount).
Though, with respect to unmapping, if PTEs aren't all unmapped at the same time, then you could end up with a case where mapcount is still incremented but nothing is really mapped. I'm not really sure what should be done there, but this problem applies to PTE-mapped THPs the same way that it applies to HGMed HugeTLB pages.
Quoted 4 lines
The patch (especially patch 1) looks good. So it's a pure question just to make sure we're on the same page. IIUC your other mapcount proposal may work, but it still needs to be able to take care of ptes in less-than-folio sizes whatever it'll look like at last.
By my "other mapcount proposal", I assume you mean the "using the PAGE_SPECIAL bit to track if mapcount has been incremented or not". It really only serves as an optimization for Matthew's scheme (see below
[2] for some more thoughts), and it doesn't have to only apply to HugeTLB.
I originally thought[1] that Matthew's scheme would be really painful for postcopy for HGM without this optimization, but it's actually not so bad. Let's assume the worst case, that we're UFFDIO_CONTINUEing from the end to the beginning, like in [1]:
First CONTINUE: pvmw finds an empty PUD, so quickly returns false. Second CONTINUE: pvmw finds 511 empty PMDs, then finds 511 empty PTEs, then finds a present PTE (from the first CONTINUE).
Third CONTINUE: pvmw finds 511 empty PMDs, then finds 510 empty PTEs.
...
514th CONTINUE: pvmw finds 510 empty PMDs, then finds 511 empty PTEs.
So it'll be slow, but it won't have to check 262k empty PTEs per CONTINUE (though you could make this possible with MADV_DONTNEED). Even with an HGM implementation that only allows PTE-mapping of HugeTLB pages, it should still behave just like this, too.
Quoted 3 lines
A trivial comment on patch 2 since we're at it: does "a future plan on some arch to support 512GB huge page" justify itself? It would be better justified, IMHO, when that support is added (and decided to use HGM)?
That's fine with me. I'm happy to drop that patch.
Quoted 7 lines
What I feel like is missing (rather than patch 2 itself) is some guard to make sure thp mapcountings will not be abused with new hugetlb sizes coming.
How about another
BUG_ON()squashed into patch 1 (probably somewhere in page_add_file|anon_rmap()) to make surefolio_size()is always smaller thanCOMPOUND_MAPPED/ 2)?
Sure, I can add that.
Thanks, Peter!
- James
[1]: https://lore.kernel.org/linux-mm/CADrL8HUrEgt+1qAtEsOHuQeA+WWnggGfLj8_nqHF0k-pqPi52w@mail.gmail.com/
[2]: Some details on what the optimization might look like:
So an excerpt of Matthew's scheme would look something like this:
/* if we're mapping < folio_nr_pages(folio) worth of PTEs. */
if (!folio_has_ptes(folio, vma))
atomic_inc(folio->_mapcount);where folio_has_ptes() is defined like:
if (!page_vma_mapped_walk(...))
return false;
page_vma_mapped_walk_done(...);
return true;You might be able to optimize folio_has_ptes() with a block like this at the beginning:
if (folio_is_naturally_aligned(folio, vma)) {
/* optimization for naturally-aligned folios. */
if (folio_test_hugetlb(folio)) {
/* check hstate-level PTE, and do a similar check as below. */
}
/* for naturally-aligned THPs: */
pmdp = mm_find_pmd(...); /* or just pass it in. */
pmd = READ_ONCE(*pmdp);
BUG_ON(!pmd_present(pmd) || pmd_leaf(pmd));
if (pmd_special(pmd))
return true;
/* we already hold the PTL for the PTE. */
ptl = pmd_lock(mm, pmdp);
/* test and set pmd_special */
pmd_unlock(ptl)
return if_we_set_pmd_special;
}(pmd_special() doesn't currently exist.) If HugeTLB walking code can be merged with generic mm, then HugeTLB wouldn't have a special case at all here.
Quoted 132 lines · James Houghton
Quoted 23 lines · Peter Xu
Quoted 17 lines · James Houghton
HugeTLB pages may soon support being mapped with PTEs. To allow for this case, merge HugeTLB's mapcount scheme with THP's.
The first patch of this series comes from the HugeTLB high-granularity mapping series[1], though with some updates, as the original version was buggy[2] and incomplete.
I am sending this change as part of this smaller series in hopes that it can be more thoroughly scrutinized.
I haven't run any THP performance tests with this series applied. HugeTLB pages don't currently support being mapped with
compound=false, but this mapcount scheme will make collapsing compound=false mappings in HugeTLB pages quite slow. This can be optimized with future patches (likely by taking advantage of HugeTLB's alignment guarantees).Quoted 3 lines
Matthew Wilcox is working on a mapcounting scheme[3] that will avoid the use of each subpage's mapcount. If this series is applied, Matthew's new scheme will automatically apply to HugeTLB pages.
Quoted 8 lines
Is this the plan?
I may have not followed closely on the latest development of Matthew's idea. The thing is if the design requires ptes being installed / removed at the same time for the whole folio, then it may not work directly for HGM if HGM wants to support at least postcopy, iiuc, because if we install the whole folio ptes at the same time it seems to beat the whole purpose of having HGM..
My understanding is that it doesn't *require* all the PTEs in a folio to be mapped at the same time. I don't see how it possibly could, given that
UFFDIO_CONTINUEexists (which can already create PTE-mapped THPs today). It would be faster to populate all the PTEs at the same time (you would only need to traverse the page table once for the entire group to see if you should be incrementing mapcount).Though, with respect to unmapping, if PTEs aren't all unmapped at the same time, then you could end up with a case where mapcount is still incremented but nothing is really mapped. I'm not really sure what should be done there, but this problem applies to PTE-mapped THPs the same way that it applies to HGMed HugeTLB pages.
Quoted 4 lines
The patch (especially patch 1) looks good. So it's a pure question just to make sure we're on the same page. IIUC your other mapcount proposal may work, but it still needs to be able to take care of ptes in less-than-folio sizes whatever it'll look like at last.
By my "other mapcount proposal", I assume you mean the "using the
PAGE_SPECIALbit to track if mapcount has been incremented or not". It really only serves as an optimization for Matthew's scheme (see below
[2] for some more thoughts), and it doesn't have to only apply to HugeTLB.I originally thought[1] that Matthew's scheme would be really painful for postcopy for HGM without this optimization, but it's actually not so bad. Let's assume the worst case, that we're
UFFDIO_CONTINUEingfrom the end to the beginning, like in [1]:First CONTINUE: pvmw finds an empty PUD, so quickly returns false. Second CONTINUE: pvmw finds 511 empty PMDs, then finds 511 empty PTEs, then finds a present PTE (from the first CONTINUE).
Third CONTINUE: pvmw finds 511 empty PMDs, then finds 510 empty PTEs.
...
514th CONTINUE: pvmw finds 510 empty PMDs, then finds 511 empty PTEs.So it'll be slow, but it won't have to check 262k empty PTEs per CONTINUE (though you could make this possible with
MADV_DONTNEED). Even with an HGM implementation that only allows PTE-mapping of HugeTLB pages, it should still behave just like this, too.Quoted 3 lines
A trivial comment on patch 2 since we're at it: does "a future plan on some arch to support 512GB huge page" justify itself? It would be better justified, IMHO, when that support is added (and decided to use HGM)?
That's fine with me. I'm happy to drop that patch.
Quoted 7 lines
What I feel like is missing (rather than patch 2 itself) is some guard to make sure thp mapcountings will not be abused with new hugetlb sizes coming.
How about another
BUG_ON()squashed into patch 1 (probably somewhere in page_add_file|anon_rmap()) to make surefolio_size()is always smaller thanCOMPOUND_MAPPED/ 2)?Sure, I can add that.
Thanks, Peter!
- James
[1]: https://lore.kernel.org/linux-mm/CADrL8HUrEgt+1qAtEsOHuQeA+WWnggGfLj8_nqHF0k-pqPi52w@mail.gmail.com/
[2]: Some details on what the optimization might look like:
So an excerpt of Matthew's scheme would look something like this:
/* if we're mapping < folio_nr_pages(folio) worth of PTEs. */ if (!folio_has_ptes(folio, vma)) atomic_inc(folio->_mapcount);where
folio_has_ptes()is defined like:if (!page_vma_mapped_walk(...)) return false; page_vma_mapped_walk_done(...); return true;You might be able to optimize
folio_has_ptes()with a block like this at the beginning:if (folio_is_naturally_aligned(folio, vma)) { /* optimization for naturally-aligned folios. */ if (folio_test_hugetlb(folio)) { /* check hstate-level PTE, and do a similar check as below. */ } /* for naturally-aligned THPs: */ pmdp = mm_find_pmd(...); /* or just pass it in. */ pmd = READ_ONCE(*pmdp); BUG_ON(!pmd_present(pmd) || pmd_leaf(pmd)); if (pmd_special(pmd)) return true; /* we already hold the PTL for the PTE. */ ptl = pmd_lock(mm, pmdp); /* test and set pmd_special */ pmd_unlock(ptl) return if_we_set_pmd_special; }
Quoted 3 lines
(
pmd_special()doesn't currently exist.) If HugeTLB walking code can be merged with generic mm, then HugeTLB wouldn't have a special case at all here.
I see what you mean now, thanks. That looks fine. I just suspect the pte_special trick will still be needed if this will start to apply to HGM, as it seems to not suite perfectly with a large folio size, still. The MADV_DONTNEED worst case of having it loop over ~folio_size() times of none pte is still possible.
Signature · 1 line
Peter Xu