convert hugetlb fault functions to folios
Subsystems (8)
v2 cover convert hugetlb fault functions to folios
============== CHANGE LOG =========================== v1 -> v2
- Add r-b to patches 1,3
- Refactor hugetlbfs_pagecache_present() to use rcu + page_cache_next_miss() per Matthew in patch 2.
- Change function name from putback_active_hugetlb_folio() to folio_putback_active_hugetlb() per Matthew in patch 3.
- Squashed v1 patches 4-5 into one patch to make for easier review as did not need to handle null page case from alloc_huge_page().
Refactor hugetlbfs_fallocate() to use same rcu + page_cache_next_miss() that Matthew suggested for hugetlbfs_pagecache_present() per Mike in patch 4.
Initialize h_cg variable to NULL in alloc_hugetlb_folio() to suppress compiler warning when CONFIG_CGROUP_HUGETLB is not set per [email protected] in patch 4.
Changed variable page_in_pagecache to folio_in_pagecache in hugetlb_mcopy_atomic_pte() as the variable is set to true once the folio variable is added to the page cache in patch 4.
- Fixed wrong change from get_page(page) -> folio_get(pagecache_folio) reported by Gerald Schaefer in patch 7.
============== OVERVIEW ===========================
This series converts the hugetlb page faulting functions to operate on folios. These include hugetlb_no_page(), hugetlb_wp(), copy_hugetlb_page_range(), and hugetlb_mcopy_atomic_pte().
patch 1-3:
- convert prerequisite helper functions to folios.
patch 4:
- add a folio variable to the hugetlb fault functions to complete conversion to folios.
patch 5-7:
- convert three functions to take in a folio rather than a page as all callers now use folios.
patch 8:
-update documentation that references alloc_huge_page
============== TEST COVERAGE ============================
Linux Test Project Hugetlb Test Fault Coverage
[opc@sidhakum-devel kernel (master)]$ sudo ./funccount 'hugetlb_no_page'
FUNC COUNThugetlb_no_page 7796
[opc@sidhakum-devel kernel (master)]$ sudo ./funccount 'hugetlb_wp'
FUNC COUNThugetlb_wp 4623
Using fallocate commands to create files on hugeltbfs
[opc@sidhakum-devel kernel (master)]$ sudo ./funccount 'hugetlbfs*'
FUNC COUNThugetlbfs_fallocate 1
Userfaultfd selftest
./userfaultfd hugetlb 256 50 /dev/hugepages/hugefile
[opc@sidhakum-devel kernel (master)]$ sudo ./funccount 'hugetlb_mcopy_atomic_pte'
FUNC COUNThugetlb_mcopy_atomic_pte 6240
[opc@sidhakum-devel kernel (master)]$ sudo ./funccount 'copy_hugetlb_page_range'
FUNC COUNTcopy_hugetlb_page_range 3
============== PERFORMANCE ============================ using bpftrace to track time spent in fault functions over 10 rounds of the LTP hugetlb tests
pre-patch:
@hugetlb_wp_nsecs:
[256, 512) 3675 |@@@@@@@@@@ | [512, 1K) 18875 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| [1K, 2K) 1366 |@@@ | [2K, 4K) 77 | | [4K, 8K) 12 | | [8K, 16K) 10 | | [16K, 32K) 2 | | [32K, 64K) 0 | | [64K, 128K) 0 | | [128K, 256K) 0 | | [256K, 512K) 0 | | [512K, 1M) 0 | | [1M, 2M) 0 | | [2M, 4M) 0 | | [4M, 8M) 0 | | [8M, 16M) 0 | | [16M, 32M) 0 | | [32M, 64M) 0 | | [64M, 128M) 0 | | [128M, 256M) 0 | | [256M, 512M) 0 | | [512M, 1G) 0 | | [1G, 2G) 0 | | [2G, 4G) 1 | |
@hugetlb_no_page_nsecs:
[64, 128) 1 | | [128, 256) 0 | | [256, 512) 67 | | [512, 1K) 66 | | [1K, 2K) 65 | | [2K, 4K) 198 | | [4K, 8K) 97 | | [8K, 16K) 3 | | [16K, 32K) 4 | | [32K, 64K) 678 | | [64K, 128K) 3401 |@ | [128K, 256K) 96746 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| [256K, 512K) 107 | | [512K, 1M) 0 | | [1M, 2M) 1 | | [2M, 4M) 0 | | [4M, 8M) 1 | | [8M, 16M) 0 | | [16M, 32M) 0 | | [32M, 64M) 0 | | [64M, 128M) 0 | | [128M, 256M) 0 | | [256M, 512M) 0 | | [512M, 1G) 0 | | [1G, 2G) 0 | | [2G, 4G) 283 | |
post-patch:
@hugetlb_wp_nsecs:
[256, 512) 22282 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| [512, 1K) 16242 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ | [1K, 2K) 1579 |@@@ | [2K, 4K) 76 | | [4K, 8K) 18 | | [8K, 16K) 30 | | [16K, 32K) 4 | |
@hugetlb_no_page_nsecs:
[256, 512) 92 | | [512, 1K) 46 | | [1K, 2K) 83 | | [2K, 4K) 188 | | [4K, 8K) 80 | | [8K, 16K) 13 | | [16K, 32K) 8 | | [32K, 64K) 2332 |@ | [64K, 128K) 7992 |@@@ | [128K, 256K) 106347 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| [256K, 512K) 143 | | [512K, 1M) 1 | | [1M, 2M) 0 | | [2M, 4M) 0 | | [4M, 8M) 0 | | [8M, 16M) 0 | | [16M, 32M) 0 | | [32M, 64M) 0 | | [64M, 128M) 0 | | [128M, 256M) 0 | | [256M, 512M) 0 | | [512M, 1G) 0 | | [1G, 2G) 0 | | [2G, 4G) 606 | |
rebased on 01/24/23 mm-unstable
Sidhartha Kumar (8):
mm/hugetlb: convert hugetlb_install_page to folios mm/hugetlb: convert hugetlbfs_pagecache_present() to folios mm/hugetlb: convert putback_active_hugepage to take in a folio mm/hugetlb: convert hugetlb fault paths to use alloc_hugetlb_folio() mm/hugetlb: convert restore_reserve_on_error to take in a folio mm/hugetlb: convert hugetlb_add_to_page_cache to take in a folio mm/hugetlb: convert hugetlb_wp() to take in a folio Documentation/mm: update hugetlbfs documentation to mention
alloc_hugetlb_folio
Documentation/mm/hugetlbfs_reserv.rst | 21 +- .../zh_CN/mm/hugetlbfs_reserv.rst | 14 +- fs/hugetlbfs/inode.c | 40 +-- include/linux/hugetlb.h | 16 +- include/linux/rmap.h | 2 +- mm/hugetlb.c | 279 +++++++++--------- mm/mempolicy.c | 6 +- mm/migrate.c | 8 +- mm/rmap.c | 6 +- 9 files changed, 196 insertions(+), 196 deletions(-)
Signature · 1 line
2.39.1
PATCH v2 1/8 mm/hugetlb: convert hugetlb_install_page to folios
Change hugetlb_install_page() to hugetlb_install_folio(). This reduces one user of the Huge Page flag macros which take in a page.
--- mm/hugetlb.c | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-)
mm/hugetlb.c +7 -7
@@ -4946,14 +4946,14 @@ static bool is_hugetlb_entry_hwpoisoned(pte_t pte) }static void -hugetlb_install_page (struct vm_area_struct *vma,pte_t *ptep,unsigned long addr, -struct page *new_page) +hugetlb_install_folio (struct vm_area_struct *vma,pte_t *ptep,unsigned long addr, +struct folio *new_folio) { -__SetPageUptodate (new_page); -hugepage_add_new_anon_rmap (new_page, vma, addr); -set_huge_pte_at (vma->vm_mm, addr, ptep,make_huge_pte (vma, new_page,1 )); +__folio_mark_uptodate (new_folio); +hugepage_add_new_anon_rmap (&new_folio->page, vma, addr); +set_huge_pte_at (vma->vm_mm, addr, ptep,make_huge_pte (vma, &new_folio->page,1 ));hugetlb_count_add (pages_per_huge_page (hstate_vma (vma)), vma->vm_mm); -SetHPageMigratable (new_page); +folio_set_hugetlb_migratable (new_folio); } int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src,
@@ -5107,7 +5107,7 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src,/* huge_ptep of dst_pte won't change as in child */ goto again; } -hugetlb_install_page (dst_vma, dst_pte, addr, new); +hugetlb_install_folio (dst_vma, dst_pte, addr,page_folio (new));spin_unlock (src_ptl);spin_unlock (dst_ptl);continue ;
Signature · 1 line
2.39.1
PATCH v2 2/8 mm/hugetlb: convert hugetlbfs_pagecache_present() to folios
In reply to Sidhartha KumarRefactor hugetlbfs_pagecache_present() to avoid getting and dropping a refcount on a page. Use RCU and page_cache_next_miss() instead.
--- mm/hugetlb.c | 16 +++++++--------- 1 file changed, 7 insertions(+), 9 deletions(-)
mm/hugetlb.c +7 -9
@@ -5651,17 +5651,15 @@ static vm_fault_t hugetlb_wp(struct mm_struct *mm, struct vm_area_struct *vma,static bool hugetlbfs_pagecache_present (struct hstate *h,struct vm_area_struct *vma,unsigned long address) { -struct address_space *mapping; -pgoff_t idx; -struct page *page; +struct address_space *mapping = vma->vm_file->f_mapping; +pgoff_t idx =vma_hugecache_offset (h, vma, address); +bool present; - mapping = vma->vm_file->f_mapping; - idx =vma_hugecache_offset (h, vma, address); +rcu_read_lock (); + present =page_cache_next_miss (mapping, idx,1 ) != idx; +rcu_read_unlock (); - page =find_get_page (mapping, idx); -if (page) -put_page (page); -return page !=NULL ; +return present; } int hugetlb_add_to_page_cache(struct page *page, struct address_space *mapping,
Signature · 1 line
2.39.1
PATCH v2 3/8 mm/hugetlb: convert putback_active_hugepage to take in a folio
In reply to Sidhartha KumarConvert putback_active_hugepage() to folio_putback_active_hugetlb(), this removes one user of the Huge Page macros which take in a page. The callers in migrate.c are also cleaned up by being able to directly use the src and dst folio variables.
---
The call to folio_putback_active_hugetlb(page_folio(page)); in putback_movable_pages() is being converted so that a folio can be passed directly. This patch has been sent by Vishal Moola and is currently under review.
include/linux/hugetlb.h | 4 ++-- mm/hugetlb.c | 8 ++++---- mm/migrate.c | 8 ++++---- 3 files changed, 10 insertions(+), 10 deletions(-)
include/linux/hugetlb.h +2 -2
@@ -174,7 +174,7 @@ int isolate_hugetlb(struct folio *folio, struct list_head *list);int get_hwpoison_hugetlb_folio (struct folio *folio,bool *hugetlb,bool unpoison);int get_huge_page_for_hwpoison (unsigned long pfn,int flags,bool *migratable_cleared); -void putback_active_hugepage (struct page *page); +void folio_putback_active_hugetlb (struct folio *folio);void move_hugetlb_state (struct folio *old_folio,struct folio *new_folio,int reason);void free_huge_page (struct page *page);void hugetlb_fix_reserve_counts (struct inode *inode);
@@ -428,7 +428,7 @@ static inline int get_huge_page_for_hwpoison(unsigned long pfn, int flags,return 0 ; } -static inline void putback_active_hugepage (struct page *page) +static inline void folio_putback_active_hugetlb (struct folio *folio) { }
mm/hugetlb.c +4 -4
@@ -7300,13 +7300,13 @@ int get_huge_page_for_hwpoison(unsigned long pfn, int flags,return ret; } -void putback_active_hugepage (struct page *page) +void folio_putback_active_hugetlb (struct folio *folio) {spin_lock_irq (&hugetlb_lock); -SetHPageMigratable (page); -list_move_tail (&page->lru, &(page_hstate (page))->hugepage_activelist); +folio_set_hugetlb_migratable (folio); +list_move_tail (&folio->lru, &(folio_hstate (folio))->hugepage_activelist);spin_unlock_irq (&hugetlb_lock); -put_page (page); +folio_put (folio); }void move_hugetlb_state (struct folio *old_folio,struct folio *new_folio,int reason)
mm/migrate.c +4 -4
@@ -151,7 +151,7 @@ void putback_movable_pages(struct list_head *l)list_for_each_entry_safe (page, page2, l, lru) {if (unlikely (PageHuge (page))) { -putback_active_hugepage (page); +folio_putback_active_hugetlb (page_folio (page));continue ; }list_del (&page->lru);
@@ -1374,7 +1374,7 @@ static int unmap_and_move_huge_page(new_page_t get_new_page,if (folio_ref_count (src) ==1 ) {/* page was freed from under us. So we are done. */ -putback_active_hugepage (hpage); +folio_putback_active_hugetlb (src);return MIGRATEPAGE_SUCCESS ; }
@@ -1459,7 +1459,7 @@ static int unmap_and_move_huge_page(new_page_t get_new_page,folio_unlock (src); out:if (rc ==MIGRATEPAGE_SUCCESS ) -putback_active_hugepage (hpage); +folio_putback_active_hugetlb (src);else if (rc != -EAGAIN )list_move_tail (&src->lru, ret);
@@ -1471,7 +1471,7 @@ static int unmap_and_move_huge_page(new_page_t get_new_page,if (put_new_page)put_new_page (new_hpage, private);else -putback_active_hugepage (new_hpage); +folio_putback_active_hugetlb (dst);return rc; }
Signature · 1 line
2.39.1
PATCH v2 4/8 mm/hugetlb: convert hugetlb fault paths to use alloc_hugetlb_folio()
In reply to Sidhartha KumarChange alloc_huge_page() to alloc_hugetlb_folio() by changing all callers to handle the now folio return type of the function. In this conversion, alloc_huge_page_vma() is also changed to alloc_hugetlb_folio_vma() and hugepage_add_new_anon_rmap() is changed to take in a folio directly. Many additions of '&folio->page' are cleaned up in subsequent patches.
hugetlbfs_fallocate() is also refactored to use the RCU + page_cache_next_miss() API.
--- fs/hugetlbfs/inode.c | 40 ++++---- include/linux/hugetlb.h | 8 +- include/linux/rmap.h | 2 +- mm/hugetlb.c | 201 ++++++++++++++++++++-------------------- mm/mempolicy.c | 6 +- mm/rmap.c | 6 +- 6 files changed, 133 insertions(+), 130 deletions(-)
fs/hugetlbfs/inode.c +21 -19
@@ -819,8 +819,9 @@ static long hugetlbfs_fallocate(struct file *file, int mode, loff_t offset,* This is supposed to be the vaddr where the page is being * faulted in, but we have no vaddr here. */ -struct page *page; +struct folio *folio;unsigned long addr; +bool present;cond_resched ();
@@ -844,48 +845,49 @@ static long hugetlbfs_fallocate(struct file *file, int mode, loff_t offset,mutex_lock (&hugetlb_fault_mutex_table[hash]);/* See if already present in mapping to avoid alloc/free */ - page =find_get_page (mapping, index); -if (page) { -put_page (page); +rcu_read_lock (); + present =page_cache_next_miss (mapping, index,1 ) != index; +rcu_read_unlock (); +if (present) {mutex_unlock (&hugetlb_fault_mutex_table[hash]);hugetlb_drop_vma_policy (&pseudo_vma);continue ; }/* -* Allocate page without setting the avoid_reserve argument. +* Allocate folio without setting the avoid_reserve argument. * There certainly are no reserves associated with the * pseudo_vma. However, there could be shared mappings with * reserves for the file at the inode level. If we fallocate -* pages in these areas, we need to consume the reserves +* folios in these areas, we need to consume the reserves * to keep reservation accounting consistent. */ - page =alloc_huge_page (&pseudo_vma, addr,0 ); + folio =alloc_hugetlb_folio (&pseudo_vma, addr,0 );hugetlb_drop_vma_policy (&pseudo_vma); -if (IS_ERR (page)) { +if (IS_ERR (folio)) {mutex_unlock (&hugetlb_fault_mutex_table[hash]); - error =PTR_ERR (page); + error =PTR_ERR (folio); goto out; } -clear_huge_page (page, addr,pages_per_huge_page (h)); -__SetPageUptodate (page); - error =hugetlb_add_to_page_cache (page, mapping, index); +clear_huge_page (&folio->page, addr,pages_per_huge_page (h)); +__folio_mark_uptodate (folio); + error =hugetlb_add_to_page_cache (&folio->page, mapping, index);if (unlikely (error)) { -restore_reserve_on_error (h, &pseudo_vma, addr, page); -put_page (page); +restore_reserve_on_error (h, &pseudo_vma, addr, &folio->page); +folio_put (folio);mutex_unlock (&hugetlb_fault_mutex_table[hash]); goto out; }mutex_unlock (&hugetlb_fault_mutex_table[hash]); -SetHPageMigratable (page); +folio_set_hugetlb_migratable (folio);/* -* unlock_page because locked by hugetlb_add_to_page_cache() -* put_page() due to reference from alloc_huge_page() +* folio_unlock because locked by hugetlb_add_to_page_cache() +* folio_put() due to reference from alloc_hugetlb_folio() */ -unlock_page (page); -put_page (page); +folio_unlock (folio); +folio_put (folio); }if (!(mode &FALLOC_FL_KEEP_SIZE ) && offset + len > inode->i_size)
include/linux/hugetlb.h +4 -4
@@ -716,11 +716,11 @@ struct huge_bootmem_page { };int isolate_or_dissolve_huge_page (struct page *page,struct list_head *list); -struct page *alloc_huge_page (struct vm_area_struct *vma, +struct folio *alloc_hugetlb_folio (struct vm_area_struct *vma,unsigned long addr,int avoid_reserve);struct folio *alloc_hugetlb_folio_nodemask (struct hstate *h,int preferred_nid,nodemask_t *nmask,gfp_t gfp_mask); -struct page *alloc_huge_page_vma (struct hstate *h,struct vm_area_struct *vma, +struct folio *alloc_hugetlb_folio_vma (struct hstate *h,struct vm_area_struct *vma,unsigned long address);int hugetlb_add_to_page_cache (struct page *page,struct address_space *mapping,pgoff_t idx);
@@ -1032,7 +1032,7 @@ static inline int isolate_or_dissolve_huge_page(struct page *page,return -ENOMEM ; } -static inline struct page *alloc_huge_page (struct vm_area_struct *vma, +static inline struct folio *alloc_hugetlb_folio (struct vm_area_struct *vma,unsigned long addr,int avoid_reserve) {
@@ -1046,7 +1046,7 @@ alloc_hugetlb_folio_nodemask(struct hstate *h, int preferred_nid,return NULL ; } -static inline struct page *alloc_huge_page_vma (struct hstate *h, +static inline struct folio *alloc_hugetlb_folio_vma (struct hstate *h,struct vm_area_struct *vma,unsigned long address) {
include/linux/rmap.h +1 -1
@@ -203,7 +203,7 @@ void page_remove_rmap(struct page *, struct vm_area_struct *,void hugepage_add_anon_rmap (struct page *,struct vm_area_struct *,unsigned long address,rmap_t flags); -void hugepage_add_new_anon_rmap (struct page *,struct vm_area_struct *, +void hugepage_add_new_anon_rmap (struct folio *,struct vm_area_struct *,unsigned long address);static inline void __page_dup_rmap (struct page *page,bool compound)
mm/hugetlb.c +101 -100
508 lines not on this page · raw message
mm/mempolicy.c +4 -2
@@ -1218,9 +1218,11 @@ static struct page *new_page(struct page *page, unsigned long start)break ; } -if (folio_test_hugetlb (src)) -return alloc_huge_page_vma (page_hstate (&src->page), +if (folio_test_hugetlb (src)) { + dst =alloc_hugetlb_folio_vma (folio_hstate (src), vma, address); +return &dst->page; + }if (folio_test_large (src)) gfp =GFP_TRANSHUGE ;
mm/rmap.c +2 -4
@@ -2547,15 +2547,13 @@ void hugepage_add_anon_rmap(struct page *page, struct vm_area_struct *vma, !!(flags & RMAP_EXCLUSIVE)); } -void hugepage_add_new_anon_rmap (struct page *page, +void hugepage_add_new_anon_rmap (struct folio *folio,struct vm_area_struct *vma,unsigned long address) { -struct folio *folio =page_folio (page); -BUG_ON (address < vma->vm_start || address >= vma->vm_end);/* increment count (starts at -1) */ atomic_set (&folio->_entire_mapcount,0 );folio_clear_hugetlb_restore_reserve (folio); -__page_set_anon_rmap (folio, page, vma, address,1 ); +__page_set_anon_rmap (folio, &folio->page, vma, address,1 ); }#endif /* CONFIG_HUGETLB_PAGE */
Signature · 1 line
2.39.1
PATCH v2 5/8 mm/hugetlb: convert restore_reserve_on_error to take in a folio
In reply to Sidhartha KumarEvery caller of restore_reserve_on_error() is now passing in &folio->page, change the function to take in a folio directly and clean up the call sites.
--- fs/hugetlbfs/inode.c | 2 +- include/linux/hugetlb.h | 2 +- mm/hugetlb.c | 21 ++++++++++----------- 3 files changed, 12 insertions(+), 13 deletions(-)
fs/hugetlbfs/inode.c +1 -1
@@ -873,7 +873,7 @@ static long hugetlbfs_fallocate(struct file *file, int mode, loff_t offset,__folio_mark_uptodate (folio); error =hugetlb_add_to_page_cache (&folio->page, mapping, index);if (unlikely (error)) { -restore_reserve_on_error (h, &pseudo_vma, addr, &folio->page); +restore_reserve_on_error (h, &pseudo_vma, addr, folio);folio_put (folio);mutex_unlock (&hugetlb_fault_mutex_table[hash]); goto out;
include/linux/hugetlb.h +1 -1
@@ -725,7 +725,7 @@ struct folio *alloc_hugetlb_folio_vma(struct hstate *h, struct vm_area_struct *vint hugetlb_add_to_page_cache (struct page *page,struct address_space *mapping,pgoff_t idx);void restore_reserve_on_error (struct hstate *h,struct vm_area_struct *vma, -unsigned long address,struct page *page); +unsigned long address,struct folio *folio);/* arch callback */ int __init __alloc_bootmem_huge_page (struct hstate *h,int nid);
mm/hugetlb.c +10 -11
@@ -2802,9 +2802,9 @@ static long vma_del_reservation(struct hstate *h,* and the hugetlb mutex should remain held when calling this routine. * * It handles two specific cases: -* 1) A reservation was in place and the page consumed the reservation. -* HPageRestoreReserve is set in the page. -* 2) No reservation was in place for the page, so HPageRestoreReserve is +* 1) A reservation was in place and the folio consumed the reservation. +* hugetlb_restore_reserve is set in the folio. +* 2) No reservation was in place for the page, so hugetlb_restore_reserve is * not set. However, alloc_hugetlb_folio always updates the reserve map. * * In case 1, free_huge_page later in the error path will increment the
@@ -2817,9 +2817,8 @@ static long vma_del_reservation(struct hstate *h,* In case 2, simply undo reserve map modifications done by alloc_hugetlb_folio. */ void restore_reserve_on_error (struct hstate *h,struct vm_area_struct *vma, -unsigned long address,struct page *page) +unsigned long address,struct folio *folio) { -struct folio *folio =page_folio (page);long rc =vma_needs_reservation (h, vma, address);if (folio_test_hugetlb_restore_reserve (folio)) {
@@ -5102,7 +5101,7 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, entry =huge_ptep_get (src_pte);if (!pte_same (src_pte_old, entry)) {restore_reserve_on_error (h, dst_vma, addr, - &new_folio->page); + new_folio);folio_put (new_folio);/* huge_ptep of dst_pte won't change as in child */ goto again;
@@ -5634,7 +5633,7 @@ static vm_fault_t hugetlb_wp(struct mm_struct *mm, struct vm_area_struct *vma,* unshare) */ if (new_folio !=page_folio (old_page)) -restore_reserve_on_error (h, vma, haddr, &new_folio->page); +restore_reserve_on_error (h, vma, haddr, new_folio);folio_put (new_folio); out_release_old:put_page (old_page);
@@ -5846,7 +5845,7 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm,* to the page cache. So it's safe to call * restore_reserve_on_error() here. */ -restore_reserve_on_error (h, vma, haddr, &folio->page); +restore_reserve_on_error (h, vma, haddr, folio);folio_put (folio); goto out; }
@@ -5947,7 +5946,7 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm,spin_unlock (ptl); backout_unlocked:if (new_folio && !new_pagecache_folio) -restore_reserve_on_error (h, vma, haddr, &folio->page); +restore_reserve_on_error (h, vma, haddr, folio);folio_unlock (folio);folio_put (folio);
@@ -6210,7 +6209,7 @@ int hugetlb_mcopy_atomic_pte(struct mm_struct *dst_mm,/* Free the allocated folio which may have * consumed a reservation. */ -restore_reserve_on_error (h, dst_vma, dst_addr, &folio->page); +restore_reserve_on_error (h, dst_vma, dst_addr, folio);folio_put (folio);/* Allocate a temporary folio to hold the copied
@@ -6339,7 +6338,7 @@ int hugetlb_mcopy_atomic_pte(struct mm_struct *dst_mm,folio_unlock (folio); out_release_nounlock:if (!folio_in_pagecache) -restore_reserve_on_error (h, dst_vma, dst_addr, &folio->page); +restore_reserve_on_error (h, dst_vma, dst_addr, folio);folio_put (folio); goto out; }
Signature · 1 line
2.39.1
PATCH v2 6/8 mm/hugetlb: convert hugetlb_add_to_page_cache to take in a folio
In reply to Sidhartha KumarEvery caller of hugetlb_add_to_page_cache() is now passing in &folio->page, change the function to take in a folio directly and clean up the call sites.
--- fs/hugetlbfs/inode.c | 2 +- include/linux/hugetlb.h | 2 +- mm/hugetlb.c | 9 ++++----- 3 files changed, 6 insertions(+), 7 deletions(-)
fs/hugetlbfs/inode.c +1 -1
@@ -871,7 +871,7 @@ static long hugetlbfs_fallocate(struct file *file, int mode, loff_t offset, }clear_huge_page (&folio->page, addr,pages_per_huge_page (h));__folio_mark_uptodate (folio); - error =hugetlb_add_to_page_cache (&folio->page, mapping, index); + error =hugetlb_add_to_page_cache (folio, mapping, index);if (unlikely (error)) {restore_reserve_on_error (h, &pseudo_vma, addr, folio);folio_put (folio);
include/linux/hugetlb.h +1 -1
@@ -722,7 +722,7 @@ struct folio *alloc_hugetlb_folio_nodemask(struct hstate *h, int preferred_nid, nodemask_t *nmask, gfp_t gfp_mask);struct folio *alloc_hugetlb_folio_vma (struct hstate *h,struct vm_area_struct *vma,unsigned long address); -int hugetlb_add_to_page_cache (struct page *page,struct address_space *mapping, +int hugetlb_add_to_page_cache (struct folio *folio,struct address_space *mapping,pgoff_t idx);void restore_reserve_on_error (struct hstate *h,struct vm_area_struct *vma,unsigned long address,struct folio *folio);
mm/hugetlb.c +4 -5
@@ -5662,10 +5662,9 @@ static bool hugetlbfs_pagecache_present(struct hstate *h,return present; } -int hugetlb_add_to_page_cache (struct page *page,struct address_space *mapping, +int hugetlb_add_to_page_cache (struct folio *folio,struct address_space *mapping,pgoff_t idx) { -struct folio *folio =page_folio (page);struct inode *inode = mapping->host;struct hstate *h =hstate_inode (inode);int err;
@@ -5677,7 +5676,7 @@ int hugetlb_add_to_page_cache(struct page *page, struct address_space *mapping,__folio_clear_locked (folio);return err; } -ClearHPageRestoreReserve (page); +folio_clear_hugetlb_restore_reserve (folio);/* * mark folio dirty so that it will not be removed from cache/file
@@ -5836,7 +5835,7 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm, new_folio = true;if (vma->vm_flags &VM_MAYSHARE ) { -int err =hugetlb_add_to_page_cache (&folio->page, mapping, idx); +int err =hugetlb_add_to_page_cache (folio, mapping, idx);if (err) {/* * err can't be -EEXIST which implies someone
@@ -6269,7 +6268,7 @@ int hugetlb_mcopy_atomic_pte(struct mm_struct *dst_mm,* hugetlb_fault_mutex_table that here must be hold by * the caller. */ - ret =hugetlb_add_to_page_cache (&folio->page, mapping, idx); + ret =hugetlb_add_to_page_cache (folio, mapping, idx);if (ret) goto out_release_nounlock; folio_in_pagecache = true;
Signature · 1 line
2.39.1
PATCH v2 7/8 mm/hugetlb: convert hugetlb_wp() to take in a folio
In reply to Sidhartha KumarChange the pagecache_page argument of hugetlb_wp to pagecache_folio. Replaces a call to find_lock_page() with filemap_lock_folio().
--- mm/hugetlb.c | 32 ++++++++++++++++---------------- 1 file changed, 16 insertions(+), 16 deletions(-)
mm/hugetlb.c +16 -16
@@ -5472,7 +5472,7 @@ static void unmap_ref_private(struct mm_struct *mm, struct vm_area_struct *vma,*/ static vm_fault_t hugetlb_wp (struct mm_struct *mm,struct vm_area_struct *vma,unsigned long address,pte_t *ptep,unsigned int flags, -struct page *pagecache_page,spinlock_t *ptl) +struct folio *pagecache_folio,spinlock_t *ptl) {const bool unshare = flags &FAULT_FLAG_UNSHARE ;pte_t pte;
@@ -5529,7 +5529,7 @@ static vm_fault_t hugetlb_wp(struct mm_struct *mm, struct vm_area_struct *vma,* of the full address range. */ if (is_vma_resv_set (vma,HPAGE_RESV_OWNER ) && - old_page != pagecache_page) +page_folio (old_page) != pagecache_folio) outside_reserve =1 ;get_page (old_page);
@@ -5922,7 +5922,7 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm,hugetlb_count_add (pages_per_huge_page (h), mm);if ((flags &FAULT_FLAG_WRITE ) && !(vma->vm_flags &VM_SHARED )) {/* Optimization, do the COW without a second fault */ - ret =hugetlb_wp (mm, vma, address, ptep, flags, &folio->page, ptl); + ret =hugetlb_wp (mm, vma, address, ptep, flags, folio, ptl); }spin_unlock (ptl);
@@ -5985,7 +5985,7 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma,u32 hash;pgoff_t idx;struct page *page =NULL ; -struct page *pagecache_page =NULL ; +struct folio *pagecache_folio =NULL ;struct hstate *h =hstate_vma (vma);struct address_space *mapping;int need_wait_lock =0 ;
@@ -6067,7 +6067,7 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma,/* Just decrements count, does not deallocate */ vma_end_reservation (h, vma, haddr); - pagecache_page =find_lock_page (mapping, idx); + pagecache_folio =filemap_lock_folio (mapping, idx); } ptl =huge_pte_lock (h, mm, ptep);
@@ -6087,9 +6087,9 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, };spin_unlock (ptl); -if (pagecache_page) { -unlock_page (pagecache_page); -put_page (pagecache_page); +if (pagecache_folio) { +folio_unlock (pagecache_folio); +folio_put (pagecache_folio); }hugetlb_vma_unlock_read (vma);mutex_unlock (&hugetlb_fault_mutex_table[hash]);
@@ -6098,11 +6098,11 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma,/* * hugetlb_wp() requires page locks of pte_page(entry) and -* pagecache_page, so here we need take the former one -* when page != pagecache_page or !pagecache_page. +* pagecache_folio, so here we need take the former one +* when page != pagecache_folio or !pagecache_folio. */ page =pte_page (entry); -if (page != pagecache_page) +if (page_folio (page) != pagecache_folio)if (!trylock_page (page)) { need_wait_lock =1 ; goto out_ptl;
@@ -6113,7 +6113,7 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma,if (flags & (FAULT_FLAG_WRITE |FAULT_FLAG_UNSHARE )) {if (!huge_pte_write (entry)) { ret =hugetlb_wp (mm, vma, address, ptep, flags, - pagecache_page, ptl); + pagecache_folio, ptl); goto out_put_page; }else if (likely (flags &FAULT_FLAG_WRITE )) { entry =huge_pte_mkdirty (entry);
@@ -6124,15 +6124,15 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, flags & FAULT_FLAG_WRITE))update_mmu_cache (vma, haddr, ptep); out_put_page: -if (page != pagecache_page) +if (page_folio (page) != pagecache_folio)unlock_page (page);put_page (page); out_ptl:spin_unlock (ptl); -if (pagecache_page) { -unlock_page (pagecache_page); -put_page (pagecache_page); +if (pagecache_folio) { +folio_unlock (pagecache_folio); +folio_put (pagecache_folio); } out_mutex:hugetlb_vma_unlock_read (vma);
Signature · 1 line
2.39.1
PATCH v2 8/8 Documentation/mm: update hugetlbfs documentation to mention alloc_hugetlb_folio
In reply to Sidhartha Kumar--- Documentation/mm/hugetlbfs_reserv.rst | 21 ++++++++++--------- .../zh_CN/mm/hugetlbfs_reserv.rst | 14 ++++++------- 2 files changed, 18 insertions(+), 17 deletions(-)
Documentation/mm/hugetlbfs_reserv.rst +11 -10
@@ -181,14 +181,14 @@ Consuming Reservations/Allocating a Huge Page Reservations are consumed when huge pages associated with the reservations are allocated and instantiated in the corresponding mapping. The allocation -is performed within the routine alloc_huge_page():: +is performed within the routine alloc_hugetlb_folio():: - struct page *alloc_huge_page(struct vm_area_struct *vma, + struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma, unsigned long addr, int avoid_reserve) -alloc_huge_page is passed a VMA pointer and a virtual address, so it can +alloc_hugetlb_folio is passed a VMA pointer and a virtual address, so it can consult the reservation map to determine if a reservation exists. In addition, -alloc_huge_page takes the argument avoid_reserve which indicates reserves +alloc_hugetlb_folio takes the argument avoid_reserve which indicates reserves should not be used even if it appears they have been set aside for the specified address. The avoid_reserve argument is most often used in the case of Copy on Write and Page Migration where additional copies of an existing
@@ -208,7 +208,8 @@ a reservation for the allocation. After determining whether a reservation exists and can be used for the allocation, the routine dequeue_huge_page_vma() is called. This routine takes two arguments related to reservations: -- avoid_reserve, this is the same value/argument passed to alloc_huge_page() +- avoid_reserve, this is the same value/argument passed to + alloc_hugetlb_folio(). - chg, even though this argument is of type long only the values 0 or 1 are passed to dequeue_huge_page_vma. If the value is 0, it indicates a reservation exists (see the section "Memory Policy and Reservations" for
@@ -233,9 +234,9 @@ the scope reservations. Even if a surplus page is allocated, the same reservation based adjustments as above will be made: SetPagePrivate(page) and resv_huge_pages--. -After obtaining a new huge page, (page)->private is set to the value of -the subpool associated with the page if it exists. This will be used for -subpool accounting when the page is freed. +After obtaining a new hugetlb folio, (folio)->_hugetlb_subpool is set to the +value of the subpool associated with the page if it exists. This will be used +for subpool accounting when the folio is freed. The routine vma_commit_reservation() is then called to adjust the reserve map based on the consumption of the reservation. In general, this involves
@@ -246,8 +247,8 @@ was no reservation in a shared mapping or this was a private mapping a new entry must be created. It is possible that the reserve map could have been changed between the call -to vma_needs_reservation() at the beginning of alloc_huge_page() and the -call to vma_commit_reservation() after the page was allocated. This would +to vma_needs_reservation() at the beginning of alloc_hugetlb_folio() and the +call to vma_commit_reservation() after the folio was allocated. This would be possible if hugetlb_reserve_pages was called for the same page in a shared mapping. In such cases, the reservation count and subpool free page count will be off by one. This rare condition can be identified by comparing the
Documentation/translations/zh_CN/mm/hugetlbfs_reserv.rst +7 -7
@@ -142,14 +142,14 @@ HPAGE_RESV_OWNER标志被设置,以表明该VMA拥有预留。 消耗预留/分配一个巨页 =========================== -当与预留相关的巨页在相应的映射中被分配和实例化时,预留就被消耗了。该分配是在函数alloc_huge_page() +当与预留相关的巨页在相应的映射中被分配和实例化时,预留就被消耗了。该分配是在函数alloc_hugetlb_folio() 中进行的:: - struct page *alloc_huge_page(struct vm_area_struct *vma, + struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma, unsigned long addr, int avoid_reserve) -alloc_huge_page被传递给一个VMA指针和一个虚拟地址,因此它可以查阅预留映射以确定是否存在预留。 -此外,alloc_huge_page需要一个参数avoid_reserve,该参数表示即使看起来已经为指定的地址预留了 +alloc_hugetlb_folio被传递给一个VMA指针和一个虚拟地址,因此它可以查阅预留映射以确定是否存在预留。 +此外,alloc_hugetlb_folio需要一个参数avoid_reserve,该参数表示即使看起来已经为指定的地址预留了 预留,也不应该使用预留。avoid_reserve参数最常被用于写时拷贝和页面迁移的情况下,即现有页面的额 外拷贝被分配。
@@ -162,7 +162,7 @@ vma_needs_reservation()返回的值通常为0或1。如果该地址存在预留 确定预留是否存在并可用于分配后,调用dequeue_huge_page_vma()函数。这个函数需要两个与预留有关 的参数: -- avoid_reserve,这是传递给alloc_huge_page()的同一个值/参数。 +- avoid_reserve,这是传递给alloc_hugetlb_folio()的同一个值/参数。 - chg,尽管这个参数的类型是long,但只有0或1的值被传递给dequeue_huge_page_vma。如果该值为0, 则表明存在预留(关于可能的问题,请参见 “预留和内存策略” 一节)。如果值 为1,则表示不存在预留,如果可能的话,必须从全局空闲池中取出该页。
@@ -179,7 +179,7 @@ free_huge_pages的值被递减。如果有一个与该页相关的预留,将 的剩余巨页和超额分配的问题。即使分配了一个多余的页面,也会进行与上面一样的基于预留的调整: SetPagePrivate(page) 和 resv_huge_pages--. -在获得一个新的巨页后,(page)->private被设置为与该页面相关的子池的值,如果它存在的话。当页 +在获得一个新的巨页后,(folio)->_hugetlb_subpool被设置为与该页面相关的子池的值,如果它存在的话。当页 面被释放时,这将被用于子池的计数。 然后调用函数vma_commit_reservation(),根据预留的消耗情况调整预留映射。一般来说,这涉及
@@ -199,7 +199,7 @@ SetPagePrivate(page)和resv_huge_pages-。 已经存在,所以不做任何改变。然而,如果共享映射中没有预留,或者这是一个私有映射,则必须创建 一个新的条目。 -在alloc_huge_page()开始调用vma_needs_reservation()和页面分配后调用 +在alloc_hugetlb_folio()开始调用vma_needs_reservation()和页面分配后调用 vma_commit_reservation()之间,预留映射有可能被改变。如果hugetlb_reserve_pages在共 享映射中为同一页面被调用,这将是可能的。在这种情况下,预留计数和子池空闲页计数会有一个偏差。 这种罕见的情况可以通过比较vma_needs_reservation和vma_commit_reservation的返回值来
Signature · 1 line
2.39.1