large folios swap-in: handle refault cases first
Subsystems (7)
v3 cover large folios swap-in: handle refault cases first
From: Barry Song <[email protected]>
This patch is extracted from the large folio swapin series[1], primarily addressing the handling of scenarios involving large folios in the swap cache. Currently, it is particularly focused on addressing the refaulting of mTHP, which is still undergoing reclamation. This approach aims to streamline code review and expedite the integration of this segment into the MM tree.
It relies on Ryan's swap-out series[2], leveraging the helper function swap_pte_batch() introduced by that series.
Presently, do_swap_page only encounters a large folio in the swap cache before the large folio is released by vmscan. However, the code should remain equally useful once we support large folio swap-in via swapin_readahead(). This approach can effectively reduce page faults and eliminate most redundant checks and early exits for MTE restoration in recent MTE patchset[3].
The large folio swap-in for SWP_SYNCHRONOUS_IO and swapin_readahead() will be split into separate patch sets and sent at a later time.
-v3:
- optimize swap_free_nr using bitmap with single one "long"; "Huang, Ying"
- drop swap_free() as suggested by "Huang, Ying", now hibernation can get batched;
- lots of cleanup in do_swap_page() as commented by Ryan Roberts and "Huang, Ying";
- handle arch_do_swap_page() with nr pages though the only platform which needs it, sparc, doesn't support THP_SWAPOUT as suggested by "Huang, Ying";
- introduce pte_move_swp_offset() as suggested by "Huang, Ying";
- drop the "any_shared" of checking swap entries with respect to David's comment;
- drop the counter of swapin_refault and keep it for debug purpose per Ying
- collect reviewed-by tags
-v2:
- rebase on top of mm-unstable in which Ryan's swap_pte_batch() has changed a lot.
- remove folio_add_new_anon_rmap() for !folio_test_anon()
as currently large folios are always anon(refault).
- add mTHP swpin refault counters
Link:
https://lore.kernel.org/linux-mm/[email protected]/
-v1:
Differences with the original large folios swap-in series
- collect r-o-b, acked;
- rename swap_nr_free to swap_free_nr, according to Ryan;
- limit the maximum kernel stack usage for swap_free_nr, Ryan;
- add output argument in swap_pte_batch to expose if all entries are exclusive
- many clean refinements, handle the corner case folio's virtual addr might not be naturally aligned
[1] https://lore.kernel.org/linux-mm/[email protected]/
[2] https://lore.kernel.org/linux-mm/[email protected]/
[3] https://lore.kernel.org/linux-mm/[email protected]/
Barry Song (3):
mm: remove swap_free() and always use swap_free_nr()
mm: introduce pte_move_swp_offset() helper which can move offset
bidirectionally
mm: introduce arch_do_swap_page_nr() which allows restore metadata for
nr pages
Chuanhua Han (3):
mm: swap: introduce swap_free_nr() for batched swap_free()
mm: swap: make should_try_to_free_swap() support large-folio
mm: swap: entirely map large folios found in swapcache
include/linux/pgtable.h | 26 ++++++++++++++---- include/linux/swap.h | 4 +-- kernel/power/swap.c | 7 ++--- mm/internal.h | 25 ++++++++++++++--- mm/memory.c | 61 +++++++++++++++++++++++++++++++++-------- mm/rmap.c | 4 +-- mm/shmem.c | 4 +-- mm/swapfile.c | 50 +++++++++++++++++++++++++++++---- 8 files changed, 143 insertions(+), 38 deletions(-)
Signature · 1 line
2.34.1
PATCH v3 1/6 mm: swap: introduce swap_free_nr() for batched swap_free()
From: Chuanhua Han <[email protected]>
While swapping in a large folio, we need to free swaps related to the whole folio. To avoid frequently acquiring and releasing swap locks, it is better to introduce an API for batched free.
Furthermore, this new function, swap_free_nr(), is designed to efficiently handle various scenarios for releasing a specified number, nr, of swap entries.
--- include/linux/swap.h | 5 +++++ mm/swapfile.c | 47 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 52 insertions(+)
include/linux/swap.h +5 -0
@@ -483,6 +483,7 @@ extern void swap_shmem_alloc(swp_entry_t);extern int swap_duplicate (swp_entry_t );extern int swapcache_prepare (swp_entry_t );extern void swap_free (swp_entry_t ); +extern void swap_free_nr (swp_entry_t entry,int nr_pages);extern void swapcache_free_entries (swp_entry_t *entries,int n);extern void free_swap_and_cache_nr (swp_entry_t entry,int nr);int swap_type_of (dev_t device,sector_t offset);
@@ -564,6 +565,10 @@ static inline void swap_free(swp_entry_t swp) { } +static inline void swap_free_nr (swp_entry_t entry,int nr_pages) +{ +} +static inline void put_swap_folio (struct folio *folio,swp_entry_t swp) { }
mm/swapfile.c +47 -0
53 lines not on this page · raw message
Signature · 1 line
2.34.1
Re: [PATCH v3 1/6] mm: swap: introduce swap_free_nr() for batched swap_free()
Quoted 9 lines · Barry Song
From: Chuanhua Han <[email protected]>
While swapping in a large folio, we need to free swaps related to the whole folio. To avoid frequently acquiring and releasing swap locks, it is better to introduce an API for batched free.
Furthermore, this new function,swap_free_nr(), is designed to efficiently handle various scenarios for releasing a specified number, nr, of swap entries.
Quoted 3 lines
Signed-off-by Chuanhua Han <[email protected]>Co-developed-by Barry Song <[email protected]>Signed-off-by Barry Song <[email protected]>
This looks much better!
Quoted 86 lines
--- include/linux/swap.h | 5 +++++ mm/swapfile.c | 47 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 52 insertions(+)include/linux/swap.h +5
-0@@ -483,6 +483,7 @@ extern void swap_shmem_alloc(swp_entry_t);extern int swap_duplicate (swp_entry_t );extern int swapcache_prepare (swp_entry_t );extern void swap_free (swp_entry_t ); +extern void swap_free_nr (swp_entry_t entry,int nr_pages);extern void swapcache_free_entries (swp_entry_t *entries,int n);extern void free_swap_and_cache_nr (swp_entry_t entry,int nr);int swap_type_of (dev_t device,sector_t offset);@@ -564,6 +565,10 @@ static inline void swap_free(swp_entry_t swp) { } +static inline void swap_free_nr (swp_entry_t entry,int nr_pages) +{ +} +static inline void put_swap_folio (struct folio *folio,swp_entry_t swp) { }mm/swapfile.c +47
-0@@ -1356,6 +1356,53 @@ void swap_free(swp_entry_t entry)__swap_entry_free (p, entry); } +static void cluster_swap_free_nr (struct swap_info_struct *sis, +unsigned long offset,int nr_pages) +{ +struct swap_cluster_info *ci; + DECLARE_BITMAP(to_free, BITS_PER_LONG) = { 0 }; +int i, nr; + + ci =lock_cluster_or_swap_info (sis, offset); +while (nr_pages) { + nr =min (BITS_PER_LONG , nr_pages); +for (i =0 ; i < nr; i++) { +if (!__swap_entry_free_locked (sis, offset + i,1 )) +bitmap_set (to_free, i,1 ); + } +if (!bitmap_empty (to_free,BITS_PER_LONG )) { +unlock_cluster_or_swap_info (sis, ci); +for_each_set_bit (i, to_free,BITS_PER_LONG ) +free_swap_slot (swp_entry (sis->type, offset + i)); +if (nr == nr_pages) +return ; +bitmap_clear (to_free,0 ,BITS_PER_LONG ); + ci =lock_cluster_or_swap_info (sis, offset); + } + offset += nr; + nr_pages -= nr; + } +unlock_cluster_or_swap_info (sis, ci); +} + +void swap_free_nr (swp_entry_t entry,int nr_pages) +{ +int nr; +struct swap_info_struct *sis; +unsigned long offset =swp_offset (entry); + + sis =_swap_info_get (entry); +if (!sis) +return ; + +while (nr_pages) { + nr =min_t (int, nr_pages,SWAPFILE_CLUSTER - offset %SWAPFILE_CLUSTER ); +cluster_swap_free_nr (sis, offset, nr); + offset += nr; + nr_pages -= nr; + } +} +/* * Called after dropping swapcache to decrease refcnt to swap entries. */
Re: [PATCH v3 1/6] mm: swap: introduce swap_free_nr() for batched swap_free()
In reply to Barry SongHi Barry,
Looks good. Looking forward to the change to batch free to skipping the swap slot cache.
All the entries are from the same swap device, it does not need the sort function. Currently it goes through a lot of locking and and unlocking inside the loop.
Chris
Quoted 102 lines · Barry Song
From: Chuanhua Han <[email protected]>
While swapping in a large folio, we need to free swaps related to the whole folio. To avoid frequently acquiring and releasing swap locks, it is better to introduce an API for batched free.
Furthermore, this new function,swap_free_nr(), is designed to efficiently handle various scenarios for releasing a specified number, nr, of swap entries.Signed-off-by Chuanhua Han <[email protected]>Co-developed-by Barry Song <[email protected]>Signed-off-by Barry Song <[email protected]>--- include/linux/swap.h | 5 +++++ mm/swapfile.c | 47 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 52 insertions(+)include/linux/swap.h +5
-0@@ -483,6 +483,7 @@ extern void swap_shmem_alloc(swp_entry_t);extern int swap_duplicate (swp_entry_t );extern int swapcache_prepare (swp_entry_t );extern void swap_free (swp_entry_t ); +extern void swap_free_nr (swp_entry_t entry,int nr_pages);extern void swapcache_free_entries (swp_entry_t *entries,int n);extern void free_swap_and_cache_nr (swp_entry_t entry,int nr);int swap_type_of (dev_t device,sector_t offset);@@ -564,6 +565,10 @@ static inline void swap_free(swp_entry_t swp) { } +static inline void swap_free_nr (swp_entry_t entry,int nr_pages) +{ +} +static inline void put_swap_folio (struct folio *folio,swp_entry_t swp) { }mm/swapfile.c +47
-1@@ -1356,6 +1356,53 @@ void swap_free(swp_entry_t entry)__swap_entry_free (p, entry); } +static void cluster_swap_free_nr (struct swap_info_struct *sis, +unsigned long offset,int nr_pages) +{ +struct swap_cluster_info *ci; + DECLARE_BITMAP(to_free, BITS_PER_LONG) = { 0 }; +int i, nr; + + ci =lock_cluster_or_swap_info (sis, offset); +while (nr_pages) { + nr =min (BITS_PER_LONG , nr_pages); +for (i =0 ; i < nr; i++) { +if (!__swap_entry_free_locked (sis, offset + i,1 )) +bitmap_set (to_free, i,1 ); + } +if (!bitmap_empty (to_free,BITS_PER_LONG )) { +unlock_cluster_or_swap_info (sis, ci); +for_each_set_bit (i, to_free,BITS_PER_LONG ) +free_swap_slot (swp_entry (sis->type, offset + i)); +if (nr == nr_pages) +return ; +bitmap_clear (to_free,0 ,BITS_PER_LONG ); + ci =lock_cluster_or_swap_info (sis, offset); + } + offset += nr; + nr_pages -= nr; + } +unlock_cluster_or_swap_info (sis, ci); +} + +void swap_free_nr (swp_entry_t entry,int nr_pages) +{ +int nr; +struct swap_info_struct *sis; +unsigned long offset =swp_offset (entry); + + sis =_swap_info_get (entry); +if (!sis) +return ; + +while (nr_pages) { + nr =min_t (int, nr_pages,SWAPFILE_CLUSTER - offset %SWAPFILE_CLUSTER ); +cluster_swap_free_nr (sis, offset, nr); + offset += nr; + nr_pages -= nr; + } +} +/* * Called after dropping swapcache to decrease refcnt to swap entries. */ --2.34.1
Re: [PATCH v3 1/6] mm: swap: introduce swap_free_nr() for batched swap_free()
In reply to Barry SongQuoted 9 lines · Barry Song
From: Chuanhua Han <[email protected]>
While swapping in a large folio, we need to free swaps related to the whole folio. To avoid frequently acquiring and releasing swap locks, it is better to introduce an API for batched free.
Furthermore, this new function,swap_free_nr(), is designed to efficiently handle various scenarios for releasing a specified number, nr, of swap entries.
Quoted 3 lines
Signed-off-by Chuanhua Han <[email protected]>Co-developed-by Barry Song <[email protected]>Signed-off-by Barry Song <[email protected]>
LGTM, Thanks!
Quoted 86 lines
--- include/linux/swap.h | 5 +++++ mm/swapfile.c | 47 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 52 insertions(+)include/linux/swap.h +5
-0@@ -483,6 +483,7 @@ extern void swap_shmem_alloc(swp_entry_t);extern int swap_duplicate (swp_entry_t );extern int swapcache_prepare (swp_entry_t );extern void swap_free (swp_entry_t ); +extern void swap_free_nr (swp_entry_t entry,int nr_pages);extern void swapcache_free_entries (swp_entry_t *entries,int n);extern void free_swap_and_cache_nr (swp_entry_t entry,int nr);int swap_type_of (dev_t device,sector_t offset);@@ -564,6 +565,10 @@ static inline void swap_free(swp_entry_t swp) { } +static inline void swap_free_nr (swp_entry_t entry,int nr_pages) +{ +} +static inline void put_swap_folio (struct folio *folio,swp_entry_t swp) { }mm/swapfile.c +47
-0@@ -1356,6 +1356,53 @@ void swap_free(swp_entry_t entry)__swap_entry_free (p, entry); } +static void cluster_swap_free_nr (struct swap_info_struct *sis, +unsigned long offset,int nr_pages) +{ +struct swap_cluster_info *ci; + DECLARE_BITMAP(to_free, BITS_PER_LONG) = { 0 }; +int i, nr; + + ci =lock_cluster_or_swap_info (sis, offset); +while (nr_pages) { + nr =min (BITS_PER_LONG , nr_pages); +for (i =0 ; i < nr; i++) { +if (!__swap_entry_free_locked (sis, offset + i,1 )) +bitmap_set (to_free, i,1 ); + } +if (!bitmap_empty (to_free,BITS_PER_LONG )) { +unlock_cluster_or_swap_info (sis, ci); +for_each_set_bit (i, to_free,BITS_PER_LONG ) +free_swap_slot (swp_entry (sis->type, offset + i)); +if (nr == nr_pages) +return ; +bitmap_clear (to_free,0 ,BITS_PER_LONG ); + ci =lock_cluster_or_swap_info (sis, offset); + } + offset += nr; + nr_pages -= nr; + } +unlock_cluster_or_swap_info (sis, ci); +} + +void swap_free_nr (swp_entry_t entry,int nr_pages) +{ +int nr; +struct swap_info_struct *sis; +unsigned long offset =swp_offset (entry); + + sis =_swap_info_get (entry); +if (!sis) +return ; + +while (nr_pages) { + nr =min_t (int, nr_pages,SWAPFILE_CLUSTER - offset %SWAPFILE_CLUSTER ); +cluster_swap_free_nr (sis, offset, nr); + offset += nr; + nr_pages -= nr; + } +} +/* * Called after dropping swapcache to decrease refcnt to swap entries. */
PATCH v3 2/6 mm: remove swap_free() and always use swap_free_nr()
In reply to Barry SongFrom: Barry Song <[email protected]>
To streamline maintenance efforts, we propose discontinuing the use of swap_free(). Instead, we can simply invoke swap_free_nr() with nr set to 1. This adjustment offers the advantage of enabling batch processing within kernel/power/swap.c. Furthermore, swap_free_nr() is designed with a bitmap consisting of only one long, resulting in overhead that can be ignored for cases where nr equals 1.
--- include/linux/swap.h | 5 ----- kernel/power/swap.c | 7 +++---- mm/memory.c | 2 +- mm/rmap.c | 4 ++-- mm/shmem.c | 4 ++-- mm/swapfile.c | 19 +++++-------------- 6 files changed, 13 insertions(+), 28 deletions(-)
include/linux/swap.h +0 -5
@@ -482,7 +482,6 @@ extern int add_swap_count_continuation(swp_entry_t, gfp_t);extern void swap_shmem_alloc (swp_entry_t );extern int swap_duplicate (swp_entry_t );extern int swapcache_prepare (swp_entry_t ); -extern void swap_free (swp_entry_t );extern void swap_free_nr (swp_entry_t entry,int nr_pages);extern void swapcache_free_entries (swp_entry_t *entries,int n);extern void free_swap_and_cache_nr (swp_entry_t entry,int nr);
@@ -561,10 +560,6 @@ static inline int swapcache_prepare(swp_entry_t swp)return 0 ; } -static inline void swap_free (swp_entry_t swp) -{ -} -static inline void swap_free_nr (swp_entry_t entry,int nr_pages) { }
kernel/power/swap.c +3 -4
@@ -181,7 +181,7 @@ sector_t alloc_swapdev_block(int swap) offset =swp_offset (get_swap_page_of_type (swap));if (offset) {if (swsusp_extents_insert (offset)) -swap_free (swp_entry (swap, offset)); +swap_free_nr (swp_entry (swap, offset),1 );else return swapdev_block (swap, offset); }
@@ -200,12 +200,11 @@ void free_all_swap_pages(int swap)while ((node = swsusp_extents.rb_node)) {struct swsusp_extent *ext; -unsigned long offset; ext = rb_entry(node, struct swsusp_extent, node);rb_erase (node, &swsusp_extents); -for (offset = ext->start; offset <= ext->end; offset++) -swap_free (swp_entry (swap, offset)); +swap_free_nr (swp_entry (swap, ext->start), + ext->end - ext->start +1 );kfree (ext); }
mm/memory.c +1 -1
@@ -4225,7 +4225,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)* We're already holding a reference on the page but haven't mapped it * yet. */ -swap_free (entry); +swap_free_nr (entry,1 );if (should_try_to_free_swap (folio, vma, vmf->flags))folio_free_swap (folio);
mm/rmap.c +2 -2
@@ -1865,7 +1865,7 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, goto walk_done_err; }if (arch_unmap_one (mm, vma, address, pteval) <0 ) { -swap_free (entry); +swap_free_nr (entry,1 );set_pte_at (mm, address, pvmw.pte, pteval); goto walk_done_err; }
@@ -1873,7 +1873,7 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,/* See folio_try_share_anon_rmap(): clear PTE first. */ if (anon_exclusive &&folio_try_share_anon_rmap_pte (folio, subpage)) { -swap_free (entry); +swap_free_nr (entry,1 );set_pte_at (mm, address, pvmw.pte, pteval); goto walk_done_err; }
mm/shmem.c +2 -2
@@ -1836,7 +1836,7 @@ static void shmem_set_folio_swapin_error(struct inode *inode, pgoff_t index,* in shmem_evict_inode(). */ shmem_recalc_inode (inode,-1 ,-1 ); -swap_free (swap); +swap_free_nr (swap,1 ); }/*
@@ -1927,7 +1927,7 @@ static int shmem_swapin_folio(struct inode *inode, pgoff_t index,delete_from_swap_cache (folio);folio_mark_dirty (folio); -swap_free (swap); +swap_free_nr (swap,1 );put_swap_device (si); *foliop = folio;
mm/swapfile.c +5 -14
@@ -1343,19 +1343,6 @@ static void swap_entry_free(struct swap_info_struct *p, swp_entry_t entry)swap_range_free (p, offset,1 ); } -/* -* Caller has made sure that the swap device corresponding to entry -* is still around or has not been recycled. -*/ -void swap_free (swp_entry_t entry) -{ -struct swap_info_struct *p; - - p =_swap_info_get (entry); -if (p) -__swap_entry_free (p, entry); -} -static void cluster_swap_free_nr (struct swap_info_struct *sis,unsigned long offset,int nr_pages) {
@@ -1385,6 +1372,10 @@ static void cluster_swap_free_nr(struct swap_info_struct *sis,unlock_cluster_or_swap_info (sis, ci); } +/* +* Caller has made sure that the swap device corresponding to entry +* is still around or has not been recycled. +*/ void swap_free_nr (swp_entry_t entry,int nr_pages) {int nr;
@@ -1930,7 +1921,7 @@ static int unuse_pte(struct vm_area_struct *vma, pmd_t *pmd, new_pte =pte_mkuffd_wp (new_pte); setpte:set_pte_at (vma->vm_mm, addr, pte, new_pte); -swap_free (entry); +swap_free_nr (entry,1 ); out:if (pte)pte_unmap_unlock (pte, ptl);
Signature · 1 line
2.34.1
Re: [PATCH v3 2/6] mm: remove swap_free() and always use swap_free_nr()
Quoted 29 lines · Barry Song
From: Barry Song <[email protected]>
To streamline maintenance efforts, we propose discontinuing the use of
swap_free(). Instead, we can simply invokeswap_free_nr()with nr set to 1. This adjustment offers the advantage of enabling batch processing withinkernel/power/swap.c. Furthermore,swap_free_nr()is designed with a bitmap consisting of only one long, resulting in overhead that can be ignored for cases where nr equals 1.Suggested-by "Huang, Ying" <[email protected]>Signed-off-by Barry Song <[email protected]>Cc "Rafael J. Wysocki" <[email protected]>Cc Pavel Machek <[email protected]>Cc Len Brown <[email protected]>Cc Hugh Dickins <[email protected]>--- include/linux/swap.h | 5 ----- kernel/power/swap.c | 7 +++---- mm/memory.c | 2 +- mm/rmap.c | 4 ++-- mm/shmem.c | 4 ++-- mm/swapfile.c | 19 +++++-------------- 6 files changed, 13 insertions(+), 28 deletions(-)include/linux/swap.h +0
-0
include/linux/swap.h extern int add_swap_count_continuation(swp_entry_t, gfp_t); · line 485
Quoted 4 lines
include/linux/swap.h +0
-1@@ -482,7 +482,6 @@ extern int add_swap_count_continuation(swp_entry_t, gfp_t);extern void swap_shmem_alloc (swp_entry_t );extern int swap_duplicate (swp_entry_t );extern int swapcache_prepare (swp_entry_t ); -extern void swap_free (swp_entry_t );
I wonder if it would be cleaner to:
#define swap_free(entry) swap_free_nr((entry), 1)
To save all the churn for the callsites that just want to pass a single entry?
Quoted 143 lines
include/linux/swap.h +0
-4extern void swap_free_nr (swp_entry_t entry,int nr_pages);extern void swapcache_free_entries (swp_entry_t *entries,int n);extern void free_swap_and_cache_nr (swp_entry_t entry,int nr);@@ -561,10 +560,6 @@ static inline int swapcache_prepare(swp_entry_t swp)return 0 ; } -static inline void swap_free (swp_entry_t swp) -{ -} -static inline void swap_free_nr (swp_entry_t entry,int nr_pages) { }kernel/power/swap.c +3
-4@@ -181,7 +181,7 @@ sector_t alloc_swapdev_block(int swap) offset =swp_offset (get_swap_page_of_type (swap));if (offset) {if (swsusp_extents_insert (offset)) -swap_free (swp_entry (swap, offset)); +swap_free_nr (swp_entry (swap, offset),1 );else return swapdev_block (swap, offset); }@@ -200,12 +200,11 @@ void free_all_swap_pages(int swap)while ((node = swsusp_extents.rb_node)) {struct swsusp_extent *ext; -unsigned long offset; ext = rb_entry(node, struct swsusp_extent, node);rb_erase (node, &swsusp_extents); -for (offset = ext->start; offset <= ext->end; offset++) -swap_free (swp_entry (swap, offset)); +swap_free_nr (swp_entry (swap, ext->start), + ext->end - ext->start +1 );kfree (ext); }mm/memory.c +1
-1@@ -4225,7 +4225,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)* We're already holding a reference on the page but haven't mapped it * yet. */ -swap_free (entry); +swap_free_nr (entry,1 );if (should_try_to_free_swap (folio, vma, vmf->flags))folio_free_swap (folio);mm/rmap.c +2
-2@@ -1865,7 +1865,7 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, goto walk_done_err; }if (arch_unmap_one (mm, vma, address, pteval) <0 ) { -swap_free (entry); +swap_free_nr (entry,1 );set_pte_at (mm, address, pvmw.pte, pteval); goto walk_done_err; }@@ -1873,7 +1873,7 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,/* See folio_try_share_anon_rmap(): clear PTE first. */ if (anon_exclusive &&folio_try_share_anon_rmap_pte (folio, subpage)) { -swap_free (entry); +swap_free_nr (entry,1 );set_pte_at (mm, address, pvmw.pte, pteval); goto walk_done_err; }mm/shmem.c +2
-2@@ -1836,7 +1836,7 @@ static void shmem_set_folio_swapin_error(struct inode *inode, pgoff_t index,* in shmem_evict_inode(). */ shmem_recalc_inode (inode,-1 ,-1 ); -swap_free (swap); +swap_free_nr (swap,1 ); }/* @@ -1927,7 +1927,7 @@ static int shmem_swapin_folio(struct inode *inode, pgoff_t index,delete_from_swap_cache (folio);folio_mark_dirty (folio); -swap_free (swap); +swap_free_nr (swap,1 );put_swap_device (si); *foliop = folio;mm/swapfile.c +5
-14@@ -1343,19 +1343,6 @@ static void swap_entry_free(struct swap_info_struct *p, swp_entry_t entry)swap_range_free (p, offset,1 ); } -/* -* Caller has made sure that the swap device corresponding to entry -* is still around or has not been recycled. -*/ -void swap_free (swp_entry_t entry) -{ -struct swap_info_struct *p; - - p =_swap_info_get (entry); -if (p) -__swap_entry_free (p, entry); -} -static void cluster_swap_free_nr (struct swap_info_struct *sis,unsigned long offset,int nr_pages) {@@ -1385,6 +1372,10 @@ static void cluster_swap_free_nr(struct swap_info_struct *sis,unlock_cluster_or_swap_info (sis, ci); } +/* +* Caller has made sure that the swap device corresponding to entry +* is still around or has not been recycled. +*/ void swap_free_nr (swp_entry_t entry,int nr_pages) {int nr;@@ -1930,7 +1921,7 @@ static int unuse_pte(struct vm_area_struct *vma, pmd_t *pmd, new_pte =pte_mkuffd_wp (new_pte); setpte:set_pte_at (vma->vm_mm, addr, pte, new_pte); -swap_free (entry); +swap_free_nr (entry,1 ); out:if (pte)pte_unmap_unlock (pte, ptl);
Re: [PATCH v3 2/6] mm: remove swap_free() and always use swap_free_nr()
Quoted 38 lines · Ryan Roberts
Quoted 29 lines · Barry Song
From: Barry Song <[email protected]>
To streamline maintenance efforts, we propose discontinuing the use of
swap_free(). Instead, we can simply invokeswap_free_nr()with nr set to 1. This adjustment offers the advantage of enabling batch processing withinkernel/power/swap.c. Furthermore,swap_free_nr()is designed with a bitmap consisting of only one long, resulting in overhead that can be ignored for cases where nr equals 1.Suggested-by "Huang, Ying" <[email protected]>Signed-off-by Barry Song <[email protected]>Cc "Rafael J. Wysocki" <[email protected]>Cc Pavel Machek <[email protected]>Cc Len Brown <[email protected]>Cc Hugh Dickins <[email protected]>--- include/linux/swap.h | 5 ----- kernel/power/swap.c | 7 +++---- mm/memory.c | 2 +- mm/rmap.c | 4 ++-- mm/shmem.c | 4 ++-- mm/swapfile.c | 19 +++++-------------- 6 files changed, 13 insertions(+), 28 deletions(-)include/linux/swap.h +0
-0include/linux/swap.h extern int add_swap_count_continuation(swp_entry_t, gfp_t); · line 485
Quoted 4 lines
include/linux/swap.h +0
-1@@ -482,7 +482,6 @@ extern int add_swap_count_continuation(swp_entry_t, gfp_t);extern void swap_shmem_alloc (swp_entry_t );extern int swap_duplicate (swp_entry_t );extern int swapcache_prepare (swp_entry_t ); -extern void swap_free (swp_entry_t );I wonder if it would be cleaner to:
Quoted 4 lines
#define
swap_free(entry)swap_free_nr((entry), 1)To save all the churn for the callsites that just want to pass a single entry?
Either way works. It will produce the same machine code. I have a slight inclination to just drop swap_free(entry) API so that it discourages the caller to do a for loop over swap_free().
Chris
Re: [PATCH v3 2/6] mm: remove swap_free() and always use swap_free_nr()
Quoted 3 lines · Chris Li
Either way works. It will produce the same machine code. I have a slight inclination to just drop
swap_free(entry)API so that it discourages the caller to do a for loop overswap_free().
Then just ad the number of entries parameter to swap_free and do away with the separate swap_free_nr.
Re: [PATCH v3 2/6] mm: remove swap_free() and always use swap_free_nr()
Quoted 8 lines · Christoph Hellwig
Quoted 3 lines · Chris Li
Either way works. It will produce the same machine code. I have a slight inclination to just drop
swap_free(entry)API so that it discourages the caller to do a for loop overswap_free().Then just ad the number of entries parameter to
swap_freeand do away with the separateswap_free_nr.
swap_free_nr() isn't separate, after this patch, it is the only one left. there won't be swap_free() any more. it seems you want to directly "rename" it to swap_free()?
Re: [PATCH v3 2/6] mm: remove swap_free() and always use swap_free_nr()
Quoted 3 lines · Barry Song
swap_free_nr()isn't separate, after this patch, it is the only one left. there won't beswap_free()any more. it seems you want to directly "rename" it toswap_free()?
Yes. Avoid the pointless suffix if it is the only variant.
Re: [PATCH v3 2/6] mm: remove swap_free() and always use swap_free_nr()
Quoted 7 lines · Christoph Hellwig
Quoted 3 lines · Barry Song
swap_free_nr()isn't separate, after this patch, it is the only one left. there won't beswap_free()any more. it seems you want to directly "rename" it toswap_free()?Yes. Avoid the pointless suffix if it is the only variant.
well. it seems you are right. We usually use a suffix to differentiate two or more cases, but now, there is only one case left, the suffix seems no longer useful.
one more problem is that free_swap_and_cache_nr() and swap_free_nr() are not quite aligned.
extern void free_swap_and_cache_nr(swp_entry_t entry, int nr);
static inline void free_swap_and_cache(swp_entry_t entry)
{
free_swap_and_cache_nr(entry, 1);
}The problem space is the same. I feel like in that case, we can also drop free_swap_and_cache_nr() and simply add the nr parameter?
Huang, Ying
Reviewed-byRe: [PATCH v3 2/6] mm: remove swap_free() and always use swap_free_nr()
In reply to Ryan RobertsQuoted 37 lines · Ryan Roberts
Quoted 29 lines · Barry Song
From: Barry Song <[email protected]>
To streamline maintenance efforts, we propose discontinuing the use of
swap_free(). Instead, we can simply invokeswap_free_nr()with nr set to 1. This adjustment offers the advantage of enabling batch processing withinkernel/power/swap.c. Furthermore,swap_free_nr()is designed with a bitmap consisting of only one long, resulting in overhead that can be ignored for cases where nr equals 1.Suggested-by "Huang, Ying" <[email protected]>Signed-off-by Barry Song <[email protected]>Cc "Rafael J. Wysocki" <[email protected]>Cc Pavel Machek <[email protected]>Cc Len Brown <[email protected]>Cc Hugh Dickins <[email protected]>--- include/linux/swap.h | 5 ----- kernel/power/swap.c | 7 +++---- mm/memory.c | 2 +- mm/rmap.c | 4 ++-- mm/shmem.c | 4 ++-- mm/swapfile.c | 19 +++++-------------- 6 files changed, 13 insertions(+), 28 deletions(-)include/linux/swap.h +0
-0include/linux/swap.h extern int add_swap_count_continuation(swp_entry_t, gfp_t); · line 485
Quoted 4 lines
include/linux/swap.h +0
-1@@ -482,7 +482,6 @@ extern int add_swap_count_continuation(swp_entry_t, gfp_t);extern void swap_shmem_alloc (swp_entry_t );extern int swap_duplicate (swp_entry_t );extern int swapcache_prepare (swp_entry_t ); -extern void swap_free (swp_entry_t );I wonder if it would be cleaner to:
Quoted 3 lines
#define
swap_free(entry)swap_free_nr((entry), 1)To save all the churn for the callsites that just want to pass a single entry?
I prefer this way. Although I prefer inline functions.
Otherwise, LGTM. Feel free to add
in the future version.
Quoted 127 lines
Quoted 127 lines
include/linux/swap.h +0
-4extern void swap_free_nr (swp_entry_t entry,int nr_pages);extern void swapcache_free_entries (swp_entry_t *entries,int n);extern void free_swap_and_cache_nr (swp_entry_t entry,int nr);@@ -561,10 +560,6 @@ static inline int swapcache_prepare(swp_entry_t swp)return 0 ; } -static inline void swap_free (swp_entry_t swp) -{ -} -static inline void swap_free_nr (swp_entry_t entry,int nr_pages) { }kernel/power/swap.c +3
-4@@ -181,7 +181,7 @@ sector_t alloc_swapdev_block(int swap) offset =swp_offset (get_swap_page_of_type (swap));if (offset) {if (swsusp_extents_insert (offset)) -swap_free (swp_entry (swap, offset)); +swap_free_nr (swp_entry (swap, offset),1 );else return swapdev_block (swap, offset); }@@ -200,12 +200,11 @@ void free_all_swap_pages(int swap)while ((node = swsusp_extents.rb_node)) {struct swsusp_extent *ext; -unsigned long offset; ext = rb_entry(node, struct swsusp_extent, node);rb_erase (node, &swsusp_extents); -for (offset = ext->start; offset <= ext->end; offset++) -swap_free (swp_entry (swap, offset)); +swap_free_nr (swp_entry (swap, ext->start), + ext->end - ext->start +1 );kfree (ext); }mm/memory.c +1
-1@@ -4225,7 +4225,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)* We're already holding a reference on the page but haven't mapped it * yet. */ -swap_free (entry); +swap_free_nr (entry,1 );if (should_try_to_free_swap (folio, vma, vmf->flags))folio_free_swap (folio);mm/rmap.c +2
-2@@ -1865,7 +1865,7 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, goto walk_done_err; }if (arch_unmap_one (mm, vma, address, pteval) <0 ) { -swap_free (entry); +swap_free_nr (entry,1 );set_pte_at (mm, address, pvmw.pte, pteval); goto walk_done_err; }@@ -1873,7 +1873,7 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,/* See folio_try_share_anon_rmap(): clear PTE first. */ if (anon_exclusive &&folio_try_share_anon_rmap_pte (folio, subpage)) { -swap_free (entry); +swap_free_nr (entry,1 );set_pte_at (mm, address, pvmw.pte, pteval); goto walk_done_err; }mm/shmem.c +2
-2@@ -1836,7 +1836,7 @@ static void shmem_set_folio_swapin_error(struct inode *inode, pgoff_t index,* in shmem_evict_inode(). */ shmem_recalc_inode (inode,-1 ,-1 ); -swap_free (swap); +swap_free_nr (swap,1 ); }/* @@ -1927,7 +1927,7 @@ static int shmem_swapin_folio(struct inode *inode, pgoff_t index,delete_from_swap_cache (folio);folio_mark_dirty (folio); -swap_free (swap); +swap_free_nr (swap,1 );put_swap_device (si); *foliop = folio;mm/swapfile.c +0
-13@@ -1343,19 +1343,6 @@ static void swap_entry_free(struct swap_info_struct *p, swp_entry_t entry)swap_range_free (p, offset,1 ); } -/* -* Caller has made sure that the swap device corresponding to entry -* is still around or has not been recycled. -*/ -void swap_free (swp_entry_t entry) -{ -struct swap_info_struct *p; - - p =_swap_info_get (entry); -if (p) -__swap_entry_free (p, entry); -} -static void cluster_swap_free_nr (struct swap_info_struct *sis,unsigned long offset,int nr_pages) {@@ -1385,6 +1372,10 @@ static void cluster_swap_free_nr(struct swap_info_struct *sis,unlock_cluster_or_swap_info (sis, ci); }
Quoted 16 lines
Quoted 8 lines
mm/swapfile.c +4
-0+/* +* Caller has made sure that the swap device corresponding to entry +* is still around or has not been recycled. +*/ void swap_free_nr (swp_entry_t entry,int nr_pages) {int nr;mm/swapfile.c static int unuse_pte(struct vm_area_struct *vma, pmd_t *pmd, · line 1927
Quoted 8 lines
mm/swapfile.c +1
-1@@ -1930,7 +1921,7 @@ static int unuse_pte(struct vm_area_struct *vma, pmd_t *pmd, new_pte =pte_mkuffd_wp (new_pte); setpte:set_pte_at (vma->vm_mm, addr, pte, new_pte); -swap_free (entry); +swap_free_nr (entry,1 ); out:if (pte)pte_unmap_unlock (pte, ptl);
--
Best Regards,
Huang, Ying
Re: [PATCH v3 2/6] mm: remove swap_free() and always use swap_free_nr()
Quoted 40 lines · Ying
Quoted 37 lines · Ryan Roberts
Quoted 29 lines · Barry Song
From: Barry Song <[email protected]>
To streamline maintenance efforts, we propose discontinuing the use of
swap_free(). Instead, we can simply invokeswap_free_nr()with nr set to 1. This adjustment offers the advantage of enabling batch processing withinkernel/power/swap.c. Furthermore,swap_free_nr()is designed with a bitmap consisting of only one long, resulting in overhead that can be ignored for cases where nr equals 1.Suggested-by "Huang, Ying" <[email protected]>Signed-off-by Barry Song <[email protected]>Cc "Rafael J. Wysocki" <[email protected]>Cc Pavel Machek <[email protected]>Cc Len Brown <[email protected]>Cc Hugh Dickins <[email protected]>--- include/linux/swap.h | 5 ----- kernel/power/swap.c | 7 +++---- mm/memory.c | 2 +- mm/rmap.c | 4 ++-- mm/shmem.c | 4 ++-- mm/swapfile.c | 19 +++++-------------- 6 files changed, 13 insertions(+), 28 deletions(-)include/linux/swap.h +0
-0include/linux/swap.h extern int add_swap_count_continuation(swp_entry_t, gfp_t); · line 485
Quoted 4 lines
include/linux/swap.h +0
-1@@ -482,7 +482,6 @@ extern int add_swap_count_continuation(swp_entry_t, gfp_t);extern void swap_shmem_alloc (swp_entry_t );extern int swap_duplicate (swp_entry_t );extern int swapcache_prepare (swp_entry_t ); -extern void swap_free (swp_entry_t );I wonder if it would be cleaner to:
Quoted 5 lines
Quoted 3 lines
#define
swap_free(entry)swap_free_nr((entry), 1)To save all the churn for the callsites that just want to pass a single entry?
I prefer this way. Although I prefer inline functions.
Yes, using static inline is preferable. I've recently submitted a checkpatch/codestyle for this, which can be found at:
https://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm.git/commit/?h=mm-everything&id=39c58d5ed036
https://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm.git/commit/?h=mm-everything&id=8379bf0b0e1f5
Using static inline aligns with the established rule.
Quoted 4 lines
Otherwise, LGTM. Feel free to add
Reviewed-by "Huang, Ying" <[email protected]>
Thanks!
Quoted 2 lines
in the future version.
I believe Christoph's vote leans towards simply removing swap_free_nr and renaming it to swap_free, while adding a new parameter as follows.
void swap_free(swp_entry_t entry, int nr);
{
}
now I see Ryan and you prefer
static inline swap_free()
{
swap_free_nr(...., 1)
}Chris slightly favors discouraging the use of swap_free() without the new parameter. Removing swap_free() can address this concern.
It seems that maintaining swap_free() and having it call swap_free_nr() with a default value of 1 received the most support.
To align with free_swap_and_cache() and free_swap_and_cache_nr(), I'll proceed with the "static inline" approach in the new version. Please voice any objections you may have, Christoph, Chris.
Quoted 140 lines
Quoted 140 lines
Quoted 140 lines
include/linux/swap.h +0
-4extern void swap_free_nr (swp_entry_t entry,int nr_pages);extern void swapcache_free_entries (swp_entry_t *entries,int n);extern void free_swap_and_cache_nr (swp_entry_t entry,int nr);@@ -561,10 +560,6 @@ static inline int swapcache_prepare(swp_entry_t swp)return 0 ; } -static inline void swap_free (swp_entry_t swp) -{ -} -static inline void swap_free_nr (swp_entry_t entry,int nr_pages) { }kernel/power/swap.c +3
-4@@ -181,7 +181,7 @@ sector_t alloc_swapdev_block(int swap) offset =swp_offset (get_swap_page_of_type (swap));if (offset) {if (swsusp_extents_insert (offset)) -swap_free (swp_entry (swap, offset)); +swap_free_nr (swp_entry (swap, offset),1 );else return swapdev_block (swap, offset); }@@ -200,12 +200,11 @@ void free_all_swap_pages(int swap)while ((node = swsusp_extents.rb_node)) {struct swsusp_extent *ext; -unsigned long offset; ext = rb_entry(node, struct swsusp_extent, node);rb_erase (node, &swsusp_extents); -for (offset = ext->start; offset <= ext->end; offset++) -swap_free (swp_entry (swap, offset)); +swap_free_nr (swp_entry (swap, ext->start), + ext->end - ext->start +1 );kfree (ext); }mm/memory.c +1
-1@@ -4225,7 +4225,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)* We're already holding a reference on the page but haven't mapped it * yet. */ -swap_free (entry); +swap_free_nr (entry,1 );if (should_try_to_free_swap (folio, vma, vmf->flags))folio_free_swap (folio);mm/rmap.c +2
-2@@ -1865,7 +1865,7 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, goto walk_done_err; }if (arch_unmap_one (mm, vma, address, pteval) <0 ) { -swap_free (entry); +swap_free_nr (entry,1 );set_pte_at (mm, address, pvmw.pte, pteval); goto walk_done_err; }@@ -1873,7 +1873,7 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,/* See folio_try_share_anon_rmap(): clear PTE first. */ if (anon_exclusive &&folio_try_share_anon_rmap_pte (folio, subpage)) { -swap_free (entry); +swap_free_nr (entry,1 );set_pte_at (mm, address, pvmw.pte, pteval); goto walk_done_err; }mm/shmem.c +2
-2@@ -1836,7 +1836,7 @@ static void shmem_set_folio_swapin_error(struct inode *inode, pgoff_t index,* in shmem_evict_inode(). */ shmem_recalc_inode (inode,-1 ,-1 ); -swap_free (swap); +swap_free_nr (swap,1 ); }/* @@ -1927,7 +1927,7 @@ static int shmem_swapin_folio(struct inode *inode, pgoff_t index,delete_from_swap_cache (folio);folio_mark_dirty (folio); -swap_free (swap); +swap_free_nr (swap,1 );put_swap_device (si); *foliop = folio;mm/swapfile.c +4
-14@@ -1343,19 +1343,6 @@ static void swap_entry_free(struct swap_info_struct *p, swp_entry_t entry)swap_range_free (p, offset,1 ); } -/* -* Caller has made sure that the swap device corresponding to entry -* is still around or has not been recycled. -*/ -void swap_free (swp_entry_t entry) -{ -struct swap_info_struct *p; - - p =_swap_info_get (entry); -if (p) -__swap_entry_free (p, entry); -} -static void cluster_swap_free_nr (struct swap_info_struct *sis,unsigned long offset,int nr_pages) {@@ -1385,6 +1372,10 @@ static void cluster_swap_free_nr(struct swap_info_struct *sis,unlock_cluster_or_swap_info (sis, ci); } +/* +* Caller has made sure that the swap device corresponding to entry +* is still around or has not been recycled. +*/ void swap_free_nr (swp_entry_t entry,int nr_pages) {int nr;@@ -1930,7 +1921,7 @@ static int unuse_pte(struct vm_area_struct *vma, pmd_t *pmd, new_pte =pte_mkuffd_wp (new_pte); setpte:set_pte_at (vma->vm_mm, addr, pte, new_pte); -swap_free (entry);
Quoted 8 lines
Quoted 4 lines
mm/swapfile.c static int unuse_pte(struct vm_area_struct *vma, pmd_t *pmd, · line 1927
Quoted 4 lines
mm/swapfile.c +1
-0+swap_free_nr (entry,1 ); out:if (pte)pte_unmap_unlock (pte, ptl);--
Best Regards,
Huang, Ying
Thanks
Barry
Re: [PATCH v3 2/6] mm: remove swap_free() and always use swap_free_nr()
Quoted 41 lines · Barry Song
Quoted 40 lines · Ying
Quoted 37 lines · Ryan Roberts
Quoted 29 lines · Barry Song
From: Barry Song <[email protected]>
To streamline maintenance efforts, we propose discontinuing the use of
swap_free(). Instead, we can simply invokeswap_free_nr()with nr set to 1. This adjustment offers the advantage of enabling batch processing withinkernel/power/swap.c. Furthermore,swap_free_nr()is designed with a bitmap consisting of only one long, resulting in overhead that can be ignored for cases where nr equals 1.Suggested-by "Huang, Ying" <[email protected]>Signed-off-by Barry Song <[email protected]>Cc "Rafael J. Wysocki" <[email protected]>Cc Pavel Machek <[email protected]>Cc Len Brown <[email protected]>Cc Hugh Dickins <[email protected]>--- include/linux/swap.h | 5 ----- kernel/power/swap.c | 7 +++---- mm/memory.c | 2 +- mm/rmap.c | 4 ++-- mm/shmem.c | 4 ++-- mm/swapfile.c | 19 +++++-------------- 6 files changed, 13 insertions(+), 28 deletions(-)include/linux/swap.h +0
-0include/linux/swap.h extern int add_swap_count_continuation(swp_entry_t, gfp_t); · line 485
Quoted 4 lines
include/linux/swap.h +0
-1@@ -482,7 +482,6 @@ extern int add_swap_count_continuation(swp_entry_t, gfp_t);extern void swap_shmem_alloc (swp_entry_t );extern int swap_duplicate (swp_entry_t );extern int swapcache_prepare (swp_entry_t ); -extern void swap_free (swp_entry_t );I wonder if it would be cleaner to:
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
#define
swap_free(entry)swap_free_nr((entry), 1)To save all the churn for the callsites that just want to pass a single entry?
I prefer this way. Although I prefer inline functions.
Yes, I agree inline function is the better approach.
Quoted 35 lines
Yes, using static inline is preferable. I've recently submitted a checkpatch/codestyle for this, which can be found at:
https://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm.git/commit/?h=mm-everything&id=39c58d5ed036
https://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm.git/commit/?h=mm-everything&id=8379bf0b0e1f5Using static inline aligns with the established rule.
Quoted 4 lines
Otherwise, LGTM. Feel free to add
Reviewed-by "Huang, Ying" <[email protected]>Thanks!
Quoted 2 lines
in the future version.
I believe Christoph's vote leans towards simply removing
swap_free_nrand renaming it toswap_free, while adding a new parameter as follows.void
swap_free(swp_entry_t entry, int nr);
{
}now I see Ryan and you prefer
static inline
swap_free(){ swap_free_nr(...., 1) }Chris slightly favors discouraging the use of
swap_free()without the new parameter. Removingswap_free()can address this concern.
Quoted 6 lines
It seems that maintaining
swap_free()and having it callswap_free_nr()with a default value of 1 received the most support.To align with
free_swap_and_cache()andfree_swap_and_cache_nr(), I'll proceed with the "static inline" approach in the new version. Please voice any objections you may have, Christoph, Chris.
I'm happy with either route. If you end up adding a nr param to swap_free() then it would also be good to give free_swap_and_cache_nr() the same treatment.
Quoted 152 lines
Quoted 141 lines
Quoted 141 lines
Quoted 141 lines
include/linux/swap.h +0
-4extern void swap_free_nr (swp_entry_t entry,int nr_pages);extern void swapcache_free_entries (swp_entry_t *entries,int n);extern void free_swap_and_cache_nr (swp_entry_t entry,int nr);@@ -561,10 +560,6 @@ static inline int swapcache_prepare(swp_entry_t swp)return 0 ; } -static inline void swap_free (swp_entry_t swp) -{ -} -static inline void swap_free_nr (swp_entry_t entry,int nr_pages) { }kernel/power/swap.c +3
-4@@ -181,7 +181,7 @@ sector_t alloc_swapdev_block(int swap) offset =swp_offset (get_swap_page_of_type (swap));if (offset) {if (swsusp_extents_insert (offset)) -swap_free (swp_entry (swap, offset)); +swap_free_nr (swp_entry (swap, offset),1 );else return swapdev_block (swap, offset); }@@ -200,12 +200,11 @@ void free_all_swap_pages(int swap)while ((node = swsusp_extents.rb_node)) {struct swsusp_extent *ext; -unsigned long offset; ext = rb_entry(node, struct swsusp_extent, node);rb_erase (node, &swsusp_extents); -for (offset = ext->start; offset <= ext->end; offset++) -swap_free (swp_entry (swap, offset)); +swap_free_nr (swp_entry (swap, ext->start), + ext->end - ext->start +1 );kfree (ext); }mm/memory.c +1
-1@@ -4225,7 +4225,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)* We're already holding a reference on the page but haven't mapped it * yet. */ -swap_free (entry); +swap_free_nr (entry,1 );if (should_try_to_free_swap (folio, vma, vmf->flags))folio_free_swap (folio);mm/rmap.c +2
-2@@ -1865,7 +1865,7 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, goto walk_done_err; }if (arch_unmap_one (mm, vma, address, pteval) <0 ) { -swap_free (entry); +swap_free_nr (entry,1 );set_pte_at (mm, address, pvmw.pte, pteval); goto walk_done_err; }@@ -1873,7 +1873,7 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,/* See folio_try_share_anon_rmap(): clear PTE first. */ if (anon_exclusive &&folio_try_share_anon_rmap_pte (folio, subpage)) { -swap_free (entry); +swap_free_nr (entry,1 );set_pte_at (mm, address, pvmw.pte, pteval); goto walk_done_err; }mm/shmem.c +2
-2@@ -1836,7 +1836,7 @@ static void shmem_set_folio_swapin_error(struct inode *inode, pgoff_t index,* in shmem_evict_inode(). */ shmem_recalc_inode (inode,-1 ,-1 ); -swap_free (swap); +swap_free_nr (swap,1 ); }/* @@ -1927,7 +1927,7 @@ static int shmem_swapin_folio(struct inode *inode, pgoff_t index,delete_from_swap_cache (folio);folio_mark_dirty (folio); -swap_free (swap); +swap_free_nr (swap,1 );put_swap_device (si); *foliop = folio;mm/swapfile.c +4
-14@@ -1343,19 +1343,6 @@ static void swap_entry_free(struct swap_info_struct *p, swp_entry_t entry)swap_range_free (p, offset,1 ); } -/* -* Caller has made sure that the swap device corresponding to entry -* is still around or has not been recycled. -*/ -void swap_free (swp_entry_t entry) -{ -struct swap_info_struct *p; - - p =_swap_info_get (entry); -if (p) -__swap_entry_free (p, entry); -} -static void cluster_swap_free_nr (struct swap_info_struct *sis,unsigned long offset,int nr_pages) {@@ -1385,6 +1372,10 @@ static void cluster_swap_free_nr(struct swap_info_struct *sis,unlock_cluster_or_swap_info (sis, ci); } +/* +* Caller has made sure that the swap device corresponding to entry +* is still around or has not been recycled. +*/ void swap_free_nr (swp_entry_t entry,int nr_pages) {int nr;@@ -1930,7 +1921,7 @@ static int unuse_pte(struct vm_area_struct *vma, pmd_t *pmd, new_pte =pte_mkuffd_wp (new_pte); setpte:set_pte_at (vma->vm_mm, addr, pte, new_pte); -swap_free (entry);Quoted 8 lines
Quoted 4 lines
mm/swapfile.c static int unuse_pte(struct vm_area_struct *vma, pmd_t *pmd, · line 1927
Quoted 4 lines
mm/swapfile.c +1
-0+swap_free_nr (entry,1 ); out:if (pte)pte_unmap_unlock (pte, ptl);--
Best Regards,
Huang, YingThanks
Barry
PATCH v3 3/6 mm: introduce pte_move_swp_offset() helper which can move offset bidirectionally
In reply to Barry SongFrom: Barry Song <[email protected]>
There could arise a necessity to obtain the first pte_t from a swap pte_t located in the middle. For instance, this may occur within the context of do_swap_page(), where a page fault can potentially occur in any PTE of a large folio. To address this, the following patch introduces pte_move_swp_offset(), a function capable of bidirectional movement by a specified delta argument. Consequently, pte_increment_swp_offset() will directly invoke it with delta = 1.
--- mm/internal.h | 25 +++++++++++++++++++++---- 1 file changed, 21 insertions(+), 4 deletions(-)
mm/internal.h +21 -4
@@ -211,18 +211,21 @@ static inline int folio_pte_batch(struct folio *folio, unsigned long addr, }/** -* pte_next_swp_offset - Increment the swap entry offset field of a swap pte. +* pte_move_swp_offset - Move the swap entry offset field of a swap pte +* forward or backward by delta * @pte: The initial pte state; is_swap_pte(pte) must be true and * non_swap_entry() must be false. +* @delta: The direction and the offset we are moving; forward if delta +* is positive; backward if delta is negative * -* Increments the swap offset, while maintaining all other fields, including +* Moves the swap offset, while maintaining all other fields, including * swap type, and any swp pte bits. The resulting pte is returned. */ -static inline pte_t pte_next_swp_offset (pte_t pte) +static inline pte_t pte_move_swp_offset (pte_t pte,long delta) {swp_entry_t entry =pte_to_swp_entry (pte);pte_t new =__swp_entry_to_pte (__swp_entry (swp_type (entry), - (swp_offset (entry) +1 ))); + (swp_offset (entry) + delta)));if (pte_swp_soft_dirty (pte)) new =pte_swp_mksoft_dirty (new);
@@ -234,6 +237,20 @@ static inline pte_t pte_next_swp_offset(pte_t pte)return new; } + +/** +* pte_next_swp_offset - Increment the swap entry offset field of a swap pte. +* @pte: The initial pte state; is_swap_pte(pte) must be true and +* non_swap_entry() must be false. +* +* Increments the swap offset, while maintaining all other fields, including +* swap type, and any swp pte bits. The resulting pte is returned. +*/ +static inline pte_t pte_next_swp_offset (pte_t pte) +{ +return pte_move_swp_offset (pte,1 ); +} +/** * swap_pte_batch - detect a PTE batch for a set of contiguous swap entries * @start_ptep: Page table pointer for the first entry.
Signature · 1 line
2.34.1
Re: [PATCH v3 3/6] mm: introduce pte_move_swp_offset() helper which can move offset bidirectionally
Quoted 8 lines · Barry Song
From: Barry Song <[email protected]>
There could arise a necessity to obtain the first
pte_tfrom a swappte_tlocated in the middle. For instance, this may occur within the context ofdo_swap_page(), where a page fault can potentially occur in any PTE of a large folio. To address this, the following patch introducespte_move_swp_offset(), a function capable of bidirectional movement by a specified delta argument. Consequently,pte_increment_swp_offset()
You mean pte_next_swp_offset()?
Quoted 20 lines
will directly invoke it with delta = 1.
Suggested-by "Huang, Ying" <[email protected]>Signed-off-by Barry Song <[email protected]>--- mm/internal.h | 25 +++++++++++++++++++++---- 1 file changed, 21 insertions(+), 4 deletions(-)mm/internal.h +2
-1@@ -211,18 +211,21 @@ static inline int folio_pte_batch(struct folio *folio, unsigned long addr, }/** -* pte_next_swp_offset - Increment the swap entry offset field of a swap pte. +* pte_move_swp_offset - Move the swap entry offset field of a swap pte +* forward or backward by delta * @pte: The initial pte state; is_swap_pte(pte) must be true and
mm/internal.h static inline int folio_pte_batch(struct folio *folio, unsigned long addr, · line 224
Quoted 10 lines
mm/internal.h +4
-2* non_swap_entry() must be false. +* @delta: The direction and the offset we are moving; forward if delta +* is positive; backward if delta is negative * -* Increments the swap offset, while maintaining all other fields, including +* Moves the swap offset, while maintaining all other fields, including * swap type, and any swp pte bits. The resulting pte is returned. */ -static inline pte_t pte_next_swp_offset (pte_t pte) +static inline pte_t pte_move_swp_offset (pte_t pte,long delta)
We have equivalent functions for pfn:
pte_next_pfn()
pte_advance_pfn()
Although the latter takes an unsigned long and only moves forward currently. I wonder if it makes sense to have their naming and semantics match? i.e. change pte_advance_pfn() to pte_move_pfn() and let it move backwards too.
I guess we don't have a need for that and it adds more churn.
Anyway:
Quoted 29 lines
mm/internal.h +15
-1{swp_entry_t entry =pte_to_swp_entry (pte);pte_t new =__swp_entry_to_pte (__swp_entry (swp_type (entry), - (swp_offset (entry) +1 ))); + (swp_offset (entry) + delta)));if (pte_swp_soft_dirty (pte)) new =pte_swp_mksoft_dirty (new);@@ -234,6 +237,20 @@ static inline pte_t pte_next_swp_offset(pte_t pte)return new; } + +/** +* pte_next_swp_offset - Increment the swap entry offset field of a swap pte. +* @pte: The initial pte state; is_swap_pte(pte) must be true and +* non_swap_entry() must be false. +* +* Increments the swap offset, while maintaining all other fields, including +* swap type, and any swp pte bits. The resulting pte is returned. +*/ +static inline pte_t pte_next_swp_offset (pte_t pte) +{ +return pte_move_swp_offset (pte,1 ); +} +/** * swap_pte_batch - detect a PTE batch for a set of contiguous swap entries * @start_ptep: Page table pointer for the first entry.
Re: [PATCH v3 3/6] mm: introduce pte_move_swp_offset() helper which can move offset bidirectionally
Quoted 4 lines · Ryan Roberts
Quoted 2 lines · Barry Song
From: Barry Song <[email protected]>
Quoted 8 lines
Quoted 6 lines
There could arise a necessity to obtain the first
pte_tfrom a swappte_tlocated in the middle. For instance, this may occur within the context ofdo_swap_page(), where a page fault can potentially occur in any PTE of a large folio. To address this, the following patch introducespte_move_swp_offset(), a function capable of bidirectional movement by a specified delta argument. Consequently,pte_increment_swp_offset()You mean
pte_next_swp_offset()?
yes.
Quoted 34 lines
Quoted 21 lines
will directly invoke it with delta = 1.
Suggested-by "Huang, Ying" <[email protected]>Signed-off-by Barry Song <[email protected]>--- mm/internal.h | 25 +++++++++++++++++++++---- 1 file changed, 21 insertions(+), 4 deletions(-)mm/internal.h +2
-1@@ -211,18 +211,21 @@ static inline int folio_pte_batch(struct folio *folio, unsigned long addr, }/** -* pte_next_swp_offset - Increment the swap entry offset field of a swap pte. +* pte_move_swp_offset - Move the swap entry offset field of a swap pte +* forward or backward by delta * @pte: The initial pte state; is_swap_pte(pte) must be true and mm/internal.h static inline int folio_pte_batch(struct folio *folio, unsigned long addr, · line 224
Quoted 10 lines
mm/internal.h +4
-2* non_swap_entry() must be false. +* @delta: The direction and the offset we are moving; forward if delta +* is positive; backward if delta is negative * -* Increments the swap offset, while maintaining all other fields, including +* Moves the swap offset, while maintaining all other fields, including * swap type, and any swp pte bits. The resulting pte is returned. */ -static inline pte_t pte_next_swp_offset (pte_t pte) +static inline pte_t pte_move_swp_offset (pte_t pte,long delta)We have equivalent functions for pfn:
Quoted 8 lines
pte_next_pfn()
pte_advance_pfn()Although the latter takes an unsigned long and only moves forward currently. I wonder if it makes sense to have their naming and semantics match? i.e. change
pte_advance_pfn()topte_move_pfn()and let it move backwards too.I guess we don't have a need for that and it adds more churn.
we might have a need in the below case.
A forks B, then A and B share large folios. B unmap/exit, then large folios of process
A become single-mapped.
Right now, while writing A's folios, we are CoWing A's large folios into many small
folios. I believe we can reuse the entire large folios instead of doing nr_pages CoW and page faults.
In this case, we might want to get the first PTE from vmf->pte.
Another case, might be
A forks B, and we write either A or B, we might CoW an entire large folios instead
CoWing nr_pages small folios.
case 1 seems more useful, I might have a go after some days. then we might see pte_move_pfn().
Quoted 4 lines
Anyway:
Reviewed-by Ryan Roberts <[email protected]>
thanks!
Quoted 23 lines
Quoted 23 lines
mm/internal.h +10
-1{swp_entry_t entry =pte_to_swp_entry (pte);pte_t new =__swp_entry_to_pte (__swp_entry (swp_type (entry), - (swp_offset (entry) +1 ))); + (swp_offset (entry) + delta)));if (pte_swp_soft_dirty (pte)) new =pte_swp_mksoft_dirty (new);@@ -234,6 +237,20 @@ static inline pte_t pte_next_swp_offset(pte_t pte)return new; } + +/** +* pte_next_swp_offset - Increment the swap entry offset field of a swap pte. +* @pte: The initial pte state; is_swap_pte(pte) must be true and +* non_swap_entry() must be false. +* +* Increments the swap offset, while maintaining all other fields, including +* swap type, and any swp pte bits. The resulting pte is returned. +*/
Quoted 9 lines
mm/internal.h static inline pte_t pte_next_swp_offset(pte_t pte) · line 256
Quoted 8 lines
mm/internal.h +5
-0+static inline pte_t pte_next_swp_offset (pte_t pte) +{ +return pte_move_swp_offset (pte,1 ); +} +/** * swap_pte_batch - detect a PTE batch for a set of contiguous swap entries * @start_ptep: Page table pointer for the first entry.
Barry
Re: [PATCH v3 3/6] mm: introduce pte_move_swp_offset() helper which can move offset bidirectionally
Quoted 59 lines · Barry Song
Quoted 4 lines · Ryan Roberts
Quoted 2 lines · Barry Song
From: Barry Song <[email protected]>
Quoted 8 lines
Quoted 6 lines
There could arise a necessity to obtain the first
pte_tfrom a swappte_tlocated in the middle. For instance, this may occur within the context ofdo_swap_page(), where a page fault can potentially occur in any PTE of a large folio. To address this, the following patch introducespte_move_swp_offset(), a function capable of bidirectional movement by a specified delta argument. Consequently,pte_increment_swp_offset()You mean
pte_next_swp_offset()?yes.
Quoted 34 lines
Quoted 21 lines
will directly invoke it with delta = 1.
Suggested-by "Huang, Ying" <[email protected]>Signed-off-by Barry Song <[email protected]>--- mm/internal.h | 25 +++++++++++++++++++++---- 1 file changed, 21 insertions(+), 4 deletions(-)mm/internal.h +2
-1@@ -211,18 +211,21 @@ static inline int folio_pte_batch(struct folio *folio, unsigned long addr, }/** -* pte_next_swp_offset - Increment the swap entry offset field of a swap pte. +* pte_move_swp_offset - Move the swap entry offset field of a swap pte +* forward or backward by delta * @pte: The initial pte state; is_swap_pte(pte) must be true and mm/internal.h static inline int folio_pte_batch(struct folio *folio, unsigned long addr, · line 224
Quoted 10 lines
mm/internal.h +4
-2* non_swap_entry() must be false. +* @delta: The direction and the offset we are moving; forward if delta +* is positive; backward if delta is negative * -* Increments the swap offset, while maintaining all other fields, including +* Moves the swap offset, while maintaining all other fields, including * swap type, and any swp pte bits. The resulting pte is returned. */ -static inline pte_t pte_next_swp_offset (pte_t pte) +static inline pte_t pte_move_swp_offset (pte_t pte,long delta)We have equivalent functions for pfn:
Quoted 8 lines
pte_next_pfn() pte_advance_pfn()Although the latter takes an unsigned long and only moves forward currently. I wonder if it makes sense to have their naming and semantics match? i.e. change
pte_advance_pfn()topte_move_pfn()and let it move backwards too.I guess we don't have a need for that and it adds more churn.
Quoted 9 lines
we might have a need in the below case.
A forks B, then A and B share large folios. B unmap/exit, then large folios of process
A become single-mapped.
Right now, while writing A's folios, we are CoWing A's large folios into many small
folios. I believe we can reuse the entire large folios instead of doingnr_pagesCoW and page faults.
In this case, we might want to get the first PTE fromvmf->pte.
Once we have COW reuse for large folios in place (I think you know that I am working on that), it might make sense to "COW-reuse around", meaning we look if some neighboring PTEs map the same large folio and map them writable as well. But if it's really worth it, increasing page fault latency, is to be decided separately.
Quoted 8 lines
Another case, might be
A forks B, and we write either A or B, we might CoW an entire large folios instead
CoWingnr_pagessmall folios.case 1 seems more useful, I might have a go after some days. then we might see
pte_move_pfn().
pte_move_pfn() does sound odd to me. It might not be required to implement the optimization described above. (it's easier to simply read another PTE, check if it maps the same large folio, and to batch from there)
Signature · 3 lines
Cheers, David / dhildenb
Re: [PATCH v3 3/6] mm: introduce pte_move_swp_offset() helper which can move offset bidirectionally
Quoted 71 lines · David Hildenbrand
Quoted 59 lines · Barry Song
Quoted 4 lines · Ryan Roberts
Quoted 2 lines · Barry Song
From: Barry Song <[email protected]>
Quoted 8 lines
Quoted 6 lines
There could arise a necessity to obtain the first
pte_tfrom a swappte_tlocated in the middle. For instance, this may occur within the context ofdo_swap_page(), where a page fault can potentially occur in any PTE of a large folio. To address this, the following patch introducespte_move_swp_offset(), a function capable of bidirectional movement by a specified delta argument. Consequently,pte_increment_swp_offset()You mean
pte_next_swp_offset()?yes.
Quoted 34 lines
Quoted 21 lines
will directly invoke it with delta = 1.
Suggested-by "Huang, Ying" <[email protected]>Signed-off-by Barry Song <[email protected]>--- mm/internal.h | 25 +++++++++++++++++++++---- 1 file changed, 21 insertions(+), 4 deletions(-)mm/internal.h +2
-1@@ -211,18 +211,21 @@ static inline int folio_pte_batch(struct folio *folio, unsigned long addr, }/** -* pte_next_swp_offset - Increment the swap entry offset field of a swap pte. +* pte_move_swp_offset - Move the swap entry offset field of a swap pte +* forward or backward by delta * @pte: The initial pte state; is_swap_pte(pte) must be true and mm/internal.h static inline int folio_pte_batch(struct folio *folio, unsigned long addr, · line 224
Quoted 10 lines
mm/internal.h +4
-2* non_swap_entry() must be false. +* @delta: The direction and the offset we are moving; forward if delta +* is positive; backward if delta is negative * -* Increments the swap offset, while maintaining all other fields, including +* Moves the swap offset, while maintaining all other fields, including * swap type, and any swp pte bits. The resulting pte is returned. */ -static inline pte_t pte_next_swp_offset (pte_t pte) +static inline pte_t pte_move_swp_offset (pte_t pte,long delta)We have equivalent functions for pfn:
Quoted 8 lines
pte_next_pfn() pte_advance_pfn()Although the latter takes an unsigned long and only moves forward currently. I wonder if it makes sense to have their naming and semantics match? i.e. change
pte_advance_pfn()topte_move_pfn()and let it move backwards too.I guess we don't have a need for that and it adds more churn.
Quoted 9 lines
we might have a need in the below case.
A forks B, then A and B share large folios. B unmap/exit, then large folios of process
A become single-mapped.
Right now, while writing A's folios, we are CoWing A's large folios into many small
folios. I believe we can reuse the entire large folios instead of doingnr_pagesCoW and page faults.
In this case, we might want to get the first PTE fromvmf->pte.
Quoted 2 lines
Once we have COW reuse for large folios in place (I think you know that I am working on that), it might make sense to "COW-reuse around",
TBH, I don't know if you are working on that. please Cc me next time :-)
Quoted 3 lines
meaning we look if some neighboring PTEs map the same large folio and map them writable as well. But if it's really worth it, increasing page fault latency, is to be decided separately.
On the other hand, we eliminate latency for the remaining nr_pages - 1 PTEs. Perhaps we can discover a more cost-effective method to signify that a large folio is probably singly mapped? and only call "multi-PTEs" reuse while that condition is true in PF and avoid increasing latency always?
Quoted 8 lines
Quoted 8 lines
Another case, might be
A forks B, and we write either A or B, we might CoW an entire large folios instead
CoWingnr_pagessmall folios.
Quoted 6 lines
Quoted 2 lines
case 1 seems more useful, I might have a go after some days. then we might see
pte_move_pfn().
pte_move_pfn()does sound odd to me. It might not be required to implement the optimization described above. (it's easier to simply read another PTE, check if it maps the same large folio, and to batch from there)
It appears that your proposal suggests potential reusability as follows: if we have a large folio containing 16 PTEs, you might consider reusing only 4 by examining PTEs "around" but not necessarily all 16 PTEs. please correct me if my understanding is wrong.
Initially, my idea was to obtain the first PTE using pte_move_pfn() and then utilize folio_pte_batch() with the first PTE as arguments to ensure consistency in nr_pages, thus enabling complete reuse of the whole folio.
Quoted 4 lines
--
Cheers,David / dhildenb
Thanks
Barry
Re: [PATCH v3 3/6] mm: introduce pte_move_swp_offset() helper which can move offset bidirectionally
Quoted 72 lines · Barry Song
Quoted 71 lines · David Hildenbrand
Quoted 68 lines · Barry Song
Quoted 4 lines · Ryan Roberts
Quoted 2 lines · Barry Song
From: Barry Song <[email protected]>
Quoted 8 lines
Quoted 6 lines
There could arise a necessity to obtain the first
pte_tfrom a swappte_tlocated in the middle. For instance, this may occur within the context ofdo_swap_page(), where a page fault can potentially occur in any PTE of a large folio. To address this, the following patch introducespte_move_swp_offset(), a function capable of bidirectional movement by a specified delta argument. Consequently,pte_increment_swp_offset()You mean
pte_next_swp_offset()?yes.
Quoted 34 lines
Quoted 21 lines
will directly invoke it with delta = 1.
Suggested-by "Huang, Ying" <[email protected]>Signed-off-by Barry Song <[email protected]>--- mm/internal.h | 25 +++++++++++++++++++++---- 1 file changed, 21 insertions(+), 4 deletions(-)mm/internal.h +2
-1@@ -211,18 +211,21 @@ static inline int folio_pte_batch(struct folio *folio, unsigned long addr, }/** -* pte_next_swp_offset - Increment the swap entry offset field of a swap pte. +* pte_move_swp_offset - Move the swap entry offset field of a swap pte +* forward or backward by delta * @pte: The initial pte state; is_swap_pte(pte) must be true and mm/internal.h static inline int folio_pte_batch(struct folio *folio, unsigned long addr, · line 224
Quoted 10 lines
mm/internal.h +4
-2* non_swap_entry() must be false. +* @delta: The direction and the offset we are moving; forward if delta +* is positive; backward if delta is negative * -* Increments the swap offset, while maintaining all other fields, including +* Moves the swap offset, while maintaining all other fields, including * swap type, and any swp pte bits. The resulting pte is returned. */ -static inline pte_t pte_next_swp_offset (pte_t pte) +static inline pte_t pte_move_swp_offset (pte_t pte,long delta)We have equivalent functions for pfn:
Quoted 8 lines
pte_next_pfn() pte_advance_pfn()Although the latter takes an unsigned long and only moves forward currently. I wonder if it makes sense to have their naming and semantics match? i.e. change
pte_advance_pfn()topte_move_pfn()and let it move backwards too.I guess we don't have a need for that and it adds more churn.
we might have a need in the below case.
A forks B, then A and B share large folios. B unmap/exit, then large folios of process
A become single-mapped.
Right now, while writing A's folios, we are CoWing A's large folios into many small
folios. I believe we can reuse the entire large folios instead of doingnr_pagesCoW and page faults.
In this case, we might want to get the first PTE fromvmf->pte.
Quoted 4 lines
Quoted 2 lines
Once we have COW reuse for large folios in place (I think you know that I am working on that), it might make sense to "COW-reuse around",
TBH, I don't know if you are working on that. please Cc me next time :-)
I could have sworn I mentioned it to you already :)
See
https://lore.kernel.org/linux-mm/[email protected]/T/
I'll follow-up on that soonish (now that batching is upstream and the large mapcount is on its way upstream).
Quoted 8 lines
Quoted 4 lines
meaning we look if some neighboring PTEs map the same large folio and map them writable as well. But if it's really worth it, increasing page fault latency, is to be decided separately.
On the other hand, we eliminate latency for the remaining
nr_pages- 1 PTEs. Perhaps we can discover a more cost-effective method to signify that a large folio is probably singly mapped?
Yes, precisely what I am up to!
Quoted 2 lines
and only call "multi-PTEs" reuse while that condition is true in PF and avoid increasing latency always?
I'm thinking along those lines:
If we detect that it's exclusive, we can certainly mapped the current PTE writable. Then, we can decide how much (and if) we want to fault-around writable as an optimization.
For smallish large folios, it might make sense to try faulting around most of the folio.
For large large folios (e.g., PTE-mapped 2MiB THP and bigger), we might not want to fault around the whole thing -- especially if there is little benefit to be had from contig-pte bits.
Quoted 16 lines
Quoted 9 lines
Quoted 9 lines
Another case, might be
A forks B, and we write either A or B, we might CoW an entire large folios instead
CoWingnr_pagessmall folios.Quoted 6 lines
Quoted 2 lines
case 1 seems more useful, I might have a go after some days. then we might see
pte_move_pfn().
pte_move_pfn()does sound odd to me. It might not be required to implement the optimization described above. (it's easier to simply read another PTE, check if it maps the same large folio, and to batch from there)
Quoted 8 lines
It appears that your proposal suggests potential reusability as follows: if we have a large folio containing 16 PTEs, you might consider reusing only 4 by examining PTEs "around" but not necessarily all 16 PTEs. please correct me if my understanding is wrong.
Initially, my idea was to obtain the first PTE using
pte_move_pfn()and then utilizefolio_pte_batch()with the first PTE as arguments to ensure consistency innr_pages, thus enabling complete reuse of the whole folio.
Simply doing an vm_normal_folio(pte - X) == folio and then trying to batch from there might be easier and cleaner.
Signature · 3 lines
Cheers, David / dhildenb
Re: [PATCH v3 3/6] mm: introduce pte_move_swp_offset() helper which can move offset bidirectionally
Quoted 110 lines · David Hildenbrand
Quoted 75 lines · Barry Song
Quoted 74 lines · David Hildenbrand
Quoted 71 lines · Barry Song
Quoted 4 lines · Ryan Roberts
Quoted 2 lines · Barry Song
From: Barry Song <[email protected]>
Quoted 8 lines
Quoted 6 lines
There could arise a necessity to obtain the first
pte_tfrom a swappte_tlocated in the middle. For instance, this may occur within the context ofdo_swap_page(), where a page fault can potentially occur in any PTE of a large folio. To address this, the following patch introducespte_move_swp_offset(), a function capable of bidirectional movement by a specified delta argument. Consequently,pte_increment_swp_offset()You mean
pte_next_swp_offset()?yes.
Quoted 36 lines
Quoted 31 lines
will directly invoke it with delta = 1.
Suggested-by "Huang, Ying" <[email protected]>Signed-off-by Barry Song <[email protected]>---
mm/internal.h| 25 +++++++++++++++++++++----1 file changed, 21 insertions(+), 4 deletions(-)mm/internal.h +0
-1@@ -211,18 +211,21 @@ static inline int folio_pte_batch(struct folio *folio, unsigned long addr, } /** -* pte_next_swp_offset - Increment the swap entry offset field of a swap pte.
+ *pte_move_swp_offset- Move the swap entry offset field of a swap pte
+ * forward or backward by delta
* @pte: The initial pte state;is_swap_pte(pte)must be true and *non_swap_entry()must be false.
+ * @delta: The direction and the offset we are moving; forward if delta
+ * is positive; backward if delta is negative
*
- * Increments the swap offset, while maintaining all other fields, including
+ * Moves the swap offset, while maintaining all other fields, including
* swap type, and any swp pte bits. The resulting pte is returned. */mm/internal.h static inline int folio_pte_batch(struct folio *folio, unsigned long addr, · line 224
Quoted 2 lines
mm/internal.h +1
-1-static inline pte_t pte_next_swp_offset (pte_t pte) +static inline pte_t pte_move_swp_offset (pte_t pte,long delta)We have equivalent functions for pfn:
Quoted 8 lines
pte_next_pfn()
pte_advance_pfn()Although the latter takes an unsigned long and only moves forward currently. I wonder if it makes sense to have their naming and semantics match? i.e. change
pte_advance_pfn()topte_move_pfn()and let it move backwards too.I guess we don't have a need for that and it adds more churn.
we might have a need in the below case.
A forks B, then A and B share large folios. B unmap/exit, then large folios of process
A become single-mapped.
Right now, while writing A's folios, we are CoWing A's large folios into many small
folios. I believe we can reuse the entire large folios instead of doingnr_pages
CoW and page faults.
In this case, we might want to get the first PTE fromvmf->pte.Quoted 4 lines
Quoted 2 lines
Once we have COW reuse for large folios in place (I think you know that I am working on that), it might make sense to "COW-reuse around",
TBH, I don't know if you are working on that. please Cc me next time :-)
I could have sworn I mentioned it to you already :)
See
https://lore.kernel.org/linux-mm/[email protected]/T/
I'll follow-up on that soonish (now that batching is upstream and the large mapcount is on its way upstream).
Quoted 8 lines
Quoted 4 lines
meaning we look if some neighboring PTEs map the same large folio and map them writable as well. But if it's really worth it, increasing page fault latency, is to be decided separately.
On the other hand, we eliminate latency for the remaining
nr_pages- 1 PTEs. Perhaps we can discover a more cost-effective method to signify that a large folio is probably singly mapped?Yes, precisely what I am up to!
Quoted 2 lines
and only call "multi-PTEs" reuse while that condition is true in PF and avoid increasing latency always?
I'm thinking along those lines:
If we detect that it's exclusive, we can certainly mapped the current PTE writable. Then, we can decide how much (and if) we want to fault-around writable as an optimization.
Quoted 19 lines
For smallish large folios, it might make sense to try faulting around most of the folio.
For large large folios (e.g., PTE-mapped 2MiB THP and bigger), we might not want to fault around the whole thing -- especially if there is little benefit to be had from contig-pte bits.
Quoted 12 lines
Quoted 12 lines
Quoted 11 lines
Another case, might be
A forks B, and we write either A or B, we might CoW an entire large folios instead
CoWingnr_pagessmall folios.case 1 seems more useful, I might have a go after some days. then we might see
pte_move_pfn().
pte_move_pfn()does sound odd to me.
Yes, I agree the name is odd. pte_move_swp_offset() sounds similarly odd tbh. Perhaps just pte_advance_swp_offset() with a negative value is clearer about what its doing?
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
It might not be required to
implement the optimization described above. (it's easier to simply read another PTE, check if it maps the same large folio, and to batch from there)
Yes agreed.
Quoted 14 lines
Quoted 10 lines
Quoted 1 line
It appears that your proposal suggests potential reusability as follows: if we have a large folio containing 16 PTEs, you might consider reusing only 4 by examining PTEs "around" but not necessarily all 16 PTEs. please correct me if my understanding is wrong.
Initially, my idea was to obtain the first PTE using
pte_move_pfn()and then utilizefolio_pte_batch()with the first PTE as arguments to ensure consistency innr_pages, thus enabling complete reuse of the whole folio.Simply doing an
vm_normal_folio(pte - X)== folio and then trying to batch from there might be easier and cleaner.
Re: [PATCH v3 3/6] mm: introduce pte_move_swp_offset() helper which can move offset bidirectionally
Quoted 128 lines · Ryan Roberts
Quoted 126 lines · David Hildenbrand
Quoted 75 lines · Barry Song
Quoted 74 lines · David Hildenbrand
Quoted 71 lines · Barry Song
Quoted 4 lines · Ryan Roberts
Quoted 2 lines · Barry Song
From: Barry Song <[email protected]>
Quoted 8 lines
Quoted 6 lines
There could arise a necessity to obtain the first
pte_tfrom a swappte_tlocated in the middle. For instance, this may occur within the context ofdo_swap_page(), where a page fault can potentially occur in any PTE of a large folio. To address this, the following patch introducespte_move_swp_offset(), a function capable of bidirectional movement by a specified delta argument. Consequently,pte_increment_swp_offset()You mean
pte_next_swp_offset()?yes.
Quoted 36 lines
Quoted 31 lines
will directly invoke it with delta = 1.
Suggested-by "Huang, Ying" <[email protected]>Signed-off-by Barry Song <[email protected]>--- mm/internal.h | 25 +++++++++++++++++++++---- 1 file changed, 21 insertions(+), 4 deletions(-)mm/internal.h +0
-1@@ -211,18 +211,21 @@ static inline int folio_pte_batch(struct folio *folio, unsigned long addr, }/** -* pte_next_swp_offset - Increment the swap entry offset field of a swap pte.
+ *pte_move_swp_offset- Move the swap entry offset field of a swap pte
+ * forward or backward by delta
* @pte: The initial pte state;is_swap_pte(pte)must be true and
*non_swap_entry()must be false.
+ * @delta: The direction and the offset we are moving; forward if delta
+ * is positive; backward if delta is negative*- * Increments the swap offset, while maintaining all other fields, including
+ * Moves the swap offset, while maintaining all other fields, including
* swap type, and any swp pte bits. The resulting pte is returned. */mm/internal.h
Quoted 2 lines
mm/internal.h +1
-1-static inline pte_t pte_next_swp_offset (pte_t pte) +static inline pte_t pte_move_swp_offset (pte_t pte,long delta)We have equivalent functions for pfn:
Quoted 8 lines
pte_next_pfn() pte_advance_pfn()Although the latter takes an unsigned long and only moves forward currently. I wonder if it makes sense to have their naming and semantics match? i.e. change
pte_advance_pfn()topte_move_pfn()and let it move backwards too.I guess we don't have a need for that and it adds more churn.
we might have a need in the below case.
A forks B, then A and B share large folios. B unmap/exit, then large folios of process
A become single-mapped.
Right now, while writing A's folios, we are CoWing A's large folios into many small
folios. I believe we can reuse the entire large folios instead of doingnr_pages
CoW and page faults.
In this case, we might want to get the first PTE fromvmf->pte.Quoted 4 lines
Quoted 2 lines
Once we have COW reuse for large folios in place (I think you know that I am working on that), it might make sense to "COW-reuse around",
TBH, I don't know if you are working on that. please Cc me next time :-)
I could have sworn I mentioned it to you already :)
See
https://lore.kernel.org/linux-mm/[email protected]/T/
I'll follow-up on that soonish (now that batching is upstream and the large mapcount is on its way upstream).
Quoted 8 lines
Quoted 4 lines
meaning we look if some neighboring PTEs map the same large folio and map them writable as well. But if it's really worth it, increasing page fault latency, is to be decided separately.
On the other hand, we eliminate latency for the remaining
nr_pages- 1 PTEs. Perhaps we can discover a more cost-effective method to signify that a large folio is probably singly mapped?Yes, precisely what I am up to!
Quoted 2 lines
and only call "multi-PTEs" reuse while that condition is true in PF and avoid increasing latency always?
I'm thinking along those lines:
If we detect that it's exclusive, we can certainly mapped the current PTE writable. Then, we can decide how much (and if) we want to fault-around writable as an optimization.
For smallish large folios, it might make sense to try faulting around most of the folio.
For large large folios (e.g., PTE-mapped 2MiB THP and bigger), we might not want to fault around the whole thing -- especially if there is little benefit to be had from contig-pte bits.
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Another case, might be
A forks B, and we write either A or B, we might CoW an entire large folios instead
CoWingnr_pagessmall folios.
Quoted 8 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 2 lines
case 1 seems more useful, I might have a go after some days. then we might see
pte_move_pfn().
pte_move_pfn()does sound odd to me.Yes, I agree the name is odd.
pte_move_swp_offset()sounds similarly odd tbh. Perhaps justpte_advance_swp_offset()with a negative value is clearer about what its doing?
I am not a native speaker. but dictionary says
advance:
move forward in a purposeful way.
a forward movement.
Now we are moving backward or forward :-)
Quoted 21 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
It might not be required to
implement the optimization described above. (it's easier to simply read another PTE, check if it maps the same large folio, and to batch from there)Yes agreed.
Quoted 14 lines
Quoted 10 lines
Quoted 1 line
It appears that your proposal suggests potential reusability as follows: if we have a large folio containing 16 PTEs, you might consider reusing only 4 by examining PTEs "around" but not necessarily all 16 PTEs. please correct me if my understanding is wrong.
Initially, my idea was to obtain the first PTE using
pte_move_pfn()and then utilizefolio_pte_batch()with the first PTE as arguments to ensure consistency innr_pages, thus enabling complete reuse of the whole folio.Simply doing an
vm_normal_folio(pte - X)== folio and then trying to batch from there might be easier and cleaner.
Re: [PATCH v3 3/6] mm: introduce pte_move_swp_offset() helper which can move offset bidirectionally
Quoted 138 lines · Barry Song
Quoted 128 lines · Ryan Roberts
Quoted 126 lines · David Hildenbrand
Quoted 75 lines · Barry Song
Quoted 74 lines · David Hildenbrand
Quoted 71 lines · Barry Song
Quoted 4 lines · Ryan Roberts
Quoted 2 lines · Barry Song
From: Barry Song <[email protected]>
Quoted 8 lines
Quoted 6 lines
There could arise a necessity to obtain the first
pte_tfrom a swappte_tlocated in the middle. For instance, this may occur within the context ofdo_swap_page(), where a page fault can potentially occur in any PTE of a large folio. To address this, the following patch introducespte_move_swp_offset(), a function capable of bidirectional movement by a specified delta argument. Consequently,pte_increment_swp_offset()You mean
pte_next_swp_offset()?yes.
Quoted 36 lines
Quoted 31 lines
will directly invoke it with delta = 1.
Suggested-by "Huang, Ying" <[email protected]>Signed-off-by Barry Song <[email protected]>--- mm/internal.h | 25 +++++++++++++++++++++---- 1 file changed, 21 insertions(+), 4 deletions(-)mm/internal.h +0
-1@@ -211,18 +211,21 @@ static inline int folio_pte_batch(struct folio *folio, unsigned long addr, }/** -* pte_next_swp_offset - Increment the swap entry offset field of a swap pte.
+ *pte_move_swp_offset- Move the swap entry offset field of a swap pte
+ * forward or backward by delta
* @pte: The initial pte state;is_swap_pte(pte)must be true and
*non_swap_entry()must be false.
+ * @delta: The direction and the offset we are moving; forward if delta
+ * is positive; backward if delta is negative*- * Increments the swap offset, while maintaining all other fields, including
+ * Moves the swap offset, while maintaining all other fields, including
* swap type, and any swp pte bits. The resulting pte is returned. */mm/internal.h
Quoted 2 lines
mm/internal.h +1
-1-static inline pte_t pte_next_swp_offset (pte_t pte) +static inline pte_t pte_move_swp_offset (pte_t pte,long delta)We have equivalent functions for pfn:
Quoted 8 lines
pte_next_pfn() pte_advance_pfn()Although the latter takes an unsigned long and only moves forward currently. I wonder if it makes sense to have their naming and semantics match? i.e. change
pte_advance_pfn()topte_move_pfn()and let it move backwards too.I guess we don't have a need for that and it adds more churn.
we might have a need in the below case.
A forks B, then A and B share large folios. B unmap/exit, then large folios of process
A become single-mapped.
Right now, while writing A's folios, we are CoWing A's large folios into many small
folios. I believe we can reuse the entire large folios instead of doingnr_pages
CoW and page faults.
In this case, we might want to get the first PTE fromvmf->pte.Quoted 4 lines
Quoted 2 lines
Once we have COW reuse for large folios in place (I think you know that I am working on that), it might make sense to "COW-reuse around",
TBH, I don't know if you are working on that. please Cc me next time :-)
I could have sworn I mentioned it to you already :)
See
https://lore.kernel.org/linux-mm/[email protected]/T/
I'll follow-up on that soonish (now that batching is upstream and the large mapcount is on its way upstream).
Quoted 8 lines
Quoted 4 lines
meaning we look if some neighboring PTEs map the same large folio and map them writable as well. But if it's really worth it, increasing page fault latency, is to be decided separately.
On the other hand, we eliminate latency for the remaining
nr_pages- 1 PTEs. Perhaps we can discover a more cost-effective method to signify that a large folio is probably singly mapped?Yes, precisely what I am up to!
Quoted 2 lines
and only call "multi-PTEs" reuse while that condition is true in PF and avoid increasing latency always?
I'm thinking along those lines:
If we detect that it's exclusive, we can certainly mapped the current PTE writable. Then, we can decide how much (and if) we want to fault-around writable as an optimization.
For smallish large folios, it might make sense to try faulting around most of the folio.
For large large folios (e.g., PTE-mapped 2MiB THP and bigger), we might not want to fault around the whole thing -- especially if there is little benefit to be had from contig-pte bits.
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Another case, might be
A forks B, and we write either A or B, we might CoW an entire large folios instead
CoWingnr_pagessmall folios.Quoted 8 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 2 lines
case 1 seems more useful, I might have a go after some days. then we might see
pte_move_pfn().
pte_move_pfn()does sound odd to me.Yes, I agree the name is odd.
pte_move_swp_offset()sounds similarly odd tbh. Perhaps justpte_advance_swp_offset()with a negative value is clearer about what its doing?
Quoted 7 lines
I am not a native speaker. but dictionary says
advance:
move forward in a purposeful way.
a forward movement.Now we are moving backward or forward :-)
Sure, but if you pass a negative value then you are moving forwards by a negative amount ;-)
Anyway, forget I said anything - its not important.
Quoted 22 lines
Quoted 22 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
It might not be required to
implement the optimization described above. (it's easier to simply read another PTE, check if it maps the same large folio, and to batch from there)Yes agreed.
Quoted 14 lines
Quoted 10 lines
Quoted 1 line
It appears that your proposal suggests potential reusability as follows: if we have a large folio containing 16 PTEs, you might consider reusing only 4 by examining PTEs "around" but not necessarily all 16 PTEs. please correct me if my understanding is wrong.
Initially, my idea was to obtain the first PTE using
pte_move_pfn()and then utilizefolio_pte_batch()with the first PTE as arguments to ensure consistency innr_pages, thus enabling complete reuse of the whole folio.Simply doing an
vm_normal_folio(pte - X)== folio and then trying to batch from there might be easier and cleaner.
Chris Li
maintainerRe: [PATCH v3 3/6] mm: introduce pte_move_swp_offset() helper which can move offset bidirectionally
In reply to Barry SongQuoted 10 lines · Barry Song
From: Barry Song <[email protected]>
There could arise a necessity to obtain the first
pte_tfrom a swappte_tlocated in the middle. For instance, this may occur within the context ofdo_swap_page(), where a page fault can potentially occur in any PTE of a large folio. To address this, the following patch introducespte_move_swp_offset(), a function capable of bidirectional movement by a specified delta argument. Consequently,pte_increment_swp_offset()will directly invoke it with delta = 1.
BTW, this patch has conflict with the latest mm-unstable. You might need to refresh it.
I do not see the delta = -1 usage case here. Maybe merge the patch with the follow up patch that uses the delta = -1?
Does delta only make sense as 1 and -1?
This patch doesn't seem straightly necessary to me.
Chris
Chris
Quoted 62 lines
Suggested-by "Huang, Ying" <[email protected]>Signed-off-by Barry Song <[email protected]>--- mm/internal.h | 25 +++++++++++++++++++++---- 1 file changed, 21 insertions(+), 4 deletions(-)mm/internal.h +21
-5@@ -211,18 +211,21 @@ static inline int folio_pte_batch(struct folio *folio, unsigned long addr, }/** -* pte_next_swp_offset - Increment the swap entry offset field of a swap pte. +* pte_move_swp_offset - Move the swap entry offset field of a swap pte +* forward or backward by delta * @pte: The initial pte state; is_swap_pte(pte) must be true and * non_swap_entry() must be false. +* @delta: The direction and the offset we are moving; forward if delta +* is positive; backward if delta is negative * -* Increments the swap offset, while maintaining all other fields, including +* Moves the swap offset, while maintaining all other fields, including * swap type, and any swp pte bits. The resulting pte is returned. */ -static inline pte_t pte_next_swp_offset (pte_t pte) +static inline pte_t pte_move_swp_offset (pte_t pte,long delta) {swp_entry_t entry =pte_to_swp_entry (pte);pte_t new =__swp_entry_to_pte (__swp_entry (swp_type (entry), - (swp_offset (entry) +1 ))); + (swp_offset (entry) + delta)));if (pte_swp_soft_dirty (pte)) new =pte_swp_mksoft_dirty (new);@@ -234,6 +237,20 @@ static inline pte_t pte_next_swp_offset(pte_t pte)return new; } + +/** +* pte_next_swp_offset - Increment the swap entry offset field of a swap pte. +* @pte: The initial pte state; is_swap_pte(pte) must be true and +* non_swap_entry() must be false. +* +* Increments the swap offset, while maintaining all other fields, including +* swap type, and any swp pte bits. The resulting pte is returned. +*/ +static inline pte_t pte_next_swp_offset (pte_t pte) +{ +return pte_move_swp_offset (pte,1 ); +} +/** * swap_pte_batch - detect a PTE batch for a set of contiguous swap entries * @start_ptep: Page table pointer for the first entry. -- 2.34.1
Re: [PATCH v3 3/6] mm: introduce pte_move_swp_offset() helper which can move offset bidirectionally
Quoted 13 lines · Chris Li
Quoted 10 lines · Barry Song
From: Barry Song <[email protected]>
There could arise a necessity to obtain the first
pte_tfrom a swappte_tlocated in the middle. For instance, this may occur within the context ofdo_swap_page(), where a page fault can potentially occur in any PTE of a large folio. To address this, the following patch introducespte_move_swp_offset(), a function capable of bidirectional movement by a specified delta argument. Consequently,pte_increment_swp_offset()will directly invoke it with delta = 1.
Quoted 6 lines
BTW, this patch has conflict with the latest mm-unstable. You might need to refresh it.
I do not see the delta = -1 usage case here. Maybe merge the patch with the follow up patch that uses the delta = -1?
Does delta only make sense as 1 and -1?
nop. delta can be any value from -1 to -(nr_pages - 1). This is used to get the first pte from vmf->pte in patdh6.
it might be better to be a separate patch as it is introducing a new helper? do_swap_page() might not be the only user in the future?
Quoted 60 lines
This patch doesn't seem straightly necessary to me.
Chris
Chris
Quoted 52 lines
Suggested-by "Huang, Ying" <[email protected]>Signed-off-by Barry Song <[email protected]>--- mm/internal.h | 25 +++++++++++++++++++++---- 1 file changed, 21 insertions(+), 4 deletions(-)mm/internal.h +18
-4@@ -211,18 +211,21 @@ static inline int folio_pte_batch(struct folio *folio, unsigned long addr, }/** -* pte_next_swp_offset - Increment the swap entry offset field of a swap pte. +* pte_move_swp_offset - Move the swap entry offset field of a swap pte +* forward or backward by delta * @pte: The initial pte state; is_swap_pte(pte) must be true and * non_swap_entry() must be false. +* @delta: The direction and the offset we are moving; forward if delta +* is positive; backward if delta is negative * -* Increments the swap offset, while maintaining all other fields, including +* Moves the swap offset, while maintaining all other fields, including * swap type, and any swp pte bits. The resulting pte is returned. */ -static inline pte_t pte_next_swp_offset (pte_t pte) +static inline pte_t pte_move_swp_offset (pte_t pte,long delta) {swp_entry_t entry =pte_to_swp_entry (pte);pte_t new =__swp_entry_to_pte (__swp_entry (swp_type (entry), - (swp_offset (entry) +1 ))); + (swp_offset (entry) + delta)));if (pte_swp_soft_dirty (pte)) new =pte_swp_mksoft_dirty (new);@@ -234,6 +237,20 @@ static inline pte_t pte_next_swp_offset(pte_t pte)return new; } + +/** +* pte_next_swp_offset - Increment the swap entry offset field of a swap pte. +* @pte: The initial pte state; is_swap_pte(pte) must be true and +* non_swap_entry() must be false. +* +* Increments the swap offset, while maintaining all other fields, including +* swap type, and any swp pte bits. The resulting pte is returned. +*/ +static inline pte_t pte_next_swp_offset (pte_t pte) +{
Quoted 8 lines
Quoted 8 lines
mm/internal.h +3
-1+return pte_move_swp_offset (pte,1 ); +} +/** * swap_pte_batch - detect a PTE batch for a set of contiguous swap entries * @start_ptep: Page table pointer for the first entry. -- 2.34.1
Thanks
Barry
Re: [PATCH v3 3/6] mm: introduce pte_move_swp_offset() helper which can move offset bidirectionally
In reply to Barry SongQuoted 10 lines · Barry Song
From: Barry Song <[email protected]>
There could arise a necessity to obtain the first
pte_tfrom a swappte_tlocated in the middle. For instance, this may occur within the context ofdo_swap_page(), where a page fault can potentially occur in any PTE of a large folio. To address this, the following patch introducespte_move_swp_offset(), a function capable of bidirectional movement by a specified delta argument. Consequently,pte_increment_swp_offset()will directly invoke it with delta = 1.
Quoted 2 lines
Suggested-by "Huang, Ying" <[email protected]>Signed-off-by Barry Song <[email protected]>
LGTM, Thanks! Feel free to add
in the future version.
Quoted 45 lines
--- mm/internal.h | 25 +++++++++++++++++++++---- 1 file changed, 21 insertions(+), 4 deletions(-)mm/internal.h +14
-4@@ -211,18 +211,21 @@ static inline int folio_pte_batch(struct folio *folio, unsigned long addr, }/** -* pte_next_swp_offset - Increment the swap entry offset field of a swap pte. +* pte_move_swp_offset - Move the swap entry offset field of a swap pte +* forward or backward by delta * @pte: The initial pte state; is_swap_pte(pte) must be true and * non_swap_entry() must be false. +* @delta: The direction and the offset we are moving; forward if delta +* is positive; backward if delta is negative * -* Increments the swap offset, while maintaining all other fields, including +* Moves the swap offset, while maintaining all other fields, including * swap type, and any swp pte bits. The resulting pte is returned. */ -static inline pte_t pte_next_swp_offset (pte_t pte) +static inline pte_t pte_move_swp_offset (pte_t pte,long delta) {swp_entry_t entry =pte_to_swp_entry (pte);pte_t new =__swp_entry_to_pte (__swp_entry (swp_type (entry), - (swp_offset (entry) +1 ))); + (swp_offset (entry) + delta)));if (pte_swp_soft_dirty (pte)) new =pte_swp_mksoft_dirty (new);@@ -234,6 +237,20 @@ static inline pte_t pte_next_swp_offset(pte_t pte)return new; } + +/** +* pte_next_swp_offset - Increment the swap entry offset field of a swap pte. +* @pte: The initial pte state; is_swap_pte(pte) must be true and +* non_swap_entry() must be false. +* +* Increments the swap offset, while maintaining all other fields, including
mm/internal.h static inline pte_t pte_next_swp_offset(pte_t pte) · line 256
Quoted 10 lines
mm/internal.h +7
-0+* swap type, and any swp pte bits. The resulting pte is returned. +*/ +static inline pte_t pte_next_swp_offset (pte_t pte) +{ +return pte_move_swp_offset (pte,1 ); +} +/** * swap_pte_batch - detect a PTE batch for a set of contiguous swap entries * @start_ptep: Page table pointer for the first entry.
--
Best Regards,
Huang, Ying
PATCH v3 4/6 mm: introduce arch_do_swap_page_nr() which allows restore metadata for nr pages
In reply to Barry SongFrom: Barry Song <[email protected]>
Should do_swap_page() have the capability to directly map a large folio, metadata restoration becomes necessary for a specified number of pages denoted as nr. It's important to highlight that metadata restoration is solely required by the SPARC platform, which, however, does not enable THP_SWAP. Consequently, in the present kernel configuration, there exists no practical scenario where users necessitate the restoration of nr metadata. Platforms implementing THP_SWAP might invoke this function with nr values exceeding 1, subsequent to do_swap_page() successfully mapping an entire large folio. Nonetheless, their arch_do_swap_page_nr() functions remain empty.
--- include/linux/pgtable.h | 26 ++++++++++++++++++++------ mm/memory.c | 3 ++- 2 files changed, 22 insertions(+), 7 deletions(-)
include/linux/pgtable.h +20 -6
@@ -1084,6 +1084,15 @@ static inline int pgd_same(pgd_t pgd_a, pgd_t pgd_b) })#ifndef __HAVE_ARCH_DO_SWAP_PAGE +static inline void arch_do_swap_page_nr (struct mm_struct *mm, +struct vm_area_struct *vma, +unsigned long addr, +pte_t pte,pte_t oldpte, +int nr) +{ + +} +#else /* * Some architectures support metadata associated with a page. When a * page is being swapped out, this metadata must be saved so it can be
@@ -1092,12 +1101,17 @@ static inline int pgd_same(pgd_t pgd_a, pgd_t pgd_b)* page as metadata for the page. arch_do_swap_page() can restore this * metadata when a page is swapped back in. */ -static inline void arch_do_swap_page (struct mm_struct *mm, -struct vm_area_struct *vma, -unsigned long addr, -pte_t pte,pte_t oldpte) -{ - +static inline void arch_do_swap_page_nr (struct mm_struct *mm, +struct vm_area_struct *vma, +unsigned long addr, +pte_t pte,pte_t oldpte, +int nr) +{ +for (int i =0 ; i < nr; i++) { +arch_do_swap_page (vma->vm_mm, vma, addr + i *PAGE_SIZE , +pte_advance_pfn (pte, i), +pte_advance_pfn (oldpte, i)); + } }#endif
mm/memory.c +2 -1
@@ -4266,7 +4266,8 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)VM_BUG_ON (!folio_test_anon (folio) || (pte_write (pte) && !PageAnonExclusive (page)));set_pte_at (vma->vm_mm, vmf->address, vmf->pte, pte); -arch_do_swap_page (vma->vm_mm, vma, vmf->address, pte, vmf->orig_pte); +arch_do_swap_page_nr (vma->vm_mm, vma, vmf->address, + pte, vmf->orig_pte,1 );folio_unlock (folio);if (folio != swapcache && swapcache) {
Signature · 1 line
2.34.1
Re: [PATCH v3 4/6] mm: introduce arch_do_swap_page_nr() which allows restore metadata for nr pages
Quoted 43 lines · Barry Song
From: Barry Song <[email protected]>
Should
do_swap_page()have the capability to directly map a large folio, metadata restoration becomes necessary for a specified number of pages denoted as nr. It's important to highlight that metadata restoration is solely required by the SPARC platform, which, however, does not enableTHP_SWAP. Consequently, in the present kernel configuration, there exists no practical scenario where users necessitate the restoration of nr metadata. Platforms implementingTHP_SWAPmight invoke this function with nr values exceeding 1, subsequent todo_swap_page()successfully mapping an entire large folio. Nonetheless, theirarch_do_swap_page_nr()functions remain empty.Cc Khalid Aziz <[email protected]>Cc "David S. Miller" <[email protected]>Cc Andreas Larsson <[email protected]>Signed-off-by Barry Song <[email protected]>--- include/linux/pgtable.h | 26 ++++++++++++++++++++------ mm/memory.c | 3 ++- 2 files changed, 22 insertions(+), 7 deletions(-)include/linux/pgtable.h +9
-0@@ -1084,6 +1084,15 @@ static inline int pgd_same(pgd_t pgd_a, pgd_t pgd_b) })#ifndef __HAVE_ARCH_DO_SWAP_PAGE +static inline void arch_do_swap_page_nr (struct mm_struct *mm, +struct vm_area_struct *vma, +unsigned long addr, +pte_t pte,pte_t oldpte, +int nr) +{ + +} +#else /* * Some architectures support metadata associated with a page. When a * page is being swapped out, this metadata must be saved so it can be
include/linux/pgtable.h static inline int pgd_same(pgd_t pgd_a, pgd_t pgd_b) · line 1098
Quoted 7 lines
include/linux/pgtable.h +0
-4@@ -1092,12 +1101,17 @@ static inline int pgd_same(pgd_t pgd_a, pgd_t pgd_b)* page as metadata for the page. arch_do_swap_page() can restore this * metadata when a page is swapped back in. */ -static inline void arch_do_swap_page (struct mm_struct *mm, -struct vm_area_struct *vma, -unsigned long addr, -pte_t pte,pte_t oldpte)
This hook seems to be very similar to arch_swap_restore(), I wonder if it makes sense to merge them. Out of scope for this patch series though.
include/linux/pgtable.h static inline int pgd_same(pgd_t pgd_a, pgd_t pgd_b) · line 1113
Quoted 12 lines
include/linux/pgtable.h +10
-2-{ - +static inline void arch_do_swap_page_nr (struct mm_struct *mm, +struct vm_area_struct *vma, +unsigned long addr, +pte_t pte,pte_t oldpte, +int nr) +{ +for (int i =0 ; i < nr; i++) { +arch_do_swap_page (vma->vm_mm, vma, addr + i *PAGE_SIZE , +pte_advance_pfn (pte, i), +pte_advance_pfn (oldpte, i));
It seems a bit odd to create a batched version of this, but not allow arches to take advantage. Although I guess your point is that only SPARC implements it and on that platform nr will always be 1? So no point right now? So this is just a convenience for do_swap_page()? Makes sense.
Quoted 18 lines
include/linux/pgtable.h +1
-0+ } }#endif mm/memory.c +2
-1@@ -4266,7 +4266,8 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)VM_BUG_ON (!folio_test_anon (folio) || (pte_write (pte) && !PageAnonExclusive (page)));set_pte_at (vma->vm_mm, vmf->address, vmf->pte, pte); -arch_do_swap_page (vma->vm_mm, vma, vmf->address, pte, vmf->orig_pte); +arch_do_swap_page_nr (vma->vm_mm, vma, vmf->address, + pte, vmf->orig_pte,1 );folio_unlock (folio);if (folio != swapcache && swapcache) {
Re: [PATCH v3 4/6] mm: introduce arch_do_swap_page_nr() which allows restore metadata for nr pages
In reply to Barry SongQuoted 13 lines · Barry Song
From: Barry Song <[email protected]>
Should
do_swap_page()have the capability to directly map a large folio, metadata restoration becomes necessary for a specified number of pages denoted as nr. It's important to highlight that metadata restoration is solely required by the SPARC platform, which, however, does not enableTHP_SWAP. Consequently, in the present kernel configuration, there exists no practical scenario where users necessitate the restoration of nr metadata. Platforms implementingTHP_SWAPmight invoke this function with nr values exceeding 1, subsequent todo_swap_page()successfully mapping an entire large folio. Nonetheless, theirarch_do_swap_page_nr()functions remain empty.
Quoted 4 lines
Cc Khalid Aziz <[email protected]>Cc "David S. Miller" <[email protected]>Cc Andreas Larsson <[email protected]>Signed-off-by Barry Song <[email protected]>
Looks good to me.
Quoted 63 lines
--- include/linux/pgtable.h | 26 ++++++++++++++++++++------ mm/memory.c | 3 ++- 2 files changed, 22 insertions(+), 7 deletions(-)include/linux/pgtable.h +20
-6@@ -1084,6 +1084,15 @@ static inline int pgd_same(pgd_t pgd_a, pgd_t pgd_b) })#ifndef __HAVE_ARCH_DO_SWAP_PAGE +static inline void arch_do_swap_page_nr (struct mm_struct *mm, +struct vm_area_struct *vma, +unsigned long addr, +pte_t pte,pte_t oldpte, +int nr) +{ + +} +#else /* * Some architectures support metadata associated with a page. When a * page is being swapped out, this metadata must be saved so it can be @@ -1092,12 +1101,17 @@ static inline int pgd_same(pgd_t pgd_a, pgd_t pgd_b)* page as metadata for the page. arch_do_swap_page() can restore this * metadata when a page is swapped back in. */ -static inline void arch_do_swap_page (struct mm_struct *mm, -struct vm_area_struct *vma, -unsigned long addr, -pte_t pte,pte_t oldpte) -{ - +static inline void arch_do_swap_page_nr (struct mm_struct *mm, +struct vm_area_struct *vma, +unsigned long addr, +pte_t pte,pte_t oldpte, +int nr) +{ +for (int i =0 ; i < nr; i++) { +arch_do_swap_page (vma->vm_mm, vma, addr + i *PAGE_SIZE , +pte_advance_pfn (pte, i), +pte_advance_pfn (oldpte, i)); + } }#endif mm/memory.c +2
-1@@ -4266,7 +4266,8 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)VM_BUG_ON (!folio_test_anon (folio) || (pte_write (pte) && !PageAnonExclusive (page)));set_pte_at (vma->vm_mm, vmf->address, vmf->pte, pte); -arch_do_swap_page (vma->vm_mm, vma, vmf->address, pte, vmf->orig_pte); +arch_do_swap_page_nr (vma->vm_mm, vma, vmf->address, + pte, vmf->orig_pte,1 );folio_unlock (folio);if (folio != swapcache && swapcache) {
PATCH v3 5/6 mm: swap: make should_try_to_free_swap() support large-folio
In reply to Barry SongFrom: Chuanhua Han <[email protected]>
The function should_try_to_free_swap() operates under the assumption that swap-in always occurs at the normal page granularity, i.e., folio_nr_pages() = 1. However, in reality, for large folios, add_to_swap_cache() will invoke folio_ref_add(folio, nr). To accommodate large folio swap-in, this patch eliminates this assumption.
--- mm/memory.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
mm/memory.c +1 -1
@@ -3877,7 +3877,7 @@ static inline bool should_try_to_free_swap(struct folio *folio,* reference only in case it's likely that we'll be the exlusive user. */ return (fault_flags &FAULT_FLAG_WRITE ) && !folio_test_ksm (folio) && -folio_ref_count (folio) ==2 ; +folio_ref_count (folio) == (1 +folio_nr_pages (folio)); }static vm_fault_t pte_marker_clear (struct vm_fault *vmf)
Signature · 1 line
2.34.1
Re: [PATCH v3 5/6] mm: swap: make should_try_to_free_swap() support large-folio
Quoted 23 lines · Barry Song
From: Chuanhua Han <[email protected]>
The function
should_try_to_free_swap()operates under the assumption that swap-in always occurs at the normal page granularity, i.e.,folio_nr_pages()= 1. However, in reality, for large folios,add_to_swap_cache()will invokefolio_ref_add(folio, nr). To accommodate large folio swap-in, this patch eliminates this assumption.Signed-off-by Chuanhua Han <[email protected]>Co-developed-by Barry Song <[email protected]>Signed-off-by Barry Song <[email protected]>Acked-by Chris Li <[email protected]>Reviewed-by Ryan Roberts <[email protected]>Reviewed-by "Huang, Ying" <[email protected]>--- mm/memory.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)mm/memory.c +0
-0
mm/memory.c static inline bool should_try_to_free_swap(struct folio *folio, · line 3883
Quoted 8 lines
mm/memory.c +1
-1@@ -3877,7 +3877,7 @@ static inline bool should_try_to_free_swap(struct folio *folio,* reference only in case it's likely that we'll be the exlusive user. */ return (fault_flags &FAULT_FLAG_WRITE ) && !folio_test_ksm (folio) && -folio_ref_count (folio) ==2 ; +folio_ref_count (folio) == (1 +folio_nr_pages (folio)); }static vm_fault_t pte_marker_clear (struct vm_fault *vmf)
Signature · 3 lines
Cheers, David / dhildenb
PATCH v3 6/6 mm: swap: entirely map large folios found in swapcache
In reply to Barry SongFrom: Chuanhua Han <[email protected]>
When a large folio is found in the swapcache, the current implementation requires calling do_swap_page() nr_pages times, resulting in nr_pages page faults. This patch opts to map the entire large folio at once to minimize page faults. Additionally, redundant checks and early exits for ARM64 MTE restoring are removed.
--- mm/memory.c | 60 ++++++++++++++++++++++++++++++++++++++++++----------- 1 file changed, 48 insertions(+), 12 deletions(-)
mm/memory.c +48 -12
@@ -3968,6 +3968,10 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)pte_t pte;vm_fault_t ret =0 ;void *shadow =NULL ; +int nr_pages =1 ; +unsigned long page_idx =0 ; +unsigned long address = vmf->address; +pte_t *ptep;if (!pte_unmap_same (vmf)) goto out;
@@ -4166,6 +4170,36 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) goto out_nomap; } + ptep = vmf->pte; +if (folio_test_large (folio) &&folio_test_swapcache (folio)) { +int nr =folio_nr_pages (folio); +unsigned long idx =folio_page_idx (folio, page); +unsigned long folio_start = vmf->address - idx *PAGE_SIZE ; +unsigned long folio_end = folio_start + nr *PAGE_SIZE ; +pte_t *folio_ptep; +pte_t folio_pte; + +if (unlikely (folio_start <max (vmf->address &PMD_MASK , vma->vm_start))) + goto check_folio; +if (unlikely (folio_end >pmd_addr_end (vmf->address, vma->vm_end))) + goto check_folio; + + folio_ptep = vmf->pte - idx; + folio_pte =ptep_get (folio_ptep); +if (!pte_same (folio_pte,pte_move_swp_offset (vmf->orig_pte, -idx)) || +swap_pte_batch (folio_ptep, nr, folio_pte) != nr) + goto check_folio; + + page_idx = idx; + address = folio_start; + ptep = folio_ptep; + nr_pages = nr; + entry = folio->swap; + page = &folio->page; + } + +check_folio: +/* * PG_anon_exclusive reuses PG_mappedtodisk for anon pages. A swap pte * must never point at an anonymous page in the swapcache that is
@@ -4225,12 +4259,13 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)* We're already holding a reference on the page but haven't mapped it * yet. */ -swap_free_nr (entry,1 ); +swap_free_nr (entry, nr_pages);if (should_try_to_free_swap (folio, vma, vmf->flags))folio_free_swap (folio); -inc_mm_counter (vma->vm_mm,MM_ANONPAGES ); -dec_mm_counter (vma->vm_mm,MM_SWAPENTS ); +folio_ref_add (folio, nr_pages -1 ); +add_mm_counter (vma->vm_mm,MM_ANONPAGES , nr_pages); +add_mm_counter (vma->vm_mm,MM_SWAPENTS , -nr_pages); pte =mk_pte (page, vma->vm_page_prot);/*
@@ -4240,34 +4275,35 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)* exclusivity. */ if (!folio_test_ksm (folio) && - (exclusive ||folio_ref_count (folio) ==1 )) { + (exclusive || (folio_ref_count (folio) == nr_pages && +folio_nr_pages (folio) == nr_pages))) {if (vmf->flags &FAULT_FLAG_WRITE ) { pte =maybe_mkwrite (pte_mkdirty (pte), vma); vmf->flags &= ~FAULT_FLAG_WRITE ; } rmap_flags |=RMAP_EXCLUSIVE ; } -flush_icache_page (vma, page); +flush_icache_pages (vma, page, nr_pages);if (pte_swp_soft_dirty (vmf->orig_pte)) pte =pte_mksoft_dirty (pte);if (pte_swp_uffd_wp (vmf->orig_pte)) pte =pte_mkuffd_wp (pte); - vmf->orig_pte = pte; + vmf->orig_pte =pte_advance_pfn (pte, page_idx);/* ksm created a completely new copy */ if (unlikely (folio != swapcache && swapcache)) { -folio_add_new_anon_rmap (folio, vma, vmf->address); +folio_add_new_anon_rmap (folio, vma, address);folio_add_lru_vma (folio, vma); }else { -folio_add_anon_rmap_pte (folio, page, vma, vmf->address, +folio_add_anon_rmap_ptes (folio, page, nr_pages, vma, address, rmap_flags); }VM_BUG_ON (!folio_test_anon (folio) || (pte_write (pte) && !PageAnonExclusive (page))); -set_pte_at (vma->vm_mm, vmf->address, vmf->pte, pte); -arch_do_swap_page_nr (vma->vm_mm, vma, vmf->address, - pte, vmf->orig_pte,1 ); +set_ptes (vma->vm_mm, address, ptep, pte, nr_pages); +arch_do_swap_page_nr (vma->vm_mm, vma, address, + pte, pte, nr_pages);folio_unlock (folio);if (folio != swapcache && swapcache) {
@@ -4291,7 +4327,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) }/* No need to invalidate - it was non-present before */ -update_mmu_cache_range (vmf, vma, vmf->address, vmf->pte,1 ); +update_mmu_cache_range (vmf, vma, address, ptep, nr_pages); unlock:if (vmf->pte)pte_unmap_unlock (vmf->pte, vmf->ptl);
Signature · 1 line
2.34.1
Re: [PATCH v3 6/6] mm: swap: entirely map large folios found in swapcache
Quoted 11 lines · Barry Song
From: Chuanhua Han <[email protected]>
When a large folio is found in the swapcache, the current implementation requires calling
do_swap_page()nr_pagestimes, resulting innr_pagespage faults. This patch opts to map the entire large folio at once to minimize page faults. Additionally, redundant checks and early exits for ARM64 MTE restoring are removed.Signed-off-by Chuanhua Han <[email protected]>Co-developed-by Barry Song <[email protected]>Signed-off-by Barry Song <[email protected]>
With the suggested changes below:
Quoted 9 lines
--- mm/memory.c | 60 ++++++++++++++++++++++++++++++++++++++++++----------- 1 file changed, 48 insertions(+), 12 deletions(-)mm/memory.c +0
-0
mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 3974
Quoted 7 lines
mm/memory.c +4
-0@@ -3968,6 +3968,10 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)pte_t pte;vm_fault_t ret =0 ;void *shadow =NULL ; +int nr_pages =1 ; +unsigned long page_idx =0 ; +unsigned long address = vmf->address; +pte_t *ptep;
nit: Personally I'd prefer all these to get initialised just before the "if (folio_test_large()..." block below. That way it is clear they are fresh (incase any logic between here and there makes an adjustment) and its clear that they are only to be used after that block (the compiler will warn if using an uninitialized value).
Quoted 26 lines
mm/memory.c +19
-0if (!pte_unmap_same (vmf)) goto out;@@ -4166,6 +4170,36 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) goto out_nomap; } + ptep = vmf->pte; +if (folio_test_large (folio) &&folio_test_swapcache (folio)) { +int nr =folio_nr_pages (folio); +unsigned long idx =folio_page_idx (folio, page); +unsigned long folio_start = vmf->address - idx *PAGE_SIZE ; +unsigned long folio_end = folio_start + nr *PAGE_SIZE ; +pte_t *folio_ptep; +pte_t folio_pte; + +if (unlikely (folio_start <max (vmf->address &PMD_MASK , vma->vm_start))) + goto check_folio; +if (unlikely (folio_end >pmd_addr_end (vmf->address, vma->vm_end))) + goto check_folio; + + folio_ptep = vmf->pte - idx; + folio_pte =ptep_get (folio_ptep); +if (!pte_same (folio_pte,pte_move_swp_offset (vmf->orig_pte, -idx)) || +swap_pte_batch (folio_ptep, nr, folio_pte) != nr) + goto check_folio;
mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 4201
Quoted 10 lines
mm/memory.c +10
-0+ + page_idx = idx; + address = folio_start; + ptep = folio_ptep; + nr_pages = nr; + entry = folio->swap; + page = &folio->page; + } + +check_folio:
Is this still the correct label name, given the checks are now above the new block? Perhaps "one_page" or something like that?
Quoted 22 lines
mm/memory.c +5
-3+/* * PG_anon_exclusive reuses PG_mappedtodisk for anon pages. A swap pte * must never point at an anonymous page in the swapcache that is @@ -4225,12 +4259,13 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)* We're already holding a reference on the page but haven't mapped it * yet. */ -swap_free_nr (entry,1 ); +swap_free_nr (entry, nr_pages);if (should_try_to_free_swap (folio, vma, vmf->flags))folio_free_swap (folio); -inc_mm_counter (vma->vm_mm,MM_ANONPAGES ); -dec_mm_counter (vma->vm_mm,MM_SWAPENTS ); +folio_ref_add (folio, nr_pages -1 ); +add_mm_counter (vma->vm_mm,MM_ANONPAGES , nr_pages); +add_mm_counter (vma->vm_mm,MM_SWAPENTS , -nr_pages); pte =mk_pte (page, vma->vm_page_prot);/*
mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 4279
Quoted 6 lines
mm/memory.c +2
-1@@ -4240,34 +4275,35 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)* exclusivity. */ if (!folio_test_ksm (folio) && - (exclusive ||folio_ref_count (folio) ==1 )) { + (exclusive || (folio_ref_count (folio) == nr_pages && +folio_nr_pages (folio) == nr_pages))) {
I think in practice there is no change here? If nr_pages > 1 then the folio is in the swapcache, so there is an extra ref on it? I agree with the change for robustness sake. Just checking my understanding.
Quoted 46 lines
mm/memory.c +8
-8if (vmf->flags &FAULT_FLAG_WRITE ) { pte =maybe_mkwrite (pte_mkdirty (pte), vma); vmf->flags &= ~FAULT_FLAG_WRITE ; } rmap_flags |=RMAP_EXCLUSIVE ; } -flush_icache_page (vma, page); +flush_icache_pages (vma, page, nr_pages);if (pte_swp_soft_dirty (vmf->orig_pte)) pte =pte_mksoft_dirty (pte);if (pte_swp_uffd_wp (vmf->orig_pte)) pte =pte_mkuffd_wp (pte); - vmf->orig_pte = pte; + vmf->orig_pte =pte_advance_pfn (pte, page_idx);/* ksm created a completely new copy */ if (unlikely (folio != swapcache && swapcache)) { -folio_add_new_anon_rmap (folio, vma, vmf->address); +folio_add_new_anon_rmap (folio, vma, address);folio_add_lru_vma (folio, vma); }else { -folio_add_anon_rmap_pte (folio, page, vma, vmf->address, +folio_add_anon_rmap_ptes (folio, page, nr_pages, vma, address, rmap_flags); }VM_BUG_ON (!folio_test_anon (folio) || (pte_write (pte) && !PageAnonExclusive (page))); -set_pte_at (vma->vm_mm, vmf->address, vmf->pte, pte); -arch_do_swap_page_nr (vma->vm_mm, vma, vmf->address, - pte, vmf->orig_pte,1 ); +set_ptes (vma->vm_mm, address, ptep, pte, nr_pages); +arch_do_swap_page_nr (vma->vm_mm, vma, address, + pte, pte, nr_pages);folio_unlock (folio);if (folio != swapcache && swapcache) {@@ -4291,7 +4327,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) }/* No need to invalidate - it was non-present before */ -update_mmu_cache_range (vmf, vma, vmf->address, vmf->pte,1 ); +update_mmu_cache_range (vmf, vma, address, ptep, nr_pages); unlock:if (vmf->pte)pte_unmap_unlock (vmf->pte, vmf->ptl);
Re: [PATCH v3 6/6] mm: swap: entirely map large folios found in swapcache
Quoted 32 lines · Ryan Roberts
Quoted 11 lines · Barry Song
From: Chuanhua Han <[email protected]>
When a large folio is found in the swapcache, the current implementation requires calling
do_swap_page()nr_pagestimes, resulting innr_pagespage faults. This patch opts to map the entire large folio at once to minimize page faults. Additionally, redundant checks and early exits for ARM64 MTE restoring are removed.Signed-off-by Chuanhua Han <[email protected]>Co-developed-by Barry Song <[email protected]>Signed-off-by Barry Song <[email protected]>With the suggested changes below:
Reviewed-by Ryan Roberts <[email protected]>Quoted 14 lines
--- mm/memory.c | 60 ++++++++++++++++++++++++++++++++++++++++++----------- 1 file changed, 48 insertions(+), 12 deletions(-)mm/memory.c +2
-0@@ -3968,6 +3968,10 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)pte_t pte;vm_fault_t ret =0 ;void *shadow =NULL ; +int nr_pages =1 ; +unsigned long page_idx =0 ;
Quoted 8 lines
mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 3974
Quoted 2 lines
mm/memory.c +2
-0+unsigned long address = vmf->address; +pte_t *ptep;nit: Personally I'd prefer all these to get initialised just before the "if (
folio_test_large()..." block below. That way it is clear they are fresh (incase any logic between here and there makes an adjustment) and its clear that they are only to be used after that block (the compiler will warn if using an uninitialized value).
right. I agree this will make the code more readable.
Quoted 32 lines
Quoted 32 lines
mm/memory.c +24
-0if (!pte_unmap_same (vmf)) goto out;@@ -4166,6 +4170,36 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) goto out_nomap; } + ptep = vmf->pte; +if (folio_test_large (folio) &&folio_test_swapcache (folio)) { +int nr =folio_nr_pages (folio); +unsigned long idx =folio_page_idx (folio, page); +unsigned long folio_start = vmf->address - idx *PAGE_SIZE ; +unsigned long folio_end = folio_start + nr *PAGE_SIZE ; +pte_t *folio_ptep; +pte_t folio_pte; + +if (unlikely (folio_start <max (vmf->address &PMD_MASK , vma->vm_start))) + goto check_folio; +if (unlikely (folio_end >pmd_addr_end (vmf->address, vma->vm_end))) + goto check_folio; + + folio_ptep = vmf->pte - idx; + folio_pte =ptep_get (folio_ptep); +if (!pte_same (folio_pte,pte_move_swp_offset (vmf->orig_pte, -idx)) || +swap_pte_batch (folio_ptep, nr, folio_pte) != nr) + goto check_folio; + + page_idx = idx; + address = folio_start; + ptep = folio_ptep; + nr_pages = nr;
Quoted 8 lines
mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 4201
Quoted 5 lines
mm/memory.c +5
-0+ entry = folio->swap; + page = &folio->page; + } + +check_folio:Is this still the correct label name, given the checks are now above the new block? Perhaps "
one_page" or something like that?
not quite sure about this, as the code after one_page can be multiple_pages. On the other hand, it seems we are really checking folio after "check_folio" :-)
BUG_ON(!folio_test_anon(folio) && folio_test_mappedtodisk(folio)); BUG_ON(folio_test_anon(folio) && PageAnonExclusive(page));
/*
* Check under PT lock (to protect against concurrent fork() sharing
* the swap entry concurrently) for certainly exclusive pages. */
if (!folio_test_ksm(folio)) {
Quoted 21 lines
Quoted 21 lines
mm/memory.c +5
-3+/* * PG_anon_exclusive reuses PG_mappedtodisk for anon pages. A swap pte * must never point at an anonymous page in the swapcache that is @@ -4225,12 +4259,13 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)* We're already holding a reference on the page but haven't mapped it * yet. */ -swap_free_nr (entry,1 ); +swap_free_nr (entry, nr_pages);if (should_try_to_free_swap (folio, vma, vmf->flags))folio_free_swap (folio); -inc_mm_counter (vma->vm_mm,MM_ANONPAGES ); -dec_mm_counter (vma->vm_mm,MM_SWAPENTS ); +folio_ref_add (folio, nr_pages -1 ); +add_mm_counter (vma->vm_mm,MM_ANONPAGES , nr_pages); +add_mm_counter (vma->vm_mm,MM_SWAPENTS , -nr_pages); pte =mk_pte (page, vma->vm_page_prot);
Quoted 12 lines
mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 4279
Quoted 8 lines
mm/memory.c +2
-1/* @@ -4240,34 +4275,35 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)* exclusivity. */ if (!folio_test_ksm (folio) && - (exclusive ||folio_ref_count (folio) ==1 )) { + (exclusive || (folio_ref_count (folio) == nr_pages && +folio_nr_pages (folio) == nr_pages))) {I think in practice there is no change here? If
nr_pages> 1 then the folio is in the swapcache, so there is an extra ref on it? I agree with the change for robustness sake. Just checking my understanding.
This is the code showing we are reusing/(mkwrite) a folio either
1. we meet a small folio and we are the only one hitting the small folio
2. we meet a large folio and we are the only one hitting the large folio
any corner cases besides the above two seems difficult. for example,
while we hit a large folio in swapcache but if we can't entirely map it (nr_pages==1) due to partial unmap, we will have folio_ref_count(folio) == nr_pages == 1, in this case, lacking folio_nr_pages(folio) == nr_pages might lead to mkwrite() on a single pte within a partially unmapped large folio. not quite sure this is wrong, but seems buggy and arduous.
Quoted 41 lines
Quoted 41 lines
mm/memory.c +7
-7if (vmf->flags &FAULT_FLAG_WRITE ) { pte =maybe_mkwrite (pte_mkdirty (pte), vma); vmf->flags &= ~FAULT_FLAG_WRITE ; } rmap_flags |=RMAP_EXCLUSIVE ; } -flush_icache_page (vma, page); +flush_icache_pages (vma, page, nr_pages);if (pte_swp_soft_dirty (vmf->orig_pte)) pte =pte_mksoft_dirty (pte);if (pte_swp_uffd_wp (vmf->orig_pte)) pte =pte_mkuffd_wp (pte); - vmf->orig_pte = pte; + vmf->orig_pte =pte_advance_pfn (pte, page_idx);/* ksm created a completely new copy */ if (unlikely (folio != swapcache && swapcache)) { -folio_add_new_anon_rmap (folio, vma, vmf->address); +folio_add_new_anon_rmap (folio, vma, address);folio_add_lru_vma (folio, vma); }else { -folio_add_anon_rmap_pte (folio, page, vma, vmf->address, +folio_add_anon_rmap_ptes (folio, page, nr_pages, vma, address, rmap_flags); }VM_BUG_ON (!folio_test_anon (folio) || (pte_write (pte) && !PageAnonExclusive (page))); -set_pte_at (vma->vm_mm, vmf->address, vmf->pte, pte); -arch_do_swap_page_nr (vma->vm_mm, vma, vmf->address, - pte, vmf->orig_pte,1 ); +set_ptes (vma->vm_mm, address, ptep, pte, nr_pages); +arch_do_swap_page_nr (vma->vm_mm, vma, address, + pte, pte, nr_pages);folio_unlock (folio);if (folio != swapcache && swapcache) {@@ -4291,7 +4327,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) }
Quoted 7 lines
mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 4333
Quoted 6 lines
mm/memory.c +1
-1/* No need to invalidate - it was non-present before */ -update_mmu_cache_range (vmf, vma, vmf->address, vmf->pte,1 ); +update_mmu_cache_range (vmf, vma, address, ptep, nr_pages); unlock:if (vmf->pte)pte_unmap_unlock (vmf->pte, vmf->ptl);
Thanks
Barry
Re: [PATCH v3 6/6] mm: swap: entirely map large folios found in swapcache
Quoted 138 lines · Barry Song
Quoted 32 lines · Ryan Roberts
Quoted 11 lines · Barry Song
From: Chuanhua Han <[email protected]>
When a large folio is found in the swapcache, the current implementation requires calling
do_swap_page()nr_pagestimes, resulting innr_pagespage faults. This patch opts to map the entire large folio at once to minimize page faults. Additionally, redundant checks and early exits for ARM64 MTE restoring are removed.Signed-off-by Chuanhua Han <[email protected]>Co-developed-by Barry Song <[email protected]>Signed-off-by Barry Song <[email protected]>With the suggested changes below:
Reviewed-by Ryan Roberts <[email protected]>Quoted 14 lines
--- mm/memory.c | 60 ++++++++++++++++++++++++++++++++++++++++++----------- 1 file changed, 48 insertions(+), 12 deletions(-)mm/memory.c +2
-0@@ -3968,6 +3968,10 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)pte_t pte;vm_fault_t ret =0 ;void *shadow =NULL ; +int nr_pages =1 ; +unsigned long page_idx =0 ;Quoted 8 lines
mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 3974
Quoted 2 lines
mm/memory.c +2
-0+unsigned long address = vmf->address; +pte_t *ptep;nit: Personally I'd prefer all these to get initialised just before the "if (
folio_test_large()..." block below. That way it is clear they are fresh (incase any logic between here and there makes an adjustment) and its clear that they are only to be used after that block (the compiler will warn if using an uninitialized value).right. I agree this will make the code more readable.
Quoted 32 lines
Quoted 32 lines
mm/memory.c +24
-0if (!pte_unmap_same (vmf)) goto out;@@ -4166,6 +4170,36 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) goto out_nomap; } + ptep = vmf->pte; +if (folio_test_large (folio) &&folio_test_swapcache (folio)) { +int nr =folio_nr_pages (folio); +unsigned long idx =folio_page_idx (folio, page); +unsigned long folio_start = vmf->address - idx *PAGE_SIZE ; +unsigned long folio_end = folio_start + nr *PAGE_SIZE ; +pte_t *folio_ptep; +pte_t folio_pte; + +if (unlikely (folio_start <max (vmf->address &PMD_MASK , vma->vm_start))) + goto check_folio; +if (unlikely (folio_end >pmd_addr_end (vmf->address, vma->vm_end))) + goto check_folio; + + folio_ptep = vmf->pte - idx; + folio_pte =ptep_get (folio_ptep); +if (!pte_same (folio_pte,pte_move_swp_offset (vmf->orig_pte, -idx)) || +swap_pte_batch (folio_ptep, nr, folio_pte) != nr) + goto check_folio; + + page_idx = idx; + address = folio_start; + ptep = folio_ptep; + nr_pages = nr;Quoted 8 lines
mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 4201
Quoted 5 lines
mm/memory.c +5
-0+ entry = folio->swap; + page = &folio->page; + } + +check_folio:Is this still the correct label name, given the checks are now above the new block? Perhaps "
one_page" or something like that?not quite sure about this, as the code after
one_pagecan bemultiple_pages. On the other hand, it seems we are really checking folio after "check_folio" :-)
BUG_ON(!folio_test_anon(folio) && folio_test_mappedtodisk(folio));BUG_ON(folio_test_anon(folio) && PageAnonExclusive(page));/*
* Check under PT lock (to protect against concurrentfork()sharing
* the swap entry concurrently) for certainly exclusive pages. */
if (!folio_test_ksm(folio)) {Quoted 21 lines
Quoted 21 lines
mm/memory.c +5
-3+/* * PG_anon_exclusive reuses PG_mappedtodisk for anon pages. A swap pte * must never point at an anonymous page in the swapcache that is @@ -4225,12 +4259,13 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)* We're already holding a reference on the page but haven't mapped it * yet. */ -swap_free_nr (entry,1 ); +swap_free_nr (entry, nr_pages);if (should_try_to_free_swap (folio, vma, vmf->flags))folio_free_swap (folio); -inc_mm_counter (vma->vm_mm,MM_ANONPAGES ); -dec_mm_counter (vma->vm_mm,MM_SWAPENTS ); +folio_ref_add (folio, nr_pages -1 ); +add_mm_counter (vma->vm_mm,MM_ANONPAGES , nr_pages); +add_mm_counter (vma->vm_mm,MM_SWAPENTS , -nr_pages); pte =mk_pte (page, vma->vm_page_prot);Quoted 12 lines
mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 4279
Quoted 8 lines
mm/memory.c +2
-1/* @@ -4240,34 +4275,35 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)* exclusivity. */ if (!folio_test_ksm (folio) && - (exclusive ||folio_ref_count (folio) ==1 )) { + (exclusive || (folio_ref_count (folio) == nr_pages && +folio_nr_pages (folio) == nr_pages))) {I think in practice there is no change here? If
nr_pages> 1 then the folio is in the swapcache, so there is an extra ref on it? I agree with the change for robustness sake. Just checking my understanding.This is the code showing we are reusing/(mkwrite) a folio either
1. we meet a small folio and we are the only one hitting the small folio
2. we meet a large folio and we are the only one hitting the large folio
Quoted 5 lines
any corner cases besides the above two seems difficult. for example,
while we hit a large folio in swapcache but if we can't entirely map it (nr_pages==1) due to partial unmap, we will have
folio_ref_count(folio)==nr_pages== 1
No, there would be other references from the swapcache and folio_ref_count(folio) > 1. See my other reply.
Signature · 3 lines
Cheers, David / dhildenb
Re: [PATCH v3 6/6] mm: swap: entirely map large folios found in swapcache
Quoted 140 lines · David Hildenbrand
Quoted 138 lines · Barry Song
Quoted 32 lines · Ryan Roberts
Quoted 11 lines · Barry Song
From: Chuanhua Han <[email protected]>
When a large folio is found in the swapcache, the current implementation requires calling
do_swap_page()nr_pagestimes, resulting innr_pagespage faults. This patch opts to map the entire large folio at once to minimize page faults. Additionally, redundant checks and early exits for ARM64 MTE restoring are removed.Signed-off-by Chuanhua Han <[email protected]>Co-developed-by Barry Song <[email protected]>Signed-off-by Barry Song <[email protected]>With the suggested changes below:
Reviewed-by Ryan Roberts <[email protected]>Quoted 14 lines
--- mm/memory.c | 60 ++++++++++++++++++++++++++++++++++++++++++----------- 1 file changed, 48 insertions(+), 12 deletions(-)mm/memory.c +2
-0@@ -3968,6 +3968,10 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)pte_t pte;vm_fault_t ret =0 ;void *shadow =NULL ; +int nr_pages =1 ; +unsigned long page_idx =0 ;Quoted 8 lines
mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 3974
Quoted 2 lines
mm/memory.c +2
-0+unsigned long address = vmf->address; +pte_t *ptep;nit: Personally I'd prefer all these to get initialised just before the "if (
folio_test_large()..." block below. That way it is clear they are fresh (incase any logic between here and there makes an adjustment) and its clear that they are only to be used after that block (the compiler will warn if using an uninitialized value).right. I agree this will make the code more readable.
Quoted 32 lines
Quoted 32 lines
mm/memory.c +24
-0if (!pte_unmap_same (vmf)) goto out;@@ -4166,6 +4170,36 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) goto out_nomap; } + ptep = vmf->pte; +if (folio_test_large (folio) &&folio_test_swapcache (folio)) { +int nr =folio_nr_pages (folio); +unsigned long idx =folio_page_idx (folio, page); +unsigned long folio_start = vmf->address - idx *PAGE_SIZE ; +unsigned long folio_end = folio_start + nr *PAGE_SIZE ; +pte_t *folio_ptep; +pte_t folio_pte; + +if (unlikely (folio_start <max (vmf->address &PMD_MASK , vma->vm_start))) + goto check_folio; +if (unlikely (folio_end >pmd_addr_end (vmf->address, vma->vm_end))) + goto check_folio; + + folio_ptep = vmf->pte - idx; + folio_pte =ptep_get (folio_ptep); +if (!pte_same (folio_pte,pte_move_swp_offset (vmf->orig_pte, -idx)) || +swap_pte_batch (folio_ptep, nr, folio_pte) != nr) + goto check_folio; + + page_idx = idx; + address = folio_start; + ptep = folio_ptep; + nr_pages = nr;Quoted 8 lines
mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 4201
Quoted 5 lines
mm/memory.c +5
-0+ entry = folio->swap; + page = &folio->page; + } + +check_folio:Is this still the correct label name, given the checks are now above the new block? Perhaps "
one_page" or something like that?not quite sure about this, as the code after
one_pagecan bemultiple_pages. On the other hand, it seems we are really checking folio after "check_folio" :-)
BUG_ON(!folio_test_anon(folio) && folio_test_mappedtodisk(folio));BUG_ON(folio_test_anon(folio) && PageAnonExclusive(page));/*
* Check under PT lock (to protect against concurrentfork()sharing
* the swap entry concurrently) for certainly exclusive pages. */
if (!folio_test_ksm(folio)) {Quoted 21 lines
Quoted 21 lines
mm/memory.c +5
-3+/* * PG_anon_exclusive reuses PG_mappedtodisk for anon pages. A swap pte * must never point at an anonymous page in the swapcache that is @@ -4225,12 +4259,13 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)* We're already holding a reference on the page but haven't mapped it * yet. */ -swap_free_nr (entry,1 ); +swap_free_nr (entry, nr_pages);if (should_try_to_free_swap (folio, vma, vmf->flags))folio_free_swap (folio); -inc_mm_counter (vma->vm_mm,MM_ANONPAGES ); -dec_mm_counter (vma->vm_mm,MM_SWAPENTS ); +folio_ref_add (folio, nr_pages -1 ); +add_mm_counter (vma->vm_mm,MM_ANONPAGES , nr_pages); +add_mm_counter (vma->vm_mm,MM_SWAPENTS , -nr_pages); pte =mk_pte (page, vma->vm_page_prot);Quoted 12 lines
mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 4279
Quoted 8 lines
mm/memory.c +2
-1/* @@ -4240,34 +4275,35 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)* exclusivity. */ if (!folio_test_ksm (folio) && - (exclusive ||folio_ref_count (folio) ==1 )) { + (exclusive || (folio_ref_count (folio) == nr_pages && +folio_nr_pages (folio) == nr_pages))) {I think in practice there is no change here? If
nr_pages> 1 then the folio is in the swapcache, so there is an extra ref on it? I agree with the change for robustness sake. Just checking my understanding.This is the code showing we are reusing/(mkwrite) a folio either
1. we meet a small folio and we are the only one hitting the small folio
2. we meet a large folio and we are the only one hitting the large folio
Quoted 8 lines
Quoted 5 lines
any corner cases besides the above two seems difficult. for example,
while we hit a large folio in swapcache but if we can't entirely map it (nr_pages==1) due to partial unmap, we will have
folio_ref_count(folio)==nr_pages== 1No, there would be other references from the swapcache and
folio_ref_count(folio)> 1. See my other reply.
right. can be clearer by:
@@ -4263,7 +4264,6 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)
if (should_try_to_free_swap(folio, vma, vmf->flags))
folio_free_swap(folio);- folio_ref_add(folio, nr_pages - 1);
add_mm_counter(vma->vm_mm, MM_ANONPAGES, nr_pages);
add_mm_counter(vma->vm_mm, MM_SWAPENTS, -nr_pages);
pte = mk_pte(page, vma->vm_page_prot);
@@ -4275,14 +4275,14 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)
* exclusivity.
*/
if (!folio_test_ksm(folio) &&
- (exclusive || (folio_ref_count(folio) == nr_pages &&
- folio_nr_pages(folio) == nr_pages))) {
+ (exclusive || folio_ref_count(folio) == 1)) {
if (vmf->flags & FAULT_FLAG_WRITE) {
pte = maybe_mkwrite(pte_mkdirty(pte), vma);
vmf->flags &= ~FAULT_FLAG_WRITE;
}
rmap_flags |= RMAP_EXCLUSIVE;
}
+ folio_ref_add(folio, nr_pages - 1);
flush_icache_pages(vma, page, nr_pages);
if (pte_swp_soft_dirty(vmf->orig_pte))
pte = pte_mksoft_dirty(pte);Quoted 6 lines
--
Cheers,David / dhildenb
Re: [PATCH v3 6/6] mm: swap: entirely map large folios found in swapcache
Quoted 144 lines · Barry Song
Quoted 142 lines · David Hildenbrand
Quoted 140 lines · Barry Song
Quoted 32 lines · Ryan Roberts
Quoted 11 lines · Barry Song
From: Chuanhua Han <[email protected]>
When a large folio is found in the swapcache, the current implementation requires calling
do_swap_page()nr_pagestimes, resulting innr_pagespage faults. This patch opts to map the entire large folio at once to minimize page faults. Additionally, redundant checks and early exits for ARM64 MTE restoring are removed.Signed-off-by Chuanhua Han <[email protected]>Co-developed-by Barry Song <[email protected]>Signed-off-by Barry Song <[email protected]>With the suggested changes below:
Reviewed-by Ryan Roberts <[email protected]>Quoted 14 lines
--- mm/memory.c | 60 ++++++++++++++++++++++++++++++++++++++++++----------- 1 file changed, 48 insertions(+), 12 deletions(-)mm/memory.c +2
-0@@ -3968,6 +3968,10 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)pte_t pte;vm_fault_t ret =0 ;void *shadow =NULL ; +int nr_pages =1 ; +unsigned long page_idx =0 ;Quoted 8 lines
mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 3974
Quoted 2 lines
mm/memory.c +2
-0+unsigned long address = vmf->address; +pte_t *ptep;nit: Personally I'd prefer all these to get initialised just before the "if (
folio_test_large()..." block below. That way it is clear they are fresh (incase any logic between here and there makes an adjustment) and its clear that they are only to be used after that block (the compiler will warn if using an uninitialized value).right. I agree this will make the code more readable.
Quoted 32 lines
Quoted 32 lines
mm/memory.c +24
-0if (!pte_unmap_same (vmf)) goto out;@@ -4166,6 +4170,36 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) goto out_nomap; } + ptep = vmf->pte; +if (folio_test_large (folio) &&folio_test_swapcache (folio)) { +int nr =folio_nr_pages (folio); +unsigned long idx =folio_page_idx (folio, page); +unsigned long folio_start = vmf->address - idx *PAGE_SIZE ; +unsigned long folio_end = folio_start + nr *PAGE_SIZE ; +pte_t *folio_ptep; +pte_t folio_pte; + +if (unlikely (folio_start <max (vmf->address &PMD_MASK , vma->vm_start))) + goto check_folio; +if (unlikely (folio_end >pmd_addr_end (vmf->address, vma->vm_end))) + goto check_folio; + + folio_ptep = vmf->pte - idx; + folio_pte =ptep_get (folio_ptep); +if (!pte_same (folio_pte,pte_move_swp_offset (vmf->orig_pte, -idx)) || +swap_pte_batch (folio_ptep, nr, folio_pte) != nr) + goto check_folio; + + page_idx = idx; + address = folio_start; + ptep = folio_ptep; + nr_pages = nr;Quoted 8 lines
mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 4201
Quoted 5 lines
mm/memory.c +5
-0+ entry = folio->swap; + page = &folio->page; + } + +check_folio:Is this still the correct label name, given the checks are now above the new block? Perhaps "
one_page" or something like that?not quite sure about this, as the code after
one_pagecan bemultiple_pages. On the other hand, it seems we are really checking folio after "check_folio" :-)
BUG_ON(!folio_test_anon(folio) && folio_test_mappedtodisk(folio));BUG_ON(folio_test_anon(folio) && PageAnonExclusive(page));/*
* Check under PT lock (to protect against concurrentfork()sharing
* the swap entry concurrently) for certainly exclusive pages. */
if (!folio_test_ksm(folio)) {Quoted 21 lines
Quoted 21 lines
mm/memory.c +5
-3+/* * PG_anon_exclusive reuses PG_mappedtodisk for anon pages. A swap pte * must never point at an anonymous page in the swapcache that is @@ -4225,12 +4259,13 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)* We're already holding a reference on the page but haven't mapped it * yet. */ -swap_free_nr (entry,1 ); +swap_free_nr (entry, nr_pages);if (should_try_to_free_swap (folio, vma, vmf->flags))folio_free_swap (folio); -inc_mm_counter (vma->vm_mm,MM_ANONPAGES ); -dec_mm_counter (vma->vm_mm,MM_SWAPENTS ); +folio_ref_add (folio, nr_pages -1 ); +add_mm_counter (vma->vm_mm,MM_ANONPAGES , nr_pages); +add_mm_counter (vma->vm_mm,MM_SWAPENTS , -nr_pages); pte =mk_pte (page, vma->vm_page_prot);Quoted 12 lines
mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 4279
Quoted 8 lines
mm/memory.c +2
-1/* @@ -4240,34 +4275,35 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)* exclusivity. */ if (!folio_test_ksm (folio) && - (exclusive ||folio_ref_count (folio) ==1 )) { + (exclusive || (folio_ref_count (folio) == nr_pages && +folio_nr_pages (folio) == nr_pages))) {I think in practice there is no change here? If
nr_pages> 1 then the folio is in the swapcache, so there is an extra ref on it? I agree with the change for robustness sake. Just checking my understanding.This is the code showing we are reusing/(mkwrite) a folio either
1. we meet a small folio and we are the only one hitting the small folio
2. we meet a large folio and we are the only one hitting the large folioany corner cases besides the above two seems difficult. for example,
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
while we hit a large folio in swapcache but if we can't entirely map it (nr_pages==1) due to partial unmap, we will have
folio_ref_count(folio)==nr_pages== 1No, there would be other references from the swapcache and
folio_ref_count(folio)> 1. See my other reply.right. can be clearer by:
Wait, do we still need folio_nr_pages(folio) == nr_pages even if we use folio_ref_count(folio) == 1 and moving folio_ref_add(folio, nr_pages - 1)?
one case is that we have a large folio with 16 PTEs, and we unmap 15 swap PTE entries, thus we have only one swap entry left. Then we hit the large folio in swapcache. but we have only one PTE thus we will map only one PTE. lacking folio_nr_pages(folio) == nr_pages, we reuse the large folio for one PTE. with it, do_wp_page() will migrate the large folio to a small one?
1AM, tired and sleepy. not quite sure I am correct. I look forward to seeing your reply tomorrow morning :-)
Quoted 34 lines
mm/memory.c +2
-3@@ -4263,7 +4264,6 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)if (should_try_to_free_swap (folio, vma, vmf->flags))folio_free_swap (folio); -folio_ref_add (folio, nr_pages -1 );add_mm_counter (vma->vm_mm,MM_ANONPAGES , nr_pages);add_mm_counter (vma->vm_mm,MM_SWAPENTS , -nr_pages); pte =mk_pte (page, vma->vm_page_prot);@@ -4275,14 +4275,14 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)* exclusivity. */ if (!folio_test_ksm (folio) && - (exclusive || (folio_ref_count (folio) == nr_pages && -folio_nr_pages (folio) == nr_pages))) { + (exclusive ||folio_ref_count (folio) ==1 )) {if (vmf->flags &FAULT_FLAG_WRITE ) { pte =maybe_mkwrite (pte_mkdirty (pte), vma); vmf->flags &= ~FAULT_FLAG_WRITE ; } rmap_flags |=RMAP_EXCLUSIVE ; } +folio_ref_add (folio, nr_pages -1 );flush_icache_pages (vma, page, nr_pages);if (pte_swp_soft_dirty (vmf->orig_pte)) pte =pte_mksoft_dirty (pte);Quoted 8 lines
--
Cheers,David / dhildenb
Re: [PATCH v3 6/6] mm: swap: entirely map large folios found in swapcache
Quoted 149 lines · Barry Song
Quoted 148 lines · Barry Song
Quoted 146 lines · David Hildenbrand
Quoted 143 lines · Barry Song
Quoted 32 lines · Ryan Roberts
Quoted 11 lines · Barry Song
From: Chuanhua Han <[email protected]>
When a large folio is found in the swapcache, the current implementation requires calling
do_swap_page()nr_pagestimes, resulting innr_pagespage faults. This patch opts to map the entire large folio at once to minimize page faults. Additionally, redundant checks and early exits for ARM64 MTE restoring are removed.Signed-off-by Chuanhua Han <[email protected]>Co-developed-by Barry Song <[email protected]>Signed-off-by Barry Song <[email protected]>With the suggested changes below:
Reviewed-by Ryan Roberts <[email protected]>Quoted 14 lines
--- mm/memory.c | 60 ++++++++++++++++++++++++++++++++++++++++++----------- 1 file changed, 48 insertions(+), 12 deletions(-)mm/memory.c +2
-0@@ -3968,6 +3968,10 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)pte_t pte;vm_fault_t ret =0 ;void *shadow =NULL ; +int nr_pages =1 ; +unsigned long page_idx =0 ;Quoted 8 lines
mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 3974
Quoted 2 lines
mm/memory.c +2
-0+unsigned long address = vmf->address; +pte_t *ptep;nit: Personally I'd prefer all these to get initialised just before the "if (
folio_test_large()..." block below. That way it is clear they are fresh (incase any logic between here and there makes an adjustment) and its clear that they are only to be used after that block (the compiler will warn if using an uninitialized value).right. I agree this will make the code more readable.
Quoted 32 lines
Quoted 32 lines
mm/memory.c +24
-0if (!pte_unmap_same (vmf)) goto out;@@ -4166,6 +4170,36 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) goto out_nomap; } + ptep = vmf->pte; +if (folio_test_large (folio) &&folio_test_swapcache (folio)) { +int nr =folio_nr_pages (folio); +unsigned long idx =folio_page_idx (folio, page); +unsigned long folio_start = vmf->address - idx *PAGE_SIZE ; +unsigned long folio_end = folio_start + nr *PAGE_SIZE ; +pte_t *folio_ptep; +pte_t folio_pte; + +if (unlikely (folio_start <max (vmf->address &PMD_MASK , vma->vm_start))) + goto check_folio; +if (unlikely (folio_end >pmd_addr_end (vmf->address, vma->vm_end))) + goto check_folio; + + folio_ptep = vmf->pte - idx; + folio_pte =ptep_get (folio_ptep); +if (!pte_same (folio_pte,pte_move_swp_offset (vmf->orig_pte, -idx)) || +swap_pte_batch (folio_ptep, nr, folio_pte) != nr) + goto check_folio; + + page_idx = idx; + address = folio_start; + ptep = folio_ptep; + nr_pages = nr;Quoted 8 lines
mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 4201
Quoted 5 lines
mm/memory.c +5
-0+ entry = folio->swap; + page = &folio->page; + } + +check_folio:Is this still the correct label name, given the checks are now above the new block? Perhaps "
one_page" or something like that?not quite sure about this, as the code after
one_pagecan bemultiple_pages. On the other hand, it seems we are really checking folio after "check_folio" :-)
BUG_ON(!folio_test_anon(folio) && folio_test_mappedtodisk(folio));BUG_ON(folio_test_anon(folio) && PageAnonExclusive(page));/*
* Check under PT lock (to protect against concurrentfork()sharing
* the swap entry concurrently) for certainly exclusive pages. */
if (!folio_test_ksm(folio)) {Quoted 21 lines
Quoted 21 lines
mm/memory.c +5
-3+/* * PG_anon_exclusive reuses PG_mappedtodisk for anon pages. A swap pte * must never point at an anonymous page in the swapcache that is @@ -4225,12 +4259,13 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)* We're already holding a reference on the page but haven't mapped it * yet. */ -swap_free_nr (entry,1 ); +swap_free_nr (entry, nr_pages);if (should_try_to_free_swap (folio, vma, vmf->flags))folio_free_swap (folio); -inc_mm_counter (vma->vm_mm,MM_ANONPAGES ); -dec_mm_counter (vma->vm_mm,MM_SWAPENTS ); +folio_ref_add (folio, nr_pages -1 ); +add_mm_counter (vma->vm_mm,MM_ANONPAGES , nr_pages); +add_mm_counter (vma->vm_mm,MM_SWAPENTS , -nr_pages); pte =mk_pte (page, vma->vm_page_prot);Quoted 12 lines
mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 4279
Quoted 8 lines
mm/memory.c +2
-1/* @@ -4240,34 +4275,35 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)* exclusivity. */ if (!folio_test_ksm (folio) && - (exclusive ||folio_ref_count (folio) ==1 )) { + (exclusive || (folio_ref_count (folio) == nr_pages && +folio_nr_pages (folio) == nr_pages))) {I think in practice there is no change here? If
nr_pages> 1 then the folio is in the swapcache, so there is an extra ref on it? I agree with the change for robustness sake. Just checking my understanding.This is the code showing we are reusing/(mkwrite) a folio either
1. we meet a small folio and we are the only one hitting the small folio
2. we meet a large folio and we are the only one hitting the large folioany corner cases besides the above two seems difficult. for example,
while we hit a large folio in swapcache but if we can't entirely map it (nr_pages==1) due to partial unmap, we will have
folio_ref_count(folio)==nr_pages== 1
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
No, there would be other references from the swapcache and
folio_ref_count(folio)> 1. See my other reply.right. can be clearer by:
Wait, do we still need
folio_nr_pages(folio)==nr_pageseven if we usefolio_ref_count(folio)== 1 and movingfolio_ref_add(folio, nr_pages - 1)?
I don't think that we will "need" it.
Quoted 7 lines
one case is that we have a large folio with 16 PTEs, and we unmap 15 swap PTE entries, thus we have only one swap entry left. Then we hit the large folio in swapcache. but we have only one PTE thus we will map only one PTE. lacking
folio_nr_pages(folio)==nr_pages, we reuse the large folio for one PTE. with it,do_wp_page()will migrate the large folio to a small one?
We will set PAE bit and do_wp_page() will unconditionally reuse that page.
Note that this is the same as if we had pte_swp_exclusive() set and would have run into "exclusive=true" here.
If we'd want a similar "optimization" as we have in wp_can_reuse_anon_folio(), you'd want something like
exclusive || (folio_ref_count(folio) == 1 &&
(!folio_test_large(folio) || nr_pages > 1)
... but I am not sure if that is really worth the complexity here.
Quoted 3 lines
1AM, tired and sleepy. not quite sure I am correct. I look forward to seeing your reply tomorrow morning :-)
Heh, no need to dream about this ;)
Signature · 3 lines
Cheers, David / dhildenb
Re: [PATCH v3 6/6] mm: swap: entirely map large folios found in swapcache
Quoted 171 lines · David Hildenbrand
Quoted 149 lines · Barry Song
Quoted 148 lines · Barry Song
Quoted 146 lines · David Hildenbrand
Quoted 143 lines · Barry Song
Quoted 32 lines · Ryan Roberts
Quoted 11 lines · Barry Song
From: Chuanhua Han <[email protected]>
When a large folio is found in the swapcache, the current implementation requires calling
do_swap_page()nr_pagestimes, resulting innr_pagespage faults. This patch opts to map the entire large folio at once to minimize page faults. Additionally, redundant checks and early exits for ARM64 MTE restoring are removed.Signed-off-by Chuanhua Han <[email protected]>Co-developed-by Barry Song <[email protected]>Signed-off-by Barry Song <[email protected]>With the suggested changes below:
Reviewed-by Ryan Roberts <[email protected]>Quoted 14 lines
--- mm/memory.c | 60 ++++++++++++++++++++++++++++++++++++++++++----------- 1 file changed, 48 insertions(+), 12 deletions(-)mm/memory.c +2
-0@@ -3968,6 +3968,10 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)pte_t pte;vm_fault_t ret =0 ;void *shadow =NULL ; +int nr_pages =1 ; +unsigned long page_idx =0 ;Quoted 8 lines
mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 3974
Quoted 2 lines
mm/memory.c +2
-0+unsigned long address = vmf->address; +pte_t *ptep;nit: Personally I'd prefer all these to get initialised just before the "if (
folio_test_large()..." block below. That way it is clear they are fresh (incase any logic between here and there makes an adjustment) and its clear that they are only to be used after that block (the compiler will warn if using an uninitialized value).right. I agree this will make the code more readable.
Quoted 32 lines
Quoted 32 lines
mm/memory.c +24
-0if (!pte_unmap_same (vmf)) goto out;@@ -4166,6 +4170,36 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) goto out_nomap; } + ptep = vmf->pte; +if (folio_test_large (folio) &&folio_test_swapcache (folio)) { +int nr =folio_nr_pages (folio); +unsigned long idx =folio_page_idx (folio, page); +unsigned long folio_start = vmf->address - idx *PAGE_SIZE ; +unsigned long folio_end = folio_start + nr *PAGE_SIZE ; +pte_t *folio_ptep; +pte_t folio_pte; + +if (unlikely (folio_start <max (vmf->address &PMD_MASK , vma->vm_start))) + goto check_folio; +if (unlikely (folio_end >pmd_addr_end (vmf->address, vma->vm_end))) + goto check_folio; + + folio_ptep = vmf->pte - idx; + folio_pte =ptep_get (folio_ptep); +if (!pte_same (folio_pte,pte_move_swp_offset (vmf->orig_pte, -idx)) || +swap_pte_batch (folio_ptep, nr, folio_pte) != nr) + goto check_folio; + + page_idx = idx; + address = folio_start; + ptep = folio_ptep; + nr_pages = nr;Quoted 8 lines
mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 4201
Quoted 5 lines
mm/memory.c +5
-0+ entry = folio->swap; + page = &folio->page; + } + +check_folio:Is this still the correct label name, given the checks are now above the new block? Perhaps "
one_page" or something like that?not quite sure about this, as the code after
one_pagecan bemultiple_pages. On the other hand, it seems we are really checking folio after "check_folio" :-)
BUG_ON(!folio_test_anon(folio) && folio_test_mappedtodisk(folio));BUG_ON(folio_test_anon(folio) && PageAnonExclusive(page));/*
* Check under PT lock (to protect against concurrentfork()sharing
* the swap entry concurrently) for certainly exclusive pages. */
if (!folio_test_ksm(folio)) {Quoted 21 lines
Quoted 21 lines
mm/memory.c +5
-3+/* * PG_anon_exclusive reuses PG_mappedtodisk for anon pages. A swap pte * must never point at an anonymous page in the swapcache that is @@ -4225,12 +4259,13 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)* We're already holding a reference on the page but haven't mapped it * yet. */ -swap_free_nr (entry,1 ); +swap_free_nr (entry, nr_pages);if (should_try_to_free_swap (folio, vma, vmf->flags))folio_free_swap (folio); -inc_mm_counter (vma->vm_mm,MM_ANONPAGES ); -dec_mm_counter (vma->vm_mm,MM_SWAPENTS ); +folio_ref_add (folio, nr_pages -1 ); +add_mm_counter (vma->vm_mm,MM_ANONPAGES , nr_pages); +add_mm_counter (vma->vm_mm,MM_SWAPENTS , -nr_pages); pte =mk_pte (page, vma->vm_page_prot);Quoted 12 lines
mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 4279
Quoted 8 lines
mm/memory.c +2
-1/* @@ -4240,34 +4275,35 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)* exclusivity. */ if (!folio_test_ksm (folio) && - (exclusive ||folio_ref_count (folio) ==1 )) { + (exclusive || (folio_ref_count (folio) == nr_pages && +folio_nr_pages (folio) == nr_pages))) {I think in practice there is no change here? If
nr_pages> 1 then the folio is in the swapcache, so there is an extra ref on it? I agree with the change for robustness sake. Just checking my understanding.This is the code showing we are reusing/(mkwrite) a folio either
1. we meet a small folio and we are the only one hitting the small folio
2. we meet a large folio and we are the only one hitting the large folioany corner cases besides the above two seems difficult. for example,
while we hit a large folio in swapcache but if we can't entirely map it (nr_pages==1) due to partial unmap, we will have
folio_ref_count(folio)==nr_pages== 1Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
No, there would be other references from the swapcache and
folio_ref_count(folio)> 1. See my other reply.right. can be clearer by:
Wait, do we still need
folio_nr_pages(folio)==nr_pageseven if we usefolio_ref_count(folio)== 1 and movingfolio_ref_add(folio, nr_pages - 1)?I don't think that we will "need" it.
Quoted 7 lines
one case is that we have a large folio with 16 PTEs, and we unmap 15 swap PTE entries, thus we have only one swap entry left. Then we hit the large folio in swapcache. but we have only one PTE thus we will map only one PTE. lacking
folio_nr_pages(folio)==nr_pages, we reuse the large folio for one PTE. with it,do_wp_page()will migrate the large folio to a small one?We will set PAE bit and
do_wp_page()will unconditionally reuse that page.
Quoted 8 lines
Note that this is the same as if we had
pte_swp_exclusive()set and would have run into "exclusive=true" here.If we'd want a similar "optimization" as we have in
wp_can_reuse_anon_folio(), you'd want something likeexclusive || (
folio_ref_count(folio)== 1 &&(!folio_test_large(folio) || nr_pages > 1)
I feel like
A : !folio_test_large(folio) || nr_pages > 1
equals
B: folio_nr_pages(folio) == nr_pages
if folio is small, folio_test_large(folio) is false, both A and B will be true; if folio is large, and we map the whole large folio, A will be true because of nr_pages > 1;
B is also true;
if folio is large, and we map single one PTE, A will be false;
B is also false, because nr_pages == 1 but folio_nr_pages(folio) > 1;
right?
However, I agree that delving into this complexity might not be necessary at the moment.
Quoted 7 lines
... but I am not sure if that is really worth the complexity here.
Quoted 3 lines
1AM, tired and sleepy. not quite sure I am correct. I look forward to seeing your reply tomorrow morning :-)
Quoted 6 lines
Heh, no need to dream about this ;)
--
Cheers,David / dhildenb
Thanks
Barry
Re: [PATCH v3 6/6] mm: swap: entirely map large folios found in swapcache
Quoted 189 lines · Barry Song
Quoted 171 lines · David Hildenbrand
Quoted 149 lines · Barry Song
Quoted 148 lines · Barry Song
Quoted 146 lines · David Hildenbrand
Quoted 143 lines · Barry Song
Quoted 32 lines · Ryan Roberts
Quoted 11 lines · Barry Song
From: Chuanhua Han <[email protected]>
When a large folio is found in the swapcache, the current implementation requires calling
do_swap_page()nr_pagestimes, resulting innr_pagespage faults. This patch opts to map the entire large folio at once to minimize page faults. Additionally, redundant checks and early exits for ARM64 MTE restoring are removed.Signed-off-by Chuanhua Han <[email protected]>Co-developed-by Barry Song <[email protected]>Signed-off-by Barry Song <[email protected]>With the suggested changes below:
Reviewed-by Ryan Roberts <[email protected]>Quoted 14 lines
--- mm/memory.c | 60 ++++++++++++++++++++++++++++++++++++++++++----------- 1 file changed, 48 insertions(+), 12 deletions(-)mm/memory.c +2
-0@@ -3968,6 +3968,10 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)pte_t pte;vm_fault_t ret =0 ;void *shadow =NULL ; +int nr_pages =1 ; +unsigned long page_idx =0 ;Quoted 8 lines
mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 3974
Quoted 2 lines
mm/memory.c +2
-0+unsigned long address = vmf->address; +pte_t *ptep;nit: Personally I'd prefer all these to get initialised just before the "if (
folio_test_large()..." block below. That way it is clear they are fresh (incase any logic between here and there makes an adjustment) and its clear that they are only to be used after that block (the compiler will warn if using an uninitialized value).right. I agree this will make the code more readable.
Quoted 32 lines
Quoted 32 lines
mm/memory.c +24
-0if (!pte_unmap_same (vmf)) goto out;@@ -4166,6 +4170,36 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) goto out_nomap; } + ptep = vmf->pte; +if (folio_test_large (folio) &&folio_test_swapcache (folio)) { +int nr =folio_nr_pages (folio); +unsigned long idx =folio_page_idx (folio, page); +unsigned long folio_start = vmf->address - idx *PAGE_SIZE ; +unsigned long folio_end = folio_start + nr *PAGE_SIZE ; +pte_t *folio_ptep; +pte_t folio_pte; + +if (unlikely (folio_start <max (vmf->address &PMD_MASK , vma->vm_start))) + goto check_folio; +if (unlikely (folio_end >pmd_addr_end (vmf->address, vma->vm_end))) + goto check_folio; + + folio_ptep = vmf->pte - idx; + folio_pte =ptep_get (folio_ptep); +if (!pte_same (folio_pte,pte_move_swp_offset (vmf->orig_pte, -idx)) || +swap_pte_batch (folio_ptep, nr, folio_pte) != nr) + goto check_folio; + + page_idx = idx; + address = folio_start; + ptep = folio_ptep; + nr_pages = nr;Quoted 8 lines
mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 4201
Quoted 5 lines
mm/memory.c +5
-0+ entry = folio->swap; + page = &folio->page; + } + +check_folio:Is this still the correct label name, given the checks are now above the new block? Perhaps "
one_page" or something like that?not quite sure about this, as the code after
one_pagecan bemultiple_pages. On the other hand, it seems we are really checking folio after "check_folio" :-)
BUG_ON(!folio_test_anon(folio) && folio_test_mappedtodisk(folio));BUG_ON(folio_test_anon(folio) && PageAnonExclusive(page));/*
* Check under PT lock (to protect against concurrentfork()sharing
* the swap entry concurrently) for certainly exclusive pages. */
if (!folio_test_ksm(folio)) {Quoted 21 lines
Quoted 21 lines
mm/memory.c +5
-3+/* * PG_anon_exclusive reuses PG_mappedtodisk for anon pages. A swap pte * must never point at an anonymous page in the swapcache that is @@ -4225,12 +4259,13 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)* We're already holding a reference on the page but haven't mapped it * yet. */ -swap_free_nr (entry,1 ); +swap_free_nr (entry, nr_pages);if (should_try_to_free_swap (folio, vma, vmf->flags))folio_free_swap (folio); -inc_mm_counter (vma->vm_mm,MM_ANONPAGES ); -dec_mm_counter (vma->vm_mm,MM_SWAPENTS ); +folio_ref_add (folio, nr_pages -1 ); +add_mm_counter (vma->vm_mm,MM_ANONPAGES , nr_pages); +add_mm_counter (vma->vm_mm,MM_SWAPENTS , -nr_pages); pte =mk_pte (page, vma->vm_page_prot);Quoted 12 lines
mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 4279
Quoted 8 lines
mm/memory.c +2
-1/* @@ -4240,34 +4275,35 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)* exclusivity. */ if (!folio_test_ksm (folio) && - (exclusive ||folio_ref_count (folio) ==1 )) { + (exclusive || (folio_ref_count (folio) == nr_pages && +folio_nr_pages (folio) == nr_pages))) {I think in practice there is no change here? If
nr_pages> 1 then the folio is in the swapcache, so there is an extra ref on it? I agree with the change for robustness sake. Just checking my understanding.This is the code showing we are reusing/(mkwrite) a folio either
1. we meet a small folio and we are the only one hitting the small folio
2. we meet a large folio and we are the only one hitting the large folioany corner cases besides the above two seems difficult. for example,
while we hit a large folio in swapcache but if we can't entirely map it (nr_pages==1) due to partial unmap, we will have
folio_ref_count(folio)==nr_pages== 1Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
No, there would be other references from the swapcache and
folio_ref_count(folio)> 1. See my other reply.right. can be clearer by:
Wait, do we still need
folio_nr_pages(folio)==nr_pageseven if we usefolio_ref_count(folio)== 1 and movingfolio_ref_add(folio, nr_pages - 1)?I don't think that we will "need" it.
Quoted 7 lines
one case is that we have a large folio with 16 PTEs, and we unmap 15 swap PTE entries, thus we have only one swap entry left. Then we hit the large folio in swapcache. but we have only one PTE thus we will map only one PTE. lacking
folio_nr_pages(folio)==nr_pages, we reuse the large folio for one PTE. with it,do_wp_page()will migrate the large folio to a small one?We will set PAE bit and
do_wp_page()will unconditionally reuse that page.Quoted 8 lines
Note that this is the same as if we had
pte_swp_exclusive()set and would have run into "exclusive=true" here.If we'd want a similar "optimization" as we have in
wp_can_reuse_anon_folio(), you'd want something likeexclusive || (
folio_ref_count(folio)== 1 &&(!folio_test_large(folio) || nr_pages > 1)I feel like
A : !folio_test_large(folio) ||
nr_pages> 1equals
B:
folio_nr_pages(folio)==nr_pages
Quoted 8 lines
if folio is small,
folio_test_large(folio)is false, both A and B will be true; if folio is large, and we map the whole large folio, A will be true because ofnr_pages> 1;
B is also true;
if folio is large, and we map single one PTE, A will be false;
B is also false, becausenr_pages== 1 butfolio_nr_pages(folio)> 1;right?
Let's assume a single subpage of a large folio is no longer mapped. Then, we'd have:
nr_pages == folio_nr_pages(folio) - 1.
You could simply map+reuse most of the folio without COWing.
Once we support COW reuse of PTE-mapped THP we'd do the same. Here, it's just easy to detect that the folio is exclusive (folio_ref_count(folio) == 1 before mapping anything).
If you really want to mimic what do_wp_page() currently does, you should have:
exclusive || (folio_ref_count(folio) == 1 && !folio_test_large(folio))
Personally, I think we should keep it simple here and use:
exclusive || folio_ref_count(folio) == 1
IMHO, that's as clear as it gets.
Signature · 3 lines
Cheers, David / dhildenb
Re: [PATCH v3 6/6] mm: swap: entirely map large folios found in swapcache
Quoted 198 lines · David Hildenbrand
Quoted 196 lines · Barry Song
Quoted 171 lines · David Hildenbrand
Quoted 149 lines · Barry Song
Quoted 148 lines · Barry Song
Quoted 146 lines · David Hildenbrand
Quoted 143 lines · Barry Song
Quoted 32 lines · Ryan Roberts
Quoted 11 lines · Barry Song
From: Chuanhua Han <[email protected]>
When a large folio is found in the swapcache, the current implementation requires calling
do_swap_page()nr_pagestimes, resulting innr_pagespage faults. This patch opts to map the entire large folio at once to minimize page faults. Additionally, redundant checks and early exits for ARM64 MTE restoring are removed.Signed-off-by Chuanhua Han <[email protected]>Co-developed-by Barry Song <[email protected]>Signed-off-by Barry Song <[email protected]>With the suggested changes below:
Reviewed-by Ryan Roberts <[email protected]>Quoted 14 lines
--- mm/memory.c | 60 ++++++++++++++++++++++++++++++++++++++++++----------- 1 file changed, 48 insertions(+), 12 deletions(-)mm/memory.c +2
-0@@ -3968,6 +3968,10 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)pte_t pte;vm_fault_t ret =0 ;void *shadow =NULL ; +int nr_pages =1 ; +unsigned long page_idx =0 ;Quoted 8 lines
mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 3974
Quoted 2 lines
mm/memory.c +2
-0+unsigned long address = vmf->address; +pte_t *ptep;nit: Personally I'd prefer all these to get initialised just before the "if (
folio_test_large()..." block below. That way it is clear they are fresh (incase any logic between here and there makes an adjustment) and its clear that they are only to be used after that block (the compiler will warn if using an uninitialized value).right. I agree this will make the code more readable.
Quoted 32 lines
Quoted 32 lines
mm/memory.c +24
-0if (!pte_unmap_same (vmf)) goto out;@@ -4166,6 +4170,36 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) goto out_nomap; } + ptep = vmf->pte; +if (folio_test_large (folio) &&folio_test_swapcache (folio)) { +int nr =folio_nr_pages (folio); +unsigned long idx =folio_page_idx (folio, page); +unsigned long folio_start = vmf->address - idx *PAGE_SIZE ; +unsigned long folio_end = folio_start + nr *PAGE_SIZE ; +pte_t *folio_ptep; +pte_t folio_pte; + +if (unlikely (folio_start <max (vmf->address &PMD_MASK , vma->vm_start))) + goto check_folio; +if (unlikely (folio_end >pmd_addr_end (vmf->address, vma->vm_end))) + goto check_folio; + + folio_ptep = vmf->pte - idx; + folio_pte =ptep_get (folio_ptep); +if (!pte_same (folio_pte,pte_move_swp_offset (vmf->orig_pte, -idx)) || +swap_pte_batch (folio_ptep, nr, folio_pte) != nr) + goto check_folio; + + page_idx = idx; + address = folio_start; + ptep = folio_ptep; + nr_pages = nr;Quoted 8 lines
mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 4201
Quoted 5 lines
mm/memory.c +5
-0+ entry = folio->swap; + page = &folio->page; + } + +check_folio:Is this still the correct label name, given the checks are now above the new block? Perhaps "
one_page" or something like that?not quite sure about this, as the code after
one_pagecan bemultiple_pages. On the other hand, it seems we are really checking folio after "check_folio" :-)
BUG_ON(!folio_test_anon(folio) && folio_test_mappedtodisk(folio));BUG_ON(folio_test_anon(folio) && PageAnonExclusive(page));/*
* Check under PT lock (to protect against concurrentfork()sharing
* the swap entry concurrently) for certainly exclusive pages. */
if (!folio_test_ksm(folio)) {Quoted 21 lines
Quoted 21 lines
mm/memory.c +5
-3+/* * PG_anon_exclusive reuses PG_mappedtodisk for anon pages. A swap pte * must never point at an anonymous page in the swapcache that is @@ -4225,12 +4259,13 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)* We're already holding a reference on the page but haven't mapped it * yet. */ -swap_free_nr (entry,1 ); +swap_free_nr (entry, nr_pages);if (should_try_to_free_swap (folio, vma, vmf->flags))folio_free_swap (folio); -inc_mm_counter (vma->vm_mm,MM_ANONPAGES ); -dec_mm_counter (vma->vm_mm,MM_SWAPENTS ); +folio_ref_add (folio, nr_pages -1 ); +add_mm_counter (vma->vm_mm,MM_ANONPAGES , nr_pages); +add_mm_counter (vma->vm_mm,MM_SWAPENTS , -nr_pages); pte =mk_pte (page, vma->vm_page_prot);Quoted 12 lines
mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 4279
Quoted 8 lines
mm/memory.c +2
-1/* @@ -4240,34 +4275,35 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)* exclusivity. */ if (!folio_test_ksm (folio) && - (exclusive ||folio_ref_count (folio) ==1 )) { + (exclusive || (folio_ref_count (folio) == nr_pages && +folio_nr_pages (folio) == nr_pages))) {I think in practice there is no change here? If
nr_pages> 1 then the folio is in the swapcache, so there is an extra ref on it? I agree with the change for robustness sake. Just checking my understanding.This is the code showing we are reusing/(mkwrite) a folio either
1. we meet a small folio and we are the only one hitting the small folio
2. we meet a large folio and we are the only one hitting the large folioany corner cases besides the above two seems difficult. for example,
while we hit a large folio in swapcache but if we can't entirely map it (nr_pages==1) due to partial unmap, we will have
folio_ref_count(folio)==nr_pages== 1Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
No, there would be other references from the swapcache and
folio_ref_count(folio)> 1. See my other reply.right. can be clearer by:
Wait, do we still need
folio_nr_pages(folio)==nr_pageseven if we usefolio_ref_count(folio)== 1 and movingfolio_ref_add(folio, nr_pages - 1)?I don't think that we will "need" it.
Quoted 7 lines
one case is that we have a large folio with 16 PTEs, and we unmap 15 swap PTE entries, thus we have only one swap entry left. Then we hit the large folio in swapcache. but we have only one PTE thus we will map only one PTE. lacking
folio_nr_pages(folio)==nr_pages, we reuse the large folio for one PTE. with it,do_wp_page()will migrate the large folio to a small one?We will set PAE bit and
do_wp_page()will unconditionally reuse that page.Quoted 8 lines
Note that this is the same as if we had
pte_swp_exclusive()set and would have run into "exclusive=true" here.If we'd want a similar "optimization" as we have in
wp_can_reuse_anon_folio(), you'd want something likeexclusive || (
folio_ref_count(folio)== 1 &&(!folio_test_large(folio) || nr_pages > 1)I feel like
A : !folio_test_large(folio) ||
nr_pages> 1equals
B:
folio_nr_pages(folio)==nr_pagesif folio is small,
folio_test_large(folio)is false, both A and B will be true; if folio is large, and we map the whole large folio, A will be true because ofnr_pages> 1;
B is also true;
if folio is large, and we map single one PTE, A will be false;
B is also false, becausenr_pages== 1 butfolio_nr_pages(folio)> 1;
Quoted 8 lines
Quoted 1 line
right?
Let's assume a single subpage of a large folio is no longer mapped. Then, we'd have:
nr_pages==folio_nr_pages(folio)- 1.You could simply map+reuse most of the folio without COWing.
yes. This is good but the pte which is no longer mapped could be anyone within the nr_pages PTEs. so it could be quite tricky for set_ptes.
Quoted 9 lines
Once we support COW reuse of PTE-mapped THP we'd do the same. Here, it's just easy to detect that the folio is exclusive (
folio_ref_count(folio)== 1 before mapping anything).If you really want to mimic what
do_wp_page()currently does, you should have:exclusive || (
folio_ref_count(folio)== 1 && !folio_test_large(folio))
I actually dislike the part that do_wp_page() handles the reuse of a large folio which is entirely mapped. For example, A forks B, B exit, we write A's large folio, we get nr_pages CoW of small folios. Ideally, we can reuse the whole folios for writing.
Quoted 4 lines
Personally, I think we should keep it simple here and use:
exclusive ||
folio_ref_count(folio)== 1
I feel this is still better than
exclusive || (folio_ref_count(folio) == 1 && !folio_test_large(folio)) as we reuse the whole large folio. the do_wp_page() behaviour doesn't have this.
Quoted 2 lines
IMHO, that's as clear as it gets.
I agree this is clear. But I wonder if there is a possibility to optimize.
Using your instance,
"Let's assume a single subpage of a large folio is no longer mapped."
For a large folio with 16 PTEs and in case we have unmapped one of them. Thus, we have 15 swap entries left.
The first PTE which gets page faults will reuse the whole large folio having "exclusive || folio_ref_count(folio) == 1" only. The left 14 will allocate 14 small
folios(swapcache has been dropped), thus, we use 16 + 14 = 30 pages memory.
with either
A : !folio_test_large(folio) || nr_pages > 1
or
B: folio_nr_pages(folio) == nr_pages
We consume 15 pages.
Quoted 5 lines
--
Cheers,David / dhildenb
Thanks
Barry
Re: [PATCH v3 6/6] mm: swap: entirely map large folios found in swapcache
Quoted 10 lines
Quoted 6 lines
Let's assume a single subpage of a large folio is no longer mapped. Then, we'd have:
nr_pages==folio_nr_pages(folio)- 1.You could simply map+reuse most of the folio without COWing.
yes. This is good but the pte which is no longer mapped could be anyone within the
nr_pagesPTEs. so it could be quite tricky forset_ptes.
The swap batching logic should take care of that, otherwise it would be buggy.
Quoted 9 lines
Quoted 9 lines
Once we support COW reuse of PTE-mapped THP we'd do the same. Here, it's just easy to detect that the folio is exclusive (
folio_ref_count(folio)== 1 before mapping anything).If you really want to mimic what
do_wp_page()currently does, you should have:
Quoted 6 lines
Quoted 1 line
exclusive || (
folio_ref_count(folio)== 1 && !folio_test_large(folio))I actually dislike the part that
do_wp_page()handles the reuse of a large folio which is entirely mapped. For example, A forks B, B exit, we write A's large folio, we getnr_pagesCoW of small folios. Ideally, we can reuse the whole folios for writing.
Yes, see the link I shared to what I am working on. There isn't really a question if what we do right now needs to be improved and all these scenarios are pretty obvious clear.
Quoted 10 lines
Quoted 5 lines
Personally, I think we should keep it simple here and use:
exclusive ||
folio_ref_count(folio)== 1I feel this is still better than
exclusive || (folio_ref_count(folio)== 1 && !folio_test_large(folio)) as we reuse the whole large folio. thedo_wp_page()behaviour doesn't have this.
Yes, but there is the comment "Same logic as in do_wp_page();". We already ran into issues having different COW reuse logic all over the place. For this case here, I don't care if we leave it as
"exclusive || folio_ref_count(folio) == 1"
But let's not try inventing new stuff here.
Signature · 3 lines
Cheers, David / dhildenb
Re: [PATCH v3 6/6] mm: swap: entirely map large folios found in swapcache
Quoted 6 lines · David Hildenbrand
Quoted 6 lines
Quoted 6 lines
Let's assume a single subpage of a large folio is no longer mapped. Then, we'd have:
nr_pages==folio_nr_pages(folio)- 1.
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
You could simply map+reuse most of the folio without COWing.
yes. This is good but the pte which is no longer mapped could be anyone within the
nr_pagesPTEs. so it could be quite tricky forset_ptes.The swap batching logic should take care of that, otherwise it would be buggy.
When you mention "it would be buggy," are you also referring to the current fallback approach? or only refer to the future patch which might be able to map/reuse "nr_pages - 1" pages?
The current patch falls back to setting nr_pages = 1 without mapping or reusing nr_pages - 1. I feel your concern doesn't refer to this fallback?
Quoted 12 lines
Quoted 12 lines
Quoted 11 lines
Once we support COW reuse of PTE-mapped THP we'd do the same. Here, it's just easy to detect that the folio is exclusive (
folio_ref_count(folio)== 1 before mapping anything).If you really want to mimic what
do_wp_page()currently does, you should have:exclusive || (
folio_ref_count(folio)== 1 && !folio_test_large(folio))
Quoted 8 lines
Quoted 4 lines
I actually dislike the part that
do_wp_page()handles the reuse of a large folio which is entirely mapped. For example, A forks B, B exit, we write A's large folio, we getnr_pagesCoW of small folios. Ideally, we can reuse the whole folios for writing.Yes, see the link I shared to what I am working on. There isn't really a question if what we do right now needs to be improved and all these scenarios are pretty obvious clear.
Great! I plan to dedicate more time to reviewing your work.
Quoted 12 lines
Quoted 11 lines
Quoted 6 lines
Personally, I think we should keep it simple here and use:
exclusive ||
folio_ref_count(folio)== 1I feel this is still better than
exclusive || (folio_ref_count(folio)== 1 && !folio_test_large(folio)) as we reuse the whole large folio. thedo_wp_page()behaviour doesn't have this.
Quoted 5 lines
Yes, but there is the comment "Same logic as in
do_wp_page();". We already ran into issues having different COW reuse logic all over the place. For this case here, I don't care if we leave it as"exclusive ||
folio_ref_count(folio)== 1"
I'm perfectly fine with using the code for this patchset and maybe looking for other
opportunities for potential optimization as an incremental patchset, for example,
reusing the remaining PTEs as suggested by you - "simply map+reuse most of the folio without COWing."
Quoted 2 lines
But let's not try inventing new stuff here.
It seems you ignored and snipped my "16 + 14" pages and "15" pages example though. but once we support "simply map+reuse most of the folio without COWing", the "16+14" problem can be resolved, instead, we consume 16 pages.
Quoted 5 lines
--
Cheers,David / dhildenb
Thanks
Barry
Re: [PATCH v3 6/6] mm: swap: entirely map large folios found in swapcache
Quoted 13 lines · Barry Song
Quoted 12 lines · David Hildenbrand
Quoted 11 lines
Quoted 7 lines
Let's assume a single subpage of a large folio is no longer mapped. Then, we'd have:
nr_pages==folio_nr_pages(folio)- 1.You could simply map+reuse most of the folio without COWing.
yes. This is good but the pte which is no longer mapped could be anyone within the
nr_pagesPTEs. so it could be quite tricky forset_ptes.
Quoted 6 lines
Quoted 2 lines
The swap batching logic should take care of that, otherwise it would be buggy.
When you mention "it would be buggy," are you also referring to the current fallback approach? or only refer to the future patch which might be able to map/reuse "
nr_pages- 1" pages?
swap_pte_batch() should not skip any holes. So consequently, set_ptes() should do the right thing. (regarding your comment "could be quite ricky for set_ptes")
So I think that should be working as expected.
Quoted 21 lines
The current patch falls back to setting
nr_pages= 1 without mapping or reusingnr_pages- 1. I feel your concern doesn't refer to this fallback?Quoted 17 lines
Quoted 16 lines
Quoted 11 lines
Once we support COW reuse of PTE-mapped THP we'd do the same. Here, it's just easy to detect that the folio is exclusive (
folio_ref_count(folio)== 1 before mapping anything).If you really want to mimic what
do_wp_page()currently does, you should have:exclusive || (
folio_ref_count(folio)== 1 && !folio_test_large(folio))I actually dislike the part that
do_wp_page()handles the reuse of a large folio which is entirely mapped. For example, A forks B, B exit, we write A's large folio, we getnr_pagesCoW of small folios. Ideally, we can reuse the whole folios for writing.
Quoted 5 lines
Quoted 3 lines
Yes, see the link I shared to what I am working on. There isn't really a question if what we do right now needs to be improved and all these scenarios are pretty obvious clear.
Great! I plan to dedicate more time to reviewing your work.
Nice! And there will be a lot of follow-up optimization work I won't tackle immediately regarding COW (COW-reuse around, maybe sometimes we want to COW bigger chunks).
I still have making PageAnonExclusive a per-folio flag on my TODO list, that will help the COW-reuse around case a lot.
Quoted 27 lines
Quoted 13 lines
Quoted 4 lines
Quoted 4 lines
Quoted 8 lines
Quoted 3 lines
Personally, I think we should keep it simple here and use:
exclusive ||
folio_ref_count(folio)== 1I feel this is still better than
exclusive || (folio_ref_count(folio)== 1 && !folio_test_large(folio)) as we reuse the whole large folio. thedo_wp_page()behaviour doesn't have this.Quoted 5 lines
Yes, but there is the comment "Same logic as in
do_wp_page();". We already ran into issues having different COW reuse logic all over the place. For this case here, I don't care if we leave it as"exclusive ||
folio_ref_count(folio)== 1"I'm perfectly fine with using the code for this patchset and maybe looking for other
opportunities for potential optimization as an incremental patchset, for example,
reusing the remaining PTEs as suggested by you - "simply map+reuse most of the folio without COWing."Quoted 1 line
Quoted 6 lines
Quoted 1 line
But let's not try inventing new stuff here.
It seems you ignored and snipped my "16 + 14" pages and "15" pages example though. but once we support "simply map+reuse most of the folio without COWing", the "16+14" problem can be resolved, instead, we consume 16 pages.
Oh, sorry for skipping that, for me it was rather clear: the partially mapped folios will be on the deferred split list and the excess memory can (and will be) reclaimed when there is need. So this temporary memory consumption is usually not a problem in practice. But yes, something to optimize (just like COW reuse in general).
Signature · 3 lines
Cheers, David / dhildenb
Re: [PATCH v3 6/6] mm: swap: entirely map large folios found in swapcache
Quoted 22 lines · David Hildenbrand
Quoted 13 lines · Barry Song
Quoted 12 lines · David Hildenbrand
Quoted 11 lines
Quoted 7 lines
Let's assume a single subpage of a large folio is no longer mapped. Then, we'd have:
nr_pages==folio_nr_pages(folio)- 1.You could simply map+reuse most of the folio without COWing.
yes. This is good but the pte which is no longer mapped could be anyone within the
nr_pagesPTEs. so it could be quite tricky forset_ptes.Quoted 6 lines
Quoted 2 lines
The swap batching logic should take care of that, otherwise it would be buggy.
When you mention "it would be buggy," are you also referring to the current fallback approach? or only refer to the future patch which might be able to map/reuse "
nr_pages- 1" pages?
Quoted 5 lines
swap_pte_batch()should not skip any holes. So consequently,set_ptes()should do the right thing. (regarding your comment "could be quite ricky forset_ptes")So I think that should be working as expected.
maybe not. take a look at my current code, I am goto check_folio with nr_pages = 1
if swap_pte_batch(folio_ptep, nr, folio_pte) != folio_nr_pages(folio).
+ nr_pages = 1;
+ ...
+ if (folio_test_large(folio) && folio_test_swapcache(folio)) {
+ int nr = folio_nr_pages(folio);
+ ...
+ if (!pte_same(folio_pte,pte_move_swp_offset(vmf->orig_pte, -idx)) ||
+ swap_pte_batch(folio_ptep, nr, folio_pte) != nr)
+ goto check_folio; /* read here, i am falling
back nr_pages = 1 */
+
+
+ ...
+ nr_pages = nr;
The fallback(=1) works. but it seems you are proposing set nr_pages = swap_pte_batch(folio_ptep, nr, folio_pte)
if (swap_pte_batch(folio_ptep, nr, folio_pte) > 1 && swap_pte_batch(folio_ptep, nr, folio_pte) <nr_pages) ?
Quoted 81 lines
Quoted 22 lines
The current patch falls back to setting
nr_pages= 1 without mapping or reusingnr_pages- 1. I feel your concern doesn't refer to this fallback?Quoted 17 lines
Quoted 16 lines
Quoted 11 lines
Once we support COW reuse of PTE-mapped THP we'd do the same. Here, it's just easy to detect that the folio is exclusive (
folio_ref_count(folio)== 1 before mapping anything).If you really want to mimic what
do_wp_page()currently does, you should have:exclusive || (
folio_ref_count(folio)== 1 && !folio_test_large(folio))I actually dislike the part that
do_wp_page()handles the reuse of a large folio which is entirely mapped. For example, A forks B, B exit, we write A's large folio, we getnr_pagesCoW of small folios. Ideally, we can reuse the whole folios for writing.Quoted 5 lines
Quoted 3 lines
Yes, see the link I shared to what I am working on. There isn't really a question if what we do right now needs to be improved and all these scenarios are pretty obvious clear.
Great! I plan to dedicate more time to reviewing your work.
Nice! And there will be a lot of follow-up optimization work I won't tackle immediately regarding COW (COW-reuse around, maybe sometimes we want to COW bigger chunks).
I still have making PageAnonExclusive a per-folio flag on my TODO list, that will help the COW-reuse around case a lot.
Quoted 27 lines
Quoted 13 lines
Quoted 4 lines
Quoted 4 lines
Quoted 8 lines
Quoted 3 lines
Personally, I think we should keep it simple here and use:
exclusive ||
folio_ref_count(folio)== 1I feel this is still better than
exclusive || (folio_ref_count(folio)== 1 && !folio_test_large(folio)) as we reuse the whole large folio. thedo_wp_page()behaviour doesn't have this.Quoted 5 lines
Yes, but there is the comment "Same logic as in
do_wp_page();". We already ran into issues having different COW reuse logic all over the place. For this case here, I don't care if we leave it as"exclusive ||
folio_ref_count(folio)== 1"I'm perfectly fine with using the code for this patchset and maybe looking for other
opportunities for potential optimization as an incremental patchset, for example,
reusing the remaining PTEs as suggested by you - "simply map+reuse most of the folio without COWing."Quoted 1 line
Quoted 6 lines
Quoted 1 line
But let's not try inventing new stuff here.
It seems you ignored and snipped my "16 + 14" pages and "15" pages example though. but once we support "simply map+reuse most of the folio without COWing", the "16+14" problem can be resolved, instead, we consume 16 pages.
Oh, sorry for skipping that, for me it was rather clear: the partially mapped folios will be on the deferred split list and the excess memory can (and will be) reclaimed when there is need. So this temporary memory consumption is usually not a problem in practice. But yes, something to optimize (just like COW reuse in general).
--
Cheers,David / dhildenb
Re: [PATCH v3 6/6] mm: swap: entirely map large folios found in swapcache
In reply to Barry SongQuoted 85 lines · Barry Song
Quoted 32 lines · Ryan Roberts
Quoted 11 lines · Barry Song
From: Chuanhua Han <[email protected]>
When a large folio is found in the swapcache, the current implementation requires calling
do_swap_page()nr_pagestimes, resulting innr_pagespage faults. This patch opts to map the entire large folio at once to minimize page faults. Additionally, redundant checks and early exits for ARM64 MTE restoring are removed.Signed-off-by Chuanhua Han <[email protected]>Co-developed-by Barry Song <[email protected]>Signed-off-by Barry Song <[email protected]>With the suggested changes below:
Reviewed-by Ryan Roberts <[email protected]>Quoted 14 lines
--- mm/memory.c | 60 ++++++++++++++++++++++++++++++++++++++++++----------- 1 file changed, 48 insertions(+), 12 deletions(-)mm/memory.c +2
-0@@ -3968,6 +3968,10 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)pte_t pte;vm_fault_t ret =0 ;void *shadow =NULL ; +int nr_pages =1 ; +unsigned long page_idx =0 ;Quoted 8 lines
mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 3974
Quoted 2 lines
mm/memory.c +2
-0+unsigned long address = vmf->address; +pte_t *ptep;nit: Personally I'd prefer all these to get initialised just before the "if (
folio_test_large()..." block below. That way it is clear they are fresh (incase any logic between here and there makes an adjustment) and its clear that they are only to be used after that block (the compiler will warn if using an uninitialized value).right. I agree this will make the code more readable.
Quoted 32 lines
Quoted 32 lines
mm/memory.c +24
-0if (!pte_unmap_same (vmf)) goto out;@@ -4166,6 +4170,36 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) goto out_nomap; } + ptep = vmf->pte; +if (folio_test_large (folio) &&folio_test_swapcache (folio)) { +int nr =folio_nr_pages (folio); +unsigned long idx =folio_page_idx (folio, page); +unsigned long folio_start = vmf->address - idx *PAGE_SIZE ; +unsigned long folio_end = folio_start + nr *PAGE_SIZE ; +pte_t *folio_ptep; +pte_t folio_pte; + +if (unlikely (folio_start <max (vmf->address &PMD_MASK , vma->vm_start))) + goto check_folio; +if (unlikely (folio_end >pmd_addr_end (vmf->address, vma->vm_end))) + goto check_folio; + + folio_ptep = vmf->pte - idx; + folio_pte =ptep_get (folio_ptep); +if (!pte_same (folio_pte,pte_move_swp_offset (vmf->orig_pte, -idx)) || +swap_pte_batch (folio_ptep, nr, folio_pte) != nr) + goto check_folio; + + page_idx = idx; + address = folio_start; + ptep = folio_ptep; + nr_pages = nr;Quoted 8 lines
mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 4201
Quoted 5 lines
mm/memory.c +5
-0+ entry = folio->swap; + page = &folio->page; + } + +check_folio:Is this still the correct label name, given the checks are now above the new block? Perhaps "
one_page" or something like that?
Quoted 3 lines
not quite sure about this, as the code after
one_pagecan bemultiple_pages. On the other hand, it seems we are really checking folio after "check_folio" :-)
Yeah fair enough. Ignore my comment.
Quoted 109 lines
BUG_ON(!folio_test_anon(folio) && folio_test_mappedtodisk(folio));BUG_ON(folio_test_anon(folio) && PageAnonExclusive(page));/*
* Check under PT lock (to protect against concurrentfork()sharing
* the swap entry concurrently) for certainly exclusive pages. */
if (!folio_test_ksm(folio)) {Quoted 21 lines
Quoted 21 lines
mm/memory.c +5
-3+/* * PG_anon_exclusive reuses PG_mappedtodisk for anon pages. A swap pte * must never point at an anonymous page in the swapcache that is @@ -4225,12 +4259,13 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)* We're already holding a reference on the page but haven't mapped it * yet. */ -swap_free_nr (entry,1 ); +swap_free_nr (entry, nr_pages);if (should_try_to_free_swap (folio, vma, vmf->flags))folio_free_swap (folio); -inc_mm_counter (vma->vm_mm,MM_ANONPAGES ); -dec_mm_counter (vma->vm_mm,MM_SWAPENTS ); +folio_ref_add (folio, nr_pages -1 ); +add_mm_counter (vma->vm_mm,MM_ANONPAGES , nr_pages); +add_mm_counter (vma->vm_mm,MM_SWAPENTS , -nr_pages); pte =mk_pte (page, vma->vm_page_prot);Quoted 12 lines
mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 4279
Quoted 8 lines
mm/memory.c +2
-1/* @@ -4240,34 +4275,35 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)* exclusivity. */ if (!folio_test_ksm (folio) && - (exclusive ||folio_ref_count (folio) ==1 )) { + (exclusive || (folio_ref_count (folio) == nr_pages && +folio_nr_pages (folio) == nr_pages))) {I think in practice there is no change here? If
nr_pages> 1 then the folio is in the swapcache, so there is an extra ref on it? I agree with the change for robustness sake. Just checking my understanding.This is the code showing we are reusing/(mkwrite) a folio either
1. we meet a small folio and we are the only one hitting the small folio
2. we meet a large folio and we are the only one hitting the large folioany corner cases besides the above two seems difficult. for example,
while we hit a large folio in swapcache but if we can't entirely map it (nr_pages==1) due to partial unmap, we will have
folio_ref_count(folio)==nr_pages== 1, in this case, lackingfolio_nr_pages(folio)==nr_pagesmight lead tomkwrite()on a single pte within a partially unmapped large folio. not quite sure this is wrong, but seems buggy and arduous.Quoted 41 lines
Quoted 41 lines
mm/memory.c +7
-7if (vmf->flags &FAULT_FLAG_WRITE ) { pte =maybe_mkwrite (pte_mkdirty (pte), vma); vmf->flags &= ~FAULT_FLAG_WRITE ; } rmap_flags |=RMAP_EXCLUSIVE ; } -flush_icache_page (vma, page); +flush_icache_pages (vma, page, nr_pages);if (pte_swp_soft_dirty (vmf->orig_pte)) pte =pte_mksoft_dirty (pte);if (pte_swp_uffd_wp (vmf->orig_pte)) pte =pte_mkuffd_wp (pte); - vmf->orig_pte = pte; + vmf->orig_pte =pte_advance_pfn (pte, page_idx);/* ksm created a completely new copy */ if (unlikely (folio != swapcache && swapcache)) { -folio_add_new_anon_rmap (folio, vma, vmf->address); +folio_add_new_anon_rmap (folio, vma, address);folio_add_lru_vma (folio, vma); }else { -folio_add_anon_rmap_pte (folio, page, vma, vmf->address, +folio_add_anon_rmap_ptes (folio, page, nr_pages, vma, address, rmap_flags); }VM_BUG_ON (!folio_test_anon (folio) || (pte_write (pte) && !PageAnonExclusive (page))); -set_pte_at (vma->vm_mm, vmf->address, vmf->pte, pte); -arch_do_swap_page_nr (vma->vm_mm, vma, vmf->address, - pte, vmf->orig_pte,1 ); +set_ptes (vma->vm_mm, address, ptep, pte, nr_pages); +arch_do_swap_page_nr (vma->vm_mm, vma, address, + pte, pte, nr_pages);folio_unlock (folio);if (folio != swapcache && swapcache) {@@ -4291,7 +4327,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) }Quoted 7 lines
mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 4333
Quoted 6 lines
mm/memory.c +1
-1/* No need to invalidate - it was non-present before */ -update_mmu_cache_range (vmf, vma, vmf->address, vmf->pte,1 ); +update_mmu_cache_range (vmf, vma, address, ptep, nr_pages); unlock:if (vmf->pte)pte_unmap_unlock (vmf->pte, vmf->ptl);Thanks
Barry
Re: [PATCH v3 6/6] mm: swap: entirely map large folios found in swapcache
In reply to Barry SongQuoted 85 lines · Barry Song
From: Chuanhua Han <[email protected]>
When a large folio is found in the swapcache, the current implementation requires calling
do_swap_page()nr_pagestimes, resulting innr_pagespage faults. This patch opts to map the entire large folio at once to minimize page faults. Additionally, redundant checks and early exits for ARM64 MTE restoring are removed.Signed-off-by Chuanhua Han <[email protected]>Co-developed-by Barry Song <[email protected]>Signed-off-by Barry Song <[email protected]>--- mm/memory.c | 60 ++++++++++++++++++++++++++++++++++++++++++----------- 1 file changed, 48 insertions(+), 12 deletions(-)mm/memory.c +38
-3@@ -3968,6 +3968,10 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)pte_t pte;vm_fault_t ret =0 ;void *shadow =NULL ; +int nr_pages =1 ; +unsigned long page_idx =0 ; +unsigned long address = vmf->address; +pte_t *ptep;if (!pte_unmap_same (vmf)) goto out;@@ -4166,6 +4170,36 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) goto out_nomap; } + ptep = vmf->pte; +if (folio_test_large (folio) &&folio_test_swapcache (folio)) { +int nr =folio_nr_pages (folio); +unsigned long idx =folio_page_idx (folio, page); +unsigned long folio_start = vmf->address - idx *PAGE_SIZE ; +unsigned long folio_end = folio_start + nr *PAGE_SIZE ; +pte_t *folio_ptep; +pte_t folio_pte; + +if (unlikely (folio_start <max (vmf->address &PMD_MASK , vma->vm_start))) + goto check_folio; +if (unlikely (folio_end >pmd_addr_end (vmf->address, vma->vm_end))) + goto check_folio; + + folio_ptep = vmf->pte - idx; + folio_pte =ptep_get (folio_ptep); +if (!pte_same (folio_pte,pte_move_swp_offset (vmf->orig_pte, -idx)) || +swap_pte_batch (folio_ptep, nr, folio_pte) != nr) + goto check_folio; + + page_idx = idx; + address = folio_start; + ptep = folio_ptep; + nr_pages = nr; + entry = folio->swap; + page = &folio->page; + } + +check_folio: +/* * PG_anon_exclusive reuses PG_mappedtodisk for anon pages. A swap pte * must never point at an anonymous page in the swapcache that is @@ -4225,12 +4259,13 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)* We're already holding a reference on the page but haven't mapped it * yet. */ -swap_free_nr (entry,1 ); +swap_free_nr (entry, nr_pages);if (should_try_to_free_swap (folio, vma, vmf->flags))folio_free_swap (folio); -inc_mm_counter (vma->vm_mm,MM_ANONPAGES ); -dec_mm_counter (vma->vm_mm,MM_SWAPENTS ); +folio_ref_add (folio, nr_pages -1 ); +add_mm_counter (vma->vm_mm,MM_ANONPAGES , nr_pages); +add_mm_counter (vma->vm_mm,MM_SWAPENTS , -nr_pages); pte =mk_pte (page, vma->vm_page_prot);/*
mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 4282
Quoted 9 lines
mm/memory.c +2
-1@@ -4240,34 +4275,35 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)* exclusivity. */ if (!folio_test_ksm (folio) && - (exclusive ||folio_ref_count (folio) ==1 )) { + (exclusive || (folio_ref_count (folio) == nr_pages && +folio_nr_pages (folio) == nr_pages))) {if (vmf->flags &FAULT_FLAG_WRITE ) { pte =maybe_mkwrite (pte_mkdirty (pte), vma); vmf->flags &= ~FAULT_FLAG_WRITE ;
I fail to convince myself that this change is correct, and if it is correct, it's confusing (I think there is a dependency on folio_free_swap() having been called and succeeding, such that we don't have a folio that is in the swapcache at this point).
Why can't we move the folio_ref_add() after this check and just leave the check as it is?
"folio_ref_count(folio) == 1" is as clear as it gets: we hold the single reference, so we can do with this thing whatever we want: it's certainly exclusive. No swapcache, no other people mapping it.
Signature · 3 lines
Cheers, David / dhildenb
Re: [PATCH v3 6/6] mm: swap: entirely map large folios found in swapcache
Quoted 102 lines · David Hildenbrand
Quoted 85 lines · Barry Song
From: Chuanhua Han <[email protected]>
When a large folio is found in the swapcache, the current implementation requires calling
do_swap_page()nr_pagestimes, resulting innr_pagespage faults. This patch opts to map the entire large folio at once to minimize page faults. Additionally, redundant checks and early exits for ARM64 MTE restoring are removed.Signed-off-by Chuanhua Han <[email protected]>Co-developed-by Barry Song <[email protected]>Signed-off-by Barry Song <[email protected]>--- mm/memory.c | 60 ++++++++++++++++++++++++++++++++++++++++++----------- 1 file changed, 48 insertions(+), 12 deletions(-)mm/memory.c +38
-3@@ -3968,6 +3968,10 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)pte_t pte;vm_fault_t ret =0 ;void *shadow =NULL ; +int nr_pages =1 ; +unsigned long page_idx =0 ; +unsigned long address = vmf->address; +pte_t *ptep;if (!pte_unmap_same (vmf)) goto out;@@ -4166,6 +4170,36 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) goto out_nomap; } + ptep = vmf->pte; +if (folio_test_large (folio) &&folio_test_swapcache (folio)) { +int nr =folio_nr_pages (folio); +unsigned long idx =folio_page_idx (folio, page); +unsigned long folio_start = vmf->address - idx *PAGE_SIZE ; +unsigned long folio_end = folio_start + nr *PAGE_SIZE ; +pte_t *folio_ptep; +pte_t folio_pte; + +if (unlikely (folio_start <max (vmf->address &PMD_MASK , vma->vm_start))) + goto check_folio; +if (unlikely (folio_end >pmd_addr_end (vmf->address, vma->vm_end))) + goto check_folio; + + folio_ptep = vmf->pte - idx; + folio_pte =ptep_get (folio_ptep); +if (!pte_same (folio_pte,pte_move_swp_offset (vmf->orig_pte, -idx)) || +swap_pte_batch (folio_ptep, nr, folio_pte) != nr) + goto check_folio; + + page_idx = idx; + address = folio_start; + ptep = folio_ptep; + nr_pages = nr; + entry = folio->swap; + page = &folio->page; + } + +check_folio: +/* * PG_anon_exclusive reuses PG_mappedtodisk for anon pages. A swap pte * must never point at an anonymous page in the swapcache that is @@ -4225,12 +4259,13 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)* We're already holding a reference on the page but haven't mapped it * yet. */ -swap_free_nr (entry,1 ); +swap_free_nr (entry, nr_pages);if (should_try_to_free_swap (folio, vma, vmf->flags))folio_free_swap (folio); -inc_mm_counter (vma->vm_mm,MM_ANONPAGES ); -dec_mm_counter (vma->vm_mm,MM_SWAPENTS ); +folio_ref_add (folio, nr_pages -1 ); +add_mm_counter (vma->vm_mm,MM_ANONPAGES , nr_pages); +add_mm_counter (vma->vm_mm,MM_SWAPENTS , -nr_pages); pte =mk_pte (page, vma->vm_page_prot);/* mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 4282
Quoted 9 lines
mm/memory.c +2
-1@@ -4240,34 +4275,35 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)* exclusivity. */ if (!folio_test_ksm (folio) && - (exclusive ||folio_ref_count (folio) ==1 )) { + (exclusive || (folio_ref_count (folio) == nr_pages && +folio_nr_pages (folio) == nr_pages))) {if (vmf->flags &FAULT_FLAG_WRITE ) { pte =maybe_mkwrite (pte_mkdirty (pte), vma); vmf->flags &= ~FAULT_FLAG_WRITE ;I fail to convince myself that this change is correct, and if it is correct, it's confusing (I think there is a dependency on
folio_free_swap()having been called and succeeding, such that we don't have a folio that is in the swapcache at this point).
Quoted 6 lines
Why can't we move the
folio_ref_add()after this check and just leave the check as it is?"
folio_ref_count(folio)== 1" is as clear as it gets: we hold the single reference, so we can do with this thing whatever we want: it's certainly exclusive. No swapcache, no other people mapping it.
Right.
I believe the code works correctly but is a bit confusing. as you said, we might move folio_ref_add() behind folio_ref_count(folio) == 1.
Quoted 7 lines
--
Cheers,David / dhildenb
Thanks
Barry