support large folio swap-out and swap-in for shmem
Subsystems (8)
Baolin Wang
reviewerPATCH v3 3/10 mm: shmem: extend shmem_partial_swap_usage() to support large folio swap
In reply to Baolin WangTo support shmem large folio swapout in the following patches, using xa_get_order() to get the order of the swap entry to calculate the swap usage of shmem.
--- mm/shmem.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
mm/shmem.c +1 -1
@@ -881,7 +881,7 @@ unsigned long shmem_partial_swap_usage(struct address_space *mapping,if (xas_retry (&xas, page))continue ;if (xa_is_value (page)) - swapped++; + swapped +=1 <<xa_get_order (xas.xa, xas.xa_index);if (xas.xa_index == max)break ;if (need_resched ()) {
Signature · 1 line
2.39.3
Baolin Wang
reviewerPATCH v3 4/10 mm: shmem: return number of pages beeing freed in shmem_free_swap
In reply to Baolin WangFrom: Daniel Gomez <[email protected]>
Both shmem_free_swap callers expect the number of pages being freed. In the large folios context, this needs to support larger values other than 0 (used as 1 page being freed) and -ENOENT (used as 0 pages being freed). In preparation for large folios adoption, make shmem_free_swap routine return the number of pages being freed. So, returning 0 in this context, means 0 pages being freed.
While we are at it, changing to use free_swap_and_cache_nr() to free large order swap entry by Baolin Wang.
--- mm/shmem.c | 25 ++++++++++++++++--------- 1 file changed, 16 insertions(+), 9 deletions(-)
mm/shmem.c +16 -9
@@ -847,18 +847,22 @@ static void shmem_delete_from_page_cache(struct folio *folio, void *radswap) }/* -* Remove swap entry from page cache, free the swap and its page cache. +* Remove swap entry from page cache, free the swap and its page cache. Returns +* the number of pages being freed. 0 means entry not found in XArray (0 pages +* being freed). */ -static int shmem_free_swap (struct address_space *mapping, -pgoff_t index,void *radswap) +static long shmem_free_swap (struct address_space *mapping, +pgoff_t index,void *radswap) { +int order =xa_get_order (&mapping->i_pages, index);void *old; old =xa_cmpxchg_irq (&mapping->i_pages, index, radswap,NULL ,0 );if (old != radswap) -return -ENOENT ; -free_swap_and_cache (radix_to_swp_entry (radswap)); -return 0 ; +return 0 ; +free_swap_and_cache_nr (radix_to_swp_entry (radswap),1 << order); + +return 1 << order; }/*
@@ -1010,7 +1014,7 @@ static void shmem_undo_range(struct inode *inode, loff_t lstart, loff_t lend,if (xa_is_value (folio)) {if (unfalloc)continue ; - nr_swaps_freed += !shmem_free_swap (mapping, + nr_swaps_freed +=shmem_free_swap (mapping, indices[i], folio);continue ; }
@@ -1077,14 +1081,17 @@ static void shmem_undo_range(struct inode *inode, loff_t lstart, loff_t lend, folio = fbatch.folios[i];if (xa_is_value (folio)) { +long swaps_freed; +if (unfalloc)continue ; -if (shmem_free_swap (mapping, indices[i], folio)) { + swaps_freed =shmem_free_swap (mapping, indices[i], folio); +if (!swaps_freed) {/* Swap was replaced by page: retry */ index = indices[i];break ; } - nr_swaps_freed++; + nr_swaps_freed += swaps_freed;continue ; }
Signature · 1 line
2.39.3
Baolin Wang
reviewerv3 cover support large folio swap-out and swap-in for shmem
Shmem will support large folio allocation [1] [2] to get a better performance, however, the memory reclaim still splits the precious large folios when trying to swap-out shmem, which may lead to the memory fragmentation issue and can not take advantage of the large folio for shmeme.
Moreover, the swap code already supports for swapping out large folio without split, and large folio swap-in[3] series is queued into mm-unstable branch. Hence this patch set also supports the large folio swap-out and swap-in for shmem.
Note: this series is targeted for v6.12. Please help to review. Thanks.
Functional testing
==================
I use the latest mm-unstable branch to test with reverting Chris's "mm: swap: mTHP swap allocator base on swap cluster order" series which can cause some problems (Hugh also reported in [4]).
Machine environment: 32 Arm cores, 120G memory and 50G swap device.
1. Run xfstests suite to test tmpfs filesystem, and I did not catch any regressions with this patch set.
FSTYP=tmpfs
export TEST_DIR=/mnt/tempfs_mnt
export TEST_DEV=/mnt/tempfs_mnt
export SCRATCH_MNT=/mnt/scratchdir
export SCRATCH_DEV=/mnt/scratchdir
2. Run all mm selftests in tools/testing/selftests/mm/, and no regressions found.
3. I also wrote several shmem swap test cases, including shmem splitting, shmem swapout, shmem swapin, swapoff during shmem swapout, shmem reclaim, shmem swapin replacement, etc. I tested these cases under 4K and 64K shmem folio sizes with a swap device, and shmem swap functionality works well on my machine.
[1] https://lore.kernel.org/all/[email protected]/
[2] https://lore.kernel.org/all/[email protected]/
[3] https://lore.kernel.org/all/[email protected]/T/
[4] https://lore.kernel.org/all/[email protected]/
Chagens from v2:
- Add new patch to split large swap entry if swapin folio is order 0 folio.
- Update some commit message.
Changes from v1:
- Remove useless 'order' variable in shmem_partial_swap_usage(), per Daniel.
- Add a new patch to return number of pages beeing freed in shmem_free_swap(), per Daniel.
- Drop 'orders' parameter for find_get_entries() and find_lock_entries().
- Round down the index when adding the swapin folio into the pagecache, suggested by Hugh.
- Fix the reference issue when removing folio from pagecache in patch 8.
- Fix replacing old folio in swap cache in patch 7.
Changes from RFC:
- Rebased to the latest mm-unstable.
- Drop the counter name fixing patch, which was queued into mm-hotfixes-stable branch.
Baolin Wang (9):
mm: vmscan: add validation before spliting shmem large folio
mm: swap: extend swap_shmem_alloc() to support batch SWAP_MAP_SHMEM
flag setting
mm: shmem: extend shmem_partial_swap_usage() to support large folio
swap
mm: filemap: use xa_get_order() to get the swap entry order
mm: shmem: use swap_free_nr() to free shmem swap entries
mm: shmem: support large folio allocation for shmem_replace_folio()
mm: shmem: drop folio reference count using 'nr_pages' in
shmem_delete_from_page_cache()
mm: shmem: split large entry if the swapin folio is not large
mm: shmem: support large folio swap out
Daniel Gomez (1):
mm: shmem: return number of pages beeing freed in shmem_free_swap
drivers/gpu/drm/i915/gem/i915_gem_shmem.c | 1 + include/linux/swap.h | 4 +- include/linux/writeback.h | 1 + mm/filemap.c | 4 + mm/shmem.c | 203 +++++++++++++++++----- mm/swapfile.c | 98 ++++++----- mm/vmscan.c | 22 ++- 7 files changed, 238 insertions(+), 95 deletions(-)
Signature · 1 line
2.39.3
Baolin Wang
reviewerPATCH v3 5/10 mm: filemap: use xa_get_order() to get the swap entry order
In the following patches, shmem will support the swap out of large folios, which means the shmem mappings may contain large order swap entries, so using xa_get_order() to get the folio order of the shmem swap entry to update the '*start' correctly.
--- mm/filemap.c | 4 ++++ 1 file changed, 4 insertions(+)
mm/filemap.c +4 -0
@@ -2056,6 +2056,8 @@ unsigned find_get_entries(struct address_space *mapping, pgoff_t *start, folio = fbatch->folios[idx];if (!xa_is_value (folio)) nr =folio_nr_pages (folio); +else + nr =1 <<xa_get_order (&mapping->i_pages, indices[idx]); *start = indices[idx] + nr; }return folio_batch_count (fbatch);
@@ -2120,6 +2122,8 @@ unsigned find_lock_entries(struct address_space *mapping, pgoff_t *start, folio = fbatch->folios[idx];if (!xa_is_value (folio)) nr =folio_nr_pages (folio); +else + nr =1 <<xa_get_order (&mapping->i_pages, indices[idx]); *start = indices[idx] + nr; }return folio_batch_count (fbatch);
Signature · 1 line
2.39.3
Baolin Wang
reviewerPATCH v3 1/10 mm: vmscan: add validation before spliting shmem large folio
In reply to Baolin WangPage reclaim will not scan anon LRU if no swap space, however MADV_PAGEOUT can still split shmem large folios even without a swap device. Thus add swap available space validation before spliting shmem large folio to avoid redundant split.
--- mm/vmscan.c | 8 ++++++++ 1 file changed, 8 insertions(+)
mm/vmscan.c +8 -0
@@ -1237,6 +1237,14 @@ static unsigned int shrink_folio_list(struct list_head *folio_list, } }else if (folio_test_swapbacked (folio) &&folio_test_large (folio)) { + +/* +* Do not split shmem folio if no swap memory +* available. +*/ +if (!total_swap_pages) + goto activate_locked; +/* Split shmem folio */ if (split_folio_to_list (folio, folio_list)) goto keep_locked;
Signature · 1 line
2.39.3
Baolin Wang
reviewerPATCH v3 2/10 mm: swap: extend swap_shmem_alloc() to support batch SWAP_MAP_SHMEM flag setting
In reply to Baolin WangTo support shmem large folio swap operations, add a new parameter to swap_shmem_alloc() that allows batch SWAP_MAP_SHMEM flag setting for shmem swap entries.
While we are at it, using folio_nr_pages() to get the number of pages of the folio as a preparation.
--- include/linux/swap.h | 4 +- mm/shmem.c | 6 ++- mm/swapfile.c | 98 +++++++++++++++++++++++--------------------- 3 files changed, 57 insertions(+), 51 deletions(-)
include/linux/swap.h +2 -2
@@ -475,7 +475,7 @@ void put_swap_folio(struct folio *folio, swp_entry_t entry);extern swp_entry_t get_swap_page_of_type (int );extern int get_swap_pages (int n,swp_entry_t swp_entries[],int order);extern int add_swap_count_continuation (swp_entry_t ,gfp_t ); -extern void swap_shmem_alloc (swp_entry_t ); +extern void swap_shmem_alloc (swp_entry_t ,int );extern int swap_duplicate (swp_entry_t );extern int swapcache_prepare (swp_entry_t );extern void swap_free_nr (swp_entry_t entry,int nr_pages);
@@ -542,7 +542,7 @@ static inline int add_swap_count_continuation(swp_entry_t swp, gfp_t gfp_mask)return 0 ; } -static inline void swap_shmem_alloc (swp_entry_t swp) +static inline void swap_shmem_alloc (swp_entry_t swp,int nr) { }
mm/shmem.c +4 -2
@@ -1443,6 +1443,7 @@ static int shmem_writepage(struct page *page, struct writeback_control *wbc)struct shmem_sb_info *sbinfo =SHMEM_SB (inode->i_sb);swp_entry_t swap;pgoff_t index; +int nr_pages;/* * Our capabilities prevent regular writeback or sync from ever calling
@@ -1475,6 +1476,7 @@ static int shmem_writepage(struct page *page, struct writeback_control *wbc) } index = folio->index; + nr_pages =folio_nr_pages (folio);/* * This is somewhat ridiculous, but without plumbing a SWAP_MAP_FALLOC
@@ -1527,8 +1529,8 @@ static int shmem_writepage(struct page *page, struct writeback_control *wbc)if (add_to_swap_cache (folio, swap, __GFP_HIGH | __GFP_NOMEMALLOC | __GFP_NOWARN,NULL ) ==0 ) { -shmem_recalc_inode (inode,0 ,1 ); -swap_shmem_alloc (swap); +shmem_recalc_inode (inode,0 , nr_pages); +swap_shmem_alloc (swap, nr_pages);shmem_delete_from_page_cache (folio,swp_to_radix_entry (swap));mutex_unlock (&shmem_swaplist_mutex);
mm/swapfile.c +51 -47
138 lines not on this page · raw message
Signature · 1 line
2.39.3
Baolin Wang
reviewerPATCH v3 6/10 mm: shmem: use swap_free_nr() to free shmem swap entries
In reply to Baolin WangAs a preparation for supporting shmem large folio swapout, use swap_free_nr() to free some continuous swap entries of the shmem large folio when the large folio was swapped in from the swap cache. In addition, the index should also be round down to the number of pages when adding the swapin folio into the pagecache.
--- mm/shmem.c | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-)
mm/shmem.c +10 -6
@@ -1961,6 +1961,7 @@ static void shmem_set_folio_swapin_error(struct inode *inode, pgoff_t index,struct address_space *mapping = inode->i_mapping;swp_entry_t swapin_error;void *old; +int nr_pages; swapin_error =make_poisoned_swp_entry (); old = xa_cmpxchg_irq(&mapping->i_pages, index,
@@ -1969,6 +1970,7 @@ static void shmem_set_folio_swapin_error(struct inode *inode, pgoff_t index,if (old !=swp_to_radix_entry (swap))return ; + nr_pages =folio_nr_pages (folio);folio_wait_writeback (folio);delete_from_swap_cache (folio);/*
@@ -1976,8 +1978,8 @@ static void shmem_set_folio_swapin_error(struct inode *inode, pgoff_t index,* won't be 0 when inode is released and thus trigger WARN_ON(i_blocks) * in shmem_evict_inode(). */ -shmem_recalc_inode (inode,-1 ,-1 ); -swap_free (swap); +shmem_recalc_inode (inode, -nr_pages, -nr_pages); +swap_free_nr (swap, nr_pages); }/*
@@ -1996,7 +1998,7 @@ static int shmem_swapin_folio(struct inode *inode, pgoff_t index,struct swap_info_struct *si;struct folio *folio =NULL ;swp_entry_t swap; -int error; +int error, nr_pages;VM_BUG_ON (!*foliop || !xa_is_value (*foliop)); swap =radix_to_swp_entry (*foliop);
@@ -2043,6 +2045,7 @@ static int shmem_swapin_folio(struct inode *inode, pgoff_t index, goto failed; }folio_wait_writeback (folio); + nr_pages =folio_nr_pages (folio);/* * Some architectures may have to restore extra metadata to the
@@ -2056,19 +2059,20 @@ static int shmem_swapin_folio(struct inode *inode, pgoff_t index, goto failed; } - error =shmem_add_to_page_cache (folio, mapping, index, + error =shmem_add_to_page_cache (folio, mapping, +round_down (index, nr_pages),swp_to_radix_entry (swap), gfp);if (error) goto failed; -shmem_recalc_inode (inode,0 ,-1 ); +shmem_recalc_inode (inode,0 , -nr_pages);if (sgp ==SGP_WRITE )folio_mark_accessed (folio);delete_from_swap_cache (folio);folio_mark_dirty (folio); -swap_free (swap); +swap_free_nr (swap, nr_pages);put_swap_device (si); *foliop = folio;
Signature · 1 line
2.39.3
Baolin Wang
reviewerPATCH v3 9/10 mm: shmem: split large entry if the swapin folio is not large
In reply to Baolin WangNow the swap device can only swap-in order 0 folio, even though a large folio is swapped out. This requires us to split the large entry previously saved in the shmem pagecache to support the swap in of small folios.
--- mm/shmem.c | 95 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 95 insertions(+)
mm/shmem.c +95 -0
@@ -1990,6 +1990,82 @@ static void shmem_set_folio_swapin_error(struct inode *inode, pgoff_t index,swap_free_nr (swap, nr_pages); } +static int shmem_split_large_entry (struct inode *inode,pgoff_t index, +swp_entry_t swap,int new_order,gfp_t gfp) +{ +struct address_space *mapping = inode->i_mapping; +XA_STATE_ORDER (xas, &mapping->i_pages, index, new_order); +void *alloced_shadow =NULL ; +int alloced_order =0 , i; + +for (;;) { +int order =-1 , split_order =0 ; +void *old =NULL ; + +xas_lock_irq (&xas); + old =xas_load (&xas); +if (!xa_is_value (old) ||swp_to_radix_entry (swap) != old) { +xas_set_err (&xas, -EEXIST ); + goto unlock; + } + +if (order ==-1 ) + order =xas_get_order (&xas); + +/* Swap entry may have changed before we re-acquire the lock */ +if (alloced_order && + (old != alloced_shadow || order != alloced_order)) { +xas_destroy (&xas); + alloced_order =0 ; + } + +/* Try to split large swap entry in pagecache */ +if (order >0 && order > new_order) { +if (!alloced_order) { + split_order = order; + goto unlock; + } +xas_split (&xas, old, order); + +/* +* Re-set the swap entry after splitting, and the swap +* offset of the original large entry must be continuous. +*/ +for (i =0 ; i <1 << order; i += (1 << new_order)) { +pgoff_t aligned_index =round_down (index,1 << order); +swp_entry_t tmp; + + tmp =swp_entry (swp_type (swap),swp_offset (swap) + i); +__xa_store (&mapping->i_pages, aligned_index + i, +swp_to_radix_entry (tmp),0 ); + } + } + +unlock: +xas_unlock_irq (&xas); + +/* split needed, alloc here and retry. */ +if (split_order) { +xas_split_alloc (&xas, old, split_order, gfp); +if (xas_error (&xas)) + goto error; + alloced_shadow = old; + alloced_order = split_order; +xas_reset (&xas); +continue ; + } + +if (!xas_nomem (&xas, gfp)) +break ; + } + +error: +if (xas_error (&xas)) +return xas_error (&xas); + +return alloced_order; +} +/* * Swap in the folio pointed to by *foliop. * Caller has to make sure that *foliop contains a valid swapped folio.
@@ -2026,12 +2102,31 @@ static int shmem_swapin_folio(struct inode *inode, pgoff_t index,/* Look it up and read it in.. */ folio =swap_cache_get_folio (swap,NULL ,0 );if (!folio) { +int split_order, offset; +/* Or update major stats only when swapin succeeds?? */ if (fault_type) { *fault_type |=VM_FAULT_MAJOR ;count_vm_event (PGMAJFAULT );count_memcg_event_mm (fault_mm,PGMAJFAULT ); } + +/* +* Now swap device can only swap in order 0 folio, then we +* should split the large swap entry stored in the pagecache +* if necessary. +*/ + split_order =shmem_split_large_entry (inode, index, swap,0 , gfp); +if (split_order <0 ) { + error = split_order; + goto failed; + } + +if (split_order >0 ) { + offset = index -round_down (index,1 << split_order); + swap =swp_entry (swp_type (swap),swp_offset (swap) + offset); + } +/* Here we actually start the io */ folio =shmem_swapin_cluster (swap, gfp, info, index);if (!folio) {
Signature · 1 line
2.39.3
Baolin Wang
reviewerPATCH v3 10/10 mm: shmem: support large folio swap out
In reply to Baolin WangShmem will support large folio allocation [1] [2] to get a better performance, however, the memory reclaim still splits the precious large folios when trying to swap out shmem, which may lead to the memory fragmentation issue and can not take advantage of the large folio for shmeme.
Moreover, the swap code already supports for swapping out large folio without split, hence this patch set supports the large folio swap out for shmem.
Note the i915_gem_shmem driver still need to be split when swapping, thus add a new flag 'split_large_folio' for writeback_control to indicate spliting the large folio.
[1] https://lore.kernel.org/all/[email protected]/
[2] https://lore.kernel.org/all/[email protected]/
--- drivers/gpu/drm/i915/gem/i915_gem_shmem.c | 1 + include/linux/writeback.h | 1 + mm/shmem.c | 3 +-- mm/vmscan.c | 14 ++++++++++++-- 4 files changed, 15 insertions(+), 4 deletions(-)
drivers/gpu/drm/i915/gem/i915_gem_shmem.c +1 -0
@@ -308,6 +308,7 @@ void __shmem_writeback(size_t size, struct address_space *mapping) .range_start = 0, .range_end = LLONG_MAX, .for_reclaim = 1, + .split_large_folio = 1, };unsigned long i;
include/linux/writeback.h +1 -0
@@ -63,6 +63,7 @@ struct writeback_control { unsigned range_cyclic:1; /* range_start is cyclic */ unsigned for_sync:1; /* sync(2) WB_SYNC_ALL writeback */ unsigned unpinned_netfs_wb:1; /* Cleared I_PINNING_NETFS_WB */ + unsigned split_large_folio:1; /* Split large folio for shmem writeback *//* * When writeback IOs are bounced through async layers, only the
mm/shmem.c +1 -2
@@ -786,7 +786,6 @@ static int shmem_add_to_page_cache(struct folio *folio,VM_BUG_ON_FOLIO (index !=round_down (index, nr), folio);VM_BUG_ON_FOLIO (!folio_test_locked (folio), folio);VM_BUG_ON_FOLIO (!folio_test_swapbacked (folio), folio); -VM_BUG_ON (expected &&folio_test_large (folio));folio_ref_add (folio, nr); folio->mapping = mapping;
@@ -1473,7 +1472,7 @@ static int shmem_writepage(struct page *page, struct writeback_control *wbc)* "force", drivers/gpu/drm/i915/gem/i915_gem_shmem.c gets huge pages, * and its shmem_writeback() needs them to be split when swapping. */ -if (folio_test_large (folio)) { +if (wbc->split_large_folio &&folio_test_large (folio)) {/* Ensure the subpages are still dirty */ folio_test_set_dirty (folio);if (split_huge_page (page) <0 )
mm/vmscan.c +12 -2
@@ -1245,8 +1245,12 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,if (!total_swap_pages) goto activate_locked; -/* Split shmem folio */ -if (split_folio_to_list (folio, folio_list)) +/* +* Only split shmem folio when CONFIG_THP_SWAP +* is not enabled. +*/ +if (!IS_ENABLED (CONFIG_THP_SWAP ) && +split_folio_to_list (folio, folio_list)) goto keep_locked; }
@@ -1348,10 +1352,16 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,* starts and then write it out here. */ try_to_unmap_flush_dirty (); +try_pageout:switch (pageout (folio, mapping, &plug)) {case PAGE_KEEP : goto keep_locked;case PAGE_ACTIVATE : +if (shmem_mapping (mapping) &&folio_test_large (folio) && + !split_folio_to_list (folio, folio_list)) { + nr_pages =1 ; + goto try_pageout; + } goto activate_locked;case PAGE_SUCCESS : stat->nr_pageout += nr_pages;
Signature · 1 line
2.39.3
Baolin Wang
reviewerPATCH v3 7/10 mm: shmem: support large folio allocation for shmem_replace_folio()
In reply to Baolin WangTo support large folio swapin for shmem in the following patches, add large folio allocation for the new replacement folio in shmem_replace_folio(). Moreover large folios occupy N consecutive entries in the swap cache instead of using multi-index entries like the page cache, therefore we should replace each consecutive entries in the swap cache instead of using the shmem_replace_entry().
As well as updating statistics and folio reference count using the number of pages in the folio.
--- mm/shmem.c | 54 +++++++++++++++++++++++++++++++----------------------- 1 file changed, 31 insertions(+), 23 deletions(-)
mm/shmem.c +31 -23
@@ -1889,28 +1889,24 @@ static bool shmem_should_replace_folio(struct folio *folio, gfp_t gfp)static int shmem_replace_folio (struct folio **foliop,gfp_t gfp,struct shmem_inode_info *info,pgoff_t index) { -struct folio *old, *new; -struct address_space *swap_mapping; -swp_entry_t entry; -pgoff_t swap_index; -int error; - - old = *foliop; - entry = old->swap; - swap_index =swap_cache_index (entry); - swap_mapping =swap_address_space (entry); +struct folio *new, *old = *foliop; +swp_entry_t entry = old->swap; +struct address_space *swap_mapping =swap_address_space (entry); +pgoff_t swap_index =swap_cache_index (entry); +XA_STATE (xas, &swap_mapping->i_pages, swap_index); +int nr_pages =folio_nr_pages (old); +int error =0 , i;/* * We have arrived here because our zones are constrained, so don't * limit chance of success by further cpuset and node constraints. */ gfp &= ~GFP_CONSTRAINT_MASK ; -VM_BUG_ON_FOLIO (folio_test_large (old), old); - new =shmem_alloc_folio (gfp,0 , info, index); + new =shmem_alloc_folio (gfp,folio_order (old), info, index);if (!new)return -ENOMEM ; -folio_get (new); +folio_ref_add (new, nr_pages);folio_copy (new, old);flush_dcache_folio (new);
@@ -1920,18 +1916,25 @@ static int shmem_replace_folio(struct folio **foliop, gfp_t gfp, new->swap = entry;folio_set_swapcache (new); -/* -* Our caller will very soon move newpage out of swapcache, but it's -* a nice clean interface for us to replace oldpage by newpage there. -*/ +/* Swap cache still stores N entries instead of a high-order entry */ xa_lock_irq (&swap_mapping->i_pages); - error =shmem_replace_entry (swap_mapping, swap_index, old, new); +for (i =0 ; i < nr_pages; i++) { +void *item =xas_load (&xas); + +if (item != old) { + error = -ENOENT ; +break ; + } + +xas_store (&xas, new); +xas_next (&xas); + }if (!error) {mem_cgroup_replace_folio (old, new); -__lruvec_stat_mod_folio (new,NR_FILE_PAGES ,1 ); -__lruvec_stat_mod_folio (new,NR_SHMEM ,1 ); -__lruvec_stat_mod_folio (old,NR_FILE_PAGES ,-1 ); -__lruvec_stat_mod_folio (old,NR_SHMEM ,-1 ); +__lruvec_stat_mod_folio (new,NR_FILE_PAGES , nr_pages); +__lruvec_stat_mod_folio (new,NR_SHMEM , nr_pages); +__lruvec_stat_mod_folio (old,NR_FILE_PAGES , -nr_pages); +__lruvec_stat_mod_folio (old,NR_SHMEM , -nr_pages); }xa_unlock_irq (&swap_mapping->i_pages);
@@ -1951,7 +1954,12 @@ static int shmem_replace_folio(struct folio **foliop, gfp_t gfp, old->private =NULL ;folio_unlock (old); -folio_put_refs (old,2 ); +/* +* The old folio are removed from swap cache, drop the 'nr_pages' +* reference, as well as one temporary reference getting from swap +* cache. +*/ +folio_put_refs (old, nr_pages +1 );return error; }
Signature · 1 line
2.39.3
Baolin Wang
reviewerPATCH v3 8/10 mm: shmem: drop folio reference count using 'nr_pages' in shmem_delete_from_page_cache()
In reply to Baolin WangTo support large folio swapin/swapout for shmem in the following patches, drop the folio's reference count by the number of pages contained in the folio when a shmem folio is deleted from shmem pagecache after adding into swap cache.
--- mm/shmem.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
mm/shmem.c +1 -1
@@ -842,7 +842,7 @@ static void shmem_delete_from_page_cache(struct folio *folio, void *radswap)__lruvec_stat_mod_folio (folio,NR_FILE_PAGES , -nr);__lruvec_stat_mod_folio (folio,NR_SHMEM , -nr);xa_unlock_irq (&mapping->i_pages); -folio_put (folio); +folio_put_refs (folio, nr);BUG_ON (error); }
Signature · 1 line
2.39.3