Skip to contentSkip to conversation

mm/swap: optimize swap cache search space

PATCH In review Kairui Song 6 participants 19 messages Apr 23, 2024 – Apr 28, 2024

Subsystems (15)
lore
Share & tools
lore
#0 lore raw

v2 cover mm/swap: optimize swap cache search space

From: Kairui Song <[email protected]>

Currently we use one swap_address_space for every 64M chunk to reduce lock contention, this is like having a set of smaller swap files inside one big swap file. But when doing swap cache look up or insert, we are still using the offset of the whole large swap file. This is OK for correctness, as the offset (key) is unique.

But Xarray is specially optimized for small indexes, it creates the redix tree levels lazily to be just enough to fit the largest key stored in one Xarray. So we are wasting tree nodes unnecessarily.

For 64M chunk it should only take at most 3 level to contain everything. But we are using the offset from the whole swap file, so the offset (key) value will be way beyond 64M, and so will the tree level.

Optimize this by reduce the swap cache search space into 64M scope.

Test with time memhog 128G inside a 8G memcg using 128G swap (ramdisk with SWP_SYNCHRONOUS_IO dropped, tested 3 times, results are stable. The test result is similar but the improvement is smaller if SWP_SYNCHRONOUS_IO is enabled, as swap out path can never skip swap cache):

Before:
6.07user 250.74system 4:17.26elapsed 99%CPU (0avgtext+0avgdata 8373376maxresident)k 0inputs+0outputs (55major+33555018minor)pagefaults 0swaps

After (+1.8% faster):
6.08user 246.09system 4:12.58elapsed 99%CPU (0avgtext+0avgdata 8373248maxresident)k 0inputs+0outputs (54major+33555027minor)pagefaults 0swaps

Similar result with MySQL and sysbench using swap:
Before:
94055.61 qps

After (+0.8% faster):
94834.91 qps

There is alse a very slight drop of radix tree node slab usage:

Before: 303952K
After:  302224K

For this series:

There are multiple places that expect mixed type of pages (page cache or swap cache), eg. migration, huge memory split; There are four helpers for that:

- page_index
- page_file_offset
- folio_index
- folio_file_pos

To keep the code clean and compatible, this series first cleaned up usage of page_index and page_file_offset, then convert folio_index and folio_file_pos to be compatible with separate offsets. Current users won't be effected. And introduce two new helper swap_cache_index and swap_dev_pos for swap internal usage. Replace swp_offset with swap_cache_index when used to retrieve folio from swap cache, and use swap_dev_pos when needed to retrieve the device position of a swap entry.

Idealy, in the future, we may want to reduce SWAP_ADDRESS_SPACE_SHIFT from 14 to 12: Default Xarray chunk offset is 6, so we have 3 level trees instead of 2 level trees just for 2 extra bits. But swap cache is based on address_space struct, with 4 times more metadata sparsely distributed in memory it waste more cacheline, the performance gain from this series is almost canceled according to my test. So first, just have a cleaner seperation of offsets and smaller search space.

Patch 1/8 - 6/8: Clean up usage of page_index and page_file_offset Patch 7/8: Convert folio_index and folio_dev_pos to be compatible with
separate offset.
Patch 8/8: Introduce swap_cache_index and use it when doing lookup in
swap cache.

V1: https://lore.kernel.org/all/[email protected]/
Update from V1:
- Convert more users to use folio directly when possible [Matthew Wilcox]
- Rename swap_file_pos to swap_dev_pos [Huang, Ying]
- Update comments and commit message.
- Adjust headers and add dummy function to fix build error.

This series is part of effort to reduce swap cache overhead, and ultimately remove SWP_SYNCHRONOUS_IO and unify swap cache usage as proposed before:
https://lore.kernel.org/lkml/[email protected]/

Kairui Song (8):
NFS: remove nfs_page_lengthg and usage of page_index
nilfs2: drop usage of page_index
f2fs: drop usage of page_index
ceph: drop usage of page_index
cifs: drop usage of page_file_offset
mm/swap: get the swap file offset directly
mm: drop page_index/page_file_offset and convert swap helpers to use
folio
mm/swap: reduce swap cache search space

 fs/ceph/dir.c           |  2 +-
 fs/ceph/inode.c         |  2 +-
 fs/f2fs/data.c          |  5 ++---
 fs/nfs/internal.h       | 19 -------------------
 fs/nilfs2/bmap.c        |  3 +--
 fs/smb/client/file.c    |  2 +-
 include/linux/mm.h      | 13 -------------
 include/linux/pagemap.h | 19 +++++++++----------
 mm/huge_memory.c        |  2 +-
 mm/memcontrol.c         |  2 +-
 mm/mincore.c            |  2 +-
 mm/page_io.c            |  6 +++---
 mm/shmem.c              |  2 +-
 mm/swap.h               | 24 ++++++++++++++++++++++++
 mm/swap_state.c         | 12 ++++++------
 mm/swapfile.c           | 17 +++++++++++------
 16 files changed, 63 insertions(+), 69 deletions(-)
Signature · 1 line
2.44.0
#1 lore raw

PATCH v2 1/8 NFS: remove nfs_page_lengthg and usage of page_index

From: Kairui Song <[email protected]>

This function is no longer used after
commit 4fa7a717b432 ("NFS: Fix up nfs_vm_page_mkwrite() for folios"), all users have been converted to use folio instead, just delete it to remove usage of page_index.

Cc Trond Myklebust <[email protected]>
Cc Anna Schumaker <[email protected]>
---
 fs/nfs/internal.h | 19 -------------------
 1 file changed, 19 deletions(-)
fs/nfs/internal.h +0 -19
index 06253695fe53..deac98dce6ac 100644
@@ -790,25 +790,6 @@ static inline void nfs_folio_mark_unstable(struct folio *folio,
 	}
 }
 
-/*
- * Determine the number of bytes of data the page contains
- */
-static inline
-unsigned int nfs_page_length(struct page *page)
-{
-	loff_t i_size = i_size_read(page_file_mapping(page)->host);
-
-	if (i_size > 0) {
-		pgoff_t index = page_index(page);
-		pgoff_t end_index = (i_size - 1) >> PAGE_SHIFT;
-		if (index < end_index)
-			return PAGE_SIZE;
-		if (index == end_index)
-			return ((i_size - 1) & ~PAGE_MASK) + 1;
-	}
-	return 0;
-}
-
 /*
  * Determine the number of bytes of data the page contains
  */
Signature · 1 line
2.44.0
#2 lore raw

PATCH v2 2/8 nilfs2: drop usage of page_index

In reply to Kairui Song

From: Kairui Song <[email protected]>

page_index is only for mixed usage of page cache and swap cache, for pure page cache usage, the caller can just use page->index instead.

It can't be a swap cache page here (being part of buffer head), so just drop it, also convert it to use folio.

---
 fs/nilfs2/bmap.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
fs/nilfs2/bmap.c +1 -2
index 383f0afa2cea..9f561afe864f 100644
@@ -453,8 +453,7 @@ __u64 nilfs_bmap_data_get_key(const struct nilfs_bmap *bmap,
 	struct buffer_head *pbh;
 	__u64 key;
 
-	key = page_index(bh->b_page) << (PAGE_SHIFT -
-					 bmap->b_inode->i_blkbits);
+	key = bh->b_folio->index << (PAGE_SHIFT - bmap->b_inode->i_blkbits);
 	for (pbh = page_buffers(bh->b_page); pbh != bh; pbh = pbh->b_this_page)
 		key++;
 
Signature · 1 line
2.44.0
maintainer
#3 lore raw

Re: [PATCH v2 2/8] nilfs2: drop usage of page_index

Quoted 21 lines · Kairui Song

From: Kairui Song <[email protected]>

page_index is only for mixed usage of page cache and swap cache, for pure page cache usage, the caller can just use page->index instead.

It can't be a swap cache page here (being part of buffer head), so just drop it, also convert it to use folio.

---
 fs/nilfs2/bmap.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
fs/nilfs2/bmap.c +0 -0
index 383f0afa2cea..9f561afe864f 100644

fs/nilfs2/bmap.c __u64 nilfs_bmap_data_get_key(const struct nilfs_bmap *bmap, · line 458

Quoted 8 lines
fs/nilfs2/bmap.c +1 -2
@@ -453,8 +453,7 @@ __u64 nilfs_bmap_data_get_key(const struct nilfs_bmap *bmap,
        struct buffer_head *pbh;
        __u64 key;
 
-       key = page_index(bh->b_page) << (PAGE_SHIFT -
-                                        bmap->b_inode->i_blkbits);
+       key = bh->b_folio->index << (PAGE_SHIFT - bmap->b_inode->i_blkbits);
        for (pbh = page_buffers(bh->b_page); pbh != bh; pbh = pbh->b_this_page)
                key++;

This conversion mixes the use of page and folio within the function. Would you like to take the opportunity to convert "page_buffers(bh->b_page)" to "folio_buffers(bh->b_folio)" as well?

Thanks,
Ryusuke Konishi

#4 lore raw

Re: [PATCH v2 2/8] nilfs2: drop usage of page_index

Quoted 26 lines · Ryusuke Konishi
Quoted 24 lines · Kairui Song

From: Kairui Song <[email protected]>

page_index is only for mixed usage of page cache and swap cache, for pure page cache usage, the caller can just use page->index instead.

It can't be a swap cache page here (being part of buffer head), so just drop it, also convert it to use folio.

---
 fs/nilfs2/bmap.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
fs/nilfs2/bmap.c +0 -0
index 383f0afa2cea..9f561afe864f 100644
@@ -453,8 +453,7 @@ __u64 nilfs_bmap_data_get_key(const struct nilfs_bmap *bmap,
        struct buffer_head *pbh;
        __u64 key;
 
Quoted 9 lines

fs/nilfs2/bmap.c __u64 nilfs_bmap_data_get_key(const struct nilfs_bmap *bmap, · line 458

Quoted 5 lines
fs/nilfs2/bmap.c +1 -2
-       key = page_index(bh->b_page) << (PAGE_SHIFT -
-                                        bmap->b_inode->i_blkbits);
+       key = bh->b_folio->index << (PAGE_SHIFT - bmap->b_inode->i_blkbits);
        for (pbh = page_buffers(bh->b_page); pbh != bh; pbh = pbh->b_this_page)
                key++;

This conversion mixes the use of page and folio within the function. Would you like to take the opportunity to convert "page_buffers(bh->b_page)" to "folio_buffers(bh->b_folio)" as well?

OK, will update this part.

#5 lore raw

PATCH v2 3/8 f2fs: drop usage of page_index

In reply to Kairui Song

From: Kairui Song <[email protected]>

page_index is needed for mixed usage of page cache and swap cache, for pure page cache usage, the caller can just use page->index instead.

It can't be a swap cache page here, so just drop it.

---
 fs/f2fs/data.c | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)
fs/f2fs/data.c +2 -3
index d9494b5fc7c1..12d5bbd18755 100644
@@ -2057,7 +2057,7 @@ static int f2fs_read_single_page(struct inode *inode, struct page *page,
 	sector_t block_nr;
 	int ret = 0;
 
-	block_in_file = (sector_t)page_index(page);
+	block_in_file = (sector_t)page->index;
 	last_block = block_in_file + nr_pages;
 	last_block_in_file = bytes_to_blks(inode,
 			f2fs_readpage_limit(inode) + blocksize - 1);
@@ -4086,8 +4086,7 @@ void f2fs_clear_page_cache_dirty_tag(struct page *page)
 	unsigned long flags;
 
 	xa_lock_irqsave(&mapping->i_pages, flags);
-	__xa_clear_mark(&mapping->i_pages, page_index(page),
-						PAGECACHE_TAG_DIRTY);
+	__xa_clear_mark(&mapping->i_pages, page->index, PAGECACHE_TAG_DIRTY);
 	xa_unlock_irqrestore(&mapping->i_pages, flags);
 }
 
Signature · 1 line
2.44.0
maintainer
#6 lore raw

Re: [PATCH v2 3/8] f2fs: drop usage of page_index

fs/f2fs/data.c void f2fs_clear_page_cache_dirty_tag(struct page *page) · line 4091

Quoted 9 lines · Kairui Song
fs/f2fs/data.c +1 -2
@@ -4086,8 +4086,7 @@ void f2fs_clear_page_cache_dirty_tag(struct page *page)
 	unsigned long flags;
 
 	xa_lock_irqsave(&mapping->i_pages, flags);
-	__xa_clear_mark(&mapping->i_pages, page_index(page),
-						PAGECACHE_TAG_DIRTY);
+	__xa_clear_mark(&mapping->i_pages, page->index, PAGECACHE_TAG_DIRTY);
 	xa_unlock_irqrestore(&mapping->i_pages, flags);
 }

I just sent a patch which is going to conflict with this:

https://lore.kernel.org/linux-mm/[email protected]/

Chao Yu, Jaegeuk Kim; what are your plans for converting f2fs to use folios? This is getting quite urgent.

maintainer
#7 lore raw

Re: [PATCH v2 3/8] f2fs: drop usage of page_index

Quoted 13 lines · Matthew Wilcox

fs/f2fs/data.c void f2fs_clear_page_cache_dirty_tag(struct page *page) · line 4091

Quoted 9 lines · Kairui Song
fs/f2fs/data.c +1 -2
@@ -4086,8 +4086,7 @@ void f2fs_clear_page_cache_dirty_tag(struct page *page)
  	unsigned long flags;
 
  	xa_lock_irqsave(&mapping->i_pages, flags);
-	__xa_clear_mark(&mapping->i_pages, page_index(page),
-						PAGECACHE_TAG_DIRTY);
+	__xa_clear_mark(&mapping->i_pages, page->index, PAGECACHE_TAG_DIRTY);
  	xa_unlock_irqrestore(&mapping->i_pages, flags);
  }

I just sent a patch which is going to conflict with this:

Quoted 3 lines

https://lore.kernel.org/linux-mm/[email protected]/

Chao Yu, Jaegeuk Kim; what are your plans for converting f2fs to use

Hi Matthew,

I've converted .read_folio and .readahead of f2fs to use folio w/ below patchset, and let me take a look how to support and enable large folio...

https://lore.kernel.org/linux-f2fs-devel/[email protected]/

Thanks,

Quoted 1 line

folios? This is getting quite urgent.

#8 lore raw

PATCH v2 4/8 ceph: drop usage of page_index

In reply to Kairui Song

From: Kairui Song <[email protected]>

page_index is needed for mixed usage of page cache and swap cache, for pure page cache usage, the caller can just use page->index instead.

It can't be a swap cache page here, so just drop it.

---
 fs/ceph/dir.c   | 2 +-
 fs/ceph/inode.c | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
fs/ceph/dir.c +1 -1
index 0e9f56eaba1e..570a9d634cc5 100644
@@ -141,7 +141,7 @@ __dcache_find_get_entry(struct dentry *parent, u64 idx,
 	if (ptr_pos >= i_size_read(dir))
 		return NULL;
 
-	if (!cache_ctl->page || ptr_pgoff != page_index(cache_ctl->page)) {
+	if (!cache_ctl->page || ptr_pgoff != cache_ctl->page->index) {
 		ceph_readdir_cache_release(cache_ctl);
 		cache_ctl->page = find_lock_page(&dir->i_data, ptr_pgoff);
 		if (!cache_ctl->page) {
fs/ceph/inode.c +1 -1
index 7b2e77517f23..1f92d3faaa6b 100644
@@ -1861,7 +1861,7 @@ static int fill_readdir_cache(struct inode *dir, struct dentry *dn,
 	unsigned idx = ctl->index % nsize;
 	pgoff_t pgoff = ctl->index / nsize;
 
-	if (!ctl->page || pgoff != page_index(ctl->page)) {
+	if (!ctl->page || pgoff != ctl->page->index) {
 		ceph_readdir_cache_release(ctl);
 		if (idx == 0)
 			ctl->page = grab_cache_page(&dir->i_data, pgoff);
Signature · 1 line
2.44.0
Reviewed-by on v2 4/8
#9 lore raw

Re: [PATCH v2 4/8] ceph: drop usage of page_index

Quoted 35 lines · Kairui Song

From: Kairui Song <[email protected]>

page_index is needed for mixed usage of page cache and swap cache, for pure page cache usage, the caller can just use page->index instead.

It can't be a swap cache page here, so just drop it.

---
  fs/ceph/dir.c   | 2 +-
  fs/ceph/inode.c | 2 +-
  2 files changed, 2 insertions(+), 2 deletions(-)
fs/ceph/dir.c +1 -1
index 0e9f56eaba1e..570a9d634cc5 100644
@@ -141,7 +141,7 @@ __dcache_find_get_entry(struct dentry *parent, u64 idx,
  	if (ptr_pos >= i_size_read(dir))
  		return NULL;
 
-	if (!cache_ctl->page || ptr_pgoff != page_index(cache_ctl->page)) {
+	if (!cache_ctl->page || ptr_pgoff != cache_ctl->page->index) {
  		ceph_readdir_cache_release(cache_ctl);
  		cache_ctl->page = find_lock_page(&dir->i_data, ptr_pgoff);
  		if (!cache_ctl->page) {
fs/ceph/inode.c +0 -0
index 7b2e77517f23..1f92d3faaa6b 100644

fs/ceph/inode.c static int fill_readdir_cache(struct inode *dir, struct dentry *dn, · line 1867

Quoted 8 lines
fs/ceph/inode.c +1 -1
@@ -1861,7 +1861,7 @@ static int fill_readdir_cache(struct inode *dir, struct dentry *dn,
  	unsigned idx = ctl->index % nsize;
  	pgoff_t pgoff = ctl->index / nsize;
 
-	if (!ctl->page || pgoff != page_index(ctl->page)) {
+	if (!ctl->page || pgoff != ctl->page->index) {
  		ceph_readdir_cache_release(ctl);
  		if (idx == 0)
  			ctl->page = grab_cache_page(&dir->i_data, pgoff);

LGTM.

#10 lore raw

PATCH v2 5/8 cifs: drop usage of page_file_offset

In reply to Kairui Song

From: Kairui Song <[email protected]>

page_file_offset is only needed for mixed usage of page cache and swap cache, for pure page cache usage, the caller can just use page_offset instead.

It can't be a swap cache page here, so just drop it and convert it to use folio.

Cc Steve French <[email protected]>
Cc Namjae Jeon <[email protected]>
Cc Paulo Alcantara <[email protected]>
Cc Shyam Prasad N <[email protected]>
Cc Bharath SM <[email protected]>
---
 fs/smb/client/file.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
fs/smb/client/file.c +1 -1
index 16aadce492b2..56ebc0b1444d 100644
@@ -4749,7 +4749,7 @@ static int cifs_readpage_worker(struct file *file, struct page *page,
 static int cifs_read_folio(struct file *file, struct folio *folio)
 {
 	struct page *page = &folio->page;
-	loff_t offset = page_file_offset(page);
+	loff_t offset = folio_pos(folio);
 	int rc = -EACCES;
 	unsigned int xid;
 
Signature · 1 line
2.44.0
#11 lore raw

PATCH v2 6/8 mm/swap: get the swap file offset directly

In reply to Kairui Song

From: Kairui Song <[email protected]>

folio_file_pos and page_file_offset are for mixed usage of swap cache and page cache, it can't be page cache here, so introduce a new helper to get the swap offset in swap file directly.

Need to include swapops.h in mm/swap.h to ensure swp_offset is always defined before use.

---
 mm/page_io.c | 6 +++---
 mm/swap.h    | 9 +++++++++
 2 files changed, 12 insertions(+), 3 deletions(-)
mm/page_io.c +3 -3
index ae2b49055e43..615812cfe4ca 100644
@@ -279,7 +279,7 @@ static void sio_write_complete(struct kiocb *iocb, long ret)
 		 * be temporary.
 		 */
 		pr_err_ratelimited("Write error %ld on dio swapfile (%llu)\n",
-				   ret, page_file_offset(page));
+				   ret, swap_dev_pos(page_swap_entry(page)));
 		for (p = 0; p < sio->pages; p++) {
 			page = sio->bvec[p].bv_page;
 			set_page_dirty(page);
@@ -298,7 +298,7 @@ static void swap_writepage_fs(struct folio *folio, struct writeback_control *wbc
 	struct swap_iocb *sio = NULL;
 	struct swap_info_struct *sis = swp_swap_info(folio->swap);
 	struct file *swap_file = sis->swap_file;
-	loff_t pos = folio_file_pos(folio);
+	loff_t pos = swap_dev_pos(folio->swap);
 
 	count_swpout_vm_event(folio);
 	folio_start_writeback(folio);
@@ -429,7 +429,7 @@ static void swap_read_folio_fs(struct folio *folio, struct swap_iocb **plug)
 {
 	struct swap_info_struct *sis = swp_swap_info(folio->swap);
 	struct swap_iocb *sio = NULL;
-	loff_t pos = folio_file_pos(folio);
+	loff_t pos = swap_dev_pos(folio->swap);
 
 	if (plug)
 		sio = *plug;
mm/swap.h +9 -0
index fc2f6ade7f80..82023ab93205 100644
@@ -5,6 +5,7 @@
 struct mempolicy;
 
 #ifdef CONFIG_SWAP
+#include <linux/swapops.h> /* for swp_offset */
 #include <linux/blk_types.h> /* for bio_end_io_t */
 
 /* linux/mm/page_io.c */
@@ -31,6 +32,14 @@ extern struct address_space *swapper_spaces[];
 	(&swapper_spaces[swp_type(entry)][swp_offset(entry) \
 		>> SWAP_ADDRESS_SPACE_SHIFT])
 
+/*
+ * Return the swap device position of the swap entry.
+ */
+static inline loff_t swap_dev_pos(swp_entry_t entry)
+{
+	return ((loff_t)swp_offset(entry)) << PAGE_SHIFT;
+}
+
 void show_swap_cache_info(void);
 bool add_to_swap(struct folio *folio);
 void *get_shadow_from_swap_cache(swp_entry_t entry);
Signature · 1 line
2.44.0
#12 lore raw

PATCH v2 7/8 mm: drop page_index/page_file_offset and convert swap helpers to use folio

In reply to Kairui Song

From: Kairui Song <[email protected]>

There are four helpers for retrieving the page index within address space, or offset within mapped file:

- page_index
- page_file_offset
- folio_index (equivalence of page_index)
- folio_file_pos (equivalence of page_file_offset)

And they are only needed for mixed usage of swap cache & page cache (eg. migration, huge memory split). Else users can just use folio->index or folio_pos.

This commit drops page_index and page_file_offset as we have eliminated all users, and converts folio_index and folio_file_pos (they were simply wrappers of page_index and page_file_offset, and implemented in a not very clean way) to use folio internally.

After this commit, there will be only two helpers for users that may encounter mixed usage of swap cache and page cache:

- folio_index (calls __folio_swap_cache_index for swap cache folio)
- folio_file_pos (calls __folio_swap_dev_pos for swap cache folio)

The index in swap cache and offset in swap device are still basically the same thing, but will be different in following commits.

---
 include/linux/mm.h      | 13 -------------
 include/linux/pagemap.h | 19 +++++++++----------
 mm/swapfile.c           | 13 +++++++++----
 3 files changed, 18 insertions(+), 27 deletions(-)
include/linux/mm.h +0 -13
index 0436b919f1c7..797480e76c9c 100644
@@ -2245,19 +2245,6 @@ static inline void *folio_address(const struct folio *folio)
 	return page_address(&folio->page);
 }
 
-extern pgoff_t __page_file_index(struct page *page);
-
-/*
- * Return the pagecache index of the passed page.  Regular pagecache pages
- * use ->index whereas swapcache pages use swp_offset(->private)
- */
-static inline pgoff_t page_index(struct page *page)
-{
-	if (unlikely(PageSwapCache(page)))
-		return __page_file_index(page);
-	return page->index;
-}
-
 /*
  * Return true only if the page has been allocated with
  * ALLOC_NO_WATERMARKS and the low watermark was not
include/linux/pagemap.h +9 -10
index 2df35e65557d..a7d025571ee6 100644
@@ -780,7 +780,7 @@ static inline struct page *grab_cache_page_nowait(struct address_space *mapping,
 			mapping_gfp_mask(mapping));
 }
 
-#define swapcache_index(folio)	__page_file_index(&(folio)->page)
+extern pgoff_t __folio_swap_cache_index(struct folio *folio);
 
 /**
  * folio_index - File index of a folio.
@@ -795,9 +795,9 @@ static inline struct page *grab_cache_page_nowait(struct address_space *mapping,
  */
 static inline pgoff_t folio_index(struct folio *folio)
 {
-        if (unlikely(folio_test_swapcache(folio)))
-                return swapcache_index(folio);
-        return folio->index;
+	if (unlikely(folio_test_swapcache(folio)))
+		return __folio_swap_cache_index(folio);
+	return folio->index;
 }
 
 /**
@@ -920,11 +920,6 @@ static inline loff_t page_offset(struct page *page)
 	return ((loff_t)page->index) << PAGE_SHIFT;
 }
 
-static inline loff_t page_file_offset(struct page *page)
-{
-	return ((loff_t)page_index(page)) << PAGE_SHIFT;
-}
-
 /**
  * folio_pos - Returns the byte position of this folio in its file.
  * @folio: The folio.
@@ -934,6 +929,8 @@ static inline loff_t folio_pos(struct folio *folio)
 	return page_offset(&folio->page);
 }
 
+extern loff_t __folio_swap_dev_pos(struct folio *folio);
+
 /**
  * folio_file_pos - Returns the byte position of this folio in its file.
  * @folio: The folio.
@@ -943,7 +940,9 @@ static inline loff_t folio_pos(struct folio *folio)
  */
 static inline loff_t folio_file_pos(struct folio *folio)
 {
-	return page_file_offset(&folio->page);
+	if (unlikely(folio_test_swapcache(folio)))
+		return __folio_swap_dev_pos(folio);
+	return ((loff_t)folio->index << PAGE_SHIFT);
 }
 
 /*
mm/swapfile.c +9 -4
index 4919423cce76..2387f5e131d7 100644
@@ -3419,12 +3419,17 @@ struct address_space *swapcache_mapping(struct folio *folio)
 }
 EXPORT_SYMBOL_GPL(swapcache_mapping);
 
-pgoff_t __page_file_index(struct page *page)
+pgoff_t __folio_swap_cache_index(struct folio *folio)
 {
-	swp_entry_t swap = page_swap_entry(page);
-	return swp_offset(swap);
+	return swp_offset(folio->swap);
 }
-EXPORT_SYMBOL_GPL(__page_file_index);
+EXPORT_SYMBOL_GPL(__folio_swap_cache_index);
+
+loff_t __folio_swap_dev_pos(struct folio *folio)
+{
+	return swap_dev_pos(folio->swap);
+}
+EXPORT_SYMBOL_GPL(__folio_swap_dev_pos);
 
 /*
  * add_swap_count_continuation - called when a swap count is duplicated
Signature · 1 line
2.44.0
#13 lore raw

Re: [PATCH v2 7/8] mm: drop page_index/page_file_offset and convert swap helpers to use folio

Quoted 107 lines · Kairui Song

From: Kairui Song <[email protected]>

There are four helpers for retrieving the page index within address space, or offset within mapped file:

- page_index
- page_file_offset
- folio_index (equivalence of page_index)
- folio_file_pos (equivalence of page_file_offset)

And they are only needed for mixed usage of swap cache & page cache (eg. migration, huge memory split). Else users can just use folio->index or folio_pos.

This commit drops page_index and page_file_offset as we have eliminated all users, and converts folio_index and folio_file_pos (they were simply wrappers of page_index and page_file_offset, and implemented in a not very clean way) to use folio internally.

After this commit, there will be only two helpers for users that may encounter mixed usage of swap cache and page cache:

- folio_index (calls __folio_swap_cache_index for swap cache folio)
- folio_file_pos (calls __folio_swap_dev_pos for swap cache folio)

The index in swap cache and offset in swap device are still basically the same thing, but will be different in following commits.

---
 include/linux/mm.h      | 13 -------------
 include/linux/pagemap.h | 19 +++++++++----------
 mm/swapfile.c           | 13 +++++++++----
 3 files changed, 18 insertions(+), 27 deletions(-)
include/linux/mm.h +0 -13
index 0436b919f1c7..797480e76c9c 100644
@@ -2245,19 +2245,6 @@ static inline void *folio_address(const struct folio *folio)
 	return page_address(&folio->page);
 }
 
-extern pgoff_t __page_file_index(struct page *page);
-
-/*
- * Return the pagecache index of the passed page.  Regular pagecache pages
- * use ->index whereas swapcache pages use swp_offset(->private)
- */
-static inline pgoff_t page_index(struct page *page)
-{
-	if (unlikely(PageSwapCache(page)))
-		return __page_file_index(page);
-	return page->index;
-}
-
 /*
  * Return true only if the page has been allocated with
  * ALLOC_NO_WATERMARKS and the low watermark was not
include/linux/pagemap.h +6 -9
index 2df35e65557d..a7d025571ee6 100644
@@ -780,7 +780,7 @@ static inline struct page *grab_cache_page_nowait(struct address_space *mapping,
 			mapping_gfp_mask(mapping));
 }
 
-#define swapcache_index(folio)	__page_file_index(&(folio)->page)
+extern pgoff_t __folio_swap_cache_index(struct folio *folio);
 
 /**
  * folio_index - File index of a folio.
@@ -795,9 +795,9 @@ static inline struct page *grab_cache_page_nowait(struct address_space *mapping,
  */
 static inline pgoff_t folio_index(struct folio *folio)
 {
-        if (unlikely(folio_test_swapcache(folio)))
-                return swapcache_index(folio);
-        return folio->index;
+	if (unlikely(folio_test_swapcache(folio)))
+		return __folio_swap_cache_index(folio);
+	return folio->index;
 }
 
 /**
@@ -920,11 +920,6 @@ static inline loff_t page_offset(struct page *page)
 	return ((loff_t)page->index) << PAGE_SHIFT;
 }
 
-static inline loff_t page_file_offset(struct page *page)
-{
-	return ((loff_t)page_index(page)) << PAGE_SHIFT;
-}
-
 /**
  * folio_pos - Returns the byte position of this folio in its file.
  * @folio: The folio.
@@ -934,6 +929,8 @@ static inline loff_t folio_pos(struct folio *folio)
 	return page_offset(&folio->page);
 }
 
+extern loff_t __folio_swap_dev_pos(struct folio *folio);
+
 /**
  * folio_file_pos - Returns the byte position of this folio in its file.
  * @folio: The folio.

include/linux/pagemap.h static inline loff_t folio_pos(struct folio *folio) · line 945

Quoted 7 lines
include/linux/pagemap.h +3 -1
@@ -943,7 +940,9 @@ static inline loff_t folio_pos(struct folio *folio)
  */
 static inline loff_t folio_file_pos(struct folio *folio)
 {
-	return page_file_offset(&folio->page);
+	if (unlikely(folio_test_swapcache(folio)))
+		return __folio_swap_dev_pos(folio);
+	return ((loff_t)folio->index << PAGE_SHIFT);

This still looks confusing for me. The function returns the byte position of the folio in its file. But we returns the swap device position of the folio.

Tried to search folio_file_pos() usage. The 2 usage in page_io.c is swap specific, we can use swap_dev_pos() directly.

There are also other file system users (NFS and AFS) of folio_file_pos(), I don't know why they need to work with swap cache. Cced file system maintainers for help.

Quoted 19 lines
include/linux/pagemap.h +0 -0
 }
 
 /*
mm/swapfile.c +2 -4
index 4919423cce76..2387f5e131d7 100644
@@ -3419,12 +3419,17 @@ struct address_space *swapcache_mapping(struct folio *folio)
 }
 EXPORT_SYMBOL_GPL(swapcache_mapping);
 
-pgoff_t __page_file_index(struct page *page)
+pgoff_t __folio_swap_cache_index(struct folio *folio)
 {
-	swp_entry_t swap = page_swap_entry(page);
-	return swp_offset(swap);
+	return swp_offset(folio->swap);
 }
-EXPORT_SYMBOL_GPL(__page_file_index);

mm/swapfile.c struct address_space *swapcache_mapping(struct folio *folio) · line 3435

Quoted 10 lines
mm/swapfile.c +7 -0
+EXPORT_SYMBOL_GPL(__folio_swap_cache_index);
+
+loff_t __folio_swap_dev_pos(struct folio *folio)
+{
+	return swap_dev_pos(folio->swap);
+}
+EXPORT_SYMBOL_GPL(__folio_swap_dev_pos);
 
 /*
  * add_swap_count_continuation - called when a swap count is duplicated

--
Best Regards,
Huang, Ying

#14 lore raw

Re: [PATCH v2 7/8] mm: drop page_index/page_file_offset and convert swap helpers to use folio

Quoted 113 lines · Ying
Quoted 110 lines · Kairui Song

From: Kairui Song <[email protected]>

There are four helpers for retrieving the page index within address space, or offset within mapped file:

- page_index
- page_file_offset
- folio_index (equivalence of page_index)
- folio_file_pos (equivalence of page_file_offset)

And they are only needed for mixed usage of swap cache & page cache (eg. migration, huge memory split). Else users can just use folio->index or folio_pos.

This commit drops page_index and page_file_offset as we have eliminated all users, and converts folio_index and folio_file_pos (they were simply wrappers of page_index and page_file_offset, and implemented in a not very clean way) to use folio internally.

After this commit, there will be only two helpers for users that may encounter mixed usage of swap cache and page cache:

- folio_index (calls __folio_swap_cache_index for swap cache folio)
- folio_file_pos (calls __folio_swap_dev_pos for swap cache folio)

The index in swap cache and offset in swap device are still basically the same thing, but will be different in following commits.

---
 include/linux/mm.h      | 13 -------------
 include/linux/pagemap.h | 19 +++++++++----------
 mm/swapfile.c           | 13 +++++++++----
 3 files changed, 18 insertions(+), 27 deletions(-)
include/linux/mm.h +0 -13
index 0436b919f1c7..797480e76c9c 100644
@@ -2245,19 +2245,6 @@ static inline void *folio_address(const struct folio *folio)
      return page_address(&folio->page);
 }
 
-extern pgoff_t __page_file_index(struct page *page);
-
-/*
- * Return the pagecache index of the passed page.  Regular pagecache pages
- * use ->index whereas swapcache pages use swp_offset(->private)
- */
-static inline pgoff_t page_index(struct page *page)
-{
-     if (unlikely(PageSwapCache(page)))
-             return __page_file_index(page);
-     return page->index;
-}
-
 /*
  * Return true only if the page has been allocated with
  * ALLOC_NO_WATERMARKS and the low watermark was not
include/linux/pagemap.h +6 -9
index 2df35e65557d..a7d025571ee6 100644
@@ -780,7 +780,7 @@ static inline struct page *grab_cache_page_nowait(struct address_space *mapping,
                      mapping_gfp_mask(mapping));
 }
 
-#define swapcache_index(folio)       __page_file_index(&(folio)->page)
+extern pgoff_t __folio_swap_cache_index(struct folio *folio);
 
 /**
  * folio_index - File index of a folio.
@@ -795,9 +795,9 @@ static inline struct page *grab_cache_page_nowait(struct address_space *mapping,
  */
 static inline pgoff_t folio_index(struct folio *folio)
 {
-        if (unlikely(folio_test_swapcache(folio)))
-                return swapcache_index(folio);
-        return folio->index;
+     if (unlikely(folio_test_swapcache(folio)))
+             return __folio_swap_cache_index(folio);
+     return folio->index;
 }
 
 /**
@@ -920,11 +920,6 @@ static inline loff_t page_offset(struct page *page)
      return ((loff_t)page->index) << PAGE_SHIFT;
 }
 
-static inline loff_t page_file_offset(struct page *page)
-{
-     return ((loff_t)page_index(page)) << PAGE_SHIFT;
-}
-
 /**
  * folio_pos - Returns the byte position of this folio in its file.
  * @folio: The folio.
@@ -934,6 +929,8 @@ static inline loff_t folio_pos(struct folio *folio)
      return page_offset(&folio->page);
 }
 
+extern loff_t __folio_swap_dev_pos(struct folio *folio);
+
 /**
  * folio_file_pos - Returns the byte position of this folio in its file.
  * @folio: The folio.
@@ -943,7 +940,9 @@ static inline loff_t folio_pos(struct folio *folio)
  */
 static inline loff_t folio_file_pos(struct folio *folio)
 {
Quoted 8 lines

include/linux/pagemap.h static inline loff_t folio_pos(struct folio *folio) · line 945

Quoted 4 lines
include/linux/pagemap.h +3 -1
-     return page_file_offset(&folio->page);
+     if (unlikely(folio_test_swapcache(folio)))
+             return __folio_swap_dev_pos(folio);
+     return ((loff_t)folio->index << PAGE_SHIFT);

This still looks confusing for me. The function returns the byte position of the folio in its file. But we returns the swap device position of the folio.

Thanks for the comment.

This doesn't look too confusing to me, __folio_swap_dev_pos -> swap_dev_pos also returns the byte position of the folio in swap device. If we agree swap device is kind of equivalent to the page cache file here, it shouldn't be too hard to understand.

Quoted 3 lines

Tried to search folio_file_pos() usage. The 2 usage in page_io.c is swap specific, we can use swap_dev_pos() directly.

The 2 usage in page_io.c is already converted to use swap_dev_pos directly in this series (patch 6/8).

Quoted 5 lines

There are also other file system users (NFS and AFS) of folio_file_pos(), I don't know why they need to work with swap cache. Cced file system maintainers for help.

Thanks, I'm not very sure if we can just drop folio_file_pos and convert all users to use folio_pos directly. Swap cache mapping shouldn't be exposed to fs, but I'm not confident enough that this is a safe move. It looks OK to do so just by examining NFS code, but let's wait for feedback from FS people.

maintainer
#15 lore raw

Re: [PATCH v2 7/8] mm: drop page_index/page_file_offset and convert swap helpers to use folio

In reply to Huang, Ying
Quoted 12 lines · Ying

include/linux/pagemap.h static inline loff_t folio_pos(struct folio *folio) · line 945

Quoted 6 lines · Kairui Song

static inline loff_t folio_file_pos(struct folio *folio) {

include/linux/pagemap.h +3 -1
-	return page_file_offset(&folio->page);
+	if (unlikely(folio_test_swapcache(folio)))
+		return __folio_swap_dev_pos(folio);
+	return ((loff_t)folio->index << PAGE_SHIFT);

This still looks confusing for me. The function returns the byte position of the folio in its file. But we returns the swap device position of the folio.

Quoted 6 lines

Tried to search folio_file_pos() usage. The 2 usage in page_io.c is swap specific, we can use swap_dev_pos() directly.

There are also other file system users (NFS and AFS) of folio_file_pos(), I don't know why they need to work with swap cache. Cced file system maintainers for help.

Time for a history lesson!

In d56b4ddf7781 (2012) we introduced page_file_index() and page_file_mapping() to support swap-over-NFS. Writes to the swapfile went through ->direct_IO but reads went through ->readpage. So NFS was changed to remove direct references to page->mapping and page->index because those aren't right for anon pages (or shmem pages being swapped out).

In e1209d3a7a67 (2022), we stopped using ->readpage in favour of using ->swap_rw. Now we don't need to use page_file_*(); we get the swap_file and ki_pos directly in the swap_iocb. But there are still relics in NFS that nobody has dared rip out. And there are all the copy-and-pasted filesystems that use page_file_* because they don't know any better.

We should delete page_file_*() and folio_file_*(). They shouldn't be needed any more.

#16 lore raw

Re: [PATCH v2 7/8] mm: drop page_index/page_file_offset and convert swap helpers to use folio

Hi, Matthew,

Quoted 28 lines · Matthew Wilcox
Quoted 12 lines · Ying

include/linux/pagemap.h static inline loff_t folio_pos(struct folio *folio) · line 945

Quoted 6 lines · Kairui Song

static inline loff_t folio_file_pos(struct folio *folio) {

include/linux/pagemap.h +3 -1
-	return page_file_offset(&folio->page);
+	if (unlikely(folio_test_swapcache(folio)))
+		return __folio_swap_dev_pos(folio);
+	return ((loff_t)folio->index << PAGE_SHIFT);

This still looks confusing for me. The function returns the byte position of the folio in its file. But we returns the swap device position of the folio.

Quoted 6 lines

Tried to search folio_file_pos() usage. The 2 usage in page_io.c is swap specific, we can use swap_dev_pos() directly.

There are also other file system users (NFS and AFS) of folio_file_pos(), I don't know why they need to work with swap cache. Cced file system maintainers for help.

Time for a history lesson!

In d56b4ddf7781 (2012) we introduced page_file_index() and page_file_mapping() to support swap-over-NFS. Writes to the swapfile went through ->direct_IO but reads went through ->readpage. So NFS was changed to remove direct references to page->mapping and page->index because those aren't right for anon pages (or shmem pages being swapped out).

Quoted 8 lines

In e1209d3a7a67 (2022), we stopped using ->readpage in favour of using ->swap_rw. Now we don't need to use page_file_*(); we get the swap_file and ki_pos directly in the swap_iocb. But there are still relics in NFS that nobody has dared rip out. And there are all the copy-and-pasted filesystems that use page_file_* because they don't know any better.

We should delete page_file_*() and folio_file_*(). They shouldn't be needed any more.

Thanks a lot for your detailed explanation! Yes, this will simplify the semantics and improve the readability of the corresponding code.

--
Best Regards,
Huang, Ying

#17 lore raw

Re: [PATCH v2 7/8] mm: drop page_index/page_file_offset and convert swap helpers to use folio

In reply to Matthew Wilcox
Quoted 29 lines · Matthew Wilcox
Quoted 12 lines · Ying

include/linux/pagemap.h static inline loff_t folio_pos(struct folio *folio) · line 945

Quoted 6 lines · Kairui Song

static inline loff_t folio_file_pos(struct folio *folio) {

include/linux/pagemap.h +3 -1
-   return page_file_offset(&folio->page);
+   if (unlikely(folio_test_swapcache(folio)))
+           return __folio_swap_dev_pos(folio);
+   return ((loff_t)folio->index << PAGE_SHIFT);

This still looks confusing for me. The function returns the byte position of the folio in its file. But we returns the swap device position of the folio.

Quoted 6 lines

Tried to search folio_file_pos() usage. The 2 usage in page_io.c is swap specific, we can use swap_dev_pos() directly.

There are also other file system users (NFS and AFS) of folio_file_pos(), I don't know why they need to work with swap cache. Cced file system maintainers for help.

Time for a history lesson!

In d56b4ddf7781 (2012) we introduced page_file_index() and page_file_mapping() to support swap-over-NFS. Writes to the swapfile went through ->direct_IO but reads went through ->readpage. So NFS was changed to remove direct references to page->mapping and page->index because those aren't right for anon pages (or shmem pages being swapped out).

Quoted 8 lines

In e1209d3a7a67 (2022), we stopped using ->readpage in favour of using ->swap_rw. Now we don't need to use page_file_*(); we get the swap_file and ki_pos directly in the swap_iocb. But there are still relics in NFS that nobody has dared rip out. And there are all the copy-and-pasted filesystems that use page_file_* because they don't know any better.

We should delete page_file_*() and folio_file_*(). They shouldn't be needed any more.

Thanks for the explanation! I'll update the series, and just delete paeg_file_offset and folio_file_pos with more auditing, to make the code cleaner. Should I add a suggest-by for the removal?

#18 lore raw

PATCH v2 8/8 mm/swap: reduce swap cache search space

In reply to Kairui Song

From: Kairui Song <[email protected]>

Currently we use one swap_address_space for every 64M chunk to reduce lock contention, this is like having a set of smaller swap files inside one big swap file. But when doing swap cache look up or insert, we are still using the offset of the whole large swap file. This is OK for correctness, as the offset (key) is unique.

But Xarray is specially optimized for small indexes, it creates the radix tree levels lazily to be just enough to fit the largest key stored in one Xarray. So we are wasting tree nodes unnecessarily.

For 64M chunk it should only take at most 3 levels to contain everything. But we are using the offset from the whole swap file, so the offset (key) value will be way beyond 64M, and so will the tree level.

Optimize this by using a new helper swap_cache_index to get a swap entry's unique offset in its own 64M swap_address_space.

I see a ~1% performance gain in benchmark and actual workload with high memory pressure.

Test with time memhog 128G inside a 8G memcg using 128G swap (ramdisk with SWP_SYNCHRONOUS_IO dropped, tested 3 times, results are stable. The test result is similar but the improvement is smaller if SWP_SYNCHRONOUS_IO is enabled, as swap out path can never skip swap cache):

Before:
6.07user 250.74system 4:17.26elapsed 99%CPU (0avgtext+0avgdata 8373376maxresident)k 0inputs+0outputs (55major+33555018minor)pagefaults 0swaps

After (1.8% faster):
6.08user 246.09system 4:12.58elapsed 99%CPU (0avgtext+0avgdata 8373248maxresident)k 0inputs+0outputs (54major+33555027minor)pagefaults 0swaps

Similar result with MySQL and sysbench using swap:
Before:
94055.61 qps

After (0.8% faster):
94834.91 qps

Radix tree slab usage is also very slightly lower.

---
 mm/huge_memory.c |  2 +-
 mm/memcontrol.c  |  2 +-
 mm/mincore.c     |  2 +-
 mm/shmem.c       |  2 +-
 mm/swap.h        | 15 +++++++++++++++
 mm/swap_state.c  | 12 ++++++------
 mm/swapfile.c    |  6 +++---
 7 files changed, 28 insertions(+), 13 deletions(-)
mm/huge_memory.c +1 -1
index 9859aa4f7553..1208d60792f0 100644
@@ -2903,7 +2903,7 @@ static void __split_huge_page(struct page *page, struct list_head *list,
 	split_page_memcg(head, order, new_order);
 
 	if (folio_test_anon(folio) && folio_test_swapcache(folio)) {
-		offset = swp_offset(folio->swap);
+		offset = swap_cache_index(folio->swap);
 		swap_cache = swap_address_space(folio->swap);
 		xa_lock(&swap_cache->i_pages);
 	}
mm/memcontrol.c +1 -1
index fabce2b50c69..04d7be7f30dc 100644
@@ -5934,7 +5934,7 @@ static struct page *mc_handle_swap_pte(struct vm_area_struct *vma,
 	 * Because swap_cache_get_folio() updates some statistics counter,
 	 * we call find_get_page() with swapper_space directly.
 	 */
-	page = find_get_page(swap_address_space(ent), swp_offset(ent));
+	page = find_get_page(swap_address_space(ent), swap_cache_index(ent));
 	entry->val = ent.val;
 
 	return page;
mm/mincore.c +1 -1
index dad3622cc963..e31cf1bde614 100644
@@ -139,7 +139,7 @@ static int mincore_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end,
 			} else {
 #ifdef CONFIG_SWAP
 				*vec = mincore_page(swap_address_space(entry),
-						    swp_offset(entry));
+						    swap_cache_index(entry));
 #else
 				WARN_ON(1);
 				*vec = 1;
mm/shmem.c +1 -1
index 0aad0d9a621b..cbe33ab52a73 100644
@@ -1762,7 +1762,7 @@ static int shmem_replace_folio(struct folio **foliop, gfp_t gfp,
 
 	old = *foliop;
 	entry = old->swap;
-	swap_index = swp_offset(entry);
+	swap_index = swap_cache_index(entry);
 	swap_mapping = swap_address_space(entry);
 
 	/*
mm/swap.h +15 -0
index 82023ab93205..93e3e1b58a7f 100644
@@ -27,6 +27,7 @@ void __swap_writepage(struct folio *folio, struct writeback_control *wbc);
 /* One swap address space for each 64M swap space */
 #define SWAP_ADDRESS_SPACE_SHIFT	14
 #define SWAP_ADDRESS_SPACE_PAGES	(1 << SWAP_ADDRESS_SPACE_SHIFT)
+#define SWAP_ADDRESS_SPACE_MASK		(BIT(SWAP_ADDRESS_SPACE_SHIFT) - 1)
 extern struct address_space *swapper_spaces[];
 #define swap_address_space(entry)			    \
 	(&swapper_spaces[swp_type(entry)][swp_offset(entry) \
@@ -40,6 +41,15 @@ static inline loff_t swap_dev_pos(swp_entry_t entry)
 	return ((loff_t)swp_offset(entry)) << PAGE_SHIFT;
 }
 
+/*
+ * Return the swap cache index of the swap entry.
+ */
+static inline pgoff_t swap_cache_index(swp_entry_t entry)
+{
+	BUILD_BUG_ON((SWP_OFFSET_MASK | SWAP_ADDRESS_SPACE_MASK) != SWP_OFFSET_MASK);
+	return swp_offset(entry) & SWAP_ADDRESS_SPACE_MASK;
+}
+
 void show_swap_cache_info(void);
 bool add_to_swap(struct folio *folio);
 void *get_shadow_from_swap_cache(swp_entry_t entry);
@@ -86,6 +96,11 @@ static inline struct address_space *swap_address_space(swp_entry_t entry)
 	return NULL;
 }
 
+static inline pgoff_t swap_cache_index(swp_entry_t entry)
+{
+	return 0;
+}
+
 static inline void show_swap_cache_info(void)
 {
 }
mm/swap_state.c +6 -6
index bfc7e8c58a6d..9dbb54c72770 100644
@@ -72,7 +72,7 @@ void show_swap_cache_info(void)
 void *get_shadow_from_swap_cache(swp_entry_t entry)
 {
 	struct address_space *address_space = swap_address_space(entry);
-	pgoff_t idx = swp_offset(entry);
+	pgoff_t idx = swap_cache_index(entry);
 	struct page *page;
 
 	page = xa_load(&address_space->i_pages, idx);
@@ -89,7 +89,7 @@ int add_to_swap_cache(struct folio *folio, swp_entry_t entry,
 			gfp_t gfp, void **shadowp)
 {
 	struct address_space *address_space = swap_address_space(entry);
-	pgoff_t idx = swp_offset(entry);
+	pgoff_t idx = swap_cache_index(entry);
 	XA_STATE_ORDER(xas, &address_space->i_pages, idx, folio_order(folio));
 	unsigned long i, nr = folio_nr_pages(folio);
 	void *old;
@@ -144,7 +144,7 @@ void __delete_from_swap_cache(struct folio *folio,
 	struct address_space *address_space = swap_address_space(entry);
 	int i;
 	long nr = folio_nr_pages(folio);
-	pgoff_t idx = swp_offset(entry);
+	pgoff_t idx = swap_cache_index(entry);
 	XA_STATE(xas, &address_space->i_pages, idx);
 
 	xas_set_update(&xas, workingset_update_node);
@@ -350,7 +350,7 @@ struct folio *swap_cache_get_folio(swp_entry_t entry,
 {
 	struct folio *folio;
 
-	folio = filemap_get_folio(swap_address_space(entry), swp_offset(entry));
+	folio = filemap_get_folio(swap_address_space(entry), swap_cache_index(entry));
 	if (!IS_ERR(folio)) {
 		bool vma_ra = swap_use_vma_readahead();
 		bool readahead;
@@ -420,7 +420,7 @@ struct folio *filemap_get_incore_folio(struct address_space *mapping,
 	si = get_swap_device(swp);
 	if (!si)
 		return ERR_PTR(-ENOENT);
-	index = swp_offset(swp);
+	index = swap_cache_index(swp);
 	folio = filemap_get_folio(swap_address_space(swp), index);
 	put_swap_device(si);
 	return folio;
@@ -447,7 +447,7 @@ struct folio *__read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask,
 		 * that would confuse statistics.
 		 */
 		folio = filemap_get_folio(swap_address_space(entry),
-						swp_offset(entry));
+					  swap_cache_index(entry));
 		if (!IS_ERR(folio))
 			goto got_folio;
 
mm/swapfile.c +3 -3
index 2387f5e131d7..3d838659f55f 100644
@@ -138,7 +138,7 @@ static int __try_to_reclaim_swap(struct swap_info_struct *si,
 	struct folio *folio;
 	int ret = 0;
 
-	folio = filemap_get_folio(swap_address_space(entry), offset);
+	folio = filemap_get_folio(swap_address_space(entry), swap_cache_index(entry));
 	if (IS_ERR(folio))
 		return 0;
 	/*
@@ -2110,7 +2110,7 @@ static int try_to_unuse(unsigned int type)
 	       (i = find_next_to_unuse(si, i)) != 0) {
 
 		entry = swp_entry(type, i);
-		folio = filemap_get_folio(swap_address_space(entry), i);
+		folio = filemap_get_folio(swap_address_space(entry), swap_cache_index(entry));
 		if (IS_ERR(folio))
 			continue;
 
@@ -3421,7 +3421,7 @@ EXPORT_SYMBOL_GPL(swapcache_mapping);
 
 pgoff_t __folio_swap_cache_index(struct folio *folio)
 {
-	return swp_offset(folio->swap);
+	return swap_cache_index(folio->swap);
 }
 EXPORT_SYMBOL_GPL(__folio_swap_cache_index);
 
Signature · 1 line
2.44.0