Skip to content

mm/swap: clean up and optimize swap cache index

Kairui Song ceph-devel linux-f2fs-devel linux-fsdevel linux-mm linux-nfs linux-nilfs lkml netfs 4 versions – In review

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v6 Latest11 patches 1 replyIn review 10 of 11 applied
  2. v512 patches 2 repliesSuperseded
  3. v412 patches 8 repliesSuperseded
Earlier versions · 1
VersionPostedPatchesRepliesStatus
v312 patches14 repliesSuperseded

Patches in v6 10 of 11 applied

#Subject and filesSizeRepliesApplied
1/11f2fs: drop usage of page_indexfs/f2fs/data.c+1 -10 replies
2/11nilfs2: drop usage of page_indexfs/nilfs2/bmap.c+2 -80 repliesApplied
3/11ceph: drop usage of page_indexfs/ceph/dir.c fs/ceph/inode.c+2 -21 replyApplied
4/11NFS: remove nfs_page_lengthg and usage of page_indexfs/nfs/internal.h+0 -190 repliesApplied
5/11afs: drop usage of folio_file_posfs/afs/dir.c fs/afs/dir_edit.c+5 -50 repliesApplied
6/11netfs: drop usage of folio_file_posfs/netfs/buffered_read.c fs/netfs/buffered_write.c+3 -30 repliesApplied
7/11nfs: drop usage of folio_file_posfs/nfs/file.c fs/nfs/nfstrace.h fs/nfs/write.c+6 -60 repliesApplied
8/11mm/swap: get the swap device offset directlymm/page_io.c mm/swap.h+12 -30 repliesApplied
9/11mm: remove page_file_offset and folio_file_posinclude/linux/pagemap.h+0 -170 repliesApplied
10/11mm: drop page_index and simplify folio_indexinclude/linux/mm.h include/linux/pagemap.h mm/swapfile.c+7 -210 repliesApplied
11/11mm/swap: reduce swap cache search spacemm/huge_memory.c mm/memcontrol.c mm/mincore.c +4 more+31 -150 repliesApplied

Reviews trailers per patch

6 of 11 patches have a Reviewed-by or Acked-by.

PatchRyusuke KonishiHuang, YingXiubo Li
1/11
2/11A-b
3/11R-b
4/11
5/11
6/11
7/11
8/11R-b
9/11R-b
10/11R-b
11/11R-b

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes. Scroll the table sideways for all 3 reviewers.

Read the v6 thread

Cover letter [PATCH v6 00/11] mm/swap: clean up and optimize swap cache index

From: Kairui Song <[email protected]>

Patch 1/11 is not needed if f2fs converted .readahead to use folio, I included it for easier test and review.

Currently we use one swap_address_space for every 64M chunk to reduce lock contention, this is like having a set of smaller files inside a swap device. But when doing swap cache look up or insert, we are still using the offset of the whole large swap device. This is OK for correctness, as the offset (key) is unique.

But Xarray is specially optimized for small indexes, it creates the redix tree levels lazily to be just enough to fit the largest key stored in one Xarray. So we are wasting tree nodes unnecessarily.

For 64M chunk it should only take at most 3 level to contain everything. But if we are using the offset from the whole swap device, the offset (key) value will be way beyond 64M, and so will the tree level.

Optimize this by reduce the swap cache search space into 64M scope.

Test with time memhog 128G inside a 8G memcg using 128G swap (ramdisk with SWP_SYNCHRONOUS_IO dropped, tested 3 times, results are stable. The test result is similar but the improvement is smaller if SWP_SYNCHRONOUS_IO is enabled, as swap out path can never skip swap cache):

Before:
6.07user 250.74system 4:17.26elapsed 99%CPU (0avgtext+0avgdata 8373376maxresident)k 0inputs+0outputs (55major+33555018minor)pagefaults 0swaps

After (+1.8% faster):
6.08user 246.09system 4:12.58elapsed 99%CPU (0avgtext+0avgdata 8373248maxresident)k 0inputs+0outputs (54major+33555027minor)pagefaults 0swaps

Similar result with MySQL and sysbench using swap:
Before:
94055.61 qps

After (+0.8% faster):
94834.91 qps

There is alse a very slight drop of radix tree node slab usage:

Before: 303952K
After:  302224K

For this series:

There are multiple places that expect mixed type of pages (page cache or swap cache), eg. migration, huge memory split; There are four helpers for that:

- page_index
- page_file_offset
- folio_index
- folio_file_pos

To keep the code clean and compatible, this series first cleaned up usage of them.

page_file_offset and folio_file_pos are historical helpes that can be simply dropped after clean up. And page_index can be all converted to folio_index or folio->index.

Then introduce two new helpers swap_cache_index and swap_dev_pos for swap. Replace swp_offset with swap_cache_index when used to retrieve folio from swap cache, and use swap_dev_pos when needed to retrieve the device position of a swap entry. This way, swap_cache_index can return the optimized value with no compatibility issue.

The result is better performance and reduced LOC.

Idealy, in the future, we may want to reduce SWAP_ADDRESS_SPACE_SHIFT from 14 to 12: Default Xarray chunk offset is 6, so we have 3 level trees instead of 2 level trees just for 2 extra bits. But swap cache is based on address_space struct, with 4 times more metadata sparsely distributed in memory it waste more cacheline, the performance gain from this series is almost canceled according to my test. So first, just have a cleaner seperation of offsets and smaller search space.

Patch 1/11 - 10/11: Clean up usage of above helpers. Patch 11/11: Apply the optmization.

V5: https://lore.kernel.org/linux-mm/[email protected]/
Update from V5:
- Rebase to latest mainline.
- Drop cifs patch as is have already converted to use folio.
- Collect Review-by [Huang, Ying].

V4: https://lore.kernel.org/all/[email protected]/
Update from V4:
- Collect Review-by and Acked-by.
- Fix a leftover error in commit message found by [David Hildenbrand].
- A few code clean up for better readability [Huang, Ying]

V3: https://lore.kernel.org/all/[email protected]/
Update from V3:
- Help remove a redundant loop in nilfs2 [Matthew Wilcox]
- Update commit message, use the term swap device instead of swap file to avoid confusion [Huang, Ying]
- Add more details in commit message about folio_file_pos usage in NFS.
- Fix a shadow leak in clear_shadow_from_swap_cache.

V2: https://lore.kernel.org/linux-mm/[email protected]/
Update from V2:
- Clean up usage of page_file_offset and folio_file_pos [Matthew Wilcox]
https://lore.kernel.org/linux-mm/[email protected]/
- Use folio in nilfs_bmap_data_get_key [Ryusuke Konishi]

V1: https://lore.kernel.org/all/[email protected]/
Update from V1:
- Convert more users to use folio directly when possible [Matthew Wilcox]
- Rename swap_file_pos to swap_dev_pos [Huang, Ying]
- Update comments and commit message.
- Adjust headers and add dummy function to fix build error.

This series is part of effort to reduce swap cache overhead, and ultimately remove SWP_SYNCHRONOUS_IO and unify swap cache usage as proposed before:
https://lore.kernel.org/lkml/[email protected]/

Kairui Song (11):
f2fs: drop usage of page_index
nilfs2: drop usage of page_index
ceph: drop usage of page_index
NFS: remove nfs_page_lengthg and usage of page_index
afs: drop usage of folio_file_pos
netfs: drop usage of folio_file_pos
nfs: drop usage of folio_file_pos
mm/swap: get the swap device offset directly
mm: remove page_file_offset and folio_file_pos
mm: drop page_index and simplify folio_index mm/swap: reduce swap cache search space

 fs/afs/dir.c              |  6 +++---
 fs/afs/dir_edit.c         |  4 ++--
 fs/ceph/dir.c             |  2 +-
 fs/ceph/inode.c           |  2 +-
 fs/f2fs/data.c            |  2 +-
 fs/netfs/buffered_read.c  |  4 ++--
 fs/netfs/buffered_write.c |  2 +-
 fs/nfs/file.c             |  2 +-
 fs/nfs/internal.h         | 19 -------------------
 fs/nfs/nfstrace.h         |  4 ++--
 fs/nfs/write.c            |  6 +++---
 fs/nilfs2/bmap.c          | 10 ++--------
 include/linux/mm.h        | 13 -------------
 include/linux/pagemap.h   | 25 ++++---------------------
 mm/huge_memory.c          |  2 +-
 mm/memcontrol.c           |  2 +-
 mm/mincore.c              |  2 +-
 mm/page_io.c              |  6 +++---
 mm/shmem.c                |  2 +-
 mm/swap.h                 | 24 ++++++++++++++++++++++++
 mm/swap_state.c           | 17 +++++++++--------
 mm/swapfile.c             | 11 +++++------
 22 files changed, 68 insertions(+), 99 deletions(-)
Signature · 1 line
2.45.0