mm, swap: introduce swap table as swap cache (phase I)
Subsystems (15)
v4 cover mm, swap: introduce swap table as swap cache (phase I)
From: Kairui Song <[email protected]>
This is the first phase of the bigger series implementing basic infrastructures for the Swap Table idea proposed at the LSF/MM/BPF topic "Integrate swap cache, swap maps with swap allocator" [1]. To give credit where it is due, this is based on Chris Li's idea and a prototype of using cluster size atomic arrays to implement swap cache.
This phase I contains 15 patches, introduces the swap table infrastructure and uses it as the swap cache backend. By doing so, we have up to ~5-20% performance gain in throughput, RPS or build time for benchmark and workload tests. The speed up is due to less contention on the swap cache access and shallower swap cache lookup path. The cluster size is much finer-grained than the 64M address space split, which is removed in this phase I. It also unifies and cleans up the swap code base.
Each swap cluster will dynamically allocate the swap table, which is an atomic array to cover every swap slot in the cluster. It replaces the swap cache backed by XArray. In phase I, the static allocated swap_map still co-exists with the swap table. The memory usage is about the same as the original on average. A few exception test cases show about 1% higher in memory usage. In the following phases of the series, swap_map will merge into the swap table without additional memory allocation. It will result in net memory reduction compared to the original swap cache.
Testing has shown that phase I has a significant performance improvement from 8c/1G ARM machine to 48c96t/128G x86_64 servers in many practical workloads.
The full picture with a summary can be found at [2]. An older bigger series of 28 patches is posted at [3].
vm-scability test:
==================
Test with:
usemem --init-time -O -y -x -n 31 1G (4G memcg, PMEM as swap)
Before: After: System time: 219.12s 158.16s (-27.82%) Sum Throughput: 4767.13 MB/s 6128.59 MB/s (+28.55%) Single process Throughput: 150.21 MB/s 196.52 MB/s (+30.83%) Free latency: 175047.58 us 131411.87 us (-24.92%)
usemem --init-time -O -y -x -n 32 1536M (16G memory, global pressure, PMEM as swap)
Before: After: System time: 356.16s 284.68s (-20.06%) Sum Throughput: 4648.35 MB/s 5453.52 MB/s (+17.32%) Single process Throughput: 141.63 MB/s 168.35 MB/s (+18.86%) Free latency: 499907.71 us 484977.03 us (-2.99%)
This shows an improvement of more than 20% improvement in most readings.
Build kernel test:
==================
The following result matrix is from building kernel with defconfig on tmpfs with ZSWAP / ZRAM, using different memory pressure and setups. Measuring sys and real time in seconds, less is better (user time is almost identical as expected):
-j<NR> / Mem | Sys before / after | Real before / after
Using 16G ZRAM with memcg limit:
6 / 192M | 9686 / 9472 -2.21% | 2130 / 2096 -1.59%
12 / 256M | 6610 / 6451 -2.41% | 827 / 812 -1.81%
24 / 384M | 5938 / 5701 -3.37% | 414 / 405 -2.17%
48 / 768M | 4696 / 4409 -6.11% | 188 / 182 -3.19%
With 64k folio:
24 / 512M | 4222 / 4162 -1.42% | 326 / 321 -1.53%
48 / 1G | 3688 / 3622 -1.79% | 151 / 149 -1.32%
With ZSWAP with 3G memcg (using higher limit due to kmem account):
48 / 3G | 603 / 581 -3.65% | 81 / 80 -1.23%Testing extremely high global memory and schedule pressure: Using ZSWAP with 32G NVMEs in a 48c VM that has 4G memory, no memcg limit, system components take up about 1.5G already, using make -j48 to build defconfig:
Before: sys time: 2069.53s real time: 135.76s After: sys time: 2021.13s (-2.34%) real time: 134.23s (-1.12%)
On another 48c 4G memory VM, using 16G ZRAM as swap, testing make -j48 with same config:
Before: sys time: 1756.96s real time: 111.01s After: sys time: 1715.90s (-2.34%) real time: 109.51s (-1.35%)
All cases are more or less faster, and no regression even under extremely heavy global memory pressure.
Redis / Valkey bench:
=====================
The test machine is a ARM64 VM with 1536M memory 12 cores, Redis is set to use 2500M memory, and ZRAM swap size is set to 5G:
Testing with:
redis-benchmark -r 2000000 -n 2000000 -d 1024 -c 12 -P 32 -t get
no BGSAVE with BGSAVE Before: 487576.06 RPS 280016.02 RPS After: 487541.76 RPS (-0.01%) 300155.32 RPS (+7.19%)
Testing with:
redis-benchmark -r 2500000 -n 2500000 -d 1024 -c 12 -P 32 -t get
no BGSAVE with BGSAVE Before: 466789.59 RPS 281213.92 RPS After: 466402.89 RPS (-0.08%) 298411.84 RPS (+6.12%)
With BGSAVE enabled, most Redis memory will have a swap count > 1 so swap cache is heavily in use. We can see a about 6% performance gain. No BGSAVE is very slightly slower (<0.1%) due to the higher memory pressure of the co-existence of swap_map and swap table. This will be optimzed into a net gain and up to 20% gain in BGSAVE case in the following phases.
HDD swap is also ~40% faster with usemem because we removed an old contention workaround.
---
V4 changes:
- Patch 14: fix potential cluster leak when attemp a sleep allocation of swap table: Just remove the logic that check and return the percpu cluster, it was trying to avoid fragmentation, which wasn't very successfully and may not work at all if there are multiple devices. The fragmentation is not a serious issue and given the chance of hitting that race is extremely low, let's just ignore that [ Chris Meson ].
- Patch 8 & 9: move some changes from Patch 9 to Patch 8 to avoid build error, no code change. [ Baolin Wang ]
- Patch 9: Fix locking section issue, should protect the shmem statistic update with spin lock irq. Also fix an warn on condition.
Link to V3:
- https://lore.kernel.org/linux-mm/[email protected]/
V3 changes:
There is basically no code change, mostly clean up and comment changes.
- Renames a few variable and functions [ David Hildenbrand, Chris Li ]
- Move the folio_matches_swap_entry check under unuse_pte in patch 5 [ David Hildenbrand ].
- Remove two redundant function params for the folio replace helper in patch 10, and add comment about folio setup [ David Hildenbrand ].
- Move the shmem clean up patch after the API rename patch to fix build error, no code change. [ Baolin Wang ]
- Fix build error with !SWAP and !SHMEM config. [ Klara Modin, SeongJae Park ]
- Fix a few typo and blank lines [ David Hildenbrand ]
- Minor code style change [ Chris Li ].
- Fix documentation warning from bot. [ Chris Li ]
Link to V2:
- https://lore.kernel.org/linux-mm/[email protected]/
V2 main updates:
- Added some more non-debug atomic sanity checks, which very slightly slow down the performance by a bit compared to V1 for some tests. But let's be more cautious in phase I, in case there is any hidden bug in swap, since it has a rather complex historical baggage. And phase I is a critical step for follow up swap table series [ Chris Li ]. These sanity checks can be removed very easily later or wrapped with a DEBUG_VM_SWAP as more swap table features land in.
- The code is basically same as V1, except one missing shmem statistic fixed thanks to [ Baolin Wang ]
- Added a lot of kernel docs, and more constify of sturct folio arguments, and better function names, thanks to [ David Hildenbrand ] and [ Chris Li ].
- Removed the arguable readahead change thanks to [ Chris Li ].
- Add an IRQ sanity check, have been testing with this check on for a long time but forgot to include it in V1.
Other updates:
- One VM_WARN_ON_ONCE to VM_WARN_ON_ONCE_FOLIO and minor adjust [ Barry Song ]
- Split out 4 patches from the old patches. The end result is still the same but should be easier to review [ Chris Li, David Hildenbrand ]
- Avoid some trivial blank line or variable name changes and comment fix [ Baoquan He, Chris Li, David Hildenbrand ]
- Added HDD test benchmark result [ Chris Li ]
- Rename and simplified folio_matches_swap_entry a little bit. Link to V1:
- https://lore.kernel.org/linux-mm/[email protected]/
Chris Li (1):
docs/mm: add document for swap table
Kairui Song (14):
mm, swap: use unified helper for swap cache look up
mm, swap: fix swap cache index error when retrying reclaim mm, swap: check page poison flag after locking it
mm, swap: always lock and check the swap cache folio before use mm, swap: rename and move some swap cluster definition and helpers mm, swap: tidy up swap device and cluster info helpers mm, swap: cleanup swap cache API and add kerneldoc mm/shmem, swap: remove redundant error handling for replacing folio mm, swap: wrap swap cache replacement with a helper
mm, swap: use the swap table for the swap cache and switch API mm, swap: mark swap address space ro and add context debug check mm, swap: remove contention workaround for swap cache mm, swap: implement dynamic allocation of swap table
mm, swap: use a single page for swap table when the size fits
Documentation/mm/index.rst | 1 + Documentation/mm/swap-table.rst | 72 +++++ MAINTAINERS | 2 + include/linux/swap.h | 42 --- mm/filemap.c | 2 +- mm/huge_memory.c | 15 +- mm/memory-failure.c | 2 +- mm/memory.c | 27 +- mm/migrate.c | 28 +- mm/mincore.c | 3 +- mm/page_io.c | 12 +- mm/shmem.c | 59 ++--- mm/swap.h | 311 +++++++++++++++++++--- mm/swap_state.c | 450 ++++++++++++++++--------------- mm/swap_table.h | 130 +++++++++ mm/swapfile.c | 451 +++++++++++++++++++++----------- mm/userfaultfd.c | 5 +- mm/vmscan.c | 20 +- mm/zswap.c | 9 +- 19 files changed, 1106 insertions(+), 535 deletions(-) create mode 100644 Documentation/mm/swap-table.rst create mode 100644 mm/swap_table.h
Signature · 1 line
2.51.0
PATCH v4 1/15 docs/mm: add document for swap table
From: Chris Li <[email protected]>
Swap table is the new swap cache.
--- Documentation/mm/index.rst | 1 + Documentation/mm/swap-table.rst | 72 +++++++++++++++++++++++++++++++++ MAINTAINERS | 1 + 3 files changed, 74 insertions(+) create mode 100644 Documentation/mm/swap-table.rst
Documentation/mm/index.rst +1 -0
@@ -57,6 +57,7 @@ documentation, or deleted if it has served its purpose. page_table_check remap_file_pages split_page_table_lock + swap-table transhuge unevictable-lru vmalloced-kernel-stacks
Documentation/mm/swap-table.rst new file +72 -0
72 lines not on this page · raw message
MAINTAINERS +1 -0
@@ -16225,6 +16225,7 @@ R: Barry Song <[email protected]> R: Chris Li <[email protected]> L: [email protected] S: Maintained +F: Documentation/mm/swap-table.rst F: include/linux/swap.h F: include/linux/swapfile.h F: include/linux/swapops.h
Signature · 1 line
2.51.0
Re: [PATCH v4 01/15] docs/mm: add document for swap table
Quoted 55 lines · Kairui Song
From: Chris Li <[email protected]>
Swap table is the new swap cache.
Signed-off-by Chris Li <[email protected]>Signed-off-by Kairui Song <[email protected]>--- Documentation/mm/index.rst | 1 + Documentation/mm/swap-table.rst | 72 +++++++++++++++++++++++++++++++++ MAINTAINERS | 1 + 3 files changed, 74 insertions(+) create mode 100644 Documentation/mm/swap-table.rstDocumentation/mm/index.rst +1
-0@@ -57,6 +57,7 @@ documentation, or deleted if it has served its purpose. page_table_check remap_file_pages split_page_table_lock + swap-table transhuge unevictable-lru vmalloced-kernel-stacksDocumentation/mm/swap-table.rst new file +23
-0@@ -0,0 +1,72 @@ +.. SPDX-License-Identifier: GPL-2.0 + +:Author: Chris Li <[email protected]>, Kairui Song <[email protected]> + +========== +Swap Table +========== + +Swap table implements swap cache as a per-cluster swap cache value array. + +Swap Entry +---------- + +A swap entry contains the information required to serve the anonymous page +fault. + +Swap entry is encoded as two parts: swap type and swap offset. + +The swap type indicates which swap device to use. +The swap offset is the offset of the swap file to read the page data from. + +Swap Cache +----------
Documentation/mm/swap-table.rst line 33
Quoted 10 lines
Documentation/mm/swap-table.rst new file +10
-0+ +Swap cache is a map to look up folios using swap entry as the key. The result +value can have three possible types depending on which stage of this swap entry +was in. + +1. NULL: This swap entry is not used. + +2. folio: A folio has been allocated and bound to this swap entry. This is + the transient state of swap out or swap in. The folio data can be in + the folio or swap file, or both.
This doesn’t look quite right.
the folio’s data must reside within the folio itself? The data might also be in a swap file, or not.
Quoted 15 lines
Documentation/mm/swap-table.rst new file +15
-0+ +3. shadow: The shadow contains the working set information of the swapped + out folio. This is the normal state for a swapped out page. + +Swap Table Internals +-------------------- + +The previous swap cache is implemented by XArray. The XArray is a tree +structure. Each lookup will go through multiple nodes. Can we do better? + +Notice that most of the time when we look up the swap cache, we are either +in a swap in or swap out path. We should already have the swap cluster, +which contains the swap entry. + +If we have a per-cluster array to store swap cache value in the cluster.
Documentation/mm/swap-table.rst line 58
Quoted 10 lines
Documentation/mm/swap-table.rst new file +10
-0+Swap cache lookup within the cluster can be a very simple array lookup. + +We give such a per-cluster swap cache value array a name: the swap table. + +Each swap cluster contains 512 entries, so a swap table stores one cluster +worth of swap cache values, which is exactly one page. This is not +coincidental because the cluster size is determined by the huge page size. +The swap table is holding an array of pointers. The pointer has the same +size as the PTE. The size of the swap table should match to the second +last level of the page table page, exactly one page.
On a 32-bit system, I’m guessing the swap table is 2 KB, which is about half of a page?
Documentation/mm/swap-table.rst line 62
Quoted 4 lines
Documentation/mm/swap-table.rst new file +4
-0+ +With swap table, swap cache lookup can achieve great locality, simpler, +and faster. +
Thanks
Barry
Chris Li
maintainerRe: [PATCH v4 01/15] docs/mm: add document for swap table
Quoted 70 lines · Barry Song
Quoted 55 lines · Kairui Song
From: Chris Li <[email protected]>
Swap table is the new swap cache.
Signed-off-by Chris Li <[email protected]>Signed-off-by Kairui Song <[email protected]>--- Documentation/mm/index.rst | 1 + Documentation/mm/swap-table.rst | 72 +++++++++++++++++++++++++++++++++ MAINTAINERS | 1 + 3 files changed, 74 insertions(+) create mode 100644 Documentation/mm/swap-table.rstDocumentation/mm/index.rst +1
-0@@ -57,6 +57,7 @@ documentation, or deleted if it has served its purpose. page_table_check remap_file_pages split_page_table_lock + swap-table transhuge unevictable-lru vmalloced-kernel-stacksDocumentation/mm/swap-table.rst new file +23
-0@@ -0,0 +1,72 @@ +.. SPDX-License-Identifier: GPL-2.0 + +:Author: Chris Li <[email protected]>, Kairui Song <[email protected]> + +========== +Swap Table +========== + +Swap table implements swap cache as a per-cluster swap cache value array. + +Swap Entry +---------- + +A swap entry contains the information required to serve the anonymous page +fault. + +Swap entry is encoded as two parts: swap type and swap offset. + +The swap type indicates which swap device to use. +The swap offset is the offset of the swap file to read the page data from. + +Swap Cache +----------Documentation/mm/swap-table.rst line 33
Quoted 10 lines
Documentation/mm/swap-table.rst new file +10
-0+ +Swap cache is a map to look up folios using swap entry as the key. The result +value can have three possible types depending on which stage of this swap entry +was in. + +1. NULL: This swap entry is not used. + +2. folio: A folio has been allocated and bound to this swap entry. This is + the transient state of swap out or swap in. The folio data can be in + the folio or swap file, or both.This doesn’t look quite right.
Quoted 1 line
the folio’s data must reside within the folio itself?
For swap out cases that is true. The swap in case you allocate the folio first then read data from swap file to folio. There is a window swap file that has the data and folio does not.
Quoted 1 line
The data might also be in a swap file, or not.
The data only in swap file is covered by "data can be in the folio or swap file", it is an OR relationship.
I think my previous statement still stands correct considering both swap out and swap in. Of course there is always room for improvement to make it more clear. But folio always has the data is not true for swap in. If you have other ways to improve it, please feel free to suggest.
Quoted 2 lines
On a 32-bit system, I’m guessing the swap table is 2 KB, which is about half of a page?
Yes, true. I consider that but decide to leave it out of the document. There are a lot of other implementation details the document does not cover, not just this aspect. This document provides a simple abstracted view (might not cover all the detail cases). One way to address that is add a qualification "on a 64 bit system". What do you say? I don't want to talk about the 32 bit system having half of a page in this document, I consider that too much detail. The 32 bit system is pretty rare nowadays.
Chris
Re: [PATCH v4 01/15] docs/mm: add document for swap table
Quoted 80 lines · Chris Li
Quoted 70 lines · Barry Song
Quoted 55 lines · Kairui Song
From: Chris Li <[email protected]>
Swap table is the new swap cache.
Signed-off-by Chris Li <[email protected]>Signed-off-by Kairui Song <[email protected]>--- Documentation/mm/index.rst | 1 + Documentation/mm/swap-table.rst | 72 +++++++++++++++++++++++++++++++++ MAINTAINERS | 1 + 3 files changed, 74 insertions(+) create mode 100644 Documentation/mm/swap-table.rstDocumentation/mm/index.rst +1
-0@@ -57,6 +57,7 @@ documentation, or deleted if it has served its purpose. page_table_check remap_file_pages split_page_table_lock + swap-table transhuge unevictable-lru vmalloced-kernel-stacksDocumentation/mm/swap-table.rst new file +23
-0@@ -0,0 +1,72 @@ +.. SPDX-License-Identifier: GPL-2.0 + +:Author: Chris Li <[email protected]>, Kairui Song <[email protected]> + +========== +Swap Table +========== + +Swap table implements swap cache as a per-cluster swap cache value array. + +Swap Entry +---------- + +A swap entry contains the information required to serve the anonymous page +fault. + +Swap entry is encoded as two parts: swap type and swap offset. + +The swap type indicates which swap device to use. +The swap offset is the offset of the swap file to read the page data from. + +Swap Cache +----------Documentation/mm/swap-table.rst line 33
Quoted 10 lines
Documentation/mm/swap-table.rst new file +10
-0+ +Swap cache is a map to look up folios using swap entry as the key. The result +value can have three possible types depending on which stage of this swap entry +was in. + +1. NULL: This swap entry is not used. + +2. folio: A folio has been allocated and bound to this swap entry. This is + the transient state of swap out or swap in. The folio data can be in + the folio or swap file, or both.This doesn’t look quite right.
Quoted 1 line
the folio’s data must reside within the folio itself?
For swap out cases that is true. The swap in case you allocate the folio first then read data from swap file to folio. There is a window swap file that has the data and folio does not.
Quoted 1 line
The data might also be in a swap file, or not.
Quoted 8 lines
The data only in swap file is covered by "data can be in the folio or swap file", it is an OR relationship.
I think my previous statement still stands correct considering both swap out and swap in. Of course there is always room for improvement to make it more clear. But folio always has the data is not true for swap in. If you have other ways to improve it, please feel free to suggest.
I assume you’re referring to the swapin case where a folio has been allocated and added to the swap cache, but it’s still being read and hasn’t been updated yet?
I assume it could be something like:
The data may be in the folio or will be placed there later. It could also reside in the swap file.
Alternatively, leave it unchanged.
Quoted 5 lines
Quoted 4 lines
On a 32-bit system, I’m guessing the swap table is 2 KB, which is about half of a page?
Quoted 8 lines
Yes, true. I consider that but decide to leave it out of the document. There are a lot of other implementation details the document does not cover, not just this aspect. This document provides a simple abstracted view (might not cover all the detail cases). One way to address that is add a qualification "on a 64 bit system". What do you say? I don't want to talk about the 32 bit system having half of a page in this document, I consider that too much detail. The 32 bit system is pretty rare nowadays.
I’d prefer that we remove all descriptions about matching PAGE_SIZE, since we would need to double-check every case, like 16 KB or 64 KB pages.
For ARM64 with a 16 KB page size, the last-level index uses 24:14. For ARM64 with a 64 KB page size, it uses 28:16[1]. For them, 512 entries are not one PAGE.
[1] https://developer.arm.com/documentation/101811/0104/Translation-granule
Thanks
Barry
Chris Li
maintainerRe: [PATCH v4 01/15] docs/mm: add document for swap table
Quoted 9 lines · Barry Song
Quoted 5 lines
I think my previous statement still stands correct considering both swap out and swap in. Of course there is always room for improvement to make it more clear. But folio always has the data is not true for swap in. If you have other ways to improve it, please feel free to suggest.
I assume you’re referring to the swapin case where a folio has been allocated and added to the swap cache, but it’s still being read and hasn’t been updated yet?
Right. That is the case swapfile has the data and folio does not.
Quoted 3 lines
I assume it could be something like:
The data may be in the folio or will be placed there later. It could
This is for swap in only, does not describe the swap out case.
Quoted 1 line
also reside in the swap file.
Right and it did not have the same coverage about data that can be both in the folio and swapfile. Sorry about the pedantic. If we want to improve it, we might want to cover the same level of detail.
Quoted 1 line
Alternatively, leave it unchanged.
I think considering the swap out and swap in case, the original is fine. The reader will need to make some effort to map to what it does in the code, at least the description is correct.
Quoted 15 lines
Quoted 6 lines
Quoted 5 lines
On a 32-bit system, I’m guessing the swap table is 2 KB, which is about half of a page?
Quoted 8 lines
Yes, true. I consider that but decide to leave it out of the document. There are a lot of other implementation details the document does not cover, not just this aspect. This document provides a simple abstracted view (might not cover all the detail cases). One way to address that is add a qualification "on a 64 bit system". What do you say? I don't want to talk about the 32 bit system having half of a page in this document, I consider that too much detail. The 32 bit system is pretty rare nowadays.
Quoted 1 line
I’d prefer that we remove all descriptions about matching
PAGE_SIZE,
I am fine with that as well.
Quoted 1 line
since we would need to double-check every case, like 16 KB or 64 KB pages.
The cluster size is determined by the last to second level page table page size. I fail to see why 16KB matters here for the cluster. Are you saying in the 16KB page size case, the custer size is 512/4 = 128 swap entry per cluster?
Quoted 3 lines
For ARM64 with a 16 KB page size, the last-level index uses 24:14. For ARM64 with a 64 KB page size, it uses 28:16[1]. For them, 512 entries are not one PAGE.
Now you got me curioused.
In your above two examples, what is the respected swap cluster swap entry size? In other words, how much entry does one swap cluster hold?
Sorry I am not very familiar with the ARM page tables.
Chris
Re: [PATCH v4 01/15] docs/mm: add document for swap table
Quoted 21 lines · Chris Li
Quoted 9 lines · Barry Song
Quoted 5 lines
I think my previous statement still stands correct considering both swap out and swap in. Of course there is always room for improvement to make it more clear. But folio always has the data is not true for swap in. If you have other ways to improve it, please feel free to suggest.
I assume you’re referring to the swapin case where a folio has been allocated and added to the swap cache, but it’s still being read and hasn’t been updated yet?
Right. That is the case swapfile has the data and folio does not.
Quoted 3 lines
I assume it could be something like:
The data may be in the folio or will be placed there later. It couldThis is for swap in only, does not describe the swap out case.
Quoted 1 line
also reside in the swap file.
Quoted 8 lines
Right and it did not have the same coverage about data that can be both in the folio and swapfile. Sorry about the pedantic. If we want to improve it, we might want to cover the same level of detail.
Quoted 1 line
Alternatively, leave it unchanged.
I think considering the swap out and swap in case, the original is fine. The reader will need to make some effort to map to what it does in the code, at least the description is correct.
Ok.
Quoted 31 lines
Quoted 16 lines
Quoted 7 lines
Quoted 6 lines
On a 32-bit system, I’m guessing the swap table is 2 KB, which is about half of a page?
Quoted 8 lines
Yes, true. I consider that but decide to leave it out of the document. There are a lot of other implementation details the document does not cover, not just this aspect. This document provides a simple abstracted view (might not cover all the detail cases). One way to address that is add a qualification "on a 64 bit system". What do you say? I don't want to talk about the 32 bit system having half of a page in this document, I consider that too much detail. The 32 bit system is pretty rare nowadays.
Quoted 1 line
I’d prefer that we remove all descriptions about matching
PAGE_SIZE,I am fine with that as well.
Quoted 1 line
since we would need to double-check every case, like 16 KB or 64 KB pages.
The cluster size is determined by the last to second level page table page size. I fail to see why 16KB matters here for the cluster. Are you saying in the 16KB page size case, the custer size is 512/4 = 128 swap entry per cluster?
Quoted 3 lines
For ARM64 with a 16 KB page size, the last-level index uses 24:14. For ARM64 with a 64 KB page size, it uses 28:16[1]. For them, 512 entries are not one PAGE.
Quoted 6 lines
Now you got me curioused.
In your above two examples, what is the respected swap cluster swap entry size? In other words, how much entry does one swap cluster hold?
Sorry I am not very familiar with the ARM page tables.
Oh, my mistake—I recalculated:
For a 16 KB page size, SWAPCLUSTER_SIZE will be 2^11 = 2048, so the swap table is 2048 * 8 = 16 KB.
For a 64 KB page size, SWAPCLUSTER_SIZE will be 2^13 = 8192, so the swap table is 8192 * 8 = 64 KB.
This approach still seems to work, so the 32-bit system appears to be the only exception. However, I’m not entirely sure that your description of “the second last level” is correct. I believe it refers to the PTE, which corresponds to the last level, not the second-to-last. In other words, how do you define the second-to-last level page table?
Thanks
Barry
Chris Li
maintainerRe: [PATCH v4 01/15] docs/mm: add document for swap table
Quoted 9 lines · Barry Song
Quoted 4 lines
In your above two examples, what is the respected swap cluster swap entry size? In other words, how much entry does one swap cluster hold?
Sorry I am not very familiar with the ARM page tables.
Oh, my mistake—I recalculated:
For a 16 KB page size,
SWAPCLUSTER_SIZEwill be 2^11 = 2048, so the swap table is 2048 * 8 = 16 KB.
So my original description is correct in the sense that, the 16KB page size, swap table is 16KB, this is not coincidental.
Quoted 2 lines
For a 64 KB page size,
SWAPCLUSTER_SIZEwill be 2^13 = 8192, so the swap table is 8192 * 8 = 64 KB.
Same here. For 64 KB page size, the swap table is 64KB as you just told me. I am just trying to give a bit of a glimpse of where I get the intuition for swap tables.
Quoted 6 lines
This approach still seems to work, so the 32-bit system appears to be the only exception. However, I’m not entirely sure that your description of “the second last level” is correct. I believe it refers to the PTE, which corresponds to the last level, not the second-to-last. In other words, how do you define the second-to-last level page table?
The second-to-last level page table page holds the PMD. The last level page table holds PTE.
Cluster size is HPAGE_PMD_NR = 1<<HPAGE_PMD_ORDER
I was thinking of a PMD entry but the actual page table page it points to is the last level.
That is a good catch. Let me see how to fix it.
What I am trying to say is that, swap table size should match to the PTE page table page size which determines the cluster size. An alternative to understanding the swap table is that swap table is a shadow PTE page table containing the shadow PTE matching to the page that gets swapped out to the swapfile. It is arranged in the swapfile swap offset order. The intuition is simple once you find the right angle to view it. However it might be a mouthful to explain.
I am fine with removing it, on the other hand it removes the only bit of secret sauce which I try to give the reader a glimpse of my intuition of the swap table.
Thanks for catching that.
Chris
Re: [PATCH v4 01/15] docs/mm: add document for swap table
Quoted 21 lines
Quoted 5 lines
This approach still seems to work, so the 32-bit system appears to be the only exception. However, I’m not entirely sure that your description of “the second last level” is correct. I believe it refers to the PTE, which corresponds to the last level, not the second-to-last. In other words, how do you define the second-to-last level page table?
The second-to-last level page table page holds the PMD. The last level page table holds PTE.
Cluster size isHPAGE_PMD_NR= 1<<HPAGE_PMD_ORDER
I was thinking of a PMD entry but the actual page table page it points to is the last level.
That is a good catch. Let me see how to fix it.What I am trying to say is that, swap table size should match to the PTE page table page size which determines the cluster size. An alternative to understanding the swap table is that swap table is a shadow PTE page table containing the shadow PTE matching to the page that gets swapped out to the swapfile. It is arranged in the swapfile swap offset order. The intuition is simple once you find the right angle to view it. However it might be a mouthful to explain.
Quoted 3 lines
I am fine with removing it, on the other hand it removes the only bit of secret sauce which I try to give the reader a glimpse of my intuition of the swap table.
Perhaps you could describe the swap table as similar to a PTE page table representing the swap cache mapping.
That is correct for most 32-bit and 64-bit systems,
but not for every machine.
The only exception is a 32-bit system with a 64-bit physical address (Large Physical Address Extension, LPAE), which uses a 4 KB PTE table but a 2 KB swap table because the pointer is 32 bit while each page table entry is 64 bit.
Maybe we can simply say that the number of entries in the swap table is the same as in a PTE page table?
Quoted 4 lines
Thanks for catching that.
Chris
Thanks
Barry
Re: [PATCH v4 01/15] docs/mm: add document for swap table
Quoted 5 lines
Perhaps you could describe the swap table as similar to a PTE page table representing the swap cache mapping.
That is correct for most 32-bit and 64-bit systems,
but not for every machine.
Quoted 7 lines
The only exception is a 32-bit system with a 64-bit physical address (Large Physical Address Extension, LPAE), which uses a 4 KB PTE table but a 2 KB swap table because the pointer is 32 bit while each page table entry is 64 bit.
Maybe we can simply say that the number of entries in the swap table is the same as in a PTE page table?
BTW, as Kairui mentioned, you plan to store the PFN instead of a pointer in phase 2.
I wonder whether we need to switch to atomic64_t on systems where the physical address is 64 bit but the virtual address is 32 bit :-)
Thanks
Barry
Chris Li
maintainerRe: [PATCH v4 01/15] docs/mm: add document for swap table
Quoted 11 lines · Barry Song
Quoted 11 lines
Perhaps you could describe the swap table as similar to a PTE page table representing the swap cache mapping.
That is correct for most 32-bit and 64-bit systems,
but not for every machine.The only exception is a 32-bit system with a 64-bit physical address (Large Physical Address Extension, LPAE), which uses a 4 KB PTE table but a 2 KB swap table because the pointer is 32 bit while each page table entry is 64 bit.
Quoted 5 lines
Quoted 2 lines
Maybe we can simply say that the number of entries in the swap table is the same as in a PTE page table?
BTW, as Kairui mentioned, you plan to store the PFN instead of a pointer in phase 2.
Yes, let's update the document then and only then. Otherwise the document will be mismatching the code and confuse the reader.
Quoted 3 lines
I wonder whether we need to switch to
atomic64_ton systems where the physical address is 64 bit but the virtual address is 32 bit :-)
It is possible we need 64 bit for the swap cache anyway for other reasons when we get into the later phases. Again, let's deal with it later.
Chris
Chris Li
maintainerRe: [PATCH v4 01/15] docs/mm: add document for swap table
In reply to Barry SongQuoted 22 lines · Barry Song
Quoted 22 lines
Quoted 6 lines
This approach still seems to work, so the 32-bit system appears to be the only exception. However, I’m not entirely sure that your description of “the second last level” is correct. I believe it refers to the PTE, which corresponds to the last level, not the second-to-last. In other words, how do you define the second-to-last level page table?
The second-to-last level page table page holds the PMD. The last level page table holds PTE.
Cluster size isHPAGE_PMD_NR= 1<<HPAGE_PMD_ORDER
I was thinking of a PMD entry but the actual page table page it points to is the last level.
That is a good catch. Let me see how to fix it.What I am trying to say is that, swap table size should match to the PTE page table page size which determines the cluster size. An alternative to understanding the swap table is that swap table is a shadow PTE page table containing the shadow PTE matching to the page that gets swapped out to the swapfile. It is arranged in the swapfile swap offset order. The intuition is simple once you find the right angle to view it. However it might be a mouthful to explain.
Quoted 6 lines
Quoted 3 lines
I am fine with removing it, on the other hand it removes the only bit of secret sauce which I try to give the reader a glimpse of my intuition of the swap table.
Perhaps you could describe the swap table as similar to a PTE page table representing the swap cache mapping.
Hard to qualify what is "similar", in what way it is similar. Different readers will have different interpretations of what similar means to them.
Quoted 2 lines
That is correct for most 32-bit and 64-bit systems, but not for every machine.
I think I will leave it as for most 64 bit systems, the swap table size is exactly one page table page size and that is not coincidental.
Quoted 4 lines
The only exception is a 32-bit system with a 64-bit physical address (Large Physical Address Extension, LPAE), which uses a 4 KB PTE table but a 2 KB swap table because the pointer is 32 bit while each page table entry is 64 bit.
I feel that is a very corner case. I will leave it out of the document. I want to present a simplified abstracted view. There is always more detail to distract the simple abstracted view. That is why we have physics.
Quoted 2 lines
Maybe we can simply say that the number of entries in the swap table is the same as in a PTE page table?
Yes, that is what I want to say, for most modern 64 bit systems.
Chris
Chris Li
maintainerRe: [PATCH v4 01/15] docs/mm: add document for swap table
Hi Barry,
How about this:
A swap table stores one cluster worth of swap cache values, which is exactly one page table page on most morden 64 bit systems. This is not coincidental because the cluster size is determined by the huge page size. The swap table is holding an array of pointers, which have the same size as the PTE. The size of the swap table should match the page table page.
If that sounds OK, I will send an incremental patch to Andrew.
Chris
Quoted 57 lines · Chris Li
Quoted 22 lines · Barry Song
Quoted 22 lines
Quoted 6 lines
This approach still seems to work, so the 32-bit system appears to be the only exception. However, I’m not entirely sure that your description of “the second last level” is correct. I believe it refers to the PTE, which corresponds to the last level, not the second-to-last. In other words, how do you define the second-to-last level page table?
The second-to-last level page table page holds the PMD. The last level page table holds PTE.
Cluster size isHPAGE_PMD_NR= 1<<HPAGE_PMD_ORDER
I was thinking of a PMD entry but the actual page table page it points to is the last level.
That is a good catch. Let me see how to fix it.What I am trying to say is that, swap table size should match to the PTE page table page size which determines the cluster size. An alternative to understanding the swap table is that swap table is a shadow PTE page table containing the shadow PTE matching to the page that gets swapped out to the swapfile. It is arranged in the swapfile swap offset order. The intuition is simple once you find the right angle to view it. However it might be a mouthful to explain.
Quoted 6 lines
Quoted 3 lines
I am fine with removing it, on the other hand it removes the only bit of secret sauce which I try to give the reader a glimpse of my intuition of the swap table.
Perhaps you could describe the swap table as similar to a PTE page table representing the swap cache mapping.
Hard to qualify what is "similar", in what way it is similar. Different readers will have different interpretations of what similar means to them.
Quoted 2 lines
That is correct for most 32-bit and 64-bit systems, but not for every machine.
I think I will leave it as for most 64 bit systems, the swap table size is exactly one page table page size and that is not coincidental.
Quoted 4 lines
The only exception is a 32-bit system with a 64-bit physical address (Large Physical Address Extension, LPAE), which uses a 4 KB PTE table but a 2 KB swap table because the pointer is 32 bit while each page table entry is 64 bit.
I feel that is a very corner case. I will leave it out of the document. I want to present a simplified abstracted view. There is always more detail to distract the simple abstracted view. That is why we have physics.
Quoted 2 lines
Maybe we can simply say that the number of entries in the swap table is the same as in a PTE page table?
Yes, that is what I want to say, for most modern 64 bit systems.
Chris
Re: [PATCH v4 01/15] docs/mm: add document for swap table
Quoted 8 lines · Chris Li
Hi Barry,
How about this:
A swap table stores one cluster worth of swap cache values, which is exactly one page table page on most morden 64 bit systems. This is not coincidental because the cluster size is determined by the huge page size.
I’d phrase it as “PMD huge page,” since we also have “PUD huge page.”
Quoted 4 lines
The swap table is holding an array of pointers, which have the same size as the PTE. The size of the swap table should match the page table page.
I’m not entirely sure what you mean by “page table page.”
My understanding is that you’re saying:
The swap table contains an array of pointers, each the same size as a PTE, so its total size typically matches a PTE page table—one page on modern 64-bit systems.
Quoted 3 lines
If that sounds OK, I will send an incremental patch to Andrew.
Chris
Thanks
Barry
Chris Li
maintainerRe: [PATCH v4 01/15] docs/mm: add document for swap table
Quoted 5 lines · Barry Song
Quoted 3 lines · Chris Li
Hi Barry,
Quoted 7 lines
Quoted 5 lines
How about this:
A swap table stores one cluster worth of swap cache values, which is exactly one page table page on most morden 64 bit systems. This is not coincidental because the cluster size is determined by the huge page size.
I’d phrase it as “PMD huge page,” since we also have “PUD huge page.”
Good point. Will do.
Quoted 7 lines
Quoted 5 lines
The swap table is holding an array of pointers, which have the same size as the PTE. The size of the swap table should match the page table page.
I’m not entirely sure what you mean by “page table page.”
The page that gets pointed by the page table, or the page that holds the PTE.
Quoted 4 lines
My understanding is that you’re saying:
The swap table contains an array of pointers, each the same size as a PTE, so its total size typically matches a PTE page table—one page on modern 64-bit systems.
That sounds good. Thanks for the suggestion.
I take your suggestion with some small modifications, mostly to clarify the total size is the total size of one cluster of swap tables. The total size of all swap tables in a swap file is much bigger.
How about this:
A swap table is an array of pointers. Each pointer is the same size as a PTE. The size of a swap table for one swap cluster typically matches a PTE page table,
which is one page on modern 64-bit systems.
Chris
Re: [PATCH v4 01/15] docs/mm: add document for swap table
Quoted 15 lines
Quoted 1 line
I’m not entirely sure what you mean by “page table page.”
The page that gets pointed by the page table, or the page that holds the PTE.
Quoted 4 lines
My understanding is that you’re saying:
The swap table contains an array of pointers, each the same size as a PTE, so its total size typically matches a PTE page table—one page on modern 64-bit systems.That sounds good. Thanks for the suggestion.
I take your suggestion with some small modifications, mostly to clarify the total size is the total size of one cluster of swap tables. The total size of all swap tables in a swap file is much bigger.
Quoted 6 lines
How about this:
A swap table is an array of pointers. Each pointer is the same size as a PTE. The size of a swap table for one swap cluster typically matches a PTE page table,
which is one page on modern 64-bit systems.
Acked.
Quoted 2 lines
Chris
Thanks
Barry
Chris Li
maintainerRe: [PATCH v4 01/15] docs/mm: add document for swap table
Hi Andrew,
Can you please apply this incremental fix up commit on the document patch of the swap table series?
Just folding it into the original patch is fine.
Here is the change log:
- Move the swap table document to the mm main section. [SeongJae Park]
- Rewrite the swap table size sentence for easier to understand. [ Barry]
Thanks
Chris
Quoted 23 lines · Barry Song
Quoted 16 lines
Quoted 2 lines
I’m not entirely sure what you mean by “page table page.”
The page that gets pointed by the page table, or the page that holds the PTE.
Quoted 4 lines
My understanding is that you’re saying:
The swap table contains an array of pointers, each the same size as a PTE, so its total size typically matches a PTE page table—one page on modern 64-bit systems.That sounds good. Thanks for the suggestion.
I take your suggestion with some small modifications, mostly to clarify the total size is the total size of one cluster of swap tables. The total size of all swap tables in a swap file is much bigger.Quoted 6 lines
How about this:
A swap table is an array of pointers. Each pointer is the same size as a PTE. The size of a swap table for one swap cluster typically matches a PTE page table,
which is one page on modern 64-bit systems.
Quoted 7 lines
Acked.
Quoted 2 lines
Chris
Thanks
Barry
From b03a9b53dfcb1558feb01188f657340bc3414a22 Mon Sep 17 00:00:00 2001
From: Chris Li <[email protected]>
Date: Fri, 19 Sep 2025 18:12:04 -0700
Subject: [PATCH] SQUASH: docs: fix up for the swap table document
Make swap table into the main section.
Adjust wording on the swap table size vs PTE page table size.
--- Documentation/mm/index.rst | 2 +- Documentation/mm/swap-table.rst | 9 +++------ 2 files changed, 4 insertions(+), 7 deletions(-)
Documentation/mm/index.rst +1 -1
@@ -20,6 +20,7 @@ see the :doc:`admin guide <../admin-guide/mm/index>`. highmem page_reclaim swap + swap-table page_cache shmfs oom
@@ -57,7 +58,6 @@ documentation, or deleted if it has served its purpose. page_table_check remap_file_pages split_page_table_lock - swap-table transhuge unevictable-lru vmalloced-kernel-stacks
Documentation/mm/swap-table.rst +3 -6
@@ -50,12 +50,9 @@ Swap cache lookup within the cluster can be a very simple array lookup. We give such a per-cluster swap cache value array a name: the swap table. -Each swap cluster contains 512 entries, so a swap table stores one cluster -worth of swap cache values, which is exactly one page. This is not -coincidental because the cluster size is determined by the huge page size. -The swap table is holding an array of pointers. The pointer has the same -size as the PTE. The size of the swap table should match to the second -last level of the page table page, exactly one page. +A swap table is an array of pointers. Each pointer is the same size as a +PTE. The size of a swap table for one swap cluster typically matches a PTE +page table, which is one page on modern 64-bit systems. With swap table, swap cache lookup can achieve great locality, simpler, and faster.
Signature · 1 line
2.51.0.470.ga7dc726c21-goog
Re: [PATCH v4 01/15] docs/mm: add document for swap table
In reply to Kairui SongQuoted 3 lines · On Wed, 17 Sep 2025 00:00:46 +0800 Kairui Song <[email protected]> wrote:
From: Chris Li <[email protected]>
Swap table is the new swap cache.
Thank you very much for doing this great work!
Quoted 15 lines
Signed-off-by Chris Li <[email protected]>Signed-off-by Kairui Song <[email protected]>--- Documentation/mm/index.rst | 1 + Documentation/mm/swap-table.rst | 72 +++++++++++++++++++++++++++++++++ MAINTAINERS | 1 + 3 files changed, 74 insertions(+) create mode 100644 Documentation/mm/swap-table.rstDocumentation/mm/index.rst +0
-0
Documentation/mm/index.rst documentation, or deleted if it has served its purpose. · line 63
Quoted 7 lines
Documentation/mm/index.rst +1
-0@@ -57,6 +57,7 @@ documentation, or deleted if it has served its purpose. page_table_check remap_file_pages split_page_table_lock + swap-table transhuge unevictable-lru vmalloced-kernel-stacks
The above diff is adding this great document on 'Unsorted Documentation' section. Could we add the document on the main section? I think swap.rst on the main section could be a good place, and wondering what others think.
Thanks,
SJ
[...]
Chris Li
maintainerRe: [PATCH v4 01/15] docs/mm: add document for swap table
Quoted 8 lines · SeongJae Park
Quoted 3 lines · On Wed, 17 Sep 2025 00:00:46 +0800 Kairui Song <[email protected]> wrote:
From: Chris Li <[email protected]>
Swap table is the new swap cache.
Thank you very much for doing this great work!
I only did the prototype. Kairui did the heavy lifting.
Quoted 9 lines
Quoted 9 lines
Signed-off-by Chris Li <[email protected]>Signed-off-by Kairui Song <[email protected]>--- Documentation/mm/index.rst | 1 + Documentation/mm/swap-table.rst | 72 +++++++++++++++++++++++++++++++++ MAINTAINERS | 1 + 3 files changed, 74 insertions(+) create mode 100644 Documentation/mm/swap-table.rst
Quoted 16 lines
Documentation/mm/index.rst documentation, or deleted if it has served its purpose. · line 63
Quoted 12 lines
Documentation/mm/index.rst +1
-0@@ -57,6 +57,7 @@ documentation, or deleted if it has served its purpose. page_table_check remap_file_pages split_page_table_lock + swap-table transhuge unevictable-lru vmalloced-kernel-stacksThe above diff is adding this great document on 'Unsorted Documentation' section. Could we add the document on the main section? I think
swap.rston the main section could be a good place, and wondering what others think.
That is a good idea. Will do together with another minor fix reported by Barry. I will move it after "swap.rst".
Thanks for the suggestion.
Chris
PATCH v4 2/15 mm, swap: use unified helper for swap cache look up
In reply to Kairui SongFrom: Kairui Song <[email protected]>
The swap cache lookup helper swap_cache_get_folio currently does readahead updates as well, so callers that are not doing swapin from any VMA or mapping are forced to reuse filemap helpers instead, and have to access the swap cache space directly.
So decouple readahead update with swap cache lookup. Move the readahead update part into a standalone helper. Let the caller call the readahead update helper if they do readahead. And convert all swap cache lookups to use swap_cache_get_folio.
After this commit, there are only three special cases for accessing swap cache space now: huge memory splitting, migration, and shmem replacing, because they need to lock the XArray. The following commits will wrap their accesses to the swap cache too, with special helpers.
And worth noting, currently dropbehind is not supported for anon folio, and we will never see a dropbehind folio in swap cache. The unified helper can be updated later to handle that.
While at it, add proper kernedoc for touched helpers.
No functional change.
--- mm/memory.c | 6 ++- mm/mincore.c | 3 +- mm/shmem.c | 4 +- mm/swap.h | 13 ++++-- mm/swap_state.c | 109 +++++++++++++++++++++++++---------------------- mm/swapfile.c | 11 +++-- mm/userfaultfd.c | 5 +-- 7 files changed, 81 insertions(+), 70 deletions(-)
mm/memory.c +4 -2
@@ -4660,9 +4660,11 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)if (unlikely (!si)) goto out; - folio =swap_cache_get_folio (entry, vma, vmf->address); -if (folio) + folio =swap_cache_get_folio (entry); +if (folio) { +swap_update_readahead (folio, vma, vmf->address); page =folio_file_page (folio,swp_offset (entry)); + } swapcache = folio;if (!folio) {
mm/mincore.c +1 -2
@@ -76,8 +76,7 @@ static unsigned char mincore_swap(swp_entry_t entry, bool shmem)if (!si)return 0 ; } - folio =filemap_get_entry (swap_address_space (entry), -swap_cache_index (entry)); + folio =swap_cache_get_folio (entry);if (shmem)put_swap_device (si);/* The swap cache space contains either folio, shadow or NULL */
mm/shmem.c +3 -1
@@ -2317,7 +2317,7 @@ static int shmem_swapin_folio(struct inode *inode, pgoff_t index, }/* Look it up and read it in.. */ - folio =swap_cache_get_folio (swap,NULL ,0 ); + folio =swap_cache_get_folio (swap);if (!folio) {if (data_race (si->flags &SWP_SYNCHRONOUS_IO )) {/* Direct swapin skipping swap cache & readahead */
@@ -2342,6 +2342,8 @@ static int shmem_swapin_folio(struct inode *inode, pgoff_t index,count_vm_event (PGMAJFAULT );count_memcg_event_mm (fault_mm,PGMAJFAULT ); } + }else { +swap_update_readahead (folio,NULL ,0 ); }if (order >folio_order (folio)) {
mm/swap.h +9 -4
@@ -62,8 +62,7 @@ void delete_from_swap_cache(struct folio *folio);void clear_shadow_from_swap_cache (int type,unsigned long begin,unsigned long end);void swapcache_clear (struct swap_info_struct *si,swp_entry_t entry,int nr); -struct folio *swap_cache_get_folio (swp_entry_t entry, -struct vm_area_struct *vma,unsigned long addr); +struct folio *swap_cache_get_folio (swp_entry_t entry);struct folio *read_swap_cache_async (swp_entry_t entry,gfp_t gfp_mask,struct vm_area_struct *vma,unsigned long addr,struct swap_iocb **plug);
@@ -74,6 +73,8 @@ struct folio *swap_cluster_readahead(swp_entry_t entry, gfp_t flag, struct mempolicy *mpol, pgoff_t ilx);struct folio *swapin_readahead (swp_entry_t entry,gfp_t flag,struct vm_fault *vmf); +void swap_update_readahead (struct folio *folio,struct vm_area_struct *vma, +unsigned long addr);static inline unsigned int folio_swap_flags (struct folio *folio) {
@@ -159,6 +160,11 @@ static inline struct folio *swapin_readahead(swp_entry_t swp, gfp_t gfp_mask,return NULL ; } +static inline void swap_update_readahead (struct folio *folio, +struct vm_area_struct *vma,unsigned long addr) +{ +} +static inline int swap_writeout (struct folio *folio,struct swap_iocb **swap_plug) {
@@ -169,8 +175,7 @@ static inline void swapcache_clear(struct swap_info_struct *si, swp_entry_t entr { } -static inline struct folio *swap_cache_get_folio (swp_entry_t entry, -struct vm_area_struct *vma,unsigned long addr) +static inline struct folio *swap_cache_get_folio (swp_entry_t entry) {return NULL ; }
mm/swap_state.c +57 -52
132 lines not on this page · raw message
mm/swapfile.c +5 -6
@@ -213,15 +213,14 @@ static int __try_to_reclaim_swap(struct swap_info_struct *si, unsigned long offset, unsigned long flags) {swp_entry_t entry =swp_entry (si->type, offset); -struct address_space *address_space =swap_address_space (entry);struct swap_cluster_info *ci;struct folio *folio;int ret, nr_pages;bool need_reclaim; again: - folio =filemap_get_folio (address_space,swap_cache_index (entry)); -if (IS_ERR (folio)) + folio =swap_cache_get_folio (entry); +if (!folio)return 0 ; nr_pages =folio_nr_pages (folio);
@@ -2131,7 +2130,7 @@ static int unuse_pte_range(struct vm_area_struct *vma, pmd_t *pmd,pte_unmap (pte); pte =NULL ; - folio =swap_cache_get_folio (entry, vma, addr); + folio =swap_cache_get_folio (entry);if (!folio) {struct vm_fault vmf = { .vma = vma,
@@ -2357,8 +2356,8 @@ static int try_to_unuse(unsigned int type) (i = find_next_to_unuse(si, i)) != 0) { entry =swp_entry (type, i); - folio =filemap_get_folio (swap_address_space (entry),swap_cache_index (entry)); -if (IS_ERR (folio)) + folio =swap_cache_get_folio (entry); +if (!folio)continue ;/*
mm/userfaultfd.c +2 -3
@@ -1489,9 +1489,8 @@ static long move_pages_ptes(struct mm_struct *mm, pmd_t *dst_pmd, pmd_t *src_pmd* separately to allow proper handling. */ if (!src_folio) - folio =filemap_get_folio (swap_address_space (entry), -swap_cache_index (entry)); -if (!IS_ERR_OR_NULL (folio)) { + folio =swap_cache_get_folio (entry); +if (folio) {if (folio_test_large (folio)) { ret = -EBUSY ;folio_put (folio);
Signature · 1 line
2.51.0
PATCH v4 3/15 mm, swap: fix swap cache index error when retrying reclaim
In reply to Kairui SongFrom: Kairui Song <[email protected]>
The allocator will reclaim cached slots while scanning. Currently, it will try again if reclaim found a folio that is already removed from the swap cache due to a race. But the following lookup will be using the wrong index. It won't cause any OOB issue since the swap cache index is truncated upon lookup, but it may lead to reclaiming of an irrelevant folio.
This should not cause a measurable issue, but we should fix it.
--- mm/swapfile.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
mm/swapfile.c +4 -4
@@ -212,7 +212,7 @@ static bool swap_is_last_map(struct swap_info_struct *si,static int __try_to_reclaim_swap (struct swap_info_struct *si,unsigned long offset,unsigned long flags) { -swp_entry_t entry =swp_entry (si->type, offset); +const swp_entry_t entry =swp_entry (si->type, offset);struct swap_cluster_info *ci;struct folio *folio;int ret, nr_pages;
@@ -240,13 +240,13 @@ static int __try_to_reclaim_swap(struct swap_info_struct *si,* Offset could point to the middle of a large folio, or folio * may no longer point to the expected offset before it's locked. */ - entry = folio->swap; -if (offset <swp_offset (entry) || offset >=swp_offset (entry) + nr_pages) { +if (offset <swp_offset (folio->swap) || + offset >=swp_offset (folio->swap) + nr_pages) {folio_unlock (folio);folio_put (folio); goto again; } - offset =swp_offset (entry); + offset =swp_offset (folio->swap); need_reclaim = ((flags & TTRS_ANYWAY) || ((flags & TTRS_UNMAPPED) && !folio_mapped(folio)) ||
Signature · 1 line
2.51.0
PATCH v4 4/15 mm, swap: check page poison flag after locking it
In reply to Kairui SongFrom: Kairui Song <[email protected]>
Instead of checking the poison flag only in the fast swap cache lookup path, always check the poison flags after locking a swap cache folio.
There are two reasons to do so.
The folio is unstable and could be removed from the swap cache anytime, so it's totally possible that the folio is no longer the backing folio of a swap entry, and could be an irrelevant poisoned folio. We might mistakenly kill a faulting process.
And it's totally possible or even common for the slow swap in path (swapin_readahead) to bring in a cached folio. The cache folio could be poisoned, too. Only checking the poison flag in the fast path will miss such folios.
The race window is tiny, so it's very unlikely to happen, though. While at it, also add a unlikely prefix.
--- mm/memory.c | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-)
mm/memory.c +11 -11
@@ -4661,10 +4661,8 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) goto out; folio =swap_cache_get_folio (entry); -if (folio) { +if (folio)swap_update_readahead (folio, vma, vmf->address); - page =folio_file_page (folio,swp_offset (entry)); - } swapcache = folio;if (!folio) {
@@ -4735,20 +4733,13 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) ret =VM_FAULT_MAJOR ;count_vm_event (PGMAJFAULT );count_memcg_event_mm (vma->vm_mm,PGMAJFAULT ); - page =folio_file_page (folio,swp_offset (entry)); - }else if (PageHWPoison (page)) { -/* -* hwpoisoned dirty swapcache pages are kept for killing -* owner processes (which may be unknown at hwpoison time) -*/ - ret =VM_FAULT_HWPOISON ; - goto out_release; } ret |=folio_lock_or_retry (folio, vmf);if (ret &VM_FAULT_RETRY ) goto out_release; + page =folio_file_page (folio,swp_offset (entry));if (swapcache) {/* * Make sure folio_free_swap() or swapoff did not release the
@@ -4761,6 +4752,15 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) page_swap_entry(page).val != entry.val)) goto out_page; +if (unlikely (PageHWPoison (page))) { +/* +* hwpoisoned dirty swapcache pages are kept for killing +* owner processes (which may be unknown at hwpoison time) +*/ + ret =VM_FAULT_HWPOISON ; + goto out_page; + } +/* * KSM sometimes has to copy on read faults, for example, if * folio->index of non-ksm folios would be nonlinear inside the
Signature · 1 line
2.51.0
PATCH v4 5/15 mm, swap: always lock and check the swap cache folio before use
In reply to Kairui SongFrom: Kairui Song <[email protected]>
Swap cache lookup only increases the reference count of the returned folio. That's not enough to ensure a folio is stable in the swap cache, so the folio could be removed from the swap cache at any time. The caller should always lock and check the folio before using it.
We have just documented this in kerneldoc, now introduce a helper for swap cache folio verification with proper sanity checks.
Also, sanitize a few current users to use this convention and the new helper for easier debugging. They were not having observable problems yet, only trivial issues like wasted CPU cycles on swapoff or reclaiming. They would fail in some other way, but it is still better to always follow this convention to make things robust and make later commits easier to do.
--- mm/memory.c | 3 +-- mm/swap.h | 27 +++++++++++++++++++++++++++ mm/swap_state.c | 7 +++++-- mm/swapfile.c | 10 ++++++++-- 4 files changed, 41 insertions(+), 6 deletions(-)
mm/memory.c +1 -2
@@ -4748,8 +4748,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)* swapcache, we need to check that the page's swap has not * changed. */ -if (unlikely (!folio_test_swapcache (folio) || -page_swap_entry (page).val != entry.val)) +if (unlikely (!folio_matches_swap_entry (folio, entry))) goto out_page;if (unlikely (PageHWPoison (page))) {
mm/swap.h +27 -0
@@ -52,6 +52,28 @@ static inline pgoff_t swap_cache_index(swp_entry_t entry)return swp_offset (entry) &SWAP_ADDRESS_SPACE_MASK ; } +/** +* folio_matches_swap_entry - Check if a folio matches a given swap entry. +* @folio: The folio. +* @entry: The swap entry to check against. +* +* Context: The caller should have the folio locked to ensure it's stable +* and nothing will move it in or out of the swap cache. +* Return: true or false. +*/ +static inline bool folio_matches_swap_entry (const struct folio *folio, +swp_entry_t entry) +{ +swp_entry_t folio_entry = folio->swap; +long nr_pages =folio_nr_pages (folio); + +VM_WARN_ON_ONCE_FOLIO (!folio_test_locked (folio), folio); +if (!folio_test_swapcache (folio)) +return false; +VM_WARN_ON_ONCE_FOLIO (!IS_ALIGNED (folio_entry.val, nr_pages), folio); +return folio_entry.val ==round_down (entry.val, nr_pages); +} +void show_swap_cache_info (void );void *get_shadow_from_swap_cache (swp_entry_t entry); int add_to_swap_cache(struct folio *folio, swp_entry_t entry,
@@ -144,6 +166,11 @@ static inline pgoff_t swap_cache_index(swp_entry_t entry)return 0 ; } +static inline bool folio_matches_swap_entry (const struct folio *folio,swp_entry_t entry) +{ +return false; +} +static inline void show_swap_cache_info (void ) { }
mm/swap_state.c +5 -2
@@ -79,7 +79,7 @@ void show_swap_cache_info(void)* with reference count or locks. * Return: Returns the found folio on success, NULL otherwise. The caller * must lock and check if the folio still matches the swap entry before -* use. +* use (e.g. with folio_matches_swap_entry). */ struct folio *swap_cache_get_folio (swp_entry_t entry) {
@@ -346,7 +346,10 @@ struct folio *__read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask,for (;;) {int err; -/* Check the swap cache in case the folio is already there */ +/* +* Check the swap cache first, if a cached folio is found, +* return it unlocked. The caller will lock and check it. +*/ folio =swap_cache_get_folio (entry);if (folio) goto got_folio;
mm/swapfile.c +8 -2
@@ -240,8 +240,7 @@ static int __try_to_reclaim_swap(struct swap_info_struct *si,* Offset could point to the middle of a large folio, or folio * may no longer point to the expected offset before it's locked. */ -if (offset <swp_offset (folio->swap) || - offset >=swp_offset (folio->swap) + nr_pages) { +if (!folio_matches_swap_entry (folio, entry)) {folio_unlock (folio);folio_put (folio); goto again;
@@ -2004,6 +2003,13 @@ static int unuse_pte(struct vm_area_struct *vma, pmd_t *pmd,bool hwpoisoned = false;int ret =1 ; +/* +* If the folio is removed from swap cache by others, continue to +* unuse other PTEs. try_to_unuse may try again if we missed this one. +*/ +if (!folio_matches_swap_entry (folio, entry)) +return 0 ; + swapcache = folio; folio =ksm_might_need_to_copy (folio, vma, addr);if (unlikely (!folio))
Signature · 1 line
2.51.0
Re: [PATCH v4 05/15] mm, swap: always lock and check the swap cache folio before use
Quoted 18 lines · Kairui Song
From: Kairui Song <[email protected]>
Swap cache lookup only increases the reference count of the returned folio. That's not enough to ensure a folio is stable in the swap cache, so the folio could be removed from the swap cache at any time. The caller should always lock and check the folio before using it.
We have just documented this in kerneldoc, now introduce a helper for swap cache folio verification with proper sanity checks.
Also, sanitize a few current users to use this convention and the new helper for easier debugging. They were not having observable problems yet, only trivial issues like wasted CPU cycles on swapoff or reclaiming. They would fail in some other way, but it is still better to always follow this convention to make things robust and make later commits easier to do.
Quoted 4 lines
Signed-off-by Kairui Song <[email protected]>Acked-by Chris Li <[email protected]>Acked-by Nhat Pham <[email protected]>
Thanks
Barry
PATCH v4 6/15 mm, swap: rename and move some swap cluster definition and helpers
In reply to Kairui SongFrom: Kairui Song <[email protected]>
No feature change, move cluster related definitions and helpers to mm/swap.h, also tidy up and add a "swap_" prefix for cluster lock/unlock helpers, so they can be used outside of swap files. And while at it, add kerneldoc.
--- include/linux/swap.h | 34 ---------------- mm/swap.h | 70 ++++++++++++++++++++++++++++++++ mm/swapfile.c | 97 +++++++++++++------------------------------- 3 files changed, 99 insertions(+), 102 deletions(-)
include/linux/swap.h +0 -34
@@ -235,40 +235,6 @@ enum {/* Special value in each swap_map continuation */ #define SWAP_CONT_MAX 0x7f/* Max count */ -/* -* We use this to track usage of a cluster. A cluster is a block of swap disk -* space with SWAPFILE_CLUSTER pages long and naturally aligns in disk. All -* free clusters are organized into a list. We fetch an entry from the list to -* get a free cluster. -* -* The flags field determines if a cluster is free. This is -* protected by cluster lock. -*/ -struct swap_cluster_info { -spinlock_t lock;/* -* Protect swap_cluster_info fields -* other than list, and swap_info_struct->swap_map -* elements corresponding to the swap cluster. -*/ -u16 count; -u8 flags; -u8 order; -struct list_head list; -}; - -/* All on-list cluster must have a non-zero flag. */ -enum swap_cluster_flags { -CLUSTER_FLAG_NONE =0 ,/* For temporary off-list cluster */ -CLUSTER_FLAG_FREE , -CLUSTER_FLAG_NONFULL , -CLUSTER_FLAG_FRAG , -/* Clusters with flags above are allocatable */ -CLUSTER_FLAG_USABLE =CLUSTER_FLAG_FRAG , -CLUSTER_FLAG_FULL , -CLUSTER_FLAG_DISCARD , -CLUSTER_FLAG_MAX , -}; -/* * The first page in the swap file is the swap header, which is always marked * bad to prevent it from being allocated as an entry. This also prevents the
mm/swap.h +70 -0
80 lines not on this page · raw message
mm/swapfile.c +29 -68
264 lines not on this page · raw message
Signature · 1 line
2.51.0
Re: [PATCH v4 06/15] mm, swap: rename and move some swap cluster definition and helpers
Quoted 8 lines · Kairui Song
From: Kairui Song <[email protected]>
No feature change, move cluster related definitions and helpers to
mm/swap.h, also tidy up and add a "swap_" prefix for cluster lock/unlock helpers, so they can be used outside of swap files. And while at it, add kerneldoc.
Quoted 5 lines
Signed-off-by Kairui Song <[email protected]>Reviewed-by Baolin Wang <[email protected]>Reviewed-by Barry Song <[email protected]>Acked-by Chris Li <[email protected]>
PATCH v4 7/15 mm, swap: tidy up swap device and cluster info helpers
In reply to Kairui SongFrom: Kairui Song <[email protected]>
swp_swap_info is the most commonly used helper for retrieving swap info. It has an internal check that may lead to a NULL return value, but almost none of its caller checks the return value, making the internal check pointless. In fact, most of these callers already ensured the entry is valid and never expect a NULL value.
Tidy this up and improve the function names. If the caller can make sure the swap entry/type is valid and the device is pinned, use the new introduced __swap_entry_to_info/__swap_type_to_info instead. They have more debug sanity checks and lower overhead as they are inlined.
Callers that may expect a NULL value should use
swap_entry_to_info/swap_type_to_info instead.
No feature change. The rearranged codes should have had no effect, or they should have been hitting NULL de-ref bugs already. Only some new sanity checks are added so potential issues may show up in debug build.
The new helpers will be frequently used with swap table later when working with swap cache folios. A locked swap cache folio ensures the entries are valid and stable so these helpers are very helpful.
--- include/linux/swap.h | 6 ------ mm/page_io.c | 12 ++++++------ mm/swap.h | 38 +++++++++++++++++++++++++++++++++----- mm/swap_state.c | 4 ++-- mm/swapfile.c | 37 +++++++++++++++++++------------------ 5 files changed, 60 insertions(+), 37 deletions(-)
include/linux/swap.h +0 -6
@@ -479,7 +479,6 @@ extern sector_t swapdev_block(int, pgoff_t);extern int __swap_count (swp_entry_t entry);extern bool swap_entry_swapped (struct swap_info_struct *si,swp_entry_t entry);extern int swp_swapcount (swp_entry_t entry); -struct swap_info_struct *swp_swap_info (swp_entry_t entry);struct backing_dev_info ;extern int init_swap_address_space (unsigned int type,unsigned long nr_pages);extern void exit_swap_address_space (unsigned int type);
@@ -492,11 +491,6 @@ static inline void put_swap_device(struct swap_info_struct *si) }#else /* CONFIG_SWAP */ -static inline struct swap_info_struct *swp_swap_info (swp_entry_t entry) -{ -return NULL ; -} -static inline struct swap_info_struct *get_swap_device (swp_entry_t entry) {return NULL ;
mm/page_io.c +6 -6
@@ -204,7 +204,7 @@ static bool is_folio_zero_filled(struct folio *folio)static void swap_zeromap_folio_set (struct folio *folio) {struct obj_cgroup *objcg =get_obj_cgroup_from_folio (folio); -struct swap_info_struct *sis =swp_swap_info (folio->swap); +struct swap_info_struct *sis =__swap_entry_to_info (folio->swap);int nr_pages =folio_nr_pages (folio);swp_entry_t entry;unsigned int i;
@@ -223,7 +223,7 @@ static void swap_zeromap_folio_set(struct folio *folio)static void swap_zeromap_folio_clear (struct folio *folio) { -struct swap_info_struct *sis =swp_swap_info (folio->swap); +struct swap_info_struct *sis =__swap_entry_to_info (folio->swap);swp_entry_t entry;unsigned int i;
@@ -374,7 +374,7 @@ static void sio_write_complete(struct kiocb *iocb, long ret)static void swap_writepage_fs (struct folio *folio,struct swap_iocb **swap_plug) {struct swap_iocb *sio = swap_plug ? *swap_plug :NULL ; -struct swap_info_struct *sis =swp_swap_info (folio->swap); +struct swap_info_struct *sis =__swap_entry_to_info (folio->swap);struct file *swap_file = sis->swap_file;loff_t pos =swap_dev_pos (folio->swap);
@@ -446,7 +446,7 @@ static void swap_writepage_bdev_async(struct folio *folio,void __swap_writepage (struct folio *folio,struct swap_iocb **swap_plug) { -struct swap_info_struct *sis =swp_swap_info (folio->swap); +struct swap_info_struct *sis =__swap_entry_to_info (folio->swap);VM_BUG_ON_FOLIO (!folio_test_swapcache (folio), folio);/*
@@ -537,7 +537,7 @@ static bool swap_read_folio_zeromap(struct folio *folio)static void swap_read_folio_fs (struct folio *folio,struct swap_iocb **plug) { -struct swap_info_struct *sis =swp_swap_info (folio->swap); +struct swap_info_struct *sis =__swap_entry_to_info (folio->swap);struct swap_iocb *sio =NULL ;loff_t pos =swap_dev_pos (folio->swap);
@@ -608,7 +608,7 @@ static void swap_read_folio_bdev_async(struct folio *folio,void swap_read_folio (struct folio *folio,struct swap_iocb **plug) { -struct swap_info_struct *sis =swp_swap_info (folio->swap); +struct swap_info_struct *sis =__swap_entry_to_info (folio->swap);bool synchronous = sis->flags &SWP_SYNCHRONOUS_IO ;bool workingset =folio_test_workingset (folio);unsigned long pflags;
mm/swap.h +33 -5
@@ -15,6 +15,8 @@ extern int page_cluster;#define swap_entry_order (order) 0#endif +extern struct swap_info_struct *swap_info[]; +/* * We use this to track usage of a cluster. A cluster is a block of swap disk * space with SWAPFILE_CLUSTER pages long and naturally aligns in disk. All
@@ -53,9 +55,29 @@ enum swap_cluster_flags {#include <linux/swapops.h> /* for swp_offset */ #include <linux/blk_types.h> /* for bio_end_io_t */ -static inline struct swap_cluster_info *swp_offset_cluster ( +/* +* Callers of all helpers below must ensure the entry, type, or offset is +* valid, and protect the swap device with reference count or locks. +*/ +static inline struct swap_info_struct *__swap_type_to_info (int type) +{ +struct swap_info_struct *si; + + si =READ_ONCE (swap_info[type]);/* rcu_dereference() */ +VM_WARN_ON_ONCE (percpu_ref_is_zero (&si->users));/* race with swapoff */ +return si; +} + +static inline struct swap_info_struct *__swap_entry_to_info (swp_entry_t entry) +{ +return __swap_type_to_info (swp_type (entry)); +} + +static inline struct swap_cluster_info *__swap_offset_to_cluster (struct swap_info_struct *si,pgoff_t offset) { +VM_WARN_ON_ONCE (percpu_ref_is_zero (&si->users));/* race with swapoff */ +VM_WARN_ON_ONCE (offset >= si->max);return &si->cluster_info[offset /SWAPFILE_CLUSTER ]; }
@@ -70,8 +92,9 @@ static inline struct swap_cluster_info *swp_offset_cluster(static inline struct swap_cluster_info *swap_cluster_lock (struct swap_info_struct *si,unsigned long offset) { -struct swap_cluster_info *ci =swp_offset_cluster (si, offset); +struct swap_cluster_info *ci =__swap_offset_to_cluster (si, offset); +VM_WARN_ON_ONCE (percpu_ref_is_zero (&si->users));/* race with swapoff */ spin_lock (&ci->lock);return ci; }
@@ -170,7 +193,7 @@ void swap_update_readahead(struct folio *folio, struct vm_area_struct *vma,static inline unsigned int folio_swap_flags (struct folio *folio) { -return swp_swap_info (folio->swap)->flags; +return __swap_entry_to_info (folio->swap)->flags; }/*
@@ -181,7 +204,7 @@ static inline unsigned int folio_swap_flags(struct folio *folio)static inline int swap_zeromap_batch (swp_entry_t entry,int max_nr,bool *is_zeromap) { -struct swap_info_struct *sis =swp_swap_info (entry); +struct swap_info_struct *sis =__swap_entry_to_info (entry);unsigned long start =swp_offset (entry);unsigned long end = start + max_nr;bool first_bit;
@@ -200,7 +223,7 @@ static inline int swap_zeromap_batch(swp_entry_t entry, int max_nr,static inline int non_swapcache_batch (swp_entry_t entry,int max_nr) { -struct swap_info_struct *si =swp_swap_info (entry); +struct swap_info_struct *si =__swap_entry_to_info (entry);pgoff_t offset =swp_offset (entry);int i;
@@ -219,6 +242,11 @@ static inline int non_swapcache_batch(swp_entry_t entry, int max_nr)#else /* CONFIG_SWAP */ struct swap_iocb ; +static inline struct swap_info_struct *__swap_entry_to_info (swp_entry_t entry) +{ +return NULL ; +} +static inline void swap_read_folio (struct folio *folio,struct swap_iocb **plug) { }
mm/swap_state.c +2 -2
@@ -336,7 +336,7 @@ struct folio *__read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask, struct mempolicy *mpol, pgoff_t ilx, bool *new_page_allocated, bool skip_if_exists) { - struct swap_info_struct *si = swp_swap_info(entry); + struct swap_info_struct *si = __swap_entry_to_info(entry);struct folio *folio;struct folio *new_folio =NULL ;struct folio *result =NULL ;
@@ -560,7 +560,7 @@ struct folio *swap_cluster_readahead(swp_entry_t entry, gfp_t gfp_mask,unsigned long offset = entry_offset;unsigned long start_offset, end_offset;unsigned long mask; -struct swap_info_struct *si =swp_swap_info (entry); +struct swap_info_struct *si =__swap_entry_to_info (entry);struct blk_plug plug;struct swap_iocb *splug =NULL ;bool page_allocated;
mm/swapfile.c +19 -18
@@ -102,7 +102,7 @@ static PLIST_HEAD(swap_active_head);static struct plist_head *swap_avail_heads;static DEFINE_SPINLOCK (swap_avail_lock); -static struct swap_info_struct *swap_info[MAX_SWAPFILES ]; +struct swap_info_struct *swap_info[MAX_SWAPFILES ];static DEFINE_MUTEX (swapon_mutex);
@@ -124,14 +124,20 @@ static DEFINE_PER_CPU(struct percpu_swap_cluster, percpu_swap_cluster) = { .lock = INIT_LOCAL_LOCK(), }; -static struct swap_info_struct *swap_type_to_swap_info (int type) +/* May return NULL on invalid type, caller must check for NULL return */ +static struct swap_info_struct *swap_type_to_info (int type) {if (type >=MAX_SWAPFILES )return NULL ; -return READ_ONCE (swap_info[type]);/* rcu_dereference() */ } +/* May return NULL on invalid entry, caller must check for NULL return */ +static struct swap_info_struct *swap_entry_to_info (swp_entry_t entry) +{ +return swap_type_to_info (swp_type (entry)); +} +static inline unsigned char swap_count (unsigned char ent) {return ent & ~SWAP_HAS_CACHE ;/* may include COUNT_CONTINUED flag */
@@ -342,7 +348,7 @@ offset_to_swap_extent(struct swap_info_struct *sis, unsigned long offset)sector_t swap_folio_sector (struct folio *folio) { -struct swap_info_struct *sis =swp_swap_info (folio->swap); +struct swap_info_struct *sis =__swap_entry_to_info (folio->swap);struct swap_extent *se;sector_t sector;pgoff_t offset;
@@ -1300,7 +1306,7 @@ static struct swap_info_struct *_swap_info_get(swp_entry_t entry)if (!entry.val) goto out; - si =swp_swap_info (entry); + si =swap_entry_to_info (entry);if (!si) goto bad_nofile;if (data_race (!(si->flags &SWP_USED )))
@@ -1415,7 +1421,7 @@ struct swap_info_struct *get_swap_device(swp_entry_t entry)if (!entry.val) goto out; - si =swp_swap_info (entry); + si =swap_entry_to_info (entry);if (!si) goto bad_nofile;if (!get_swap_device_info (si))
@@ -1538,7 +1544,7 @@ static void swap_entries_free(struct swap_info_struct *si,unsigned char *map_end = map + nr_pages;/* It should never free entries across different clusters */ -VM_BUG_ON (ci !=swp_offset_cluster (si, offset + nr_pages -1 )); +VM_BUG_ON (ci !=__swap_offset_to_cluster (si, offset + nr_pages -1 ));VM_BUG_ON (cluster_is_empty (ci));VM_BUG_ON (ci->count < nr_pages);
@@ -1596,7 +1602,7 @@ void put_swap_folio(struct folio *folio, swp_entry_t entry)int __swap_count (swp_entry_t entry) { -struct swap_info_struct *si =swp_swap_info (entry); +struct swap_info_struct *si =__swap_entry_to_info (entry);pgoff_t offset =swp_offset (entry);return swap_count (si->swap_map[offset]);
@@ -1827,7 +1833,7 @@ void free_swap_and_cache_nr(swp_entry_t entry, int nr)swp_entry_t get_swap_page_of_type (int type) { -struct swap_info_struct *si =swap_type_to_swap_info (type); +struct swap_info_struct *si =swap_type_to_info (type);unsigned long offset;swp_entry_t entry = {0 };
@@ -1908,7 +1914,7 @@ int find_first_swap(dev_t *device)*/ sector_t swapdev_block (int type,pgoff_t offset) { -struct swap_info_struct *si =swap_type_to_swap_info (type); +struct swap_info_struct *si =swap_type_to_info (type);struct swap_extent *se;if (!si || !(si->flags &SWP_WRITEOK ))
@@ -2837,7 +2843,7 @@ static void *swap_start(struct seq_file *swap, loff_t *pos)if (!l)return SEQ_START_TOKEN ; -for (type =0 ; (si =swap_type_to_swap_info (type)); type++) { +for (type =0 ; (si =swap_type_to_info (type)); type++) {if (!(si->flags &SWP_USED ) || !si->swap_map)continue ;if (!--l)
@@ -2858,7 +2864,7 @@ static void *swap_next(struct seq_file *swap, void *v, loff_t *pos) type = si->type +1 ; ++(*pos); -for (; (si =swap_type_to_swap_info (type)); type++) { +for (; (si =swap_type_to_info (type)); type++) {if (!(si->flags &SWP_USED ) || !si->swap_map)continue ;return si;
@@ -3531,7 +3537,7 @@ static int __swap_duplicate(swp_entry_t entry, unsigned char usage, int nr)unsigned char has_cache;int err, i; - si =swp_swap_info (entry); + si =swap_entry_to_info (entry);if (WARN_ON_ONCE (!si)) {pr_err ("%s%08lx\n" , Bad_file, entry.val);return -EINVAL ;
@@ -3646,11 +3652,6 @@ void swapcache_clear(struct swap_info_struct *si, swp_entry_t entry, int nr)swap_entries_put_cache (si, entry, nr); } -struct swap_info_struct *swp_swap_info (swp_entry_t entry) -{ -return swap_type_to_swap_info (swp_type (entry)); -} -/* * add_swap_count_continuation - called when a swap count is duplicated * beyond SWAP_MAP_MAX, it allocates a new page and links that to the entry's
Signature · 1 line
2.51.0
PATCH v4 8/15 mm, swap: cleanup swap cache API and add kerneldoc
In reply to Kairui SongFrom: Kairui Song <[email protected]>
In preparation for replacing the swap cache backend with the swap table, clean up and add proper kernel doc for all swap cache APIs. Now all swap cache APIs are well-defined with consistent names.
No feature change, only renaming and documenting.
--- mm/filemap.c | 2 +- mm/memory-failure.c | 2 +- mm/memory.c | 2 +- mm/shmem.c | 10 +++--- mm/swap.h | 48 ++++++++++++++----------- mm/swap_state.c | 86 ++++++++++++++++++++++++++++++++------------- mm/swapfile.c | 8 ++--- mm/vmscan.c | 2 +- mm/zswap.c | 2 +- 9 files changed, 103 insertions(+), 59 deletions(-)
mm/filemap.c +1 -1
@@ -4525,7 +4525,7 @@ static void filemap_cachestat(struct address_space *mapping,* invalidation, so there might not be * a shadow in the swapcache (yet). */ - shadow =get_shadow_from_swap_cache (swp); + shadow =swap_cache_get_shadow (swp);if (!shadow) goto resched; }
mm/memory-failure.c +1 -1
@@ -1127,7 +1127,7 @@ static int me_swapcache_clean(struct page_state *ps, struct page *p)struct folio *folio =page_folio (p);int ret; -delete_from_swap_cache (folio); +swap_cache_del_folio (folio); ret =delete_from_lru_cache (folio) ?MF_FAILED :MF_RECOVERED ;folio_unlock (folio);
mm/memory.c +1 -1
@@ -4699,7 +4699,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)memcg1_swapin (entry, nr_pages); - shadow =get_shadow_from_swap_cache (entry); + shadow =swap_cache_get_shadow (entry);if (shadow)workingset_refault (folio, shadow);
mm/shmem.c +5 -5
@@ -1661,13 +1661,13 @@ int shmem_writeout(struct folio *folio, struct swap_iocb **plug, }/* -* The delete_from_swap_cache() below could be left for +* The swap_cache_del_folio() below could be left for * shrink_folio_list()'s folio_free_swap() to dispose of; * but I'm a little nervous about letting this folio out of * shmem_writeout() in a hybrid half-tmpfs-half-swap state * e.g. folio_mapping(folio) might give an unexpected answer. */ -delete_from_swap_cache (folio); +swap_cache_del_folio (folio); goto redirty; }if (nr_pages >1 )
@@ -2045,7 +2045,7 @@ static struct folio *shmem_swap_alloc_folio(struct inode *inode, new->swap = entry;memcg1_swapin (entry, nr_pages); - shadow =get_shadow_from_swap_cache (entry); + shadow =swap_cache_get_shadow (entry);if (shadow)workingset_refault (new, shadow);folio_add_lru (new);
@@ -2183,7 +2183,7 @@ static void shmem_set_folio_swapin_error(struct inode *inode, pgoff_t index, nr_pages =folio_nr_pages (folio);folio_wait_writeback (folio);if (!skip_swapcache) -delete_from_swap_cache (folio); +swap_cache_del_folio (folio);/* * Don't treat swapin error folio as alloced. Otherwise inode->i_blocks * won't be 0 when inode is released and thus trigger WARN_ON(i_blocks)
@@ -2422,7 +2422,7 @@ static int shmem_swapin_folio(struct inode *inode, pgoff_t index, folio->swap.val =0 ;swapcache_clear (si, swap, nr_pages); }else { -delete_from_swap_cache (folio); +swap_cache_del_folio (folio); }folio_mark_dirty (folio);swap_free_nr (swap, nr_pages);
mm/swap.h +27 -21
72 lines not on this page · raw message
mm/swap_state.c +62 -24
142 lines not on this page · raw message
mm/swapfile.c +4 -4
@@ -267,7 +267,7 @@ static int __try_to_reclaim_swap(struct swap_info_struct *si,if (!need_reclaim) goto out_unlock; -delete_from_swap_cache (folio); +swap_cache_del_folio (folio);folio_set_dirty (folio); ret = nr_pages; out_unlock:
@@ -1124,7 +1124,7 @@ static void swap_range_free(struct swap_info_struct *si, unsigned long offset,swap_slot_free_notify (si->bdev, offset); offset++; } -clear_shadow_from_swap_cache (si->type, begin, end); +swap_cache_clear_shadow (si->type, begin, end);/* * Make sure that try_to_unuse() observes si->inuse_pages reaching 0
@@ -1289,7 +1289,7 @@ int folio_alloc_swap(struct folio *folio, gfp_t gfp)* TODO: this could cause a theoretical memory reclaim * deadlock in the swap out path. */ -if (add_to_swap_cache (folio, entry, gfp | __GFP_NOMEMALLOC,NULL )) +if (swap_cache_add_folio (folio, entry, gfp | __GFP_NOMEMALLOC,NULL )) goto out_free;return 0 ;
@@ -1759,7 +1759,7 @@ bool folio_free_swap(struct folio *folio)if (folio_swapped (folio))return false; -delete_from_swap_cache (folio); +swap_cache_del_folio (folio);folio_set_dirty (folio);return true; }
mm/vmscan.c +1 -1
@@ -776,7 +776,7 @@ static int __remove_mapping(struct address_space *mapping, struct folio *folio,if (reclaimed && !mapping_exiting (mapping)) shadow =workingset_eviction (folio, target_memcg); -__delete_from_swap_cache (folio, swap, shadow); +__swap_cache_del_folio (folio, swap, shadow);memcg1_swapout (folio, swap);xa_unlock_irq (&mapping->i_pages);put_swap_folio (folio, swap);
mm/zswap.c +1 -1
@@ -1069,7 +1069,7 @@ static int zswap_writeback_entry(struct zswap_entry *entry, out:if (ret && ret != -EEXIST ) { -delete_from_swap_cache (folio); +swap_cache_del_folio (folio);folio_unlock (folio); }folio_put (folio);
Signature · 1 line
2.51.0
PATCH v4 9/15 mm/shmem, swap: remove redundant error handling for replacing folio
In reply to Kairui SongFrom: Kairui Song <[email protected]>
Shmem may replace a folio in the swap cache if the cached one doesn't fit the swapin's GFP zone. When doing so, shmem has already double checked that the swap cache folio is locked, still has the swap cache flag set, and contains the wanted swap entry. So it is impossible to fail due to an XArray mismatch. There is even a comment for that.
Delete the defensive error handling path, and add a WARN_ON instead:
if that happened, something has broken the basic principle of how the swap cache works, we should catch and fix that.
--- mm/shmem.c | 32 +++++++------------------------- 1 file changed, 7 insertions(+), 25 deletions(-)
mm/shmem.c +7 -25
@@ -2121,35 +2121,17 @@ static int shmem_replace_folio(struct folio **foliop, gfp_t gfp,/* Swap cache still stores N entries instead of a high-order entry */ xa_lock_irq (&swap_mapping->i_pages);for (i =0 ; i < nr_pages; i++) { -void *item =xas_load (&xas); - -if (item != old) { - error = -ENOENT ; -break ; - } - -xas_store (&xas, new); +WARN_ON_ONCE (xas_store (&xas, new) != old);xas_next (&xas); } -if (!error) { -mem_cgroup_replace_folio (old, new); -shmem_update_stats (new, nr_pages); -shmem_update_stats (old, -nr_pages); - } + +mem_cgroup_replace_folio (old, new); +shmem_update_stats (new, nr_pages); +shmem_update_stats (old, -nr_pages);xa_unlock_irq (&swap_mapping->i_pages); -if (unlikely (error)) { -/* -* Is this possible? I think not, now that our callers -* check both the swapcache flag and folio->private -* after getting the folio lock; but be defensive. -* Reverse old to newpage for clear and free. -*/ - old = new; - }else { -folio_add_lru (new); - *foliop = new; - } +folio_add_lru (new); + *foliop = new;folio_clear_swapcache (old); old->private =NULL ;
Signature · 1 line
2.51.0
Re: [PATCH v4 09/15] mm/shmem, swap: remove redundant error handling for replacing folio
Quoted 9 lines · Kairui Song
From: Kairui Song <[email protected]>
Shmem may replace a folio in the swap cache if the cached one doesn't fit the swapin's GFP zone. When doing so, shmem has already double checked that the swap cache folio is locked, still has the swap cache flag set, and contains the wanted swap entry. So it is impossible to fail due to an XArray mismatch. There is even a comment for that.
Quoted 7 lines
Delete the defensive error handling path, and add a
WARN_ONinstead:
if that happened, something has broken the basic principle of how the swap cache works, we should catch and fix that.Signed-off-by Kairui Song <[email protected]>Reviewed-by David Hildenbrand <[email protected]>Reviewed-by Baolin Wang <[email protected]>
Chris
Quoted 55 lines
--- mm/shmem.c | 32 +++++++------------------------- 1 file changed, 7 insertions(+), 25 deletions(-)mm/shmem.c +7
-26@@ -2121,35 +2121,17 @@ static int shmem_replace_folio(struct folio **foliop, gfp_t gfp,/* Swap cache still stores N entries instead of a high-order entry */ xa_lock_irq (&swap_mapping->i_pages);for (i =0 ; i < nr_pages; i++) { -void *item =xas_load (&xas); - -if (item != old) { - error = -ENOENT ; -break ; - } - -xas_store (&xas, new); +WARN_ON_ONCE (xas_store (&xas, new) != old);xas_next (&xas); } -if (!error) { -mem_cgroup_replace_folio (old, new); -shmem_update_stats (new, nr_pages); -shmem_update_stats (old, -nr_pages); - } + +mem_cgroup_replace_folio (old, new); +shmem_update_stats (new, nr_pages); +shmem_update_stats (old, -nr_pages);xa_unlock_irq (&swap_mapping->i_pages); -if (unlikely (error)) { -/* -* Is this possible? I think not, now that our callers -* check both the swapcache flag and folio->private -* after getting the folio lock; but be defensive. -* Reverse old to newpage for clear and free. -*/ - old = new; - }else { -folio_add_lru (new); - *foliop = new; - } +folio_add_lru (new); + *foliop = new;folio_clear_swapcache (old); old->private =NULL ; --2.51.0
PATCH v4 10/15 mm, swap: wrap swap cache replacement with a helper
In reply to Kairui SongFrom: Kairui Song <[email protected]>
There are currently three swap cache users that are trying to replace an existing folio with a new one: huge memory splitting, migration, and shmem replacement. What they are doing is quite similar.
Introduce a common helper for this. In later commits, this can be easily switched to use the swap table by updating this helper.
The newly added helper also makes the swap cache API better defined, and make debugging easier by adding a few more debug checks.
Migration and shmem replace are meant to clone the folio, including content, swap entry value, and flags. And splitting will adjust each sub folio's swap entry according to order, which could be non-uniform in the future. So document it clearly that it's the caller's responsibility to set up the new folio's swap entries and flags before calling the helper. The helper will just follow the new folio's entry value.
This also prepares for replacing high-order folios in the swap cache. Currently, only splitting to order 0 is allowed for swap cache folios. Using the new helper, we can handle high-order folio splitting better.
--- mm/huge_memory.c | 4 +--- mm/migrate.c | 11 +++-------- mm/shmem.c | 11 ++--------- mm/swap.h | 5 +++++ mm/swap_state.c | 33 +++++++++++++++++++++++++++++++++ 5 files changed, 44 insertions(+), 20 deletions(-)
mm/huge_memory.c +1 -3
@@ -3798,9 +3798,7 @@ static int __folio_split(struct folio *folio, unsigned int new_order,* NOTE: shmem in swap cache is not supported yet. */ if (swap_cache) { -__xa_store (&swap_cache->i_pages, -swap_cache_index (new_folio->swap), - new_folio,0 ); +__swap_cache_replace_folio (folio, new_folio);continue ; }
mm/migrate.c +3 -8
@@ -566,7 +566,6 @@ static int __folio_migrate_mapping(struct address_space *mapping,struct zone *oldzone, *newzone;int dirty;long nr =folio_nr_pages (folio); -long entries, i;if (!mapping) {/* Take off deferred split queue while frozen and memcg set */
@@ -615,9 +614,6 @@ static int __folio_migrate_mapping(struct address_space *mapping,if (folio_test_swapcache (folio)) {folio_set_swapcache (newfolio); newfolio->private =folio_get_private (folio); - entries = nr; - }else { - entries =1 ; }/* Move dirty while folio refs frozen and newfolio not yet exposed */
@@ -627,11 +623,10 @@ static int __folio_migrate_mapping(struct address_space *mapping,folio_set_dirty (newfolio); } -/* Swap cache still stores N entries instead of a high-order entry */ -for (i =0 ; i < entries; i++) { +if (folio_test_swapcache (folio)) +__swap_cache_replace_folio (folio, newfolio); +else xas_store (&xas, newfolio); -xas_next (&xas); - }/* * Drop cache reference from old folio by unfreezing
mm/shmem.c +2 -9
@@ -2086,10 +2086,8 @@ static int shmem_replace_folio(struct folio **foliop, gfp_t gfp,struct folio *new, *old = *foliop;swp_entry_t entry = old->swap;struct address_space *swap_mapping =swap_address_space (entry); -pgoff_t swap_index =swap_cache_index (entry); -XA_STATE (xas, &swap_mapping->i_pages, swap_index);int nr_pages =folio_nr_pages (old); -int error =0 , i; +int error =0 ;/* * We have arrived here because our zones are constrained, so don't
@@ -2118,13 +2116,8 @@ static int shmem_replace_folio(struct folio **foliop, gfp_t gfp, new->swap = entry;folio_set_swapcache (new); -/* Swap cache still stores N entries instead of a high-order entry */ xa_lock_irq (&swap_mapping->i_pages); -for (i =0 ; i < nr_pages; i++) { -WARN_ON_ONCE (xas_store (&xas, new) != old); -xas_next (&xas); - } - +__swap_cache_replace_folio (old, new);mem_cgroup_replace_folio (old, new);shmem_update_stats (new, nr_pages);shmem_update_stats (old, -nr_pages);
mm/swap.h +5 -0
@@ -185,6 +185,7 @@ int swap_cache_add_folio(struct folio *folio, swp_entry_t entry,void swap_cache_del_folio (struct folio *folio);void __swap_cache_del_folio (struct folio *folio,swp_entry_t entry,void *shadow); +void __swap_cache_replace_folio (struct folio *old,struct folio *new);void swap_cache_clear_shadow (int type,unsigned long begin,unsigned long end);
@@ -336,6 +337,10 @@ static inline void __swap_cache_del_folio(struct folio *folio, swp_entry_t entry { } +static inline void __swap_cache_replace_folio (struct folio *old,struct folio *new) +{ +} +static inline unsigned int folio_swap_flags (struct folio *folio) {return 0 ;
mm/swap_state.c +33 -0
@@ -234,6 +234,39 @@ void swap_cache_del_folio(struct folio *folio)folio_ref_sub (folio,folio_nr_pages (folio)); } +/** +* __swap_cache_replace_folio - Replace a folio in the swap cache. +* @old: The old folio to be replaced. +* @new: The new folio. +* +* Replace an existing folio in the swap cache with a new folio. The +* caller is responsible for setting up the new folio's flag and swap +* entries. Replacement will take the new folio's swap entry value as +* the starting offset to override all slots covered by the new folio. +* +* Context: Caller must ensure both folios are locked, also lock the +* swap address_space that holds the old folio to avoid races. +*/ +void __swap_cache_replace_folio (struct folio *old,struct folio *new) +{ +swp_entry_t entry = new->swap; +unsigned long nr_pages =folio_nr_pages (new); +unsigned long offset =swap_cache_index (entry); +unsigned long end = offset + nr_pages; + +XA_STATE (xas, &swap_address_space (entry)->i_pages, offset); + +VM_WARN_ON_ONCE (!folio_test_swapcache (old) || !folio_test_swapcache (new)); +VM_WARN_ON_ONCE (!folio_test_locked (old) || !folio_test_locked (new)); +VM_WARN_ON_ONCE (!entry.val); + +/* Swap cache still stores N entries instead of a high-order entry */ +do { +WARN_ON_ONCE (xas_store (&xas, new) != old); +xas_next (&xas); + }while (++offset < end); +} +/** * swap_cache_clear_shadow - Clears a set of shadows in the swap cache. * @type: Indicates the swap device.
Signature · 1 line
2.51.0
PATCH v4 11/15 mm, swap: use the swap table for the swap cache and switch API
In reply to Kairui SongFrom: Kairui Song <[email protected]>
Introduce basic swap table infrastructures, which are now just a fixed-sized flat array inside each swap cluster, with access wrappers.
Each cluster contains a swap table of 512 entries. Each table entry is an opaque atomic long. It could be in 3 types: a shadow type (XA_VALUE), a folio type (pointer), or NULL.
In this first step, it only supports storing a folio or shadow, and it is a drop-in replacement for the current swap cache. Convert all swap cache users to use the new sets of APIs. Chris Li has been suggesting using a new infrastructure for swap cache for better performance, and that idea combined well with the swap table as the new backing structure. Now the lock contention range is reduced to 2M clusters, which is much smaller than the 64M address_space. And we can also drop the multiple address_space design.
All the internal works are done with swap_cache_get_* helpers. Swap cache lookup is still lock-less like before, and the helper's contexts are same with original swap cache helpers. They still require a pin on the swap device to prevent the backing data from being freed.
Swap cache updates are now protected by the swap cluster lock instead of the XArray lock. This is mostly handled internally, but new __swap_cache_* helpers require the caller to lock the cluster. So, a few new cluster access and locking helpers are also introduced.
A fully cluster-based unified swap table can be implemented on top of this to take care of all count tracking and synchronization work, with dynamic allocation. It should reduce the memory usage while making the performance even better.
--- MAINTAINERS | 1 + include/linux/swap.h | 2 - mm/huge_memory.c | 13 +- mm/migrate.c | 19 ++- mm/shmem.c | 8 +- mm/swap.h | 154 +++++++++++++++++------ mm/swap_state.c | 293 +++++++++++++++++++------------------------ mm/swap_table.h | 97 ++++++++++++++ mm/swapfile.c | 100 +++++++++++---- mm/vmscan.c | 20 ++- 10 files changed, 459 insertions(+), 248 deletions(-) create mode 100644 mm/swap_table.h
MAINTAINERS +1 -0
@@ -16232,6 +16232,7 @@ F: include/linux/swapops.h F: mm/page_io.c F: mm/swap.c F: mm/swap.h +F: mm/swap_table.h F: mm/swap_state.c F: mm/swapfile.c
include/linux/swap.h +0 -2
@@ -480,8 +480,6 @@ extern int __swap_count(swp_entry_t entry);extern bool swap_entry_swapped (struct swap_info_struct *si,swp_entry_t entry);extern int swp_swapcount (swp_entry_t entry);struct backing_dev_info ; -extern int init_swap_address_space (unsigned int type,unsigned long nr_pages); -extern void exit_swap_address_space (unsigned int type);extern struct swap_info_struct *get_swap_device (swp_entry_t entry);sector_t swap_folio_sector (struct folio *folio);
mm/huge_memory.c +6 -7
@@ -3720,7 +3720,7 @@ static int __folio_split(struct folio *folio, unsigned int new_order,/* Prevent deferred_split_scan() touching ->_refcount */ spin_lock (&ds_queue->split_queue_lock);if (folio_ref_freeze (folio,1 + extra_pins)) { -struct address_space *swap_cache =NULL ; +struct swap_cluster_info *ci =NULL ;struct lruvec *lruvec;int expected_refs;
@@ -3764,8 +3764,7 @@ static int __folio_split(struct folio *folio, unsigned int new_order, goto fail; } - swap_cache =swap_address_space (folio->swap); -xa_lock (&swap_cache->i_pages); + ci =swap_cluster_get_and_lock (folio); }/* lock lru list/PageCompound, ref frozen by page_ref_freeze */
@@ -3797,8 +3796,8 @@ static int __folio_split(struct folio *folio, unsigned int new_order,* Anonymous folio with swap cache. * NOTE: shmem in swap cache is not supported yet. */ -if (swap_cache) { -__swap_cache_replace_folio (folio, new_folio); +if (ci) { +__swap_cache_replace_folio (ci, folio, new_folio);continue ; }
@@ -3833,8 +3832,8 @@ static int __folio_split(struct folio *folio, unsigned int new_order,unlock_page_lruvec (lruvec); -if (swap_cache) -xa_unlock (&swap_cache->i_pages); +if (ci) +swap_cluster_unlock (ci); }else {spin_unlock (&ds_queue->split_queue_lock); ret = -EAGAIN ;
mm/migrate.c +15 -4
@@ -563,6 +563,7 @@ static int __folio_migrate_mapping(struct address_space *mapping, struct folio *newfolio, struct folio *folio, int expected_count) { XA_STATE(xas, &mapping->i_pages, folio_index(folio)); +struct swap_cluster_info *ci =NULL ;struct zone *oldzone, *newzone;int dirty;long nr =folio_nr_pages (folio);
@@ -591,9 +592,16 @@ static int __folio_migrate_mapping(struct address_space *mapping, oldzone =folio_zone (folio); newzone =folio_zone (newfolio); -xas_lock_irq (&xas); +if (folio_test_swapcache (folio)) + ci =swap_cluster_get_and_lock_irq (folio); +else +xas_lock_irq (&xas); +if (!folio_ref_freeze (folio, expected_count)) { -xas_unlock_irq (&xas); +if (ci) +swap_cluster_unlock_irq (ci); +else +xas_unlock_irq (&xas);return -EAGAIN ; }
@@ -624,7 +632,7 @@ static int __folio_migrate_mapping(struct address_space *mapping, }if (folio_test_swapcache (folio)) -__swap_cache_replace_folio (folio, newfolio); +__swap_cache_replace_folio (ci, folio, newfolio);else xas_store (&xas, newfolio);
@@ -635,8 +643,11 @@ static int __folio_migrate_mapping(struct address_space *mapping,*/ folio_ref_unfreeze (folio, expected_count - nr); -xas_unlock (&xas);/* Leave irq disabled to prevent preemption while updating stats */ +if (ci) +swap_cluster_unlock (ci); +else +xas_unlock (&xas);/* * If moved to a different zone then also account
mm/shmem.c +4 -4
@@ -2083,9 +2083,9 @@ static int shmem_replace_folio(struct folio **foliop, gfp_t gfp, struct shmem_inode_info *info, pgoff_t index, struct vm_area_struct *vma) { + struct swap_cluster_info *ci;struct folio *new, *old = *foliop;swp_entry_t entry = old->swap; -struct address_space *swap_mapping =swap_address_space (entry);int nr_pages =folio_nr_pages (old);int error =0 ;
@@ -2116,12 +2116,12 @@ static int shmem_replace_folio(struct folio **foliop, gfp_t gfp, new->swap = entry;folio_set_swapcache (new); -xa_lock_irq (&swap_mapping->i_pages); -__swap_cache_replace_folio (old, new); + ci =swap_cluster_get_and_lock_irq (old); +__swap_cache_replace_folio (ci, old, new);mem_cgroup_replace_folio (old, new);shmem_update_stats (new, nr_pages);shmem_update_stats (old, -nr_pages); -xa_unlock_irq (&swap_mapping->i_pages); +swap_cluster_unlock_irq (ci);folio_add_lru (new); *foliop = new;
mm/swap.h +119 -35
245 lines not on this page · raw message
mm/swap_state.c +128 -165
413 lines not on this page · raw message
mm/swap_table.h new file +97 -0
97 lines not on this page · raw message
mm/swapfile.c +75 -25
201 lines not on this page · raw message
mm/vmscan.c +14 -6
@@ -730,13 +730,18 @@ static int __remove_mapping(struct address_space *mapping, struct folio *folio, {int refcount;void *shadow =NULL ; +struct swap_cluster_info *ci;BUG_ON (!folio_test_locked (folio));BUG_ON (mapping !=folio_mapping (folio)); -if (!folio_test_swapcache (folio)) +if (folio_test_swapcache (folio)) { + ci =swap_cluster_get_and_lock_irq (folio); + }else {spin_lock (&mapping->host->i_lock); -xa_lock_irq (&mapping->i_pages); +xa_lock_irq (&mapping->i_pages); + } +/* * The non racy check for a busy folio. *
@@ -776,9 +781,9 @@ static int __remove_mapping(struct address_space *mapping, struct folio *folio,if (reclaimed && !mapping_exiting (mapping)) shadow =workingset_eviction (folio, target_memcg); -__swap_cache_del_folio (folio, swap, shadow); +__swap_cache_del_folio (ci, folio, swap, shadow);memcg1_swapout (folio, swap); -xa_unlock_irq (&mapping->i_pages); +swap_cluster_unlock_irq (ci);put_swap_folio (folio, swap); }else {void (*free_folio)(struct folio *);
@@ -816,9 +821,12 @@ static int __remove_mapping(struct address_space *mapping, struct folio *folio,return 1 ; cannot_free: -xa_unlock_irq (&mapping->i_pages); -if (!folio_test_swapcache (folio)) +if (folio_test_swapcache (folio)) { +swap_cluster_unlock_irq (ci); + }else { +xa_unlock_irq (&mapping->i_pages);spin_unlock (&mapping->host->i_lock); + }return 0 ; }
Signature · 1 line
2.51.0
PATCH v4 12/15 mm, swap: mark swap address space ro and add context debug check
In reply to Kairui SongFrom: Kairui Song <[email protected]>
Swap cache is now backed by swap table, and the address space is not holding any mutable data anymore. And swap cache is now protected by the swap cluster lock, instead of the XArray lock. All access to swap cache are wrapped by swap cache helpers. Locking is mostly handled internally by swap cache helpers, only a few __swap_cache_* helpers require the caller to lock the cluster by themselves.
Worth noting that, unlike XArray, the cluster lock is not IRQ safe. The swap cache was very different compared to filemap, and now it's completely separated from filemap. Nothing wants to mark or change anything or do a writeback callback in IRQ.
So explicitly document this and add a debug check to avoid further potential misuse. And mark the swap cache space as read-only to avoid any user wrongly mixing unexpected filemap helpers with swap cache.
--- mm/swap.h | 12 +++++++++++- mm/swap_state.c | 3 ++- 2 files changed, 13 insertions(+), 2 deletions(-)
mm/swap.h +11 -1
@@ -99,6 +99,16 @@ static __always_inline struct swap_cluster_info *__swap_cluster_lock( {struct swap_cluster_info *ci =__swap_offset_to_cluster (si, offset); +/* +* Nothing modifies swap cache in an IRQ context. All access to +* swap cache is wrapped by swap_cache_* helpers, and swap cache +* writeback is handled outside of IRQs. Swapin or swapout never +* occurs in IRQ, and neither does in-place split or replace. +* +* Besides, modifying swap cache requires synchronization with +* swap_map, which was never IRQ safe. +*/ +VM_WARN_ON_ONCE (!in_task ());VM_WARN_ON_ONCE (percpu_ref_is_zero (&si->users));/* race with swapoff */ if (irq)spin_lock_irq (&ci->lock);
@@ -192,7 +202,7 @@ void __swap_writepage(struct folio *folio, struct swap_iocb **swap_plug);#define SWAP_ADDRESS_SPACE_SHIFT 14#define SWAP_ADDRESS_SPACE_PAGES (1 << SWAP_ADDRESS_SPACE_SHIFT)#define SWAP_ADDRESS_SPACE_MASK (SWAP_ADDRESS_SPACE_PAGES - 1) -extern struct address_space swap_space; +extern struct address_space swap_space__ro_after_init ;static inline struct address_space *swap_address_space (swp_entry_t entry) {return &swap_space;
mm/swap_state.c +2 -1
@@ -37,7 +37,8 @@ static const struct address_space_operations swap_aops = {#endif }; -struct address_space swap_space__read_mostly = { +/* Set swap_space as read only as swap cache is handled by swap table */ +struct address_space swap_space__ro_after_init = { .a_ops = &swap_aops, };
Signature · 1 line
2.51.0
PATCH v4 13/15 mm, swap: remove contention workaround for swap cache
In reply to Kairui SongFrom: Kairui Song <[email protected]>
Swap cluster setup will try to shuffle the clusters on initialization. It was helpful to avoid contention for the swap cache space. The cluster size (2M) was much smaller than each swap cache space (64M), so shuffling the cluster means the allocator will try to allocate swap slots that are in different swap cache spaces for each CPU, reducing the chance of two CPUs using the same swap cache space, and hence reducing the contention.
Now, swap cache is managed by swap clusters, this shuffle is pointless. Just remove it, and clean up related macros.
This also improves the HDD swap performance as shuffling IO is a bad idea for HDD, and now the shuffling is gone. Test have shown a ~40% performance gain for HDD [1]:
Doing sequential swap in of 8G data using 8 processes with usemem, average of 3 test runs:
Before: 1270.91 KB/s per process After: 1849.54 KB/s per process
--- mm/swap.h | 4 ---- mm/swapfile.c | 32 ++++++++------------------------ mm/zswap.c | 7 +++++-- 3 files changed, 13 insertions(+), 30 deletions(-)
mm/swap.h +0 -4
@@ -198,10 +198,6 @@ int swap_writeout(struct folio *folio, struct swap_iocb **swap_plug);void __swap_writepage (struct folio *folio,struct swap_iocb **swap_plug);/* linux/mm/swap_state.c */ -/* One swap address space for each 64M swap space */ -#define SWAP_ADDRESS_SPACE_SHIFT 14 -#define SWAP_ADDRESS_SPACE_PAGES (1 << SWAP_ADDRESS_SPACE_SHIFT) -#define SWAP_ADDRESS_SPACE_MASK (SWAP_ADDRESS_SPACE_PAGES - 1)extern struct address_space swap_space__ro_after_init ;static inline struct address_space *swap_address_space (swp_entry_t entry) {
mm/swapfile.c +8 -24
@@ -3204,21 +3204,14 @@ static int setup_swap_map(struct swap_info_struct *si,return 0 ; } -#define SWAP_CLUSTER_INFO_COLS \ - DIV_ROUND_UP(L1_CACHE_BYTES, sizeof(struct swap_cluster_info)) -#define SWAP_CLUSTER_SPACE_COLS \ - DIV_ROUND_UP(SWAP_ADDRESS_SPACE_PAGES, SWAPFILE_CLUSTER) -#define SWAP_CLUSTER_COLS \ - max_t(unsigned int, SWAP_CLUSTER_INFO_COLS, SWAP_CLUSTER_SPACE_COLS) -static struct swap_cluster_info *setup_clusters (struct swap_info_struct *si,union swap_header *swap_header,unsigned long maxpages) {unsigned long nr_clusters =DIV_ROUND_UP (maxpages,SWAPFILE_CLUSTER );struct swap_cluster_info *cluster_info; -unsigned long i, j, idx;int err = -ENOMEM ; +unsigned long i; cluster_info =kvcalloc (nr_clusters,sizeof (*cluster_info),GFP_KERNEL );if (!cluster_info)
@@ -3267,22 +3260,13 @@ static struct swap_cluster_info *setup_clusters(struct swap_info_struct *si,INIT_LIST_HEAD (&si->frag_clusters[i]); } -/* -* Reduce false cache line sharing between cluster_info and -* sharing same address space. -*/ -for (j =0 ; j <SWAP_CLUSTER_COLS ; j++) { -for (i =0 ; i <DIV_ROUND_UP (nr_clusters,SWAP_CLUSTER_COLS ); i++) { -struct swap_cluster_info *ci; - idx = i *SWAP_CLUSTER_COLS + j; - ci = cluster_info + idx; -if (idx >= nr_clusters) -continue ; -if (ci->count) { - ci->flags =CLUSTER_FLAG_NONFULL ; -list_add_tail (&ci->list, &si->nonfull_clusters[0 ]); -continue ; - } +for (i =0 ; i < nr_clusters; i++) { +struct swap_cluster_info *ci = &cluster_info[i]; + +if (ci->count) { + ci->flags =CLUSTER_FLAG_NONFULL ; +list_add_tail (&ci->list, &si->nonfull_clusters[0 ]); + }else { ci->flags =CLUSTER_FLAG_FREE ;list_add_tail (&ci->list, &si->free_clusters); }
mm/zswap.c +5 -2
@@ -225,10 +225,13 @@ static bool zswap_has_pool;* helpers and fwd declarations **********************************/ +/* One swap address space for each 64M swap space */ +#define ZSWAP_ADDRESS_SPACE_SHIFT 14 +#define ZSWAP_ADDRESS_SPACE_PAGES (1 << ZSWAP_ADDRESS_SPACE_SHIFT)static inline struct xarray *swap_zswap_tree (swp_entry_t swp) {return &zswap_trees[swp_type (swp)][swp_offset (swp) - >>SWAP_ADDRESS_SPACE_SHIFT ]; + >>ZSWAP_ADDRESS_SPACE_SHIFT ]; }#define zswap_pool_debug (msg, p) \
@@ -1674,7 +1677,7 @@ int zswap_swapon(int type, unsigned long nr_pages)struct xarray *trees, *tree;unsigned int nr, i; - nr =DIV_ROUND_UP (nr_pages,SWAP_ADDRESS_SPACE_PAGES ); + nr =DIV_ROUND_UP (nr_pages,ZSWAP_ADDRESS_SPACE_PAGES ); trees =kvcalloc (nr,sizeof (*tree),GFP_KERNEL );if (!trees) {pr_err ("alloc failed, zswap disabled for swap type %d\n" , type);
Signature · 1 line
2.51.0
PATCH v4 14/15 mm, swap: implement dynamic allocation of swap table
In reply to Kairui SongFrom: Kairui Song <[email protected]>
Now swap table is cluster based, which means free clusters can free its table since no one should modify it.
There could be speculative readers, like swap cache look up, protect them by making them RCU protected. All swap table should be filled with null entries before free, so such readers will either see a NULL pointer or a null filled table being lazy freed.
On allocation, allocate the table when a cluster is used by any order.
This way, we can reduce the memory usage of large swap device significantly.
This idea to dynamically release unused swap cluster data was initially suggested by Chris Li while proposing the cluster swap allocator and it suits the swap table idea very well.
--- mm/swap.h | 2 +- mm/swap_state.c | 9 +-- mm/swap_table.h | 37 ++++++++- mm/swapfile.c | 197 +++++++++++++++++++++++++++++++++++++----------- 4 files changed, 194 insertions(+), 51 deletions(-)
mm/swap.h +1 -1
@@ -36,7 +36,7 @@ struct swap_cluster_info {u16 count;u8 flags;u8 order; -atomic_long_t *table;/* Swap table entries, see mm/swap_table.h */ +atomic_long_t __rcu *table;/* Swap table entries, see mm/swap_table.h */ struct list_head list; };
mm/swap_state.c +4 -5
@@ -91,8 +91,8 @@ struct folio *swap_cache_get_folio(swp_entry_t entry)struct folio *folio;for (;;) { - swp_tb =__swap_table_get (__swap_entry_to_cluster (entry), -swp_cluster_offset (entry)); + swp_tb =swap_table_get (__swap_entry_to_cluster (entry), +swp_cluster_offset (entry));if (!swp_tb_is_folio (swp_tb))return NULL ; folio =swp_tb_to_folio (swp_tb);
@@ -115,11 +115,10 @@ void *swap_cache_get_shadow(swp_entry_t entry) {unsigned long swp_tb; - swp_tb =__swap_table_get (__swap_entry_to_cluster (entry), -swp_cluster_offset (entry)); + swp_tb =swap_table_get (__swap_entry_to_cluster (entry), +swp_cluster_offset (entry));if (swp_tb_is_shadow (swp_tb))return swp_tb_to_shadow (swp_tb); -return NULL ; }
mm/swap_table.h +34 -3
@@ -2,8 +2,15 @@#ifndef _MM_SWAP_TABLE_H#define _MM_SWAP_TABLE_H +#include <linux/rcupdate.h> +#include <linux/atomic.h> #include "swap.h" +/* A typical flat array in each cluster as swap table */ +struct swap_table { +atomic_long_t entries[SWAPFILE_CLUSTER ]; +}; +/* * A swap table entry represents the status of a swap slot on a swap * (physical or virtual) device. The swap table in each cluster is a
@@ -76,22 +83,46 @@ static inline void *swp_tb_to_shadow(unsigned long swp_tb)static inline void __swap_table_set (struct swap_cluster_info *ci,unsigned int off,unsigned long swp_tb) { +atomic_long_t *table =rcu_dereference_protected (ci->table, true); + +lockdep_assert_held (&ci->lock);VM_WARN_ON_ONCE (off >=SWAPFILE_CLUSTER ); -atomic_long_set (&ci->table[off], swp_tb); +atomic_long_set (&table[off], swp_tb); }static inline unsigned long __swap_table_xchg (struct swap_cluster_info *ci,unsigned int off,unsigned long swp_tb) { +atomic_long_t *table =rcu_dereference_protected (ci->table, true); + +lockdep_assert_held (&ci->lock);VM_WARN_ON_ONCE (off >=SWAPFILE_CLUSTER );/* Ordering is guaranteed by cluster lock, relax */ -return atomic_long_xchg_relaxed (&ci->table[off], swp_tb); +return atomic_long_xchg_relaxed (&table[off], swp_tb); }static inline unsigned long __swap_table_get (struct swap_cluster_info *ci,unsigned int off) { +atomic_long_t *table; +VM_WARN_ON_ONCE (off >=SWAPFILE_CLUSTER ); -return atomic_long_read (&ci->table[off]); + table =rcu_dereference_check (ci->table,lockdep_is_held (&ci->lock)); + +return atomic_long_read (&table[off]); +} + +static inline unsigned long swap_table_get (struct swap_cluster_info *ci, +unsigned int off) +{ +atomic_long_t *table; +unsigned long swp_tb; + +rcu_read_lock (); + table =rcu_dereference (ci->table); + swp_tb = table ?atomic_long_read (&table[off]) :null_to_swp_tb (); +rcu_read_unlock (); + +return swp_tb; }#endif
mm/swapfile.c +155 -42
353 lines not on this page · raw message
Signature · 1 line
2.51.0
Re: [PATCH v4 14/15] mm, swap: implement dynamic allocation of swap table
Quoted 16 lines · Kairui Song
From: Kairui Song <[email protected]>
Now swap table is cluster based, which means free clusters can free its table since no one should modify it.
There could be speculative readers, like swap cache look up, protect them by making them RCU protected. All swap table should be filled with null entries before free, so such readers will either see a NULL pointer or a null filled table being lazy freed.
On allocation, allocate the table when a cluster is used by any order.
This way, we can reduce the memory usage of large swap device significantly.
Quoted 7 lines
This idea to dynamically release unused swap cluster data was initially suggested by Chris Li while proposing the cluster swap allocator and it suits the swap table idea very well.
Co-developed-by Chris Li <[email protected]>Signed-off-by Chris Li <[email protected]>Signed-off-by Kairui Song <[email protected]>
Thanks
Barry
Re: [PATCH v4 14/15] mm, swap: implement dynamic allocation of swap table
In reply to Kairui SongHi Kairui,
I assume there are no changes outside of swap_cluster_alloc_table(), if there is please let me know.
I did one more pass mostly focusing on swap_cluster_alloc_table() . Looks fine with minor Nitpicks following.
Quoted 227 lines · Kairui Song
From: Kairui Song <[email protected]>
Now swap table is cluster based, which means free clusters can free its table since no one should modify it.
There could be speculative readers, like swap cache look up, protect them by making them RCU protected. All swap table should be filled with null entries before free, so such readers will either see a NULL pointer or a null filled table being lazy freed.
On allocation, allocate the table when a cluster is used by any order.
This way, we can reduce the memory usage of large swap device significantly.
This idea to dynamically release unused swap cluster data was initially suggested by Chris Li while proposing the cluster swap allocator and it suits the swap table idea very well.
Co-developed-by Chris Li <[email protected]>Signed-off-by Chris Li <[email protected]>Signed-off-by Kairui Song <[email protected]>--- mm/swap.h | 2 +- mm/swap_state.c | 9 +-- mm/swap_table.h | 37 ++++++++- mm/swapfile.c | 197 +++++++++++++++++++++++++++++++++++++----------- 4 files changed, 194 insertions(+), 51 deletions(-)mm/swap.h +1
-1@@ -36,7 +36,7 @@ struct swap_cluster_info {u16 count;u8 flags;u8 order; -atomic_long_t *table;/* Swap table entries, see mm/swap_table.h */ +atomic_long_t __rcu *table;/* Swap table entries, see mm/swap_table.h */ struct list_head list; };mm/swap_state.c +4
-5@@ -91,8 +91,8 @@ struct folio *swap_cache_get_folio(swp_entry_t entry)struct folio *folio;for (;;) { - swp_tb =__swap_table_get (__swap_entry_to_cluster (entry), -swp_cluster_offset (entry)); + swp_tb =swap_table_get (__swap_entry_to_cluster (entry), +swp_cluster_offset (entry));if (!swp_tb_is_folio (swp_tb))return NULL ; folio =swp_tb_to_folio (swp_tb);@@ -115,11 +115,10 @@ void *swap_cache_get_shadow(swp_entry_t entry) {unsigned long swp_tb; - swp_tb =__swap_table_get (__swap_entry_to_cluster (entry), -swp_cluster_offset (entry)); + swp_tb =swap_table_get (__swap_entry_to_cluster (entry), +swp_cluster_offset (entry));if (swp_tb_is_shadow (swp_tb))return swp_tb_to_shadow (swp_tb); -return NULL ; }mm/swap_table.h +34
-3@@ -2,8 +2,15 @@#ifndef _MM_SWAP_TABLE_H#define _MM_SWAP_TABLE_H +#include <linux/rcupdate.h> +#include <linux/atomic.h> #include "swap.h" +/* A typical flat array in each cluster as swap table */ +struct swap_table { +atomic_long_t entries[SWAPFILE_CLUSTER ]; +}; +/* * A swap table entry represents the status of a swap slot on a swap * (physical or virtual) device. The swap table in each cluster is a @@ -76,22 +83,46 @@ static inline void *swp_tb_to_shadow(unsigned long swp_tb)static inline void __swap_table_set (struct swap_cluster_info *ci,unsigned int off,unsigned long swp_tb) { +atomic_long_t *table =rcu_dereference_protected (ci->table, true); + +lockdep_assert_held (&ci->lock);VM_WARN_ON_ONCE (off >=SWAPFILE_CLUSTER ); -atomic_long_set (&ci->table[off], swp_tb); +atomic_long_set (&table[off], swp_tb); }static inline unsigned long __swap_table_xchg (struct swap_cluster_info *ci,unsigned int off,unsigned long swp_tb) { +atomic_long_t *table =rcu_dereference_protected (ci->table, true); + +lockdep_assert_held (&ci->lock);VM_WARN_ON_ONCE (off >=SWAPFILE_CLUSTER );/* Ordering is guaranteed by cluster lock, relax */ -return atomic_long_xchg_relaxed (&ci->table[off], swp_tb); +return atomic_long_xchg_relaxed (&table[off], swp_tb); }static inline unsigned long __swap_table_get (struct swap_cluster_info *ci,unsigned int off) { +atomic_long_t *table; +VM_WARN_ON_ONCE (off >=SWAPFILE_CLUSTER ); -return atomic_long_read (&ci->table[off]); + table =rcu_dereference_check (ci->table,lockdep_is_held (&ci->lock)); + +return atomic_long_read (&table[off]); +} + +static inline unsigned long swap_table_get (struct swap_cluster_info *ci, +unsigned int off) +{ +atomic_long_t *table; +unsigned long swp_tb; + +rcu_read_lock (); + table =rcu_dereference (ci->table); + swp_tb = table ?atomic_long_read (&table[off]) :null_to_swp_tb (); +rcu_read_unlock (); + +return swp_tb; }#endif mm/swapfile.c +36
-11@@ -59,6 +59,9 @@ static void swap_entries_free(struct swap_info_struct *si,static void swap_range_alloc (struct swap_info_struct *si,unsigned int nr_entries);static bool folio_swapcache_freeable (struct folio *folio); +static void move_cluster (struct swap_info_struct *si, +struct swap_cluster_info *ci,struct list_head *list, +enum swap_cluster_flags new_flags);static DEFINE_SPINLOCK (swap_lock);static unsigned int nr_swapfiles;@@ -105,6 +108,8 @@ static DEFINE_SPINLOCK(swap_avail_lock);struct swap_info_struct *swap_info[MAX_SWAPFILES ]; +static struct kmem_cache *swap_table_cachep; +static DEFINE_MUTEX (swapon_mutex);static DECLARE_WAIT_QUEUE_HEAD (proc_poll_wait);@@ -401,10 +406,17 @@ static inline bool cluster_is_discard(struct swap_cluster_info *info)return info->flags ==CLUSTER_FLAG_DISCARD ; } +static inline bool cluster_table_is_alloced (struct swap_cluster_info *ci) +{ +return rcu_dereference_protected (ci->table,lockdep_is_held (&ci->lock)); +} +static inline bool cluster_is_usable (struct swap_cluster_info *ci,int order) {if (unlikely (ci->flags >CLUSTER_FLAG_USABLE ))return false; +if (!cluster_table_is_alloced (ci)) +return false;if (!order)return true;return cluster_is_empty (ci) || order == ci->order;@@ -422,32 +434,90 @@ static inline unsigned int cluster_offset(struct swap_info_struct *si,return cluster_index (si, ci) *SWAPFILE_CLUSTER ; } -static int swap_cluster_alloc_table (struct swap_cluster_info *ci) +static void swap_cluster_free_table (struct swap_cluster_info *ci) { -WARN_ON (ci->table); - ci->table =kzalloc (sizeof (unsigned long ) *SWAPFILE_CLUSTER ,GFP_KERNEL ); -if (!ci->table) -return -ENOMEM ; -return 0 ; +unsigned int ci_off; +struct swap_table *table; + +/* Only empty cluster's table is allow to be freed */ +lockdep_assert_held (&ci->lock); +VM_WARN_ON_ONCE (!cluster_is_empty (ci)); +for (ci_off =0 ; ci_off <SWAPFILE_CLUSTER ; ci_off++) +VM_WARN_ON_ONCE (!swp_tb_is_null (__swap_table_get (ci, ci_off))); + table = (void *)rcu_dereference_protected (ci->table, true); +rcu_assign_pointer (ci->table,NULL ); + +kmem_cache_free (swap_table_cachep, table); } -static void swap_cluster_free_table (struct swap_cluster_info *ci) +/* +* Allocate swap table for one cluster. Attempt an atomic allocation first, +* then fallback to sleeping allocation. +*/ +static struct swap_cluster_info * +swap_cluster_alloc_table (struct swap_info_struct *si, +struct swap_cluster_info *ci) { -unsigned int ci_off; -unsigned long swp_tb; +struct swap_table *table; -if (!ci->table) -return ; +/* +* Only cluster isolation from the allocator does table allocation. +* Swap allocator uses percpu clusters and holds the local lock.
mm/swapfile.c static inline unsigned int cluster_offset(struct swap_info_struct *si, · line 475
Quoted 10 lines
mm/swapfile.c +10
-0+*/ +lockdep_assert_held (&ci->lock); +lockdep_assert_held (&this_cpu_ptr (&percpu_swap_cluster)->lock); + +/* The cluster must be free and was just isolated from the free list. */ +VM_WARN_ON_ONCE (ci->flags || !cluster_is_empty (ci)); + + table =kmem_cache_zalloc (swap_table_cachep, + __GFP_HIGH | __GFP_NOMEMALLOC | __GFP_NOWARN); +if (table) {
Nit: this can goto the same shared exit path exit with rcu_assign_pointer(ci->table, table) at the end of the function. It looks cleaner and possibly generates better code because the compiler doesn't generate two rcu_assign_pointer() in this function.
mm/swapfile.c static inline unsigned int cluster_offset(struct swap_info_struct *si, · line 486
Quoted 11 lines
mm/swapfile.c +11
-0+rcu_assign_pointer (ci->table, table); +return ci; + } + +/* +* Try a sleep allocation. Each isolated free cluster may cause +* a sleep allocation, but there is a limited number of them, so +* the potential recursive allocation is limited. +*/ +spin_unlock (&ci->lock); +if (!(si->flags &SWP_SOLIDSTATE ))
Nit: si->flags & SWP_SOLIDSTATE should never change. However, it is better to not make assumptions about it. You'd better cache this into a local variable so that spin_unlock() and later spin_lock() are paired.
Quoted 6 lines
mm/swapfile.c +6
-0+spin_unlock (&si->global_cluster_lock); +local_unlock (&percpu_swap_cluster.lock); + + table =kmem_cache_zalloc (swap_table_cachep, + __GFP_HIGH | __GFP_NOMEMALLOC |GFP_KERNEL ); +
mm/swapfile.c static inline unsigned int cluster_offset(struct swap_info_struct *si, · line 502
Quoted 10 lines
mm/swapfile.c +10
-0+/* +* Back to atomic context. We might have migrated to a new CPU with a +* usable percpu cluster. But just keep using the isolated cluster to +* make things easier. Migration indicates a slight change of workload +* so using a new free cluster might not be a bad idea, and the worst +* could happen with ignoring the percpu cluster is fragmentation, +* which is acceptable since this fallback and race is rare. +*/ +local_lock (&percpu_swap_cluster.lock); +if (!(si->flags &SWP_SOLIDSTATE ))
Nit: here you should use the cached local variable rather than testing the !(si->flags & SWP_SOLIDSTATE) again. It is much easier to notice that the same condition was used if using the same variable as the test condition.
mm/swapfile.c static inline unsigned int cluster_offset(struct swap_info_struct *si, · line 510
Quoted 13 lines
mm/swapfile.c +7
-5+spin_lock (&si->global_cluster_lock); +spin_lock (&ci->lock); -for (ci_off =0 ; ci_off <SWAPFILE_CLUSTER ; ci_off++) { - swp_tb =__swap_table_get (ci, ci_off); -if (!swp_tb_is_null (swp_tb)) -pr_err_once ("swap: unclean swap space on swapoff: 0x%lx" , - swp_tb); +/* Nothing except this helper should touch a dangling empty cluster. */ +if (WARN_ON_ONCE (cluster_table_is_alloced (ci))) { +if (table) +kmem_cache_free (swap_table_cachep, table); +return ci;
Nit: consider goto the same exit point for return.
Right now there is no further clean up required before returning ci. Share the exit path, e.g. goto done label is in general better than shortcut the return path here for complex functions.
mm/swapfile.c static inline unsigned int cluster_offset(struct swap_info_struct *si, · line 516
Quoted 8 lines
mm/swapfile.c +4
-2} -kfree (ci->table); - ci->table =NULL ; +if (!table) { +move_cluster (si, ci, &si->free_clusters,CLUSTER_FLAG_FREE ); +spin_unlock (&ci->lock); +return NULL ;
Nit: same for here, you can set ci = NULL then goto the done label.
mm/swapfile.c static inline unsigned int cluster_offset(struct swap_info_struct *si, · line 518
Quoted 2 lines
mm/swapfile.c +2
-0+ } +
Nit: Might need another assign table label here.
mm/swapfile.c static inline unsigned int cluster_offset(struct swap_info_struct *si, · line 519
Quoted 1 line
mm/swapfile.c +1
-0+rcu_assign_pointer (ci->table, table);
Nit: I would put the "done:" label here. You are welcome to use a different label name.
Quoted 233 lines
mm/swapfile.c +68
-25+return ci; } static void move_cluster(struct swap_info_struct *si,@@ -479,7 +549,7 @@ static void swap_cluster_schedule_discard(struct swap_info_struct *si,static void __free_cluster (struct swap_info_struct *si,struct swap_cluster_info *ci) { -lockdep_assert_held (&ci->lock); +swap_cluster_free_table (ci);move_cluster (si, ci, &si->free_clusters,CLUSTER_FLAG_FREE ); ci->order =0 ; }@@ -494,15 +564,11 @@ static void __free_cluster(struct swap_info_struct *si, struct swap_cluster_info* this returns NULL for an non-empty list. */ static struct swap_cluster_info *isolate_lock_cluster ( -struct swap_info_struct *si,struct list_head *list) +struct swap_info_struct *si,struct list_head *list,int order) { -struct swap_cluster_info *ci, *ret =NULL ; +struct swap_cluster_info *ci, *found =NULL ;spin_lock (&si->lock); - -if (unlikely (!(si->flags &SWP_WRITEOK ))) - goto out; -list_for_each_entry (ci, list, list) {if (!spin_trylock (&ci->lock))continue ;@@ -514,13 +580,19 @@ static struct swap_cluster_info *isolate_lock_cluster(list_del (&ci->list); ci->flags =CLUSTER_FLAG_NONE ; - ret = ci; + found = ci;break ; } -out:spin_unlock (&si->lock); -return ret; +if (found && !cluster_table_is_alloced (found)) { +/* Only an empty free cluster's swap table can be freed. */ +VM_WARN_ON_ONCE (list != &si->free_clusters); +VM_WARN_ON_ONCE (!cluster_is_empty (found)); +return swap_cluster_alloc_table (si, found); + } + +return found; }/* @@ -653,17 +725,27 @@ static void relocate_cluster(struct swap_info_struct *si,* added to free cluster list and its usage counter will be increased by 1. * Only used for initialization. */ -static void inc_cluster_info_page (struct swap_info_struct *si, +static int inc_cluster_info_page (struct swap_info_struct *si,struct swap_cluster_info *cluster_info,unsigned long page_nr) {unsigned long idx = page_nr /SWAPFILE_CLUSTER ; +struct swap_table *table;struct swap_cluster_info *ci; ci = cluster_info + idx; +if (!ci->table) { + table =kmem_cache_zalloc (swap_table_cachep,GFP_KERNEL ); +if (!table) +return -ENOMEM ; +rcu_assign_pointer (ci->table, table); + } + ci->count++;VM_BUG_ON (ci->count >SWAPFILE_CLUSTER );VM_BUG_ON (ci->flags); + +return 0 ; } static bool cluster_reclaim_range(struct swap_info_struct *si,@@ -845,7 +927,7 @@ static unsigned int alloc_swap_scan_list(struct swap_info_struct *si,unsigned int found =SWAP_ENTRY_INVALID ; do { - struct swap_cluster_info *ci = isolate_lock_cluster(si, list); + struct swap_cluster_info *ci = isolate_lock_cluster(si, list, order); unsigned long offset; if (!ci)@@ -870,7 +952,7 @@ static void swap_reclaim_full_clusters(struct swap_info_struct *si, bool force)if (force) to_scan =swap_usage_in_pages (si) /SWAPFILE_CLUSTER ; -while ((ci =isolate_lock_cluster (si, &si->full_clusters))) { +while ((ci =isolate_lock_cluster (si, &si->full_clusters,0 ))) { offset =cluster_offset (si, ci); end =min (si->max, offset +SWAPFILE_CLUSTER ); to_scan--;@@ -1018,6 +1100,7 @@ static unsigned long cluster_alloc_swap_entry(struct swap_info_struct *si, int o done:if (!(si->flags &SWP_SOLIDSTATE ))spin_unlock (&si->global_cluster_lock); +return found; }@@ -1885,7 +1968,13 @@ swp_entry_t get_swap_page_of_type(int type)/* This is called for allocating swap entry, not cache */ if (get_swap_device_info (si)) {if (si->flags &SWP_WRITEOK ) { +/* +* Grab the local lock to be complaint +* with swap table allocation. +*/ +local_lock (&percpu_swap_cluster.lock); offset =cluster_alloc_swap_entry (si,0 ,1 ); +local_unlock (&percpu_swap_cluster.lock);if (offset) { entry =swp_entry (si->type, offset);atomic_long_dec (&nr_swap_pages);@@ -2679,12 +2768,21 @@ static void wait_for_allocation(struct swap_info_struct *si)static void free_cluster_info (struct swap_cluster_info *cluster_info,unsigned long maxpages) { +struct swap_cluster_info *ci;int i, nr_clusters =DIV_ROUND_UP (maxpages,SWAPFILE_CLUSTER );if (!cluster_info)return ; -for (i =0 ; i < nr_clusters; i++) -swap_cluster_free_table (&cluster_info[i]); +for (i =0 ; i < nr_clusters; i++) { + ci = cluster_info + i; +/* Cluster with bad marks count will have a remaining table */ +spin_lock (&ci->lock); +if (rcu_dereference_protected (ci->table, true)) { + ci->count =0 ; +swap_cluster_free_table (ci); + } +spin_unlock (&ci->lock); + }kvfree (cluster_info); }@@ -2720,6 +2818,7 @@ SYSCALL_DEFINE1(swapoff, const char __user *, specialfile)struct address_space *mapping;struct inode *inode;struct filename *pathname; +unsigned int maxpages;int err, found =0 ;if (!capable (CAP_SYS_ADMIN ))@@ -2826,8 +2925,8 @@ SYSCALL_DEFINE1(swapoff, const char __user *, specialfile) p->swap_map =NULL ; zeromap = p->zeromap; p->zeromap =NULL ; + maxpages = p->max; cluster_info = p->cluster_info; -free_cluster_info (cluster_info, p->max); p->max =0 ; p->cluster_info =NULL ;spin_unlock (&p->lock);@@ -2839,6 +2938,7 @@ SYSCALL_DEFINE1(swapoff, const char __user *, specialfile) p->global_cluster =NULL ;vfree (swap_map);kvfree (zeromap); +free_cluster_info (cluster_info, maxpages);/* Destroy swap account information */ swap_cgroup_swapoff (p->type);@@ -3217,11 +3317,8 @@ static struct swap_cluster_info *setup_clusters(struct swap_info_struct *si,if (!cluster_info) goto err; -for (i =0 ; i < nr_clusters; i++) { +for (i =0 ; i < nr_clusters; i++)spin_lock_init (&cluster_info[i].lock); -if (swap_cluster_alloc_table (&cluster_info[i])) - goto err_free; - }if (!(si->flags &SWP_SOLIDSTATE )) { si->global_cluster = kmalloc(sizeof(*si->global_cluster),@@ -3240,16 +3337,23 @@ static struct swap_cluster_info *setup_clusters(struct swap_info_struct *si,* See setup_swap_map(): header page, bad pages, * and the EOF part of the last cluster. */ -inc_cluster_info_page (si, cluster_info,0 ); + err =inc_cluster_info_page (si, cluster_info,0 ); +if (err) + goto err;for (i =0 ; i < swap_header->info.nr_badpages; i++) {unsigned int page_nr = swap_header->info.badpages[i];if (page_nr >= maxpages)continue ; -inc_cluster_info_page (si, cluster_info, page_nr); + err =inc_cluster_info_page (si, cluster_info, page_nr); +if (err) + goto err; + } +for (i = maxpages; i <round_up (maxpages,SWAPFILE_CLUSTER ); i++) { + err =inc_cluster_info_page (si, cluster_info, i); +if (err) + goto err; } -for (i = maxpages; i <round_up (maxpages,SWAPFILE_CLUSTER ); i++) -inc_cluster_info_page (si, cluster_info, i);INIT_LIST_HEAD (&si->free_clusters);INIT_LIST_HEAD (&si->full_clusters);@@ -3963,6 +4067,15 @@ static int __init swapfile_init(void) swapfile_maximum_size =arch_max_swapfile_size (); +/* +* Once a cluster is freed, it's swap table content is read +* only, and all swap cache readers (swap_cache_*) verifies +* the content before use. So it's safe to use RCU slab here. +*/ + swap_table_cachep =kmem_cache_create ("swap_table" , +sizeof (struct swap_table ), +0 ,SLAB_PANIC |SLAB_TYPESAFE_BY_RCU ,NULL ); +#ifdef CONFIG_MIGRATION if (swapfile_maximum_size >= (1UL <<SWP_MIG_TOTAL_BITS )) swap_migration_ad_supported = true; --2.51.0
PATCH v4 15/15 mm, swap: use a single page for swap table when the size fits
In reply to Kairui SongFrom: Kairui Song <[email protected]>
We have a cluster size of 512 slots. Each slot consumes 8 bytes in swap table so the swap table size of each cluster is exactly one page (4K).
If that condition is true, allocate one page direct and disable the slab cache to reduce the memory usage of swap table and avoid fragmentation.
--- mm/swap_table.h | 2 ++ mm/swapfile.c | 51 +++++++++++++++++++++++++++++++++++++++---------- 2 files changed, 43 insertions(+), 10 deletions(-)
mm/swap_table.h +2 -0
@@ -11,6 +11,8 @@ struct swap_table {atomic_long_t entries[SWAPFILE_CLUSTER ]; }; +#define SWP_TABLE_USE_PAGE (sizeof(struct swap_table) == PAGE_SIZE) +/* * A swap table entry represents the status of a swap slot on a swap * (physical or virtual) device. The swap table in each cluster is a
mm/swapfile.c +41 -10
93 lines not on this page · raw message
Signature · 1 line
2.51.0
Re: [PATCH v4 15/15] mm, swap: use a single page for swap table when the size fits
Quoted 6 lines · Kairui Song
From: Kairui Song <[email protected]>
We have a cluster size of 512 slots. Each slot consumes 8 bytes in swap table so the swap table size of each cluster is exactly one page (4K).
Quoted 7 lines
If that condition is true, allocate one page direct and disable the slab cache to reduce the memory usage of swap table and avoid fragmentation.
Co-developed-by Chris Li <[email protected]>Signed-off-by Chris Li <[email protected]>Signed-off-by Kairui Song <[email protected]>Acked-by Chris Li <[email protected]>
looks good to me, with just a minor nit noted below.
mm/swapfile.c static inline unsigned int cluster_offset(struct swap_info_struct *si, · line 467
Quoted 11 lines
---
mm/swapfile.c +10
-0+static void swap_table_free (struct swap_table *table) +{ +if (!SWP_TABLE_USE_PAGE ) { +kmem_cache_free (swap_table_cachep, table); +return ; + } + +call_rcu (&(folio_page (virt_to_folio (table),0 )->rcu_head), + swap_table_free_folio_rcu_cb); +}
Can’t it simply be:
virt_to_page(table)->rcu_head ?
Thanks
Barry
Re: [PATCH v4 15/15] mm, swap: use a single page for swap table when the size fits
Quoted 11 lines · Barry Song
Quoted 9 lines · Kairui Song
From: Kairui Song <[email protected]>
We have a cluster size of 512 slots. Each slot consumes 8 bytes in swap table so the swap table size of each cluster is exactly one page (4K).
If that condition is true, allocate one page direct and disable the slab cache to reduce the memory usage of swap table and avoid fragmentation.
Quoted 8 lines
Quoted 4 lines
Co-developed-by Chris Li <[email protected]>Signed-off-by Chris Li <[email protected]>Signed-off-by Kairui Song <[email protected]>Acked-by Chris Li <[email protected]>looks good to me, with just a minor nit noted below.
Reviewed-by Barry Song <[email protected]>
Thanks
Quoted 15 lines
mm/swapfile.c static inline unsigned int cluster_offset(struct swap_info_struct *si, · line 467
Quoted 12 lines
---
mm/swapfile.c +10
-0+static void swap_table_free (struct swap_table *table) +{ +if (!SWP_TABLE_USE_PAGE ) { +kmem_cache_free (swap_table_cachep, table); +return ; + } + +call_rcu (&(folio_page (virt_to_folio (table),0 )->rcu_head), + swap_table_free_folio_rcu_cb); +}Can’t it simply be:
virt_to_page(table)->rcu_head ?
I'm not sure if we prefer folios when possible?
Either way is fine, they are identical for this use case.
Quoted 2 lines
Thanks
Barry
Re: [PATCH v4 15/15] mm, swap: use a single page for swap table when the size fits
Quoted 16 lines
Quoted 15 lines
mm/swapfile.c static inline unsigned int cluster_offset(struct swap_info_struct *si, · line 467
Quoted 12 lines
---
mm/swapfile.c +10
-0+static void swap_table_free (struct swap_table *table) +{ +if (!SWP_TABLE_USE_PAGE ) { +kmem_cache_free (swap_table_cachep, table); +return ; + } + +call_rcu (&(folio_page (virt_to_folio (table),0 )->rcu_head), + swap_table_free_folio_rcu_cb); +}Can’t it simply be:
virt_to_page(table)->rcu_head ?
Quoted 1 line
I'm not sure if we prefer folios when possible?
Not sure either, but even Mathew still sometimes suggests using PAGE:
https://lore.kernel.org/lkml/[email protected]/
BTW, "(folio_page(folio), 0)" looks quite strange. I mean, why is it 0 instead of 1?
Although we know it’s always one PAGE, 0 still seems strange.
Quoted 3 lines
Either way is fine, they are identical for this use case.
Thanks
Barry
Re: [PATCH v4 15/15] mm, swap: use a single page for swap table when the size fits
Quoted 19 lines · Barry Song
Quoted 17 lines
Quoted 16 lines
mm/swapfile.c static inline unsigned int cluster_offset(struct swap_info_struct *si, · line 467
Quoted 13 lines
---
mm/swapfile.c +10
-0+static void swap_table_free (struct swap_table *table) +{ +if (!SWP_TABLE_USE_PAGE ) { +kmem_cache_free (swap_table_cachep, table); +return ; + } + +call_rcu (&(folio_page (virt_to_folio (table),0 )->rcu_head), + swap_table_free_folio_rcu_cb); +}Can’t it simply be:
virt_to_page(table)->rcu_head ?Quoted 1 line
I'm not sure if we prefer folios when possible?
Quoted 7 lines
Not sure either, but even Mathew still sometimes suggests using PAGE:
https://lore.kernel.org/lkml/[email protected]/
BTW, "(
folio_page(folio), 0)" looks quite strange. I mean, why is it 0 instead of 1?
Although we know it’s always one PAGE, 0 still seems strange.
I mean, we should always free an entire folio, right?
It’s impossible to free just one page from a folio unless we split a THP, etc.
So it seems more correct that rcu_head should also be promoted to the folio.
/** @rcu_head: You can use this to free a folio by RCU. */ struct rcu_head rcu_head;
Definitely outside the scope of this patchset, so not a blocker :-)
Quoted 5 lines
Quoted 4 lines
Either way is fine, they are identical for this use case.
Thanks
Barry
Hugh Dickins
maintainerQuoted 7 lines · Kairui Song
V4 changes:
- Patch 14: fix potential cluster leak when attemp a sleep allocation of swap table: Just remove the logic that check and return the percpu cluster, it was trying to avoid fragmentation, which wasn't very successfully and may not work at all if there are multiple devices. The fragmentation is not a serious issue and given the chance of hitting that race is extremely low, let's just ignore that [ Chris Meson ].
And in Patch 11 I can see that you've made an important fix to the !folio_ref_freeze() block in __folio_migrate_mapping(): there had been a swap_cluster_unlock(ci) which you've fixed to swap_cluster_unlock_irq(ci):
I was going to report that, with the WARNING and "BUG: sleeping function"s it caused, but you're ahead of me.
Thanks also for fixing the shmem_replace_folio() locking and stats update, which I'd noticed too (but never hit): looks good now.
Hugh
Quoted 6 lines
- Patch 8 & 9: move some changes from Patch 9 to Patch 8 to avoid build error, no code change. [ Baolin Wang ]
- Patch 9: Fix locking section issue, should protect the shmem statistic update with spin lock irq. Also fix an warn on condition.
Link to V3:
- https://lore.kernel.org/linux-mm/[email protected]/
Quoted 10 lines · Hugh Dickins
Quoted 7 lines · Kairui Song
V4 changes:
- Patch 14: fix potential cluster leak when attemp a sleep allocation of swap table: Just remove the logic that check and return the percpu cluster, it was trying to avoid fragmentation, which wasn't very successfully and may not work at all if there are multiple devices. The fragmentation is not a serious issue and given the chance of hitting that race is extremely low, let's just ignore that [ Chris Meson ].
Quoted 5 lines
And in Patch 11 I can see that you've made an important fix to the !folio_ref_freeze() block in
__folio_migrate_mapping(): there had been aswap_cluster_unlock(ci)which you've fixed toswap_cluster_unlock_irq(ci):
I was going to report that, with the WARNING and "BUG: sleeping function"s it caused, but you're ahead of me.
Oh thanks for pointing that out, I fixed that some time ago and forgot to include it in the change log.
Quoted 12 lines
Thanks also for fixing the
shmem_replace_folio()locking and stats update, which I'd noticed too (but never hit): looks good now.Hugh
Quoted 6 lines
- Patch 8 & 9: move some changes from Patch 9 to Patch 8 to avoid build error, no code change. [ Baolin Wang ]
- Patch 9: Fix locking section issue, should protect the shmem statistic update with spin lock irq. Also fix an warn on condition.
Link to V3:
- https://lore.kernel.org/linux-mm/[email protected]/