mm, swap: extendable swap devices (xswap phase I)
Subsystems (6)
Baoquan He
reviewerv5 cover mm, swap: extendable swap devices (xswap phase I)
From: Baoquan He <[email protected]>
A normal swap device has a fixed size, set when it is enabled. This does not fit zswap well. With zswap the swapped data stays in RAM in compressed form, so how much swap a machine can really use depends on how well the workload compresses, not on a number we pick before we know the workload. Make the device large enough for the worst case and most of it is never used. Make it just large enough for the average case and the machine runs out of swap while zswap still has free room.
xswap is a swap device with no backing file. In this phase its swap entries hold pages in zswap, and its cluster_info array lives in a VM_SPARSE area that is mapped a chunk at a time, so the device covers a large swap address space and costs only the chunks it has actually mapped. The device grows when the workload needs it and gives the tail back when it does not.
This series is the device itself: the sparse cluster array, growth and shrink, and the interfaces that create and destroy one. It is phase I of four; the physical backend, the swap accounting, and the swap-table backend records (zswap w/o xarray) build on it.
Design notes
------------
In this phase the device is only an swap address space. si->max is set when the device is created and does not change for its lifetime; what changes is how much of it is mapped. swap_info_struct carries nr_clusters_max (the swap address space) and nr_clusters_mapped (the mapped prefix), and the two walkers that touch cluster_info are bounded by the latter.
Growth starts when 85% of the mapped range is in use and maps until 65%. The call comes from the cluster allocation path. Growing maps pages into the VM_SPARSE area, and mapping can sleep, so it cannot be done inside the allocation that would otherwise fail: the swap-out path would then have nothing to fall back on. A grow maps at most XSWAP_GROW_CHUNKS chunks at a time.
Shrink gives the free tail back when the mapped range is at most 50% in use and stops at 70%. It is triggered when a cluster becomes completely free, from a workqueue, because it takes xswap_lock and sleeps in synchronize_rcu() and vm_area_unmap_pages(). Shrinking on a smaller drop is not useful: growth happens on demand, so the same clusters would just be mapped again, and every unmap also costs an RCU grace period. The two target points sit inside each other's threshold, so a grow cannot leave the range in a state the shrink would undo, and the other way around.
Configuration
-------------
The device is behind CONFIG_XSWAP, which depends on SWAP, 64BIT, ZSWAP and SYSFS. CONFIG_XSWAP_DEFAULT_MAX sets the default size of a new device; it takes a percentage of RAM, a size, or "max", and defaults to "200%":
CONFIG_XSWAP_DEFAULT_MAX="200%"
xswap.max= on the kernel command line overrides it:
xswap.max=8G 8 GiB xswap.max=300% three times RAM xswap.max=max the largest size the device can address
"max" stops at the swap offset limit, capped so that a cluster index fits in an unsigned int. A slot is still a swap entry, so that limit applies even though the data never goes to a physical address.
The swap address space cannot move once the device exists, so the size is settled at creation. That is why this is a boot parameter and not a runtime knob.
Identity
--------
A device needs a name that survives while it exists. The swap_info type is a slot in swap_info[] and is reused after swapoff, so it cannot be one. Each device instead takes an id from a per-boot counter, assigned at creation and never reused, and that id is what /proc/swaps shows:
Filename Type Size Used Priority xswap1 xswap 10485756 0 32767
The same id names the debugfs file and is what destroy takes:
# cat /sys/kernel/debug/xswap/xswap1 type 0 clusters_max 5120 clusters_mapped 73 usage_pages 0 tail_free 72 grows 0 shrinks 0 # echo 1 > /sys/kernel/mm/xswap/destroy
A file-less device has no swapon, so /sys/kernel/mm/xswap/create makes one. The only argument it takes is a priority; the size comes from xswap.max=. A priority can be given per device while a size cannot:
# echo > /sys/kernel/mm/xswap/create default priority # echo 100 > /sys/kernel/mm/xswap/create priority 100
The default is the highest priority, SWAP_FLAG_PRIO_MASK, so an xswap device is tried first, ahead of any ordinary swap device. Setting a lower priority is allowed.
The teardown of a device is shared with sys_swapoff() rather than duplicated.
swapoff
-------
sys_swapoff() opens the path first. An xswap device has no file, so the name shown in /proc/swaps cannot be opened. Only a -ENOENT from that open is taken for an xswap device, and the device with that name is torn down. A real file or directory keeps precedence: a path that fails for another reason is never matched. This is what makes both of these work:
# swapoff xswap1 # swapoff -a
The name is matched exactly, and /proc/swaps prints the one to use: a path such as ./xswap1 is not the bare name and is not recognized.
The device needs zswap. Without it swapout takes a swap entry and frees no memory, so creating one fails.
Limitations
-----------
This phase has no physical backend. If zswap refuses a page, the folio stays in memory, so an xswap-only system can run out of memory once the zswap pool is full. Keep a real swap device on such systems; phase II adds the physical backend.
xswap does not allocate large folios in this phase. The allocation walk stops at the first device for a large folio, and a new xswap device defaults to the highest priority, so the folio is offered to xswap, rejected there, and split into base pages; a lower-priority block device never gets it whole. Creating the xswap device below a block device in priority avoids this, at the cost of offering every page to the block device first. mTHP and THP swapout are both affected; the data is still swapped, in base pages. A bypass in the walk would avoid the split, but the next phase teaches xswap to take large folios itself, on a contiguous physical run, and would remove the bypass again; this phase leaves the walk unchanged.
An xswap entry stored while no real swap device exists is not put on the zswap LRU, so adding a real device later does not make it reclaimable. Phase II puts every xswap entry on the LRU.
Testing
-------
qemu KVM guest, 8G RAM, booted with xswap.max=10G. memhog is a small local helper: it faults <total_gb> of anon, fills it with a fixed pattern, and holds it.
Set up zswap and create a device:
# echo 1 > /sys/module/zswap/parameters/enabled # echo > /sys/kernel/mm/xswap/create # awk 'NR == 1 || /xswap/' /proc/swaps Filename Type Size Used Priority xswap1 xswap 10485756 0 32767
An idle device maps one chunk of cluster_info: 73 clusters with the current struct size.
Swap more anon than RAM through a small cgroup:
# mkdir -p /sys/fs/cgroup/xswap_limit # echo 2G > /sys/fs/cgroup/xswap_limit/memory.max # echo max > /sys/fs/cgroup/xswap_limit/memory.swap.max # ( echo $BASHPID > /sys/fs/cgroup/xswap_limit/cgroup.procs # exec env MEMHOG_FILL=pattern ./memhog 11 600 ) &
clusters_mapped grows to cover the pages that go to zswap (2,414,630 pages against 2,038,301 of RAM in one run), while SwapTotal stays at 10485756. Throughout, SwapTotal never moves and SwapFree stays in [0, SwapTotal]. Pushing past the size leaves the cgroup out of room and the OOM killer takes the workload, which is the pass signal there.
Growth and shrink, measured with two cgroups so that one keeps its pages while the other goes away:
idle mapped=73 occ=0.000 grows=0 shrinks=0 2G in a 1G cgroup mapped=657 occ=0.799 grows=7 shrinks=0 3G more in a second 1G cgroup mapped=1898 occ=0.829 grows=12 shrinks=0 kill the second cgroup mapped=803 occ=0.654 grows=12 shrinks=1
The second cgroup is started later, so its clusters sit at the tail, and killing it leaves a free tail to take back. The first stays alive, so usage falls but not to zero: that is the state the shrink has to land in. occ (occupancy) is usage over the mapped range, which is what the grow and shrink thresholds are expressed in.
With the first cgroup still alive, destroy the device:
# echo 1 > /sys/kernel/mm/xswap/destroy
Also tested these, and nothing crashed or warned:
- a shrink racing a swapoff of the same device, 20 rounds
- a cgroup that cannot zswap: it gets no xswap slot, and the allocator walk does not spin
- 2000 create/destroy cycles: no per-cluster state is leaked, and the ids keep increasing
- destroy with the workload alive, and a recreate at the same size
- swapoff -a and swapoff by name both tear the device down
The whole series has also been through the later phases' tests, which need the physical backend and the swap accounting; those results will be reported with the phases that own them. Nothing crashed or warned there either:
- a destroy returns the physical slots it handed out to their block device, and a THP swapout leaves the xswap device and its backend with the same usage
- a swapped THP reads back as one unit from the physical backend, and readback through zswap and through the backend both pass a byte-for-byte pattern check
- the memcg swap accounting, memory.swap.current against the backend usage, holds across charge, uncharge and destroy
Note on performance
-------------------
This phase is the device and leaves the swap store path alone: an xswap slot still goes through zswap's xarray. What it adds is covered by the growth and shrink behaviour above, not by throughput.
When an xswap device exists, the allocation-time zswap gate runs on every swapout, including the pages that end up on a real device, and in this phase its cgroup check can flush the cgroup's zswap statistics. A later phase replaces it with a lockless check.
The series as a whole (phase I+II+III+IV) has been measured on a 288-CPU server with 1.1 TiB of RAM. The numbers are repeated here only to show where this leads; each later phase will report the measurements that belong to it.
- Release path. usemem -O -y -x -n 1 20G, ten rounds, in a cgroup limited to 14G of memory and 12G of swap. The swap-out volumes are the same in both arms, 18,370,771 and 18,370,836 pages:
baseline kernel, zram device 1,067,082 us
this series, xswap device 922,396 us -13.6%
- Steady-state throughput. The same usemem runs. Per-run medians:
baseline kernel, zram device 338,260 - 338,760 KB/s
this series, xswap device 340,997 - 341,885 KB/s
The two ranges do not overlap: every pairing of a baseline run with a series run is 0.7% to 1.1% ahead, about 1% on average.
- General workload. A full kernel build, x86_64 defconfig and -j144, five rounds per kernel, best of:
baseline kernel 2,572.6 s CPU (user+sys)
this series 2,577.7 s CPU (user+sys)
The 0.2% difference is smaller than the run-to-run spread of either kernel.
The large gain is in the release path. How it splits between the swap-table backend records, the device itself and the accounting is what the later phases report, each with the measurement that belongs to it.
Overcommit
----------
An xswap device advertises its whole address space as swap: si->pages is what /proc/swaps and SwapTotal report, and what nr_swap_pages uses to decide that anonymous memory can be reclaimed. The backing is the zswap pool in this phase, and real swap devices in the later phases; the address space is not backing store in either case, yet it raises the commit limit that __vm_enough_memory() enforces.
Which semantics this should follow is not settled here. Counting the address space lets a workload commit 2xRAM it can never back; not counting it changes what SwapTotal means. swap_info_remove() skips its commit-limit probe for xswap, so destroying a device does not fail on headroom the device never provided, and real swap devices keep the probe. The limit can then fall below Committed_AS; new allocations fail until Committed_AS drops. This is left to a follow-up, together with the rest of the swap accounting.
Changelog
=========
v4 -> v5:
- Rebased onto the latest mm-new (2c5dafa7f289).
- A device now has a stable id instead of exposing the swap type:
/proc/swaps shows xswap<id>, the debugfs file is named by the id and prints the type as well, and destroy takes the id. The id is assigned per boot and never reused, so a value read earlier cannot name a different device.
- swapoff accepts the name: swapoff xswap<id> and swapoff -a tear the device down, while a real file or directory keeps precedence. A name that does not exist reports -ENOENT; the sysfs destroy keeps -ENODEV for an id it does not know. The interfaces are now documented in Documentation/ABI/testing/sysfs-kernel-mm-xswap and zswap.rst.
- CONFIG_XSWAP_DEFAULT_MAX sets the default size of a new device. It takes a percentage of RAM, a size, or "max", and defaults to "200%"; xswap.max= still overrides it.
- The cluster_info accesses are RCU-safe: the allocation fast path bounds a cached cluster against the mapped range, and the shrink drains the per-cpu cache and waits an RCU grace period before unmapping.
- xswap_grow() now links the clusters it mapped even when swapoff has already cleared SWP_WRITEOK, so a failed swapoff cannot leave them mapped and on no list.
- swap_info_remove() skips the commit limit probe for xswap; real swap devices keep it. See the overcommit note above.
- The cgroup zswap gate is checked again after the allocation when an xswap device appeared after the initial lookup, so a slot is not kept for a cgroup that cannot zswap.
- The patches are reorganized: the VM_SPARSE work is split into helpers and the xswap use, the max/pages widening moves before the sysfs interface, the shrink trigger now carries the workqueue deferral, and the unmap helper is prepared for range shrinking in its own patch. The series is now phase I of four, with the swap-table backend records as a phase of their own.
Baoquan He (18):
mm, swap: refactor free_swap_cluster_info to take swap_info_struct mm, swap: add VM_SPARSE cluster-array helpers for xswap mm, swap: use a VM_SPARSE cluster array for xswap devices mm, swap: widen swap_info_struct max/pages to unsigned long mm, swap: add a sysfs interface to create xswap devices mm, swap: refactor swapoff and add xswap_destroy
mm, swap: require zswap for xswap devices
mm, swap: use a stable id for xswap devices
mm, swap: let the command line set the xswap device size mm, swap: add CONFIG_XSWAP_DEFAULT_MAX
mm, swap: let swapoff destroy xswap devices by name
mm, swap: add xswap grow trigger on cluster allocation mm, swap: add debugfs counters for xswap
mm, swap: make xswap cluster_info accesses RCU-safe
mm, swap: prepare xswap_unmap_clusters for range shrinking mm, swap: add xswap shrink trigger and defer it to a workqueue mm, swap: do not give an xswap slot to a cgroup that cannot zswap mm, swap: skip the overcommit probe when removing xswap
Chris Li (1):
mm: xswap support for zswap
.../ABI/testing/sysfs-kernel-mm-xswap | 43 + .../admin-guide/kernel-parameters.txt | 8 + Documentation/admin-guide/mm/zswap.rst | 9 + MAINTAINERS | 1 + include/linux/swap.h | 17 +- mm/Kconfig | 20 + mm/page_io.c | 19 + mm/swap_state.c | 9 + mm/swapfile.c | 1569 +++++++++++++++-- mm/zswap.c | 11 +- 10 files changed, 1593 insertions(+), 113 deletions(-) create mode 100644 Documentation/ABI/testing/sysfs-kernel-mm-xswap
base-commit: 2c5dafa7f289d760f4492260b0988029aab478b0
Signature · 1 line
2.54.0
Baoquan He
reviewerPATCH v5 1/19 mm: xswap support for zswap
From: Chris Li <[email protected]>
Introduce extendable swap device support - xswap.
An xswap device has no backing storage and no swap data section, so it wastes no disk space. Creation is via a sysfs interface added in a later patch.
Zswap writeback is gated on whether a real (non-xswap) swap device is active. nr_real_swapfiles counts such devices and is maintained at swapon/swapoff only, so the gate reflects "a device exists to write back to" rather than "a device currently has free slots". This keeps writeback working even when the real swap device is full, and avoids a double decrement when a full device is swapped off.
An xswap entry is not added to the zswap writeback LRU: with no backing store there is nothing to write it back to. zswap_lru_del() tolerates that, since an entry that was never added is on no list.
--- include/linux/swap.h | 2 ++ mm/page_io.c | 19 +++++++++++++++++++ mm/swap_state.c | 9 +++++++++ mm/swapfile.c | 37 +++++++++++++++++++++++++++++++------ mm/zswap.c | 11 +++++++++-- 5 files changed, 70 insertions(+), 8 deletions(-)
include/linux/swap.h +2 -0
@@ -201,6 +201,7 @@ enum {SWP_STABLE_WRITES = (1 <<11 ),/* no overwrite PG_writeback pages */ SWP_SYNCHRONOUS_IO = (1 <<12 ),/* synchronous IO is efficient */ SWP_HIBERNATION = (1 <<13 ),/* pinned for hibernation */ + SWP_XSWAP = (1 << 14), /* extendable swap device */ /* add others here before... */ };
@@ -378,6 +379,7 @@ void free_folio_and_swap_cache(struct folio *folio);void free_pages_and_swap_cache (struct encoded_page **,int );/* linux/mm/swapfile.c */ extern atomic_long_t nr_swap_pages; +extern atomic_t nr_real_swapfiles;extern long total_swap_pages;extern atomic_t nr_rotate_swap;
mm/page_io.c +19 -0
@@ -258,6 +258,15 @@ int swap_writeout(struct swap_io_ctx *ctx, struct folio *folio) }rcu_read_unlock (); +/* +* xswap has no backing store: keep the folio. ctx->sis is not set +* yet, so look the device up from the entry. +*/ +if (unlikely (__swap_entry_to_info (folio->swap)->flags &SWP_XSWAP )) { +folio_mark_dirty (folio); +return AOP_WRITEPAGE_ACTIVATE ; + } +__swap_writeout (ctx, folio);return 0 ; out_unlock:
@@ -515,6 +524,16 @@ void swap_read_folio(struct swap_io_ctx *ctx, struct folio *folio)if (zswap_load (folio) != -ENOENT )return ; +if (unlikely (sis->flags &SWP_XSWAP )) { +/* +* An xswap entry only ever lives in zswap, so zswap_load() +* must have found it. Unlock and let the caller retry. +*/ +WARN_ON_ONCE (1 ); +folio_unlock (folio); +return ; + } +/* We have to read from slower devices. Increase zswap protection. */ zswap_folio_swapin (folio);swap_add_folio (ctx, folio,READ );
mm/swap_state.c +9 -0
@@ -899,6 +899,10 @@ struct folio *swap_cluster_readahead(swp_entry_t entry, gfp_t gfp_mask,struct blk_plug plug;swp_entry_t ra_entry; +/* xswap entries live only in zswap; readahead does not help. */ +if (si->flags &SWP_XSWAP ) + goto skip; + mask =swapin_nr_pages (offset) -1 ;if (!mask) goto skip;
@@ -984,6 +988,7 @@ static int swap_vma_ra_win(struct vm_fault *vmf, unsigned long *start,static struct folio *swap_vma_readahead (swp_entry_t targ_entry,gfp_t gfp_mask,struct mempolicy *mpol,pgoff_t targ_ilx,struct vm_fault *vmf) { +struct swap_info_struct *si =__swap_entry_to_info (targ_entry);struct swap_io_ctx ctx = {};struct blk_plug plug;struct folio *folio;
@@ -992,6 +997,10 @@ static struct folio *swap_vma_readahead(swp_entry_t targ_entry, gfp_t gfp_mask,unsigned long start, end, addr;pgoff_t ilx = targ_ilx; +/* xswap entries live only in zswap; readahead does not help. */ +if (si->flags &SWP_XSWAP ) + goto skip; + win =swap_vma_ra_win (vmf, &start, &end);if (win ==1 ) goto skip;
mm/swapfile.c +31 -6
@@ -66,6 +66,7 @@ static void move_cluster(struct swap_info_struct *si,static DEFINE_SPINLOCK (swap_lock);static unsigned int nr_swapfiles;atomic_long_t nr_swap_pages; +atomic_t nr_real_swapfiles;/* * Some modules use swappable objects and may try to swap them out under * memory pressure (via the shrinker). Before doing so, they may wish to
@@ -738,7 +739,8 @@ static void free_cluster(struct swap_info_struct *si, struct swap_cluster_info */* * If the swap is discardable, prepare discard the cluster * instead of free it immediately. The cluster will be freed -* after discard. +* after discard. xswap has no bdev and never sets +* SWP_PAGE_DISCARD, so it always takes the free path below. */ if ((si->flags & (SWP_WRITEOK |SWP_PAGE_DISCARD )) == (SWP_WRITEOK |SWP_PAGE_DISCARD )) {
@@ -1221,6 +1223,9 @@ static void del_from_avail_list(struct swap_info_struct *si, bool swapoff)*/ lockdep_assert_held (&si->lock); si->flags &= ~SWP_WRITEOK ; +/* Count active devices, not merely those on the avail list. */ +if (!(si->flags &SWP_XSWAP )) +atomic_sub (1 , &nr_real_swapfiles);atomic_long_or (SWAP_USAGE_OFFLIST_BIT , &si->inuse_pages); }else {/*
@@ -1254,6 +1259,12 @@ static void add_to_avail_list(struct swap_info_struct *si, bool swapon)if (swapon) {lockdep_assert_held (&si->lock); si->flags |=SWP_WRITEOK ; +/* +* Count the device here, not after adding it to the list: a +* full device is still active and skips the add below. +*/ +if (!(si->flags &SWP_XSWAP )) +atomic_add (1 , &nr_real_swapfiles); }else {if (!(READ_ONCE (si->flags) &SWP_WRITEOK )) goto skip;
@@ -1267,10 +1278,11 @@ static void add_to_avail_list(struct swap_info_struct *si, bool swapon)/* * When device is full and device is on the plist, only one updater will * see (inuse_pages == si->pages) and will call del_from_avail_list. If -* that updater happen to be here, just skip adding. +* that updater happen to be here, just skip adding. @val still has +* the off-list bit, so compare its counter part. */ pages = si->pages; -if (val == pages) { +if ((val &SWAP_USAGE_COUNTER_MASK ) == pages) {/* Just like the cmpxchg in del_from_avail_list */ if (atomic_long_try_cmpxchg (&si->inuse_pages, &pages, pages |SWAP_USAGE_OFFLIST_BIT ))
@@ -3274,7 +3286,8 @@ SYSCALL_DEFINE1(swapoff, const char __user *, specialfile)destroy_swap_extents (p, p->swap_file); -if (!(p->flags &SWP_SOLIDSTATE )) +if (!(p->flags &SWP_XSWAP ) && + !(p->flags &SWP_SOLIDSTATE ))atomic_dec (&nr_rotate_swap);mutex_lock (&swapon_mutex);
@@ -3384,6 +3397,19 @@ static void swap_stop(struct seq_file *swap, void *v)mutex_unlock (&swapon_mutex); } +static const char *swap_type_str (struct swap_info_struct *si) +{ +struct file *file = si->swap_file; + +if (si->flags &SWP_XSWAP ) +return "xswap\t" ; + +if (S_ISBLK (file_inode (file)->i_mode)) +return "partition" ; + +return "file\t" ; +} +static int swap_show (struct seq_file *swap,void *v) {struct swap_info_struct *si = v;
@@ -3403,8 +3429,7 @@ static int swap_show(struct seq_file *swap, void *v) len =seq_file_path (swap, file," \t\n\\" );seq_printf (swap,"%*s%s\t%lu\t%s%lu\t%s%d\n" , len <40 ?40 - len :1 ," " , -S_ISBLK (file_inode (file)->i_mode) ? -"partition" :"file\t" , +swap_type_str (si), bytes, bytes <10000000 ?"\t" :"" , inuse, inuse <10000000 ?"\t" :"" , si->prio);
mm/zswap.c +9 -2
@@ -1095,6 +1095,11 @@ static int zswap_writeback_entry(struct swap_io_ctx *ctx,if (IS_ERR_OR_NULL (si))return -ENOENT ; +if (si->flags &SWP_XSWAP ) { +put_swap_device (si); +return -EINVAL ; + } + mpol =get_task_policy (current); folio =__swap_cache_alloc_folio (swpentry,GFP_KERNEL ,BIT (0 ),NULL , mpol,NO_INTERLEAVE_INDEX );
@@ -1601,7 +1606,9 @@ static bool zswap_store_page(struct folio *folio, long index, entry->referenced = true;if (entry->length) {INIT_LIST_HEAD (&entry->lru); -zswap_lru_add (entry); +/* No backing store: nothing to write these back to. */ +if (!(__swap_entry_to_info (page_swpentry)->flags &SWP_XSWAP )) +zswap_lru_add (entry); }return true;
@@ -1671,7 +1678,7 @@ bool zswap_store(struct folio *folio)zswap_pool_put (pool); put_objcg:obj_cgroup_put (objcg); -if (!ret && zswap_pool_reached_full) +if (!ret && zswap_pool_reached_full &&atomic_read (&nr_real_swapfiles))queue_work (shrink_wq, &zswap_shrink_work); check_old:/*
Signature · 1 line
2.54.0
[email protected]
AI reviewRe: [PATCH v5 01/19] mm: xswap support for zswap
Thank you for your contribution! Sashiko AI review found 3 potential issues to consider: - [Medium] mm/zswap: zswap_compress() stores incompressible SWP_XSWAP pages uncompressed - [Medium]…
Thank you for your contribution! Sashiko AI review found 3 potential issues to consider:
- [Medium] mm/zswap: zswap_compress() stores incompressible SWP_XSWAP pages uncompressed
- [Medium] mm/swap_state: si shadowing and xswap neighbor readahead in swap_vma_readahead()
- [Medium] mm/zswap: infinite loop in shrink_worker() when mem_cgroup_disabled() with xswap
--
commit 5eef2108ac11c902b6d7f8f9358d479308d8aa73
Author: Chris Li <[email protected]>
mm: xswap support for zswap
This commit adds extendable swap device (xswap) support for zswap, keeping xswap entries only in zswap without backing storage, skipping swap readahead and zswap LRU insertion for them, and gating zswap writeback on active real swapfiles.
Quoted 4 lines
mm/swap_state.c +0
-0
[ ... ]
Quoted 9 lines
mm/swap_state.c +1
-0@@ -984,6 +988,7 @@ static int swap_vma_ra_win(struct vm_fault *vmf, unsigned long *start,static struct folio *swap_vma_readahead (swp_entry_t targ_entry,gfp_t gfp_mask,struct mempolicy *mpol,pgoff_t targ_ilx,struct vm_fault *vmf) { +struct swap_info_struct *si =__swap_entry_to_info (targ_entry);struct swap_io_ctx ctx = {};struct blk_plug plug;struct folio *folio;
mm/swap_state.c static struct folio *swap_vma_readahead(swp_entry_t targ_entry, gfp_t gfp_mask, · line 1006
Quoted 10 lines
mm/swap_state.c +4
-0@@ -992,6 +997,10 @@ static struct folio *swap_vma_readahead(swp_entry_t targ_entry, gfp_t gfp_mask,unsigned long start, end, addr;pgoff_t ilx = targ_ilx; +/* xswap entries live only in zswap; readahead does not help. */ +if (si->flags &SWP_XSWAP ) + goto skip; + win =swap_vma_ra_win (vmf, &start, &end);if (win ==1 ) goto skip;
[Severity: Medium]
Can swap_vma_readahead() still perform readahead on neighboring SWP_XSWAP entries when targ_entry is on a real swap device?
In the PTE loop below, neighboring entries in the VMA window may belong to another swap device:
mm/swap_state.c:swap_vma_readahead() {
...
for (addr = start; addr < end; ilx++, addr += PAGE_SIZE) {
struct swap_info_struct *si = NULL;
...
if (swp_type(entry) != swp_type(targ_entry)) {
si = get_swap_device(entry);
if (IS_ERR_OR_NULL(si))
continue;
}
folio = swap_cache_read_folio(&ctx, entry, gfp_mask, mpol, ilx,
addr != vmf->address);
}When a neighboring entry is on an SWP_XSWAP device, swap_cache_read_folio() allocates a swapcache folio and calls zswap_load(), synchronously decompressing and erasing the xswap entry from zswap during speculative readahead.
Also, the new outer struct swap_info_struct *si declaration is shadowed by the inner si declaration in the for loop, triggering a -Wshadow warning.
Quoted 4 lines
mm/zswap.c +0
-0
[ ... ]
mm/zswap.c static bool zswap_store_page(struct folio *folio, long index, · line 1614
Quoted 11 lines
mm/zswap.c +3
-1@@ -1601,7 +1606,9 @@ static bool zswap_store_page(struct folio *folio, long index, entry->referenced = true;if (entry->length) {INIT_LIST_HEAD (&entry->lru); -zswap_lru_add (entry); +/* No backing store: nothing to write these back to. */ +if (!(__swap_entry_to_info (page_swpentry)->flags &SWP_XSWAP )) +zswap_lru_add (entry); }return true;
[Severity: Medium]
Does zswap_compress() store incompressible SWP_XSWAP pages uncompressed at PAGE_SIZE when cgroup zswap writeback is enabled?
mm/zswap.c:zswap_compress() {
...
if (comp_ret) {
rcu_read_lock();
if (!mem_cgroup_zswap_writeback_enabled(folio_memcg(folio))) {
rcu_read_unlock();
goto unlock;
}
rcu_read_unlock();
comp_ret = 0;
dlen = PAGE_SIZE;
dst = kmap_local_folio(folio, index * PAGE_SIZE);
mapped = true;
}
...
}Storing PAGE_SIZE entries in zsmalloc exists only to preserve writeback LRU order, but SWP_XSWAP entries are not added to zswap_list_lru and have no backing store. Accepting incompressible SWP_XSWAP pages wastes zswap pool capacity and metadata on unreclaimable entries instead of failing zswap_store() so swap_writeout() can reactivate the folio via AOP_WRITEPAGE_ACTIVATE. Should zswap_compress() also reject comp_ret != 0 when the folio's swap device has SWP_XSWAP set?
mm/zswap.c bool zswap_store(struct folio *folio) · line 1684
Quoted 9 lines
mm/zswap.c +1
-1@@ -1671,7 +1678,7 @@ bool zswap_store(struct folio *folio)zswap_pool_put (pool); put_objcg:obj_cgroup_put (objcg); -if (!ret && zswap_pool_reached_full) +if (!ret && zswap_pool_reached_full &&atomic_read (&nr_real_swapfiles))queue_work (shrink_wq, &zswap_shrink_work); check_old:/*
[Severity: Medium]
Can shrink_worker() spin in an unbounded loop when memory cgroups are disabled (CONFIG_MEMCG=n or cgroup_disable=memory) and the zswap pool is filled above the accept threshold by SWP_XSWAP entries?
mm/zswap.c:shrink_worker() {
...
do {
...
if (!memcg && !mem_cgroup_disabled()) {
if (!attempts && ++failures == MAX_RECLAIM_RETRIES)
break;
attempts = 0;
goto resched;
}
ret = shrink_memcg(memcg);
/* drop the extra reference */
mem_cgroup_put(memcg);
...
if (ret == -ENOENT)
goto resched;
++attempts;
if (ret && ++failures == MAX_RECLAIM_RETRIES)
break;resched:
cond_resched(); } while (zswap_total_pages() > thr); }
When both a real swapfile and an xswap device are active, zswap_store() queues zswap_shrink_work once the pool is full. If zswap_list_lru is empty because the pool is occupied by SWP_XSWAP entries (which are not on the LRU), shrink_memcg(NULL) returns -ENOENT and jumps to resched without incrementing failures, while the !memcg termination check is skipped because mem_cgroup_disabled() is true. The worker then loops until external swap frees or faults bring zswap_total_pages() below thr.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
Baoquan He
reviewerPATCH v5 2/19 mm, swap: refactor free_swap_cluster_info to take swap_info_struct
In reply to Baoquan HeFrom: Baoquan He <[email protected]>
Change free_swap_cluster_info() to take struct swap_info_struct* instead of (cluster_info, maxpages). It now extracts the fields from si and clears si->cluster_info after freeing to avoid a double free on the swapon() error path.
The new parameter also lets the xswap path access si->flags in the function.
--- mm/swapfile.c | 26 ++++++++++++++------------ 1 file changed, 14 insertions(+), 12 deletions(-)
mm/swapfile.c +14 -12
@@ -3155,14 +3155,17 @@ static void wait_for_allocation(struct swap_info_struct *si) } } -static void free_swap_cluster_info (struct swap_cluster_info *cluster_info, -unsigned long maxpages) +static void free_swap_cluster_info (struct swap_info_struct *si) { +struct swap_cluster_info *cluster_info = si->cluster_info; +unsigned long maxpages = si->max;struct swap_cluster_info *ci; -int i, nr_clusters =DIV_ROUND_UP (maxpages,SWAPFILE_CLUSTER ); +int i, nr_clusters;if (!cluster_info)return ; + + nr_clusters =DIV_ROUND_UP (maxpages,SWAPFILE_CLUSTER );for (i =0 ; i < nr_clusters; i++) { ci = cluster_info + i;/* Cluster with bad marks count will have a remaining table */
@@ -3174,6 +3177,7 @@ static void free_swap_cluster_info(struct swap_cluster_info *cluster_info,spin_unlock (&ci->lock); }kvfree (cluster_info); + si->cluster_info =NULL ; }/*
@@ -3201,11 +3205,9 @@ static void flush_percpu_swap_cluster(struct swap_info_struct *si) SYSCALL_DEFINE1(swapoff, const char __user *, specialfile) {struct swap_info_struct *p =NULL ; -struct swap_cluster_info *cluster_info;struct file *swap_file, *victim;struct address_space *mapping;struct inode *inode; -unsigned int maxpages;int err, found =0 ;if (!capable (CAP_SYS_ADMIN ))
@@ -3297,10 +3299,6 @@ SYSCALL_DEFINE1(swapoff, const char __user *, specialfile) swap_file = p->swap_file; p->swap_file =NULL ; - maxpages = p->max; - cluster_info = p->cluster_info; - p->max =0 ; - p->cluster_info =NULL ;spin_unlock (&p->lock);spin_unlock (&swap_lock);arch_swap_invalidate_area (p->type);
@@ -3308,7 +3306,9 @@ SYSCALL_DEFINE1(swapoff, const char __user *, specialfile)mutex_unlock (&swapon_mutex);kfree (p->global_cluster); p->global_cluster =NULL ; -free_swap_cluster_info (cluster_info, maxpages); +free_swap_cluster_info (p); + p->max =0 ; + p->cluster_info =NULL ; inode = mapping->host;
@@ -3675,6 +3675,8 @@ static int setup_swap_clusters_info(struct swap_info_struct *si,if (!cluster_info) goto err; + si->cluster_info = cluster_info; +for (i =0 ; i < nr_clusters; i++)spin_lock_init (&cluster_info[i].lock);
@@ -3738,7 +3740,7 @@ static int setup_swap_clusters_info(struct swap_info_struct *si, si->cluster_info = cluster_info;return 0 ; err: -free_swap_cluster_info (cluster_info, maxpages); +free_swap_cluster_info (si);return err; }
@@ -3960,7 +3962,7 @@ SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags) si->global_cluster =NULL ; inode =NULL ;destroy_swap_extents (si, swap_file); -free_swap_cluster_info (si->cluster_info, si->max); +free_swap_cluster_info (si); si->cluster_info =NULL ;/* * Clear the SWP_USED flag after all resources are freed so
Signature · 1 line
2.54.0
Baoquan He
reviewerPATCH v5 3/19 mm, swap: add VM_SPARSE cluster-array helpers for xswap
In reply to Baoquan HeFrom: Baoquan He <[email protected]>
Add CONFIG_XSWAP and the state needed for a sparse cluster_info array:cluster_vm, nr_clusters_max, nr_clusters_mapped and xswap_lock.
Also add the helpers that map and unmap chunks of the VM_SPARSE area, plus the callback that finds the already-mapped prefix. The helpers have no callers yet; the next patch adds them into setup/free and the walkers.
Unmap collects the backing pages before the PTEs go away and frees them after, so it cannot fail and its callers need no retry loop. If the collection array cannot be allocated, it falls back to a bounded batch with a stack array.
They are marked __maybe_unused for this intermediate step so a CONFIG_WERROR=y build does not warn about the missing callers.
--- include/linux/swap.h | 6 ++ mm/Kconfig | 9 ++ mm/swapfile.c | 231 +++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 246 insertions(+)
include/linux/swap.h +6 -0
@@ -242,6 +242,12 @@ struct swap_info_struct {signed char type;/* strange name for an index */ unsigned int max;/* size of this swap device */ struct swap_cluster_info *cluster_info;/* array, one entry per cluster */ +#ifdef CONFIG_XSWAP +struct vm_struct *cluster_vm;/* VM_SPARSE area for cluster_info */ +unsigned long nr_clusters_max;/* total clusters in the xswap address space */ +unsigned long nr_clusters_mapped;/* currently mapped cluster count */ +struct mutex xswap_lock;/* serialize map/unmap operations */ +#endif struct list_head free_clusters;/* free clusters list */ struct list_head full_clusters;/* full clusters list */ struct list_head nonfull_clusters[SWAP_NR_ORDERS ];
mm/Kconfig +9 -0
@@ -122,6 +122,15 @@ config ZSWAP_COMPRESSOR_DEFAULT default "zstd" if ZSWAP_COMPRESSOR_DEFAULT_ZSTD default "" +config XSWAP + bool "Extendable (virtual) swap device" + depends on SWAP && 64BIT && ZSWAP && SYSFS + help + Adds support for extendable swap devices (xswap) that decouple + PTE swap entries from physical backing storage. The cluster_info + array is backed by a sparse vmalloc area that is mapped lazily + in chunks, avoiding static pre-allocation overhead. + config ZSMALLOC tristate
mm/swapfile.c +231 -0
243 lines not on this page · raw message
Signature · 1 line
2.54.0
Baoquan He
reviewerPATCH v5 4/19 mm, swap: use a VM_SPARSE cluster array for xswap devices
In reply to Baoquan HeFrom: Baoquan He <[email protected]>
An xswap device does not have to allocate cluster_info for its whole address space at once. Back it with a VM_SPARSE area and map it in chunks. At setup, map only the first chunk. The grow path maps more chunks when they are needed.
Setup, free, and the two cluster_info scans must change together. If only setup changes, free would free the wrong object, and the scans would read cluster_info entries that are not mapped yet.
The map helper rejects a stale start_idx and cleans up a partial mapping if it fails. The grow path added later depends on this check.
--- mm/swapfile.c | 152 ++++++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 141 insertions(+), 11 deletions(-)
mm/swapfile.c +141 -11
@@ -50,6 +50,17 @@#include "swap.h" #ifdef CONFIG_XSWAP +/* +* xswap: dynamically grow the cluster_info array via a VM_SPARSE area. +* +* XSWAP_GROW_CLUSTERS is the number of clusters to map in one grow +* operation. It is set to the number of cluster_info structs that +* fit in a single page (at least 16), so that the vmalloc page table +* overhead is proportional to the number of clusters mapped. +*/ +#define XSWAP_GROW_CLUSTERS \ + max_t(unsigned long, PAGE_SIZE / sizeof(struct swap_cluster_info), 16) +/* Batch size for the fallback unmap; keeps the stack array small. */ #define XSWAP_UNMAP_BATCH_PAGES 16#define XSWAP_UNMAP_BATCH_CLUSTERS \
@@ -2825,10 +2836,27 @@ static unsigned int find_next_to_unuse(struct swap_info_struct *si, unsigned int prev) {struct swap_cluster_info *ci; -unsigned long i, end; +unsigned long i, cluster_end, end;unsigned int ci_off;unsigned long swp_tb; + end = si->max; +#ifdef CONFIG_XSWAP +/* +* An xswap device maps its cluster_info in chunks as it grows, so +* the tail past nr_clusters_mapped has nothing behind it. +*/ +if (si->flags &SWP_XSWAP ) { +unsigned long mapped_end; + +/* Pairs with the smp_store_release() in xswap_map_clusters(). */ + mapped_end =smp_load_acquire (&si->nr_clusters_mapped) * +SWAPFILE_CLUSTER ; +if (mapped_end < end) + end = mapped_end; + } +#endif +/* * No need for swap_lock here: we're just looking * for whether an entry is in use, not modifying it; false
@@ -2836,11 +2864,11 @@ static unsigned int find_next_to_unuse(struct swap_info_struct *si,* allocations from this area (while holding swap_lock). */ i = prev +1 ; -while (i < si->max) { +while (i < end) { ci =__swap_offset_to_cluster (si, i); - end = min_t(unsigned long, -ALIGN_DOWN (i,SWAPFILE_CLUSTER ) +SWAPFILE_CLUSTER , - si->max); + cluster_end = min_t(unsigned long, +ALIGN_DOWN (i,SWAPFILE_CLUSTER ) +SWAPFILE_CLUSTER , + end);/* * An empty cluster has no slot in use, so skip it whole.
@@ -2850,13 +2878,13 @@ static unsigned int find_next_to_unuse(struct swap_info_struct *si,* enough, unlike in every other cluster_is_empty() caller. */ if (!READ_ONCE (ci->count)) { - i = end; + i = cluster_end;cond_resched ();continue ; } ci_off = i %SWAPFILE_CLUSTER ; -for (; i < end; ci_off++, i++) { +for (; i < cluster_end; ci_off++, i++) { swp_tb =swap_table_get (ci, ci_off);if (!swp_tb_is_null (swp_tb) && !swp_tb_is_bad (swp_tb))return i;
@@ -2866,7 +2894,6 @@ static unsigned int find_next_to_unuse(struct swap_info_struct *si,return 0 ; } -static int try_to_unuse (unsigned int type) {struct mm_struct *prev_mm;
@@ -3164,6 +3191,14 @@ static void wait_for_allocation(struct swap_info_struct *si)BUG_ON (si->flags &SWP_WRITEOK ); +#ifdef CONFIG_XSWAP +if (si->flags &SWP_XSWAP ) { +/* Pairs with the smp_store_release() in xswap_map_clusters(). */ + end =min (end,smp_load_acquire (&si->nr_clusters_mapped) * +SWAPFILE_CLUSTER ); + } +#endif +for (offset =0 ; offset < end; offset +=SWAPFILE_CLUSTER ) { ci =swap_cluster_lock (si, offset);swap_cluster_unlock (ci);
@@ -3175,11 +3210,43 @@ static void free_swap_cluster_info(struct swap_info_struct *si)struct swap_cluster_info *cluster_info = si->cluster_info;unsigned long maxpages = si->max;struct swap_cluster_info *ci; -int i, nr_clusters; +unsigned long i, nr_clusters;if (!cluster_info)return ; +#ifdef CONFIG_XSWAP +if (si->flags &SWP_XSWAP ) { +unsigned long nr_mapped; + +/* +* Cluster 0 keeps the bad header slot, so it never empties +* and __free_cluster() never frees its table. +*/ +/* Pairs with the smp_store_release() in xswap_map_clusters(). */ + nr_mapped =smp_load_acquire (&si->nr_clusters_mapped); +for (i =0 ; i < nr_mapped; i++) { + ci = &cluster_info[i]; +spin_lock (&ci->lock); +if (cluster_table_is_alloced (ci)) { +swap_cluster_assert_empty (ci,0 ,SWAPFILE_CLUSTER , true); +swap_cluster_free_table (ci); + } +spin_unlock (&ci->lock); + } +/* +* free_vm_area() drops the mapping without freeing the +* pages, and the unmap helper handles an empty range. +*/ +xswap_unmap_clusters (si,0 , si->nr_clusters_mapped); +free_vm_area (si->cluster_vm); + si->cluster_vm =NULL ; + si->cluster_info =NULL ; + si->nr_clusters_mapped =0 ; +return ; + } +#endif + nr_clusters =DIV_ROUND_UP (maxpages,SWAPFILE_CLUSTER );for (i =0 ; i < nr_clusters; i++) { ci = cluster_info + i;
@@ -3678,7 +3745,7 @@ static unsigned long read_swap_header(struct swap_info_struct *si, }#ifdef CONFIG_XSWAP -static int __maybe_unused xswap_map_clusters (struct swap_info_struct *si, +static int xswap_map_clusters (struct swap_info_struct *si,unsigned long start_idx,unsigned long nr) { unsigned long start_addr = (unsigned long)si->cluster_info +
@@ -3824,7 +3891,7 @@ static void xswap_unmap_range(struct swap_info_struct *si, }/* Caller must hold si->xswap_lock. Cannot fail. */ -static void __maybe_unused xswap_unmap_clusters (struct swap_info_struct *si, +static void xswap_unmap_clusters (struct swap_info_struct *si,unsigned long start_idx,unsigned long nr) { unsigned long start_addr = (unsigned long)si->cluster_info +
@@ -3902,6 +3969,69 @@ static int setup_swap_clusters_info(struct swap_info_struct *si,int err = -ENOMEM ;unsigned long i; +#ifdef CONFIG_XSWAP +if (si->flags &SWP_XSWAP ) { +unsigned long size =PAGE_ALIGN (nr_clusters *sizeof (*cluster_info)); +struct vm_struct *vm; + + vm =get_vm_area (size,VM_SPARSE ); +if (!vm) + goto err; + + cluster_info = vm->addr; + si->cluster_vm = vm; + si->nr_clusters_max = nr_clusters; + si->cluster_info = cluster_info; + +/* Must be initialized before xswap_map_clusters() locks it. */ +mutex_init (&si->xswap_lock); + + if (xswap_map_clusters(si, 0, min_t(unsigned long, +XSWAP_GROW_CLUSTERS , nr_clusters))) + goto err_free_vm; + +/* xswap: only cluster 0 slot 0 is bad */ + err =swap_cluster_setup_bad_slot (si, cluster_info,0 , false); +if (err) + goto err_unmap; + +INIT_LIST_HEAD (&si->free_clusters); +INIT_LIST_HEAD (&si->full_clusters); +INIT_LIST_HEAD (&si->discard_clusters); +for (i =0 ; i <SWAP_NR_ORDERS ; i++) { +INIT_LIST_HEAD (&si->nonfull_clusters[i]); +INIT_LIST_HEAD (&si->frag_clusters[i]); + } + +/* +* Cluster 0 holds the header slot, which is marked bad, so it +* is not entirely free. si->max is cluster aligned, so no +* cluster holds a slot past the end of the device. +*/ +for (i =0 ; i < si->nr_clusters_mapped; i++) { +struct swap_cluster_info *ci = &cluster_info[i]; + +if (ci->count) { + ci->flags =CLUSTER_FLAG_NONFULL ; +list_add_tail (&ci->list, &si->nonfull_clusters[0 ]); + }else { + ci->flags =CLUSTER_FLAG_FREE ; +list_add_tail (&ci->list, &si->free_clusters); + } + } + +return 0 ; + +err_unmap: +xswap_unmap_clusters (si,0 , si->nr_clusters_mapped); +err_free_vm: +free_vm_area (si->cluster_vm); + si->cluster_vm =NULL ; + si->cluster_info =NULL ; +return err; + } +#endif /* CONFIG_XSWAP */ + cluster_info =kvzalloc_objs (*cluster_info, nr_clusters);if (!cluster_info) goto err;
Signature · 1 line
2.54.0
[email protected]
AI reviewRe: [PATCH v5 04/19] mm, swap: use a VM_SPARSE cluster array for xswap devices
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider: - [High] mm, swap: missing nr_clusters_mapped check in get_swap_device() --
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider:
- [High] mm, swap: missing nr_clusters_mapped check in get_swap_device() --
commit 59c7799e61f59ce4eb079135f6f31b57258c37bd
Author: Baoquan He <[email protected]>
mm, swap: use a VM_SPARSE cluster array for xswap devices
This commit backs si->cluster_info for xswap devices with a VM_SPARSE vmalloc area mapped initially to XSWAP_GROW_CLUSTERS clusters, and bounds the cluster walks in find_next_to_unuse(), wait_for_allocation() and free_swap_cluster_info() to si->nr_clusters_mapped.
Quoted 4 lines
mm/swapfile.c +0
-0
[ ... ]
Quoted 14 lines
mm/swapfile.c +10
-0@@ -3902,6 +3969,69 @@ static int setup_swap_clusters_info(struct swap_info_struct *si,int err = -ENOMEM ;unsigned long i; +#ifdef CONFIG_XSWAP +if (si->flags &SWP_XSWAP ) { +unsigned long size =PAGE_ALIGN (nr_clusters *sizeof (*cluster_info)); +struct vm_struct *vm; + + vm =get_vm_area (size,VM_SPARSE ); +if (!vm) + goto err; + + cluster_info = vm->addr;
mm/swapfile.c static int setup_swap_clusters_info(struct swap_info_struct *si, · line 3991
Quoted 10 lines
mm/swapfile.c +10
-0+ si->cluster_vm = vm; + si->nr_clusters_max = nr_clusters; + si->cluster_info = cluster_info; + +/* Must be initialized before xswap_map_clusters() locks it. */ +mutex_init (&si->xswap_lock); + + if (xswap_map_clusters(si, 0, min_t(unsigned long, +XSWAP_GROW_CLUSTERS , nr_clusters))) + goto err_free_vm;
[Severity: High]
Can a lockless swap lookup fault on an unmapped page in si->cluster_info when offset / SWAPFILE_CLUSTER >= si->nr_clusters_mapped?
setup_swap_clusters_info() maps only the first XSWAP_GROW_CLUSTERS clusters while leaving si->max at the full device size, and get_swap_device() only checks offset against si->max:
mm/swapfile.c:get_swap_device() {
...
offset = swp_offset(entry);
if (offset >= si->max)
goto put_out;
return si;
...
}Callers such as do_swap_page() read a PTE locklessly before calling get_swap_device(entry) and re-verify the PTE under the page table lock only after looking up the swap cache. If a stale PTE from a prior swapoff and xswap activation (or after tail clusters are unmapped by commit "mm, swap: add xswap shrink trigger and defer it to a workqueue") has offset < si->max but offset / SWAPFILE_CLUSTER >= si->nr_clusters_mapped, get_swap_device() succeeds and swap_cache_get_folio() -> __swap_offset_to_cluster() dereferences an unmapped VM_SPARSE page in si->cluster_info.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=4
Baoquan He
reviewerPATCH v5 5/19 mm, swap: widen swap_info_struct max/pages to unsigned long
In reply to Baoquan HeFrom: Baoquan He <[email protected]>
si->max and si->pages are unsigned int. With 4 KiB pages, this limits an xswap device to 16 TB. xswap may need a larger address space, so change both fields to unsigned long.
Several places still use unsigned int for offsets and indexes. Widen them as well:
- the return value of cluster_offset() and the cluster scan helpers;
- the allocation offsets in cluster_alloc_swap_entry() and swap_alloc_fast();
- the indexes in find_next_to_unuse() and try_to_unuse().
Also change the related print formats from %u to %lu.
A real swap device is still limited by swap_header.info.last_page, which is __u32. So this only widens the xswap address space. The old overflow clamp in read_swap_header() is no longer needed; the header check still limits real devices.
count_swap_pages() returns unsigned int and is only used by hibernation. Hibernation never selects an xswap device. If that ever changes, WARN and return 0 instead of silently truncating the value.
--- include/linux/swap.h | 4 +-- mm/swapfile.c | 59 +++++++++++++++++++++++--------------------- 2 files changed, 33 insertions(+), 30 deletions(-)
include/linux/swap.h +2 -2
@@ -240,7 +240,7 @@ struct swap_info_struct {signed short prio;/* swap priority of this type */ struct plist_node list;/* entry in swap_active_head */ signed char type;/* strange name for an index */ -unsigned int max;/* size of this swap device */ +unsigned long max;/* size of this swap device */ struct swap_cluster_info *cluster_info;/* array, one entry per cluster */ #ifdef CONFIG_XSWAP struct vm_struct *cluster_vm;/* VM_SPARSE area for cluster_info */
@@ -254,7 +254,7 @@ struct swap_info_struct {/* list of cluster that contains at least one free slot */ struct list_head frag_clusters[SWAP_NR_ORDERS ];/* list of cluster that are fragmented or contented */ -unsigned int pages;/* total of usable pages of swap */ +unsigned long pages;/* total of usable pages of swap */ atomic_long_t inuse_pages;/* number of those currently in use */ struct swap_sequential_cluster *global_cluster;/* Use one global cluster for rotating device */ spinlock_t global_cluster_lock;/* Serialize usage of global cluster */
mm/swapfile.c +31 -28
135 lines not on this page · raw message
Signature · 1 line
2.54.0
Baoquan He
reviewerPATCH v5 6/19 mm, swap: add a sysfs interface to create xswap devices
In reply to Baoquan HeFrom: Baoquan He <[email protected]>
An xswap device has no backing file, so it cannot be created with swapon. Add a sysfs interface to create one directly:
/sys/kernel/mm/xswap/create
Write an optional priority to this file. Write nothing to use the default priority. By default the device gets the highest priority.
A new device gets si->max and nr_clusters_max equal to twice RAM. The cluster_info array is a VM_SPARSE area and is mapped lazily. At this point only the first XSWAP_GROW_CLUSTERS chunk is mapped; the grow trigger comes in a later patch. The device is shown in /proc/swaps as "xswap<N>".
Hibernation device discovery skips xswap devices because they have no bdev to store a resume image.
--- mm/swapfile.c | 175 ++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 170 insertions(+), 5 deletions(-)
mm/swapfile.c +170 -5
@@ -16,6 +16,8 @@#include <linux/kernel_stat.h> #include <linux/swap.h> #include <linux/vmalloc.h> +#include <linux/kobject.h> +#include <linux/sysfs.h> #include <linux/pagemap.h> #include <linux/namei.h> #include <linux/shmem_fs.h>
@@ -48,6 +50,15 @@#include "swap_table.h" #include "internal.h" #include "swap.h" +#define DEF_SWAP_PRIO -1 + +/* +* The highest priority a real swap device can use: one ahead of xswap +* would take the page directly, and taking that slot is what xswap +* exists to avoid. At equal priority the list is FIFO, so a real device +* that asks for this priority too and is added first wins. +*/ +#define DEF_XSWAP_PRIO SWAP_FLAG_PRIO_MASK#ifdef CONFIG_XSWAP /*
@@ -73,7 +84,67 @@ static int xswap_map_clusters(struct swap_info_struct *si,static void xswap_unmap_clusters (struct swap_info_struct *si,unsigned long start_idx,unsigned long nr);static int xswap_mapped_end (pte_t *pte,unsigned long addr,void *data); -#endif + +static int xswap_create (int prio); + +static ssize_t xswap_create_store (struct kobject *kobj, +struct kobj_attribute *attr, +const char *buf,size_t count) +{ +int prio =DEF_XSWAP_PRIO ; +const char *p; +int err; + +if (!capable (CAP_SYS_ADMIN )) +return -EPERM ; + +/* "[<prio>]" is optional; a lower one is allowed, not a higher. */ + p =skip_spaces (buf); +if (*p && *p !='\n' ) { + err =kstrtoint (p,10 , &prio); +if (err) +return err; + } + + err =xswap_create (prio); +if (err <0 ) +return err; + +return count; +} + +static struct kobj_attribute xswap_create_attr =__ATTR (create,0200 ,NULL , + xswap_create_store); + +static struct attribute *xswap_attrs[] = { + &xswap_create_attr.attr, +NULL , +}; + +static const struct attribute_group xswap_attr_group = { + .attrs = xswap_attrs, +}; + +static struct kobject *xswap_kobj; + +static void xswap_sysfs_init (void ) +{ + xswap_kobj =kobject_create_and_add ("xswap" , mm_kobj); +if (!xswap_kobj) { +pr_err ("xswap: failed to create sysfs kobject\n" ); +return ; + } +if (sysfs_create_group (xswap_kobj, &xswap_attr_group)) { +pr_err ("xswap: failed to create sysfs group\n" ); +kobject_put (xswap_kobj); + xswap_kobj =NULL ; + } +} +#else /* !CONFIG_XSWAP */ +static inline void xswap_sysfs_init (void ) +{ +} +#endif /* CONFIG_XSWAP */ static void swap_range_alloc (struct swap_info_struct *si,unsigned int nr_entries);
@@ -101,7 +172,6 @@ atomic_t nr_real_swapfiles;EXPORT_SYMBOL_GPL (nr_swap_pages);/* protected with swap_lock. reading in vm_swap_full() doesn't need lock */ long total_swap_pages; -#define DEF_SWAP_PRIO -1unsigned long swapfile_maximum_size;#ifdef CONFIG_MIGRATION bool swap_migration_ad_supported;
@@ -2284,6 +2354,9 @@ static int __find_hibernation_swap_type(dev_t device, sector_t offset)if (!(sis->flags &SWP_WRITEOK ))continue ; +/* xswap has no bdev to match a resume device */ +if (sis->flags &SWP_XSWAP ) +continue ;if (device == sis->bdev->bd_dev) {struct swap_extent *se =first_se (sis);
@@ -2471,6 +2544,8 @@ int find_first_swap(dev_t *device)if (!(sis->flags &SWP_WRITEOK ))continue ; +if (sis->flags &SWP_XSWAP ) +continue ; *device = sis->bdev->bd_dev;spin_unlock (&swap_lock);return type;
@@ -3452,7 +3527,8 @@ static void *swap_start(struct seq_file *swap, loff_t *pos)return SEQ_START_TOKEN ;for (type =0 ; (si =swap_type_to_info (type)); type++) { -if (!(si->swap_file)) +if (!si->swap_file && + !((si->flags &SWP_XSWAP ) && (si->flags &SWP_WRITEOK )))continue ;if (!--l)return si;
@@ -3473,7 +3549,8 @@ static void *swap_next(struct seq_file *swap, void *v, loff_t *pos) ++(*pos);for (; (si =swap_type_to_info (type)); type++) { -if (!(si->swap_file)) +if (!si->swap_file && + !((si->flags &SWP_XSWAP ) && (si->flags &SWP_WRITEOK )))continue ;return si; }
@@ -3515,7 +3592,14 @@ static int swap_show(struct seq_file *swap, void *v) inuse =K (swap_usage_in_pages (si)); file = si->swap_file; - len =seq_file_path (swap, file," \t\n\\" ); +if (file) + len =seq_file_path (swap, file," \t\n\\" ); +else { +char name[16 ]; + + len =scnprintf (name,sizeof (name),"xswap%d" , si->type); +seq_puts (swap, name); + } seq_printf(swap, "%*s%s\t%lu\t%s%lu\t%s%d\n", len < 40 ? 40 - len : 1, " ", swap_type_str(si),
@@ -4026,6 +4110,8 @@ static int setup_swap_clusters_info(struct swap_info_struct *si,return 0 ; err_unmap: +/* swap_cluster_setup_bad_slot() allocates cluster 0's table. */ +swap_cluster_free_table (&si->cluster_info[0 ]);xswap_unmap_clusters (si,0 , si->nr_clusters_mapped); err_free_vm:free_vm_area (si->cluster_vm);
@@ -4108,6 +4194,83 @@ static int setup_swap_clusters_info(struct swap_info_struct *si,return err; } +#ifdef CONFIG_XSWAP +/* Create a file-less xswap device, sized at twice RAM. */ +static int xswap_create (int prio) +{ +struct swap_info_struct *si; +unsigned long ram, maxpages; +int error; + +if (prio !=DEF_XSWAP_PRIO && + (prio <0 || prio >SWAP_FLAG_PRIO_MASK )) +return -EINVAL ; + + si =alloc_swap_info (); +if (IS_ERR (si)) +return PTR_ERR (si); + +INIT_WORK (&si->discard_work, swap_discard_work); +INIT_WORK (&si->reclaim_work, swap_reclaim_work); + + ram =totalram_pages (); + maxpages = min_t(unsigned long, ram * 2, swapfile_maximum_size); +/* A cluster index is an unsigned int, so cap the cluster count. */ +if (maxpages /SWAPFILE_CLUSTER >UINT_MAX ) + maxpages = (unsigned long )UINT_MAX *SWAPFILE_CLUSTER ; +/* Cluster-aligned, so no cluster holds a slot past si->max. */ +if (maxpages >SWAPFILE_CLUSTER ) + maxpages =rounddown (maxpages,SWAPFILE_CLUSTER ); +if (maxpages <2 ) + maxpages =2 ; + + si->bdev =NULL ; + si->swap_file =NULL ; + si->flags |=SWP_XSWAP |SWP_SOLIDSTATE ; + si->max = maxpages; + si->pages = maxpages -1 ; +/* +* No backing file: setup_swap_extents() is only reachable from the +* file-backed swapon() path, so set ops here. Only ops->flags is +* used, by may_enter_fs(); the IO methods are never called because +* swap_writeout()/swap_read_folio() short circuit xswap. +*/ + si->ops = &swap_bdev_ops; + + error =setup_swap_clusters_info (si,NULL , maxpages); +if (error) + goto bad_swap; + + error =zswap_swapon (si->type, si->max); +if (error) + goto bad_swap; + +mutex_lock (&swapon_mutex); + si->prio = prio; + si->list.prio = -si->prio; + si->avail_list.prio = -si->prio; +enable_swap_info (si); +pr_info ("xswap: adding extendable swap type %d (prio %d, %lu pages)\n" , + si->type, prio, si->pages); +mutex_unlock (&swapon_mutex); +atomic_inc (&proc_poll_event); +wake_up_interruptible (&proc_poll_wait); + +return si->type; + +bad_swap: +kfree (si->global_cluster); + si->global_cluster =NULL ; +destroy_swap_extents (si,NULL );/* safe: xswap never sets SWP_ACTIVATED */ +free_swap_cluster_info (si); + si->cluster_info =NULL ; +spin_lock (&swap_lock); + si->flags =0 ; +spin_unlock (&swap_lock); +return error; +} +#endif /* CONFIG_XSWAP */ + SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags) {struct swap_info_struct *si;
@@ -4454,6 +4617,8 @@ static int __init swapfile_init(void) swap_migration_ad_supported = true;#endif /* CONFIG_MIGRATION */ +xswap_sysfs_init (); +return 0 ; }subsys_initcall (swapfile_init);
Signature · 1 line
2.54.0
Baoquan He
reviewerPATCH v5 7/19 mm, swap: refactor swapoff and add xswap_destroy
In reply to Baoquan HeFrom: Baoquan He <[email protected]>
Extract __swapoff() from sys_swapoff() so the teardown logic can be shared, and make it work for file-less devices.
Add xswap_destroy() to tear down a file-less xswap device by swap type, exposed via /sys/kernel/mm/xswap/destroy. Writing a swap type tears down that device; it requires CAP_SYS_ADMIN.
--- mm/swapfile.c | 127 +++++++++++++++++++++++++++++++++++++++++++------- 1 file changed, 109 insertions(+), 18 deletions(-)
mm/swapfile.c +109 -18
@@ -86,6 +86,7 @@ static void xswap_unmap_clusters(struct swap_info_struct *si,static int xswap_mapped_end (pte_t *pte,unsigned long addr,void *data);static int xswap_create (int prio); +static int xswap_destroy (int type); static ssize_t xswap_create_store(struct kobject *kobj, struct kobj_attribute *attr,
@@ -116,8 +117,35 @@ static ssize_t xswap_create_store(struct kobject *kobj,static struct kobj_attribute xswap_create_attr =__ATTR (create,0200 ,NULL , xswap_create_store); +static ssize_t xswap_destroy_store (struct kobject *kobj, +struct kobj_attribute *attr, +const char *buf,size_t count) +{ +unsigned long type; +int err; + +if (!capable (CAP_SYS_ADMIN )) +return -EPERM ; + + err =kstrtoul (buf,0 , &type); +if (err) +return err; +if (type >=MAX_SWAPFILES ) +return -EINVAL ; + + err =xswap_destroy (type); +if (err) +return err; + +return count; +} + +static struct kobj_attribute xswap_destroy_attr =__ATTR (destroy,0200 ,NULL , + xswap_destroy_store); +static struct attribute *xswap_attrs[] = { &xswap_create_attr.attr, + &xswap_destroy_attr.attr,NULL , };
@@ -3365,13 +3393,14 @@ static void flush_percpu_swap_cluster(struct swap_info_struct *si) } } +static int swap_info_remove (struct swap_info_struct *p); +static int __swapoff (struct swap_info_struct *p); SYSCALL_DEFINE1(swapoff, const char __user *, specialfile) {struct swap_info_struct *p =NULL ; -struct file *swap_file, *victim; +struct file *victim;struct address_space *mapping; -struct inode *inode;int err, found =0 ;if (!capable (CAP_SYS_ADMIN ))
@@ -3388,7 +3417,7 @@ SYSCALL_DEFINE1(swapoff, const char __user *, specialfile)spin_lock (&swap_lock);plist_for_each_entry (p, &swap_active_head, list) {if (p->flags &SWP_WRITEOK ) { -if (p->swap_file->f_mapping == mapping) { +if (p->swap_file && p->swap_file->f_mapping == mapping) { found =1 ;break ; }
@@ -3407,20 +3436,50 @@ SYSCALL_DEFINE1(swapoff, const char __user *, specialfile) goto out_dput; } -if (!security_vm_enough_memory_mm (current->mm, p->pages)) -vm_unacct_memory (p->pages); -else { - err = -ENOMEM ; + err =swap_info_remove (p); +if (err) {spin_unlock (&swap_lock); goto out_dput; } +spin_unlock (&swap_lock); + + err =__swapoff (p); + +out_dput: +filp_close (victim,NULL ); +return err; +} + +/* +* Drop @p from the avail and active lists and undo its accounting. The +* caller must hold swap_lock and have checked that @p is WRITEOK and not +* pinned for hibernation. +* +* Returns 0, or -ENOMEM with swap_lock still held. +*/ +static int swap_info_remove (struct swap_info_struct *p) +{ +if (!security_vm_enough_memory_mm (current->mm, p->pages)) +vm_unacct_memory (p->pages); +else +return -ENOMEM ; +spin_lock (&p->lock);del_from_avail_list (p, true);plist_del (&p->list, &swap_active_head);atomic_long_sub (p->pages, &nr_swap_pages); total_swap_pages -= p->pages;spin_unlock (&p->lock); -spin_unlock (&swap_lock); +return 0 ; +} + +/* Common swap teardown after list removal; shared by sys_swapoff() and +* xswap_destroy(). +*/ +static int __swapoff (struct swap_info_struct *p) +{ +struct file *swap_file =NULL ; +int err;wait_for_allocation (p);
@@ -3431,7 +3490,7 @@ SYSCALL_DEFINE1(swapoff, const char __user *, specialfile)if (err) {/* re-insert swap space back into swap_list */ reinsert_swap_info (p); - goto out_dput; +return err; }/*
@@ -3471,15 +3530,21 @@ SYSCALL_DEFINE1(swapoff, const char __user *, specialfile)kfree (p->global_cluster); p->global_cluster =NULL ;free_swap_cluster_info (p); +/* +* The device is off swap_active_head and no longer WRITEOK, so no +* reader can observe these; clearing them here needs no lock. +*/ p->max =0 ; p->cluster_info =NULL ; - inode = mapping->host; +if (swap_file) { +struct inode *inode = swap_file->f_mapping->host; -inode_lock (inode); - inode->i_flags &= ~S_SWAPFILE ; -inode_unlock (inode); -filp_close (swap_file,NULL ); +inode_lock (inode); + inode->i_flags &= ~S_SWAPFILE ; +inode_unlock (inode); +filp_close (swap_file,NULL ); + }/* * Clear the SWP_USED flag after all resources are freed so that swapon
@@ -3490,13 +3555,10 @@ SYSCALL_DEFINE1(swapoff, const char __user *, specialfile) p->flags =0 ;spin_unlock (&swap_lock); - err =0 ;atomic_inc (&proc_poll_event);wake_up_interruptible (&proc_poll_wait); -out_dput: -filp_close (victim,NULL ); -return err; +return 0 ; }#ifdef CONFIG_PROC_FS
@@ -4269,6 +4331,35 @@ static int xswap_create(int prio)spin_unlock (&swap_lock); return error; } + +/* Tear down a file-less xswap device by its swap type. */ +static int xswap_destroy (int type) +{ +struct swap_info_struct *p; +int err; + + p =swap_type_to_info (type); +if (!p) +return -EINVAL ; + +spin_lock (&swap_lock); +if (!(p->flags &SWP_WRITEOK ) || !(p->flags &SWP_XSWAP )) { +spin_unlock (&swap_lock); +return -EINVAL ; + } +/* Refuse swapoff while the device is pinned for hibernation */ +if (p->flags &SWP_HIBERNATION ) { +spin_unlock (&swap_lock); +return -EBUSY ; + } + + err =swap_info_remove (p); +spin_unlock (&swap_lock); +if (err) +return err; + +return __swapoff (p); +}#endif /* CONFIG_XSWAP */ SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags)
Signature · 1 line
2.54.0
Baoquan He
reviewerPATCH v5 8/19 mm, swap: require zswap for xswap devices
In reply to Baoquan HeFrom: Baoquan He <[email protected]>
xswap pages live only in zswap. Without zswap, swapout cannot free the page, but still takes a swap entry. So the device consumes swap entries without freeing memory. Fail to create a device when zswap is unavailable.
--- mm/swapfile.c | 4 ++++ 1 file changed, 4 insertions(+)
mm/swapfile.c +4 -0
@@ -4268,6 +4268,10 @@ static int xswap_create(int prio) (prio < 0 || prio > SWAP_FLAG_PRIO_MASK))return -EINVAL ; +/* xswap has no backing store, it relies on zswap. */ +if (!zswap_is_enabled ()) +return -EOPNOTSUPP ; + si =alloc_swap_info ();if (IS_ERR (si))return PTR_ERR (si);
Signature · 1 line
2.54.0
Baoquan He
reviewerPATCH v5 9/19 mm, swap: use a stable id for xswap devices
In reply to Baoquan HeFrom: Baoquan He <[email protected]>
Give each xswap device a 64-bit id from a counter. The id is never reused. Show "xswap<id>" in /proc/swaps and use the same id in debugfs. Destroy matches the id, so an old name fails with -ENODEV instead of hitting a new device.
This avoids names based on the swap type, which can be reused after swapoff. The swap type is still used inside the kernel and in dmesg.
--- include/linux/swap.h | 1 + mm/swapfile.c | 59 ++++++++++++++++++++++++++++++-------------- 2 files changed, 42 insertions(+), 18 deletions(-)
include/linux/swap.h +1 -0
@@ -244,6 +244,7 @@ struct swap_info_struct {struct swap_cluster_info *cluster_info;/* array, one entry per cluster */ #ifdef CONFIG_XSWAP struct vm_struct *cluster_vm;/* VM_SPARSE area for cluster_info */ +u64 xswap_id;/* stable id, shown to userspace */ unsigned long nr_clusters_max;/* total clusters in the xswap address space */ unsigned long nr_clusters_mapped;/* currently mapped cluster count */ struct mutex xswap_lock;/* serialize map/unmap operations */
mm/swapfile.c +41 -18
110 lines not on this page · raw message
Signature · 1 line
2.54.0
[email protected]
AI reviewRe: [PATCH v5 09/19] mm, swap: use a stable id for xswap devices
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider: - [Low] mm: commit message claim about swap type in dmesg contradicts xswap_create() --
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider:
- [Low] mm: commit message claim about swap type in dmesg contradicts xswap_create()
--
commit 46956dad40c329833b6eaa3cc452fe974e0d5929
Author: Baoquan He <[email protected]>
mm, swap: use a stable id for xswap devices
This commit assigns a monotonic 64-bit xswap_id to each xswap device for use in /proc/swaps, debugfs, and xswap_destroy() instead of the reusable swap type.
Quoted 4 lines
mm/swapfile.c +0
-0
[ ... ]
mm/swapfile.c static int xswap_create(int prio) · line 4335
Quoted 11 lines
mm/swapfile.c +2
-2@@ -4316,8 +4328,8 @@ static int xswap_create(int prio) si->list.prio = -si->prio; si->avail_list.prio = -si->prio;enable_swap_info (si); -pr_info ("xswap: adding extendable swap type %d (prio %d, %lu pages)\n" , - si->type, prio, si->pages); +pr_info ("xswap: adding extendable swap device xswap%llu (prio %d, %lu pages)\n" , + (unsigned long long )si->xswap_id, prio, si->pages);mutex_unlock (&swapon_mutex);atomic_inc (&proc_poll_event);wake_up_interruptible (&proc_poll_wait);
[Severity: Low]
Is the commit message statement that "The swap type is still used inside the kernel and in dmesg" accurate? Here xswap_create() switches the dmesg output from si->type to si->xswap_id.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=9
Baoquan He
reviewerPATCH v5 10/19 mm, swap: let the command line set the xswap device size
In reply to Baoquan HeFrom: Baoquan He <[email protected]>
By default, a new xswap device is 200% of RAM, which is twice RAM. Add xswap.max= to set the size of devices created after it:
xswap.max=300% three times RAM xswap.max=8G eight GiB xswap.max=max the largest size the device can address
The default is fine for most users. But a machine with a lot of RAM and a small working set can get a SwapTotal much larger than it needs.
The size is resolved at device creation, not at parameter parsing because a percentage depends on RAM, and totalram_pages() is not ready when parameters are parsed.
An explicit size that is too large or overflows is refused. The default size is clamped to the limit instead. A size below one cluster is also refused. "max" means the largest addressable size, still bounded by the unsigned int cluster index.
--- .../admin-guide/kernel-parameters.txt | 7 + mm/swapfile.c | 130 ++++++++++++++++-- 2 files changed, 127 insertions(+), 10 deletions(-)
Documentation/admin-guide/kernel-parameters.txt +7 -0
@@ -8738,3 +8738,10 @@ Kernel parameters memory, and other data can't be written using xmon commands. off xmon is disabled. + + xswap.max= [KNL,SWAP] + Maximum size of a new xswap device. The value is a + size in bytes with an optional KMGTPE suffix, a + percentage of RAM, or "max" for the largest size + the device can address. + Format: <size>[KMGTPE] | <percent>% | max
mm/swapfile.c +120 -10
161 lines not on this page · raw message
Signature · 1 line
2.54.0
Baoquan He
reviewerPATCH v5 11/19 mm, swap: add CONFIG_XSWAP_DEFAULT_MAX
In reply to Baoquan HeFrom: Baoquan He <[email protected]>
Add a build-time default for the size of a new xswap device. The value can be a percentage of RAM, a size in bytes, or "max". "max" means the largest size the device can address. xswap.max= on the command line overrides this default.
The build-time default is clamped to the device limit. An explicit xswap.max= that is too large is refused.
If the build-time default cannot be parsed, or is a byte size below one cluster, use 200% instead. A percentage default can still be below one cluster after RAM is known. In that case, device creation fails.
Document the sysfs ABI, the stable id, and the synthetic /proc/swaps name. Also document that zswap must stay enabled while xswap devices exist.
--- .../ABI/testing/sysfs-kernel-mm-xswap | 38 +++++++++++++++++++ .../admin-guide/kernel-parameters.txt | 3 +- Documentation/admin-guide/mm/zswap.rst | 9 +++++ MAINTAINERS | 1 + mm/Kconfig | 11 ++++++ mm/swapfile.c | 22 +++++++++-- 6 files changed, 79 insertions(+), 5 deletions(-) create mode 100644 Documentation/ABI/testing/sysfs-kernel-mm-xswap
Documentation/ABI/testing/sysfs-kernel-mm-xswap new file +38 -0
@@ -0,0 +1,38 @@ +What: /sys/kernel/mm/xswap/ +Date: October 2026 +Contact: Linux memory management mailing list <[email protected]> +Description: File-less swap devices backed by zswap. + + An xswap device has no backing file. Its slots are backed + by the zswap pool, and the device size is set at boot with + xswap.max= or CONFIG_XSWAP_DEFAULT_MAX. The size is fixed + for the lifetime of the device. + + The name shown in /proc/swaps, xswap<id>, is synthetic and + is not a path. It cannot be passed to swapoff(2), so + swapoff -a reports an error for these entries. Use the + destroy attribute instead. + + An xswap device requires zswap. Disabling zswap at runtime + (enabled=0, or max_pool_percent=0) while xswap devices + exist leaves them without backing: existing entries can + still be read, but new pages cannot be stored. Destroy the + xswap devices before disabling zswap. + +What: /sys/kernel/mm/xswap/create +Date: October 2026 +Contact: Linux memory management mailing list <[email protected]> +Description: Create an xswap device. + + Write an optional priority (0..32767), or write nothing for + the default priority. The write fails if zswap is disabled + or if the configured size is smaller than one cluster. + +What: /sys/kernel/mm/xswap/destroy +Date: October 2026 +Contact: Linux memory management mailing list <[email protected]> +Description: Destroy an xswap device. + + Write the device's stable id. The id is the number after + "xswap" in /proc/swaps, for example 1 for xswap1. It is + not a swap type and is never reused.
Documentation/admin-guide/kernel-parameters.txt +2 -1
@@ -8743,5 +8743,6 @@ Kernel parameters Maximum size of a new xswap device. The value is a size in bytes with an optional KMGTPE suffix, a percentage of RAM, or "max" for the largest size - the device can address. + the device can address. The default is set by + CONFIG_XSWAP_DEFAULT_MAX. Format: <size>[KMGTPE] | <percent>% | max
Documentation/admin-guide/mm/zswap.rst +9 -0
@@ -136,3 +136,12 @@ selected. A debugfs interface is provided for various statistic about pool size, number of pages stored, same-value filled pages and various counters for the reasons pages are rejected. + +xswap devices +------------- + +An xswap device is a file-less swap device backed by zswap. While such a +device exists, do not disable zswap at runtime (``enabled=0``) or set +``max_pool_percent`` to 0. The device would have no backing: pages already +in zswap can still be read, but new pages cannot be stored. Destroy the +xswap devices first.
MAINTAINERS +1 -0
@@ -17910,6 +17910,7 @@ R: Youngjun Park <[email protected]> L: [email protected] S: Maintained F: Documentation/ABI/testing/sysfs-kernel-mm-swap +F: Documentation/ABI/testing/sysfs-kernel-mm-xswap F: Documentation/mm/swap-table.rst F: include/linux/swap.h F: include/linux/swap_ops.h
mm/Kconfig +11 -0
@@ -131,6 +131,17 @@ config XSWAP array is backed by a sparse vmalloc area that is mapped lazily in chunks, avoiding static pre-allocation overhead. +config XSWAP_DEFAULT_MAX + string "Default maximum size of a new xswap device" + depends on XSWAP + default "200%" + help + The maximum size of a newly created xswap device. It accepts a + percentage of RAM (for example "200%"), a size (for example "2T"), + or "max" for the largest size the device can address. + + xswap.max= on the kernel command line overrides this value. + config ZSMALLOC tristate
mm/swapfile.c +18 -4
@@ -161,9 +161,12 @@ static bool xswap_size_max;static bool xswap_size_set;/* xswap.max= was given */ /* -* Parse "200%", "2T" or "max". @str is not modified. +* Parse "200%", "2T" or "max". @str is not modified. @cmdline tells an +* explicit xswap.max= from the build-time default, which is clamped +* instead of refused when it is too large. */ -static int __init xswap_parse_size (const char *str,const char *what) +static int __init xswap_parse_size (const char *str,const char *what, +bool cmdline) {const char *pct;char buf[16 ];
@@ -192,12 +195,16 @@ static int __init xswap_parse_size(const char *str, const char *what) bytes =memparse (str, &end);if (*end || bytes <PAGE_SIZE ) goto bad; +/* A default below one cluster would fail every create. */ +if (!cmdline && bytes < (unsigned long )SWAPFILE_CLUSTER *PAGE_SIZE ) + goto bad; xswap_size_bytes = bytes; xswap_size_pct =0 ; xswap_size_max = false; } - xswap_size_set = true; +if (cmdline) + xswap_size_set = true;return 0 ; bad:pr_err ("xswap: bad %s=%s\n" , what, str);
@@ -206,7 +213,7 @@ static int __init xswap_parse_size(const char *str, const char *what)static int __init xswap_max_setup (char *str) { -xswap_parse_size (str,"xswap.max" ); +xswap_parse_size (str,"xswap.max" , true);return 1 ; }__setup ("xswap.max=" , xswap_max_setup);
@@ -4830,6 +4837,13 @@ static int __init swapfile_init(void) { swapfile_maximum_size =arch_max_swapfile_size (); +#ifdef CONFIG_XSWAP +/* xswap.max= on the command line overrides the build-time default. */ +if (!xswap_size_set) +xswap_parse_size (CONFIG_XSWAP_DEFAULT_MAX , +"CONFIG_XSWAP_DEFAULT_MAX" , false); +#endif +/* * Once a cluster is freed, it's swap table content is read * only, and all swap cache readers (swap_cache_*) verifies
Signature · 1 line
2.54.0
Baoquan He
reviewerPATCH v5 12/19 mm, swap: let swapoff destroy xswap devices by name
In reply to Baoquan HeFrom: Baoquan He <[email protected]>
An xswap device has no backing file, so /proc/swaps shows a synthetic name, xswap<id>, that file_open_name() cannot open. Teach swapoff to fall back to xswap_destroy() when the name is exactly "xswap<digits>" and the lookup failed with -ENOENT.
A real file or directory keeps precedence; other open errors are returned unchanged, so a file that exists but cannot be opened is not mistaken for an xswap device. The ABI documentation is updated, and the destroy attribute remains the unambiguous teardown interface.
A well-formed name for an id that does not exist asks xswap_destroy() for a device that is not there. It returns -ENODEV; map that to -ENOENT so swapoff reports a missing name like it does for a missing file.
--- .../ABI/testing/sysfs-kernel-mm-xswap | 11 +++- mm/swapfile.c | 56 ++++++++++++++++++- 2 files changed, 62 insertions(+), 5 deletions(-)
Documentation/ABI/testing/sysfs-kernel-mm-xswap +8 -3
@@ -9,9 +9,14 @@ Description: File-less swap devices backed by zswap. for the lifetime of the device. The name shown in /proc/swaps, xswap<id>, is synthetic and - is not a path. It cannot be passed to swapoff(2), so - swapoff -a reports an error for these entries. Use the - destroy attribute instead. + is not a path. swapoff(2) accepts it, so both "swapoff + xswap<id>" and swapoff -a tear the device down. A real + file or directory with that name takes precedence: if it + opens, it is looked up as a swap file and the synthetic + device is not used. Only a name that cannot be opened + because it does not exist, including a dangling symlink, + is treated as an xswap device. Use the destroy attribute + for an unambiguous teardown. An xswap device requires zswap. Disabling zswap at runtime (enabled=0, or max_pool_percent=0) while xswap devices
mm/swapfile.c +54 -2
86 lines not on this page · raw message
Signature · 1 line
2.54.0
[email protected]
AI reviewRe: [PATCH v5 12/19] mm, swap: let swapoff destroy xswap devices by name
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider: - [High] mm, swap: ERR_PTR(-ENOENT) dereference in sys_swapoff() on empty path --
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider:
- [High] mm, swap: ERR_PTR(-ENOENT) dereference in sys_swapoff() on empty path
--
commit ae71026877d501dba7d03d0b98df83479aa3a6b8
Author: Baoquan He <[email protected]>
mm, swap: let swapoff destroy xswap devices by name
This commit teaches swapoff() to fall back to destroying an xswap device when file_open_name() fails with -ENOENT and the name matches "xswap<digits>".
Quoted 4 lines
mm/swapfile.c +0
-0
[ ... ]
Quoted 7 lines
mm/swapfile.c +1
-2@@ -3513,8 +3520,21 @@ SYSCALL_DEFINE1(swapoff, const char __user *, specialfile)CLASS (filename, pathname)(specialfile); victim =file_open_name (pathname,O_RDWR |O_LARGEFILE ,0 ); -if (IS_ERR (victim)) -return PTR_ERR (victim); +if (IS_ERR (victim)) {
mm/swapfile.c SYSCALL_DEFINE1(swapoff, const char __user *, specialfile) · line 3533
Quoted 10 lines
mm/swapfile.c +10
-0+/* +* An xswap device has no file, so /proc/swaps shows a name +* that cannot be opened. Accept it here so swapoff -a works. +* Only a missing name is taken for a device; a real file or +* directory that failed for another reason must not match. +*/ +if (PTR_ERR (victim) != -ENOENT ) +return PTR_ERR (victim); + + err =xswap_destroy_by_name (pathname->name);
[Severity: High]
Can this dereference an error pointer in pathname->name when specialfile is an empty string?
When specialfile is "", getname() returns ERR_PTR(-ENOENT) in pathname, and file_open_name() -> do_file_open() returns ERR_CAST(pathname) in victim. Because PTR_ERR(victim) is -ENOENT, swapoff() then evaluates pathname->name without checking IS_ERR(pathname) and faults on ERR_PTR(-ENOENT).
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=12
Baoquan He
reviewerPATCH v5 13/19 mm, swap: add xswap grow trigger on cluster allocation
In reply to Baoquan HeFrom: Baoquan He <[email protected]>
Grow the mapped xswap range before it runs out. When usage reaches 85% of the mapped range, map more clusters from cluster_alloc_swap_entry().
Grow to about 65% usage. This leaves free space in the mapped range before the next grow at 85%, so one grow is not immediately followed by another.
Mapping pages can sleep. Grow before the allocation that needs them, not after it fails. xswap_grow() drops the per-CPU cluster lock around xswap_map_clusters(), because mapping can sleep and the lock only protects the per-CPU cache. xswap is always SWP_SOLIDSTATE, so this path does not use global_cluster_lock.
Link the new clusters even if swapoff cleared SWP_WRITEOK while mapping. The caller keeps the device alive, and the allocation path checks the flag before taking a cluster. If try_to_unuse() then fails, the reinserted device needs these clusters; otherwise they go away with the device.
After a failed scan, retry the free list once. A concurrent free or grow may have added clusters.
--- mm/swapfile.c | 100 ++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 100 insertions(+)
mm/swapfile.c +100 -0
@@ -80,7 +80,18 @@ max_t(unsigned long, \ ((XSWAP_UNMAP_BATCH_PAGES * PAGE_SIZE) / \ sizeof(struct swap_cluster_info)), 16) +/* +* Grow thresholds, as a percentage of the mapped range in use. The target +* sits above the point where the shrink starts looking (50%) and below the +* next grow trigger (85%), so a grow cannot leave the range in a state the +* shrink would undo. +*/ +#define XSWAP_GROW_UNTIL 65 +#define XSWAP_GROW_WHEN 85 +#define XSWAP_GROW_CHUNKS 4 +static bool xswap_should_grow (struct swap_info_struct *si); +static unsigned long xswap_grow (struct swap_info_struct *si);static int xswap_map_clusters (struct swap_info_struct *si,unsigned long start_idx,unsigned long nr); static void xswap_unmap_clusters(struct swap_info_struct *si,
@@ -1359,6 +1370,18 @@ static unsigned long cluster_alloc_swap_entry(struct swap_info_struct *si,if (order && !(si->flags &SWP_BLKDEV ))return 0 ; +#ifdef CONFIG_XSWAP +/* +* Top the range up early. Every path below leaves through `done`, +* so this has to come first; mapping pages can sleep, and doing it +* from the allocation that would otherwise fail leaves nothing to +* fall back on. +*/ +if ((si->flags &SWP_XSWAP ) && (si->flags &SWP_WRITEOK ) && +xswap_should_grow (si)) +xswap_grow (si); +#endif +if (!(si->flags &SWP_SOLIDSTATE )) {/* Serialize HDD SWAP allocation for each device. */ spin_lock (&si->global_cluster_lock);
@@ -1435,6 +1458,12 @@ static unsigned long cluster_alloc_swap_entry(struct swap_info_struct *si,if (found) goto done; } + +#ifdef CONFIG_XSWAP +/* A concurrent free or grow may have added clusters; retry once. */ +if (!found && (si->flags &SWP_XSWAP )) + found =alloc_swap_scan_list (si, &si->free_clusters, folio, false); +#endif done:if (!(si->flags &SWP_SOLIDSTATE ))spin_unlock (&si->global_cluster_lock);
@@ -4168,6 +4197,77 @@ static void xswap_unmap_range(struct swap_info_struct *si,__free_page (pages[i]); } +static bool xswap_should_grow (struct swap_info_struct *si) +{ +unsigned long mapped =READ_ONCE (si->nr_clusters_mapped); + +if (mapped >=READ_ONCE (si->nr_clusters_max)) +return false; + +return swap_usage_in_pages (si) *100 >= + mapped *SWAPFILE_CLUSTER *XSWAP_GROW_WHEN ; +} + +/* +* Map more of the cluster_info array, up to GROW_UNTIL in use. The caller +* holds percpu_swap_cluster.lock, which this drops while it sleeps. +*/ +static unsigned long xswap_grow (struct swap_info_struct *si) +{ +unsigned long mapped =READ_ONCE (si->nr_clusters_mapped); +unsigned long nr_new, want, i; +int ret; + +/* The unmap below is paired with the caller's lock. */ +lockdep_assert_held (&this_cpu_ptr (&percpu_swap_cluster)->lock); + + want =DIV_ROUND_UP (swap_usage_in_pages (si) *100 , +XSWAP_GROW_UNTIL *SWAPFILE_CLUSTER ); +if (want <= mapped) +return 0 ; + + nr_new =rounddown (want - mapped,XSWAP_GROW_CLUSTERS ); +if (!nr_new) + nr_new =XSWAP_GROW_CLUSTERS ; +if (nr_new >XSWAP_GROW_CHUNKS *XSWAP_GROW_CLUSTERS ) + nr_new =XSWAP_GROW_CHUNKS *XSWAP_GROW_CLUSTERS ; + nr_new =min (nr_new,READ_ONCE (si->nr_clusters_max) - mapped); +if (!nr_new) +return 0 ; + +/* +* Mapping pages can sleep. The lock only guards the per-cpu +* cluster cache, which this path does not touch. +*/ +local_unlock (&percpu_swap_cluster.lock); + ret =xswap_map_clusters (si, mapped, nr_new); +local_lock (&percpu_swap_cluster.lock); +if (ret) +return 0 ; + +/* +* Link the clusters even if swapoff cleared SWP_WRITEOK while we +* mapped. The caller keeps the device alive, and nothing takes a +* cluster while WRITEOK is clear. If try_to_unuse() fails, the +* reinserted device needs them. +*/ +for (i = mapped; i < mapped + nr_new; i++) { +struct swap_cluster_info *ci = &si->cluster_info[i]; + +/* +* A concurrent grower may have taken these already; +* only add the off-list ones. +*/ +spin_lock (&ci->lock); +if (ci->flags ==CLUSTER_FLAG_NONE ) +move_cluster (si, ci, &si->free_clusters, +CLUSTER_FLAG_FREE ); +spin_unlock (&ci->lock); + } + +return nr_new; +} +/* Caller must hold si->xswap_lock. Cannot fail. */ static void xswap_unmap_clusters (struct swap_info_struct *si,unsigned long start_idx,unsigned long nr)
Signature · 1 line
2.54.0
[email protected]
AI reviewRe: [PATCH v5 13/19] mm, swap: add xswap grow trigger on cluster allocation
Thank you for your contribution! Sashiko AI review found 2 potential issues to consider: - [Medium] mm, swap: premature nr_clusters_mapped update in xswap_map_clusters() - [Medium] mm, swap: cluster…
Thank you for your contribution! Sashiko AI review found 2 potential issues to consider:
- [Medium] mm, swap: premature nr_clusters_mapped update in xswap_map_clusters()
- [Medium] mm, swap: cluster exhaustion below 85% usage in xswap_should_grow()
--
commit 691004e6412209ac9359059a41013d8433ee5087
Author: Baoquan He <[email protected]>
mm, swap: add xswap grow trigger on cluster allocation
This commit adds on-demand growth of the mapped xswap cluster_info range from cluster_alloc_swap_entry() when swap usage reaches 85% of the currently mapped range, growing it back down to 65% usage.
Quoted 4 lines
mm/swapfile.c +0
-0
[ ... ]
Quoted 15 lines
mm/swapfile.c +11
-0@@ -1359,6 +1370,18 @@ static unsigned long cluster_alloc_swap_entry(struct swap_info_struct *si,if (order && !(si->flags &SWP_BLKDEV ))return 0 ; +#ifdef CONFIG_XSWAP +/* +* Top the range up early. Every path below leaves through `done`, +* so this has to come first; mapping pages can sleep, and doing it +* from the allocation that would otherwise fail leaves nothing to +* fall back on. +*/ +if ((si->flags &SWP_XSWAP ) && (si->flags &SWP_WRITEOK ) && +xswap_should_grow (si)) +xswap_grow (si); +#endif
[ ... ]
Quoted 17 lines
mm/swapfile.c +6
-0@@ -1435,6 +1458,12 @@ static unsigned long cluster_alloc_swap_entry(struct swap_info_struct *si,if (found) goto done; } + +#ifdef CONFIG_XSWAP +/* A concurrent free or grow may have added clusters; retry once. */ +if (!found && (si->flags &SWP_XSWAP )) + found =alloc_swap_scan_list (si, &si->free_clusters, folio, false); +#endif done:if (!(si->flags &SWP_SOLIDSTATE ))spin_unlock (&si->global_cluster_lock);@@ -4168,6 +4197,77 @@ static void xswap_unmap_range(struct swap_info_struct *si,__free_page (pages[i]); }
mm/swapfile.c static void xswap_unmap_range(struct swap_info_struct *si, · line 4209
Quoted 10 lines
mm/swapfile.c +10
-0+static bool xswap_should_grow (struct swap_info_struct *si) +{ +unsigned long mapped =READ_ONCE (si->nr_clusters_mapped); + +if (mapped >=READ_ONCE (si->nr_clusters_max)) +return false; + +return swap_usage_in_pages (si) *100 >= + mapped *SWAPFILE_CLUSTER *XSWAP_GROW_WHEN ; +}
[Severity: Medium]
Can cluster_alloc_swap_entry() run out of usable clusters and fail with SWAP_ENTRY_INVALID while swap_usage_in_pages(si) is still below 85% of the mapped range?
Each CPU that misses in swap_alloc_fast() isolates a whole 512-page cluster from si->free_clusters or si->nonfull_clusters, allocates a single slot, moves the cluster to si->frag_clusters, and caches it in percpu_swap_cluster. When the number of concurrently allocating CPUs reaches nr_clusters_mapped (initially at most XSWAP_GROW_CLUSTERS, i.e. 16 to 85 clusters), or when more than 15% of mapped clusters are held in per-CPU caches, si->free_clusters and si->nonfull_clusters are drained while every cluster in si->frag_clusters is isolated or locked by its owning CPU.
Because total page usage can still be well below 85% (and below the 65% XSWAP_GROW_UNTIL check in xswap_grow()), xswap_should_grow() returns false and the fallback at the end of cluster_alloc_swap_entry() only retries si->free_clusters without growing the mapped range, causing folio_alloc_swap() to fail prematurely even though unmapped clusters remain available.
Quoted 50 lines
mm/swapfile.c +50
-0+ +/* +* Map more of the cluster_info array, up to GROW_UNTIL in use. The caller +* holds percpu_swap_cluster.lock, which this drops while it sleeps. +*/ +static unsigned long xswap_grow (struct swap_info_struct *si) +{ +unsigned long mapped =READ_ONCE (si->nr_clusters_mapped); +unsigned long nr_new, want, i; +int ret; + +/* The unmap below is paired with the caller's lock. */ +lockdep_assert_held (&this_cpu_ptr (&percpu_swap_cluster)->lock); + + want =DIV_ROUND_UP (swap_usage_in_pages (si) *100 , +XSWAP_GROW_UNTIL *SWAPFILE_CLUSTER ); +if (want <= mapped) +return 0 ; + + nr_new =rounddown (want - mapped,XSWAP_GROW_CLUSTERS ); +if (!nr_new) + nr_new =XSWAP_GROW_CLUSTERS ; +if (nr_new >XSWAP_GROW_CHUNKS *XSWAP_GROW_CLUSTERS ) + nr_new =XSWAP_GROW_CHUNKS *XSWAP_GROW_CLUSTERS ; + nr_new =min (nr_new,READ_ONCE (si->nr_clusters_max) - mapped); +if (!nr_new) +return 0 ; + +/* +* Mapping pages can sleep. The lock only guards the per-cpu +* cluster cache, which this path does not touch. +*/ +local_unlock (&percpu_swap_cluster.lock); + ret =xswap_map_clusters (si, mapped, nr_new); +local_lock (&percpu_swap_cluster.lock); +if (ret) +return 0 ; + +/* +* Link the clusters even if swapoff cleared SWP_WRITEOK while we +* mapped. The caller keeps the device alive, and nothing takes a +* cluster while WRITEOK is clear. If try_to_unuse() fails, the +* reinserted device needs them. +*/ +for (i = mapped; i < mapped + nr_new; i++) { +struct swap_cluster_info *ci = &si->cluster_info[i]; + +/* +* A concurrent grower may have taken these already; +* only add the off-list ones.
mm/swapfile.c static void xswap_unmap_range(struct swap_info_struct *si, · line 4269
Quoted 10 lines
mm/swapfile.c +10
-0+*/ +spin_lock (&ci->lock); +if (ci->flags ==CLUSTER_FLAG_NONE ) +move_cluster (si, ci, &si->free_clusters, +CLUSTER_FLAG_FREE ); +spin_unlock (&ci->lock); + } + +return nr_new; +}
[Severity: Medium]
Can concurrent swap allocations fail in cluster_alloc_swap_entry() between xswap_map_clusters() updating si->nr_clusters_mapped and xswap_grow() linking the new clusters onto si->free_clusters?
xswap_map_clusters() publishes the new si->nr_clusters_mapped and drops si->xswap_lock before returning to xswap_grow():
mm/swapfile.c:xswap_map_clusters() {
...
smp_store_release(&si->nr_clusters_mapped, start_idx + nr);
mutex_unlock(&si->xswap_lock);
return 0;
...
}Until the winning grower re-acquires percpu_swap_cluster.lock and runs the move_cluster() loop, the newly mapped clusters remain off-list with CLUSTER_FLAG_NONE:
CPU0 CPU1cluster_alloc_swap_entry() cluster_alloc_swap_entry()
xswap_grow()
xswap_map_clusters()
mutex_lock(&si->xswap_lock)
...
smp_store_release(
&si->nr_clusters_mapped, ...)
mutex_unlock(&si->xswap_lock)
xswap_should_grow() -> false
/* scans cluster lists: empty */
alloc_swap_scan_list(
&si->free_clusters) -> 0
<-- returns SWAP_ENTRY_INVALID
local_lock(...)
move_cluster(...,
&si->free_clusters, ...)Similarly, a concurrent caller that already entered xswap_grow() and waited on si->xswap_lock wakes up, sees start_idx != si->nr_clusters_mapped, gets -EAGAIN from xswap_map_clusters(), and returns 0 from xswap_grow() before CPU0 links the clusters. When existing clusters are exhausted during the grow, these concurrent allocations return SWAP_ENTRY_INVALID despite the newly mapped capacity. Linking the new clusters onto si->free_clusters (or updating si->nr_clusters_mapped after linking) while still serialized by si->xswap_lock would avoid this window.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=13
Baoquan He
reviewerPATCH v5 14/19 mm, swap: add debugfs counters for xswap
In reply to Baoquan HeFrom: Baoquan He <[email protected]>
The mapped range and the grow count are not visible from userspace. This makes it hard to check if the grow thresholds work: we cannot see how large the mapped range is, how much it grows, or how much free tail it keeps.
Put this information in debugfs. It is a debugging aid, not a stable interface, so it does not belong in sysfs or the ABI:
# cat /sys/kernel/debug/xswap/xswap1 type 0 clusters_max 2038783 clusters_mapped 292 usage_pages 91456 tail_free 16 grows 3 shrinks 0
The shrink counter is initialized here. It will start to increase when the shrink path is added.
--- include/linux/swap.h | 3 ++ mm/swapfile.c | 78 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 81 insertions(+)
include/linux/swap.h +3 -0
@@ -248,6 +248,9 @@ struct swap_info_struct {unsigned long nr_clusters_max;/* total clusters in the xswap address space */ unsigned long nr_clusters_mapped;/* currently mapped cluster count */ struct mutex xswap_lock;/* serialize map/unmap operations */ +struct dentry *xswap_debugfs;/* mapped range and counters */ +atomic_long_t xswap_grows;/* grow operations so far */ +atomic_long_t xswap_shrinks;/* shrink operations so far */ #endif struct list_head free_clusters;/* free clusters list */ struct list_head full_clusters;/* full clusters list */
mm/swapfile.c +78 -0
126 lines not on this page · raw message
Signature · 1 line
2.54.0
Baoquan He
reviewerPATCH v5 15/19 mm, swap: make xswap cluster_info accesses RCU-safe
In reply to Baoquan HeFrom: Baoquan He <[email protected]>
An xswap cluster_info page can be unmapped at runtime. The shrink path that does this is added by a later patch; this patch establishes the contract it relies on. The allocation and reclaim paths drop ci->lock to reclaim swap cache, then take it again; in that window the cluster can be freed and its cluster_info page can go away. Cover the whole read/drop/reacquire window with RCU, so the cluster pointer stays valid until the lock is taken again.
Also take RCU in swap_alloc_fast() and the hibernation slot allocation path, which read cluster_info through a cached offset without holding ci->lock. A cached offset can outlive a shrink that did not see it, so swap_alloc_fast() now loads nr_clusters_mapped with smp_load_acquire() and fails the fast path when the offset is beyond it.
On the writer side, xswap_unmap_clusters() publishes the smaller range with smp_store_release() before the memory goes away, invalidates the per-cpu cluster cache, and waits one RCU grace period. A reader that already loaded a cached offset is then either seen by the cache flush or waited out before the PTEs are cleared.
--- mm/swapfile.c | 82 +++++++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 73 insertions(+), 9 deletions(-)
mm/swapfile.c +73 -9
@@ -1239,6 +1239,9 @@ static unsigned long alloc_swap_scan_cluster(struct swap_info_struct *si,lockdep_assert_held (&ci->lock);VM_WARN_ON (!cluster_is_usable (ci, order)); +/* ci is used without ci->lock; an xswap unmap waits for this. */ +rcu_read_lock (); +if (end < nr_pages || ci->count + nr_pages >SWAPFILE_CLUSTER ) goto out;
@@ -1267,6 +1270,7 @@ static unsigned long alloc_swap_scan_cluster(struct swap_info_struct *si, out:relocate_cluster (si, ci);swap_cluster_unlock (ci); +rcu_read_unlock ();if (si->flags &SWP_SOLIDSTATE ) {this_cpu_write (percpu_swap_cluster.offset[order], next);this_cpu_write (percpu_swap_cluster.si[order], si);
@@ -1310,6 +1314,9 @@ static void swap_reclaim_full_clusters(struct swap_info_struct *si, bool force) to_scan =swap_usage_in_pages (si) /SWAPFILE_CLUSTER ;while ((ci =isolate_lock_cluster (si, &si->full_clusters))) { +/* As in alloc_swap_scan_cluster(). */ +rcu_read_lock (); + offset =cluster_offset (si, ci); end =min (si->max, offset +SWAPFILE_CLUSTER ); to_scan--;
@@ -1334,6 +1341,7 @@ static void swap_reclaim_full_clusters(struct swap_info_struct *si, bool force)relocate_cluster (si, ci);swap_cluster_unlock (ci); +rcu_read_unlock ();if (to_scan <=0 )break ;
@@ -1664,11 +1672,36 @@ static bool swap_alloc_fast(struct folio *folio)/* * Once allocated, swap_info_struct will never be completely freed, * so checking it's liveness by get_swap_device_info is enough. +* +* The cached offset indexes si->cluster_info, which xswap can +* unmap; cover both the read and the use with RCU. */ +rcu_read_lock (); si =this_cpu_read (percpu_swap_cluster.si[order]); offset =this_cpu_read (percpu_swap_cluster.offset[order]); -if (!si || !offset || !get_swap_device_info (si)) +if (!si || !offset || !get_swap_device_info (si)) { +rcu_read_unlock ();return false; + } +#ifdef CONFIG_XSWAP +if (si->flags &SWP_XSWAP ) { +/* +* The cached offset can outlive a shrink that did not see +* it, and the cluster_info page it indexes may be unmapped. +* Fail the fast path; the slow path picks a live cluster. +*/ +unsigned long mapped; + +/* Pairs with the release store in xswap_unmap_clusters(). */ + mapped =smp_load_acquire (&si->nr_clusters_mapped); + +if (offset /SWAPFILE_CLUSTER >= mapped) { +put_swap_device (si); +rcu_read_unlock (); +return false; + } + } +#endif ci =swap_cluster_lock (si, offset);if (cluster_is_usable (ci, order)) {
@@ -1680,6 +1713,7 @@ static bool swap_alloc_fast(struct folio *folio) }put_swap_device (si); +rcu_read_unlock ();return folio_test_swapcache (folio); }
@@ -2472,8 +2506,11 @@ swp_entry_t swap_alloc_hibernation_slot(int type)/* * Try the local cluster first if it matches the device. If * not, try grab a new cluster and override local cluster. +* +* Same RCU requirement as swap_alloc_fast(). */ local_lock (&percpu_swap_cluster.lock); +rcu_read_lock (); pcp_si =this_cpu_read (percpu_swap_cluster.si[0 ]); pcp_offset =this_cpu_read (percpu_swap_cluster.offset[0 ]);if (pcp_si == si && pcp_offset) {
@@ -2483,6 +2520,7 @@ swp_entry_t swap_alloc_hibernation_slot(int type)else swap_cluster_unlock (ci); } +rcu_read_unlock ();if (!offset) offset =cluster_alloc_swap_entry (si,NULL );local_unlock (&percpu_swap_cluster.lock);
@@ -3445,7 +3483,13 @@ static void wait_for_allocation(struct swap_info_struct *si)#ifdef CONFIG_XSWAP if (si->flags &SWP_XSWAP ) { -/* Pairs with the smp_store_release() in xswap_map_clusters(). */ +/* +* Pairs with the smp_store_release() in xswap_map_clusters(). +* A grow may publish after this read, but no slot comes from +* those clusters once WRITEOK is clear. Cached clusters are +* inside the range; try_to_unuse()'s rescan catches any slot +* taken after the walk. +*/ end =min (end,smp_load_acquire (&si->nr_clusters_mapped) *SWAPFILE_CLUSTER ); }
@@ -3527,9 +3571,9 @@ static void flush_percpu_swap_cluster(struct swap_info_struct *si)for_each_possible_cpu (cpu) { pcp_si =per_cpu_ptr (percpu_swap_cluster.si, cpu);/* -* Invalidate the percpu swap cluster cache, si->users -* is dead, so no new user will point to it, just flush -* any existing user. +* Invalidate the per-cpu swap cluster cache. Any reader that +* has already loaded an entry is waited out by the caller's +* synchronize_rcu(). */ for (i =0 ; i <SWAP_NR_ORDERS ; i++)cmpxchg (&pcp_si[i], si,NULL );
@@ -4289,12 +4333,33 @@ static void xswap_unmap_clusters(struct swap_info_struct *si,mutex_lock (&si->xswap_lock); +/* +* Publish the smaller range before its memory goes away. Readers +* that use nr_clusters_mapped without xswap_lock pair with this +* store using smp_load_acquire(). +*/ +smp_store_release (&si->nr_clusters_mapped, start_idx); +if (vm_start >= vm_end) { -WRITE_ONCE (si->nr_clusters_mapped, start_idx); +/* +* The clusters sit inside one page that the preceding +* cluster shares, so there is nothing to unmap. The page +* goes away with that cluster; publishing the smaller +* cluster count is still correct. +*/ mutex_unlock (&si->xswap_lock);return ; } +/* +* A per-cpu cluster cache can still hold an offset in this range. +* Invalidate those references, then wait out the readers that have +* already loaded one, so that nobody can dereference cluster_info +* past this point. swapoff() needs the same before it releases. +*/ +flush_percpu_swap_cluster (si); +synchronize_rcu (); + npages = (vm_end - vm_start) >>PAGE_SHIFT ;if (npages <=totalram_pages ()) {
@@ -4306,7 +4371,7 @@ static void xswap_unmap_clusters(struct swap_info_struct *si,if (pages) {xswap_unmap_range (si, vm_start, vm_end, pages, npages);kvfree (pages); - goto out_store; + goto out_unlock; } }
@@ -4325,8 +4390,7 @@ static void xswap_unmap_clusters(struct swap_info_struct *si,xswap_unmap_range (si, bstart, bend, batch_pages,ARRAY_SIZE (batch_pages)); } -out_store: -WRITE_ONCE (si->nr_clusters_mapped, start_idx); +out_unlock:mutex_unlock (&si->xswap_lock); }
Signature · 1 line
2.54.0
[email protected]
AI reviewRe: [PATCH v5 15/19] mm, swap: make xswap cluster_info accesses RCU-safe
Thank you for your contribution! Sashiko AI review found 4 potential issues to consider: - [High] mm, swap: missing RCU protection in swap_put_entries_cluster() reclaim - [High] mm, swap…
Thank you for your contribution! Sashiko AI review found 4 potential issues to consider:
- [High] mm, swap: missing RCU protection in swap_put_entries_cluster() reclaim
- [High] mm, swap: xswap_unmap_clusters() skips RCU sync when vm_start
Quoted 1 line
=
vm_end
- [High] mm, swap: relocate_cluster() after lock drop re-adds freed cluster
- [High] mm, swap: unmapped xswap cluster_info access in swapin and table alloc paths
--
commit 167e6417672879cf6894286f3431ad8960ae663b
Author: Baoquan He <[email protected]>
mm, swap: make xswap cluster_info accesses RCU-safe
This commit adds RCU read-side protection, per-CPU cluster cache flushing, and acquire/release ordering on nr_clusters_mapped so that runtime xswap shrinking can safely unmap cluster_info pages.
Quoted 15 lines
mm/swapfile.c +3
-0@@ -1239,6 +1239,9 @@ static unsigned long alloc_swap_scan_cluster(struct swap_info_struct *si,lockdep_assert_held (&ci->lock);VM_WARN_ON (!cluster_is_usable (ci, order)); +/* ci is used without ci->lock; an xswap unmap waits for this. */ +rcu_read_lock (); +if (end < nr_pages || ci->count + nr_pages >SWAPFILE_CLUSTER ) goto out;
mm/swapfile.c static unsigned long alloc_swap_scan_cluster(struct swap_info_struct *si, · line 1273
Quoted 4 lines
mm/swapfile.c +1
-0@@ -1267,6 +1270,7 @@ static unsigned long alloc_swap_scan_cluster(struct swap_info_struct *si, out:relocate_cluster (si, ci);swap_cluster_unlock (ci); +rcu_read_unlock ();
[Severity: High]
Can relocate_cluster() re-add a cluster to si->free_clusters after it was freed and isolated while ci->lock was dropped?
CPU0 CPU1alloc_swap_scan_cluster()
rcu_read_lock()
cluster_reclaim_range()
spin_unlock(&ci->lock)
__try_to_reclaim_swap()
/* last slot freed */
__free_cluster()
ci->table = NULL
ci->flags = CLUSTER_FLAG_FREE
/* shrink or swap_cluster_populate()
* isolates ci from free_clusters */
ci->flags = CLUSTER_FLAG_NONE
xswap_unmap_clusters()
synchronize_rcu()
spin_lock(&ci->lock)
/* !cluster_is_usable(ci) ->
* *usable = false, goto out */ relocate_cluster(si, ci)
/* !ci->count &&
* ci->flags != CLUSTER_FLAG_FREE */
free_cluster(si, ci)
__free_cluster(si, ci) <-- NULL deref or list corruption
swap_cluster_unlock(ci)
rcu_read_unlock()
vm_area_unmap_pages()
When __free_cluster() runs a second time on a cluster whose table is already NULL, swap_cluster_assert_empty() dereferences NULL under CONFIG_DEBUG_VM, and move_cluster() links the isolated cluster back onto si->free_clusters just before its backing page is unmapped. swap_reclaim_full_clusters() has the same issue at the end of its loop when checking ci->flags == CLUSTER_FLAG_NONE without verifying cluster_table_is_alloced(ci).
[ ... ]
Quoted 11 lines
mm/swapfile.c +3
-0@@ -1310,6 +1314,9 @@ static void swap_reclaim_full_clusters(struct swap_info_struct *si, bool force) to_scan =swap_usage_in_pages (si) /SWAPFILE_CLUSTER ;while ((ci =isolate_lock_cluster (si, &si->full_clusters))) { +/* As in alloc_swap_scan_cluster(). */ +rcu_read_lock (); + offset =cluster_offset (si, ci); end =min (si->max, offset +SWAPFILE_CLUSTER ); to_scan--;
mm/swapfile.c static void swap_reclaim_full_clusters(struct swap_info_struct *si, bool force) · line 1344
Quoted 4 lines
mm/swapfile.c +1
-0@@ -1334,6 +1341,7 @@ static void swap_reclaim_full_clusters(struct swap_info_struct *si, bool force)relocate_cluster (si, ci);swap_cluster_unlock (ci); +rcu_read_unlock ();
[Severity: High]
Does swap_put_entries_cluster() need the same RCU protection around its __try_to_reclaim_swap() loop?
mm/swapfile.c:swap_put_entries_cluster() {
...
if (ci_batch != -1)
__swap_cluster_free_entries(si, ci, ci_batch, ci_off - ci_batch);
swap_cluster_unlock(ci);
if (!need_reclaim || !reclaim_cache)
return;
do {
nr_reclaimed = __try_to_reclaim_swap(si, offset,
TTRS_UNMAPPED | TTRS_FULL);
offset++;
if (nr_reclaimed)
offset = round_up(offset, abs(nr_reclaimed));
} while (offset < end);
}Callers such as shmem_free_swap() -> swap_put_entries_direct() invoke swap_put_entries_cluster() in preemptible context outside RCU. Once __swap_cluster_put_entry() drops the swap count of cached slots to 0 and swap_cluster_unlock(ci) releases ci->lock, concurrent reclaim can free the swapcache folios and empty the cluster, allowing xswap to finish synchronize_rcu() and unmap the cluster_info page before __try_to_reclaim_swap() indexes si->cluster_info.
[ ... ]
Quoted 12 lines
mm/swapfile.c +7
-0@@ -4289,12 +4333,33 @@ static void xswap_unmap_clusters(struct swap_info_struct *si,mutex_lock (&si->xswap_lock); +/* +* Publish the smaller range before its memory goes away. Readers +* that use nr_clusters_mapped without xswap_lock pair with this +* store using smp_load_acquire(). +*/ +smp_store_release (&si->nr_clusters_mapped, start_idx); +if (vm_start >= vm_end) {
mm/swapfile.c static void xswap_unmap_clusters(struct swap_info_struct *si, · line 4352
Quoted 10 lines
mm/swapfile.c +6
-1-WRITE_ONCE (si->nr_clusters_mapped, start_idx); +/* +* The clusters sit inside one page that the preceding +* cluster shares, so there is nothing to unmap. The page +* goes away with that cluster; publishing the smaller +* cluster count is still correct. +*/ mutex_unlock (&si->xswap_lock);return ; }
[Severity: High]
Can returning early when vm_start >= vm_end allow a subsequent xswap_grow() to reinitialize ci->lock while an earlier RCU reader is still using it?
CPU0 CPU1swap_alloc_fast()
rcu_read_lock()
mapped = smp_load_acquire(
&si->nr_clusters_mapped)
xswap_unmap_clusters(si, start_idx, nr)
smp_store_release(
&si->nr_clusters_mapped, start_idx)
/* vm_start >= vm_end: skips
* flush_percpu_swap_cluster() and
* synchronize_rcu() */
xswap_grow()
xswap_map_clusters(si, start_idx, ...)
/* page already mapped */
spin_lock_init(&ci->lock) ci = swap_cluster_lock(si, offset) <-- races with spin_lock_init()
Because struct swap_cluster_info (40 or 48 bytes) does not evenly divide PAGE_SIZE, shrinking a range that shares its page with start_idx - 1 takes the vm_start >= vm_end return without flushing percpu_swap_cluster or waiting for RCU readers. If xswap_grow() maps those clusters again while a reader from before the shrink is still in flight (or using an unflushed per-CPU cache entry), the mapped: path in xswap_map_clusters() runs spin_lock_init(&ci->lock) concurrently with that reader.
mm/swapfile.c static void xswap_unmap_clusters(struct swap_info_struct *si, · line 4361
Quoted 9 lines
mm/swapfile.c +8
-0+/* +* A per-cpu cluster cache can still hold an offset in this range. +* Invalidate those references, then wait out the readers that have +* already loaded one, so that nobody can dereference cluster_info +* past this point. swapoff() needs the same before it releases. +*/ +flush_percpu_swap_cluster (si); +synchronize_rcu ();
[Severity: High]
Can swapin and extend-table allocation paths still dereference an unmapped cluster_info page across sleeping allocations where RCU is not held?
CPU0 CPU1do_swap_page()
pte_unmap_same() /* drops PTL */
get_swap_device(entry)
...
__swap_cache_alloc()
spin_lock(&ci->lock)
__swap_cache_add_check()
spin_unlock(&ci->lock)
folio_alloc_mpol() /* sleeps */
/* entry freed (e.g. MADV_DONTNEED) */
__free_cluster()
xswap_unmap_clusters()
synchronize_rcu()
vm_area_unmap_pages()
spin_lock(&ci->lock) <-- page fault on unmapped ciget_swap_device() only pins si->users, which does not block runtime xswap shrinking. Once the PTL is dropped in do_swap_page() (or the xarray lock in shmem_swapin_folio()), a racing thread can free the swap entry and allow xswap_unmap_clusters() to unmap the cluster_info page while CPU0 is preempted before swap_cache_get_folio() or sleeping in __swap_cache_alloc(). The same race affects copy_pte_range() -> swap_retry_table_alloc() -> swap_extend_table_alloc(), which drops the PTL, performs a GFP_KERNEL kzalloc(), and then takes spin_lock(&ci->lock).
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=15
Baoquan He
reviewerPATCH v5 16/19 mm, swap: prepare xswap_unmap_clusters for range shrinking
In reply to Baoquan HeFrom: Baoquan He <[email protected]>
Split xswap_unmap_clusters() into a locked helper and a wrapper that takes xswap_lock. The shrink path already holds xswap_lock and must call the helper directly.
This is a pure refactor. The existing path already falls back to bounded batches when the range is too large for vm_area_unmap_pages(); the comments are expanded to explain why, and the cross-reference in swap_alloc_fast() follows the new name.
--- mm/swapfile.c | 26 +++++++++++++++++--------- 1 file changed, 17 insertions(+), 9 deletions(-)
mm/swapfile.c +17 -9
@@ -1692,7 +1692,7 @@ static bool swap_alloc_fast(struct folio *folio)*/ unsigned long mapped; -/* Pairs with the release store in xswap_unmap_clusters(). */ +/* Pairs with the release store in xswap_unmap_clusters_locked(). */ mapped =smp_load_acquire (&si->nr_clusters_mapped);if (offset /SWAPFILE_CLUSTER >= mapped) {
@@ -4318,8 +4318,8 @@ static unsigned long xswap_grow(struct swap_info_struct *si) }/* Caller must hold si->xswap_lock. Cannot fail. */ -static void xswap_unmap_clusters (struct swap_info_struct *si, -unsigned long start_idx,unsigned long nr) +static void xswap_unmap_clusters_locked (struct swap_info_struct *si, +unsigned long start_idx,unsigned long nr) {unsigned long start_addr = (unsigned long )si->cluster_info + (size_t )start_idx *sizeof (struct swap_cluster_info );
@@ -4331,8 +4331,6 @@ static void xswap_unmap_clusters(struct swap_info_struct *si,unsigned int noreclaim_flags;unsigned long npages, idx; -mutex_lock (&si->xswap_lock); -/* * Publish the smaller range before its memory goes away. Readers * that use nr_clusters_mapped without xswap_lock pair with this
@@ -4347,7 +4345,6 @@ static void xswap_unmap_clusters(struct swap_info_struct *si,* goes away with that cluster; publishing the smaller * cluster count is still correct. */ -mutex_unlock (&si->xswap_lock);return ; }
@@ -4362,6 +4359,11 @@ static void xswap_unmap_clusters(struct swap_info_struct *si, npages = (vm_end - vm_start) >>PAGE_SHIFT ; +/* +* vm_area_unmap_pages() refuses a range above totalram_pages() and +* returns without clearing the PTEs. Freeing those pages would +* leave dangling mappings, so use the batch path for that case. +*/ if (npages <=totalram_pages ()) {/* Reclaim here would re-enter xswap_map_clusters() and its lock. */ noreclaim_flags =memalloc_noreclaim_save ();
@@ -4371,11 +4373,11 @@ static void xswap_unmap_clusters(struct swap_info_struct *si,if (pages) {xswap_unmap_range (si, vm_start, vm_end, pages, npages);kvfree (pages); - goto out_unlock; +return ; } } -/* No memory for the array, or too large: unmap in bounded batches. */ +/* No memory for the array, or too large a range: unmap in batches. */ for (idx = start_idx; idx < start_idx + nr; idx +=XSWAP_UNMAP_BATCH_CLUSTERS ) {unsigned long batch =min (start_idx + nr - idx,XSWAP_UNMAP_BATCH_CLUSTERS );
@@ -4390,7 +4392,13 @@ static void xswap_unmap_clusters(struct swap_info_struct *si,xswap_unmap_range (si, bstart, bend, batch_pages,ARRAY_SIZE (batch_pages)); } -out_unlock: +} + +static void xswap_unmap_clusters (struct swap_info_struct *si, +unsigned long start_idx,unsigned long nr) +{ +mutex_lock (&si->xswap_lock); +xswap_unmap_clusters_locked (si, start_idx, nr);mutex_unlock (&si->xswap_lock); }
Signature · 1 line
2.54.0
Baoquan He
reviewerPATCH v5 17/19 mm, swap: add xswap shrink trigger and defer it to a workqueue
In reply to Baoquan HeFrom: Baoquan He <[email protected]>
The shrink cannot run directly from __free_cluster(), because that path holds ci->lock. xswap_try_shrink() takes xswap_lock and can sleep in synchronize_rcu() and vm_area_unmap_pages(). __free_cluster() schedules xswap_shrink_work instead, and teardown disables the work.
The shrink unmaps the free tail of the cluster_info mapping. It costs two RCU grace periods: one after draining the per-CPU cache and before the tail is rechecked, and one inside the unmap helper. The second one is needed because a cluster can be allocated and freed between the recheck and the unmap, and go back into the per-CPU cache. xswap devices are SWP_SOLIDSTATE, so the per-CPU cache is the only one that needs draining.
The unmap helper takes a wait flag. The shrink passes true. Teardown passes false because __swapoff() has already drained and waited. The create failure path also passes false because it has no readers.
Count shrink operations in xswap_shrinks. Initialize the work in xswap_create() so free_swap_cluster_info() can always call disable_work_sync(). Using disable_work_sync() instead of only cancel_work_sync() also rejects a schedule that read SWP_WRITEOK just before swapoff. Then no work stays pending when the swap_info is reused and INIT_WORK() runs again.
--- include/linux/swap.h | 1 + mm/swapfile.c | 233 ++++++++++++++++++++++++++++++++++++++----- 2 files changed, 211 insertions(+), 23 deletions(-)
include/linux/swap.h +1 -0
@@ -247,6 +247,7 @@ struct swap_info_struct {u64 xswap_id;/* stable id, shown to userspace */ unsigned long nr_clusters_max;/* total clusters in the xswap address space */ unsigned long nr_clusters_mapped;/* currently mapped cluster count */ +struct work_struct xswap_shrink_work;/* deferred shrink trigger */ struct mutex xswap_lock;/* serialize map/unmap operations */ struct dentry *xswap_debugfs;/* mapped range and counters */ atomic_long_t xswap_grows;/* grow operations so far */
mm/swapfile.c +210 -23
313 lines not on this page · raw message
Signature · 1 line
2.54.0
[email protected]
AI reviewRe: [PATCH v5 17/19] mm, swap: add xswap shrink trigger and defer it to a workqueue
Thank you for your contribution! Sashiko AI review found 3 potential issues to consider: - [High] mm: use-after-unmap of cluster_info in swap paths when xswap_try_shrink() runs - [High] mm: race in…
Thank you for your contribution! Sashiko AI review found 3 potential issues to consider:
- [High] mm: use-after-unmap of cluster_info in swap paths when xswap_try_shrink() runs
- [High] mm: race in xswap_try_shrink() setting CLUSTER_FLAG_NONE re-adds unmapped cluster
- [Low] mm: redundant xswap_lock acquire in __swapoff() for non-existent sysfs shrink
--
commit ff41ecd0448adc3f4a51da2b8b59751cd899add3
Author: Baoquan He <[email protected]>
mm, swap: add xswap shrink trigger and defer it to a workqueue
This commit adds deferred shrinking of the xswap cluster_info VM_SPARSE mapping via a workqueue scheduled from __free_cluster().
Quoted 4 lines
mm/swapfile.c +0
-0
[ ... ]
mm/swapfile.c static int __swapoff(struct swap_info_struct *p) · line 3711
Quoted 12 lines
mm/swapfile.c +8
-0@@ -3681,6 +3701,15 @@ static int __swapoff(struct swap_info_struct *p)struct file *swap_file =NULL ;int err; +#ifdef CONFIG_XSWAP +if (p->flags &SWP_XSWAP ) { +cancel_work_sync (&p->xswap_shrink_work); +/* Wait out a shrink racing us from the sysfs write path. */ +mutex_lock (&p->xswap_lock); +mutex_unlock (&p->xswap_lock); + } +#endif
[Severity: Low]
This isn't a bug, but is the p->xswap_lock acquisition here needed? There is no sysfs write path that triggers a shrink, and cancel_work_sync() has already waited for xswap_shrink_work after SWP_WRITEOK was cleared.
[ ... ]
Quoted 1 line
mm/swapfile.c +0
-0@@ -4413,6 +4452,153 @@ static int xswap_mapped_end(pte_t *pte, unsigned long addr, void *data)
[ ... ]
Quoted 12 lines
mm/swapfile.c +12
-0+spin_lock (&si->lock); +for (i = start_idx; i < nr_mapped; i++) { + ci = &si->cluster_info[i]; +if (READ_ONCE (ci->flags) !=CLUSTER_FLAG_FREE ) +break ; +if (!spin_trylock (&ci->lock)) { +spin_unlock (&si->lock); + goto out_unlock; + } +spin_unlock (&ci->lock); + } +if (i != nr_mapped) {
mm/swapfile.c static int xswap_mapped_end(pte_t *pte, unsigned long addr, void *data) · line 4588
Quoted 10 lines
mm/swapfile.c +10
-0+spin_unlock (&si->lock); + goto out_unlock; + } + +for (i = start_idx; i < nr_mapped; i++) { + ci = &si->cluster_info[i]; +list_del_init (&ci->list); +WRITE_ONCE (ci->flags,CLUSTER_FLAG_NONE ); + } +spin_unlock (&si->lock);
[Severity: High]
Can setting ci->flags to CLUSTER_FLAG_NONE cause a concurrent reclaimer to re-add an unmapped cluster back onto si->free_clusters?
Both swap_reclaim_full_clusters() and cluster_reclaim_range() (called from alloc_swap_scan_cluster()) isolate a cluster with ci->flags == CLUSTER_FLAG_NONE under rcu_read_lock(), temporarily drop ci->lock around __try_to_reclaim_swap(), and then re-acquire ci->lock and call relocate_cluster():
CPU0 (kworker) CPU1xswap_try_shrink()
flush_percpu_swap_cluster(si)
synchronize_rcu()
swap_reclaim_full_clusters()
rcu_read_lock()
isolate_lock_cluster()
/* ci->flags = CLUSTER_FLAG_NONE */
spin_unlock(&ci->lock)
__try_to_reclaim_swap()
/* frees last slot in ci: */
__free_cluster(si, ci)
/* ci->table = NULL */
/* ci->flags = CLUSTER_FLAG_FREE */
spin_lock(&si->lock)
/* sees CLUSTER_FLAG_FREE,
* trylocks ci->lock */
list_del_init(&ci->list)
ci->flags = CLUSTER_FLAG_NONE
spin_unlock(&si->lock)
xswap_unmap_clusters_locked()
synchronize_rcu()
spin_lock(&ci->lock)
if (ci->flags == CLUSTER_FLAG_NONE)
relocate_cluster(si, ci)
if (!ci->count &&
ci->flags != CLUSTER_FLAG_FREE)
free_cluster(si, ci)
__free_cluster(si, ci)
spin_unlock(&ci->lock)
rcu_read_unlock()
/* unmaps and frees ci pages
* while ci is on free_clusters */ <-- use-after-freeBecause xswap_try_shrink() overwrote CLUSTER_FLAG_FREE with CLUSTER_FLAG_NONE before waiting out the RCU grace period in xswap_unmap_clusters_locked(), relocate_cluster() sees !ci->count and ci->flags != CLUSTER_FLAG_FREE and calls __free_cluster() a second time. With CONFIG_DEBUG_VM=y, swap_cluster_assert_empty() immediately dereferences the NULL ci->table; otherwise move_cluster() links ci back onto si->free_clusters just before its backing pages are unmapped and freed.
mm/swapfile.c static int xswap_mapped_end(pte_t *pte, unsigned long addr, void *data) · line 4595
Quoted 7 lines
mm/swapfile.c +7
-0+ +/* +* A cluster can be taken and freed between the recheck above and the +* unmap, putting it back in the per-cpu cache. Drain and wait again. +*/ +xswap_unmap_clusters_locked (si, start_idx, nr_unmap, true); +atomic_long_inc (&si->xswap_shrinks);
[Severity: High]
Can unmapping cluster_info pages at runtime while si->users is live cause a kernel page fault in swap paths that access si->cluster_info outside rcu_read_lock()?
Several swap paths rely on get_swap_device() pinning si->users to keep si->cluster_info mapped across preemptible or sleepable sections where neither ci->lock nor rcu_read_lock() is held. For example, in do_swap_page() -> folio_alloc_swap() -> __swap_cache_alloc(), or in swap_put_entries_cluster():
mm/swapfile.c:swap_put_entries_cluster() {
...
swap_cluster_unlock(ci);
if (!need_reclaim || !reclaim_cache)
return;
do {
nr_reclaimed = __try_to_reclaim_swap(si, offset,
TTRS_UNMAPPED | TTRS_FULL);
...
}Once swap_put_entries_cluster() drops ci->lock in task context outside rcu_read_lock() after decrementing a cached slot's swap count to 0, another CPU can reclaim the swapcache folio and free the tail cluster:
CPU0 CPU1 (kworker)swap_put_entries_cluster()
/* decrements swap count to 0 */
swap_cluster_unlock(ci)
/* folio_free_swap() frees last slot
* in tail cluster -> __free_cluster() */
xswap_try_shrink()
flush_percpu_swap_cluster(si)
synchronize_rcu()
/* unlinks FREE tail clusters */
xswap_unmap_clusters_locked()
synchronize_rcu()
xswap_unmap_range() __try_to_reclaim_swap()
swap_cache_get_folio()
__swap_entry_to_cluster()
swap_table_get(ci, ...) <-- page fault on unmapped cluster_info
The same window exists when __swap_cache_alloc() drops ci->lock to sleep in folio allocation and then calls spin_lock(&ci->lock), or when swap_extend_table_alloc() sleeps in kzalloc() before locking ci->lock, if the swap entry and its tail cluster are freed concurrently.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=17
Baoquan He
reviewerPATCH v5 18/19 mm, swap: do not give an xswap slot to a cgroup that cannot zswap
In reply to Baoquan HeFrom: Baoquan He <[email protected]>
At this point, an xswap slot holds only a zswap-backed page. It has no storage of its own. If the folio's cgroup cannot take another zswap page, giving it an xswap slot is wasted: zswap refuses the page and the slot cannot hold it.
Skip xswap devices when the cgroup cannot zswap, and try a physical swap device instead.
The check is obj_cgroup_may_zswap(). It is false when memory.zswap.max is 0 anywhere in the hierarchy, or when the hierarchy has already reached that limit. So the routing follows how much the cgroup has in zswap, not only how it is configured. zswap_store() uses the same check, but it first tries to shrink the memcg. The allocation-time check does not try to shrink, so it is stricter.
Count active xswap devices, and do the cgroup lookup only when one exists. Update the counter when the device is enabled, before a full device can skip the avail-list insert, so a full device is still counted.
A device can appear between that read and the allocation. So if the slot the allocator picked is on an xswap device, check the cgroup again; when it cannot zswap, drop the slot and retry without xswap devices.
--- mm/swapfile.c | 69 ++++++++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 63 insertions(+), 6 deletions(-)
mm/swapfile.c +63 -6
@@ -326,6 +326,8 @@ static DEFINE_SPINLOCK(swap_lock);static unsigned int nr_swapfiles;atomic_long_t nr_swap_pages;atomic_t nr_real_swapfiles; +/* Active xswap devices, as in nr_real_swapfiles. */ +static atomic_t nr_xswap_files;/* * Some modules use swappable objects and may try to swap them out under * memory pressure (via the shrinker). Before doing so, they may wish to
@@ -1519,7 +1521,9 @@ static void del_from_avail_list(struct swap_info_struct *si, bool swapoff)lockdep_assert_held (&si->lock); si->flags &= ~SWP_WRITEOK ;/* Count active devices, not merely those on the avail list. */ -if (!(si->flags &SWP_XSWAP )) +if (si->flags &SWP_XSWAP ) +atomic_sub (1 , &nr_xswap_files); +else atomic_sub (1 , &nr_real_swapfiles);atomic_long_or (SWAP_USAGE_OFFLIST_BIT , &si->inuse_pages); }else {
@@ -1558,7 +1562,9 @@ static void add_to_avail_list(struct swap_info_struct *si, bool swapon)* Count the device here, not after adding it to the list: a * full device is still active and skips the add below. */ -if (!(si->flags &SWP_XSWAP )) +if (si->flags &SWP_XSWAP ) +atomic_add (1 , &nr_xswap_files); +else atomic_add (1 , &nr_real_swapfiles); }else {if (!(READ_ONCE (si->flags) &SWP_WRITEOK ))
@@ -1682,7 +1688,7 @@ static bool get_swap_device_info(struct swap_info_struct *si)* Fast path try to get swap entries with specified order from current * CPU's swap entry pool (a cluster). */ -static bool swap_alloc_fast (struct folio *folio) +static bool swap_alloc_fast (struct folio *folio,bool may_zswap) {unsigned int order =folio_order (folio);struct swap_cluster_info *ci;
@@ -1722,6 +1728,11 @@ static bool swap_alloc_fast(struct folio *folio) } }#endif +if (!may_zswap && (si->flags &SWP_XSWAP )) { +put_swap_device (si); +rcu_read_unlock (); +return false; + } ci =swap_cluster_lock (si, offset);if (cluster_is_usable (ci, order)) {
@@ -1738,13 +1749,19 @@ static bool swap_alloc_fast(struct folio *folio) }/* Rotate the device and switch to a new cluster */ -static void swap_alloc_slow (struct folio *folio) +static void swap_alloc_slow (struct folio *folio,bool may_zswap) {struct swap_info_struct *si, *next;spin_lock (&swap_avail_lock); start_over:plist_for_each_entry_safe (si, next, &swap_avail_head, avail_list) { +/* +* Do not rotate a skipped device: the walk follows the +* rotation and would spin forever. +*/ +if (!may_zswap && (si->flags &SWP_XSWAP )) +continue ;/* Rotate the device and switch to a new cluster */ plist_requeue (&si->avail_list, &swap_avail_head);spin_unlock (&swap_avail_lock);
@@ -2067,6 +2084,17 @@ static int swap_dup_entries_cluster(struct swap_info_struct *si,return err; } +/* Can the folio's cgroup take another zswap page? */ +static bool folio_may_zswap (struct folio *folio) +{ +struct obj_cgroup *objcg =get_obj_cgroup_from_folio (folio); +bool may_zswap = !objcg ||obj_cgroup_may_zswap (objcg); + +if (objcg) +obj_cgroup_put (objcg); +return may_zswap; +} +/** * folio_alloc_swap - allocate swap space for a folio * @folio: folio we want to move to swap
@@ -2083,10 +2111,23 @@ int folio_alloc_swap(struct folio *folio) {unsigned int order =folio_order (folio);unsigned int size =1 << order; +/* True unless the gate below finds an xswap device to route away from. */ +bool may_zswap = true; +bool zswap_checked = false;VM_BUG_ON_FOLIO (!folio_test_locked (folio), folio);VM_BUG_ON_FOLIO (!folio_test_uptodate (folio), folio); +/* +* A folio that zswap will not take must not get an xswap slot. With +* no xswap device to route away from, skip the lookup; a device that +* appears before we pick one is handled after the allocation. +*/ +if (atomic_read (&nr_xswap_files)) { + may_zswap =folio_may_zswap (folio); + zswap_checked = true; + } +if (order) {/* * Reject large allocation when THP_SWAP is disabled. Check below
@@ -2107,8 +2148,8 @@ int folio_alloc_swap(struct folio *folio) again:local_lock (&percpu_swap_cluster.lock); -if (!swap_alloc_fast (folio)) -swap_alloc_slow (folio); +if (!swap_alloc_fast (folio, may_zswap)) +swap_alloc_slow (folio, may_zswap);local_unlock (&percpu_swap_cluster.lock);if (!order &&unlikely (!folio_test_swapcache (folio))) {
@@ -2116,6 +2157,22 @@ int folio_alloc_swap(struct folio *folio) goto again; } +/* +* The xswap gate was skipped because there was no device when we +* looked. One may have appeared since; check before keeping the +* xswap slot the allocator may have picked. +*/ +if (unlikely (!zswap_checked &&atomic_read (&nr_xswap_files) && +folio_test_swapcache (folio) && + (__swap_entry_to_info (folio->swap)->flags &SWP_XSWAP ))) { + zswap_checked = true; + may_zswap =folio_may_zswap (folio); +if (!may_zswap) { +swap_cache_del_folio (folio); + goto again; + } + } +/* Need to call this even if allocation failed, for MEMCG_SWAP_FAIL. */ if (unlikely (mem_cgroup_try_charge_swap (folio))) {swap_cache_del_folio (folio);
Signature · 1 line
2.54.0
[email protected]
AI reviewRe: [PATCH v5 18/19] mm, swap: do not give an xswap slot to a cgroup that cannot zswap
Thank you for your contribution! Sashiko AI review found 2 potential issues to consider: - [Medium] mm, swap: folio_alloc_swap() returns -E2BIG when !may_zswap has no real swap - [Low] mm, swap…
Thank you for your contribution! Sashiko AI review found 2 potential issues to consider:
- [Medium] mm, swap: folio_alloc_swap() returns -E2BIG when !may_zswap has no real swap
- [Low] mm, swap: racy nr_xswap_files check in folio_alloc_swap() post-alloc retry
--
commit bd28f08f1cf4ece0de7cacd5579217aa023bae3f
Author: Baoquan He <[email protected]>
mm, swap: do not give an xswap slot to a cgroup that cannot zswap
This commit skips SWP_XSWAP swap devices during swap slot allocation when the folio's cgroup cannot store pages in zswap according to obj_cgroup_may_zswap().
Quoted 4 lines
mm/swapfile.c +0
-0
[ ... ]
Quoted 25 lines
mm/swapfile.c +13
-0@@ -2083,10 +2111,23 @@ int folio_alloc_swap(struct folio *folio) {unsigned int order =folio_order (folio);unsigned int size =1 << order; +/* True unless the gate below finds an xswap device to route away from. */ +bool may_zswap = true; +bool zswap_checked = false;VM_BUG_ON_FOLIO (!folio_test_locked (folio), folio);VM_BUG_ON_FOLIO (!folio_test_uptodate (folio), folio); +/* +* A folio that zswap will not take must not get an xswap slot. With +* no xswap device to route away from, skip the lookup; a device that +* appears before we pick one is handled after the allocation. +*/ +if (atomic_read (&nr_xswap_files)) { + may_zswap =folio_may_zswap (folio); + zswap_checked = true; + } +if (order) {/* * Reject large allocation when THP_SWAP is disabled. Check below
mm/swapfile.c int folio_alloc_swap(struct folio *folio) · line 2153
Quoted 8 lines
mm/swapfile.c +2
-2@@ -2107,8 +2148,8 @@ int folio_alloc_swap(struct folio *folio) again:local_lock (&percpu_swap_cluster.lock); -if (!swap_alloc_fast (folio)) -swap_alloc_slow (folio); +if (!swap_alloc_fast (folio, may_zswap)) +swap_alloc_slow (folio, may_zswap);local_unlock (&percpu_swap_cluster.lock);
[Severity: Medium]
Does folio_alloc_swap() return -E2BIG instead of -ENOMEM for a large folio when may_zswap is false and only xswap devices have free space?
When !may_zswap causes swap_alloc_fast() and swap_alloc_slow() to skip all SWP_XSWAP devices and no real swap device can satisfy the allocation, execution jumps to failed:
mm/swapfile.c:folio_alloc_swap() {
...
failed:
if (get_nr_swap_pages() <= 0)
return -ENOSPC;
if (mem_cgroup_get_folio_swap_margin(folio) <= 0)
return -ENOMEM;
return order ? -E2BIG : -ENOMEM;
}Because nr_swap_pages includes xswap capacity and mem_cgroup_get_folio_swap_margin() only checks memory.swap.max, both checks pass and folio_alloc_swap() returns -E2BIG. shrink_folio_list() and shmem_writeout() then split the large folio and retry folio_alloc_swap() on each order-0 page, which still has may_zswap == false and fails with -ENOMEM after repeating the cgroup stats flush for every subpage.
Quoted 10 lines
mm/swapfile.c +5
-0if (!order &&unlikely (!folio_test_swapcache (folio))) {@@ -2116,6 +2157,22 @@ int folio_alloc_swap(struct folio *folio) goto again; } +/* +* The xswap gate was skipped because there was no device when we +* looked. One may have appeared since; check before keeping the +* xswap slot the allocator may have picked. +*/
mm/swapfile.c int folio_alloc_swap(struct folio *folio) · line 2174
Quoted 10 lines
mm/swapfile.c +10
-0+if (unlikely (!zswap_checked &&atomic_read (&nr_xswap_files) && +folio_test_swapcache (folio) && + (__swap_entry_to_info (folio->swap)->flags &SWP_XSWAP ))) { + zswap_checked = true; + may_zswap =folio_may_zswap (folio); +if (!may_zswap) { +swap_cache_del_folio (folio); + goto again; + } + }
[Severity: Low]
Can the atomic_read(&nr_xswap_files) check here miss an xswap slot that was just allocated?
CPU0 CPU1folio_alloc_swap()
atomic_read(&nr_xswap_files) == 0
xswap_create()
...
add_to_avail_list()
atomic_add(1, &nr_xswap_files)
swap_alloc_slow()
/* allocates SWP_XSWAP slot */
__swapoff()
swap_info_remove()
del_from_avail_list()
atomic_sub(1, &nr_xswap_files)
if (!zswap_checked &&
atomic_read(&nr_xswap_files) ...)
/* reads 0, skips check */ <-- keeps xswap slot when !may_zswapdel_from_avail_list() decrements nr_xswap_files before __swapoff() runs wait_for_allocation() and try_to_unuse() (which can also fail and call reinsert_swap_info()). When that happens, folio_alloc_swap() returns 0 with an xswap slot for a cgroup that cannot zswap, and swap_writeout() later fails when zswap_store() rejects the folio.
Since folio_test_swapcache(folio) and (__swap_entry_to_info(folio->swap)->flags & SWP_XSWAP) already tell whether the allocated slot is on an xswap device, atomic_read(&nr_xswap_files) can be dropped from this condition.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=18
Baoquan He
reviewerPATCH v5 19/19 mm, swap: skip the overcommit probe when removing xswap
In reply to Baoquan HeFrom: Baoquan He <[email protected]>
swap_info_remove() probes whether Committed_AS still fits under the commit limit left after the device's pages are subtracted. An xswap device's pages are its address space, not backing storage, but they raised the commit limit at creation. A workload can commit into that headroom, after which the probe refuses to destroy the device.
Skip the probe for xswap until the overcommit accounting for it is settled. Real swap devices keep the existing protection.
--- mm/swapfile.c | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-)
mm/swapfile.c +11 -4
@@ -3736,10 +3736,17 @@ SYSCALL_DEFINE1(swapoff, const char __user *, specialfile)*/ static int swap_info_remove (struct swap_info_struct *p) { -if (!security_vm_enough_memory_mm (current->mm, p->pages)) -vm_unacct_memory (p->pages); -else -return -ENOMEM ; +/* +* The device's pages are its address space, not backing storage. +* The commit limit it added may have been committed against, so +* skip the probe until the limit stops counting that space. +*/ +if (!(p->flags &SWP_XSWAP )) { +if (!security_vm_enough_memory_mm (current->mm, p->pages)) +vm_unacct_memory (p->pages); +else +return -ENOMEM ; + }spin_lock (&p->lock);del_from_avail_list (p, true);
Signature · 1 line
2.54.0
Baoquan He
reviewerOn 11/09/26 at 01:54pm, Baoquan He wrote:
Forgot adding change history versions before v4, please refer to below v4 patchset cover letter if you want to check.
[PATCH v4 00/14] mm, swap: extendable swap devices (xswap phase I)
https://lore.kernel.org/all/[email protected]/T/#u
Quoted 85 lines
Changelog
=========
v4 -> v5:
- Rebased onto the latest mm-new (2c5dafa7f289).- A device now has a stable id instead of exposing the swap type:
/proc/swapsshows xswap<id>, the debugfs file is named by the id and prints the type as well, and destroy takes the id. The id is assigned per boot and never reused, so a value read earlier cannot name a different device.- swapoff accepts the name: swapoff xswap<id> and swapoff -a tear the device down, while a real file or directory keeps precedence. A name that does not exist reports
-ENOENT; the sysfs destroy keeps-ENODEVfor an id it does not know. The interfaces are now documented inDocumentation/ABI/testing/sysfs-kernel-mm-xswapandzswap.rst.-
CONFIG_XSWAP_DEFAULT_MAXsets the default size of a new device. It takes a percentage of RAM, a size, or "max", and defaults to "200%"; xswap.max= still overrides it.- The
cluster_infoaccesses are RCU-safe: the allocation fast path bounds a cached cluster against the mapped range, and the shrink drains the per-cpu cache and waits an RCU grace period before unmapping.-
xswap_grow()now links the clusters it mapped even when swapoff has already clearedSWP_WRITEOK, so a failed swapoff cannot leave them mapped and on no list.-
swap_info_remove()skips the commit limit probe for xswap; real swap devices keep it. See the overcommit note above.- The cgroup zswap gate is checked again after the allocation when an xswap device appeared after the initial lookup, so a slot is not kept for a cgroup that cannot zswap.
- The patches are reorganized: the
VM_SPARSEwork is split into helpers and the xswap use, the max/pages widening moves before the sysfs interface, the shrink trigger now carries the workqueue deferral, and the unmap helper is prepared for range shrinking in its own patch. The series is now phase I of four, with the swap-table backend records as a phase of their own.Baoquan He (18):
mm, swap: refactorfree_swap_cluster_infoto takeswap_info_structmm, swap: addVM_SPARSEcluster-array helpers for xswap mm, swap: use aVM_SPARSEcluster array for xswap devices mm, swap: widenswap_info_structmax/pages to unsigned long mm, swap: add a sysfs interface to create xswap devices mm, swap: refactor swapoff and addxswap_destroy
mm, swap: require zswap for xswap devices
mm, swap: use a stable id for xswap devices
mm, swap: let the command line set the xswap device size mm, swap: addCONFIG_XSWAP_DEFAULT_MAX
mm, swap: let swapoff destroy xswap devices by name
mm, swap: add xswap grow trigger on cluster allocation mm, swap: add debugfs counters for xswap
mm, swap: make xswapcluster_infoaccesses RCU-safe
mm, swap: preparexswap_unmap_clustersfor range shrinking mm, swap: add xswap shrink trigger and defer it to a workqueue mm, swap: do not give an xswap slot to a cgroup that cannot zswap mm, swap: skip the overcommit probe when removing xswapChris Li (1):
mm: xswap support for zswap.../ABI/testing/sysfs-kernel-mm-xswap | 43 + .../admin-guide/kernel-parameters.txt | 8 + Documentation/admin-guide/mm/zswap.rst | 9 + MAINTAINERS | 1 + include/linux/swap.h | 17 +- mm/Kconfig | 20 + mm/page_io.c | 19 + mm/swap_state.c | 9 + mm/swapfile.c | 1569 +++++++++++++++-- mm/zswap.c | 11 +- 10 files changed, 1593 insertions(+), 113 deletions(-) create mode 100644 Documentation/ABI/testing/sysfs-kernel-mm-xswapbase-commit:
2c5dafa7f289d760f4492260b0988029aab478b0Signature · 1 line
2.54.0
syzbot ci has tested the following series
syzbot ci has tested the following series
[v5] mm, swap: extendable swap devices (xswap phase I)
https://lore.kernel.org/all/[email protected]
* [PATCH v5 01/19] mm: xswap support for zswap
* [PATCH v5 02/19] mm, swap: refactor free_swap_cluster_info to take swap_info_struct
* [PATCH v5 03/19] mm, swap: add VM_SPARSE cluster-array helpers for xswap
* [PATCH v5 04/19] mm, swap: use a VM_SPARSE cluster array for xswap devices
* [PATCH v5 05/19] mm, swap: widen swap_info_struct max/pages to unsigned long
* [PATCH v5 06/19] mm, swap: add a sysfs interface to create xswap devices
* [PATCH v5 07/19] mm, swap: refactor swapoff and add xswap_destroy
* [PATCH v5 08/19] mm, swap: require zswap for xswap devices
* [PATCH v5 09/19] mm, swap: use a stable id for xswap devices
* [PATCH v5 10/19] mm, swap: let the command line set the xswap device size
* [PATCH v5 11/19] mm, swap: add CONFIG_XSWAP_DEFAULT_MAX
* [PATCH v5 12/19] mm, swap: let swapoff destroy xswap devices by name
* [PATCH v5 13/19] mm, swap: add xswap grow trigger on cluster allocation
* [PATCH v5 14/19] mm, swap: add debugfs counters for xswap
* [PATCH v5 15/19] mm, swap: make xswap cluster_info accesses RCU-safe
* [PATCH v5 16/19] mm, swap: prepare xswap_unmap_clusters for range shrinking
* [PATCH v5 17/19] mm, swap: add xswap shrink trigger and defer it to a workqueue
* [PATCH v5 18/19] mm, swap: do not give an xswap slot to a cgroup that cannot zswap
* [PATCH v5 19/19] mm, swap: skip the overcommit probe when removing xswap
and found the following issue:
BUG: unable to handle kernel paging request in swapoff
Full report is available here:
https://ci.syzbot.org/series/88bad2a7-a1dc-4686-80c1-647e34e90e37
***
BUG: unable to handle kernel paging request in swapoff
tree: mm-new URL: https://kernel.googlesource.com/pub/scm/linux/kernel/git/akpm/mm.git base: 2dc0136b1fdac153434ab02778fcdb32db5d0dae arch: amd64 compiler: Debian clang version 22.1.8 (1~deb13u4), Debian LLD 22.1.8 config: https://ci.syzbot.org/builds/5c923c30-8b6d-49b4-8bbb-014cb85737f7/config syz repro: https://ci.syzbot.org/findings/f14c26d8-d891-4444-9227-d55cbaf2adb7/syz_repro
BUG: unable to handle page fault for address: fffffffffffffffe #PF: supervisor read access in kernel mode #PF: error_code(0x0000) - not-present page PGD ff4b067 P4D ff4b067 PUD ff4d067 PMD 0 Oops: Oops: 0000 [#1] SMP KASAN PTI CPU: 1 UID: 0 PID: 5894 Comm: syz.2.19 Not tainted syzkaller #0 PREEMPT(full) Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.2-debian-1.16.2-1 04/01/2014 RIP: 0010:__do_sys_swapoff mm/swapfile.c:3743 [inline] RIP: 0010:__se_sys_swapoff+0x233/0x610 mm/swapfile.c:3719 Code: 99 90 ff e9 42 ff ff ff e8 2a 99 90 ff eb 6f 4c 89 e0 48 c1 e8 03 42 80 3c 28 00 74 08 4c 89 e7 e8 72 c7 08 00 4c 89 64 24 10 <4d> 8b 24 24 66 43 c7 44 2f 04 00 00 43 c6 44 2f 06 00 4c 89 3c 24 RSP: 0018:ffffc90003d1fde0 EFLAGS: 00010246 RAX: 1fffffffffffffff RBX: 0000200000000000 RCX: c8af1e162af39c00 RDX: 0000000000000000 RSI: fffffffffffffffe RDI: 00000000ffffff9c RBP: ffffc90003d1fee0 R08: 0000000000000003 R09: 0000000000000000 R10: ffffc90003d1fd40 R11: fffff520007a3faa R12: fffffffffffffffe R13: dffffc0000000000 R14: fffffffffffffffe R15: 1ffff920007a3fc0 FS: 00007f5ca62c96c0(0000) GS:ffff8882a7806000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: fffffffffffffffe CR3: 00000001023f4000 CR4: 00000000000006f0 Call Trace: <TASK> do_syscall_x64 arch/x86/entry/syscall_64.c:61 [inline] do_syscall_64+0x166/0x520 arch/x86/entry/syscall_64.c:84 entry_SYSCALL_64_after_hwframe+0x77/0x7f RIP: 0033:0x7f5ca53a8f19 Code: ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 e8 ff ff ff f7 d8 64 89 01 48 RSP: 002b:00007f5ca62c9028 EFLAGS: 00000246 ORIG_RAX: 00000000000000a8 RAX: ffffffffffffffda RBX: 00007f5ca5625fa0 RCX: 00007f5ca53a8f19 RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000200000000000 RBP: 00007f5ca5437f6d R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 R13: 00007f5ca5626038 R14: 00007f5ca5625fa0 R15: 00007ffce4301208 </TASK> Modules linked in: CR2: fffffffffffffffe ---[ end trace 0000000000000000 ]--- RIP: 0010:__do_sys_swapoff mm/swapfile.c:3743 [inline] RIP: 0010:__se_sys_swapoff+0x233/0x610 mm/swapfile.c:3719 Code: 99 90 ff e9 42 ff ff ff e8 2a 99 90 ff eb 6f 4c 89 e0 48 c1 e8 03 42 80 3c 28 00 74 08 4c 89 e7 e8 72 c7 08 00 4c 89 64 24 10 <4d> 8b 24 24 66 43 c7 44 2f 04 00 00 43 c6 44 2f 06 00 4c 89 3c 24 RSP: 0018:ffffc90003d1fde0 EFLAGS: 00010246 RAX: 1fffffffffffffff RBX: 0000200000000000 RCX: c8af1e162af39c00 RDX: 0000000000000000 RSI: fffffffffffffffe RDI: 00000000ffffff9c RBP: ffffc90003d1fee0 R08: 0000000000000003 R09: 0000000000000000 R10: ffffc90003d1fd40 R11: fffff520007a3faa R12: fffffffffffffffe R13: dffffc0000000000 R14: fffffffffffffffe R15: 1ffff920007a3fc0 FS: 00007f5ca62c96c0(0000) GS:ffff8882a7806000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: fffffffffffffffe CR3: 00000001023f4000 CR4: 00000000000006f0 ---------------- Code disassembly (best guess), 3 bytes skipped: 0: e9 42 ff ff ff jmp 0xffffff47 5: e8 2a 99 90 ff call 0xff909934 a: eb 6f jmp 0x7b c: 4c 89 e0 mov %r12,%rax f: 48 c1 e8 03 shr $0x3,%rax 13: 42 80 3c 28 00 cmpb $0x0,(%rax,%r13,1) 18: 74 08 je 0x22 1a: 4c 89 e7 mov %r12,%rdi 1d: e8 72 c7 08 00 call 0x8c794 22: 4c 89 64 24 10 mov %r12,0x10(%rsp) * 27: 4d 8b 24 24 mov (%r12),%r12 <-- trapping instruction 2b: 66 43 c7 44 2f 04 00 movw $0x0,0x4(%r15,%r13,1) 32: 00 33: 43 c6 44 2f 06 00 movb $0x0,0x6(%r15,%r13,1) 39: 4c 89 3c 24 mov %r15,(%rsp)
***
If these findings have caused you to resend the series or submit a separate fix, please add the following tag to your commit message:
---
This report is generated by a bot. It may contain errors. syzbot ci engineers can be reached at [email protected].