mm, swap: dynamic cluster management for xswap devices
Subsystems (4)
Baoquan He
reviewerv2 cover mm, swap: dynamic cluster management for xswap devices
This series implements step 3 of the incremental path proposed in [1]:
a virtual swap device with dynamic cluster growth and shrink, backed only by zswap, no writeback yet.
During the discussion in [2], Chris and Nhat debated xarray-based vs array-based cluster lookup for virtual swap. Chris argued for keeping the swap table and cluster_info in the same place with O(1) array indexing, and suggested that the kvmalloc grow approach here should provide the cluster management VS needs. He further proposed using the same grow technique for the future per-slot backend pointer array (vs_table), keeping a single coherent array layout throughout.
This series therefore serves as the foundation: once the dynamic cluster management lands, Nhat's VS series can build the per-slot backend pointer, writeback, and rmap on top of it ??? without an extra xarray lookup in the cluster access path.
A note on naming: Nhat's series uses "virtual swap" (VS), Chris originally used "ghost swap", and now suggests "xswap" (extendable swap). This series uses "xswap" as a temporary convention; the final name is open for discussion.
xswap decouple swap slots from physical backing storage. Currently zswap requires a backing swap device sized for the full virtual capacity, even when writeback is never needed ??? wasting disk space on slots that zswap will never use. An xswap device is a swapfile: a 4K on-disk header advertises the capacity, but there is no swap data section. This eliminates the wasted backing storage.
This series replaces the fixed allocation with a VM_SPARSE vmalloc area that grows on demand and shrinks when clusters are freed, within the bounds set at swapon. The cluster_info pages are mapped lazily via vm_area_map_pages() and unmapped in bulk when contiguous free clusters accumulate at the tail. Lookup is O(1) via simple array indexing, which matters on the swapout hot path. The trade-off is coarser shrink granularity (full pages, not individual clusters), but for the initial landing where swap usage tends to be ratchet-like, this is a reasonable choice.
[1] https://lore.kernel.org/all/al8ohWshSSZ64AtT@MiWiFi-R3L-srv/
[2] https://lore.kernel.org/all/CACePvbVJGVDbhvPRNsZx-f4t16TU-t6H754JOUQQ4uF2Xe6Q5w@mail.gmail.com/
Design
======
The cluster_info[] array lives in a VM_SPARSE vm_struct. Physical pages are allocated and mapped into it in chunks of XSWAP_GROW_CLUSTERS (256, configurable). When the allocator runs out of free clusters and the mapped range hasn't reached the ceiling, the grow path maps more pages. Symmetrically, when clusters are freed, a shrink path unmap pages from the tail if enough contiguous free clusters accumulate.
To avoid scanning cluster_info[] on every shrink opportunity, a nr_free_tail counter provides O(1) detection ??? it tracks how many clusters at the tail are free. Shrink fires when the counter reaches the threshold.
A per-device runtime ceiling (nr_clusters) allows userspace to cap the mapped range below the hard limit, and a debugfs knob exposes it for live tuning without swapoff/swapon.
Testing
======
1. Create xswap swapfile
touch swap.4G
truncate -s 4G swap.4G
mkswap swap.4G
dd if=swap.4G of=ghost.4G bs=4096 count=1
2. swapon
swapon ghost.4G
3. memory pressure testing
echo 1 > /sys/module/zswap/parameters/enabled stress-ng --vm 1 --vm-bytes 4G --vm-keep --timeout 120s
4. shrink and grow memory via debugfs knob
root@fedora:~# cat /sys/kernel/debug/xswap/type0_cluster_limit 4096 root@fedora:~# echo 2048 > /sys/kernel/debug/xswap/type0_cluster_limit root@fedora:~# cat /sys/kernel/debug/xswap/type0_cluster_limit 2048
I did test cases and all passed:
swapon, swapoff
swapon, during stress-ng running shrink and grow xswap size swapon, stress-ng, then poweroff when stress-ng finished
=====
Chnagelog
=========v1->v2:
- Added __GFP_HIGH | __GFP_NOMEMALLOC to alloc_page() and kmalloc_array() in the grow path, plus memalloc_noreclaim_save/restore() wrapping, to prevent the grow path from consuming emergency memory reserves or recursing into swap under PF_MEMALLOC. This is folded into patch 3. This was pointed out by Nhat.
- Folded the mutex serialization fix into the cluster grow patch (patch 3). This is suggested by Nhat.
- Fixed coding style issues: corrected indentation of declarations in xswap_unmap_clusters(), removed unnecessary block scope around the err variable in xswap_map_clusters().
- Rebased onto mm-unstable
Baoquan He (9):
mm, swap: add CONFIG_XSWAP and xswap fields to swap_info_struct mm, swap: add xswap cluster grow via VM_SPARSE vmalloc mm, swap: add xswap grow trigger on cluster allocation mm, swap: add xswap_try_shrink and shrink trigger on cluster free mm, swap: free backing pages in xswap_unmap_clusters mm, swap: add nr_free_tail for O(1) xswap shrink detection mm, swap: add adjustable runtime ceiling (nr_clusters) for xswap mm, swap: add debugfs knob for xswap per-device cluster limit mm, swap: defer xswap shrink to workqueue to avoid lock recursion
Chris Li (1):
mm: xswap support for zswap
include/linux/swap.h | 12 + mm/Kconfig | 9 + mm/page_io.c | 16 + mm/swap_state.c | 7 + mm/swapfile.c | 692 +++++++++++++++++++++++++++++++++++++++++-- mm/zswap.c | 9 +- 6 files changed, 724 insertions(+), 21 deletions(-)
Signature · 1 line
2.54.0
Baoquan He
reviewerRFC PATCH v2 1/10 mm: xswap support for zswap
From: Chris Li <[email protected]>
Introduce extendable (virtual) swap device support — xswap.
The current zswap requires a backing swapfile. The swap slot used by zswap is not able to be used by the swapfile, wasting swapfile space.
An xswap device is a swapfile that only contains the swap header, with the header indicating the size of the virtual swap space. There is no swap data section, therefore no waste of swapfile space. Any write to an xswap device will fail. To prevent accidental read or write, bdev of swap_info_struct is set to NULL. Xswap devices set the SSD flag because there is no rotational disk access when using zswap.
Zswap writeback is disabled if all swapfiles in the system are xswap devices (tracked via nr_real_swapfiles).
How to create an xswap device:
touch swap.1G
truncate -s 1G swap.1G
mkswap swap.1G
dd if=swap.1G of=xswap.1G bs=4096 count=1 # xswap.1G is 4K on disk but reports 1G capacity swapon xswap.1G
--- include/linux/swap.h | 2 ++ mm/page_io.c | 16 +++++++++++++++ mm/swap_state.c | 7 +++++++ mm/swapfile.c | 49 ++++++++++++++++++++++++++++++++++++++++---- mm/zswap.c | 9 ++++++-- 5 files changed, 77 insertions(+), 6 deletions(-)
include/linux/swap.h +2 -0
@@ -207,6 +207,7 @@ enum {SWP_STABLE_WRITES = (1 <<11 ),/* no overwrite PG_writeback pages */ SWP_SYNCHRONOUS_IO = (1 <<12 ),/* synchronous IO is efficient */ SWP_HIBERNATION = (1 <<13 ),/* pinned for hibernation */ + SWP_XSWAP = (1 << 14), /* extendable swap device */ /* add others here before... */ };
@@ -350,6 +351,7 @@ void free_folio_and_swap_cache(struct folio *folio);void free_pages_and_swap_cache (struct encoded_page **,int );/* linux/mm/swapfile.c */ extern atomic_long_t nr_swap_pages; +extern atomic_t nr_real_swapfiles;extern long total_swap_pages;extern atomic_t nr_rotate_swap;
mm/page_io.c +16 -0
@@ -247,6 +247,17 @@ int swap_writeout(struct swap_io_ctx *ctx, struct folio *folio) }rcu_read_unlock (); +/* +* ctx->sis is set by swap_add_folio() which is called from +* __swap_writepage() below. Since we must avoid the writepage +* path for xswap devices, use the swap_info from the folio's +* swap entry directly instead of going through ctx. +*/ +if (unlikely (__swap_entry_to_info (folio->swap)->flags &SWP_XSWAP )) { +folio_mark_dirty (folio); +return AOP_WRITEPAGE_ACTIVATE ; + } +__swap_writepage (ctx, folio);return 0 ; out_unlock:
@@ -479,6 +490,11 @@ void swap_read_folio(struct swap_io_ctx *ctx, struct folio *folio)if (zswap_load (folio) != -ENOENT ) goto finish; +if (unlikely (sis->flags &SWP_XSWAP )) { +folio_unlock (folio); + goto finish; + } +/* We have to read from slower devices. Increase zswap protection. */ zswap_folio_swapin (folio);swap_add_folio (ctx, folio,READ );
mm/swap_state.c +7 -0
@@ -829,6 +829,13 @@ struct folio *swap_cluster_readahead(swp_entry_t entry, gfp_t gfp_mask,struct blk_plug plug;swp_entry_t ra_entry; +/* +* The entry may have been freed by another task. Avoid swap_info_get() +* which will print error message if the race happens. +*/ +if (si->flags &SWP_XSWAP ) + goto skip; + mask =swapin_nr_pages (offset) -1 ;if (!mask) goto skip;
mm/swapfile.c +45 -4
103 lines not on this page · raw message
mm/zswap.c +7 -2
@@ -998,7 +998,12 @@ static int zswap_writeback_entry(struct zswap_entry *entry,/* try to allocate swap cache folio */ si =get_swap_device (swpentry);if (!si) -return -EEXIST ; +return -ENOENT ; + +if (si->flags &SWP_XSWAP ) { +put_swap_device (si); +return -EINVAL ; + } mpol =get_task_policy (current); folio = swap_cache_alloc_folio(swpentry, GFP_KERNEL, BIT(0), NULL, mpol,
@@ -1535,7 +1540,7 @@ bool zswap_store(struct folio *folio)zswap_pool_put (pool); put_objcg:obj_cgroup_put (objcg); -if (!ret && zswap_pool_reached_full) +if (!ret && zswap_pool_reached_full &&atomic_read (&nr_real_swapfiles))queue_work (shrink_wq, &zswap_shrink_work); check_old:/*
Signature · 1 line
2.54.0
Johannes Weiner
maintainerRe: [RFC PATCH v2 01/10] mm: xswap support for zswap
Quoted 19 lines · Baoquan He
From: Chris Li <[email protected]>
Introduce extendable (virtual) swap device support ??? xswap.
The current zswap requires a backing swapfile. The swap slot used by zswap is not able to be used by the swapfile, wasting swapfile space.
An xswap device is a swapfile that only contains the swap header, with the header indicating the size of the virtual swap space. There is no swap data section, therefore no waste of swapfile space. Any write to an xswap device will fail. To prevent accidental read or write, bdev of
swap_info_structis set to NULL. Xswap devices set the SSD flag because there is no rotational disk access when using zswap.Zswap writeback is disabled if all swapfiles in the system are xswap devices (tracked via
nr_real_swapfiles).
Quoted 7 lines
How to create an xswap device:
touch swap.1G
truncate -s 1G swap.1G
mkswap swap.1G
dd if=swap.1G of=xswap.1G bs=4096 count=1 # xswap.1G is 4K on disk but reports 1G capacity swapon xswap.1G
Sigh.
Why does the user have to go through this dance?
Why does the user have to decide in advance what size the space needs to be?
You point out no inherent limit to how much can be compressed, so there is no reason to make userspace decide on an arbitrary one.
There is no reason to tie an address space that can be managed transparently inside the kernel to TWO named files on disk.
There are plenty of past discussions on this very topic. I don't see the point in resubmitting the same thing under different names, without even a reference to previous discussions.
As a side note: if you have to add "(virtual)" after every instance of "extendable", then maybe "extendable" is a terrible name and you should just call it "virtual".
Baoquan He
reviewerRe: [RFC PATCH v2 01/10] mm: xswap support for zswap
Hi Johannes,
Quoted 32 lines · Johannes Weiner
Quoted 19 lines · Baoquan He
From: Chris Li <[email protected]>
Introduce extendable (virtual) swap device support ??? xswap.
The current zswap requires a backing swapfile. The swap slot used by zswap is not able to be used by the swapfile, wasting swapfile space.
An xswap device is a swapfile that only contains the swap header, with the header indicating the size of the virtual swap space. There is no swap data section, therefore no waste of swapfile space. Any write to an xswap device will fail. To prevent accidental read or write, bdev of
swap_info_structis set to NULL. Xswap devices set the SSD flag because there is no rotational disk access when using zswap.Zswap writeback is disabled if all swapfiles in the system are xswap devices (tracked via
nr_real_swapfiles).Quoted 7 lines
How to create an xswap device:
touch swap.1G
truncate -s 1G swap.1G
mkswap swap.1G
dd if=swap.1G of=xswap.1G bs=4096 count=1 # xswap.1G is 4K on disk but reports 1G capacity swapon xswap.1GSigh.
Why does the user have to go through this dance?
Quoted 8 lines
Why does the user have to decide in advance what size the space needs to be?
You point out no inherent limit to how much can be compressed, so there is no reason to make userspace decide on an arbitrary one.
There is no reason to tie an address space that can be managed transparently inside the kernel to TWO named files on disk.
Thanks for looking into this.
The file-based creation dance is there only because this is RFC — I wanted to reuse the existing swapon path so the core grow/shrink machinery could be measured and tested without also designing a new userspace interface. I agree it's not the right final interface.
The direction I'm thinking for the next revision:
- Drop the file requirement entirely. An xswap device has no backing store, so there is no reason it needs a file.
- Use totalram_pages as the initial per-device size. Chris suggested this, and it's a natural bound: if all anonymous memory is swapped out, that is the maximum number of swap entries zswap will ever need, assuming a reasonable compression ratio. The hard upper limit could be 2 times of system RAM, or the max system RAM memory hotplug can add to.
Doing this because we need consider swap.tier support. A single global xswap device in swap.tier would mean all memcgs compress into the same device — there is only one swap entry namespace. With per-device xswap instances, swap.tier can bind different memcgs to different xswap devices, giving each its own swap slot namespace. Total isolation on slot usage, no cross-memcg interference.
-----
Hi Chris, Joungjun,
Please correct me if I misunderstood the swap.tier concept and xswap use case in there.)
-----
For creation, something like:
1.
echo $((4 * 1024 * 1024 * 1024)) > /sys/kernel/mm/xswap/create
or
2.
even simpler, with automatic sizing:
echo 1 > /sys/kernel/mm/xswap/enable
(use totalram_pages as the default size.)
3.
swapon -t xswap xswap0
(use totalram_pages as the default size.)
I'm open to other ideas. If anyone have a preference for the interface, I'd like to hear it.
Quoted 8 lines
There are plenty of past discussions on this very topic. I don't see the point in resubmitting the same thing under different names, without even a reference to previous discussions.
As a side note: if you have to add "(virtual)" after every instance of "extendable", then maybe "extendable" is a terrible name and you should just call it "virtual".
"extendable (virtual)" wasn't meant to explain one with the other. Chris prefers "extendable", you prefer "virtual" — I put both in the cover letter so the community could weigh in. I don't have a strong preference to either.
I'll address all of this in RFC v3 — drop the file requirement, auto-size to totalram_pages (with grow/shrink for dynamic adjustment on top), and settle the name. The goal of RFC v2 was to get the core mechanics reviewed; I think that part is in reasonable shape, and the interface is exactly the kind of thing I was hoping to get feedback on. Thanks for providing it.
Thanks
Baoquan
Youngjun Park
reviewerRe: [RFC PATCH v2 01/10] mm: xswap support for zswap
Quoted 63 lines · Baoquan He
Hi Johannes,
Quoted 32 lines · Johannes Weiner
Quoted 19 lines · Baoquan He
From: Chris Li <[email protected]>
Introduce extendable (virtual) swap device support ??? xswap.
The current zswap requires a backing swapfile. The swap slot used by zswap is not able to be used by the swapfile, wasting swapfile space.
An xswap device is a swapfile that only contains the swap header, with the header indicating the size of the virtual swap space. There is no swap data section, therefore no waste of swapfile space. Any write to an xswap device will fail. To prevent accidental read or write, bdev of
swap_info_structis set to NULL. Xswap devices set the SSD flag because there is no rotational disk access when using zswap.Zswap writeback is disabled if all swapfiles in the system are xswap devices (tracked via
nr_real_swapfiles).Quoted 7 lines
How to create an xswap device:
touch swap.1G
truncate -s 1G swap.1G
mkswap swap.1G
dd if=swap.1G of=xswap.1G bs=4096 count=1 # xswap.1G is 4K on disk but reports 1G capacity swapon xswap.1GSigh.
Why does the user have to go through this dance?
Quoted 8 lines
Why does the user have to decide in advance what size the space needs to be?
You point out no inherent limit to how much can be compressed, so there is no reason to make userspace decide on an arbitrary one.
There is no reason to tie an address space that can be managed transparently inside the kernel to TWO named files on disk.
Thanks for looking into this.
The file-based creation dance is there only because this is RFC — I wanted to reuse the existing swapon path so the core grow/shrink machinery could be measured and tested without also designing a new userspace interface. I agree it's not the right final interface.
The direction I'm thinking for the next revision:
- Drop the file requirement entirely. An xswap device has no backing store, so there is no reason it needs a file.
- Use
totalram_pagesas the initial per-device size. Chris suggested this, and it's a natural bound: if all anonymous memory is swapped out, that is the maximum number of swap entries zswap will ever need, assuming a reasonable compression ratio. The hard upper limit could be 2 times of system RAM, or the max system RAM memory hotplug can add to.
Quoted 6 lines
Doing this because we need consider swap.tier support. A single global xswap device in swap.tier would mean all memcgs compress into the same device — there is only one swap entry namespace. With per-device xswap instances, swap.tier can bind different memcgs to different xswap devices, giving each its own swap slot namespace. Total isolation on slot usage, no cross-memcg interference.
Hello Baoquan :)
Is there concrete user scenario isolation is needed?
Quoted 6 lines
-----
Hi Chris, Joungjun,
Please correct me if I misunderstood the swap.tier concept and xswap use case in there.)-----
Anysway, if we want to use xswap isolation like below,
xswap t1 xswap t2 tier1 tier2
| x1 | | x2 | | dev1 | | dev2 |then each memcg may have its own xswap front-end and backing tie
memcg1: xswap t1 + tier1
memcg2: xswap t2 + tier2
However, with the current tier design, the root cgroup needs to see the whole tier layout. In that case, I think the root view may become unclear if there are multiple xswap instances. From the root cgroup point of view, it may be better to see xswap as one logical tier, not as two separate tiers.
For example, the layout could be like this:
xswap tier tier1 tier2
| xswap1 xswap2 | | dev1 | | dev2 |
Then each memcg can have its own mapping or policy:
memcg1: xswap tier + tier1 (xswap1 + dev1)
memcg2: xswap tier + tier2 (xswap2 + dev2)
With this model, the root cgroup can keep one simple global view of the xswap tier. At the same time, each memcg can still use a specific xswap area and a specific backing swap tier.
P.s I am thinking about multiple xswap usecase on tier. (this is just mind map. I don't know whether it is right or not)
Another possible layout may be to use xswap as a RAM buffer for each tier
tier1 tier2
| xswap + fast dev | | xswap + slow dev |
We can use xswap as a simple buffering layer?
In that case, we would need a clear policy
for how xswap is assigned to each tier, and how the backing swap device is selected for each tier.
So why I am saying this is that,
if xswap is managed as part of swap tiers, it would
be helpful to define a more concrete layout and policy for xswap assignment.
This would make the isolation use case much clearer
Thanks!
Youngjun
Baoquan He
reviewerRe: [RFC PATCH v2 01/10] mm: xswap support for zswap
Quoted 64 lines · Youngjun Park
Quoted 63 lines · Baoquan He
Hi Johannes,
Quoted 32 lines · Johannes Weiner
Quoted 19 lines · Baoquan He
From: Chris Li <[email protected]>
Introduce extendable (virtual) swap device support ??? xswap.
The current zswap requires a backing swapfile. The swap slot used by zswap is not able to be used by the swapfile, wasting swapfile space.
An xswap device is a swapfile that only contains the swap header, with the header indicating the size of the virtual swap space. There is no swap data section, therefore no waste of swapfile space. Any write to an xswap device will fail. To prevent accidental read or write, bdev of
swap_info_structis set to NULL. Xswap devices set the SSD flag because there is no rotational disk access when using zswap.Zswap writeback is disabled if all swapfiles in the system are xswap devices (tracked via
nr_real_swapfiles).Quoted 7 lines
How to create an xswap device:
touch swap.1G
truncate -s 1G swap.1G
mkswap swap.1G
dd if=swap.1G of=xswap.1G bs=4096 count=1 # xswap.1G is 4K on disk but reports 1G capacity swapon xswap.1GSigh.
Why does the user have to go through this dance?
Quoted 8 lines
Why does the user have to decide in advance what size the space needs to be?
You point out no inherent limit to how much can be compressed, so there is no reason to make userspace decide on an arbitrary one.
There is no reason to tie an address space that can be managed transparently inside the kernel to TWO named files on disk.
Thanks for looking into this.
The file-based creation dance is there only because this is RFC — I wanted to reuse the existing swapon path so the core grow/shrink machinery could be measured and tested without also designing a new userspace interface. I agree it's not the right final interface.
The direction I'm thinking for the next revision:
- Drop the file requirement entirely. An xswap device has no backing store, so there is no reason it needs a file.
- Use
totalram_pagesas the initial per-device size. Chris suggested this, and it's a natural bound: if all anonymous memory is swapped out, that is the maximum number of swap entries zswap will ever need, assuming a reasonable compression ratio. The hard upper limit could be 2 times of system RAM, or the max system RAM memory hotplug can add to.
Quoted 8 lines
Quoted 6 lines
Doing this because we need consider swap.tier support. A single global xswap device in swap.tier would mean all memcgs compress into the same device — there is only one swap entry namespace. With per-device xswap instances, swap.tier can bind different memcgs to different xswap devices, giving each its own swap slot namespace. Total isolation on slot usage, no cross-memcg interference.
Hello Baoquan :)
Thanks for the thoughtful input, Youngjun.
Quoted 2 lines
Is there concrete user scenario isolation is needed?
Hmm, not yet, it just comes from my personal understanding and thinking. I will exlain later.
Quoted 24 lines
Quoted 7 lines
-----
Hi Chris, Joungjun,
Please correct me if I misunderstood the swap.tier concept and xswap use case in there.)-----Anysway, if we want to use xswap isolation like below,
xswap t1 xswap t2 tier1 tier2 | x1 | | x2 | | dev1 | | dev2 |then each memcg may have its own xswap front-end and backing tie
memcg1: xswap t1 + tier1
memcg2: xswap t2 + tier2However, with the current tier design, the root cgroup needs to see the whole tier layout. In that case, I think the root view may become unclear if there are multiple xswap instances. From the root cgroup point of view, it may be better to see xswap as one logical tier, not as two separate tiers.
Quoted 4 lines
For example, the layout could be like this:
xswap tier tier1 tier2
| xswap1 xswap2 | | dev1 | | dev2 |
Exactly, this is what I am thinking about. Unless we assign each xswap different memory tier or memory type in the future. Otherwise, all xswap instances should share the same tier.
Quoted 9 lines
Then each memcg can have its own mapping or policy:
memcg1: xswap tier + tier1 (xswap1 + dev1)
memcg2: xswap tier + tier2 (xswap2 + dev2)With this model, the root cgroup can keep one simple global view of the xswap tier. At the same time, each memcg can still use a specific xswap area and a specific backing swap tier.
Exactly. This is what per xswap device is trying to do.
Quoted 7 lines
P.s I am thinking about multiple xswap usecase on tier. (this is just mind map. I don't know whether it is right or not)
Another possible layout may be to use xswap as a RAM buffer for each tier
Quoted 7 lines
tier1 tier2
| xswap + fast dev | | xswap + slow dev |We can use xswap as a simple buffering layer?
In that case, we would need a clear policy
for how xswap is assigned to each tier, and how the backing swap device is selected for each tier.
I may not get your point. Do you mean kernel will decide how to assign each xswap to tier and how to select back device via policy, but not user?
Quoted 7 lines
So why I am saying this is that,
if xswap is managed as part of swap tiers, it would
be helpful to define a more concrete layout and policy for xswap assignment.This would make the isolation use case much clearer
Here let me explain why I think per-device xswap is the right starting point.
The primary scenario I have in mind is cloud environments where different containers need different swap tier chains:
memcg A (latency-sensitive service):
xswap-A → NVMe tier (fast writeback, small capacity)
memcg B (batch processing):
xswap-B → HDD tier (slow writeback, large capacity)
memcg C (best-effort):
xswap-C → no backend (compression only, never write to disk)
With a single global xswap, all three memcgs share the same backend. If memcg B triggers heavy writeback to HDD, memcg A's swap-in path is unaffected in the IO sense, but the shared slot namespace means A and B compete for swap entries. Per-device xswap gives each its own slot pool and its own backing device — full isolation.
Beyond IO and slot isolation, per-device xswap also makes per-memcg swap statistics straightforward: slot usage, compressed page count, and writeback throughput are naturally scoped to each device.
Futhermore, the existing swap subsystem is already per-device — each swapon allocates an independent swap_info_struct with its own cluster table, slot namespace, statistics, and teardown path. Per-device xswap inherits all of this for free:
- Slot allocation via the existing per-CPU cluster allocator (no new locking)
- Per-device statistics already exposed in /proc/swaps and sysfs
- Device lifecycle via swapon/swapoff with no new management interface
- swap.tier naturally binds tiers to per-device instances
Thanks
Baoquan
Baoquan He
reviewerRFC PATCH v2 2/10 mm, swap: add CONFIG_XSWAP and xswap fields to swap_info_struct
In reply to Baoquan HeAdd CONFIG_XSWAP Kconfig option (depends on SWAP && 64BIT) for extendable (virtual) swap device support.
Add three fields to struct swap_info_struct under CONFIG_XSWAP:
- cluster_vm: the VM_SPARSE vm_struct backing the cluster_info array
- nr_clusters: total number of clusters in the xswap address space
- nr_clusters_mapped: number of clusters currently mapped (lazy grow)
These fields enable lazy vmalloc-based dynamic cluster management where the cluster_info array is backed by a sparse vmalloc area that grows on demand and shrinks when clusters are freed.
--- include/linux/swap.h | 5 +++++ mm/Kconfig | 9 +++++++++ 2 files changed, 14 insertions(+)
include/linux/swap.h +5 -0
@@ -248,6 +248,11 @@ struct swap_info_struct {signed char type;/* strange name for an index */ unsigned int max;/* size of this swap device */ struct swap_cluster_info *cluster_info;/* cluster info. Only for SSD */ +#ifdef CONFIG_XSWAP +struct vm_struct *cluster_vm;/* VM_SPARSE area for xswap dynamic cluster_info */ +unsigned long nr_clusters;/* total cluster count for xswap */ +unsigned long nr_clusters_mapped;/* currently mapped cluster count */ +#endif struct list_head free_clusters;/* free clusters list */ struct list_head full_clusters;/* full clusters list */ struct list_head nonfull_clusters[SWAP_NR_ORDERS ];
mm/Kconfig +9 -0
@@ -122,6 +122,15 @@ config ZSWAP_COMPRESSOR_DEFAULT default "zstd" if ZSWAP_COMPRESSOR_DEFAULT_ZSTD default "" +config XSWAP + bool "Extendable (virtual) swap device" + depends on SWAP && 64BIT + help + Adds support for extendable swap devices (xswap) that decouple + PTE swap entries from physical backing storage. The cluster_info + array is backed by a sparse vmalloc area that grows and shrinks + on demand, avoiding static pre-allocation overhead. + config ZSMALLOC tristate
Signature · 1 line
2.54.0
Baoquan He
reviewerRFC PATCH v2 3/10 mm, swap: add xswap cluster grow via VM_SPARSE vmalloc
In reply to Baoquan HeImplement dynamic cluster_info array growth for xswap devices using a VM_SPARSE vmalloc area:
1. xswap_map_clusters(): Allocate physical pages and map them into the pre-reserved VM_SPARSE KVA region via vm_area_map_pages().
2. xswap_unmap_clusters(): Unmap pages from the VM_SPARSE area via vm_area_unmap_pages() (used by the error/teardown paths, shrink comes later).
3. setup_swap_clusters_info() xswap path: Use get_vm_area(VM_SPARSE) for the cluster_info array, lazily mapping only the initial chunk.
4. free_swap_cluster_info(): Refactor to take swap_info_struct*. For xswap, unmap all clusters and free_vm_area().
5. swapoff: Remove snapshot locals; move p->max/p->cluster_info clearing after free_swap_cluster_info().
The grow path runs in the swap allocation context which may have PF_MEMALLOC set during reclaim, so avoid consuming emergency reserves by using __GFP_HIGH | __GFP_NOMEMALLOC on alloc_page() and kmalloc_array(), switching to vm_area_map_pages_gfp(), and wrapping the entire allocation block with memalloc_noreclaim_save() to prevent recursive reclaim from internal page table allocations.
Concurrent grow operations race on vm_area_map_pages(), triggering WARN_ON(!pte_none) in the vmap page table walk. Add a per-device mutex (xswap_lock) held across xswap_map_clusters and xswap_unmap_clusters to serialize page table modifications. The shrink path is already deferred to a workqueue so it does not contend with itself.
--- include/linux/swap.h | 1 + mm/swapfile.c | 256 +++++++++++++++++++++++++++++++++++++++++-- 2 files changed, 245 insertions(+), 12 deletions(-)
include/linux/swap.h +1 -0
@@ -252,6 +252,7 @@ struct swap_info_struct {struct vm_struct *cluster_vm;/* VM_SPARSE area for xswap dynamic cluster_info */ unsigned long nr_clusters;/* total cluster count for xswap */ unsigned long nr_clusters_mapped;/* currently mapped cluster count */ +struct mutex xswap_lock;/* serialize map/unmap operations */ #endif struct list_head free_clusters;/* free clusters list */ struct list_head full_clusters;/* full clusters list */
mm/swapfile.c +244 -12
324 lines not on this page · raw message
Signature · 1 line
2.54.0
Baoquan He
reviewerRFC PATCH v2 4/10 mm, swap: add xswap grow trigger on cluster allocation
In reply to Baoquan HeWhen cluster_alloc_swap_entry() fails to find a free cluster and the xswap device still has room to grow, expand the mapped range by XSWAP_GROW_CLUSTERS clusters.
Since xswap is always SWP_SOLIDSTATE, no locks need to be dropped before calling xswap_map_clusters() — global_cluster_lock is never held on this path.
The grow sequence:
1. Check nr_clusters_mapped < nr_clusters and free list empty
2. Call xswap_map_clusters() to allocate and map more physical pages
3. Add newly mapped clusters to si->free_clusters under si->lock
4. Retry allocation from the fresh free clusters
This makes the xswap cluster space grow transparently as swap usage increases, without any userspace intervention.
--- mm/swapfile.c | 101 +++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 100 insertions(+), 1 deletion(-)
mm/swapfile.c +100 -1
@@ -65,6 +65,7 @@ static int xswap_map_clusters(struct swap_info_struct *si, unsigned long start_idx, unsigned long nr);static void xswap_unmap_clusters (struct swap_info_struct *si,unsigned long start_idx,unsigned long nr); +static int xswap_check_mapped (pte_t *pte,unsigned long addr,void *data);#endif static void swap_range_alloc(struct swap_info_struct *si,
@@ -1204,6 +1205,48 @@ static unsigned long cluster_alloc_swap_entry(struct swap_info_struct *si,if (found) goto done; } + +#ifdef CONFIG_XSWAP +/* +* For xswap: if no free cluster was found and more clusters +* can be mapped, grow the cluster_info array and retry. +*/ +if (!found && (si->flags &SWP_XSWAP ) && +READ_ONCE (si->nr_clusters_mapped) <READ_ONCE (si->nr_clusters) && +list_empty (&si->free_clusters)) { +unsigned long nr_new =min (READ_ONCE (si->nr_clusters) - +READ_ONCE (si->nr_clusters_mapped), +XSWAP_GROW_CLUSTERS ); +unsigned long start =READ_ONCE (si->nr_clusters_mapped); +unsigned long i; + +if (!xswap_map_clusters (si, start, nr_new)) { +unsigned long added =0 ; + +spin_lock (&si->lock); +for (i = start; i < start + nr_new; i++) { +struct swap_cluster_info *ci = &si->cluster_info[i]; +spin_lock (&ci->lock); +/* +* A concurrent grower may have already added +* these clusters to the free list. Only add +* clusters that are still off-list (NONE). +*/ +if (ci->flags ==CLUSTER_FLAG_NONE ) { + ci->flags =CLUSTER_FLAG_FREE ; +list_add_tail (&ci->list, &si->free_clusters); + added++; + } +spin_unlock (&ci->lock); + } +spin_unlock (&si->lock); + +/* Retry allocation from the free list */ + found =alloc_swap_scan_list (si, &si->free_clusters, + folio, false); + } + } +#endif done:if (!(si->flags &SWP_SOLIDSTATE ))spin_unlock (&si->global_cluster_lock);
@@ -3669,7 +3712,32 @@ static int xswap_map_clusters(struct swap_info_struct *si, goto fail; } -if (vm_area_map_pages (si->cluster_vm, vm_start, vm_end, pages)) { +/* +* Check if the target pages are already mapped by a concurrent +* grower. We must do this after page allocation because +* alloc_page(GFP_KERNEL) can sleep, opening a race window. +* If someone already mapped these pages, free ours and continue. +*/ +if (apply_to_existing_page_range (&init_mm, vm_start, + vm_end - vm_start, + xswap_check_mapped,NULL )) { + i = npages; + goto fail_nounmap; + } + +int err =vm_area_map_pages (si->cluster_vm, vm_start, vm_end, pages); +if (err) { +/* +* -EBUSY means the PTEs are already present: +* another thread raced with us and mapped the +* same pages between our check above and this +* call. Treat as success — free our unused +* pages and continue. +*/ +if (err == -EBUSY ) { + i = npages; + goto fail_nounmap; + } i = npages;/* free all pages on failure */ goto fail; }
@@ -3688,6 +3756,28 @@ static int xswap_map_clusters(struct swap_info_struct *si,mutex_unlock (&si->xswap_lock);return 0 ; +fail_nounmap: +/* +* Pages already mapped by a concurrent grower. Free our unused +* pages, then fall through to initialize spinlocks. The vmalloc +* PTEs now point to the concurrent grower's pages. +*/ +while (i >0 ) { + i--; +if (pages[i]) +__free_page (pages[i]); + } +kfree (pages); +memalloc_noreclaim_restore (noreclaim_flags); + +/* Initialize spinlocks for newly mapped clusters */ +for (i = start_idx; i < start_idx + nr; i++) +spin_lock_init (&si->cluster_info[i].lock); + +WRITE_ONCE (si->nr_clusters_mapped, start_idx + nr); +mutex_unlock (&si->xswap_lock); +return 0 ; + fail:while (i >0 ) { i--;
@@ -3736,6 +3826,15 @@ static void xswap_unmap_clusters(struct swap_info_struct *si,*/ WRITE_ONCE(si->nr_clusters_mapped, start_idx); } + +/* +* Callback for apply_to_existing_page_range(): return 1 to stop at the +* first present PTE, signalling that the range is already mapped. +*/ +static int xswap_check_mapped (pte_t *pte,unsigned long addr,void *data) +{ +return 1 ; +}#endif /* CONFIG_XSWAP */ static int setup_swap_clusters_info(struct swap_info_struct *si,
Signature · 1 line
2.54.0
Baoquan He
reviewerRFC PATCH v2 5/10 mm, swap: add xswap_try_shrink and shrink trigger on cluster free
In reply to Baoquan HeAdd xswap_try_shrink() — the shrink logic that scans backwards from the tail to find contiguous free clusters, then unmaps full pages when >= XSWAP_GROW_CLUSTERS free clusters accumulate.
Wire the trigger in __free_cluster(): after a cluster is released to the free list, call xswap_try_shrink() to attempt tail shrinking. Also update the XSWAP_GROW_CLUSTERS comment to reflect both grow and shrink semantics.
--- mm/swapfile.c | 48 ++++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 44 insertions(+), 4 deletions(-)
mm/swapfile.c +44 -4
@@ -51,11 +51,13 @@#ifdef CONFIG_XSWAP /* -* xswap: dynamically grow the cluster_info array via a VM_SPARSE area. +* xswap: dynamically grow and shrink the cluster_info array via a +* VM_SPARSE area. * -* XSWAP_GROW_CLUSTERS is the number of clusters to map in one grow -* operation. It is set to the number of cluster_info structs that -* fit in a single page (at least 16), so that the vmalloc page table +* XSWAP_GROW_CLUSTERS is the number of clusters to map/unmap in one +* grow/shrink operation. It is set to the number of cluster_info +* structs that fit in a single page (at least 16), so that the vmalloc +* page table * overhead is proportional to the number of clusters mapped. */ #define XSWAP_GROW_CLUSTERS \
@@ -66,6 +68,7 @@ static int xswap_map_clusters(struct swap_info_struct *si,static void xswap_unmap_clusters (struct swap_info_struct *si,unsigned long start_idx,unsigned long nr);static int xswap_check_mapped (pte_t *pte,unsigned long addr,void *data); +static void xswap_try_shrink (struct swap_info_struct *si);#endif static void swap_range_alloc(struct swap_info_struct *si,
@@ -629,6 +632,9 @@ static void __free_cluster(struct swap_info_struct *si, struct swap_cluster_infoswap_cluster_free_table (ci);move_cluster (si, ci, &si->free_clusters,CLUSTER_FLAG_FREE ); ci->order =0 ; +#ifdef CONFIG_XSWAP +xswap_try_shrink (si); +#endif }/*
@@ -3835,6 +3841,40 @@ static int xswap_check_mapped(pte_t *pte, unsigned long addr, void *data) {return 1 ; } + +/* +* Try to shrink the cluster_info tail: unmap contiguous free clusters +* at the end of the mapped range. +*/ +static void xswap_try_shrink (struct swap_info_struct *si) +{ +struct swap_cluster_info *ci; +unsigned long last, idx; + +if (!(si->flags &SWP_XSWAP )) +return ; +if (READ_ONCE (si->nr_clusters_mapped) <=1 )/* keep cluster 0 */ +return ; + +/* Find the last non-free cluster from the tail */ + last =READ_ONCE (si->nr_clusters_mapped); +while (last >1 ) { + idx = last -1 ; + ci = &si->cluster_info[idx]; +if (ci->count || ci->flags !=CLUSTER_FLAG_FREE ) +break ; + last = idx; + } + +if (last == si->nr_clusters_mapped) +return ;/* nothing to shrink */ + +/* Only unmap if we can free at least one full page of clusters */ +if (si->nr_clusters_mapped - last <XSWAP_GROW_CLUSTERS ) +return ; + +xswap_unmap_clusters (si, last, si->nr_clusters_mapped - last); +}#endif /* CONFIG_XSWAP */ static int setup_swap_clusters_info(struct swap_info_struct *si,
Signature · 1 line
2.54.0
Baoquan He
reviewerRFC PATCH v2 6/10 mm, swap: free backing pages in xswap_unmap_clusters
In reply to Baoquan Hevm_area_unmap_pages() only clears PTEs and frees intermediate page table pages — it does not free the backing physical pages allocated by xswap_map_clusters().
Fix this by walking the page table with apply_to_existing_page_range() before the unmap to collect all struct pages in the range. After vunmap_range() clears the PTEs, free the collected pages via __free_page().
Use a simple xswap_page_data collector callback: for each present PTE, collect pte_page() into a dynamically allocated array. The array is freed after the pages are released.
--- mm/swapfile.c | 52 +++++++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 46 insertions(+), 6 deletions(-)
mm/swapfile.c +46 -6
@@ -3796,6 +3796,23 @@ static int xswap_map_clusters(struct swap_info_struct *si,return -ENOMEM ; } +struct xswap_page_data { +struct page **pages; +int nr; +int max; +}; + +static int xswap_collect_page (pte_t *pte,unsigned long addr,void *data) +{ +struct xswap_page_data *xpd = data; + +if (!pte_present (*pte)) +return 0 ; +if (xpd->nr < xpd->max) + xpd->pages[xpd->nr++] =pte_page (*pte); +return 0 ; +} +static void xswap_unmap_clusters (struct swap_info_struct *si,unsigned long start_idx,unsigned long nr) {
@@ -3805,11 +3822,15 @@ static void xswap_unmap_clusters(struct swap_info_struct *si,/* * Round to page boundaries: start up (skip partial page that may * contain clusters still in use before start_idx), end up so the -* entire range is covered. vm_area_unmap_pages() operates on -* whole pages. +* entire range is covered. vm_area_unmap_pages() and +* apply_to_existing_page_range() operate on whole pages. */ unsigned long vm_start =PAGE_ALIGN (start_addr);unsigned long vm_end =PAGE_ALIGN (end_addr); +unsigned long size; +unsigned long npages; +struct xswap_page_data xpd; +int i;mutex_lock (&si->xswap_lock);
@@ -3818,12 +3839,31 @@ static void xswap_unmap_clusters(struct swap_info_struct *si, goto out; } -vm_area_unmap_pages (si->cluster_vm, vm_start, vm_end); + size = vm_end - vm_start; + npages = size >>PAGE_SHIFT ; +/* -* vm_area_unmap_pages() only clears PTEs; it does not free the -* physical pages. Walk the page table to find and free them. +* Walk the page table to collect physical pages before unmapping. +* vm_area_unmap_pages() only clears PTEs and frees intermediate +* page table pages — it does not free backing pages. */ -/* TODO: free backing pages via page table walk or tracking bitmap */ + xpd.pages =kmalloc_array (npages,sizeof (*xpd.pages),GFP_KERNEL ); +if (xpd.pages) { + xpd.nr =0 ; + xpd.max = npages; +apply_to_existing_page_range (&init_mm, vm_start, size, + xswap_collect_page, &xpd); + } + +vm_area_unmap_pages (si->cluster_vm, vm_start, vm_end); + +/* Free the collected backing pages */ +if (xpd.pages) { +for (i =0 ; i < xpd.nr; i++) +__free_page (xpd.pages[i]); +kfree (xpd.pages); + } +mutex_unlock (&si->xswap_lock); out:
Signature · 1 line
2.54.0
Baoquan He
reviewerRFC PATCH v2 7/10 mm, swap: add nr_free_tail for O(1) xswap shrink detection
In reply to Baoquan HeTrack contiguous free clusters at the tail of the mapped range in si->nr_free_tail, maintained across alloc/free/grow paths. This eliminates the backwards scan on every shrink check.
Three paths maintain the counter:
1. xswap_update_free_tail(): called on cluster free. If the freed cluster is adjacent to the existing tail boundary, increment and extend backwards to include already-free clusters now connected.
2. xswap_trim_free_tail(): called on cluster allocation. If the allocated cluster lies within the tail free region, truncate the count to end just before it.
3. Grow path: nr_free_tail += nr_new — all newly mapped clusters are immediately free.
xswap_try_shrink() simplifies to a threshold check:
if nr_free_tail >= XSWAP_GROW_CLUSTERS → unmap
Setup initializes nr_free_tail = nr_clusters_mapped - 1 (all but cluster 0 are free at the tail).
--- include/linux/swap.h | 1 + mm/swapfile.c | 121 +++++++++++++++++++++++++++++++++++++------ 2 files changed, 105 insertions(+), 17 deletions(-)
include/linux/swap.h +1 -0
@@ -252,6 +252,7 @@ struct swap_info_struct {struct vm_struct *cluster_vm;/* VM_SPARSE area for xswap dynamic cluster_info */ unsigned long nr_clusters;/* total cluster count for xswap */ unsigned long nr_clusters_mapped;/* currently mapped cluster count */ +unsigned long nr_free_tail;/* contiguous free clusters at tail */ struct mutex xswap_lock;/* serialize map/unmap operations */ #endif struct list_head free_clusters;/* free clusters list */
mm/swapfile.c +104 -17
165 lines not on this page · raw message
Signature · 1 line
2.54.0
Baoquan He
reviewerRFC PATCH v2 8/10 mm, swap: add adjustable runtime ceiling (nr_clusters) for xswap
In reply to Baoquan HeSplit the xswap cluster limit into two fields:
- nr_clusters_max: immutable hard limit set at swapon from swap header
- nr_clusters: current growth ceiling, adjustable at runtime (≤ nr_clusters_max)
The grow path already uses nr_clusters as the ceiling. Shrink now also respects it: when nr_clusters drops below nr_clusters_mapped, shrinking fires on free until the mapped count reaches the ceiling. When nr_clusters == nr_clusters_max (default), shrink is effectively disabled — all growth and no shrink.
At swapon, nr_clusters starts at nr_clusters_max (full size).
--- include/linux/swap.h | 3 ++- mm/swapfile.c | 31 ++++++++++++++++++++++--------- 2 files changed, 24 insertions(+), 10 deletions(-)
include/linux/swap.h +2 -1
@@ -250,7 +250,8 @@ struct swap_info_struct {struct swap_cluster_info *cluster_info;/* cluster info. Only for SSD */ #ifdef CONFIG_XSWAP struct vm_struct *cluster_vm;/* VM_SPARSE area for xswap dynamic cluster_info */ -unsigned long nr_clusters;/* total cluster count for xswap */ +unsigned long nr_clusters_max;/* upper limit from swap header */ +unsigned long nr_clusters;/* current growth ceiling (≤ nr_clusters_max) */ unsigned long nr_clusters_mapped;/* currently mapped cluster count */ unsigned long nr_free_tail;/* contiguous free clusters at tail */ struct mutex xswap_lock;/* serialize map/unmap operations */
mm/swapfile.c +22 -9
@@ -72,6 +72,7 @@ static void xswap_trim_free_tail(struct swap_info_struct *si, unsigned long idx)static void xswap_update_free_tail (struct swap_info_struct *si,unsigned long freed_idx);static void xswap_try_shrink (struct swap_info_struct *si); +#endif static void swap_range_alloc(struct swap_info_struct *si,
@@ -3151,6 +3152,7 @@ static void free_swap_cluster_info(struct swap_info_struct *si)xswap_unmap_clusters (si,0 , si->nr_clusters_mapped);free_vm_area (si->cluster_vm); si->cluster_vm =NULL ; + si->nr_clusters_max =0 ; si->nr_clusters =0 ; si->nr_clusters_mapped =0 ;return ;
@@ -3972,20 +3974,31 @@ static void xswap_trim_free_tail(struct swap_info_struct *si, unsigned long idx)*/ static void xswap_try_shrink (struct swap_info_struct *si) { -unsigned long start_idx, nr_unmap, i; +unsigned long nr_mapped, nr_ceiling, nr_tail, nr_unmap; +unsigned long start_idx, i;struct swap_cluster_info *ci;if (!(si->flags &SWP_XSWAP ))return ; -if (si->nr_free_tail <XSWAP_GROW_CLUSTERS ) + + nr_mapped =READ_ONCE (si->nr_clusters_mapped); + nr_ceiling =READ_ONCE (si->nr_clusters); + nr_tail =READ_ONCE (si->nr_free_tail); + +if (nr_mapped <= nr_ceiling) +return ; +if (nr_tail <XSWAP_GROW_CLUSTERS )return ; - nr_unmap =round_down (si->nr_free_tail,XSWAP_GROW_CLUSTERS ); - start_idx = si->nr_clusters_mapped - nr_unmap; + nr_unmap =min (round_down (nr_tail,XSWAP_GROW_CLUSTERS ), + nr_mapped - nr_ceiling); +if (nr_unmap <XSWAP_GROW_CLUSTERS ) +return ; + start_idx = nr_mapped - nr_unmap;/* Verify the tail clusters are still free before unmapping */ spin_lock (&si->lock); -for (i = start_idx; i < si->nr_clusters_mapped; i++) { +for (i = start_idx; i < nr_mapped; i++) { ci = &si->cluster_info[i];if (ci->flags !=CLUSTER_FLAG_FREE ) { nr_unmap = i - start_idx;
@@ -4000,7 +4013,7 @@ static void xswap_try_shrink(struct swap_info_struct *si)return ;xswap_unmap_clusters (si, start_idx, nr_unmap); - si->nr_free_tail -= nr_unmap; + WRITE_ONCE(si->nr_free_tail, nr_tail - nr_unmap); }#endif /* CONFIG_XSWAP */
@@ -4024,6 +4037,7 @@ static int setup_swap_clusters_info(struct swap_info_struct *si, cluster_info = vm->addr; si->cluster_vm = vm; + si->nr_clusters_max = nr_clusters; si->nr_clusters = nr_clusters; si->cluster_info = cluster_info;
@@ -4058,6 +4072,8 @@ static int setup_swap_clusters_info(struct swap_info_struct *si, } } +/* All mapped clusters except cluster 0 are free at the tail */ + si->nr_free_tail = si->nr_clusters_mapped -1 ;mutex_init (&si->xswap_lock);return 0 ;
@@ -4469,7 +4485,6 @@ void __folio_throttle_swaprate(struct folio *folio, gfp_t gfp)static int __init swapfile_init (void ) { swapfile_maximum_size =arch_max_swapfile_size (); -/* * Once a cluster is freed, it's swap table content is read * only, and all swap cache readers (swap_cache_*) verifies
@@ -4479,12 +4494,10 @@ static int __init swapfile_init(void) swap_table_cachep =kmem_cache_create ("swap_table" ,sizeof (struct swap_table ),0 ,SLAB_PANIC |SLAB_TYPESAFE_BY_RCU ,NULL ); -#ifdef CONFIG_MIGRATION if (swapfile_maximum_size >= (1UL <<SWP_MIG_TOTAL_BITS )) swap_migration_ad_supported = true;#endif /* CONFIG_MIGRATION */ -return 0 ; }subsys_initcall (swapfile_init);
Signature · 1 line
2.54.0
Baoquan He
reviewerRFC PATCH v2 9/10 mm, swap: add debugfs knob for xswap per-device cluster limit
In reply to Baoquan HeAdd a per-device debugfs file for runtime adjustment of xswap cluster limit:
/sys/kernel/debug/xswap/type<N>_cluster_limit
Reading shows the current ceiling (in clusters); writing sets it (clamped to [0, nr_clusters_max]). Setting below nr_clusters_mapped triggers an immediate shrink check via xswap_try_shrink().
The debugfs entry is created at swapon and removed at swapoff.
--- include/linux/swap.h | 1 + mm/swapfile.c | 97 +++++++++++++++++++++++++++++++++++++++++++- 2 files changed, 97 insertions(+), 1 deletion(-)
include/linux/swap.h +1 -0
@@ -254,6 +254,7 @@ struct swap_info_struct {unsigned long nr_clusters;/* current growth ceiling (≤ nr_clusters_max) */ unsigned long nr_clusters_mapped;/* currently mapped cluster count */ unsigned long nr_free_tail;/* contiguous free clusters at tail */ +struct dentry *debugfs_entry;/* debugfs: type<N>_max_clusters */ struct mutex xswap_lock;/* serialize map/unmap operations */ #endif struct list_head free_clusters;/* free clusters list */
mm/swapfile.c +96 -1
139 lines not on this page · raw message
Signature · 1 line
2.54.0
Baoquan He
reviewerRFC PATCH v2 10/10 mm, swap: defer xswap shrink to workqueue to avoid lock recursion
In reply to Baoquan Hexswap_try_shrink() was called directly from __free_cluster() while holding ci->lock. The shrink path calls xswap_unmap_clusters() which unmaps vmalloc pages backing cluster_info, and on return swap_cache_del_folio() tries swap_cluster_unlock(ci) on the now- unmapped address — crashing on a not-present page.
Replace direct calls with schedule_work() so shrink runs in an independent workqueue context where no cluster locks are held. Use cancel_work_sync() during swapoff to ensure no pending shrink work races with the VM area teardown.
--- include/linux/swap.h | 1 + mm/swapfile.c | 19 +++++++++++++------ 2 files changed, 14 insertions(+), 6 deletions(-)
include/linux/swap.h +1 -0
@@ -255,6 +255,7 @@ struct swap_info_struct {unsigned long nr_clusters_mapped;/* currently mapped cluster count */ unsigned long nr_free_tail;/* contiguous free clusters at tail */ struct dentry *debugfs_entry;/* debugfs: type<N>_max_clusters */ +struct work_struct xswap_shrink_work;/* deferred shrink trigger */ struct mutex xswap_lock;/* serialize map/unmap operations */ #endif struct list_head free_clusters;/* free clusters list */
mm/swapfile.c +13 -6
@@ -127,11 +127,8 @@ static ssize_t xswap_max_clusters_write(struct file *file,spin_unlock (&swap_lock); } -/* -* Lowering the ceiling may make tail clusters eligible for -* shrinking. Trigger an immediate check. -*/ -xswap_try_shrink (si); +/* Shrink trigger: lowering the ceiling may free tail clusters. */ +schedule_work (&si->xswap_shrink_work);return count; }
@@ -726,7 +723,7 @@ static void __free_cluster(struct swap_info_struct *si, struct swap_cluster_info ci->order =0 ;#ifdef CONFIG_XSWAP xswap_update_free_tail (si, ci - si->cluster_info); -xswap_try_shrink (si); +schedule_work (&si->xswap_shrink_work);#endif }
@@ -3236,6 +3233,7 @@ static void free_swap_cluster_info(struct swap_info_struct *si)#ifdef CONFIG_XSWAP if (si->flags &SWP_XSWAP ) {xswap_debugfs_del (si); +cancel_work_sync (&si->xswap_shrink_work);/* Unmap all mapped clusters and free the VM_SPARSE area */ if (si->nr_clusters_mapped >0 )xswap_unmap_clusters (si,0 , si->nr_clusters_mapped);
@@ -4057,6 +4055,13 @@ static void xswap_trim_free_tail(struct swap_info_struct *si, unsigned long idx)WRITE_ONCE (si->nr_free_tail, nr_mapped - idx -1 ); } +static void xswap_shrink_work_fn (struct work_struct *work) +{ +struct swap_info_struct *si =container_of (work, + struct swap_info_struct, xswap_shrink_work); +xswap_try_shrink (si); +} +/* * Try to shrink the cluster_info tail. Uses si->nr_free_tail which * is maintained incrementally during alloc/free — no scanning needed.
@@ -4163,7 +4168,9 @@ static int setup_swap_clusters_info(struct swap_info_struct *si,/* All mapped clusters except cluster 0 are free at the tail */ si->nr_free_tail = si->nr_clusters_mapped -1 ; +mutex_init (&si->xswap_lock); +INIT_WORK (&si->xswap_shrink_work, xswap_shrink_work_fn);xswap_debugfs_add (si);return 0 ;
Signature · 1 line
2.54.0