Skip to contentSkip to conversation

mm/swap: introduce priority queues for swap device allocation

RFC PATCH Superseded Lian Wang 2 participants 27 messages Oct 27 – Nov 7

Subsystems (3)

v3 of 4 Superseded Latest is v4 →

lore
Share & tools
lore

Lian Wang

author
#0 lore raw

v3 cover mm/swap: introduce priority queues for swap device allocation

Hi all,

This is v3 of Kairui's swap priority queue RFC [1, 2], retaining the original authorship. It separates device selection from device-local cluster allocation to support integration with the ongoing swap tier work, while preserving existing swap priorities, the swapon interface, and memcg swap accounting and limits.

The global per-CPU cluster cache couples device choice to cluster locality. Moving that cache below device selection gives each device its own allocation hints. The available plist's shared rotation is then replaced by a priority-ordered array of same-priority device rings, with per-CPU starting offsets and quotas. A task-local cursor keeps each retry walk stable. Rotational devices retain per-device sequential hints shared across CPUs.

This remains an RFC. I would appreciate feedback on the allocation model and its integration with the ongoing swap tier work.

Following v1, the changes address related allocation problems:

1) A cache above device selection couples the chosen device to the cached cluster. Device-local ownership lets selection determine the eligible device before consulting its hints, which is useful for tiering.

2) Rotating the available plist changes shared state under swap_avail_lock. Per-CPU readers keep rotation state off that shared selection path, with a quota measured in base pages.

3) Device rotation was tied to cluster draining and the fast/slow split. Selecting a device first also lets allocation handle its pending discard directly, with a bounded amount of synchronous work.

The series is organized as follows:

01-04: Cleanup, device-state boundaries and iteration helpers. 05-07: Locking, writer-serialized publication and accounting.
08: Restore per-device per-CPU cluster locality.
09: Keep cluster metadata population out of direct reclaim.
10-11: Introduce the safely published queue and remove the avail plist.
12: Bound synchronous discard to one cluster per device attempt.
13-14: Preserve priority-aware throttle selection while removing the
active plist and unused plist_requeue API.
15-18: Reuse reader capacity, advance cursors directly, bypass singleton
rotation and redistribute reader phases on membership changes.

Test results
============
The kernel-build and BRD tests follow the v1 workloads on a newer baseline. The results below compare the unpatched baseline 4dbe24720eb3 with queue revision db66c58fac8f. The additional usemem and lock-call results compare the same baseline with 18ec4b962755. These are measurements of the named revisions; this posting contains revision fdfe5fac0d5b.

Eight-ZRAM kernel build
----------------------
The workload used eight 8-GiB ZRAM devices and make -j96 in 2-GiB and 3-GiB memcgs. Each case had 12 fresh-output builds per revision, with no warm-up. The workload source was fixed at 4dbe, using x86 defconfig and the targets bzImage modules. The revisions ran on separate boots, Before first; the 12 samples are within-boot repeats.

Mean system CPU time in seconds (lower is better):

Before Queue (db66) delta

  2-GiB memcg          4469.0100          4457.1575       -0.27%
  3-GiB memcg          2352.7325          2338.9325       -0.59%

Mean elapsed time changed from 170.1283 to 170.3675 s (+0.14%) at 2 GiB and from 136.6125 to 136.6775 s (+0.05%) at 3 GiB. No clear regression was observed in these kernel-build runs.

Twelve-BRD usemem
----------------
The same Before/db66 pair used twelve 8-GiB BRD devices and a private memcg with 24 GiB of memory and 64 GiB of swap allowance.

  usemem --init-time -O -y -x -n 32 1536M

There was one complete invocation per revision. Mapped throughput below is the requested 48 GiB divided by elapsed time, in MiB/s.

Before Queue (db66) delta
elapsed time (s) 7.00 7.11 +1.57%
mapped MiB/s 7021.71 6913.08 -1.55%

All 50 kernel-build and BRD invocations exited successfully, reported no memcg OOM events and performed swap I/O.

Additional eight-ZRAM usemem and lock-call measurements
-----------------------------------------------------
A separate experiment used eight 8-GiB equal-priority ZRAM devices and the same usemem command, with memory.max=24 GiB, memory.swap.max=64 GiB and memory.zswap.max=0 in the workload's memcg. Both kernels ran on the same x86-64 host with configs differing only in LOCALVERSION. There was one workload timing run per revision, with no warm-up, on separate boots, 18ec first.

Before Queue (18ec) delta
elapsed time (s) 7.97 8.59 +7.78%
system CPU time (s) 175.13 192.39 +9.86%
pswpout (pages) 6,317,548 6,317,684 +136

Both runs exited successfully and completed 32 workers, with no memcg OOM events. The times increased in this single-sample comparison; the cause has not been established.

Separate sampled runs recorded 12,451 swap_avail_lock acquisitions at the selected allocation callsites in Before. Source and binary inspection confirmed that these calls are absent from the 18ec allocation reader path. Availability-update callsites recorded zero acquisitions during both sampled workload windows. This shows removal of the shared selection-lock calls, rather than an end-to-end speedup.

Current revision checks
-----------------------
All 18 patches in fdfe replay to their expected intermediate trees and pass checkpatch. Static review and userspace checks of the actual helper code with ASan/UBSan cover publication, ownership, rollback, selector progression and throttle priority selection. These checks use userspace shims and do not constitute kernel runtime testing.

Cooperation with swap tiers
==========================
Device-local cluster hints remain below device selection, so a future tier policy can determine the eligible candidates before those hints are used.

V1 proposed taking tiering first and rebasing this queue onto it as an optimization [1]. I am happy to follow that route. Another option, if reviewers agree, is to take the refactoring under the existing swap priority model first, then work with Youngjun on adapting the queue and reader scope to the tier infrastructure [3].

Youngjun, which ordering would fit your current work better? Are there constraints we should preserve to make a later rebase straightforward?

The integration should have one authority for device membership, swapon/swapoff publication and lifetime. Tier policy would determine the eligible candidates before any device-local hint is used. Within-tier priority semantics need agreement with that work; this series adds no tier ABI or per-memcg device-selection control.

---
Changes in v3:
- Serialize file/ref/writable publication when removing swapon_mutex.
- Establish metadata and writer allocation safety at their first use.
- Preserve priority-aware blkcg throttling when removing the active list.
- Add reader-capacity reuse, direct cursor advancement, singleton selection and membership redistribution.
- Present the completed ZRAM/BRD results with their tested revisions and sample counts, including the observed time increases.
- Add measurements of shared selection-lock calls.

---
Kairui Song (12):
mm/swap: remove unused parameter for reading swap header mm/swap: slightly cleanup the code for hibernation error handling mm/swap: cleanup and document swap device availability flag usage mm/swap: introduce swap device iteration helper mm/swap: change the swapon lock into a percpu rwsem mm/swap: serialize swap publication before removing swapon mutex mm/swap: consolidate swap inuse accounting helpers mm/swap: add priority queue for swap device allocation mm/swap: remove available list
mm/swap: bound synchronous discard during allocation mm/swap: drop swap active plist
lib/plist.c: remove requeue function

Lian Wang (5):
mm/swap: avoid direct reclaim when populating swap cluster metadata mm/swap: reuse per-CPU queue readers across priority changes mm/swap: advance the global queue cursor directly between priority
rings
mm/swap: skip reader rotation for single-device priority rings mm/swap: redistribute queue readers after membership changes

Youngjun Park (1):
mm/swap: change back to use each swap device's percpu cluster

 include/linux/plist.h |    2 -
 include/linux/swap.h  |   43 +-
 lib/plist.c           |   64 --
 mm/swap.h             |   12 +-
 mm/swapfile.c         | 1421 ++++++++++++++++++++++++++---------------
 5 files changed, 926 insertions(+), 616 deletions(-)

---
base-commit: 4dbe24720eb3811563cd56b35d87ddf3f9beed2a

Best regards,

Signature · 1 line
Lian Wang <[email protected]>
author
#1 lore raw

RFC PATCH v3 1/18 mm/swap: remove unused parameter for reading swap header

From: Kairui Song <[email protected]>

No feature change, just a minor cleanup.

---
 mm/swapfile.c | 7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)
mm/swapfile.c +3 -4
index c3288910b3e3..7acb33cd8b71 100644
@@ -3568,9 +3568,8 @@ __weak unsigned long arch_max_swapfile_size(void)
 	return generic_max_swapfile_size();
 }
 
-static unsigned long read_swap_header(struct swap_info_struct *si,
-					union swap_header *swap_header,
-					struct inode *inode)
+static unsigned long read_swap_header(union swap_header *swap_header,
+				      struct inode *inode)
 {
 	int i;
 	unsigned long maxpages;
@@ -3801,7 +3800,7 @@ SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags)
 	}
 	swap_header = kmap_local_folio(folio, 0);
 
-	maxpages = read_swap_header(si, swap_header, inode);
+	maxpages = read_swap_header(swap_header, inode);
 	if (unlikely(!maxpages)) {
 		error = -EINVAL;
 		goto bad_swap_unlock_inode;
author
#2 lore raw

RFC PATCH v3 2/18 mm/swap: slightly cleanup the code for hibernation error handling

In reply to Lian Wang

From: Kairui Song <[email protected]>

Restructure the error paths in pin_hibernation_swap_type() using a goto out pattern to avoid repeated spin_unlock() calls and simplify the return flow. Also simplify unpin_hibernation_swap_type() by making the si NULL check inline, and clean up find_first_swap() to avoid an early return inside the loop.

---
 mm/swapfile.c | 39 +++++++++++++++++----------------------
 1 file changed, 17 insertions(+), 22 deletions(-)
mm/swapfile.c +17 -22
index 7acb33cd8b71..cf8c0999655c 100644
@@ -2279,21 +2279,18 @@ static int __find_hibernation_swap_type(dev_t device, sector_t offset)
  */
 int pin_hibernation_swap_type(dev_t device, sector_t offset)
 {
-	int type;
+	int ret;
 	struct swap_info_struct *si;
 
 	spin_lock(&swap_lock);
+	ret = __find_hibernation_swap_type(device, offset);
+	if (ret < 0)
+		goto out;
 
-	type = __find_hibernation_swap_type(device, offset);
-	if (type < 0) {
-		spin_unlock(&swap_lock);
-		return type;
-	}
-
-	si = swap_type_to_info(type);
+	si = swap_type_to_info(ret);
 	if (WARN_ON_ONCE(!si)) {
-		spin_unlock(&swap_lock);
-		return -ENODEV;
+		ret = -ENODEV;
+		goto out;
 	}
 
 	/*
@@ -2302,14 +2299,15 @@ int pin_hibernation_swap_type(dev_t device, sector_t offset)
 	 * the same session.
 	 */
 	if (WARN_ON_ONCE(si->flags & SWP_HIBERNATION)) {
-		spin_unlock(&swap_lock);
-		return -EBUSY;
+		ret = -EBUSY;
+		goto out;
 	}
 
 	si->flags |= SWP_HIBERNATION;
 
+out:
 	spin_unlock(&swap_lock);
-	return type;
+	return ret;
 }
 
 /**
@@ -2389,11 +2387,8 @@ void unpin_hibernation_swap_type(int type)
 
 	spin_lock(&swap_lock);
 	si = swap_type_to_info(type);
-	if (!si) {
-		spin_unlock(&swap_lock);
-		return;
-	}
-	si->flags &= ~SWP_HIBERNATION;
+	if (si)
+		si->flags &= ~SWP_HIBERNATION;
 	spin_unlock(&swap_lock);
 }
 
@@ -2428,7 +2423,7 @@ int find_hibernation_swap_type(dev_t device, sector_t offset)
 
 int find_first_swap(dev_t *device)
 {
-	int type;
+	int type, ret = -ENODEV;
 
 	spin_lock(&swap_lock);
 	for (type = 0; type < nr_swapfiles; type++) {
@@ -2437,11 +2432,11 @@ int find_first_swap(dev_t *device)
 		if (!(sis->flags & SWP_WRITEOK))
 			continue;
 		*device = sis->bdev->bd_dev;
-		spin_unlock(&swap_lock);
-		return type;
+		ret = type;
+		break;
 	}
 	spin_unlock(&swap_lock);
-	return -ENODEV;
+	return ret;
 }
 
 /*
author
#3 lore raw

RFC PATCH v3 3/18 mm/swap: cleanup and document swap device availability flag usage

In reply to Lian Wang

From: Kairui Song <[email protected]>

Rework swap device flag and metadata handling to make the writable publication and disable boundaries explicit.

Introduce swap_device_enable() and swap_device_disable(). Move SWP_WRITEOK updates into these boundaries while keeping availability updates serialized with swap_avail_lock.

Preserve the existing swap_lock critical section across mapping lookup, hibernation checks and the disable transition. Drain cluster allocators after dropping swap_lock, retaining the device-identity protection without holding swap_lock over all cluster locks.

Document the flag lifetime and locking rules.

---
 include/linux/swap.h |  27 ++++--
 mm/swapfile.c        | 207 ++++++++++++++++++++-----------------------
 2 files changed, 115 insertions(+), 119 deletions(-)
include/linux/swap.h +19 -8
index 4f686709d71f..66b8e602178d 100644
@@ -188,6 +188,22 @@ struct swap_extent {
 	((offsetof(union swap_header, magic.magic) - \
 	  offsetof(union swap_header, info.badpages)) / sizeof(int))
 
+/*
+ * Swap device flags, except the ones documented below, all are immutable
+ * after exposed by swap_device_enable, and until the device is freed again
+ * (SWP_USED unset). The exceptions:
+ * - SWP_USED: Protected by swap_lock. Indicates the device is inuse. Once
+ *   set, won't be cleared unless all reference to this device is freed and
+ *   swapoff finished.
+ * - SWP_WRITEOK: Protected by both swap_lock and swap_avail_lock, clearing
+ *   this flag also waits for all current cluster lock users to exit so
+ *   checking this flag while holding any of these locks ensures the device
+ *   is safe to use at the moment. Note: clearing this flag doesn't affect
+ *   pending IO or async requests, it only prevents further entry allocation
+ *   or new async request (e.g. discard) from initiating.
+ * - SWP_HIBERNATION: Protected by swap_lock. Indicates if the device
+ *   is pinned for hibernation.
+ */
 enum {
 	SWP_USED	= (1 << 0),	/* is slot in swap_info[] used? */
 	SWP_WRITEOK	= (1 << 1),	/* ok to write to this swap?	*/
@@ -256,14 +272,9 @@ struct swap_info_struct {
 	struct file *swap_file;		/* seldom referenced */
 	struct completion comp;		/* seldom referenced */
 	spinlock_t lock;		/*
-					 * protect map scan related fields like
-					 * inuse_pages and all cluster lists.
-					 * Other fields are only changed
-					 * at swapon/swapoff, so are protected
-					 * by swap_lock. changing flags need
-					 * hold this lock and swap_lock. If
-					 * both locks need hold, hold swap_lock
-					 * first.
+					 * Protect cluster lists. Other fields
+					 * are only changed at swapon/swapoff,
+					 * so are protected by swap_lock.
 					 */
 	struct work_struct discard_work; /* discard worker */
 	struct work_struct reclaim_work; /* reclaim worker */
mm/swapfile.c +96 -111

310 lines not on this page · raw message

#4 lore raw

Re: [RFC PATCH v3 03/18] mm/swap: cleanup and document swap device availability flag usage

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [High] mm/swap: swap_device_disable() leaves !SWP_WRITEOK device on swap_avail_head --

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] mm/swap: swap_device_disable() leaves !SWP_WRITEOK device on swap_avail_head --

commit a6079aaef83a224a267931b8819b00f4e3063c85
Author: Kairui Song <[email protected]>

mm/swap: cleanup and document swap device availability flag usage

This commit introduces swap_device_enable() and swap_device_disable() to make swap device writable publication and disable boundaries explicit, while moving cluster lock draining outside of swap_lock.

Quoted 4 lines
mm/swapfile.c +0 -0
index cf8c0999655ca..4adcfbbb37302 100644

[ ... ]

Quoted 1 line
mm/swapfile.c +0 -0
@@ -3078,58 +3070,87 @@ static int setup_swap_extents(struct swap_info_struct *sis,

[ ... ]

Quoted 38 lines
mm/swapfile.c +33 -0
+static int swap_device_disable(struct address_space *mapping,
+			       struct swap_info_struct **swap_info)
+{
+	struct swap_info_struct *si;
+	struct swap_cluster_info *ci;
+	unsigned long offset, end;
+	int err = -EINVAL;
 
 	spin_lock(&swap_lock);
+	plist_for_each_entry(si, &swap_active_head, list) {
+		if ((si->flags & SWP_WRITEOK) &&
+		    si->swap_file->f_mapping == mapping) {
+			err = 0;
+			break;
+		}
+	}
+	if (err)
+		goto unlock;
 
+	/*
+	 * Refuse swapoff while the device is pinned for hibernation.
+	 */
+	if (si->flags & SWP_HIBERNATION) {
+		err = -EBUSY;
+		goto unlock;
+	}
 
+	if (security_vm_enough_memory_mm(current->mm, si->pages)) {
+		err = -ENOMEM;
+		goto unlock;
+	}
+	vm_unacct_memory(si->pages);
 
+	spin_lock(&swap_avail_lock);
+	si->flags &= ~SWP_WRITEOK;
+	spin_unlock(&swap_avail_lock);
+
+	plist_del(&si->list, &swap_active_head);

mm/swapfile.c static int setup_swap_extents(struct swap_info_struct *sis, · line 3140

Quoted 10 lines
mm/swapfile.c +10 -0
+	total_swap_pages -= si->pages;
+	atomic_long_sub(si->pages, &nr_swap_pages);
+
+	end = ALIGN(si->max, SWAPFILE_CLUSTER);
+unlock:
+	spin_unlock(&swap_lock);
+	if (err)
+		return err;
+
+	del_from_avail_list(si, true);

[Severity: High]
Can splitting the clearing of SWP_WRITEOK from del_from_avail_list(si, true) cause an infinite loop in alloc_swap_scan_list() and corrupt the cluster lists?

Previously, del_from_avail_list(p, true) cleared SWP_WRITEOK, set SWAP_USAGE_OFFLIST_BIT, and removed the device from swap_avail_head within a single swap_avail_lock critical section.

With this change (which also persists when swap_lock is later converted to swapon_rwsem and swap_avail_head to swap_queue), swap_device_disable() clears SWP_WRITEOK and drops both swap_avail_lock and swap_lock before calling del_from_avail_list(si, true). During that window, the device remains visible to swap_alloc_slow() with SWP_WRITEOK cleared.

When a concurrent folio_alloc_swap() -> swap_alloc_slow() -> cluster_alloc_swap_entry() picks up this device while holding percpu_swap_cluster.lock, __swap_cluster_alloc_entries() rejects every cluster because SWP_WRITEOK is cleared:

mm/swapfile.c:alloc_swap_scan_cluster() {
	for (end -= nr_pages; offset <= end; offset += nr_pages) {
		...
		if (!__swap_cluster_alloc_entries(si, ci, folio, offset % SWAPFILE_CLUSTER))
			break;
		...
	}
out:
	relocate_cluster(si, ci);
}

Because ci->count is unchanged (0 < ci->count < SWAPFILE_CLUSTER), relocate_cluster() places each isolated cluster onto si->frag_clusters[ci->order].

First, alloc_swap_scan_list(si, &si->nonfull_clusters[order], folio, true) drains all clusters from nonfull_clusters[order] into frag_clusters[order], leaving the cluster lists disorganized if try_to_unuse() later fails and swap_device_enable() re-enables the device.

Second, for an order-0 allocation reaching the higher-order stealing loop in cluster_alloc_swap_entry():

	/* Order 0 stealing from higher order */
	for (int o = 1; o < SWAP_NR_ORDERS; o++) {
		found = alloc_swap_scan_list(si, &si->frag_clusters[o], folio, true);
		if (found)
			goto done;

		found = alloc_swap_scan_list(si, &si->nonfull_clusters[o], folio, true);
		if (found)
			goto done;
	}

if si->frag_clusters[o] is non-empty, alloc_swap_scan_list() with scan_all == true isolates a cluster from frag_clusters[o], fails __swap_cluster_alloc_entries() due to !SWP_WRITEOK, and relocate_cluster() immediately appends the same cluster back to the tail of frag_clusters[o], spinning forever in the do { ... } while (scan_all) loop with preemption disabled.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
author
#5 lore raw

Re: [RFC PATCH v3 03/18] mm/swap: cleanup and document swap device availability flag usage

Quoted 3 lines · On Wed, 28 Oct 2026 16:14:54 +0000 [email protected] wrote:

Can splitting the clearing of SWP_WRITEOK from del_from_avail_list(si, true) cause an infinite loop in alloc_swap_scan_list() and corrupt the cluster lists?

Separating these state transitions does open a selection window. I have not found evidence of list-pointer corruption.

In the next revision, clearing SWP_WRITEOK, setting the off-list bit and removing the device from the available list will again happen in one swap_avail_lock critical section, before dropping swap_lock.

At this point in the series, allocators do not take the topology lock. Moving two separately locked operations under that lock would therefore not be sufficient.

When SWP_WRITEOK is clear, an order-0 allocation scanning a partially occupied higher-order fragment cluster with scan_all=true can make no progress. The cluster lock is released on each iteration, but the cluster can keep returning to the same list.

An allocator that selected the device before disable can encounter the same underlying scan problem on the base tree. I have prepared a separate
fix: stop before isolating the next cluster when SWP_WRITEOK is clear, leaving the current iteration's normal cleanup intact.

I checked the scan helper in an isolated x86-64 KUnit guest. On v3, the disabled higher-order-fragment case did not complete; the stall stack included alloc_swap_scan_cluster(), and the host stopped the guest after 45 seconds. On the revised tree, that case and both enabled-scan controls completed successfully.

This tests the scan path directly, not the concurrent swapoff selection window through syscalls. The publication boundary is checked separately in the code and through the intermediate commits.

Thanks for the report. I will include the availability-boundary correction in the next revision and keep the base scan fix separate.

Lian

author
#6 lore raw

RFC PATCH v3 4/18 mm/swap: introduce swap device iteration helper

In reply to Lian Wang

From: Kairui Song <[email protected]>

There are many users that access the swap info array just to iterate through the swap devices to find usable ones. Introduce a generic helper for this to prepare for dropping the lock convention.

Also slightly adjust __swap_type_to_info to allow access of swapped off devices by moving the sanity check into its only current caller, and this way makes more sense too: the only caller is __swap_entry_to_info, where we should never see an entry referring to a dead device.

---
 mm/swap.h     | 12 ++++----
 mm/swapfile.c | 77 +++++++++++++++++++++++++++++++--------------------
 2 files changed, 53 insertions(+), 36 deletions(-)
mm/swap.h +6 -6
index d5bf21f517dc..d46d09b8c21f 100644
@@ -107,16 +107,16 @@ static inline unsigned int swp_cluster_offset(swp_entry_t entry)
  */
 static inline struct swap_info_struct *__swap_type_to_info(int type)
 {
-	struct swap_info_struct *si;
-
-	si = READ_ONCE(swap_info[type]); /* rcu_dereference() */
-	VM_WARN_ON_ONCE(percpu_ref_is_zero(&si->users)); /* race with swapoff */
-	return si;
+	return READ_ONCE(swap_info[type]); /* rcu_dereference() */
 }
 
 static inline struct swap_info_struct *__swap_entry_to_info(swp_entry_t entry)
 {
-	return __swap_type_to_info(swp_type(entry));
+	struct swap_info_struct *si;
+
+	si = __swap_type_to_info(swp_type(entry)); /* rcu_dereference() */
+	VM_WARN_ON_ONCE(percpu_ref_is_zero(&si->users)); /* race with swapoff */
+	return si;
 }
 
 static inline struct swap_cluster_info *__swap_offset_to_cluster(
mm/swapfile.c +47 -30

121 lines not on this page · raw message

author
#7 lore raw

RFC PATCH v3 5/18 mm/swap: change the swapon lock into a percpu rwsem

In reply to Lian Wang

From: Kairui Song <[email protected]>

Reading swap metadata is a common hot path, while swapon and swapoff are costly and very rare. Convert the existing swap_lock spinlock into a percpu rwsem so that readers can run concurrently without contention. This is a first step toward cleaning up the swap lock model and we might already see a performance gain for existing readers.

Also update the comments in swap_info_struct to reflect the lock name change.

---
 include/linux/swap.h |  10 ++--
 mm/swapfile.c        | 129 +++++++++++++++++++++++--------------------
 2 files changed, 73 insertions(+), 66 deletions(-)
include/linux/swap.h +5 -5
index 66b8e602178d..80a14a8daba2 100644
@@ -191,17 +191,17 @@ struct swap_extent {
 /*
  * Swap device flags, except the ones documented below, all are immutable
  * after exposed by swap_device_enable, and until the device is freed again
- * (SWP_USED unset). The exceptions:
- * - SWP_USED: Protected by swap_lock. Indicates the device is inuse. Once
+ * (SWP_USED unset). The exceptions are all protected by swapon_rwsem:
+ * - SWP_USED: Protected by swapon_rwsem. Indicates the device is inuse. Once
  *   set, won't be cleared unless all reference to this device is freed and
  *   swapoff finished.
- * - SWP_WRITEOK: Protected by both swap_lock and swap_avail_lock, clearing
+ * - SWP_WRITEOK: Protected by both swapon_rwsem and swap_avail_lock, clearing
  *   this flag also waits for all current cluster lock users to exit so
  *   checking this flag while holding any of these locks ensures the device
  *   is safe to use at the moment. Note: clearing this flag doesn't affect
  *   pending IO or async requests, it only prevents further entry allocation
  *   or new async request (e.g. discard) from initiating.
- * - SWP_HIBERNATION: Protected by swap_lock. Indicates if the device
+ * - SWP_HIBERNATION: Protected by swapon_rwsem. Indicates if the device
  *   is pinned for hibernation.
  */
 enum {
@@ -274,7 +274,7 @@ struct swap_info_struct {
 	spinlock_t lock;		/*
 					 * Protect cluster lists. Other fields
 					 * are only changed at swapon/swapoff,
-					 * so are protected by swap_lock.
+					 * so are protected by swapon_rwsem.
 					 */
 	struct work_struct discard_work; /* discard worker */
 	struct work_struct reclaim_work; /* reclaim worker */
mm/swapfile.c +68 -61

380 lines not on this page · raw message

#8 lore raw

Re: [RFC PATCH v3 05/18] mm/swap: change the swapon lock into a percpu rwsem

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [High] mm/swap: infinite loop in cluster_alloc_swap_entry() on disabled swap device --

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] mm/swap: infinite loop in cluster_alloc_swap_entry() on disabled swap device --

commit 75f9eec48d8910061cd39d390305f876926fa1d7
Author: Kairui Song <[email protected]>

mm/swap: change the swapon lock into a percpu rwsem

This commit converts swap_lock from a spinlock to a percpu rwsem named swapon_rwsem so that readers of swap metadata can run concurrently without contention against swapon and swapoff writers.

Quoted 4 lines
mm/swapfile.c +0 -0
index b72f4fc3e8117..4ffde6827d441 100644

[ ... ]

mm/swapfile.c static int swap_device_disable(struct address_space *mapping, · line 3162

Quoted 10 lines
mm/swapfile.c +1 -1
@@ -3152,14 +3155,14 @@ static int swap_device_disable(struct address_space *mapping,
 
 	end = ALIGN(si->max, SWAPFILE_CLUSTER);
 unlock:
-	spin_unlock(&swap_lock);
+	percpu_up_write(&swapon_rwsem);
 	if (err)
 		return err;
 
 	del_from_avail_list(si, true);

[Severity: High]
Can clearing SWP_WRITEOK and calling percpu_up_write(&swapon_rwsem) in mm/swapfile.c:swap_device_disable() before removing si from the available devices in del_from_avail_list(si, true) cause a concurrent order-0 swap allocation to enter an infinite loop in cluster_alloc_swap_entry()?

In swap_device_disable(), si->flags &= ~SWP_WRITEOK is performed under swap_avail_lock, and then both swap_avail_lock and swapon_rwsem are released before del_from_avail_list(si, true) removes si from swap_avail_head (or masks si in swap_queue after the later commit "mm/swap: add priority queue for swap device allocation", where percpu_up_write(&swapon_rwsem) also wakes readers waiting on swapon_rwsem before del_from_avail_list() runs):

mm/swapfile.c:swap_device_disable() {
	...
	spin_lock(&swap_avail_lock);
	si->flags &= ~SWP_WRITEOK;
	spin_unlock(&swap_avail_lock);

	plist_del(&si->list, &swap_active_head);
	total_swap_pages -= si->pages;
	atomic_long_sub(si->pages, &nr_swap_pages);

	end = ALIGN(si->max, SWAPFILE_CLUSTER);
unlock:
	percpu_up_write(&swapon_rwsem);
	if (err)
		return err;

	del_from_avail_list(si, true);
}

If a concurrent order-0 folio_alloc_swap() selects si while si is still visible as available but has SWP_WRITEOK cleared, it calls cluster_alloc_swap_entry(si, folio).

First, alloc_swap_scan_list(si, &si->nonfull_clusters[order], folio, true) isolates each cluster ci and sets ci->flags = CLUSTER_FLAG_NONE. Inside alloc_swap_scan_cluster(), __swap_cluster_alloc_entries() returns false because !(si->flags & SWP_WRITEOK), and relocate_cluster() moves every non-full cluster onto si->frag_clusters[ci->order].

Then, for an order-0 folio, cluster_alloc_swap_entry() enters the higher-order stealing loop:

mm/swapfile.c:cluster_alloc_swap_entry() {
	...
	/* Order 0 stealing from higher order */
	for (int o = 1; o < SWAP_NR_ORDERS; o++) {
		/*
		 * Clusters here have at least one usable slots and can't fail order 0
		 * allocation, but reclaim may drop si->lock and race with another user.
		 */
		found = alloc_swap_scan_list(si, &si->frag_clusters[o], folio, true);
		if (found)
			goto done;
		...
	}
}

And alloc_swap_scan_list() loops while scan_all is true:

mm/swapfile.c:alloc_swap_scan_list() {
	do {
		struct swap_cluster_info *ci = isolate_lock_cluster(si, list);
		unsigned long offset;

		if (!ci)
			break;
		offset = cluster_offset(si, ci);
		found = alloc_swap_scan_cluster(si, ci, folio, offset);
		if (found)
			break;
	} while (scan_all);
}

Whenever si->frag_clusters[o] is non-empty, isolate_lock_cluster() pops ci from si->frag_clusters[o] and sets ci->flags = CLUSTER_FLAG_NONE, __swap_cluster_alloc_entries() fails because !(si->flags & SWP_WRITEOK), and relocate_cluster() sees a non-empty, non-full cluster with ci->flags != CLUSTER_FLAG_FRAG and appends ci right back onto si->frag_clusters[o]. Because found is 0 and scan_all is true, wouldn't alloc_swap_scan_list() spin forever on si->frag_clusters[o] (and also leave nonfull_clusters drained into frag_clusters if swapoff later fails in try_to_unuse() and re-enables the device)?

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=5
author
#9 lore raw

Re: [RFC PATCH v3 05/18] mm/swap: change the swapon lock into a percpu rwsem

Quoted 2 lines · On Wed, 28 Oct 2026 16:14:56 +0000 [email protected] wrote:

cause a concurrent order-0 swap
allocation to enter an infinite loop in cluster_alloc_swap_entry()?

Yes. Clearing SWP_WRITEOK while leaving the device selectable opens the same availability window reported on patch 03. I will restore the atomic flag/off-list/list-removal transition in patch 03 and preserve it through this lock conversion.

The device-selection and cluster-allocation paths do not yet hold swapon_rwsem here, so merely moving the separate operations inside the writer is insufficient. The transition must stay within one swap_avail_lock critical section. The synchronous-discard fallback already takes a short reader, but it does not protect this earlier selection and scan. Once patch 10 protects allocation walks with a reader, the queue mask must also be complete before the writer is released.

I am keeping the underlying scan-liveness guard separate, since an allocator selected before disable can encounter that problem on the base tree as well.

The final revised tree passes the targeted KUnit scan cases, including the disabled fragment scan that does not complete on v3. The earlier availability-lock boundary has been checked in code; this guest comparison does not execute each intermediate commit or the syscall race.

Thanks for the report.

Lian

author
#10 lore raw

RFC PATCH v3 6/18 mm/swap: serialize swap publication before removing swapon mutex

In reply to Lian Wang

From: Kairui Song <[email protected]>

Move publication of swap_file, resurrection of the device reference and the writable transition into one swapon_rwsem writer section. Split out a writer-held enable helper so initial publication can share this section, while failed swapoff can use the locking wrapper.

With file publication and teardown serialized by the writer, remove swapon_mutex and use a topology reader across /proc/swaps iteration. Keep the SWP_USED filter when advancing the swap_info array.

The publication boundary excludes proc readers from partially published state and gives the lockless live-reference users initialized device metadata before the device becomes an allocation candidate.

---
 mm/swapfile.c | 39 +++++++++++++++++++--------------------
 1 file changed, 19 insertions(+), 20 deletions(-)
mm/swapfile.c +19 -20
index 4ffde6827d44..4c10e4a2d9b3 100644
@@ -140,9 +140,6 @@ static inline struct swap_info_struct *__swap_iter(int *i, unsigned long flag)
 
 static struct kmem_cache *swap_table_cachep;
 
-/* Protects si->swap_file for /proc/swaps usage */
-static DEFINE_MUTEX(swapon_mutex);
-
 static DECLARE_WAIT_QUEUE_HEAD(proc_poll_wait);
 /* Activity counter to indicate that a swapon or swapoff has occurred */
 static atomic_t proc_poll_event = ATOMIC_INIT(0);
@@ -3097,9 +3094,9 @@ static int setup_swap_extents(struct swap_info_struct *sis,
  * Mark a fully initialized swap device writable and expose it to the
  * allocator. The caller must have resurrected its percpu ref first.
  */
-static void swap_device_enable(struct swap_info_struct *si)
+static void __swap_device_enable(struct swap_info_struct *si)
 {
-	percpu_down_write(&swapon_rwsem);
+	lockdep_assert_held_write(&swapon_rwsem);
 	spin_lock(&swap_avail_lock);
 	si->flags |= SWP_WRITEOK;
 	spin_unlock(&swap_avail_lock);
@@ -3107,11 +3104,16 @@ static void swap_device_enable(struct swap_info_struct *si)
 	atomic_long_add(si->pages, &nr_swap_pages);
 	total_swap_pages += si->pages;
 	plist_add(&si->list, &swap_active_head);
-	percpu_up_write(&swapon_rwsem);
-
 	add_to_avail_list(si);
 }
 
+static void swap_device_enable(struct swap_info_struct *si)
+{
+	percpu_down_write(&swapon_rwsem);
+	__swap_device_enable(si);
+	percpu_up_write(&swapon_rwsem);
+}
+
 static int swap_device_disable(struct address_space *mapping,
 			       struct swap_info_struct **swap_info)
 {
@@ -3275,7 +3277,6 @@ SYSCALL_DEFINE1(swapoff, const char __user *, specialfile)
 	if (!(p->flags & SWP_SOLIDSTATE))
 		atomic_dec(&nr_rotate_swap);
 
-	mutex_lock(&swapon_mutex);
 	percpu_down_write(&swapon_rwsem);
 	spin_lock(&p->lock);
 	drain_mmlist();
@@ -3290,7 +3291,6 @@ SYSCALL_DEFINE1(swapoff, const char __user *, specialfile)
 	percpu_up_write(&swapon_rwsem);
 	arch_swap_invalidate_area(p->type);
 	zswap_swapoff(p->type);
-	mutex_unlock(&swapon_mutex);
 	kfree(p->global_cluster);
 	p->global_cluster = NULL;
 	free_swap_cluster_info(cluster_info, maxpages);
@@ -3336,20 +3336,18 @@ static __poll_t swaps_poll(struct file *file, poll_table *wait)
 	return EPOLLIN | EPOLLRDNORM;
 }
 
-/* iterator */
 static void *swap_start(struct seq_file *swap, loff_t *pos)
 {
 	struct swap_info_struct *si;
-	int type;
 	loff_t l = *pos;
 
-	mutex_lock(&swapon_mutex);
+	percpu_down_read(&swapon_rwsem);
 
 	if (!l)
 		return SEQ_START_TOKEN;
 
-	for (type = 0; (si = swap_type_to_info(type)); type++) {
-		if (!(si->swap_file))
+	for_each_swap(si) {
+		if (!si->swap_file)
 			continue;
 		if (!--l)
 			return si;
@@ -3370,7 +3368,7 @@ static void *swap_next(struct seq_file *swap, void *v, loff_t *pos)
 
 	++(*pos);
 	for (; (si = swap_type_to_info(type)); type++) {
-		if (!(si->swap_file))
+		if (!(si->flags & SWP_USED) || !(si->swap_file))
 			continue;
 		return si;
 	}
@@ -3380,7 +3378,7 @@ static void *swap_next(struct seq_file *swap, void *v, loff_t *pos)
 
 static void swap_stop(struct seq_file *swap, void *v)
 {
-	mutex_unlock(&swapon_mutex);
+	percpu_up_read(&swapon_rwsem);
 }
 
 static int swap_show(struct seq_file *swap, void *v)
@@ -3893,7 +3891,6 @@ SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags)
 		goto free_swap_zswap;
 	}
 
-	mutex_lock(&swapon_mutex);
 	prio = DEF_SWAP_PRIO;
 	if (swap_flags & SWAP_FLAG_PREFER)
 		prio = swap_flags & SWAP_FLAG_PRIO_MASK;
@@ -3905,11 +3902,14 @@ SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags)
 	si->prio = prio;
 	si->list.prio = -si->prio;
 	si->avail_list.prio = -si->prio;
+	/* Publish the file, live reference and writable state together. */
+	percpu_down_write(&swapon_rwsem);
 	si->swap_file = swap_file;
 
-	/* Sets SWP_WRITEOK, resurrect the percpu ref, expose the swap device */
+	/* Make the reference live before exposing the writable device. */
 	percpu_ref_resurrect(&si->users);
-	swap_device_enable(si);
+	__swap_device_enable(si);
+	percpu_up_write(&swapon_rwsem);
 
 	pr_info("Adding %uk swap on %s.  Priority:%d extents:%d across:%lluk %s%s%s%s\n",
 		K(si->pages), name->name, si->prio, nr_extents,
@@ -3919,7 +3919,6 @@ SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags)
 		(si->flags & SWP_AREA_DISCARD) ? "s" : "",
 		(si->flags & SWP_PAGE_DISCARD) ? "c" : "");
 
-	mutex_unlock(&swapon_mutex);
 	atomic_inc(&proc_poll_event);
 	wake_up_interruptible(&proc_poll_wait);
 
author
#11 lore raw

RFC PATCH v3 7/18 mm/swap: consolidate swap inuse accounting helpers

In reply to Lian Wang

From: Kairui Song <[email protected]>

Inline the swap_usage_add/sub helpers into their only callers and rename the callers to better reflect what they actually do: track per-device inuse page counts.

No functional change.

---
 mm/swapfile.c | 64 ++++++++++++++++++++++-----------------------------
 1 file changed, 27 insertions(+), 37 deletions(-)
mm/swapfile.c +27 -37
index 4c10e4a2d9b3..c531e0c45073 100644
@@ -49,8 +49,8 @@
 #include "internal.h"
 #include "swap.h"
 
-static void swap_range_alloc(struct swap_info_struct *si,
-			     unsigned int nr_entries);
+static void swap_device_inuse_add(struct swap_info_struct *si,
+				  unsigned int nr_entries);
 static bool folio_swapcache_freeable(struct folio *folio);
 static void move_cluster(struct swap_info_struct *si,
 			 struct swap_cluster_info *ci, struct list_head *list,
@@ -992,7 +992,7 @@ static bool __swap_cluster_alloc_entries(struct swap_info_struct *si,
 	if (cluster_is_empty(ci))
 		ci->order = order;
 	ci->count += nr_pages;
-	swap_range_alloc(si, nr_pages);
+	swap_device_inuse_add(si, nr_pages);
 
 	return true;
 }
@@ -1300,53 +1300,36 @@ skip:
 }
 
 /*
- * swap_usage_add / swap_usage_sub of each slot are serialized by ci->lock
- * within each cluster, so the total contribution to the global counter should
- * always be positive and cannot exceed the total number of usable slots.
+ * swap_device_inuse_add / swap_device_inuse_sub are called within cluster
+ * update critical sections and serialized by ci->lock within each cluster,
+ * so the total contribution to the global counter should always be positive
+ * and cannot exceed the total number of usable slots.
  */
-static bool swap_usage_add(struct swap_info_struct *si, unsigned int nr_entries)
+static void swap_device_inuse_add(struct swap_info_struct *si,
+				  unsigned int nr_entries)
 {
-	long val = atomic_long_add_return_relaxed(nr_entries, &si->inuse_pages);
+	long inuse_pages;
 
 	/*
 	 * If device is full, and SWAP_USAGE_OFFLIST_BIT is not set,
 	 * remove it from the plist.
 	 */
-	if (unlikely(val == si->pages)) {
-		del_from_avail_list(si, false);
-		return true;
-	}
-
-	return false;
-}
-
-static void swap_usage_sub(struct swap_info_struct *si, unsigned int nr_entries)
-{
-	long val = atomic_long_sub_return_relaxed(nr_entries, &si->inuse_pages);
-
-	/*
-	 * If device is not full, and SWAP_USAGE_OFFLIST_BIT is set,
-	 * add it to the plist.
-	 */
-	if (unlikely(val & SWAP_USAGE_OFFLIST_BIT))
-		add_to_avail_list(si);
-}
-
-static void swap_range_alloc(struct swap_info_struct *si,
-			     unsigned int nr_entries)
-{
-	if (swap_usage_add(si, nr_entries)) {
+	inuse_pages = atomic_long_add_return_relaxed(nr_entries, &si->inuse_pages);
+	if (unlikely(inuse_pages == si->pages)) {
 		if (vm_swap_full())
 			schedule_work(&si->reclaim_work);
+		del_from_avail_list(si, false);
 	}
+
 	atomic_long_sub(nr_entries, &nr_swap_pages);
 }
 
-static void swap_range_free(struct swap_info_struct *si, unsigned long offset,
-			    unsigned int nr_entries)
+static void swap_device_inuse_sub(struct swap_info_struct *si, unsigned long offset,
+				  unsigned int nr_entries)
 {
 	unsigned long end = offset + nr_entries - 1;
 	void (*swap_slot_free_notify)(struct block_device *, unsigned long);
+	long inuse_pages;
 
 	zswap_invalidate(si->type, offset, nr_entries);
 
@@ -1368,7 +1351,14 @@ static void swap_range_free(struct swap_info_struct *si, unsigned long offset,
 	 */
 	smp_wmb();
 	atomic_long_add(nr_entries, &nr_swap_pages);
-	swap_usage_sub(si, nr_entries);
+
+	/*
+	 * If device is not full, and SWAP_USAGE_OFFLIST_BIT is set,
+	 * add it back to the plist.
+	 */
+	inuse_pages = atomic_long_sub_return_relaxed(nr_entries, &si->inuse_pages);
+	if (unlikely(inuse_pages & SWAP_USAGE_OFFLIST_BIT))
+		add_to_avail_list(si);
 }
 
 static bool get_swap_device_info(struct swap_info_struct *si)
@@ -1994,7 +1984,7 @@ void __swap_cluster_free_entries(struct swap_info_struct *si,
 	if (batch_id)
 		mem_cgroup_uncharge_swap(batch_id, ci_off - batch_off);
 
-	swap_range_free(si, ci_head + ci_start, nr_pages);
+	swap_device_inuse_sub(si, ci_head + ci_start, nr_pages);
 	swap_cluster_assert_empty(ci, ci_start, nr_pages, false);
 
 	if (!ci->count)
@@ -2935,7 +2925,7 @@ retry:
 success:
 	/*
 	 * Make sure that further cleanups after try_to_unuse() returns happen
-	 * after swap_range_free() reduces si->inuse_pages to 0.
+	 * after swap_device_inuse_sub() reduces si->inuse_pages to 0.
 	 */
 	smp_mb();
 	return 0;
author
#12 lore raw

RFC PATCH v3 8/18 mm/swap: change back to use each swap device's percpu cluster

In reply to Lian Wang

From: Youngjun Park <[email protected]>

This reverts commit 1b7e90020eb7 ("mm, swap: use percpu cluster as allocation fast path").

The global percpu cluster causes several issues:

1) It prevents efficient swap allocation for users that do not want the default swap device policy. Because the cluster cache sits above device selection, all allocation users must stick with the default allocation policy. This fundamentally conflicts with incoming ideas like swap tiering.

2) It can cause priority inversion. Consider two memcgs: memcg1 can access devices A and B, where A has higher priority; memcg2 can only access B. Memcg2 can write the global percpu cluster with device B, then memcg1 takes B in the fast path even though higher-priority device A is not exhausted.

3) The fast-path / slow-path design bundled with the global per-CPU cache is problematic: only the fast path uses the global cache, and the slow path is forced to rotate the allocation list. This complicates consumers like discard.

Revert to per-device percpu clusters first. A later patch will introduce new infrastructure for a high-performance global allocation path.

---
 include/linux/swap.h |  13 +++-
 mm/swapfile.c        | 170 ++++++++++++++-----------------------------
 2 files changed, 63 insertions(+), 120 deletions(-)
include/linux/swap.h +10 -3
index 80a14a8daba2..1a1ed9d11ad0 100644
@@ -239,10 +239,17 @@ enum {
 #endif
 
 /*
- * We keep using same cluster for rotational device so IO will be sequential.
- * The purpose is to optimize SWAP throughput on these device.
+ * We assign a cluster to each CPU, so each CPU can allocate swap entry from
+ * its own cluster and swapout sequentially. The purpose is to optimize swapout
+ * throughput.
  */
+struct percpu_cluster {
+	local_lock_t lock; /* Protect the percpu_cluster above */
+	unsigned int next[SWAP_NR_ORDERS]; /* Likely next allocation offset */
+};
+
 struct swap_sequential_cluster {
+	spinlock_t lock; /* Serialize usage of global cluster */
 	unsigned int next[SWAP_NR_ORDERS]; /* Likely next allocation offset */
 };
 
@@ -265,8 +272,8 @@ struct swap_info_struct {
 					/* list of cluster that are fragmented or contented */
 	unsigned int pages;		/* total of usable pages of swap */
 	atomic_long_t inuse_pages;	/* number of those currently in use */
+	struct percpu_cluster	__percpu *percpu_cluster; /* per cpu's swap location */
 	struct swap_sequential_cluster *global_cluster; /* Use one global cluster for rotating device */
-	spinlock_t global_cluster_lock;	/* Serialize usage of global cluster */
 	struct rb_root swap_extent_root;/* root of the swap extent rbtree */
 	struct block_device *bdev;	/* swap device or bdev of swap file */
 	struct file *swap_file;		/* seldom referenced */
mm/swapfile.c +53 -117

293 lines not on this page · raw message

author
#13 lore raw

RFC PATCH v3 9/18 mm/swap: avoid direct reclaim when populating swap cluster metadata

In reply to Lian Wang

Swap cluster population is entered from swap allocation and may itself need memory for slot metadata. Its fallback drops the device and cluster locks before allocating with GFP_KERNEL, which can reenter swap allocation through direct reclaim.

Make this allocation boundary explicitly non-reclaiming so callers can retain outer synchronization without a recursive swap allocation. Clear __GFP_DIRECT_RECLAIM from the fallback request while retaining __GFP_HIGH and __GFP_NOMEMALLOC. The retry can still wake kswapd, but fails without direct reclaim when memory is unavailable.

This can make metadata allocation fail earlier under memory pressure. Keep the existing isolated-cluster rollback and restrict this request without adding a task-wide PF_MEMALLOC scope.

---
 mm/swapfile.c | 23 +++++++++++++----------
 1 file changed, 13 insertions(+), 10 deletions(-)
mm/swapfile.c +13 -10
index 2035513d3c61..68bed5023299 100644
@@ -545,13 +545,14 @@ static void swap_cluster_assert_empty(struct swap_cluster_info *ci,
 }
 
 /*
- * Allocate swap table for one cluster. Attempt an atomic allocation first,
- * then fallback to sleeping allocation.
+ * Allocate swap table for one cluster. Attempt allocation under the locks
+ * first, then retry with the locks dropped without entering direct reclaim.
  */
 static struct swap_cluster_info *
 swap_cluster_populate(struct swap_info_struct *si,
 			 struct swap_cluster_info *ci)
 {
+	gfp_t gfp;
 	int ret;
 
 	/*
@@ -569,9 +570,11 @@ swap_cluster_populate(struct swap_info_struct *si,
 		return ci;
 
 	/*
-	 * Try a sleep allocation. Each isolated free cluster may cause
-	 * a sleep allocation, but there is a limited number of them, so
-	 * the potential recursive allocation is limited.
+	 * Drop the device and cluster locks before retrying. Metadata allocation
+	 * must not recurse into swap allocation through direct reclaim: callers
+	 * can retain outer synchronization across cluster population. Restrict
+	 * this request instead of granting the task PF_MEMALLOC, which could
+	 * give allocator hooks and failure cleanup reserve access.
 	 */
 	spin_unlock(&ci->lock);
 	if (si->flags & SWP_SOLIDSTATE)
@@ -579,8 +582,9 @@ swap_cluster_populate(struct swap_info_struct *si,
 	else
 		spin_unlock(&si->global_cluster->lock);
 
-	ret = swap_cluster_alloc_table(ci, __GFP_HIGH | __GFP_NOMEMALLOC |
-					   GFP_KERNEL);
+	gfp = (GFP_KERNEL | __GFP_HIGH | __GFP_NOMEMALLOC) &
+		~__GFP_DIRECT_RECLAIM;
+	ret = swap_cluster_alloc_table(ci, gfp);
 
 	/*
 	 * Back to atomic context. We might have migrated to a new CPU with a
@@ -3095,9 +3099,8 @@ unlock:
 	del_from_avail_list(si, true);
 
 	/*
-	 * The swap allocator doesn't take swapon_rwsem. Looping through every
-	 * cluster lock after clearing SWP_WRITEOK ensures that allocators see
-	 * the updated flag and that no allocation remains in flight.
+	 * Drain cluster-lock users after clearing SWP_WRITEOK, so swapoff
+	 * does not race with operations that started before the transition.
 	 */
 	for (offset = 0; offset < end; offset += SWAPFILE_CLUSTER) {
 		ci = swap_cluster_lock(si, offset);
author
#14 lore raw

RFC PATCH v3 10/18 mm/swap: add priority queue for swap device allocation

In reply to Lian Wang

From: Kairui Song <[email protected]>

Swap device selection rotates swap_avail_head with plist_requeue(), serializing cluster transitions on swap_avail_lock. Separate selection from device-local cluster allocation with same-priority device rings ordered from highest priority to lowest.

Keep membership and sizes stable under swapon_rwsem during each allocation walk. Full or disabled devices remain as masked pointers. Serialize tag writers with swap_queue_update_lock and use READ_ONCE() and WRITE_ONCE() for complete tagged-pointer accesses.

Each CPU has a starting offset and base-page quota per ring. A task-local cursor snapshots its starting device so retries visit every peer once, even if another task changes the shared reader. Disable migration across the sleepable walk. Large folios retry same-priority peers before asking the caller to split instead of falling through after one fragmented device.

Prepare reader storage with GFP_NOIO outside the topology writer, then recheck priority and required length before publication. Transfer the prepared storage only on successful installation; otherwise free it after unlocking. Keep per-CPU backing allocation outside the writer's noreclaim scope because it filters caller GFP flags. Constrain small queue/ring allocations with a scoped noreclaim context and __GFP_NOMEMALLOC, restoring context before cleanup. Deletion keeps reader storage until replacement or an empty queue, avoiding shrinking backing allocations beneath the writer.

These boundaries prevent queue updates from reclaiming into a swap allocation that would acquire a reader behind the same writer. The cluster metadata fallback also avoids direct reclaim while the walk retains its outer reader. These recursive paths were identified by source inspection rather than reproduced hangs.

Publish swap_file, the live reference, queue membership and SWP_WRITEOK under the same writer. Mask the off-list flag from the old in-use counter before its full-device comparison so concurrent freeing/refilling keeps availability state consistent.

Keep the old available plist in parallel while migrating its remaining consumers.

---
 include/linux/swap.h |   5 +-
 mm/swapfile.c        | 581 ++++++++++++++++++++++++++++++++++++++-----
 2 files changed, 519 insertions(+), 67 deletions(-)
include/linux/swap.h +3 -2
index 1a1ed9d11ad0..9357a9065f37 100644
@@ -195,8 +195,9 @@ struct swap_extent {
  * - SWP_USED: Protected by swapon_rwsem. Indicates the device is inuse. Once
  *   set, won't be cleared unless all reference to this device is freed and
  *   swapoff finished.
- * - SWP_WRITEOK: Protected by both swapon_rwsem and swap_avail_lock, clearing
- *   this flag also waits for all current cluster lock users to exit so
+ * - SWP_WRITEOK: Protected by both swapon_rwsem and swap_queue_update_lock.
+ *   Clearing this flag is followed by waiting for all current cluster lock
+ *   users to exit, so
  *   checking this flag while holding any of these locks ensures the device
  *   is safe to use at the moment. Note: clearing this flag doesn't affect
  *   pending IO or async requests, it only prevents further entry allocation
mm/swapfile.c +516 -65

713 lines not on this page · raw message

#15 lore raw

Re: [RFC PATCH v3 10/18] mm/swap: add priority queue for swap device allocation

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [High] mm/swap: swap_device_disable() drops swapon_rwsem before masking device --

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] mm/swap: swap_device_disable() drops swapon_rwsem before masking device --

commit 01ef0e90906c766114a8ba5bc17c9a8b4eeba142
Author: Kairui Song <[email protected]>

mm/swap: add priority queue for swap device allocation

This patch introduces a priority queue of same-priority swap device rings for swap entry allocation, protected by swapon_rwsem and swap_queue_update_lock, with per-CPU round-robin iterators and tagged pointers to mask full or disabled devices.

Quoted 4 lines
mm/swapfile.c +0 -0
index 68bed5023299d..2065e566b7a35 100644

[ ... ]

Quoted 71 lines
mm/swapfile.c +31 -28
@@ -1375,41 +1799,44 @@ static bool get_swap_device_info(struct swap_info_struct *si)
 	return true;
 }
 
-/* Rotate the device and switch to a new cluster */
-static void swap_alloc_entry(struct folio *folio)
+static int swap_alloc_entry(struct folio *folio)
 {
-	struct swap_info_struct *si, *next;
+	struct swap_queue_cursor cursor = {};
+	long nr_pages = folio_nr_pages(folio);
+	struct swap_info_struct *si;
+	int nr_iter, ret;
 
-	spin_lock(&swap_avail_lock);
-start_over:
-	plist_for_each_entry_safe(si, next, &swap_avail_head, avail_list) {
-		/* Rotate the device and switch to a new cluster */
-		plist_requeue(&si->avail_list, &swap_avail_head);
-		spin_unlock(&swap_avail_lock);
-		if (get_swap_device_info(si)) {
-			cluster_alloc_swap_entry(si, folio);
-			put_swap_device(si);
-			if (folio_test_swapcache(folio))
-				return;
-			if (folio_test_large(folio))
-				return;
+	percpu_down_read(&swapon_rwsem);
+	migrate_disable();
+	for (nr_iter = 0;; nr_iter++) {
+		si = swap_queue_get_device(nr_pages, nr_iter, &cursor);
+		if (IS_ERR(si)) {
+			ret = PTR_ERR(si);
+			if (ret == -EBUSY)
+				continue;
+			break;
+		}
+		cluster_alloc_swap_entry(si, folio);
+
+		if (folio_test_swapcache(folio)) {
+			ret = 0;
+			break;
 		}
 
-		spin_lock(&swap_avail_lock);
 		/*
-		 * if we got here, it's likely that si was almost full before,
-		 * multiple callers probably all tried to get a page from the
-		 * same si and it filled up before we could get one; or, the si
-		 * filled up between us dropping swap_avail_lock.
-		 * Since we dropped the swap_avail_lock, the swap_avail_list
-		 * may have been modified; so if next is still in the
-		 * swap_avail_head list then try it, otherwise start over if we
-		 * have not gotten any slots.
+		 * For large allocations, try every device at the same priority,
+		 * but ask the caller to split instead of falling back to a lower
+		 * priority ring.
 		 */
-		if (plist_node_empty(&next->avail_list))
-			goto start_over;
+		if (folio_test_large(folio)) {
+			cursor.ring_only = true;
+			continue;
+		}
 	}
-	spin_unlock(&swap_avail_lock);
+
+	migrate_enable();
+	percpu_up_read(&swapon_rwsem);
+	return ret;
 }

[ ... ]

mm/swapfile.c static int swap_device_disable(struct address_space *mapping, · line 3517

Quoted 13 lines
mm/swapfile.c +2 -3
@@ -3082,10 +3509,9 @@ static int swap_device_disable(struct address_space *mapping,
 	}
 	vm_unacct_memory(si->pages);
 
-	spin_lock(&swap_avail_lock);
+	spin_lock(&swap_queue_update_lock);
 	si->flags &= ~SWP_WRITEOK;
-	spin_unlock(&swap_avail_lock);
-
+	spin_unlock(&swap_queue_update_lock);
 	plist_del(&si->list, &swap_active_head);
 	total_swap_pages -= si->pages;
 	atomic_long_sub(si->pages, &nr_swap_pages);

[Severity: High]
Should swap_device_disable() call del_from_avail_list(si, true) before releasing swapon_rwsem?

In swap_device_disable(), SWP_WRITEOK is cleared and percpu_up_write() releases swapon_rwsem before del_from_avail_list(si, true) masks si in swap_queue:

mm/swapfile.c:swap_device_disable() {
	...
	spin_lock(&swap_queue_update_lock);
	si->flags &= ~SWP_WRITEOK;
	spin_unlock(&swap_queue_update_lock);
	plist_del(&si->list, &swap_active_head);
	total_swap_pages -= si->pages;
	atomic_long_sub(si->pages, &nr_swap_pages);

	end = ALIGN(si->max, SWAPFILE_CLUSTER);
unlock:
	percpu_up_write(&swapon_rwsem);
	if (err)
		return err;

	del_from_avail_list(si, true);
	...
}

In the window between percpu_up_write(&swapon_rwsem) and del_from_avail_list(si, true) acquiring swap_queue_update_lock, a concurrent reader in folio_alloc_swap() -> swap_alloc_entry() can acquire swapon_rwsem for reading and obtain the unmasked si from swap_queue_get_device() with SWP_WRITEOK already cleared.

When swap_alloc_entry() then calls cluster_alloc_swap_entry(si, folio) on a device with !(si->flags & SWP_WRITEOK), __swap_cluster_alloc_entries() unconditionally returns false, which can cause two problems:

1. For order-0 folios, cluster_alloc_swap_entry() reaches the higher-order stealing loop:

mm/swapfile.c:cluster_alloc_swap_entry() {
	...
	/* Order 0 stealing from higher order */
	for (int o = 1; o < SWAP_NR_ORDERS; o++) {
		found = alloc_swap_scan_list(si, &si->frag_clusters[o], folio, true);
		if (found)
			goto done;

		found = alloc_swap_scan_list(si, &si->nonfull_clusters[o], folio, true);
		if (found)
			goto done;
	}
}

If si->frag_clusters[o] (or si->nonfull_clusters[o]) has any cluster with 0 < ci->count < SWAPFILE_CLUSTER, isolate_lock_cluster() isolates ci and sets ci->flags = CLUSTER_FLAG_NONE. alloc_swap_scan_cluster() finds a free slot, fails __swap_cluster_alloc_entries() because !(si->flags & SWP_WRITEOK), and calls relocate_cluster(si, ci), which puts ci right back onto si->frag_clusters[o]. Because scan_all is true, alloc_swap_scan_list() will loop forever on the same cluster while holding the cluster lock (si->percpu_cluster->lock or si->global_cluster->lock), migrate_disable(), and swapon_rwsem.

2. For large folios, swap_alloc_entry() treats the failed cluster_alloc_swap_entry(si, folio) call on the disabled device as a fragmentation failure on an active device at that priority and sets cursor.ring_only = true. If no other peer in the same priority ring satisfies the allocation, swap_queue_get_device() returns -E2BIG instead of falling back to lower-priority rings, forcing the caller to split the large folio even when lower-priority swap devices have free space.

Later in the series, commit "mm/swap: drop swap active plist" works around this window in swap_queue_first_usable_bdev() by checking SWP_WRITEOK and SWAP_USAGE_OFFLIST_BIT in addition to swap_device_masked(), but swap_device_disable() and swap_alloc_entry() remain exposed to this window.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=10
author
#16 lore raw

Re: [RFC PATCH v3 10/18] mm/swap: add priority queue for swap device allocation

Quoted 2 lines · On Wed, 28 Oct 2026 16:14:57 +0000 [email protected] wrote:

Should swap_device_disable() call del_from_avail_list(si, true) before releasing swapon_rwsem?

Yes. The next revision will clear SWP_WRITEOK, mark the device off-list and mask its queue entry before releasing the topology writer. This prevents a newly admitted allocation reader from selecting the disabled entry. Cluster-lock draining and try_to_unuse() remain outside the writer.

The preceding available-list patches will also keep their state transition atomic under swap_avail_lock, since they do not yet have this allocation reader protocol.

Closing this window also prevents a disabled candidate from being treated as an enabled but fragmented device and setting cursor.ring_only. For an enabled device with real fragmentation, trying the same-priority peers and then asking the caller to split remains the intended large-folio policy.

I checked this ordering and the large-folio control flow in the intermediate commits. The final revised tree also passes the targeted KUnit scan cases; the disabled fragment scan does not complete on v3. Those tests call the production scan helper directly, so they do not exercise the concurrent syscall selection window or the queue's large-folio retry/split policy.

Thanks for the report.

Lian

author
#17 lore raw

RFC PATCH v3 11/18 mm/swap: remove available list

In reply to Lian Wang

From: Kairui Song <[email protected]>

Pure cleanup, no functional change.

After the priority queue replaced the allocator's device selection, swap_avail_head and swap_avail_lock are no longer used. Remove them.

The throttle path now walks swap_active_head under swapon_rwsem. Check SWP_WRITEOK and the existing off-list state. This preserves the old priority ordering and does not select a full swap device.

---
 include/linux/swap.h |  1 -
 mm/swapfile.c        | 33 ++++++---------------------------
 2 files changed, 6 insertions(+), 28 deletions(-)
include/linux/swap.h +0 -1
index 9357a9065f37..17b8ec654ee2 100644
@@ -287,7 +287,6 @@ struct swap_info_struct {
 	struct work_struct discard_work; /* discard worker */
 	struct work_struct reclaim_work; /* reclaim worker */
 	struct list_head discard_clusters; /* discard clusters list */
-	struct plist_node avail_list;   /* entry in swap_avail_head */
 	const struct swap_ops *ops;
 };
 
mm/swapfile.c +6 -27
index 2065e566b7a3..0a16343a5b03 100644
@@ -96,21 +96,6 @@ static const char Bad_offset[] = "Bad swap offset entry ";
  */
 static PLIST_HEAD(swap_active_head);
 
-/*
- * all available (active, not full) swap_info_structs
- * protected with swap_avail_lock, ordered by priority.
- * This is used by folio_alloc_swap() instead of swap_active_head
- * because swap_active_head includes all swap_info_structs,
- * but folio_alloc_swap() doesn't need to look at full ones.
- * This uses its own lock instead of swapon_rwsem because when a
- * swap_info_struct changes between not-full/full, it needs to
- * add/remove itself to/from this list, but the swap_info_struct->lock
- * is held and the locking order requires swapon_rwsem to be taken
- * before any swap_info_struct->lock.
- */
-static PLIST_HEAD(swap_avail_head);
-static DEFINE_SPINLOCK(swap_avail_lock);
-
 static inline struct swap_info_struct *__swap_iter(int *i, unsigned long flag)
 {
 	lockdep_assert_held(&swapon_rwsem);
@@ -1652,7 +1637,6 @@ static void del_from_avail_list(struct swap_info_struct *si, bool swapoff)
 {
 	unsigned long pages;
 
-	spin_lock(&swap_avail_lock);
 	spin_lock(&swap_queue_update_lock);
 
 	/*
@@ -1672,10 +1656,8 @@ static void del_from_avail_list(struct swap_info_struct *si, bool swapoff)
 	}
 
 	swap_queue_mask(si);
-	plist_del(&si->avail_list, &swap_avail_head);
 skip:
 	spin_unlock(&swap_queue_update_lock);
-	spin_unlock(&swap_avail_lock);
 }
 
 /* SWAP_USAGE_OFFLIST_BIT can only be cleared by this helper. */
@@ -1684,7 +1666,6 @@ static void add_to_avail_list(struct swap_info_struct *si)
 	long val;
 	unsigned long pages;
 
-	spin_lock(&swap_avail_lock);
 	spin_lock(&swap_queue_update_lock);
 
 	/*
@@ -1715,10 +1696,8 @@ static void add_to_avail_list(struct swap_info_struct *si)
 	}
 
 	swap_queue_unmask(si);
-	plist_add(&si->avail_list, &swap_avail_head);
 skip:
 	spin_unlock(&swap_queue_update_lock);
-	spin_unlock(&swap_avail_lock);
 }
 
 /*
@@ -3844,7 +3823,6 @@ static struct swap_info_struct *alloc_swap_info(void)
 	}
 	p->swap_extent_root = RB_ROOT;
 	plist_node_init(&p->list, 0);
-	plist_node_init(&p->avail_list, 0);
 	p->flags = SWP_USED;
 	percpu_up_write(&swapon_rwsem);
 	if (defer) {
@@ -4260,7 +4238,6 @@ SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags)
 	 */
 	si->prio = prio;
 	si->list.prio = -si->prio;
-	si->avail_list.prio = -si->prio;
 
 	/*
 	 * Publish swap_file before making the percpu ref live, then add the device
@@ -4409,14 +4386,16 @@ void __folio_throttle_swaprate(struct folio *folio, gfp_t gfp)
 	if (current->throttle_disk)
 		return;
 
-	spin_lock(&swap_avail_lock);
-	plist_for_each_entry(si, &swap_avail_head, avail_list) {
-		if (si->bdev) {
+	percpu_down_read(&swapon_rwsem);
+	plist_for_each_entry(si, &swap_active_head, list) {
+		if ((si->flags & SWP_WRITEOK) &&
+		    !(atomic_long_read(&si->inuse_pages) &
+		      SWAP_USAGE_OFFLIST_BIT) && si->bdev) {
 			blkcg_schedule_throttle(si->bdev->bd_disk, true);
 			break;
 		}
 	}
-	spin_unlock(&swap_avail_lock);
+	percpu_up_read(&swapon_rwsem);
 }
 #endif
 
author
#18 lore raw

RFC PATCH v3 12/18 mm/swap: bound synchronous discard during allocation

In reply to Lian Wang

From: Kairui Song <[email protected]>

The per-device cluster cache lets the allocator select a device before handling that device's pending discards. Issue discard on the selected device when its free-cluster list is drained instead of waiting for a global fallback scan.

Do not drain a discard list that freeing paths can replenish. Factor out swap_discard_one_cluster(), isolate one cluster under si->lock, retain CLUSTER_FLAG_DISCARD while I/O is in flight, and return it to the free list after completion. The background worker keeps processing the list.

Each call to the device-cluster allocator handles at most one cluster synchronously before retrying allocation. A folio allocation can try multiple devices, so this is a per-device-attempt budget. It bounds the amount of discard work, not device I/O completion latency.

---
 mm/swapfile.c | 149 ++++++++++++++++++++++----------------------------
 1 file changed, 64 insertions(+), 85 deletions(-)
mm/swapfile.c +64 -85
index 0a16343a5b03..5c26682c0bab 100644
@@ -585,6 +585,26 @@ static long swap_usage_in_pages(struct swap_info_struct *si)
 	return atomic_long_read(&si->inuse_pages) & SWAP_USAGE_COUNTER_MASK;
 }
 
+/*
+ * Serialize the allocation on single CPU or globally to avoid
+ * fragmentation and make the workflow easier to follow.
+ */
+static void swap_alloc_lock_device(struct swap_info_struct *si)
+{
+	if (si->flags & SWP_SOLIDSTATE)
+		local_lock(&si->percpu_cluster->lock);
+	else
+		spin_lock(&si->global_cluster->lock);
+}
+
+static void swap_alloc_unlock_device(struct swap_info_struct *si)
+{
+	if (si->flags & SWP_SOLIDSTATE)
+		local_unlock(&si->percpu_cluster->lock);
+	else
+		spin_unlock(&si->global_cluster->lock);
+}
+
 /* Reclaim the swap entry anyway if possible */
 #define TTRS_ANYWAY		0x1
 /*
@@ -982,10 +1002,7 @@ swap_cluster_populate(struct swap_info_struct *si,
 	 * which could give allocator hooks and failure cleanup reserve access.
 	 */
 	spin_unlock(&ci->lock);
-	if (si->flags & SWP_SOLIDSTATE)
-		local_unlock(&si->percpu_cluster->lock);
-	else
-		spin_unlock(&si->global_cluster->lock);
+	swap_alloc_unlock_device(si);
 
 	gfp = (GFP_KERNEL | __GFP_HIGH | __GFP_NOMEMALLOC) &
 		~__GFP_DIRECT_RECLAIM;
@@ -999,10 +1016,7 @@ swap_cluster_populate(struct swap_info_struct *si,
 	 * could happen with ignoring the percpu cluster is fragmentation,
 	 * which is acceptable since this fallback and race is rare.
 	 */
-	if (si->flags & SWP_SOLIDSTATE)
-		local_lock(&si->percpu_cluster->lock);
-	else
-		spin_lock(&si->global_cluster->lock);
+	swap_alloc_lock_device(si);
 	spin_lock(&ci->lock);
 
 	if (ret) {
@@ -1092,44 +1106,40 @@ static struct swap_cluster_info *isolate_lock_cluster(
 }
 
 /*
- * Doing discard actually. After a cluster discard is finished, the cluster
- * will be added to free cluster list. Discard cluster is a bit special as
- * they don't participate in allocation or reclaim, so clusters marked as
- * CLUSTER_FLAG_DISCARD must remain off-list or on discard list.
+ * Discard one cluster. After the discard is finished, the cluster will be
+ * added to the free cluster list. Discard clusters are special because they
+ * don't participate in allocation or reclaim, so CLUSTER_FLAG_DISCARD must
+ * remain set while a cluster is either queued or being discarded.
  */
-static bool swap_do_scheduled_discard(struct swap_info_struct *si)
+static bool swap_discard_one_cluster(struct swap_info_struct *si)
 {
 	struct swap_cluster_info *ci;
-	bool ret = false;
 	unsigned int idx;
 
 	spin_lock(&si->lock);
-	while (!list_empty(&si->discard_clusters)) {
-		ci = list_first_entry(&si->discard_clusters, struct swap_cluster_info, list);
-		/*
-		 * Delete the cluster from list to prepare for discard, but keep
-		 * the CLUSTER_FLAG_DISCARD flag, there could be percpu_cluster
-		 * pointing to it, or ran into by relocate_cluster.
-		 */
-		list_del(&ci->list);
-		idx = cluster_index(si, ci);
+	if (list_empty(&si->discard_clusters)) {
 		spin_unlock(&si->lock);
-		discard_swap_cluster(si, idx * SWAPFILE_CLUSTER,
-				SWAPFILE_CLUSTER);
-
-		spin_lock(&ci->lock);
-		/*
-		 * Discard is done, clear its flags as it's off-list, then
-		 * return the cluster to allocation list.
-		 */
-		ci->flags = CLUSTER_FLAG_NONE;
-		__free_cluster(si, ci);
-		spin_unlock(&ci->lock);
-		ret = true;
-		spin_lock(&si->lock);
+		return false;
 	}
+
+	ci = list_first_entry(&si->discard_clusters,
+			      struct swap_cluster_info, list);
+	/*
+	 * Delete the cluster from the list, but keep CLUSTER_FLAG_DISCARD
+	 * set while the discard is in flight. A percpu cluster may still
+	 * point to it, or relocate_cluster() may encounter it.
+	 */
+	list_del(&ci->list);
+	idx = cluster_index(si, ci);
 	spin_unlock(&si->lock);
-	return ret;
+
+	discard_swap_cluster(si, idx * SWAPFILE_CLUSTER, SWAPFILE_CLUSTER);
+
+	spin_lock(&ci->lock);
+	ci->flags = CLUSTER_FLAG_NONE;
+	__free_cluster(si, ci);
+	spin_unlock(&ci->lock);
+	return true;
 }
 
 static void swap_discard_work(struct work_struct *work)
@@ -1138,7 +1148,8 @@ static void swap_discard_work(struct work_struct *work)
 
 	si = container_of(work, struct swap_info_struct, discard_work);
 
-	swap_do_scheduled_discard(si);
+	while (swap_discard_one_cluster(si))
+		cond_resched();
 }
 
 static void swap_users_ref_free(struct percpu_ref *ref)
@@ -1536,6 +1547,7 @@ static unsigned long cluster_alloc_swap_entry(struct swap_info_struct *si,
 	struct swap_cluster_info *ci;
 	unsigned int order = likely(folio) ? folio_order(folio) : 0;
 	unsigned int offset = SWAP_ENTRY_INVALID, found = SWAP_ENTRY_INVALID;
+	bool discarded = false;
 
 	/*
 	 * Swapfile is not block device so unable
@@ -1544,15 +1556,12 @@ static unsigned long cluster_alloc_swap_entry(struct swap_info_struct *si,
 	if (order && !(si->flags & SWP_BLKDEV))
 		return 0;
 
-	if (si->flags & SWP_SOLIDSTATE) {
-		/* Fast path using per CPU cluster */
-		local_lock(&si->percpu_cluster->lock);
+restart:
+	swap_alloc_lock_device(si);
+	if (si->flags & SWP_SOLIDSTATE)
 		offset = __this_cpu_read(si->percpu_cluster->next[order]);
-	} else {
-		/* Serialize HDD SWAP allocation for each device. */
-		spin_lock(&si->global_cluster->lock);
+	else
 		offset = si->global_cluster->next[order];
-	}
 
 	if (offset != SWAP_ENTRY_INVALID) {
 		ci = swap_cluster_lock(si, offset);
@@ -1576,6 +1585,15 @@ static unsigned long cluster_alloc_swap_entry(struct swap_info_struct *si,
 		found = alloc_swap_scan_list(si, &si->free_clusters, folio, false);
 		if (found)
 			goto done;
+
+		if (!discarded) {
+			swap_alloc_unlock_device(si);
+			if (swap_discard_one_cluster(si)) {
+				discarded = true;
+				goto restart;
+			}
+			swap_alloc_lock_device(si);
+		}
 	}
 
 	if (order < PMD_ORDER) {
@@ -1624,10 +1642,7 @@ static unsigned long cluster_alloc_swap_entry(struct swap_info_struct *si,
 			goto done;
 	}
 done:
-	if (si->flags & SWP_SOLIDSTATE)
-		local_unlock(&si->percpu_cluster->lock);
-	else
-		spin_unlock(&si->global_cluster->lock);
+	swap_alloc_unlock_device(si);
 
 	return found;
 }
@@ -1818,36 +1833,6 @@ static int swap_alloc_entry(struct folio *folio)
 	return ret;
 }
 
-/*
- * Discard pending clusters in a synchronized way when under high pressure.
- * Return: true if any cluster is discarded.
- */
-static bool swap_sync_discard(void)
-{
-	bool ret = false;
-	struct swap_info_struct *si, *next;
-
-	percpu_down_read(&swapon_rwsem);
-start_over:
-	plist_for_each_entry_safe(si, next, &swap_active_head, list) {
-		percpu_up_read(&swapon_rwsem);
-		if (get_swap_device_info(si)) {
-			if (si->flags & SWP_PAGE_DISCARD)
-				ret = swap_do_scheduled_discard(si);
-			put_swap_device(si);
-		}
-		if (ret)
-			return true;
-
-		percpu_down_read(&swapon_rwsem);
-		if (plist_node_empty(&next->list))
-			goto start_over;
-	}
-	percpu_up_read(&swapon_rwsem);
-
-	return false;
-}
-
 static int swap_extend_table_alloc(struct swap_info_struct *si,
 				   struct swap_cluster_info *ci,
 				   unsigned int ci_off, gfp_t gfp)
@@ -2149,14 +2134,8 @@ int folio_alloc_swap(struct folio *folio)
 		}
 	}
 
-again:
 	swap_alloc_entry(folio);
 
-	if (!order && unlikely(!folio_test_swapcache(folio))) {
-		if (swap_sync_discard())
-			goto again;
-	}
-
 	/* Need to call this even if allocation failed, for MEMCG_SWAP_FAIL. */
 	if (unlikely(mem_cgroup_try_charge_swap(folio))) {
 		swap_cache_del_folio(folio);
#19 lore raw

Re: [RFC PATCH v3 12/18] mm/swap: bound synchronous discard during allocation

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider: - [High] mm/swap: synchronous discard under swapon_rwsem in cluster_alloc_swap_entry() - [Medium] mm…

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [High] mm/swap: synchronous discard under swapon_rwsem in cluster_alloc_swap_entry()
- [Medium] mm: premature swap allocation failure in cluster_alloc_swap_entry() --

commit 148b9ddc65b48866a2eb76b6131e127b4a148727
Author: Kairui Song <[email protected]>

mm/swap: bound synchronous discard during allocation

This commit replaces the global synchronous discard fallback with a per-device single-cluster discard in cluster_alloc_swap_entry() when a swap device's free cluster list is empty.

Quoted 4 lines
mm/swapfile.c +0 -0
index 0a16343a5b03c..5c26682c0bab2 100644

[ ... ]

Quoted 4 lines
mm/swapfile.c +0 -0
@@ -1576,6 +1585,15 @@ static unsigned long cluster_alloc_swap_entry(struct swap_info_struct *si,
 		found = alloc_swap_scan_list(si, &si->free_clusters, folio, false);
 		if (found)
 			goto done;

mm/swapfile.c static unsigned long cluster_alloc_swap_entry(struct swap_info_struct *si, · line 1597

Quoted 10 lines
mm/swapfile.c +9 -0
+
+		if (!discarded) {
+			swap_alloc_unlock_device(si);
+			if (swap_discard_one_cluster(si)) {
+				discarded = true;
+				goto restart;
+			}
+			swap_alloc_lock_device(si);
+		}
 	}

[Severity: High]
Is it safe to call swap_discard_one_cluster() here while holding swapon_rwsem for reading?

Previously, swap_sync_discard() dropped swapon_rwsem after pinning the swap device before issuing synchronous discard I/O:

mm/swapfile.c:swap_sync_discard() {
	...
	plist_for_each_entry_safe(si, next, &swap_active_head, list) {
		percpu_up_read(&swapon_rwsem);
		if (get_swap_device_info(si)) {
			if (si->flags & SWP_PAGE_DISCARD)
				ret = swap_do_scheduled_discard(si);
			put_swap_device(si);
		}
	...
}

With this change, swap_alloc_entry() holds percpu_down_read(&swapon_rwsem) and migrate_disable() across cluster_alloc_swap_entry(), which only drops the device cluster lock via swap_alloc_unlock_device(si) before calling swap_discard_one_cluster() -> discard_swap_cluster() -> blkdev_issue_discard(..., GFP_NOIO) -> submit_bio_wait().

If a writer such as swapon(), swapoff(), or pin_hibernation_swap_type() calls percpu_down_write(&swapon_rwsem) while submit_bio_wait() is waiting for the REQ_OP_DISCARD bio to complete, new readers calling percpu_down_read(&swapon_rwsem) in swap_alloc_entry() will block behind the writer. Could this stall system-wide swap allocation, or deadlock if a block backend worker (such as ublk or nbd) needs memory reclaim and swap allocation to complete the in-flight discard request?

Also, on loop-backed swap devices, processing REQ_OP_DISCARD in the loop worker calls lo_fallocate() -> file->f_op->fallocate(), which acquires inode_lock() on the backing file. Since swapon() holds inode_lock() while calling swap_queue_lock_for_add() -> percpu_down_write(&swapon_rwsem), doesn't waiting for discard completion under percpu_down_read(&swapon_rwsem) invert the lock ordering between inode_lock() and swapon_rwsem?

[ ... ]

mm/swapfile.c int folio_alloc_swap(struct folio *folio) · line 2158

Quoted 11 lines
mm/swapfile.c +0 -5
@@ -2149,14 +2134,8 @@ int folio_alloc_swap(struct folio *folio)
 		}
 	}
 
-again:
 	swap_alloc_entry(folio);
 
-	if (!order && unlikely(!folio_test_swapcache(folio))) {
-		if (swap_sync_discard())
-			goto again;
-	}

[Severity: Medium]
Can removing the post-reclaim discard retry cause folio_alloc_swap() to fail prematurely when si->discard_clusters is populated during cluster reclaim?

In mm/swapfile.c, when cluster_alloc_swap_entry() is entered on a SWP_PAGE_DISCARD device with both si->free_clusters and si->discard_clusters initially empty, alloc_swap_scan_list() on si->free_clusters and swap_discard_one_cluster() both return false, leaving discarded set to false.

cluster_alloc_swap_entry() then scans si->nonfull_clusters, calls swap_reclaim_full_clusters(), and scans si->frag_clusters. During these steps, cluster_reclaim_range() and swap_reclaim_full_clusters() call __try_to_reclaim_swap() -> swap_cache_del_folio() -> __swap_cluster_free_entries(). When reclaiming unmapped swapcache folios drops a cluster's count to zero, free_cluster() schedules it for discard:

mm/swapfile.c:free_cluster() {
	if ((si->flags & (SWP_WRITEOK | SWP_PAGE_DISCARD)) ==
	    (SWP_WRITEOK | SWP_PAGE_DISCARD)) {
		swap_cluster_schedule_discard(si, ci);
		return;
	}
	...
}

Because swap_cluster_schedule_discard() sets ci->flags to CLUSTER_FLAG_DISCARD and moves the cluster to si->discard_clusters, cluster_reclaim_range() sees !cluster_is_usable(ci, order) and cannot allocate from the cluster it just emptied:

mm/swapfile.c:cluster_reclaim_range() {
	...
	spin_lock(&ci->lock);
	/* Recheck after lock drop, another user might have used it. */
	if (!cluster_is_usable(ci, order)) {
		*usable = false;
		return false;
	}
	...
}

Previously, folio_alloc_swap() called swap_sync_discard() after swap_alloc_entry() failed and retried the allocation, which discarded the clusters newly queued to si->discard_clusters during reclaim. Now, cluster_alloc_swap_entry() never checks si->discard_clusters again after the nonfull, full, and fragment cluster scans, returning SWAP_ENTRY_INVALID even though discarded is still false and reclaim just queued discardable clusters.

In addition, when swap_discard_one_cluster() does discard a cluster, it places the cluster onto the shared si->free_clusters list via __free_cluster() and releases ci->lock while the caller does not hold swap_alloc_lock_device(si). If another CPU isolates that cluster from si->free_clusters before the discarding task re-acquires swap_alloc_lock_device(si) at the restart label, won't the discarding task see an empty si->free_clusters list and fail the allocation even when more clusters remain on si->discard_clusters?

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=12
author
#20 lore raw

Re: [RFC PATCH v3 12/18] mm/swap: bound synchronous discard during allocation

Quoted 2 lines · On Wed, 28 Oct 2026 16:14:58 +0000 [email protected] wrote:

Is it safe to call swap_discard_one_cluster() here while holding swapon_rwsem for reading?

The reader should not span synchronous discard I/O. A writer that closes reader admission can turn one device's slow completion into a wait for new allocators on other devices as well.

I have moved the order-0 discard fallback outside the allocation reader. The fallback pins a live device under a short reader, drops the reader before issuing one cluster's discard, then releases the device reference. Each retry starts a fresh task-local allocation walk, without reusing a ring pointer or cursor saved before releasing the reader for I/O.

The per-request bitmap permits at most one cluster to be processed synchronously per swap type. Empty discard candidates do not consume that budget. Large folios retain the pre-patch-12 retry/split policy; they do not enter this order-0 fallback.

The generic reclaim-to-swap deadlock needs qualification: the loop worker's PF_MEMALLOC_NOIO prevents its ordinary direct reclaim from allocating anonymous swap backing. There is a separate conditional inode/topology lock-order concern when loop discard and a concurrent swapon use the same backing inode. I am not claiming a runtime reproduction of that lock cycle.

Quoted 3 lines

[Severity: Medium]
fail prematurely when si->discard_clusters is populated during cluster reclaim?

Yes. With an invalid cached hint, a nonfull-cluster scan can reclaim its last cache-only folio after the early discard check. The newly empty cluster then enters DISCARD and cannot satisfy that allocation. The reader-external fallback catches this case and retries allocation after discard, while retaining the bounded work per request.

A peer can still take the freed cluster before the retry under a one-cluster budget. The bound does not guarantee that the caller consumes the freed cluster, and we should not turn it into an unbounded drain to guarantee allocation success. This is distinct from missing a newly queued cluster without using any budget.

For the paired KUnit comparison, I used identical fixture sources on v3 and the revised tree in isolated x86-64 guests, with configurations differing only in LOCALVERSION and the expected-outcome selector. The tests use the production allocator, swap table, accounting, locks and discard BIO path on a private test SI; no production return value is overridden.

For late nonfull reclaim, v3 returns -ENOMEM without caching the new folio and leaves one empty cluster pending discard. The revised tree returns 0, caches the new folio and leaves no pending cluster after one 4096-sector discard (512 pages at 4 KiB per page).

In the already-queued-discard control, both trees issue one 4096-sector BIO. At submission, lockdep records all 4096 sectors under the reader on v3, and none under either the reader or writer on the revised tree. The three-pending-cluster control processes one cluster and leaves two pending on both trees. Lockdep remains enabled, with no unknown lock state.

All ten cases pass on the revised tree: four scan cases and six discard cases. These are helper-level tests, not a swapon/swapoff syscall race, page-table swapout, or a complete backend deadlock reproduction. They do not provide large-folio performance results or prove the budget against an indefinitely replenishing producer.

Separately, syscall/data smoke tests on the revised tree passed with a single device, equal-priority devices, different-priority devices and MADV_HUGEPAGE advice. Each phase verified 900 MiB of anonymous data twice, with nonzero pswpin/pswpout and no OOM kills. Both equal-priority devices were used; with different priorities the lower device remained unused. Swapon and swapoff succeeded. This is additional correctness coverage, not an old/new performance comparison, a concurrent swapoff-race reproduction, or proof of specific mTHP swap orders.

Thanks for the reports.

Lian

author
#21 lore raw

RFC PATCH v3 13/18 mm/swap: drop swap active plist

In reply to Lian Wang

From: Kairui Song <[email protected]>

Remove swap_active_head and the associated plist_node. Device identity lookups can use the writable swap_info iterator without ordering.

Keep priority ordering for the block-cgroup swap throttle target by walking the priority queue under the topology reader. Choose the first unmasked, writable, non-offlist device with a backing block device. Check WRITEOK and OFFLIST as well as the pointer tag because disable and full transitions can temporarily leave an unmasked entry.

This read-only lookup does not consume per-CPU allocation quotas or change retry cursors. It avoids making the throttle disk depend on swapon's swap-type assignment order when devices have different priorities.

---
 include/linux/swap.h |  1 -
 mm/swapfile.c        | 59 ++++++++++++++++++++++++--------------------
 2 files changed, 32 insertions(+), 28 deletions(-)
include/linux/swap.h +0 -1
index 17b8ec654ee2..bfca46e1fbc6 100644
@@ -261,7 +261,6 @@ struct swap_info_struct {
 	struct percpu_ref users;	/* indicate and keep swap device valid. */
 	unsigned long	flags;		/* SWP_USED etc: see above */
 	signed short	prio;		/* swap priority of this type */
-	struct plist_node list;		/* entry in swap_active_head */
 	signed char	type;		/* strange name for an index */
 	unsigned int	max;		/* size of this swap device */
 	struct swap_cluster_info *cluster_info; /* array, one entry per cluster */
mm/swapfile.c +32 -27

114 lines not on this page · raw message

author
#22 lore raw

RFC PATCH v3 14/18 lib/plist.c: remove requeue function

In reply to Lian Wang

From: Kairui Song <[email protected]>

Now the last user of plist requeue is gone, this function can be removed.

---
 include/linux/plist.h |  2 --
 lib/plist.c           | 64 -------------------------------------------
 2 files changed, 66 deletions(-)
include/linux/plist.h +0 -2
index 16cf4355b5c1..73d359c1a053 100644
@@ -132,8 +132,6 @@ static inline void plist_node_init(struct plist_node *node, int prio)
 extern void plist_add(struct plist_node *node, struct plist_head *head);
 extern void plist_del(struct plist_node *node, struct plist_head *head);
 
-extern void plist_requeue(struct plist_node *node, struct plist_head *head);
-
 /**
  * plist_for_each - iterate over the plist
  * @pos:	the type * to use as a loop counter
lib/plist.c +0 -64

82 lines not on this page · raw message

author
#23 lore raw

RFC PATCH v3 15/18 mm/swap: reuse per-CPU queue readers across priority changes

In reply to Lian Wang

Inserting a new priority allocates a replacement per-CPU reader array even when earlier ring removal left enough storage. An unnecessary allocation can add swapon work or fail despite sufficient capacity.

Track the installed reader capacity and retain its high-water mark until the queue becomes empty. Reuse existing storage when the new ring fits, initializing every valid ring index for the new topology. Prepare larger storage outside the writer only when growth is required and recheck capacity under the writer before publishing.

Preserve the existing prepare/install ownership and failure rollback protocol. Device removal continues to avoid per-CPU shrinking allocation under the writer.

---
 mm/swapfile.c | 28 ++++++++++++++++++----------
 1 file changed, 18 insertions(+), 10 deletions(-)
mm/swapfile.c +18 -10
index 7b3520ce1e41..5b6492b1bdc6 100644
@@ -201,6 +201,8 @@ struct swap_queue_cursor {
 };
 
 static __percpu struct swap_queue_reader *swap_queue_readers;
+/* Keep the high-water capacity until the last priority ring is removed. */
+static unsigned int swap_queue_readers_capacity;
 
 static inline bool swap_device_masked(struct swap_info_struct *si)
 {
@@ -240,7 +242,7 @@ static bool swap_queue_needs_readers(int prio)
 			return false;
 	}
 
-	return true;
+	return swap_queue_len >= swap_queue_readers_capacity;
 }
 
 /*
@@ -288,13 +290,15 @@ static void swap_queue_reset_readers(void)
 	}
 }
 
-static void swap_queue_install_readers(struct swap_queue_reader __percpu *readers)
+static void swap_queue_install_readers(struct swap_queue_reader __percpu *readers,
+				       unsigned int capacity)
 {
 	int cpu;
 
 	lockdep_assert_held_write(&swapon_rwsem);
 	free_percpu(swap_queue_readers);
 	swap_queue_readers = readers;
+	swap_queue_readers_capacity = capacity;
 	if (!readers)
 		return;
 
@@ -431,7 +435,8 @@ static int swap_queue_add(struct swap_info_struct *si,
 	}
 
 	/* No ring at this priority: insert a new one at pos. */
-	if (!*readers || nr_readers <= swap_queue_len)
+	if (swap_queue_len >= swap_queue_readers_capacity &&
+	    (!*readers || nr_readers <= swap_queue_len))
 		goto failed;
 	new_queue = kmalloc_array(swap_queue_len + 1, sizeof(*swap_queue), gfp);
 	if (!new_queue)
@@ -459,8 +464,12 @@ static int swap_queue_add(struct swap_info_struct *si,
 	memalloc_noreclaim_restore(noreclaim_flag);
 	kfree(old_queue);
 
-	swap_queue_install_readers(*readers);
-	*readers = NULL;
+	if (swap_queue_len > swap_queue_readers_capacity) {
+		swap_queue_install_readers(*readers, nr_readers);
+		*readers = NULL;
+	} else {
+		swap_queue_reset_readers();
+	}
 	return 0;
 
 add_to_ring:
@@ -503,10 +512,9 @@ static void swap_queue_del(struct swap_info_struct *si)
 	}
 
 	/*
-	 * Shrinking the small topology arrays is optional. Keep reader storage
-	 * until a new priority is inserted or the queue is empty: per-CPU
-	 * allocation must not run under this writer's noreclaim scope, which
-	 * can grant backing pages reserve access.
+	 * Shrinking the small topology arrays is optional. Keep reader capacity
+	 * until the queue is empty: per-CPU allocation must not run under this
+	 * writer's noreclaim scope, which can grant backing pages reserve access.
 	 */
 	gfp = GFP_NOIO | __GFP_HIGH | __GFP_NOMEMALLOC;
 	noreclaim_flag = memalloc_noreclaim_save();
@@ -549,7 +557,7 @@ static void swap_queue_del(struct swap_info_struct *si)
 			swap_queue = NULL;
 		}
 		if (!swap_queue_len)
-			swap_queue_install_readers(NULL);
+			swap_queue_install_readers(NULL, 0);
 	}
 	spin_unlock(&swap_queue_update_lock);
 
author
#24 lore raw

RFC PATCH v3 16/18 mm/swap: advance the global queue cursor directly between priority rings

In reply to Lian Wang

The allocation walk previously passed a total retry index to the selector. Each retry translated it back to a priority ring by scanning from the beginning of swap_queue. Keep the ring index and the number of attempted peers in the task-local cursor instead.

Snapshot the starting device only on the first visit to each ring and advance each peer exactly once, including masked entries. Preserve the large-folio same-ring boundary and the stale reader-offset correction. The selector's complete walk is linear in the devices and priority groups; device cluster scanning and availability updates have their own costs.

Keep the topology read lock and CPU migration protection across the walk. The global selector uses only existing priorities and adds no per-memcg device-selection state or ABI. Existing swap charging and limits remain in folio_alloc_swap(), whose failure path classifies allocation errors.

The walk no longer returns an ignored internal error to that caller.

---
 mm/swapfile.c | 49 +++++++++++++++++++++++--------------------------
 1 file changed, 23 insertions(+), 26 deletions(-)
mm/swapfile.c +23 -26
index 5b6492b1bdc6..bbd114f043ab 100644
@@ -194,10 +194,10 @@ struct swap_queue_reader {
 };
 
 struct swap_queue_cursor {
-	bool valid;
 	bool ring_only;
 	unsigned int ring_idx;
 	unsigned int offset;
+	unsigned int nr_iter;
 };
 
 static __percpu struct swap_queue_reader *swap_queue_readers;
@@ -307,28 +307,32 @@ static void swap_queue_install_readers(struct swap_queue_reader __percpu *reader
 	swap_queue_reset_readers();
 }
 
-static struct swap_info_struct *swap_queue_get_device(long nr_alloc, int nr_iter,
-						      struct swap_queue_cursor *cursor)
+static struct swap_info_struct *
+swap_queue_get_device(long nr_alloc, struct swap_queue_cursor *cursor)
 {
 	bool rotate = false;
 	struct swap_info_struct *si;
 	struct swap_ring_iterator *ri;
 	struct swap_prio_ring *ring;
-	unsigned int dev_idx, queue_idx;
+	unsigned int dev_idx, queue_idx, nr_iter;
 
-	if (!swap_queue_len)
+	queue_idx = cursor->ring_idx;
+	if (queue_idx >= swap_queue_len)
 		return ERR_PTR(-ENOENT);
 
-	queue_idx = 0;
-	while (nr_iter >= swap_queue[queue_idx]->size) {
-		nr_iter -= swap_queue[queue_idx]->size;
-		if (++queue_idx >= swap_queue_len)
-			return ERR_PTR(cursor->ring_only ? -E2BIG : -ENOENT);
+	ring = swap_queue[queue_idx];
+	if (cursor->nr_iter >= ring->size) {
+		if (cursor->ring_only)
+			return ERR_PTR(-E2BIG);
+		queue_idx = ++cursor->ring_idx;
+		if (queue_idx >= swap_queue_len)
+			return ERR_PTR(-ENOENT);
+		cursor->nr_iter = 0;
+		ring = swap_queue[queue_idx];
 	}
-	if (cursor->ring_only && cursor->ring_idx != queue_idx)
-		return ERR_PTR(-E2BIG);
 
-	ring = swap_queue[queue_idx];
+	/* Consume masked peers too, so every walk remains bounded. */
+	nr_iter = cursor->nr_iter++;
 	local_lock(&swap_queue_readers->lock);
 	ri = this_cpu_ptr(&swap_queue_readers->ri[queue_idx]);
 	/*
@@ -336,9 +340,7 @@ static struct swap_info_struct *swap_queue_get_device(long nr_alloc, int nr_iter
 	 * iterator can move between retries, but the cursor must visit every
 	 * device in the ring exactly once before falling through.
 	 */
-	if (!cursor->valid || cursor->ring_idx != queue_idx) {
-		cursor->valid = true;
-		cursor->ring_idx = queue_idx;
+	if (!nr_iter) {
 		if (ri->rr_counter < nr_alloc)
 			rotate = true;
 		else if (ri->offset >= ring->size)
@@ -1794,29 +1796,25 @@ static bool get_swap_device_info(struct swap_info_struct *si)
 	return true;
 }
 
-static int swap_alloc_entry(struct folio *folio)
+static void swap_alloc_entry(struct folio *folio)
 {
 	struct swap_queue_cursor cursor = {};
 	long nr_pages = folio_nr_pages(folio);
 	struct swap_info_struct *si;
-	int nr_iter, ret;
 
 	percpu_down_read(&swapon_rwsem);
 	migrate_disable();
-	for (nr_iter = 0;; nr_iter++) {
-		si = swap_queue_get_device(nr_pages, nr_iter, &cursor);
+	for (;;) {
+		si = swap_queue_get_device(nr_pages, &cursor);
 		if (IS_ERR(si)) {
-			ret = PTR_ERR(si);
-			if (ret == -EBUSY)
+			if (PTR_ERR(si) == -EBUSY)
 				continue;
 			break;
 		}
 		cluster_alloc_swap_entry(si, folio);
 
-		if (folio_test_swapcache(folio)) {
-			ret = 0;
+		if (folio_test_swapcache(folio))
 			break;
-		}
 
 		/*
 		 * For large allocations, try every device at the same priority,
@@ -1831,7 +1829,6 @@ static int swap_alloc_entry(struct folio *folio)
 
 	migrate_enable();
 	percpu_up_read(&swapon_rwsem);
-	return ret;
 }
 
 static int swap_extend_table_alloc(struct swap_info_struct *si,
author
#25 lore raw

RFC PATCH v3 17/18 mm/swap: skip reader rotation for single-device priority rings

In reply to Lian Wang

A priority ring with one device cannot benefit from rotating its starting offset or maintaining a quota. Read its only entry directly while the caller continues to hold the topology read lock and migration protection.

Consume the cursor attempt and check the full/disabled pointer tag through the common path. A masked singleton therefore terminates or falls through with the same bounded behavior as a multi-device ring. Keep the existing large-folio same-priority failure boundary.

This removes reader-local state updates for the common single-device case without bypassing device lifetime or availability checks.

---
 mm/swapfile.c | 6 ++++++
 1 file changed, 6 insertions(+)
mm/swapfile.c +6 -0
index bbd114f043ab..9e90f815f99c 100644
@@ -333,6 +333,11 @@ swap_queue_get_device(long nr_alloc, struct swap_queue_cursor *cursor)
 
 	/* Consume masked peers too, so every walk remains bounded. */
 	nr_iter = cursor->nr_iter++;
+	if (ring->size == 1) {
+		si = READ_ONCE(ring->dev[0]);
+		goto check_device;
+	}
+
 	local_lock(&swap_queue_readers->lock);
 	ri = this_cpu_ptr(&swap_queue_readers->ri[queue_idx]);
 	/*
@@ -362,6 +367,7 @@ swap_queue_get_device(long nr_alloc, struct swap_queue_cursor *cursor)
 	si = READ_ONCE(ring->dev[dev_idx]);
 	local_unlock(&swap_queue_readers->lock);
 
+check_device:
 	if (swap_device_masked(si))
 		return ERR_PTR(-EBUSY);
 
author
#26 lore raw

RFC PATCH v3 18/18 mm/swap: redistribute queue readers after membership changes

In reply to Lian Wang

Adding a device to an existing priority ring reuses the per-CPU reader storage. A ring that previously contained one device can therefore keep every CPU at the old common starting offset after peers are added. Deleting devices or rings can likewise leave offsets and quotas that describe the previous membership.

Reset the valid per-CPU ring offsets and quotas after peer insertion and every nonempty deletion. Run the initialization helper under the topology writer, which excludes allocation walks, without allocating new reader storage or adding selector hot-path work.

Use cpu-id modulo placement for a fresh best-effort rotation across all valid rings. This does not move existing slots or guarantee strict balance for every active CPU subset. Mask and availability transitions do not trigger the topology reset.

---
 mm/swapfile.c | 3 +++
 1 file changed, 3 insertions(+)
mm/swapfile.c +3 -0
index 9e90f815f99c..d659afb574b3 100644
@@ -496,6 +496,7 @@ add_to_ring:
 	spin_unlock(&swap_queue_update_lock);
 	memalloc_noreclaim_restore(noreclaim_flag);
 	kfree(old_ring);
+	swap_queue_reset_readers();
 	return 0;
 
 failed:
@@ -568,6 +569,8 @@ static void swap_queue_del(struct swap_info_struct *si)
 			swap_queue_install_readers(NULL, 0);
 	}
 	spin_unlock(&swap_queue_update_lock);
+	if (swap_queue_len)
+		swap_queue_reset_readers();
 
 	kfree(old_ring);
 	kfree(old_queue);