Skip to contentSkip to conversation

mm/swap: introduce priority queue to remove global cluster cache and plist

RFC PATCH In review Kairui Song 4 participants 18 messages Jul 13 – Jul 15

Subsystems (3)
lore
Share & tools
lore

Kairui Song

author maintainer
#0 lore raw

cover mm/swap: introduce priority queue to remove global cluster cache and plist

Hi all,

This series introduces a new way to fairly distribute the IO to multiple swap devices, completely gets rid of the mutable plist and global lock, hence removes the global cluster cache and allows allocation and rotation to happen more freely. This is especially helpful for swap tiering or layered swap. Sending as RFC for review first as tiering should land first, this can rebase and join later as an optimization, and there are a few items that can still be improved based on this series. I specifically kept the tiering and layered ideas in mind to make the new design friendly for new ideas like these and tested a few different appraoches locally.

As a dependency, this also cleanup and rework the locking pattern for swap device management. All swap devices and the newly introduced percpu reader priority queue are now guarded by a percpu rwsem for better scalability and a more future-proof design.

Previously, each swap device had its own percpu cluster cache. To improve performance, commit 1b7e90020eb7 ("mm, swap: use percpu cluster as allocation fast path") moved the cluster cache to a global scope and let every device share the same global cache, so we don't need to always touch the plist. Fast swap allocation will just use each CPU's cluster cache. Device rotation is still always handled by the plist when one cluster is drained in the cluster cache, which bundled cluster allocation with rotation. It works, but this whole design caused several long-standing issues:

1) The global percpu cluster cache sitting above device selection fundamentally conflicts with concepts like swap tiering and layered swap allocation. We need to bypass certain swap devices, so we have to bypass the cache as well because the cache doesn't have stable ownership. That would result in very poor performance. And there are other problems like priority inversion and device rotation. These are described in more detail in the swap tier series [1].

2) The plist itself requires plist_requeue() to rotate devices for round-robin, which needs swap_avail_lock held. Allocators must
cycle: lock -> pick -> unlock -> allocate -> lock -> rotate. All CPUs contend on the same global list head, and the design is neither flexible nor robust.

3) Rotation is considered the slow path. The fast / slow path design to avoid touching the plist has caused workarounds like commit 9fb749cd15078 ("mm, swap: do not perform synchronous discard during allocation"), we can't stably grab an actual device before touching the plist, so the design is suboptimal and has many constraints.

4) We don't have a clear rule of how rotation works. The round-robin rotation is strongly bundled to the cluster allocation algorithm, and cluster drain is unpredictable. We will have a roughly 2M rotation block size but that varies from time to time.

This series gets rid of all these issues step by step:

Patches 1-4: Cleanups and infrastructure
Patches 5-7: Lock and accounting consolidation
Patch 8: Revert the global percpu cluster by YoungJun Patch 9: This is the core change: percpu reader swap queue Patch 10: Remove the old swap_avail_head plist (no longer needed) Patch 11: Drop the global sync discard workaround Patches 12-13: Drop plist entirely

The design is described in Patch 9. In general, devices of the same priority are arranged in the same static ring. The ring itself is mostly read only, but each CPU's reader of the ring rotates, with a counter built into each reader. The counter is adjustable and controls the rotation pace. The priority ring design should fit well with the current tiering design.

The performance is looking on par with the old global cluster cache design, while it enables more potential features like weighted interleave or adjustable shard value for device rotation of same priority (by simply adjusting the counter value, SWAP_ROUND_ROBIN_QUOTA).

Testing with 8 ZRAM devices, running a typical build linux kernel test, with make -j96 in a 2G memcg (Using 8 ZRAM, might not be a very common setup but there are such configs to avoid lock contention inside each ZRAM device, and this also simulates having multiple high speed block devices as SWAP), measure average system time for 12 testruns:

Before:          6633.09s
After patch  8: 12294.74s (Move the global cluster cache back to device scope)
After patch 13:  6551.27s (Introduce the percpu queue to avoid device contention)

And doing the workload in a 3G memcg, avg sys time of 12 testrun:

Before:          3312.63s
After patch  8:  6450.35s
After patch 13:  3311.38s

Using 12 brd, testing with: usemem --init-time -O -y -x -n 32 1536M

Before:         4347.58 MB/s
After patch  8: 2773.39 MB/s
After patch 13: 4345.04 MB/s

Patch 8 removed the global percpu cluster cache, so we are touching the plist before accessing the cache, and it's very clear that the performance is really bad. Then with the new percpu reader queue design, we got rid of the plist, performance is on-par or better, while there is no more slow / fast path design, and cache stays inside each device. We can always select a device based on priority before accessing the cache.

And the shard size is adjustable through SWAP_ROUND_ROBIN_QUOTA which is hardcoded for now to 2M, changing that effect the performance in different ways too.

And the distributions are fair among devices during the test (similiar with brd):

NAME TYPE SIZE USED PRIO

/dev/zram0 partition   8G 692.4M   -1
/dev/zram1 partition   8G 693.2M   -1
/dev/zram2 partition   8G 694.9M   -1
/dev/zram3 partition   8G 694.1M   -1
/dev/zram4 partition   8G 695.5M   -1
/dev/zram5 partition   8G 695.1M   -1
/dev/zram6 partition   8G 694.3M   -1
/dev/zram7 partition   8G 693.6M   -1

This work was discussed several times on the mailing list [2] and rebased and reworked to leave room for future tiering, optimization, e.g. by assigning separate iterators per priority ring.

One patch from YoungJun is included as well [3].

---
Kairui Song (12):
mm/swap: remove unused parameter for reading swap header mm/swap: slightly cleanup the code for hibernation error handling mm/swap: cleanup and document swap device availability flag usage mm/swap: introduce swap device iteration helper mm/swap: change the swapon lock into a percpu rwsem mm/swap: remove swapon mutex and update proc reader mm/swap: consolidate swap inuse accounting helpers mm/swap: add priority queue for swap device allocation mm/swap: remove available list
mm/swap: perform sync discard on single device more proactively mm/swap: drop swap active plist
lib/plist.c: remove requeue function

Youngjun Park (1):
mm/swap: change back to use each swap device's percpu cluster

 include/linux/plist.h |    2 -
 include/linux/swap.h  |   42 +-
 lib/plist.c           |   64 ---
 mm/swap.h             |   12 +-
 mm/swapfile.c         | 1165 +++++++++++++++++++++++++++++--------------------
 5 files changed, 718 insertions(+), 567 deletions(-)

---
base-commit: bdc38bfc1262e3d1432afadd2aa2ffd83d139dbb
change-id: 20260529-swap-pcp-priq-68d3d925578c

Best regards,

Signature · 1 line
Kairui Song <[email protected]>
author maintainer
#1 lore raw

RFC PATCH 1/13 mm/swap: remove unused parameter for reading swap header

No feature change, just a minor cleanup.

---
 mm/swapfile.c | 7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)
mm/swapfile.c +3 -4
index 775c385490a5..27b4d11125d6 100644
@@ -3453,9 +3453,8 @@ __weak unsigned long arch_max_swapfile_size(void)
 	return generic_max_swapfile_size();
 }
 
-static unsigned long read_swap_header(struct swap_info_struct *si,
-					union swap_header *swap_header,
-					struct inode *inode)
+static unsigned long read_swap_header(union swap_header *swap_header,
+				      struct inode *inode)
 {
 	int i;
 	unsigned long maxpages;
@@ -3679,7 +3678,7 @@ SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags)
 	}
 	swap_header = kmap_local_folio(folio, 0);
 
-	maxpages = read_swap_header(si, swap_header, inode);
+	maxpages = read_swap_header(swap_header, inode);
 	if (unlikely(!maxpages)) {
 		error = -EINVAL;
 		goto bad_swap_unlock_inode;
Signature · 1 line
2.55.0
author maintainer
#2 lore raw

RFC PATCH 2/13 mm/swap: slightly cleanup the code for hibernation error handling

In reply to Kairui Song

Restructure the error paths in pin_hibernation_swap_type() using a goto out pattern to avoid repeated spin_unlock() calls and simplify the return flow. Also simplify unpin_hibernation_swap_type() by making the si NULL check inline, and clean up find_first_swap() to avoid an early return inside the loop.

---
 mm/swapfile.c | 39 +++++++++++++++++----------------------
 1 file changed, 17 insertions(+), 22 deletions(-)
mm/swapfile.c +17 -22
index 27b4d11125d6..30ee75bdec40 100644
@@ -2247,21 +2247,18 @@ static int __find_hibernation_swap_type(dev_t device, sector_t offset)
  */
 int pin_hibernation_swap_type(dev_t device, sector_t offset)
 {
-	int type;
+	int ret;
 	struct swap_info_struct *si;
 
 	spin_lock(&swap_lock);
+	ret = __find_hibernation_swap_type(device, offset);
+	if (ret < 0)
+		goto out;
 
-	type = __find_hibernation_swap_type(device, offset);
-	if (type < 0) {
-		spin_unlock(&swap_lock);
-		return type;
-	}
-
-	si = swap_type_to_info(type);
+	si = swap_type_to_info(ret);
 	if (WARN_ON_ONCE(!si)) {
-		spin_unlock(&swap_lock);
-		return -ENODEV;
+		ret = -ENODEV;
+		goto out;
 	}
 
 	/*
@@ -2270,14 +2267,15 @@ int pin_hibernation_swap_type(dev_t device, sector_t offset)
 	 * the same session.
 	 */
 	if (WARN_ON_ONCE(si->flags & SWP_HIBERNATION)) {
-		spin_unlock(&swap_lock);
-		return -EBUSY;
+		ret = -EBUSY;
+		goto out;
 	}
 
 	si->flags |= SWP_HIBERNATION;
 
+out:
 	spin_unlock(&swap_lock);
-	return type;
+	return ret;
 }
 
 /**
@@ -2296,11 +2294,8 @@ void unpin_hibernation_swap_type(int type)
 
 	spin_lock(&swap_lock);
 	si = swap_type_to_info(type);
-	if (!si) {
-		spin_unlock(&swap_lock);
-		return;
-	}
-	si->flags &= ~SWP_HIBERNATION;
+	if (si)
+		si->flags &= ~SWP_HIBERNATION;
 	spin_unlock(&swap_lock);
 }
 
@@ -2335,7 +2330,7 @@ int find_hibernation_swap_type(dev_t device, sector_t offset)
 
 int find_first_swap(dev_t *device)
 {
-	int type;
+	int type, ret = -ENODEV;
 
 	spin_lock(&swap_lock);
 	for (type = 0; type < nr_swapfiles; type++) {
@@ -2344,11 +2339,11 @@ int find_first_swap(dev_t *device)
 		if (!(sis->flags & SWP_WRITEOK))
 			continue;
 		*device = sis->bdev->bd_dev;
-		spin_unlock(&swap_lock);
-		return type;
+		ret = type;
+		break;
 	}
 	spin_unlock(&swap_lock);
-	return -ENODEV;
+	return ret;
 }
 
 /*
Signature · 1 line
2.55.0
author maintainer
#3 lore raw

RFC PATCH 3/13 mm/swap: cleanup and document swap device availability flag usage

In reply to Kairui Song

Rework swap device flag and metedata handling and swapon/swapoff to be cleaner and better documented, in preparation for locking cleanup.

Consolidate existing routines and introduce swap_device_enable and swap_device_disable as the clean boundary of exposing or isolating a swap device. swap_device_enable sets the proper flags and exposes the device as an allocation candidate, while swap_device_disable clears related flags and ensures no more allocations will happen.

And add comment blocks documenting the lifetime and locking rules for swap device flags and their locking conventions.

No feature change, only code rearrangement and documentation.

---
 include/linux/swap.h |  27 +++++---
 mm/swapfile.c        | 189 +++++++++++++++++++++++----------------------------
 2 files changed, 104 insertions(+), 112 deletions(-)
include/linux/swap.h +19 -8
index 696ed01709c2..a8ce05024cfe 100644
@@ -194,6 +194,22 @@ struct swap_extent {
 	((offsetof(union swap_header, magic.magic) - \
 	  offsetof(union swap_header, info.badpages)) / sizeof(int))
 
+/*
+ * Swap device flags, except the ones documented below, all are immutable
+ * after exposed by swap_device_enable, and until the device is freed again
+ * (SWP_USED unset). The exceptions:
+ * - SWP_USED: Protected by swap_lock. Indicates the device is inuse. Once
+ *   set, won't be cleared unless all reference to this device is freed and
+ *   swapoff finished.
+ * - SWP_WRITEOK: Protected by both swap_lock and swap_avail_lock, clearing
+ *   this flag also waits for all current cluster lock users to exit so
+ *   checking this flag while holding any of these locks ensures the device
+ *   is safe to use at the moment. Note: clearing this flag doesn't affect
+ *   pending IO or async requests, it only prevents further entry allocation
+ *   or new async request (e.g. discard) from initiating.
+ * - SWP_HIBERNATION: Protected by swap_lock. Indicates if the device
+ *   is pinned for hibernation.
+ */
 enum {
 	SWP_USED	= (1 << 0),	/* is slot in swap_info[] used? */
 	SWP_WRITEOK	= (1 << 1),	/* ok to write to this swap?	*/
@@ -263,14 +279,9 @@ struct swap_info_struct {
 	struct file *swap_file;		/* seldom referenced */
 	struct completion comp;		/* seldom referenced */
 	spinlock_t lock;		/*
-					 * protect map scan related fields like
-					 * inuse_pages and all cluster lists.
-					 * Other fields are only changed
-					 * at swapon/swapoff, so are protected
-					 * by swap_lock. changing flags need
-					 * hold this lock and swap_lock. If
-					 * both locks need hold, hold swap_lock
-					 * first.
+					 * Protect cluster lists. Other fields
+					 * are only changed at swapon/swapoff,
+					 * so are protected by swap_lock.
 					 */
 	struct work_struct discard_work; /* discard worker */
 	struct work_struct reclaim_work; /* reclaim worker */
mm/swapfile.c +85 -104

290 lines not on this page · raw message

Signature · 1 line
2.55.0
author maintainer
#4 lore raw

RFC PATCH 4/13 mm/swap: introduce swap device iteration helper

In reply to Kairui Song

There are many users that access the swap info array just to iterate through the swap devices to find usable ones. Introduce a generic helper for this to prepare for dropping the lock convention.

Also slightly adjust __swap_type_to_info to allow access of swapped off devices by moving the sanity check into its only current caller, and this way makes more sense too: the only caller is __swap_entry_to_info, where we should never see an entry referring to a dead device.

---
 mm/swap.h     | 12 +++++-----
 mm/swapfile.c | 77 ++++++++++++++++++++++++++++++++++++-----------------------
 2 files changed, 53 insertions(+), 36 deletions(-)
mm/swap.h +6 -6
index b51ad3071a73..08a782f3d5ac 100644
@@ -91,16 +91,16 @@ static inline unsigned int swp_cluster_offset(swp_entry_t entry)
  */
 static inline struct swap_info_struct *__swap_type_to_info(int type)
 {
-	struct swap_info_struct *si;
-
-	si = READ_ONCE(swap_info[type]); /* rcu_dereference() */
-	VM_WARN_ON_ONCE(percpu_ref_is_zero(&si->users)); /* race with swapoff */
-	return si;
+	return READ_ONCE(swap_info[type]); /* rcu_dereference() */
 }
 
 static inline struct swap_info_struct *__swap_entry_to_info(swp_entry_t entry)
 {
-	return __swap_type_to_info(swp_type(entry));
+	struct swap_info_struct *si;
+
+	si = __swap_type_to_info(swp_type(entry)); /* rcu_dereference() */
+	VM_WARN_ON_ONCE(percpu_ref_is_zero(&si->users)); /* race with swapoff */
+	return si;
 }
 
 static inline struct swap_cluster_info *__swap_offset_to_cluster(
mm/swapfile.c +47 -30

121 lines not on this page · raw message

Signature · 1 line
2.55.0
author maintainer
#5 lore raw

RFC PATCH 5/13 mm/swap: change the swapon lock into a percpu rwsem

In reply to Kairui Song

Reading swap metadata is a common hot path, while swapon and swapoff are costly and very rare. Convert the existing swap_lock spinlock into a percpu rwsem so that readers can run concurrently without contention. This is a first step toward cleaning up the swap lock model and we might already see a performance gain for existing readers.

Also update the comments in swap_info_struct to reflect the lock name change.

---
 include/linux/swap.h |  10 ++---
 mm/swapfile.c        | 121 +++++++++++++++++++++++++++------------------------
 2 files changed, 69 insertions(+), 62 deletions(-)
include/linux/swap.h +5 -5
index a8ce05024cfe..d0c4e0ab5806 100644
@@ -197,17 +197,17 @@ struct swap_extent {
 /*
  * Swap device flags, except the ones documented below, all are immutable
  * after exposed by swap_device_enable, and until the device is freed again
- * (SWP_USED unset). The exceptions:
- * - SWP_USED: Protected by swap_lock. Indicates the device is inuse. Once
+ * (SWP_USED unset). The exceptions are all protected by swapon_rwsem:
+ * - SWP_USED: Protected by swapon_rwsem. Indicates the device is inuse. Once
  *   set, won't be cleared unless all reference to this device is freed and
  *   swapoff finished.
- * - SWP_WRITEOK: Protected by both swap_lock and swap_avail_lock, clearing
+ * - SWP_WRITEOK: Protected by both swapon_rwsem and swap_avail_lock, clearing
  *   this flag also waits for all current cluster lock users to exit so
  *   checking this flag while holding any of these locks ensures the device
  *   is safe to use at the moment. Note: clearing this flag doesn't affect
  *   pending IO or async requests, it only prevents further entry allocation
  *   or new async request (e.g. discard) from initiating.
- * - SWP_HIBERNATION: Protected by swap_lock. Indicates if the device
+ * - SWP_HIBERNATION: Protected by swapon_rwsem. Indicates if the device
  *   is pinned for hibernation.
  */
 enum {
@@ -281,7 +281,7 @@ struct swap_info_struct {
 	spinlock_t lock;		/*
 					 * Protect cluster lists. Other fields
 					 * are only changed at swapon/swapoff,
-					 * so are protected by swap_lock.
+					 * so are protected by swapon_rwsem.
 					 */
 	struct work_struct discard_work; /* discard worker */
 	struct work_struct reclaim_work; /* reclaim worker */
mm/swapfile.c +64 -57

349 lines not on this page · raw message

Signature · 1 line
2.55.0
author maintainer
#6 lore raw

RFC PATCH 6/13 mm/swap: remove swapon mutex and update proc reader

In reply to Kairui Song

Now that swapon_rwsem protects all swapon/swapoff-sensitive data, the old swapon_mutex is redundant. Remove it and convert the /proc/swaps seq_file reader to use the rwsem instead. Also convert the swap_start iterator to use for_each_swap() for consistency.

Also adapt swap_next, it needs an explicit SWP_USED check. swap_file is not reliably protected by swapon_rwsem, so checking SWP_USED is more accurate and stable for filtering in-use devices.

---
 mm/swapfile.c | 19 +++++--------------
 1 file changed, 5 insertions(+), 14 deletions(-)
mm/swapfile.c +5 -14
index d75fad161ba5..8b073569d1d9 100644
@@ -140,9 +140,6 @@ static inline struct swap_info_struct *__swap_iter(int *i, unsigned long flag)
 
 static struct kmem_cache *swap_table_cachep;
 
-/* Protects si->swap_file for /proc/swaps usage */
-static DEFINE_MUTEX(swapon_mutex);
-
 static DECLARE_WAIT_QUEUE_HEAD(proc_poll_wait);
 /* Activity counter to indicate that a swapon or swapoff has occurred */
 static atomic_t proc_poll_event = ATOMIC_INIT(0);
@@ -3156,7 +3153,6 @@ SYSCALL_DEFINE1(swapoff, const char __user *, specialfile)
 	if (!(p->flags & SWP_SOLIDSTATE))
 		atomic_dec(&nr_rotate_swap);
 
-	mutex_lock(&swapon_mutex);
 	percpu_down_write(&swapon_rwsem);
 	spin_lock(&p->lock);
 	drain_mmlist();
@@ -3171,7 +3167,6 @@ SYSCALL_DEFINE1(swapoff, const char __user *, specialfile)
 	percpu_up_write(&swapon_rwsem);
 	arch_swap_invalidate_area(p->type);
 	zswap_swapoff(p->type);
-	mutex_unlock(&swapon_mutex);
 	kfree(p->global_cluster);
 	p->global_cluster = NULL;
 	free_swap_cluster_info(cluster_info, maxpages);
@@ -3217,20 +3212,18 @@ static __poll_t swaps_poll(struct file *file, poll_table *wait)
 	return EPOLLIN | EPOLLRDNORM;
 }
 
-/* iterator */
 static void *swap_start(struct seq_file *swap, loff_t *pos)
 {
 	struct swap_info_struct *si;
-	int type;
 	loff_t l = *pos;
 
-	mutex_lock(&swapon_mutex);
+	percpu_down_read(&swapon_rwsem);
 
 	if (!l)
 		return SEQ_START_TOKEN;
 
-	for (type = 0; (si = swap_type_to_info(type)); type++) {
-		if (!(si->swap_file))
+	for_each_swap(si) {
+		if (!si->swap_file)
 			continue;
 		if (!--l)
 			return si;
@@ -3251,7 +3244,7 @@ static void *swap_next(struct seq_file *swap, void *v, loff_t *pos)
 
 	++(*pos);
 	for (; (si = swap_type_to_info(type)); type++) {
-		if (!(si->swap_file))
+		if (!(si->flags & SWP_USED) || !(si->swap_file))
 			continue;
 		return si;
 	}
@@ -3261,7 +3254,7 @@ static void *swap_next(struct seq_file *swap, void *v, loff_t *pos)
 
 static void swap_stop(struct seq_file *swap, void *v)
 {
-	mutex_unlock(&swapon_mutex);
+	percpu_up_read(&swapon_rwsem);
 }
 
 static int swap_show(struct seq_file *swap, void *v)
@@ -3764,7 +3757,6 @@ SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags)
 		goto free_swap_zswap;
 	}
 
-	mutex_lock(&swapon_mutex);
 	prio = DEF_SWAP_PRIO;
 	if (swap_flags & SWAP_FLAG_PREFER)
 		prio = swap_flags & SWAP_FLAG_PRIO_MASK;
@@ -3790,7 +3782,6 @@ SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags)
 		(si->flags & SWP_AREA_DISCARD) ? "s" : "",
 		(si->flags & SWP_PAGE_DISCARD) ? "c" : "");
 
-	mutex_unlock(&swapon_mutex);
 	atomic_inc(&proc_poll_event);
 	wake_up_interruptible(&proc_poll_wait);
 
Signature · 1 line
2.55.0
author maintainer
#7 lore raw

RFC PATCH 7/13 mm/swap: consolidate swap inuse accounting helpers

In reply to Kairui Song

Inline the swap_usage_add/sub helpers into their only callers and rename the callers to better reflect what they actually do: track per-device inuse page counts.

No functional change.

---
 mm/swapfile.c | 64 +++++++++++++++++++++++++----------------------------------
 1 file changed, 27 insertions(+), 37 deletions(-)
mm/swapfile.c +27 -37
index 8b073569d1d9..df9855401a2e 100644
@@ -49,8 +49,8 @@
 #include "internal.h"
 #include "swap.h"
 
-static void swap_range_alloc(struct swap_info_struct *si,
-			     unsigned int nr_entries);
+static void swap_device_inuse_add(struct swap_info_struct *si,
+				  unsigned int nr_entries);
 static bool folio_swapcache_freeable(struct folio *folio);
 static void move_cluster(struct swap_info_struct *si,
 			 struct swap_cluster_info *ci, struct list_head *list,
@@ -984,7 +984,7 @@ static bool __swap_cluster_alloc_entries(struct swap_info_struct *si,
 	if (cluster_is_empty(ci))
 		ci->order = order;
 	ci->count += nr_pages;
-	swap_range_alloc(si, nr_pages);
+	swap_device_inuse_add(si, nr_pages);
 
 	return true;
 }
@@ -1286,53 +1286,36 @@ static void add_to_avail_list(struct swap_info_struct *si)
 }
 
 /*
- * swap_usage_add / swap_usage_sub of each slot are serialized by ci->lock
- * within each cluster, so the total contribution to the global counter should
- * always be positive and cannot exceed the total number of usable slots.
+ * swap_device_inuse_add / swap_device_inuse_sub are called within cluster
+ * update critical sections and serialized by ci->lock within each cluster,
+ * so the total contribution to the global counter should always be positive
+ * and cannot exceed the total number of usable slots.
  */
-static bool swap_usage_add(struct swap_info_struct *si, unsigned int nr_entries)
+static void swap_device_inuse_add(struct swap_info_struct *si,
+				  unsigned int nr_entries)
 {
-	long val = atomic_long_add_return_relaxed(nr_entries, &si->inuse_pages);
+	long inuse_pages;
 
 	/*
 	 * If device is full, and SWAP_USAGE_OFFLIST_BIT is not set,
 	 * remove it from the plist.
 	 */
-	if (unlikely(val == si->pages)) {
-		del_from_avail_list(si, false);
-		return true;
-	}
-
-	return false;
-}
-
-static void swap_usage_sub(struct swap_info_struct *si, unsigned int nr_entries)
-{
-	long val = atomic_long_sub_return_relaxed(nr_entries, &si->inuse_pages);
-
-	/*
-	 * If device is not full, and SWAP_USAGE_OFFLIST_BIT is set,
-	 * add it to the plist.
-	 */
-	if (unlikely(val & SWAP_USAGE_OFFLIST_BIT))
-		add_to_avail_list(si);
-}
-
-static void swap_range_alloc(struct swap_info_struct *si,
-			     unsigned int nr_entries)
-{
-	if (swap_usage_add(si, nr_entries)) {
+	inuse_pages = atomic_long_add_return_relaxed(nr_entries, &si->inuse_pages);
+	if (unlikely(inuse_pages == si->pages)) {
 		if (vm_swap_full())
 			schedule_work(&si->reclaim_work);
+		del_from_avail_list(si, false);
 	}
+
 	atomic_long_sub(nr_entries, &nr_swap_pages);
 }
 
-static void swap_range_free(struct swap_info_struct *si, unsigned long offset,
-			    unsigned int nr_entries)
+static void swap_device_inuse_sub(struct swap_info_struct *si, unsigned long offset,
+				  unsigned int nr_entries)
 {
 	unsigned long end = offset + nr_entries - 1;
 	void (*swap_slot_free_notify)(struct block_device *, unsigned long);
+	long inuse_pages;
 	unsigned int i;
 
 	for (i = 0; i < nr_entries; i++)
@@ -1356,7 +1339,14 @@ static void swap_range_free(struct swap_info_struct *si, unsigned long offset,
 	 */
 	smp_wmb();
 	atomic_long_add(nr_entries, &nr_swap_pages);
-	swap_usage_sub(si, nr_entries);
+
+	/*
+	 * If device is not full, and SWAP_USAGE_OFFLIST_BIT is set,
+	 * add it back to the plist.
+	 */
+	inuse_pages = atomic_long_sub_return_relaxed(nr_entries, &si->inuse_pages);
+	if (unlikely(inuse_pages & SWAP_USAGE_OFFLIST_BIT))
+		add_to_avail_list(si);
 }
 
 static bool get_swap_device_info(struct swap_info_struct *si)
@@ -1969,7 +1959,7 @@ void __swap_cluster_free_entries(struct swap_info_struct *si,
 	if (batch_id)
 		mem_cgroup_uncharge_swap(batch_id, ci_off - batch_off);
 
-	swap_range_free(si, ci_head + ci_start, nr_pages);
+	swap_device_inuse_sub(si, ci_head + ci_start, nr_pages);
 	swap_cluster_assert_empty(ci, ci_start, nr_pages, false);
 
 	if (!ci->count)
@@ -2821,7 +2811,7 @@ static int try_to_unuse(unsigned int type)
 success:
 	/*
 	 * Make sure that further cleanups after try_to_unuse() returns happen
-	 * after swap_range_free() reduces si->inuse_pages to 0.
+	 * after swap_device_inuse_sub() reduces si->inuse_pages to 0.
 	 */
 	smp_mb();
 	return 0;
Signature · 1 line
2.55.0
author maintainer
#8 lore raw

RFC PATCH 8/13 mm/swap: change back to use each swap device's percpu cluster

In reply to Kairui Song

From: Youngjun Park <[email protected]>

This reverts commit 1b7e90020eb7 ("mm, swap: use percpu cluster as allocation fast path").

The global percpu cluster causes several issues:

1) It prevents efficient swap allocation for users that do not want the default swap device policy. Because the cluster cache sits above device selection, all allocation users must stick with the default allocation policy. This fundamentally conflicts with incoming ideas like swap tiering.

2) It can cause priority inversion. Consider two memcgs: memcg1 can access devices A and B, where A has higher priority; memcg2 can only access B. Memcg2 can write the global percpu cluster with device B, then memcg1 takes B in the fast path even though higher-priority device A is not exhausted.

3) The fast-path / slow-path design bundled with the global per-CPU cache is problematic: only the fast path uses the global cache, and the slow path is forced to rotate the allocation list. This complicates consumers like discard.

Revert to per-device percpu clusters first. A later patch will introduce new infrastructure for a high-performance global allocation path.

---
 include/linux/swap.h |  13 +++-
 mm/swapfile.c        | 184 +++++++++++++++++----------------------------------
 2 files changed, 72 insertions(+), 125 deletions(-)
include/linux/swap.h +10 -3
index d0c4e0ab5806..6733a8979aa2 100644
@@ -246,10 +246,17 @@ enum {
 #endif
 
 /*
- * We keep using same cluster for rotational device so IO will be sequential.
- * The purpose is to optimize SWAP throughput on these device.
+ * We assign a cluster to each CPU, so each CPU can allocate swap entry from
+ * its own cluster and swapout sequentially. The purpose is to optimize swapout
+ * throughput.
  */
+struct percpu_cluster {
+	local_lock_t lock; /* Protect the percpu_cluster above */
+	unsigned int next[SWAP_NR_ORDERS]; /* Likely next allocation offset */
+};
+
 struct swap_sequential_cluster {
+	spinlock_t lock; /* Serialize usage of global cluster */
 	unsigned int next[SWAP_NR_ORDERS]; /* Likely next allocation offset */
 };
 
@@ -272,8 +279,8 @@ struct swap_info_struct {
 					/* list of cluster that are fragmented or contented */
 	unsigned int pages;		/* total of usable pages of swap */
 	atomic_long_t inuse_pages;	/* number of those currently in use */
+	struct percpu_cluster	__percpu *percpu_cluster; /* per cpu's swap location */
 	struct swap_sequential_cluster *global_cluster; /* Use one global cluster for rotating device */
-	spinlock_t global_cluster_lock;	/* Serialize usage of global cluster */
 	struct rb_root swap_extent_root;/* root of the swap extent rbtree */
 	struct block_device *bdev;	/* swap device or bdev of swap file */
 	struct file *swap_file;		/* seldom referenced */
mm/swapfile.c +62 -122

319 lines not on this page · raw message

Signature · 1 line
2.55.0
author maintainer
#9 lore raw

RFC PATCH 9/13 mm/swap: add priority queue for swap device allocation

In reply to Kairui Song

The swap allocator uses a plist (swap_avail_head) ordered by priority to pick the next device. To spread IO across devices at the same priority it rotates the current device to the tail via plist_requeue(), which requires holding swap_avail_lock. It then drops the lock for the allocation attempt and reacquires to check whether the next device is still on the list. This per-allocation lock cycling and the single global rotation point mean all CPUs contend for the same list head.

Add a custom priority queue that groups devices into rings, ordered by priority to form a static array as the queue. Devices in the same priority ring also form a static array.

To eliminate CPU contention on a global lock, swapon_rwsem is reused to protect the queue. The queue stays read-only, only swapon and swapoff can adjust the length of the queue or rings in it. The writer lock is costly but swapon and swapoff are rare so this is acceptable.

Since the queue is read-only, rotation is done from the reader side. Each CPU maintains its own iterator using a counter that counts down to advance to the next device in the same ring, so allocation is round-robin while retaining locality for clustering. After a failed attempt the iterator also advances so the caller does not retry the same full or fragmented device.

If a whole ring is attempted and allocation could not be made, the iterator falls through to the next ring. This almost never happens because we always want to allocate from the device with the highest priority, unless that device is full.

To speed up iteration when there are multiple full devices, a device pointer stored in the ring can be marked as unavailable so the iterator skips past it. Fully used devices can mark their pointer as unavailable without taking the rwsem lock. The marking is done using a spin lock to sync with potential writer because percpu rwsem is too heavy for writers, and full device transitions are more frequent and performance-sensitive compared to swapon and swapoff.

With this patch, swap allocation should be faster and cleaner. The percpu rwsem has very low overhead for the allocator, and it can now hold swapon_rwsem for the entire allocation loop, no longer needing to keep dropping and reacquiring a lock to rotate list entries.

The old swap_avail_head plist is still maintained in parallel for now, updated alongside the queue in del_from_avail_list() and add_to_avail_list(). It will be removed once the remaining consumer is converted in a follow-up patch.

---
 mm/swapfile.c | 455 ++++++++++++++++++++++++++++++++++++++++++++++++++--------
 1 file changed, 398 insertions(+), 57 deletions(-)
mm/swapfile.c +398 -57
index b2adf36a0884..f6dd85968928 100644
@@ -55,6 +55,7 @@ static bool folio_swapcache_freeable(struct folio *folio);
 static void move_cluster(struct swap_info_struct *si,
 			 struct swap_cluster_info *ci, struct list_head *list,
 			 enum swap_cluster_flags new_flags);
+static bool get_swap_device_info(struct swap_info_struct *si);
 
 /*
  * Serializes swapon/swapoff (writers) and protects the swap_info
@@ -160,18 +161,350 @@ static struct swap_info_struct *swap_entry_to_info(swp_entry_t entry)
 	return swap_type_to_info(swp_type(entry));
 }
 
+/*
+ * All available swap_info_structs are grouped by priority rings, the rings
+ * are ordered in a queue by priority (lower prio value = higher priority).
+ * The allocator iterates and rotates devices within each priority ring.
+ * When all devices in a ring are iterated, it goes to the next lower
+ * priority ring.
+ */
+struct swap_prio_ring {
+	int prio;
+	unsigned int size;
+	struct swap_info_struct *dev[] __counted_by(size);
+};
+
+/*
+ * The ring is protected by swapon_rwsem so updating it is costly. To make
+ * the allocator and other users skip full devices faster, the lowest bit of
+ * a device pointer is used to mark it disabled (temporarily unavailable).
+ * This relies on struct swap_info_struct being sufficiently
+ * aligned (guaranteed by kmalloc).
+ */
+#define SWAP_DEVICE_MASKED_SHIFT	0
+#define SWAP_DEVICE_MASKED_BIT		BIT(SWAP_DEVICE_MASKED_SHIFT)
+
+/*
+ * Serializes queue content mutations and keeps SWAP_USAGE_OFFLIST_BIT
+ * consistent with the masked state of each device pointer.
+ */
+static DEFINE_SPINLOCK(swap_queue_update_lock);
+
+/*
+ * Swap queue is protected by both swap_queue_update_lock and swapon_rwsem.
+ * Only swapon/swapoff will take the write lock, and modify the queue length
+ * or any ring's length. swap_queue_update_lock protects the content so
+ * devices can be masked easily without taking the writelock, which is heavy.
+ */
+static struct swap_prio_ring **swap_queue;
+static unsigned int swap_queue_len;
+
+/*
+ * Each CPU has its read iterator, so the queue itself will remain read
+ * only and the CPU side reader rotates by iterating the devices
+ * periodically using the counter.
+ */
+#define SWAP_ROUND_ROBIN_QUOTA SWAPFILE_CLUSTER
+struct swap_ring_iterator {
+	int offset;
+	long rr_counter;
+};
+
+struct swap_queue_reader {
+	local_lock_t lock;
+	struct swap_ring_iterator ri[];
+};
+
+static __percpu struct swap_queue_reader *swap_queue_readers;
+
+static inline bool swap_device_masked(struct swap_info_struct *si)
+{
+	return (unsigned long)si & SWAP_DEVICE_MASKED_BIT;
+}
+
+static inline struct swap_info_struct *swap_device_unmask_ptr(struct swap_info_struct *si)
+{
+	return (struct swap_info_struct *)((unsigned long)si & ~SWAP_DEVICE_MASKED_BIT);
+}
+
+static struct swap_queue_reader __percpu *swap_queue_prealloc_readers(int nr_rings, gfp_t gfp)
+{
+	struct swap_queue_reader __percpu *readers;
+
+	if (!nr_rings)
+		return NULL;
+
+	readers = __alloc_percpu_gfp(struct_size(readers, ri, nr_rings),
+				     __alignof__(*readers), gfp);
+	return readers;
+}
+
+static void swap_queue_install_readers(struct swap_queue_reader __percpu *readers)
+{
+	int ring_idx, cpu;
+	struct swap_prio_ring *ring;
+
+	free_percpu(swap_queue_readers);
+	swap_queue_readers = readers;
+	if (!readers)
+		return;
+
+	/* Distribute each CPU's swap IO fairly across devices. */
+	for_each_possible_cpu(cpu) {
+		local_lock_init(&per_cpu_ptr(readers, cpu)->lock);
+
+		for (ring_idx = 0; ring_idx < swap_queue_len; ring_idx++) {
+			ring = swap_queue[ring_idx];
+			per_cpu_ptr(readers, cpu)->ri[ring_idx].offset =
+				    cpu % ring->size;
+		}
+	}
+}
+
+static struct swap_info_struct *swap_queue_get_device(long nr_alloc, int nr_iter)
+{
+	bool rotate = false;
+	struct swap_info_struct *si;
+	struct swap_ring_iterator *ri;
+	struct swap_prio_ring *ring;
+	unsigned int queue_idx;
+
+	if (!swap_queue_len)
+		return ERR_PTR(-ENOENT);
+
+	queue_idx = 0;
+	while (nr_iter >= swap_queue[queue_idx]->size) {
+		nr_iter -= swap_queue[queue_idx]->size;
+		if (++queue_idx >= swap_queue_len)
+			return ERR_PTR(-ENOENT);
+	}
+
+	ring = swap_queue[queue_idx];
+	local_lock(&swap_queue_readers->lock);
+	ri = this_cpu_ptr(&swap_queue_readers->ri[queue_idx]);
+	/* Rotate while iterating the ring, just not on the first try */
+	if (nr_iter)
+		rotate = true;
+	else if (ri->rr_counter < nr_alloc)
+		rotate = true;
+	else if (ri->offset >= ring->size)
+		rotate = true;
+	if (rotate) {
+		ri->offset++;
+		ri->offset %= ring->size;
+		ri->rr_counter = SWAP_ROUND_ROBIN_QUOTA;
+	}
+	ri->rr_counter -= nr_alloc;
+	si = READ_ONCE(ring->dev[ri->offset]);
+	local_unlock(&swap_queue_readers->lock);
+
+	if (swap_device_masked(si))
+		return ERR_PTR(-EBUSY);
+
+	si = swap_device_unmask_ptr(si);
+	return si;
+}
+
+static bool swap_queue_find(struct swap_info_struct *si,
+			    unsigned int *ring_idx, unsigned int *dev_idx)
+{
+	unsigned int i, j;
+	struct swap_prio_ring *ring;
+
+	lockdep_assert(lockdep_is_held(&swapon_rwsem) ||
+		       lockdep_is_held(&swap_queue_update_lock));
+
+	for (i = 0; i < swap_queue_len; i++) {
+		ring = swap_queue[i];
+		if (ring->prio != si->prio)
+			continue;
+		for (j = 0; j < ring->size; j++) {
+			if (swap_device_unmask_ptr(READ_ONCE(ring->dev[j])) != si)
+				continue;
+			*ring_idx = i;
+			*dev_idx = j;
+			return true;
+		}
+	}
+	return false;
+}
+
+static void swap_queue_mask(struct swap_info_struct *si)
+{
+	unsigned int ring_idx, dev_idx;
+
+	lockdep_assert_held(&swap_queue_update_lock);
+	if (swap_queue_find(si, &ring_idx, &dev_idx))
+		__set_bit(SWAP_DEVICE_MASKED_SHIFT,
+			  (unsigned long *)&swap_queue[ring_idx]->dev[dev_idx]);
+}
+
+static void swap_queue_unmask(struct swap_info_struct *si)
+{
+	unsigned int ring_idx, dev_idx;
+
+	lockdep_assert_held(&swap_queue_update_lock);
+	if (swap_queue_find(si, &ring_idx, &dev_idx))
+		__clear_bit(SWAP_DEVICE_MASKED_SHIFT,
+			    (unsigned long *)&swap_queue[ring_idx]->dev[dev_idx]);
+}
+
+static int swap_queue_add(struct swap_info_struct *si)
+{
+	struct swap_prio_ring **new_queue = NULL, **old_queue = NULL;
+	struct swap_queue_reader __percpu *new_readers = NULL;
+	struct swap_prio_ring *ring, *new_ring = NULL, *old_ring = NULL;
+	int prio = si->prio;
+	int i, pos, err = -ENOMEM;
+	gfp_t gfp;
+
+	/* Swap not usable here because this is swap, just reclaim cache. */
+	gfp = GFP_NOIO | __GFP_HIGH;
+	lockdep_assert_held_write(&swapon_rwsem);
+
+	for (pos = 0; pos < swap_queue_len; pos++) {
+		if (swap_queue[pos]->prio == prio)
+			goto add_to_ring;
+		if (swap_queue[pos]->prio < prio)
+			break;
+	}
+
+	/* No ring at this priority: insert a new one at pos. */
+	new_readers = swap_queue_prealloc_readers(swap_queue_len + 1, gfp);
+	if (!new_readers)
+		goto failed;
+	new_queue = kmalloc_array(swap_queue_len + 1, sizeof(*swap_queue), gfp);
+	if (!new_queue)
+		goto failed;
+	new_ring = kmalloc(struct_size(new_ring, dev, 1), gfp);
+	if (!new_ring)
+		goto failed;
+	if (!get_swap_device_info(si))
+		goto failed;
+
+	new_ring->prio = prio;
+	new_ring->size = 1;
+	new_ring->dev[0] = si;
+	for (i = 0; i < pos; i++)
+		new_queue[i] = swap_queue[i];
+	new_queue[pos] = new_ring;
+	for (i = pos; i < swap_queue_len; i++)
+		new_queue[i + 1] = swap_queue[i];
+
+	spin_lock(&swap_queue_update_lock);
+	old_queue = swap_queue;
+	swap_queue = new_queue;
+	swap_queue_len++;
+	spin_unlock(&swap_queue_update_lock);
+	kfree(old_queue);
+
+	swap_queue_install_readers(new_readers);
+	return 0;
+
+add_to_ring:
+	ring = swap_queue[pos];
+	new_ring = kmalloc(struct_size(ring, dev, ring->size + 1), gfp);
+	if (!new_ring)
+		goto failed;
+	if (!get_swap_device_info(si))
+		goto failed;
+	spin_lock(&swap_queue_update_lock);
+	memcpy(new_ring, ring, struct_size(ring, dev, ring->size));
+	new_ring->size++;
+	new_ring->dev[new_ring->size - 1] = si;
+	old_ring = swap_queue[pos];
+	swap_queue[pos] = new_ring;
+	spin_unlock(&swap_queue_update_lock);
+	kfree(old_ring);
+	return 0;
+
+failed:
+	free_percpu(new_readers);
+	kfree(new_queue);
+	kfree(new_ring);
+	return err;
+}
+
+static void swap_queue_del(struct swap_info_struct *si)
+{
+	gfp_t gfp;
+	unsigned int ring_idx, dev_idx;
+	struct swap_queue_reader __percpu *new_readers = NULL;
+	struct swap_prio_ring *ring, *new_ring = NULL, *old_ring = NULL;
+	struct swap_prio_ring **new_queue = NULL, **old_queue = NULL;
+
+	lockdep_assert_held_write(&swapon_rwsem);
+	if (!swap_queue_find(si, &ring_idx, &dev_idx)) {
+		WARN_ON(1);
+		return;
+	}
+
+	/*
+	 * To shrink memory usage, pre-allocate new smaller data before
+	 * locking. Failure is fine, swapoff will release them anyway.
+	 */
+	gfp = GFP_NOIO | __GFP_HIGH;
+	ring = swap_queue[ring_idx];
+	if (ring->size > 1)
+		new_ring = kmalloc(struct_size(ring, dev, ring->size - 1), gfp);
+	if (ring->size == 1 && swap_queue_len > 1) {
+		new_readers = swap_queue_prealloc_readers(
+					swap_queue_len - 1, gfp);
+		new_queue = kmalloc(sizeof(*swap_queue) *
+				    (swap_queue_len - 1), gfp);
+	}
+
+	spin_lock(&swap_queue_update_lock);
+	if (ring->size > 1) {
+		/* Shift trailing devices left to fill the gap. */
+		while (++dev_idx < ring->size)
+			ring->dev[dev_idx - 1] =
+				ring->dev[dev_idx];
+		ring->size--;
+		if (new_ring) {
+			memcpy(new_ring, ring,
+			       struct_size(ring, dev, ring->size));
+			old_ring = ring;
+			swap_queue[ring_idx] = new_ring;
+		}
+	} else {
+		/* Last device in this ring: remove the ring. */
+		old_ring = ring;
+		swap_queue_len--;
+		while (++ring_idx <= swap_queue_len)
+			swap_queue[ring_idx - 1] =
+				swap_queue[ring_idx];
+		if (new_queue) {
+			memcpy(new_queue, swap_queue,
+			       sizeof(*swap_queue) * swap_queue_len);
+			old_queue = swap_queue;
+			swap_queue = new_queue;
+		} else if (!swap_queue_len) {
+			old_queue = swap_queue;
+			swap_queue = NULL;
+		}
+		if (new_readers || !swap_queue_len)
+			swap_queue_install_readers(new_readers);
+	}
+	spin_unlock(&swap_queue_update_lock);
+
+	kfree(old_ring);
+	kfree(old_queue);
+	put_swap_device(si);
+}
+
 /*
  * Use the second highest bit of inuse_pages counter as the indicator
- * if one swap device is on the available plist, so the atomic can
+ * if one swap device is unavailable for allocation, so the atomic can
  * still be updated arithmetically while having special data embedded.
  *
  * inuse_pages counter is the only thing indicating if a device should
- * be on avail_lists or not (except swapon / swapoff). By embedding the
- * off-list bit in the atomic counter, updates no longer need any lock
- * to check the list status.
+ * be in the available queue or not (except swapon / swapoff). By
+ * embedding the off-list bit in the atomic counter, updates no longer
+ * need any lock to check the list status.
  *
- * This bit will be set if the device is not on the plist and not
- * usable, will be cleared if the device is on the plist.
+ * This bit will be set if the device is not in the available queue
+ * and not usable, will be cleared if the device is in the queue.
  */
 #define SWAP_USAGE_OFFLIST_BIT (1UL << (BITS_PER_TYPE(atomic_t) - 2))
 #define SWAP_USAGE_COUNTER_MASK (~SWAP_USAGE_OFFLIST_BIT)
@@ -1215,6 +1548,7 @@ static void del_from_avail_list(struct swap_info_struct *si, bool swapoff)
 	unsigned long pages;
 
 	spin_lock(&swap_avail_lock);
+	spin_lock(&swap_queue_update_lock);
 
 	/*
 	 * Force remove it only for swapoff. Else, take it off-list only if
@@ -1232,9 +1566,10 @@ static void del_from_avail_list(struct swap_info_struct *si, bool swapoff)
 		atomic_long_or(SWAP_USAGE_OFFLIST_BIT, &si->inuse_pages);
 	}
 
+	swap_queue_mask(si);
 	plist_del(&si->avail_list, &swap_avail_head);
-
 skip:
+	spin_unlock(&swap_queue_update_lock);
 	spin_unlock(&swap_avail_lock);
 }
 
@@ -1245,12 +1580,12 @@ static void add_to_avail_list(struct swap_info_struct *si)
 	unsigned long pages;
 
 	spin_lock(&swap_avail_lock);
+	spin_lock(&swap_queue_update_lock);
 
 	/*
-	 * Add the device to the avail list if SWP_WRITEOK is set and
-	 * SWAP_USAGE_OFFLIST_BIT is still set. Swapoff clears
-	 * SWP_WRITEOK first, so the device won't be re-added after
-	 * swapoff starts unless swap_device_enable resurrects it.
+	 * Mark the device as avail if SWP_WRITEOK is set. Swapoff clears
+	 * SWP_WRITEOK first, so check that first so the device won't be
+	 * re-added after swapoff started.
 	 */
 	if (!(si->flags & SWP_WRITEOK))
 		goto skip;
@@ -1261,9 +1596,10 @@ static void add_to_avail_list(struct swap_info_struct *si)
 	val = atomic_long_fetch_and_relaxed(~SWAP_USAGE_OFFLIST_BIT, &si->inuse_pages);
 
 	/*
-	 * When device is full and device is on the plist, only one updater will
-	 * see (inuse_pages == si->pages) and will call del_from_avail_list. If
-	 * that updater happen to be here, just skip adding.
+	 * When device is full and marked as available, one reader will see
+	 * (inuse_pages == si->pages) and should mark it as unavailable and
+	 * set SWAP_USAGE_OFFLIST_BIT. If that updater happens to be here, just
+	 * skip the rest.
 	 */
 	pages = si->pages;
 	if (val == pages) {
@@ -1273,10 +1609,10 @@ static void add_to_avail_list(struct swap_info_struct *si)
 			goto skip;
 	}
 
+	swap_queue_unmask(si);
 	plist_add(&si->avail_list, &swap_avail_head);
-
 skip:
-	spin_unlock(&swap_avail_lock);
+	spin_unlock(&swap_queue_update_lock);
 }
 
 /*
@@ -1292,7 +1628,7 @@ static void swap_device_inuse_add(struct swap_info_struct *si,
 
 	/*
 	 * If device is full, and SWAP_USAGE_OFFLIST_BIT is not set,
-	 * remove it from the plist.
+	 * mark it unavailable.
 	 */
 	inuse_pages = atomic_long_add_return_relaxed(nr_entries, &si->inuse_pages);
 	if (unlikely(inuse_pages == si->pages)) {
@@ -1336,7 +1672,7 @@ static void swap_device_inuse_sub(struct swap_info_struct *si, unsigned long off
 
 	/*
 	 * If device is not full, and SWAP_USAGE_OFFLIST_BIT is set,
-	 * add it back to the plist.
+	 * add it back to the available queue.
 	 */
 	inuse_pages = atomic_long_sub_return_relaxed(nr_entries, &si->inuse_pages);
 	if (unlikely(inuse_pages & SWAP_USAGE_OFFLIST_BIT))
@@ -1359,41 +1695,40 @@ static bool get_swap_device_info(struct swap_info_struct *si)
 	return true;
 }
 
-/* Rotate the device and switch to a new cluster */
-static void swap_alloc_entry(struct folio *folio)
+static int swap_alloc_entry(struct folio *folio)
 {
-	struct swap_info_struct *si, *next;
+	long nr_pages = folio_nr_pages(folio);
+	struct swap_info_struct *si;
+	int nr_iter, ret;
 
-	spin_lock(&swap_avail_lock);
-start_over:
-	plist_for_each_entry_safe(si, next, &swap_avail_head, avail_list) {
-		/* Rotate the device and switch to a new cluster */
-		plist_requeue(&si->avail_list, &swap_avail_head);
-		spin_unlock(&swap_avail_lock);
-		if (get_swap_device_info(si)) {
-			cluster_alloc_swap_entry(si, folio);
-			put_swap_device(si);
-			if (folio_test_swapcache(folio))
-				return;
-			if (folio_test_large(folio))
-				return;
+	percpu_down_read(&swapon_rwsem);
+	for (nr_iter = 0;; nr_iter++) {
+		si = swap_queue_get_device(nr_pages, nr_iter);
+		if (IS_ERR(si)) {
+			ret = PTR_ERR(si);
+			if (ret == -EBUSY)
+				continue;
+			break;
+		}
+		cluster_alloc_swap_entry(si, folio);
+
+		if (folio_test_swapcache(folio)) {
+			ret = 0;
+			break;
 		}
 
-		spin_lock(&swap_avail_lock);
 		/*
-		 * if we got here, it's likely that si was almost full before,
-		 * multiple callers probably all tried to get a page from the
-		 * same si and it filled up before we could get one; or, the si
-		 * filled up between us dropping swap_avail_lock.
-		 * Since we dropped the swap_avail_lock, the swap_avail_list
-		 * may have been modified; so if next is still in the
-		 * swap_avail_head list then try it, otherwise start over if we
-		 * have not gotten any slots.
+		 * For large allocation, return error directly to inform the
+		 * caller to split it instead of fallback to other devices.
 		 */
-		if (plist_node_empty(&next->avail_list))
-			goto start_over;
+		if (folio_test_large(folio)) {
+			ret = -E2BIG;
+			break;
+		}
 	}
-	spin_unlock(&swap_avail_lock);
+
+	percpu_up_read(&swapon_rwsem);
+	return ret;
 }
 
 /*
@@ -1707,6 +2042,7 @@ int folio_alloc_swap(struct folio *folio)
 {
 	unsigned int order = folio_order(folio);
 	unsigned int size = 1 << order;
+	int ret;
 
 	VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio);
 	VM_BUG_ON_FOLIO(!folio_test_uptodate(folio), folio);
@@ -1730,7 +2066,7 @@ int folio_alloc_swap(struct folio *folio)
 	}
 
 again:
-	swap_alloc_entry(folio);
+	ret = swap_alloc_entry(folio);
 
 	if (!order && unlikely(!folio_test_swapcache(folio))) {
 		if (swap_sync_discard())
@@ -1742,7 +2078,7 @@ int folio_alloc_swap(struct folio *folio)
 		swap_cache_del_folio(folio);
 
 	if (unlikely(!folio_test_swapcache(folio)))
-		return -ENOMEM;
+		return ret ? ret : -ENOMEM;
 
 	return 0;
 }
@@ -2913,10 +3249,9 @@ static int setup_swap_extents(struct swap_info_struct *sis,
 static void swap_device_enable(struct swap_info_struct *si)
 {
 	percpu_down_write(&swapon_rwsem);
-	spin_lock(&swap_avail_lock);
+	spin_lock(&swap_queue_update_lock);
 	si->flags |= SWP_WRITEOK;
-	spin_unlock(&swap_avail_lock);
-
+	spin_unlock(&swap_queue_update_lock);
 	atomic_long_add(si->pages, &nr_swap_pages);
 	total_swap_pages += si->pages;
 	plist_add(&si->list, &swap_active_head);
@@ -2948,16 +3283,14 @@ static int swap_device_disable(struct swap_info_struct *si)
 	}
 	vm_unacct_memory(si->pages);
 
-	spin_lock(&swap_avail_lock);
+	spin_lock(&swap_queue_update_lock);
 	si->flags &= ~SWP_WRITEOK;
-	spin_unlock(&swap_avail_lock);
-
+	spin_unlock(&swap_queue_update_lock);
 	plist_del(&si->list, &swap_active_head);
 	total_swap_pages -= si->pages;
 	atomic_long_sub(si->pages, &nr_swap_pages);
-	percpu_up_write(&swapon_rwsem);
-
 	del_from_avail_list(si, true);
+	percpu_up_write(&swapon_rwsem);
 
 	/*
 	 * Swap allocator doesn't touch si lock, so looping through all
@@ -3042,6 +3375,9 @@ SYSCALL_DEFINE1(swapoff, const char __user *, specialfile)
 		return err;
 	}
 
+	percpu_down_write(&swapon_rwsem);
+	swap_queue_del(p);
+
 	/*
 	 * Wait for swap operations protected by get/put_swap_device()
 	 * to complete.  Because of synchronize_rcu() here, all swap
@@ -3062,7 +3398,6 @@ SYSCALL_DEFINE1(swapoff, const char __user *, specialfile)
 	if (!(p->flags & SWP_SOLIDSTATE))
 		atomic_dec(&nr_rotate_swap);
 
-	percpu_down_write(&swapon_rwsem);
 	spin_lock(&p->lock);
 	drain_mmlist();
 
@@ -3700,6 +4035,12 @@ SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags)
 
 	/* Sets SWP_WRITEOK, resurrect the percpu ref, expose the swap device */
 	percpu_ref_resurrect(&si->users);
+	percpu_down_write(&swapon_rwsem);
+	error = swap_queue_add(si);
+	percpu_up_write(&swapon_rwsem);
+	if (error)
+		goto free_swap_zswap;
+
 	swap_device_enable(si);
 
 	pr_info("Adding %uk swap on %s.  Priority:%d extents:%d across:%lluk %s%s%s%s\n",
Signature · 1 line
2.55.0
reviewer
#10 lore raw

Re: [PATCH RFC 09/13] mm/swap: add priority queue for swap device allocation

Quoted 1 line · On Tue, Jul 14, 2026 at 01:25:47AM +0800, Kairui Song via B4 Relay wrote:

From: Kairui Song <[email protected]>

Hi Kairui,

Thanks for the patch :)
I like the idea. A few questions and comments on the core implementation below.

I'll share my higher-level thoughts (e.g. on a tunable interface for the round-robin policy) separately in a follow-up.

Quoted 3 lines

+/*
+ * All available swap_info_structs are grouped by priority rings, the rings
+ * are ordered in a queue by priority (lower prio value = higher priority).

looks like comment updated needed :)

[...]

mm/swapfile.c static struct swap_info_struct *swap_entry_to_info(swp_entry_t entry) · line 185

Quoted 10 lines
mm/swapfile.c +10 -0
+
+/*
+ * The ring is protected by swapon_rwsem so updating it is costly. To make
+ * the allocator and other users skip full devices faster, the lowest bit of
+ * a device pointer is used to mark it disabled (temporarily unavailable).
+ * This relies on struct swap_info_struct being sufficiently
+ * aligned (guaranteed by kmalloc).
+ */
+#define SWAP_DEVICE_MASKED_SHIFT	0
+#define SWAP_DEVICE_MASKED_BIT		BIT(SWAP_DEVICE_MASKED_SHIFT)

Unlike the old plist, full devices stay in the ring, so every allocation keeps walking past them. When the higher priority devices are all full, each swap out pays one masked pointer check per full device before reaching a usable one. That is also the steady state for swap tiering, so I want to be sure it stays cheap. It looks like a deliberate trade-off of the static ring design. What do you think?

[...]

Quoted 33 lines
mm/swapfile.c +33 -0
+static struct swap_info_struct *swap_queue_get_device(long nr_alloc, int nr_iter)
+{
+	bool rotate = false;
+	struct swap_info_struct *si;
+	struct swap_ring_iterator *ri;
+	struct swap_prio_ring *ring;
+	unsigned int queue_idx;
+
+	if (!swap_queue_len)
+		return ERR_PTR(-ENOENT);
+
+	queue_idx = 0;
+	while (nr_iter >= swap_queue[queue_idx]->size) {
+		nr_iter -= swap_queue[queue_idx]->size;
+		if (++queue_idx >= swap_queue_len)
+			return ERR_PTR(-ENOENT);
+	}
+
+	ring = swap_queue[queue_idx];
+	local_lock(&swap_queue_readers->lock);
+	ri = this_cpu_ptr(&swap_queue_readers->ri[queue_idx]);
+	/* Rotate while iterating the ring, just not on the first try */
+	if (nr_iter)
+		rotate = true;
+	else if (ri->rr_counter < nr_alloc)
+		rotate = true;
+	else if (ri->offset >= ring->size)
+		rotate = true;
+	if (rotate) {
+		ri->offset++;
+		ri->offset %= ring->size;
+		ri->rr_counter = SWAP_ROUND_ROBIN_QUOTA;
+	}

mm/swapfile.c static struct swap_info_struct *swap_entry_to_info(swp_entry_t entry) · line 306

Quoted 10 lines
mm/swapfile.c +10 -0
+	ri->rr_counter -= nr_alloc;
+	si = READ_ONCE(ring->dev[ri->offset]);
+	local_unlock(&swap_queue_readers->lock);
+
+	if (swap_device_masked(si))
+		return ERR_PTR(-EBUSY);
+
+	si = swap_device_unmask_ptr(si);
+	return si;
+}

Tasks interleaving on the same CPU can rotate the ring more than once for a single failure, since the lock is dropped between two get_device() calls of one allocation. A retry walk can then visit one device twice and skip another. It is transient and looks harmless, and every fix I could think of costs more than it is worth....

Unless you have a better idea, how about noting it in a comment?

Quoted 13 lines
mm/swapfile.c +4 -7
-/* Rotate the device and switch to a new cluster */
-static void swap_alloc_entry(struct folio *folio)
+static int swap_alloc_entry(struct folio *folio)
 {
-	struct swap_info_struct *si, *next;
+	long nr_pages = folio_nr_pages(folio);
+	struct swap_info_struct *si;
+	int nr_iter, ret;
 
-	spin_lock(&swap_avail_lock);
-start_over:
-	plist_for_each_entry_safe(si, next, &swap_avail_head, avail_list) {
-		/* Rotate the device and switch to a new cluster */

mm/swapfile.c static bool get_swap_device_info(struct swap_info_struct *si) · line 1704

Quoted 10 lines
mm/swapfile.c +1 -9
-		plist_requeue(&si->avail_list, &swap_avail_head);
-		spin_unlock(&swap_avail_lock);
-		if (get_swap_device_info(si)) {
-			cluster_alloc_swap_entry(si, folio);
-			put_swap_device(si);
-			if (folio_test_swapcache(folio))
-				return;
-			if (folio_test_large(folio))
-				return;
+	percpu_down_read(&swapon_rwsem);

The local lock is only held inside swap_queue_get_device() now, so the task can migrate between device selection and cluster_alloc_swap_entry(). Then rr_counter is charged on one CPU while the pages land in another CPU's cluster, so the pacing of one cluster per visit does not hold anymore.

Since you want the loop to stay sleepable, how about migrate_disable() around the loop?

mm/swapfile.c static bool get_swap_device_info(struct swap_info_struct *si) · line 1706

Quoted 2 lines
mm/swapfile.c +2 -0
+	for (nr_iter = 0;; nr_iter++) {
+		si = swap_queue_get_device(nr_pages, nr_iter);

When a ring has a single device, which is probably the most common setup, the iterator does nothing useful. Would a fast path that skips the local_lock and just reads ring->dev[0] for size == 1 rings sequentially be worth it? swapon_rwsem keeps the queue stable, so choosing between the two paths should be race free, I think.

Quoted 12 lines
mm/swapfile.c +11 -0
+		if (IS_ERR(si)) {
+			ret = PTR_ERR(si);
+			if (ret == -EBUSY)
+				continue;
+			break;
+		}
+		cluster_alloc_swap_entry(si, folio);
+
+		if (folio_test_swapcache(folio)) {
+			ret = 0;
+			break;
		}

[...]

Quoted 16 lines
	del_from_avail_list(si, true);
(quoted diff) +1 -0
+	percpu_up_write(&swapon_rwsem);
	/*
	 * Swap allocator doesn't touch si lock, so looping through all
mm/swapfile.c +3 -0
@@ -3042,6 +3375,9 @@ SYSCALL_DEFINE1(swapoff, const char __user *, specialfile)
 		return err;
 	}
 
+	percpu_down_write(&swapon_rwsem);
+	swap_queue_del(p);
+
 	/*
 	 * Wait for swap operations protected by get/put_swap_device()
 	 * to complete.  Because of synchronize_rcu() here, all swap

mm/swapfile.c SYSCALL_DEFINE1(swapoff, const char __user *, specialfile) · line 3402

Quoted 6 lines
mm/swapfile.c +0 -1
@@ -3062,7 +3398,6 @@ SYSCALL_DEFINE1(swapoff, const char __user *, specialfile)
 	if (!(p->flags & SWP_SOLIDSTATE))
 		atomic_dec(&nr_rotate_swap);
 
-	percpu_down_write(&swapon_rwsem);
 	spin_lock(&p->lock);
 	drain_mmlist();

synchronize_rcu() can take a while and it runs here with the write lock held, together with wait_for_completion() and flush_work(), so a single swapoff can stall all swap allocation during a little bit long time.

Only swap_queue_del() and the final teardown seem to need the write lock. If my assumption is right, how about dropping it after swap_queue_del() and re-taking it for the teardown?

mm/swapfile.c SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags) · line 4044

Quoted 11 lines
mm/swapfile.c +6 -0
@@ -3700,6 +4035,12 @@ SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags)
 
 	/* Sets SWP_WRITEOK, resurrect the percpu ref, expose the swap device */
 	percpu_ref_resurrect(&si->users);
+	percpu_down_write(&swapon_rwsem);
+	error = swap_queue_add(si);
+	percpu_up_write(&swapon_rwsem);
+	if (error)
+		goto free_swap_zswap;
+
 	swap_device_enable(si);

Thanks!
Youngjun Park

#11 lore raw

Re: [PATCH RFC 09/13] mm/swap: add priority queue for swap device allocation

Quoted 11 lines · Youngjun Park
Quoted 1 line · On Tue, Jul 14, 2026 at 01:25:47AM +0800, Kairui Song via B4 Relay wrote:

From: Kairui Song <[email protected]>

Hi Kairui,

Thanks for the patch :)
I like the idea. A few questions and comments on the core implementation below.

I'll share my higher-level thoughts (e.g. on a tunable interface for the round-robin policy) separately in a follow-up.

Thanks for the review!

Quoted 5 lines
Quoted 3 lines

+/*
+ * All available swap_info_structs are grouped by priority rings, the rings
+ * are ordered in a queue by priority (lower prio value = higher priority).

looks like comment updated needed :)

Nice catch :), I got confused by the priority defination again, will fix it.

Quoted 12 lines

[...]

mm/swapfile.c static struct swap_info_struct *swap_entry_to_info(swp_entry_t entry) · line 185

Quoted 10 lines
mm/swapfile.c +10 -0
+
+/*
+ * The ring is protected by swapon_rwsem so updating it is costly. To make
+ * the allocator and other users skip full devices faster, the lowest bit of
+ * a device pointer is used to mark it disabled (temporarily unavailable).
+ * This relies on struct swap_info_struct being sufficiently
+ * aligned (guaranteed by kmalloc).
+ */
+#define SWAP_DEVICE_MASKED_SHIFT     0
+#define SWAP_DEVICE_MASKED_BIT               BIT(SWAP_DEVICE_MASKED_SHIFT)
Quoted 7 lines

Unlike the old plist, full devices stay in the ring, so every allocation keeps walking past them. When the higher priority devices are all full, each swap out pays one masked pointer check per full device before reaching a usable one. That is also the steady state for swap tiering, so I want to be sure it stays cheap. It looks like a deliberate trade-off of the static ring design. What do you think?

Right, this should be super cheap as modern processors are really good at walking arrays. We have at most 32 swap devices in most cases, only 4 cache lines if they are in the same top ring and I think the CPU will only spend a dozen cycles on them. And rings are mostly read-only; they will stay cached for hot access.

If they fallback to secondary rings... I think these devices are mostly performance insensitive anyway, and CPUs are still pretty good at reading the readonly arrays.

I did plan to add a ring->avail to skip completely empty rings, or use cmpxchg & a seq style lock (don't shrink the ring, just shift the content on full device removal) to update the ring content, to archive 0 overhead if we got more full devices, it works but that's ends up super complex with no measurable gain.

Maybe we can implement these if this percpu reader queue can be used as a generic infrastructure for other components, or if we grow to support hundreds of swaps someday, not really a issue for now I think.

Quoted 45 lines

[...]

Quoted 42 lines
mm/swapfile.c +42 -0
+static struct swap_info_struct *swap_queue_get_device(long nr_alloc, int nr_iter)
+{
+     bool rotate = false;
+     struct swap_info_struct *si;
+     struct swap_ring_iterator *ri;
+     struct swap_prio_ring *ring;
+     unsigned int queue_idx;
+
+     if (!swap_queue_len)
+             return ERR_PTR(-ENOENT);
+
+     queue_idx = 0;
+     while (nr_iter >= swap_queue[queue_idx]->size) {
+             nr_iter -= swap_queue[queue_idx]->size;
+             if (++queue_idx >= swap_queue_len)
+                     return ERR_PTR(-ENOENT);
+     }
+
+     ring = swap_queue[queue_idx];
+     local_lock(&swap_queue_readers->lock);
+     ri = this_cpu_ptr(&swap_queue_readers->ri[queue_idx]);
+     /* Rotate while iterating the ring, just not on the first try */
+     if (nr_iter)
+             rotate = true;
+     else if (ri->rr_counter < nr_alloc)
+             rotate = true;
+     else if (ri->offset >= ring->size)
+             rotate = true;
+     if (rotate) {
+             ri->offset++;
+             ri->offset %= ring->size;
+             ri->rr_counter = SWAP_ROUND_ROBIN_QUOTA;
+     }
+     ri->rr_counter -= nr_alloc;
+     si = READ_ONCE(ring->dev[ri->offset]);
+     local_unlock(&swap_queue_readers->lock);
+
+     if (swap_device_masked(si))
+             return ERR_PTR(-EBUSY);
+
+     si = swap_device_unmask_ptr(si);
+     return si;
Quoted 8 lines
Quoted 1 line

+}

Tasks interleaving on the same CPU can rotate the ring more than once for a single failure, since the lock is dropped between two get_device() calls of one allocation. A retry walk can then visit one device twice and skip another. It is transient and looks harmless, and every fix I could think of costs more than it is worth....

You mean dropping the local lock? Rotating it a bit more might be harmless I think, if that happens randomly, then the rotation is still fair.

I did think about keep holding the local lock during the whole process, but making this unsleepable is a trade-off and reduce future flexibility... the local lock of the cluster below is already causing some trouble for swap table allocation so I hoped we could avoid that. So far, stress test didn't show any fragmentation issue.

Quoted 3 lines

Unless you have a better idea, how about noting it in a comment?

Yeah, we can definitely doc that more cleanly.

Quoted 30 lines
Quoted 13 lines
mm/swapfile.c +4 -7
-/* Rotate the device and switch to a new cluster */
-static void swap_alloc_entry(struct folio *folio)
+static int swap_alloc_entry(struct folio *folio)
 {
-     struct swap_info_struct *si, *next;
+     long nr_pages = folio_nr_pages(folio);
+     struct swap_info_struct *si;
+     int nr_iter, ret;
 
-     spin_lock(&swap_avail_lock);
-start_over:
-     plist_for_each_entry_safe(si, next, &swap_avail_head, avail_list) {
-             /* Rotate the device and switch to a new cluster */

mm/swapfile.c static bool get_swap_device_info(struct swap_info_struct *si) · line 1704

Quoted 10 lines
mm/swapfile.c +1 -9
-             plist_requeue(&si->avail_list, &swap_avail_head);
-             spin_unlock(&swap_avail_lock);
-             if (get_swap_device_info(si)) {
-                     cluster_alloc_swap_entry(si, folio);
-                     put_swap_device(si);
-                     if (folio_test_swapcache(folio))
-                             return;
-                     if (folio_test_large(folio))
-                             return;
+     percpu_down_read(&swapon_rwsem);

The local lock is only held inside swap_queue_get_device() now, so the task can migrate between device selection and cluster_alloc_swap_entry(). Then rr_counter is charged on one CPU while the pages land in another CPU's cluster, so the pacing of one cluster per visit does not hold anymore.

Quoted 2 lines

Since you want the loop to stay sleepable, how about migrate_disable() around the loop?

That sounds like a good idea to avoid a potential race. I think these races are harmless though.

Quoted 9 lines

mm/swapfile.c static bool get_swap_device_info(struct swap_info_struct *si) · line 1706

Quoted 3 lines
mm/swapfile.c +2 -0
+     for (nr_iter = 0;; nr_iter++) {
+             si = swap_queue_get_device(nr_pages, nr_iter);

When a ring has a single device, which is probably the most common setup, the iterator does nothing useful. Would a fast path that skips the local_lock and just reads ring->dev[0] for size == 1 rings sequentially be worth it? swapon_rwsem keeps the queue stable, so choosing between the two paths should be race free, I think.

Ah, that's a good idea indeed.

Quoted 27 lines
Quoted 16 lines
      del_from_avail_list(si, true);
mm/swapfile.c +4 -0
+     percpu_up_write(&swapon_rwsem);
 
      /*
       * Swap allocator doesn't touch si lock, so looping through all
@@ -3042,6 +3375,9 @@ SYSCALL_DEFINE1(swapoff, const char __user *, specialfile)
              return err;
      }
 
+     percpu_down_write(&swapon_rwsem);
+     swap_queue_del(p);
+
      /*
       * Wait for swap operations protected by get/put_swap_device()
       * to complete.  Because of synchronize_rcu() here, all swap

mm/swapfile.c SYSCALL_DEFINE1(swapoff, const char __user *, specialfile) · line 3402

Quoted 6 lines
mm/swapfile.c +0 -1
@@ -3062,7 +3398,6 @@ SYSCALL_DEFINE1(swapoff, const char __user *, specialfile)
      if (!(p->flags & SWP_SOLIDSTATE))
              atomic_dec(&nr_rotate_swap);
 
-     percpu_down_write(&swapon_rwsem);
      spin_lock(&p->lock);
      drain_mmlist();

synchronize_rcu() can take a while and it runs here with the write lock held, together with wait_for_completion() and flush_work(), so a single swapoff can stall all swap allocation during a little bit long time.

Quoted 3 lines

Only swap_queue_del() and the final teardown seem to need the write lock. If my assumption is right, how about dropping it after swap_queue_del() and re-taking it for the teardown?

Good observation! I noticed we could tidy the locking scope, I skipped that optimization for the RFC :P. Swapoff is very performance insensitive IMO, it will put a huge pressure to the system and IO, so an RCU sync / flush hardly matters. The design is heavily optimized toward readers (ordinary swapin/swapout). But a tidier implementation and optimizations are definitely great to have.

Thanks again for the detailed review! I think this can be rebased on top of your tiering work, there are some conflicts but nothing fundamental, and I think the ring design actually fits well. I've been experiement this for a while so I think better send it out to plan out the next steps, and this series can be used to optimize the tiering alloc as follow up.

#12 lore raw

Re: [PATCH RFC 09/13] mm/swap: add priority queue for swap device allocation

In reply to Kairui Song

Hi Kairui,

Thanks for this series! I noticed a few issues in this patch:

[...]

Quoted 45 lines
mm/swapfile.c +45 -0
+static void swap_queue_install_readers(struct swap_queue_reader __percpu *readers)
+{
+	int ring_idx, cpu;
+	struct swap_prio_ring *ring;
+
+	free_percpu(swap_queue_readers);
+	swap_queue_readers = readers;
+	if (!readers)
+		return;
+
+	/* Distribute each CPU's swap IO fairly across devices. */
+	for_each_possible_cpu(cpu) {
+		local_lock_init(&per_cpu_ptr(readers, cpu)->lock);
+
+		for (ring_idx = 0; ring_idx < swap_queue_len; ring_idx++) {
+			ring = swap_queue[ring_idx];
+			per_cpu_ptr(readers, cpu)->ri[ring_idx].offset =
+				    cpu % ring->size;
+		}
+	}
+}
+static struct swap_info_struct *swap_queue_get_device(long nr_alloc, int nr_iter)
+{
+	bool rotate = false;
+	struct swap_info_struct *si;
+	struct swap_ring_iterator *ri;
+	struct swap_prio_ring *ring;
+	unsigned int queue_idx;
+
+	if (!swap_queue_len)
+		return ERR_PTR(-ENOENT);
+
+	queue_idx = 0;
+	while (nr_iter >= swap_queue[queue_idx]->size) {
+		nr_iter -= swap_queue[queue_idx]->size;
+		if (++queue_idx >= swap_queue_len)
+			return ERR_PTR(-ENOENT);
+	}
+
+	ring = swap_queue[queue_idx];
+	local_lock(&swap_queue_readers->lock);
+	ri = this_cpu_ptr(&swap_queue_readers->ri[queue_idx]);
+	/* Rotate while iterating the ring, just not on the first try */
+	if (nr_iter)
+		rotate = true;

mm/swapfile.c static struct swap_info_struct *swap_entry_to_info(swp_entry_t entry) · line 297

Quoted 10 lines
mm/swapfile.c +10 -0
+	else if (ri->rr_counter < nr_alloc)
+		rotate = true;
+	else if (ri->offset >= ring->size)
+		rotate = true;
+	if (rotate) {
+		ri->offset++;
+		ri->offset %= ring->size;
+		ri->rr_counter = SWAP_ROUND_ROBIN_QUOTA;
+	}
+	ri->rr_counter -= nr_alloc;

ri->rr_counter is left at 0 so the first allocation on each CPU hits ri->rr_counter < nr_alloc and rotates, shifting offset away from intended cpu % ring->size start. Initializing rr_counter to SWAP_ROUND_ROBIN_QUOTA would avoid this, or is this initial rotation perhaps intentional?

[...]

Quoted 41 lines
mm/swapfile.c +9 -8
@@ -1245,12 +1580,12 @@ static void add_to_avail_list(struct swap_info_struct *si)
 	unsigned long pages;
 
 	spin_lock(&swap_avail_lock);
+	spin_lock(&swap_queue_update_lock);
 
 	/*
-	 * Add the device to the avail list if SWP_WRITEOK is set and
-	 * SWAP_USAGE_OFFLIST_BIT is still set. Swapoff clears
-	 * SWP_WRITEOK first, so the device won't be re-added after
-	 * swapoff starts unless swap_device_enable resurrects it.
+	 * Mark the device as avail if SWP_WRITEOK is set. Swapoff clears
+	 * SWP_WRITEOK first, so check that first so the device won't be
+	 * re-added after swapoff started.
 	 */
 	if (!(si->flags & SWP_WRITEOK))
 		goto skip;
@@ -1261,9 +1596,10 @@ static void add_to_avail_list(struct swap_info_struct *si)
 	val = atomic_long_fetch_and_relaxed(~SWAP_USAGE_OFFLIST_BIT, &si->inuse_pages);
 
 	/*
-	 * When device is full and device is on the plist, only one updater will
-	 * see (inuse_pages == si->pages) and will call del_from_avail_list. If
-	 * that updater happen to be here, just skip adding.
+	 * When device is full and marked as available, one reader will see
+	 * (inuse_pages == si->pages) and should mark it as unavailable and
+	 * set SWAP_USAGE_OFFLIST_BIT. If that updater happens to be here, just
+	 * skip the rest.
 	 */
 	pages = si->pages;
 	if (val == pages) {
@@ -1273,10 +1609,10 @@ static void add_to_avail_list(struct swap_info_struct *si)
 			goto skip;
 	}
 
+	swap_queue_unmask(si);
 	plist_add(&si->avail_list, &swap_avail_head);
-

skip:

(quoted diff) +1 -1
-	spin_unlock(&swap_avail_lock);
+	spin_unlock(&swap_queue_update_lock);
Quoted 1 line

}

swap_avail_lock is acquired but never released when add_to_avail_list() returns.

[...]

mm/swapfile.c SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags) · line 4044

Quoted 11 lines
mm/swapfile.c +6 -0
@@ -3700,6 +4035,12 @@ SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags)
 
 	/* Sets SWP_WRITEOK, resurrect the percpu ref, expose the swap device */
 	percpu_ref_resurrect(&si->users);
+	percpu_down_write(&swapon_rwsem);
+	error = swap_queue_add(si);
+	percpu_up_write(&swapon_rwsem);
+	if (error)
+		goto free_swap_zswap;
+
 	swap_device_enable(si);

The swapfile is inserted into the ring unmasked before SWP_WRITEOK was set in swap_device_enable(). Allocators can pick it and get -EBUSY from get_swap_device_info(). How about adding masked and then letting swap_device_enable() unmask the new swapfile?

---
Thanks!
Jihan LIN

#13 lore raw

Re: [PATCH RFC 09/13] mm/swap: add priority queue for swap device allocation

Quoted 4 lines · Jihan LIN

Hi Kairui,

Thanks for this series! I noticed a few issues in this patch:

Thanks for the review!

Quoted 32 lines
Quoted 32 lines
mm/swapfile.c +32 -0
+static struct swap_info_struct *swap_queue_get_device(long nr_alloc, int nr_iter)
+{
+      bool rotate = false;
+      struct swap_info_struct *si;
+      struct swap_ring_iterator *ri;
+      struct swap_prio_ring *ring;
+      unsigned int queue_idx;
+
+      if (!swap_queue_len)
+              return ERR_PTR(-ENOENT);
+
+      queue_idx = 0;
+      while (nr_iter >= swap_queue[queue_idx]->size) {
+              nr_iter -= swap_queue[queue_idx]->size;
+              if (++queue_idx >= swap_queue_len)
+                      return ERR_PTR(-ENOENT);
+      }
+
+      ring = swap_queue[queue_idx];
+      local_lock(&swap_queue_readers->lock);
+      ri = this_cpu_ptr(&swap_queue_readers->ri[queue_idx]);
+      /* Rotate while iterating the ring, just not on the first try */
+      if (nr_iter)
+              rotate = true;
+      else if (ri->rr_counter < nr_alloc)
+              rotate = true;
+      else if (ri->offset >= ring->size)
+              rotate = true;
+      if (rotate) {
+              ri->offset++;
+              ri->offset %= ring->size;
+              ri->rr_counter = SWAP_ROUND_ROBIN_QUOTA;
Quoted 8 lines

Review

Quoted 2 lines
(quoted diff) +2 -0
+      }
+      ri->rr_counter -= nr_alloc;

ri->rr_counter is left at 0 so the first allocation on each CPU hits ri->rr_counter < nr_alloc and rotates, shifting offset away from intended cpu % ring->size start. Initializing rr_counter to SWAP_ROUND_ROBIN_QUOTA would avoid this, or is this initial rotation perhaps intentional?

Yeah, that's true, it hardly matters as we will do million of allocation and rotates a lot randomly, so the first allocation is really trivial. But initlalize it as SWAP_ROUND_ROBIN_QUOTA definately make it more consistent. Agree on that.

Quoted 37 lines

[...]

Quoted 35 lines
mm/swapfile.c +8 -7
@@ -1245,12 +1580,12 @@ static void add_to_avail_list(struct swap_info_struct *si)
      unsigned long pages;
 
      spin_lock(&swap_avail_lock);
+      spin_lock(&swap_queue_update_lock);
 
      /*
-       * Add the device to the avail list if SWP_WRITEOK is set and
-       * SWAP_USAGE_OFFLIST_BIT is still set. Swapoff clears
-       * SWP_WRITEOK first, so the device won't be re-added after
-       * swapoff starts unless swap_device_enable resurrects it.
+       * Mark the device as avail if SWP_WRITEOK is set. Swapoff clears
+       * SWP_WRITEOK first, so check that first so the device won't be
+       * re-added after swapoff started.
       */
      if (!(si->flags & SWP_WRITEOK))
              goto skip;
@@ -1261,9 +1596,10 @@ static void add_to_avail_list(struct swap_info_struct *si)
      val = atomic_long_fetch_and_relaxed(~SWAP_USAGE_OFFLIST_BIT, &si->inuse_pages);
 
      /*
-       * When device is full and device is on the plist, only one updater will
-       * see (inuse_pages == si->pages) and will call del_from_avail_list. If
-       * that updater happen to be here, just skip adding.
+       * When device is full and marked as available, one reader will see
+       * (inuse_pages == si->pages) and should mark it as unavailable and
+       * set SWAP_USAGE_OFFLIST_BIT. If that updater happens to be here, just
+       * skip the rest.
       */
      pages = si->pages;
      if (val == pages) {
@@ -1273,10 +1609,10 @@ static void add_to_avail_list(struct swap_info_struct *si)
                      goto skip;
      }
 
Quoted 9 lines
Quoted 7 lines
mm/swapfile.c +1 -1
+      swap_queue_unmask(si);
      plist_add(&si->avail_list, &swap_avail_head);
-

skip:

(quoted diff) +1 -1
-      spin_unlock(&swap_avail_lock);
+      spin_unlock(&swap_queue_update_lock);

}

swap_avail_lock is acquired but never released when add_to_avail_list() returns.

Ah, that really a silly error of mine, in the following commit swap_avail_lock is completely removed, so I didn't notice that during review or test. Need to fix this intermeidate commit. Thanks!

Quoted 5 lines

[...]

Quoted 2 lines
(quoted diff) +0 -0
@@ -3700,6 +4035,12 @@ SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags)
 
Quoted 13 lines

mm/swapfile.c SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags) · line 4044

Quoted 9 lines
mm/swapfile.c +6 -0
      /* Sets SWP_WRITEOK, resurrect the percpu ref, expose the swap device */
      percpu_ref_resurrect(&si->users);
+      percpu_down_write(&swapon_rwsem);
+      error = swap_queue_add(si);
+      percpu_up_write(&swapon_rwsem);
+      if (error)
+              goto free_swap_zswap;
+
      swap_device_enable(si);

The swapfile is inserted into the ring unmasked before SWP_WRITEOK was set in swap_device_enable(). Allocators can pick it and get -EBUSY from get_swap_device_info(). How about adding masked and then letting swap_device_enable() unmask the new swapfile?

Yeah, that's a good idea. I considered it but thought it wasn't necessary for the RFC, so I skipped that part. We can definitely do that, it will also help for some other synchronization issues that are still present.

Thanks again!

author maintainer
#14 lore raw

RFC PATCH 10/13 mm/swap: remove available list

In reply to Kairui Song

Pure cleanup, no functional change.

After the priority queue replaced the allocator's device selection, swap_avail_head and swap_avail_lock are useless now. Remove them.

__folio_throttle_swaprate() now iterates swap_active_head under swapon_rwsem instead, the SWP_WRITEOK filtering makes it equivalent to before.

---
 include/linux/swap.h |  1 -
 mm/swapfile.c        | 30 ++++--------------------------
 2 files changed, 4 insertions(+), 27 deletions(-)
include/linux/swap.h +0 -1
index 6733a8979aa2..189c9898641b 100644
@@ -293,7 +293,6 @@ struct swap_info_struct {
 	struct work_struct discard_work; /* discard worker */
 	struct work_struct reclaim_work; /* reclaim worker */
 	struct list_head discard_clusters; /* discard clusters list */
-	struct plist_node avail_list;   /* entry in swap_avail_head */
 };
 
 static inline swp_entry_t page_swap_entry(struct page *page)
mm/swapfile.c +4 -26
index f6dd85968928..01240c4c9db3 100644
@@ -96,21 +96,6 @@ static const char Bad_offset[] = "Bad swap offset entry ";
  */
 static PLIST_HEAD(swap_active_head);
 
-/*
- * all available (active, not full) swap_info_structs
- * protected with swap_avail_lock, ordered by priority.
- * This is used by folio_alloc_swap() instead of swap_active_head
- * because swap_active_head includes all swap_info_structs,
- * but folio_alloc_swap() doesn't need to look at full ones.
- * This uses its own lock instead of swapon_rwsem because when a
- * swap_info_struct changes between not-full/full, it needs to
- * add/remove itself to/from this list, but the swap_info_struct->lock
- * is held and the locking order requires swapon_rwsem to be taken
- * before any swap_info_struct->lock.
- */
-static PLIST_HEAD(swap_avail_head);
-static DEFINE_SPINLOCK(swap_avail_lock);
-
 static inline struct swap_info_struct *__swap_iter(int *i, unsigned long flag)
 {
 	lockdep_assert_held(&swapon_rwsem);
@@ -1547,7 +1532,6 @@ static void del_from_avail_list(struct swap_info_struct *si, bool swapoff)
 {
 	unsigned long pages;
 
-	spin_lock(&swap_avail_lock);
 	spin_lock(&swap_queue_update_lock);
 
 	/*
@@ -1567,10 +1551,8 @@ static void del_from_avail_list(struct swap_info_struct *si, bool swapoff)
 	}
 
 	swap_queue_mask(si);
-	plist_del(&si->avail_list, &swap_avail_head);
 skip:
 	spin_unlock(&swap_queue_update_lock);
-	spin_unlock(&swap_avail_lock);
 }
 
 /* SWAP_USAGE_OFFLIST_BIT can only be cleared by this helper. */
@@ -1579,7 +1561,6 @@ static void add_to_avail_list(struct swap_info_struct *si)
 	long val;
 	unsigned long pages;
 
-	spin_lock(&swap_avail_lock);
 	spin_lock(&swap_queue_update_lock);
 
 	/*
@@ -1610,7 +1591,6 @@ static void add_to_avail_list(struct swap_info_struct *si)
 	}
 
 	swap_queue_unmask(si);
-	plist_add(&si->avail_list, &swap_avail_head);
 skip:
 	spin_unlock(&swap_queue_update_lock);
 }
@@ -3622,7 +3602,6 @@ static struct swap_info_struct *alloc_swap_info(void)
 	}
 	p->swap_extent_root = RB_ROOT;
 	plist_node_init(&p->list, 0);
-	plist_node_init(&p->avail_list, 0);
 	p->flags = SWP_USED;
 	percpu_up_write(&swapon_rwsem);
 	if (defer) {
@@ -4030,7 +4009,6 @@ SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags)
 	 */
 	si->prio = prio;
 	si->list.prio = -si->prio;
-	si->avail_list.prio = -si->prio;
 	si->swap_file = swap_file;
 
 	/* Sets SWP_WRITEOK, resurrect the percpu ref, expose the swap device */
@@ -4163,14 +4141,14 @@ void __folio_throttle_swaprate(struct folio *folio, gfp_t gfp)
 	if (current->throttle_disk)
 		return;
 
-	spin_lock(&swap_avail_lock);
-	plist_for_each_entry(si, &swap_avail_head, avail_list) {
-		if (si->bdev) {
+	percpu_down_read(&swapon_rwsem);
+	plist_for_each_entry(si, &swap_active_head, list) {
+		if ((si->flags & SWP_WRITEOK) && si->bdev) {
 			blkcg_schedule_throttle(si->bdev->bd_disk, true);
 			break;
 		}
 	}
-	spin_unlock(&swap_avail_lock);
+	percpu_up_read(&swapon_rwsem);
 }
 #endif
 
Signature · 1 line
2.55.0
author maintainer
#15 lore raw

RFC PATCH 11/13 mm/swap: perform sync discard on single device more proactively

In reply to Kairui Song

Previous commit 9fb749cd15078 ("mm, swap: do not perform synchronous discard during allocation") added a workaround-style global sync discard when all devices are drained to prevent OOM. It wasn't an optimal solution and in the discussion [1], it's preferred to discard more proactively, and in the scope of a single device instead of globally. That wasn't achievable due to the limitations of the previous swap allocation design, or would have ended up too complex.

Now with the new workflow and swap cluster cache moved inside the device scope, we can easily achieve that. So drop the old workaround and implement a more proactive and simpler solution. Doing discard when the free list is drained will prevent fragmentation better and avoid a global discard scan.

---
 mm/swapfile.c | 88 ++++++++++++++++++++++-------------------------------------
 1 file changed, 33 insertions(+), 55 deletions(-)
mm/swapfile.c +33 -55
index 01240c4c9db3..74a50c39da60 100644
@@ -498,6 +498,26 @@ static long swap_usage_in_pages(struct swap_info_struct *si)
 	return atomic_long_read(&si->inuse_pages) & SWAP_USAGE_COUNTER_MASK;
 }
 
+/*
+ * Serialize the allocation on single CPU or globally to avoid
+ * fragmentation and make the workflow easier to follow.
+ */
+static void swap_alloc_lock_device(struct swap_info_struct *si)
+{
+	if (si->flags & SWP_SOLIDSTATE)
+		local_lock(&si->percpu_cluster->lock);
+	else
+		spin_lock(&si->global_cluster->lock);
+}
+
+static void swap_alloc_unlock_device(struct swap_info_struct *si)
+{
+	if (si->flags & SWP_SOLIDSTATE)
+		local_unlock(&si->percpu_cluster->lock);
+	else
+		spin_unlock(&si->global_cluster->lock);
+}
+
 /* Reclaim the swap entry anyway if possible */
 #define TTRS_ANYWAY		0x1
 /*
@@ -884,10 +904,7 @@ swap_cluster_populate(struct swap_info_struct *si,
 	 * the potential recursive allocation is limited.
 	 */
 	spin_unlock(&ci->lock);
-	if (si->flags & SWP_SOLIDSTATE)
-		local_unlock(&si->percpu_cluster->lock);
-	else
-		spin_unlock(&si->global_cluster->lock);
+	swap_alloc_unlock_device(si);
 
 	ret = swap_cluster_alloc_table(ci, __GFP_HIGH | __GFP_NOMEMALLOC |
 					   GFP_KERNEL);
@@ -900,10 +917,7 @@ swap_cluster_populate(struct swap_info_struct *si,
 	 * could happen with ignoring the percpu cluster is fragmentation,
 	 * which is acceptable since this fallback and race is rare.
 	 */
-	if (si->flags & SWP_SOLIDSTATE)
-		local_lock(&si->percpu_cluster->lock);
-	else
-		spin_lock(&si->global_cluster->lock);
+	swap_alloc_lock_device(si);
 	spin_lock(&ci->lock);
 
 	if (ret) {
@@ -1439,15 +1453,12 @@ static unsigned long cluster_alloc_swap_entry(struct swap_info_struct *si,
 	if (order && !(si->flags & SWP_BLKDEV))
 		return 0;
 
-	if (si->flags & SWP_SOLIDSTATE) {
-		/* Fast path using per CPU cluster */
-		local_lock(&si->percpu_cluster->lock);
+restart:
+	swap_alloc_lock_device(si);
+	if (si->flags & SWP_SOLIDSTATE)
 		offset = __this_cpu_read(si->percpu_cluster->next[order]);
-	} else {
-		/* Serialize HDD SWAP allocation for each device. */
-		spin_lock(&si->global_cluster->lock);
+	else
 		offset = si->global_cluster->next[order];
-	}
 
 	if (offset != SWAP_ENTRY_INVALID) {
 		ci = swap_cluster_lock(si, offset);
@@ -1471,6 +1482,12 @@ static unsigned long cluster_alloc_swap_entry(struct swap_info_struct *si,
 		found = alloc_swap_scan_list(si, &si->free_clusters, folio, false);
 		if (found)
 			goto done;
+
+		if (!list_empty(&si->discard_clusters)) {
+			swap_alloc_unlock_device(si);
+			swap_do_scheduled_discard(si);
+			goto restart;
+		}
 	}
 
 	if (order < PMD_ORDER) {
@@ -1519,10 +1536,7 @@ static unsigned long cluster_alloc_swap_entry(struct swap_info_struct *si,
 			goto done;
 	}
 done:
-	if (si->flags & SWP_SOLIDSTATE)
-		local_unlock(&si->percpu_cluster->lock);
-	else
-		spin_unlock(&si->global_cluster->lock);
+	swap_alloc_unlock_device(si);
 
 	return found;
 }
@@ -1711,36 +1725,6 @@ static int swap_alloc_entry(struct folio *folio)
 	return ret;
 }
 
-/*
- * Discard pending clusters in a synchronized way when under high pressure.
- * Return: true if any cluster is discarded.
- */
-static bool swap_sync_discard(void)
-{
-	bool ret = false;
-	struct swap_info_struct *si, *next;
-
-	percpu_down_read(&swapon_rwsem);
-start_over:
-	plist_for_each_entry_safe(si, next, &swap_active_head, list) {
-		percpu_up_read(&swapon_rwsem);
-		if (get_swap_device_info(si)) {
-			if (si->flags & SWP_PAGE_DISCARD)
-				ret = swap_do_scheduled_discard(si);
-			put_swap_device(si);
-		}
-		if (ret)
-			return true;
-
-		percpu_down_read(&swapon_rwsem);
-		if (plist_node_empty(&next->list))
-			goto start_over;
-	}
-	percpu_up_read(&swapon_rwsem);
-
-	return false;
-}
-
 static int swap_extend_table_alloc(struct swap_info_struct *si,
 				   struct swap_cluster_info *ci,
 				   unsigned int ci_off, gfp_t gfp)
@@ -2045,14 +2029,8 @@ int folio_alloc_swap(struct folio *folio)
 		}
 	}
 
-again:
 	ret = swap_alloc_entry(folio);
 
-	if (!order && unlikely(!folio_test_swapcache(folio))) {
-		if (swap_sync_discard())
-			goto again;
-	}
-
 	/* Need to call this even if allocation failed, for MEMCG_SWAP_FAIL. */
 	if (unlikely(mem_cgroup_try_charge_swap(folio)))
 		swap_cache_del_folio(folio);
Signature · 1 line
2.55.0
author maintainer
#16 lore raw

RFC PATCH 12/13 mm/swap: drop swap active plist

In reply to Kairui Song

The swap_active_head plist tracked all writable swap devices, ordered by priority. Now that every consumer that needed priority ordering has moved to the priority queue, the remaining users only need to find any writable device.

Remove swap_active_head, the associated plist_node from swap_info_struct, and migrate the remaining plist operations.

---
 include/linux/swap.h |  1 -
 mm/swapfile.c        | 26 +++++---------------------
 2 files changed, 5 insertions(+), 22 deletions(-)
include/linux/swap.h +0 -1
index 189c9898641b..04767540972b 100644
@@ -267,7 +267,6 @@ struct swap_info_struct {
 	struct percpu_ref users;	/* indicate and keep swap device valid. */
 	unsigned long	flags;		/* SWP_USED etc: see above */
 	signed short	prio;		/* swap priority of this type */
-	struct plist_node list;		/* entry in swap_active_head */
 	signed char	type;		/* strange name for an index */
 	unsigned int	max;		/* size of this swap device */
 	struct swap_cluster_info *cluster_info; /* cluster info. Only for SSD */
mm/swapfile.c +5 -21
index 74a50c39da60..f93863309a5a 100644
@@ -41,7 +41,6 @@
 #include <linux/completion.h>
 #include <linux/suspend.h>
 #include <linux/zswap.h>
-#include <linux/plist.h>
 
 #include <asm/tlbflush.h>
 #include <linux/leafops.h>
@@ -90,12 +89,6 @@ bool swap_migration_ad_supported;
 static const char Bad_file[] = "Bad swap file entry ";
 static const char Bad_offset[] = "Bad swap offset entry ";
 
-/*
- * all active swap_info_structs
- * protected with swapon_rwsem, and ordered by priority.
- */
-static PLIST_HEAD(swap_active_head);
-
 static inline struct swap_info_struct *__swap_iter(int *i, unsigned long flag)
 {
 	lockdep_assert_held(&swapon_rwsem);
@@ -3212,7 +3205,6 @@ static void swap_device_enable(struct swap_info_struct *si)
 	spin_unlock(&swap_queue_update_lock);
 	atomic_long_add(si->pages, &nr_swap_pages);
 	total_swap_pages += si->pages;
-	plist_add(&si->list, &swap_active_head);
 	percpu_up_write(&swapon_rwsem);
 
 	add_to_avail_list(si);
@@ -3244,7 +3236,6 @@ static int swap_device_disable(struct swap_info_struct *si)
 	spin_lock(&swap_queue_update_lock);
 	si->flags &= ~SWP_WRITEOK;
 	spin_unlock(&swap_queue_update_lock);
-	plist_del(&si->list, &swap_active_head);
 	total_swap_pages -= si->pages;
 	atomic_long_sub(si->pages, &nr_swap_pages);
 	del_from_avail_list(si, true);
@@ -3306,9 +3297,8 @@ SYSCALL_DEFINE1(swapoff, const char __user *, specialfile)
 
 	mapping = victim->f_mapping;
 	percpu_down_read(&swapon_rwsem);
-	plist_for_each_entry(p, &swap_active_head, list) {
-		if (p->flags & SWP_WRITEOK &&
-		    p->swap_file->f_mapping == mapping) {
+	for_each_avail_swap(p) {
+		if (p->swap_file->f_mapping == mapping) {
 			found = 1;
 			break;
 		}
@@ -3579,7 +3569,6 @@ static struct swap_info_struct *alloc_swap_info(void)
 		 */
 	}
 	p->swap_extent_root = RB_ROOT;
-	plist_node_init(&p->list, 0);
 	p->flags = SWP_USED;
 	percpu_up_write(&swapon_rwsem);
 	if (defer) {
@@ -3981,12 +3970,7 @@ SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags)
 	if (swap_flags & SWAP_FLAG_PREFER)
 		prio = swap_flags & SWAP_FLAG_PRIO_MASK;
 
-	/*
-	 * The plist prio is negated because plist ordering is
-	 * low-to-high, while swap ordering is high-to-low
-	 */
 	si->prio = prio;
-	si->list.prio = -si->prio;
 	si->swap_file = swap_file;
 
 	/* Sets SWP_WRITEOK, resurrect the percpu ref, expose the swap device */
@@ -4096,7 +4080,7 @@ int swap_dup_entry_direct(swp_entry_t entry)
 #if defined(CONFIG_MEMCG) && defined(CONFIG_BLK_CGROUP)
 static bool __has_usable_swap(void)
 {
-	return !plist_head_empty(&swap_active_head);
+	return READ_ONCE(total_swap_pages) > 0;
 }
 
 void __folio_throttle_swaprate(struct folio *folio, gfp_t gfp)
@@ -4120,8 +4104,8 @@ void __folio_throttle_swaprate(struct folio *folio, gfp_t gfp)
 		return;
 
 	percpu_down_read(&swapon_rwsem);
-	plist_for_each_entry(si, &swap_active_head, list) {
-		if ((si->flags & SWP_WRITEOK) && si->bdev) {
+	for_each_avail_swap(si) {
+		if (si->bdev) {
 			blkcg_schedule_throttle(si->bdev->bd_disk, true);
 			break;
 		}
Signature · 1 line
2.55.0
author maintainer
#17 lore raw

RFC PATCH 13/13 lib/plist.c: remove requeue function

In reply to Kairui Song

Now the last user of plist requeue is gone, this function can be removed.

---
 include/linux/plist.h |  2 --
 lib/plist.c           | 64 ---------------------------------------------------
 2 files changed, 66 deletions(-)
include/linux/plist.h +0 -2
index 16cf4355b5c1..73d359c1a053 100644
@@ -132,8 +132,6 @@ static inline void plist_node_init(struct plist_node *node, int prio)
 extern void plist_add(struct plist_node *node, struct plist_head *head);
 extern void plist_del(struct plist_node *node, struct plist_head *head);
 
-extern void plist_requeue(struct plist_node *node, struct plist_head *head);
-
 /**
  * plist_for_each - iterate over the plist
  * @pos:	the type * to use as a loop counter
lib/plist.c +0 -64

82 lines not on this page · raw message

Signature · 1 line
2.55.0