Skip to contentSkip to conversation

mm, swap: rework of swap allocator locks

PATCH Superseded Kairui Song 2 participants 16 messages Dec 24, 2024

Subsystems (4)

v2 of 4 Superseded Latest is v4 →

lore
Share & tools
lore
#0 lore raw

v2 cover mm, swap: rework of swap allocator locks

From: Kairui Song <[email protected]>

This series greatly improved swap performance by reworking the locking design and simplify a lot of code path. Test showed a up to 400% vm-scalability improvement with pmem as SWAP, and up to 37% reduce of kernel compile real time with ZRAM as SWAP (up to 60% improvement in system time).

This is part of the new swap allocator discussed during the "Swap Abstraction" discussion at LSF/MM 2024, and "mTHP and swap allocator" discussion at LPC 2024.

This is a follow up of previous swap cluster allocator series:
https://lore.kernel.org/linux-mm/[email protected]/
Also enables further optimizations which will come later.

Previous series introduced a fully cluster based allocator, this series completely get rid of the old allocator and makes the new allocator avoid touching the si->lock unless needed. This bring huge performance gain and get rid of slot cache for freeing path.

Currently, swap locking is mainly composed of two locks, cluster lock (ci->lock) and device lock (si->lock). The device lock is widely used to protect many things, causing it to be the main bottleneck for SWAP.

Cluster lock is much more fine-grained, so it will be best to use ci->lock instead of si->lock as much as possible.

perf lock indicates this issue clearly. Doing linux kernel build using tmpfs and ZRAM with limited memory (make -j64 with 1G memcg and 4k pages), result of "perf lock contention -ab sleep 3" shows:

  contended   total wait     max wait     avg wait         type   caller
     34948     53.63 s       7.11 ms      1.53 ms     spinlock   free_swap_and_cache_nr+0x350
     16569     40.05 s       6.45 ms      2.42 ms     spinlock   get_swap_pages+0x231
     11191     28.41 s       7.03 ms      2.54 ms     spinlock   swapcache_free_entries+0x59
      4147     22.78 s     122.66 ms      5.49 ms     spinlock   page_vma_mapped_walk+0x6f3
      4595      7.17 s       6.79 ms      1.56 ms     spinlock   swapcache_free_entries+0x59
    406027      2.74 s       2.59 ms      6.74 us     spinlock   list_lru_add+0x39

...snip...

The top 5 caller are all users of si->lock, total wait time sums to several minutes in the 3 seconds time window.

Following the new allocator design, many operation doesn't need to touch si->lock at all. We only need to take si->lock when doing operations across multiple clusters (changing the cluster list). So ideally allocator should always take ci->lock first, then take si->lock only if needed. But due to historical reasons, ci->lock is used inside si->lock critical section, causing lock inversion if we simply try to acquire si->lock after acquiring ci->lock.

This series audited all si->lock usage, clean up legacy codes, eliminate usage of si->lock as much as possible by introducing new designs based on the new cluster allocator.

Old HDD allocation codes are removed, cluster allocator is adapted with small changes for HDD usage, test is looking OK.

And this also removed slot cache for freeing path. The performance is even better without it now, and this enables other clean up and optimizations as discussed before:

https://lore.kernel.org/all/CAMgjq7ACohT_uerSz8E_994ZZCv709Zor+43hdmesW_59W1BWw@mail.gmail.com/

After this series, lock contention on si->lock is nearly unobservable with perf lock with the same test above:

contended total wait max wait avg wait type caller
... snip ...

         91    204.62 us      4.51 us      2.25 us     spinlock   cluster_move+0x2e
  ... snip ...
         47    125.62 us      4.47 us      2.67 us     spinlock   cluster_move+0x2e
  ... snip ...
         23     63.15 us      3.95 us      2.74 us     spinlock   cluster_move+0x2e
  ... snip ...
         17     41.26 us      4.58 us      2.43 us     spinlock   cluster_isolate_lock+0x1d

... snip ...

cluster_move and cluster_isolate_lock (two new introduced helper) are basically the only users of si->lock now, performance gain is huge, and LOC is reduced.

Tests Results:

vm-scalability
==============
Running usemem --init-time -O -y -x -R -31 1G from vm-scalability in a 12G memory cgroup using simulated pmem as SWAP backend (32G pmem, 32 CPUs).

Using 4K folio by default, 64k mTHP and sequential access (!-R) results are also provided. 6 test runs for each case, Total Throughput:

Test Before (KB/s) (stdev) After (KB/s) (stdev) Delta
---------------------------------------------------------------------------

Random (4K):     69937.11 (16449.77)    369816.17  (24476.68)  +428.78%
Random (64k):    123442.83 (13207.51)   216379.00  (25024.83)  +75.28%
Sequential (4K): 6313909.83 (148856.12) 6419860.66 (183563.38) +1.7%

Sequential access will cause lower stress for the allocator so the gain is limited, but with random access (which is much closer to real workloads) the performance gain is huge.

Build kernel with defconfig on tmpfs with ZRAM
==============================================
Below results shows a test matrix using different memory cgroup limit and job numbets, and scaled up progressive for a intuitive result. Done on a 48c96t system.

6 test run for each case, it can be seen clearly that as concurrent job number goes higher the performance gain is higher, but even -j6 is showing slight improvement.

   make -j<NR>     |   System Time (seconds)  |   Total Time (seconds)
 (NR / Mem / ZRAM) | (Before / After / Delta) | (Before / After / Delta)
 With 4k pages only:
  6 / 192M / 3G    |    1533 /  1522 / -0.7%  |    1420 /  1414 / -0.3%
 12 / 256M / 4G    |    2275 /  2226 / -2.2%  |     758 /   742 / -2.1%
 24 / 384M / 5G    |    3596 /  3154 / -12.3% |     476 /   422 / -11.3%
 48 / 768M / 7G    |    8159 /  3605 / -55.8% |     330 /   221 / -33.0%
 96 / 1.5G / 10G   |   18541 /  6462 / -65.1% |     283 /   180 / -36.4%
 With 64k mTHP:
 24 / 512M / 5G    |    3585 /  3469 /  -3.2% |     293 /   290 / -0.1%
 48 /   1G / 7G    |    8173 /  3607 / -55.9% |     251 /   158 / -37.0%
 96 /   2G / 10G   |   16305 /  7791 / -52.2% |     226 /   144 / -36.3%

The fragmentation are reduced too:
With: make -j96 / 1152M memcg, 64K mTHP:
(avg of 4 test run)
Before:
hugepages-64kB/stats/swpout: 1696184
hugepages-64kB/stats/swpout_fallback: 414318
After: (-63.2% mTHP swapout failure)
hugepages-64kB/stats/swpout: 1866267
hugepages-64kB/stats/swpout_fallback: 158330

There is a up to 65.1% improvement in sys time for build kernel test, and lower fragmentation rate.

Build kernel with tinyconfig on tmpfs with HDD as swap:
=======================================================

This test is similar to above, but HDD test is very noisy and slow, the deviation is huge, so just use tinyconfig instead and take the median test result of 3 test run, which looks OK:

Before this series:
114.44user 29.11system 39:42.90elapsed 6%CPU 2901232inputs+0outputs (238877major+4227640minor)pagefaults

After this commit:
113.90user 23.81system 38:11.77elapsed 6%CPU 2548728inputs+0outputs (235471major+4238110minor)pagefaults

Single thread SWAP:
===================

Sequential SWAP should also be slightly faster as we removed a lot of unnecessary parts. Test using micro benchmark for swapout/in 4G zero memory using ZRAM, 10 test runs:

Swapout Before (avg. 3359304):
3353796 3358551 3371305 3356043 3367524 3355303 3355924 3354513 3360776

Swapin Before (avg. 1928698):
1920283 1927183 1934105 1921373 1926562 1938261 1927726 1928636 1934155

Swapout After (avg. 3347511, -0.4%):
3337863 3347948 3355235 3339081 3333134 3353006 3354917 3346055 3360359

Swapin After (avg. 1922290, -0.3%):
1919101 1925743 1916810 1917007 1923930 1935152 1917403 1923549 1921913

The gain is limited at noise level but seems slightly better.

V1: https://lore.kernel.org/linux-mm/[email protected]/
Updates:
- Retest some tests after rebase on top of latest mm-unstable, the new Cgroup lock removal increased the performance gain of this series too, some results are basically same as before so unchanged:
https://lore.kernel.org/linux-mm/[email protected]/
- Rework the off-list bit handling, make it easier to review and more robust, also reduce LOC [Chris Li].
- Code style improvements and minor code optimizations. [Chris Li].
- Fixing a potential swapoff race issue due to missing SWP_WRITEOK check [Huang Ying].
- Added vm-scalability test with pmem [Huang Ying].

Kairui Song (13):
mm, swap: minor clean up for swap entry allocation mm, swap: fold swap_info_get_cont in the only caller mm, swap: remove old allocation path for HDD
mm, swap: use cluster lock for HDD
mm, swap: clean up device availability check
mm, swap: clean up plist removal and adding
mm, swap: hold a reference during scan and cleanup flag usage mm, swap: use an enum to define all cluster flags and wrap flags
changes
mm, swap: reduce contention on device lock
mm, swap: simplify percpu cluster updating
mm, swap: introduce a helper for retrieving cluster from offset mm, swap: use a global swap cluster for non-rotation devices mm, swap_slots: remove slot cache for freeing path

 fs/btrfs/inode.c           |    1 -
 fs/iomap/swapfile.c        |    1 -
 include/linux/swap.h       |   34 +-
 include/linux/swap_slots.h |    3 -
 mm/page_io.c               |    1 -
 mm/swap_slots.c            |   78 +--
 mm/swapfile.c              | 1246 ++++++++++++++++--------------------
 7 files changed, 591 insertions(+), 773 deletions(-)
Signature · 1 line
2.47.1
#1 lore raw

PATCH v2 1/13 mm, swap: minor clean up for swap entry allocation

From: Kairui Song <[email protected]>

Direct reclaim can skip the whole folio after reclaimed a set of folio based slots. Also simplify the code for allocation, reduce indention.

---
 mm/swapfile.c | 59 +++++++++++++++++++++++++--------------------------
 1 file changed, 29 insertions(+), 30 deletions(-)
mm/swapfile.c +29 -30
index b0a9071cfe1d..f8002f110104 100644
@@ -604,23 +604,28 @@ static bool cluster_reclaim_range(struct swap_info_struct *si,
 				  unsigned long start, unsigned long end)
 {
 	unsigned char *map = si->swap_map;
-	unsigned long offset;
+	unsigned long offset = start;
+	int nr_reclaim;
 
 	spin_unlock(&ci->lock);
 	spin_unlock(&si->lock);
 
-	for (offset = start; offset < end; offset++) {
+	do {
 		switch (READ_ONCE(map[offset])) {
 		case 0:
-			continue;
+			offset++;
+			break;
 		case SWAP_HAS_CACHE:
-			if (__try_to_reclaim_swap(si, offset, TTRS_ANYWAY | TTRS_DIRECT) > 0)
-				continue;
-			goto out;
+			nr_reclaim = __try_to_reclaim_swap(si, offset, TTRS_ANYWAY | TTRS_DIRECT);
+			if (nr_reclaim > 0)
+				offset += nr_reclaim;
+			else
+				goto out;
+			break;
 		default:
 			goto out;
 		}
-	}
+	} while (offset < end);
 out:
 	spin_lock(&si->lock);
 	spin_lock(&ci->lock);
@@ -838,35 +843,30 @@ static unsigned long cluster_alloc_swap_entry(struct swap_info_struct *si, int o
 							 &found, order, usage);
 			frags++;
 			if (found)
-				break;
+				goto done;
 		}
 
-		if (!found) {
+		/*
+		 * Nonfull clusters are moved to frag tail if we reached
+		 * here, count them too, don't over scan the frag list.
+		 */
+		while (frags < si->frag_cluster_nr[order]) {
+			ci = list_first_entry(&si->frag_clusters[order],
+					      struct swap_cluster_info, list);
 			/*
-			 * Nonfull clusters are moved to frag tail if we reached
-			 * here, count them too, don't over scan the frag list.
+			 * Rotate the frag list to iterate, they were all failing
+			 * high order allocation or moved here due to per-CPU usage,
+			 * this help keeping usable cluster ahead.
 			 */
-			while (frags < si->frag_cluster_nr[order]) {
-				ci = list_first_entry(&si->frag_clusters[order],
-						      struct swap_cluster_info, list);
-				/*
-				 * Rotate the frag list to iterate, they were all failing
-				 * high order allocation or moved here due to per-CPU usage,
-				 * this help keeping usable cluster ahead.
-				 */
-				list_move_tail(&ci->list, &si->frag_clusters[order]);
-				offset = alloc_swap_scan_cluster(si, cluster_offset(si, ci),
-								 &found, order, usage);
-				frags++;
-				if (found)
-					break;
-			}
+			list_move_tail(&ci->list, &si->frag_clusters[order]);
+			offset = alloc_swap_scan_cluster(si, cluster_offset(si, ci),
+							 &found, order, usage);
+			frags++;
+			if (found)
+				goto done;
 		}
 	}
 
-	if (found)
-		goto done;
-
 	if (!list_empty(&si->discard_clusters)) {
 		/*
 		 * we don't have free cluster but have some clusters in
@@ -904,7 +904,6 @@ static unsigned long cluster_alloc_swap_entry(struct swap_info_struct *si, int o
 				goto done;
 		}
 	}
-
 done:
 	cluster->next[order] = offset;
 	return found;
Signature · 1 line
2.47.1
#2 lore raw

PATCH v2 2/13 mm, swap: fold swap_info_get_cont in the only caller

In reply to Kairui Song

From: Kairui Song <[email protected]>

The name of the function is confusing, and the code is much easier to follow after folding, also rename the confusing naming "p" to more meaningful "si".

---
 mm/swapfile.c | 39 +++++++++++++++------------------------
 1 file changed, 15 insertions(+), 24 deletions(-)
mm/swapfile.c +15 -24
index f8002f110104..574059158627 100644
@@ -1375,22 +1375,6 @@ static struct swap_info_struct *_swap_info_get(swp_entry_t entry)
 	return NULL;
 }
 
-static struct swap_info_struct *swap_info_get_cont(swp_entry_t entry,
-					struct swap_info_struct *q)
-{
-	struct swap_info_struct *p;
-
-	p = _swap_info_get(entry);
-
-	if (p != q) {
-		if (q != NULL)
-			spin_unlock(&q->lock);
-		if (p != NULL)
-			spin_lock(&p->lock);
-	}
-	return p;
-}
-
 static unsigned char __swap_entry_free_locked(struct swap_info_struct *si,
 					      unsigned long offset,
 					      unsigned char usage)
@@ -1687,14 +1671,14 @@ static int swp_entry_cmp(const void *ent1, const void *ent2)
 
 void swapcache_free_entries(swp_entry_t *entries, int n)
 {
-	struct swap_info_struct *p, *prev;
+	struct swap_info_struct *si, *prev;
 	int i;
 
 	if (n <= 0)
 		return;
 
 	prev = NULL;
-	p = NULL;
+	si = NULL;
 
 	/*
 	 * Sort swap entries by swap device, so each lock is only taken once.
@@ -1704,13 +1688,20 @@ void swapcache_free_entries(swp_entry_t *entries, int n)
 	if (nr_swapfiles > 1)
 		sort(entries, n, sizeof(entries[0]), swp_entry_cmp, NULL);
 	for (i = 0; i < n; ++i) {
-		p = swap_info_get_cont(entries[i], prev);
-		if (p)
-			swap_entry_range_free(p, entries[i], 1);
-		prev = p;
+		si = _swap_info_get(entries[i]);
+
+		if (si != prev) {
+			if (prev != NULL)
+				spin_unlock(&prev->lock);
+			if (si != NULL)
+				spin_lock(&si->lock);
+		}
+		if (si)
+			swap_entry_range_free(si, entries[i], 1);
+		prev = si;
 	}
-	if (p)
-		spin_unlock(&p->lock);
+	if (si)
+		spin_unlock(&si->lock);
 }
 
 int __swap_count(swp_entry_t entry)
Signature · 1 line
2.47.1
#3 lore raw

PATCH v2 3/13 mm, swap: remove old allocation path for HDD

In reply to Kairui Song

From: Kairui Song <[email protected]>

We are currently using different swap allocation algorithm for HDD and non-HDD. This leads to the existence of a different set of locks, and the code path is heavily bloated, causing difficulties for further optimization and maintenance.

This commit removes all HDD swap allocation and related dead code, and uses the cluster allocation algorithm instead.

The performance may drop temporarily, but this should be negligible:
The main advantage of the legacy HDD allocation algorithm is that it tends to use continuous slots, but swap device gets fragmented quickly anyway, and the attempt to use continuous slots will fail easily.

This commit also enables mTHP swap on HDD, which is expected to be beneficial, and following commits will adapt and optimize the cluster allocator for HDD.

---
 include/linux/swap.h |   3 -
 mm/swapfile.c        | 235 ++-----------------------------------------
 2 files changed, 9 insertions(+), 229 deletions(-)
include/linux/swap.h +0 -3
index 187715eec3cb..0c681aa5cb98 100644
@@ -310,9 +310,6 @@ struct swap_info_struct {
 	unsigned int highest_bit;	/* index of last free in swap_map */
 	unsigned int pages;		/* total of usable pages of swap */
 	unsigned int inuse_pages;	/* number of those currently in use */
-	unsigned int cluster_next;	/* likely index for next allocation */
-	unsigned int cluster_nr;	/* countdown to next cluster search */
-	unsigned int __percpu *cluster_next_cpu; /*percpu index for next allocation */
 	struct percpu_cluster __percpu *percpu_cluster; /* per cpu's swap location */
 	struct rb_root swap_extent_root;/* root of the swap extent rbtree */
 	struct block_device *bdev;	/* swap device or bdev of swap file */
mm/swapfile.c +9 -226

307 lines not on this page · raw message

Signature · 1 line
2.47.1
#4 lore raw

PATCH v2 4/13 mm, swap: use cluster lock for HDD

In reply to Kairui Song

From: Kairui Song <[email protected]>

Cluster lock (ci->lock) was introduce to reduce contention for certain operations. Using cluster lock for HDD is not helpful as HDD have a poor performance, so locking isn't the bottleneck. But having different set of locks for HDD / non-HDD prevents further rework of device lock (si->lock).

This commit just changed all lock_cluster_or_swap_info to lock_cluster, which is a safe and straight conversion since cluster info is always allocated now, also removed all cluster_info related checks.

---
 mm/swapfile.c | 107 ++++++++++++++++----------------------------------
 1 file changed, 34 insertions(+), 73 deletions(-)
mm/swapfile.c +34 -73
index fca58d43b836..d0e5b9fa0c48 100644
@@ -58,10 +58,9 @@ static void swap_entry_range_free(struct swap_info_struct *si, swp_entry_t entry
 static void swap_range_alloc(struct swap_info_struct *si, unsigned long offset,
 			     unsigned int nr_entries);
 static bool folio_swapcache_freeable(struct folio *folio);
-static struct swap_cluster_info *lock_cluster_or_swap_info(
-		struct swap_info_struct *si, unsigned long offset);
-static void unlock_cluster_or_swap_info(struct swap_info_struct *si,
-					struct swap_cluster_info *ci);
+static struct swap_cluster_info *lock_cluster(struct swap_info_struct *si,
+					      unsigned long offset);
+static void unlock_cluster(struct swap_cluster_info *ci);
 
 static DEFINE_SPINLOCK(swap_lock);
 static unsigned int nr_swapfiles;
@@ -222,9 +221,9 @@ static int __try_to_reclaim_swap(struct swap_info_struct *si,
 	 * swap_map is HAS_CACHE only, which means the slots have no page table
 	 * reference or pending writeback, and can't be allocated to others.
 	 */
-	ci = lock_cluster_or_swap_info(si, offset);
+	ci = lock_cluster(si, offset);
 	need_reclaim = swap_is_has_cache(si, offset, nr_pages);
-	unlock_cluster_or_swap_info(si, ci);
+	unlock_cluster(ci);
 	if (!need_reclaim)
 		goto out_unlock;
 
@@ -404,45 +403,15 @@ static inline struct swap_cluster_info *lock_cluster(struct swap_info_struct *si
 {
 	struct swap_cluster_info *ci;
 
-	ci = si->cluster_info;
-	if (ci) {
-		ci += offset / SWAPFILE_CLUSTER;
-		spin_lock(&ci->lock);
-	}
-	return ci;
-}
-
-static inline void unlock_cluster(struct swap_cluster_info *ci)
-{
-	if (ci)
-		spin_unlock(&ci->lock);
-}
-
-/*
- * Determine the locking method in use for this device.  Return
- * swap_cluster_info if SSD-style cluster-based locking is in place.
- */
-static inline struct swap_cluster_info *lock_cluster_or_swap_info(
-		struct swap_info_struct *si, unsigned long offset)
-{
-	struct swap_cluster_info *ci;
-
-	/* Try to use fine-grained SSD-style locking if available: */
-	ci = lock_cluster(si, offset);
-	/* Otherwise, fall back to traditional, coarse locking: */
-	if (!ci)
-		spin_lock(&si->lock);
+	ci = &si->cluster_info[offset / SWAPFILE_CLUSTER];
+	spin_lock(&ci->lock);
 
 	return ci;
 }
 
-static inline void unlock_cluster_or_swap_info(struct swap_info_struct *si,
-					       struct swap_cluster_info *ci)
+static inline void unlock_cluster(struct swap_cluster_info *ci)
 {
-	if (ci)
-		unlock_cluster(ci);
-	else
-		spin_unlock(&si->lock);
+	spin_unlock(&ci->lock);
 }
 
 /* Add a cluster to discard list and schedule it to do discard */
@@ -558,9 +527,6 @@ static void inc_cluster_info_page(struct swap_info_struct *si,
 	unsigned long idx = page_nr / SWAPFILE_CLUSTER;
 	struct swap_cluster_info *ci;
 
-	if (!cluster_info)
-		return;
-
 	ci = cluster_info + idx;
 	ci->count++;
 
@@ -576,9 +542,6 @@ static void inc_cluster_info_page(struct swap_info_struct *si,
 static void dec_cluster_info_page(struct swap_info_struct *si,
 				  struct swap_cluster_info *ci, int nr_pages)
 {
-	if (!si->cluster_info)
-		return;
-
 	VM_BUG_ON(ci->count < nr_pages);
 	VM_BUG_ON(cluster_is_free(ci));
 	lockdep_assert_held(&si->lock);
@@ -1007,8 +970,6 @@ static int cluster_alloc_swap(struct swap_info_struct *si,
 {
 	int n_ret = 0;
 
-	VM_BUG_ON(!si->cluster_info);
-
 	si->flags += SWP_SCANNING;
 
 	while (n_ret < nr) {
@@ -1052,10 +1013,10 @@ static int scan_swap_map_slots(struct swap_info_struct *si,
 		}
 
 		/*
-		 * Swapfile is not block device or not using clusters so unable
+		 * Swapfile is not block device so unable
 		 * to allocate large entries.
 		 */
-		if (!(si->flags & SWP_BLKDEV) || !si->cluster_info)
+		if (!(si->flags & SWP_BLKDEV))
 			return 0;
 	}
 
@@ -1295,9 +1256,9 @@ static unsigned char __swap_entry_free(struct swap_info_struct *si,
 	unsigned long offset = swp_offset(entry);
 	unsigned char usage;
 
-	ci = lock_cluster_or_swap_info(si, offset);
+	ci = lock_cluster(si, offset);
 	usage = __swap_entry_free_locked(si, offset, 1);
-	unlock_cluster_or_swap_info(si, ci);
+	unlock_cluster(ci);
 	if (!usage)
 		free_swap_slot(entry);
 
@@ -1320,14 +1281,14 @@ static bool __swap_entries_free(struct swap_info_struct *si,
 	if (nr > SWAPFILE_CLUSTER - offset % SWAPFILE_CLUSTER)
 		goto fallback;
 
-	ci = lock_cluster_or_swap_info(si, offset);
+	ci = lock_cluster(si, offset);
 	if (!swap_is_last_map(si, offset, nr, &has_cache)) {
-		unlock_cluster_or_swap_info(si, ci);
+		unlock_cluster(ci);
 		goto fallback;
 	}
 	for (i = 0; i < nr; i++)
 		WRITE_ONCE(si->swap_map[offset + i], SWAP_HAS_CACHE);
-	unlock_cluster_or_swap_info(si, ci);
+	unlock_cluster(ci);
 
 	if (!has_cache) {
 		for (i = 0; i < nr; i++)
@@ -1383,7 +1344,7 @@ static void cluster_swap_free_nr(struct swap_info_struct *si,
 	DECLARE_BITMAP(to_free, BITS_PER_LONG) = { 0 };
 	int i, nr;
 
-	ci = lock_cluster_or_swap_info(si, offset);
+	ci = lock_cluster(si, offset);
 	while (nr_pages) {
 		nr = min(BITS_PER_LONG, nr_pages);
 		for (i = 0; i < nr; i++) {
@@ -1391,18 +1352,18 @@ static void cluster_swap_free_nr(struct swap_info_struct *si,
 				bitmap_set(to_free, i, 1);
 		}
 		if (!bitmap_empty(to_free, BITS_PER_LONG)) {
-			unlock_cluster_or_swap_info(si, ci);
+			unlock_cluster(ci);
 			for_each_set_bit(i, to_free, BITS_PER_LONG)
 				free_swap_slot(swp_entry(si->type, offset + i));
 			if (nr == nr_pages)
 				return;
 			bitmap_clear(to_free, 0, BITS_PER_LONG);
-			ci = lock_cluster_or_swap_info(si, offset);
+			ci = lock_cluster(si, offset);
 		}
 		offset += nr;
 		nr_pages -= nr;
 	}
-	unlock_cluster_or_swap_info(si, ci);
+	unlock_cluster(ci);
 }
 
 /*
@@ -1441,9 +1402,9 @@ void put_swap_folio(struct folio *folio, swp_entry_t entry)
 	if (!si)
 		return;
 
-	ci = lock_cluster_or_swap_info(si, offset);
+	ci = lock_cluster(si, offset);
 	if (size > 1 && swap_is_has_cache(si, offset, size)) {
-		unlock_cluster_or_swap_info(si, ci);
+		unlock_cluster(ci);
 		spin_lock(&si->lock);
 		swap_entry_range_free(si, entry, size);
 		spin_unlock(&si->lock);
@@ -1451,14 +1412,14 @@ void put_swap_folio(struct folio *folio, swp_entry_t entry)
 	}
 	for (int i = 0; i < size; i++, entry.val++) {
 		if (!__swap_entry_free_locked(si, offset + i, SWAP_HAS_CACHE)) {
-			unlock_cluster_or_swap_info(si, ci);
+			unlock_cluster(ci);
 			free_swap_slot(entry);
 			if (i == size - 1)
 				return;
-			lock_cluster_or_swap_info(si, offset);
+			lock_cluster(si, offset);
 		}
 	}
-	unlock_cluster_or_swap_info(si, ci);
+	unlock_cluster(ci);
 }
 
 static int swp_entry_cmp(const void *ent1, const void *ent2)
@@ -1522,9 +1483,9 @@ int swap_swapcount(struct swap_info_struct *si, swp_entry_t entry)
 	struct swap_cluster_info *ci;
 	int count;
 
-	ci = lock_cluster_or_swap_info(si, offset);
+	ci = lock_cluster(si, offset);
 	count = swap_count(si->swap_map[offset]);
-	unlock_cluster_or_swap_info(si, ci);
+	unlock_cluster(ci);
 	return count;
 }
 
@@ -1547,7 +1508,7 @@ int swp_swapcount(swp_entry_t entry)
 
 	offset = swp_offset(entry);
 
-	ci = lock_cluster_or_swap_info(si, offset);
+	ci = lock_cluster(si, offset);
 
 	count = swap_count(si->swap_map[offset]);
 	if (!(count & COUNT_CONTINUED))
@@ -1570,7 +1531,7 @@ int swp_swapcount(swp_entry_t entry)
 		n *= (SWAP_CONT_MAX + 1);
 	} while (tmp_count & COUNT_CONTINUED);
 out:
-	unlock_cluster_or_swap_info(si, ci);
+	unlock_cluster(ci);
 	return count;
 }
 
@@ -1585,8 +1546,8 @@ static bool swap_page_trans_huge_swapped(struct swap_info_struct *si,
 	int i;
 	bool ret = false;
 
-	ci = lock_cluster_or_swap_info(si, offset);
-	if (!ci || nr_pages == 1) {
+	ci = lock_cluster(si, offset);
+	if (nr_pages == 1) {
 		if (swap_count(map[roffset]))
 			ret = true;
 		goto unlock_out;
@@ -1598,7 +1559,7 @@ static bool swap_page_trans_huge_swapped(struct swap_info_struct *si,
 		}
 	}
 unlock_out:
-	unlock_cluster_or_swap_info(si, ci);
+	unlock_cluster(ci);
 	return ret;
 }
 
@@ -3428,7 +3389,7 @@ static int __swap_duplicate(swp_entry_t entry, unsigned char usage, int nr)
 	offset = swp_offset(entry);
 	VM_WARN_ON(nr > SWAPFILE_CLUSTER - offset % SWAPFILE_CLUSTER);
 	VM_WARN_ON(usage == 1 && nr > 1);
-	ci = lock_cluster_or_swap_info(si, offset);
+	ci = lock_cluster(si, offset);
 
 	err = 0;
 	for (i = 0; i < nr; i++) {
@@ -3483,7 +3444,7 @@ static int __swap_duplicate(swp_entry_t entry, unsigned char usage, int nr)
 	}
 
 unlock_out:
-	unlock_cluster_or_swap_info(si, ci);
+	unlock_cluster(ci);
 	return err;
 }
 
Signature · 1 line
2.47.1
#5 lore raw

PATCH v2 5/13 mm, swap: clean up device availability check

In reply to Kairui Song

From: Kairui Song <[email protected]>

Remove highest_bit and lowest_bit. After the HDD allocation path has been removed, the only purpose of these two fields is to determine whether the device is full or not, which can instead be determined by checking the inuse_pages.

---
 fs/btrfs/inode.c     |  1 -
 fs/iomap/swapfile.c  |  1 -
 include/linux/swap.h |  2 --
 mm/page_io.c         |  1 -
 mm/swapfile.c        | 38 ++++++++------------------------------
 5 files changed, 8 insertions(+), 35 deletions(-)
fs/btrfs/inode.c +0 -1
index 488edca8333a..a1ba78afab2c 100644
@@ -10044,7 +10044,6 @@ static int btrfs_swap_activate(struct swap_info_struct *sis, struct file *file,
 	*span = bsi.highest_ppage - bsi.lowest_ppage + 1;
 	sis->max = bsi.nr_pages;
 	sis->pages = bsi.nr_pages - 1;
-	sis->highest_bit = bsi.nr_pages - 1;
 	return bsi.nr_extents;
 }
 #else
fs/iomap/swapfile.c +0 -1
index 5fc0ac36dee3..b90d0eda9e51 100644
@@ -189,7 +189,6 @@ int iomap_swapfile_activate(struct swap_info_struct *sis,
 	*pagespan = 1 + isi.highest_ppage - isi.lowest_ppage;
 	sis->max = isi.nr_pages;
 	sis->pages = isi.nr_pages - 1;
-	sis->highest_bit = isi.nr_pages - 1;
 	return isi.nr_extents;
 }
 EXPORT_SYMBOL_GPL(iomap_swapfile_activate);
include/linux/swap.h +0 -2
index 0c681aa5cb98..0c222017b5c6 100644
@@ -306,8 +306,6 @@ struct swap_info_struct {
 	struct list_head frag_clusters[SWAP_NR_ORDERS];
 					/* list of cluster that are fragmented or contented */
 	unsigned int frag_cluster_nr[SWAP_NR_ORDERS];
-	unsigned int lowest_bit;	/* index of first free in swap_map */
-	unsigned int highest_bit;	/* index of last free in swap_map */
 	unsigned int pages;		/* total of usable pages of swap */
 	unsigned int inuse_pages;	/* number of those currently in use */
 	struct percpu_cluster __percpu *percpu_cluster; /* per cpu's swap location */
mm/page_io.c +0 -1
index 4b4ea8e49cf6..9b983de351f9 100644
@@ -163,7 +163,6 @@ int generic_swapfile_activate(struct swap_info_struct *sis,
 		page_no = 1;	/* force Empty message */
 	sis->max = page_no;
 	sis->pages = page_no - 1;
-	sis->highest_bit = page_no - 1;
 out:
 	return ret;
 bad_bmap:
mm/swapfile.c +8 -30
index d0e5b9fa0c48..7963a0c646a4 100644
@@ -55,7 +55,7 @@ static bool swap_count_continued(struct swap_info_struct *, pgoff_t,
 static void free_swap_count_continuations(struct swap_info_struct *);
 static void swap_entry_range_free(struct swap_info_struct *si, swp_entry_t entry,
 				  unsigned int nr_pages);
-static void swap_range_alloc(struct swap_info_struct *si, unsigned long offset,
+static void swap_range_alloc(struct swap_info_struct *si,
 			     unsigned int nr_entries);
 static bool folio_swapcache_freeable(struct folio *folio);
 static struct swap_cluster_info *lock_cluster(struct swap_info_struct *si,
@@ -650,7 +650,7 @@ static bool cluster_alloc_range(struct swap_info_struct *si, struct swap_cluster
 	}
 
 	memset(si->swap_map + start, usage, nr_pages);
-	swap_range_alloc(si, start, nr_pages);
+	swap_range_alloc(si, nr_pages);
 	ci->count += nr_pages;
 
 	if (ci->count == SWAPFILE_CLUSTER) {
@@ -888,19 +888,11 @@ static void del_from_avail_list(struct swap_info_struct *si)
 	spin_unlock(&swap_avail_lock);
 }
 
-static void swap_range_alloc(struct swap_info_struct *si, unsigned long offset,
+static void swap_range_alloc(struct swap_info_struct *si,
 			     unsigned int nr_entries)
 {
-	unsigned int end = offset + nr_entries - 1;
-
-	if (offset == si->lowest_bit)
-		si->lowest_bit += nr_entries;
-	if (end == si->highest_bit)
-		WRITE_ONCE(si->highest_bit, si->highest_bit - nr_entries);
 	WRITE_ONCE(si->inuse_pages, si->inuse_pages + nr_entries);
 	if (si->inuse_pages == si->pages) {
-		si->lowest_bit = si->max;
-		si->highest_bit = 0;
 		del_from_avail_list(si);
 
 		if (si->cluster_info && vm_swap_full())
@@ -933,15 +925,8 @@ static void swap_range_free(struct swap_info_struct *si, unsigned long offset,
 	for (i = 0; i < nr_entries; i++)
 		clear_bit(offset + i, si->zeromap);
 
-	if (offset < si->lowest_bit)
-		si->lowest_bit = offset;
-	if (end > si->highest_bit) {
-		bool was_full = !si->highest_bit;
-
-		WRITE_ONCE(si->highest_bit, end);
-		if (was_full && (si->flags & SWP_WRITEOK))
-			add_to_avail_list(si);
-	}
+	if (si->inuse_pages == si->pages)
+		add_to_avail_list(si);
 	if (si->flags & SWP_BLKDEV)
 		swap_slot_free_notify =
 			si->bdev->bd_disk->fops->swap_slot_free_notify;
@@ -1051,15 +1036,12 @@ int get_swap_pages(int n_goal, swp_entry_t swp_entries[], int entry_order)
 		plist_requeue(&si->avail_lists[node], &swap_avail_heads[node]);
 		spin_unlock(&swap_avail_lock);
 		spin_lock(&si->lock);
-		if (!si->highest_bit || !(si->flags & SWP_WRITEOK)) {
+		if ((si->inuse_pages == si->pages) || !(si->flags & SWP_WRITEOK)) {
 			spin_lock(&swap_avail_lock);
 			if (plist_node_empty(&si->avail_lists[node])) {
 				spin_unlock(&si->lock);
 				goto nextsi;
 			}
-			WARN(!si->highest_bit,
-			     "swap_info %d in list but !highest_bit\n",
-			     si->type);
 			WARN(!(si->flags & SWP_WRITEOK),
 			     "swap_info %d in list but !SWP_WRITEOK\n",
 			     si->type);
@@ -2441,8 +2423,8 @@ static void _enable_swap_info(struct swap_info_struct *si)
 	 */
 	plist_add(&si->list, &swap_active_head);
 
-	/* add to available list iff swap device is not full */
-	if (si->highest_bit)
+	/* add to available list if swap device is not full */
+	if (si->inuse_pages < si->pages)
 		add_to_avail_list(si);
 }
 
@@ -2606,7 +2588,6 @@ SYSCALL_DEFINE1(swapoff, const char __user *, specialfile)
 	drain_mmlist();
 
 	/* wait for anyone still in scan_swap_map_slots */
-	p->highest_bit = 0;		/* cuts scans short */
 	while (p->flags >= SWP_SCANNING) {
 		spin_unlock(&p->lock);
 		spin_unlock(&swap_lock);
@@ -2941,8 +2922,6 @@ static unsigned long read_swap_header(struct swap_info_struct *si,
 		return 0;
 	}
 
-	si->lowest_bit  = 1;
-
 	maxpages = swapfile_maximum_size;
 	last_page = swap_header->info.last_page;
 	if (!last_page) {
@@ -2959,7 +2938,6 @@ static unsigned long read_swap_header(struct swap_info_struct *si,
 		if ((unsigned int)maxpages == 0)
 			maxpages = UINT_MAX;
 	}
-	si->highest_bit = maxpages - 1;
 
 	if (!maxpages)
 		return 0;
Signature · 1 line
2.47.1
#6 lore raw

Re: [PATCH v2 05/13] mm, swap: clean up device availability check

kernel test robot noticed the following build errors:

Hi Kairui,

kernel test robot noticed the following build errors:

[auto build test ERROR on akpm-mm/mm-everything]

url:    https://github.com/intel-lab-lkp/linux/commits/Kairui-Song/mm-swap-minor-clean-up-for-swap-entry-allocation/20241224-224136
base:   https://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm.git mm-everything
patch link:    https://lore.kernel.org/r/20241224143811.33462-6-ryncsn%40gmail.com
patch subject: [PATCH v2 05/13] mm, swap: clean up device availability check
config: arm64-randconfig-001-20241224 (https://download.01.org/0day-ci/archive/20241225/[email protected]/config)
compiler: aarch64-linux-gcc (GCC) 14.2.0
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20241225/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All errors (new ones prefixed by >>):

fs/f2fs/data.c: In function 'check_swap_activate':

>> fs/f2fs/data.c:4046:12: error: 'struct swap_info_struct' has no member named 'highest_bit'
    4046 |         sis->highest_bit = cur_lblock - 1;
         |            ^~

vim +4046 fs/f2fs/data.c

859fca6b706e00 Chao Yu              2021-05-26  3940  
0b8fc00601c0d8 Chao Yu              2021-05-26  3941  static int check_swap_activate(struct swap_info_struct *sis,
af4b6b8edf6aad Chao Yu              2020-10-12  3942  				struct file *swap_file, sector_t *span)
af4b6b8edf6aad Chao Yu              2020-10-12  3943  {
af4b6b8edf6aad Chao Yu              2020-10-12  3944  	struct address_space *mapping = swap_file->f_mapping;
af4b6b8edf6aad Chao Yu              2020-10-12  3945  	struct inode *inode = mapping->host;
36e4d95891ed37 [email protected] 2021-03-01  3946  	struct f2fs_sb_info *sbi = F2FS_I_SB(inode);
aa4074e8fec4d2 Wu Bo                2024-04-28  3947  	block_t cur_lblock;
aa4074e8fec4d2 Wu Bo                2024-04-28  3948  	block_t last_lblock;
aa4074e8fec4d2 Wu Bo                2024-04-28  3949  	block_t pblock;
aa4074e8fec4d2 Wu Bo                2024-04-28  3950  	block_t lowest_pblock = -1;
aa4074e8fec4d2 Wu Bo                2024-04-28  3951  	block_t highest_pblock = 0;
af4b6b8edf6aad Chao Yu              2020-10-12  3952  	int nr_extents = 0;
aa4074e8fec4d2 Wu Bo                2024-04-28  3953  	unsigned int nr_pblocks;
859fca6b706e00 Chao Yu              2021-05-26  3954  	unsigned int blks_per_sec = BLKS_PER_SEC(sbi);
ca298241bc2293 Jaegeuk Kim          2021-05-11  3955  	unsigned int not_aligned = 0;
36e4d95891ed37 [email protected] 2021-03-01  3956  	int ret = 0;
af4b6b8edf6aad Chao Yu              2020-10-12  3957  
af4b6b8edf6aad Chao Yu              2020-10-12  3958  	/*
af4b6b8edf6aad Chao Yu              2020-10-12  3959  	 * Map all the blocks into the extent list.  This code doesn't try
af4b6b8edf6aad Chao Yu              2020-10-12  3960  	 * to be very smart.
af4b6b8edf6aad Chao Yu              2020-10-12  3961  	 */
af4b6b8edf6aad Chao Yu              2020-10-12  3962  	cur_lblock = 0;
7461f370941802 Chao Yu              2024-11-08  3963  	last_lblock = F2FS_BYTES_TO_BLK(i_size_read(inode));
af4b6b8edf6aad Chao Yu              2020-10-12  3964  
1da66103837077 [email protected] 2021-02-27  3965  	while (cur_lblock < last_lblock && cur_lblock < sis->max) {
b876f4c94c3d16 Jaegeuk Kim          2020-11-24  3966  		struct f2fs_map_blocks map;
859fca6b706e00 Chao Yu              2021-05-26  3967  retry:
af4b6b8edf6aad Chao Yu              2020-10-12  3968  		cond_resched();
af4b6b8edf6aad Chao Yu              2020-10-12  3969  
b876f4c94c3d16 Jaegeuk Kim          2020-11-24  3970  		memset(&map, 0, sizeof(map));
b876f4c94c3d16 Jaegeuk Kim          2020-11-24  3971  		map.m_lblk = cur_lblock;
36e4d95891ed37 [email protected] 2021-03-01  3972  		map.m_len = last_lblock - cur_lblock;
36e4d95891ed37 [email protected] 2021-03-01  3973  		map.m_next_pgofs = NULL;
36e4d95891ed37 [email protected] 2021-03-01  3974  		map.m_next_extent = NULL;
b876f4c94c3d16 Jaegeuk Kim          2020-11-24  3975  		map.m_seg_type = NO_CHECK_TYPE;
36e4d95891ed37 [email protected] 2021-03-01  3976  		map.m_may_create = false;
af4b6b8edf6aad Chao Yu              2020-10-12  3977  
cd8fc5226bef3a Christoph Hellwig    2022-11-28  3978  		ret = f2fs_map_blocks(inode, &map, F2FS_GET_BLOCK_FIEMAP);
af4b6b8edf6aad Chao Yu              2020-10-12  3979  		if (ret)
36e4d95891ed37 [email protected] 2021-03-01  3980  			goto out;
af4b6b8edf6aad Chao Yu              2020-10-12  3981  
af4b6b8edf6aad Chao Yu              2020-10-12  3982  		/* hole */
36e4d95891ed37 [email protected] 2021-03-01  3983  		if (!(map.m_flags & F2FS_MAP_FLAGS)) {
833dcd35453713 Joe Perches          2021-05-26  3984  			f2fs_err(sbi, "Swapfile has holes");
f395183f9544ba Jaegeuk Kim          2021-05-12  3985  			ret = -EINVAL;
36e4d95891ed37 [email protected] 2021-03-01  3986  			goto out;
36e4d95891ed37 [email protected] 2021-03-01  3987  		}
af4b6b8edf6aad Chao Yu              2020-10-12  3988  
b876f4c94c3d16 Jaegeuk Kim          2020-11-24  3989  		pblock = map.m_pblk;
b876f4c94c3d16 Jaegeuk Kim          2020-11-24  3990  		nr_pblocks = map.m_len;
af4b6b8edf6aad Chao Yu              2020-10-12  3991  
aa4074e8fec4d2 Wu Bo                2024-04-28  3992  		if ((pblock - SM_I(sbi)->main_blkaddr) % blks_per_sec ||
aa4074e8fec4d2 Wu Bo                2024-04-28  3993  				nr_pblocks % blks_per_sec ||
9703d69d9d153b Daeho Jeong          2024-02-13  3994  				!f2fs_valid_pinned_area(sbi, pblock)) {
9703d69d9d153b Daeho Jeong          2024-02-13  3995  			bool last_extent = false;
9703d69d9d153b Daeho Jeong          2024-02-13  3996  
ca298241bc2293 Jaegeuk Kim          2021-05-11  3997  			not_aligned++;
859fca6b706e00 Chao Yu              2021-05-26  3998  
859fca6b706e00 Chao Yu              2021-05-26  3999  			nr_pblocks = roundup(nr_pblocks, blks_per_sec);
859fca6b706e00 Chao Yu              2021-05-26  4000  			if (cur_lblock + nr_pblocks > sis->max)
859fca6b706e00 Chao Yu              2021-05-26  4001  				nr_pblocks -= blks_per_sec;
859fca6b706e00 Chao Yu              2021-05-26  4002  
859fca6b706e00 Chao Yu              2021-05-26  4003  			/* this extent is last one */
9703d69d9d153b Daeho Jeong          2024-02-13  4004  			if (!nr_pblocks) {
9703d69d9d153b Daeho Jeong          2024-02-13  4005  				nr_pblocks = last_lblock - cur_lblock;
9703d69d9d153b Daeho Jeong          2024-02-13  4006  				last_extent = true;
ca298241bc2293 Jaegeuk Kim          2021-05-11  4007  			}
36e4d95891ed37 [email protected] 2021-03-01  4008  
859fca6b706e00 Chao Yu              2021-05-26  4009  			ret = f2fs_migrate_blocks(inode, cur_lblock,
859fca6b706e00 Chao Yu              2021-05-26  4010  							nr_pblocks);
9703d69d9d153b Daeho Jeong          2024-02-13  4011  			if (ret) {
9703d69d9d153b Daeho Jeong          2024-02-13  4012  				if (ret == -ENOENT)
9703d69d9d153b Daeho Jeong          2024-02-13  4013  					ret = -EINVAL;
859fca6b706e00 Chao Yu              2021-05-26  4014  				goto out;
9703d69d9d153b Daeho Jeong          2024-02-13  4015  			}
9703d69d9d153b Daeho Jeong          2024-02-13  4016  
9703d69d9d153b Daeho Jeong          2024-02-13  4017  			if (!last_extent)
859fca6b706e00 Chao Yu              2021-05-26  4018  				goto retry;
859fca6b706e00 Chao Yu              2021-05-26  4019  		}
9703d69d9d153b Daeho Jeong          2024-02-13  4020  
af4b6b8edf6aad Chao Yu              2020-10-12  4021  		if (cur_lblock + nr_pblocks >= sis->max)
af4b6b8edf6aad Chao Yu              2020-10-12  4022  			nr_pblocks = sis->max - cur_lblock;
af4b6b8edf6aad Chao Yu              2020-10-12  4023  
af4b6b8edf6aad Chao Yu              2020-10-12  4024  		if (cur_lblock) {	/* exclude the header page */
af4b6b8edf6aad Chao Yu              2020-10-12  4025  			if (pblock < lowest_pblock)
af4b6b8edf6aad Chao Yu              2020-10-12  4026  				lowest_pblock = pblock;
af4b6b8edf6aad Chao Yu              2020-10-12  4027  			if (pblock + nr_pblocks - 1 > highest_pblock)
af4b6b8edf6aad Chao Yu              2020-10-12  4028  				highest_pblock = pblock + nr_pblocks - 1;
af4b6b8edf6aad Chao Yu              2020-10-12  4029  		}
af4b6b8edf6aad Chao Yu              2020-10-12  4030  
af4b6b8edf6aad Chao Yu              2020-10-12  4031  		/*
af4b6b8edf6aad Chao Yu              2020-10-12  4032  		 * We found a PAGE_SIZE-length, PAGE_SIZE-aligned run of blocks
af4b6b8edf6aad Chao Yu              2020-10-12  4033  		 */
af4b6b8edf6aad Chao Yu              2020-10-12  4034  		ret = add_swap_extent(sis, cur_lblock, nr_pblocks, pblock);
af4b6b8edf6aad Chao Yu              2020-10-12  4035  		if (ret < 0)
af4b6b8edf6aad Chao Yu              2020-10-12  4036  			goto out;
af4b6b8edf6aad Chao Yu              2020-10-12  4037  		nr_extents += ret;
af4b6b8edf6aad Chao Yu              2020-10-12  4038  		cur_lblock += nr_pblocks;
af4b6b8edf6aad Chao Yu              2020-10-12  4039  	}
af4b6b8edf6aad Chao Yu              2020-10-12  4040  	ret = nr_extents;
af4b6b8edf6aad Chao Yu              2020-10-12  4041  	*span = 1 + highest_pblock - lowest_pblock;
af4b6b8edf6aad Chao Yu              2020-10-12  4042  	if (cur_lblock == 0)
af4b6b8edf6aad Chao Yu              2020-10-12  4043  		cur_lblock = 1;	/* force Empty message */
af4b6b8edf6aad Chao Yu              2020-10-12  4044  	sis->max = cur_lblock;
af4b6b8edf6aad Chao Yu              2020-10-12  4045  	sis->pages = cur_lblock - 1;
af4b6b8edf6aad Chao Yu              2020-10-12 @4046  	sis->highest_bit = cur_lblock - 1;
af4b6b8edf6aad Chao Yu              2020-10-12  4047  out:
859fca6b706e00 Chao Yu              2021-05-26  4048  	if (not_aligned)
d7e9a9037de27b Daniel Rosenberg     2023-10-02  4049  		f2fs_warn(sbi, "Swapfile (%u) is not align to section: 1) creat(), 2) ioctl(F2FS_IOC_SET_PIN_FILE), 3) fallocate(%lu * N)",
859fca6b706e00 Chao Yu              2021-05-26  4050  			  not_aligned, blks_per_sec * F2FS_BLKSIZE);
af4b6b8edf6aad Chao Yu              2020-10-12  4051  	return ret;
af4b6b8edf6aad Chao Yu              2020-10-12  4052  }
af4b6b8edf6aad Chao Yu              2020-10-12  4053  
Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
#7 lore raw

Re: [PATCH v2 05/13] mm, swap: clean up device availability check

In reply to Kairui Song
kernel test robot noticed the following build errors:

Hi Kairui,

kernel test robot noticed the following build errors:

[auto build test ERROR on akpm-mm/mm-everything]

url:    https://github.com/intel-lab-lkp/linux/commits/Kairui-Song/mm-swap-minor-clean-up-for-swap-entry-allocation/20241224-224136
base:   https://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm.git mm-everything
patch link:    https://lore.kernel.org/r/20241224143811.33462-6-ryncsn%40gmail.com
patch subject: [PATCH v2 05/13] mm, swap: clean up device availability check
config: i386-buildonly-randconfig-001-20241224 (https://download.01.org/0day-ci/archive/20241225/[email protected]/config)
compiler: clang version 19.1.3 (https://github.com/llvm/llvm-project ab51eccf88f5321e7c60591c5546b254b6afab99)
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20241225/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All errors (new ones prefixed by >>):

>> fs/f2fs/data.c:4046:7: error: no member named 'highest_bit' in 'struct swap_info_struct'
    4046 |         sis->highest_bit = cur_lblock - 1;
         |         ~~~  ^

1 error generated.

vim +4046 fs/f2fs/data.c

859fca6b706e00 Chao Yu              2021-05-26  3940  
0b8fc00601c0d8 Chao Yu              2021-05-26  3941  static int check_swap_activate(struct swap_info_struct *sis,
af4b6b8edf6aad Chao Yu              2020-10-12  3942  				struct file *swap_file, sector_t *span)
af4b6b8edf6aad Chao Yu              2020-10-12  3943  {
af4b6b8edf6aad Chao Yu              2020-10-12  3944  	struct address_space *mapping = swap_file->f_mapping;
af4b6b8edf6aad Chao Yu              2020-10-12  3945  	struct inode *inode = mapping->host;
36e4d95891ed37 [email protected] 2021-03-01  3946  	struct f2fs_sb_info *sbi = F2FS_I_SB(inode);
aa4074e8fec4d2 Wu Bo                2024-04-28  3947  	block_t cur_lblock;
aa4074e8fec4d2 Wu Bo                2024-04-28  3948  	block_t last_lblock;
aa4074e8fec4d2 Wu Bo                2024-04-28  3949  	block_t pblock;
aa4074e8fec4d2 Wu Bo                2024-04-28  3950  	block_t lowest_pblock = -1;
aa4074e8fec4d2 Wu Bo                2024-04-28  3951  	block_t highest_pblock = 0;
af4b6b8edf6aad Chao Yu              2020-10-12  3952  	int nr_extents = 0;
aa4074e8fec4d2 Wu Bo                2024-04-28  3953  	unsigned int nr_pblocks;
859fca6b706e00 Chao Yu              2021-05-26  3954  	unsigned int blks_per_sec = BLKS_PER_SEC(sbi);
ca298241bc2293 Jaegeuk Kim          2021-05-11  3955  	unsigned int not_aligned = 0;
36e4d95891ed37 [email protected] 2021-03-01  3956  	int ret = 0;
af4b6b8edf6aad Chao Yu              2020-10-12  3957  
af4b6b8edf6aad Chao Yu              2020-10-12  3958  	/*
af4b6b8edf6aad Chao Yu              2020-10-12  3959  	 * Map all the blocks into the extent list.  This code doesn't try
af4b6b8edf6aad Chao Yu              2020-10-12  3960  	 * to be very smart.
af4b6b8edf6aad Chao Yu              2020-10-12  3961  	 */
af4b6b8edf6aad Chao Yu              2020-10-12  3962  	cur_lblock = 0;
7461f370941802 Chao Yu              2024-11-08  3963  	last_lblock = F2FS_BYTES_TO_BLK(i_size_read(inode));
af4b6b8edf6aad Chao Yu              2020-10-12  3964  
1da66103837077 [email protected] 2021-02-27  3965  	while (cur_lblock < last_lblock && cur_lblock < sis->max) {
b876f4c94c3d16 Jaegeuk Kim          2020-11-24  3966  		struct f2fs_map_blocks map;
859fca6b706e00 Chao Yu              2021-05-26  3967  retry:
af4b6b8edf6aad Chao Yu              2020-10-12  3968  		cond_resched();
af4b6b8edf6aad Chao Yu              2020-10-12  3969  
b876f4c94c3d16 Jaegeuk Kim          2020-11-24  3970  		memset(&map, 0, sizeof(map));
b876f4c94c3d16 Jaegeuk Kim          2020-11-24  3971  		map.m_lblk = cur_lblock;
36e4d95891ed37 [email protected] 2021-03-01  3972  		map.m_len = last_lblock - cur_lblock;
36e4d95891ed37 [email protected] 2021-03-01  3973  		map.m_next_pgofs = NULL;
36e4d95891ed37 [email protected] 2021-03-01  3974  		map.m_next_extent = NULL;
b876f4c94c3d16 Jaegeuk Kim          2020-11-24  3975  		map.m_seg_type = NO_CHECK_TYPE;
36e4d95891ed37 [email protected] 2021-03-01  3976  		map.m_may_create = false;
af4b6b8edf6aad Chao Yu              2020-10-12  3977  
cd8fc5226bef3a Christoph Hellwig    2022-11-28  3978  		ret = f2fs_map_blocks(inode, &map, F2FS_GET_BLOCK_FIEMAP);
af4b6b8edf6aad Chao Yu              2020-10-12  3979  		if (ret)
36e4d95891ed37 [email protected] 2021-03-01  3980  			goto out;
af4b6b8edf6aad Chao Yu              2020-10-12  3981  
af4b6b8edf6aad Chao Yu              2020-10-12  3982  		/* hole */
36e4d95891ed37 [email protected] 2021-03-01  3983  		if (!(map.m_flags & F2FS_MAP_FLAGS)) {
833dcd35453713 Joe Perches          2021-05-26  3984  			f2fs_err(sbi, "Swapfile has holes");
f395183f9544ba Jaegeuk Kim          2021-05-12  3985  			ret = -EINVAL;
36e4d95891ed37 [email protected] 2021-03-01  3986  			goto out;
36e4d95891ed37 [email protected] 2021-03-01  3987  		}
af4b6b8edf6aad Chao Yu              2020-10-12  3988  
b876f4c94c3d16 Jaegeuk Kim          2020-11-24  3989  		pblock = map.m_pblk;
b876f4c94c3d16 Jaegeuk Kim          2020-11-24  3990  		nr_pblocks = map.m_len;
af4b6b8edf6aad Chao Yu              2020-10-12  3991  
aa4074e8fec4d2 Wu Bo                2024-04-28  3992  		if ((pblock - SM_I(sbi)->main_blkaddr) % blks_per_sec ||
aa4074e8fec4d2 Wu Bo                2024-04-28  3993  				nr_pblocks % blks_per_sec ||
9703d69d9d153b Daeho Jeong          2024-02-13  3994  				!f2fs_valid_pinned_area(sbi, pblock)) {
9703d69d9d153b Daeho Jeong          2024-02-13  3995  			bool last_extent = false;
9703d69d9d153b Daeho Jeong          2024-02-13  3996  
ca298241bc2293 Jaegeuk Kim          2021-05-11  3997  			not_aligned++;
859fca6b706e00 Chao Yu              2021-05-26  3998  
859fca6b706e00 Chao Yu              2021-05-26  3999  			nr_pblocks = roundup(nr_pblocks, blks_per_sec);
859fca6b706e00 Chao Yu              2021-05-26  4000  			if (cur_lblock + nr_pblocks > sis->max)
859fca6b706e00 Chao Yu              2021-05-26  4001  				nr_pblocks -= blks_per_sec;
859fca6b706e00 Chao Yu              2021-05-26  4002  
859fca6b706e00 Chao Yu              2021-05-26  4003  			/* this extent is last one */
9703d69d9d153b Daeho Jeong          2024-02-13  4004  			if (!nr_pblocks) {
9703d69d9d153b Daeho Jeong          2024-02-13  4005  				nr_pblocks = last_lblock - cur_lblock;
9703d69d9d153b Daeho Jeong          2024-02-13  4006  				last_extent = true;
ca298241bc2293 Jaegeuk Kim          2021-05-11  4007  			}
36e4d95891ed37 [email protected] 2021-03-01  4008  
859fca6b706e00 Chao Yu              2021-05-26  4009  			ret = f2fs_migrate_blocks(inode, cur_lblock,
859fca6b706e00 Chao Yu              2021-05-26  4010  							nr_pblocks);
9703d69d9d153b Daeho Jeong          2024-02-13  4011  			if (ret) {
9703d69d9d153b Daeho Jeong          2024-02-13  4012  				if (ret == -ENOENT)
9703d69d9d153b Daeho Jeong          2024-02-13  4013  					ret = -EINVAL;
859fca6b706e00 Chao Yu              2021-05-26  4014  				goto out;
9703d69d9d153b Daeho Jeong          2024-02-13  4015  			}
9703d69d9d153b Daeho Jeong          2024-02-13  4016  
9703d69d9d153b Daeho Jeong          2024-02-13  4017  			if (!last_extent)
859fca6b706e00 Chao Yu              2021-05-26  4018  				goto retry;
859fca6b706e00 Chao Yu              2021-05-26  4019  		}
9703d69d9d153b Daeho Jeong          2024-02-13  4020  
af4b6b8edf6aad Chao Yu              2020-10-12  4021  		if (cur_lblock + nr_pblocks >= sis->max)
af4b6b8edf6aad Chao Yu              2020-10-12  4022  			nr_pblocks = sis->max - cur_lblock;
af4b6b8edf6aad Chao Yu              2020-10-12  4023  
af4b6b8edf6aad Chao Yu              2020-10-12  4024  		if (cur_lblock) {	/* exclude the header page */
af4b6b8edf6aad Chao Yu              2020-10-12  4025  			if (pblock < lowest_pblock)
af4b6b8edf6aad Chao Yu              2020-10-12  4026  				lowest_pblock = pblock;
af4b6b8edf6aad Chao Yu              2020-10-12  4027  			if (pblock + nr_pblocks - 1 > highest_pblock)
af4b6b8edf6aad Chao Yu              2020-10-12  4028  				highest_pblock = pblock + nr_pblocks - 1;
af4b6b8edf6aad Chao Yu              2020-10-12  4029  		}
af4b6b8edf6aad Chao Yu              2020-10-12  4030  
af4b6b8edf6aad Chao Yu              2020-10-12  4031  		/*
af4b6b8edf6aad Chao Yu              2020-10-12  4032  		 * We found a PAGE_SIZE-length, PAGE_SIZE-aligned run of blocks
af4b6b8edf6aad Chao Yu              2020-10-12  4033  		 */
af4b6b8edf6aad Chao Yu              2020-10-12  4034  		ret = add_swap_extent(sis, cur_lblock, nr_pblocks, pblock);
af4b6b8edf6aad Chao Yu              2020-10-12  4035  		if (ret < 0)
af4b6b8edf6aad Chao Yu              2020-10-12  4036  			goto out;
af4b6b8edf6aad Chao Yu              2020-10-12  4037  		nr_extents += ret;
af4b6b8edf6aad Chao Yu              2020-10-12  4038  		cur_lblock += nr_pblocks;
af4b6b8edf6aad Chao Yu              2020-10-12  4039  	}
af4b6b8edf6aad Chao Yu              2020-10-12  4040  	ret = nr_extents;
af4b6b8edf6aad Chao Yu              2020-10-12  4041  	*span = 1 + highest_pblock - lowest_pblock;
af4b6b8edf6aad Chao Yu              2020-10-12  4042  	if (cur_lblock == 0)
af4b6b8edf6aad Chao Yu              2020-10-12  4043  		cur_lblock = 1;	/* force Empty message */
af4b6b8edf6aad Chao Yu              2020-10-12  4044  	sis->max = cur_lblock;
af4b6b8edf6aad Chao Yu              2020-10-12  4045  	sis->pages = cur_lblock - 1;
af4b6b8edf6aad Chao Yu              2020-10-12 @4046  	sis->highest_bit = cur_lblock - 1;
af4b6b8edf6aad Chao Yu              2020-10-12  4047  out:
859fca6b706e00 Chao Yu              2021-05-26  4048  	if (not_aligned)
d7e9a9037de27b Daniel Rosenberg     2023-10-02  4049  		f2fs_warn(sbi, "Swapfile (%u) is not align to section: 1) creat(), 2) ioctl(F2FS_IOC_SET_PIN_FILE), 3) fallocate(%lu * N)",
859fca6b706e00 Chao Yu              2021-05-26  4050  			  not_aligned, blks_per_sec * F2FS_BLKSIZE);
af4b6b8edf6aad Chao Yu              2020-10-12  4051  	return ret;
af4b6b8edf6aad Chao Yu              2020-10-12  4052  }
af4b6b8edf6aad Chao Yu              2020-10-12  4053  
Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
#8 lore raw

PATCH v2 6/13 mm, swap: clean up plist removal and adding

In reply to Kairui Song

From: Kairui Song <[email protected]>

When the swap device is full (inuse_pages == pages), it should be removed from the allocation available plist. If any slot is freed, the swap device should be added back to the plist. Additionally, during swapon or swapoff, the swap device is forcefully added or removed.

Currently, the condition (inuse_pages == pages) is checked after every counter update, then remove or add the device accordingly. This is serialized by si->lock.

This commit decouples it from the protection of si->lock and reworked plist removal and adding, making it possible to get rid of the hard dependency on si->lock in allocation path in later commits.

To achieve this, simply using another lock is not an optimal approach, as the overhead is observable for a hot counter, and may cause complex locking issues. Thus, this commit manages to make it a lock-free atomic operation, by embedding the plist state into the second highest bit of the atomic counter.

Simply making the counter an atomic will not work, if the update and plist status check are not performed atomically, we may miss an addition or removal. With the embedded info we can update the counter and check the plist status with single atomic operations, and avoid any extra overheads:

If the counter is full (inuse_pages == pages) and the off-list bit is unset, we attempt to remove it from the plist. If the counter is not full (inuse_pages != pages) and the off-list bit is set, we attempt to add it to the plist. Removing, adding and bit update is serialized with a lock, which is a cold path. Ordinary counter updates will be lock-free.

---
 include/linux/swap.h |   2 +-
 mm/swapfile.c        | 184 +++++++++++++++++++++++++++++++------------
 2 files changed, 135 insertions(+), 51 deletions(-)
include/linux/swap.h +1 -1
index 0c222017b5c6..e1eeea6307cd 100644
@@ -307,7 +307,7 @@ struct swap_info_struct {
 					/* list of cluster that are fragmented or contented */
 	unsigned int frag_cluster_nr[SWAP_NR_ORDERS];
 	unsigned int pages;		/* total of usable pages of swap */
-	unsigned int inuse_pages;	/* number of those currently in use */
+	atomic_long_t inuse_pages;	/* number of those currently in use */
 	struct percpu_cluster __percpu *percpu_cluster; /* per cpu's swap location */
 	struct rb_root swap_extent_root;/* root of the swap extent rbtree */
 	struct block_device *bdev;	/* swap device or bdev of swap file */
mm/swapfile.c +134 -50

329 lines not on this page · raw message

Signature · 1 line
2.47.1
#9 lore raw

PATCH v2 7/13 mm, swap: hold a reference during scan and cleanup flag usage

In reply to Kairui Song

From: Kairui Song <[email protected]>

The flag SWP_SCANNING was used as an indicator of whether a device is being scanned for allocation, and prevents swapoff. Combined with SWP_WRITEOK, they work as a set of barriers for a clean swapoff:

1. Swapoff clears SWP_WRITEOK, allocation requests will see ~SWP_WRITEOK and abort as it's serialized by si->lock.
2. Swapoff unuses all allocated entries.
3. Swapoff waits for SWP_SCANNING flag to be cleared, so ongoing allocations will stop, preventing UAF.
4. Now swapoff can free everything safely.

This will make the allocation path have a hard dependency on si->lock. Allocation always have to acquire si->lock first for setting SWP_SCANNING and checking SWP_WRITEOK.

This commit removes this flag, and just uses the existing per-CPU refcount instead to prevent UAF in step 3, which serves well for such usage without dependency on si->lock, and scales very well too. Just hold a reference during the whole scan and allocation process. Swapoff will kill and wait for the counter.

And for preventing any allocation from happening after step 1 so the unuse in step 2 can ensure all slots are free, swapoff will acquire the ci->lock of each cluster one by one to ensure all allocations see ~SWP_WRITEOK and abort.

This way these dependences on si->lock are gone. And worth noting we can't kill the refcount as the first step for swapoff as the unuse process have to acquire the refcount.

---
 include/linux/swap.h |  1 -
 mm/swapfile.c        | 90 ++++++++++++++++++++++++++++----------------
 2 files changed, 57 insertions(+), 34 deletions(-)
include/linux/swap.h +0 -1
index e1eeea6307cd..02120f1005d5 100644
@@ -219,7 +219,6 @@ enum {
 	SWP_STABLE_WRITES = (1 << 11),	/* no overwrite PG_writeback pages */
 	SWP_SYNCHRONOUS_IO = (1 << 12),	/* synchronous IO is efficient */
 					/* add others here before... */
-	SWP_SCANNING	= (1 << 14),	/* refcount in scan_swap_map */
 };
 
 #define SWAP_CLUSTER_MAX 32UL
mm/swapfile.c +57 -33

151 lines not on this page · raw message

Signature · 1 line
2.47.1
#10 lore raw

PATCH v2 8/13 mm, swap: use an enum to define all cluster flags and wrap flags changes

In reply to Kairui Song

From: Kairui Song <[email protected]>

Currently, we are only using flags to indicate which list the cluster is on. Using one bit for each list type might be a waste, as the list type grows, we will consume too many bits. Additionally, the current mixed usage of '&' and '==' is a bit confusing.

Make it clean by using an enum to define all possible cluster statuses. Only an off-list cluster will have the NONE (0) flag. And use a wrapper to annotate and sanitize all flag settings and list movements.

---
 include/linux/swap.h | 17 +++++++---
 mm/swapfile.c        | 75 +++++++++++++++++++++++---------------------
 2 files changed, 52 insertions(+), 40 deletions(-)
include/linux/swap.h +13 -4
index 02120f1005d5..339d7f0192ff 100644
@@ -257,10 +257,19 @@ struct swap_cluster_info {
 	u8 order;
 	struct list_head list;
 };
-#define CLUSTER_FLAG_FREE 1 /* This cluster is free */
-#define CLUSTER_FLAG_NONFULL 2 /* This cluster is on nonfull list */
-#define CLUSTER_FLAG_FRAG 4 /* This cluster is on nonfull list */
-#define CLUSTER_FLAG_FULL 8 /* This cluster is on full list */
+
+/* All on-list cluster must have a non-zero flag. */
+enum swap_cluster_flags {
+	CLUSTER_FLAG_NONE = 0, /* For temporary off-list cluster */
+	CLUSTER_FLAG_FREE,
+	CLUSTER_FLAG_NONFULL,
+	CLUSTER_FLAG_FRAG,
+	/* Clusters with flags above are allocatable */
+	CLUSTER_FLAG_USABLE = CLUSTER_FLAG_FRAG,
+	CLUSTER_FLAG_FULL,
+	CLUSTER_FLAG_DISCARD,
+	CLUSTER_FLAG_MAX,
+};
 
 /*
  * The first page in the swap file is the swap header, which is always marked
mm/swapfile.c +39 -36

136 lines not on this page · raw message

Signature · 1 line
2.47.1
#11 lore raw

PATCH v2 9/13 mm, swap: reduce contention on device lock

In reply to Kairui Song

From: Kairui Song <[email protected]>

Currently, swap locking is mainly composed of two locks: the cluster lock (ci->lock) and the device lock (si->lock).

The cluster lock is much more fine-grained, so it is best to use ci->lock instead of si->lock as much as possible.

We have cleaned up other hard dependencies on si->lock. Following the new cluster allocator design, most operations don't need to touch si->lock at all. In practice, we only need to take si->lock when moving clusters between lists.

To achieve this, this commit reworks the locking pattern of all si->lock and ci->lock users, eliminates all usage of ci->lock inside si->lock, and introduces a new design to avoid touching si->lock unless needed.

For minimal contention and easier understanding of the system, two ideas are introduced with the corresponding helpers: isolation and relocation.

- Clusters will be isolated from the list when iterating the list to search for an allocatable cluster.

This ensures other CPUs won't walk into the same cluster easily, and it releases si->lock after acquiring ci->lock, providing the only place that handles the inversion of two locks, and avoids contention.

Iterating the cluster list almost always moves the cluster (free -> nonfull, nonfull -> frag, frag -> frag tail), but it doesn't know where the cluster should be moved to until scanning is done. So keeping the cluster off-list is a good option with low overhead.

The off-list time window of a cluster is also minimal. In the worst case, one CPU will return the cluster after scanning the 512 entries on it, which we used to busy wait with a spin lock.

This is done with the new helper cluster_isolate_lock.

- Clusters will be relocated after allocation or freeing, according to their usage count and status.

Allocations no longer hold si->lock now, and may drop ci->lock for reclaim, so the cluster could be moved to any location while no lock is held. Besides, isolation clears all flags when it takes the cluster off the list (the flags must be in sync with the list status, so cluster users don't need to touch si->lock for checking its list status). So the cluster has to be relocated to the right list according to its usage after allocation or freeing.

Relocation is optional, if the cluster flags indicate it's already on the right list, it will skip touching the list or si->lock.

This is done with relocate_cluster after allocation or with [partial_]free_cluster after freeing.

This handled usage of all kinds of clusters in a clean way.

Scanning and allocation by iterating the cluster list is handled by "isolate - <scan / allocate> - relocate".

Scanning and allocation of per-CPU clusters will only involve "<scan / allocate> - relocate", as it knows which cluster to lock and use.

Freeing will only involve "relocate".

Each CPU will keep using its per-CPU cluster until the 512 entries are all consumed. Freeing also has to free 512 entries to trigger cluster movement in the best case, so si->lock is rarely touched.

Testing with building the Linux kernel with defconfig showed huge improvement:

tiem make -j96 / 768M memcg, 4K pages, 10G ZRAM, on Intel 8255C:
Before:
Sys time: 73578.30, Real time: 864.05
After: (-50.7% sys time, -44.8% real time)
Sys time: 36227.49, Real time: 476.66

time make -j96 / 1152M memcg, 64K mTHP, 10G ZRAM, on Intel 8255C:
(avg of 4 test run)
Before:
Sys time: 74044.85, Real time: 846.51 hugepages-64kB/stats/swpout: 1735216 hugepages-64kB/stats/swpout_fallback: 430333

After: (-40.4% sys time, -37.1% real time)
Sys time: 44160.56, Real time: 532.07
hugepages-64kB/stats/swpout: 1786288
hugepages-64kB/stats/swpout_fallback: 243384

time make -j32 / 512M memcg, 4K pages, 5G ZRAM, on AMD 7K62:
Before:
Sys time: 8098.21, Real time: 401.3
After: (-22.6% sys time, -12.8% real time )
Sys time: 6265.02, Real time: 349.83

The allocation success rate also slightly improved as we sanitized the usage of clusters with new defined helpers, previously dropping si->lock or ci->lock during scan will cause cluster order shuffle.

---
 include/linux/swap.h |   3 +-
 mm/swapfile.c        | 435 ++++++++++++++++++++++++-------------------
 2 files changed, 246 insertions(+), 192 deletions(-)
include/linux/swap.h +2 -1
index 339d7f0192ff..c4ff31cb6bde 100644
@@ -291,6 +291,7 @@ enum swap_cluster_flags {
  * throughput.
  */
 struct percpu_cluster {
+	local_lock_t lock; /* Protect the percpu_cluster above */
 	unsigned int next[SWAP_NR_ORDERS]; /* Likely next allocation offset */
 };
 
@@ -313,7 +314,7 @@ struct swap_info_struct {
 					/* list of cluster that contains at least one free slot */
 	struct list_head frag_clusters[SWAP_NR_ORDERS];
 					/* list of cluster that are fragmented or contented */
-	unsigned int frag_cluster_nr[SWAP_NR_ORDERS];
+	atomic_long_t frag_cluster_nr[SWAP_NR_ORDERS];
 	unsigned int pages;		/* total of usable pages of swap */
 	atomic_long_t inuse_pages;	/* number of those currently in use */
 	struct percpu_cluster __percpu *percpu_cluster; /* per cpu's swap location */
mm/swapfile.c +244 -191

753 lines not on this page · raw message

Signature · 1 line
2.47.1
#12 lore raw

PATCH v2 10/13 mm, swap: simplify percpu cluster updating

In reply to Kairui Song

From: Kairui Song <[email protected]>

Instead of using a returning argument, we can simply store the next cluster offset to the fixed percpu location, which reduce the stack usage and simplify the function:

Object size:
./scripts/bloat-o-meter mm/swapfile.o mm/swapfile.o.new add/remove: 0/0 grow/shrink: 0/2 up/down: 0/-271 (-271)
Function old new delta
get_swap_pages 2847 2733 -114
alloc_swap_scan_cluster 894 737 -157
Total: Before=30833, After=30562, chg -0.88%

Stack usage:
Before:
swapfile.c:1190:5:get_swap_pages 240 static

After:
swapfile.c:1185:5:get_swap_pages 216 static

---
 include/linux/swap.h |  4 +--
 mm/swapfile.c        | 66 +++++++++++++++++++-------------------------
 2 files changed, 31 insertions(+), 39 deletions(-)
include/linux/swap.h +2 -2
index c4ff31cb6bde..4c1d2e69689f 100644
@@ -275,9 +275,9 @@ enum swap_cluster_flags {
  * The first page in the swap file is the swap header, which is always marked
  * bad to prevent it from being allocated as an entry. This also prevents the
  * cluster to which it belongs being marked free. Therefore 0 is safe to use as
- * a sentinel to indicate next is not valid in percpu_cluster.
+ * a sentinel to indicate an entry is not valid.
  */
-#define SWAP_NEXT_INVALID	0
+#define SWAP_ENTRY_INVALID	0
 
 #ifdef CONFIG_THP_SWAP
 #define SWAP_NR_ORDERS		(PMD_ORDER + 1)
mm/swapfile.c +29 -37

146 lines not on this page · raw message

Signature · 1 line
2.47.1
#13 lore raw

PATCH v2 11/13 mm, swap: introduce a helper for retrieving cluster from offset

In reply to Kairui Song

From: Kairui Song <[email protected]>

It's a common operation to retrieve the cluster info from offset, introduce a helper for this.

---
 mm/swapfile.c | 14 ++++++++++----
 1 file changed, 10 insertions(+), 4 deletions(-)
mm/swapfile.c +10 -4
index ac170acf55a7..0445a2db8492 100644
@@ -424,6 +424,12 @@ static inline unsigned int cluster_index(struct swap_info_struct *si,
 	return ci - si->cluster_info;
 }
 
+static inline struct swap_cluster_info *offset_to_cluster(struct swap_info_struct *si,
+							  unsigned long offset)
+{
+	return &si->cluster_info[offset / SWAPFILE_CLUSTER];
+}
+
 static inline unsigned int cluster_offset(struct swap_info_struct *si,
 					  struct swap_cluster_info *ci)
 {
@@ -435,7 +441,7 @@ static inline struct swap_cluster_info *lock_cluster(struct swap_info_struct *si
 {
 	struct swap_cluster_info *ci;
 
-	ci = &si->cluster_info[offset / SWAPFILE_CLUSTER];
+	ci = offset_to_cluster(si, offset);
 	spin_lock(&ci->lock);
 
 	return ci;
@@ -1473,10 +1479,10 @@ static void swap_entry_range_free(struct swap_info_struct *si, swp_entry_t entry
 	unsigned char *map_end = map + nr_pages;
 	struct swap_cluster_info *ci;
 
-	/* It should never free entries across different clusters */
-	VM_BUG_ON((offset / SWAPFILE_CLUSTER) != ((offset + nr_pages - 1) / SWAPFILE_CLUSTER));
-
 	ci = lock_cluster(si, offset);
+
+	/* It should never free entries across different clusters */
+	VM_BUG_ON(ci != offset_to_cluster(si, offset + nr_pages - 1));
 	VM_BUG_ON(cluster_is_free(ci));
 	VM_BUG_ON(ci->count < nr_pages);
 
Signature · 1 line
2.47.1
#14 lore raw

PATCH v2 12/13 mm, swap: use a global swap cluster for non-rotation devices

In reply to Kairui Song

From: Kairui Song <[email protected]>

Non-rotational devices (SSD / ZRAM) can tolerate fragmentation, so the goal of the SWAP allocator is to avoid contention for clusters. It uses a per-CPU cluster design, and each CPU will use a different cluster as much as possible.

However, HDDs are very sensitive to fragmentation, contention is trivial in comparison. Therefore, we use one global cluster instead. This ensures that each order will be written to the same cluster as much as possible, which helps make the I/O more continuous.

This ensures that the performance of the cluster allocator is as good as that of the old allocator. Tests after this commit compared to those before this series:

Tested using 'make -j32' with tinyconfig, a 1G memcg limit, and HDD swap:

make -j32 with tinyconfig, using 1G memcg limit and HDD swap:

Before this series:
114.44user 29.11system 39:42.90elapsed 6%CPU (0avgtext+0avgdata 157284maxresident)k 2901232inputs+0outputs (238877major+4227640minor)pagefaults

After this commit:
113.90user 23.81system 38:11.77elapsed 6%CPU (0avgtext+0avgdata 157260maxresident)k 2548728inputs+0outputs (235471major+4238110minor)pagefaults

---
 include/linux/swap.h |  2 ++
 mm/swapfile.c        | 51 ++++++++++++++++++++++++++++++++------------
 2 files changed, 39 insertions(+), 14 deletions(-)
include/linux/swap.h +2 -0
index 4c1d2e69689f..b13b72645db3 100644
@@ -318,6 +318,8 @@ struct swap_info_struct {
 	unsigned int pages;		/* total of usable pages of swap */
 	atomic_long_t inuse_pages;	/* number of those currently in use */
 	struct percpu_cluster __percpu *percpu_cluster; /* per cpu's swap location */
+	struct percpu_cluster *global_cluster; /* Use one global cluster for rotating device */
+	spinlock_t global_cluster_lock;	/* Serialize usage of global cluster */
 	struct rb_root swap_extent_root;/* root of the swap extent rbtree */
 	struct block_device *bdev;	/* swap device or bdev of swap file */
 	struct file *swap_file;		/* seldom referenced */
mm/swapfile.c +37 -14

90 lines not on this page · raw message

Signature · 1 line
2.47.1
#15 lore raw

PATCH v2 13/13 mm, swap_slots: remove slot cache for freeing path

In reply to Kairui Song

From: Kairui Song <[email protected]>

The slot cache for freeing path is mostly for reducing the overhead of si->lock. As we have basically eliminated the si->lock usage for freeing path, it can be removed.

This helps simplify the code, and avoids swap entries from being hold in cache upon freeing. The delayed freeing of entries have been causing trouble for further optimizations for zswap [1] and in theory will also cause more fragmentation, and extra overhead.

Test with build linux kernel showed both performance and fragmentation is better without the cache:

tiem make -j96 / 768M memcg, 4K pages, 10G ZRAM, avg of 4 test run::
Before:
Sys time: 36047.78, Real time: 472.43
After: (-7.6% sys time, -7.3% real time)
Sys time: 33314.76, Real time: 437.67

time make -j96 / 1152M memcg, 64K mTHP, 10G ZRAM, avg of 4 test run:
Before:
Sys time: 46859.04, Real time: 562.63 hugepages-64kB/stats/swpout: 1783392 hugepages-64kB/stats/swpout_fallback: 240875
After: (-23.3% sys time, -21.3% real time)
Sys time: 35958.87, Real time: 442.69 hugepages-64kB/stats/swpout: 1866267 hugepages-64kB/stats/swpout_fallback: 158330

Sequential SWAP should be also slightly faster, tests didn't show a measurable difference though, at least no regression:

Swapin 4G zero page on ZRAM (time in us):
Before (avg. 1923756)
1912391 1927023 1927957 1916527 1918263 1914284 1934753 1940813 1921791 After (avg. 1922290):
1919101 1925743 1916810 1917007 1923930 1935152 1917403 1923549 1921913

---
 include/linux/swap_slots.h |  3 --
 mm/swap_slots.c            | 78 +++++----------------------------
 mm/swapfile.c              | 89 +++++++++++++++-----------------------
 3 files changed, 44 insertions(+), 126 deletions(-)
include/linux/swap_slots.h +0 -3
index 15adfb8c813a..840aec3523b2 100644
@@ -16,15 +16,12 @@ struct swap_slots_cache {
 	swp_entry_t	*slots;
 	int		nr;
 	int		cur;
-	spinlock_t	free_lock;  /* protects slots_ret, n_ret */
-	swp_entry_t	*slots_ret;
 	int		n_ret;
 };
 
 void disable_swap_slots_cache_lock(void);
 void reenable_swap_slots_cache_unlock(void);
 void enable_swap_slots_cache(void);
-void free_swap_slot(swp_entry_t entry);
 
 extern bool swap_slot_cache_enabled;
 
mm/swap_slots.c +10 -68

160 lines not on this page · raw message

mm/swapfile.c +34 -55

170 lines not on this page · raw message

Signature · 1 line
2.47.1