Skip to contentSkip to conversation

mm, swap: rework of swap allocator locks

PATCH Superseded Kairui Song 4 participants 21 messages Oct 22, 2024 – Oct 24, 2024

Subsystems (4)

v1 of 4 Superseded Latest is v4 →

lore
Share & tools
lore
#0 lore raw

cover mm, swap: rework of swap allocator locks

From: Kairui Song <[email protected]>

This series improved the swap allocator performance greatly by reworking the locking design and simplify a lot of code path.

This is follow up of previous swap cluster allocator series:
https://lore.kernel.org/linux-mm/[email protected]/

And this series is based on an follow up fix of the swap cluster allocator:
https://lore.kernel.org/linux-mm/[email protected]/

This is part of the new swap allocator work item discussed in Chris's "Swap Abstraction" discussion at LSF/MM 2024, and "mTHP and swap allocator" discussion at LPC 2024.

Previous series introduced a fully cluster based allocation algorithm, this series completely get rid of the old allocation path and makes the allocator avoid grabbing the si->lock unless needed. This bring huge performance gain and get rid of slot cache on freeing path.

Currently, swap locking is mainly composed of two locks, cluster lock (ci->lock) and device lock (si->lock). The device lock is widely used to protect many things, causing it to be the main bottleneck for SWAP.

Cluster lock is much more fine-grained, so it will be best to use ci->lock instead of si->lock as much as possible.

perf lock indicates this issue clearly. Doing linux kernel build using tmpfs and ZRAM with limited memory (make -j64 with 1G memcg and 4k pages), result of "perf lock contention -ab sleep 3":

contended total wait max wait avg wait type caller

     34948     53.63 s       7.11 ms      1.53 ms     spinlock   free_swap_and_cache_nr+0x350
     16569     40.05 s       6.45 ms      2.42 ms     spinlock   get_swap_pages+0x231
     11191     28.41 s       7.03 ms      2.54 ms     spinlock   swapcache_free_entries+0x59
      4147     22.78 s     122.66 ms      5.49 ms     spinlock   page_vma_mapped_walk+0x6f3
      4595      7.17 s       6.79 ms      1.56 ms     spinlock   swapcache_free_entries+0x59
    406027      2.74 s       2.59 ms      6.74 us     spinlock   list_lru_add+0x39

...snip...

The top 5 caller are all users of si->lock, total wait time up sums to several minutes in the 3 seconds time window.

Following the new allocator design, many operation doesn't need to touch si->lock at all. We only need to take si->lock when doing operations across multiple clusters (eg. changing the cluster list), other operations only need to take ci->lock. So ideally allocator should always take ci->lock first, then, if needed, take si->lock. But due to historical reasons, ci->lock is used inside si->lock by design, causing lock inversion if we simply try to acquire si->lock after acquiring ci->lock.

This series audited all si->lock usage, simplify legacy codes, eliminate usage of si->lock as much as possible by introducing new designs based on the new cluster allocator.

Old HDD allocation codes are removed, cluster allocator is adapted with small changes for HDD usage, test is looking OK.

And this also removed slot cache for freeing path. The performance is better without it, and this enables other clean up and optimizations as discussed before:
https://lore.kernel.org/all/CAMgjq7ACohT_uerSz8E_994ZZCv709Zor+43hdmesW_59W1BWw@mail.gmail.com/

After this series, lock contention on si->lock is nearly unobservable with perf lock with the same test above :

contended total wait max wait avg wait type caller
... snip ...

         91    204.62 us      4.51 us      2.25 us     spinlock   cluster_move+0x2e
  ... snip ...
         47    125.62 us      4.47 us      2.67 us     spinlock   cluster_move+0x2e
  ... snip ...
         23     63.15 us      3.95 us      2.74 us     spinlock   cluster_move+0x2e
  ... snip ...
         17     41.26 us      4.58 us      2.43 us     spinlock   cluster_isolate_lock+0x1d

... snip ...

cluster_move and cluster_isolate_lock are basically the only users of si->lock now, performance gain is huge with reduced LOC.

Tests
===

Build kernel with defconfig on tmpfs with ZRAM as swap:
---

Running a test matrix which is scaled up progressive for a intuitive result. The test are ran on top of tmpfs, using memory cgroup for memory limitation, on a 48c96t system.

12 test run for each case, it can be seen clearly that as concurrent job number goes higher the performance gain is higher, the performance is higher even with low concurrency.

   make -j<NR>     |   System Time (seconds)  |   Total Time (seconds)
 (NR / Mem / ZRAM) | (Before / After / Delta) | (Before / After / Delta)
 With 4k pages only:
  6 / 192M / 3G    |    5258 /  5235 / -0.3%  |    1420 /  1414 / -0.3%
 12 / 256M / 4G    |    5518 /  5337 / -3.3%  |     758 /   742 / -2.1%
 24 / 384M / 5G    |    7091 /  5766 / -18.7% |     476 /   422 / -11.3%
 48 / 768M / 7G    |   11139 /  5831 / -47.7% |     330 /   221 / -33.0%
 96 / 1.5G / 10G   |   21303 / 11353 / -46.7% |     283 /   180 / -36.4%
 With 64k mTHP:
 24 / 512M / 5G    |    5104 /  4641 / -18.7% |     376 /   358 / -4.8%
 48 /   1G / 7G    |    8693 /  4662 / -18.7% |     257 /   176 / -31.5%
 96 /   2G / 10G   |   17056 / 10263 / -39.8% |     234 /   169 / -27.8%

With more aggressive setup, it shows clearly both the performance and fragmentation are better:

tiem make -j96 / 768M memcg, 4K pages, 10G ZRAM, on Intel 8255C * 2:
(avg of 4 test run)
Before:
Sys time: 73578.30, Real time: 864.05
tiem make -j96 / 1G memcg, 4K pages, 10G ZRAM:
After: (-54.7% sys time, -49.3% real time)
Sys time: 33314.76, Real time: 437.67

time make -j96 / 1152M memcg, 64K mTHP, 10G ZRAM, on Intel 8255C * 2:
(avg of 4 test run)
Before:
Sys time: 74044.85, Real time: 846.51 hugepages-64kB/stats/swpout: 1735216 hugepages-64kB/stats/swpout_fallback: 430333
After: (-51.4% sys time, -47.7% real time, -63.2% mTHP failure) Sys time: 35958.87, Real time: 442.69 hugepages-64kB/stats/swpout: 1866267 hugepages-64kB/stats/swpout_fallback: 158330

There is a up to 54.7% improvement for build kernel test, and lower fragmentation rate. Performance improvement should be even larger for micro benchmarks

Build kernel with tinyconfig on tmpfs with HDD as swap:
---

This test is similar to above, but HDD test is very noisy and slow, the deviation is huge, so just use tinyconfig instead and take the median test result of 3 test run, which looks OK:

Before this series:
114.44user 29.11system 39:42.90elapsed 6%CPU 2901232inputs+0outputs (238877major+4227640minor)pagefaults

After this commit:
113.90user 23.81system 38:11.77elapsed 6%CPU 2548728inputs+0outputs (235471major+4238110minor)pagefaults

Single thread SWAP:
---

Sequential SWAP should also be slightly faster as we removed a lot of unnecessary parts. Test using micro benchmark for swapout/in 4G zero memory using ZRAM, 10 test runs:

Swapout Before (avg. 3359304):
3353796 3358551 3371305 3356043 3367524 3355303 3355924 3354513 3360776

Swapin Before (avg. 1928698):
1920283 1927183 1934105 1921373 1926562 1938261 1927726 1928636 1934155

Swapout After (avg. 3347511, -0.4%):
3337863 3347948 3355235 3339081 3333134 3353006 3354917 3346055 3360359

Swapin After (avg. 1922290, -0.3%):
1919101 1925743 1916810 1917007 1923930 1935152 1917403 1923549 1921913

Worth noticing the patch "mm, swap: use a global swap cluster for non-rotation device" introduced minor overhead for certain tests (see the test results in commit message), but the gain from later commit covered that, it can be further improved later.

Kairui Song (13):
mm, swap: minor clean up for swap entry allocation mm, swap: fold swap_info_get_cont in the only caller mm, swap: remove old allocation path for HDD
mm, swap: use cluster lock for HDD
mm, swap: clean up device availability check
mm, swap: clean up plist removal and adding
mm, swap: hold a reference of si during scan and clean up flags mm, swap: use an enum to define all cluster flags and wrap flags
changes
mm, swap: reduce contention on device lock
mm, swap: simplify percpu cluster updating
mm, swap: introduce a helper for retrieving cluster from offset mm, swap: use a global swap cluster for non-rotation device mm, swap_slots: remove slot cache for freeing path

 fs/btrfs/inode.c           |    1 -
 fs/iomap/swapfile.c        |    1 -
 include/linux/swap.h       |   36 +-
 include/linux/swap_slots.h |    3 -
 mm/page_io.c               |    1 -
 mm/swap_slots.c            |   78 +--
 mm/swapfile.c              | 1198 ++++++++++++++++--------------------
 7 files changed, 558 insertions(+), 760 deletions(-)
Signature · 1 line
2.47.0
#1 lore raw

PATCH 1/13 mm, swap: minor clean up for swap entry allocation

From: Kairui Song <[email protected]>

Direct reclaim can skip the whole folio after reclaimed a set of folio based slots. Also simplify the code for allocation, reduce indention.

---
 mm/swapfile.c | 59 +++++++++++++++++++++++++--------------------------
 1 file changed, 29 insertions(+), 30 deletions(-)
mm/swapfile.c +29 -30
index 46bd4b1a3c07..1128cea95c47 100644
@@ -604,23 +604,28 @@ static bool cluster_reclaim_range(struct swap_info_struct *si,
 				  unsigned long start, unsigned long end)
 {
 	unsigned char *map = si->swap_map;
-	unsigned long offset;
+	unsigned long offset = start;
+	int nr_reclaim;
 
 	spin_unlock(&ci->lock);
 	spin_unlock(&si->lock);
 
-	for (offset = start; offset < end; offset++) {
+	do {
 		switch (READ_ONCE(map[offset])) {
 		case 0:
-			continue;
+			offset++;
+			break;
 		case SWAP_HAS_CACHE:
-			if (__try_to_reclaim_swap(si, offset, TTRS_ANYWAY | TTRS_DIRECT) > 0)
-				continue;
-			goto out;
+			nr_reclaim = __try_to_reclaim_swap(si, offset, TTRS_ANYWAY | TTRS_DIRECT);
+			if (nr_reclaim > 0)
+				offset += nr_reclaim;
+			else
+				goto out;
+			break;
 		default:
 			goto out;
 		}
-	}
+	} while (offset < end);
 out:
 	spin_lock(&si->lock);
 	spin_lock(&ci->lock);
@@ -826,35 +831,30 @@ static unsigned long cluster_alloc_swap_entry(struct swap_info_struct *si, int o
 							 &found, order, usage);
 			frags++;
 			if (found)
-				break;
+				goto done;
 		}
 
-		if (!found) {
+		/*
+		 * Nonfull clusters are moved to frag tail if we reached
+		 * here, count them too, don't over scan the frag list.
+		 */
+		while (frags < si->frag_cluster_nr[order]) {
+			ci = list_first_entry(&si->frag_clusters[order],
+					      struct swap_cluster_info, list);
 			/*
-			 * Nonfull clusters are moved to frag tail if we reached
-			 * here, count them too, don't over scan the frag list.
+			 * Rotate the frag list to iterate, they were all failing
+			 * high order allocation or moved here due to per-CPU usage,
+			 * this help keeping usable cluster ahead.
 			 */
-			while (frags < si->frag_cluster_nr[order]) {
-				ci = list_first_entry(&si->frag_clusters[order],
-						      struct swap_cluster_info, list);
-				/*
-				 * Rotate the frag list to iterate, they were all failing
-				 * high order allocation or moved here due to per-CPU usage,
-				 * this help keeping usable cluster ahead.
-				 */
-				list_move_tail(&ci->list, &si->frag_clusters[order]);
-				offset = alloc_swap_scan_cluster(si, cluster_offset(si, ci),
-								 &found, order, usage);
-				frags++;
-				if (found)
-					break;
-			}
+			list_move_tail(&ci->list, &si->frag_clusters[order]);
+			offset = alloc_swap_scan_cluster(si, cluster_offset(si, ci),
+							 &found, order, usage);
+			frags++;
+			if (found)
+				goto done;
 		}
 	}
 
-	if (found)
-		goto done;
-
 	if (!list_empty(&si->discard_clusters)) {
 		/*
 		 * we don't have free cluster but have some clusters in
@@ -892,7 +892,6 @@ static unsigned long cluster_alloc_swap_entry(struct swap_info_struct *si, int o
 				goto done;
 		}
 	}
-
 done:
 	cluster->next[order] = offset;
 	return found;
Signature · 1 line
2.47.0
#2 lore raw

PATCH 2/13 mm, swap: fold swap_info_get_cont in the only caller

In reply to Kairui Song

From: Kairui Song <[email protected]>

The name of the function is confusing, and the code is much easier to follow after folding, also rename the confusing naming "p" to more meaningful "si".

---
 mm/swapfile.c | 39 +++++++++++++++------------------------
 1 file changed, 15 insertions(+), 24 deletions(-)
mm/swapfile.c +15 -24
index 1128cea95c47..e1e4a1ba4fc5 100644
@@ -1359,22 +1359,6 @@ static struct swap_info_struct *_swap_info_get(swp_entry_t entry)
 	return NULL;
 }
 
-static struct swap_info_struct *swap_info_get_cont(swp_entry_t entry,
-					struct swap_info_struct *q)
-{
-	struct swap_info_struct *p;
-
-	p = _swap_info_get(entry);
-
-	if (p != q) {
-		if (q != NULL)
-			spin_unlock(&q->lock);
-		if (p != NULL)
-			spin_lock(&p->lock);
-	}
-	return p;
-}
-
 static unsigned char __swap_entry_free_locked(struct swap_info_struct *si,
 					      unsigned long offset,
 					      unsigned char usage)
@@ -1671,14 +1655,14 @@ static int swp_entry_cmp(const void *ent1, const void *ent2)
 
 void swapcache_free_entries(swp_entry_t *entries, int n)
 {
-	struct swap_info_struct *p, *prev;
+	struct swap_info_struct *si, *prev;
 	int i;
 
 	if (n <= 0)
 		return;
 
 	prev = NULL;
-	p = NULL;
+	si = NULL;
 
 	/*
 	 * Sort swap entries by swap device, so each lock is only taken once.
@@ -1688,13 +1672,20 @@ void swapcache_free_entries(swp_entry_t *entries, int n)
 	if (nr_swapfiles > 1)
 		sort(entries, n, sizeof(entries[0]), swp_entry_cmp, NULL);
 	for (i = 0; i < n; ++i) {
-		p = swap_info_get_cont(entries[i], prev);
-		if (p)
-			swap_entry_range_free(p, entries[i], 1);
-		prev = p;
+		si = _swap_info_get(entries[i]);
+
+		if (si != prev) {
+			if (prev != NULL)
+				spin_unlock(&prev->lock);
+			if (si != NULL)
+				spin_lock(&si->lock);
+		}
+		if (si)
+			swap_entry_range_free(si, entries[i], 1);
+		prev = si;
 	}
-	if (p)
-		spin_unlock(&p->lock);
+	if (si)
+		spin_unlock(&si->lock);
 }
 
 int __swap_count(swp_entry_t entry)
Signature · 1 line
2.47.0
#3 lore raw

PATCH 3/13 mm, swap: remove old allocation path for HDD

In reply to Kairui Song

From: Kairui Song <[email protected]>

We are currently using different swap allocation algorithm for HDD and non-HDD. This leads to the existing of different set of locking, and the code path is heavily bloated, causing troubles for further optimization and maintenance.

This commit removes all HDD swap allocation and related dead code, and use cluster allocation algorithm instead.

The performance may drop a little bit temporarily, and should be
negligible: The main advantage of legacy HDD allocation algorithm is that is tend to use continuous slots, but swap device gets fragmented quickly anyway, and the attempt to use continuous slots will fail easily.

This commit also enables mTHP swap on HDD, which should be beneficial, and following commits will adapt and optimize the cluster allocator for HDD.

---
 include/linux/swap.h |   3 -
 mm/swapfile.c        | 235 ++-----------------------------------------
 2 files changed, 9 insertions(+), 229 deletions(-)
include/linux/swap.h +0 -3
index f3e0ac20c2e8..3a71198a6957 100644
@@ -309,9 +309,6 @@ struct swap_info_struct {
 	unsigned int highest_bit;	/* index of last free in swap_map */
 	unsigned int pages;		/* total of usable pages of swap */
 	unsigned int inuse_pages;	/* number of those currently in use */
-	unsigned int cluster_next;	/* likely index for next allocation */
-	unsigned int cluster_nr;	/* countdown to next cluster search */
-	unsigned int __percpu *cluster_next_cpu; /*percpu index for next allocation */
 	struct percpu_cluster __percpu *percpu_cluster; /* per cpu's swap location */
 	struct rb_root swap_extent_root;/* root of the swap extent rbtree */
 	struct block_device *bdev;	/* swap device or bdev of swap file */
mm/swapfile.c +9 -226

307 lines not on this page · raw message

Signature · 1 line
2.47.0
#4 lore raw

PATCH 4/13 mm, swap: use cluster lock for HDD

In reply to Kairui Song

From: Kairui Song <[email protected]>

Cluster lock (ci->lock) was introduce to reduce contention for certain operations. Using cluster lock for HDD is not helpful as HDD have a poor performance, so locking isn't the bottleneck. But having different set of locks for HDD / non-HDD prevents further rework of device lock (si->lock).

This commit just changed all lock_cluster_or_swap_info to lock_cluster, which is a safe and straight conversion since cluster info is always allocated now, also removed all cluster_info related checks.

---
 mm/swapfile.c | 107 ++++++++++++++++----------------------------------
 1 file changed, 34 insertions(+), 73 deletions(-)
mm/swapfile.c +34 -73
index ffdf7eedecb5..f8e70bb5f1d7 100644
@@ -58,10 +58,9 @@ static void swap_entry_range_free(struct swap_info_struct *si, swp_entry_t entry
 static void swap_range_alloc(struct swap_info_struct *si, unsigned long offset,
 			     unsigned int nr_entries);
 static bool folio_swapcache_freeable(struct folio *folio);
-static struct swap_cluster_info *lock_cluster_or_swap_info(
-		struct swap_info_struct *si, unsigned long offset);
-static void unlock_cluster_or_swap_info(struct swap_info_struct *si,
-					struct swap_cluster_info *ci);
+static struct swap_cluster_info *lock_cluster(struct swap_info_struct *si,
+					      unsigned long offset);
+static void unlock_cluster(struct swap_cluster_info *ci);
 
 static DEFINE_SPINLOCK(swap_lock);
 static unsigned int nr_swapfiles;
@@ -222,9 +221,9 @@ static int __try_to_reclaim_swap(struct swap_info_struct *si,
 	 * swap_map is HAS_CACHE only, which means the slots have no page table
 	 * reference or pending writeback, and can't be allocated to others.
 	 */
-	ci = lock_cluster_or_swap_info(si, offset);
+	ci = lock_cluster(si, offset);
 	need_reclaim = swap_is_has_cache(si, offset, nr_pages);
-	unlock_cluster_or_swap_info(si, ci);
+	unlock_cluster(ci);
 	if (!need_reclaim)
 		goto out_unlock;
 
@@ -404,45 +403,15 @@ static inline struct swap_cluster_info *lock_cluster(struct swap_info_struct *si
 {
 	struct swap_cluster_info *ci;
 
-	ci = si->cluster_info;
-	if (ci) {
-		ci += offset / SWAPFILE_CLUSTER;
-		spin_lock(&ci->lock);
-	}
-	return ci;
-}
-
-static inline void unlock_cluster(struct swap_cluster_info *ci)
-{
-	if (ci)
-		spin_unlock(&ci->lock);
-}
-
-/*
- * Determine the locking method in use for this device.  Return
- * swap_cluster_info if SSD-style cluster-based locking is in place.
- */
-static inline struct swap_cluster_info *lock_cluster_or_swap_info(
-		struct swap_info_struct *si, unsigned long offset)
-{
-	struct swap_cluster_info *ci;
-
-	/* Try to use fine-grained SSD-style locking if available: */
-	ci = lock_cluster(si, offset);
-	/* Otherwise, fall back to traditional, coarse locking: */
-	if (!ci)
-		spin_lock(&si->lock);
+	ci = &si->cluster_info[offset / SWAPFILE_CLUSTER];
+	spin_lock(&ci->lock);
 
 	return ci;
 }
 
-static inline void unlock_cluster_or_swap_info(struct swap_info_struct *si,
-					       struct swap_cluster_info *ci)
+static inline void unlock_cluster(struct swap_cluster_info *ci)
 {
-	if (ci)
-		unlock_cluster(ci);
-	else
-		spin_unlock(&si->lock);
+	spin_unlock(&ci->lock);
 }
 
 /* Add a cluster to discard list and schedule it to do discard */
@@ -558,9 +527,6 @@ static void inc_cluster_info_page(struct swap_info_struct *si,
 	unsigned long idx = page_nr / SWAPFILE_CLUSTER;
 	struct swap_cluster_info *ci;
 
-	if (!cluster_info)
-		return;
-
 	ci = cluster_info + idx;
 	ci->count++;
 
@@ -576,9 +542,6 @@ static void inc_cluster_info_page(struct swap_info_struct *si,
 static void dec_cluster_info_page(struct swap_info_struct *si,
 				  struct swap_cluster_info *ci, int nr_pages)
 {
-	if (!si->cluster_info)
-		return;
-
 	VM_BUG_ON(ci->count < nr_pages);
 	VM_BUG_ON(cluster_is_free(ci));
 	lockdep_assert_held(&si->lock);
@@ -995,8 +958,6 @@ static int cluster_alloc_swap(struct swap_info_struct *si,
 {
 	int n_ret = 0;
 
-	VM_BUG_ON(!si->cluster_info);
-
 	while (n_ret < nr) {
 		unsigned long offset = cluster_alloc_swap_entry(si, order, usage);
 
@@ -1036,10 +997,10 @@ static int scan_swap_map_slots(struct swap_info_struct *si,
 		}
 
 		/*
-		 * Swapfile is not block device or not using clusters so unable
+		 * Swapfile is not block device so unable
 		 * to allocate large entries.
 		 */
-		if (!(si->flags & SWP_BLKDEV) || !si->cluster_info)
+		if (!(si->flags & SWP_BLKDEV))
 			return 0;
 	}
 
@@ -1279,9 +1240,9 @@ static unsigned char __swap_entry_free(struct swap_info_struct *si,
 	unsigned long offset = swp_offset(entry);
 	unsigned char usage;
 
-	ci = lock_cluster_or_swap_info(si, offset);
+	ci = lock_cluster(si, offset);
 	usage = __swap_entry_free_locked(si, offset, 1);
-	unlock_cluster_or_swap_info(si, ci);
+	unlock_cluster(ci);
 	if (!usage)
 		free_swap_slot(entry);
 
@@ -1304,14 +1265,14 @@ static bool __swap_entries_free(struct swap_info_struct *si,
 	if (nr > SWAPFILE_CLUSTER - offset % SWAPFILE_CLUSTER)
 		goto fallback;
 
-	ci = lock_cluster_or_swap_info(si, offset);
+	ci = lock_cluster(si, offset);
 	if (!swap_is_last_map(si, offset, nr, &has_cache)) {
-		unlock_cluster_or_swap_info(si, ci);
+		unlock_cluster(ci);
 		goto fallback;
 	}
 	for (i = 0; i < nr; i++)
 		WRITE_ONCE(si->swap_map[offset + i], SWAP_HAS_CACHE);
-	unlock_cluster_or_swap_info(si, ci);
+	unlock_cluster(ci);
 
 	if (!has_cache) {
 		for (i = 0; i < nr; i++)
@@ -1367,7 +1328,7 @@ static void cluster_swap_free_nr(struct swap_info_struct *si,
 	DECLARE_BITMAP(to_free, BITS_PER_LONG) = { 0 };
 	int i, nr;
 
-	ci = lock_cluster_or_swap_info(si, offset);
+	ci = lock_cluster(si, offset);
 	while (nr_pages) {
 		nr = min(BITS_PER_LONG, nr_pages);
 		for (i = 0; i < nr; i++) {
@@ -1375,18 +1336,18 @@ static void cluster_swap_free_nr(struct swap_info_struct *si,
 				bitmap_set(to_free, i, 1);
 		}
 		if (!bitmap_empty(to_free, BITS_PER_LONG)) {
-			unlock_cluster_or_swap_info(si, ci);
+			unlock_cluster(ci);
 			for_each_set_bit(i, to_free, BITS_PER_LONG)
 				free_swap_slot(swp_entry(si->type, offset + i));
 			if (nr == nr_pages)
 				return;
 			bitmap_clear(to_free, 0, BITS_PER_LONG);
-			ci = lock_cluster_or_swap_info(si, offset);
+			ci = lock_cluster(si, offset);
 		}
 		offset += nr;
 		nr_pages -= nr;
 	}
-	unlock_cluster_or_swap_info(si, ci);
+	unlock_cluster(ci);
 }
 
 /*
@@ -1425,9 +1386,9 @@ void put_swap_folio(struct folio *folio, swp_entry_t entry)
 	if (!si)
 		return;
 
-	ci = lock_cluster_or_swap_info(si, offset);
+	ci = lock_cluster(si, offset);
 	if (size > 1 && swap_is_has_cache(si, offset, size)) {
-		unlock_cluster_or_swap_info(si, ci);
+		unlock_cluster(ci);
 		spin_lock(&si->lock);
 		swap_entry_range_free(si, entry, size);
 		spin_unlock(&si->lock);
@@ -1435,14 +1396,14 @@ void put_swap_folio(struct folio *folio, swp_entry_t entry)
 	}
 	for (int i = 0; i < size; i++, entry.val++) {
 		if (!__swap_entry_free_locked(si, offset + i, SWAP_HAS_CACHE)) {
-			unlock_cluster_or_swap_info(si, ci);
+			unlock_cluster(ci);
 			free_swap_slot(entry);
 			if (i == size - 1)
 				return;
-			lock_cluster_or_swap_info(si, offset);
+			lock_cluster(si, offset);
 		}
 	}
-	unlock_cluster_or_swap_info(si, ci);
+	unlock_cluster(ci);
 }
 
 static int swp_entry_cmp(const void *ent1, const void *ent2)
@@ -1506,9 +1467,9 @@ int swap_swapcount(struct swap_info_struct *si, swp_entry_t entry)
 	struct swap_cluster_info *ci;
 	int count;
 
-	ci = lock_cluster_or_swap_info(si, offset);
+	ci = lock_cluster(si, offset);
 	count = swap_count(si->swap_map[offset]);
-	unlock_cluster_or_swap_info(si, ci);
+	unlock_cluster(ci);
 	return count;
 }
 
@@ -1531,7 +1492,7 @@ int swp_swapcount(swp_entry_t entry)
 
 	offset = swp_offset(entry);
 
-	ci = lock_cluster_or_swap_info(si, offset);
+	ci = lock_cluster(si, offset);
 
 	count = swap_count(si->swap_map[offset]);
 	if (!(count & COUNT_CONTINUED))
@@ -1554,7 +1515,7 @@ int swp_swapcount(swp_entry_t entry)
 		n *= (SWAP_CONT_MAX + 1);
 	} while (tmp_count & COUNT_CONTINUED);
 out:
-	unlock_cluster_or_swap_info(si, ci);
+	unlock_cluster(ci);
 	return count;
 }
 
@@ -1569,8 +1530,8 @@ static bool swap_page_trans_huge_swapped(struct swap_info_struct *si,
 	int i;
 	bool ret = false;
 
-	ci = lock_cluster_or_swap_info(si, offset);
-	if (!ci || nr_pages == 1) {
+	ci = lock_cluster(si, offset);
+	if (nr_pages == 1) {
 		if (swap_count(map[roffset]))
 			ret = true;
 		goto unlock_out;
@@ -1582,7 +1543,7 @@ static bool swap_page_trans_huge_swapped(struct swap_info_struct *si,
 		}
 	}
 unlock_out:
-	unlock_cluster_or_swap_info(si, ci);
+	unlock_cluster(ci);
 	return ret;
 }
 
@@ -3412,7 +3373,7 @@ static int __swap_duplicate(swp_entry_t entry, unsigned char usage, int nr)
 	offset = swp_offset(entry);
 	VM_WARN_ON(nr > SWAPFILE_CLUSTER - offset % SWAPFILE_CLUSTER);
 	VM_WARN_ON(usage == 1 && nr > 1);
-	ci = lock_cluster_or_swap_info(si, offset);
+	ci = lock_cluster(si, offset);
 
 	err = 0;
 	for (i = 0; i < nr; i++) {
@@ -3467,7 +3428,7 @@ static int __swap_duplicate(swp_entry_t entry, unsigned char usage, int nr)
 	}
 
 unlock_out:
-	unlock_cluster_or_swap_info(si, ci);
+	unlock_cluster(ci);
 	return err;
 }
 
Signature · 1 line
2.47.0
#5 lore raw

PATCH 5/13 mm, swap: clean up device availability check

In reply to Kairui Song

From: Kairui Song <[email protected]>

Remove highest_bit and lowest_bit. After HDD allocation path is removed, only purpose of these two fields is to judge if the device is full or not, which can be done by checking inuse_pages instead.

---
 fs/btrfs/inode.c     |  1 -
 fs/iomap/swapfile.c  |  1 -
 include/linux/swap.h |  2 --
 mm/page_io.c         |  1 -
 mm/swapfile.c        | 38 ++++++++------------------------------
 5 files changed, 8 insertions(+), 35 deletions(-)
fs/btrfs/inode.c +0 -1
index 5618ca02934a..aba9c0d58998 100644
@@ -10023,7 +10023,6 @@ static int btrfs_swap_activate(struct swap_info_struct *sis, struct file *file,
 	*span = bsi.highest_ppage - bsi.lowest_ppage + 1;
 	sis->max = bsi.nr_pages;
 	sis->pages = bsi.nr_pages - 1;
-	sis->highest_bit = bsi.nr_pages - 1;
 	return bsi.nr_extents;
 }
 #else
fs/iomap/swapfile.c +0 -1
index 5fc0ac36dee3..b90d0eda9e51 100644
@@ -189,7 +189,6 @@ int iomap_swapfile_activate(struct swap_info_struct *sis,
 	*pagespan = 1 + isi.highest_ppage - isi.lowest_ppage;
 	sis->max = isi.nr_pages;
 	sis->pages = isi.nr_pages - 1;
-	sis->highest_bit = isi.nr_pages - 1;
 	return isi.nr_extents;
 }
 EXPORT_SYMBOL_GPL(iomap_swapfile_activate);
include/linux/swap.h +0 -2
index 3a71198a6957..c0d49dad7a4b 100644
@@ -305,8 +305,6 @@ struct swap_info_struct {
 	struct list_head frag_clusters[SWAP_NR_ORDERS];
 					/* list of cluster that are fragmented or contented */
 	unsigned int frag_cluster_nr[SWAP_NR_ORDERS];
-	unsigned int lowest_bit;	/* index of first free in swap_map */
-	unsigned int highest_bit;	/* index of last free in swap_map */
 	unsigned int pages;		/* total of usable pages of swap */
 	unsigned int inuse_pages;	/* number of those currently in use */
 	struct percpu_cluster __percpu *percpu_cluster; /* per cpu's swap location */
mm/page_io.c +0 -1
index a28d28b6b3ce..c8a25203bcf4 100644
@@ -163,7 +163,6 @@ int generic_swapfile_activate(struct swap_info_struct *sis,
 		page_no = 1;	/* force Empty message */
 	sis->max = page_no;
 	sis->pages = page_no - 1;
-	sis->highest_bit = page_no - 1;
 out:
 	return ret;
 bad_bmap:
mm/swapfile.c +8 -30
index f8e70bb5f1d7..e620b41c3120 100644
@@ -55,7 +55,7 @@ static bool swap_count_continued(struct swap_info_struct *, pgoff_t,
 static void free_swap_count_continuations(struct swap_info_struct *);
 static void swap_entry_range_free(struct swap_info_struct *si, swp_entry_t entry,
 				  unsigned int nr_pages);
-static void swap_range_alloc(struct swap_info_struct *si, unsigned long offset,
+static void swap_range_alloc(struct swap_info_struct *si,
 			     unsigned int nr_entries);
 static bool folio_swapcache_freeable(struct folio *folio);
 static struct swap_cluster_info *lock_cluster(struct swap_info_struct *si,
@@ -647,7 +647,7 @@ static void cluster_alloc_range(struct swap_info_struct *si, struct swap_cluster
 	}
 
 	memset(si->swap_map + start, usage, nr_pages);
-	swap_range_alloc(si, start, nr_pages);
+	swap_range_alloc(si, nr_pages);
 	ci->count += nr_pages;
 
 	if (ci->count == SWAPFILE_CLUSTER) {
@@ -876,19 +876,11 @@ static void del_from_avail_list(struct swap_info_struct *si)
 	spin_unlock(&swap_avail_lock);
 }
 
-static void swap_range_alloc(struct swap_info_struct *si, unsigned long offset,
+static void swap_range_alloc(struct swap_info_struct *si,
 			     unsigned int nr_entries)
 {
-	unsigned int end = offset + nr_entries - 1;
-
-	if (offset == si->lowest_bit)
-		si->lowest_bit += nr_entries;
-	if (end == si->highest_bit)
-		WRITE_ONCE(si->highest_bit, si->highest_bit - nr_entries);
 	WRITE_ONCE(si->inuse_pages, si->inuse_pages + nr_entries);
 	if (si->inuse_pages == si->pages) {
-		si->lowest_bit = si->max;
-		si->highest_bit = 0;
 		del_from_avail_list(si);
 
 		if (vm_swap_full())
@@ -921,15 +913,8 @@ static void swap_range_free(struct swap_info_struct *si, unsigned long offset,
 	for (i = 0; i < nr_entries; i++)
 		clear_bit(offset + i, si->zeromap);
 
-	if (offset < si->lowest_bit)
-		si->lowest_bit = offset;
-	if (end > si->highest_bit) {
-		bool was_full = !si->highest_bit;
-
-		WRITE_ONCE(si->highest_bit, end);
-		if (was_full && (si->flags & SWP_WRITEOK))
-			add_to_avail_list(si);
-	}
+	if (si->inuse_pages == si->pages)
+		add_to_avail_list(si);
 	if (si->flags & SWP_BLKDEV)
 		swap_slot_free_notify =
 			si->bdev->bd_disk->fops->swap_slot_free_notify;
@@ -1035,15 +1020,12 @@ int get_swap_pages(int n_goal, swp_entry_t swp_entries[], int entry_order)
 		plist_requeue(&si->avail_lists[node], &swap_avail_heads[node]);
 		spin_unlock(&swap_avail_lock);
 		spin_lock(&si->lock);
-		if (!si->highest_bit || !(si->flags & SWP_WRITEOK)) {
+		if ((si->inuse_pages == si->pages) || !(si->flags & SWP_WRITEOK)) {
 			spin_lock(&swap_avail_lock);
 			if (plist_node_empty(&si->avail_lists[node])) {
 				spin_unlock(&si->lock);
 				goto nextsi;
 			}
-			WARN(!si->highest_bit,
-			     "swap_info %d in list but !highest_bit\n",
-			     si->type);
 			WARN(!(si->flags & SWP_WRITEOK),
 			     "swap_info %d in list but !SWP_WRITEOK\n",
 			     si->type);
@@ -2425,8 +2407,8 @@ static void _enable_swap_info(struct swap_info_struct *si)
 	 */
 	plist_add(&si->list, &swap_active_head);
 
-	/* add to available list iff swap device is not full */
-	if (si->highest_bit)
+	/* add to available list if swap device is not full */
+	if (si->inuse_pages < si->pages)
 		add_to_avail_list(si);
 }
 
@@ -2590,7 +2572,6 @@ SYSCALL_DEFINE1(swapoff, const char __user *, specialfile)
 	drain_mmlist();
 
 	/* wait for anyone still in scan_swap_map_slots */
-	p->highest_bit = 0;		/* cuts scans short */
 	while (p->flags >= SWP_SCANNING) {
 		spin_unlock(&p->lock);
 		spin_unlock(&swap_lock);
@@ -2925,8 +2906,6 @@ static unsigned long read_swap_header(struct swap_info_struct *si,
 		return 0;
 	}
 
-	si->lowest_bit  = 1;
-
 	maxpages = swapfile_maximum_size;
 	last_page = swap_header->info.last_page;
 	if (!last_page) {
@@ -2943,7 +2922,6 @@ static unsigned long read_swap_header(struct swap_info_struct *si,
 		if ((unsigned int)maxpages == 0)
 			maxpages = UINT_MAX;
 	}
-	si->highest_bit = maxpages - 1;
 
 	if (!maxpages)
 		return 0;
Signature · 1 line
2.47.0
#6 lore raw

PATCH 6/13 mm, swap: clean up plist removal and adding

In reply to Kairui Song

From: Kairui Song <[email protected]>

When swap device is full (inuse_pages == pages), it should be removed from the plist. And if any slot is freed, the swap device should be added back to the plist. On swapoff / swapon, the swap device will also be force removed / added.

This is currently serialized by si->lock, and some historical sanity check code are still here. This commit decouple this from the protection of si->lock and clean it up to prepare for si->lock rework.

Noticing inuse_pages counter is the only thing decides if a device should be removed from or added to the plist (except swapon / swapoff as a special case), and inuse_pages is a very hot counter. So to avoid extra overhead on the counter update hot path, and make it possible to check and update the plist when the counter value changes, embed the plist state into the inuse_pages counter, and turn the counter into an atomic. This way we can check and update the counter with one CAS and avoid any extra synchronization.

If the counter is full (inuse_pages == pages) with the off-list bit unset, try to remove it from the plist. If the counter is not full (inuse_pages != pages) with the off-list bit set, try to add it to the plist. Removing and adding is serialized with lock as well as the bit setting. Ordinary counter updates will be lockless.

---
 include/linux/swap.h |   2 +-
 mm/swapfile.c        | 182 +++++++++++++++++++++++++++++++------------
 2 files changed, 132 insertions(+), 52 deletions(-)
include/linux/swap.h +1 -1
index c0d49dad7a4b..16dcf8bd1a4e 100644
@@ -306,7 +306,7 @@ struct swap_info_struct {
 					/* list of cluster that are fragmented or contented */
 	unsigned int frag_cluster_nr[SWAP_NR_ORDERS];
 	unsigned int pages;		/* total of usable pages of swap */
-	unsigned int inuse_pages;	/* number of those currently in use */
+	atomic_long_t inuse_pages;	/* number of those currently in use */
 	struct percpu_cluster __percpu *percpu_cluster; /* per cpu's swap location */
 	struct rb_root swap_extent_root;/* root of the swap extent rbtree */
 	struct block_device *bdev;	/* swap device or bdev of swap file */
mm/swapfile.c +131 -51

332 lines not on this page · raw message

Signature · 1 line
2.47.0
#7 lore raw

PATCH 7/13 mm, swap: hold a reference of si during scan and clean up flags

In reply to Kairui Song

From: Kairui Song <[email protected]>

The flag SWP_SCANNING was used as an indicator of whether a device is being scanned, and prevents swap off. But it's already no longer used. The only thing protects the scanning now is the si lock.

However allocation path may drop the si lock, in theory this could leaf to UAF.

So clean this up, just hold a reference for whole allocation path. So per CPU counter killing will wait for existing scan and other usage. The flag SWP_SCANNING can also be dropped.

---
 include/linux/swap.h |  1 -
 mm/swapfile.c        | 62 +++++++++++++++++++++++---------------------
 2 files changed, 33 insertions(+), 30 deletions(-)
include/linux/swap.h +0 -1
index 16dcf8bd1a4e..1651174959c8 100644
@@ -219,7 +219,6 @@ enum {
 	SWP_STABLE_WRITES = (1 << 11),	/* no overwrite PG_writeback pages */
 	SWP_SYNCHRONOUS_IO = (1 << 12),	/* synchronous IO is efficient */
 					/* add others here before... */
-	SWP_SCANNING	= (1 << 14),	/* refcount in scan_swap_map */
 };
 
 #define SWAP_CLUSTER_MAX 32UL
mm/swapfile.c +33 -29

93 lines not on this page · raw message

Signature · 1 line
2.47.0
#8 lore raw

PATCH 8/13 mm, swap: use an enum to define all cluster flags and wrap flags changes

In reply to Kairui Song

From: Kairui Song <[email protected]>

Currently we are only using flags to indicate which list the cluster is on, using one bit for each list type might be a waste as the list type grows we will consume too much bits. And current the mixed usage of "&" and "==" is a bit confusing.

Make it clean by using an enum to define all possible cluster status, only an off-list cluster will have the NONE (0) flag. And use a wrapper to annotate and sanitize all flag setting and list movement.

---
 include/linux/swap.h | 17 +++++++---
 mm/swapfile.c        | 76 +++++++++++++++++++++++---------------------
 2 files changed, 53 insertions(+), 40 deletions(-)
include/linux/swap.h +13 -4
index 1651174959c8..75fc2da1767d 100644
@@ -256,10 +256,19 @@ struct swap_cluster_info {
 	u8 order;
 	struct list_head list;
 };
-#define CLUSTER_FLAG_FREE 1 /* This cluster is free */
-#define CLUSTER_FLAG_NONFULL 2 /* This cluster is on nonfull list */
-#define CLUSTER_FLAG_FRAG 4 /* This cluster is on nonfull list */
-#define CLUSTER_FLAG_FULL 8 /* This cluster is on full list */
+
+/*
+ * All on-list cluster must have a non-zero flag.
+ */
+enum swap_cluster_flags {
+	CLUSTER_FLAG_NONE = 0, /* For temporary off-list cluster */
+	CLUSTER_FLAG_FREE,
+	CLUSTER_FLAG_NONFULL,
+	CLUSTER_FLAG_FRAG,
+	CLUSTER_FLAG_FULL,
+	CLUSTER_FLAG_DISCARD,
+	CLUSTER_FLAG_MAX,
+};
 
 /*
  * The first page in the swap file is the swap header, which is always marked
mm/swapfile.c +40 -36

137 lines not on this page · raw message

Signature · 1 line
2.47.0
#9 lore raw

PATCH 9/13 mm, swap: reduce contention on device lock

In reply to Kairui Song

From: Kairui Song <[email protected]>

Currently swap locking is mainly composed of two locks, cluster lock (ci->lock) and device lock (si->lock).

Cluster lock is much more fine-grained, so it will be best to use ci->lock instead of si->lock as much as possible.

Following the new cluster allocator design, many operation doesn't need to touch si->lock at all. In practise, we only need to take si->lock when moving clusters between lists.

To archive it, this commit reworked the locking pattern of all si->lock and ci->lock users, eliminated all usage of ci->lock inside si->lock, introduce new design to avoid touching si->lock as much as possible.

For minimal contention for allocation and easier understanding, two ideas are introduced with the corresponding helpers: isolation and relocation:

- Clusters will be isolated from list upon being scanned for allocation, so scanning of on-list cluster no longer need to hold the si->lock except the very moment, and hence removed the ci->lock usage inside si->lock.

In the new allocator design, one cluster always get moved after scanning (free -> nonfull, nonfull -> frag, frag -> frag tail) so this introduces no extra overhead. This also greatly reduced the contention of both si->lock and ci->lock as other CPUs won't walk onto the same cluster by iterating the list.

The off-list time window of a cluster is also minimal, one CPU can at most hold one cluster while scanning the 512 entries on it, which we used to busy wait with a spin lock.

This is done with cluster_isolate_lock on scanning of a new cluster.

Note: Scanning of per CPU cluster is a special case, it doesn't isolation the cluster. That's because it doesn't need to hold the si->lock at all, it simply acquire the ci->lock of previously used cluster and use it.

- Cluster will be relocated after allocation or freeing according to it's count and status.

Allocations no longer holds si->lock now, and may drop ci->lock for reclaim, so the cluster could be moved to anywhere. Besides, isolation clears all flags when it takes the cluster off list (The flag must be in-sync with list status, so cluster users don't need to touch si->lock for checking its list status. This is important for reducing contention on si->lock). So the cluster have to be relocated according to its usage after being allocation to the right list.

This is done with relocate_cluster after allocation, or [partial_]free_cluster after freeing.

Now except swapon / swapoff and discard, isolation and relocation are the only two places that need to take si->lock. And as each CPU will keep using its per-CPU cluster as much as possible and a cluster have 512 entries to be consumed, si->lock is rarely touched.

The lock contention of si->lock is now barely observable. Test with build linux kernel with defconfig showed huge performance improvement:

tiem make -j96 / 768M memcg, 4K pages, 10G ZRAM, on Intel 8255C:
Before:
Sys time: 73578.30, Real time: 864.05
After: (-50.7% sys time, -44.8% real time)
Sys time: 36227.49, Real time: 476.66

time make -j96 / 1152M memcg, 64K mTHP, 10G ZRAM, on Intel 8255C:
(avg of 4 test run)
Before:
Sys time: 74044.85, Real time: 846.51 hugepages-64kB/stats/swpout: 1735216 hugepages-64kB/stats/swpout_fallback: 430333

After: (-40.4% sys time, -37.1% real time)
Sys time: 44160.56, Real time: 532.07
hugepages-64kB/stats/swpout: 1786288
hugepages-64kB/stats/swpout_fallback: 243384

time make -j32 / 512M memcg, 4K pages, 5G ZRAM, on AMD 7K62:
Before:
Sys time: 8098.21, Real time: 401.3
After: (-22.6% sys time, -12.8% real time )
Sys time: 6265.02, Real time: 349.83

The allocation success rate also slightly improved as we sanitized the usage of clusters with new defined helpers and locks, so temporarily dropped si->lock or ci->lock won't cause cluster order shuffle.

---
 include/linux/swap.h |   5 +-
 mm/swapfile.c        | 418 ++++++++++++++++++++++++-------------------
 2 files changed, 239 insertions(+), 184 deletions(-)
include/linux/swap.h +4 -1
index 75fc2da1767d..a3b5d74b095a 100644
@@ -265,6 +265,8 @@ enum swap_cluster_flags {
 	CLUSTER_FLAG_FREE,
 	CLUSTER_FLAG_NONFULL,
 	CLUSTER_FLAG_FRAG,
+	/* Clusters with flags above are allocatable */
+	CLUSTER_FLAG_USABLE = CLUSTER_FLAG_FRAG,
 	CLUSTER_FLAG_FULL,
 	CLUSTER_FLAG_DISCARD,
 	CLUSTER_FLAG_MAX,
@@ -290,6 +292,7 @@ enum swap_cluster_flags {
  * throughput.
  */
 struct percpu_cluster {
+	local_lock_t lock; /* Protect the percpu_cluster above */
 	unsigned int next[SWAP_NR_ORDERS]; /* Likely next allocation offset */
 };
 
@@ -312,7 +315,7 @@ struct swap_info_struct {
 					/* list of cluster that contains at least one free slot */
 	struct list_head frag_clusters[SWAP_NR_ORDERS];
 					/* list of cluster that are fragmented or contented */
-	unsigned int frag_cluster_nr[SWAP_NR_ORDERS];
+	atomic_long_t frag_cluster_nr[SWAP_NR_ORDERS];
 	unsigned int pages;		/* total of usable pages of swap */
 	atomic_long_t inuse_pages;	/* number of those currently in use */
 	struct percpu_cluster __percpu *percpu_cluster; /* per cpu's swap location */
mm/swapfile.c +235 -183

735 lines not on this page · raw message

Signature · 1 line
2.47.0
#10 lore raw

PATCH 10/13 mm, swap: simplify percpu cluster updating

In reply to Kairui Song

From: Kairui Song <[email protected]>

Instead of using a returning argument, we can simply store the next cluster offset to the fixed percpu location, which reduce the stack usage and simplify the function:

Object size:
./scripts/bloat-o-meter mm/swapfile.o mm/swapfile.o.new add/remove: 0/0 grow/shrink: 0/2 up/down: 0/-271 (-271)
Function old new delta
get_swap_pages 2847 2733 -114
alloc_swap_scan_cluster 894 737 -157
Total: Before=30833, After=30562, chg -0.88%

Stack usage:
Before:
swapfile.c:1190:5:get_swap_pages 240 static

After:
swapfile.c:1185:5:get_swap_pages 216 static

---
 include/linux/swap.h |  4 ++--
 mm/swapfile.c        | 57 ++++++++++++++++++++------------------------
 2 files changed, 28 insertions(+), 33 deletions(-)
include/linux/swap.h +2 -2
index a3b5d74b095a..0e6c6bb385f0 100644
@@ -276,9 +276,9 @@ enum swap_cluster_flags {
  * The first page in the swap file is the swap header, which is always marked
  * bad to prevent it from being allocated as an entry. This also prevents the
  * cluster to which it belongs being marked free. Therefore 0 is safe to use as
- * a sentinel to indicate next is not valid in percpu_cluster.
+ * a sentinel to indicate an entry is not valid.
  */
-#define SWAP_NEXT_INVALID	0
+#define SWAP_ENTRY_INVALID	0
 
 #ifdef CONFIG_THP_SWAP
 #define SWAP_NR_ORDERS		(PMD_ORDER + 1)
mm/swapfile.c +26 -31

137 lines not on this page · raw message

Signature · 1 line
2.47.0
#11 lore raw

PATCH 11/13 mm, swap: introduce a helper for retrieving cluster from offset

In reply to Kairui Song

From: Kairui Song <[email protected]>

It's a common operation to retrieve the cluster info from offset, introduce a helper for this.

---
 mm/swapfile.c | 16 +++++++++++-----
 1 file changed, 11 insertions(+), 5 deletions(-)
mm/swapfile.c +11 -5
index f529e2ce2019..f25d697f6736 100644
@@ -423,6 +423,12 @@ static inline unsigned int cluster_index(struct swap_info_struct *si,
 	return ci - si->cluster_info;
 }
 
+static inline struct swap_cluster_info *offset_to_cluster(struct swap_info_struct *si,
+							  unsigned long offset)
+{
+	return &si->cluster_info[offset / SWAPFILE_CLUSTER];
+}
+
 static inline unsigned int cluster_offset(struct swap_info_struct *si,
 					  struct swap_cluster_info *ci)
 {
@@ -434,7 +440,7 @@ static inline struct swap_cluster_info *lock_cluster(struct swap_info_struct *si
 {
 	struct swap_cluster_info *ci;
 
-	ci = &si->cluster_info[offset / SWAPFILE_CLUSTER];
+	ci = offset_to_cluster(si, offset);
 	spin_lock(&ci->lock);
 
 	return ci;
@@ -756,7 +762,7 @@ static unsigned int alloc_swap_scan_cluster(struct swap_info_struct *si,
 	bool need_reclaim, ret;
 	struct swap_cluster_info *ci;
 
-	ci = &si->cluster_info[offset / SWAPFILE_CLUSTER];
+	ci = offset_to_cluster(si, offset);
 	lockdep_assert_held(&ci->lock);
 
 	if (end < nr_pages || ci->count + nr_pages > SWAPFILE_CLUSTER)
@@ -1457,10 +1463,10 @@ static void swap_entry_range_free(struct swap_info_struct *si, swp_entry_t entry
 	unsigned char *map_end = map + nr_pages;
 	struct swap_cluster_info *ci;
 
-	/* It should never free entries across different clusters */
-	VM_BUG_ON((offset / SWAPFILE_CLUSTER) != ((offset + nr_pages - 1) / SWAPFILE_CLUSTER));
-
 	ci = lock_cluster(si, offset);
+
+	/* It should never free entries across different clusters */
+	VM_BUG_ON(ci != offset_to_cluster(si, offset + nr_pages - 1));
 	VM_BUG_ON(cluster_is_free(ci));
 	VM_BUG_ON(ci->count < nr_pages);
 
Signature · 1 line
2.47.0
#12 lore raw

PATCH 12/13 mm, swap: use a global swap cluster for non-rotation device

In reply to Kairui Song

From: Kairui Song <[email protected]>

Non-rotation (SSD / ZRAM) device can tolerate fragmentations so the goal of SWAP allocator is to avoid contention of clusters. So it used a per-CPU cluster design, and each CPU will be using a different cluster as much as possible.

But HDD is very sensitive to fragmentations, contention is trivial compared to this. So just use one global cluster instead. This ensured each order will be wring to a same cluster as much as possible, which helps to make the IO more continuous.

This ensures the performance of cluster allocator is as good as the old allocator. Test after this commit compared to before this series:

make -j32 with tinyconfig, using 1G memcg limit and HDD swap:

Before this series:
114.44user 29.11system 39:42.90elapsed 6%CPU (0avgtext+0avgdata 157284maxresident)k 2901232inputs+0outputs (238877major+4227640minor)pagefaults

After this commit:
113.90user 23.81system 38:11.77elapsed 6%CPU (0avgtext+0avgdata 157260maxresident)k 2548728inputs+0outputs (235471major+4238110minor)pagefaults

---
 include/linux/swap.h |  2 ++
 mm/swapfile.c        | 48 ++++++++++++++++++++++++++++++++------------
 2 files changed, 37 insertions(+), 13 deletions(-)
include/linux/swap.h +2 -0
index 0e6c6bb385f0..9898b1881d4d 100644
@@ -319,6 +319,8 @@ struct swap_info_struct {
 	unsigned int pages;		/* total of usable pages of swap */
 	atomic_long_t inuse_pages;	/* number of those currently in use */
 	struct percpu_cluster __percpu *percpu_cluster; /* per cpu's swap location */
+	struct percpu_cluster *global_cluster; /* Use one global cluster for rotating device */
+	spinlock_t global_cluster_lock;	/* Serialize usage of global cluster */
 	struct rb_root swap_extent_root;/* root of the swap extent rbtree */
 	struct block_device *bdev;	/* swap device or bdev of swap file */
 	struct file *swap_file;		/* seldom referenced */
mm/swapfile.c +35 -13

87 lines not on this page · raw message

Signature · 1 line
2.47.0
#13 lore raw

PATCH 13/13 mm, swap_slots: remove slot cache for freeing path

In reply to Kairui Song

From: Kairui Song <[email protected]>

The slot cache for freeing path is mostly for reducing the overhead of si->lock. As we have basically eliminated the si->lock usage for freeing path, it can be just removed.

This helps simplify the code, and avoids swap entries from being hold in cache upon freeing. The delayed freeing of entries have been causing trouble for further optimizations for zswap [1] and in theory will also cause more fragmentation, and extra overhead.

Test with build linux kernel showed both performance and fragmentation is better without the cache:

tiem make -j96 / 768M memcg, 4K pages, 10G ZRAM, avg of 4 test run::
Before:
Sys time: 36047.78, Real time: 472.43
After: (-7.6% sys time, -7.3% real time)
Sys time: 33314.76, Real time: 437.67

time make -j96 / 1152M memcg, 64K mTHP, 10G ZRAM, avg of 4 test run:
Before:
Sys time: 46859.04, Real time: 562.63 hugepages-64kB/stats/swpout: 1783392 hugepages-64kB/stats/swpout_fallback: 240875
After: (-23.3% sys time, -21.3% real time)
Sys time: 35958.87, Real time: 442.69 hugepages-64kB/stats/swpout: 1866267 hugepages-64kB/stats/swpout_fallback: 158330

Sequential SWAP should be also slightly faster, tests didn't show a measurable difference though, at least no regression:

Swapin 4G zero page on ZRAM (time in us):
Before (avg. 1923756)
1912391 1927023 1927957 1916527 1918263 1914284 1934753 1940813 1921791 After (avg. 1922290):
1919101 1925743 1916810 1917007 1923930 1935152 1917403 1923549 1921913

---
 include/linux/swap_slots.h |  3 --
 mm/swap_slots.c            | 78 +++++----------------------------
 mm/swapfile.c              | 89 +++++++++++++++-----------------------
 3 files changed, 44 insertions(+), 126 deletions(-)
include/linux/swap_slots.h +0 -3
index 15adfb8c813a..840aec3523b2 100644
@@ -16,15 +16,12 @@ struct swap_slots_cache {
 	swp_entry_t	*slots;
 	int		nr;
 	int		cur;
-	spinlock_t	free_lock;  /* protects slots_ret, n_ret */
-	swp_entry_t	*slots_ret;
 	int		n_ret;
 };
 
 void disable_swap_slots_cache_lock(void);
 void reenable_swap_slots_cache_unlock(void);
 void enable_swap_slots_cache(void);
-void free_swap_slot(swp_entry_t entry);
 
 extern bool swap_slot_cache_enabled;
 
mm/swap_slots.c +10 -68

160 lines not on this page · raw message

mm/swapfile.c +34 -55

170 lines not on this page · raw message

Signature · 1 line
2.47.0
#14 lore raw
In reply to Kairui Song

Hi, Kairui,

Quoted 12 lines · Kairui Song

From: Kairui Song <[email protected]>

This series improved the swap allocator performance greatly by reworking the locking design and simplify a lot of code path.

This is follow up of previous swap cluster allocator series:
https://lore.kernel.org/linux-mm/[email protected]/

And this series is based on an follow up fix of the swap cluster allocator:
https://lore.kernel.org/linux-mm/[email protected]/

Quoted 8 lines

This is part of the new swap allocator work item discussed in Chris's "Swap Abstraction" discussion at LSF/MM 2024, and "mTHP and swap allocator" discussion at LPC 2024.

Previous series introduced a fully cluster based allocation algorithm, this series completely get rid of the old allocation path and makes the allocator avoid grabbing the si->lock unless needed. This bring huge performance gain and get rid of slot cache on freeing path.

Great!

Quoted 3 lines

Currently, swap locking is mainly composed of two locks, cluster lock (ci->lock) and device lock (si->lock). The device lock is widely used to protect many things, causing it to be the main bottleneck for SWAP.

Device lock can be confusing with another device lock for struct device. Better to call it swap device lock?

Quoted 15 lines

Cluster lock is much more fine-grained, so it will be best to use ci->lock instead of si->lock as much as possible.

perf lock indicates this issue clearly. Doing linux kernel build using tmpfs and ZRAM with limited memory (make -j64 with 1G memcg and 4k pages), result of "perf lock contention -ab sleep 3":

contended total wait max wait avg wait type caller

     34948     53.63 s       7.11 ms      1.53 ms     spinlock   free_swap_and_cache_nr+0x350
     16569     40.05 s       6.45 ms      2.42 ms     spinlock   get_swap_pages+0x231
     11191     28.41 s       7.03 ms      2.54 ms     spinlock   swapcache_free_entries+0x59
      4147     22.78 s     122.66 ms      5.49 ms     spinlock   page_vma_mapped_walk+0x6f3
      4595      7.17 s       6.79 ms      1.56 ms     spinlock   swapcache_free_entries+0x59
    406027      2.74 s       2.59 ms      6.74 us     spinlock   list_lru_add+0x39
Quoted 4 lines

...snip...

The top 5 caller are all users of si->lock, total wait time up sums to several minutes in the 3 seconds time window.

Can you show results of perf record -g, perf report -g too? I have interest to check hot spot shifting too.

Quoted 9 lines

Following the new allocator design, many operation doesn't need to touch si->lock at all. We only need to take si->lock when doing operations across multiple clusters (eg. changing the cluster list), other operations only need to take ci->lock. So ideally allocator should always take ci->lock first, then, if needed, take si->lock. But due to historical reasons, ci->lock is used inside si->lock by design, causing lock inversion if we simply try to acquire si->lock after acquiring ci->lock.

Quoted 6 lines

This series audited all si->lock usage, simplify legacy codes, eliminate usage of si->lock as much as possible by introducing new designs based on the new cluster allocator.

Old HDD allocation codes are removed, cluster allocator is adapted with small changes for HDD usage, test is looking OK.

I think that it's a good idea to remove HDD allocation specific code. Can you check the performance of swapping to HDD? However, I understand that many people have no HDD in hand.

Quoted 36 lines

And this also removed slot cache for freeing path. The performance is better without it, and this enables other clean up and optimizations as discussed before:
https://lore.kernel.org/all/CAMgjq7ACohT_uerSz8E_994ZZCv709Zor+43hdmesW_59W1BWw@mail.gmail.com/

After this series, lock contention on si->lock is nearly unobservable with perf lock with the same test above :

contended total wait max wait avg wait type caller
... snip ...

         91    204.62 us      4.51 us      2.25 us     spinlock   cluster_move+0x2e
  ... snip ...
         47    125.62 us      4.47 us      2.67 us     spinlock   cluster_move+0x2e
  ... snip ...
         23     63.15 us      3.95 us      2.74 us     spinlock   cluster_move+0x2e
  ... snip ...
         17     41.26 us      4.58 us      2.43 us     spinlock   cluster_isolate_lock+0x1d

... snip ...

cluster_move and cluster_isolate_lock are basically the only users of si->lock now, performance gain is huge with reduced LOC.

Tests
===

Build kernel with defconfig on tmpfs with ZRAM as swap:
---

Running a test matrix which is scaled up progressive for a intuitive result. The test are ran on top of tmpfs, using memory cgroup for memory limitation, on a 48c96t system.

12 test run for each case, it can be seen clearly that as concurrent job number goes higher the performance gain is higher, the performance is higher even with low concurrency.

Quoted 12 lines
   make -j<NR>     |   System Time (seconds)  |   Total Time (seconds)
 (NR / Mem / ZRAM) | (Before / After / Delta) | (Before / After / Delta)
 With 4k pages only:
  6 / 192M / 3G    |    5258 /  5235 / -0.3%  |    1420 /  1414 / -0.3%
 12 / 256M / 4G    |    5518 /  5337 / -3.3%  |     758 /   742 / -2.1%
 24 / 384M / 5G    |    7091 /  5766 / -18.7% |     476 /   422 / -11.3%
 48 / 768M / 7G    |   11139 /  5831 / -47.7% |     330 /   221 / -33.0%
 96 / 1.5G / 10G   |   21303 / 11353 / -46.7% |     283 /   180 / -36.4%
 With 64k mTHP:
 24 / 512M / 5G    |    5104 /  4641 / -18.7% |     376 /   358 / -4.8%
 48 /   1G / 7G    |    8693 /  4662 / -18.7% |     257 /   176 / -31.5%
 96 /   2G / 10G   |   17056 / 10263 / -39.8% |     234 /   169 / -27.8%

How much is the swap in/out throughput before/after the change?

When I worked on swap in/out performance before, the hot spot shifts from swap related code to LRU lock and zone lock. Things may change a lot now.

If zram is used as swap device, the hot spot may become compression/decompression after solving the swap lock contention. To stress swap subsystem further, we may use a ram disk as swap. Previously, we have used a simulated pmem device (backed by DRAM). That can be setup as in,

https://pmem.io/blog/2016/02/how-to-emulate-persistent-memory/

After creating the raw block device: /dev/pmem0, we can do

$ mkswap /dev/pmem0
$ swapon /dev/pmem0

Can you use something similar if necessary?

Quoted 22 lines

With more aggressive setup, it shows clearly both the performance and fragmentation are better:

tiem make -j96 / 768M memcg, 4K pages, 10G ZRAM, on Intel 8255C * 2:
(avg of 4 test run)
Before:
Sys time: 73578.30, Real time: 864.05
tiem make -j96 / 1G memcg, 4K pages, 10G ZRAM:
After: (-54.7% sys time, -49.3% real time)
Sys time: 33314.76, Real time: 437.67

time make -j96 / 1152M memcg, 64K mTHP, 10G ZRAM, on Intel 8255C * 2:
(avg of 4 test run)
Before:
Sys time: 74044.85, Real time: 846.51 hugepages-64kB/stats/swpout: 1735216 hugepages-64kB/stats/swpout_fallback: 430333
After: (-51.4% sys time, -47.7% real time, -63.2% mTHP failure) Sys time: 35958.87, Real time: 442.69 hugepages-64kB/stats/swpout: 1866267 hugepages-64kB/stats/swpout_fallback: 158330

Quoted 3 lines

There is a up to 54.7% improvement for build kernel test, and lower fragmentation rate. Performance improvement should be even larger for micro benchmarks

Very good result!

Quoted 58 lines

Build kernel with tinyconfig on tmpfs with HDD as swap:
---

This test is similar to above, but HDD test is very noisy and slow, the deviation is huge, so just use tinyconfig instead and take the median test result of 3 test run, which looks OK:

Before this series:
114.44user 29.11system 39:42.90elapsed 6%CPU 2901232inputs+0outputs (238877major+4227640minor)pagefaults

After this commit:
113.90user 23.81system 38:11.77elapsed 6%CPU 2548728inputs+0outputs (235471major+4238110minor)pagefaults

Single thread SWAP:
---

Sequential SWAP should also be slightly faster as we removed a lot of unnecessary parts. Test using micro benchmark for swapout/in 4G zero memory using ZRAM, 10 test runs:

Swapout Before (avg. 3359304):
3353796 3358551 3371305 3356043 3367524 3355303 3355924 3354513 3360776

Swapin Before (avg. 1928698):
1920283 1927183 1934105 1921373 1926562 1938261 1927726 1928636 1934155

Swapout After (avg. 3347511, -0.4%):
3337863 3347948 3355235 3339081 3333134 3353006 3354917 3346055 3360359

Swapin After (avg. 1922290, -0.3%):
1919101 1925743 1916810 1917007 1923930 1935152 1917403 1923549 1921913

Worth noticing the patch "mm, swap: use a global swap cluster for non-rotation device" introduced minor overhead for certain tests (see the test results in commit message), but the gain from later commit covered that, it can be further improved later.

Kairui Song (13):
mm, swap: minor clean up for swap entry allocation mm, swap: fold swap_info_get_cont in the only caller mm, swap: remove old allocation path for HDD
mm, swap: use cluster lock for HDD
mm, swap: clean up device availability check
mm, swap: clean up plist removal and adding
mm, swap: hold a reference of si during scan and clean up flags mm, swap: use an enum to define all cluster flags and wrap flags
changes
mm, swap: reduce contention on device lock
mm, swap: simplify percpu cluster updating
mm, swap: introduce a helper for retrieving cluster from offset mm, swap: use a global swap cluster for non-rotation device mm, swap_slots: remove slot cache for freeing path

Quoted 8 lines
 fs/btrfs/inode.c           |    1 -
 fs/iomap/swapfile.c        |    1 -
 include/linux/swap.h       |   36 +-
 include/linux/swap_slots.h |    3 -
 mm/page_io.c               |    1 -
 mm/swap_slots.c            |   78 +--
 mm/swapfile.c              | 1198 ++++++++++++++++--------------------
 7 files changed, 558 insertions(+), 760 deletions(-)

--
Best Regards,
Huang, Ying

#15 lore raw
Quoted 2 lines · Ying

Hi, Kairui,

Hi Ying,

Quoted 24 lines
Quoted 12 lines · Kairui Song

From: Kairui Song <[email protected]>

This series improved the swap allocator performance greatly by reworking the locking design and simplify a lot of code path.

This is follow up of previous swap cluster allocator series:
https://lore.kernel.org/linux-mm/[email protected]/

And this series is based on an follow up fix of the swap cluster allocator:
https://lore.kernel.org/linux-mm/[email protected]/

Quoted 8 lines

This is part of the new swap allocator work item discussed in Chris's "Swap Abstraction" discussion at LSF/MM 2024, and "mTHP and swap allocator" discussion at LPC 2024.

Previous series introduced a fully cluster based allocation algorithm, this series completely get rid of the old allocation path and makes the allocator avoid grabbing the si->lock unless needed. This bring huge performance gain and get rid of slot cache on freeing path.

Quoted 8 lines

Great!

Quoted 3 lines

Currently, swap locking is mainly composed of two locks, cluster lock (ci->lock) and device lock (si->lock). The device lock is widely used to protect many things, causing it to be the main bottleneck for SWAP.

Device lock can be confusing with another device lock for struct device. Better to call it swap device lock?

Good idea, I'll use the term swap device lock then.

Quoted 18 lines
Quoted 18 lines

Cluster lock is much more fine-grained, so it will be best to use ci->lock instead of si->lock as much as possible.

perf lock indicates this issue clearly. Doing linux kernel build using tmpfs and ZRAM with limited memory (make -j64 with 1G memcg and 4k pages), result of "perf lock contention -ab sleep 3":

contended total wait max wait avg wait type caller

     34948     53.63 s       7.11 ms      1.53 ms     spinlock   free_swap_and_cache_nr+0x350
     16569     40.05 s       6.45 ms      2.42 ms     spinlock   get_swap_pages+0x231
     11191     28.41 s       7.03 ms      2.54 ms     spinlock   swapcache_free_entries+0x59
      4147     22.78 s     122.66 ms      5.49 ms     spinlock   page_vma_mapped_walk+0x6f3
      4595      7.17 s       6.79 ms      1.56 ms     spinlock   swapcache_free_entries+0x59
    406027      2.74 s       2.59 ms      6.74 us     spinlock   list_lru_add+0x39

...snip...

Quoted 5 lines
Quoted 2 lines

The top 5 caller are all users of si->lock, total wait time up sums to several minutes in the 3 seconds time window.

Can you show results of perf record -g, perf report -g too? I have interest to check hot spot shifting too.

Sure. I think perf lock result is already good enough and cleaner. My test environment are mostly VM based so spinlock slow path may get offloaded to host, and can't be see by perf record, I collected following data after disabled paravirt spinlock:

The time consumption and stack trace of a page fault before:
- 78.45% 0.17% cc1 [kernel.kallsyms]
[k] asm_exc_page_fault
- 78.28% asm_exc_page_fault
- 78.18% exc_page_fault
- 78.17% do_user_addr_fault
- 78.09% handle_mm_fault
- 78.06% __handle_mm_fault
- 69.69% do_swap_page
- 55.87% alloc_swap_folio
- 55.60% mem_cgroup_swapin_charge_folio
- 55.48% charge_memcg
- 55.45% try_charge_memcg
- 55.36% try_to_free_mem_cgroup_pages
- do_try_to_free_pages
- 55.35% shrink_node
- 55.27% shrink_lruvec
- 55.13% try_to_shrink_lruvec
- 54.79% evict_folios
- 54.35% shrink_folio_list
- 30.01% add_to_swap
- 29.77%
folio_alloc_swap
- 29.50%
get_swap_pages

25.03% queued_spin_lock_slowpath
- 2.71%
alloc_swap_scan_cluster

1.80% queued_spin_lock_slowpath

                                                                  +

0.89% __try_to_reclaim_swap
- 1.74%
swap_reclaim_full_clusters

1.74% queued_spin_lock_slowpath
- 10.88%
try_to_unmap_flush_dirty
- 10.87%
arch_tlbbatch_flush
- 10.85%
on_each_cpu_cond_mask

smp_call_function_many_cond
+ 7.45% pageout
+ 2.71% try_to_unmap_flush
+ 1.90% try_to_unmap
+ 0.78% folio_referenced
- 9.41% cluster_swap_free_nr
- 9.39% free_swap_slot
- 9.35% swapcache_free_entries
8.40% queued_spin_lock_slowpath 0.93% swap_entry_range_free
- 3.61% swap_read_folio_bdev_sync
- 3.55% submit_bio_wait
- 3.51% submit_bio_noacct_nocheck
+ 3.46% __submit_bio
+ 7.71% do_pte_missing
+ 0.61% wp_page_copy

The queued_spin_lock_slowpath above is the si->lock, and there are multiple users of it so the total overhead is higher than shown.

After:
- 75.05% 0.43% cc1 [kernel.kallsyms]
[k] asm_exc_page_fault
- 74.62% asm_exc_page_fault
- 74.36% exc_page_fault
- 74.34% do_user_addr_fault
- 74.10% handle_mm_fault
- 73.96% __handle_mm_fault
- 67.55% do_swap_page
- 45.92% alloc_swap_folio
- 45.03% mem_cgroup_swapin_charge_folio
- 44.58% charge_memcg
- 44.44% try_charge_memcg
- 44.12% try_to_free_mem_cgroup_pages
- do_try_to_free_pages
- 44.10% shrink_node
- 43.86% shrink_lruvec
- 41.92% try_to_shrink_lruvec
- 40.67% evict_folios
- 37.12% shrink_folio_list
- 20.88% pageout
+ 20.02% swap_writepage
+ 0.72% shmem_writepage
- 4.08% add_to_swap
- 2.48%
folio_alloc_swap
- 2.12%
__mem_cgroup_try_charge_swap
- 1.47%
swap_cgroup_record

                                                                  +

1.32% _raw_spin_lock_irqsave
- 1.56%
add_to_swap_cache
- 1.04% xas_store
+ 1.01%
workingset_update_node
+ 3.97%
try_to_unmap_flush_dirty
+ 3.51% folio_referenced
+ 2.24% __remove_mapping
+ 1.16% try_to_unmap
+ 0.52% try_to_unmap_flush
2.50%
queued_spin_lock_slowpath
0.79% scan_folios
+ 1.20% try_to_inc_max_seq
+ 1.92% lru_add_drain
+ 0.73% vma_alloc_folio_noprof
- 9.81% swap_read_folio_bdev_sync
- 9.61% submit_bio_wait
+ 9.49% submit_bio_noacct_nocheck
- 8.06% cluster_swap_free_nr
- 8.02% swap_entry_range_free
+ 3.92% __mem_cgroup_uncharge_swap
+ 2.90% zram_slot_free_notify
0.58% clear_shadow_from_swap_cache
- 1.32% __folio_batch_add_and_move
- 1.30% folio_batch_move_lru
+ 1.10% folio_lruvec_lock_irqsave

spin_lock usage is much lower.

I prefer the perf lock output as it shows the exact time and user of locks.

Quoted 14 lines
Quoted 14 lines

Following the new allocator design, many operation doesn't need to touch si->lock at all. We only need to take si->lock when doing operations across multiple clusters (eg. changing the cluster list), other operations only need to take ci->lock. So ideally allocator should always take ci->lock first, then, if needed, take si->lock. But due to historical reasons, ci->lock is used inside si->lock by design, causing lock inversion if we simply try to acquire si->lock after acquiring ci->lock.

This series audited all si->lock usage, simplify legacy codes, eliminate usage of si->lock as much as possible by introducing new designs based on the new cluster allocator.

Quoted 6 lines
Quoted 2 lines

Old HDD allocation codes are removed, cluster allocator is adapted with small changes for HDD usage, test is looking OK.

I think that it's a good idea to remove HDD allocation specific code. Can you check the performance of swapping to HDD? However, I understand that many people have no HDD in hand.

It's not hard to make cluster allocator work well with HDD in theory, see the commit "mm, swap: use a global swap cluster for non-rotation device".
The testing is not very reliable though, I found HDD swap performance is very unstable because of the IO pattern of HDD, so it's just a best effort try.

Quoted 49 lines
Quoted 36 lines

And this also removed slot cache for freeing path. The performance is better without it, and this enables other clean up and optimizations as discussed before:
https://lore.kernel.org/all/CAMgjq7ACohT_uerSz8E_994ZZCv709Zor+43hdmesW_59W1BWw@mail.gmail.com/

After this series, lock contention on si->lock is nearly unobservable with perf lock with the same test above :

contended total wait max wait avg wait type caller
... snip ...

         91    204.62 us      4.51 us      2.25 us     spinlock   cluster_move+0x2e
  ... snip ...
         47    125.62 us      4.47 us      2.67 us     spinlock   cluster_move+0x2e
  ... snip ...
         23     63.15 us      3.95 us      2.74 us     spinlock   cluster_move+0x2e
  ... snip ...
         17     41.26 us      4.58 us      2.43 us     spinlock   cluster_isolate_lock+0x1d

... snip ...

cluster_move and cluster_isolate_lock are basically the only users of si->lock now, performance gain is huge with reduced LOC.

Tests
===

Build kernel with defconfig on tmpfs with ZRAM as swap:
---

Running a test matrix which is scaled up progressive for a intuitive result. The test are ran on top of tmpfs, using memory cgroup for memory limitation, on a 48c96t system.

12 test run for each case, it can be seen clearly that as concurrent job number goes higher the performance gain is higher, the performance is higher even with low concurrency.

Quoted 12 lines
   make -j<NR>     |   System Time (seconds)  |   Total Time (seconds)
 (NR / Mem / ZRAM) | (Before / After / Delta) | (Before / After / Delta)
 With 4k pages only:
  6 / 192M / 3G    |    5258 /  5235 / -0.3%  |    1420 /  1414 / -0.3%
 12 / 256M / 4G    |    5518 /  5337 / -3.3%  |     758 /   742 / -2.1%
 24 / 384M / 5G    |    7091 /  5766 / -18.7% |     476 /   422 / -11.3%
 48 / 768M / 7G    |   11139 /  5831 / -47.7% |     330 /   221 / -33.0%
 96 / 1.5G / 10G   |   21303 / 11353 / -46.7% |     283 /   180 / -36.4%
 With 64k mTHP:
 24 / 512M / 5G    |    5104 /  4641 / -18.7% |     376 /   358 / -4.8%
 48 /   1G / 7G    |    8693 /  4662 / -18.7% |     257 /   176 / -31.5%
 96 /   2G / 10G   |   17056 / 10263 / -39.8% |     234 /   169 / -27.8%
Quoted 1 line

How much is the swap in/out throughput before/after the change?

This may not be too beneficial for typical throughput measurement:
- For example doing the same test with brd will only show a ~20% performance improvement, still a big gain though. I think the si->lock spinlock wasting CPU cycles may effect CPU sensitive things like ZRAM even more.
- And simple benchmarks which just do multiple sequential swaps in/out in multiple thread hardly stress the allocator. I haven't found a good benchmark to simulate random parallel IOs on SWAP yet, I can write one later.

A more close to real word benchmark like build kernel test, or mysql/sysbench all showed great improment.

Quoted 11 lines

When I worked on swap in/out performance before, the hot spot shifts from swap related code to LRU lock and zone lock. Things may change a lot now.

If zram is used as swap device, the hot spot may become compression/decompression after solving the swap lock contention. To stress swap subsystem further, we may use a ram disk as swap. Previously, we have used a simulated pmem device (backed by DRAM). That can be setup as in,

Quoted 8 lines

https://pmem.io/blog/2016/02/how-to-emulate-persistent-memory/

After creating the raw block device: /dev/pmem0, we can do

$ mkswap /dev/pmem0
$ swapon /dev/pmem0

Can you use something similar if necessary?

I used to test with brd, as described above, I think using ZRAM with test simulating real workload is more useful.
And I did include a Sequential SWAP test, the result is looking OK (no regression, minor to none improvement).

I can have a try with the pmem setup later, I guess the result will be similar to brd test.

Quoted 99 lines
Quoted 23 lines

With more aggressive setup, it shows clearly both the performance and fragmentation are better:

tiem make -j96 / 768M memcg, 4K pages, 10G ZRAM, on Intel 8255C * 2:
(avg of 4 test run)
Before:
Sys time: 73578.30, Real time: 864.05
tiem make -j96 / 1G memcg, 4K pages, 10G ZRAM:
After: (-54.7% sys time, -49.3% real time)
Sys time: 33314.76, Real time: 437.67

time make -j96 / 1152M memcg, 64K mTHP, 10G ZRAM, on Intel 8255C * 2:
(avg of 4 test run)
Before:
Sys time: 74044.85, Real time: 846.51 hugepages-64kB/stats/swpout: 1735216 hugepages-64kB/stats/swpout_fallback: 430333
After: (-51.4% sys time, -47.7% real time, -63.2% mTHP failure) Sys time: 35958.87, Real time: 442.69 hugepages-64kB/stats/swpout: 1866267 hugepages-64kB/stats/swpout_fallback: 158330

Quoted 3 lines

There is a up to 54.7% improvement for build kernel test, and lower fragmentation rate. Performance improvement should be even larger for micro benchmarks

Very good result!

Quoted 58 lines

Build kernel with tinyconfig on tmpfs with HDD as swap:
---

This test is similar to above, but HDD test is very noisy and slow, the deviation is huge, so just use tinyconfig instead and take the median test result of 3 test run, which looks OK:

Before this series:
114.44user 29.11system 39:42.90elapsed 6%CPU 2901232inputs+0outputs (238877major+4227640minor)pagefaults

After this commit:
113.90user 23.81system 38:11.77elapsed 6%CPU 2548728inputs+0outputs (235471major+4238110minor)pagefaults

Single thread SWAP:
---

Sequential SWAP should also be slightly faster as we removed a lot of unnecessary parts. Test using micro benchmark for swapout/in 4G zero memory using ZRAM, 10 test runs:

Swapout Before (avg. 3359304):
3353796 3358551 3371305 3356043 3367524 3355303 3355924 3354513 3360776

Swapin Before (avg. 1928698):
1920283 1927183 1934105 1921373 1926562 1938261 1927726 1928636 1934155

Swapout After (avg. 3347511, -0.4%):
3337863 3347948 3355235 3339081 3333134 3353006 3354917 3346055 3360359

Swapin After (avg. 1922290, -0.3%):
1919101 1925743 1916810 1917007 1923930 1935152 1917403 1923549 1921913

Worth noticing the patch "mm, swap: use a global swap cluster for non-rotation device" introduced minor overhead for certain tests (see the test results in commit message), but the gain from later commit covered that, it can be further improved later.

Kairui Song (13):
mm, swap: minor clean up for swap entry allocation mm, swap: fold swap_info_get_cont in the only caller mm, swap: remove old allocation path for HDD
mm, swap: use cluster lock for HDD
mm, swap: clean up device availability check
mm, swap: clean up plist removal and adding
mm, swap: hold a reference of si during scan and clean up flags mm, swap: use an enum to define all cluster flags and wrap flags
changes
mm, swap: reduce contention on device lock
mm, swap: simplify percpu cluster updating
mm, swap: introduce a helper for retrieving cluster from offset mm, swap: use a global swap cluster for non-rotation device mm, swap_slots: remove slot cache for freeing path

Quoted 8 lines
 fs/btrfs/inode.c           |    1 -
 fs/iomap/swapfile.c        |    1 -
 include/linux/swap.h       |   36 +-
 include/linux/swap_slots.h |    3 -
 mm/page_io.c               |    1 -
 mm/swap_slots.c            |   78 +--
 mm/swapfile.c              | 1198 ++++++++++++++++--------------------
 7 files changed, 558 insertions(+), 760 deletions(-)

--
Best Regards,
Huang, Ying

#16 lore raw
Quoted 186 lines · Kairui Song
Quoted 2 lines · Ying

Hi, Kairui,

Hi Ying,

Quoted 24 lines
Quoted 12 lines · Kairui Song

From: Kairui Song <[email protected]>

This series improved the swap allocator performance greatly by reworking the locking design and simplify a lot of code path.

This is follow up of previous swap cluster allocator series:
https://lore.kernel.org/linux-mm/[email protected]/

And this series is based on an follow up fix of the swap cluster allocator:
https://lore.kernel.org/linux-mm/[email protected]/

Quoted 8 lines

This is part of the new swap allocator work item discussed in Chris's "Swap Abstraction" discussion at LSF/MM 2024, and "mTHP and swap allocator" discussion at LPC 2024.

Previous series introduced a fully cluster based allocation algorithm, this series completely get rid of the old allocation path and makes the allocator avoid grabbing the si->lock unless needed. This bring huge performance gain and get rid of slot cache on freeing path.

Quoted 8 lines

Great!

Quoted 3 lines

Currently, swap locking is mainly composed of two locks, cluster lock (ci->lock) and device lock (si->lock). The device lock is widely used to protect many things, causing it to be the main bottleneck for SWAP.

Device lock can be confusing with another device lock for struct device. Better to call it swap device lock?

Good idea, I'll use the term swap device lock then.

Quoted 18 lines
Quoted 18 lines

Cluster lock is much more fine-grained, so it will be best to use ci->lock instead of si->lock as much as possible.

perf lock indicates this issue clearly. Doing linux kernel build using tmpfs and ZRAM with limited memory (make -j64 with 1G memcg and 4k pages), result of "perf lock contention -ab sleep 3":

contended total wait max wait avg wait type caller

     34948     53.63 s       7.11 ms      1.53 ms     spinlock   free_swap_and_cache_nr+0x350
     16569     40.05 s       6.45 ms      2.42 ms     spinlock   get_swap_pages+0x231
     11191     28.41 s       7.03 ms      2.54 ms     spinlock   swapcache_free_entries+0x59
      4147     22.78 s     122.66 ms      5.49 ms     spinlock   page_vma_mapped_walk+0x6f3
      4595      7.17 s       6.79 ms      1.56 ms     spinlock   swapcache_free_entries+0x59
    406027      2.74 s       2.59 ms      6.74 us     spinlock   list_lru_add+0x39

...snip...

Quoted 5 lines
Quoted 2 lines

The top 5 caller are all users of si->lock, total wait time up sums to several minutes in the 3 seconds time window.

Can you show results of perf record -g, perf report -g too? I have interest to check hot spot shifting too.

Sure. I think perf lock result is already good enough and cleaner. My test environment are mostly VM based so spinlock slow path may get offloaded to host, and can't be see by perf record, I collected following data after disabled paravirt spinlock:

The time consumption and stack trace of a page fault before:
- 78.45% 0.17% cc1 [kernel.kallsyms]
[k] asm_exc_page_fault
- 78.28% asm_exc_page_fault
- 78.18% exc_page_fault
- 78.17% do_user_addr_fault
- 78.09% handle_mm_fault
- 78.06% __handle_mm_fault
- 69.69% do_swap_page
- 55.87% alloc_swap_folio
- 55.60% mem_cgroup_swapin_charge_folio
- 55.48% charge_memcg
- 55.45% try_charge_memcg
- 55.36% try_to_free_mem_cgroup_pages
- do_try_to_free_pages
- 55.35% shrink_node
- 55.27% shrink_lruvec
- 55.13% try_to_shrink_lruvec
- 54.79% evict_folios
- 54.35% shrink_folio_list
- 30.01% add_to_swap
- 29.77%
folio_alloc_swap
- 29.50%
get_swap_pages

25.03% queued_spin_lock_slowpath
- 2.71%
alloc_swap_scan_cluster

1.80% queued_spin_lock_slowpath

                                                                  +

0.89% __try_to_reclaim_swap
- 1.74%
swap_reclaim_full_clusters

1.74% queued_spin_lock_slowpath
- 10.88%
try_to_unmap_flush_dirty
- 10.87%
arch_tlbbatch_flush
- 10.85%
on_each_cpu_cond_mask

smp_call_function_many_cond
+ 7.45% pageout
+ 2.71% try_to_unmap_flush
+ 1.90% try_to_unmap
+ 0.78% folio_referenced
- 9.41% cluster_swap_free_nr
- 9.39% free_swap_slot
- 9.35% swapcache_free_entries
8.40% queued_spin_lock_slowpath 0.93% swap_entry_range_free
- 3.61% swap_read_folio_bdev_sync
- 3.55% submit_bio_wait
- 3.51% submit_bio_noacct_nocheck
+ 3.46% __submit_bio
+ 7.71% do_pte_missing
+ 0.61% wp_page_copy

The queued_spin_lock_slowpath above is the si->lock, and there are multiple users of it so the total overhead is higher than shown.

After:
- 75.05% 0.43% cc1 [kernel.kallsyms]
[k] asm_exc_page_fault
- 74.62% asm_exc_page_fault
- 74.36% exc_page_fault
- 74.34% do_user_addr_fault
- 74.10% handle_mm_fault
- 73.96% __handle_mm_fault
- 67.55% do_swap_page
- 45.92% alloc_swap_folio
- 45.03% mem_cgroup_swapin_charge_folio
- 44.58% charge_memcg
- 44.44% try_charge_memcg
- 44.12% try_to_free_mem_cgroup_pages
- do_try_to_free_pages
- 44.10% shrink_node
- 43.86% shrink_lruvec
- 41.92% try_to_shrink_lruvec
- 40.67% evict_folios
- 37.12% shrink_folio_list
- 20.88% pageout
+ 20.02% swap_writepage
+ 0.72% shmem_writepage
- 4.08% add_to_swap
- 2.48%
folio_alloc_swap
- 2.12%
__mem_cgroup_try_charge_swap
- 1.47%
swap_cgroup_record

                                                                  +

1.32% _raw_spin_lock_irqsave
- 1.56%
add_to_swap_cache
- 1.04% xas_store
+ 1.01%
workingset_update_node
+ 3.97%
try_to_unmap_flush_dirty
+ 3.51% folio_referenced
+ 2.24% __remove_mapping
+ 1.16% try_to_unmap
+ 0.52% try_to_unmap_flush
2.50%
queued_spin_lock_slowpath
0.79% scan_folios
+ 1.20% try_to_inc_max_seq
+ 1.92% lru_add_drain
+ 0.73% vma_alloc_folio_noprof
- 9.81% swap_read_folio_bdev_sync
- 9.61% submit_bio_wait
+ 9.49% submit_bio_noacct_nocheck

Quoted 8 lines

- 8.06% cluster_swap_free_nr
- 8.02% swap_entry_range_free
+ 3.92% __mem_cgroup_uncharge_swap
+ 2.90% zram_slot_free_notify 0.58% clear_shadow_from_swap_cache
- 1.32% __folio_batch_add_and_move
- 1.30% folio_batch_move_lru
+ 1.10% folio_lruvec_lock_irqsave

Thanks for data.

It seems that the cycles shifts from spinning to memory compression. That is expected.

Quoted 3 lines

spin_lock usage is much lower.

I prefer the perf lock output as it shows the exact time and user of locks.

perf cycles data is more complete. You can find which part becomes new hot spot.

Quoted 21 lines
Quoted 14 lines
Quoted 14 lines

Following the new allocator design, many operation doesn't need to touch si->lock at all. We only need to take si->lock when doing operations across multiple clusters (eg. changing the cluster list), other operations only need to take ci->lock. So ideally allocator should always take ci->lock first, then, if needed, take si->lock. But due to historical reasons, ci->lock is used inside si->lock by design, causing lock inversion if we simply try to acquire si->lock after acquiring ci->lock.

This series audited all si->lock usage, simplify legacy codes, eliminate usage of si->lock as much as possible by introducing new designs based on the new cluster allocator.

Quoted 6 lines
Quoted 2 lines

Old HDD allocation codes are removed, cluster allocator is adapted with small changes for HDD usage, test is looking OK.

I think that it's a good idea to remove HDD allocation specific code. Can you check the performance of swapping to HDD? However, I understand that many people have no HDD in hand.

Quoted 6 lines

It's not hard to make cluster allocator work well with HDD in theory, see the commit "mm, swap: use a global swap cluster for non-rotation device".
The testing is not very reliable though, I found HDD swap performance is very unstable because of the IO pattern of HDD, so it's just a best effort try.

Just to check whether code change cause something too bad for HDD. No measurable difference is a good news.

Quoted 49 lines
Quoted 49 lines
Quoted 48 lines

And this also removed slot cache for freeing path. The performance is better without it, and this enables other clean up and optimizations as discussed before:
https://lore.kernel.org/all/CAMgjq7ACohT_uerSz8E_994ZZCv709Zor+43hdmesW_59W1BWw@mail.gmail.com/

After this series, lock contention on si->lock is nearly unobservable with perf lock with the same test above :

contended total wait max wait avg wait type caller
... snip ...

         91    204.62 us      4.51 us      2.25 us     spinlock   cluster_move+0x2e
  ... snip ...
         47    125.62 us      4.47 us      2.67 us     spinlock   cluster_move+0x2e
  ... snip ...
         23     63.15 us      3.95 us      2.74 us     spinlock   cluster_move+0x2e
  ... snip ...
         17     41.26 us      4.58 us      2.43 us     spinlock   cluster_isolate_lock+0x1d

... snip ...

cluster_move and cluster_isolate_lock are basically the only users of si->lock now, performance gain is huge with reduced LOC.

Tests
===

Build kernel with defconfig on tmpfs with ZRAM as swap:
---

Running a test matrix which is scaled up progressive for a intuitive result. The test are ran on top of tmpfs, using memory cgroup for memory limitation, on a 48c96t system.

12 test run for each case, it can be seen clearly that as concurrent job number goes higher the performance gain is higher, the performance is higher even with low concurrency.

   make -j<NR>     |   System Time (seconds)  |   Total Time (seconds)
 (NR / Mem / ZRAM) | (Before / After / Delta) | (Before / After / Delta)
 With 4k pages only:
  6 / 192M / 3G    |    5258 /  5235 / -0.3%  |    1420 /  1414 / -0.3%
 12 / 256M / 4G    |    5518 /  5337 / -3.3%  |     758 /   742 / -2.1%
 24 / 384M / 5G    |    7091 /  5766 / -18.7% |     476 /   422 / -11.3%
 48 / 768M / 7G    |   11139 /  5831 / -47.7% |     330 /   221 / -33.0%
 96 / 1.5G / 10G   |   21303 / 11353 / -46.7% |     283 /   180 / -36.4%
 With 64k mTHP:
 24 / 512M / 5G    |    5104 /  4641 / -18.7% |     376 /   358 / -4.8%
 48 /   1G / 7G    |    8693 /  4662 / -18.7% |     257 /   176 / -31.5%
 96 /   2G / 10G   |   17056 / 10263 / -39.8% |     234 /   169 / -27.8%
Quoted 7 lines
Quoted 1 line

How much is the swap in/out throughput before/after the change?

This may not be too beneficial for typical throughput measurement:
- For example doing the same test with brd will only show a ~20% performance improvement, still a big gain though. I think the si->lock spinlock wasting CPU cycles may effect CPU sensitive things like ZRAM even more.

20% is a good data. You don't need to guess. perf cycles profiling can show the hot spot.

Quoted 6 lines

- And simple benchmarks which just do multiple sequential swaps in/out in multiple thread hardly stress the allocator.

I haven't found a good
benchmark to simulate random parallel IOs on SWAP yet, I can write one later.

I have used anon-w-rand test case of vm-scalability to simulate random parallel swap out.

https://git.kernel.org/pub/scm/linux/kernel/git/wfg/vm-scalability.git/tree/case-anon-w-rand

Quoted 2 lines

A more close to real word benchmark like build kernel test, or mysql/sysbench all showed great improment.

Yes. Real work load is good. We can use micro-benchmark to find out some performance limit, for example, max possible throughput.

Quoted 13 lines
Quoted 13 lines

When I worked on swap in/out performance before, the hot spot shifts from swap related code to LRU lock and zone lock. Things may change a lot now.

If zram is used as swap device, the hot spot may become compression/decompression after solving the swap lock contention. To stress swap subsystem further, we may use a ram disk as swap. Previously, we have used a simulated pmem device (backed by DRAM). That can be setup as in,

https://pmem.io/blog/2016/02/how-to-emulate-persistent-memory/

Quoted 8 lines
Quoted 6 lines

After creating the raw block device: /dev/pmem0, we can do

$ mkswap /dev/pmem0
$ swapon /dev/pmem0

Can you use something similar if necessary?

I used to test with brd, as described above,

brd will allocate memory during running, pmem can avoid that. perf profile is your friends to root cause the possible issue.

Quoted 2 lines

I think using ZRAM with
test simulating real workload is more useful.

Yes. And, as I said before. Micro-benchmark has its own value.

Quoted 2 lines

And I did include a Sequential SWAP test, the result is looking OK (no regression, minor to none improvement).

Good. At least we have no regression here.

Quoted 99 lines

I can have a try with the pmem setup later, I guess the result will be similar to brd test.

Quoted 95 lines
Quoted 23 lines

With more aggressive setup, it shows clearly both the performance and fragmentation are better:

tiem make -j96 / 768M memcg, 4K pages, 10G ZRAM, on Intel 8255C * 2:
(avg of 4 test run)
Before:
Sys time: 73578.30, Real time: 864.05
tiem make -j96 / 1G memcg, 4K pages, 10G ZRAM:
After: (-54.7% sys time, -49.3% real time)
Sys time: 33314.76, Real time: 437.67

time make -j96 / 1152M memcg, 64K mTHP, 10G ZRAM, on Intel 8255C * 2:
(avg of 4 test run)
Before:
Sys time: 74044.85, Real time: 846.51 hugepages-64kB/stats/swpout: 1735216 hugepages-64kB/stats/swpout_fallback: 430333
After: (-51.4% sys time, -47.7% real time, -63.2% mTHP failure) Sys time: 35958.87, Real time: 442.69 hugepages-64kB/stats/swpout: 1866267 hugepages-64kB/stats/swpout_fallback: 158330

Quoted 3 lines

There is a up to 54.7% improvement for build kernel test, and lower fragmentation rate. Performance improvement should be even larger for micro benchmarks

Very good result!

Quoted 66 lines

Build kernel with tinyconfig on tmpfs with HDD as swap:
---

This test is similar to above, but HDD test is very noisy and slow, the deviation is huge, so just use tinyconfig instead and take the median test result of 3 test run, which looks OK:

Before this series:
114.44user 29.11system 39:42.90elapsed 6%CPU 2901232inputs+0outputs (238877major+4227640minor)pagefaults

After this commit:
113.90user 23.81system 38:11.77elapsed 6%CPU 2548728inputs+0outputs (235471major+4238110minor)pagefaults

Single thread SWAP:
---

Sequential SWAP should also be slightly faster as we removed a lot of unnecessary parts. Test using micro benchmark for swapout/in 4G zero memory using ZRAM, 10 test runs:

Swapout Before (avg. 3359304):
3353796 3358551 3371305 3356043 3367524 3355303 3355924 3354513 3360776

Swapin Before (avg. 1928698):
1920283 1927183 1934105 1921373 1926562 1938261 1927726 1928636 1934155

Swapout After (avg. 3347511, -0.4%):
3337863 3347948 3355235 3339081 3333134 3353006 3354917 3346055 3360359

Swapin After (avg. 1922290, -0.3%):
1919101 1925743 1916810 1917007 1923930 1935152 1917403 1923549 1921913

Worth noticing the patch "mm, swap: use a global swap cluster for non-rotation device" introduced minor overhead for certain tests (see the test results in commit message), but the gain from later commit covered that, it can be further improved later.

Kairui Song (13):
mm, swap: minor clean up for swap entry allocation mm, swap: fold swap_info_get_cont in the only caller mm, swap: remove old allocation path for HDD
mm, swap: use cluster lock for HDD
mm, swap: clean up device availability check
mm, swap: clean up plist removal and adding
mm, swap: hold a reference of si during scan and clean up flags mm, swap: use an enum to define all cluster flags and wrap flags
changes
mm, swap: reduce contention on device lock
mm, swap: simplify percpu cluster updating
mm, swap: introduce a helper for retrieving cluster from offset mm, swap: use a global swap cluster for non-rotation device mm, swap_slots: remove slot cache for freeing path

 fs/btrfs/inode.c           |    1 -
 fs/iomap/swapfile.c        |    1 -
 include/linux/swap.h       |   36 +-
 include/linux/swap_slots.h |    3 -
 mm/page_io.c               |    1 -
 mm/swap_slots.c            |   78 +--
 mm/swapfile.c              | 1198 ++++++++++++++++--------------------
 7 files changed, 558 insertions(+), 760 deletions(-)

--
Best Regards,
Huang, Ying

#17 lore raw
Quoted 341 lines · Ying
Quoted 186 lines · Kairui Song
Quoted 2 lines · Ying

Hi, Kairui,

Hi Ying,

Quoted 24 lines
Quoted 12 lines · Kairui Song

From: Kairui Song <[email protected]>

This series improved the swap allocator performance greatly by reworking the locking design and simplify a lot of code path.

This is follow up of previous swap cluster allocator series:
https://lore.kernel.org/linux-mm/[email protected]/

And this series is based on an follow up fix of the swap cluster allocator:
https://lore.kernel.org/linux-mm/[email protected]/

Quoted 8 lines

This is part of the new swap allocator work item discussed in Chris's "Swap Abstraction" discussion at LSF/MM 2024, and "mTHP and swap allocator" discussion at LPC 2024.

Previous series introduced a fully cluster based allocation algorithm, this series completely get rid of the old allocation path and makes the allocator avoid grabbing the si->lock unless needed. This bring huge performance gain and get rid of slot cache on freeing path.

Quoted 8 lines

Great!

Quoted 3 lines

Currently, swap locking is mainly composed of two locks, cluster lock (ci->lock) and device lock (si->lock). The device lock is widely used to protect many things, causing it to be the main bottleneck for SWAP.

Device lock can be confusing with another device lock for struct device. Better to call it swap device lock?

Good idea, I'll use the term swap device lock then.

Quoted 18 lines
Quoted 18 lines

Cluster lock is much more fine-grained, so it will be best to use ci->lock instead of si->lock as much as possible.

perf lock indicates this issue clearly. Doing linux kernel build using tmpfs and ZRAM with limited memory (make -j64 with 1G memcg and 4k pages), result of "perf lock contention -ab sleep 3":

contended total wait max wait avg wait type caller

     34948     53.63 s       7.11 ms      1.53 ms     spinlock   free_swap_and_cache_nr+0x350
     16569     40.05 s       6.45 ms      2.42 ms     spinlock   get_swap_pages+0x231
     11191     28.41 s       7.03 ms      2.54 ms     spinlock   swapcache_free_entries+0x59
      4147     22.78 s     122.66 ms      5.49 ms     spinlock   page_vma_mapped_walk+0x6f3
      4595      7.17 s       6.79 ms      1.56 ms     spinlock   swapcache_free_entries+0x59
    406027      2.74 s       2.59 ms      6.74 us     spinlock   list_lru_add+0x39

...snip...

Quoted 5 lines
Quoted 2 lines

The top 5 caller are all users of si->lock, total wait time up sums to several minutes in the 3 seconds time window.

Can you show results of perf record -g, perf report -g too? I have interest to check hot spot shifting too.

Sure. I think perf lock result is already good enough and cleaner. My test environment are mostly VM based so spinlock slow path may get offloaded to host, and can't be see by perf record, I collected following data after disabled paravirt spinlock:

The time consumption and stack trace of a page fault before:
- 78.45% 0.17% cc1 [kernel.kallsyms]
[k] asm_exc_page_fault
- 78.28% asm_exc_page_fault
- 78.18% exc_page_fault
- 78.17% do_user_addr_fault
- 78.09% handle_mm_fault
- 78.06% __handle_mm_fault
- 69.69% do_swap_page
- 55.87% alloc_swap_folio
- 55.60% mem_cgroup_swapin_charge_folio
- 55.48% charge_memcg
- 55.45% try_charge_memcg
- 55.36% try_to_free_mem_cgroup_pages
- do_try_to_free_pages
- 55.35% shrink_node
- 55.27% shrink_lruvec
- 55.13% try_to_shrink_lruvec
- 54.79% evict_folios
- 54.35% shrink_folio_list
- 30.01% add_to_swap
- 29.77%
folio_alloc_swap
- 29.50%
get_swap_pages

25.03% queued_spin_lock_slowpath
- 2.71%
alloc_swap_scan_cluster

1.80% queued_spin_lock_slowpath

                                                                  +

0.89% __try_to_reclaim_swap
- 1.74%
swap_reclaim_full_clusters

1.74% queued_spin_lock_slowpath
- 10.88%
try_to_unmap_flush_dirty
- 10.87%
arch_tlbbatch_flush
- 10.85%
on_each_cpu_cond_mask

smp_call_function_many_cond
+ 7.45% pageout
+ 2.71% try_to_unmap_flush
+ 1.90% try_to_unmap
+ 0.78% folio_referenced
- 9.41% cluster_swap_free_nr
- 9.39% free_swap_slot
- 9.35% swapcache_free_entries
8.40% queued_spin_lock_slowpath 0.93% swap_entry_range_free
- 3.61% swap_read_folio_bdev_sync
- 3.55% submit_bio_wait
- 3.51% submit_bio_noacct_nocheck
+ 3.46% __submit_bio
+ 7.71% do_pte_missing
+ 0.61% wp_page_copy

The queued_spin_lock_slowpath above is the si->lock, and there are multiple users of it so the total overhead is higher than shown.

After:
- 75.05% 0.43% cc1 [kernel.kallsyms]
[k] asm_exc_page_fault
- 74.62% asm_exc_page_fault
- 74.36% exc_page_fault
- 74.34% do_user_addr_fault
- 74.10% handle_mm_fault
- 73.96% __handle_mm_fault
- 67.55% do_swap_page
- 45.92% alloc_swap_folio
- 45.03% mem_cgroup_swapin_charge_folio
- 44.58% charge_memcg
- 44.44% try_charge_memcg
- 44.12% try_to_free_mem_cgroup_pages
- do_try_to_free_pages
- 44.10% shrink_node
- 43.86% shrink_lruvec
- 41.92% try_to_shrink_lruvec
- 40.67% evict_folios
- 37.12% shrink_folio_list
- 20.88% pageout
+ 20.02% swap_writepage
+ 0.72% shmem_writepage
- 4.08% add_to_swap
- 2.48%
folio_alloc_swap
- 2.12%
__mem_cgroup_try_charge_swap
- 1.47%
swap_cgroup_record

                                                                  +

1.32% _raw_spin_lock_irqsave
- 1.56%
add_to_swap_cache
- 1.04% xas_store
+ 1.01%
workingset_update_node
+ 3.97%
try_to_unmap_flush_dirty
+ 3.51% folio_referenced
+ 2.24% __remove_mapping
+ 1.16% try_to_unmap
+ 0.52% try_to_unmap_flush
2.50%
queued_spin_lock_slowpath
0.79% scan_folios
+ 1.20% try_to_inc_max_seq
+ 1.92% lru_add_drain
+ 0.73% vma_alloc_folio_noprof
- 9.81% swap_read_folio_bdev_sync
- 9.61% submit_bio_wait
+ 9.49% submit_bio_noacct_nocheck

Quoted 8 lines

- 8.06% cluster_swap_free_nr
- 8.02% swap_entry_range_free
+ 3.92% __mem_cgroup_uncharge_swap
+ 2.90% zram_slot_free_notify 0.58% clear_shadow_from_swap_cache
- 1.32% __folio_batch_add_and_move
- 1.30% folio_batch_move_lru
+ 1.10% folio_lruvec_lock_irqsave

Thanks for data.

It seems that the cycles shifts from spinning to memory compression. That is expected.

Quoted 3 lines

spin_lock usage is much lower.

I prefer the perf lock output as it shows the exact time and user of locks.

perf cycles data is more complete. You can find which part becomes new hot spot.

Quoted 21 lines
Quoted 14 lines
Quoted 14 lines

Following the new allocator design, many operation doesn't need to touch si->lock at all. We only need to take si->lock when doing operations across multiple clusters (eg. changing the cluster list), other operations only need to take ci->lock. So ideally allocator should always take ci->lock first, then, if needed, take si->lock. But due to historical reasons, ci->lock is used inside si->lock by design, causing lock inversion if we simply try to acquire si->lock after acquiring ci->lock.

This series audited all si->lock usage, simplify legacy codes, eliminate usage of si->lock as much as possible by introducing new designs based on the new cluster allocator.

Quoted 6 lines
Quoted 2 lines

Old HDD allocation codes are removed, cluster allocator is adapted with small changes for HDD usage, test is looking OK.

I think that it's a good idea to remove HDD allocation specific code. Can you check the performance of swapping to HDD? However, I understand that many people have no HDD in hand.

Quoted 6 lines

It's not hard to make cluster allocator work well with HDD in theory, see the commit "mm, swap: use a global swap cluster for non-rotation device".
The testing is not very reliable though, I found HDD swap performance is very unstable because of the IO pattern of HDD, so it's just a best effort try.

Just to check whether code change cause something too bad for HDD. No measurable difference is a good news.

Quoted 49 lines
Quoted 49 lines
Quoted 48 lines

And this also removed slot cache for freeing path. The performance is better without it, and this enables other clean up and optimizations as discussed before:
https://lore.kernel.org/all/CAMgjq7ACohT_uerSz8E_994ZZCv709Zor+43hdmesW_59W1BWw@mail.gmail.com/

After this series, lock contention on si->lock is nearly unobservable with perf lock with the same test above :

contended total wait max wait avg wait type caller
... snip ...

         91    204.62 us      4.51 us      2.25 us     spinlock   cluster_move+0x2e
  ... snip ...
         47    125.62 us      4.47 us      2.67 us     spinlock   cluster_move+0x2e
  ... snip ...
         23     63.15 us      3.95 us      2.74 us     spinlock   cluster_move+0x2e
  ... snip ...
         17     41.26 us      4.58 us      2.43 us     spinlock   cluster_isolate_lock+0x1d

... snip ...

cluster_move and cluster_isolate_lock are basically the only users of si->lock now, performance gain is huge with reduced LOC.

Tests
===

Build kernel with defconfig on tmpfs with ZRAM as swap:
---

Running a test matrix which is scaled up progressive for a intuitive result. The test are ran on top of tmpfs, using memory cgroup for memory limitation, on a 48c96t system.

12 test run for each case, it can be seen clearly that as concurrent job number goes higher the performance gain is higher, the performance is higher even with low concurrency.

   make -j<NR>     |   System Time (seconds)  |   Total Time (seconds)
 (NR / Mem / ZRAM) | (Before / After / Delta) | (Before / After / Delta)
 With 4k pages only:
  6 / 192M / 3G    |    5258 /  5235 / -0.3%  |    1420 /  1414 / -0.3%
 12 / 256M / 4G    |    5518 /  5337 / -3.3%  |     758 /   742 / -2.1%
 24 / 384M / 5G    |    7091 /  5766 / -18.7% |     476 /   422 / -11.3%
 48 / 768M / 7G    |   11139 /  5831 / -47.7% |     330 /   221 / -33.0%
 96 / 1.5G / 10G   |   21303 / 11353 / -46.7% |     283 /   180 / -36.4%
 With 64k mTHP:
 24 / 512M / 5G    |    5104 /  4641 / -18.7% |     376 /   358 / -4.8%
 48 /   1G / 7G    |    8693 /  4662 / -18.7% |     257 /   176 / -31.5%
 96 /   2G / 10G   |   17056 / 10263 / -39.8% |     234 /   169 / -27.8%
Quoted 7 lines
Quoted 1 line

How much is the swap in/out throughput before/after the change?

This may not be too beneficial for typical throughput measurement:
- For example doing the same test with brd will only show a ~20% performance improvement, still a big gain though. I think the si->lock spinlock wasting CPU cycles may effect CPU sensitive things like ZRAM even more.

20% is a good data. You don't need to guess. perf cycles profiling can show the hot spot.

Quoted 6 lines

- And simple benchmarks which just do multiple sequential swaps in/out in multiple thread hardly stress the allocator.

I haven't found a good
benchmark to simulate random parallel IOs on SWAP yet, I can write one later.

I have used anon-w-rand test case of vm-scalability to simulate random parallel swap out.

https://git.kernel.org/pub/scm/linux/kernel/git/wfg/vm-scalability.git/tree/case-anon-w-rand

Quoted 2 lines

A more close to real word benchmark like build kernel test, or mysql/sysbench all showed great improment.

Yes. Real work load is good. We can use micro-benchmark to find out some performance limit, for example, max possible throughput.

Quoted 13 lines
Quoted 13 lines

When I worked on swap in/out performance before, the hot spot shifts from swap related code to LRU lock and zone lock. Things may change a lot now.

If zram is used as swap device, the hot spot may become compression/decompression after solving the swap lock contention. To stress swap subsystem further, we may use a ram disk as swap. Previously, we have used a simulated pmem device (backed by DRAM). That can be setup as in,

https://pmem.io/blog/2016/02/how-to-emulate-persistent-memory/

Quoted 8 lines
Quoted 6 lines

After creating the raw block device: /dev/pmem0, we can do

$ mkswap /dev/pmem0
$ swapon /dev/pmem0

Can you use something similar if necessary?

I used to test with brd, as described above,

Quoted 7 lines

brd will allocate memory during running, pmem can avoid that. perf profile is your friends to root cause the possible issue.

Quoted 2 lines

I think using ZRAM with
test simulating real workload is more useful.

Yes. And, as I said before. Micro-benchmark has its own value.

Hi Ying,

Thank you very much for the suggestion, I didn't mean I'm against micro benchmarks in any way, just a lot of effort was spent on other tests so I skipped that part for V1.

As you mentioned vm-scalability, I think this is definitely a good idea to include that test when pmem simulation.

There are still some bottlenecks of SWAP, beside compression and page fault / tlb, mostly cgroup lock and list lru locks. I have some ideas to optimize these too, could be next steps.

Quoted 8 lines
Quoted 2 lines

And I did include a Sequential SWAP test, the result is looking OK (no regression, minor to none improvement).

Good. At least we have no regression here.

--
Best Regards,
Huang, Ying

maintainer
#18 lore raw
In reply to Kairui Song
Quoted 5 lines · On Wed, 23 Oct 2024 03:24:38 +0800 Kairui Song <[email protected]> wrote:

After this series, lock contention on si->lock is nearly unobservable with perf lock with the same test above :

contended total wait max wait avg wait type caller
... snip ...

Quoted 8 lines
         91    204.62 us      4.51 us      2.25 us     spinlock   cluster_move+0x2e
  ... snip ...
         47    125.62 us      4.47 us      2.67 us     spinlock   cluster_move+0x2e
  ... snip ...
         23     63.15 us      3.95 us      2.74 us     spinlock   cluster_move+0x2e
  ... snip ...
         17     41.26 us      4.58 us      2.43 us     spinlock   cluster_isolate_lock+0x1d

... snip ...

Were any overall runtime benefits observed?

#19 lore raw
Quoted 17 lines · Andrew Morton
Quoted 5 lines · On Wed, 23 Oct 2024 03:24:38 +0800 Kairui Song <[email protected]> wrote:

After this series, lock contention on si->lock is nearly unobservable with perf lock with the same test above :

contended total wait max wait avg wait type caller
... snip ...

Quoted 8 lines
         91    204.62 us      4.51 us      2.25 us     spinlock   cluster_move+0x2e
  ... snip ...
         47    125.62 us      4.47 us      2.67 us     spinlock   cluster_move+0x2e
  ... snip ...
         23     63.15 us      3.95 us      2.74 us     spinlock   cluster_move+0x2e
  ... snip ...
         17     41.26 us      4.58 us      2.43 us     spinlock   cluster_isolate_lock+0x1d

... snip ...

Quoted 1 line

Were any overall runtime benefits observed?

Yes, see the "Tests" results in the cover letter (summary: up to 50% build time saved for build linux kernel test when under pressure, with either mTHP or 4K pages):

time make -j96 / 768M memcg, 4K pages, 10G ZRAM, on Intel 8255C * 2 in VM:
(avg of 4 test run)
Before:
Sys time: 73578.30, Real time: 864.05
After: (-54.7% sys time, -49.3% real time)
Sys time: 33314.76, Real time: 437.67

time make -j96 / 1152M memcg, 64K mTHP, 10G ZRAM, on Intel 8255C * 2 in VM:
(avg of 4 test run)
Before:
Sys time: 74044.85, Real time: 846.51
After: (-51.4% sys time, -47.7% real time, -63.2% mTHP failure) Sys time: 35958.87, Real time: 442.69

Tests on the host bare metal showed similar results.

There are some other test results I didn't include in the cover letter for V1 yet and I'm still testing more scenarios, eg. mysql test in 1G memcg and with 96 workers and ZRAM swap:
before:

    transactions:                        755630 (6292.11 per sec.)
    queries:                             12090080 (100673.69 per sec.)
after:
    transactions:                        1077156 (8972.73 per sec.)
    queries:                             17234496 (143563.65 per sec.)

~30% faster.

Also the mTHP swap allocation success rate is higher, I can highlight these changes in V2.

#20 lore raw
In reply to Kairui Song
Quoted 162 lines · Kairui Song

From: Kairui Song <[email protected]>

This series improved the swap allocator performance greatly by reworking the locking design and simplify a lot of code path.

This is follow up of previous swap cluster allocator series:
https://lore.kernel.org/linux-mm/[email protected]/

And this series is based on an follow up fix of the swap cluster allocator:
https://lore.kernel.org/linux-mm/[email protected]/

This is part of the new swap allocator work item discussed in Chris's "Swap Abstraction" discussion at LSF/MM 2024, and "mTHP and swap allocator" discussion at LPC 2024.

Previous series introduced a fully cluster based allocation algorithm, this series completely get rid of the old allocation path and makes the allocator avoid grabbing the si->lock unless needed. This bring huge performance gain and get rid of slot cache on freeing path.

Currently, swap locking is mainly composed of two locks, cluster lock (ci->lock) and device lock (si->lock). The device lock is widely used to protect many things, causing it to be the main bottleneck for SWAP.

Cluster lock is much more fine-grained, so it will be best to use ci->lock instead of si->lock as much as possible.

perf lock indicates this issue clearly. Doing linux kernel build using tmpfs and ZRAM with limited memory (make -j64 with 1G memcg and 4k pages), result of "perf lock contention -ab sleep 3":

contended total wait max wait avg wait type caller

     34948     53.63 s       7.11 ms      1.53 ms     spinlock   free_swap_and_cache_nr+0x350
     16569     40.05 s       6.45 ms      2.42 ms     spinlock   get_swap_pages+0x231
     11191     28.41 s       7.03 ms      2.54 ms     spinlock   swapcache_free_entries+0x59
      4147     22.78 s     122.66 ms      5.49 ms     spinlock   page_vma_mapped_walk+0x6f3
      4595      7.17 s       6.79 ms      1.56 ms     spinlock   swapcache_free_entries+0x59
    406027      2.74 s       2.59 ms      6.74 us     spinlock   list_lru_add+0x39

...snip...

The top 5 caller are all users of si->lock, total wait time up sums to several minutes in the 3 seconds time window.

Following the new allocator design, many operation doesn't need to touch si->lock at all. We only need to take si->lock when doing operations across multiple clusters (eg. changing the cluster list), other operations only need to take ci->lock. So ideally allocator should always take ci->lock first, then, if needed, take si->lock. But due to historical reasons, ci->lock is used inside si->lock by design, causing lock inversion if we simply try to acquire si->lock after acquiring ci->lock.

This series audited all si->lock usage, simplify legacy codes, eliminate usage of si->lock as much as possible by introducing new designs based on the new cluster allocator.

Old HDD allocation codes are removed, cluster allocator is adapted with small changes for HDD usage, test is looking OK.

And this also removed slot cache for freeing path. The performance is better without it, and this enables other clean up and optimizations as discussed before:
https://lore.kernel.org/all/CAMgjq7ACohT_uerSz8E_994ZZCv709Zor+43hdmesW_59W1BWw@mail.gmail.com/

After this series, lock contention on si->lock is nearly unobservable with perf lock with the same test above :

contended total wait max wait avg wait type caller
... snip ...

         91    204.62 us      4.51 us      2.25 us     spinlock   cluster_move+0x2e
  ... snip ...
         47    125.62 us      4.47 us      2.67 us     spinlock   cluster_move+0x2e
  ... snip ...
         23     63.15 us      3.95 us      2.74 us     spinlock   cluster_move+0x2e
  ... snip ...
         17     41.26 us      4.58 us      2.43 us     spinlock   cluster_isolate_lock+0x1d

... snip ...

cluster_move and cluster_isolate_lock are basically the only users of si->lock now, performance gain is huge with reduced LOC.

Tests
===

Build kernel with defconfig on tmpfs with ZRAM as swap:
---

Running a test matrix which is scaled up progressive for a intuitive result. The test are ran on top of tmpfs, using memory cgroup for memory limitation, on a 48c96t system.

12 test run for each case, it can be seen clearly that as concurrent job number goes higher the performance gain is higher, the performance is higher even with low concurrency.

   make -j<NR>     |   System Time (seconds)  |   Total Time (seconds)
 (NR / Mem / ZRAM) | (Before / After / Delta) | (Before / After / Delta)
 With 4k pages only:
  6 / 192M / 3G    |    5258 /  5235 / -0.3%  |    1420 /  1414 / -0.3%
 12 / 256M / 4G    |    5518 /  5337 / -3.3%  |     758 /   742 / -2.1%
 24 / 384M / 5G    |    7091 /  5766 / -18.7% |     476 /   422 / -11.3%
 48 / 768M / 7G    |   11139 /  5831 / -47.7% |     330 /   221 / -33.0%
 96 / 1.5G / 10G   |   21303 / 11353 / -46.7% |     283 /   180 / -36.4%
 With 64k mTHP:
 24 / 512M / 5G    |    5104 /  4641 / -18.7% |     376 /   358 / -4.8%
 48 /   1G / 7G    |    8693 /  4662 / -18.7% |     257 /   176 / -31.5%
 96 /   2G / 10G   |   17056 / 10263 / -39.8% |     234 /   169 / -27.8%

With more aggressive setup, it shows clearly both the performance and fragmentation are better:

tiem make -j96 / 768M memcg, 4K pages, 10G ZRAM, on Intel 8255C * 2:
(avg of 4 test run)
Before:
Sys time: 73578.30, Real time: 864.05
tiem make -j96 / 1G memcg, 4K pages, 10G ZRAM:
After: (-54.7% sys time, -49.3% real time)
Sys time: 33314.76, Real time: 437.67

time make -j96 / 1152M memcg, 64K mTHP, 10G ZRAM, on Intel 8255C * 2:
(avg of 4 test run)
Before:
Sys time: 74044.85, Real time: 846.51 hugepages-64kB/stats/swpout: 1735216 hugepages-64kB/stats/swpout_fallback: 430333
After: (-51.4% sys time, -47.7% real time, -63.2% mTHP failure) Sys time: 35958.87, Real time: 442.69 hugepages-64kB/stats/swpout: 1866267 hugepages-64kB/stats/swpout_fallback: 158330

There is a up to 54.7% improvement for build kernel test, and lower fragmentation rate. Performance improvement should be even larger for micro benchmarks

Build kernel with tinyconfig on tmpfs with HDD as swap:
---

This test is similar to above, but HDD test is very noisy and slow, the deviation is huge, so just use tinyconfig instead and take the median test result of 3 test run, which looks OK:

Before this series:
114.44user 29.11system 39:42.90elapsed 6%CPU 2901232inputs+0outputs (238877major+4227640minor)pagefaults

After this commit:
113.90user 23.81system 38:11.77elapsed 6%CPU 2548728inputs+0outputs (235471major+4238110minor)pagefaults

Single thread SWAP:
---

Sequential SWAP should also be slightly faster as we removed a lot of unnecessary parts. Test using micro benchmark for swapout/in 4G zero memory using ZRAM, 10 test runs:

Swapout Before (avg. 3359304):
3353796 3358551 3371305 3356043 3367524 3355303 3355924 3354513 3360776

Quoted 8 lines

Swapin Before (avg. 1928698):
1920283 1927183 1934105 1921373 1926562 1938261 1927726 1928636 1934155

Swapout After (avg. 3347511, -0.4%):
3337863 3347948 3355235 3339081 3333134 3353006 3354917 3346055 3360359

Swapin After (avg. 1922290, -0.3%):
1919101 1925743 1916810 1917007 1923930 1935152 1917403 1923549 1921913

Unfortunately I don't have the time to go through this series, but I just wanted to say that this awesome work, Kairui.

Selfishly, I especially like cleaning up the swap slot freeing path, and having a centralized freeing path with a single call to zswap_invalidate().

Thanks for doing this :)