mm, swap: rework of swap allocator locks
Subsystems (4)
cover mm, swap: rework of swap allocator locks
From: Kairui Song <[email protected]>
This series improved the swap allocator performance greatly by reworking the locking design and simplify a lot of code path.
This is follow up of previous swap cluster allocator series:
https://lore.kernel.org/linux-mm/[email protected]/
And this series is based on an follow up fix of the swap cluster allocator:
https://lore.kernel.org/linux-mm/[email protected]/
This is part of the new swap allocator work item discussed in Chris's "Swap Abstraction" discussion at LSF/MM 2024, and "mTHP and swap allocator" discussion at LPC 2024.
Previous series introduced a fully cluster based allocation algorithm, this series completely get rid of the old allocation path and makes the allocator avoid grabbing the si->lock unless needed. This bring huge performance gain and get rid of slot cache on freeing path.
Currently, swap locking is mainly composed of two locks, cluster lock (ci->lock) and device lock (si->lock). The device lock is widely used to protect many things, causing it to be the main bottleneck for SWAP.
Cluster lock is much more fine-grained, so it will be best to use ci->lock instead of si->lock as much as possible.
perf lock indicates this issue clearly. Doing linux kernel build using tmpfs and ZRAM with limited memory (make -j64 with 1G memcg and 4k pages), result of "perf lock contention -ab sleep 3":
contended total wait max wait avg wait type caller
34948 53.63 s 7.11 ms 1.53 ms spinlock free_swap_and_cache_nr+0x350
16569 40.05 s 6.45 ms 2.42 ms spinlock get_swap_pages+0x231
11191 28.41 s 7.03 ms 2.54 ms spinlock swapcache_free_entries+0x59
4147 22.78 s 122.66 ms 5.49 ms spinlock page_vma_mapped_walk+0x6f3
4595 7.17 s 6.79 ms 1.56 ms spinlock swapcache_free_entries+0x59
406027 2.74 s 2.59 ms 6.74 us spinlock list_lru_add+0x39...snip...
The top 5 caller are all users of si->lock, total wait time up sums to several minutes in the 3 seconds time window.
Following the new allocator design, many operation doesn't need to touch si->lock at all. We only need to take si->lock when doing operations across multiple clusters (eg. changing the cluster list), other operations only need to take ci->lock. So ideally allocator should always take ci->lock first, then, if needed, take si->lock. But due to historical reasons, ci->lock is used inside si->lock by design, causing lock inversion if we simply try to acquire si->lock after acquiring ci->lock.
This series audited all si->lock usage, simplify legacy codes, eliminate usage of si->lock as much as possible by introducing new designs based on the new cluster allocator.
Old HDD allocation codes are removed, cluster allocator is adapted with small changes for HDD usage, test is looking OK.
And this also removed slot cache for freeing path. The performance is better without it, and this enables other clean up and optimizations as discussed before:
https://lore.kernel.org/all/CAMgjq7ACohT_uerSz8E_994ZZCv709Zor+43hdmesW_59W1BWw@mail.gmail.com/
After this series, lock contention on si->lock is nearly unobservable with perf lock with the same test above :
contended total wait max wait avg wait type caller
... snip ...
91 204.62 us 4.51 us 2.25 us spinlock cluster_move+0x2e
... snip ...
47 125.62 us 4.47 us 2.67 us spinlock cluster_move+0x2e
... snip ...
23 63.15 us 3.95 us 2.74 us spinlock cluster_move+0x2e
... snip ...
17 41.26 us 4.58 us 2.43 us spinlock cluster_isolate_lock+0x1d... snip ...
cluster_move and cluster_isolate_lock are basically the only users of si->lock now, performance gain is huge with reduced LOC.
Tests
===
Build kernel with defconfig on tmpfs with ZRAM as swap:
---
Running a test matrix which is scaled up progressive for a intuitive result. The test are ran on top of tmpfs, using memory cgroup for memory limitation, on a 48c96t system.
12 test run for each case, it can be seen clearly that as concurrent job number goes higher the performance gain is higher, the performance is higher even with low concurrency.
make -j<NR> | System Time (seconds) | Total Time (seconds) (NR / Mem / ZRAM) | (Before / After / Delta) | (Before / After / Delta) With 4k pages only: 6 / 192M / 3G | 5258 / 5235 / -0.3% | 1420 / 1414 / -0.3% 12 / 256M / 4G | 5518 / 5337 / -3.3% | 758 / 742 / -2.1% 24 / 384M / 5G | 7091 / 5766 / -18.7% | 476 / 422 / -11.3% 48 / 768M / 7G | 11139 / 5831 / -47.7% | 330 / 221 / -33.0% 96 / 1.5G / 10G | 21303 / 11353 / -46.7% | 283 / 180 / -36.4% With 64k mTHP: 24 / 512M / 5G | 5104 / 4641 / -18.7% | 376 / 358 / -4.8% 48 / 1G / 7G | 8693 / 4662 / -18.7% | 257 / 176 / -31.5% 96 / 2G / 10G | 17056 / 10263 / -39.8% | 234 / 169 / -27.8%
With more aggressive setup, it shows clearly both the performance and fragmentation are better:
tiem make -j96 / 768M memcg, 4K pages, 10G ZRAM, on Intel 8255C * 2:
(avg of 4 test run)
Before:
Sys time: 73578.30, Real time: 864.05
tiem make -j96 / 1G memcg, 4K pages, 10G ZRAM:
After: (-54.7% sys time, -49.3% real time)
Sys time: 33314.76, Real time: 437.67
time make -j96 / 1152M memcg, 64K mTHP, 10G ZRAM, on Intel 8255C * 2:
(avg of 4 test run)
Before:
Sys time: 74044.85, Real time: 846.51 hugepages-64kB/stats/swpout: 1735216 hugepages-64kB/stats/swpout_fallback: 430333
After: (-51.4% sys time, -47.7% real time, -63.2% mTHP failure) Sys time: 35958.87, Real time: 442.69 hugepages-64kB/stats/swpout: 1866267 hugepages-64kB/stats/swpout_fallback: 158330
There is a up to 54.7% improvement for build kernel test, and lower fragmentation rate. Performance improvement should be even larger for micro benchmarks
Build kernel with tinyconfig on tmpfs with HDD as swap:
---
This test is similar to above, but HDD test is very noisy and slow, the deviation is huge, so just use tinyconfig instead and take the median test result of 3 test run, which looks OK:
Before this series:
114.44user 29.11system 39:42.90elapsed 6%CPU 2901232inputs+0outputs (238877major+4227640minor)pagefaults
After this commit:
113.90user 23.81system 38:11.77elapsed 6%CPU 2548728inputs+0outputs (235471major+4238110minor)pagefaults
Single thread SWAP:
---
Sequential SWAP should also be slightly faster as we removed a lot of unnecessary parts. Test using micro benchmark for swapout/in 4G zero memory using ZRAM, 10 test runs:
Swapout Before (avg. 3359304):
3353796 3358551 3371305 3356043 3367524 3355303 3355924 3354513 3360776
Swapin Before (avg. 1928698):
1920283 1927183 1934105 1921373 1926562 1938261 1927726 1928636 1934155
Swapout After (avg. 3347511, -0.4%):
3337863 3347948 3355235 3339081 3333134 3353006 3354917 3346055 3360359
Swapin After (avg. 1922290, -0.3%):
1919101 1925743 1916810 1917007 1923930 1935152 1917403 1923549 1921913
Worth noticing the patch "mm, swap: use a global swap cluster for non-rotation device" introduced minor overhead for certain tests (see the test results in commit message), but the gain from later commit covered that, it can be further improved later.
Kairui Song (13):
mm, swap: minor clean up for swap entry allocation mm, swap: fold swap_info_get_cont in the only caller mm, swap: remove old allocation path for HDD
mm, swap: use cluster lock for HDD
mm, swap: clean up device availability check
mm, swap: clean up plist removal and adding
mm, swap: hold a reference of si during scan and clean up flags mm, swap: use an enum to define all cluster flags and wrap flags
changes
mm, swap: reduce contention on device lock
mm, swap: simplify percpu cluster updating
mm, swap: introduce a helper for retrieving cluster from offset mm, swap: use a global swap cluster for non-rotation device mm, swap_slots: remove slot cache for freeing path
fs/btrfs/inode.c | 1 - fs/iomap/swapfile.c | 1 - include/linux/swap.h | 36 +- include/linux/swap_slots.h | 3 - mm/page_io.c | 1 - mm/swap_slots.c | 78 +-- mm/swapfile.c | 1198 ++++++++++++++++-------------------- 7 files changed, 558 insertions(+), 760 deletions(-)
Signature · 1 line
2.47.0
PATCH 1/13 mm, swap: minor clean up for swap entry allocation
From: Kairui Song <[email protected]>
Direct reclaim can skip the whole folio after reclaimed a set of folio based slots. Also simplify the code for allocation, reduce indention.
--- mm/swapfile.c | 59 +++++++++++++++++++++++++-------------------------- 1 file changed, 29 insertions(+), 30 deletions(-)
mm/swapfile.c +29 -30
@@ -604,23 +604,28 @@ static bool cluster_reclaim_range(struct swap_info_struct *si, unsigned long start, unsigned long end) {unsigned char *map = si->swap_map; -unsigned long offset; +unsigned long offset = start; +int nr_reclaim;spin_unlock (&ci->lock);spin_unlock (&si->lock); -for (offset = start; offset < end; offset++) { +do {switch (READ_ONCE (map[offset])) {case 0 : -continue ; + offset++; +break ;case SWAP_HAS_CACHE : -if (__try_to_reclaim_swap (si, offset,TTRS_ANYWAY |TTRS_DIRECT ) >0 ) -continue ; - goto out; + nr_reclaim =__try_to_reclaim_swap (si, offset,TTRS_ANYWAY |TTRS_DIRECT ); +if (nr_reclaim >0 ) + offset += nr_reclaim; +else + goto out; +break ;default : goto out; } - } + }while (offset < end); out:spin_lock (&si->lock);spin_lock (&ci->lock);
@@ -826,35 +831,30 @@ static unsigned long cluster_alloc_swap_entry(struct swap_info_struct *si, int o &found, order, usage); frags++;if (found) -break ; + goto done; } -if (!found) { +/* +* Nonfull clusters are moved to frag tail if we reached +* here, count them too, don't over scan the frag list. +*/ +while (frags < si->frag_cluster_nr[order]) { + ci =list_first_entry (&si->frag_clusters[order], + struct swap_cluster_info, list);/* -* Nonfull clusters are moved to frag tail if we reached -* here, count them too, don't over scan the frag list. +* Rotate the frag list to iterate, they were all failing +* high order allocation or moved here due to per-CPU usage, +* this help keeping usable cluster ahead. */ -while (frags < si->frag_cluster_nr[order]) { - ci =list_first_entry (&si->frag_clusters[order], - struct swap_cluster_info, list); -/* -* Rotate the frag list to iterate, they were all failing -* high order allocation or moved here due to per-CPU usage, -* this help keeping usable cluster ahead. -*/ -list_move_tail (&ci->list, &si->frag_clusters[order]); - offset =alloc_swap_scan_cluster (si,cluster_offset (si, ci), - &found, order, usage); - frags++; -if (found) -break ; - } +list_move_tail (&ci->list, &si->frag_clusters[order]); + offset =alloc_swap_scan_cluster (si,cluster_offset (si, ci), + &found, order, usage); + frags++; +if (found) + goto done; } } -if (found) - goto done; -if (!list_empty (&si->discard_clusters)) {/* * we don't have free cluster but have some clusters in
@@ -892,7 +892,6 @@ static unsigned long cluster_alloc_swap_entry(struct swap_info_struct *si, int o goto done; } } - done: cluster->next[order] = offset;return found;
Signature · 1 line
2.47.0
PATCH 2/13 mm, swap: fold swap_info_get_cont in the only caller
In reply to Kairui SongFrom: Kairui Song <[email protected]>
The name of the function is confusing, and the code is much easier to follow after folding, also rename the confusing naming "p" to more meaningful "si".
--- mm/swapfile.c | 39 +++++++++++++++------------------------ 1 file changed, 15 insertions(+), 24 deletions(-)
mm/swapfile.c +15 -24
@@ -1359,22 +1359,6 @@ static struct swap_info_struct *_swap_info_get(swp_entry_t entry)return NULL ; } -static struct swap_info_struct *swap_info_get_cont (swp_entry_t entry, -struct swap_info_struct *q) -{ -struct swap_info_struct *p; - - p =_swap_info_get (entry); - -if (p != q) { -if (q !=NULL ) -spin_unlock (&q->lock); -if (p !=NULL ) -spin_lock (&p->lock); - } -return p; -} -static unsigned char __swap_entry_free_locked (struct swap_info_struct *si,unsigned long offset,unsigned char usage)
@@ -1671,14 +1655,14 @@ static int swp_entry_cmp(const void *ent1, const void *ent2)void swapcache_free_entries (swp_entry_t *entries,int n) { -struct swap_info_struct *p, *prev; +struct swap_info_struct *si, *prev;int i;if (n <=0 )return ; prev =NULL ; - p =NULL ; + si =NULL ;/* * Sort swap entries by swap device, so each lock is only taken once.
@@ -1688,13 +1672,20 @@ void swapcache_free_entries(swp_entry_t *entries, int n)if (nr_swapfiles >1 )sort (entries, n,sizeof (entries[0 ]), swp_entry_cmp,NULL );for (i =0 ; i < n; ++i) { - p =swap_info_get_cont (entries[i], prev); -if (p) -swap_entry_range_free (p, entries[i],1 ); - prev = p; + si =_swap_info_get (entries[i]); + +if (si != prev) { +if (prev !=NULL ) +spin_unlock (&prev->lock); +if (si !=NULL ) +spin_lock (&si->lock); + } +if (si) +swap_entry_range_free (si, entries[i],1 ); + prev = si; } -if (p) -spin_unlock (&p->lock); +if (si) +spin_unlock (&si->lock); }int __swap_count (swp_entry_t entry)
Signature · 1 line
2.47.0
PATCH 3/13 mm, swap: remove old allocation path for HDD
In reply to Kairui SongFrom: Kairui Song <[email protected]>
We are currently using different swap allocation algorithm for HDD and non-HDD. This leads to the existing of different set of locking, and the code path is heavily bloated, causing troubles for further optimization and maintenance.
This commit removes all HDD swap allocation and related dead code, and use cluster allocation algorithm instead.
The performance may drop a little bit temporarily, and should be
negligible: The main advantage of legacy HDD allocation algorithm is that is tend to use continuous slots, but swap device gets fragmented quickly anyway, and the attempt to use continuous slots will fail easily.
This commit also enables mTHP swap on HDD, which should be beneficial, and following commits will adapt and optimize the cluster allocator for HDD.
--- include/linux/swap.h | 3 - mm/swapfile.c | 235 ++----------------------------------------- 2 files changed, 9 insertions(+), 229 deletions(-)
include/linux/swap.h +0 -3
@@ -309,9 +309,6 @@ struct swap_info_struct {unsigned int highest_bit;/* index of last free in swap_map */ unsigned int pages;/* total of usable pages of swap */ unsigned int inuse_pages;/* number of those currently in use */ -unsigned int cluster_next;/* likely index for next allocation */ -unsigned int cluster_nr;/* countdown to next cluster search */ -unsigned int __percpu *cluster_next_cpu;/*percpu index for next allocation */ struct percpu_cluster __percpu *percpu_cluster;/* per cpu's swap location */ struct rb_root swap_extent_root;/* root of the swap extent rbtree */ struct block_device *bdev;/* swap device or bdev of swap file */
mm/swapfile.c +9 -226
307 lines not on this page · raw message
Signature · 1 line
2.47.0
PATCH 4/13 mm, swap: use cluster lock for HDD
In reply to Kairui SongFrom: Kairui Song <[email protected]>
Cluster lock (ci->lock) was introduce to reduce contention for certain operations. Using cluster lock for HDD is not helpful as HDD have a poor performance, so locking isn't the bottleneck. But having different set of locks for HDD / non-HDD prevents further rework of device lock (si->lock).
This commit just changed all lock_cluster_or_swap_info to lock_cluster, which is a safe and straight conversion since cluster info is always allocated now, also removed all cluster_info related checks.
--- mm/swapfile.c | 107 ++++++++++++++++---------------------------------- 1 file changed, 34 insertions(+), 73 deletions(-)
mm/swapfile.c +34 -73
@@ -58,10 +58,9 @@ static void swap_entry_range_free(struct swap_info_struct *si, swp_entry_t entrystatic void swap_range_alloc (struct swap_info_struct *si,unsigned long offset,unsigned int nr_entries);static bool folio_swapcache_freeable (struct folio *folio); -static struct swap_cluster_info *lock_cluster_or_swap_info ( -struct swap_info_struct *si,unsigned long offset); -static void unlock_cluster_or_swap_info (struct swap_info_struct *si, -struct swap_cluster_info *ci); +static struct swap_cluster_info *lock_cluster (struct swap_info_struct *si, +unsigned long offset); +static void unlock_cluster (struct swap_cluster_info *ci);static DEFINE_SPINLOCK (swap_lock);static unsigned int nr_swapfiles;
@@ -222,9 +221,9 @@ static int __try_to_reclaim_swap(struct swap_info_struct *si,* swap_map is HAS_CACHE only, which means the slots have no page table * reference or pending writeback, and can't be allocated to others. */ - ci =lock_cluster_or_swap_info (si, offset); + ci =lock_cluster (si, offset); need_reclaim =swap_is_has_cache (si, offset, nr_pages); -unlock_cluster_or_swap_info (si, ci); +unlock_cluster (ci);if (!need_reclaim) goto out_unlock;
@@ -404,45 +403,15 @@ static inline struct swap_cluster_info *lock_cluster(struct swap_info_struct *si {struct swap_cluster_info *ci; - ci = si->cluster_info; -if (ci) { - ci += offset /SWAPFILE_CLUSTER ; -spin_lock (&ci->lock); - } -return ci; -} - -static inline void unlock_cluster (struct swap_cluster_info *ci) -{ -if (ci) -spin_unlock (&ci->lock); -} - -/* -* Determine the locking method in use for this device. Return -* swap_cluster_info if SSD-style cluster-based locking is in place. -*/ -static inline struct swap_cluster_info *lock_cluster_or_swap_info ( -struct swap_info_struct *si,unsigned long offset) -{ -struct swap_cluster_info *ci; - -/* Try to use fine-grained SSD-style locking if available: */ - ci =lock_cluster (si, offset); -/* Otherwise, fall back to traditional, coarse locking: */ -if (!ci) -spin_lock (&si->lock); + ci = &si->cluster_info[offset /SWAPFILE_CLUSTER ]; +spin_lock (&ci->lock);return ci; } -static inline void unlock_cluster_or_swap_info (struct swap_info_struct *si, -struct swap_cluster_info *ci) +static inline void unlock_cluster (struct swap_cluster_info *ci) { -if (ci) -unlock_cluster (ci); -else -spin_unlock (&si->lock); +spin_unlock (&ci->lock); }/* Add a cluster to discard list and schedule it to do discard */
@@ -558,9 +527,6 @@ static void inc_cluster_info_page(struct swap_info_struct *si,unsigned long idx = page_nr /SWAPFILE_CLUSTER ;struct swap_cluster_info *ci; -if (!cluster_info) -return ; - ci = cluster_info + idx; ci->count++;
@@ -576,9 +542,6 @@ static void inc_cluster_info_page(struct swap_info_struct *si,static void dec_cluster_info_page (struct swap_info_struct *si,struct swap_cluster_info *ci,int nr_pages) { -if (!si->cluster_info) -return ; -VM_BUG_ON (ci->count < nr_pages);VM_BUG_ON (cluster_is_free (ci));lockdep_assert_held (&si->lock);
@@ -995,8 +958,6 @@ static int cluster_alloc_swap(struct swap_info_struct *si, {int n_ret =0 ; -VM_BUG_ON (!si->cluster_info); -while (n_ret < nr) {unsigned long offset =cluster_alloc_swap_entry (si, order, usage);
@@ -1036,10 +997,10 @@ static int scan_swap_map_slots(struct swap_info_struct *si, }/* -* Swapfile is not block device or not using clusters so unable +* Swapfile is not block device so unable * to allocate large entries. */ -if (!(si->flags &SWP_BLKDEV ) || !si->cluster_info) +if (!(si->flags &SWP_BLKDEV ))return 0 ; }
@@ -1279,9 +1240,9 @@ static unsigned char __swap_entry_free(struct swap_info_struct *si,unsigned long offset =swp_offset (entry);unsigned char usage; - ci =lock_cluster_or_swap_info (si, offset); + ci =lock_cluster (si, offset); usage =__swap_entry_free_locked (si, offset,1 ); -unlock_cluster_or_swap_info (si, ci); +unlock_cluster (ci);if (!usage)free_swap_slot (entry);
@@ -1304,14 +1265,14 @@ static bool __swap_entries_free(struct swap_info_struct *si,if (nr >SWAPFILE_CLUSTER - offset %SWAPFILE_CLUSTER ) goto fallback; - ci =lock_cluster_or_swap_info (si, offset); + ci =lock_cluster (si, offset);if (!swap_is_last_map (si, offset, nr, &has_cache)) { -unlock_cluster_or_swap_info (si, ci); +unlock_cluster (ci); goto fallback; }for (i =0 ; i < nr; i++)WRITE_ONCE (si->swap_map[offset + i],SWAP_HAS_CACHE ); -unlock_cluster_or_swap_info (si, ci); +unlock_cluster (ci);if (!has_cache) {for (i =0 ; i < nr; i++)
@@ -1367,7 +1328,7 @@ static void cluster_swap_free_nr(struct swap_info_struct *si, DECLARE_BITMAP(to_free, BITS_PER_LONG) = { 0 };int i, nr; - ci =lock_cluster_or_swap_info (si, offset); + ci =lock_cluster (si, offset);while (nr_pages) { nr =min (BITS_PER_LONG , nr_pages);for (i =0 ; i < nr; i++) {
@@ -1375,18 +1336,18 @@ static void cluster_swap_free_nr(struct swap_info_struct *si,bitmap_set (to_free, i,1 ); }if (!bitmap_empty (to_free,BITS_PER_LONG )) { -unlock_cluster_or_swap_info (si, ci); +unlock_cluster (ci);for_each_set_bit (i, to_free,BITS_PER_LONG )free_swap_slot (swp_entry (si->type, offset + i));if (nr == nr_pages)return ;bitmap_clear (to_free,0 ,BITS_PER_LONG ); - ci =lock_cluster_or_swap_info (si, offset); + ci =lock_cluster (si, offset); } offset += nr; nr_pages -= nr; } -unlock_cluster_or_swap_info (si, ci); +unlock_cluster (ci); }/*
@@ -1425,9 +1386,9 @@ void put_swap_folio(struct folio *folio, swp_entry_t entry)if (!si)return ; - ci =lock_cluster_or_swap_info (si, offset); + ci =lock_cluster (si, offset);if (size >1 &&swap_is_has_cache (si, offset, size)) { -unlock_cluster_or_swap_info (si, ci); +unlock_cluster (ci);spin_lock (&si->lock);swap_entry_range_free (si, entry, size);spin_unlock (&si->lock);
@@ -1435,14 +1396,14 @@ void put_swap_folio(struct folio *folio, swp_entry_t entry) }for (int i =0 ; i < size; i++, entry.val++) {if (!__swap_entry_free_locked (si, offset + i,SWAP_HAS_CACHE )) { -unlock_cluster_or_swap_info (si, ci); +unlock_cluster (ci);free_swap_slot (entry);if (i == size -1 )return ; -lock_cluster_or_swap_info (si, offset); +lock_cluster (si, offset); } } -unlock_cluster_or_swap_info (si, ci); +unlock_cluster (ci); }static int swp_entry_cmp (const void *ent1,const void *ent2)
@@ -1506,9 +1467,9 @@ int swap_swapcount(struct swap_info_struct *si, swp_entry_t entry)struct swap_cluster_info *ci;int count; - ci =lock_cluster_or_swap_info (si, offset); + ci =lock_cluster (si, offset); count =swap_count (si->swap_map[offset]); -unlock_cluster_or_swap_info (si, ci); +unlock_cluster (ci);return count; }
@@ -1531,7 +1492,7 @@ int swp_swapcount(swp_entry_t entry) offset =swp_offset (entry); - ci =lock_cluster_or_swap_info (si, offset); + ci =lock_cluster (si, offset); count =swap_count (si->swap_map[offset]);if (!(count &COUNT_CONTINUED ))
@@ -1554,7 +1515,7 @@ int swp_swapcount(swp_entry_t entry) n *= (SWAP_CONT_MAX +1 ); }while (tmp_count &COUNT_CONTINUED ); out: -unlock_cluster_or_swap_info (si, ci); +unlock_cluster (ci);return count; }
@@ -1569,8 +1530,8 @@ static bool swap_page_trans_huge_swapped(struct swap_info_struct *si,int i;bool ret = false; - ci =lock_cluster_or_swap_info (si, offset); -if (!ci || nr_pages ==1 ) { + ci =lock_cluster (si, offset); +if (nr_pages ==1 ) {if (swap_count (map[roffset])) ret = true; goto unlock_out;
@@ -1582,7 +1543,7 @@ static bool swap_page_trans_huge_swapped(struct swap_info_struct *si, } } unlock_out: -unlock_cluster_or_swap_info (si, ci); +unlock_cluster (ci);return ret; }
@@ -3412,7 +3373,7 @@ static int __swap_duplicate(swp_entry_t entry, unsigned char usage, int nr) offset =swp_offset (entry);VM_WARN_ON (nr >SWAPFILE_CLUSTER - offset %SWAPFILE_CLUSTER );VM_WARN_ON (usage ==1 && nr >1 ); - ci =lock_cluster_or_swap_info (si, offset); + ci =lock_cluster (si, offset); err =0 ;for (i =0 ; i < nr; i++) {
@@ -3467,7 +3428,7 @@ static int __swap_duplicate(swp_entry_t entry, unsigned char usage, int nr) } unlock_out: -unlock_cluster_or_swap_info (si, ci); +unlock_cluster (ci);return err; }
Signature · 1 line
2.47.0
PATCH 5/13 mm, swap: clean up device availability check
In reply to Kairui SongFrom: Kairui Song <[email protected]>
Remove highest_bit and lowest_bit. After HDD allocation path is removed, only purpose of these two fields is to judge if the device is full or not, which can be done by checking inuse_pages instead.
--- fs/btrfs/inode.c | 1 - fs/iomap/swapfile.c | 1 - include/linux/swap.h | 2 -- mm/page_io.c | 1 - mm/swapfile.c | 38 ++++++++------------------------------ 5 files changed, 8 insertions(+), 35 deletions(-)
fs/btrfs/inode.c +0 -1
@@ -10023,7 +10023,6 @@ static int btrfs_swap_activate(struct swap_info_struct *sis, struct file *file, *span = bsi.highest_ppage - bsi.lowest_ppage +1 ; sis->max = bsi.nr_pages; sis->pages = bsi.nr_pages -1 ; - sis->highest_bit = bsi.nr_pages -1 ; return bsi.nr_extents; }#else
fs/iomap/swapfile.c +0 -1
@@ -189,7 +189,6 @@ int iomap_swapfile_activate(struct swap_info_struct *sis, *pagespan =1 + isi.highest_ppage - isi.lowest_ppage; sis->max = isi.nr_pages; sis->pages = isi.nr_pages -1 ; - sis->highest_bit = isi.nr_pages -1 ;return isi.nr_extents; }EXPORT_SYMBOL_GPL (iomap_swapfile_activate);
include/linux/swap.h +0 -2
@@ -305,8 +305,6 @@ struct swap_info_struct {struct list_head frag_clusters[SWAP_NR_ORDERS ];/* list of cluster that are fragmented or contented */ unsigned int frag_cluster_nr[SWAP_NR_ORDERS ]; -unsigned int lowest_bit;/* index of first free in swap_map */ -unsigned int highest_bit;/* index of last free in swap_map */ unsigned int pages;/* total of usable pages of swap */ unsigned int inuse_pages;/* number of those currently in use */ struct percpu_cluster __percpu *percpu_cluster;/* per cpu's swap location */
mm/page_io.c +0 -1
@@ -163,7 +163,6 @@ int generic_swapfile_activate(struct swap_info_struct *sis, page_no =1 ;/* force Empty message */ sis->max = page_no; sis->pages = page_no -1 ; - sis->highest_bit = page_no -1 ; out:return ret; bad_bmap:
mm/swapfile.c +8 -30
@@ -55,7 +55,7 @@ static bool swap_count_continued(struct swap_info_struct *, pgoff_t,static void free_swap_count_continuations (struct swap_info_struct *);static void swap_entry_range_free (struct swap_info_struct *si,swp_entry_t entry,unsigned int nr_pages); -static void swap_range_alloc (struct swap_info_struct *si,unsigned long offset, +static void swap_range_alloc (struct swap_info_struct *si,unsigned int nr_entries);static bool folio_swapcache_freeable (struct folio *folio); static struct swap_cluster_info *lock_cluster(struct swap_info_struct *si,
@@ -647,7 +647,7 @@ static void cluster_alloc_range(struct swap_info_struct *si, struct swap_cluster }memset (si->swap_map + start, usage, nr_pages); -swap_range_alloc (si, start, nr_pages); +swap_range_alloc (si, nr_pages); ci->count += nr_pages;if (ci->count ==SWAPFILE_CLUSTER ) {
@@ -876,19 +876,11 @@ static void del_from_avail_list(struct swap_info_struct *si)spin_unlock (&swap_avail_lock); } -static void swap_range_alloc (struct swap_info_struct *si,unsigned long offset, +static void swap_range_alloc (struct swap_info_struct *si,unsigned int nr_entries) { -unsigned int end = offset + nr_entries -1 ; - -if (offset == si->lowest_bit) - si->lowest_bit += nr_entries; -if (end == si->highest_bit) -WRITE_ONCE (si->highest_bit, si->highest_bit - nr_entries);WRITE_ONCE (si->inuse_pages, si->inuse_pages + nr_entries);if (si->inuse_pages == si->pages) { - si->lowest_bit = si->max; - si->highest_bit =0 ;del_from_avail_list (si);if (vm_swap_full ())
@@ -921,15 +913,8 @@ static void swap_range_free(struct swap_info_struct *si, unsigned long offset,for (i =0 ; i < nr_entries; i++)clear_bit (offset + i, si->zeromap); -if (offset < si->lowest_bit) - si->lowest_bit = offset; -if (end > si->highest_bit) { -bool was_full = !si->highest_bit; - -WRITE_ONCE (si->highest_bit, end); -if (was_full && (si->flags &SWP_WRITEOK )) -add_to_avail_list (si); - } +if (si->inuse_pages == si->pages) +add_to_avail_list (si);if (si->flags &SWP_BLKDEV ) swap_slot_free_notify = si->bdev->bd_disk->fops->swap_slot_free_notify;
@@ -1035,15 +1020,12 @@ int get_swap_pages(int n_goal, swp_entry_t swp_entries[], int entry_order)plist_requeue (&si->avail_lists[node], &swap_avail_heads[node]);spin_unlock (&swap_avail_lock);spin_lock (&si->lock); -if (!si->highest_bit || !(si->flags &SWP_WRITEOK )) { +if ((si->inuse_pages == si->pages) || !(si->flags &SWP_WRITEOK )) {spin_lock (&swap_avail_lock);if (plist_node_empty (&si->avail_lists[node])) {spin_unlock (&si->lock); goto nextsi; } -WARN (!si->highest_bit, -"swap_info %d in list but !highest_bit\n" , - si->type);WARN (!(si->flags &SWP_WRITEOK ),"swap_info %d in list but !SWP_WRITEOK\n" , si->type);
@@ -2425,8 +2407,8 @@ static void _enable_swap_info(struct swap_info_struct *si)*/ plist_add (&si->list, &swap_active_head); -/* add to available list iff swap device is not full */ -if (si->highest_bit) +/* add to available list if swap device is not full */ +if (si->inuse_pages < si->pages)add_to_avail_list (si); }
@@ -2590,7 +2572,6 @@ SYSCALL_DEFINE1(swapoff, const char __user *, specialfile)drain_mmlist ();/* wait for anyone still in scan_swap_map_slots */ - p->highest_bit =0 ;/* cuts scans short */ while (p->flags >=SWP_SCANNING ) {spin_unlock (&p->lock);spin_unlock (&swap_lock);
@@ -2925,8 +2906,6 @@ static unsigned long read_swap_header(struct swap_info_struct *si,return 0 ; } - si->lowest_bit =1 ; - maxpages = swapfile_maximum_size; last_page = swap_header->info.last_page;if (!last_page) {
@@ -2943,7 +2922,6 @@ static unsigned long read_swap_header(struct swap_info_struct *si,if ((unsigned int )maxpages ==0 ) maxpages =UINT_MAX ; } - si->highest_bit = maxpages -1 ;if (!maxpages)return 0 ;
Signature · 1 line
2.47.0
PATCH 6/13 mm, swap: clean up plist removal and adding
In reply to Kairui SongFrom: Kairui Song <[email protected]>
When swap device is full (inuse_pages == pages), it should be removed from the plist. And if any slot is freed, the swap device should be added back to the plist. On swapoff / swapon, the swap device will also be force removed / added.
This is currently serialized by si->lock, and some historical sanity check code are still here. This commit decouple this from the protection of si->lock and clean it up to prepare for si->lock rework.
Noticing inuse_pages counter is the only thing decides if a device should be removed from or added to the plist (except swapon / swapoff as a special case), and inuse_pages is a very hot counter. So to avoid extra overhead on the counter update hot path, and make it possible to check and update the plist when the counter value changes, embed the plist state into the inuse_pages counter, and turn the counter into an atomic. This way we can check and update the counter with one CAS and avoid any extra synchronization.
If the counter is full (inuse_pages == pages) with the off-list bit unset, try to remove it from the plist. If the counter is not full (inuse_pages != pages) with the off-list bit set, try to add it to the plist. Removing and adding is serialized with lock as well as the bit setting. Ordinary counter updates will be lockless.
--- include/linux/swap.h | 2 +- mm/swapfile.c | 182 +++++++++++++++++++++++++++++++------------ 2 files changed, 132 insertions(+), 52 deletions(-)
include/linux/swap.h +1 -1
@@ -306,7 +306,7 @@ struct swap_info_struct {/* list of cluster that are fragmented or contented */ unsigned int frag_cluster_nr[SWAP_NR_ORDERS ];unsigned int pages;/* total of usable pages of swap */ -unsigned int inuse_pages;/* number of those currently in use */ +atomic_long_t inuse_pages;/* number of those currently in use */ struct percpu_cluster __percpu *percpu_cluster;/* per cpu's swap location */ struct rb_root swap_extent_root;/* root of the swap extent rbtree */ struct block_device *bdev;/* swap device or bdev of swap file */
mm/swapfile.c +131 -51
332 lines not on this page · raw message
Signature · 1 line
2.47.0
PATCH 7/13 mm, swap: hold a reference of si during scan and clean up flags
In reply to Kairui SongFrom: Kairui Song <[email protected]>
The flag SWP_SCANNING was used as an indicator of whether a device is being scanned, and prevents swap off. But it's already no longer used. The only thing protects the scanning now is the si lock.
However allocation path may drop the si lock, in theory this could leaf to UAF.
So clean this up, just hold a reference for whole allocation path. So per CPU counter killing will wait for existing scan and other usage. The flag SWP_SCANNING can also be dropped.
--- include/linux/swap.h | 1 - mm/swapfile.c | 62 +++++++++++++++++++++++--------------------- 2 files changed, 33 insertions(+), 30 deletions(-)
include/linux/swap.h +0 -1
@@ -219,7 +219,6 @@ enum {SWP_STABLE_WRITES = (1 <<11 ),/* no overwrite PG_writeback pages */ SWP_SYNCHRONOUS_IO = (1 << 12), /* synchronous IO is efficient */ /* add others here before... */ - SWP_SCANNING = (1 << 14), /* refcount in scan_swap_map */ };#define SWAP_CLUSTER_MAX 32UL
mm/swapfile.c +33 -29
93 lines not on this page · raw message
Signature · 1 line
2.47.0
PATCH 8/13 mm, swap: use an enum to define all cluster flags and wrap flags changes
In reply to Kairui SongFrom: Kairui Song <[email protected]>
Currently we are only using flags to indicate which list the cluster is on, using one bit for each list type might be a waste as the list type grows we will consume too much bits. And current the mixed usage of "&" and "==" is a bit confusing.
Make it clean by using an enum to define all possible cluster status, only an off-list cluster will have the NONE (0) flag. And use a wrapper to annotate and sanitize all flag setting and list movement.
--- include/linux/swap.h | 17 +++++++--- mm/swapfile.c | 76 +++++++++++++++++++++++--------------------- 2 files changed, 53 insertions(+), 40 deletions(-)
include/linux/swap.h +13 -4
@@ -256,10 +256,19 @@ struct swap_cluster_info {u8 order;struct list_head list; }; -#define CLUSTER_FLAG_FREE 1/* This cluster is free */ -#define CLUSTER_FLAG_NONFULL 2/* This cluster is on nonfull list */ -#define CLUSTER_FLAG_FRAG 4/* This cluster is on nonfull list */ -#define CLUSTER_FLAG_FULL 8/* This cluster is on full list */ + +/* +* All on-list cluster must have a non-zero flag. +*/ +enum swap_cluster_flags { +CLUSTER_FLAG_NONE =0 ,/* For temporary off-list cluster */ +CLUSTER_FLAG_FREE , +CLUSTER_FLAG_NONFULL , +CLUSTER_FLAG_FRAG , +CLUSTER_FLAG_FULL , +CLUSTER_FLAG_DISCARD , +CLUSTER_FLAG_MAX , +};/* * The first page in the swap file is the swap header, which is always marked
mm/swapfile.c +40 -36
137 lines not on this page · raw message
Signature · 1 line
2.47.0
PATCH 9/13 mm, swap: reduce contention on device lock
In reply to Kairui SongFrom: Kairui Song <[email protected]>
Currently swap locking is mainly composed of two locks, cluster lock (ci->lock) and device lock (si->lock).
Cluster lock is much more fine-grained, so it will be best to use ci->lock instead of si->lock as much as possible.
Following the new cluster allocator design, many operation doesn't need to touch si->lock at all. In practise, we only need to take si->lock when moving clusters between lists.
To archive it, this commit reworked the locking pattern of all si->lock and ci->lock users, eliminated all usage of ci->lock inside si->lock, introduce new design to avoid touching si->lock as much as possible.
For minimal contention for allocation and easier understanding, two ideas are introduced with the corresponding helpers: isolation and relocation:
- Clusters will be isolated from list upon being scanned for allocation, so scanning of on-list cluster no longer need to hold the si->lock except the very moment, and hence removed the ci->lock usage inside si->lock.
In the new allocator design, one cluster always get moved after scanning (free -> nonfull, nonfull -> frag, frag -> frag tail) so this introduces no extra overhead. This also greatly reduced the contention of both si->lock and ci->lock as other CPUs won't walk onto the same cluster by iterating the list.
The off-list time window of a cluster is also minimal, one CPU can at most hold one cluster while scanning the 512 entries on it, which we used to busy wait with a spin lock.
This is done with cluster_isolate_lock on scanning of a new cluster.
Note: Scanning of per CPU cluster is a special case, it doesn't isolation the cluster. That's because it doesn't need to hold the si->lock at all, it simply acquire the ci->lock of previously used cluster and use it.
- Cluster will be relocated after allocation or freeing according to it's count and status.
Allocations no longer holds si->lock now, and may drop ci->lock for reclaim, so the cluster could be moved to anywhere. Besides, isolation clears all flags when it takes the cluster off list (The flag must be in-sync with list status, so cluster users don't need to touch si->lock for checking its list status. This is important for reducing contention on si->lock). So the cluster have to be relocated according to its usage after being allocation to the right list.
This is done with relocate_cluster after allocation, or [partial_]free_cluster after freeing.
Now except swapon / swapoff and discard, isolation and relocation are the only two places that need to take si->lock. And as each CPU will keep using its per-CPU cluster as much as possible and a cluster have 512 entries to be consumed, si->lock is rarely touched.
The lock contention of si->lock is now barely observable. Test with build linux kernel with defconfig showed huge performance improvement:
tiem make -j96 / 768M memcg, 4K pages, 10G ZRAM, on Intel 8255C:
Before:
Sys time: 73578.30, Real time: 864.05
After: (-50.7% sys time, -44.8% real time)
Sys time: 36227.49, Real time: 476.66
time make -j96 / 1152M memcg, 64K mTHP, 10G ZRAM, on Intel 8255C:
(avg of 4 test run)
Before:
Sys time: 74044.85, Real time: 846.51 hugepages-64kB/stats/swpout: 1735216 hugepages-64kB/stats/swpout_fallback: 430333
After: (-40.4% sys time, -37.1% real time)
Sys time: 44160.56, Real time: 532.07
hugepages-64kB/stats/swpout: 1786288
hugepages-64kB/stats/swpout_fallback: 243384
time make -j32 / 512M memcg, 4K pages, 5G ZRAM, on AMD 7K62:
Before:
Sys time: 8098.21, Real time: 401.3
After: (-22.6% sys time, -12.8% real time )
Sys time: 6265.02, Real time: 349.83
The allocation success rate also slightly improved as we sanitized the usage of clusters with new defined helpers and locks, so temporarily dropped si->lock or ci->lock won't cause cluster order shuffle.
--- include/linux/swap.h | 5 +- mm/swapfile.c | 418 ++++++++++++++++++++++++------------------- 2 files changed, 239 insertions(+), 184 deletions(-)
include/linux/swap.h +4 -1
@@ -265,6 +265,8 @@ enum swap_cluster_flags {CLUSTER_FLAG_FREE ,CLUSTER_FLAG_NONFULL ,CLUSTER_FLAG_FRAG , +/* Clusters with flags above are allocatable */ +CLUSTER_FLAG_USABLE =CLUSTER_FLAG_FRAG ,CLUSTER_FLAG_FULL ,CLUSTER_FLAG_DISCARD , CLUSTER_FLAG_MAX,
@@ -290,6 +292,7 @@ enum swap_cluster_flags {* throughput. */ struct percpu_cluster { +local_lock_t lock;/* Protect the percpu_cluster above */ unsigned int next[SWAP_NR_ORDERS ];/* Likely next allocation offset */ };
@@ -312,7 +315,7 @@ struct swap_info_struct {/* list of cluster that contains at least one free slot */ struct list_head frag_clusters[SWAP_NR_ORDERS ];/* list of cluster that are fragmented or contented */ -unsigned int frag_cluster_nr[SWAP_NR_ORDERS ]; +atomic_long_t frag_cluster_nr[SWAP_NR_ORDERS ];unsigned int pages;/* total of usable pages of swap */ atomic_long_t inuse_pages;/* number of those currently in use */ struct percpu_cluster __percpu *percpu_cluster;/* per cpu's swap location */
mm/swapfile.c +235 -183
735 lines not on this page · raw message
Signature · 1 line
2.47.0
PATCH 10/13 mm, swap: simplify percpu cluster updating
In reply to Kairui SongFrom: Kairui Song <[email protected]>
Instead of using a returning argument, we can simply store the next cluster offset to the fixed percpu location, which reduce the stack usage and simplify the function:
Object size:./scripts/bloat-o-meter mm/swapfile.o mm/swapfile.o.new add/remove: 0/0 grow/shrink: 0/2 up/down: 0/-271 (-271)
Function old new deltaget_swap_pages 2847 2733 -114alloc_swap_scan_cluster 894 737 -157
Total: Before=30833, After=30562, chg -0.88%
Stack usage:
Before:
swapfile.c:1190:5:get_swap_pages 240 static
After:
swapfile.c:1185:5:get_swap_pages 216 static
--- include/linux/swap.h | 4 ++-- mm/swapfile.c | 57 ++++++++++++++++++++------------------------ 2 files changed, 28 insertions(+), 33 deletions(-)
include/linux/swap.h +2 -2
@@ -276,9 +276,9 @@ enum swap_cluster_flags {* The first page in the swap file is the swap header, which is always marked * bad to prevent it from being allocated as an entry. This also prevents the * cluster to which it belongs being marked free. Therefore 0 is safe to use as -* a sentinel to indicate next is not valid in percpu_cluster. +* a sentinel to indicate an entry is not valid. */ -#define SWAP_NEXT_INVALID 0 +#define SWAP_ENTRY_INVALID 0#ifdef CONFIG_THP_SWAP #define SWAP_NR_ORDERS (PMD_ORDER + 1)
mm/swapfile.c +26 -31
137 lines not on this page · raw message
Signature · 1 line
2.47.0
PATCH 11/13 mm, swap: introduce a helper for retrieving cluster from offset
In reply to Kairui SongFrom: Kairui Song <[email protected]>
It's a common operation to retrieve the cluster info from offset, introduce a helper for this.
--- mm/swapfile.c | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-)
mm/swapfile.c +11 -5
@@ -423,6 +423,12 @@ static inline unsigned int cluster_index(struct swap_info_struct *si,return ci - si->cluster_info; } +static inline struct swap_cluster_info *offset_to_cluster (struct swap_info_struct *si, +unsigned long offset) +{ +return &si->cluster_info[offset /SWAPFILE_CLUSTER ]; +} +static inline unsigned int cluster_offset (struct swap_info_struct *si,struct swap_cluster_info *ci) {
@@ -434,7 +440,7 @@ static inline struct swap_cluster_info *lock_cluster(struct swap_info_struct *si {struct swap_cluster_info *ci; - ci = &si->cluster_info[offset /SWAPFILE_CLUSTER ]; + ci =offset_to_cluster (si, offset);spin_lock (&ci->lock);return ci;
@@ -756,7 +762,7 @@ static unsigned int alloc_swap_scan_cluster(struct swap_info_struct *si,bool need_reclaim, ret;struct swap_cluster_info *ci; - ci = &si->cluster_info[offset /SWAPFILE_CLUSTER ]; + ci =offset_to_cluster (si, offset);lockdep_assert_held (&ci->lock);if (end < nr_pages || ci->count + nr_pages >SWAPFILE_CLUSTER )
@@ -1457,10 +1463,10 @@ static void swap_entry_range_free(struct swap_info_struct *si, swp_entry_t entryunsigned char *map_end = map + nr_pages;struct swap_cluster_info *ci; -/* It should never free entries across different clusters */ -VM_BUG_ON ((offset /SWAPFILE_CLUSTER ) != ((offset + nr_pages -1 ) /SWAPFILE_CLUSTER )); - ci =lock_cluster (si, offset); + +/* It should never free entries across different clusters */ +VM_BUG_ON (ci !=offset_to_cluster (si, offset + nr_pages -1 ));VM_BUG_ON (cluster_is_free (ci));VM_BUG_ON (ci->count < nr_pages);
Signature · 1 line
2.47.0
PATCH 12/13 mm, swap: use a global swap cluster for non-rotation device
In reply to Kairui SongFrom: Kairui Song <[email protected]>
Non-rotation (SSD / ZRAM) device can tolerate fragmentations so the goal of SWAP allocator is to avoid contention of clusters. So it used a per-CPU cluster design, and each CPU will be using a different cluster as much as possible.
But HDD is very sensitive to fragmentations, contention is trivial compared to this. So just use one global cluster instead. This ensured each order will be wring to a same cluster as much as possible, which helps to make the IO more continuous.
This ensures the performance of cluster allocator is as good as the old allocator. Test after this commit compared to before this series:
make -j32 with tinyconfig, using 1G memcg limit and HDD swap:
Before this series:
114.44user 29.11system 39:42.90elapsed 6%CPU (0avgtext+0avgdata 157284maxresident)k 2901232inputs+0outputs (238877major+4227640minor)pagefaults
After this commit:
113.90user 23.81system 38:11.77elapsed 6%CPU (0avgtext+0avgdata 157260maxresident)k 2548728inputs+0outputs (235471major+4238110minor)pagefaults
--- include/linux/swap.h | 2 ++ mm/swapfile.c | 48 ++++++++++++++++++++++++++++++++------------ 2 files changed, 37 insertions(+), 13 deletions(-)
include/linux/swap.h +2 -0
@@ -319,6 +319,8 @@ struct swap_info_struct {unsigned int pages;/* total of usable pages of swap */ atomic_long_t inuse_pages;/* number of those currently in use */ struct percpu_cluster __percpu *percpu_cluster;/* per cpu's swap location */ +struct percpu_cluster *global_cluster;/* Use one global cluster for rotating device */ +spinlock_t global_cluster_lock;/* Serialize usage of global cluster */ struct rb_root swap_extent_root;/* root of the swap extent rbtree */ struct block_device *bdev;/* swap device or bdev of swap file */ struct file *swap_file;/* seldom referenced */
mm/swapfile.c +35 -13
87 lines not on this page · raw message
Signature · 1 line
2.47.0
PATCH 13/13 mm, swap_slots: remove slot cache for freeing path
In reply to Kairui SongFrom: Kairui Song <[email protected]>
The slot cache for freeing path is mostly for reducing the overhead of si->lock. As we have basically eliminated the si->lock usage for freeing path, it can be just removed.
This helps simplify the code, and avoids swap entries from being hold in cache upon freeing. The delayed freeing of entries have been causing trouble for further optimizations for zswap [1] and in theory will also cause more fragmentation, and extra overhead.
Test with build linux kernel showed both performance and fragmentation is better without the cache:
tiem make -j96 / 768M memcg, 4K pages, 10G ZRAM, avg of 4 test run::
Before:
Sys time: 36047.78, Real time: 472.43
After: (-7.6% sys time, -7.3% real time)
Sys time: 33314.76, Real time: 437.67
time make -j96 / 1152M memcg, 64K mTHP, 10G ZRAM, avg of 4 test run:
Before:
Sys time: 46859.04, Real time: 562.63 hugepages-64kB/stats/swpout: 1783392 hugepages-64kB/stats/swpout_fallback: 240875
After: (-23.3% sys time, -21.3% real time)
Sys time: 35958.87, Real time: 442.69 hugepages-64kB/stats/swpout: 1866267 hugepages-64kB/stats/swpout_fallback: 158330
Sequential SWAP should be also slightly faster, tests didn't show a measurable difference though, at least no regression:
Swapin 4G zero page on ZRAM (time in us):
Before (avg. 1923756)
1912391 1927023 1927957 1916527 1918263 1914284 1934753 1940813 1921791 After (avg. 1922290):
1919101 1925743 1916810 1917007 1923930 1935152 1917403 1923549 1921913
--- include/linux/swap_slots.h | 3 -- mm/swap_slots.c | 78 +++++---------------------------- mm/swapfile.c | 89 +++++++++++++++----------------------- 3 files changed, 44 insertions(+), 126 deletions(-)
include/linux/swap_slots.h +0 -3
@@ -16,15 +16,12 @@ struct swap_slots_cache {swp_entry_t *slots;int nr;int cur; -spinlock_t free_lock;/* protects slots_ret, n_ret */ -swp_entry_t *slots_ret;int n_ret; };void disable_swap_slots_cache_lock (void );void reenable_swap_slots_cache_unlock (void );void enable_swap_slots_cache (void ); -void free_swap_slot (swp_entry_t entry);extern bool swap_slot_cache_enabled;
mm/swap_slots.c +10 -68
160 lines not on this page · raw message
mm/swapfile.c +34 -55
170 lines not on this page · raw message
Signature · 1 line
2.47.0
Hi, Kairui,
Quoted 12 lines · Kairui Song
From: Kairui Song <[email protected]>
This series improved the swap allocator performance greatly by reworking the locking design and simplify a lot of code path.
This is follow up of previous swap cluster allocator series:
https://lore.kernel.org/linux-mm/[email protected]/And this series is based on an follow up fix of the swap cluster allocator:
https://lore.kernel.org/linux-mm/[email protected]/
Quoted 8 lines
This is part of the new swap allocator work item discussed in Chris's "Swap Abstraction" discussion at LSF/MM 2024, and "mTHP and swap allocator" discussion at LPC 2024.
Previous series introduced a fully cluster based allocation algorithm, this series completely get rid of the old allocation path and makes the allocator avoid grabbing the
si->lockunless needed. This bring huge performance gain and get rid of slot cache on freeing path.
Great!
Quoted 3 lines
Currently, swap locking is mainly composed of two locks, cluster lock (
ci->lock) and device lock (si->lock). The device lock is widely used to protect many things, causing it to be the main bottleneck for SWAP.
Device lock can be confusing with another device lock for struct device. Better to call it swap device lock?
Quoted 15 lines
Cluster lock is much more fine-grained, so it will be best to use
ci->lockinstead ofsi->lockas much as possible.
perf lockindicates this issue clearly. Doing linux kernel build using tmpfs and ZRAM with limited memory (make -j64 with 1G memcg and 4k pages), result of "perf lock contention -ab sleep 3":contended total wait max wait avg wait type caller
34948 53.63 s 7.11 ms 1.53 ms spinlock free_swap_and_cache_nr+0x350 16569 40.05 s 6.45 ms 2.42 ms spinlock get_swap_pages+0x231 11191 28.41 s 7.03 ms 2.54 ms spinlock swapcache_free_entries+0x59 4147 22.78 s 122.66 ms 5.49 ms spinlock page_vma_mapped_walk+0x6f3 4595 7.17 s 6.79 ms 1.56 ms spinlock swapcache_free_entries+0x59 406027 2.74 s 2.59 ms 6.74 us spinlock list_lru_add+0x39
Quoted 4 lines
...snip...
The top 5 caller are all users of
si->lock, total wait time up sums to several minutes in the 3 seconds time window.
Can you show results of perf record -g, perf report -g too? I have interest to check hot spot shifting too.
Quoted 9 lines
Following the new allocator design, many operation doesn't need to touch
si->lockat all. We only need to takesi->lockwhen doing operations across multiple clusters (eg. changing the cluster list), other operations only need to takeci->lock. So ideally allocator should always takeci->lockfirst, then, if needed, takesi->lock. But due to historical reasons,ci->lockis used insidesi->lockby design, causing lock inversion if we simply try to acquiresi->lockafter acquiringci->lock.
Quoted 6 lines
This series audited all
si->lockusage, simplify legacy codes, eliminate usage ofsi->lockas much as possible by introducing new designs based on the new cluster allocator.Old HDD allocation codes are removed, cluster allocator is adapted with small changes for HDD usage, test is looking OK.
I think that it's a good idea to remove HDD allocation specific code. Can you check the performance of swapping to HDD? However, I understand that many people have no HDD in hand.
Quoted 36 lines
And this also removed slot cache for freeing path. The performance is better without it, and this enables other clean up and optimizations as discussed before:
https://lore.kernel.org/all/CAMgjq7ACohT_uerSz8E_994ZZCv709Zor+43hdmesW_59W1BWw@mail.gmail.com/After this series, lock contention on
si->lockis nearly unobservable withperf lockwith the same test above :contended total wait max wait avg wait type caller
... snip ...91 204.62 us 4.51 us 2.25 us spinlock cluster_move+0x2e ... snip ... 47 125.62 us 4.47 us 2.67 us spinlock cluster_move+0x2e ... snip ... 23 63.15 us 3.95 us 2.74 us spinlock cluster_move+0x2e ... snip ... 17 41.26 us 4.58 us 2.43 us spinlock cluster_isolate_lock+0x1d... snip ...
cluster_moveandcluster_isolate_lockare basically the only users ofsi->locknow, performance gain is huge with reduced LOC.Tests
===Build kernel with defconfig on tmpfs with ZRAM as swap:
---Running a test matrix which is scaled up progressive for a intuitive result. The test are ran on top of tmpfs, using memory cgroup for memory limitation, on a 48c96t system.
12 test run for each case, it can be seen clearly that as concurrent job number goes higher the performance gain is higher, the performance is higher even with low concurrency.
Quoted 12 lines
make -j<NR> | System Time (seconds) | Total Time (seconds) (NR / Mem / ZRAM) | (Before / After / Delta) | (Before / After / Delta) With 4k pages only: 6 / 192M / 3G | 5258 / 5235 / -0.3% | 1420 / 1414 / -0.3% 12 / 256M / 4G | 5518 / 5337 / -3.3% | 758 / 742 / -2.1% 24 / 384M / 5G | 7091 / 5766 / -18.7% | 476 / 422 / -11.3% 48 / 768M / 7G | 11139 / 5831 / -47.7% | 330 / 221 / -33.0% 96 / 1.5G / 10G | 21303 / 11353 / -46.7% | 283 / 180 / -36.4% With 64k mTHP: 24 / 512M / 5G | 5104 / 4641 / -18.7% | 376 / 358 / -4.8% 48 / 1G / 7G | 8693 / 4662 / -18.7% | 257 / 176 / -31.5% 96 / 2G / 10G | 17056 / 10263 / -39.8% | 234 / 169 / -27.8%
How much is the swap in/out throughput before/after the change?
When I worked on swap in/out performance before, the hot spot shifts from swap related code to LRU lock and zone lock. Things may change a lot now.
If zram is used as swap device, the hot spot may become compression/decompression after solving the swap lock contention. To stress swap subsystem further, we may use a ram disk as swap. Previously, we have used a simulated pmem device (backed by DRAM). That can be setup as in,
https://pmem.io/blog/2016/02/how-to-emulate-persistent-memory/
After creating the raw block device: /dev/pmem0, we can do
$ mkswap /dev/pmem0 $ swapon /dev/pmem0
Can you use something similar if necessary?
Quoted 22 lines
With more aggressive setup, it shows clearly both the performance and fragmentation are better:
tiem make -j96 / 768M memcg, 4K pages, 10G ZRAM, on Intel 8255C * 2:
(avg of 4 test run)
Before:
Sys time: 73578.30, Real time: 864.05
tiem make -j96 / 1G memcg, 4K pages, 10G ZRAM:
After: (-54.7% sys time, -49.3% real time)
Sys time: 33314.76, Real time: 437.67time make -j96 / 1152M memcg, 64K mTHP, 10G ZRAM, on Intel 8255C * 2:
(avg of 4 test run)
Before:
Sys time: 74044.85, Real time: 846.51 hugepages-64kB/stats/swpout: 1735216 hugepages-64kB/stats/swpout_fallback: 430333
After: (-51.4% sys time, -47.7% real time, -63.2% mTHP failure) Sys time: 35958.87, Real time: 442.69 hugepages-64kB/stats/swpout: 1866267 hugepages-64kB/stats/swpout_fallback: 158330
Quoted 3 lines
There is a up to 54.7% improvement for build kernel test, and lower fragmentation rate. Performance improvement should be even larger for micro benchmarks
Very good result!
Quoted 58 lines
Build kernel with tinyconfig on tmpfs with HDD as swap:
---This test is similar to above, but HDD test is very noisy and slow, the deviation is huge, so just use tinyconfig instead and take the median test result of 3 test run, which looks OK:
Before this series:
114.44user 29.11system 39:42.90elapsed 6%CPU 2901232inputs+0outputs (238877major+4227640minor)pagefaultsAfter this commit:
113.90user 23.81system 38:11.77elapsed 6%CPU 2548728inputs+0outputs (235471major+4238110minor)pagefaultsSingle thread SWAP:
---Sequential SWAP should also be slightly faster as we removed a lot of unnecessary parts. Test using micro benchmark for swapout/in 4G zero memory using ZRAM, 10 test runs:
Swapout Before (avg. 3359304):
3353796 3358551 3371305 3356043 3367524 3355303 3355924 3354513 3360776Swapin Before (avg. 1928698):
1920283 1927183 1934105 1921373 1926562 1938261 1927726 1928636 1934155Swapout After (avg. 3347511, -0.4%):
3337863 3347948 3355235 3339081 3333134 3353006 3354917 3346055 3360359Swapin After (avg. 1922290, -0.3%):
1919101 1925743 1916810 1917007 1923930 1935152 1917403 1923549 1921913Worth noticing the patch "mm, swap: use a global swap cluster for non-rotation device" introduced minor overhead for certain tests (see the test results in commit message), but the gain from later commit covered that, it can be further improved later.
Suggested-by Chris Li <[email protected]>Signed-off-by Kairui Song <[email protected]>Kairui Song (13):
mm, swap: minor clean up for swap entry allocation mm, swap: foldswap_info_get_contin the only caller mm, swap: remove old allocation path for HDD
mm, swap: use cluster lock for HDD
mm, swap: clean up device availability check
mm, swap: clean up plist removal and adding
mm, swap: hold a reference of si during scan and clean up flags mm, swap: use an enum to define all cluster flags and wrap flags
changes
mm, swap: reduce contention on device lock
mm, swap: simplify percpu cluster updating
mm, swap: introduce a helper for retrieving cluster from offset mm, swap: use a global swap cluster for non-rotation device mm,swap_slots: remove slot cache for freeing path
Quoted 8 lines
fs/btrfs/inode.c | 1 - fs/iomap/swapfile.c | 1 - include/linux/swap.h | 36 +- include/linux/swap_slots.h | 3 - mm/page_io.c | 1 - mm/swap_slots.c | 78 +-- mm/swapfile.c | 1198 ++++++++++++++++-------------------- 7 files changed, 558 insertions(+), 760 deletions(-)
--
Best Regards,
Huang, Ying
Quoted 2 lines · Ying
Hi, Kairui,
Hi Ying,
Quoted 24 lines
Quoted 12 lines · Kairui Song
From: Kairui Song <[email protected]>
This series improved the swap allocator performance greatly by reworking the locking design and simplify a lot of code path.
This is follow up of previous swap cluster allocator series:
https://lore.kernel.org/linux-mm/[email protected]/And this series is based on an follow up fix of the swap cluster allocator:
https://lore.kernel.org/linux-mm/[email protected]/Quoted 8 lines
This is part of the new swap allocator work item discussed in Chris's "Swap Abstraction" discussion at LSF/MM 2024, and "mTHP and swap allocator" discussion at LPC 2024.
Previous series introduced a fully cluster based allocation algorithm, this series completely get rid of the old allocation path and makes the allocator avoid grabbing the
si->lockunless needed. This bring huge performance gain and get rid of slot cache on freeing path.
Quoted 8 lines
Great!
Quoted 3 lines
Currently, swap locking is mainly composed of two locks, cluster lock (
ci->lock) and device lock (si->lock). The device lock is widely used to protect many things, causing it to be the main bottleneck for SWAP.Device lock can be confusing with another device lock for struct device. Better to call it swap device lock?
Good idea, I'll use the term swap device lock then.
Quoted 18 lines
Quoted 18 lines
Cluster lock is much more fine-grained, so it will be best to use
ci->lockinstead ofsi->lockas much as possible.
perf lockindicates this issue clearly. Doing linux kernel build using tmpfs and ZRAM with limited memory (make -j64 with 1G memcg and 4k pages), result of "perf lock contention -ab sleep 3":contended total wait max wait avg wait type caller
34948 53.63 s 7.11 ms 1.53 ms spinlock free_swap_and_cache_nr+0x350 16569 40.05 s 6.45 ms 2.42 ms spinlock get_swap_pages+0x231 11191 28.41 s 7.03 ms 2.54 ms spinlock swapcache_free_entries+0x59 4147 22.78 s 122.66 ms 5.49 ms spinlock page_vma_mapped_walk+0x6f3 4595 7.17 s 6.79 ms 1.56 ms spinlock swapcache_free_entries+0x59 406027 2.74 s 2.59 ms 6.74 us spinlock list_lru_add+0x39...snip...
Quoted 5 lines
Quoted 2 lines
The top 5 caller are all users of
si->lock, total wait time up sums to several minutes in the 3 seconds time window.Can you show results of
perf record -g,perf report -gtoo? I have interest to check hot spot shifting too.
Sure. I think perf lock result is already good enough and cleaner. My test environment are mostly VM based so spinlock slow path may get offloaded to host, and can't be see by perf record, I collected following data after disabled paravirt spinlock:
The time consumption and stack trace of a page fault before:
- 78.45% 0.17% cc1 [kernel.kallsyms]
[k] asm_exc_page_fault
- 78.28% asm_exc_page_fault
- 78.18% exc_page_fault
- 78.17% do_user_addr_fault
- 78.09% handle_mm_fault
- 78.06% __handle_mm_fault
- 69.69% do_swap_page
- 55.87% alloc_swap_folio
- 55.60% mem_cgroup_swapin_charge_folio
- 55.48% charge_memcg
- 55.45% try_charge_memcg
- 55.36% try_to_free_mem_cgroup_pages
- do_try_to_free_pages
- 55.35% shrink_node
- 55.27% shrink_lruvec
- 55.13% try_to_shrink_lruvec
- 54.79% evict_folios
- 54.35% shrink_folio_list
- 30.01% add_to_swap
- 29.77%folio_alloc_swap
- 29.50%get_swap_pages
25.03% queued_spin_lock_slowpath
- 2.71%alloc_swap_scan_cluster
1.80% queued_spin_lock_slowpath
+
0.89% __try_to_reclaim_swap
- 1.74%swap_reclaim_full_clusters
1.74% queued_spin_lock_slowpath
- 10.88%try_to_unmap_flush_dirty
- 10.87%arch_tlbbatch_flush
- 10.85%on_each_cpu_cond_mask
smp_call_function_many_cond
+ 7.45% pageout
+ 2.71% try_to_unmap_flush
+ 1.90% try_to_unmap
+ 0.78% folio_referenced
- 9.41% cluster_swap_free_nr
- 9.39% free_swap_slot
- 9.35% swapcache_free_entries
8.40% queued_spin_lock_slowpath 0.93% swap_entry_range_free
- 3.61% swap_read_folio_bdev_sync
- 3.55% submit_bio_wait
- 3.51% submit_bio_noacct_nocheck
+ 3.46% __submit_bio
+ 7.71% do_pte_missing
+ 0.61% wp_page_copy
The queued_spin_lock_slowpath above is the si->lock, and there are multiple users of it so the total overhead is higher than shown.
After:
- 75.05% 0.43% cc1 [kernel.kallsyms]
[k] asm_exc_page_fault
- 74.62% asm_exc_page_fault
- 74.36% exc_page_fault
- 74.34% do_user_addr_fault
- 74.10% handle_mm_fault
- 73.96% __handle_mm_fault
- 67.55% do_swap_page
- 45.92% alloc_swap_folio
- 45.03% mem_cgroup_swapin_charge_folio
- 44.58% charge_memcg
- 44.44% try_charge_memcg
- 44.12% try_to_free_mem_cgroup_pages
- do_try_to_free_pages
- 44.10% shrink_node
- 43.86% shrink_lruvec
- 41.92% try_to_shrink_lruvec
- 40.67% evict_folios
- 37.12% shrink_folio_list
- 20.88% pageout
+ 20.02% swap_writepage
+ 0.72% shmem_writepage
- 4.08% add_to_swap
- 2.48%folio_alloc_swap
- 2.12%__mem_cgroup_try_charge_swap
- 1.47%swap_cgroup_record
+
1.32% _raw_spin_lock_irqsave
- 1.56%add_to_swap_cache
- 1.04% xas_store
+ 1.01%workingset_update_node
+ 3.97%try_to_unmap_flush_dirty
+ 3.51% folio_referenced
+ 2.24% __remove_mapping
+ 1.16% try_to_unmap
+ 0.52% try_to_unmap_flush
2.50%queued_spin_lock_slowpath
0.79% scan_folios
+ 1.20% try_to_inc_max_seq
+ 1.92% lru_add_drain
+ 0.73% vma_alloc_folio_noprof
- 9.81% swap_read_folio_bdev_sync
- 9.61% submit_bio_wait
+ 9.49% submit_bio_noacct_nocheck
- 8.06% cluster_swap_free_nr
- 8.02% swap_entry_range_free
+ 3.92% __mem_cgroup_uncharge_swap
+ 2.90% zram_slot_free_notify
0.58% clear_shadow_from_swap_cache
- 1.32% __folio_batch_add_and_move
- 1.30% folio_batch_move_lru
+ 1.10% folio_lruvec_lock_irqsave
spin_lock usage is much lower.
I prefer the perf lock output as it shows the exact time and user of locks.
Quoted 14 lines
Quoted 14 lines
Following the new allocator design, many operation doesn't need to touch
si->lockat all. We only need to takesi->lockwhen doing operations across multiple clusters (eg. changing the cluster list), other operations only need to takeci->lock. So ideally allocator should always takeci->lockfirst, then, if needed, takesi->lock. But due to historical reasons,ci->lockis used insidesi->lockby design, causing lock inversion if we simply try to acquiresi->lockafter acquiringci->lock.This series audited all
si->lockusage, simplify legacy codes, eliminate usage ofsi->lockas much as possible by introducing new designs based on the new cluster allocator.
Quoted 6 lines
Quoted 2 lines
Old HDD allocation codes are removed, cluster allocator is adapted with small changes for HDD usage, test is looking OK.
I think that it's a good idea to remove HDD allocation specific code. Can you check the performance of swapping to HDD? However, I understand that many people have no HDD in hand.
It's not hard to make cluster allocator work well with HDD in theory, see the commit "mm, swap: use a global swap cluster for non-rotation device".
The testing is not very reliable though, I found HDD swap performance is very unstable because of the IO pattern of HDD, so it's just a best effort try.
Quoted 49 lines
Quoted 36 lines
And this also removed slot cache for freeing path. The performance is better without it, and this enables other clean up and optimizations as discussed before:
https://lore.kernel.org/all/CAMgjq7ACohT_uerSz8E_994ZZCv709Zor+43hdmesW_59W1BWw@mail.gmail.com/After this series, lock contention on
si->lockis nearly unobservable withperf lockwith the same test above :contended total wait max wait avg wait type caller
... snip ...91 204.62 us 4.51 us 2.25 us spinlock cluster_move+0x2e ... snip ... 47 125.62 us 4.47 us 2.67 us spinlock cluster_move+0x2e ... snip ... 23 63.15 us 3.95 us 2.74 us spinlock cluster_move+0x2e ... snip ... 17 41.26 us 4.58 us 2.43 us spinlock cluster_isolate_lock+0x1d... snip ...
cluster_moveandcluster_isolate_lockare basically the only users ofsi->locknow, performance gain is huge with reduced LOC.Tests
===Build kernel with defconfig on tmpfs with ZRAM as swap:
---Running a test matrix which is scaled up progressive for a intuitive result. The test are ran on top of tmpfs, using memory cgroup for memory limitation, on a 48c96t system.
12 test run for each case, it can be seen clearly that as concurrent job number goes higher the performance gain is higher, the performance is higher even with low concurrency.
Quoted 12 lines
make -j<NR> | System Time (seconds) | Total Time (seconds) (NR / Mem / ZRAM) | (Before / After / Delta) | (Before / After / Delta) With 4k pages only: 6 / 192M / 3G | 5258 / 5235 / -0.3% | 1420 / 1414 / -0.3% 12 / 256M / 4G | 5518 / 5337 / -3.3% | 758 / 742 / -2.1% 24 / 384M / 5G | 7091 / 5766 / -18.7% | 476 / 422 / -11.3% 48 / 768M / 7G | 11139 / 5831 / -47.7% | 330 / 221 / -33.0% 96 / 1.5G / 10G | 21303 / 11353 / -46.7% | 283 / 180 / -36.4% With 64k mTHP: 24 / 512M / 5G | 5104 / 4641 / -18.7% | 376 / 358 / -4.8% 48 / 1G / 7G | 8693 / 4662 / -18.7% | 257 / 176 / -31.5% 96 / 2G / 10G | 17056 / 10263 / -39.8% | 234 / 169 / -27.8%
Quoted 1 line
How much is the swap in/out throughput before/after the change?
This may not be too beneficial for typical throughput measurement:
- For example doing the same test with brd will only show a ~20% performance improvement, still a big gain though. I think the si->lock spinlock wasting CPU cycles may effect CPU sensitive things like ZRAM even more.
- And simple benchmarks which just do multiple sequential swaps in/out in multiple thread hardly stress the allocator. I haven't found a good benchmark to simulate random parallel IOs on SWAP yet, I can write one later.
A more close to real word benchmark like build kernel test, or mysql/sysbench all showed great improment.
Quoted 11 lines
When I worked on swap in/out performance before, the hot spot shifts from swap related code to LRU lock and zone lock. Things may change a lot now.
If zram is used as swap device, the hot spot may become compression/decompression after solving the swap lock contention. To stress swap subsystem further, we may use a ram disk as swap. Previously, we have used a simulated pmem device (backed by DRAM). That can be setup as in,
Quoted 8 lines
https://pmem.io/blog/2016/02/how-to-emulate-persistent-memory/
After creating the raw block device:
/dev/pmem0, we can do$ mkswap /dev/pmem0 $ swapon /dev/pmem0Can you use something similar if necessary?
I used to test with brd, as described above, I think using ZRAM with test simulating real workload is more useful.
And I did include a Sequential SWAP test, the result is looking OK (no regression, minor to none improvement).
I can have a try with the pmem setup later, I guess the result will be similar to brd test.
Quoted 99 lines
Quoted 23 lines
With more aggressive setup, it shows clearly both the performance and fragmentation are better:
tiem make -j96 / 768M memcg, 4K pages, 10G ZRAM, on Intel 8255C * 2:
(avg of 4 test run)
Before:
Sys time: 73578.30, Real time: 864.05
tiem make -j96 / 1G memcg, 4K pages, 10G ZRAM:
After: (-54.7% sys time, -49.3% real time)
Sys time: 33314.76, Real time: 437.67time make -j96 / 1152M memcg, 64K mTHP, 10G ZRAM, on Intel 8255C * 2:
(avg of 4 test run)
Before:
Sys time: 74044.85, Real time: 846.51 hugepages-64kB/stats/swpout: 1735216 hugepages-64kB/stats/swpout_fallback: 430333
After: (-51.4% sys time, -47.7% real time, -63.2% mTHP failure) Sys time: 35958.87, Real time: 442.69 hugepages-64kB/stats/swpout: 1866267 hugepages-64kB/stats/swpout_fallback: 158330Quoted 3 lines
There is a up to 54.7% improvement for build kernel test, and lower fragmentation rate. Performance improvement should be even larger for micro benchmarks
Very good result!
Quoted 58 lines
Build kernel with tinyconfig on tmpfs with HDD as swap:
---This test is similar to above, but HDD test is very noisy and slow, the deviation is huge, so just use tinyconfig instead and take the median test result of 3 test run, which looks OK:
Before this series:
114.44user 29.11system 39:42.90elapsed 6%CPU 2901232inputs+0outputs (238877major+4227640minor)pagefaultsAfter this commit:
113.90user 23.81system 38:11.77elapsed 6%CPU 2548728inputs+0outputs (235471major+4238110minor)pagefaultsSingle thread SWAP:
---Sequential SWAP should also be slightly faster as we removed a lot of unnecessary parts. Test using micro benchmark for swapout/in 4G zero memory using ZRAM, 10 test runs:
Swapout Before (avg. 3359304):
3353796 3358551 3371305 3356043 3367524 3355303 3355924 3354513 3360776Swapin Before (avg. 1928698):
1920283 1927183 1934105 1921373 1926562 1938261 1927726 1928636 1934155Swapout After (avg. 3347511, -0.4%):
3337863 3347948 3355235 3339081 3333134 3353006 3354917 3346055 3360359Swapin After (avg. 1922290, -0.3%):
1919101 1925743 1916810 1917007 1923930 1935152 1917403 1923549 1921913Worth noticing the patch "mm, swap: use a global swap cluster for non-rotation device" introduced minor overhead for certain tests (see the test results in commit message), but the gain from later commit covered that, it can be further improved later.
Suggested-by Chris Li <[email protected]>Signed-off-by Kairui Song <[email protected]>Kairui Song (13):
mm, swap: minor clean up for swap entry allocation mm, swap: foldswap_info_get_contin the only caller mm, swap: remove old allocation path for HDD
mm, swap: use cluster lock for HDD
mm, swap: clean up device availability check
mm, swap: clean up plist removal and adding
mm, swap: hold a reference of si during scan and clean up flags mm, swap: use an enum to define all cluster flags and wrap flags
changes
mm, swap: reduce contention on device lock
mm, swap: simplify percpu cluster updating
mm, swap: introduce a helper for retrieving cluster from offset mm, swap: use a global swap cluster for non-rotation device mm,swap_slots: remove slot cache for freeing pathQuoted 8 lines
fs/btrfs/inode.c | 1 - fs/iomap/swapfile.c | 1 - include/linux/swap.h | 36 +- include/linux/swap_slots.h | 3 - mm/page_io.c | 1 - mm/swap_slots.c | 78 +-- mm/swapfile.c | 1198 ++++++++++++++++-------------------- 7 files changed, 558 insertions(+), 760 deletions(-)--
Best Regards,
Huang, Ying
Quoted 186 lines · Kairui Song
Quoted 2 lines · Ying
Hi, Kairui,
Hi Ying,
Quoted 24 lines
Quoted 12 lines · Kairui Song
From: Kairui Song <[email protected]>
This series improved the swap allocator performance greatly by reworking the locking design and simplify a lot of code path.
This is follow up of previous swap cluster allocator series:
https://lore.kernel.org/linux-mm/[email protected]/And this series is based on an follow up fix of the swap cluster allocator:
https://lore.kernel.org/linux-mm/[email protected]/Quoted 8 lines
This is part of the new swap allocator work item discussed in Chris's "Swap Abstraction" discussion at LSF/MM 2024, and "mTHP and swap allocator" discussion at LPC 2024.
Previous series introduced a fully cluster based allocation algorithm, this series completely get rid of the old allocation path and makes the allocator avoid grabbing the
si->lockunless needed. This bring huge performance gain and get rid of slot cache on freeing path.Quoted 8 lines
Great!
Quoted 3 lines
Currently, swap locking is mainly composed of two locks, cluster lock (
ci->lock) and device lock (si->lock). The device lock is widely used to protect many things, causing it to be the main bottleneck for SWAP.Device lock can be confusing with another device lock for struct device. Better to call it swap device lock?
Good idea, I'll use the term swap device lock then.
Quoted 18 lines
Quoted 18 lines
Cluster lock is much more fine-grained, so it will be best to use
ci->lockinstead ofsi->lockas much as possible.
perf lockindicates this issue clearly. Doing linux kernel build using tmpfs and ZRAM with limited memory (make -j64 with 1G memcg and 4k pages), result of "perf lock contention -ab sleep 3":contended total wait max wait avg wait type caller
34948 53.63 s 7.11 ms 1.53 ms spinlock free_swap_and_cache_nr+0x350 16569 40.05 s 6.45 ms 2.42 ms spinlock get_swap_pages+0x231 11191 28.41 s 7.03 ms 2.54 ms spinlock swapcache_free_entries+0x59 4147 22.78 s 122.66 ms 5.49 ms spinlock page_vma_mapped_walk+0x6f3 4595 7.17 s 6.79 ms 1.56 ms spinlock swapcache_free_entries+0x59 406027 2.74 s 2.59 ms 6.74 us spinlock list_lru_add+0x39...snip...
Quoted 5 lines
Quoted 2 lines
The top 5 caller are all users of
si->lock, total wait time up sums to several minutes in the 3 seconds time window.Can you show results of
perf record -g,perf report -gtoo? I have interest to check hot spot shifting too.Sure. I think
perf lockresult is already good enough and cleaner. My test environment are mostly VM based so spinlock slow path may get offloaded to host, and can't be see by perf record, I collected following data after disabled paravirt spinlock:The time consumption and stack trace of a page fault before:
- 78.45% 0.17% cc1 [kernel.kallsyms]
[k]asm_exc_page_fault
- 78.28%asm_exc_page_fault
- 78.18%exc_page_fault
- 78.17%do_user_addr_fault
- 78.09%handle_mm_fault
- 78.06%__handle_mm_fault
- 69.69%do_swap_page
- 55.87%alloc_swap_folio
- 55.60%mem_cgroup_swapin_charge_folio
- 55.48%charge_memcg
- 55.45%try_charge_memcg
- 55.36%try_to_free_mem_cgroup_pages
-do_try_to_free_pages
- 55.35%shrink_node
- 55.27%shrink_lruvec
- 55.13%try_to_shrink_lruvec
- 54.79%evict_folios
- 54.35%shrink_folio_list
- 30.01%add_to_swap
- 29.77%folio_alloc_swap
- 29.50%get_swap_pages25.03%
queued_spin_lock_slowpath
- 2.71%alloc_swap_scan_cluster1.80%
queued_spin_lock_slowpath+0.89%
__try_to_reclaim_swap
- 1.74%swap_reclaim_full_clusters1.74%
queued_spin_lock_slowpath
- 10.88%try_to_unmap_flush_dirty
- 10.87%arch_tlbbatch_flush
- 10.85%on_each_cpu_cond_mask
smp_call_function_many_cond
+ 7.45% pageout
+ 2.71%try_to_unmap_flush
+ 1.90%try_to_unmap
+ 0.78%folio_referenced
- 9.41%cluster_swap_free_nr
- 9.39%free_swap_slot
- 9.35%swapcache_free_entries
8.40%queued_spin_lock_slowpath0.93%swap_entry_range_free
- 3.61%swap_read_folio_bdev_sync
- 3.55%submit_bio_wait
- 3.51%submit_bio_noacct_nocheck
+ 3.46%__submit_bio
+ 7.71%do_pte_missing
+ 0.61%wp_page_copyThe
queued_spin_lock_slowpathabove is thesi->lock, and there are multiple users of it so the total overhead is higher than shown.After:
- 75.05% 0.43% cc1 [kernel.kallsyms]
[k]asm_exc_page_fault
- 74.62%asm_exc_page_fault
- 74.36%exc_page_fault
- 74.34%do_user_addr_fault
- 74.10%handle_mm_fault
- 73.96%__handle_mm_fault
- 67.55%do_swap_page
- 45.92%alloc_swap_folio
- 45.03%mem_cgroup_swapin_charge_folio
- 44.58%charge_memcg
- 44.44%try_charge_memcg
- 44.12%try_to_free_mem_cgroup_pages
-do_try_to_free_pages
- 44.10%shrink_node
- 43.86%shrink_lruvec
- 41.92%try_to_shrink_lruvec
- 40.67%evict_folios
- 37.12%shrink_folio_list
- 20.88% pageout
+ 20.02%swap_writepage
+ 0.72%shmem_writepage
- 4.08%add_to_swap
- 2.48%folio_alloc_swap
- 2.12%__mem_cgroup_try_charge_swap
- 1.47%swap_cgroup_record+1.32%
_raw_spin_lock_irqsave
- 1.56%add_to_swap_cache
- 1.04%xas_store
+ 1.01%workingset_update_node
+ 3.97%try_to_unmap_flush_dirty
+ 3.51%folio_referenced
+ 2.24%__remove_mapping
+ 1.16%try_to_unmap
+ 0.52%try_to_unmap_flush
2.50%queued_spin_lock_slowpath
0.79%scan_folios
+ 1.20%try_to_inc_max_seq
+ 1.92%lru_add_drain
+ 0.73%vma_alloc_folio_noprof
- 9.81%swap_read_folio_bdev_sync
- 9.61%submit_bio_wait
+ 9.49%submit_bio_noacct_nocheck
Quoted 8 lines
- 8.06%
cluster_swap_free_nr
- 8.02%swap_entry_range_free
+ 3.92%__mem_cgroup_uncharge_swap
+ 2.90%zram_slot_free_notify0.58%clear_shadow_from_swap_cache
- 1.32%__folio_batch_add_and_move
- 1.30%folio_batch_move_lru
+ 1.10%folio_lruvec_lock_irqsave
Thanks for data.
It seems that the cycles shifts from spinning to memory compression. That is expected.
Quoted 3 lines
spin_lockusage is much lower.I prefer the perf lock output as it shows the exact time and user of locks.
perf cycles data is more complete. You can find which part becomes new hot spot.
Quoted 21 lines
Quoted 14 lines
Quoted 14 lines
Following the new allocator design, many operation doesn't need to touch
si->lockat all. We only need to takesi->lockwhen doing operations across multiple clusters (eg. changing the cluster list), other operations only need to takeci->lock. So ideally allocator should always takeci->lockfirst, then, if needed, takesi->lock. But due to historical reasons,ci->lockis used insidesi->lockby design, causing lock inversion if we simply try to acquiresi->lockafter acquiringci->lock.This series audited all
si->lockusage, simplify legacy codes, eliminate usage ofsi->lockas much as possible by introducing new designs based on the new cluster allocator.Quoted 6 lines
Quoted 2 lines
Old HDD allocation codes are removed, cluster allocator is adapted with small changes for HDD usage, test is looking OK.
I think that it's a good idea to remove HDD allocation specific code. Can you check the performance of swapping to HDD? However, I understand that many people have no HDD in hand.
Quoted 6 lines
It's not hard to make cluster allocator work well with HDD in theory, see the commit "mm, swap: use a global swap cluster for non-rotation device".
The testing is not very reliable though, I found HDD swap performance is very unstable because of the IO pattern of HDD, so it's just a best effort try.
Just to check whether code change cause something too bad for HDD. No measurable difference is a good news.
Quoted 49 lines
Quoted 49 lines
Quoted 48 lines
And this also removed slot cache for freeing path. The performance is better without it, and this enables other clean up and optimizations as discussed before:
https://lore.kernel.org/all/CAMgjq7ACohT_uerSz8E_994ZZCv709Zor+43hdmesW_59W1BWw@mail.gmail.com/After this series, lock contention on
si->lockis nearly unobservable withperf lockwith the same test above :contended total wait max wait avg wait type caller
... snip ...91 204.62 us 4.51 us 2.25 us spinlock cluster_move+0x2e ... snip ... 47 125.62 us 4.47 us 2.67 us spinlock cluster_move+0x2e ... snip ... 23 63.15 us 3.95 us 2.74 us spinlock cluster_move+0x2e ... snip ... 17 41.26 us 4.58 us 2.43 us spinlock cluster_isolate_lock+0x1d... snip ...
cluster_moveandcluster_isolate_lockare basically the only users ofsi->locknow, performance gain is huge with reduced LOC.Tests
===Build kernel with defconfig on tmpfs with ZRAM as swap:
---Running a test matrix which is scaled up progressive for a intuitive result. The test are ran on top of tmpfs, using memory cgroup for memory limitation, on a 48c96t system.
12 test run for each case, it can be seen clearly that as concurrent job number goes higher the performance gain is higher, the performance is higher even with low concurrency.
make -j<NR> | System Time (seconds) | Total Time (seconds) (NR / Mem / ZRAM) | (Before / After / Delta) | (Before / After / Delta) With 4k pages only: 6 / 192M / 3G | 5258 / 5235 / -0.3% | 1420 / 1414 / -0.3% 12 / 256M / 4G | 5518 / 5337 / -3.3% | 758 / 742 / -2.1% 24 / 384M / 5G | 7091 / 5766 / -18.7% | 476 / 422 / -11.3% 48 / 768M / 7G | 11139 / 5831 / -47.7% | 330 / 221 / -33.0% 96 / 1.5G / 10G | 21303 / 11353 / -46.7% | 283 / 180 / -36.4% With 64k mTHP: 24 / 512M / 5G | 5104 / 4641 / -18.7% | 376 / 358 / -4.8% 48 / 1G / 7G | 8693 / 4662 / -18.7% | 257 / 176 / -31.5% 96 / 2G / 10G | 17056 / 10263 / -39.8% | 234 / 169 / -27.8%
Quoted 7 lines
Quoted 1 line
How much is the swap in/out throughput before/after the change?
This may not be too beneficial for typical throughput measurement:
- For example doing the same test with brd will only show a ~20% performance improvement, still a big gain though. I think thesi->lockspinlock wasting CPU cycles may effect CPU sensitive things like ZRAM even more.
20% is a good data. You don't need to guess. perf cycles profiling can show the hot spot.
Quoted 6 lines
- And simple benchmarks which just do multiple sequential swaps in/out in multiple thread hardly stress the allocator.
I haven't found a good
benchmark to simulate random parallel IOs on SWAP yet, I can write one later.
I have used anon-w-rand test case of vm-scalability to simulate random parallel swap out.
https://git.kernel.org/pub/scm/linux/kernel/git/wfg/vm-scalability.git/tree/case-anon-w-rand
Quoted 2 lines
A more close to real word benchmark like build kernel test, or mysql/sysbench all showed great improment.
Yes. Real work load is good. We can use micro-benchmark to find out some performance limit, for example, max possible throughput.
Quoted 13 lines
Quoted 13 lines
When I worked on swap in/out performance before, the hot spot shifts from swap related code to LRU lock and zone lock. Things may change a lot now.
If zram is used as swap device, the hot spot may become compression/decompression after solving the swap lock contention. To stress swap subsystem further, we may use a ram disk as swap. Previously, we have used a simulated pmem device (backed by DRAM). That can be setup as in,
https://pmem.io/blog/2016/02/how-to-emulate-persistent-memory/
Quoted 8 lines
Quoted 6 lines
After creating the raw block device:
/dev/pmem0, we can do$ mkswap /dev/pmem0 $ swapon /dev/pmem0Can you use something similar if necessary?
I used to test with brd, as described above,
brd will allocate memory during running, pmem can avoid that. perf profile is your friends to root cause the possible issue.
Quoted 2 lines
I think using ZRAM with
test simulating real workload is more useful.
Yes. And, as I said before. Micro-benchmark has its own value.
Quoted 2 lines
And I did include a Sequential SWAP test, the result is looking OK (no regression, minor to none improvement).
Good. At least we have no regression here.
Quoted 99 lines
I can have a try with the pmem setup later, I guess the result will be similar to brd test.
Quoted 95 lines
Quoted 23 lines
With more aggressive setup, it shows clearly both the performance and fragmentation are better:
tiem make -j96 / 768M memcg, 4K pages, 10G ZRAM, on Intel 8255C * 2:
(avg of 4 test run)
Before:
Sys time: 73578.30, Real time: 864.05
tiem make -j96 / 1G memcg, 4K pages, 10G ZRAM:
After: (-54.7% sys time, -49.3% real time)
Sys time: 33314.76, Real time: 437.67time make -j96 / 1152M memcg, 64K mTHP, 10G ZRAM, on Intel 8255C * 2:
(avg of 4 test run)
Before:
Sys time: 74044.85, Real time: 846.51 hugepages-64kB/stats/swpout: 1735216 hugepages-64kB/stats/swpout_fallback: 430333
After: (-51.4% sys time, -47.7% real time, -63.2% mTHP failure) Sys time: 35958.87, Real time: 442.69 hugepages-64kB/stats/swpout: 1866267 hugepages-64kB/stats/swpout_fallback: 158330Quoted 3 lines
There is a up to 54.7% improvement for build kernel test, and lower fragmentation rate. Performance improvement should be even larger for micro benchmarks
Very good result!
Quoted 66 lines
Build kernel with tinyconfig on tmpfs with HDD as swap:
---This test is similar to above, but HDD test is very noisy and slow, the deviation is huge, so just use tinyconfig instead and take the median test result of 3 test run, which looks OK:
Before this series:
114.44user 29.11system 39:42.90elapsed 6%CPU 2901232inputs+0outputs (238877major+4227640minor)pagefaultsAfter this commit:
113.90user 23.81system 38:11.77elapsed 6%CPU 2548728inputs+0outputs (235471major+4238110minor)pagefaultsSingle thread SWAP:
---Sequential SWAP should also be slightly faster as we removed a lot of unnecessary parts. Test using micro benchmark for swapout/in 4G zero memory using ZRAM, 10 test runs:
Swapout Before (avg. 3359304):
3353796 3358551 3371305 3356043 3367524 3355303 3355924 3354513 3360776Swapin Before (avg. 1928698):
1920283 1927183 1934105 1921373 1926562 1938261 1927726 1928636 1934155Swapout After (avg. 3347511, -0.4%):
3337863 3347948 3355235 3339081 3333134 3353006 3354917 3346055 3360359Swapin After (avg. 1922290, -0.3%):
1919101 1925743 1916810 1917007 1923930 1935152 1917403 1923549 1921913Worth noticing the patch "mm, swap: use a global swap cluster for non-rotation device" introduced minor overhead for certain tests (see the test results in commit message), but the gain from later commit covered that, it can be further improved later.
Suggested-by Chris Li <[email protected]>Signed-off-by Kairui Song <[email protected]>Kairui Song (13):
mm, swap: minor clean up for swap entry allocation mm, swap: foldswap_info_get_contin the only caller mm, swap: remove old allocation path for HDD
mm, swap: use cluster lock for HDD
mm, swap: clean up device availability check
mm, swap: clean up plist removal and adding
mm, swap: hold a reference of si during scan and clean up flags mm, swap: use an enum to define all cluster flags and wrap flags
changes
mm, swap: reduce contention on device lock
mm, swap: simplify percpu cluster updating
mm, swap: introduce a helper for retrieving cluster from offset mm, swap: use a global swap cluster for non-rotation device mm,swap_slots: remove slot cache for freeing pathfs/btrfs/inode.c | 1 - fs/iomap/swapfile.c | 1 - include/linux/swap.h | 36 +- include/linux/swap_slots.h | 3 - mm/page_io.c | 1 - mm/swap_slots.c | 78 +-- mm/swapfile.c | 1198 ++++++++++++++++-------------------- 7 files changed, 558 insertions(+), 760 deletions(-)
--
Best Regards,
Huang, Ying
Quoted 341 lines · Ying
Quoted 186 lines · Kairui Song
Quoted 2 lines · Ying
Hi, Kairui,
Hi Ying,
Quoted 24 lines
Quoted 12 lines · Kairui Song
From: Kairui Song <[email protected]>
This series improved the swap allocator performance greatly by reworking the locking design and simplify a lot of code path.
This is follow up of previous swap cluster allocator series:
https://lore.kernel.org/linux-mm/[email protected]/And this series is based on an follow up fix of the swap cluster allocator:
https://lore.kernel.org/linux-mm/[email protected]/Quoted 8 lines
This is part of the new swap allocator work item discussed in Chris's "Swap Abstraction" discussion at LSF/MM 2024, and "mTHP and swap allocator" discussion at LPC 2024.
Previous series introduced a fully cluster based allocation algorithm, this series completely get rid of the old allocation path and makes the allocator avoid grabbing the
si->lockunless needed. This bring huge performance gain and get rid of slot cache on freeing path.Quoted 8 lines
Great!
Quoted 3 lines
Currently, swap locking is mainly composed of two locks, cluster lock (
ci->lock) and device lock (si->lock). The device lock is widely used to protect many things, causing it to be the main bottleneck for SWAP.Device lock can be confusing with another device lock for struct device. Better to call it swap device lock?
Good idea, I'll use the term swap device lock then.
Quoted 18 lines
Quoted 18 lines
Cluster lock is much more fine-grained, so it will be best to use
ci->lockinstead ofsi->lockas much as possible.
perf lockindicates this issue clearly. Doing linux kernel build using tmpfs and ZRAM with limited memory (make -j64 with 1G memcg and 4k pages), result of "perf lock contention -ab sleep 3":contended total wait max wait avg wait type caller
34948 53.63 s 7.11 ms 1.53 ms spinlock free_swap_and_cache_nr+0x350 16569 40.05 s 6.45 ms 2.42 ms spinlock get_swap_pages+0x231 11191 28.41 s 7.03 ms 2.54 ms spinlock swapcache_free_entries+0x59 4147 22.78 s 122.66 ms 5.49 ms spinlock page_vma_mapped_walk+0x6f3 4595 7.17 s 6.79 ms 1.56 ms spinlock swapcache_free_entries+0x59 406027 2.74 s 2.59 ms 6.74 us spinlock list_lru_add+0x39...snip...
Quoted 5 lines
Quoted 2 lines
The top 5 caller are all users of
si->lock, total wait time up sums to several minutes in the 3 seconds time window.Can you show results of
perf record -g,perf report -gtoo? I have interest to check hot spot shifting too.Sure. I think
perf lockresult is already good enough and cleaner. My test environment are mostly VM based so spinlock slow path may get offloaded to host, and can't be see by perf record, I collected following data after disabled paravirt spinlock:The time consumption and stack trace of a page fault before:
- 78.45% 0.17% cc1 [kernel.kallsyms]
[k]asm_exc_page_fault
- 78.28%asm_exc_page_fault
- 78.18%exc_page_fault
- 78.17%do_user_addr_fault
- 78.09%handle_mm_fault
- 78.06%__handle_mm_fault
- 69.69%do_swap_page
- 55.87%alloc_swap_folio
- 55.60%mem_cgroup_swapin_charge_folio
- 55.48%charge_memcg
- 55.45%try_charge_memcg
- 55.36%try_to_free_mem_cgroup_pages
-do_try_to_free_pages
- 55.35%shrink_node
- 55.27%shrink_lruvec
- 55.13%try_to_shrink_lruvec
- 54.79%evict_folios
- 54.35%shrink_folio_list
- 30.01%add_to_swap
- 29.77%folio_alloc_swap
- 29.50%get_swap_pages25.03%
queued_spin_lock_slowpath
- 2.71%alloc_swap_scan_cluster1.80%
queued_spin_lock_slowpath+0.89%
__try_to_reclaim_swap
- 1.74%swap_reclaim_full_clusters1.74%
queued_spin_lock_slowpath
- 10.88%try_to_unmap_flush_dirty
- 10.87%arch_tlbbatch_flush
- 10.85%on_each_cpu_cond_mask
smp_call_function_many_cond
+ 7.45% pageout
+ 2.71%try_to_unmap_flush
+ 1.90%try_to_unmap
+ 0.78%folio_referenced
- 9.41%cluster_swap_free_nr
- 9.39%free_swap_slot
- 9.35%swapcache_free_entries
8.40%queued_spin_lock_slowpath0.93%swap_entry_range_free
- 3.61%swap_read_folio_bdev_sync
- 3.55%submit_bio_wait
- 3.51%submit_bio_noacct_nocheck
+ 3.46%__submit_bio
+ 7.71%do_pte_missing
+ 0.61%wp_page_copyThe
queued_spin_lock_slowpathabove is thesi->lock, and there are multiple users of it so the total overhead is higher than shown.After:
- 75.05% 0.43% cc1 [kernel.kallsyms]
[k]asm_exc_page_fault
- 74.62%asm_exc_page_fault
- 74.36%exc_page_fault
- 74.34%do_user_addr_fault
- 74.10%handle_mm_fault
- 73.96%__handle_mm_fault
- 67.55%do_swap_page
- 45.92%alloc_swap_folio
- 45.03%mem_cgroup_swapin_charge_folio
- 44.58%charge_memcg
- 44.44%try_charge_memcg
- 44.12%try_to_free_mem_cgroup_pages
-do_try_to_free_pages
- 44.10%shrink_node
- 43.86%shrink_lruvec
- 41.92%try_to_shrink_lruvec
- 40.67%evict_folios
- 37.12%shrink_folio_list
- 20.88% pageout
+ 20.02%swap_writepage
+ 0.72%shmem_writepage
- 4.08%add_to_swap
- 2.48%folio_alloc_swap
- 2.12%__mem_cgroup_try_charge_swap
- 1.47%swap_cgroup_record+1.32%
_raw_spin_lock_irqsave
- 1.56%add_to_swap_cache
- 1.04%xas_store
+ 1.01%workingset_update_node
+ 3.97%try_to_unmap_flush_dirty
+ 3.51%folio_referenced
+ 2.24%__remove_mapping
+ 1.16%try_to_unmap
+ 0.52%try_to_unmap_flush
2.50%queued_spin_lock_slowpath
0.79%scan_folios
+ 1.20%try_to_inc_max_seq
+ 1.92%lru_add_drain
+ 0.73%vma_alloc_folio_noprof
- 9.81%swap_read_folio_bdev_sync
- 9.61%submit_bio_wait
+ 9.49%submit_bio_noacct_nocheckQuoted 8 lines
- 8.06%
cluster_swap_free_nr
- 8.02%swap_entry_range_free
+ 3.92%__mem_cgroup_uncharge_swap
+ 2.90%zram_slot_free_notify0.58%clear_shadow_from_swap_cache
- 1.32%__folio_batch_add_and_move
- 1.30%folio_batch_move_lru
+ 1.10%folio_lruvec_lock_irqsaveThanks for data.
It seems that the cycles shifts from spinning to memory compression. That is expected.
Quoted 3 lines
spin_lockusage is much lower.I prefer the perf lock output as it shows the exact time and user of locks.
perf cycles data is more complete. You can find which part becomes new hot spot.
Quoted 21 lines
Quoted 14 lines
Quoted 14 lines
Following the new allocator design, many operation doesn't need to touch
si->lockat all. We only need to takesi->lockwhen doing operations across multiple clusters (eg. changing the cluster list), other operations only need to takeci->lock. So ideally allocator should always takeci->lockfirst, then, if needed, takesi->lock. But due to historical reasons,ci->lockis used insidesi->lockby design, causing lock inversion if we simply try to acquiresi->lockafter acquiringci->lock.This series audited all
si->lockusage, simplify legacy codes, eliminate usage ofsi->lockas much as possible by introducing new designs based on the new cluster allocator.Quoted 6 lines
Quoted 2 lines
Old HDD allocation codes are removed, cluster allocator is adapted with small changes for HDD usage, test is looking OK.
I think that it's a good idea to remove HDD allocation specific code. Can you check the performance of swapping to HDD? However, I understand that many people have no HDD in hand.
Quoted 6 lines
It's not hard to make cluster allocator work well with HDD in theory, see the commit "mm, swap: use a global swap cluster for non-rotation device".
The testing is not very reliable though, I found HDD swap performance is very unstable because of the IO pattern of HDD, so it's just a best effort try.Just to check whether code change cause something too bad for HDD. No measurable difference is a good news.
Quoted 49 lines
Quoted 49 lines
Quoted 48 lines
And this also removed slot cache for freeing path. The performance is better without it, and this enables other clean up and optimizations as discussed before:
https://lore.kernel.org/all/CAMgjq7ACohT_uerSz8E_994ZZCv709Zor+43hdmesW_59W1BWw@mail.gmail.com/After this series, lock contention on
si->lockis nearly unobservable withperf lockwith the same test above :contended total wait max wait avg wait type caller
... snip ...91 204.62 us 4.51 us 2.25 us spinlock cluster_move+0x2e ... snip ... 47 125.62 us 4.47 us 2.67 us spinlock cluster_move+0x2e ... snip ... 23 63.15 us 3.95 us 2.74 us spinlock cluster_move+0x2e ... snip ... 17 41.26 us 4.58 us 2.43 us spinlock cluster_isolate_lock+0x1d... snip ...
cluster_moveandcluster_isolate_lockare basically the only users ofsi->locknow, performance gain is huge with reduced LOC.Tests
===Build kernel with defconfig on tmpfs with ZRAM as swap:
---Running a test matrix which is scaled up progressive for a intuitive result. The test are ran on top of tmpfs, using memory cgroup for memory limitation, on a 48c96t system.
12 test run for each case, it can be seen clearly that as concurrent job number goes higher the performance gain is higher, the performance is higher even with low concurrency.
make -j<NR> | System Time (seconds) | Total Time (seconds) (NR / Mem / ZRAM) | (Before / After / Delta) | (Before / After / Delta) With 4k pages only: 6 / 192M / 3G | 5258 / 5235 / -0.3% | 1420 / 1414 / -0.3% 12 / 256M / 4G | 5518 / 5337 / -3.3% | 758 / 742 / -2.1% 24 / 384M / 5G | 7091 / 5766 / -18.7% | 476 / 422 / -11.3% 48 / 768M / 7G | 11139 / 5831 / -47.7% | 330 / 221 / -33.0% 96 / 1.5G / 10G | 21303 / 11353 / -46.7% | 283 / 180 / -36.4% With 64k mTHP: 24 / 512M / 5G | 5104 / 4641 / -18.7% | 376 / 358 / -4.8% 48 / 1G / 7G | 8693 / 4662 / -18.7% | 257 / 176 / -31.5% 96 / 2G / 10G | 17056 / 10263 / -39.8% | 234 / 169 / -27.8%Quoted 7 lines
Quoted 1 line
How much is the swap in/out throughput before/after the change?
This may not be too beneficial for typical throughput measurement:
- For example doing the same test with brd will only show a ~20% performance improvement, still a big gain though. I think thesi->lockspinlock wasting CPU cycles may effect CPU sensitive things like ZRAM even more.20% is a good data. You don't need to guess. perf cycles profiling can show the hot spot.
Quoted 6 lines
- And simple benchmarks which just do multiple sequential swaps in/out in multiple thread hardly stress the allocator.
I haven't found a good
benchmark to simulate random parallel IOs on SWAP yet, I can write one later.I have used anon-w-rand test case of vm-scalability to simulate random parallel swap out.
https://git.kernel.org/pub/scm/linux/kernel/git/wfg/vm-scalability.git/tree/case-anon-w-rand
Quoted 2 lines
A more close to real word benchmark like build kernel test, or mysql/sysbench all showed great improment.
Yes. Real work load is good. We can use micro-benchmark to find out some performance limit, for example, max possible throughput.
Quoted 13 lines
Quoted 13 lines
When I worked on swap in/out performance before, the hot spot shifts from swap related code to LRU lock and zone lock. Things may change a lot now.
If zram is used as swap device, the hot spot may become compression/decompression after solving the swap lock contention. To stress swap subsystem further, we may use a ram disk as swap. Previously, we have used a simulated pmem device (backed by DRAM). That can be setup as in,
https://pmem.io/blog/2016/02/how-to-emulate-persistent-memory/
Quoted 8 lines
Quoted 6 lines
After creating the raw block device:
/dev/pmem0, we can do$ mkswap /dev/pmem0 $ swapon /dev/pmem0Can you use something similar if necessary?
I used to test with brd, as described above,
Quoted 7 lines
brd will allocate memory during running, pmem can avoid that. perf profile is your friends to root cause the possible issue.
Quoted 2 lines
I think using ZRAM with
test simulating real workload is more useful.Yes. And, as I said before. Micro-benchmark has its own value.
Hi Ying,
Thank you very much for the suggestion, I didn't mean I'm against micro benchmarks in any way, just a lot of effort was spent on other tests so I skipped that part for V1.
As you mentioned vm-scalability, I think this is definitely a good idea to include that test when pmem simulation.
There are still some bottlenecks of SWAP, beside compression and page fault / tlb, mostly cgroup lock and list lru locks. I have some ideas to optimize these too, could be next steps.
Quoted 8 lines
Quoted 2 lines
And I did include a Sequential SWAP test, the result is looking OK (no regression, minor to none improvement).
Good. At least we have no regression here.
--
Best Regards,
Huang, Ying
Andrew Morton
maintainerQuoted 5 lines · On Wed, 23 Oct 2024 03:24:38 +0800 Kairui Song <[email protected]> wrote:
After this series, lock contention on
si->lockis nearly unobservable withperf lockwith the same test above :contended total wait max wait avg wait type caller
... snip ...
Quoted 8 lines
91 204.62 us 4.51 us 2.25 us spinlock cluster_move+0x2e ... snip ... 47 125.62 us 4.47 us 2.67 us spinlock cluster_move+0x2e ... snip ... 23 63.15 us 3.95 us 2.74 us spinlock cluster_move+0x2e ... snip ... 17 41.26 us 4.58 us 2.43 us spinlock cluster_isolate_lock+0x1d... snip ...
Were any overall runtime benefits observed?
Quoted 17 lines · Andrew Morton
Quoted 5 lines · On Wed, 23 Oct 2024 03:24:38 +0800 Kairui Song <[email protected]> wrote:
After this series, lock contention on
si->lockis nearly unobservable withperf lockwith the same test above :contended total wait max wait avg wait type caller
... snip ...Quoted 8 lines
91 204.62 us 4.51 us 2.25 us spinlock cluster_move+0x2e ... snip ... 47 125.62 us 4.47 us 2.67 us spinlock cluster_move+0x2e ... snip ... 23 63.15 us 3.95 us 2.74 us spinlock cluster_move+0x2e ... snip ... 17 41.26 us 4.58 us 2.43 us spinlock cluster_isolate_lock+0x1d... snip ...
Quoted 1 line
Were any overall runtime benefits observed?
Yes, see the "Tests" results in the cover letter (summary: up to 50% build time saved for build linux kernel test when under pressure, with either mTHP or 4K pages):
time make -j96 / 768M memcg, 4K pages, 10G ZRAM, on Intel 8255C * 2 in VM:
(avg of 4 test run)
Before:
Sys time: 73578.30, Real time: 864.05
After: (-54.7% sys time, -49.3% real time)
Sys time: 33314.76, Real time: 437.67
time make -j96 / 1152M memcg, 64K mTHP, 10G ZRAM, on Intel 8255C * 2 in VM:
(avg of 4 test run)
Before:
Sys time: 74044.85, Real time: 846.51
After: (-51.4% sys time, -47.7% real time, -63.2% mTHP failure) Sys time: 35958.87, Real time: 442.69
Tests on the host bare metal showed similar results.
There are some other test results I didn't include in the cover letter for V1 yet and I'm still testing more scenarios, eg. mysql test in 1G memcg and with 96 workers and ZRAM swap:
before:
transactions: 755630 (6292.11 per sec.)
queries: 12090080 (100673.69 per sec.)
after:
transactions: 1077156 (8972.73 per sec.)
queries: 17234496 (143563.65 per sec.)~30% faster.
Also the mTHP swap allocation success rate is higher, I can highlight these changes in V2.
Quoted 162 lines · Kairui Song
From: Kairui Song <[email protected]>
This series improved the swap allocator performance greatly by reworking the locking design and simplify a lot of code path.
This is follow up of previous swap cluster allocator series:
https://lore.kernel.org/linux-mm/[email protected]/And this series is based on an follow up fix of the swap cluster allocator:
https://lore.kernel.org/linux-mm/[email protected]/This is part of the new swap allocator work item discussed in Chris's "Swap Abstraction" discussion at LSF/MM 2024, and "mTHP and swap allocator" discussion at LPC 2024.
Previous series introduced a fully cluster based allocation algorithm, this series completely get rid of the old allocation path and makes the allocator avoid grabbing the
si->lockunless needed. This bring huge performance gain and get rid of slot cache on freeing path.Currently, swap locking is mainly composed of two locks, cluster lock (
ci->lock) and device lock (si->lock). The device lock is widely used to protect many things, causing it to be the main bottleneck for SWAP.Cluster lock is much more fine-grained, so it will be best to use
ci->lockinstead ofsi->lockas much as possible.
perf lockindicates this issue clearly. Doing linux kernel build using tmpfs and ZRAM with limited memory (make -j64 with 1G memcg and 4k pages), result of "perf lock contention -ab sleep 3":contended total wait max wait avg wait type caller
34948 53.63 s 7.11 ms 1.53 ms spinlock free_swap_and_cache_nr+0x350 16569 40.05 s 6.45 ms 2.42 ms spinlock get_swap_pages+0x231 11191 28.41 s 7.03 ms 2.54 ms spinlock swapcache_free_entries+0x59 4147 22.78 s 122.66 ms 5.49 ms spinlock page_vma_mapped_walk+0x6f3 4595 7.17 s 6.79 ms 1.56 ms spinlock swapcache_free_entries+0x59 406027 2.74 s 2.59 ms 6.74 us spinlock list_lru_add+0x39...snip...
The top 5 caller are all users of
si->lock, total wait time up sums to several minutes in the 3 seconds time window.Following the new allocator design, many operation doesn't need to touch
si->lockat all. We only need to takesi->lockwhen doing operations across multiple clusters (eg. changing the cluster list), other operations only need to takeci->lock. So ideally allocator should always takeci->lockfirst, then, if needed, takesi->lock. But due to historical reasons,ci->lockis used insidesi->lockby design, causing lock inversion if we simply try to acquiresi->lockafter acquiringci->lock.This series audited all
si->lockusage, simplify legacy codes, eliminate usage ofsi->lockas much as possible by introducing new designs based on the new cluster allocator.Old HDD allocation codes are removed, cluster allocator is adapted with small changes for HDD usage, test is looking OK.
And this also removed slot cache for freeing path. The performance is better without it, and this enables other clean up and optimizations as discussed before:
https://lore.kernel.org/all/CAMgjq7ACohT_uerSz8E_994ZZCv709Zor+43hdmesW_59W1BWw@mail.gmail.com/After this series, lock contention on
si->lockis nearly unobservable withperf lockwith the same test above :contended total wait max wait avg wait type caller
... snip ...91 204.62 us 4.51 us 2.25 us spinlock cluster_move+0x2e ... snip ... 47 125.62 us 4.47 us 2.67 us spinlock cluster_move+0x2e ... snip ... 23 63.15 us 3.95 us 2.74 us spinlock cluster_move+0x2e ... snip ... 17 41.26 us 4.58 us 2.43 us spinlock cluster_isolate_lock+0x1d... snip ...
cluster_moveandcluster_isolate_lockare basically the only users ofsi->locknow, performance gain is huge with reduced LOC.Tests
===Build kernel with defconfig on tmpfs with ZRAM as swap:
---Running a test matrix which is scaled up progressive for a intuitive result. The test are ran on top of tmpfs, using memory cgroup for memory limitation, on a 48c96t system.
12 test run for each case, it can be seen clearly that as concurrent job number goes higher the performance gain is higher, the performance is higher even with low concurrency.
make -j<NR> | System Time (seconds) | Total Time (seconds) (NR / Mem / ZRAM) | (Before / After / Delta) | (Before / After / Delta) With 4k pages only: 6 / 192M / 3G | 5258 / 5235 / -0.3% | 1420 / 1414 / -0.3% 12 / 256M / 4G | 5518 / 5337 / -3.3% | 758 / 742 / -2.1% 24 / 384M / 5G | 7091 / 5766 / -18.7% | 476 / 422 / -11.3% 48 / 768M / 7G | 11139 / 5831 / -47.7% | 330 / 221 / -33.0% 96 / 1.5G / 10G | 21303 / 11353 / -46.7% | 283 / 180 / -36.4% With 64k mTHP: 24 / 512M / 5G | 5104 / 4641 / -18.7% | 376 / 358 / -4.8% 48 / 1G / 7G | 8693 / 4662 / -18.7% | 257 / 176 / -31.5% 96 / 2G / 10G | 17056 / 10263 / -39.8% | 234 / 169 / -27.8%With more aggressive setup, it shows clearly both the performance and fragmentation are better:
tiem make -j96 / 768M memcg, 4K pages, 10G ZRAM, on Intel 8255C * 2:
(avg of 4 test run)
Before:
Sys time: 73578.30, Real time: 864.05
tiem make -j96 / 1G memcg, 4K pages, 10G ZRAM:
After: (-54.7% sys time, -49.3% real time)
Sys time: 33314.76, Real time: 437.67time make -j96 / 1152M memcg, 64K mTHP, 10G ZRAM, on Intel 8255C * 2:
(avg of 4 test run)
Before:
Sys time: 74044.85, Real time: 846.51 hugepages-64kB/stats/swpout: 1735216 hugepages-64kB/stats/swpout_fallback: 430333
After: (-51.4% sys time, -47.7% real time, -63.2% mTHP failure) Sys time: 35958.87, Real time: 442.69 hugepages-64kB/stats/swpout: 1866267 hugepages-64kB/stats/swpout_fallback: 158330There is a up to 54.7% improvement for build kernel test, and lower fragmentation rate. Performance improvement should be even larger for micro benchmarks
Build kernel with tinyconfig on tmpfs with HDD as swap:
---This test is similar to above, but HDD test is very noisy and slow, the deviation is huge, so just use tinyconfig instead and take the median test result of 3 test run, which looks OK:
Before this series:
114.44user 29.11system 39:42.90elapsed 6%CPU 2901232inputs+0outputs (238877major+4227640minor)pagefaultsAfter this commit:
113.90user 23.81system 38:11.77elapsed 6%CPU 2548728inputs+0outputs (235471major+4238110minor)pagefaultsSingle thread SWAP:
---Sequential SWAP should also be slightly faster as we removed a lot of unnecessary parts. Test using micro benchmark for swapout/in 4G zero memory using ZRAM, 10 test runs:
Swapout Before (avg. 3359304):
3353796 3358551 3371305 3356043 3367524 3355303 3355924 3354513 3360776
Quoted 8 lines
Swapin Before (avg. 1928698):
1920283 1927183 1934105 1921373 1926562 1938261 1927726 1928636 1934155Swapout After (avg. 3347511, -0.4%):
3337863 3347948 3355235 3339081 3333134 3353006 3354917 3346055 3360359Swapin After (avg. 1922290, -0.3%):
1919101 1925743 1916810 1917007 1923930 1935152 1917403 1923549 1921913
Unfortunately I don't have the time to go through this series, but I just wanted to say that this awesome work, Kairui.
Selfishly, I especially like cleaning up the swap slot freeing path, and having a centralized freeing path with a single call to zswap_invalidate().
Thanks for doing this :)