Skip to contentSkip to conversation

improving dynamic zswap shrinker protection scheme

PATCH Applied Nhat Pham 3 participants 12 messages Aug 5, 2024 – Aug 7, 2024

Subsystems (3)
lore
Share & tools
lore

Nhat Pham

author maintainer
#0 lore raw

v3 cover improving dynamic zswap shrinker protection scheme

v3: No (intended) functional change
* Small cleanups, renamings, etc. (suggested by Yosry Ahmed)
v2:
* Add more details in comments, patch changelog, documentation, etc. about the second chance scheme and its ability to modulate the writeback rate (patch 1) (suggested by Yosry Ahmed).
* Move the referenced bit (patch 1) (suggested by Yosry Ahmed).

When experimenting with the memory-pressure based (i.e "dynamic") zswap shrinker in production, we observed a sharp increase in the number of swapins, which led to performance regression. We were able to trace this regression to the following problems with the shrinker's warm pages protection scheme:

1. The protection decays way too rapidly, and the decaying is coupled with zswap stores, leading to anomalous patterns, in which a small batch of zswap stores effectively erase all the protection in place for the warmer pages in the zswap LRU.

This observation has also been corroborated upstream by Takero Funaki (in [1]).

2. We inaccurately track the number of swapped in pages, missing the non-pivot pages that are part of the readahead window, while counting the pages that are found in the zswap pool.

To alleviate these two issues, this patch series improve the dynamic zswap shrinker in the following manner:

1. Replace the protection size tracking scheme with a second chance algorithm. This new scheme removes the need for haphazard stats decaying, and automatically adjusts the pace of pages aging with memory pressure, and writeback rate with pool activities: slowing down when the pool is dominated with zswpouts, and speeding up when the pool is dominated with stale entries.

2. Fix the tracking of the number of swapins to take into account non-pivot pages in the readahead window.

With these two changes in place, in a kernel-building benchmark without any cold data added, the number of swapins is reduced by 64.12%. This translate to a 10.32% reduction in build time. We also observe a 3% reduction in kernel CPU time.

In another benchmark, with cold data added (to gauge the new algorithm's ability to offload cold data), the new second chance scheme outperforms the old protection scheme by around 0.7%, and actually written back around 21% more pages to backing swap device. So the new scheme is just as good, if not even better than the old scheme on this front as well.

[1]: https://lore.kernel.org/linux-mm/CAPpodddcGsK=0Xczfuk8usgZ47xeyf4ZjiofdT+ujiyz6V2pFQ@mail.gmail.com/

Nhat Pham (2):
zswap: implement a second chance algorithm for dynamic zswap shrinker
zswap: track swapins from disk more accurately

 include/linux/zswap.h |  16 +++----
 mm/page_io.c          |  11 ++++-
 mm/swap_state.c       |   8 +---
 mm/zswap.c            | 108 ++++++++++++++++++++++++------------------
 4 files changed, 82 insertions(+), 61 deletions(-)

base-commit: cca1345bd26a67fc61a92ff0c6d81766c259e522

Signature · 1 line
2.43.0
author maintainer
#1 lore raw

PATCH v3 1/2 zswap: implement a second chance algorithm for dynamic zswap shrinker

Current zswap shrinker's heuristics to prevent overshrinking is brittle and inaccurate, specifically in the way we decay the protection size (i.e making pages in the zswap LRU eligible for reclaim).

We currently decay protection aggressively in zswap_lru_add() calls. This leads to the following unfortunate effect: when a new batch of pages enter zswap, the protection size rapidly decays to below 25% of the zswap LRU size, which is way too low.

We have observed this effect in production, when experimenting with the zswap shrinker: the rate of shrinking shoots up massively right after a new batch of zswap stores. This is somewhat the opposite of what we want originally - when new pages enter zswap, we want to protect both these new pages AND the pages that are already protected in the zswap LRU.

Replace existing heuristics with a second chance algorithm

1. When a new zswap entry is stored in the zswap pool, its referenced bit is set.
2. When the zswap shrinker encounters a zswap entry with the referenced bit set, give it a second chance - only flips the referenced bit and rotate it in the LRU.
3. If the shrinker encounters the entry again, this time with its referenced bit unset, then it can reclaim the entry.

In this manner, the aging of the pages in the zswap LRUs are decoupled from zswap stores, and picks up the pace with increasing memory pressure (which is what we want).

The second chance scheme allows us to modulate the writeback rate based on recent pool activities. Entries that recently entered the pool will be protected, so if the pool is dominated by such entries the writeback rate will reduce proportionally, protecting the workload's workingset.On the other hand, stale entries will be written back quickly, which increases the effective writeback rate.

The referenced bit is added at the hole after the length field of struct zswap_entry, so there is no extra space overhead for this algorithm.

We will still maintain the count of swapins, which is consumed and subtracted from the lru size in zswap_shrinker_count(), to further penalize past overshrinking that led to disk swapins. The idea is that had we considered this many more pages in the LRU active/protected, they would not have been written back and we would not have had to swapped them in.

To test this new heuristics, I built the kernel under a cgroup with memory.max set to 2G, on a host with 36 cores:

With the old shrinker:

real: 263.89s
user: 4318.11s
sys: 673.29s
swapins: 227300.5

With the second chance algorithm:

real: 244.85s
user: 4327.22s
sys: 664.39s
swapins: 94663

(average over 5 runs)

We observe an 1.3% reduction in kernel CPU usage, and around 7.2% reduction in real time. Note that the number of swapped in pages dropped by 58%.

---
 include/linux/zswap.h |  16 +++----
 mm/zswap.c            | 108 ++++++++++++++++++++++++------------------
 2 files changed, 70 insertions(+), 54 deletions(-)
include/linux/zswap.h +7 -9
index 6cecb4a4f68b..9cd1beef0654 100644
@@ -13,17 +13,15 @@ extern atomic_t zswap_stored_pages;
 
 struct zswap_lruvec_state {
 	/*
-	 * Number of pages in zswap that should be protected from the shrinker.
-	 * This number is an estimate of the following counts:
+	 * Number of swapped in pages from disk, i.e not found in the zswap pool.
 	 *
-	 * a) Recent page faults.
-	 * b) Recent insertion to the zswap LRU. This includes new zswap stores,
-	 *    as well as recent zswap LRU rotations.
-	 *
-	 * These pages are likely to be warm, and might incur IO if the are written
-	 * to swap.
+	 * This is consumed and subtracted from the lru size in
+	 * zswap_shrinker_count() to penalize past overshrinking that led to disk
+	 * swapins. The idea is that had we considered this many more pages in the
+	 * LRU active/protected and not written them back, we would not have had to
+	 * swapped them in.
 	 */
-	atomic_long_t nr_zswap_protected;
+	atomic_long_t nr_disk_swapins;
 };
 
 unsigned long zswap_total_pages(void);
mm/zswap.c +63 -45

191 lines not on this page · raw message

Signature · 1 line
2.43.0
Acked-by on v3 1/2
#2 lore raw

Re: [PATCH v3 1/2] zswap: implement a second chance algorithm for dynamic zswap shrinker

Quoted 112 lines · Nhat Pham

Current zswap shrinker's heuristics to prevent overshrinking is brittle and inaccurate, specifically in the way we decay the protection size (i.e making pages in the zswap LRU eligible for reclaim).

We currently decay protection aggressively in zswap_lru_add() calls. This leads to the following unfortunate effect: when a new batch of pages enter zswap, the protection size rapidly decays to below 25% of the zswap LRU size, which is way too low.

We have observed this effect in production, when experimenting with the zswap shrinker: the rate of shrinking shoots up massively right after a new batch of zswap stores. This is somewhat the opposite of what we want originally - when new pages enter zswap, we want to protect both these new pages AND the pages that are already protected in the zswap LRU.

Replace existing heuristics with a second chance algorithm

1. When a new zswap entry is stored in the zswap pool, its referenced bit is set.
2. When the zswap shrinker encounters a zswap entry with the referenced bit set, give it a second chance - only flips the referenced bit and rotate it in the LRU.
3. If the shrinker encounters the entry again, this time with its referenced bit unset, then it can reclaim the entry.

In this manner, the aging of the pages in the zswap LRUs are decoupled from zswap stores, and picks up the pace with increasing memory pressure (which is what we want).

The second chance scheme allows us to modulate the writeback rate based on recent pool activities. Entries that recently entered the pool will be protected, so if the pool is dominated by such entries the writeback rate will reduce proportionally, protecting the workload's workingset.On the other hand, stale entries will be written back quickly, which increases the effective writeback rate.

The referenced bit is added at the hole after the length field of struct zswap_entry, so there is no extra space overhead for this algorithm.

We will still maintain the count of swapins, which is consumed and subtracted from the lru size in zswap_shrinker_count(), to further penalize past overshrinking that led to disk swapins. The idea is that had we considered this many more pages in the LRU active/protected, they would not have been written back and we would not have had to swapped them in.

To test this new heuristics, I built the kernel under a cgroup with memory.max set to 2G, on a host with 36 cores:

With the old shrinker:

real: 263.89s
user: 4318.11s
sys: 673.29s
swapins: 227300.5

With the second chance algorithm:

real: 244.85s
user: 4327.22s
sys: 664.39s
swapins: 94663

(average over 5 runs)

We observe an 1.3% reduction in kernel CPU usage, and around 7.2% reduction in real time. Note that the number of swapped in pages dropped by 58%.

---
 include/linux/zswap.h |  16 +++----
 mm/zswap.c            | 108 ++++++++++++++++++++++++------------------
 2 files changed, 70 insertions(+), 54 deletions(-)
include/linux/zswap.h +7 -9
index 6cecb4a4f68b..9cd1beef0654 100644
@@ -13,17 +13,15 @@ extern atomic_t zswap_stored_pages;
 
 struct zswap_lruvec_state {
        /*
-        * Number of pages in zswap that should be protected from the shrinker.
-        * This number is an estimate of the following counts:
+        * Number of swapped in pages from disk, i.e not found in the zswap pool.
         *
-        * a) Recent page faults.
-        * b) Recent insertion to the zswap LRU. This includes new zswap stores,
-        *    as well as recent zswap LRU rotations.
-        *
-        * These pages are likely to be warm, and might incur IO if the are written
-        * to swap.
+        * This is consumed and subtracted from the lru size in
+        * zswap_shrinker_count() to penalize past overshrinking that led to disk
+        * swapins. The idea is that had we considered this many more pages in the
+        * LRU active/protected and not written them back, we would not have had to
+        * swapped them in.
         */
-       atomic_long_t nr_zswap_protected;
+       atomic_long_t nr_disk_swapins;
 };
 
 unsigned long zswap_total_pages(void);
mm/zswap.c +0 -0
index adeaf9c97fde..fb3d9cb88785 100644

mm/zswap.c static struct shrinker *zswap_shrinker; · line 193

Quoted 7 lines
mm/zswap.c +4 -0
@@ -187,6 +187,10 @@ static struct shrinker *zswap_shrinker;
  * length - the length in bytes of the compressed page data.  Needed during
  *          decompression. For a same value filled page length is 0, and both
  *          pool and lru are invalid and must be ignored.
+ * referenced - true if the entry recently entered the zswap pool. Unset by the
+ *              dynamic shrinker. The entry is only reclaimed by the dynamic
+ *              shrinker if referenced is unset. See comments in the shrinker
+ *              section for context.

Nit: It is unset and reclaimed by the writeback logic in general, which isn't necessarily triggered from the dynamic shrinker, right?

Quoted 30 lines
mm/zswap.c +1 -6
  * pool - the zswap_pool the entry's data is in
  * handle - zpool allocation handle that stores the compressed page data
  * value - value of the same-value filled pages which have same content
@@ -196,6 +200,7 @@ static struct shrinker *zswap_shrinker;
 struct zswap_entry {
        swp_entry_t swpentry;
        unsigned int length;
+       bool referenced;
        struct zswap_pool *pool;
        union {
                unsigned long handle;
@@ -700,11 +705,8 @@ static inline int entry_to_nid(struct zswap_entry *entry)
 
 static void zswap_lru_add(struct list_lru *list_lru, struct zswap_entry *entry)
 {
-       atomic_long_t *nr_zswap_protected;
-       unsigned long lru_size, old, new;
        int nid = entry_to_nid(entry);
        struct mem_cgroup *memcg;
-       struct lruvec *lruvec;
 
        /*
         * Note that it is safe to use rcu_read_lock() here, even in the face of
@@ -722,19 +724,6 @@ static void zswap_lru_add(struct list_lru *list_lru, struct zswap_entry *entry)
        memcg = mem_cgroup_from_entry(entry);
        /* will always succeed */
        list_lru_add(list_lru, &entry->lru, nid, memcg);
-
-       /* Update the protection area */
-       lru_size = list_lru_count_one(list_lru, nid, memcg);

mm/zswap.c static void zswap_lru_add(struct list_lru *list_lru, struct zswap_entry *entry) · line 737

Quoted 10 lines
mm/zswap.c +0 -10
-       lruvec = mem_cgroup_lruvec(memcg, NODE_DATA(nid));
-       nr_zswap_protected = &lruvec->zswap_lruvec_state.nr_zswap_protected;
-       old = atomic_long_inc_return(nr_zswap_protected);
-       /*
-        * Decay to avoid overflow and adapt to changing workloads.
-        * This is based on LRU reclaim cost decaying heuristics.
-        */
-       do {
-               new = old > lru_size / 4 ? old / 2 : old;
-       } while (!atomic_long_try_cmpxchg(nr_zswap_protected, &old, new));

Nice, arcane heuristics gone :)

LGTM with the above nit:

author maintainer
#3 lore raw

Re: [PATCH v3 1/2] zswap: implement a second chance algorithm for dynamic zswap shrinker

Quoted 4 lines · Yosry Ahmed

Nit: It is unset and reclaimed by the writeback logic in general, which isn't necessarily triggered from the dynamic shrinker, right?

Yeah you're right. Just sent a fixlet correcting this :)

author maintainer
#4 lore raw

PATCH v3 1/2 zswap: implement a second chance algorithm for dynamic zswap shrinker (fix)

In reply to Nhat Pham

Fix a small mistake in the referenced bit documentation.

---
 mm/zswap.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
mm/zswap.c +2 -2
index fb3d9cb88785..44a7234a7b11 100644
@@ -188,8 +188,8 @@ static struct shrinker *zswap_shrinker;
  *          decompression. For a same value filled page length is 0, and both
  *          pool and lru are invalid and must be ignored.
  * referenced - true if the entry recently entered the zswap pool. Unset by the
- *              dynamic shrinker. The entry is only reclaimed by the dynamic
- *              shrinker if referenced is unset. See comments in the shrinker
+ *              writeback logic. The entry is only reclaimed by the writeback
+ *              logic if referenced is unset. See comments in the shrinker
  *              section for context.
  * pool - the zswap_pool the entry's data is in
  * handle - zpool allocation handle that stores the compressed page data
Signature · 1 line
2.43.0
Acked-by on v3 1/2
#5 lore raw

Re: [PATCH v3 1/2] zswap: implement a second chance algorithm for dynamic zswap shrinker (fix)

Quoted 4 lines · Nhat Pham

Fix a small mistake in the referenced bit documentation.

Quoted 21 lines
---
 mm/zswap.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
mm/zswap.c +2 -3
index fb3d9cb88785..44a7234a7b11 100644
@@ -188,8 +188,8 @@ static struct shrinker *zswap_shrinker;
  *          decompression. For a same value filled page length is 0, and both
  *          pool and lru are invalid and must be ignored.
  * referenced - true if the entry recently entered the zswap pool. Unset by the
- *              dynamic shrinker. The entry is only reclaimed by the dynamic
- *              shrinker if referenced is unset. See comments in the shrinker
+ *              writeback logic. The entry is only reclaimed by the writeback
+ *              logic if referenced is unset. See comments in the shrinker
  *              section for context.
  * pool - the zswap_pool the entry's data is in
  * handle - zpool allocation handle that stores the compressed page data
--

2.43.0

maintainer Acked-by on v3 1/2
#6 lore raw

Re: [PATCH v3 1/2] zswap: implement a second chance algorithm for dynamic zswap shrinker

In reply to Nhat Pham
Quoted 64 lines · Nhat Pham

Current zswap shrinker's heuristics to prevent overshrinking is brittle and inaccurate, specifically in the way we decay the protection size (i.e making pages in the zswap LRU eligible for reclaim).

We currently decay protection aggressively in zswap_lru_add() calls. This leads to the following unfortunate effect: when a new batch of pages enter zswap, the protection size rapidly decays to below 25% of the zswap LRU size, which is way too low.

We have observed this effect in production, when experimenting with the zswap shrinker: the rate of shrinking shoots up massively right after a new batch of zswap stores. This is somewhat the opposite of what we want originally - when new pages enter zswap, we want to protect both these new pages AND the pages that are already protected in the zswap LRU.

Replace existing heuristics with a second chance algorithm

1. When a new zswap entry is stored in the zswap pool, its referenced bit is set.
2. When the zswap shrinker encounters a zswap entry with the referenced bit set, give it a second chance - only flips the referenced bit and rotate it in the LRU.
3. If the shrinker encounters the entry again, this time with its referenced bit unset, then it can reclaim the entry.

In this manner, the aging of the pages in the zswap LRUs are decoupled from zswap stores, and picks up the pace with increasing memory pressure (which is what we want).

The second chance scheme allows us to modulate the writeback rate based on recent pool activities. Entries that recently entered the pool will be protected, so if the pool is dominated by such entries the writeback rate will reduce proportionally, protecting the workload's workingset.On the other hand, stale entries will be written back quickly, which increases the effective writeback rate.

The referenced bit is added at the hole after the length field of struct zswap_entry, so there is no extra space overhead for this algorithm.

We will still maintain the count of swapins, which is consumed and subtracted from the lru size in zswap_shrinker_count(), to further penalize past overshrinking that led to disk swapins. The idea is that had we considered this many more pages in the LRU active/protected, they would not have been written back and we would not have had to swapped them in.

To test this new heuristics, I built the kernel under a cgroup with memory.max set to 2G, on a host with 36 cores:

With the old shrinker:

real: 263.89s
user: 4318.11s
sys: 673.29s
swapins: 227300.5

With the second chance algorithm:

real: 244.85s
user: 4327.22s
sys: 664.39s
swapins: 94663

Quoted 8 lines

(average over 5 runs)

We observe an 1.3% reduction in kernel CPU usage, and around 7.2% reduction in real time. Note that the number of swapped in pages dropped by 58%.

author maintainer
#7 lore raw

PATCH v3 2/2 zswap: track swapins from disk more accurately

In reply to Nhat Pham

Currently, there are a couple of issues with our disk swapin tracking for dynamic zswap shrinker heuristics:

1. We only increment the swapin counter on pivot pages. This means we are not taking into account pages that also need to be swapped in, but are already taken care of as part of the readahead window.

2. We are also incrementing when the pages are read from the zswap pool, which is inaccurate.

This patch rectifies these issues by incrementing the counter whenever we need to perform a non-zswap read. Note that we are slightly overcounting, as a page might be read into memory by the readahead algorithm even though it will not be neeeded by users - however, this is an acceptable inaccuracy, as the readahead logic itself will adapt to these kind of scenarios.

To test this change, I built the kernel under a cgroup with its memory.max set to 2 GB:

real: 236.66s
user: 4286.06s
sys: 652.86s
swapins: 81552

For comparison, with just the new second chance algorithm, the build time is as follows:

real: 244.85s
user: 4327.22s
sys: 664.39s
swapins: 94663

Without neither:

real: 263.89s
user: 4318.11s
sys: 673.29s
swapins: 227300.5

(average over 5 runs)

With this change, the kernel CPU time reduces by a further 1.7%, and the real time is reduced by another 3.3%, compared to just the second chance algorithm by itself. The swapins count also reduces by another 13.85%.

Combinng the two changes, we reduce the real time by 10.32%, kernel CPU time by 3%, and number of swapins by 64.12%.

To gauge the new scheme's ability to offload cold data, I ran another benchmark, in which the kernel was built under a cgroup with memory.max set to 3 GB, but with 0.5 GB worth of cold data allocated before each build (in a shmem file).

Under the old scheme:

real: 197.18s
user: 4365.08s
sys: 289.02s
zswpwb: 72115.2

Under the new scheme:

real: 195.8s
user: 4362.25s
sys: 290.14s
zswpwb: 87277.8

(average over 5 runs)

Notice that we actually observe a 21% increase in the number of written back pages - so the new scheme is just as good, if not better at offloading pages from the zswap pool when they are cold. Build time reduces by around 0.7% as a result.

Fixes b5ba474f3f51 ("zswap: shrink zswap pool based on memory pressure")
---
 mm/page_io.c    | 11 ++++++++++-
 mm/swap_state.c |  8 ++------
 2 files changed, 12 insertions(+), 7 deletions(-)
mm/page_io.c +10 -1
index ff8c99ee3af7..0004c9fbf7e8 100644
@@ -521,7 +521,15 @@ void swap_read_folio(struct folio *folio, struct swap_iocb **plug)
 
 	if (zswap_load(folio)) {
 		folio_unlock(folio);
-	} else if (data_race(sis->flags & SWP_FS_OPS)) {
+		goto finish;
+	}
+
+	/*
+	 * We have to read the page from slower devices. Increase zswap protection.
+	 */
+	zswap_folio_swapin(folio);
+
+	if (data_race(sis->flags & SWP_FS_OPS)) {
 		swap_read_folio_fs(folio, plug);
 	} else if (synchronous) {
 		swap_read_folio_bdev_sync(folio, sis);
@@ -529,6 +537,7 @@ void swap_read_folio(struct folio *folio, struct swap_iocb **plug)
 		swap_read_folio_bdev_async(folio, sis);
 	}
 
+finish:
 	if (workingset) {
 		delayacct_thrashing_end(&in_thrashing);
 		psi_memstall_leave(&pflags);
mm/swap_state.c +2 -6
index a1726e49a5eb..3a0cf965f32b 100644
@@ -698,10 +698,8 @@ struct folio *swap_cluster_readahead(swp_entry_t entry, gfp_t gfp_mask,
 	/* The page was likely read above, so no need for plugging here */
 	folio = __read_swap_cache_async(entry, gfp_mask, mpol, ilx,
 					&page_allocated, false);
-	if (unlikely(page_allocated)) {
-		zswap_folio_swapin(folio);
+	if (unlikely(page_allocated))
 		swap_read_folio(folio, NULL);
-	}
 	return folio;
 }
 
@@ -850,10 +848,8 @@ static struct folio *swap_vma_readahead(swp_entry_t targ_entry, gfp_t gfp_mask,
 	/* The folio was likely read above, so no need for plugging here */
 	folio = __read_swap_cache_async(targ_entry, gfp_mask, mpol, targ_ilx,
 					&page_allocated, false);
-	if (unlikely(page_allocated)) {
-		zswap_folio_swapin(folio);
+	if (unlikely(page_allocated))
 		swap_read_folio(folio, NULL);
-	}
 	return folio;
 }
 
Signature · 1 line
2.43.0
Acked-by on v3 2/2
#8 lore raw

Re: [PATCH v3 2/2] zswap: track swapins from disk more accurately

Quoted 8 lines · Nhat Pham

Currently, there are a couple of issues with our disk swapin tracking for dynamic zswap shrinker heuristics:

1. We only increment the swapin counter on pivot pages. This means we are not taking into account pages that also need to be swapped in, but are already taken care of as part of the readahead window.

Quoted 7 lines

2. We are also incrementing when the pages are read from the zswap pool, which is inaccurate.

This patch rectifies these issues by incrementing the counter whenever we need to perform a non-zswap read. Note that we are slightly overcounting, as a page might be read into memory by the readahead algorithm even though it will not be neeeded by users - however, this is

needed*

Quoted 26 lines

an acceptable inaccuracy, as the readahead logic itself will adapt to these kind of scenarios.

To test this change, I built the kernel under a cgroup with its memory.max set to 2 GB:

real: 236.66s
user: 4286.06s
sys: 652.86s
swapins: 81552

For comparison, with just the new second chance algorithm, the build time is as follows:

real: 244.85s
user: 4327.22s
sys: 664.39s
swapins: 94663

Without neither:

real: 263.89s
user: 4318.11s
sys: 673.29s
swapins: 227300.5

Quoted 8 lines

(average over 5 runs)

With this change, the kernel CPU time reduces by a further 1.7%, and the real time is reduced by another 3.3%, compared to just the second chance algorithm by itself. The swapins count also reduces by another 13.85%.

Combinng the two changes, we reduce the real time by 10.32%, kernel CPU

Combining*

Quoted 41 lines

time by 3%, and number of swapins by 64.12%.

To gauge the new scheme's ability to offload cold data, I ran another benchmark, in which the kernel was built under a cgroup with memory.max set to 3 GB, but with 0.5 GB worth of cold data allocated before each build (in a shmem file).

Under the old scheme:

real: 197.18s
user: 4365.08s
sys: 289.02s
zswpwb: 72115.2

Under the new scheme:

real: 195.8s
user: 4362.25s
sys: 290.14s
zswpwb: 87277.8

(average over 5 runs)

Notice that we actually observe a 21% increase in the number of written back pages - so the new scheme is just as good, if not better at offloading pages from the zswap pool when they are cold. Build time reduces by around 0.7% as a result.

Fixes b5ba474f3f51 ("zswap: shrink zswap pool based on memory pressure")
---
 mm/page_io.c    | 11 ++++++++++-
 mm/swap_state.c |  8 ++------
 2 files changed, 12 insertions(+), 7 deletions(-)
mm/page_io.c +0 -0
index ff8c99ee3af7..0004c9fbf7e8 100644

mm/page_io.c void swap_read_folio(struct folio *folio, struct swap_iocb **plug) · line 529

Quoted 10 lines
mm/page_io.c +6 -1
@@ -521,7 +521,15 @@ void swap_read_folio(struct folio *folio, struct swap_iocb **plug)
 
        if (zswap_load(folio)) {
                folio_unlock(folio);
-       } else if (data_race(sis->flags & SWP_FS_OPS)) {
+               goto finish;
+       }
+
+       /*
+        * We have to read the page from slower devices. Increase zswap protection.
+        */

Can we fit this on a single line?

Anyway, LGTM:

author maintainer
#9 lore raw

Re: [PATCH v3 2/2] zswap: track swapins from disk more accurately

Quoted 5 lines · Yosry Ahmed
Quoted 2 lines

algorithm even though it will not be neeeded by users - however, this is

needed*

[...]

Quoted 5 lines
Quoted 2 lines

Combinng the two changes, we reduce the real time by 10.32%, kernel CPU

Combining*

My bad - English is hard :) Thanks for picking these out, Yosry!

Andrew, would you mind fixing these two typos for me?

Quoted 2 lines

Can we fit this on a single line?

Just sent a fixlet to squeeze it into a single line :)

Quoted 3 lines

Anyway, LGTM:

Thanks for the review and feedback on the patch series, Yosry!

author maintainer
#10 lore raw

PATCH v3 2/2 zswap: track swapins from disk more accurately (fix)

In reply to Nhat Pham

Squeeze a comment into a single line.

---
 mm/page_io.c | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)
mm/page_io.c +1 -3
index 0004c9fbf7e8..aa190e3cb050 100644
@@ -524,9 +524,7 @@ void swap_read_folio(struct folio *folio, struct swap_iocb **plug)
 		goto finish;
 	}
 
-	/*
-	 * We have to read the page from slower devices. Increase zswap protection.
-	 */
+	/* We have to read from slower devices. Increase zswap protection. */
 	zswap_folio_swapin(folio);
 
 	if (data_race(sis->flags & SWP_FS_OPS)) {
Signature · 1 line
2.43.0
maintainer Acked-by on v3 2/2
#11 lore raw

Re: [PATCH v3 2/2] zswap: track swapins from disk more accurately

In reply to Nhat Pham
Quoted 71 lines · Nhat Pham

Currently, there are a couple of issues with our disk swapin tracking for dynamic zswap shrinker heuristics:

1. We only increment the swapin counter on pivot pages. This means we are not taking into account pages that also need to be swapped in, but are already taken care of as part of the readahead window.

2. We are also incrementing when the pages are read from the zswap pool, which is inaccurate.

This patch rectifies these issues by incrementing the counter whenever we need to perform a non-zswap read. Note that we are slightly overcounting, as a page might be read into memory by the readahead algorithm even though it will not be neeeded by users - however, this is an acceptable inaccuracy, as the readahead logic itself will adapt to these kind of scenarios.

To test this change, I built the kernel under a cgroup with its memory.max set to 2 GB:

real: 236.66s
user: 4286.06s
sys: 652.86s
swapins: 81552

For comparison, with just the new second chance algorithm, the build time is as follows:

real: 244.85s
user: 4327.22s
sys: 664.39s
swapins: 94663

Without neither:

real: 263.89s
user: 4318.11s
sys: 673.29s
swapins: 227300.5

(average over 5 runs)

With this change, the kernel CPU time reduces by a further 1.7%, and the real time is reduced by another 3.3%, compared to just the second chance algorithm by itself. The swapins count also reduces by another 13.85%.

Combinng the two changes, we reduce the real time by 10.32%, kernel CPU time by 3%, and number of swapins by 64.12%.

To gauge the new scheme's ability to offload cold data, I ran another benchmark, in which the kernel was built under a cgroup with memory.max set to 3 GB, but with 0.5 GB worth of cold data allocated before each build (in a shmem file).

Under the old scheme:

real: 197.18s
user: 4365.08s
sys: 289.02s
zswpwb: 72115.2

Under the new scheme:

real: 195.8s
user: 4362.25s
sys: 290.14s
zswpwb: 87277.8

(average over 5 runs)

Quoted 8 lines

Notice that we actually observe a 21% increase in the number of written back pages - so the new scheme is just as good, if not better at offloading pages from the zswap pool when they are cold. Build time reduces by around 0.7% as a result.

Fixes b5ba474f3f51 ("zswap: shrink zswap pool based on memory pressure")