Skip to contentSkip to conversation

mm: zsmalloc: make shrinker compaction budget-aware

RFC PATCH In review [email protected] 6 participants 40 messages Aug 6 – Nov 18

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

cover mm: zsmalloc: make shrinker compaction budget-aware

From: Xueyuan Chen <[email protected]>

zs_shrinker_scan() currently calls zs_compact(), which compacts the whole pool regardless of sc->nr_to_scan. A single shrinker callback can therefore reclaim much more than requested and spend a long time in compaction.

On an Android device with 12 GB of RAM, observed zsmalloc compaction durations had a p95 of 38.86 ms and a maximum of 269.68 ms, motivating a per-scan reclaim goal.

This series uses sc->nr_to_scan as a reclaimed-page goal and keeps a per-pool size-class cursor so later scans resume where the previous scan stopped. The existing full-pool behavior of zs_compact() is unchanged.

Xueyuan Chen (2):
mm: zsmalloc: add a page limit to pool compaction
mm: zsmalloc: use the shrinker reclaim budget

 mm/zsmalloc.c | 46 +++++++++++++++++++++++++++++++---------------
 1 file changed, 31 insertions(+), 15 deletions(-)

--
2.47.3

#1 lore raw

RFC PATCH 1/2 mm: zsmalloc: add a page limit to pool compaction

From: Xueyuan Chen <[email protected]>

Allow pool compaction to stop after reclaiming a requested number of pages. This prepares it for budgeted callers while keeping zs_compact() behavior unchanged.

---
 mm/zsmalloc.c | 21 +++++++++++++++++----
 1 file changed, 17 insertions(+), 4 deletions(-)
mm/zsmalloc.c +17 -4
index 83f5820c45f9..334d24093c64 100644
@@ -1884,7 +1884,8 @@ static unsigned long zs_can_compact(struct size_class *class)
 }
 
 static unsigned long __zs_compact(struct zs_pool *pool,
-				  struct size_class *class)
+				  struct size_class *class,
+				  unsigned long max_pages)
 {
 	struct zspage *src_zspage = NULL;
 	struct zspage *dst_zspage = NULL;
@@ -1896,7 +1897,7 @@ static unsigned long __zs_compact(struct zs_pool *pool,
 	 */
 	write_lock(&pool->lock);
 	spin_lock(&class->lock);
-	while (zs_can_compact(class)) {
+	while ((pages_freed < max_pages) && zs_can_compact(class)) {
 		int fg;
 
 		if (!dst_zspage) {
@@ -1947,12 +1948,16 @@ static unsigned long __zs_compact(struct zs_pool *pool,
 	return pages_freed;
 }
 
-unsigned long zs_compact(struct zs_pool *pool)
+static unsigned long zs_compact_pool(struct zs_pool *pool,
+					unsigned long max_pages)
 {
 	int i;
 	struct size_class *class;
 	unsigned long pages_freed = 0;
 
+	if (!max_pages)
+		return 0;
+
 	/*
 	 * Pool compaction is performed under pool->lock so it is basically
 	 * single-threaded. Having more than one thread in __zs_compact()
@@ -1966,13 +1971,21 @@ unsigned long zs_compact(struct zs_pool *pool)
 		class = pool->size_class[i];
 		if (class->index != i)
 			continue;
-		pages_freed += __zs_compact(pool, class);
+		pages_freed += __zs_compact(pool, class,
+					max_pages - pages_freed);
+		if (pages_freed >= max_pages)
+			break;
 	}
 	atomic_long_add(pages_freed, &pool->stats.pages_compacted);
 	atomic_set(&pool->compaction_in_progress, 0);
 
 	return pages_freed;
 }
+
+unsigned long zs_compact(struct zs_pool *pool)
+{
+	return zs_compact_pool(pool, ULONG_MAX);
+}
 EXPORT_SYMBOL_GPL(zs_compact);
 
 void zs_pool_stats(struct zs_pool *pool, struct zs_pool_stats *stats)
Signature · 1 line
2.47.3
#2 lore raw

RFC PATCH 2/2 mm: zsmalloc: use the shrinker reclaim budget

In reply to [email protected]

From: Xueyuan Chen <[email protected]>

The zsmalloc shrinker currently compacts the whole pool for every scan, which can do much more work than reclaim requires.

Use the requested page count as the compaction goal and continue later scans from the previous position.

---
 mm/zsmalloc.c | 33 ++++++++++++++++++---------------
 1 file changed, 18 insertions(+), 15 deletions(-)
mm/zsmalloc.c +18 -15
index 334d24093c64..aa5900ce2620 100644
@@ -217,6 +217,8 @@ struct zs_pool {
 	/* protect zspage migration/compaction */
 	rwlock_t lock;
 	atomic_t compaction_in_progress;
+	/* next class shrinker triggered compaction */
+	unsigned int compact_cursor;
 };
 
 static inline void zpdesc_set_first(struct zpdesc *zpdesc)
@@ -1949,9 +1951,10 @@ static unsigned long __zs_compact(struct zs_pool *pool,
 }
 
 static unsigned long zs_compact_pool(struct zs_pool *pool,
-					unsigned long max_pages)
+					unsigned long max_pages,
+					bool use_cursor)
 {
-	int i;
+	unsigned int index, nr_scanned;
 	struct size_class *class;
 	unsigned long pages_freed = 0;
 
@@ -1967,15 +1970,19 @@ static unsigned long zs_compact_pool(struct zs_pool *pool,
 	if (atomic_xchg(&pool->compaction_in_progress, 1))
 		return 0;
 
-	for (i = ZS_SIZE_CLASSES - 1; i >= 0; i--) {
-		class = pool->size_class[i];
-		if (class->index != i)
-			continue;
-		pages_freed += __zs_compact(pool, class,
-					max_pages - pages_freed);
+	index = use_cursor ? pool->compact_cursor : ZS_SIZE_CLASSES - 1;
+	for (nr_scanned = ZS_SIZE_CLASSES; nr_scanned; nr_scanned--) {
+		class = pool->size_class[index];
+		if (class->index == index)
+			pages_freed += __zs_compact(pool, class,
+						max_pages - pages_freed);
+		index = index ? index - 1 : ZS_SIZE_CLASSES - 1;
 		if (pages_freed >= max_pages)
 			break;
 	}
+	if (use_cursor)
+		pool->compact_cursor = index;
+
 	atomic_long_add(pages_freed, &pool->stats.pages_compacted);
 	atomic_set(&pool->compaction_in_progress, 0);
 
@@ -1984,7 +1991,7 @@ static unsigned long zs_compact_pool(struct zs_pool *pool,
 
 unsigned long zs_compact(struct zs_pool *pool)
 {
-	return zs_compact_pool(pool, ULONG_MAX);
+	return zs_compact_pool(pool, ULONG_MAX, false);
 }
 EXPORT_SYMBOL_GPL(zs_compact);
 
@@ -2000,12 +2007,7 @@ static unsigned long zs_shrinker_scan(struct shrinker *shrinker,
 	unsigned long pages_freed;
 	struct zs_pool *pool = shrinker->private_data;
 
-	/*
-	 * Compact classes and calculate compaction delta.
-	 * Can run concurrently with a manually triggered
-	 * (by user) compaction.
-	 */
-	pages_freed = zs_compact(pool);
+	pages_freed = zs_compact_pool(pool, sc->nr_to_scan, true);
 
 	return pages_freed ? pages_freed : SHRINK_STOP;
 }
@@ -2094,6 +2096,7 @@ struct zs_pool *zs_create_pool(const char *name)
 	init_deferred_free(pool);
 	rwlock_init(&pool->lock);
 	atomic_set(&pool->compaction_in_progress, 0);
+	pool->compact_cursor = ZS_SIZE_CLASSES - 1;
 
 	pool->name = kstrdup(name, GFP_KERNEL);
 	if (!pool->name)
Signature · 1 line
2.47.3
#3 lore raw
In reply to [email protected]
Quoted 3 lines · On (26/08/06 16:27), [email protected] wrote:

On an Android device with 12 GB of RAM, observed zsmalloc compaction durations had a p95 of 38.86 ms and a maximum of 269.68 ms, motivating a per-scan reclaim goal.

Would it be possible to give a little more data? What was the fragmentation ratio, how much memory was saved during that auto-compaction, etc. If possible.

Somewhere in the back of my mind I was thinking about, maybe, disabling (removing) zsmalloc shrinker callbacks, in other words disabling auto-compaction. We have a sysfs knob for pool compaction for system that still want to run compaction. So I'm leaning towards removal of shrinker callbacks from zsmalloc.

#4 lore raw

Hi Sergey,

Here is some additional data:

I used the following definitions:
  compactable ratio = freeable_pages / total_pages memory reclaimed  = pages_freed * PAGE_SIZE

freeable_pages is the estimate before compaction, based on the same calculation as zs_shrinker_count(), while pages_freed is the actual number of backing pages released.

There were 264 callbacks in the trace:
  callback elapsed time:
    median:    5.77 ms
    p95:      55.82 ms
    maximum: 271.36 ms

  compactable ratio before compaction:
    median:   0.32%
    p95:      2.86%
    maximum:  8.33%

  memory reclaimed per callback:
    median:    3.80 MiB
    p95:      30.45 MiB
    maximum:  92.73 MiB

The longest callback took 271.36 ms. Its compactable ratio was 3.25%, and it released 7,650 pages, or about 29.88 MiB.

There was also a 241.91 ms callback (with 30 schedule-outs) with a compactable ratio of 0.44%. It released 1,019 pages, or about 3.98 MiB.

Based on this data, it seems better to remove the shrinker.

Would you prefer that I change v2 to remove the zsmalloc shrinker callbacks directly?

Thanks,
Xueyuan

Quoted 13 lines · Sergey Senozhatsky
Quoted 3 lines · On (26/08/06 16:27), [email protected] wrote:

On an Android device with 12 GB of RAM, observed zsmalloc compaction durations had a p95 of 38.86 ms and a maximum of 269.68 ms, motivating a per-scan reclaim goal.

Would it be possible to give a little more data? What was the fragmentation ratio, how much memory was saved during that auto-compaction, etc. If possible.

Somewhere in the back of my mind I was thinking about, maybe, disabling (removing) zsmalloc shrinker callbacks, in other words disabling auto-compaction. We have a sysfs knob for pool compaction for system that still want to run compaction. So I'm leaning towards removal of shrinker callbacks from zsmalloc.

#5 lore raw
Quoted 31 lines · Xueyuan Chen

Hi Sergey,

Here is some additional data:

I used the following definitions:
  compactable ratio = freeable_pages / total_pages memory reclaimed  = pages_freed * PAGE_SIZE

freeable_pages is the estimate before compaction, based on the same calculation as zs_shrinker_count(), while pages_freed is the actual number of backing pages released.

There were 264 callbacks in the trace:
  callback elapsed time:
    median:    5.77 ms
    p95:      55.82 ms
    maximum: 271.36 ms

  compactable ratio before compaction:
    median:   0.32%
    p95:      2.86%
    maximum:  8.33%

  memory reclaimed per callback:
    median:    3.80 MiB
    p95:      30.45 MiB
    maximum:  92.73 MiB

The longest callback took 271.36 ms. Its compactable ratio was 3.25%, and it released 7,650 pages, or about 29.88 MiB.

Quoted 8 lines

There was also a 241.91 ms callback (with 30 schedule-outs) with a compactable ratio of 0.44%. It released 1,019 pages, or about 3.98 MiB.

Based on this data, it seems better to remove the shrinker.

Would you prefer that I change v2 to remove the zsmalloc shrinker callbacks directly?

Let's bring in heavy artillery to this discussion, in addition to Andrew and Minchan, adding Nhat, Yosry, Barry, Johannes, Brian (random order).

Folks, I'm bullish on removal of zsmalloc shrinker callbacks. I don't think those buy us much apart from memcpy-s and lock contention. Systems that want to compact zsmalloc have a sysfs knob (and API) to do so (based on zram mm_stat numbers).

Any thoughts?

#6 lore raw
Quoted 42 lines · Sergey Senozhatsky
Quoted 31 lines · Xueyuan Chen

Hi Sergey,

Here is some additional data:

I used the following definitions:
compactable ratio = freeable_pages / total_pages memory reclaimed = pages_freed * PAGE_SIZE

freeable_pages is the estimate before compaction, based on the same calculation as zs_shrinker_count(), while pages_freed is the actual number of backing pages released.

There were 264 callbacks in the trace:
callback elapsed time:

    median:    5.77 ms
    p95:      55.82 ms
    maximum: 271.36 ms

compactable ratio before compaction:

    median:   0.32%
    p95:      2.86%
    maximum:  8.33%

memory reclaimed per callback:

    median:    3.80 MiB
    p95:      30.45 MiB
    maximum:  92.73 MiB

The longest callback took 271.36 ms. Its compactable ratio was 3.25%, and it released 7,650 pages, or about 29.88 MiB.

Quoted 8 lines

There was also a 241.91 ms callback (with 30 schedule-outs) with a compactable ratio of 0.44%. It released 1,019 pages, or about 3.98 MiB.

Based on this data, it seems better to remove the shrinker.

Would you prefer that I change v2 to remove the zsmalloc shrinker callbacks directly?

Quoted 7 lines

Let's bring in heavy artillery to this discussion, in addition to Andrew and Minchan, adding Nhat, Yosry, Barry, Johannes, Brian (random order).

Folks, I'm bullish on removal of zsmalloc shrinker callbacks. I don't think those buy us much apart from memcpy-s and lock contention. Systems that want to compact zsmalloc have a sysfs knob (and API) to do so (based on zram mm_stat numbers).

Hmm we'd need to collect more data in our workloads to determine, but if we can compute compactable ratio (freeable_pages / total_pages) on a per size class basis, can we just skip the size class whose ratio is too bad? Would that at least cut down on the vast majority of fruitless memcpys and lock acquisitions etc?

I'm always a bit hesitant to over-rely on userspace, especially when kernel has information to do something smart about it. It might not react in time, and many proactive reclaiming schemes back off under heavy memory pressure.

Quoted 2 lines

Any thoughts?

#7 lore raw
Quoted 55 lines · Nhat Pham
Quoted 40 lines · Sergey Senozhatsky
Quoted 31 lines · Xueyuan Chen

Hi Sergey,

Here is some additional data:

I used the following definitions:
compactable ratio = freeable_pages / total_pages memory reclaimed = pages_freed * PAGE_SIZE

freeable_pages is the estimate before compaction, based on the same calculation as zs_shrinker_count(), while pages_freed is the actual number of backing pages released.

There were 264 callbacks in the trace:

   callback elapsed time:
     median:    5.77 ms
     p95:      55.82 ms
     maximum: 271.36 ms

   compactable ratio before compaction:
     median:   0.32%
     p95:      2.86%
     maximum:  8.33%

   memory reclaimed per callback:
     median:    3.80 MiB
     p95:      30.45 MiB
     maximum:  92.73 MiB

The longest callback took 271.36 ms. Its compactable ratio was 3.25%, and it released 7,650 pages, or about 29.88 MiB.

Quoted 8 lines

There was also a 241.91 ms callback (with 30 schedule-outs) with a compactable ratio of 0.44%. It released 1,019 pages, or about 3.98 MiB.

Based on this data, it seems better to remove the shrinker.

Would you prefer that I change v2 to remove the zsmalloc shrinker callbacks directly?

Quoted 7 lines

Let's bring in heavy artillery to this discussion, in addition to Andrew and Minchan, adding Nhat, Yosry, Barry, Johannes, Brian (random order).

Folks, I'm bullish on removal of zsmalloc shrinker callbacks. I don't think those buy us much apart from memcpy-s and lock contention. Systems that want to compact zsmalloc have a sysfs knob (and API) to do so (based on zram mm_stat numbers).

Hmm we'd need to collect more data in our workloads to determine, but if we can compute compactable ratio (freeable_pages / total_pages) on a per size class basis, can we just skip the size class whose ratio is too bad? Would that at least cut down on the vast majority of fruitless memcpys and lock acquisitions etc?

Quoted 4 lines

I'm always a bit hesitant to over-rely on userspace, especially when kernel has information to do something smart about it. It might not react in time, and many proactive reclaiming schemes back off under heavy memory pressure.

Hi Nhat,

I collected per size-class data. 53.7% of size classes have freeable==0,
so skipping them saves some time. However, 94.7% of zs_compact() time is
spent on classes where freeable > 0 — meaning the cost is dominated by the actual compaction work, not by scanning empty classes.

That said, skipping freeable == 0 classes is still a worthwhile optimization on its own — it avoids unnecessary write_lock,spin_lock. But it would not meaningfully reduce the worst-case latency.

Thanks,
Xueyuan

Quoted 1 line
Quoted 1 line

Any thoughts?

#8 lore raw
Quoted 63 lines · Xueyuan Chen
Quoted 55 lines · Nhat Pham
Quoted 40 lines · Sergey Senozhatsky
Quoted 31 lines · Xueyuan Chen

Hi Sergey,

Here is some additional data:

I used the following definitions:
compactable ratio = freeable_pages / total_pages memory reclaimed = pages_freed * PAGE_SIZE

freeable_pages is the estimate before compaction, based on the same calculation as zs_shrinker_count(), while pages_freed is the actual number of backing pages released.

There were 264 callbacks in the trace:

   callback elapsed time:
     median:    5.77 ms
     p95:      55.82 ms
     maximum: 271.36 ms

   compactable ratio before compaction:
     median:   0.32%
     p95:      2.86%
     maximum:  8.33%

   memory reclaimed per callback:
     median:    3.80 MiB
     p95:      30.45 MiB
     maximum:  92.73 MiB

The longest callback took 271.36 ms. Its compactable ratio was 3.25%, and it released 7,650 pages, or about 29.88 MiB.

Quoted 8 lines

There was also a 241.91 ms callback (with 30 schedule-outs) with a compactable ratio of 0.44%. It released 1,019 pages, or about 3.98 MiB.

Based on this data, it seems better to remove the shrinker.

Would you prefer that I change v2 to remove the zsmalloc shrinker callbacks directly?

Quoted 7 lines

Let's bring in heavy artillery to this discussion, in addition to Andrew and Minchan, adding Nhat, Yosry, Barry, Johannes, Brian (random order).

Folks, I'm bullish on removal of zsmalloc shrinker callbacks. I don't think those buy us much apart from memcpy-s and lock contention. Systems that want to compact zsmalloc have a sysfs knob (and API) to do so (based on zram mm_stat numbers).

Hmm we'd need to collect more data in our workloads to determine, but if we can compute compactable ratio (freeable_pages / total_pages) on a per size class basis, can we just skip the size class whose ratio is too bad? Would that at least cut down on the vast majority of fruitless memcpys and lock acquisitions etc?

Quoted 4 lines

I'm always a bit hesitant to over-rely on userspace, especially when kernel has information to do something smart about it. It might not react in time, and many proactive reclaiming schemes back off under heavy memory pressure.

Quoted 6 lines

Hi Nhat,

I collected per size-class data. 53.7% of size classes have freeable==0, so skipping them saves some time. However, 94.7% of zs_compact() time is spent on classes where freeable > 0 — meaning the cost is dominated by the actual compaction work, not by scanning empty classes.

What's the distribution of these? Say if I were to skip all size class with compactable ratio below 20%, or those where we can free at least one full zspage? (IIUC, we only free any memory at all if a zspage got freed up from our internal compaction, correct?).

#9 lore raw
Quoted 65 lines · Nhat Pham
Quoted 63 lines · Xueyuan Chen
Quoted 59 lines · Nhat Pham
Quoted 40 lines · Sergey Senozhatsky
Quoted 31 lines · Xueyuan Chen

Hi Sergey,

Here is some additional data:

I used the following definitions:
compactable ratio = freeable_pages / total_pages memory reclaimed = pages_freed * PAGE_SIZE

freeable_pages is the estimate before compaction, based on the same calculation as zs_shrinker_count(), while pages_freed is the actual number of backing pages released.

There were 264 callbacks in the trace:

   callback elapsed time:
     median:    5.77 ms
     p95:      55.82 ms
     maximum: 271.36 ms

   compactable ratio before compaction:
     median:   0.32%
     p95:      2.86%
     maximum:  8.33%

   memory reclaimed per callback:
     median:    3.80 MiB
     p95:      30.45 MiB
     maximum:  92.73 MiB

The longest callback took 271.36 ms. Its compactable ratio was 3.25%, and it released 7,650 pages, or about 29.88 MiB.

Quoted 8 lines

There was also a 241.91 ms callback (with 30 schedule-outs) with a compactable ratio of 0.44%. It released 1,019 pages, or about 3.98 MiB.

Based on this data, it seems better to remove the shrinker.

Would you prefer that I change v2 to remove the zsmalloc shrinker callbacks directly?

Quoted 7 lines

Let's bring in heavy artillery to this discussion, in addition to Andrew and Minchan, adding Nhat, Yosry, Barry, Johannes, Brian (random order).

Folks, I'm bullish on removal of zsmalloc shrinker callbacks. I don't think those buy us much apart from memcpy-s and lock contention. Systems that want to compact zsmalloc have a sysfs knob (and API) to do so (based on zram mm_stat numbers).

Hmm we'd need to collect more data in our workloads to determine, but if we can compute compactable ratio (freeable_pages / total_pages) on a per size class basis, can we just skip the size class whose ratio is too bad? Would that at least cut down on the vast majority of fruitless memcpys and lock acquisitions etc?

I'm always a bit hesitant to over-rely on userspace, especially when kernel has information to do something smart about it. It might not react in time, and many proactive reclaiming schemes back off under heavy memory pressure.

Quoted 7 lines
Quoted 6 lines

Hi Nhat,

I collected per size-class data. 53.7% of size classes have freeable==0, so skipping them saves some time. However, 94.7% of zs_compact() time is spent on classes where freeable > 0 — meaning the cost is dominated by the actual compaction work, not by scanning empty classes.

Hi Nhat,

Sorry for the late reply.

Quoted 4 lines

What's the distribution of these? Say if I were to skip all size class with compactable ratio below 20%, or those where we can free at least one full zspage? (IIUC, we only free any memory at all if a zspage got freed up from our internal compaction, correct?).

You are right, we only free the backing pages when a full zspage is emptied by compaction.

My test device had some problems, I need some time to collect new data. The good news is the per-size-class distribution should be ready next week.

Thanks

#10 lore raw
In reply to Nhat Pham
Quoted 64 lines · Nhat Pham
Quoted 63 lines · Xueyuan Chen
Quoted 55 lines · Nhat Pham
Quoted 40 lines · Sergey Senozhatsky
Quoted 31 lines · Xueyuan Chen

Hi Sergey,

Here is some additional data:

I used the following definitions:
compactable ratio = freeable_pages / total_pages memory reclaimed = pages_freed * PAGE_SIZE

freeable_pages is the estimate before compaction, based on the same calculation as zs_shrinker_count(), while pages_freed is the actual number of backing pages released.

There were 264 callbacks in the trace:

    callback elapsed time:
      median:    5.77 ms
      p95:      55.82 ms
      maximum: 271.36 ms

    compactable ratio before compaction:
      median:   0.32%
      p95:      2.86%
      maximum:  8.33%

    memory reclaimed per callback:
      median:    3.80 MiB
      p95:      30.45 MiB
      maximum:  92.73 MiB

The longest callback took 271.36 ms. Its compactable ratio was 3.25%, and it released 7,650 pages, or about 29.88 MiB.

Quoted 8 lines

There was also a 241.91 ms callback (with 30 schedule-outs) with a compactable ratio of 0.44%. It released 1,019 pages, or about 3.98 MiB.

Based on this data, it seems better to remove the shrinker.

Would you prefer that I change v2 to remove the zsmalloc shrinker callbacks directly?

Quoted 7 lines

Let's bring in heavy artillery to this discussion, in addition to Andrew and Minchan, adding Nhat, Yosry, Barry, Johannes, Brian (random order).

Folks, I'm bullish on removal of zsmalloc shrinker callbacks. I don't think those buy us much apart from memcpy-s and lock contention. Systems that want to compact zsmalloc have a sysfs knob (and API) to do so (based on zram mm_stat numbers).

Hmm we'd need to collect more data in our workloads to determine, but if we can compute compactable ratio (freeable_pages / total_pages) on a per size class basis, can we just skip the size class whose ratio is too bad? Would that at least cut down on the vast majority of fruitless memcpys and lock acquisitions etc?

Quoted 4 lines

I'm always a bit hesitant to over-rely on userspace, especially when kernel has information to do something smart about it. It might not react in time, and many proactive reclaiming schemes back off under heavy memory pressure.

Hi Nhat,

Quoted 8 lines
Quoted 4 lines

I collected per size-class data. 53.7% of size classes have freeable==0, so skipping them saves some time. However, 94.7% of zs_compact() time is spent on classes where freeable > 0 — meaning the cost is dominated by the actual compaction work, not by scanning empty classes.

What's the distribution of these? Say if I were to skip all size class with compactable ratio below 20%, or those where we can free at least one full zspage? (IIUC, we only free any memory at all if a zspage got freed up from our internal compaction, correct?).

Hi Nhat,

Across 106 zram0 compaction callbacks, there were 6,246 per-class events with freeable > 0, covering 118 size classes.

The data by object size was:

  | object size (bytes) | classes | events | total pages | freeable pages |
  | [0, 256)            |   14    |   570  |    182297   | 6728      |
  | [256, 512)          |   16    |   748  |    480127   |  14526      |
  | [512, 1024)         |   30    |  1586  |   2677075   |  71934      |
  | [1024, 2048)        |   36    |  2180  |  11990170   | 294133      |
  | [2048, 3072)        |   15    |   821  |   5089301   | 106204      |
  | [3072, 4096]        |    7    |   341  |   1170843   |  20012      |

The compactable-ratio distribution at class-compaction entry was:

  p25:       0.75%
  median:    1.65%
  p90:       6.76%
  p95:       9.42%
  p99:      14.52%
  maximum:  30.99%

Only 12 out of 6,246 events had a ratio of 20% or higher, so a 20% threshold would skip nearly all of them.

          | freeable pages | total pages | compactable ratio |
  class A |       10       |      20     |        50%        |
  class B |       20       |      50     |        40%        |

In this example, class B has more freeable pages even though its ratio is lower. So I don't think the ratio alone is enough to decide whether to compact a class.

Thanks,
Xueyuan

#11 lore raw
In reply to Nhat Pham
Quoted 55 lines · Nhat Pham
Quoted 41 lines · Xueyuan Chen
Quoted 31 lines

Hi Sergey,

Here is some additional data:

I used the following definitions:
compactable ratio = freeable_pages / total_pages memory reclaimed = pages_freed * PAGE_SIZE

freeable_pages is the estimate before compaction, based on the same calculation as zs_shrinker_count(), while pages_freed is the actual number of backing pages released.

There were 264 callbacks in the trace:
callback elapsed time:

    median:    5.77 ms
    p95:      55.82 ms
    maximum: 271.36 ms

compactable ratio before compaction:

    median:   0.32%
    p95:      2.86%
    maximum:  8.33%

memory reclaimed per callback:

    median:    3.80 MiB
    p95:      30.45 MiB
    maximum:  92.73 MiB

The longest callback took 271.36 ms. Its compactable ratio was 3.25%, and it released 7,650 pages, or about 29.88 MiB.

Quoted 8 lines

There was also a 241.91 ms callback (with 30 schedule-outs) with a compactable ratio of 0.44%. It released 1,019 pages, or about 3.98 MiB.

Based on this data, it seems better to remove the shrinker.

Would you prefer that I change v2 to remove the zsmalloc shrinker callbacks directly?

Quoted 7 lines

Let's bring in heavy artillery to this discussion, in addition to Andrew and Minchan, adding Nhat, Yosry, Barry, Johannes, Brian (random order).

Folks, I'm bullish on removal of zsmalloc shrinker callbacks. I don't think those buy us much apart from memcpy-s and lock contention. Systems that want to compact zsmalloc have a sysfs knob (and API) to do so (based on zram mm_stat numbers).

Hmm we'd need to collect more data in our workloads to determine, but if we can compute compactable ratio (freeable_pages / total_pages) on a per size class basis, can we just skip the size class whose ratio is too bad? Would that at least cut down on the vast majority of fruitless memcpys and lock acquisitions etc?

Quoted 4 lines

I'm always a bit hesitant to over-rely on userspace, especially when kernel has information to do something smart about it. It might not react in time, and many proactive reclaiming schemes back off under heavy memory pressure.

Apologies for the delay.

The thing about shrinker compaction is that it's sort of on the wrong end. It doesn't trigger compaction when memory is plentiful, no matter the internal fragmentation rate, and start firing compaction scans when memory pressure hits. The problem with this is that, now compaction moves objects (if can compact zspages) and releases empty zspages, in parallel with potentially heavy swapping out activities, which can simply allocate those released pages back. Which seems like a useless exercise and needless memcpy-s. So I was thinking about either putting it under config or just removing it (this might be a harder decision).

#12 lore raw

On (26/10/29 10:43), Sergey Senozhatsky wrote:
[..]

Quoted 13 lines
Quoted 10 lines

Hmm we'd need to collect more data in our workloads to determine, but if we can compute compactable ratio (freeable_pages / total_pages) on a per size class basis, can we just skip the size class whose ratio is too bad? Would that at least cut down on the vast majority of fruitless memcpys and lock acquisitions etc?

I'm always a bit hesitant to over-rely on userspace, especially when kernel has information to do something smart about it. It might not react in time, and many proactive reclaiming schemes back off under heavy memory pressure.

Apologies for the delay.

Quoted 10 lines

The thing about shrinker compaction is that it's sort of on the wrong end. It doesn't trigger compaction when memory is plentiful, no matter the internal fragmentation rate, and start firing compaction scans when memory pressure hits. The problem with this is that, now compaction moves objects (if can compact zspages) and releases empty zspages, in parallel with potentially heavy swapping out activities, which can simply allocate those released pages back. Which seems like a useless exercise and needless memcpy-s. So I was thinking about either putting it under config or just removing it (this might be a harder decision).

In embedded (android/bookOS/etc.) there is a daemon, that triggers various zram activities. We can wire in pool-compaction (once under memory pressure), which is a zram sysfs device attribute, instead of pondering zsmalloc with scan callbacks.

#13 lore raw
Quoted 26 lines · Sergey Senozhatsky

On (26/10/29 10:43), Sergey Senozhatsky wrote:
[..]

Quoted 13 lines
Quoted 10 lines

Hmm we'd need to collect more data in our workloads to determine, but if we can compute compactable ratio (freeable_pages / total_pages) on a per size class basis, can we just skip the size class whose ratio is too bad? Would that at least cut down on the vast majority of fruitless memcpys and lock acquisitions etc?

I'm always a bit hesitant to over-rely on userspace, especially when kernel has information to do something smart about it. It might not react in time, and many proactive reclaiming schemes back off under heavy memory pressure.

Apologies for the delay.

Quoted 10 lines

The thing about shrinker compaction is that it's sort of on the wrong end. It doesn't trigger compaction when memory is plentiful, no matter the internal fragmentation rate, and start firing compaction scans when memory pressure hits. The problem with this is that, now compaction moves objects (if can compact zspages) and releases empty zspages, in parallel with potentially heavy swapping out activities, which can simply allocate those released pages back. Which seems like a useless exercise and needless memcpy-s. So I was thinking about either putting it under config or just removing it (this might be a harder decision).

Quoted 5 lines

In embedded (android/bookOS/etc.) there is a daemon, that triggers various zram activities. We can wire in pool-compaction (once under memory pressure), which is a zram sysfs device attribute, instead of pondering zsmalloc with scan callbacks.

What triggering policy do you have in mind for the daemon?

If compaction is still triggered when the system is under memory pressure, wouldn't it still overlap with heavy swap-out? The shrinker can already run from kswapd, so moving it to a daemon alone would not necessarily change that timing.

#14 lore raw
In reply to Sergey Senozhatsky
Quoted 26 lines · Sergey Senozhatsky

On (26/10/29 10:43), Sergey Senozhatsky wrote:
[..]

Quoted 23 lines
Quoted 10 lines

Hmm we'd need to collect more data in our workloads to determine, but if we can compute compactable ratio (freeable_pages / total_pages) on a per size class basis, can we just skip the size class whose ratio is too bad? Would that at least cut down on the vast majority of fruitless memcpys and lock acquisitions etc?

I'm always a bit hesitant to over-rely on userspace, especially when kernel has information to do something smart about it. It might not react in time, and many proactive reclaiming schemes back off under heavy memory pressure.

Apologies for the delay.

The thing about shrinker compaction is that it's sort of on the wrong end. It doesn't trigger compaction when memory is plentiful, no matter the internal fragmentation rate, and start firing compaction scans when memory pressure hits. The problem with this is that, now compaction moves objects (if can compact zspages) and releases empty zspages, in parallel with potentially heavy swapping out activities, which can simply allocate those released pages back. Which seems like a useless exercise and needless memcpy-s. So I was thinking about either putting it under config or just removing it (this might be a harder decision).

Hmmm I suppose if the new objects belong to the fragmented size class, then there is no point in doing defragmentation, because the new objects will fill the holes.

If they are of different size classes though it's a win to free these fragmented zspages and reallocate them to the size classes of the incoming objects right?

I can't really say which is more likely without data... And it might be workload/reclaim-behavior dependent...

(The size class check for fragmentation level to skip internal compaction is nice here because is agnostic to distribution and trigger points. We should land that regardless).

Quoted 5 lines

In embedded (android/bookOS/etc.) there is a daemon, that triggers various zram activities. We can wire in pool-compaction (once under memory pressure), which is a zram sysfs device attribute, instead of pondering zsmalloc with scan callbacks.

Internally we use senpai, which trigger memory reclaim when there is no memory pressure, to reclaim cold data (which in turns trigger zsmalloc compaction).

It still does both zswapping and zsmalloc compaction in the same round-ish though.

#15 lore raw
In reply to Sergey Senozhatsky
Quoted 61 lines · Sergey Senozhatsky
Quoted 55 lines · Nhat Pham
Quoted 41 lines · Xueyuan Chen
Quoted 31 lines

Hi Sergey,

Here is some additional data:

I used the following definitions:
compactable ratio = freeable_pages / total_pages memory reclaimed = pages_freed * PAGE_SIZE

freeable_pages is the estimate before compaction, based on the same calculation as zs_shrinker_count(), while pages_freed is the actual number of backing pages released.

There were 264 callbacks in the trace:
callback elapsed time:

    median:    5.77 ms
    p95:      55.82 ms
    maximum: 271.36 ms

compactable ratio before compaction:

    median:   0.32%
    p95:      2.86%
    maximum:  8.33%

memory reclaimed per callback:

    median:    3.80 MiB
    p95:      30.45 MiB
    maximum:  92.73 MiB

The longest callback took 271.36 ms. Its compactable ratio was 3.25%, and it released 7,650 pages, or about 29.88 MiB.

Quoted 8 lines

There was also a 241.91 ms callback (with 30 schedule-outs) with a compactable ratio of 0.44%. It released 1,019 pages, or about 3.98 MiB.

Based on this data, it seems better to remove the shrinker.

Would you prefer that I change v2 to remove the zsmalloc shrinker callbacks directly?

Quoted 7 lines

Let's bring in heavy artillery to this discussion, in addition to Andrew and Minchan, adding Nhat, Yosry, Barry, Johannes, Brian (random order).

Folks, I'm bullish on removal of zsmalloc shrinker callbacks. I don't think those buy us much apart from memcpy-s and lock contention. Systems that want to compact zsmalloc have a sysfs knob (and API) to do so (based on zram mm_stat numbers).

Hmm we'd need to collect more data in our workloads to determine, but if we can compute compactable ratio (freeable_pages / total_pages) on a per size class basis, can we just skip the size class whose ratio is too bad? Would that at least cut down on the vast majority of fruitless memcpys and lock acquisitions etc?

Quoted 4 lines

I'm always a bit hesitant to over-rely on userspace, especially when kernel has information to do something smart about it. It might not react in time, and many proactive reclaiming schemes back off under heavy memory pressure.

Quoted 5 lines

Apologies for the delay.

The thing about shrinker compaction is that it's sort of on the wrong end. It doesn't trigger compaction when memory is plentiful, no matter the internal fragmentation rate, and start firing compaction

Makes sense to me. The fragmentation ratio alone doesn't determine the latency impact, that also depends on how muchdata compaction needs to move and whether it overlaps with foreground zram activity.

Signature · 2 lines
Thanks,
Xueyuan
#16 lore raw
In reply to Nhat Pham
Quoted 5 lines · Nhat Pham

Hmm we'd need to collect more data in our workloads to determine, but if we can compute compactable ratio (freeable_pages / total_pages) on a per size class basis, can we just skip the size class whose ratio is too bad? Would that at least cut down on the vast majority of fruitless memcpys and lock acquisitions etc?

To calculate those we need to tool pool/class locks. But technically possible. Not sure, how would be determine the threshold tho.

On the other alternative is quite simple. We can make it "default ZSWAP", for example.

---

mm/Kconfig +11 -0
index afff1b25b543..72542946bc53 100644
@@ -157,6 +157,17 @@ config ZSMALLOC_CHAIN_SIZE
 
 	  For more information, see zsmalloc documentation.
 
+config ZSMALLOC_SHRINKER
+	bool "Enable zsmalloc pool shrinker"
+	default y
+	help
+	  This option enables the zsmalloc pool shrinker, which allows
+	  the zsmalloc pool to be shrunk when the system is under memory
+	  pressure. The shrinker will attempt to free up memory by
+	  compacting zspages and releasing empty zspages back to the system.
+
+	  If unsure, say N.
+
 endmenu
 
 menu "Slab allocator options"
mm/zsmalloc.c +9 -2
index 7ef80e0da626..e0d7ab9142bb 100644
@@ -276,8 +276,10 @@ struct zs_pool {
 
 	struct zs_pool_stats stats;
 
+#ifdef CONFIG_ZSMALLOC_SHRINKER
 	/* Compact classes */
 	struct shrinker *shrinker;
+#endif
 
 #ifdef CONFIG_ZSMALLOC_STAT
 	struct dentry *stat_dentry;
@@ -2050,8 +2052,9 @@ void zs_pool_stats(struct zs_pool *pool, struct zs_pool_stats *stats)
 }
 EXPORT_SYMBOL_GPL(zs_pool_stats);
 
+#ifdef CONFIG_ZSMALLOC_SHRINKER
 static unsigned long zs_shrinker_scan(struct shrinker *shrinker,
-		struct shrink_control *sc)
+				      struct shrink_control *sc)
 {
 	unsigned long pages_freed;
 	struct zs_pool *pool = shrinker->private_data;
@@ -2067,7 +2070,7 @@ static unsigned long zs_shrinker_scan(struct shrinker *shrinker,
 }
 
 static unsigned long zs_shrinker_count(struct shrinker *shrinker,
-		struct shrink_control *sc)
+				       struct shrink_control *sc)
 {
 	int i;
 	struct size_class *class;
@@ -2105,6 +2108,10 @@ static int zs_register_shrinker(struct zs_pool *pool)
 
 	return 0;
 }
+#else
+static void zs_unregister_shrinker(struct zs_pool *pool) { }
+static int zs_register_shrinker(struct zs_pool *pool) { return 0; }
+#endif
 
 static int calculate_zspage_chain_size(int class_size)
 {
#17 lore raw

Adding Chris for visibility

Quoted 6 lines · Sergey Senozhatsky
Quoted 5 lines · Nhat Pham

Hmm we'd need to collect more data in our workloads to determine, but if we can compute compactable ratio (freeable_pages / total_pages) on a per size class basis, can we just skip the size class whose ratio is too bad? Would that at least cut down on the vast majority of fruitless memcpys and lock acquisitions etc?

[..]

Quoted 70 lines

---

mm/Kconfig +11 -0
index afff1b25b543..72542946bc53 100644
@@ -157,6 +157,17 @@ config ZSMALLOC_CHAIN_SIZE
 
 	  For more information, see zsmalloc documentation.
 
+config ZSMALLOC_SHRINKER
+	bool "Enable zsmalloc pool shrinker"
+	default y
+	help
+	  This option enables the zsmalloc pool shrinker, which allows
+	  the zsmalloc pool to be shrunk when the system is under memory
+	  pressure. The shrinker will attempt to free up memory by
+	  compacting zspages and releasing empty zspages back to the system.
+
+	  If unsure, say N.
+
 endmenu
 
 menu "Slab allocator options"
mm/zsmalloc.c +9 -2
index 7ef80e0da626..e0d7ab9142bb 100644
@@ -276,8 +276,10 @@ struct zs_pool {
 
 	struct zs_pool_stats stats;
 
+#ifdef CONFIG_ZSMALLOC_SHRINKER
 	/* Compact classes */
 	struct shrinker *shrinker;
+#endif
 
 #ifdef CONFIG_ZSMALLOC_STAT
 	struct dentry *stat_dentry;
@@ -2050,8 +2052,9 @@ void zs_pool_stats(struct zs_pool *pool, struct zs_pool_stats *stats)
 }
 EXPORT_SYMBOL_GPL(zs_pool_stats);
 
+#ifdef CONFIG_ZSMALLOC_SHRINKER
 static unsigned long zs_shrinker_scan(struct shrinker *shrinker,
-		struct shrink_control *sc)
+				      struct shrink_control *sc)
 {
 	unsigned long pages_freed;
 	struct zs_pool *pool = shrinker->private_data;
@@ -2067,7 +2070,7 @@ static unsigned long zs_shrinker_scan(struct shrinker *shrinker,
 }
 
 static unsigned long zs_shrinker_count(struct shrinker *shrinker,
-		struct shrink_control *sc)
+				       struct shrink_control *sc)
 {
 	int i;
 	struct size_class *class;
@@ -2105,6 +2108,10 @@ static int zs_register_shrinker(struct zs_pool *pool)
 
 	return 0;
 }
+#else
+static void zs_unregister_shrinker(struct zs_pool *pool) { }
+static int zs_register_shrinker(struct zs_pool *pool) { return 0; }
+#endif
 
 static int calculate_zspage_chain_size(int class_size)
 {
#18 lore raw
Quoted 1 line · Sergey Senozhatsky

Adding Chris for visibility

Thanks for looping me in, Sergey.

I tested your CONFIG_ZSMALLOC_SHRINKER=n patch in ARM64 QEMU (7.3.0+, 16 boots per arm) by fragmenting zram0, triggering a manual compaction pass at pressure onset via /sys/block/zram0/compact, driving ~240 MB of swap-outs under memory pressure, and running a post-swap manual compaction pass.

Your theory holds up: with the shrinker enabled, the kernel compacted 2,218 pages (~9 MB) in the middle of the swap-out, only for incoming swap-outs to immediately reuse those pages. With CONFIG_ZSMALLOC_SHRINKER=n, mid-swap compaction dropped to 0 and zram pool memory was identical both during the swap-out (120.9 MB vs 121.0 MB) and after manual compaction (110.9 MB vs 111.2 MB).

I also tested adding the unlocked zs_can_compact() pre-check in __zs_compact() on top of CONFIG_ZSMALLOC_SHRINKER=n, which cuts each userspace /sys/block/zram0/compact scan from 97 us to 27 us:

Shrinker ON Shrinker OFF Shrinker OFF + pre-check
-------------------------------------------------------------------------------------
Pages compacted mid-swap 2,218 0 0

  zram mem_used_total (mid-swap)     121.0 MB       120.9 MB                   121.1 MB
  zram mem_used_total (post-compact) 111.2 MB       110.9 MB                   111.1 MB

/sys/block/zram0/compact scan time 96 us 97 us 27 us

Representative /sys/block/zram0/mm_stat snapshots:

BASE (CONFIG_ZSMALLOC_SHRINKER=y)

  pre_onset:         184553472  58747279   82350080  0   82374656  0      0   0   0
  onset_compact:     184553472  58747279   60456960  0   82374656  0   5345   0   0
  after_swap_out:    334196736 107289761  120946688  0  120946688  0   7570  29  29
  post_compact:      334749696 107472834  110907392  0  120946688  0  10016  29  29

PATCHED (CONFIG_ZSMALLOC_SHRINKER=n)

  pre_onset:         184553472  58747279   82350080  0   82374656  0      0   0   0
  onset_compact:     184553472  58747279   60456960  0   82374656  0   5345   0   0
  after_swap_out:    334372864 107347183  120922112  0  120922112  0   5345  29  29
  post_compact:      334995456 107550158  111009792  0  120922112  0   7767  29  29
#19 lore raw
Quoted 19 lines · Chris Wiseman
Quoted 1 line · Sergey Senozhatsky

Adding Chris for visibility

Thanks for looping me in, Sergey.

I tested your CONFIG_ZSMALLOC_SHRINKER=n patch in ARM64 QEMU (7.3.0+, 16 boots per arm) by fragmenting zram0, triggering a manual compaction pass at pressure onset via /sys/block/zram0/compact, driving ~240 MB of swap-outs under memory pressure, and running a post-swap manual compaction pass.

Your theory holds up: with the shrinker enabled, the kernel compacted 2,218 pages (~9 MB) in the middle of the swap-out, only for incoming swap-outs to immediately reuse those pages. With CONFIG_ZSMALLOC_SHRINKER=n, mid-swap compaction dropped to 0 and zram pool memory was identical both during the swap-out (120.9 MB vs 121.0 MB) and after manual compaction (110.9 MB vs 111.2 MB).

Quoted 3 lines

I also tested adding the unlocked zs_can_compact() pre-check in __zs_compact() on top of CONFIG_ZSMALLOC_SHRINKER=n, which cuts each userspace /sys/block/zram0/compact scan from 97 us to 27 us:

Is this precheck basically checking if we can free at least one zspage of the size class before proceeding? Yeah the number looks good - let's add that, in parallel!

Quoted 16 lines

Shrinker ON Shrinker OFF Shrinker OFF + pre-check
-------------------------------------------------------------------------------------
Pages compacted mid-swap 2,218 0 0

  zram mem_used_total (mid-swap)     121.0 MB       120.9 MB                   121.1 MB
  zram mem_used_total (post-compact) 111.2 MB       110.9 MB                   111.1 MB

/sys/block/zram0/compact scan time 96 us 97 us 27 us

Representative /sys/block/zram0/mm_stat snapshots:

BASE (CONFIG_ZSMALLOC_SHRINKER=y)

  pre_onset:         184553472  58747279   82350080  0   82374656  0      0   0   0
  onset_compact:     184553472  58747279   60456960  0   82374656  0   5345   0   0
  after_swap_out:    334196736 107289761  120946688  0  120946688  0   7570  29  29
  post_compact:      334749696 107472834  110907392  0  120946688  0  10016  29  29
Quoted 5 lines

PATCHED (CONFIG_ZSMALLOC_SHRINKER=n)

  pre_onset:         184553472  58747279   82350080  0   82374656  0      0   0   0
  onset_compact:     184553472  58747279   60456960  0   82374656  0   5345   0   0
  after_swap_out:    334372864 107347183  120922112  0  120922112  0   5345  29  29
  post_compact:      334995456 107550158  111009792  0  120922112  0   7767  29  29

Ah ok hmm. Sounds like internal compaction isn't that helpful, i.e zsmalloc already does a pretty decent job at packing objects that there aren't a terrible amount of internal fragmentation in the first place...?

Should we do CONFIG_ZSMALLOC_SHRINKER (default to N or to CONFIG_ZSWAP) for now, and remove it in a couple of releases if users in the wild do not complain?

#20 lore raw
Quoted 4 lines · Nhat Pham

Is this precheck basically checking if we can free at least one zspage of the size class before proceeding? Yeah the number looks good - let's add that, in parallel!

Yeah, it just checks if a bucket has enough wasted slots to free at least one full page before taking the locks. Sergey pointed out those two counters aren't atomic together so an unlocked check is just a best-effort hint - if you and Sergey are good with that, happy to send a patch.

Quoted 7 lines

Ah ok hmm. Sounds like internal compaction isn't that helpful, i.e zsmalloc already does a pretty decent job at packing objects that there aren't a terrible amount of internal fragmentation in the first place...?
Should we do CONFIG_ZSMALLOC_SHRINKER (default to N or to CONFIG_ZSWAP) for now, and remove it in a couple of releases if users in the wild do not complain?

Compaction still helps when we trigger it between bursts (freed ~21 MB before the swap storm and ~10 MB after). It just wastes time running mid-swap when new writes are filling those open slots right back up. Defaulting CONFIG_ZSMALLOC_SHRINKER to N (or CONFIG_ZSWAP) for a couple releases makes sense to me.

#21 lore raw
Quoted 9 lines · Chris Wiseman
Quoted 4 lines · Nhat Pham

Is this precheck basically checking if we can free at least one zspage of the size class before proceeding? Yeah the number looks good - let's add that, in parallel!

Yeah, it just checks if a bucket has enough wasted slots to free at least one full page before taking the locks. Sergey pointed out those

You meant full zspage here right? IIUC we keep the entire chain of zpages around until the entire zspage is gone?

Quoted 3 lines

two counters aren't atomic together so an unlocked check is just a best-effort hint - if you and Sergey are good with that, happy to send a patch.

Personally I'm happy with it being a best-effort hint!

We can also recheck after we acquire the lock (but I suspect it's unlikely we'll get radically different decision?)

Sergey, how do you feel about this?

Quoted 9 lines
Quoted 8 lines

Ah ok hmm. Sounds like internal compaction isn't that helpful, i.e zsmalloc already does a pretty decent job at packing objects that there aren't a terrible amount of internal fragmentation in the first place...?
Should we do CONFIG_ZSMALLOC_SHRINKER (default to N or to CONFIG_ZSWAP) for now, and remove it in a couple of releases if users in the wild do not complain?

Quoted 5 lines

Compaction still helps when we trigger it between bursts (freed ~21 MB before the swap storm and ~10 MB after). It just wastes time running mid-swap when new writes are filling those open slots right back up. Defaulting CONFIG_ZSMALLOC_SHRINKER to N (or CONFIG_ZSWAP) for a couple releases makes sense to me.

SGTM, but I'll let Sergey chime in!

#22 lore raw
Quoted 18 lines · Nhat Pham
Quoted 9 lines · Chris Wiseman
Quoted 4 lines · Nhat Pham

Is this precheck basically checking if we can free at least one zspage of the size class before proceeding? Yeah the number looks good - let's add that, in parallel!

Yeah, it just checks if a bucket has enough wasted slots to free at least one full page before taking the locks. Sergey pointed out those

You meant full zspage here right? IIUC we keep the entire chain of zpages around until the entire zspage is gone?

Quoted 3 lines

two counters aren't atomic together so an unlocked check is just a best-effort hint - if you and Sergey are good with that, happy to send a patch.

Quoted 22 lines

Personally I'm happy with it being a best-effort hint!

We can also recheck after we acquire the lock (but I suspect it's unlikely we'll get radically different decision?)

Sergey, how do you feel about this?

Quoted 14 lines
Quoted 8 lines

Ah ok hmm. Sounds like internal compaction isn't that helpful, i.e zsmalloc already does a pretty decent job at packing objects that there aren't a terrible amount of internal fragmentation in the first place...?
Should we do CONFIG_ZSMALLOC_SHRINKER (default to N or to CONFIG_ZSWAP) for now, and remove it in a couple of releases if users in the wild do not complain?

Compaction still helps when we trigger it between bursts (freed ~21 MB before the swap storm and ~10 MB after). It just wastes time running mid-swap when new writes are filling those open slots right back up. Defaulting CONFIG_ZSMALLOC_SHRINKER to N (or CONFIG_ZSWAP) for a couple releases makes sense to me.

Hi Nhat, Chris,

Quoted 1 line

SGTM, but I'll let Sergey chime in!

Thanks, Nhat and Chris, for the discussion.

If Sergey has no objections, I'd be happy to send a follow-up series to make the shrinker configurable and add the unlocked pre-check as Nhat suggested.

#23 lore raw

On (26/11/05 09:10), Xueyuan Chen wrote:
[..]

Quoted 7 lines
Quoted 1 line

SGTM, but I'll let Sergey chime in!

Thanks, Nhat and Chris, for the discussion.

If Sergey has no objections, I'd be happy to send a follow-up series to make the shrinker configurable and add the unlocked pre-check as Nhat suggested.

Thanks. Sounds like you and Chris will to have "Rock-Paper-Scissors" to determine who sends a oatch ;) Just joking, let's first figure out how exactly we want that zsmalloc-shrinker thing to behave wrt zswap expectations.

#24 lore raw
Quoted 7 lines · Sergey Senozhatsky

On (26/11/05 09:10), Xueyuan Chen wrote:
[..]

Quoted 4 lines
Quoted 1 line

SGTM, but I'll let Sergey chime in!

Thanks, Nhat and Chris, for the discussion.

Quoted 8 lines
Quoted 3 lines

If Sergey has no objections, I'd be happy to send a follow-up series to make the shrinker configurable and add the unlocked pre-check as Nhat suggested.

Thanks. Sounds like you and Chris will to have "Rock-Paper-Scissors" to determine who sends a oatch ;) Just joking, let's first figure out how exactly we want that zsmalloc-shrinker thing to behave wrt zswap expectations.

Haha, Xueyuan is welcome to send the patches!

Dumb question on the zswap side - aren't the zswap shrinker and the zsmalloc shrinker independent? Even if zswap's shrinker is turned off at runtime, normal swap-ins still free slots and leave fragmentation behind, and unlike zram, zswap doesn't have a manual sysfs compact knob. Would it be simpler to just keep CONFIG_ZSMALLOC_SHRINKER as its own standalone config rather than tying it to zswap's runtime knob?

#25 lore raw
Quoted 18 lines · Chris Wiseman
Quoted 7 lines · Sergey Senozhatsky

On (26/11/05 09:10), Xueyuan Chen wrote:
[..]

Quoted 4 lines
Quoted 1 line

SGTM, but I'll let Sergey chime in!

Thanks, Nhat and Chris, for the discussion.

Quoted 8 lines
Quoted 3 lines

If Sergey has no objections, I'd be happy to send a follow-up series to make the shrinker configurable and add the unlocked pre-check as Nhat suggested.

Thanks. Sounds like you and Chris will to have "Rock-Paper-Scissors" to determine who sends a oatch ;) Just joking, let's first figure out how exactly we want that zsmalloc-shrinker thing to behave wrt zswap expectations.

Quoted 8 lines

Haha, Xueyuan is welcome to send the patches!

Dumb question on the zswap side - aren't the zswap shrinker and the zsmalloc shrinker independent? Even if zswap's shrinker is turned off at runtime, normal swap-ins still free slots and leave fragmentation behind, and unlike zram, zswap doesn't have a manual sysfs compact knob. Would it be simpler to just keep CONFIG_ZSMALLOC_SHRINKER as its own standalone config rather than tying it to zswap's runtime knob?

Yeah those 2 shrinkers are independent! I think we should just let ZSMALLOC_SHRINKER default to the value of ZSWAP_DEFAULT_ON. zswap_load() (i.e zswap-in) by itself is enough to create fragmentation. The zswap shrinker is just another trigger (and it's also irrelevant if zswap is disabled anyway).

How do you feel about this, folks?

#26 lore raw
Quoted 28 lines · Nhat Pham
Quoted 18 lines · Chris Wiseman
Quoted 7 lines · Sergey Senozhatsky

On (26/11/05 09:10), Xueyuan Chen wrote:
[..]

Quoted 4 lines
Quoted 1 line

SGTM, but I'll let Sergey chime in!

Thanks, Nhat and Chris, for the discussion.

Quoted 8 lines
Quoted 3 lines

If Sergey has no objections, I'd be happy to send a follow-up series to make the shrinker configurable and add the unlocked pre-check as Nhat suggested.

Thanks. Sounds like you and Chris will to have "Rock-Paper-Scissors" to determine who sends a oatch ;) Just joking, let's first figure out how exactly we want that zsmalloc-shrinker thing to behave wrt zswap expectations.

Quoted 8 lines

Haha, Xueyuan is welcome to send the patches!

Dumb question on the zswap side - aren't the zswap shrinker and the zsmalloc shrinker independent? Even if zswap's shrinker is turned off at runtime, normal swap-ins still free slots and leave fragmentation behind, and unlike zram, zswap doesn't have a manual sysfs compact knob. Would it be simpler to just keep CONFIG_ZSMALLOC_SHRINKER as its own standalone config rather than tying it to zswap's runtime knob?

Quoted 5 lines

Yeah those 2 shrinkers are independent! I think we should just let ZSMALLOC_SHRINKER default to the value of ZSWAP_DEFAULT_ON. zswap_load() (i.e zswap-in) by itself is enough to create fragmentation. The zswap shrinker is just another trigger (and it's also irrelevant if zswap is disabled anyway).

Do we have any numbers for the zsmalloc shrinker with zswap? If it doesn't help much there either, do we need to keep it behind a config option?

Quoted 4 lines

How do you feel about this, folks?

#27 lore raw
In reply to Nhat Pham
Quoted 29 lines · Nhat Pham
Quoted 18 lines · Chris Wiseman
Quoted 7 lines · Sergey Senozhatsky

On (26/11/05 09:10), Xueyuan Chen wrote:
[..]

Quoted 4 lines
Quoted 1 line

SGTM, but I'll let Sergey chime in!

Thanks, Nhat and Chris, for the discussion.

Quoted 8 lines
Quoted 3 lines

If Sergey has no objections, I'd be happy to send a follow-up series to make the shrinker configurable and add the unlocked pre-check as Nhat suggested.

Thanks. Sounds like you and Chris will to have "Rock-Paper-Scissors" to determine who sends a oatch ;) Just joking, let's first figure out how exactly we want that zsmalloc-shrinker thing to behave wrt zswap expectations.

Quoted 8 lines

Haha, Xueyuan is welcome to send the patches!

Dumb question on the zswap side - aren't the zswap shrinker and the zsmalloc shrinker independent? Even if zswap's shrinker is turned off at runtime, normal swap-ins still free slots and leave fragmentation behind, and unlike zram, zswap doesn't have a manual sysfs compact knob. Would it be simpler to just keep CONFIG_ZSMALLOC_SHRINKER as its own standalone config rather than tying it to zswap's runtime knob?

Quoted 5 lines

Yeah those 2 shrinkers are independent! I think we should just let ZSMALLOC_SHRINKER default to the value of ZSWAP_DEFAULT_ON. zswap_load() (i.e zswap-in) by itself is enough to create fragmentation. The zswap shrinker is just another trigger (and it's also irrelevant if zswap is disabled anyway).

If zram doesn't benefit from zsmalloc shrinker, especially during reclaim, I wonder how much zswap benefits. AFAICT, it was hooked up before zsmalloc was hooked into compaction, and compaction seems like the right place to do this although it technically is freeing pages from the page allocator's perspective.

I wonder if this is something we can collect some data about, and maybe kill it off instead of putting off behind a config option. If hooking it into compaction is not enough, maybe we should add a similar interface in zswap to do proactive pool compaction from userspace, or maybe move the interface to zsmalloc itself.

Quoted 2 lines

How do you feel about this, folks?

#28 lore raw
Quoted 43 lines · Yosry Ahmed
Quoted 29 lines · Nhat Pham
Quoted 18 lines · Chris Wiseman
Quoted 7 lines · Sergey Senozhatsky

On (26/11/05 09:10), Xueyuan Chen wrote:
[..]

Quoted 4 lines
Quoted 1 line

SGTM, but I'll let Sergey chime in!

Thanks, Nhat and Chris, for the discussion.

Quoted 8 lines
Quoted 3 lines

If Sergey has no objections, I'd be happy to send a follow-up series to make the shrinker configurable and add the unlocked pre-check as Nhat suggested.

Thanks. Sounds like you and Chris will to have "Rock-Paper-Scissors" to determine who sends a oatch ;) Just joking, let's first figure out how exactly we want that zsmalloc-shrinker thing to behave wrt zswap expectations.

Quoted 8 lines

Haha, Xueyuan is welcome to send the patches!

Dumb question on the zswap side - aren't the zswap shrinker and the zsmalloc shrinker independent? Even if zswap's shrinker is turned off at runtime, normal swap-ins still free slots and leave fragmentation behind, and unlike zram, zswap doesn't have a manual sysfs compact knob. Would it be simpler to just keep CONFIG_ZSMALLOC_SHRINKER as its own standalone config rather than tying it to zswap's runtime knob?

Quoted 5 lines

Yeah those 2 shrinkers are independent! I think we should just let ZSMALLOC_SHRINKER default to the value of ZSWAP_DEFAULT_ON. zswap_load() (i.e zswap-in) by itself is enough to create fragmentation. The zswap shrinker is just another trigger (and it's also irrelevant if zswap is disabled anyway).

If zram doesn't benefit from zsmalloc shrinker, especially during reclaim, I wonder how much zswap benefits. AFAICT, it was hooked up before zsmalloc was hooked into compaction, and compaction seems like the right place to do this although it technically is freeing pages from the page allocator's perspective.

Quoted 5 lines

I wonder if this is something we can collect some data about, and maybe kill it off instead of putting off behind a config option. If hooking it into compaction is not enough, maybe we should add a similar interface in zswap to do proactive pool compaction from userspace, or maybe move the interface to zsmalloc itself.

Never mind the compaction argument, this is just to move zsmalloc pages around, it doesn't help with internal fragmentation. -ENOCOFFEE.

I still think we should check if the zsmalloc shrinker actually helps zswap or not tho!

#29 lore raw
Quoted 6 lines · Yosry Ahmed

Never mind the compaction argument, this is just to move zsmalloc pages around, it doesn't help with internal fragmentation. -ENOCOFFEE.

I still think we should check if the zsmalloc shrinker actually helps zswap or not tho!

I ran a quick test with zswap turned on to check.

Yosry is right - as long as new pages are swapping out, zswap naturally refills the freed slots on its own. Both runs ended up at the exact same 95 MB pool size, and turning the zsmalloc shrinker off cut page fault latency by 26% because reclaim isn't fighting itself for the same locks.

The only time the shrinker actually saved memory in zswap (~20.8 MB) was when I blocked new swap-outs completely after punching holes, since zswap doesn't have a manual sysfs compact knob like zram.

ARM64 QEMU (Linux 7.3.0+, 4 vCPUs, zswap lz4):
Shrinker ON Shrinker OFF

Active swap-out pool size:      95.9 MB         95.1 MB
Active swap-out p50 fault:     3,262 us        2,414 us  (-26%)
Idle pool (swap-out blocked):   44.2 MB         65.1 MB  (+20.8 MB)

Maybe we go with Nhat's idea (CONFIG_ZSMALLOC_SHRINKER defaulting to ZSWAP_DEFAULT_ON) for now, and if zswap adds a proactive compact knob later we can just kill the shrinker?

#30 lore raw
Quoted 23 lines · Chris Wiseman
Quoted 6 lines · Yosry Ahmed

Never mind the compaction argument, this is just to move zsmalloc pages around, it doesn't help with internal fragmentation. -ENOCOFFEE.

I still think we should check if the zsmalloc shrinker actually helps zswap or not tho!

I ran a quick test with zswap turned on to check.

Yosry is right - as long as new pages are swapping out, zswap naturally refills the freed slots on its own. Both runs ended up at the exact same 95 MB pool size, and turning the zsmalloc shrinker off cut page fault latency by 26% because reclaim isn't fighting itself for the same locks.

The only time the shrinker actually saved memory in zswap (~20.8 MB) was when I blocked new swap-outs completely after punching holes, since zswap doesn't have a manual sysfs compact knob like zram.

ARM64 QEMU (Linux 7.3.0+, 4 vCPUs, zswap lz4):
Shrinker ON Shrinker OFF

Quoted 7 lines
Active swap-out pool size:      95.9 MB         95.1 MB
Active swap-out p50 fault:     3,262 us        2,414 us  (-26%)
Idle pool (swap-out blocked):   44.2 MB         65.1 MB  (+20.8 MB)

Maybe we go with Nhat's idea (CONFIG_ZSMALLOC_SHRINKER defaulting to ZSWAP_DEFAULT_ON) for now, and if zswap adds a proactive compact knob later we can just kill the shrinker?

Unless we lose something by dropping the shrinker that we can only get back through proactive compaction, I would argue that adding the proactive compaction knob (in zswap or zsmalloc) is orthogonal. If we can establish that zsmalloc compaction during reclaim (via the shrinker) is useless/harmful, that should be enough to drop it. Adding a way to invoke proactive compaction should be a separate feature judged based on its merits.

#31 lore raw
In reply to Chris Wiseman

On (26/11/05 16:53), Chris Wiseman wrote:
[..]

Quoted 3 lines

Maybe we go with Nhat's idea (CONFIG_ZSMALLOC_SHRINKER defaulting to ZSWAP_DEFAULT_ON) for now, and if zswap adds a proactive compact knob later we can just kill the shrinker?

Sounds good to me (on the zram side).

#32 lore raw
Quoted 8 lines · Sergey Senozhatsky

On (26/11/05 16:53), Chris Wiseman wrote:
[..]

Quoted 3 lines

Maybe we go with Nhat's idea (CONFIG_ZSMALLOC_SHRINKER defaulting to ZSWAP_DEFAULT_ON) for now, and if zswap adds a proactive compact knob later we can just kill the shrinker?

Sounds good to me (on the zram side).

Thanks Sergey. Xueyuan, are you still planning to send the 2-patch series (CONFIG_ZSMALLOC_SHRINKER + unlocked zs_can_compact() pre-check), or want me to spin and post it?

Yosry / Nhat - any objection to starting with CONFIG_ZSMALLOC_SHRINKER defaulting to ZSWAP_DEFAULT_ON so zram users can turn it off right away while we decide whether to drop the shrinker completely?

#33 lore raw
Quoted 11 lines · Chris Wiseman
Quoted 8 lines · Sergey Senozhatsky

On (26/11/05 16:53), Chris Wiseman wrote:
[..]

Quoted 3 lines

Maybe we go with Nhat's idea (CONFIG_ZSMALLOC_SHRINKER defaulting to ZSWAP_DEFAULT_ON) for now, and if zswap adds a proactive compact knob later we can just kill the shrinker?

Sounds good to me (on the zram side).

Quoted 7 lines

Thanks Sergey. Xueyuan, are you still planning to send the 2-patch series (CONFIG_ZSMALLOC_SHRINKER + unlocked zs_can_compact() pre-check), or want me to spin and post it?

Yosry / Nhat - any objection to starting with CONFIG_ZSMALLOC_SHRINKER defaulting to ZSWAP_DEFAULT_ON so zram users can turn it off right away while we decide whether to drop the shrinker completely?

I would rather drop it right away, but I am fine if you don't wanna bother with zswap.

#34 lore raw
Quoted 3 lines · Yosry Ahmed

I would rather drop it right away, but I am fine if you don't wanna bother with zswap.

Works for me - dropping it outright is way cleaner than adding a Kconfig knob nobody wants to maintain.

Nhat / Johannes / Sergey - good with a 2-patch series that just removes the zsmalloc shrinker and adds the unlocked zs_can_compact() pre-check for manual zram compaction? If Xueyuan doesn't grab it I'm happy to post it.

#35 lore raw
Quoted 5 lines · Chris Wiseman
Quoted 3 lines · Yosry Ahmed

I would rather drop it right away, but I am fine if you don't wanna bother with zswap.

Quoted 9 lines

Works for me - dropping it outright is way cleaner than adding a Kconfig knob nobody wants to maintain.

Nhat / Johannes / Sergey - good with a 2-patch series that just removes the zsmalloc shrinker and adds the unlocked zs_can_compact() pre-check for manual zram compaction? If Xueyuan doesn't grab it I'm happy to post it.

:) Thanks, Chris. Feel free to take over the series.

#36 lore raw
In reply to Chris Wiseman
Quoted 3 lines · Chris Wiseman

Nhat / Johannes / Sergey - good with a 2-patch series that just removes the zsmalloc shrinker and adds the unlocked zs_can_compact() pre-check for manual zram compaction?

If there is no shrinker now then does unlocked zs_can_compact() matter that much? I assume zs_can_compact() is now called only from external code, so it's not expected to be executed nearly as often as before, or is there some other reason to make it unlocked?

#37 lore raw
Quoted 10 lines · Sergey Senozhatsky
Quoted 3 lines · Chris Wiseman

Nhat / Johannes / Sergey - good with a 2-patch series that just removes the zsmalloc shrinker and adds the unlocked zs_can_compact() pre-check for manual zram compaction?

If there is no shrinker now then does unlocked zs_can_compact() matter that much? I assume zs_can_compact() is now called only from external code, so it's not expected to be executed nearly as often as before, or is there some other reason to make it unlocked?

Ha, fair enough - started this whole thread trying to shave 70 us off the shrinker and somehow talked ourselves into deleting the whole thing. Definitely not worth adding lockless reads just for the occasional manual zram compact.

Happy to drop the pre-check and just send a single patch removing the zsmalloc shrinker once Nhat and Johannes are good with it.

#38 lore raw
In reply to Sergey Senozhatsky
Quoted 6 lines · Sergey Senozhatsky

On (26/11/05 09:10), Xueyuan Chen wrote:
[..]

Quoted 4 lines
Quoted 1 line

SGTM, but I'll let Sergey chime in!

Thanks, Nhat and Chris, for the discussion.

Quoted 9 lines
Quoted 3 lines

If Sergey has no objections, I'd be happy to send a follow-up series to make the shrinker configurable and add the unlocked pre-check as Nhat suggested.

Thanks. Sounds like you and Chris will to have "Rock-Paper-Scissors" to determine who sends a oatch ;) Just joking, let's first figure out how exactly we want that zsmalloc-shrinker thing to behave wrt zswap expectations.

Haha :) Either works for me.

#39 lore raw
In reply to Nhat Pham
Quoted 8 lines · Nhat Pham
Quoted 7 lines · Chris Wiseman
Quoted 4 lines · Nhat Pham

Is this precheck basically checking if we can free at least one zspage of the size class before proceeding? Yeah the number looks good - let's add that, in parallel!

Quoted 5 lines
Quoted 2 lines

Yeah, it just checks if a bucket has enough wasted slots to free at least one full page before taking the locks. Sergey pointed out those

You meant full zspage here right? IIUC we keep the entire chain of zpages around until the entire zspage is gone?

Full zspage page, right.

Quoted 10 lines
Quoted 3 lines

two counters aren't atomic together so an unlocked check is just a best-effort hint - if you and Sergey are good with that, happy to send a patch.

Personally I'm happy with it being a best-effort hint!

We can also recheck after we acquire the lock (but I suspect it's unlikely we'll get radically different decision?)

Sergey, how do you feel about this?

Well, I guess... I don't mind the lockless best effort hint.

I suppose another way could be an introduction of ->can_compact bool flag, which we can set under the lock in zs_free() (that's where fragmentation is created). zs_can_compact() doesn't seem too expensive, but calling it on every zs_free() looks a bit bad, we sort of want to do less work in zs_free().

Quoted 9 lines
Quoted 9 lines
Quoted 8 lines

Ah ok hmm. Sounds like internal compaction isn't that helpful, i.e zsmalloc already does a pretty decent job at packing objects that there aren't a terrible amount of internal fragmentation in the first place...?
Should we do CONFIG_ZSMALLOC_SHRINKER (default to N or to CONFIG_ZSWAP) for now, and remove it in a couple of releases if users in the wild do not complain?

Quoted 7 lines
Quoted 5 lines

Compaction still helps when we trigger it between bursts (freed ~21 MB before the swap storm and ~10 MB after). It just wastes time running mid-swap when new writes are filling those open slots right back up. Defaulting CONFIG_ZSMALLOC_SHRINKER to N (or CONFIG_ZSWAP) for a couple releases makes sense to me.

SGTM, but I'll let Sergey chime in!

So "depends on ZSWAP" looks a bit problematic, because zswap has both ZSWAP_SHRINKER_DEFAULT_ON (which can select CONFIG_ZSMALLOC_SHRINKER), but then zswap also has module_param_named(shrinker_enabled). So if want to tie zsmalloc to zswap shrinker then we probably need to wire in that run-time knob as well. If shrinker_enabled should imply zsmalloc shrinker (which I think it didn't so far). zswap folks, do you have any preferences?