zram: support asynchronous GC for lazy slot freeing
Subsystems (3)
RFC PATCH 1/1 zram: support asynchronous GC for lazy slot freeing
Swap freeing can be expensive when unmapping a VMA containing many swap entries. This has been reported to significantly delay memory reclamation during Android’s low-memory killing, especially when multiple processes are terminated to free memory, with slot_free() accounting for more than 80% of the total cost of freeing swap entries.
Two earlier attempts by Lei and Zhiguo added a new thread in the mm core to asynchronously collect and free swap entries [1][2], but the design itself is fairly complex.
When anon folios and swap entries are mixed within a process, reclaiming anon folios from killed processes helps return memory to the system as quickly as possible, so that newly launched applications can satisfy their memory demands. It is not ideal for swap freeing to block anon folio freeing. On the other hand, swap freeing can still return memory to the system, although at a slower rate due to memory compression.
Therefore, in zram, we introduce a GC worker to allow anon folio freeing and slot_free to run in parallel, since slot_free is performed asynchronously, maximizing the rate at which memory is returned to the system.
Xueyuan’s test on RK3588 shows that unmapping a 256MB swap-filled VMA becomes 3.4× faster when pinning tasks to CPU2, reducing the execution time from 63,102,982 ns to 18,570,726 ns.
A positive side effect is that async GC also slightly improves do_swap_page() performance, as it no longer has to wait for slot_free() to complete.
Xueyuan’s test shows that swapping in 256MB of data (each page filled with repeating patterns such as “1024 one”, “1024 two”, “1024 three”, and “1024 four”) reduces execution time from 1,358,133,886 ns to 1,104,315,986 ns, achieving a 1.22× speedup.
[1] https://lore.kernel.org/all/[email protected]/
[2] https://lore.kernel.org/all/[email protected]/
--- drivers/block/zram/zram_drv.c | 56 ++++++++++++++++++++++++++++++++++- drivers/block/zram/zram_drv.h | 3 ++ 2 files changed, 58 insertions(+), 1 deletion(-)
drivers/block/zram/zram_drv.c +55 -1
@@ -1958,6 +1958,23 @@ static ssize_t debug_stat_show(struct device *dev,return ret; } +static void gc_slots_free (struct zram *zram) +{ +size_t num_pages = zram->disksize >>PAGE_SHIFT ; +unsigned long index; + + index =find_next_bit (zram->gc_map, num_pages,0 ); +while (index < num_pages) { +if (slot_trylock (zram, index)) { +if (test_bit (index, zram->gc_map)) +slot_free (zram, index); +slot_unlock (zram, index); +cond_resched (); + } + index =find_next_bit (zram->gc_map, num_pages, index +1 ); + } +} +static void zram_meta_free (struct zram *zram,u64 disksize) {size_t num_pages = disksize >>PAGE_SHIFT ;
@@ -1966,6 +1983,9 @@ static void zram_meta_free(struct zram *zram, u64 disksize)if (!zram->table)return ; +flush_work (&zram->gc_work); +gc_slots_free (zram); +/* Free all pages that are still in this zram device */ for (index =0 ; index < num_pages; index++)slot_free (zram, index);
@@ -1973,6 +1993,14 @@ static void zram_meta_free(struct zram *zram, u64 disksize)zs_destroy_pool (zram->mem_pool);vfree (zram->table); zram->table =NULL ; +bitmap_free (zram->gc_map); +} + +static void zram_gc_work (struct work_struct *work) +{ + struct zram *zram = container_of(work, struct zram, gc_work); + +gc_slots_free (zram); }static bool zram_meta_alloc (struct zram *zram,u64 disksize)
@@ -1991,12 +2019,22 @@ static bool zram_meta_alloc(struct zram *zram, u64 disksize)return false; } + zram->gc_map =bitmap_zalloc (num_pages,GFP_KERNEL ); +if (!zram->gc_map) { +zs_destroy_pool (zram->mem_pool); +vfree (zram->table); + zram->table =NULL ; +return false; + } +if (!huge_class_size) huge_class_size =zs_huge_class_size (zram->mem_pool);for (index =0 ; index < num_pages; index++)slot_lock_init (zram, index); +INIT_WORK (&zram->gc_work, zram_gc_work); +return true; }
@@ -2008,6 +2046,8 @@ static void slot_free(struct zram *zram, u32 index) zram->table[index].attr.ac_time =0 ;#endif +if (test_and_clear_bit (index, zram->gc_map)) +atomic64_dec (&zram->stats.gc_slots);clear_slot_flag (zram, index,ZRAM_IDLE );clear_slot_flag (zram, index,ZRAM_INCOMPRESSIBLE );clear_slot_flag (zram, index,ZRAM_PP_SLOT );
@@ -2781,6 +2821,19 @@ static void zram_submit_bio(struct bio *bio) } } +static bool try_slot_lazy_free (struct zram *zram,unsigned long index) +{ +/* too many lazy-free slots, perform direct free */ +if (atomic64_read (&zram->stats.gc_slots) >30000 ) +return false; +if (test_and_set_bit (index, zram->gc_map)) +return false; +/* accumulated lazy-free slots, wake up GC worker */ +if (atomic64_inc_return (&zram->stats.gc_slots) >200 ) +queue_work (system_dfl_wq, &zram->gc_work); +return true; +} +static void zram_slot_free_notify (struct block_device *bdev,unsigned long index) {
@@ -2794,7 +2847,8 @@ static void zram_slot_free_notify(struct block_device *bdev,return ; } -slot_free (zram, index); +if (!try_slot_lazy_free (zram, index)) +slot_free (zram, index);slot_unlock (zram, index); }
drivers/block/zram/zram_drv.h +3 -0
@@ -88,6 +88,7 @@ struct zram_stats {atomic64_t pages_stored;/* no. of pages currently stored */ atomic_long_t max_used_pages;/* no. of maximum pages stored */ atomic64_t miss_free;/* no. of missed free */ +atomic64_t gc_slots;/* no. of queued for lazy free by gc */ #ifdef CONFIG_ZRAM_WRITEBACK atomic64_t bd_count;/* no. of pages in backing device */ atomic64_t bd_reads;/* no. of reads from backing device */
@@ -142,5 +143,7 @@ struct zram {#ifdef CONFIG_ZRAM_MEMORY_TRACKING struct dentry *debugfs_dir;#endif +unsigned long *gc_map; + struct work_struct gc_work; };#endif
Signature · 1 line
2.39.3 (Apple Git-146)
Quoted 38 lines · Barry Song (Xiaomi)
Swap freeing can be expensive when unmapping a VMA containing many swap entries. This has been reported to significantly delay memory reclamation during Android’s low-memory killing, especially when multiple processes are terminated to free memory, with
slot_free()accounting for more than 80% of the total cost of freeing swap entries.Two earlier attempts by Lei and Zhiguo added a new thread in the mm core to asynchronously collect and free swap entries [1][2], but the design itself is fairly complex.
When anon folios and swap entries are mixed within a process, reclaiming anon folios from killed processes helps return memory to the system as quickly as possible, so that newly launched applications can satisfy their memory demands. It is not ideal for swap freeing to block anon folio freeing. On the other hand, swap freeing can still return memory to the system, although at a slower rate due to memory compression.
Therefore, in zram, we introduce a GC worker to allow anon folio freeing and
slot_freeto run in parallel, sinceslot_freeis performed asynchronously, maximizing the rate at which memory is returned to the system.Xueyuan’s test on RK3588 shows that unmapping a 256MB swap-filled VMA becomes 3.4× faster when pinning tasks to CPU2, reducing the execution time from 63,102,982 ns to 18,570,726 ns.
A positive side effect is that async GC also slightly improves
do_swap_page()performance, as it no longer has to wait forslot_free()to complete.Xueyuan’s test shows that swapping in 256MB of data (each page filled with repeating patterns such as “1024 one”, “1024 two”, “1024 three”, and “1024 four”) reduces execution time from 1,358,133,886 ns to 1,104,315,986 ns, achieving a 1.22× speedup.
Quoted 5 lines
[1] https://lore.kernel.org/all/[email protected]/
[2] https://lore.kernel.org/all/[email protected]/Tested-by Xueyuan Chen <[email protected]>Signed-off-by Barry Song (Xiaomi) <[email protected]>
Hi Barry
This looks an interesting idea to me.
Quoted 20 lines
--- drivers/block/zram/zram_drv.c | 56 ++++++++++++++++++++++++++++++++++- drivers/block/zram/zram_drv.h | 3 ++ 2 files changed, 58 insertions(+), 1 deletion(-)drivers/block/zram/zram_drv.c +7
-0@@ -1958,6 +1958,23 @@ static ssize_t debug_stat_show(struct device *dev,return ret; } +static void gc_slots_free (struct zram *zram) +{ +size_t num_pages = zram->disksize >>PAGE_SHIFT ; +unsigned long index; + + index =find_next_bit (zram->gc_map, num_pages,0 ); +while (index < num_pages) {
drivers/block/zram/zram_drv.c static ssize_t debug_stat_show(struct device *dev, · line 1977
Quoted 10 lines
drivers/block/zram/zram_drv.c +10
-0+if (slot_trylock (zram, index)) { +if (test_bit (index, zram->gc_map)) +slot_free (zram, index); +slot_unlock (zram, index); +cond_resched (); + } + index =find_next_bit (zram->gc_map, num_pages, index +1 ); + } +} +
The ideas looks interesting but the implementation looks not that optimal to me. find_next_bit does a O(n) looks up for every gc call looks really expensive if the pending slot is at tail.
Perhaps a percpu stack can be used, something like the folio batch?
drivers/block/zram/zram_drv.c static void zram_slot_free_notify(struct block_device *bdev, · line 2851
Quoted 3 lines
drivers/block/zram/zram_drv.c +2
-1-slot_free (zram, index); +if (!try_slot_lazy_free (zram, index)) +slot_free (zram, index);
What is making this slot_free so costly? zs_free?
drivers/block/zram/zram_drv.h struct zram_stats { · line 91
Quoted 12 lines
drivers/block/zram/zram_drv.c +0
-0slot_unlock (zram, index); }drivers/block/zram/zram_drv.h +1
-0@@ -88,6 +88,7 @@ struct zram_stats {atomic64_t pages_stored;/* no. of pages currently stored */ atomic_long_t max_used_pages;/* no. of maximum pages stored */ atomic64_t miss_free;/* no. of missed free */ +atomic64_t gc_slots;/* no. of queued for lazy free by gc */
Maybe we want to track the size of content being delayed instead of slots number? I saw there is a 30000 hard limit for that.
Perhaps it will make more sense if we have a "buffer size" (e.g. 64M), seems more intuitive to me. e.g. the ZRAM module can occupy at most 64M of memory, so the delayed free won't cause a significant global pressure.
Also I think this patch is batching the memory free operations, so the workqueue or design can also be further optimized for batching, for example if the zs_free is the expensive part then maybe we shall just clear the handler for the freeing slot and leave the handler in a percpu stack, then batch free these handlers. zsmalloc might make use some batch optimization based on that too, something like kmem_cache_free_bulk but for zsmalloc?
if zs_free is not all the expensive part, I took a look at slot_free maybe a lot of read / write of slot data can be merged.
This patch currently doesn't reduce the total amount of work, but if above idea works, a lot of redundant operations might be be dropped, result in better performance in every case.
Just my two cents and ideas, not sure if I got everything correct. Looking forward for more disscussion on this :)
On Sun, Apr 12, 2026 at 07:48:48PM +0800, Kairui Song wrote:
[...]
Quoted 1 line
What is making this
slot_freeso costly?zs_free?
Yes, I've captured some perf data on RK3588 cpu2:
- 3.79% 0.42% zram [zram] [k] slot_free
- 89.04% slot_free
- 65.40% zs_free
+ 77.29% free_zspage
+ 21.75% kmem_cache_free
0.68% __kern_my_cpu_offset
+ 13.19% _raw_spin_unlock
+ 4.86% _raw_read_unlock
4.75% obj_free
+ 4.72% _raw_read_lock
3.64% fix_fullness_group
+ 2.02% _raw_spin_lock
+ 1.31% kmem_cache_free
It's clear that zs_free is the primary hotspot, accounting for ~65.40% of the total slot_free cycles. Beyond that, have some read and spin lock in slot_free.
--
Thanks,
Xueyuan
Sergey Senozhatsky
maintainerQuoted 20 lines · Xueyuan Chen
On Sun, Apr 12, 2026 at 07:48:48PM +0800, Kairui Song wrote:
[...]Quoted 1 line
What is making this
slot_freeso costly?zs_free?Yes, I've captured some perf data on RK3588 cpu2:
- 3.79% 0.42% zram [zram] [k]
slot_free
- 89.04%slot_free
- 65.40%zs_free
+ 77.29%free_zspage
+ 21.75%kmem_cache_free
0.68%__kern_my_cpu_offset
+ 13.19%_raw_spin_unlock
+ 4.86%_raw_read_unlock
4.75%obj_free
+ 4.72%_raw_read_lock
3.64%fix_fullness_group
+ 2.02%_raw_spin_lock
+ 1.31%kmem_cache_free
Quoted 3 lines
It's clear that
zs_freeis the primary hotspot, accounting for ~65.40% of the totalslot_freecycles. Beyond that, have some read and spin lock inslot_free.
Just a random thought, if zs_free() is costly then it likely also affects zswap, which makes me wonder if doing something on the zsmalloc side is a "batter" way forward.
Quoted 22 lines · Sergey Senozhatsky
Quoted 20 lines · Xueyuan Chen
On Sun, Apr 12, 2026 at 07:48:48PM +0800, Kairui Song wrote:
[...]Quoted 1 line
What is making this
slot_freeso costly?zs_free?Yes, I've captured some perf data on RK3588 cpu2:
- 3.79% 0.42% zram [zram] [k]
slot_free
- 89.04%slot_free
- 65.40%zs_free
+ 77.29%free_zspage
+ 21.75%kmem_cache_free
0.68%__kern_my_cpu_offset
+ 13.19%_raw_spin_unlock
+ 4.86%_raw_read_unlock
4.75%obj_free
+ 4.72%_raw_read_lock
3.64%fix_fullness_group
+ 2.02%_raw_spin_lock
+ 1.31%kmem_cache_free
Quoted 7 lines
Quoted 3 lines
It's clear that
zs_freeis the primary hotspot, accounting for ~65.40% of the totalslot_freecycles. Beyond that, have some read and spin lock inslot_free.Just a random thought, if
zs_free()is costly then it likely also affects zswap, which makes me wonder if doing something on the zsmalloc side is a "batter" way forward.
Xueyuan's perf shows that 65.4% of slot_free is spent in zs_free, so there is still around 35% elsewhere. However, this might be a measurement issue. If we confirm the number is >=90% or so, moving GC into zsmalloc seems like a better option. My real use case is zram rather than zswap in the Android industry, but this would benefit both zswap and zram.
Meanwhile, I would also like to try whether combining many bit operations, such as clear_slot_flag(), can further help.
Thanks
Barry
Quoted 76 lines · Kairui Song
Quoted 38 lines · Barry Song (Xiaomi)
Swap freeing can be expensive when unmapping a VMA containing many swap entries. This has been reported to significantly delay memory reclamation during Android’s low-memory killing, especially when multiple processes are terminated to free memory, with
slot_free()accounting for more than 80% of the total cost of freeing swap entries.Two earlier attempts by Lei and Zhiguo added a new thread in the mm core to asynchronously collect and free swap entries [1][2], but the design itself is fairly complex.
When anon folios and swap entries are mixed within a process, reclaiming anon folios from killed processes helps return memory to the system as quickly as possible, so that newly launched applications can satisfy their memory demands. It is not ideal for swap freeing to block anon folio freeing. On the other hand, swap freeing can still return memory to the system, although at a slower rate due to memory compression.
Therefore, in zram, we introduce a GC worker to allow anon folio freeing and
slot_freeto run in parallel, sinceslot_freeis performed asynchronously, maximizing the rate at which memory is returned to the system.Xueyuan’s test on RK3588 shows that unmapping a 256MB swap-filled VMA becomes 3.4× faster when pinning tasks to CPU2, reducing the execution time from 63,102,982 ns to 18,570,726 ns.
A positive side effect is that async GC also slightly improves
do_swap_page()performance, as it no longer has to wait forslot_free()to complete.Xueyuan’s test shows that swapping in 256MB of data (each page filled with repeating patterns such as “1024 one”, “1024 two”, “1024 three”, and “1024 four”) reduces execution time from 1,358,133,886 ns to 1,104,315,986 ns, achieving a 1.22× speedup.
Quoted 5 lines
[1] https://lore.kernel.org/all/[email protected]/
[2] https://lore.kernel.org/all/[email protected]/Tested-by Xueyuan Chen <[email protected]>Signed-off-by Barry Song (Xiaomi) <[email protected]>Hi Barry
This looks an interesting idea to me.
Quoted 26 lines
--- drivers/block/zram/zram_drv.c | 56 ++++++++++++++++++++++++++++++++++- drivers/block/zram/zram_drv.h | 3 ++ 2 files changed, 58 insertions(+), 1 deletion(-)drivers/block/zram/zram_drv.c +13
-0@@ -1958,6 +1958,23 @@ static ssize_t debug_stat_show(struct device *dev,return ret; } +static void gc_slots_free (struct zram *zram) +{ +size_t num_pages = zram->disksize >>PAGE_SHIFT ; +unsigned long index; + + index =find_next_bit (zram->gc_map, num_pages,0 ); +while (index < num_pages) { +if (slot_trylock (zram, index)) { +if (test_bit (index, zram->gc_map)) +slot_free (zram, index); +slot_unlock (zram, index); +cond_resched (); + }
Quoted 8 lines
drivers/block/zram/zram_drv.c static ssize_t debug_stat_show(struct device *dev, · line 1977
Quoted 4 lines
drivers/block/zram/zram_drv.c +4
-0+ index =find_next_bit (zram->gc_map, num_pages, index +1 ); + } +} +The ideas looks interesting but the implementation looks not that optimal to me.
find_next_bitdoes a O(n) looks up for every gc call looks really expensive if the pending slot is at tail.
Agreed. It’s essentially a prototype at this stage to demonstrate the idea.
Quoted 2 lines
Perhaps a percpu stack can be used, something like the folio batch?
I guess a major difference is that folio batching aims to reduce lruvec lock contention. Once a CPU’s slot space is empty, it batches draining folios into the lruvec by checking whether some slots can share the same lock. This procedure is synchronous within folio_batch_move_lru(this_cpu_ptr(fbatch), move_fn);
In our case, we might not want a synchronous procedure, so each CPU could launch its own workqueue. I’m not sure whether this is actually beneficial, as it might trigger the zsmalloc lock contention we are trying to eliminate.
If we end up wanting to drain all CPUs together, that would make things quite complex again.
So I guess a hierarchical bitmap, an XArray, or even a simple array could work. If we cap it at 64MB, the array would be at most 128KB on a PAGE_SIZE=4KB system.
I am CC’ing Wenchao, who may be interested in further measurements and also involved in a more efficient implementation.
Quoted 10 lines
drivers/block/zram/zram_drv.c static void zram_slot_free_notify(struct block_device *bdev, · line 2851
Quoted 4 lines
drivers/block/zram/zram_drv.c +2
-1-slot_free (zram, index); +if (!try_slot_lazy_free (zram, index)) +slot_free (zram, index);What is making this
slot_freeso costly?zs_free?Quoted 3 lines
drivers/block/zram/zram_drv.c +0
-0slot_unlock (zram, index); }
Quoted 12 lines
drivers/block/zram/zram_drv.h struct zram_stats { · line 91
Quoted 9 lines
drivers/block/zram/zram_drv.h +1
-0@@ -88,6 +88,7 @@ struct zram_stats {atomic64_t pages_stored;/* no. of pages currently stored */ atomic_long_t max_used_pages;/* no. of maximum pages stored */ atomic64_t miss_free;/* no. of missed free */ +atomic64_t gc_slots;/* no. of queued for lazy free by gc */ Maybe we want to track the size of content being delayed instead of slots number? I saw there is a 30000 hard limit for that.
Yep, definitely we want size, not number of pages, since PAGE_SIZE is not constant.
Quoted 6 lines
Perhaps it will make more sense if we have a "buffer size" (e.g. 64M), seems more intuitive to me. e.g. the ZRAM module can occupy at most 64M of memory, so the delayed free won't cause a significant global pressure.
Quoted 7 lines
Also I think this patch is batching the memory free operations, so the workqueue or design can also be further optimized for batching, for example if the
zs_freeis the expensive part then maybe we shall just clear the handler for the freeing slot and leave the handler in a percpu stack, then batch free these handlers. zsmalloc might make use some batch optimization based on that too, something likekmem_cache_free_bulkbut for zsmalloc?
I’m not really sure a per-CPU approach is the right direction, since zsmalloc already has a lot of contention we may want to eliminate. If we introduce per-CPU workqueues or similar mechanisms, we might end up increasing contention rather than reducing it.
A kmem_cache_free_bulk()-like approach might be a good direction to investigate for zsmalloc. I guess Xueyuan is also thinking about it? Right now, zsmalloc frequently takes and releases multiple locks for each individual.
Quoted 7 lines
if
zs_freeis not all the expensive part, I took a look atslot_freemaybe a lot of read / write of slot data can be merged.This patch currently doesn't reduce the total amount of work, but if above idea works, a lot of redundant operations might be be dropped, result in better performance in every case.
Yep, hopefully we can optimize for every case. Of course, that will take a lot of time :-)
Quoted 3 lines
Just my two cents and ideas, not sure if I got everything correct. Looking forward for more disscussion on this :)
Thanks for your suggestions—they are always welcome. We may discuss this further.
Best Regards
Barry