mm/zswap: shrink zswap_entry via a fixed pool index
Subsystems (2)
v4 cover mm/zswap: shrink zswap_entry via a fixed pool index
Every stored page has a struct zswap_entry, so its size is pure per-page overhead. On x86_64 it is currently 56 bytes, of which 8 bytes are a pointer to the owning zswap_pool.
Only a handful of pools are ever live: a new pool is created only when the compressor is (re)set, and pools are reused across compressor switches. That makes a per-entry pool pointer more expensive than it needs to be, and the RCU list that currently tracks pools is more machinery than this needs once each pool already has a stable slot.
This series:
1. Releases retired pools with queue_rcu_work() instead of synchronize_rcu() so the last put no longer blocks on an RCU grace period.
2. Replaces the zswap_pools list with a fixed ZSWAP_MAX_POOLS (16) array and a separate RCU-protected current-pool pointer, giving each pool a stable slot index. Slot 0 is left unused, and pool creation publishes the fully constructed pool directly into a free slot.
3. Stores that u8 slot index in each zswap_entry instead of the pool pointer. The u8 fits in padding after the bool referenced field, so the entry shrinks from 56 to 48 bytes on x86_64 (~2MiB of metadata saved per 1GiB of data held in zswap).
Runtime compressor switching is preserved, but the fixed array now bounds the number of simultaneously live distinct compressor pools to 15 (slot 0 is reserved). If all usable slots are full, creating a pool for another compressor fails and the compressor switch is rejected.
Benchmark (x86_64, compressor=lzo, MADV_PAGEOUT store + fault-in load):
- zswap_entry object_size: 56 -> 48 bytes
- e2e store+load median latency: no measurable regression vs baseline at matched stored_delta
The extra cost per store/free/decompress is one array-index load instead of a pointer dereference. With a single (or few) live pool(s) that does not show up against (de)compression.
Testing
=======
- sizeof_check: 56 -> 48 bytes on x86_64
- Boot with DEBUG_ATOMIC_SLEEP + lockdep/PROVE_RCU + KASAN:
zswap store/load and compressor switch (retire + reuse) pass
This series is based on akpm/mm-unstable as of 2026-08-30 (42d64d4fef83).
Changes since RFC v3:
- Retire pools with queue_rcu_work() instead of call_rcu().
- Queue the deferred RCU release work on system_percpu_wq.
- Use rcu_dereference_check() with entry->pool_idx != 0 in zswap_entry_pool() instead of rcu_dereference_protected().
- Simplify pool-slot publishing after confirming compressor parameter updates are serialized by the module parameter lock.
Jianyue Wu (3):
mm/zswap: release retired pools via queue_rcu_work() instead of
synchronize_rcu()
mm/zswap: replace the zswap_pools list with a fixed pools array mm/zswap: reference the pool by index to shrink struct zswap_entry
mm/zswap.c | 140 ++++++++++++++++++++++++++++++++++++++--------------- 1 file changed, 101 insertions(+), 39 deletions(-)
base-commit: 42d64d4fef83a241c919c8693fdf0a21b2cb6061
Signature · 1 line
2.43.0
RFC PATCH v4 1/3 mm/zswap: release retired pools via queue_rcu_work() instead of synchronize_rcu()
When a pool's last reference is dropped, __zswap_pool_empty() removes it from the pool list and schedules __zswap_pool_release(), which calls synchronize_rcu() to wait for readers before tearing the pool down.
synchronize_rcu() is a synchronous, potentially long wait. Replace it with queue_rcu_work(): __zswap_pool_empty() hands the pool to queue_rcu_work(), which waits for a grace period asynchronously and then runs __zswap_pool_release() from a worker for the sleepable teardown (__zswap_pool_empty() can run in atomic context and must not block). The grace-period guarantee is unchanged; the retirement path just no longer blocks on it.
--- mm/zswap.c | 12 +++++------- 1 file changed, 5 insertions(+), 7 deletions(-)
mm/zswap.c +5 -7
@@ -155,7 +155,7 @@ struct zswap_pool {struct crypto_acomp_ctx __percpu *acomp_ctx;struct percpu_ref ref;struct list_head list; -struct work_struct release_work; +struct rcu_work release_work;struct hlist_node node;char tfm_name[CRYPTO_MAX_ALG_NAME ]; };
@@ -379,10 +379,8 @@ static void zswap_pool_destroy(struct zswap_pool *pool)static void __zswap_pool_release (struct work_struct *work) { -struct zswap_pool *pool =container_of (work,typeof (*pool), - release_work); - -synchronize_rcu (); +struct zswap_pool *pool =container_of (to_rcu_work (work), +typeof (*pool), release_work);/* nobody should have been able to get a ref... */ WARN_ON (!percpu_ref_is_zero (&pool->ref));
@@ -406,8 +404,8 @@ static void __zswap_pool_empty(struct percpu_ref *ref)list_del_rcu (&pool->list); -INIT_WORK (&pool->release_work, __zswap_pool_release); -schedule_work (&pool->release_work); +INIT_RCU_WORK (&pool->release_work, __zswap_pool_release); +queue_rcu_work (system_percpu_wq, &pool->release_work);spin_unlock_bh (&zswap_pools_lock); }
Signature · 1 line
2.43.0
Re: [RFC PATCH v4 1/3] mm/zswap: release retired pools via queue_rcu_work() instead of synchronize_rcu()
Quoted 13 lines · Jianyue Wu
When a pool's last reference is dropped,
__zswap_pool_empty()removes it from the pool list and schedules__zswap_pool_release(), which callssynchronize_rcu()to wait for readers before tearing the pool down.
synchronize_rcu()is a synchronous, potentially long wait. Replace it withqueue_rcu_work():__zswap_pool_empty()hands the pool toqueue_rcu_work(), which waits for a grace period asynchronously and then runs__zswap_pool_release()from a worker for the sleepable teardown (__zswap_pool_empty()can run in atomic context and must not block). The grace-period guarantee is unchanged; the retirement path just no longer blocks on it.
Quoted 2 lines
Suggested-by Yosry Ahmed <[email protected]>Signed-off-by Jianyue Wu <[email protected]>
Why are the patches still tagged RFC?
Anyway, with one nit below:
Quoted 9 lines
--- mm/zswap.c | 12 +++++------- 1 file changed, 5 insertions(+), 7 deletions(-)mm/zswap.c +0
-0
mm/zswap.c struct zswap_pool { · line 158
Quoted 5 lines
mm/zswap.c +1
-1@@ -155,7 +155,7 @@ struct zswap_pool {struct crypto_acomp_ctx __percpu *acomp_ctx;struct percpu_ref ref;struct list_head list; -struct work_struct release_work; +struct rcu_work release_work;
Seems like the convention is to use rwork in the name instead of work.
Quoted 30 lines
mm/zswap.c +4
-7struct hlist_node node;char tfm_name[CRYPTO_MAX_ALG_NAME ]; };@@ -379,10 +379,8 @@ static void zswap_pool_destroy(struct zswap_pool *pool)static void __zswap_pool_release (struct work_struct *work) { -struct zswap_pool *pool =container_of (work,typeof (*pool), - release_work); - -synchronize_rcu (); +struct zswap_pool *pool =container_of (to_rcu_work (work), +typeof (*pool), release_work);/* nobody should have been able to get a ref... */ WARN_ON (!percpu_ref_is_zero (&pool->ref));@@ -406,8 +404,8 @@ static void __zswap_pool_empty(struct percpu_ref *ref)list_del_rcu (&pool->list); -INIT_WORK (&pool->release_work, __zswap_pool_release); -schedule_work (&pool->release_work); +INIT_RCU_WORK (&pool->release_work, __zswap_pool_release); +queue_rcu_work (system_percpu_wq, &pool->release_work);spin_unlock_bh (&zswap_pools_lock); } --2.43.0
Re: [RFC PATCH v4 1/3] mm/zswap: release retired pools via queue_rcu_work() instead of synchronize_rcu()
Quoted 15 lines · Yosry Ahmed
Quoted 13 lines · Jianyue Wu
When a pool's last reference is dropped,
__zswap_pool_empty()removes it from the pool list and schedules__zswap_pool_release(), which callssynchronize_rcu()to wait for readers before tearing the pool down.
synchronize_rcu()is a synchronous, potentially long wait. Replace it withqueue_rcu_work():__zswap_pool_empty()hands the pool toqueue_rcu_work(), which waits for a grace period asynchronously and then runs__zswap_pool_release()from a worker for the sleepable teardown (__zswap_pool_empty()can run in atomic context and must not block). The grace-period guarantee is unchanged; the retirement path just no longer blocks on it.
Quoted 4 lines
Quoted 2 lines
Suggested-by Yosry Ahmed <[email protected]>Signed-off-by Jianyue Wu <[email protected]>Why are the patches still tagged RFC?
Exactly, should be removed, I dropped the RFC tag in the new version.
Quoted 8 lines
Anyway, with one nit below:
Acked-by Yosry Ahmed <[email protected]>Quoted 4 lines
--- mm/zswap.c | 12 +++++------- 1 file changed, 5 insertions(+), 7 deletions(-)
Quoted 12 lines
mm/zswap.c struct zswap_pool { · line 158
Quoted 10 lines
mm/zswap.c +1
-1@@ -155,7 +155,7 @@ struct zswap_pool {struct crypto_acomp_ctx __percpu *acomp_ctx;struct percpu_ref ref;struct list_head list; -struct work_struct release_work; +struct rcu_work release_work;Seems like the convention is to use rwork in the name instead of work.
Good point, I renamed it to release_rwork in the new version.
Best regards,
Jianyue
Re: [RFC PATCH v4 1/3] mm/zswap: release retired pools via queue_rcu_work() instead of synchronize_rcu()
In reply to Jianyue WuQuoted 12 lines · Jianyue Wu
When a pool's last reference is dropped,
__zswap_pool_empty()removes it from the pool list and schedules__zswap_pool_release(), which callssynchronize_rcu()to wait for readers before tearing the pool down.
synchronize_rcu()is a synchronous, potentially long wait. Replace it withqueue_rcu_work():__zswap_pool_empty()hands the pool toqueue_rcu_work(), which waits for a grace period asynchronously and then runs__zswap_pool_release()from a worker for the sleepable teardown (__zswap_pool_empty()can run in atomic context and must not block). The grace-period guarantee is unchanged; the retirement path just no longer blocks on it.
Quoted 2 lines
Suggested-by Yosry Ahmed <[email protected]>Signed-off-by Jianyue Wu <[email protected]>
With "release_rwork",
Re: [RFC PATCH v4 1/3] mm/zswap: release retired pools via queue_rcu_work() instead of synchronize_rcu()
Quoted 14 lines · Johannes Weiner
Quoted 12 lines · Jianyue Wu
When a pool's last reference is dropped,
__zswap_pool_empty()removes it from the pool list and schedules__zswap_pool_release(), which callssynchronize_rcu()to wait for readers before tearing the pool down.
synchronize_rcu()is a synchronous, potentially long wait. Replace it withqueue_rcu_work():__zswap_pool_empty()hands the pool toqueue_rcu_work(), which waits for a grace period asynchronously and then runs__zswap_pool_release()from a worker for the sleepable teardown (__zswap_pool_empty()can run in atomic context and must not block). The grace-period guarantee is unchanged; the retirement path just no longer blocks on it.
Quoted 6 lines
Quoted 2 lines
Suggested-by Yosry Ahmed <[email protected]>Signed-off-by Jianyue Wu <[email protected]>With "
release_rwork",Reviewed-by Johannes Weiner <[email protected]>
Thanks, I will use release_rwork instead.
Best regards,
Jianyue
RFC PATCH v4 2/3 mm/zswap: replace the zswap_pools list with a fixed pools array
In reply to Jianyue WuOriginally zswap holds its pools on an RCU list whose head also serves as the "current pool". Only a handful of pools are ever live at once, since a new pool is only created when the compressor is (re)set and pools are reused across compressor switches.
Hold the pools in a fixed ZSWAP_MAX_POOLS-element array so each pool has a stable slot number, and track the current pool with a separate rcu-protected pointer.
Slot 0 is intentionally left unused (always NULL): a zeroed or incorrectly initialized pool index then resolves to NULL and trips a WARN rather than silently aliasing a live pool in another slot.
The array keeps the same RCU publish/retire discipline the list had, so lookup and teardown stay equivalent. A fully-constructed pool is stored into its slot as the last step of zswap_pool_create(), so array walkers only ever observe a NULL slot or a ready pool. Pool creation is serialized by the module-wide kernel param mutex (all built-in params share one lock) and otherwise only happens during single-threaded init, so no two creators race for a slot. zswap_pools_lock still serializes the store against a retiring pool clearing its slot in __zswap_pool_empty().
Behavior change: the fixed array bounds the number of simultaneously live pools at ZSWAP_MAX_POOLS - 1 (15, since slot 0 is reserved), whereas the old list was unbounded. A pool is only live while it is the current pool or still has stored pages referencing it, and pools are reused across compressor switches, so 15 is far more than any real configuration needs. Once all slots are occupied, creating a pool for a 16th distinct compressor fails: zswap_pool_create() errors and returns NULL, and the compressor switch is rejected with -EINVAL rather than silently succeeding. The cap can be raised by increasing ZSWAP_MAX_POOLS (bounded by the u8 slot index, so up to 256).
--- mm/zswap.c | 97 ++++++++++++++++++++++++++++++++++++++++-------------- 1 file changed, 72 insertions(+), 25 deletions(-)
mm/zswap.c +72 -25
@@ -13,6 +13,7 @@#define pr_fmt (fmt) KBUILD_MODNAME ": " fmt +#include <linux/cleanup.h> #include <linux/module.h> #include <linux/cpu.h> #include <linux/highmem.h>
@@ -154,12 +155,27 @@ struct zswap_pool {struct zs_pool *zs_pool;struct crypto_acomp_ctx __percpu *acomp_ctx;struct percpu_ref ref; -struct list_head list;struct rcu_work release_work;struct hlist_node node; +u8 idx;char tfm_name[CRYPTO_MAX_ALG_NAME ]; }; +#define ZSWAP_MAX_POOLS 16 +/* +* Slot 0 is intentionally never used: it stays NULL so that a zeroed or +* incorrectly initialized pool->idx resolves to NULL (and trips a WARN) +* instead of silently aliasing a live pool in another slot. +*/ +#define ZSWAP_FIRST_POOL_SLOT 1 +static struct zswap_pool __rcu *zswap_pools[ZSWAP_MAX_POOLS ]; +static_assert (ZSWAP_MAX_POOLS -1 <=U8_MAX ); +/* +* The current pool (NULL if none): an alias of one zswap_pools[] slot. +* It always holds a ref, so a pool is never retired while it is current. +*/ +static struct zswap_pool __rcu *zswap_current_pool; +/* Global LRU lists shared by all zswap pools. */ static struct list_lru zswap_list_lru;
@@ -200,9 +216,6 @@ struct zswap_entry {static struct xarray *zswap_trees[MAX_SWAPFILES ];static unsigned int nr_zswap_trees[MAX_SWAPFILES ]; -/* RCU-protected iteration */ -static LIST_HEAD (zswap_pools); -/* protects zswap_pools list modification */ static DEFINE_SPINLOCK (zswap_pools_lock);/* pool counter to provide unique names to zsmalloc */ static atomic_t zswap_pools_count =ATOMIC_INIT (0 );
@@ -270,6 +283,31 @@ static void acomp_ctx_free(struct crypto_acomp_ctx *acomp_ctx) acomp_ctx->buffer =NULL ; } +/* +* Publish a fully-constructed pool into a free array slot. Pool creation is +* serialized by the module-wide kernel param mutex (all built-in params share +* one lock) and only otherwise happens during single-threaded init, so no two +* creators race for a slot. The pool is complete before it is stored, and +* zswap_pools_lock still serializes this store against a concurrent retiring +* pool clearing its slot in __zswap_pool_empty(), so array walkers only ever +* observe a NULL slot or a ready pool. +*/ +static int zswap_pool_assign_slot (struct zswap_pool *pool) +{ +int i; + +guard (spinlock_bh)(&zswap_pools_lock); +for (i =ZSWAP_FIRST_POOL_SLOT ; i <ZSWAP_MAX_POOLS ; i++) { +if (!rcu_access_pointer (zswap_pools[i])) { + pool->idx = i; +rcu_assign_pointer (zswap_pools[i], pool); +return i; + } + } + +return -ENOSPC ; +} +static struct zswap_pool *zswap_pool_create (char *compressor) {struct zswap_pool *pool;
@@ -313,19 +351,29 @@ static struct zswap_pool *zswap_pool_create(char *compressor)if (ret) goto cpuhp_add_fail; -/* being the current pool takes 1 ref; this func expects the -* caller to always add the new pool as the current pool +/* +* The initial ref keeps the pool alive while it is current. Stored +* entries take additional refs so a retired pool remains alive while +* any entries still reference it. */ ret =percpu_ref_init (&pool->ref, __zswap_pool_empty,PERCPU_REF_ALLOW_REINIT ,GFP_KERNEL );if (ret) goto ref_fail; -INIT_LIST_HEAD (&pool->list); + + ret =zswap_pool_assign_slot (pool); +if (ret <0 ) { +pr_err ("cannot create more than %d pools\n" , +ZSWAP_MAX_POOLS -ZSWAP_FIRST_POOL_SLOT ); + goto slot_fail; + }zswap_pool_debug ("created" , pool);return pool; +slot_fail: +percpu_ref_exit (&pool->ref); ref_fail:cpuhp_state_remove_instance (CPUHP_MM_ZSWP_POOL_PREPARE , &pool->node);
@@ -386,7 +434,6 @@ static void __zswap_pool_release(struct work_struct *work)WARN_ON (!percpu_ref_is_zero (&pool->ref));percpu_ref_exit (&pool->ref); -/* pool is now off zswap_pools list and has no references. */ zswap_pool_destroy (pool); }
@@ -402,7 +449,7 @@ static void __zswap_pool_empty(struct percpu_ref *ref)WARN_ON (pool ==zswap_pool_current ()); -list_del_rcu (&pool->list); +rcu_assign_pointer (zswap_pools[pool->idx],NULL );INIT_RCU_WORK (&pool->release_work, __zswap_pool_release);queue_rcu_work (system_percpu_wq, &pool->release_work);
@@ -433,7 +480,8 @@ static struct zswap_pool *__zswap_pool_current(void) {struct zswap_pool *pool; - pool =list_first_or_null_rcu (&zswap_pools,typeof (*pool), list); + pool =rcu_dereference_check (zswap_current_pool, +lockdep_is_held (&zswap_pools_lock));WARN_ONCE (!pool && zswap_has_pool,"%s: no page storage pool!\n" , __func__);
@@ -466,11 +514,12 @@ static struct zswap_pool *zswap_pool_current_get(void)static struct zswap_pool *zswap_pool_find_get (char *compressor) {struct zswap_pool *pool; +int i; -assert_spin_locked (&zswap_pools_lock); - -list_for_each_entry_rcu (pool, &zswap_pools, list) { -if (strcmp (pool->tfm_name, compressor)) +for (i =ZSWAP_FIRST_POOL_SLOT ; i <ZSWAP_MAX_POOLS ; i++) { + pool =rcu_dereference_protected (zswap_pools[i], +lockdep_is_held (&zswap_pools_lock)); +if (!pool ||strcmp (pool->tfm_name, compressor))continue ;/* if we can't get it, it's about to be destroyed */ if (!zswap_pool_tryget (pool))
@@ -495,10 +544,14 @@ unsigned long zswap_total_pages(void) {struct zswap_pool *pool;unsigned long total =0 ; +int i;rcu_read_lock (); -list_for_each_entry_rcu (pool, &zswap_pools, list) - total +=zs_get_total_pages (pool->zs_pool); +for (i =ZSWAP_FIRST_POOL_SLOT ; i <ZSWAP_MAX_POOLS ; i++) { + pool =rcu_dereference (zswap_pools[i]); +if (pool) + total +=zs_get_total_pages (pool->zs_pool); + }rcu_read_unlock ();return total;
@@ -560,7 +613,6 @@ static int zswap_compressor_param_set(const char *val, const struct kernel_paramif (pool) {zswap_pool_debug ("using existing" , pool);WARN_ON (pool ==zswap_pool_current ()); -list_del_rcu (&pool->list); }spin_unlock_bh (&zswap_pools_lock);
@@ -588,15 +640,9 @@ static int zswap_compressor_param_set(const char *val, const struct kernel_paramif (!ret) { put_pool =zswap_pool_current (); -list_add_rcu (&pool->list, &zswap_pools); +rcu_assign_pointer (zswap_current_pool, pool); zswap_has_pool = true; }else if (pool) { -/* -* Add the possibly pre-existing pool to the end of the pools -* list; if it's new (and empty) then it'll be removed and -* destroyed by the put after we drop the lock -*/ -list_add_tail_rcu (&pool->list, &zswap_pools); put_pool = pool; }
@@ -1801,7 +1847,8 @@ static int zswap_setup(void) pool =__zswap_pool_create_fallback ();if (pool) {pr_info ("loaded using pool %s\n" , pool->tfm_name); -list_add (&pool->list, &zswap_pools); +/* zswap_pool_create() already stored the pool in its array slot. */ +rcu_assign_pointer (zswap_current_pool, pool); zswap_has_pool = true;static_branch_enable (&zswap_ever_enabled); }else {
Signature · 1 line
2.43.0
Re: [RFC PATCH v4 2/3] mm/zswap: replace the zswap_pools list with a fixed pools array
Quoted 35 lines · Jianyue Wu
Originally zswap holds its pools on an RCU list whose head also serves as the "current pool". Only a handful of pools are ever live at once, since a new pool is only created when the compressor is (re)set and pools are reused across compressor switches.
Hold the pools in a fixed ZSWAP_MAX_POOLS-element array so each pool has a stable slot number, and track the current pool with a separate rcu-protected pointer.
Slot 0 is intentionally left unused (always NULL): a zeroed or incorrectly initialized pool index then resolves to NULL and trips a WARN rather than silently aliasing a live pool in another slot.
The array keeps the same RCU publish/retire discipline the list had, so lookup and teardown stay equivalent. A fully-constructed pool is stored into its slot as the last step of
zswap_pool_create(), so array walkers only ever observe a NULL slot or a ready pool. Pool creation is serialized by the module-wide kernel param mutex (all built-in params share one lock) and otherwise only happens during single-threaded init, so no two creators race for a slot.zswap_pools_lockstill serializes the store against a retiring pool clearing its slot in__zswap_pool_empty().Behavior change: the fixed array bounds the number of simultaneously live pools at
ZSWAP_MAX_POOLS- 1 (15, since slot 0 is reserved), whereas the old list was unbounded. A pool is only live while it is the current pool or still has stored pages referencing it, and pools are reused across compressor switches, so 15 is far more than any real configuration needs. Once all slots are occupied, creating a pool for a 16th distinct compressor fails:zswap_pool_create()errors and returns NULL, and the compressor switch is rejected with-EINVALrather than silently succeeding. The cap can be raised by increasingZSWAP_MAX_POOLS(bounded by the u8 slot index, so up to 256).
Quoted 3 lines
Suggested-by Nhat Pham <[email protected]>Suggested-by Yosry Ahmed <[email protected]>Signed-off-by Jianyue Wu <[email protected]>
Johannes Weiner
maintainerRe: [RFC PATCH v4 2/3] mm/zswap: replace the zswap_pools list with a fixed pools array
In reply to Jianyue WuQuoted 55 lines · Jianyue Wu
Originally zswap holds its pools on an RCU list whose head also serves as the "current pool". Only a handful of pools are ever live at once, since a new pool is only created when the compressor is (re)set and pools are reused across compressor switches.
Hold the pools in a fixed ZSWAP_MAX_POOLS-element array so each pool has a stable slot number, and track the current pool with a separate rcu-protected pointer.
Slot 0 is intentionally left unused (always NULL): a zeroed or incorrectly initialized pool index then resolves to NULL and trips a WARN rather than silently aliasing a live pool in another slot.
The array keeps the same RCU publish/retire discipline the list had, so lookup and teardown stay equivalent. A fully-constructed pool is stored into its slot as the last step of
zswap_pool_create(), so array walkers only ever observe a NULL slot or a ready pool. Pool creation is serialized by the module-wide kernel param mutex (all built-in params share one lock) and otherwise only happens during single-threaded init, so no two creators race for a slot.zswap_pools_lockstill serializes the store against a retiring pool clearing its slot in__zswap_pool_empty().Behavior change: the fixed array bounds the number of simultaneously live pools at
ZSWAP_MAX_POOLS- 1 (15, since slot 0 is reserved), whereas the old list was unbounded. A pool is only live while it is the current pool or still has stored pages referencing it, and pools are reused across compressor switches, so 15 is far more than any real configuration needs. Once all slots are occupied, creating a pool for a 16th distinct compressor fails:zswap_pool_create()errors and returns NULL, and the compressor switch is rejected with-EINVALrather than silently succeeding. The cap can be raised by increasingZSWAP_MAX_POOLS(bounded by the u8 slot index, so up to 256).Suggested-by Nhat Pham <[email protected]>Suggested-by Yosry Ahmed <[email protected]>Signed-off-by Jianyue Wu <[email protected]>--- mm/zswap.c | 97 ++++++++++++++++++++++++++++++++++++++++-------------- 1 file changed, 72 insertions(+), 25 deletions(-)mm/zswap.c +1
-0@@ -13,6 +13,7 @@#define pr_fmt (fmt) KBUILD_MODNAME ": " fmt +#include <linux/cleanup.h> #include <linux/module.h> #include <linux/cpu.h> #include <linux/highmem.h> @@ -154,12 +155,27 @@ struct zswap_pool {struct zs_pool *zs_pool;
mm/zswap.c struct zswap_pool { · line 164
Quoted 10 lines
mm/zswap.c +2
-1struct crypto_acomp_ctx __percpu *acomp_ctx;struct percpu_ref ref; -struct list_head list;struct rcu_work release_work;struct hlist_node node; +u8 idx;char tfm_name[CRYPTO_MAX_ALG_NAME ]; }; +#define ZSWAP_MAX_POOLS 16
It's unlikely to happen, but this is a super annoying failure mode. User would have to kill something, delete shmem/tmpfs, or swapoff. And it's not obvious which entries are in which pool.
Wouldn't an idr make more sense?
Quoted 18 lines
mm/zswap.c +14
-0@@ -270,6 +283,31 @@ static void acomp_ctx_free(struct crypto_acomp_ctx *acomp_ctx) acomp_ctx->buffer =NULL ; } +/* +* Publish a fully-constructed pool into a free array slot. Pool creation is +* serialized by the module-wide kernel param mutex (all built-in params share +* one lock) and only otherwise happens during single-threaded init, so no two +* creators race for a slot. The pool is complete before it is stored, and +* zswap_pools_lock still serializes this store against a concurrent retiring +* pool clearing its slot in __zswap_pool_empty(), so array walkers only ever +* observe a NULL slot or a ready pool. +*/ +static int zswap_pool_assign_slot (struct zswap_pool *pool) +{ +int i; + +guard (spinlock_bh)(&zswap_pools_lock);
mm/zswap.c static void acomp_ctx_free(struct crypto_acomp_ctx *acomp_ctx) · line 309
Quoted 10 lines
mm/zswap.c +10
-0+for (i =ZSWAP_FIRST_POOL_SLOT ; i <ZSWAP_MAX_POOLS ; i++) { +if (!rcu_access_pointer (zswap_pools[i])) { + pool->idx = i; +rcu_assign_pointer (zswap_pools[i], pool); +return i; + } + } + +return -ENOSPC ; +}
It was kind of overdue, but with this now requiring a pool walk as well, it would be better to factor out a find_or_create function?
Something like:
static struct zswap_pool *zswap_pool_find_or_create(char *compressor)
{
struct zswap_pool *pool, *new_pool = NULL;
u8 id, new_id = 0;insert_new:
spin_lock_bh(&zswap_pools_lock);
idr_for_each_entry(&zswap_pools, pool, id) {
if (pool && !strcmp(pool->tfm_name, compressor) && zswap_pool_tryget(pool)) {
if (new_pool) {
pool_put(new_pool);
idr_free(&zswap_pools, new_id);
}
spin_unlock_bh(&zswap_pools_lock);
return pool;
}
}
if (new_pool) {
idr_replace(&zswap_pools, new_pool, new_id);
spin_unlock_bh(&zswap_pools_lock);
return new_pool;
}
spin_unlock_bh(&zswap_pools_lock);
new_pool = pool_alloc();
if (!new_pool)
...
new_id = idr_alloc(&zswap_pools, NULL, 1, 256, GFP_KERNEL);
if (new_id < 0)
...
goto insert_new;
} Re: [RFC PATCH v4 2/3] mm/zswap: replace the zswap_pools list with a fixed pools array
Quoted 132 lines · Johannes Weiner
Quoted 55 lines · Jianyue Wu
Originally zswap holds its pools on an RCU list whose head also serves as the "current pool". Only a handful of pools are ever live at once, since a new pool is only created when the compressor is (re)set and pools are reused across compressor switches.
Hold the pools in a fixed ZSWAP_MAX_POOLS-element array so each pool has a stable slot number, and track the current pool with a separate rcu-protected pointer.
Slot 0 is intentionally left unused (always NULL): a zeroed or incorrectly initialized pool index then resolves to NULL and trips a WARN rather than silently aliasing a live pool in another slot.
The array keeps the same RCU publish/retire discipline the list had, so lookup and teardown stay equivalent. A fully-constructed pool is stored into its slot as the last step of
zswap_pool_create(), so array walkers only ever observe a NULL slot or a ready pool. Pool creation is serialized by the module-wide kernel param mutex (all built-in params share one lock) and otherwise only happens during single-threaded init, so no two creators race for a slot.zswap_pools_lockstill serializes the store against a retiring pool clearing its slot in__zswap_pool_empty().Behavior change: the fixed array bounds the number of simultaneously live pools at
ZSWAP_MAX_POOLS- 1 (15, since slot 0 is reserved), whereas the old list was unbounded. A pool is only live while it is the current pool or still has stored pages referencing it, and pools are reused across compressor switches, so 15 is far more than any real configuration needs. Once all slots are occupied, creating a pool for a 16th distinct compressor fails:zswap_pool_create()errors and returns NULL, and the compressor switch is rejected with-EINVALrather than silently succeeding. The cap can be raised by increasingZSWAP_MAX_POOLS(bounded by the u8 slot index, so up to 256).Suggested-by Nhat Pham <[email protected]>Suggested-by Yosry Ahmed <[email protected]>Signed-off-by Jianyue Wu <[email protected]>--- mm/zswap.c | 97 ++++++++++++++++++++++++++++++++++++++++-------------- 1 file changed, 72 insertions(+), 25 deletions(-)mm/zswap.c +1
-0@@ -13,6 +13,7 @@#define pr_fmt (fmt) KBUILD_MODNAME ": " fmt +#include <linux/cleanup.h> #include <linux/module.h> #include <linux/cpu.h> #include <linux/highmem.h> @@ -154,12 +155,27 @@ struct zswap_pool {struct zs_pool *zs_pool;mm/zswap.c struct zswap_pool { · line 164
Quoted 10 lines
mm/zswap.c +2
-1struct crypto_acomp_ctx __percpu *acomp_ctx;struct percpu_ref ref; -struct list_head list;struct rcu_work release_work;struct hlist_node node; +u8 idx;char tfm_name[CRYPTO_MAX_ALG_NAME ]; }; +#define ZSWAP_MAX_POOLS 16It's unlikely to happen, but this is a super annoying failure mode. User would have to kill something, delete shmem/tmpfs, or swapoff. And it's not obvious which entries are in which pool.
Wouldn't an idr make more sense?
Quoted 18 lines
mm/zswap.c +14
-0@@ -270,6 +283,31 @@ static void acomp_ctx_free(struct crypto_acomp_ctx *acomp_ctx) acomp_ctx->buffer =NULL ; } +/* +* Publish a fully-constructed pool into a free array slot. Pool creation is +* serialized by the module-wide kernel param mutex (all built-in params share +* one lock) and only otherwise happens during single-threaded init, so no two +* creators race for a slot. The pool is complete before it is stored, and +* zswap_pools_lock still serializes this store against a concurrent retiring +* pool clearing its slot in __zswap_pool_empty(), so array walkers only ever +* observe a NULL slot or a ready pool. +*/ +static int zswap_pool_assign_slot (struct zswap_pool *pool) +{ +int i; + +guard (spinlock_bh)(&zswap_pools_lock);mm/zswap.c static void acomp_ctx_free(struct crypto_acomp_ctx *acomp_ctx) · line 309
Quoted 10 lines
mm/zswap.c +10
-0+for (i =ZSWAP_FIRST_POOL_SLOT ; i <ZSWAP_MAX_POOLS ; i++) { +if (!rcu_access_pointer (zswap_pools[i])) { + pool->idx = i; +rcu_assign_pointer (zswap_pools[i], pool); +return i; + } + } + +return -ENOSPC ; +}It was kind of overdue, but with this now requiring a pool walk as well, it would be better to factor out a
find_or_createfunction?Something like:
static struct
zswap_pool*zswap_pool_find_or_create(char *compressor){ struct zswap_pool *pool, *new_pool = NULL; u8 id, new_id = 0;
insert_new:spin_lock_bh(&zswap_pools_lock); idr_for_each_entry(&zswap_pools, pool, id) { if (pool && !strcmp(pool->tfm_name, compressor) && zswap_pool_tryget(pool)) { if (new_pool) { pool_put(new_pool); idr_free(&zswap_pools, new_id); } spin_unlock_bh(&zswap_pools_lock); return pool; } } if (new_pool) { idr_replace(&zswap_pools, new_pool, new_id); spin_unlock_bh(&zswap_pools_lock); return new_pool; } spin_unlock_bh(&zswap_pools_lock);
Quoted 8 lines
new_pool = pool_alloc();
if (!new_pool)
...
new_id = idr_alloc(&zswap_pools, NULL, 1, 256, GFP_KERNEL);
if (new_id < 0)
...
goto insert_new;
}Agreed. Hitting the 15-pool cap would be an annoying failure mode, even if it should be rare.
I will switch this to an IDR with IDs in the 1..255 range, keeping 0 as the invalid entry value, and factor the lookup/allocation path into a zswap_pool_find_or_create() helper.
Best regards,
Jianyue
Re: [RFC PATCH v4 2/3] mm/zswap: replace the zswap_pools list with a fixed pools array
Quoted 142 lines · Jianyue Wu
Quoted 132 lines · Johannes Weiner
Quoted 55 lines · Jianyue Wu
Originally zswap holds its pools on an RCU list whose head also serves as the "current pool". Only a handful of pools are ever live at once, since a new pool is only created when the compressor is (re)set and pools are reused across compressor switches.
Hold the pools in a fixed ZSWAP_MAX_POOLS-element array so each pool has a stable slot number, and track the current pool with a separate rcu-protected pointer.
Slot 0 is intentionally left unused (always NULL): a zeroed or incorrectly initialized pool index then resolves to NULL and trips a WARN rather than silently aliasing a live pool in another slot.
The array keeps the same RCU publish/retire discipline the list had, so lookup and teardown stay equivalent. A fully-constructed pool is stored into its slot as the last step of
zswap_pool_create(), so array walkers only ever observe a NULL slot or a ready pool. Pool creation is serialized by the module-wide kernel param mutex (all built-in params share one lock) and otherwise only happens during single-threaded init, so no two creators race for a slot.zswap_pools_lockstill serializes the store against a retiring pool clearing its slot in__zswap_pool_empty().Behavior change: the fixed array bounds the number of simultaneously live pools at
ZSWAP_MAX_POOLS- 1 (15, since slot 0 is reserved), whereas the old list was unbounded. A pool is only live while it is the current pool or still has stored pages referencing it, and pools are reused across compressor switches, so 15 is far more than any real configuration needs. Once all slots are occupied, creating a pool for a 16th distinct compressor fails:zswap_pool_create()errors and returns NULL, and the compressor switch is rejected with-EINVALrather than silently succeeding. The cap can be raised by increasingZSWAP_MAX_POOLS(bounded by the u8 slot index, so up to 256).Suggested-by Nhat Pham <[email protected]>Suggested-by Yosry Ahmed <[email protected]>Signed-off-by Jianyue Wu <[email protected]>--- mm/zswap.c | 97 ++++++++++++++++++++++++++++++++++++++++-------------- 1 file changed, 72 insertions(+), 25 deletions(-)mm/zswap.c +1
-0@@ -13,6 +13,7 @@#define pr_fmt (fmt) KBUILD_MODNAME ": " fmt +#include <linux/cleanup.h> #include <linux/module.h> #include <linux/cpu.h> #include <linux/highmem.h> @@ -154,12 +155,27 @@ struct zswap_pool {struct zs_pool *zs_pool;mm/zswap.c struct zswap_pool { · line 164
Quoted 10 lines
mm/zswap.c +2
-1struct crypto_acomp_ctx __percpu *acomp_ctx;struct percpu_ref ref; -struct list_head list;struct rcu_work release_work;struct hlist_node node; +u8 idx;char tfm_name[CRYPTO_MAX_ALG_NAME ]; }; +#define ZSWAP_MAX_POOLS 16It's unlikely to happen, but this is a super annoying failure mode. User would have to kill something, delete shmem/tmpfs, or swapoff. And it's not obvious which entries are in which pool.
Wouldn't an idr make more sense?
Quoted 18 lines
mm/zswap.c +14
-0@@ -270,6 +283,31 @@ static void acomp_ctx_free(struct crypto_acomp_ctx *acomp_ctx) acomp_ctx->buffer =NULL ; } +/* +* Publish a fully-constructed pool into a free array slot. Pool creation is +* serialized by the module-wide kernel param mutex (all built-in params share +* one lock) and only otherwise happens during single-threaded init, so no two +* creators race for a slot. The pool is complete before it is stored, and +* zswap_pools_lock still serializes this store against a concurrent retiring +* pool clearing its slot in __zswap_pool_empty(), so array walkers only ever +* observe a NULL slot or a ready pool. +*/ +static int zswap_pool_assign_slot (struct zswap_pool *pool) +{ +int i; + +guard (spinlock_bh)(&zswap_pools_lock);mm/zswap.c static void acomp_ctx_free(struct crypto_acomp_ctx *acomp_ctx) · line 309
Quoted 10 lines
mm/zswap.c +10
-0+for (i =ZSWAP_FIRST_POOL_SLOT ; i <ZSWAP_MAX_POOLS ; i++) { +if (!rcu_access_pointer (zswap_pools[i])) { + pool->idx = i; +rcu_assign_pointer (zswap_pools[i], pool); +return i; + } + } + +return -ENOSPC ; +}It was kind of overdue, but with this now requiring a pool walk as well, it would be better to factor out a
find_or_createfunction?Something like:
static struct
zswap_pool*zswap_pool_find_or_create(char *compressor){ struct zswap_pool *pool, *new_pool = NULL; u8 id, new_id = 0;
insert_new:spin_lock_bh(&zswap_pools_lock); idr_for_each_entry(&zswap_pools, pool, id) { if (pool && !strcmp(pool->tfm_name, compressor) && zswap_pool_tryget(pool)) { if (new_pool) { pool_put(new_pool); idr_free(&zswap_pools, new_id); } spin_unlock_bh(&zswap_pools_lock); return pool; } } if (new_pool) { idr_replace(&zswap_pools, new_pool, new_id); spin_unlock_bh(&zswap_pools_lock); return new_pool; } spin_unlock_bh(&zswap_pools_lock);Quoted 8 lines
new_pool = pool_alloc(); if (!new_pool) ... new_id = idr_alloc(&zswap_pools, NULL, 1, 256, GFP_KERNEL); if (new_id < 0) ... goto insert_new; }
Quoted 8 lines
Agreed. Hitting the 15-pool cap would be an annoying failure mode, even if it should be rare.
I will switch this to an IDR with IDs in the 1..255 range, keeping 0 as the invalid entry value, and factor the lookup/allocation path into azswap_pool_find_or_create()helper.Best regards,
Jianyue
Hello Johannes,
I have this working locally, but I am still testing and some parts still need clarification.
1. Could I use xarray here? Similar to idr, can naturally start from 1, and zswap already uses xarray for the entry trees.
i.e. the pool table is an allocating xarray (XA_FLAGS_ALLOC1, ids 1..255, 0 reserved).
2. Compressor updates are already serialized by the param lock, and the only other create path is zswap_setup() during init (or when enabling zswap). That does not run alongside another creator, so I think we can keep the simpler lookup-then-create flow. It is almost the same as the original list code, just using the xarray API.
Best regards,
Jianyue
RFC PATCH v4 3/3 mm/zswap: reference the pool by index to shrink struct zswap_entry
In reply to Jianyue Wustruct zswap_entry is one allocation per stored page, so its size is pure overhead. It currently embeds an 8-byte pool pointer, even though the live pools now sit in a small fixed array indexed by a u8 slot number.
Replace the per-entry pool pointer with that u8 slot index and resolve it through the fixed pool array. A live entry holds a reference to its pool, so the slot cannot be reused under it. The lookup therefore needs no RCU read-side section or zswap_pools_lock.
The u8 fits in the padding after the bool referenced field, shrinking the entry from 56 to 48 bytes on x86_64. This raises objs_per_slab from 73 to 85 and saves about 2MiB of metadata per 1GiB of data held in zswap.
--- mm/zswap.c | 31 ++++++++++++++++++++++++------- 1 file changed, 24 insertions(+), 7 deletions(-)
mm/zswap.c +24 -7
@@ -198,7 +198,7 @@ static struct shrinker *zswap_shrinker;* writeback logic. The entry is only reclaimed by the writeback * logic if referenced is unset. See comments in the shrinker * section for context. -* pool - the zswap_pool the entry's data is in +* pool_idx - slot of the zswap_pool that the entry's data is in. * handle - zsmalloc allocation handle that stores the compressed page data * objcg - the obj_cgroup that the compressed memory is charged to * lru - handle to the pool's lru used to evict pages.
@@ -207,12 +207,22 @@ struct zswap_entry {swp_entry_t swpentry;unsigned int length;bool referenced; -struct zswap_pool *pool; +u8 pool_idx;unsigned long handle;struct obj_cgroup *objcg;struct list_head lru; }; +static struct zswap_pool *zswap_entry_pool (struct zswap_entry *entry) +{ +/* +* A live entry holds a pool reference, so the slot stays valid with no +* RCU read-side section. The != 0 check marks access protected by +* the reference. A live entry never uses the reserved slot 0. +*/ +return rcu_dereference_check (zswap_pools[entry->pool_idx], entry->pool_idx !=0 ); +} +static struct xarray *zswap_trees[MAX_SWAPFILES ];static unsigned int nr_zswap_trees[MAX_SWAPFILES ];
@@ -808,9 +818,13 @@ static void zswap_entry_cache_free(struct zswap_entry *entry)*/ static void zswap_entry_free (struct zswap_entry *entry) { +struct zswap_pool *pool =zswap_entry_pool (entry); +zswap_lru_del (entry); -zs_free (entry->pool->zs_pool, entry->handle); -zswap_pool_put (entry->pool); +if (!WARN_ON_ONCE (!pool)) { +zs_free (pool->zs_pool, entry->handle); +zswap_pool_put (pool); + }if (entry->objcg) {obj_cgroup_uncharge_zswap (entry->objcg, entry->length);obj_cgroup_put (entry->objcg);
@@ -967,12 +981,15 @@ static bool zswap_compress(struct page *page, struct zswap_entry *entry,static bool zswap_decompress (struct zswap_entry *entry,struct folio *folio) { -struct zswap_pool *pool = entry->pool; +struct zswap_pool *pool =zswap_entry_pool (entry);struct scatterlist input[2 ];/* zsmalloc returns an SG list 1-2 entries */ struct scatterlist output;struct crypto_acomp_ctx *acomp_ctx;int ret =0 , dlen; +if (WARN_ON_ONCE (!pool)) +return false; + acomp_ctx =raw_cpu_ptr (pool->acomp_ctx);mutex_lock (&acomp_ctx->mutex);zs_obj_read_sg_begin (pool->zs_pool, entry->handle, input, entry->length);
@@ -1008,7 +1025,7 @@ static bool zswap_decompress(struct zswap_entry *entry, struct folio *folio)pr_alert_ratelimited ("Decompression error from zswap (%d:%lu %s %u->%d)\n" ,swp_type (entry->swpentry),swp_offset (entry->swpentry), - entry->pool->tfm_name, + pool->tfm_name, entry->length, dlen);return false; }
@@ -1511,7 +1528,7 @@ static bool zswap_store_page(struct page *page,* The publishing order matters to prevent writeback from seeing * an incoherent entry. */ - entry->pool = pool; + entry->pool_idx = pool->idx; entry->swpentry = page_swpentry; entry->objcg = objcg; entry->referenced = true;
Signature · 1 line
2.43.0
Re: [RFC PATCH v4 3/3] mm/zswap: reference the pool by index to shrink struct zswap_entry
Quoted 11 lines · Jianyue Wu
struct
zswap_entryis one allocation per stored page, so its size is pure overhead. It currently embeds an 8-byte pool pointer, even though the live pools now sit in a small fixed array indexed by a u8 slot number.Replace the per-entry pool pointer with that u8 slot index and resolve it through the fixed pool array. A live entry holds a reference to its pool, so the slot cannot be reused under it. The lookup therefore needs no RCU read-side section or
zswap_pools_lock.
Quoted 7 lines
The u8 fits in the padding after the bool referenced field, shrinking the entry from 56 to 48 bytes on
x86_64. This raisesobjs_per_slabfrom 73 to 85 and saves about 2MiB of metadata per 1GiB of data held in zswap.Suggested-by Chris Li <[email protected]>Signed-off-by Jianyue Wu <[email protected]>
v5 cover mm/zswap: shrink zswap_entry via a pool id
In reply to Jianyue WuEvery stored page has a struct zswap_entry, so its size is pure per-page overhead. On 64-bit it is currently 56 bytes, of which 8 bytes are a pointer to the owning zswap_pool.
Only a handful of pools are ever live: a new pool is created only when the compressor is (re)set, and pools are reused across compressor switches. That makes a per-entry pool pointer more expensive than it needs to be, and the RCU list that currently tracks pools is more machinery than this needs once each pool already has a stable id.
This series:
1. Releases retired pools with queue_rcu_work() instead of a worker calling synchronize_rcu(), so the release worker no longer blocks on an RCU grace period.
2. Replaces the zswap_pools list with an allocating xarray (XA_FLAGS_ALLOC1 | XA_FLAGS_LOCK_BH) and a separate RCU-protected current-pool pointer, giving each pool a stable small id. Ids start at 1. The reserved id 0 is never allocated, so looking it up resolves to NULL. The table grows as needed up to 255 live pools (u8 pool_idx), not a fixed slot array.
3. Stores that u8 pool id in each zswap_entry instead of the pool pointer. The u8 fits in padding after the bool referenced field, so the entry shrinks from 56 to 48 bytes on 64-bit (~2MiB of metadata saved per 1GiB of data held in zswap).
Runtime compressor switching is preserved. Pool ids are bounded to 1..255 because struct zswap_entry stores the id in a u8. The cap counts every id still in the xarray, including a killed pool that still has entries. Ids are reused when a pool is erased. Switching back to a compressor that still has a pool in the xarray resurrects it rather than allocating a new id. If all usable ids are full, creating a pool for another compressor fails and the compressor switch is rejected.
On 64-bit, struct zswap_entry is 56 -> 48 bytes, which fits 73 -> 85 objects in a 4K slab.
Benchmark (x86_64, compressor=lzo, MADV_PAGEOUT store + fault-in load):
- e2e store+load median latency: no measurable regression vs baseline at matched stored_delta
Each store, free, and decompress looks up the pool with xa_load() instead of following a pointer. With only a handful of live pools the xarray walk is short.
Testing
=======
- Boot with DEBUG_ATOMIC_SLEEP + lockdep/PROVE_RCU + KASAN:
zswap store/load, shrinker writeback, and compressor switch (retire, then switch back to resurrect) pass
This series is based on akpm/mm-unstable as of 2026-09-04 (20cab322c95c).
To: Johannes Weiner <[email protected]>
To: Yosry Ahmed <[email protected]>
To: Nhat Pham <[email protected]>
To: Chengming Zhou <[email protected]>
To: Andrew Morton <[email protected]>
Changes since RFC v4:
- Replace the RCU pool list with an allocating xarray (XA_FLAGS_ALLOC1 | XA_FLAGS_LOCK_BH) and a separate RCU-protected current-pool pointer.
- Resolve entry->pool_idx with xa_load() under rcu_read_lock().
- Write pool_idx before the entry is stored in the swap tree, so a lookup cannot see a stale pool_idx left over from slab reuse.
- Retire pools with queue_rcu_work() on system_percpu_wq.
- Drop the RFC tag.
---
Jianyue Wu (3):
mm/zswap: release retired pools via queue_rcu_work() instead of
synchronize_rcu()
mm/zswap: replace the zswap_pools list with an allocating xarray mm/zswap: reference the pool by id to shrink struct zswap_entry
mm/zswap.c | 144 ++++++++++++++++++++++++++++++++++++----------------- 1 file changed, 97 insertions(+), 47 deletions(-)
base-commit: 20cab322c95cea0327215bb81a05df69336032dd
Signature · 1 line
2.43.0
PATCH v5 1/3 mm/zswap: release retired pools via queue_rcu_work() instead of synchronize_rcu()
When a pool's last reference is dropped, __zswap_pool_empty() removes it from the pool list and schedules __zswap_pool_release(), which calls synchronize_rcu() to wait for readers before tearing the pool down.
synchronize_rcu() is a synchronous, potentially long wait. Replace it with queue_rcu_work(): __zswap_pool_empty() hands the pool to queue_rcu_work(), which waits for a grace period asynchronously and then runs __zswap_pool_release() from a worker for the sleepable teardown (__zswap_pool_empty() can run in atomic context and must not block). The grace-period guarantee is unchanged; the retirement path just no longer blocks on it.
--- mm/zswap.c | 12 +++++------- 1 file changed, 5 insertions(+), 7 deletions(-)
mm/zswap.c +5 -7
@@ -155,7 +155,7 @@ struct zswap_pool {struct crypto_acomp_ctx __percpu *acomp_ctx;struct percpu_ref ref;struct list_head list; -struct work_struct release_work; +struct rcu_work release_rwork;struct hlist_node node;char tfm_name[CRYPTO_MAX_ALG_NAME ]; };
@@ -379,10 +379,8 @@ static void zswap_pool_destroy(struct zswap_pool *pool)static void __zswap_pool_release (struct work_struct *work) { -struct zswap_pool *pool =container_of (work,typeof (*pool), - release_work); - -synchronize_rcu (); +struct zswap_pool *pool =container_of (to_rcu_work (work), +typeof (*pool), release_rwork);/* nobody should have been able to get a ref... */ WARN_ON (!percpu_ref_is_zero (&pool->ref));
@@ -406,8 +404,8 @@ static void __zswap_pool_empty(struct percpu_ref *ref)list_del_rcu (&pool->list); -INIT_WORK (&pool->release_work, __zswap_pool_release); -schedule_work (&pool->release_work); +INIT_RCU_WORK (&pool->release_rwork, __zswap_pool_release); +queue_rcu_work (system_percpu_wq, &pool->release_rwork);spin_unlock_bh (&zswap_pools_lock); }
Signature · 1 line
2.43.0
PATCH v5 2/3 mm/zswap: replace the zswap_pools list with an allocating xarray
In reply to Jianyue WuOriginally zswap keeps its pools on an RCU list whose head also serves as the current pool. Convert the pool table to an allocating xarray keyed by a small integer id, and track the current pool with a separate RCU-protected pointer.
The xarray gives each pool a stable id for a later zswap_entry shrink. XA_FLAGS_ALLOC1 starts ids at 1, so id 0 remains reserved. The id range is bounded by ZSWAP_MAX_POOL_ID because the later entry field is a u8.
Keep compressor switching close to the previous flow: look up an existing pool under xa_lock, resurrect it outside the lock if reused, or create a new one. zswap_pool_create() allocates the pool's id and publishes it into the xarray as its final step, so the create call is itself atomic: it either fully builds the pool and publishes it, or unwinds completely on failure. Publishing makes the pool live, so a caller that later fails (e.g. param_set_charp()) must still kill the pool to erase it from the xarray. Runtime compressor parameter updates are serialized by the module parameter lock, so no speculative loser path is needed.
A retiring pool is erased from the xarray in __zswap_pool_empty() and freed via queue_rcu_work(), preserving the old RCU teardown ordering.
--- mm/zswap.c | 87 +++++++++++++++++++++++++++++++++--------------------- 1 file changed, 54 insertions(+), 33 deletions(-)
mm/zswap.c +54 -33
@@ -34,6 +34,7 @@#include <linux/writeback.h> #include <linux/pagemap.h> #include <linux/workqueue.h> +#include <linux/xarray.h> #include <linux/list_lru.h> #include <linux/zsmalloc.h>
@@ -154,12 +155,24 @@ struct zswap_pool {struct zs_pool *zs_pool;struct crypto_acomp_ctx __percpu *acomp_ctx;struct percpu_ref ref; -struct list_head list;struct rcu_work release_rwork;struct hlist_node node; +u8 idx;char tfm_name[CRYPTO_MAX_ALG_NAME ]; }; +/* +* Live pools keyed by id (1..ZSWAP_MAX_POOL_ID). XA_FLAGS_ALLOC1 keeps +* the reserved id 0 unallocated, so looking it up never aliases a live +* pool. XA_FLAGS_LOCK_BH makes the xa_lock softirq-safe: it is taken +* from __zswap_pool_empty(), which runs from a percpu_ref release +* callback in softirq context. +*/ +#define ZSWAP_FIRST_POOL_ID 1 +#define ZSWAP_MAX_POOL_ID U8_MAX +static DEFINE_XARRAY_FLAGS(zswap_pools, XA_FLAGS_ALLOC1 | XA_FLAGS_LOCK_BH); +static struct zswap_pool __rcu *zswap_current_pool; +/* Global LRU lists shared by all zswap pools. */ static struct list_lru zswap_list_lru;
@@ -200,10 +213,6 @@ struct zswap_entry {static struct xarray *zswap_trees[MAX_SWAPFILES ];static unsigned int nr_zswap_trees[MAX_SWAPFILES ]; -/* RCU-protected iteration */ -static LIST_HEAD (zswap_pools); -/* protects zswap_pools list modification */ -static DEFINE_SPINLOCK (zswap_pools_lock);/* pool counter to provide unique names to zsmalloc */ static atomic_t zswap_pools_count =ATOMIC_INIT (0 );
@@ -275,6 +284,7 @@ static struct zswap_pool *zswap_pool_create(char *compressor)struct zswap_pool *pool;char name[38 ];/* 'zswap' + 32 char (max) num + \0 */ int ret, cpu; +u32 id;if (!zswap_has_pool && !strcmp (compressor,ZSWAP_PARAM_UNSET ))return NULL ;
@@ -320,12 +330,24 @@ static struct zswap_pool *zswap_pool_create(char *compressor) PERCPU_REF_ALLOW_REINIT, GFP_KERNEL);if (ret) goto ref_fail; -INIT_LIST_HEAD (&pool->list); + + ret =xa_alloc_bh (&zswap_pools, &id, pool, +XA_LIMIT (ZSWAP_FIRST_POOL_ID ,ZSWAP_MAX_POOL_ID ), +GFP_KERNEL ); +if (ret) { +if (ret == -EBUSY ) +pr_err ("cannot allocate pool id (max %d live pools)\n" , +ZSWAP_MAX_POOL_ID -ZSWAP_FIRST_POOL_ID +1 ); + goto xa_fail; + } + pool->idx = id;zswap_pool_debug ("created" , pool);return pool; +xa_fail: +percpu_ref_exit (&pool->ref); ref_fail:cpuhp_state_remove_instance (CPUHP_MM_ZSWP_POOL_PREPARE , &pool->node);
@@ -386,7 +408,7 @@ static void __zswap_pool_release(struct work_struct *work)WARN_ON (!percpu_ref_is_zero (&pool->ref));percpu_ref_exit (&pool->ref); -/* pool is now off zswap_pools list and has no references. */ +/* The pool is no longer in zswap_pools and has no references. */ zswap_pool_destroy (pool); }
@@ -398,16 +420,16 @@ static void __zswap_pool_empty(struct percpu_ref *ref) pool =container_of (ref,typeof (*pool), ref); -spin_lock_bh (&zswap_pools_lock); +xa_lock_bh (&zswap_pools);WARN_ON (pool ==zswap_pool_current ()); -list_del_rcu (&pool->list); +__xa_erase (&zswap_pools, pool->idx);INIT_RCU_WORK (&pool->release_rwork, __zswap_pool_release);queue_rcu_work (system_percpu_wq, &pool->release_rwork); -spin_unlock_bh (&zswap_pools_lock); +xa_unlock_bh (&zswap_pools); }static int __must_check zswap_pool_tryget (struct zswap_pool *pool)
@@ -433,7 +455,8 @@ static struct zswap_pool *__zswap_pool_current(void) {struct zswap_pool *pool; - pool =list_first_or_null_rcu (&zswap_pools,typeof (*pool), list); + pool =rcu_dereference_check (zswap_current_pool, +lockdep_is_held (&zswap_pools.xa_lock));WARN_ONCE (!pool && zswap_has_pool,"%s: no page storage pool!\n" , __func__);
@@ -442,7 +465,7 @@ static struct zswap_pool *__zswap_pool_current(void)static struct zswap_pool *zswap_pool_current (void ) { -assert_spin_locked (&zswap_pools_lock); +lockdep_assert_held (&zswap_pools.xa_lock);return __zswap_pool_current (); }
@@ -462,14 +485,15 @@ static struct zswap_pool *zswap_pool_current_get(void)return pool; } -/* type and compressor must be null-terminated */ +/* compressor must be null-terminated */ static struct zswap_pool *zswap_pool_find_get (char *compressor) {struct zswap_pool *pool; +unsigned long id; -assert_spin_locked (&zswap_pools_lock); +lockdep_assert_held (&zswap_pools.xa_lock); -list_for_each_entry_rcu (pool, &zswap_pools, list) { +xa_for_each (&zswap_pools, id, pool) {if (strcmp (pool->tfm_name, compressor))continue ;/* if we can't get it, it's about to be destroyed */
@@ -495,9 +519,15 @@ unsigned long zswap_total_pages(void) {struct zswap_pool *pool;unsigned long total =0 ; +unsigned long id; +/* +* rcu_read_lock() is required here, not just for xa_for_each(): it also +* keeps each pool alive while it is dereferenced, since a concurrently +* retired pool is freed via queue_rcu_work() after a grace period. +*/ rcu_read_lock (); -list_for_each_entry_rcu (pool, &zswap_pools, list) +xa_for_each (&zswap_pools, id, pool) total +=zs_get_total_pages (pool->zs_pool);rcu_read_unlock ();
@@ -554,20 +584,17 @@ static int zswap_compressor_param_set(const char *val, const struct kernel_paramreturn -ENOENT ; } -spin_lock_bh (&zswap_pools_lock); - +xa_lock_bh (&zswap_pools); pool =zswap_pool_find_get (s);if (pool) {zswap_pool_debug ("using existing" , pool);WARN_ON (pool ==zswap_pool_current ()); -list_del_rcu (&pool->list); } +xa_unlock_bh (&zswap_pools); -spin_unlock_bh (&zswap_pools_lock); - -if (!pool) +if (!pool) { pool =zswap_pool_create (s); -else { + }else {/* * Restore the initial ref dropped by percpu_ref_kill() * when the pool was decommissioned and switch it again
@@ -584,23 +611,17 @@ static int zswap_compressor_param_set(const char *val, const struct kernel_paramelse ret = -EINVAL ; -spin_lock_bh (&zswap_pools_lock); +xa_lock_bh (&zswap_pools);if (!ret) { put_pool =zswap_pool_current (); -list_add_rcu (&pool->list, &zswap_pools); +rcu_assign_pointer (zswap_current_pool, pool); zswap_has_pool = true; }else if (pool) { -/* -* Add the possibly pre-existing pool to the end of the pools -* list; if it's new (and empty) then it'll be removed and -* destroyed by the put after we drop the lock -*/ -list_add_tail_rcu (&pool->list, &zswap_pools); put_pool = pool; } -spin_unlock_bh (&zswap_pools_lock); +xa_unlock_bh (&zswap_pools);/* * Drop the ref from either the old current pool,
@@ -1788,7 +1809,7 @@ static int zswap_setup(void) pool =__zswap_pool_create_fallback ();if (pool) {pr_info ("loaded using pool %s\n" , pool->tfm_name); -list_add (&pool->list, &zswap_pools); +rcu_assign_pointer (zswap_current_pool, pool); zswap_has_pool = true;static_branch_enable (&zswap_ever_enabled); }else {
Signature · 1 line
2.43.0
Yosry Ahmed
maintainerRe: [PATCH v5 2/3] mm/zswap: replace the zswap_pools list with an allocating xarray
Quoted 131 lines · Jianyue Wu
Originally zswap keeps its pools on an RCU list whose head also serves as the current pool. Convert the pool table to an allocating xarray keyed by a small integer id, and track the current pool with a separate RCU-protected pointer.
The xarray gives each pool a stable id for a later
zswap_entryshrink.XA_FLAGS_ALLOC1starts ids at 1, so id 0 remains reserved. The id range is bounded byZSWAP_MAX_POOL_IDbecause the later entry field is a u8.Keep compressor switching close to the previous flow: look up an existing pool under
xa_lock, resurrect it outside the lock if reused, or create a new one.zswap_pool_create()allocates the pool's id and publishes it into the xarray as its final step, so the create call is itself atomic: it either fully builds the pool and publishes it, or unwinds completely on failure. Publishing makes the pool live, so a caller that later fails (e.g.param_set_charp()) must still kill the pool to erase it from the xarray. Runtime compressor parameter updates are serialized by the module parameter lock, so no speculative loser path is needed.A retiring pool is erased from the xarray in
__zswap_pool_empty()and freed viaqueue_rcu_work(), preserving the old RCU teardown ordering.Suggested-by Nhat Pham <[email protected]>Suggested-by Yosry Ahmed <[email protected]>Suggested-by Johannes Weiner <[email protected]>Signed-off-by Jianyue Wu <[email protected]>--- mm/zswap.c | 87 +++++++++++++++++++++++++++++++++--------------------- 1 file changed, 54 insertions(+), 33 deletions(-)mm/zswap.c +30
-8@@ -34,6 +34,7 @@#include <linux/writeback.h> #include <linux/pagemap.h> #include <linux/workqueue.h> +#include <linux/xarray.h> #include <linux/list_lru.h> #include <linux/zsmalloc.h> @@ -154,12 +155,24 @@ struct zswap_pool {struct zs_pool *zs_pool;struct crypto_acomp_ctx __percpu *acomp_ctx;struct percpu_ref ref; -struct list_head list;struct rcu_work release_rwork;struct hlist_node node; +u8 idx;char tfm_name[CRYPTO_MAX_ALG_NAME ]; }; +/* +* Live pools keyed by id (1..ZSWAP_MAX_POOL_ID). XA_FLAGS_ALLOC1 keeps +* the reserved id 0 unallocated, so looking it up never aliases a live +* pool. XA_FLAGS_LOCK_BH makes the xa_lock softirq-safe: it is taken +* from __zswap_pool_empty(), which runs from a percpu_ref release +* callback in softirq context. +*/ +#define ZSWAP_FIRST_POOL_ID 1 +#define ZSWAP_MAX_POOL_ID U8_MAX +static DEFINE_XARRAY_FLAGS(zswap_pools, XA_FLAGS_ALLOC1 | XA_FLAGS_LOCK_BH); +static struct zswap_pool __rcu *zswap_current_pool; +/* Global LRU lists shared by all zswap pools. */ static struct list_lru zswap_list_lru;@@ -200,10 +213,6 @@ struct zswap_entry {static struct xarray *zswap_trees[MAX_SWAPFILES ];static unsigned int nr_zswap_trees[MAX_SWAPFILES ]; -/* RCU-protected iteration */ -static LIST_HEAD (zswap_pools); -/* protects zswap_pools list modification */ -static DEFINE_SPINLOCK (zswap_pools_lock);/* pool counter to provide unique names to zsmalloc */ static atomic_t zswap_pools_count =ATOMIC_INIT (0 );@@ -275,6 +284,7 @@ static struct zswap_pool *zswap_pool_create(char *compressor)struct zswap_pool *pool;char name[38 ];/* 'zswap' + 32 char (max) num + \0 */ int ret, cpu; +u32 id;if (!zswap_has_pool && !strcmp (compressor,ZSWAP_PARAM_UNSET ))return NULL ;@@ -320,12 +330,24 @@ static struct zswap_pool *zswap_pool_create(char *compressor) PERCPU_REF_ALLOW_REINIT, GFP_KERNEL);if (ret) goto ref_fail; -INIT_LIST_HEAD (&pool->list); + + ret =xa_alloc_bh (&zswap_pools, &id, pool, +XA_LIMIT (ZSWAP_FIRST_POOL_ID ,ZSWAP_MAX_POOL_ID ), +GFP_KERNEL ); +if (ret) { +if (ret == -EBUSY ) +pr_err ("cannot allocate pool id (max %d live pools)\n" , +ZSWAP_MAX_POOL_ID -ZSWAP_FIRST_POOL_ID +1 ); + goto xa_fail; + } + pool->idx = id;zswap_pool_debug ("created" , pool);return pool; +xa_fail: +percpu_ref_exit (&pool->ref); ref_fail:cpuhp_state_remove_instance (CPUHP_MM_ZSWP_POOL_PREPARE , &pool->node);@@ -386,7 +408,7 @@ static void __zswap_pool_release(struct work_struct *work)WARN_ON (!percpu_ref_is_zero (&pool->ref));percpu_ref_exit (&pool->ref); -/* pool is now off zswap_pools list and has no references. */ +/* The pool is no longer in zswap_pools and has no references. */ zswap_pool_destroy (pool); }@@ -398,16 +420,16 @@ static void __zswap_pool_empty(struct percpu_ref *ref) pool =container_of (ref,typeof (*pool), ref); -spin_lock_bh (&zswap_pools_lock); +xa_lock_bh (&zswap_pools);
mm/zswap.c static void __zswap_pool_empty(struct percpu_ref *ref) · line 432
Quoted 10 lines
mm/zswap.c +2
-2WARN_ON (pool ==zswap_pool_current ()); -list_del_rcu (&pool->list); +__xa_erase (&zswap_pools, pool->idx);INIT_RCU_WORK (&pool->release_rwork, __zswap_pool_release);queue_rcu_work (system_percpu_wq, &pool->release_rwork); -spin_unlock_bh (&zswap_pools_lock); +xa_unlock_bh (&zswap_pools);
Do we need to call queue_rcu_work() under the lock? I assume not. Can we just call xa_erase_bh()?
Quoted 42 lines
mm/zswap.c +7
-5}static int __must_check zswap_pool_tryget (struct zswap_pool *pool)@@ -433,7 +455,8 @@ static struct zswap_pool *__zswap_pool_current(void) {struct zswap_pool *pool; - pool =list_first_or_null_rcu (&zswap_pools,typeof (*pool), list); + pool =rcu_dereference_check (zswap_current_pool, +lockdep_is_held (&zswap_pools.xa_lock));WARN_ONCE (!pool && zswap_has_pool,"%s: no page storage pool!\n" , __func__);@@ -442,7 +465,7 @@ static struct zswap_pool *__zswap_pool_current(void)static struct zswap_pool *zswap_pool_current (void ) { -assert_spin_locked (&zswap_pools_lock); +lockdep_assert_held (&zswap_pools.xa_lock);return __zswap_pool_current (); }@@ -462,14 +485,15 @@ static struct zswap_pool *zswap_pool_current_get(void)return pool; } -/* type and compressor must be null-terminated */ +/* compressor must be null-terminated */ static struct zswap_pool *zswap_pool_find_get (char *compressor) {struct zswap_pool *pool; +unsigned long id; -assert_spin_locked (&zswap_pools_lock); +lockdep_assert_held (&zswap_pools.xa_lock); -list_for_each_entry_rcu (pool, &zswap_pools, list) { +xa_for_each (&zswap_pools, id, pool) {if (strcmp (pool->tfm_name, compressor))continue ;/* if we can't get it, it's about to be destroyed */
mm/zswap.c unsigned long zswap_total_pages(void) · line 528
Quoted 10 lines
mm/zswap.c +6
-0@@ -495,9 +519,15 @@ unsigned long zswap_total_pages(void) {struct zswap_pool *pool;unsigned long total =0 ; +unsigned long id; +/* +* rcu_read_lock() is required here, not just for xa_for_each(): it also +* keeps each pool alive while it is dereferenced, since a concurrently +* retired pool is freed via queue_rcu_work() after a grace period. +*/
Doesn't xa_for_each() already handle RCU locking?
Quoted 7 lines
mm/zswap.c +1
-1rcu_read_lock (); -list_for_each_entry_rcu (pool, &zswap_pools, list) +xa_for_each (&zswap_pools, id, pool) total +=zs_get_total_pages (pool->zs_pool);rcu_read_unlock ();
mm/zswap.c static int zswap_compressor_param_set(const char *val, const struct kernel_param · line 588
Quoted 7 lines
mm/zswap.c +1
-2@@ -554,20 +584,17 @@ static int zswap_compressor_param_set(const char *val, const struct kernel_paramreturn -ENOENT ; } -spin_lock_bh (&zswap_pools_lock); - +xa_lock_bh (&zswap_pools); pool =zswap_pool_find_get (s);
This is the only caller of zswap_pool_find_get(), and since we remove list_del_rcu() below we have no reason for holding the lock here other than zswap_pool_find_get(). So let's move the locking inside? I am also not really sure if we actually need to hold the lock here anymore to begin with.
Quoted 18 lines
mm/zswap.c +3
-5if (pool) {zswap_pool_debug ("using existing" , pool);WARN_ON (pool ==zswap_pool_current ()); -list_del_rcu (&pool->list); } +xa_unlock_bh (&zswap_pools); -spin_unlock_bh (&zswap_pools_lock); - -if (!pool) +if (!pool) { pool =zswap_pool_create (s); -else { + }else {/* * Restore the initial ref dropped by percpu_ref_kill() * when the pool was decommissioned and switch it again
mm/zswap.c static int zswap_compressor_param_set(const char *val, const struct kernel_param · line 614
Quoted 5 lines
mm/zswap.c +1
-1@@ -584,23 +611,17 @@ static int zswap_compressor_param_set(const char *val, const struct kernel_paramelse ret = -EINVAL ; -spin_lock_bh (&zswap_pools_lock); +xa_lock_bh (&zswap_pools);
Do we still need to hold the lock here?
Quoted 33 lines
mm/zswap.c +3
-10if (!ret) { put_pool =zswap_pool_current (); -list_add_rcu (&pool->list, &zswap_pools); +rcu_assign_pointer (zswap_current_pool, pool); zswap_has_pool = true; }else if (pool) { -/* -* Add the possibly pre-existing pool to the end of the pools -* list; if it's new (and empty) then it'll be removed and -* destroyed by the put after we drop the lock -*/ -list_add_tail_rcu (&pool->list, &zswap_pools); put_pool = pool; } -spin_unlock_bh (&zswap_pools_lock); +xa_unlock_bh (&zswap_pools);/* * Drop the ref from either the old current pool, @@ -1788,7 +1809,7 @@ static int zswap_setup(void) pool =__zswap_pool_create_fallback ();if (pool) {pr_info ("loaded using pool %s\n" , pool->tfm_name); -list_add (&pool->list, &zswap_pools); +rcu_assign_pointer (zswap_current_pool, pool); zswap_has_pool = true;static_branch_enable (&zswap_ever_enabled); }else { --2.43.0
Re: [PATCH v5 2/3] mm/zswap: replace the zswap_pools list with an allocating xarray
Quoted 4 lines · Yosry Ahmed
Quoted 4 lines
mm/zswap.c +1
-1-list_del_rcu (&pool->list); +__xa_erase (&zswap_pools, pool->idx);
Quoted 8 lines
Review
Quoted 5 lines
INIT_RCU_WORK(&pool->release_rwork, __zswap_pool_release); queue_rcu_work(system_percpu_wq, &pool->release_rwork);(quoted diff) +1
-1- spin_unlock_bh(&zswap_pools_lock); + xa_unlock_bh(&zswap_pools);Do we need to call
queue_rcu_work()under the lock? I assume not. Can we just callxa_erase_bh()?
Right, no need. queue_rcu_work() only touches the retiring pool's own release_rwork. xa_erase_bh(&zswap_pools, pool->idx) is enough.
Quoted 8 lines
Doesn't
xa_for_each()already handle RCU locking?Quoted 5 lines
rcu_read_lock();mm/zswap.c +1
-1-list_for_each_entry_rcu (pool, &zswap_pools, list) +xa_for_each (&zswap_pools, id, pool) total +=zs_get_total_pages (pool->zs_pool);rcu_read_unlock ();
xa_for_each() only takes RCU around each internal lookup, not across the loop body, so the explicit rcu_read_lock() is still needed to keep a concurrently retired pool alive while we dereference pool->zs_pool.
Quoted 4 lines
Quoted 4 lines
mm/zswap.c +0
-0@@ -554,20 +584,17 @@ static int zswap_compressor_param_set(const char *val, const struct kernel_paramreturn -ENOENT ; }
Quoted 10 lines
mm/zswap.c static int zswap_compressor_param_set(const char *val, const struct kernel_param · line 588
Quoted 4 lines
mm/zswap.c +1
-2-spin_lock_bh (&zswap_pools_lock); - +xa_lock_bh (&zswap_pools); pool =zswap_pool_find_get (s);This is the only caller of
zswap_pool_find_get(), and since we removelist_del_rcu()below we have no reason for holding the lock here other thanzswap_pool_find_get(). So let's move the locking inside? I am also not really sure if we actually need to hold the lock here anymore to begin with.
Agreed, and it turns out we don't need xa_lock here at all.
Quoted 9 lines
mm/zswap.c static int zswap_compressor_param_set(const char *val, const struct kernel_param · line 614
Quoted 6 lines
mm/zswap.c +1
-1@@ -584,23 +611,17 @@ static int zswap_compressor_param_set(const char *val, const struct kernel_paramelse ret = -EINVAL ; -spin_lock_bh (&zswap_pools_lock); +xa_lock_bh (&zswap_pools);Do we still need to hold the lock here?
Agreed, no need lock here. Compressor switches are serialized by the module parameter lock, so this is the only writer of zswap_current_pool.
Thanks for the review!
Best regards,
Jianyue
PATCH v5 3/3 mm/zswap: reference the pool by id to shrink struct zswap_entry
In reply to Jianyue Wustruct zswap_entry is one allocation per stored page, so its size is pure overhead. It currently embeds an 8-byte pool pointer, even though the live pools now sit in an allocating xarray keyed by a small integer id that fits in a u8.
Replace the per-entry pool pointer with that u8 id and resolve it through the xarray with xa_load(). A live entry holds a reference to its pool, so the id cannot be reused under it; xa_load() is lockless and only needs an rcu_read_lock() section, no zswap_pools_lock. A live entry never uses the reserved id 0, so looking up that id resolves to NULL and trips a WARN rather than aliasing a live pool.
The u8 fits in the padding after the bool referenced field, shrinking the entry from 56 to 48 bytes on 64-bit. This raises objs_per_slab from 73 to 85 and saves about 2MiB of metadata per 1GiB of data held in zswap.
--- mm/zswap.c | 45 ++++++++++++++++++++++++++++++++++++++------- 1 file changed, 38 insertions(+), 7 deletions(-)
mm/zswap.c +38 -7
@@ -195,7 +195,7 @@ static struct shrinker *zswap_shrinker;* writeback logic. The entry is only reclaimed by the writeback * logic if referenced is unset. See comments in the shrinker * section for context. -* pool - the zswap_pool the entry's data is in +* pool_idx - id of the zswap_pool that the entry's data is in. * handle - zsmalloc allocation handle that stores the compressed page data * objcg - the obj_cgroup that the compressed memory is charged to * lru - handle to the pool's lru used to evict pages.
@@ -204,12 +204,27 @@ struct zswap_entry {swp_entry_t swpentry;unsigned int length;bool referenced; -struct zswap_pool *pool; +u8 pool_idx;unsigned long handle;struct obj_cgroup *objcg;struct list_head lru; }; +/* +* The pool stays alive after this returns because a stored entry holds a +* reference to its pool (taken in zswap_store_page()). +*/ +static struct zswap_pool *zswap_entry_pool (struct zswap_entry *entry) +{ +struct zswap_pool *pool; + +rcu_read_lock (); + pool =xa_load (&zswap_pools, entry->pool_idx); +rcu_read_unlock (); + +return pool; +} +static struct xarray *zswap_trees[MAX_SWAPFILES ];static unsigned int nr_zswap_trees[MAX_SWAPFILES ];
@@ -770,9 +785,13 @@ static void zswap_entry_cache_free(struct zswap_entry *entry)*/ static void zswap_entry_free (struct zswap_entry *entry) { +struct zswap_pool *pool =zswap_entry_pool (entry); +zswap_lru_del (entry); -zs_free (entry->pool->zs_pool, entry->handle); -zswap_pool_put (entry->pool); +if (!WARN_ON_ONCE (!pool)) { +zs_free (pool->zs_pool, entry->handle); +zswap_pool_put (pool); + }if (entry->objcg) {obj_cgroup_uncharge_zswap (entry->objcg, entry->length);obj_cgroup_put (entry->objcg);
@@ -929,12 +948,15 @@ static bool zswap_compress(struct page *page, struct zswap_entry *entry,static bool zswap_decompress (struct zswap_entry *entry,struct folio *folio) { -struct zswap_pool *pool = entry->pool; +struct zswap_pool *pool =zswap_entry_pool (entry);struct scatterlist input[2 ];/* zsmalloc returns an SG list 1-2 entries */ struct scatterlist output;struct crypto_acomp_ctx *acomp_ctx;int ret =0 , dlen; +if (WARN_ON_ONCE (!pool)) +return false; + acomp_ctx =raw_cpu_ptr (pool->acomp_ctx);mutex_lock (&acomp_ctx->mutex);zs_obj_read_sg_begin (pool->zs_pool, entry->handle, input, entry->length);
@@ -970,7 +992,7 @@ static bool zswap_decompress(struct zswap_entry *entry, struct folio *folio)pr_alert_ratelimited ("Decompression error from zswap (%d:%lu %s %u->%d)\n" ,swp_type (entry->swpentry),swp_offset (entry->swpentry), - entry->pool->tfm_name, + pool->tfm_name, entry->length, dlen);return false; }
@@ -1428,6 +1450,16 @@ static bool zswap_store_page(struct page *page,if (!zswap_compress (page, entry, pool)) goto compress_failed; +/* +* Set pool_idx before publishing the entry: compression has +* succeeded and the pool is already pinned by this store, so the id is +* final. Doing it here (rather than after xa_store()) means the entry +* is never briefly visible with a stale pool_idx left over from slab +* reuse, which zswap_entry_pool() would otherwise resolve to an +* unrelated live pool. +*/ + entry->pool_idx = pool->idx; + old =xa_store (swap_zswap_tree (page_swpentry),swp_offset (page_swpentry), entry,GFP_KERNEL );
@@ -1473,7 +1505,6 @@ static bool zswap_store_page(struct page *page,* The publishing order matters to prevent writeback from seeing * an incoherent entry. */ - entry->pool = pool; entry->swpentry = page_swpentry; entry->objcg = objcg; entry->referenced = true;
Signature · 1 line
2.43.0
Yosry Ahmed
maintainerRe: [PATCH v5 3/3] mm/zswap: reference the pool by id to shrink struct zswap_entry
Quoted 49 lines · Jianyue Wu
struct
zswap_entryis one allocation per stored page, so its size is pure overhead. It currently embeds an 8-byte pool pointer, even though the live pools now sit in an allocating xarray keyed by a small integer id that fits in a u8.Replace the per-entry pool pointer with that u8 id and resolve it through the xarray with
xa_load(). A live entry holds a reference to its pool, so the id cannot be reused under it;xa_load()is lockless and only needs anrcu_read_lock()section, nozswap_pools_lock. A live entry never uses the reserved id 0, so looking up that id resolves to NULL and trips a WARN rather than aliasing a live pool.The u8 fits in the padding after the bool referenced field, shrinking the entry from 56 to 48 bytes on 64-bit. This raises
objs_per_slabfrom 73 to 85 and saves about 2MiB of metadata per 1GiB of data held in zswap.Suggested-by Chris Li <[email protected]>Signed-off-by Jianyue Wu <[email protected]>--- mm/zswap.c | 45 ++++++++++++++++++++++++++++++++++++++------- 1 file changed, 38 insertions(+), 7 deletions(-)mm/zswap.c +3
-2@@ -195,7 +195,7 @@ static struct shrinker *zswap_shrinker;* writeback logic. The entry is only reclaimed by the writeback * logic if referenced is unset. See comments in the shrinker * section for context. -* pool - the zswap_pool the entry's data is in +* pool_idx - id of the zswap_pool that the entry's data is in. * handle - zsmalloc allocation handle that stores the compressed page data * objcg - the obj_cgroup that the compressed memory is charged to * lru - handle to the pool's lru used to evict pages. @@ -204,12 +204,27 @@ struct zswap_entry {swp_entry_t swpentry;unsigned int length;bool referenced; -struct zswap_pool *pool; +u8 pool_idx;unsigned long handle;struct obj_cgroup *objcg;struct list_head lru; }; +/*
mm/zswap.c struct zswap_entry { · line 223
Quoted 10 lines
mm/zswap.c +10
-0+* The pool stays alive after this returns because a stored entry holds a +* reference to its pool (taken in zswap_store_page()). +*/ +static struct zswap_pool *zswap_entry_pool (struct zswap_entry *entry) +{ +struct zswap_pool *pool; + +rcu_read_lock (); + pool =xa_load (&zswap_pools, entry->pool_idx); +rcu_read_unlock ();
Doesn't xa_load() already call rcu_read_[un]lock()?
Quoted 77 lines
mm/zswap.c +25
-6+ +return pool; +} +static struct xarray *zswap_trees[MAX_SWAPFILES ];static unsigned int nr_zswap_trees[MAX_SWAPFILES ];@@ -770,9 +785,13 @@ static void zswap_entry_cache_free(struct zswap_entry *entry)*/ static void zswap_entry_free (struct zswap_entry *entry) { +struct zswap_pool *pool =zswap_entry_pool (entry); +zswap_lru_del (entry); -zs_free (entry->pool->zs_pool, entry->handle); -zswap_pool_put (entry->pool); +if (!WARN_ON_ONCE (!pool)) { +zs_free (pool->zs_pool, entry->handle); +zswap_pool_put (pool); + }if (entry->objcg) {obj_cgroup_uncharge_zswap (entry->objcg, entry->length);obj_cgroup_put (entry->objcg);@@ -929,12 +948,15 @@ static bool zswap_compress(struct page *page, struct zswap_entry *entry,static bool zswap_decompress (struct zswap_entry *entry,struct folio *folio) { -struct zswap_pool *pool = entry->pool; +struct zswap_pool *pool =zswap_entry_pool (entry);struct scatterlist input[2 ];/* zsmalloc returns an SG list 1-2 entries */ struct scatterlist output;struct crypto_acomp_ctx *acomp_ctx;int ret =0 , dlen; +if (WARN_ON_ONCE (!pool)) +return false; + acomp_ctx =raw_cpu_ptr (pool->acomp_ctx);mutex_lock (&acomp_ctx->mutex);zs_obj_read_sg_begin (pool->zs_pool, entry->handle, input, entry->length);@@ -970,7 +992,7 @@ static bool zswap_decompress(struct zswap_entry *entry, struct folio *folio)pr_alert_ratelimited ("Decompression error from zswap (%d:%lu %s %u->%d)\n" ,swp_type (entry->swpentry),swp_offset (entry->swpentry), - entry->pool->tfm_name, + pool->tfm_name, entry->length, dlen);return false; }@@ -1428,6 +1450,16 @@ static bool zswap_store_page(struct page *page,if (!zswap_compress (page, entry, pool)) goto compress_failed; +/* +* Set pool_idx before publishing the entry: compression has +* succeeded and the pool is already pinned by this store, so the id is +* final. Doing it here (rather than after xa_store()) means the entry +* is never briefly visible with a stale pool_idx left over from slab +* reuse, which zswap_entry_pool() would otherwise resolve to an +* unrelated live pool. +*/ + entry->pool_idx = pool->idx; + old =xa_store (swap_zswap_tree (page_swpentry),swp_offset (page_swpentry), entry,GFP_KERNEL );@@ -1473,7 +1505,6 @@ static bool zswap_store_page(struct page *page,* The publishing order matters to prevent writeback from seeing * an incoherent entry. */ - entry->pool = pool; entry->swpentry = page_swpentry; entry->objcg = objcg; entry->referenced = true; --2.43.0
Re: [PATCH v5 3/3] mm/zswap: reference the pool by id to shrink struct zswap_entry
Quoted 5 lines · Yosry Ahmed
Quoted 5 lines
mm/zswap.c +5
-0+/* +* The pool stays alive after this returns because a stored entry holds a +* reference to its pool (taken in zswap_store_page()). +*/ +static struct zswap_pool *zswap_entry_pool (struct zswap_entry *entry)
Quoted 9 lines
mm/zswap.c struct zswap_entry { · line 223
Quoted 6 lines
mm/zswap.c +6
-0+{ +struct zswap_pool *pool; + +rcu_read_lock (); + pool =xa_load (&zswap_pools, entry->pool_idx); +rcu_read_unlock ();Doesn't
xa_load()already call rcu_read_[un]lock()?
Yes, xa_load() does the lookup under RCU internally, so the wrapping rcu_read_lock()/unlock() is redundant, and will remove it.
Best regards,
Jianyue
v6 cover mm/zswap: shrink zswap_entry via a pool id
In reply to Jianyue WuEvery stored page has a struct zswap_entry, so its size is pure per-page overhead. On 64-bit it is currently 56 bytes, of which 8 bytes are a pointer to the owning zswap_pool.
Only a handful of pools are ever live: a new pool is created only when the compressor is (re)set, and pools are reused across compressor switches. A list cannot look a pool up by id. An allocating xarray can, which lets each zswap_entry store a u8 instead of a pointer.
This series:
1. Releases retired pools with queue_rcu_work() instead of a worker calling synchronize_rcu(), so the release worker no longer blocks on an RCU grace period.
2. Replaces the zswap_pools list with an allocating xarray (XA_FLAGS_ALLOC1 | XA_FLAGS_LOCK_BH) and a separate RCU-protected current-pool pointer, giving each pool a stable small id. Ids start at 1. The reserved id 0 is never allocated, so looking it up resolves to NULL. The table grows as needed up to 255 live pools (u8 pool_idx), not a fixed slot array.
3. Stores that u8 pool id in each zswap_entry instead of the pool pointer. The u8 fits in padding after the bool referenced field.
On 64-bit that shrinks the entry from 56 to 48 bytes, which fits 73 to 85 objects in a 4K slab (~2MiB of metadata saved per 1GiB of data held in zswap).
The 255-id cap counts every pool still in the xarray. Switching compressor kills the old pool, but that pool stays in the table until its last entry drops the pool's ref, so a draining pool still occupies an id. The id is reused only after xa_erase. Switching back to a compressor whose pool is still in the table resurrects it instead of allocating a new id. If every id is occupied, creating a pool for another compressor fails and the switch is rejected.
Pool table locking: xa_for_each() walks and the current-pool pointer use an explicit rcu_read_lock(), because xa_for_each()'s own RCU does not span the loop body. xa_load() takes RCU around the lookup itself, so zswap_entry_pool() needs no extra rcu_read_lock(). The returned pool stays valid because a live entry pins it via percpu_ref, so the id cannot be reused under it. xa_lock is taken only in xa_alloc_bh() and xa_erase_bh().
Benchmark (x86_64, compressor=lzo, MADV_PAGEOUT store + fault-in load):
- e2e store+load median latency: no measurable regression vs baseline at matched stored_delta
Each store, free, and decompress looks up the pool with xa_load() instead of following a pointer. With only a handful of live pools the xarray walk is short.
Testing
=======
- Boot with DEBUG_ATOMIC_SLEEP + lockdep/PROVE_RCU + KASAN:
zswap store/load, shrinker writeback, and compressor switch (retire, then switch back to resurrect) pass
This series is based on akpm/mm-unstable as of 2026-09-05 (d118502628f8).
To: Johannes Weiner <[email protected]>
To: Yosry Ahmed <[email protected]>
To: Nhat Pham <[email protected]>
To: Chengming Zhou <[email protected]>
To: Andrew Morton <[email protected]>
Changes since v5:
- Use xa_erase_bh() and move queue_rcu_work() out of xa_lock() in
__zswap_pool_empty().
- Drop the redundant rcu_read_lock() around zswap_entry_pool(). xa_load()
already takes RCU and a live entry pins its pool via percpu_ref.
- Replace xa_lock() protected pool walks with RCU. xa_lock() is now only for
xa_alloc_bh() and xa_erase_bh(). - Drop zswap_pool_current(), it only read the current pool under xa_lock(), which this path no longer takes.
- Add a comment in zswap_total_pages() on why the outer rcu_read_lock() remains (xa_for_each()'s RCU does not span the loop body).
- Rebase onto akpm/mm-unstable d118502628f8.
---
Jianyue Wu (3):
mm/zswap: release retired pools via queue_rcu_work() instead of synchronize_rcu() mm/zswap: replace the zswap_pools list with an allocating xarray mm/zswap: reference the pool by id to shrink struct zswap_entry
mm/zswap.c | 171 ++++++++++++++++++++++++++++++++++++------------------------- 1 file changed, 101 insertions(+), 70 deletions(-)
---
base-commit: d118502628f8b673be9023db8bdf878f64a7ed45
change-id: 20260906-shrink_zswap_entry_v6-ed3d14c9001f
Best regards,
Signature · 1 line
Jianyue Wu <[email protected]>
PATCH v6 1/3 mm/zswap: release retired pools via queue_rcu_work() instead of synchronize_rcu()
When a pool's last reference is dropped, __zswap_pool_empty() removes it from the pool list and schedules __zswap_pool_release(), which calls synchronize_rcu() to wait for readers before tearing the pool down.
synchronize_rcu() is a synchronous, potentially long wait. Replace it with queue_rcu_work(): __zswap_pool_empty() hands the pool to queue_rcu_work(), which waits for a grace period asynchronously and then runs __zswap_pool_release() from a worker for the sleepable teardown (__zswap_pool_empty() can run in atomic context and must not block). The grace-period guarantee is unchanged; the retirement path just no longer blocks on it.
--- mm/zswap.c | 12 +++++------- 1 file changed, 5 insertions(+), 7 deletions(-)
mm/zswap.c +5 -7
@@ -155,7 +155,7 @@ struct zswap_pool {struct crypto_acomp_ctx __percpu *acomp_ctx;struct percpu_ref ref;struct list_head list; -struct work_struct release_work; +struct rcu_work release_rwork;struct hlist_node node;char tfm_name[CRYPTO_MAX_ALG_NAME ]; };
@@ -379,10 +379,8 @@ static void zswap_pool_destroy(struct zswap_pool *pool)static void __zswap_pool_release (struct work_struct *work) { -struct zswap_pool *pool =container_of (work,typeof (*pool), - release_work); - -synchronize_rcu (); +struct zswap_pool *pool =container_of (to_rcu_work (work), +typeof (*pool), release_rwork);/* nobody should have been able to get a ref... */ WARN_ON (!percpu_ref_is_zero (&pool->ref));
@@ -406,8 +404,8 @@ static void __zswap_pool_empty(struct percpu_ref *ref)list_del_rcu (&pool->list); -INIT_WORK (&pool->release_work, __zswap_pool_release); -schedule_work (&pool->release_work); +INIT_RCU_WORK (&pool->release_rwork, __zswap_pool_release); +queue_rcu_work (system_percpu_wq, &pool->release_rwork);spin_unlock_bh (&zswap_pools_lock); }
Signature · 1 line
2.43.0
PATCH v6 2/3 mm/zswap: replace the zswap_pools list with an allocating xarray
In reply to Jianyue WuOriginally zswap kept its pools on an RCU list whose head also served as the current pool. Convert the pool table to an allocating xarray keyed by a small integer id, and track the current pool with a separate RCU-protected pointer.
The xarray gives each pool a stable id for a later zswap_entry shrink. XA_FLAGS_ALLOC1 starts ids at 1, so id 0 remains reserved. The id range is bounded by ZSWAP_MAX_POOL_ID because the later entry field is a u8.
Keep compressor switching close to the previous flow: look up an existing pool with xa_for_each(), resurrect it if reused, or create a new one. zswap_pool_create() allocates the pool's id and publishes it into the xarray as its final step, so the create call either fully publishes or fully unwinds on failure. Publishing makes the pool live, so a caller that later fails (e.g. param_set_charp()) must still kill the pool to erase it from the xarray.
Compressor switches update zswap_current_pool with rcu_assign_pointer(), serialized by the module parameter lock, so no xa_lock is needed for that update. The pool walk above is lockless under RCU. xa_lock is taken only to allocate (xa_alloc_bh()) and erase (xa_erase_bh()) xarray entries.
--- mm/zswap.c | 122 +++++++++++++++++++++++++++++++++---------------------------- 1 file changed, 66 insertions(+), 56 deletions(-)
mm/zswap.c +66 -56
@@ -34,6 +34,7 @@#include <linux/writeback.h> #include <linux/pagemap.h> #include <linux/workqueue.h> +#include <linux/xarray.h> #include <linux/list_lru.h> #include <linux/zsmalloc.h>
@@ -154,12 +155,23 @@ struct zswap_pool {struct zs_pool *zs_pool;struct crypto_acomp_ctx __percpu *acomp_ctx;struct percpu_ref ref; -struct list_head list;struct rcu_work release_rwork;struct hlist_node node; +u8 idx;char tfm_name[CRYPTO_MAX_ALG_NAME ]; }; +/* +* Live pools keyed by id (1..ZSWAP_MAX_POOL_ID). XA_FLAGS_ALLOC1 keeps id 0 +* reserved so it is never handed to a live pool. XA_FLAGS_LOCK_BH makes the +* xa_lock softirq-safe: it is taken from __zswap_pool_empty(), which runs from +* a percpu_ref release callback in softirq context. +*/ +#define ZSWAP_FIRST_POOL_ID 1 +#define ZSWAP_MAX_POOL_ID U8_MAX +static DEFINE_XARRAY_FLAGS(zswap_pools, XA_FLAGS_ALLOC1 | XA_FLAGS_LOCK_BH); +static struct zswap_pool __rcu *zswap_current_pool; +/* Global LRU lists shared by all zswap pools. */ static struct list_lru zswap_list_lru;
@@ -200,10 +212,6 @@ struct zswap_entry {static struct xarray *zswap_trees[MAX_SWAPFILES ];static unsigned int nr_zswap_trees[MAX_SWAPFILES ]; -/* RCU-protected iteration */ -static LIST_HEAD (zswap_pools); -/* protects zswap_pools list modification */ -static DEFINE_SPINLOCK (zswap_pools_lock);/* pool counter to provide unique names to zsmalloc */ static atomic_t zswap_pools_count =ATOMIC_INIT (0 );
@@ -275,6 +283,7 @@ static struct zswap_pool *zswap_pool_create(char *compressor)struct zswap_pool *pool;char name[38 ];/* 'zswap' + 32 char (max) num + \0 */ int ret, cpu; +u32 id;if (!zswap_has_pool && !strcmp (compressor,ZSWAP_PARAM_UNSET ))return NULL ;
@@ -320,12 +329,29 @@ static struct zswap_pool *zswap_pool_create(char *compressor) PERCPU_REF_ALLOW_REINIT, GFP_KERNEL);if (ret) goto ref_fail; -INIT_LIST_HEAD (&pool->list); + +/* +* Publish only after the pool is fully built, so lockless walkers +* never see a half-initialized pool. The _bh variant pairs with the +* softirq-context xa_lock taken in __zswap_pool_empty(). +*/ + ret =xa_alloc_bh (&zswap_pools, &id, pool, +XA_LIMIT (ZSWAP_FIRST_POOL_ID ,ZSWAP_MAX_POOL_ID ), +GFP_KERNEL ); +if (ret) { +if (ret == -EBUSY ) +pr_err ("cannot allocate pool id (max %d live pools)\n" , +ZSWAP_MAX_POOL_ID -ZSWAP_FIRST_POOL_ID +1 ); + goto xa_fail; + } + pool->idx = id;zswap_pool_debug ("created" , pool);return pool; +xa_fail: +percpu_ref_exit (&pool->ref); ref_fail:cpuhp_state_remove_instance (CPUHP_MM_ZSWP_POOL_PREPARE , &pool->node);
@@ -386,28 +412,22 @@ static void __zswap_pool_release(struct work_struct *work)WARN_ON (!percpu_ref_is_zero (&pool->ref));percpu_ref_exit (&pool->ref); -/* pool is now off zswap_pools list and has no references. */ +/* The pool is no longer in zswap_pools and has no references. */ zswap_pool_destroy (pool); } -static struct zswap_pool *zswap_pool_current (void ); -static void __zswap_pool_empty (struct percpu_ref *ref) {struct zswap_pool *pool; pool =container_of (ref,typeof (*pool), ref); -spin_lock_bh (&zswap_pools_lock); - -WARN_ON (pool ==zswap_pool_current ()); +WARN_ON (pool ==rcu_access_pointer (zswap_current_pool)); -list_del_rcu (&pool->list); +xa_erase_bh (&zswap_pools, pool->idx);INIT_RCU_WORK (&pool->release_rwork, __zswap_pool_release);queue_rcu_work (system_percpu_wq, &pool->release_rwork); - -spin_unlock_bh (&zswap_pools_lock); }static int __must_check zswap_pool_tryget (struct zswap_pool *pool)
@@ -433,20 +453,13 @@ static struct zswap_pool *__zswap_pool_current(void) {struct zswap_pool *pool; - pool =list_first_or_null_rcu (&zswap_pools,typeof (*pool), list); + pool =rcu_dereference (zswap_current_pool);WARN_ONCE (!pool && zswap_has_pool,"%s: no page storage pool!\n" , __func__);return pool; } -static struct zswap_pool *zswap_pool_current (void ) -{ -assert_spin_locked (&zswap_pools_lock); - -return __zswap_pool_current (); -} -static struct zswap_pool *zswap_pool_current_get (void ) {struct zswap_pool *pool;
@@ -462,23 +475,28 @@ static struct zswap_pool *zswap_pool_current_get(void)return pool; } -/* type and compressor must be null-terminated */ +/* compressor must be null-terminated */ static struct zswap_pool *zswap_pool_find_get (char *compressor) {struct zswap_pool *pool; +unsigned long id; -assert_spin_locked (&zswap_pools_lock); - -list_for_each_entry_rcu (pool, &zswap_pools, list) { +/* +* __zswap_pool_empty() can erase from zswap_pools in softirq while we +* walk. rcu_read_lock() keeps the walk consistent and each pool alive +* across tryget(). xa_for_each()'s own RCU does not span the loop body. +*/ +rcu_read_lock (); +xa_for_each (&zswap_pools, id, pool) {if (strcmp (pool->tfm_name, compressor))continue ;/* if we can't get it, it's about to be destroyed */ -if (!zswap_pool_tryget (pool)) -continue ; -return pool; +if (zswap_pool_tryget (pool)) +break ; } +rcu_read_unlock (); -return NULL ; +return pool; }static unsigned long zswap_max_pages (void )
@@ -495,9 +513,14 @@ unsigned long zswap_total_pages(void) {struct zswap_pool *pool;unsigned long total =0 ; +unsigned long id; +/* +* rcu_read_lock() keeps each pool alive across zs_get_total_pages(). +* xa_for_each()'s own RCU does not span the loop body. +*/ rcu_read_lock (); -list_for_each_entry_rcu (pool, &zswap_pools, list) +xa_for_each (&zswap_pools, id, pool) total +=zs_get_total_pages (pool->zs_pool);rcu_read_unlock ();
@@ -554,20 +577,13 @@ static int zswap_compressor_param_set(const char *val, const struct kernel_paramreturn -ENOENT ; } -spin_lock_bh (&zswap_pools_lock); - pool =zswap_pool_find_get (s); -if (pool) { +if (!pool) { + pool =zswap_pool_create (s); + }else {zswap_pool_debug ("using existing" , pool); -WARN_ON (pool ==zswap_pool_current ()); -list_del_rcu (&pool->list); - } - -spin_unlock_bh (&zswap_pools_lock); +WARN_ON (pool ==rcu_access_pointer (zswap_current_pool)); -if (!pool) - pool =zswap_pool_create (s); -else {/* * Restore the initial ref dropped by percpu_ref_kill() * when the pool was decommissioned and switch it again
@@ -584,24 +600,18 @@ static int zswap_compressor_param_set(const char *val, const struct kernel_paramelse ret = -EINVAL ; -spin_lock_bh (&zswap_pools_lock); - +/* +* Compressor switches are serialized by the kernel param lock, so this +* is the only writer of zswap_current_pool: no xa_lock needed. +*/ if (!ret) { - put_pool =zswap_pool_current (); -list_add_rcu (&pool->list, &zswap_pools); + put_pool =rcu_access_pointer (zswap_current_pool); +rcu_assign_pointer (zswap_current_pool, pool); zswap_has_pool = true; }else if (pool) { -/* -* Add the possibly pre-existing pool to the end of the pools -* list; if it's new (and empty) then it'll be removed and -* destroyed by the put after we drop the lock -*/ -list_add_tail_rcu (&pool->list, &zswap_pools); put_pool = pool; } -spin_unlock_bh (&zswap_pools_lock); -/* * Drop the ref from either the old current pool, * or the new pool we failed to add
@@ -1788,7 +1798,7 @@ static int zswap_setup(void) pool =__zswap_pool_create_fallback ();if (pool) {pr_info ("loaded using pool %s\n" , pool->tfm_name); -list_add (&pool->list, &zswap_pools); +rcu_assign_pointer (zswap_current_pool, pool); zswap_has_pool = true;static_branch_enable (&zswap_ever_enabled); }else {
Signature · 1 line
2.43.0
Re: [PATCH v6 2/3] mm/zswap: replace the zswap_pools list with an allocating xarray
Quoted 24 lines · Jianyue Wu
Originally zswap kept its pools on an RCU list whose head also served as the current pool. Convert the pool table to an allocating xarray keyed by a small integer id, and track the current pool with a separate RCU-protected pointer.
The xarray gives each pool a stable id for a later
zswap_entryshrink.XA_FLAGS_ALLOC1starts ids at 1, so id 0 remains reserved. The id range is bounded byZSWAP_MAX_POOL_IDbecause the later entry field is a u8.Keep compressor switching close to the previous flow: look up an existing pool with
xa_for_each(), resurrect it if reused, or create a new one.zswap_pool_create()allocates the pool's id and publishes it into the xarray as its final step, so the create call either fully publishes or fully unwinds on failure. Publishing makes the pool live, so a caller that later fails (e.g.param_set_charp()) must still kill the pool to erase it from the xarray.Compressor switches update
zswap_current_poolwithrcu_assign_pointer(), serialized by the module parameter lock, so noxa_lockis needed for that update. The pool walk above is lockless under RCU.xa_lockis taken only to allocate (xa_alloc_bh()) and erase (xa_erase_bh()) xarray entries.
Quoted 4 lines
Suggested-by Nhat Pham <[email protected]>Suggested-by Yosry Ahmed <[email protected]>Suggested-by Johannes Weiner <[email protected]>Signed-off-by Jianyue Wu <[email protected]>
PATCH v6 3/3 mm/zswap: reference the pool by id to shrink struct zswap_entry
In reply to Jianyue Wustruct zswap_entry is one allocation per stored page, so its size is pure overhead. It currently embeds an 8-byte pool pointer, even though the live pools now sit in an allocating xarray keyed by a small integer id that fits in a u8.
Replace the per-entry pool pointer with that u8 id and resolve it through the xarray with xa_load(). xa_load() does its own RCU-protected lookup, so the caller needs no rcu_read_lock() section of its own. The resolved pool stays valid because a live entry pins it via percpu_ref (taken in zswap_store_page()), so its id cannot be reused. A live entry never uses the reserved id 0, so a zeroed id resolves to NULL and trips a WARN rather than aliasing a live pool.
The u8 fits in the padding after the bool referenced field, shrinking the entry from 56 to 48 bytes on 64-bit. This raises objs_per_slab from 73 to 85 and saves about 2MiB of metadata per 1GiB of data held in zswap.
--- mm/zswap.c | 37 ++++++++++++++++++++++++++++++------- 1 file changed, 30 insertions(+), 7 deletions(-)
mm/zswap.c +30 -7
@@ -194,7 +194,7 @@ static struct shrinker *zswap_shrinker;* writeback logic. The entry is only reclaimed by the writeback * logic if referenced is unset. See comments in the shrinker * section for context. -* pool - the zswap_pool the entry's data is in +* pool_idx - id of the zswap_pool that the entry's data is in. * handle - zsmalloc allocation handle that stores the compressed page data * objcg - the obj_cgroup that the compressed memory is charged to * lru - handle to the pool's lru used to evict pages.
@@ -203,12 +203,22 @@ struct zswap_entry {swp_entry_t swpentry;unsigned int length;bool referenced; -struct zswap_pool *pool; +u8 pool_idx;unsigned long handle;struct obj_cgroup *objcg;struct list_head lru; }; +/* +* No RCU section is needed around the returned pointer: a stored entry pins +* its pool via percpu_ref (taken in zswap_store_page()), so the id cannot be +* reused under us. Callers WARN and handle a NULL from a corrupt pool_idx. +*/ +static struct zswap_pool *zswap_entry_pool (struct zswap_entry *entry) +{ +return xa_load (&zswap_pools, entry->pool_idx); +} +static struct xarray *zswap_trees[MAX_SWAPFILES ];static unsigned int nr_zswap_trees[MAX_SWAPFILES ];
@@ -759,9 +769,13 @@ static void zswap_entry_cache_free(struct zswap_entry *entry)*/ static void zswap_entry_free (struct zswap_entry *entry) { +struct zswap_pool *pool =zswap_entry_pool (entry); +zswap_lru_del (entry); -zs_free (entry->pool->zs_pool, entry->handle); -zswap_pool_put (entry->pool); +if (!WARN_ON_ONCE (!pool)) { +zs_free (pool->zs_pool, entry->handle); +zswap_pool_put (pool); + }if (entry->objcg) {obj_cgroup_uncharge_zswap (entry->objcg, entry->length);obj_cgroup_put (entry->objcg);
@@ -918,12 +932,15 @@ static bool zswap_compress(struct page *page, struct zswap_entry *entry,static bool zswap_decompress (struct zswap_entry *entry,struct folio *folio) { -struct zswap_pool *pool = entry->pool; +struct zswap_pool *pool =zswap_entry_pool (entry);struct scatterlist input[2 ];/* zsmalloc returns an SG list 1-2 entries */ struct scatterlist output;struct crypto_acomp_ctx *acomp_ctx;int ret =0 , dlen; +if (WARN_ON_ONCE (!pool)) +return false; + acomp_ctx =raw_cpu_ptr (pool->acomp_ctx);mutex_lock (&acomp_ctx->mutex);zs_obj_read_sg_begin (pool->zs_pool, entry->handle, input, entry->length);
@@ -959,7 +976,7 @@ static bool zswap_decompress(struct zswap_entry *entry, struct folio *folio)pr_alert_ratelimited ("Decompression error from zswap (%d:%lu %s %u->%d)\n" ,swp_type (entry->swpentry),swp_offset (entry->swpentry), - entry->pool->tfm_name, + pool->tfm_name, entry->length, dlen);return false; }
@@ -1417,6 +1434,13 @@ static bool zswap_store_page(struct page *page,if (!zswap_compress (page, entry, pool)) goto compress_failed; +/* +* Set pool_idx before the xa_store() below publishes the entry, or a +* concurrent reader could resolve a stale pool_idx left by slab reuse +* to an unrelated live pool. +*/ + entry->pool_idx = pool->idx; + old =xa_store (swap_zswap_tree (page_swpentry),swp_offset (page_swpentry), entry,GFP_KERNEL );
@@ -1462,7 +1486,6 @@ static bool zswap_store_page(struct page *page,* The publishing order matters to prevent writeback from seeing * an incoherent entry. */ - entry->pool = pool; entry->swpentry = page_swpentry; entry->objcg = objcg; entry->referenced = true;
Signature · 1 line
2.43.0
Re: [PATCH v6 3/3] mm/zswap: reference the pool by id to shrink struct zswap_entry
Quoted 14 lines · Jianyue Wu
struct
zswap_entryis one allocation per stored page, so its size is pure overhead. It currently embeds an 8-byte pool pointer, even though the live pools now sit in an allocating xarray keyed by a small integer id that fits in a u8.Replace the per-entry pool pointer with that u8 id and resolve it through the xarray with
xa_load().xa_load()does its own RCU-protected lookup, so the caller needs norcu_read_lock()section of its own. The resolved pool stays valid because a live entry pins it viapercpu_ref(taken inzswap_store_page()), so its id cannot be reused. A live entry never uses the reserved id 0, so a zeroed id resolves to NULL and trips a WARN rather than aliasing a live pool.
Quoted 7 lines
The u8 fits in the padding after the bool referenced field, shrinking the entry from 56 to 48 bytes on 64-bit. This raises
objs_per_slabfrom 73 to 85 and saves about 2MiB of metadata per 1GiB of data held in zswap.Suggested-by Chris Li <[email protected]>Signed-off-by Jianyue Wu <[email protected]>