Batched zswap_store() framework with compression batching.
Subsystems (4)
v3 cover Batched zswap_store() framework with compression batching.
v3: Batched zswap_store() framework with compression batching
=============================================================
This series is a spin-off of "v14: zswap compression batching with optimized iaa_crypto driver" [1], and focuses on the generic crypto_acomp and zswap compression batching enablers. Resuming work on this series following a gap since v14 (January 2026). Please note that my email address and affiliation have changed from [email protected] to [email protected]. I have retained the original Intel sign-offs on prior patches for historical attribution while signing off v3 of this series with my current address.
The main focus of v3 was to incorporate Sashiko's code review comments and gather results by disabling memory.high overage-driven stalls, based on Yosry's comment in [3]. The rest of the validation methodology remains the same:
2 usemem processes, PMD THP enabled in an experiment with 56% over-commit.
To summarize the main contribution of this patch series: Batching is introduced in zswap_store() of large folios: we carve out a large folio into batches of ZSWAP_MAX_BATCH_SIZE (4 pages) for software compressors, and store each batch. Moving the fundamental operations that need to be happen to store a page to now work at the granularity of a batch offers opportunities to optimize them from these perspectives:
1) Utilize existing kernel API that minimize obtaining locks for bulk operations, e.g. kmem_cache_alloc_bulk()/kmem_cache_free_bulk().
2) We keep the working set for each of the core store functionalities (alloc/initialize zswap entries, compress pages, store pages in zsmalloc, pool/objcg reference increments, objcg charging) to be focused on a batch, thereby minimizing cache evictions and benefiting from cache locality.
3) Along the lines of (1), the main zswap_compress() batching routine obtains the comp_ctx mutex lock once per batch, and releases it after it has processed all pages in that batch.
4) Any zswap store functionality that works on a batch can potentially be optimized via bulk-processing/parallelized as a next step, to further improve performance.
Performance data: usemem with 56% over-commit:
==============================================
Validation was done with mm-unstable as of 10-25-2026 (Baseline) and with this patch series. To minimize variation in the data and to help drive useful conclusions, I made these changes to my performance measurement methodology:
1) Reduced the number of usemem processes to 2. Each process tries to allocate 2400M, in a 3G cgroup memory.high budget, effectively accomplishing a 56.25% over-commit.
2) Modified memcontrol.c/calculate_high_delay() to always return 0 to mirror the "hack" noted by Yosry in [3] to minimize variation due to process throttling when memory.high is breached.
3) Enabled PMD THP, which forces zswap_store()/zswap_compress() latency to be in the critical path, and also generates sufficient samples for reliable bpftrace measurements.
4) v3 does not report usemem throughput metrics, instead focuses solely on bpftrace p50/p95/p99 latencies of zswap_store() and zswap_compress() to validate the benefit of store batching and compress batching respectively.
5) I used lhist() instead of hist() in bpftrace to gather latencies at a finer level of granularity and accuracy.
Testing was done on a Chromebook with a QEMU "vng" VM running in Crostini.
Chromebook Diagnostics reports:
Intel(R) Core(TM) Ultra 5 115U (10 CPUs, 4.200GHz)
RAM: 16GB
vng setup:
Memory: 7G
Swap: 3G
Pinned CPUs: 2 (E-Cores: 2,3)zswap compressor:
ZSTD
mTHP enabled:
2048kB (PMD THP)
I used the help of an LLM to write the bpftrace data post-processing scripts that generate latency percentiles.
vm-scalability/usemem setup:
cgroup memory.high: 3G
2 usemem processes, each allocates 2400M anonymous memory, reads the Silesia dataset [2] from a pre-fork-allocated buffer into the 2400M, writes to each unsigned long in the 2400M, sleeps for 10 sec, then frees the memory and exits.
usemem -w --read-silesia -O -b 1 -s 10 -n 2 2400M
usemem metrics are reported from a single run after starting the QEMU VM. Further, I have tabulated the bpftrace p50/p95/p99 latencies with these labels:
"zswap_store_higher_order_folio":
zswap_store() of PMD folios.
"zswap_store_order0_folio":
zswap_store() of order-0 folios.
"zswap_batch_compress_per_page":
zswap_compress() per-page latency of store batches generated from PMD folios.
"zswap_single_compress_per_page":
zswap_compress() per-page latency of order-0 folios.
=============================================================================
usemem PMD THP BPFTRACE
=============================================================================
Baseline zswap zswap Change Change
mm-unstable store compress with with
10-25-2026 batching batching compress compress
(patch 6-7) (patch 8) batching batching
(ns) %
=============================================================================
zswap_store_order0_folio (ns):
=============================================================================
p50 39,631 37,538 37,765 -1,866 -4.71% p95 53,088 49,690 49,349 -3,739 -7.04% p99 65,065 63,529 57,404 -7,661 -11.77% ============================================================================= zswap_store_higher_order_folio (ns): ============================================================================= p50 21,460,366 21,201,411 20,383,233 -1,077,133 -5.02% p95 25,975,341 27,558,253 25,045,350 -929,991 -3.58% p99 29,538,493 61,926,648 27,184,835 -2,353,658 -7.97% ============================================================================= zswap_single_compress_per_page (ns): ============================================================================= p50 37,545 39,320 37,218 -327 -0.87% p95 51,549 66,503 51,481 -68 -0.13% p99 89,866 134,854 87,091 -2,775 -3.09% ============================================================================= zswap_batch_compress_per_page (ns): ============================================================================= p50 40,493 2,948 7.85% p95 56,944 5,395 10.47% p99 76,242 -13,624 -15.16%
=============================================================================
==========================================================================
vmstat summary:
==========================================================================
usemem Baseline zswap zswap
PMD THP mm-unstable store compress
10-25-2026 batching batching
(patch 6-7) (patch 8)
==========================================================================
zswpin 989,367 1,050,081 949,046
zswpout 1,567,607 1,634,772 1,533,088
2048kB-alloc 2,500 2,504 2,500 2048kB-zswpout 1,557 1,552 1,539 2048kB-swpout-fallback 793 834 743 2048kB-swpout 0 0 0 pswpin 0 0 0 pswpout 0 0 0 zswpwb 0 0 0
==========================================================================
==========================================================================
zswap stats after workload finishes:
==========================================================================
usemem Baseline zswap zswap
PMD THP mm-unstable store compress
10-25-2026 batching batching
(patch 6-7) (patch 8)
==========================================================================
pool_limit_hit 0 0 0 written_back_pages 0 0 0 reject_alloc_fail 0 0 0 reject_kmemcache_fail 0 0 0 reject_reclaim_fail 0 0 0 reject_compress_fail 0 0 0 reject_compress_poor 0 0 0 decompress_fail 0 0 0 stored_incompressible_pages 0 0 0 stored_pages 0 0 0 pool_total_size 0 0 0
==========================================================================
Conclusions based on the data:
==============================
The batching of zswap_store() phases, and compress batching optimizations benefit zswap_store() and zswap_compress_per_page() median and tail latencies for PMD folios and base 4K folios. The benefits are attributable to the store batching framework added in patches 6-7, that is fully utilized to improve zswap_compress() latency by giving it a batch of pages to be compressed. As the bpftrace data shows, patches 6-7 alone are not sufficient for the gains. Batch compression added in patch 8 benefits latency significantly, for these reasons:
1) Per-CPU sg_table with ZSWAP_MAX_BATCH_SIZE scatterlists that are statically mapped to compression dst buffers for that CPU, thereby minimizing local SG lists initialization, and using compact new crypto_acomp API introduced in patches 1-4 that set up an acomp_req's src/dst to be SG lists.
2) Patch 6 allocates ZSWAP_MAX_BATCH_SIZE destination buffers in the per-CPU acomp_ctx, for an added memory overhead of ~12 KB per-CPU. The trade-off pays off in significant latency improvements which can potentially be used to drive higher memory over-commit.
3) A one-shot zswap_update_entries() does all the zsmalloc handle allocation and copying for the batch. The key concept driving efficiency is that the working set is kept focused on batch-centric data pertaining to a handful of operations on the batch, to minimize cache evictions and benefit from cache locality.
4) By making zswap_compress() more efficient, we are able to lock the comp mutex per batch of ZSWAP_MAX_BATCH_SIZE (4) pages instead of per-page.
The exceptions are p50/p95 batch compress latencies (normalized per-page), even though the p99 batch compress latency improves significantly, by 15% over mm-unstable. Batch compress latency can be optimized further in subsequent patches. The batching framework established in this series allows us to make locking and bulk-processing improvements in each stage of zswap_store(). One of the ideas that shown very promising results is to replace zsmalloc with SLUB compressed caches and will be submitted separately. This is expected to resolve the p50 batch compress latency issue since it will be directly optimizing zswap_update_entries(), where compressed buffers can be binned based on size and bulk allocated from SLUB.
The speedup with zswap store batching of large folios has a propelling effect on other parts of mm, especially, by enabling large folios to be swapped out without splitting, as the data shows. This enables other mm components that are optimized for large folios, such as munmap(), folio refcounts/mapcounts updates, PTE invalidation upon freeing memory, zswap_invalidate(); which benefit from xarray offsets/range continuity. As a result, subsequent large folio swapouts are more likely to find contiguous swap slots, there are fewer large folio splits during reclaim and the cycle continues. As the data demonstrates, this directly contributes to more memory over-commit opportunity.
[1]: https://patchwork.kernel.org/project/linux-mm/cover/[email protected]/
[2]: https://wanos.co/assets/silesia.tar
[3]: https://lore.kernel.org/all/[email protected]/
Changes since v2:
=================
Resolved v2 code review issues highlighted by Sashiko. Baseline mm-unstable as of 10-25-2026, commit a2abad0d7f45.
1) Patch 3:
- Re-wrote the fast path conditional to be !acomp_is_async(tfm) instead of
!acomp_req_on_stack(req).
- Moved the state-flag changes from acomp_save_req() back to
acomp_virt_to_sg().2) Patch 4:
- Replaced the ambiguous use of "cra_flags & CRYPTO_ALG_TYPE_ACOMPRESS" with
directly checkinf for "cra_type == &crypto_acomp_type" to detect an
acomp_alg.3) Patch 5:
- Fixed the off-by-one bitfield width by modifying ZSWAP_MAX_COMPR_LENGTH_SHIFT to 19U and the build time assertion check for BUILD_BUG_ON(ZSWAP_MAX_COMPR_LENGTH_SHIFT <= PAGE_SHIFT)).
4) Patch 6:
- Moved the "u8 compr_batch_size" to struct zswap_comp_ctx, which is per-CPU to address the "unsynchronized write to pool->compr_batch_size on CPU hotplug" issue Sashiko highlighted.
5) Patch 7:
- The special casing for comp_ret=0 with dlen == PAGE_SIZE (among other end-cases) when writeback is disabled, is replicated in the batched version of zswap_compress() as well.
6) Patch 8:
- Added obj_cgroup_charge_zswap_many() to correctly account for MEMCG_ZSWAP* counters with bulk-charging total compressed bytes to zswap.
- Resolved zsmalloc() handles leak issue in case of compress batch failures in zswap_update_entries().
- Deleted zswap_pool_get().
Changes since v1:
=================
1) Rebased to mm-unstable as of 10-25-2026, commit a2abad0d7f45.
2) Addressed Herbert's comments on checking for the req->unit_size being equal to the req->slen to identify singleton acomp_reqs and to quickly short-circuit with setting the dst SG list length instead of going through the request chaining flow. Further, the most common code path is the very first check, with the short-circuit, in crypto_acomp_compress().
3) Addressed Yosry's comments wrt:
- struct zswap_entry "length" field build-time assertion to have sufficient bit-fields for the architecture's PAGE_SIZE.
- Use kmem_cache_alloc() instead of kmem_cache_alloc_node() in zswap_entry_cache_alloc().
4) Established a reliable and accurate performance validation methodology, based on Yosry's comments.
5) Refactored the sub-components of zswap_store_pages() where each routine processes a batch. The new zswap_account_stores() gets refs, updates statistics and charges cgroup for the batch, rather than for a page at a time. zswap_store_pages() is more readable and maintainable. This batching framework can be immediately useful for implementing more ideas such as SLUB bulk allocation of compressed data, reclaiming a batch of folios from shrink_folio_list(), etc.
Changes since [1]:
==================
1) Better cacheline placement of struct zswap_pool and struct zswap_entry members for temporal memory access efficiency in zswap store/load hot paths, while minimizing holes.
2) Refactored the core compression batching functionality to a separate __zswap_compress() function, which is called by zswap_compress(), as per Nhat's and Yosry's comments. Hopefully this helps explain the outer "store batching" zswap_compress() vis-a-vis the inner "compress batching" __zswap_compress() distinction.
3) The acomp_ctx is locked once and is passed to the routine that compresses a batch, per the algorithm's batch-size. Similarly, the folio's node id and writeback_enabled status are computed once and passed to the routines that implement the different store functionalities, to save latency by avoid localized recomputes of information that doesn't change.
4) Moved the folio node id being stored in zswap_entry to a separate patch, per Yosry's comment.
5) Addressed the kunmap_local() issue flagged by Jixin He in the IS_ERR_VALUE(handle) error path
(https://patchwork.kernel.org/comment/27095925) - thanks!
6) Moved comments inline, per Yosry's suggestions.
Thanks,
Kanchana
Kanchana P. Sridhar (8):
crypto: acomp - Define a unit_size in struct acomp_req to enable
batching. mm: zswap: Set the unit size for zswap to PAGE_SIZE.
crypto: acomp - Add bit to indicate segmentation support.
crypto: acomp - Add trivial segmentation wrapper.
crypto: acomp - Add API to get a compression algorithm's batch-size.
mm: zswap: Store the folio's node id in the zswap_entry.
mm: zswap: Store large folios in batches.
mm: zswap: Refactor and optimize zswap_store_pages() batch operations.
mm: zswap: Batched zswap_compress() for storing large folios in
batches.
crypto/acompress.c | 48 +- include/crypto/acompress.h | 66 ++- include/crypto/algapi.h | 5 + include/crypto/internal/acompress.h | 10 +- include/linux/crypto.h | 5 +- include/linux/memcontrol.h | 7 + mm/memcontrol.c | 36 ++ mm/zswap.c | 671 +++++++++++++++++++++------- 8 files changed, 673 insertions(+), 175 deletions(-)
Signature · 1 line
2.39.5
PATCH v3 1/8 crypto: acomp - Define a unit_size in struct acomp_req to enable batching. mm: zswap: Set the unit size for zswap to PAGE_SIZE.
We add a new @unit_size data member to struct acomp_req along with a helper function acomp_request_set_unit_size() for kernel modules to set the unit size to use while breaking down the request's src/dst scatterlists.
An acomp_alg can implement batching by using the @req->unit_size to break down the SG lists passed in via @req->dst and/or @req->src, to submit individual @req->slen/@req->unit_size compress jobs or @req->dlen/@req->unit_size decompress jobs, for batch compression and batch decompression respectively.
In case of batch compression, the folio's pages for the batch can be retrieved from the @req->src scatterlist by using a struct sg_page_iter after determining the number of pages as @req->slen/@req->unit_size.
Please note that acomp_request_set_callback() sets the @req->unit_size to 0.
In zswap_acomp_req_init(), after the call to zswap_set_acomp_req_callback(), we set the unit size for zswap to PAGE_SIZE.
acomp_request_set_unit_size(areq->req, PAGE_SIZE);
--- include/crypto/acompress.h | 49 ++++++++++++++++++++++++++++++++++++++ mm/zswap.c | 6 +++++ 2 files changed, 55 insertions(+)
include/crypto/acompress.h +49 -0
@@ -79,6 +79,7 @@ struct acomp_req_chain {* @dvirt: Destination virtual address * @slen: Size of the input buffer * @dlen: Size of the output buffer and number of bytes produced +* @unit_size: Unit size for the request for use in batching * @chain: Private API code data, do not use * @__ctx: Start of private context data */
@@ -94,6 +95,7 @@ struct acomp_req { };unsigned int slen;unsigned int dlen; +unsigned int unit_size;struct acomp_req_chain chain;
@@ -329,9 +331,56 @@ static inline void acomp_request_set_callback(struct acomp_req *req, { flgs &= ~CRYPTO_ACOMP_REQ_PRIVATE ; flgs |= req->base.flags &CRYPTO_ACOMP_REQ_PRIVATE ; + req->unit_size =0 ;crypto_request_set_callback (&req->base, flgs, cmpl, data); } +/** +* acomp_request_set_unit_size() -- Sets the unit size for the request. +* +* This is a helper function that enables batching for zswap, IPComp, etc. +* It allows multiple independent compression (or decompression) operations to +* be submitted in a single request's SG lists, where each SG list ("segment") +* is processed independently. The unit size helps derive segments from a +* single request. crypto_acomp does not expect the segments to be related in +* any way. +* +* Example usage model: +* +* A module such as zswap that's configured to use a batching compressor, can +* accomplish batch compression of "nr_pages" with crypto_acomp by creating an +* output SG table for the batch, initialized to contain "nr_pages" SG +* lists. Each scatterlist is mapped to the nth destination buffer for the +* batch. Depending on whether the @req is used for batch compress/decompress, +* zswap must set the @req's source/destination length to be +* "nr_pages * @req->unit_size" respectively. +* +* An acomp_alg can implement batch compression by using the @req->unit_size +* to break down the SG lists passed in via @req->dst to submit individual +* "@req->slen/@req->unit_size" compress jobs to be processed as a batch. +* +* Similarly, zswap can implement batch decompression by passing an +* SG table with "nr_pages" SG lists via @req->src to process +* "@req->dlen/@req->unit_size" decompress jobs as a batch. +* +* This API must be called after acomp_request_set_callback(), +* which sets @req->unit_size to 0. This makes it easy for users of +* crypto_acomp to rely on a default of not opting in to batching. +* Users such as zswap opt in to batching by defining @req->unit_size +* to a non-zero value for use by acomp_algs supporting batching. +* +* @du would be PAGE_SIZE for zswap, it could be the MTU for IPsec. +* +* @req: asynchronous compress/decompress request +* @du: data unit size of the input/output buffer scatterlist. +*/ +static inline void acomp_request_set_unit_size (struct acomp_req *req, +unsigned int du) +{ + req->unit_size = du; +} + +/** * acomp_request_set_params() -- Sets request parameters *
mm/zswap.c +6 -0
@@ -826,6 +826,12 @@ static int zswap_acomp_req_init(struct zswap_acomp_req *areq,crypto_init_wait (&areq->wait);zswap_set_acomp_req_callback (areq->req, &areq->wait); +/* +* This should be called after acomp_request_set_callback(), +* which initializes req->unit_size to 0. +*/ +acomp_request_set_unit_size (areq->req,PAGE_SIZE ); +mutex_init (&areq->mutex);return 0 ; }
Signature · 1 line
2.39.5
PATCH v3 2/8 crypto: acomp - Add bit to indicate segmentation support.
In reply to Kanchana P. SridharThis patch adds segmentation support for compression.
Add a bit to the crypto_alg flags to indicate support for segmentation. Also add a helper for acomp to test whether a given tfm supports segmentation.
--- include/crypto/algapi.h | 5 +++++ include/crypto/internal/acompress.h | 5 +++++ include/linux/crypto.h | 3 +++ 3 files changed, 13 insertions(+)
include/crypto/algapi.h +5 -0
@@ -280,6 +280,11 @@ static inline bool crypto_tfm_req_virt(struct crypto_tfm *tfm)return tfm->__crt_alg->cra_flags &CRYPTO_ALG_REQ_VIRT ; } +static inline bool crypto_tfm_req_seg (struct crypto_tfm *tfm) +{ +return tfm->__crt_alg->cra_flags &CRYPTO_ALG_REQ_SEG ; +} +static inline u32 crypto_request_flags (struct crypto_async_request *req) {return req->flags & ~CRYPTO_TFM_REQ_ON_STACK ;
include/crypto/internal/acompress.h +5 -0
@@ -189,6 +189,11 @@ static inline bool crypto_acomp_req_virt(struct crypto_acomp *tfm)return crypto_tfm_req_virt (&tfm->base); } +static inline bool crypto_acomp_req_seg (struct crypto_acomp *tfm) +{ +return crypto_tfm_req_seg (&tfm->base); +} +void crypto_acomp_free_streams (struct crypto_acomp_streams *s);int crypto_acomp_alloc_streams (struct crypto_acomp_streams *s);
include/linux/crypto.h +3 -0
@@ -139,6 +139,9 @@/* Set if the algorithm cannot have a fallback (e.g., phmac). */ #define CRYPTO_ALG_NO_FALLBACK 0x00080000 +/* Set if the algorithm supports segmentation. */ +#define CRYPTO_ALG_REQ_SEG 0x00100000 +/* The high bits 0xff000000 are reserved for type-specific flags. */ /*
Signature · 1 line
2.39.5
PATCH v3 3/8 crypto: acomp - Add trivial segmentation wrapper.
In reply to Kanchana P. SridharThis patch provides a wrapper for existing algorithms so that they can accept a single segment while returning the compressed length or error through the dst SG list length.
This trivial segmentation wrapper only supports compression with a segment count of exactly one.
The reason is that the first user zswap will only allocate the extra memory if the underlying algorithm supports segmentation, and otherwise only one segment will be given at a time.
Having this wrapper means that the same calling convention can be used for all algorithms, regardless of segmentation support.
To gain latency in the zswap_store() hot path impacted by this patch:
1) The most common code path is invoked first in
crypto_acomp_compress().
2) The helper functions used in (1) are made __always_inline.
--- crypto/acompress.c | 34 ++++++++++++++++++++++++++--- include/crypto/acompress.h | 5 +++-- include/crypto/internal/acompress.h | 4 ++-- include/linux/crypto.h | 2 +- 4 files changed, 37 insertions(+), 8 deletions(-)
crypto/acompress.c +31 -3
@@ -166,14 +166,18 @@ static void acomp_save_req(struct acomp_req *req, crypto_completion_t cplt) state->compl = req->base.complete; state->data = req->base.data; + state->unit_size = req->unit_size; + req->base.complete = cplt; req->base.data = req; + req->unit_size =0 ; }static void acomp_restore_req (struct acomp_req *req) { req->base.complete = req->chain.compl; req->base.data = req->chain.data; + req->unit_size = req->chain.unit_size; }static void acomp_reqchain_virt (struct acomp_req *req)
@@ -242,6 +246,10 @@ static int acomp_reqchain_finish(struct acomp_req *req, int err) {acomp_reqchain_virt (req);acomp_restore_req (req); + +if (req->unit_size) + req->dst->length =unlikely (err) ? err : req->dlen; +return err; }
@@ -266,10 +274,13 @@ static int acomp_do_req_chain(struct acomp_req *req, bool comp) {int err; +if (unlikely (req->unit_size && req->slen > req->unit_size)) +return -ENOSYS ; +acomp_save_req (req, acomp_reqchain_done); err =acomp_do_one_req (req, comp); -if (err == -EBUSY || err == -EINPROGRESS ) +if (unlikely (err == -EBUSY || err == -EINPROGRESS ))return err;return acomp_reqchain_finish (req, err);
@@ -279,10 +290,27 @@ int crypto_acomp_compress(struct acomp_req *req) {struct crypto_acomp *tfm =crypto_acomp_reqtfm (req); +if (likely ((req->unit_size == req->slen) &&acomp_request_issg (req) && + !crypto_acomp_req_seg (tfm) && !acomp_is_async (tfm))) { +int err = tfm->compress (req); + + req->dst->length =unlikely (err) ? err : req->dlen; +return err; + } +if (acomp_req_on_stack (req) &&acomp_is_async (tfm))return -EAGAIN ; + +if (req->unit_size &&acomp_request_issg (req)) { +if (!crypto_acomp_req_seg (tfm)) +return acomp_do_req_chain (req, true); + +return tfm->compress (req); + } +if (crypto_acomp_req_virt (tfm) ||acomp_request_issg (req)) -return crypto_acomp_reqtfm (req)->compress (req); +return tfm->compress (req); +return acomp_do_req_chain (req, true); }EXPORT_SYMBOL_GPL (crypto_acomp_compress);
@@ -294,7 +322,7 @@ int crypto_acomp_decompress(struct acomp_req *req)if (acomp_req_on_stack (req) &&acomp_is_async (tfm))return -EAGAIN ;if (crypto_acomp_req_virt (tfm) ||acomp_request_issg (req)) -return crypto_acomp_reqtfm (req)->decompress (req); +return tfm->decompress (req);return acomp_do_req_chain (req, false); }EXPORT_SYMBOL_GPL (crypto_acomp_decompress);
include/crypto/acompress.h +3 -2
@@ -67,6 +67,7 @@ struct acomp_req_chain {struct folio *dfolio; };u32 flags; +u32 unit_size; };/**
@@ -199,7 +200,7 @@ static inline void acomp_request_set_tfm(struct acomp_req *req,crypto_request_set_tfm (&req->base,crypto_acomp_tfm (tfm)); } -static inline bool acomp_is_async (struct crypto_acomp *tfm) +static __always_inline bool acomp_is_async(struct crypto_acomp *tfm) {return crypto_comp_alg_common (tfm)->base.cra_flags &CRYPTO_ALG_ASYNC ;
@@ -294,7 +295,7 @@ static inline void *acomp_request_extra(struct acomp_req *req)return (void *)((char *)req + len); } -static inline bool acomp_req_on_stack (struct acomp_req *req) +static __always_inline bool acomp_req_on_stack(struct acomp_req *req) {return crypto_req_on_stack (&req->base); }
include/crypto/internal/acompress.h +2 -2
@@ -146,7 +146,7 @@ void crypto_unregister_acomp(struct acomp_alg *alg);int crypto_register_acomps (struct acomp_alg *algs,int count);void crypto_unregister_acomps (struct acomp_alg *algs,int count); -static inline bool acomp_request_issg (struct acomp_req *req) +static __always_inline bool acomp_request_issg(struct acomp_req *req) {return !(req->base.flags & (CRYPTO_ACOMP_REQ_SRC_VIRT |CRYPTO_ACOMP_REQ_DST_VIRT ));
@@ -189,7 +189,7 @@ static inline bool crypto_acomp_req_virt(struct crypto_acomp *tfm)return crypto_tfm_req_virt (&tfm->base); } -static inline bool crypto_acomp_req_seg (struct crypto_acomp *tfm) +static __always_inline bool crypto_acomp_req_seg(struct crypto_acomp *tfm) {return crypto_tfm_req_seg (&tfm->base); }
include/linux/crypto.h +1 -1
@@ -492,7 +492,7 @@ static inline bool crypto_tfm_is_async(struct crypto_tfm *tfm)return tfm->__crt_alg->cra_flags &CRYPTO_ALG_ASYNC ; } -static inline bool crypto_req_on_stack (struct crypto_async_request *req) +static __always_inline bool crypto_req_on_stack(struct crypto_async_request *req) {return req->flags &CRYPTO_TFM_REQ_ON_STACK ; }
Signature · 1 line
2.39.5
PATCH v3 4/8 crypto: acomp - Add API to get a compression algorithm's batch-size.
In reply to Kanchana P. SridharThis commit adds a @batch_size data member to struct acomp_alg.
An acomp_alg compression algorithm that supports batching of compressions and decompressions must provide a @batch_size greater than 1, representing the maximum batch-size that the compressor supports, so that kernel users of crypto_acomp, such as zswap, can allocate resources for submitting multiple compress/decompress jobs that can be batched, and invoke batching of [de]compressions.
The new crypto_acomp_batch_size() API queries the crypto_acomp's acomp_alg for the batch-size. If the acomp_alg has registered a @batch_size greater than 1, this is returned. If not, a default of "1" is returned.
zswap can invoke crypto_acomp_batch_size() to query the maximum number of requests that can be batch [de]compressed. Based on this, zswap can use the minimum of any zswap-specific upper limits for batch-size and the compressor's max @batch_size, to allocate batching resources.
This enables zswap to compress/decompress multiple pages as a batch to improve swapout/swapin performance and memory savings.
--- crypto/acompress.c | 14 ++++++++++++++ include/crypto/acompress.h | 12 ++++++++++++ include/crypto/internal/acompress.h | 3 +++ 3 files changed, 29 insertions(+)
crypto/acompress.c +14 -0
@@ -327,6 +327,20 @@ int crypto_acomp_decompress(struct acomp_req *req) }EXPORT_SYMBOL_GPL (crypto_acomp_decompress); +unsigned int crypto_acomp_batch_size (struct crypto_acomp *tfm) +{ +if (acomp_is_async (tfm) && + (crypto_comp_alg_common (tfm)->base.cra_type == &crypto_acomp_type)) { +struct acomp_alg *alg =crypto_acomp_alg (tfm); + +if (alg && alg->batch_size >1 ) +return alg->batch_size; + } + +return 1 ; +} +EXPORT_SYMBOL_GPL (crypto_acomp_batch_size); +void comp_prepare_alg (struct comp_alg_common *alg) {struct crypto_alg *base = &alg->base;
include/crypto/acompress.h +12 -0
@@ -593,6 +593,18 @@ int crypto_acomp_compress(struct acomp_req *req);*/ int crypto_acomp_decompress (struct acomp_req *req); +/** +* crypto_acomp_batch_size() -- Get the algorithm's batch size +* +* Function returns the algorithm's batch size for batching operations +* +* @tfm: ACOMPRESS tfm handle allocated with crypto_alloc_acomp() +* +* Return: @tfm's acomp_alg's @batch_size, if it has defined a +* @batch_size greater than 1; else return 1. +*/ +unsigned int crypto_acomp_batch_size (struct crypto_acomp *tfm); +static inline struct acomp_req *acomp_request_on_stack_init (char *buf,struct crypto_acomp *tfm) {
include/crypto/internal/acompress.h +3 -0
@@ -28,6 +28,8 @@* * @compress: Function performs a compress operation * @decompress: Function performs a de-compress operation +* @batch_size: Maximum batch-size for batching compress/decompress +* operations. * @init: Initialize the cryptographic transformation object. * This function is used to initialize the cryptographic * transformation object. This function is called only once at
@@ -47,6 +49,7 @@struct acomp_alg {int (*compress)(struct acomp_req *req);int (*decompress)(struct acomp_req *req); +unsigned int batch_size;int (*init)(struct crypto_acomp *tfm);void (*exit)(struct crypto_acomp *tfm);
Signature · 1 line
2.39.5
PATCH v3 5/8 mm: zswap: Store the folio's node id in the zswap_entry.
In reply to Kanchana P. SridharWhen storing a folio in zswap, save the folio's node id in the zswap entry instead of relying on the zswap_entry being allocated on the same node. This enables optimizations such as batch allocation of zswap entries for large folios using kmem_cache_alloc_bulk() that does not support specifying a node id.
Accordingly, replace the SLUB allocation in zswap_entry_cache_alloc() to be node-agnostic.
The size of the zswap_entry remains unchanged as 'referenced' is lumped in with the 'length' (as it doesn't need a full unsigned int anyway). A build time assertion checks for the zswap_entry's 'length' bitfield being large enough to represent PAGE_SIZE, which can be 256KB on some architectures.
--- mm/zswap.c | 33 ++++++++++++++++++--------------- 1 file changed, 18 insertions(+), 15 deletions(-)
mm/zswap.c +18 -15
@@ -180,6 +180,7 @@ struct zswap_pool {*/ #define ZSWAP_FIRST_POOL_ID 1#define ZSWAP_MAX_POOL_ID U8_MAX +#define ZSWAP_MAX_COMPR_LENGTH_SHIFT 19U static DEFINE_XARRAY_FLAGS(zswap_pools, XA_FLAGS_ALLOC1 | XA_FLAGS_LOCK_BH);static struct zswap_pool __rcu *zswap_current_pool;
@@ -206,15 +207,19 @@ static struct shrinker *zswap_shrinker;* logic if referenced is unset. See comments in the shrinker * section for context. * pool_idx - id of the zswap_pool that the entry's data is in. +* nid - NUMA node id of the page for which this is the zswap entry. * handle - zsmalloc allocation handle that stores the compressed page data * objcg - the obj_cgroup that the compressed memory is charged to * lru - handle to the pool's lru used to evict pages. */ struct zswap_entry {swp_entry_t swpentry; -unsigned int length; -bool referenced; -u8 pool_idx; +struct { +unsigned int length:ZSWAP_MAX_COMPR_LENGTH_SHIFT ; +bool referenced:1U ; +u8 pool_idx; + }; +int nid;unsigned long handle;struct obj_cgroup *objcg;struct list_head lru;
@@ -679,14 +684,8 @@ static int zswap_enabled_param_set(const char *val,* lru functions **********************************/ -static inline int entry_to_nid (struct zswap_entry *entry) -{ -return page_to_nid (virt_to_page (entry)); -} -static void zswap_lru_add (struct zswap_entry *entry) { -int nid =entry_to_nid (entry);struct mem_cgroup *memcg;/*
@@ -703,19 +702,18 @@ static void zswap_lru_add(struct zswap_entry *entry)rcu_read_lock (); memcg =obj_cgroup_memcg (entry->objcg);/* will always succeed */ -list_lru_add (&zswap_list_lru, &entry->lru, nid, memcg); +list_lru_add (&zswap_list_lru, &entry->lru, entry->nid, memcg);rcu_read_unlock (); }static void zswap_lru_del (struct zswap_entry *entry) { -int nid =entry_to_nid (entry);struct mem_cgroup *memcg;rcu_read_lock (); memcg =obj_cgroup_memcg (entry->objcg);/* will always succeed */ -list_lru_del (&zswap_list_lru, &entry->lru, nid, memcg); +list_lru_del (&zswap_list_lru, &entry->lru, entry->nid, memcg);rcu_read_unlock (); }
@@ -763,10 +761,11 @@ void zswap_memcg_offline_cleanup(struct mem_cgroup *memcg)**********************************/ static struct kmem_cache *zswap_entry_cache; -static struct zswap_entry *zswap_entry_cache_alloc (gfp_t gfp,int nid) +static struct zswap_entry *zswap_entry_cache_alloc (gfp_t gfp) {struct zswap_entry *entry; - entry =kmem_cache_alloc_node (zswap_entry_cache, gfp, nid); + + entry =kmem_cache_alloc (zswap_entry_cache, gfp);if (!entry)return NULL ;return entry;
@@ -1507,9 +1506,10 @@ static bool zswap_store_page(struct folio *folio, long index, {swp_entry_t page_swpentry =folio_swap_entry (folio, index);struct zswap_entry *entry, *old; +int nid =folio_nid (folio);/* allocate entry */ - entry =zswap_entry_cache_alloc (GFP_KERNEL ,folio_nid (folio)); + entry =zswap_entry_cache_alloc (GFP_KERNEL );if (!entry) { zswap_reject_kmemcache_fail++;return false;
@@ -1524,6 +1524,7 @@ static bool zswap_store_page(struct folio *folio, long index,* to an unrelated live pool. */ entry->pool_idx = pool->idx; + entry->nid = nid; old = xa_store(swap_zswap_tree(page_swpentry), swp_offset(page_swpentry),
@@ -1897,6 +1898,8 @@ static int zswap_setup(void)struct zswap_pool *pool;int ret; +BUILD_BUG_ON (ZSWAP_MAX_COMPR_LENGTH_SHIFT <=PAGE_SHIFT ); + zswap_entry_cache =KMEM_CACHE (zswap_entry,0 );if (!zswap_entry_cache) {pr_err ("entry cache creation failed\n" );
Signature · 1 line
2.39.5
PATCH v3 6/8 mm: zswap: Store large folios in batches.
In reply to Kanchana P. SridharUse batching when storing large folios in zswap. A new constant, ZSWAP_MAX_BATCH_SIZE is defined as 4U to support processing batches of up to 4 pages at a time while storing large folios. This commit modifies the "store single page" approach to "store a batch of pages".
As part of CPU onlining per-CPU acomp_ctx setup, ZSWAP_MAX_BATCH_SIZE compression buffers are allocated to eventually be able to compress and store ZSWAP_MAX_BATCH_SIZE pages. This happens in subsequent commits.
For existing software compressors, the extra overhead is the the extra 'buffers' pointer, 3 additional elements in 'buffers' and 3 additional 4K space per 'buffers[i]': in all ~12 KB per-CPU on x86_64. Only the first buffer is currently used.
zswap_store_page() is replaced with zswap_store_pages(), which processes a batch of pages and allows for batching optimizations. For now, only optimize allocating entries by using batch allocations from the slab cache.
Avoid repeatedly calling mem_cgroup_zswap_writeback_enabled() for every page and only call it once for the folio, since the entire folio is charged to a single memcg. Similarly, obtain the nid once to use during each batch store of a folio. To facilitate this, a "struct zswap_store_context" is introduced to keep function parameter lists concise.
--- mm/zswap.c | 307 +++++++++++++++++++++++++++++++++++++---------------- 1 file changed, 216 insertions(+), 91 deletions(-)
mm/zswap.c +216 -91
@@ -84,6 +84,9 @@ static bool zswap_pool_reached_full;#define ZSWAP_PARAM_UNSET "" +/* Limit the batch size to limit per-CPU memory usage for dst buffers. */ +#define ZSWAP_MAX_BATCH_SIZE 4U +static int zswap_setup (void );/* Enable/disable zswap */
@@ -143,10 +146,12 @@ struct zswap_acomp_req {struct mutex mutex; }; -/* The compression mutex also protects the output buffer. */ +/* +* The compression mutex also protects the output buffer(s). +*/ struct zswap_comp_ctx {struct zswap_acomp_req areq; -u8 *buffer; +u8 **buffers; };/* Separate requests, so that decompression does not wait for compression. */
@@ -161,6 +166,7 @@ struct crypto_acomp_ctx {* The only case where lru_lock is not acquired while holding tree.lock is * when a zswap_entry is taken off the lru for writeback, in that case it * needs to be verified that it's still valid in the tree. +* */ struct zswap_pool {struct zs_pool *zs_pool;
@@ -225,6 +231,15 @@ struct zswap_entry {struct list_head lru; }; +struct zswap_store_context { +struct folio *folio; +long folio_start; +bool wb_enabled; +int nid; +u8 nr_pages; +struct zswap_entry *entries[ZSWAP_MAX_BATCH_SIZE ]; +}; +/* * No RCU section is needed around the returned pointer: a stored entry pins * its pool via percpu_ref (taken in zswap_store_page()), so the id cannot be
@@ -283,6 +298,8 @@ static void __zswap_pool_empty(struct percpu_ref *ref);static void acomp_ctx_free (struct crypto_acomp_ctx *acomp_ctx) { +u8 i; +if (!acomp_ctx)return ;
@@ -301,8 +318,12 @@ static void acomp_ctx_free(struct crypto_acomp_ctx *acomp_ctx) acomp_ctx->acomp =NULL ; -kfree (acomp_ctx->comp.buffer); - acomp_ctx->comp.buffer =NULL ; +if (acomp_ctx->comp.buffers) { +for (i =0 ; i <ZSWAP_MAX_BATCH_SIZE ; ++i) +kfree (acomp_ctx->comp.buffers[i]); +kfree (acomp_ctx->comp.buffers); + } + acomp_ctx->comp.buffers =NULL ; }static struct zswap_pool *zswap_pool_create (char *compressor)
@@ -387,6 +408,7 @@ static struct zswap_pool *zswap_pool_create(char *compressor) cpuhp_add_fail:for_each_possible_cpu (cpu)acomp_ctx_free (per_cpu_ptr (pool->acomp_ctx, cpu)); + error:if (pool->acomp_ctx)free_percpu (pool->acomp_ctx);
@@ -776,6 +798,41 @@ static void zswap_entry_cache_free(struct zswap_entry *entry)kmem_cache_free (zswap_entry_cache, entry); } +static inline void zswap_entries_cache_free_batch ( +struct zswap_entry **entries, +u8 nr_entries) +{ +/* +* It is okay to use this to free entries allocated separately +* by zswap_entry_cache_alloc(). +*/ +kmem_cache_free_bulk (zswap_entry_cache, nr_entries, (void **)entries); +} + +static bool zswap_entries_cache_alloc_batch ( +struct zswap_store_context *zc, +gfp_t gfp) +{ +bool ret =kmem_cache_alloc_bulk (zswap_entry_cache, gfp, + zc->nr_pages, (void **)zc->entries); + +if (unlikely (!ret)) { +u8 i; + +for (i =0 ; i < zc->nr_pages; ++i) { + zc->entries[i] =zswap_entry_cache_alloc (GFP_KERNEL ); + +if (unlikely (!zc->entries[i])) { + zswap_reject_kmemcache_fail++; +zswap_entries_cache_free_batch (zc->entries, i); +return false; + } + } + } + +return true; +} +/* * Carries out the common pattern of freeing an entry's zsmalloc allocation, * freeing the entry itself, and decrementing the number of stored pages.
@@ -785,7 +842,7 @@ static void zswap_entry_free(struct zswap_entry *entry)struct zswap_pool *pool =zswap_entry_pool (entry);zswap_lru_del (entry); -if (!WARN_ON_ONCE (!pool)) { +if (likely (!WARN_ON_ONCE (!pool))) {zs_free (pool->zs_pool, entry->handle);zswap_pool_put (pool); }
@@ -839,7 +896,9 @@ static int zswap_cpu_comp_prepare(unsigned int cpu, struct hlist_node *node) { struct zswap_pool *pool = hlist_entry(node, struct zswap_pool, node);struct crypto_acomp_ctx *acomp_ctx =per_cpu_ptr (pool->acomp_ctx, cpu); +int nid =cpu_to_node (cpu);int ret = -ENOMEM ; +u8 i;/* * To handle cases where the CPU goes through online-offline-online
@@ -850,15 +909,11 @@ static int zswap_cpu_comp_prepare(unsigned int cpu, struct hlist_node *node)return 0 ; } - acomp_ctx->comp.buffer =kmalloc_node (PAGE_SIZE ,GFP_KERNEL ,cpu_to_node (cpu)); -if (!acomp_ctx->comp.buffer) -return ret; -/* * In case of an error, crypto_alloc_acomp_node() returns an * error pointer, never NULL. */ - acomp_ctx->acomp =crypto_alloc_acomp_node (pool->tfm_name,0 ,0 ,cpu_to_node (cpu)); + acomp_ctx->acomp =crypto_alloc_acomp_node (pool->tfm_name,0 ,0 , nid);if (IS_ERR (acomp_ctx->acomp)) {pr_err ("could not alloc crypto acomp %s : %pe\n" , pool->tfm_name, acomp_ctx->acomp);
@@ -870,6 +925,24 @@ static int zswap_cpu_comp_prepare(unsigned int cpu, struct hlist_node *node)if (ret) goto req_fail; +/* +* Allocate up to ZSWAP_MAX_BATCH_SIZE dst buffers for all compressors. +*/ + acomp_ctx->comp.buffers =kcalloc_node (ZSWAP_MAX_BATCH_SIZE , +sizeof (u8 *),GFP_KERNEL , nid); +if (!acomp_ctx->comp.buffers) { + ret = -ENOMEM ; + goto req_fail; + } + +for (i =0 ; i <ZSWAP_MAX_BATCH_SIZE ; ++i) { + acomp_ctx->comp.buffers[i] =kmalloc_node (PAGE_SIZE ,GFP_KERNEL , nid); +if (!acomp_ctx->comp.buffers[i]) { + ret = -ENOMEM ; + goto req_fail; + } + } +/* Skip the per-CPU request when decompression can use the stack. */ if (acomp_is_async (acomp_ctx->acomp)) { ret =zswap_acomp_req_init (&acomp_ctx->decomp, acomp_ctx->acomp);
@@ -887,7 +960,8 @@ static int zswap_cpu_comp_prepare(unsigned int cpu, struct hlist_node *node) }static bool zswap_compress (struct folio *folio,long index, -struct zswap_entry *entry,struct zswap_pool *pool) +struct zswap_entry *entry,struct zswap_pool *pool, +bool wb_enabled) {struct zswap_comp_ctx *comp_ctx;struct scatterlist input, output;
@@ -901,7 +975,7 @@ static bool zswap_compress(struct folio *folio, long index, comp_ctx = &raw_cpu_ptr (pool->acomp_ctx)->comp;mutex_lock (&comp_ctx->areq.mutex); - dst = comp_ctx->buffer; + dst = comp_ctx->buffers[0 ];sg_init_table (&input,1 );sg_set_folio (&input, folio,PAGE_SIZE , index *PAGE_SIZE );
@@ -940,12 +1014,8 @@ static bool zswap_compress(struct folio *folio, long index,* to the active LRU list in the case. */ if (comp_ret) { -rcu_read_lock (); -if (!mem_cgroup_zswap_writeback_enabled (folio_memcg (folio))) { -rcu_read_unlock (); +if (!wb_enabled) goto unlock; - } -rcu_read_unlock (); comp_ret =0 ; dlen =PAGE_SIZE ; dst =kmap_local_folio (folio, index *PAGE_SIZE );
@@ -1500,91 +1570,126 @@ static void shrink_worker(struct work_struct *w)* main API **********************************/ -static bool zswap_store_page (struct folio *folio,long index, -struct obj_cgroup *objcg, -struct zswap_pool *pool) +/* +* Store multiple pages in @zc->folio, starting from the page at index @start up +* to the page at index @end-1. +*/ +static bool zswap_store_pages (struct zswap_store_context *zc, +long start, +long end, +struct zswap_pool *pool, +struct obj_cgroup *objcg) { -swp_entry_t page_swpentry =folio_swap_entry (folio, index); -struct zswap_entry *entry, *old; -int nid =folio_nid (folio); +u8 i, store_fail_idx =0 ; +bool ret; -/* allocate entry */ - entry =zswap_entry_cache_alloc (GFP_KERNEL ); -if (!entry) { - zswap_reject_kmemcache_fail++; -return false; - } + zc->nr_pages = end - start; + zc->folio_start = start; + +VM_WARN_ON_ONCE (zc->nr_pages <=0 || zc->nr_pages >ZSWAP_MAX_BATCH_SIZE ); -if (!zswap_compress (folio, index, entry, pool)) - goto compress_failed; + ret =zswap_entries_cache_alloc_batch (zc,GFP_KERNEL ); +if (unlikely (!ret)) +return false;/* -* Set pool_idx before the xa_store() below publishes the entry, or a -* concurrent reader could resolve a stale pool_idx left by slab reuse -* to an unrelated live pool. +* We co-locate entry initialization as much as possible here to +* minimize potential cache misses. */ - entry->pool_idx = pool->idx; - entry->nid = nid; - - old =xa_store (swap_zswap_tree (page_swpentry), -swp_offset (page_swpentry), - entry,GFP_KERNEL ); -if (xa_is_err (old)) { -int err =xa_err (old); +for (i =0 ; i < zc->nr_pages; ++i) { + zc->entries[i]->handle = (unsigned long )ERR_PTR (-EINVAL ); +/* +* Set pool_idx before the xa_store() below publishes the entry, or a +* concurrent reader could resolve a stale pool_idx left by slab reuse +* to an unrelated live pool. +*/ + zc->entries[i]->pool_idx = pool->idx; + zc->entries[i]->swpentry =folio_swap_entry (zc->folio, + zc->folio_start + i); + zc->entries[i]->objcg = objcg; + zc->entries[i]->referenced = true; + zc->entries[i]->nid = zc->nid; +INIT_LIST_HEAD (&zc->entries[i]->lru); + } -WARN_ONCE (err != -ENOMEM ,"unexpected xarray error: %d\n" , err); - zswap_reject_alloc_fail++; - goto store_failed; +for (i =0 ; i < zc->nr_pages; ++i) { +if (!zswap_compress (zc->folio, zc->folio_start + i, + zc->entries[i], pool, zc->wb_enabled)) + goto store_pages_failed; } -/* -* We may have had an existing entry that became stale when -* the folio was redirtied and now the new version is being -* swapped out. Get rid of the old. -*/ -if (old) -zswap_entry_free (old); +for (i =0 ; i < zc->nr_pages; ++i) { +struct zswap_entry *old, *entry = zc->entries[i]; + + old =xa_store (swap_zswap_tree (entry->swpentry), +swp_offset (entry->swpentry), + entry,GFP_KERNEL ); +if (unlikely (xa_is_err (old))) { +int err =xa_err (old); + +WARN_ONCE (err != -ENOMEM ,"unexpected xarray error: %d\n" , err); + zswap_reject_alloc_fail++; +/* +* Entries up to this point have been stored in the +* xarray. zswap_store() will erase them from the xarray +* and call zswap_entry_free(). Local cleanup in +* 'store_pages_failed' only needs to happen for +* entries from [@i to @nr_pages). +*/ + store_fail_idx = i; + goto store_pages_failed; + } -/* -* The entry is successfully compressed and stored in the tree, there is -* no further possibility of failure. Grab refs to the pool and objcg, -* charge zswap memory, and increment zswap_stored_pages. -* The opposite actions will be performed by zswap_entry_free() -* when the entry is removed from the tree. -*/ -zswap_pool_get (pool); -if (objcg) { -obj_cgroup_get (objcg); -obj_cgroup_charge_zswap (objcg, entry->length); - } -atomic_long_inc (&zswap_stored_pages); -if (entry->length ==PAGE_SIZE ) -atomic_long_inc (&zswap_stored_incompressible_pages); +/* +* We may have had an existing entry that became stale when +* the folio was redirtied and now the new version is being +* swapped out. Get rid of the old. +*/ +if (unlikely (old)) +zswap_entry_free (old); -/* -* We finish initializing the entry while it's already in xarray. -* This is safe because: -* -* 1. Concurrent stores and invalidations are excluded by folio lock. -* -* 2. Writeback is excluded by the entry not being on the LRU yet. -* The publishing order matters to prevent writeback from seeing -* an incoherent entry. -*/ - entry->swpentry = page_swpentry; - entry->objcg = objcg; - entry->referenced = true; -if (entry->length) { -INIT_LIST_HEAD (&entry->lru); -zswap_lru_add (entry); +/* +* The entry is successfully compressed and stored in the tree, +* and further failures will be cleaned up in zswap_store(). +* Grab refs to the pool and objcg, charge zswap memory, and +* increment zswap_stored_pages. The opposite actions will be +* performed by zswap_entry_free() when the entry is removed +* from the tree. +*/ +zswap_pool_get (pool); +if (objcg) { +obj_cgroup_get (objcg); +obj_cgroup_charge_zswap (objcg, entry->length); + } +atomic_long_inc (&zswap_stored_pages); +if (unlikely (entry->length ==PAGE_SIZE )) +atomic_long_inc (&zswap_stored_incompressible_pages); + +/* +* We finish by adding the entry to the LRU while it's already +* in xarray. This is safe because: +* +* 1. Concurrent stores and invalidations are excluded by folio lock. +* +* 2. Writeback is excluded by the entry not being on the LRU yet. +* The publishing order matters to prevent writeback from seeing +* an incoherent entry. +*/ +if (likely (entry->length)) +zswap_lru_add (entry); }return true; -store_failed: -zs_free (pool->zs_pool, entry->handle); -compress_failed: -zswap_entry_cache_free (entry); +store_pages_failed: +for (i = store_fail_idx; i < zc->nr_pages; ++i) { +if (!IS_ERR_VALUE (zc->entries[i]->handle)) +zs_free (pool->zs_pool, zc->entries[i]->handle); + } + +zswap_entries_cache_free_batch (&zc->entries[store_fail_idx], + zc->nr_pages - store_fail_idx); +return false; }
@@ -1594,9 +1699,14 @@ bool zswap_store(struct folio *folio)swp_entry_t swp = folio->swap;struct obj_cgroup *objcg =NULL ;struct mem_cgroup *memcg =NULL ; -struct zswap_pool *pool; +int nid =folio_nid (folio);bool ret = false; -long index; +struct zswap_pool *pool; +long start, end; +struct zswap_store_context zc = { + .folio = folio, + .nid = nid, + };VM_WARN_ON_ONCE (!folio_test_locked (folio));VM_WARN_ON_ONCE (!folio_test_swapcache (folio));
@@ -1630,8 +1740,23 @@ bool zswap_store(struct folio *folio)mem_cgroup_put (memcg); } -for (index =0 ; index < nr_pages; ++index) { -if (!zswap_store_page (folio, index, objcg, pool)) +rcu_read_lock (); + zc.wb_enabled =mem_cgroup_zswap_writeback_enabled (folio_memcg (folio)); +rcu_read_unlock (); + +/* +* For batching and non-batching compressors, store the folio in +* batches of ZSWAP_MAX_BATCH_SIZE, where the pages in the batch +* are compressed in batches of the compressor's batch_size. This gives +* better performance than invoking zswap_store_pages() per-page, +* due to cache locality of working set structures. +*/ +for (start =0 ; start < nr_pages; start +=ZSWAP_MAX_BATCH_SIZE ) { + end =min (start +ZSWAP_MAX_BATCH_SIZE , nr_pages); + + ret =zswap_store_pages (&zc, start, end, pool, objcg); + +if (unlikely (!ret)) goto put_pool; }
Signature · 1 line
2.39.5
PATCH v3 7/8 mm: zswap: Refactor and optimize zswap_store_pages() batch operations.
In reply to Kanchana P. SridharThe various stages of zswap_store_pages() are broken down into routines that work on a batch of data when a large folio is being processed:
zswap_init_entries()
zswap_tree_stores()
The struct zswap_store_context is used to pass constraints and data among these batching components. The resulting code is highly simplified and maintainable, where each of the batching sub-routines of zswap_store() can be optimized modularly.
For instance, the zswap_tree_stores() now does the zswap atomics counter increments, getting pool references, getting objcg references and objcg zswap charging at the batch level. A new obj_cgroup_charge_zswap_many() is added to enable batch charging of compressed bytes for multiple pages/incompressible pages in the batch, in order to accurately update the MEMCG_ZSWAP* counters.
The next commit will implement batching of zswap_compress().
Subsequent patch series will be submitted, that implement ideas building upon this batching framework:
1) Replacing zsmalloc with SLUB allocator, comprising of kmem_caches for a finite # of compressed size bins. This enables bulk allocation of compressed buffers in a batch that belong to the same size bin. It gets rid of significant zsmalloc latency overhead (locks, compressed data spanning zspages, etc.), without compromising memory efficiency.
2) vmscan reclaim batching of hybrid any-order folios, that will submit a batch of folios to zswap_store()'s batching framework to optimize storing of batches of pages from different folios, in addition to within large folios.
--- include/linux/memcontrol.h | 7 ++ mm/memcontrol.c | 36 ++++++++ mm/zswap.c | 182 ++++++++++++++++++++++++------------- 3 files changed, 164 insertions(+), 61 deletions(-)
include/linux/memcontrol.h +7 -0
@@ -1946,6 +1946,8 @@ static inline void mem_cgroup_calculate_protection_path(struct mem_cgroup *root,#if defined(CONFIG_MEMCG ) && defined(CONFIG_ZSWAP )bool obj_cgroup_may_zswap (struct obj_cgroup *objcg);void obj_cgroup_charge_zswap (struct obj_cgroup *objcg,size_t size); +void obj_cgroup_charge_zswap_many (struct obj_cgroup *objcg,size_t size, +u8 nr_pages,u8 nr_incomp_pages);void obj_cgroup_uncharge_zswap (struct obj_cgroup *objcg,size_t size);bool mem_cgroup_zswap_writeback_enabled (const struct mem_cgroup *memcg);#else
@@ -1957,6 +1959,11 @@ static inline void obj_cgroup_charge_zswap(struct obj_cgroup *objcg, size_t size) { } +static inline void obj_cgroup_charge_zswap_many (struct obj_cgroup *objcg, +size_t size,u8 nr_pages, +u8 nr_incomp_pages) +{ +}static inline void obj_cgroup_uncharge_zswap (struct obj_cgroup *objcg,size_t size) {
mm/memcontrol.c +36 -0
@@ -6345,6 +6345,42 @@ void obj_cgroup_charge_zswap(struct obj_cgroup *objcg, size_t size)rcu_read_unlock (); } +/** +* obj_cgroup_charge_zswap_many - bulk charge compression backend memory +* from compressing a batch of pages. +* @objcg: the object cgroup +* @size: total compressed length of a batch of compressed objects +* @nr_pages: # of pages that were compressed to store in zswap +* @nr_incomp_pages: # of incompressible pages to store in zswap +* +* This forces the charge after obj_cgroup_may_zswap() allowed +* compression and storage in zswap for this cgroup to go ahead. +*/ +void obj_cgroup_charge_zswap_many (struct obj_cgroup *objcg,size_t size, +u8 nr_pages,u8 nr_incomp_pages) +{ +struct mem_cgroup *memcg; + +if (!cgroup_subsys_on_dfl (memory_cgrp_subsys)) +return ; + +if (obj_cgroup_is_root (objcg)) +return ; + +VM_WARN_ON_ONCE (!(current->flags &PF_MEMALLOC )); + +/* PF_MEMALLOC context, charging must succeed */ +if (obj_cgroup_charge (objcg,GFP_KERNEL , size)) +VM_WARN_ON_ONCE (1 ); + +rcu_read_lock (); + memcg =obj_cgroup_memcg (objcg); +mod_memcg_state (memcg,MEMCG_ZSWAP_B , size); +mod_memcg_state (memcg,MEMCG_ZSWAPPED , nr_pages); +mod_memcg_state (memcg,MEMCG_ZSWAP_INCOMP , nr_incomp_pages); +rcu_read_unlock (); +} +/** * obj_cgroup_uncharge_zswap - uncharge compression backend memory * @objcg: the object cgroup
mm/zswap.c +121 -61
241 lines not on this page · raw message
Signature · 1 line
2.39.5
PATCH v3 8/8 mm: zswap: Batched zswap_compress() for storing large folios in batches.
In reply to Kanchana P. SridharA new batching implementation of zswap_compress() is provided for compressors that do and do not support batching. This eliminates code duplication and facilitates code maintainability with the introduction of compress batching.
The vectorized implementation of calling the earlier zswap_compress() sequentially, one page at a time in zswap_store_pages(), is replaced with this new version of zswap_compress() that accepts multiple pages to compress as a batch.
When the crypto_acomp is used to set up the per-CPU acomp_ctx during CPU onlining, the compressor's batch_size is obtained by calling:
crypto_acomp_batch_size(acomp_ctx->acomp)
This is bounded by ZSWAP_MAX_BATCH_SIZE to limit the memory overhead and to reflect the store batch constraint of max # of pages in a batch. The clamped batch_size is stored in the per-CPU acomp_ctx->comp.compr_batch_size.
As noted in previous commits in this series, ZSWAP_MAX_BATCH_SIZE per-CPU compression buffers are available for sequential and batching compression algorithms.
Regardless of compression batching, we always process large folios in batches of ZSWAP_MAX_BATCH_SIZE.
If the compressor does not support batching, each page in the batch is compressed sequentially. For batching compressors, up to acomp_ctx->comp.compr_batch_size pages in the batch are compressed per call to crypto_acomp_compress(). The first phase of zswap_compress() thus only calls crypto_acomp_compress() until all pages in the batch have been compressed. When this finishes, the compressed buffers and compressed lengths for the batch are available in the acomp_ctx->comp.sg_table->sgl scatterlists.
A new zswap_update_entries() then iterates through the (up to) ZSWAP_MAX_BATCH_SIZE SG lists in acomp_ctx->comp.sg_table->sgl, allocates zsmalloc memory handles and copies all compressed buffers to the compressed pool. All existing functionality on zswap writeback_enabled (or not) fallbacks under various error conditions denoted by the crypto_acomp_compress() return value, and with this commit, in the individual acomp_ctx->comp.sg_table->sgl "sg->length"s set in the crypto_acomp_compress() fast path, is essentially preserved.
The most significant change is that the batched zswap_compress() incorporates Herbert's suggestion for SG lists to represent the batch's inputs/outputs to interface with the crypto API [1].
The introduction of zswap_update_entries() presents an opportunity to optimize the task of storing a batch of compressed buffers. Subsequent patch series will be submitted to optimize this phase by replacing zsmalloc with the SLUB allocator to improve zswap_store()/zswap_load() latencies.
The batching framework introduced in this series improves zswap_store() latency by a lot.
Performance data: usemem with 56% over-commit:
==============================================
Validation was done with mm-unstable as of 10-25-2026 (Baseline) and with this patch series. To minimize variation in the data and to help drive useful conclusions, I made these changes to my performance measurement methodology:
1) Reduced the number of usemem processes to 2. Each process tries to allocate 2400M, in a 3G cgroup memory.high budget, effectively accomplishing a 56.25% over-commit.
2) Modified memcontrol.c/calculate_high_delay() to always return 0 to mirror the "hack" noted by Yosry in [3] to minimize variation due to process throttling when memory.high is breached.
3) Enabled PMD THP, which forces zswap_store()/zswap_compress() latency to be in the critical path, and also generates sufficient samples for reliable bpftrace measurements.
4) v3 does not report usemem throughput metrics, instead focuses solely on bpftrace p50/p95/p99 latencies of zswap_store() and zswap_compress() to validate the benefit of store batching and compress batching respectively.
5) I used lhist() instead of hist() in bpftrace to gather latencies at a finer level of granularity and accuracy.
Testing was done on a Chromebook with a QEMU "vng" VM running in Crostini.
Chromebook Diagnostics reports:
Intel(R) Core(TM) Ultra 5 115U (10 CPUs, 4.200GHz)
RAM: 16GB
vng setup:
Memory: 7G
Swap: 3G
Pinned CPUs: 2 (E-Cores: 2,3)zswap compressor:
ZSTD
mTHP enabled:
2048kB (PMD THP)
I used the help of an LLM to write the bpftrace data post-processing scripts that generate latency percentiles.
vm-scalability/usemem setup:
cgroup memory.high: 3G
2 usemem processes, each allocates 2400M anonymous memory, reads the Silesia dataset [2] from a pre-fork-allocated buffer into the 2400M, writes to each unsigned long in the 2400M, sleeps for 10 sec, then frees the memory and exits.
usemem -w --read-silesia -O -b 1 -s 10 -n 2 2400M
usemem metrics are reported from a single run after starting the QEMU VM. Further, I have tabulated the bpftrace p50/p95/p99 latencies with these labels:
"zswap_store_higher_order_folio":
zswap_store() of PMD folios.
"zswap_store_order0_folio":
zswap_store() of order-0 folios.
"zswap_batch_compress_per_page":
zswap_compress() per-page latency of store batches generated from PMD folios.
"zswap_single_compress_per_page":
zswap_compress() per-page latency of order-0 folios.
=============================================================================
usemem PMD THP BPFTRACE
=============================================================================
Baseline zswap zswap Change Change
mm-unstable store compress with with
10-25-2026 batching batching compress compress
(patch 6-7) (patch 8) batching batching
(ns) %
=============================================================================
zswap_store_order0_folio (ns):
=============================================================================
p50 39,631 37,538 37,765 -1,866 -4.71% p95 53,088 49,690 49,349 -3,739 -7.04% p99 65,065 63,529 57,404 -7,661 -11.77% ============================================================================= zswap_store_higher_order_folio (ns): ============================================================================= p50 21,460,366 21,201,411 20,383,233 -1,077,133 -5.02% p95 25,975,341 27,558,253 25,045,350 -929,991 -3.58% p99 29,538,493 61,926,648 27,184,835 -2,353,658 -7.97% ============================================================================= zswap_single_compress_per_page (ns): ============================================================================= p50 37,545 39,320 37,218 -327 -0.87% p95 51,549 66,503 51,481 -68 -0.13% p99 89,866 134,854 87,091 -2,775 -3.09% ============================================================================= zswap_batch_compress_per_page (ns): ============================================================================= p50 40,493 2,948 7.85% p95 56,944 5,395 10.47% p99 76,242 -13,624 -15.16%
=============================================================================
==========================================================================
vmstat summary:
==========================================================================
usemem Baseline zswap zswap
PMD THP mm-unstable store compress
10-25-2026 batching batching
(patch 6-7) (patch 8)
==========================================================================
zswpin 989,367 1,050,081 949,046
zswpout 1,567,607 1,634,772 1,533,088
2048kB-alloc 2,500 2,504 2,500 2048kB-zswpout 1,557 1,552 1,539 2048kB-swpout-fallback 793 834 743 2048kB-swpout 0 0 0 pswpin 0 0 0 pswpout 0 0 0 zswpwb 0 0 0
==========================================================================
Conclusions based on the data:
==============================
The batching of zswap_store() phases, and compress batching optimizations benefit zswap_store() and zswap_compress_per_page() median and tail latencies for PMD folios and base 4K folios. The benefits are attributable to the store batching framework added in patches 6-7, that is fully utilized to improve zswap_compress() latency by giving it a batch of pages to be compressed. As the bpftrace data shows, patches 6-7 alone are not sufficient for the gains. Batch compression added in patch 8 benefits latency significantly, for these reasons:
1) Per-CPU sg_table with ZSWAP_MAX_BATCH_SIZE scatterlists that are statically mapped to compression dst buffers for that CPU, thereby minimizing local SG lists initialization, and using compact new crypto_acomp API introduced in patches 1-4 that set up an acomp_req's src/dst to be SG lists.
2) Patch 6 allocates ZSWAP_MAX_BATCH_SIZE destination buffers in the per-CPU acomp_ctx, for an added memory overhead of ~12 KB per-CPU. The trade-off pays off in significant latency improvements which can potentially be used to drive higher memory over-commit.
3) A one-shot zswap_update_entries() does all the zsmalloc handle allocation and copying for the batch. The key concept driving efficiency is that the working set is kept focused on batch-centric data pertaining to a handful of operations on the batch, to minimize cache evictions and benefit from cache locality.
4) By making zswap_compress() more efficient, we are able to lock the comp mutex per batch of ZSWAP_MAX_BATCH_SIZE (4) pages instead of per-page.
The exceptions are p50/p95 batch compress latencies (normalized per-page), even though the p99 batch compress latency improves significantly, by 15% over mm-unstable. Batch compress latency can be optimized further in subsequent patches. The batching framework established in this series allows us to make locking and bulk-processing improvements in each stage of zswap_store(). One of the ideas that shown very promising results is to replace zsmalloc with SLUB compressed caches and will be submitted separately. This is expected to resolve the p50 batch compress latency issue since it will be directly optimizing zswap_update_entries(), where compressed buffers can be binned based on size and bulk allocated from SLUB.
The speedup with zswap store batching of large folios has a propelling effect on other parts of mm, especially, by enabling large folios to be swapped out without splitting, as the data shows. This enables other mm components that are optimized for large folios, such as munmap(), folio refcounts/mapcounts updates, PTE invalidation upon freeing memory, zswap_invalidate(); which benefit from xarray offsets/range continuity. As a result, subsequent large folio swapouts are more likely to find contiguous swap slots, there are fewer large folio splits during reclaim and the cycle continues. As the data demonstrates, this directly contributes to more memory over-commit opportunity.
Architectural considerations for the zswap batching framework:
==============================================================
The zswap batching framework is designed to be hardware-agnostic. It can be leveraged by any hardware accelerator or software-based compressor.
Potential future clients of the batching framework:
===================================================
This patch-series demonstrates the performance benefits of compression batching when used in zswap_store() of large folios. Compression batching can be used for other use cases such as batching compression in zram, batch compression of different folios during reclaim, kcompressd, file systems, etc. Decompression batching can be used to improve efficiency of zswap writeback (Thanks Nhat for this idea), batching decompressions in zram, etc.
[1]: https://lore.kernel.org/all/[email protected]/T/#m99aea2ce3d284e6c5a3253061d97b08c4752a798
[2]: https://wanos.co/assets/silesia.tar
--- mm/zswap.c | 293 +++++++++++++++++++++++++++++++++++++++-------------- 1 file changed, 217 insertions(+), 76 deletions(-)
mm/zswap.c +217 -76
@@ -148,9 +148,14 @@ struct zswap_acomp_req {/* * The compression mutex also protects the output buffer(s). +* +* @compr_batch_size: The max batch size of the compression algorithm, +* bounded by ZSWAP_MAX_BATCH_SIZE. */ struct zswap_comp_ctx {struct zswap_acomp_req areq; +u8 compr_batch_size; +struct sg_table *sg_table;u8 **buffers; };
@@ -234,9 +239,13 @@ struct zswap_entry {struct zswap_store_context {struct folio *folio;long folio_start; +unsigned int slen; +u8 nr_batch_pages;bool wb_enabled;int nid;u8 nr_pages; +struct zs_pool *zs_pool; +struct zswap_comp_ctx *comp_ctx;struct zswap_entry *entries[ZSWAP_MAX_BATCH_SIZE ]; };
@@ -324,6 +333,13 @@ static void acomp_ctx_free(struct crypto_acomp_ctx *acomp_ctx)kfree (acomp_ctx->comp.buffers); } acomp_ctx->comp.buffers =NULL ; + acomp_ctx->comp.compr_batch_size =0 ; + +if (acomp_ctx->comp.sg_table) { +sg_free_table (acomp_ctx->comp.sg_table); +kfree (acomp_ctx->comp.sg_table); + } + acomp_ctx->comp.sg_table =NULL ; }static struct zswap_pool *zswap_pool_create (char *compressor)
@@ -897,6 +913,7 @@ static int zswap_cpu_comp_prepare(unsigned int cpu, struct hlist_node *node) struct zswap_pool *pool = hlist_entry(node, struct zswap_pool, node);struct crypto_acomp_ctx *acomp_ctx =per_cpu_ptr (pool->acomp_ctx, cpu);int nid =cpu_to_node (cpu); +struct scatterlist *sg;int ret = -ENOMEM ;u8 i;
@@ -921,6 +938,13 @@ static int zswap_cpu_comp_prepare(unsigned int cpu, struct hlist_node *node) goto fail; } +/* +* Constrain the crypto algorithm's batch size by ZSWAP_MAX_BATCH_SIZE, +* store this in the per-CPU zswap_comp_ctx's @compr_batch_size. +*/ + acomp_ctx->comp.compr_batch_size =min (ZSWAP_MAX_BATCH_SIZE , +crypto_acomp_batch_size (acomp_ctx->acomp)); + ret =zswap_acomp_req_init (&acomp_ctx->comp.areq, acomp_ctx->acomp);if (ret) goto req_fail;
@@ -943,6 +967,26 @@ static int zswap_cpu_comp_prepare(unsigned int cpu, struct hlist_node *node) } } + acomp_ctx->comp.sg_table =kmalloc (sizeof (*acomp_ctx->comp.sg_table), +GFP_KERNEL ); +if (!acomp_ctx->comp.sg_table) { + ret = -ENOMEM ; + goto req_fail; + } + +if (sg_alloc_table (acomp_ctx->comp.sg_table,ZSWAP_MAX_BATCH_SIZE , +GFP_KERNEL )) { + ret = -ENOMEM ; + goto req_fail; + } + +/* +* Statically map the per-CPU destination buffers to the per-CPU +* SG lists; crucial for performance. +*/ +for_each_sg (acomp_ctx->comp.sg_table->sgl, sg,ZSWAP_MAX_BATCH_SIZE , i) +sg_set_buf (sg, acomp_ctx->comp.buffers[i],PAGE_SIZE ); +/* Skip the per-CPU request when decompression can use the stack. */ if (acomp_is_async (acomp_ctx->acomp)) { ret =zswap_acomp_req_init (&acomp_ctx->decomp, acomp_ctx->acomp);
@@ -959,92 +1003,180 @@ static int zswap_cpu_comp_prepare(unsigned int cpu, struct hlist_node *node)return ret; } -static bool zswap_compress (struct folio *folio,long index, -struct zswap_entry *entry,struct zswap_pool *pool, -bool wb_enabled) +/* +* zswap_update_entries() is called only when all pages in a store batch have +* been successfully compressed (subsumes the writeback_enabled fallback in case +* of compress errors). The only error that can occur in this phase is zs_malloc +* allocation failure, upon which an error status is returned to +* zswap_store_pages(). +*/ +static inline bool zswap_update_entries (struct zswap_store_context *zc) { -struct zswap_comp_ctx *comp_ctx; -struct scatterlist input, output; -int comp_ret =0 , alloc_ret =0 ; -unsigned int dlen =PAGE_SIZE ; +gfp_t gfp =GFP_NOWAIT | __GFP_NORETRY | __GFP_HIGHMEM | __GFP_MOVABLE; +struct zswap_entry *entry; +struct scatterlist *sg;unsigned long handle; -gfp_t gfp; -u8 *dst; -bool mapped = false; +unsigned int dlen; +u8 batch_iter; +u8 *buf; + +/* +* For each output SG list in @acomp_ctx->req->sg_table->sgl, +* the @sg->length should be set to either the page's compressed +* length (success), or it's negative value compression error. +*/ +for_each_sg (zc->comp_ctx->sg_table->sgl, sg, zc->nr_pages, batch_iter) { + entry = zc->entries[batch_iter]; + dlen = sg->length; + +prefetchw (entry); - comp_ctx = &raw_cpu_ptr (pool->acomp_ctx)->comp; -mutex_lock (&comp_ctx->areq.mutex); + handle =zs_malloc (zc->zs_pool, dlen, gfp, zc->nid); - dst = comp_ctx->buffers[0 ]; -sg_init_table (&input,1 ); -sg_set_folio (&input, folio,PAGE_SIZE , index *PAGE_SIZE ); +if (unlikely (IS_ERR_VALUE (handle))) { +if (PTR_ERR ((void *)handle) == -ENOSPC ) + zswap_reject_compress_poor++; +else + zswap_reject_alloc_fail++; + +return false; + } -sg_init_one (&output, dst,PAGE_SIZE ); -acomp_request_set_params (comp_ctx->areq.req, &input, &output, -PAGE_SIZE , dlen); + entry->length = dlen; + entry->handle = handle; + +if (unlikely (dlen ==PAGE_SIZE )) { + buf =kmap_local_folio (zc->folio, + (zc->folio_start + batch_iter) *PAGE_SIZE ); +zs_obj_write (zc->zs_pool, handle, buf, dlen); +kunmap_local (buf); + }else { + buf = zc->comp_ctx->buffers[batch_iter]; +zs_obj_write (zc->zs_pool, handle, buf, dlen); + } + } + +return true; +} + +/* +* zswap_compress() batching implementation for sequential and batching +* compressors. +* +* Description: +* ============ +* Compress multiple @zc->nr_pages in @zc->folio starting from the +* @zc->folio_start index in batches of @zc->nr_batch_pages; the latter +* representing the lesser of the @zc->nr_pages to be batch compressed and the +* compressor's batch size. +* +* @zc->nr_pages can be in (1, ZSWAP_MAX_BATCH_SIZE] even if the compressor does +* not support batching. +* +* If @nr_batch_pages is 1, each page is compressed sequentially. +* +* If @nr_batch_pages is > 1, the compressor may use batching to optimize +* compression. +* +* In all cases, the store processing that needs to happen post-compression +* occurs in batches of @zc->nr_pages. Error handling is minimal, mostly to +* update zswap statistics, with catch-all cleanup for the batch occuring in the +* @store_pages_failed label in zswap_store_pages(). +*/ +static bool zswap_compress (struct zswap_store_context *zc) +{ +u8 batch_start, comp_batch_iter; +struct scatterlist *sg, *dst_sg; +int err;/* -* it maybe looks a little bit silly that we send an asynchronous request, -* then wait for its completion synchronously. This makes the process look -* synchronous in fact. -* Theoretically, acomp supports users send multiple acomp requests in one -* acomp instance, then get those requests done simultaneously. but in this -* case, zswap actually does store and load page by page, there is no -* existing method to send the second page before the first page is done -* in one thread doing zswap. -* but in different threads running on different cpu, we have different -* acomp instance, and compression and decompression use separate -* requests, so multiple threads can do (de)compression in parallel. +* @nr_batch_pages is constrained by the per-CPU comp_ctx's compressor +* batch size and the @zc->nr_pages in this store batch. @slen indicates +* the total source length bytes for @nr_batch_pages. */ - comp_ret =crypto_wait_req (crypto_acomp_compress (comp_ctx->areq.req), - &comp_ctx->areq.wait); - dlen = comp_ctx->areq.req->dlen; -if (!comp_ret) { -if (dlen ==PAGE_SIZE ) - comp_ret = -ENOSPC ; -else if (WARN_ON_ONCE (!dlen || dlen >PAGE_SIZE )) - comp_ret = -EINVAL ; - } + zc->nr_batch_pages =min (zc->comp_ctx->compr_batch_size, zc->nr_pages); + zc->slen = zc->nr_batch_pages *PAGE_SIZE ;/* -* If a page cannot be compressed into a size smaller than PAGE_SIZE, -* save the content as is without a compression, to keep the LRU order -* of writebacks. If writeback is disabled, reject the page since it -* only adds metadata overhead. swap_writeout() will put the page back -* to the active LRU list in the case. +* Compress the @nr_pages in @folio starting at index @folio_start +* in batches of @nr_batch_pages. */ -if (comp_ret) { -if (!wb_enabled) - goto unlock; - comp_ret =0 ; - dlen =PAGE_SIZE ; - dst =kmap_local_folio (folio, index *PAGE_SIZE ); - mapped = true; - } +for (batch_start =0 ; batch_start < zc->nr_pages; + batch_start += zc->nr_batch_pages) { +/* +* Send @nr_batch_pages to crypto_acomp for compression: +* +* These pages are in @folio's range of indices in the interval +* [@folio_start + @batch_start, +* @folio_start + @batch_start + @nr_batch_pages). +* +* The source folio pages in the batch are directly submitted to +* crypto_acomp via acomp_request_set_src_folio(). +*/ +acomp_request_set_src_folio (zc->comp_ctx->areq.req, zc->folio, + (zc->folio_start + batch_start) *PAGE_SIZE , + zc->slen); + + dst_sg = zc->comp_ctx->sg_table->sgl + batch_start; + dst_sg->length = zc->slen; + +/* +* The per-CPU @zc->comp_ctx->sg_table scatterlists are statically +* mapped to the per-CPU dst @buffers at pool creation time. +* There are ZSWAP_MAX_BATCH_SIZE per-CPU SG lists and dst +* buffers allocated for all compressors. +*/ +acomp_request_set_dst_sg (zc->comp_ctx->areq.req, + dst_sg, zc->slen); + + err =crypto_wait_req (crypto_acomp_compress (zc->comp_ctx->areq.req), + &zc->comp_ctx->areq.wait); + +/* +* If a page cannot be compressed into a size smaller than +* PAGE_SIZE, save the content as is without a compression, to +* keep the LRU order of writebacks. If writeback is disabled, +* reject the page since it only adds metadata overhead. +* swap_writeout() will put the page back to the active LRU list +* in the case. +*/ +if (err && !zc->wb_enabled) + goto compress_error; - gfp =GFP_NOWAIT | __GFP_NORETRY | __GFP_HIGHMEM | __GFP_MOVABLE; - handle =zs_malloc (pool->zs_pool, dlen, gfp,folio_nid (folio)); -if (IS_ERR_VALUE (handle)) { - alloc_ret =PTR_ERR ((void *)handle); - goto unlock; +/* +* For each output SG list in @acomp_ctx->req->sg_table->sgl, +* the @sg->length should be set to either the page's compressed +* length (success), or it's negative value compression error. +*/ +for_each_sg (dst_sg, sg, zc->nr_batch_pages, comp_batch_iter) { + +if (sg->length <=0 || sg->length >=PAGE_SIZE ) { +if (!zc->wb_enabled) { +if (sg->length ==PAGE_SIZE ) + sg->length = -ENOSPC ; +else if (WARN_ON_ONCE (!sg->length || sg->length >PAGE_SIZE )) + sg->length = -EINVAL ; + goto compress_error; + } + + sg->length =PAGE_SIZE ; + } + } } -zs_obj_write (pool->zs_pool, handle, dst, dlen); - entry->handle = handle; - entry->length = dlen; +return zswap_update_entries (zc); -unlock: -if (mapped) -kunmap_local (dst); -if (comp_ret == -ENOSPC || alloc_ret == -ENOSPC ) - zswap_reject_compress_poor++; -else if (comp_ret) - zswap_reject_compress_fail++; -else if (alloc_ret) - zswap_reject_alloc_fail++; - -mutex_unlock (&comp_ctx->areq.mutex); -return comp_ret ==0 && alloc_ret ==0 ; +compress_error: +for_each_sg (dst_sg, sg, zc->nr_batch_pages, comp_batch_iter) { +if ((int )sg->length <0 ) { +if ((int )sg->length == -ENOSPC ) + zswap_reject_compress_poor++; +else + zswap_reject_compress_fail++; + } + } + +return false; } static bool __zswap_decompress(struct zswap_entry *entry,
@@ -1730,11 +1862,19 @@ static bool zswap_store_pages(struct zswap_store_context *zc,zswap_init_entries (zc, pool, objcg); -for (i =0 ; i < zc->nr_pages; ++i) { -if (!zswap_compress (zc->folio, zc->folio_start + i, - zc->entries[i], pool, zc->wb_enabled)) - goto store_pages_failed; - } +/* +* Locking the comp_ctx mutex once per store batch results in better +* performance as compared to locking per compress batch. +*/ + zc->comp_ctx = &raw_cpu_ptr (pool->acomp_ctx)->comp; +mutex_lock (&zc->comp_ctx->areq.mutex); + + ret =zswap_compress (zc); + +mutex_unlock (&zc->comp_ctx->areq.mutex); + +if (unlikely (!ret)) + goto store_pages_failed; ret =zswap_tree_stores (zc, pool, objcg);
@@ -1800,6 +1940,7 @@ bool zswap_store(struct folio *folio)mem_cgroup_put (memcg); } + zc.zs_pool = pool->zs_pool;rcu_read_lock (); zc.wb_enabled =mem_cgroup_zswap_writeback_enabled (folio_memcg (folio));rcu_read_unlock ();
Signature · 1 line
2.39.5
Yosry Ahmed
maintainerQuoted 32 lines · Kanchana P. Sridhar
=============================================================================
usemem PMD THP BPFTRACE
=============================================================================
Baseline zswap zswap Change Change
mm-unstable store compress with with
10-25-2026 batching batching compress compress
(patch 6-7) (patch 8) batching batching
(ns) %
=============================================================================
zswap_store_order0_folio(ns):
=============================================================================p50 39,631 37,538 37,765 -1,866 -4.71% p95 53,088 49,690 49,349 -3,739 -7.04% p99 65,065 63,529 57,404 -7,661 -11.77% ============================================================================= zswap_store_higher_order_folio (ns): ============================================================================= p50 21,460,366 21,201,411 20,383,233 -1,077,133 -5.02% p95 25,975,341 27,558,253 25,045,350 -929,991 -3.58% p99 29,538,493 61,926,648 27,184,835 -2,353,658 -7.97% ============================================================================= zswap_single_compress_per_page (ns): ============================================================================= p50 37,545 39,320 37,218 -327 -0.87% p95 51,549 66,503 51,481 -68 -0.13% p99 89,866 134,854 87,091 -2,775 -3.09% ============================================================================= zswap_batch_compress_per_page (ns): ============================================================================= p50 40,493 2,948 7.85% p95 56,944 5,395 10.47% p99 76,242 -13,624 -15.16%
Quoted 1 line
=============================================================================
As with previous versions, I cannot reproduce these results. I tried with a QEMU VM on a Cascade Lake host. I saw marginal improvements with patch 6 and no significant gain with patches 7-8.
I noticed that you omitted high-level metrics (e.g. throughput), I didn't see significant changes in throughput, and I also ran a kernel build and saw a small regression in some cases (probably with order-0 folios) and no significant improvement otherwise. Even if latency for some functions does improve slightly, it doesn't matter if it doesn't show up on higher-level metrics.
It could be the difference in setups, or the overhead of bpftrace hooks for zswap_compress() might affect measured zswap_store() latencies, as you will have fewer zswap_compress() calls with patch 8.
As advised previously, I would trim down patch 6 to the bare minimum (without all the scaffolding and prep work for patches 7-8), and send it separately (with patch 5), and we can check if it's worth adding, I am still concerned it might be adding overhead for order-0 to be honest.
I am questioning the benefit of batching here without HW batching, so far the results are more concerning than encouraging. I already spent too much time trying to reproduce the performance gains. Also, sending new versions without resolving discussions on previous ones is not really productive.
Quoted 56 lines · Yosry Ahmed
Quoted 32 lines · Kanchana P. Sridhar
=============================================================================
usemem PMD THP BPFTRACE
=============================================================================
Baseline zswap zswap Change Change
mm-unstable store compress with with
10-25-2026 batching batching compress compress
(patch 6-7) (patch 8) batching batching
(ns) %
=============================================================================
zswap_store_order0_folio(ns):
=============================================================================p50 39,631 37,538 37,765 -1,866 -4.71% p95 53,088 49,690 49,349 -3,739 -7.04% p99 65,065 63,529 57,404 -7,661 -11.77% ============================================================================= zswap_store_higher_order_folio (ns): ============================================================================= p50 21,460,366 21,201,411 20,383,233 -1,077,133 -5.02% p95 25,975,341 27,558,253 25,045,350 -929,991 -3.58% p99 29,538,493 61,926,648 27,184,835 -2,353,658 -7.97% ============================================================================= zswap_single_compress_per_page (ns): ============================================================================= p50 37,545 39,320 37,218 -327 -0.87% p95 51,549 66,503 51,481 -68 -0.13% p99 89,866 134,854 87,091 -2,775 -3.09% ============================================================================= zswap_batch_compress_per_page (ns): ============================================================================= p50 40,493 2,948 7.85% p95 56,944 5,395 10.47% p99 76,242 -13,624 -15.16%Quoted 1 line
=============================================================================
As with previous versions, I cannot reproduce these results. I tried with a QEMU VM on a Cascade Lake host. I saw marginal improvements with patch 6 and no significant gain with patches 7-8.
I noticed that you omitted high-level metrics (e.g. throughput), I didn't see significant changes in throughput, and I also ran a kernel build and saw a small regression in some cases (probably with order-0 folios) and no significant improvement otherwise. Even if latency for some functions does improve slightly, it doesn't matter if it doesn't show up on higher-level metrics.
It could be the difference in setups, or the overhead of bpftrace hooks for
zswap_compress()might affect measuredzswap_store()latencies, as you will have fewerzswap_compress()calls with patch 8.As advised previously, I would trim down patch 6 to the bare minimum (without all the scaffolding and prep work for patches 7-8), and send it separately (with patch 5), and we can check if it's worth adding, I am still concerned it might be adding overhead for order-0 to be honest.
Quoted 5 lines
I am questioning the benefit of batching here without HW batching, so far the results are more concerning than encouraging. I already spent too much time trying to reproduce the performance gains. Also, sending new versions without resolving discussions on previous ones is not really productive.
Thanks for taking the time to test and provide feedback, Yosry.
I omitted high-level metrics like throughput since you raised a concern about variation. Hence, I decided to go with the "ultimate truth" which is bpftrace latency measurements.
For performance validation, I only have access to a 16G Chromebook with a Crostini VM to which 13G is allocated. I am setting aside 3G for swap and 7G for QEMU. Within the QEMU VM, I am selecting 2 E-cores to run the 2 usemem processes to minimize variation due to frequency fluctuations. This setup has its own limitations such as being unable to fix the frequency.
Despite all this, I have been able to consistently replicate the bpftrace latency improvements with patch 8. My results show regression with patch 6-7, and improvement only with patch 8. As noted in the cover-letter, patch 8 adds significant latency improvement for order-0 folios. This is with ZSTD. FWIW, these are bpftrace latency improvements in a significantly memory constrained environment and this ought to be a useful data point. I have been posting general vmstat "system health" metrics to prove that nothing unexpected is going on.
My data shows that order-0 zswap_store() tail latency improves by 11% over mm-unstable with patch 8, although p50 is roughly similar between patches 6-7 and 8. PMD store latency is significantly better at p50 and p99 with patch 8.
Likewise, p99 latency of zswap_compress() alone improves by 3% with patch 8, while it degrades with patch 6-7. p50 is almost the same with patch 8, although patch 6-7 regresses. The page-normalized batch_compress() p99 latency is better by 15%. All comparisons are wrt mm-unstable.
Hence, I believe even in a memory-constrained setup such as mine, batching significantly (7-15%) improves p99 in all cases, with 4-5% gains in p50: for order-0 and PMD folios.
In addition to the latency benefits shown through my validation, I would like to also emphasize the strategic benefit of moving zswap to use batching, as noted under "Conclusions" in my cover letter. IOW, batching within large folios to improve workload metrics can only do so much (in proportion to the # of large folios reclaimed and contribution of zswap_store() to the overall allocation-reclaim path). What this patch series attempts to provide is a zswap batching framework upon which additional improvements can be built, such as reclaiming multiple folios as a batch. I believe that compress batching is essential to support these next steps, which improve workload throughput/latency metrics even more than simply batching within folios. This is with ZSTD.
Due to the environmental constraints, I haven't been able to reliably use kernel compilation for validation.
I have been doing my best to resolve previous discussions by incorporating the calculate_high_delay() hack that you suggested to reduce variation. The data posted in v3 comprehends this suggestion.
I would like to suggest/request the following, as next steps:
1) Can we reconcile our methodologies for usemem bpftrace
measurements: # of vCPU, usemem command line, reading Silesia, etc. Also swap space and QEMU memory allocation, using linear lhists() in bpftrace, etc. If you can share these details, it would be very helpful. Thanks again for your time!
2) Does the kernel community offer servers for contributors to test their optimizations at scale?
3) Can I please request the help of other zswap maintainers/reviewers to validate this series with ZSTD on a server setup with usemem running on 30+ cores, under high over-commit ratios; through bpftrace and/or throughput metrics.
Thanks,
Kanchana
[email protected]
AI reviewRe: [PATCH v3 3/8] crypto: acomp - Add trivial segmentation wrapper.
In reply to Kanchana P. SridharThank you for your contribution! Sashiko AI review found 2 potential issues to consider: - [Medium] crypto: acomp - missing non-zero unit_size check in crypto_acomp_compress() - [Medium] crypto…
Thank you for your contribution! Sashiko AI review found 2 potential issues to consider:
- [Medium] crypto: acomp - missing non-zero unit_size check in crypto_acomp_compress()
- [Medium] crypto: acomp - union type confusion in acomp_reqchain_finish() on virt dst
--
commit c2a605231cabd5e0e15d6d778b3943a2c118d756
Author: Kanchana P. Sridhar <[email protected]>
crypto: acomp - Add trivial segmentation wrapper.
This commit adds a single-segment compression wrapper for algorithms that do not natively support segmentation, returning the compressed length or error in req->dst->length when req->unit_size is set.
Quoted 4 lines
crypto/acompress.c +0
-0
[ ... ]
Quoted 12 lines
crypto/acompress.c +4
-0@@ -242,6 +246,10 @@ static int acomp_reqchain_finish(struct acomp_req *req, int err) {acomp_reqchain_virt (req);acomp_restore_req (req); + +if (req->unit_size) + req->dst->length =unlikely (err) ? err : req->dlen; +return err; }
crypto/acompress.c static int acomp_do_req_chain(struct acomp_req *req, bool comp) · line 278
Quoted 5 lines
crypto/acompress.c +2
-0@@ -266,10 +274,13 @@ static int acomp_do_req_chain(struct acomp_req *req, bool comp) {int err; +if (unlikely (req->unit_size && req->slen > req->unit_size)) +return -ENOSYS ;
[Severity: Medium]
Can acomp_reqchain_finish() corrupt the destination buffer when a request with a non-zero req->unit_size uses virtual addresses?
In struct acomp_req, dst and dvirt share an anonymous union. When acomp_do_req_chain() handles a virtual-address request (CRYPTO_ACOMP_REQ_DST_VIRT), acomp_reqchain_virt() restores the raw u8 * buffer pointer into req->dvirt before acomp_restore_req() restores req->unit_size. Writing to req->dst->length then treats req->dvirt as a struct scatterlist * and overwrites 4 bytes inside the caller's output buffer.
In addition, acomp_do_req_chain() is called for decompression (comp == false), so a decompression request with req->unit_size set will also hit the compression-only req->slen > req->unit_size check and write to req->dst->length.
Quoted 9 lines
crypto/acompress.c +1
-1acomp_save_req (req, acomp_reqchain_done); err =acomp_do_one_req (req, comp); -if (err == -EBUSY || err == -EINPROGRESS ) +if (unlikely (err == -EBUSY || err == -EINPROGRESS ))return err;return acomp_reqchain_finish (req, err);
crypto/acompress.c int crypto_acomp_compress(struct acomp_req *req) · line 299
Quoted 10 lines
crypto/acompress.c +7
-0@@ -279,10 +290,27 @@ int crypto_acomp_compress(struct acomp_req *req) {struct crypto_acomp *tfm =crypto_acomp_reqtfm (req); +if (likely ((req->unit_size == req->slen) &&acomp_request_issg (req) && + !crypto_acomp_req_seg (tfm) && !acomp_is_async (tfm))) { +int err = tfm->compress (req); + + req->dst->length =unlikely (err) ? err : req->dlen; +return err; + }
[Severity: Medium]
Should this fast-path check also verify that req->unit_size is non-zero?
Callers that do not use segmentation leave req->unit_size at 0. If such a caller submits a scatterlist request with req->slen == 0 on a synchronous non-segmented transform, req->unit_size == req->slen is true, so crypto_acomp_compress() enters the segmentation path and writes the negative error from tfm->compress(req) into req->dst->length, corrupting the caller's scatterlist or causing a NULL pointer dereference if req->dst is NULL (which can also happen when req->unit_size is non-zero and req->dst is NULL here or in acomp_reqchain_finish()).
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
[email protected]
AI reviewRe: [PATCH v3 4/8] crypto: acomp - Add API to get a compression algorithm's batch-size.
In reply to Kanchana P. SridharThank you for your contribution! Sashiko AI review found 1 potential issue to consider: - [Medium] crypto: acomp: batch_size gated on acomp_is_async() instead of REQ_SEG --
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider:
- [Medium] crypto: acomp: batch_size gated on acomp_is_async() instead of REQ_SEG
--
commit 2fa95d403dcc5d6c7b702698b0e306a5d643d3b3
Author: Kanchana P. Sridhar <[email protected]>
crypto: acomp - Add API to get a compression algorithm's batch-size.
This commit adds a batch_size field to struct acomp_alg and introduces crypto_acomp_batch_size() to query the maximum batch size supported by an acomp transform.
Quoted 11 lines
crypto/acompress.c +3
-0@@ -327,6 +327,20 @@ int crypto_acomp_decompress(struct acomp_req *req) }EXPORT_SYMBOL_GPL (crypto_acomp_decompress); +unsigned int crypto_acomp_batch_size (struct crypto_acomp *tfm) +{ + if (acomp_is_async(tfm) &&
crypto/acompress.c int crypto_acomp_decompress(struct acomp_req *req) · line 342
Quoted 10 lines
crypto/acompress.c +10
-0+ (crypto_comp_alg_common(tfm)->base.cra_type == &crypto_acomp_type)) { +struct acomp_alg *alg =crypto_acomp_alg (tfm); + +if (alg && alg->batch_size >1 ) +return alg->batch_size; + } + +return 1 ; +} +EXPORT_SYMBOL_GPL (crypto_acomp_batch_size);
[Severity: Medium]
Should crypto_acomp_batch_size() check crypto_acomp_req_seg(tfm) (or have crypto_register_acomp() enforce CRYPTO_ALG_REQ_SEG when batch_size
Quoted 1 line
1) instead of gating on
acomp_is_async(tfm)?
crypto_acomp_compress() dispatches segmented requests based on crypto_acomp_req_seg(tfm):
crypto/acompress.c:crypto_acomp_compress() {
...
if (req->unit_size && acomp_request_issg(req)) {
if (!crypto_acomp_req_seg(tfm))
return acomp_do_req_chain(req, true);
return tfm->compress(req);
}
...
}If an asynchronous acomp_alg sets batch_size > 1 without setting CRYPTO_ALG_REQ_SEG in cra_flags, crypto_acomp_batch_size() returns alg->batch_size, but crypto_acomp_compress() routes batched requests (req->slen > req->unit_size) to acomp_do_req_chain(), which fails with -ENOSYS. Conversely, requiring acomp_is_async(tfm) forces crypto_acomp_batch_size() to return 1 for synchronous acomp_alg transforms even when they set batch_size > 1 and CRYPTO_ALG_REQ_SEG.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=4
[email protected]
AI reviewRe: [PATCH v3 5/8] mm: zswap: Store the folio's node id in the zswap_entry.
In reply to Kanchana P. SridharThank you for your contribution! Sashiko AI review found 1 potential issue to consider: - [Medium] mm: zswap: shrink_memcg() skips entries on nodes outside N_NORMAL_MEMORY --
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider:
- [Medium] mm: zswap: shrink_memcg() skips entries on nodes outside N_NORMAL_MEMORY
--
commit f6e1c6117279d2be409376eb57c05492fc654dea
Author: Kanchana P. Sridhar <[email protected]>
mm: zswap: Store the folio's node id in the zswap_entry.
This commit stores the swapped-out folio's NUMA node ID in struct zswap_entry for list_lru tracking and switches zswap_entry slab allocations to be node-agnostic.
Quoted 4 lines
mm/zswap.c +0
-0
[ ... ]
Quoted 23 lines
mm/zswap.c +1
-7@@ -680,14 +685,8 @@ static int zswap_enabled_param_set(const char *val,* lru functions **********************************/ -static inline int entry_to_nid (struct zswap_entry *entry) -{ -return page_to_nid (virt_to_page (entry)); -} -static void zswap_lru_add (struct zswap_entry *entry) { -int nid =entry_to_nid (entry);struct mem_cgroup *memcg;/* @@ -704,19 +703,18 @@ static void zswap_lru_add(struct zswap_entry *entry)rcu_read_lock (); memcg =obj_cgroup_memcg (entry->objcg);/* will always succeed */ -list_lru_add (&zswap_list_lru, &entry->lru, nid, memcg); +list_lru_add (&zswap_list_lru, &entry->lru, entry->nid, memcg);rcu_read_unlock (); }
[ ... ]
Quoted 13 lines
mm/zswap.c +2
-1@@ -1539,9 +1538,10 @@ static bool zswap_store_page(struct folio *folio, long index, {swp_entry_t page_swpentry =folio_swap_entry (folio, index);struct zswap_entry *entry, *old; +int nid =folio_nid (folio);/* allocate entry */ - entry =zswap_entry_cache_alloc (GFP_KERNEL ,folio_nid (folio)); + entry =zswap_entry_cache_alloc (GFP_KERNEL );if (!entry) { zswap_reject_kmemcache_fail++;return false;
mm/zswap.c static bool zswap_store_page(struct folio *folio, long index, · line 1559
Quoted 4 lines
mm/zswap.c +1
-0@@ -1556,6 +1556,7 @@ static bool zswap_store_page(struct folio *folio, long index,* to an unrelated live pool. */ entry->pool_idx = pool->idx; + entry->nid = nid;
[Severity: Medium]
Can shrink_memcg() miss zswap_list_lru entries when a folio comes from a node outside N_NORMAL_MEMORY?
Previously, entry_to_nid() derived the list_lru node ID from the GFP_KERNEL slab page backing struct zswap_entry, which was guaranteed to be on a node in N_NORMAL_MEMORY. Now entry->nid is set to folio_nid(folio), and movable swap-backed folios can reside on ZONE_MOVABLE-only or ZONE_HIGHMEM-only nodes that are in N_MEMORY but not in N_NORMAL_MEMORY. Because shrink_memcg() only walks N_NORMAL_MEMORY nodes:
mm/zswap.c:shrink_memcg() {
...
for_each_node_state(nid, N_NORMAL_MEMORY) {
unsigned long nr_to_walk = SWAP_CLUSTER_MAX;
shrunk += list_lru_walk_one(&zswap_list_lru, nid, memcg,
&shrink_memcg_cb, &ctl, &nr_to_walk);
scanned += SWAP_CLUSTER_MAX - nr_to_walk;
}
...
}entries added under entry->nid for nodes outside N_NORMAL_MEMORY are skipped by both per-cgroup and global pool limit reclaim, leaving them unreclaimable by shrink_memcg() and causing premature zswap_store() rejections once limits are reached. Should shrink_memcg() iterate over N_MEMORY instead?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=5
[email protected]
AI reviewRe: [PATCH v3 7/8] mm: zswap: Refactor and optimize zswap_store_pages() batch operations.
In reply to Kanchana P. SridharThank you for your contribution! Sashiko AI review found 3 potential issues to consider: - [High] mm: zswap: missing !CONFIG_MEMCG stub for obj_cgroup_get_many() - [Low] mm: zswap: unconditional…
Thank you for your contribution! Sashiko AI review found 3 potential issues to consider:
- [High] mm: zswap: missing !CONFIG_MEMCG stub for obj_cgroup_get_many()
- [Low] mm: zswap: unconditional incompressible page stat updates on zero delta
- [Low] mm: memcontrol: unused and duplicated obj_cgroup_charge_zswap() --
commit 55351c710f10c5fbb79d9201d80496dc658bbc19
Author: Kanchana P. Sridhar <[email protected]>
mm: zswap: Refactor and optimize zswap_store_pages() batch operations.
This commit factors zswap_store_pages() into batch helpers zswap_init_entries() and zswap_tree_stores(), and introduces obj_cgroup_charge_zswap_many() to charge and account zswap batches at once.
Quoted 33 lines
mm/memcontrol.c +25
-0@@ -6368,6 +6368,42 @@ void obj_cgroup_charge_zswap(struct obj_cgroup *objcg, size_t size)rcu_read_unlock (); } +/** +* obj_cgroup_charge_zswap_many - bulk charge compression backend memory +* from compressing a batch of pages. +* @objcg: the object cgroup +* @size: total compressed length of a batch of compressed objects +* @nr_pages: # of pages that were compressed to store in zswap +* @nr_incomp_pages: # of incompressible pages to store in zswap +* +* This forces the charge after obj_cgroup_may_zswap() allowed +* compression and storage in zswap for this cgroup to go ahead. +*/ +void obj_cgroup_charge_zswap_many (struct obj_cgroup *objcg,size_t size, +u8 nr_pages,u8 nr_incomp_pages) +{ +struct mem_cgroup *memcg; + +if (!cgroup_subsys_on_dfl (memory_cgrp_subsys)) +return ; + +if (obj_cgroup_is_root (objcg)) +return ; + +VM_WARN_ON_ONCE (!(current->flags &PF_MEMALLOC )); + +/* PF_MEMALLOC context, charging must succeed */
mm/memcontrol.c void obj_cgroup_charge_zswap(struct obj_cgroup *objcg, size_t size) · line 6405
Quoted 10 lines
mm/memcontrol.c +10
-0+if (obj_cgroup_charge (objcg,GFP_KERNEL , size)) +VM_WARN_ON_ONCE (1 ); + +rcu_read_lock (); + memcg =obj_cgroup_memcg (objcg); +mod_memcg_state (memcg,MEMCG_ZSWAP_B , size); +mod_memcg_state (memcg,MEMCG_ZSWAPPED , nr_pages); +mod_memcg_state (memcg,MEMCG_ZSWAP_INCOMP , nr_incomp_pages); +rcu_read_unlock (); +}
[Severity: Low]
Could obj_cgroup_charge_zswap() be removed now that its only caller in mm/zswap.c has switched to obj_cgroup_charge_zswap_many(), just as zswap_pool_get() was replaced by zswap_pool_get_many()? Leaving obj_cgroup_charge_zswap() in mm/memcontrol.c and include/linux/memcontrol.h duplicates the charging logic with no remaining callers.
Quoted 4 lines
mm/zswap.c +0
-0
[ ... ]
Quoted 6 lines
mm/zswap.c +2
-0@@ -1711,17 +1702,86 @@ static bool zswap_store_pages(struct zswap_store_context *zc,zswap_lru_add (entry); } +/* +* Each batch entry is successfully compressed and stored in the tree,
mm/zswap.c static bool zswap_store_pages(struct zswap_store_context *zc, · line 1716
Quoted 10 lines
mm/zswap.c +10
-0+* and further failures will be cleaned up in zswap_store(). +* Next, we do bulk accounting for the batch: grab refs to the pool and +* objcg, charge zswap memory, increment zswap_stored_pages and +* zswap_stored_incompressible_pages. The opposite actions will be +* performed by zswap_entry_free() when the entry is removed from the +* tree. +*/ +zswap_pool_get_many (pool, zc->nr_pages); +if (objcg) { +obj_cgroup_get_many (objcg, zc->nr_pages);
[Severity: High]
Does calling obj_cgroup_get_many() here break the build when CONFIG_ZSWAP=y and CONFIG_MEMCG=n?
include/linux/memcontrol.h only defines obj_cgroup_get_many() inside the #ifdef CONFIG_MEMCG block, while the #else section only provides stubs for obj_cgroup_get() and obj_cgroup_put().
mm/zswap.c static bool zswap_store_pages(struct zswap_store_context *zc, · line 1689
Quoted 5 lines
mm/zswap.c +5
-0+obj_cgroup_charge_zswap_many (objcg, total_batch_comp_length, + zc->nr_pages, nr_incomp); + } +atomic_long_add (zc->nr_pages, &zswap_stored_pages); +atomic_long_add (nr_incomp, &zswap_stored_incompressible_pages);
[Severity: Low]
Should the incompressible page updates here and in obj_cgroup_charge_zswap_many() be skipped when nr_incomp is 0?
In the common case where all pages in the batch are compressible, atomic_long_add(0, &zswap_stored_incompressible_pages) still performs an atomic read-modify-write on the shared global counter, and mod_memcg_state(memcg, MEMCG_ZSWAP_INCOMP, 0) still runs get_cpu(), updates per-CPU stats by 0, and fires trace_mod_memcg_state() with val == 0.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=7
[email protected]
AI reviewRe: [PATCH v3 8/8] mm: zswap: Batched zswap_compress() for storing large folios in batches.
In reply to Kanchana P. SridharThank you for your contribution! Sashiko AI review found 3 potential issues to consider: - [High] mm: zswap: corrupted and unterminated dst scatterlist in zswap_compress() - [Medium] mm: zswap…
Thank you for your contribution! Sashiko AI review found 3 potential issues to consider:
- [High] mm: zswap: corrupted and unterminated dst scatterlist in zswap_compress()
- [Medium] mm: zswap: unclamped nr_batch_pages on remainder batch in zswap_compress()
- [Medium] mm: zswap: unsigned sg->length check mishandles negative errors in zswap_compress()
--
commit 212f5740068394b4c82b7b1e5452b1ac6625322f
Author: Kanchana P. Sridhar <[email protected]>
mm: zswap: Batched zswap_compress() for storing large folios in batches.
This commit replaces per-page zswap_compress() calls with a batched implementation that compresses up to ZSWAP_MAX_BATCH_SIZE pages using a per-CPU scatterlist table and updates the batch entries in zswap_update_entries().
Quoted 4 lines
mm/zswap.c +0
-0
[ ... ]
Quoted 1 line
mm/zswap.c +0
-0@@ -898,6 +914,7 @@ static int zswap_cpu_comp_prepare(unsigned int cpu, struct hlist_node *node)
[ ... ]
Quoted 19 lines
mm/zswap.c +19
-0+ acomp_ctx->comp.sg_table =kmalloc (sizeof (*acomp_ctx->comp.sg_table), +GFP_KERNEL ); +if (!acomp_ctx->comp.sg_table) { + ret = -ENOMEM ; + goto req_fail; + } + +if (sg_alloc_table (acomp_ctx->comp.sg_table,ZSWAP_MAX_BATCH_SIZE , +GFP_KERNEL )) { + ret = -ENOMEM ; + goto req_fail; + } + +/* +* Statically map the per-CPU destination buffers to the per-CPU +* SG lists; crucial for performance. +*/ +for_each_sg (acomp_ctx->comp.sg_table->sgl, sg,ZSWAP_MAX_BATCH_SIZE , i) +sg_set_buf (sg, acomp_ctx->comp.buffers[i],PAGE_SIZE );
[ ... ]
Quoted 21 lines
mm/zswap.c +8
-11+static bool zswap_compress (struct zswap_store_context *zc) +{ +u8 batch_start, comp_batch_iter; +struct scatterlist *sg, *dst_sg; +int err;/* -* it maybe looks a little bit silly that we send an asynchronous request, -* then wait for its completion synchronously. This makes the process look -* synchronous in fact. -* Theoretically, acomp supports users send multiple acomp requests in one -* acomp instance, then get those requests done simultaneously. but in this -* case, zswap actually does store and load page by page, there is no -* existing method to send the second page before the first page is done -* in one thread doing zswap. -* but in different threads running on different cpu, we have different -* acomp instance, and compression and decompression use separate -* requests, so multiple threads can do (de)compression in parallel. +* @nr_batch_pages is constrained by the per-CPU comp_ctx's compressor +* batch size and the @zc->nr_pages in this store batch. @slen indicates +* the total source length bytes for @nr_batch_pages.
mm/zswap.c static int zswap_cpu_comp_prepare(unsigned int cpu, struct hlist_node *node) · line 1105
Quoted 10 lines
mm/zswap.c +6
-0*/ + zc->nr_batch_pages =min (zc->comp_ctx->compr_batch_size, zc->nr_pages); + zc->slen = zc->nr_batch_pages *PAGE_SIZE ;/* +* Compress the @nr_pages in @folio starting at index @folio_start +* in batches of @nr_batch_pages. */ +for (batch_start =0 ; batch_start < zc->nr_pages; + batch_start += zc->nr_batch_pages) {
[Severity: Medium]
Can zc->nr_batch_pages exceed the remaining pages in the store batch on the final loop iteration?
zc->nr_batch_pages and zc->slen are computed once before the loop instead of being clamped to zc->nr_pages - batch_start on each iteration. If a compressor has a compr_batch_size that does not evenly divide zc->nr_pages (for example, compr_batch_size == 3 with zc->nr_pages == 4), the final iteration reads past the end of the folio batch and walks past the SG_END terminator of zc->comp_ctx->sg_table->sgl in for_each_sg(), causing a NULL pointer dereference when sg_next() returns NULL.
Quoted 15 lines
mm/zswap.c +15
-0+/* +* Send @nr_batch_pages to crypto_acomp for compression: +* +* These pages are in @folio's range of indices in the interval +* [@folio_start + @batch_start, +* @folio_start + @batch_start + @nr_batch_pages). +* +* The source folio pages in the batch are directly submitted to +* crypto_acomp via acomp_request_set_src_folio(). +*/ +acomp_request_set_src_folio (zc->comp_ctx->areq.req, zc->folio, + (zc->folio_start + batch_start) *PAGE_SIZE , + zc->slen); + + dst_sg = zc->comp_ctx->sg_table->sgl + batch_start;
mm/zswap.c static int zswap_cpu_comp_prepare(unsigned int cpu, struct hlist_node *node) · line 1130
Quoted 10 lines
mm/zswap.c +10
-0+ dst_sg->length = zc->slen; + +/* +* The per-CPU @zc->comp_ctx->sg_table scatterlists are statically +* mapped to the per-CPU dst @buffers at pool creation time. +* There are ZSWAP_MAX_BATCH_SIZE per-CPU SG lists and dst +* buffers allocated for all compressors. +*/ +acomp_request_set_dst_sg (zc->comp_ctx->areq.req, + dst_sg, zc->slen);
[Severity: High]
Does setting dst_sg->length = zc->slen corrupt the per-CPU scatterlist table in zc->comp_ctx->sg_table?
In zswap_cpu_comp_prepare(), each entry in sg_table->sgl is mapped to a separate PAGE_SIZE buffer and only sgl[ZSWAP_MAX_BATCH_SIZE - 1] has SG_END set. During compression, each entry's sg->length is overwritten with the compressed size or a negative error code.
When zc->nr_batch_pages > 1, setting dst_sg->length = zc->slen tells the compressor that dst_sg[0]'s single PAGE_SIZE buffer is nr_batch_pages * PAGE_SIZE bytes long, while dst_sg[1 .. nr_batch_pages - 1].length are never reset to PAGE_SIZE and retain stale lengths or negative error codes from the previous batch. If crypto_acomp_compress() fails while zc->wb_enabled is true without updating every entry's sg->length, any entry still holding a stale length in (0, PAGE_SIZE) bypasses the sg->length >= PAGE_SIZE check and causes zswap_update_entries() to store stale data from the previous folio.
In addition, because SG_END is only set on sgl[3], passing dst_sg = sgl + batch_start leaves 4 - batch_start entries linked in req->dst. Hardware acomp drivers that check sg_nents(req->dst) or DMA-map every entry in the scatterlist walk past the current batch into trailing entries whose sg->length may still hold PAGE_SIZE or (unsigned int)-ENOSPC (0xffffffe4) from an earlier failed compression.
Quoted 40 lines
mm/zswap.c +36
-0+ + err =crypto_wait_req (crypto_acomp_compress (zc->comp_ctx->areq.req), + &zc->comp_ctx->areq.wait); + +/* +* If a page cannot be compressed into a size smaller than +* PAGE_SIZE, save the content as is without a compression, to +* keep the LRU order of writebacks. If writeback is disabled, +* reject the page since it only adds metadata overhead. +* swap_writeout() will put the page back to the active LRU list +* in the case. +*/ +if (err && !zc->wb_enabled) + goto compress_error; +/* +* For each output SG list in @acomp_ctx->req->sg_table->sgl, +* the @sg->length should be set to either the page's compressed +* length (success), or it's negative value compression error. +*/ +for_each_sg (dst_sg, sg, zc->nr_batch_pages, comp_batch_iter) { + +if (sg->length <=0 || sg->length >=PAGE_SIZE ) { +if (!zc->wb_enabled) { +if (sg->length ==PAGE_SIZE ) + sg->length = -ENOSPC ; +else if (WARN_ON_ONCE (!sg->length || sg->length >PAGE_SIZE )) + sg->length = -EINVAL ; + goto compress_error; + } + + sg->length =PAGE_SIZE ; + } + } } +return zswap_update_entries (zc); +compress_error: +for_each_sg (dst_sg, sg, zc->nr_batch_pages, comp_batch_iter) {
mm/zswap.c static int zswap_cpu_comp_prepare(unsigned int cpu, struct hlist_node *node) · line 1180
Quoted 10 lines
mm/zswap.c +9
-0+if ((int )sg->length <0 ) { +if ((int )sg->length == -ENOSPC ) + zswap_reject_compress_poor++; +else + zswap_reject_compress_fail++; + } + } + +return false; }
[Severity: Medium]
Does checking sg->length without casting to int mishandle negative compression error codes when !zc->wb_enabled?
scatterlist.length is an unsigned int, so a negative error code stored in sg->length (such as -ENOSPC) is never <= 0 and instead triggers WARN_ON_ONCE(!sg->length || sg->length > PAGE_SIZE) and overwrites sg->length with -EINVAL before compress_error: checks for -ENOSPC. Conversely, if crypto_acomp_compress() returns an error before setting negative values in sg->length, jumping directly to compress_error: finds no entries with (int)sg->length < 0 and increments neither rejection counter.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=8
Yosry Ahmed
maintainerQuoted 26 lines
Quoted 20 lines
As with previous versions, I cannot reproduce these results. I tried with a QEMU VM on a Cascade Lake host. I saw marginal improvements with patch 6 and no significant gain with patches 7-8.
I noticed that you omitted high-level metrics (e.g. throughput), I didn't see significant changes in throughput, and I also ran a kernel build and saw a small regression in some cases (probably with order-0 folios) and no significant improvement otherwise. Even if latency for some functions does improve slightly, it doesn't matter if it doesn't show up on higher-level metrics.
It could be the difference in setups, or the overhead of bpftrace hooks for
zswap_compress()might affect measuredzswap_store()latencies, as you will have fewerzswap_compress()calls with patch 8.As advised previously, I would trim down patch 6 to the bare minimum (without all the scaffolding and prep work for patches 7-8), and send it separately (with patch 5), and we can check if it's worth adding, I am still concerned it might be adding overhead for order-0 to be honest.
Quoted 5 lines
I am questioning the benefit of batching here without HW batching, so far the results are more concerning than encouraging. I already spent too much time trying to reproduce the performance gains. Also, sending new versions without resolving discussions on previous ones is not really productive.
Quoted 5 lines
Thanks for taking the time to test and provide feedback, Yosry.
I omitted high-level metrics like throughput since you raised a concern about variation. Hence, I decided to go with the "ultimate truth" which is bpftrace latency measurements.
I didn't mean that we shouldn't look at high-level metrics, it's just that the test setup had too much variance for the high-level metrics to be useful so I checked the low-level latency. Ideally we'd test performance with benchmarks that should have stable high-level metrics.
Quoted 32 lines
For performance validation, I only have access to a 16G Chromebook with a Crostini VM to which 13G is allocated. I am setting aside 3G for swap and 7G for QEMU. Within the QEMU VM, I am selecting 2 E-cores to run the 2 usemem processes to minimize variation due to frequency fluctuations. This setup has its own limitations such as being unable to fix the frequency.
Despite all this, I have been able to consistently replicate the bpftrace latency improvements with patch 8. My results show regression with patch 6-7, and improvement only with patch 8. As noted in the cover-letter, patch 8 adds significant latency improvement for order-0 folios. This is with ZSTD. FWIW, these are bpftrace latency improvements in a significantly memory constrained environment and this ought to be a useful data point. I have been posting general vmstat "system health" metrics to prove that nothing unexpected is going on.
My data shows that order-0
zswap_store()tail latency improves by 11% over mm-unstable with patch 8, although p50 is roughly similar between patches 6-7 and 8. PMD store latency is significantly better at p50 and p99 with patch 8.Likewise, p99 latency of
zswap_compress()alone improves by 3% with patch 8, while it degrades with patch 6-7. p50 is almost the same with patch 8, although patch 6-7 regresses. The page-normalizedbatch_compress()p99 latency is better by 15%. All comparisons are wrt mm-unstable.Hence, I believe even in a memory-constrained setup such as mine, batching significantly (7-15%) improves p99 in all cases, with 4-5% gains in p50: for order-0 and PMD folios.
Quoted 12 lines
In addition to the latency benefits shown through my validation, I would like to also emphasize the strategic benefit of moving zswap to use batching, as noted under "Conclusions" in my cover letter. IOW, batching within large folios to improve workload metrics can only do so much (in proportion to the # of large folios reclaimed and contribution of
zswap_store()to the overall allocation-reclaim path). What this patch series attempts to provide is a zswap batching framework upon which additional improvements can be built, such as reclaiming multiple folios as a batch. I believe that compress batching is essential to support these next steps, which improve workload throughput/latency metrics even more than simply batching within folios. This is with ZSTD.
Without a direct line of sight for such features and consensus from the community, landing the framework in zswap is not very useful, it actually increases maintenance burden.
Quoted 6 lines
Due to the environmental constraints, I haven't been able to reliably use kernel compilation for validation.
I have been doing my best to resolve previous discussions by incorporating the
calculate_high_delay()hack that you suggested to reduce variation. The data posted in v3 comprehends this suggestion.
Oh I wasn't suggesting you generally test with this hack, I just noticed that the variance was resulting from memory.high throttling and nullified it to see what the numbers look like without it. As I mentioned above, ideally we see stable improvement in high-level metrics. usemem is already a microbenchmark, if we can't demonstrate stable improvements in its metrics and have to measure latency of specific functions, the significance of the optimizations is questionable.
It's a tradeoff between maintainability and performance. It doesn't really make sense to add complexity for marginal performance benefits on a small subset of setups.
Quoted 8 lines
I would like to suggest/request the following, as next steps:
1) Can we reconcile our methodologies for usemem bpftrace
measurements: # of vCPU, usemem command line, reading Silesia, etc. Also swap space and QEMU memory allocation, using linearlhists()in bpftrace, etc. If you can share these details, it would be very helpful. Thanks again for your time!
I ran a VM with 2 vCPUs, 8G of memory, and a 25G swapfile (although the size of the swapfile doesn't matter much).
I used zstd with the default pool limit (20%) and the shrinker off, with 2M THP enabled. I set memory.high to 3G for usemem like you.
I used this usemem command with the diff you posted in the last version:
$ usemem -w --read-silesia -O -b 1 -s 10 -n 2 2400M
For kernel build, I used:
$ kcbench --llvm -s 6.8 --nodownload -j 8 -i 1 -o <PATH>--detailed-results
I don't have the bpftrace commands but it should be pretty standard (I think I used hist() not lhist() tho).
All that being said, I don't think I have more time to spend on this tbh.
Quoted 3 lines
2) Does the kernel community offer servers for contributors to test their optimizations at scale?
I don't believe so, but I am not sure.
Quoted 4 lines
3) Can I please request the help of other zswap maintainers/reviewers to validate this series with ZSTD on a server setup with usemem running on 30+ cores, under high over-commit ratios; through bpftrace and/or throughput metrics.
I can't ask people to spend time testing this, I already spent more time than I have and I am sure other people are also busy. If someone is interested in testing this and collecting performance numbers, that would be up to them.