crypto: zstd - avoid initializing the workspace twice
Subsystems (2)
cover crypto: zstd - avoid initializing the workspace twice
Both zstd_compress() and zstd_decompress() set up the shared per-CPU workspace as a C/DStream before walking the request, and then, when the first source and destination fragments each span the whole request, hand off to zstd_compress_one()/zstd_decompress_one(), which immediately overwrite that same ctx->wksp with a CCtx/DCtx. The stream setup is discarded without a byte having been processed.
That one-shot path is not a corner case: zswap always takes it when storing, and takes it for a load whenever the stored object lies within a single zsmalloc page.
These two patches defer the stream initialization to the first walk iteration that actually streams, guarded by a flag because that iteration can be reached more than once.
A 4 KiB crypto_acomp benchmark [1], twelve runs of nine 30,000-operation rounds. Bare metal is an Intel Xeon Platinum 8321HC, turbo off, performance governor, pinned to one core; the VM is a one-vCPU KVM guest on a faster host.
baseline patched delta
bare metal
compress 52,283 ns 51,038 ns 1,245 ns 2.4%
decompress 2,317 ns 1,998 ns 319 ns 13.8%
one-vCPU KVM
compress 16,675 ns 15,050 ns 1,625 ns 9.8%
decompress 3,516 ns 2,265 ns 1,251 ns 35.6%The guest numbers are larger because the two CPUID instructions in ZSTD_cpuid() become unconditional VM exits there.
[1] https://gist.github.com/uarif1/5cf02f0e22c23f0d1b3d84348f12914c
Usama Arif (2):
crypto: zstd - Avoid redundant cstream initialization
crypto: zstd - Avoid redundant dstream initialization
crypto/zstd.c | 40 ++++++++++++++++++++++++++++------------ 1 file changed, 28 insertions(+), 12 deletions(-)
base-commit: 4b18edbd8e70f7e6860d56370f13244896d0f95c
Signature · 1 line
2.53.0-Meta
PATCH 1/2 crypto: zstd - Avoid redundant cstream initialization
zstd_compress() initializes the shared workspace as a CStream before entering the walk loop. If the first source and destination fragments each span the whole request it then hands off to zstd_compress_one(), which initializes that same ctx->wksp as a CCtx, discarding the CStream setup without having compressed a byte. zswap always takes this one-shot path when storing, so every page it stores paid for both.
Neither is cheap: zstd_init_cstream() redoes the cwksp layout, zeroes the ZSTD_CCtx, probes for BMI2 through ZSTD_cpuid(), then resets the session and parameters and replays ten validated ZSTD_CCtx_setParameter() calls.
Defer the CStream initialization to the first walk iteration that needs it, guarded by a flag because that iteration can be reached more than once. The first inner iteration either takes the one-shot path and returns or initializes the CStream, so the trailing zstd_end_stream() cannot pick up the stale context left in ctx->cctx by an earlier request. Unlike the old call site the new one runs with the walk's fragments mapped, so it has to release them before failing.
For a 4 KB crypto_acomp benchmark for compression, twelve runs of nine 30K operation rounds, on the bare-metal host the median per-round mean request time fell from 52,283 ns to 51,038 ns (2.4%). In the one-vCPU KVM guest it fell from 16,675 ns to 15,050 ns (9.8%).
The larger improvement in guest is because of the pair of CPUID instructions in ZSTD_cpuid() that the removed initialization runs.
--- crypto/zstd.c | 20 ++++++++++++++------ 1 file changed, 14 insertions(+), 6 deletions(-)
crypto/zstd.c +14 -6
@@ -96,6 +96,7 @@ static int zstd_compress_one(struct acomp_req *req, struct zstd_ctx *ctx,static int zstd_compress (struct acomp_req *req) { +bool stream_initialized = false;struct crypto_acomp_stream *s;unsigned int pos, scur, dcur;unsigned int total_out =0 ;
@@ -115,12 +116,6 @@ static int zstd_compress(struct acomp_req *req)if (ret) goto out; - ctx->cctx =zstd_init_cstream (&ctx->params,0 , ctx->wksp, ctx->wksp_size); -if (!ctx->cctx) { - ret = -EINVAL ; - goto out; - } - do { dcur = acomp_walk_next_dst(&walk); if (!dcur) {
@@ -142,6 +137,19 @@ static int zstd_compress(struct acomp_req *req) goto out; } +if (!stream_initialized) { + ctx->cctx =zstd_init_cstream (&ctx->params,0 , + ctx->wksp, ctx->wksp_size); +if (!ctx->cctx) { +/* Release in the reverse of the map order. */ +acomp_walk_done_src (&walk,0 ); +acomp_walk_done_dst (&walk,0 ); + ret = -EINVAL ; + goto out; + } + stream_initialized = true; + } +if (scur) { inbuf.pos =0 ; inbuf.src = walk.src.virt.addr;
Signature · 1 line
2.53.0-Meta
Re: [PATCH 1/2] crypto: zstd - Avoid redundant cstream initialization
Quoted 19 lines · Usama Arif
zstd_compress()initializes the shared workspace as a CStream before entering the walk loop. If the first source and destination fragments each span the whole request it then hands off tozstd_compress_one(), which initializes that samectx->wkspas a CCtx, discarding the CStream setup without having compressed a byte. zswap always takes this one-shot path when storing, so every page it stores paid for both.Neither is cheap:
zstd_init_cstream()redoes the cwksp layout, zeroes theZSTD_CCtx, probes for BMI2 throughZSTD_cpuid(), then resets the session and parameters and replays ten validatedZSTD_CCtx_setParameter()calls.Defer the CStream initialization to the first walk iteration that needs it, guarded by a flag because that iteration can be reached more than once. The first inner iteration either takes the one-shot path and returns or initializes the CStream, so the trailing
zstd_end_stream()cannot pick up the stale context left inctx->cctxby an earlier request. Unlike the old call site the new one runs with the walk's fragments mapped, so it has to release them before failing.
Quoted 8 lines
For a 4 KB
crypto_acompbenchmark for compression, twelve runs of nine 30K operation rounds, on the bare-metal host the median per-round mean request time fell from 52,283 ns to 51,038 ns (2.4%). In the one-vCPU KVM guest it fell from 16,675 ns to 15,050 ns (9.8%).
The larger improvement in guest is because of the pair of CPUID instructions inZSTD_cpuid()that the removed initialization runs.Signed-off-by Usama Arif <[email protected]>
This optimization is a clear win, and makes sense to apply even after the BMI2 patches in zstd, as it also removes other initialization work such as zeroing buffers. Thanks for finding & fixing this!
Quoted 32 lines
--- crypto/zstd.c | 20 ++++++++++++++------ 1 file changed, 14 insertions(+), 6 deletions(-)crypto/zstd.c +1
-6@@ -96,6 +96,7 @@ static int zstd_compress_one(struct acomp_req *req, struct zstd_ctx *ctx,static int zstd_compress (struct acomp_req *req) { +bool stream_initialized = false;struct crypto_acomp_stream *s;unsigned int pos, scur, dcur;unsigned int total_out =0 ;@@ -115,12 +116,6 @@ static int zstd_compress(struct acomp_req *req)if (ret) goto out; - ctx->cctx =zstd_init_cstream (&ctx->params,0 , ctx->wksp, ctx->wksp_size); -if (!ctx->cctx) { - ret = -EINVAL ; - goto out; - } - do { dcur = acomp_walk_next_dst(&walk); if (!dcur) {@@ -142,6 +137,19 @@ static int zstd_compress(struct acomp_req *req) goto out; }
crypto/zstd.c static int zstd_compress(struct acomp_req *req) · line 148
Quoted 10 lines
crypto/zstd.c +9
-0+if (!stream_initialized) { + ctx->cctx =zstd_init_cstream (&ctx->params,0 , + ctx->wksp, ctx->wksp_size); +if (!ctx->cctx) { +/* Release in the reverse of the map order. */ +acomp_walk_done_src (&walk,0 ); +acomp_walk_done_dst (&walk,0 ); + ret = -EINVAL ; + goto out;
The other error paths, e.g. when ZSTD_isError() is true below, don't call acomp_walk_done_src() or acomp_walk_done_dst(). The difference is that they are returning -EIO rather than -EINVAL.
I don't know enough about the acomp API to know which behavior is correct, but want to flag this as it seems like one of them may be incorrect.
The same question applies to the next patch for decompression.
Quoted 10 lines
crypto/zstd.c +4
-1+ } + stream_initialized = true; + } +if (scur) { inbuf.pos =0 ; inbuf.src = walk.src.virt.addr; --2.53.0-Meta
PATCH 2/2 crypto: zstd - Avoid redundant dstream initialization
In reply to Usama Arifzstd_decompress() initializes the shared workspace as a DStream before entering the walk loop. If the first source and destination fragments each span the whole request it then hands off to zstd_decompress_one(), which initializes that same ctx->wksp as a DCtx, discarding the DStream setup without having decompressed a byte. The two are in fact the same routine, as ZSTD_initStaticDStream() is a tail call to ZSTD_initStaticDCtx() and zstd_init_dstream() discards its max_window_size argument, so the work was done twice byte for byte.
zswap takes this path whenever the stored object lies within a single zsmalloc page, which is the common case; an object straddling a page boundary comes back from zs_obj_read_sg_begin() as a two-entry source scatterlist and streams instead.
Defer the DStream initialization to the first walk iteration that reaches the streaming path, guarded by a flag because that iteration can be reached more than once. Within this function ctx->dctx is read only by the zstd_decompress_stream() call immediately below, so no stale context can be picked up. As in the previous patch the new call site runs with the walk's fragments mapped and has to release them before failing.
For a 4 KB crypto_acomp benchmark for decompression, twelve runs of nine 30K operation rounds, on the bare-metal host the median per-round mean request time fell from 2,317 ns to 1,998 ns (13.8%).
In the one-vCPU KVM guest it fell from 3,516 ns to 2,265 ns (35.6%).
--- crypto/zstd.c | 20 ++++++++++++++------ 1 file changed, 14 insertions(+), 6 deletions(-)
crypto/zstd.c +14 -6
@@ -215,6 +215,7 @@ static int zstd_decompress_one(struct acomp_req *req, struct zstd_ctx *ctx,static int zstd_decompress (struct acomp_req *req) { +bool stream_initialized = false;struct crypto_acomp_stream *s;unsigned int total_out =0 ;unsigned int scur, dcur;
@@ -232,12 +233,6 @@ static int zstd_decompress(struct acomp_req *req)if (ret) goto out; - ctx->dctx =zstd_init_dstream (ZSTD_MAX_SIZE , ctx->wksp, ctx->wksp_size); -if (!ctx->dctx) { - ret = -EINVAL ; - goto out; - } - do { scur = acomp_walk_next_src(&walk); if (scur) {
@@ -263,6 +258,19 @@ static int zstd_decompress(struct acomp_req *req) goto out; } +if (!stream_initialized) { + ctx->dctx =zstd_init_dstream (ZSTD_MAX_SIZE , ctx->wksp, + ctx->wksp_size); +if (!ctx->dctx) { +/* Release in the reverse of the map order. */ +acomp_walk_done_dst (&walk,0 ); +acomp_walk_done_src (&walk,0 ); + ret = -EINVAL ; + goto out; + } + stream_initialized = true; + } + outbuf.pos =0 ; outbuf.dst = (u8 *)walk.dst.virt.addr; outbuf.size = dcur;
Signature · 1 line
2.53.0-Meta
Quoted 21 lines · Usama Arif
Both
zstd_compress()andzstd_decompress()set up the shared per-CPU workspace as a C/DStream before walking the request, and then, when the first source and destination fragments each span the whole request, hand off to zstd_compress_one()/zstd_decompress_one(), which immediately overwrite that samectx->wkspwith a CCtx/DCtx. The stream setup is discarded without a byte having been processed.That one-shot path is not a corner case: zswap always takes it when storing, and takes it for a load whenever the stored object lies within a single zsmalloc page.
These two patches defer the stream initialization to the first walk iteration that actually streams, guarded by a flag because that iteration can be reached more than once.
A 4 KiB
crypto_acompbenchmark [1], twelve runs of nine 30,000-operation rounds. Bare metal is an Intel Xeon Platinum 8321HC, turbo off, performance governor, pinned to one core; the VM is a one-vCPU KVM guest on a faster host.
Quoted 7 lines
baseline patched delta
bare metalcompress 52,283 ns 51,038 ns 1,245 ns 2.4% decompress 2,317 ns 1,998 ns 319 ns 13.8% one-vCPU KVM compress 16,675 ns 15,050 ns 1,625 ns 9.8% decompress 3,516 ns 2,265 ns 1,251 ns 35.6%
Nice!
Quoted 3 lines
The guest numbers are larger because the two CPUID instructions in
ZSTD_cpuid()become unconditional VM exits there.
I wonder how much benefit we get from just doing CPUID once in ZSTD_cpuid() and cache the results. There might still be value in avoiding the overall initialization, but I think ZSTD_cpuid() should only be executing CPUID once anyway (e.g. in case it's called in other paths)?
Quoted 15 lines
[1] https://gist.github.com/uarif1/5cf02f0e22c23f0d1b3d84348f12914c
Usama Arif (2):
crypto: zstd - Avoid redundant cstream initialization
crypto: zstd - Avoid redundant dstream initializationcrypto/zstd.c | 40 ++++++++++++++++++++++++++++------------ 1 file changed, 28 insertions(+), 12 deletions(-)base-commit:
4b18edbd8e70f7e6860d56370f13244896d0f95c--
2.53.0-Meta
Quoted 33 lines · Yosry Ahmed
Quoted 21 lines · Usama Arif
Both
zstd_compress()andzstd_decompress()set up the shared per-CPU workspace as a C/DStream before walking the request, and then, when the first source and destination fragments each span the whole request, hand off to zstd_compress_one()/zstd_decompress_one(), which immediately overwrite that samectx->wkspwith a CCtx/DCtx. The stream setup is discarded without a byte having been processed.That one-shot path is not a corner case: zswap always takes it when storing, and takes it for a load whenever the stored object lies within a single zsmalloc page.
These two patches defer the stream initialization to the first walk iteration that actually streams, guarded by a flag because that iteration can be reached more than once.
A 4 KiB
crypto_acompbenchmark [1], twelve runs of nine 30,000-operation rounds. Bare metal is an Intel Xeon Platinum 8321HC, turbo off, performance governor, pinned to one core; the VM is a one-vCPU KVM guest on a faster host.Quoted 7 lines
baseline patched delta
bare metalcompress 52,283 ns 51,038 ns 1,245 ns 2.4% decompress 2,317 ns 1,998 ns 319 ns 13.8% one-vCPU KVM compress 16,675 ns 15,050 ns 1,625 ns 9.8% decompress 3,516 ns 2,265 ns 1,251 ns 35.6%Nice!
Quoted 1 line
Quoted 9 lines
Quoted 2 lines
The guest numbers are larger because the two CPUID instructions in
ZSTD_cpuid()become unconditional VM exits there.I wonder how much benefit we get from just doing CPUID once in
ZSTD_cpuid()and cache the results. There might still be value in avoiding the overall initialization, but I thinkZSTD_cpuid()should only be executing CPUID once anyway (e.g. in case it's called in other paths)?
So I tried something like below and the numbers look really good on a VM:
compression decompression Before 14523ns 2250ns After 13554ns (-6.7%) 973ns (-56.8%)
commit c169757935b9c4f4d0765227cdf989515f1987a2 (HEAD -> zstd_cpuid_cache)
Author: Usama Arif <[email protected]> Date: Wed Aug 26 04:03:35 2026 -0700
zstd: probe the CPU for BMI2 support only once
ZSTD_cpuSupportsBmi2() issues CPUID on every context setup for an answer that cannot change while the kernel is running. On x86 that is two serializing CPUID instructions, and the callers are not rare:
squashfs, erofs, btrfs, f2fs and crypto/zstd all initialise a context per operation, so a busy squashfs or zswap workload pays for it per block or per page. Under KVM it is worse, because CPUID is an unconditional VM exit.
Cache the result. Keeping the cache as a single int with a negative sentinel, rather than a copy of ZSTD_cpuid_t, keeps it to one word: a racing pair of probes computes the same value from the same CPUID leaf, so the unsynchronized access is benign, and READ_ONCE()/WRITE_ONCE() keep the compiler and KCSAN in agreement about that.
ZSTD_cpuSupportsBmi2() is MEM_STATIC, so each translation unit that inlines it gets its own cache - three in a modular build, plus one in each preboot decompressor. That is a handful of ints in bss and one extra probe apiece, not worth avoiding.
The cached answer is the one the probing CPU reported. zstd could already be migrated between the probe and the use of the flag, so this does not introduce a heterogeneity question that was not there before.
lib/zstd/common/zstd_internal.h +15 -2
@@ -312,8 +312,21 @@ size_t ZSTD_decodeSeqHeaders(ZSTD_DCtx* dctx, int* nbSeqPtr, MEM_STATIC int ZSTD_cpuSupportsBmi2(void) {#if DYNAMIC_BMI2 -ZSTD_cpuid_t cpuid =ZSTD_cpuid (); -return ZSTD_cpuid_bmi1 (cpuid) &&ZSTD_cpuid_bmi2 (cpuid); +/* +* The answer cannot change over the life of the kernel, so probe +* once. Racing probes compute the same value, so the unsynchronized +* access is benign; the annotations are there to keep it that way. +*/ +static int supported =-1 ; +int s =READ_ONCE (supported); + +if (s <0 ) { +ZSTD_cpuid_t const cpuid =ZSTD_cpuid (); + + s =ZSTD_cpuid_bmi1 (cpuid) &&ZSTD_cpuid_bmi2 (cpuid); +WRITE_ONCE (supported, s); + } +return s;#else /* Nothing looks at the flag in this configuration. */ return 0 ;
Quoted 15 lines
Quoted 15 lines
[1] https://gist.github.com/uarif1/5cf02f0e22c23f0d1b3d84348f12914c
Usama Arif (2):
crypto: zstd - Avoid redundant cstream initialization
crypto: zstd - Avoid redundant dstream initializationcrypto/zstd.c | 40 ++++++++++++++++++++++++++++------------ 1 file changed, 28 insertions(+), 12 deletions(-)base-commit:
4b18edbd8e70f7e6860d56370f13244896d0f95c--
2.53.0-Meta
Quoted 28 lines · On Tue, 25 Aug 2026 15:06:00 -0700 Usama Arif <[email protected]> wrote:
Both
zstd_compress()andzstd_decompress()set up the shared per-CPU workspace as a C/DStream before walking the request, and then, when the first source and destination fragments each span the whole request, hand off to zstd_compress_one()/zstd_decompress_one(), which immediately overwrite that samectx->wkspwith a CCtx/DCtx. The stream setup is discarded without a byte having been processed.That one-shot path is not a corner case: zswap always takes it when storing, and takes it for a load whenever the stored object lies within a single zsmalloc page.
These two patches defer the stream initialization to the first walk iteration that actually streams, guarded by a flag because that iteration can be reached more than once.
A 4 KiB
crypto_acompbenchmark [1], twelve runs of nine 30,000-operation rounds. Bare metal is an Intel Xeon Platinum 8321HC, turbo off, performance governor, pinned to one core; the VM is a one-vCPU KVM guest on a faster host.baseline patched delta
bare metalcompress 52,283 ns 51,038 ns 1,245 ns 2.4% decompress 2,317 ns 1,998 ns 319 ns 13.8% one-vCPU KVM compress 16,675 ns 15,050 ns 1,625 ns 9.8% decompress 3,516 ns 2,265 ns 1,251 ns 35.6%
Quoted 4 lines
The guest numbers are larger because the two CPUID instructions in
ZSTD_cpuid()become unconditional VM exits there.[1] https://gist.github.com/uarif1/5cf02f0e22c23f0d1b3d84348f12914c
Hi,
Just wanted to check if there was any feedback or review of the series.
I think its a nice optimization and even with the CPUID instructions getting cached [1], this is still needed. The improvement in baremetal is not coming (just) from CPUID instructions.
[1] https://lore.kernel.org/all/[email protected]/
Thanks,
Usama
Quoted 35 lines · Usama Arif
Quoted 28 lines · On Tue, 25 Aug 2026 15:06:00 -0700 Usama Arif <[email protected]> wrote:
Both
zstd_compress()andzstd_decompress()set up the shared per-CPU workspace as a C/DStream before walking the request, and then, when the first source and destination fragments each span the whole request, hand off to zstd_compress_one()/zstd_decompress_one(), which immediately overwrite that samectx->wkspwith a CCtx/DCtx. The stream setup is discarded without a byte having been processed.That one-shot path is not a corner case: zswap always takes it when storing, and takes it for a load whenever the stored object lies within a single zsmalloc page.
These two patches defer the stream initialization to the first walk iteration that actually streams, guarded by a flag because that iteration can be reached more than once.
A 4 KiB
crypto_acompbenchmark [1], twelve runs of nine 30,000-operation rounds. Bare metal is an Intel Xeon Platinum 8321HC, turbo off, performance governor, pinned to one core; the VM is a one-vCPU KVM guest on a faster host.baseline patched delta
bare metalcompress 52,283 ns 51,038 ns 1,245 ns 2.4% decompress 2,317 ns 1,998 ns 319 ns 13.8% one-vCPU KVM compress 16,675 ns 15,050 ns 1,625 ns 9.8% decompress 3,516 ns 2,265 ns 1,251 ns 35.6%Quoted 4 lines
The guest numbers are larger because the two CPUID instructions in
ZSTD_cpuid()become unconditional VM exits there.[1] https://gist.github.com/uarif1/5cf02f0e22c23f0d1b3d84348f12914c
Quoted 7 lines
Hi,
Just wanted to check if there was any feedback or review of the series.
I think its a nice optimization and even with the CPUID instructions getting cached [1], this is still needed. The improvement in baremetal is not coming (just) from CPUID instructions.
Agreed, this optimization makes sense to me as well.
Quoted 4 lines
[1] https://lore.kernel.org/all/[email protected]/
Thanks,
Usama
Herbert Xu
maintainerQuoted 37 lines · Usama Arif
Both
zstd_compress()andzstd_decompress()set up the shared per-CPU workspace as a C/DStream before walking the request, and then, when the first source and destination fragments each span the whole request, hand off to zstd_compress_one()/zstd_decompress_one(), which immediately overwrite that samectx->wkspwith a CCtx/DCtx. The stream setup is discarded without a byte having been processed.That one-shot path is not a corner case: zswap always takes it when storing, and takes it for a load whenever the stored object lies within a single zsmalloc page.
These two patches defer the stream initialization to the first walk iteration that actually streams, guarded by a flag because that iteration can be reached more than once.
A 4 KiB
crypto_acompbenchmark [1], twelve runs of nine 30,000-operation rounds. Bare metal is an Intel Xeon Platinum 8321HC, turbo off, performance governor, pinned to one core; the VM is a one-vCPU KVM guest on a faster host.baseline patched delta
bare metalcompress 52,283 ns 51,038 ns 1,245 ns 2.4% decompress 2,317 ns 1,998 ns 319 ns 13.8% one-vCPU KVM compress 16,675 ns 15,050 ns 1,625 ns 9.8% decompress 3,516 ns 2,265 ns 1,251 ns 35.6%The guest numbers are larger because the two CPUID instructions in
ZSTD_cpuid()become unconditional VM exits there.[1] https://gist.github.com/uarif1/5cf02f0e22c23f0d1b3d84348f12914c
Usama Arif (2):
crypto: zstd - Avoid redundant cstream initialization
crypto: zstd - Avoid redundant dstream initialization
Quoted 7 lines
crypto/zstd.c | 40 ++++++++++++++++++++++++++++------------ 1 file changed, 28 insertions(+), 12 deletions(-)base-commit:
4b18edbd8e70f7e6860d56370f13244896d0f95cSignature · 1 line
2.53.0-Meta
All applied. Thanks.
Signature · 3 lines
Email: Herbert Xu <[email protected]> Home Page: http://gondor.apana.org.au/~herbert/ PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt