crypto: use vmalloc() for large stream workspaces
Subsystems (3)
cover crypto: use vmalloc() for large stream workspaces
The zstd and deflate acomp implementations retain per-CPU stream workspaces until the algorithms are unregistered. On x86-64 with 4 KiB pages, kvmalloc() can back them with order-9 and order-6 kmalloc() blocks, respectively.
Use vmalloc() and vfree() instead. For a server with 384 CPUs, if we have streams on all of the CPUs and kvmalloc() takes the kmalloc() path for every stream, their workspaces use 193.5 MiB more for zstd and 42 MiB for deflate.
In five before/after boot pairs on a 96-vCPU x86-64 VM with 8 GiB RAM, the test populated both algorithms crypto workspaces on every CPU. All 192 baseline workspaces used kmalloc() without these patches. The patches reduced workspace backing by 48.375 MiB for zstd and 10.5 MiB for deflate —58.875 MiB combined—in each pair.
In five before/after runs in a 4-vCPU x86-64 VM, median compression and decompression throughput changed by less than 1% (-0.93% to +0.64%). Inputs were 4 KiB and 64 KiB synthetic buffers containing pseudorandom data or a 50/50 mix of repeating-pattern and pseudorandom bytes. The differences were within the observed run-to-run variation.
Usama Arif (2):
crypto: zstd - use vmalloc() for stream workspaces
crypto: deflate - use vmalloc() for stream workspaces
crypto/deflate.c | 5 +++-- crypto/zstd.c | 4 ++-- 2 files changed, 5 insertions(+), 4 deletions(-)
Signature · 1 line
2.53.0-Meta
PATCH 1/2 crypto: zstd - use vmalloc() for stream workspaces
The zstd acomp streams retain their workspaces after the last transform has been freed. The first is allocated during transform initialization; additional per-CPU streams are allocated on demand. They are released when the algorithm is unregistered.
In a 4 KiB-page x86-64 build, each stream requests 1,566,281 bytes. If kvmalloc() succeeds via kmalloc(), it occupies an order-9 block (2 MiB). vmalloc() needs 383 backing pages (1,532 KiB), 516 KiB fewer per stream. Across 384 populated streams, that is 193.5 MiB less workspace backing if all would have used kmalloc(). There is no such gain when kvmalloc() already falls back to vmalloc().
Use vmalloc() and vfree() directly to avoid the high-order allocation. This uses vmalloc address space, page tables and bookkeeping; the figures above compare workspace backing only.
--- crypto/zstd.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
crypto/zstd.c +2 -2
@@ -44,7 +44,7 @@ static void *zstd_alloc_stream(void)if (!wksp_size)return ERR_PTR (-EINVAL ); - ctx =kvmalloc_flex (*ctx, wksp, wksp_size); + ctx =vmalloc (struct_size (ctx, wksp, wksp_size));if (!ctx)return ERR_PTR (-ENOMEM );
@@ -56,7 +56,7 @@ static void *zstd_alloc_stream(void)static void zstd_free_stream (void *ctx) { -kvfree (ctx); +vfree (ctx); }static struct crypto_acomp_streams zstd_streams = {
Signature · 1 line
2.53.0-Meta
Re: [PATCH 1/2] crypto: zstd - use vmalloc() for stream workspaces
Quoted 7 lines · Usama Arif
The zstd acomp streams retain their workspaces after the last transform
has been freed. The first is allocated during transform initialization;
additional per-CPU streams are allocated on demand. They are released when the algorithm is unregistered.
Quoted 5 lines
In a 4 KiB-page x86-64 build, each stream requests 1,566,281 bytes. If
kvmalloc()succeeds viakmalloc(), it occupies an order-9 block (2 MiB).vmalloc()needs 383 backing pages (1,532 KiB), 516 KiB fewer per stream. Across 384 populated streams, that is 193.5 MiB less
Woah. Nice catch
Quoted 2 lines
Signed-off-by Usama Arif <[email protected]>
Signature · 1 line
All Rights Reversed.
Re: [PATCH 1/2] crypto: zstd - use vmalloc() for stream workspaces
In reply to Usama ArifQuoted 12 lines · Usama Arif
The zstd acomp streams retain their workspaces after the last transform has been freed. The first is allocated during transform initialization; additional per-CPU streams are allocated on demand. They are released when the algorithm is unregistered.
In a 4 KiB-page x86-64 build, each stream requests 1,566,281 bytes. If
kvmalloc()succeeds viakmalloc(), it occupies an order-9 block (2 MiB).vmalloc()needs 383 backing pages (1,532 KiB), 516 KiB fewer per stream. Across 384 populated streams, that is 193.5 MiB less workspace backing if all would have usedkmalloc(). There is no such gain whenkvmalloc()already falls back tovmalloc().
Quoted 5 lines
Use
vmalloc()andvfree()directly to avoid the high-order allocation. This uses vmalloc address space, page tables and bookkeeping; the figures above compare workspace backing only.Signed-off-by Usama Arif <[email protected]>
Nice catch. Given the lifetime of these streams, there really isn't enough turnover to justify the kmalloc optimization to begin with.
Re: [PATCH 1/2] crypto: zstd - use vmalloc() for stream workspaces
In reply to Usama ArifHi Usama,
Quoted 1 line · Usama Arif
Use
vmalloc()andvfree()directly to avoid the high-order allocation.
vmalloc() doesn't avoid high-order allocations any more. Since a06157804399 ("mm/vmalloc: request large order pages from buddy allocator"), vm_area_alloc_pages() first tries the largest order that fits, and splits the block up afterwards. So for these 383 pages it still asks for an order-8 block first, then order 6, 5, 4, 3, 2 and 1.
What the patch does avoid is kmalloc() rounding the 1.5 MiB up to a 2 MiB block, which is where the 516 KiB per stream comes from. Maybe say that instead?
Once that's fixed, please feel free to add:
Chris
PATCH 2/2 crypto: deflate - use vmalloc() for stream workspaces
In reply to Usama ArifThe deflate acomp streams outlive the transforms that use them. One stream is allocated during transform initialization; additional per-CPU streams are allocated on demand. They are freed only when the algorithm is unregistered.
In a 4 KiB-page x86-64 build, each stream requests 145,320 bytes. If kvmalloc() succeeds via kmalloc(), it occupies an order-6 block (256 KiB). vmalloc() needs 36 backing pages (144 KiB), 112 KiB fewer per stream. Across 384 populated streams, that is 42 MiB less workspace backing if all would have used kmalloc(). There is no such gain when kvmalloc() already falls back to vmalloc().
Use vmalloc() and vfree() directly, as the crypto lz4 implementations already do. This uses vmalloc address space, page tables and bookkeeping; the figures above compare workspace backing only.
--- crypto/deflate.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
crypto/deflate.c +3 -2
@@ -20,6 +20,7 @@#include <linux/scatterlist.h> #include <linux/slab.h> #include <linux/spinlock.h> +#include <linux/vmalloc.h> #include <linux/zlib.h> #define DEFLATE_DEF_LEVEL Z_DEFAULT_COMPRESSION
@@ -40,7 +41,7 @@ static void *deflate_alloc_stream(void) DEFLATE_DEF_MEMLEVEL));struct deflate_stream *ctx; - ctx =kvmalloc_flex (*ctx, workspace, size); + ctx =vmalloc (struct_size (ctx, workspace, size));if (!ctx)return ERR_PTR (-ENOMEM );
@@ -51,7 +52,7 @@ static void *deflate_alloc_stream(void)static void deflate_free_stream (void *ctx) { -kvfree (ctx); +vfree (ctx); }static struct crypto_acomp_streams deflate_streams = {
Signature · 1 line
2.53.0-Meta
Re: [PATCH 2/2] crypto: deflate - use vmalloc() for stream workspaces
Quoted 12 lines · Usama Arif
The deflate acomp streams outlive the transforms that use them. One stream is allocated during transform initialization; additional per-CPU streams are allocated on demand. They are freed only when the algorithm is unregistered.
In a 4 KiB-page x86-64 build, each stream requests 145,320 bytes. If
kvmalloc()succeeds viakmalloc(), it occupies an order-6 block (256 KiB).vmalloc()needs 36 backing pages (144 KiB), 112 KiB fewer per stream. Across 384 populated streams, that is 42 MiB less workspace backing if all would have usedkmalloc(). There is no such gain whenkvmalloc()already falls back tovmalloc().
Quoted 5 lines
Use
vmalloc()andvfree()directly, as the crypto lz4 implementations already do. This uses vmalloc address space, page tables and bookkeeping; the figures above compare workspace backing only.Signed-off-by Usama Arif <[email protected]>
Signature · 1 line
All Rights Reversed.
Re: [PATCH 2/2] crypto: deflate - use vmalloc() for stream workspaces
In reply to Usama ArifQuoted 12 lines · Usama Arif
The deflate acomp streams outlive the transforms that use them. One stream is allocated during transform initialization; additional per-CPU streams are allocated on demand. They are freed only when the algorithm is unregistered.
In a 4 KiB-page x86-64 build, each stream requests 145,320 bytes. If
kvmalloc()succeeds viakmalloc(), it occupies an order-6 block (256 KiB).vmalloc()needs 36 backing pages (144 KiB), 112 KiB fewer per stream. Across 384 populated streams, that is 42 MiB less workspace backing if all would have usedkmalloc(). There is no such gain whenkvmalloc()already falls back tovmalloc().
Quoted 5 lines
Use
vmalloc()andvfree()directly, as the crypto lz4 implementations already do. This uses vmalloc address space, page tables and bookkeeping; the figures above compare workspace backing only.Signed-off-by Usama Arif <[email protected]>
Re: [PATCH 2/2] crypto: deflate - use vmalloc() for stream workspaces
In reply to Usama ArifHi Usama,
Thanks! I've tested this with zswap using deflate, and it works well.
Chris