Skip to contentSkip to conversation

crypto: use vmalloc() for large stream workspaces

PATCH In review Usama Arif 4 participants 9 messages Nov 11 – Nov 13

Subsystems (3)
lore
Share & tools
lore

Usama Arif

author
#0 lore raw

cover crypto: use vmalloc() for large stream workspaces

The zstd and deflate acomp implementations retain per-CPU stream workspaces until the algorithms are unregistered. On x86-64 with 4 KiB pages, kvmalloc() can back them with order-9 and order-6 kmalloc() blocks, respectively.

Use vmalloc() and vfree() instead. For a server with 384 CPUs, if we have streams on all of the CPUs and kvmalloc() takes the kmalloc() path for every stream, their workspaces use 193.5 MiB more for zstd and 42 MiB for deflate.

In five before/after boot pairs on a 96-vCPU x86-64 VM with 8 GiB RAM, the test populated both algorithms crypto workspaces on every CPU. All 192 baseline workspaces used kmalloc() without these patches. The patches reduced workspace backing by 48.375 MiB for zstd and 10.5 MiB for deflate —58.875 MiB combined—in each pair.

In five before/after runs in a 4-vCPU x86-64 VM, median compression and decompression throughput changed by less than 1% (-0.93% to +0.64%). Inputs were 4 KiB and 64 KiB synthetic buffers containing pseudorandom data or a 50/50 mix of repeating-pattern and pseudorandom bytes. The differences were within the observed run-to-run variation.

Usama Arif (2):
crypto: zstd - use vmalloc() for stream workspaces
crypto: deflate - use vmalloc() for stream workspaces

 crypto/deflate.c | 5 +++--
 crypto/zstd.c    | 4 ++--
 2 files changed, 5 insertions(+), 4 deletions(-)
Signature · 1 line
2.53.0-Meta
author
#1 lore raw

PATCH 1/2 crypto: zstd - use vmalloc() for stream workspaces

The zstd acomp streams retain their workspaces after the last transform has been freed. The first is allocated during transform initialization; additional per-CPU streams are allocated on demand. They are released when the algorithm is unregistered.

In a 4 KiB-page x86-64 build, each stream requests 1,566,281 bytes. If kvmalloc() succeeds via kmalloc(), it occupies an order-9 block (2 MiB). vmalloc() needs 383 backing pages (1,532 KiB), 516 KiB fewer per stream. Across 384 populated streams, that is 193.5 MiB less workspace backing if all would have used kmalloc(). There is no such gain when kvmalloc() already falls back to vmalloc().

Use vmalloc() and vfree() directly to avoid the high-order allocation. This uses vmalloc address space, page tables and bookkeeping; the figures above compare workspace backing only.

---
 crypto/zstd.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
crypto/zstd.c +2 -2
index 76bdaca327330..f012293919368 100644
@@ -44,7 +44,7 @@ static void *zstd_alloc_stream(void)
 	if (!wksp_size)
 		return ERR_PTR(-EINVAL);
 
-	ctx = kvmalloc_flex(*ctx, wksp, wksp_size);
+	ctx = vmalloc(struct_size(ctx, wksp, wksp_size));
 	if (!ctx)
 		return ERR_PTR(-ENOMEM);
 
@@ -56,7 +56,7 @@ static void *zstd_alloc_stream(void)
 
 static void zstd_free_stream(void *ctx)
 {
-	kvfree(ctx);
+	vfree(ctx);
 }
 
 static struct crypto_acomp_streams zstd_streams = {
Signature · 1 line
2.53.0-Meta
Reviewed-by on v1 1/2
#2 lore raw

Re: [PATCH 1/2] crypto: zstd - use vmalloc() for stream workspaces

Quoted 7 lines · Usama Arif

The zstd acomp streams retain their workspaces after the last transform
has been freed. The first is allocated during transform initialization;
additional per-CPU streams are allocated on demand. They are released when the algorithm is unregistered.

Quoted 5 lines

In a 4 KiB-page x86-64 build, each stream requests 1,566,281 bytes. If kvmalloc() succeeds via kmalloc(), it occupies an order-9 block (2 MiB). vmalloc() needs 383 backing pages (1,532 KiB), 516 KiB fewer per stream. Across 384 populated streams, that is 193.5 MiB less

Woah. Nice catch

Quoted 2 lines
Signature · 1 line
All Rights Reversed.
Reviewed-by on v1 1/2
#3 lore raw

Re: [PATCH 1/2] crypto: zstd - use vmalloc() for stream workspaces

In reply to Usama Arif
Quoted 12 lines · Usama Arif

The zstd acomp streams retain their workspaces after the last transform has been freed. The first is allocated during transform initialization; additional per-CPU streams are allocated on demand. They are released when the algorithm is unregistered.

In a 4 KiB-page x86-64 build, each stream requests 1,566,281 bytes. If kvmalloc() succeeds via kmalloc(), it occupies an order-9 block (2 MiB). vmalloc() needs 383 backing pages (1,532 KiB), 516 KiB fewer per stream. Across 384 populated streams, that is 193.5 MiB less workspace backing if all would have used kmalloc(). There is no such gain when kvmalloc() already falls back to vmalloc().

Quoted 5 lines

Use vmalloc() and vfree() directly to avoid the high-order allocation. This uses vmalloc address space, page tables and bookkeeping; the figures above compare workspace backing only.

Nice catch. Given the lifetime of these streams, there really isn't enough turnover to justify the kmalloc optimization to begin with.

Reviewed-by Tested-by on v1 1/2
#4 lore raw

Re: [PATCH 1/2] crypto: zstd - use vmalloc() for stream workspaces

In reply to Usama Arif

Hi Usama,

Quoted 1 line · Usama Arif

Use vmalloc() and vfree() directly to avoid the high-order allocation.

vmalloc() doesn't avoid high-order allocations any more. Since a06157804399 ("mm/vmalloc: request large order pages from buddy allocator"), vm_area_alloc_pages() first tries the largest order that fits, and splits the block up afterwards. So for these 383 pages it still asks for an order-8 block first, then order 6, 5, 4, 3, 2 and 1.

What the patch does avoid is kmalloc() rounding the 1.5 MiB up to a 2 MiB block, which is where the 516 KiB per stream comes from. Maybe say that instead?

Once that's fixed, please feel free to add:

Chris

author
#5 lore raw

PATCH 2/2 crypto: deflate - use vmalloc() for stream workspaces

In reply to Usama Arif

The deflate acomp streams outlive the transforms that use them. One stream is allocated during transform initialization; additional per-CPU streams are allocated on demand. They are freed only when the algorithm is unregistered.

In a 4 KiB-page x86-64 build, each stream requests 145,320 bytes. If kvmalloc() succeeds via kmalloc(), it occupies an order-6 block (256 KiB). vmalloc() needs 36 backing pages (144 KiB), 112 KiB fewer per stream. Across 384 populated streams, that is 42 MiB less workspace backing if all would have used kmalloc(). There is no such gain when kvmalloc() already falls back to vmalloc().

Use vmalloc() and vfree() directly, as the crypto lz4 implementations already do. This uses vmalloc address space, page tables and bookkeeping; the figures above compare workspace backing only.

---
 crypto/deflate.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
crypto/deflate.c +3 -2
index 710ebba7ce850..a8692532a65da 100644
@@ -20,6 +20,7 @@
 #include <linux/scatterlist.h>
 #include <linux/slab.h>
 #include <linux/spinlock.h>
+#include <linux/vmalloc.h>
 #include <linux/zlib.h>
 
 #define DEFLATE_DEF_LEVEL		Z_DEFAULT_COMPRESSION
@@ -40,7 +41,7 @@ static void *deflate_alloc_stream(void)
 						     DEFLATE_DEF_MEMLEVEL));
 	struct deflate_stream *ctx;
 
-	ctx = kvmalloc_flex(*ctx, workspace, size);
+	ctx = vmalloc(struct_size(ctx, workspace, size));
 	if (!ctx)
 		return ERR_PTR(-ENOMEM);
 
@@ -51,7 +52,7 @@ static void *deflate_alloc_stream(void)
 
 static void deflate_free_stream(void *ctx)
 {
-	kvfree(ctx);
+	vfree(ctx);
 }
 
 static struct crypto_acomp_streams deflate_streams = {
Signature · 1 line
2.53.0-Meta
Reviewed-by on v1 2/2
#6 lore raw

Re: [PATCH 2/2] crypto: deflate - use vmalloc() for stream workspaces

Quoted 12 lines · Usama Arif

The deflate acomp streams outlive the transforms that use them. One stream is allocated during transform initialization; additional per-CPU streams are allocated on demand. They are freed only when the algorithm is unregistered.

In a 4 KiB-page x86-64 build, each stream requests 145,320 bytes. If kvmalloc() succeeds via kmalloc(), it occupies an order-6 block (256 KiB). vmalloc() needs 36 backing pages (144 KiB), 112 KiB fewer per stream. Across 384 populated streams, that is 42 MiB less workspace backing if all would have used kmalloc(). There is no such gain when kvmalloc() already falls back to vmalloc().

Quoted 5 lines

Use vmalloc() and vfree() directly, as the crypto lz4 implementations already do. This uses vmalloc address space, page tables and bookkeeping; the figures above compare workspace backing only.

Signature · 1 line
All Rights Reversed.
Reviewed-by on v1 2/2
#7 lore raw

Re: [PATCH 2/2] crypto: deflate - use vmalloc() for stream workspaces

In reply to Usama Arif
Quoted 12 lines · Usama Arif

The deflate acomp streams outlive the transforms that use them. One stream is allocated during transform initialization; additional per-CPU streams are allocated on demand. They are freed only when the algorithm is unregistered.

In a 4 KiB-page x86-64 build, each stream requests 145,320 bytes. If kvmalloc() succeeds via kmalloc(), it occupies an order-6 block (256 KiB). vmalloc() needs 36 backing pages (144 KiB), 112 KiB fewer per stream. Across 384 populated streams, that is 42 MiB less workspace backing if all would have used kmalloc(). There is no such gain when kvmalloc() already falls back to vmalloc().

Quoted 5 lines

Use vmalloc() and vfree() directly, as the crypto lz4 implementations already do. This uses vmalloc address space, page tables and bookkeeping; the figures above compare workspace backing only.