Skip to contentSkip to conversation

crypto/arm64: Reduce stack bloat from scoped ksimd

PATCH Applied Ard Biesheuvel 3 participants 6 messages Dec 3, 2025 – Dec 8, 2025

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

cover crypto/arm64: Reduce stack bloat from scoped ksimd

Arnd reports that the new scoped ksimd changes result in excessive stack bloat in the XTS routines in some cases. Fix this for AES-XTS and SM4-XTS.

Note that the offending patches went in via the libcrypto tree, so these changes should either go via the same route, or wait for -rc1

Cc Arnd Bergmann <[email protected]>
Cc Eric Biggers <[email protected]>
Cc Herbert Xu <[email protected]>

Ard Biesheuvel (2):
crypto/arm64: aes/xts - Using single ksimd scope to reduce stack bloat crypto/arm64: sm4/xts: Merge ksimd scopes to reduce stack bloat

 arch/arm64/crypto/aes-glue.c        | 75 ++++++++++----------
 arch/arm64/crypto/aes-neonbs-glue.c | 44 ++++++------
 arch/arm64/crypto/sm4-ce-glue.c     | 42 ++++++-----
 3 files changed, 77 insertions(+), 84 deletions(-)
Signature · 1 line
2.47.3
#1 lore raw

PATCH 1/2 crypto/arm64: aes/xts - Using single ksimd scope to reduce stack bloat

The ciphertext stealing logic in the AES-XTS implementation creates a separate ksimd scope to call into the FP/SIMD core routines, and in some cases (CONFIG_KASAN_STACK is one, but there might be others), the 528 byte kernel mode FP/SIMD buffer that is allocated inside this scope is not shared with the preceding ksimd scope, resulting in unnecessary stack bloat.

Considering that

a) the XTS ciphertext stealing logic is never called for block encryption use cases, and XTS is rarely used for anything else,

b) in the vast majority of cases, the entire input block is processed during the first iteration of the loop,

we can combine both ksimd scopes into a single one with no practical impact on how often/how long FP/SIMD is en/disabled, allowing us to reuse the same stack slot for both FP/SIMD routine calls.

Fixes ba3c1b3b5ac9 ("crypto/arm64: aes-blk - Switch to 'ksimd' scoped guard API")
---
 arch/arm64/crypto/aes-glue.c        | 75 ++++++++++----------
 arch/arm64/crypto/aes-neonbs-glue.c | 44 ++++++------
 2 files changed, 57 insertions(+), 62 deletions(-)
arch/arm64/crypto/aes-glue.c +36 -39
index b087b900d279..c51d4487e9e9 100644
@@ -549,38 +549,37 @@ static int __maybe_unused xts_encrypt(struct skcipher_request *req)
 		tail = 0;
 	}
 
-	for (first = 1; walk.nbytes >= AES_BLOCK_SIZE; first = 0) {
-		int nbytes = walk.nbytes;
+	scoped_ksimd() {
+		for (first = 1; walk.nbytes >= AES_BLOCK_SIZE; first = 0) {
+			int nbytes = walk.nbytes;
 
-		if (walk.nbytes < walk.total)
-			nbytes &= ~(AES_BLOCK_SIZE - 1);
+			if (walk.nbytes < walk.total)
+				nbytes &= ~(AES_BLOCK_SIZE - 1);
 
-		scoped_ksimd()
 			aes_xts_encrypt(walk.dst.virt.addr, walk.src.virt.addr,
 					ctx->key1.key_enc, rounds, nbytes,
 					ctx->key2.key_enc, walk.iv, first);
-		err = skcipher_walk_done(&walk, walk.nbytes - nbytes);
-	}
+			err = skcipher_walk_done(&walk, walk.nbytes - nbytes);
+		}
 
-	if (err || likely(!tail))
-		return err;
+		if (err || likely(!tail))
+			return err;
 
-	dst = src = scatterwalk_ffwd(sg_src, req->src, req->cryptlen);
-	if (req->dst != req->src)
-		dst = scatterwalk_ffwd(sg_dst, req->dst, req->cryptlen);
+		dst = src = scatterwalk_ffwd(sg_src, req->src, req->cryptlen);
+		if (req->dst != req->src)
+			dst = scatterwalk_ffwd(sg_dst, req->dst, req->cryptlen);
 
-	skcipher_request_set_crypt(req, src, dst, AES_BLOCK_SIZE + tail,
-				   req->iv);
+		skcipher_request_set_crypt(req, src, dst, AES_BLOCK_SIZE + tail,
+					   req->iv);
 
-	err = skcipher_walk_virt(&walk, &subreq, false);
-	if (err)
-		return err;
+		err = skcipher_walk_virt(&walk, &subreq, false);
+		if (err)
+			return err;
 
-	scoped_ksimd()
 		aes_xts_encrypt(walk.dst.virt.addr, walk.src.virt.addr,
 				ctx->key1.key_enc, rounds, walk.nbytes,
 				ctx->key2.key_enc, walk.iv, first);
-
+	}
 	return skcipher_walk_done(&walk, 0);
 }
 
@@ -619,39 +618,37 @@ static int __maybe_unused xts_decrypt(struct skcipher_request *req)
 		tail = 0;
 	}
 
-	for (first = 1; walk.nbytes >= AES_BLOCK_SIZE; first = 0) {
-		int nbytes = walk.nbytes;
+	scoped_ksimd() {
+		for (first = 1; walk.nbytes >= AES_BLOCK_SIZE; first = 0) {
+			int nbytes = walk.nbytes;
 
-		if (walk.nbytes < walk.total)
-			nbytes &= ~(AES_BLOCK_SIZE - 1);
+			if (walk.nbytes < walk.total)
+				nbytes &= ~(AES_BLOCK_SIZE - 1);
 
-		scoped_ksimd()
 			aes_xts_decrypt(walk.dst.virt.addr, walk.src.virt.addr,
 					ctx->key1.key_dec, rounds, nbytes,
 					ctx->key2.key_enc, walk.iv, first);
-		err = skcipher_walk_done(&walk, walk.nbytes - nbytes);
-	}
+			err = skcipher_walk_done(&walk, walk.nbytes - nbytes);
+		}
 
-	if (err || likely(!tail))
-		return err;
-
-	dst = src = scatterwalk_ffwd(sg_src, req->src, req->cryptlen);
-	if (req->dst != req->src)
-		dst = scatterwalk_ffwd(sg_dst, req->dst, req->cryptlen);
+		if (err || likely(!tail))
+			return err;
 
-	skcipher_request_set_crypt(req, src, dst, AES_BLOCK_SIZE + tail,
-				   req->iv);
+		dst = src = scatterwalk_ffwd(sg_src, req->src, req->cryptlen);
+		if (req->dst != req->src)
+			dst = scatterwalk_ffwd(sg_dst, req->dst, req->cryptlen);
 
-	err = skcipher_walk_virt(&walk, &subreq, false);
-	if (err)
-		return err;
+		skcipher_request_set_crypt(req, src, dst, AES_BLOCK_SIZE + tail,
+					   req->iv);
 
+		err = skcipher_walk_virt(&walk, &subreq, false);
+		if (err)
+			return err;
 
-	scoped_ksimd()
 		aes_xts_decrypt(walk.dst.virt.addr, walk.src.virt.addr,
 				ctx->key1.key_dec, rounds, walk.nbytes,
 				ctx->key2.key_enc, walk.iv, first);
-
+	}
 	return skcipher_walk_done(&walk, 0);
 }
 
arch/arm64/crypto/aes-neonbs-glue.c +21 -23

64 lines not on this page · raw message

Signature · 1 line
2.47.3
#2 lore raw

PATCH 2/2 crypto/arm64: sm4/xts: Merge ksimd scopes to reduce stack bloat

In reply to Ard Biesheuvel

Merge the two ksimd scopes in the implementation of SM4-XTS to prevent stack bloat in cases where the compiler fails to combine the stack slots for the kernel mode FP/SIMD buffers.

---
 arch/arm64/crypto/sm4-ce-glue.c | 42 ++++++++++----------
 1 file changed, 20 insertions(+), 22 deletions(-)
arch/arm64/crypto/sm4-ce-glue.c +20 -22
index 5569cece5a0b..0933ba45fbe7 100644
@@ -346,11 +346,11 @@ static int sm4_xts_crypt(struct skcipher_request *req, bool encrypt)
 		tail = 0;
 	}
 
-	while ((nbytes = walk.nbytes) >= SM4_BLOCK_SIZE) {
-		if (nbytes < walk.total)
-			nbytes &= ~(SM4_BLOCK_SIZE - 1);
+	scoped_ksimd() {
+		while ((nbytes = walk.nbytes) >= SM4_BLOCK_SIZE) {
+			if (nbytes < walk.total)
+				nbytes &= ~(SM4_BLOCK_SIZE - 1);
 
-		scoped_ksimd() {
 			if (encrypt)
 				sm4_ce_xts_enc(ctx->key1.rkey_enc, walk.dst.virt.addr,
 						walk.src.virt.addr, walk.iv, nbytes,
@@ -359,32 +359,30 @@ static int sm4_xts_crypt(struct skcipher_request *req, bool encrypt)
 				sm4_ce_xts_dec(ctx->key1.rkey_dec, walk.dst.virt.addr,
 						walk.src.virt.addr, walk.iv, nbytes,
 						rkey2_enc);
-		}
 
-		rkey2_enc = NULL;
+			rkey2_enc = NULL;
 
-		err = skcipher_walk_done(&walk, walk.nbytes - nbytes);
-		if (err)
-			return err;
-	}
+			err = skcipher_walk_done(&walk, walk.nbytes - nbytes);
+			if (err)
+				return err;
+		}
 
-	if (likely(tail == 0))
-		return 0;
+		if (likely(tail == 0))
+			return 0;
 
-	/* handle ciphertext stealing */
+		/* handle ciphertext stealing */
 
-	dst = src = scatterwalk_ffwd(sg_src, req->src, subreq.cryptlen);
-	if (req->dst != req->src)
-		dst = scatterwalk_ffwd(sg_dst, req->dst, subreq.cryptlen);
+		dst = src = scatterwalk_ffwd(sg_src, req->src, subreq.cryptlen);
+		if (req->dst != req->src)
+			dst = scatterwalk_ffwd(sg_dst, req->dst, subreq.cryptlen);
 
-	skcipher_request_set_crypt(&subreq, src, dst, SM4_BLOCK_SIZE + tail,
-				   req->iv);
+		skcipher_request_set_crypt(&subreq, src, dst, SM4_BLOCK_SIZE + tail,
+					   req->iv);
 
-	err = skcipher_walk_virt(&walk, &subreq, false);
-	if (err)
-		return err;
+		err = skcipher_walk_virt(&walk, &subreq, false);
+		if (err)
+			return err;
 
-	scoped_ksimd() {
 		if (encrypt)
 			sm4_ce_xts_enc(ctx->key1.rkey_enc, walk.dst.virt.addr,
 					walk.src.virt.addr, walk.iv, walk.nbytes,
Signature · 1 line
2.47.3
#3 lore raw
In reply to Ard Biesheuvel
Quoted 7 lines · Ard Biesheuvel

Arnd reports that the new scoped ksimd changes result in excessive stack bloat in the XTS routines in some cases. Fix this for AES-XTS and SM4-XTS.

Note that the offending patches went in via the libcrypto tree, so these changes should either go via the same route, or wait for -rc1

Quoted 8 lines
Cc Arnd Bergmann <[email protected]>
Cc Eric Biggers <[email protected]>
Cc Herbert Xu <[email protected]>

Ard Biesheuvel (2):
crypto/arm64: aes/xts - Using single ksimd scope to reduce stack bloat crypto/arm64: sm4/xts: Merge ksimd scopes to reduce stack bloat

Thanks, looks good to me. I'll plan to take these and send another pull request probably sometime next week.

- Eric

Tested-by on the v1 series
#4 lore raw
In reply to Ard Biesheuvel
Quoted 7 lines · Ard Biesheuvel

Arnd reports that the new scoped ksimd changes result in excessive stack bloat in the XTS routines in some cases. Fix this for AES-XTS and SM4-XTS.

Note that the offending patches went in via the libcrypto tree, so these changes should either go via the same route, or wait for -rc1

Quoted 7 lines
Cc Arnd Bergmann <[email protected]>
Cc Eric Biggers <[email protected]>
Cc Herbert Xu <[email protected]>

Ard Biesheuvel (2):
crypto/arm64: aes/xts - Using single ksimd scope to reduce stack bloat crypto/arm64: sm4/xts: Merge ksimd scopes to reduce stack bloat

I've tested a few of the configurations that I saw issues on and they all look good so far.

I'll leave randconfig builds running with this enabled to see if anything was missing.

Arnd

Applied
#5 lore raw
In reply to Ard Biesheuvel
Quoted 11 lines · Ard Biesheuvel

Arnd reports that the new scoped ksimd changes result in excessive stack bloat in the XTS routines in some cases. Fix this for AES-XTS and SM4-XTS.

Note that the offending patches went in via the libcrypto tree, so these changes should either go via the same route, or wait for -rc1

Cc Arnd Bergmann <[email protected]>
Cc Eric Biggers <[email protected]>
Cc Herbert Xu <[email protected]>
Quoted 8 lines

Ard Biesheuvel (2):
crypto/arm64: aes/xts - Using single ksimd scope to reduce stack bloat crypto/arm64: sm4/xts: Merge ksimd scopes to reduce stack bloat

 arch/arm64/crypto/aes-glue.c        | 75 ++++++++++----------
 arch/arm64/crypto/aes-neonbs-glue.c | 44 ++++++------
 arch/arm64/crypto/sm4-ce-glue.c     | 42 ++++++-----
 3 files changed, 77 insertions(+), 84 deletions(-)

Applied to https://git.kernel.org/pub/scm/linux/kernel/git/ebiggers/linux.git/log/?h=libcrypto-fixes

- Eric