Skip to contentSkip to conversation

FAILED: patch "[PATCH] staging: rtl8723bs: Avoid memset() in aes_cipher() and" failed to apply to 5.10-stable tree

PATCH In review [email protected] 4 participants 4 messages Jun 30, 2025 – Jul 2, 2025

Subsystems (2)
lore
Share & tools
lore

[email protected]

author maintainer
#0 lore raw

The patch below does not apply to the 5.10-stable tree.
If someone wants it applied there, or to any other stable or longterm tree, then please email the backport, including the original git commit id to <[email protected]>.

To reproduce the conflict and resubmit, you may use the following commands:

git fetch https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/ linux-5.10.y
git checkout FETCH_HEAD
git cherry-pick -x a55bc4ffc06d8c965a7d6f0a01ed0ed41380df28
# <resolve conflicts, build, test, etc.>
git commit -s
git send-email --to '<[email protected]>' --in-reply-to '2025063055-overfed-dispute-71ba@gregkh' --subject-prefix 'PATCH 5.10.y' HEAD^..

Possible dependencies:

thanks,

greg k-h

------------------ original commit in Linus's tree ------------------

From a55bc4ffc06d8c965a7d6f0a01ed0ed41380df28 Mon Sep 17 00:00:00 2001
From: Nathan Chancellor <[email protected]>
Date: Mon, 9 Jun 2025 14:13:14 -0700
Subject: [PATCH] staging: rtl8723bs: Avoid memset() in aes_cipher() and
aes_decipher()

After commit 6f110a5e4f99 ("Disable SLUB_TINY for build testing"), which causes CONFIG_KASAN to be enabled in allmodconfig again, arm64 allmodconfig builds with older versions of clang (15 through 17) show an instance of -Wframe-larger-than (which breaks the build with CONFIG_WERROR=y):

drivers/staging/rtl8723bs/core/rtw_security.c:1287:5: error: stack frame size (2208) exceeds limit (2048) in 'rtw_aes_decrypt' [-Werror,-Wframe-larger-than]
1287 | u32 rtw_aes_decrypt(struct adapter *padapter, u8 *precvframe)

        |     ^

This comes from aes_decipher() being inlined in rtw_aes_decrypt(). Running the same build with CONFIG_FRAME_WARN=128 shows aes_cipher() also uses a decent amount of stack, just under the limit of 2048:

drivers/staging/rtl8723bs/core/rtw_security.c:864:19: warning: stack frame size (1952) exceeds limit (128) in 'aes_cipher' [-Wframe-larger-than]
864 | static signed int aes_cipher(u8 *key, uint hdrlen,
| ^

-Rpass-analysis=stack-frame-layout only shows one large structure on the stack, which is the ctx variable inlined from aes128k128d(). A good number of the other variables come from the additional checks of fortified string routines, which are present in memset(), which both aes_cipher() and aes_decipher() use to initialize some temporary buffers. In this case, since the size is known at compile time, these additional checks should not result in any code generation changes but allmodconfig has several sanitizers enabled, which may make it harder for the compiler to eliminate the compile time checks and the variables that come about from them.

The memset() calls are just initializing these buffers to zero, so use '= {}' instead, which is used all over the kernel and does the exact same thing as memset() without the fortify checks, which drops the stack usage of these functions by a few hundred kilobytes.

drivers/staging/rtl8723bs/core/rtw_security.c:864:19: warning: stack frame size (1584) exceeds limit (128) in 'aes_cipher' [-Wframe-larger-than]
864 | static signed int aes_cipher(u8 *key, uint hdrlen,
| ^
drivers/staging/rtl8723bs/core/rtw_security.c:1271:5: warning: stack frame size (1456) exceeds limit (128) in 'rtw_aes_decrypt' [-Wframe-larger-than]
1271 | u32 rtw_aes_decrypt(struct adapter *padapter, u8 *precvframe)

        |     ^
drivers/staging/rtl8723bs/core/rtw_security.c +14 -30
index 1e9eff01b1aa..e9f382c280d9 100644
@@ -868,29 +868,21 @@ static signed int aes_cipher(u8 *key, uint	hdrlen,
 		num_blocks, payload_index;
 
 	u8 pn_vector[6];
-	u8 mic_iv[16];
-	u8 mic_header1[16];
-	u8 mic_header2[16];
-	u8 ctr_preload[16];
+	u8 mic_iv[16] = {};
+	u8 mic_header1[16] = {};
+	u8 mic_header2[16] = {};
+	u8 ctr_preload[16] = {};
 
 	/* Intermediate Buffers */
-	u8 chain_buffer[16];
-	u8 aes_out[16];
-	u8 padded_buffer[16];
+	u8 chain_buffer[16] = {};
+	u8 aes_out[16] = {};
+	u8 padded_buffer[16] = {};
 	u8 mic[8];
 	uint	frtype  = GetFrameType(pframe);
 	uint	frsubtype  = GetFrameSubType(pframe);
 
 	frsubtype = frsubtype>>4;
 
-	memset((void *)mic_iv, 0, 16);
-	memset((void *)mic_header1, 0, 16);
-	memset((void *)mic_header2, 0, 16);
-	memset((void *)ctr_preload, 0, 16);
-	memset((void *)chain_buffer, 0, 16);
-	memset((void *)aes_out, 0, 16);
-	memset((void *)padded_buffer, 0, 16);
-
 	if ((hdrlen == WLAN_HDR_A3_LEN) || (hdrlen ==  WLAN_HDR_A3_QOS_LEN))
 		a4_exists = 0;
 	else
@@ -1080,15 +1072,15 @@ static signed int aes_decipher(u8 *key, uint	hdrlen,
 			num_blocks, payload_index;
 	signed int res = _SUCCESS;
 	u8 pn_vector[6];
-	u8 mic_iv[16];
-	u8 mic_header1[16];
-	u8 mic_header2[16];
-	u8 ctr_preload[16];
+	u8 mic_iv[16] = {};
+	u8 mic_header1[16] = {};
+	u8 mic_header2[16] = {};
+	u8 ctr_preload[16] = {};
 
 		/* Intermediate Buffers */
-	u8 chain_buffer[16];
-	u8 aes_out[16];
-	u8 padded_buffer[16];
+	u8 chain_buffer[16] = {};
+	u8 aes_out[16] = {};
+	u8 padded_buffer[16] = {};
 	u8 mic[8];
 
 	uint frtype  = GetFrameType(pframe);
@@ -1096,14 +1088,6 @@ static signed int aes_decipher(u8 *key, uint	hdrlen,
 
 	frsubtype = frsubtype>>4;
 
-	memset((void *)mic_iv, 0, 16);
-	memset((void *)mic_header1, 0, 16);
-	memset((void *)mic_header2, 0, 16);
-	memset((void *)ctr_preload, 0, 16);
-	memset((void *)chain_buffer, 0, 16);
-	memset((void *)aes_out, 0, 16);
-	memset((void *)padded_buffer, 0, 16);
-
 	/* start to decrypt the payload */
 
 	num_blocks = (plen-8) / 16; /* plen including LLC, payload_length and mic) */
#1 lore raw

PATCH 1/1 staging: rtl8723bs: Avoid memset() in aes_cipher() and aes_decipher()

commit a55bc4ffc06d8c965a7d6f0a01ed0ed41380df28 upstream.

After commit 6f110a5e4f99 ("Disable SLUB_TINY for build testing"), which causes CONFIG_KASAN to be enabled in allmodconfig again, arm64 allmodconfig builds with older versions of clang (15 through 17) show an instance of -Wframe-larger-than (which breaks the build with CONFIG_WERROR=y):

drivers/staging/rtl8723bs/core/rtw_security.c:1287:5: error: stack frame size (2208) exceeds limit (2048) in 'rtw_aes_decrypt' [-Werror,-Wframe-larger-than]
1287 | u32 rtw_aes_decrypt(struct adapter *padapter, u8 *precvframe)

        |     ^

This comes from aes_decipher() being inlined in rtw_aes_decrypt(). Running the same build with CONFIG_FRAME_WARN=128 shows aes_cipher() also uses a decent amount of stack, just under the limit of 2048:

drivers/staging/rtl8723bs/core/rtw_security.c:864:19: warning: stack frame size (1952) exceeds limit (128) in 'aes_cipher' [-Wframe-larger-than]
864 | static signed int aes_cipher(u8 *key, uint hdrlen,
| ^

-Rpass-analysis=stack-frame-layout only shows one large structure on the stack, which is the ctx variable inlined from aes128k128d(). A good number of the other variables come from the additional checks of fortified string routines, which are present in memset(), which both aes_cipher() and aes_decipher() use to initialize some temporary buffers. In this case, since the size is known at compile time, these additional checks should not result in any code generation changes but allmodconfig has several sanitizers enabled, which may make it harder for the compiler to eliminate the compile time checks and the variables that come about from them.

The memset() calls are just initializing these buffers to zero, so use '= {}' instead, which is used all over the kernel and does the exact same thing as memset() without the fortify checks, which drops the stack usage of these functions by a few hundred kilobytes.

drivers/staging/rtl8723bs/core/rtw_security.c:864:19: warning: stack frame size (1584) exceeds limit (128) in 'aes_cipher' [-Wframe-larger-than]
864 | static signed int aes_cipher(u8 *key, uint hdrlen,
| ^
drivers/staging/rtl8723bs/core/rtw_security.c:1271:5: warning: stack frame size (1456) exceeds limit (128) in 'rtw_aes_decrypt' [-Wframe-larger-than]
1271 | u32 rtw_aes_decrypt(struct adapter *padapter, u8 *precvframe)

        |     ^
---
 drivers/staging/rtl8723bs/core/rtw_security.c | 46 ++++++-------------
 1 file changed, 14 insertions(+), 32 deletions(-)
drivers/staging/rtl8723bs/core/rtw_security.c +14 -32
index 159d32ace2bc..cc709e849f39 100644
@@ -1299,30 +1299,21 @@ static sint aes_cipher(u8 *key, uint	hdrlen,
 		num_blocks, payload_index;
 
 	u8 pn_vector[6];
-	u8 mic_iv[16];
-	u8 mic_header1[16];
-	u8 mic_header2[16];
-	u8 ctr_preload[16];
+	u8 mic_iv[16] = {};
+	u8 mic_header1[16] = {};
+	u8 mic_header2[16] = {};
+	u8 ctr_preload[16] = {};
 
 	/* Intermediate Buffers */
-	u8 chain_buffer[16];
-	u8 aes_out[16];
-	u8 padded_buffer[16];
+	u8 chain_buffer[16] = {};
+	u8 aes_out[16] = {};
+	u8 padded_buffer[16] = {};
 	u8 mic[8];
 	uint	frtype  = GetFrameType(pframe);
 	uint	frsubtype  = GetFrameSubType(pframe);
 
 	frsubtype = frsubtype>>4;
 
-
-	memset((void *)mic_iv, 0, 16);
-	memset((void *)mic_header1, 0, 16);
-	memset((void *)mic_header2, 0, 16);
-	memset((void *)ctr_preload, 0, 16);
-	memset((void *)chain_buffer, 0, 16);
-	memset((void *)aes_out, 0, 16);
-	memset((void *)padded_buffer, 0, 16);
-
 	if ((hdrlen == WLAN_HDR_A3_LEN) || (hdrlen ==  WLAN_HDR_A3_QOS_LEN))
 		a4_exists = 0;
 	else
@@ -1540,15 +1531,15 @@ static sint aes_decipher(u8 *key, uint	hdrlen,
 			num_blocks, payload_index;
 	sint res = _SUCCESS;
 	u8 pn_vector[6];
-	u8 mic_iv[16];
-	u8 mic_header1[16];
-	u8 mic_header2[16];
-	u8 ctr_preload[16];
+	u8 mic_iv[16] = {};
+	u8 mic_header1[16] = {};
+	u8 mic_header2[16] = {};
+	u8 ctr_preload[16] = {};
 
 		/* Intermediate Buffers */
-	u8 chain_buffer[16];
-	u8 aes_out[16];
-	u8 padded_buffer[16];
+	u8 chain_buffer[16] = {};
+	u8 aes_out[16] = {};
+	u8 padded_buffer[16] = {};
 	u8 mic[8];
 
 
@@ -1557,15 +1548,6 @@ static sint aes_decipher(u8 *key, uint	hdrlen,
 
 	frsubtype = frsubtype>>4;
 
-
-	memset((void *)mic_iv, 0, 16);
-	memset((void *)mic_header1, 0, 16);
-	memset((void *)mic_header2, 0, 16);
-	memset((void *)ctr_preload, 0, 16);
-	memset((void *)chain_buffer, 0, 16);
-	memset((void *)aes_out, 0, 16);
-	memset((void *)padded_buffer, 0, 16);
-
 	/* start to decrypt the payload */
 
 	num_blocks = (plen-8) / 16; /* plen including LLC, payload_length and mic) */
Signature · 1 line
2.50.0
maintainer
#2 lore raw

Re: [PATCH 5.10 and 5.4] staging: rtl8723bs: Avoid memset() in aes_cipher() and aes_decipher()

[ Sasha's backport helper bot ]

Hi,

✅ All tests passed successfully. No issues detected. No action required from the submitter.

The upstream commit SHA1 provided is correct: a55bc4ffc06d8c965a7d6f0a01ed0ed41380df28

Status in newer kernel trees:
6.15.y | Present (different SHA1: 53d7bf452fe7) 6.12.y | Present (different SHA1: 5da335f62003) 6.6.y | Present (different SHA1: b62980fa236b) 6.1.y | Present (different SHA1: 5d678ffa4843) 5.15.y | Present (different SHA1: 4b29ab1d5c42) 5.10.y | Not found

Note: The patch differs from the upstream commit:
---
1: a55bc4ffc06d8 ! 1: 72e918d867583 staging: rtl8723bs: Avoid memset() in aes_cipher() and aes_decipher()

    @@ Metadata
      ## Commit message ##
         staging: rtl8723bs: Avoid memset() in aes_cipher() and aes_decipher()
     
    +    commit a55bc4ffc06d8c965a7d6f0a01ed0ed41380df28 upstream.
    +
         After commit 6f110a5e4f99 ("Disable SLUB_TINY for build testing"), which
         causes CONFIG_KASAN to be enabled in allmodconfig again, arm64
         allmodconfig builds with older versions of clang (15 through 17) show an
    @@ Commit message
         Reviewed-by: Dan Carpenter <[email protected]>
         Link: https://lore.kernel.org/r/20250609-rtl8723bs-fix-clang-arm64-wflt-v1-1-e2accba43def@kernel.org
         Signed-off-by: Greg Kroah-Hartman <[email protected]>
    +    Signed-off-by: Nathan Chancellor <[email protected]>
     
      ## drivers/staging/rtl8723bs/core/rtw_security.c ##
    -@@ drivers/staging/rtl8723bs/core/rtw_security.c: static signed int aes_cipher(u8 *key, uint	hdrlen,
    +@@ drivers/staging/rtl8723bs/core/rtw_security.c: static sint aes_cipher(u8 *key, uint	hdrlen,
      		num_blocks, payload_index;
      
      	u8 pn_vector[6];
    @@ drivers/staging/rtl8723bs/core/rtw_security.c: static signed int aes_cipher(u8 *
      
      	frsubtype = frsubtype>>4;
      
    +-
     -	memset((void *)mic_iv, 0, 16);
     -	memset((void *)mic_header1, 0, 16);
     -	memset((void *)mic_header2, 0, 16);
    @@ drivers/staging/rtl8723bs/core/rtw_security.c: static signed int aes_cipher(u8 *
      	if ((hdrlen == WLAN_HDR_A3_LEN) || (hdrlen ==  WLAN_HDR_A3_QOS_LEN))
      		a4_exists = 0;
      	else
    -@@ drivers/staging/rtl8723bs/core/rtw_security.c: static signed int aes_decipher(u8 *key, uint	hdrlen,
    +@@ drivers/staging/rtl8723bs/core/rtw_security.c: static sint aes_decipher(u8 *key, uint	hdrlen,
      			num_blocks, payload_index;
    - 	signed int res = _SUCCESS;
    + 	sint res = _SUCCESS;
      	u8 pn_vector[6];
     -	u8 mic_iv[16];
     -	u8 mic_header1[16];
    @@ drivers/staging/rtl8723bs/core/rtw_security.c: static signed int aes_decipher(u8
     +	u8 padded_buffer[16] = {};
      	u8 mic[8];
      
    - 	uint frtype  = GetFrameType(pframe);
    -@@ drivers/staging/rtl8723bs/core/rtw_security.c: static signed int aes_decipher(u8 *key, uint	hdrlen,
    + 
    +@@ drivers/staging/rtl8723bs/core/rtw_security.c: static sint aes_decipher(u8 *key, uint	hdrlen,
      
      	frsubtype = frsubtype>>4;
      
    +-
     -	memset((void *)mic_iv, 0, 16);
     -	memset((void *)mic_header1, 0, 16);
     -	memset((void *)mic_header2, 0, 16);

---

Results of testing on various branches:

| Branch                    | Patch Apply | Build Test |
|---------------------------|-------------|------------|
| stable/linux-5.4.y        |  Success    |  Success   |
| stable/linux-5.10.y       |  Success    |  Success   |
#3 lore raw

Re: [PATCH 5.10 and 5.4] staging: rtl8723bs: Avoid memset() in aes_cipher() and aes_decipher()

In reply to Nathan Chancellor
Quoted 20 lines · On Tue, 1 Jul 2025 08:23:24 -0700 Nathan Chancellor <[email protected]> wrote:

commit a55bc4ffc06d8c965a7d6f0a01ed0ed41380df28 upstream.

After commit 6f110a5e4f99 ("Disable SLUB_TINY for build testing"), which causes CONFIG_KASAN to be enabled in allmodconfig again, arm64 allmodconfig builds with older versions of clang (15 through 17) show an instance of -Wframe-larger-than (which breaks the build with CONFIG_WERROR=y):

drivers/staging/rtl8723bs/core/rtw_security.c:1287:5: error: stack frame size (2208) exceeds limit (2048) in 'rtw_aes_decrypt' [-Werror,-Wframe-larger-than]
1287 | u32 rtw_aes_decrypt(struct adapter *padapter, u8 *precvframe)

        |     ^

This comes from aes_decipher() being inlined in rtw_aes_decrypt(). Running the same build with CONFIG_FRAME_WARN=128 shows aes_cipher() also uses a decent amount of stack, just under the limit of 2048:

drivers/staging/rtl8723bs/core/rtw_security.c:864:19: warning: stack frame size (1952) exceeds limit (128) in 'aes_cipher' [-Wframe-larger-than]
864 | static signed int aes_cipher(u8 *key, uint hdrlen,
| ^

...

Quoted 5 lines

The memset() calls are just initializing these buffers to zero, so use '= {}' instead, which is used all over the kernel and does the exact same thing as memset() without the fortify checks, which drops the stack usage of these functions by a few hundred kilobytes.

I suspect you mean bytes....

David

Quoted 8 lines

drivers/staging/rtl8723bs/core/rtw_security.c:864:19: warning: stack frame size (1584) exceeds limit (128) in 'aes_cipher' [-Wframe-larger-than]
864 | static signed int aes_cipher(u8 *key, uint hdrlen,
| ^
drivers/staging/rtl8723bs/core/rtw_security.c:1271:5: warning: stack frame size (1456) exceeds limit (128) in 'rtw_aes_decrypt' [-Wframe-larger-than]
1271 | u32 rtw_aes_decrypt(struct adapter *padapter, u8 *precvframe)

        |     ^