Skip to contentSkip to conversation

raid6: add missing vzeroupper to AVX-512 code

PATCH RESEND Applied Eric Biggers 2 participants 3 messages Aug 31

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH RESEND 1/1 raid6: add missing vzeroupper to AVX-512 code

Since the AVX-512 optimized RAID6 code uses ZMM registers, execute vzeroupper before returning from it. This is needed to avoid degrading the performance of any later SSE code that may happen to be executed.

Fixes e0a491c12968 ("lib/raid6: Add AVX512 optimized gen_syndrome functions")
Fixes 13c520b2993c ("lib/raid6: Add AVX512 optimized recovery functions")

---

This didn't get taken through the x86 tree. Andrew, it seems you're taking patches to lib/raid/. Can you apply this one?

 lib/raid/raid6/x86/avx512.c       | 6 ++++++
 lib/raid/raid6/x86/recov_avx512.c | 2 ++
 2 files changed, 8 insertions(+)
lib/raid/raid6/x86/avx512.c +6 -0
index e671eb5bde63..772bfc4af6df 100644
@@ -78,6 +78,7 @@ static void raid6_avx5121_gen_syndrome(int disks, size_t bytes, void **ptrs)
 	}
 
 	asm volatile("sfence" : : : "memory");
+	asm volatile("vzeroupper");
 	kernel_fpu_end();
 }
 
@@ -137,6 +138,7 @@ static void raid6_avx5121_xor_syndrome(int disks, int start, int stop,
 	}
 
 	asm volatile("sfence" : : : "memory");
+	asm volatile("vzeroupper");
 	kernel_fpu_end();
 }
 
@@ -208,6 +210,7 @@ static void raid6_avx5122_gen_syndrome(int disks, size_t bytes, void **ptrs)
 	}
 
 	asm volatile("sfence" : : : "memory");
+	asm volatile("vzeroupper");
 	kernel_fpu_end();
 }
 
@@ -292,6 +295,7 @@ static void raid6_avx5122_xor_syndrome(int disks, int start, int stop,
 	}
 
 	asm volatile("sfence" : : : "memory");
+	asm volatile("vzeroupper");
 	kernel_fpu_end();
 }
 
@@ -396,6 +400,7 @@ static void raid6_avx5124_gen_syndrome(int disks, size_t bytes, void **ptrs)
 	}
 
 	asm volatile("sfence" : : : "memory");
+	asm volatile("vzeroupper");
 	kernel_fpu_end();
 }
 
@@ -529,6 +534,7 @@ static void raid6_avx5124_xor_syndrome(int disks, int start, int stop,
 			       "m" (q[d+128]), "m" (q[d+192]));
 	}
 	asm volatile("sfence" : : : "memory");
+	asm volatile("vzeroupper");
 	kernel_fpu_end();
 }
 const struct raid6_calls raid6_avx512x4 = {
lib/raid/raid6/x86/recov_avx512.c +2 -0
index ec72d5a30c01..299a3f044d61 100644
@@ -211,6 +211,7 @@ static void raid6_2data_recov_avx512(int disks, size_t bytes, int faila,
 #endif
 	}
 
+	asm volatile("vzeroupper");
 	kernel_fpu_end();
 }
 
@@ -353,6 +354,7 @@ static void raid6_datap_recov_avx512(int disks, size_t bytes, int faila,
 #endif
 	}
 
+	asm volatile("vzeroupper");
 	kernel_fpu_end();
 }
 

base-commit: cee9395acd8043be0644b25c34bfa86623f2b935
prerequisite-patch-id: ebf41d9b9e92708245f6ae0657896ff3f8ef7d0d
prerequisite-patch-id: f85f9c7e8bdd6fcfd394edb80f2263d7189e138d

Signature · 1 line
2.55.0
#1 lore raw
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:

Pre-existing issues:
- [High] Missing vzeroupper in AVX and AVX2 RAID6 and XOR implementations
- [High] Missing vzeroupper in ARIA GFNI AVX512 crypto routines --

commit c86f8a80ee519a29f4ac71611d625cca2868db38
Author: Eric Biggers <[email protected]>
raid6: add missing vzeroupper to AVX-512 code

This commit adds vzeroupper to the AVX-512 optimized RAID6 routines before returning, preventing performance degradation of subsequent SSE code.

lib/raid/raid6/x86/avx512.c static void raid6_avx5121_gen_syndrome(int disks, size_t bytes, void **ptrs) · line 83

Quoted 10 lines
lib/raid/raid6/x86/avx512.c +1 -0
@@ -78,6 +78,7 @@ static void raid6_avx5121_gen_syndrome(int disks, size_t bytes, void **ptrs)
 	}
 
 	asm volatile("sfence" : : : "memory");
+	asm volatile("vzeroupper");
 	kernel_fpu_end();
 }

[Severity: High]
This is a pre-existing issue, but does this incomplete fix also need to address the conceptually identical AVX/AVX2 implementations?

For example, in lib/raid/raid6/x86/avx2.c:raid6_avx21_gen_syndrome():

	asm volatile("sfence" : : : "memory");
	kernel_fpu_end();

And similarly in lib/raid/raid6/x86/recov_avx2.c:raid6_2data_recov_avx2() and lib/raid/xor/x86/xor-avx.c:xor_gen_avx(). These routines use YMM registers but return to other kernel paths without clearing the upper state.

[Severity: High]
This isn't a bug introduced by this patch, but are there other areas in the kernel that need a similar fix? For example, in the crypto subsystem, the ARIA GFNI AVX512 routines dirty ZMM registers but omit the vzeroupper.

arch/x86/crypto/aria_gfni_avx512_glue.c:aria_avx512_ctr_encrypt():

			kernel_fpu_begin();
			aria_ops.aria_ctr_crypt_64way(ctx, dst, src,
						      &req_ctx->keystream[0],
						      walk.iv);
			kernel_fpu_end();

And in arch/x86/crypto/aria-gfni-avx512-asm_64.S:aria_gfni_avx512_encrypt_64way(), the assembly implementation returns without issuing a vzeroupper to clear the upper state of the ZMM registers.

[ ... ]

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
#2 lore raw
Quoted 5 lines · On Mon, Aug 31, 2026 at 10:07:54PM +0000, [email protected] wrote:

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:

Pre-existing issues:
- [High] Missing vzeroupper in AVX and AVX2 RAID6 and XOR implementations
- [High] Missing vzeroupper in ARIA GFNI AVX512 crypto routines

Already addressed by other patches I sent.

- Eric