Skip to contentSkip to conversation

raid6: add missing vzeroupper to AVX2 code

PATCH RESEND Applied Eric Biggers 2 participants 2 messages Aug 31

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH RESEND 1/1 raid6: add missing vzeroupper to AVX2 code

Since the AVX2 optimized RAID6 code uses YMM registers, execute vzeroupper before returning from it. This is needed to avoid degrading the performance of any later SSE code that may happen to be executed.

Fixes 2c935842bdb4 ("lib/raid6: Add AVX2 optimized gen_syndrome functions")
Fixes 7056741fd9fc ("lib/raid6: Add AVX2 optimized recovery functions")

---

This didn't get taken through the x86 tree. Andrew, it seems you're taking patches to lib/raid/. Can you apply this one?

 lib/raid/raid6/x86/avx2.c       | 6 ++++++
 lib/raid/raid6/x86/recov_avx2.c | 2 ++
 2 files changed, 8 insertions(+)
lib/raid/raid6/x86/avx2.c +6 -0
index 7d829c669ea7..3cc2fe7ac42c 100644
@@ -67,6 +67,7 @@ static void raid6_avx21_gen_syndrome(int disks, size_t bytes, void **ptrs)
 	}
 
 	asm volatile("sfence" : : : "memory");
+	asm volatile("vzeroupper");
 	kernel_fpu_end();
 }
 
@@ -115,6 +116,7 @@ static void raid6_avx21_xor_syndrome(int disks, int start, int stop,
 	}
 
 	asm volatile("sfence" : : : "memory");
+	asm volatile("vzeroupper");
 	kernel_fpu_end();
 }
 
@@ -175,6 +177,7 @@ static void raid6_avx22_gen_syndrome(int disks, size_t bytes, void **ptrs)
 	}
 
 	asm volatile("sfence" : : : "memory");
+	asm volatile("vzeroupper");
 	kernel_fpu_end();
 }
 
@@ -243,6 +246,7 @@ static void raid6_avx22_xor_syndrome(int disks, int start, int stop,
 	}
 
 	asm volatile("sfence" : : : "memory");
+	asm volatile("vzeroupper");
 	kernel_fpu_end();
 }
 
@@ -334,6 +338,7 @@ static void raid6_avx24_gen_syndrome(int disks, size_t bytes, void **ptrs)
 	}
 
 	asm volatile("sfence" : : : "memory");
+	asm volatile("vzeroupper");
 	kernel_fpu_end();
 }
 
@@ -444,6 +449,7 @@ static void raid6_avx24_xor_syndrome(int disks, int start, int stop,
 		asm volatile("vmovntdq %%ymm14,%0" : "=m" (q[d+96]));
 	}
 	asm volatile("sfence" : : : "memory");
+	asm volatile("vzeroupper");
 	kernel_fpu_end();
 }
 
lib/raid/raid6/x86/recov_avx2.c +2 -0
index a714a780a2d8..820871046e30 100644
@@ -176,6 +176,7 @@ static void raid6_2data_recov_avx2(int disks, size_t bytes, int faila,
 #endif
 	}
 
+	asm volatile("vzeroupper");
 	kernel_fpu_end();
 }
 
@@ -293,6 +294,7 @@ static void raid6_datap_recov_avx2(int disks, size_t bytes, int faila,
 #endif
 	}
 
+	asm volatile("vzeroupper");
 	kernel_fpu_end();
 }
 

base-commit: cee9395acd8043be0644b25c34bfa86623f2b935
prerequisite-patch-id: ebf41d9b9e92708245f6ae0657896ff3f8ef7d0d

Signature · 1 line
2.55.0
#1 lore raw
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:

Pre-existing issues:
- [High] The patch adds vzeroupper to AVX2 functions to avoid AVX-to-SSE transition penalties, but fails to apply the same necessary fix to the corresponding AVX-512 functions which suffer from the exact same issue. --

commit 51aca22e4c3be53f90a7f330d0ca103da02290ab
Author: Eric Biggers <[email protected]>

raid6: add missing vzeroupper to AVX2 code

This patch adds the vzeroupper instruction to the AVX2 optimized RAID6 code before returning. This prevents degrading the performance of subsequent legacy SSE code by avoiding hardware transition penalties.

lib/raid/raid6/x86/avx2.c static void raid6_avx21_gen_syndrome(int disks, size_t bytes, void **ptrs) · line 70

Quoted 9 lines
lib/raid/raid6/x86/avx2.c +1 -0
index 7d829c669ea79..3cc2fe7ac42c5 100644
@@ -67,6 +67,7 @@ static void raid6_avx21_gen_syndrome(int disks, size_t bytes, void **ptrs)
 	}
 
 	asm volatile("sfence" : : : "memory");
+	asm volatile("vzeroupper");

[Severity: High]
This isn't a bug introduced by this patch, but do the AVX-512 versions of this code need the same fix?

In lib/raid/raid6/x86/avx512.c:raid6_avx5121_gen_syndrome():

	asm volatile("sfence" : : : "memory");
	kernel_fpu_end();

And in lib/raid/raid6/x86/recov_avx512.c:raid6_2data_recov_avx512():

	}

	kernel_fpu_end();

The AVX-512 syndrome generation and recovery functions exit leaving ZMM registers dirty without calling vzeroupper. Without vzeroupper after using ZMM0-ZMM15 registers, the CPU remains in a dirty upper state. This flaw appears to be mirrored across all AVX-512 functions in these files.

If subsequent kernel code on the same CPU core executes legacy 128-bit SSE instructions before the state is cleared, could this cause severe hardware transition penalties that degrade overall crypto or networking performance and cause a performance regression?

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1