x86/mm: Fix check/use ordering in switch_mm_irqs_off()

From: Stephen Dolan <[email protected]>

This is a stable-specific fix. There is no single upstream commit to cherry-pick; the equivalent mainline fix is commit 83b0177a6c48 ("x86/mm: Fix SMP ordering in switch_mm_irqs_off()"), which is in v6.18 but does not apply to these trees because the surrounding code was refactored.

The backports of commit fea4e317f9e7 ("x86/mm: Eliminate window where TLB flushes may be inadvertently skipped") -- the fix for CVE-2025-37964 -- to the 6.1, 6.6 and 6.12 trees ended up with two code blocks in the wrong order relative to mainline. The fix depends on setting cpu_tlbstate.loaded_mm to LOADED_MM_SWITCHING *before* reading tlb_gen, so that a concurrent TLB shootdown either sees LOADED_MM_SWITCHING and sends the IPI, or the switching CPU sees the updated tlb_gen. In the stable trees the write of LOADED_MM_SWITCHING was placed after the tlb_gen read, so the race window fea4e317f9e7 was meant to close is still open and CVE-2025-37964 is unfixed there:
a process can be left running with stale TLB entries, which typically manifests as rare, hard-to-bisect memory corruption or segfaults.

This has been confirmed by several independent parties:

- reproduced on 6.1.y/6.6.y/6.12.y with the test program in [1], and confirmed fixed by this patch
- Seth Forshee saw segfaults on 6.12.y within ~30 minutes of running a test workload; with this patch it ran 18 hours cleanly [2]
- Greg Thelen reports 6.6.y- and 6.12.y-based test failures fixed by this patch [3]

Dave Hansen acked the patch [4] and has no objection to it going into stable [5].

The patch below is against 6.12.y; the identical change applies to 6.1.y and 6.6.y. (The cpumask_test_cpu()/smp_mb() portion of the mainline fix is not needed here because commit 209954cbc7d0 ("x86/mm/tlb: Update mm_cpumask lazily") was never backported to these trees.)

[1] https://lore.kernel.org/lkml/CAHDw0oGd0B4=uuv8NGqbUQ_ZVmSheU2bN70e4QhFXWvuAZdt2w@mail.gmail.com/
[2] https://lore.kernel.org/lkml/aZYWXe739XUJrBld@do-x1carbon/
[3] https://lore.kernel.org/lkml/CAHH2K0brx9omC9QYyB6Lio3t_1Lf8v=VaFoiaG23UgQ-aec89Q@mail.gmail.com/
[4] https://lore.kernel.org/lkml/[email protected]/
[5] https://lore.kernel.org/lkml/[email protected]/

[Lee: The v5.15.y backport of fea4e317f9e7 (12f703811af0) placed the
LOADED_MM_SWITCHING write after the tlb_gen read too, so v5.15.y suffers from the same issue. Applies cleanly]

---
 arch/x86/mm/tlb.c | 16 ++++++++--------
 1 file changed, 8 insertions(+), 8 deletions(-)

index 11a43d373bae..f6e541c24ae2 100644
@@ -596,6 +596,14 @@ void switch_mm_irqs_off(struct mm_struct *prev, struct mm_struct *next,
 		 */
 		cond_mitigation(tsk);
 
+		/*
+		 * Indicate that CR3 is about to change. nmi_uaccess_okay()
+		 * and others are sensitive to the window where mm_cpumask(),
+		 * CR3 and cpu_tlbstate.loaded_mm are not all in sync.
+		 */
+		this_cpu_write(cpu_tlbstate.loaded_mm, LOADED_MM_SWITCHING);
+		barrier();
+
 		/*
 		 * Stop remote flushes for the previous mm.
 		 * Skip kernel threads; we never send init_mm TLB flushing IPIs,

@@ -615,14 +623,6 @@ void switch_mm_irqs_off(struct mm_struct *prev, struct mm_struct *next,
 		next_tlb_gen = atomic64_read(&next->context.tlb_gen);
 
 		choose_new_asid(next, next_tlb_gen, &new_asid, &need_flush);
-
-		/*
-		 * Indicate that CR3 is about to change. nmi_uaccess_okay()
-		 * and others are sensitive to the window where mm_cpumask(),
-		 * CR3 and cpu_tlbstate.loaded_mm are not all in sync.
- 		 */
-		this_cpu_write(cpu_tlbstate.loaded_mm, LOADED_MM_SWITCHING);
-		barrier();
 	}
 
 	if (need_flush) {

2.56.0.rc1.315.gc6ed9934b7-goog

Queued for 5.15, thanks.

Thanks,
Sasha