Skip to contentSkip to conversation

arm64: mm: Fix the lockless page-table walk in show_pte()

PATCH Applied Karl Mehltretter 2 participants 2 messages Aug 28 – Sep 3 arm64

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 arm64: mm: Fix the lockless page-table walk in show_pte()

show_pte() walks page tables locklessly and can run with interrupts enabled. A concurrent teardown can free a table page while it is being walked. It can also clear a parent entry after show_pte() checked it; the regular pXd_offset() helpers then reread the cleared entry and can derive a bogus lower-level pointer and fault again.

Use the lockless offset helpers with the saved parent entries, as gup_fast() does, and pass the saved PMD to pte_offset_map().

For task page tables, arm64 selects MMU_GATHER_RCU_TABLE_FREE. Disable local interrupts around the walk to hold off RCU-deferred table frees and block the tlb_remove_table_sync_one() IPI until the walk is finished.

Place the IRQ guard after the header print. This does not make the output a consistent snapshot, but prevents the task page-table walk from dereferencing a released table page or deriving a pointer from a different parent value.

Fixes 1d18c47c735e ("arm64: MMU fault handling and page table management")
Assisted-by LLM

---
Changes in v2:
- Move the IRQ guard below the header print, as suggested by Will Deacon.
- Use saved parent entries for the lockless descent after testing exposed the separate race in the regular offset helpers.
- Retested with GCC, Clang and Sparse; expanded runtime coverage to deterministic and randomized races, KASAN, 4K/16K/64K, PREEMPT_RT, and 50 native/KVM Pi 400 boots.

Link to v1: https://lore.kernel.org/r/[email protected]

 arch/arm64/mm/fault.c | 12 ++++++++----
 1 file changed, 8 insertions(+), 4 deletions(-)
arch/arm64/mm/fault.c +8 -4
index 0b52557652be..75c3e463df2e 100644
@@ -16,6 +16,7 @@
 #include <linux/mm.h>
 #include <linux/hardirq.h>
 #include <linux/init.h>
+#include <linux/irqflags.h>
 #include <linux/kasan.h>
 #include <linux/kprobes.h>
 #include <linux/uaccess.h>
@@ -154,6 +155,9 @@ static void show_pte(unsigned long addr)
 	pr_alert("%s pgtable: %luk pages, %llu-bit VAs, pgdp=%016lx\n",
 		 mm == &init_mm ? "swapper" : "user", PAGE_SIZE / SZ_1K,
 		 vabits_actual, mm_to_pgd_phys(mm));
+
+	guard(irqsave)();
+
 	pgdp = pgd_offset(mm, addr);
 	pgd = READ_ONCE(*pgdp);
 	pr_alert("[%016lx] pgd=%016llx", addr, pgd_val(pgd));
@@ -167,25 +171,25 @@ static void show_pte(unsigned long addr)
 		if (pgd_none(pgd) || pgd_bad(pgd))
 			break;
 
-		p4dp = p4d_offset(pgdp, addr);
+		p4dp = p4d_offset_lockless(pgdp, pgd, addr);
 		p4d = READ_ONCE(*p4dp);
 		pr_cont(", p4d=%016llx", p4d_val(p4d));
 		if (p4d_none(p4d) || p4d_bad(p4d))
 			break;
 
-		pudp = pud_offset(p4dp, addr);
+		pudp = pud_offset_lockless(p4dp, p4d, addr);
 		pud = READ_ONCE(*pudp);
 		pr_cont(", pud=%016llx", pud_val(pud));
 		if (pud_none(pud) || pud_bad(pud))
 			break;
 
-		pmdp = pmd_offset(pudp, addr);
+		pmdp = pmd_offset_lockless(pudp, pud, addr);
 		pmd = READ_ONCE(*pmdp);
 		pr_cont(", pmd=%016llx", pmd_val(pmd));
 		if (pmd_none(pmd) || pmd_bad(pmd))
 			break;
 
-		ptep = pte_offset_map(pmdp, addr);
+		ptep = pte_offset_map(&pmd, addr);
 		if (!ptep)
 			break;
 

base-commit: 1b78070aaef63512688aebfbc82365ef9d6660f1

Signature · 1 line
2.39.5 (Apple Git-154)
maintainer Applied
#1 lore raw
Quoted 10 lines · Karl Mehltretter

show_pte() walks page tables locklessly and can run with interrupts enabled. A concurrent teardown can free a table page while it is being walked. It can also clear a parent entry after show_pte() checked it; the regular pXd_offset() helpers then reread the cleared entry and can derive a bogus lower-level pointer and fault again.

Use the lockless offset helpers with the saved parent entries, as gup_fast() does, and pass the saved PMD to pte_offset_map().

[...]

Applied to arm64 (for-next/fixes), thanks!

[1/1] arm64: mm: Fix the lockless page-table walk in show_pte()

      https://git.kernel.org/arm64/c/a77644d009de

Cheers,

Signature · 5 lines
Will

https://fixes.arm64.dev
https://next.arm64.dev
https://will.arm64.dev