arm64: mm: Fix the lockless page-table walk in show_pte()
PATCH v2 1/1 arm64: mm: Fix the lockless page-table walk in show_pte()
show_pte() walks page tables locklessly and can run with interrupts enabled. A concurrent teardown can free a table page while it is being walked. It can also clear a parent entry after show_pte() checked it; the regular pXd_offset() helpers then reread the cleared entry and can derive a bogus lower-level pointer and fault again.
Use the lockless offset helpers with the saved parent entries, as gup_fast() does, and pass the saved PMD to pte_offset_map().
For task page tables, arm64 selects MMU_GATHER_RCU_TABLE_FREE. Disable local interrupts around the walk to hold off RCU-deferred table frees and block the tlb_remove_table_sync_one() IPI until the walk is finished.
Place the IRQ guard after the header print. This does not make the output a consistent snapshot, but prevents the task page-table walk from dereferencing a released table page or deriving a pointer from a different parent value.
---
Changes in v2:
- Move the IRQ guard below the header print, as suggested by Will Deacon.
- Use saved parent entries for the lockless descent after testing exposed the separate race in the regular offset helpers.
- Retested with GCC, Clang and Sparse; expanded runtime coverage to deterministic and randomized races, KASAN, 4K/16K/64K, PREEMPT_RT, and 50 native/KVM Pi 400 boots.
Link to v1: https://lore.kernel.org/r/[email protected]
arch/arm64/mm/fault.c | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-)
arch/arm64/mm/fault.c +8 -4
@@ -16,6 +16,7 @@#include <linux/mm.h> #include <linux/hardirq.h> #include <linux/init.h> +#include <linux/irqflags.h> #include <linux/kasan.h> #include <linux/kprobes.h> #include <linux/uaccess.h>
@@ -154,6 +155,9 @@ static void show_pte(unsigned long addr)pr_alert ("%s pgtable: %luk pages, %llu-bit VAs, pgdp=%016lx\n" , mm == &init_mm ?"swapper" :"user" ,PAGE_SIZE /SZ_1K , vabits_actual,mm_to_pgd_phys (mm)); + +guard (irqsave)(); + pgdp =pgd_offset (mm, addr); pgd =READ_ONCE (*pgdp);pr_alert ("[%016lx] pgd=%016llx" , addr,pgd_val (pgd));
@@ -167,25 +171,25 @@ static void show_pte(unsigned long addr)if (pgd_none (pgd) ||pgd_bad (pgd))break ; - p4dp =p4d_offset (pgdp, addr); + p4dp =p4d_offset_lockless (pgdp, pgd, addr); p4d =READ_ONCE (*p4dp);pr_cont (", p4d=%016llx" ,p4d_val (p4d));if (p4d_none (p4d) ||p4d_bad (p4d))break ; - pudp =pud_offset (p4dp, addr); + pudp =pud_offset_lockless (p4dp, p4d, addr); pud =READ_ONCE (*pudp);pr_cont (", pud=%016llx" ,pud_val (pud));if (pud_none (pud) ||pud_bad (pud))break ; - pmdp =pmd_offset (pudp, addr); + pmdp =pmd_offset_lockless (pudp, pud, addr); pmd =READ_ONCE (*pmdp);pr_cont (", pmd=%016llx" ,pmd_val (pmd));if (pmd_none (pmd) ||pmd_bad (pmd))break ; - ptep =pte_offset_map (pmdp, addr); + ptep =pte_offset_map (&pmd, addr);if (!ptep)break ;
base-commit: 1b78070aaef63512688aebfbc82365ef9d6660f1
Signature · 1 line
2.39.5 (Apple Git-154)
Quoted 10 lines · Karl Mehltretter
show_pte()walks page tables locklessly and can run with interrupts enabled. A concurrent teardown can free a table page while it is being walked. It can also clear a parent entry aftershow_pte()checked it; the regularpXd_offset()helpers then reread the cleared entry and can derive a bogus lower-level pointer and fault again.Use the lockless offset helpers with the saved parent entries, as
gup_fast()does, and pass the saved PMD topte_offset_map().[...]
Applied to arm64 (for-next/fixes), thanks!
[1/1] arm64: mm: Fix the lockless page-table walk in show_pte()
https://git.kernel.org/arm64/c/a77644d009de
Cheers,
Signature · 5 lines
Will https://fixes.arm64.dev https://next.arm64.dev https://will.arm64.dev