Skip to contentSkip to conversation

arm64: mm: Make flush_tlb_fix_spurious_fault() a no-op

PATCH In review Will Deacon 2 participants 2 messages Sep 30, 2020 arm64

lore
Share & tools
lore

Will Deacon

author maintainer
#0 lore raw

PATCH 1/1 arm64: mm: Make flush_tlb_fix_spurious_fault() a no-op

Our use of broadcast TLB maintenance means that spurious page-faults that have been handled already by another CPU do not require additional TLB maintenance.

Make flush_tlb_fix_spurious_fault() a no-op and rely on the existing TLB invalidation instead. Add an explicit flush_tlb_page() when making a page dirty, as the TLB is permitted to cache the old read-only entry.

Cc Catalin Marinas <[email protected]>
Link https://lore.kernel.org/r/20200728092220.GA21800@willie-the-truck open here
Signed-off-by Will Deacon <[email protected]>
---
 arch/arm64/include/asm/pgtable.h | 8 ++++++++
 arch/arm64/mm/fault.c            | 4 +++-
 2 files changed, 11 insertions(+), 1 deletion(-)
arch/arm64/include/asm/pgtable.h +8 -0
index bc68da9f5706..02ad3105c14c 100644
@@ -50,6 +50,14 @@ extern void __pgd_error(const char *file, int line, unsigned long val);
 	__flush_tlb_range(vma, addr, end, PUD_SIZE, false, 1)
 #endif /* CONFIG_TRANSPARENT_HUGEPAGE */
 
+/*
+ * Outside of a few very special situations (e.g. hibernation), we always
+ * use broadcast TLB invalidation instructions, therefore a spurious page
+ * fault on one CPU which has been handled concurrently by another CPU
+ * does not need to perform additional invalidation.
+ */
+#define flush_tlb_fix_spurious_fault(vma, address) do { } while (0)
+
 /*
  * ZERO_PAGE is a global shared page that is always zero: used
  * for zero-mapped memory areas etc..
arch/arm64/mm/fault.c +3 -1
index f07333e86c2f..a696a7921da4 100644
@@ -218,7 +218,9 @@ int ptep_set_access_flags(struct vm_area_struct *vma,
 		pteval = cmpxchg_relaxed(&pte_val(*ptep), old_pteval, pteval);
 	} while (pteval != old_pteval);
 
-	flush_tlb_fix_spurious_fault(vma, address);
+	/* Invalidate a stale read-only entry */
+	if (dirty)
+		flush_tlb_page(vma, address);
 	return 1;
 }
Signature · 1 line
2.28.0.709.gb0816b6eb0-goog
maintainer Reviewed-by on v1 1/1
#1 lore raw

arch/arm64/mm/fault.c int ptep_set_access_flags(struct vm_area_struct *vma, · line 224

Quoted 13 lines · Will Deacon
arch/arm64/mm/fault.c +3 -1
index f07333e86c2f..a696a7921da4 100644
@@ -218,7 +218,9 @@ int ptep_set_access_flags(struct vm_area_struct *vma,
 		pteval = cmpxchg_relaxed(&pte_val(*ptep), old_pteval, pteval);
 	} while (pteval != old_pteval);
 
-	flush_tlb_fix_spurious_fault(vma, address);
+	/* Invalidate a stale read-only entry */
+	if (dirty)
+		flush_tlb_page(vma, address);
 	return 1;

In my proposal I had a pte_accessible(pte) check instead of dirty here since we may go for an old pte directly to a writable one and a TLBI wouldn't be needed. Not that it matters from a performance perspective.

Either way,

Reviewed-by Catalin Marinas <[email protected]>