Skip to contentSkip to conversation

arm64/mm: Elide TLB flush in certain pte protection transitions

PATCH RESEND Applied Dev Jain 4 participants 6 messages Oct 17, 2025 – Nov 13, 2025 arm64

lore
Share & tools
lore

Dev Jain

author
#0 lore raw

PATCH RESEND v2 1/1 arm64/mm: Elide TLB flush in certain pte protection transitions

Currently arm64 does an unconditional TLB flush in mprotect(). This is not required for some cases, for example, when changing from PROT_NONE to PROT_READ | PROT_WRITE (a real usecase - glibc malloc does this to emulate growing into the non-main heaps), and unsetting uffd-wp in a range.

Therefore, implement pte_needs_flush() for arm64, which is already implemented by some other arches as well.

Running a userspace program changing permissions back and forth between PROT_NONE and PROT_READ | PROT_WRITE, and measuring the average time taken for the none->rw transition, I get a reduction from 3.2 microseconds to 2.85 microseconds, giving a 12.3% improvement.

Reviewed-by Kefeng Wang <[email protected]>
Signed-off-by Dev Jain <[email protected]>

---
mm-selftests pass. Based on 6.18-rc1.

v1->v2:
- Drop PTE_PRESENT_INVALID and PTE_AF checks, use ptdesc_t instead of pteval_t, return !!diff (Ryan)

 arch/arm64/include/asm/tlbflush.h | 27 +++++++++++++++++++++++++++
 1 file changed, 27 insertions(+)
arch/arm64/include/asm/tlbflush.h +27 -0
index 18a5dc0c9a54..40df783ba09a 100644
@@ -524,6 +524,33 @@ static inline void arch_tlbbatch_add_pending(struct arch_tlbflush_unmap_batch *b
 {
 	__flush_tlb_range_nosync(mm, start, end, PAGE_SIZE, true, 3);
 }
+
+static inline bool __pte_flags_need_flush(ptdesc_t oldval, ptdesc_t newval)
+{
+	ptdesc_t diff = oldval ^ newval;
+
+	/* invalid to valid transition requires no flush */
+	if (!(oldval & PTE_VALID))
+		return false;
+
+	/* Transition in the SW bits requires no flush */
+	diff &= ~PTE_SWBITS_MASK;
+
+	return !!diff;
+}
+
+static inline bool pte_needs_flush(pte_t oldpte, pte_t newpte)
+{
+	return __pte_flags_need_flush(pte_val(oldpte), pte_val(newpte));
+}
+#define pte_needs_flush pte_needs_flush
+
+static inline bool huge_pmd_needs_flush(pmd_t oldpmd, pmd_t newpmd)
+{
+	return __pte_flags_need_flush(pmd_val(oldpmd), pmd_val(newpmd));
+}
+#define huge_pmd_needs_flush huge_pmd_needs_flush
+
 #endif
 
 #endif
Signature · 1 line
2.30.2
#1 lore raw
Quoted 5 lines · Dev Jain

Currently arm64 does an unconditional TLB flush in mprotect(). This is not required for some cases, for example, when changing from PROT_NONE to PROT_READ | PROT_WRITE (a real usecase - glibc malloc does this to emulate growing into the non-main heaps), and unsetting uffd-wp in a range.

I recall seeing this pattern frequently in multi-threaded programs. The stacks for the threads can be observed by running strace on a multi-threaded app:

mmap(NULL, 20480, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x7fbfcd14c000 mprotect(0x7fbfcd14d000, 16384, PROT_READ|PROT_WRITE) = 0

The stack guard page in the mmap region remains non-READ/WRITE, while the rest of the area is set to RW after initially being PROT_NONE.

Quoted 11 lines

Therefore, implement pte_needs_flush() for arm64, which is already implemented by some other arches as well.

Running a userspace program changing permissions back and forth between PROT_NONE and PROT_READ | PROT_WRITE, and measuring the average time taken for the none->rw transition, I get a reduction from 3.2 microseconds to 2.85 microseconds, giving a 12.3% improvement.

Reviewed-by Kefeng Wang <[email protected]>
Signed-off-by Dev Jain <[email protected]>

The patch seems to make a lot of sense to me, but I’m not an ARM architecture expert and cannot judge whether __pte_flags_need_flush() is correct. I’ll leave that to Will and Catalin.

Quoted 22 lines

---
mm-selftests pass. Based on 6.18-rc1.

v1->v2:
- Drop PTE_PRESENT_INVALID and PTE_AF checks, use ptdesc_t instead of pteval_t, return !!diff (Ryan)

 arch/arm64/include/asm/tlbflush.h | 27 +++++++++++++++++++++++++++
 1 file changed, 27 insertions(+)
arch/arm64/include/asm/tlbflush.h +4 -0
index 18a5dc0c9a54..40df783ba09a 100644
@@ -524,6 +524,33 @@ static inline void arch_tlbbatch_add_pending(struct arch_tlbflush_unmap_batch *b
 {
        __flush_tlb_range_nosync(mm, start, end, PAGE_SIZE, true, 3);
 }
+
+static inline bool __pte_flags_need_flush(ptdesc_t oldval, ptdesc_t newval)
+{
+       ptdesc_t diff = oldval ^ newval;

arch/arm64/include/asm/tlbflush.h static inline void arch_tlbbatch_add_pending(struct arch_tlbflush_unmap_batch *b · line 540

Quoted 10 lines
arch/arm64/include/asm/tlbflush.h +10 -0
+
+       /* invalid to valid transition requires no flush */
+       if (!(oldval & PTE_VALID))
+               return false;
+
+       /* Transition in the SW bits requires no flush */
+       diff &= ~PTE_SWBITS_MASK;
+
+       return !!diff;
+}

Thanks
Barry

author
#2 lore raw
In reply to Dev Jain

Gentle ping

Quoted 63 lines · Dev Jain

Currently arm64 does an unconditional TLB flush in mprotect(). This is not required for some cases, for example, when changing from PROT_NONE to PROT_READ | PROT_WRITE (a real usecase - glibc malloc does this to emulate growing into the non-main heaps), and unsetting uffd-wp in a range.

Therefore, implement pte_needs_flush() for arm64, which is already implemented by some other arches as well.

Running a userspace program changing permissions back and forth between PROT_NONE and PROT_READ | PROT_WRITE, and measuring the average time taken for the none->rw transition, I get a reduction from 3.2 microseconds to 2.85 microseconds, giving a 12.3% improvement.

Reviewed-by Kefeng Wang <[email protected]>
Signed-off-by Dev Jain <[email protected]>

---
mm-selftests pass. Based on 6.18-rc1.

v1->v2:
- Drop PTE_PRESENT_INVALID and PTE_AF checks, use ptdesc_t instead of pteval_t, return !!diff (Ryan)

  arch/arm64/include/asm/tlbflush.h | 27 +++++++++++++++++++++++++++
  1 file changed, 27 insertions(+)
arch/arm64/include/asm/tlbflush.h +27 -0
index 18a5dc0c9a54..40df783ba09a 100644
@@ -524,6 +524,33 @@ static inline void arch_tlbbatch_add_pending(struct arch_tlbflush_unmap_batch *b
  {
  	__flush_tlb_range_nosync(mm, start, end, PAGE_SIZE, true, 3);
  }
+
+static inline bool __pte_flags_need_flush(ptdesc_t oldval, ptdesc_t newval)
+{
+	ptdesc_t diff = oldval ^ newval;
+
+	/* invalid to valid transition requires no flush */
+	if (!(oldval & PTE_VALID))
+		return false;
+
+	/* Transition in the SW bits requires no flush */
+	diff &= ~PTE_SWBITS_MASK;
+
+	return !!diff;
+}
+
+static inline bool pte_needs_flush(pte_t oldpte, pte_t newpte)
+{
+	return __pte_flags_need_flush(pte_val(oldpte), pte_val(newpte));
+}
+#define pte_needs_flush pte_needs_flush
+
+static inline bool huge_pmd_needs_flush(pmd_t oldpmd, pmd_t newpmd)
+{
+	return __pte_flags_need_flush(pmd_val(oldpmd), pmd_val(newpmd));
+}
+#define huge_pmd_needs_flush huge_pmd_needs_flush
+
  #endif
 
  #endif
#3 lore raw
In reply to Dev Jain
Quoted 30 lines · Dev Jain

Currently arm64 does an unconditional TLB flush in mprotect(). This is not required for some cases, for example, when changing from PROT_NONE to PROT_READ | PROT_WRITE (a real usecase - glibc malloc does this to emulate growing into the non-main heaps), and unsetting uffd-wp in a range.

Therefore, implement pte_needs_flush() for arm64, which is already implemented by some other arches as well.

Running a userspace program changing permissions back and forth between PROT_NONE and PROT_READ | PROT_WRITE, and measuring the average time taken for the none->rw transition, I get a reduction from 3.2 microseconds to 2.85 microseconds, giving a 12.3% improvement.

Reviewed-by Kefeng Wang <[email protected]>
Signed-off-by Dev Jain <[email protected]>

---
mm-selftests pass. Based on 6.18-rc1.

v1->v2:
- Drop PTE_PRESENT_INVALID and PTE_AF checks, use ptdesc_t instead of pteval_t, return !!diff (Ryan)

 arch/arm64/include/asm/tlbflush.h | 27 +++++++++++++++++++++++++++
 1 file changed, 27 insertions(+)
arch/arm64/include/asm/tlbflush.h +0 -0
index 18a5dc0c9a54..40df783ba09a 100644

arch/arm64/include/asm/tlbflush.h static inline void arch_tlbbatch_add_pending(struct arch_tlbflush_unmap_batch *b · line 533

Quoted 10 lines
arch/arm64/include/asm/tlbflush.h +7 -0
@@ -524,6 +524,33 @@ static inline void arch_tlbbatch_add_pending(struct arch_tlbflush_unmap_batch *b
 {
 	__flush_tlb_range_nosync(mm, start, end, PAGE_SIZE, true, 3);
 }
+
+static inline bool __pte_flags_need_flush(ptdesc_t oldval, ptdesc_t newval)
+{
+	ptdesc_t diff = oldval ^ newval;
+
+	/* invalid to valid transition requires no flush */
+	if (!(oldval & PTE_VALID))

Using pte_valid() helper would be better.

	if (!pte_valid(oldval))
		return false;
Quoted 23 lines
arch/arm64/include/asm/tlbflush.h +20 -0
+		return false;
+
+	/* Transition in the SW bits requires no flush */
+	diff &= ~PTE_SWBITS_MASK;
+
+	return !!diff;
+}
+
+static inline bool pte_needs_flush(pte_t oldpte, pte_t newpte)
+{
+	return __pte_flags_need_flush(pte_val(oldpte), pte_val(newpte));
+}
+#define pte_needs_flush pte_needs_flush
+
+static inline bool huge_pmd_needs_flush(pmd_t oldpmd, pmd_t newpmd)
+{
+	return __pte_flags_need_flush(pmd_val(oldpmd), pmd_val(newpmd));
+}
+#define huge_pmd_needs_flush huge_pmd_needs_flush
+
 #endif
 
 #endif
author
#4 lore raw
Quoted 44 lines · Anshuman Khandual
Quoted 30 lines · Dev Jain

Currently arm64 does an unconditional TLB flush in mprotect(). This is not required for some cases, for example, when changing from PROT_NONE to PROT_READ | PROT_WRITE (a real usecase - glibc malloc does this to emulate growing into the non-main heaps), and unsetting uffd-wp in a range.

Therefore, implement pte_needs_flush() for arm64, which is already implemented by some other arches as well.

Running a userspace program changing permissions back and forth between PROT_NONE and PROT_READ | PROT_WRITE, and measuring the average time taken for the none->rw transition, I get a reduction from 3.2 microseconds to 2.85 microseconds, giving a 12.3% improvement.

Reviewed-by Kefeng Wang <[email protected]>
Signed-off-by Dev Jain <[email protected]>

---
mm-selftests pass. Based on 6.18-rc1.

v1->v2:
- Drop PTE_PRESENT_INVALID and PTE_AF checks, use ptdesc_t instead of pteval_t, return !!diff (Ryan)

  arch/arm64/include/asm/tlbflush.h | 27 +++++++++++++++++++++++++++
  1 file changed, 27 insertions(+)
arch/arm64/include/asm/tlbflush.h +0 -0
index 18a5dc0c9a54..40df783ba09a 100644

arch/arm64/include/asm/tlbflush.h static inline void arch_tlbbatch_add_pending(struct arch_tlbflush_unmap_batch *b · line 533

Quoted 10 lines
arch/arm64/include/asm/tlbflush.h +7 -0
@@ -524,6 +524,33 @@ static inline void arch_tlbbatch_add_pending(struct arch_tlbflush_unmap_batch *b
  {
  	__flush_tlb_range_nosync(mm, start, end, PAGE_SIZE, true, 3);
  }
+
+static inline bool __pte_flags_need_flush(ptdesc_t oldval, ptdesc_t newval)
+{
+	ptdesc_t diff = oldval ^ newval;
+
+	/* invalid to valid transition requires no flush */
+	if (!(oldval & PTE_VALID))

Using pte_valid() helper would be better.

Quoted 2 lines
	if (!pte_valid(oldval))
		return false;

Unfortunately, that would need asm/pgtable.h, which already includes asm/tlbflush.h, leading to circular include.

If I put all this code into asm/pgtable.h, then pte_valid() needs a pte_t or a pmd_t to work on, which forces us to break away from the nice abstraction of directing both pte_needs_flush and huge_pmd_needs_flush to __pte_flags_need_flush.

Quoted 23 lines
Quoted 23 lines
arch/arm64/include/asm/tlbflush.h +20 -0
+		return false;
+
+	/* Transition in the SW bits requires no flush */
+	diff &= ~PTE_SWBITS_MASK;
+
+	return !!diff;
+}
+
+static inline bool pte_needs_flush(pte_t oldpte, pte_t newpte)
+{
+	return __pte_flags_need_flush(pte_val(oldpte), pte_val(newpte));
+}
+#define pte_needs_flush pte_needs_flush
+
+static inline bool huge_pmd_needs_flush(pmd_t oldpmd, pmd_t newpmd)
+{
+	return __pte_flags_need_flush(pmd_val(oldpmd), pmd_val(newpmd));
+}
+#define huge_pmd_needs_flush huge_pmd_needs_flush
+
  #endif
 
  #endif
Applied
#5 lore raw
In reply to Dev Jain

From: Catalin Marinas <[email protected]>

Quoted 9 lines · Dev Jain

Currently arm64 does an unconditional TLB flush in mprotect(). This is not required for some cases, for example, when changing from PROT_NONE to PROT_READ | PROT_WRITE (a real usecase - glibc malloc does this to emulate growing into the non-main heaps), and unsetting uffd-wp in a range.

Therefore, implement pte_needs_flush() for arm64, which is already implemented by some other arches as well.

[...]

Applied to arm64 (for-next/misc), thanks!

[1/1] arm64/mm: Elide TLB flush in certain pte protection transitions

      https://git.kernel.org/arm64/c/c320dbb7c80d
Signature · 1 line
Catalin