arm64: mm: Optimise tlb flush logic where we have >4K granule

PATCH Superseded Steve Capper 2 participants 5 messages May 2, 2014 arm64

Share & tools

Steve Capper

author

2014-05-02 10:37 UTC #0 lore raw

PATCH v2 1/1 arm64: mm: Optimise tlb flush logic where we have >4K granule

The tlb maintainence functions: __cpu_flush_user_tlb_range and __cpu_flush_kern_tlb_range do not take into consideration the page granule when looping through the address range, and repeatedly flush tlb entries for the same page when operating with 64K pages.

This patch re-works the logic s.t. we instead advance the loop by
1 << (PAGE_SHIFT - 12), so avoid repeating ourselves.

Also the routines have been converted from assembler to static inline functions to aid with legibility and potential compiler optimisations.

Signed-off-by Steve Capper <[email protected]>

Acked-by Will Deacon <[email protected]>

---
Changed in V2: added the missing isb(.) to the kernel tlb flush.

---
 arch/arm64/include/asm/tlbflush.h | 31 ++++++++++++++---
 arch/arm64/mm/Makefile            |  2 +-
 arch/arm64/mm/tlb.S               | 71 ---------------------------------------
 3 files changed, 27 insertions(+), 77 deletions(-)
 delete mode 100644 arch/arm64/mm/tlb.S

arch/arm64/include/asm/tlbflush.h +26 -5

index 8b48203..941c615 100644
@@ -98,11 +98,32 @@ static inline void flush_tlb_page(struct vm_area_struct *vma,
 	dsb();
 }
 
-/*
- * Convert calls to our calling convention.
- */
-#define flush_tlb_range(vma,start,end)	__cpu_flush_user_tlb_range(start,end,vma)
-#define flush_tlb_kernel_range(s,e)	__cpu_flush_kern_tlb_range(s,e)
+static inline void flush_tlb_range(struct vm_area_struct *vma,
+					unsigned long start, unsigned long end)
+{
+	unsigned long asid = (unsigned long)ASID(vma->vm_mm) << 48;
+	unsigned long addr;
+	start = asid | (start >> 12);
+	end = asid | (end >> 12);
+
+	dsb(ishst);
+	for (addr = start; addr < end; addr += 1 << (PAGE_SHIFT - 12))
+		asm("tlbi vae1is, %0" : : "r"(addr));
+	dsb(ish);
+}
+
+static inline void flush_tlb_kernel_range(unsigned long start, unsigned long end)
+{
+	unsigned long addr;
+	start >>= 12;
+	end >>= 12;
+
+	dsb(ishst);
+	for (addr = start; addr < end; addr += 1 << (PAGE_SHIFT - 12))
+		asm("tlbi vaae1is, %0" : : "r"(addr));
+	dsb(ish);
+	isb();
+}
 
 /*
  * On AArch64, the cache coherency is handled via the set_pte_at() function.

arch/arm64/mm/Makefile +1 -1

index b51d364..3ecb56c 100644
@@ -1,5 +1,5 @@
 obj-y				:= dma-mapping.o extable.o fault.o init.o \
 				   cache.o copypage.o flush.o \
 				   ioremap.o mmap.o pgd.o mmu.o \
-				   context.o tlb.o proc.o
+				   context.o proc.o
 obj-$(CONFIG_HUGETLB_PAGE)	+= hugetlbpage.o

arch/arm64/mm/tlb.S deleted +0 ~~-71~~

71 lines not on this page · raw message

Signature · 1 line

1.8.1.4

Will Deacon

2014-05-02 11:26 UTC #1 lore raw

PATCH v2 1/1 arm64: mm: Optimise tlb flush logic where we have >4K granule

Quoted 8 lines · Steve Capper

The tlb maintainence functions: __cpu_flush_user_tlb_range and __cpu_flush_kern_tlb_range do not take into consideration the page granule when looping through the address range, and repeatedly flush tlb entries for the same page when operating with 64K pages.
This patch re-works the logic s.t. we instead advance the loop by
1 << (PAGE_SHIFT - 12), so avoid repeating ourselves.

Quoted 7 lines

Also the routines have been converted from assembler to static inline functions to aid with legibility and potential compiler optimisations.
Signed-off-by Steve Capper <[email protected]>
Acked-by Will Deacon <[email protected]>
---
Changed in V2: added the missing isb(.) to the kernel tlb flush.

Hold your horses ;)

You mentioned remapping kernel text rw/ro, but if you think about it, it's still executable for both of these, so the isb() isn't needed. Do we have a case for changing whether or not something is executable?

Will

Steve Capper

author

2014-05-02 12:31 UTC #2 lore raw

PATCH v2 1/1 arm64: mm: Optimise tlb flush logic where we have >4K granule

Quoted 12 lines · Will Deacon

Quoted 11 lines · Steve Capper
The tlb maintainence functions: __cpu_flush_user_tlb_range and __cpu_flush_kern_tlb_range do not take into consideration the page granule when looping through the address range, and repeatedly flush tlb entries for the same page when operating with 64K pages.
This patch re-works the logic s.t. we instead advance the loop by
1 << (PAGE_SHIFT - 12), so avoid repeating ourselves.
Also the routines have been converted from assembler to static inline functions to aid with legibility and potential compiler optimisations.

Quoted 6 lines

Quoted 4 lines
Signed-off-by Steve Capper <[email protected]>
Acked-by Will Deacon <[email protected]>
---
Changed in V2: added the missing isb(.) to the kernel tlb flush.
Hold your horses ;)

:-)

Quoted 4 lines

You mentioned remapping kernel text rw/ro, but if you think about it, it's still executable for both of these, so the isb() isn't needed. Do we have a case for changing whether or not something is executable?

I think module loading and unloading in future would likely need this. i.e. if we get stuff like set_memory_nx and friends for ARM64.

We've just had this added to ARM:
75374ad ARM: mm: Define set_memory_* functions for ARM

Cheers,

Signature · 1 line

Steve

Quoted 2 lines

Will

Will Deacon

2014-05-02 12:59 UTC #3 lore raw

PATCH v2 1/1 arm64: mm: Optimise tlb flush logic where we have >4K granule

Quoted 27 lines · Steve Capper

Quoted 12 lines · Will Deacon
Quoted 11 lines · Steve Capper
The tlb maintainence functions: __cpu_flush_user_tlb_range and __cpu_flush_kern_tlb_range do not take into consideration the page granule when looping through the address range, and repeatedly flush tlb entries for the same page when operating with 64K pages.
This patch re-works the logic s.t. we instead advance the loop by
1 << (PAGE_SHIFT - 12), so avoid repeating ourselves.
Also the routines have been converted from assembler to static inline functions to aid with legibility and potential compiler optimisations.
Quoted 6 lines
Quoted 4 lines
Signed-off-by Steve Capper <[email protected]>
Acked-by Will Deacon <[email protected]>
---
Changed in V2: added the missing isb(.) to the kernel tlb flush.
Hold your horses ;)
:-)
Quoted 4 lines
You mentioned remapping kernel text rw/ro, but if you think about it, it's still executable for both of these, so the isb() isn't needed. Do we have a case for changing whether or not something is executable?

Quoted 5 lines

I think module loading and unloading in future would likely need this. i.e. if we get stuff like set_memory_nx and friends for ARM64.
We've just had this added to ARM:
75374ad ARM: mm: Define set_memory_* functions for ARM

Ok, but if it's just set_memory_nx that needs this, I'd rather put the isb() there instead of penalising all kernel TLB flushes.

Will

Steve Capper

author

2014-05-02 13:27 UTC #4 lore raw

PATCH v2 1/1 arm64: mm: Optimise tlb flush logic where we have >4K granule

Quoted 28 lines · Will Deacon

Quoted 27 lines · Steve Capper
Quoted 12 lines · Will Deacon
Quoted 11 lines · Steve Capper
The tlb maintainence functions: __cpu_flush_user_tlb_range and __cpu_flush_kern_tlb_range do not take into consideration the page granule when looping through the address range, and repeatedly flush tlb entries for the same page when operating with 64K pages.
This patch re-works the logic s.t. we instead advance the loop by
1 << (PAGE_SHIFT - 12), so avoid repeating ourselves.
Also the routines have been converted from assembler to static inline functions to aid with legibility and potential compiler optimisations.
Quoted 6 lines
Quoted 4 lines
Signed-off-by Steve Capper <[email protected]>
Acked-by Will Deacon <[email protected]>
---
Changed in V2: added the missing isb(.) to the kernel tlb flush.
Hold your horses ;)
:-)
Quoted 4 lines
You mentioned remapping kernel text rw/ro, but if you think about it, it's still executable for both of these, so the isb() isn't needed. Do we have a case for changing whether or not something is executable?

Quoted 8 lines

Quoted 5 lines
I think module loading and unloading in future would likely need this. i.e. if we get stuff like set_memory_nx and friends for ARM64.
We've just had this added to ARM:
75374ad ARM: mm: Define set_memory_* functions for ARM
Ok, but if it's just set_memory_nx that needs this, I'd rather put the isb() there instead of penalising all kernel TLB flushes.

Sure that makes sense, I can't think of any other scenarios that would require this isb(.). I will document the fact that the isb has been removed in the commit log for V3. Also...

Laura,
I'll add you to CC for the patch, as you're working on the set_memory_* functions for ARM64.

Cheers,

Signature · 1 line

Steve

Quoted 2 lines

Will