arm64: tlbflush: Ensure start/end of address range are aligned to stride
PATCH 1/1 arm64: tlbflush: Ensure start/end of address range are aligned to stride
Since commit 3d65b6bbc01e ("arm64: tlbi: Set MAX_TLBI_OPS to PTRS_PER_PTE"), we resort to per-ASID invalidation when attempting to perform more than PTRS_PER_PTE invalidation instructions in a single call to __flush_tlb_range(). Whilst this is beneficial, the mmu_gather code does not ensure that the end address of the range is rounded-up to the stride when freeing intermediate page tables in pXX_free_tlb(), which defeats our range checking.
Align the bounds passed into __flush_tlb_range().
--- arch/arm64/include/asm/tlbflush.h | 3 +++ 1 file changed, 3 insertions(+)
arch/arm64/include/asm/tlbflush.h +3 -0
@@ -195,6 +195,9 @@ static inline void __flush_tlb_range(struct vm_area_struct *vma,unsigned long asid =ASID (vma->vm_mm);unsigned long addr; + start =round_down (start, stride); + end =round_up (end, stride); +if ((end - start) >= (MAX_TLBI_OPS * stride)) {flush_tlb_mm (vma->vm_mm);return ;
Signature · 7 lines
2.11.0 _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Hi Will,
Quoted 8 lines · Will Deacon
Since commit
3d65b6bbc01e("arm64: tlbi: SetMAX_TLBI_OPStoPTRS_PER_PTE"), we resort to per-ASID invalidation when attempting to perform more thanPTRS_PER_PTEinvalidation instructions in a single call to__flush_tlb_range(). Whilst this is beneficial, themmu_gathercode does not ensure that the end address of the range is rounded-up to the stride when freeing intermediate page tables inpXX_free_tlb(), which defeats our range checking.
Quoted 5 lines
Align the bounds passed into
__flush_tlb_range().Cc Catalin Marinas <[email protected]>Cc Peter Zijlstra <[email protected]>Reported-by Hanjun Guo <[email protected]>
Thanks for the patch,I will test the patch tomorrow my local time as it's late here, and will update you when I get the results.
Thanks
Hanjun
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Quoted 11 lines · Hanjun Guo
Hi Will,
Quoted 8 lines · Will Deacon
Since commit
3d65b6bbc01e("arm64: tlbi: SetMAX_TLBI_OPStoPTRS_PER_PTE"), we resort to per-ASID invalidation when attempting to perform more thanPTRS_PER_PTEinvalidation instructions in a single call to__flush_tlb_range(). Whilst this is beneficial, themmu_gathercode does not ensure that the end address of the range is rounded-up to the stride when freeing intermediate page tables inpXX_free_tlb(), which defeats our range checking.
Quoted 8 lines
Quoted 5 lines
Align the bounds passed into
__flush_tlb_range().Cc Catalin Marinas <[email protected]>Cc Peter Zijlstra <[email protected]>Reported-by Hanjun Guo <[email protected]>Thanks for the patch,I will test the patch tomorrow my local time as it's late here, and will update you when I get the results.
I tested this patch on top of 5.2-rc1, and on the Kunpeng920 ARM64 server platform, with test case I reported before, I can see about 100% speedup for munmap() (from about 47us to 25us), that's great!
Thanks
Hanjun
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel