mm,unmap: avoid flushing TLB in batch if PTE is inaccessible
Subsystems (2)
PATCH 1/1 mm,unmap: avoid flushing TLB in batch if PTE is inaccessible
Hi, Andrew,
The version 1 of this patch was merged in mm-unstable branch. If you want to move that patch into mm-stable recently, it may be better to update that patch with this new version firstly. If you want to do that after v6.4-rc1, I will rebase this patch and resend it after v6.4-rc1 is released.
Hi, Amit,
The patch has been changed based on comments from Xin. I keep your "reviewed-by" because I think the change is trivial. But if you think it's inappropriate, I will change that.
Best Regards,
Huang, Ying
------------------------------->8------------------------------------------ 0Day/LKP reported a performance regression for commit 7e12beb8ca2a ("migrate_pages: batch flushing TLB"). In the commit, the TLB flushing during page migration is batched. So, in try_to_migrate_one(), ptep_clear_flush() is replaced with set_tlb_ubc_flush_pending(). In further investigation, it is found that the TLB flushing can be avoided in ptep_clear_flush() if the PTE is inaccessible. In fact, we can optimize in similar way for the batched TLB flushing too to improve the performance.
So in this patch, we check pte_accessible() before set_tlb_ubc_flush_pending() in try_to_unmap/migrate_one(). Tests show that the benchmark score of the anon-cow-rand-mt test case of vm-scalability test suite can improve up to 2.1% with the patch on a Intel server machine. The TLB flushing IPI can reduce up to 44.3%.
--- mm/rmap.c | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-)
mm/rmap.c +8 -4
@@ -641,10 +641,14 @@ void try_to_unmap_flush_dirty(void)#define TLB_FLUSH_BATCH_PENDING_LARGE \ (TLB_FLUSH_BATCH_PENDING_MASK / 2) -static void set_tlb_ubc_flush_pending (struct mm_struct *mm,bool writable) +static void set_tlb_ubc_flush_pending (struct mm_struct *mm,pte_t pteval) {struct tlbflush_unmap_batch *tlb_ubc = ¤t->tlb_ubc;int batch, nbatch; +bool writable =pte_dirty (pteval); + +if (!pte_accessible (mm, pteval)) +return ;arch_tlbbatch_add_mm (&tlb_ubc->arch, mm); tlb_ubc->flush_required = true;
@@ -731,7 +735,7 @@ void flush_tlb_batched_pending(struct mm_struct *mm) } }#else -static void set_tlb_ubc_flush_pending (struct mm_struct *mm,bool writable) +static void set_tlb_ubc_flush_pending (struct mm_struct *mm,pte_t pteval) { }
@@ -1582,7 +1586,7 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,*/ pteval =ptep_get_and_clear (mm, address, pvmw.pte); -set_tlb_ubc_flush_pending (mm,pte_dirty (pteval)); +set_tlb_ubc_flush_pending (mm, pteval); }else { pteval =ptep_clear_flush (vma, address, pvmw.pte); }
@@ -1963,7 +1967,7 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma,*/ pteval =ptep_get_and_clear (mm, address, pvmw.pte); -set_tlb_ubc_flush_pending (mm,pte_dirty (pteval)); +set_tlb_ubc_flush_pending (mm, pteval); }else { pteval =ptep_clear_flush (vma, address, pvmw.pte); }
Signature · 1 line
2.39.2
Quoted 55 lines · Huang Ying
Hi, Andrew,
The version 1 of this patch was merged in mm-unstable branch. If you want to move that patch into mm-stable recently, it may be better to update that patch with this new version firstly. If you want to do that after v6.4-rc1, I will rebase this patch and resend it after v6.4-rc1 is released.
Hi, Amit,
The patch has been changed based on comments from Xin. I keep your "reviewed-by" because I think the change is trivial. But if you think it's inappropriate, I will change that.
Best Regards,
Huang, Ying------------------------------->8------------------------------------------ 0Day/LKP reported a performance regression for commit
7e12beb8ca2a("migrate_pages: batch flushing TLB"). In the commit, the TLB flushing during page migration is batched. So, intry_to_migrate_one(),ptep_clear_flush()is replaced withset_tlb_ubc_flush_pending(). In further investigation, it is found that the TLB flushing can be avoided inptep_clear_flush()if the PTE is inaccessible. In fact, we can optimize in similar way for the batched TLB flushing too to improve the performance.So in this patch, we check
pte_accessible()beforeset_tlb_ubc_flush_pending()in try_to_unmap/migrate_one(). Tests show that the benchmark score of the anon-cow-rand-mt test case of vm-scalability test suite can improve up to 2.1% with the patch on a Intel server machine. The TLB flushing IPI can reduce up to 44.3%.Fixes 7e12beb8ca2a ("migrate_pages: batch flushing TLB")Reported-by kernel test robot <[email protected]>Signed-off-by "Huang, Ying" <[email protected]>Reviewed-by Nadav Amit <[email protected]>Cc haoxin <[email protected]>Cc Mel Gorman <[email protected]>Cc Hugh Dickins <[email protected]>Cc Matthew Wilcox (Oracle) <[email protected]>Cc David Hildenbrand <[email protected]>--- mm/rmap.c | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-)mm/rmap.c +0
-0@@ -641,10 +641,14 @@ void try_to_unmap_flush_dirty(void)#define TLB_FLUSH_BATCH_PENDING_LARGE \ (TLB_FLUSH_BATCH_PENDING_MASK / 2)
mm/rmap.c void try_to_unmap_flush_dirty(void) · line 651
Quoted 10 lines
mm/rmap.c +5
-1-static void set_tlb_ubc_flush_pending (struct mm_struct *mm,bool writable) +static void set_tlb_ubc_flush_pending (struct mm_struct *mm,pte_t pteval) {struct tlbflush_unmap_batch *tlb_ubc = ¤t->tlb_ubc;int batch, nbatch; +bool writable =pte_dirty (pteval); + +if (!pte_accessible (mm, pteval)) +return ;
LGTM
Quoted 30 lines
mm/rmap.c +3
-3arch_tlbbatch_add_mm (&tlb_ubc->arch, mm); tlb_ubc->flush_required = true;@@ -731,7 +735,7 @@ void flush_tlb_batched_pending(struct mm_struct *mm) } }#else -static void set_tlb_ubc_flush_pending (struct mm_struct *mm,bool writable) +static void set_tlb_ubc_flush_pending (struct mm_struct *mm,pte_t pteval) { }@@ -1582,7 +1586,7 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,*/ pteval =ptep_get_and_clear (mm, address, pvmw.pte); -set_tlb_ubc_flush_pending (mm,pte_dirty (pteval)); +set_tlb_ubc_flush_pending (mm, pteval); }else { pteval =ptep_clear_flush (vma, address, pvmw.pte); }@@ -1963,7 +1967,7 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma,*/ pteval =ptep_get_and_clear (mm, address, pvmw.pte); -set_tlb_ubc_flush_pending (mm,pte_dirty (pteval)); +set_tlb_ubc_flush_pending (mm, pteval); }else { pteval =ptep_clear_flush (vma, address, pvmw.pte); }