Skip to contentSkip to conversation

mm,unmap: avoid flushing TLB in batch if PTE is inaccessible

PATCH In review Huang Ying 3 participants 10 messages Apr 10, 2023 – Apr 20, 2023

Subsystems (2)
lore
Share & tools
lore

Huang Ying

author
#0 lore raw

PATCH 1/1 mm,unmap: avoid flushing TLB in batch if PTE is inaccessible

0Day/LKP reported a performance regression for commit 7e12beb8ca2a ("migrate_pages: batch flushing TLB"). In the commit, the TLB flushing during page migration is batched. So, in try_to_migrate_one(), ptep_clear_flush() is replaced with set_tlb_ubc_flush_pending(). In further investigation, it is found that the TLB flushing can be avoided in ptep_clear_flush() if the PTE is inaccessible. In fact, we can optimize in similar way for the batched TLB flushing too to improve the performance.

So in this patch, we check pte_accessible() before set_tlb_ubc_flush_pending() in try_to_unmap/migrate_one(). Tests show that the benchmark score of the anon-cow-rand-mt test case of vm-scalability test suite can improve up to 2.1% with the patch on a Intel server machine. The TLB flushing IPI can reduce up to 44.3%.

Fixes 7e12beb8ca2a ("migrate_pages: batch flushing TLB")
Cc Nadav Amit <[email protected]>
Cc Mel Gorman <[email protected]>
Cc Hugh Dickins <[email protected]>
Cc Matthew Wilcox (Oracle) <[email protected]>
Cc David Hildenbrand <[email protected]>
---
 mm/rmap.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
mm/rmap.c +4 -2
index 8632e02661ac..3c7c43642d7c 100644
@@ -1582,7 +1582,8 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,
 				 */
 				pteval = ptep_get_and_clear(mm, address, pvmw.pte);
 
-				set_tlb_ubc_flush_pending(mm, pte_dirty(pteval));
+				if (pte_accessible(mm, pteval))
+					set_tlb_ubc_flush_pending(mm, pte_dirty(pteval));
 			} else {
 				pteval = ptep_clear_flush(vma, address, pvmw.pte);
 			}
@@ -1963,7 +1964,8 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma,
 				 */
 				pteval = ptep_get_and_clear(mm, address, pvmw.pte);
 
-				set_tlb_ubc_flush_pending(mm, pte_dirty(pteval));
+				if (pte_accessible(mm, pteval))
+					set_tlb_ubc_flush_pending(mm, pte_dirty(pteval));
 			} else {
 				pteval = ptep_clear_flush(vma, address, pvmw.pte);
 			}
Signature · 1 line
2.39.2
#1 lore raw
Quoted 13 lines · Huang Ying

0Day/LKP reported a performance regression for commit 7e12beb8ca2a ("migrate_pages: batch flushing TLB"). In the commit, the TLB flushing during page migration is batched. So, in try_to_migrate_one(), ptep_clear_flush() is replaced with set_tlb_ubc_flush_pending(). In further investigation, it is found that the TLB flushing can be avoided in ptep_clear_flush() if the PTE is inaccessible. In fact, we can optimize in similar way for the batched TLB flushing too to improve the performance.

Quoted 5 lines

So in this patch, we check pte_accessible() before set_tlb_ubc_flush_pending() in try_to_unmap/migrate_one(). Tests show that the benchmark score of the anon-cow-rand-mt test case of vm-scalability test suite can improve up to 2.1% with the patch on a Intel server machine. The TLB flushing IPI can reduce up to 44.3%.

LGTM.

I know it’s meaningless for x86 (but perhaps ARM would use this infra too): do we need smp_mb__after_atomic() after ptep_get_and_clear() and before pte_accessible()?

In addition, if this goes into stable (based on the Fixes tag), consider breaking it into 2 patches, when only one would be backported.

#2 lore raw

Hi, Amit,

Thank you very much for review!

Quoted 11 lines · Nadav Amit
Quoted 11 lines · Huang Ying

0Day/LKP reported a performance regression for commit 7e12beb8ca2a ("migrate_pages: batch flushing TLB"). In the commit, the TLB flushing during page migration is batched. So, in try_to_migrate_one(), ptep_clear_flush() is replaced with set_tlb_ubc_flush_pending(). In further investigation, it is found that the TLB flushing can be avoided in ptep_clear_flush() if the PTE is inaccessible. In fact, we can optimize in similar way for the batched TLB flushing too to improve the performance.

Quoted 7 lines
Quoted 5 lines

So in this patch, we check pte_accessible() before set_tlb_ubc_flush_pending() in try_to_unmap/migrate_one(). Tests show that the benchmark score of the anon-cow-rand-mt test case of vm-scalability test suite can improve up to 2.1% with the patch on a Intel server machine. The TLB flushing IPI can reduce up to 44.3%.

LGTM.

Thanks!

Quoted 3 lines

I know it’s meaningless for x86 (but perhaps ARM would use this infra too): do we need smp_mb__after_atomic() after ptep_get_and_clear() and before pte_accessible()?

Why do we need the memory barrier? IIUC, the PTL is locked, so PTE value will not be changed under us. Anything else?

Quoted 2 lines

In addition, if this goes into stable (based on the Fixes tag), consider breaking it into 2 patches, when only one would be backported.

The fixed commit (7e12beb8ca2a ("migrate_pages: batch flushing TLB")) is merged by v6.3-rc1. So this patch will only be backported to v6.3 and later. Is it OK?

Best Regards,
Huang, Ying

#3 lore raw
Quoted 31 lines · Ying

!! External Email

Hi, Amit,

Thank you very much for review!

Quoted 11 lines · Nadav Amit
Quoted 11 lines · Huang Ying

0Day/LKP reported a performance regression for commit 7e12beb8ca2a ("migrate_pages: batch flushing TLB"). In the commit, the TLB flushing during page migration is batched. So, in try_to_migrate_one(), ptep_clear_flush() is replaced with set_tlb_ubc_flush_pending(). In further investigation, it is found that the TLB flushing can be avoided in ptep_clear_flush() if the PTE is inaccessible. In fact, we can optimize in similar way for the batched TLB flushing too to improve the performance.

Quoted 7 lines
Quoted 5 lines

So in this patch, we check pte_accessible() before set_tlb_ubc_flush_pending() in try_to_unmap/migrate_one(). Tests show that the benchmark score of the anon-cow-rand-mt test case of vm-scalability test suite can improve up to 2.1% with the patch on a Intel server machine. The TLB flushing IPI can reduce up to 44.3%.

LGTM.

Quoted 8 lines

Thanks!

Quoted 3 lines

I know it’s meaningless for x86 (but perhaps ARM would use this infra too): do we need smp_mb__after_atomic() after ptep_get_and_clear() and before pte_accessible()?

Why do we need the memory barrier? IIUC, the PTL is locked, so PTE value will not be changed under us. Anything else?

I was thinking about the ordering with respect to atomic_read(&mm->tlb_flush_pending), which is not protected by the PTL. I guess you can correctly argue that because of other control-flow dependencies, the barrier is not necessary.

Quoted 7 lines
Quoted 3 lines

In addition, if this goes into stable (based on the Fixes tag), consider breaking it into 2 patches, when only one would be backported.

The fixed commit (7e12beb8ca2a ("migrate_pages: batch flushing TLB")) is merged by v6.3-rc1. So this patch will only be backported to v6.3 and later. Is it OK?

Of course. I wasn’t sure when the bug was introduced.

#4 lore raw
Quoted 35 lines · Nadav Amit
Quoted 35 lines · Ying

!! External Email

Hi, Amit,

Thank you very much for review!

Quoted 11 lines · Nadav Amit
Quoted 11 lines · Huang Ying

0Day/LKP reported a performance regression for commit 7e12beb8ca2a ("migrate_pages: batch flushing TLB"). In the commit, the TLB flushing during page migration is batched. So, in try_to_migrate_one(), ptep_clear_flush() is replaced with set_tlb_ubc_flush_pending(). In further investigation, it is found that the TLB flushing can be avoided in ptep_clear_flush() if the PTE is inaccessible. In fact, we can optimize in similar way for the batched TLB flushing too to improve the performance.

Quoted 7 lines
Quoted 5 lines

So in this patch, we check pte_accessible() before set_tlb_ubc_flush_pending() in try_to_unmap/migrate_one(). Tests show that the benchmark score of the anon-cow-rand-mt test case of vm-scalability test suite can improve up to 2.1% with the patch on a Intel server machine. The TLB flushing IPI can reduce up to 44.3%.

LGTM.

Thanks!

Quoted 3 lines

I know it’s meaningless for x86 (but perhaps ARM would use this infra too): do we need smp_mb__after_atomic() after ptep_get_and_clear() and before pte_accessible()?

Quoted 7 lines
Quoted 2 lines

Why do we need the memory barrier? IIUC, the PTL is locked, so PTE value will not be changed under us. Anything else?

I was thinking about the ordering with respect to atomic_read(&mm->tlb_flush_pending), which is not protected by the PTL. I guess you can correctly argue that because of other control-flow dependencies, the barrier is not necessary.

For ordering between ptep_get_and_clear() and atomic_read(&mm->tlb_flush_pending), I think PTL has provided the necessary protection already. The code path to write mm->tlb_flush_pending is,

tlb_gather_mmu
inc_tlb_flush_pending a)
lock PTL
change PTE b)
unlock PTL
tlb_finish_mmu
dec_tlb_flush_pending c)

While code path of try_to_unmap/migrate_one is,

lock PTL
read and change PTE d)
read mm->tlb_flush_pending e)
unlock PTL

Even if e) occurs before d), they cannot occur at the same time of b). Do I miss anything?

Best Regards,
Huang, Ying

[snip]

#5 lore raw
Quoted 66 lines · Ying

!! External Email

Quoted 35 lines · Nadav Amit
Quoted 35 lines · Ying

!! External Email

Hi, Amit,

Thank you very much for review!

Quoted 11 lines · Nadav Amit
Quoted 11 lines · Huang Ying

0Day/LKP reported a performance regression for commit 7e12beb8ca2a ("migrate_pages: batch flushing TLB"). In the commit, the TLB flushing during page migration is batched. So, in try_to_migrate_one(), ptep_clear_flush() is replaced with set_tlb_ubc_flush_pending(). In further investigation, it is found that the TLB flushing can be avoided in ptep_clear_flush() if the PTE is inaccessible. In fact, we can optimize in similar way for the batched TLB flushing too to improve the performance.

Quoted 7 lines
Quoted 5 lines

So in this patch, we check pte_accessible() before set_tlb_ubc_flush_pending() in try_to_unmap/migrate_one(). Tests show that the benchmark score of the anon-cow-rand-mt test case of vm-scalability test suite can improve up to 2.1% with the patch on a Intel server machine. The TLB flushing IPI can reduce up to 44.3%.

LGTM.

Thanks!

Quoted 3 lines

I know it’s meaningless for x86 (but perhaps ARM would use this infra too): do we need smp_mb__after_atomic() after ptep_get_and_clear() and before pte_accessible()?

Quoted 7 lines
Quoted 2 lines

Why do we need the memory barrier? IIUC, the PTL is locked, so PTE value will not be changed under us. Anything else?

I was thinking about the ordering with respect to atomic_read(&mm->tlb_flush_pending), which is not protected by the PTL. I guess you can correctly argue that because of other control-flow dependencies, the barrier is not necessary.

For ordering between ptep_get_and_clear() and atomic_read(&mm->tlb_flush_pending), I think PTL has provided the necessary protection already. The code path to write mm->tlb_flush_pending is,

tlb_gather_mmu
inc_tlb_flush_pending a)
lock PTL
change PTE b)
unlock PTL
tlb_finish_mmu
dec_tlb_flush_pending c)

While code path of try_to_unmap/migrate_one is,

Quoted 7 lines

lock PTL
read and change PTE d)
read mm->tlb_flush_pending e)
unlock PTL

Even if e) occurs before d), they cannot occur at the same time of b). Do I miss anything?

You didn’t miss anything. I went over the comment on inc_tlb_flush_pending() and you follow the scheme.

#6 lore raw
Quoted 69 lines · Nadav Amit
Quoted 69 lines · Ying

!! External Email

Quoted 35 lines · Nadav Amit
Quoted 35 lines · Ying

!! External Email

Hi, Amit,

Thank you very much for review!

Quoted 11 lines · Nadav Amit
Quoted 11 lines · Huang Ying

0Day/LKP reported a performance regression for commit 7e12beb8ca2a ("migrate_pages: batch flushing TLB"). In the commit, the TLB flushing during page migration is batched. So, in try_to_migrate_one(), ptep_clear_flush() is replaced with set_tlb_ubc_flush_pending(). In further investigation, it is found that the TLB flushing can be avoided in ptep_clear_flush() if the PTE is inaccessible. In fact, we can optimize in similar way for the batched TLB flushing too to improve the performance.

Quoted 7 lines
Quoted 5 lines

So in this patch, we check pte_accessible() before set_tlb_ubc_flush_pending() in try_to_unmap/migrate_one(). Tests show that the benchmark score of the anon-cow-rand-mt test case of vm-scalability test suite can improve up to 2.1% with the patch on a Intel server machine. The TLB flushing IPI can reduce up to 44.3%.

LGTM.

Thanks!

Quoted 3 lines

I know it’s meaningless for x86 (but perhaps ARM would use this infra too): do we need smp_mb__after_atomic() after ptep_get_and_clear() and before pte_accessible()?

Quoted 7 lines
Quoted 2 lines

Why do we need the memory barrier? IIUC, the PTL is locked, so PTE value will not be changed under us. Anything else?

I was thinking about the ordering with respect to atomic_read(&mm->tlb_flush_pending), which is not protected by the PTL. I guess you can correctly argue that because of other control-flow dependencies, the barrier is not necessary.

For ordering between ptep_get_and_clear() and atomic_read(&mm->tlb_flush_pending), I think PTL has provided the necessary protection already. The code path to write mm->tlb_flush_pending is,

tlb_gather_mmu
inc_tlb_flush_pending a)
lock PTL
change PTE b)
unlock PTL
tlb_finish_mmu
dec_tlb_flush_pending c)

While code path of try_to_unmap/migrate_one is,

lock PTL
read and change PTE d)
read mm->tlb_flush_pending e)
unlock PTL

Quoted 5 lines
Quoted 2 lines

Even if e) occurs before d), they cannot occur at the same time of b). Do I miss anything?

You didn’t miss anything. I went over the comment on inc_tlb_flush_pending() and you follow the scheme.

Thanks! Can I get your acked-by or reviewed-by for this patch?

Best Regards,
Huang, Ying

Reviewed-by
#7 lore raw
Quoted 12 lines · Ying

!! External Email

Quoted 3 lines · Nadav Amit

You didn’t miss anything. I went over the comment on inc_tlb_flush_pending() and you follow the scheme.

Thanks! Can I get your acked-by or reviewed-by for this patch?

Thanks,
Nadav

#8 lore raw
In reply to Huang Ying
Quoted 44 lines · Huang Ying

0Day/LKP reported a performance regression for commit 7e12beb8ca2a ("migrate_pages: batch flushing TLB"). In the commit, the TLB flushing during page migration is batched. So, in try_to_migrate_one(), ptep_clear_flush() is replaced with set_tlb_ubc_flush_pending(). In further investigation, it is found that the TLB flushing can be avoided in ptep_clear_flush() if the PTE is inaccessible. In fact, we can optimize in similar way for the batched TLB flushing too to improve the performance.

So in this patch, we check pte_accessible() before set_tlb_ubc_flush_pending() in try_to_unmap/migrate_one(). Tests show that the benchmark score of the anon-cow-rand-mt test case of vm-scalability test suite can improve up to 2.1% with the patch on a Intel server machine. The TLB flushing IPI can reduce up to 44.3%.

Fixes 7e12beb8ca2a ("migrate_pages: batch flushing TLB")
Cc Nadav Amit <[email protected]>
Cc Mel Gorman <[email protected]>
Cc Hugh Dickins <[email protected]>
Cc Matthew Wilcox (Oracle) <[email protected]>
Cc David Hildenbrand <[email protected]>
---
  mm/rmap.c | 6 ++++--
  1 file changed, 4 insertions(+), 2 deletions(-)
mm/rmap.c +2 -1
index 8632e02661ac..3c7c43642d7c 100644
@@ -1582,7 +1582,8 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,
  				 */
  				pteval = ptep_get_and_clear(mm, address, pvmw.pte);
 
-				set_tlb_ubc_flush_pending(mm, pte_dirty(pteval));
+				if (pte_accessible(mm, pteval))
+					set_tlb_ubc_flush_pending(mm, pte_dirty(pteval));
  			} else {
  				pteval = ptep_clear_flush(vma, address, pvmw.pte);
  			}

mm/rmap.c static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma, · line 1968

Quoted 6 lines
mm/rmap.c +2 -1
@@ -1963,7 +1964,8 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma,
  				 */
  				pteval = ptep_get_and_clear(mm, address, pvmw.pte);
 
-				set_tlb_ubc_flush_pending(mm, pte_dirty(pteval));
+				if (pte_accessible(mm, pteval))
+					set_tlb_ubc_flush_pending(mm, pte_dirty(pteval));

Just a advice, can you put pte_accessible() into set_tlb_ubc_flush_pendin(), just like ptep_clear_flush(); so that we no need to add  if (pte_accessible()) in per place

where call set_tlb_ubc_flush_pending();

Quoted 3 lines
mm/rmap.c +0 -0
  			} else {
  				pteval = ptep_clear_flush(vma, address, pvmw.pte);
  			}
#9 lore raw
Quoted 56 lines · haoxin

( 2023/4/10 H3:52, Huang Ying S:

Quoted 44 lines

0Day/LKP reported a performance regression for commit 7e12beb8ca2a ("migrate_pages: batch flushing TLB"). In the commit, the TLB flushing during page migration is batched. So, in try_to_migrate_one(), ptep_clear_flush() is replaced with set_tlb_ubc_flush_pending(). In further investigation, it is found that the TLB flushing can be avoided in ptep_clear_flush() if the PTE is inaccessible. In fact, we can optimize in similar way for the batched TLB flushing too to improve the performance.

So in this patch, we check pte_accessible() before set_tlb_ubc_flush_pending() in try_to_unmap/migrate_one(). Tests show that the benchmark score of the anon-cow-rand-mt test case of vm-scalability test suite can improve up to 2.1% with the patch on a Intel server machine. The TLB flushing IPI can reduce up to 44.3%.

Fixes 7e12beb8ca2a ("migrate_pages: batch flushing TLB")
Cc Nadav Amit <[email protected]>
Cc Mel Gorman <[email protected]>
Cc Hugh Dickins <[email protected]>
Cc Matthew Wilcox (Oracle) <[email protected]>
Cc David Hildenbrand <[email protected]>
---
  mm/rmap.c | 6 ++++--
  1 file changed, 4 insertions(+), 2 deletions(-)
mm/rmap.c +2 -0
index 8632e02661ac..3c7c43642d7c 100644
@@ -1582,7 +1582,8 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,
  				 */
  				pteval = ptep_get_and_clear(mm, address, pvmw.pte);
  -				set_tlb_ubc_flush_pending(mm,
 pte_dirty(pteval));
+				if (pte_accessible(mm, pteval))
+					set_tlb_ubc_flush_pending(mm, pte_dirty(pteval));
  			} else {
  				pteval = ptep_clear_flush(vma, address, pvmw.pte);
  			}

mm/rmap.c static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma, · line 1969

Quoted 6 lines
mm/rmap.c +2 -0
@@ -1963,7 +1964,8 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma,
  				 */
  				pteval = ptep_get_and_clear(mm, address, pvmw.pte);
  -				set_tlb_ubc_flush_pending(mm,
 pte_dirty(pteval));
+				if (pte_accessible(mm, pteval))
+					set_tlb_ubc_flush_pending(mm, pte_dirty(pteval));

Just a advice, can you put pte_accessible() into set_tlb_ubc_flush_pendin(), just like ptep_clear_flush(); so that we no need to add if (pte_accessible()) in per place

Quoted 1 line

where call set_tlb_ubc_flush_pending();

Sounds reasonable for me, will do that in the next version. Thanks for suggestion.

Best Regards,
Huang, Ying

Quoted 3 lines
Quoted 3 lines
mm/rmap.c +0 -0
  			} else {
  				pteval = ptep_clear_flush(vma, address, pvmw.pte);
  			}