Skip to contentSkip to conversation

mm/khugepaged: move tlb_remove_table_sync_one out from under PTL

PATCH In review Lance Yang 2 participants 7 messages Jan 15 – Jan 20

Subsystems (3)
lore
Share & tools
lore

Lance Yang

author reviewer
#0 lore raw

PATCH 1/1 mm/khugepaged: move tlb_remove_table_sync_one out from under PTL

From: Lance Yang <[email protected]>

tlb_remove_table_sync_one() sends IPIs to all CPUs and waits for them, which we really don't want to do while holding PTL.

Just move the call to after we release PTL, and drop the macro wrapper while we're at it.

---
 include/linux/pgtable.h | 4 ----
 mm/khugepaged.c         | 5 +++--
 2 files changed, 3 insertions(+), 6 deletions(-)
include/linux/pgtable.h +0 -4
index eb8aacba3698..fb04ed22052c 100644
@@ -755,7 +755,6 @@ static inline pmd_t pmdp_get_lockless(pmd_t *pmdp)
 	return pmd;
 }
 #define pmdp_get_lockless pmdp_get_lockless
-#define pmdp_get_lockless_sync() tlb_remove_table_sync_one()
 #endif /* CONFIG_PGTABLE_LEVELS > 2 */
 #endif /* CONFIG_GUP_GET_PXX_LOW_HIGH */
 
@@ -774,9 +773,6 @@ static inline pmd_t pmdp_get_lockless(pmd_t *pmdp)
 {
 	return pmdp_get(pmdp);
 }
-static inline void pmdp_get_lockless_sync(void)
-{
-}
 #endif
 
 #ifdef CONFIG_TRANSPARENT_HUGEPAGE
mm/khugepaged.c +3 -2
index 9f790ec34400..0a6cebf880e0 100644
@@ -1664,10 +1664,10 @@ static enum scan_result try_collapse_pte_mapped_thp(struct mm_struct *mm, unsign
 		}
 	}
 	pgt_pmd = pmdp_collapse_flush(vma, haddr, pmd);
-	pmdp_get_lockless_sync();
 	pte_unmap_unlock(start_pte, ptl);
 	if (ptl != pml)
 		spin_unlock(pml);
+	tlb_remove_table_sync_one();
 
 	mmu_notifier_invalidate_range_end(&range);
 
@@ -1818,7 +1818,6 @@ static void retract_page_tables(struct address_space *mapping, pgoff_t pgoff)
 		 */
 		if (likely(file_backed_vma_is_retractable(vma))) {
 			pgt_pmd = pmdp_collapse_flush(vma, addr, pmd);
-			pmdp_get_lockless_sync();
 			success = true;
 		}
 
@@ -1826,6 +1825,8 @@ static void retract_page_tables(struct address_space *mapping, pgoff_t pgoff)
 			spin_unlock(ptl);
 drop_pml:
 		spin_unlock(pml);
+		if (success)
+			tlb_remove_table_sync_one();
 
 		mmu_notifier_invalidate_range_end(&range);
 
Signature · 1 line
2.49.0
reviewer
#1 lore raw

Hi Lance,

Quoted 4 lines · Lance Yang

From: Lance Yang <[email protected]>

tlb_remove_table_sync_one() sends IPIs to all CPUs and waits for them, which we really don't want to do while holding PTL.

Could you add more comments to explain why this is safe for the PAE case?

For the non-PAE case, you added a new tlb_remove_table_sync_one(), why we need this (to solve what problem)? Please also add more comments to explain.

Quoted 64 lines

Just move the call to after we release PTL, and drop the macro wrapper while we're at it.

---
  include/linux/pgtable.h | 4 ----
  mm/khugepaged.c         | 5 +++--
  2 files changed, 3 insertions(+), 6 deletions(-)
include/linux/pgtable.h +0 -4
index eb8aacba3698..fb04ed22052c 100644
@@ -755,7 +755,6 @@ static inline pmd_t pmdp_get_lockless(pmd_t *pmdp)
  	return pmd;
  }
  #define pmdp_get_lockless pmdp_get_lockless
-#define pmdp_get_lockless_sync() tlb_remove_table_sync_one()
  #endif /* CONFIG_PGTABLE_LEVELS > 2 */
  #endif /* CONFIG_GUP_GET_PXX_LOW_HIGH */
 
@@ -774,9 +773,6 @@ static inline pmd_t pmdp_get_lockless(pmd_t *pmdp)
  {
  	return pmdp_get(pmdp);
  }
-static inline void pmdp_get_lockless_sync(void)
-{
-}
  #endif
 
  #ifdef CONFIG_TRANSPARENT_HUGEPAGE
mm/khugepaged.c +3 -2
index 9f790ec34400..0a6cebf880e0 100644
@@ -1664,10 +1664,10 @@ static enum scan_result try_collapse_pte_mapped_thp(struct mm_struct *mm, unsign
  		}
  	}
  	pgt_pmd = pmdp_collapse_flush(vma, haddr, pmd);
-	pmdp_get_lockless_sync();
  	pte_unmap_unlock(start_pte, ptl);
  	if (ptl != pml)
  		spin_unlock(pml);
+	tlb_remove_table_sync_one();
 
  	mmu_notifier_invalidate_range_end(&range);
 
@@ -1818,7 +1818,6 @@ static void retract_page_tables(struct address_space *mapping, pgoff_t pgoff)
  		 */
  		if (likely(file_backed_vma_is_retractable(vma))) {
  			pgt_pmd = pmdp_collapse_flush(vma, addr, pmd);
-			pmdp_get_lockless_sync();
  			success = true;
  		}
 
@@ -1826,6 +1825,8 @@ static void retract_page_tables(struct address_space *mapping, pgoff_t pgoff)
  			spin_unlock(ptl);
  drop_pml:
  		spin_unlock(pml);
+		if (success)
+			tlb_remove_table_sync_one();
 
  		mmu_notifier_invalidate_range_end(&range);
 
author reviewer
#2 lore raw
Quoted 9 lines · Baolin Wang

Hi Lance,

Quoted 4 lines · Lance Yang

From: Lance Yang <[email protected]>

tlb_remove_table_sync_one() sends IPIs to all CPUs and waits for them, which we really don't want to do while holding PTL.

Could you add more comments to explain why this is safe for the PAE case?

Yep, IIUC, it is safe because we've already done pmdp_collapse_flush() which ensures the PMD change is visible.

pmdp_get_lockless_sync() (which calls tlb_remove_table_sync_one() on PAE) is just to ensure any ongoing lockless pmd readers (e.g., GUP-fast) complete before we proceed. It sends IPIs to all CPUs and waits for responses - a CPU can only respond when it's not between local_irq_save() and local_irq_restore().

Moving it out from under PTL doesn't change the synchronization semantics, since lockless readers don't depend on PTL anyway.

Quoted 2 lines

For the non-PAE case, you added a new tlb_remove_table_sync_one(), why we need this (to solve what problem)? Please also add more comments to explain.

Oops, you're right, the original macro was a no-op for non-PAE.

I should just move the macro call out from under PTL, rather than replacing it with direct tlb_remove_table_sync_one() calls.

Thanks,
Lance

Quoted 62 lines
Quoted 61 lines

Just move the call to after we release PTL, and drop the macro wrapper while we're at it.

---
  include/linux/pgtable.h | 4 ----
  mm/khugepaged.c         | 5 +++--

  2 files changed, 3 insertions(+), 6 deletions(-)
include/linux/pgtable.h +0 -4
index eb8aacba3698..fb04ed22052c 100644
@@ -755,7 +755,6 @@ static inline pmd_t pmdp_get_lockless(pmd_t *pmdp)
       return pmd;
   }
   #define pmdp_get_lockless pmdp_get_lockless
-#define pmdp_get_lockless_sync() tlb_remove_table_sync_one()
   #endif /* CONFIG_PGTABLE_LEVELS > 2 */
   #endif /* CONFIG_GUP_GET_PXX_LOW_HIGH */
@@ -774,9 +773,6 @@ static inline pmd_t pmdp_get_lockless(pmd_t *pmdp)
   {
       return pmdp_get(pmdp);
   }
-static inline void pmdp_get_lockless_sync(void)
-{
-}
   #endif
   #ifdef CONFIG_TRANSPARENT_HUGEPAGE
mm/khugepaged.c +0 -0
index 9f790ec34400..0a6cebf880e0 100644
@@ -1664,10 +1664,10 @@ static enum scan_result 

try_collapse_pte_mapped_thp(struct mm_struct *mm, unsign }
      }
      pgt_pmd = pmdp_collapse_flush(vma, haddr, pmd); -    pmdp_get_lockless_sync(); pte_unmap_unlock(start_pte, ptl);
      if (ptl != pml)
          spin_unlock(pml);
+    tlb_remove_table_sync_one(); mmu_notifier_invalidate_range_end(&range);

mm/khugepaged.c +0 -1
@@ -1818,7 +1818,6 @@ static void retract_page_tables(struct 
 address_space *mapping, pgoff_t pgoff)
            */
           if (likely(file_backed_vma_is_retractable(vma))) {
               pgt_pmd = pmdp_collapse_flush(vma, addr, pmd);
-            pmdp_get_lockless_sync();
               success = true;
           }
@@ -1826,6 +1825,8 @@ static void retract_page_tables(struct 
 address_space *mapping, pgoff_t pgoff)
               spin_unlock(ptl);

  drop_pml:
          spin_unlock(pml);
+        if (success)
+            tlb_remove_table_sync_one(); mmu_notifier_invalidate_range_end(&range);

reviewer
#3 lore raw
Quoted 16 lines · Lance Yang
Quoted 9 lines · Baolin Wang

Hi Lance,

Quoted 4 lines · Lance Yang

From: Lance Yang <[email protected]>

tlb_remove_table_sync_one() sends IPIs to all CPUs and waits for them, which we really don't want to do while holding PTL.

Could you add more comments to explain why this is safe for the PAE case?

Yep, IIUC, it is safe because we've already done pmdp_collapse_flush() which ensures the PMD change is visible.

Quoted 7 lines

pmdp_get_lockless_sync() (which calls tlb_remove_table_sync_one() on PAE) is just to ensure any ongoing lockless pmd readers (e.g., GUP-fast) complete before we proceed. It sends IPIs to all CPUs and waits for responses - a CPU can only respond when it's not between local_irq_save() and local_irq_restore().

Moving it out from under PTL doesn't change the synchronization semantics, since lockless readers don't depend on PTL anyway.

Cc Hugh who introduced the pmdp_get_lockless_sync(), to double check.

Sounds reasonable to me, please add these comments into the commit message. Thanks.

Quoted 6 lines
Quoted 1 line

For the non-PAE case, you added a new tlb_remove_table_sync_one(), why we need this (to solve what problem)? Please also add more comments to explain.

Oops, you're right, the original macro was a no-op for non-PAE.

I should just move the macro call out from under PTL, rather than replacing it with direct tlb_remove_table_sync_one() calls.

OK.

Quoted 61 lines
Quoted 61 lines
Quoted 60 lines

Just move the call to after we release PTL, and drop the macro wrapper while we're at it.

---
  include/linux/pgtable.h | 4 ----
  mm/khugepaged.c         | 5 +++--

  2 files changed, 3 insertions(+), 6 deletions(-)
include/linux/pgtable.h +0 -4
index eb8aacba3698..fb04ed22052c 100644
@@ -755,7 +755,6 @@ static inline pmd_t pmdp_get_lockless(pmd_t *pmdp)
       return pmd;
   }
   #define pmdp_get_lockless pmdp_get_lockless
-#define pmdp_get_lockless_sync() tlb_remove_table_sync_one()
   #endif /* CONFIG_PGTABLE_LEVELS > 2 */
   #endif /* CONFIG_GUP_GET_PXX_LOW_HIGH */
@@ -774,9 +773,6 @@ static inline pmd_t pmdp_get_lockless(pmd_t *pmdp)
   {
       return pmdp_get(pmdp);
   }
-static inline void pmdp_get_lockless_sync(void)
-{
-}
   #endif
   #ifdef CONFIG_TRANSPARENT_HUGEPAGE
mm/khugepaged.c +0 -0
index 9f790ec34400..0a6cebf880e0 100644
@@ -1664,10 +1664,10 @@ static enum scan_result 

try_collapse_pte_mapped_thp(struct mm_struct *mm, unsign }
      }
      pgt_pmd = pmdp_collapse_flush(vma, haddr, pmd); -    pmdp_get_lockless_sync(); pte_unmap_unlock(start_pte, ptl);
      if (ptl != pml)
          spin_unlock(pml);
+    tlb_remove_table_sync_one(); mmu_notifier_invalidate_range_end(&range);

mm/khugepaged.c +0 -1
@@ -1818,7 +1818,6 @@ static void retract_page_tables(struct 
 address_space *mapping, pgoff_t pgoff)
            */
           if (likely(file_backed_vma_is_retractable(vma))) {
               pgt_pmd = pmdp_collapse_flush(vma, addr, pmd);
-            pmdp_get_lockless_sync();
               success = true;
           }
@@ -1826,6 +1825,8 @@ static void retract_page_tables(struct 
 address_space *mapping, pgoff_t pgoff)
               spin_unlock(ptl);

  drop_pml:
          spin_unlock(pml);
+        if (success)
+            tlb_remove_table_sync_one(); mmu_notifier_invalidate_range_end(&range);

author reviewer
#4 lore raw
Quoted 24 lines · Baolin Wang
Quoted 21 lines · Lance Yang
Quoted 9 lines · Baolin Wang

Hi Lance,

Quoted 4 lines · Lance Yang

From: Lance Yang <[email protected]>

tlb_remove_table_sync_one() sends IPIs to all CPUs and waits for them, which we really don't want to do while holding PTL.

Could you add more comments to explain why this is safe for the PAE case?

Yep, IIUC, it is safe because we've already done pmdp_collapse_flush() which ensures the PMD change is visible.

pmdp_get_lockless_sync() (which calls tlb_remove_table_sync_one() on PAE) is just to ensure any ongoing lockless pmd readers (e.g., GUP-fast) complete before we proceed. It sends IPIs to all CPUs and waits for responses - a CPU can only respond when it's not between local_irq_save() and local_irq_restore().

Quoted 6 lines
Quoted 2 lines

Moving it out from under PTL doesn't change the synchronization semantics, since lockless readers don't depend on PTL anyway.

Cc Hugh who introduced the pmdp_get_lockless_sync(), to double check.

Sounds reasonable to me, please add these comments into the commit message. Thanks.

Yes, will do. Thanks!

Quoted 9 lines
Quoted 7 lines
Quoted 2 lines

For the non-PAE case, you added a new tlb_remove_table_sync_one(), why we need this (to solve what problem)? Please also add more comments to explain.

Oops, you're right, the original macro was a no-op for non-PAE.

I should just move the macro call out from under PTL, rather than replacing it with direct tlb_remove_table_sync_one() calls.

OK.

Cheers,
Lance

author reviewer
#5 lore raw

Re: [PATCH v1 1/1] mm/khugepaged: move tlb_remove_table_sync_one out

Hi Hugh,

Could you check if my understanding is correct?

On PAE, pmdp_get_lockless() reads pmd_low first, then pmd_high. There's a risk of reading mismatched values if another CPU modifies the PMD between the two reads.

Commit 146b42e07494[1] introduced local_irq_save() to protect the split-read, blocking TLB flush IPIs during the operation.

After modifying the PMD, pmdp_get_lockless_sync() sends an IPI to ensure all ongoing split-reads complete before proceeding with pte_free_defer().

As commit 146b42e07494[1] says:

```
Complement this pmdp_get_lockless_start() and pmdp_get_lockless_end(),
used only locally in __pte_offset_map(), with a pmdp_get_lockless_sync()
synonym for tlb_remove_table_sync_one(): to send the necessary interrupt
at the right moment on those configs which do not already send it.
```

And commit 1043173eb5eb[2] says:

```
Follow the pattern in retract_page_tables(); and using pte_free_defer()
removes most of the need for tlb_remove_table_sync_one() here; but call
pmdp_get_lockless_sync() to use it in the PAE case.
```

Regarding moving pmdp_get_lockless_sync() out from under PTL: Since lockless readers (e.g., GUP-fast, __pte_offset_map()) are protected by local_irq_save() rather than PTL, pmdp_get_lockless_sync() can be called outside PTL as long as it's before pte_free_defer().

In contrast, for non-PAE, PMD reads are atomic, so pmdp_get_lockless_sync() is a no-op.

[1] https://github.com/torvalds/linux/commit/146b42e07494e45f7c7bcf2cbf7afd1424afd78e
[2] https://github.com/torvalds/linux/commit/1043173eb5eb351a1dba11cca12705075fe74a9e

Thanks,
Lance

Quoted 57 lines · Lance Yang
Quoted 28 lines · Baolin Wang
Quoted 25 lines · Lance Yang
Quoted 10 lines · Baolin Wang

Hi Lance,

Quoted 4 lines · Lance Yang

From: Lance Yang <[email protected]>

tlb_remove_table_sync_one() sends IPIs to all CPUs and waits for them, which we really don't want to do while holding PTL.

Could you add more comments to explain why this is safe for the PAE case?

Yep, IIUC, it is safe because we've already done pmdp_collapse_flush() which ensures the PMD change is visible.

pmdp_get_lockless_sync() (which calls tlb_remove_table_sync_one() on PAE) is just to ensure any ongoing lockless pmd readers (e.g., GUP-fast) complete
before we proceed. It sends IPIs to all CPUs and waits for responses - a CPU
can only respond when it's not between local_irq_save() and local_irq_restore().

Quoted 8 lines
Quoted 3 lines

Moving it out from under PTL doesn't change the synchronization semantics,
since lockless readers don't depend on PTL anyway.

Cc Hugh who introduced the pmdp_get_lockless_sync(), to double check.

Sounds reasonable to me, please add these comments into the commit message. Thanks.

Yes, will do. Thanks!

Quoted 11 lines
Quoted 9 lines
Quoted 4 lines

For the non-PAE case, you added a new tlb_remove_table_sync_one(), why we need this (to solve what problem)? Please also add more comments to explain.

Oops, you're right, the original macro was a no-op for non-PAE.

I should just move the macro call out from under PTL, rather than replacing it with direct tlb_remove_table_sync_one() calls.

OK.

Cheers,
Lance

author reviewer
#6 lore raw

Re: [PATCH v1 1/1] mm/khugepaged: move tlb_remove_table_sync_one out

Quoted 31 lines · Lance Yang

Hi Hugh,

Could you check if my understanding is correct?

On PAE, pmdp_get_lockless() reads pmd_low first, then pmd_high. There's a risk of reading mismatched values if another CPU modifies the PMD between the two reads.

Commit 146b42e07494[1] introduced local_irq_save() to protect the split-read, blocking TLB flush IPIs during the operation.

After modifying the PMD, pmdp_get_lockless_sync() sends an IPI to ensure all ongoing split-reads complete before proceeding with pte_free_defer().

As commit 146b42e07494[1] says:

```
Complement this pmdp_get_lockless_start() and pmdp_get_lockless_end(),
used only locally in __pte_offset_map(), with a pmdp_get_lockless_sync()
synonym for tlb_remove_table_sync_one(): to send the necessary interrupt
at the right moment on those configs which do not already send it.
```

And commit 1043173eb5eb[2] says:

```
Follow the pattern in retract_page_tables(); and using pte_free_defer()
removes most of the need for tlb_remove_table_sync_one() here; but call
pmdp_get_lockless_sync() to use it in the PAE case.
```
Quoted 4 lines

Regarding moving pmdp_get_lockless_sync() out from under PTL: Since lockless readers (e.g., GUP-fast, __pte_offset_map()) are protected by local_irq_save() rather than PTL, pmdp_get_lockless_sync() can be called outside PTL as long as it's before pte_free_defer().

Looking at commit 146b42e07494[1] again, it says pmdp_get_lockless_sync() should be called "at the right moment". I now realize moving it outside PTL might not be safe.

If we release PTL before calling pmdp_get_lockless_sync(), another CPU could set a new PMD while a lockless reader is still in local_irq_save() reading the old PMD (split-read). I'm not sure if this race is actually possible, but if it is, it would hit the ABA problem where the reader gets mismatched pmd_low (old) and pmd_high (new) - the "faint risk" mentioned in commit 146b42e07494[1].

On Native x86 PAE, pmdp_collapse_flush() sends IPI and waits, preventing this race. But on PV, the hypercall returns immediately, so we need pmdp_get_lockless_sync() to ensure all IRQ-off readers complete before releasing PTL.

I should keep it under PTL to be safe.

Sorry for the churn :(

[1] https://github.com/torvalds/linux/commit/146b42e07494e45f7c7bcf2cbf7afd1424afd78e

Thanks,
Lance

Quoted 69 lines

In contrast, for non-PAE, PMD reads are atomic, so pmdp_get_lockless_sync() is a no-op.

[1] https://github.com/torvalds/linux/commit/146b42e07494e45f7c7bcf2cbf7afd1424afd78e
[2] https://github.com/torvalds/linux/commit/1043173eb5eb351a1dba11cca12705075fe74a9e

Thanks,
Lance

Quoted 57 lines · Lance Yang
Quoted 28 lines · Baolin Wang
Quoted 25 lines · Lance Yang
Quoted 10 lines · Baolin Wang

Hi Lance,

Quoted 4 lines · Lance Yang

From: Lance Yang <[email protected]>

tlb_remove_table_sync_one() sends IPIs to all CPUs and waits for them, which we really don't want to do while holding PTL.

Could you add more comments to explain why this is safe for the PAE case?

Yep, IIUC, it is safe because we've already done pmdp_collapse_flush() which ensures the PMD change is visible.

pmdp_get_lockless_sync() (which calls tlb_remove_table_sync_one() on PAE) is just to ensure any ongoing lockless pmd readers (e.g., GUP-fast) complete
before we proceed. It sends IPIs to all CPUs and waits for responses - a CPU
can only respond when it's not between local_irq_save() and local_irq_restore().

Quoted 8 lines
Quoted 3 lines

Moving it out from under PTL doesn't change the synchronization semantics,
since lockless readers don't depend on PTL anyway.

Cc Hugh who introduced the pmdp_get_lockless_sync(), to double check.

Sounds reasonable to me, please add these comments into the commit message. Thanks.

Yes, will do. Thanks!

Quoted 11 lines
Quoted 9 lines
Quoted 4 lines

For the non-PAE case, you added a new tlb_remove_table_sync_one(), why we need this (to solve what problem)? Please also add more comments to explain.

Oops, you're right, the original macro was a no-op for non-PAE.

I should just move the macro call out from under PTL, rather than replacing it with direct tlb_remove_table_sync_one() calls.

OK.

Cheers,
Lance