skip redundant TLB sync IPIs

From: Lance Yang <[email protected]>

When unsharing hugetlb PMD page tables, we currently send two IPIs: one for TLB invalidation, and another to synchronize with concurrent GUP-fast walkers.

However, if the TLB flush already reaches all CPUs, the second IPI is redundant. GUP-fast runs with IRQs disabled, so when the TLB flush IPI completes, any concurrent GUP-fast must have finished.

Add tlb_table_flush_implies_ipi_broadcast() to let architectures indicate their TLB flush provides full synchronization, enabling the redundant IPI to be skipped.

Suggested-by David Hildenbrand (Red Hat) <[email protected]>

Signed-off-by Lance Yang <[email protected]>

---
  include/asm-generic/tlb.h | 14 ++++++++++++++
  1 file changed, 14 insertions(+)

include/asm-generic/tlb.h +7 -0

index 4d679d2a206b..e8d99b5e831f 100644
@@ -261,6 +261,20 @@ static inline void tlb_remove_table_sync_one(void) { }
 
  #endif /* CONFIG_MMU_GATHER_RCU_TABLE_FREE */
 
+/*
+ * Architectures can override if their TLB flush already broadcasts IPIs to all
+ * CPUs when freeing or unsharing page tables.
+ *
+ * Return true only when the flush guarantees:
+ * - IPIs reach all CPUs with potentially stale paging-structure cache entries
+ * - Synchronization with IRQ-disabled code like GUP-fast

From: Lance Yang <[email protected]>

Embed the tlb_table_flush_implies_ipi_broadcast() check directly inside tlb_remove_table_sync_one() instead of requiring every caller to check it explicitly. This relies on callers to do the right thing: flush with freed_tables=true or unshared_tables=true beforehand.

All existing callers satisfy this requirement:

1. mm/khugepaged.c:1188 (collapse_huge_page):

    pmdp_collapse_flush(vma, address, pmd)
    -> flush_tlb_range(vma, address, address + HPAGE_PMD_SIZE)
       -> flush_tlb_mm_range(mm, ..., freed_tables = true)
          -> flush_tlb_multi(mm_cpumask(mm), info)

    So freed_tables=true before calling tlb_remove_table_sync_one().

2. include/asm-generic/tlb.h:861 (tlb_flush_unshared_tables):

    tlb_flush_mmu_tlbonly(tlb)
    -> tlb_flush(tlb)
       -> flush_tlb_mm_range(mm, ..., unshared_tables = true)
          -> flush_tlb_multi(mm_cpumask(mm), info)

    unshared_tables=true (equivalent to freed_tables for sending IPIs).

3. mm/mmu_gather.c:341 (__tlb_remove_table_one):

When we can't allocate a batch page in tlb_remove_table(), we do:

tlb_table_invalidate(tlb)
-> tlb_flush_mmu_tlbonly(tlb)
-> flush_tlb_mm_range(mm, ..., freed_tables = true)
-> flush_tlb_multi(mm_cpumask(mm), info)

Then:
tlb_remove_table_one(table)
-> __tlb_remove_table_one(table) // if !CONFIG_PT_RECLAIM
-> tlb_remove_table_sync_one()

freed_tables=true, and this should work too.

Why is tlb->freed_tables guaranteed? Because callers like pte_free_tlb() (via free_pte_range) set freed_tables=true before calling __pte_free_tlb(), which then calls tlb_remove_table(). We cannot free page tables without freed_tables=true.

Note that tlb_remove_table_sync_one() was a NOP on bare metal x86 (CONFIG_MMU_GATHER_RCU_TABLE_FREE=n) before commit a37259732a7d ("x86/mm: Make MMU_GATHER_RCU_TABLE_FREE unconditional").

4-5. mm/khugepaged.c:1683,1819 (pmdp_get_lockless_sync macro):

Same as #1. These also use pmdp_collapse_flush() beforehand.

Suggested-by David Hildenbrand (Red Hat) <[email protected]>

Signed-off-by Lance Yang <[email protected]>

---
  mm/mmu_gather.c | 4 ++++
  1 file changed, 4 insertions(+)

mm/mmu_gather.c +4 -0

index 7468ec388455..7b588643cbae 100644
@@ -276,6 +276,10 @@ static void tlb_remove_table_smp_sync(void *arg)
 
  void tlb_remove_table_sync_one(void)
  {
+	/* Skip the IPI if the TLB flush already synchronized with other CPUs. */
+	if (tlb_table_flush_implies_ipi_broadcast())
+		return;
+
  	/*
  	 * This isn't an RCU grace period and hence the page-tables cannot be
  	 * assumed to be actually RCU-freed.

Quoted 43 lines · Lance Yang

From: Lance Yang <[email protected]>

When unsharing hugetlb PMD page tables, we currently send two IPIs: one for TLB invalidation, and another to synchronize with concurrent GUP-fast walkers.

However, if the TLB flush already reaches all CPUs, the second IPI is redundant. GUP-fast runs with IRQs disabled, so when the TLB flush IPI completes, any concurrent GUP-fast must have finished.

Add tlb_table_flush_implies_ipi_broadcast() to let architectures indicate their TLB flush provides full synchronization, enabling the redundant IPI to be skipped.

Suggested-by David Hildenbrand (Red Hat) <[email protected]>

Signed-off-by Lance Yang <[email protected]>

---
  include/asm-generic/tlb.h | 14 ++++++++++++++

  1 file changed, 14 insertions(+)

include/asm-generic/tlb.h +2 -0

index 4d679d2a206b..e8d99b5e831f 100644
@@ -261,6 +261,20 @@ static inline void 
 tlb_remove_table_sync_one(void) { }
   #endif /* CONFIG_MMU_GATHER_RCU_TABLE_FREE */
+/*
+ * Architectures can override if their TLB flush already broadcasts 

IPIs to all
+ * CPUs when freeing or unsharing page tables.
+ *
+ * Return true only when the flush guarantees:
+ * - IPIs reach all CPUs with potentially stale paging-structure cache entries
+ * - Synchronization with IRQ-disabled code like GUP-fast
+ */

include/asm-generic/tlb.h +6 -0

+#ifndef tlb_table_flush_implies_ipi_broadcast
+static inline bool tlb_table_flush_implies_ipi_broadcast(void)
+{
+    return false;
+}
+#endif

Quoted 2 lines

  #ifndef CONFIG_MMU_GATHER_NO_GATHER
  /*

This should likely get squashed into patch #3. Patch #1 itself does not add a lot of value to be had separately.

So best to squash both and have them as #1, to then implement it in #2 for x86.

Quoted 52 lines · Lance Yang

From: Lance Yang <[email protected]>

Embed the tlb_table_flush_implies_ipi_broadcast() check directly inside tlb_remove_table_sync_one() instead of requiring every caller to check it explicitly. This relies on callers to do the right thing: flush with freed_tables=true or unshared_tables=true beforehand.

All existing callers satisfy this requirement:

1. mm/khugepaged.c:1188 (collapse_huge_page):

    pmdp_collapse_flush(vma, address, pmd)
    -> flush_tlb_range(vma, address, address + HPAGE_PMD_SIZE) -> flush_tlb_mm_range(mm, ..., freed_tables = true) -> flush_tlb_multi(mm_cpumask(mm), info)

    So freed_tables=true before calling tlb_remove_table_sync_one().

2. include/asm-generic/tlb.h:861 (tlb_flush_unshared_tables):

    tlb_flush_mmu_tlbonly(tlb)
    -> tlb_flush(tlb)
       -> flush_tlb_mm_range(mm, ..., unshared_tables = true) -> flush_tlb_multi(mm_cpumask(mm), info)

    unshared_tables=true (equivalent to freed_tables for sending IPIs).

3. mm/mmu_gather.c:341 (__tlb_remove_table_one):

    When we can't allocate a batch page in tlb_remove_table(), we do:

    tlb_table_invalidate(tlb)
    -> tlb_flush_mmu_tlbonly(tlb)
       -> flush_tlb_mm_range(mm, ..., freed_tables = true) -> flush_tlb_multi(mm_cpumask(mm), info)

    Then:
    tlb_remove_table_one(table)
    -> __tlb_remove_table_one(table) // if !CONFIG_PT_RECLAIM -> tlb_remove_table_sync_one()

    freed_tables=true, and this should work too.

    Why is tlb->freed_tables guaranteed? Because callers like pte_free_tlb() (via free_pte_range) set freed_tables=true before calling __pte_free_tlb(), which then calls tlb_remove_table(). We cannot free page tables without freed_tables=true.

    Note that tlb_remove_table_sync_one() was a NOP on bare metal x86 (CONFIG_MMU_GATHER_RCU_TABLE_FREE=n) before commit a37259732a7d ("x86/mm: Make MMU_GATHER_RCU_TABLE_FREE unconditional").

Quoted 6 lines

4-5. mm/khugepaged.c:1683,1819 (pmdp_get_lockless_sync macro):

    Same as #1. These also use pmdp_collapse_flush() beforehand.

Suggested-by David Hildenbrand (Red Hat) <[email protected]>

Signed-off-by Lance Yang <[email protected]>

LGTM. I think we should document that somewhere. Can we add some

kerneldoc for tlb_remove_table_sync_one() where we document that it doesn't to any sync if a previous TLB flush when removing/unsharing page tables would have already performed an IPI?

Quoted 20 lines

---
  mm/mmu_gather.c | 4 ++++

  1 file changed, 4 insertions(+)

mm/mmu_gather.c +1 -0

index 7468ec388455..7b588643cbae 100644
@@ -276,6 +276,10 @@ static void tlb_remove_table_smp_sync(void *arg)
   void tlb_remove_table_sync_one(void)
   {
+    /* Skip the IPI if the TLB flush already synchronized with other 

CPUs. */
+    if (tlb_table_flush_implies_ipi_broadcast())
+        return;
+
      /*
       * This isn't an RCU grace period and hence the page-tables cannot be
       * assumed to be actually RCU-freed.

This series introduces a way for architectures to indicate their TLB flush already provides full synchronization, allowing the redundant IPI to be skipped. For now, the optimization is implemented for x86 first and applied to all page table operations that free or unshare tables.

When unsharing hugetlb PMD page tables or collapsing pages in khugepaged, we send two IPIs: one for TLB invalidation, and another to synchronize with concurrent GUP-fast walkers.

On 12/29/25 06:52, Lance Yang wrote:
...

Quoted 4 lines

This series introduces a way for architectures to indicate their TLB flush already provides full synchronization, allowing the redundant IPI to be skipped. For now, the optimization is implemented for x86 first and applied to all page table operations that free or unshare tables.

I really don't like all the complexity here. Even on x86, there are three or more ways of deriving this. Having the pv_ops check the value of another pv op is also a bit unsettling.

That said, complexity can be worth it with sufficient demonstrated gains. But:

Quoted 3 lines

When unsharing hugetlb PMD page tables or collapsing pages in khugepaged, we send two IPIs: one for TLB invalidation, and another to synchronize with concurrent GUP-fast walkers.

Those aren't exactly hot paths. khugepaged is fundamentally rate limited. I don't think unsharing hugetlb PMD page tables just is all that common either.

Quoted 9 lines · Dave Hansen

On 12/29/25 06:52, Lance Yang wrote:
...

Quoted 6 lines

This series introduces a way for architectures to indicate their TLB flush
already provides full synchronization, allowing the redundant IPI to be skipped. For now, the optimization is implemented for x86 first and applied
to all page table operations that free or unshare tables.

Quoted 3 lines

I really don't like all the complexity here. Even on x86, there are three or more ways of deriving this. Having the pv_ops check the value of another pv op is also a bit unsettling.

Right. What I actually meant is that we simply have a property "bool flush_tlb_multi_implies_ipi_broadcast" that we set only to true from the initialization code.

Without comparing the pv_ops.

That should reduce the complexity quite a bit IMHO.

But maybe you have an even better way on how to indicate support, in a very simple way.

Quoted 11 lines

That said, complexity can be worth it with sufficient demonstrated gains. But:

Quoted 4 lines

When unsharing hugetlb PMD page tables or collapsing pages in khugepaged,
we send two IPIs: one for TLB invalidation, and another to synchronize with concurrent GUP-fast walkers.

Those aren't exactly hot paths. khugepaged is fundamentally rate limited. I don't think unsharing hugetlb PMD page tables just is all that common either.

Given that the added IPIs during unsharing broke Oracle DBs rather badly [1], I think this is actually a case worth optimizing.

[1] https://lkml.kernel.org/r/[email protected]

Quoted 18 lines · David Hildenbrand (Red Hat)

Quoted 9 lines · Dave Hansen

On 12/29/25 06:52, Lance Yang wrote:
...

Quoted 6 lines

This series introduces a way for architectures to indicate their TLB flush
already provides full synchronization, allowing the redundant IPI to be skipped. For now, the optimization is implemented for x86 first and applied
to all page table operations that free or unshare tables.

Quoted 3 lines

I really don't like all the complexity here. Even on x86, there are three or more ways of deriving this. Having the pv_ops check the value of another pv op is also a bit unsettling.

Right. What I actually meant is that we simply have a property "bool flush_tlb_multi_implies_ipi_broadcast" that we set only to true from the initialization code.

Quoted 3 lines

Without comparing the pv_ops.

That should reduce the complexity quite a bit IMHO.

Yeah, that sounds promising.

Quoted 3 lines

But maybe you have an even better way on how to indicate support, in a very simple way.

Rather than having some kind of explicit support enumeration, the other idea I had would be to actually track the state about what needs to get flushed somewhere. For instance, even CPUs with enabled INVLPGB support still use IPIs sometimes. That makes the tlb_table_flush_implies_ipi_broadcast() check a bit imperfect as is because it will for the extra sync IPI even when INVLPGB isn't being used for an mm.

First, we already save some semblance of support for doing different flushes when freeing page tables mmu_gather->freed_tables. But, the call sites in question here are for a single flush and don't use mmu_gathers.

The other pretty straightforward thing to do would be to add something to mm->context that indicates that page tables need to be freed but there might still be wild gup walkers out there that need an IPI. It would get set when the page tables are modified and cleared at all the sites where an IPIs are sent.

Quoted 16 lines

Quoted 5 lines

That said, complexity can be worth it with sufficient demonstrated gains. But:

Quoted 8 lines

Quoted 4 lines

When unsharing hugetlb PMD page tables or collapsing pages in khugepaged,
we send two IPIs: one for TLB invalidation, and another to synchronize with concurrent GUP-fast walkers.

Those aren't exactly hot paths. khugepaged is fundamentally rate limited. I don't think unsharing hugetlb PMD page tables just is all that common either.

Given that the added IPIs during unsharing broke Oracle DBs rather badly [1], I think this is actually a case worth optimizing.

...

Quoted 1 line

[1] https://lkml.kernel.org/r/[email protected]

Gah, that's good context, thanks.

Are there any tests out there that might catch these this case better? It might be something good to have 0day watch for.

Maybe we could do that as a follow-up. I'd like to keep things simple for now, so we just add a bool property to skip redundant TLB sync IPIs on systems without INVLPGB support.

Then we could add the mm->context (or something similar) tracking later to handle things more precisely.

Anyway, I'm open to going straight to the mm->context approach as well and happy to do that instead :D

On 1/3/26 00:39, Lance Yang wrote:
...

Quoted 3 lines

Maybe we could do that as a follow-up. I'd like to keep things simple for now, so we just add a bool property to skip redundant TLB sync IPIs on systems without INVLPGB support.

It's not just INVLPGB support. Take a look at hyperv_flush_tlb_multi(), for instance. It can eventually land back in native_flush_tlb_multi(), but would also "fail" the pv_ops check in all cases.

It's not that Hyper-V performance is super important, it just that the semantics of the chosen approach here are rather complicated.

Quoted 6 lines

Then we could add the mm->context (or something similar) tracking later to handle things more precisely.

Anyway, I'm open to going straight to the mm->context approach as well and happy to do that instead :D

I'd really like to see what an mm->context approach looks like before we go forward with what is being proposed here.

Is there some kind of hurry to get this done immediately?

Quoted 9 lines · Dave Hansen

On 1/3/26 00:39, Lance Yang wrote:
...

Quoted 3 lines

Maybe we could do that as a follow-up. I'd like to keep things simple for now, so we just add a bool property to skip redundant TLB sync IPIs on systems without INVLPGB support.

It's not just INVLPGB support. Take a look at hyperv_flush_tlb_multi(), for instance. It can eventually land back in native_flush_tlb_multi(), but would also "fail" the pv_ops check in all cases.

Thanks for pointing that out!

Quoted 3 lines

It's not that Hyper-V performance is super important, it just that the semantics of the chosen approach here are rather complicated.

Yep, got it ;)

Quoted 9 lines

Quoted 6 lines

Then we could add the mm->context (or something similar) tracking later to handle things more precisely.

Anyway, I'm open to going straight to the mm->context approach as well and happy to do that instead :D

I'd really like to see what an mm->context approach looks like before we go forward with what is being proposed here.

Actually, I went ahead and tried a simialr approach using tlb_gather to track IPI sends dynamically/precisely.

Seems simpler than the mm->context approach because:

1) IIUC, mm->context tracking would need proper synchronization (CAS, handling concurrent flushes, etc.) which adds more complexity :)

2) With tlb_gather we already have the right context at the right time - we just pass the tlb pointer through flush_tlb_mm_range() and set a flag when IPIs are actually sent.

The first one adds a tlb_flush_sent_ipi flag to mmu_gather and wires it through flush_tlb_mm_range(). When we call flush_tlb_multi(), we set the flag. Then tlb_gather_remove_table_sync_one() checks it and skips the IPI if it's set.

---8<---
When unsharing hugetlb PMD page tables, we currently send two IPIs: one for TLB invalidation, and another to synchronize with concurrent GUP-fast walkers via tlb_remove_table_sync_one().

However, if the TLB flush already sent IPIs to all CPUs (when freed_tables or unshared_tables is true), the second IPI is redundant. GUP-fast runs with IRQs disabled, so when the TLB flush IPI completes, any concurrent GUP-fast must have finished.

Add a tlb_flush_sent_ipi flag to struct mmu_gather to track whether IPIs were actually sent.

Introduce tlb_gather_remove_table_sync_one() which checks tlb_flush_sent_ipi and skips the IPI if redundant.

Suggested-by David Hildenbrand <[email protected]>

Suggested-by Dave Hansen <[email protected]>

Signed-off-by Lance Yang <[email protected]>

---
  arch/x86/include/asm/tlb.h      |  3 ++-
  arch/x86/include/asm/tlbflush.h |  8 ++++----
  arch/x86/kernel/alternative.c   |  2 +-
  arch/x86/kernel/ldt.c           |  2 +-
  arch/x86/mm/tlb.c               |  6 ++++--
  include/asm-generic/tlb.h       | 14 +++++++++-----
  mm/mmu_gather.c                 | 24 ++++++++++++++++++------
  7 files changed, 39 insertions(+), 20 deletions(-)

arch/x86/include/asm/tlb.h +2 -1

index 866ea78ba156..c5950a92058c 100644
@@ -20,7 +20,8 @@ static inline void tlb_flush(struct mmu_gather *tlb)
          end = tlb->end;
      }
 
-    flush_tlb_mm_range(tlb->mm, start, end, stride_shift, tlb- 
 >freed_tables);
+    flush_tlb_mm_range(tlb->mm, start, end, stride_shift,
+               tlb->freed_tables || tlb->unshared_tables, tlb);
  }
 

 static inline void invlpg(unsigned long addr)

arch/x86/include/asm/tlbflush.h +4 -4

index 00daedfefc1b..9524105659c3 100644
@@ -305,23 +305,23 @@ static inline bool mm_in_asid_transition(struct 
 mm_struct *mm) { return false; }
  #endif
 
  #define flush_tlb_mm(mm)                        \
-        flush_tlb_mm_range(mm, 0UL, TLB_FLUSH_ALL, 0UL, true)
+        flush_tlb_mm_range(mm, 0UL, TLB_FLUSH_ALL, 0UL, true, NULL)
 
  #define flush_tlb_range(vma, start, end)                \
      flush_tlb_mm_range((vma)->vm_mm, start, end,            \
                 ((vma)->vm_flags & VM_HUGETLB)        \
                  ? huge_page_shift(hstate_vma(vma))    \
-                : PAGE_SHIFT, true)
+                : PAGE_SHIFT, true, NULL)
 
  extern void flush_tlb_all(void);
  extern void flush_tlb_mm_range(struct mm_struct *mm, unsigned long start,
                  unsigned long end, unsigned int stride_shift,
-                bool freed_tables);
+                bool freed_tables, struct mmu_gather *tlb);
  extern void flush_tlb_kernel_range(unsigned long start, unsigned long 
 end);
 
  static inline void flush_tlb_page(struct vm_area_struct *vma, unsigned 
 long a)
  {
-    flush_tlb_mm_range(vma->vm_mm, a, a + PAGE_SIZE, PAGE_SHIFT, false);
+    flush_tlb_mm_range(vma->vm_mm, a, a + PAGE_SIZE, PAGE_SHIFT, false, 

NULL);
 }

 static inline bool arch_tlbbatch_should_defer(struct mm_struct *mm)

arch/x86/kernel/alternative.c +0 -0

index 28518371d8bf..006f3705b616 100644
@@ -2572,7 +2572,7 @@ static void *__text_poke(text_poke_f func, void 
 *addr, const void *src, size_t l
       */
      flush_tlb_mm_range(text_poke_mm, text_poke_mm_addr, 

text_poke_mm_addr +
                (cross_page_boundary ? 2 : 1) * PAGE_SIZE,
-               PAGE_SHIFT, false);
+               PAGE_SHIFT, false, NULL);

     if (func == text_poke_memcpy) {
         /*

arch/x86/kernel/ldt.c +1 -1

index 0f19ef355f5f..d8494706fec5 100644
@@ -374,7 +374,7 @@ static void unmap_ldt_struct(struct mm_struct *mm, 
 struct ldt_struct *ldt)
      }
 
      va = (unsigned long)ldt_slot_va(ldt->slot);
-    flush_tlb_mm_range(mm, va, va + nr_pages * PAGE_SIZE, PAGE_SHIFT, 
 false);
+    flush_tlb_mm_range(mm, va, va + nr_pages * PAGE_SIZE, PAGE_SHIFT, 
 false, NULL);

 }

 #else /* !CONFIG_MITIGATION_PAGE_TABLE_ISOLATION */

arch/x86/mm/tlb.c +2 -2

index f5b93e01e347..099f8d61be1a 100644
@@ -1447,8 +1447,8 @@ static void put_flush_tlb_info(void)
  }
 
  void flush_tlb_mm_range(struct mm_struct *mm, unsigned long start,
-                unsigned long end, unsigned int stride_shift,
-                bool freed_tables)
+            unsigned long end, unsigned int stride_shift,
+            bool freed_tables, struct mmu_gather *tlb)
  {
      struct flush_tlb_info *info;
      int cpu = get_cpu();

arch/x86/mm/tlb.c +0 -0

      } else if (mm == this_cpu_read(cpu_tlbstate.loaded_mm)) {
          lockdep_assert_irqs_enabled();

         local_irq_disable();

include/asm-generic/tlb.h +1 -0

index 4d679d2a206b..0ec35699da99 100644
@@ -249,6 +249,7 @@ static inline void tlb_remove_table(struct 
 mmu_gather *tlb, void *table)
  #define tlb_needs_table_invalidate() (true)
  #endif
 
+void tlb_gather_remove_table_sync_one(struct mmu_gather *tlb);
  void tlb_remove_table_sync_one(void);
 

 #else

include/asm-generic/tlb.h +1 -0

@@ -257,6 +258,7 @@ void tlb_remove_table_sync_one(void);
  #error tlb_needs_table_invalidate() requires MMU_GATHER_RCU_TABLE_FREE
  #endif
 
+static inline void tlb_gather_remove_table_sync_one(struct mmu_gather 
 *tlb) { }
  static inline void tlb_remove_table_sync_one(void) { }
 

 #endif /* CONFIG_MMU_GATHER_RCU_TABLE_FREE */

include/asm-generic/tlb.h +3 -0

@@ -379,6 +381,12 @@ struct mmu_gather {
       */
      unsigned int        fully_unshared_tables : 1;
 
+    /*
+     * Did the TLB flush for freed/unshared tables send IPIs to all CPUs?
+     * If true, we can skip the redundant IPI in 

tlb_remove_table_sync_one().
+     */
+    unsigned int        tlb_flush_sent_ipi : 1;

include/asm-generic/tlb.h +1 -3

+
      unsigned int        batch_count;
 
  #ifndef CONFIG_MMU_GATHER_NO_GATHER

@@ -834,13 +842,9 @@ static inline void tlb_flush_unshared_tables(struct 
 mmu_gather *tlb)
       *
       * We only perform this when we are the last sharer of a page table,
       * as the IPI will reach all CPUs: any GUP-fast.
-     *
-     * Note that on configs where tlb_remove_table_sync_one() is a NOP,
-     * the expectation is that the tlb_flush_mmu_tlbonly() would have 

issued
-     * required IPIs already for us.
      */
     if (tlb->fully_unshared_tables) {
-        tlb_remove_table_sync_one();
+        tlb_gather_remove_table_sync_one(tlb);
         tlb->fully_unshared_tables = false;
     }
 }

mm/mmu_gather.c +2 -1

index 7468ec388455..288c281b2ca4 100644
@@ -274,8 +274,14 @@ static void tlb_remove_table_smp_sync(void *arg)
      /* Simply deliver the interrupt */
  }
 
-void tlb_remove_table_sync_one(void)
+void tlb_gather_remove_table_sync_one(struct mmu_gather *tlb)
  {
+    /* Skip the IPI if the TLB flush already synchronized with other 

CPUs */
+    if (tlb && tlb->tlb_flush_sent_ipi) {
+        tlb->tlb_flush_sent_ipi = false;
+        return;
+    }

mm/mmu_gather.c +6 -0

+
      /*
       * This isn't an RCU grace period and hence the page-tables cannot be
       * assumed to be actually RCU-freed.

@@ -286,6 +292,11 @@ void tlb_remove_table_sync_one(void)
      smp_call_function(tlb_remove_table_smp_sync, NULL, 1);
  }
 
+void tlb_remove_table_sync_one(void)
+{
+    tlb_gather_remove_table_sync_one(NULL);
+}
+
  static void tlb_remove_table_rcu(struct rcu_head *head)
  {
      __tlb_remove_table_free(container_of(head, struct mmu_table_batch, 

rcu));

mm/mmu_gather.c +4 -4

@@ -337,16 +348,16 @@ static inline void __tlb_remove_table_one(void 
 *table)
      call_rcu(&ptdesc->pt_rcu_head, __tlb_remove_table_one_rcu);
  }
  #else
-static inline void __tlb_remove_table_one(void *table)
+static inline void __tlb_remove_table_one(void *table, struct 
 mmu_gather *tlb)
  {
-    tlb_remove_table_sync_one();
+    tlb_gather_remove_table_sync_one(tlb);
      __tlb_remove_table(table);
  }
  #endif /* CONFIG_PT_RECLAIM */
 
-static void tlb_remove_table_one(void *table)
+static void tlb_remove_table_one(void *table, struct mmu_gather *tlb)
  {
-    __tlb_remove_table_one(table);
+    __tlb_remove_table_one(table, tlb);
  }

 static void tlb_table_flush(struct mmu_gather *tlb)

mm/mmu_gather.c +1 -1

@@ -368,7 +379,7 @@ void tlb_remove_table(struct mmu_gather *tlb, void 
 *table)
          *batch = (struct mmu_table_batch *)__get_free_page(GFP_NOWAIT);
          if (*batch == NULL) {
              tlb_table_invalidate(tlb);
-            tlb_remove_table_one(table);
+            tlb_remove_table_one(table, tlb);
              return;
          }

         (*batch)->nr = 0;

mm/mmu_gather.c +1 -0

@@ -428,6 +439,7 @@ static void __tlb_gather_mmu(struct mmu_gather *tlb, 
 struct mm_struct *mm,
      tlb->vma_pfn = 0;
 
      tlb->fully_unshared_tables = 0;
+    tlb->tlb_flush_sent_ipi = 0;
      __tlb_reset_range(tlb);
      inc_tlb_flush_pending(tlb->mm);

 }
---

The second one optimizes khugepaged by using mmu_gather to track IPI sends. This makes the approach work across all paths ;)

---8<---
pmdp_collapse_flush() may already send IPIs to flush TLBs, and then callers send another IPI via tlb_remove_table_sync_one() or pmdp_get_lockless_sync() to synchronize with concurrent GUP-fast walkers.

However, since GUP-fast runs with IRQs disabled, the TLB flush IPI already provides the necessary synchronization. We can avoid the redundant second IPI.

Introduce pmdp_collapse_flush_sync() which combines flush and sync:

- For architectures using the generic pmdp_collapse_flush() implementation (e.g., x86): Use mmu_gather to track IPI sends. If the TLB flush sent an IPI, tlb_gather_remove_table_sync_one() will skip the redundant one.

- For architectures with custom pmdp_collapse_flush() (s390, riscv, powerpc): Fall back to calling pmdp_collapse_flush() followed by tlb_remove_table_sync_one(). No behavior change.

Update khugepaged to use pmdp_collapse_flush_sync() instead of separate flush and sync calls. Remove the now-unused pmdp_get_lockless_sync() macro.

Suggested-by David Hildenbrand <[email protected]>

Suggested-by Dave Hansen <[email protected]>

Signed-off-by Lance Yang <[email protected]>

---
  include/linux/pgtable.h | 13 +++++++++----
  mm/khugepaged.c         |  9 +++------
  mm/pgtable-generic.c    | 34 ++++++++++++++++++++++++++++++++++
  3 files changed, 46 insertions(+), 10 deletions(-)

include/linux/pgtable.h +2 -4

index eb8aacba3698..b42758197d47 100644
@@ -755,7 +755,6 @@ static inline pmd_t pmdp_get_lockless(pmd_t *pmdp)
      return pmd;
  }
  #define pmdp_get_lockless pmdp_get_lockless
-#define pmdp_get_lockless_sync() tlb_remove_table_sync_one()
  #endif /* CONFIG_PGTABLE_LEVELS > 2 */
  #endif /* CONFIG_GUP_GET_PXX_LOW_HIGH */
 

@@ -774,9 +773,6 @@ static inline pmd_t pmdp_get_lockless(pmd_t *pmdp)
  {
      return pmdp_get(pmdp);
  }
-static inline void pmdp_get_lockless_sync(void)
-{
-}
  #endif
 
  #ifdef CONFIG_TRANSPARENT_HUGEPAGE

@@ -1174,6 +1170,8 @@ static inline void pudp_set_wrprotect(struct 
 mm_struct *mm,
  #ifdef CONFIG_TRANSPARENT_HUGEPAGE
  extern pmd_t pmdp_collapse_flush(struct vm_area_struct *vma,
                   unsigned long address, pmd_t *pmdp);
+extern pmd_t pmdp_collapse_flush_sync(struct vm_area_struct *vma,
+                      unsigned long address, pmd_t *pmdp);
  #else
  static inline pmd_t pmdp_collapse_flush(struct vm_area_struct *vma,

                     unsigned long address,

include/linux/pgtable.h +7 -0

@@ -1182,6 +1180,13 @@ static inline pmd_t pmdp_collapse_flush(struct 
 vm_area_struct *vma,
      BUILD_BUG();
      return *pmdp;
  }
+static inline pmd_t pmdp_collapse_flush_sync(struct vm_area_struct *vma,
+                         unsigned long address,
+                         pmd_t *pmdp)
+{
+    BUILD_BUG();
+    return *pmdp;
+}
  #define pmdp_collapse_flush pmdp_collapse_flush
  #endif /* CONFIG_TRANSPARENT_HUGEPAGE */

 #endif

mm/khugepaged.c +0 -0

index 9f790ec34400..0a98afc85c50 100644
@@ -1177,10 +1177,9 @@ static enum scan_result collapse_huge_page(struct 

mm_struct *mm, unsigned long a
      * Parallel GUP-fast is fine since GUP-fast will back off when
      * it detects PMD is changed.
      */
-    _pmd = pmdp_collapse_flush(vma, address, pmd);
+    _pmd = pmdp_collapse_flush_sync(vma, address, pmd);
     spin_unlock(pmd_ptl); mmu_notifier_invalidate_range_end(&range);
-    tlb_remove_table_sync_one();

     pte = pte_offset_map_lock(mm, &_pmd, address, &pte_ptl); if (pte) {

mm/khugepaged.c +0 -0

@@ -1663,8 +1662,7 @@ static enum scan_result 

try_collapse_pte_mapped_thp(struct mm_struct *mm, unsign
             }
         }
     }
-    pgt_pmd = pmdp_collapse_flush(vma, haddr, pmd); -    pmdp_get_lockless_sync();
+    pgt_pmd = pmdp_collapse_flush_sync(vma, haddr, pmd);
     pte_unmap_unlock(start_pte, ptl);
     if (ptl != pml)
         spin_unlock(pml);

mm/khugepaged.c +1 -2

@@ -1817,8 +1815,7 @@ static void retract_page_tables(struct 
 address_space *mapping, pgoff_t pgoff)
           * races against the prior checks.
           */
          if (likely(file_backed_vma_is_retractable(vma))) {
-            pgt_pmd = pmdp_collapse_flush(vma, addr, pmd);
-            pmdp_get_lockless_sync();
+            pgt_pmd = pmdp_collapse_flush_sync(vma, addr, pmd);
              success = true;
          }

mm/pgtable-generic.c +34 -0

index d3aec7a9926a..be2ee82e6fc4 100644
@@ -233,6 +233,40 @@ pmd_t pmdp_collapse_flush(struct vm_area_struct 
 *vma, unsigned long address,
      flush_tlb_range(vma, address, address + HPAGE_PMD_SIZE);
      return pmd;
  }
+
+pmd_t pmdp_collapse_flush_sync(struct vm_area_struct *vma, unsigned 
 long address,
+                   pmd_t *pmdp)
+{
+    struct mmu_gather tlb;
+    pmd_t pmd;
+
+    VM_BUG_ON(address & ~HPAGE_PMD_MASK);
+    VM_BUG_ON(pmd_trans_huge(*pmdp));
+
+    tlb_gather_mmu(&tlb, vma->vm_mm);
+    pmd = pmdp_huge_get_and_clear(vma->vm_mm, address, pmdp);
+
+    flush_tlb_mm_range(vma->vm_mm, address, address + HPAGE_PMD_SIZE,
+               PAGE_SHIFT, true, &tlb);
+
+    /*
+     * Synchronize with GUP-fast. If the flush sent IPIs, skip the
+     * redundant sync IPI.
+     */
+    tlb_gather_remove_table_sync_one(&tlb);
+    tlb_finish_mmu(&tlb);
+    return pmd;
+}
+#else
+pmd_t pmdp_collapse_flush_sync(struct vm_area_struct *vma, unsigned 
 long address,
+                   pmd_t *pmdp)
+{
+    pmd_t pmd;
+
+    pmd = pmdp_collapse_flush(vma, address, pmdp);
+    tlb_remove_table_sync_one();
+    return pmd;
+}

 #endif

 /* arch define pte_free_defer in asm/pgalloc.h for its own
implementation */
---

Quoted 2 lines

Is there some kind of hurry to get this done immediately?

No rush at all - just wanted to explore what works best and keep things simpler as well ;)

What do you think?