Skip to contentSkip to conversation

mm/mmu_gather: replace IPI with synchronize_rcu() when batch allocation fails

PATCH Superseded Lance Yang 3 participants 8 messages Feb 23 – Feb 24

Subsystems (3)

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Lance Yang

author
#0 lore raw

PATCH 1/1 mm/mmu_gather: replace IPI with synchronize_rcu() when batch allocation fails

From: Lance Yang <[email protected]>

When freeing page tables, we try to batch them. If batch allocation fails (GFP_NOWAIT), __tlb_remove_table_one() immediately frees the one without batching.

On !CONFIG_PT_RECLAIM, the fallback sends an IPI to all CPUs via tlb_remove_table_sync_one(). It disrupts all CPUs even when only a single process is unmapping memory. IPI broadcast was reported to hurt RT workloads[1].

tlb_remove_table_sync_one() synchronizes with lockless page-table walkers (e.g. GUP-fast) that rely on IRQ disabling. These walkers use local_irq_disable(), which is also an RCU read-side critical section. synchronize_rcu() waits for all such sections to complete, providing the same guarantee as IPI but without disrupting all CPUs.

Since batch allocation already failed, we are in a way slow path, so replacing the IPI with synchronize_rcu() is fine.

We are in process context (unmap_region, exit_mmap) with only mmap_lock held, a sleeping lock. synchronize_rcu() will catch any invalid context via might_sleep().

[1] https://lore.kernel.org/linux-mm/[email protected]/

---
 mm/mmu_gather.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
mm/mmu_gather.c +2 -1
index fe5b6a031717..df670c219260 100644
@@ -339,7 +339,8 @@ static inline void __tlb_remove_table_one(void *table)
 #else
 static inline void __tlb_remove_table_one(void *table)
 {
-	tlb_remove_table_sync_one();
+	if (IS_ENABLED(CONFIG_MMU_GATHER_RCU_TABLE_FREE))
+		synchronize_rcu();
 	__tlb_remove_table(table);
 }
 #endif /* CONFIG_PT_RECLAIM */
Signature · 1 line
2.49.0
#1 lore raw
Quoted 24 lines · Lance Yang

From: Lance Yang <[email protected]>

When freeing page tables, we try to batch them. If batch allocation fails (GFP_NOWAIT), __tlb_remove_table_one() immediately frees the one without batching.

On !CONFIG_PT_RECLAIM, the fallback sends an IPI to all CPUs via tlb_remove_table_sync_one(). It disrupts all CPUs even when only a single process is unmapping memory. IPI broadcast was reported to hurt RT workloads[1].

tlb_remove_table_sync_one() synchronizes with lockless page-table walkers (e.g. GUP-fast) that rely on IRQ disabling. These walkers use local_irq_disable(), which is also an RCU read-side critical section. synchronize_rcu() waits for all such sections to complete, providing the same guarantee as IPI but without disrupting all CPUs.

Since batch allocation already failed, we are in a way slow path, so replacing the IPI with synchronize_rcu() is fine.

We are in process context (unmap_region, exit_mmap) with only mmap_lock held, a sleeping lock. synchronize_rcu() will catch any invalid context via might_sleep().

Quoted 8 lines

[1] https://lore.kernel.org/linux-mm/[email protected]/

I think it was primarily Peter and Dave suggesting that :)

Quoted 10 lines
---
  mm/mmu_gather.c | 3 ++-
  1 file changed, 2 insertions(+), 1 deletion(-)
mm/mmu_gather.c +0 -0
index fe5b6a031717..df670c219260 100644

mm/mmu_gather.c static inline void __tlb_remove_table_one(void *table) · line 343

Quoted 6 lines
mm/mmu_gather.c +2 -1
@@ -339,7 +339,8 @@ static inline void __tlb_remove_table_one(void *table)
  #else
  static inline void __tlb_remove_table_one(void *table)
  {
-	tlb_remove_table_sync_one();
+	if (IS_ENABLED(CONFIG_MMU_GATHER_RCU_TABLE_FREE))
+		synchronize_rcu();

That should work.

Reading all the comments for tlb_remove_table_smp_sync(), I wonder whether we should wrap that in a tlb_remove_table_sync_rcu() function, with a proper kerneldoc for the CONFIG_MMU_GATHER_RCU_TABLE_FREE variant where we discuss how this relates to tlb_remove_table_sync_one (and tlb_remove_table_smp_sync() .

Signature · 3 lines
Cheers,

David
author
#2 lore raw
Quoted 27 lines · David Hildenbrand (Arm)
Quoted 26 lines · Lance Yang

From: Lance Yang <[email protected]>

When freeing page tables, we try to batch them. If batch allocation fails (GFP_NOWAIT), __tlb_remove_table_one() immediately frees the one without batching.

On !CONFIG_PT_RECLAIM, the fallback sends an IPI to all CPUs via tlb_remove_table_sync_one(). It disrupts all CPUs even when only a single process is unmapping memory. IPI broadcast was reported to hurt RT workloads[1].

tlb_remove_table_sync_one() synchronizes with lockless page-table walkers (e.g. GUP-fast) that rely on IRQ disabling. These walkers use local_irq_disable(), which is also an RCU read-side critical section. synchronize_rcu() waits for all such sections to complete, providing the same guarantee as IPI but without disrupting all CPUs.

Since batch allocation already failed, we are in a way slow path, so replacing the IPI with synchronize_rcu() is fine.

We are in process context (unmap_region, exit_mmap) with only mmap_lock held, a sleeping lock. synchronize_rcu() will catch any invalid context via might_sleep().

[1] https://lore.kernel.org/linux-mm/[email protected]/

Quoted 8 lines
Quoted 6 lines

I think it was primarily Peter and Dave suggesting that :)

:)

Quoted 19 lines
Quoted 10 lines
---
  mm/mmu_gather.c | 3 ++-
  1 file changed, 2 insertions(+), 1 deletion(-)
mm/mmu_gather.c +0 -0
index fe5b6a031717..df670c219260 100644

mm/mmu_gather.c static inline void __tlb_remove_table_one(void *table) · line 343

Quoted 6 lines
mm/mmu_gather.c +2 -1
@@ -339,7 +339,8 @@ static inline void __tlb_remove_table_one(void *table)
  #else
  static inline void __tlb_remove_table_one(void *table)
  {
-	tlb_remove_table_sync_one();
+	if (IS_ENABLED(CONFIG_MMU_GATHER_RCU_TABLE_FREE))
+		synchronize_rcu();

That should work.

Quoted 5 lines

Reading all the comments for tlb_remove_table_smp_sync(), I wonder whether we should wrap that in a tlb_remove_table_sync_rcu() function, with a proper kerneldoc for the CONFIG_MMU_GATHER_RCU_TABLE_FREE variant where we discuss how this relates to tlb_remove_table_sync_one (and tlb_remove_table_smp_sync() .

Good point! That would be cleaner and better ;)

How about the following:

---8<---

mm/mmu_gather.c +23 -1
index fe5b6a031717..ea5503d3e650 100644
@@ -296,6 +296,24 @@ static void tlb_remove_table_free(struct mmu_table_batch *batch)
 	call_rcu(&batch->rcu, tlb_remove_table_rcu);
 }
 
+/**
+ * tlb_remove_table_sync_rcu() - synchronize with software page-table walkers
+ *
+ * Like tlb_remove_table_sync_one() but uses RCU grace period instead of IPI
+ * broadcast. Should be used in slow paths where sleeping is acceptable.
+ *
+ * Software/Lockless page-table walkers use local_irq_disable(), which is also
+ * an RCU read-side critical section. synchronize_rcu() waits for all such
+ * sections, providing the same guarantee as tlb_remove_table_sync_one() but
+ * without disrupting all CPUs with IPIs.
+ *
+ * Context: Can sleep/block. Cannot be called from any atomic context.
+ */
+static void tlb_remove_table_sync_rcu(void)
+{
+	synchronize_rcu();
+}
+
 #else /* !CONFIG_MMU_GATHER_RCU_TABLE_FREE */
 
 static void tlb_remove_table_free(struct mmu_table_batch *batch)
@@ -303,6 +321,10 @@ static void tlb_remove_table_free(struct mmu_table_batch *batch)
 	__tlb_remove_table_free(batch);
 }
 
+static void tlb_remove_table_sync_rcu(void)
+{
+}
+
 #endif /* CONFIG_MMU_GATHER_RCU_TABLE_FREE */
 
 /*
@@ -339,7 +361,7 @@ static inline void __tlb_remove_table_one(void *table)
 #else
 static inline void __tlb_remove_table_one(void *table)
 {
-	tlb_remove_table_sync_one();
+	tlb_remove_table_sync_rcu();
 	__tlb_remove_table(table);
 }
 #endif /* CONFIG_PT_RECLAIM */

---

Thanks for the suggestion!
Lance

#3 lore raw
Quoted 121 lines · Lance Yang
Quoted 27 lines · David Hildenbrand (Arm)
Quoted 26 lines · Lance Yang

From: Lance Yang <[email protected]>

When freeing page tables, we try to batch them. If batch allocation fails (GFP_NOWAIT), __tlb_remove_table_one() immediately frees the one without batching.

On !CONFIG_PT_RECLAIM, the fallback sends an IPI to all CPUs via tlb_remove_table_sync_one(). It disrupts all CPUs even when only a single process is unmapping memory. IPI broadcast was reported to hurt RT workloads[1].

tlb_remove_table_sync_one() synchronizes with lockless page-table walkers (e.g. GUP-fast) that rely on IRQ disabling. These walkers use local_irq_disable(), which is also an RCU read-side critical section. synchronize_rcu() waits for all such sections to complete, providing the same guarantee as IPI but without disrupting all CPUs.

Since batch allocation already failed, we are in a way slow path, so replacing the IPI with synchronize_rcu() is fine.

We are in process context (unmap_region, exit_mmap) with only mmap_lock held, a sleeping lock. synchronize_rcu() will catch any invalid context via might_sleep().

[1] https://lore.kernel.org/linux-mm/[email protected]/

Quoted 8 lines
Quoted 6 lines

I think it was primarily Peter and Dave suggesting that :)

:)

Quoted 19 lines
Quoted 10 lines
---
   mm/mmu_gather.c | 3 ++-
   1 file changed, 2 insertions(+), 1 deletion(-)
mm/mmu_gather.c +0 -0
index fe5b6a031717..df670c219260 100644

mm/mmu_gather.c static inline void __tlb_remove_table_one(void *table) · line 343

Quoted 6 lines
mm/mmu_gather.c +2 -1
@@ -339,7 +339,8 @@ static inline void __tlb_remove_table_one(void *table)
   #else
   static inline void __tlb_remove_table_one(void *table)
   {
-	tlb_remove_table_sync_one();
+	if (IS_ENABLED(CONFIG_MMU_GATHER_RCU_TABLE_FREE))
+		synchronize_rcu();

That should work.

Quoted 5 lines

Reading all the comments for tlb_remove_table_smp_sync(), I wonder whether we should wrap that in a tlb_remove_table_sync_rcu() function, with a proper kerneldoc for the CONFIG_MMU_GATHER_RCU_TABLE_FREE variant where we discuss how this relates to tlb_remove_table_sync_one (and tlb_remove_table_smp_sync() .

Good point! That would be cleaner and better ;)

How about the following:

---8<---

mm/mmu_gather.c +23 -2
index fe5b6a031717..ea5503d3e650 100644
@@ -296,6 +296,24 @@ static void tlb_remove_table_free(struct mmu_table_batch *batch)
  	call_rcu(&batch->rcu, tlb_remove_table_rcu);
  }
 
+/**
+ * tlb_remove_table_sync_rcu() - synchronize with software page-table walkers
+ *
+ * Like tlb_remove_table_sync_one() but uses RCU grace period instead of IPI
+ * broadcast. Should be used in slow paths where sleeping is acceptable.
+ *
+ * Software/Lockless page-table walkers use local_irq_disable(), which is also
+ * an RCU read-side critical section. synchronize_rcu() waits for all such
+ * sections, providing the same guarantee as tlb_remove_table_sync_one() but
+ * without disrupting all CPUs with IPIs.
+ *
+ * Context: Can sleep/block. Cannot be called from any atomic context.
+ */
+static void tlb_remove_table_sync_rcu(void)
+{
+	synchronize_rcu();
+}
+
  #else /* !CONFIG_MMU_GATHER_RCU_TABLE_FREE */
 
  static void tlb_remove_table_free(struct mmu_table_batch *batch)
@@ -303,6 +321,10 @@ static void tlb_remove_table_free(struct mmu_table_batch *batch)
  	__tlb_remove_table_free(batch);
  }
 
+static void tlb_remove_table_sync_rcu(void)
+{
+}
+
  #endif /* CONFIG_MMU_GATHER_RCU_TABLE_FREE */
 
  /*
@@ -339,7 +361,7 @@ static inline void __tlb_remove_table_one(void *table)
  #else
  static inline void __tlb_remove_table_one(void *table)
  {
-	tlb_remove_table_sync_one();
+	tlb_remove_table_sync_rcu();
  	__tlb_remove_table(table);
  }
  #endif /* CONFIG_PT_RECLAIM */
---
Quoted 2 lines

Thanks for the suggestion!
Lance

LGTM, but let's hear other options.

Signature · 3 lines
Cheers,

David
#4 lore raw
In reply to Lance Yang

On 2/23/26 04:58, Lance Yang wrote:
...

Quoted 5 lines

+/**
+ * tlb_remove_table_sync_rcu() - synchronize with software page-table walkers
+ *
+ * Like tlb_remove_table_sync_one() but uses RCU grace period instead of IPI
+ * broadcast. Should be used in slow paths where sleeping is acceptable.

Just a nit on comments: Use imperative voice:

... Use in slow paths where sleeping is acceptable.

Quoted 4 lines

+ * Software/Lockless page-table walkers use local_irq_disable(), which is also
+ * an RCU read-side critical section. synchronize_rcu() waits for all such
+ * sections, providing the same guarantee as tlb_remove_table_sync_one() but
+ * without disrupting all CPUs with IPIs.

Yep, synchronize_rcu() is likely slower (longer wall clock time) but less disruptive to other CPUs.

Is it worth explaining here that this should be used when code really needs to _wait_ and *not* for freeing memory? Freeing memory should use RCU callbacks that don't cause latency spikes in this thread, not this.

Quoted 1 line

+ * Context: Can sleep/block. Cannot be called from any atomic context.

As a general rule, expressing constraints like this is best done in code, not comments, so:

	might_sleep();
or
	WARN_ON_ONCE(!in_atomic());

seem appropriate.

I didn't see any obvious warning like that in the top levels of synchronize_rcu().

Quoted 9 lines
mm/mmu_gather.c +5 -0
+static void tlb_remove_table_sync_rcu(void)
+{
+	synchronize_rcu();
+}
+
 #else /* !CONFIG_MMU_GATHER_RCU_TABLE_FREE */
 
 static void tlb_remove_table_free(struct mmu_table_batch *batch)

mm/mmu_gather.c static void tlb_remove_table_free(struct mmu_table_batch *batch) · line 328

Quoted 8 lines
mm/mmu_gather.c +4 -0
@@ -303,6 +321,10 @@ static void tlb_remove_table_free(struct mmu_table_batch *batch)
 	__tlb_remove_table_free(batch);
 }
 
+static void tlb_remove_table_sync_rcu(void)
+{
+}
+
 #endif /* CONFIG_MMU_GATHER_RCU_TABLE_FREE */

This seems a _little_ dangerous to even define. We don't want this sneaking into use when it doesn't do anything.

author
#5 lore raw
Quoted 11 lines · Dave Hansen

On 2/23/26 04:58, Lance Yang wrote:
...

Quoted 5 lines

+/**
+ * tlb_remove_table_sync_rcu() - synchronize with software page-table walkers
+ *
+ * Like tlb_remove_table_sync_one() but uses RCU grace period instead of IPI
+ * broadcast. Should be used in slow paths where sleeping is acceptable.

Just a nit on comments: Use imperative voice:

... Use in slow paths where sleeping is acceptable.

Okay, thanks.

Quoted 11 lines
Quoted 4 lines

+ * Software/Lockless page-table walkers use local_irq_disable(), which is also
+ * an RCU read-side critical section. synchronize_rcu() waits for all such
+ * sections, providing the same guarantee as tlb_remove_table_sync_one() but
+ * without disrupting all CPUs with IPIs.

Yep, synchronize_rcu() is likely slower (longer wall clock time) but less disruptive to other CPUs.

Is it worth explaining here that this should be used when code really needs to _wait_ and *not* for freeing memory? Freeing memory should use RCU callbacks that don't cause latency spikes in this thread, not this.

Good point! Worth clarifying. Something like:

Note: Use this when code really needs to wait for synchronization, *not* for freeing memory. Memory freeing should use RCU callbacks that don't cause latency spikes in this thread.

Quoted 5 lines
Quoted 1 line

+ * Context: Can sleep/block. Cannot be called from any atomic context.

As a general rule, expressing constraints like this is best done in code, not comments, so:

Quoted 8 lines
	might_sleep();
or
	WARN_ON_ONCE(!in_atomic());

seem appropriate.

I didn't see any obvious warning like that in the top levels of synchronize_rcu().

Yep, synchronize_rcu() does call might_sleep() internally:

synchronize_rcu()
-> synchronize_rcu_normal()

    -> wait_rcu_gp()      -> __wait_rcu_gp()
        -> might_sleep()

But adding an explicit might_sleep() here makes the constraint more obvious. I'll add it :)

Quoted 12 lines
Quoted 12 lines
mm/mmu_gather.c +5 -0
+static void tlb_remove_table_sync_rcu(void)
+{
+	synchronize_rcu();
+}
+
  #else /* !CONFIG_MMU_GATHER_RCU_TABLE_FREE */
 
  static void tlb_remove_table_free(struct mmu_table_batch *batch)
@@ -303,6 +321,10 @@ static void tlb_remove_table_free(struct mmu_table_batch *batch)
  	__tlb_remove_table_free(batch);
  }
 
Quoted 7 lines

mm/mmu_gather.c static void tlb_remove_table_free(struct mmu_table_batch *batch) · line 328

Quoted 5 lines
mm/mmu_gather.c +4 -0
+static void tlb_remove_table_sync_rcu(void)
+{
+}
+
  #endif /* CONFIG_MMU_GATHER_RCU_TABLE_FREE */

This seems a _little_ dangerous to even define. We don't want this sneaking into use when it doesn't do anything.

This follows the same pattern as tlb_remove_table_sync_one(), which also has an empty stub in the !CONFIG_MMU_GATHER_RCU_TABLE_FREE path.

It should be in tlb.h like tlb_remove_table_sync_one(). Will put it there.

Thanks,
Lance

#6 lore raw
Quoted 4 lines · Lance Yang

Note: Use this when code really needs to wait for synchronization, *not* for freeing memory. Memory freeing should use RCU callbacks that don't cause latency spikes in this thread.

Yeah, but I'd probably mix it in with the other chit chat about non-atomic contexts. I'd also make it something like:

Do not use for freeing memory. Use RCU callbacks instead to avoid latency spikes.

to make the commands clear.

author
#7 lore raw
Quoted 6 lines · Dave Hansen
Quoted 4 lines · Lance Yang

Note: Use this when code really needs to wait for synchronization, *not* for freeing memory. Memory freeing should use RCU callbacks that don't cause latency spikes in this thread.

Quoted 7 lines

Yeah, but I'd probably mix it in with the other chit chat about non-atomic contexts. I'd also make it something like:

Do not use for freeing memory. Use RCU callbacks instead to avoid latency spikes.

to make the commands clear.

Got it, thanks. I'll update it in v2.