Skip to contentSkip to conversation

mm/mmu_gather: replace IPI with synchronize_rcu() when batch allocation fails

PATCH Superseded Lance Yang 4 participants 9 messages Feb 24

Subsystems (3)

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Lance Yang

author
#0 lore raw

PATCH v2 1/1 mm/mmu_gather: replace IPI with synchronize_rcu() when batch allocation fails

From: Lance Yang <[email protected]>

When freeing page tables, we try to batch them. If batch allocation fails (GFP_NOWAIT), __tlb_remove_table_one() immediately frees the one without batching.

On !CONFIG_PT_RECLAIM, the fallback sends an IPI to all CPUs via tlb_remove_table_sync_one(). It disrupts all CPUs even when only a single process is unmapping memory. IPI broadcast was reported to hurt RT workloads[1].

tlb_remove_table_sync_one() synchronizes with lockless page-table walkers (e.g. GUP-fast) that rely on IRQ disabling. These walkers use local_irq_disable(), which is also an RCU read-side critical section.

This patch introduces tlb_remove_table_sync_rcu() which uses RCU grace period (synchronize_rcu()) instead of IPI broadcast. This provides the same guarantee as IPI but without disrupting all CPUs. Since batch allocation already failed, we are in a way slow path where sleeping is acceptable - we are in process context (unmap_region, exit_mmap) with only mmap_lock held. might_sleep() will catch any invalid context.

[1] https://lore.kernel.org/linux-mm/[email protected]/

---
v1 -> v2:
- Wrap synchronize_rcu() in tlb_remove_table_sync_rcu() with proper kerneldoc (per David)
- Add might_sleep() to make sleeping constraint explicit (per Dave)
- Clarify this is for synchronization, not memory freeing (per Dave)
- https://lore.kernel.org/linux-mm/[email protected]/

 include/asm-generic/tlb.h |  4 ++++
 mm/mmu_gather.c           | 22 +++++++++++++++++++++-
 2 files changed, 25 insertions(+), 1 deletion(-)
include/asm-generic/tlb.h +4 -0
index 4aeac0c3d3f0..bdcc2778ac64 100644
@@ -251,6 +251,8 @@ static inline void tlb_remove_table(struct mmu_gather *tlb, void *table)
 
 void tlb_remove_table_sync_one(void);
 
+void tlb_remove_table_sync_rcu(void);
+
 #else
 
 #ifdef tlb_needs_table_invalidate
@@ -259,6 +261,8 @@ void tlb_remove_table_sync_one(void);
 
 static inline void tlb_remove_table_sync_one(void) { }
 
+static inline void tlb_remove_table_sync_rcu(void) { }
+
 #endif /* CONFIG_MMU_GATHER_RCU_TABLE_FREE */
 
 
mm/mmu_gather.c +21 -1
index fe5b6a031717..2c6fa8db55df 100644
@@ -296,6 +296,26 @@ static void tlb_remove_table_free(struct mmu_table_batch *batch)
 	call_rcu(&batch->rcu, tlb_remove_table_rcu);
 }
 
+/**
+ * tlb_remove_table_sync_rcu() - synchronize with software page-table walkers
+ *
+ * Like tlb_remove_table_sync_one() but uses RCU grace period instead of IPI
+ * broadcast. Use in slow paths where sleeping is acceptable.
+ *
+ * Software/Lockless page-table walkers use local_irq_disable(), which is also
+ * an RCU read-side critical section. synchronize_rcu() waits for all such
+ * sections, providing the same guarantee as tlb_remove_table_sync_one() but
+ * without disrupting all CPUs with IPIs.
+ *
+ * Do not use for freeing memory. Use RCU callbacks instead to avoid latency
+ * spikes. Cannot be called from any atomic context.
+ */
+void tlb_remove_table_sync_rcu(void)
+{
+	might_sleep();
+	synchronize_rcu();
+}
+
 #else /* !CONFIG_MMU_GATHER_RCU_TABLE_FREE */
 
 static void tlb_remove_table_free(struct mmu_table_batch *batch)
@@ -339,7 +359,7 @@ static inline void __tlb_remove_table_one(void *table)
 #else
 static inline void __tlb_remove_table_one(void *table)
 {
-	tlb_remove_table_sync_one();
+	tlb_remove_table_sync_rcu();
 	__tlb_remove_table(table);
 }
 #endif /* CONFIG_PT_RECLAIM */
Signature · 1 line
2.49.0
Acked-by on v2 1/1
#1 lore raw
Quoted 70 lines · Lance Yang

From: Lance Yang <[email protected]>

When freeing page tables, we try to batch them. If batch allocation fails (GFP_NOWAIT), __tlb_remove_table_one() immediately frees the one without batching.

On !CONFIG_PT_RECLAIM, the fallback sends an IPI to all CPUs via tlb_remove_table_sync_one(). It disrupts all CPUs even when only a single process is unmapping memory. IPI broadcast was reported to hurt RT workloads[1].

tlb_remove_table_sync_one() synchronizes with lockless page-table walkers (e.g. GUP-fast) that rely on IRQ disabling. These walkers use local_irq_disable(), which is also an RCU read-side critical section.

This patch introduces tlb_remove_table_sync_rcu() which uses RCU grace period (synchronize_rcu()) instead of IPI broadcast. This provides the same guarantee as IPI but without disrupting all CPUs. Since batch allocation already failed, we are in a way slow path where sleeping is acceptable - we are in process context (unmap_region, exit_mmap) with only mmap_lock held. might_sleep() will catch any invalid context.

[1] https://lore.kernel.org/linux-mm/[email protected]/

---
v1 -> v2:
- Wrap synchronize_rcu() in tlb_remove_table_sync_rcu() with proper kerneldoc (per David)
- Add might_sleep() to make sleeping constraint explicit (per Dave)
- Clarify this is for synchronization, not memory freeing (per Dave)
- https://lore.kernel.org/linux-mm/[email protected]/

 include/asm-generic/tlb.h |  4 ++++
 mm/mmu_gather.c           | 22 +++++++++++++++++++++-
 2 files changed, 25 insertions(+), 1 deletion(-)
include/asm-generic/tlb.h +4 -0
index 4aeac0c3d3f0..bdcc2778ac64 100644
@@ -251,6 +251,8 @@ static inline void tlb_remove_table(struct mmu_gather *tlb, void *table)
 
 void tlb_remove_table_sync_one(void);
 
+void tlb_remove_table_sync_rcu(void);
+
 #else
 
 #ifdef tlb_needs_table_invalidate
@@ -259,6 +261,8 @@ void tlb_remove_table_sync_one(void);
 
 static inline void tlb_remove_table_sync_one(void) { }
 
+static inline void tlb_remove_table_sync_rcu(void) { }
+
 #endif /* CONFIG_MMU_GATHER_RCU_TABLE_FREE */
 
 
mm/mmu_gather.c +0 -0
index fe5b6a031717..2c6fa8db55df 100644

mm/mmu_gather.c static void tlb_remove_table_free(struct mmu_table_batch *batch) · line 300

Quoted 5 lines
mm/mmu_gather.c +2 -0
@@ -296,6 +296,26 @@ static void tlb_remove_table_free(struct mmu_table_batch *batch)
 	call_rcu(&batch->rcu, tlb_remove_table_rcu);
 }
 
+/**
+ * tlb_remove_table_sync_rcu() - synchronize with software page-table walkers

Nit: no need for the "()"

Thanks!

Signature · 3 lines
Cheers,

David
author
#2 lore raw
Quoted 74 lines · David Hildenbrand (Arm)
Quoted 73 lines · Lance Yang

From: Lance Yang <[email protected]>

When freeing page tables, we try to batch them. If batch allocation fails (GFP_NOWAIT), __tlb_remove_table_one() immediately frees the one without batching.

On !CONFIG_PT_RECLAIM, the fallback sends an IPI to all CPUs via tlb_remove_table_sync_one(). It disrupts all CPUs even when only a single process is unmapping memory. IPI broadcast was reported to hurt RT workloads[1].

tlb_remove_table_sync_one() synchronizes with lockless page-table walkers (e.g. GUP-fast) that rely on IRQ disabling. These walkers use local_irq_disable(), which is also an RCU read-side critical section.

This patch introduces tlb_remove_table_sync_rcu() which uses RCU grace period (synchronize_rcu()) instead of IPI broadcast. This provides the same guarantee as IPI but without disrupting all CPUs. Since batch allocation already failed, we are in a way slow path where sleeping is acceptable - we are in process context (unmap_region, exit_mmap) with only mmap_lock held. might_sleep() will catch any invalid context.

[1] https://lore.kernel.org/linux-mm/[email protected]/

---
v1 -> v2:
- Wrap synchronize_rcu() in tlb_remove_table_sync_rcu() with proper kerneldoc (per David)
- Add might_sleep() to make sleeping constraint explicit (per Dave)
- Clarify this is for synchronization, not memory freeing (per Dave)
- https://lore.kernel.org/linux-mm/[email protected]/

  include/asm-generic/tlb.h |  4 ++++
  mm/mmu_gather.c           | 22 +++++++++++++++++++++-
  2 files changed, 25 insertions(+), 1 deletion(-)
include/asm-generic/tlb.h +4 -0
index 4aeac0c3d3f0..bdcc2778ac64 100644
@@ -251,6 +251,8 @@ static inline void tlb_remove_table(struct mmu_gather *tlb, void *table)
 
  void tlb_remove_table_sync_one(void);
 
+void tlb_remove_table_sync_rcu(void);
+
  #else
 
  #ifdef tlb_needs_table_invalidate
@@ -259,6 +261,8 @@ void tlb_remove_table_sync_one(void);
 
  static inline void tlb_remove_table_sync_one(void) { }
 
+static inline void tlb_remove_table_sync_rcu(void) { }
+
  #endif /* CONFIG_MMU_GATHER_RCU_TABLE_FREE */
 
 
mm/mmu_gather.c +0 -0
index fe5b6a031717..2c6fa8db55df 100644
@@ -296,6 +296,26 @@ static void tlb_remove_table_free(struct mmu_table_batch *batch)
  	call_rcu(&batch->rcu, tlb_remove_table_rcu);
  }
 
Quoted 4 lines

mm/mmu_gather.c static void tlb_remove_table_free(struct mmu_table_batch *batch) · line 300

Quoted 2 lines
mm/mmu_gather.c +2 -0
+/**
+ * tlb_remove_table_sync_rcu() - synchronize with software page-table walkers

Nit: no need for the "()"

Oops, @Andrew could you drop the "()" here when applying?

Quoted 4 lines

Thanks!

Thanks for taking time to review!

maintainer
#3 lore raw
In reply to Lance Yang
Quoted 15 lines · Lance Yang

From: Lance Yang <[email protected]>

When freeing page tables, we try to batch them. If batch allocation fails (GFP_NOWAIT), __tlb_remove_table_one() immediately frees the one without batching.

On !CONFIG_PT_RECLAIM, the fallback sends an IPI to all CPUs via tlb_remove_table_sync_one(). It disrupts all CPUs even when only a single process is unmapping memory. IPI broadcast was reported to hurt RT workloads[1].

tlb_remove_table_sync_one() synchronizes with lockless page-table walkers (e.g. GUP-fast) that rely on IRQ disabling. These walkers use local_irq_disable(), which is also an RCU read-side critical section.

Quoted 6 lines

This patch introduces tlb_remove_table_sync_rcu() which uses RCU grace period (synchronize_rcu()) instead of IPI broadcast. This provides the same guarantee as IPI but without disrupting all CPUs. Since batch allocation already failed, we are in a way slow path where sleeping is acceptable - we are in process context (unmap_region, exit_mmap) with only mmap_lock held. might_sleep() will catch any invalid context.

So sending the IPIs also requires non-atomic context, so change there.

What isn't explained, and very much not clear to me, is why tlb_remove_table_sync_one() is retained?

Quoted 38 lines
include/asm-generic/tlb.h +4 -0
index 4aeac0c3d3f0..bdcc2778ac64 100644
@@ -251,6 +251,8 @@ static inline void tlb_remove_table(struct mmu_gather *tlb, void *table)
 
 void tlb_remove_table_sync_one(void);
 
+void tlb_remove_table_sync_rcu(void);
+
 #else
 
 #ifdef tlb_needs_table_invalidate
@@ -259,6 +261,8 @@ void tlb_remove_table_sync_one(void);
 
 static inline void tlb_remove_table_sync_one(void) { }
 
+static inline void tlb_remove_table_sync_rcu(void) { }
+
 #endif /* CONFIG_MMU_GATHER_RCU_TABLE_FREE */
 
 
mm/mmu_gather.c +8 -0
index fe5b6a031717..2c6fa8db55df 100644
@@ -296,6 +296,26 @@ static void tlb_remove_table_free(struct mmu_table_batch *batch)
 	call_rcu(&batch->rcu, tlb_remove_table_rcu);
 }
 
+/**
+ * tlb_remove_table_sync_rcu() - synchronize with software page-table walkers
+ *
+ * Like tlb_remove_table_sync_one() but uses RCU grace period instead of IPI
+ * broadcast. Use in slow paths where sleeping is acceptable.
+ *
+ * Software/Lockless page-table walkers use local_irq_disable(), which is also
+ * an RCU read-side critical section. synchronize_rcu() waits for all such

mm/mmu_gather.c static void tlb_remove_table_free(struct mmu_table_batch *batch) · line 316

Quoted 10 lines
mm/mmu_gather.c +10 -0
+ * sections, providing the same guarantee as tlb_remove_table_sync_one() but
+ * without disrupting all CPUs with IPIs.
+ *
+ * Do not use for freeing memory. Use RCU callbacks instead to avoid latency
+ * spikes. Cannot be called from any atomic context.
+ */
+void tlb_remove_table_sync_rcu(void)
+{
+	might_sleep();
+	synchronize_rcu();

synchronize_rcu() should end up in a might_sleep() at some point if it blocks (which it typically will).

Quoted 17 lines
mm/mmu_gather.c +3 -1
+}
+
 #else /* !CONFIG_MMU_GATHER_RCU_TABLE_FREE */
 
 static void tlb_remove_table_free(struct mmu_table_batch *batch)
@@ -339,7 +359,7 @@ static inline void __tlb_remove_table_one(void *table)
 #else
 static inline void __tlb_remove_table_one(void *table)
 {
-	tlb_remove_table_sync_one();
+	tlb_remove_table_sync_rcu();
 	__tlb_remove_table(table);
 }
 #endif /* CONFIG_PT_RECLAIM */
Signature · 1 line
2.49.0
maintainer
#4 lore raw
Quoted 16 lines · Peter Zijlstra
Quoted 15 lines · Lance Yang

From: Lance Yang <[email protected]>

When freeing page tables, we try to batch them. If batch allocation fails (GFP_NOWAIT), __tlb_remove_table_one() immediately frees the one without batching.

On !CONFIG_PT_RECLAIM, the fallback sends an IPI to all CPUs via tlb_remove_table_sync_one(). It disrupts all CPUs even when only a single process is unmapping memory. IPI broadcast was reported to hurt RT workloads[1].

tlb_remove_table_sync_one() synchronizes with lockless page-table walkers (e.g. GUP-fast) that rely on IRQ disabling. These walkers use local_irq_disable(), which is also an RCU read-side critical section.

Quoted 8 lines
Quoted 6 lines

This patch introduces tlb_remove_table_sync_rcu() which uses RCU grace period (synchronize_rcu()) instead of IPI broadcast. This provides the same guarantee as IPI but without disrupting all CPUs. Since batch allocation already failed, we are in a way slow path where sleeping is acceptable - we are in process context (unmap_region, exit_mmap) with only mmap_lock held. might_sleep() will catch any invalid context.

So sending the IPIs also requires non-atomic context, so change there.

s/so/no/ -- typing hard

author
#5 lore raw
In reply to Peter Zijlstra
Quoted 16 lines · Peter Zijlstra
Quoted 15 lines · Lance Yang

From: Lance Yang <[email protected]>

When freeing page tables, we try to batch them. If batch allocation fails (GFP_NOWAIT), __tlb_remove_table_one() immediately frees the one without batching.

On !CONFIG_PT_RECLAIM, the fallback sends an IPI to all CPUs via tlb_remove_table_sync_one(). It disrupts all CPUs even when only a single process is unmapping memory. IPI broadcast was reported to hurt RT workloads[1].

tlb_remove_table_sync_one() synchronizes with lockless page-table walkers (e.g. GUP-fast) that rely on IRQ disabling. These walkers use local_irq_disable(), which is also an RCU read-side critical section.

Quoted 8 lines
Quoted 6 lines

This patch introduces tlb_remove_table_sync_rcu() which uses RCU grace period (synchronize_rcu()) instead of IPI broadcast. This provides the same guarantee as IPI but without disrupting all CPUs. Since batch allocation already failed, we are in a way slow path where sleeping is acceptable - we are in process context (unmap_region, exit_mmap) with only mmap_lock held. might_sleep() will catch any invalid context.

So sending the IPIs also requires non-atomic context, so change there.

Yeah, you're right!

Quoted 2 lines

What isn't explained, and very much not clear to me, is why tlb_remove_table_sync_one() is retained?

Good point. tlb_remove_table_sync_one() is still needed in:

1) khugepaged (mm/khugepaged.c) - after pmdp_collapse_flush()
2) tlb_finish_mmu() (tlb.h) - when tlb->fully_unshared_tables
3) ...

These are not slow paths like batch allocation failure. This patch only converts this obvious slow path first.

I'm working on converting the remaining callers as well, but not with RCU, looking at other options (e.g. targeted IPI).

Quoted 44 lines
Quoted 44 lines
include/asm-generic/tlb.h +4 -0
index 4aeac0c3d3f0..bdcc2778ac64 100644
@@ -251,6 +251,8 @@ static inline void tlb_remove_table(struct mmu_gather *tlb, void *table)
 
  void tlb_remove_table_sync_one(void);
 
+void tlb_remove_table_sync_rcu(void);
+
  #else
 
  #ifdef tlb_needs_table_invalidate
@@ -259,6 +261,8 @@ void tlb_remove_table_sync_one(void);
 
  static inline void tlb_remove_table_sync_one(void) { }
 
+static inline void tlb_remove_table_sync_rcu(void) { }
+
  #endif /* CONFIG_MMU_GATHER_RCU_TABLE_FREE */
 
 
mm/mmu_gather.c +13 -0
index fe5b6a031717..2c6fa8db55df 100644
@@ -296,6 +296,26 @@ static void tlb_remove_table_free(struct mmu_table_batch *batch)
  	call_rcu(&batch->rcu, tlb_remove_table_rcu);
  }
 
+/**
+ * tlb_remove_table_sync_rcu() - synchronize with software page-table walkers
+ *
+ * Like tlb_remove_table_sync_one() but uses RCU grace period instead of IPI
+ * broadcast. Use in slow paths where sleeping is acceptable.
+ *
+ * Software/Lockless page-table walkers use local_irq_disable(), which is also
+ * an RCU read-side critical section. synchronize_rcu() waits for all such
+ * sections, providing the same guarantee as tlb_remove_table_sync_one() but
+ * without disrupting all CPUs with IPIs.
+ *
+ * Do not use for freeing memory. Use RCU callbacks instead to avoid latency
+ * spikes. Cannot be called from any atomic context.
Quoted 8 lines

mm/mmu_gather.c static void tlb_remove_table_free(struct mmu_table_batch *batch) · line 316

Quoted 5 lines
mm/mmu_gather.c +5 -0
+ */
+void tlb_remove_table_sync_rcu(void)
+{
+	might_sleep();
+	synchronize_rcu();

synchronize_rcu() should end up in a might_sleep() at some point if it blocks (which it typically will).

Right, will drop the explicit might_sleep() and "Cannot be called from any atomic context" from kerneldoc since both have the same requirements.

Thanks,
Lance

maintainer Acked-by
#6 lore raw
Quoted 39 lines · Lance Yang
Quoted 16 lines · Peter Zijlstra
Quoted 15 lines · Lance Yang

From: Lance Yang <[email protected]>

When freeing page tables, we try to batch them. If batch allocation fails (GFP_NOWAIT), __tlb_remove_table_one() immediately frees the one without batching.

On !CONFIG_PT_RECLAIM, the fallback sends an IPI to all CPUs via tlb_remove_table_sync_one(). It disrupts all CPUs even when only a single process is unmapping memory. IPI broadcast was reported to hurt RT workloads[1].

tlb_remove_table_sync_one() synchronizes with lockless page-table walkers (e.g. GUP-fast) that rely on IRQ disabling. These walkers use local_irq_disable(), which is also an RCU read-side critical section.

Quoted 8 lines
Quoted 6 lines

This patch introduces tlb_remove_table_sync_rcu() which uses RCU grace period (synchronize_rcu()) instead of IPI broadcast. This provides the same guarantee as IPI but without disrupting all CPUs. Since batch allocation already failed, we are in a way slow path where sleeping is acceptable - we are in process context (unmap_region, exit_mmap) with only mmap_lock held. might_sleep() will catch any invalid context.

So sending the IPIs also requires non-atomic context, so change there.

Yeah, you're right!

Quoted 2 lines

What isn't explained, and very much not clear to me, is why tlb_remove_table_sync_one() is retained?

Good point. tlb_remove_table_sync_one() is still needed in:

1) khugepaged (mm/khugepaged.c) - after pmdp_collapse_flush()
2) tlb_finish_mmu() (tlb.h) - when tlb->fully_unshared_tables
3) ...

Quoted 5 lines

These are not slow paths like batch allocation failure. This patch only converts this obvious slow path first.

I'm working on converting the remaining callers as well, but not with RCU, looking at other options (e.g. targeted IPI).

OK, so with that addition to the Changelog,

author
#7 lore raw
Quoted 43 lines · Peter Zijlstra
Quoted 42 lines · Lance Yang
Quoted 16 lines · Peter Zijlstra
Quoted 15 lines · Lance Yang

From: Lance Yang <[email protected]>

When freeing page tables, we try to batch them. If batch allocation fails (GFP_NOWAIT), __tlb_remove_table_one() immediately frees the one without batching.

On !CONFIG_PT_RECLAIM, the fallback sends an IPI to all CPUs via tlb_remove_table_sync_one(). It disrupts all CPUs even when only a single process is unmapping memory. IPI broadcast was reported to hurt RT workloads[1].

tlb_remove_table_sync_one() synchronizes with lockless page-table walkers (e.g. GUP-fast) that rely on IRQ disabling. These walkers use local_irq_disable(), which is also an RCU read-side critical section.

Quoted 8 lines
Quoted 6 lines

This patch introduces tlb_remove_table_sync_rcu() which uses RCU grace period (synchronize_rcu()) instead of IPI broadcast. This provides the same guarantee as IPI but without disrupting all CPUs. Since batch allocation already failed, we are in a way slow path where sleeping is acceptable - we are in process context (unmap_region, exit_mmap) with only mmap_lock held. might_sleep() will catch any invalid context.

So sending the IPIs also requires non-atomic context, so change there.

Yeah, you're right!

Quoted 2 lines

What isn't explained, and very much not clear to me, is why tlb_remove_table_sync_one() is retained?

Good point. tlb_remove_table_sync_one() is still needed in:

1) khugepaged (mm/khugepaged.c) - after pmdp_collapse_flush()
2) tlb_finish_mmu() (tlb.h) - when tlb->fully_unshared_tables
3) ...

These are not slow paths like batch allocation failure. This patch only converts this obvious slow path first.

Quoted 6 lines
Quoted 2 lines

I'm working on converting the remaining callers as well, but not with RCU, looking at other options (e.g. targeted IPI).

OK, so with that addition to the Changelog,

Thanks for taking time to review!

#8 lore raw
In reply to Peter Zijlstra
Quoted 6 lines · Peter Zijlstra

mm/mmu_gather.c static void tlb_remove_table_free(struct mmu_table_batch *batch) · line 316

Quoted 4 lines
mm/mmu_gather.c +4 -0
+void tlb_remove_table_sync_rcu(void)
+{
+	might_sleep();
+	synchronize_rcu();

synchronize_rcu() should end up in a might_sleep() at some point if it blocks (which it typically will).

FWIW, I do prefer the explicit might_sleep() rather than leaving it to just the documentation. It just makes it easier to find bugs. I'm sure there's some crazy RCU variant that doesn't often sleep in synchronize_rcu(). ;)

If it's worth adding a line of comment, it's worth adding a line of code to actually keep folks honest. This is also going to be a pretty darn slow path so it shouldn't bloat anything too much.