mm/mmu_gather: replace IPI with synchronize_rcu() when batch allocation fails
PATCH 1/1 mm/mmu_gather: replace IPI with synchronize_rcu() when batch allocation fails
From: Lance Yang <[email protected]>
When freeing page tables, we try to batch them. If batch allocation fails (GFP_NOWAIT), __tlb_remove_table_one() immediately frees the one without batching.
On !CONFIG_PT_RECLAIM, the fallback sends an IPI to all CPUs via tlb_remove_table_sync_one(). It disrupts all CPUs even when only a single process is unmapping memory. IPI broadcast was reported to hurt RT workloads[1].
tlb_remove_table_sync_one() synchronizes with lockless page-table walkers (e.g. GUP-fast) that rely on IRQ disabling. These walkers use local_irq_disable(), which is also an RCU read-side critical section. synchronize_rcu() waits for all such sections to complete, providing the same guarantee as IPI but without disrupting all CPUs.
Since batch allocation already failed, we are in a way slow path, so replacing the IPI with synchronize_rcu() is fine.
We are in process context (unmap_region, exit_mmap) with only mmap_lock held, a sleeping lock. synchronize_rcu() will catch any invalid context via might_sleep().
[1] https://lore.kernel.org/linux-mm/[email protected]/
--- mm/mmu_gather.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
mm/mmu_gather.c +2 -1
@@ -339,7 +339,8 @@ static inline void __tlb_remove_table_one(void *table)#else static inline void __tlb_remove_table_one (void *table) { -tlb_remove_table_sync_one (); +if (IS_ENABLED (CONFIG_MMU_GATHER_RCU_TABLE_FREE )) +synchronize_rcu ();__tlb_remove_table (table); }#endif /* CONFIG_PT_RECLAIM */
Signature · 1 line
2.49.0
Quoted 24 lines · Lance Yang
From: Lance Yang <[email protected]>
When freeing page tables, we try to batch them. If batch allocation fails (
GFP_NOWAIT),__tlb_remove_table_one()immediately frees the one without batching.On !CONFIG_PT_RECLAIM, the fallback sends an IPI to all CPUs via
tlb_remove_table_sync_one(). It disrupts all CPUs even when only a single process is unmapping memory. IPI broadcast was reported to hurt RT workloads[1].
tlb_remove_table_sync_one()synchronizes with lockless page-table walkers (e.g. GUP-fast) that rely on IRQ disabling. These walkers uselocal_irq_disable(), which is also an RCU read-side critical section.synchronize_rcu()waits for all such sections to complete, providing the same guarantee as IPI but without disrupting all CPUs.Since batch allocation already failed, we are in a way slow path, so replacing the IPI with
synchronize_rcu()is fine.We are in process context (
unmap_region,exit_mmap) with onlymmap_lockheld, a sleeping lock.synchronize_rcu()will catch any invalid context viamight_sleep().
Quoted 8 lines
[1] https://lore.kernel.org/linux-mm/[email protected]/
Suggested-by Peter Zijlstra <[email protected]>Suggested-by Dave Hansen <[email protected]>Suggested-by David Hildenbrand (Arm) <[email protected]>
I think it was primarily Peter and Dave suggesting that :)
Quoted 10 lines
Signed-off-by Lance Yang <[email protected]>--- mm/mmu_gather.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)mm/mmu_gather.c +0
-0
mm/mmu_gather.c static inline void __tlb_remove_table_one(void *table) · line 343
Quoted 6 lines
mm/mmu_gather.c +2
-1@@ -339,7 +339,8 @@ static inline void __tlb_remove_table_one(void *table)#else static inline void __tlb_remove_table_one (void *table) { -tlb_remove_table_sync_one (); +if (IS_ENABLED (CONFIG_MMU_GATHER_RCU_TABLE_FREE )) +synchronize_rcu ();
That should work.
Reading all the comments for tlb_remove_table_smp_sync(), I wonder whether we should wrap that in a tlb_remove_table_sync_rcu() function, with a proper kerneldoc for the CONFIG_MMU_GATHER_RCU_TABLE_FREE variant where we discuss how this relates to tlb_remove_table_sync_one (and tlb_remove_table_smp_sync() .
Signature · 3 lines
Cheers, David
Quoted 27 lines · David Hildenbrand (Arm)
Quoted 26 lines · Lance Yang
From: Lance Yang <[email protected]>
When freeing page tables, we try to batch them. If batch allocation fails (
GFP_NOWAIT),__tlb_remove_table_one()immediately frees the one without batching.On !CONFIG_PT_RECLAIM, the fallback sends an IPI to all CPUs via
tlb_remove_table_sync_one(). It disrupts all CPUs even when only a single process is unmapping memory. IPI broadcast was reported to hurt RT workloads[1].
tlb_remove_table_sync_one()synchronizes with lockless page-table walkers (e.g. GUP-fast) that rely on IRQ disabling. These walkers uselocal_irq_disable(), which is also an RCU read-side critical section.synchronize_rcu()waits for all such sections to complete, providing the same guarantee as IPI but without disrupting all CPUs.Since batch allocation already failed, we are in a way slow path, so replacing the IPI with
synchronize_rcu()is fine.We are in process context (
unmap_region,exit_mmap) with onlymmap_lockheld, a sleeping lock.synchronize_rcu()will catch any invalid context viamight_sleep().
Quoted 8 lines
Quoted 6 lines
Suggested-by Peter Zijlstra <[email protected]>Suggested-by Dave Hansen <[email protected]>Suggested-by David Hildenbrand (Arm) <[email protected]>I think it was primarily Peter and Dave suggesting that :)
:)
Quoted 19 lines
Quoted 10 lines
Signed-off-by Lance Yang <[email protected]>--- mm/mmu_gather.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)mm/mmu_gather.c +0
-0mm/mmu_gather.c static inline void __tlb_remove_table_one(void *table) · line 343
Quoted 6 lines
mm/mmu_gather.c +2
-1@@ -339,7 +339,8 @@ static inline void __tlb_remove_table_one(void *table)#else static inline void __tlb_remove_table_one (void *table) { -tlb_remove_table_sync_one (); +if (IS_ENABLED (CONFIG_MMU_GATHER_RCU_TABLE_FREE )) +synchronize_rcu ();That should work.
Quoted 5 lines
Reading all the comments for
tlb_remove_table_smp_sync(), I wonder whether we should wrap that in atlb_remove_table_sync_rcu()function, with a proper kerneldoc for theCONFIG_MMU_GATHER_RCU_TABLE_FREEvariant where we discuss how this relates totlb_remove_table_sync_one(andtlb_remove_table_smp_sync().
Good point! That would be cleaner and better ;)
How about the following:
---8<---
mm/mmu_gather.c +23 -1
@@ -296,6 +296,24 @@ static void tlb_remove_table_free(struct mmu_table_batch *batch) call_rcu(&batch->rcu, tlb_remove_table_rcu); } +/** +* tlb_remove_table_sync_rcu() - synchronize with software page-table walkers +* +* Like tlb_remove_table_sync_one() but uses RCU grace period instead of IPI +* broadcast. Should be used in slow paths where sleeping is acceptable. +* +* Software/Lockless page-table walkers use local_irq_disable(), which is also +* an RCU read-side critical section. synchronize_rcu() waits for all such +* sections, providing the same guarantee as tlb_remove_table_sync_one() but +* without disrupting all CPUs with IPIs. +* +* Context: Can sleep/block. Cannot be called from any atomic context. +*/ +static void tlb_remove_table_sync_rcu (void ) +{ +synchronize_rcu (); +} +#else /* !CONFIG_MMU_GATHER_RCU_TABLE_FREE */ static void tlb_remove_table_free (struct mmu_table_batch *batch)
@@ -303,6 +321,10 @@ static void tlb_remove_table_free(struct mmu_table_batch *batch) __tlb_remove_table_free(batch); } +static void tlb_remove_table_sync_rcu (void ) +{ +} +#endif /* CONFIG_MMU_GATHER_RCU_TABLE_FREE */ /*
@@ -339,7 +361,7 @@ static inline void __tlb_remove_table_one(void *table)#else static inline void __tlb_remove_table_one (void *table) { -tlb_remove_table_sync_one (); +tlb_remove_table_sync_rcu ();__tlb_remove_table (table); }#endif /* CONFIG_PT_RECLAIM */
---
Thanks for the suggestion!
Lance
Quoted 121 lines · Lance Yang
Quoted 27 lines · David Hildenbrand (Arm)
Quoted 26 lines · Lance Yang
From: Lance Yang <[email protected]>
When freeing page tables, we try to batch them. If batch allocation fails (
GFP_NOWAIT),__tlb_remove_table_one()immediately frees the one without batching.On !CONFIG_PT_RECLAIM, the fallback sends an IPI to all CPUs via
tlb_remove_table_sync_one(). It disrupts all CPUs even when only a single process is unmapping memory. IPI broadcast was reported to hurt RT workloads[1].
tlb_remove_table_sync_one()synchronizes with lockless page-table walkers (e.g. GUP-fast) that rely on IRQ disabling. These walkers uselocal_irq_disable(), which is also an RCU read-side critical section.synchronize_rcu()waits for all such sections to complete, providing the same guarantee as IPI but without disrupting all CPUs.Since batch allocation already failed, we are in a way slow path, so replacing the IPI with
synchronize_rcu()is fine.We are in process context (
unmap_region,exit_mmap) with onlymmap_lockheld, a sleeping lock.synchronize_rcu()will catch any invalid context viamight_sleep().Quoted 8 lines
Quoted 6 lines
Suggested-by Peter Zijlstra <[email protected]>Suggested-by Dave Hansen <[email protected]>Suggested-by David Hildenbrand (Arm) <[email protected]>I think it was primarily Peter and Dave suggesting that :)
:)
Quoted 19 lines
Quoted 10 lines
Signed-off-by Lance Yang <[email protected]>--- mm/mmu_gather.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)mm/mmu_gather.c +0
-0mm/mmu_gather.c static inline void __tlb_remove_table_one(void *table) · line 343
Quoted 6 lines
mm/mmu_gather.c +2
-1@@ -339,7 +339,8 @@ static inline void __tlb_remove_table_one(void *table)#else static inline void __tlb_remove_table_one (void *table) { -tlb_remove_table_sync_one (); +if (IS_ENABLED (CONFIG_MMU_GATHER_RCU_TABLE_FREE )) +synchronize_rcu ();That should work.
Quoted 5 lines
Reading all the comments for
tlb_remove_table_smp_sync(), I wonder whether we should wrap that in atlb_remove_table_sync_rcu()function, with a proper kerneldoc for theCONFIG_MMU_GATHER_RCU_TABLE_FREEvariant where we discuss how this relates totlb_remove_table_sync_one(andtlb_remove_table_smp_sync().Good point! That would be cleaner and better ;)
How about the following:
---8<---
mm/mmu_gather.c +23
-2@@ -296,6 +296,24 @@ static void tlb_remove_table_free(struct mmu_table_batch *batch) call_rcu(&batch->rcu, tlb_remove_table_rcu); } +/** +* tlb_remove_table_sync_rcu() - synchronize with software page-table walkers +* +* Like tlb_remove_table_sync_one() but uses RCU grace period instead of IPI +* broadcast. Should be used in slow paths where sleeping is acceptable. +* +* Software/Lockless page-table walkers use local_irq_disable(), which is also +* an RCU read-side critical section. synchronize_rcu() waits for all such +* sections, providing the same guarantee as tlb_remove_table_sync_one() but +* without disrupting all CPUs with IPIs. +* +* Context: Can sleep/block. Cannot be called from any atomic context. +*/ +static void tlb_remove_table_sync_rcu (void ) +{ +synchronize_rcu (); +} +#else /* !CONFIG_MMU_GATHER_RCU_TABLE_FREE */ static void tlb_remove_table_free (struct mmu_table_batch *batch)@@ -303,6 +321,10 @@ static void tlb_remove_table_free(struct mmu_table_batch *batch) __tlb_remove_table_free(batch); } +static void tlb_remove_table_sync_rcu (void ) +{ +} +#endif /* CONFIG_MMU_GATHER_RCU_TABLE_FREE */ /* @@ -339,7 +361,7 @@ static inline void __tlb_remove_table_one(void *table)#else static inline void __tlb_remove_table_one (void *table) { -tlb_remove_table_sync_one (); +tlb_remove_table_sync_rcu ();__tlb_remove_table (table); }#endif /* CONFIG_PT_RECLAIM */ ---
Quoted 2 lines
Thanks for the suggestion!
Lance
LGTM, but let's hear other options.
Signature · 3 lines
Cheers, David
On 2/23/26 04:58, Lance Yang wrote:
...
Quoted 5 lines
+/**
+ *tlb_remove_table_sync_rcu()- synchronize with software page-table walkers
+ *
+ * Liketlb_remove_table_sync_one()but uses RCU grace period instead of IPI
+ * broadcast. Should be used in slow paths where sleeping is acceptable.
Just a nit on comments: Use imperative voice:
... Use in slow paths where sleeping is acceptable.
Quoted 4 lines
+ * Software/Lockless page-table walkers use
local_irq_disable(), which is also
+ * an RCU read-side critical section.synchronize_rcu()waits for all such
+ * sections, providing the same guarantee astlb_remove_table_sync_one()but
+ * without disrupting all CPUs with IPIs.
Yep, synchronize_rcu() is likely slower (longer wall clock time) but less disruptive to other CPUs.
Is it worth explaining here that this should be used when code really needs to _wait_ and *not* for freeing memory? Freeing memory should use RCU callbacks that don't cause latency spikes in this thread, not this.
Quoted 1 line
+ * Context: Can sleep/block. Cannot be called from any atomic context.
As a general rule, expressing constraints like this is best done in code, not comments, so:
might_sleep(); or WARN_ON_ONCE(!in_atomic());
seem appropriate.
I didn't see any obvious warning like that in the top levels of synchronize_rcu().
Quoted 9 lines
mm/mmu_gather.c +5
-0+static void tlb_remove_table_sync_rcu (void ) +{ +synchronize_rcu (); +} +#else /* !CONFIG_MMU_GATHER_RCU_TABLE_FREE */ static void tlb_remove_table_free (struct mmu_table_batch *batch)
mm/mmu_gather.c static void tlb_remove_table_free(struct mmu_table_batch *batch) · line 328
Quoted 8 lines
mm/mmu_gather.c +4
-0@@ -303,6 +321,10 @@ static void tlb_remove_table_free(struct mmu_table_batch *batch) __tlb_remove_table_free(batch); } +static void tlb_remove_table_sync_rcu (void ) +{ +} +#endif /* CONFIG_MMU_GATHER_RCU_TABLE_FREE */
This seems a _little_ dangerous to even define. We don't want this sneaking into use when it doesn't do anything.
Quoted 11 lines · Dave Hansen
On 2/23/26 04:58, Lance Yang wrote:
...Quoted 5 lines
+/**
+ *tlb_remove_table_sync_rcu()- synchronize with software page-table walkers
+ *
+ * Liketlb_remove_table_sync_one()but uses RCU grace period instead of IPI
+ * broadcast. Should be used in slow paths where sleeping is acceptable.Just a nit on comments: Use imperative voice:
... Use in slow paths where sleeping is acceptable.
Okay, thanks.
Quoted 11 lines
Quoted 4 lines
+ * Software/Lockless page-table walkers use
local_irq_disable(), which is also
+ * an RCU read-side critical section.synchronize_rcu()waits for all such
+ * sections, providing the same guarantee astlb_remove_table_sync_one()but
+ * without disrupting all CPUs with IPIs.Yep,
synchronize_rcu()is likely slower (longer wall clock time) but less disruptive to other CPUs.Is it worth explaining here that this should be used when code really needs to _wait_ and *not* for freeing memory? Freeing memory should use RCU callbacks that don't cause latency spikes in this thread, not this.
Good point! Worth clarifying. Something like:
Note: Use this when code really needs to wait for synchronization, *not* for freeing memory. Memory freeing should use RCU callbacks that don't cause latency spikes in this thread.
Quoted 5 lines
Quoted 1 line
+ * Context: Can sleep/block. Cannot be called from any atomic context.
As a general rule, expressing constraints like this is best done in code, not comments, so:
Quoted 8 lines
might_sleep(); or WARN_ON_ONCE(!in_atomic());seem appropriate.
I didn't see any obvious warning like that in the top levels of
synchronize_rcu().
Yep, synchronize_rcu() does call might_sleep() internally:
synchronize_rcu()
-> synchronize_rcu_normal()
-> wait_rcu_gp() -> __wait_rcu_gp()
-> might_sleep()But adding an explicit might_sleep() here makes the constraint more obvious. I'll add it :)
Quoted 12 lines
Quoted 12 lines
mm/mmu_gather.c +5
-0+static void tlb_remove_table_sync_rcu (void ) +{ +synchronize_rcu (); +} +#else /* !CONFIG_MMU_GATHER_RCU_TABLE_FREE */ static void tlb_remove_table_free (struct mmu_table_batch *batch)@@ -303,6 +321,10 @@ static void tlb_remove_table_free(struct mmu_table_batch *batch)__tlb_remove_table_free (batch); }
Quoted 7 lines
mm/mmu_gather.c static void tlb_remove_table_free(struct mmu_table_batch *batch) · line 328
Quoted 5 lines
mm/mmu_gather.c +4
-0+static void tlb_remove_table_sync_rcu (void ) +{ +} +#endif /* CONFIG_MMU_GATHER_RCU_TABLE_FREE */ This seems a _little_ dangerous to even define. We don't want this sneaking into use when it doesn't do anything.
This follows the same pattern as tlb_remove_table_sync_one(), which also has an empty stub in the !CONFIG_MMU_GATHER_RCU_TABLE_FREE path.
It should be in tlb.h like tlb_remove_table_sync_one(). Will put it there.
Thanks,
Lance
Quoted 4 lines · Lance Yang
Note: Use this when code really needs to wait for synchronization, *not* for freeing memory. Memory freeing should use RCU callbacks that don't cause latency spikes in this thread.
Yeah, but I'd probably mix it in with the other chit chat about non-atomic contexts. I'd also make it something like:
Do not use for freeing memory. Use RCU callbacks instead to avoid latency spikes.
to make the commands clear.
Quoted 6 lines · Dave Hansen
Quoted 4 lines · Lance Yang
Note: Use this when code really needs to wait for synchronization, *not* for freeing memory. Memory freeing should use RCU callbacks that don't cause latency spikes in this thread.
Quoted 7 lines
Yeah, but I'd probably mix it in with the other chit chat about non-atomic contexts. I'd also make it something like:
Do not use for freeing memory. Use RCU callbacks instead to avoid latency spikes.
to make the commands clear.
Got it, thanks. I'll update it in v2.