mm/mmu_gather: replace IPI with synchronize_rcu() when batch allocation fails
PATCH v2 1/1 mm/mmu_gather: replace IPI with synchronize_rcu() when batch allocation fails
From: Lance Yang <[email protected]>
When freeing page tables, we try to batch them. If batch allocation fails (GFP_NOWAIT), __tlb_remove_table_one() immediately frees the one without batching.
On !CONFIG_PT_RECLAIM, the fallback sends an IPI to all CPUs via tlb_remove_table_sync_one(). It disrupts all CPUs even when only a single process is unmapping memory. IPI broadcast was reported to hurt RT workloads[1].
tlb_remove_table_sync_one() synchronizes with lockless page-table walkers (e.g. GUP-fast) that rely on IRQ disabling. These walkers use local_irq_disable(), which is also an RCU read-side critical section.
This patch introduces tlb_remove_table_sync_rcu() which uses RCU grace period (synchronize_rcu()) instead of IPI broadcast. This provides the same guarantee as IPI but without disrupting all CPUs. Since batch allocation already failed, we are in a way slow path where sleeping is acceptable - we are in process context (unmap_region, exit_mmap) with only mmap_lock held. might_sleep() will catch any invalid context.
[1] https://lore.kernel.org/linux-mm/[email protected]/
---
v1 -> v2:
- Wrap synchronize_rcu() in tlb_remove_table_sync_rcu() with proper kerneldoc (per David)
- Add might_sleep() to make sleeping constraint explicit (per Dave)
- Clarify this is for synchronization, not memory freeing (per Dave)
- https://lore.kernel.org/linux-mm/[email protected]/
include/asm-generic/tlb.h | 4 ++++ mm/mmu_gather.c | 22 +++++++++++++++++++++- 2 files changed, 25 insertions(+), 1 deletion(-)
include/asm-generic/tlb.h +4 -0
@@ -251,6 +251,8 @@ static inline void tlb_remove_table(struct mmu_gather *tlb, void *table)void tlb_remove_table_sync_one (void ); +void tlb_remove_table_sync_rcu (void ); +#else #ifdef tlb_needs_table_invalidate
@@ -259,6 +261,8 @@ void tlb_remove_table_sync_one(void);static inline void tlb_remove_table_sync_one (void ) { } +static inline void tlb_remove_table_sync_rcu (void ) { } +#endif /* CONFIG_MMU_GATHER_RCU_TABLE_FREE */
mm/mmu_gather.c +21 -1
@@ -296,6 +296,26 @@ static void tlb_remove_table_free(struct mmu_table_batch *batch) call_rcu(&batch->rcu, tlb_remove_table_rcu); } +/** +* tlb_remove_table_sync_rcu() - synchronize with software page-table walkers +* +* Like tlb_remove_table_sync_one() but uses RCU grace period instead of IPI +* broadcast. Use in slow paths where sleeping is acceptable. +* +* Software/Lockless page-table walkers use local_irq_disable(), which is also +* an RCU read-side critical section. synchronize_rcu() waits for all such +* sections, providing the same guarantee as tlb_remove_table_sync_one() but +* without disrupting all CPUs with IPIs. +* +* Do not use for freeing memory. Use RCU callbacks instead to avoid latency +* spikes. Cannot be called from any atomic context. +*/ +void tlb_remove_table_sync_rcu (void ) +{ +might_sleep (); +synchronize_rcu (); +} +#else /* !CONFIG_MMU_GATHER_RCU_TABLE_FREE */ static void tlb_remove_table_free (struct mmu_table_batch *batch)
@@ -339,7 +359,7 @@ static inline void __tlb_remove_table_one(void *table)#else static inline void __tlb_remove_table_one (void *table) { -tlb_remove_table_sync_one (); +tlb_remove_table_sync_rcu ();__tlb_remove_table (table); }#endif /* CONFIG_PT_RECLAIM */
Signature · 1 line
2.49.0
Quoted 70 lines · Lance Yang
From: Lance Yang <[email protected]>
When freeing page tables, we try to batch them. If batch allocation fails (
GFP_NOWAIT),__tlb_remove_table_one()immediately frees the one without batching.On !CONFIG_PT_RECLAIM, the fallback sends an IPI to all CPUs via
tlb_remove_table_sync_one(). It disrupts all CPUs even when only a single process is unmapping memory. IPI broadcast was reported to hurt RT workloads[1].
tlb_remove_table_sync_one()synchronizes with lockless page-table walkers (e.g. GUP-fast) that rely on IRQ disabling. These walkers uselocal_irq_disable(), which is also an RCU read-side critical section.This patch introduces
tlb_remove_table_sync_rcu()which uses RCU grace period (synchronize_rcu()) instead of IPI broadcast. This provides the same guarantee as IPI but without disrupting all CPUs. Since batch allocation already failed, we are in a way slow path where sleeping is acceptable - we are in process context (unmap_region,exit_mmap) with onlymmap_lockheld.might_sleep()will catch any invalid context.[1] https://lore.kernel.org/linux-mm/[email protected]/
Suggested-by Peter Zijlstra <[email protected]>Suggested-by Dave Hansen <[email protected]>Suggested-by David Hildenbrand (Arm) <[email protected]>Signed-off-by Lance Yang <[email protected]>---
v1 -> v2:
- Wrapsynchronize_rcu()intlb_remove_table_sync_rcu()with proper kerneldoc (per David)
- Addmight_sleep()to make sleeping constraint explicit (per Dave)
- Clarify this is for synchronization, not memory freeing (per Dave)
- https://lore.kernel.org/linux-mm/[email protected]/include/asm-generic/tlb.h | 4 ++++ mm/mmu_gather.c | 22 +++++++++++++++++++++- 2 files changed, 25 insertions(+), 1 deletion(-)include/asm-generic/tlb.h +4
-0@@ -251,6 +251,8 @@ static inline void tlb_remove_table(struct mmu_gather *tlb, void *table)void tlb_remove_table_sync_one (void ); +void tlb_remove_table_sync_rcu (void ); +#else #ifdef tlb_needs_table_invalidate@@ -259,6 +261,8 @@ void tlb_remove_table_sync_one(void);static inline void tlb_remove_table_sync_one (void ) { } +static inline void tlb_remove_table_sync_rcu (void ) { } +#endif /* CONFIG_MMU_GATHER_RCU_TABLE_FREE */ mm/mmu_gather.c +0
-0
mm/mmu_gather.c static void tlb_remove_table_free(struct mmu_table_batch *batch) · line 300
Quoted 5 lines
mm/mmu_gather.c +2
-0@@ -296,6 +296,26 @@ static void tlb_remove_table_free(struct mmu_table_batch *batch)call_rcu (&batch->rcu, tlb_remove_table_rcu); } +/** +* tlb_remove_table_sync_rcu() - synchronize with software page-table walkers
Nit: no need for the "()"
Thanks!
Signature · 3 lines
Cheers, David
Quoted 74 lines · David Hildenbrand (Arm)
Quoted 73 lines · Lance Yang
From: Lance Yang <[email protected]>
When freeing page tables, we try to batch them. If batch allocation fails (
GFP_NOWAIT),__tlb_remove_table_one()immediately frees the one without batching.On !CONFIG_PT_RECLAIM, the fallback sends an IPI to all CPUs via
tlb_remove_table_sync_one(). It disrupts all CPUs even when only a single process is unmapping memory. IPI broadcast was reported to hurt RT workloads[1].
tlb_remove_table_sync_one()synchronizes with lockless page-table walkers (e.g. GUP-fast) that rely on IRQ disabling. These walkers uselocal_irq_disable(), which is also an RCU read-side critical section.This patch introduces
tlb_remove_table_sync_rcu()which uses RCU grace period (synchronize_rcu()) instead of IPI broadcast. This provides the same guarantee as IPI but without disrupting all CPUs. Since batch allocation already failed, we are in a way slow path where sleeping is acceptable - we are in process context (unmap_region,exit_mmap) with onlymmap_lockheld.might_sleep()will catch any invalid context.[1] https://lore.kernel.org/linux-mm/[email protected]/
Suggested-by Peter Zijlstra <[email protected]>Suggested-by Dave Hansen <[email protected]>Suggested-by David Hildenbrand (Arm) <[email protected]>Signed-off-by Lance Yang <[email protected]>---
v1 -> v2:
- Wrapsynchronize_rcu()intlb_remove_table_sync_rcu()with proper kerneldoc (per David)
- Addmight_sleep()to make sleeping constraint explicit (per Dave)
- Clarify this is for synchronization, not memory freeing (per Dave)
- https://lore.kernel.org/linux-mm/[email protected]/include/asm-generic/tlb.h | 4 ++++ mm/mmu_gather.c | 22 +++++++++++++++++++++- 2 files changed, 25 insertions(+), 1 deletion(-)include/asm-generic/tlb.h +4
-0@@ -251,6 +251,8 @@ static inline void tlb_remove_table(struct mmu_gather *tlb, void *table)void tlb_remove_table_sync_one (void ); +void tlb_remove_table_sync_rcu (void ); +#else #ifdef tlb_needs_table_invalidate@@ -259,6 +261,8 @@ void tlb_remove_table_sync_one(void);static inline void tlb_remove_table_sync_one (void ) { } +static inline void tlb_remove_table_sync_rcu (void ) { } +#endif /* CONFIG_MMU_GATHER_RCU_TABLE_FREE */ mm/mmu_gather.c +0
-0@@ -296,6 +296,26 @@ static void tlb_remove_table_free(struct mmu_table_batch *batch)call_rcu (&batch->rcu, tlb_remove_table_rcu); }
Quoted 4 lines
mm/mmu_gather.c static void tlb_remove_table_free(struct mmu_table_batch *batch) · line 300
Quoted 2 lines
mm/mmu_gather.c +2
-0+/** +* tlb_remove_table_sync_rcu() - synchronize with software page-table walkers Nit: no need for the "()"
Oops, @Andrew could you drop the "()" here when applying?
Quoted 4 lines
Thanks!
Thanks for taking time to review!
Peter Zijlstra
maintainerQuoted 15 lines · Lance Yang
From: Lance Yang <[email protected]>
When freeing page tables, we try to batch them. If batch allocation fails (
GFP_NOWAIT),__tlb_remove_table_one()immediately frees the one without batching.On !CONFIG_PT_RECLAIM, the fallback sends an IPI to all CPUs via
tlb_remove_table_sync_one(). It disrupts all CPUs even when only a single process is unmapping memory. IPI broadcast was reported to hurt RT workloads[1].
tlb_remove_table_sync_one()synchronizes with lockless page-table walkers (e.g. GUP-fast) that rely on IRQ disabling. These walkers uselocal_irq_disable(), which is also an RCU read-side critical section.
Quoted 6 lines
This patch introduces
tlb_remove_table_sync_rcu()which uses RCU grace period (synchronize_rcu()) instead of IPI broadcast. This provides the same guarantee as IPI but without disrupting all CPUs. Since batch allocation already failed, we are in a way slow path where sleeping is acceptable - we are in process context (unmap_region,exit_mmap) with onlymmap_lockheld.might_sleep()will catch any invalid context.
So sending the IPIs also requires non-atomic context, so change there.
What isn't explained, and very much not clear to me, is why tlb_remove_table_sync_one() is retained?
Quoted 38 lines
include/asm-generic/tlb.h +4
-0@@ -251,6 +251,8 @@ static inline void tlb_remove_table(struct mmu_gather *tlb, void *table)void tlb_remove_table_sync_one (void ); +void tlb_remove_table_sync_rcu (void ); +#else #ifdef tlb_needs_table_invalidate@@ -259,6 +261,8 @@ void tlb_remove_table_sync_one(void);static inline void tlb_remove_table_sync_one (void ) { } +static inline void tlb_remove_table_sync_rcu (void ) { } +#endif /* CONFIG_MMU_GATHER_RCU_TABLE_FREE */ mm/mmu_gather.c +8
-0@@ -296,6 +296,26 @@ static void tlb_remove_table_free(struct mmu_table_batch *batch)call_rcu (&batch->rcu, tlb_remove_table_rcu); } +/** +* tlb_remove_table_sync_rcu() - synchronize with software page-table walkers +* +* Like tlb_remove_table_sync_one() but uses RCU grace period instead of IPI +* broadcast. Use in slow paths where sleeping is acceptable. +* +* Software/Lockless page-table walkers use local_irq_disable(), which is also +* an RCU read-side critical section. synchronize_rcu() waits for all such
mm/mmu_gather.c static void tlb_remove_table_free(struct mmu_table_batch *batch) · line 316
Quoted 10 lines
mm/mmu_gather.c +10
-0+* sections, providing the same guarantee as tlb_remove_table_sync_one() but +* without disrupting all CPUs with IPIs. +* +* Do not use for freeing memory. Use RCU callbacks instead to avoid latency +* spikes. Cannot be called from any atomic context. +*/ +void tlb_remove_table_sync_rcu (void ) +{ +might_sleep (); +synchronize_rcu ();
synchronize_rcu() should end up in a might_sleep() at some point if it blocks (which it typically will).
Quoted 17 lines
mm/mmu_gather.c +3
-1+} +#else /* !CONFIG_MMU_GATHER_RCU_TABLE_FREE */ static void tlb_remove_table_free (struct mmu_table_batch *batch)@@ -339,7 +359,7 @@ static inline void __tlb_remove_table_one(void *table)#else static inline void __tlb_remove_table_one (void *table) { -tlb_remove_table_sync_one (); +tlb_remove_table_sync_rcu ();__tlb_remove_table (table); }#endif /* CONFIG_PT_RECLAIM */ Signature · 1 line
2.49.0
Peter Zijlstra
maintainerQuoted 16 lines · Peter Zijlstra
Quoted 15 lines · Lance Yang
From: Lance Yang <[email protected]>
When freeing page tables, we try to batch them. If batch allocation fails (
GFP_NOWAIT),__tlb_remove_table_one()immediately frees the one without batching.On !CONFIG_PT_RECLAIM, the fallback sends an IPI to all CPUs via
tlb_remove_table_sync_one(). It disrupts all CPUs even when only a single process is unmapping memory. IPI broadcast was reported to hurt RT workloads[1].
tlb_remove_table_sync_one()synchronizes with lockless page-table walkers (e.g. GUP-fast) that rely on IRQ disabling. These walkers uselocal_irq_disable(), which is also an RCU read-side critical section.
Quoted 8 lines
Quoted 6 lines
This patch introduces
tlb_remove_table_sync_rcu()which uses RCU grace period (synchronize_rcu()) instead of IPI broadcast. This provides the same guarantee as IPI but without disrupting all CPUs. Since batch allocation already failed, we are in a way slow path where sleeping is acceptable - we are in process context (unmap_region,exit_mmap) with onlymmap_lockheld.might_sleep()will catch any invalid context.So sending the IPIs also requires non-atomic context, so change there.
s/so/no/ -- typing hard
Quoted 16 lines · Peter Zijlstra
Quoted 15 lines · Lance Yang
From: Lance Yang <[email protected]>
When freeing page tables, we try to batch them. If batch allocation fails (
GFP_NOWAIT),__tlb_remove_table_one()immediately frees the one without batching.On !CONFIG_PT_RECLAIM, the fallback sends an IPI to all CPUs via
tlb_remove_table_sync_one(). It disrupts all CPUs even when only a single process is unmapping memory. IPI broadcast was reported to hurt RT workloads[1].
tlb_remove_table_sync_one()synchronizes with lockless page-table walkers (e.g. GUP-fast) that rely on IRQ disabling. These walkers uselocal_irq_disable(), which is also an RCU read-side critical section.
Quoted 8 lines
Quoted 6 lines
This patch introduces
tlb_remove_table_sync_rcu()which uses RCU grace period (synchronize_rcu()) instead of IPI broadcast. This provides the same guarantee as IPI but without disrupting all CPUs. Since batch allocation already failed, we are in a way slow path where sleeping is acceptable - we are in process context (unmap_region,exit_mmap) with onlymmap_lockheld.might_sleep()will catch any invalid context.So sending the IPIs also requires non-atomic context, so change there.
Yeah, you're right!
Quoted 2 lines
What isn't explained, and very much not clear to me, is why
tlb_remove_table_sync_one()is retained?
Good point. tlb_remove_table_sync_one() is still needed in:
1) khugepaged (mm/khugepaged.c) - after pmdp_collapse_flush()
2) tlb_finish_mmu() (tlb.h) - when tlb->fully_unshared_tables
3) ...
These are not slow paths like batch allocation failure. This patch only converts this obvious slow path first.
I'm working on converting the remaining callers as well, but not with RCU, looking at other options (e.g. targeted IPI).
Quoted 44 lines
Quoted 44 lines
include/asm-generic/tlb.h +4
-0@@ -251,6 +251,8 @@ static inline void tlb_remove_table(struct mmu_gather *tlb, void *table)void tlb_remove_table_sync_one (void ); +void tlb_remove_table_sync_rcu (void ); +#else #ifdef tlb_needs_table_invalidate@@ -259,6 +261,8 @@ void tlb_remove_table_sync_one(void);static inline void tlb_remove_table_sync_one (void ) { } +static inline void tlb_remove_table_sync_rcu (void ) { } +#endif /* CONFIG_MMU_GATHER_RCU_TABLE_FREE */ mm/mmu_gather.c +13
-0@@ -296,6 +296,26 @@ static void tlb_remove_table_free(struct mmu_table_batch *batch)call_rcu (&batch->rcu, tlb_remove_table_rcu); } +/** +* tlb_remove_table_sync_rcu() - synchronize with software page-table walkers +* +* Like tlb_remove_table_sync_one() but uses RCU grace period instead of IPI +* broadcast. Use in slow paths where sleeping is acceptable. +* +* Software/Lockless page-table walkers use local_irq_disable(), which is also +* an RCU read-side critical section. synchronize_rcu() waits for all such +* sections, providing the same guarantee as tlb_remove_table_sync_one() but +* without disrupting all CPUs with IPIs. +* +* Do not use for freeing memory. Use RCU callbacks instead to avoid latency +* spikes. Cannot be called from any atomic context.
Quoted 8 lines
mm/mmu_gather.c static void tlb_remove_table_free(struct mmu_table_batch *batch) · line 316
Quoted 5 lines
mm/mmu_gather.c +5
-0+*/ +void tlb_remove_table_sync_rcu (void ) +{ +might_sleep (); +synchronize_rcu ();
synchronize_rcu()should end up in amight_sleep()at some point if it blocks (which it typically will).
Right, will drop the explicit might_sleep() and "Cannot be called from any atomic context" from kerneldoc since both have the same requirements.
Thanks,
Lance
Quoted 39 lines · Lance Yang
Quoted 16 lines · Peter Zijlstra
Quoted 15 lines · Lance Yang
From: Lance Yang <[email protected]>
When freeing page tables, we try to batch them. If batch allocation fails (
GFP_NOWAIT),__tlb_remove_table_one()immediately frees the one without batching.On !CONFIG_PT_RECLAIM, the fallback sends an IPI to all CPUs via
tlb_remove_table_sync_one(). It disrupts all CPUs even when only a single process is unmapping memory. IPI broadcast was reported to hurt RT workloads[1].
tlb_remove_table_sync_one()synchronizes with lockless page-table walkers (e.g. GUP-fast) that rely on IRQ disabling. These walkers uselocal_irq_disable(), which is also an RCU read-side critical section.Quoted 8 lines
Quoted 6 lines
This patch introduces
tlb_remove_table_sync_rcu()which uses RCU grace period (synchronize_rcu()) instead of IPI broadcast. This provides the same guarantee as IPI but without disrupting all CPUs. Since batch allocation already failed, we are in a way slow path where sleeping is acceptable - we are in process context (unmap_region,exit_mmap) with onlymmap_lockheld.might_sleep()will catch any invalid context.So sending the IPIs also requires non-atomic context, so change there.
Yeah, you're right!
Quoted 2 lines
What isn't explained, and very much not clear to me, is why
tlb_remove_table_sync_one()is retained?Good point.
tlb_remove_table_sync_one()is still needed in:1) khugepaged (
mm/khugepaged.c) - afterpmdp_collapse_flush()
2)tlb_finish_mmu()(tlb.h) - whentlb->fully_unshared_tables
3) ...
Quoted 5 lines
These are not slow paths like batch allocation failure. This patch only converts this obvious slow path first.
I'm working on converting the remaining callers as well, but not with RCU, looking at other options (e.g. targeted IPI).
OK, so with that addition to the Changelog,
Quoted 43 lines · Peter Zijlstra
Quoted 42 lines · Lance Yang
Quoted 16 lines · Peter Zijlstra
Quoted 15 lines · Lance Yang
From: Lance Yang <[email protected]>
When freeing page tables, we try to batch them. If batch allocation fails (
GFP_NOWAIT),__tlb_remove_table_one()immediately frees the one without batching.On !CONFIG_PT_RECLAIM, the fallback sends an IPI to all CPUs via
tlb_remove_table_sync_one(). It disrupts all CPUs even when only a single process is unmapping memory. IPI broadcast was reported to hurt RT workloads[1].
tlb_remove_table_sync_one()synchronizes with lockless page-table walkers (e.g. GUP-fast) that rely on IRQ disabling. These walkers uselocal_irq_disable(), which is also an RCU read-side critical section.Quoted 8 lines
Quoted 6 lines
This patch introduces
tlb_remove_table_sync_rcu()which uses RCU grace period (synchronize_rcu()) instead of IPI broadcast. This provides the same guarantee as IPI but without disrupting all CPUs. Since batch allocation already failed, we are in a way slow path where sleeping is acceptable - we are in process context (unmap_region,exit_mmap) with onlymmap_lockheld.might_sleep()will catch any invalid context.So sending the IPIs also requires non-atomic context, so change there.
Yeah, you're right!
Quoted 2 lines
What isn't explained, and very much not clear to me, is why
tlb_remove_table_sync_one()is retained?Good point.
tlb_remove_table_sync_one()is still needed in:1) khugepaged (
mm/khugepaged.c) - afterpmdp_collapse_flush()
2)tlb_finish_mmu()(tlb.h) - whentlb->fully_unshared_tables
3) ...These are not slow paths like batch allocation failure. This patch only converts this obvious slow path first.
Quoted 6 lines
Quoted 2 lines
I'm working on converting the remaining callers as well, but not with RCU, looking at other options (e.g. targeted IPI).
OK, so with that addition to the Changelog,
Thanks for taking time to review!
Quoted 6 lines · Peter Zijlstra
mm/mmu_gather.c static void tlb_remove_table_free(struct mmu_table_batch *batch) · line 316
Quoted 4 lines
mm/mmu_gather.c +4
-0+void tlb_remove_table_sync_rcu (void ) +{ +might_sleep (); +synchronize_rcu ();
synchronize_rcu()should end up in amight_sleep()at some point if it blocks (which it typically will).
FWIW, I do prefer the explicit might_sleep() rather than leaving it to just the documentation. It just makes it easier to find bugs. I'm sure there's some crazy RCU variant that doesn't often sleep in synchronize_rcu(). ;)
If it's worth adding a line of comment, it's worth adding a line of code to actually keep folks honest. This is also going to be a pretty darn slow path so it shouldn't bloat anything too much.