shoot lazy tlbs
Subsystems (13)
Nicholas Piggin
reviewerv6 cover shoot lazy tlbs
It's time for that annual flamewar. Nothing really changed in core code to clean things up or make it better for x86 last year, so I think we can table that objection.
IIRC the situation left off with Andy proposing a different approach, and Linus preferring to shoot the lazies at exit time (piggybacking on the TLB flush IPI), which is what this series allows an arch to do. Discussion thread here:
I don't think there was any movement on this or other alternatives, or code cleanups since then, but correct me if I'm wrong.
Since v5 of this series, there has just been a minor rebase to upstream, and some tweaking of comments and code style. No functional changes.
Also included patch 5 which is the optimisation that combines final TLB shootdown with the lazy tlb mm shootdown IPIs. Included because Linus expected to see it. It works fine, but I have some other powerpc changes I would like to go ahead of it so I would like to take those through the powerpc tree. And actually giving it a release cycle without that optimization will help stress test the final IPI cleanup path too, which I would like.
Even without the last patch, the additional IPIs caused by shoot lazy is down in the noise so I'm not too concerned about it.
Thanks,
Nick
Nicholas Piggin (5):
lazy tlb: introduce lazy tlb mm refcount helper functions lazy tlb: allow lazy tlb mm refcounting to be configurable lazy tlb: shoot lazies, non-refcounting lazy tlb mm reference handling
scheme
powerpc/64s: enable MMU_LAZY_TLB_SHOOTDOWN powerpc/64s/radix: combine final TLB flush and lazy tlb mm shootdown
IPIs
Documentation/mm/active_mm.rst | 6 +++ arch/Kconfig | 32 ++++++++++++++ arch/arm/mach-rpc/ecard.c | 2 +- arch/powerpc/Kconfig | 1 + arch/powerpc/kernel/smp.c | 2 +- arch/powerpc/mm/book3s64/radix_tlb.c | 30 +++++++++++-- fs/exec.c | 2 +- include/linux/sched/mm.h | 28 ++++++++++++ kernel/cpu.c | 2 +- kernel/exit.c | 2 +- kernel/fork.c | 65 ++++++++++++++++++++++++++++ kernel/kthread.c | 21 +++++---- kernel/sched/core.c | 35 ++++++++++----- kernel/sched/sched.h | 4 +- 14 files changed, 205 insertions(+), 27 deletions(-)
Signature · 1 line
2.37.2
Nicholas Piggin
reviewerPATCH v6 1/5 lazy tlb: introduce lazy tlb mm refcount helper functions
Add explicit _lazy_tlb annotated functions for lazy tlb mm refcounting. This makes the lazy tlb mm references more obvious, and allows the refcounting scheme to be modified in later changes.
The only functional change is in kthread_use_mm/kthread_unuse_mm is because it is clever with refcounting: If it happens that the kthread's lazy tlb mm (active_mm) is the same as the mm to be used, the code doesn't touch the refcount but rather transfers the lazy refcount to used-mm refcount. If the lazy tlb mm refcount is no longer equivalent to the regular refcount, this trick can not be used. mmgrab a regular reference on mm to use, and mmdrop_lazy_tlb the previous active_mm.
--- arch/arm/mach-rpc/ecard.c | 2 +- arch/powerpc/kernel/smp.c | 2 +- arch/powerpc/mm/book3s64/radix_tlb.c | 4 ++-- fs/exec.c | 2 +- include/linux/sched/mm.h | 16 ++++++++++++++++ kernel/cpu.c | 2 +- kernel/exit.c | 2 +- kernel/kthread.c | 21 +++++++++++++-------- kernel/sched/core.c | 15 ++++++++------- 9 files changed, 44 insertions(+), 22 deletions(-)
arch/arm/mach-rpc/ecard.c +1 -1
@@ -253,7 +253,7 @@ static int ecard_init_mm(void) current->mm = mm; current->active_mm = mm;activate_mm (active_mm, mm); -mmdrop (active_mm); +mmdrop_lazy_tlb (active_mm);ecard_init_pgtables (mm);return 0 ; }
arch/powerpc/kernel/smp.c +1 -1
@@ -1611,7 +1611,7 @@ void start_secondary(void *unused)if (IS_ENABLED (CONFIG_PPC32 ))setup_kup (); -mmgrab (&init_mm); +mmgrab_lazy_tlb (&init_mm); current->active_mm = &init_mm;smp_store_cpu_info (cpu);
arch/powerpc/mm/book3s64/radix_tlb.c +2 -2
@@ -794,10 +794,10 @@ void exit_lazy_flush_tlb(struct mm_struct *mm, bool always_flush)if (current->active_mm == mm) {WARN_ON_ONCE (current->mm !=NULL );/* Is a kernel thread and is using mm as the lazy tlb */ -mmgrab (&init_mm); +mmgrab_lazy_tlb (&init_mm); current->active_mm = &init_mm;switch_mm_irqs_off (mm, &init_mm, current); -mmdrop (mm); +mmdrop_lazy_tlb (mm); }/*
fs/exec.c +1 -1
@@ -1033,7 +1033,7 @@ static int exec_mmap(struct mm_struct *mm)mmput (old_mm);return 0 ; } -mmdrop (active_mm); +mmdrop_lazy_tlb (active_mm);return 0 ; }
include/linux/sched/mm.h +16 -0
@@ -79,6 +79,22 @@ static inline void mmdrop_sched(struct mm_struct *mm) }#endif +/* Helpers for lazy TLB mm refcounting */ +static inline void mmgrab_lazy_tlb (struct mm_struct *mm) +{ +mmgrab (mm); +} + +static inline void mmdrop_lazy_tlb (struct mm_struct *mm) +{ +mmdrop (mm); +} + +static inline void mmdrop_lazy_tlb_sched (struct mm_struct *mm) +{ +mmdrop_sched (mm); +} +/** * mmget() - Pin the address space associated with a &struct mm_struct. * @mm: The address space to pin.
kernel/cpu.c +1 -1
@@ -623,7 +623,7 @@ static int finish_cpu(unsigned int cpu)*/ if (mm != &init_mm) idle->active_mm = &init_mm; -mmdrop (mm); +mmdrop_lazy_tlb (mm);return 0 ; }
kernel/exit.c +1 -1
@@ -537,7 +537,7 @@ static void exit_mm(void)return ;sync_mm_rss (mm);mmap_read_lock (mm); -mmgrab (mm); +mmgrab_lazy_tlb (mm);BUG_ON (mm != current->active_mm);/* more a memory barrier than a real lock */ task_lock (current);
kernel/kthread.c +13 -8
@@ -1410,14 +1410,19 @@ void kthread_use_mm(struct mm_struct *mm)WARN_ON_ONCE (!(tsk->flags &PF_KTHREAD ));WARN_ON_ONCE (tsk->mm); +/* +* It's possible that tsk->active_mm == mm here, but we must +* still mmgrab(mm) and mmdrop_lazy_tlb(active_mm), because lazy +* mm may not have its own refcount (see mmgrab/drop_lazy_tlb()). +*/ +mmgrab (mm); +task_lock (tsk);/* Hold off tlb flush IPIs while switching mm's */ local_irq_disable (); active_mm = tsk->active_mm; -if (active_mm != mm) { -mmgrab (mm); +if (active_mm != mm) tsk->active_mm = mm; - } tsk->mm = mm;membarrier_update_current_mm (mm);switch_mm_irqs_off (active_mm, mm, tsk);
@@ -1434,12 +1439,9 @@ void kthread_use_mm(struct mm_struct *mm)* memory barrier after storing to tsk->mm, before accessing * user-space memory. A full memory barrier for membarrier * {PRIVATE,GLOBAL}_EXPEDITED is implicitly provided by -* mmdrop(), or explicitly with smp_mb(). +* mmdrop_lazy_tlb(). */ -if (active_mm != mm) -mmdrop (active_mm); -else -smp_mb (); +mmdrop_lazy_tlb (active_mm); }EXPORT_SYMBOL_GPL (kthread_use_mm);
@@ -1467,10 +1469,13 @@ void kthread_unuse_mm(struct mm_struct *mm)local_irq_disable (); tsk->mm =NULL ;membarrier_update_current_mm (NULL ); +mmgrab_lazy_tlb (mm);/* active_mm is still 'mm' */ enter_lazy_tlb (mm, tsk);local_irq_enable ();task_unlock (tsk); + +mmdrop (mm); }EXPORT_SYMBOL_GPL (kthread_unuse_mm);
kernel/sched/core.c +8 -7
@@ -5140,13 +5140,14 @@ static struct rq *finish_task_switch(struct task_struct *prev)* rq->curr, before returning to userspace, so provide them here: * * - a full memory barrier for {PRIVATE,GLOBAL}_EXPEDITED, implicitly -* provided by mmdrop(), +* provided by mmdrop_lazy_tlb(), * - a sync_core for SYNC_CORE. */ if (mm) {membarrier_mm_sync_core_before_usermode (mm); -mmdrop_sched (mm); +mmdrop_lazy_tlb_sched (mm); } +if (unlikely (prev_state ==TASK_DEAD )) {if (prev->sched_class->task_dead) prev->sched_class->task_dead (prev);
@@ -5203,9 +5204,9 @@ context_switch(struct rq *rq, struct task_struct *prev,/* * kernel -> kernel lazy + transfer active -* user -> kernel lazy + mmgrab() active +* user -> kernel lazy + mmgrab_lazy_tlb() active * -* kernel -> user switch + mmdrop() active +* kernel -> user switch + mmdrop_lazy_tlb() active * user -> user switch */ if (!next->mm) {// to kernel
@@ -5213,7 +5214,7 @@ context_switch(struct rq *rq, struct task_struct *prev, next->active_mm = prev->active_mm;if (prev->mm)// from user -mmgrab (prev->active_mm); +mmgrab_lazy_tlb (prev->active_mm);else prev->active_mm =NULL ; }else {// to user
@@ -5230,7 +5231,7 @@ context_switch(struct rq *rq, struct task_struct *prev,lru_gen_use_mm (next->mm);if (!prev->mm) {// from kernel -/* will mmdrop() in finish_task_switch(). */ +/* will mmdrop_lazy_tlb() in finish_task_switch(). */ rq->prev_mm = prev->active_mm; prev->active_mm =NULL ; }
@@ -9859,7 +9860,7 @@ void __init sched_init(void)/* * The boot idle thread does lazy MMU switching as well: */ -mmgrab (&init_mm); +mmgrab_lazy_tlb (&init_mm);enter_lazy_tlb (&init_mm, current);/*
Signature · 1 line
2.37.2
Nicholas Piggin
reviewerPATCH v6 2/5 lazy tlb: allow lazy tlb mm refcounting to be configurable
In reply to Nicholas PigginAdd CONFIG_MMU_TLB_REFCOUNT which enables refcounting of the lazy tlb mm when it is context switched. This can be disabled by architectures that don't require this refcounting if they clean up lazy tlb mms when the last refcount is dropped. Currently this is always enabled, which is what existing code does, so the patch is effectively a no-op.
Rename rq->prev_mm to rq->prev_lazy_mm, because that's what it is.
--- Documentation/mm/active_mm.rst | 6 ++++++ arch/Kconfig | 17 +++++++++++++++++ include/linux/sched/mm.h | 18 +++++++++++++++--- kernel/sched/core.c | 22 ++++++++++++++++++---- kernel/sched/sched.h | 4 +++- 5 files changed, 59 insertions(+), 8 deletions(-)
Documentation/mm/active_mm.rst +6 -0
@@ -4,6 +4,12 @@ Active MM ========= +Note, the mm_count refcount may no longer include the "lazy" users +(running tasks with ->active_mm == mm && ->mm == NULL) on kernels +with CONFIG_MMU_LAZY_TLB_REFCOUNT=n. Taking and releasing these lazy +references must be done with mmgrab_lazy_tlb() and mmdrop_lazy_tlb() +helpers which abstracts this config option. + :: List: linux-kernel
arch/Kconfig +17 -0
@@ -465,6 +465,23 @@ config ARCH_WANT_IRQS_OFF_ACTIVATE_MM irqs disabled over activate_mm. Architectures that do IPI based TLB shootdowns should enable this. +# Use normal mm refcounting for MMU_LAZY_TLB kernel thread references. +# MMU_LAZY_TLB_REFCOUNT=n can improve the scalability of context switching +# to/from kernel threads when the same mm is running on a lot of CPUs (a large +# multi-threaded application), by reducing contention on the mm refcount. +# +# This can be disabled if the architecture ensures no CPUs are using an mm as a +# "lazy tlb" beyond its final refcount (i.e., by the time __mmdrop frees the mm +# or its kernel page tables). This could be arranged by arch_exit_mmap(), or +# final exit(2) TLB flush, for example. +# +# To implement this, an arch *must*: +# Ensure the _lazy_tlb variants of mmgrab/mmdrop are used when dropping the +# lazy reference of a kthread's ->active_mm (non-arch code has been converted +# already). +config MMU_LAZY_TLB_REFCOUNT + def_bool y + config ARCH_HAVE_NMI_SAFE_CMPXCHG bool
include/linux/sched/mm.h +15 -3
@@ -82,17 +82,29 @@ static inline void mmdrop_sched(struct mm_struct *mm)/* Helpers for lazy TLB mm refcounting */ static inline void mmgrab_lazy_tlb (struct mm_struct *mm) { -mmgrab (mm); +if (IS_ENABLED (CONFIG_MMU_LAZY_TLB_REFCOUNT )) +mmgrab (mm); }static inline void mmdrop_lazy_tlb (struct mm_struct *mm) { -mmdrop (mm); +if (IS_ENABLED (CONFIG_MMU_LAZY_TLB_REFCOUNT )) { +mmdrop (mm); + }else { +/* +* mmdrop_lazy_tlb must provide a full memory barrier, see the +* membarrier comment finish_task_switch which relies on this. +*/ +smp_mb (); + } }static inline void mmdrop_lazy_tlb_sched (struct mm_struct *mm) { -mmdrop_sched (mm); +if (IS_ENABLED (CONFIG_MMU_LAZY_TLB_REFCOUNT )) +mmdrop_sched (mm); +else +smp_mb ();// see above }/**
kernel/sched/core.c +18 -4
@@ -5081,7 +5081,7 @@ static struct rq *finish_task_switch(struct task_struct *prev) __releases(rq->lock) {struct rq *rq =this_rq (); -struct mm_struct *mm = rq->prev_mm; +struct mm_struct *mm =NULL ;unsigned int prev_state;/*
@@ -5100,7 +5100,10 @@ static struct rq *finish_task_switch(struct task_struct *prev) current->comm, current->pid, preempt_count()))preempt_count_set (FORK_PREEMPT_COUNT ); - rq->prev_mm =NULL ; +#ifdef CONFIG_MMU_LAZY_TLB_REFCOUNT + mm = rq->prev_lazy_mm; + rq->prev_lazy_mm =NULL ; +#endif /* * A task struct has one reference for the use as "current".
@@ -5231,9 +5234,20 @@ context_switch(struct rq *rq, struct task_struct *prev,lru_gen_use_mm (next->mm);if (!prev->mm) {// from kernel -/* will mmdrop_lazy_tlb() in finish_task_switch(). */ - rq->prev_mm = prev->active_mm; +#ifdef CONFIG_MMU_LAZY_TLB_REFCOUNT +/* Will mmdrop_lazy_tlb() in finish_task_switch(). */ + rq->prev_lazy_mm = prev->active_mm; prev->active_mm =NULL ; +#else +/* +* Without MMU_LAZY_TLB_REFCOUNT there is no lazy +* tracking (because no rq->prev_lazy_mm) in +* finish_task_switch, so no mmdrop_lazy_tlb(), so no +* memory barrier for membarrier (see the membarrier +* comment in finish_task_switch()). Do it here. +*/ +smp_mb (); +#endif } }
kernel/sched/sched.h +3 -1
@@ -1009,7 +1009,9 @@ struct rq {struct task_struct *idle;struct task_struct *stop;unsigned long next_balance; -struct mm_struct *prev_mm; +#ifdef CONFIG_MMU_LAZY_TLB_REFCOUNT +struct mm_struct *prev_lazy_mm; +#endif unsigned int clock_update_flags;u64 clock;
Signature · 1 line
2.37.2
Re: [PATCH v6 2/5] lazy tlb: allow lazy tlb mm refcounting to be configurable
Quoted 78 lines · Nicholas Piggin
Add
CONFIG_MMU_TLB_REFCOUNTwhich enables refcounting of the lazy tlb mm when it is context switched. This can be disabled by architectures that don't require this refcounting if they clean up lazy tlb mms when the last refcount is dropped. Currently this is always enabled, which is what existing code does, so the patch is effectively a no-op.Rename
rq->prev_mmtorq->prev_lazy_mm, because that's what it is.Signed-off-by Nicholas Piggin <[email protected]>--- Documentation/mm/active_mm.rst | 6 ++++++ arch/Kconfig | 17 +++++++++++++++++ include/linux/sched/mm.h | 18 +++++++++++++++--- kernel/sched/core.c | 22 ++++++++++++++++++---- kernel/sched/sched.h | 4 +++- 5 files changed, 59 insertions(+), 8 deletions(-)Documentation/mm/active_mm.rst +6
-0@@ -4,6 +4,12 @@ Active MM ========= +Note, the mm_count refcount may no longer include the "lazy" users +(running tasks with ->active_mm == mm && ->mm == NULL) on kernels +with CONFIG_MMU_LAZY_TLB_REFCOUNT=n. Taking and releasing these lazy +references must be done with mmgrab_lazy_tlb() and mmdrop_lazy_tlb() +helpers which abstracts this config option. + :: List: linux-kernelarch/Kconfig +17
-0@@ -465,6 +465,23 @@ config ARCH_WANT_IRQS_OFF_ACTIVATE_MM irqs disabled over activate_mm. Architectures that do IPI based TLB shootdowns should enable this. +# Use normal mm refcounting for MMU_LAZY_TLB kernel thread references. +# MMU_LAZY_TLB_REFCOUNT=n can improve the scalability of context switching +# to/from kernel threads when the same mm is running on a lot of CPUs (a large +# multi-threaded application), by reducing contention on the mm refcount. +# +# This can be disabled if the architecture ensures no CPUs are using an mm as a +# "lazy tlb" beyond its final refcount (i.e., by the time __mmdrop frees the mm +# or its kernel page tables). This could be arranged by arch_exit_mmap(), or +# final exit(2) TLB flush, for example. +# +# To implement this, an arch *must*: +# Ensure the _lazy_tlb variants of mmgrab/mmdrop are used when dropping the +# lazy reference of a kthread's ->active_mm (non-arch code has been converted +# already). +config MMU_LAZY_TLB_REFCOUNT + def_bool y + config ARCH_HAVE_NMI_SAFE_CMPXCHG boolinclude/linux/sched/mm.h +2
-2@@ -82,17 +82,29 @@ static inline void mmdrop_sched(struct mm_struct *mm)/* Helpers for lazy TLB mm refcounting */ static inline void mmgrab_lazy_tlb (struct mm_struct *mm) { -mmgrab (mm); +if (IS_ENABLED (CONFIG_MMU_LAZY_TLB_REFCOUNT )) +mmgrab (mm); }static inline void mmdrop_lazy_tlb (struct mm_struct *mm) { -mmdrop (mm);
include/linux/sched/mm.h static inline void mmdrop_sched(struct mm_struct *mm) · line 100
Quoted 10 lines
include/linux/sched/mm.h +9
-0+if (IS_ENABLED (CONFIG_MMU_LAZY_TLB_REFCOUNT )) { +mmdrop (mm); + }else { +/* +* mmdrop_lazy_tlb must provide a full memory barrier, see the +* membarrier comment finish_task_switch which relies on this. +*/ +smp_mb (); + } }
Considering the fact that mmdrop_lazy_tlb() replaced mmdrop() in various locations in which smp_mb() was not required, this comment might be confusing. IOW, for the cases in most cases where mmdrop_lazy_tlb() replaced mmdrop(), this comment was irrelevant, and therefore it now becomes confusing.
I am not sure the include the smp_mb() here instead of "open-coding" it helps.
include/linux/sched/mm.h static inline void mmdrop_sched(struct mm_struct *mm) · line 108
Quoted 9 lines
include/linux/sched/mm.h +4
-1static inline void mmdrop_lazy_tlb_sched (struct mm_struct *mm) { -mmdrop_sched (mm); +if (IS_ENABLED (CONFIG_MMU_LAZY_TLB_REFCOUNT )) +mmdrop_sched (mm); +else +smp_mb ();// see above }
Wrong style of comment.
Re: [PATCH v6 2/5] lazy tlb: allow lazy tlb mm refcounting to be configurable
Quoted 11 lines · Nadav Amit
Quoted 10 lines
+ if (
IS_ENABLED(CONFIG_MMU_LAZY_TLB_REFCOUNT)) {
+ mmdrop(mm);
+ } else {
+ /*
+ *mmdrop_lazy_tlbmust provide a full memory barrier, see the + * membarrier commentfinish_task_switchwhich relies on this. + */
+smp_mb();
+ }
}
Quoted 3 lines
Considering the fact that
mmdrop_lazy_tlb()replacedmmdrop()in various locations in whichsmp_mb()was not required, this comment might be confusing. IOW, for the cases in most cases wheremmdrop_lazy_tlb()replacedmmdrop(), this comment was irrelevant, and therefore it now becomes confusing.I am not sure the include the
smp_mb()here instead of "open-coding" it helps.
I think that I now understand why you do need the smp_mb() here, so ignore my comment.
Nicholas Piggin
reviewerRe: [PATCH v6 2/5] lazy tlb: allow lazy tlb mm refcounting to be configurable
Quoted 20 lines · Nadav Amit
Quoted 17 lines · Nadav Amit
Quoted 10 lines
+ if (
IS_ENABLED(CONFIG_MMU_LAZY_TLB_REFCOUNT)) {
+ mmdrop(mm);
+ } else {
+ /*
+ *mmdrop_lazy_tlbmust provide a full memory barrier, see the + * membarrier commentfinish_task_switchwhich relies on this. + */
+smp_mb();
+ }
}Considering the fact that
mmdrop_lazy_tlb()replacedmmdrop()in various locations in whichsmp_mb()was not required, this comment might be confusing. IOW, for the cases in most cases wheremmdrop_lazy_tlb()replacedmmdrop(), this comment was irrelevant, and therefore it now becomes confusing.
Quoted 4 lines
Quoted 2 lines
I am not sure the include the
smp_mb()here instead of "open-coding" it helps.I think that I now understand why you do need the
smp_mb()here, so ignore my comment.
For the moment it's basically a convenience thing so the caller does not have to care what option is configured. Possibly we could weaken it and do necessary barriers in callers if we consolidated to one option, but I'd have to be convinced it'd be worthwhile, because it would still make it deviate from mmdrop(), and we'd probably at least need a release barrier to drop the reference.
Thanks,
Nick
Nicholas Piggin
reviewerPATCH v6 3/5 lazy tlb: shoot lazies, non-refcounting lazy tlb mm reference handling scheme
In reply to Nicholas PigginOn big systems, the mm refcount can become highly contented when doing a lot of context switching with threaded applications (particularly switching between the idle thread and an application thread).
Abandoning lazy tlb slows switching down quite a bit in the important user->idle->user cases, so instead implement a non-refcounted scheme that causes __mmdrop() to IPI all CPUs in the mm_cpumask and shoot down any remaining lazy ones.
Shootdown IPIs cost could be an issue, but they have not been observed to be a serious problem with this scheme, because short-lived processes tend not to migrate CPUs much, therefore they don't get much chance to leave lazy tlb mm references on remote CPUs. There are a lot of options to reduce them if necessary.
--- arch/Kconfig | 15 ++++++++++++ kernel/fork.c | 65 +++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 80 insertions(+)
arch/Kconfig +15 -0
@@ -481,6 +481,21 @@ config ARCH_WANT_IRQS_OFF_ACTIVATE_MM # already). config MMU_LAZY_TLB_REFCOUNT def_bool y + depends on !MMU_LAZY_TLB_SHOOTDOWN + +# This option allows MMU_LAZY_TLB_REFCOUNT=n. It ensures no CPUs are using an +# mm as a lazy tlb beyond its last reference count, by shooting down these +# users before the mm is deallocated. __mmdrop() first IPIs all CPUs that may +# be using the mm as a lazy tlb, so that they may switch themselves to using +# init_mm for their active mm. mm_cpumask(mm) is used to determine which CPUs +# may be using mm as a lazy tlb mm. +# +# To implement this, an arch *must*: +# - At the time of the final mmdrop of the mm, ensure mm_cpumask(mm) contains +# at least all possible CPUs in which the mm is lazy. +# - It must meet the requirements for MMU_LAZY_TLB_REFCOUNT=n (see above). +config MMU_LAZY_TLB_SHOOTDOWN + bool config ARCH_HAVE_NMI_SAFE_CMPXCHG bool
kernel/fork.c +65 -0
77 lines not on this page · raw message
Signature · 1 line
2.37.2
Re: [PATCH v6 3/5] lazy tlb: shoot lazies, non-refcounting lazy tlb mm reference handling scheme
Quoted 4 lines · Nicholas Piggin
kernel/fork.c static void check_mm(struct mm_struct *mm) · line 799
Quoted 10 lines
kernel/fork.c +10
-0+static void do_shoot_lazy_tlb (void *arg) +{ +struct mm_struct *mm = arg; + +if (current->active_mm == mm) { +WARN_ON_ONCE (current->mm); + current->active_mm = &init_mm; +switch_mm (mm, &init_mm, current); + } +}
I might be out of touch - doesn’t a flush already take place when we free the page-tables, at least on common cases on x86?
IIUC exit_mmap() would free page-tables, and whenever page-tables are freed, on x86, we do shootdown regardless to whether the target CPU TLB state marks is_lazy. Then, flush_tlb_func() should call switch_mm_irqs_off() and everything should be fine, no?
[ I understand you care about powerpc, just wondering on the effect on x86 ]
Nicholas Piggin
reviewerRe: [PATCH v6 3/5] lazy tlb: shoot lazies, non-refcounting lazy tlb mm reference handling scheme
Quoted 18 lines · Nadav Amit
Quoted 4 lines · Nicholas Piggin
kernel/fork.c static void check_mm(struct mm_struct *mm) · line 799
Quoted 10 lines
kernel/fork.c +10
-0+static void do_shoot_lazy_tlb (void *arg) +{ +struct mm_struct *mm = arg; + +if (current->active_mm == mm) { +WARN_ON_ONCE (current->mm); + current->active_mm = &init_mm; +switch_mm (mm, &init_mm, current); + } +}I might be out of touch - doesn’t a flush already take place when we free the page-tables, at least on common cases on x86?
Quoted 6 lines
IIUC
exit_mmap()would free page-tables, and whenever page-tables are freed, on x86, we do shootdown regardless to whether the target CPU TLB state marksis_lazy. Then,flush_tlb_func()should callswitch_mm_irqs_off()and everything should be fine, no?[ I understand you care about powerpc, just wondering on the effect on x86 ]
If you can easily piggyback on IPI work you already do in exit_mmap then that's likely to be preferable. I don't know the details of x86 these days but there is some discussion about it in last year's thread, it sounded quite feasible.
This is stil required at final __mmdrop() time because it's still possible that lazy mm refs will need to be cleaned. exit_mmap() itself explicitly creates one, so if the __mmdrop() runs on a different CPU, then there's one. kthreads using the mm could create others. If that part of it is unclear or under-commented, I can try improve it.
Thanks,
Nick
Nicholas Piggin
reviewerRe: [PATCH v6 3/5] lazy tlb: shoot lazies, non-refcounting lazy tlb mm reference handling scheme
In reply to Nadav AmitQuoted 18 lines · Nadav Amit
Quoted 4 lines · Nicholas Piggin
kernel/fork.c static void check_mm(struct mm_struct *mm) · line 799
Quoted 10 lines
kernel/fork.c +10
-0+static void do_shoot_lazy_tlb (void *arg) +{ +struct mm_struct *mm = arg; + +if (current->active_mm == mm) { +WARN_ON_ONCE (current->mm); + current->active_mm = &init_mm; +switch_mm (mm, &init_mm, current); + } +}I might be out of touch - doesn’t a flush already take place when we free the page-tables, at least on common cases on x86?
Quoted 6 lines
IIUC
exit_mmap()would free page-tables, and whenever page-tables are freed, on x86, we do shootdown regardless to whether the target CPU TLB state marksis_lazy. Then,flush_tlb_func()should callswitch_mm_irqs_off()and everything should be fine, no?[ I understand you care about powerpc, just wondering on the effect on x86 ]
Now I come to think of it, Rik had done this for x86 a while back.
https://lore.kernel.org/all/[email protected]/
I didn't know about it when I wrote this, so I never dug into why it didn't get merged. It might have missed the final __mmdrop races but I'm not not sure, x86 lazy tlb mode is too complicated to know at a glance. I would check with him though.
Thanks,
Nick
Re: [PATCH v6 3/5] lazy tlb: shoot lazies, non-refcounting lazy tlb mm reference handling scheme
Quoted 26 lines · Nicholas Piggin
Quoted 18 lines · Nadav Amit
Quoted 4 lines · Nicholas Piggin
kernel/fork.c static void check_mm(struct mm_struct *mm) · line 799
Quoted 10 lines
kernel/fork.c +10
-0+static void do_shoot_lazy_tlb (void *arg) +{ +struct mm_struct *mm = arg; + +if (current->active_mm == mm) { +WARN_ON_ONCE (current->mm); + current->active_mm = &init_mm; +switch_mm (mm, &init_mm, current); + } +}I might be out of touch - doesn’t a flush already take place when we free the page-tables, at least on common cases on x86?
Quoted 6 lines
IIUC
exit_mmap()would free page-tables, and whenever page-tables are freed, on x86, we do shootdown regardless to whether the target CPU TLB state marksis_lazy. Then,flush_tlb_func()should callswitch_mm_irqs_off()and everything should be fine, no?[ I understand you care about powerpc, just wondering on the effect on x86 ]
Quoted 8 lines
Now I come to think of it, Rik had done this for x86 a while back.
https://lore.kernel.org/all/[email protected]/
I didn't know about it when I wrote this, so I never dug into why it didn't get merged. It might have missed the final
__mmdropraces but I'm not not sure, x86 lazy tlb mode is too complicated to know at a glance. I would check with him though.
My point was that naturally (i.e., as done today), when exit_mmap() is done, you release the page tables (not just the pages). On x86 it means that you also send shootdown IPI to all the *lazy* CPUs to perform a flush, so they would exit the lazy mode.
[ this should be true for 99% of the cases, excluding cases where there
were not page-tables, for instance ]
So the patch of Rik, I think, does not help in the common cases, although it may perhaps make implicit actions more explicit in the code.
Nicholas Piggin
reviewerRe: [PATCH v6 3/5] lazy tlb: shoot lazies, non-refcounting lazy tlb mm reference handling scheme
Quoted 43 lines · Nadav Amit
Quoted 26 lines · Nicholas Piggin
Quoted 18 lines · Nadav Amit
Quoted 4 lines · Nicholas Piggin
kernel/fork.c static void check_mm(struct mm_struct *mm) · line 799
Quoted 10 lines
kernel/fork.c +10
-0+static void do_shoot_lazy_tlb (void *arg) +{ +struct mm_struct *mm = arg; + +if (current->active_mm == mm) { +WARN_ON_ONCE (current->mm); + current->active_mm = &init_mm; +switch_mm (mm, &init_mm, current); + } +}I might be out of touch - doesn’t a flush already take place when we free the page-tables, at least on common cases on x86?
Quoted 6 lines
IIUC
exit_mmap()would free page-tables, and whenever page-tables are freed, on x86, we do shootdown regardless to whether the target CPU TLB state marksis_lazy. Then,flush_tlb_func()should callswitch_mm_irqs_off()and everything should be fine, no?[ I understand you care about powerpc, just wondering on the effect on x86 ]
Quoted 8 lines
Now I come to think of it, Rik had done this for x86 a while back.
https://lore.kernel.org/all/[email protected]/
I didn't know about it when I wrote this, so I never dug into why it didn't get merged. It might have missed the final
__mmdropraces but I'm not not sure, x86 lazy tlb mode is too complicated to know at a glance. I would check with him though.My point was that naturally (i.e., as done today), when
exit_mmap()is done, you release the page tables (not just the pages). On x86 it means that you also send shootdown IPI to all the *lazy* CPUs to perform a flush, so they would exit the lazy mode.
Quoted 5 lines
[ this should be true for 99% of the cases, excluding cases where there
were not page-tables, for instance ]So the patch of Rik, I think, does not help in the common cases, although it may perhaps make implicit actions more explicit in the code.
If that's what it does, then sure. IIRC x86 didn't used to work that way long ago, but you would know what it does today. You might find it doesn't need much arch change to work. OTOH Andy has major problems with active_mm and some other x86 use-after-free weirdness that that I wasn't able to comprehend. He'll be naking x86 implementation until that's all cleaned up so better try to understand what's going on with that first.
Thanks,
Nick
Nicholas Piggin
reviewerPATCH v6 4/5 powerpc/64s: enable MMU_LAZY_TLB_SHOOTDOWN
In reply to Nicholas PigginOn a 16-socket 192-core POWER8 system, a context switching benchmark with as many software threads as CPUs (so each switch will go in and out of idle), upstream can achieve a rate of about 1 million context switches per second, due to contention on the mm refcount.
64s meets the prerequisites for CONFIG_MMU_LAZY_TLB_SHOOTDOWN, so enable the option. This increases the above benchmark to 118 million context switches per second.
This generates 314 additional IPI interrupts on a 144 CPU system doing a kernel compile, which is in the noise in terms of kernel cycles.
--- arch/powerpc/Kconfig | 1 + 1 file changed, 1 insertion(+)
arch/powerpc/Kconfig +1 -0
@@ -265,6 +265,7 @@ config PPC select MMU_GATHER_PAGE_SIZE select MMU_GATHER_RCU_TABLE_FREE select MMU_GATHER_MERGE_VMAS + select MMU_LAZY_TLB_SHOOTDOWN if PPC_BOOK3S_64 select MODULES_USE_ELF_RELA select NEED_DMA_MAP_STATE if PPC64 || NOT_COHERENT_CACHE select NEED_PER_CPU_EMBED_FIRST_CHUNK if PPC64
Signature · 1 line
2.37.2
Re: [PATCH v6 4/5] powerpc/64s: enable MMU_LAZY_TLB_SHOOTDOWN
[ Adding a few more x86 and arm64 maintainers - while linux-arch is the right mailing list, I'm not convinced people actually follow it all that closely ]
Quoted 9 lines · Nicholas Piggin
On a 16-socket 192-core POWER8 system, a context switching benchmark with as many software threads as CPUs (so each switch will go in and out of idle), upstream can achieve a rate of about 1 million context switches per second, due to contention on the mm refcount.
64s meets the prerequisites for
CONFIG_MMU_LAZY_TLB_SHOOTDOWN, so enable the option. This increases the above benchmark to 118 million context switches per second.
Well, the 1M -> 118M change does seem like a good reason for this series.
The patches certainly don't look offensive to me, so Ack as far as I'm concerned, but honestly, it's been some time since I've personally been active on the idle and lazy TLB code, so that ack is probably largely worthless.
If anything, my main reaction to this all is to wonder whether the config option is a good idea - maybe we could do this unconditionally, and make the source code (and logic) simpler to follow when you don't have to worry about the CONFIG_MMU_LAZY_TLB_REFCOUNT option.
I wouldn't be surprised to hear that x86 can have the same issue where the mm_struct refcount is a bigger issue than the possibility of an extra TLB shootdown at the final exit time.
But having the config options as a way to switch people over gradually (and perhaps then removing it later) doesn't sound wrong to me either.
And I personally find the argument in patch 3/5 fairly convincing:
Shootdown IPIs cost could be an issue, but they have not been observed to be a serious problem with this scheme, because short-lived processes tend not to migrate CPUs much, therefore they don't get much chance to leave lazy tlb mm references on remote CPUs.
Andy? PeterZ? Catalin?
Nick - it might be good to link to the actual benchmark, and let people who have access to big machines perhaps just try it out on non-powerpc platforms...
Linus
Nicholas Piggin
reviewerRe: [PATCH v6 4/5] powerpc/64s: enable MMU_LAZY_TLB_SHOOTDOWN
Quoted 11 lines · Linus Torvalds
[ Adding a few more x86 and arm64 maintainers - while linux-arch is the right mailing list, I'm not convinced people actually follow it all that closely ]
Quoted 6 lines · Nicholas Piggin
On a 16-socket 192-core POWER8 system, a context switching benchmark with as many software threads as CPUs (so each switch will go in and out of idle), upstream can achieve a rate of about 1 million context switches per second, due to contention on the mm refcount.
Quoted 5 lines
Quoted 3 lines
64s meets the prerequisites for
CONFIG_MMU_LAZY_TLB_SHOOTDOWN, so enable the option. This increases the above benchmark to 118 million context switches per second.Well, the 1M -> 118M change does seem like a good reason for this series.
It was an artificial corner case, mind you. I don't think it's a reason to panic and likely smaller systems with faster atomics will care far less than our big 2-hop systems.
Benchmark is will-it-scale:
./context_switch1_threads -t 768
min:2174 max:2690 total:1827952
33.52% [k] finish_task_switch
27.26% [k] interrupt_return
22.66% [k] __schedule
2.30% [k] _raw_spin_trylock ./context_switch1_threads -t 1536
min:103000 max:120100 total:177201906
The top case has 1/2 the switching pairs to available CPU, which makes them all switch the same mm between real and lazy. Bottom case is just switching between user threads so that doesn't hit the lazy refcount.
Quoted 10 lines
The patches certainly don't look offensive to me, so Ack as far as I'm concerned, but honestly, it's been some time since I've personally been active on the idle and lazy TLB code, so that ack is probably largely worthless.
If anything, my main reaction to this all is to wonder whether the config option is a good idea - maybe we could do this unconditionally, and make the source code (and logic) simpler to follow when you don't have to worry about the
CONFIG_MMU_LAZY_TLB_REFCOUNToption.
Quoted 6 lines
I wouldn't be surprised to hear that x86 can have the same issue where the
mm_structrefcount is a bigger issue than the possibility of an extra TLB shootdown at the final exit time.But having the config options as a way to switch people over gradually (and perhaps then removing it later) doesn't sound wrong to me either.
IMO it's trivial enough that we could carry both, but everything's a straw on the camel's back so if we can consolidate it would always be preferebale. Let's see how it plays out for a few releases.
Quoted 7 lines
And I personally find the argument in patch 3/5 fairly convincing:
Shootdown IPIs cost could be an issue, but they have not been observed to be a serious problem with this scheme, because short-lived processes tend not to migrate CPUs much, therefore they don't get much chance to leave lazy tlb mm references on remote CPUs.
Quoted 5 lines
Andy? PeterZ? Catalin?
Nick - it might be good to link to the actual benchmark, and let people who have access to big machines perhaps just try it out on non-powerpc platforms...
Yep good point, I'll put it in the changelog. I might submit another round to Andrew in a bit with acks and any minor tweaks and minus the last patch, assuming no major changes or objections.
Thanks,
Nick
Nicholas Piggin
reviewerPATCH v6 5/5 powerpc/64s/radix: combine final TLB flush and lazy tlb mm shootdown IPIs
In reply to Nicholas Piggin** Not for merge **
CONFIG_MMU_LAZY_TLB_SHOOTDOWN that requires IPIs to clear the "lazy tlb" references to an mm that is being freed. With the radix MMU, the final userspace exit TLB flush can be performed with IPIs, and those IPIs can also clear lazy tlb mm references, which mostly eliminates the final IPIs required by MMU_LAZY_TLB_SHOOTDOWN.
This does mean the final TLB flush is not done with TLBIE, which can be faster than IPI+TLBIEL, but we would have to do those IPIs for lazy shootdown so using TLBIEL should be a win.
The final cpumask test and possible IPIs are still needed to clean up some rare race cases. We could prevent those entirely (e.g., prevent new lazy tlb mm references if userspace has gone away, or move the final TLB flush later), but I'd have to see actual numbers that matter before adding any more complexity for it. I can't imagine it would ever be worthwhile.
This takes lazy tlb mm shootdown IPI interrupts from 314 to 3 on a 144 CPU system doing a kernel compile. It also takes care of the one potential problem workload which is a short-lived process with multiple CPU-bound threads that want to be spread to other CPUs, because the mm exit happens after the process is back to single-threaded.
--- arch/powerpc/mm/book3s64/radix_tlb.c | 26 +++++++++++++++++++++++++- 1 file changed, 25 insertions(+), 1 deletion(-)
arch/powerpc/mm/book3s64/radix_tlb.c +25 -1
@@ -1303,7 +1303,31 @@ void radix__tlb_flush(struct mmu_gather *tlb)* See the comment for radix in arch_exit_mmap(). */ if (tlb->fullmm || tlb->need_flush_all) { -__flush_all_mm (mm, true); +if (IS_ENABLED (CONFIG_MMU_LAZY_TLB_SHOOTDOWN )) { +/* +* Shootdown based lazy tlb mm refcounting means we +* have to IPI everyone in the mm_cpumask anyway soon +* when the mm goes away, so might as well do it as +* part of the final flush now. +* +* If lazy shootdown was improved to reduce IPIs (e.g., +* by batching), then it may end up being better to use +* tlbies here instead. +*/ +smp_mb ();/* see radix__flush_tlb_mm */ +exit_flush_lazy_tlbs (mm); +_tlbiel_pid (mm->context.id,RIC_FLUSH_ALL ); + +/* +* It should not be possible to have coprocessors still +* attached here. +*/ +if (WARN_ON_ONCE (atomic_read (&mm->context.copros) >0 )) +__flush_all_mm (mm, true); + }else { +__flush_all_mm (mm, true); + } + }else if ( (psize =radix_get_mmu_psize (page_size)) ==-1 ) {if (!tlb->freed_tables)radix__flush_tlb_mm (mm);
Signature · 1 line
2.37.2