mmu context cleanup, lazy tlb cleanup,
Subsystems (29)
Nicholas Piggin
reviewercover mmu context cleanup, lazy tlb cleanup,
This blew up a bit bigger than I thought, so I'd like to get some comments as to whether people agree with the direction it's going.
The patches aren't cleanly split out by arch, but as it is now it's probably easier to get a quick overview of the changes at a glance anyway.
So there's a few different things here.
1. Clean up and use asm-generic for no-op mmu context functions (so not just for nommu architectures). This should be functionally a no-op for everybody. This allows exit_lazy_tlb to easily be added.
2. Add exit_lazy_tlb and use it for x86, so this is x86 and membarrier specific changes. I _may_ have spotted a small membarrier / core sync bug here when adding exit_lazy_tlb.
3. Tidy up lazy tlb a little bit, have its own refcount function and allow it to be selected out. We can audit the nommu archs and deselect it for those.
4. Add a non-refcounting lazy mmu mode, to help scalability when the same mm is used for a lot of lazy mmu switching.
Comments, questions on anything would be much appreciated.
Thanks,
Nick
Nicholas Piggin (7):
asm-generic: add generic MMU versions of mmu context functions
arch: use asm-generic mmu context for no-op implementations
mm: introduce exit_lazy_tlb
x86: use exit_lazy_tlb rather than
membarrier_mm_sync_core_before_usermode
lazy tlb: introduce lazy mm refcount helper functions lazy tlb: allow lazy tlb mm switching to be configurable lazy tlb: shoot lazies, a non-refcounting lazy tlb option
.../membarrier-sync-core/arch-support.txt | 6 +- arch/Kconfig | 23 +++++ arch/alpha/include/asm/mmu_context.h | 12 +-- arch/arc/include/asm/mmu_context.h | 16 ++-- arch/arm/include/asm/mmu_context.h | 26 +----- arch/arm64/include/asm/mmu_context.h | 7 +- arch/csky/include/asm/mmu_context.h | 8 +- arch/hexagon/include/asm/mmu_context.h | 33 ++------ arch/ia64/include/asm/mmu_context.h | 17 +--- arch/m68k/include/asm/mmu_context.h | 47 ++--------- arch/microblaze/include/asm/mmu_context.h | 2 +- arch/microblaze/include/asm/mmu_context_mm.h | 8 +- arch/microblaze/include/asm/processor.h | 3 - arch/mips/include/asm/mmu_context.h | 11 +-- arch/nds32/include/asm/mmu_context.h | 10 +-- arch/nios2/include/asm/mmu_context.h | 21 +---- arch/nios2/mm/mmu_context.c | 1 + arch/openrisc/include/asm/mmu_context.h | 8 +- arch/openrisc/mm/tlb.c | 2 + arch/parisc/include/asm/mmu_context.h | 12 +-- arch/powerpc/Kconfig | 1 + arch/powerpc/include/asm/mmu_context.h | 22 ++--- arch/powerpc/kernel/smp.c | 2 +- arch/powerpc/mm/book3s64/radix_tlb.c | 4 +- arch/riscv/include/asm/mmu_context.h | 22 +---- arch/s390/include/asm/mmu_context.h | 9 +- arch/sh/include/asm/mmu_context.h | 7 +- arch/sh/include/asm/mmu_context_32.h | 9 -- arch/sparc/include/asm/mmu_context_32.h | 10 +-- arch/sparc/include/asm/mmu_context_64.h | 10 +-- arch/um/include/asm/mmu_context.h | 12 ++- arch/unicore32/include/asm/mmu_context.h | 24 +----- arch/x86/include/asm/mmu_context.h | 41 +++++++++ arch/x86/include/asm/sync_core.h | 28 ------- arch/xtensa/include/asm/mmu_context.h | 11 +-- arch/xtensa/include/asm/nommu_context.h | 26 +----- fs/exec.c | 5 +- include/asm-generic/mmu_context.h | 77 +++++++++++++---- include/asm-generic/nommu_context.h | 19 +++++ include/linux/sched/mm.h | 35 ++++---- include/linux/sync_core.h | 21 ----- kernel/cpu.c | 6 +- kernel/exit.c | 2 +- kernel/fork.c | 39 +++++++++ kernel/kthread.c | 12 ++- kernel/sched/core.c | 84 ++++++++++++------- kernel/sched/sched.h | 4 +- 47 files changed, 388 insertions(+), 427 deletions(-) delete mode 100644 arch/x86/include/asm/sync_core.h create mode 100644 include/asm-generic/nommu_context.h delete mode 100644 include/linux/sync_core.h
Signature · 1 line
2.23.0
Nicholas Piggin
reviewerRFC PATCH 1/7 asm-generic: add generic MMU versions of mmu context functions
Many of these are no-ops on many architectures, so extend mmu_context.h to cover MMU and NOMMU, and split the NOMMU bits out to nommu_context.h
--- arch/microblaze/include/asm/mmu_context.h | 2 +- arch/sh/include/asm/mmu_context.h | 2 +- include/asm-generic/mmu_context.h | 57 +++++++++++++++++------ include/asm-generic/nommu_context.h | 19 ++++++++ 4 files changed, 64 insertions(+), 16 deletions(-) create mode 100644 include/asm-generic/nommu_context.h
arch/microblaze/include/asm/mmu_context.h +1 -1
@@ -2,5 +2,5 @@#ifdef CONFIG_MMU # include <asm/mmu_context_mm.h> #else -# include <asm-generic/mmu_context.h> +# include <asm-generic/nommu_context.h> #endif
arch/sh/include/asm/mmu_context.h +1 -1
@@ -134,7 +134,7 @@ static inline void switch_mm(struct mm_struct *prev,#define set_TTB (pgd) do { } while (0)#define get_TTB () (0) -#include <asm-generic/mmu_context.h> +#include <asm-generic/nommu_context.h> #endif /* CONFIG_MMU */
include/asm-generic/mmu_context.h +43 -14
87 lines not on this page · raw message
include/asm-generic/nommu_context.h new file +19 -0
@@ -0,0 +1,19 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef __ASM_GENERIC_NOMMU_H +#define __ASM_GENERIC_NOMMU_H + +/* +* Generic hooks for NOMMU architectures, which do not need to do +* anything special here. +*/ + +#include <asm-generic/mm_hooks.h> +#include <asm-generic/mmu_context.h> + +static inline void switch_mm (struct mm_struct *prev, +struct mm_struct *next, +struct task_struct *tsk) +{ +} + +#endif /* __ASM_GENERIC_NOMMU_H */
Signature · 1 line
2.23.0
Nicholas Piggin
reviewerRFC PATCH 2/7 arch: use asm-generic mmu context for no-op implementations
In reply to Nicholas PigginThis patch bunches all architectures together. If the general idea is accepted I will split them individually. Some architectures can go further e.g., with consolidating switch_mm and activate_mm but I only did the more obvious ones.
--- arch/alpha/include/asm/mmu_context.h | 12 ++--- arch/arc/include/asm/mmu_context.h | 16 +++---- arch/arm/include/asm/mmu_context.h | 26 ++--------- arch/arm64/include/asm/mmu_context.h | 7 ++- arch/csky/include/asm/mmu_context.h | 8 ++-- arch/hexagon/include/asm/mmu_context.h | 33 +++----------- arch/ia64/include/asm/mmu_context.h | 17 ++----- arch/m68k/include/asm/mmu_context.h | 47 ++++---------------- arch/microblaze/include/asm/mmu_context_mm.h | 8 ++-- arch/microblaze/include/asm/processor.h | 3 -- arch/mips/include/asm/mmu_context.h | 11 ++--- arch/nds32/include/asm/mmu_context.h | 10 +---- arch/nios2/include/asm/mmu_context.h | 21 ++------- arch/nios2/mm/mmu_context.c | 1 + arch/openrisc/include/asm/mmu_context.h | 8 ++-- arch/openrisc/mm/tlb.c | 2 + arch/parisc/include/asm/mmu_context.h | 12 ++--- arch/powerpc/include/asm/mmu_context.h | 22 +++------ arch/riscv/include/asm/mmu_context.h | 22 +-------- arch/s390/include/asm/mmu_context.h | 9 ++-- arch/sh/include/asm/mmu_context.h | 5 +-- arch/sh/include/asm/mmu_context_32.h | 9 ---- arch/sparc/include/asm/mmu_context_32.h | 10 ++--- arch/sparc/include/asm/mmu_context_64.h | 10 ++--- arch/um/include/asm/mmu_context.h | 12 +++-- arch/unicore32/include/asm/mmu_context.h | 24 ++-------- arch/x86/include/asm/mmu_context.h | 6 +++ arch/xtensa/include/asm/mmu_context.h | 11 ++--- arch/xtensa/include/asm/nommu_context.h | 26 +---------- 29 files changed, 106 insertions(+), 302 deletions(-)
arch/alpha/include/asm/mmu_context.h +4 -8
@@ -214,8 +214,6 @@ ev4_activate_mm(struct mm_struct *prev_mm, struct mm_struct *next_mm)tbiap (); } -#define deactivate_mm (tsk,mm) do { } while (0) -#ifdef CONFIG_ALPHA_GENERIC # define switch_mm (a,b,c) alpha_mv.mv_switch_mm((a),(b),(c))# define activate_mm (x,y) alpha_mv.mv_activate_mm((x),(y))
@@ -229,6 +227,7 @@ ev4_activate_mm(struct mm_struct *prev_mm, struct mm_struct *next_mm)# endif #endif +#define init_new_context init_new_contextstatic inline int init_new_context (struct task_struct *tsk,struct mm_struct *mm) {
@@ -242,12 +241,7 @@ init_new_context(struct task_struct *tsk, struct mm_struct *mm)return 0 ; } -extern inline void -destroy_context (struct mm_struct *mm) -{ -/* Nothing to do. */ -} - +#define enter_lazy_tlb enter_lazy_tlbstatic inline void enter_lazy_tlb (struct mm_struct *mm,struct task_struct *tsk) {
@@ -255,6 +249,8 @@ enter_lazy_tlb(struct mm_struct *mm, struct task_struct *tsk) = ((unsigned long)mm->pgd - IDENT_ADDR) >> PAGE_SHIFT; } +#include <asm-generic/mmu_context.h> +#ifdef __MMU_EXTERN_INLINE#undef __EXTERN_INLINE#undef __MMU_EXTERN_INLINE
arch/arc/include/asm/mmu_context.h +8 -8
@@ -102,6 +102,7 @@ static inline void get_new_mmu_context(struct mm_struct *mm)* Initialize the context related info for a new mm_struct * instance. */ +#define init_new_context init_new_contextstatic inline int init_new_context (struct task_struct *tsk,struct mm_struct *mm) {
@@ -113,6 +114,7 @@ init_new_context(struct task_struct *tsk, struct mm_struct *mm)return 0 ; } +#define destroy_context destroy_contextstatic inline void destroy_context (struct mm_struct *mm) {unsigned long flags;
@@ -153,13 +155,12 @@ static inline void switch_mm(struct mm_struct *prev, struct mm_struct *next, }/* -* Called at the time of execve() to get a new ASID -* Note the subtlety here: get_new_mmu_context() behaves differently here -* vs. in switch_mm(). Here it always returns a new ASID, because mm has -* an unallocated "initial" value, while in latter, it moves to a new ASID, -* only if it was unallocated +* activate_mm defaults to switch_mm and is called at the time of execve() to +* get a new ASID Note the subtlety here: get_new_mmu_context() behaves +* differently here vs. in switch_mm(). Here it always returns a new ASID, +* because mm has an unallocated "initial" value, while in latter, it moves to +* a new ASID, only if it was unallocated */ -#define activate_mm (prev, next) switch_mm(prev, next, NULL)/* it seemed that deactivate_mm( ) is a reasonable place to do book-keeping * for retiring-mm. However destroy_context( ) still needs to do that because
@@ -168,8 +169,7 @@ static inline void switch_mm(struct mm_struct *prev, struct mm_struct *next,* there is a good chance that task gets sched-out/in, making it's ASID valid * again (this teased me for a whole day). */ -#define deactivate_mm (tsk, mm) do { } while (0) -#define enter_lazy_tlb (mm, tsk) +#include <asm-generic/mmu_context.h> #endif /* __ASM_ARC_MMU_CONTEXT_H */
arch/arm/include/asm/mmu_context.h +3 -23
@@ -26,6 +26,8 @@ void __check_vmalloc_seq(struct mm_struct *mm);#ifdef CONFIG_CPU_HAS_ASID void check_and_switch_context (struct mm_struct *mm,struct task_struct *tsk); + +#define init_new_context init_new_contextstatic inline int init_new_context (struct task_struct *tsk,struct mm_struct *mm) {
@@ -92,32 +94,10 @@ static inline void finish_arch_post_lock_switch(void)#endif /* CONFIG_MMU */ -static inline int -init_new_context (struct task_struct *tsk,struct mm_struct *mm) -{ -return 0 ; -} - -#endif /* CONFIG_CPU_HAS_ASID */ -#define destroy_context (mm) do { } while(0)#define activate_mm (prev,next) switch_mm(prev, next, NULL) -/* -* This is called when "tsk" is about to enter lazy TLB mode. -* -* mm: describes the currently active mm context -* tsk: task which is entering lazy tlb -* cpu: cpu number which is entering lazy tlb -* -* tsk->mm will be NULL -*/ -static inline void -enter_lazy_tlb (struct mm_struct *mm,struct task_struct *tsk) -{ -} -/* * This is the actual mm switch as far as the scheduler * is concerned. No registers are touched. We avoid
@@ -149,6 +129,6 @@ switch_mm(struct mm_struct *prev, struct mm_struct *next, #endif } -#define deactivate_mm (tsk,mm) do { } while (0) +#include <asm-generic/mmu_context.h> #endif
arch/arm64/include/asm/mmu_context.h +3 -4
@@ -174,7 +174,6 @@ static inline void cpu_replace_ttbr1(pgd_t *pgdp)* Setting a reserved TTBR0 or EPD0 would work, but it all gets ugly when you * take CPU migration into account. */ -#define destroy_context (mm) do { } while(0)void check_and_switch_context (struct mm_struct *mm,unsigned int cpu);#define init_new_context (tsk,mm) ({ atomic64_set(&(mm)->context.id, 0); 0; })
@@ -202,6 +201,7 @@ static inline void update_saved_ttbr0(struct task_struct *tsk, }#endif +#define enter_lazy_tlb enter_lazy_tlbstatic inline void enter_lazy_tlb (struct mm_struct *mm,struct task_struct *tsk) {
@@ -244,12 +244,11 @@ switch_mm(struct mm_struct *prev, struct mm_struct *next,update_saved_ttbr0 (tsk, next); } -#define deactivate_mm (tsk,mm) do { } while (0) -#define activate_mm (prev,next) switch_mm(prev, next, current) -void verify_cpu_asid_bits (void );void post_ttbr_update_workaround (void ); +#include <asm-generic/mmu_context.h> +#endif /* !__ASSEMBLY__ */ #endif /* !__ASM_MMU_CONTEXT_H */
arch/csky/include/asm/mmu_context.h +3 -5
@@ -24,11 +24,6 @@#define cpu_asid (mm) (atomic64_read(&mm->context.asid) & ASID_MASK)#define init_new_context (tsk,mm) ({ atomic64_set(&(mm)->context.asid, 0); 0; }) -#define activate_mm (prev,next) switch_mm(prev, next, current) - -#define destroy_context (mm) do {} while (0) -#define enter_lazy_tlb (mm, tsk) do {} while (0) -#define deactivate_mm (tsk, mm) do {} while (0)void check_and_switch_context (struct mm_struct *mm,unsigned int cpu);
@@ -46,4 +41,7 @@ switch_mm(struct mm_struct *prev, struct mm_struct *next, flush_icache_deferred(next); } + +#include <asm-generic/mmu_context.h> +#endif /* __ASM_CSKY_MMU_CONTEXT_H */
arch/hexagon/include/asm/mmu_context.h +5 -28
@@ -15,39 +15,13 @@#include <asm/pgalloc.h> #include <asm/mem-layout.h> -static inline void destroy_context (struct mm_struct *mm) -{ -} -/* * VM port hides all TLB management, so "lazy TLB" isn't very * meaningful. Even for ports to architectures with visble TLBs, * this is almost invariably a null function. +* +* mm->context is set up by pgd_alloc, so no init_new_context required. */ -static inline void enter_lazy_tlb (struct mm_struct *mm, -struct task_struct *tsk) -{ -} - -/* -* Architecture-specific actions, if any, for memory map deactivation. -*/ -static inline void deactivate_mm (struct task_struct *tsk, -struct mm_struct *mm) -{ -} - -/** -* init_new_context - initialize context related info for new mm_struct instance -* @tsk: pointer to a task struct -* @mm: pointer to a new mm struct -*/ -static inline int init_new_context (struct task_struct *tsk, -struct mm_struct *mm) -{ -/* mm->context is set up by pgd_alloc */ -return 0 ; -}/* * Switch active mm context
@@ -74,6 +48,7 @@ static inline void switch_mm(struct mm_struct *prev, struct mm_struct *next,/* * Activate new memory map for task */ +#define activate_mm activate_mmstatic inline void activate_mm (struct mm_struct *prev,struct mm_struct *next) {unsigned long flags;
@@ -86,4 +61,6 @@ static inline void activate_mm(struct mm_struct *prev, struct mm_struct *next)/* Generic hooks for arch_dup_mmap and arch_exit_mmap */ #include <asm-generic/mm_hooks.h> +#include <asm-generic/mmu_context.h> +#endif
arch/ia64/include/asm/mmu_context.h +4 -13
@@ -49,11 +49,6 @@ DECLARE_PER_CPU(u8, ia64_need_tlb_flush);extern void mmu_context_init (void );extern void wrap_mmu_context (struct mm_struct *mm); -static inline void -enter_lazy_tlb (struct mm_struct *mm,struct task_struct *tsk) -{ -} -/* * When the context counter wraps around all TLBs need to be flushed because * an old context number might have been reused. This is signalled by the
@@ -116,6 +111,7 @@ get_mmu_context (struct mm_struct *mm)* Initialize context number to some sane value. MM is guaranteed to be a * brand-new address-space, so no TLB flushing is needed, ever. */ +#define init_new_context init_new_contextstatic inline int init_new_context (struct task_struct *p,struct mm_struct *mm) {
@@ -123,12 +119,6 @@ init_new_context (struct task_struct *p, struct mm_struct *mm)return 0 ; } -static inline void -destroy_context (struct mm_struct *mm) -{ -/* Nothing to do. */ -} -static inline void reload_context (nv_mm_context_t context) {
@@ -178,11 +168,10 @@ activate_context (struct mm_struct *mm) }while (unlikely (context != mm->context)); } -#define deactivate_mm (tsk,mm) do { } while (0) -/* * Switch from address space PREV to address space NEXT. */ +#define activate_mm activate_mmstatic inline void activate_mm (struct mm_struct *prev,struct mm_struct *next) {
@@ -196,5 +185,7 @@ activate_mm (struct mm_struct *prev, struct mm_struct *next)#define switch_mm (prev_mm,next_mm,next_task) activate_mm(prev_mm, next_mm) +#include <asm-generic/mmu_context.h> +# endif /* ! __ASSEMBLY__ */ #endif /* _ASM_IA64_MMU_CONTEXT_H */
arch/m68k/include/asm/mmu_context.h +8 -39
103 lines not on this page · raw message
arch/microblaze/include/asm/mmu_context_mm.h +4 -4
@@ -33,10 +33,6 @@to represent all kernel pages as shared among all contexts. */ -static inline void enter_lazy_tlb (struct mm_struct *mm,struct task_struct *tsk) -{ -} -# define NO_CONTEXT 256# define LAST_CONTEXT 255# define FIRST_CONTEXT 1
@@ -105,6 +101,7 @@ static inline void get_mmu_context(struct mm_struct *mm)/* * We're finished using the context for an address space. */ +#define destroy_context destroy_contextstatic inline void destroy_context (struct mm_struct *mm) {if (mm->context !=NO_CONTEXT ) {
@@ -126,6 +123,7 @@ static inline void switch_mm(struct mm_struct *prev, struct mm_struct *next,* After we have set current->mm to a new value, this activates * the context for the new mm so we see the new mappings. */ +#define activate_mm activate_mmstatic inline void activate_mm (struct mm_struct *active_mm,struct mm_struct *mm) {
@@ -136,5 +134,7 @@ static inline void activate_mm(struct mm_struct *active_mm,extern void mmu_context_init (void ); +#include <asm-generic/mmu_context.h> +# endif /* __KERNEL__ */ #endif /* _ASM_MICROBLAZE_MMU_CONTEXT_H */
arch/microblaze/include/asm/processor.h +0 -3
@@ -122,9 +122,6 @@ unsigned long get_wchan(struct task_struct *p);# define KSTK_EIP (task) (task_pc(task))# define KSTK_ESP (task) (task_sp(task)) -/* FIXME */ -# define deactivate_mm (tsk, mm) do { } while (0) -# define STACK_TOP TASK_SIZE# define STACK_TOP_MAX STACK_TOP
arch/mips/include/asm/mmu_context.h +4 -7
@@ -124,10 +124,6 @@ static inline void set_cpu_context(unsigned int cpu,#define cpu_asid (cpu, mm) \ (cpu_context((cpu), (mm)) & cpu_asid_mask(&cpu_data[cpu])) -static inline void enter_lazy_tlb (struct mm_struct *mm,struct task_struct *tsk) -{ -} -extern void get_new_mmu_context (struct mm_struct *mm);extern void check_mmu_context (struct mm_struct *mm);extern void check_switch_mmu_context (struct mm_struct *mm);
@@ -136,6 +132,7 @@ extern void check_switch_mmu_context(struct mm_struct *mm);* Initialize the context related info for a new mm_struct * instance. */ +#define init_new_context init_new_contextstatic inline int init_new_context (struct task_struct *tsk,struct mm_struct *mm) {
@@ -180,14 +177,12 @@ static inline void switch_mm(struct mm_struct *prev, struct mm_struct *next,* Destroy context related info for an mm_struct that is about * to be put to rest. */ +#define destroy_context destroy_contextstatic inline void destroy_context (struct mm_struct *mm) {dsemul_mm_cleanup (mm); } -#define activate_mm (prev, next) switch_mm(prev, next, current) -#define deactivate_mm (tsk, mm) do { } while (0) -static inline void drop_mmu_context (struct mm_struct *mm) {
@@ -237,4 +232,6 @@ drop_mmu_context(struct mm_struct *mm) local_irq_restore(flags); } +#include <asm-generic/mmu_context.h> +#endif /* _ASM_MMU_CONTEXT_H */
arch/nds32/include/asm/mmu_context.h +2 -8
@@ -9,6 +9,7 @@#include <asm/proc-fns.h> #include <asm-generic/mm_hooks.h> +#define init_new_context init_new_contextstatic inline int init_new_context (struct task_struct *tsk,struct mm_struct *mm) {
@@ -16,8 +17,6 @@ init_new_context(struct task_struct *tsk, struct mm_struct *mm)return 0 ; } -#define destroy_context (mm) do { } while(0) -#define CID_BITS 9extern spinlock_t cid_lock;extern unsigned int cpu_last_cid;
@@ -47,10 +46,6 @@ static inline void check_context(struct mm_struct *mm)__new_context (mm); } -static inline void enter_lazy_tlb (struct mm_struct *mm,struct task_struct *tsk) -{ -} -static inline void switch_mm (struct mm_struct *prev,struct mm_struct *next,struct task_struct *tsk) {
@@ -62,7 +57,6 @@ static inline void switch_mm(struct mm_struct *prev, struct mm_struct *next, } } -#define deactivate_mm (tsk,mm) do { } while (0) -#define activate_mm (prev,next) switch_mm(prev, next, NULL) +#include <asm-generic/mmu_context.h> #endif
arch/nios2/include/asm/mmu_context.h +4 -17
@@ -26,16 +26,13 @@ extern unsigned long get_pid_from_context(mm_context_t *ctx);*/ extern pgd_t *pgd_current; -static inline void enter_lazy_tlb (struct mm_struct *mm,struct task_struct *tsk) -{ -} -/* * Initialize the context related info for a new mm_struct instance. * * Set all new contexts to 0, that way the generation will never match * the currently running generation when this context is switched in. */ +#define init_new_context init_new_contextstatic inline int init_new_context (struct task_struct *tsk,struct mm_struct *mm) {
@@ -43,26 +40,16 @@ static inline int init_new_context(struct task_struct *tsk, return 0; } -/* -* Destroy context related info for an mm_struct that is about -* to be put to rest. -*/ -static inline void destroy_context (struct mm_struct *mm) -{ -} -void switch_mm (struct mm_struct *prev,struct mm_struct *next,struct task_struct *tsk); -static inline void deactivate_mm (struct task_struct *tsk, -struct mm_struct *mm) -{ -} -/* * After we have set current->mm to a new value, this activates * the context for the new mm so we see the new mappings. */ +#define activate_mm activate_mmvoid activate_mm (struct mm_struct *prev,struct mm_struct *next); +#include <asm-generic/mmu_context.h> +#endif /* _ASM_NIOS2_MMU_CONTEXT_H */
arch/nios2/mm/mmu_context.c +1 -0
@@ -103,6 +103,7 @@ void switch_mm(struct mm_struct *prev, struct mm_struct *next,* After we have set current->mm to a new value, this activates * the context for the new mm so we see the new mappings. */ +#define activate_mm activate_mmvoid activate_mm (struct mm_struct *prev,struct mm_struct *next) { next->context =get_new_context ();
arch/openrisc/include/asm/mmu_context.h +3 -5
@@ -17,13 +17,13 @@#include <asm-generic/mm_hooks.h> +#define init_new_context init_new_contextextern int init_new_context (struct task_struct *tsk,struct mm_struct *mm); +#define destroy_context destroy_contextextern void destroy_context (struct mm_struct *mm);extern void switch_mm (struct mm_struct *prev,struct mm_struct *next,struct task_struct *tsk); -#define deactivate_mm (tsk, mm) do { } while (0) -#define activate_mm (prev, next) switch_mm((prev), (next), NULL)/* current active pgd - this is similar to other processors pgd
@@ -32,8 +32,6 @@ extern void switch_mm(struct mm_struct *prev, struct mm_struct *next,extern volatile pgd_t *current_pgd[];/* defined in arch/openrisc/mm/fault.c */ -static inline void enter_lazy_tlb (struct mm_struct *mm,struct task_struct *tsk) -{ -} +#include <asm-generic/mmu_context.h> #endif
arch/openrisc/mm/tlb.c +2 -0
@@ -159,6 +159,7 @@ void switch_mm(struct mm_struct *prev, struct mm_struct *next,* instance. */ +#define init_new_context init_new_contextint init_new_context (struct task_struct *tsk,struct mm_struct *mm) { mm->context =NO_CONTEXT ;
@@ -170,6 +171,7 @@ int init_new_context(struct task_struct *tsk, struct mm_struct *mm)* drops it. */ +#define destroy_context destroy_contextvoid destroy_context (struct mm_struct *mm) {flush_tlb_mm (mm);
arch/parisc/include/asm/mmu_context.h +6 -6
@@ -8,16 +8,13 @@#include <asm/pgalloc.h> #include <asm-generic/mm_hooks.h> -static inline void enter_lazy_tlb (struct mm_struct *mm,struct task_struct *tsk) -{ -} -/* on PA-RISC, we actually have enough contexts to justify an allocator * for them. prumpf */ extern unsigned long alloc_sid (void );extern void free_sid (unsigned long ); +#define init_new_context init_new_contextstatic inline int init_new_context (struct task_struct *tsk,struct mm_struct *mm) {
@@ -27,6 +24,7 @@ init_new_context(struct task_struct *tsk, struct mm_struct *mm)return 0 ; } +#define destroy_context destroy_contextstatic inline void destroy_context (struct mm_struct *mm) {
@@ -72,8 +70,7 @@ static inline void switch_mm(struct mm_struct *prev, }#define switch_mm_irqs_off switch_mm_irqs_off -#define deactivate_mm (tsk,mm) do { } while (0) - +#define activate_mm activate_mmstatic inline void activate_mm (struct mm_struct *prev,struct mm_struct *next) {/*
@@ -91,4 +88,7 @@ static inline void activate_mm(struct mm_struct *prev, struct mm_struct *next) switch_mm(prev,next,current); } + +#include <asm-generic/mmu_context.h> +#endif
arch/powerpc/include/asm/mmu_context.h +7 -15
@@ -14,7 +14,9 @@/* * Most if the context management is out of line */ +#define init_new_context init_new_contextextern int init_new_context (struct task_struct *tsk,struct mm_struct *mm); +#define destroy_context destroy_contextextern void destroy_context (struct mm_struct *mm);#ifdef CONFIG_SPAPR_TCE_IOMMU struct mm_iommu_table_group_mem_t ;
@@ -237,27 +239,15 @@ static inline void switch_mm(struct mm_struct *prev, struct mm_struct *next, }#define switch_mm_irqs_off switch_mm_irqs_off - -#define deactivate_mm (tsk,mm) do { } while (0) - -/* -* After we have set current->mm to a new value, this activates -* the context for the new mm so we see the new mappings. -*/ -static inline void activate_mm (struct mm_struct *prev,struct mm_struct *next) -{ -switch_mm (prev, next, current); -} - -/* We don't currently use enter_lazy_tlb() for anything */ +#ifdef CONFIG_PPC_BOOK3E_64 +#define enter_lazy_tlb enter_lazy_tlbstatic inline void enter_lazy_tlb (struct mm_struct *mm,struct task_struct *tsk) {/* 64-bit Book3E keeps track of current PGD in the PACA */ -#ifdef CONFIG_PPC_BOOK3E_64 get_paca ()->pgd =NULL ; -#endif } +#endif extern void arch_exit_mmap (struct mm_struct *mm);
@@ -300,5 +290,7 @@ static inline int arch_dup_mmap(struct mm_struct *oldmm,return 0 ; } +#include <asm-generic/mmu_context.h> +#endif /* __KERNEL__ */ #endif /* __ASM_POWERPC_MMU_CONTEXT_H */
arch/riscv/include/asm/mmu_context.h +2 -20
@@ -13,34 +13,16 @@#include <linux/mm.h> #include <linux/sched.h> -static inline void enter_lazy_tlb (struct mm_struct *mm, -struct task_struct *task) -{ -} - -/* Initialize context-related info for a new mm_struct */ -static inline int init_new_context (struct task_struct *task, -struct mm_struct *mm) -{ -return 0 ; -} - -static inline void destroy_context (struct mm_struct *mm) -{ -} -void switch_mm (struct mm_struct *prev,struct mm_struct *next,struct task_struct *task); +#define activate_mm activate_mmstatic inline void activate_mm (struct mm_struct *prev,struct mm_struct *next) {switch_mm (prev, next,NULL ); } -static inline void deactivate_mm (struct task_struct *task, -struct mm_struct *mm) -{ -} +#include <asm-generic/mmu_context.h> #endif /* _ASM_RISCV_MMU_CONTEXT_H */
arch/s390/include/asm/mmu_context.h +4 -5
@@ -15,6 +15,7 @@#include <asm/ctl_reg.h> #include <asm-generic/mm_hooks.h> +#define init_new_context init_new_contextstatic inline int init_new_context (struct task_struct *tsk,struct mm_struct *mm) {
@@ -69,8 +70,6 @@ static inline int init_new_context(struct task_struct *tsk,return 0 ; } -#define destroy_context (mm) do { } while (0) -static inline void set_user_asce (struct mm_struct *mm) { S390_lowcore.user_asce = mm->context.asce;
@@ -125,9 +124,7 @@ static inline void finish_arch_post_lock_switch(void)set_fs (current->thread.mm_segment); } -#define enter_lazy_tlb (mm,tsk) do { } while (0) -#define deactivate_mm (tsk,mm) do { } while (0) - +#define activate_mm activate_mmstatic inline void activate_mm (struct mm_struct *prev,struct mm_struct *next) {
@@ -136,4 +133,6 @@ static inline void activate_mm(struct mm_struct *prev, set_user_asce(next); } +#include <asm-generic/mmu_context.h> +#endif /* __S390_MMU_CONTEXT_H */
arch/sh/include/asm/mmu_context.h +2 -3
@@ -85,6 +85,7 @@ static inline void get_mmu_context(struct mm_struct *mm, unsigned int cpu)* Initialize the context related info for a new mm_struct * instance. */ +#define init_new_context init_new_contextstatic inline int init_new_context (struct task_struct *tsk,struct mm_struct *mm) {
@@ -121,9 +122,7 @@ static inline void switch_mm(struct mm_struct *prev, activate_context(next, cpu); } -#define activate_mm (prev, next) switch_mm((prev),(next),NULL) -#define deactivate_mm (tsk,mm) do { } while (0) -#define enter_lazy_tlb (mm,tsk) do { } while (0) +#include <asm-generic/mmu_context.h> #else
arch/sh/include/asm/mmu_context_32.h +0 -9
@@ -2,15 +2,6 @@ #ifndef __ASM_SH_MMU_CONTEXT_32_H #define __ASM_SH_MMU_CONTEXT_32_H -/* - * Destroy context related info for an mm_struct that is about - * to be put to rest. - */ -static inline void destroy_context(struct mm_struct *mm) -{ - /* Do nothing */ -} - #ifdef CONFIG_CPU_HAS_PTEAEX static inline void set_asid(unsigned long asid) {
arch/sparc/include/asm/mmu_context_32.h +4 -6
@@ -6,13 +6,10 @@#include <asm-generic/mm_hooks.h> -static inline void enter_lazy_tlb (struct mm_struct *mm,struct task_struct *tsk) -{ -} -/* Initialize a new mmu context. This is invoked when a new * address space instance (unique or shared) is instantiated. */ +#define init_new_context init_new_contextint init_new_context (struct task_struct *tsk,struct mm_struct *mm);/* Destroy a dead context. This occurs when mmput drops the
@@ -20,17 +17,18 @@ int init_new_context(struct task_struct *tsk, struct mm_struct *mm);* all the page tables have been flushed. Our job is to destroy * any remaining processor-specific state. */ +#define destroy_context destroy_contextvoid destroy_context (struct mm_struct *mm);/* Switch the current MM context. */ void switch_mm (struct mm_struct *old_mm,struct mm_struct *mm,struct task_struct *tsk); -#define deactivate_mm (tsk,mm) do { } while (0) -/* Activate a new MM instance for the current task. */ #define activate_mm (active_mm, mm) switch_mm((active_mm), (mm), NULL) +#include <asm-generic/mmu_context.h> +#endif /* !(__ASSEMBLY__) */ #endif /* !(__SPARC_MMU_CONTEXT_H) */
arch/sparc/include/asm/mmu_context_64.h +5 -5
@@ -16,17 +16,16 @@#include <asm-generic/mm_hooks.h> #include <asm/percpu.h> -static inline void enter_lazy_tlb (struct mm_struct *mm,struct task_struct *tsk) -{ -} -extern spinlock_t ctx_alloc_lock;extern unsigned long tlb_context_cache;extern unsigned long mmu_context_bmap[]; DECLARE_PER_CPU(struct mm_struct *, per_cpu_secondary_mm);void get_new_mmu_context (struct mm_struct *mm); + +#define init_new_context init_new_contextint init_new_context (struct task_struct *tsk,struct mm_struct *mm); +#define destroy_context destroy_contextvoid destroy_context (struct mm_struct *mm); void __tsb_context_switch(unsigned long pgd_pa,
@@ -136,7 +135,6 @@ static inline void switch_mm(struct mm_struct *old_mm, struct mm_struct *mm, strspin_unlock_irqrestore (&mm->context.lock, flags); } -#define deactivate_mm (tsk,mm) do { } while (0)#define activate_mm (active_mm, mm) switch_mm(active_mm, mm, NULL)#define __HAVE_ARCH_START_CONTEXT_SWITCH
@@ -187,6 +185,8 @@ static inline void finish_arch_post_lock_switch(void) } } +#include <asm-generic/mmu_context.h> +#endif /* !(__ASSEMBLY__) */ #endif /* !(__SPARC64_MMU_CONTEXT_H) */
arch/um/include/asm/mmu_context.h +5 -7
@@ -37,10 +37,9 @@ static inline bool arch_vma_access_permitted(struct vm_area_struct *vma,* end asm-generic/mm_hooks.h functions */ -#define deactivate_mm (tsk,mm) do { } while (0) -extern void force_flush_all (void ); +#define activate_mm activate_mmstatic inline void activate_mm (struct mm_struct *old,struct mm_struct *new) {/*
@@ -66,13 +65,12 @@ static inline void switch_mm(struct mm_struct *prev, struct mm_struct *next, } } -static inline void enter_lazy_tlb (struct mm_struct *mm, -struct task_struct *tsk) -{ -} - +#define init_new_context init_new_contextextern int init_new_context (struct task_struct *task,struct mm_struct *mm); +#define destroy_context destroy_contextextern void destroy_context (struct mm_struct *mm); +#include <asm-generic/mmu_context.h> +#endif
arch/unicore32/include/asm/mmu_context.h +3 -21
@@ -18,24 +18,6 @@#include <asm/cacheflush.h> #include <asm/cpu-single.h> -#define init_new_context (tsk, mm) 0 - -#define destroy_context (mm) do { } while (0) - -/* -* This is called when "tsk" is about to enter lazy TLB mode. -* -* mm: describes the currently active mm context -* tsk: task which is entering lazy tlb -* cpu: cpu number which is entering lazy tlb -* -* tsk->mm will be NULL -*/ -static inline void -enter_lazy_tlb (struct mm_struct *mm,struct task_struct *tsk) -{ -} -/* * This is the actual mm switch as far as the scheduler * is concerned. No registers are touched. We avoid
@@ -52,9 +34,6 @@ switch_mm(struct mm_struct *prev, struct mm_struct *next,cpu_switch_mm (next->pgd, next); } -#define deactivate_mm (tsk, mm) do { } while (0) -#define activate_mm (prev, next) switch_mm(prev, next, NULL) -/* * We are inserting a "fake" vma for the user-accessible vector page so * gdb and friends can get to it through ptrace and /proc/<pid>/mem.
@@ -95,4 +74,7 @@ static inline bool arch_vma_access_permitted(struct vm_area_struct *vma,/* by default, allow everything */ return true; } + +#include <asm-generic/mmu_context.h> +#endif
arch/x86/include/asm/mmu_context.h +6 -0
@@ -92,12 +92,14 @@ static inline void switch_ldt(struct mm_struct *prev, struct mm_struct *next) }#endif +#define enter_lazy_tlb enter_lazy_tlbextern void enter_lazy_tlb (struct mm_struct *mm,struct task_struct *tsk);/* * Init a new mm. Used on mm copies, like at fork() * and on mm's that are brand-new, like at execve(). */ +#define init_new_context init_new_contextstatic inline int init_new_context (struct task_struct *tsk,struct mm_struct *mm) {
@@ -117,6 +119,8 @@ static inline int init_new_context(struct task_struct *tsk,init_new_context_ldt (mm);return 0 ; } + +#define destroy_context destroy_contextstatic inline void destroy_context (struct mm_struct *mm) {destroy_context_ldt (mm);
@@ -215,4 +219,6 @@ static inline bool arch_vma_access_permitted(struct vm_area_struct *vma,unsigned long __get_current_cr3_fast (void ); +#include <asm-generic/mmu_context.h> +#endif /* _ASM_X86_MMU_CONTEXT_H */
arch/xtensa/include/asm/mmu_context.h +3 -8
@@ -111,6 +111,7 @@ static inline void activate_context(struct mm_struct *mm, unsigned int cpu)* to -1 says the process has never run on any core. */ +#define init_new_context init_new_contextstatic inline int init_new_context (struct task_struct *tsk,struct mm_struct *mm) {
@@ -136,24 +137,18 @@ static inline void switch_mm(struct mm_struct *prev, struct mm_struct *next,activate_context (next, cpu); } -#define activate_mm (prev, next) switch_mm((prev), (next), NULL) -#define deactivate_mm (tsk, mm) do { } while (0) -/* * Destroy context related info for an mm_struct that is about * to be put to rest. */ +#define destroy_context destroy_contextstatic inline void destroy_context (struct mm_struct *mm) {invalidate_page_directory (); } -static inline void enter_lazy_tlb (struct mm_struct *mm,struct task_struct *tsk) -{ -/* Nothing to do. */ - -} +#include <asm-generic/mmu_context.h> #endif /* CONFIG_MMU */ #endif /* _XTENSA_MMU_CONTEXT_H */
arch/xtensa/include/asm/nommu_context.h +1 -25
@@ -7,28 +7,4 @@ static inline void init_kio(void) { } -static inline void enter_lazy_tlb (struct mm_struct *mm,struct task_struct *tsk) -{ -} - -static inline int init_new_context (struct task_struct *tsk,struct mm_struct *mm) -{ -return 0 ; -} - -static inline void destroy_context (struct mm_struct *mm) -{ -} - -static inline void activate_mm (struct mm_struct *prev,struct mm_struct *next) -{ -} - -static inline void switch_mm (struct mm_struct *prev,struct mm_struct *next, -struct task_struct *tsk) -{ -} - -static inline void deactivate_mm (struct task_struct *tsk,struct mm_struct *mm) -{ -} +#include <asm-generic/nommu_context.h>
Signature · 1 line
2.23.0
Nicholas Piggin
reviewerRFC PATCH 3/7 mm: introduce exit_lazy_tlb
In reply to Nicholas Piggin--- fs/exec.c | 5 +++-- include/asm-generic/mmu_context.h | 20 ++++++++++++++++++++ kernel/kthread.c | 1 + kernel/sched/core.c | 2 ++ 4 files changed, 26 insertions(+), 2 deletions(-)
fs/exec.c +3 -2
@@ -1117,9 +1117,10 @@ static int exec_mmap(struct mm_struct *mm)setmax_mm_hiwater_rss (&tsk->signal->maxrss, old_mm);mm_update_next_owner (old_mm);mmput (old_mm); -return 0 ; + }else { +exit_lazy_tlb (active_mm, tsk); +mmdrop (active_mm); } -mmdrop (active_mm);return 0 ; }
include/asm-generic/mmu_context.h +20 -0
@@ -24,6 +24,26 @@ static inline void enter_lazy_tlb(struct mm_struct *mm, }#endif +/* +* exit_lazy_tlb - Called after switching away from a lazy TLB mode mm. +* +* mm: the lazy mm context that was switched away from +* tsk: the task that was switched to non-lazy mm +* +* tsk->mm will not be NULL. +* +* Note this is not symmetrical to enter_lazy_tlb, this is not +* called when tasks switch into the lazy mm, it's called after the +* lazy mm becomes non-lazy (either switched to a different mm or the +* owner of the mm returns). +*/ +#ifndef exit_lazy_tlb +static inline void exit_lazy_tlb (struct mm_struct *mm, +struct task_struct *tsk) +{ +} +#endif +/** * init_new_context - Initialize context of a new mm_struct. * @tsk: task struct for the mm
kernel/kthread.c +1 -0
@@ -1253,6 +1253,7 @@ void kthread_use_mm(struct mm_struct *mm)if (active_mm != mm)mmdrop (active_mm); +exit_lazy_tlb (active_mm, tsk);to_kthread (tsk)->oldfs =get_fs ();set_fs (USER_DS );
kernel/sched/core.c +2 -0
@@ -3439,6 +3439,8 @@ context_switch(struct rq *rq, struct task_struct *prev,switch_mm_irqs_off (prev->active_mm, next->mm, next);if (!prev->mm) {// from kernel +exit_lazy_tlb (prev->active_mm, next); +/* will mmdrop() in finish_task_switch(). */ rq->prev_mm = prev->active_mm; prev->active_mm =NULL ;
Signature · 1 line
2.23.0
Nicholas Piggin
reviewerRFC PATCH 4/7 x86: use exit_lazy_tlb rather than membarrier_mm_sync_core_before_usermode
In reply to Nicholas PigginAnd get rid of the generic sync_core_before_usermode facility.
This helper is the wrong way around I think. The idea that membarrier state requires a core sync before returning to user is the easy one that does not need hiding behind membarrier calls. The gap in core synchronization due to x86's sysret/sysexit and lazy tlb mode, is the tricky detail that is better put in x86 lazy tlb code.
Consider if an arch did not synchronize core in switch_mm either, then membarrier_mm_sync_core_before_usermode would be in the wrong place but arch specific mmu context functions would still be the right place. There is also a exit_lazy_tlb case that is not covered by this call, which could be a bugs (kthread use mm the membarrier process's mm then context switch back to the process without switching mm or lazy mm switch).
This makes lazy tlb code a bit more modular.
--- .../membarrier-sync-core/arch-support.txt | 6 +++- arch/x86/include/asm/mmu_context.h | 35 +++++++++++++++++++ arch/x86/include/asm/sync_core.h | 28 --------------- include/linux/sched/mm.h | 14 -------- include/linux/sync_core.h | 21 ----------- kernel/cpu.c | 4 ++- kernel/kthread.c | 2 +- kernel/sched/core.c | 16 ++++----- 8 files changed, 51 insertions(+), 75 deletions(-) delete mode 100644 arch/x86/include/asm/sync_core.h delete mode 100644 include/linux/sync_core.h
Documentation/features/sched/membarrier-sync-core/arch-support.txt +5 -1
@@ -5,6 +5,10 @@ # # Architecture requirements # +# If your architecture returns to user-space through non-core-serializing +# instructions, you need to ensure these are done in switch_mm and exit_lazy_tlb +# (if lazy tlb switching is implemented). +# # * arm/arm64/powerpc # # Rely on implicit context synchronization as a result of exception return
@@ -24,7 +28,7 @@ # instead on write_cr3() performed by switch_mm() to provide core serialization # after changing the current mm, and deal with the special case of kthread -> # uthread (temporarily keeping current mm into active_mm) by issuing a -# sync_core_before_usermode() in that specific case. +# serializing instruction in exit_lazy_mm() in that specific case. # ----------------------- | arch |status|
arch/x86/include/asm/mmu_context.h +35 -0
@@ -6,6 +6,7 @@#include <linux/atomic.h> #include <linux/mm_types.h> #include <linux/pkeys.h> +#include <linux/sched/mm.h> #include <trace/events/tlb.h>
@@ -95,6 +96,40 @@ static inline void switch_ldt(struct mm_struct *prev, struct mm_struct *next)#define enter_lazy_tlb enter_lazy_tlbextern void enter_lazy_tlb (struct mm_struct *mm,struct task_struct *tsk); +#ifdef CONFIG_MEMBARRIER +/* +* Ensure that a core serializing instruction is issued before returning +* to user-mode, if a SYNC_CORE was requested. x86 implements return to +* user-space through sysexit, sysrel, and sysretq, which are not core +* serializing. +* +* See the membarrier comment in finish_task_switch as to why this is done +* in exit_lazy_tlb. +*/ +#define exit_lazy_tlb exit_lazy_tlb +static inline void exit_lazy_tlb (struct mm_struct *mm,struct task_struct *tsk) +{ +/* Switching mm is serializing with write_cr3 */ +if (tsk->mm != mm) +return ; + +if (likely (!(atomic_read (&mm->membarrier_state) & +MEMBARRIER_STATE_PRIVATE_EXPEDITED_SYNC_CORE ))) +return ; + +/* With PTI, we unconditionally serialize before running user code. */ +if (static_cpu_has (X86_FEATURE_PTI )) +return ; +/* +* Return from interrupt and NMI is done through iret, which is core +* serializing. +*/ +if (in_irq () ||in_nmi ()) +return ; +sync_core (); +} +#endif +/* * Init a new mm. Used on mm copies, like at fork() * and on mm's that are brand-new, like at execve().
arch/x86/include/asm/sync_core.h deleted +0 -28
@@ -1,28 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0 */ -#ifndef _ASM_X86_SYNC_CORE_H -#define _ASM_X86_SYNC_CORE_H - -#include <linux/preempt.h> -#include <asm/processor.h> -#include <asm/cpufeature.h> - -/* -* Ensure that a core serializing instruction is issued before returning -* to user-mode. x86 implements return to user-space through sysexit, -* sysrel, and sysretq, which are not core serializing. -*/ -static inline void sync_core_before_usermode (void ) -{ -/* With PTI, we unconditionally serialize before running user code. */ -if (static_cpu_has (X86_FEATURE_PTI )) -return ; -/* -* Return from interrupt and NMI is done through iret, which is core -* serializing. -*/ -if (in_irq () ||in_nmi ()) -return ; -sync_core (); -} - -#endif /* _ASM_X86_SYNC_CORE_H */
include/linux/sched/mm.h +0 -14
@@ -7,7 +7,6 @@#include <linux/sched.h> #include <linux/mm_types.h> #include <linux/gfp.h> -#include <linux/sync_core.h> /* * Routines for handling mm_structs
@@ -364,16 +363,6 @@ enum {#include <asm/membarrier.h> #endif -static inline void membarrier_mm_sync_core_before_usermode (struct mm_struct *mm) -{ -if (current->mm != mm) -return ; -if (likely (!(atomic_read (&mm->membarrier_state) & -MEMBARRIER_STATE_PRIVATE_EXPEDITED_SYNC_CORE ))) -return ; -sync_core_before_usermode (); -} -extern void membarrier_exec_mmap (struct mm_struct *mm);#else
@@ -387,9 +376,6 @@ static inline void membarrier_arch_switch_mm(struct mm_struct *prev,static inline void membarrier_exec_mmap (struct mm_struct *mm) { } -static inline void membarrier_mm_sync_core_before_usermode (struct mm_struct *mm) -{ -}#endif #endif /* _LINUX_SCHED_MM_H */
include/linux/sync_core.h deleted +0 -21
@@ -1,21 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0 */ -#ifndef _LINUX_SYNC_CORE_H -#define _LINUX_SYNC_CORE_H - -#ifdef CONFIG_ARCH_HAS_SYNC_CORE_BEFORE_USERMODE -#include <asm/sync_core.h> -#else -/* -* This is a dummy sync_core_before_usermode() implementation that can be used -* on all architectures which return to user-space through core serializing -* instructions. -* If your architecture returns to user-space through non-core-serializing -* instructions, you need to write your own functions. -*/ -static inline void sync_core_before_usermode (void ) -{ -} -#endif - -#endif /* _LINUX_SYNC_CORE_H */ -
kernel/cpu.c +3 -1
@@ -572,7 +572,9 @@ static int finish_cpu(unsigned int cpu)/* * idle_task_exit() will have switched to &init_mm, now -* clean up any remaining active_mm state. +* clean up any remaining active_mm state. exit_lazy_tlb +* is not done, if an arch did any accounting in these +* functions it would have to be added. */ if (mm != &init_mm) idle->active_mm = &init_mm;
kernel/kthread.c +1 -1
@@ -1251,9 +1251,9 @@ void kthread_use_mm(struct mm_struct *mm)finish_arch_post_lock_switch ();#endif +exit_lazy_tlb (active_mm, tsk);if (active_mm != mm)mmdrop (active_mm); -exit_lazy_tlb (active_mm, tsk);to_kthread (tsk)->oldfs =get_fs ();set_fs (USER_DS );
kernel/sched/core.c +7 -9
@@ -3294,22 +3294,19 @@ static struct rq *finish_task_switch(struct task_struct *prev)kcov_finish_switch (current);fire_sched_in_preempt_notifiers (current); +/* * When switching through a kernel thread, the loop in * membarrier_{private,global}_expedited() may have observed that * kernel thread and not issued an IPI. It is therefore possible to * schedule between user->kernel->user threads without passing though -* switch_mm(). Membarrier requires a barrier after storing to -* rq->curr, before returning to userspace, so provide them here: -* -* - a full memory barrier for {PRIVATE,GLOBAL}_EXPEDITED, implicitly -* provided by mmdrop(), -* - a sync_core for SYNC_CORE. +* switch_mm(). Membarrier requires a full barrier after storing to +* rq->curr, before returning to userspace, for +* {PRIVATE,GLOBAL}_EXPEDITED. This is implicitly provided by mmdrop(). */ -if (mm) { -membarrier_mm_sync_core_before_usermode (mm); +if (mm)mmdrop (mm); - } +if (unlikely (prev_state ==TASK_DEAD )) {if (prev->sched_class->task_dead) prev->sched_class->task_dead (prev);
@@ -6292,6 +6289,7 @@ void idle_task_exit(void)BUG_ON (current !=this_rq ()->idle);if (mm != &init_mm) { +/* enter_lazy_tlb is not done because we're about to go down */ switch_mm (mm, &init_mm, current);finish_arch_post_lock_switch (); }
Signature · 1 line
2.23.0
Nicholas Piggin
reviewerRFC PATCH 5/7 lazy tlb: introduce lazy mm refcount helper functions
In reply to Nicholas PigginAdd explicit _lazy_tlb annotated functions for lazy mm refcounting. This makes things a bit more explicit, and allows explicit refcounting to be removed if it is not used.
--- arch/powerpc/kernel/smp.c | 2 +- arch/powerpc/mm/book3s64/radix_tlb.c | 4 ++-- fs/exec.c | 2 +- include/linux/sched/mm.h | 17 +++++++++++++++++ kernel/cpu.c | 2 +- kernel/exit.c | 2 +- kernel/kthread.c | 11 +++++++---- kernel/sched/core.c | 13 +++++++------ 8 files changed, 37 insertions(+), 16 deletions(-)
arch/powerpc/kernel/smp.c +1 -1
@@ -1253,7 +1253,7 @@ void start_secondary(void *unused)unsigned int cpu =smp_processor_id ();struct cpumask *(*sibling_mask)(int ) = cpu_sibling_mask; -mmgrab (&init_mm); +mmgrab (&init_mm);/* XXX: where is the mmput for this? */ current->active_mm = &init_mm;smp_store_cpu_info (cpu);
arch/powerpc/mm/book3s64/radix_tlb.c +2 -2
@@ -652,10 +652,10 @@ static void do_exit_flush_lazy_tlb(void *arg)* Must be a kernel thread because sender is single-threaded. */ BUG_ON (current->mm); -mmgrab (&init_mm); +mmgrab_lazy_tlb (&init_mm);switch_mm (mm, &init_mm, current); current->active_mm = &init_mm; -mmdrop (mm); +mmdrop_lazy_tlb (mm); }_tlbiel_pid (pid,RIC_FLUSH_ALL ); }
fs/exec.c +1 -1
@@ -1119,7 +1119,7 @@ static int exec_mmap(struct mm_struct *mm)mmput (old_mm); }else {exit_lazy_tlb (active_mm, tsk); -mmdrop (active_mm); +mmdrop_lazy_tlb (active_mm); }return 0 ; }
include/linux/sched/mm.h +17 -0
@@ -50,6 +50,23 @@ static inline void mmdrop(struct mm_struct *mm)void mmdrop (struct mm_struct *mm); +/* Helpers for lazy TLB mm refcounting */ +static inline void mmgrab_lazy_tlb (struct mm_struct *mm) +{ +mmgrab (mm); +} + +static inline void mmdrop_lazy_tlb (struct mm_struct *mm) +{ +mmdrop (mm); +} + +static inline void mmdrop_lazy_tlb_smp_mb (struct mm_struct *mm) +{ +/* This depends on mmdrop providing a full smp_mb() */ +mmdrop (mm); +} +/* * This has to be called after a get_task_mm()/mmget_not_zero() * followed by taking the mmap_lock for writing before modifying the
kernel/cpu.c +1 -1
@@ -578,7 +578,7 @@ static int finish_cpu(unsigned int cpu)*/ if (mm != &init_mm) idle->active_mm = &init_mm; -mmdrop (mm); +mmdrop_lazy_tlb (mm);return 0 ; }
kernel/exit.c +1 -1
@@ -470,7 +470,7 @@ static void exit_mm(void)__set_current_state (TASK_RUNNING );mmap_read_lock (mm); } -mmgrab (mm); +mmgrab_lazy_tlb (mm);BUG_ON (mm != current->active_mm);/* more a memory barrier than a real lock */ task_lock (current);
kernel/kthread.c +7 -4
@@ -1238,12 +1238,12 @@ void kthread_use_mm(struct mm_struct *mm)WARN_ON_ONCE (!(tsk->flags &PF_KTHREAD ));WARN_ON_ONCE (tsk->mm); +mmgrab (mm); +task_lock (tsk); active_mm = tsk->active_mm; -if (active_mm != mm) { -mmgrab (mm); +if (active_mm != mm) tsk->active_mm = mm; - } tsk->mm = mm;switch_mm (active_mm, mm, tsk);task_unlock (tsk);
@@ -1253,7 +1253,7 @@ void kthread_use_mm(struct mm_struct *mm)exit_lazy_tlb (active_mm, tsk);if (active_mm != mm) -mmdrop (active_mm); +mmdrop_lazy_tlb (active_mm);to_kthread (tsk)->oldfs =get_fs ();set_fs (USER_DS );
@@ -1276,9 +1276,12 @@ void kthread_unuse_mm(struct mm_struct *mm)task_lock (tsk);sync_mm_rss (mm); tsk->mm =NULL ; +mmgrab_lazy_tlb (mm);/* active_mm is still 'mm' */ enter_lazy_tlb (mm, tsk);task_unlock (tsk); + +mmdrop (mm); }EXPORT_SYMBOL_GPL (kthread_unuse_mm);
kernel/sched/core.c +7 -6
@@ -3302,10 +3302,11 @@ static struct rq *finish_task_switch(struct task_struct *prev)* schedule between user->kernel->user threads without passing though * switch_mm(). Membarrier requires a full barrier after storing to * rq->curr, before returning to userspace, for -* {PRIVATE,GLOBAL}_EXPEDITED. This is implicitly provided by mmdrop(). +* {PRIVATE,GLOBAL}_EXPEDITED. This is implicitly provided by +* mmdrop_lazy_tlb_smp_mb(). */ if (mm) -mmdrop (mm); +mmdrop_lazy_tlb_smp_mb (mm);if (unlikely (prev_state ==TASK_DEAD )) {if (prev->sched_class->task_dead)
@@ -3410,9 +3411,9 @@ context_switch(struct rq *rq, struct task_struct *prev,/* * kernel -> kernel lazy + transfer active -* user -> kernel lazy + mmgrab() active +* user -> kernel lazy + mmgrab_lazy_tlb() active * -* kernel -> user switch + mmdrop() active +* kernel -> user switch + mmdrop_lazy_tlb() active * user -> user switch */ if (!next->mm) {// to kernel
@@ -3420,7 +3421,7 @@ context_switch(struct rq *rq, struct task_struct *prev, next->active_mm = prev->active_mm;if (prev->mm)// from user -mmgrab (prev->active_mm); +mmgrab_lazy_tlb (prev->active_mm);else prev->active_mm =NULL ; }else {// to user
@@ -3438,7 +3439,7 @@ context_switch(struct rq *rq, struct task_struct *prev,if (!prev->mm) {// from kernel exit_lazy_tlb (prev->active_mm, next); -/* will mmdrop() in finish_task_switch(). */ +/* will mmdrop_lazy_tlb() in finish_task_switch(). */ rq->prev_mm = prev->active_mm; prev->active_mm =NULL ; }
Signature · 1 line
2.23.0
Nicholas Piggin
reviewerRFC PATCH 6/7 lazy tlb: allow lazy tlb mm switching to be configurable
In reply to Nicholas PigginNOMMU systems could easily go without this and save a bit of code and the mm refcounting, because their mm switch is a no-op. I haven't flipped them over because haven't audited all arch code to convert over to using the _lazy_tlb refcounting.
--- arch/Kconfig | 7 +++++ include/linux/sched/mm.h | 12 ++++++--- kernel/sched/core.c | 55 +++++++++++++++++++++++++++------------- kernel/sched/sched.h | 4 ++- 4 files changed, 55 insertions(+), 23 deletions(-)
arch/Kconfig +7 -0
@@ -411,6 +411,13 @@ config MMU_GATHER_NO_GATHER bool depends on MMU_GATHER_TABLE_FREE +# Would like to make this depend on MMU, because there is little use for lazy mm switching +# with NOMMU, but have to audit NOMMU architecture code first. +config MMU_LAZY_TLB + def_bool y + help + Enable "lazy TLB" mmu context switching for kernel threads. + config ARCH_HAVE_NMI_SAFE_CMPXCHG bool
include/linux/sched/mm.h +8 -4
@@ -53,18 +53,22 @@ void mmdrop(struct mm_struct *mm);/* Helpers for lazy TLB mm refcounting */ static inline void mmgrab_lazy_tlb (struct mm_struct *mm) { -mmgrab (mm); +if (IS_ENABLED (CONFIG_MMU_LAZY_TLB )) +mmgrab (mm); }static inline void mmdrop_lazy_tlb (struct mm_struct *mm) { -mmdrop (mm); +if (IS_ENABLED (CONFIG_MMU_LAZY_TLB )) +mmdrop (mm); }static inline void mmdrop_lazy_tlb_smp_mb (struct mm_struct *mm) { -/* This depends on mmdrop providing a full smp_mb() */ -mmdrop (mm); +if (IS_ENABLED (CONFIG_MMU_LAZY_TLB )) +mmdrop (mm);/* This depends on mmdrop providing a full smp_mb() */ +else +smp_mb (); }/*
kernel/sched/core.c +37 -18
83 lines not on this page · raw message
kernel/sched/sched.h +3 -1
@@ -929,7 +929,9 @@ struct rq {struct task_struct *idle;struct task_struct *stop;unsigned long next_balance; -struct mm_struct *prev_mm; +#ifdef CONFIG_MMU_LAZY_TLB +struct mm_struct *prev_lazy_mm; +#endif unsigned int clock_update_flags;u64 clock;
Signature · 1 line
2.23.0
Nicholas Piggin
reviewerRFC PATCH 7/7 lazy tlb: shoot lazies, a non-refcounting lazy tlb option
In reply to Nicholas PigginOn big systems, the mm refcount can become highly contented when doing a lot of context switching with threaded applications (particularly switching between the idle thread and an application thread).
Abandoning lazy tlb slows switching down quite a bit in the important user->idle->user cases, so so instead implement a non-refcounted scheme that causes __mmdrop() to IPI all CPUs in the mm_cpumask and shoot down any remaining lazy ones.
On a 16-socket 192-core POWER8 system, a context switching benchmark with as many software threads as CPUs (so each switch will go in and out of idle), upstream can achieve a rate of about 1 million context switches per second. After this patch it goes up to 118 million.
--- arch/Kconfig | 16 ++++++++++++++++ arch/powerpc/Kconfig | 1 + include/linux/sched/mm.h | 6 +++--- kernel/fork.c | 39 +++++++++++++++++++++++++++++++++++++++ 4 files changed, 59 insertions(+), 3 deletions(-)
arch/Kconfig +16 -0
@@ -418,6 +418,22 @@ config MMU_LAZY_TLB help Enable "lazy TLB" mmu context switching for kernel threads. +config MMU_LAZY_TLB_REFCOUNT + def_bool y + depends on MMU_LAZY_TLB + depends on !MMU_LAZY_TLB_SHOOTDOWN + +config MMU_LAZY_TLB_SHOOTDOWN + bool + depends on MMU_LAZY_TLB + help + Instead of refcounting the "lazy tlb" mm struct, which can cause + contention with multi-threaded apps on large multiprocessor systems, + this option causes __mmdrop to IPI all CPUs in the mm_cpumask and + switch to init_mm if they were using the to-be-freed mm as the lazy + tlb. Architectures which do not track all possible lazy tlb CPUs in + mm_cpumask can not use this (without modification). + config ARCH_HAVE_NMI_SAFE_CMPXCHG bool
arch/powerpc/Kconfig +1 -0
@@ -225,6 +225,7 @@ config PPC select HAVE_PERF_USER_STACK_DUMP select MMU_GATHER_RCU_TABLE_FREE select MMU_GATHER_PAGE_SIZE + select MMU_LAZY_TLB_SHOOTDOWN select HAVE_REGS_AND_STACK_ACCESS_API select HAVE_RELIABLE_STACKTRACE if PPC_BOOK3S_64 && CPU_LITTLE_ENDIAN select HAVE_SYSCALL_TRACEPOINTS
include/linux/sched/mm.h +3 -3
@@ -53,19 +53,19 @@ void mmdrop(struct mm_struct *mm);/* Helpers for lazy TLB mm refcounting */ static inline void mmgrab_lazy_tlb (struct mm_struct *mm) { -if (IS_ENABLED (CONFIG_MMU_LAZY_TLB )) +if (IS_ENABLED (CONFIG_MMU_LAZY_TLB_REFCOUNT ))mmgrab (mm); }static inline void mmdrop_lazy_tlb (struct mm_struct *mm) { -if (IS_ENABLED (CONFIG_MMU_LAZY_TLB )) +if (IS_ENABLED (CONFIG_MMU_LAZY_TLB_REFCOUNT ))mmdrop (mm); }static inline void mmdrop_lazy_tlb_smp_mb (struct mm_struct *mm) { -if (IS_ENABLED (CONFIG_MMU_LAZY_TLB )) +if (IS_ENABLED (CONFIG_MMU_LAZY_TLB_REFCOUNT ))mmdrop (mm);/* This depends on mmdrop providing a full smp_mb() */ else smp_mb ();
kernel/fork.c +39 -0
@@ -685,6 +685,40 @@ static void check_mm(struct mm_struct *mm)#define allocate_mm () (kmem_cache_alloc(mm_cachep, GFP_KERNEL))#define free_mm (mm) (kmem_cache_free(mm_cachep, (mm))) +static void do_shoot_lazy_tlb (void *arg) +{ +struct mm_struct *mm = arg; + +if (current->active_mm == mm) { +BUG_ON (current->mm); +switch_mm (mm, &init_mm, current); + current->active_mm = &init_mm; + } +} + +static void do_check_lazy_tlb (void *arg) +{ +struct mm_struct *mm = arg; + +BUG_ON (current->active_mm == mm); +} + +static void shoot_lazy_tlbs (struct mm_struct *mm) +{ +if (IS_ENABLED (CONFIG_MMU_LAZY_TLB_SHOOTDOWN )) { +smp_call_function_many (mm_cpumask (mm), do_shoot_lazy_tlb, (void *)mm,1 ); +do_shoot_lazy_tlb (mm); + } +} + +static void check_lazy_tlbs (struct mm_struct *mm) +{ +if (IS_ENABLED (CONFIG_DEBUG_VM )) { +smp_call_function (do_check_lazy_tlb, (void *)mm,1 ); +do_check_lazy_tlb (mm); + } +} +/* * Called when the last reference to the mm * is dropped: either by a lazy thread or by
@@ -695,6 +729,11 @@ void __mmdrop(struct mm_struct *mm)BUG_ON (mm == &init_mm);WARN_ON_ONCE (mm == current->mm);WARN_ON_ONCE (mm == current->active_mm); + +/* Ensure no CPUs are using this as their lazy tlb mm */ +shoot_lazy_tlbs (mm); +check_lazy_tlbs (mm); +mm_free_pgd (mm);destroy_context (mm);mmu_notifier_subscriptions_destroy (mm);
Signature · 1 line
2.23.0
Peter Zijlstra
maintainerRe: [RFC PATCH 7/7] lazy tlb: shoot lazies, a non-refcounting lazy tlb option
Quoted 9 lines · Nicholas Piggin
On big systems, the mm refcount can become highly contented when doing a lot of context switching with threaded applications (particularly switching between the idle thread and an application thread).
Abandoning lazy tlb slows switching down quite a bit in the important
user->idle->usercases, so so instead implement a non-refcounted scheme that causes__mmdrop()to IPI all CPUs in themm_cpumaskand shoot down any remaining lazy ones.
Quoted 4 lines
On a 16-socket 192-core POWER8 system, a context switching benchmark with as many software threads as CPUs (so each switch will go in and out of idle), upstream can achieve a rate of about 1 million context switches per second. After this patch it goes up to 118 million.
That's mighty impressive, however:
kernel/fork.c static void check_mm(struct mm_struct *mm) · line 712
Quoted 7 lines
kernel/fork.c +7
-0+static void shoot_lazy_tlbs (struct mm_struct *mm) +{ +if (IS_ENABLED (CONFIG_MMU_LAZY_TLB_SHOOTDOWN )) { +smp_call_function_many (mm_cpumask (mm), do_shoot_lazy_tlb, (void *)mm,1 ); +do_shoot_lazy_tlb (mm); + } +}
IIRC you (power) never clear a CPU from that mask, so for other workloads I can see this resulting in massive amounts of IPIs.
For instance, take as many processes as you have CPUs. For each, manually walk the task across all CPUs and exit. Again.
Clearly, that's an extreme, but still...
Peter Zijlstra
maintainerRe: [RFC PATCH 4/7] x86: use exit_lazy_tlb rather than membarrier_mm_sync_core_before_usermode
In reply to Nicholas PigginQuoted 8 lines · Nicholas Piggin
And get rid of the generic
sync_core_before_usermodefacility.This helper is the wrong way around I think. The idea that membarrier state requires a core sync before returning to user is the easy one that does not need hiding behind membarrier calls. The gap in core synchronization due to x86's sysret/sysexit and lazy tlb mode, is the tricky detail that is better put in x86 lazy tlb code.
Quoted 8 lines
Consider if an arch did not synchronize core in
switch_mmeither, thenmembarrier_mm_sync_core_before_usermodewould be in the wrong place but arch specific mmu context functions would still be the right place. There is also aexit_lazy_tlbcase that is not covered by this call, which could be a bugs (kthread use mm the membarrier process's mm then context switch back to the process without switching mm or lazy mm switch).This makes lazy tlb code a bit more modular.
Hurmph, I know I've been staring at this at some point. I think I meant to have a TIF to force the IRET path in the case of MEMBAR_SYNC_CORE. But I was discouraged by amluto.
Peter Zijlstra
maintainerRe: [RFC PATCH 5/7] lazy tlb: introduce lazy mm refcount helper functions
In reply to Nicholas Pigginarch/powerpc/kernel/smp.c void start_secondary(void *unused) · line 1259
Quoted 13 lines · Nicholas Piggin
arch/powerpc/kernel/smp.c +1
-1@@ -1253,7 +1253,7 @@ void start_secondary(void *unused)unsigned int cpu =smp_processor_id ();struct cpumask *(*sibling_mask)(int ) = cpu_sibling_mask; -mmgrab (&init_mm); +mmgrab (&init_mm);/* XXX: where is the mmput for this? */ current->active_mm = &init_mm;smp_store_cpu_info (cpu);
Right; so IIRC it should be this one:
Quoted 12 lines
kernel/cpu.c +1
-1@@ -578,7 +578,7 @@ static int finish_cpu(unsigned int cpu)*/ if (mm != &init_mm) idle->active_mm = &init_mm; -mmdrop (mm); +mmdrop_lazy_tlb (mm);return 0 ; }
Mathieu Desnoyers
maintainerRe: [RFC PATCH 4/7] x86: use exit_lazy_tlb rather than membarrier_mm_sync_core_before_usermode
In reply to Nicholas PigginQuoted 8 lines · ----- On Jul 9, 2020, at 9:56 PM, Nicholas Piggin [email protected] wrote:
And get rid of the generic
sync_core_before_usermodefacility.This helper is the wrong way around I think. The idea that membarrier state requires a core sync before returning to user is the easy one that does not need hiding behind membarrier calls. The gap in core synchronization due to x86's sysret/sysexit and lazy tlb mode, is the tricky detail that is better put in x86 lazy tlb code.
Quoted 8 lines
Consider if an arch did not synchronize core in
switch_mmeither, thenmembarrier_mm_sync_core_before_usermodewould be in the wrong place but arch specific mmu context functions would still be the right place. There is also aexit_lazy_tlbcase that is not covered by this call, which could be a bugs (kthread use mm the membarrier process's mm then context switch back to the process without switching mm or lazy mm switch).This makes lazy tlb code a bit more modular.
I agree that moving this logic to exit_lazy_tlb is much more modular and cleaner.
Thanks,
Mathieu
Quoted 273 lines
Signed-off-by Nicholas Piggin <[email protected]>---
.../membarrier-sync-core/arch-support.txt | 6 +++- arch/x86/include/asm/mmu_context.h | 35 +++++++++++++++++++ arch/x86/include/asm/sync_core.h | 28 --------------- include/linux/sched/mm.h | 14 -------- include/linux/sync_core.h | 21 ----------- kernel/cpu.c | 4 ++- kernel/kthread.c | 2 +- kernel/sched/core.c | 16 ++++----- 8 files changed, 51 insertions(+), 75 deletions(-)delete mode 100644
arch/x86/include/asm/sync_core.hdelete mode 100644include/linux/sync_core.hDocumentation/features/sched/membarrier-sync-core/arch-support.txt +2
-0@@ -5,6 +5,10 @@ # # Architecture requirements # +# If your architecture returns to user-space through non-core-serializing +# instructions, you need to ensure these are done in switch_mm and
exit_lazy_tlbDocumentation/features/sched/membarrier-sync-core/arch-support.txt +2
-0+# (if lazy tlb switching is implemented). +## * arm/arm64/powerpc
#
# Rely on implicit context synchronization as a result of exception returnDocumentation/features/sched/membarrier-sync-core/arch-support.txt +1
-1@@ -24,7 +28,7 @@ # instead on write_cr3() performed by switch_mm() to provide core serialization # after changing the current mm, and deal with the special case of kthread -> # uthread (temporarily keeping current mm into active_mm) by issuing a -# sync_core_before_usermode() in that specific case. +# serializing instruction in exit_lazy_mm() in that specific case. # ----------------------- | arch |status|arch/x86/include/asm/mmu_context.h +35
-0@@ -6,6 +6,7 @@#include <linux/atomic.h> #include <linux/mm_types.h> #include <linux/pkeys.h> +#include <linux/sched/mm.h> #include <trace/events/tlb.h> @@ -95,6 +96,40 @@ static inline void switch_ldt(struct mm_struct *prev, struct mm_struct *next)#define enter_lazy_tlb enter_lazy_tlbextern void enter_lazy_tlb (struct mm_struct *mm,struct task_struct *tsk); +#ifdef CONFIG_MEMBARRIER +/* +* Ensure that a core serializing instruction is issued before returning +* to user-mode, if a SYNC_CORE was requested. x86 implements return to +* user-space through sysexit, sysrel, and sysretq, which are not core +* serializing. +* +* See the membarrier comment in finish_task_switch as to why this is done +* in exit_lazy_tlb. +*/ +#define exit_lazy_tlb exit_lazy_tlb +static inline void exit_lazy_tlb (struct mm_struct *mm,struct task_struct *tsk) +{ +/* Switching mm is serializing with write_cr3 */ +if (tsk->mm != mm) +return ; + +if (likely (!(atomic_read (&mm->membarrier_state) & +MEMBARRIER_STATE_PRIVATE_EXPEDITED_SYNC_CORE ))) +return ; + +/* With PTI, we unconditionally serialize before running user code. */ +if (static_cpu_has (X86_FEATURE_PTI )) +return ; +/* +* Return from interrupt and NMI is done through iret, which is core +* serializing. +*/ +if (in_irq () ||in_nmi ()) +return ; +sync_core (); +} +#endif +/* * Init a new mm. Used on mm copies, like at fork() * and on mm's that are brand-new, like at
execve().arch/x86/include/asm/sync_core.h deleted +0
-28@@ -1,28 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0 */ -#ifndef _ASM_X86_SYNC_CORE_H -#define _ASM_X86_SYNC_CORE_H - -#include <linux/preempt.h> -#include <asm/processor.h> -#include <asm/cpufeature.h> - -/* -* Ensure that a core serializing instruction is issued before returning -* to user-mode. x86 implements return to user-space through sysexit, -* sysrel, and sysretq, which are not core serializing. -*/ -static inline void sync_core_before_usermode (void ) -{ -/* With PTI, we unconditionally serialize before running user code. */ -if (static_cpu_has (X86_FEATURE_PTI )) -return ; -/* -* Return from interrupt and NMI is done through iret, which is core -* serializing. -*/ -if (in_irq () ||in_nmi ()) -return ; -sync_core (); -} - -#endif /* _ASM_X86_SYNC_CORE_H */ include/linux/sched/mm.h +0
-11@@ -7,7 +7,6 @@#include <linux/sched.h> #include <linux/mm_types.h> #include <linux/gfp.h> -#include <linux/sync_core.h> /* * Routines for handling mm_structs @@ -364,16 +363,6 @@ enum {#include <asm/membarrier.h> #endif -static inline void membarrier_mm_sync_core_before_usermode (struct mm_struct *mm) -{ -if (current->mm != mm) -return ; -if (likely (!(atomic_read (&mm->membarrier_state) & -MEMBARRIER_STATE_PRIVATE_EXPEDITED_SYNC_CORE ))) -return ; -sync_core_before_usermode (); -} -extern void membarrier_exec_mmap (struct mm_struct *mm);#else
include/linux/sched/mm.h +0
-3@@ -387,9 +376,6 @@ static inline void membarrier_arch_switch_mm(structmm_struct *prev, static inline void membarrier_exec_mmap(struct mm_struct *mm) { } -static inline void membarrier_mm_sync_core_before_usermode (struct mm_struct *mm) -{ -} #endif#endif /*
_LINUX_SCHED_MM_H*/include/linux/sync_core.h deleted +0
-21@@ -1,21 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0 */ -#ifndef _LINUX_SYNC_CORE_H -#define _LINUX_SYNC_CORE_H - -#ifdef CONFIG_ARCH_HAS_SYNC_CORE_BEFORE_USERMODE -#include <asm/sync_core.h> -#else -/* -* This is a dummy sync_core_before_usermode() implementation that can be used -* on all architectures which return to user-space through core serializing -* instructions. -* If your architecture returns to user-space through non-core-serializing -* instructions, you need to write your own functions. -*/ -static inline void sync_core_before_usermode (void ) -{ -} -#endif - -#endif /* _LINUX_SYNC_CORE_H */ -kernel/cpu.c +3
-1@@ -572,7 +572,9 @@ static int finish_cpu(unsigned int cpu)/* * idle_task_exit() will have switched to &init_mm, now -* clean up any remaining active_mm state. +* clean up any remaining active_mm state. exit_lazy_tlb +* is not done, if an arch did any accounting in these +* functions it would have to be added. */ if (mm != &init_mm) idle->active_mm = &init_mm;kernel/kthread.c +1
-1@@ -1251,9 +1251,9 @@ void kthread_use_mm(struct mm_struct *mm)finish_arch_post_lock_switch ();#endif +exit_lazy_tlb (active_mm, tsk);if (active_mm != mm)mmdrop (active_mm); -exit_lazy_tlb (active_mm, tsk);to_kthread (tsk)->oldfs =get_fs ();set_fs (USER_DS );kernel/sched/core.c +6
-9@@ -3294,22 +3294,19 @@ static struct rq *finish_task_switch(struct task_struct *prev)kcov_finish_switch (current);fire_sched_in_preempt_notifiers (current); +/* * When switching through a kernel thread, the loop in * membarrier_{private,global}_expedited() may have observed that * kernel thread and not issued an IPI. It is therefore possible to * schedule between user->kernel->user threads without passing though -* switch_mm(). Membarrier requires a barrier after storing to -* rq->curr, before returning to userspace, so provide them here: -* -* - a full memory barrier for {PRIVATE,GLOBAL}_EXPEDITED, implicitly -* provided by mmdrop(), -* - a sync_core for SYNC_CORE. +* switch_mm(). Membarrier requires a full barrier after storing to +* rq->curr, before returning to userspace, for +* {PRIVATE,GLOBAL}_EXPEDITED. This is implicitly provided by mmdrop(). */ -if (mm) { -membarrier_mm_sync_core_before_usermode (mm); +if (mm)mmdrop (mm); - } +if (unlikely (prev_state ==TASK_DEAD )) {if (prev->sched_class->task_dead)prev->sched_class->task_dead(prev);kernel/sched/core.c +1
-1@@ -6292,6 +6289,7 @@ void idle_task_exit(void)BUG_ON (current !=this_rq ()->idle);if (mm != &init_mm) { +/* enter_lazy_tlb is not done because we're about to go down */ switch_mm (mm, &init_mm, current);finish_arch_post_lock_switch (); } --2.23.0
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com
Re: [RFC PATCH 4/7] x86: use exit_lazy_tlb rather than membarrier_mm_sync_core_before_usermode
In reply to Nicholas PigginQuoted 9 lines · Nicholas Piggin
And get rid of the generic
sync_core_before_usermodefacility.This helper is the wrong way around I think. The idea that membarrier state requires a core sync before returning to user is the easy one that does not need hiding behind membarrier calls. The gap in core synchronization due to x86's sysret/sysexit and lazy tlb mode, is the tricky detail that is better put in x86 lazy tlb code.
Quoted 8 lines
Consider if an arch did not synchronize core in
switch_mmeither, thenmembarrier_mm_sync_core_before_usermodewould be in the wrong place but arch specific mmu context functions would still be the right place. There is also aexit_lazy_tlbcase that is not covered by this call, which could be a bugs (kthread use mm the membarrier process's mm then context switch back to the process without switching mm or lazy mm switch).This makes lazy tlb code a bit more modular.
The mm-switching and TLB-management has often had the regrettable property that it gets wired up in a way that seems to work at the time but doesn't have clear semantics, and I'm a bit concerned that this patch is in that category. If I'm understanding right, you're trying to enforce the property that exiting lazy TLB mode will promise to sync the core eventually. But this has all kinds of odd properties:
- Why is exit_lazy_tlb() getting called at all in the relevant cases? When is it permissible to call it? I look at your new code and see:
Quoted 5 lines
arch/x86/include/asm/mmu_context.h +5
-0+/* +* Ensure that a core serializing instruction is issued before returning +* to user-mode, if a SYNC_CORE was requested. x86 implements return to +* user-space through sysexit, sysrel, and sysretq, which are not core +* serializing.
arch/x86/include/asm/mmu_context.h static inline void switch_ldt(struct mm_struct *prev, struct mm_struct *next) · line 114
Quoted 10 lines
arch/x86/include/asm/mmu_context.h +10
-0+* +* See the membarrier comment in finish_task_switch as to why this is done +* in exit_lazy_tlb. +*/ +#define exit_lazy_tlb exit_lazy_tlb +static inline void exit_lazy_tlb (struct mm_struct *mm,struct task_struct *tsk) +{ +/* Switching mm is serializing with write_cr3 */ +if (tsk->mm != mm) +return ;
And my brain says WTF? Surely you meant something like if (WARN_ON_ONCE(tsk->mm != mm)) { /* egads, what even happened? how do we try to recover well enough to get a crashed logged at least? */ }
So this needs actual documentation, preferably in comments near the function, of what the preconditions are and what this mm parameter is. Once that's done, then we could consider whether it's appropriate to have this function promise to sync the core under some conditions.
- This whole structure seems to rely on the idea that switching mm
syncs something. I periodically ask chip vendor for non-serializing mm switches. Specifically, in my dream world, we have totally separate user and kernel page tables. Changing out the user tables doesn't serialize or even create a fence. Instead it creates the minimum required pipeline hazard such that user memory access and switches to usermode will make sure they access memory through the correct page tables. I haven't convinced a chip vendor yet, but there are quite a few hundreds of cycles to be saved here. With this in mind, I see the fencing aspects of the TLB handling code as somewhat of an accident. I'm fine with documenting them and using them to optimize other paths, but I think it should be explicit. For example:
/* Also does a full barrier? (Or a sync_core()-style barrier.) However, if you rely on this, you must document it in a comment where you call this function. *?
void switch_mm_irqs_off()
{
}
This is kind of like how we strongly encourage anyone using smp_?mb() to document what they are fencing against.
Also, as it stands, I can easily see in_irq() ceasing to promise to serialize. There are older kernels for which it does not promise to serialize. And I have plans to make it stop serializing in the nearish future.
--Andy
Nicholas Piggin
reviewerRe: [RFC PATCH 4/7] x86: use exit_lazy_tlb rather than membarrier_mm_sync_core_before_usermode
Excerpts from Andy Lutomirski's message of July 11, 2020 3:04 am:
Quoted 17 lines · Nicholas Piggin
Quoted 16 lines
And get rid of the generic
sync_core_before_usermodefacility.This helper is the wrong way around I think. The idea that membarrier state requires a core sync before returning to user is the easy one that does not need hiding behind membarrier calls. The gap in core synchronization due to x86's sysret/sysexit and lazy tlb mode, is the tricky detail that is better put in x86 lazy tlb code.
Consider if an arch did not synchronize core in
switch_mmeither, thenmembarrier_mm_sync_core_before_usermodewould be in the wrong place but arch specific mmu context functions would still be the right place. There is also aexit_lazy_tlbcase that is not covered by this call, which could be a bugs (kthread use mm the membarrier process's mm then context switch back to the process without switching mm or lazy mm switch).
Quoted 6 lines
Quoted 1 line
This makes lazy tlb code a bit more modular.
The mm-switching and TLB-management has often had the regrettable property that it gets wired up in a way that seems to work at the time but doesn't have clear semantics, and I'm a bit concerned that this patch is in that category.
It's much more explicit in the core code about where hooks are called after this patch. And then the x86 membarrier implementation details are contained to the x86 code where they belong, and we don't have the previous hook with unclear semantics missing from core code.
Quoted 5 lines
If I'm understanding right, you're trying
to enforce the property that exiting lazy TLB mode will promise to sync the core eventually. But this has all kinds of odd properties:- Why is
exit_lazy_tlb()getting called at all in the relevant cases?
It's a property of how MEMBARRIER_SYNC_CORE is implemented by arch/x86, see the membarrier comment in finish_task_switch (for analogous reason).
Quoted 1 line
When is it permissible to call it?
Comment for the asm-generic code says it's to be called when the lazy active mm becomes non-lazy.
Quoted 13 lines
I look at your new code and see:
Quoted 11 lines
arch/x86/include/asm/mmu_context.h +11
-0+/* +* Ensure that a core serializing instruction is issued before returning +* to user-mode, if a SYNC_CORE was requested. x86 implements return to +* user-space through sysexit, sysrel, and sysretq, which are not core +* serializing. +* +* See the membarrier comment in finish_task_switch as to why this is done +* in exit_lazy_tlb. +*/ +#define exit_lazy_tlb exit_lazy_tlb +static inline void exit_lazy_tlb (struct mm_struct *mm,struct task_struct *tsk)
Quoted 8 lines
arch/x86/include/asm/mmu_context.h static inline void switch_ldt(struct mm_struct *prev, struct mm_struct *next) · line 114
Quoted 4 lines
arch/x86/include/asm/mmu_context.h +4
-0+{ +/* Switching mm is serializing with write_cr3 */ +if (tsk->mm != mm) +return ;And my brain says WTF? Surely you meant something like if (
WARN_ON_ONCE(tsk->mm != mm)) { /* egads, what even happened? how do we try to recover well enough to get a crashed logged at least? */ }
No, the active mm can be unlazied by switching to a different mm.
Quoted 4 lines
So this needs actual documentation, preferably in comments near the function, of what the preconditions are and what this mm parameter is. Once that's done, then we could consider whether it's appropriate to have this function promise to sync the core under some conditions.
It's documented in generic code. I prefer not to duplicate comments too much but I can add a "refer to asm-generic version for usage" or something if you'd like?
Quoted 2 lines
- This whole structure seems to rely on the idea that switching mm
syncs something.
Which whole structure? The x86 implementation of sync core explicitly does rely on that, yes. But I've pulled that out of core code with this patch.
Quoted 8 lines
I periodically ask chip vendor for non-serializing
mm switches. Specifically, in my dream world, we have totally separate user and kernel page tables. Changing out the user tables doesn't serialize or even create a fence. Instead it creates the minimum required pipeline hazard such that user memory access and switches to usermode will make sure they access memory through the correct page tables. I haven't convinced a chip vendor yet, but there are quite a few hundreds of cycles to be saved here.
The fundmaental difficulty is that the kernel can still access user mappings any time after the switch. We can probably handwave ways around it by serializing lazily when encountering the next user access and hoping that most of your mm switches result in a kernel exit that serializes or some other unavoidable serialize so you can avoid the mm switch one. In practice it sounds like a lot of trouble. But anyway the sync core could presumably be adjusted or reasoned to still be correct, depending on how it works.
Quoted 12 lines
With this in
mind, I see the fencing aspects of the TLB handling code as somewhat of an accident. I'm fine with documenting them and using them to optimize other paths, but I think it should be explicit. For example:/* Also does a full barrier? (Or a sync_core()-style barrier.) However, if you rely on this, you must document it in a comment where you call this function. *?
voidswitch_mm_irqs_off()
{
}
Quoted 2 lines
This is kind of like how we strongly encourage anyone using smp_?mb() to document what they are fencing against.
Hmm. I don't think anything outside core scheduler/arch code is allowed to assume that, because they don't really know if schedule() will cause a switch. Hopefully nobody does, I would agree it shouldn't be encouraged.
It is pretty fundamental to how we do task CPU migration so I don't see it ever going away. A push model where the source CPU has to release tasks that it last ran before they can be run elsewhere is unworkable. (Or maybe it's not, but no getting around that would require careful audits of said low level code).
Quoted 4 lines
Also, as it stands, I can easily see
in_irq()ceasing to promise to serialize. There are older kernels for which it does not promise to serialize. And I have plans to make it stop serializing in the nearish future.
You mean x86's return from interrupt? Sounds fun... you'll konw where to update the membarrier sync code, at least :)
Thanks,
Nick
Nicholas Piggin
reviewerRe: [RFC PATCH 7/7] lazy tlb: shoot lazies, a non-refcounting lazy tlb option
In reply to Peter ZijlstraExcerpts from Peter Zijlstra's message of July 10, 2020 7:35 pm:
Quoted 10 lines · Nicholas Piggin
Quoted 9 lines
On big systems, the mm refcount can become highly contented when doing a lot of context switching with threaded applications (particularly switching between the idle thread and an application thread).
Abandoning lazy tlb slows switching down quite a bit in the important
user->idle->usercases, so so instead implement a non-refcounted scheme that causes__mmdrop()to IPI all CPUs in themm_cpumaskand shoot down any remaining lazy ones.
Quoted 6 lines
Quoted 4 lines
On a 16-socket 192-core POWER8 system, a context switching benchmark with as many software threads as CPUs (so each switch will go in and out of idle), upstream can achieve a rate of about 1 million context switches per second. After this patch it goes up to 118 million.
That's mighty impressive, however:
Well, it's the usual case of "find a bouncing line and scale up the machine size until you achieve your desired improvements" :) But we are looking at some fundamental scalabilities and seeing if we can improve a few things.
Quoted 12 lines
kernel/fork.c static void check_mm(struct mm_struct *mm) · line 712
Quoted 8 lines
kernel/fork.c +7
-0+static void shoot_lazy_tlbs (struct mm_struct *mm) +{ +if (IS_ENABLED (CONFIG_MMU_LAZY_TLB_SHOOTDOWN )) { +smp_call_function_many (mm_cpumask (mm), do_shoot_lazy_tlb, (void *)mm,1 ); +do_shoot_lazy_tlb (mm); + } +}IIRC you (power) never clear a CPU from that mask, so for other workloads I can see this resulting in massive amounts of IPIs.
Quoted 4 lines
For instance, take as many processes as you have CPUs. For each, manually walk the task across all CPUs and exit. Again.
Clearly, that's an extreme, but still...
We do have some issues with that, it does tend to be very self-limiting though, short lived tasks that can drive lots of exits won't get to run on a lot of cores.
It's worth keeping an eye on, it may not be too hard to mitigate the IPIs doing something dumb like collecting a queue of mms before killing a batch of them.
Thanks,
Nick
Nicholas Piggin
reviewerRe: [RFC PATCH 4/7] x86: use exit_lazy_tlb rather than membarrier_mm_sync_core_before_usermode
In reply to Nicholas PigginExcerpts from Nicholas Piggin's message of July 13, 2020 2:45 pm:
Quoted 8 lines
Excerpts from Andy Lutomirski's message of July 11, 2020 3:04 am:
Quoted 4 lines
Also, as it stands, I can easily see
in_irq()ceasing to promise to serialize. There are older kernels for which it does not promise to serialize. And I have plans to make it stop serializing in the nearish future.You mean x86's return from interrupt? Sounds fun... you'll konw where to update the membarrier sync code, at least :)
Oh, I should actually say Mathieu recently clarified a return from interrupt doesn't fundamentally need to serialize in order to support membarrier sync core.
https://lists.ozlabs.org/pipermail/linuxppc-dev/2020-July/214171.html
So you may not need to do anything more if you relaxed it.
Thanks,
Nick
Mathieu Desnoyers
maintainerRe: [RFC PATCH 4/7] x86: use exit_lazy_tlb rather than membarrier_mm_sync_core_before_usermode
Quoted 7 lines · ----- On Jul 13, 2020, at 9:47 AM, Nicholas Piggin [email protected] wrote:
Excerpts from Nicholas Piggin's message of July 13, 2020 2:45 pm:
Quoted 6 lines
Excerpts from Andy Lutomirski's message of July 11, 2020 3:04 am:
Quoted 4 lines
Also, as it stands, I can easily see
in_irq()ceasing to promise to serialize. There are older kernels for which it does not promise to serialize. And I have plans to make it stop serializing in the nearish future.
Quoted 6 lines
Quoted 2 lines
You mean x86's return from interrupt? Sounds fun... you'll konw where to update the membarrier sync code, at least :)
Oh, I should actually say Mathieu recently clarified a return from interrupt doesn't fundamentally need to serialize in order to support membarrier sync core.
Clarification to your statement:
Return from interrupt to kernel code does not need to be context serializing as long as kernel serializes before returning to user-space.
However, return from interrupt to user-space needs to be context serializing.
Thanks,
Mathieu
Quoted 7 lines
https://lists.ozlabs.org/pipermail/linuxppc-dev/2020-July/214171.html
So you may not need to do anything more if you relaxed it.
Thanks,
Nick
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com
Re: [RFC PATCH 4/7] x86: use exit_lazy_tlb rather than membarrier_mm_sync_core_before_usermode
Quoted 17 lines · Mathieu Desnoyers
Quoted 7 lines · ----- On Jul 13, 2020, at 9:47 AM, Nicholas Piggin [email protected] wrote:
Excerpts from Nicholas Piggin's message of July 13, 2020 2:45 pm:
Quoted 6 lines
Excerpts from Andy Lutomirski's message of July 11, 2020 3:04 am:
Quoted 4 lines
Also, as it stands, I can easily see
in_irq()ceasing to promise to serialize. There are older kernels for which it does not promise to serialize. And I have plans to make it stop serializing in the nearish future.Quoted 6 lines
Quoted 2 lines
You mean x86's return from interrupt? Sounds fun... you'll konw where to update the membarrier sync code, at least :)
Oh, I should actually say Mathieu recently clarified a return from interrupt doesn't fundamentally need to serialize in order to support membarrier sync core.
Quoted 7 lines
Clarification to your statement:
Return from interrupt to kernel code does not need to be context serializing as long as kernel serializes before returning to user-space.
However, return from interrupt to user-space needs to be context serializing.
Indeed, and I figured this out on the first read through because I'm quite familiar with the x86 entry code. But Nick somehow missed this, and Nick is the one who wrote the patch.
Nick, I think this helps prove my point. The code you're submitting may well be correct, but it's unmaintainable. At the very least, this needs a comment explaining, from the perspective of x86, *exactly* what exit_lazy_tlb() is promising, why it's promising it, how it achieves that promise, and what code cares about it. Or we could do something with TIF flags and make this all less magical, although that will probably end up very slightly slower.
--Andy
Re: [RFC PATCH 7/7] lazy tlb: shoot lazies, a non-refcounting lazy tlb option
In reply to Nicholas PigginQuoted 10 lines · Nicholas Piggin
On big systems, the mm refcount can become highly contented when doing a lot of context switching with threaded applications (particularly switching between the idle thread and an application thread).
Abandoning lazy tlb slows switching down quite a bit in the important
user->idle->usercases, so so instead implement a non-refcounted scheme that causes__mmdrop()to IPI all CPUs in themm_cpumaskand shoot down any remaining lazy ones.
Quoted 5 lines
On a 16-socket 192-core POWER8 system, a context switching benchmark with as many software threads as CPUs (so each switch will go in and out of idle), upstream can achieve a rate of about 1 million context switches per second. After this patch it goes up to 118 million.
I read the patch a couple of times, and I have a suggestion that could be nonsense. You are, effectively, using mm_cpumask() as a sort of refcount. You're saying "hey, this mm has no more references, but it still has nonempty mm_cpumask(), so let's send an IPI and shoot down those references too." I'm wondering whether you actually need the IPI. What if, instead, you actually treated mm_cpumask as a refcount for real? Roughly, in __mmdrop(), you would only free the page tables if mm_cpumask() is empty. And, in the code that removes a CPU from mm_cpumask(), you would check if mm_users == 0 and, if so, check if you just removed the last bit from mm_cpumask and potentially free the mm.
Getting the locking right here could be a bit tricky -- you need to avoid two CPUs simultaneously exiting lazy TLB and thinking they should free the mm, and you also need to avoid an mm with mm_users hitting zero concurrently with the last remote CPU using it lazily exiting lazy TLB. Perhaps this could be resolved by having mm_count == 1 mean "mm_cpumask() is might contain bits and, if so, it owns the mm" and mm_count == 0 meaning "now it's dead" and using some careful cmpxchg or dec_return to make sure that only one CPU frees it.
Or maybe you'd need a lock or RCU for this, but the idea would be to only ever take the lock after mm_users goes to zero.
--Andy
Nicholas Piggin
reviewerRe: [RFC PATCH 4/7] x86: use exit_lazy_tlb rather than membarrier_mm_sync_core_before_usermode
In reply to Andy LutomirskiExcerpts from Andy Lutomirski's message of July 14, 2020 1:48 am:
Quoted 27 lines
Quoted 17 lines · Mathieu Desnoyers
Quoted 7 lines · ----- On Jul 13, 2020, at 9:47 AM, Nicholas Piggin [email protected] wrote:
Excerpts from Nicholas Piggin's message of July 13, 2020 2:45 pm:
Quoted 6 lines
Excerpts from Andy Lutomirski's message of July 11, 2020 3:04 am:
Quoted 4 lines
Also, as it stands, I can easily see
in_irq()ceasing to promise to serialize. There are older kernels for which it does not promise to serialize. And I have plans to make it stop serializing in the nearish future.Quoted 6 lines
Quoted 2 lines
You mean x86's return from interrupt? Sounds fun... you'll konw where to update the membarrier sync code, at least :)
Oh, I should actually say Mathieu recently clarified a return from interrupt doesn't fundamentally need to serialize in order to support membarrier sync core.
Quoted 7 lines
Clarification to your statement:
Return from interrupt to kernel code does not need to be context serializing as long as kernel serializes before returning to user-space.
However, return from interrupt to user-space needs to be context serializing.
Quoted 6 lines
Indeed, and I figured this out on the first read through because I'm quite familiar with the x86 entry code. But Nick somehow missed this, and Nick is the one who wrote the patch.
Nick, I think this helps prove my point. The code you're submitting may well be correct, but it's unmaintainable.
It's not. The patch I wrote for x86 is a no-op, it just moves existing x86 hook and code that's already there to a different name.
Actually it's not quite a no-op, it't changes it to use hooks that are actually called in the right places. Because previously it was unmaintainable from point of view of generic mm -- it was not clear at all that the old one should have been called in other places where the mm goes non-lazy. Now with the exit_lazy_tlb hook, it can quite easily be spotted where it is missing.
And x86 keeps their membarrier code in x86, and uses nice well defined lazy tlb mm hooks.
Quoted 6 lines
At the very least, this
needs a comment explaining, from the perspective of x86, *exactly* whatexit_lazy_tlb()is promising, why it's promising it, how it achieves that promise, and what code cares about it. Or we could do something with TIF flags and make this all less magical, although that will probably end up very slightly slower.
It's all documented there in existing comments plus the asm-generic exit_lazy_tlb specification added AFAIKS.
Is the membarrier comment in finish_task_switch plus these ones not enough?
Thanks,
Nick
Nicholas Piggin
reviewerRe: [RFC PATCH 7/7] lazy tlb: shoot lazies, a non-refcounting lazy tlb option
In reply to Andy LutomirskiExcerpts from Andy Lutomirski's message of July 14, 2020 1:59 am:
Quoted 38 lines · Nicholas Piggin
Quoted 10 lines
On big systems, the mm refcount can become highly contented when doing a lot of context switching with threaded applications (particularly switching between the idle thread and an application thread).
Abandoning lazy tlb slows switching down quite a bit in the important
user->idle->usercases, so so instead implement a non-refcounted scheme that causes__mmdrop()to IPI all CPUs in themm_cpumaskand shoot down any remaining lazy ones.Quoted 5 lines
On a 16-socket 192-core POWER8 system, a context switching benchmark with as many software threads as CPUs (so each switch will go in and out of idle), upstream can achieve a rate of about 1 million context switches per second. After this patch it goes up to 118 million.
I read the patch a couple of times, and I have a suggestion that could be nonsense. You are, effectively, using
mm_cpumask()as a sort of refcount. You're saying "hey, this mm has no more references, but it still has nonemptymm_cpumask(), so let's send an IPI and shoot down those references too." I'm wondering whether you actually need the IPI. What if, instead, you actually treatedmm_cpumaskas a refcount for real? Roughly, in__mmdrop(), you would only free the page tables ifmm_cpumask()is empty. And, in the code that removes a CPU frommm_cpumask(), you would check ifmm_users== 0 and, if so, check if you just removed the last bit frommm_cpumaskand potentially free the mm.Getting the locking right here could be a bit tricky -- you need to avoid two CPUs simultaneously exiting lazy TLB and thinking they should free the mm, and you also need to avoid an mm with
mm_usershitting zero concurrently with the last remote CPU using it lazily exiting lazy TLB. Perhaps this could be resolved by havingmm_count== 1 mean "mm_cpumask()is might contain bits and, if so, it owns the mm" andmm_count== 0 meaning "now it's dead" and using some careful cmpxchg ordec_returnto make sure that only one CPU frees it.
Quoted 2 lines
Or maybe you'd need a lock or RCU for this, but the idea would be to only ever take the lock after
mm_usersgoes to zero.
I don't think it's nonsense, it could be a good way to avoid IPIs.
I haven't seen much problem here that made me too concerned about IPIs yet, so I think the simple patch may be good enough to start with for powerpc. I'm looking at avoiding/reducing the IPIs by combining the unlazying with the exit TLB flush without doing anything fancy with ref counting, but we'll see.
Thanks,
Nick
Re: [RFC PATCH 7/7] lazy tlb: shoot lazies, a non-refcounting lazy tlb option
Quoted 45 lines · Nicholas Piggin
Excerpts from Andy Lutomirski's message of July 14, 2020 1:59 am:
Quoted 38 lines
Quoted 11 lines · Nicholas Piggin
On big systems, the mm refcount can become highly contented when doing a lot of context switching with threaded applications (particularly switching between the idle thread and an application thread).
Abandoning lazy tlb slows switching down quite a bit in the important
user->idle->usercases, so so instead implement a non-refcounted scheme that causes__mmdrop()to IPI all CPUs in themm_cpumaskand shoot down any remaining lazy ones.Quoted 5 lines
On a 16-socket 192-core POWER8 system, a context switching benchmark with as many software threads as CPUs (so each switch will go in and out of idle), upstream can achieve a rate of about 1 million context switches per second. After this patch it goes up to 118 million.
I read the patch a couple of times, and I have a suggestion that could be nonsense. You are, effectively, using
mm_cpumask()as a sort of refcount. You're saying "hey, this mm has no more references, but it still has nonemptymm_cpumask(), so let's send an IPI and shoot down those references too." I'm wondering whether you actually need the IPI. What if, instead, you actually treatedmm_cpumaskas a refcount for real? Roughly, in__mmdrop(), you would only free the page tables ifmm_cpumask()is empty. And, in the code that removes a CPU frommm_cpumask(), you would check ifmm_users== 0 and, if so, check if you just removed the last bit frommm_cpumaskand potentially free the mm.Getting the locking right here could be a bit tricky -- you need to avoid two CPUs simultaneously exiting lazy TLB and thinking they should free the mm, and you also need to avoid an mm with
mm_usershitting zero concurrently with the last remote CPU using it lazily exiting lazy TLB. Perhaps this could be resolved by havingmm_count== 1 mean "mm_cpumask()is might contain bits and, if so, it owns the mm" andmm_count== 0 meaning "now it's dead" and using some careful cmpxchg ordec_returnto make sure that only one CPU frees it.Quoted 2 lines
Or maybe you'd need a lock or RCU for this, but the idea would be to only ever take the lock after
mm_usersgoes to zero.
Quoted 7 lines
I don't think it's nonsense, it could be a good way to avoid IPIs.
I haven't seen much problem here that made me too concerned about IPIs yet, so I think the simple patch may be good enough to start with for powerpc. I'm looking at avoiding/reducing the IPIs by combining the unlazying with the exit TLB flush without doing anything fancy with ref counting, but we'll see.
I would be cautious with benchmarking here. I would expect that the nasty cases may affect power consumption more than performance — the specific issue is IPIs hitting idle cores, and the main effects are to slow down exit() a bit but also to kick the idle core out of idle. Although, if the idle core is in a deep sleep, that IPI could be *very* slow.
So I think it’s worth at least giving this a try.
Quoted 3 lines
Thanks,
Nick
Nicholas Piggin
reviewerRe: [RFC PATCH 7/7] lazy tlb: shoot lazies, a non-refcounting lazy tlb option
Excerpts from Andy Lutomirski's message of July 14, 2020 4:18 am:
Quoted 53 lines
Quoted 45 lines · Nicholas Piggin
Excerpts from Andy Lutomirski's message of July 14, 2020 1:59 am:
Quoted 38 lines
Quoted 11 lines · Nicholas Piggin
On big systems, the mm refcount can become highly contented when doing a lot of context switching with threaded applications (particularly switching between the idle thread and an application thread).
Abandoning lazy tlb slows switching down quite a bit in the important
user->idle->usercases, so so instead implement a non-refcounted scheme that causes__mmdrop()to IPI all CPUs in themm_cpumaskand shoot down any remaining lazy ones.Quoted 5 lines
On a 16-socket 192-core POWER8 system, a context switching benchmark with as many software threads as CPUs (so each switch will go in and out of idle), upstream can achieve a rate of about 1 million context switches per second. After this patch it goes up to 118 million.
I read the patch a couple of times, and I have a suggestion that could be nonsense. You are, effectively, using
mm_cpumask()as a sort of refcount. You're saying "hey, this mm has no more references, but it still has nonemptymm_cpumask(), so let's send an IPI and shoot down those references too." I'm wondering whether you actually need the IPI. What if, instead, you actually treatedmm_cpumaskas a refcount for real? Roughly, in__mmdrop(), you would only free the page tables ifmm_cpumask()is empty. And, in the code that removes a CPU frommm_cpumask(), you would check ifmm_users== 0 and, if so, check if you just removed the last bit frommm_cpumaskand potentially free the mm.Getting the locking right here could be a bit tricky -- you need to avoid two CPUs simultaneously exiting lazy TLB and thinking they should free the mm, and you also need to avoid an mm with
mm_usershitting zero concurrently with the last remote CPU using it lazily exiting lazy TLB. Perhaps this could be resolved by havingmm_count== 1 mean "mm_cpumask()is might contain bits and, if so, it owns the mm" andmm_count== 0 meaning "now it's dead" and using some careful cmpxchg ordec_returnto make sure that only one CPU frees it.Quoted 2 lines
Or maybe you'd need a lock or RCU for this, but the idea would be to only ever take the lock after
mm_usersgoes to zero.Quoted 7 lines
I don't think it's nonsense, it could be a good way to avoid IPIs.
I haven't seen much problem here that made me too concerned about IPIs yet, so I think the simple patch may be good enough to start with for powerpc. I'm looking at avoiding/reducing the IPIs by combining the unlazying with the exit TLB flush without doing anything fancy with ref counting, but we'll see.
Quoted 6 lines
I would be cautious with benchmarking here. I would expect that the nasty cases may affect power consumption more than performance — the specific issue is IPIs hitting idle cores, and the main effects are to slow down
exit()a bit but also to kick the idle core out of idle. Although, if the idle core is in a deep sleep, that IPI could be *very* slow.
It will tend to be self-limiting to some degree (deeper idle cores would tend to have less chance of IPI) but we have bigger issues on powerpc with that, like broadcast IPIs to the mm cpumask for THP management. Power hasn't really shown up as an issue but powerpc CPUs may have their own requirements and issues there, shall we say.
Quoted 1 line
So I think it’s worth at least giving this a try.
To be clear it's not a complete solution itself. The problem is of course that mm cpumask gives you false negatives, so the bits won't always clean up after themselves as CPUs switch away from their lazy tlb mms.
I would suspect it _may_ help with garbage collecting some remainders nicely after exit, but only with somewhat of a different accounting system than powerpc uses -- we tie mm_cpumask to TLB valids, so it can become spread over CPUs that don't (and even have never) used that mm as a lazy mm I don't know that the self-culling trick would help a great deal within that scheme.
So powerpc needs a bit more work on that side of things too, hence looking at doing more of this in the final TLB shootdown.
There's actually a lot of other things we can do as well to reduce IPIs, batching being a simple hammer, some kind of quiescing, testing the remote CPU to check what active mm it is using, doing the un-lazy at certain defined points etc, so I'm actually not worried about IPIs suddenly popping up and rendering the whole concept unworkable. At some point (unless we go something pretty complex like a SRCU type thing, or adding extra locking .e.g, to use_mm()), then at least sometimes an IPI will be required so I think it's reasonable to start here and introduce complexity more slowly if it's justified.
Thanks,
Nick
Nicholas Piggin
reviewerRe: [RFC PATCH 7/7] lazy tlb: shoot lazies, a non-refcounting lazy tlb option
Excerpts from Nicholas Piggin's message of July 14, 2020 3:04 pm:
Quoted 67 lines
Excerpts from Andy Lutomirski's message of July 14, 2020 4:18 am:
Quoted 53 lines
Quoted 45 lines · Nicholas Piggin
Excerpts from Andy Lutomirski's message of July 14, 2020 1:59 am:
Quoted 38 lines
Quoted 11 lines · Nicholas Piggin
On big systems, the mm refcount can become highly contented when doing a lot of context switching with threaded applications (particularly switching between the idle thread and an application thread).
Abandoning lazy tlb slows switching down quite a bit in the important
user->idle->usercases, so so instead implement a non-refcounted scheme that causes__mmdrop()to IPI all CPUs in themm_cpumaskand shoot down any remaining lazy ones.Quoted 5 lines
On a 16-socket 192-core POWER8 system, a context switching benchmark with as many software threads as CPUs (so each switch will go in and out of idle), upstream can achieve a rate of about 1 million context switches per second. After this patch it goes up to 118 million.
I read the patch a couple of times, and I have a suggestion that could be nonsense. You are, effectively, using
mm_cpumask()as a sort of refcount. You're saying "hey, this mm has no more references, but it still has nonemptymm_cpumask(), so let's send an IPI and shoot down those references too." I'm wondering whether you actually need the IPI. What if, instead, you actually treatedmm_cpumaskas a refcount for real? Roughly, in__mmdrop(), you would only free the page tables ifmm_cpumask()is empty. And, in the code that removes a CPU frommm_cpumask(), you would check ifmm_users== 0 and, if so, check if you just removed the last bit frommm_cpumaskand potentially free the mm.Getting the locking right here could be a bit tricky -- you need to avoid two CPUs simultaneously exiting lazy TLB and thinking they should free the mm, and you also need to avoid an mm with
mm_usershitting zero concurrently with the last remote CPU using it lazily exiting lazy TLB. Perhaps this could be resolved by havingmm_count== 1 mean "mm_cpumask()is might contain bits and, if so, it owns the mm" andmm_count== 0 meaning "now it's dead" and using some careful cmpxchg ordec_returnto make sure that only one CPU frees it.Quoted 2 lines
Or maybe you'd need a lock or RCU for this, but the idea would be to only ever take the lock after
mm_usersgoes to zero.Quoted 7 lines
I don't think it's nonsense, it could be a good way to avoid IPIs.
I haven't seen much problem here that made me too concerned about IPIs yet, so I think the simple patch may be good enough to start with for powerpc. I'm looking at avoiding/reducing the IPIs by combining the unlazying with the exit TLB flush without doing anything fancy with ref counting, but we'll see.
Quoted 6 lines
I would be cautious with benchmarking here. I would expect that the nasty cases may affect power consumption more than performance — the specific issue is IPIs hitting idle cores, and the main effects are to slow down
exit()a bit but also to kick the idle core out of idle. Although, if the idle core is in a deep sleep, that IPI could be *very* slow.It will tend to be self-limiting to some degree (deeper idle cores would tend to have less chance of IPI) but we have bigger issues on powerpc with that, like broadcast IPIs to the mm cpumask for THP management. Power hasn't really shown up as an issue but powerpc CPUs may have their own requirements and issues there, shall we say.
Quoted 6 lines
Quoted 1 line
So I think it’s worth at least giving this a try.
To be clear it's not a complete solution itself. The problem is of course that mm cpumask gives you false negatives, so the bits won't always clean up after themselves as CPUs switch away from their lazy tlb mms.
^^
False positives: CPU is in the mm_cpumask, but is not using the mm as a lazy tlb. So there can be bits left and never freed.
If you closed the false positives, you're back to a shared mm cache line on lazy mm context switches.
Thanks,
Nick
Re: [RFC PATCH 7/7] lazy tlb: shoot lazies, a non-refcounting lazy tlb option
Quoted 79 lines · Nicholas Piggin
Excerpts from Nicholas Piggin's message of July 14, 2020 3:04 pm:
Quoted 67 lines
Excerpts from Andy Lutomirski's message of July 14, 2020 4:18 am:
Quoted 53 lines
Quoted 45 lines · Nicholas Piggin
Excerpts from Andy Lutomirski's message of July 14, 2020 1:59 am:
Quoted 38 lines
Quoted 11 lines · Nicholas Piggin
On big systems, the mm refcount can become highly contented when doing a lot of context switching with threaded applications (particularly switching between the idle thread and an application thread).
Abandoning lazy tlb slows switching down quite a bit in the important
user->idle->usercases, so so instead implement a non-refcounted scheme that causes__mmdrop()to IPI all CPUs in themm_cpumaskand shoot down any remaining lazy ones.Quoted 5 lines
On a 16-socket 192-core POWER8 system, a context switching benchmark with as many software threads as CPUs (so each switch will go in and out of idle), upstream can achieve a rate of about 1 million context switches per second. After this patch it goes up to 118 million.
I read the patch a couple of times, and I have a suggestion that could be nonsense. You are, effectively, using
mm_cpumask()as a sort of refcount. You're saying "hey, this mm has no more references, but it still has nonemptymm_cpumask(), so let's send an IPI and shoot down those references too." I'm wondering whether you actually need the IPI. What if, instead, you actually treatedmm_cpumaskas a refcount for real? Roughly, in__mmdrop(), you would only free the page tables ifmm_cpumask()is empty. And, in the code that removes a CPU frommm_cpumask(), you would check ifmm_users== 0 and, if so, check if you just removed the last bit frommm_cpumaskand potentially free the mm.Getting the locking right here could be a bit tricky -- you need to avoid two CPUs simultaneously exiting lazy TLB and thinking they should free the mm, and you also need to avoid an mm with
mm_usershitting zero concurrently with the last remote CPU using it lazily exiting lazy TLB. Perhaps this could be resolved by havingmm_count== 1 mean "mm_cpumask()is might contain bits and, if so, it owns the mm" andmm_count== 0 meaning "now it's dead" and using some careful cmpxchg ordec_returnto make sure that only one CPU frees it.Quoted 2 lines
Or maybe you'd need a lock or RCU for this, but the idea would be to only ever take the lock after
mm_usersgoes to zero.Quoted 7 lines
I don't think it's nonsense, it could be a good way to avoid IPIs.
I haven't seen much problem here that made me too concerned about IPIs yet, so I think the simple patch may be good enough to start with for powerpc. I'm looking at avoiding/reducing the IPIs by combining the unlazying with the exit TLB flush without doing anything fancy with ref counting, but we'll see.
Quoted 6 lines
I would be cautious with benchmarking here. I would expect that the nasty cases may affect power consumption more than performance — the specific issue is IPIs hitting idle cores, and the main effects are to slow down
exit()a bit but also to kick the idle core out of idle. Although, if the idle core is in a deep sleep, that IPI could be *very* slow.It will tend to be self-limiting to some degree (deeper idle cores would tend to have less chance of IPI) but we have bigger issues on powerpc with that, like broadcast IPIs to the mm cpumask for THP management. Power hasn't really shown up as an issue but powerpc CPUs may have their own requirements and issues there, shall we say.
Quoted 6 lines
Quoted 1 line
So I think it’s worth at least giving this a try.
To be clear it's not a complete solution itself. The problem is of course that mm cpumask gives you false negatives, so the bits won't always clean up after themselves as CPUs switch away from their lazy tlb mms.
Quoted 7 lines
^^
False positives: CPU is in the
mm_cpumask, but is not using the mm as a lazy tlb. So there can be bits left and never freed.If you closed the false positives, you're back to a shared mm cache line on lazy mm context switches.
x86 has this exact problem. At least no more than 64*8 CPUs share the cache line :)
Can your share your benchmark?
Quoted 3 lines
Thanks,
Nick
Peter Zijlstra
maintainerRe: [RFC PATCH 7/7] lazy tlb: shoot lazies, a non-refcounting lazy tlb option
Quoted 1 line · Andy Lutomirski
x86 has this exact problem. At least no more than 64*8 CPUs share the cache line :)
I've seen patches for a 'sparse' bitmap to solve related problems.
It's basically the same code, except it multiplies everything (size, bit-nr) by a constant to reduce the number of active bits per line.
This sadly doesn't take topology into account, but reducing contention is still good ofcourse.
Nicholas Piggin
reviewerRe: [RFC PATCH 7/7] lazy tlb: shoot lazies, a non-refcounting lazy tlb option
In reply to Andy LutomirskiExcerpts from Andy Lutomirski's message of July 14, 2020 10:46 pm:
Quoted 84 lines
Quoted 84 lines · Nicholas Piggin
Excerpts from Nicholas Piggin's message of July 14, 2020 3:04 pm:
Quoted 67 lines
Excerpts from Andy Lutomirski's message of July 14, 2020 4:18 am:
Quoted 53 lines
Quoted 45 lines · Nicholas Piggin
Excerpts from Andy Lutomirski's message of July 14, 2020 1:59 am:
Quoted 38 lines
Quoted 11 lines · Nicholas Piggin
On big systems, the mm refcount can become highly contented when doing a lot of context switching with threaded applications (particularly switching between the idle thread and an application thread).
Abandoning lazy tlb slows switching down quite a bit in the important
user->idle->usercases, so so instead implement a non-refcounted scheme that causes__mmdrop()to IPI all CPUs in themm_cpumaskand shoot down any remaining lazy ones.Quoted 5 lines
On a 16-socket 192-core POWER8 system, a context switching benchmark with as many software threads as CPUs (so each switch will go in and out of idle), upstream can achieve a rate of about 1 million context switches per second. After this patch it goes up to 118 million.
I read the patch a couple of times, and I have a suggestion that could be nonsense. You are, effectively, using
mm_cpumask()as a sort of refcount. You're saying "hey, this mm has no more references, but it still has nonemptymm_cpumask(), so let's send an IPI and shoot down those references too." I'm wondering whether you actually need the IPI. What if, instead, you actually treatedmm_cpumaskas a refcount for real? Roughly, in__mmdrop(), you would only free the page tables ifmm_cpumask()is empty. And, in the code that removes a CPU frommm_cpumask(), you would check ifmm_users== 0 and, if so, check if you just removed the last bit frommm_cpumaskand potentially free the mm.Getting the locking right here could be a bit tricky -- you need to avoid two CPUs simultaneously exiting lazy TLB and thinking they should free the mm, and you also need to avoid an mm with
mm_usershitting zero concurrently with the last remote CPU using it lazily exiting lazy TLB. Perhaps this could be resolved by havingmm_count== 1 mean "mm_cpumask()is might contain bits and, if so, it owns the mm" andmm_count== 0 meaning "now it's dead" and using some careful cmpxchg ordec_returnto make sure that only one CPU frees it.Quoted 2 lines
Or maybe you'd need a lock or RCU for this, but the idea would be to only ever take the lock after
mm_usersgoes to zero.Quoted 7 lines
I don't think it's nonsense, it could be a good way to avoid IPIs.
I haven't seen much problem here that made me too concerned about IPIs yet, so I think the simple patch may be good enough to start with for powerpc. I'm looking at avoiding/reducing the IPIs by combining the unlazying with the exit TLB flush without doing anything fancy with ref counting, but we'll see.
Quoted 6 lines
I would be cautious with benchmarking here. I would expect that the nasty cases may affect power consumption more than performance — the specific issue is IPIs hitting idle cores, and the main effects are to slow down
exit()a bit but also to kick the idle core out of idle. Although, if the idle core is in a deep sleep, that IPI could be *very* slow.It will tend to be self-limiting to some degree (deeper idle cores would tend to have less chance of IPI) but we have bigger issues on powerpc with that, like broadcast IPIs to the mm cpumask for THP management. Power hasn't really shown up as an issue but powerpc CPUs may have their own requirements and issues there, shall we say.
Quoted 6 lines
Quoted 1 line
So I think it’s worth at least giving this a try.
To be clear it's not a complete solution itself. The problem is of course that mm cpumask gives you false negatives, so the bits won't always clean up after themselves as CPUs switch away from their lazy tlb mms.
^^
False positives: CPU is in the
mm_cpumask, but is not using the mm as a lazy tlb. So there can be bits left and never freed.
Quoted 6 lines
Quoted 2 lines
If you closed the false positives, you're back to a shared mm cache line on lazy mm context switches.
x86 has this exact problem. At least no more than 64*8 CPUs share the cache line :)
Can your share your benchmark?
It's just context_switch1_threads from will-it-scale, running with 1/2 the number of CPUs, and core affinity with an SMT processor (so each thread from the switching pairs gets spread to their own CPU and so you get the task->idle->task switching.
It's really just about the worst case, so I wouldn't say it's something to panic about.
Thanks,
Nick
Nicholas Piggin
reviewerRe: [RFC PATCH 7/7] lazy tlb: shoot lazies, a non-refcounting lazy tlb option
In reply to Andy LutomirskiExcerpts from Andy Lutomirski's message of July 14, 2020 10:46 pm:
Quoted 84 lines
Quoted 84 lines · Nicholas Piggin
Excerpts from Nicholas Piggin's message of July 14, 2020 3:04 pm:
Quoted 67 lines
Excerpts from Andy Lutomirski's message of July 14, 2020 4:18 am:
Quoted 53 lines
Quoted 45 lines · Nicholas Piggin
Excerpts from Andy Lutomirski's message of July 14, 2020 1:59 am:
Quoted 38 lines
Quoted 11 lines · Nicholas Piggin
On big systems, the mm refcount can become highly contented when doing a lot of context switching with threaded applications (particularly switching between the idle thread and an application thread).
Abandoning lazy tlb slows switching down quite a bit in the important
user->idle->usercases, so so instead implement a non-refcounted scheme that causes__mmdrop()to IPI all CPUs in themm_cpumaskand shoot down any remaining lazy ones.Quoted 5 lines
On a 16-socket 192-core POWER8 system, a context switching benchmark with as many software threads as CPUs (so each switch will go in and out of idle), upstream can achieve a rate of about 1 million context switches per second. After this patch it goes up to 118 million.
I read the patch a couple of times, and I have a suggestion that could be nonsense. You are, effectively, using
mm_cpumask()as a sort of refcount. You're saying "hey, this mm has no more references, but it still has nonemptymm_cpumask(), so let's send an IPI and shoot down those references too." I'm wondering whether you actually need the IPI. What if, instead, you actually treatedmm_cpumaskas a refcount for real? Roughly, in__mmdrop(), you would only free the page tables ifmm_cpumask()is empty. And, in the code that removes a CPU frommm_cpumask(), you would check ifmm_users== 0 and, if so, check if you just removed the last bit frommm_cpumaskand potentially free the mm.Getting the locking right here could be a bit tricky -- you need to avoid two CPUs simultaneously exiting lazy TLB and thinking they should free the mm, and you also need to avoid an mm with
mm_usershitting zero concurrently with the last remote CPU using it lazily exiting lazy TLB. Perhaps this could be resolved by havingmm_count== 1 mean "mm_cpumask()is might contain bits and, if so, it owns the mm" andmm_count== 0 meaning "now it's dead" and using some careful cmpxchg ordec_returnto make sure that only one CPU frees it.Quoted 2 lines
Or maybe you'd need a lock or RCU for this, but the idea would be to only ever take the lock after
mm_usersgoes to zero.Quoted 7 lines
I don't think it's nonsense, it could be a good way to avoid IPIs.
I haven't seen much problem here that made me too concerned about IPIs yet, so I think the simple patch may be good enough to start with for powerpc. I'm looking at avoiding/reducing the IPIs by combining the unlazying with the exit TLB flush without doing anything fancy with ref counting, but we'll see.
Quoted 6 lines
I would be cautious with benchmarking here. I would expect that the nasty cases may affect power consumption more than performance — the specific issue is IPIs hitting idle cores, and the main effects are to slow down
exit()a bit but also to kick the idle core out of idle. Although, if the idle core is in a deep sleep, that IPI could be *very* slow.It will tend to be self-limiting to some degree (deeper idle cores would tend to have less chance of IPI) but we have bigger issues on powerpc with that, like broadcast IPIs to the mm cpumask for THP management. Power hasn't really shown up as an issue but powerpc CPUs may have their own requirements and issues there, shall we say.
Quoted 6 lines
Quoted 1 line
So I think it’s worth at least giving this a try.
To be clear it's not a complete solution itself. The problem is of course that mm cpumask gives you false negatives, so the bits won't always clean up after themselves as CPUs switch away from their lazy tlb mms.
^^
False positives: CPU is in the
mm_cpumask, but is not using the mm as a lazy tlb. So there can be bits left and never freed.
Quoted 6 lines
Quoted 2 lines
If you closed the false positives, you're back to a shared mm cache line on lazy mm context switches.
x86 has this exact problem. At least no more than 64*8 CPUs share the cache line :)
Can your share your benchmark?
Just testing the IPI rates (on a smaller 176 CPU system), on a kernel compile, it causes about 300 shootdown interrupts (not 300 broadcasts but total interrupts).
And very short lived fork;exec;exit things like typical scripting commands doesn't typically generate any.
So yeah the really high exit rate things self-limit pretty well.
I documented the concern and added a few of the possible ways to further reduce IPIs in the comments though.
Thanks,
Nick
Nicholas Piggin
reviewerRe: [RFC PATCH 4/7] x86: use exit_lazy_tlb rather than membarrier_mm_sync_core_before_usermode
In reply to Mathieu DesnoyersExcerpts from Mathieu Desnoyers's message of July 14, 2020 12:13 am:
Quoted 16 lines
Quoted 7 lines · ----- On Jul 13, 2020, at 9:47 AM, Nicholas Piggin [email protected] wrote:
Excerpts from Nicholas Piggin's message of July 13, 2020 2:45 pm:
Quoted 6 lines
Excerpts from Andy Lutomirski's message of July 11, 2020 3:04 am:
Quoted 4 lines
Also, as it stands, I can easily see
in_irq()ceasing to promise to serialize. There are older kernels for which it does not promise to serialize. And I have plans to make it stop serializing in the nearish future.Quoted 6 lines
Quoted 2 lines
You mean x86's return from interrupt? Sounds fun... you'll konw where to update the membarrier sync code, at least :)
Oh, I should actually say Mathieu recently clarified a return from interrupt doesn't fundamentally need to serialize in order to support membarrier sync core.
Quoted 6 lines
Clarification to your statement:
Return from interrupt to kernel code does not need to be context serializing as long as kernel serializes before returning to user-space.
However, return from interrupt to user-space needs to be context serializing.
Hmm, I'm not sure it's enough even with the sync in the exit_lazy_tlb in the right places.
A kernel thread does a use_mm, then it blocks and the user process with the same mm runs on that CPU, and then it calls into the kernel, blocks, the kernel thread runs again, another CPU issues a membarrier which does not IPI this one because it's running a kthread, and then the kthread switches back to the user process (still without having unused the mm), and then the user process returns from syscall without having done a core synchronising instruction.
The cause of the problem is you want to avoid IPI'ing kthreads. Why? I'm guessing it really only matters as an optimisation in case of idle threads. Idle thread is easy (well, easier) because it won't use_mm, so you could check for rq->curr == rq->idle in your loop (in a suitable sched accessor function).
But... I'm not really liking this subtlety in the scheduler for all this (the scheduler still needs the barriers when switching out of idle).
Can it be improved somehow? Let me forget x86 core sync problem for now (that _may_ be a bit harder), and step back and look at what we're doing. The memory barrier case would actually suffer from the same problem as core sync, because in the same situation it has no implicit mmdrop in the scheduler switch code either.
So what are we doing with membarrier? We want any activity caused by the set of CPUs/threads specified that can be observed by this thread before calling membarrier is appropriately fenced from activity that can be observed to happen after the call returns.
CPU0 CPU1
1. user stuff
a. membarrier() 2. enter kernel b. read rq->curr 3. rq->curr switched to kthread c. is kthread, skip IPI 4. switch_to kthread d. return to user 5. rq->curr switched to user thread
6. switch_to user thread
7. exit kernel
8. more user stuff
As far as I can see, the problem is CPU1 might reorder step 5 and step 8, so you have mmdrop of lazy mm be a mb after step 6.
But why? The membarrier call only cares that there is a full barrier between 1 and 8, right? Which it will get from the previous context switch to the kthread.
I must say the memory barrier comments in membarrier could be improved a bit (unless I'm missing where the main comment is). It's fine to know what barriers pair with one another, but we need to know which exact memory accesses it is ordering
/*
* Matches memory barriers around rq->curr modification in
* scheduler.
*/
Sure, but it doesn't say what else is being ordered. I think it's just the user memory accesses, but would be nice to make that a bit more explicit. If we had such comments then we might know this case is safe.
I think the funny powerpc barrier is a similar case of this. If we ever see remote_rq->curr->flags & PF_KTHREAD, then we _know_ that CPU has or will have issued a memory barrier between running user code.
So AFAIKS all this membarrier stuff in kernel/sched/core.c could just go away. Except x86 because thread switch doesn't imply core sync, so CPU1 between 1 and 8 may never issue a core sync instruction the same way a context switch must be a full mb.
Before getting to x86 -- Am I right, or way off track here?
Thanks,
Nick
Nicholas Piggin
reviewerRe: [RFC PATCH 4/7] x86: use exit_lazy_tlb rather than membarrier_mm_sync_core_before_usermode
Excerpts from Nicholas Piggin's message of July 16, 2020 2:15 pm:
Quoted 65 lines
Excerpts from Mathieu Desnoyers's message of July 14, 2020 12:13 am:
Quoted 16 lines
Quoted 7 lines · ----- On Jul 13, 2020, at 9:47 AM, Nicholas Piggin [email protected] wrote:
Excerpts from Nicholas Piggin's message of July 13, 2020 2:45 pm:
Quoted 6 lines
Excerpts from Andy Lutomirski's message of July 11, 2020 3:04 am:
Quoted 4 lines
Also, as it stands, I can easily see
in_irq()ceasing to promise to serialize. There are older kernels for which it does not promise to serialize. And I have plans to make it stop serializing in the nearish future.Quoted 6 lines
Quoted 2 lines
You mean x86's return from interrupt? Sounds fun... you'll konw where to update the membarrier sync code, at least :)
Oh, I should actually say Mathieu recently clarified a return from interrupt doesn't fundamentally need to serialize in order to support membarrier sync core.
Quoted 6 lines
Clarification to your statement:
Return from interrupt to kernel code does not need to be context serializing as long as kernel serializes before returning to user-space.
However, return from interrupt to user-space needs to be context serializing.
Hmm, I'm not sure it's enough even with the sync in the
exit_lazy_tlbin the right places.A kernel thread does a
use_mm, then it blocks and the user process with the same mm runs on that CPU, and then it calls into the kernel, blocks, the kernel thread runs again, another CPU issues a membarrier which does not IPI this one because it's running a kthread, and then the kthread switches back to the user process (still without having unused the mm), and then the user process returns from syscall without having done a core synchronising instruction.The cause of the problem is you want to avoid IPI'ing kthreads. Why? I'm guessing it really only matters as an optimisation in case of idle threads. Idle thread is easy (well, easier) because it won't
use_mm, so you could check forrq->curr==rq->idlein your loop (in a suitable sched accessor function).But... I'm not really liking this subtlety in the scheduler for all this (the scheduler still needs the barriers when switching out of idle).
Can it be improved somehow? Let me forget x86 core sync problem for now (that _may_ be a bit harder), and step back and look at what we're doing. The memory barrier case would actually suffer from the same problem as core sync, because in the same situation it has no implicit mmdrop in the scheduler switch code either.
So what are we doing with membarrier? We want any activity caused by the set of CPUs/threads specified that can be observed by this thread before calling membarrier is appropriately fenced from activity that can be observed to happen after the call returns.
CPU0 CPU1
1. user stuffa. membarrier() 2. enter kernel b. read rq->curr 3. rq->curr switched to kthread c. is kthread, skip IPI 4. switch_to kthread d. return to user 5. rq->curr switched to user thread6.
switch_touser thread
7. exit kernel
8. more user stuff
Quoted 6 lines
As far as I can see, the problem is CPU1 might reorder step 5 and step 8, so you have mmdrop of lazy mm be a mb after step 6.
But why? The membarrier call only cares that there is a full barrier between 1 and 8, right? Which it will get from the previous context switch to the kthread.
I should be more complete here, especially since I was complaining about unclear barrier comment :)
CPU0 CPU1 a. user stuff 1. user stuff b. membarrier() 2. enter kernel c. smp_mb() 3. smp_mb__after_spinlock(); // in __schedule d. read rq->curr 4. rq->curr switched to kthread e. is kthread, skip IPI 5. switch_to kthread f. return to user 6. rq->curr switched to user thread g. user stuff 7. switch_to user thread
8. exit kernel
9. more user stuff
What you're really ordering is a, g vs 1, 9 right?
In other words, 9 must see a if it sees g, g must see 1 if it saw 9, etc.
Userspace does not care where the barriers are exactly or what kernel memory accesses might be being ordered by them, so long as there is a mb somewhere between a and g, and 1 and 9. Right?
Re: [RFC PATCH 4/7] x86: use exit_lazy_tlb rather than membarrier_mm_sync_core_before_usermode
In reply to Nicholas PigginQuoted 95 lines · Nicholas Piggin
Excerpts from Mathieu Desnoyers's message of July 14, 2020 12:13 am:
Quoted 16 lines
Quoted 7 lines · ----- On Jul 13, 2020, at 9:47 AM, Nicholas Piggin [email protected] wrote:
Excerpts from Nicholas Piggin's message of July 13, 2020 2:45 pm:
Quoted 6 lines
Excerpts from Andy Lutomirski's message of July 11, 2020 3:04 am:
Quoted 4 lines
Also, as it stands, I can easily see
in_irq()ceasing to promise to serialize. There are older kernels for which it does not promise to serialize. And I have plans to make it stop serializing in the nearish future.Quoted 6 lines
Quoted 2 lines
You mean x86's return from interrupt? Sounds fun... you'll konw where to update the membarrier sync code, at least :)
Oh, I should actually say Mathieu recently clarified a return from interrupt doesn't fundamentally need to serialize in order to support membarrier sync core.
Quoted 6 lines
Clarification to your statement:
Return from interrupt to kernel code does not need to be context serializing as long as kernel serializes before returning to user-space.
However, return from interrupt to user-space needs to be context serializing.
Hmm, I'm not sure it's enough even with the sync in the
exit_lazy_tlbin the right places.A kernel thread does a
use_mm, then it blocks and the user process with the same mm runs on that CPU, and then it calls into the kernel, blocks, the kernel thread runs again, another CPU issues a membarrier which does not IPI this one because it's running a kthread, and then the kthread switches back to the user process (still without having unused the mm), and then the user process returns from syscall without having done a core synchronising instruction.The cause of the problem is you want to avoid IPI'ing kthreads. Why? I'm guessing it really only matters as an optimisation in case of idle threads. Idle thread is easy (well, easier) because it won't
use_mm, so you could check forrq->curr==rq->idlein your loop (in a suitable sched accessor function).But... I'm not really liking this subtlety in the scheduler for all this (the scheduler still needs the barriers when switching out of idle).
Can it be improved somehow? Let me forget x86 core sync problem for now (that _may_ be a bit harder), and step back and look at what we're doing. The memory barrier case would actually suffer from the same problem as core sync, because in the same situation it has no implicit mmdrop in the scheduler switch code either.
So what are we doing with membarrier? We want any activity caused by the set of CPUs/threads specified that can be observed by this thread before calling membarrier is appropriately fenced from activity that can be observed to happen after the call returns.
CPU0 CPU1
1. user stuffa. membarrier() 2. enter kernel b. read rq->curr 3. rq->curr switched to kthread c. is kthread, skip IPI 4. switch_to kthread d. return to user 5. rq->curr switched to user thread6.
switch_touser thread
7. exit kernel
8. more user stuffAs far as I can see, the problem is CPU1 might reorder step 5 and step 8, so you have mmdrop of lazy mm be a mb after step 6.
But why? The membarrier call only cares that there is a full barrier between 1 and 8, right? Which it will get from the previous context switch to the kthread.
I must say the memory barrier comments in membarrier could be improved a bit (unless I'm missing where the main comment is). It's fine to know what barriers pair with one another, but we need to know which exact memory accesses it is ordering
/*
* Matches memory barriers aroundrq->currmodification in
* scheduler.
*/Sure, but it doesn't say what else is being ordered. I think it's just the user memory accesses, but would be nice to make that a bit more explicit. If we had such comments then we might know this case is safe.
I think the funny powerpc barrier is a similar case of this. If we ever see
remote_rq->curr->flags&PF_KTHREAD, then we _know_ that CPU has or will have issued a memory barrier between running user code.
Quoted 6 lines
So AFAIKS all this membarrier stuff in
kernel/sched/core.ccould just go away. Except x86 because thread switch doesn't imply core sync, so CPU1 between 1 and 8 may never issue a core sync instruction the same way a context switch must be a full mb.Before getting to x86 -- Am I right, or way off track here?
I find it hard to believe that this is x86 only. Why would thread switch imply core sync on any architecture? Is x86 unique in having a stupid expensive core sync that is heavier than smp_mb()?
But I’m wondering if all this deferred sync stuff is wrong. In the brave new world of io_uring and such, perhaps kernel access matter too. Heck, even:
int a[2];
Thread A:
a[0] = 1;
a[1] = 2:
Thread B:
write(fd, a, sizeof(a));
Doesn’t do what thread A is expecting. Admittedly this particular example is nonsense, but maybe there are sensible cases that matter to someone.
—Andy
Quoted 3 lines
Thanks,
Nick
Nicholas Piggin
reviewerRe: [RFC PATCH 4/7] x86: use exit_lazy_tlb rather than membarrier_mm_sync_core_before_usermode
Excerpts from Andy Lutomirski's message of July 16, 2020 3:18 pm:
Quoted 95 lines
Quoted 95 lines · Nicholas Piggin
Excerpts from Mathieu Desnoyers's message of July 14, 2020 12:13 am:
Quoted 16 lines
Quoted 7 lines · ----- On Jul 13, 2020, at 9:47 AM, Nicholas Piggin [email protected] wrote:
Excerpts from Nicholas Piggin's message of July 13, 2020 2:45 pm:
Quoted 6 lines
Excerpts from Andy Lutomirski's message of July 11, 2020 3:04 am:
Quoted 4 lines
Also, as it stands, I can easily see
in_irq()ceasing to promise to serialize. There are older kernels for which it does not promise to serialize. And I have plans to make it stop serializing in the nearish future.Quoted 6 lines
Quoted 2 lines
You mean x86's return from interrupt? Sounds fun... you'll konw where to update the membarrier sync code, at least :)
Oh, I should actually say Mathieu recently clarified a return from interrupt doesn't fundamentally need to serialize in order to support membarrier sync core.
Quoted 6 lines
Clarification to your statement:
Return from interrupt to kernel code does not need to be context serializing as long as kernel serializes before returning to user-space.
However, return from interrupt to user-space needs to be context serializing.
Hmm, I'm not sure it's enough even with the sync in the
exit_lazy_tlbin the right places.A kernel thread does a
use_mm, then it blocks and the user process with the same mm runs on that CPU, and then it calls into the kernel, blocks, the kernel thread runs again, another CPU issues a membarrier which does not IPI this one because it's running a kthread, and then the kthread switches back to the user process (still without having unused the mm), and then the user process returns from syscall without having done a core synchronising instruction.The cause of the problem is you want to avoid IPI'ing kthreads. Why? I'm guessing it really only matters as an optimisation in case of idle threads. Idle thread is easy (well, easier) because it won't
use_mm, so you could check forrq->curr==rq->idlein your loop (in a suitable sched accessor function).But... I'm not really liking this subtlety in the scheduler for all this (the scheduler still needs the barriers when switching out of idle).
Can it be improved somehow? Let me forget x86 core sync problem for now (that _may_ be a bit harder), and step back and look at what we're doing. The memory barrier case would actually suffer from the same problem as core sync, because in the same situation it has no implicit mmdrop in the scheduler switch code either.
So what are we doing with membarrier? We want any activity caused by the set of CPUs/threads specified that can be observed by this thread before calling membarrier is appropriately fenced from activity that can be observed to happen after the call returns.
CPU0 CPU1
1. user stuffa. membarrier() 2. enter kernel b. read rq->curr 3. rq->curr switched to kthread c. is kthread, skip IPI 4. switch_to kthread d. return to user 5. rq->curr switched to user thread6.
switch_touser thread
7. exit kernel
8. more user stuffAs far as I can see, the problem is CPU1 might reorder step 5 and step 8, so you have mmdrop of lazy mm be a mb after step 6.
But why? The membarrier call only cares that there is a full barrier between 1 and 8, right? Which it will get from the previous context switch to the kthread.
I must say the memory barrier comments in membarrier could be improved a bit (unless I'm missing where the main comment is). It's fine to know what barriers pair with one another, but we need to know which exact memory accesses it is ordering
/*
* Matches memory barriers aroundrq->currmodification in
* scheduler.
*/Sure, but it doesn't say what else is being ordered. I think it's just the user memory accesses, but would be nice to make that a bit more explicit. If we had such comments then we might know this case is safe.
I think the funny powerpc barrier is a similar case of this. If we ever see
remote_rq->curr->flags&PF_KTHREAD, then we _know_ that CPU has or will have issued a memory barrier between running user code.
Quoted 8 lines
Quoted 6 lines
So AFAIKS all this membarrier stuff in
kernel/sched/core.ccould just go away. Except x86 because thread switch doesn't imply core sync, so CPU1 between 1 and 8 may never issue a core sync instruction the same way a context switch must be a full mb.Before getting to x86 -- Am I right, or way off track here?
I find it hard to believe that this is x86 only. Why would thread switch imply core sync on any architecture? Is x86 unique in having a stupid expensive core sync that is heavier than
smp_mb()?
It's not the thread switch but the return from kernel to user -- at least of architectures that implement membarrier SYNC_CORE, x86 can do that without serializing.
The thread switch is muddying the waters a bit, it's not the actual thread switch we care about, that just happens to be used as a point where we try to catch the membarrier IPIs that were skipped due to the PF_KTHREAD optimisation.
I think that doing said check in the lazy tlb exit code is both unnecessary for the memory ordering and insufficient for pipeline serialization.
Quoted 8 lines
But I’m wondering if all this deferred sync stuff is wrong. In the brave new world of
io_uringand such, perhaps kernel access matter too. Heck, even:int a[2];
Thread A:
a[0] = 1;
a[1] = 2:
Quoted 5 lines
Thread B:
write(fd, a, sizeof(a));Doesn’t do what thread A is expecting. Admittedly this particular example is nonsense, but maybe there are sensible cases that matter to someone.
I think kernel accesses probably do matter (or at least they should by principle of least surprise). And so I was doubly misleading by labeling it as "user stuff". I should have distinguished between previous user or kernel accesses, as opposed to the kernel accesses specifically for the implementation of the membarrier call.
So I think the membarrier code gets *that* part right (modulo what we have seen already) if the kernel access is being done from process context.
But yes if the access is coming from io_uring that has done kthread_use_mm or some other random code running in a kernel thread working on get_user_pages memory or any similar shared vm_insert_pfn memory, then it goes completely to hell.
So good catch, PF_KTHREAD check is problematic there even if no actual users exist today. rq->curr == rq->idle test might be better, but can we have interrupts writing completions into user memory? For performance I would hope so, so that makes even that test problematic.
Maybe membarrier should close that gap entirely, and work around performance issue by adding _USER_ONLY flags which explicitly only order user mode accesess vs other user accesses.
Thanks,
Nick
Peter Zijlstra
maintainerRe: [RFC PATCH 4/7] x86: use exit_lazy_tlb rather than membarrier_mm_sync_core_before_usermode
In reply to Andy LutomirskiQuoted 12 lines · Andy Lutomirski
Quoted 11 lines · Nicholas Piggin
CPU0 CPU1
1. user stuffa. membarrier() 2. enter kernel b. read rq->curr 3. rq->curr switched to kthread c. is kthread, skip IPI 4. switch_to kthread d. return to user 5. rq->curr switched to user thread6.
switch_touser thread
7. exit kernel
8. more user stuff
Quoted 3 lines
I find it hard to believe that this is x86 only. Why would thread switch imply core sync on any architecture? Is x86 unique in having a stupid expensive core sync that is heavier than
smp_mb()?
smp_mb() is nowhere near the most expensive barrier we have in Linux, mb() might qualify, since that has some completion requirements since it needs to serialize against external actors.
On x86_64 things are rather murky, we have:
LOCK prefix -- which implies smp_mb() before and after RmW LFENCE -- which used to be rmb like, until Spectre, and now it
is ISYNC like. Since ISYNC ensures an empty pipeline, it also implies all loads are retired (and therefore complete) it implies rmb.
MFENCE -- which is a memop completion barrier like, it makes
sure all previously issued memops are complete.
if you read that carefully, you'll note you'll have to use LFENCE + MFENCE to order against non-memops instructions.
But none of them imply dumping the instruction decoder caches, that only happens on core serializing instructions like CR3 writes, IRET, CPUID and a few others, I think we recently got a SERIALIZE instruction to add to this list.
On ARM64 there's something a whole different set of barriers, and again smp_mb() isn't nowhere near the top of the list. They have roughly 3 classes:
ISB -- instruction sync barrier
DMB(x) -- memory ordering in domain x
DSB(x) -- memory completion in domain x
And they have at least 3 domains (IIRC), system, outer, inner.
The ARM64 __switch_to() includes a dsb(sy), just like PowerPC used to have a SYNC, but since PowerPC is rare for only having one rediculously heavy serializing instruction, we got to re-use the smp_mb() early in __schedule() instead, but ARM64 can't do that.
So rather than say that x86 is special here, I'd say that PowerPC is special here.
Quoted 3 lines
But I’m wondering if all this deferred sync stuff is wrong. In the brave new world of
io_uringand such, perhaps kernel access matter too. Heck, even:
IIRC the membarrier SYNC_CORE use-case is about user-space self-modifying code.
Userspace re-uses a text address and needs to SYNC_CORE before it can be sure the old text is forgotten. Nothing the kernel does matters there.
I suppose the manpage could be more clear there.
Nicholas Piggin
reviewerRe: [RFC PATCH 4/7] x86: use exit_lazy_tlb rather than membarrier_mm_sync_core_before_usermode
Excerpts from Peter Zijlstra's message of July 16, 2020 6:50 pm:
Quoted 50 lines · Andy Lutomirski
Quoted 12 lines
Quoted 11 lines · Nicholas Piggin
CPU0 CPU1
1. user stuffa. membarrier() 2. enter kernel b. read rq->curr 3. rq->curr switched to kthread c. is kthread, skip IPI 4. switch_to kthread d. return to user 5. rq->curr switched to user thread6.
switch_touser thread
7. exit kernel
8. more user stuffQuoted 3 lines
I find it hard to believe that this is x86 only. Why would thread switch imply core sync on any architecture? Is x86 unique in having a stupid expensive core sync that is heavier than
smp_mb()?
smp_mb()is nowhere near the most expensive barrier we have in Linux,mb()might qualify, since that has some completion requirements since it needs to serialize against external actors.On
x86_64things are rather murky, we have:LOCK prefix -- which implies
smp_mb()before and after RmW LFENCE -- which used to be rmb like, until Spectre, and now it
is ISYNC like. Since ISYNC ensures an empty pipeline, it also implies all loads are retired (and therefore complete) it implies rmb.
MFENCE -- which is a memop completion barrier like, it makes
sure all previously issued memops are complete.if you read that carefully, you'll note you'll have to use LFENCE + MFENCE to order against non-memops instructions.
But none of them imply dumping the instruction decoder caches, that only happens on core serializing instructions like CR3 writes, IRET, CPUID and a few others, I think we recently got a SERIALIZE instruction to add to this list.
On ARM64 there's something a whole different set of barriers, and again
smp_mb()isn't nowhere near the top of the list. They have roughly 3 classes:ISB -- instruction sync barrier
DMB(x) -- memory ordering in domain x
DSB(x) -- memory completion in domain xAnd they have at least 3 domains (IIRC), system, outer, inner.
Quoted 8 lines
The ARM64
__switch_to()includes a dsb(sy), just like PowerPC used to have a SYNC, but since PowerPC is rare for only having one rediculously heavy serializing instruction, we got to re-use thesmp_mb()early in__schedule()instead, but ARM64 can't do that.So rather than say that x86 is special here, I'd say that PowerPC is special here.
PowerPC is "special", I'll agree with you there :)
It does have a SYNC (HWSYNC) instruction that is mb(). It does not serialize the core.
ISYNC is a nop. ICBI ; ISYNC does serialize the core.
Difference between them is probably much the same as difference between MFENCE and CPUID on x86 CPUs. Serializing the core is almost always pretty expensive. HWSYNC/MFENCE can be expensive if you have a lot of or difficult (not exclusive in cache) outstanding with critical reads after the barrier, but it can also be somewhat cheap if there are few writes, and executed past, it only needs to hold up subsequent reads.
That said... implementation details. powerpc CPUs have traditionally had fairly costly HWSYNC.
Quoted 11 lines
Quoted 3 lines
But I’m wondering if all this deferred sync stuff is wrong. In the brave new world of
io_uringand such, perhaps kernel access matter too. Heck, even:IIRC the membarrier
SYNC_COREuse-case is about user-space self-modifying code.Userspace re-uses a text address and needs to
SYNC_COREbefore it can be sure the old text is forgotten. Nothing the kernel does matters there.I suppose the manpage could be more clear there.
True, but memory ordering of kernel stores from kernel threads for regular mem barrier is the concern here.
Does io_uring update completion queue from kernel thread or interrupt, for example? If it does, then membarrier will not order such stores with user memory accesses.
Thanks,
Nick
[email protected]
maintainerRe: [RFC PATCH 4/7] x86: use exit_lazy_tlb rather than membarrier_mm_sync_core_before_usermode
Quoted 14 lines · Nicholas Piggin
Excerpts from Peter Zijlstra's message of July 16, 2020 6:50 pm:
Quoted 13 lines · Andy Lutomirski
Quoted 5 lines
Quoted 1 line · Nicholas Piggin
But I’m wondering if all this deferred sync stuff is wrong. In the brave new world of
io_uringand such, perhaps kernel access matter too. Heck, even:IIRC the membarrier
SYNC_COREuse-case is about user-space self-modifying code.Userspace re-uses a text address and needs to
SYNC_COREbefore it can be sure the old text is forgotten. Nothing the kernel does matters there.
Quoted 8 lines
Quoted 1 line
I suppose the manpage could be more clear there.
True, but memory ordering of kernel stores from kernel threads for regular mem barrier is the concern here.
Does
io_uringupdate completion queue from kernel thread or interrupt, for example? If it does, then membarrier will not order such stores with user memory accesses.
So we're talking about regular membarrier() then? Not the SYNC_CORE variant per-se.
Even there, I'll argue we don't care, but perhaps Mathieu has a different opinion. All we care about is that all other threads (or CPUs for GLOBAL) observe an smp_mb() before it returns.
Any serialization against whatever those other threads/CPUs are running at the instant of the syscall is external to the syscall, we make no gauarantees about that. That is, we can fundamentally not say what another CPU is executing concurrently. Nor should we want to.
So if you feel that your membarrier() ought to serialize against remote execution, you need to arrange a quiecent state on the remote side yourself.
Now, normally membarrier() is used to implement userspace RCU like things, and there all that matters is that the remote CPUs observe the beginngin of the new grace-period, ie counter flip, and we observe their read-side critical sections, or smething like that, it's been a while since I looked at all that.
It's always been the case that concurrent syscalls could change user memory, io_uring doesn't change that, it just makes it even less well defined when that would happen. If you want to serialize against that, you need to arrange that externally.
Mathieu Desnoyers
maintainerRe: [RFC PATCH 4/7] x86: use exit_lazy_tlb rather than membarrier_mm_sync_core_before_usermode
Quoted 24 lines · ----- On Jul 16, 2020, at 7:00 AM, Peter Zijlstra [email protected] wrote:
Quoted 14 lines · Nicholas Piggin
Excerpts from Peter Zijlstra's message of July 16, 2020 6:50 pm:
Quoted 13 lines · Andy Lutomirski
Quoted 5 lines
Quoted 1 line · Nicholas Piggin
But I’m wondering if all this deferred sync stuff is wrong. In the brave new world of
io_uringand such, perhaps kernel access matter too. Heck, even:IIRC the membarrier
SYNC_COREuse-case is about user-space self-modifying code.Userspace re-uses a text address and needs to
SYNC_COREbefore it can be sure the old text is forgotten. Nothing the kernel does matters there.Quoted 8 lines
Quoted 1 line
I suppose the manpage could be more clear there.
True, but memory ordering of kernel stores from kernel threads for regular mem barrier is the concern here.
Does
io_uringupdate completion queue from kernel thread or interrupt, for example? If it does, then membarrier will not order such stores with user memory accesses.
Quoted 5 lines
So we're talking about regular
membarrier()then? Not theSYNC_COREvariant per-se.Even there, I'll argue we don't care, but perhaps Mathieu has a different opinion.
I agree with Peter that we don't care about accesses to user-space memory performed concurrently with membarrier.
What we'd care about in terms of accesses to user-space memory from the kernel is something that would be clearly ordered as happening before or after the membarrier call, for instance a read(2) followed by membarrier(2) after the read returns, or a read(2) issued after return from membarrier(2). The other scenario we'd care about is with the compiler barrier paired with membarrier: e.g. read(2) returns, compiler barrier, followed by a store. Or load, compiler barrier, followed by write(2).
All those scenarios imply before/after ordering wrt either membarrier or the compiler barrier. I notice that io_uring has a "completion" queue. Let's try to come up with realistic usage scenarios.
So the dependency chain would be provided by e.g.:
* Infrequent read / Frequent write, communicating read completion through variable X
wait for io_uring read request completion -> membarrier -> store X=1
with matching
load from X (waiting for X==1) -> asm volatile (::: "memory") -> submit io_uring write request
or this other scenario:
* Frequent read / Infrequent write, communicating read completion through variable X
load from X (waiting for X==1) -> membarrier -> submit io_uring write request
with matching
wait for io_uring read request completion -> asm volatile (::: "memory") -> store X=1
Thanks,
Mathieu
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com
Mathieu Desnoyers
maintainerRe: [RFC PATCH 4/7] x86: use exit_lazy_tlb rather than membarrier_mm_sync_core_before_usermode
In reply to Nicholas PigginQuoted 15 lines · ----- On Jul 16, 2020, at 12:42 AM, Nicholas Piggin [email protected] wrote:
I should be more complete here, especially since I was complaining about unclear barrier comment :)
CPU0 CPU1 a. user stuff 1. user stuff b. membarrier() 2. enter kernel c. smp_mb() 3. smp_mb__after_spinlock(); // in __schedule d. read rq->curr 4. rq->curr switched to kthread e. is kthread, skip IPI 5. switch_to kthread f. return to user 6. rq->curr switched to user thread g. user stuff 7. switch_to user thread8. exit kernel
9. more user stuff
Quoted 8 lines
What you're really ordering is a, g vs 1, 9 right?
In other words, 9 must see a if it sees g, g must see 1 if it saw 9, etc.
Userspace does not care where the barriers are exactly or what kernel memory accesses might be being ordered by them, so long as there is a mb somewhere between a and g, and 1 and 9. Right?
This is correct. Note that the accesses to user-space memory can be done either by user-space code or kernel code, it doesn't matter. However, in order to be considered as happening before/after either membarrier or the matching compiler barrier, kernel code needs to have causality relationship with user-space execution, e.g. user-space does a system call, or returns from a system call.
In the case of io_uring, submitting a request or returning from waiting on request completion appear to provide this causality relationship.
Thanks,
Mathieu
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com
Mathieu Desnoyers
maintainerRe: [RFC PATCH 4/7] x86: use exit_lazy_tlb rather than membarrier_mm_sync_core_before_usermode
Quoted 18 lines · ----- On Jul 16, 2020, at 11:46 AM, Mathieu Desnoyers…
Quoted 17 lines · ----- On Jul 16, 2020, at 12:42 AM, Nicholas Piggin [email protected] wrote:
I should be more complete here, especially since I was complaining about unclear barrier comment :)
CPU0 CPU1 a. user stuff 1. user stuff b. membarrier() 2. enter kernel c. smp_mb() 3. smp_mb__after_spinlock(); // in __schedule d. read rq->curr 4. rq->curr switched to kthread e. is kthread, skip IPI 5. switch_to kthread f. return to user 6. rq->curr switched to user thread g. user stuff 7. switch_to user thread8. exit kernel
9. more user stuffWhat you're really ordering is a, g vs 1, 9 right?
Quoted 8 lines
Quoted 6 lines
In other words, 9 must see a if it sees g, g must see 1 if it saw 9, etc.
Userspace does not care where the barriers are exactly or what kernel memory accesses might be being ordered by them, so long as there is a mb somewhere between a and g, and 1 and 9. Right?
This is correct.
Actually, sorry, the above is not quite right. It's been a while since I looked into the details of membarrier.
The smp_mb() at the beginning of membarrier() needs to be paired with a smp_mb() _after_ rq->curr is switched back to the user thread, so the memory barrier is between store to rq->curr and following user-space accesses.
The smp_mb() at the end of membarrier() needs to be paired with the smp_mb__after_spinlock() at the beginning of schedule, which is between accesses to userspace memory and switching rq->curr to kthread.
As to *why* this ordering is needed, I'd have to dig through additional scenarios from https://lwn.net/Articles/573436/. Or maybe Paul remembers ?
Thanks,
Mathieu
Quoted 19 lines
Note that the accesses to user-space memory can be
done either by user-space code or kernel code, it doesn't matter. However, in order to be considered as happening before/after either membarrier or the matching compiler barrier, kernel code needs to have causality relationship with user-space execution, e.g. user-space does a system call, or returns from a system call.In the case of
io_uring, submitting a request or returning from waiting on request completion appear to provide this causality relationship.Thanks,
Mathieu
--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com
Mathieu Desnoyers
maintainerRe: [RFC PATCH 4/7] x86: use exit_lazy_tlb rather than membarrier_mm_sync_core_before_usermode
Quoted 38 lines · ----- On Jul 16, 2020, at 12:03 PM, Mathieu Desnoyers…
----- On Jul 16, 2020, at 11:46 AM, Mathieu Desnoyers
Quoted 18 lines · [email protected] wrote:
Quoted 17 lines · ----- On Jul 16, 2020, at 12:42 AM, Nicholas Piggin [email protected] wrote:
I should be more complete here, especially since I was complaining about unclear barrier comment :)
CPU0 CPU1 a. user stuff 1. user stuff b. membarrier() 2. enter kernel c. smp_mb() 3. smp_mb__after_spinlock(); // in __schedule d. read rq->curr 4. rq->curr switched to kthread e. is kthread, skip IPI 5. switch_to kthread f. return to user 6. rq->curr switched to user thread g. user stuff 7. switch_to user thread8. exit kernel
9. more user stuffWhat you're really ordering is a, g vs 1, 9 right?
Quoted 8 lines
Quoted 6 lines
In other words, 9 must see a if it sees g, g must see 1 if it saw 9, etc.
Userspace does not care where the barriers are exactly or what kernel memory accesses might be being ordered by them, so long as there is a mb somewhere between a and g, and 1 and 9. Right?
This is correct.
Actually, sorry, the above is not quite right. It's been a while since I looked into the details of membarrier.
The
smp_mb()at the beginning ofmembarrier()needs to be paired with asmp_mb()_after_rq->curris switched back to the user thread, so the memory barrier is between store torq->currand following user-space accesses.
Quoted 6 lines
The
smp_mb()at the end ofmembarrier()needs to be paired with thesmp_mb__after_spinlock()at the beginning of schedule, which is between accesses to userspace memory and switchingrq->currto kthread.As to *why* this ordering is needed, I'd have to dig through additional scenarios from https://lwn.net/Articles/573436/. Or maybe Paul remembers ?
Thinking further about this, I'm beginning to consider that maybe we have been overly cautious by requiring memory barriers before and after store to rq->curr.
If CPU0 observes a CPU1's rq->curr->mm which differs from its own process (current) while running the membarrier system call, it necessarily means that CPU1 had to issue smp_mb__after_spinlock when entering the scheduler, between any user-space loads/stores and update of rq->curr.
Requiring a memory barrier between update of rq->curr (back to current process's thread) and following user-space memory accesses does not seem to guarantee anything more than what the initial barrier at the beginning of __schedule already provides, because the guarantees are only about accesses to user-space memory.
Therefore, with the memory barrier at the beginning of __schedule, just observing that CPU1's rq->curr differs from current should guarantee that a memory barrier was issued between any sequentially consistent instructions belonging to the current process on CPU1.
Or am I missing/misremembering an important point here ?
Thanks,
Mathieu
Quoted 30 lines
Thanks,
Mathieu
Quoted 12 lines
Note that the accesses to user-space memory can be
done either by user-space code or kernel code, it doesn't matter. However, in order to be considered as happening before/after either membarrier or the matching compiler barrier, kernel code needs to have causality relationship with user-space execution, e.g. user-space does a system call, or returns from a system call.In the case of
io_uring, submitting a request or returning from waiting on request completion appear to provide this causality relationship.Thanks,
Quoted 7 lines
Mathieu
--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com
Re: [RFC PATCH 4/7] x86: use exit_lazy_tlb rather than membarrier_mm_sync_core_before_usermode
Quoted 60 lines · Mathieu Desnoyers
Quoted 38 lines · ----- On Jul 16, 2020, at 12:03 PM, Mathieu Desnoyers…
----- On Jul 16, 2020, at 11:46 AM, Mathieu Desnoyers
Quoted 18 lines · [email protected] wrote:
Quoted 17 lines · ----- On Jul 16, 2020, at 12:42 AM, Nicholas Piggin [email protected] wrote:
I should be more complete here, especially since I was complaining about unclear barrier comment :)
CPU0 CPU1 a. user stuff 1. user stuff b. membarrier() 2. enter kernel c. smp_mb() 3. smp_mb__after_spinlock(); // in __schedule d. read rq->curr 4. rq->curr switched to kthread e. is kthread, skip IPI 5. switch_to kthread f. return to user 6. rq->curr switched to user thread g. user stuff 7. switch_to user thread8. exit kernel
9. more user stuffWhat you're really ordering is a, g vs 1, 9 right?
Quoted 8 lines
Quoted 6 lines
In other words, 9 must see a if it sees g, g must see 1 if it saw 9, etc.
Userspace does not care where the barriers are exactly or what kernel memory accesses might be being ordered by them, so long as there is a mb somewhere between a and g, and 1 and 9. Right?
This is correct.
Actually, sorry, the above is not quite right. It's been a while since I looked into the details of membarrier.
The
smp_mb()at the beginning ofmembarrier()needs to be paired with asmp_mb()_after_rq->curris switched back to the user thread, so the memory barrier is between store torq->currand following user-space accesses.Quoted 6 lines
The
smp_mb()at the end ofmembarrier()needs to be paired with thesmp_mb__after_spinlock()at the beginning of schedule, which is between accesses to userspace memory and switchingrq->currto kthread.As to *why* this ordering is needed, I'd have to dig through additional scenarios from https://lwn.net/Articles/573436/. Or maybe Paul remembers ?
Thinking further about this, I'm beginning to consider that maybe we have been overly cautious by requiring memory barriers before and after store to
rq->curr.If CPU0 observes a CPU1's
rq->curr->mmwhich differs from its own process (current) while running the membarrier system call, it necessarily means that CPU1 had to issuesmp_mb__after_spinlockwhen entering the scheduler, between any user-space loads/stores and update ofrq->curr.Requiring a memory barrier between update of
rq->curr(back to current process's thread) and following user-space memory accesses does not seem to guarantee anything more than what the initial barrier at the beginning of__schedulealready provides, because the guarantees are only about accesses to user-space memory.
Quoted 6 lines
Therefore, with the memory barrier at the beginning of
__schedule, just observing that CPU1'srq->currdiffers from current should guarantee that a memory barrier was issued between any sequentially consistent instructions belonging to the current process on CPU1.Or am I missing/misremembering an important point here ?
Is it correct to say that the switch_to operations in 5 and 7 include memory barriers? If they do, then skipping the IPI should be okay.
The reason is as follows: The guarantee you need to enforce is that anything written by CPU0 before the membarrier() will be visible to CPU1 after it returns to user mode. Let's say that a writes to X and 9 reads from X.
Then we have an instance of the Store Buffer pattern:
CPU0 CPU1 a. Write X 6. Write rq->curr for user thread c. smp_mb() 7. switch_to memory barrier d. Read rq->curr 9. Read X
In this pattern, the memory barriers make it impossible for both reads to miss their corresponding writes. Since d does fail to read 6 (it sees the earlier value stored by 4), 9 must read a.
The other guarantee you need is that g on CPU0 will observe anything written by CPU1 in 1. This is easier to see, using the fact that 3 is a memory barrier and d reads from 4.
Alan Stern
Nicholas Piggin
reviewerRe: [RFC PATCH 4/7] x86: use exit_lazy_tlb rather than membarrier_mm_sync_core_before_usermode
In reply to [email protected]Excerpts from [email protected]'s message of July 16, 2020 9:00 pm:
Quoted 20 lines · Nicholas Piggin
Quoted 19 lines
Excerpts from Peter Zijlstra's message of July 16, 2020 6:50 pm:
Quoted 7 lines · Andy Lutomirski
Quoted 5 lines
Quoted 1 line · Nicholas Piggin
But I’m wondering if all this deferred sync stuff is wrong. In the brave new world of
io_uringand such, perhaps kernel access matter too. Heck, even:Quoted 7 lines
IIRC the membarrier
SYNC_COREuse-case is about user-space self-modifying code.Userspace re-uses a text address and needs to
SYNC_COREbefore it can be sure the old text is forgotten. Nothing the kernel does matters there.I suppose the manpage could be more clear there.
True, but memory ordering of kernel stores from kernel threads for regular mem barrier is the concern here.
Quoted 6 lines
Quoted 3 lines
Does
io_uringupdate completion queue from kernel thread or interrupt, for example? If it does, then membarrier will not order such stores with user memory accesses.So we're talking about regular
membarrier()then? Not theSYNC_COREvariant per-se.
Well, both but Andy in this case was wondering about kernel writes vs user.
Quoted 20 lines
Even there, I'll argue we don't care, but perhaps Mathieu has a different opinion. All we care about is that all other threads (or CPUs for GLOBAL) observe an
smp_mb()before it returns.Any serialization against whatever those other threads/CPUs are running at the instant of the syscall is external to the syscall, we make no gauarantees about that. That is, we can fundamentally not say what another CPU is executing concurrently. Nor should we want to.
So if you feel that your
membarrier()ought to serialize against remote execution, you need to arrange a quiecent state on the remote side yourself.Now, normally
membarrier()is used to implement userspace RCU like things, and there all that matters is that the remote CPUs observe the beginngin of the new grace-period, ie counter flip, and we observe their read-side critical sections, or smething like that, it's been a while since I looked at all that.
Quoted 4 lines
It's always been the case that concurrent syscalls could change user memory,
io_uringdoesn't change that, it just makes it even less well defined when that would happen. If you want to serialize against that, you need to arrange that externally.
membarrier does replace barrier instructions on remote CPUs, which do order accesses performed by the kernel on the user address space. So membarrier should too I guess.
Normal process context accesses like read(2) will do so because they don't get filtered out from IPIs, but kernel threads using the mm may not.
Thanks,
Nick
Nicholas Piggin
reviewerRe: [RFC PATCH 4/7] x86: use exit_lazy_tlb rather than membarrier_mm_sync_core_before_usermode
In reply to Mathieu DesnoyersExcerpts from Mathieu Desnoyers's message of July 17, 2020 4:58 am:
Quoted 60 lines
Quoted 38 lines · ----- On Jul 16, 2020, at 12:03 PM, Mathieu Desnoyers…
----- On Jul 16, 2020, at 11:46 AM, Mathieu Desnoyers
Quoted 18 lines · [email protected] wrote:
Quoted 17 lines · ----- On Jul 16, 2020, at 12:42 AM, Nicholas Piggin [email protected] wrote:
I should be more complete here, especially since I was complaining about unclear barrier comment :)
CPU0 CPU1 a. user stuff 1. user stuff b. membarrier() 2. enter kernel c. smp_mb() 3. smp_mb__after_spinlock(); // in __schedule d. read rq->curr 4. rq->curr switched to kthread e. is kthread, skip IPI 5. switch_to kthread f. return to user 6. rq->curr switched to user thread g. user stuff 7. switch_to user thread8. exit kernel
9. more user stuffWhat you're really ordering is a, g vs 1, 9 right?
Quoted 8 lines
Quoted 6 lines
In other words, 9 must see a if it sees g, g must see 1 if it saw 9, etc.
Userspace does not care where the barriers are exactly or what kernel memory accesses might be being ordered by them, so long as there is a mb somewhere between a and g, and 1 and 9. Right?
This is correct.
Actually, sorry, the above is not quite right. It's been a while since I looked into the details of membarrier.
The
smp_mb()at the beginning ofmembarrier()needs to be paired with asmp_mb()_after_rq->curris switched back to the user thread, so the memory barrier is between store torq->currand following user-space accesses.Quoted 6 lines
The
smp_mb()at the end ofmembarrier()needs to be paired with thesmp_mb__after_spinlock()at the beginning of schedule, which is between accesses to userspace memory and switchingrq->currto kthread.As to *why* this ordering is needed, I'd have to dig through additional scenarios from https://lwn.net/Articles/573436/. Or maybe Paul remembers ?
Thinking further about this, I'm beginning to consider that maybe we have been overly cautious by requiring memory barriers before and after store to
rq->curr.If CPU0 observes a CPU1's
rq->curr->mmwhich differs from its own process (current) while running the membarrier system call, it necessarily means that CPU1 had to issuesmp_mb__after_spinlockwhen entering the scheduler, between any user-space loads/stores and update ofrq->curr.Requiring a memory barrier between update of
rq->curr(back to current process's thread) and following user-space memory accesses does not seem to guarantee anything more than what the initial barrier at the beginning of__schedulealready provides, because the guarantees are only about accesses to user-space memory.
Quoted 6 lines
Therefore, with the memory barrier at the beginning of
__schedule, just observing that CPU1'srq->currdiffers from current should guarantee that a memory barrier was issued between any sequentially consistent instructions belonging to the current process on CPU1.Or am I missing/misremembering an important point here ?
I might have mislead you.
CPU0 CPU1 r1=y x=1 membarrier() y=1
r2=x
membarrier provides if r1==1 then r2==1 (right?)
CPU0
r1=y
membarrier()
smp_mb();
t = cpu_rq(1)->curr;
if (t->mm == mm)
IPI(CPU1);
smp_mb()
r2=x
vs
CPU1
...
__schedule()
smp_mb__after_spinlock()
rq->curr = kthread
...
__schedule()
smp_mb__after_spinlock()
rq->curr = user thread exit kernel
x=1
y=1
Now these last 3 stores are not ordered, so CPU0 might see y==1 but rq->curr == kthread, right? Then it will skip the IPI and stores to x and y will not be ordered.
So we do need a mb after rq->curr store when mm is switching.
I believe for the global membarrier PF_KTHREAD optimisation, we also need a barrier when switching from a kernel thread to user, for the same reason.
So I think I was wrong to say the barrier is not necessary.
I haven't quite worked out why two mb()s are required in membarrier(), but at least that's less of a performance concern.
Thanks,
Nick
Mathieu Desnoyers
maintainerRe: [RFC PATCH 4/7] x86: use exit_lazy_tlb rather than membarrier_mm_sync_core_before_usermode
In reply to Alan SternQuoted 90 lines · ----- On Jul 16, 2020, at 5:24 PM, Alan Stern [email protected] wrote:
Quoted 72 lines · Mathieu Desnoyers
----- On Jul 16, 2020, at 12:03 PM, Mathieu Desnoyers
Quoted 38 lines · [email protected] wrote:
----- On Jul 16, 2020, at 11:46 AM, Mathieu Desnoyers
Quoted 18 lines · [email protected] wrote:
Quoted 17 lines · ----- On Jul 16, 2020, at 12:42 AM, Nicholas Piggin [email protected] wrote:
I should be more complete here, especially since I was complaining about unclear barrier comment :)
CPU0 CPU1 a. user stuff 1. user stuff b. membarrier() 2. enter kernel c. smp_mb() 3. smp_mb__after_spinlock(); // in __schedule d. read rq->curr 4. rq->curr switched to kthread e. is kthread, skip IPI 5. switch_to kthread f. return to user 6. rq->curr switched to user thread g. user stuff 7. switch_to user thread8. exit kernel
9. more user stuffWhat you're really ordering is a, g vs 1, 9 right?
Quoted 8 lines
Quoted 6 lines
In other words, 9 must see a if it sees g, g must see 1 if it saw 9, etc.
Userspace does not care where the barriers are exactly or what kernel memory accesses might be being ordered by them, so long as there is a mb somewhere between a and g, and 1 and 9. Right?
This is correct.
Actually, sorry, the above is not quite right. It's been a while since I looked into the details of membarrier.
The
smp_mb()at the beginning ofmembarrier()needs to be paired with asmp_mb()_after_rq->curris switched back to the user thread, so the memory barrier is between store torq->currand following user-space accesses.Quoted 6 lines
The
smp_mb()at the end ofmembarrier()needs to be paired with thesmp_mb__after_spinlock()at the beginning of schedule, which is between accesses to userspace memory and switchingrq->currto kthread.As to *why* this ordering is needed, I'd have to dig through additional scenarios from https://lwn.net/Articles/573436/. Or maybe Paul remembers ?
Thinking further about this, I'm beginning to consider that maybe we have been overly cautious by requiring memory barriers before and after store to
rq->curr.If CPU0 observes a CPU1's
rq->curr->mmwhich differs from its own process (current)
while running the membarrier system call, it necessarily means that CPU1 had to issuesmp_mb__after_spinlockwhen entering the scheduler, between any user-space
loads/stores and update ofrq->curr.Requiring a memory barrier between update of
rq->curr(back to current process's thread) and following user-space memory accesses does not seem to guarantee anything more than what the initial barrier at the beginning of__schedulealready
provides, because the guarantees are only about accesses to user-space memory.Therefore, with the memory barrier at the beginning of
__schedule, just observing that
CPU1'srq->currdiffers from current should guarantee that a memory barrier was issued
between any sequentially consistent instructions belonging to the current process on
CPU1.Quoted 1 line
Or am I missing/misremembering an important point here ?
Is it correct to say that the
switch_tooperations in 5 and 7 include memory barriers? If they do, then skipping the IPI should be okay.The reason is as follows: The guarantee you need to enforce is that anything written by CPU0 before the
membarrier()will be visible to CPU1 after it returns to user mode. Let's say that a writes to X and 9 reads from X.Then we have an instance of the Store Buffer pattern:
CPU0 CPU1 a. Write X 6. Write rq->curr for user thread c. smp_mb() 7. switch_to memory barrier d. Read rq->curr 9. Read X
Quoted 7 lines
In this pattern, the memory barriers make it impossible for both reads to miss their corresponding writes. Since d does fail to read 6 (it sees the earlier value stored by 4), 9 must read a.
The other guarantee you need is that g on CPU0 will observe anything written by CPU1 in 1. This is easier to see, using the fact that 3 is a memory barrier and d reads from 4.
Right, and Nick's reply involving pairs of loads/stores on each side clarifies the situation even further.
Thanks,
Mathieu
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com
Mathieu Desnoyers
maintainerRe: [RFC PATCH 4/7] x86: use exit_lazy_tlb rather than membarrier_mm_sync_core_before_usermode
In reply to Nicholas Piggin----- On Jul 16, 2020, at 7:26 PM, Nicholas Piggin [email protected] wrote:
[...]
Quoted 8 lines
membarrier does replace barrier instructions on remote CPUs, which do order accesses performed by the kernel on the user address space. So membarrier should too I guess.
Normal process context accesses like read(2) will do so because they don't get filtered out from IPIs, but kernel threads using the mm may not.
But it should not be an issue, because membarrier's ordering is only with respect to submit and completion of io_uring requests, which are performed through system calls from the context of user-space threads, which are called from the right mm.
Thanks,
Mathieu
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com
Re: [RFC PATCH 4/7] x86: use exit_lazy_tlb rather than membarrier_mm_sync_core_before_usermode
In reply to Mathieu DesnoyersQuoted 24 lines · Mathieu Desnoyers
Quoted 22 lines · ----- On Jul 16, 2020, at 5:24 PM, Alan Stern [email protected] wrote:
Quoted 21 lines · Mathieu Desnoyers
----- On Jul 16, 2020, at 12:03 PM, Mathieu Desnoyers
Quoted 18 lines · [email protected] wrote:
----- On Jul 16, 2020, at 11:46 AM, Mathieu Desnoyers
Quoted 15 lines · [email protected] wrote:
Quoted 14 lines · ----- On Jul 16, 2020, at 12:42 AM, Nicholas Piggin [email protected] wrote:
I should be more complete here, especially since I was complaining about unclear barrier comment :)
CPU0 CPU1 a. user stuff 1. user stuff b. membarrier() 2. enter kernel c. smp_mb() 3. smp_mb__after_spinlock(); // in __schedule d. read rq->curr 4. rq->curr switched to kthread e. is kthread, skip IPI 5. switch_to kthread f. return to user 6. rq->curr switched to user thread g. user stuff 7. switch_to user thread8. exit kernel
9. more user stuff
...
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Requiring a memory barrier between update of
rq->curr(back to current process's thread) and following user-space memory accesses does not seem to guarantee anything more than what the initial barrier at the beginning of__schedulealready
provides, because the guarantees are only about accesses to user-space memory.
...
Quoted 19 lines
Quoted 19 lines
Is it correct to say that the
switch_tooperations in 5 and 7 include memory barriers? If they do, then skipping the IPI should be okay.The reason is as follows: The guarantee you need to enforce is that anything written by CPU0 before the
membarrier()will be visible to CPU1 after it returns to user mode. Let's say that a writes to X and 9 reads from X.Then we have an instance of the Store Buffer pattern:
CPU0 CPU1 a. Write X 6. Write rq->curr for user thread c. smp_mb() 7. switch_to memory barrier d. Read rq->curr 9. Read XIn this pattern, the memory barriers make it impossible for both reads to miss their corresponding writes. Since d does fail to read 6 (it sees the earlier value stored by 4), 9 must read a.
Quoted 6 lines
Quoted 3 lines
The other guarantee you need is that g on CPU0 will observe anything written by CPU1 in 1. This is easier to see, using the fact that 3 is a memory barrier and d reads from 4.
Right, and Nick's reply involving pairs of loads/stores on each side clarifies the situation even further.
The key part of my reply was the question: "Is it correct to say that the switch_to operations in 5 and 7 include memory barriers?" From the text quoted above and from Nick's reply, it seems clear that they do not.
I agree with Nick: A memory barrier is needed somewhere between the assignment at 6 and the return to user mode at 8. Otherwise you end up with the Store Buffer pattern having a memory barrier on only one side, and it is well known that this arrangement does not guarantee any ordering.
One thing I don't understand about all this: Any context switch has to include a memory barrier somewhere, but both you and Nick seem to be saying that steps 6 and 7 don't include (or don't need) any memory barriers. What am I missing?
Alan Stern
Mathieu Desnoyers
maintainerRe: [RFC PATCH 4/7] x86: use exit_lazy_tlb rather than membarrier_mm_sync_core_before_usermode
Quoted 59 lines · ----- On Jul 17, 2020, at 10:51 AM, Alan Stern [email protected] wrote:
Quoted 24 lines · Mathieu Desnoyers
Quoted 22 lines · ----- On Jul 16, 2020, at 5:24 PM, Alan Stern [email protected] wrote:
Quoted 21 lines · Mathieu Desnoyers
----- On Jul 16, 2020, at 12:03 PM, Mathieu Desnoyers
Quoted 18 lines · [email protected] wrote:
----- On Jul 16, 2020, at 11:46 AM, Mathieu Desnoyers
Quoted 15 lines · [email protected] wrote:
Quoted 14 lines · ----- On Jul 16, 2020, at 12:42 AM, Nicholas Piggin [email protected] wrote:
I should be more complete here, especially since I was complaining about unclear barrier comment :)
CPU0 CPU1 a. user stuff 1. user stuff b. membarrier() 2. enter kernel c. smp_mb() 3. smp_mb__after_spinlock(); // in __schedule d. read rq->curr 4. rq->curr switched to kthread e. is kthread, skip IPI 5. switch_to kthread f. return to user 6. rq->curr switched to user thread g. user stuff 7. switch_to user thread8. exit kernel
9. more user stuff...
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Requiring a memory barrier between update of
rq->curr(back to current process's thread) and following user-space memory accesses does not seem to guarantee anything more than what the initial barrier at the beginning of__schedulealready
provides, because the guarantees are only about accesses to user-space memory....
Quoted 23 lines
Quoted 22 lines
Is it correct to say that the
switch_tooperations in 5 and 7 include memory barriers? If they do, then skipping the IPI should be okay.The reason is as follows: The guarantee you need to enforce is that anything written by CPU0 before the
membarrier()will be visible to CPU1 after it returns to user mode. Let's say that a writes to X and 9 reads from X.Then we have an instance of the Store Buffer pattern:
CPU0 CPU1 a. Write X 6. Write rq->curr for user thread c. smp_mb() 7. switch_to memory barrier d. Read rq->curr 9. Read XIn this pattern, the memory barriers make it impossible for both reads to miss their corresponding writes. Since d does fail to read 6 (it sees the earlier value stored by 4), 9 must read a.
The other guarantee you need is that g on CPU0 will observe anything written by CPU1 in 1. This is easier to see, using the fact that 3 is a memory barrier and d reads from 4.
Quoted 7 lines
Quoted 2 lines
Right, and Nick's reply involving pairs of loads/stores on each side clarifies the situation even further.
The key part of my reply was the question: "Is it correct to say that the
switch_tooperations in 5 and 7 include memory barriers?" From the text quoted above and from Nick's reply, it seems clear that they do not.
I remember that switch_mm implies it, but not switch_to.
The scenario that triggered this discussion is when the scheduler does a lazy tlb entry/exit, which is basically switch from a user task to a kernel thread without changing the mm, and usually switching back afterwards. This optimization means the rq->curr mm temporarily differs, which prevent IPIs from being sent by membarrier, but without involving a switch_mm. This requires explicit memory barriers either on entry/exit of lazy tlb mode, or explicit barriers in the scheduler for those special-cases.
Quoted 5 lines
I agree with Nick: A memory barrier is needed somewhere between the assignment at 6 and the return to user mode at 8. Otherwise you end up with the Store Buffer pattern having a memory barrier on only one side, and it is well known that this arrangement does not guarantee any ordering.
Yes, I see this now. I'm still trying to wrap my head around why the memory barrier at the end of membarrier() needs to be paired with a scheduler barrier though.
Quoted 4 lines
One thing I don't understand about all this: Any context switch has to include a memory barrier somewhere, but both you and Nick seem to be saying that steps 6 and 7 don't include (or don't need) any memory barriers. What am I missing?
All context switch have the smp_mb__before_spinlock at the beginning of __schedule(), which I suspect is what you refer to. However this barrier is before the store to rq->curr, not after.
Thanks,
Mathieu
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com
Re: [RFC PATCH 4/7] x86: use exit_lazy_tlb rather than membarrier_mm_sync_core_before_usermode
Quoted 9 lines
Quoted 5 lines
I agree with Nick: A memory barrier is needed somewhere between the assignment at 6 and the return to user mode at 8. Otherwise you end up with the Store Buffer pattern having a memory barrier on only one side, and it is well known that this arrangement does not guarantee any ordering.
Yes, I see this now. I'm still trying to wrap my head around why the memory barrier at the end of
membarrier()needs to be paired with a scheduler barrier though.
The memory barrier at the end of membarrier() on CPU0 is necessary in order to enforce the guarantee that any writes occurring on CPU1 before the membarrier() is executed will be visible to any code executing on CPU0 after the membarrier(). Ignoring the kthread issue, we can have:
CPU0 CPU1 x = 1 barrier() y = 1 r2 = y membarrier(): a: smp_mb() b: send IPI IPI-induced mb c: smp_mb() r1 = x
The writes to x and y are unordered by the hardware, so it's possible to have r2 = 1 even though the write to x doesn't execute until b. If the memory barrier at c is omitted then "r1 = x" can be reordered before b (although not before a), so we get r1 = 0. This violates the guarantee that membarrier() is supposed to provide.
The timing of the memory barrier at c has to ensure that it executes after the IPI-induced memory barrier on CPU1. If it happened before then we could still end up with r1 = 0. That's why the pairing matters.
I hope this helps your head get properly wrapped. :-)
Alan Stern
Mathieu Desnoyers
maintainerRe: [RFC PATCH 4/7] x86: use exit_lazy_tlb rather than membarrier_mm_sync_core_before_usermode
Quoted 32 lines · ----- On Jul 17, 2020, at 12:11 PM, Alan Stern [email protected] wrote:
Quoted 9 lines
Quoted 5 lines
I agree with Nick: A memory barrier is needed somewhere between the assignment at 6 and the return to user mode at 8. Otherwise you end up with the Store Buffer pattern having a memory barrier on only one side, and it is well known that this arrangement does not guarantee any ordering.
Yes, I see this now. I'm still trying to wrap my head around why the memory barrier at the end of
membarrier()needs to be paired with a scheduler barrier though.The memory barrier at the end of
membarrier()on CPU0 is necessary in order to enforce the guarantee that any writes occurring on CPU1 before themembarrier()is executed will be visible to any code executing on CPU0 after themembarrier(). Ignoring the kthread issue, we can have:CPU0 CPU1 x = 1 barrier() y = 1 r2 = y membarrier(): a: smp_mb() b: send IPI IPI-induced mb c: smp_mb() r1 = xThe writes to x and y are unordered by the hardware, so it's possible to have r2 = 1 even though the write to x doesn't execute until b. If the memory barrier at c is omitted then "r1 = x" can be reordered before b (although not before a), so we get r1 = 0. This violates the guarantee that
membarrier()is supposed to provide.
Quoted 5 lines
The timing of the memory barrier at c has to ensure that it executes after the IPI-induced memory barrier on CPU1. If it happened before then we could still end up with r1 = 0. That's why the pairing matters.
I hope this helps your head get properly wrapped. :-)
It does help a bit! ;-)
This explains this part of the comment near the smp_mb at the end of membarrier:
* Memory barrier on the caller thread _after_ we finished
* waiting for the last IPI. [...]
However, it does not explain why it needs to be paired with a barrier in the scheduler, clearly for the case where the IPI is skipped. I wonder whether this part of the comment is factually correct:
* [...] Matches memory barriers around rq->curr modification in scheduler.
Thanks,
Mathieu
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com
Re: [RFC PATCH 4/7] x86: use exit_lazy_tlb rather than membarrier_mm_sync_core_before_usermode
Quoted 44 lines · Mathieu Desnoyers
Quoted 32 lines · ----- On Jul 17, 2020, at 12:11 PM, Alan Stern [email protected] wrote:
Quoted 9 lines
Quoted 5 lines
I agree with Nick: A memory barrier is needed somewhere between the assignment at 6 and the return to user mode at 8. Otherwise you end up with the Store Buffer pattern having a memory barrier on only one side, and it is well known that this arrangement does not guarantee any ordering.
Yes, I see this now. I'm still trying to wrap my head around why the memory barrier at the end of
membarrier()needs to be paired with a scheduler barrier though.The memory barrier at the end of
membarrier()on CPU0 is necessary in order to enforce the guarantee that any writes occurring on CPU1 before themembarrier()is executed will be visible to any code executing on CPU0 after themembarrier(). Ignoring the kthread issue, we can have:CPU0 CPU1 x = 1 barrier() y = 1 r2 = y membarrier(): a: smp_mb() b: send IPI IPI-induced mb c: smp_mb() r1 = xThe writes to x and y are unordered by the hardware, so it's possible to have r2 = 1 even though the write to x doesn't execute until b. If the memory barrier at c is omitted then "r1 = x" can be reordered before b (although not before a), so we get r1 = 0. This violates the guarantee that
membarrier()is supposed to provide.Quoted 5 lines
The timing of the memory barrier at c has to ensure that it executes after the IPI-induced memory barrier on CPU1. If it happened before then we could still end up with r1 = 0. That's why the pairing matters.
I hope this helps your head get properly wrapped. :-)
It does help a bit! ;-)
This explains this part of the comment near the
smp_mbat the end of membarrier:
Quoted 8 lines
* Memory barrier on the caller thread _after_ we finished
* waiting for the last IPI. [...]However, it does not explain why it needs to be paired with a barrier in the scheduler, clearly for the case where the IPI is skipped. I wonder whether this part of the comment is factually correct:
* [...] Matches memory barriers around
rq->currmodification in scheduler.
The reasoning is pretty much the same as above:
CPU0 CPU1 x = 1 barrier() y = 1 r2 = y membarrier(): a: smp_mb() switch to kthread (includes mb) b: read rq->curr == kthread switch to user (includes mb) c: smp_mb() r1 = x
Once again, it is possible that x = 1 doesn't become visible to CPU0 until shortly before b. But if c is omitted then "r1 = x" can be reordered before b (to any time after a), so we can have r1 = 0.
Here the timing requirement is that c executes after the first memory barrier on CPU1 -- which is one of the ones around the rq->curr modification. (In fact, in this scenario CPU1's switch back to the user process is irrelevant.)
Alan Stern
Mathieu Desnoyers
maintainerRe: [RFC PATCH 4/7] x86: use exit_lazy_tlb rather than membarrier_mm_sync_core_before_usermode
Quoted 70 lines · ----- On Jul 17, 2020, at 1:44 PM, Alan Stern [email protected] wrote:
Quoted 47 lines · Mathieu Desnoyers
Quoted 32 lines · ----- On Jul 17, 2020, at 12:11 PM, Alan Stern [email protected] wrote:
Quoted 9 lines
Quoted 5 lines
I agree with Nick: A memory barrier is needed somewhere between the assignment at 6 and the return to user mode at 8. Otherwise you end up with the Store Buffer pattern having a memory barrier on only one side, and it is well known that this arrangement does not guarantee any ordering.
Yes, I see this now. I'm still trying to wrap my head around why the memory barrier at the end of
membarrier()needs to be paired with a scheduler barrier though.The memory barrier at the end of
membarrier()on CPU0 is necessary in order to enforce the guarantee that any writes occurring on CPU1 before themembarrier()is executed will be visible to any code executing on CPU0 after themembarrier(). Ignoring the kthread issue, we can have:CPU0 CPU1 x = 1 barrier() y = 1 r2 = y membarrier(): a: smp_mb() b: send IPI IPI-induced mb c: smp_mb() r1 = xThe writes to x and y are unordered by the hardware, so it's possible to have r2 = 1 even though the write to x doesn't execute until b. If the memory barrier at c is omitted then "r1 = x" can be reordered before b (although not before a), so we get r1 = 0. This violates the guarantee that
membarrier()is supposed to provide.Quoted 5 lines
The timing of the memory barrier at c has to ensure that it executes after the IPI-induced memory barrier on CPU1. If it happened before then we could still end up with r1 = 0. That's why the pairing matters.
I hope this helps your head get properly wrapped. :-)
It does help a bit! ;-)
This explains this part of the comment near the
smp_mbat the end of membarrier:* Memory barrier on the caller thread _after_ we finished
* waiting for the last IPI. [...]Quoted 6 lines
However, it does not explain why it needs to be paired with a barrier in the scheduler, clearly for the case where the IPI is skipped. I wonder whether this part
of the comment is factually correct:* [...] Matches memory barriers around
rq->currmodification in scheduler.The reasoning is pretty much the same as above:
CPU0 CPU1 x = 1 barrier() y = 1 r2 = y membarrier(): a: smp_mb() switch to kthread (includes mb) b: read rq->curr == kthread switch to user (includes mb) c: smp_mb() r1 = x
Quoted 8 lines
Once again, it is possible that x = 1 doesn't become visible to CPU0 until shortly before b. But if c is omitted then "r1 = x" can be reordered before b (to any time after a), so we can have r1 = 0.
Here the timing requirement is that c executes after the first memory barrier on CPU1 -- which is one of the ones around the
rq->currmodification. (In fact, in this scenario CPU1's switch back to the user process is irrelevant.)
That indeed covers the last scenario I was wondering about. Thanks Alan!
Mathieu
Quoted 2 lines
Alan Stern
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com
Nicholas Piggin
reviewerRe: [RFC PATCH 4/7] x86: use exit_lazy_tlb rather than membarrier_mm_sync_core_before_usermode
In reply to Mathieu DesnoyersExcerpts from Mathieu Desnoyers's message of July 17, 2020 11:42 pm:
Quoted 7 lines
----- On Jul 16, 2020, at 7:26 PM, Nicholas Piggin [email protected] wrote:
[...]Quoted 5 lines
membarrier does replace barrier instructions on remote CPUs, which do order accesses performed by the kernel on the user address space. So membarrier should too I guess.
Quoted 8 lines
Quoted 3 lines
Normal process context accesses like read(2) will do so because they don't get filtered out from IPIs, but kernel threads using the mm may not.
But it should not be an issue, because membarrier's ordering is only with respect to submit and completion of
io_uringrequests, which are performed through system calls from the context of user-space threads, which are called from the right mm.
Is that true? Can io completions be written into an address space via a kernel thread? I don't know the io_uring code well but it looks like that's asynchonously using the user mm context.
How about other memory accesses via kthread_use_mm? Presumably there is still ordering requirement there for membarrier, so I really think it's a fragile interface with no real way for the user to know how kernel threads may use its mm for any particular reason, so membarrier should synchronize all possible kernel users as well.
Thanks,
Nick
Mathieu Desnoyers
maintainerRe: [RFC PATCH 4/7] x86: use exit_lazy_tlb rather than membarrier_mm_sync_core_before_usermode
Quoted 18 lines · ----- On Jul 19, 2020, at 11:03 PM, Nicholas Piggin [email protected] wrote:
Excerpts from Mathieu Desnoyers's message of July 17, 2020 11:42 pm:
Quoted 11 lines
----- On Jul 16, 2020, at 7:26 PM, Nicholas Piggin [email protected] wrote:
[...]Quoted 8 lines
membarrier does replace barrier instructions on remote CPUs, which do order accesses performed by the kernel on the user address space. So membarrier should too I guess.
Normal process context accesses like read(2) will do so because they don't get filtered out from IPIs, but kernel threads using the mm may not.
Quoted 5 lines
But it should not be an issue, because membarrier's ordering is only with respect
to submit and completion ofio_uringrequests, which are performed through system calls from the context of user-space threads, which are called from the right mm.
Quoted 3 lines
Is that true? Can io completions be written into an address space via a kernel thread? I don't know the
io_uringcode well but it looks like that's asynchonously using the user mm context.
Indeed, the io completion appears to be signaled asynchronously between kernel and user-space. Therefore, both kernel and userspace code need to have proper memory barriers in place to signal completion, otherwise user-space could read garbage after it notices completion of a read.
I did not review the entire io_uring implementation, but the publish side for completion appears to be:
static void __io_commit_cqring(struct io_ring_ctx *ctx)
{
struct io_rings *rings = ctx->rings;
/* order cqe stores with ring update */
smp_store_release(&rings->cq.tail, ctx->cached_cq_tail);
if (wq_has_sleeper(&ctx->cq_wait)) {
wake_up_interruptible(&ctx->cq_wait);
kill_fasync(&ctx->cq_fasync, SIGIO, POLL_IN);
}
}The store-release on tail should be paired with a load_acquire on the reader-side (it's called "read_barrier()" in the code):
tools/io_uring/queue.c:
static int __io_uring_get_cqe(struct io_uring *ring,
struct io_uring_cqe **cqe_ptr, int wait)
{
struct io_uring_cq *cq = &ring->cq;
const unsigned mask = *cq->kring_mask;
unsigned head;
int ret;
*cqe_ptr = NULL;
head = *cq->khead;
do {
/*
* It's necessary to use a read_barrier() before reading
* the CQ tail, since the kernel updates it locklessly. The
* kernel has the matching store barrier for the update. The
* kernel also ensures that previous stores to CQEs are ordered
* with the tail update.
*/
read_barrier();
if (head != *cq->ktail) {
*cqe_ptr = &cq->cqes[head & mask];
break;
}
if (!wait)
break;
ret = io_uring_enter(ring->ring_fd, 0, 1,
IORING_ENTER_GETEVENTS, NULL);
if (ret < 0)
return -errno;
} while (1);
return 0;
}So as far as membarrier memory ordering dependencies are concerned, it relies on the store-release/load-acquire dependency chain in the completion queue to order against anything that was done prior to the completed requests.
What is in-flight while the requests are being serviced provides no memory ordering guarantee whatsoever.
Quoted 2 lines
How about other memory accesses via
kthread_use_mm? Presumably there is still ordering requirement there for membarrier,
Please provide an example case with memory accesses via kthread_use_mm where ordering matters to support your concern.
Quoted 4 lines
so I really think
it's a fragile interface with no real way for the user to know how kernel threads may use its mm for any particular reason, so membarrier should synchronize all possible kernel users as well.
I strongly doubt so, but perhaps something should be clarified in the documentation if you have that feeling.
Thanks,
Mathieu
Quoted 3 lines
Thanks,
Nick
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com
Nicholas Piggin
reviewerRe: [RFC PATCH 4/7] x86: use exit_lazy_tlb rather than membarrier_mm_sync_core_before_usermode
Excerpts from Mathieu Desnoyers's message of July 21, 2020 2:46 am:
Quoted 20 lines
Quoted 18 lines · ----- On Jul 19, 2020, at 11:03 PM, Nicholas Piggin [email protected] wrote:
Excerpts from Mathieu Desnoyers's message of July 17, 2020 11:42 pm:
Quoted 16 lines
----- On Jul 16, 2020, at 7:26 PM, Nicholas Piggin [email protected] wrote:
[...]Quoted 8 lines
membarrier does replace barrier instructions on remote CPUs, which do order accesses performed by the kernel on the user address space. So membarrier should too I guess.
Normal process context accesses like read(2) will do so because they don't get filtered out from IPIs, but kernel threads using the mm may not.
But it should not be an issue, because membarrier's ordering is only with respect
to submit and completion ofio_uringrequests, which are performed through system calls from the context of user-space threads, which are called from the right mm.
Quoted 6 lines
Quoted 3 lines
Is that true? Can io completions be written into an address space via a kernel thread? I don't know the
io_uringcode well but it looks like that's asynchonously using the user mm context.Indeed, the io completion appears to be signaled asynchronously between kernel and user-space.
Yep, many other places do similar with use_mm.
[snip]
Quoted 6 lines
So as far as membarrier memory ordering dependencies are concerned, it relies on the store-release/load-acquire dependency chain in the completion queue to order against anything that was done prior to the completed requests.
What is in-flight while the requests are being serviced provides no memory ordering guarantee whatsoever.
Yeah you're probably right in this case I think. Quite likely most kernel tasks that asynchronously write to user memory would at least have some kind of producer-consumer barriers.
But is that restriction of all async modifications documented and enforced anywhere?
Quoted 5 lines
Quoted 2 lines
How about other memory accesses via
kthread_use_mm? Presumably there is still ordering requirement there for membarrier,Please provide an example case with memory accesses via
kthread_use_mmwhere ordering matters to support your concern.
I think the concern Andy raised with io_uring was less a specific problem he saw and more a general concern that we have these memory accesses which are not synchronized with membarrier.
Quoted 7 lines
Quoted 4 lines
so I really think
it's a fragile interface with no real way for the user to know how kernel threads may use its mm for any particular reason, so membarrier should synchronize all possible kernel users as well.I strongly doubt so, but perhaps something should be clarified in the documentation if you have that feeling.
I'd rather go the other way and say if you have reasoning or numbers for why PF_KTHREAD is an important optimisation above rq->curr == rq->idle then we could think about keeping this subtlety with appropriate documentation added, otherwise we can just kill it and remove all doubt.
That being said, the x86 sync core gap that I imagined could be fixed by changing to rq->curr == rq->idle test does not actually exist because the global membarrier does not have a sync core option. So fixing the exit_lazy_tlb points that this series does *should* fix that. So PF_KTHREAD may be less problematic than I thought from implementation point of view, only semantics.
Thanks,
Nick
Mathieu Desnoyers
maintainerRe: [RFC PATCH 4/7] x86: use exit_lazy_tlb rather than membarrier_mm_sync_core_before_usermode
Quoted 1 line · ----- On Jul 21, 2020, at 6:04 AM, Nicholas Piggin [email protected] wrote:
Excerpts from Mathieu Desnoyers's message of July 21, 2020 2:46 am:
[...]
Quoted 32 lines
Yeah you're probably right in this case I think. Quite likely most kernel tasks that asynchronously write to user memory would at least have some kind of producer-consumer barriers.
But is that restriction of all async modifications documented and enforced anywhere?
Quoted 5 lines
Quoted 2 lines
How about other memory accesses via
kthread_use_mm? Presumably there is still ordering requirement there for membarrier,Please provide an example case with memory accesses via
kthread_use_mmwhere ordering matters to support your concern.I think the concern Andy raised with
io_uringwas less a specific problem he saw and more a general concern that we have these memory accesses which are not synchronized with membarrier.Quoted 8 lines
Quoted 4 lines
so I really think
it's a fragile interface with no real way for the user to know how kernel threads may use its mm for any particular reason, so membarrier should synchronize all possible kernel users as well.I strongly doubt so, but perhaps something should be clarified in the documentation
if you have that feeling.I'd rather go the other way and say if you have reasoning or numbers for why
PF_KTHREADis an important optimisation aboverq->curr==rq->idlethen we could think about keeping this subtlety with appropriate documentation added, otherwise we can just kill it and remove all doubt.
Quoted 6 lines
That being said, the x86 sync core gap that I imagined could be fixed by changing to
rq->curr==rq->idletest does not actually exist because the global membarrier does not have a sync core option. So fixing theexit_lazy_tlbpoints that this series does *should* fix that. SoPF_KTHREADmay be less problematic than I thought from implementation point of view, only semantics.
Today, the membarrier global expedited command explicitly skips kernel threads, but it happens that membarrier private expedited considers those with the same mm as target for the IPI.
So we already implement a semantic which differs between private and global expedited membarriers. This can be explained in part by the fact that kthread_use_mm was introduced after 4.16, where the most recent membarrier commands where introduced. It seems that the effect on membarrier was not considered when kthread_use_mm was introduced.
Looking at membarrier(2) documentation, it states that IPIs are only sent to threads belonging to the same process as the calling thread. If my understanding of the notion of process is correct, this should rule out sending the IPI to kernel threads, given they are not "part" of the same process, only borrowing the mm. But I agree that the distinction is moot, and should be clarified.
Without a clear use-case to justify adding a constraint on membarrier, I am tempted to simply clarify documentation of current membarrier commands, stating clearly that they are not guaranteed to affect kernel threads. Then, if we have a compelling use-case to implement a different behavior which covers kthreads, this could be added consistently across membarrier commands with a flag (or by adding new commands).
Does this approach make sense ?
Thanks,
Mathieu
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com
Nicholas Piggin
reviewerRe: [RFC PATCH 4/7] x86: use exit_lazy_tlb rather than membarrier_mm_sync_core_before_usermode
Excerpts from Mathieu Desnoyers's message of July 21, 2020 11:11 pm:
Quoted 43 lines
Quoted 1 line · ----- On Jul 21, 2020, at 6:04 AM, Nicholas Piggin [email protected] wrote:
Excerpts from Mathieu Desnoyers's message of July 21, 2020 2:46 am:
[...]
Quoted 32 lines
Yeah you're probably right in this case I think. Quite likely most kernel tasks that asynchronously write to user memory would at least have some kind of producer-consumer barriers.
But is that restriction of all async modifications documented and enforced anywhere?
Quoted 5 lines
Quoted 2 lines
How about other memory accesses via
kthread_use_mm? Presumably there is still ordering requirement there for membarrier,Please provide an example case with memory accesses via
kthread_use_mmwhere ordering matters to support your concern.I think the concern Andy raised with
io_uringwas less a specific problem he saw and more a general concern that we have these memory accesses which are not synchronized with membarrier.Quoted 8 lines
Quoted 4 lines
so I really think
it's a fragile interface with no real way for the user to know how kernel threads may use its mm for any particular reason, so membarrier should synchronize all possible kernel users as well.I strongly doubt so, but perhaps something should be clarified in the documentation
if you have that feeling.I'd rather go the other way and say if you have reasoning or numbers for why
PF_KTHREADis an important optimisation aboverq->curr==rq->idlethen we could think about keeping this subtlety with appropriate documentation added, otherwise we can just kill it and remove all doubt.Quoted 6 lines
That being said, the x86 sync core gap that I imagined could be fixed by changing to
rq->curr==rq->idletest does not actually exist because the global membarrier does not have a sync core option. So fixing theexit_lazy_tlbpoints that this series does *should* fix that. SoPF_KTHREADmay be less problematic than I thought from implementation point of view, only semantics.
Quoted 6 lines
Today, the membarrier global expedited command explicitly skips kernel threads, but it happens that membarrier private expedited considers those with the same mm as target for the IPI.
So we already implement a semantic which differs between private and global expedited membarriers.
Which is not a good thing.
Quoted 4 lines
This can be explained in part by the fact that
kthread_use_mmwas introduced after 4.16, where the most recent membarrier commands where introduced. It seems that the effect on membarrier was not considered whenkthread_use_mmwas introduced.
No it was just renamed, it used to be called use_mm and has been in the kernel for ~ever.
That you hadn't considered this is actually weight for my point, which is that there's so much subtle behaviour that's easy to miss we're better off with simpler and fewer special cases until it's proven they're needed. Not the other way around.
Quoted 6 lines
Looking at membarrier(2) documentation, it states that IPIs are only sent to threads belonging to the same process as the calling thread. If my understanding of the notion of process is correct, this should rule out sending the IPI to kernel threads, given they are not "part" of the same process, only borrowing the mm. But I agree that the distinction is moot, and should be clarified.
It does if you read it in a user-hostile legalistic way. The reality is userspace shouldn't and can't know about how the kernel might implement functionality.
Quoted 8 lines
Without a clear use-case to justify adding a constraint on membarrier, I am tempted to simply clarify documentation of current membarrier commands, stating clearly that they are not guaranteed to affect kernel threads. Then, if we have a compelling use-case to implement a different behavior which covers kthreads, this could be added consistently across membarrier commands with a flag (or by adding new commands).
Does this approach make sense ?
The other position is without a clear use case for PF_KTHREAD, seeing as async kernel accesses had not been considered before now, we limit the optimision to only skipping the idle thread. I think that makes more sense (unless you have a reason for PF_KTHREAD but it doesn't seem like there is much of one).
Thanks,
Nick
[email protected]
maintainerRe: [RFC PATCH 4/7] x86: use exit_lazy_tlb rather than membarrier_mm_sync_core_before_usermode
In reply to Nicholas PigginQuoted 6 lines · Nicholas Piggin
That being said, the x86 sync core gap that I imagined could be fixed by changing to
rq->curr==rq->idletest does not actually exist because the global membarrier does not have a sync core option. So fixing theexit_lazy_tlbpoints that this series does *should* fix that. SoPF_KTHREADmay be less problematic than I thought from implementation point of view, only semantics.
So I've been trying to figure out where that PF_KTHREAD comes from, commit 227a4aadc75b ("sched/membarrier: Fix p->mm->membarrier_state racy load") changed 'p->mm' to '!(p->flags & PF_KTHREAD)'.
So the first version:
https://lkml.kernel.org/r/[email protected]
appears to unconditionally send the IPI and checks p->mm in the IPI context, but then v2:
https://lkml.kernel.org/r/[email protected]
has the current code. But I've been unable to find the reason the 'p->mm' test changed into '!(p->flags & PF_KTHREAD)'.
The comment doesn't really help either; sure we have the whole lazy mm thing, but that's ->active_mm, not ->mm.
Possibly it is because {,un}use_mm() do not have sufficient barriers to make the remote p->mm test work? Or were we over-eager with the !p->mm doesn't imply kthread 'cleanups' at the time?
Also, I just realized, I still have a fix for use_mm() now kthread_use_mm() that seems to have been lost.
Mathieu Desnoyers
maintainerRe: [RFC PATCH 4/7] x86: use exit_lazy_tlb rather than membarrier_mm_sync_core_before_usermode
Quoted 17 lines · ----- On Jul 21, 2020, at 11:06 AM, Peter Zijlstra [email protected] wrote:
Quoted 6 lines · Nicholas Piggin
That being said, the x86 sync core gap that I imagined could be fixed by changing to
rq->curr==rq->idletest does not actually exist because the global membarrier does not have a sync core option. So fixing theexit_lazy_tlbpoints that this series does *should* fix that. SoPF_KTHREADmay be less problematic than I thought from implementation point of view, only semantics.So I've been trying to figure out where that
PF_KTHREADcomes from, commit227a4aadc75b("sched/membarrier: Fixp->mm->membarrier_stateracy load") changed 'p->mm' to '!(p->flags &PF_KTHREAD)'.So the first version:
Quoted 7 lines
appears to unconditionally send the IPI and checks
p->mmin the IPI context, but then v2:https://lkml.kernel.org/r/[email protected]
has the current code. But I've been unable to find the reason the '
p->mm' test changed into '!(p->flags &PF_KTHREAD)'.
Looking back at my inbox, it seems like you are the one who proposed to skip all kthreads:
https://lkml.kernel.org/r/[email protected]
Quoted 7 lines
The comment doesn't really help either; sure we have the whole lazy mm thing, but that's ->active_mm, not ->mm.
Possibly it is because {,un}use_mm() do not have sufficient barriers to make the remote
p->mmtest work? Or were we over-eager with the !p->mm doesn't imply kthread 'cleanups' at the time?
The nice thing about adding back kthreads to the threads considered for membarrier IPI is that it has no observable effect on the user-space ABI. No pre-existing kthread rely on this, and we just provide an additional guarantee for future kthread implementations.
Quoted 2 lines
Also, I just realized, I still have a fix for
use_mm()nowkthread_use_mm()that seems to have been lost.
I suspect we need to at least document the memory barriers in kthread_use_mm and kthread_unuse_mm to state that they are required by membarrier if we want to ipi kthreads as well.
Thanks,
Mathieu
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com
Peter Zijlstra
maintainerRe: [RFC PATCH 4/7] x86: use exit_lazy_tlb rather than membarrier_mm_sync_core_before_usermode
Quoted 24 lines · Mathieu Desnoyers
Quoted 22 lines · ----- On Jul 21, 2020, at 11:06 AM, Peter Zijlstra [email protected] wrote:
Quoted 6 lines · Nicholas Piggin
That being said, the x86 sync core gap that I imagined could be fixed by changing to
rq->curr==rq->idletest does not actually exist because the global membarrier does not have a sync core option. So fixing theexit_lazy_tlbpoints that this series does *should* fix that. SoPF_KTHREADmay be less problematic than I thought from implementation point of view, only semantics.So I've been trying to figure out where that
PF_KTHREADcomes from, commit227a4aadc75b("sched/membarrier: Fixp->mm->membarrier_stateracy load") changed 'p->mm' to '!(p->flags &PF_KTHREAD)'.So the first version:
https://lkml.kernel.org/r/[email protected]
appears to unconditionally send the IPI and checks
p->mmin the IPI context, but then v2:
Quoted 7 lines
Quoted 2 lines
has the current code. But I've been unable to find the reason the '
p->mm' test changed into '!(p->flags &PF_KTHREAD)'.Looking back at my inbox, it seems like you are the one who proposed to skip all kthreads:
I had a feeling it might've been me ;-) I just couldn't find the email.
Quoted 12 lines
Quoted 6 lines
The comment doesn't really help either; sure we have the whole lazy mm thing, but that's ->active_mm, not ->mm.
Possibly it is because {,un}use_mm() do not have sufficient barriers to make the remote
p->mmtest work? Or were we over-eager with the !p->mm doesn't imply kthread 'cleanups' at the time?The nice thing about adding back kthreads to the threads considered for membarrier IPI is that it has no observable effect on the user-space ABI. No pre-existing kthread rely on this, and we just provide an additional guarantee for future kthread implementations.
Quoted 6 lines
Quoted 2 lines
Also, I just realized, I still have a fix for
use_mm()nowkthread_use_mm()that seems to have been lost.I suspect we need to at least document the memory barriers in
kthread_use_mmandkthread_unuse_mmto state that they are required by membarrier if we want to ipi kthreads as well.
Right, so going by that email you found it was mostly a case of being lazy, but yes, if we audit the kthread_{,un}use_mm() barriers and add any other bits that might be needed, covering kthreads should be possible.
No objections from me for making it so.
Mathieu Desnoyers
maintainerRe: [RFC PATCH 4/7] x86: use exit_lazy_tlb rather than membarrier_mm_sync_core_before_usermode
Quoted 56 lines · ----- On Jul 21, 2020, at 11:19 AM, Peter Zijlstra [email protected] wrote:
Quoted 24 lines · Mathieu Desnoyers
Quoted 22 lines · ----- On Jul 21, 2020, at 11:06 AM, Peter Zijlstra [email protected] wrote:
Quoted 6 lines · Nicholas Piggin
That being said, the x86 sync core gap that I imagined could be fixed by changing to
rq->curr==rq->idletest does not actually exist because the global membarrier does not have a sync core option. So fixing theexit_lazy_tlbpoints that this series does *should* fix that. SoPF_KTHREADmay be less problematic than I thought from implementation point of view, only semantics.So I've been trying to figure out where that
PF_KTHREADcomes from, commit227a4aadc75b("sched/membarrier: Fixp->mm->membarrier_stateracy load") changed 'p->mm' to '!(p->flags &PF_KTHREAD)'.So the first version:
https://lkml.kernel.org/r/[email protected]
appears to unconditionally send the IPI and checks
p->mmin the IPI context, but then v2:Quoted 7 lines
Quoted 2 lines
has the current code. But I've been unable to find the reason the '
p->mm' test changed into '!(p->flags &PF_KTHREAD)'.Looking back at my inbox, it seems like you are the one who proposed to skip all kthreads:
I had a feeling it might've been me ;-) I just couldn't find the email.
Quoted 14 lines
Quoted 6 lines
The comment doesn't really help either; sure we have the whole lazy mm thing, but that's ->active_mm, not ->mm.
Possibly it is because {,un}use_mm() do not have sufficient barriers to make the remote
p->mmtest work? Or were we over-eager with the !p->mm doesn't imply kthread 'cleanups' at the time?The nice thing about adding back kthreads to the threads considered for membarrier
IPI is that it has no observable effect on the user-space ABI. No pre-existing kthread
rely on this, and we just provide an additional guarantee for future kthread implementations.Quoted 6 lines
Quoted 2 lines
Also, I just realized, I still have a fix for
use_mm()nowkthread_use_mm()that seems to have been lost.I suspect we need to at least document the memory barriers in
kthread_use_mmandkthread_unuse_mmto state that they are required by membarrier if we want to ipi kthreads as well.
Quoted 6 lines
Right, so going by that email you found it was mostly a case of being lazy, but yes, if we audit the kthread_{,un}use_mm() barriers and add any other bits that might be needed, covering kthreads should be possible.
No objections from me for making it so.
I'm OK on making membarrier cover kthreads using mm as well, provided we audit kthread_{,un}use_mm() to make sure the proper barriers are in place after setting task->mm and before clearing it.
Thanks,
Mathieu
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com