AMD broadcast TLB invalidation
Subsystems (4)
v2 cover AMD broadcast TLB invalidation
Add support for broadcast TLB invalidation using AMD's INVLPGB instruction.
This allows the kernel to invalidate TLB entries on remote CPUs without needing to send IPIs, without having to wait for remote CPUs to handle those interrupts, and with less interruption to what was running on those CPUs.
Because x86 PCID space is limited, and there are some very large systems out there, broadcast TLB invalidation is only used for processes that are active on 3 or more CPUs, with the threshold being gradually increased the more the PCID space gets exhausted.
Combined with the removal of unnecessary lru_add_drain calls (see https://lkml.org/lkml/2024/12/19/1388) this results in a nice performance boost for the will-it-scale tlb_flush2_threads test on an AMD Milan system with 36 cores:
- vanilla kernel: 527k loops/second - lru_add_drain removal: 731k loops/second - only INVLPGB: 527k loops/second
- lru_add_drain + INVLPGB: 1157k loops/second
Profiling with only the INVLPGB changes showed while TLB invalidation went down from 40% of the total CPU time to only around 4% of CPU time, the contention simply moved to the LRU lock.
Fixing both at the same time about doubles the
number of iterations per second from this case.
v2:
- Apply suggestions by Peter and Borislav (thank you!)
- Fix bug in arch_tlbbatch_flush, where we need to do both the TLBSYNC, and flush the CPUs that are in the cpumask.
- Some updates to comments and changelogs based on questions.
PATCH v2 1/11 x86/mm: make MMU_GATHER_RCU_TABLE_FREE unconditional
Currently x86 usse CONFIG_MMU_GATHER_TABLE_FREE when using paravirt, and not when running on bare metal.
There is no real good reason to do things differently for each setup. Make them all the same.
--- arch/x86/Kconfig | 2 +- arch/x86/kernel/paravirt.c | 7 +------ 2 files changed, 2 insertions(+), 7 deletions(-)
arch/x86/Kconfig +1 -1
@@ -274,7 +274,7 @@ config X86 select HAVE_PCI select HAVE_PERF_REGS select HAVE_PERF_USER_STACK_DUMP - select MMU_GATHER_RCU_TABLE_FREE if PARAVIRT + select MMU_GATHER_RCU_TABLE_FREE select MMU_GATHER_MERGE_VMAS select HAVE_POSIX_CPU_TIMERS_TASK_WORK select HAVE_REGS_AND_STACK_ACCESS_API
arch/x86/kernel/paravirt.c +1 -6
@@ -59,11 +59,6 @@ void __init native_pv_lock_init(void)static_branch_enable (&virt_spin_lock_key); } -static void native_tlb_remove_table (struct mmu_gather *tlb,void *table) -{ -tlb_remove_page (tlb, table); -} -struct static_key paravirt_steal_enabled;struct static_key paravirt_steal_rq_enabled;
@@ -191,7 +186,7 @@ struct paravirt_patch_template pv_ops = { .mmu.flush_tlb_kernel = native_flush_tlb_global, .mmu.flush_tlb_one_user = native_flush_tlb_one_user, .mmu.flush_tlb_multi = native_flush_tlb_multi, - .mmu.tlb_remove_table = native_tlb_remove_table, + .mmu.tlb_remove_table = tlb_remove_table, .mmu.exit_mmap = paravirt_nop, .mmu.notify_page_enc_status_changed = paravirt_nop,
Signature · 1 line
2.47.1
Re: [PATCH 01/11] x86/mm: make MMU_GATHER_RCU_TABLE_FREE unconditional
Very happy to see this change!
Quoted 1 line · Rik van Riel
Currently x86 usse
CONFIG_MMU_GATHER_TABLE_FREEwhen using
^
useQuoted 42 lines
paravirt, and not when running on bare metal.
There is no real good reason to do things differently for each setup. Make them all the same.
Signed-off-by Rik van Riel <[email protected]>Suggested-by Peter Zijlstra <[email protected]>--- arch/x86/Kconfig | 2 +- arch/x86/kernel/paravirt.c | 7 +------ 2 files changed, 2 insertions(+), 7 deletions(-)arch/x86/Kconfig +1
-1@@ -274,7 +274,7 @@ config X86 select HAVE_PCI select HAVE_PERF_REGS select HAVE_PERF_USER_STACK_DUMP - select MMU_GATHER_RCU_TABLE_FREE if PARAVIRT + select MMU_GATHER_RCU_TABLE_FREE select MMU_GATHER_MERGE_VMAS select HAVE_POSIX_CPU_TIMERS_TASK_WORK select HAVE_REGS_AND_STACK_ACCESS_APIarch/x86/kernel/paravirt.c +0
-5@@ -59,11 +59,6 @@ void __init native_pv_lock_init(void)static_branch_enable (&virt_spin_lock_key); } -static void native_tlb_remove_table (struct mmu_gather *tlb,void *table) -{ -tlb_remove_page (tlb, table); -} -struct static_key paravirt_steal_enabled;struct static_key paravirt_steal_rq_enabled;
arch/x86/kernel/paravirt.c struct paravirt_patch_template pv_ops = { · line 192
Quoted 8 lines
arch/x86/kernel/paravirt.c +1
-1@@ -191,7 +186,7 @@ struct paravirt_patch_template pv_ops = { .mmu.flush_tlb_kernel = native_flush_tlb_global, .mmu.flush_tlb_one_user = native_flush_tlb_one_user, .mmu.flush_tlb_multi = native_flush_tlb_multi, - .mmu.tlb_remove_table = native_tlb_remove_table, + .mmu.tlb_remove_table = tlb_remove_table, .mmu.exit_mmap = paravirt_nop, .mmu.notify_page_enc_status_changed = paravirt_nop,
It look like this patch series is not based on the latest linux-next.
In addition to the above case, maybe the paravirt_tlb_remove_table() in arch/x86/mm/pgtable.c also needs to be changed to tlb_remove_table()?
Thanks,
Qi
Re: [PATCH 01/11] x86/mm: make MMU_GATHER_RCU_TABLE_FREE unconditional
Quoted 16 lines · Qi Zheng
Very happy to see this change!
Quoted 10 lines
arch/x86/kernel/paravirt.c +0
-0@@ -191,7 +186,7 @@ struct paravirt_patch_template pv_ops = { .mmu.flush_tlb_kernel = native_flush_tlb_global,.mmu.flush_tlb_one_user =
native_flush_tlb_one_user,
.mmu.flush_tlb_multi =native_flush_tlb_multi,arch/x86/kernel/paravirt.c +1
-1- .mmu.tlb_remove_table = native_tlb_remove_table, + .mmu.tlb_remove_table = tlb_remove_table,
.mmu.exit_mmap =paravirt_nop,
.mmu.notify_page_enc_status_changed =paravirt_nop,It look like this patch series is not based on the latest linux-next.
That is correct. I based this on tip.git x86/mm, since that seems like the most likely destination for this code.
Quoted 3 lines
In addition to the above case, maybe the
paravirt_tlb_remove_table()inarch/x86/mm/pgtable.calso needs to be changed totlb_remove_table()?
I'll get that in the next version.
Thank you for reviewing the patch!
Signature · 1 line
All Rights Reversed.
Peter Zijlstra
maintainerRe: [PATCH 01/11] x86/mm: make MMU_GATHER_RCU_TABLE_FREE unconditional
In reply to Rik van RielQuoted 5 lines · Rik van Riel
Currently x86 usse
CONFIG_MMU_GATHER_TABLE_FREEwhen using paravirt, and not when running on bare metal.There is no real good reason to do things differently for each setup. Make them all the same.
More importantly, the changes you're proposing very much rely on this. Without TLBi IPIs nothing serializes GUP-fast vs TLBi and this RCU-ish table free scheme is required.
Quoted 48 lines
Signed-off-by Rik van Riel <[email protected]>Suggested-by Peter Zijlstra <[email protected]>--- arch/x86/Kconfig | 2 +- arch/x86/kernel/paravirt.c | 7 +------ 2 files changed, 2 insertions(+), 7 deletions(-)arch/x86/Kconfig +1
-1@@ -274,7 +274,7 @@ config X86 select HAVE_PCI select HAVE_PERF_REGS select HAVE_PERF_USER_STACK_DUMP - select MMU_GATHER_RCU_TABLE_FREE if PARAVIRT + select MMU_GATHER_RCU_TABLE_FREE select MMU_GATHER_MERGE_VMAS select HAVE_POSIX_CPU_TIMERS_TASK_WORK select HAVE_REGS_AND_STACK_ACCESS_APIarch/x86/kernel/paravirt.c +1
-6@@ -59,11 +59,6 @@ void __init native_pv_lock_init(void)static_branch_enable (&virt_spin_lock_key); } -static void native_tlb_remove_table (struct mmu_gather *tlb,void *table) -{ -tlb_remove_page (tlb, table); -} -struct static_key paravirt_steal_enabled;struct static_key paravirt_steal_rq_enabled;@@ -191,7 +186,7 @@ struct paravirt_patch_template pv_ops = { .mmu.flush_tlb_kernel = native_flush_tlb_global, .mmu.flush_tlb_one_user = native_flush_tlb_one_user, .mmu.flush_tlb_multi = native_flush_tlb_multi, - .mmu.tlb_remove_table = native_tlb_remove_table, + .mmu.tlb_remove_table = tlb_remove_table, .mmu.exit_mmap = paravirt_nop, .mmu.notify_page_enc_status_changed = paravirt_nop,Signature · 1 line
2.47.1
PATCH v2 2/11 x86/mm: add X86_FEATURE_INVLPGB definition.
In reply to Rik van RielAdd the INVPLGB CPUID definition, allowing the kernel to recognize whether the CPU supports the INVLPGB instruction.
--- arch/x86/include/asm/cpufeatures.h | 1 + 1 file changed, 1 insertion(+)
arch/x86/include/asm/cpufeatures.h +1 -0
@@ -338,6 +338,7 @@#define X86_FEATURE_CLZERO (13*32+ 0)/* "clzero" CLZERO instruction */ #define X86_FEATURE_IRPERF (13*32+ 1)/* "irperf" Instructions Retired Count */ #define X86_FEATURE_XSAVEERPTR (13*32+ 2)/* "xsaveerptr" Always save/restore FP error pointers */ +#define X86_FEATURE_INVLPGB (13*32+ 3)/* "invlpgb" INVLPGB instruction */ #define X86_FEATURE_RDPRU (13*32+ 4)/* "rdpru" Read processor register at user level */ #define X86_FEATURE_WBNOINVD (13*32+ 9)/* "wbnoinvd" WBNOINVD instruction */ #define X86_FEATURE_AMD_IBPB (13*32+12)/* Indirect Branch Prediction Barrier */
Signature · 1 line
2.47.1
PATCH v2 3/11 x86/mm: get INVLPGB count max from CPUID
In reply to Rik van RielThe CPU advertises the maximum number of pages that can be shot down with one INVLPGB instruction in the CPUID data.
Save that information for later use.
--- arch/x86/include/asm/tlbflush.h | 1 + arch/x86/kernel/cpu/amd.c | 8 ++++++++ arch/x86/kernel/setup.c | 4 ++++ 3 files changed, 13 insertions(+)
arch/x86/include/asm/tlbflush.h +1 -0
@@ -182,6 +182,7 @@ static inline void cr4_init_shadow(void)extern unsigned long mmu_cr4_features;extern u32 *trampoline_cr4_features; +extern u16 invlpgb_count_max;extern void initialize_tlbstate_and_flush (void );
arch/x86/kernel/cpu/amd.c +8 -0
@@ -1135,6 +1135,14 @@ static void cpu_detect_tlb_amd(struct cpuinfo_x86 *c) tlb_lli_2m[ENTRIES ] = eax & mask; tlb_lli_4m[ENTRIES ] = tlb_lli_2m[ENTRIES ] >>1 ; + +if (c->extended_cpuid_level <0x80000008 ) +return ; + +cpuid (0x80000008 , &eax, &ebx, &ecx, &edx); + +/* Max number of pages INVLPGB can invalidate in one shot */ + invlpgb_count_max = (edx &0xffff ) +1 ; }static const struct cpu_dev amd_cpu_dev = {
arch/x86/kernel/setup.c +4 -0
@@ -138,6 +138,10 @@ __visible unsigned long mmu_cr4_features __ro_after_init;__visible unsigned long mmu_cr4_features__ro_after_init =X86_CR4_PAE ;#endif +#ifdef CONFIG_CPU_SUP_AMD +u16 invlpgb_count_max; +#endif +#ifdef CONFIG_IMA static phys_addr_t ima_kexec_buffer_phys;static size_t ima_kexec_buffer_size;
Signature · 1 line
2.47.1
Re: [PATCH 03/11] x86/mm: get INVLPGB count max from CPUID
arch/x86/kernel/setup.c __visible unsigned long mmu_cr4_features __ro_after_init; · line 142
Quoted 12 lines · Rik van Riel
arch/x86/kernel/setup.c +2
-0@@ -138,6 +138,10 @@ __visible unsigned long mmu_cr4_features __ro_after_init;__visible unsigned long mmu_cr4_features__ro_after_init =X86_CR4_PAE ;#endif +#ifdef CONFIG_CPU_SUP_AMD +u16 invlpgb_count_max;
Any reason to mark it also as __ro_after_init?
PATCH v2 4/11 x86/mm: add INVLPGB support code
In reply to Rik van RielAdd invlpgb.h with the helper functions and definitions needed to use broadcast TLB invalidation on AMD EPYC 3 and newer CPUs.
--- arch/x86/include/asm/invlpgb.h | 93 +++++++++++++++++++++++++++++++++ arch/x86/include/asm/tlbflush.h | 1 + 2 files changed, 94 insertions(+) create mode 100644 arch/x86/include/asm/invlpgb.h
arch/x86/include/asm/invlpgb.h new file +93 -0
@@ -0,0 +1,93 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ASM_X86_INVLPGB +#define _ASM_X86_INVLPGB + +#include <vdso/bits.h> + +/* +* INVLPGB does broadcast TLB invalidation across all the CPUs in the system. +* +* The INVLPGB instruction is weakly ordered, and a batch of invalidations can +* be done in a parallel fashion. +* +* TLBSYNC is used to ensure that pending INVLPGB invalidations initiated from +* this CPU have completed. +*/ +static inline void __invlpgb (unsigned long asid,unsigned long pcid,unsigned long addr, +int extra_count,bool pmd_stride,unsigned long flags) +{ +u64 rax = addr | flags; +u32 ecx = (pmd_stride <<31 ) | extra_count; +u32 edx = (pcid <<16 ) | asid; + + asmvolatile ("invlpgb" : :"a" (rax),"c" (ecx),"d" (edx)); +} + +/* +* INVLPGB can be targeted by virtual address, PCID, ASID, or any combination +* of the three. For example: +* - INVLPGB_VA | INVLPGB_INCLUDE_GLOBAL: invalidate all TLB entries at the address +* - INVLPGB_PCID: invalidate all TLB entries matching the PCID +* +* The first can be used to invalidate (kernel) mappings at a particular +* address across all processes. +* +* The latter invalidates all TLB entries matching a PCID. +*/ +#define INVLPGB_VA BIT(0) +#define INVLPGB_PCID BIT(1) +#define INVLPGB_ASID BIT(2) +#define INVLPGB_INCLUDE_GLOBAL BIT(3) +#define INVLPGB_FINAL_ONLY BIT(4) +#define INVLPGB_INCLUDE_NESTED BIT(5) + +/* Flush all mappings for a given pcid and addr, not including globals. */ +static inline void invlpgb_flush_user (unsigned long pcid, +unsigned long addr) +{ +__invlpgb (0 , pcid, addr,0 ,0 ,INVLPGB_PCID |INVLPGB_VA ); +} + +static inline void invlpgb_flush_user_nr (unsigned long pcid,unsigned long addr, +int nr,bool pmd_stride) +{ +__invlpgb (0 , pcid, addr, nr -1 , pmd_stride,INVLPGB_PCID |INVLPGB_VA ); +} + +/* Flush all mappings for a given ASID, not including globals. */ +static inline void invlpgb_flush_single_asid (unsigned long asid) +{ +__invlpgb (asid,0 ,0 ,0 ,0 ,INVLPGB_ASID ); +} + +/* Flush all mappings for a given PCID, not including globals. */ +static inline void invlpgb_flush_single_pcid (unsigned long pcid) +{ +__invlpgb (0 , pcid,0 ,0 ,0 ,INVLPGB_PCID ); +} + +/* Flush all mappings, including globals, for all PCIDs. */ +static inline void invlpgb_flush_all (void ) +{ +__invlpgb (0 ,0 ,0 ,0 ,0 ,INVLPGB_INCLUDE_GLOBAL ); +} + +/* Flush addr, including globals, for all PCIDs. */ +static inline void invlpgb_flush_addr (unsigned long addr,int nr) +{ +__invlpgb (0 ,0 , addr, nr -1 ,0 ,INVLPGB_INCLUDE_GLOBAL ); +} + +/* Flush all mappings for all PCIDs except globals. */ +static inline void invlpgb_flush_all_nonglobals (void ) +{ +__invlpgb (0 ,0 ,0 ,0 ,0 ,0 ); +} + +/* Wait for INVLPGB originated by this CPU to complete. */ +static inline void tlbsync (void ) +{ + asmvolatile ("tlbsync" ); +} + +#endif /* _ASM_X86_INVLPGB */
arch/x86/include/asm/tlbflush.h +1 -0
@@ -10,6 +10,7 @@#include <asm/cpufeature.h> #include <asm/special_insns.h> #include <asm/smp.h> +#include <asm/invlpgb.h> #include <asm/invpcid.h> #include <asm/pti.h> #include <asm/processor-flags.h>
Signature · 1 line
2.47.1
PATCH v2 5/11 x86/mm: use INVLPGB for kernel TLB flushes
In reply to Rik van RielUse broadcast TLB invalidation for kernel addresses when available.
This stops us from having to send IPIs for kernel TLB flushes.
--- arch/x86/mm/tlb.c | 31 +++++++++++++++++++++++++++++++ 1 file changed, 31 insertions(+)
arch/x86/mm/tlb.c +31 -0
@@ -1077,6 +1077,32 @@ void flush_tlb_all(void)on_each_cpu (do_flush_tlb_all,NULL ,1 ); } +static void broadcast_kernel_range_flush (unsigned long start,unsigned long end) +{ +unsigned long addr; +unsigned long maxnr = invlpgb_count_max; +unsigned long threshold = tlb_single_page_flush_ceiling * maxnr; + +/* +* TLBSYNC only waits for flushes originating on the same CPU. +* Disabling migration allows us to wait on all flushes. +*/ +guard (preempt)(); + +if (end ==TLB_FLUSH_ALL || + (end - start) > threshold <<PAGE_SHIFT ) { +invlpgb_flush_all (); + }else { +unsigned long nr; +for (addr = start; addr < end; addr += nr <<PAGE_SHIFT ) { + nr =min ((end - addr) >>PAGE_SHIFT , maxnr); +invlpgb_flush_addr (addr, nr); + } + } + +tlbsync (); +} +static void do_kernel_range_flush (void *info) {struct flush_tlb_info *f = info;
@@ -1089,6 +1115,11 @@ static void do_kernel_range_flush(void *info)void flush_tlb_kernel_range (unsigned long start,unsigned long end) { +if (cpu_feature_enabled (X86_FEATURE_INVLPGB )) { +broadcast_kernel_range_flush (start, end); +return ; + } +/* Balance as user space task's flush, a bit conservative */ if (end ==TLB_FLUSH_ALL || (end - start) > tlb_single_page_flush_ceiling <<PAGE_SHIFT ) {
Signature · 1 line
2.47.1
PATCH v2 6/11 x86/tlb: use INVLPGB in flush_tlb_all
In reply to Rik van RielThe flush_tlb_all() function is not used a whole lot, but we might as well use broadcast TLB flushing there, too.
--- arch/x86/mm/tlb.c | 6 ++++++ 1 file changed, 6 insertions(+)
arch/x86/mm/tlb.c +6 -0
@@ -1074,6 +1074,12 @@ static void do_flush_tlb_all(void *info)void flush_tlb_all (void ) {count_vm_tlb_event (NR_TLB_REMOTE_FLUSH ); +if (cpu_feature_enabled (X86_FEATURE_INVLPGB )) { +guard (preempt)(); +invlpgb_flush_all (); +tlbsync (); +return ; + }on_each_cpu (do_flush_tlb_all,NULL ,1 ); }
Signature · 1 line
2.47.1
PATCH v2 7/11 x86/mm: use broadcast TLB flushing for page reclaim TLB flushing
In reply to Rik van RielIn the page reclaim code, we only track the CPU(s) where the TLB needs to be flushed, rather than all the individual mappings that may be getting invalidated.
Use broadcast TLB flushing when that is available.
--- arch/x86/mm/tlb.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)
arch/x86/mm/tlb.c +9 -1
@@ -1310,8 +1310,16 @@ EXPORT_SYMBOL_GPL(__flush_tlb_all);void arch_tlbbatch_flush (struct arch_tlbflush_unmap_batch *batch) {struct flush_tlb_info *info; +int cpu; + +if (cpu_feature_enabled (X86_FEATURE_INVLPGB )) { +guard (preempt)(); +invlpgb_flush_all_nonglobals (); +tlbsync (); +return ; + } -int cpu =get_cpu (); + cpu =get_cpu (); info =get_flush_tlb_info (NULL ,0 ,TLB_FLUSH_ALL ,0 , false,TLB_GENERATION_INVALID );
Signature · 1 line
2.47.1
PATCH v2 8/11 x86/mm: enable broadcast TLB invalidation for multi-threaded processes
In reply to Rik van RielUse broadcast TLB invalidation, using the INVPLGB instruction, on AMD EPYC 3 and newer CPUs.
In order to not exhaust PCID space, and keep TLB flushes local for single threaded processes, we only hand out broadcast ASIDs to processes active on 3 or more CPUs, and gradually increase the threshold as broadcast ASID space is depleted.
--- arch/x86/include/asm/mmu.h | 6 + arch/x86/include/asm/mmu_context.h | 12 ++ arch/x86/include/asm/tlbflush.h | 15 ++ arch/x86/mm/tlb.c | 313 ++++++++++++++++++++++++++++- 4 files changed, 337 insertions(+), 9 deletions(-)
arch/x86/include/asm/mmu.h +6 -0
@@ -48,6 +48,12 @@ typedef struct {unsigned long flags;#endif +#ifdef CONFIG_CPU_SUP_AMD +struct list_head broadcast_asid_list; +u16 broadcast_asid; +bool asid_transition; +#endif +#ifdef CONFIG_ADDRESS_MASKING /* Active LAM mode: X86_CR3_LAM_U48 or X86_CR3_LAM_U57 or 0 (disabled) */ unsigned long lam_cr3_mask;
arch/x86/include/asm/mmu_context.h +12 -0
@@ -139,6 +139,8 @@ static inline void mm_reset_untag_mask(struct mm_struct *mm)#define enter_lazy_tlb enter_lazy_tlbextern void enter_lazy_tlb (struct mm_struct *mm,struct task_struct *tsk); +extern void destroy_context_free_broadcast_asid (struct mm_struct *mm); +/* * Init a new mm. Used on mm copies, like at fork() * and on mm's that are brand-new, like at execve().
@@ -161,6 +163,13 @@ static inline int init_new_context(struct task_struct *tsk, mm->context.execute_only_pkey = -1; }#endif + +#ifdef CONFIG_CPU_SUP_AMD +INIT_LIST_HEAD (&mm->context.broadcast_asid_list); + mm->context.broadcast_asid =0 ; + mm->context.asid_transition = false; +#endif +mm_reset_untag_mask (mm);init_new_context_ldt (mm);return 0 ;
@@ -170,6 +179,9 @@ static inline int init_new_context(struct task_struct *tsk,static inline void destroy_context (struct mm_struct *mm) {destroy_context_ldt (mm); +#ifdef CONFIG_CPU_SUP_AMD +destroy_context_free_broadcast_asid (mm); +#endif } extern void switch_mm(struct mm_struct *prev, struct mm_struct *next,
arch/x86/include/asm/tlbflush.h +15 -0
@@ -65,6 +65,21 @@ static inline void cr4_clear_bits(unsigned long mask)*/ #define TLB_NR_DYN_ASIDS 6 +#ifdef CONFIG_CPU_SUP_AMD +#define is_dyn_asid (asid) (asid) < TLB_NR_DYN_ASIDS +#define is_broadcast_asid (asid) (asid) >= TLB_NR_DYN_ASIDS +#define in_asid_transition (info) (info->mm && info->mm->context.asid_transition) +#else +#define is_dyn_asid (asid) true +#define is_broadcast_asid (asid) false +#define in_asid_transition (info) false + +inline bool needs_broadcast_asid_reload (struct mm_struct *next,u16 prev_asid) +{ +return false; +} +#endif +struct tlb_context {u64 ctx_id;u64 tlb_gen;
arch/x86/mm/tlb.c +304 -9
400 lines not on this page · raw message
Signature · 1 line
2.47.1
Re: [PATCH 08/11] x86/mm: enable broadcast TLB invalidation for multi-threaded processes
Quoted 7 lines · Rik van Riel
arch/x86/mm/tlb.c +7
-0+static int mm_active_cpus (struct mm_struct *mm) +{ +int count =0 ; +int cpu; + +for_each_cpu (cpu,mm_cpumask (mm)) { +/* Skip the CPUs that aren't really running this process. */
arch/x86/mm/tlb.c static void choose_new_asid(struct mm_struct *next, u64 next_tlb_gen, · line 367
Quoted 10 lines
arch/x86/mm/tlb.c +10
-0+if (per_cpu (cpu_tlbstate.loaded_mm, cpu) != mm) +continue ; + +if (per_cpu (cpu_tlbstate_shared.is_lazy, cpu)) +continue ; + + count++; + } +return count; +}
Since you are only interested in checking whether the number of “mm active CPUs" is greater than a certain threshold, don’t you want to add some checks for early termination? This can allow to avoid cachelines of cpu_tlbstate traversing back and forth.
For instance, by running cpumask_weight() first, if the weight is lower than the threshold, no need to loop. Similarly, if inside the loop the threshold has already been crossed, no need for more iterations.
Re: [PATCH 08/11] x86/mm: enable broadcast TLB invalidation for multi-threaded processes
In reply to Rik van Rielarch/x86/mm/tlb.c void flush_tlb_mm_range(struct mm_struct *mm, unsigned long start, · line 1346
Quoted 12 lines · Rik van Riel
arch/x86/mm/tlb.c +2
-1@@ -1049,9 +1341,12 @@ void flush_tlb_mm_range(struct mm_struct *mm, unsigned long start,* a local TLB flush is needed. Optimize this use-case by calling * flush_tlb_func_local() directly in this case. */ -if (cpumask_any_but (mm_cpumask (mm), cpu) < nr_cpu_ids) { +if (IS_ENABLED (CONFIG_CPU_SUP_AMD ) && mm->context.broadcast_asid) { +broadcast_tlb_flush (info);
I think broadcast_asid is defined within an ifdef, so the IS_ENABLED() here would not save you from having to use ifdef.
PATCH v2 9/11 x86,tlb: do targeted broadcast flushing from tlbbatch code
In reply to Rik van RielInstead of doing a system-wide TLB flush from arch_tlbbatch_flush, queue up asynchronous, targeted flushes from arch_tlbbatch_add_pending.
This also allows us to avoid adding the CPUs of processes using broadcast flushing to the batch->cpumask, and will hopefully further reduce TLB flushing from the reclaim and compaction paths.
--- arch/x86/include/asm/tlbbatch.h | 1 + arch/x86/include/asm/tlbflush.h | 12 +++------ arch/x86/mm/tlb.c | 48 ++++++++++++++++++++++++++------- 3 files changed, 42 insertions(+), 19 deletions(-)
arch/x86/include/asm/tlbbatch.h +1 -0
@@ -10,6 +10,7 @@ struct arch_tlbflush_unmap_batch {* the PFNs being flushed.. */ struct cpumask cpumask; + bool used_invlpgb; };#endif /* _ARCH_X86_TLBBATCH_H */
arch/x86/include/asm/tlbflush.h +3 -9
@@ -295,21 +295,15 @@ static inline u64 inc_mm_tlb_gen(struct mm_struct *mm)return atomic64_inc_return (&mm->context.tlb_gen); } -static inline void arch_tlbbatch_add_pending (struct arch_tlbflush_unmap_batch *batch, -struct mm_struct *mm, -unsigned long uaddr) -{ -inc_mm_tlb_gen (mm); -cpumask_or (&batch->cpumask, &batch->cpumask,mm_cpumask (mm)); -mmu_notifier_arch_invalidate_secondary_tlbs (mm,0 ,-1UL ); -} -static inline void arch_flush_tlb_batched_pending (struct mm_struct *mm) {flush_tlb_mm (mm); }extern void arch_tlbbatch_flush (struct arch_tlbflush_unmap_batch *batch); +extern void arch_tlbbatch_add_pending (struct arch_tlbflush_unmap_batch *batch, +struct mm_struct *mm, +unsigned long uaddr); static inline bool pte_flags_need_flush(unsigned long oldflags, unsigned long newflags,
arch/x86/mm/tlb.c +38 -10
66 lines not on this page · raw message
Signature · 1 line
2.47.1
PATCH v2 10/11 x86/mm: enable AMD translation cache extensions
In reply to Rik van RielWith AMD TCE (translation cache extensions) only the intermediate mappings that cover the address range zapped by INVLPG / INVLPGB get invalidated, rather than all intermediate mappings getting zapped at every TLB invalidation.
This can help reduce the TLB miss rate, by keeping more intermediate mappings in the cache.
From the AMD manual:
Translation Cache Extension (TCE) Bit. Bit 15, read/write. Setting this bit to 1 changes how the INVLPG, INVLPGB, and INVPCID instructions operate on TLB entries. When this bit is 0, these instructions remove the target PTE from the TLB as well as all upper-level table entries that are cached in the TLB, whether or not they are associated with the target PTE. When this bit is set, these instructions will remove the target PTE and only those upper-level entries that lead to the target PTE in the page table hierarchy, leaving unrelated upper-level entries intact.
--- arch/x86/kernel/cpu/amd.c | 8 ++++++++ arch/x86/mm/tlb.c | 10 +++++++--- 2 files changed, 15 insertions(+), 3 deletions(-)
arch/x86/kernel/cpu/amd.c +8 -0
@@ -1143,6 +1143,14 @@ static void cpu_detect_tlb_amd(struct cpuinfo_x86 *c)/* Max number of pages INVLPGB can invalidate in one shot */ invlpgb_count_max = (edx &0xffff ) +1 ; + +/* If supported, enable translation cache extensions (TCE) */ +cpuid (0x80000001 , &eax, &ebx, &ecx, &edx); +if (ecx &BIT (17 )) { +u64 msr =native_read_msr (MSR_EFER );; + msr |=BIT (15 ); +wrmsrl (MSR_EFER , msr); + } }static const struct cpu_dev amd_cpu_dev = {
arch/x86/mm/tlb.c +7 -3
@@ -480,7 +480,7 @@ static void broadcast_tlb_flush(struct flush_tlb_info *info)if (info->stride_shift >PMD_SHIFT ) maxnr =1 ; -if (info->end ==TLB_FLUSH_ALL ) { +if (info->end ==TLB_FLUSH_ALL || info->freed_tables) {invlpgb_flush_single_pcid (kern_pcid (asid));/* Do any CPUs supporting INVLPGB need PTI? */ if (static_cpu_has (X86_FEATURE_PTI ))
@@ -1113,7 +1113,7 @@ static void flush_tlb_func(void *info)* * The only question is whether to do a full or partial flush. * -* We do a partial flush if requested and two extra conditions +* We do a partial flush if requested and three extra conditions * are met: * * 1. f->new_tlb_gen == local_tlb_gen + 1. We have an invariant that
@@ -1140,10 +1140,14 @@ static void flush_tlb_func(void *info)* date. By doing a full flush instead, we can increase * local_tlb_gen all the way to mm_tlb_gen and we can probably * avoid another flush in the very near future. +* +* 3. No page tables were freed. If page tables were freed, a full +* flush ensures intermediate translations in the TLB get flushed. */ if (f->end !=TLB_FLUSH_ALL && f->new_tlb_gen == local_tlb_gen +1 && - f->new_tlb_gen == mm_tlb_gen) { + f->new_tlb_gen == mm_tlb_gen && + !f->freed_tables) {/* Partial flush */ unsigned long addr = f->start;
Signature · 1 line
2.47.1
PATCH v2 11/11 x86/mm: only invalidate final translations with INVLPGB
In reply to Rik van RielUse the INVLPGB_FINAL_ONLY flag when invalidating mappings with INVPLGB. This way only leaf mappings get removed from the TLB, leaving intermediate translations cached.
On the (rare) occasions where we free page tables we do a full flush, ensuring intermediate translations get flushed from the TLB.
--- arch/x86/include/asm/invlpgb.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/include/asm/invlpgb.h +1 -1
@@ -51,7 +51,7 @@ static inline void invlpgb_flush_user(unsigned long pcid,static inline void invlpgb_flush_user_nr (unsigned long pcid,unsigned long addr,int nr,bool pmd_stride) { -__invlpgb (0 , pcid, addr, nr -1 , pmd_stride,INVLPGB_PCID |INVLPGB_VA ); +__invlpgb (0 , pcid, addr, nr -1 , pmd_stride,INVLPGB_PCID |INVLPGB_VA |INVLPGB_FINAL_ONLY ); }/* Flush all mappings for a given ASID, not including globals. */
Signature · 1 line
2.47.1
From: [email protected] <[email protected]> Sent: Sunday, December 22, 2024 6:55 PM
Quoted 8 lines
Add support for broadcast TLB invalidation using AMD's INVLPGB instruction.
This allows the kernel to invalidate TLB entries on remote CPUs without needing to send IPIs, without having to wait for remote CPUs to handle those interrupts, and with less interruption to what was running on those CPUs.
Quoted 4 lines
Because x86 PCID space is limited, and there are some very large systems out there, broadcast TLB invalidation is only used for processes that are active on 3 or more CPUs, with the threshold being gradually increased the more the PCID space gets exhausted.
Rik --
What is this patch set's expectation about INVLPGB and TLBSYNC availability and usage in a VM? I see that INVLPGB and TLBYSNC behavior in a VM is spec'ed in the AMD Programmer's Manual, but I wonder about their impact in a multi-tenant host like in a public cloud environment. And given what this patch set does in assigning global ASIDs, should X86_FEATURE_INVLPGB be disabled if running in a VM where the hypervisor for whatever reason has enabled INVLPGB/TLBSYNC in its VMs?
My knowledge of the details here is pretty limited, so my question may just reflect my ignorance. But it would be good for the code comments and/or commit messages to include explicit statements about what is expected in a VM.
Michael
Quoted 25 lines
Combined with the removal of unnecessary
lru_add_draincalls (see https://lkml.org/lkml/2024/12/19/1388) this results in a nice performance boost for the will-it-scaletlb_flush2_threadstest on an AMD Milan system with 36 cores:- vanilla kernel: 527k loops/second - lru_add_drain removal: 731k loops/second - only INVLPGB: 527k loops/second-
lru_add_drain+ INVLPGB: 1157k loops/secondProfiling with only the INVLPGB changes showed while TLB invalidation went down from 40% of the total CPU time to only around 4% of CPU time, the contention simply moved to the LRU lock.
Fixing both at the same time about doubles the
number of iterations per second from this case.v2:
- Apply suggestions by Peter and Borislav (thank you!)
- Fix bug inarch_tlbbatch_flush, where we need to do both the TLBSYNC, and flush the CPUs that are in the cpumask.
- Some updates to comments and changelogs based on questions.
Quoted 21 lines · Michael Kelley
From: [email protected] <[email protected]> Sent: Sunday, December 22, 2024 6:55 PM
Quoted 11 lines
Add support for broadcast TLB invalidation using AMD's INVLPGB instruction.
This allows the kernel to invalidate TLB entries on remote CPUs without
needing to send IPIs, without having to wait for remote CPUs to handle
those interrupts, and with less interruption to what was running on those CPUs.Quoted 4 lines
Because x86 PCID space is limited, and there are some very large systems out there, broadcast TLB invalidation is only used for processes that are active on 3 or more CPUs, with the threshold being gradually increased the more the PCID space gets exhausted.
Rik --
Quoted 9 lines
What is this patch set's expectation about INVLPGB and TLBSYNC availability and usage in a VM? I see that INVLPGB and TLBYSNC behavior in a VM is spec'ed in the AMD Programmer's Manual, but I wonder about their impact in a multi-tenant host like in a public cloud environment. And given what this patch set does in assigning global ASIDs, should
X86_FEATURE_INVLPGBbe disabled if running in a VM where the hypervisor for whatever reason has enabled INVLPGB/TLBSYNC in its VMs?
This patch series enables bare metal INVLPGB functionality.
Virtual machines should probably not expose the INVPLGB CPUID feature bit to guests, since virtual machine invalidation seems to work differently than bare metal invalidation.
For one, the ASID seems to actually mean something in SVM context, while trying to use the ASID in bare metal blows up :)
Signature · 1 line
All Rights Reversed.
Quoted 33 lines · Rik van Riel
Quoted 21 lines · Michael Kelley
From: [email protected] <[email protected]> Sent: Sunday, December 22, 2024 6:55 PM
Quoted 11 lines
Add support for broadcast TLB invalidation using AMD's INVLPGB instruction.
This allows the kernel to invalidate TLB entries on remote CPUs without
needing to send IPIs, without having to wait for remote CPUs to handle
those interrupts, and with less interruption to what was running on those CPUs.Quoted 4 lines
Because x86 PCID space is limited, and there are some very large systems out there, broadcast TLB invalidation is only used for processes that are active on 3 or more CPUs, with the threshold being gradually increased the more the PCID space gets exhausted.
Rik --
Quoted 9 lines
What is this patch set's expectation about INVLPGB and TLBSYNC availability and usage in a VM? I see that INVLPGB and TLBYSNC behavior in a VM is spec'ed in the AMD Programmer's Manual, but I wonder about their impact in a multi-tenant host like in a public cloud environment. And given what this patch set does in assigning global ASIDs, should
X86_FEATURE_INVLPGBbe disabled if running in a VM where the hypervisor for whatever reason has enabled INVLPGB/TLBSYNC in its VMs?This patch series enables bare metal INVLPGB functionality.
Quoted 8 lines
Virtual machines should probably not expose the INVPLGB CPUID feature bit to guests, since virtual machine invalidation seems to work differently than bare metal invalidation.
For one, the ASID seems to actually mean something in SVM context, while trying to use the ASID in bare metal blows up :)
Note that global ASIDs (relative to VMs) are different from the broadcast ASIDs being used here. IIUC, the broadcast ASIDs here get translated to a PCID value (kern_pcid(asid) or user_pcid(asid) in patch #9).
Thanks,
Tom