AMD broadcast TLB invalidation
Subsystems (7)
v11 cover AMD broadcast TLB invalidation
Add support for broadcast TLB invalidation using AMD's INVLPGB instruction.
This allows the kernel to invalidate TLB entries on remote CPUs without needing to send IPIs, without having to wait for remote CPUs to handle those interrupts, and with less interruption to what was running on those CPUs.
Because x86 PCID space is limited, and there are some very large systems out there, broadcast TLB invalidation is only used for processes that are active on 3 or more CPUs, with the threshold being gradually increased the more the PCID space gets exhausted.
Combined with the removal of unnecessary lru_add_drain calls (see https://lkml.org/lkml/2024/12/19/1388) this results in a nice performance boost for the will-it-scale tlb_flush2_threads test on an AMD Milan system with 36 cores:
- vanilla kernel: 527k loops/second - lru_add_drain removal: 731k loops/second - only INVLPGB: 527k loops/second
- lru_add_drain + INVLPGB: 1157k loops/second
Profiling with only the INVLPGB changes showed while TLB invalidation went down from 40% of the total CPU time to only around 4% of CPU time, the contention simply moved to the LRU lock.
Fixing both at the same time about doubles the
number of iterations per second from this case.
Some numbers closer to real world performance
can be found at Phoronix, thanks to Michael:
https://www.phoronix.com/news/AMD-INVLPGB-Linux-Benefits
My current plan is to implement support for Intel's RAR (Remote Action Request) TLB flushing in a follow-up series, after this thing has been merged into -tip. Making things any larger would just be unwieldy for reviewers.
v11:
- resolve conflict with CONFIG_PT_RECLAIM code
- a few more cleanups (Peter, Brendan, Nadav)
v10:
- simplify partial pages with min(nr, 1) in the invlpgb loop (Peter)
- document x86 paravirt, AMD invlpgb, and ARM64 flush without IPI (Brendan)
- remove IS_ENABLED(CONFIG_X86_BROADCAST_TLB_FLUSH) (Brendan)
- various cleanups (Brendan)
v9:
- print warning when start or end address was rounded (Peter)
- in the reclaim code, tlbsync at context switch time (Peter)
- fix !CONFIG_CPU_SUP_AMD compile error in arch_tlbbatch_add_pending (Jan)
v8:
- round start & end to handle non-page-aligned callers (Steven & Jan)
- fix up changelog & add tested-by tags (Manali)
v7:
- a few small code cleanups (Nadav)
- fix spurious VM_WARN_ON_ONCE in mm_global_asid
- code simplifications & better barriers (Peter & Dave)
v6:
- fix info->end check in flush_tlb_kernel_range (Michael)
- disable broadcast TLB flushing on 32 bit x86
v5:
- use byte assembly for compatibility with older toolchains (Borislav, Michael)
- ensure a panic on an invalid number of extra pages (Dave, Tom)
- add cant_migrate() assertion to tlbsync (Jann)
- a bunch more cleanups (Nadav)
- key TCE enabling off X86_FEATURE_TCE (Andrew)
- fix a race between reclaim and ASID transition (Jann)
v4:
- Use only bitmaps to track free global ASIDs (Nadav)
- Improved AMD initialization (Borislav & Tom)
- Various naming and documentation improvements (Peter, Nadav, Tom, Dave)
- Fixes for subtle race conditions (Jann)
v3:
- Remove paravirt tlb_remove_table call (thank you Qi Zheng)
- More suggested cleanups and changelog fixes by Peter and Nadav
v2:
- Apply suggestions by Peter and Borislav (thank you!)
- Fix bug in arch_tlbbatch_flush, where we need to do both
the TLBSYNC, and flush the CPUs that are in the cpumask.
- Some updates to comments and changelogs based on questions.
PATCH v11 1/12 x86/mm: make MMU_GATHER_RCU_TABLE_FREE unconditional
Currently x86 uses CONFIG_MMU_GATHER_TABLE_FREE when using paravirt, and not when running on bare metal.
There is no real good reason to do things differently for each setup. Make them all the same.
Currently get_user_pages_fast synchronizes against page table freeing in two different ways:
- on bare metal, by blocking IRQs, which block TLB flush IPIs
- on paravirt, with MMU_GATHER_RCU_TABLE_FREE
This is done because some paravirt TLB flush implementations handle the TLB flush in the hypervisor, and will do the flush even when the target CPU has interrupts disabled.
Always handle page table freeing with MMU_GATHER_RCU_TABLE_FREE. Using RCU synchronization between page table freeing and get_user_pages_fast() allows bare metal to also do TLB flushing while interrupts are disabled.
Various places in the mm do still block IRQs or disable preemption as an implicit way to block RCU frees.
That makes it safe to use INVLPGB on AMD CPUs.
--- arch/x86/Kconfig | 2 +- arch/x86/kernel/paravirt.c | 17 +---------------- arch/x86/mm/pgtable.c | 27 ++++----------------------- 3 files changed, 6 insertions(+), 40 deletions(-)
arch/x86/Kconfig +1 -1
@@ -278,7 +278,7 @@ config X86 select HAVE_PCI select HAVE_PERF_REGS select HAVE_PERF_USER_STACK_DUMP - select MMU_GATHER_RCU_TABLE_FREE if PARAVIRT + select MMU_GATHER_RCU_TABLE_FREE select MMU_GATHER_MERGE_VMAS select HAVE_POSIX_CPU_TIMERS_TASK_WORK select HAVE_REGS_AND_STACK_ACCESS_API
arch/x86/kernel/paravirt.c +1 -16
@@ -59,21 +59,6 @@ void __init native_pv_lock_init(void)static_branch_enable (&virt_spin_lock_key); } -#ifndef CONFIG_PT_RECLAIM -static void native_tlb_remove_table (struct mmu_gather *tlb,void *table) -{ -struct ptdesc *ptdesc = (struct ptdesc *)table; - -pagetable_dtor (ptdesc); -tlb_remove_page (tlb,ptdesc_page (ptdesc)); -} -#else -static void native_tlb_remove_table (struct mmu_gather *tlb,void *table) -{ -tlb_remove_table (tlb, table); -} -#endif -struct static_key paravirt_steal_enabled;struct static_key paravirt_steal_rq_enabled;
@@ -195,7 +180,7 @@ struct paravirt_patch_template pv_ops = { .mmu.flush_tlb_kernel = native_flush_tlb_global, .mmu.flush_tlb_one_user = native_flush_tlb_one_user, .mmu.flush_tlb_multi = native_flush_tlb_multi, - .mmu.tlb_remove_table = native_tlb_remove_table, + .mmu.tlb_remove_table = tlb_remove_table, .mmu.exit_mmap = paravirt_nop, .mmu.notify_page_enc_status_changed = paravirt_nop,
arch/x86/mm/pgtable.c +4 -23
@@ -18,25 +18,6 @@ EXPORT_SYMBOL(physical_mask);#define PGTABLE_HIGHMEM 0#endif -#ifndef CONFIG_PARAVIRT -#ifndef CONFIG_PT_RECLAIM -static inline -void paravirt_tlb_remove_table (struct mmu_gather *tlb,void *table) -{ -struct ptdesc *ptdesc = (struct ptdesc *)table; - -pagetable_dtor (ptdesc); -tlb_remove_page (tlb,ptdesc_page (ptdesc)); -} -#else -static inline -void paravirt_tlb_remove_table (struct mmu_gather *tlb,void *table) -{ -tlb_remove_table (tlb, table); -} -#endif /* !CONFIG_PT_RECLAIM */ -#endif /* !CONFIG_PARAVIRT */ -gfp_t __userpte_alloc_gfp =GFP_PGTABLE_USER |PGTABLE_HIGHMEM ;pgtable_t pte_alloc_one (struct mm_struct *mm)
@@ -64,7 +45,7 @@ early_param("userpte", setup_userpte);void ___pte_free_tlb (struct mmu_gather *tlb,struct page *pte) {paravirt_release_pte (page_to_pfn (pte)); -paravirt_tlb_remove_table (tlb,page_ptdesc (pte)); +tlb_remove_table (tlb,page_ptdesc (pte)); }#if CONFIG_PGTABLE_LEVELS >2
@@ -78,21 +59,21 @@ void ___pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd)#ifdef CONFIG_X86_PAE tlb->need_flush_all =1 ;#endif -paravirt_tlb_remove_table (tlb,virt_to_ptdesc (pmd)); +tlb_remove_table (tlb,virt_to_ptdesc (pmd)); }#if CONFIG_PGTABLE_LEVELS >3 void ___pud_free_tlb (struct mmu_gather *tlb,pud_t *pud) {paravirt_release_pud (__pa (pud) >>PAGE_SHIFT ); -paravirt_tlb_remove_table (tlb,virt_to_ptdesc (pud)); +tlb_remove_table (tlb,virt_to_ptdesc (pud)); }#if CONFIG_PGTABLE_LEVELS >4 void ___p4d_free_tlb (struct mmu_gather *tlb,p4d_t *p4d) {paravirt_release_p4d (__pa (p4d) >>PAGE_SHIFT ); -paravirt_tlb_remove_table (tlb,virt_to_ptdesc (p4d)); +tlb_remove_table (tlb,virt_to_ptdesc (p4d)); }#endif /* CONFIG_PGTABLE_LEVELS > 4 */ #endif /* CONFIG_PGTABLE_LEVELS > 3 */
Signature · 1 line
2.47.1
[tip: x86/mm] x86/mm: Make MMU_GATHER_RCU_TABLE_FREE unconditional
The following commit has been merged into the x86/mm branch of tip:
The following commit has been merged into the x86/mm branch of tip:
Commit-ID: a37259732a7dc33047fa1e4f9a338088f452e017 Gitweb: https://git.kernel.org/tip/a37259732a7dc33047fa1e4f9a338088f452e017 Author: Rik van Riel <[email protected]> AuthorDate: Thu, 13 Feb 2025 11:13:52 -05:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Fri, 21 Feb 2025 16:20:11 +01:00
x86/mm: Make MMU_GATHER_RCU_TABLE_FREE unconditional
Currently x86 uses CONFIG_MMU_GATHER_TABLE_FREE when using paravirt, and not when running on bare metal.
There is no real good reason to do things differently for each setup. Make them all the same.
Currently get_user_pages_fast synchronizes against page table freeing in two different ways:
- on bare metal, by blocking IRQs, which block TLB flush IPIs
- on paravirt, with MMU_GATHER_RCU_TABLE_FREE
This is done because some paravirt TLB flush implementations handle the TLB flush in the hypervisor, and will do the flush even when the target CPU has interrupts disabled.
Always handle page table freeing with MMU_GATHER_RCU_TABLE_FREE. Using RCU synchronization between page table freeing and get_user_pages_fast() allows bare metal to also do TLB flushing while interrupts are disabled.
Various places in the mm do still block IRQs or disable preemption as an implicit way to block RCU frees.
That makes it safe to use INVLPGB on AMD CPUs.
--- arch/x86/Kconfig | 2 +- arch/x86/kernel/paravirt.c | 17 +---------------- arch/x86/mm/pgtable.c | 27 ++++----------------------- 3 files changed, 6 insertions(+), 40 deletions(-)
arch/x86/Kconfig +1 -1
@@ -278,7 +278,7 @@ config X86 select HAVE_PCI select HAVE_PERF_REGS select HAVE_PERF_USER_STACK_DUMP - select MMU_GATHER_RCU_TABLE_FREE if PARAVIRT + select MMU_GATHER_RCU_TABLE_FREE select MMU_GATHER_MERGE_VMAS select HAVE_POSIX_CPU_TIMERS_TASK_WORK select HAVE_REGS_AND_STACK_ACCESS_API
arch/x86/kernel/paravirt.c +1 -16
@@ -59,21 +59,6 @@ void __init native_pv_lock_init(void)static_branch_enable (&virt_spin_lock_key); } -#ifndef CONFIG_PT_RECLAIM -static void native_tlb_remove_table (struct mmu_gather *tlb,void *table) -{ -struct ptdesc *ptdesc = (struct ptdesc *)table; - -pagetable_dtor (ptdesc); -tlb_remove_page (tlb,ptdesc_page (ptdesc)); -} -#else -static void native_tlb_remove_table (struct mmu_gather *tlb,void *table) -{ -tlb_remove_table (tlb, table); -} -#endif -struct static_key paravirt_steal_enabled;struct static_key paravirt_steal_rq_enabled;
@@ -195,7 +180,7 @@ struct paravirt_patch_template pv_ops = { .mmu.flush_tlb_kernel = native_flush_tlb_global, .mmu.flush_tlb_one_user = native_flush_tlb_one_user, .mmu.flush_tlb_multi = native_flush_tlb_multi, - .mmu.tlb_remove_table = native_tlb_remove_table, + .mmu.tlb_remove_table = tlb_remove_table, .mmu.exit_mmap = paravirt_nop, .mmu.notify_page_enc_status_changed = paravirt_nop,
arch/x86/mm/pgtable.c +4 -23
@@ -18,25 +18,6 @@ EXPORT_SYMBOL(physical_mask);#define PGTABLE_HIGHMEM 0#endif -#ifndef CONFIG_PARAVIRT -#ifndef CONFIG_PT_RECLAIM -static inline -void paravirt_tlb_remove_table (struct mmu_gather *tlb,void *table) -{ -struct ptdesc *ptdesc = (struct ptdesc *)table; - -pagetable_dtor (ptdesc); -tlb_remove_page (tlb,ptdesc_page (ptdesc)); -} -#else -static inline -void paravirt_tlb_remove_table (struct mmu_gather *tlb,void *table) -{ -tlb_remove_table (tlb, table); -} -#endif /* !CONFIG_PT_RECLAIM */ -#endif /* !CONFIG_PARAVIRT */ -gfp_t __userpte_alloc_gfp =GFP_PGTABLE_USER |PGTABLE_HIGHMEM ;pgtable_t pte_alloc_one (struct mm_struct *mm)
@@ -64,7 +45,7 @@ early_param("userpte", setup_userpte);void ___pte_free_tlb (struct mmu_gather *tlb,struct page *pte) {paravirt_release_pte (page_to_pfn (pte)); -paravirt_tlb_remove_table (tlb,page_ptdesc (pte)); +tlb_remove_table (tlb,page_ptdesc (pte)); }#if CONFIG_PGTABLE_LEVELS >2
@@ -78,21 +59,21 @@ void ___pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd)#ifdef CONFIG_X86_PAE tlb->need_flush_all =1 ;#endif -paravirt_tlb_remove_table (tlb,virt_to_ptdesc (pmd)); +tlb_remove_table (tlb,virt_to_ptdesc (pmd)); }#if CONFIG_PGTABLE_LEVELS >3 void ___pud_free_tlb (struct mmu_gather *tlb,pud_t *pud) {paravirt_release_pud (__pa (pud) >>PAGE_SHIFT ); -paravirt_tlb_remove_table (tlb,virt_to_ptdesc (pud)); +tlb_remove_table (tlb,virt_to_ptdesc (pud)); }#if CONFIG_PGTABLE_LEVELS >4 void ___p4d_free_tlb (struct mmu_gather *tlb,p4d_t *p4d) {paravirt_release_p4d (__pa (p4d) >>PAGE_SHIFT ); -paravirt_tlb_remove_table (tlb,virt_to_ptdesc (p4d)); +tlb_remove_table (tlb,virt_to_ptdesc (p4d)); }#endif /* CONFIG_PGTABLE_LEVELS > 4 */ #endif /* CONFIG_PGTABLE_LEVELS > 3 */
PATCH v11 2/12 x86/mm: remove pv_ops.mmu.tlb_remove_table call
In reply to Rik van RielEvery pv_ops.mmu.tlb_remove_table call ends up calling tlb_remove_table.
Get rid of the indirection by simply calling tlb_remove_table directly, and not going through the paravirt function pointers.
--- arch/x86/hyperv/mmu.c | 1 - arch/x86/include/asm/paravirt.h | 5 ----- arch/x86/include/asm/paravirt_types.h | 2 -- arch/x86/kernel/kvm.c | 1 - arch/x86/kernel/paravirt.c | 1 - arch/x86/xen/mmu_pv.c | 1 - 6 files changed, 11 deletions(-)
arch/x86/hyperv/mmu.c +0 -1
@@ -239,5 +239,4 @@ void hyperv_setup_mmu_ops(void)pr_info ("Using hypercall for remote TLB flush\n" ); pv_ops.mmu.flush_tlb_multi = hyperv_flush_tlb_multi; - pv_ops.mmu.tlb_remove_table = tlb_remove_table; }
arch/x86/include/asm/paravirt.h +0 -5
@@ -91,11 +91,6 @@ static inline void __flush_tlb_multi(const struct cpumask *cpumask,PVOP_VCALL2 (mmu.flush_tlb_multi, cpumask, info); } -static inline void paravirt_tlb_remove_table (struct mmu_gather *tlb,void *table) -{ -PVOP_VCALL2 (mmu.tlb_remove_table, tlb, table); -} -static inline void paravirt_arch_exit_mmap (struct mm_struct *mm) {PVOP_VCALL1 (mmu.exit_mmap, mm);
arch/x86/include/asm/paravirt_types.h +0 -2
@@ -134,8 +134,6 @@ struct pv_mmu_ops {void (*flush_tlb_multi)(const struct cpumask *cpus,const struct flush_tlb_info *info); -void (*tlb_remove_table)(struct mmu_gather *tlb,void *table); -/* Hook for intercepting the destruction of an mm_struct. */ void (*exit_mmap)(struct mm_struct *mm);void (*notify_page_enc_status_changed)(unsigned long pfn,int npages,bool enc);
arch/x86/kernel/kvm.c +0 -1
@@ -838,7 +838,6 @@ static void __init kvm_guest_init(void)#ifdef CONFIG_SMP if (pv_tlb_flush_supported ()) { pv_ops.mmu.flush_tlb_multi = kvm_flush_tlb_multi; - pv_ops.mmu.tlb_remove_table = tlb_remove_table;pr_info ("KVM setup pv remote TLB flush\n" ); }
arch/x86/kernel/paravirt.c +0 -1
@@ -180,7 +180,6 @@ struct paravirt_patch_template pv_ops = { .mmu.flush_tlb_kernel = native_flush_tlb_global, .mmu.flush_tlb_one_user = native_flush_tlb_one_user, .mmu.flush_tlb_multi = native_flush_tlb_multi, - .mmu.tlb_remove_table = tlb_remove_table, .mmu.exit_mmap = paravirt_nop, .mmu.notify_page_enc_status_changed = paravirt_nop,
arch/x86/xen/mmu_pv.c +0 -1
@@ -2141,7 +2141,6 @@ static const typeof(pv_ops) xen_mmu_ops __initconst = { .flush_tlb_kernel = xen_flush_tlb, .flush_tlb_one_user = xen_flush_tlb_one_user, .flush_tlb_multi = xen_flush_tlb_multi, - .tlb_remove_table = tlb_remove_table, .pgd_alloc = xen_pgd_alloc, .pgd_free = xen_pgd_free,
Signature · 1 line
2.47.1
[tip: x86/mm] x86/mm: Remove pv_ops.mmu.tlb_remove_table call
The following commit has been merged into the x86/mm branch of tip:
The following commit has been merged into the x86/mm branch of tip:
Commit-ID: f2c5c21058270167ce23172022da083b62e5ad4c Gitweb: https://git.kernel.org/tip/f2c5c21058270167ce23172022da083b62e5ad4c Author: Rik van Riel <[email protected]> AuthorDate: Thu, 13 Feb 2025 11:13:53 -05:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Fri, 21 Feb 2025 16:20:12 +01:00
x86/mm: Remove pv_ops.mmu.tlb_remove_table call
Every pv_ops.mmu.tlb_remove_table call ends up calling tlb_remove_table.
Get rid of the indirection by simply calling tlb_remove_table directly, and not going through the paravirt function pointers.
--- arch/x86/hyperv/mmu.c | 1 - arch/x86/include/asm/paravirt.h | 5 ----- arch/x86/include/asm/paravirt_types.h | 2 -- arch/x86/kernel/kvm.c | 1 - arch/x86/kernel/paravirt.c | 1 - arch/x86/xen/mmu_pv.c | 1 - 6 files changed, 11 deletions(-)
arch/x86/hyperv/mmu.c +0 -1
@@ -239,5 +239,4 @@ void hyperv_setup_mmu_ops(void)pr_info ("Using hypercall for remote TLB flush\n" ); pv_ops.mmu.flush_tlb_multi = hyperv_flush_tlb_multi; - pv_ops.mmu.tlb_remove_table = tlb_remove_table; }
arch/x86/include/asm/paravirt.h +0 -5
@@ -91,11 +91,6 @@ static inline void __flush_tlb_multi(const struct cpumask *cpumask,PVOP_VCALL2 (mmu.flush_tlb_multi, cpumask, info); } -static inline void paravirt_tlb_remove_table (struct mmu_gather *tlb,void *table) -{ -PVOP_VCALL2 (mmu.tlb_remove_table, tlb, table); -} -static inline void paravirt_arch_exit_mmap (struct mm_struct *mm) {PVOP_VCALL1 (mmu.exit_mmap, mm);
arch/x86/include/asm/paravirt_types.h +0 -2
@@ -134,8 +134,6 @@ struct pv_mmu_ops {void (*flush_tlb_multi)(const struct cpumask *cpus,const struct flush_tlb_info *info); -void (*tlb_remove_table)(struct mmu_gather *tlb,void *table); -/* Hook for intercepting the destruction of an mm_struct. */ void (*exit_mmap)(struct mm_struct *mm);void (*notify_page_enc_status_changed)(unsigned long pfn,int npages,bool enc);
arch/x86/kernel/kvm.c +0 -1
@@ -838,7 +838,6 @@ static void __init kvm_guest_init(void)#ifdef CONFIG_SMP if (pv_tlb_flush_supported ()) { pv_ops.mmu.flush_tlb_multi = kvm_flush_tlb_multi; - pv_ops.mmu.tlb_remove_table = tlb_remove_table;pr_info ("KVM setup pv remote TLB flush\n" ); }
arch/x86/kernel/paravirt.c +0 -1
@@ -180,7 +180,6 @@ struct paravirt_patch_template pv_ops = { .mmu.flush_tlb_kernel = native_flush_tlb_global, .mmu.flush_tlb_one_user = native_flush_tlb_one_user, .mmu.flush_tlb_multi = native_flush_tlb_multi, - .mmu.tlb_remove_table = tlb_remove_table, .mmu.exit_mmap = paravirt_nop, .mmu.notify_page_enc_status_changed = paravirt_nop,
arch/x86/xen/mmu_pv.c +0 -1
@@ -2189,7 +2189,6 @@ static const typeof(pv_ops) xen_mmu_ops __initconst = { .flush_tlb_kernel = xen_flush_tlb, .flush_tlb_one_user = xen_flush_tlb_one_user, .flush_tlb_multi = xen_flush_tlb_multi, - .tlb_remove_table = tlb_remove_table, .pgd_alloc = xen_pgd_alloc, .pgd_free = xen_pgd_free,
PATCH v11 3/12 x86/mm: consolidate full flush threshold decision
In reply to Rik van RielReduce code duplication by consolidating the decision point for whether to do individual invalidations or a full flush inside get_flush_tlb_info.
--- arch/x86/mm/tlb.c | 41 +++++++++++++++++++---------------------- 1 file changed, 19 insertions(+), 22 deletions(-)
arch/x86/mm/tlb.c +19 -22
@@ -1009,6 +1009,15 @@ static struct flush_tlb_info *get_flush_tlb_info(struct mm_struct *mm, info->initiating_cpu =smp_processor_id (); info->trim_cpumask =0 ; +/* +* If the number of flushes is so large that a full flush +* would be faster, do a full flush. +*/ +if ((end - start) >> stride_shift > tlb_single_page_flush_ceiling) { + info->start =0 ; + info->end =TLB_FLUSH_ALL ; + } +return info; }
@@ -1026,17 +1035,8 @@ void flush_tlb_mm_range(struct mm_struct *mm, unsigned long start, bool freed_tables) {struct flush_tlb_info *info; +int cpu =get_cpu ();u64 new_tlb_gen; -int cpu; - - cpu =get_cpu (); - -/* Should we flush just the requested range? */ -if ((end ==TLB_FLUSH_ALL ) || - ((end - start) >> stride_shift) > tlb_single_page_flush_ceiling) { - start =0 ; - end =TLB_FLUSH_ALL ; - }/* This is also a barrier that synchronizes with switch_mm(). */ new_tlb_gen =inc_mm_tlb_gen (mm);
@@ -1089,22 +1089,19 @@ static void do_kernel_range_flush(void *info)void flush_tlb_kernel_range (unsigned long start,unsigned long end) { -/* Balance as user space task's flush, a bit conservative */ -if (end ==TLB_FLUSH_ALL || - (end - start) > tlb_single_page_flush_ceiling <<PAGE_SHIFT ) { -on_each_cpu (do_flush_tlb_all,NULL ,1 ); - }else { -struct flush_tlb_info *info; +struct flush_tlb_info *info; -preempt_disable (); - info =get_flush_tlb_info (NULL , start, end,0 , false, -TLB_GENERATION_INVALID ); +guard (preempt)(); + info =get_flush_tlb_info (NULL , start, end,PAGE_SHIFT , false, +TLB_GENERATION_INVALID ); + +if (info->end ==TLB_FLUSH_ALL ) +on_each_cpu (do_flush_tlb_all,NULL ,1 ); +else on_each_cpu (do_kernel_range_flush, info,1 ); -put_flush_tlb_info (); -preempt_enable (); - } +put_flush_tlb_info (); }/*
Signature · 1 line
2.47.1
Re: [PATCH v11 03/12] x86/mm: consolidate full flush threshold decision
Quoted 3 lines · Rik van Riel
Reduce code duplication by consolidating the decision point for whether to do individual invalidations or a full flush inside
get_flush_tlb_info.
Looks sane. Not a ton of code savings, but it does look a lot nicer:
Re: [PATCH v11 03/12] x86/mm: consolidate full flush threshold decision
In reply to Rik van Rielarch/x86/mm/tlb.c static struct flush_tlb_info *get_flush_tlb_info(struct mm_struct *mm, · line 1019
Quoted 12 lines · Rik van Riel
arch/x86/mm/tlb.c +8
-0@@ -1009,6 +1009,15 @@ static struct flush_tlb_info *get_flush_tlb_info(struct mm_struct *mm, info->initiating_cpu =smp_processor_id (); info->trim_cpumask =0 ; +/* +* If the number of flushes is so large that a full flush +* would be faster, do a full flush. +*/ +if ((end - start) >> stride_shift > tlb_single_page_flush_ceiling) { + info->start =0 ; + info->end =TLB_FLUSH_ALL ; + }
And if you move the range decision before the info-> struct members assignment, it becomes even more readable because you're using start and end in the check and then you assign it so a reader doesn't have to go and look whether start and end are the same as info->start and info->end:
/*
* If the number of flushes is so large that a full flush
* would be faster, do a full flush.
*/
if ((end - start) >> stride_shift > tlb_single_page_flush_ceiling) {
start = 0;
end = TLB_FLUSH_ALL;
}
info->start = start;
info->end = end;
info->mm = mm;
info->stride_shift = stride_shift;
info->freed_tables = freed_tables;
info->new_tlb_gen = new_tlb_gen;
info->initiating_cpu = smp_processor_id();
info->trim_cpumask = 0;
return info;
}with that:
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquettePATCH v11 4/12 x86/mm: get INVLPGB count max from CPUID
In reply to Rik van RielThe CPU advertises the maximum number of pages that can be shot down with one INVLPGB instruction in the CPUID data.
Save that information for later use.
--- arch/x86/Kconfig.cpu | 5 +++++ arch/x86/include/asm/cpufeatures.h | 1 + arch/x86/include/asm/tlbflush.h | 7 +++++++ arch/x86/kernel/cpu/amd.c | 8 ++++++++ 4 files changed, 21 insertions(+)
arch/x86/Kconfig.cpu +5 -0
@@ -395,6 +395,10 @@ config X86_VMX_FEATURE_NAMES def_bool y depends on IA32_FEAT_CTL +config X86_BROADCAST_TLB_FLUSH + def_bool y + depends on CPU_SUP_AMD && 64BIT + menuconfig PROCESSOR_SELECT bool "Supported processor vendors" if EXPERT help
@@ -431,6 +435,7 @@ config CPU_SUP_CYRIX_32 config CPU_SUP_AMD default y bool "Support AMD processors" if PROCESSOR_SELECT + select X86_BROADCAST_TLB_FLUSH help This enables detection, tunings and quirks for AMD processors
arch/x86/include/asm/cpufeatures.h +1 -0
@@ -338,6 +338,7 @@#define X86_FEATURE_CLZERO (13*32+ 0)/* "clzero" CLZERO instruction */ #define X86_FEATURE_IRPERF (13*32+ 1)/* "irperf" Instructions Retired Count */ #define X86_FEATURE_XSAVEERPTR (13*32+ 2)/* "xsaveerptr" Always save/restore FP error pointers */ +#define X86_FEATURE_INVLPGB (13*32+ 3)/* INVLPGB and TLBSYNC instruction supported. */ #define X86_FEATURE_RDPRU (13*32+ 4)/* "rdpru" Read processor register at user level */ #define X86_FEATURE_WBNOINVD (13*32+ 9)/* "wbnoinvd" WBNOINVD instruction */ #define X86_FEATURE_AMD_IBPB (13*32+12)/* Indirect Branch Prediction Barrier */
arch/x86/include/asm/tlbflush.h +7 -0
@@ -183,6 +183,13 @@ static inline void cr4_init_shadow(void)extern unsigned long mmu_cr4_features;extern u32 *trampoline_cr4_features; +/* How many pages can we invalidate with one INVLPGB. */ +#ifdef CONFIG_X86_BROADCAST_TLB_FLUSH +extern u16 invlpgb_count_max; +#else +#define invlpgb_count_max 1 +#endif +extern void initialize_tlbstate_and_flush (void );/*
arch/x86/kernel/cpu/amd.c +8 -0
@@ -29,6 +29,8 @@#include "cpu.h" +u16 invlpgb_count_max__ro_after_init ; +static inline int rdmsrl_amd_safe (unsigned msr,unsigned long long *p) {u32 gprs[8 ] = {0 };
@@ -1139,6 +1141,12 @@ static void cpu_detect_tlb_amd(struct cpuinfo_x86 *c) tlb_lli_2m[ENTRIES ] = eax & mask; tlb_lli_4m[ENTRIES ] = tlb_lli_2m[ENTRIES ] >>1 ; + +/* Max number of pages INVLPGB can invalidate in one shot */ +if (boot_cpu_has (X86_FEATURE_INVLPGB )) { +cpuid (0x80000008 , &eax, &ebx, &ecx, &edx); + invlpgb_count_max = (edx &0xffff ) +1 ; + } }static const struct cpu_dev amd_cpu_dev = {
Signature · 1 line
2.47.1
Re: [PATCH v11 04/12] x86/mm: get INVLPGB count max from CPUID
arch/x86/kernel/cpu/amd.c static void cpu_detect_tlb_amd(struct cpuinfo_x86 *c) · line 1150
Quoted 6 lines · Rik van Riel
arch/x86/kernel/cpu/amd.c +5
-0+/* Max number of pages INVLPGB can invalidate in one shot */ +if (boot_cpu_has (X86_FEATURE_INVLPGB )) { +cpuid (0x80000008 , &eax, &ebx, &ecx, &edx); + invlpgb_count_max = (edx &0xffff ) +1 ; + } }
My only nit with this is that this adds a seventh place that the kernel reads leaf 0x80000008. The users seem really random, though. Some cross-vendor stuff, some very AMD-specific stuff.
So, I'll leave that at a nit since I don't have a good suggestion for solving it:
Borislav Petkov
maintainerRe: [PATCH v11 04/12] x86/mm: get INVLPGB count max from CPUID
In reply to Rik van RielQuoted 32 lines · Rik van Riel
The CPU advertises the maximum number of pages that can be shot down with one INVLPGB instruction in the CPUID data.
Save that information for later use.
Signed-off-by Rik van Riel <[email protected]>Tested-by Manali Shukla <[email protected]>Tested-by Brendan Jackman <[email protected]>Tested-by Michael Kelley <[email protected]>--- arch/x86/Kconfig.cpu | 5 +++++ arch/x86/include/asm/cpufeatures.h | 1 + arch/x86/include/asm/tlbflush.h | 7 +++++++ arch/x86/kernel/cpu/amd.c | 8 ++++++++ 4 files changed, 21 insertions(+)arch/x86/Kconfig.cpu +4
-0@@ -395,6 +395,10 @@ config X86_VMX_FEATURE_NAMES def_bool y depends on IA32_FEAT_CTL +config X86_BROADCAST_TLB_FLUSH + def_bool y + depends on CPU_SUP_AMD && 64BIT + menuconfig PROCESSOR_SELECT bool "Supported processor vendors" if EXPERT help
arch/x86/Kconfig.cpu config CPU_SUP_CYRIX_32 · line 438
Quoted 4 lines
arch/x86/Kconfig.cpu +1
-0@@ -431,6 +435,7 @@ config CPU_SUP_CYRIX_32 config CPU_SUP_AMD default y bool "Support AMD processors" if PROCESSOR_SELECT + select X86_BROADCAST_TLB_FLUSH
CPU_SUP_AMD selects X86_BROADCAST_TLB_FLUSH which depends on CPU_SUP_AMD which selects X86_BROADCAST_TLB_FLUSH which depends on CPU_SUP_AMD...
Why do you really need yet another Kconfig symbol? Just whack X86_BROADCAST_TLB_FLUSH - it'll be enabled by default on everything anyway.
arch/x86/Kconfig.cpu static void cpu_detect_tlb_amd(struct cpuinfo_x86 *c) · line 1149
Quoted 10 lines
arch/x86/Kconfig.cpu +6
-0@@ -1139,6 +1141,12 @@ static void cpu_detect_tlb_amd(struct cpuinfo_x86 *c) tlb_lli_2m[ENTRIES] = eax & mask; tlb_lli_4m[ENTRIES] = tlb_lli_2m[ENTRIES] >> 1; + + /* Max number of pages INVLPGB can invalidate in one shot */ + if (boot_cpu_has(X86_FEATURE_INVLPGB)) { + cpuid(0x80000008, &eax, &ebx, &ecx, &edx); + invlpgb_count_max = (edx & 0xffff) + 1; + }
get_cpu_cap() already reads that leaf. You don't need to do it here again.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteRe: [PATCH v11 04/12] x86/mm: get INVLPGB count max from CPUID
Quoted 43 lines · Borislav Petkov
Quoted 36 lines · Rik van Riel
The CPU advertises the maximum number of pages that can be shot down
with one INVLPGB instruction in the CPUID data.Save that information for later use.
Signed-off-by Rik van Riel <[email protected]>Tested-by Manali Shukla <[email protected]>Tested-by Brendan Jackman <[email protected]>Tested-by Michael Kelley <[email protected]>---
arch/x86/Kconfig.cpu | 5 +++++arch/x86/include/asm/cpufeatures.h| 1 + arch/x86/include/asm/tlbflush.h | 7 +++++++ arch/x86/kernel/cpu/amd.c | 8 ++++++++4 files changed, 21 insertions(+)arch/x86/Kconfig.cpu +0
-0@@ -395,6 +395,10 @@ config X86_VMX_FEATURE_NAMES
def_booly
depends onIA32_FEAT_CTLarch/x86/Kconfig.cpu +4
-0+config X86_BROADCAST_TLB_FLUSH + def_bool y + depends on CPU_SUP_AMD && 64BIT +menuconfig
PROCESSOR_SELECT
bool "Supported processor vendors" if EXPERT helparch/x86/Kconfig.cpu +0
-0@@ -431,6 +435,7 @@ config CPU_SUP_CYRIX_32config
CPU_SUP_AMD
default y
bool "Support AMD processors" ifPROCESSOR_SELECTarch/x86/Kconfig.cpu config CPU_SUP_CYRIX_32 · line 438
Quoted 1 line
arch/x86/Kconfig.cpu +1
-0+ select X86_BROADCAST_TLB_FLUSH
CPU_SUP_AMDselectsX86_BROADCAST_TLB_FLUSHwhich depends onCPU_SUP_AMDwhich
selectsX86_BROADCAST_TLB_FLUSHwhich depends onCPU_SUP_AMD...
Quoted 3 lines
Why do you really need yet another
Kconfigsymbol? Just whackX86_BROADCAST_TLB_FLUSH- it'll be enabled by default on everything anyway.
Dave specifically asked me to do things that way.
Quoted 17 lines
arch/x86/Kconfig.cpu
Quoted 13 lines
arch/x86/Kconfig.cpu +0
-0@@ -1139,6 +1141,12 @@ static void cpu_detect_tlb_amd(struct cpuinfo_x86 *c) tlb_lli_2m[ENTRIES] = eax & mask;
tlb_lli_4m[ENTRIES] = tlb_lli_2m[ENTRIES] >> 1;arch/x86/Kconfig.cpu +2
-0+ + /* Max number of pages INVLPGB can invalidate in one shot*/
arch/x86/Kconfig.cpu +4
-0+ if (boot_cpu_has(X86_FEATURE_INVLPGB)) { + cpuid(0x80000008, &eax, &ebx, &ecx, &edx); + invlpgb_count_max = (edx & 0xffff) + 1; + }
get_cpu_cap()already reads that leaf. You don't need to do it here again.
Should I modify get_cpu_cap() to store
c->x86_capabilities[CPUID_8000_0008_EDX] ?
Currently only the EBX data is stored there,
while invlpgb_count_max comes from EDX.
Signature · 1 line
All Rights Reversed.
Borislav Petkov
maintainerRe: [PATCH v11 04/12] x86/mm: get INVLPGB count max from CPUID
Quoted 5 lines · Rik van Riel
Should I modify
get_cpu_cap()to store
c->x86_capabilities[CPUID_8000_0008_EDX] ?Currently only the EBX data is stored there,
whileinvlpgb_count_maxcomes from EDX.
No, you should simply assign invlpgb_count_max there.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteRe: [PATCH v11 04/12] x86/mm: get INVLPGB count max from CPUID
In reply to Rik van RielQuoted 8 lines · Rik van Riel
Quoted 7 lines
CPU_SUP_AMDselectsX86_BROADCAST_TLB_FLUSHwhich depends onCPU_SUP_AMDwhich
selectsX86_BROADCAST_TLB_FLUSHwhich depends onCPU_SUP_AMD...Why do you really need yet another
Kconfigsymbol? Just whackX86_BROADCAST_TLB_FLUSH- it'll be enabled by default on everything anyway.Dave specifically asked me to do things that way.
Yeah, I'm in camp "moar Kconfig!" and Boris is in camp "too many Kconfigs!".
I'm OK getting rid of the Kconfig as long as we have _some_ other way to nicely identify the INVLPGB code (which I think the X86_FEATURE gives us) _and_ that the overhead isn't too bad from leaving it compiled in all the time.
The mmu_context is the one place that's kinda nasty without a Kconfig option. I guess it's only a few bytes but it's a super silly waste of a few bytes on i386, for example.
I also really like the _logical_ clarity that comes from something like:
+#ifdef CONFIG_X86_BROADCAST_TLB_FLUSH
+ u16 global_asid;
+ bool asid_transition;
+#endif
as opposed to:
+#ifdef CONFIG_64BIT
+ /* Only used with X86_FEATURE_INVLPGB */
+ u16 global_asid;
+ bool asid_transition;
+#endif
So, while I respect Boris's concern about Kconfig proliferation, I see the user-invisible ones (the ones without a prompt) as pretty harmless.
Sorry for the maintainer crossfire, btw. I definitely don't want to be thrashing you around too much.
PATCH v11 5/12 x86/mm: add INVLPGB support code
In reply to Rik van RielAdd invlpgb.h with the helper functions and definitions needed to use broadcast TLB invalidation on AMD EPYC 3 and newer CPUs.
--- arch/x86/include/asm/disabled-features.h | 8 +- arch/x86/include/asm/invlpgb.h | 101 +++++++++++++++++++++++ arch/x86/include/asm/tlbflush.h | 1 + 3 files changed, 109 insertions(+), 1 deletion(-) create mode 100644 arch/x86/include/asm/invlpgb.h
arch/x86/include/asm/disabled-features.h +7 -1
@@ -129,6 +129,12 @@#define DISABLE_SEV_SNP (1 << (X86_FEATURE_SEV_SNP & 31))#endif +#ifdef CONFIG_X86_BROADCAST_TLB_FLUSH +#define DISABLE_INVLPGB 0 +#else +#define DISABLE_INVLPGB (1 << (X86_FEATURE_INVLPGB & 31)) +#endif +/* * Make sure to add features to the correct mask */
@@ -146,7 +152,7 @@#define DISABLED_MASK11 (DISABLE_RETPOLINE|DISABLE_RETHUNK|DISABLE_UNRET| \ DISABLE_CALL_DEPTH_TRACKING|DISABLE_USER_SHSTK)#define DISABLED_MASK12 (DISABLE_FRED|DISABLE_LAM) -#define DISABLED_MASK13 0 +#define DISABLED_MASK13 (DISABLE_INVLPGB)#define DISABLED_MASK14 0#define DISABLED_MASK15 0#define DISABLED_MASK16 (DISABLE_PKU|DISABLE_OSPKE|DISABLE_LA57|DISABLE_UMIP| \
arch/x86/include/asm/invlpgb.h new file +101 -0
101 lines not on this page · raw message
arch/x86/include/asm/tlbflush.h +1 -0
@@ -10,6 +10,7 @@#include <asm/cpufeature.h> #include <asm/special_insns.h> #include <asm/smp.h> +#include <asm/invlpgb.h> #include <asm/invpcid.h> #include <asm/pti.h> #include <asm/processor-flags.h>
Signature · 1 line
2.47.1
Re: [PATCH v11 05/12] x86/mm: add INVLPGB support code
Quoted 2 lines · Rik van Riel
Add
invlpgb.hwith the helper functions and definitions needed to use broadcast TLB invalidation on AMD EPYC 3 and newer CPUs.
You should also note here that (or if??) all these functions get used later in the series.
arch/x86/include/asm/invlpgb.h line 92
Quoted 5 lines
arch/x86/include/asm/invlpgb.h new file +5
-0+/* Flush addr, including globals, for all PCIDs. */ +static inline void invlpgb_flush_addr_nosync (unsigned long addr,u16 nr) +{ +__invlpgb (0 ,0 , addr, nr -1 ,0 ,INVLPGB_INCLUDE_GLOBAL ); +}
Something about the "nr - 1"'s needs to get mentioned *somewhere*. I think the best place is actually in __invlpgb(). Basically make the calling convention for __invlpgb() be the _sane_ thing where nr==1 flushes 1 page. Then do the nr-=1 in __invlpgb() and document why.
I don't mean to insult the AMD ISA designers here. I might have done the same thing. But the software to use the instruction ends up looking really funky. It would be great to limit the number of places that deal with the funkiness to exactly 1.
With those two nits addressed:
Re: [PATCH v11 05/12] x86/mm: add INVLPGB support code
Quoted 8 lines · Dave Hansen
Quoted 3 lines · Rik van Riel
Add
invlpgb.hwith the helper functions and definitions needed to use
broadcast TLB invalidation on AMD EPYC 3 and newer CPUs.You should also note here that (or if??) all these functions get used later in the series.
You made me look, and there was indeed one helper function that is not being used. I removed it, and added in the commit message that all the (remaining) functions are used.
Quoted 19 lines
arch/x86/include/asm/invlpgb.h line 92
Quoted 6 lines
(quoted diff) +2
-0+/* Flush addr, including globals, for all PCIDs. */ +static inline void invlpgb_flush_addr_nosync(unsigned long addr,u16 nr)
arch/x86/include/asm/invlpgb.h new file +3
-0+{ +__invlpgb (0 ,0 , addr, nr -1 ,0 ,INVLPGB_INCLUDE_GLOBAL ); +}Something about the "nr - 1"'s needs to get mentioned *somewhere*. I think the best place is actually in
__invlpgb(). Basically make the calling convention for__invlpgb()be the _sane_ thing where nr==1 flushes 1 page. Then do the nr-=1 in__invlpgb()and document why.I don't mean to insult the AMD ISA designers here. I might have done the
same thing. But the software to use the instruction ends up looking really funky. It would be great to limit the number of places that deal
with the funkiness to exactly 1.
Quoted 3 lines
With those two nits addressed:
Acked-by Dave Hansen <[email protected]>
Nits addressed, and the Acked-by header added to the next version :)
Thank you!
Signature · 1 line
All Rights Reversed.
Borislav Petkov
maintainerRe: [PATCH v11 05/12] x86/mm: add INVLPGB support code
In reply to Rik van RielQuoted 19 lines · Rik van Riel
Add
invlpgb.hwith the helper functions and definitions needed to use broadcast TLB invalidation on AMD EPYC 3 and newer CPUs.Signed-off-by Rik van Riel <[email protected]>Tested-by Manali Shukla <[email protected]>Tested-by Brendan Jackman <[email protected]>Tested-by Michael Kelley <[email protected]>--- arch/x86/include/asm/disabled-features.h | 8 +- arch/x86/include/asm/invlpgb.h | 101 +++++++++++++++++++++++ arch/x86/include/asm/tlbflush.h | 1 + 3 files changed, 109 insertions(+), 1 deletion(-) create mode 100644 arch/x86/include/asm/invlpgb.harch/x86/include/asm/disabled-features.h +0
-0
arch/x86/include/asm/disabled-features.h line 137
Quoted 9 lines
arch/x86/include/asm/disabled-features.h +6
-0@@ -129,6 +129,12 @@#define DISABLE_SEV_SNP (1 << (X86_FEATURE_SEV_SNP & 31))#endif +#ifdef CONFIG_X86_BROADCAST_TLB_FLUSH +#define DISABLE_INVLPGB 0 +#else +#define DISABLE_INVLPGB (1 << (X86_FEATURE_INVLPGB & 31)) +#endif +
What does that bring you really in savings?
arch/x86/include/asm/invlpgb.h
Quoted 17 lines
arch/x86/include/asm/disabled-features.h +1
-1/* * Make sure to add features to the correct mask */ @@ -146,7 +152,7 @@#define DISABLED_MASK11 (DISABLE_RETPOLINE|DISABLE_RETHUNK|DISABLE_UNRET| \ DISABLE_CALL_DEPTH_TRACKING|DISABLE_USER_SHSTK)#define DISABLED_MASK12 (DISABLE_FRED|DISABLE_LAM) -#define DISABLED_MASK13 0 +#define DISABLED_MASK13 (DISABLE_INVLPGB)#define DISABLED_MASK14 0#define DISABLED_MASK15 0#define DISABLED_MASK16 (DISABLE_PKU|DISABLE_OSPKE|DISABLE_LA57|DISABLE_UMIP| \arch/x86/include/asm/invlpgb.h new file +0
-0
I remember asking you to add all that gunk to arch/x86/include/asm/tlb.h. Please do so.
arch/x86/include/asm/invlpgb.h line 61
Quoted 10 lines
arch/x86/include/asm/invlpgb.h new file +10
-0+#define INVLPGB_VA BIT(0) +#define INVLPGB_PCID BIT(1) +#define INVLPGB_ASID BIT(2) +#define INVLPGB_INCLUDE_GLOBAL BIT(3) +#define INVLPGB_FINAL_ONLY BIT(4) +#define INVLPGB_INCLUDE_NESTED BIT(5) + +/* Flush all mappings for a given pcid and addr, not including globals. */ +static inline void invlpgb_flush_user (unsigned long pcid, +unsigned long addr)
Please drop this unused function.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteRe: [PATCH v11 05/12] x86/mm: add INVLPGB support code
Quoted 9 lines · Borislav Petkov
arch/x86/include/asm/invlpgb.h
Quoted 4 lines · Rik van Riel
index 000000000000..a1d5dedd5217
arch/x86/include/asm/invlpgb.h new file +0
-0I remember asking you to add all that gunk to
arch/x86/include/asm/tlb.h.
Please do so.
Dave just asked me to split out more things into
their own files.
I'm happy to do whatever the maintainers want,
but when you both want the opposite from each
other, I won't be able to make you both happy.
What should I be doing here?
Signature · 1 line
All Rights Reversed.
Re: [PATCH v11 05/12] x86/mm: add INVLPGB support code
Quoted 11 lines · Rik van Riel
Quoted 9 lines · Borislav Petkov
arch/x86/include/asm/invlpgb.h
Quoted 4 lines · Rik van Riel
index 000000000000..a1d5dedd5217
arch/x86/include/asm/invlpgb.h new file +0
-0I remember asking you to add all that gunk to
arch/x86/include/asm/tlb.h.
Please do so.
Quoted 8 lines
Dave just asked me to split out more things into
their own files.I'm happy to do whatever the maintainers want,
but when you both want the opposite from each
other, I won't be able to make you both happy.What should I be doing here?
I think you're referring to this:
Quoted 1 line
I don't have a strong preference about creating an invlpgb.h header. As long as the header is still relatively small the #ifdef pile is readable, it's fine to stick in an existing header.
The thing I raised about the 09/12 patch was a large #ifdef in a .c file. We have a general rule to avoid #ifdefs in .c files. Specifically, I find that having large #ifdef's regions in a .c file means that you literally can't see both sides of the #ifdef at once or easily understand which code belongs to the #ifdef. The result is invariably weird compile issues that pop up.
But, either way, #ifdefs are a sign of weakness. Less so in a header and more so in a .c file.
Borislav Petkov
maintainerRe: [PATCH v11 05/12] x86/mm: add INVLPGB support code
Quoted 2 lines · Dave Hansen
But, either way, #ifdefs are a sign of weakness. Less so in a header and more so in a .c file.
... and, as we just discussed and agreed on chat, we don't need the Kconfig option either.
Thx.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteRe: [PATCH v11 05/12] x86/mm: add INVLPGB support code
Quoted 9 lines · Borislav Petkov
Quoted 3 lines · Dave Hansen
But, either way, #ifdefs are a sign of weakness. Less so in a header and
more so in a .c file.... and, as we just discussed and agreed on chat, we don't need the
Kconfig
option either.
I'll re-do the series without ifdefs, except for
the one masking out X86_FEATURE_INVLPGB (on 32
bit systems).
Then the compiler should be able to get rid of the other code, when the feature is hard-disabled.
Signature · 1 line
All Rights Reversed.
Borislav Petkov
maintainerRe: [PATCH v11 05/12] x86/mm: add INVLPGB support code
Quoted 6 lines · Rik van Riel
I'll re-do the series without ifdefs, except for
the one masking outX86_FEATURE_INVLPGB(on 32
bit systems).Then the compiler should be able to get rid of the other code, when the feature is hard-disabled.
Thanks.
In talking to Dave last night, I think we are fine with having the majority of ifdeffery in headers and keep it away from .c files.
Any new Kconfig symbol adds more configurations we must build and test but... oh well.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquettePATCH v11 6/12 x86/mm: use INVLPGB for kernel TLB flushes
In reply to Rik van RielUse broadcast TLB invalidation for kernel addresses when available.
Remove the need to send IPIs for kernel TLB flushes.
--- arch/x86/mm/tlb.c | 26 +++++++++++++++++++++++++- 1 file changed, 25 insertions(+), 1 deletion(-)
arch/x86/mm/tlb.c +25 -1
@@ -1077,6 +1077,28 @@ void flush_tlb_all(void)on_each_cpu (do_flush_tlb_all,NULL ,1 ); } +static bool broadcast_kernel_range_flush (struct flush_tlb_info *info) +{ +unsigned long addr; +unsigned long nr; + +if (!cpu_feature_enabled (X86_FEATURE_INVLPGB )) +return false; + +if (info->end ==TLB_FLUSH_ALL ) { +invlpgb_flush_all (); +return true; + } + +for (addr = info->start; addr < info->end; addr += nr <<PAGE_SHIFT ) { + nr = (info->end - addr) >>PAGE_SHIFT ; + nr =clamp_val (nr,1 , invlpgb_count_max); +invlpgb_flush_addr_nosync (addr, nr); + } +tlbsync (); +return true; +} +static void do_kernel_range_flush (void *info) {struct flush_tlb_info *f = info;
@@ -1096,7 +1118,9 @@ void flush_tlb_kernel_range(unsigned long start, unsigned long end) info =get_flush_tlb_info (NULL , start, end,PAGE_SHIFT , false,TLB_GENERATION_INVALID ); -if (info->end ==TLB_FLUSH_ALL ) +if (broadcast_kernel_range_flush (info)) + ;/* Fall through. */ +else if (info->end ==TLB_FLUSH_ALL )on_each_cpu (do_flush_tlb_all,NULL ,1 );else on_each_cpu (do_kernel_range_flush, info,1 );
Signature · 1 line
2.47.1
Re: [PATCH v11 06/12] x86/mm: use INVLPGB for kernel TLB flushes
arch/x86/mm/tlb.c void flush_tlb_kernel_range(unsigned long start, unsigned long end) · line 1126
Quoted 7 lines · Rik van Riel
arch/x86/mm/tlb.c +3
-1-if (info->end ==TLB_FLUSH_ALL ) +if (broadcast_kernel_range_flush (info)) + ;/* Fall through. */ +else if (info->end ==TLB_FLUSH_ALL )on_each_cpu (do_flush_tlb_all,NULL ,1 );else on_each_cpu (do_kernel_range_flush, info,1 );
We've got to find a better name for broadcast_kernel_range_flush(). Because IPIs are broadcast too. The naming makes it confusing. Why would be broadcast, and then start trying IPIs that are also broadcast?
This hunk really the crux of the patch and we need to make sure the semantics are crystal clear. What do folks think about something like this:
/*
* Try to use a hardware assist for flushing the TLB.
* This is expected to be cheaper than using an IPI
* to do flushes.
*
* Returns True if the assisted method succeeded.
*/
static bool assisted_kernel_range_flush(struct flush_tlb_info *info)
{
...
}static do_ipi_tlb_flush(...info)
{
if (info->end == TLB_FLUSH_ALL)
on_each_cpu(do_flush_tlb_all, NULL, 1);
else
on_each_cpu(do_kernel_range_flush, info, 1);
}Then we end up with:
/* * Assisted flushes are assumed to be faster. Try * them first and fall back to IPIs if not available. */ flush_success = assisted_kernel_range_flush(info); if (!flush_success) do_ipi_tlb_flush(info);
I think that's a *LOT* more clear:
1. Try assisted flush
2. If it fails fall back to an IPI-based flush
Peter Zijlstra
maintainerRe: [PATCH v11 06/12] x86/mm: use INVLPGB for kernel TLB flushes
Quoted 12 lines · Dave Hansen
arch/x86/mm/tlb.c void flush_tlb_kernel_range(unsigned long start, unsigned long end) · line 1126
Quoted 7 lines · Rik van Riel
arch/x86/mm/tlb.c +3
-1-if (info->end ==TLB_FLUSH_ALL ) +if (broadcast_kernel_range_flush (info)) + ;/* Fall through. */ +else if (info->end ==TLB_FLUSH_ALL )on_each_cpu (do_flush_tlb_all,NULL ,1 );else on_each_cpu (do_kernel_range_flush, info,1 );We've got to find a better name for
broadcast_kernel_range_flush(). Because IPIs are broadcast too. The naming makes it confusing. Why would be broadcast, and then start trying IPIs that are also broadcast?
IIRC the more general name is indeed broadcast tlbi; as in other architectures use this naming to mean this very thing too.
But yes, I see the confusion, but I don't think changing the naming really helps a lot here :-/
Re: [PATCH v11 06/12] x86/mm: use INVLPGB for kernel TLB flushes
Quoted 12 lines · Peter Zijlstra
Quoted 11 lines · Dave Hansen
arch/x86/mm/tlb.c void flush_tlb_kernel_range(unsigned long start, unsigned long end) · line 1126
Quoted 7 lines · Rik van Riel
arch/x86/mm/tlb.c +3
-1-if (info->end ==TLB_FLUSH_ALL ) +if (broadcast_kernel_range_flush (info)) + ;/* Fall through. */ +else if (info->end ==TLB_FLUSH_ALL )on_each_cpu (do_flush_tlb_all,NULL ,1 );else on_each_cpu (do_kernel_range_flush, info,1 );We've got to find a better name for
broadcast_kernel_range_flush(). Because IPIs are broadcast too. The naming makes it confusing. Why would be broadcast, and then start trying IPIs that are also broadcast?
Quoted 5 lines
IIRC the more general name is indeed broadcast tlbi; as in other architectures use this naming to mean this very thing too.
But yes, I see the confusion, but I don't think changing the naming really helps a lot here :-/
Fair enough. If we don't have a better name, we can at least do:
if (new_bad_name()) {
new_thing();
} else {
old_thing();
}My real heartburn is with:
if (new_bad_name()) {
new_thing();
} else if (need_thing_1()) {
old_thing1();
} else {
old_thing2();
}Where new and old are logically squished together.
Re: [PATCH v11 06/12] x86/mm: use INVLPGB for kernel TLB flushes
Quoted 19 lines · Dave Hansen
Fair enough. If we don't have a better name, we can at least do:
if (new_bad_name()) { new_thing(); } else { old_thing(); }My real heartburn is with:
if (new_bad_name()) { new_thing(); } else if (need_thing_1()) { old_thing1(); } else { old_thing2(); }
Quoted 2 lines
Where new and old are logically squished together.
Do we want to group this code by history, or by function?
I would argue that new_thing() and old_thing1() are closer to each other functionally (they both do remote TLB invalidation) than they are to old_thing2(), which does local-only invalidation.
I can organize the code however people want, but I would like a second opinion on this idea :)
Signature · 1 line
All Rights Reversed.
Re: [PATCH v11 06/12] x86/mm: use INVLPGB for kernel TLB flushes
Quoted 25 lines · Rik van Riel
Quoted 19 lines · Dave Hansen
Fair enough. If we don't have a better name, we can at least do:
if (new_bad_name()) { new_thing(); } else { old_thing(); }My real heartburn is with:
if (new_bad_name()) { new_thing(); } else if (need_thing_1()) { old_thing1(); } else { old_thing2(); }Quoted 2 lines
Where new and old are logically squished together.
Do we want to group this code by history, or by function?
Quoted 7 lines
I would argue that
new_thing()andold_thing1()are closer to each other functionally (they both do remote TLB invalidation) than they are toold_thing2(), which does local-only invalidation.I can organize the code however people want, but I would like a second opinion on this idea :)
IIUC the discussion is about:
if (broadcast_kernel_range_flush(info))
; /* Fall through. */
else if (info->end == TLB_FLUSH_ALL)
on_each_cpu(do_flush_tlb_all, NULL, 1);
else
on_each_cpu(do_kernel_range_flush, info, 1);In this case I agree with Dave. old_thing1() and old_thing2() are both sending IPIs, the difference is that old_thing1() is doing a full flush while old_thing2() is doing a range flush. Not sure why you mentioned that old_thing2() does local invalidation.
broadcast_kernel_range_flush() also decides between full and range flushes internally. So the main difference between 'new' and 'old' here is using the broadcast flush vs the IPI flush.
So I think what Dave wants (and I agree) is:
if (!broadcast_kernel_range_flush(info))
ipi_kernel_range_flush(info)
Where ipi_kernel_range_flush() contains old_thing1() and oldthing2().
Quoted 3 lines
Signature · 1 line
All Rights Reversed.
Re: [PATCH v11 06/12] x86/mm: use INVLPGB for kernel TLB flushes
Quoted 6 lines · Yosry Ahmed
So I think what Dave wants (and I agree) is:
if (!broadcast_kernel_range_flush(info))
ipi_kernel_range_flush(info)Where
ipi_kernel_range_flush()containsold_thing1()andoldthing2().
I like this idea. I changed the code to add that
for the next version! Thank you.
Signature · 1 line
All Rights Reversed.
Re: [PATCH v11 06/12] x86/mm: use INVLPGB for kernel TLB flushes
Quoted 6 lines · Rik van Riel
Quoted 5 lines · Yosry Ahmed
So I think what Dave wants (and I agree) is:
if (!broadcast_kernel_range_flush(info))
ipi_kernel_range_flush(info)Where
ipi_kernel_range_flush()containsold_thing1()andoldthing2().
That's OK-ish. But it still smells of hacking in the new concept without refactoring things properly.
Let's logically inline the code that we've got. I think it actually ends up looking something like this:
if (cpu_feature_enabled(X86_FEATURE_INVLPGB)) {
if (info->end == TLB_FLUSH_ALL) {
invlpgb_flush_all();
} else {
for_each(addr)
invlpgb_flush_addr_nosync(addr, nr);
}
} else {
if (info->end == TLB_FLUSH_ALL)
on_each_cpu(do_flush_tlb_all, NULL, 1);
else
on_each_cpu(do_kernel_range_flush, info, 1);
}Where we've got two inputs:
1. INVLPGB support (or not)
2. TLB_FLUSH_ALL (basically ranged or full flush)
So I think we should group by *one* of those. The above groups by INVLPGB support and this groups by TLB_FLUSH_ALL:
if (info->end == TLB_FLUSH_ALL) {
if (cpu_feature_enabled(X86_FEATURE_INVLPGB)) {
invlpgb_flush_all();
} else {
on_each_cpu(do_flush_tlb_all, NULL, 1);
}
} else {
if (cpu_feature_enabled(X86_FEATURE_INVLPGB))
for_each(addr)
invlpgb_flush_addr_nosync(addr, nr);
else
on_each_cpu(do_kernel_range_flush, info, 1);
}So, if we create some helpers that give some consistent naming:
static tlb_flush_all_ipi(...)
{
on_each_cpu(do_flush_tlb_all, NULL, 1);
}static tlb_flush_all(...)
{
if (cpu_feature_enabled(X86_FEATURE_INVLPGB))
invlpgb_flush_all(...);
else
tlb_flush_all_ipi(...);
}and then also create the ranged equivalents (which internally have the same cpu_feature_enabled() check):
tlb_flush_range_ipi(...) invlpgb_flush_range(...)
Then we can have the top-level code be:
if (info->end == TLB_FLUSH_ALL) tlb_flush_all(info); else tlb_flush_range(info);
That actually looks way nicer than what we have today. For bonus points, if a third way of flushing the TLB showed up, it would slot right in:
static tlb_flush_all(...)
{
if (cpu_feature_enabled(X86_FEATURE_INVLPGB))
invlpgb_flush_all(...);
+ else if cpu_feature_enabled(X86_FEATURE_RAR))
+ rar_flush_all(...);
else
tlb_flush_all_ipi(...);}
That's *exactly* the way we want the code to read. At the higher level, it's deciding based on the generic thing that *everybody* cares about:
ranged or full flush. Then, at the lower level, it's deciding how to implement that high-level flush concept.
Re: [PATCH v11 06/12] x86/mm: use INVLPGB for kernel TLB flushes
Quoted 36 lines · Dave Hansen
Quoted 6 lines · Rik van Riel
Quoted 5 lines · Yosry Ahmed
So I think what Dave wants (and I agree) is:
if (!broadcast_kernel_range_flush(info))
ipi_kernel_range_flush(info)Where
ipi_kernel_range_flush()containsold_thing1()andoldthing2().That's OK-ish. But it still smells of hacking in the new concept without refactoring things properly.
Let's logically inline the code that we've got. I think it actually ends up looking something like this:
if (cpu_feature_enabled(X86_FEATURE_INVLPGB)) { if (info->end == TLB_FLUSH_ALL) { invlpgb_flush_all(); } else { for_each(addr) invlpgb_flush_addr_nosync(addr, nr); } } else { if (info->end == TLB_FLUSH_ALL) on_each_cpu(do_flush_tlb_all, NULL, 1); else on_each_cpu(do_kernel_range_flush, info, 1); }Where we've got two inputs:
1. INVLPGB support (or not)
2.TLB_FLUSH_ALL(basically ranged or full flush)So I think we should group by *one* of those. The above groups by INVLPGB support and this groups by
TLB_FLUSH_ALL:
Quoted 13 lines
if (info->end == TLB_FLUSH_ALL) {
if (cpu_feature_enabled(X86_FEATURE_INVLPGB)) {
invlpgb_flush_all();
} else {
on_each_cpu(do_flush_tlb_all, NULL, 1);
}
} else {
if (cpu_feature_enabled(X86_FEATURE_INVLPGB))
for_each(addr)
invlpgb_flush_addr_nosync(addr, nr);
else
on_each_cpu(do_kernel_range_flush, info, 1);
}Yeah an if/else structure is better than using the invlpgb helper and falling back to IPIs if it returns false, and I also prefer grouping by the flush scope (range/flush).
Thanks for the illustrations :)
Quoted 47 lines
So, if we create some helpers that give some consistent naming:
static
tlb_flush_all_ipi(...){ on_each_cpu(do_flush_tlb_all, NULL, 1); }static
tlb_flush_all(...){ if (cpu_feature_enabled(X86_FEATURE_INVLPGB)) invlpgb_flush_all(...); else tlb_flush_all_ipi(...); }and then also create the ranged equivalents (which internally have the same
cpu_feature_enabled()check):tlb_flush_range_ipi(...) invlpgb_flush_range(...)Then we can have the top-level code be:
if (info->end == TLB_FLUSH_ALL) tlb_flush_all(info); else tlb_flush_range(info);That actually looks way nicer than what we have today. For bonus points, if a third way of flushing the TLB showed up, it would slot right in:
static
tlb_flush_all(...){ if (cpu_feature_enabled(X86_FEATURE_INVLPGB)) invlpgb_flush_all(...);(quoted diff) +2
-0+ else if cpu_feature_enabled(X86_FEATURE_RAR)) + rar_flush_all(...);else tlb_flush_all_ipi(...);}
That's *exactly* the way we want the code to read. At the higher level, it's deciding based on the generic thing that *everybody* cares about:
ranged or full flush. Then, at the lower level, it's deciding how to implement that high-level flush concept.
PATCH v11 7/12 x86/mm: use INVLPGB in flush_tlb_all
In reply to Rik van RielThe flush_tlb_all() function is not used a whole lot, but we might as well use broadcast TLB flushing there, too.
--- arch/x86/mm/tlb.c | 12 ++++++++++++ 1 file changed, 12 insertions(+)
arch/x86/mm/tlb.c +12 -0
@@ -1065,6 +1065,16 @@ void flush_tlb_mm_range(struct mm_struct *mm, unsigned long start, } +static bool broadcast_flush_tlb_all (void ) +{ +if (!cpu_feature_enabled (X86_FEATURE_INVLPGB )) +return false; + +guard (preempt)(); +invlpgb_flush_all (); +return true; +} +static void do_flush_tlb_all (void *info) {count_vm_tlb_event (NR_TLB_REMOTE_FLUSH_RECEIVED );
@@ -1073,6 +1083,8 @@ static void do_flush_tlb_all(void *info)void flush_tlb_all (void ) { +if (broadcast_flush_tlb_all ()) +return ;count_vm_tlb_event (NR_TLB_REMOTE_FLUSH );on_each_cpu (do_flush_tlb_all,NULL ,1 ); }
Signature · 1 line
2.47.1
Re: [PATCH v11 07/12] x86/mm: use INVLPGB in flush_tlb_all
arch/x86/mm/tlb.c static void do_flush_tlb_all(void *info) · line 1090
Quoted 7 lines · Rik van Riel
void
flush_tlb_all(void)
{arch/x86/mm/tlb.c +2
-0+if (broadcast_flush_tlb_all ()) +return ;count_vm_tlb_event (NR_TLB_REMOTE_FLUSH );on_each_cpu (do_flush_tlb_all,NULL ,1 ); }
This could use a couple of one-line comments.
Also, let's just keep the NR_TLB_REMOTE_FLUSH manipulation even when using the INVLPGB. It's still logically a remote flush even if it doesn't use an IPI.
PATCH v11 8/12 x86/mm: use broadcast TLB flushing for page reclaim TLB flushing
In reply to Rik van RielIn the page reclaim code, we only track the CPU(s) where the TLB needs to be flushed, rather than all the individual mappings that may be getting invalidated.
Use broadcast TLB flushing when that is available.
--- arch/x86/mm/tlb.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
arch/x86/mm/tlb.c +3 -1
@@ -1316,7 +1316,9 @@ void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch)* a local TLB flush is needed. Optimize this use-case by calling * flush_tlb_func_local() directly in this case. */ -if (cpumask_any_but (&batch->cpumask, cpu) < nr_cpu_ids) { +if (cpu_feature_enabled (X86_FEATURE_INVLPGB )) { +invlpgb_flush_all_nonglobals (); + }else if (cpumask_any_but (&batch->cpumask, cpu) < nr_cpu_ids) {flush_tlb_multi (&batch->cpumask, info); }else if (cpumask_test_cpu (cpu, &batch->cpumask)) {lockdep_assert_irqs_enabled ();
Signature · 1 line
2.47.1
Re: [PATCH v11 08/12] x86/mm: use broadcast TLB flushing for page reclaim TLB flushing
Quoted 5 lines · Rik van Riel
In the page reclaim code, we only track the CPU(s) where the TLB needs to be flushed, rather than all the individual mappings that may be getting invalidated.
Use broadcast TLB flushing when that is available.
The changelog here is a little light. This patch is doing a *ton* of stuff.
The existing code has two cases where it is doing a full TLB flush, not a ranged flush.
1. An actual IPI to some CPUs in batch->cpumask
2. A local flush, no IPI
The change here eliminates both of those options, even the "common case" which is not sending an IPI at all. So this replaces a CPU-local (aka. 1 logical CPU) TLB flush with a broadcast to the *ENTIRE* system. That's a really really big change to not be noted. It's not something that's an obvious win to me.
Quoted 5 lines
arch/x86/mm/tlb.c +0
-0
arch/x86/mm/tlb.c void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch) · line 1324
Quoted 10 lines
arch/x86/mm/tlb.c +3
-1@@ -1316,7 +1316,9 @@ void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch)* a local TLB flush is needed. Optimize this use-case by calling * flush_tlb_func_local() directly in this case. */ -if (cpumask_any_but (&batch->cpumask, cpu) < nr_cpu_ids) { +if (cpu_feature_enabled (X86_FEATURE_INVLPGB )) { +invlpgb_flush_all_nonglobals (); + }else if (cpumask_any_but (&batch->cpumask, cpu) < nr_cpu_ids) {flush_tlb_multi (&batch->cpumask, info); }else if (cpumask_test_cpu (cpu, &batch->cpumask)) {lockdep_assert_irqs_enabled ();
The structure of the code is also a bit off to me. O'd kinda prefer that we stick the pattern of (logically):
if (cpu_feature_enabled(X86_FEATURE_INVLPGB)) {
invlpgb_...();
} else {
on_each_cpu*();
}This patch is going a couple of functions up in the call chain above the on_each_cpu()'s.
It would be more consistent with the previous modifications in this series if the X86_FEATURE_INVLPGB check was in native_flush_tlb_multi(), instead.
Would that make sense here? It would also preserve the "common case" optimization that's in arch_tlbbatch_flush().
Re: [PATCH v11 08/12] x86/mm: use broadcast TLB flushing for page reclaim TLB flushing
Quoted 4 lines · Dave Hansen
Would that make sense here? It would also preserve the "common case" optimization that's in
arch_tlbbatch_flush().
What we do in this patch goes out the window in patch
10. This is just a temporary stage along the way to what we have at the end of the series, needed to make sure we don't break bisect partway through the series.
I'm not sure we should be doing much with this patch.
I could fold it into the next patch, but that would make things harder to read.
Signature · 1 line
All Rights Reversed.
Re: [PATCH v11 08/12] x86/mm: use broadcast TLB flushing for page reclaim TLB flushing
Quoted 9 lines · Rik van Riel
Quoted 3 lines · Dave Hansen
Would that make sense here? It would also preserve the "common case" optimization that's in
arch_tlbbatch_flush().What we do in this patch goes out the window in patch
10. This is just a temporary stage along the way to what we have at the end of the series, needed to make sure we don't break bisect partway through the series.
Quoted 4 lines
I'm not sure we should be doing much with this patch.
I could fold it into the next patch, but that would make things harder to read.
Ahh, that makes sense. If it's going out the window, could it be temporarily commented, please? Add the comment here and remove it in patch 10 (or whatever).
Re: [PATCH v11 08/12] x86/mm: use broadcast TLB flushing for page reclaim TLB flushing
Quoted 5 lines · Dave Hansen
Ahh, that makes sense. If it's going out the window, could it be temporarily commented, please? Add the comment here and remove it in patch 10 (or whatever).
I added an explanation in the commit message.
You are right that it wasn't totally clear.
Signature · 1 line
All Rights Reversed.
PATCH v11 9/12 x86/mm: enable broadcast TLB invalidation for multi-threaded processes
In reply to Rik van RielUse broadcast TLB invalidation, using the INVPLGB instruction, on AMD EPYC 3 and newer CPUs.
In order to not exhaust PCID space, and keep TLB flushes local for single threaded processes, we only hand out broadcast ASIDs to processes active on 4 or more CPUs.
--- arch/x86/include/asm/mmu.h | 6 + arch/x86/include/asm/mmu_context.h | 14 ++ arch/x86/include/asm/tlbflush.h | 73 ++++++ arch/x86/mm/tlb.c | 341 ++++++++++++++++++++++++++++- 4 files changed, 422 insertions(+), 12 deletions(-)
arch/x86/include/asm/mmu.h +6 -0
@@ -69,6 +69,12 @@ typedef struct {u16 pkey_allocation_map;s16 execute_only_pkey;#endif + +#ifdef CONFIG_X86_BROADCAST_TLB_FLUSH +u16 global_asid; +bool asid_transition; +#endif + } mm_context_t;#define INIT_MM_CONTEXT (mm) \
arch/x86/include/asm/mmu_context.h +14 -0
@@ -139,6 +139,8 @@ static inline void mm_reset_untag_mask(struct mm_struct *mm)#define enter_lazy_tlb enter_lazy_tlbextern void enter_lazy_tlb (struct mm_struct *mm,struct task_struct *tsk); +extern void destroy_context_free_global_asid (struct mm_struct *mm); +/* * Init a new mm. Used on mm copies, like at fork() * and on mm's that are brand-new, like at execve().
@@ -161,6 +163,14 @@ static inline int init_new_context(struct task_struct *tsk, mm->context.execute_only_pkey = -1; }#endif + +#ifdef CONFIG_X86_BROADCAST_TLB_FLUSH +if (cpu_feature_enabled (X86_FEATURE_INVLPGB )) { + mm->context.global_asid =0 ; + mm->context.asid_transition = false; + } +#endif +mm_reset_untag_mask (mm);init_new_context_ldt (mm);return 0 ;
@@ -170,6 +180,10 @@ static inline int init_new_context(struct task_struct *tsk,static inline void destroy_context (struct mm_struct *mm) {destroy_context_ldt (mm); +#ifdef CONFIG_X86_BROADCAST_TLB_FLUSH +if (cpu_feature_enabled (X86_FEATURE_INVLPGB )) +destroy_context_free_global_asid (mm); +#endif } extern void switch_mm(struct mm_struct *prev, struct mm_struct *next,
arch/x86/include/asm/tlbflush.h +73 -0
85 lines not on this page · raw message
arch/x86/mm/tlb.c +329 -12
423 lines not on this page · raw message
Signature · 1 line
2.47.1
Re: [PATCH v11 09/12] x86/mm: enable broadcast TLB invalidation for multi-threaded processes
Quoted 2 lines · Rik van Riel
Use broadcast TLB invalidation, using the INVPLGB instruction, on AMD EPYC 3 and newer CPUs.
Could we please just zap the "on AMD EPYC 3 and newer CPUs" from all of these patches? It can be mentioned once in the cover letter or something, but it doesn't need to be repeated.
Quoted 3 lines
In order to not exhaust PCID space, and keep TLB flushes local for single threaded processes, we only hand out broadcast ASIDs to processes active on 4 or more CPUs.
Please no "we's". Use imperative voice. This also needs some fleshing out. Perhaps:
There is not enough room in the 12-bit ASID address space to hand out broadcast ASIDs to every process. Only hand out broadcast ASIDs to processes when they are observed to be simultaneously running on 4 or more CPUs.
Most importantly, this ensures that single threaded processes continue to use the cheaper, legacy, local TLB invalidation instructions like INVLPG.
arch/x86/include/asm/mmu.h typedef struct { · line 78
Quoted 6 lines
arch/x86/include/asm/mmu.h +5
-0+#ifdef CONFIG_X86_BROADCAST_TLB_FLUSH +u16 global_asid; +bool asid_transition; +#endif + } mm_context_t;
Please give these at least a line or two comment explaining what they do.
Quoted 54 lines
#define
INIT_MM_CONTEXT(mm)\arch/x86/include/asm/mmu_context.h +14
-0@@ -139,6 +139,8 @@ static inline void mm_reset_untag_mask(struct mm_struct *mm)#define enter_lazy_tlb enter_lazy_tlbextern void enter_lazy_tlb (struct mm_struct *mm,struct task_struct *tsk); +extern void destroy_context_free_global_asid (struct mm_struct *mm); +/* * Init a new mm. Used on mm copies, like at fork() * and on mm's that are brand-new, like at execve(). @@ -161,6 +163,14 @@ static inline int init_new_context(struct task_struct *tsk, mm->context.execute_only_pkey = -1; }#endif + +#ifdef CONFIG_X86_BROADCAST_TLB_FLUSH +if (cpu_feature_enabled (X86_FEATURE_INVLPGB )) { + mm->context.global_asid =0 ; + mm->context.asid_transition = false; + } +#endif +mm_reset_untag_mask (mm);init_new_context_ldt (mm);return 0 ;@@ -170,6 +180,10 @@ static inline int init_new_context(struct task_struct *tsk,static inline void destroy_context (struct mm_struct *mm) {destroy_context_ldt (mm); +#ifdef CONFIG_X86_BROADCAST_TLB_FLUSH +if (cpu_feature_enabled (X86_FEATURE_INVLPGB )) +destroy_context_free_global_asid (mm); +#endif } extern void switch_mm(struct mm_struct *prev, struct mm_struct *next,arch/x86/include/asm/tlbflush.h +1
-0@@ -6,6 +6,7 @@#include <linux/mmu_notifier.h> #include <linux/sched.h> +#include <asm/barrier.h> #include <asm/processor.h> #include <asm/cpufeature.h> #include <asm/special_insns.h> @@ -239,6 +240,78 @@ void flush_tlb_one_kernel(unsigned long addr); void flush_tlb_multi(const struct cpumask *cpumask,
arch/x86/include/asm/tlbflush.h void flush_tlb_one_kernel(unsigned long addr); · line 250
Quoted 10 lines
arch/x86/include/asm/tlbflush.h +8
-0const struct flush_tlb_info *info); +#ifdef CONFIG_X86_BROADCAST_TLB_FLUSH +static inline bool is_dyn_asid (u16 asid) +{ +if (!cpu_feature_enabled (X86_FEATURE_INVLPGB )) +return true; + +return asid <TLB_NR_DYN_ASIDS ; +}
If possible, could you avoid double-defining the helpers that will compile with and without CONFIG_X86_BROADCAST_TLB_FLUSH? Just put the #ifdef around the ones that *need* it.
...
Quoted 12 lines
arch/x86/include/asm/tlbflush.h +12
-0+static inline bool in_asid_transition (struct mm_struct *mm) +{ +if (!cpu_feature_enabled (X86_FEATURE_INVLPGB )) +return false; + +return mm &&READ_ONCE (mm->context.asid_transition); +} + +static inline u16 mm_global_asid (struct mm_struct *mm) +{ +u16 asid; +
arch/x86/include/asm/tlbflush.h void flush_tlb_one_kernel(unsigned long addr); · line 278
Quoted 10 lines
arch/x86/include/asm/tlbflush.h +10
-0+if (!cpu_feature_enabled (X86_FEATURE_INVLPGB )) +return 0 ; + + asid =smp_load_acquire (&mm->context.global_asid); + +/* mm->context.global_asid is either 0, or a global ASID */ +VM_WARN_ON_ONCE (asid &&is_dyn_asid (asid)); + +return asid; +}
Yay, some kind of custom lock. :)
Could give us a little idea what the locking rules are here and why this neds the READ_ONCE() and smp_load_acquire()?
...
arch/x86/include/asm/tlbflush.h void flush_tlb_one_kernel(unsigned long addr); · line 276
Quoted 4 lines
arch/x86/include/asm/tlbflush.h +4
-0+/* +* TLB consistency for global ASIDs is maintained with broadcast TLB +* flushing. The TLB is never outdated, and does not need flushing. +*/
This is another case where I think using the word "broadcast" is not helping.
Here's the problem: INVLPGB is a "INVLPG" that's broadcast. So the name INVLPGB is fine. INVLPGB is *a* way to broadcast INVLPG which is *a* kind of TLB invalidation.
But, to me "broadcast TLB flushing" is a broad term. In arguably includes INVLPGB and normal IPI-based flushing. Just like the function naming in the earlier patch, I think we need a better term here.
arch/x86/include/asm/tlbflush.h void flush_tlb_one_kernel(unsigned long addr); · line 289
Quoted 13 lines
arch/x86/include/asm/tlbflush.h +10
-0+if (static_cpu_has (X86_FEATURE_INVLPGB )) { +u16 global_asid =mm_global_asid (next); + +if (global_asid) { + *new_asid = global_asid; + *need_flush = false; +return ; + } + } +#ifdef CONFIG_X86_BROADCAST_TLB_FLUSH
How cleanly could we throw this hunk in a new file? I always dislike big #ifdefs like this. They seem like magnets for causing weird compile problems.
arch/x86/include/asm/tlbflush.h void flush_tlb_one_kernel(unsigned long addr); · line 293
Quoted 4 lines
arch/x86/include/asm/tlbflush.h +4
-0+/* +* Logic for broadcast TLB invalidation. +*/ +static DEFINE_RAW_SPINLOCK (global_asid_lock);
The global lock definitely needs some discussion in the changelog.
arch/x86/include/asm/tlbflush.h void flush_tlb_one_kernel(unsigned long addr); · line 296
Quoted 3 lines
arch/x86/include/asm/tlbflush.h +3
-0+static u16 last_global_asid =MAX_ASID_AVAILABLE ; +static DECLARE_BITMAP(global_asid_used, MAX_ASID_AVAILABLE) = { 0 }; +static DECLARE_BITMAP(global_asid_freed, MAX_ASID_AVAILABLE) = { 0 };
Isn't the initialization to all 0's superfluous for a global variable?
arch/x86/include/asm/tlbflush.h void flush_tlb_one_kernel(unsigned long addr); · line 308
Quoted 12 lines
arch/x86/include/asm/tlbflush.h +12
-0+static int global_asid_available =MAX_ASID_AVAILABLE -TLB_NR_DYN_ASIDS -1 ; + +static void reset_global_asid_space (void ) +{ +lockdep_assert_held (&global_asid_lock); + +/* +* A global TLB flush guarantees that any stale entries from +* previously freed global ASIDs get flushed from the TLB +* everywhere, making these global ASIDs safe to reuse. +*/ +invlpgb_flush_all_nonglobals ();
Ugh, my suggestion to use the term "global ASID" really doesn't work here, does it?
Also, isn't a invlpgb_flush_all_nonglobals() _relatively_ slow? It has to go out and talk over the fabric to every CPU, right? This is also holding a global lock.
That's seems worrisome.
Quoted 52 lines
arch/x86/include/asm/tlbflush.h +52
-0+/* +* Clear all the previously freed global ASIDs from the +* broadcast_asid_used bitmap, now that the global TLB flush +* has made them actually available for re-use. +*/ +bitmap_andnot (global_asid_used, global_asid_used, + global_asid_freed,MAX_ASID_AVAILABLE ); +bitmap_clear (global_asid_freed,0 ,MAX_ASID_AVAILABLE ); + +/* +* ASIDs 0-TLB_NR_DYN_ASIDS are used for CPU-local ASID +* assignments, for tasks doing IPI based TLB shootdowns. +* Restart the search from the start of the global ASID space. +*/ + last_global_asid =TLB_NR_DYN_ASIDS ; +} + +static u16 get_global_asid (void ) +{ + +u16 asid; + +lockdep_assert_held (&global_asid_lock); + +/* The previous allocated ASID is at the top of the address space. */ +if (last_global_asid >=MAX_ASID_AVAILABLE -1 ) +reset_global_asid_space (); + + asid =find_next_zero_bit (global_asid_used,MAX_ASID_AVAILABLE , last_global_asid); + +if (asid >=MAX_ASID_AVAILABLE ) { +/* This should never happen. */ +VM_WARN_ONCE (1 ,"Unable to allocate global ASID despite %d available\n" , + global_asid_available); +return 0 ; + } + +/* Claim this global ASID. */ +__set_bit (asid, global_asid_used); + last_global_asid = asid; + global_asid_available--; +return asid; +} + +/* +* Returns true if the mm is transitioning from a CPU-local ASID to a global +* (INVLPGB) ASID, or the other way around. +*/ +static bool needs_global_asid_reload (struct mm_struct *next,u16 prev_asid) +{ +u16 global_asid =mm_global_asid (next); +
arch/x86/include/asm/tlbflush.h void flush_tlb_one_kernel(unsigned long addr);
Quoted 10 lines
arch/x86/include/asm/tlbflush.h +10
-0+/* Process is transitioning to a global ASID */ +if (global_asid && prev_asid != global_asid) +return true; + +/* Transition from global->local ASID does not currently happen. */ +if (!global_asid &&is_global_asid (prev_asid)) +return true; + +return false; +}
I'm going to throw in the towel at this point. This patch needs to get broken up. It's more at once than my poor little brain can handle.
The _least_ it can do is introduce the stub functions and injection into existing code changes, first. Then, in a second patch, introduce the real implementation.
I also suspect that a big chunk of the ASID allocator could be broken out and introduced separately.
Another example is broadcast_tlb_flush(). To reduce complexity in _this_ patch, it could do something suboptimal like always do a invlpgb_flush_all_nonglobals() regardless of the kind of flush it gets. Then, in a later patch, you could optimize it.
Re: [PATCH v11 09/12] x86/mm: enable broadcast TLB invalidation for multi-threaded processes
Quoted 8 lines · Dave Hansen
arch/x86/mm/tlb.c static void choose_new_asid(struct mm_struct *next, u64 next_tlb_gen, · line 233
Quoted 4 lines
arch/x86/mm/tlb.c +4
-0+/* +* TLB consistency for global ASIDs is maintained with broadcast TLB +* flushing. The TLB is never outdated, and does not need flushing. +*/ This is another case where I think using the word "broadcast" is not helping.
Quoted 7 lines
Here's the problem: INVLPGB is a "INVLPG" that's broadcast. So the name INVLPGB is fine. INVLPGB is *a* way to broadcast INVLPG which is *a* kind of TLB invalidation.
But, to me "broadcast TLB flushing" is a broad term. In arguably includes INVLPGB and normal IPI-based flushing. Just like the function naming in the earlier patch, I think we need a better term here.
If we wanna refer to invlpgb-type things without saying invlpgb, and as you pointed out "broadcast" is too general, I think we just need the terminology to refer to the fact it's a special "hardware" feature.
So "hardware-synchronized" or something like that. The former would abbrev to hwsync in identifiers, without any loss of clarity.
Re: [PATCH v11 09/12] x86/mm: enable broadcast TLB invalidation for multi-threaded processes
In reply to Dave HansenQuoted 10 lines · Dave Hansen
Quoted 1 line · Rik van Riel
+#ifdef
CONFIG_X86_BROADCAST_TLB_FLUSHHow cleanly could we throw this hunk in a new file? I always dislike big
#ifdefs like this. They seem like magnets for causing weird compile problems.
It looks like if we compile the X86_FEATURE_INVLPGB out through arch/x86/include/asm/disabled-features.h, the compiler can be smart enough to elide the no longer necessary code ... but only if we have these functions in the same compilation unit as their callers.
That means we should be able to get rid of the ifdef, if we keep these functions in arch/x86/mm/tlb.c
Quoted 8 lines
arch/x86/mm/tlb.c static void choose_new_asid(struct mm_struct *next, u64 next_tlb_gen, · line 274
Quoted 5 lines
arch/x86/mm/tlb.c +4
-0+/* +* Logic for broadcast TLB invalidation. +*/ +static DEFINE_RAW_SPINLOCK (global_asid_lock);The global lock definitely needs some discussion in the changelog.
Quoted 9 lines
Quoted 5 lines
(quoted diff) +2
-0+static u16 last_global_asid = MAX_ASID_AVAILABLE; +static DECLARE_BITMAP(global_asid_used, MAX_ASID_AVAILABLE) = { 0};
+staticDECLARE_BITMAP(global_asid_freed, MAX_ASID_AVAILABLE)= { 0 };Isn't the initialization to all 0's superfluous for a global variable?
I'll remove that, and will add documentation for the spinlock.
Quoted 18 lines
Quoted 10 lines
+static int
global_asid_available=MAX_ASID_AVAILABLE-TLB_NR_DYN_ASIDS- 1;arch/x86/mm/tlb.c +7
-0+ +static void reset_global_asid_space (void ) +{ +lockdep_assert_held (&global_asid_lock); + +/* +* A global TLB flush guarantees that any stale entries from
arch/x86/mm/tlb.c static void choose_new_asid(struct mm_struct *next, u64 next_tlb_gen, · line 289
Quoted 4 lines
arch/x86/mm/tlb.c +4
-0+* previously freed global ASIDs get flushed from the TLB +* everywhere, making these global ASIDs safe to reuse. +*/ +invlpgb_flush_all_nonglobals ();Ugh, my suggestion to use the term "global ASID" really doesn't work here, does it?
Quoted 6 lines
Also, isn't a
invlpgb_flush_all_nonglobals()_relatively_ slow? It has
to go out and talk over the fabric to every CPU, right? This is also holding a global lock.That's seems worrisome.
This only happens on ASID rollover, when we have reached the end of global ASID space, and are about to restart the search from the beginning.
We do not do a flush at every allocation, but only once every few thousand global ASID allocations.
Quoted 29 lines
Quoted 12 lines
+static bool
needs_global_asid_reload(struct mm_struct *next, u16 prev_asid)arch/x86/mm/tlb.c +8
-0+{ +u16 global_asid =mm_global_asid (next); + +/* Process is transitioning to a global ASID */ +if (global_asid && prev_asid != global_asid) +return true; + +/* Transition from global->local ASID does not currently happen. */
arch/x86/mm/tlb.c static void choose_new_asid(struct mm_struct *next, u64 next_tlb_gen, · line 352
Quoted 5 lines
arch/x86/mm/tlb.c +5
-0+if (!global_asid &&is_global_asid (prev_asid)) +return true; + +return false; +}I'm going to throw in the towel at this point. This patch needs to get
broken up. It's more at once than my poor little brain can handle.The _least_ it can do is introduce the stub functions and injection into
existing code changes, first. Then, in a second patch, introduce the real implementation.I also suspect that a big chunk of the ASID allocator could be broken out and introduced separately.
Quoted 7 lines
Another example is
broadcast_tlb_flush(). To reduce complexity in _this_
patch, it could do something suboptimal like always do ainvlpgb_flush_all_nonglobals()regardless of the kind of flush it gets.
Then, in a later patch, you could optimize it.
With the config option and ifdefs (mostly) gone, I think the split would probably have to be in two patches:
1) Modify existing code to call non-functional stub functions.
2) Modify the stub functions to then do something.
I'm not sure quite how much this will help with review, since to review the second patch you'll have to go back to the first patch, but I might as well try...
Signature · 1 line
All Rights Reversed.
PATCH v11 10/12 x86/mm: do targeted broadcast flushing from tlbbatch code
In reply to Rik van RielInstead of doing a system-wide TLB flush from arch_tlbbatch_flush, queue up asynchronous, targeted flushes from arch_tlbbatch_add_pending.
This also allows us to avoid adding the CPUs of processes using broadcast flushing to the batch->cpumask, and will hopefully further reduce TLB flushing from the reclaim and compaction paths.
--- arch/x86/include/asm/invlpgb.h | 20 ++++----- arch/x86/include/asm/tlbflush.h | 19 ++++---- arch/x86/mm/tlb.c | 79 +++++++++++++++++++++++++++++++-- 3 files changed, 96 insertions(+), 22 deletions(-)
arch/x86/include/asm/invlpgb.h +10 -10
@@ -31,7 +31,7 @@ static inline void __invlpgb(unsigned long asid, unsigned long pcid, }/* Wait for INVLPGB originated by this CPU to complete. */ -static inline void tlbsync (void ) +static inline void __tlbsync (void ) {cant_migrate ();/* TLBSYNC: supported in binutils >= 0.36. */
@@ -61,19 +61,19 @@ static inline void invlpgb_flush_user(unsigned long pcid, unsigned long addr) {__invlpgb (0 , pcid, addr,0 ,0 ,INVLPGB_PCID |INVLPGB_VA ); -tlbsync (); +__tlbsync (); } -static inline void invlpgb_flush_user_nr_nosync (unsigned long pcid, -unsigned long addr, -u16 nr, -bool pmd_stride) +static inline void __invlpgb_flush_user_nr_nosync (unsigned long pcid, +unsigned long addr, +u16 nr, +bool pmd_stride) {__invlpgb (0 , pcid, addr, nr -1 , pmd_stride,INVLPGB_PCID |INVLPGB_VA ); }/* Flush all mappings for a given PCID, not including globals. */ -static inline void invlpgb_flush_single_pcid_nosync (unsigned long pcid) +static inline void __invlpgb_flush_single_pcid_nosync (unsigned long pcid) {__invlpgb (0 , pcid,0 ,0 ,0 ,INVLPGB_PCID ); }
@@ -82,11 +82,11 @@ static inline void invlpgb_flush_single_pcid_nosync(unsigned long pcid)static inline void invlpgb_flush_all (void ) {__invlpgb (0 ,0 ,0 ,0 ,0 ,INVLPGB_INCLUDE_GLOBAL ); -tlbsync (); +__tlbsync (); }/* Flush addr, including globals, for all PCIDs. */ -static inline void invlpgb_flush_addr_nosync (unsigned long addr,u16 nr) +static inline void __invlpgb_flush_addr_nosync (unsigned long addr,u16 nr) {__invlpgb (0 ,0 , addr, nr -1 ,0 ,INVLPGB_INCLUDE_GLOBAL ); }
@@ -95,7 +95,7 @@ static inline void invlpgb_flush_addr_nosync(unsigned long addr, u16 nr)static inline void invlpgb_flush_all_nonglobals (void ) {__invlpgb (0 ,0 ,0 ,0 ,0 ,0 ); -tlbsync (); +__tlbsync (); }#endif /* _ASM_X86_INVLPGB */
arch/x86/include/asm/tlbflush.h +10 -9
@@ -106,6 +106,9 @@ struct tlb_state {* need to be invalidated. */ bool invalidate_other; +#ifdef CONFIG_X86_BROADCAST_TLB_FLUSH +bool need_tlbsync; +#endif #ifdef CONFIG_ADDRESS_MASKING /*
@@ -310,6 +313,10 @@ static inline void broadcast_tlb_flush(struct flush_tlb_info *info)static inline void consider_global_asid (struct mm_struct *mm) { } + +static inline void tlbsync (void ) +{ +}#endif #ifdef CONFIG_PARAVIRT
@@ -359,21 +366,15 @@ static inline u64 inc_mm_tlb_gen(struct mm_struct *mm)return atomic64_inc_return (&mm->context.tlb_gen); } -static inline void arch_tlbbatch_add_pending (struct arch_tlbflush_unmap_batch *batch, -struct mm_struct *mm, -unsigned long uaddr) -{ -inc_mm_tlb_gen (mm); -cpumask_or (&batch->cpumask, &batch->cpumask,mm_cpumask (mm)); -mmu_notifier_arch_invalidate_secondary_tlbs (mm,0 ,-1UL ); -} -static inline void arch_flush_tlb_batched_pending (struct mm_struct *mm) {flush_tlb_mm (mm); }extern void arch_tlbbatch_flush (struct arch_tlbflush_unmap_batch *batch); +extern void arch_tlbbatch_add_pending (struct arch_tlbflush_unmap_batch *batch, +struct mm_struct *mm, +unsigned long uaddr); static inline bool pte_flags_need_flush(unsigned long oldflags, unsigned long newflags,
arch/x86/mm/tlb.c +76 -3
115 lines not on this page · raw message
Signature · 1 line
2.47.1
PATCH v11 11/12 x86/mm: enable AMD translation cache extensions
In reply to Rik van RielWith AMD TCE (translation cache extensions) only the intermediate mappings that cover the address range zapped by INVLPG / INVLPGB get invalidated, rather than all intermediate mappings getting zapped at every TLB invalidation.
This can help reduce the TLB miss rate, by keeping more intermediate mappings in the cache.
From the AMD manual:
Translation Cache Extension (TCE) Bit. Bit 15, read/write. Setting this bit to 1 changes how the INVLPG, INVLPGB, and INVPCID instructions operate on TLB entries. When this bit is 0, these instructions remove the target PTE from the TLB as well as all upper-level table entries that are cached in the TLB, whether or not they are associated with the target PTE. When this bit is set, these instructions will remove the target PTE and only those upper-level entries that lead to the target PTE in the page table hierarchy, leaving unrelated upper-level entries intact.
--- arch/x86/include/asm/msr-index.h | 2 ++ arch/x86/kernel/cpu/amd.c | 4 ++++ tools/arch/x86/include/asm/msr-index.h | 2 ++ 3 files changed, 8 insertions(+)
arch/x86/include/asm/msr-index.h +2 -0
@@ -25,6 +25,7 @@#define _EFER_SVME 12/* Enable virtualization */ #define _EFER_LMSLE 13/* Long Mode Segment Limit Enable */ #define _EFER_FFXSR 14/* Enable Fast FXSAVE/FXRSTOR */ +#define _EFER_TCE 15/* Enable Translation Cache Extensions */ #define _EFER_AUTOIBRS 21/* Enable Automatic IBRS */ #define EFER_SCE (1<<_EFER_SCE)
@@ -34,6 +35,7 @@#define EFER_SVME (1<<_EFER_SVME)#define EFER_LMSLE (1<<_EFER_LMSLE)#define EFER_FFXSR (1<<_EFER_FFXSR) +#define EFER_TCE (1<<_EFER_TCE)#define EFER_AUTOIBRS (1<<_EFER_AUTOIBRS)/*
arch/x86/kernel/cpu/amd.c +4 -0
@@ -1075,6 +1075,10 @@ static void init_amd(struct cpuinfo_x86 *c)/* AMD CPUs don't need fencing after x2APIC/TSC_DEADLINE MSR writes. */ clear_cpu_cap (c,X86_FEATURE_APIC_MSRS_FENCE ); + +/* Enable Translation Cache Extension */ +if (cpu_feature_enabled (X86_FEATURE_TCE )) +msr_set_bit (MSR_EFER , _EFER_TCE); }#ifdef CONFIG_X86_32
tools/arch/x86/include/asm/msr-index.h +2 -0
@@ -25,6 +25,7 @@#define _EFER_SVME 12/* Enable virtualization */ #define _EFER_LMSLE 13/* Long Mode Segment Limit Enable */ #define _EFER_FFXSR 14/* Enable Fast FXSAVE/FXRSTOR */ +#define _EFER_TCE 15/* Enable Translation Cache Extensions */ #define _EFER_AUTOIBRS 21/* Enable Automatic IBRS */ #define EFER_SCE (1<<_EFER_SCE)
@@ -34,6 +35,7 @@#define EFER_SVME (1<<_EFER_SVME)#define EFER_LMSLE (1<<_EFER_LMSLE)#define EFER_FFXSR (1<<_EFER_FFXSR) +#define EFER_TCE (1<<_EFER_TCE)#define EFER_AUTOIBRS (1<<_EFER_AUTOIBRS)/*
Signature · 1 line
2.47.1
PATCH v11 12/12 x86/mm: only invalidate final translations with INVLPGB
In reply to Rik van RielUse the INVLPGB_FINAL_ONLY flag when invalidating mappings with INVPLGB. This way only leaf mappings get removed from the TLB, leaving intermediate translations cached.
On the (rare) occasions where we free page tables we do a full flush, ensuring intermediate translations get flushed from the TLB.
--- arch/x86/include/asm/invlpgb.h | 10 ++++++++-- arch/x86/mm/tlb.c | 13 +++++++------ 2 files changed, 15 insertions(+), 8 deletions(-)
arch/x86/include/asm/invlpgb.h +8 -2
@@ -67,9 +67,15 @@ static inline void invlpgb_flush_user(unsigned long pcid,static inline void __invlpgb_flush_user_nr_nosync (unsigned long pcid,unsigned long addr,u16 nr, -bool pmd_stride) +bool pmd_stride, +bool freed_tables) { -__invlpgb (0 , pcid, addr, nr -1 , pmd_stride,INVLPGB_PCID |INVLPGB_VA ); +u8 flags =INVLPGB_PCID |INVLPGB_VA ; + +if (!freed_tables) + flags |=INVLPGB_FINAL_ONLY ; + +__invlpgb (0 , pcid, addr, nr -1 , pmd_stride, flags); }/* Flush all mappings for a given PCID, not including globals. */
arch/x86/mm/tlb.c +7 -6
@@ -502,9 +502,10 @@ static inline void tlbsync(void)static inline void invlpgb_flush_user_nr_nosync (unsigned long pcid,unsigned long addr, -u16 nr,bool pmd_stride) +u16 nr,bool pmd_stride, +bool freed_tables) { -__invlpgb_flush_user_nr_nosync (pcid, addr, nr, pmd_stride); +__invlpgb_flush_user_nr_nosync (pcid, addr, nr, pmd_stride, freed_tables);if (!this_cpu_read (cpu_tlbstate.need_tlbsync))this_cpu_write (cpu_tlbstate.need_tlbsync, true); }
@@ -547,9 +548,9 @@ static void broadcast_tlb_flush(struct flush_tlb_info *info) nr =clamp_val (nr,1 , invlpgb_count_max); } -invlpgb_flush_user_nr_nosync (kern_pcid (asid), addr, nr, pmd); +invlpgb_flush_user_nr_nosync (kern_pcid (asid), addr, nr, pmd, info->freed_tables);if (static_cpu_has (X86_FEATURE_PTI )) -invlpgb_flush_user_nr_nosync (user_pcid (asid), addr, nr, pmd); +invlpgb_flush_user_nr_nosync (user_pcid (asid), addr, nr, pmd, info->freed_tables); addr += nr << info->stride_shift; }while (addr < info->end);
@@ -1697,10 +1698,10 @@ void arch_tlbbatch_add_pending(struct arch_tlbflush_unmap_batch *batch,u16 asid =mm_global_asid (mm);if (asid) { -invlpgb_flush_user_nr_nosync (kern_pcid (asid), uaddr,1 , false); +invlpgb_flush_user_nr_nosync (kern_pcid (asid), uaddr,1 , false, false);/* Do any CPUs supporting INVLPGB need PTI? */ if (static_cpu_has (X86_FEATURE_PTI )) -invlpgb_flush_user_nr_nosync (user_pcid (asid), uaddr,1 , false); +invlpgb_flush_user_nr_nosync (user_pcid (asid), uaddr,1 , false, false);/* * Some CPUs might still be using a local ASID for this
Signature · 1 line
2.47.1
Quoted 2 lines · Rik van Riel
Add support for broadcast TLB invalidation using AMD's INVLPGB instruction.
Oh - what if the user sets nopcid. We need to disable invlpgb in that case right?
(I assume it could be made to work, but doesn't seem like a case worth optimising).
Quoted 10 lines · Brendan Jackman
Quoted 2 lines · Rik van Riel
Add support for broadcast TLB invalidation using AMD's INVLPGB instruction.
Oh - what if the user sets nopcid. We need to disable invlpgb in that case right?
(I assume it could be made to work, but doesn't seem like a case worth optimising).
BTW there are still a couple of unresolved threads from previous versions.
KPTI comment:
https://lore.kernel.org/linux-kernel/CA+i-1C2zuctxx6oPVVu0zBJ=Q=Hs73mgrWs5jsp8obARNcUS9g@mail.gmail.com/
and
https://lore.kernel.org/linux-kernel/[email protected]/
Clarify context-switch sync:
(there's another mention of it somewhere that I can't find)
Suggestion for WARN_ON_ONCE:
Quoted 8 lines · Brendan Jackman
Quoted 3 lines · Rik van Riel
Add support for broadcast TLB invalidation using AMD's INVLPGB instruction.
Oh - what if the user sets nopcid. We need to disable invlpgb in that case right?
That's automatic, when !X86_FEATURE_PCID, choose_new_asid() always returns 0, and we never assign a global ASID to a task on any CPU.
I suppose we could make "nopcid" a little more efficient by short-circuiting the code in consider_global_asid() as well, but it should already work correctly.
Peter, do you prefer a v12, or should additional small fixes and improvements just be sent in follow-up patches at this point?
Signature · 1 line
All Rights Reversed.
Peter Zijlstra
maintainerQuoted 4 lines · Rik van Riel
Peter, do you prefer a v12, or should
additional small fixes and improvements
just be sent in follow-up patches at
this point?
I think I'm mostly okay at this point, but DaveH was wanting to have a look at this thing. Dave did you get around to reading one of these recent versions, wdyt?
Brendan Jackman
Reviewed-byQuoted 11 lines · Peter Zijlstra
Quoted 4 lines · Rik van Riel
Peter, do you prefer a v12, or should
additional small fixes and improvements
just be sent in follow-up patches at
this point?I think I'm mostly okay at this point, but DaveH was wanting to have a look at this thing. Dave did you get around to reading one of these recent versions, wdyt?
I was waiting until my nitpicks were resolved but if you merge it before then, please add my: