Skip to contentSkip to conversation

AMD broadcast TLB invalidation

PATCH Superseded Rik van Riel 13 participants 90 messages Dec 30, 2024 – Jan 14, 2025

Subsystems (8)
Memory management x86 KVM x86 MM +4 more subsystems

v3 of 14 Superseded Latest is v14 →

lore
Share & tools
lore
#0 lore raw

v3 cover AMD broadcast TLB invalidation

Subject: [RFC PATCH 00/10] AMD broadcast TLB invalidation

Add support for broadcast TLB invalidation using AMD's INVLPGB instruction.

This allows the kernel to invalidate TLB entries on remote CPUs without needing to send IPIs, without having to wait for remote CPUs to handle those interrupts, and with less interruption to what was running on those CPUs.

Because x86 PCID space is limited, and there are some very large systems out there, broadcast TLB invalidation is only used for processes that are active on 3 or more CPUs, with the threshold being gradually increased the more the PCID space gets exhausted.

Combined with the removal of unnecessary lru_add_drain calls (see https://lkml.org/lkml/2024/12/19/1388) this results in a nice performance boost for the will-it-scale tlb_flush2_threads test on an AMD Milan system with 36 cores:

- vanilla kernel:           527k loops/second
- lru_add_drain removal:    731k loops/second
- only INVLPGB:             527k loops/second

- lru_add_drain + INVLPGB: 1157k loops/second

Profiling with only the INVLPGB changes showed while TLB invalidation went down from 40% of the total CPU time to only around 4% of CPU time, the contention simply moved to the LRU lock.

Fixing both at the same time about doubles the
number of iterations per second from this case.

v3:
- Remove paravirt tlb_remove_table call (thank you Qi Zheng)
- More suggested cleanups and changelog fixes by Peter and Nadav
v2:
- Apply suggestions by Peter and Borislav (thank you!)
- Fix bug in arch_tlbbatch_flush, where we need to do both the TLBSYNC, and flush the CPUs that are in the cpumask.
- Some updates to comments and changelogs based on questions.

#1 lore raw

PATCH v3 1/12 x86/mm: make MMU_GATHER_RCU_TABLE_FREE unconditional

Currently x86 uses CONFIG_MMU_GATHER_TABLE_FREE when using paravirt, and not when running on bare metal.

There is no real good reason to do things differently for each setup. Make them all the same.

After this change, the synchronization between get_user_pages_fast and page table freeing is handled by RCU, which prevents page tables from being reused for other data while get_user_pages_fast is walking them.

This allows us to invalidate page tables while other CPUs have interrupts disabled.

---
 arch/x86/Kconfig           | 2 +-
 arch/x86/kernel/paravirt.c | 7 +------
 2 files changed, 2 insertions(+), 7 deletions(-)
arch/x86/Kconfig +1 -1
index 9d7bd0ae48c4..e8743f8c9fd0 100644
@@ -274,7 +274,7 @@ config X86
 	select HAVE_PCI
 	select HAVE_PERF_REGS
 	select HAVE_PERF_USER_STACK_DUMP
-	select MMU_GATHER_RCU_TABLE_FREE	if PARAVIRT
+	select MMU_GATHER_RCU_TABLE_FREE
 	select MMU_GATHER_MERGE_VMAS
 	select HAVE_POSIX_CPU_TIMERS_TASK_WORK
 	select HAVE_REGS_AND_STACK_ACCESS_API
arch/x86/kernel/paravirt.c +1 -6
index fec381533555..2b78a6b466ed 100644
@@ -59,11 +59,6 @@ void __init native_pv_lock_init(void)
 		static_branch_enable(&virt_spin_lock_key);
 }
 
-static void native_tlb_remove_table(struct mmu_gather *tlb, void *table)
-{
-	tlb_remove_page(tlb, table);
-}
-
 struct static_key paravirt_steal_enabled;
 struct static_key paravirt_steal_rq_enabled;
 
@@ -191,7 +186,7 @@ struct paravirt_patch_template pv_ops = {
 	.mmu.flush_tlb_kernel	= native_flush_tlb_global,
 	.mmu.flush_tlb_one_user	= native_flush_tlb_one_user,
 	.mmu.flush_tlb_multi	= native_flush_tlb_multi,
-	.mmu.tlb_remove_table	= native_tlb_remove_table,
+	.mmu.tlb_remove_table	= tlb_remove_table,
 
 	.mmu.exit_mmap		= paravirt_nop,
 	.mmu.notify_page_enc_status_changed	= paravirt_nop,
Signature · 1 line
2.47.1
maintainer
#2 lore raw

Re: [PATCH 01/12] x86/mm: make MMU_GATHER_RCU_TABLE_FREE unconditional

Quoted 10 lines · Rik van Riel

Currently x86 uses CONFIG_MMU_GATHER_TABLE_FREE when using paravirt, and not when running on bare metal.

There is no real good reason to do things differently for each setup. Make them all the same.

After this change, the synchronization between get_user_pages_fast and page table freeing is handled by RCU, which prevents page tables from being reused for other data while get_user_pages_fast is walking them.

I'd rather like to read here why this is not a problem anymore and why

48a8b97cfd80 ("x86/mm: Only use tlb_remove_table() for paravirt")

is not relevant anymore.

Quoted 1 line

This allows us to invalidate page tables while other CPUs have

	      ^^

Please use passive voice in your commit message: no "we" or "I", etc, and describe your changes in imperative mood.

Personal pronouns are ambiguous in text, especially with so many parties/companies/etc developing the kernel so let's avoid them please.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#3 lore raw

Re: [PATCH 01/12] x86/mm: make MMU_GATHER_RCU_TABLE_FREE unconditional

Quoted 14 lines · Borislav Petkov
Quoted 6 lines · Rik van Riel

Currently x86 uses CONFIG_MMU_GATHER_TABLE_FREE when using paravirt, and not when running on bare metal.

There is no real good reason to do things differently for each setup. Make them all the same.

Quoted 6 lines

After this change, the synchronization between get_user_pages_fast and page table freeing is handled by RCU, which prevents page tables
from being reused for other data while get_user_pages_fast is walking
them.

Quoted 6 lines

I'd rather like to read here why this is not a problem anymore and why

  48a8b97cfd80 ("x86/mm: Only use tlb_remove_table() for paravirt")

is not relevant anymore.

That would be a question for Peter :)

Quoted 6 lines
Quoted 2 lines

This allows us to invalidate page tables while other CPUs have

	      ^^

Please use passive voice in your commit message: no "we" or "I", etc, and describe your changes in imperative mood.

Will do. Between your feedback, the suggestions from Qi and Nadav, and the kernel test robot flagging some build issues without CONFIG_CPU_SUP_AMD, there's enough to warrant a v4 :)

Signature · 1 line
All Rights Reversed.
maintainer
#4 lore raw

Re: [PATCH 01/12] x86/mm: make MMU_GATHER_RCU_TABLE_FREE unconditional

Quoted 4 lines · Rik van Riel

Will do. Between your feedback, the suggestions from Qi and Nadav, and the kernel test robot flagging some build issues without CONFIG_CPU_SUP_AMD, there's enough to warrant a v4 :)

Yes, just take your time pls. I need to get through the rest, first, in the coming days.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#5 lore raw

Re: [PATCH 01/12] x86/mm: make MMU_GATHER_RCU_TABLE_FREE unconditional

Quoted 10 lines · Borislav Petkov
Quoted 4 lines · Rik van Riel

Will do. Between your feedback, the suggestions from Qi and Nadav, and the kernel test robot flagging some build issues without CONFIG_CPU_SUP_AMD, there's enough to warrant a v4 :)

Yes, just take your time pls. I need to get through the rest, first, in the
coming days.

I'm incorporating the feedback I have so far, and will test with those improvements.

I'll wait for you to finish making your way through before coming up with a v4 :)

I do have a question about the second to last patch "x86/mm: enable AMD translation cache extensions"

It looks like with this patch the translation cache extensions get enabled (when I read back the MSR), but the AMD manual suggests I may need to enable EFER.TCE separately on every CPU?

Signature · 1 line
All Rights Reversed.
maintainer
#6 lore raw

Re: [PATCH 01/12] x86/mm: make MMU_GATHER_RCU_TABLE_FREE unconditional

Quoted 2 lines · Rik van Riel

I'll wait for you to finish making your way through before coming up with a v4 :)

Thanks!

Quoted 4 lines

It looks like with this patch the translation cache extensions get enabled (when I read back the MSR), but the AMD manual suggests I may need to enable EFER.TCE separately on every CPU?

Yes, you do. EFER is a per-thread MSR.

cpu/amd.c is where we usually do enable features.

Thx.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
maintainer
#7 lore raw

Re: [PATCH 01/12] x86/mm: make MMU_GATHER_RCU_TABLE_FREE unconditional

In reply to Rik van Riel
Quoted 15 lines · Rik van Riel
Quoted 14 lines · Borislav Petkov
Quoted 12 lines · Rik van Riel

Currently x86 uses CONFIG_MMU_GATHER_TABLE_FREE when using paravirt, and not when running on bare metal.

There is no real good reason to do things differently for each setup. Make them all the same.

After this change, the synchronization between get_user_pages_fast and page table freeing is handled by RCU, which prevents page tables
from being reused for other data while get_user_pages_fast is walking
them.

Quoted 8 lines
Quoted 6 lines

I'd rather like to read here why this is not a problem anymore and why

  48a8b97cfd80 ("x86/mm: Only use tlb_remove_table() for paravirt")

is not relevant anymore.

That would be a question for Peter :)

Well, I've already answered why we need this in the previous thread but it wasn't preserved :-(

Currently GUP-fast serializes against table-free by disabling interrupts, which in turn holds of the TLBI-IPIs.

Since you're going to be doing broadcast TLBI -- without IPIs, this no longer works and we need other means of serializing GUP-fast vs table-free.

MMU_GATHER_RCU_TABLE_FREE is that means.

So where previously paravirt implementations of tlb_flush_multi might require this (because of virt optimizations that avoided the TLBI-IPI), this broadcast invalidate now very much requires this for native.

maintainer
#8 lore raw

Re: [PATCH 01/12] x86/mm: make MMU_GATHER_RCU_TABLE_FREE unconditional

Quoted 2 lines · Peter Zijlstra

Well, I've already answered why we need this in the previous thread but it wasn't preserved :-(

... and this needs to be part of the commit message. And there's a similar comment over tlb_remove_table_smp_sync() in mm/mmu_gather.c which pretty much explains the same thing.

Quoted 7 lines

Currently GUP-fast serializes against table-free by disabling interrupts, which in turn holds of the TLBI-IPIs.

Since you're going to be doing broadcast TLBI -- without IPIs, this no longer works and we need other means of serializing GUP-fast vs table-free.

Quoted 5 lines

MMU_GATHER_RCU_TABLE_FREE is that means.

So where previously paravirt implementations of tlb_flush_multi might require this (because of virt optimizations that avoided the TLBI-IPI), this broadcast invalidate now very much requires this for native.

Right, so this begs the question: we probably should do this dynamically only on TLBI systems - not on everything native - due to the overhead of this batching - I'm looking at tlb_remove_table().

Or should we make this unconditional on all native because we don't care about the overhead and would like to have simpler code. I mean, disabling IRQs vs batching and allocating memory...?

Meh.

Thx.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
maintainer
#9 lore raw

Re: [PATCH 01/12] x86/mm: make MMU_GATHER_RCU_TABLE_FREE unconditional

Quoted 21 lines · Borislav Petkov
Quoted 2 lines · Peter Zijlstra

Well, I've already answered why we need this in the previous thread but it wasn't preserved :-(

... and this needs to be part of the commit message. And there's a similar comment over tlb_remove_table_smp_sync() in mm/mmu_gather.c which pretty much explains the same thing.

Quoted 7 lines

Currently GUP-fast serializes against table-free by disabling interrupts, which in turn holds of the TLBI-IPIs.

Since you're going to be doing broadcast TLBI -- without IPIs, this no longer works and we need other means of serializing GUP-fast vs table-free.

Quoted 5 lines

MMU_GATHER_RCU_TABLE_FREE is that means.

So where previously paravirt implementations of tlb_flush_multi might require this (because of virt optimizations that avoided the TLBI-IPI), this broadcast invalidate now very much requires this for native.

Quoted 7 lines

Right, so this begs the question: we probably should do this dynamically only on TLBI systems - not on everything native - due to the overhead of this batching - I'm looking at tlb_remove_table().

Or should we make this unconditional on all native because we don't care about the overhead and would like to have simpler code. I mean, disabling IRQs vs batching and allocating memory...?

The disabling IRQs on the GUP-fast side stays, it acts as a RCU-read-side section -- also mmu_gather reverts to sending IPIs if it runs out of memory (extremely rare).

I don't think there is measurable overhead from doing the separate table batching, but I'm sure the robots will tell us.

#10 lore raw

Re: [PATCH 01/12] x86/mm: make MMU_GATHER_RCU_TABLE_FREE unconditional

Quoted 9 lines · Peter Zijlstra
Quoted 3 lines

Or should we make this unconditional on all native because we don't care about the overhead and would like to have simpler code. I mean, disabling IRQs vs batching and allocating memory...?

The disabling IRQs on the GUP-fast side stays, it acts as a RCU-read-side section -- also mmu_gather reverts to sending IPIs if it runs out of memory (extremely rare).

I don't think there is measurable overhead from doing the separate table batching, but I'm sure the robots will tell us.

We should _try_ to make it unconditional for simplicity if nothing else.

BTW, a few years back, some folks at Intel turned on MMU_GATHER_RCU_TABLE_FREE and ran the usual 0day/LKP tests. I _think_ it was when we were exploring the benefits of Intel's IPI-free TLB flushing mechanism. We didn't find anything remarkable either way (IIRC).

#11 lore raw

Re: [PATCH 01/12] x86/mm: make MMU_GATHER_RCU_TABLE_FREE unconditional

In reply to Borislav Petkov
Quoted 33 lines · Borislav Petkov
Quoted 3 lines · Peter Zijlstra

Well, I've already answered why we need this in the previous thread but
it wasn't preserved :-(

... and this needs to be part of the commit message. And there's a similar
comment over tlb_remove_table_smp_sync() in mm/mmu_gather.c which pretty much
explains the same thing.

Quoted 8 lines

Currently GUP-fast serializes against table-free by disabling interrupts, which in turn holds of the TLBI-IPIs.

Since you're going to be doing broadcast TLBI -- without IPIs, this no
longer works and we need other means of serializing GUP-fast vs table-free.

Quoted 7 lines

MMU_GATHER_RCU_TABLE_FREE is that means.

So where previously paravirt implementations of tlb_flush_multi might
require this (because of virt optimizations that avoided the TLBI- IPI),
this broadcast invalidate now very much requires this for native.

Right, so this begs the question: we probably should do this dynamically only
on TLBI systems - not on everything native - due to the overhead of this
batching - I'm looking at tlb_remove_table().

Quoted 5 lines

Or should we make this unconditional on all native because we don't care about
the overhead and would like to have simpler code. I mean, disabling IRQs vs
batching and allocating memory...?

Given the cost of IPIs on systems where we don't
technically need MMU_GATHER_RCU_TABLE_FREE, the
batching might still be cheaper.

Signature · 1 line
All Rights Reversed.
#12 lore raw

PATCH v3 2/12 x86/mm: remove pv_ops.mmu.tlb_remove_table call

In reply to Rik van Riel

Every pv_ops.mmu.tlb_remove_table call ends up calling tlb_remove_table.

Get rid of the indirection by simply calling tlb_remove_table directly, and not going through the paravirt function pointers.

---
 arch/x86/hyperv/mmu.c                 |  1 -
 arch/x86/include/asm/paravirt.h       |  5 -----
 arch/x86/include/asm/paravirt_types.h |  2 --
 arch/x86/kernel/kvm.c                 |  1 -
 arch/x86/kernel/paravirt.c            |  1 -
 arch/x86/mm/pgtable.c                 | 16 ++++------------
 arch/x86/xen/mmu_pv.c                 |  1 -
 7 files changed, 4 insertions(+), 23 deletions(-)
arch/x86/hyperv/mmu.c +0 -1
index 1cc113200ff5..cbe6c71e17c1 100644
@@ -240,5 +240,4 @@ void hyperv_setup_mmu_ops(void)
 
 	pr_info("Using hypercall for remote TLB flush\n");
 	pv_ops.mmu.flush_tlb_multi = hyperv_flush_tlb_multi;
-	pv_ops.mmu.tlb_remove_table = tlb_remove_table;
 }
arch/x86/include/asm/paravirt.h +0 -5
index d4eb9e1d61b8..794ba3647c6c 100644
@@ -91,11 +91,6 @@ static inline void __flush_tlb_multi(const struct cpumask *cpumask,
 	PVOP_VCALL2(mmu.flush_tlb_multi, cpumask, info);
 }
 
-static inline void paravirt_tlb_remove_table(struct mmu_gather *tlb, void *table)
-{
-	PVOP_VCALL2(mmu.tlb_remove_table, tlb, table);
-}
-
 static inline void paravirt_arch_exit_mmap(struct mm_struct *mm)
 {
 	PVOP_VCALL1(mmu.exit_mmap, mm);
arch/x86/include/asm/paravirt_types.h +0 -2
index 8d4fbe1be489..13405959e4db 100644
@@ -136,8 +136,6 @@ struct pv_mmu_ops {
 	void (*flush_tlb_multi)(const struct cpumask *cpus,
 				const struct flush_tlb_info *info);
 
-	void (*tlb_remove_table)(struct mmu_gather *tlb, void *table);
-
 	/* Hook for intercepting the destruction of an mm_struct. */
 	void (*exit_mmap)(struct mm_struct *mm);
 	void (*notify_page_enc_status_changed)(unsigned long pfn, int npages, bool enc);
arch/x86/kernel/kvm.c +0 -1
index 7a422a6c5983..3be9b3342c67 100644
@@ -838,7 +838,6 @@ static void __init kvm_guest_init(void)
 #ifdef CONFIG_SMP
 	if (pv_tlb_flush_supported()) {
 		pv_ops.mmu.flush_tlb_multi = kvm_flush_tlb_multi;
-		pv_ops.mmu.tlb_remove_table = tlb_remove_table;
 		pr_info("KVM setup pv remote TLB flush\n");
 	}
 
arch/x86/kernel/paravirt.c +0 -1
index 2b78a6b466ed..c019771e0123 100644
@@ -186,7 +186,6 @@ struct paravirt_patch_template pv_ops = {
 	.mmu.flush_tlb_kernel	= native_flush_tlb_global,
 	.mmu.flush_tlb_one_user	= native_flush_tlb_one_user,
 	.mmu.flush_tlb_multi	= native_flush_tlb_multi,
-	.mmu.tlb_remove_table	= tlb_remove_table,
 
 	.mmu.exit_mmap		= paravirt_nop,
 	.mmu.notify_page_enc_status_changed	= paravirt_nop,
arch/x86/mm/pgtable.c +4 -12
index 5745a354a241..3dc4af1f7868 100644
@@ -18,14 +18,6 @@ EXPORT_SYMBOL(physical_mask);
 #define PGTABLE_HIGHMEM 0
 #endif
 
-#ifndef CONFIG_PARAVIRT
-static inline
-void paravirt_tlb_remove_table(struct mmu_gather *tlb, void *table)
-{
-	tlb_remove_page(tlb, table);
-}
-#endif
-
 gfp_t __userpte_alloc_gfp = GFP_PGTABLE_USER | PGTABLE_HIGHMEM;
 
 pgtable_t pte_alloc_one(struct mm_struct *mm)
@@ -54,7 +46,7 @@ void ___pte_free_tlb(struct mmu_gather *tlb, struct page *pte)
 {
 	pagetable_pte_dtor(page_ptdesc(pte));
 	paravirt_release_pte(page_to_pfn(pte));
-	paravirt_tlb_remove_table(tlb, pte);
+	tlb_remove_table(tlb, pte);
 }
 
 #if CONFIG_PGTABLE_LEVELS > 2
@@ -70,7 +62,7 @@ void ___pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd)
 	tlb->need_flush_all = 1;
 #endif
 	pagetable_pmd_dtor(ptdesc);
-	paravirt_tlb_remove_table(tlb, ptdesc_page(ptdesc));
+	tlb_remove_table(tlb, ptdesc_page(ptdesc));
 }
 
 #if CONFIG_PGTABLE_LEVELS > 3
@@ -80,14 +72,14 @@ void ___pud_free_tlb(struct mmu_gather *tlb, pud_t *pud)
 
 	pagetable_pud_dtor(ptdesc);
 	paravirt_release_pud(__pa(pud) >> PAGE_SHIFT);
-	paravirt_tlb_remove_table(tlb, virt_to_page(pud));
+	tlb_remove_table(tlb, virt_to_page(pud));
 }
 
 #if CONFIG_PGTABLE_LEVELS > 4
 void ___p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d)
 {
 	paravirt_release_p4d(__pa(p4d) >> PAGE_SHIFT);
-	paravirt_tlb_remove_table(tlb, virt_to_page(p4d));
+	tlb_remove_table(tlb, virt_to_page(p4d));
 }
 #endif	/* CONFIG_PGTABLE_LEVELS > 4 */
 #endif	/* CONFIG_PGTABLE_LEVELS > 3 */
arch/x86/xen/mmu_pv.c +0 -1
index 55a4996d0c04..041e17282af0 100644
@@ -2137,7 +2137,6 @@ static const typeof(pv_ops) xen_mmu_ops __initconst = {
 		.flush_tlb_kernel = xen_flush_tlb,
 		.flush_tlb_one_user = xen_flush_tlb_one_user,
 		.flush_tlb_multi = xen_flush_tlb_multi,
-		.tlb_remove_table = tlb_remove_table,
 
 		.pgd_alloc = xen_pgd_alloc,
 		.pgd_free = xen_pgd_free,
Signature · 1 line
2.47.1
#13 lore raw

Re: [PATCH 02/12] x86/mm: remove pv_ops.mmu.tlb_remove_table call

Quoted 7 lines · Rik van Riel

Every pv_ops.mmu.tlb_remove_table call ends up calling tlb_remove_table.

Get rid of the indirection by simply calling tlb_remove_table directly, and not going through the paravirt function pointers.

Quoted 9 lines
---
  arch/x86/hyperv/mmu.c                 |  1 -
  arch/x86/include/asm/paravirt.h       |  5 -----
  arch/x86/include/asm/paravirt_types.h |  2 --
  arch/x86/kernel/kvm.c                 |  1 -
  arch/x86/kernel/paravirt.c            |  1 -
  arch/x86/mm/pgtable.c                 | 16 ++++------------
  arch/x86/xen/mmu_pv.c                 |  1 -
  7 files changed, 4 insertions(+), 23 deletions(-)

This change looks good to me. Thanks!

In addition, since I also made relevant changes [1], there will be conflict when our code is merged into the linux-next branch. Of course, this is very easy to fix. ;)

[1]. https://lore.kernel.org/all/0287d442a973150b0e1019cc406e6322d148277a.1733305182.git.zhengqi.arch@bytedance.com/
(This patch has been merged into the mm-unstable branch, and we can
also see it in the linux-next: https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/commit/?id=62e76fb4ff704945b5cf9411dda448fefb6618f9)

Thanks,
Qi

#14 lore raw

PATCH v3 3/12 x86/mm: add X86_FEATURE_INVLPGB definition.

In reply to Rik van Riel

Add the INVPLGB CPUID definition, allowing the kernel to recognize whether the CPU supports the INVLPGB instruction.

---
 arch/x86/include/asm/cpufeatures.h | 1 +
 1 file changed, 1 insertion(+)
arch/x86/include/asm/cpufeatures.h +1 -0
index 17b6590748c0..b7209d6c3a5f 100644
@@ -338,6 +338,7 @@
 #define X86_FEATURE_CLZERO		(13*32+ 0) /* "clzero" CLZERO instruction */
 #define X86_FEATURE_IRPERF		(13*32+ 1) /* "irperf" Instructions Retired Count */
 #define X86_FEATURE_XSAVEERPTR		(13*32+ 2) /* "xsaveerptr" Always save/restore FP error pointers */
+#define X86_FEATURE_INVLPGB		(13*32+ 3) /* "invlpgb" INVLPGB instruction */
 #define X86_FEATURE_RDPRU		(13*32+ 4) /* "rdpru" Read processor register at user level */
 #define X86_FEATURE_WBNOINVD		(13*32+ 9) /* "wbnoinvd" WBNOINVD instruction */
 #define X86_FEATURE_AMD_IBPB		(13*32+12) /* Indirect Branch Prediction Barrier */
Signature · 1 line
2.47.1
maintainer
#15 lore raw

Re: [PATCH 03/12] x86/mm: add X86_FEATURE_INVLPGB definition.

Quoted 13 lines · Rik van Riel

Add the INVPLGB CPUID definition, allowing the kernel to recognize whether the CPU supports the INVLPGB instruction.

---
 arch/x86/include/asm/cpufeatures.h | 1 +
 1 file changed, 1 insertion(+)
arch/x86/include/asm/cpufeatures.h +0 -0
index 17b6590748c0..b7209d6c3a5f 100644

arch/x86/include/asm/cpufeatures.h line 341

Quoted 4 lines
arch/x86/include/asm/cpufeatures.h +1 -0
@@ -338,6 +338,7 @@
 #define X86_FEATURE_CLZERO		(13*32+ 0) /* "clzero" CLZERO instruction */
 #define X86_FEATURE_IRPERF		(13*32+ 1) /* "irperf" Instructions Retired Count */
 #define X86_FEATURE_XSAVEERPTR		(13*32+ 2) /* "xsaveerptr" Always save/restore FP error pointers */
+#define X86_FEATURE_INVLPGB		(13*32+ 3) /* "invlpgb" INVLPGB instruction */
						      ^^^^^^^^^

We don't show random CPUID bits in /proc/cpuinfo anymore so you can remove that.

arch/x86/include/asm/cpufeatures.h line 344

Quoted 4 lines
arch/x86/include/asm/cpufeatures.h +0 -0
 #define X86_FEATURE_RDPRU		(13*32+ 4) /* "rdpru" Read processor register at user level */
 #define X86_FEATURE_WBNOINVD		(13*32+ 9) /* "wbnoinvd" WBNOINVD instruction */
 #define X86_FEATURE_AMD_IBPB		(13*32+12) /* Indirect Branch Prediction Barrier */

Also, merge this patch with the patch which uses the flag pls.

Thx.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#16 lore raw

Re: [PATCH 03/12] x86/mm: add X86_FEATURE_INVLPGB definition.

Quoted 14 lines · Borislav Petkov
Quoted 11 lines · Rik van Riel

+++ b/arch/x86/include/asm/cpufeatures.h

(quoted diff) +0 -0
@@ -338,6 +338,7 @@
  #define X86_FEATURE_CLZERO		(13*32+ 0) /* "clzero"

CLZERO instruction */
 #define X86_FEATURE_IRPERF (13*32+ 1) /* "irperf"
Instructions Retired Count */
 #define X86_FEATURE_XSAVEERPTR (13*32+ 2) /* "xsaveerptr"
Always save/restore FP error pointers */
+#define X86_FEATURE_INVLPGB (13*32+ 3) /* "invlpgb"
INVLPGB instruction */

						      ^^^^^^^^^
Quoted 3 lines

We don't show random CPUID bits in /proc/cpuinfo anymore so you can remove
that.

I still see dozens of flags in /proc/cpuinfo here on 6.11. When did that change?

Quoted 10 lines
Quoted 8 lines

 #define X86_FEATURE_RDPRU (13*32+ 4) /* "rdpru" Read
processor register at user level */
 #define X86_FEATURE_WBNOINVD (13*32+ 9) /* "wbnoinvd"
WBNOINVD instruction */
 #define X86_FEATURE_AMD_IBPB (13*32+12) /* Indirect
Branch Prediction Barrier */

Also, merge this patch with the patch which uses the flag pls.

The first real use is 3 patches further down into the series.

I'm not convinced things will be more readable if 4 patches get squashed down into one.

Are you sure you want that?

Signature · 4 lines
All Rights Reversed.

-- 
All Rights Reversed.
maintainer
#17 lore raw

Re: [PATCH 03/12] x86/mm: add X86_FEATURE_INVLPGB definition.

Quoted 2 lines · Rik van Riel

I still see dozens of flags in /proc/cpuinfo here on 6.11.

We can't remove those flags because they're an ABI.

Quoted 1 line

When did that change?

This should explain: Documentation/arch/x86/cpuinfo.rst

Quoted 2 lines

I'm not convinced things will be more readable if 4 patches get squashed down into one.

Not 4 patches - you merge this one with patch 6 - the first patch that uses the flag.

Thx.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#18 lore raw

PATCH v3 4/12 x86/mm: get INVLPGB count max from CPUID

In reply to Rik van Riel

The CPU advertises the maximum number of pages that can be shot down with one INVLPGB instruction in the CPUID data.

Save that information for later use.

---
 arch/x86/include/asm/tlbflush.h | 1 +
 arch/x86/kernel/cpu/amd.c       | 8 ++++++++
 arch/x86/kernel/setup.c         | 4 ++++
 3 files changed, 13 insertions(+)
arch/x86/include/asm/tlbflush.h +1 -0
index 02fc2aa06e9e..7d1468a3967b 100644
@@ -182,6 +182,7 @@ static inline void cr4_init_shadow(void)
 
 extern unsigned long mmu_cr4_features;
 extern u32 *trampoline_cr4_features;
+extern u16 invlpgb_count_max;
 
 extern void initialize_tlbstate_and_flush(void);
 
arch/x86/kernel/cpu/amd.c +8 -0
index 79d2e17f6582..226b8fc64bfc 100644
@@ -1135,6 +1135,14 @@ static void cpu_detect_tlb_amd(struct cpuinfo_x86 *c)
 		tlb_lli_2m[ENTRIES] = eax & mask;
 
 	tlb_lli_4m[ENTRIES] = tlb_lli_2m[ENTRIES] >> 1;
+
+	if (c->extended_cpuid_level < 0x80000008)
+		return;
+
+	cpuid(0x80000008, &eax, &ebx, &ecx, &edx);
+
+	/* Max number of pages INVLPGB can invalidate in one shot */
+	invlpgb_count_max = (edx & 0xffff) + 1;
 }
 
 static const struct cpu_dev amd_cpu_dev = {
arch/x86/kernel/setup.c +4 -0
index f1fea506e20f..6c4d08f8f7b1 100644
@@ -138,6 +138,10 @@ __visible unsigned long mmu_cr4_features __ro_after_init;
 __visible unsigned long mmu_cr4_features __ro_after_init = X86_CR4_PAE;
 #endif
 
+#ifdef CONFIG_CPU_SUP_AMD
+u16 invlpgb_count_max __ro_after_init;
+#endif
+
 #ifdef CONFIG_IMA
 static phys_addr_t ima_kexec_buffer_phys;
 static size_t ima_kexec_buffer_size;
Signature · 1 line
2.47.1
maintainer
#19 lore raw

Re: [PATCH 04/12] x86/mm: get INVLPGB count max from CPUID

arch/x86/kernel/setup.c __visible unsigned long mmu_cr4_features __ro_after_init; · line 143

Quoted 11 lines · Rik van Riel
arch/x86/kernel/setup.c +3 -0
index f1fea506e20f..6c4d08f8f7b1 100644
@@ -138,6 +138,10 @@ __visible unsigned long mmu_cr4_features __ro_after_init;
 __visible unsigned long mmu_cr4_features __ro_after_init = X86_CR4_PAE;
 #endif
 
+#ifdef CONFIG_CPU_SUP_AMD
+u16 invlpgb_count_max __ro_after_init;
+#endif

You can define this in amd.c and put the ifdeffery in the header. Something like:

#ifdef CONFIG_CPU_SUP_AMD
extern u16 invlpgb_count_max __ro_after_init;
#else
#define invlpgb_count_max 0
#endif

or so and use it freely in the remaining places.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#20 lore raw

Re: [PATCH 04/12] x86/mm: get INVLPGB count max from CPUID

In reply to Rik van Riel
Quoted 29 lines · Rik van Riel

The CPU advertises the maximum number of pages that can be shot down with one INVLPGB instruction in the CPUID data.

Save that information for later use.

---
 arch/x86/include/asm/tlbflush.h | 1 +
 arch/x86/kernel/cpu/amd.c       | 8 ++++++++
 arch/x86/kernel/setup.c         | 4 ++++
 3 files changed, 13 insertions(+)
arch/x86/include/asm/tlbflush.h +1 -0
index 02fc2aa06e9e..7d1468a3967b 100644
@@ -182,6 +182,7 @@ static inline void cr4_init_shadow(void)
 
 extern unsigned long mmu_cr4_features;
 extern u32 *trampoline_cr4_features;
+extern u16 invlpgb_count_max;
 
 extern void initialize_tlbstate_and_flush(void);
 
arch/x86/kernel/cpu/amd.c +0 -0
index 79d2e17f6582..226b8fc64bfc 100644

arch/x86/kernel/cpu/amd.c static void cpu_detect_tlb_amd(struct cpuinfo_x86 *c) · line 1140

Quoted 6 lines
arch/x86/kernel/cpu/amd.c +3 -0
@@ -1135,6 +1135,14 @@ static void cpu_detect_tlb_amd(struct cpuinfo_x86 *c)
 		tlb_lli_2m[ENTRIES] = eax & mask;
 
 	tlb_lli_4m[ENTRIES] = tlb_lli_2m[ENTRIES] >> 1;
+
+	if (c->extended_cpuid_level < 0x80000008)
+		return;

Can this just be based on cpu_feature_enabled(X86_FEATURE_TLBI), e.g:

	if (cpu_feature_enabled(X86_FEATURE_TLBI))
		invlpgb_count_max = (cpuid_edx(0x80000008) & 0xffff) + 1

Then you can squash this and the previous patch.

Thanks,
Tom

Quoted 23 lines
arch/x86/kernel/cpu/amd.c +5 -0
+
+	cpuid(0x80000008, &eax, &ebx, &ecx, &edx);
+
+	/* Max number of pages INVLPGB can invalidate in one shot */
+	invlpgb_count_max = (edx & 0xffff) + 1;
 }
 
 static const struct cpu_dev amd_cpu_dev = {
arch/x86/kernel/setup.c +4 -0
index f1fea506e20f..6c4d08f8f7b1 100644
@@ -138,6 +138,10 @@ __visible unsigned long mmu_cr4_features __ro_after_init;
 __visible unsigned long mmu_cr4_features __ro_after_init = X86_CR4_PAE;
 #endif
 
+#ifdef CONFIG_CPU_SUP_AMD
+u16 invlpgb_count_max __ro_after_init;
+#endif
+
 #ifdef CONFIG_IMA
 static phys_addr_t ima_kexec_buffer_phys;
 static size_t ima_kexec_buffer_size;
#21 lore raw

Re: [PATCH 04/12] x86/mm: get INVLPGB count max from CPUID

Quoted 14 lines · Tom Lendacky

arch/x86/kernel/cpu/amd.c static void cpu_detect_tlb_amd(struct cpuinfo_x86 *c) · line 1140

Quoted 10 lines · Rik van Riel

+++ b/arch/x86/kernel/cpu/amd.c

(quoted diff) +0 -0
@@ -1135,6 +1135,14 @@ static void cpu_detect_tlb_amd(struct
 cpuinfo_x86 *c)
  		tlb_lli_2m[ENTRIES] = eax & mask;


  tlb_lli_4m[ENTRIES] = tlb_lli_2m[ENTRIES] >> 1;

arch/x86/kernel/cpu/amd.c +3 -0
+
+	if (c->extended_cpuid_level < 0x80000008)
+		return;

Can this just be based on cpu_feature_enabled(X86_FEATURE_TLBI), e.g:

Quoted 4 lines
	if (cpu_feature_enabled(X86_FEATURE_TLBI))
		invlpgb_count_max = (cpuid_edx(0x80000008) & 0xffff)

+ 1

I don't see X86_FEATURE_TLBI defined in the tip
tree. Which CPUID bit does that need to be?

Quoted 1 line

Then you can squash this and the previous patch.

Heh, I already squashed the previous commit into
what is patch 6 in this series, as requested by
Borislav :)

Signature · 1 line
All Rights Reversed.
#22 lore raw

Re: [PATCH 04/12] x86/mm: get INVLPGB count max from CPUID

Quoted 15 lines · Rik van Riel
Quoted 14 lines · Tom Lendacky

arch/x86/kernel/cpu/amd.c static void cpu_detect_tlb_amd(struct cpuinfo_x86 *c) · line 1140

Quoted 10 lines · Rik van Riel

+++ b/arch/x86/kernel/cpu/amd.c

(quoted diff) +0 -0
@@ -1135,6 +1135,14 @@ static void cpu_detect_tlb_amd(struct
 cpuinfo_x86 *c)
  		tlb_lli_2m[ENTRIES] = eax & mask;


  tlb_lli_4m[ENTRIES] = tlb_lli_2m[ENTRIES] >> 1;

arch/x86/kernel/cpu/amd.c +3 -0
+
+	if (c->extended_cpuid_level < 0x80000008)
+		return;

Can this just be based on cpu_feature_enabled(X86_FEATURE_TLBI), e.g:

Quoted 6 lines
Quoted 4 lines
	if (cpu_feature_enabled(X86_FEATURE_TLBI))
		invlpgb_count_max = (cpuid_edx(0x80000008) & 0xffff)

+ 1

I don't see X86_FEATURE_TLBI defined in the tip
tree. Which CPUID bit does that need to be?

Sorry, I meant X86_FEATURE_INVLPGB.

Thanks,
Tom

Quoted 7 lines
Quoted 2 lines

Then you can squash this and the previous patch.

Heh, I already squashed the previous commit into
what is patch 6 in this series, as requested by
Borislav :)

maintainer
#23 lore raw

Re: [PATCH 04/12] x86/mm: get INVLPGB count max from CPUID

In reply to Rik van Riel
Quoted 2 lines · Rik van Riel

I don't see X86_FEATURE_TLBI defined in the tip
tree. Which CPUID bit does that need to be?

It should be added as:

#define X86_FEATURE_INVLPGB (13*32+ 3) /* INVLPGB and TLBSYNC instruction supported. */

in arch/x86/include/asm/cpufeatures.h

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#24 lore raw

PATCH v3 5/12 x86/mm: add INVLPGB support code

In reply to Rik van Riel

Add invlpgb.h with the helper functions and definitions needed to use broadcast TLB invalidation on AMD EPYC 3 and newer CPUs.

---
 arch/x86/include/asm/invlpgb.h  | 93 +++++++++++++++++++++++++++++++++
 arch/x86/include/asm/tlbflush.h |  1 +
 2 files changed, 94 insertions(+)
 create mode 100644 arch/x86/include/asm/invlpgb.h
arch/x86/include/asm/invlpgb.h new file +93 -0
new file mode 100644
index 000000000000..862775897a54
@@ -0,0 +1,93 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _ASM_X86_INVLPGB
+#define _ASM_X86_INVLPGB
+
+#include <vdso/bits.h>
+
+/*
+ * INVLPGB does broadcast TLB invalidation across all the CPUs in the system.
+ *
+ * The INVLPGB instruction is weakly ordered, and a batch of invalidations can
+ * be done in a parallel fashion.
+ *
+ * TLBSYNC is used to ensure that pending INVLPGB invalidations initiated from
+ * this CPU have completed.
+ */
+static inline void __invlpgb(unsigned long asid, unsigned long pcid, unsigned long addr,
+			    int extra_count, bool pmd_stride, unsigned long flags)
+{
+	u64 rax = addr | flags;
+	u32 ecx = (pmd_stride << 31) | extra_count;
+	u32 edx = (pcid << 16) | asid;
+
+	asm volatile("invlpgb" : : "a" (rax), "c" (ecx), "d" (edx));
+}
+
+/*
+ * INVLPGB can be targeted by virtual address, PCID, ASID, or any combination
+ * of the three. For example:
+ * - INVLPGB_VA | INVLPGB_INCLUDE_GLOBAL: invalidate all TLB entries at the address
+ * - INVLPGB_PCID:              	  invalidate all TLB entries matching the PCID
+ *
+ * The first can be used to invalidate (kernel) mappings at a particular
+ * address across all processes.
+ *
+ * The latter invalidates all TLB entries matching a PCID.
+ */
+#define INVLPGB_VA			BIT(0)
+#define INVLPGB_PCID			BIT(1)
+#define INVLPGB_ASID			BIT(2)
+#define INVLPGB_INCLUDE_GLOBAL		BIT(3)
+#define INVLPGB_FINAL_ONLY		BIT(4)
+#define INVLPGB_INCLUDE_NESTED		BIT(5)
+
+/* Flush all mappings for a given pcid and addr, not including globals. */
+static inline void invlpgb_flush_user(unsigned long pcid,
+				      unsigned long addr)
+{
+	__invlpgb(0, pcid, addr, 0, 0, INVLPGB_PCID | INVLPGB_VA);
+}
+
+static inline void invlpgb_flush_user_nr(unsigned long pcid, unsigned long addr,
+					 int nr, bool pmd_stride)
+{
+	__invlpgb(0, pcid, addr, nr - 1, pmd_stride, INVLPGB_PCID | INVLPGB_VA);
+}
+
+/* Flush all mappings for a given ASID, not including globals. */
+static inline void invlpgb_flush_single_asid(unsigned long asid)
+{
+	__invlpgb(asid, 0, 0, 0, 0, INVLPGB_ASID);
+}
+
+/* Flush all mappings for a given PCID, not including globals. */
+static inline void invlpgb_flush_single_pcid(unsigned long pcid)
+{
+	__invlpgb(0, pcid, 0, 0, 0, INVLPGB_PCID);
+}
+
+/* Flush all mappings, including globals, for all PCIDs. */
+static inline void invlpgb_flush_all(void)
+{
+	__invlpgb(0, 0, 0, 0, 0, INVLPGB_INCLUDE_GLOBAL);
+}
+
+/* Flush addr, including globals, for all PCIDs. */
+static inline void invlpgb_flush_addr(unsigned long addr, int nr)
+{
+	__invlpgb(0, 0, addr, nr - 1, 0, INVLPGB_INCLUDE_GLOBAL);
+}
+
+/* Flush all mappings for all PCIDs except globals. */
+static inline void invlpgb_flush_all_nonglobals(void)
+{
+	__invlpgb(0, 0, 0, 0, 0, 0);
+}
+
+/* Wait for INVLPGB originated by this CPU to complete. */
+static inline void tlbsync(void)
+{
+	asm volatile("tlbsync");
+}
+
+#endif /* _ASM_X86_INVLPGB */
arch/x86/include/asm/tlbflush.h +1 -0
index 7d1468a3967b..20074f17fbcd 100644
@@ -10,6 +10,7 @@
 #include <asm/cpufeature.h>
 #include <asm/special_insns.h>
 #include <asm/smp.h>
+#include <asm/invlpgb.h>
 #include <asm/invpcid.h>
 #include <asm/pti.h>
 #include <asm/processor-flags.h>
Signature · 1 line
2.47.1
maintainer
#25 lore raw

Re: [PATCH 05/12] x86/mm: add INVLPGB support code

arch/x86/include/asm/invlpgb.h

Quoted 15 lines · Rik van Riel

Add invlpgb.h with the helper functions and definitions needed to use broadcast TLB invalidation on AMD EPYC 3 and newer CPUs.

---
 arch/x86/include/asm/invlpgb.h  | 93 +++++++++++++++++++++++++++++++++
 arch/x86/include/asm/tlbflush.h |  1 +
 2 files changed, 94 insertions(+)
 create mode 100644 arch/x86/include/asm/invlpgb.h
arch/x86/include/asm/invlpgb.h new file +0 -0
new file mode 100644
index 000000000000..862775897a54

I don't see the point for a separate header just for that. We have arch/x86/include/asm/tlb.h.

Quoted 8 lines
arch/x86/include/asm/invlpgb.h new file +7 -0
@@ -0,0 +1,93 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _ASM_X86_INVLPGB
+#define _ASM_X86_INVLPGB
+
+#include <vdso/bits.h>
+
+/*

arch/x86/include/asm/invlpgb.h line 17

Quoted 10 lines
arch/x86/include/asm/invlpgb.h new file +10 -0
+ * INVLPGB does broadcast TLB invalidation across all the CPUs in the system.
+ *
+ * The INVLPGB instruction is weakly ordered, and a batch of invalidations can
+ * be done in a parallel fashion.
+ *
+ * TLBSYNC is used to ensure that pending INVLPGB invalidations initiated from
+ * this CPU have completed.
+ */
+static inline void __invlpgb(unsigned long asid, unsigned long pcid, unsigned long addr,
+			    int extra_count, bool pmd_stride, unsigned long flags)

See below. Once you prune the functions you're not using in your patchset, this argument list will drop too. We can always extend it later, if really needed, so let's keep it simple here.

I had slimmed it down to this internally:

static inline void invlpgb(unsigned long va, unsigned long count, unsigned long id)

{
       /* INVLPGB; supported in binutils >= 2.36. */
	asm volatile(".byte 0x0f, 0x01, 0xfe"
		     : : "a" (va), "c" (count), "d" (id)
		     : "memory");
}

I had the memory clobber too but now that I think of it, it probably isn't needed because even if the compiler reorders INVLPGB, it is weakly-ordered anyway.

TLBSYNC should probably have a memory clobber tho, to prevent the compiler from doing funky stuff...

arch/x86/include/asm/invlpgb.h line 23

Quoted 6 lines
arch/x86/include/asm/invlpgb.h new file +6 -0
+{
+	u64 rax = addr | flags;
+	u32 ecx = (pmd_stride << 31) | extra_count;
+	u32 edx = (pcid << 16) | asid;
+
+	asm volatile("invlpgb" : : "a" (rax), "c" (ecx), "d" (edx));

No, you do:

	/* INVLPGB; supported in binutils >= 2.36. */
 	asm volatile(".byte 0x0f, 0x01, 0xfe"
	...

arch/x86/include/asm/invlpgb.h line 30

Quoted 5 lines
arch/x86/include/asm/invlpgb.h new file +5 -0
+/*
+ * INVLPGB can be targeted by virtual address, PCID, ASID, or any combination
+ * of the three. For example:
+ * - INVLPGB_VA | INVLPGB_INCLUDE_GLOBAL: invalidate all TLB entries at the address
+ * - INVLPGB_PCID:              	  invalidate all TLB entries matching the PCID
		     ^^^^^^^^^^^^^^^^^^^^^^

Whitespace damage here. Needs tabs.

arch/x86/include/asm/invlpgb.h line 42

Quoted 12 lines
arch/x86/include/asm/invlpgb.h new file +12 -0
+ *
+ * The first can be used to invalidate (kernel) mappings at a particular
+ * address across all processes.
+ *
+ * The latter invalidates all TLB entries matching a PCID.
+ */
+#define INVLPGB_VA			BIT(0)
+#define INVLPGB_PCID			BIT(1)
+#define INVLPGB_ASID			BIT(2)
+#define INVLPGB_INCLUDE_GLOBAL		BIT(3)
+#define INVLPGB_FINAL_ONLY		BIT(4)
+#define INVLPGB_INCLUDE_NESTED		BIT(5)

Please add only the defines which are actually being used. Ditto for the functions.

arch/x86/include/asm/invlpgb.h line 91

Quoted 5 lines
arch/x86/include/asm/invlpgb.h new file +5 -0
+/* Wait for INVLPGB originated by this CPU to complete. */
+static inline void tlbsync(void)
+{
+	asm volatile("tlbsync");
+}
	/* TLBSYNC; supported in binutils >= 2.36. */

asm volatile(".byte 0x0f, 0x01, 0xff" ::: "memory");

Quoted 17 lines
arch/x86/include/asm/invlpgb.h new file +2 -0
+
+#endif /* _ASM_X86_INVLPGB */
arch/x86/include/asm/tlbflush.h +1 -0
index 7d1468a3967b..20074f17fbcd 100644
@@ -10,6 +10,7 @@
 #include <asm/cpufeature.h>
 #include <asm/special_insns.h>
 #include <asm/smp.h>
+#include <asm/invlpgb.h>
 #include <asm/invpcid.h>
 #include <asm/pti.h>
 #include <asm/processor-flags.h>
Signature · 1 line
2.47.1
Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#26 lore raw

Re: [PATCH 05/12] x86/mm: add INVLPGB support code

Quoted 8 lines · Borislav Petkov

arch/x86/include/asm/invlpgb.h line 42

Quoted 6 lines
arch/x86/include/asm/invlpgb.h new file +6 -0
+#define INVLPGB_VA			BIT(0)
+#define INVLPGB_PCID			BIT(1)
+#define INVLPGB_ASID			BIT(2)
+#define INVLPGB_INCLUDE_GLOBAL		BIT(3)
+#define INVLPGB_FINAL_ONLY		BIT(4)
+#define INVLPGB_INCLUDE_NESTED		BIT(5)

Please add only the defines which are actually being used. Ditto for the functions.

There's some precedent for defining them all up front, like we did for invpcid_flush_*().

For INVPCID, there are four variants and two of them got used up front. But I get that it's a balancing act between having untested code that might bitrot and introducing helpers at a time when someone (Rik) is very likely to get all the variants coded up correctly.

Rik, how many of these end up being used by the end of the series?

#27 lore raw

Re: [PATCH 05/12] x86/mm: add INVLPGB support code

Quoted 15 lines · Dave Hansen
Quoted 9 lines · Borislav Petkov

arch/x86/include/asm/invlpgb.h line 42

Quoted 6 lines
arch/x86/include/asm/invlpgb.h new file +6 -0
+#define INVLPGB_VA			BIT(0)
+#define INVLPGB_PCID			BIT(1)
+#define INVLPGB_ASID			BIT(2)
+#define INVLPGB_INCLUDE_GLOBAL		BIT(3)
+#define INVLPGB_FINAL_ONLY		BIT(4)
+#define INVLPGB_INCLUDE_NESTED		BIT(5)

Please add only the defines which are actually being used. Ditto for the
functions.

There's some precedent for defining them all up front, like we did for
invpcid_flush_*().

Quoted 8 lines

For INVPCID, there are four variants and two of them got used up front.
But I get that it's a balancing act between having untested code that might bitrot and introducing helpers at a time when someone (Rik) is very likely to get all the variants coded up correctly.

Rik, how many of these end up being used by the end of the series?

Only invlpgb_flush_single_asid is unused at the end of the series.

I'll remove that one.

As for the bit flags, those are a hardware interface. I can remove the unused ones, but would like to know why :)

Signature · 1 line
All Rights Reversed.
maintainer
#28 lore raw

Re: [PATCH 05/12] x86/mm: add INVLPGB support code

In reply to Dave Hansen
Quoted 7 lines · Dave Hansen

There's some precedent for defining them all up front, like we did for invpcid_flush_*().

For INVPCID, there are four variants and two of them got used up front. But I get that it's a balancing act between having untested code that might bitrot and introducing helpers at a time when someone (Rik) is very likely to get all the variants coded up correctly.

That's just silly. We don't add all possible hw interface bits, defines etc, when they're not going to be used. If we did, the kernel would be an unwieldy mess of MSRs and their bits, unused insn opcodes and the like.

If someone wants to use them, someone can add them *when* they're needed - not preemptively, in anticipation that *someone* *might* use them in the future.

Guys, I can't believe I'm actually arguing for something so obvious.

:-\

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#29 lore raw

Re: [PATCH 05/12] x86/mm: add INVLPGB support code

In reply to Borislav Petkov
Quoted 13 lines · Borislav Petkov
Quoted 7 lines · Rik van Riel
arch/x86/include/asm/invlpgb.h new file +6 -0
+{
+	u64 rax = addr | flags;
+	u32 ecx = (pmd_stride << 31) | extra_count;
+	u32 edx = (pcid << 16) | asid;
+
+	asm volatile("invlpgb" : : "a" (rax), "c" (ecx), "d"

(edx));

No, you do:

Quoted 3 lines
	/* INVLPGB; supported in binutils >= 2.36. */
 	asm volatile(".byte 0x0f, 0x01, 0xfe"
	...

Sorry, this one fell through the cracks.

I've replaced both of the asm mnemonics with
byte code for the next version of the series.

Signature · 1 line
All Rights Reversed.
maintainer
#30 lore raw

Re: [PATCH 05/12] x86/mm: add INVLPGB support code

In reply to Rik van Riel

arch/x86/include/asm/invlpgb.h line 17

Quoted 2 lines · Rik van Riel
arch/x86/include/asm/invlpgb.h new file +2 -0
+static inline void __invlpgb(unsigned long asid, unsigned long pcid, unsigned long addr,
+			    int extra_count, bool pmd_stride, unsigned long flags)

That pmd_stride thing - the callers should supply a bool: true/false instead of 0/1 for more clarity at the call sites.

arch/x86/include/asm/invlpgb.h line 20

Quoted 3 lines
arch/x86/include/asm/invlpgb.h new file +3 -0
+{
+	u64 rax = addr | flags;
+	u32 ecx = (pmd_stride << 31) | extra_count;

You need to handle the case where extra_count becomes negative because callers supply nr=0 and you do "nr - 1" below and then you'll end up flushing 0b1111_1111_1111_1111 TLB entries.

"ECX[15:0] contains a count of the number of sequential pages to invalidate in addition to the original virtual address, starting from the virtual address specified in rAX."

Quoted 4 lines
arch/x86/include/asm/invlpgb.h new file +4 -0
+	u32 edx = (pcid << 16) | asid;
+
+	asm volatile("invlpgb" : : "a" (rax), "c" (ecx), "d" (edx));
+}
Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#31 lore raw

PATCH v3 6/12 x86/mm: use INVLPGB for kernel TLB flushes

In reply to Rik van Riel

Use broadcast TLB invalidation for kernel addresses when available.

This stops us from having to send IPIs for kernel TLB flushes.

---
 arch/x86/mm/tlb.c | 31 +++++++++++++++++++++++++++++++
 1 file changed, 31 insertions(+)
arch/x86/mm/tlb.c +31 -0
index 6cf881a942bb..29207dc5b807 100644
@@ -1077,6 +1077,32 @@ void flush_tlb_all(void)
 	on_each_cpu(do_flush_tlb_all, NULL, 1);
 }
 
+static void broadcast_kernel_range_flush(unsigned long start, unsigned long end)
+{
+	unsigned long addr;
+	unsigned long maxnr = invlpgb_count_max;
+	unsigned long threshold = tlb_single_page_flush_ceiling * maxnr;
+
+	/*
+	 * TLBSYNC only waits for flushes originating on the same CPU.
+	 * Disabling migration allows us to wait on all flushes.
+	 */
+	guard(preempt)();
+
+	if (end == TLB_FLUSH_ALL ||
+	    (end - start) > threshold << PAGE_SHIFT) {
+		invlpgb_flush_all();
+	} else {
+		unsigned long nr;
+		for (addr = start; addr < end; addr += nr << PAGE_SHIFT) {
+			nr = min((end - addr) >> PAGE_SHIFT, maxnr);
+			invlpgb_flush_addr(addr, nr);
+		}
+	}
+
+	tlbsync();
+}
+
 static void do_kernel_range_flush(void *info)
 {
 	struct flush_tlb_info *f = info;
@@ -1089,6 +1115,11 @@ static void do_kernel_range_flush(void *info)
 
 void flush_tlb_kernel_range(unsigned long start, unsigned long end)
 {
+	if (cpu_feature_enabled(X86_FEATURE_INVLPGB)) {
+		broadcast_kernel_range_flush(start, end);
+		return;
+	}
+
 	/* Balance as user space task's flush, a bit conservative */
 	if (end == TLB_FLUSH_ALL ||
 	    (end - start) > tlb_single_page_flush_ceiling << PAGE_SHIFT) {
Signature · 1 line
2.47.1
maintainer
#32 lore raw

Re: [PATCH 06/12] x86/mm: use INVLPGB for kernel TLB flushes

Quoted 14 lines · Rik van Riel

Use broadcast TLB invalidation for kernel addresses when available.

This stops us from having to send IPIs for kernel TLB flushes.

---
 arch/x86/mm/tlb.c | 31 +++++++++++++++++++++++++++++++
 1 file changed, 31 insertions(+)
arch/x86/mm/tlb.c +0 -0
index 6cf881a942bb..29207dc5b807 100644

arch/x86/mm/tlb.c void flush_tlb_all(void) · line 1084

Quoted 8 lines
arch/x86/mm/tlb.c +5 -0
@@ -1077,6 +1077,32 @@ void flush_tlb_all(void)
 	on_each_cpu(do_flush_tlb_all, NULL, 1);
 }
 
+static void broadcast_kernel_range_flush(unsigned long start, unsigned long end)
+{
+	unsigned long addr;
+	unsigned long maxnr = invlpgb_count_max;
+	unsigned long threshold = tlb_single_page_flush_ceiling * maxnr;

The tip-tree preferred ordering of variable declarations at the beginning of a function is reverse fir tree order::

	struct long_struct_name *descriptive_name;
	unsigned long foo, bar;
	unsigned int tmp;
	int ret;

The above is faster to parse than the reverse ordering::

	int ret;
	unsigned int tmp;
	unsigned long foo, bar;
	struct long_struct_name *descriptive_name;

And even more so than random ordering::

	unsigned long foo, bar;
	int ret;
	struct long_struct_name *descriptive_name;
	unsigned int tmp;

And you can get rid of maxnr and get the reversed xmas tree order:

arch/x86/mm/tlb.c +2 -3
index 29207dc5b807..8a85acd9483d 100644
@@ -1079,9 +1079,8 @@ void flush_tlb_all(void)
 
 static void broadcast_kernel_range_flush(unsigned long start, unsigned long end)
 {
+	unsigned long threshold = tlb_single_page_flush_ceiling * invlpgb_count_max;
 	unsigned long addr;
-	unsigned long maxnr = invlpgb_count_max;
-	unsigned long threshold = tlb_single_page_flush_ceiling * maxnr;
 
 	/*
 	 * TLBSYNC only waits for flushes originating on the same CPU.
@@ -1095,7 +1094,7 @@ static void broadcast_kernel_range_flush(unsigned long start, unsigned long end)
 	} else {
 		unsigned long nr;
 		for (addr = start; addr < end; addr += nr << PAGE_SHIFT) {
-			nr = min((end - addr) >> PAGE_SHIFT, maxnr);
+			nr = min((end - addr) >> PAGE_SHIFT, invlpgb_count_max);
 			invlpgb_flush_addr(addr, nr);
 		}
 	}

arch/x86/mm/tlb.c void flush_tlb_all(void) · line 1089

Quoted 5 lines
arch/x86/mm/tlb.c +5 -0
+	/*
+	 * TLBSYNC only waits for flushes originating on the same CPU.
+	 * Disabling migration allows us to wait on all flushes.
+	 */
+	guard(preempt)();

Migration?

Why not migrate_disable() then?

Although there's a big, thorny comment in include/linux/preempt.h about its influence on sched.

Quoted 14 lines
arch/x86/mm/tlb.c +14 -0
+
+	if (end == TLB_FLUSH_ALL ||
+	    (end - start) > threshold << PAGE_SHIFT) {
+		invlpgb_flush_all();
+	} else {
+		unsigned long nr;
+		for (addr = start; addr < end; addr += nr << PAGE_SHIFT) {
+			nr = min((end - addr) >> PAGE_SHIFT, maxnr);
+			invlpgb_flush_addr(addr, nr);
+		}
+	}
+
+	tlbsync();
+}
Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#33 lore raw

Re: [PATCH 06/12] x86/mm: use INVLPGB for kernel TLB flushes

In reply to Rik van Riel
Quoted 3 lines · Rik van Riel

Use broadcast TLB invalidation for kernel addresses when available.

This stops us from having to send IPIs for kernel TLB flushes.

Could this be changed to imperative voice, please?

Remove the need to send IPIs for kernel TLB flushes.

arch/x86/mm/tlb.c void flush_tlb_all(void) · line 1084

Quoted 5 lines
arch/x86/mm/tlb.c +5 -0
+static void broadcast_kernel_range_flush(unsigned long start, unsigned long end)
+{
+	unsigned long addr;
+	unsigned long maxnr = invlpgb_count_max;
+	unsigned long threshold = tlb_single_page_flush_ceiling * maxnr;

The 'tlb_single_page_flush_ceiling' value was determined by looking at _local_ invalidation cost. Could you talk a bit about why it's also a good value to use for remote invalidations? Does it hold up for INVLPGB the same way it did for good ol' INVLPG? Has there been any explicit testing here to find a good value?

I'm also confused by the multiplication here. Let's say invlpgb_count_max==20 and tlb_single_page_flush_ceiling==30.

You would need to switch away from single-address invalidation when the number of addresses is >20 for INVLPGB functional reasons. But you'd also need to switch away when >30 for performance reasons (tlb_single_page_flush_ceiling).

But I don't understand how that would make the threshold 20*30=600 invalidations.

arch/x86/mm/tlb.c void flush_tlb_all(void) · line 1089

Quoted 4 lines
arch/x86/mm/tlb.c +4 -0
+	/*
+	 * TLBSYNC only waits for flushes originating on the same CPU.
+	 * Disabling migration allows us to wait on all flushes.
+	 */

Imperative voice here too, please:

Disable migration to wait on all flushes.

arch/x86/mm/tlb.c void flush_tlb_all(void) · line 1093

Quoted 4 lines
arch/x86/mm/tlb.c +4 -0
+	guard(preempt)();
+
+	if (end == TLB_FLUSH_ALL ||
+	    (end - start) > threshold << PAGE_SHIFT) {

This is basically a copy-and-paste of the "range vs. global" flush logic, but taking 'invlpgb_count_max' into account.

It would be ideal if those limit checks could be consolidated. I suspect that when the 'threshold' calculation above gets clarified that they may be easier to consolidate.

BTW, what is a typical value for 'invlpgb_count_max'? Is it more or less than the typical value for 'tlb_single_page_flush_ceiling'?

Maybe we should just lower 'tlb_single_page_flush_ceiling' if 'invlpgb_count_max' falls below it so we only have _one_ runtime value to consider.

Quoted 17 lines
arch/x86/mm/tlb.c +12 -0
+		invlpgb_flush_all();
+	} else {
+		unsigned long nr;
+		for (addr = start; addr < end; addr += nr << PAGE_SHIFT) {
+			nr = min((end - addr) >> PAGE_SHIFT, maxnr);
+			invlpgb_flush_addr(addr, nr);
+		}
+	}
+
+	tlbsync();
+}
+
 static void do_kernel_range_flush(void *info)
 {
 	struct flush_tlb_info *f = info;
@@ -1089,6 +1115,11 @@ static void do_kernel_range_flush(void *info)
 

arch/x86/mm/tlb.c static void do_kernel_range_flush(void *info) · line 1125

Quoted 10 lines
arch/x86/mm/tlb.c +5 -0
 void flush_tlb_kernel_range(unsigned long start, unsigned long end)
 {
+	if (cpu_feature_enabled(X86_FEATURE_INVLPGB)) {
+		broadcast_kernel_range_flush(start, end);
+		return;
+	}
+
 	/* Balance as user space task's flush, a bit conservative */
 	if (end == TLB_FLUSH_ALL ||
 	    (end - start) > tlb_single_page_flush_ceiling << PAGE_SHIFT) {

I also wonder if this would all get simpler if we give in and *always* call get_flush_tlb_info(). That would provide a nice single place to consolidate the "all vs. ranged" flush logic.

#34 lore raw

Re: [PATCH 06/12] x86/mm: use INVLPGB for kernel TLB flushes

Quoted 22 lines · Dave Hansen
Quoted 7 lines · Rik van Riel

+static void broadcast_kernel_range_flush(unsigned long start, unsigned long end)

arch/x86/mm/tlb.c +4 -0
+{
+	unsigned long addr;
+	unsigned long maxnr = invlpgb_count_max;
+	unsigned long threshold = tlb_single_page_flush_ceiling *

maxnr;

The 'tlb_single_page_flush_ceiling' value was determined by looking at
_local_ invalidation cost. Could you talk a bit about why it's also a good value to use for remote invalidations? Does it hold up for INVLPGB
the same way it did for good ol' INVLPG? Has there been any explicit testing here to find a good value?

I'm also confused by the multiplication here. Let's say invlpgb_count_max==20 and tlb_single_page_flush_ceiling==30.

Quoted 8 lines

You would need to switch away from single-address invalidation when the
number of addresses is >20 for INVLPGB functional reasons. But you'd also need to switch away when >30 for performance reasons (tlb_single_page_flush_ceiling).

But I don't understand how that would make the threshold 20*30=600 invalidations.

I have not done any measurement to see how flushing with INVLPGB stacks up versus local TLB flushes.

What makes INVLPGB potentially slower:
- These flushes are done globally

What makes INVLPGB potentially faster:
- Multiple flushes can be pending simultaneously, and executed in any convenient order by the CPUs.
- Wait once on completion of all the queued flushes.

Another thing that makes things interesting is the TLB entry coalescing done by AMD CPUs.

When multiple pages are both virtually and physically contiguous in memory (which is fairly common), the CPU can use a single TLB entry to map up to 8 of them.

That means if we issue eg. 20 INVLPGB flushes for 8 4kB pages each, instead of the CPUs needing to remove 160 TLB entries, there might only be 50.

I just guessed at the numbers used in my code, while trying to sort out the details elsewhere in the code.

How should we go about measuring the tradeoffs between invalidation time, and the time spent in TLB misses from flushing unnecessary stuff?

Quoted 19 lines

arch/x86/mm/tlb.c void flush_tlb_all(void) · line 1089

Quoted 6 lines
(quoted diff) +2 -0
+	/*
+	 * TLBSYNC only waits for flushes originating on the same

CPU.

arch/x86/mm/tlb.c +2 -0
+	 * Disabling migration allows us to wait on all flushes.
+	 */

Imperative voice here too, please:

Disable migration to wait on all flushes.

arch/x86/mm/tlb.c void flush_tlb_all(void) · line 1093

Quoted 4 lines
arch/x86/mm/tlb.c +4 -0
+	guard(preempt)();
+
+	if (end == TLB_FLUSH_ALL ||
+	    (end - start) > threshold << PAGE_SHIFT) {

This is basically a copy-and-paste of the "range vs. global" flush logic, but taking 'invlpgb_count_max' into account.

Quoted 5 lines

It would be ideal if those limit checks could be consolidated. I suspect
that when the 'threshold' calculation above gets clarified that they may
be easier to consolidate.

Maybe?

I implemented another suggestion in the code, and the start of flush_tlb_kernel_range() now looks like this:

void flush_tlb_kernel_range(unsigned long start, unsigned long end) {
if (broadcast_kernel_range_flush(start, end))
return;

If we are to consolidate the limit check, it should probably be in a helper function somewhere, and not by spreading out the broadcast flush calls.

Quoted 9 lines

BTW, what is a typical value for 'invlpgb_count_max'? Is it more or less
than the typical value for 'tlb_single_page_flush_ceiling'?

Maybe we should just lower 'tlb_single_page_flush_ceiling' if 'invlpgb_count_max' falls below it so we only have _one_ runtime value
to consider.

The value for invlpgb_count_max on both Milan and Bergamo CPUs appears to be 8. That is, the CPU reports we can flush 7 additional pages beyond a single page.

This matches the number of PTEs that can be cached in one TLB entry if they are contiguous and aligned, and matches one cache line worth of PTE entries.

Quoted 32 lines
Quoted 32 lines
arch/x86/mm/tlb.c +4 -0
+		invlpgb_flush_all();
+	} else {
+		unsigned long nr;
+		for (addr = start; addr < end; addr += nr <<

PAGE_SHIFT) {

(quoted diff) +1 -0
+			nr = min((end - addr) >> PAGE_SHIFT,

maxnr);

arch/x86/mm/tlb.c +7 -0
+			invlpgb_flush_addr(addr, nr);
+		}
+	}
+
+	tlbsync();
+}
+

 static void do_kernel_range_flush(void *info)
 {
  struct flush_tlb_info *f = info;

(quoted diff) +0 -0
@@ -1089,6 +1115,11 @@ static void do_kernel_range_flush(void
 *info)


 void flush_tlb_kernel_range(unsigned long start, unsigned long end)
 {

arch/x86/mm/tlb.c +5 -0
+	if (cpu_feature_enabled(X86_FEATURE_INVLPGB)) {
+		broadcast_kernel_range_flush(start, end);
+		return;
+	}
+

  /* Balance as user space task's flush, a bit conservative */

Quoted 8 lines
Quoted 3 lines

  if (end == TLB_FLUSH_ALL ||
      (end - start) > tlb_single_page_flush_ceiling << PAGE_SHIFT) {

I also wonder if this would all get simpler if we give in and *always*
call get_flush_tlb_info(). That would provide a nice single place to consolidate the "all vs. ranged" flush logic.

Possibly. That might be a good way to unify that threshold check?

That should probably be a separate patch, though.

Signature · 1 line
All Rights Reversed.
#35 lore raw

Re: [PATCH 06/12] x86/mm: use INVLPGB for kernel TLB flushes

Quoted 44 lines · Rik van Riel
Quoted 20 lines · Dave Hansen
Quoted 7 lines · Rik van Riel

+static void broadcast_kernel_range_flush(unsigned long start, unsigned long end)

arch/x86/mm/tlb.c +4 -0
+{
+	unsigned long addr;
+	unsigned long maxnr = invlpgb_count_max;
+	unsigned long threshold = tlb_single_page_flush_ceiling *

maxnr;

The 'tlb_single_page_flush_ceiling' value was determined by looking at _local_ invalidation cost. Could you talk a bit about why it's also a good value to use for remote invalidations? Does it hold up for INVLPGB the same way it did for good ol' INVLPG? Has there been any explicit testing here to find a good value?

I'm also confused by the multiplication here. Let's say invlpgb_count_max==20 and tlb_single_page_flush_ceiling==30.

Quoted 7 lines

You would need to switch away from single-address invalidation when the number of addresses is >20 for INVLPGB functional reasons. But you'd also need to switch away when >30 for performance reasons (tlb_single_page_flush_ceiling).

But I don't understand how that would make the threshold 20*30=600 invalidations.

I have not done any measurement to see how flushing with INVLPGB stacks up versus
local TLB flushes.

What makes INVLPGB potentially slower:
- These flushes are done globally

What makes INVLPGB potentially faster:
- Multiple flushes can be pending simultaneously, and executed in any convenient order by the CPUs.
- Wait once on completion of all the queued flushes.

Another thing that makes things interesting is the TLB entry coalescing done by AMD CPUs.

Quoted 7 lines

When multiple pages are both virtually and physically contiguous in memory (which is fairly common), the CPU can use a single TLB entry to map up to 8 of them.

That means if we issue eg. 20 INVLPGB flushes for 8 4kB pages each, instead of the CPUs needing to remove 160 TLB entries, there might only be 50.

I honestly don't expect there to be any real difference in INVLPGB execution on the sender side based on what the receivers have in their TLB.

Quoted 7 lines

I just guessed at the numbers used in my code, while trying to sort out the details elsewhere in the code.

How should we go about measuring the tradeoffs between invalidation time, and the time spent in TLB misses from flushing unnecessary stuff?

Well, we did a bunch of benchmarks for INVLPG. We could dig that back up and repeat some of it.

But actually I think INVLPGB is *WAY* better than INVLPG here. INVLPG doesn't have ranged invalidation. It will only architecturally invalidate multiple 4K entries when the hardware fractured them in the first place. I think we should probably take advantage of what INVLPGB can do instead of following the INVLPG approach.

INVLPGB will invalidate a range no matter where the underlying entries came from. Its "increment the virtual address at the 2M boundary" mode will invalidate entries of any size. That's my reading of the docs at least. Is that everyone else's reading too?

So, let's pick a number "Z" which is >= invlpgb_count_max. Z could arguably be set to tlb_single_page_flush_ceiling. Then do this:

	   4k -> Z*4k => use 4k step
	>Z*4k -> Z*2M => use 2M step
	>Z*2M	      => invalidate everything

Invalidations <=Z*4k are exact. They never zap extra TLB entries.

Invalidations that use the 2M step *might* unnecessarily zap some extra 4k mappings in the last 2M, but this is *WAY* better than invalidating everything.

"Invalidate everything" obviously stinks, but it should only be for pretty darn big invalidations. This approach can also do a true ranged INVLPGB for many more cases than the existing proposal. The only issue would be if the 2M step is substantially more expensive than the 4k step.

...

Quoted 7 lines
Quoted 3 lines

I also wonder if this would all get simpler if we give in and *always* call get_flush_tlb_info(). That would provide a nice single place to consolidate the "all vs. ranged" flush logic.

Possibly. That might be a good way to unify that threshold check?

That should probably be a separate patch, though.

Yes, it should be part of refactoring that comes before the INVLPGB enabling.

#36 lore raw

Re: [PATCH 06/12] x86/mm: use INVLPGB for kernel TLB flushes

Quoted 10 lines · Dave Hansen

But actually I think INVLPGB is *WAY* better than INVLPG here. INVLPG
doesn't have ranged invalidation. It will only architecturally invalidate multiple 4K entries when the hardware fractured them in the
first place. I think we should probably take advantage of what INVLPGB
can do instead of following the INVLPG approach.

Quoted 6 lines

INVLPGB will invalidate a range no matter where the underlying entries
came from. Its "increment the virtual address at the 2M boundary" mode
will invalidate entries of any size. That's my reading of the docs at least. Is that everyone else's reading too?

Ohhhh, good point! I glossed over that the first half dozen times I was reading the document, because I was trying to use the ASID, and working to figure out why things kept crashing (turns out I can only use the PCID on bare metal)

Quoted 8 lines

So, let's pick a number "Z" which is >= invlpgb_count_max. Z could arguably be set to tlb_single_page_flush_ceiling. Then do this:

	   4k -> Z*4k => use 4k step
	>Z*4k -> Z*2M => use 2M step
	>Z*2M	      => invalidate everything
Quoted 8 lines

Invalidations <=Z*4k are exact. They never zap extra TLB entries.

Invalidations that use the 2M step *might* unnecessarily zap some extra
4k mappings in the last 2M, but this is *WAY* better than invalidating
everything.

This is a great idea.

Then the code in get_flush_tlb_info can adjust start, end, and stride_shift as needed.

INVLPGB also supports invalidation of an entire 1GB region, so we can take your idea one step further :)

With up to 8 pages zapped by a single INVLPGB instruction, and multiple in flight simultaneously, maybe we could set the threshold to 64, for 8 INVLPGBs in flight at once?

That way we can invalidate up to 1/8th of a 512 entry range with individual zaps, before just zapping the higher level entry.

Quoted 2 lines

"Invalidate everything" obviously stinks, but it should only be for pretty darn big invalidations.

That would only come into play when we get past several GB worth of invalidation.

Signature · 1 line
All Rights Reversed.
#37 lore raw

Re: [PATCH 06/12] x86/mm: use INVLPGB for kernel TLB flushes

In reply to Dave Hansen
Quoted 10 lines · Dave Hansen

But actually I think INVLPGB is *WAY* better than INVLPG here. INVLPG doesn't have ranged invalidation. It will only architecturally invalidate multiple 4K entries when the hardware fractured them in the first place. I think we should probably take advantage of what INVLPGB can do instead of following the INVLPG approach.

Quoted 4 lines

INVLPGB will invalidate a range no matter where the underlying entries came from. Its "increment the virtual address at the 2M boundary" mode will invalidate entries of any size. That's my reading of the docs at least. Is that everyone else's reading too?

This is not my reading. I think that this reading assumes that besides the broadcast, some new “range flush” was added to the TLB. My guess is that this not the case, since presumably it would require a different TLB structure (and who does 2 changes at once ;-) ).

My understanding is therefore that it’s all in microcode. There is a “stride” and “number” which are used by the microcode for iterating and on every iteration the microcode issues a TLB invalidation. This invalidation is similar to INVLPG, just as it was always done (putting aside the variants that do not invalidate the PWC). IOW, the page-size is not given as part of the INVLPG and not as part of INVLPGB (regardless of the stride) for whatever entries used to invalidate a given address.

I think my understanding is backed by the wording of "regardless of the page size” appearing for INVLPG as well in AMD’s manual.

My guess is that invalidating more entries will take longer time, maybe not on the sender, but at least on the receiver. I also guess that in certain setups - big NUMA machines - INVLPGB might perform worse. I remember vaguely ARM guys writing something about such behavior.

#38 lore raw

Re: [PATCH 06/12] x86/mm: use INVLPGB for kernel TLB flushes

Quoted 4 lines · Nadav Amit

This is not my reading. I think that this reading assumes that besides the broadcast, some new “range flush” was added to the TLB. My guess is that this not the case, since presumably it would require a different TLB structure (and who does 2 changes at once 😉 ).

Reading it again, I think you're right.

The INVLPG and INVLPGB language is too close. It would also _talk_ about invalidating a range rather than just incrementing an address to invalidate.

I think the key thing we need to decide is whether to treat a single INVLPGB(stride=8) more like a single INVLPGB or eight INVLPGBs. Measuring a bunch of invalidation looks should tell us that.

#39 lore raw

Re: [PATCH 06/12] x86/mm: use INVLPGB for kernel TLB flushes

Quoted 11 lines · Dave Hansen
Quoted 7 lines · Nadav Amit

This is not my reading. I think that this reading assumes that besides
the broadcast, some new “range flush” was added to the TLB. My guess
is that this not the case, since presumably it would require a different
TLB structure (and who does 2 changes at once 😉 ).

Reading it again, I think you're right.

Quoted 8 lines

The INVLPG and INVLPGB language is too close. It would also _talk_ about
invalidating a range rather than just incrementing an address to invalidate.

I think the key thing we need to decide is whether to treat a single INVLPGB(stride=8) more like a single INVLPGB or eight INVLPGBs. Measuring a bunch of invalidation looks should tell us that.

Would I be wrong to assume that the CPUs have some optimizations built in to efficiently execute an invalidation for "everything in a PCID"?

The "global invalidate" we send does not zap everything in the TLB, but only the translations for a single PCID.

I suppose we should measure these things at some point (after I do the other cleanups?), because the CPUs may well have made a bunch of optimizations that we don't know about.

Signature · 1 line
All Rights Reversed.
#40 lore raw

Re: [PATCH 06/12] x86/mm: use INVLPGB for kernel TLB flushes

Quoted 1 line · Rik van Riel

On Fri, 2025-01-10 at 07:14 -0800, Dave Hansen wrote:

...

Quoted 8 lines
Quoted 3 lines

I think the key thing we need to decide is whether to treat a single INVLPGB(stride=8) more like a single INVLPGB or eight INVLPGBs. Measuring a bunch of invalidation looks should tell us that.

Would I be wrong to assume that the CPUs have some optimizations built in to efficiently execute an invalidation for "everything in a PCID"?

There's only a few bits in the actual TLBs to store the PCID (or VPID), roughly 3 on Intel. Then there's another structure to map between the architectural PCID and the 3 bits of actual hardware alias.

That's what I know. The rest is pure speculation:

All you have to do in theory is zap the one entry in the PCID=>HW mapping structure to invalidate a whole PCID. You don't need to run through the TLB itself to invalidate it.

You need to do something else to make sure that the now-unused 3-bit hardware identifier gets reused at _some_ point, but there may be other tricks for that.

Quoted 9 lines

The "global invalidate" we send does not zap everything in the TLB, but only the translations for a single PCID.

I suppose we should measure these things at some point (after I do the other cleanups?), because the CPUs may well have made a bunch of optimizations that we don't know about.

IIRC, the "big" invalidation modes are pretty cheap to execute. Most of the cost comes from the TLB refill, not the flush itself.

But there's no substitute for actually measuring it. There's some wonky stuff out there. The last time Andy L. went and looked at it, there were oddities like INVPCID's "Individual-address invalidation" and INVLPG having surprisingly different performance.

#41 lore raw

Re: [PATCH 06/12] x86/mm: use INVLPGB for kernel TLB flushes

Quoted 6 lines · Dave Hansen
Quoted 1 line

about.

IIRC, the "big" invalidation modes are pretty cheap to execute. Most of
the cost comes from the TLB refill, not the flush itself.

Quoted 7 lines

But there's no substitute for actually measuring it. There's some wonky
stuff out there. The last time Andy L. went and looked at it, there were
oddities like INVPCID's "Individual-address invalidation" and INVLPG having surprisingly different performance.

Agreed on all points.

If you guys don't mind, I'm going to keep this on the back burner, while I work my way through all the suggested code cleanups and improvements first.

Signature · 1 line
All Rights Reversed.
#42 lore raw

Re: [PATCH 06/12] x86/mm: use INVLPGB for kernel TLB flushes

In reply to Rik van Riel
Quoted 28 lines · Rik van Riel

Use broadcast TLB invalidation for kernel addresses when available.

This stops us from having to send IPIs for kernel TLB flushes.

---
 arch/x86/mm/tlb.c | 31 +++++++++++++++++++++++++++++++
 1 file changed, 31 insertions(+)
arch/x86/mm/tlb.c +11 -0
index 6cf881a942bb..29207dc5b807 100644
@@ -1077,6 +1077,32 @@ void flush_tlb_all(void)
 	on_each_cpu(do_flush_tlb_all, NULL, 1);
 }
 
+static void broadcast_kernel_range_flush(unsigned long start, unsigned long end)
+{
+	unsigned long addr;
+	unsigned long maxnr = invlpgb_count_max;
+	unsigned long threshold = tlb_single_page_flush_ceiling * maxnr;
+
+	/*
+	 * TLBSYNC only waits for flushes originating on the same CPU.
+	 * Disabling migration allows us to wait on all flushes.
+	 */
+	guard(preempt)();

arch/x86/mm/tlb.c void flush_tlb_all(void) · line 1100

Quoted 10 lines
arch/x86/mm/tlb.c +10 -0
+
+	if (end == TLB_FLUSH_ALL ||
+	    (end - start) > threshold << PAGE_SHIFT) {
+		invlpgb_flush_all();
+	} else {
+		unsigned long nr;
+		for (addr = start; addr < end; addr += nr << PAGE_SHIFT) {
+			nr = min((end - addr) >> PAGE_SHIFT, maxnr);
+			invlpgb_flush_addr(addr, nr);
+		}

Would it be better to put this loop in the actual invlpgb_flush* function(s)? Then callers don't have to worry about it, similar to what is done in clflush_cache_range() / clflush_cache_range_opt().

Thanks,
Tom

Quoted 20 lines
arch/x86/mm/tlb.c +10 -0
+	}
+
+	tlbsync();
+}
+
 static void do_kernel_range_flush(void *info)
 {
 	struct flush_tlb_info *f = info;
@@ -1089,6 +1115,11 @@ static void do_kernel_range_flush(void *info)
 
 void flush_tlb_kernel_range(unsigned long start, unsigned long end)
 {
+	if (cpu_feature_enabled(X86_FEATURE_INVLPGB)) {
+		broadcast_kernel_range_flush(start, end);
+		return;
+	}
+
 	/* Balance as user space task's flush, a bit conservative */
 	if (end == TLB_FLUSH_ALL ||
 	    (end - start) > tlb_single_page_flush_ceiling << PAGE_SHIFT) {
#43 lore raw

Re: [PATCH 06/12] x86/mm: use INVLPGB for kernel TLB flushes

Quoted 26 lines · Tom Lendacky
Quoted 25 lines · Rik van Riel

Use broadcast TLB invalidation for kernel addresses when available.

+static void broadcast_kernel_range_flush(unsigned long start, unsigned long end)

arch/x86/mm/tlb.c +4 -0
+{
+	unsigned long addr;
+	unsigned long maxnr = invlpgb_count_max;
+	unsigned long threshold = tlb_single_page_flush_ceiling *

maxnr;

arch/x86/mm/tlb.c +3 -0
+
+	/*
+	 * TLBSYNC only waits for flushes originating on the same

CPU.

arch/x86/mm/tlb.c +10 -0
+	 * Disabling migration allows us to wait on all flushes.
+	 */
+	guard(preempt)();
+
+	if (end == TLB_FLUSH_ALL ||
+	    (end - start) > threshold << PAGE_SHIFT) {
+		invlpgb_flush_all();
+	} else {
+		unsigned long nr;
+		for (addr = start; addr < end; addr += nr <<

PAGE_SHIFT) {

(quoted diff) +1 -0
+			nr = min((end - addr) >> PAGE_SHIFT,
Quoted 8 lines

Review

Quoted 3 lines

maxnr);

(quoted diff) +2 -0
+			invlpgb_flush_addr(addr, nr);
+		}

Would it be better to put this loop in the actual invlpgb_flush* function(s)? Then callers don't have to worry about it, similar to what is
done in clflush_cache_range() / clflush_cache_range_opt().

Maybe?

We only have one caller, though.

Signature · 1 line
All Rights Reversed.
#44 lore raw

PATCH v3 7/12 x86/tlb: use INVLPGB in flush_tlb_all

In reply to Rik van Riel

The flush_tlb_all() function is not used a whole lot, but we might as well use broadcast TLB flushing there, too.

---
 arch/x86/mm/tlb.c | 6 ++++++
 1 file changed, 6 insertions(+)
arch/x86/mm/tlb.c +6 -0
index 29207dc5b807..266d5174fc7b 100644
@@ -1074,6 +1074,12 @@ static void do_flush_tlb_all(void *info)
 void flush_tlb_all(void)
 {
 	count_vm_tlb_event(NR_TLB_REMOTE_FLUSH);
+	if (cpu_feature_enabled(X86_FEATURE_INVLPGB)) {
+		guard(preempt)();
+		invlpgb_flush_all();
+		tlbsync();
+		return;
+	}
 	on_each_cpu(do_flush_tlb_all, NULL, 1);
 }
 
Signature · 1 line
2.47.1
#45 lore raw

Re: [PATCH 07/12] x86/tlb: use INVLPGB in flush_tlb_all

arch/x86/mm/tlb.c static void do_flush_tlb_all(void *info) · line 1082

Quoted 12 lines · Rik van Riel
arch/x86/mm/tlb.c +6 -0
@@ -1074,6 +1074,12 @@ static void do_flush_tlb_all(void *info)
 void flush_tlb_all(void)
 {
 	count_vm_tlb_event(NR_TLB_REMOTE_FLUSH);
+	if (cpu_feature_enabled(X86_FEATURE_INVLPGB)) {
+		guard(preempt)();
+		invlpgb_flush_all();
+		tlbsync();
+		return;
+	}

After seeing a few of these, I'd really prefer that the preempt and tlbsync() logic be hidden in the invlpgb_*() helper, or *a* helper at least.

This would be a lot easier on the eyes if it were something like:

	flushed = invlpgb_flush_all();
	if (flushed)
		return;

or even:

	if (cpu_feature_enabled(X86_FEATURE_INVLPGB)) {
		invlpgb_flush_all();
		return;
	}
#46 lore raw

Re: [PATCH 07/12] x86/tlb: use INVLPGB in flush_tlb_all

Quoted 18 lines · Dave Hansen

arch/x86/mm/tlb.c static void do_flush_tlb_all(void *info) · line 1082

Quoted 12 lines · Rik van Riel
arch/x86/mm/tlb.c +6 -0
@@ -1074,6 +1074,12 @@ static void do_flush_tlb_all(void *info)
  void flush_tlb_all(void)
  {
  	count_vm_tlb_event(NR_TLB_REMOTE_FLUSH);
+	if (cpu_feature_enabled(X86_FEATURE_INVLPGB)) {
+		guard(preempt)();
+		invlpgb_flush_all();
+		tlbsync();
+		return;
+	}

After seeing a few of these, I'd really prefer that the preempt and tlbsync() logic be hidden in the invlpgb_*() helper, or *a* helper at least.

Quoted 5 lines

This would be a lot easier on the eyes if it were something like:

	flushed = invlpgb_flush_all();
	if (flushed)
		return;

One issue here is that some of the invlpgb helpers are supposed to be asynchronous, because we can have multiple of those flushes pending simultaneously, and then wait for them to complete with a tlbsync.

How would we avoid the confusion between the two types (async vs sync) invlpgb helpers?

I'm all for cleaning this up, but I have not thought of a good idea yet...

Signature · 1 line
All Rights Reversed.
#47 lore raw

Re: [PATCH 07/12] x86/tlb: use INVLPGB in flush_tlb_all

Quoted 25 lines · Rik van Riel
Quoted 18 lines · Dave Hansen

arch/x86/mm/tlb.c static void do_flush_tlb_all(void *info) · line 1082

Quoted 12 lines · Rik van Riel
arch/x86/mm/tlb.c +6 -0
@@ -1074,6 +1074,12 @@ static void do_flush_tlb_all(void *info)
  void flush_tlb_all(void)
  {
  	count_vm_tlb_event(NR_TLB_REMOTE_FLUSH);
+	if (cpu_feature_enabled(X86_FEATURE_INVLPGB)) {
+		guard(preempt)();
+		invlpgb_flush_all();
+		tlbsync();
+		return;
+	}

After seeing a few of these, I'd really prefer that the preempt and tlbsync() logic be hidden in the invlpgb_*() helper, or *a* helper at least.

Quoted 5 lines

This would be a lot easier on the eyes if it were something like:

	flushed = invlpgb_flush_all();
	if (flushed)
		return;
Quoted 7 lines

One issue here is that some of the invlpgb helpers are supposed to be asynchronous, because we can have multiple of those flushes pending simultaneously, and then wait for them to complete with a tlbsync.

How would we avoid the confusion between the two types (async vs sync) invlpgb helpers?

It could be done with naming. Either preface things with __ or give them "sync" suffixes.

We could also do it with a calling convention:

	struct invlpgb_seq;

	start_invlpgb(&invlpgb_seq);
	invlpgb_flush_addr(&invlpgb_seq, start, end);
	end_invlpgb(&invlpgb_seq);

The things that can logically get done in sequence need to have the start/end, and need to have the struct passed in. The ones that have the internal sync don't have the argument.

#48 lore raw

PATCH v3 8/12 x86/mm: use broadcast TLB flushing for page reclaim TLB flushing

In reply to Rik van Riel

In the page reclaim code, we only track the CPU(s) where the TLB needs to be flushed, rather than all the individual mappings that may be getting invalidated.

Use broadcast TLB flushing when that is available.

---
 arch/x86/mm/tlb.c | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)
arch/x86/mm/tlb.c +9 -1
index 266d5174fc7b..64f1679c37e1 100644
@@ -1310,8 +1310,16 @@ EXPORT_SYMBOL_GPL(__flush_tlb_all);
 void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch)
 {
 	struct flush_tlb_info *info;
+	int cpu;
+
+	if (cpu_feature_enabled(X86_FEATURE_INVLPGB)) {
+		guard(preempt)();
+		invlpgb_flush_all_nonglobals();
+		tlbsync();
+		return;
+	}
 
-	int cpu = get_cpu();
+	cpu = get_cpu();
 
 	info = get_flush_tlb_info(NULL, 0, TLB_FLUSH_ALL, 0, false,
 				  TLB_GENERATION_INVALID);
Signature · 1 line
2.47.1
#49 lore raw

PATCH v3 9/12 x86/mm: enable broadcast TLB invalidation for multi-threaded processes

In reply to Rik van Riel

Use broadcast TLB invalidation, using the INVPLGB instruction, on AMD EPYC 3 and newer CPUs.

In order to not exhaust PCID space, and keep TLB flushes local for single threaded processes, we only hand out broadcast ASIDs to processes active on 3 or more CPUs, and gradually increase the threshold as broadcast ASID space is depleted.

---
 arch/x86/include/asm/mmu.h         |   6 +
 arch/x86/include/asm/mmu_context.h |  12 ++
 arch/x86/include/asm/tlbflush.h    |  17 ++
 arch/x86/mm/tlb.c                  | 310 ++++++++++++++++++++++++++++-
 4 files changed, 336 insertions(+), 9 deletions(-)
arch/x86/include/asm/mmu.h +6 -0
index 3b496cdcb74b..a8e8dfa5a520 100644
@@ -48,6 +48,12 @@ typedef struct {
 	unsigned long flags;
 #endif
 
+#ifdef CONFIG_CPU_SUP_AMD
+	struct list_head broadcast_asid_list;
+	u16 broadcast_asid;
+	bool asid_transition;
+#endif
+
 #ifdef CONFIG_ADDRESS_MASKING
 	/* Active LAM mode:  X86_CR3_LAM_U48 or X86_CR3_LAM_U57 or 0 (disabled) */
 	unsigned long lam_cr3_mask;
arch/x86/include/asm/mmu_context.h +12 -0
index 795fdd53bd0a..0dc446c427d2 100644
@@ -139,6 +139,8 @@ static inline void mm_reset_untag_mask(struct mm_struct *mm)
 #define enter_lazy_tlb enter_lazy_tlb
 extern void enter_lazy_tlb(struct mm_struct *mm, struct task_struct *tsk);
 
+extern void destroy_context_free_broadcast_asid(struct mm_struct *mm);
+
 /*
  * Init a new mm.  Used on mm copies, like at fork()
  * and on mm's that are brand-new, like at execve().
@@ -161,6 +163,13 @@ static inline int init_new_context(struct task_struct *tsk,
 		mm->context.execute_only_pkey = -1;
 	}
 #endif
+
+#ifdef CONFIG_CPU_SUP_AMD
+	INIT_LIST_HEAD(&mm->context.broadcast_asid_list);
+	mm->context.broadcast_asid = 0;
+	mm->context.asid_transition = false;
+#endif
+
 	mm_reset_untag_mask(mm);
 	init_new_context_ldt(mm);
 	return 0;
@@ -170,6 +179,9 @@ static inline int init_new_context(struct task_struct *tsk,
 static inline void destroy_context(struct mm_struct *mm)
 {
 	destroy_context_ldt(mm);
+#ifdef CONFIG_CPU_SUP_AMD
+	destroy_context_free_broadcast_asid(mm);
+#endif
 }
 
 extern void switch_mm(struct mm_struct *prev, struct mm_struct *next,
arch/x86/include/asm/tlbflush.h +17 -0
index 20074f17fbcd..5e9956af98d1 100644
@@ -65,6 +65,23 @@ static inline void cr4_clear_bits(unsigned long mask)
  */
 #define TLB_NR_DYN_ASIDS	6
 
+#ifdef CONFIG_CPU_SUP_AMD
+#define is_dyn_asid(asid) (asid) < TLB_NR_DYN_ASIDS
+#define is_broadcast_asid(asid) (asid) >= TLB_NR_DYN_ASIDS
+#define in_asid_transition(info) (info->mm && info->mm->context.asid_transition)
+#define mm_broadcast_asid(mm) (mm->context.broadcast_asid)
+#else
+#define is_dyn_asid(asid) true
+#define is_broadcast_asid(asid) false
+#define in_asid_transition(info) false
+#define mm_broadcast_asid(mm) 0
+
+inline bool needs_broadcast_asid_reload(struct mm_struct *next, u16 prev_asid)
+{
+	return false;
+}
+#endif
+
 struct tlb_context {
 	u64 ctx_id;
 	u64 tlb_gen;
arch/x86/mm/tlb.c +301 -9

397 lines not on this page · raw message

Signature · 1 line
2.47.1
#50 lore raw

Re: [PATCH 09/12] x86/mm: enable broadcast TLB invalidation for multi-threaded processes

Quoted 15 lines · Rik van Riel

Use broadcast TLB invalidation, using the INVPLGB instruction, on AMD EPYC 3 and newer CPUs.

In order to not exhaust PCID space, and keep TLB flushes local for single threaded processes, we only hand out broadcast ASIDs to processes active on 3 or more CPUs, and gradually increase the threshold as broadcast ASID space is depleted.

---

[snip]

Quoted 26 lines
arch/x86/include/asm/mmu_context.h +9 -0
@@ -139,6 +139,8 @@ static inline void mm_reset_untag_mask(struct mm_struct *mm)
 #define enter_lazy_tlb enter_lazy_tlb
 extern void enter_lazy_tlb(struct mm_struct *mm, struct task_struct *tsk);
 
+extern void destroy_context_free_broadcast_asid(struct mm_struct *mm);
+
 /*
 * Init a new mm.  Used on mm copies, like at fork()
 * and on mm's that are brand-new, like at execve().
@@ -161,6 +163,13 @@ static inline int init_new_context(struct task_struct *tsk,
 		mm->context.execute_only_pkey = -1;
 	}
 #endif
+
+#ifdef CONFIG_CPU_SUP_AMD
+	INIT_LIST_HEAD(&mm->context.broadcast_asid_list);
+	mm->context.broadcast_asid = 0;
+	mm->context.asid_transition = false;
+#endif
+
 	mm_reset_untag_mask(mm);
 	init_new_context_ldt(mm);
 	return 0;

arch/x86/include/asm/mmu_context.h static inline int init_new_context(struct task_struct *tsk, · line 184

Quoted 6 lines
arch/x86/include/asm/mmu_context.h +3 -0
@@ -170,6 +179,9 @@ static inline int init_new_context(struct task_struct *tsk,
 static inline void destroy_context(struct mm_struct *mm)
 {
 	destroy_context_ldt(mm);
+#ifdef CONFIG_CPU_SUP_AMD
+	destroy_context_free_broadcast_asid(mm);
+#endif

This ifdef’ry is not great. I think it’s better to have entire functions in ifdef than put ifdef’s within the code.

Quoted 11 lines
arch/x86/include/asm/mmu_context.h +0 -0
 }
 
 extern void switch_mm(struct mm_struct *prev, struct mm_struct *next,
arch/x86/include/asm/tlbflush.h +0 -0
index 20074f17fbcd..5e9956af98d1 100644
@@ -65,6 +65,23 @@ static inline void cr4_clear_bits(unsigned long mask)
 */
 #define TLB_NR_DYN_ASIDS	6
 

arch/x86/include/asm/tlbflush.h static inline void cr4_clear_bits(unsigned long mask) · line 77

Quoted 10 lines
arch/x86/include/asm/tlbflush.h +10 -0
+#ifdef CONFIG_CPU_SUP_AMD
+#define is_dyn_asid(asid) (asid) < TLB_NR_DYN_ASIDS
+#define is_broadcast_asid(asid) (asid) >= TLB_NR_DYN_ASIDS
+#define in_asid_transition(info) (info->mm && info->mm->context.asid_transition)
+#define mm_broadcast_asid(mm) (mm->context.broadcast_asid)
+#else
+#define is_dyn_asid(asid) true
+#define is_broadcast_asid(asid) false
+#define in_asid_transition(info) false
+#define mm_broadcast_asid(mm) 0

I don’t see a reason why those should be #define instead of inline functions. Arguably, those are better due to type-checking, etc. For instance is_dyn_asid() is missing brackets to be safe.

Quoted 34 lines
arch/x86/include/asm/tlbflush.h +7 -0
+
+inline bool needs_broadcast_asid_reload(struct mm_struct *next, u16 prev_asid)
+{
+	return false;
+}
+#endif
+
 struct tlb_context {
 	u64 ctx_id;
 	u64 tlb_gen;
arch/x86/mm/tlb.c +5 -3
index 64f1679c37e1..eb83391385ce 100644
@@ -74,13 +74,15 @@
 * use different names for each of them:
 *
 * ASID  - [0, TLB_NR_DYN_ASIDS-1]
- *         the canonical identifier for an mm
+ *         the canonical identifier for an mm, dynamically allocated on each CPU
+ *         [TLB_NR_DYN_ASIDS, MAX_ASID_AVAILABLE-1]
+ *         the canonical, global identifier for an mm, identical across all CPUs
 *
- * kPCID - [1, TLB_NR_DYN_ASIDS]
+ * kPCID - [1, MAX_ASID_AVAILABLE]
 *         the value we write into the PCID part of CR3; corresponds to the
 *         ASID+1, because PCID 0 is special.
 *
- * uPCID - [2048 + 1, 2048 + TLB_NR_DYN_ASIDS]
+ * uPCID - [2048 + 1, 2048 + MAX_ASID_AVAILABLE]
 *         for KPTI each mm has two address spaces and thus needs two
 *         PCID values, but we can still do with a single ASID denomination
 *         for each mm. Corresponds to kPCID + 2048.

arch/x86/mm/tlb.c static void choose_new_asid(struct mm_struct *next, u64 next_tlb_gen, · line 234

Quoted 8 lines
arch/x86/mm/tlb.c +5 -0
@@ -225,6 +227,18 @@ static void choose_new_asid(struct mm_struct *next, u64 next_tlb_gen,
 		return;
 	}
 
+	/*
+	 * TLB consistency for this ASID is maintained with INVLPGB;
+	 * TLB flushes happen even while the process isn't running.
+	 */
+#ifdef CONFIG_CPU_SUP_AMD

I’m pretty sure IS_ENABLED() can be used here.

arch/x86/mm/tlb.c static void choose_new_asid(struct mm_struct *next, u64 next_tlb_gen, · line 236

Quoted 2 lines
arch/x86/mm/tlb.c +2 -0
+	if (static_cpu_has(X86_FEATURE_INVLPGB) && mm_broadcast_asid(next)) {
+		*new_asid = mm_broadcast_asid(next);

Isn’t there a risk of a race changing broadcast_asid between the two reads?

Maybe use READ_ONCE() also since the value is modified asynchronously?

Quoted 11 lines
arch/x86/mm/tlb.c +5 -0
+		*need_flush = false;
+		return;
+	}
+#endif
+
 	if (this_cpu_read(cpu_tlbstate.invalidate_other))
 		clear_asid_other();
 
@@ -251,6 +265,245 @@ static void choose_new_asid(struct mm_struct *next, u64 next_tlb_gen,
 	*need_flush = true;
 }

arch/x86/mm/tlb.c static void choose_new_asid(struct mm_struct *next, u64 next_tlb_gen, · line 276

Quoted 10 lines
arch/x86/mm/tlb.c +9 -0
 
+#ifdef CONFIG_CPU_SUP_AMD
+/*
+ * Logic for AMD INVLPGB support.
+ */
+static DEFINE_RAW_SPINLOCK(broadcast_asid_lock);
+static u16 last_broadcast_asid = TLB_NR_DYN_ASIDS;
+static DECLARE_BITMAP(broadcast_asid_used, MAX_ASID_AVAILABLE) = { 0 };
+static LIST_HEAD(broadcast_asid_list);
+static int broadcast_asid_available = MAX_ASID_AVAILABLE - TLB_NR_DYN_ASIDS - 1;

Presumably some of these data structures are shared, and some are accessed frequently together. Wouldn’t it make more sense to put them inside a struct(s) and make it cacheline aligned?

Quoted 30 lines
arch/x86/mm/tlb.c +30 -0
+
+static void reset_broadcast_asid_space(void)
+{
+	mm_context_t *context;
+
+	lockdep_assert_held(&broadcast_asid_lock);
+
+	/*
+	 * Flush once when we wrap around the ASID space, so we won't need
+	 * to flush every time we allocate an ASID for boradcast flushing.
+	 */
+	invlpgb_flush_all_nonglobals();
+	tlbsync();
+
+	/*
+	 * Leave the currently used broadcast ASIDs set in the bitmap, since
+	 * those cannot be reused before the next wraparound and flush..
+	 */
+	bitmap_clear(broadcast_asid_used, 0, MAX_ASID_AVAILABLE);
+	list_for_each_entry(context, &broadcast_asid_list, broadcast_asid_list)
+		__set_bit(context->broadcast_asid, broadcast_asid_used);
+
+	last_broadcast_asid = TLB_NR_DYN_ASIDS;
+}
+
+static u16 get_broadcast_asid(void)
+{
+	lockdep_assert_held(&broadcast_asid_lock);
+
+	do {

arch/x86/mm/tlb.c static void choose_new_asid(struct mm_struct *next, u64 next_tlb_gen, · line 316

Quoted 10 lines
arch/x86/mm/tlb.c +10 -0
+		u16 start = last_broadcast_asid;
+		u16 asid = find_next_zero_bit(broadcast_asid_used, MAX_ASID_AVAILABLE, start);
+
+		if (asid >= MAX_ASID_AVAILABLE) {
+			reset_broadcast_asid_space();
+			continue;
+		}
+
+		/* Try claiming this broadcast ASID. */
+		if (!test_and_set_bit(asid, broadcast_asid_used)) {

IIUC, broadcast_asid_used is always protected with broadcast_asid_lock. So why test_and_set_bit ?

Quoted 16 lines
arch/x86/mm/tlb.c +16 -0
+			last_broadcast_asid = asid;
+			return asid;
+		}
+	} while (1);
+}
+
+/*
+ * Returns true if the mm is transitioning from a CPU-local ASID to a broadcast
+ * (INVLPGB) ASID, or the other way around.
+ */
+static bool needs_broadcast_asid_reload(struct mm_struct *next, u16 prev_asid)
+{
+	u16 broadcast_asid = mm_broadcast_asid(next);
+
+	if (broadcast_asid && prev_asid != broadcast_asid)
+		return true;

arch/x86/mm/tlb.c static void choose_new_asid(struct mm_struct *next, u64 next_tlb_gen, · line 342

Quoted 10 lines
arch/x86/mm/tlb.c +10 -0
+
+	if (!broadcast_asid && is_broadcast_asid(prev_asid))
+		return true;
+
+	return false;
+}
+
+void destroy_context_free_broadcast_asid(struct mm_struct *mm)
+{
+	if (!mm->context.broadcast_asid)

mm_broadcast_asid()?

Quoted 33 lines
arch/x86/mm/tlb.c +33 -0
+		return;
+
+	guard(raw_spinlock_irqsave)(&broadcast_asid_lock);
+	mm->context.broadcast_asid = 0;
+	list_del(&mm->context.broadcast_asid_list);
+	broadcast_asid_available++;
+}
+
+static bool mm_active_cpus_exceeds(struct mm_struct *mm, int threshold)
+{
+	int count = 0;
+	int cpu;
+
+	if (cpumask_weight(mm_cpumask(mm)) <= threshold)
+		return false;
+
+	for_each_cpu(cpu, mm_cpumask(mm)) {
+		/* Skip the CPUs that aren't really running this process. */
+		if (per_cpu(cpu_tlbstate.loaded_mm, cpu) != mm)
+			continue;
+
+		if (per_cpu(cpu_tlbstate_shared.is_lazy, cpu))
+			continue;
+
+		if (++count > threshold)
+			return true;
+	}
+	return false;
+}
+
+/*
+ * Assign a broadcast ASID to the current process, protecting against
+ * races between multiple threads in the process.

arch/x86/mm/tlb.c static void choose_new_asid(struct mm_struct *next, u64 next_tlb_gen, · line 385

Quoted 10 lines
arch/x86/mm/tlb.c +10 -0
+ */
+static void use_broadcast_asid(struct mm_struct *mm)
+{
+	guard(raw_spinlock_irqsave)(&broadcast_asid_lock);
+
+	/* This process is already using broadcast TLB invalidation. */
+	if (mm->context.broadcast_asid)
+		return;
+
+	mm->context.broadcast_asid = get_broadcast_asid();

This is read without the lock, so do you want WRITE_ONCE() here?

arch/x86/mm/tlb.c static void choose_new_asid(struct mm_struct *next, u64 next_tlb_gen, · line 386

Quoted 1 line
arch/x86/mm/tlb.c +1 -0
+	mm->context.asid_transition = true;

And what about asid_transition? Presumably also need WRITE_ONCE(). But more importantly than this theoretical compiler optimization, is there some assumed ordering with setting broadcast_asid?

Quoted 42 lines
arch/x86/mm/tlb.c +42 -0
+	list_add(&mm->context.broadcast_asid_list, &broadcast_asid_list);
+	broadcast_asid_available--;
+}
+
+/*
+ * Figure out whether to assign a broadcast (global) ASID to a process.
+ * We vary the threshold by how empty or full broadcast ASID space is.
+ * 1/4 full: >= 4 active threads
+ * 1/2 full: >= 8 active threads
+ * 3/4 full: >= 16 active threads
+ * 7/8 full: >= 32 active threads
+ * etc
+ *
+ * This way we should never exhaust the broadcast ASID space, even on very
+ * large systems, and the processes with the largest number of active
+ * threads should be able to use broadcast TLB invalidation.
+ */
+#define HALFFULL_THRESHOLD 8
+static bool meets_broadcast_asid_threshold(struct mm_struct *mm)
+{
+	int avail = broadcast_asid_available;
+	int threshold = HALFFULL_THRESHOLD;
+
+	if (!avail)
+		return false;
+
+	if (avail > MAX_ASID_AVAILABLE * 3 / 4) {
+		threshold = HALFFULL_THRESHOLD / 4;
+	} else if (avail > MAX_ASID_AVAILABLE / 2) {
+		threshold = HALFFULL_THRESHOLD / 2;
+	} else if (avail < MAX_ASID_AVAILABLE / 3) {
+		do {
+			avail *= 2;
+			threshold *= 2;
+		} while ((avail + threshold) < MAX_ASID_AVAILABLE / 2);
+	}
+
+	return mm_active_cpus_exceeds(mm, threshold);
+}
+
+static void count_tlb_flush(struct mm_struct *mm)
+{

arch/x86/mm/tlb.c static void choose_new_asid(struct mm_struct *next, u64 next_tlb_gen, · line 438

Quoted 10 lines
arch/x86/mm/tlb.c +10 -0
+	if (!static_cpu_has(X86_FEATURE_INVLPGB))
+		return;
+
+	/* Check every once in a while. */
+	if ((current->pid & 0x1f) != (jiffies & 0x1f))
+		return;
+
+	if (meets_broadcast_asid_threshold(mm))
+		use_broadcast_asid(mm);
+}

I don’t think count_tlb_flush() is a name that reflects what this function does.

arch/x86/mm/tlb.c static void choose_new_asid(struct mm_struct *next, u64 next_tlb_gen, · line 446

Quoted 8 lines
arch/x86/mm/tlb.c +8 -0
+
+static void finish_asid_transition(struct flush_tlb_info *info)
+{
+	struct mm_struct *mm = info->mm;
+	int bc_asid = mm_broadcast_asid(mm);
+	int cpu;
+
+	if (!mm->context.asid_transition)

is_asid_transition()?

Quoted 29 lines
arch/x86/mm/tlb.c +29 -0
+		return;
+
+	for_each_cpu(cpu, mm_cpumask(mm)) {
+		if (READ_ONCE(per_cpu(cpu_tlbstate.loaded_mm, cpu)) != mm)
+			continue;
+
+		/*
+		 * If at least one CPU is not using the broadcast ASID yet,
+		 * send a TLB flush IPI. The IPI should cause stragglers
+		 * to transition soon.
+		 */
+		if (per_cpu(cpu_tlbstate.loaded_mm_asid, cpu) != bc_asid) {
+			flush_tlb_multi(mm_cpumask(info->mm), info);
+			return;
+		}
+	}
+
+	/* All the CPUs running this process are using the broadcast ASID. */
+	mm->context.asid_transition = 0;
+}
+
+static void broadcast_tlb_flush(struct flush_tlb_info *info)
+{
+	bool pmd = info->stride_shift == PMD_SHIFT;
+	unsigned long maxnr = invlpgb_count_max;
+	unsigned long asid = info->mm->context.broadcast_asid;
+	unsigned long addr = info->start;
+	unsigned long nr;
+

arch/x86/mm/tlb.c static void choose_new_asid(struct mm_struct *next, u64 next_tlb_gen, · line 485

Quoted 10 lines
arch/x86/mm/tlb.c +10 -0
+	/* Flushing multiple pages at once is not supported with 1GB pages. */
+	if (info->stride_shift > PMD_SHIFT)
+		maxnr = 1;
+
+	if (info->end == TLB_FLUSH_ALL) {
+		invlpgb_flush_single_pcid(kern_pcid(asid));
+		/* Do any CPUs supporting INVLPGB need PTI? */
+		if (static_cpu_has(X86_FEATURE_PTI))
+			invlpgb_flush_single_pcid(user_pcid(asid));
+	} else do {

I couldn’t find any use of “else do” in the kernel. Might it be confusing?

Quoted 25 lines
arch/x86/mm/tlb.c +21 -0
+		/*
+		 * Calculate how many pages can be flushed at once; if the
+		 * remainder of the range is less than one page, flush one.
+		 */
+		nr = min(maxnr, (info->end - addr) >> info->stride_shift);
+		nr = max(nr, 1);
+
+		invlpgb_flush_user_nr(kern_pcid(asid), addr, nr, pmd);
+		/* Do any CPUs supporting INVLPGB need PTI? */
+		if (static_cpu_has(X86_FEATURE_PTI))
+			invlpgb_flush_user_nr(user_pcid(asid), addr, nr, pmd);
+		addr += nr << info->stride_shift;
+	} while (addr < info->end);
+
+	finish_asid_transition(info);
+
+	/* Wait for the INVLPGBs kicked off above to finish. */
+	tlbsync();
+}
+#endif /* CONFIG_CPU_SUP_AMD */
+
 /*
 * Given an ASID, flush the corresponding user ASID.  We can delay this
 * until the next time we switch to it.

arch/x86/mm/tlb.c void switch_mm_irqs_off(struct mm_struct *unused, struct mm_struct *next, · line 814

Quoted 8 lines
arch/x86/mm/tlb.c +3 -2
@@ -556,8 +809,9 @@ void switch_mm_irqs_off(struct mm_struct *unused, struct mm_struct *next,
 	 */
 	if (prev == next) {
 		/* Not actually switching mm's */
-		VM_WARN_ON(this_cpu_read(cpu_tlbstate.ctxs[prev_asid].ctx_id) !=
-			   next->context.ctx_id);
+		if (is_dyn_asid(prev_asid))
+			VM_WARN_ON(this_cpu_read(cpu_tlbstate.ctxs[prev_asid].ctx_id) !=
+				   next->context.ctx_id);

Why not to add the condition into the VM_WARN_ON and avoid the nesting?

Quoted 83 lines
arch/x86/mm/tlb.c +33 -2
 		/*
 		 * If this races with another thread that enables lam, 'new_lam'
@@ -573,6 +827,23 @@ void switch_mm_irqs_off(struct mm_struct *unused, struct mm_struct *next,
 				 !cpumask_test_cpu(cpu, mm_cpumask(next))))
 			cpumask_set_cpu(cpu, mm_cpumask(next));
 
+		/*
+		 * Check if the current mm is transitioning to a new ASID.
+		 */
+		if (needs_broadcast_asid_reload(next, prev_asid)) {
+			next_tlb_gen = atomic64_read(&next->context.tlb_gen);
+
+			choose_new_asid(next, next_tlb_gen, &new_asid, &need_flush);
+			goto reload_tlb;
+		}
+
+		/*
+		 * Broadcast TLB invalidation keeps this PCID up to date
+		 * all the time.
+		 */
+		if (is_broadcast_asid(prev_asid))
+			return;
+
 		/*
 		 * If the CPU is not in lazy TLB mode, we are just switching
 		 * from one thread in a process to another thread in the same
@@ -626,8 +897,10 @@ void switch_mm_irqs_off(struct mm_struct *unused, struct mm_struct *next,
 		barrier();
 	}
 
+reload_tlb:
 	new_lam = mm_lam_cr3_mask(next);
 	if (need_flush) {
+		VM_BUG_ON(is_broadcast_asid(new_asid));
 		this_cpu_write(cpu_tlbstate.ctxs[new_asid].ctx_id, next->context.ctx_id);
 		this_cpu_write(cpu_tlbstate.ctxs[new_asid].tlb_gen, next_tlb_gen);
 		load_new_mm_cr3(next->pgd, new_asid, new_lam, true);
@@ -746,7 +1019,7 @@ static void flush_tlb_func(void *info)
 	const struct flush_tlb_info *f = info;
 	struct mm_struct *loaded_mm = this_cpu_read(cpu_tlbstate.loaded_mm);
 	u32 loaded_mm_asid = this_cpu_read(cpu_tlbstate.loaded_mm_asid);
-	u64 local_tlb_gen = this_cpu_read(cpu_tlbstate.ctxs[loaded_mm_asid].tlb_gen);
+	u64 local_tlb_gen;
 	bool local = smp_processor_id() == f->initiating_cpu;
 	unsigned long nr_invalidate = 0;
 	u64 mm_tlb_gen;
@@ -769,6 +1042,16 @@ static void flush_tlb_func(void *info)
 	if (unlikely(loaded_mm == &init_mm))
 		return;
 
+	/* Reload the ASID if transitioning into or out of a broadcast ASID */
+	if (needs_broadcast_asid_reload(loaded_mm, loaded_mm_asid)) {
+		switch_mm_irqs_off(NULL, loaded_mm, NULL);
+		loaded_mm_asid = this_cpu_read(cpu_tlbstate.loaded_mm_asid);
+	}
+
+	/* Broadcast ASIDs are always kept up to date with INVLPGB. */
+	if (is_broadcast_asid(loaded_mm_asid))
+		return;
+
 	VM_WARN_ON(this_cpu_read(cpu_tlbstate.ctxs[loaded_mm_asid].ctx_id) !=
 		   loaded_mm->context.ctx_id);
 
@@ -786,6 +1069,8 @@ static void flush_tlb_func(void *info)
 		return;
 	}
 
+	local_tlb_gen = this_cpu_read(cpu_tlbstate.ctxs[loaded_mm_asid].tlb_gen);
+
 	if (unlikely(f->new_tlb_gen != TLB_GENERATION_INVALID &&
 		     f->new_tlb_gen <= local_tlb_gen)) {
 		/*
@@ -953,7 +1238,7 @@ STATIC_NOPV void native_flush_tlb_multi(const struct cpumask *cpumask,
 	 * up on the new contents of what used to be page tables, while
 	 * doing a speculative memory access.
 	 */
-	if (info->freed_tables)
+	if (info->freed_tables || in_asid_transition(info))
 		on_each_cpu_mask(cpumask, flush_tlb_func, (void *)info, true);
 	else
 		on_each_cpu_cond_mask(should_flush_tlb, flush_tlb_func,

arch/x86/mm/tlb.c void flush_tlb_mm_range(struct mm_struct *mm, unsigned long start, · line 1319

Quoted 9 lines
arch/x86/mm/tlb.c +3 -0
@@ -1026,14 +1311,18 @@ void flush_tlb_mm_range(struct mm_struct *mm, unsigned long start,
 				bool freed_tables)
 {
 	struct flush_tlb_info *info;
+	unsigned long threshold = tlb_single_page_flush_ceiling;
 	u64 new_tlb_gen;
 	int cpu;
 
+	if (static_cpu_has(X86_FEATURE_INVLPGB))
+		threshold *= invlpgb_count_max;

I know it’s not really impacting performance, but it is hard for me to see such calculations happening unnecessarily every time...

#51 lore raw

Re: [PATCH 09/12] x86/mm: enable broadcast TLB invalidation for multi-threaded processes

Quoted 8 lines · Nadav Amit
Quoted 8 lines
arch/x86/include/asm/tlbflush.h +0 -0
@@ -65,6 +65,23 @@ static inline void cr4_clear_bits(unsigned long
 mask)
  */
 #define TLB_NR_DYN_ASIDS	6
 
Quoted 10 lines

arch/x86/include/asm/tlbflush.h static inline void cr4_clear_bits(unsigned long mask)

Quoted 3 lines
arch/x86/include/asm/tlbflush.h +2 -0
+#ifdef CONFIG_CPU_SUP_AMD
+#define is_dyn_asid(asid) (asid) < TLB_NR_DYN_ASIDS
 

I don’t see a reason why those should be #define instead of inline functions.
Arguably, those are better due to type-checking, etc. For instance is_dyn_asid()
is missing brackets to be safe.

I've turned these all into inline functions for the next version.

Quoted 22 lines

arch/x86/include/asm/tlbflush.h

Quoted 13 lines
arch/x86/include/asm/tlbflush.h +3 -0
@@ -225,6 +227,18 @@ static void choose_new_asid(struct mm_struct
 *next, u64 next_tlb_gen,
 		return;
 	}
 
+	/*
+	 * TLB consistency for this ASID is maintained with
 INVLPGB;
+	 * TLB flushes happen even while the process isn't

running.

arch/x86/include/asm/tlbflush.h +2 -0
+	 */
+#ifdef CONFIG_CPU_SUP_AMD

I’m pretty sure IS_ENABLED() can be used here.

arch/x86/include/asm/tlbflush.h

Quoted 3 lines
arch/x86/include/asm/tlbflush.h +1 -0
+	if (static_cpu_has(X86_FEATURE_INVLPGB) &&

mm_broadcast_asid(next)) {

arch/x86/include/asm/tlbflush.h +1 -0
+		*new_asid = mm_broadcast_asid(next);

Isn’t there a risk of a race changing broadcast_asid between the two reads?

Quoted 3 lines

Maybe use READ_ONCE() also since the value is modified asynchronously?

In the current code, the broadcast_asid only ever changes from zero to a non-zero value, which continues to be used for the lifetime of the process.

It will not change from one non-zero to another non-zero value.

I have cleaned up the code a bit by having just one single read, though.

Quoted 15 lines
Quoted 15 lines
arch/x86/include/asm/tlbflush.h +8 -0
@@ -251,6 +265,245 @@ static void choose_new_asid(struct mm_struct
 *next, u64 next_tlb_gen,
 	*need_flush = true;
 }
 
+#ifdef CONFIG_CPU_SUP_AMD
+/*
+ * Logic for AMD INVLPGB support.
+ */
+static DEFINE_RAW_SPINLOCK(broadcast_asid_lock);
+static u16 last_broadcast_asid = TLB_NR_DYN_ASIDS;
+static DECLARE_BITMAP(broadcast_asid_used, MAX_ASID_AVAILABLE) = {
 0 };
+static LIST_HEAD(broadcast_asid_list);
Quoted 9 lines

arch/x86/include/asm/tlbflush.h static void choose_new_asid(struct mm_struct · line 279

Quoted 2 lines
arch/x86/include/asm/tlbflush.h +1 -0
+static int broadcast_asid_available = MAX_ASID_AVAILABLE -
 TLB_NR_DYN_ASIDS - 1;

Presumably some of these data structures are shared, and some are accessed
frequently together. Wouldn’t it make more sense to put them inside a struct(s)
and make it cacheline aligned?

Maybe?

I'm not sure any of these are particularly frequently used, at least not compared to things like TLB invalidations.

Quoted 8 lines
Quoted 4 lines
arch/x86/include/asm/tlbflush.h +2 -0
+		/* Try claiming this broadcast ASID. */
+		if (!test_and_set_bit(asid, broadcast_asid_used))

{

IIUC, broadcast_asid_used is always protected with broadcast_asid_lock.
So why test_and_set_bit  ?

Thanks for spotting that one. I cleaned that up for the next version.

Quoted 6 lines

arch/x86/include/asm/tlbflush.h

Quoted 4 lines
arch/x86/include/asm/tlbflush.h +3 -0
+void destroy_context_free_broadcast_asid(struct mm_struct *mm)
+{
+	if (!mm->context.broadcast_asid)

mm_broadcast_asid()?

I've intentionally kept this one in the same "shape" as the assignment a few lines lower. That might be cleaner than reading it one way, and then writing it another way.

Quoted 14 lines

arch/x86/include/asm/tlbflush.h

Quoted 11 lines
arch/x86/include/asm/tlbflush.h +5 -0
+static void use_broadcast_asid(struct mm_struct *mm)
+{
+	guard(raw_spinlock_irqsave)(&broadcast_asid_lock);
+
+	/* This process is already using broadcast TLB

invalidation. */

arch/x86/include/asm/tlbflush.h +4 -0
+	if (mm->context.broadcast_asid)
+		return;
+
+	mm->context.broadcast_asid = get_broadcast_asid();

This is read without the lock, so do you want WRITE_ONCE() here?

Quoted 7 lines

arch/x86/include/asm/tlbflush.h

Quoted 1 line
arch/x86/include/asm/tlbflush.h +1 -0
+	mm->context.asid_transition = true;

And what about asid_transition? Presumably also need WRITE_ONCE(). But more
importantly than this theoretical compiler optimization, is there some assumed
ordering with setting broadcast_asid?

I changed both to WRITE_ONCE.

Thinking about ordering, maybe we need to set asid_transition before setting broadcast_asid?

That way when we see a non-zero broadcast ASID, we are guaranteed to try finish_asid_transition() from broadcast_tlb_flush().

Fixed for the next version.

Quoted 9 lines

arch/x86/include/asm/tlbflush.h

Quoted 4 lines
arch/x86/include/asm/tlbflush.h +3 -0
+	if (meets_broadcast_asid_threshold(mm))
+		use_broadcast_asid(mm);
+}

I don’t think count_tlb_flush() is a name that reflects what this function
does.

Agreed. I've renamed it to the equally lame consider_broadcast_asid :)

Quoted 10 lines

arch/x86/include/asm/tlbflush.h

Quoted 8 lines
arch/x86/include/asm/tlbflush.h +8 -0
+
+static void finish_asid_transition(struct flush_tlb_info *info)
+{
+	struct mm_struct *mm = info->mm;
+	int bc_asid = mm_broadcast_asid(mm);
+	int cpu;
+
+	if (!mm->context.asid_transition)

is_asid_transition()?

I'm not convinced that for a thing we access in so few places, having a different "shape" for the read than we have for the write would make the code easier to follow.

I'm open to arguments, though ;)

Quoted 11 lines

arch/x86/include/asm/tlbflush.h

Quoted 8 lines
arch/x86/include/asm/tlbflush.h +5 -0
+	if (info->end == TLB_FLUSH_ALL) {
+		invlpgb_flush_single_pcid(kern_pcid(asid));
+		/* Do any CPUs supporting INVLPGB need PTI? */
+		if (static_cpu_has(X86_FEATURE_PTI))
+			invlpgb_flush_single_pcid(user_pcid(asid))

;

arch/x86/include/asm/tlbflush.h +1 -0
+	} else do {

I couldn’t find any use of “else do” in the kernel. Might it be confusing?

I could replace it with a goto finish_asid_transition. That's what I had there before. I'm not sure it was better, though :/

Quoted 27 lines
Quoted 26 lines
arch/x86/include/asm/tlbflush.h +2 -0
+		/*
+		 * Calculate how many pages can be flushed at

once; if the

arch/x86/include/asm/tlbflush.h +1 -0
+		 * remainder of the range is less than one page,

flush one.

arch/x86/include/asm/tlbflush.h +2 -0
+		 */
+		nr = min(maxnr, (info->end - addr) >> info-
Quoted 1 line

stride_shift);

arch/x86/include/asm/tlbflush.h +3 -0
+		nr = max(nr, 1);
+
+		invlpgb_flush_user_nr(kern_pcid(asid), addr, nr,

pmd);

arch/x86/include/asm/tlbflush.h +3 -0
+		/* Do any CPUs supporting INVLPGB need PTI? */
+		if (static_cpu_has(X86_FEATURE_PTI))
+			invlpgb_flush_user_nr(user_pcid(asid),

addr, nr, pmd);

arch/x86/include/asm/tlbflush.h +8 -0
+		addr += nr << info->stride_shift;
+	} while (addr < info->end);
+
+	finish_asid_transition(info);
+
+	/* Wait for the INVLPGBs kicked off above to finish. */
+	tlbsync();
+}
Quoted 15 lines

arch/x86/include/asm/tlbflush.h

Quoted 12 lines
arch/x86/include/asm/tlbflush.h +2 -2
@@ -556,8 +809,9 @@ void switch_mm_irqs_off(struct mm_struct
 *unused, struct mm_struct *next,
 	 */
 	if (prev == next) {
 		/* Not actually switching mm's */
-
 		VM_WARN_ON(this_cpu_read(cpu_tlbstate.ctxs[prev_asid].ctx_id) !=
-			   next->context.ctx_id);
+		if (is_dyn_asid(prev_asid))
+			VM_WARN_ON(this_cpu_read(cpu_tlbstate.ctxs

[prev_asid].ctx_id) !=

arch/x86/include/asm/tlbflush.h +1 -0
+				   next->context.ctx_id);

Why not to add the condition into the VM_WARN_ON and avoid the nesting?

Done. Thank you.

Quoted 10 lines
Quoted 10 lines
arch/x86/include/asm/tlbflush.h +1 -0
@@ -1026,14 +1311,18 @@ void flush_tlb_mm_range(struct mm_struct
 *mm, unsigned long start,
 				bool freed_tables)
 {
 	struct flush_tlb_info *info;
+	unsigned long threshold = tlb_single_page_flush_ceiling;
 	u64 new_tlb_gen;
 	int cpu;
 
Quoted 7 lines

arch/x86/include/asm/tlbflush.h void flush_tlb_mm_range(struct mm_struct · line 1320

Quoted 2 lines
arch/x86/include/asm/tlbflush.h +2 -0
+	if (static_cpu_has(X86_FEATURE_INVLPGB))
+		threshold *= invlpgb_count_max;

I know it’s not really impacting performance, but it is hard for me to see
such calculations happening unnecessarily every time...

Thanks for pointing out this code.

We should only do the multiplication if this mm is using broadcast TLB invalidation.

Fixed for the next version.

I left the multiplication for now, since that is smaller than adding code to the debugfs tlb_single_page_flush_ceiling handler to store a multiplied value in another variable.

Signature · 1 line
All Rights Reversed.
#52 lore raw

Re: [PATCH 09/12] x86/mm: enable broadcast TLB invalidation for multi-threaded processes

Quoted 1 line · Rik van Riel

Fixed for the next version.

Thanks Rik,

Admittedly, I don't feel great about my overall last review - it mostly focused on style and common BKMs.

I still don't quite get the entire logic. To name one thing that I don't understand: why do we need broadcast_asid_list and the complicated games of syncing it with broadcast_asid_used. Why wouldn't broadcast_asid_used suffice?

#53 lore raw

Re: [PATCH 09/12] x86/mm: enable broadcast TLB invalidation for multi-threaded processes

Quoted 7 lines · Nadav Amit
Quoted 1 line · Rik van Riel

Fixed for the next version.

Thanks Rik,

Quoted 7 lines

Admittedly, I don't feel great about my overall last review - it mostly focused on style and common BKMs.

I still don't quite get the entire logic. To name one thing that I don't
understand: why do we need broadcast_asid_list and the complicated games of syncing it with broadcast_asid_used. Why wouldn't broadcast_asid_used suffice?

If I uderstand correctly from Rik's patch, I think the list is needed to save the flush for only when we run out of the ASID space (wrap around). Without the list, whenever the ASID bit is cleared, you also have to flush the TLBs.

Quoted 3 lines
Signature · 3 lines
Best,
Karim
Edinburgh University
#54 lore raw

Re: [PATCH 09/12] x86/mm: enable broadcast TLB invalidation for multi-threaded processes

Quoted 20 lines · Karim Manaouil
Quoted 11 lines · Nadav Amit
Quoted 1 line · Rik van Riel

Fixed for the next version.

Thanks Rik,

Admittedly, I don't feel great about my overall last review - it mostly
focused on style and common BKMs.

Quoted 7 lines

I still don't quite get the entire logic. To name one thing that I don't
understand: why do we need broadcast_asid_list and the complicated games of
syncing it with broadcast_asid_used. Why wouldn't broadcast_asid_used
suffice?

Quoted 7 lines

If I uderstand correctly from Rik's patch, I think the list is needed to
save the flush for only when we run out of the ASID space (wrap around).
Without the list, whenever the ASID bit is cleared, you also have to flush
the TLBs.

That's exactly it.

The list will only contain processes that are active on multiple CPUs, and hit a TLB flush "at the right moment" to be assigned a broadcast ASID, which will be true for essentially every process that does a lot of TLB flushes and is long lived.

However, something like a kernel build has lots of short lived, single threaded processes, for which we should not be using broadcast TLB flushing, and which will not need to remove themselves from the list at exit time.

Signature · 1 line
All Rights Reversed.
#55 lore raw

Re: [PATCH 09/12] x86/mm: enable broadcast TLB invalidation for multi-threaded processes

Quoted 37 lines · Rik van Riel
Quoted 20 lines · Karim Manaouil
Quoted 11 lines · Nadav Amit
Quoted 1 line · Rik van Riel

Fixed for the next version.

Thanks Rik,

Admittedly, I don't feel great about my overall last review - it mostly
focused on style and common BKMs.

Quoted 7 lines

I still don't quite get the entire logic. To name one thing that I don't
understand: why do we need broadcast_asid_list and the complicated games of
syncing it with broadcast_asid_used. Why wouldn't broadcast_asid_used
suffice?

Quoted 7 lines

If I uderstand correctly from Rik's patch, I think the list is needed to
save the flush for only when we run out of the ASID space (wrap around).
Without the list, whenever the ASID bit is cleared, you also have to flush
the TLBs.

That's exactly it.

The list will only contain processes that are active on multiple CPUs, and hit a TLB flush "at the right moment" to be assigned a broadcast ASID, which will be true for essentially every process that does a lot of TLB flushes and is long lived.

Quoted 5 lines

However, something like a kernel build has lots of short lived, single threaded processes, for which we should not be using broadcast TLB flushing, and which will not need to remove themselves from the list at exit time.

Thank you Karim and Rik for the patient explanations.

But IIUC, it does seem a bit overly complicated (and when I say complicated, I mostly refer to traversing the broadcast_asid_list and its overhead).

It seems to me that basically you want to have two pieces of data that can easily be kept in bitmaps.

1. (existing) broadcast_asid_used - those that are currently "busy" and cannot be allocated.

2. (new) broadcast_asid_pending_flush - those that are logically free but were still not flushed. This would allow removing broadcast_asid_list.

Then in reset_broadcast_asid_space(), you just do something like:

bitmap_andnot(broadcast_asid_used, broadcast_asid_used,

		broadcast_asid_pending_flush, MAX_ASID_AVAILABLE);

bitmap_clear(broadcast_asid_pending_flush, MAX_ASID_AVAILABLE);

Seems to me as simpler to understand, faster, and may even allow in the future to avoid locking in fast-paths (would require different ordering and some thought). Of course, I might be missing something...

#56 lore raw

Re: [PATCH 09/12] x86/mm: enable broadcast TLB invalidation for multi-threaded processes

In reply to Rik van Riel
kernel test robot noticed the following build errors:

Hi Rik,

kernel test robot noticed the following build errors:

[auto build test ERROR on tip/x86/mm]
[also build test ERROR on tip/master]
[cannot apply to tip/x86/core tip/auto-latest linus/master v6.13-rc5 next-20241220] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Rik-van-Riel/x86-mm-make-MMU_GATHER_RCU_TABLE_FREE-unconditional/20241231-015909
base:   tip/x86/mm
patch link:    https://lore.kernel.org/r/20241230175550.4046587-10-riel%40surriel.com
patch subject: [PATCH 09/12] x86/mm: enable broadcast TLB invalidation for multi-threaded processes
config: x86_64-buildonly-randconfig-002-20241231 (https://download.01.org/0day-ci/archive/20241231/[email protected]/config)
compiler: gcc-12 (Debian 12.2.0-14) 12.2.0
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20241231/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All errors (new ones prefixed by >>):

arch/x86/mm/tlb.c: In function 'flush_tlb_mm_range':

>> arch/x86/mm/tlb.c:1342:17: error: implicit declaration of function 'broadcast_tlb_flush'; did you mean 'trace_tlb_flush'? [-Werror=implicit-function-declaration]
    1342 |                 broadcast_tlb_flush(info);
         |                 ^~~~~~~~~~~~~~~~~~~
         |                 trace_tlb_flush
>> arch/x86/mm/tlb.c:1346:17: error: implicit declaration of function 'count_tlb_flush'; did you mean 'trace_tlb_flush'? [-Werror=implicit-function-declaration]
    1346 |                 count_tlb_flush(mm);
         |                 ^~~~~~~~~~~~~~~
         |                 trace_tlb_flush

cc1: some warnings being treated as errors

vim +1342 arch/x86/mm/tlb.c

1308
1309 void flush_tlb_mm_range(struct mm_struct *mm, unsigned long start,
1310 unsigned long end, unsigned int stride_shift,
1311 bool freed_tables)
1312 {
1313 struct flush_tlb_info *info;
1314 unsigned long threshold = tlb_single_page_flush_ceiling;
1315 u64 new_tlb_gen;
1316 int cpu;
1317
1318 if (static_cpu_has(X86_FEATURE_INVLPGB))
1319 threshold *= invlpgb_count_max;
1320
1321 cpu = get_cpu();
1322
1323 /* Should we flush just the requested range? */
1324 if ((end == TLB_FLUSH_ALL) ||
1325 ((end - start) >> stride_shift) > threshold) {
1326 start = 0;
1327 end = TLB_FLUSH_ALL;
1328 }
1329
1330 /* This is also a barrier that synchronizes with switch_mm(). */
1331 new_tlb_gen = inc_mm_tlb_gen(mm);
1332
1333 info = get_flush_tlb_info(mm, start, end, stride_shift, freed_tables,
1334 new_tlb_gen);
1335
1336 /*
1337 * flush_tlb_multi() is not optimized for the common case in which only
1338 * a local TLB flush is needed. Optimize this use-case by calling
1339 * flush_tlb_func_local() directly in this case.
1340 */
1341 if (IS_ENABLED(CONFIG_CPU_SUP_AMD) && mm_broadcast_asid(mm)) {

> 1342			broadcast_tlb_flush(info);
  1343		} else if (cpumask_any_but(mm_cpumask(mm), cpu) < nr_cpu_ids) {
  1344			info->trim_cpumask = should_trim_cpumask(mm);
  1345			flush_tlb_multi(mm_cpumask(mm), info);
> 1346			count_tlb_flush(mm);

1347 } else if (mm == this_cpu_read(cpu_tlbstate.loaded_mm)) {
1348 lockdep_assert_irqs_enabled();
1349 local_irq_disable();
1350 flush_tlb_func(info);
1351 local_irq_enable();
1352 }
1353
1354 put_flush_tlb_info();
1355 put_cpu();
1356 mmu_notifier_arch_invalidate_secondary_tlbs(mm, start, end);
1357 }
1358

Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
#57 lore raw

Re: [PATCH 09/12] x86/mm: enable broadcast TLB invalidation for multi-threaded processes

In reply to Rik van Riel
Quoted 7 lines · Rik van Riel

Use broadcast TLB invalidation, using the INVPLGB instruction, on AMD EPYC 3 and newer CPUs.

In order to not exhaust PCID space, and keep TLB flushes local for single threaded processes, we only hand out broadcast ASIDs to processes active on 3 or more CPUs, and gradually increase the threshold as broadcast ASID space is depleted.

[...]

Quoted 12 lines
---
 arch/x86/include/asm/mmu.h         |   6 +
 arch/x86/include/asm/mmu_context.h |  12 ++
 arch/x86/include/asm/tlbflush.h    |  17 ++
 arch/x86/mm/tlb.c                  | 310 ++++++++++++++++++++++++++++-
 4 files changed, 336 insertions(+), 9 deletions(-)
arch/x86/include/asm/mmu.h +0 -0
index 3b496cdcb74b..a8e8dfa5a520 100644

arch/x86/include/asm/mmu.h typedef struct { · line 54

Quoted 7 lines
arch/x86/include/asm/mmu.h +4 -0
@@ -48,6 +48,12 @@ typedef struct {
        unsigned long flags;
 #endif
 
+#ifdef CONFIG_CPU_SUP_AMD
+       struct list_head broadcast_asid_list;
+       u16 broadcast_asid;
+       bool asid_transition;

Please add a comment on the semantics of the "asid_transition" field here after addressing the comments below.

Quoted 5 lines
arch/x86/include/asm/mmu.h +2 -0
+#endif
+
 #ifdef CONFIG_ADDRESS_MASKING
        /* Active LAM mode:  X86_CR3_LAM_U48 or X86_CR3_LAM_U57 or 0 (disabled) */
        unsigned long lam_cr3_mask;

[...]

arch/x86/include/asm/mmu.h

Quoted 6 lines
arch/x86/include/asm/mmu.h +6 -0
+#ifdef CONFIG_CPU_SUP_AMD
+/*
+ * Logic for AMD INVLPGB support.
+ */
+static DEFINE_RAW_SPINLOCK(broadcast_asid_lock);
+static u16 last_broadcast_asid = TLB_NR_DYN_ASIDS;

I wonder if this should be set to MAX_ASID_AVAILABLE or such to ensure that we do a flush before we start using the broadcast ASID space the first time... Or is there something else that already guarantees that all ASIDs of the TLB are flushed during kernel boot?

arch/x86/include/asm/mmu.h

Quoted 13 lines
arch/x86/include/asm/mmu.h +13 -0
+static DECLARE_BITMAP(broadcast_asid_used, MAX_ASID_AVAILABLE) = { 0 };
+static LIST_HEAD(broadcast_asid_list);
+static int broadcast_asid_available = MAX_ASID_AVAILABLE - TLB_NR_DYN_ASIDS - 1;
+
+static void reset_broadcast_asid_space(void)
+{
+       mm_context_t *context;
+
+       lockdep_assert_held(&broadcast_asid_lock);
+
+       /*
+        * Flush once when we wrap around the ASID space, so we won't need
+        * to flush every time we allocate an ASID for boradcast flushing.

nit: typoed "broadcast"

Quoted 16 lines
arch/x86/include/asm/mmu.h +16 -0
+        */
+       invlpgb_flush_all_nonglobals();
+       tlbsync();
+
+       /*
+        * Leave the currently used broadcast ASIDs set in the bitmap, since
+        * those cannot be reused before the next wraparound and flush..
+        */
+       bitmap_clear(broadcast_asid_used, 0, MAX_ASID_AVAILABLE);
+       list_for_each_entry(context, &broadcast_asid_list, broadcast_asid_list)
+               __set_bit(context->broadcast_asid, broadcast_asid_used);
+
+       last_broadcast_asid = TLB_NR_DYN_ASIDS;
+}
+
+static u16 get_broadcast_asid(void)

arch/x86/include/asm/mmu.h

Quoted 10 lines
arch/x86/include/asm/mmu.h +10 -0
+{
+       lockdep_assert_held(&broadcast_asid_lock);
+
+       do {
+               u16 start = last_broadcast_asid;
+               u16 asid = find_next_zero_bit(broadcast_asid_used, MAX_ASID_AVAILABLE, start);
+
+               if (asid >= MAX_ASID_AVAILABLE) {
+                       reset_broadcast_asid_space();
+                       continue;

Can this loop endlessly without making forward progress if we have a few thousand processes on the system that are multi-threaded (or used to be multi-threaded) and race the wrong way? meets_broadcast_asid_threshold() checks if we have free IDs remaining, but that check happens before broadcast_asid_lock is held, so we could theoretically race such that no free IDs are available, right?

Quoted 9 lines
arch/x86/include/asm/mmu.h +9 -0
+               }
+
+               /* Try claiming this broadcast ASID. */
+               if (!test_and_set_bit(asid, broadcast_asid_used)) {
+                       last_broadcast_asid = asid;
+                       return asid;
+               }
+       } while (1);
+}

[...]

Quoted 4 lines
arch/x86/include/asm/mmu.h +4 -0
+/*
+ * Assign a broadcast ASID to the current process, protecting against
+ * races between multiple threads in the process.
+ */

arch/x86/include/asm/mmu.h

Quoted 10 lines
arch/x86/include/asm/mmu.h +10 -0
+static void use_broadcast_asid(struct mm_struct *mm)
+{
+       guard(raw_spinlock_irqsave)(&broadcast_asid_lock);
+
+       /* This process is already using broadcast TLB invalidation. */
+       if (mm->context.broadcast_asid)
+               return;
+
+       mm->context.broadcast_asid = get_broadcast_asid();
+       mm->context.asid_transition = true;

This looks buggy to me: If we first set mm->context.broadcast_asid and then later set mm->context.asid_transition, then a flush_tlb_mm_range() that happens in between will observe mm_broadcast_asid() being true (meaning broadcast invalidation should be used) while mm->context.asid_transition is false (meaning broadcast invalidation alone is sufficient); but actually we haven't even started transitioning CPUs over to the new ASID yet, so I think the flush does nothing?

Maybe change how mm->context.asid_transition works such that it is immediately set on mm creation and cleared when the transition is done, so that you don't have to touch it here?

Also, please use at least WRITE_ONCE() for writes here, and add comments documenting ordering requirements.

Quoted 3 lines
arch/x86/include/asm/mmu.h +3 -0
+       list_add(&mm->context.broadcast_asid_list, &broadcast_asid_list);
+       broadcast_asid_available--;
+}

[...]

arch/x86/include/asm/mmu.h

Quoted 7 lines
arch/x86/include/asm/mmu.h +7 -0
+static void finish_asid_transition(struct flush_tlb_info *info)
+{
+       struct mm_struct *mm = info->mm;
+       int bc_asid = mm_broadcast_asid(mm);
+       int cpu;
+
+       if (!mm->context.asid_transition)

AFAIU this can be accessed concurrently - please use at least READ_ONCE(). (I think in the current version of the patch, this needs to be ordered against the preceding mm_broadcast_asid() read, but that's implicit on x86, so I guess writing a barrier here would be superfluous.)

arch/x86/include/asm/mmu.h

Quoted 5 lines
arch/x86/include/asm/mmu.h +5 -0
+               return;
+
+       for_each_cpu(cpu, mm_cpumask(mm)) {
+               if (READ_ONCE(per_cpu(cpu_tlbstate.loaded_mm, cpu)) != mm)
+                       continue;

switch_mm_irqs_off() picks an ASID and writes CR3 before writing loaded_mm:
"/* Make sure we write CR3 before loaded_mm. */"

Can we race with a concurrent switch_mm_irqs_off() on the other CPU such that the other CPU has already switched CR3 to our MM using the old ASID, but has not yet written loaded_mm, such that we skip it here? And then we'll think we finished the ASID transition, and the next time we do a flush, we'll wrongly omit the flush for that other CPU even though it's still using the old ASID?

arch/x86/include/asm/mmu.h

Quoted 7 lines
arch/x86/include/asm/mmu.h +7 -0
+
+               /*
+                * If at least one CPU is not using the broadcast ASID yet,
+                * send a TLB flush IPI. The IPI should cause stragglers
+                * to transition soon.
+                */
+               if (per_cpu(cpu_tlbstate.loaded_mm_asid, cpu) != bc_asid) {

READ_ONCE()? Also, I think this needs a comment explaining that this can race with concurrent MM switches such that we wrongly think that there's a straggler (because we're not reading the loaded_mm and the loaded_mm_asid as one atomic combination).

arch/x86/include/asm/mmu.h

Quoted 7 lines
arch/x86/include/asm/mmu.h +7 -0
+                       flush_tlb_multi(mm_cpumask(info->mm), info);
+                       return;
+               }
+       }
+
+       /* All the CPUs running this process are using the broadcast ASID. */
+       mm->context.asid_transition = 0;

WRITE_ONCE()?
Also: This is a bool, please use "false".

Quoted 6 lines
arch/x86/include/asm/mmu.h +6 -0
+}
+
+static void broadcast_tlb_flush(struct flush_tlb_info *info)
+{
+       bool pmd = info->stride_shift == PMD_SHIFT;
+       unsigned long maxnr = invlpgb_count_max;

arch/x86/include/asm/mmu.h

Quoted 10 lines
arch/x86/include/asm/mmu.h +10 -0
+       unsigned long asid = info->mm->context.broadcast_asid;
+       unsigned long addr = info->start;
+       unsigned long nr;
+
+       /* Flushing multiple pages at once is not supported with 1GB pages. */
+       if (info->stride_shift > PMD_SHIFT)
+               maxnr = 1;
+
+       if (info->end == TLB_FLUSH_ALL) {
+               invlpgb_flush_single_pcid(kern_pcid(asid));

What orders this flush with the preceding page table update? Does the instruction implicitly get ordered after preceding memory writes, or do we get that ordering from inc_mm_tlb_gen() or something like that?

Quoted 24 lines
arch/x86/include/asm/mmu.h +24 -0
+               /* Do any CPUs supporting INVLPGB need PTI? */
+               if (static_cpu_has(X86_FEATURE_PTI))
+                       invlpgb_flush_single_pcid(user_pcid(asid));
+       } else do {
+               /*
+                * Calculate how many pages can be flushed at once; if the
+                * remainder of the range is less than one page, flush one.
+                */
+               nr = min(maxnr, (info->end - addr) >> info->stride_shift);
+               nr = max(nr, 1);
+
+               invlpgb_flush_user_nr(kern_pcid(asid), addr, nr, pmd);
+               /* Do any CPUs supporting INVLPGB need PTI? */
+               if (static_cpu_has(X86_FEATURE_PTI))
+                       invlpgb_flush_user_nr(user_pcid(asid), addr, nr, pmd);
+               addr += nr << info->stride_shift;
+       } while (addr < info->end);
+
+       finish_asid_transition(info);
+
+       /* Wait for the INVLPGBs kicked off above to finish. */
+       tlbsync();
+}
+#endif /* CONFIG_CPU_SUP_AMD */

[...]

arch/x86/include/asm/mmu.h static void flush_tlb_func(void *info) · line 1053

Quoted 13 lines
arch/x86/include/asm/mmu.h +9 -0
@@ -769,6 +1042,16 @@ static void flush_tlb_func(void *info)
        if (unlikely(loaded_mm == &init_mm))
                return;
 
+       /* Reload the ASID if transitioning into or out of a broadcast ASID */
+       if (needs_broadcast_asid_reload(loaded_mm, loaded_mm_asid)) {
+               switch_mm_irqs_off(NULL, loaded_mm, NULL);
+               loaded_mm_asid = this_cpu_read(cpu_tlbstate.loaded_mm_asid);
+       }
+
+       /* Broadcast ASIDs are always kept up to date with INVLPGB. */
+       if (is_broadcast_asid(loaded_mm_asid))
+               return;

This relies on the mm_broadcast_asid() read in flush_tlb_mm_range() being ordered after the page table update, correct? And we get that required ordering from the inc_mm_tlb_gen(), which implies a full barrier? It might be nice if there were some more comments on this.

Quoted 3 lines
arch/x86/include/asm/mmu.h +0 -0
        VM_WARN_ON(this_cpu_read(cpu_tlbstate.ctxs[loaded_mm_asid].ctx_id) !=
                   loaded_mm->context.ctx_id);
 
#58 lore raw

Re: [PATCH 09/12] x86/mm: enable broadcast TLB invalidation for multi-threaded processes

Quoted 6 lines · Jann Horn
Quoted 6 lines

+++ b/arch/x86/include/asm/mmu.h

arch/x86/include/asm/mmu.h +0 -0
@@ -48,6 +48,12 @@ typedef struct {
         unsigned long flags;
  #endif
 
Quoted 7 lines

arch/x86/include/asm/mmu.h typedef struct { · line 54

Quoted 4 lines
arch/x86/include/asm/mmu.h +4 -0
+#ifdef CONFIG_CPU_SUP_AMD
+       struct list_head broadcast_asid_list;
+       u16 broadcast_asid;
+       bool asid_transition;

Please add a comment on the semantics of the "asid_transition" field here after addressing the comments below.

Will do.

Quoted 8 lines
Quoted 7 lines
arch/x86/include/asm/mmu.h +2 -0
+#endif
+
  #ifdef CONFIG_ADDRESS_MASKING
         /* Active LAM mode:  X86_CR3_LAM_U48 or X86_CR3_LAM_U57 or

0 (disabled) */
        unsigned long lam_cr3_mask;

[...]

Quoted 12 lines

arch/x86/mm/tlb.c static void choose_new_asid(struct mm_struct *next, u64 next_tlb_gen, · line 273

Quoted 6 lines
arch/x86/mm/tlb.c +6 -0
+#ifdef CONFIG_CPU_SUP_AMD
+/*
+ * Logic for AMD INVLPGB support.
+ */
+static DEFINE_RAW_SPINLOCK(broadcast_asid_lock);
+static u16 last_broadcast_asid = TLB_NR_DYN_ASIDS;

I wonder if this should be set to MAX_ASID_AVAILABLE or such to ensure
that we do a flush before we start using the broadcast ASID space the first time... Or is there something else that already guarantees that all ASIDs of the TLB are flushed during kernel boot?

That is a good idea. I don't know if the TLBs always get flushed on every kexec, for example, and having the wraparound code exercised early on every boot will be good as a self test for future proofing the code.

I'll do that in the next version.

Quoted 14 lines
Quoted 8 lines
arch/x86/mm/tlb.c +7 -0
+static u16 get_broadcast_asid(void)
+{
+       lockdep_assert_held(&broadcast_asid_lock);
+
+       do {
+               u16 start = last_broadcast_asid;
+               u16 asid = find_next_zero_bit(broadcast_asid_used,
Quoted 5 lines

MAX_ASID_AVAILABLE, start);
+
+               if (asid >= MAX_ASID_AVAILABLE) { +                       reset_broadcast_asid_space(); +                       continue;

Quoted 8 lines

Can this loop endlessly without making forward progress if we have a few thousand processes on the system that are multi-threaded (or used to be multi-threaded) and race the wrong way? meets_broadcast_asid_threshold() checks if we have free IDs remaining,
but that check happens before broadcast_asid_lock is held, so we could
theoretically race such that no free IDs are available, right?

You are right, I need to duplicate that check under the spinlock! I'll get that done in the next version.

Quoted 5 lines
Quoted 3 lines

+       mm->context.broadcast_asid = get_broadcast_asid(); +       mm->context.asid_transition = true;

This looks buggy to me:

Nadav found the same issue. I've fixed it locally already for the next version.

Quoted 4 lines

Maybe change how mm->context.asid_transition works such that it is immediately set on mm creation and cleared when the transition is done, so that you don't have to touch it here?

If we want to document the ordering, won't it be better to keep both assignments close to each other (with WRITE_ONCE), so the code stays easier to understand for future maintenance?

Quoted 2 lines

Also, please use at least WRITE_ONCE() for writes here, and add comments documenting ordering requirements.

I'll add a comment.

Quoted 14 lines

arch/x86/mm/tlb.c static void choose_new_asid(struct mm_struct *next, u64 next_tlb_gen, · line 446

Quoted 8 lines
arch/x86/mm/tlb.c +7 -0
+static void finish_asid_transition(struct flush_tlb_info *info)
+{
+       struct mm_struct *mm = info->mm;
+       int bc_asid = mm_broadcast_asid(mm);
+       int cpu;
+
+       if (!mm->context.asid_transition)

AFAIU this can be accessed concurrently - please use at least READ_ONCE(). (I think in the current version of the patch, this needs to be ordered against the preceding mm_broadcast_asid() read, but that's implicit on x86, so I guess writing a barrier here would be superfluous.)

I'll add a READ_ONCE here. Good point.

Quoted 12 lines
Quoted 7 lines

+               return;
+
+       for_each_cpu(cpu, mm_cpumask(mm)) {
+               if (READ_ONCE(per_cpu(cpu_tlbstate.loaded_mm, cpu)) != mm)
+                       continue;

switch_mm_irqs_off() picks an ASID and writes CR3 before writing loaded_mm:
"/* Make sure we write CR3 before loaded_mm. */"

Quoted 6 lines

Can we race with a concurrent switch_mm_irqs_off() on the other CPU such that the other CPU has already switched CR3 to our MM using the old ASID, but has not yet written loaded_mm, such that we skip it here? And then we'll think we finished the ASID transition, and the next time we do a flush, we'll wrongly omit the flush for that other CPU even though it's still using the old ASID?

That is a very good question.

I suppose we need to check against LOADED_MM_SWITCHING too, and possibly wait to see what mm shows up on that CPU before proceeding?

Maybe as simple as this?

for_each_cpu(cpu, mm_cpumask(mm)) {
while (READ_ONCE(per_cpu(cpu_tlbstate.loaded_mm, cpu)
== LOADED_MM_SWITCHING)

			cpu_relax();

                if (READ_ONCE(per_cpu(cpu_tlbstate.loaded_mm, cpu)) !=
mm)
                        continue;

                /*
                 * If at least one CPU is not using the broadcast ASID
yet,
                 * send a TLB flush IPI. The IPI should cause
stragglers
                 * to transition soon.
                 */
                if (per_cpu(cpu_tlbstate.loaded_mm_asid, cpu) !=
bc_asid) {
                        flush_tlb_multi(mm_cpumask(info->mm), info);
                        return;
                }
        }

Then the only change needed to switch_mm_irqs_off would be to move the LOADED_MM_SWITCHING line to before choose_new_asid, to fully close the window.

Am I overlooking anything here?

Quoted 12 lines
Quoted 11 lines

+
+               /*
+                * If at least one CPU is not using the broadcast ASID yet,
+                * send a TLB flush IPI. The IPI should cause stragglers
+                * to transition soon.
+                */
+               if (per_cpu(cpu_tlbstate.loaded_mm_asid, cpu) != bc_asid) {

Quoted 4 lines

READ_ONCE()? Also, I think this needs a comment explaining that this can race with concurrent MM switches such that we wrongly think that there's a straggler (because we're not reading the loaded_mm and the loaded_mm_asid as one atomic combination).

I'll add the READ_ONCE.

Will the race still exist if we wait on
LOADED_MM_SWITCHING as proposed above?

Quoted 11 lines
Quoted 10 lines

+                       flush_tlb_multi(mm_cpumask(info->mm), info);
+                       return;
+               }
+       }
+
+       /* All the CPUs running this process are using the broadcast ASID. */
+       mm->context.asid_transition = 0;

Quoted 2 lines

WRITE_ONCE()?
Also: This is a bool, please use "false".

Will do.

Quoted 14 lines
Quoted 14 lines
arch/x86/mm/tlb.c +11 -0
+}
+
+static void broadcast_tlb_flush(struct flush_tlb_info *info)
+{
+       bool pmd = info->stride_shift == PMD_SHIFT;
+       unsigned long maxnr = invlpgb_count_max;
+       unsigned long asid = info->mm->context.broadcast_asid;
+       unsigned long addr = info->start;
+       unsigned long nr;
+
+       /* Flushing multiple pages at once is not supported with

1GB pages. */
+       if (info->stride_shift > PMD_SHIFT)

Quoted 8 lines
Quoted 4 lines

+               maxnr = 1;
+
+       if (info->end == TLB_FLUSH_ALL) {
+               invlpgb_flush_single_pcid(kern_pcid(asid));

What orders this flush with the preceding page table update? Does the instruction implicitly get ordered after preceding memory writes, or do we get that ordering from inc_mm_tlb_gen() or something like that?

I believe inc_mm_tlb_gen() should provide the ordering.

You are right that it should be documented.

Quoted 10 lines
Quoted 5 lines

+       /* Broadcast ASIDs are always kept up to date with INVLPGB. */
+       if (is_broadcast_asid(loaded_mm_asid))
+               return;

This relies on the mm_broadcast_asid() read in flush_tlb_mm_range() being ordered after the page table update, correct? And we get that required ordering from the inc_mm_tlb_gen(), which implies a full barrier? It might be nice if there were some more comments on this.

I will add some comments, and I hope you can review those in the next series, because I'll no doubt forget to explain something important.

Thank you for the thorough review!

Signature · 1 line
All Rights Reversed.
#59 lore raw

Re: [PATCH 09/12] x86/mm: enable broadcast TLB invalidation for multi-threaded processes

Quoted 8 lines · Rik van Riel
Quoted 4 lines · Jann Horn

Maybe change how mm->context.asid_transition works such that it is immediately set on mm creation and cleared when the transition is done, so that you don't have to touch it here?

If we want to document the ordering, won't it be better to keep both assignments close to each other (with WRITE_ONCE), so the code stays easier to understand for future maintenance?

You have a point there. I was thinking that if asid_transition is set on mm creation, we don't have to think about the ordering properties as hard; but I guess you're right that it would be more clean/future-proof to do the writes together here.

Quoted 49 lines
Quoted 11 lines
Quoted 6 lines
arch/x86/mm/tlb.c +4 -0
+               return;
+
+       for_each_cpu(cpu, mm_cpumask(mm)) {
+               if (READ_ONCE(per_cpu(cpu_tlbstate.loaded_mm, cpu))

!= mm)
+ continue;

switch_mm_irqs_off() picks an ASID and writes CR3 before writing loaded_mm:
"/* Make sure we write CR3 before loaded_mm. */"

Quoted 6 lines

Can we race with a concurrent switch_mm_irqs_off() on the other CPU such that the other CPU has already switched CR3 to our MM using the old ASID, but has not yet written loaded_mm, such that we skip it here? And then we'll think we finished the ASID transition, and the next time we do a flush, we'll wrongly omit the flush for that other CPU even though it's still using the old ASID?

That is a very good question.

I suppose we need to check against LOADED_MM_SWITCHING too, and possibly wait to see what mm shows up on that CPU before proceeding?

Maybe as simple as this?

for_each_cpu(cpu, mm_cpumask(mm)) {
while (READ_ONCE(per_cpu(cpu_tlbstate.loaded_mm, cpu)
== LOADED_MM_SWITCHING)

                        cpu_relax();

                if (READ_ONCE(per_cpu(cpu_tlbstate.loaded_mm, cpu)) !=
mm)
                        continue;

                /*
                 * If at least one CPU is not using the broadcast ASID
yet,
                 * send a TLB flush IPI. The IPI should cause
stragglers
                 * to transition soon.
                 */
                if (per_cpu(cpu_tlbstate.loaded_mm_asid, cpu) !=
bc_asid) {
                        flush_tlb_multi(mm_cpumask(info->mm), info);
                        return;
                }
        }
Quoted 5 lines

Then the only change needed to switch_mm_irqs_off would be to move the LOADED_MM_SWITCHING line to before choose_new_asid, to fully close the window.

Am I overlooking anything here?

I think that might require having a full memory barrier in switch_mm_irqs_off to ensure that the write of LOADED_MM_SWITCHING can't be reordered after reads in choose_new_asid(). Which wouldn't be very nice; we probably should avoid adding heavy barriers to the task switch path...

Hmm, but I think luckily the cpumask_set_cpu() already implies a relaxed RMW atomic, which I think on X86 is actually the same as a sequentially consistent atomic, so as long as you put the LOADED_MM_SWITCHING line before that, it might do the job? Maybe with an smp_mb__after_atomic() and/or an explainer comment. (smp_mb__after_atomic() is a no-op on x86, so maybe just a comment is the right way. Documentation/memory-barriers.txt says smp_mb__after_atomic() can be used together with atomic RMW bitop functions.)

Quoted 16 lines
Quoted 11 lines
Quoted 10 lines
arch/x86/mm/tlb.c +3 -0
+
+               /*
+                * If at least one CPU is not using the broadcast

ASID yet,
+ * send a TLB flush IPI. The IPI should cause
stragglers

arch/x86/mm/tlb.c +3 -0
+                * to transition soon.
+                */
+               if (per_cpu(cpu_tlbstate.loaded_mm_asid, cpu) !=

bc_asid) {

Quoted 4 lines

READ_ONCE()? Also, I think this needs a comment explaining that this can race with concurrent MM switches such that we wrongly think that there's a straggler (because we're not reading the loaded_mm and the loaded_mm_asid as one atomic combination).

Quoted 4 lines

I'll add the READ_ONCE.

Will the race still exist if we wait on
LOADED_MM_SWITCHING as proposed above?

I think so, since between reading the loaded_mm and reading the loaded_mm_asid, the remote CPU might go through an entire task switch. Like:

1. We read the loaded_mm, and see that the remote CPU is currently running in our mm_struct.
2. The remote CPU does a task switch to another process with a different mm_struct.
3. We read the loaded_mm_asid, and see an ASID that does not match our broadcast ASID (because the loaded ASID is not for our mm_struct).

#60 lore raw

Re: [PATCH 09/12] x86/mm: enable broadcast TLB invalidation for multi-threaded processes

Quoted 16 lines · Jann Horn
Quoted 6 lines · Rik van Riel

Then the only change needed to switch_mm_irqs_off would be to move the LOADED_MM_SWITCHING line to before choose_new_asid, to fully close the window.

Am I overlooking anything here?

I think that might require having a full memory barrier in switch_mm_irqs_off to ensure that the write of LOADED_MM_SWITCHING can't be reordered after reads in choose_new_asid(). Which wouldn't be
very nice; we probably should avoid adding heavy barriers to the task switch path...

Quoted 10 lines

Hmm, but I think luckily the cpumask_set_cpu() already implies a relaxed RMW atomic, which I think on X86 is actually the same as a sequentially consistent atomic, so as long as you put the LOADED_MM_SWITCHING line before that, it might do the job? Maybe with an smp_mb__after_atomic() and/or an explainer comment. (smp_mb__after_atomic() is a no-op on x86, so maybe just a comment is the right way. Documentation/memory-barriers.txt says smp_mb__after_atomic() can be used together with atomic RMW bitop functions.)

That noop smp_mb__after_atomic() might be the way to go, since we do not actually use the mm_cpumask with INVLPGB, and we could conceivably skip updates to the bitmask for tasks using broadcast TLB flushing.

Quoted 12 lines
Quoted 6 lines
Quoted 1 line

I'll add the READ_ONCE.

Will the race still exist if we wait on LOADED_MM_SWITCHING as proposed above?

I think so, since between reading the loaded_mm and reading the loaded_mm_asid, the remote CPU might go through an entire task switch.
Like:

Quoted 8 lines

1. We read the loaded_mm, and see that the remote CPU is currently running in our mm_struct.
2. The remote CPU does a task switch to another process with a different mm_struct.
3. We read the loaded_mm_asid, and see an ASID that does not match our
broadcast ASID (because the loaded ASID is not for our mm_struct).

A false positive, where we do not clear the
asid_transition field, and will check again
in the future should be harmless, though.

The worry is false negatives, where we fail
to detect an out-of-sync CPU, yet still clear
the asid_transition field.

Signature · 1 line
All Rights Reversed.
#61 lore raw

Re: [PATCH 09/12] x86/mm: enable broadcast TLB invalidation for multi-threaded processes

In reply to Rik van Riel
Quoted 27 lines · Rik van Riel
arch/x86/mm/tlb.c +25 -0
+/*
+ * Figure out whether to assign a broadcast (global) ASID to a process.
+ * We vary the threshold by how empty or full broadcast ASID space is.
+ * 1/4 full: >= 4 active threads
+ * 1/2 full: >= 8 active threads
+ * 3/4 full: >= 16 active threads
+ * 7/8 full: >= 32 active threads
+ * etc
+ *
+ * This way we should never exhaust the broadcast ASID space, even on very
+ * large systems, and the processes with the largest number of active
+ * threads should be able to use broadcast TLB invalidation.
+ */
+#define HALFFULL_THRESHOLD 8
+static bool meets_broadcast_asid_threshold(struct mm_struct *mm)
+{
+	int avail = broadcast_asid_available;
+	int threshold = HALFFULL_THRESHOLD;
+
+	if (!avail)
+		return false;
+
+	if (avail > MAX_ASID_AVAILABLE * 3 / 4) {
+		threshold = HALFFULL_THRESHOLD / 4;
+	} else if (avail > MAX_ASID_AVAILABLE / 2) {

arch/x86/mm/tlb.c static void choose_new_asid(struct mm_struct *next, u64 next_tlb_gen, · line 425

Quoted 10 lines
arch/x86/mm/tlb.c +10 -0
+		threshold = HALFFULL_THRESHOLD / 2;
+	} else if (avail < MAX_ASID_AVAILABLE / 3) {
+		do {
+			avail *= 2;
+			threshold *= 2;
+		} while ((avail + threshold) < MAX_ASID_AVAILABLE / 2);
+	}
+
+	return mm_active_cpus_exceeds(mm, threshold);
+}

Rik,

I thought about it further and I am not sure this approach is so great. It reminds me the technique of eating chocolate forever: each day eat half of the previous day. It works in theory, but less in practice.

IOW, I mean it seems likely that early processes would get and hog all broadcast ASIDs. It seems necessary to be able to revoke broadcast ASIDs, although I understand it can be complicated.

Do you have any other resource in mind that Linux manages in a similar way (avoids revoking)?

#62 lore raw

Re: [PATCH 09/12] x86/mm: enable broadcast TLB invalidation for multi-threaded processes

Quoted 7 lines · Nadav Amit

I thought about it further and I am not sure this approach is so great.
It reminds me the technique of eating chocolate forever: each day eat half of the previous day. It works in theory, but less in practice.

The PCID space might be just large enough that we can get away with it, even on extremely large systems.

Say that we have a system with 8192 CPUs, where we are not using the PTI mitigation, giving us 4086 or so available PCIDs.

If the system runs 4k 2-thread processes, most of them get to use INVLPGB.

If the system runs 4 processes with 2k threads each, all of those large processes get to use INVLPGB.

If a few smaller processes do not get to use INVLPGB, it may not matter much, since the large (and presumably more important) processes in the system do get to use it.

Quoted 6 lines

IOW, I mean it seems likely that early processes would get and hog all
broadcast ASIDs. It seems necessary to be able to revoke broadcast ASIDs,
although I understand it can be complicated.

Revoking broadcast ASIDs works. An earlier prototype of these patches assigned broadcast ASIDs only to the top 8 TLB flushing processes on the system, and would kick tasks out of the top 8 when a more active flusher showed up.

However, given that INVLPGB seems to give only a few percent performance boost in the Phoronix tests, having some processes use INVPLGB, and some use TLB flushing might be a perfectly reasonable fallback.

https://www.phoronix.com/news/AMD-INVLPGB-Linux-Benefits

Signature · 1 line
All Rights Reversed.
#63 lore raw

Re: [PATCH 09/12] x86/mm: enable broadcast TLB invalidation for multi-threaded processes

In reply to Rik van Riel

On 12/30/24 09:53, Rik van Riel wrote:
...

arch/x86/include/asm/mmu.h typedef struct { · line 55

Quoted 5 lines
arch/x86/include/asm/mmu.h +5 -0
+#ifdef CONFIG_CPU_SUP_AMD
+	struct list_head broadcast_asid_list;
+	u16 broadcast_asid;
+	bool asid_transition;
+#endif

Could we either do:

config X86_TLB_FLUSH_BROADCAST_HW
bool
depends on CONFIG_CPU_SUP_AMD

or even

#define X86_TLB_FLUSH_BROADCAST_HW CONFIG_CPU_SUP_AMD

for the whole series please?

There are a non-trivial number of #ifdefs here and it would be nice to know what there're for, logically.

This is a completely selfish request because Intel has a similar feature and we're surely going to give this approach a try on Intel CPUs too.

Second, is there something that prevents you from defining a new MM_CONTEXT_* flag instead of a new bool? It might save bloating the context by a few words.

Quoted 17 lines

#ifdef CONFIG_ADDRESS_MASKING

arch/x86/include/asm/mmu.h +0 -0
 	/* Active LAM mode:  X86_CR3_LAM_U48 or X86_CR3_LAM_U57 or 0 (disabled) */
 	unsigned long lam_cr3_mask;
arch/x86/include/asm/mmu_context.h +2 -0
index 795fdd53bd0a..0dc446c427d2 100644
@@ -139,6 +139,8 @@ static inline void mm_reset_untag_mask(struct mm_struct *mm)
 #define enter_lazy_tlb enter_lazy_tlb
 extern void enter_lazy_tlb(struct mm_struct *mm, struct task_struct *tsk);
 
+extern void destroy_context_free_broadcast_asid(struct mm_struct *mm);
+
 /*
  * Init a new mm.  Used on mm copies, like at fork()
  * and on mm's that are brand-new, like at execve().

arch/x86/include/asm/mmu_context.h static inline int init_new_context(struct task_struct *tsk, · line 171

Quoted 9 lines
arch/x86/include/asm/mmu_context.h +6 -0
@@ -161,6 +163,13 @@ static inline int init_new_context(struct task_struct *tsk,
 		mm->context.execute_only_pkey = -1;
 	}
 #endif
+
+#ifdef CONFIG_CPU_SUP_AMD
+	INIT_LIST_HEAD(&mm->context.broadcast_asid_list);
+	mm->context.broadcast_asid = 0;
+	mm->context.asid_transition = false;
+#endif

We've been inconsistent about it, but I think I'd prefer that this had a:

	if (cpu_feature_enabled(X86_FEATURE_INVLPGB)) {
		...
	}

wrapper as opposed to CONFIG_CPU_SUP_AMD. It might save dirtying a cacheline on all the CPUs that don't care. cpu_feature_enabled() would also function the same as the #ifdef.

Quoted 21 lines
arch/x86/include/asm/mmu_context.h +3 -0
 	mm_reset_untag_mask(mm);
 	init_new_context_ldt(mm);
 	return 0;
@@ -170,6 +179,9 @@ static inline int init_new_context(struct task_struct *tsk,
 static inline void destroy_context(struct mm_struct *mm)
 {
 	destroy_context_ldt(mm);
+#ifdef CONFIG_CPU_SUP_AMD
+	destroy_context_free_broadcast_asid(mm);
+#endif
 }
 
 extern void switch_mm(struct mm_struct *prev, struct mm_struct *next,
arch/x86/include/asm/tlbflush.h +0 -0
index 20074f17fbcd..5e9956af98d1 100644
@@ -65,6 +65,23 @@ static inline void cr4_clear_bits(unsigned long mask)
  */
 #define TLB_NR_DYN_ASIDS	6
 

arch/x86/include/asm/tlbflush.h static inline void cr4_clear_bits(unsigned long mask) · line 77

Quoted 10 lines
arch/x86/include/asm/tlbflush.h +10 -0
+#ifdef CONFIG_CPU_SUP_AMD
+#define is_dyn_asid(asid) (asid) < TLB_NR_DYN_ASIDS
+#define is_broadcast_asid(asid) (asid) >= TLB_NR_DYN_ASIDS
+#define in_asid_transition(info) (info->mm && info->mm->context.asid_transition)
+#define mm_broadcast_asid(mm) (mm->context.broadcast_asid)
+#else
+#define is_dyn_asid(asid) true
+#define is_broadcast_asid(asid) false
+#define in_asid_transition(info) false
+#define mm_broadcast_asid(mm) 0

I think it was said elsewhere, but I also prefer static inlines for these instead of macros. The type checking that you get from the compiler in _both_ compile configurations is much more valuable than brevity.

...

arch/x86/include/asm/tlbflush.h static inline void cr4_clear_bits(unsigned long mask) · line 81

Quoted 4 lines
arch/x86/include/asm/tlbflush.h +4 -0
+	/*
+	 * TLB consistency for this ASID is maintained with INVLPGB;
+	 * TLB flushes happen even while the process isn't running.
+	 */

I'm not sure this comment helps much. The thing that matters here is that a broadcast ASID is asigned from a global namespace and not from a per-cpu namespace.

Quoted 12 lines
arch/x86/include/asm/tlbflush.h +8 -0
+#ifdef CONFIG_CPU_SUP_AMD
+	if (static_cpu_has(X86_FEATURE_INVLPGB) && mm_broadcast_asid(next)) {
+		*new_asid = mm_broadcast_asid(next);
+		*need_flush = false;
+		return;
+	}
+#endif
+
 	if (this_cpu_read(cpu_tlbstate.invalidate_other))
 		clear_asid_other();

arch/x86/include/asm/tlbflush.h static void choose_new_asid(struct mm_struct *next, u64 next_tlb_gen, · line 271

Quoted 7 lines
arch/x86/include/asm/tlbflush.h +4 -0
@@ -251,6 +265,245 @@ static void choose_new_asid(struct mm_struct *next, u64 next_tlb_gen,
 	*need_flush = true;
 }
 
+#ifdef CONFIG_CPU_SUP_AMD
+/*
+ * Logic for AMD INVLPGB support.
+ */

This comment is another indication that this shouldn't all be crammed under CONFIG_CPU_SUP_AMD.

arch/x86/include/asm/tlbflush.h static void choose_new_asid(struct mm_struct *next, u64 next_tlb_gen, · line 274

Quoted 3 lines
arch/x86/include/asm/tlbflush.h +3 -0
+static DEFINE_RAW_SPINLOCK(broadcast_asid_lock);
+static u16 last_broadcast_asid = TLB_NR_DYN_ASIDS;
+static DECLARE_BITMAP(broadcast_asid_used, MAX_ASID_AVAILABLE) = { 0 };

I'm debating whether this should be a bitmap for "broadcast" ASIDs alone or for all ASIDs.

arch/x86/include/asm/tlbflush.h static void choose_new_asid(struct mm_struct *next, u64 next_tlb_gen, · line 286

Quoted 12 lines
arch/x86/include/asm/tlbflush.h +12 -0
+static LIST_HEAD(broadcast_asid_list);
+static int broadcast_asid_available = MAX_ASID_AVAILABLE - TLB_NR_DYN_ASIDS - 1;
+
+static void reset_broadcast_asid_space(void)
+{
+	mm_context_t *context;
+
+	lockdep_assert_held(&broadcast_asid_lock);
+
+	/*
+	 * Flush once when we wrap around the ASID space, so we won't need
+	 * to flush every time we allocate an ASID for boradcast flushing.

^ broadcast

Quoted 4 lines
arch/x86/include/asm/tlbflush.h +4 -0
+	 */
+	invlpgb_flush_all_nonglobals();
+	tlbsync();
+

arch/x86/include/asm/tlbflush.h static void choose_new_asid(struct mm_struct *next, u64 next_tlb_gen, · line 300

Quoted 10 lines
arch/x86/include/asm/tlbflush.h +10 -0
+	/*
+	 * Leave the currently used broadcast ASIDs set in the bitmap, since
+	 * those cannot be reused before the next wraparound and flush..
+	 */
+	bitmap_clear(broadcast_asid_used, 0, MAX_ASID_AVAILABLE);
+	list_for_each_entry(context, &broadcast_asid_list, broadcast_asid_list)
+		__set_bit(context->broadcast_asid, broadcast_asid_used);
+
+	last_broadcast_asid = TLB_NR_DYN_ASIDS;
+}

'TLB_NR_DYN_ASIDS' is special here. Could it please be made more clear what it means *logically*?

Quoted 10 lines
arch/x86/include/asm/tlbflush.h +10 -0
+static u16 get_broadcast_asid(void)
+{
+	lockdep_assert_held(&broadcast_asid_lock);
+
+	do {
+		u16 start = last_broadcast_asid;
+		u16 asid = find_next_zero_bit(broadcast_asid_used, MAX_ASID_AVAILABLE, start);
+
+		if (asid >= MAX_ASID_AVAILABLE) {
+			reset_broadcast_asid_space();

arch/x86/include/asm/tlbflush.h static void choose_new_asid(struct mm_struct *next, u64 next_tlb_gen, · line 320

Quoted 10 lines
arch/x86/include/asm/tlbflush.h +10 -0
+			continue;
+		}
+
+		/* Try claiming this broadcast ASID. */
+		if (!test_and_set_bit(asid, broadcast_asid_used)) {
+			last_broadcast_asid = asid;
+			return asid;
+		}
+	} while (1);
+}

I think it was said elsewhere, but the "try" logic doesn't make a lot of sense to me when it's all protected by a global lock.

Quoted 20 lines
arch/x86/include/asm/tlbflush.h +20 -0
+/*
+ * Returns true if the mm is transitioning from a CPU-local ASID to a broadcast
+ * (INVLPGB) ASID, or the other way around.
+ */
+static bool needs_broadcast_asid_reload(struct mm_struct *next, u16 prev_asid)
+{
+	u16 broadcast_asid = mm_broadcast_asid(next);
+
+	if (broadcast_asid && prev_asid != broadcast_asid)
+		return true;
+
+	if (!broadcast_asid && is_broadcast_asid(prev_asid))
+		return true;
+
+	return false;
+}
+
+void destroy_context_free_broadcast_asid(struct mm_struct *mm)
+{
+	if (!mm->context.broadcast_asid)

arch/x86/include/asm/tlbflush.h static void choose_new_asid(struct mm_struct *next, u64 next_tlb_gen, · line 350

Quoted 10 lines
arch/x86/include/asm/tlbflush.h +10 -0
+		return;
+
+	guard(raw_spinlock_irqsave)(&broadcast_asid_lock);
+	mm->context.broadcast_asid = 0;
+	list_del(&mm->context.broadcast_asid_list);
+	broadcast_asid_available++;
+}
+
+static bool mm_active_cpus_exceeds(struct mm_struct *mm, int threshold)
+{

This function is pretty important. It's kinda missing a comment about its theory of operation.

arch/x86/include/asm/tlbflush.h static void choose_new_asid(struct mm_struct *next, u64 next_tlb_gen, · line 355

Quoted 5 lines
arch/x86/include/asm/tlbflush.h +5 -0
+	int count = 0;
+	int cpu;
+
+	if (cpumask_weight(mm_cpumask(mm)) <= threshold)
+		return false;

There's a lot of potential redundancy between this check and the one below. I assume this sequence was desinged for performance: first, do a cheap, one-stop-shopping check on mm_cpumask(). If it looks, ok, then go marauding around in a bunch of per_cpu() cachelines in a much more expensive but precise search.

Could we spell some of that out explicitly, please?

arch/x86/include/asm/tlbflush.h static void choose_new_asid(struct mm_struct *next, u64 next_tlb_gen, · line 359

Quoted 4 lines
arch/x86/include/asm/tlbflush.h +4 -0
+	for_each_cpu(cpu, mm_cpumask(mm)) {
+		/* Skip the CPUs that aren't really running this process. */
+		if (per_cpu(cpu_tlbstate.loaded_mm, cpu) != mm)
+			continue;

This is the only place I know of where 'cpu_tlbstate' is read from a non-local CPU. This is fundamentally racy as hell and needs some heavy commenting about why this raciness is OK.

#64 lore raw

Re: [PATCH 09/12] x86/mm: enable broadcast TLB invalidation for multi-threaded processes

Quoted 10 lines · Dave Hansen

On 12/30/24 09:53, Rik van Riel wrote:
...

arch/x86/include/asm/mmu.h typedef struct { · line 55

Quoted 5 lines
arch/x86/include/asm/mmu.h +5 -0
+#ifdef CONFIG_CPU_SUP_AMD
+	struct list_head broadcast_asid_list;
+	u16 broadcast_asid;
+	bool asid_transition;
+#endif

Could we either do:

Quoted 3 lines

config X86_TLB_FLUSH_BROADCAST_HW
bool
depends on CONFIG_CPU_SUP_AMD

I went with this option.

I also got rid of the list_head (thanks Nadav), and moved the remaining two items into the same word as the protection key bits.

I think I have everybody's comments addressed in my code base now. I'll post a new version soon-ish :)

Signature · 1 line
All Rights Reversed.
#65 lore raw

PATCH v3 10/12 x86,tlb: do targeted broadcast flushing from tlbbatch code

In reply to Rik van Riel

Instead of doing a system-wide TLB flush from arch_tlbbatch_flush, queue up asynchronous, targeted flushes from arch_tlbbatch_add_pending.

This also allows us to avoid adding the CPUs of processes using broadcast flushing to the batch->cpumask, and will hopefully further reduce TLB flushing from the reclaim and compaction paths.

---
 arch/x86/include/asm/tlbbatch.h |  1 +
 arch/x86/include/asm/tlbflush.h | 12 +++------
 arch/x86/mm/tlb.c               | 48 ++++++++++++++++++++++++++-------
 3 files changed, 42 insertions(+), 19 deletions(-)
arch/x86/include/asm/tlbbatch.h +1 -0
index 1ad56eb3e8a8..f9a17edf63ad 100644
@@ -10,6 +10,7 @@ struct arch_tlbflush_unmap_batch {
 	 * the PFNs being flushed..
 	 */
 	struct cpumask cpumask;
+	bool used_invlpgb;
 };
 
 #endif /* _ARCH_X86_TLBBATCH_H */
arch/x86/include/asm/tlbflush.h +3 -9
index 5e9956af98d1..17ec1b169ebd 100644
@@ -297,21 +297,15 @@ static inline u64 inc_mm_tlb_gen(struct mm_struct *mm)
 	return atomic64_inc_return(&mm->context.tlb_gen);
 }
 
-static inline void arch_tlbbatch_add_pending(struct arch_tlbflush_unmap_batch *batch,
-					     struct mm_struct *mm,
-					     unsigned long uaddr)
-{
-	inc_mm_tlb_gen(mm);
-	cpumask_or(&batch->cpumask, &batch->cpumask, mm_cpumask(mm));
-	mmu_notifier_arch_invalidate_secondary_tlbs(mm, 0, -1UL);
-}
-
 static inline void arch_flush_tlb_batched_pending(struct mm_struct *mm)
 {
 	flush_tlb_mm(mm);
 }
 
 extern void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch);
+extern void arch_tlbbatch_add_pending(struct arch_tlbflush_unmap_batch *batch,
+					     struct mm_struct *mm,
+					     unsigned long uaddr);
 
 static inline bool pte_flags_need_flush(unsigned long oldflags,
 					unsigned long newflags,
arch/x86/mm/tlb.c +38 -10

66 lines not on this page · raw message

Signature · 1 line
2.47.1
#66 lore raw

PATCH v3 11/12 x86/mm: enable AMD translation cache extensions

In reply to Rik van Riel

With AMD TCE (translation cache extensions) only the intermediate mappings that cover the address range zapped by INVLPG / INVLPGB get invalidated, rather than all intermediate mappings getting zapped at every TLB invalidation.

This can help reduce the TLB miss rate, by keeping more intermediate mappings in the cache.

From the AMD manual:

Translation Cache Extension (TCE) Bit. Bit 15, read/write. Setting this bit to 1 changes how the INVLPG, INVLPGB, and INVPCID instructions operate on TLB entries. When this bit is 0, these instructions remove the target PTE from the TLB as well as all upper-level table entries that are cached in the TLB, whether or not they are associated with the target PTE. When this bit is set, these instructions will remove the target PTE and only those upper-level entries that lead to the target PTE in the page table hierarchy, leaving unrelated upper-level entries intact.

---
 arch/x86/kernel/cpu/amd.c |  8 ++++++++
 arch/x86/mm/tlb.c         | 10 +++++++---
 2 files changed, 15 insertions(+), 3 deletions(-)
arch/x86/kernel/cpu/amd.c +8 -0
index 226b8fc64bfc..4dc42705aaca 100644
@@ -1143,6 +1143,14 @@ static void cpu_detect_tlb_amd(struct cpuinfo_x86 *c)
 
 	/* Max number of pages INVLPGB can invalidate in one shot */
 	invlpgb_count_max = (edx & 0xffff) + 1;
+
+	/* If supported, enable translation cache extensions (TCE) */
+	cpuid(0x80000001, &eax, &ebx, &ecx, &edx);
+	if (ecx & BIT(17)) {
+		u64 msr = native_read_msr(MSR_EFER);;
+		msr |= BIT(15);
+		wrmsrl(MSR_EFER, msr);
+	}
 }
 
 static const struct cpu_dev amd_cpu_dev = {
arch/x86/mm/tlb.c +7 -3
index 454a370494d3..585d0731ca9f 100644
@@ -477,7 +477,7 @@ static void broadcast_tlb_flush(struct flush_tlb_info *info)
 	if (info->stride_shift > PMD_SHIFT)
 		maxnr = 1;
 
-	if (info->end == TLB_FLUSH_ALL) {
+	if (info->end == TLB_FLUSH_ALL || info->freed_tables) {
 		invlpgb_flush_single_pcid(kern_pcid(asid));
 		/* Do any CPUs supporting INVLPGB need PTI? */
 		if (static_cpu_has(X86_FEATURE_PTI))
@@ -1110,7 +1110,7 @@ static void flush_tlb_func(void *info)
 	 *
 	 * The only question is whether to do a full or partial flush.
 	 *
-	 * We do a partial flush if requested and two extra conditions
+	 * We do a partial flush if requested and three extra conditions
 	 * are met:
 	 *
 	 * 1. f->new_tlb_gen == local_tlb_gen + 1.  We have an invariant that
@@ -1137,10 +1137,14 @@ static void flush_tlb_func(void *info)
 	 *    date.  By doing a full flush instead, we can increase
 	 *    local_tlb_gen all the way to mm_tlb_gen and we can probably
 	 *    avoid another flush in the very near future.
+	 *
+	 * 3. No page tables were freed. If page tables were freed, a full
+	 *    flush ensures intermediate translations in the TLB get flushed.
 	 */
 	if (f->end != TLB_FLUSH_ALL &&
 	    f->new_tlb_gen == local_tlb_gen + 1 &&
-	    f->new_tlb_gen == mm_tlb_gen) {
+	    f->new_tlb_gen == mm_tlb_gen &&
+	    !f->freed_tables) {
 		/* Partial flush */
 		unsigned long addr = f->start;
 
Signature · 1 line
2.47.1
#67 lore raw

Re: [PATCH 11/12] x86/mm: enable AMD translation cache extensions

Quoted 8 lines · Rik van Riel
arch/x86/kernel/cpu/amd.c +0 -0
@@ -1143,6 +1143,14 @@ static void cpu_detect_tlb_amd(struct cpuinfo_x86 *c)
 
 	/* Max number of pages INVLPGB can invalidate in one shot */

arch/x86/kernel/cpu/amd.c static void cpu_detect_tlb_amd(struct cpuinfo_x86 *c) · line 1154

Quoted 10 lines
arch/x86/kernel/cpu/amd.c +8 -0
 	invlpgb_count_max = (edx & 0xffff) + 1;
+
+	/* If supported, enable translation cache extensions (TCE) */
+	cpuid(0x80000001, &eax, &ebx, &ecx, &edx);
+	if (ecx & BIT(17)) {
+		u64 msr = native_read_msr(MSR_EFER);;
+		msr |= BIT(15);
+		wrmsrl(MSR_EFER, msr);
+	}
 }

Sorry for the gradual/delayed feedback.

Is it possible to avoid the BIT(x) and just add the bits to arch/x86/include/asm/msr-index.h like EFER_FFXSR ?

#68 lore raw

Re: [PATCH 11/12] x86/mm: enable AMD translation cache extensions

Quoted 24 lines · Nadav Amit
Quoted 20 lines · Rik van Riel
arch/x86/kernel/cpu/amd.c +8 -0
@@ -1143,6 +1143,14 @@ static void cpu_detect_tlb_amd(struct
 cpuinfo_x86 *c)
 
 	/* Max number of pages INVLPGB can invalidate in one shot
 */
 	invlpgb_count_max = (edx & 0xffff) + 1;
+
+	/* If supported, enable translation cache extensions (TCE)
 */
+	cpuid(0x80000001, &eax, &ebx, &ecx, &edx);
+	if (ecx & BIT(17)) {
+		u64 msr = native_read_msr(MSR_EFER);;
+		msr |= BIT(15);
+		wrmsrl(MSR_EFER, msr);
+	}
Quoted 1 line

}

Sorry for the gradual/delayed feedback.

Quoted 3 lines

Is it possible to avoid the BIT(x) and just add the bits to arch/x86/include/asm/msr-index.h like EFER_FFXSR ?

Of course!

I'd be happy to send that either as part of the next version of the patch series, or as a separate cleanup patch later. Whatever is more convenient for the x86 maintainers.

Signature · 1 line
All Rights Reversed.
#69 lore raw

Re: [PATCH 11/12] x86/mm: enable AMD translation cache extensions

In reply to Rik van Riel
Quoted 9 lines · Rik van Riel

With AMD TCE (translation cache extensions) only the intermediate mappings that cover the address range zapped by INVLPG / INVLPGB get invalidated, rather than all intermediate mappings getting zapped at every TLB invalidation.

This can help reduce the TLB miss rate, by keeping more intermediate mappings in the cache.

Quoted 1 line

From the AMD manual:

Quoted 8 lines

Translation Cache Extension (TCE) Bit. Bit 15, read/write. Setting this bit to 1 changes how the INVLPG, INVLPGB, and INVPCID instructions operate on TLB entries. When this bit is 0, these instructions remove the target PTE from the TLB as well as all upper-level table entries that are cached in the TLB, whether or not they are associated with the target PTE. When this bit is set, these instructions will remove the target PTE and only those upper-level entries that lead to the target PTE in the page table hierarchy, leaving unrelated upper-level entries intact.

How does this patch interact with KVM SVM guests?
In particular, will this patch cause TLB flushes performed by guest kernels to behave differently?

[...]

Quoted 23 lines
arch/x86/mm/tlb.c +2 -2
index 454a370494d3..585d0731ca9f 100644
@@ -477,7 +477,7 @@ static void broadcast_tlb_flush(struct flush_tlb_info *info)
        if (info->stride_shift > PMD_SHIFT)
                maxnr = 1;
 
-       if (info->end == TLB_FLUSH_ALL) {
+       if (info->end == TLB_FLUSH_ALL || info->freed_tables) {
                invlpgb_flush_single_pcid(kern_pcid(asid));
                /* Do any CPUs supporting INVLPGB need PTI? */
                if (static_cpu_has(X86_FEATURE_PTI))
@@ -1110,7 +1110,7 @@ static void flush_tlb_func(void *info)
         *
         * The only question is whether to do a full or partial flush.
         *
-        * We do a partial flush if requested and two extra conditions
+        * We do a partial flush if requested and three extra conditions
         * are met:
         *
         * 1. f->new_tlb_gen == local_tlb_gen + 1.  We have an invariant that

arch/x86/mm/tlb.c static void flush_tlb_func(void *info) · line 1143

Quoted 7 lines
arch/x86/mm/tlb.c +3 -0
@@ -1137,10 +1137,14 @@ static void flush_tlb_func(void *info)
         *    date.  By doing a full flush instead, we can increase
         *    local_tlb_gen all the way to mm_tlb_gen and we can probably
         *    avoid another flush in the very near future.
+        *
+        * 3. No page tables were freed. If page tables were freed, a full
+        *    flush ensures intermediate translations in the TLB get flushed.
         */

Why is this necessary - do we ever issue TLB flushes that are intended to zap upper-level entries which are not covered by the specified address range?

When, for example, free_pmd_range() gets rid of a page table, it calls pmd_free_tlb(), which sets tlb->freed_tables and does tlb_flush_pud_range(tlb, address, PAGE_SIZE).

#70 lore raw

Re: [PATCH 11/12] x86/mm: enable AMD translation cache extensions

Quoted 9 lines · Jann Horn

On Mon, Dec 30, 2024 at 6:53 PM Rik van Riel <[email protected]>

Quoted 3 lines

only those upper-level entries that lead to the target PTE in the page table hierarchy, leaving unrelated upper-level entries intact.

How does this patch interact with KVM SVM guests?
In particular, will this patch cause TLB flushes performed by guest kernels to behave differently?

That is a good question.

A Linux guest should be fine, since Linux already flushes the parts of the TLB where page tables are being freed.

I don't know whether this could potentially break some non-Linux guests, though.

Quoted 45 lines

[...]

Quoted 30 lines
arch/x86/mm/tlb.c +1 -1
index 454a370494d3..585d0731ca9f 100644
@@ -477,7 +477,7 @@ static void broadcast_tlb_flush(struct
 flush_tlb_info *info)
         if (info->stride_shift > PMD_SHIFT)
                 maxnr = 1;
 
-       if (info->end == TLB_FLUSH_ALL) {
+       if (info->end == TLB_FLUSH_ALL || info->freed_tables) {
                 invlpgb_flush_single_pcid(kern_pcid(asid));
                 /* Do any CPUs supporting INVLPGB need PTI? */

                if (static_cpu_has(X86_FEATURE_PTI))

arch/x86/mm/tlb.c +0 -0
@@ -1110,7 +1110,7 @@ static void flush_tlb_func(void *info)
          *
          * The only question is whether to do a full or partial

flush.
         *
-        * We do a partial flush if requested and two extra conditions
+        * We do a partial flush if requested and three extra conditions
         * are met:
         *
         * 1. f->new_tlb_gen == local_tlb_gen + 1.  We have an invariant that

arch/x86/mm/tlb.c +0 -0
@@ -1137,10 +1137,14 @@ static void flush_tlb_func(void *info)
          *    date.  By doing a full flush instead, we can increase
          *    local_tlb_gen all the way to mm_tlb_gen and we can
Quoted 8 lines

probably
         *    avoid another flush in the very near future. +        *
+        * 3. No page tables were freed. If page tables were freed, a full
+        *    flush ensures intermediate translations in the TLB get flushed.
         */

Why is this necessary - do we ever issue TLB flushes that are intended
to zap upper-level entries which are not covered by the specified address range?

Quoted 5 lines

When, for example, free_pmd_range() gets rid of a page table, it calls
pmd_free_tlb(), which sets tlb->freed_tables and does tlb_flush_pud_range(tlb, address, PAGE_SIZE).

I missed those calls.

It looks like this change is not needed.

Of course, the way pmd_free_tlb() operates, the partial zaps done in that code will exceed the (default 33 pages) value of tlb_single_page_flush_ceiling, and the code in flush_tlb_mm_range() will already do a full flush by default today.

I'll leave out these unnecessary changes in the next version.

Signature · 1 line
All Rights Reversed.
#71 lore raw

Re: [PATCH 11/12] x86/mm: enable AMD translation cache extensions

+KVM/SVM folks in case they know more about how enabling CPU features interacts with virtualization; original patch is at
https://lore.kernel.org/all/[email protected]/

Quoted 18 lines · Rik van Riel
Quoted 9 lines · Jann Horn

On Mon, Dec 30, 2024 at 6:53 PM Rik van Riel <[email protected]>

Quoted 3 lines

only those upper-level entries that lead to the target PTE in the page table hierarchy, leaving unrelated upper-level entries intact.

How does this patch interact with KVM SVM guests?
In particular, will this patch cause TLB flushes performed by guest kernels to behave differently?

That is a good question.

A Linux guest should be fine, since Linux already flushes the parts of the TLB where page tables are being freed.

I don't know whether this could potentially break
some non-Linux guests, though.

#72 lore raw

Re: [PATCH 11/12] x86/mm: enable AMD translation cache extensions

Quoted 4 lines · Jann Horn

+KVM/SVM folks in case they know more about how enabling CPU features interacts with virtualization; original patch is at
https://lore.kernel.org/all/[email protected]/

Quoted 10 lines
Quoted 9 lines · Rik van Riel
Quoted 8 lines · Jann Horn

On Mon, Dec 30, 2024 at 6:53 PM Rik van Riel <[email protected]>

Quoted 3 lines

only those upper-level entries that lead to the target PTE in the page table hierarchy, leaving unrelated upper-level entries intact.

How does this patch interact with KVM SVM guests?
In particular, will this patch cause TLB flushes performed by guest kernels to behave differently?

No. EFER is context switched by hardware on VMRUN and #VMEXIT, i.e. the guest runs with its own EFER, and thus will get the targeted flushes if and only if the hypervisor virtualizes EFER.TCE *and* the guest explicitly enables EFER.TCE.

Quoted 7 lines
Quoted 7 lines

That is a good question.

A Linux guest should be fine, since Linux already flushes the parts of the TLB where page tables are being freed.

I don't know whether this could potentially break some non-Linux guests, though.

#73 lore raw

Re: [PATCH 11/12] x86/mm: enable AMD translation cache extensions

In reply to Rik van Riel
Quoted 29 lines · Rik van Riel

With AMD TCE (translation cache extensions) only the intermediate mappings that cover the address range zapped by INVLPG / INVLPGB get invalidated, rather than all intermediate mappings getting zapped at every TLB invalidation.

This can help reduce the TLB miss rate, by keeping more intermediate mappings in the cache.

From the AMD manual:

Translation Cache Extension (TCE) Bit. Bit 15, read/write. Setting this bit to 1 changes how the INVLPG, INVLPGB, and INVPCID instructions operate on TLB entries. When this bit is 0, these instructions remove the target PTE from the TLB as well as all upper-level table entries that are cached in the TLB, whether or not they are associated with the target PTE. When this bit is set, these instructions will remove the target PTE and only those upper-level entries that lead to the target PTE in the page table hierarchy, leaving unrelated upper-level entries intact.

---
 arch/x86/kernel/cpu/amd.c |  8 ++++++++
 arch/x86/mm/tlb.c         | 10 +++++++---
 2 files changed, 15 insertions(+), 3 deletions(-)
arch/x86/kernel/cpu/amd.c +0 -0
index 226b8fc64bfc..4dc42705aaca 100644

arch/x86/kernel/cpu/amd.c static void cpu_detect_tlb_amd(struct cpuinfo_x86 *c) · line 1149

Quoted 7 lines
arch/x86/kernel/cpu/amd.c +4 -0
@@ -1143,6 +1143,14 @@ static void cpu_detect_tlb_amd(struct cpuinfo_x86 *c)
 
 	/* Max number of pages INVLPGB can invalidate in one shot */
 	invlpgb_count_max = (edx & 0xffff) + 1;
+
+	/* If supported, enable translation cache extensions (TCE) */
+	cpuid(0x80000001, &eax, &ebx, &ecx, &edx);
+	if (ecx & BIT(17)) {

Back to my comment from patch #4, you can put this under the cpu_feature_enabled() check and just set it.

arch/x86/kernel/cpu/amd.c static void cpu_detect_tlb_amd(struct cpuinfo_x86 *c) · line 1152

Quoted 3 lines
arch/x86/kernel/cpu/amd.c +3 -0
+		u64 msr = native_read_msr(MSR_EFER);;
+		msr |= BIT(15);
+		wrmsrl(MSR_EFER, msr);

msr_set_bit() ?

Thanks,
Tom

Quoted 42 lines
arch/x86/kernel/cpu/amd.c +1 -0
+	}
 }
 
 static const struct cpu_dev amd_cpu_dev = {
arch/x86/mm/tlb.c +7 -3
index 454a370494d3..585d0731ca9f 100644
@@ -477,7 +477,7 @@ static void broadcast_tlb_flush(struct flush_tlb_info *info)
 	if (info->stride_shift > PMD_SHIFT)
 		maxnr = 1;
 
-	if (info->end == TLB_FLUSH_ALL) {
+	if (info->end == TLB_FLUSH_ALL || info->freed_tables) {
 		invlpgb_flush_single_pcid(kern_pcid(asid));
 		/* Do any CPUs supporting INVLPGB need PTI? */
 		if (static_cpu_has(X86_FEATURE_PTI))
@@ -1110,7 +1110,7 @@ static void flush_tlb_func(void *info)
 	 *
 	 * The only question is whether to do a full or partial flush.
 	 *
-	 * We do a partial flush if requested and two extra conditions
+	 * We do a partial flush if requested and three extra conditions
 	 * are met:
 	 *
 	 * 1. f->new_tlb_gen == local_tlb_gen + 1.  We have an invariant that
@@ -1137,10 +1137,14 @@ static void flush_tlb_func(void *info)
 	 *    date.  By doing a full flush instead, we can increase
 	 *    local_tlb_gen all the way to mm_tlb_gen and we can probably
 	 *    avoid another flush in the very near future.
+	 *
+	 * 3. No page tables were freed. If page tables were freed, a full
+	 *    flush ensures intermediate translations in the TLB get flushed.
 	 */
 	if (f->end != TLB_FLUSH_ALL &&
 	    f->new_tlb_gen == local_tlb_gen + 1 &&
-	    f->new_tlb_gen == mm_tlb_gen) {
+	    f->new_tlb_gen == mm_tlb_gen &&
+	    !f->freed_tables) {
 		/* Partial flush */
 		unsigned long addr = f->start;
 
#74 lore raw

Re: [PATCH 11/12] x86/mm: enable AMD translation cache extensions

Quoted 17 lines · Tom Lendacky

arch/x86/kernel/cpu/amd.c static void cpu_detect_tlb_amd(struct cpuinfo_x86 *c) · line 1149

Quoted 13 lines

+++ b/arch/x86/kernel/cpu/amd.c

(quoted diff) +0 -0
@@ -1143,6 +1143,14 @@ static void cpu_detect_tlb_amd(struct
 cpuinfo_x86 *c)


  /* Max number of pages INVLPGB can invalidate in one shot */
  invlpgb_count_max = (edx & 0xffff) + 1;

(quoted diff) +2 -0
+
+	/* If supported, enable translation cache extensions (TCE)

*/

arch/x86/kernel/cpu/amd.c +2 -0
+	cpuid(0x80000001, &eax, &ebx, &ecx, &edx);
+	if (ecx & BIT(17)) {

Back to my comment from patch #4, you can put this under the cpu_feature_enabled() check and just set it.

Ohhh nice, so I can just add a CPUID feature bit for TCE, and then have this?

if(cpu_feature_enabled(X86_FEATURE_TCE))

    msr_set_bit(MSR_EFER, EFER_TCE);

That is much nicer.

Is this the right location for that code, or do I need to move it somewhere else to guarantee TCE gets enabled on every CPU?

Quoted 61 lines

arch/x86/kernel/cpu/amd.c static void cpu_detect_tlb_amd(struct cpuinfo_x86 *c) · line 1152

Quoted 3 lines
arch/x86/kernel/cpu/amd.c +3 -0
+		u64 msr = native_read_msr(MSR_EFER);;
+		msr |= BIT(15);
+		wrmsrl(MSR_EFER, msr);

msr_set_bit() ?

Thanks,
Tom

Quoted 51 lines
arch/x86/kernel/cpu/amd.c +1 -0
+	}

 }

 static const struct cpu_dev amd_cpu_dev = {

arch/x86/mm/tlb.c +0 -0
index 454a370494d3..585d0731ca9f 100644
@@ -477,7 +477,7 @@ static void broadcast_tlb_flush(struct
 flush_tlb_info *info)
  	if (info->stride_shift > PMD_SHIFT)
  		maxnr = 1;

arch/x86/mm/tlb.c +1 -1
-	if (info->end == TLB_FLUSH_ALL) {
+	if (info->end == TLB_FLUSH_ALL || info->freed_tables) {

  invlpgb_flush_single_pcid(kern_pcid(asid)); /* Do any CPUs supporting INVLPGB need PTI? */ if (static_cpu_has(X86_FEATURE_PTI))

arch/x86/mm/tlb.c +0 -0
@@ -1110,7 +1110,7 @@ static void flush_tlb_func(void *info)
  	 *
  	 * The only question is whether to do a full or partial

flush.
  *

arch/x86/mm/tlb.c +0 -1
-	 * We do a partial flush if requested and two extra

conditions

arch/x86/mm/tlb.c +1 -0
+	 * We do a partial flush if requested and three extra

conditions
  * are met:
  *
  * 1. f->new_tlb_gen == local_tlb_gen + 1.  We have an invariant that

arch/x86/mm/tlb.c +0 -0
@@ -1137,10 +1137,14 @@ static void flush_tlb_func(void *info)
  	 *    date.  By doing a full flush instead, we can

increase
  *    local_tlb_gen all the way to mm_tlb_gen and we can probably
  *    avoid another flush in the very near future.

arch/x86/mm/tlb.c +2 -0
+	 *
+	 * 3. No page tables were freed. If page tables were

freed, a full

arch/x86/mm/tlb.c +1 -0
+	 *    flush ensures intermediate translations in the TLB

get flushed.
  */
  if (f->end != TLB_FLUSH_ALL && f->new_tlb_gen == local_tlb_gen + 1 &&

arch/x86/mm/tlb.c +2 -1
-	    f->new_tlb_gen == mm_tlb_gen) {
+	    f->new_tlb_gen == mm_tlb_gen &&
+	    !f->freed_tables) {

  /* Partial flush */
  unsigned long addr = f->start;

Signature · 1 line
All Rights Reversed.
maintainer
#75 lore raw

Re: [PATCH 11/12] x86/mm: enable AMD translation cache extensions

Quoted 3 lines · Rik van Riel

Is this the right location for that code, or
do I need to move it somewhere else to
guarantee TCE gets enabled on every CPU?

Didn't I answer this already?

https://lore.kernel.org/r/20250102115237.GGZ3Z-BUqBauHEfB10@fat_crate.local

And you should put it in init_amd() as it runs on every CPU.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#76 lore raw

Re: [PATCH 11/12] x86/mm: enable AMD translation cache extensions

Quoted 11 lines · Borislav Petkov
Quoted 3 lines · Rik van Riel

Is this the right location for that code, or
do I need to move it somewhere else to
guarantee TCE gets enabled on every CPU?

Didn't I answer this already?

https://lore.kernel.org/r/20250102115237.GGZ3Z-BUqBauHEfB10@fat_crate.local

And you should put it in init_amd() as it runs on every CPU.

Thank you for telling me about init_amd().

I wasn't sure where in amd.c to put it.

Signature · 1 line
All Rights Reversed.
#77 lore raw

PATCH v3 12/12 x86/mm: only invalidate final translations with INVLPGB

In reply to Rik van Riel

Use the INVLPGB_FINAL_ONLY flag when invalidating mappings with INVPLGB. This way only leaf mappings get removed from the TLB, leaving intermediate translations cached.

On the (rare) occasions where we free page tables we do a full flush, ensuring intermediate translations get flushed from the TLB.

---
 arch/x86/include/asm/invlpgb.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/include/asm/invlpgb.h +1 -1
index 862775897a54..2669ebfffe81 100644
@@ -51,7 +51,7 @@ static inline void invlpgb_flush_user(unsigned long pcid,
 static inline void invlpgb_flush_user_nr(unsigned long pcid, unsigned long addr,
 					 int nr, bool pmd_stride)
 {
-	__invlpgb(0, pcid, addr, nr - 1, pmd_stride, INVLPGB_PCID | INVLPGB_VA);
+	__invlpgb(0, pcid, addr, nr - 1, pmd_stride, INVLPGB_PCID | INVLPGB_VA | INVLPGB_FINAL_ONLY);
 }
 
 /* Flush all mappings for a given ASID, not including globals. */
Signature · 1 line
2.47.1
#78 lore raw

Re: [PATCH 12/12] x86/mm: only invalidate final translations with INVLPGB

Quoted 17 lines · Rik van Riel

Use the INVLPGB_FINAL_ONLY flag when invalidating mappings with INVPLGB. This way only leaf mappings get removed from the TLB, leaving intermediate translations cached.

On the (rare) occasions where we free page tables we do a full flush, ensuring intermediate translations get flushed from the TLB.

---
 arch/x86/include/asm/invlpgb.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/include/asm/invlpgb.h +0 -0
index 862775897a54..2669ebfffe81 100644

arch/x86/include/asm/invlpgb.h static inline void invlpgb_flush_user(unsigned long pcid, · line 55

Quoted 6 lines
arch/x86/include/asm/invlpgb.h +1 -1
@@ -51,7 +51,7 @@ static inline void invlpgb_flush_user(unsigned long pcid,
 static inline void invlpgb_flush_user_nr(unsigned long pcid, unsigned long addr,
                                         int nr, bool pmd_stride)
 {
-       __invlpgb(0, pcid, addr, nr - 1, pmd_stride, INVLPGB_PCID | INVLPGB_VA);
+       __invlpgb(0, pcid, addr, nr - 1, pmd_stride, INVLPGB_PCID | INVLPGB_VA | INVLPGB_FINAL_ONLY);
 }

Please note this final-only behavior in a comment above the function and/or rename the function to make this clear.

I think this currently interacts badly with pmdp_collapse_flush(), which is used by retract_page_tables(). pmdp_collapse_flush() removes a PMD entry pointing to a page table with pmdp_huge_get_and_clear(), then calls flush_tlb_range(), which on x86 calls flush_tlb_mm_range() with the "freed_tables" parameter set to false. But that's really a preexisting bug, not something introduced by your series. I've sent a patch for that, see
<https://lore.kernel.org/r/[email protected]>.

#79 lore raw

Re: [PATCH 12/12] x86/mm: only invalidate final translations with INVLPGB

Quoted 19 lines · Jann Horn
Quoted 11 lines · Rik van Riel

+++ b/arch/x86/include/asm/invlpgb.h

(quoted diff) +1 -1
@@ -51,7 +51,7 @@ static inline void invlpgb_flush_user(unsigned
 long pcid,
  static inline void invlpgb_flush_user_nr(unsigned long pcid,
 unsigned long addr,
                                          int nr, bool pmd_stride)
  {
-       __invlpgb(0, pcid, addr, nr - 1, pmd_stride, INVLPGB_PCID |
 INVLPGB_VA);
+       __invlpgb(0, pcid, addr, nr - 1, pmd_stride, INVLPGB_PCID |
Quoted 2 lines

INVLPGB_VA | INVLPGB_FINAL_ONLY);
 }

Please note this final-only behavior in a comment above the function and/or rename the function to make this clear.

Quoted 2 lines

I think this currently interacts badly with pmdp_collapse_flush(), which is used by retract_page_tables(). pmdp_collapse_flush() removes

I've added a freed_tables argument to invlpgb_flush_user_nr_nosync

Quoted 9 lines

a PMD entry pointing to a page table with pmdp_huge_get_and_clear(), then calls flush_tlb_range(), which on x86 calls flush_tlb_mm_range() with the "freed_tables" parameter set to false. But that's really a preexisting bug, not something introduced by your series. I've sent a patch for that, see
<
https://lore.kernel.org/r/[email protected]

Quoted 1 line

.

With your change, I believe the next version of my patch series should handle this case correctly, too.

Signature · 1 line
All Rights Reversed.
#80 lore raw
In reply to Rik van Riel

A couple of high level things we need to address:

First, I'm OK calling this approach "broadcast TLB invalidation". But I don't think the ASIDs should be called "broadcast ASIDs". I'd much rather that they are called something which makes it clear that they are from a different namespace than the existing ASIDs.

After this series there will be three classes:

0: Special ASID used for the kernel, basically 1->TLB_NR_DYN_ASIDS: Allocated from private, per-cpu space. Meaningless
when compared between CPUs.

Quoted 1 line

TLB_NR_DYN_ASIDS: Allocated from shared, kernel-wide space. All CPUs

share this space and must all agree on what the values mean.

The fact that the "shared" ones are system-wide obviously allows INVLPGB to be used. The hardware feature also obviously "broadcasts" things more than plain old INVLPG did. But I don't think that makes the ASIDs "broadcast" ASIDs.

It's much more important to know that they are shared across the system instead of per-cpu than the fact that the deep implementation manages them with an instruction that is "broadcast" by hardware.

So can we call them "global", "shared" or "system" ASIDs, please?

Second, the TLB_NR_DYN_ASIDS was picked because it's roughly the number of distinct PCIDs that the CPU can keep in the TLB at once (at least on Intel). Let's say a CPU has 6 mm's in the per-cpu ASID space and another 6 in the shared/broadcast space. At that point, PCIDs might not be doing much good because the TLB can't store entries for 12 PCIDs.

Is there any comprehension in this series? Should we be indexing cpu_tlbstate.ctxs[] by a *context* number rather than by the ASID that it's running as?

Last, I'm not 100% convinced we want to do this whole thing. The will-it-scale numbers are nice. But given the complexity of this, I think we need some actual, real end users to stand up and say exactly how this is important in *PRODUCTION* to them.

#81 lore raw
Quoted 3 lines · Dave Hansen

So can we call them "global", "shared" or "system" ASIDs, please?

I have renamed them to global ASIDs.

Quoted 10 lines

Second, the TLB_NR_DYN_ASIDS was picked because it's roughly the number
of distinct PCIDs that the CPU can keep in the TLB at once (at least on
Intel). Let's say a CPU has 6 mm's in the per-cpu ASID space and another
6 in the shared/broadcast space. At that point, PCIDs might not be doing
much good because the TLB can't store entries for 12 PCIDs.

If the CPU has 12 runnable processes, we may have various other performance issues, too, like the system simply not having enough CPU power to run all the runnable tasks.

Most of the systems I have looked at seem to average between .2 and 2 runnable tasks per CPU, depending on whether the workload is CPU bound, or memory/IO bound.

Quoted 5 lines

Is there any comprehension in this series? Should we be indexing cpu_tlbstate.ctxs[] by a *context* number rather than by the ASID that
it's running as?

We only need the cpu_tlbstate.ctxs[] for the per-CPU ASID space, in order to look up what process is assigned which slot.

We do not need it for global ASID numbers, which are always the same everywhere.

Quoted 5 lines

Last, I'm not 100% convinced we want to do this whole thing. The will-it-scale numbers are nice. But given the complexity of this, I think we need some actual, real end users to stand up and say exactly how this is important in *PRODUCTION* to them.

Do any of these count? :)

https://www.phoronix.com/review/amd-invlpgb-linux
I am hoping to gather some real world numbers as well, and will work with some workload owners to get some numbers.

Signature · 1 line
All Rights Reversed.
#82 lore raw
In reply to Rik van Riel
Quoted 23 lines · Rik van Riel

Subject: [RFC PATCH 00/10] AMD broadcast TLB invalidation

Add support for broadcast TLB invalidation using AMD's INVLPGB instruction.

This allows the kernel to invalidate TLB entries on remote CPUs without needing to send IPIs, without having to wait for remote CPUs to handle those interrupts, and with less interruption to what was running on those CPUs.

Because x86 PCID space is limited, and there are some very large systems out there, broadcast TLB invalidation is only used for processes that are active on 3 or more CPUs, with the threshold being gradually increased the more the PCID space gets exhausted.

Combined with the removal of unnecessary lru_add_drain calls (see https://lkml.org/lkml/2024/12/19/1388) this results in a nice performance boost for the will-it-scale tlb_flush2_threads test on an AMD Milan system with 36 cores:

- vanilla kernel:           527k loops/second
- lru_add_drain removal:    731k loops/second
- only INVLPGB:             527k loops/second
Quoted 6 lines

- lru_add_drain + INVLPGB: 1157k loops/second

Profiling with only the INVLPGB changes showed while TLB invalidation went down from 40% of the total CPU time to only around 4% of CPU time, the contention simply moved to the LRU lock.

We briefly looked at using INVLPGB/TLBSYNC as part of the ASI work to optimize away the async freeing logic which sends TLB flush IPIs.

I have a high-level question about INVLPGB/TLBSYNC that I could not immediately find the answer to in the AMD manual. Sorry if I missed the answer or if I missed something obvious.

Do we know what the underlying mechanism for delivering the TLB flushes is? If a CPU has interrupts disabled, does it still receive the broadcast TLB flush request and handle it?

My main concern is that TLBSYNC is a single instruction that seems like it will wait for an arbitrary amount of time, and IIUC interrupts (and NMIs) will not be delivered to the running CPU until after the instruction completes execution (only at an instruction boundary).

Are there any guarantees about other CPUs handling the broadcast TLB flush in a timely manner, or an explanation of how CPUs handle the incoming requests in general?

Quoted 14 lines

Fixing both at the same time about doubles the
number of iterations per second from this case.

v3:
- Remove paravirt tlb_remove_table call (thank you Qi Zheng)
- More suggested cleanups and changelog fixes by Peter and Nadav
v2:
- Apply suggestions by Peter and Borislav (thank you!)
- Fix bug in arch_tlbbatch_flush, where we need to do both the TLBSYNC, and flush the CPUs that are in the cpumask.
- Some updates to comments and changelogs based on questions.

#83 lore raw
Quoted 11 lines · Yosry Ahmed

We briefly looked at using INVLPGB/TLBSYNC as part of the ASI work to optimize away the async freeing logic which sends TLB flush IPIs.

I have a high-level question about INVLPGB/TLBSYNC that I could not immediately find the answer to in the AMD manual. Sorry if I missed the answer or if I missed something obvious.

Do we know what the underlying mechanism for delivering the TLB flushes is? If a CPU has interrupts disabled, does it still receive the broadcast TLB flush request and handle it?

I assume TLB invalidation is probably handled similarly to how cache coherency is handled between CPUs.

However, it probably does not need to be quite as fast, since cache coherency traffic is probably 2-6 orders of magnitude more common than TLB invalidation traffic.

Quoted 10 lines

My main concern is that TLBSYNC is a single instruction that seems like it will wait for an arbitrary amount of time, and IIUC interrupts
(and NMIs) will not be delivered to the running CPU until after the instruction completes execution (only at an instruction boundary).

Are there any guarantees about other CPUs handling the broadcast TLB flush in a timely manner, or an explanation of how CPUs handle the incoming requests in general?

The performance numbers I got with the tlb_flush2_threads microbenchmark strongly suggest that INVLPGB flushes are handled by the receiving CPUs even while interrupts are disabled.

CPU time spent in flush_tlb_mm_range goes down with INVLPGB, compared with IPI based TLB flushing, even when the IPIs only go to a subset of CPUs.

I have no idea whether the invalidation is handled by something like microcode in the CPU, by the (more external?) logic that handles cache coherency, or something else entirely.

I suspect AMD wouldn't tell us exactly ;)

Signature · 1 line
All Rights Reversed.
#84 lore raw
Quoted 41 lines · Rik van Riel
Quoted 11 lines · Yosry Ahmed

We briefly looked at using INVLPGB/TLBSYNC as part of the ASI work to optimize away the async freeing logic which sends TLB flush IPIs.

I have a high-level question about INVLPGB/TLBSYNC that I could not immediately find the answer to in the AMD manual. Sorry if I missed the answer or if I missed something obvious.

Do we know what the underlying mechanism for delivering the TLB flushes is? If a CPU has interrupts disabled, does it still receive the broadcast TLB flush request and handle it?

I assume TLB invalidation is probably handled similarly to how cache coherency is handled between CPUs.

However, it probably does not need to be quite as fast, since cache coherency traffic is probably 2-6 orders of magnitude more common than TLB invalidation traffic.

Quoted 10 lines

My main concern is that TLBSYNC is a single instruction that seems like it will wait for an arbitrary amount of time, and IIUC interrupts
(and NMIs) will not be delivered to the running CPU until after the instruction completes execution (only at an instruction boundary).

Are there any guarantees about other CPUs handling the broadcast TLB flush in a timely manner, or an explanation of how CPUs handle the incoming requests in general?

The performance numbers I got with the tlb_flush2_threads microbenchmark strongly suggest that INVLPGB flushes are handled by the receiving CPUs even while interrupts are disabled.

CPU time spent in flush_tlb_mm_range goes down with INVLPGB, compared with IPI based TLB flushing, even when the IPIs only go to a subset of CPUs.

Quoted 6 lines

I have no idea whether the invalidation is handled by something like microcode in the CPU, by the (more external?) logic that handles cache coherency, or something else entirely.

I suspect AMD wouldn't tell us exactly ;)

Well, ideally they would just tell us the conditions under which CPUs respond to the broadcast TLB flush or the expectations around latency. I am also wondering if a CPU can respond to an INVLPGB while running TLBSYNC, specifically if it's possible for two CPUs to send broadcasts to one another and then execute TLBSYNC to wait for each other. Could this lead to a deadlock? I think the answer is no but we have little understanding about what's going on under the hood to know for sure (or at least I do).

Quoted 3 lines

--
All Rights Reversed.

#85 lore raw
Quoted 4 lines
Quoted 1 line

I suspect AMD wouldn't tell us exactly ;)

Well, ideally they would just tell us the conditions under which CPUs respond to the broadcast TLB flush or the expectations around latency.

Disclaimer.  I'm not at AMD; I don't know how they implement it; I'm just a random person on the internet.  But, here are a few things that might be relevant to know.

AMD's SEV-SNP whitepaper [1] states that RMP permissions "are cached in the CPU TLB and related structures" and also "When required, hardware automatically performs TLB invalidations to ensure that all processors in the system see the updated RMP entry information."

That sentence doesn't use "broadcast" or "remote", but "all processors" is a pretty clear clue.  Broadcast TLB invalidations are a building block of all the RMP-manipulation instructions.

Furthermore, to be useful in this context, they need to be ordered with memory.  Specifically, a new pagewalk mustn't start after an invalidation, yet observe the stale RMP entry.

x86 CPUs do have reasonable forward-progress guarantees, but in order to achieve forward progress, they need to e.g. guarantee that one memory access doesn't displace the TLB entry backing a different memory access from the same instruction, or you could livelock while trying to complete a single instruction.

A consequence is that you can't safely invalidate a TLB entry of an in-progress instruction (although this means only the oldest instruction in the pipeline, because everything else is speculative and potentially transient).

INVLPGB invalidations are interrupt-like from the point of view of the remote core, but can be processed

~Andrew

[1]
https://www.amd.com/content/dam/amd/en/documents/epyc-business-docs/white-papers/SEV-SNP-strengthening-vm-isolation-with-integrity-protection-and-more.pdf

#86 lore raw
In reply to Yosry Ahmed
Quoted 4 lines
Quoted 1 line

I suspect AMD wouldn't tell us exactly ;)

Well, ideally they would just tell us the conditions under which CPUs respond to the broadcast TLB flush or the expectations around latency.

[Resend, complete this time]

Disclaimer.  I'm not at AMD; I don't know how they implement it; I'm just a random person on the internet.  But, here are a few things that might be relevant to know.

AMD's SEV-SNP whitepaper [1] states that RMP permissions "are cached in the CPU TLB and related structures" and also "When required, hardware automatically performs TLB invalidations to ensure that all processors in the system see the updated RMP entry information."

That sentence doesn't use "broadcast" or "remote", but "all processors" is a pretty clear clue.  Broadcast TLB invalidations are a building block of all the RMP-manipulation instructions.

Furthermore, to be useful in this context, they need to be ordered with memory.  Specifically, a new pagewalk mustn't start after an invalidation, yet observe the stale RMP entry.

x86 CPUs do have reasonable forward-progress guarantees, but in order to achieve forward progress, they need to e.g. guarantee that one memory access doesn't displace the TLB entry backing a different memory access from the same instruction, or you could livelock while trying to complete a single instruction.

A consequence is that you can't safely invalidate a TLB entry of an in-progress instruction (although this means only the oldest instruction in the pipeline, because everything else is speculative and potentially transient).

INVLPGB invalidations are interrupt-like from the point of view of the remote core, but are microarchitectural and can be taken irrespective of the architectural Interrupt and Global Interrupt Flags.  As a consequence, they'll need wait until an instruction boundary to be processed.  While not AMD, the Intel RAR whitepaper [2] discusses the handling of RARs on the remote processor, and they share a number of constraints in common with INVLPGB.

Overall, I'd expect the INVLPGB instructions to be pretty quick in and of themselves; interestingly, they're not identified as architecturally serialising.  The broadcast is probably posted, and will be dealt with by remote processors on the subsequent instruction boundary.  TLBSYNC is the barrier to wait until the invalidations have been processed, and this will block for an unspecified length of time, probably bounded by the "longest" instruction in progress on a remote CPU.  e.g. I expect it probably will suck if you have to wait for a WBINVD instruction to complete on a remote CPU.

That said, architectural IPIs have the same conditions too, except on top of that you've got to run a whole interrupt handler.  So, with reasonable confidence, however slow TLBSYNC might be in the worst case, it's got absolutely nothing on the overhead of doing invalidations the old fashioned way.

~Andrew

[1]
https://www.amd.com/content/dam/amd/en/documents/epyc-business-docs/white-papers/SEV-SNP-strengthening-vm-isolation-with-integrity-protection-and-more.pdf
[2]
https://www.intel.com/content/dam/develop/external/us/en/documents/341431-remote-action-request-white-paper.pdf

#87 lore raw
Quoted 57 lines · Andrew Cooper
Quoted 5 lines
Quoted 2 lines

I suspect AMD wouldn't tell us exactly ;)

Well, ideally they would just tell us the conditions under which CPUs respond to the broadcast TLB flush or the expectations around latency.

[Resend, complete this time]

Disclaimer. I'm not at AMD; I don't know how they implement it; I'm just a random person on the internet. But, here are a few things that might be relevant to know.

AMD's SEV-SNP whitepaper [1] states that RMP permissions "are cached in the CPU TLB and related structures" and also "When required, hardware automatically performs TLB invalidations to ensure that all processors in the system see the updated RMP entry information."

That sentence doesn't use "broadcast" or "remote", but "all processors" is a pretty clear clue. Broadcast TLB invalidations are a building block of all the RMP-manipulation instructions.

Furthermore, to be useful in this context, they need to be ordered with memory. Specifically, a new pagewalk mustn't start after an invalidation, yet observe the stale RMP entry.

x86 CPUs do have reasonable forward-progress guarantees, but in order to achieve forward progress, they need to e.g. guarantee that one memory access doesn't displace the TLB entry backing a different memory access from the same instruction, or you could livelock while trying to complete a single instruction.

A consequence is that you can't safely invalidate a TLB entry of an in-progress instruction (although this means only the oldest instruction in the pipeline, because everything else is speculative and potentially transient).

INVLPGB invalidations are interrupt-like from the point of view of the remote core, but are microarchitectural and can be taken irrespective of the architectural Interrupt and Global Interrupt Flags. As a consequence, they'll need wait until an instruction boundary to be processed. While not AMD, the Intel RAR whitepaper [2] discusses the handling of RARs on the remote processor, and they share a number of constraints in common with INVLPGB.

Overall, I'd expect the INVLPGB instructions to be pretty quick in and of themselves; interestingly, they're not identified as architecturally serialising. The broadcast is probably posted, and will be dealt with by remote processors on the subsequent instruction boundary. TLBSYNC is the barrier to wait until the invalidations have been processed, and this will block for an unspecified length of time, probably bounded by the "longest" instruction in progress on a remote CPU. e.g. I expect it probably will suck if you have to wait for a WBINVD instruction to complete on a remote CPU.

Quoted 5 lines

That said, architectural IPIs have the same conditions too, except on top of that you've got to run a whole interrupt handler. So, with reasonable confidence, however slow TLBSYNC might be in the worst case, it's got absolutely nothing on the overhead of doing invalidations the old fashioned way.

Generally speaking, I am not arguing that TLB flush IPIs are worse than INLPGB/TLBSYNC, I think we should expect the latter to perform better in most cases.

But there is a difference here because the processor executing TLBSYNC cannot serve interrupts or NMIs while waiting for remote CPUs, because they have to be served at an instruction boundary, right? Unless TLBSYNC is an exception to that rule, or its execution is considered completed before remote CPUs respond (i.e. the CPU executes it quickly then enters into a wait doing "nothing").

There are also intriguing corner cases that are not documented. For example, you mention that it's reasonable to expect that a remote CPU does not serve TLBSYNC except at the instruction boundary. What if that CPU is executing TLBSYNC? Do we have to wait for its execution to complete? Is it possible to end up in a deadlock? This goes back to my previous point about whether TLBSYNC is a special case or when it's considered to have finished executing.

I am sure people thought about that and I am probably worried over nothing, but there's little details here so one has to speculate.

Again, sorry if I am making a fuss over nothing and it's all in my head.

Quoted 8 lines

~Andrew

[1]
https://www.amd.com/content/dam/amd/en/documents/epyc-business-docs/white-papers/SEV-SNP-strengthening-vm-isolation-with-integrity-protection-and-more.pdf
[2]
https://www.intel.com/content/dam/develop/external/us/en/documents/341431-remote-action-request-white-paper.pdf

#88 lore raw
Quoted 60 lines · Yosry Ahmed
Quoted 54 lines · Andrew Cooper
Quoted 3 lines
Quoted 1 line

I suspect AMD wouldn't tell us exactly ;)

Well, ideally they would just tell us the conditions under which CPUs respond to the broadcast TLB flush or the expectations around latency.

[Resend, complete this time]

Disclaimer. I'm not at AMD; I don't know how they implement it; I'm just a random person on the internet. But, here are a few things that might be relevant to know.

AMD's SEV-SNP whitepaper [1] states that RMP permissions "are cached in the CPU TLB and related structures" and also "When required, hardware automatically performs TLB invalidations to ensure that all processors in the system see the updated RMP entry information."

That sentence doesn't use "broadcast" or "remote", but "all processors" is a pretty clear clue. Broadcast TLB invalidations are a building block of all the RMP-manipulation instructions.

Furthermore, to be useful in this context, they need to be ordered with memory. Specifically, a new pagewalk mustn't start after an invalidation, yet observe the stale RMP entry.

x86 CPUs do have reasonable forward-progress guarantees, but in order to achieve forward progress, they need to e.g. guarantee that one memory access doesn't displace the TLB entry backing a different memory access from the same instruction, or you could livelock while trying to complete a single instruction.

A consequence is that you can't safely invalidate a TLB entry of an in-progress instruction (although this means only the oldest instruction in the pipeline, because everything else is speculative and potentially transient).

INVLPGB invalidations are interrupt-like from the point of view of the remote core, but are microarchitectural and can be taken irrespective of the architectural Interrupt and Global Interrupt Flags. As a consequence, they'll need wait until an instruction boundary to be processed. While not AMD, the Intel RAR whitepaper [2] discusses the handling of RARs on the remote processor, and they share a number of constraints in common with INVLPGB.

Overall, I'd expect the INVLPGB instructions to be pretty quick in and of themselves; interestingly, they're not identified as architecturally serialising. The broadcast is probably posted, and will be dealt with by remote processors on the subsequent instruction boundary. TLBSYNC is the barrier to wait until the invalidations have been processed, and this will block for an unspecified length of time, probably bounded by the "longest" instruction in progress on a remote CPU. e.g. I expect it probably will suck if you have to wait for a WBINVD instruction to complete on a remote CPU.

Quoted 5 lines

That said, architectural IPIs have the same conditions too, except on top of that you've got to run a whole interrupt handler. So, with reasonable confidence, however slow TLBSYNC might be in the worst case, it's got absolutely nothing on the overhead of doing invalidations the old fashioned way.

Quoted 7 lines

Generally speaking, I am not arguing that TLB flush IPIs are worse than INLPGB/TLBSYNC, I think we should expect the latter to perform better in most cases.

But there is a difference here because the processor executing TLBSYNC cannot serve interrupts or NMIs while waiting for remote CPUs, because they have to be served at an instruction boundary, right?

That's as per the architecture, yes.  NMIs do have to be served on instruction boundaries.  An NMI that becomes pending while a TLBSYNC is in progress will have to wait until the TLBSYNC completes.

(Probably.  REP string instructions and AVX scatter/gather have explicit behaviours that them them be interrupted, and to continue from where they left off when the interrupt handler returns.  Depending on how TLBSYNC is implemented, it's just possible it has this property too.)

Quoted 8 lines

Unless
TLBSYNC is an exception to that rule, or its execution is considered completed before remote CPUs respond (i.e. the CPU executes it quickly then enters into a wait doing "nothing").

There are also intriguing corner cases that are not documented. For example, you mention that it's reasonable to expect that a remote CPU does not serve TLBSYNC except at the instruction boundary.

INVLPGB needs to wait for an instruction boundary in order to be processed.

All TLBSYNC needs to do is wait until it's certain that all the prior INVLPGBs issued by this CPU have been serviced.

Quoted 5 lines

What if
that CPU is executing TLBSYNC? Do we have to wait for its execution to complete? Is it possible to end up in a deadlock? This goes back to my previous point about whether TLBSYNC is a special case or when it's considered to have finished executing.

Remember that the SEV-SNP instruction (PSMASH, PVALIDATE, RMP{ADJUST,UPDATE,QUERY,READ}) have an INVLPGB/TLBSYNC pair under the hood.  You can execute these instructions on different CPUs in parallel.

It's certainly possible AMD missed something and there's and there's a deadlock case in there.  But Google do offer SEV-SNP VMs and have the data and scale to know whether such a deadlock is happening in practice.

Quoted 5 lines

I am sure people thought about that and I am probably worried over nothing, but there's little details here so one has to speculate.

Again, sorry if I am making a fuss over nothing and it's all in my head.

It's absolutely a valid question to ask.

But x86 is full of longer delays than this.  The GIF for example can block NMIs until the hypervisor is complete with the world switch, and it's left as an exercise to software not to abuse this.  Taking an SMI will be orders of magnitude more expensive than anything discussed here.

~Andrew

#89 lore raw
Quoted 70 lines · Andrew Cooper
Quoted 60 lines · Yosry Ahmed
Quoted 54 lines · Andrew Cooper
Quoted 3 lines
Quoted 1 line

I suspect AMD wouldn't tell us exactly ;)

Well, ideally they would just tell us the conditions under which CPUs respond to the broadcast TLB flush or the expectations around latency.

[Resend, complete this time]

Disclaimer. I'm not at AMD; I don't know how they implement it; I'm just a random person on the internet. But, here are a few things that might be relevant to know.

AMD's SEV-SNP whitepaper [1] states that RMP permissions "are cached in the CPU TLB and related structures" and also "When required, hardware automatically performs TLB invalidations to ensure that all processors in the system see the updated RMP entry information."

That sentence doesn't use "broadcast" or "remote", but "all processors" is a pretty clear clue. Broadcast TLB invalidations are a building block of all the RMP-manipulation instructions.

Furthermore, to be useful in this context, they need to be ordered with memory. Specifically, a new pagewalk mustn't start after an invalidation, yet observe the stale RMP entry.

x86 CPUs do have reasonable forward-progress guarantees, but in order to achieve forward progress, they need to e.g. guarantee that one memory access doesn't displace the TLB entry backing a different memory access from the same instruction, or you could livelock while trying to complete a single instruction.

A consequence is that you can't safely invalidate a TLB entry of an in-progress instruction (although this means only the oldest instruction in the pipeline, because everything else is speculative and potentially transient).

INVLPGB invalidations are interrupt-like from the point of view of the remote core, but are microarchitectural and can be taken irrespective of the architectural Interrupt and Global Interrupt Flags. As a consequence, they'll need wait until an instruction boundary to be processed. While not AMD, the Intel RAR whitepaper [2] discusses the handling of RARs on the remote processor, and they share a number of constraints in common with INVLPGB.

Overall, I'd expect the INVLPGB instructions to be pretty quick in and of themselves; interestingly, they're not identified as architecturally serialising. The broadcast is probably posted, and will be dealt with by remote processors on the subsequent instruction boundary. TLBSYNC is the barrier to wait until the invalidations have been processed, and this will block for an unspecified length of time, probably bounded by the "longest" instruction in progress on a remote CPU. e.g. I expect it probably will suck if you have to wait for a WBINVD instruction to complete on a remote CPU.

Quoted 5 lines

That said, architectural IPIs have the same conditions too, except on top of that you've got to run a whole interrupt handler. So, with reasonable confidence, however slow TLBSYNC might be in the worst case, it's got absolutely nothing on the overhead of doing invalidations the old fashioned way.

Quoted 7 lines

Generally speaking, I am not arguing that TLB flush IPIs are worse than INLPGB/TLBSYNC, I think we should expect the latter to perform better in most cases.

But there is a difference here because the processor executing TLBSYNC cannot serve interrupts or NMIs while waiting for remote CPUs, because they have to be served at an instruction boundary, right?

Quoted 8 lines

That's as per the architecture, yes. NMIs do have to be served on instruction boundaries. An NMI that becomes pending while a TLBSYNC is in progress will have to wait until the TLBSYNC completes.

(Probably. REP string instructions and AVX scatter/gather have explicit behaviours that them them be interrupted, and to continue from where they left off when the interrupt handler returns. Depending on how TLBSYNC is implemented, it's just possible it has this property too.)

That would be great actually, if that's the case all my concerns go away.

Quoted 21 lines
Quoted 9 lines

Unless
TLBSYNC is an exception to that rule, or its execution is considered completed before remote CPUs respond (i.e. the CPU executes it quickly then enters into a wait doing "nothing").

There are also intriguing corner cases that are not documented. For example, you mention that it's reasonable to expect that a remote CPU does not serve TLBSYNC except at the instruction boundary.

INVLPGB needs to wait for an instruction boundary in order to be processed.

All TLBSYNC needs to do is wait until it's certain that all the prior INVLPGBs issued by this CPU have been serviced.

Quoted 5 lines

What if
that CPU is executing TLBSYNC? Do we have to wait for its execution to complete? Is it possible to end up in a deadlock? This goes back to my previous point about whether TLBSYNC is a special case or when it's considered to have finished executing.

Quoted 7 lines

Remember that the SEV-SNP instruction (PSMASH, PVALIDATE, RMP{ADJUST,UPDATE,QUERY,READ}) have an INVLPGB/TLBSYNC pair under the hood. You can execute these instructions on different CPUs in parallel.

It's certainly possible AMD missed something and there's and there's a deadlock case in there. But Google do offer SEV-SNP VMs and have the data and scale to know whether such a deadlock is happening in practice.

I am not familiar with SEV-SNP so excuse my ignorance. I am also pretty sure that the percentage of SEV-SNP workloads is very low compared to the workloads that would start using INVLPGB/TLBSYNC after this series. So if there's a dormant bug or a rare scenario where the TLBSYNC latency is massive, it may very well be newly uncovered now.

Quoted 5 lines
Quoted 5 lines

I am sure people thought about that and I am probably worried over nothing, but there's little details here so one has to speculate.

Quoted 8 lines
Quoted 1 line

Again, sorry if I am making a fuss over nothing and it's all in my head.

It's absolutely a valid question to ask.

But x86 is full of longer delays than this. The GIF for example can block NMIs until the hypervisor is complete with the world switch, and it's left as an exercise to software not to abuse this. Taking an SMI will be orders of magnitude more expensive than anything discussed here.

Right. What is happening here just seems like something that happens more frequently and therefore is more likely to run into cases with absurd delays.

It would be great if someone from AMD could shed some light on what is to be reasonably expected from TLBSYNC here.

Anyway, thanks a lot for all your (very informative) responses :)