Skip to contentSkip to conversation

AMD broadcast TLB invalidation

RFC PATCH Superseded Rik van Riel 6 participants 21 messages Dec 23, 2024 – Jan 10, 2025

Subsystems (4)

v2 of 14 Superseded Latest is v14 →

lore
Share & tools
lore
#0 lore raw

v2 cover AMD broadcast TLB invalidation

Add support for broadcast TLB invalidation using AMD's INVLPGB instruction.

This allows the kernel to invalidate TLB entries on remote CPUs without needing to send IPIs, without having to wait for remote CPUs to handle those interrupts, and with less interruption to what was running on those CPUs.

Because x86 PCID space is limited, and there are some very large systems out there, broadcast TLB invalidation is only used for processes that are active on 3 or more CPUs, with the threshold being gradually increased the more the PCID space gets exhausted.

Combined with the removal of unnecessary lru_add_drain calls (see https://lkml.org/lkml/2024/12/19/1388) this results in a nice performance boost for the will-it-scale tlb_flush2_threads test on an AMD Milan system with 36 cores:

- vanilla kernel:           527k loops/second
- lru_add_drain removal:    731k loops/second
- only INVLPGB:             527k loops/second

- lru_add_drain + INVLPGB: 1157k loops/second

Profiling with only the INVLPGB changes showed while TLB invalidation went down from 40% of the total CPU time to only around 4% of CPU time, the contention simply moved to the LRU lock.

Fixing both at the same time about doubles the
number of iterations per second from this case.

v2:
- Apply suggestions by Peter and Borislav (thank you!)
- Fix bug in arch_tlbbatch_flush, where we need to do both the TLBSYNC, and flush the CPUs that are in the cpumask.
- Some updates to comments and changelogs based on questions.

#1 lore raw

PATCH v2 1/11 x86/mm: make MMU_GATHER_RCU_TABLE_FREE unconditional

Currently x86 usse CONFIG_MMU_GATHER_TABLE_FREE when using paravirt, and not when running on bare metal.

There is no real good reason to do things differently for each setup. Make them all the same.

---
 arch/x86/Kconfig           | 2 +-
 arch/x86/kernel/paravirt.c | 7 +------
 2 files changed, 2 insertions(+), 7 deletions(-)
arch/x86/Kconfig +1 -1
index 9d7bd0ae48c4..e8743f8c9fd0 100644
@@ -274,7 +274,7 @@ config X86
 	select HAVE_PCI
 	select HAVE_PERF_REGS
 	select HAVE_PERF_USER_STACK_DUMP
-	select MMU_GATHER_RCU_TABLE_FREE	if PARAVIRT
+	select MMU_GATHER_RCU_TABLE_FREE
 	select MMU_GATHER_MERGE_VMAS
 	select HAVE_POSIX_CPU_TIMERS_TASK_WORK
 	select HAVE_REGS_AND_STACK_ACCESS_API
arch/x86/kernel/paravirt.c +1 -6
index fec381533555..2b78a6b466ed 100644
@@ -59,11 +59,6 @@ void __init native_pv_lock_init(void)
 		static_branch_enable(&virt_spin_lock_key);
 }
 
-static void native_tlb_remove_table(struct mmu_gather *tlb, void *table)
-{
-	tlb_remove_page(tlb, table);
-}
-
 struct static_key paravirt_steal_enabled;
 struct static_key paravirt_steal_rq_enabled;
 
@@ -191,7 +186,7 @@ struct paravirt_patch_template pv_ops = {
 	.mmu.flush_tlb_kernel	= native_flush_tlb_global,
 	.mmu.flush_tlb_one_user	= native_flush_tlb_one_user,
 	.mmu.flush_tlb_multi	= native_flush_tlb_multi,
-	.mmu.tlb_remove_table	= native_tlb_remove_table,
+	.mmu.tlb_remove_table	= tlb_remove_table,
 
 	.mmu.exit_mmap		= paravirt_nop,
 	.mmu.notify_page_enc_status_changed	= paravirt_nop,
Signature · 1 line
2.47.1
#2 lore raw

Re: [PATCH 01/11] x86/mm: make MMU_GATHER_RCU_TABLE_FREE unconditional

Very happy to see this change!

Quoted 1 line · Rik van Riel

Currently x86 usse CONFIG_MMU_GATHER_TABLE_FREE when using

                ^
                use
Quoted 42 lines

paravirt, and not when running on bare metal.

There is no real good reason to do things differently for each setup. Make them all the same.

---
  arch/x86/Kconfig           | 2 +-
  arch/x86/kernel/paravirt.c | 7 +------
  2 files changed, 2 insertions(+), 7 deletions(-)
arch/x86/Kconfig +1 -1
index 9d7bd0ae48c4..e8743f8c9fd0 100644
@@ -274,7 +274,7 @@ config X86
  	select HAVE_PCI
  	select HAVE_PERF_REGS
  	select HAVE_PERF_USER_STACK_DUMP
-	select MMU_GATHER_RCU_TABLE_FREE	if PARAVIRT
+	select MMU_GATHER_RCU_TABLE_FREE
  	select MMU_GATHER_MERGE_VMAS
  	select HAVE_POSIX_CPU_TIMERS_TASK_WORK
  	select HAVE_REGS_AND_STACK_ACCESS_API
arch/x86/kernel/paravirt.c +0 -5
index fec381533555..2b78a6b466ed 100644
@@ -59,11 +59,6 @@ void __init native_pv_lock_init(void)
  		static_branch_enable(&virt_spin_lock_key);
  }
 
-static void native_tlb_remove_table(struct mmu_gather *tlb, void *table)
-{
-	tlb_remove_page(tlb, table);
-}
-
  struct static_key paravirt_steal_enabled;
  struct static_key paravirt_steal_rq_enabled;
 

arch/x86/kernel/paravirt.c struct paravirt_patch_template pv_ops = { · line 192

Quoted 8 lines
arch/x86/kernel/paravirt.c +1 -1
@@ -191,7 +186,7 @@ struct paravirt_patch_template pv_ops = {
  	.mmu.flush_tlb_kernel	= native_flush_tlb_global,
  	.mmu.flush_tlb_one_user	= native_flush_tlb_one_user,
  	.mmu.flush_tlb_multi	= native_flush_tlb_multi,
-	.mmu.tlb_remove_table	= native_tlb_remove_table,
+	.mmu.tlb_remove_table	= tlb_remove_table,
 
  	.mmu.exit_mmap		= paravirt_nop,
  	.mmu.notify_page_enc_status_changed	= paravirt_nop,

It look like this patch series is not based on the latest linux-next.

In addition to the above case, maybe the paravirt_tlb_remove_table() in arch/x86/mm/pgtable.c also needs to be changed to tlb_remove_table()?

Thanks,
Qi

#3 lore raw

Re: [PATCH 01/11] x86/mm: make MMU_GATHER_RCU_TABLE_FREE unconditional

Quoted 16 lines · Qi Zheng

Very happy to see this change!

Quoted 10 lines
arch/x86/kernel/paravirt.c +0 -0
@@ -191,7 +186,7 @@ struct paravirt_patch_template pv_ops = {
   	.mmu.flush_tlb_kernel	= native_flush_tlb_global,

   .mmu.flush_tlb_one_user =
native_flush_tlb_one_user,
   .mmu.flush_tlb_multi = native_flush_tlb_multi,

arch/x86/kernel/paravirt.c +1 -1
-	.mmu.tlb_remove_table	= native_tlb_remove_table,
+	.mmu.tlb_remove_table	= tlb_remove_table,


   .mmu.exit_mmap = paravirt_nop,
   .mmu.notify_page_enc_status_changed = paravirt_nop,

It look like this patch series is not based on the latest linux-next.

That is correct. I based this on tip.git x86/mm, since that seems like the most likely destination for this code.

Quoted 3 lines

In addition to the above case, maybe the paravirt_tlb_remove_table() in arch/x86/mm/pgtable.c also needs to be changed to tlb_remove_table()?

I'll get that in the next version.

Thank you for reviewing the patch!

Signature · 1 line
All Rights Reversed.
maintainer
#4 lore raw

Re: [PATCH 01/11] x86/mm: make MMU_GATHER_RCU_TABLE_FREE unconditional

In reply to Rik van Riel
Quoted 5 lines · Rik van Riel

Currently x86 usse CONFIG_MMU_GATHER_TABLE_FREE when using paravirt, and not when running on bare metal.

There is no real good reason to do things differently for each setup. Make them all the same.

More importantly, the changes you're proposing very much rely on this. Without TLBi IPIs nothing serializes GUP-fast vs TLBi and this RCU-ish table free scheme is required.

Quoted 48 lines
---
 arch/x86/Kconfig           | 2 +-
 arch/x86/kernel/paravirt.c | 7 +------
 2 files changed, 2 insertions(+), 7 deletions(-)
arch/x86/Kconfig +1 -1
index 9d7bd0ae48c4..e8743f8c9fd0 100644
@@ -274,7 +274,7 @@ config X86
 	select HAVE_PCI
 	select HAVE_PERF_REGS
 	select HAVE_PERF_USER_STACK_DUMP
-	select MMU_GATHER_RCU_TABLE_FREE	if PARAVIRT
+	select MMU_GATHER_RCU_TABLE_FREE
 	select MMU_GATHER_MERGE_VMAS
 	select HAVE_POSIX_CPU_TIMERS_TASK_WORK
 	select HAVE_REGS_AND_STACK_ACCESS_API
arch/x86/kernel/paravirt.c +1 -6
index fec381533555..2b78a6b466ed 100644
@@ -59,11 +59,6 @@ void __init native_pv_lock_init(void)
 		static_branch_enable(&virt_spin_lock_key);
 }
 
-static void native_tlb_remove_table(struct mmu_gather *tlb, void *table)
-{
-	tlb_remove_page(tlb, table);
-}
-
 struct static_key paravirt_steal_enabled;
 struct static_key paravirt_steal_rq_enabled;
 
@@ -191,7 +186,7 @@ struct paravirt_patch_template pv_ops = {
 	.mmu.flush_tlb_kernel	= native_flush_tlb_global,
 	.mmu.flush_tlb_one_user	= native_flush_tlb_one_user,
 	.mmu.flush_tlb_multi	= native_flush_tlb_multi,
-	.mmu.tlb_remove_table	= native_tlb_remove_table,
+	.mmu.tlb_remove_table	= tlb_remove_table,
 
 	.mmu.exit_mmap		= paravirt_nop,
 	.mmu.notify_page_enc_status_changed	= paravirt_nop,
Signature · 1 line
2.47.1
#5 lore raw

PATCH v2 2/11 x86/mm: add X86_FEATURE_INVLPGB definition.

In reply to Rik van Riel

Add the INVPLGB CPUID definition, allowing the kernel to recognize whether the CPU supports the INVLPGB instruction.

---
 arch/x86/include/asm/cpufeatures.h | 1 +
 1 file changed, 1 insertion(+)
arch/x86/include/asm/cpufeatures.h +1 -0
index 17b6590748c0..b7209d6c3a5f 100644
@@ -338,6 +338,7 @@
 #define X86_FEATURE_CLZERO		(13*32+ 0) /* "clzero" CLZERO instruction */
 #define X86_FEATURE_IRPERF		(13*32+ 1) /* "irperf" Instructions Retired Count */
 #define X86_FEATURE_XSAVEERPTR		(13*32+ 2) /* "xsaveerptr" Always save/restore FP error pointers */
+#define X86_FEATURE_INVLPGB		(13*32+ 3) /* "invlpgb" INVLPGB instruction */
 #define X86_FEATURE_RDPRU		(13*32+ 4) /* "rdpru" Read processor register at user level */
 #define X86_FEATURE_WBNOINVD		(13*32+ 9) /* "wbnoinvd" WBNOINVD instruction */
 #define X86_FEATURE_AMD_IBPB		(13*32+12) /* Indirect Branch Prediction Barrier */
Signature · 1 line
2.47.1
#6 lore raw

PATCH v2 3/11 x86/mm: get INVLPGB count max from CPUID

In reply to Rik van Riel

The CPU advertises the maximum number of pages that can be shot down with one INVLPGB instruction in the CPUID data.

Save that information for later use.

---
 arch/x86/include/asm/tlbflush.h | 1 +
 arch/x86/kernel/cpu/amd.c       | 8 ++++++++
 arch/x86/kernel/setup.c         | 4 ++++
 3 files changed, 13 insertions(+)
arch/x86/include/asm/tlbflush.h +1 -0
index 02fc2aa06e9e..7d1468a3967b 100644
@@ -182,6 +182,7 @@ static inline void cr4_init_shadow(void)
 
 extern unsigned long mmu_cr4_features;
 extern u32 *trampoline_cr4_features;
+extern u16 invlpgb_count_max;
 
 extern void initialize_tlbstate_and_flush(void);
 
arch/x86/kernel/cpu/amd.c +8 -0
index 79d2e17f6582..226b8fc64bfc 100644
@@ -1135,6 +1135,14 @@ static void cpu_detect_tlb_amd(struct cpuinfo_x86 *c)
 		tlb_lli_2m[ENTRIES] = eax & mask;
 
 	tlb_lli_4m[ENTRIES] = tlb_lli_2m[ENTRIES] >> 1;
+
+	if (c->extended_cpuid_level < 0x80000008)
+		return;
+
+	cpuid(0x80000008, &eax, &ebx, &ecx, &edx);
+
+	/* Max number of pages INVLPGB can invalidate in one shot */
+	invlpgb_count_max = (edx & 0xffff) + 1;
 }
 
 static const struct cpu_dev amd_cpu_dev = {
arch/x86/kernel/setup.c +4 -0
index f1fea506e20f..ef2b49edca25 100644
@@ -138,6 +138,10 @@ __visible unsigned long mmu_cr4_features __ro_after_init;
 __visible unsigned long mmu_cr4_features __ro_after_init = X86_CR4_PAE;
 #endif
 
+#ifdef CONFIG_CPU_SUP_AMD
+u16 invlpgb_count_max;
+#endif
+
 #ifdef CONFIG_IMA
 static phys_addr_t ima_kexec_buffer_phys;
 static size_t ima_kexec_buffer_size;
Signature · 1 line
2.47.1
#7 lore raw

Re: [PATCH 03/11] x86/mm: get INVLPGB count max from CPUID

arch/x86/kernel/setup.c __visible unsigned long mmu_cr4_features __ro_after_init; · line 142

Quoted 12 lines · Rik van Riel
arch/x86/kernel/setup.c +2 -0
@@ -138,6 +138,10 @@ __visible unsigned long mmu_cr4_features __ro_after_init;
 __visible unsigned long mmu_cr4_features __ro_after_init = X86_CR4_PAE;
 #endif
 
+#ifdef CONFIG_CPU_SUP_AMD
+u16 invlpgb_count_max;

Any reason to mark it also as __ro_after_init?

#8 lore raw

PATCH v2 4/11 x86/mm: add INVLPGB support code

In reply to Rik van Riel

Add invlpgb.h with the helper functions and definitions needed to use broadcast TLB invalidation on AMD EPYC 3 and newer CPUs.

---
 arch/x86/include/asm/invlpgb.h  | 93 +++++++++++++++++++++++++++++++++
 arch/x86/include/asm/tlbflush.h |  1 +
 2 files changed, 94 insertions(+)
 create mode 100644 arch/x86/include/asm/invlpgb.h
arch/x86/include/asm/invlpgb.h new file +93 -0
new file mode 100644
index 000000000000..862775897a54
@@ -0,0 +1,93 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _ASM_X86_INVLPGB
+#define _ASM_X86_INVLPGB
+
+#include <vdso/bits.h>
+
+/*
+ * INVLPGB does broadcast TLB invalidation across all the CPUs in the system.
+ *
+ * The INVLPGB instruction is weakly ordered, and a batch of invalidations can
+ * be done in a parallel fashion.
+ *
+ * TLBSYNC is used to ensure that pending INVLPGB invalidations initiated from
+ * this CPU have completed.
+ */
+static inline void __invlpgb(unsigned long asid, unsigned long pcid, unsigned long addr,
+			    int extra_count, bool pmd_stride, unsigned long flags)
+{
+	u64 rax = addr | flags;
+	u32 ecx = (pmd_stride << 31) | extra_count;
+	u32 edx = (pcid << 16) | asid;
+
+	asm volatile("invlpgb" : : "a" (rax), "c" (ecx), "d" (edx));
+}
+
+/*
+ * INVLPGB can be targeted by virtual address, PCID, ASID, or any combination
+ * of the three. For example:
+ * - INVLPGB_VA | INVLPGB_INCLUDE_GLOBAL: invalidate all TLB entries at the address
+ * - INVLPGB_PCID:              	  invalidate all TLB entries matching the PCID
+ *
+ * The first can be used to invalidate (kernel) mappings at a particular
+ * address across all processes.
+ *
+ * The latter invalidates all TLB entries matching a PCID.
+ */
+#define INVLPGB_VA			BIT(0)
+#define INVLPGB_PCID			BIT(1)
+#define INVLPGB_ASID			BIT(2)
+#define INVLPGB_INCLUDE_GLOBAL		BIT(3)
+#define INVLPGB_FINAL_ONLY		BIT(4)
+#define INVLPGB_INCLUDE_NESTED		BIT(5)
+
+/* Flush all mappings for a given pcid and addr, not including globals. */
+static inline void invlpgb_flush_user(unsigned long pcid,
+				      unsigned long addr)
+{
+	__invlpgb(0, pcid, addr, 0, 0, INVLPGB_PCID | INVLPGB_VA);
+}
+
+static inline void invlpgb_flush_user_nr(unsigned long pcid, unsigned long addr,
+					 int nr, bool pmd_stride)
+{
+	__invlpgb(0, pcid, addr, nr - 1, pmd_stride, INVLPGB_PCID | INVLPGB_VA);
+}
+
+/* Flush all mappings for a given ASID, not including globals. */
+static inline void invlpgb_flush_single_asid(unsigned long asid)
+{
+	__invlpgb(asid, 0, 0, 0, 0, INVLPGB_ASID);
+}
+
+/* Flush all mappings for a given PCID, not including globals. */
+static inline void invlpgb_flush_single_pcid(unsigned long pcid)
+{
+	__invlpgb(0, pcid, 0, 0, 0, INVLPGB_PCID);
+}
+
+/* Flush all mappings, including globals, for all PCIDs. */
+static inline void invlpgb_flush_all(void)
+{
+	__invlpgb(0, 0, 0, 0, 0, INVLPGB_INCLUDE_GLOBAL);
+}
+
+/* Flush addr, including globals, for all PCIDs. */
+static inline void invlpgb_flush_addr(unsigned long addr, int nr)
+{
+	__invlpgb(0, 0, addr, nr - 1, 0, INVLPGB_INCLUDE_GLOBAL);
+}
+
+/* Flush all mappings for all PCIDs except globals. */
+static inline void invlpgb_flush_all_nonglobals(void)
+{
+	__invlpgb(0, 0, 0, 0, 0, 0);
+}
+
+/* Wait for INVLPGB originated by this CPU to complete. */
+static inline void tlbsync(void)
+{
+	asm volatile("tlbsync");
+}
+
+#endif /* _ASM_X86_INVLPGB */
arch/x86/include/asm/tlbflush.h +1 -0
index 7d1468a3967b..20074f17fbcd 100644
@@ -10,6 +10,7 @@
 #include <asm/cpufeature.h>
 #include <asm/special_insns.h>
 #include <asm/smp.h>
+#include <asm/invlpgb.h>
 #include <asm/invpcid.h>
 #include <asm/pti.h>
 #include <asm/processor-flags.h>
Signature · 1 line
2.47.1
#9 lore raw

PATCH v2 5/11 x86/mm: use INVLPGB for kernel TLB flushes

In reply to Rik van Riel

Use broadcast TLB invalidation for kernel addresses when available.

This stops us from having to send IPIs for kernel TLB flushes.

---
 arch/x86/mm/tlb.c | 31 +++++++++++++++++++++++++++++++
 1 file changed, 31 insertions(+)
arch/x86/mm/tlb.c +31 -0
index 6cf881a942bb..29207dc5b807 100644
@@ -1077,6 +1077,32 @@ void flush_tlb_all(void)
 	on_each_cpu(do_flush_tlb_all, NULL, 1);
 }
 
+static void broadcast_kernel_range_flush(unsigned long start, unsigned long end)
+{
+	unsigned long addr;
+	unsigned long maxnr = invlpgb_count_max;
+	unsigned long threshold = tlb_single_page_flush_ceiling * maxnr;
+
+	/*
+	 * TLBSYNC only waits for flushes originating on the same CPU.
+	 * Disabling migration allows us to wait on all flushes.
+	 */
+	guard(preempt)();
+
+	if (end == TLB_FLUSH_ALL ||
+	    (end - start) > threshold << PAGE_SHIFT) {
+		invlpgb_flush_all();
+	} else {
+		unsigned long nr;
+		for (addr = start; addr < end; addr += nr << PAGE_SHIFT) {
+			nr = min((end - addr) >> PAGE_SHIFT, maxnr);
+			invlpgb_flush_addr(addr, nr);
+		}
+	}
+
+	tlbsync();
+}
+
 static void do_kernel_range_flush(void *info)
 {
 	struct flush_tlb_info *f = info;
@@ -1089,6 +1115,11 @@ static void do_kernel_range_flush(void *info)
 
 void flush_tlb_kernel_range(unsigned long start, unsigned long end)
 {
+	if (cpu_feature_enabled(X86_FEATURE_INVLPGB)) {
+		broadcast_kernel_range_flush(start, end);
+		return;
+	}
+
 	/* Balance as user space task's flush, a bit conservative */
 	if (end == TLB_FLUSH_ALL ||
 	    (end - start) > tlb_single_page_flush_ceiling << PAGE_SHIFT) {
Signature · 1 line
2.47.1
#10 lore raw

PATCH v2 6/11 x86/tlb: use INVLPGB in flush_tlb_all

In reply to Rik van Riel

The flush_tlb_all() function is not used a whole lot, but we might as well use broadcast TLB flushing there, too.

---
 arch/x86/mm/tlb.c | 6 ++++++
 1 file changed, 6 insertions(+)
arch/x86/mm/tlb.c +6 -0
index 29207dc5b807..266d5174fc7b 100644
@@ -1074,6 +1074,12 @@ static void do_flush_tlb_all(void *info)
 void flush_tlb_all(void)
 {
 	count_vm_tlb_event(NR_TLB_REMOTE_FLUSH);
+	if (cpu_feature_enabled(X86_FEATURE_INVLPGB)) {
+		guard(preempt)();
+		invlpgb_flush_all();
+		tlbsync();
+		return;
+	}
 	on_each_cpu(do_flush_tlb_all, NULL, 1);
 }
 
Signature · 1 line
2.47.1
#11 lore raw

PATCH v2 7/11 x86/mm: use broadcast TLB flushing for page reclaim TLB flushing

In reply to Rik van Riel

In the page reclaim code, we only track the CPU(s) where the TLB needs to be flushed, rather than all the individual mappings that may be getting invalidated.

Use broadcast TLB flushing when that is available.

---
 arch/x86/mm/tlb.c | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)
arch/x86/mm/tlb.c +9 -1
index 266d5174fc7b..64f1679c37e1 100644
@@ -1310,8 +1310,16 @@ EXPORT_SYMBOL_GPL(__flush_tlb_all);
 void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch)
 {
 	struct flush_tlb_info *info;
+	int cpu;
+
+	if (cpu_feature_enabled(X86_FEATURE_INVLPGB)) {
+		guard(preempt)();
+		invlpgb_flush_all_nonglobals();
+		tlbsync();
+		return;
+	}
 
-	int cpu = get_cpu();
+	cpu = get_cpu();
 
 	info = get_flush_tlb_info(NULL, 0, TLB_FLUSH_ALL, 0, false,
 				  TLB_GENERATION_INVALID);
Signature · 1 line
2.47.1
#12 lore raw

PATCH v2 8/11 x86/mm: enable broadcast TLB invalidation for multi-threaded processes

In reply to Rik van Riel

Use broadcast TLB invalidation, using the INVPLGB instruction, on AMD EPYC 3 and newer CPUs.

In order to not exhaust PCID space, and keep TLB flushes local for single threaded processes, we only hand out broadcast ASIDs to processes active on 3 or more CPUs, and gradually increase the threshold as broadcast ASID space is depleted.

---
 arch/x86/include/asm/mmu.h         |   6 +
 arch/x86/include/asm/mmu_context.h |  12 ++
 arch/x86/include/asm/tlbflush.h    |  15 ++
 arch/x86/mm/tlb.c                  | 313 ++++++++++++++++++++++++++++-
 4 files changed, 337 insertions(+), 9 deletions(-)
arch/x86/include/asm/mmu.h +6 -0
index 3b496cdcb74b..a8e8dfa5a520 100644
@@ -48,6 +48,12 @@ typedef struct {
 	unsigned long flags;
 #endif
 
+#ifdef CONFIG_CPU_SUP_AMD
+	struct list_head broadcast_asid_list;
+	u16 broadcast_asid;
+	bool asid_transition;
+#endif
+
 #ifdef CONFIG_ADDRESS_MASKING
 	/* Active LAM mode:  X86_CR3_LAM_U48 or X86_CR3_LAM_U57 or 0 (disabled) */
 	unsigned long lam_cr3_mask;
arch/x86/include/asm/mmu_context.h +12 -0
index 795fdd53bd0a..0dc446c427d2 100644
@@ -139,6 +139,8 @@ static inline void mm_reset_untag_mask(struct mm_struct *mm)
 #define enter_lazy_tlb enter_lazy_tlb
 extern void enter_lazy_tlb(struct mm_struct *mm, struct task_struct *tsk);
 
+extern void destroy_context_free_broadcast_asid(struct mm_struct *mm);
+
 /*
  * Init a new mm.  Used on mm copies, like at fork()
  * and on mm's that are brand-new, like at execve().
@@ -161,6 +163,13 @@ static inline int init_new_context(struct task_struct *tsk,
 		mm->context.execute_only_pkey = -1;
 	}
 #endif
+
+#ifdef CONFIG_CPU_SUP_AMD
+	INIT_LIST_HEAD(&mm->context.broadcast_asid_list);
+	mm->context.broadcast_asid = 0;
+	mm->context.asid_transition = false;
+#endif
+
 	mm_reset_untag_mask(mm);
 	init_new_context_ldt(mm);
 	return 0;
@@ -170,6 +179,9 @@ static inline int init_new_context(struct task_struct *tsk,
 static inline void destroy_context(struct mm_struct *mm)
 {
 	destroy_context_ldt(mm);
+#ifdef CONFIG_CPU_SUP_AMD
+	destroy_context_free_broadcast_asid(mm);
+#endif
 }
 
 extern void switch_mm(struct mm_struct *prev, struct mm_struct *next,
arch/x86/include/asm/tlbflush.h +15 -0
index 20074f17fbcd..074f46b74b92 100644
@@ -65,6 +65,21 @@ static inline void cr4_clear_bits(unsigned long mask)
  */
 #define TLB_NR_DYN_ASIDS	6
 
+#ifdef CONFIG_CPU_SUP_AMD
+#define is_dyn_asid(asid) (asid) < TLB_NR_DYN_ASIDS
+#define is_broadcast_asid(asid) (asid) >= TLB_NR_DYN_ASIDS
+#define in_asid_transition(info) (info->mm && info->mm->context.asid_transition)
+#else
+#define is_dyn_asid(asid) true
+#define is_broadcast_asid(asid) false
+#define in_asid_transition(info) false
+
+inline bool needs_broadcast_asid_reload(struct mm_struct *next, u16 prev_asid)
+{
+	return false;
+}
+#endif
+
 struct tlb_context {
 	u64 ctx_id;
 	u64 tlb_gen;
arch/x86/mm/tlb.c +304 -9

400 lines not on this page · raw message

Signature · 1 line
2.47.1
#13 lore raw

Re: [PATCH 08/11] x86/mm: enable broadcast TLB invalidation for multi-threaded processes

Quoted 7 lines · Rik van Riel
arch/x86/mm/tlb.c +7 -0
+static int mm_active_cpus(struct mm_struct *mm)
+{
+	int count = 0;
+	int cpu;
+
+	for_each_cpu(cpu, mm_cpumask(mm)) {
+		/* Skip the CPUs that aren't really running this process. */

arch/x86/mm/tlb.c static void choose_new_asid(struct mm_struct *next, u64 next_tlb_gen, · line 367

Quoted 10 lines
arch/x86/mm/tlb.c +10 -0
+		if (per_cpu(cpu_tlbstate.loaded_mm, cpu) != mm)
+			continue;
+
+		if (per_cpu(cpu_tlbstate_shared.is_lazy, cpu))
+			continue;
+
+		count++;
+	}
+	return count;
+}

Since you are only interested in checking whether the number of “mm active CPUs" is greater than a certain threshold, don’t you want to add some checks for early termination? This can allow to avoid cachelines of cpu_tlbstate traversing back and forth.

For instance, by running cpumask_weight() first, if the weight is lower than the threshold, no need to loop. Similarly, if inside the loop the threshold has already been crossed, no need for more iterations.

#14 lore raw

Re: [PATCH 08/11] x86/mm: enable broadcast TLB invalidation for multi-threaded processes

In reply to Rik van Riel

arch/x86/mm/tlb.c void flush_tlb_mm_range(struct mm_struct *mm, unsigned long start, · line 1346

Quoted 12 lines · Rik van Riel
arch/x86/mm/tlb.c +2 -1
@@ -1049,9 +1341,12 @@ void flush_tlb_mm_range(struct mm_struct *mm, unsigned long start,
 	 * a local TLB flush is needed. Optimize this use-case by calling
 	 * flush_tlb_func_local() directly in this case.
 	 */
-	if (cpumask_any_but(mm_cpumask(mm), cpu) < nr_cpu_ids) {
+	if (IS_ENABLED(CONFIG_CPU_SUP_AMD) && mm->context.broadcast_asid) {
+		broadcast_tlb_flush(info);
 

I think broadcast_asid is defined within an ifdef, so the IS_ENABLED() here would not save you from having to use ifdef.

#15 lore raw

PATCH v2 9/11 x86,tlb: do targeted broadcast flushing from tlbbatch code

In reply to Rik van Riel

Instead of doing a system-wide TLB flush from arch_tlbbatch_flush, queue up asynchronous, targeted flushes from arch_tlbbatch_add_pending.

This also allows us to avoid adding the CPUs of processes using broadcast flushing to the batch->cpumask, and will hopefully further reduce TLB flushing from the reclaim and compaction paths.

---
 arch/x86/include/asm/tlbbatch.h |  1 +
 arch/x86/include/asm/tlbflush.h | 12 +++------
 arch/x86/mm/tlb.c               | 48 ++++++++++++++++++++++++++-------
 3 files changed, 42 insertions(+), 19 deletions(-)
arch/x86/include/asm/tlbbatch.h +1 -0
index 1ad56eb3e8a8..f9a17edf63ad 100644
@@ -10,6 +10,7 @@ struct arch_tlbflush_unmap_batch {
 	 * the PFNs being flushed..
 	 */
 	struct cpumask cpumask;
+	bool used_invlpgb;
 };
 
 #endif /* _ARCH_X86_TLBBATCH_H */
arch/x86/include/asm/tlbflush.h +3 -9
index 074f46b74b92..71d094841356 100644
@@ -295,21 +295,15 @@ static inline u64 inc_mm_tlb_gen(struct mm_struct *mm)
 	return atomic64_inc_return(&mm->context.tlb_gen);
 }
 
-static inline void arch_tlbbatch_add_pending(struct arch_tlbflush_unmap_batch *batch,
-					     struct mm_struct *mm,
-					     unsigned long uaddr)
-{
-	inc_mm_tlb_gen(mm);
-	cpumask_or(&batch->cpumask, &batch->cpumask, mm_cpumask(mm));
-	mmu_notifier_arch_invalidate_secondary_tlbs(mm, 0, -1UL);
-}
-
 static inline void arch_flush_tlb_batched_pending(struct mm_struct *mm)
 {
 	flush_tlb_mm(mm);
 }
 
 extern void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch);
+extern void arch_tlbbatch_add_pending(struct arch_tlbflush_unmap_batch *batch,
+					     struct mm_struct *mm,
+					     unsigned long uaddr);
 
 static inline bool pte_flags_need_flush(unsigned long oldflags,
 					unsigned long newflags,
arch/x86/mm/tlb.c +38 -10

66 lines not on this page · raw message

Signature · 1 line
2.47.1
#16 lore raw

PATCH v2 10/11 x86/mm: enable AMD translation cache extensions

In reply to Rik van Riel

With AMD TCE (translation cache extensions) only the intermediate mappings that cover the address range zapped by INVLPG / INVLPGB get invalidated, rather than all intermediate mappings getting zapped at every TLB invalidation.

This can help reduce the TLB miss rate, by keeping more intermediate mappings in the cache.

From the AMD manual:

Translation Cache Extension (TCE) Bit. Bit 15, read/write. Setting this bit to 1 changes how the INVLPG, INVLPGB, and INVPCID instructions operate on TLB entries. When this bit is 0, these instructions remove the target PTE from the TLB as well as all upper-level table entries that are cached in the TLB, whether or not they are associated with the target PTE. When this bit is set, these instructions will remove the target PTE and only those upper-level entries that lead to the target PTE in the page table hierarchy, leaving unrelated upper-level entries intact.

---
 arch/x86/kernel/cpu/amd.c |  8 ++++++++
 arch/x86/mm/tlb.c         | 10 +++++++---
 2 files changed, 15 insertions(+), 3 deletions(-)
arch/x86/kernel/cpu/amd.c +8 -0
index 226b8fc64bfc..4dc42705aaca 100644
@@ -1143,6 +1143,14 @@ static void cpu_detect_tlb_amd(struct cpuinfo_x86 *c)
 
 	/* Max number of pages INVLPGB can invalidate in one shot */
 	invlpgb_count_max = (edx & 0xffff) + 1;
+
+	/* If supported, enable translation cache extensions (TCE) */
+	cpuid(0x80000001, &eax, &ebx, &ecx, &edx);
+	if (ecx & BIT(17)) {
+		u64 msr = native_read_msr(MSR_EFER);;
+		msr |= BIT(15);
+		wrmsrl(MSR_EFER, msr);
+	}
 }
 
 static const struct cpu_dev amd_cpu_dev = {
arch/x86/mm/tlb.c +7 -3
index c5459516a72e..f1e2358616e5 100644
@@ -480,7 +480,7 @@ static void broadcast_tlb_flush(struct flush_tlb_info *info)
 	if (info->stride_shift > PMD_SHIFT)
 		maxnr = 1;
 
-	if (info->end == TLB_FLUSH_ALL) {
+	if (info->end == TLB_FLUSH_ALL || info->freed_tables) {
 		invlpgb_flush_single_pcid(kern_pcid(asid));
 		/* Do any CPUs supporting INVLPGB need PTI? */
 		if (static_cpu_has(X86_FEATURE_PTI))
@@ -1113,7 +1113,7 @@ static void flush_tlb_func(void *info)
 	 *
 	 * The only question is whether to do a full or partial flush.
 	 *
-	 * We do a partial flush if requested and two extra conditions
+	 * We do a partial flush if requested and three extra conditions
 	 * are met:
 	 *
 	 * 1. f->new_tlb_gen == local_tlb_gen + 1.  We have an invariant that
@@ -1140,10 +1140,14 @@ static void flush_tlb_func(void *info)
 	 *    date.  By doing a full flush instead, we can increase
 	 *    local_tlb_gen all the way to mm_tlb_gen and we can probably
 	 *    avoid another flush in the very near future.
+	 *
+	 * 3. No page tables were freed. If page tables were freed, a full
+	 *    flush ensures intermediate translations in the TLB get flushed.
 	 */
 	if (f->end != TLB_FLUSH_ALL &&
 	    f->new_tlb_gen == local_tlb_gen + 1 &&
-	    f->new_tlb_gen == mm_tlb_gen) {
+	    f->new_tlb_gen == mm_tlb_gen &&
+	    !f->freed_tables) {
 		/* Partial flush */
 		unsigned long addr = f->start;
 
Signature · 1 line
2.47.1
#17 lore raw

PATCH v2 11/11 x86/mm: only invalidate final translations with INVLPGB

In reply to Rik van Riel

Use the INVLPGB_FINAL_ONLY flag when invalidating mappings with INVPLGB. This way only leaf mappings get removed from the TLB, leaving intermediate translations cached.

On the (rare) occasions where we free page tables we do a full flush, ensuring intermediate translations get flushed from the TLB.

---
 arch/x86/include/asm/invlpgb.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/include/asm/invlpgb.h +1 -1
index 862775897a54..2669ebfffe81 100644
@@ -51,7 +51,7 @@ static inline void invlpgb_flush_user(unsigned long pcid,
 static inline void invlpgb_flush_user_nr(unsigned long pcid, unsigned long addr,
 					 int nr, bool pmd_stride)
 {
-	__invlpgb(0, pcid, addr, nr - 1, pmd_stride, INVLPGB_PCID | INVLPGB_VA);
+	__invlpgb(0, pcid, addr, nr - 1, pmd_stride, INVLPGB_PCID | INVLPGB_VA | INVLPGB_FINAL_ONLY);
 }
 
 /* Flush all mappings for a given ASID, not including globals. */
Signature · 1 line
2.47.1
#18 lore raw
In reply to Rik van Riel

From: [email protected] <[email protected]> Sent: Sunday, December 22, 2024 6:55 PM

Quoted 8 lines

Add support for broadcast TLB invalidation using AMD's INVLPGB instruction.

This allows the kernel to invalidate TLB entries on remote CPUs without needing to send IPIs, without having to wait for remote CPUs to handle those interrupts, and with less interruption to what was running on those CPUs.

Quoted 4 lines

Because x86 PCID space is limited, and there are some very large systems out there, broadcast TLB invalidation is only used for processes that are active on 3 or more CPUs, with the threshold being gradually increased the more the PCID space gets exhausted.

Rik --

What is this patch set's expectation about INVLPGB and TLBSYNC availability and usage in a VM? I see that INVLPGB and TLBYSNC behavior in a VM is spec'ed in the AMD Programmer's Manual, but I wonder about their impact in a multi-tenant host like in a public cloud environment. And given what this patch set does in assigning global ASIDs, should X86_FEATURE_INVLPGB be disabled if running in a VM where the hypervisor for whatever reason has enabled INVLPGB/TLBSYNC in its VMs?

My knowledge of the details here is pretty limited, so my question may just reflect my ignorance. But it would be good for the code comments and/or commit messages to include explicit statements about what is expected in a VM.

Michael

Quoted 25 lines

Combined with the removal of unnecessary lru_add_drain calls (see https://lkml.org/lkml/2024/12/19/1388) this results in a nice performance boost for the will-it-scale tlb_flush2_threads test on an AMD Milan system with 36 cores:

- vanilla kernel:           527k loops/second
- lru_add_drain removal:    731k loops/second
- only INVLPGB:             527k loops/second

- lru_add_drain + INVLPGB: 1157k loops/second

Profiling with only the INVLPGB changes showed while TLB invalidation went down from 40% of the total CPU time to only around 4% of CPU time, the contention simply moved to the LRU lock.

Fixing both at the same time about doubles the
number of iterations per second from this case.

v2:
- Apply suggestions by Peter and Borislav (thank you!)
- Fix bug in arch_tlbbatch_flush, where we need to do both the TLBSYNC, and flush the CPUs that are in the cpumask.
- Some updates to comments and changelogs based on questions.

#19 lore raw
Quoted 21 lines · Michael Kelley

From: [email protected] <[email protected]> Sent: Sunday, December 22, 2024 6:55 PM

Quoted 11 lines

Add support for broadcast TLB invalidation using AMD's INVLPGB instruction.

This allows the kernel to invalidate TLB entries on remote CPUs without
needing to send IPIs, without having to wait for remote CPUs to handle
those interrupts, and with less interruption to what was running on those CPUs.

Quoted 4 lines

Because x86 PCID space is limited, and there are some very large systems out there, broadcast TLB invalidation is only used for processes that are active on 3 or more CPUs, with the threshold being gradually increased the more the PCID space gets exhausted.

Rik --

Quoted 9 lines

What is this patch set's expectation about INVLPGB and TLBSYNC availability and usage in a VM? I see that INVLPGB and TLBYSNC behavior in a VM is spec'ed in the AMD Programmer's Manual, but I wonder about their impact in a multi-tenant host like in a public cloud environment. And given what this patch set does in assigning global ASIDs, should X86_FEATURE_INVLPGB be disabled if running in a VM where the hypervisor for whatever reason has enabled INVLPGB/TLBSYNC in its VMs?

This patch series enables bare metal INVLPGB functionality.

Virtual machines should probably not expose the INVPLGB CPUID feature bit to guests, since virtual machine invalidation seems to work differently than bare metal invalidation.

For one, the ASID seems to actually mean something in SVM context, while trying to use the ASID in bare metal blows up :)

Signature · 1 line
All Rights Reversed.
#20 lore raw
Quoted 33 lines · Rik van Riel
Quoted 21 lines · Michael Kelley

From: [email protected] <[email protected]> Sent: Sunday, December 22, 2024 6:55 PM

Quoted 11 lines

Add support for broadcast TLB invalidation using AMD's INVLPGB instruction.

This allows the kernel to invalidate TLB entries on remote CPUs without
needing to send IPIs, without having to wait for remote CPUs to handle
those interrupts, and with less interruption to what was running on those CPUs.

Quoted 4 lines

Because x86 PCID space is limited, and there are some very large systems out there, broadcast TLB invalidation is only used for processes that are active on 3 or more CPUs, with the threshold being gradually increased the more the PCID space gets exhausted.

Rik --

Quoted 9 lines

What is this patch set's expectation about INVLPGB and TLBSYNC availability and usage in a VM? I see that INVLPGB and TLBYSNC behavior in a VM is spec'ed in the AMD Programmer's Manual, but I wonder about their impact in a multi-tenant host like in a public cloud environment. And given what this patch set does in assigning global ASIDs, should X86_FEATURE_INVLPGB be disabled if running in a VM where the hypervisor for whatever reason has enabled INVLPGB/TLBSYNC in its VMs?

This patch series enables bare metal INVLPGB functionality.

Quoted 8 lines

Virtual machines should probably not expose the INVPLGB CPUID feature bit to guests, since virtual machine invalidation seems to work differently than bare metal invalidation.

For one, the ASID seems to actually mean something in SVM context, while trying to use the ASID in bare metal blows up :)

Note that global ASIDs (relative to VMs) are different from the broadcast ASIDs being used here. IIUC, the broadcast ASIDs here get translated to a PCID value (kern_pcid(asid) or user_pcid(asid) in patch #9).

Thanks,
Tom

Quoted 2 lines