Skip to contentSkip to conversation

Intel RAR TLB invalidation

RFC PATCH Superseded Rik van Riel 4 participants 15 messages Jun 5, 2025 – Jun 6, 2025

Subsystems (2)

v3 of 5 Superseded Latest is v5 →

lore
Share & tools
lore
#0 lore raw

v3 cover Intel RAR TLB invalidation

This patch series adds support for IPI-less TLB invalidation using Intel RAR technology.

Intel RAR differs from AMD INVLPGB in a few ways:
- RAR goes through (emulated?) APIC writes, not instructions
- RAR flushes go through a memory table with 64 entries
- RAR flushes can be targeted to a cpumask
- The RAR functionality must be set up at boot time before it can be used

The cpumask targeting has resulted in Intel RAR and AMD INVLPGB having slightly different rules:
- Processes with dynamic ASIDs use IPI based shootdowns
- INVLPGB: processes with a global ASID
- always have the TLB up to date, on every CPU
- never need to flush the TLB at context switch time
- RAR: processes with global ASIDs
- have the TLB up to date on CPUs in the mm_cpumask
- can skip a TLB flush at context switch time if the CPU is in the mm_cpumask
- need to flush the TLB when scheduled on a cpu not in the mm_cpumask, in case it used to run there before and the TLB has stale entries

RAR functionality is present on Sapphire Rapids and newer CPUs.

Information about Intel RAR can be found in this whitepaper.

https://www.intel.com/content/dam/develop/external/us/en/documents/341431-remote-action-request-white-paper.pdf

This patch series is based off a 2019 patch series created by Intel, with patches later in the series modified to fit into the TLB flush code structure we have after AMD INVLPGB functionality was integrated.

TODO:
- when flushing multiple CPUs anyway, use RAR to invalidate our own TLB (needs a chicken-egg issue with early boot solved)
- some sort of optimization to avoid sending RARs to CPUs in deeper idle states when they have init_mm loaded (flush when switching to init_mm?)

v3:
- move cpa_flush() change out of this patch series
- use MSR_IA32_CORE_CAPS definition, merge first two patches together
- move RAR initialization to early_init_intel()
- remove single-CPU "fast path" from smp_call_rar_many
- remove smp call table RAR entries, just do a direct call
- cleanups suggested (Ingo, Nadav, Dave, Thomas, Borislav, Sean)
- fix !CONFIG_SMP compile in Kconfig
- match RAR definitions to the names & numbers in the documentation
- the code seems to work now
v2:
- Cleanups suggested by Ingo and Nadav (thank you)
- Basic RAR code seems to actually work now.
- Kernel TLB flushes with RAR seem to work correctly.
- User TLB flushes with RAR are still broken, with two symptoms:
- The !is_lazy WARN_ON in leave_mm() is tripped
- Random segfaults.

#1 lore raw

RFC PATCH v3 1/7 x86/mm: Introduce Remote Action Request MSRs

From: Yu-cheng Yu <[email protected]>

Remote Action Request (RAR) is a model-specific feature to speed up inter-processor operations by moving parts of those operations from software to hardware.

The current RAR implementation handles TLB flushes and MSR writes.

This patch introduces RAR MSRs. RAR is introduced in later patches.

There are five RAR MSRs:

MSR_CORE_CAPABILITIES
MSR_IA32_RAR_CTRL
MSR_IA32_RAR_ACT_VEC
MSR_IA32_RAR_PAYLOAD_BASE
MSR_IA32_RAR_INFO

---
 arch/x86/include/asm/msr-index.h | 13 +++++++++++++
 1 file changed, 13 insertions(+)
arch/x86/include/asm/msr-index.h +13 -0
index b7dded3c8113..367a62c50aa2 100644
@@ -110,6 +110,8 @@
 
 /* Abbreviated from Intel SDM name IA32_CORE_CAPABILITIES */
 #define MSR_IA32_CORE_CAPS			  0x000000cf
+#define MSR_IA32_CORE_CAPS_RAR_BIT		  1
+#define MSR_IA32_CORE_CAPS_RAR			  BIT(MSR_IA32_CORE_CAPS_RAR_BIT)
 #define MSR_IA32_CORE_CAPS_INTEGRITY_CAPS_BIT	  2
 #define MSR_IA32_CORE_CAPS_INTEGRITY_CAPS	  BIT(MSR_IA32_CORE_CAPS_INTEGRITY_CAPS_BIT)
 #define MSR_IA32_CORE_CAPS_SPLIT_LOCK_DETECT_BIT  5
@@ -122,6 +124,17 @@
 #define SNB_C3_AUTO_UNDEMOTE		(1UL << 27)
 #define SNB_C1_AUTO_UNDEMOTE		(1UL << 28)
 
+/*
+ * Remote Action Requests (RAR) MSRs
+ */
+#define MSR_IA32_RAR_CTRL		0x000000ed
+#define MSR_IA32_RAR_ACT_VEC		0x000000ee
+#define MSR_IA32_RAR_PAYLOAD_BASE	0x000000ef
+#define MSR_IA32_RAR_INFO		0x000000f0
+
+#define RAR_CTRL_ENABLE			BIT(31)
+#define RAR_CTRL_IGNORE_IF		BIT(30)
+
 #define MSR_MTRRcap			0x000000fe
 
 #define MSR_IA32_ARCH_CAPABILITIES	0x0000010a
Signature · 1 line
2.49.0
#2 lore raw

RFC PATCH v3 2/7 x86/mm: enable BROADCAST_TLB_FLUSH on Intel, too

In reply to Rik van Riel

From: Rik van Riel <[email protected]>

Much of the code for Intel RAR and AMD INVLPGB is shared.

Place both under the same config option.

---
 arch/x86/Kconfig.cpu | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/Kconfig.cpu +1 -1
index f928cf6e3252..ab763f69f54d 100644
@@ -360,7 +360,7 @@ menuconfig PROCESSOR_SELECT
 
 config BROADCAST_TLB_FLUSH
 	def_bool y
-	depends on CPU_SUP_AMD && 64BIT
+	depends on (CPU_SUP_AMD || CPU_SUP_INTEL) && 64BIT && SMP
 
 config CPU_SUP_INTEL
 	default y
Signature · 1 line
2.49.0
#3 lore raw

RFC PATCH v3 3/7 x86/mm: Introduce X86_FEATURE_RAR

In reply to Rik van Riel

From: Yu-cheng Yu <[email protected]>

Introduce X86_FEATURE_RAR and enumeration of the feature.

[riel: moved initialization to intel.c and disabling to Kconfig.cpufeatures]

---
 arch/x86/Kconfig.cpufeatures       | 4 ++++
 arch/x86/include/asm/cpufeatures.h | 2 +-
 arch/x86/kernel/cpu/intel.c        | 9 +++++++++
 3 files changed, 14 insertions(+), 1 deletion(-)
arch/x86/Kconfig.cpufeatures +4 -0
index 250c10627ab3..7d459b5f47f7 100644
@@ -195,3 +195,7 @@ config X86_DISABLED_FEATURE_SEV_SNP
 config X86_DISABLED_FEATURE_INVLPGB
 	def_bool y
 	depends on !BROADCAST_TLB_FLUSH
+
+config X86_DISABLED_FEATURE_RAR
+	def_bool y
+	depends on !BROADCAST_TLB_FLUSH
arch/x86/include/asm/cpufeatures.h +1 -1
index ee176236c2be..e6781541ffce 100644
@@ -76,7 +76,7 @@
 #define X86_FEATURE_K8			( 3*32+ 4) /* Opteron, Athlon64 */
 #define X86_FEATURE_ZEN5		( 3*32+ 5) /* CPU based on Zen5 microarchitecture */
 #define X86_FEATURE_ZEN6		( 3*32+ 6) /* CPU based on Zen6 microarchitecture */
-/* Free                                 ( 3*32+ 7) */
+#define X86_FEATURE_RAR			( 3*32+ 7) /* Intel Remote Action Request */
 #define X86_FEATURE_CONSTANT_TSC	( 3*32+ 8) /* "constant_tsc" TSC ticks at a constant rate */
 #define X86_FEATURE_UP			( 3*32+ 9) /* "up" SMP kernel running on UP */
 #define X86_FEATURE_ART			( 3*32+10) /* "art" Always running timer (ART) */
arch/x86/kernel/cpu/intel.c +9 -0
index 076eaa41b8c8..f5cac46e1b91 100644
@@ -335,6 +335,15 @@ static void early_init_intel(struct cpuinfo_x86 *c)
 	 */
 	if (cpu_has(c, X86_FEATURE_TME))
 		detect_tme_early(c);
+
+	if (cpu_has(c, X86_FEATURE_CORE_CAPABILITIES)) {
+		u64 msr;
+
+		rdmsrl(MSR_IA32_CORE_CAPS, msr);
+
+		if (msr & MSR_IA32_CORE_CAPS_RAR)
+			setup_force_cpu_cap(X86_FEATURE_RAR);
+	}
 }
 
 static void bsp_init_intel(struct cpuinfo_x86 *c)
Signature · 1 line
2.49.0
#4 lore raw

RFC PATCH v3 4/7 x86/apic: Introduce Remote Action Request Operations

In reply to Rik van Riel

From: Yu-cheng Yu <[email protected]>

RAR TLB flushing is started by sending a command to the APIC. This patch adds Remote Action Request commands.

Because RAR_VECTOR is hardcoded at 0xe0, POSTED_MSI_NOTIFICATION_VECTOR has to be lowered to 0xdf, reducing the number of available vectors by 13.

[riel: refactor after 6 years of changes, lower POSTED_MSI_NOTIFICATION_VECTOR]

---
 arch/x86/include/asm/apicdef.h     | 1 +
 arch/x86/include/asm/irq_vectors.h | 7 ++++++-
 arch/x86/include/asm/smp.h         | 1 +
 arch/x86/kernel/apic/ipi.c         | 5 +++++
 arch/x86/kernel/apic/local.h       | 3 +++
 5 files changed, 16 insertions(+), 1 deletion(-)
arch/x86/include/asm/apicdef.h +1 -0
index 094106b6a538..b152d45af91a 100644
@@ -92,6 +92,7 @@
 #define		APIC_DM_LOWEST		0x00100
 #define		APIC_DM_SMI		0x00200
 #define		APIC_DM_REMRD		0x00300
+#define		APIC_DM_RAR		0x00300
 #define		APIC_DM_NMI		0x00400
 #define		APIC_DM_INIT		0x00500
 #define		APIC_DM_STARTUP		0x00600
arch/x86/include/asm/irq_vectors.h +6 -1
index 47051871b436..52a0cf56562a 100644
@@ -97,11 +97,16 @@
 
 #define LOCAL_TIMER_VECTOR		0xec
 
+/*
+ * RAR (remote action request) TLB flush
+ */
+#define RAR_VECTOR			0xe0
+
 /*
  * Posted interrupt notification vector for all device MSIs delivered to
  * the host kernel.
  */
-#define POSTED_MSI_NOTIFICATION_VECTOR	0xeb
+#define POSTED_MSI_NOTIFICATION_VECTOR	0xdf
 
 #define NR_VECTORS			 256
 
arch/x86/include/asm/smp.h +1 -0
index 0c1c68039d6f..0e5ad0dc987a 100644
@@ -120,6 +120,7 @@ void __noreturn mwait_play_dead(unsigned int eax_hint);
 void native_smp_send_reschedule(int cpu);
 void native_send_call_func_ipi(const struct cpumask *mask);
 void native_send_call_func_single_ipi(int cpu);
+void native_send_rar_ipi(const struct cpumask *mask);
 
 asmlinkage __visible void smp_reboot_interrupt(void);
 __visible void smp_reschedule_interrupt(struct pt_regs *regs);
arch/x86/kernel/apic/ipi.c +5 -0
index 98a57cb4aa86..9983c42619ef 100644
@@ -106,6 +106,11 @@ void apic_send_nmi_to_offline_cpu(unsigned int cpu)
 		return;
 	apic->send_IPI(cpu, NMI_VECTOR);
 }
+
+void native_send_rar_ipi(const struct cpumask *mask)
+{
+	__apic_send_IPI_mask(mask, RAR_VECTOR);
+}
 #endif /* CONFIG_SMP */
 
 static inline int __prepare_ICR2(unsigned int mask)
arch/x86/kernel/apic/local.h +3 -0
index bdcf609eb283..833669174267 100644
@@ -38,6 +38,9 @@ static inline unsigned int __prepare_ICR(unsigned int shortcut, int vector,
 	case NMI_VECTOR:
 		icr |= APIC_DM_NMI;
 		break;
+	case RAR_VECTOR:
+		icr |= APIC_DM_RAR;
+		break;
 	}
 	return icr;
 }
Signature · 1 line
2.49.0
#5 lore raw

RFC PATCH v3 5/7 x86/mm: Introduce Remote Action Request

In reply to Rik van Riel

From: Yu-cheng Yu <[email protected]>

Remote Action Request (RAR) is a TLB flushing broadcast facility. To start a TLB flush, the initiator CPU creates a RAR payload and sends a command to the APIC. The receiving CPUs automatically flush TLBs as specified in the payload without the kernel's involement.

[ riel: add pcid parameter to smp_call_rar_many so other mms can be flushed ]

---
 arch/x86/include/asm/rar.h   |  69 +++++++++++
 arch/x86/kernel/cpu/common.c |   4 +
 arch/x86/mm/Makefile         |   1 +
 arch/x86/mm/rar.c            | 217 +++++++++++++++++++++++++++++++++++
 4 files changed, 291 insertions(+)
 create mode 100644 arch/x86/include/asm/rar.h
 create mode 100644 arch/x86/mm/rar.c
arch/x86/include/asm/rar.h new file +69 -0
new file mode 100644
index 000000000000..78c039e40e81
@@ -0,0 +1,69 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _ASM_X86_RAR_H
+#define _ASM_X86_RAR_H
+
+/*
+ * RAR payload types
+ */
+#define RAR_TYPE_INVPG		0
+#define RAR_TYPE_INVPG_NO_CR3	1
+#define RAR_TYPE_INVPCID	2
+#define RAR_TYPE_INVEPT		3
+#define RAR_TYPE_INVVPID	4
+#define RAR_TYPE_WRMSR		5
+
+/*
+ * Subtypes for RAR_TYPE_INVLPG
+ */
+#define RAR_INVPG_ADDR			0 /* address specific */
+#define RAR_INVPG_ALL			2 /* all, include global */
+#define RAR_INVPG_ALL_NO_GLOBAL		3 /* all, exclude global */
+
+/*
+ * Subtypes for RAR_TYPE_INVPCID
+ */
+#define RAR_INVPCID_ADDR		0 /* address specific */
+#define RAR_INVPCID_PCID		1 /* all of PCID */
+#define RAR_INVPCID_ALL			2 /* all, include global */
+#define RAR_INVPCID_ALL_NO_GLOBAL	3 /* all, exclude global */
+
+/*
+ * Page size for RAR_TYPE_INVLPG
+ */
+#define RAR_INVLPG_PAGE_SIZE_4K		0
+#define RAR_INVLPG_PAGE_SIZE_2M		1
+#define RAR_INVLPG_PAGE_SIZE_1G		2
+
+/*
+ * Max number of pages per payload
+ */
+#define RAR_INVLPG_MAX_PAGES 63
+
+struct rar_payload {
+	u64 for_sw		: 8;
+	u64 type		: 8;
+	u64 must_be_zero_1	: 16;
+	u64 subtype		: 3;
+	u64 page_size		: 2;
+	u64 num_pages		: 6;
+	u64 must_be_zero_2	: 21;
+
+	u64 must_be_zero_3;
+
+	/*
+	 * Starting address
+	 */
+	u64 initiator_cr3;
+	u64 linear_address;
+
+	/*
+	 * Padding
+	 */
+	u64 padding[4];
+};
+
+void rar_cpu_init(void);
+void smp_call_rar_many(const struct cpumask *mask, u16 pcid,
+		       unsigned long start, unsigned long end);
+
+#endif /* _ASM_X86_RAR_H */
arch/x86/kernel/cpu/common.c +4 -0
index 8feb8fd2957a..d68a0a9b2aa2 100644
@@ -71,6 +71,7 @@
 #include <asm/tdx.h>
 #include <asm/posted_intr.h>
 #include <asm/runtime-const.h>
+#include <asm/rar.h>
 
 #include "cpu.h"
 
@@ -2425,6 +2426,9 @@ void cpu_init(void)
 	if (is_uv_system())
 		uv_cpu_init();
 
+	if (cpu_feature_enabled(X86_FEATURE_RAR))
+		rar_cpu_init();
+
 	load_fixmap_gdt(cpu);
 }
 
arch/x86/mm/Makefile +1 -0
index 5b9908f13dcf..f36fc99e8b10 100644
@@ -52,6 +52,7 @@ obj-$(CONFIG_ACPI_NUMA)		+= srat.o
 obj-$(CONFIG_X86_INTEL_MEMORY_PROTECTION_KEYS)	+= pkeys.o
 obj-$(CONFIG_RANDOMIZE_MEMORY)			+= kaslr.o
 obj-$(CONFIG_MITIGATION_PAGE_TABLE_ISOLATION)	+= pti.o
+obj-$(CONFIG_BROADCAST_TLB_FLUSH)		+= rar.o
 
 obj-$(CONFIG_X86_MEM_ENCRYPT)	+= mem_encrypt.o
 obj-$(CONFIG_AMD_MEM_ENCRYPT)	+= mem_encrypt_amd.o
arch/x86/mm/rar.c new file +217 -0

217 lines not on this page · raw message

Signature · 1 line
2.49.0
#6 lore raw

Re: [RFC PATCH v3 5/7] x86/mm: Introduce Remote Action Request

Just few small things that jump out…

Quoted 8 lines · Rik van Riel
arch/x86/mm/rar.c new file +6 -0
+void rar_cpu_init(void)
+{
+	u64 r;
+	u8 *bitmap;
+	int max_payloads;
+	int this_cpu = smp_processor_id();

arch/x86/mm/rar.c line 137

Quoted 10 lines
arch/x86/mm/rar.c new file +10 -0
+
+	cpumask_clear(&per_cpu(rar_cpu_mask, this_cpu));
+
+	/* The MSR contains N defining the max [0-N] rar payload slots. */
+	rdmsrl(MSR_IA32_RAR_INFO, r);
+	max_payloads = (r >> 32) + 1;
+
+	/* If this CPU supports less than RAR_MAX_PAYLOADS, lower our limit. */
+	if (max_payloads < rar_max_payloads)
+		rar_max_payloads = max_payloads;

Unless I am missing something, this looks very racy.

BTW: should rar_max_payloads be ro_after_init?

arch/x86/mm/rar.c line 140

Quoted 3 lines
arch/x86/mm/rar.c new file +3 -0
+	pr_info_once("RAR: support %d payloads\n", max_payloads);
+
+	bitmap = (u8 *)per_cpu(rar_action, this_cpu);

this_cpu_ptr() would be cleaner (here and when using rar_cpu_mask).

arch/x86/mm/rar.c line 151

Quoted 11 lines
arch/x86/mm/rar.c new file +11 -0
+	memset(bitmap, 0, RAR_MAX_PAYLOADS);
+	wrmsrl(MSR_IA32_RAR_ACT_VEC, (u64)virt_to_phys(bitmap));
+	wrmsrl(MSR_IA32_RAR_PAYLOAD_BASE, (u64)virt_to_phys(rar_payload));
+
+	/*
+	 * Allow RAR events to be processed while interrupts are disabled on
+	 * a target CPU. This prevents "pileups" where many CPUs are waiting
+	 * on one CPU that has IRQs blocked for too long, and should reduce
+	 * contention on the rar_payload table.
+	 */
+	r = RAR_CTRL_ENABLE | RAR_CTRL_IGNORE_IF;

Do we really need r ?

Quoted 2 lines
(quoted diff) +2 -0
+	wrmsrl(MSR_IA32_RAR_CTRL, r);
+}
#7 lore raw

Re: [RFC PATCH v3 5/7] x86/mm: Introduce Remote Action Request

Quoted 16 lines · Nadav Amit

Just few small things that jump out…

Quoted 14 lines · Rik van Riel
arch/x86/mm/rar.c new file +10 -0
+void rar_cpu_init(void)
+{
+	u64 r;
+	u8 *bitmap;
+	int max_payloads;
+	int this_cpu = smp_processor_id();
+
+	cpumask_clear(&per_cpu(rar_cpu_mask, this_cpu));
+
+	/* The MSR contains N defining the max [0-N] rar payload

slots. */

arch/x86/mm/rar.c new file +1 -0
+	rdmsrl(MSR_IA32_RAR_INFO, r);
Quoted 9 lines

arch/x86/mm/rar.c line 137

Quoted 6 lines
arch/x86/mm/rar.c new file +3 -0
+	max_payloads = (r >> 32) + 1;
+
+	/* If this CPU supports less than RAR_MAX_PAYLOADS, lower

our limit. */

arch/x86/mm/rar.c new file +2 -0
+	if (max_payloads < rar_max_payloads)
+		rar_max_payloads = max_payloads;

Unless I am missing something, this looks very racy.

All the CPUs in the system should support the same number rar_max_payloads, since they share the same rar_action table.

Quoted 7 lines

BTW: should rar_max_payloads be ro_after_init?

arch/x86/mm/rar.c line 140

Quoted 3 lines
arch/x86/mm/rar.c new file +3 -0
+	pr_info_once("RAR: support %d payloads\n", max_payloads);
+
+	bitmap = (u8 *)per_cpu(rar_action, this_cpu);

this_cpu_ptr() would be cleaner (here and when using rar_cpu_mask).

A CPU cannot start using the rar_action table until the wrmsl below.

That should ensure there is no race here.

Thank you for the cleanup ideas. I'll apply those for v4.

Quoted 4 lines
Quoted 4 lines
(quoted diff) +2 -0
+	/*
+	 * Allow RAR events to be processed while interrupts are

disabled on

(quoted diff) +1 -0
+	 * a target CPU. This prevents "pileups" where many CPUs
Quoted 8 lines

arch/x86/mm/rar.c line 151

Quoted 6 lines

are waiting

(quoted diff) +1 -0
+	 * on one CPU that has IRQs blocked for too long, and

should reduce

arch/x86/mm/rar.c new file +3 -0
+	 * contention on the rar_payload table.
+	 */
+	r = RAR_CTRL_ENABLE | RAR_CTRL_IGNORE_IF;

Do we really need r ?

I suppose not. The original code added another
value into r, but that hangs hard with today's
microcode :)

Quoted 5 lines
Quoted 3 lines
(quoted diff) +2 -0
+	wrmsrl(MSR_IA32_RAR_CTRL, r);
+}
Signature · 1 line
All Rights Reversed.
#8 lore raw

Re: [RFC PATCH v3 5/7] x86/mm: Introduce Remote Action Request

Quoted 33 lines · Rik van Riel
Quoted 19 lines · Nadav Amit

Just few small things that jump out…

Quoted 17 lines · Rik van Riel

On 5 Jun 2025, at 19:35, Rik van Riel <[email protected]> wrote:

arch/x86/mm/rar.c new file +10 -0
+void rar_cpu_init(void)
+{
+	u64 r;
+	u8 *bitmap;
+	int max_payloads;
+	int this_cpu = smp_processor_id();
+
+	cpumask_clear(&per_cpu(rar_cpu_mask, this_cpu));
+
+	/* The MSR contains N defining the max [0-N] rar payload

slots. */

arch/x86/mm/rar.c new file +1 -0
+	rdmsrl(MSR_IA32_RAR_INFO, r);
Quoted 9 lines

arch/x86/mm/rar.c line 137

Quoted 6 lines
arch/x86/mm/rar.c new file +3 -0
+	max_payloads = (r >> 32) + 1;
+
+	/* If this CPU supports less than RAR_MAX_PAYLOADS, lower

our limit. */

arch/x86/mm/rar.c new file +2 -0
+	if (max_payloads < rar_max_payloads)
+		rar_max_payloads = max_payloads;

Unless I am missing something, this looks very racy.

Quoted 4 lines

All the CPUs in the system should support the same number rar_max_payloads, since they share the same rar_action table.

Usually you don’t want even benign data-races because it might cause tools to shout for no reason. So you would want to assist both other people and tools such as KCSAN by marking such accesses with data_race().

However, I think perhaps the bigger issue is that you want to assume all cores have the same RAR settings, and right now it might be a bit inconsistent.

So you may want to do some initial checks on the BSP as for whether RAR is supported and what rar_max_payloads is (e.g., in bsp_init_intel() ). And then on each AP, in something like init_intel() you’d call setup_clear_cpu_cap() to disable RAR if any CPU's max_payloads is different than the BSP.

[ BTW: further regarding patch 4, it seems cleaner to call
rar_cpu_init() from Intel specific code like init_intel() ? ]

Just sharing my thoughts (and further clarifying them), Nadav

#9 lore raw

Re: [RFC PATCH v3 5/7] x86/mm: Introduce Remote Action Request

Quoted 4 lines · Nadav Amit

However, I think perhaps the bigger issue is that you want to assume all cores have the same RAR settings, and right now it might be a bit inconsistent.

The documentation strongly suggests that RAR_INFO.TableMaxIndex is architecturally determined, and always 64 on Sapphire Rapids.

I'm not sure we want code to handle a case the Intel documentation seems to suggest cannot exist. Maybe somebody from Intel has some firmer ideas here?

Quoted 6 lines

So you may want to do some initial checks on the BSP as for whether RAR is supported and what rar_max_payloads is (e.g., in bsp_init_intel() ). And then on each AP, in something like init_intel() you’d call setup_clear_cpu_cap() to disable RAR if any CPU's max_payloads is different than the BSP.

Grabbing the value of rar_max_payloads from early_init_intel() might make sense, since it is supposed to be a system-wide value, and not a per CPU thing.

That seems like it would solve data race issues?

Quoted 4 lines

[ BTW: further regarding patch 4, it seems cleaner to call rar_cpu_init() from Intel specific code like init_intel() ? ]

Good idea, I'll move the call there.

Quoted 2 lines

Just sharing my thoughts (and further clarifying them),

I appreciate the comments and suggestions!

Now that the code seems to (finally) work reliably, I should have a faster turnaround time incorporating people's suggestions, too.

Signature · 1 line
All Rights Reversed.
#10 lore raw

Re: [RFC PATCH v3 5/7] x86/mm: Introduce Remote Action Request

Quoted 12 lines · Rik van Riel
Quoted 6 lines · Nadav Amit

So you may want to do some initial checks on the BSP as for whether RAR is supported and what rar_max_payloads is (e.g., in bsp_init_intel() ). And then on each AP, in something like init_intel() you’d call setup_clear_cpu_cap() to disable RAR if any CPU's max_payloads is different than the BSP.

Quoted 6 lines

Grabbing the value of rar_max_payloads from early_init_intel() might make sense, since it is supposed to be a system-wide value, and not a per CPU thing.

That seems like it would solve data race issues?

early_init_intel() is called from init_intel() so it would be called both for the BSP and the APs; IOW, it won’t solve the race.

Looking again at the code, I think that you should be able to assume this data is the same on all cores and move the X86_FEATURE_RAR enabling and rar_max_payloads setting into intel_detect_tlb() which appears both to be the most logical place to put it and only runs on the BSP.

#11 lore raw

RFC PATCH v3 6/7 x86/mm: use RAR for kernel TLB flushes

In reply to Rik van Riel

From: Rik van Riel <[email protected]>

Use Intel RAR for kernel TLB flushes, when enabled.

Pass in PCID 0 to smp_call_rar_many() to flush the specified addresses, regardless of which PCID they might be cached under in any destination CPU.

---
 arch/x86/mm/tlb.c | 38 ++++++++++++++++++++++++++++++++++++++
 1 file changed, 38 insertions(+)
arch/x86/mm/tlb.c +38 -0
index 39f80111e6f1..8931f7029d6c 100644
@@ -21,6 +21,7 @@
 #include <asm/apic.h>
 #include <asm/msr.h>
 #include <asm/perf_event.h>
+#include <asm/rar.h>
 #include <asm/tlb.h>
 
 #include "mm_internal.h"
@@ -1468,6 +1469,18 @@ static void do_flush_tlb_all(void *info)
 	__flush_tlb_all();
 }
 
+static void rar_full_flush(const cpumask_t *cpumask)
+{
+	guard(preempt)();
+	smp_call_rar_many(cpumask, 0, 0, TLB_FLUSH_ALL);
+	invpcid_flush_all();
+}
+
+static void rar_flush_all(void)
+{
+	rar_full_flush(cpu_online_mask);
+}
+
 void flush_tlb_all(void)
 {
 	count_vm_tlb_event(NR_TLB_REMOTE_FLUSH);
@@ -1475,6 +1488,8 @@ void flush_tlb_all(void)
 	/* First try (faster) hardware-assisted TLB invalidation. */
 	if (cpu_feature_enabled(X86_FEATURE_INVLPGB))
 		invlpgb_flush_all();
+	else if (cpu_feature_enabled(X86_FEATURE_RAR))
+		rar_flush_all();
 	else
 		/* Fall back to the IPI-based invalidation. */
 		on_each_cpu(do_flush_tlb_all, NULL, 1);
@@ -1504,15 +1519,36 @@ static void do_kernel_range_flush(void *info)
 	struct flush_tlb_info *f = info;
 	unsigned long addr;
 
+	/*
+	 * With PTI kernel TLB entries in all PCIDs need to be flushed.
+	 * With RAR the PCID space becomes so large, we might as well flush it all.
+	 *
+	 * Either of the two by itself works with targeted flushes.
+	 */
+	if (cpu_feature_enabled(X86_FEATURE_RAR) &&
+	    cpu_feature_enabled(X86_FEATURE_PTI)) {
+		invpcid_flush_all();
+		return;
+	}
+
 	/* flush range by one by one 'invlpg' */
 	for (addr = f->start; addr < f->end; addr += PAGE_SIZE)
 		flush_tlb_one_kernel(addr);
 }
 
+static void rar_kernel_range_flush(struct flush_tlb_info *info)
+{
+	guard(preempt)();
+	smp_call_rar_many(cpu_online_mask, 0, info->start, info->end);
+	do_kernel_range_flush(info);
+}
+
 static void kernel_tlb_flush_all(struct flush_tlb_info *info)
 {
 	if (cpu_feature_enabled(X86_FEATURE_INVLPGB))
 		invlpgb_flush_all();
+	else if (cpu_feature_enabled(X86_FEATURE_RAR))
+		rar_flush_all();
 	else
 		on_each_cpu(do_flush_tlb_all, NULL, 1);
 }
@@ -1521,6 +1557,8 @@ static void kernel_tlb_flush_range(struct flush_tlb_info *info)
 {
 	if (cpu_feature_enabled(X86_FEATURE_INVLPGB))
 		invlpgb_kernel_range_flush(info);
+	else if (cpu_feature_enabled(X86_FEATURE_RAR))
+		rar_kernel_range_flush(info);
 	else
 		on_each_cpu(do_kernel_range_flush, info, 1);
 }
Signature · 1 line
2.49.0
#12 lore raw

RFC PATCH v3 7/7 x86/mm: userspace & pageout flushing using Intel RAR

In reply to Rik van Riel

From: Rik van Riel <[email protected]>

Use Intel RAR to flush userspace mappings.

Because RAR flushes are targeted using a cpu bitmap, the rules are a little bit different than for true broadcast TLB invalidation.

For true broadcast TLB invalidation, like done with AMD INVLPGB, a global ASID always has up to date TLB entries on every CPU. The context switch code never has to flush the TLB when switching to a global ASID on any CPU with INVLPGB.

For RAR, the TLB mappings for a global ASID are kept up to date only on CPUs within the mm_cpumask, which lazily follows the threads around the system. The context switch code does not need to flush the TLB if the CPU is in the mm_cpumask, and the PCID used stays the same.

However, a CPU that falls outside of the mm_cpumask can have out of date TLB mappings for this task. When switching to that task on a CPU not in the mm_cpumask, the TLB does need to be flushed.

---
 arch/x86/include/asm/tlbflush.h |   9 +-
 arch/x86/mm/tlb.c               | 177 ++++++++++++++++++++++++--------
 2 files changed, 141 insertions(+), 45 deletions(-)
arch/x86/include/asm/tlbflush.h +6 -3
index e9b81876ebe4..21bd9162df38 100644
@@ -250,7 +250,8 @@ static inline u16 mm_global_asid(struct mm_struct *mm)
 {
 	u16 asid;
 
-	if (!cpu_feature_enabled(X86_FEATURE_INVLPGB))
+	if (!cpu_feature_enabled(X86_FEATURE_INVLPGB) &&
+	    !cpu_feature_enabled(X86_FEATURE_RAR))
 		return 0;
 
 	asid = smp_load_acquire(&mm->context.global_asid);
@@ -263,7 +264,8 @@ static inline u16 mm_global_asid(struct mm_struct *mm)
 
 static inline void mm_init_global_asid(struct mm_struct *mm)
 {
-	if (cpu_feature_enabled(X86_FEATURE_INVLPGB)) {
+	if (cpu_feature_enabled(X86_FEATURE_INVLPGB) ||
+	    cpu_feature_enabled(X86_FEATURE_RAR)) {
 		mm->context.global_asid = 0;
 		mm->context.asid_transition = false;
 	}
@@ -287,7 +289,8 @@ static inline void mm_clear_asid_transition(struct mm_struct *mm)
 
 static inline bool mm_in_asid_transition(struct mm_struct *mm)
 {
-	if (!cpu_feature_enabled(X86_FEATURE_INVLPGB))
+	if (!cpu_feature_enabled(X86_FEATURE_INVLPGB) &&
+	    !cpu_feature_enabled(X86_FEATURE_RAR))
 		return false;
 
 	return mm && READ_ONCE(mm->context.asid_transition);
arch/x86/mm/tlb.c +135 -42

345 lines not on this page · raw message

Signature · 1 line
2.49.0
#13 lore raw
In reply to Rik van Riel
Quoted 21 lines · Rik van Riel

This patch series adds support for IPI-less TLB invalidation using Intel RAR technology.

Intel RAR differs from AMD INVLPGB in a few ways:
- RAR goes through (emulated?) APIC writes, not instructions
- RAR flushes go through a memory table with 64 entries
- RAR flushes can be targeted to a cpumask
- The RAR functionality must be set up at boot time before it can be used

The cpumask targeting has resulted in Intel RAR and AMD INVLPGB having slightly different rules:
- Processes with dynamic ASIDs use IPI based shootdowns
- INVLPGB: processes with a global ASID
- always have the TLB up to date, on every CPU
- never need to flush the TLB at context switch time
- RAR: processes with global ASIDs
- have the TLB up to date on CPUs in the mm_cpumask
- can skip a TLB flush at context switch time if the CPU is in the mm_cpumask
- need to flush the TLB when scheduled on a cpu not in the mm_cpumask, in case it used to run there before and the TLB has stale entries

Quoted 7 lines

RAR functionality is present on Sapphire Rapids and newer CPUs.

Information about Intel RAR can be found in this whitepaper.

https://www.intel.com/content/dam/develop/external/us/en/documents/341431-remote-action-request-white-paper.pdf

This patch series is based off a 2019 patch series created by Intel

Do you have performance numbers? IIRC, the reason that 2019 series never went anywhere is because RAR wasn't a win for bare metal. Though I believe it _was_ a win for KVM, due to the cost of an IPI being significantly higher (requires a VM-Exit => VM-Enter roundtrip).

Quoted 3 lines

with patches later in the series modified to fit into
the TLB flush code structure we have after AMD INVLPGB functionality was integrated.

Please provide diff stats in the cover letter, e.g. so that sub-maintainers like myself don't have to scroll through every patch to figure out whether or not there's something that requires their review/ack.

#14 lore raw
Quoted 5 lines · Sean Christopherson
Quoted 1 line

This patch series is based off a 2019 patch series created by Intel

Do you have performance numbers? IIRC, the reason that 2019 series never went anywhere is because RAR wasn't a win for bare metal. Though I believe it _was_ a win for KVM, due to the cost of an IPI being significantly higher (requires a VM-Exit => VM-Enter roundtrip).

I'd love to see some performance numbers too.

But the real problem with the 2019 series was that it didn't have a global PCID space at all, only the per-cpu space. That greatly limited how RAR could be used.

I suspect now that Rik added the global space for INVLPGB that we'll have better luck with RAR.