Skip to contentSkip to conversation

Intel RAR TLB invalidation

RFC PATCH Superseded Rik van Riel 8 participants 36 messages May 20, 2025 – Jun 4, 2025

Subsystems (3)

v2 of 5 Superseded Latest is v5 →

lore
Share & tools
lore
#0 lore raw

v2 cover Intel RAR TLB invalidation

This patch series adds support for IPI-less TLB invalidation using Intel RAR technology.

Intel RAR differs from AMD INVLPGB in a few ways:
- RAR goes through (emulated?) APIC writes, not instructions
- RAR flushes go through a memory table with 64 entries
- RAR flushes can be targeted to a cpumask
- The RAR functionality must be set up at boot time before it can be used

The cpumask targeting has resulted in Intel RAR and AMD INVLPGB having slightly different rules:
- Processes with dynamic ASIDs use IPI based shootdowns
- INVLPGB: processes with a global ASID
- always have the TLB up to date, on every CPU
- never need to flush the TLB at context switch time
- RAR: processes with global ASIDs
- have the TLB up to date on CPUs in the mm_cpumask
- can skip a TLB flush at context switch time if the CPU is in the mm_cpumask
- need to flush the TLB when scheduled on a cpu not in the mm_cpumask, in case it used to run there before and the TLB has stale entries

RAR functionality is present on Sapphire Rapids and newer CPUs.

Information about Intel RAR can be found in this whitepaper.

https://www.intel.com/content/dam/develop/external/us/en/documents/341431-remote-action-request-white-paper.pdf

This patch series is based off a 2019 patch series created by Intel, with patches later in the series modified to fit into the TLB flush code structure we have after AMD INVLPGB functionality was integrated.

RFC v2:
- Cleanups suggested by Ingo and Nadav (thank you)
- Basic RAR code seems to actually work now
- Kernel TLB flushes with RAR seem to work correctly
- User TLB flushes with RAR are still broken, with two symptoms:
- The !is_lazy WARN_ON in leave_mm() is tripped
- Random segfaults

#1 lore raw

RFC v2 1/9 x86/mm: Introduce MSR_IA32_CORE_CAPABILITIES

From: Yu-cheng Yu <[email protected]>

MSR_IA32_CORE_CAPABILITIES indicates the existence of other MSRs. Bit[1] indicates Remote Action Request (RAR) TLB registers.

---
 arch/x86/include/asm/msr-index.h | 6 ++++++
 1 file changed, 6 insertions(+)
arch/x86/include/asm/msr-index.h +6 -0
index b7dded3c8113..c848dd4bfceb 100644
@@ -220,6 +220,12 @@
 						     * their affected status.
 						     */
 
+#define MSR_IA32_CORE_CAPABILITIES	0x000000cf
+#define CORE_CAP_RAR			BIT(1)	/*
+						 * Remote Action Request. Used to directly
+						 * flush the TLB on remote CPUs.
+						 */
+
 #define MSR_IA32_FLUSH_CMD		0x0000010b
 #define L1D_FLUSH			BIT(0)	/*
 						 * Writeback and invalidate the
Signature · 1 line
2.49.0
#2 lore raw

Re: [RFC v2 1/9] x86/mm: Introduce MSR_IA32_CORE_CAPABILITIES

Quoted 2 lines · Rik van Riel

MSR_IA32_CORE_CAPABILITIES indicates the existence of other MSRs. Bit[1] indicates Remote Action Request (RAR) TLB registers.

Nit: This may have changed from when Yu-cheng wrote the changelog, but RAR can do more than just flush the TLB. This probably needs to get a refresh.

#3 lore raw

Re: [RFC v2 1/9] x86/mm: Introduce MSR_IA32_CORE_CAPABILITIES

In reply to Rik van Riel
Quoted 16 lines · Rik van Riel

From: Yu-cheng Yu <[email protected]>

MSR_IA32_CORE_CAPABILITIES indicates the existence of other MSRs. Bit[1] indicates Remote Action Request (RAR) TLB registers.

---
 arch/x86/include/asm/msr-index.h | 6 ++++++
 1 file changed, 6 insertions(+)
arch/x86/include/asm/msr-index.h +0 -0
index b7dded3c8113..c848dd4bfceb 100644

arch/x86/include/asm/msr-index.h line 227

Quoted 8 lines
arch/x86/include/asm/msr-index.h +5 -0
@@ -220,6 +220,12 @@
 						     * their affected status.
 						     */
 
+#define MSR_IA32_CORE_CAPABILITIES	0x000000cf
+#define CORE_CAP_RAR			BIT(1)	/*
+						 * Remote Action Request. Used to directly
+						 * flush the TLB on remote CPUs.
+						 */

CORE_CAPABILITIES is already supported and enumerated, it's just abbreviated:

/* Abbreviated from Intel SDM name IA32_CORE_CAPABILITIES */
#define MSR_IA32_CORE_CAPS 0x000000cf
#define MSR_IA32_CORE_CAPS_INTEGRITY_CAPS_BIT 2
#define MSR_IA32_CORE_CAPS_INTEGRITY_CAPS BIT(MSR_IA32_CORE_CAPS_INTEGRITY_CAPS_BIT)
#define MSR_IA32_CORE_CAPS_SPLIT_LOCK_DETECT_BIT 5
#define MSR_IA32_CORE_CAPS_SPLIT_LOCK_DETECT BIT(MSR_IA32_CORE_CAPS_SPLIT_LOCK_DETECT_BIT)

#4 lore raw

RFC v2 2/9 x86/mm: Introduce Remote Action Request MSRs

In reply to Rik van Riel

From: Yu-cheng Yu <[email protected]>

Remote Action Request (RAR) is a TLB flushing broadcast facility. This patch introduces RAR MSRs. RAR is introduced in later patches.

There are five RAR MSRs:

MSR_CORE_CAPABILITIES
MSR_IA32_RAR_CTRL
MSR_IA32_RAR_ACT_VEC
MSR_IA32_RAR_PAYLOAD_BASE
MSR_IA32_RAR_INFO

---
 arch/x86/include/asm/msr-index.h | 11 +++++++++++
 1 file changed, 11 insertions(+)
arch/x86/include/asm/msr-index.h +11 -0
index c848dd4bfceb..adff8f0dc7bb 100644
@@ -122,6 +122,17 @@
 #define SNB_C3_AUTO_UNDEMOTE		(1UL << 27)
 #define SNB_C1_AUTO_UNDEMOTE		(1UL << 28)
 
+/*
+ * Remote Action Requests (RAR) MSRs
+ */
+#define MSR_IA32_RAR_CTRL		0x000000ed
+#define MSR_IA32_RAR_ACT_VEC		0x000000ee
+#define MSR_IA32_RAR_PAYLOAD_BASE	0x000000ef
+#define MSR_IA32_RAR_INFO		0x000000f0
+
+#define RAR_CTRL_ENABLE			BIT(31)
+#define RAR_CTRL_IGNORE_IF		BIT(30)
+
 #define MSR_MTRRcap			0x000000fe
 
 #define MSR_IA32_ARCH_CAPABILITIES	0x0000010a
Signature · 1 line
2.49.0
maintainer
#5 lore raw

Re: [RFC v2 2/9] x86/mm: Introduce Remote Action Request MSRs

Quoted 13 lines · Rik van Riel

From: Yu-cheng Yu <[email protected]>

Remote Action Request (RAR) is a TLB flushing broadcast facility. This patch introduces RAR MSRs. RAR is introduced in later patches.

There are five RAR MSRs:

MSR_CORE_CAPABILITIES
MSR_IA32_RAR_CTRL
MSR_IA32_RAR_ACT_VEC
MSR_IA32_RAR_PAYLOAD_BASE
MSR_IA32_RAR_INFO

Quoted 5 lines
---
 arch/x86/include/asm/msr-index.h | 11 +++++++++++
 1 file changed, 11 insertions(+)

You can merge this one with the previous one.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#6 lore raw

RFC v2 3/9 x86/mm: enable BROADCAST_TLB_FLUSH on Intel, too

In reply to Rik van Riel

From: Rik van Riel <[email protected]>

Much of the code for Intel RAR and AMD INVLPGB is shared.

Place both under the same config option.

---
 arch/x86/Kconfig.cpu | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/Kconfig.cpu +1 -1
index f928cf6e3252..f9cdd145abba 100644
@@ -360,7 +360,7 @@ menuconfig PROCESSOR_SELECT
 
 config BROADCAST_TLB_FLUSH
 	def_bool y
-	depends on CPU_SUP_AMD && 64BIT
+	depends on (CPU_SUP_AMD || CPU_SUP_INTEL) && 64BIT
 
 config CPU_SUP_INTEL
 	default y
Signature · 1 line
2.49.0
#7 lore raw

RFC v2 4/9 x86/mm: Introduce X86_FEATURE_RAR

In reply to Rik van Riel

From: Rik van Riel <[email protected]>

Introduce X86_FEATURE_RAR and enumeration of the feature.

[riel: moved initialization to intel.c and disabling to Kconfig.cpufeatures]

---
 arch/x86/Kconfig.cpufeatures       |  4 ++++
 arch/x86/include/asm/cpufeatures.h |  2 +-
 arch/x86/kernel/cpu/common.c       | 13 +++++++++++++
 3 files changed, 18 insertions(+), 1 deletion(-)
arch/x86/Kconfig.cpufeatures +4 -0
index 250c10627ab3..7d459b5f47f7 100644
@@ -195,3 +195,7 @@ config X86_DISABLED_FEATURE_SEV_SNP
 config X86_DISABLED_FEATURE_INVLPGB
 	def_bool y
 	depends on !BROADCAST_TLB_FLUSH
+
+config X86_DISABLED_FEATURE_RAR
+	def_bool y
+	depends on !BROADCAST_TLB_FLUSH
arch/x86/include/asm/cpufeatures.h +1 -1
index 5b50e0e35129..0729c2d54109 100644
@@ -76,7 +76,7 @@
 #define X86_FEATURE_K8			( 3*32+ 4) /* Opteron, Athlon64 */
 #define X86_FEATURE_ZEN5		( 3*32+ 5) /* CPU based on Zen5 microarchitecture */
 #define X86_FEATURE_ZEN6		( 3*32+ 6) /* CPU based on Zen6 microarchitecture */
-/* Free                                 ( 3*32+ 7) */
+#define X86_FEATURE_RAR			( 3*32+ 7) /* Intel Remote Action Request */
 #define X86_FEATURE_CONSTANT_TSC	( 3*32+ 8) /* "constant_tsc" TSC ticks at a constant rate */
 #define X86_FEATURE_UP			( 3*32+ 9) /* "up" SMP kernel running on UP */
 #define X86_FEATURE_ART			( 3*32+10) /* "art" Always running timer (ART) */
arch/x86/kernel/cpu/common.c +13 -0
index 8feb8fd2957a..dd662c42f510 100644
@@ -1545,6 +1545,18 @@ static void __init cpu_set_bug_bits(struct cpuinfo_x86 *c)
 	setup_force_cpu_bug(X86_BUG_L1TF);
 }
 
+static void __init detect_rar(struct cpuinfo_x86 *c)
+{
+	u64 msr;
+
+	if (cpu_has(c, X86_FEATURE_CORE_CAPABILITIES)) {
+		rdmsrl(MSR_IA32_CORE_CAPABILITIES, msr);
+
+		if (msr & CORE_CAP_RAR)
+			setup_force_cpu_cap(X86_FEATURE_RAR);
+	}
+}
+
 /*
  * The NOPL instruction is supposed to exist on all CPUs of family >= 6;
  * unfortunately, that's not true in practice because of early VIA
@@ -1771,6 +1783,7 @@ static void __init early_identify_cpu(struct cpuinfo_x86 *c)
 		setup_clear_cpu_cap(X86_FEATURE_LA57);
 
 	detect_nopl();
+	detect_rar(c);
 }
 
 void __init init_cpu_devs(void)
Signature · 1 line
2.49.0
maintainer
#8 lore raw

Re: [RFC v2 4/9] x86/mm: Introduce X86_FEATURE_RAR

Quoted 7 lines · Rik van Riel

From: Rik van Riel <[email protected]>

Introduce X86_FEATURE_RAR and enumeration of the feature.

[riel: moved initialization to intel.c and disabling to Kconfig.cpufeatures]

I'm guessing Yu-cheng is the original author - that's expressed differently.

Quoted 56 lines
---
 arch/x86/Kconfig.cpufeatures       |  4 ++++
 arch/x86/include/asm/cpufeatures.h |  2 +-
 arch/x86/kernel/cpu/common.c       | 13 +++++++++++++
 3 files changed, 18 insertions(+), 1 deletion(-)
arch/x86/Kconfig.cpufeatures +4 -0
index 250c10627ab3..7d459b5f47f7 100644
@@ -195,3 +195,7 @@ config X86_DISABLED_FEATURE_SEV_SNP
 config X86_DISABLED_FEATURE_INVLPGB
 	def_bool y
 	depends on !BROADCAST_TLB_FLUSH
+
+config X86_DISABLED_FEATURE_RAR
+	def_bool y
+	depends on !BROADCAST_TLB_FLUSH
arch/x86/include/asm/cpufeatures.h +1 -1
index 5b50e0e35129..0729c2d54109 100644
@@ -76,7 +76,7 @@
 #define X86_FEATURE_K8			( 3*32+ 4) /* Opteron, Athlon64 */
 #define X86_FEATURE_ZEN5		( 3*32+ 5) /* CPU based on Zen5 microarchitecture */
 #define X86_FEATURE_ZEN6		( 3*32+ 6) /* CPU based on Zen6 microarchitecture */
-/* Free                                 ( 3*32+ 7) */
+#define X86_FEATURE_RAR			( 3*32+ 7) /* Intel Remote Action Request */
 #define X86_FEATURE_CONSTANT_TSC	( 3*32+ 8) /* "constant_tsc" TSC ticks at a constant rate */
 #define X86_FEATURE_UP			( 3*32+ 9) /* "up" SMP kernel running on UP */
 #define X86_FEATURE_ART			( 3*32+10) /* "art" Always running timer (ART) */
arch/x86/kernel/cpu/common.c +12 -0
index 8feb8fd2957a..dd662c42f510 100644
@@ -1545,6 +1545,18 @@ static void __init cpu_set_bug_bits(struct cpuinfo_x86 *c)
 	setup_force_cpu_bug(X86_BUG_L1TF);
 }
 
+static void __init detect_rar(struct cpuinfo_x86 *c)
+{
+	u64 msr;
+
+	if (cpu_has(c, X86_FEATURE_CORE_CAPABILITIES)) {
+		rdmsrl(MSR_IA32_CORE_CAPABILITIES, msr);
+
+		if (msr & CORE_CAP_RAR)
+			setup_force_cpu_cap(X86_FEATURE_RAR);
+	}
+}
+
 /*
  * The NOPL instruction is supposed to exist on all CPUs of family >= 6;
  * unfortunately, that's not true in practice because of early VIA

arch/x86/kernel/cpu/common.c static void __init early_identify_cpu(struct cpuinfo_x86 *c) · line 1787

Quoted 5 lines
arch/x86/kernel/cpu/common.c +1 -0
@@ -1771,6 +1783,7 @@ static void __init early_identify_cpu(struct cpuinfo_x86 *c)
 		setup_clear_cpu_cap(X86_FEATURE_LA57);
 
 	detect_nopl();
+	detect_rar(c);
 }

Move all this gunk into early_init_intel().

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#9 lore raw

Re: [RFC v2 4/9] x86/mm: Introduce X86_FEATURE_RAR

Quoted 5 lines · Borislav Petkov
Quoted 4 lines · Rik van Riel

From: Rik van Riel <[email protected]>

Introduce X86_FEATURE_RAR and enumeration of the feature.

Quoted 7 lines
Quoted 4 lines

[riel: moved initialization to intel.c and disabling to Kconfig.cpufeatures]

I'm guessing Yu-cheng is the original author - that's expressed differently.

I will fix that up!

Quoted 11 lines
Quoted 8 lines
(quoted diff) +0 -0
@@ -1771,6 +1783,7 @@ static void __init early_identify_cpu(struct
 cpuinfo_x86 *c)
  		setup_clear_cpu_cap(X86_FEATURE_LA57);


  detect_nopl();

arch/x86/kernel/cpu/common.c +1 -0
+	detect_rar(c);

 }

Move all this gunk into early_init_intel().

I had the same thought, and tried that already.

It didn't work.

Signature · 1 line
All Rights Reversed.
maintainer
#10 lore raw

Re: [RFC v2 4/9] x86/mm: Introduce X86_FEATURE_RAR

Quoted 3 lines · Rik van Riel

I had the same thought, and tried that already.

It didn't work.

Care to share why?

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#11 lore raw

Re: [RFC v2 4/9] x86/mm: Introduce X86_FEATURE_RAR

Quoted 7 lines · Borislav Petkov
Quoted 3 lines · Rik van Riel

I had the same thought, and tried that already.

It didn't work.

Care to share why?

It resulted in RAR not being properly initialized, and the system hanging when trying to use RAR to flush the TLB.

I don't remember exactly what sequence of events was happening here, maybe something with the boot CPU per-cpu RAR initialization being called (or not, due to X86_FEATURE_RAR not being set) before the systemwide initialization?

Signature · 1 line
All Rights Reversed.
maintainer
#12 lore raw

Re: [RFC v2 4/9] x86/mm: Introduce X86_FEATURE_RAR

Quoted 12 lines · Rik van Riel
Quoted 7 lines · Borislav Petkov
Quoted 3 lines · Rik van Riel

I had the same thought, and tried that already.

It didn't work.

Care to share why?

It resulted in RAR not being properly initialized, and the system hanging when trying to use RAR to flush the TLB.

Quoted 5 lines

I don't remember exactly what sequence of events was happening here, maybe something with the boot CPU per-cpu RAR initialization being called (or not, due to X86_FEATURE_RAR not being set) before the systemwide initialization?

I'm asking you to move it from this path to

                if (this_cpu->c_early_init)
                        this_cpu->c_early_init(c);

or

                if (this_cpu->c_bsp_init)
                        this_cpu->c_bsp_init(c);

a couple of lines above.

This doesn't change anything: you're still running it on the BSP once. So I don't see how any of the above confusion would happen.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#13 lore raw

RFC v2 5/9 x86/mm: Change cpa_flush() to call flush_kernel_range() directly

In reply to Rik van Riel

From: Rik van Riel <[email protected]>

The function cpa_flush() calls __flush_tlb_one_kernel() and flush_tlb_all().

Replacing that with a call to flush_tlb_kernel_range() allows cpa_flush() to make use of INVLPGB or RAR without any additional changes.

Initialize invlpgb_count_max to 1, since flush_tlb_kernel_range() can now be called before invlpgb_count_max has been initialized to the value read from CPUID.

[riel: remove now unused __cpa_flush_tlb]

---
 arch/x86/kernel/cpu/amd.c    |  2 +-
 arch/x86/mm/pat/set_memory.c | 20 +++++++-------------
 2 files changed, 8 insertions(+), 14 deletions(-)
arch/x86/kernel/cpu/amd.c +1 -1
index 93da466dfe2c..b2ad8d13211a 100644
@@ -31,7 +31,7 @@
 
 #include "cpu.h"
 
-u16 invlpgb_count_max __ro_after_init;
+u16 invlpgb_count_max __ro_after_init = 1;
 
 static inline int rdmsrq_amd_safe(unsigned msr, u64 *p)
 {
arch/x86/mm/pat/set_memory.c +7 -13
index 30ab4aced761..2454f5249329 100644
@@ -399,15 +399,6 @@ static void cpa_flush_all(unsigned long cache)
 	on_each_cpu(__cpa_flush_all, (void *) cache, 1);
 }
 
-static void __cpa_flush_tlb(void *data)
-{
-	struct cpa_data *cpa = data;
-	unsigned int i;
-
-	for (i = 0; i < cpa->numpages; i++)
-		flush_tlb_one_kernel(fix_addr(__cpa_addr(cpa, i)));
-}
-
 static int collapse_large_pages(unsigned long addr, struct list_head *pgtables);
 
 static void cpa_collapse_large_pages(struct cpa_data *cpa)
@@ -444,6 +435,7 @@ static void cpa_collapse_large_pages(struct cpa_data *cpa)
 
 static void cpa_flush(struct cpa_data *cpa, int cache)
 {
+	unsigned long start, end;
 	unsigned int i;
 
 	BUG_ON(irqs_disabled() && !early_boot_irqs_disabled);
@@ -453,10 +445,12 @@ static void cpa_flush(struct cpa_data *cpa, int cache)
 		goto collapse_large_pages;
 	}
 
-	if (cpa->force_flush_all || cpa->numpages > tlb_single_page_flush_ceiling)
-		flush_tlb_all();
-	else
-		on_each_cpu(__cpa_flush_tlb, cpa, 1);
+	start = fix_addr(__cpa_addr(cpa, 0));
+	end = fix_addr(__cpa_addr(cpa, cpa->numpages));
+	if (cpa->force_flush_all)
+		end = TLB_FLUSH_ALL;
+
+	flush_tlb_kernel_range(start, end);
 
 	if (!cache)
 		goto collapse_large_pages;
Signature · 1 line
2.49.0
maintainer
#14 lore raw

Re: [RFC v2 5/9] x86/mm: Change cpa_flush() to call flush_kernel_range() directly

Quoted 9 lines · Rik van Riel

From: Rik van Riel <[email protected]>

The function cpa_flush() calls __flush_tlb_one_kernel() and flush_tlb_all().

Replacing that with a call to flush_tlb_kernel_range() allows cpa_flush() to make use of INVLPGB or RAR without any additional changes.

Quoted 8 lines

Initialize invlpgb_count_max to 1, since flush_tlb_kernel_range() can now be called before invlpgb_count_max has been initialized to the value read from CPUID.

[riel: remove now unused __cpa_flush_tlb]

Please audit all your SOB chains.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#15 lore raw

Re: [RFC v2 5/9] x86/mm: Change cpa_flush() to call flush_kernel_range() directly

In reply to Rik van Riel
Quoted 6 lines · Rik van Riel

The function cpa_flush() calls __flush_tlb_one_kernel() and flush_tlb_all().

Replacing that with a call to flush_tlb_kernel_range() allows cpa_flush() to make use of INVLPGB or RAR without any additional changes.

Yeah, the pageattr.c flushing code has gone through some twists and turns over the years but it does indeed look like it has converged to be awfully close to the other flushing code. It used to do wbinvd() and a full flush, but the wbinvd() disappeared at some point.

I don't immediately see any downsides to doing this. You could probably even hoist this up to the top of the series. I think it's a good cleanup on its own.

Also, I'd make the point in the subject and changelog that this isn't just changing one function to call another, it's removing some duplicated functionality and consolidating it to existing common code.

Maybe this for the subject:

x86/mm: Have cpa_flush() use common TLB flushing infrastructure

One super nit:

arch/x86/mm/pat/set_memory.c static void cpa_flush(struct cpa_data *cpa, int cache) · line 449

Quoted 2 lines
arch/x86/mm/pat/set_memory.c +2 -0
+	start = fix_addr(__cpa_addr(cpa, 0));
+	end = fix_addr(__cpa_addr(cpa, cpa->numpages));

Please vertically align the fix_addr()'s.

#16 lore raw

RFC v2 6/9 x86/apic: Introduce Remote Action Request Operations

In reply to Rik van Riel

From: Rik van Riel <[email protected]>

RAR TLB flushing is started by sending a command to the APIC. This patch adds Remote Action Request commands.

[riel: move some things around to acount for 6 years of changes]

---
 arch/x86/include/asm/apicdef.h     |  1 +
 arch/x86/include/asm/irq_vectors.h |  5 +++++
 arch/x86/include/asm/smp.h         | 15 +++++++++++++++
 arch/x86/kernel/apic/ipi.c         | 23 +++++++++++++++++++----
 arch/x86/kernel/apic/local.h       |  3 +++
 arch/x86/kernel/smp.c              |  3 +++
 6 files changed, 46 insertions(+), 4 deletions(-)
arch/x86/include/asm/apicdef.h +1 -0
index 094106b6a538..b152d45af91a 100644
@@ -92,6 +92,7 @@
 #define		APIC_DM_LOWEST		0x00100
 #define		APIC_DM_SMI		0x00200
 #define		APIC_DM_REMRD		0x00300
+#define		APIC_DM_RAR		0x00300
 #define		APIC_DM_NMI		0x00400
 #define		APIC_DM_INIT		0x00500
 #define		APIC_DM_STARTUP		0x00600
arch/x86/include/asm/irq_vectors.h +5 -0
index 47051871b436..c417b0015304 100644
@@ -103,6 +103,11 @@
  */
 #define POSTED_MSI_NOTIFICATION_VECTOR	0xeb
 
+/*
+ * RAR (remote action request) TLB flush
+ */
+#define RAR_VECTOR			0xe0
+
 #define NR_VECTORS			 256
 
 #ifdef CONFIG_X86_LOCAL_APIC
arch/x86/include/asm/smp.h +15 -0
index 0c1c68039d6f..1ab9f5fcac8a 100644
@@ -40,6 +40,9 @@ struct smp_ops {
 
 	void (*send_call_func_ipi)(const struct cpumask *mask);
 	void (*send_call_func_single_ipi)(int cpu);
+
+	void (*send_rar_ipi)(const struct cpumask *mask);
+	void (*send_rar_single_ipi)(int cpu);
 };
 
 /* Globals due to paravirt */
@@ -100,6 +103,16 @@ static inline void arch_send_call_function_ipi_mask(const struct cpumask *mask)
 	smp_ops.send_call_func_ipi(mask);
 }
 
+static inline void arch_send_rar_single_ipi(int cpu)
+{
+	smp_ops.send_rar_single_ipi(cpu);
+}
+
+static inline void arch_send_rar_ipi_mask(const struct cpumask *mask)
+{
+	smp_ops.send_rar_ipi(mask);
+}
+
 void cpu_disable_common(void);
 void native_smp_prepare_boot_cpu(void);
 void smp_prepare_cpus_common(void);
@@ -120,6 +133,8 @@ void __noreturn mwait_play_dead(unsigned int eax_hint);
 void native_smp_send_reschedule(int cpu);
 void native_send_call_func_ipi(const struct cpumask *mask);
 void native_send_call_func_single_ipi(int cpu);
+void native_send_rar_ipi(const struct cpumask *mask);
+void native_send_rar_single_ipi(int cpu);
 
 asmlinkage __visible void smp_reboot_interrupt(void);
 __visible void smp_reschedule_interrupt(struct pt_regs *regs);
arch/x86/kernel/apic/ipi.c +19 -4
index 98a57cb4aa86..e5e9fc08f86c 100644
@@ -79,7 +79,7 @@ void native_send_call_func_single_ipi(int cpu)
 	__apic_send_IPI(cpu, CALL_FUNCTION_SINGLE_VECTOR);
 }
 
-void native_send_call_func_ipi(const struct cpumask *mask)
+static void do_native_send_ipi(const struct cpumask *mask, int vector)
 {
 	if (static_branch_likely(&apic_use_ipi_shorthand)) {
 		unsigned int cpu = smp_processor_id();
@@ -88,14 +88,19 @@ void native_send_call_func_ipi(const struct cpumask *mask)
 			goto sendmask;
 
 		if (cpumask_test_cpu(cpu, mask))
-			__apic_send_IPI_all(CALL_FUNCTION_VECTOR);
+			__apic_send_IPI_all(vector);
 		else if (num_online_cpus() > 1)
-			__apic_send_IPI_allbutself(CALL_FUNCTION_VECTOR);
+			__apic_send_IPI_allbutself(vector);
 		return;
 	}
 
 sendmask:
-	__apic_send_IPI_mask(mask, CALL_FUNCTION_VECTOR);
+	__apic_send_IPI_mask(mask, vector);
+}
+
+void native_send_call_func_ipi(const struct cpumask *mask)
+{
+	do_native_send_ipi(mask, CALL_FUNCTION_VECTOR);
 }
 
 void apic_send_nmi_to_offline_cpu(unsigned int cpu)
@@ -106,6 +111,16 @@ void apic_send_nmi_to_offline_cpu(unsigned int cpu)
 		return;
 	apic->send_IPI(cpu, NMI_VECTOR);
 }
+
+void native_send_rar_single_ipi(int cpu)
+{
+	apic->send_IPI_mask(cpumask_of(cpu), RAR_VECTOR);
+}
+
+void native_send_rar_ipi(const struct cpumask *mask)
+{
+	do_native_send_ipi(mask, RAR_VECTOR);
+}
 #endif /* CONFIG_SMP */
 
 static inline int __prepare_ICR2(unsigned int mask)
arch/x86/kernel/apic/local.h +3 -0
index bdcf609eb283..833669174267 100644
@@ -38,6 +38,9 @@ static inline unsigned int __prepare_ICR(unsigned int shortcut, int vector,
 	case NMI_VECTOR:
 		icr |= APIC_DM_NMI;
 		break;
+	case RAR_VECTOR:
+		icr |= APIC_DM_RAR;
+		break;
 	}
 	return icr;
 }
arch/x86/kernel/smp.c +3 -0
index 18266cc3d98c..2c51ed6aaf03 100644
@@ -297,5 +297,8 @@ struct smp_ops smp_ops = {
 
 	.send_call_func_ipi	= native_send_call_func_ipi,
 	.send_call_func_single_ipi = native_send_call_func_single_ipi,
+
+	.send_rar_ipi		= native_send_rar_ipi,
+	.send_rar_single_ipi	= native_send_rar_single_ipi,
 };
 EXPORT_SYMBOL_GPL(smp_ops);
Signature · 1 line
2.49.0
#17 lore raw

Re: [RFC v2 6/9] x86/apic: Introduce Remote Action Request Operations

Quoted 5 lines · Rik van Riel
arch/x86/include/asm/irq_vectors.h +0 -0
index 47051871b436..c417b0015304 100644

arch/x86/include/asm/irq_vectors.h line 111

Quoted 9 lines
arch/x86/include/asm/irq_vectors.h +5 -0
@@ -103,6 +103,11 @@
  */
 #define POSTED_MSI_NOTIFICATION_VECTOR	0xeb
 
+/*
+ * RAR (remote action request) TLB flush
+ */
+#define RAR_VECTOR			0xe0
+
 #define NR_VECTORS			 256

This subtly breaks x86 IRQ vector allocation AFAICS.

Right now device IRQ vectors are allocated from 0x81 to FIRST_SYSTEM_VECTOR (POSTED_MSI_NOTIFICATION_VECTOR) or 0xeb.

But RAR_VECTOR is within that range, the the IRQ allocator will overlap it and result in what I guess will be misbehaving RAR code and misbehaving device IRQ handling once it hands out 0xeb as well.

So you need to lower NR_EXTERNAL_VECTORS for there to be no overlap between device IRQ vectors and system IRQ vectors.

This will substantially compresses the available device vector space from ~108 vectors to ~95 vectors, a ~12% reduction. RAR, under the current device IRQ vector allocator, will effectively reduce the number of vectors not by 1 vector, but by 13 vectors. This should be pointed out in the changelog.

It probably doesn't matter much due to MSI multiplexing, but should nevertheless be implemented correctly and should be documented.

Thanks,

Ingo

#18 lore raw

Re: [RFC v2 6/9] x86/apic: Introduce Remote Action Request Operations

Quoted 21 lines · Ingo Molnar
Quoted 14 lines · Rik van Riel
arch/x86/include/asm/irq_vectors.h +0 -0
index 47051871b436..c417b0015304 100644
@@ -103,6 +103,11 @@
   */
  #define POSTED_MSI_NOTIFICATION_VECTOR	0xeb

arch/x86/include/asm/irq_vectors.h +5 -0
+/*
+ * RAR (remote action request) TLB flush
+ */
+#define RAR_VECTOR			0xe0
+
Quoted 1 line

 #define NR_VECTORS 256

This subtly breaks x86 IRQ vector allocation AFAICS.

Quoted 7 lines

Right now device IRQ vectors are allocated from 0x81 to FIRST_SYSTEM_VECTOR (POSTED_MSI_NOTIFICATION_VECTOR) or 0xeb.

But RAR_VECTOR is within that range, the the IRQ allocator will overlap
it and result in what I guess will be misbehaving RAR code and misbehaving device IRQ handling once it hands out 0xeb as well.

Sure enough! After fixing this issue, the nearly instant segfaults for programs using RAR are no longer happening.

I'll let it run tests overnight, and will hopefully be able to post a reliable v3 tomorrow.

Thank you!

Signature · 1 line
All Rights Reversed.
#19 lore raw

Re: [RFC v2 6/9] x86/apic: Introduce Remote Action Request Operations

In reply to Rik van Riel

arch/x86/include/asm/smp.h struct smp_ops { · line 46

Quoted 12 lines
arch/x86/include/asm/smp.h +3 -0
index 0c1c68039d6f..1ab9f5fcac8a 100644
@@ -40,6 +40,9 @@ struct smp_ops {
 
 	void (*send_call_func_ipi)(const struct cpumask *mask);
 	void (*send_call_func_single_ipi)(int cpu);
+
+	void (*send_rar_ipi)(const struct cpumask *mask);
+	void (*send_rar_single_ipi)(int cpu);
 };

I assume Yu-cheng did it this way.

I'm curios why new smp_ops are needed for this, though. It's not like there are a bunch of different implementations to pick between.

Quoted 16 lines
arch/x86/include/asm/smp.h +1 -1
-void native_send_call_func_ipi(const struct cpumask *mask)
+static void do_native_send_ipi(const struct cpumask *mask, int vector)
 {
arch/x86/include/asm/smp.h +2 -2
 	if (static_branch_likely(&apic_use_ipi_shorthand)) {
 		unsigned int cpu = smp_processor_id();
@@ -88,14 +88,19 @@ void native_send_call_func_ipi(const struct cpumask *mask)
 			goto sendmask;
 
 		if (cpumask_test_cpu(cpu, mask))
-			__apic_send_IPI_all(CALL_FUNCTION_VECTOR);
+			__apic_send_IPI_all(vector);
 		else if (num_online_cpus() > 1)
-			__apic_send_IPI_allbutself(CALL_FUNCTION_VECTOR);
+			__apic_send_IPI_allbutself(vector);
 		return;
 	}

arch/x86/include/asm/smp.h void native_send_call_func_ipi(const struct cpumask *mask) · line 104

Quoted 10 lines
arch/x86/include/asm/smp.h +6 -1
 
 sendmask:
-	__apic_send_IPI_mask(mask, CALL_FUNCTION_VECTOR);
+	__apic_send_IPI_mask(mask, vector);
+}
+
+void native_send_call_func_ipi(const struct cpumask *mask)
+{
+	do_native_send_ipi(mask, CALL_FUNCTION_VECTOR);
 }

This refactoring probably belongs in a separate patch.

Quoted 23 lines
arch/x86/include/asm/smp.h +10 -0
 void apic_send_nmi_to_offline_cpu(unsigned int cpu)
@@ -106,6 +111,16 @@ void apic_send_nmi_to_offline_cpu(unsigned int cpu)
 		return;
 	apic->send_IPI(cpu, NMI_VECTOR);
 }
+
+void native_send_rar_single_ipi(int cpu)
+{
+	apic->send_IPI_mask(cpumask_of(cpu), RAR_VECTOR);
+}
+
+void native_send_rar_ipi(const struct cpumask *mask)
+{
+	do_native_send_ipi(mask, RAR_VECTOR);
+}
 #endif /* CONFIG_SMP */
 
 static inline int __prepare_ICR2(unsigned int mask)
arch/x86/kernel/apic/local.h +0 -0
index bdcf609eb283..833669174267 100644

arch/x86/kernel/apic/local.h static inline unsigned int __prepare_ICR(unsigned int shortcut, int vector, · line 46

Quoted 9 lines
arch/x86/kernel/apic/local.h +3 -0
@@ -38,6 +38,9 @@ static inline unsigned int __prepare_ICR(unsigned int shortcut, int vector,
 	case NMI_VECTOR:
 		icr |= APIC_DM_NMI;
 		break;
+	case RAR_VECTOR:
+		icr |= APIC_DM_RAR;
+		break;
 	}
 	return icr;
 }

I feel like this patch is doing three separate things:

1. Adds smp_ops
2. Refactors native_send_call_func_ipi()
3. Adds RAR support

None of those are huge, but it would make a lot more sense to break those out. I'm also still not sure of the point of the smp_ops.

#20 lore raw

Re: [RFC v2 6/9] x86/apic: Introduce Remote Action Request Operations

Quoted 16 lines · Dave Hansen
Quoted 12 lines
arch/x86/include/asm/smp.h +0 -0
index 0c1c68039d6f..1ab9f5fcac8a 100644
@@ -40,6 +40,9 @@ struct smp_ops {


  void (*send_call_func_ipi)(const struct cpumask *mask); void (*send_call_func_single_ipi)(int cpu);

arch/x86/include/asm/smp.h +3 -0
+
+	void (*send_rar_ipi)(const struct cpumask *mask);
+	void (*send_rar_single_ipi)(int cpu);
Quoted 1 line

 };

I assume Yu-cheng did it this way.

Quoted 3 lines

I'm curios why new smp_ops are needed for this, though. It's not like there are a bunch of different implementations to pick between.

You are right, this was in the code I received.

Quoted 9 lines

I feel like this patch is doing three separate things:

1. Adds smp_ops
2. Refactors native_send_call_func_ipi()
3. Adds RAR support

None of those are huge, but it would make a lot more sense to break those out. I'm also still not sure of the point of the smp_ops.

I am not very familiar with this part of the kernel, but would be happy to make whatever changes the maintainers want to see.

Signature · 1 line
All Rights Reversed.
#21 lore raw

RFC v2 7/9 x86/mm: Introduce Remote Action Request

In reply to Rik van Riel

From: Yu-cheng Yu <[email protected]>

Remote Action Request (RAR) is a TLB flushing broadcast facility. To start a TLB flush, the initiator CPU creates a RAR payload and sends a command to the APIC. The receiving CPUs automatically flush TLBs as specified in the payload without the kernel's involement.

[ riel: add pcid parameter to smp_call_rar_many so other mms can be flushed ]

---
 arch/x86/include/asm/rar.h   |  69 +++++++++++++
 arch/x86/kernel/cpu/common.c |   4 +
 arch/x86/mm/Makefile         |   1 +
 arch/x86/mm/rar.c            | 195 +++++++++++++++++++++++++++++++++++
 4 files changed, 269 insertions(+)
 create mode 100644 arch/x86/include/asm/rar.h
 create mode 100644 arch/x86/mm/rar.c
arch/x86/include/asm/rar.h new file +69 -0
new file mode 100644
index 000000000000..78c039e40e81
@@ -0,0 +1,69 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _ASM_X86_RAR_H
+#define _ASM_X86_RAR_H
+
+/*
+ * RAR payload types
+ */
+#define RAR_TYPE_INVPG		0
+#define RAR_TYPE_INVPG_NO_CR3	1
+#define RAR_TYPE_INVPCID	2
+#define RAR_TYPE_INVEPT		3
+#define RAR_TYPE_INVVPID	4
+#define RAR_TYPE_WRMSR		5
+
+/*
+ * Subtypes for RAR_TYPE_INVLPG
+ */
+#define RAR_INVPG_ADDR			0 /* address specific */
+#define RAR_INVPG_ALL			2 /* all, include global */
+#define RAR_INVPG_ALL_NO_GLOBAL		3 /* all, exclude global */
+
+/*
+ * Subtypes for RAR_TYPE_INVPCID
+ */
+#define RAR_INVPCID_ADDR		0 /* address specific */
+#define RAR_INVPCID_PCID		1 /* all of PCID */
+#define RAR_INVPCID_ALL			2 /* all, include global */
+#define RAR_INVPCID_ALL_NO_GLOBAL	3 /* all, exclude global */
+
+/*
+ * Page size for RAR_TYPE_INVLPG
+ */
+#define RAR_INVLPG_PAGE_SIZE_4K		0
+#define RAR_INVLPG_PAGE_SIZE_2M		1
+#define RAR_INVLPG_PAGE_SIZE_1G		2
+
+/*
+ * Max number of pages per payload
+ */
+#define RAR_INVLPG_MAX_PAGES 63
+
+struct rar_payload {
+	u64 for_sw		: 8;
+	u64 type		: 8;
+	u64 must_be_zero_1	: 16;
+	u64 subtype		: 3;
+	u64 page_size		: 2;
+	u64 num_pages		: 6;
+	u64 must_be_zero_2	: 21;
+
+	u64 must_be_zero_3;
+
+	/*
+	 * Starting address
+	 */
+	u64 initiator_cr3;
+	u64 linear_address;
+
+	/*
+	 * Padding
+	 */
+	u64 padding[4];
+};
+
+void rar_cpu_init(void);
+void smp_call_rar_many(const struct cpumask *mask, u16 pcid,
+		       unsigned long start, unsigned long end);
+
+#endif /* _ASM_X86_RAR_H */
arch/x86/kernel/cpu/common.c +4 -0
index dd662c42f510..b1e1b9afb2ac 100644
@@ -71,6 +71,7 @@
 #include <asm/tdx.h>
 #include <asm/posted_intr.h>
 #include <asm/runtime-const.h>
+#include <asm/rar.h>
 
 #include "cpu.h"
 
@@ -2438,6 +2439,9 @@ void cpu_init(void)
 	if (is_uv_system())
 		uv_cpu_init();
 
+	if (cpu_feature_enabled(X86_FEATURE_RAR))
+		rar_cpu_init();
+
 	load_fixmap_gdt(cpu);
 }
 
arch/x86/mm/Makefile +1 -0
index 5b9908f13dcf..f36fc99e8b10 100644
@@ -52,6 +52,7 @@ obj-$(CONFIG_ACPI_NUMA)		+= srat.o
 obj-$(CONFIG_X86_INTEL_MEMORY_PROTECTION_KEYS)	+= pkeys.o
 obj-$(CONFIG_RANDOMIZE_MEMORY)			+= kaslr.o
 obj-$(CONFIG_MITIGATION_PAGE_TABLE_ISOLATION)	+= pti.o
+obj-$(CONFIG_BROADCAST_TLB_FLUSH)		+= rar.o
 
 obj-$(CONFIG_X86_MEM_ENCRYPT)	+= mem_encrypt.o
 obj-$(CONFIG_AMD_MEM_ENCRYPT)	+= mem_encrypt_amd.o
arch/x86/mm/rar.c new file +195 -0

195 lines not on this page · raw message

Signature · 1 line
2.49.0
#22 lore raw

Re: [RFC v2 7/9] x86/mm: Introduce Remote Action Request

Quoted 8 lines · Rik van Riel

From: Yu-cheng Yu <[email protected]>

Remote Action Request (RAR) is a TLB flushing broadcast facility. To start a TLB flush, the initiator CPU creates a RAR payload and sends a command to the APIC. The receiving CPUs automatically flush TLBs as specified in the payload without the kernel's involement.

[ riel: add pcid parameter to smp_call_rar_many so other mms can be flushed ]

Please actually review & tidy up patches that you pass through, don't just hack them up minimally and slap your tag and SOB on top of it.

One example, of many:

arch/x86/mm/rar.c line 141

Quoted 1 line
arch/x86/mm/rar.c new file +1 -0
+	 * We allow cpu's that are not yet online though, as no one else can

Here the comment has 'CPU' in lowercase, and with a grammar mistake.

arch/x86/mm/rar.c line 142

Quoted 1 line
arch/x86/mm/rar.c new file +1 -0
+	 * send smp call function interrupt to this cpu and as such deadlocks

Here 'CPU' is in lowercase.

arch/x86/mm/rar.c line 148

Quoted 1 line
arch/x86/mm/rar.c new file +1 -0
+	/* Try to fastpath.  So, what's a CPU they want?  Ignoring this one. */

Oh, here 'CPU' is uppercase again! What happened?

arch/x86/mm/rar.c line 153

Quoted 1 line
arch/x86/mm/rar.c new file +1 -0
+	/* No online cpus?  We're done. */

Lowercase again. Damn, I thought we settled on a way to spell this thing already.

arch/x86/mm/rar.c line 157

Quoted 1 line
arch/x86/mm/rar.c new file +1 -0
+	/* Do we have another CPU which isn't us? */

And uppercase. What a roller-coaster.

arch/x86/mm/rar.c line 177

Quoted 2 lines
arch/x86/mm/rar.c new file +2 -0
+	/* Fastpath: do that cpu by itself. */
+	/* Some callers race with other cpus changing the passed mask */

And lowercase.

arch/x86/mm/rar.c line 187

Quoted 1 line
arch/x86/mm/rar.c new file +1 -0
+	/* Send a message to all CPUs in the map */

And uppercase.

It's almost as if nobody has ever read these comments after writing them.

There's like a zillion small random-noise details through the entire series that insert unnecessary extra white noise in critical system code that should be a lot more carefully written, which emits a foul aura of carelessness. Reviewers should not be forced to point these out to you, in fact reviewers should not be exposed to such noise at all.

Please review the entire thing *much* more carefully before submitting -v3.

Thanks,

Ingo

#23 lore raw

Re: [RFC v2 7/9] x86/mm: Introduce Remote Action Request

Quoted 11 lines · Ingo Molnar
Quoted 8 lines · Rik van Riel

From: Yu-cheng Yu <[email protected]>

Remote Action Request (RAR) is a TLB flushing broadcast facility. To start a TLB flush, the initiator CPU creates a RAR payload and sends a command to the APIC.  The receiving CPUs automatically flush
TLBs as specified in the payload without the kernel's involement.

Quoted 5 lines
Quoted 2 lines

[ riel: add pcid parameter to smp_call_rar_many so other mms can be flushed ]

Please actually review & tidy up patches that you pass through, don't just hack them up minimally and slap your tag and SOB on top of it.

I'm happy to do that now that the code is finally working.

v3 will have much more cleanups, and hopefully a few optimizations.

Signature · 1 line
All Rights Reversed.
#24 lore raw

Re: [RFC v2 7/9] x86/mm: Introduce Remote Action Request

Quoted 1 line · Rik van Riel

v3 will have much more cleanups, and hopefully a few optimizations.

Thanks!

Ingo

#25 lore raw

Re: [RFC v2 7/9] x86/mm: Introduce Remote Action Request

In reply to Rik van Riel

Not a full review, but..

Quoted 4 lines · Rik van Riel

+/*
+ * This is a modified version of smp_call_function_many() of kernel/smp.c,

The updated function names is smp_call_function_many_cond() and it is not aligned with smp_call_rar_many. I think the new version is (suprisingly) better, so it’d be beneficial to bring smp_call_rar_many() to be like the updated one in smp.c.

Quoted 11 lines

+ * without a function pointer, because the RAR handler is the ucode.
+ */

arch/x86/mm/rar.c new file +9 -0
+void smp_call_rar_many(const struct cpumask *mask, u16 pcid,
+		       unsigned long start, unsigned long end)
+{
+	unsigned long pages = (end - start + PAGE_SIZE) / PAGE_SIZE;
+	int cpu, next_cpu, this_cpu = smp_processor_id();
+	cpumask_t *dest_mask;
+	unsigned long idx;
+
+	if (pages > RAR_INVLPG_MAX_PAGES || end == TLB_FLUSH_ALL)

arch/x86/mm/rar.c line 146

Quoted 10 lines
arch/x86/mm/rar.c new file +10 -0
+		pages = RAR_INVLPG_MAX_PAGES;
+
+	/*
+	 * Can deadlock when called with interrupts disabled.
+	 * We allow cpu's that are not yet online though, as no one else can
+	 * send smp call function interrupt to this cpu and as such deadlocks
+	 * can't happen.
+	 */
+	WARN_ON_ONCE(cpu_online(this_cpu) && irqs_disabled()
+		     && !oops_in_progress && !early_boot_irqs_disabled);

I thought you agreed to change it to make it use lockdep instead (so it will be compiled out without LOCKDEP), like done in smp_call_function_many_cond()

arch/x86/mm/rar.c line 152

Quoted 6 lines
arch/x86/mm/rar.c new file +6 -0
+
+	/* Try to fastpath.  So, what's a CPU they want?  Ignoring this one. */
+	cpu = cpumask_first_and(mask, cpu_online_mask);
+	if (cpu == this_cpu)
+		cpu = cpumask_next_and(cpu, mask, cpu_online_mask);
+

Putting aside the rest of the code, I see you don’t call should_flush_tlb(). I think it is worth mentioning in commit log or comment the rationale behind it (and maybe benchmarks to justify it).

#26 lore raw

Re: [RFC v2 7/9] x86/mm: Introduce Remote Action Request

Quoted 8 lines · Nadav Amit

Not a full review, but..

Quoted 5 lines · Rik van Riel

+/*
+ * This is a modified version of smp_call_function_many() of kernel/smp.c,

Quoted 7 lines

The updated function names is smp_call_function_many_cond() and it is not aligned with smp_call_rar_many. I think the new version is (suprisingly)
better, so it’d be beneficial to bring smp_call_rar_many() to be like the
updated one in smp.c.

Agreed, it will be good to conditionally not send the RAR vector to some CPUs, especially ones that are in deeper idle states.

That means structuring the code more like
smp_call_function_many_cond()

Quoted 8 lines
Quoted 8 lines
arch/x86/mm/rar.c new file +3 -0
+	/*
+	 * Can deadlock when called with interrupts disabled.
+	 * We allow cpu's that are not yet online though, as no

one else can

(quoted diff) +1 -0
+	 * send smp call function interrupt to this cpu and as

such deadlocks

arch/x86/mm/rar.c new file +2 -0
+	 * can't happen.
+	 */
Quoted 9 lines
Quoted 3 lines
(quoted diff) +2 -0
+	WARN_ON_ONCE(cpu_online(this_cpu) && irqs_disabled()
+		     && !oops_in_progress &&

!early_boot_irqs_disabled);

I thought you agreed to change it to make it use lockdep instead (so it will
be compiled out without LOCKDEP), like done in smp_call_function_many_cond()

I thought I had made that change in my tree.

I guess I lost it in a rebase :(

Quoted 6 lines
Quoted 6 lines
(quoted diff) +2 -0
+
+	/* Try to fastpath.  So, what's a CPU they want?  Ignoring

this one. */

arch/x86/mm/rar.c new file +3 -0
+	cpu = cpumask_first_and(mask, cpu_online_mask);
+	if (cpu == this_cpu)
+		cpu = cpumask_next_and(cpu, mask,
Quoted 10 lines
Quoted 2 lines

cpu_online_mask);
+

Putting aside the rest of the code, I see you don’t call should_flush_tlb().
I think it is worth mentioning in commit log or comment the rationale behind
it (and maybe benchmarks to justify it).

The long term plan here is to simply have the originating CPU included in the cpumask, and have it send a RAR request to itself.

That way all the CPUs can invalidate their entries in parallel, without any extra code.

Signature · 1 line
All Rights Reversed.
#27 lore raw

Re: [RFC v2 7/9] x86/mm: Introduce Remote Action Request

Quoted 11 lines · Rik van Riel
Quoted 8 lines

Putting aside the rest of the code, I see you don’t call should_flush_tlb().
I think it is worth mentioning in commit log or comment the rationale behind
it (and maybe benchmarks to justify it).

Quoted 3 lines

The long term plan here is to simply have the originating CPU included in the cpumask, and have it send a RAR request to itself.

That’s unrelated. I was referring to considering supporting some sort of lazy TLB to eliminate sending RAR to cores that do not care about it. Is there a cost of RAR to more cores than needed? My guess is that there is one, and maybe in such cases you would want actual IPI and special handling.

#28 lore raw

Re: [RFC v2 7/9] x86/mm: Introduce Remote Action Request

Quoted 15 lines · Nadav Amit
Quoted 11 lines · Rik van Riel
Quoted 9 lines

Putting aside the rest of the code, I see you don’t call should_flush_tlb().
I think it is worth mentioning in commit log or comment the rationale
behind
it (and maybe benchmarks to justify it).

Quoted 3 lines

The long term plan here is to simply have the originating CPU included in the cpumask, and have it send a RAR request to itself.

Quoted 5 lines

That’s unrelated. I was referring to considering supporting some sort of lazy TLB to eliminate sending RAR to cores that do not care about it. Is there a cost of RAR to more cores than needed? My guess is that there is one, and maybe in such cases you would want actual IPI and special handling.

For RAR, I suspect the big cost is waking up CPUs in idle states, and waiting for them to wake up.

One possibility may be to change leave_mm() to have an argument to set some flag that the RAR code can read to see whether or not to send a RAR interrupt to that CPU, even if it is in the cpumask.

I don't think we can use the exact same should_flush_tlb() logic, because the tlb_gen is not updated by a RAR flush, and the should_flush_tlb() logic is somewhat intertwined with the tlb_gen logic.

Signature · 1 line
All Rights Reversed.
#29 lore raw

Re: [RFC v2 7/9] x86/mm: Introduce Remote Action Request

In reply to Rik van Riel
kernel test robot noticed the following build errors:

Hi Rik,

[This is a private test report for your RFC patch.] kernel test robot noticed the following build errors:

[auto build test ERROR on tip/master]
[also build test ERROR on linus/master v6.15-rc7 next-20250516] [cannot apply to tip/x86/core tip/x86/mm tip/auto-latest] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Rik-van-Riel/x86-mm-Introduce-MSR_IA32_CORE_CAPABILITIES/20250520-090600
base:   tip/master
patch link:    https://lore.kernel.org/r/20250520010350.1740223-8-riel%40surriel.com
patch subject: [RFC v2 7/9] x86/mm: Introduce Remote Action Request
config: x86_64-buildonly-randconfig-001-20250520 (https://download.01.org/0day-ci/archive/20250520/[email protected]/config)
compiler: gcc-12 (Debian 12.2.0-14) 12.2.0
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20250520/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All errors (new ones prefixed by >>):

arch/x86/mm/rar.c: In function 'smp_call_rar_many':

>> arch/x86/mm/rar.c:167:17: error: implicit declaration of function 'arch_send_rar_single_ipi' [-Werror=implicit-function-declaration]
     167 |                 arch_send_rar_single_ipi(cpu);
         |                 ^~~~~~~~~~~~~~~~~~~~~~~~
>> arch/x86/mm/rar.c:188:9: error: implicit declaration of function 'arch_send_rar_ipi_mask' [-Werror=implicit-function-declaration]
     188 |         arch_send_rar_ipi_mask(dest_mask);
         |         ^~~~~~~~~~~~~~~~~~~~~~

cc1: some warnings being treated as errors

vim +/arch_send_rar_single_ipi +167 arch/x86/mm/rar.c

123
124 /*
125 * This is a modified version of smp_call_function_many() of kernel/smp.c,
126 * without a function pointer, because the RAR handler is the ucode.
127 */
128 void smp_call_rar_many(const struct cpumask *mask, u16 pcid,
129 unsigned long start, unsigned long end)
130 {
131 unsigned long pages = (end - start + PAGE_SIZE) / PAGE_SIZE;
132 int cpu, next_cpu, this_cpu = smp_processor_id();
133 cpumask_t *dest_mask;
134 unsigned long idx;
135
136 if (pages > RAR_INVLPG_MAX_PAGES || end == TLB_FLUSH_ALL)
137 pages = RAR_INVLPG_MAX_PAGES;
138
139 /*
140 * Can deadlock when called with interrupts disabled.
141 * We allow cpu's that are not yet online though, as no one else can
142 * send smp call function interrupt to this cpu and as such deadlocks
143 * can't happen.
144 */
145 WARN_ON_ONCE(cpu_online(this_cpu) && irqs_disabled()
146 && !oops_in_progress && !early_boot_irqs_disabled);
147
148 /* Try to fastpath. So, what's a CPU they want? Ignoring this one. */
149 cpu = cpumask_first_and(mask, cpu_online_mask);
150 if (cpu == this_cpu)
151 cpu = cpumask_next_and(cpu, mask, cpu_online_mask);
152
153 /* No online cpus? We're done. */
154 if (cpu >= nr_cpu_ids)
155 return;
156
157 /* Do we have another CPU which isn't us? */
158 next_cpu = cpumask_next_and(cpu, mask, cpu_online_mask);
159 if (next_cpu == this_cpu)
160 next_cpu = cpumask_next_and(next_cpu, mask, cpu_online_mask);
161
162 /* Fastpath: do that cpu by itself. */
163 if (next_cpu >= nr_cpu_ids) {
164 idx = get_payload();
165 set_payload(idx, pcid, start, pages);
166 set_action_entry(idx, cpu);

 > 167			arch_send_rar_single_ipi(cpu);
   168			wait_for_done(idx, cpu);
   169			free_payload(idx);
   170			return;
   171		}
   172	
   173		dest_mask = this_cpu_ptr(&rar_cpu_mask);
   174		cpumask_and(dest_mask, mask, cpu_online_mask);
   175		cpumask_clear_cpu(this_cpu, dest_mask);
   176	
   177		/* Some callers race with other cpus changing the passed mask */
   178		if (unlikely(!cpumask_weight(dest_mask)))
   179			return;
   180	
   181		idx = get_payload();
   182		set_payload(idx, pcid, start, pages);
   183	
   184		for_each_cpu(cpu, dest_mask)
   185			set_action_entry(idx, cpu);
   186	
   187		/* Send a message to all CPUs in the map */
 > 188		arch_send_rar_ipi_mask(dest_mask);
Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
#30 lore raw

Re: [RFC v2 7/9] x86/mm: Introduce Remote Action Request

In reply to Rik van Riel
Quoted 143 lines · Rik van Riel

From: Yu-cheng Yu <[email protected]>

Remote Action Request (RAR) is a TLB flushing broadcast facility. To start a TLB flush, the initiator CPU creates a RAR payload and sends a command to the APIC. The receiving CPUs automatically flush TLBs as specified in the payload without the kernel's involement.

[ riel: add pcid parameter to smp_call_rar_many so other mms can be flushed ]

---
 arch/x86/include/asm/rar.h   |  69 +++++++++++++
 arch/x86/kernel/cpu/common.c |   4 +
 arch/x86/mm/Makefile         |   1 +
 arch/x86/mm/rar.c            | 195 +++++++++++++++++++++++++++++++++++
 4 files changed, 269 insertions(+)
 create mode 100644 arch/x86/include/asm/rar.h
 create mode 100644 arch/x86/mm/rar.c
arch/x86/include/asm/rar.h new file +69 -0
new file mode 100644
index 000000000000..78c039e40e81
@@ -0,0 +1,69 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _ASM_X86_RAR_H
+#define _ASM_X86_RAR_H
+
+/*
+ * RAR payload types
+ */
+#define RAR_TYPE_INVPG		0
+#define RAR_TYPE_INVPG_NO_CR3	1
+#define RAR_TYPE_INVPCID	2
+#define RAR_TYPE_INVEPT		3
+#define RAR_TYPE_INVVPID	4
+#define RAR_TYPE_WRMSR		5
+
+/*
+ * Subtypes for RAR_TYPE_INVLPG
+ */
+#define RAR_INVPG_ADDR			0 /* address specific */
+#define RAR_INVPG_ALL			2 /* all, include global */
+#define RAR_INVPG_ALL_NO_GLOBAL		3 /* all, exclude global */
+
+/*
+ * Subtypes for RAR_TYPE_INVPCID
+ */
+#define RAR_INVPCID_ADDR		0 /* address specific */
+#define RAR_INVPCID_PCID		1 /* all of PCID */
+#define RAR_INVPCID_ALL			2 /* all, include global */
+#define RAR_INVPCID_ALL_NO_GLOBAL	3 /* all, exclude global */
+
+/*
+ * Page size for RAR_TYPE_INVLPG
+ */
+#define RAR_INVLPG_PAGE_SIZE_4K		0
+#define RAR_INVLPG_PAGE_SIZE_2M		1
+#define RAR_INVLPG_PAGE_SIZE_1G		2
+
+/*
+ * Max number of pages per payload
+ */
+#define RAR_INVLPG_MAX_PAGES 63
+
+struct rar_payload {
+	u64 for_sw		: 8;
+	u64 type		: 8;
+	u64 must_be_zero_1	: 16;
+	u64 subtype		: 3;
+	u64 page_size		: 2;
+	u64 num_pages		: 6;
+	u64 must_be_zero_2	: 21;
+
+	u64 must_be_zero_3;
+
+	/*
+	 * Starting address
+	 */
+	u64 initiator_cr3;
+	u64 linear_address;
+
+	/*
+	 * Padding
+	 */
+	u64 padding[4];
+};
+
+void rar_cpu_init(void);
+void smp_call_rar_many(const struct cpumask *mask, u16 pcid,
+		       unsigned long start, unsigned long end);
+
+#endif /* _ASM_X86_RAR_H */
arch/x86/kernel/cpu/common.c +4 -0
index dd662c42f510..b1e1b9afb2ac 100644
@@ -71,6 +71,7 @@
 #include <asm/tdx.h>
 #include <asm/posted_intr.h>
 #include <asm/runtime-const.h>
+#include <asm/rar.h>
 
 #include "cpu.h"
 
@@ -2438,6 +2439,9 @@ void cpu_init(void)
 	if (is_uv_system())
 		uv_cpu_init();
 
+	if (cpu_feature_enabled(X86_FEATURE_RAR))
+		rar_cpu_init();
+
 	load_fixmap_gdt(cpu);
 }
 
arch/x86/mm/Makefile +1 -0
index 5b9908f13dcf..f36fc99e8b10 100644
@@ -52,6 +52,7 @@ obj-$(CONFIG_ACPI_NUMA)		+= srat.o
 obj-$(CONFIG_X86_INTEL_MEMORY_PROTECTION_KEYS)	+= pkeys.o
 obj-$(CONFIG_RANDOMIZE_MEMORY)			+= kaslr.o
 obj-$(CONFIG_MITIGATION_PAGE_TABLE_ISOLATION)	+= pti.o
+obj-$(CONFIG_BROADCAST_TLB_FLUSH)		+= rar.o
 
 obj-$(CONFIG_X86_MEM_ENCRYPT)	+= mem_encrypt.o
 obj-$(CONFIG_AMD_MEM_ENCRYPT)	+= mem_encrypt_amd.o
arch/x86/mm/rar.c new file +8 -0
new file mode 100644
index 000000000000..16dc9b889cbd
@@ -0,0 +1,195 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * RAR TLB shootdown
+ */
+#include <linux/sched.h>
+#include <linux/bug.h>
+#include <asm/current.h>
+#include <asm/io.h>

arch/x86/mm/rar.c line 18

Quoted 10 lines
arch/x86/mm/rar.c new file +10 -0
+#include <asm/sync_bitops.h>
+#include <asm/rar.h>
+#include <asm/tlbflush.h>
+
+static DEFINE_PER_CPU(struct cpumask, rar_cpu_mask);
+
+#define RAR_ACTION_OK		0x00
+#define RAR_ACTION_START	0x01
+#define RAR_ACTION_ACKED	0x02
+#define RAR_ACTION_FAIL		0x80

These don't match up with the names that ended up in the public documentation. Could we realign them, please?

arch/x86/mm/rar.c line 24

Quoted 5 lines
arch/x86/mm/rar.c new file +5 -0
+#define RAR_MAX_PAYLOADS 32UL
+
+static unsigned long rar_in_use = ~(RAR_MAX_PAYLOADS - 1);
+static struct rar_payload rar_payload[RAR_MAX_PAYLOADS] __page_aligned_bss;
+static DEFINE_PER_CPU_ALIGNED(u8[RAR_MAX_PAYLOADS], rar_action);

At some point, there needs to be a description of the data structures. For instance, there's nothing architecturally requiring all CPUs to share a payload table. But this implementation chooses to have them share. We need a discussion somewhere of those design decisions.

One thing that also needs discussion: 'rar_in_use' isn't really about RAR itself. It's a bitmap of whether the payload is allocated.

arch/x86/mm/rar.c line 25

Quoted 2 lines
arch/x86/mm/rar.c new file +2 -0
+static unsigned long get_payload(void)
+{

This is more like "allocate a payload slot" than a "get payload" operation, IMNHO.

Quoted 6 lines
arch/x86/mm/rar.c new file +6 -0
+	while (1) {
+		unsigned long bit;
+
+		/*
+		 * Find a free bit and confirm it with
+		 * test_and_set_bit() below.

arch/x86/mm/rar.c line 43

Quoted 10 lines
arch/x86/mm/rar.c new file +10 -0
+		 */
+		bit = ffz(READ_ONCE(rar_in_use));
+
+		if (bit >= RAR_MAX_PAYLOADS)
+			continue;
+
+		if (!test_and_set_bit((long)bit, &rar_in_use))
+			return bit;
+	}
+}

This also serves like a kind of spinlock to wait for a payload slot to become free.

arch/x86/mm/rar.c line 53

Quoted 9 lines
arch/x86/mm/rar.c new file +9 -0
+static void free_payload(unsigned long idx)
+{
+	clear_bit(idx, &rar_in_use);
+}
+
+static void set_payload(unsigned long idx, u16 pcid, unsigned long start,
+			uint32_t pages)
+{
+	struct rar_payload *p = &rar_payload[idx];

I'd _probably_ just pass the 'struct rar_payload *' instead of an index. It's harder to screw up a pointer.

Quoted 15 lines
arch/x86/mm/rar.c new file +15 -0
+	p->must_be_zero_1	= 0;
+	p->must_be_zero_2	= 0;
+	p->must_be_zero_3	= 0;
+	p->page_size		= RAR_INVLPG_PAGE_SIZE_4K;
+	p->type			= RAR_TYPE_INVPCID;
+	p->num_pages		= pages;
+	p->initiator_cr3	= pcid;
+	p->linear_address	= start;
+
+	if (pcid) {
+		/* RAR invalidation of the mapping of a specific process. */
+		if (pages >= RAR_INVLPG_MAX_PAGES)
+			p->subtype = RAR_INVPCID_PCID;
+		else
+			p->subtype = RAR_INVPCID_ADDR;

arch/x86/mm/rar.c line 79

Quoted 10 lines
arch/x86/mm/rar.c new file +10 -0
+	} else {
+		/*
+		 * Unfortunately RAR_INVPCID_ADDR excludes global translations.
+		 * Always do a full flush for kernel invalidations.
+		 */
+		p->subtype = RAR_INVPCID_ALL;
+	}
+
+	smp_wmb();
+}

The barrier could use a comment too.

arch/x86/mm/rar.c line 81

Quoted 1 line
arch/x86/mm/rar.c new file +1 -0
+static void set_action_entry(unsigned long idx, int target_cpu)

Just trying to read this, I think we probably should remove the 'idx' nomenclature and call them "payload_nr"'s or something more descriptive.

arch/x86/mm/rar.c line 86

Quoted 5 lines
arch/x86/mm/rar.c new file +5 -0
+{
+	u8 *bitmap = per_cpu(rar_action, target_cpu);
+
+	WRITE_ONCE(bitmap[idx], RAR_ACTION_START);
+}

Maybe a comment like this for set_action_entry() would be helpful:

/*
* Given a remote CPU, "arm" its action vector to ensure it
* handles payload number 'idx' when it receives the RAR signal.
* The remote CPU will overwrite RAR_ACTION_START when it handles
* the request.
*/

arch/x86/mm/rar.c line 95

Quoted 8 lines
arch/x86/mm/rar.c new file +8 -0
+static void wait_for_done(unsigned long idx, int target_cpu)
+{
+	u8 status;
+	u8 *rar_actions = per_cpu(rar_action, target_cpu);
+
+	status = READ_ONCE(rar_actions[idx]);
+
+	while ((status != RAR_ACTION_OK) && (status != RAR_ACTION_FAIL)) {

Should this be:

	while (status == RAR_ACTION_START) {
	...

? That would more clearly link it to set_action_entry() and would also be shorter.

Quoted 7 lines
arch/x86/mm/rar.c new file +7 -0
+		cpu_relax();
+		status = READ_ONCE(rar_actions[idx]);
+	}
+
+	WARN_ON_ONCE(rar_actions[idx] == RAR_ACTION_FAIL);
+}
+

arch/x86/mm/rar.c line 112

Quoted 10 lines
arch/x86/mm/rar.c new file +10 -0
+void rar_cpu_init(void)
+{
+	u64 r;
+	u8 *bitmap;
+	int this_cpu = smp_processor_id();
+
+	cpumask_clear(&per_cpu(rar_cpu_mask, this_cpu));
+
+	rdmsrl(MSR_IA32_RAR_INFO, r);
+	pr_info_once("RAR: support %lld payloads\n", r >> 32);

Doesn't this need to get coordinated or checked against RAR_MAX_PAYLOADS?

It might also be nice to use one of the mask functions for this. It's nice when you see a spec say "37:32" and then you see code actually see a GENMASK(37, 32) somewhere to match it.

arch/x86/mm/rar.c line 117

Quoted 4 lines
arch/x86/mm/rar.c new file +4 -0
+	bitmap = (u8 *)per_cpu(rar_action, this_cpu);
+	memset(bitmap, 0, RAR_MAX_PAYLOADS);
+	wrmsrl(MSR_IA32_RAR_ACT_VEC, (u64)virt_to_phys(bitmap));
+	wrmsrl(MSR_IA32_RAR_PAYLOAD_BASE, (u64)virt_to_phys(rar_payload));

please vertically align the virt_to_phys() ^

arch/x86/mm/rar.c line 112

Quoted 2 lines
arch/x86/mm/rar.c new file +2 -0
+
+	r = RAR_CTRL_ENABLE | RAR_CTRL_IGNORE_IF;

Setting RAR_CTRL_IGNORE_IF is probably worth a _little_ discussion in the changelog.

arch/x86/mm/rar.c line 120

Quoted 1 line
arch/x86/mm/rar.c new file +1 -0
+	// reserved bits!!! r |= (RAR_VECTOR & 0xff);

Is this just some cruft from testing?

arch/x86/mm/rar.c line 127

Quoted 7 lines
arch/x86/mm/rar.c new file +7 -0
+	wrmsrl(MSR_IA32_RAR_CTRL, r);
+}
+
+/*
+ * This is a modified version of smp_call_function_many() of kernel/smp.c,
+ * without a function pointer, because the RAR handler is the ucode.
+ */

It doesn't look _that_ much like smp_call_function_many(). I don't see much that can be consolidated.

Quoted 4 lines
arch/x86/mm/rar.c new file +4 -0
+void smp_call_rar_many(const struct cpumask *mask, u16 pcid,
+		       unsigned long start, unsigned long end)
+{
+	unsigned long pages = (end - start + PAGE_SIZE) / PAGE_SIZE;

arch/x86/mm/rar.c line 141

Quoted 10 lines
arch/x86/mm/rar.c new file +10 -0
+	int cpu, next_cpu, this_cpu = smp_processor_id();
+	cpumask_t *dest_mask;
+	unsigned long idx;
+
+	if (pages > RAR_INVLPG_MAX_PAGES || end == TLB_FLUSH_ALL)
+		pages = RAR_INVLPG_MAX_PAGES;
+
+	/*
+	 * Can deadlock when called with interrupts disabled.
+	 * We allow cpu's that are not yet online though, as no one else can

Nit: at some point all of the "we's" need to be excised and moved over to imperative voice.

Quoted 4 lines
arch/x86/mm/rar.c new file +4 -0
+	 * send smp call function interrupt to this cpu and as such deadlocks
+	 * can't happen.
+	 */
+	WARN_ON_ONCE(cpu_online(this_cpu) && irqs_disabled()

arch/x86/mm/rar.c line 155

Quoted 10 lines
arch/x86/mm/rar.c new file +10 -0
+		     && !oops_in_progress && !early_boot_irqs_disabled);
+
+	/* Try to fastpath.  So, what's a CPU they want?  Ignoring this one. */
+	cpu = cpumask_first_and(mask, cpu_online_mask);
+	if (cpu == this_cpu)
+		cpu = cpumask_next_and(cpu, mask, cpu_online_mask);
+
+	/* No online cpus?  We're done. */
+	if (cpu >= nr_cpu_ids)
+		return;

This little idiom _is_ in smp_call_function_many_cond(). I wonder if it can be refactored out.

Quoted 5 lines
arch/x86/mm/rar.c new file +5 -0
+	/* Do we have another CPU which isn't us? */
+	next_cpu = cpumask_next_and(cpu, mask, cpu_online_mask);
+	if (next_cpu == this_cpu)
+		next_cpu = cpumask_next_and(next_cpu, mask, cpu_online_mask);
+

arch/x86/mm/rar.c line 171

Quoted 10 lines
arch/x86/mm/rar.c new file +10 -0
+	/* Fastpath: do that cpu by itself. */
+	if (next_cpu >= nr_cpu_ids) {
+		idx = get_payload();
+		set_payload(idx, pcid, start, pages);
+		set_action_entry(idx, cpu);
+		arch_send_rar_single_ipi(cpu);
+		wait_for_done(idx, cpu);
+		free_payload(idx);
+		return;
+	}

FWIW, I'm not sure this is that much of a fast path. I wouldn't be shocked if _some_ hardware has a much faster way of IPI'ing a single CPU versus a bunch. But I think arch_send_rar_single_ipi() and arch_send_rar_ipi_mask() end up frobbing the hardware in pretty similar ways.

I'd probably just axe this in the name of simplification unless there are numbers behind it.

Quoted 9 lines
arch/x86/mm/rar.c new file +9 -0
+	dest_mask = this_cpu_ptr(&rar_cpu_mask);
+	cpumask_and(dest_mask, mask, cpu_online_mask);
+	cpumask_clear_cpu(this_cpu, dest_mask);
+
+	/* Some callers race with other cpus changing the passed mask */
+	if (unlikely(!cpumask_weight(dest_mask)))
+		return;
+
+	idx = get_payload();

arch/x86/mm/rar.c line 191

Quoted 10 lines
arch/x86/mm/rar.c new file +10 -0
+	set_payload(idx, pcid, start, pages);
+
+	for_each_cpu(cpu, dest_mask)
+		set_action_entry(idx, cpu);
+
+	/* Send a message to all CPUs in the map */
+	arch_send_rar_ipi_mask(dest_mask);
+
+	for_each_cpu(cpu, dest_mask)
+		wait_for_done(idx, cpu);

Naming nit: Let's give wait_for_done() a more RAR-specific name. It'll make it clear that this is a RAR opertion and not soemthing generic.

Quoted 3 lines
arch/x86/mm/rar.c new file +3 -0
+	free_payload(idx);
+}
+EXPORT_SYMBOL(smp_call_rar_many);
#31 lore raw

Re: [RFC v2 7/9] x86/mm: Introduce Remote Action Request

Quoted 8 lines · Dave Hansen
Quoted 4 lines · Rik van Riel

+/*
+ * This is a modified version of smp_call_function_many() of kernel/smp.c,
+ * without a function pointer, because the RAR handler is the ucode.
+ */

It doesn't look _that_ much like smp_call_function_many(). I don't see much that can be consolidated.

It does not look like it because it has a gazillion of function arguments, which can all be packed into a data structure, i.e. the function argument of smp_call_function_many().

There is zero justification to reinvent the wheel and create another source of hard to debug problems.

IMNSHO it's absolutely not rocket science to reuse smp_call_function_many() for this, but I might be missing something as always and I'm happy to be enlightenend.

Just for the record: The changelog contains an utter void of information why this modified version of well established common code is required and desired.

Thanks,

tglx

#32 lore raw

Re: [RFC v2 7/9] x86/mm: Introduce Remote Action Request

In reply to Dave Hansen
Quoted 13 lines · Dave Hansen
Quoted 10 lines
arch/x86/mm/rar.c new file +8 -0
+static void wait_for_done(unsigned long idx, int target_cpu)
+{
+	u8 status;
+	u8 *rar_actions = per_cpu(rar_action, target_cpu);
+
+	status = READ_ONCE(rar_actions[idx]);
+
+	while ((status != RAR_ACTION_OK) && (status !=

RAR_ACTION_FAIL)) {

Should this be:

Quoted 7 lines
	while (status == RAR_ACTION_START) {
	...

? That would more clearly link it to set_action_entry() and would also
be shorter.

That is a very good question. The old RAR code suggests there might be some intermediate state when the target CPU works on processing the RAR entry, but the current documentation only shows RAR_SUCCESS, RAR_PENDING, and RAR_FAILURE as possible values.

Lets try with status == RAR_ACTION_PENDING.

Quoted 14 lines

arch/x86/mm/rar.c line 112

Quoted 11 lines
arch/x86/mm/rar.c new file +10 -0
+void rar_cpu_init(void)
+{
+	u64 r;
+	u8 *bitmap;
+	int this_cpu = smp_processor_id();
+
+	cpumask_clear(&per_cpu(rar_cpu_mask, this_cpu));
+
+	rdmsrl(MSR_IA32_RAR_INFO, r);
+	pr_info_once("RAR: support %lld payloads\n", r >> 32);

Doesn't this need to get coordinated or checked against RAR_MAX_PAYLOADS?

I just added that in, and also applied all the cleanups from your email.

Quoted 5 lines

arch/x86/mm/rar.c line 120

Quoted 2 lines
arch/x86/mm/rar.c new file +1 -0
+	// reserved bits!!! r |= (RAR_VECTOR & 0xff);

Is this just some cruft from testing?

I'm kind of guessing the old code might have used this value to specify which IRQ vector to use for RAR, but modern microcode hardcodes the RAR_VECTOR value.

Quoted 5 lines
Quoted 5 lines
arch/x86/mm/rar.c new file +5 -0
+	wrmsrl(MSR_IA32_RAR_CTRL, r);
+}
+
+/*
+ * This is a modified version of smp_call_function_many() of
Quoted 8 lines
Quoted 4 lines

kernel/smp.c,
+ * without a function pointer, because the RAR handler is the ucode.
+ */

It doesn't look _that_ much like smp_call_function_many(). I don't see
much that can be consolidated.

Agreed. It looks even less like it after some more simplifications.

Quoted 8 lines

arch/x86/mm/rar.c line 155

Quoted 4 lines
arch/x86/mm/rar.c new file +3 -0
+	/* No online cpus?  We're done. */
+	if (cpu >= nr_cpu_ids)
+		return;

This little idiom _is_ in smp_call_function_many_cond(). I wonder if it
can be refactored out.

Removing the arch_send_rar_single_ipi fast path gets rid of this code completely.

Once we cpumask_and with the cpu_online_mask, the cpumask_weight should end up as 0 if no online CPUs are in the mask.

Thank you for all the cleanup suggestions. I've tried to address them all for v3.

Signature · 1 line
All Rights Reversed.
#33 lore raw

RFC v2 8/9 x86/mm: use RAR for kernel TLB flushes

In reply to Rik van Riel

From: Rik van Riel <[email protected]>

Use Intel RAR for kernel TLB flushes, when enabled.

Pass in PCID 0 to smp_call_rar_many() to flush the specified addresses, regardless of which PCID they might be cached under in any destination CPU.

---
 arch/x86/mm/rar.c |  4 ++--
 arch/x86/mm/tlb.c | 38 ++++++++++++++++++++++++++++++++++++++
 2 files changed, 40 insertions(+), 2 deletions(-)
arch/x86/mm/rar.c +2 -2
index 16dc9b889cbd..9a18c926ea7b 100644
@@ -142,8 +142,8 @@ void smp_call_rar_many(const struct cpumask *mask, u16 pcid,
 	 * send smp call function interrupt to this cpu and as such deadlocks
 	 * can't happen.
 	 */
-	WARN_ON_ONCE(cpu_online(this_cpu) && irqs_disabled()
-		     && !oops_in_progress && !early_boot_irqs_disabled);
+	if (cpu_online(this_cpu) && !oops_in_progress && !early_boot_irqs_disabled)
+		lockdep_assert_irqs_enabled();
 
 	/* Try to fastpath.  So, what's a CPU they want?  Ignoring this one. */
 	cpu = cpumask_first_and(mask, cpu_online_mask);
arch/x86/mm/tlb.c +38 -0
index f5761e8be77f..35489df811dc 100644
@@ -21,6 +21,7 @@
 #include <asm/apic.h>
 #include <asm/msr.h>
 #include <asm/perf_event.h>
+#include <asm/rar.h>
 #include <asm/tlb.h>
 
 #include "mm_internal.h"
@@ -1446,6 +1447,18 @@ static void do_flush_tlb_all(void *info)
 	__flush_tlb_all();
 }
 
+static void rar_full_flush(const cpumask_t *cpumask)
+{
+	guard(preempt)();
+	smp_call_rar_many(cpumask, 0, 0, TLB_FLUSH_ALL);
+	invpcid_flush_all();
+}
+
+static void rar_flush_all(void)
+{
+	rar_full_flush(cpu_online_mask);
+}
+
 void flush_tlb_all(void)
 {
 	count_vm_tlb_event(NR_TLB_REMOTE_FLUSH);
@@ -1453,6 +1466,8 @@ void flush_tlb_all(void)
 	/* First try (faster) hardware-assisted TLB invalidation. */
 	if (cpu_feature_enabled(X86_FEATURE_INVLPGB))
 		invlpgb_flush_all();
+	else if (cpu_feature_enabled(X86_FEATURE_RAR))
+		rar_flush_all();
 	else
 		/* Fall back to the IPI-based invalidation. */
 		on_each_cpu(do_flush_tlb_all, NULL, 1);
@@ -1482,15 +1497,36 @@ static void do_kernel_range_flush(void *info)
 	struct flush_tlb_info *f = info;
 	unsigned long addr;
 
+	/*
+	 * With PTI kernel TLB entries in all PCIDs need to be flushed.
+	 * With RAR the PCID space becomes so large, we might as well flush it all.
+	 *
+	 * Either of the two by itself works with targeted flushes.
+	 */
+	if (cpu_feature_enabled(X86_FEATURE_RAR) &&
+	    cpu_feature_enabled(X86_FEATURE_PTI)) {
+		invpcid_flush_all();
+		return;
+	}
+
 	/* flush range by one by one 'invlpg' */
 	for (addr = f->start; addr < f->end; addr += PAGE_SIZE)
 		flush_tlb_one_kernel(addr);
 }
 
+static void rar_kernel_range_flush(struct flush_tlb_info *info)
+{
+	guard(preempt)();
+	smp_call_rar_many(cpu_online_mask, 0, info->start, info->end);
+	do_kernel_range_flush(info);
+}
+
 static void kernel_tlb_flush_all(struct flush_tlb_info *info)
 {
 	if (cpu_feature_enabled(X86_FEATURE_INVLPGB))
 		invlpgb_flush_all();
+	else if (cpu_feature_enabled(X86_FEATURE_RAR))
+		rar_flush_all();
 	else
 		on_each_cpu(do_flush_tlb_all, NULL, 1);
 }
@@ -1499,6 +1535,8 @@ static void kernel_tlb_flush_range(struct flush_tlb_info *info)
 {
 	if (cpu_feature_enabled(X86_FEATURE_INVLPGB))
 		invlpgb_kernel_range_flush(info);
+	else if (cpu_feature_enabled(X86_FEATURE_RAR))
+		rar_kernel_range_flush(info);
 	else
 		on_each_cpu(do_kernel_range_flush, info, 1);
 }
Signature · 1 line
2.49.0
#34 lore raw

RFC v2 9/9 x86/mm: userspace & pageout flushing using Intel RAR

In reply to Rik van Riel

From: Rik van Riel <[email protected]>

Use Intel RAR to flush userspace mappings.

Because RAR flushes are targeted using a cpu bitmap, the rules are a little bit different than for true broadcast TLB invalidation.

For true broadcast TLB invalidation, like done with AMD INVLPGB, a global ASID always has up to date TLB entries on every CPU. The context switch code never has to flush the TLB when switching to a global ASID on any CPU with INVLPGB.

For RAR, the TLB mappings for a global ASID are kept up to date only on CPUs within the mm_cpumask, which lazily follows the threads around the system. The context switch code does not need to flush the TLB if the CPU is in the mm_cpumask, and the PCID used stays the same.

However, a CPU that falls outside of the mm_cpumask can have out of date TLB mappings for this task. When switching to that task on a CPU not in the mm_cpumask, the TLB does need to be flushed.

---
 arch/x86/include/asm/tlbflush.h |   9 ++-
 arch/x86/mm/tlb.c               | 121 ++++++++++++++++++++++++++------
 2 files changed, 104 insertions(+), 26 deletions(-)
arch/x86/include/asm/tlbflush.h +6 -3
index cc9935bbbd45..bdde3ce6c9b1 100644
@@ -276,7 +276,8 @@ static inline u16 mm_global_asid(struct mm_struct *mm)
 {
 	u16 asid;
 
-	if (!cpu_feature_enabled(X86_FEATURE_INVLPGB))
+	if (!cpu_feature_enabled(X86_FEATURE_INVLPGB) &&
+	    !cpu_feature_enabled(X86_FEATURE_RAR))
 		return 0;
 
 	asid = smp_load_acquire(&mm->context.global_asid);
@@ -289,7 +290,8 @@ static inline u16 mm_global_asid(struct mm_struct *mm)
 
 static inline void mm_init_global_asid(struct mm_struct *mm)
 {
-	if (cpu_feature_enabled(X86_FEATURE_INVLPGB)) {
+	if (cpu_feature_enabled(X86_FEATURE_INVLPGB) ||
+	    cpu_feature_enabled(X86_FEATURE_RAR)) {
 		mm->context.global_asid = 0;
 		mm->context.asid_transition = false;
 	}
@@ -313,7 +315,8 @@ static inline void mm_clear_asid_transition(struct mm_struct *mm)
 
 static inline bool mm_in_asid_transition(struct mm_struct *mm)
 {
-	if (!cpu_feature_enabled(X86_FEATURE_INVLPGB))
+	if (!cpu_feature_enabled(X86_FEATURE_INVLPGB) &&
+	    !cpu_feature_enabled(X86_FEATURE_RAR))
 		return false;
 
 	return mm && READ_ONCE(mm->context.asid_transition);
arch/x86/mm/tlb.c +98 -23

249 lines not on this page · raw message

Signature · 1 line
2.49.0
#35 lore raw

Re: [RFC v2.1 9/9] x86/mm: userspace & pageout flushing using Intel RAR

Quoted 3 lines · On Mon, 19 May 2025 21:02:34 -0400 Rik van Riel <[email protected]> wrote:

From: Rik van Riel <[email protected]>

Use Intel RAR to flush userspace mappings.

The version below no longer segfaults.

However, I am still hitting the WARN_ON() in leave_mm(), when called from the idle task through cpuidle_enter_state().

---8<---
From e80e10cdb6f15d29a65ab438cb07ba4b99f64b6e Mon Sep 17 00:00:00 2001
From: Rik van Riel <[email protected]>
Date: Thu, 24 Apr 2025 07:15:44 -0700
Subject: [PATCH 10/11] x86/mm: userspace & pageout flushing using Intel RAR

Use Intel RAR to flush userspace mappings.

Because RAR flushes are targeted using a cpu bitmap, the rules are a little bit different than for true broadcast TLB invalidation.

For true broadcast TLB invalidation, like done with AMD INVLPGB, a global ASID always has up to date TLB entries on every CPU. The context switch code never has to flush the TLB when switching to a global ASID on any CPU with INVLPGB.

For RAR, the TLB mappings for a global ASID are kept up to date only on CPUs within the mm_cpumask, which lazily follows the threads around the system. The context switch code does not need to flush the TLB if the CPU is in the mm_cpumask, and the PCID used stays the same.

However, a CPU that falls outside of the mm_cpumask can have out of date TLB mappings for this task. When switching to that task on a CPU not in the mm_cpumask, the TLB does need to be flushed.

---
 arch/x86/include/asm/tlbflush.h |   9 ++-
 arch/x86/mm/tlb.c               | 133 +++++++++++++++++++++++++-------
 2 files changed, 111 insertions(+), 31 deletions(-)
arch/x86/include/asm/tlbflush.h +6 -3
index cc9935bbbd45..bdde3ce6c9b1 100644
@@ -276,7 +276,8 @@ static inline u16 mm_global_asid(struct mm_struct *mm)
 {
 	u16 asid;
 
-	if (!cpu_feature_enabled(X86_FEATURE_INVLPGB))
+	if (!cpu_feature_enabled(X86_FEATURE_INVLPGB) &&
+	    !cpu_feature_enabled(X86_FEATURE_RAR))
 		return 0;
 
 	asid = smp_load_acquire(&mm->context.global_asid);
@@ -289,7 +290,8 @@ static inline u16 mm_global_asid(struct mm_struct *mm)
 
 static inline void mm_init_global_asid(struct mm_struct *mm)
 {
-	if (cpu_feature_enabled(X86_FEATURE_INVLPGB)) {
+	if (cpu_feature_enabled(X86_FEATURE_INVLPGB) ||
+	    cpu_feature_enabled(X86_FEATURE_RAR)) {
 		mm->context.global_asid = 0;
 		mm->context.asid_transition = false;
 	}
@@ -313,7 +315,8 @@ static inline void mm_clear_asid_transition(struct mm_struct *mm)
 
 static inline bool mm_in_asid_transition(struct mm_struct *mm)
 {
-	if (!cpu_feature_enabled(X86_FEATURE_INVLPGB))
+	if (!cpu_feature_enabled(X86_FEATURE_INVLPGB) &&
+	    !cpu_feature_enabled(X86_FEATURE_RAR))
 		return false;
 
 	return mm && READ_ONCE(mm->context.asid_transition);
arch/x86/mm/tlb.c +105 -28

281 lines not on this page · raw message

Signature · 1 line
2.47.1