x86/mm: Intel RAR TLB invalidation support
Subsystems (3)
v5 cover x86/mm: Intel RAR TLB invalidation support
This patch series adds support for IPI-less TLB invalidation using Intel RAR technology.
Intel RAR differs from AMD INVLPGB in a few ways:
- RAR goes through (emulated?) APIC writes, not instructions
- RAR flushes go through a memory table with 64 entries
- RAR flushes can be targeted to a cpumask
- The RAR functionality must be set up at boot time before it can be used
The cpumask targeting has resulted in Intel RAR and AMD INVLPGB having slightly different rules:
- Processes with dynamic ASIDs use IPI based shootdowns
- INVLPGB: processes with a global ASID
- always have the TLB up to date, on every CPU
- never need to flush the TLB at context switch time
- RAR: processes with global ASIDs
- have the TLB up to date on CPUs in the mm_cpumask
- can skip a TLB flush at context switch time if the CPU is in the mm_cpumask
- need to flush the TLB when scheduled on a cpu not in the mm_cpumask, in case it used to run there before and the TLB has stale entries
RAR functionality is present on Sapphire Rapids and newer CPUs.
Information about Intel RAR can be found in this whitepaper.
This patch series is based off a 2019 patch series created by Intel, with patches later in the series modified to fit into the TLB flush code structure we have after AMD INVLPGB functionality was integrated.
The stock hardware needed some tweaks from Intel before this series will work. Contact Dave Hansen ofr details if you want to run this on real hardware.
TODO:
- some sort of optimization to avoid sending RARs to CPUs in deeper idle states when they have init_mm loaded (flush when switching to init_mm?)
- figure out if there is a way to make RAR actually go faster than IPI-based TLB invalidation (getting closer?)
v5:
- Add optimization to avoid heavy contention in apic_mem_wait_irc_idle() v4:
- remove chicken/egg problem that made it impossible to use RAR early in bootup, now RAR can be used to flush the local TLB (but it's broken?)
- always flush other CPUs with RAR, no more periodic flush_tlb_func
- separate, simplified cpumask trimming code
- attempt to use RAR to flush the local TLB, which should work according to the documentation
- add a DEBUG patch to flush the local TLB with RAR and again locally, may need some help from Intel to figure out why this makes a difference
- memory dumps of rar_payload[] suggest we are sending valid RARs
- receiving CPUs set the status from RAR_PENDING to RAR_SUCCESS
- unclear whether the TLB is actually flushed correctly :(
v3:
- move cpa_flush() change out of this patch series
- use MSR_IA32_CORE_CAPS definition, merge first two patches together
- move RAR initialization to early_init_intel()
- remove single-CPU "fast path" from smp_call_rar_many
- remove smp call table RAR entries, just do a direct call
- cleanups suggested (Ingo, Nadav, Dave, Thomas, Borislav, Sean)
- fix !CONFIG_SMP compile in Kconfig
- match RAR definitions to the names & numbers in the documentation
- the code seems to work now
v2:
- Cleanups suggested by Ingo and Nadav (thank you)
- Basic RAR code seems to actually work now.
- Kernel TLB flushes with RAR seem to work correctly.
- User TLB flushes with RAR are still broken, with two symptoms:
- The !is_lazy WARN_ON in leave_mm() is tripped
- Random segfaults.
RFC v5 1/8 x86/mm: Introduce Remote Action Request MSRs
From: Yu-cheng Yu <[email protected]>
Remote Action Request (RAR) is a model-specific feature to speed up inter-processor operations by moving parts of those operations from software to hardware.
The current RAR implementation handles TLB flushes and MSR writes.
This patch introduces RAR MSRs. RAR is introduced in later patches.
There are five RAR MSRs:
MSR_CORE_CAPABILITIES
MSR_IA32_RAR_CTRL
MSR_IA32_RAR_ACT_VEC
MSR_IA32_RAR_PAYLOAD_BASE
MSR_IA32_RAR_INFO
--- arch/x86/include/asm/msr-index.h | 13 +++++++++++++ 1 file changed, 13 insertions(+)
arch/x86/include/asm/msr-index.h +13 -0
@@ -110,6 +110,8 @@/* Abbreviated from Intel SDM name IA32_CORE_CAPABILITIES */ #define MSR_IA32_CORE_CAPS 0x000000cf +#define MSR_IA32_CORE_CAPS_RAR_BIT 1 +#define MSR_IA32_CORE_CAPS_RAR BIT(MSR_IA32_CORE_CAPS_RAR_BIT)#define MSR_IA32_CORE_CAPS_INTEGRITY_CAPS_BIT 2#define MSR_IA32_CORE_CAPS_INTEGRITY_CAPS BIT(MSR_IA32_CORE_CAPS_INTEGRITY_CAPS_BIT)#define MSR_IA32_CORE_CAPS_SPLIT_LOCK_DETECT_BIT 5
@@ -122,6 +124,17 @@#define SNB_C3_AUTO_UNDEMOTE (1UL << 27)#define SNB_C1_AUTO_UNDEMOTE (1UL << 28) +/* +* Remote Action Requests (RAR) MSRs +*/ +#define MSR_IA32_RAR_CTRL 0x000000ed +#define MSR_IA32_RAR_ACT_VEC 0x000000ee +#define MSR_IA32_RAR_PAYLOAD_BASE 0x000000ef +#define MSR_IA32_RAR_INFO 0x000000f0 + +#define RAR_CTRL_ENABLE BIT(31) +#define RAR_CTRL_IGNORE_IF BIT(30) +#define MSR_MTRRcap 0x000000fe#define MSR_IA32_ARCH_CAPABILITIES 0x0000010a
Signature · 1 line
2.51.1
RFC v5 2/8 x86/mm: enable BROADCAST_TLB_FLUSH on Intel, too
In reply to Rik van RielFrom: Rik van Riel <[email protected]>
Much of the code for Intel RAR and AMD INVLPGB is shared.
Place both under the same config option.
--- arch/x86/Kconfig.cpu | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/Kconfig.cpu +1 -1
@@ -360,7 +360,7 @@ menuconfig PROCESSOR_SELECT config BROADCAST_TLB_FLUSH def_bool y - depends on CPU_SUP_AMD && 64BIT + depends on (CPU_SUP_AMD || CPU_SUP_INTEL) && 64BIT && SMP config CPU_SUP_INTEL default y
Signature · 1 line
2.51.1
RFC v5 3/8 x86/mm: Introduce X86_FEATURE_RAR
In reply to Rik van RielFrom: Yu-cheng Yu <[email protected]>
Introduce X86_FEATURE_RAR and enumeration of the feature.
[riel: moved initialization to intel.c and disabling to Kconfig.cpufeatures]
--- arch/x86/Kconfig.cpufeatures | 4 ++++ arch/x86/include/asm/cpufeatures.h | 2 +- arch/x86/kernel/cpu/intel.c | 9 +++++++++ 3 files changed, 14 insertions(+), 1 deletion(-)
arch/x86/Kconfig.cpufeatures +4 -0
@@ -199,3 +199,7 @@ config X86_DISABLED_FEATURE_SEV_SNP config X86_DISABLED_FEATURE_INVLPGB def_bool y depends on !BROADCAST_TLB_FLUSH + +config X86_DISABLED_FEATURE_RAR + def_bool y + depends on !BROADCAST_TLB_FLUSH
arch/x86/include/asm/cpufeatures.h +1 -1
@@ -76,7 +76,7 @@#define X86_FEATURE_K8 ( 3*32+ 4)/* Opteron, Athlon64 */ #define X86_FEATURE_ZEN5 ( 3*32+ 5)/* CPU based on Zen5 microarchitecture */ #define X86_FEATURE_ZEN6 ( 3*32+ 6)/* CPU based on Zen6 microarchitecture */ -/* Free ( 3*32+ 7) */ +#define X86_FEATURE_RAR ( 3*32+ 7)/* Intel Remote Action Request */ #define X86_FEATURE_CONSTANT_TSC ( 3*32+ 8)/* "constant_tsc" TSC ticks at a constant rate */ #define X86_FEATURE_UP ( 3*32+ 9)/* "up" SMP kernel running on UP */ #define X86_FEATURE_ART ( 3*32+10)/* "art" Always running timer (ART) */
arch/x86/kernel/cpu/intel.c +9 -0
@@ -719,6 +719,15 @@ static void intel_detect_tlb(struct cpuinfo_x86 *c)cpuid_leaf_0x2 (®s);for_each_cpuid_0x2_desc (regs, ptr, desc)intel_tlb_lookup (desc); + +if (cpu_has (c,X86_FEATURE_CORE_CAPABILITIES )) { +u64 msr; + +rdmsrl (MSR_IA32_CORE_CAPS , msr); + +if (msr &MSR_IA32_CORE_CAPS_RAR ) +setup_force_cpu_cap (X86_FEATURE_RAR ); + } }static const struct cpu_dev intel_cpu_dev = {
Signature · 1 line
2.51.1
RFC v5 4/8 x86/apic: Introduce Remote Action Request Operations
In reply to Rik van RielFrom: Yu-cheng Yu <[email protected]>
RAR TLB flushing is started by sending a command to the APIC. This patch adds Remote Action Request commands.
Because RAR_VECTOR is hardcoded at 0xe0, POSTED_MSI_NOTIFICATION_VECTOR has to be lowered to 0xdf, reducing the number of available vectors by 13.
[riel: refactor after 6 years of changes, lower POSTED_MSI_NOTIFICATION_VECTOR]
--- arch/x86/include/asm/apicdef.h | 1 + arch/x86/include/asm/irq_vectors.h | 7 ++++++- arch/x86/include/asm/smp.h | 1 + arch/x86/kernel/apic/ipi.c | 5 +++++ arch/x86/kernel/apic/local.h | 3 +++ 5 files changed, 16 insertions(+), 1 deletion(-)
arch/x86/include/asm/apicdef.h +1 -0
@@ -92,6 +92,7 @@#define APIC_DM_LOWEST 0x00100#define APIC_DM_SMI 0x00200#define APIC_DM_REMRD 0x00300 +#define APIC_DM_RAR 0x00300#define APIC_DM_NMI 0x00400#define APIC_DM_INIT 0x00500#define APIC_DM_STARTUP 0x00600
arch/x86/include/asm/irq_vectors.h +6 -1
@@ -97,11 +97,16 @@#define LOCAL_TIMER_VECTOR 0xec +/* +* RAR (remote action request) TLB flush +*/ +#define RAR_VECTOR 0xe0 +/* * Posted interrupt notification vector for all device MSIs delivered to * the host kernel. */ -#define POSTED_MSI_NOTIFICATION_VECTOR 0xeb +#define POSTED_MSI_NOTIFICATION_VECTOR 0xdf#define NR_VECTORS 256
arch/x86/include/asm/smp.h +1 -0
@@ -123,6 +123,7 @@ void __noreturn mwait_play_dead(unsigned int eax_hint);void native_smp_send_reschedule (int cpu);void native_send_call_func_ipi (const struct cpumask *mask);void native_send_call_func_single_ipi (int cpu); +void native_send_rar_ipi (const struct cpumask *mask); asmlinkage __visible void smp_reboot_interrupt(void); __visible void smp_reschedule_interrupt(struct pt_regs *regs);
arch/x86/kernel/apic/ipi.c +5 -0
@@ -106,6 +106,11 @@ void apic_send_nmi_to_offline_cpu(unsigned int cpu)return ; apic->send_IPI(cpu, NMI_VECTOR); } + +void native_send_rar_ipi (const struct cpumask *mask) +{ +__apic_send_IPI_mask (mask,RAR_VECTOR ); +}#endif /* CONFIG_SMP */ static inline int __prepare_ICR2 (unsigned int mask)
arch/x86/kernel/apic/local.h +3 -0
@@ -38,6 +38,9 @@ static inline unsigned int __prepare_ICR(unsigned int shortcut, int vector,case NMI_VECTOR : icr |=APIC_DM_NMI ;break ; +case RAR_VECTOR : + icr |=APIC_DM_RAR ; +break ; }return icr; }
Signature · 1 line
2.51.1
RFC v5 5/8 x86/mm: Introduce Remote Action Request
In reply to Rik van RielFrom: Yu-cheng Yu <[email protected]>
Remote Action Request (RAR) is a TLB flushing broadcast facility. To start a TLB flush, the initiator CPU creates a RAR payload and sends a command to the APIC. The receiving CPUs automatically flush TLBs as specified in the payload without the kernel's involement.
[ riel: add pcid parameter to smp_call_rar_many so other mms can be flushed ]
--- arch/x86/include/asm/rar.h | 76 ++++++++++++ arch/x86/kernel/cpu/intel.c | 8 +- arch/x86/mm/Makefile | 1 + arch/x86/mm/rar.c | 236 ++++++++++++++++++++++++++++++++++++ 4 files changed, 320 insertions(+), 1 deletion(-) create mode 100644 arch/x86/include/asm/rar.h create mode 100644 arch/x86/mm/rar.c
arch/x86/include/asm/rar.h new file +76 -0
@@ -0,0 +1,76 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ASM_X86_RAR_H +#define _ASM_X86_RAR_H + +/* +* RAR payload types +*/ +#define RAR_TYPE_INVPG 0 +#define RAR_TYPE_INVPG_NO_CR3 1 +#define RAR_TYPE_INVPCID 2 +#define RAR_TYPE_INVEPT 3 +#define RAR_TYPE_INVVPID 4 +#define RAR_TYPE_WRMSR 5 + +/* +* Subtypes for RAR_TYPE_INVLPG +*/ +#define RAR_INVPG_ADDR 0/* address specific */ +#define RAR_INVPG_ALL 2/* all, include global */ +#define RAR_INVPG_ALL_NO_GLOBAL 3/* all, exclude global */ + +/* +* Subtypes for RAR_TYPE_INVPCID +*/ +#define RAR_INVPCID_ADDR 0/* address specific */ +#define RAR_INVPCID_PCID 1/* all of PCID */ +#define RAR_INVPCID_ALL 2/* all, include global */ +#define RAR_INVPCID_ALL_NO_GLOBAL 3/* all, exclude global */ + +/* +* Page size for RAR_TYPE_INVLPG +*/ +#define RAR_INVLPG_PAGE_SIZE_4K 0 +#define RAR_INVLPG_PAGE_SIZE_2M 1 +#define RAR_INVLPG_PAGE_SIZE_1G 2 + +/* +* Max number of pages per payload +*/ +#define RAR_INVLPG_MAX_PAGES 63 + +struct rar_payload { +u64 for_sw :8 ; +u64 type :8 ; +u64 must_be_zero_1 :16 ; +u64 subtype :3 ; +u64 page_size :2 ; +u64 num_pages :6 ; +u64 must_be_zero_2 :21 ; + +u64 must_be_zero_3; + +/* +* Starting address +*/ +union { +u64 initiator_cr3; +struct { +u64 pcid :12 ; +u64 ignored :52 ; + }; + }; +u64 linear_address; + +/* +* Padding +*/ +u64 padding[4 ]; +}; + +void rar_cpu_init (void ); +void rar_boot_cpu_init (void ); +void smp_call_rar_many (const struct cpumask *mask,u16 pcid, +unsigned long start,unsigned long end); + +#endif /* _ASM_X86_RAR_H */
arch/x86/kernel/cpu/intel.c +7 -1
@@ -22,6 +22,7 @@#include <asm/microcode.h> #include <asm/msr.h> #include <asm/numa.h> +#include <asm/rar.h> #include <asm/resctrl.h> #include <asm/thermal.h> #include <asm/uaccess.h>
@@ -624,6 +625,9 @@ static void init_intel(struct cpuinfo_x86 *c)split_lock_init ();intel_init_thermal (c); + +if (cpu_feature_enabled (X86_FEATURE_RAR )) +rar_cpu_init (); }#ifdef CONFIG_X86_32
@@ -725,8 +729,10 @@ static void intel_detect_tlb(struct cpuinfo_x86 *c)rdmsrl (MSR_IA32_CORE_CAPS , msr); -if (msr &MSR_IA32_CORE_CAPS_RAR ) +if (msr &MSR_IA32_CORE_CAPS_RAR ) {setup_force_cpu_cap (X86_FEATURE_RAR ); +rar_boot_cpu_init (); + } } }
arch/x86/mm/Makefile +1 -0
@@ -52,6 +52,7 @@ obj-$(CONFIG_ACPI_NUMA) += srat.o obj-$(CONFIG_X86_INTEL_MEMORY_PROTECTION_KEYS) += pkeys.o obj-$(CONFIG_RANDOMIZE_MEMORY) += kaslr.o obj-$(CONFIG_MITIGATION_PAGE_TABLE_ISOLATION) += pti.o +obj-$(CONFIG_BROADCAST_TLB_FLUSH) += rar.o obj-$(CONFIG_X86_MEM_ENCRYPT) += mem_encrypt.o obj-$(CONFIG_AMD_MEM_ENCRYPT) += mem_encrypt_amd.o
arch/x86/mm/rar.c new file +236 -0
236 lines not on this page · raw message
Signature · 1 line
2.51.1
kernel test robot
build botRe: [RFC v5 5/8] x86/mm: Introduce Remote Action Request
kernel test robot noticed the following build errors:
Hi Rik,
[This is a private test report for your RFC patch.] kernel test robot noticed the following build errors:
[auto build test ERROR on tip/x86/mm]
[also build test ERROR on tip/master linus/master v6.18-rc6 next-20251121] [cannot apply to tip/x86/core tip/auto-latest]
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Rik-van-Riel/x86-mm-Introduce-Remote-Action-Request-MSRs/20251122-025719 base: tip/x86/mm patch link: https://lore.kernel.org/r/20251121185530.21876-6-riel%40surriel.com patch subject: [RFC v5 5/8] x86/mm: Introduce Remote Action Request config: i386-allnoconfig (https://download.01.org/0day-ci/archive/20251122/[email protected]/config) compiler: gcc-14 (Debian 14.2.0-19) 14.2.0 reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20251122/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All errors (new ones prefixed by >>):
ld: arch/x86/kernel/cpu/intel.o: in function `intel_detect_tlb':
>> intel.c:(.text+0x2c3): undefined reference to `rar_boot_cpu_init'
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
RFC v5 6/8 x86/mm: use RAR for kernel TLB flushes
In reply to Rik van RielFrom: Rik van Riel <[email protected]>
Use Intel RAR for kernel TLB flushes, when enabled.
Pass in PCID 0 to smp_call_rar_many() to flush the specified addresses, regardless of which PCID they might be cached under in any destination CPU.
--- arch/x86/mm/tlb.c | 36 ++++++++++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+)
arch/x86/mm/tlb.c +36 -0
@@ -22,6 +22,7 @@#include <asm/apic.h> #include <asm/msr.h> #include <asm/perf_event.h> +#include <asm/rar.h> #include <asm/tlb.h> #include "mm_internal.h"
@@ -1489,6 +1490,17 @@ static void do_flush_tlb_all(void *info)__flush_tlb_all (); } +static void rar_full_flush (const cpumask_t *cpumask) +{ +guard (preempt)(); +smp_call_rar_many (cpumask,0 ,0 ,TLB_FLUSH_ALL ); +} + +static void rar_flush_all (void ) +{ +rar_full_flush (cpu_online_mask); +} +void flush_tlb_all (void ) {count_vm_tlb_event (NR_TLB_REMOTE_FLUSH );
@@ -1496,6 +1508,8 @@ void flush_tlb_all(void)/* First try (faster) hardware-assisted TLB invalidation. */ if (cpu_feature_enabled (X86_FEATURE_INVLPGB ))invlpgb_flush_all (); +else if (cpu_feature_enabled (X86_FEATURE_RAR )) +rar_flush_all ();else /* Fall back to the IPI-based invalidation. */ on_each_cpu (do_flush_tlb_all,NULL ,1 );
@@ -1525,15 +1539,35 @@ static void do_kernel_range_flush(void *info)struct flush_tlb_info *f = info;unsigned long addr; +/* +* With PTI kernel TLB entries in all PCIDs need to be flushed. +* With RAR the PCID space becomes so large, we might as well flush it all. +* +* Either of the two by itself works with targeted flushes. +*/ +if (cpu_feature_enabled (X86_FEATURE_RAR ) && +cpu_feature_enabled (X86_FEATURE_PTI )) { +invpcid_flush_all (); +return ; + } +/* flush range by one by one 'invlpg' */ for (addr = f->start; addr < f->end; addr +=PAGE_SIZE )flush_tlb_one_kernel (addr); } +static void rar_kernel_range_flush (struct flush_tlb_info *info) +{ +guard (preempt)(); +smp_call_rar_many (cpu_online_mask,0 , info->start, info->end); +} +static void kernel_tlb_flush_all (struct flush_tlb_info *info) {if (cpu_feature_enabled (X86_FEATURE_INVLPGB ))invlpgb_flush_all (); +else if (cpu_feature_enabled (X86_FEATURE_RAR )) +rar_flush_all ();else on_each_cpu (do_flush_tlb_all,NULL ,1 ); }
@@ -1542,6 +1576,8 @@ static void kernel_tlb_flush_range(struct flush_tlb_info *info) {if (cpu_feature_enabled (X86_FEATURE_INVLPGB ))invlpgb_kernel_range_flush (info); +else if (cpu_feature_enabled (X86_FEATURE_RAR )) +rar_kernel_range_flush (info);else on_each_cpu (do_kernel_range_flush, info,1 ); }
Signature · 1 line
2.51.1
RFC v5 7/8 x86/mm: userspace & pageout flushing using Intel RAR
In reply to Rik van RielFrom: Rik van Riel <[email protected]>
Use Intel RAR to flush userspace mappings.
Because RAR flushes are targeted using a cpu bitmap, the rules are a little bit different than for true broadcast TLB invalidation.
For true broadcast TLB invalidation, like done with AMD INVLPGB, a global ASID always has up to date TLB entries on every CPU. The context switch code never has to flush the TLB when switching to a global ASID on any CPU with INVLPGB.
For RAR, the TLB mappings for a global ASID are kept up to date only on CPUs within the mm_cpumask, which lazily follows the threads around the system. The context switch code does not need to flush the TLB if the CPU is in the mm_cpumask, and the PCID used stays the same.
However, a CPU that falls outside of the mm_cpumask can have out of date TLB mappings for this task. When switching to that task on a CPU not in the mm_cpumask, the TLB does need to be flushed.
--- arch/x86/include/asm/tlbflush.h | 9 +- arch/x86/mm/tlb.c | 216 ++++++++++++++++++++++++++------ 2 files changed, 181 insertions(+), 44 deletions(-)
arch/x86/include/asm/tlbflush.h +6 -3
@@ -250,7 +250,8 @@ static inline u16 mm_global_asid(struct mm_struct *mm) {u16 asid; -if (!cpu_feature_enabled (X86_FEATURE_INVLPGB )) +if (!cpu_feature_enabled (X86_FEATURE_INVLPGB ) && + !cpu_feature_enabled (X86_FEATURE_RAR ))return 0 ; asid =smp_load_acquire (&mm->context.global_asid);
@@ -263,7 +264,8 @@ static inline u16 mm_global_asid(struct mm_struct *mm)static inline void mm_init_global_asid (struct mm_struct *mm) { -if (cpu_feature_enabled (X86_FEATURE_INVLPGB )) { +if (cpu_feature_enabled (X86_FEATURE_INVLPGB ) || +cpu_feature_enabled (X86_FEATURE_RAR )) { mm->context.global_asid =0 ; mm->context.asid_transition = false; }
@@ -287,7 +289,8 @@ static inline void mm_clear_asid_transition(struct mm_struct *mm)static inline bool mm_in_asid_transition (struct mm_struct *mm) { -if (!cpu_feature_enabled (X86_FEATURE_INVLPGB )) +if (!cpu_feature_enabled (X86_FEATURE_INVLPGB ) && + !cpu_feature_enabled (X86_FEATURE_RAR ))return false;return mm &&READ_ONCE (mm->context.asid_transition);
arch/x86/mm/tlb.c +175 -41
390 lines not on this page · raw message
Signature · 1 line
2.51.1
RFC v5 8/8 x86/mm: make RAR invalidation scalable by skipping duplicate APIC pokes
In reply to Rik van RielFrom: Rik van Riel <[email protected]>
The naive RAR implementation suffers from heavy contention in apic_mem_wait_irc_idle(), when multiple CPUs send our RAR interrupts simultaneously.
When a CPU receives a RAR, it will scan its action vector, and process all the rar_payload entries where the corresponding action vector is set to RAR_PENDING. After processing each payload, it will set the corresponding action vector to RAR_SUCCESS.
That means sending one single RAR to a CPU is enough for that CPU to process all the pending RAR payloads, and other CPUs do not usually need to send additional RARs to that CPU.
Optimistically avoid sending RAR interrupts to CPUs that are already processing a RAR, looping back only if our request went unprocessed, but the remote CPU is no longer processing any RARs.
This changes will-it-scale tlb_flush2_threads numbers like this:
loops/sec IPI flush naive RAR optimized RAR
threads
1 175k 174k 170k 5 337k 345k 321k 10 530k 469k 497k 20 752k 363k 616k 30 922k 259k 754k 40 1005k 205k 779k 50 1073k 164k 883k 60 1040k 141k 813k
The numbers above are on a 30 core / 60 thread, single socket Sapphire Rapids system. Average of 4 runs.
This exact same code reached up to 1200k loops/second on a -tip kernel from a few weeks ago, and did so reliably across several reboots. I have no good explanation for the difference.
--- arch/x86/mm/rar.c | 60 ++++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 54 insertions(+), 6 deletions(-)
arch/x86/mm/rar.c +54 -6
@@ -11,6 +11,7 @@#include <asm/tlbflush.h> static DEFINE_PER_CPU(struct cpumask, rar_cpu_mask); +static DEFINE_PER_CPU(struct cpumask, apic_cpu_mask);#define RAR_SUCCESS 0x00#define RAR_PENDING 0x01
@@ -47,6 +48,32 @@ static struct rar_lock rar_locks[RAR_MAX_PAYLOADS] __cacheline_aligned;*/ static DEFINE_PER_CPU_ALIGNED(u8[RAR_MAX_PAYLOADS], rar_action); +/* +* Tracks whether a RAR is in flight to this CPU. This is used +* to avoid sending another RAR (waiting on the APIC) when the +* target CPU is already handling RARs. +*/ +static DEFINE_PER_CPU(int, rar_pending) = -1; + +static bool get_rar_pending (int target_cpu,int this_cpu) +{ +int *this_rar_pending = &per_cpu (rar_pending, target_cpu); + +/* Another CPU is flushing this CPU already. */ +if (*this_rar_pending !=-1 ) +return false; + +/* Is this_cpu the one that needs to send a RAR to target_cpu? */ +return cmpxchg (this_rar_pending,-1 , this_cpu) ==-1 ; +} + +static void release_rar_pending (int target_cpu,int this_cpu) +{ +/* If this_cpu sent the RAR to target_cpu, clear rar_pending */ +if (READ_ONCE (per_cpu (rar_pending, target_cpu)) == this_cpu) +WRITE_ONCE (per_cpu (rar_pending, target_cpu),-1 ); +} +/* * TODO: group CPUs together based on locality in the system instead * of CPU number, to further reduce the cost of contention.
@@ -113,7 +140,7 @@ static void set_action_entry(unsigned long payload_nr, int target_cpu)WRITE_ONCE (bitmap[payload_nr],RAR_PENDING ); } -static void wait_for_action_done (unsigned long payload_nr,int target_cpu) +static u8 wait_for_action_done (unsigned long payload_nr,int target_cpu) {u8 status;u8 *rar_actions =per_cpu (rar_action, target_cpu);
@@ -123,9 +150,14 @@ static void wait_for_action_done(unsigned long payload_nr, int target_cpu)while (status ==RAR_PENDING ) {cpu_relax (); status =READ_ONCE (rar_actions[payload_nr]); +/* Target CPU is not processing RARs right now. */ +if (READ_ONCE (per_cpu (rar_pending, target_cpu)) ==-1 ) +return status; }WARN_ON_ONCE (rar_actions[payload_nr] !=RAR_SUCCESS ); + +return status; }void rar_cpu_init (void )
@@ -183,7 +215,7 @@ void smp_call_rar_many(const struct cpumask *mask, u16 pcid, {unsigned long pages = (end - start +PAGE_SIZE ) /PAGE_SIZE ;int cpu, this_cpu =smp_processor_id (); -cpumask_t *dest_mask; +cpumask_t *dest_mask, *apic_mask;unsigned long payload_nr;/* Catch the "end - start + PAGE_SIZE" overflow above. */
@@ -213,7 +245,9 @@ void smp_call_rar_many(const struct cpumask *mask, u16 pcid,* flushes at context switch time. */ dest_mask =this_cpu_ptr (&rar_cpu_mask); + apic_mask =this_cpu_ptr (&apic_cpu_mask);cpumask_and (dest_mask, mask, cpu_online_mask); +cpumask_clear (apic_mask);/* Some callers race with other CPUs changing the passed mask */ if (unlikely (!cpumask_weight (dest_mask)))
@@ -225,11 +259,25 @@ void smp_call_rar_many(const struct cpumask *mask, u16 pcid,for_each_cpu (cpu, dest_mask)set_action_entry (payload_nr, cpu); -/* Send a message to all CPUs in the map */ -native_send_rar_ipi (dest_mask); +do { +for_each_cpu (cpu, dest_mask) { +/* Track the CPUs that have no RAR pending (yet). */ +if (get_rar_pending (cpu, this_cpu)) +__cpumask_set_cpu (cpu, apic_mask); + } -for_each_cpu (cpu, dest_mask) -wait_for_action_done (payload_nr, cpu); +/* Send a message to the CPUs not processing RARs yet */ +native_send_rar_ipi (apic_mask); + +for_each_cpu (cpu, dest_mask) { +u8 status =wait_for_action_done (payload_nr, cpu); +if (status ==RAR_SUCCESS ) { +release_rar_pending (cpu, this_cpu); +__cpumask_clear_cpu (cpu, dest_mask); +__cpumask_clear_cpu (cpu, apic_mask); + } + } + }while (unlikely (cpumask_weight (dest_mask)));free_payload_slot (payload_nr); }
Signature · 1 line
2.51.1