Intel RAR TLB invalidation
Subsystems (3)
cover Intel RAR TLB invalidation
This patch series adds support for IPI-less TLB invalidation using Intel RAR technology.
Intel RAR differs from AMD INVLPGB in a few ways:
- RAR goes through (emulated?) APIC writes, not instructions
- RAR flushes go through a memory table with 64 entries
- RAR flushes can be targeted to a cpumask
- The RAR functionality must be set up at boot time before it can be used
The cpumask targeting has resulted in Intel RAR and AMD INVLPGB having slightly different rules:
- Processes with dynamic ASIDs use IPI based shootdowns
- INVLPGB: processes with a global ASID
- always have the TLB up to date, on every CPU
- never need to flush the TLB at context switch time
- RAR: processes with global ASIDs
- have the TLB up to date on CPUs in the mm_cpumask
- can skip a TLB flush at context switch time if the CPU is in the mm_cpumask
- need to flush the TLB when scheduled on a cpu not in the mm_cpumask, in case it used to run there before and the TLB has stale entries
RAR functionality is present on Sapphire Rapids and newer CPUs.
Information about Intel RAR can be found in this whitepaper.
This patch series is based off a 2019 patch series created by Intel, with patches later in the series modified to fit into the TLB flush code structure we have after AMD INVLPGB functionality was integrated.
This first version of the code still has issues with segfaults and kernel oopses. Clearly something is still wrong with how or when flushes are done, but the code could use more eyeballs.
I have some idas for additional code cleanups as well, but would like to get the last bugs sorted out first...
RFC PATCH 1/9 x86/mm: Introduce MSR_IA32_CORE_CAPABILITIES
From: Yu-cheng Yu <[email protected]>
MSR_IA32_CORE_CAPABILITIES indicates the existence of other MSRs. Bit[1] indicates Remote Action Request (RAR) TLB registers.
--- arch/x86/include/asm/msr-index.h | 6 ++++++ 1 file changed, 6 insertions(+)
arch/x86/include/asm/msr-index.h +6 -0
@@ -212,6 +212,12 @@* File. */ +#define MSR_IA32_CORE_CAPABILITIES 0x000000cf +#define CORE_CAP_RAR BIT(1)/* +* Remote Action Request. Used to directly +* flush the TLB on remote CPUs. +*/ +#define MSR_IA32_FLUSH_CMD 0x0000010b#define L1D_FLUSH BIT(0)/* * Writeback and invalidate the
Signature · 1 line
2.49.0
RFC PATCH 2/9 x86/mm: Introduce Remote Action Request MSRs
In reply to Rik van RielFrom: Yu-cheng Yu <[email protected]>
Remote Action Request (RAR) is a TLB flushing broadcast facility. This patch introduces RAR MSRs. RAR is introduced in later patches.
There are five RAR MSRs:
MSR_CORE_CAPABILITIES
MSR_IA32_RAR_CTRL
MSR_IA32_RAR_ACT_VEC
MSR_IA32_RAR_PAYLOAD_BASE
MSR_IA32_RAR_INFO
--- arch/x86/include/asm/msr-index.h | 11 +++++++++++ 1 file changed, 11 insertions(+)
arch/x86/include/asm/msr-index.h +11 -0
@@ -122,6 +122,17 @@#define SNB_C3_AUTO_UNDEMOTE (1UL << 27)#define SNB_C1_AUTO_UNDEMOTE (1UL << 28) +/* +* Remote Action Requests (RAR) MSRs +*/ +#define MSR_IA32_RAR_CTRL 0x000000ed +#define MSR_IA32_RAR_ACT_VEC 0x000000ee +#define MSR_IA32_RAR_PAYLOAD_BASE 0x000000ef +#define MSR_IA32_RAR_INFO 0x000000f0 + +#define RAR_CTRL_ENABLE BIT(31) +#define RAR_CTRL_IGNORE_IF BIT(30) +#define MSR_MTRRcap 0x000000fe#define MSR_IA32_ARCH_CAPABILITIES 0x0000010a
Signature · 1 line
2.49.0
RFC PATCH 3/9 x86/mm: enable BROADCAST_TLB_FLUSH on Intel, too
In reply to Rik van RielFrom: Rik van Riel <[email protected]>
Much of the code for Intel RAR and AMD INVLPGB is shared.
Place both under the same config option.
--- arch/x86/Kconfig.cpu | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/Kconfig.cpu +1 -1
@@ -360,7 +360,7 @@ menuconfig PROCESSOR_SELECT config BROADCAST_TLB_FLUSH def_bool y - depends on CPU_SUP_AMD && 64BIT + depends on (CPU_SUP_AMD || CPU_SUP_INTEL) && 64BIT config CPU_SUP_INTEL default y
Signature · 1 line
2.49.0
RFC PATCH 4/9 x86/mm: Introduce X86_FEATURE_RAR
In reply to Rik van RielFrom: Rik van Riel <[email protected]>
Introduce X86_FEATURE_RAR and enumeration of the feature.
[riel: move disabling to Kconfig.cpufeatures]
--- arch/x86/Kconfig.cpufeatures | 4 ++++ arch/x86/include/asm/cpufeatures.h | 2 +- arch/x86/kernel/cpu/common.c | 13 +++++++++++++ 3 files changed, 18 insertions(+), 1 deletion(-)
arch/x86/Kconfig.cpufeatures +4 -0
@@ -199,3 +199,7 @@ config X86_DISABLED_FEATURE_SEV_SNP config X86_DISABLED_FEATURE_INVLPGB def_bool y depends on !BROADCAST_TLB_FLUSH + +config X86_DISABLED_FEATURE_RAR + def_bool y + depends on !BROADCAST_TLB_FLUSH
arch/x86/include/asm/cpufeatures.h +1 -1
@@ -75,7 +75,7 @@#define X86_FEATURE_CENTAUR_MCR ( 3*32+ 3)/* "centaur_mcr" Centaur MCRs (= MTRRs) */ #define X86_FEATURE_K8 ( 3*32+ 4)/* Opteron, Athlon64 */ #define X86_FEATURE_ZEN5 ( 3*32+ 5)/* CPU based on Zen5 microarchitecture */ -/* Free ( 3*32+ 6) */ +#define X86_FEATURE_RAR ( 3*32+ 6)/* Intel Remote Action Request */ /* Free ( 3*32+ 7) */ #define X86_FEATURE_CONSTANT_TSC ( 3*32+ 8)/* "constant_tsc" TSC ticks at a constant rate */ #define X86_FEATURE_UP ( 3*32+ 9)/* "up" SMP kernel running on UP */
arch/x86/kernel/cpu/common.c +13 -0
@@ -1502,6 +1502,18 @@ static void __init cpu_set_bug_bits(struct cpuinfo_x86 *c)setup_force_cpu_bug (X86_BUG_L1TF ); } +static void __init detect_rar (struct cpuinfo_x86 *c) +{ +u64 msr; + +if (cpu_has (c,X86_FEATURE_CORE_CAPABILITIES )) { +rdmsrl (MSR_IA32_CORE_CAPABILITIES , msr); + +if (msr &CORE_CAP_RAR ) +setup_force_cpu_cap (X86_FEATURE_RAR ); + } +} +/* * The NOPL instruction is supposed to exist on all CPUs of family >= 6; * unfortunately, that's not true in practice because of early VIA
@@ -1728,6 +1740,7 @@ static void __init early_identify_cpu(struct cpuinfo_x86 *c)setup_clear_cpu_cap (X86_FEATURE_LA57 );detect_nopl (); +detect_rar (c); }void __init init_cpu_devs (void )
Signature · 1 line
2.49.0
RFC PATCH 5/9 x86/mm: Change cpa_flush() to call flush_kernel_range() directly
In reply to Rik van RielFrom: Rik van Riel <[email protected]>
The function cpa_flush() calls __flush_tlb_one_kernel() and flush_tlb_all().
Replacing that with a call to flush_tlb_kernel_range() allows cpa_flush() to make use of INVLPGB or RAR without any additional changes.
Initialize invlpgb_count_max to 1, since flush_tlb_kernel_range() can now be called before invlpgb_count_max has been initialized to the value read from CPUID.
[riel: remove now unused __cpa_flush_tlb]
--- arch/x86/kernel/cpu/amd.c | 2 +- arch/x86/mm/pat/set_memory.c | 20 +++++++------------- 2 files changed, 8 insertions(+), 14 deletions(-)
arch/x86/kernel/cpu/amd.c +1 -1
@@ -31,7 +31,7 @@#include "cpu.h" -u16 invlpgb_count_max__ro_after_init ; +u16 invlpgb_count_max__ro_after_init =1 ;static inline int rdmsrq_amd_safe (unsigned msr,u64 *p) {
arch/x86/mm/pat/set_memory.c +7 -13
@@ -399,15 +399,6 @@ static void cpa_flush_all(unsigned long cache)on_each_cpu (__cpa_flush_all, (void *) cache,1 ); } -static void __cpa_flush_tlb (void *data) -{ -struct cpa_data *cpa = data; -unsigned int i; - -for (i =0 ; i < cpa->numpages; i++) -flush_tlb_one_kernel (fix_addr (__cpa_addr (cpa, i))); -} -static int collapse_large_pages (unsigned long addr,struct list_head *pgtables);static void cpa_collapse_large_pages (struct cpa_data *cpa)
@@ -444,6 +435,7 @@ static void cpa_collapse_large_pages(struct cpa_data *cpa)static void cpa_flush (struct cpa_data *cpa,int cache) { +unsigned long start, end;unsigned int i;BUG_ON (irqs_disabled () && !early_boot_irqs_disabled);
@@ -453,10 +445,12 @@ static void cpa_flush(struct cpa_data *cpa, int cache) goto collapse_large_pages; } -if (cpa->force_flush_all || cpa->numpages > tlb_single_page_flush_ceiling) -flush_tlb_all (); -else -on_each_cpu (__cpa_flush_tlb, cpa,1 ); + start =fix_addr (__cpa_addr (cpa,0 )); + end =fix_addr (__cpa_addr (cpa, cpa->numpages)); +if (cpa->force_flush_all) + end =TLB_FLUSH_ALL ; + +flush_tlb_kernel_range (start, end);if (!cache) goto collapse_large_pages;
Signature · 1 line
2.49.0
RFC PATCH 6/9 x86/apic: Introduce Remote Action Request Operations
In reply to Rik van RielFrom: Rik van Riel <[email protected]>
RAR TLB flushing is started by sending a command to the APIC. This patch adds Remote Action Request commands.
[riel: move some things around to acount for 6 years of changes]
--- arch/x86/include/asm/apicdef.h | 1 + arch/x86/include/asm/irq_vectors.h | 5 +++++ arch/x86/include/asm/smp.h | 15 +++++++++++++++ arch/x86/kernel/apic/ipi.c | 23 +++++++++++++++++++---- arch/x86/kernel/apic/local.h | 3 +++ arch/x86/kernel/smp.c | 3 +++ 6 files changed, 46 insertions(+), 4 deletions(-)
arch/x86/include/asm/apicdef.h +1 -0
@@ -92,6 +92,7 @@#define APIC_DM_LOWEST 0x00100#define APIC_DM_SMI 0x00200#define APIC_DM_REMRD 0x00300 +#define APIC_DM_RAR 0x00300#define APIC_DM_NMI 0x00400#define APIC_DM_INIT 0x00500#define APIC_DM_STARTUP 0x00600
arch/x86/include/asm/irq_vectors.h +5 -0
@@ -103,6 +103,11 @@*/ #define POSTED_MSI_NOTIFICATION_VECTOR 0xeb +/* +* RAR (remote action request) TLB flush +*/ +#define RAR_VECTOR 0xe0 +#define NR_VECTORS 256#ifdef CONFIG_X86_LOCAL_APIC
arch/x86/include/asm/smp.h +15 -0
@@ -40,6 +40,9 @@ struct smp_ops {void (*send_call_func_ipi)(const struct cpumask *mask);void (*send_call_func_single_ipi)(int cpu); + +void (*send_rar_ipi)(const struct cpumask *mask); +void (*send_rar_single_ipi)(int cpu); };/* Globals due to paravirt */
@@ -100,6 +103,16 @@ static inline void arch_send_call_function_ipi_mask(const struct cpumask *mask) smp_ops.send_call_func_ipi (mask); } +static inline void arch_send_rar_single_ipi (int cpu) +{ + smp_ops.send_rar_single_ipi (cpu); +} + +static inline void arch_send_rar_ipi_mask (const struct cpumask *mask) +{ + smp_ops.send_rar_ipi (mask); +} +void cpu_disable_common (void );void native_smp_prepare_boot_cpu (void );void smp_prepare_cpus_common (void );
@@ -120,6 +133,8 @@ void __noreturn mwait_play_dead(unsigned int eax_hint);void native_smp_send_reschedule (int cpu);void native_send_call_func_ipi (const struct cpumask *mask);void native_send_call_func_single_ipi (int cpu); +void native_send_rar_ipi (const struct cpumask *mask); +void native_send_rar_single_ipi (int cpu); asmlinkage __visible void smp_reboot_interrupt(void); __visible void smp_reschedule_interrupt(struct pt_regs *regs);
arch/x86/kernel/apic/ipi.c +19 -4
@@ -79,7 +79,7 @@ void native_send_call_func_single_ipi(int cpu)__apic_send_IPI (cpu,CALL_FUNCTION_SINGLE_VECTOR ); } -void native_send_call_func_ipi (const struct cpumask *mask) +static void do_native_send_ipi (const struct cpumask *mask,int vector) {if (static_branch_likely (&apic_use_ipi_shorthand)) {unsigned int cpu =smp_processor_id ();
@@ -88,14 +88,19 @@ void native_send_call_func_ipi(const struct cpumask *mask) goto sendmask;if (cpumask_test_cpu (cpu, mask)) -__apic_send_IPI_all (CALL_FUNCTION_VECTOR ); +__apic_send_IPI_all (vector);else if (num_online_cpus () >1 ) -__apic_send_IPI_allbutself (CALL_FUNCTION_VECTOR ); +__apic_send_IPI_allbutself (vector);return ; } sendmask: -__apic_send_IPI_mask (mask,CALL_FUNCTION_VECTOR ); +__apic_send_IPI_mask (mask, vector); +} + +void native_send_call_func_ipi (const struct cpumask *mask) +{ +do_native_send_ipi (mask,CALL_FUNCTION_VECTOR ); }void apic_send_nmi_to_offline_cpu (unsigned int cpu)
@@ -106,6 +111,16 @@ void apic_send_nmi_to_offline_cpu(unsigned int cpu)return ; apic->send_IPI(cpu, NMI_VECTOR); } + +void native_send_rar_single_ipi (int cpu) +{ + apic->send_IPI_mask (cpumask_of (cpu),RAR_VECTOR ); +} + +void native_send_rar_ipi (const struct cpumask *mask) +{ +do_native_send_ipi (mask,RAR_VECTOR ); +}#endif /* CONFIG_SMP */ static inline int __prepare_ICR2 (unsigned int mask)
arch/x86/kernel/apic/local.h +3 -0
@@ -38,6 +38,9 @@ static inline unsigned int __prepare_ICR(unsigned int shortcut, int vector,case NMI_VECTOR : icr |=APIC_DM_NMI ;break ; +case RAR_VECTOR : + icr |=APIC_DM_RAR ; +break ; }return icr; }
arch/x86/kernel/smp.c +3 -0
@@ -297,5 +297,8 @@ struct smp_ops smp_ops = { .send_call_func_ipi = native_send_call_func_ipi, .send_call_func_single_ipi = native_send_call_func_single_ipi, + + .send_rar_ipi = native_send_rar_ipi, + .send_rar_single_ipi = native_send_rar_single_ipi, };EXPORT_SYMBOL_GPL (smp_ops);
Signature · 1 line
2.49.0
RFC PATCH 7/9 x86/mm: Introduce Remote Action Request
In reply to Rik van RielFrom: Yu-cheng Yu <[email protected]>
Remote Action Request (RAR) is a TLB flushing broadcast facility. To start a TLB flush, the initiator CPU creates a RAR payload and sends a command to the APIC. The receiving CPUs automatically flush TLBs as specified in the payload without the kernel's involement.
[ riel:
- add pcid parameter to smp_call_rar_many so other mms can be flushed
- ensure get_payload only allocates valid indices
- make sure rar_cpu_init does not write reserved bits
- fix overflow in range vs full flush decision ]
--- arch/x86/include/asm/rar.h | 69 +++++++++++ arch/x86/kernel/cpu/common.c | 4 + arch/x86/mm/Makefile | 1 + arch/x86/mm/rar.c | 226 +++++++++++++++++++++++++++++++++++ 4 files changed, 300 insertions(+) create mode 100644 arch/x86/include/asm/rar.h create mode 100644 arch/x86/mm/rar.c
arch/x86/include/asm/rar.h new file +69 -0
@@ -0,0 +1,69 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ASM_X86_RAR_H +#define _ASM_X86_RAR_H + +/* +* RAR payload types +*/ +#define RAR_TYPE_INVPG 0 +#define RAR_TYPE_INVPG_NO_CR3 1 +#define RAR_TYPE_INVPCID 2 +#define RAR_TYPE_INVEPT 3 +#define RAR_TYPE_INVVPID 4 +#define RAR_TYPE_WRMSR 5 + +/* +* Subtypes for RAR_TYPE_INVLPG +*/ +#define RAR_INVPG_ADDR 0/* address specific */ +#define RAR_INVPG_ALL 2/* all, include global */ +#define RAR_INVPG_ALL_NO_GLOBAL 3/* all, exclude global */ + +/* +* Subtypes for RAR_TYPE_INVPCID +*/ +#define RAR_INVPCID_ADDR 0/* address specific */ +#define RAR_INVPCID_PCID 1/* all of PCID */ +#define RAR_INVPCID_ALL 2/* all, include global */ +#define RAR_INVPCID_ALL_NO_GLOBAL 3/* all, exclude global */ + +/* +* Page size for RAR_TYPE_INVLPG +*/ +#define RAR_INVLPG_PAGE_SIZE_4K 0 +#define RAR_INVLPG_PAGE_SIZE_2M 1 +#define RAR_INVLPG_PAGE_SIZE_1G 2 + +/* +* Max number of pages per payload +*/ +#define RAR_INVLPG_MAX_PAGES 63 + +typedef struct { +uint64_t for_sw :8 ; +uint64_t type :8 ; +uint64_t must_be_zero_1 :16 ; +uint64_t subtype :3 ; +uint64_t page_size:2 ; +uint64_t num_pages :6 ; +uint64_t must_be_zero_2 :21 ; + +uint64_t must_be_zero_3; + +/* +* Starting address +*/ +uint64_t initiator_cr3; +uint64_t linear_address; + +/* +* Padding +*/ +uint64_t padding[4 ]; +}rar_payload_t ; + +void rar_cpu_init (void ); +void smp_call_rar_many (const struct cpumask *mask,u16 pcid, +unsigned long start,unsigned long end); + +#endif /* _ASM_X86_RAR_H */
arch/x86/kernel/cpu/common.c +4 -0
@@ -71,6 +71,7 @@#include <asm/tdx.h> #include <asm/posted_intr.h> #include <asm/runtime-const.h> +#include <asm/rar.h> #include "cpu.h"
@@ -2395,6 +2396,9 @@ void cpu_init(void)if (is_uv_system ())uv_cpu_init (); +if (cpu_feature_enabled (X86_FEATURE_RAR )) +rar_cpu_init (); +load_fixmap_gdt (cpu); }
arch/x86/mm/Makefile +1 -0
@@ -54,6 +54,7 @@ obj-$(CONFIG_ACPI_NUMA) += srat.o obj-$(CONFIG_X86_INTEL_MEMORY_PROTECTION_KEYS) += pkeys.o obj-$(CONFIG_RANDOMIZE_MEMORY) += kaslr.o obj-$(CONFIG_MITIGATION_PAGE_TABLE_ISOLATION) += pti.o +obj-$(CONFIG_BROADCAST_TLB_FLUSH) += rar.o obj-$(CONFIG_X86_MEM_ENCRYPT) += mem_encrypt.o obj-$(CONFIG_AMD_MEM_ENCRYPT) += mem_encrypt_amd.o
arch/x86/mm/rar.c new file +226 -0
226 lines not on this page · raw message
Signature · 1 line
2.49.0
Re: [RFC PATCH 7/9] x86/mm: Introduce Remote Action Request
Quoted 13 lines · Rik van Riel
arch/x86/mm/rar.c new file +9
-0+void smp_call_rar_many (const struct cpumask *mask,u16 pcid, +unsigned long start,unsigned long end) +{ +unsigned long pages = (end - start +PAGE_SIZE ) /PAGE_SIZE ; +int cpu, next_cpu, this_cpu =smp_processor_id (); +cpumask_t *dest_mask; +unsigned long idx; + +if (pages >RAR_INVLPG_MAX_PAGES || end ==TLB_FLUSH_ALL )
arch/x86/mm/rar.c line 173
Quoted 10 lines
arch/x86/mm/rar.c new file +10
-0+ pages =RAR_INVLPG_MAX_PAGES ; + +/* +* Can deadlock when called with interrupts disabled. +* We allow cpu's that are not yet online though, as no one else can +* send smp call function interrupt to this cpu and as such deadlocks +* can't happen. +*/ +WARN_ON_ONCE (cpu_online (this_cpu) &&irqs_disabled () + && !oops_in_progress && !early_boot_irqs_disabled);
To ease it for the reader, consider using the updated version from smp.c (or - even better - refactor into common inline function):
if (cpu_online(this_cpu) && !oops_in_progress && !early_boot_irqs_disabled) lockdep_assert_irqs_enabled();
Quoted 6 lines
arch/x86/mm/rar.c new file +6
-0+ +/* Try to fastpath. So, what's a CPU they want? Ignoring this one. */ + cpu =cpumask_first_and (mask, cpu_online_mask); +if (cpu == this_cpu) + cpu =cpumask_next_and (cpu, mask, cpu_online_mask); +
arch/x86/mm/rar.c line 189
Quoted 10 lines
arch/x86/mm/rar.c new file +10
-0+/* No online cpus? We're done. */ +if (cpu >= nr_cpu_ids) +return ; + +/* Do we have another CPU which isn't us? */ + next_cpu =cpumask_next_and (cpu, mask, cpu_online_mask); +if (next_cpu == this_cpu) + next_cpu =cpumask_next_and (next_cpu, mask, cpu_online_mask); + +/* Fastpath: do that cpu by itself. */
If you follow my comment (suggestion) about the concurrent flushes, then this part should be moved to be in the same was as done in the updated smp_call_function_many_cond().
IOW, the main difference between this path and the “slow path” is arch_send_rar_ipi_mask() vs arch_send_rar_single_ipi() (and maybe “and” with cpu_online_mask).
Quoted 19 lines
arch/x86/mm/rar.c new file +19
-0+if (next_cpu >= nr_cpu_ids) { +lock (this_cpu_ptr (&rar_lock)); + idx =get_payload (); +set_payload (idx, pcid, start, pages); +set_action_entry (idx, cpu); +arch_send_rar_single_ipi (cpu); +wait_for_done (idx, cpu); +free_payload (idx); +unlock (this_cpu_ptr (&rar_lock)); +return ; + } + + dest_mask =this_cpu_ptr (&rar_cpu_mask); +cpumask_and (dest_mask, mask, cpu_online_mask); +cpumask_clear_cpu (this_cpu, dest_mask); + +/* Some callers race with other cpus changing the passed mask */ +if (unlikely (!cpumask_weight (dest_mask))) +return ;
arch/x86/mm/rar.c line 218
Quoted 10 lines
arch/x86/mm/rar.c new file +10
-0+ +lock (this_cpu_ptr (&rar_lock)); + idx =get_payload (); +set_payload (idx, pcid, start, pages); + +for_each_cpu (cpu, dest_mask) +set_action_entry (idx, cpu); + +/* Send a message to all CPUs in the map */ +arch_send_rar_ipi_mask (dest_mask);
Since 2019 we have move into “multi” TLB flush instead of “many”.
This means that try to take advantage of the time between sending the IPI and the indication it was completed to do the local TLB flush. For both consistency and performance, I recommend you’d follow this approach and do the local TLB flush (if needed) here, instead of doing it in the caller.
arch/x86/mm/rar.c line 224
Quoted 6 lines
arch/x86/mm/rar.c new file +6
-0+ +for_each_cpu (cpu, dest_mask) +wait_for_done (idx, cpu); + +free_payload (idx); +unlock (this_cpu_ptr (&rar_lock));
We don’t do lock/unlock on kernel/smp.c . So I would expect at least a comment as for why it is required.
Quoted 2 lines
(quoted diff) +2
-0+} +EXPORT_SYMBOL(smp_call_rar_many);
Re: [RFC PATCH 7/9] x86/mm: Introduce Remote Action Request
Quoted 32 lines · Nadav Amit
Quoted 26 lines · Rik van Riel
arch/x86/mm/rar.c new file +4
-0+void smp_call_rar_many (const struct cpumask *mask,u16 pcid, + unsigned long start, unsigned long end) +{ + unsigned long pages = (end - start + PAGE_SIZE) /
PAGE_SIZE;arch/x86/mm/rar.c new file +10
-0+int cpu, next_cpu, this_cpu =smp_processor_id (); +cpumask_t *dest_mask; +unsigned long idx; + +if (pages >RAR_INVLPG_MAX_PAGES || end ==TLB_FLUSH_ALL ) + pages =RAR_INVLPG_MAX_PAGES ; + +/* +* Can deadlock when called with interrupts disabled. +* We allow cpu's that are not yet online though, as no one else can
(quoted diff) +1
-0+ * send smp call function interrupt to this cpu and assuch deadlocks
arch/x86/mm/rar.c new file +4
-0+* can't happen. +*/ + WARN_ON_ONCE(cpu_online(this_cpu) && irqs_disabled() + && !oops_in_progress &&Quoted 1 line
!early_boot_irqs_disabled);
To ease it for the reader, consider using the updated version from
smp.c
(or - even better - refactor into common inline function):
Quoted 3 lines
if (cpu_online(this_cpu) && !oops_in_progress && !early_boot_irqs_disabled) lockdep_assert_irqs_enabled();
Nice cleanup. I will change this. Thank you.
Quoted 28 lines
Quoted 19 lines
(quoted diff) +2
-0+ + /* Try to fastpath. So, what's a CPU they want? Ignoringthis one. */
arch/x86/mm/rar.c new file +3
-0+ cpu =cpumask_first_and (mask, cpu_online_mask); +if (cpu == this_cpu) + cpu = cpumask_next_and(cpu, mask,
cpu_online_mask);arch/x86/mm/rar.c new file +9
-0+ +/* No online cpus? We're done. */ +if (cpu >= nr_cpu_ids) +return ; + +/* Do we have another CPU which isn't us? */ + next_cpu =cpumask_next_and (cpu, mask, cpu_online_mask); +if (next_cpu == this_cpu) + next_cpu = cpumask_next_and(next_cpu, mask,
cpu_online_mask);arch/x86/mm/rar.c line 189
Quoted 2 lines
arch/x86/mm/rar.c new file +2
-0+ +/* Fastpath: do that cpu by itself. */ If you follow my comment (suggestion) about the concurrent flushes, then
this part should be moved to be in the same was as done in the updatedsmp_call_function_many_cond().
Quoted 3 lines
IOW, the main difference between this path and the “slow path” is
arch_send_rar_ipi_mask()vsarch_send_rar_single_ipi()(and maybe “and” withcpu_online_mask).
It gets better. Page 8 of the RAR whitepaper tells us that we can simply use RAR to have a CPU send itself TLB flush instructions, and the microcode will do the flush at the same time the other CPUs handle theirs.
"At this point, the ILP may invalidate its own TLB by
signaling RAR to itself in order to invoke the RAR handler locally as well"
I tried this, but things blew up very early in boot, presumably due to the CPU trying to send itself a RAR before it was fully configured to handle them.
The code may need a better decision point than cpu_feature_enabled(X86_FEATURE_RAR) to decide whether or not to use RAR.
Probably something that indicates RAR is actually ready to use on all CPUs.
Quoted 6 lines
Since 2019 we have move into “multi” TLB flush instead of “many”.
This means that try to take advantage of the time between sending the IPI
and the indication it was completed to do the local TLB flush.
I think we have 3 cases here:
1) Only the local TLB needs to be flushed. In this case we can INVPCID locally, and skip any potential contention on the RAR payload table.
2) Only one remote CPU needs to be flushed (no local). This can use the arch_rar_send_single_ipi() thing.
3) Multiple CPUs need to be flushed. This could include the local CPU, or be only multiple remote CPUs. For this case we could just use arch_send_rar_ipi_mask(), including sending a RAR request to the local CPU, which should handle it concurrently with the other CPUs.
Does that seem like a reasonable way to handle things?
Quoted 11 lines
arch/x86/mm/rar.c line 224
Quoted 6 lines
arch/x86/mm/rar.c new file +6
-0+ +for_each_cpu (cpu, dest_mask) +wait_for_done (idx, cpu); + +free_payload (idx); +unlock (this_cpu_ptr (&rar_lock));We don’t do lock/unlock on
kernel/smp.c. So I would expect at least a
comment as for why it is required.
That is a very good question!
It is locking a per-cpu lock, which no other code path takes.
It looks like it could protect against preemption, on a kernel with full preemption enabled, but that should not be needed since the code in arch/x86/mm/tlb.c disables preemption around every call to the RAR code.
I suspect that lock is no longer needed, but maybe somebody at Intel has a reason why we still do?
Signature · 1 line
All Rights Reversed.
Re: [RFC PATCH 7/9] x86/mm: Introduce Remote Action Request
Quoted 2 lines · Rik van Riel
I suspect that lock is no longer needed, but maybe somebody at Intel has a reason why we still do?
I just took a quick look at the locking. It doesn't make any sense to me either.
I suspect it's just plain not needed.
Re: [RFC PATCH 7/9] x86/mm: Introduce Remote Action Request
In reply to Rik van RielQuoted 19 lines · Rik van Riel
It gets better. Page 8 of the RAR whitepaper tells us that we can simply use RAR to have a CPU send itself TLB flush instructions, and the microcode will do the flush at the same time the other CPUs handle theirs.
"At this point, the ILP may invalidate its own TLB by signaling RAR to itself in order to invoke the RAR handler locally as well"
I tried this, but things blew up very early in boot, presumably due to the CPU trying to send itself a RAR before it was fully configured to handle them.
Quoted 7 lines
The code may need a better decision point than
cpu_feature_enabled(X86_FEATURE_RAR)to decide whether or not to use RAR.Probably something that indicates RAR is actually ready to use on all CPUs.
Once you get something working (perhaps with a branch for now) you can take the static-key/static-call path, presumably. I would first try to get something working properly.
BTW: I suspect that the RAR approach might not handle TLB storms worse than the IPI approach, in which once the handler sees such a storm, it does full TLB flush and skips flushes of “older” generations. You may want to benchmark this scenario (IIRC one of the will-it-scale does something similar).
Quoted 5 lines
I think we have 3 cases here:
1) Only the local TLB needs to be flushed. In this case we can INVPCID locally, and skip any potential contention on the RAR payload table.
More like INVLPG (and INVPCID to the user PTI). AFAIK, Andy said INVLPG performs better than INVPCID for a single entry. But yes, this is a simple and hot scenario that should have a separate code-path.
Quoted 11 lines
2) Only one remote CPU needs to be flushed (no local). This can use the
arch_rar_send_single_ipi()thing.3) Multiple CPUs need to be flushed. This could include the local CPU, or be only multiple remote CPUs. For this case we could just use
arch_send_rar_ipi_mask(), including sending a RAR request to the local CPU, which should handle it concurrently with the other CPUs.Does that seem like a reasonable way to handle things?
It it. It is just that code-wise, I think the 2nd and 3rd cases are similar, and it can be better to distinguish the differences between them without creating two completely separate code-paths. This makes maintenance and reasoning more simple, I think.
Consider having a look at smp_call_function_many_cond(). I think it handles the 2nd and 3rd cases nicely in the manner I just described. Admittedly, I am a bit biased…
Re: [RFC PATCH 7/9] x86/mm: Introduce Remote Action Request
Quoted 23 lines · Nadav Amit
Quoted 23 lines · Rik van Riel
It gets better. Page 8 of the RAR whitepaper tells us that we can simply use RAR to have a CPU send itself TLB flush instructions, and the microcode will do the flush at the same time the other CPUs handle theirs.
"At this point, the ILP may invalidate its own TLB by signaling RAR to itself in order to invoke the RAR handler locally as well"
I tried this, but things blew up very early in boot, presumably due to the CPU trying to send itself a RAR before it was fully configured to handle them.
The code may need a better decision point than
cpu_feature_enabled(X86_FEATURE_RAR)to decide whether or not to use RAR.
Quoted 8 lines
Quoted 3 lines
Probably something that indicates RAR is actually ready to use on all CPUs.
Once you get something working (perhaps with a branch for now) you can take the static-key/static-call path, presumably. I would first try to get something working properly.
The static-key code is implemented with alternatives, which call flush_tlb_mm_range.
I've not spent the time digging into whether that creates any chicken-egg scenarios yet :)
Quoted 10 lines
Quoted 5 lines
I think we have 3 cases here:
1) Only the local TLB needs to be flushed. In this case we can INVPCID locally, and skip any potential contention on the RAR payload table.
More like INVLPG (and INVPCID to the user PTI). AFAIK, Andy said INVLPG performs better than INVPCID for a single entry. But yes, this is a simple and hot scenario that should have a separate code-path.
I think this can probably be handled in flush_tlb_mm_range(), so the RAR code is only called for cases (2) and (3) to begin with.
Quoted 13 lines
Quoted 5 lines
2) Only one remote CPU needs to be flushed (no local). This can use the
arch_rar_send_single_ipi()thing.Quoted 7 lines
3) Multiple CPUs need to be flushed. This could include the local CPU, or be only multiple remote CPUs. For this case we could just use
arch_send_rar_ipi_mask(), including sending a RAR request to the local CPU, which should handle it concurrently with the other CPUs.Does that seem like a reasonable way to handle things?
Quoted 8 lines
It it. It is just that code-wise, I think the 2nd and 3rd cases are similar, and it can be better to distinguish the differences between them without creating two completely separate code-paths. This makes maintenance and reasoning more simple, I think.
Consider having a look at
smp_call_function_many_cond(). I think it handles the 2nd and 3rd cases nicely in the manner I just described. Admittedly, I am a bit biased…
I need to use smp_call_function_many_cond() anyway, to prevent sending RARs to CPUs that are in lazy TLB mode (and possibly in a power saving idle state).
IPI TLB flushing and RAR can probably both use the same should_flush_tlb() helper function.
Signature · 1 line
All Rights Reversed.
Re: [RFC PATCH 7/9] x86/mm: Introduce Remote Action Request
In reply to Rik van RielQuoted 55 lines · Rik van Riel
+++
b/arch/x86/include/asm/rar.harch/x86/include/asm/rar.h new file +53
-0@@ -0,0 +1,69 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ASM_X86_RAR_H +#define _ASM_X86_RAR_H + +/* + * RAR payload types + */ +#define RAR_TYPE_INVPG 0 +#define RAR_TYPE_INVPG_NO_CR3 1 +#define RAR_TYPE_INVPCID 2 +#define RAR_TYPE_INVEPT 3 +#define RAR_TYPE_INVVPID 4 +#define RAR_TYPE_WRMSR 5 + +/* + * Subtypes for RAR_TYPE_INVLPG + */ +#define RAR_INVPG_ADDR 0 /* address specific */ +#define RAR_INVPG_ALL 2 /* all, include global */ +#define RAR_INVPG_ALL_NO_GLOBAL 3 /* all, exclude global */ + +/* + * Subtypes for RAR_TYPE_INVPCID + */ +#define RAR_INVPCID_ADDR 0 /* address specific */ +#define RAR_INVPCID_PCID 1 /* all of PCID */ +#define RAR_INVPCID_ALL 2 /* all, include global */ +#define RAR_INVPCID_ALL_NO_GLOBAL 3 /* all, exclude global */ + +/* + * Page size for RAR_TYPE_INVLPG + */ +#define RAR_INVLPG_PAGE_SIZE_4K 0 +#define RAR_INVLPG_PAGE_SIZE_2M 1 +#define RAR_INVLPG_PAGE_SIZE_1G 2 + +/* + * Max number of pages per payload + */ +#define RAR_INVLPG_MAX_PAGES 63 + +typedef struct { + uint64_t for_sw : 8; + uint64_t type : 8; + uint64_t must_be_zero_1 : 16; + uint64_t subtype : 3; + uint64_t page_size: 2; + uint64_t num_pages : 6; + uint64_t must_be_zero_2 : 21; + + uint64_t must_be_zero_3; + + /*
arch/x86/include/asm/rar.h line 63
Quoted 10 lines
arch/x86/include/asm/rar.h new file +10
-0+* Starting address +*/ +uint64_t initiator_cr3; +uint64_t linear_address; + +/* +* Padding +*/ +uint64_t padding[4 ]; +} rar_payload_t;
- Please don't use _t typedefs for complex types. 'struct rar_payload' should be good enough.
- Please use u64/u32 for HW ABI definitions.
- Please align bitfield definitions vertically, for better readability:
u64 for_sw : 8; u64 type : 8; u64 must_be_zero_1 : 16; u64 subtype : 3; u64 page_size : 2; u64 num_pages : 6; u64 must_be_zero_2 : 21;
arch/x86/mm/rar.c line 3
Quoted 5 lines
arch/x86/mm/rar.c new file +4
-0+++ b/arch/x86/mm/rar.c@@ -0,0 +1,226 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* +* RAR Tlb shootdown
s/Tlb
/TLB
arch/x86/mm/rar.c line 6
Quoted 1 line
arch/x86/mm/rar.c new file +1
-0+#include <linux/kgdb.h>
Is this really needed? There's nothing KGDB specific in here AFAICS.
arch/x86/mm/rar.c line 141
Quoted 9 lines
arch/x86/mm/rar.c new file +6
-0+static DEFINE_PER_CPU_ALIGNED(u64[(RAR_MAX_PAYLOADS + 8) / 8], rar_action); +static void set_action_entry (unsigned long idx,int target_cpu) +{ +u8 *bitmap = (u8 *)per_cpu (rar_action, target_cpu); +u8 *bitmap = (u8 *)per_cpu (rar_action, target_cpu); + bitmap = (u8 *)per_cpu (rar_action, this_cpu);
So AFAICS all these ugly, forced type casts tp (u8 *) are needed only because rar_action has the wrong type: if it were an u8[], then these lines could be:
static DEFINE_PER_CPU_ALIGNED(u8[RAR_MAX_PAYLOADS], rar_action); ... u8 *bitmap = per_cpu(rar_action, target_cpu);
right?
Thanks,
Ingo
RFC PATCH 8/9 x86/mm: use RAR for kernel TLB flushes
In reply to Rik van RielFrom: Rik van Riel <[email protected]>
Use Intel RAR for kernel TLB flushes, when enabled.
Pass in PCID 0 to smp_call_rar_many() to flush kernel memory.
Unfortunately RAR_INVPCID_ADDR excludes global PTE mappings, so only full flushes with RAR_INVPCID_ALL will flush kernel mappings.
--- arch/x86/mm/tlb.c | 38 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 38 insertions(+)
arch/x86/mm/tlb.c +38 -0
@@ -21,6 +21,7 @@#include <asm/apic.h> #include <asm/msr.h> #include <asm/perf_event.h> +#include <asm/rar.h> #include <asm/tlb.h> #include "mm_internal.h"
@@ -1451,6 +1452,18 @@ static void do_flush_tlb_all(void *info)__flush_tlb_all (); } +static void rar_full_flush (const cpumask_t *cpumask) +{ +guard (preempt)(); +smp_call_rar_many (cpumask,0 ,0 ,TLB_FLUSH_ALL ); +invpcid_flush_all (); +} + +static void rar_flush_all (void ) +{ +rar_full_flush (cpu_online_mask); +} +void flush_tlb_all (void ) {count_vm_tlb_event (NR_TLB_REMOTE_FLUSH );
@@ -1458,6 +1471,8 @@ void flush_tlb_all(void)/* First try (faster) hardware-assisted TLB invalidation. */ if (cpu_feature_enabled (X86_FEATURE_INVLPGB ))invlpgb_flush_all (); +else if (cpu_feature_enabled (X86_FEATURE_RAR )) +rar_flush_all ();else /* Fall back to the IPI-based invalidation. */ on_each_cpu (do_flush_tlb_all,NULL ,1 );
@@ -1487,15 +1502,36 @@ static void do_kernel_range_flush(void *info)struct flush_tlb_info *f = info;unsigned long addr; +/* +* With PTI kernel TLB entries in all PCIDs need to be flushed. +* With RAR the PCID space becomes so large, we might as well flush it all. +* +* Either of the two by itself works with targeted flushes. +*/ +if (cpu_feature_enabled (X86_FEATURE_RAR ) && +cpu_feature_enabled (X86_FEATURE_PTI )) { +invpcid_flush_all (); +return ; + } +/* flush range by one by one 'invlpg' */ for (addr = f->start; addr < f->end; addr +=PAGE_SIZE )flush_tlb_one_kernel (addr); } +static void rar_kernel_range_flush (struct flush_tlb_info *info) +{ +guard (preempt)(); +smp_call_rar_many (cpu_online_mask,0 , info->start, info->end); +do_kernel_range_flush (info); +} +static void kernel_tlb_flush_all (struct flush_tlb_info *info) {if (cpu_feature_enabled (X86_FEATURE_INVLPGB ))invlpgb_flush_all (); +else if (cpu_feature_enabled (X86_FEATURE_RAR )) +rar_flush_all ();else on_each_cpu (do_flush_tlb_all,NULL ,1 ); }
@@ -1504,6 +1540,8 @@ static void kernel_tlb_flush_range(struct flush_tlb_info *info) {if (cpu_feature_enabled (X86_FEATURE_INVLPGB ))invlpgb_kernel_range_flush (info); +else if (cpu_feature_enabled (X86_FEATURE_RAR )) +rar_kernel_range_flush (info);else on_each_cpu (do_kernel_range_flush, info,1 ); }
Signature · 1 line
2.49.0
RFC PATCH 9/9 x86/mm: userspace & pageout flushing using Intel RAR
In reply to Rik van RielFrom: Rik van Riel <[email protected]>
Use Intel RAR to flush userspace mappings.
Because RAR flushes are targeted using a cpu bitmap, the rules are a little bit different than for true broadcast TLB invalidation.
For true broadcast TLB invalidation, like done with AMD INVLPGB, a global ASID always has up to date TLB entries on every CPU. The context switch code never has to flush the TLB when switching to a global ASID on any CPU with INVLPGB.
For RAR, the TLB mappings for a global ASID are kept up to date only on CPUs within the mm_cpumask, which lazily follows the threads around the system. The context switch code does not need to flush the TLB if the CPU is in the mm_cpumask, and the PCID used stays the same.
However, a CPU that falls outside of the mm_cpumask can have out of date TLB mappings for this task. When switching to that task on a CPU not in the mm_cpumask, the TLB does need to be flushed.
--- arch/x86/include/asm/tlbflush.h | 9 ++- arch/x86/mm/tlb.c | 119 +++++++++++++++++++++++++------- 2 files changed, 99 insertions(+), 29 deletions(-)
arch/x86/include/asm/tlbflush.h +6 -3
@@ -250,7 +250,8 @@ static inline u16 mm_global_asid(struct mm_struct *mm) {u16 asid; -if (!cpu_feature_enabled (X86_FEATURE_INVLPGB )) +if (!cpu_feature_enabled (X86_FEATURE_INVLPGB ) && + !cpu_feature_enabled (X86_FEATURE_RAR ))return 0 ; asid =smp_load_acquire (&mm->context.global_asid);
@@ -263,7 +264,8 @@ static inline u16 mm_global_asid(struct mm_struct *mm)static inline void mm_init_global_asid (struct mm_struct *mm) { -if (cpu_feature_enabled (X86_FEATURE_INVLPGB )) { +if (!cpu_feature_enabled (X86_FEATURE_INVLPGB ) && + !cpu_feature_enabled (X86_FEATURE_RAR )) { mm->context.global_asid =0 ; mm->context.asid_transition = false; }
@@ -287,7 +289,8 @@ static inline void mm_clear_asid_transition(struct mm_struct *mm)static inline bool mm_in_asid_transition (struct mm_struct *mm) { -if (!cpu_feature_enabled (X86_FEATURE_INVLPGB )) +if (!cpu_feature_enabled (X86_FEATURE_INVLPGB ) && + !cpu_feature_enabled (X86_FEATURE_RAR ))return false;return mm &&READ_ONCE (mm->context.asid_transition);
arch/x86/mm/tlb.c +93 -26
224 lines not on this page · raw message
Signature · 1 line
2.49.0