arch/x86: Optionally flush L1D on context switch
Subsystems (4)
v2 cover arch/x86: Optionally flush L1D on context switch
This patch is a continuation of RFC/PoC to start the discussion on optionally flushing L1D cache. The goal is to allow tasks that are paranoid due to the recent snoop assisted data sampling vulnerabilites, to flush their L1D on being switched out. This protects their data from being snooped or leaked via side channels after the task has context switched out.
The points of discussion/review are (with updates):
1. Discuss the use case and the right approach to address this A. Generally there seems to be consensus that we need this
2. Does an arch prctl allowing for opt-in flushing make sense, would other arches care about something similar?
A. We definitely build this for x86, have not heard from any other arch
maintainers. There was suggestion to make this a prctl and let each arch implement L1D flushing if needed, there is no arch agnostic software L1D flush.
3. There is a fallback software L1D load, similar to what L1TF does, but we don't prefetch the TLB, is that sufficient?
A. There was no conclusion, I suspect we don't need this
4. Should we consider cleaning up the L1D on arrival of tasks? A. For now, we think this case is not the priority for this patchset.
In summary, this is an early PoC to start the discussion on the need for conditional L1D flushing based on the security posture of the application and the sensitivity of the data it has access to or might have access to.
Changelog v2:
- Reuse existing code for allocation and flush
- Simplify the goto logic in the actual l1d_flush function
- Optimize the code path with jump labels/static functions
Balbir Singh (4):
arch/x86/kvm: Refactor l1d flush lifecycle management arch/x86: Refactor tlbflush and l1d flush arch/x86: Optionally flush L1D on context switch arch/x86: L1D flush, optimize the context switch
arch/x86/include/asm/cacheflush.h | 6 ++ arch/x86/include/asm/nospec-branch.h | 2 + arch/x86/include/asm/thread_info.h | 4 ++ arch/x86/include/asm/tlbflush.h | 6 ++ arch/x86/include/uapi/asm/prctl.h | 3 + arch/x86/kernel/Makefile | 1 + arch/x86/kernel/l1d_flush.c | 85 ++++++++++++++++++++++++++++ arch/x86/kernel/process_64.c | 10 +++- arch/x86/kvm/vmx/vmx.c | 56 +++--------------- arch/x86/mm/tlb.c | 85 ++++++++++++++++++++++++++++ 10 files changed, 209 insertions(+), 49 deletions(-) create mode 100644 arch/x86/kernel/l1d_flush.c
Signature · 1 line
2.17.1
RFC PATCH v2 1/4 arch/x86/kvm: Refactor l1d flush lifecycle management
Split out the allocation and free routines to be used in a follow up set of patches (to reuse for L1D flushing).
--- arch/x86/include/asm/cacheflush.h | 3 +++ arch/x86/kernel/Makefile | 1 + arch/x86/kernel/l1d_flush.c | 36 +++++++++++++++++++++++++++++++ arch/x86/kvm/vmx/vmx.c | 25 +++------------------ 4 files changed, 43 insertions(+), 22 deletions(-) create mode 100644 arch/x86/kernel/l1d_flush.c
arch/x86/include/asm/cacheflush.h +3 -0
@@ -6,6 +6,9 @@#include <asm-generic/cacheflush.h> #include <asm/special_insns.h> +#define L1D_CACHE_ORDER 4void clflush_cache_range (void *addr,unsigned int size); +void *alloc_l1d_flush_pages (void ); +void cleanup_l1d_flush_pages (void *l1d_flush_pages);#endif /* _ASM_X86_CACHEFLUSH_H */
arch/x86/kernel/Makefile +1 -0
@@ -159,3 +159,4 @@ ifeq ($(CONFIG_X86_64),y) endif obj-$(CONFIG_IMA_SECURE_AND_OR_TRUSTED_BOOT) += ima_arch.o +obj-y += l1d_flush.o
arch/x86/kernel/l1d_flush.c new file +36 -0
@@ -0,0 +1,36 @@ +#include <linux/mm.h> +#include <asm/cacheflush.h> + +void *alloc_l1d_flush_pages (void ) +{ +struct page *page; +void *l1d_flush_pages =NULL ; +int i; + +/* +* This allocation for l1d_flush_pages is not tied to a VM/task's +* lifetime and so should not be charged to a memcg. +*/ + page =alloc_pages (GFP_KERNEL ,L1D_CACHE_ORDER ); +if (!page) +return NULL ; + l1d_flush_pages =page_address (page); + +/* +* Initialize each page with a different pattern in +* order to protect against KSM in the nested +* virtualization case. +*/ +for (i =0 ; i <1u <<L1D_CACHE_ORDER ; ++i) { +memset (l1d_flush_pages + i *PAGE_SIZE , i +1 , +PAGE_SIZE ); + } +return l1d_flush_pages; +} +EXPORT_SYMBOL_GPL (alloc_l1d_flush_pages); + +void cleanup_l1d_flush_pages (void *l1d_flush_pages) +{ +free_pages ((unsigned long )l1d_flush_pages,L1D_CACHE_ORDER ); +} +EXPORT_SYMBOL_GPL (cleanup_l1d_flush_pages);
arch/x86/kvm/vmx/vmx.c +3 -22
@@ -202,14 +202,10 @@ static const struct { [VMENTER_L1D_FLUSH_NOT_REQUIRED] = {"not required", false}, }; -#define L1D_CACHE_ORDER 4static void *vmx_l1d_flush_pages;static int vmx_setup_l1d_flush (enum vmx_l1d_flush_state l1tf) { -struct page *page; -unsigned int i; -if (!boot_cpu_has_bug (X86_BUG_L1TF )) { l1tf_vmx_mitigation =VMENTER_L1D_FLUSH_NOT_REQUIRED ;return 0 ;
@@ -252,24 +248,9 @@ static int vmx_setup_l1d_flush(enum vmx_l1d_flush_state l1tf)if (l1tf !=VMENTER_L1D_FLUSH_NEVER && !vmx_l1d_flush_pages && !boot_cpu_has (X86_FEATURE_FLUSH_L1D )) { -/* -* This allocation for vmx_l1d_flush_pages is not tied to a VM -* lifetime and so should not be charged to a memcg. -*/ - page =alloc_pages (GFP_KERNEL ,L1D_CACHE_ORDER ); -if (!page) + vmx_l1d_flush_pages =alloc_l1d_flush_pages (); +if (!vmx_l1d_flush_pages)return -ENOMEM ; - vmx_l1d_flush_pages =page_address (page); - -/* -* Initialize each page with a different pattern in -* order to protect against KSM in the nested -* virtualization case. -*/ -for (i =0 ; i <1u <<L1D_CACHE_ORDER ; ++i) { -memset (vmx_l1d_flush_pages + i *PAGE_SIZE , i +1 , -PAGE_SIZE ); - } } l1tf_vmx_mitigation = l1tf;
@@ -7999,7 +7980,7 @@ static struct kvm_x86_ops vmx_x86_ops __ro_after_init = {static void vmx_cleanup_l1d_flush (void ) {if (vmx_l1d_flush_pages) { -free_pages ((unsigned long )vmx_l1d_flush_pages,L1D_CACHE_ORDER ); +cleanup_l1d_flush_pages (vmx_l1d_flush_pages); vmx_l1d_flush_pages =NULL ; }/* Restore state so sysfs ignores VMX */
Signature · 1 line
2.17.1
RFC PATCH v2 2/4 arch/x86: Refactor tlbflush and l1d flush
In reply to Balbir SinghRefactor the existing assembly bits into smaller helper functions and also abstract L1D_FLUSH into a helper function. Use these functions in kvm for L1D flushing.
--- arch/x86/include/asm/cacheflush.h | 3 ++ arch/x86/kernel/l1d_flush.c | 49 +++++++++++++++++++++++++++++++ arch/x86/kvm/vmx/vmx.c | 31 ++++--------------- 3 files changed, 57 insertions(+), 26 deletions(-)
arch/x86/include/asm/cacheflush.h +3 -0
@@ -10,5 +10,8 @@void clflush_cache_range (void *addr,unsigned int size);void *alloc_l1d_flush_pages (void );void cleanup_l1d_flush_pages (void *l1d_flush_pages); +void populate_tlb_with_flush_pages (void *l1d_flush_pages); +void flush_l1d_cache_sw (void *l1d_flush_pages); +int flush_l1d_cache_hw (void );#endif /* _ASM_X86_CACHEFLUSH_H */
arch/x86/kernel/l1d_flush.c +49 -0
52 lines not on this page · raw message
arch/x86/kvm/vmx/vmx.c +5 -26
@@ -5963,8 +5963,6 @@ static int vmx_handle_exit(struct kvm_vcpu *vcpu,*/ static void vmx_l1d_flush (struct kvm_vcpu *vcpu) { -int size =PAGE_SIZE <<L1D_CACHE_ORDER ; -/* * This code is only executed when the the flush mode is 'cond' or * 'always'
@@ -5993,32 +5991,13 @@ static void vmx_l1d_flush(struct kvm_vcpu *vcpu) vcpu->stat.l1d_flush++; -if (static_cpu_has (X86_FEATURE_FLUSH_L1D )) { -wrmsrl (MSR_IA32_FLUSH_CMD ,L1D_FLUSH ); +if (flush_l1d_cache_hw ())return ; - } - asmvolatile ( -/* First ensure the pages are in the TLB */ -"xorl %%eax, %%eax\n" -".Lpopulate_tlb:\n\t" -"movzbl (%[flush_pages], %%" _ASM_AX"), %%ecx\n\t" -"addl $4096, %%eax\n\t" -"cmpl %%eax, %[size]\n\t" -"jne .Lpopulate_tlb\n\t" -"xorl %%eax, %%eax\n\t" -"cpuid\n\t" -/* Now fill the cache */ -"xorl %%eax, %%eax\n" -".Lfill_cache:\n" -"movzbl (%[flush_pages], %%" _ASM_AX"), %%ecx\n\t" -"addl $64, %%eax\n\t" -"cmpl %%eax, %[size]\n\t" -"jne .Lfill_cache\n\t" -"lfence\n" - :: [flush_pages]"r" (vmx_l1d_flush_pages), - [size]"r" (size) - :"eax" ,"ebx" ,"ecx" ,"edx" ); +preempt_disable (); +populate_tlb_with_flush_pages (vmx_l1d_flush_pages); +flush_l1d_cache_sw (vmx_l1d_flush_pages); +preempt_enable (); }static void update_cr8_intercept (struct kvm_vcpu *vcpu,int tpr,int irr)
Signature · 1 line
2.17.1
RFC PATCH v2 3/4 arch/x86: Optionally flush L1D on context switch
In reply to Balbir SinghThis patch implements a mechanisn to selectively flush the L1D cache. The goal is to allow tasks that are paranoid due to the recent snoop assisted data sampling vulnerabilites, to flush their L1D on being switched out. This protects their data from being snooped or leaked via side channels after the task has context switched out.
There are two scenarios we might want to protect against, a task leaving the CPU with data still in L1D (which is the main concern of this patch), the second scenario is a malicious task coming in (not so well trusted) for which we want to clean up the cache before it starts
This patch adds an arch specific prctl() to flush L1D cache on context switch out, the existing mechanisms of tracking prev_mm via cpu_tlbstate is reused (very similar to the cond_ipbp() changes). The patch has been lighted tested.
The current version benefited from discussions with Kees and Thomas.
--- arch/x86/include/asm/thread_info.h | 4 ++ arch/x86/include/asm/tlbflush.h | 6 +++ arch/x86/include/uapi/asm/prctl.h | 3 ++ arch/x86/kernel/process_64.c | 10 ++++- arch/x86/mm/tlb.c | 72 ++++++++++++++++++++++++++++++ 5 files changed, 94 insertions(+), 1 deletion(-)
arch/x86/include/asm/thread_info.h +4 -0
@@ -96,6 +96,7 @@ struct thread_info {#define TIF_MEMDIE 20/* is terminating due to OOM killer */ #define TIF_POLLING_NRFLAG 21/* idle is polling for TIF_NEED_RESCHED */ #define TIF_IO_BITMAP 22/* uses I/O bitmap */ +#define TIF_L1D_FLUSH 23/* Flush L1D on mm switches (processes) */ #define TIF_FORCED_TF 24/* true if TF in eflags artificially */ #define TIF_BLOCKSTEP 25/* set when we want DEBUGCTLMSR_BTF */ #define TIF_LAZY_MMU_UPDATES 27/* task is updating the mmu lazily */
@@ -132,6 +133,7 @@ struct thread_info {#define _TIF_ADDR32 (1 << TIF_ADDR32)#define _TIF_X32 (1 << TIF_X32)#define _TIF_FSCHECK (1 << TIF_FSCHECK) +#define _TIF_L1D_FLUSH (1 << TIF_L1D_FLUSH)/* Work to do before invoking the actual syscall. */ #define _TIF_WORK_SYSCALL_ENTRY \
@@ -239,6 +241,8 @@ extern void arch_task_cache_init(void);extern int arch_dup_task_struct (struct task_struct *dst,struct task_struct *src);extern void arch_release_task_struct (struct task_struct *tsk);extern void arch_setup_new_exec (void ); +extern int enable_l1d_flush_for_task (struct task_struct *tsk); +extern int disable_l1d_flush_for_task (struct task_struct *tsk);#define arch_setup_new_exec arch_setup_new_exec#endif /* !__ASSEMBLY__ */
arch/x86/include/asm/tlbflush.h +6 -0
@@ -219,6 +219,12 @@ struct tlb_state {*/ unsigned long cr4; +/* +* Flush the L1D cache on switch_mm_irqs_off() for a +* task getting off the CPU, if it opted in to do so +*/ +bool last_user_mm_l1d_flush; +/* * This is a list of all contexts that might exist in the TLB. * There is one per ASID that we use, and the ASID (what the
arch/x86/include/uapi/asm/prctl.h +3 -0
@@ -14,4 +14,7 @@#define ARCH_MAP_VDSO_32 0x2002#define ARCH_MAP_VDSO_64 0x2003 +#define ARCH_SET_L1D_FLUSH 0x3001 +#define ARCH_GET_L1D_FLUSH 0x3002 +#endif /* _ASM_X86_PRCTL_H */
arch/x86/kernel/process_64.c +9 -1
@@ -700,7 +700,15 @@ long do_arch_prctl_64(struct task_struct *task, int option, unsigned long arg2)case ARCH_MAP_VDSO_64 :return prctl_map_vdso (&vdso_image_64, arg2);#endif - +case ARCH_GET_L1D_FLUSH : +return test_ti_thread_flag (&task->thread_info,TIF_L1D_FLUSH ); +case ARCH_SET_L1D_FLUSH : { +if (arg2 >=1 ) +return enable_l1d_flush_for_task (task); +else +return disable_l1d_flush_for_task (task); +break ; + }default : ret = -EINVAL ;break ;
arch/x86/mm/tlb.c +72 -0
96 lines not on this page · raw message
Signature · 1 line
2.17.1
Quoted 1 line · Balbir Singh
This patch implements a mechanisn to selectively flush the L1D cache.
git grep 'This patch' Documentation/process/
Quoted 13 lines
arch/x86/include/asm/tlbflush.h +5
-0@@ -219,6 +219,12 @@ struct tlb_state {*/ unsigned long cr4; +/* +* Flush the L1D cache on switch_mm_irqs_off() for a +* task getting off the CPU, if it opted in to do so +*/ +bool last_user_mm_l1d_flush;
...
Quoted 5 lines
+/*
+ * Flush the L1D cache for this CPU. We want to this at switch mm time,
+ * this is a pessimistic security measure and an opt-in for those tasks
+ * that host sensitive information and there are concerns about spills
+ * from fill buffers.
Color me confused, but how is L1D flush mitigating fill buffer spills (MFBDS)? The MDS family is mitigated by MD_CLEAR, i.e VERW.
arch/x86/include/asm/tlbflush.h
Quoted 10 lines
+ */
arch/x86/include/asm/tlbflush.h +9
-0+static void l1d_flush (struct mm_struct *next,struct task_struct *tsk) +{ +struct mm_struct *real_prev =this_cpu_read (cpu_tlbstate.loaded_mm); + +/* +* If we are not really switching mm's, we can just return +*/ +if (real_prev == next) +return ;
Instead of having the same check here, please stick the call into the corresponding path in switch_mm_irqs_off(), i.e. where we already have the cond_ibpb() invocation.
arch/x86/include/asm/tlbflush.h
Quoted 2 lines
arch/x86/include/asm/tlbflush.h +2
-0+/* +* Do we need flushing for by the previous task
for by? Perhaps:
Did the previous task request L1D flush when it scheduled in?
arch/x86/include/asm/tlbflush.h
Quoted 2 lines
arch/x86/include/asm/tlbflush.h +2
-0+*/ +if (this_cpu_read (cpu_tlbstate.last_user_mm_l1d_flush) !=0 ) {
This is a bool, so != 0 is pointless.
arch/x86/include/asm/tlbflush.h
Quoted 3 lines
arch/x86/include/asm/tlbflush.h +3
-0+if (!flush_l1d_cache_hw ()) +flush_l1d_cache_sw (l1d_flush_pages); +this_cpu_write (cpu_tlbstate.last_user_mm_l1d_flush,0 );
s/0/false/
arch/x86/include/asm/tlbflush.h
Quoted 10 lines
arch/x86/include/asm/tlbflush.h +10
-0+/* Make sure we clear the values before we set it again */ +barrier (); + } + +if (tsk ==NULL ) +return ; + +/* We don't need stringent checks as we opt-in/opt-out */ +if (test_ti_thread_flag (&tsk->thread_info,TIF_L1D_FLUSH )) +this_cpu_write (cpu_tlbstate.last_user_mm_l1d_flush,1 );
s/1/true/
That aside looking at the gazillion of conditionals here. That's 4 in the worst case. So how about extending cond_ibpb() like the below?
Thanks,
tglx
8<---------------------
arch/x86/include/asm/thread_info.h +2 -1
@@ -84,7 +84,7 @@ struct thread_info {#define TIF_SYSCALL_AUDIT 7/* syscall auditing active */ #define TIF_SECCOMP 8/* secure computing */ #define TIF_SPEC_IB 9/* Indirect branch speculation mitigation */ -#define TIF_SPEC_FORCE_UPDATE 10/* Force speculation MSR update in context switch */ +#define TIF_SPEC_FLUSH_L1D 10/* L1D Flush in switch_mm() */ #define TIF_USER_RETURN_NOTIFY 11/* notify kernel of userspace return */ #define TIF_UPROBE 12/* breakpointed or singlestepping */ #define TIF_PATCH_PENDING 13/* pending live patching update */
@@ -96,6 +96,7 @@ struct thread_info {#define TIF_MEMDIE 20/* is terminating due to OOM killer */ #define TIF_POLLING_NRFLAG 21/* idle is polling for TIF_NEED_RESCHED */ #define TIF_IO_BITMAP 22/* uses I/O bitmap */ +#define TIF_SPEC_FORCE_UPDATE 23/* Force speculation MSR update in context switch */ #define TIF_FORCED_TF 24/* true if TF in eflags artificially */ #define TIF_BLOCKSTEP 25/* set when we want DEBUGCTLMSR_BTF */ #define TIF_LAZY_MMU_UPDATES 27/* task is updating the mmu lazily */
arch/x86/include/asm/tlbflush.h +1 -1
@@ -172,7 +172,7 @@ struct tlb_state {/* Last user mm for optimizing IBPB */ union {struct mm_struct *last_user_mm; -unsigned long last_user_mm_ibpb; +unsigned long last_user_mm_spec; };u16 loaded_mm_asid;
arch/x86/mm/tlb.c +28 -26
113 lines not on this page · raw message
Quoted 7 lines · Thomas Gleixner
Quoted 1 line · Balbir Singh
This patch implements a mechanisn to selectively flush the L1D cache.
git grep 'This patch'
Documentation/process/
I'll get more imperative, thanks!
Quoted 17 lines
Quoted 14 lines
arch/x86/include/asm/tlbflush.h +5
-0@@ -219,6 +219,12 @@ struct tlb_state {*/ unsigned long cr4; +/* +* Flush the L1D cache on switch_mm_irqs_off() for a +* task getting off the CPU, if it opted in to do so +*/ +bool last_user_mm_l1d_flush;...
Quoted 8 lines
Quoted 5 lines
+/*
+ * Flush the L1D cache for this CPU. We want to this at switch mm time,
+ * this is a pessimistic security measure and an opt-in for those tasks
+ * that host sensitive information and there are concerns about spills
+ * from fill buffers.Color me confused, but how is L1D flush mitigating fill buffer spills (MFBDS)? The MDS family is mitigated by
MD_CLEAR, i.e VERW.
I should reword that sentence to say snoop from L1D
Quoted 16 lines
arch/x86/include/asm/tlbflush.h
Quoted 11 lines
+ */
arch/x86/include/asm/tlbflush.h +9
-0+static void l1d_flush (struct mm_struct *next,struct task_struct *tsk) +{ +struct mm_struct *real_prev =this_cpu_read (cpu_tlbstate.loaded_mm); + +/* +* If we are not really switching mm's, we can just return +*/ +if (real_prev == next) +return ;Instead of having the same check here, please stick the call into the corresponding path in
switch_mm_irqs_off(), i.e. where we already have thecond_ibpb()invocation.
Sure, will do
Quoted 4 lines
arch/x86/include/asm/tlbflush.h
Quoted 2 lines
arch/x86/include/asm/tlbflush.h +2
-0+/* +* Do we need flushing for by the previous task for by? Perhaps:
I'll fix that comment up
Quoted 6 lines
Did the previous task request L1D flush when it scheduled in?
arch/x86/include/asm/tlbflush.h
Quoted 2 lines
arch/x86/include/asm/tlbflush.h +2
-0+*/ +if (this_cpu_read (cpu_tlbstate.last_user_mm_l1d_flush) !=0 ) {
Quoted 8 lines
This is a bool, so != 0 is pointless.
arch/x86/include/asm/tlbflush.h
Quoted 3 lines
arch/x86/include/asm/tlbflush.h +3
-0+if (!flush_l1d_cache_hw ()) +flush_l1d_cache_sw (l1d_flush_pages); +this_cpu_write (cpu_tlbstate.last_user_mm_l1d_flush,0 );s/0/false/
Will do
Quoted 13 lines
arch/x86/include/asm/tlbflush.h
Quoted 10 lines
arch/x86/include/asm/tlbflush.h +10
-0+/* Make sure we clear the values before we set it again */ +barrier (); + } + +if (tsk ==NULL ) +return ; + +/* We don't need stringent checks as we opt-in/opt-out */ +if (test_ti_thread_flag (&tsk->thread_info,TIF_L1D_FLUSH )) +this_cpu_write (cpu_tlbstate.last_user_mm_l1d_flush,1 );s/1/true/
Quoted 7 lines
That aside looking at the gazillion of conditionals here. That's 4 in the worst case. So how about extending
cond_ibpb()like the below?Thanks,
tglx
Makes sense, it mostly looks good! Let me refactor the comments and code.
Balbir
Quoted 165 lines
8<---------------------
arch/x86/include/asm/thread_info.h +0
-0@@ -84,7 +84,7 @@ struct thread_info {#define TIF_SYSCALL_AUDIT 7/* syscall auditing active */ #define TIF_SECCOMP 8/* secure computing */ #define TIF_SPEC_IB 9/* Indirect branch speculation mitigation */
-#defineTIF_SPEC_FORCE_UPDATE10 /* Force speculation MSR update in
context switch */
+#defineTIF_SPEC_FLUSH_L1D10 /* L1D Flush inswitch_mm()*/
#defineTIF_USER_RETURN_NOTIFY11 /* notify kernel of userspace return
*/#define TIF_UPROBE 12 /* breakpointed or singlestepping */ #define TIF_PATCH_PENDING 13 /* pending live patching update */arch/x86/include/asm/thread_info.h +0
-0@@ -96,6 +96,7 @@ struct thread_info {#define TIF_MEMDIE 20/* is terminating due to OOM killer */ #define TIF_POLLING_NRFLAG 21/* idle is polling for
TIF_NEED_RESCHED*/#define TIF_IO_BITMAP 22 /* uses I/O bitmap */ +#define TIF_SPEC_FORCE_UPDATE 23 /* Force speculation MSR update incontext switch */
#defineTIF_FORCED_TF24 /* true if TF in eflags artificially
*/
#defineTIF_BLOCKSTEP25 /* set when we wantDEBUGCTLMSR_BTF
*/
#defineTIF_LAZY_MMU_UPDATES27 /* task is updating the mmu lazily
*/arch/x86/include/asm/tlbflush.h +1
-1@@ -172,7 +172,7 @@ struct tlb_state {/* Last user mm for optimizing IBPB */ union {struct mm_struct *last_user_mm; -unsigned long last_user_mm_ibpb; +unsigned long last_user_mm_spec; };u16 loaded_mm_asid;arch/x86/mm/tlb.c +15
-8@@ -33,10 +33,13 @@*/ /* -* Use bit 0 to mangle the TIF_SPEC_IB state into the mm pointer which is -* stored in cpu_tlb_state.last_user_mm_ibpb. +* Bits mangle the TIF_SPEC_* state into the mm pointer which is +* stored in cpu_tlb_state.last_user_mm_spec. */ #define LAST_USER_MM_IBPB 0x1UL +#define LAST_USER_MM_FLUSH_L1D 0x2UL + +#define LAST_USER_MM_SPEC_MASK (LAST_USER_MM_IBPB | LAST_USER_MM_FLUSH_L1D)/* * We get here when we do something requiring a TLB invalidation @@ -189,18 +192,22 @@ static void sync_current_stack_to_mm(str } } -static inline unsigned long mm_mangle_tif_spec_ib (struct task_struct *next) +static inline unsigned long mm_mangle_tif_spec (struct task_struct *next) {unsigned long next_tif =task_thread_info (next)->flags; -unsigned long ibpb = (next_tif >>TIF_SPEC_IB ) &LAST_USER_MM_IBPB ; +unsigned long bits = (next_tif >>TIF_SPEC_IB ) &LAST_USER_MM_SPEC_MASK ; + +BUILD_BUG_ON (TIF_SPEC_FLUSH_L1D !=TIF_SPEC_IB +1 ); -return (unsigned long )next->mm | ibpb; +return (unsigned long )next->mm | bits; } -static void cond_ibpb (struct task_struct *next) +static void cond_mitigations (struct task_struct *next) { -if (!next || !next->mm) -return ; +unsigned long prev_mm, next_mm; + + prev_mm =this_cpu_read (cpu_tlbstate.last_user_mm_spec); + next_mm =mm_mangle_tif_spec (next);/* * Both, the conditional and the always IBPB mode use the mm
arch/x86/mm/tlb.c +13
-18@@ -212,8 +219,6 @@ static void cond_ibpb(struct task_struct* exposed data is not really interesting. */ if (static_branch_likely (&switch_mm_cond_ibpb)) { -unsigned long prev_mm, next_mm; -/* * This is a bit more complex than the always mode because * it has to handle two cases: @@ -243,20 +248,14 @@ static void cond_ibpb(struct task_struct* Optimize this with reasonably small overhead for the * above cases. Mangle the TIF_SPEC_IB bit into the mm * pointer of the incoming task which is stored in -* cpu_tlbstate.last_user_mm_ibpb for comparison. -*/ - next_mm =mm_mangle_tif_spec_ib (next); - prev_mm =this_cpu_read (cpu_tlbstate.last_user_mm_ibpb); - -/* +* cpu_tlbstate.last_user_mm_spec for comparison. +* * Issue IBPB only if the mm's are different and one or * both have the IBPB bit set. */ if (next_mm != prev_mm && (next_mm | prev_mm) &LAST_USER_MM_IBPB )indirect_branch_prediction_barrier (); - -this_cpu_write (cpu_tlbstate.last_user_mm_ibpb, next_mm); }if (static_branch_unlikely (&switch_mm_always_ibpb)) {@@ -265,11 +264,15 @@ static void cond_ibpb(struct task_struct* different context than the user space task which ran * last on this CPU. */ -if (this_cpu_read (cpu_tlbstate.last_user_mm) != next->mm) { +if ((prev_mm & ~LAST_USER_MM_SPEC_MASK ) != + (unsigned long )next->mm)indirect_branch_prediction_barrier (); -this_cpu_write (cpu_tlbstate.last_user_mm, next->mm); - } } + +if (prev_mm &LAST_USER_MM_FLUSH_L1D ) +flush_l1d (); + +this_cpu_write (cpu_tlbstate.last_user_mm_spec, next_mm); } void switch_mm_irqs_off(struct mm_struct *prev, struct mm_struct *next,@@ -371,11 +374,10 @@ void switch_mm_irqs_off(struct mm_struct need_flush = true; }else {/* -* Avoid user/user BTB poisoning by flushing the branch -* predictor when switching between processes. This stops -* one process from doing Spectre-v2 attacks on another. +* Speculation vulnerability mitigations when switching +* to a different user space process. */ -cond_ibpb (tsk); +cond_mitigations (tsk);if (IS_ENABLED (CONFIG_VMAP_STACK )) {/* @@ -501,7 +503,7 @@ void initialize_tlbstate_and_flush(void)write_cr3 (build_cr3 (mm->pgd,0 ));/* Reinitialize tlbstate. */ -this_cpu_write (cpu_tlbstate.last_user_mm_ibpb,LAST_USER_MM_IBPB ); +this_cpu_write (cpu_tlbstate.last_user_mm_spec,LAST_USER_MM_SPEC_MASK );this_cpu_write (cpu_tlbstate.loaded_mm_asid,0 );this_cpu_write (cpu_tlbstate.next_asid,1 );this_cpu_write(cpu_tlbstate.ctxs[0].ctx_id, mm->context.ctx_id);
RFC PATCH v2 4/4 arch/x86: L1D flush, optimize the context switch
In reply to Balbir SinghUse a static branch/jump label to optimize the code. Right now we don't ref count the users, but that could be done if needed in the future.
--- arch/x86/include/asm/nospec-branch.h | 2 ++ arch/x86/mm/tlb.c | 13 +++++++++++++ 2 files changed, 15 insertions(+)
arch/x86/include/asm/nospec-branch.h +2 -0
@@ -310,6 +310,8 @@ DECLARE_STATIC_KEY_FALSE(switch_mm_always_ibpb);DECLARE_STATIC_KEY_FALSE (mds_user_clear);DECLARE_STATIC_KEY_FALSE (mds_idle_clear); +DECLARE_STATIC_KEY_FALSE (switch_mm_l1d_flush); +#include <asm/segment.h> /**
arch/x86/mm/tlb.c +13 -0
@@ -155,6 +155,11 @@ EXPORT_SYMBOL_GPL(leave_mm);static void *l1d_flush_pages;static DEFINE_MUTEX (l1d_flush_mutex); +/* Flush L1D on switch_mm() */ +DEFINE_STATIC_KEY_FALSE (switch_mm_l1d_flush); +EXPORT_SYMBOL_GPL (switch_mm_l1d_flush); + +int enable_l1d_flush_for_task (struct task_struct *tsk) {struct page *page;
@@ -170,6 +175,11 @@ int enable_l1d_flush_for_task(struct task_struct *tsk) l1d_flush_pages =alloc_l1d_flush_pages ();if (!l1d_flush_pages)return -ENOMEM ; +/* +* We could do more accurate ref counting +* if needed +*/ +static_branch_enable (&switch_mm_l1d_flush); }mutex_unlock (&l1d_flush_mutex); }
@@ -195,6 +205,9 @@ static void l1d_flush(struct mm_struct *next, struct task_struct *tsk) {struct mm_struct *real_prev =this_cpu_read (cpu_tlbstate.loaded_mm); +if (static_branch_unlikely (&switch_mm_l1d_flush)) +return ; +/* * If we are not really switching mm's, we can just return */
Signature · 1 line
2.17.1
Re: [RFC PATCH v2 4/4] arch/x86: L1D flush, optimize the context switch
Quoted 6 lines · Balbir Singh
Use a static branch/jump label to optimize the code. Right now we don't ref count the users, but that could be done if needed in the future.
Signed-off-by Balbir Singh <[email protected]>
I sent an older version of the patch, here is the updated version
Balbir Singh
From fa80e9691202183a2879a013e497f221b22305a6 Mon Sep 17 00:00:00 2001
From: Balbir Singh <[email protected]>
Date: Wed, 25 Mar 2020 16:41:18 +1100
Subject: [RFC PATCH v2 4/4] arch/x86: L1D flush, optimize the context switch
Use a static branch/jump label to optimize the code. Right now we don't ref count the users, but that could be done if needed in the future.
--- arch/x86/include/asm/nospec-branch.h | 2 ++ arch/x86/mm/tlb.c | 52 +++++++++++++++++----------- 2 files changed, 34 insertions(+), 20 deletions(-)
arch/x86/include/asm/nospec-branch.h +2 -0
@@ -310,6 +310,8 @@ DECLARE_STATIC_KEY_FALSE(switch_mm_always_ibpb);DECLARE_STATIC_KEY_FALSE (mds_user_clear);DECLARE_STATIC_KEY_FALSE (mds_idle_clear); +DECLARE_STATIC_KEY_FALSE (switch_mm_l1d_flush); +#include <asm/segment.h> /**
arch/x86/mm/tlb.c +32 -20
73 lines not on this page · raw message
Signature · 1 line
2.17.1
Quoted 37 lines · Balbir Singh
This patch is a continuation of RFC/PoC to start the discussion on optionally
flushing L1D cache. The goal is to allow tasks that are paranoid due to the recent snoop assisted data sampling vulnerabilites, to flush their L1D on being
switched out. This protects their data from being snooped or leaked via side channels after the task has context switched out.The points of discussion/review are (with updates):
1. Discuss the use case and the right approach to address this A. Generally there seems to be consensus that we need this
2. Does an arch prctl allowing for opt-in flushing make sense, would other arches care about something similar?
A. We definitely build this for x86, have not heard from any other arch
maintainers. There was suggestion to make this a prctl and let each arch implement L1D flushing if needed, there is no arch agnostic software L1D flush.3. There is a fallback software L1D load, similar to what L1TF does, but we don't prefetch the TLB, is that sufficient?
A. There was no conclusion, I suspect we don't need this4. Should we consider cleaning up the L1D on arrival of tasks? A. For now, we think this case is not the priority for this patchset.
In summary, this is an early PoC to start the discussion on the need for conditional L1D flushing based on the security posture of the application and the sensitivity of the data it has access to or might have access to.
Changelog v2:
- Reuse existing code for allocation and flush
- Simplify the goto logic in the actuall1d_flushfunction
- Optimize the code path with jump labels/static functions
Quoted 8 lines
Balbir Singh (4):
arch/x86/kvm: Refactor l1d flush lifecycle managementarch/x86: Refactor tlbflush and l1d flusharch/x86: Optionally flush L1D on context switcharch/x86: L1D flush, optimize the context switch
Ping, looking for comments and criticism of the approach. I understand with the merge window around the corner everyone is busy. There is a bug in the v2 RFC series, I am happy to post a version without the RFC for broader testing and feedback.
I am quite keen to hear about the interface and any concerns with the arch_prctl() interface.
Balbir Singh.