Skip to contentSkip to conversation

Next revision of the L1D flush patches

PATCH Applied Balbir Singh 6 participants 21 messages Jan 8, 2021 – Jul 28, 2021 1 message not in the archive

Subsystems (6)
lore
Share & tools
lore
#0 lore raw

v4 cover Next revision of the L1D flush patches

Implement a mechanism that allows tasks to conditionally flush their L1D cache (mitigation mechanism suggested in [2]). The previous posts of these patches were sent for inclusion (see [3]) and were not included due to the concern for the need for additional checks, those checks were:

1. Implement this mechanism only for CPUs affected by the L1TF bug
2. Disable the software fallback
3. Provide an override to enable this mechanism
4. Be SMT aware in the implementation

The patches support a use case where the entire system is not in non SMT mode, but rather a few CPUs can have their SMT turned off and processes that want to opt-in are expected to run on non SMT cores. This gives the administrator complete control over setting up the mitigation for the issue. In addition, the administrator has a boot time override (l1d_flush=on) to turn on the mechanism without which this mechanism will not work.

To implement these efficiently, a new per cpu view of whether the core is in SMT mode or not is implemented in patch 1. The code is refactored in patch 2 so that the existing code can allow for other speculation related checks when switching mm between tasks, this mechanism has not changed since the last post. The ability to flush L1D for tasks if the TIF_SPEC_L1D_FLUSH bit is set and the task has context switched out of a non SMT core is provided by patch 3. Hooks for the user space API, for this feature to be invoked via prctl are provided in patch 4, along with the checks described above (1, 2, and 3). Documentation updates are in patch 5, with updates on l1d_flush, the prctl changes and updates to the kernel-parameters (l1d_flush_out).

The checks for opting into L1D flushing are:
a. If the CPU is affected by L1TF
b. Hardware L1D flush mechanism is available

A task running on a core with SMT enabled and opting into this feature will receive a SIGBUS.

References
[1] https://software.intel.com/security-software-guidance/software-guidance/snoop-assisted-l1-data-sampling
[2] https://software.intel.com/security-software-guidance/insights/deep-dive-snoop-assisted-l1-data-sampling
[3] https://lkml.org/lkml/2020/6/2/1150
[4] https://lore.kernel.org/lkml/[email protected]/
[5] https://lore.kernel.org/lkml/[email protected]/

Reviewers guide to v4
- The key patch in the series and most of the changes to this revision are to patch 4. patches 3 and 5 have been modified to keep them consistent with the changes to patch 4.

Changelog v4:
- Use a static key to enable the mechanism (remove overheads)
- By default have the mechanism turned off, so there are two opt-ins needed, one by the administrator at boot time, second by the application
- Rename l1d_flush_out/L1D_FLUSH_OUT to l1d_flush/L1D_FLUSH
- Implement other review recommendations
Changelog v3:
- Implement the SIGBUS mechansim
- Update and fix the documentation

Balbir Singh (5):
x86/smp: Add a per-cpu view of SMT state x86/mm: Refactor cond_ibpb() to support other use cases x86/mm: Optionally flush L1D on context switch
prctl: Hook L1D flushing in via prctl
Documentation: Add L1D flushing Documentation

 Documentation/admin-guide/hw-vuln/index.rst   |  1 +
 .../admin-guide/hw-vuln/l1d_flush.rst         | 70 +++++++++++++++
 .../admin-guide/kernel-parameters.txt         | 17 ++++
 Documentation/userspace-api/spec_ctrl.rst     |  8 ++
 arch/Kconfig                                  |  4 +
 arch/x86/Kconfig                              |  1 +
 arch/x86/include/asm/cacheflush.h             |  8 ++
 arch/x86/include/asm/nospec-branch.h          |  2 +
 arch/x86/include/asm/processor.h              |  2 +
 arch/x86/include/asm/thread_info.h            |  6 +-
 arch/x86/include/asm/tlbflush.h               |  2 +-
 arch/x86/kernel/cpu/bugs.c                    | 71 +++++++++++++++
 arch/x86/kernel/smpboot.c                     | 10 ++-
 arch/x86/mm/tlb.c                             | 88 ++++++++++++++-----
 include/linux/sched.h                         | 10 +++
 include/uapi/linux/prctl.h                    |  1 +
 16 files changed, 273 insertions(+), 28 deletions(-)
 create mode 100644 Documentation/admin-guide/hw-vuln/l1d_flush.rst
Signature · 1 line
2.17.1
#1 lore raw

PATCH v4 1/5 x86/smp: Add a per-cpu view of SMT state

A new field smt_active in cpuinfo_x86 identifies if the current core/cpu is in SMT mode or not. This can be very helpful if the system has some of its cores with threads offlined and can be used for cases where action is taken based on the state of SMT. The follow up patches use this feature.

---
 arch/x86/include/asm/processor.h |  2 ++
 arch/x86/kernel/smpboot.c        | 10 +++++++++-
 2 files changed, 11 insertions(+), 1 deletion(-)
arch/x86/include/asm/processor.h +2 -0
index c20a52b5534b..a411466a6e74 100644
@@ -136,6 +136,8 @@ struct cpuinfo_x86 {
 	u16			logical_die_id;
 	/* Index into per_cpu list: */
 	u16			cpu_index;
+	/*  Is SMT active on this core? */
+	bool			smt_active;
 	u32			microcode;
 	/* Address space bits used by the cache internally */
 	u8			x86_cache_bits;
arch/x86/kernel/smpboot.c +9 -1
index 8ca66af96a54..5f6df298d785 100644
@@ -640,6 +640,9 @@ void set_cpu_sibling_map(int cpu)
 	threads = cpumask_weight(topology_sibling_cpumask(cpu));
 	if (threads > __max_smt_threads)
 		__max_smt_threads = threads;
+
+	for_each_cpu(i, topology_sibling_cpumask(cpu))
+		cpu_data(i).smt_active = threads > 1;
 }
 
 /* maps the cpu to the sched domain representing multi-core */
@@ -1551,8 +1554,13 @@ static void remove_siblinginfo(int cpu)
 
 	for_each_cpu(sibling, topology_die_cpumask(cpu))
 		cpumask_clear_cpu(cpu, topology_die_cpumask(sibling));
-	for_each_cpu(sibling, topology_sibling_cpumask(cpu))
+
+	for_each_cpu(sibling, topology_sibling_cpumask(cpu)) {
 		cpumask_clear_cpu(cpu, topology_sibling_cpumask(sibling));
+		if (cpumask_weight(topology_sibling_cpumask(sibling)) == 1)
+			cpu_data(sibling).smt_active = false;
+	}
+
 	for_each_cpu(sibling, cpu_llc_shared_mask(cpu))
 		cpumask_clear_cpu(cpu, cpu_llc_shared_mask(sibling));
 	cpumask_clear(cpu_llc_shared_mask(cpu));
Signature · 1 line
2.17.1
merge bot Applied
#2 lore raw

[tip: x86/cpu] x86/smp: Add a per-cpu view of SMT state

The following commit has been merged into the x86/cpu branch of tip:

The following commit has been merged into the x86/cpu branch of tip:

Commit-ID:     c52787b590634646d4da3d8f23c4532ba050d40d
Gitweb:        https://git.kernel.org/tip/c52787b590634646d4da3d8f23c4532ba050d40d
Author:        Balbir Singh <[email protected]>
AuthorDate:    Fri, 08 Jan 2021 23:10:52 +11:00
Committer:     Thomas Gleixner <[email protected]>
CommitterDate: Wed, 28 Jul 2021 11:42:23 +02:00

x86/smp: Add a per-cpu view of SMT state

A new field smt_active in cpuinfo_x86 identifies if the current core/cpu is in SMT mode or not.

This is helpful when the system has some of its cores with threads offlined and can be used for cases where action is taken based on the state of SMT.

The upcoming support for paranoid L1D flush will make use of this information.

---
 arch/x86/include/asm/processor.h |  2 ++
 arch/x86/kernel/smpboot.c        | 10 +++++++++-
 2 files changed, 11 insertions(+), 1 deletion(-)
arch/x86/include/asm/processor.h +2 -0
index f3020c5..1e0d13c 100644
@@ -136,6 +136,8 @@ struct cpuinfo_x86 {
 	u16			logical_die_id;
 	/* Index into per_cpu list: */
 	u16			cpu_index;
+	/*  Is SMT active on this core? */
+	bool			smt_active;
 	u32			microcode;
 	/* Address space bits used by the cache internally */
 	u8			x86_cache_bits;
arch/x86/kernel/smpboot.c +9 -1
index 9320285..85f6e24 100644
@@ -610,6 +610,9 @@ void set_cpu_sibling_map(int cpu)
 	if (threads > __max_smt_threads)
 		__max_smt_threads = threads;
 
+	for_each_cpu(i, topology_sibling_cpumask(cpu))
+		cpu_data(i).smt_active = threads > 1;
+
 	/*
 	 * This needs a separate iteration over the cpus because we rely on all
 	 * topology_sibling_cpumask links to be set-up.
@@ -1552,8 +1555,13 @@ static void remove_siblinginfo(int cpu)
 
 	for_each_cpu(sibling, topology_die_cpumask(cpu))
 		cpumask_clear_cpu(cpu, topology_die_cpumask(sibling));
-	for_each_cpu(sibling, topology_sibling_cpumask(cpu))
+
+	for_each_cpu(sibling, topology_sibling_cpumask(cpu)) {
 		cpumask_clear_cpu(cpu, topology_sibling_cpumask(sibling));
+		if (cpumask_weight(topology_sibling_cpumask(sibling)) == 1)
+			cpu_data(sibling).smt_active = false;
+	}
+
 	for_each_cpu(sibling, cpu_llc_shared_mask(cpu))
 		cpumask_clear_cpu(cpu, cpu_llc_shared_mask(sibling));
 	cpumask_clear(cpu_llc_shared_mask(cpu));
#3 lore raw

PATCH v4 2/5 x86/mm: Refactor cond_ibpb() to support other use cases

In reply to Balbir Singh

cond_ibpb() has the necessary bits required to track the previous mm in switch_mm_irqs_off(). This can be reused for other use cases like L1D flushing on context switch.

[ tglx: Moved comment, added a separate define for state (re)initialization ]

---
 arch/x86/include/asm/tlbflush.h |  2 +-
 arch/x86/mm/tlb.c               | 53 ++++++++++++++++++---------------
 2 files changed, 30 insertions(+), 25 deletions(-)
arch/x86/include/asm/tlbflush.h +1 -1
index 8c87a2e0b660..a927d40664df 100644
@@ -83,7 +83,7 @@ struct tlb_state {
 	/* Last user mm for optimizing IBPB */
 	union {
 		struct mm_struct	*last_user_mm;
-		unsigned long		last_user_mm_ibpb;
+		unsigned long		last_user_mm_spec;
 	};
 
 	u16 loaded_mm_asid;
arch/x86/mm/tlb.c +29 -24

116 lines not on this page · raw message

Signature · 1 line
2.17.1
merge bot Applied
#4 lore raw

[tip: x86/cpu] x86/mm: Refactor cond_ibpb() to support other use cases

The following commit has been merged into the x86/cpu branch of tip:

The following commit has been merged into the x86/cpu branch of tip:

Commit-ID:     371b09c6fdc436f2c7bb67fc90df5eec8ce90f06
Gitweb:        https://git.kernel.org/tip/371b09c6fdc436f2c7bb67fc90df5eec8ce90f06
Author:        Balbir Singh <[email protected]>
AuthorDate:    Fri, 08 Jan 2021 23:10:53 +11:00
Committer:     Thomas Gleixner <[email protected]>
CommitterDate: Wed, 28 Jul 2021 11:42:24 +02:00

x86/mm: Refactor cond_ibpb() to support other use cases

cond_ibpb() has the necessary bits required to track the previous mm in switch_mm_irqs_off(). This can be reused for other use cases like L1D flushing on context switch.

---
 arch/x86/include/asm/tlbflush.h |  2 +-
 arch/x86/mm/tlb.c               | 53 +++++++++++++++++---------------
 2 files changed, 30 insertions(+), 25 deletions(-)
arch/x86/include/asm/tlbflush.h +1 -1
index fa952ea..b587a9e 100644
@@ -83,7 +83,7 @@ struct tlb_state {
 	/* Last user mm for optimizing IBPB */
 	union {
 		struct mm_struct	*last_user_mm;
-		unsigned long		last_user_mm_ibpb;
+		unsigned long		last_user_mm_spec;
 	};
 
 	u16 loaded_mm_asid;
arch/x86/mm/tlb.c +29 -24

116 lines not on this page · raw message

#5 lore raw

PATCH v4 3/5 x86/mm: Optionally flush L1D on context switch

In reply to Balbir Singh

Implement a mechanism to selectively flush the L1D cache. The goal is to allow tasks that want to save sensitive information, found by the recent snoop assisted data sampling vulnerabilites, to flush their L1D on being switched out. This protects their data from being snooped or leaked via side channels after the task has context switched out.

There are two scenarios we might want to protect against, a task leaving the CPU with data still in L1D (which is the main concern of this patch), the second scenario is a malicious task coming in (not so well trusted) for which we want to clean up the cache before it starts. Only the case for the former is addressed.

A new thread_info flag TIF_SPEC_L1D_FLUSH is added to track tasks which opt-into L1D flushing. cpu_tlbstate.last_user_mm_spec is used to convert the TIF flags into mm state (per cpu via last_user_mm_spec) in cond_mitigation(), which then used to do decide when to flush the L1D cache.

A new helper inline function l1d_flush_hw() has been introduced. Currently it returns an error code if hardware flushing is not supported. The caller currently does not check the return value, in the context of these patches, the routine is called only when HW assisted flushing is available.

---
 arch/x86/include/asm/cacheflush.h  |  8 ++++++++
 arch/x86/include/asm/thread_info.h |  9 +++++++--
 arch/x86/mm/tlb.c                  | 16 ++++++++++++++--
 3 files changed, 29 insertions(+), 4 deletions(-)
arch/x86/include/asm/cacheflush.h +8 -0
index b192d917a6d0..554eaf697f3f 100644
@@ -10,4 +10,12 @@
 
 void clflush_cache_range(void *addr, unsigned int size);
 
+static inline int l1d_flush_hw(void)
+{
+	if (static_cpu_has(X86_FEATURE_FLUSH_L1D)) {
+		wrmsrl(MSR_IA32_FLUSH_CMD, L1D_FLUSH);
+		return 0;
+	}
+	return -EOPNOTSUPP;
+}
 #endif /* _ASM_X86_CACHEFLUSH_H */
arch/x86/include/asm/thread_info.h +7 -2
index 0d751d5da702..33b637442b9e 100644
@@ -81,7 +81,7 @@ struct thread_info {
 #define TIF_SINGLESTEP		4	/* reenable singlestep on user return*/
 #define TIF_SSBD		5	/* Speculative store bypass disable */
 #define TIF_SPEC_IB		9	/* Indirect branch speculation mitigation */
-#define TIF_SPEC_FORCE_UPDATE	10	/* Force speculation MSR update in context switch */
+#define TIF_SPEC_L1D_FLUSH	10	/* Flush L1D on mm switches (processes) */
 #define TIF_USER_RETURN_NOTIFY	11	/* notify kernel of userspace return */
 #define TIF_UPROBE		12	/* breakpointed or singlestepping */
 #define TIF_PATCH_PENDING	13	/* pending live patching update */
@@ -93,6 +93,7 @@ struct thread_info {
 #define TIF_MEMDIE		20	/* is terminating due to OOM killer */
 #define TIF_POLLING_NRFLAG	21	/* idle is polling for TIF_NEED_RESCHED */
 #define TIF_IO_BITMAP		22	/* uses I/O bitmap */
+#define TIF_SPEC_FORCE_UPDATE	23	/* Force speculation MSR update in context switch */
 #define TIF_FORCED_TF		24	/* true if TF in eflags artificially */
 #define TIF_BLOCKSTEP		25	/* set when we want DEBUGCTLMSR_BTF */
 #define TIF_LAZY_MMU_UPDATES	27	/* task is updating the mmu lazily */
@@ -104,7 +105,7 @@ struct thread_info {
 #define _TIF_SINGLESTEP		(1 << TIF_SINGLESTEP)
 #define _TIF_SSBD		(1 << TIF_SSBD)
 #define _TIF_SPEC_IB		(1 << TIF_SPEC_IB)
-#define _TIF_SPEC_FORCE_UPDATE	(1 << TIF_SPEC_FORCE_UPDATE)
+#define _TIF_SPEC_L1D_FLUSH	(1 << TIF_SPEC_L1D_FLUSH)
 #define _TIF_USER_RETURN_NOTIFY	(1 << TIF_USER_RETURN_NOTIFY)
 #define _TIF_UPROBE		(1 << TIF_UPROBE)
 #define _TIF_PATCH_PENDING	(1 << TIF_PATCH_PENDING)
@@ -115,6 +116,7 @@ struct thread_info {
 #define _TIF_SLD		(1 << TIF_SLD)
 #define _TIF_POLLING_NRFLAG	(1 << TIF_POLLING_NRFLAG)
 #define _TIF_IO_BITMAP		(1 << TIF_IO_BITMAP)
+#define _TIF_SPEC_FORCE_UPDATE	(1 << TIF_SPEC_FORCE_UPDATE)
 #define _TIF_FORCED_TF		(1 << TIF_FORCED_TF)
 #define _TIF_BLOCKSTEP		(1 << TIF_BLOCKSTEP)
 #define _TIF_LAZY_MMU_UPDATES	(1 << TIF_LAZY_MMU_UPDATES)
@@ -217,6 +219,9 @@ static inline int arch_within_stack_frames(const void * const stack,
 			   current_thread_info()->status & TS_COMPAT)
 #endif
 
+extern int enable_l1d_flush_for_task(struct task_struct *tsk);
+extern int disable_l1d_flush_for_task(struct task_struct *tsk);
+
 extern void arch_task_cache_init(void);
 extern int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src);
 extern void arch_release_task_struct(struct task_struct *tsk);
arch/x86/mm/tlb.c +14 -2
index 7320348b5a61..f67c5bd58158 100644
@@ -8,11 +8,13 @@
 #include <linux/export.h>
 #include <linux/cpu.h>
 #include <linux/debugfs.h>
+#include <linux/sched/smt.h>
 
 #include <asm/tlbflush.h>
 #include <asm/mmu_context.h>
 #include <asm/nospec-branch.h>
 #include <asm/cache.h>
+#include <asm/cacheflush.h>
 #include <asm/apic.h>
 
 #include "mm_internal.h"
@@ -42,11 +44,12 @@
  */
 
 /*
- * Bits to mangle the TIF_SPEC_IB state into the mm pointer which is
+ * Bits to mangle the TIF_SPEC_* state into the mm pointer which is
  * stored in cpu_tlb_state.last_user_mm_spec.
  */
 #define LAST_USER_MM_IBPB	0x1UL
-#define LAST_USER_MM_SPEC_MASK	(LAST_USER_MM_IBPB)
+#define LAST_USER_MM_L1D_FLUSH	0x2UL
+#define LAST_USER_MM_SPEC_MASK	(LAST_USER_MM_IBPB | LAST_USER_MM_L1D_FLUSH)
 
 /* Bits to set when tlbstate and flush is (re)initialized */
 #define LAST_USER_MM_INIT	LAST_USER_MM_IBPB
@@ -325,6 +328,7 @@ static inline unsigned long mm_mangle_tif_spec_bits(struct task_struct *next)
 	unsigned long next_tif = task_thread_info(next)->flags;
 	unsigned long spec_bits = (next_tif >> TIF_SPEC_IB) & LAST_USER_MM_SPEC_MASK;
 
+	BUILD_BUG_ON(TIF_SPEC_L1D_FLUSH != TIF_SPEC_IB + 1);
 	return (unsigned long)next->mm | spec_bits;
 }
 
@@ -402,6 +406,14 @@ static void cond_mitigation(struct task_struct *next)
 			indirect_branch_prediction_barrier();
 	}
 
+	/*
+	 * Flush only if SMT is disabled as per the contract, which is checked
+	 * when the feature is enabled.
+	 */
+	if (!this_cpu_read(cpu_info.smt_active) &&
+		(prev_mm & LAST_USER_MM_L1D_FLUSH))
+		l1d_flush_hw();
+
 	this_cpu_write(cpu_tlbstate.last_user_mm_spec, next_mm);
 }
 
Signature · 1 line
2.17.1
#6 lore raw

Re: [PATCH v4 3/5] x86/mm: Optionally flush L1D on context switch

I love your patch! Perhaps something to improve:

Hi Balbir,

I love your patch! Perhaps something to improve:

[auto build test WARNING on next-20210108]
[also build test WARNING on v5.11-rc2]
[cannot apply to tip/x86/mm tip/master linus/master tip/x86/core v5.11-rc2 v5.11-rc1 v5.10] [If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]

url:    https://github.com/0day-ci/linux/commits/Balbir-Singh/Next-revision-of-the-L1D-flush-patches/20210108-201551
base:    1c925d2030afd354a02c23500386e620e662622b
config: x86_64-randconfig-s022-20210108 (attached as .config)
compiler: gcc-9 (Debian 9.3.0-15) 9.3.0
reproduce:
        # apt-get install sparse
        # sparse version: v0.6.3-208-g46a52ca4-dirty
        # https://github.com/0day-ci/linux/commit/9dcce59b808dbd79d6535898f9262c0e4073aff0
        git remote add linux-review https://github.com/0day-ci/linux
        git fetch --no-tags linux-review Balbir-Singh/Next-revision-of-the-L1D-flush-patches/20210108-201551
        git checkout 9dcce59b808dbd79d6535898f9262c0e4073aff0
        # save the attached .config to linux build tree
        make W=1 C=1 CF='-fdiagnostic-prefix -D__CHECK_ENDIAN__' ARCH=x86_64 

If you fix the issue, kindly add following tag as appropriate

"sparse warnings: (new ones prefixed by >>)"

>> arch/x86/mm/tlb.c:413:14: sparse: sparse: incorrect type in initializer (different address spaces) @@     expected void const [noderef] __percpu *__vpp_verify @@     got bool * @@

arch/x86/mm/tlb.c:413:14: sparse: expected void const [noderef] __percpu *__vpp_verify
arch/x86/mm/tlb.c:413:14: sparse: got bool *

vim +413 arch/x86/mm/tlb.c

334
335 static void cond_mitigation(struct task_struct *next)
336 {
337 unsigned long prev_mm, next_mm;
338
339 if (!next || !next->mm)
340 return;
341
342 next_mm = mm_mangle_tif_spec_bits(next);
343 prev_mm = this_cpu_read(cpu_tlbstate.last_user_mm_spec);
344
345 /*
346 * Avoid user/user BTB poisoning by flushing the branch predictor
347 * when switching between processes. This stops one process from
348 * doing Spectre-v2 attacks on another.
349 *
350 * Both, the conditional and the always IBPB mode use the mm
351 * pointer to avoid the IBPB when switching between tasks of the
352 * same process. Using the mm pointer instead of mm->context.ctx_id
353 * opens a hypothetical hole vs. mm_struct reuse, which is more or
354 * less impossible to control by an attacker. Aside of that it
355 * would only affect the first schedule so the theoretically
356 * exposed data is not really interesting.
357 */
358 if (static_branch_likely(&switch_mm_cond_ibpb)) {
359 /*
360 * This is a bit more complex than the always mode because
361 * it has to handle two cases:
362 *
363 * 1) Switch from a user space task (potential attacker)
364 * which has TIF_SPEC_IB set to a user space task
365 * (potential victim) which has TIF_SPEC_IB not set.
366 *
367 * 2) Switch from a user space task (potential attacker)
368 * which has TIF_SPEC_IB not set to a user space task
369 * (potential victim) which has TIF_SPEC_IB set.
370 *
371 * This could be done by unconditionally issuing IBPB when
372 * a task which has TIF_SPEC_IB set is either scheduled in
373 * or out. Though that results in two flushes when:
374 *
375 * - the same user space task is scheduled out and later
376 * scheduled in again and only a kernel thread ran in
377 * between.
378 *
379 * - a user space task belonging to the same process is
380 * scheduled in after a kernel thread ran in between
381 *
382 * - a user space task belonging to the same process is
383 * scheduled in immediately.
384 *
385 * Optimize this with reasonably small overhead for the
386 * above cases. Mangle the TIF_SPEC_IB bit into the mm
387 * pointer of the incoming task which is stored in
388 * cpu_tlbstate.last_user_mm_spec for comparison.
389 *
390 * Issue IBPB only if the mm's are different and one or
391 * both have the IBPB bit set.
392 */
393 if (next_mm != prev_mm &&
394 (next_mm | prev_mm) & LAST_USER_MM_IBPB)
395 indirect_branch_prediction_barrier();
396 }
397
398 if (static_branch_unlikely(&switch_mm_always_ibpb)) {
399 /*
400 * Only flush when switching to a user space task with a
401 * different context than the user space task which ran
402 * last on this CPU.
403 */
404 if ((prev_mm & ~LAST_USER_MM_SPEC_MASK) !=
405 (unsigned long)next->mm)
406 indirect_branch_prediction_barrier();
407 }
408
409 /*
410 * Flush only if SMT is disabled as per the contract, which is checked
411 * when the feature is enabled.
412 */

 > 413		if (!this_cpu_read(cpu_info.smt_active) &&

414 (prev_mm & LAST_USER_MM_L1D_FLUSH))
415 l1d_flush_hw();
416
417 this_cpu_write(cpu_tlbstate.last_user_mm_spec, next_mm);
418 }
419

---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org

#7 lore raw

PATCH v4 4/5 prctl: Hook L1D flushing in via prctl

In reply to Balbir Singh

Use the existing PR_GET/SET_SPECULATION_CTRL API to expose the L1D flush capability. For L1D flushing PR_SPEC_FORCE_DISABLE and PR_SPEC_DISABLE_NOEXEC are not supported.

Enabling L1D flush does not check if the task is running on an SMT enabled core, rather a check is done at runtime (at the time of flush), if the task runs on a non SMT enabled core then the task is sent a SIGBUS (this is done prior to the task executing on the core, so no data is leaked). This is better than the other alternatives of

a. Ensuring strict affinity of the task (hard to enforce without further changes in the scheduler)
b. Silently skipping flush for tasks that move to SMT enabled cores.

An arch config ARCH_HAS_PARANOID_L1D_FLUSH has been added and struct task carries a callback_head for arch's that support this config (currently on x86), this callback head is used to schedule task work (SIGBUS delivery).

There is also no seccomp integration for the feature.

---
 arch/Kconfig                         |  4 ++
 arch/x86/Kconfig                     |  1 +
 arch/x86/include/asm/nospec-branch.h |  2 +
 arch/x86/include/asm/thread_info.h   |  3 --
 arch/x86/kernel/cpu/bugs.c           | 71 ++++++++++++++++++++++++++++
 arch/x86/mm/tlb.c                    | 29 ++++++++++--
 include/linux/sched.h                | 10 ++++
 include/uapi/linux/prctl.h           |  1 +
 8 files changed, 115 insertions(+), 6 deletions(-)
arch/Kconfig +4 -0
index 7091f7187951..0a0701e0a1ed 100644
@@ -327,6 +327,10 @@ config ARCH_32BIT_OFF_T
 	  still support 32-bit off_t. This option is enabled for all such
 	  architectures explicitly.
 
+config ARCH_HAS_PARANOID_L1D_FLUSH
+	bool
+	default n
+
 config HAVE_ASM_MODVERSIONS
 	bool
 	help
arch/x86/Kconfig +1 -0
index bd4993b276fd..2bb53bfaea02 100644
@@ -107,6 +107,7 @@ config X86
 	select ARCH_WANT_HUGE_PMD_SHARE
 	select ARCH_WANT_LD_ORPHAN_WARN
 	select ARCH_WANTS_THP_SWAP		if X86_64
+	select ARCH_HAS_PARANOID_L1D_FLUSH
 	select BUILDTIME_TABLE_SORT
 	select CLKEVT_I8253
 	select CLOCKSOURCE_VALIDATE_LAST_CYCLE
arch/x86/include/asm/nospec-branch.h +2 -0
index cb9ad6b73973..cd60934c6075 100644
@@ -253,6 +253,8 @@ DECLARE_STATIC_KEY_FALSE(switch_mm_always_ibpb);
 DECLARE_STATIC_KEY_FALSE(mds_user_clear);
 DECLARE_STATIC_KEY_FALSE(mds_idle_clear);
 
+DECLARE_STATIC_KEY_FALSE(l1d_flush_enabled);
+
 #include <asm/segment.h>
 
 /**
arch/x86/include/asm/thread_info.h +0 -3
index 33b637442b9e..054dc0f58ac4 100644
@@ -219,9 +219,6 @@ static inline int arch_within_stack_frames(const void * const stack,
 			   current_thread_info()->status & TS_COMPAT)
 #endif
 
-extern int enable_l1d_flush_for_task(struct task_struct *tsk);
-extern int disable_l1d_flush_for_task(struct task_struct *tsk);
-
 extern void arch_task_cache_init(void);
 extern int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src);
 extern void arch_release_task_struct(struct task_struct *tsk);
arch/x86/kernel/cpu/bugs.c +71 -0

113 lines not on this page · raw message

arch/x86/mm/tlb.c +26 -3
index f67c5bd58158..aa9286b83f8f 100644
@@ -323,6 +323,28 @@ void switch_mm(struct mm_struct *prev, struct mm_struct *next,
 	local_irq_restore(flags);
 }
 
+/*
+ * Sent to a task that opts into L1D flushing via the prctl interface
+ * but ends up running on an SMT enabled core.
+ */
+static void l1d_flush_kill(struct callback_head *ch)
+{
+	force_sig(SIGBUS);
+}
+
+static void  l1d_flush_evaluate(unsigned long prev_mm, unsigned long next_mm,
+				struct task_struct *next)
+{
+	if (prev_mm & LAST_USER_MM_L1D_FLUSH)
+		l1d_flush_hw();
+
+	if ((next_mm & LAST_USER_MM_L1D_FLUSH) && this_cpu_read(cpu_info.smt_active)) {
+		clear_ti_thread_flag(&next->thread_info, TIF_SPEC_L1D_FLUSH);
+		next->l1d_flush_kill.func = l1d_flush_kill;
+		task_work_add(next, &next->l1d_flush_kill, TWA_RESUME);
+	}
+}
+
 static inline unsigned long mm_mangle_tif_spec_bits(struct task_struct *next)
 {
 	unsigned long next_tif = task_thread_info(next)->flags;
@@ -410,9 +432,10 @@ static void cond_mitigation(struct task_struct *next)
 	 * Flush only if SMT is disabled as per the contract, which is checked
 	 * when the feature is enabled.
 	 */
-	if (!this_cpu_read(cpu_info.smt_active) &&
-		(prev_mm & LAST_USER_MM_L1D_FLUSH))
-		l1d_flush_hw();
+	if (static_branch_unlikely(&l1d_flush_enabled)) {
+		if (unlikely((prev_mm | next_mm) & LAST_USER_MM_L1D_FLUSH))
+			l1d_flush_evaluate(prev_mm, next_mm, next);
+	}
 
 	this_cpu_write(cpu_tlbstate.last_user_mm_spec, next_mm);
 }
include/linux/sched.h +10 -0
index e5ad6d354b7b..77e9d32d70ca 100644
@@ -1376,6 +1376,16 @@ struct task_struct {
 	unsigned long			getblk_bh_state;
 #endif
 
+#ifdef CONFIG_ARCH_HAS_PARANOID_L1D_FLUSH
+	/*
+	 * If L1D flush is supported on mm context switch
+	 * then we use this callback head to queue kill work
+	 * to kill tasks that are not running on SMT disabled
+	 * cores
+	 */
+	struct callback_head		l1d_flush_kill;
+#endif
+
 	/*
 	 * New fields for task_struct should be added above here, so that
 	 * they are included in the randomized portion of task_struct.
include/uapi/linux/prctl.h +1 -0
index 90deb41c8a34..44adcae6641c 100644
@@ -213,6 +213,7 @@ struct prctl_mm_map {
 /* Speculation control variants */
 # define PR_SPEC_STORE_BYPASS		0
 # define PR_SPEC_INDIRECT_BRANCH	1
+# define PR_SPEC_L1D_FLUSH		2
 /* Return and control values for PR_SET/GET_SPECULATION_CTRL */
 # define PR_SPEC_NOT_AFFECTED		0
 # define PR_SPEC_PRCTL			(1UL << 0)
Signature · 1 line
2.17.1
merge bot Applied
#8 lore raw

[tip: x86/cpu] x86, prctl: Hook L1D flushing in via prctl

The following commit has been merged into the x86/cpu branch of tip:

The following commit has been merged into the x86/cpu branch of tip:

Commit-ID:     e893bb1bb4d2eb635eba61e5d9c5135d96855773
Gitweb:        https://git.kernel.org/tip/e893bb1bb4d2eb635eba61e5d9c5135d96855773
Author:        Balbir Singh <[email protected]>
AuthorDate:    Fri, 08 Jan 2021 23:10:55 +11:00
Committer:     Thomas Gleixner <[email protected]>
CommitterDate: Wed, 28 Jul 2021 11:42:25 +02:00

x86, prctl: Hook L1D flushing in via prctl

Use the existing PR_GET/SET_SPECULATION_CTRL API to expose the L1D flush capability. For L1D flushing PR_SPEC_FORCE_DISABLE and PR_SPEC_DISABLE_NOEXEC are not supported.

Enabling L1D flush does not check if the task is running on an SMT enabled core, rather a check is done at runtime (at the time of flush), if the task runs on a SMT sibling then the task is sent a SIGBUS which is executed before the task returns to user space or to a guest.

This is better than the other alternatives of:

a. Ensuring strict affinity of the task (hard to enforce without further
changes in the scheduler)

b. Silently skipping flush for tasks that move to SMT enabled cores.

Hook up the core prctl and implement the x86 specific parts which in turn makes it functional.

---
 arch/x86/kernel/cpu/bugs.c | 33 +++++++++++++++++++++++++++++++++
 include/uapi/linux/prctl.h |  1 +
 2 files changed, 34 insertions(+)
arch/x86/kernel/cpu/bugs.c +33 -0
index 1a5a1b0..ecfca3b 100644
@@ -1252,6 +1252,24 @@ static void task_update_spec_tif(struct task_struct *tsk)
 		speculation_ctrl_update_current();
 }
 
+static int l1d_flush_prctl_set(struct task_struct *task, unsigned long ctrl)
+{
+
+	if (!static_branch_unlikely(&switch_mm_cond_l1d_flush))
+		return -EPERM;
+
+	switch (ctrl) {
+	case PR_SPEC_ENABLE:
+		set_ti_thread_flag(&task->thread_info, TIF_SPEC_L1D_FLUSH);
+		return 0;
+	case PR_SPEC_DISABLE:
+		clear_ti_thread_flag(&task->thread_info, TIF_SPEC_L1D_FLUSH);
+		return 0;
+	default:
+		return -ERANGE;
+	}
+}
+
 static int ssb_prctl_set(struct task_struct *task, unsigned long ctrl)
 {
 	if (ssb_mode != SPEC_STORE_BYPASS_PRCTL &&
@@ -1361,6 +1379,8 @@ int arch_prctl_spec_ctrl_set(struct task_struct *task, unsigned long which,
 		return ssb_prctl_set(task, ctrl);
 	case PR_SPEC_INDIRECT_BRANCH:
 		return ib_prctl_set(task, ctrl);
+	case PR_SPEC_L1D_FLUSH:
+		return l1d_flush_prctl_set(task, ctrl);
 	default:
 		return -ENODEV;
 	}
@@ -1377,6 +1397,17 @@ void arch_seccomp_spec_mitigate(struct task_struct *task)
 }
 #endif
 
+static int l1d_flush_prctl_get(struct task_struct *task)
+{
+	if (!static_branch_unlikely(&switch_mm_cond_l1d_flush))
+		return PR_SPEC_FORCE_DISABLE;
+
+	if (test_ti_thread_flag(&task->thread_info, TIF_SPEC_L1D_FLUSH))
+		return PR_SPEC_PRCTL | PR_SPEC_ENABLE;
+	else
+		return PR_SPEC_PRCTL | PR_SPEC_DISABLE;
+}
+
 static int ssb_prctl_get(struct task_struct *task)
 {
 	switch (ssb_mode) {
@@ -1427,6 +1458,8 @@ int arch_prctl_spec_ctrl_get(struct task_struct *task, unsigned long which)
 		return ssb_prctl_get(task);
 	case PR_SPEC_INDIRECT_BRANCH:
 		return ib_prctl_get(task);
+	case PR_SPEC_L1D_FLUSH:
+		return l1d_flush_prctl_get(task);
 	default:
 		return -ENODEV;
 	}
include/uapi/linux/prctl.h +1 -0
index 967d9c5..964c41e 100644
@@ -213,6 +213,7 @@ struct prctl_mm_map {
 /* Speculation control variants */
 # define PR_SPEC_STORE_BYPASS		0
 # define PR_SPEC_INDIRECT_BRANCH	1
+# define PR_SPEC_L1D_FLUSH		2
 /* Return and control values for PR_SET/GET_SPECULATION_CTRL */
 # define PR_SPEC_NOT_AFFECTED		0
 # define PR_SPEC_PRCTL			(1UL << 0)
#9 lore raw

PATCH v4 5/5 Documentation: Add L1D flushing Documentation

In reply to Balbir Singh

Add documentation of l1d flushing, explain the need for the feature and how it can be used.

---
 Documentation/admin-guide/hw-vuln/index.rst   |  1 +
 .../admin-guide/hw-vuln/l1d_flush.rst         | 70 +++++++++++++++++++
 .../admin-guide/kernel-parameters.txt         | 17 +++++
 Documentation/userspace-api/spec_ctrl.rst     |  8 +++
 4 files changed, 96 insertions(+)
 create mode 100644 Documentation/admin-guide/hw-vuln/l1d_flush.rst
Documentation/admin-guide/hw-vuln/index.rst +1 -0
index ca4dbdd9016d..21710f8609fe 100644
@@ -15,3 +15,4 @@ are configurable at compile, boot or run time.
    tsx_async_abort
    multihit.rst
    special-register-buffer-data-sampling.rst
+   l1d_flush.rst
Documentation/admin-guide/hw-vuln/l1d_flush.rst new file +70 -0

70 lines not on this page · raw message

Documentation/admin-guide/kernel-parameters.txt +17 -0
index bc20e2f4677f..bd1e8e329727 100644
@@ -2356,6 +2356,23 @@
 			feature (tagged TLBs) on capable Intel chips.
 			Default is 1 (enabled)
 
+	l1d_flush=	[X86,INTEL]
+			Control mitigation for L1D based snooping vulnerability.
+
+			Certain CPUs are vulnerable to an exploit against CPU
+			internal buffers which can forward information to a
+			disclosure gadget under certain conditions.
+
+			In vulnerable processors, the speculatively
+			forwarded data can be used in a cache side channel
+			attack, to access data to which the attacker does
+			not have direct access.
+
+			This parameter controls the mitigation. The
+			options are:
+
+			on         - enable the interface for the mitigation
+
 	l1tf=           [X86] Control mitigation of the L1TF vulnerability on
 			      affected CPUs
 
Documentation/userspace-api/spec_ctrl.rst +8 -0
index 7ddd8f667459..5e8ed9eef9aa 100644
@@ -106,3 +106,11 @@ Speculation misfeature controls
    * prctl(PR_SET_SPECULATION_CTRL, PR_SPEC_INDIRECT_BRANCH, PR_SPEC_ENABLE, 0, 0);
    * prctl(PR_SET_SPECULATION_CTRL, PR_SPEC_INDIRECT_BRANCH, PR_SPEC_DISABLE, 0, 0);
    * prctl(PR_SET_SPECULATION_CTRL, PR_SPEC_INDIRECT_BRANCH, PR_SPEC_FORCE_DISABLE, 0, 0);
+
+- PR_SPEC_L1D_FLUSH: Flush L1D Cache on context switch out of the task
+                        (works only when tasks run on non SMT cores)
+
+  Invocations:
+   * prctl(PR_GET_SPECULATION_CTRL, PR_SPEC_L1D_FLUSH, 0, 0, 0);
+   * prctl(PR_SET_SPECULATION_CTRL, PR_SPEC_L1D_FLUSH, PR_SPEC_ENABLE, 0, 0);
+   * prctl(PR_SET_SPECULATION_CTRL, PR_SPEC_L1D_FLUSH, PR_SPEC_DISABLE, 0, 0);
Signature · 1 line
2.17.1
merge bot Applied
#10 lore raw

[tip: x86/cpu] Documentation: Add L1D flushing Documentation

The following commit has been merged into the x86/cpu branch of tip:

The following commit has been merged into the x86/cpu branch of tip:

Commit-ID:     b7fe54f6c2d437082dcbecfbd832f38edd9caaf4
Gitweb:        https://git.kernel.org/tip/b7fe54f6c2d437082dcbecfbd832f38edd9caaf4
Author:        Balbir Singh <[email protected]>
AuthorDate:    Fri, 08 Jan 2021 23:10:56 +11:00
Committer:     Thomas Gleixner <[email protected]>
CommitterDate: Wed, 28 Jul 2021 11:42:25 +02:00

Documentation: Add L1D flushing Documentation

Add documentation of l1d flushing, explain the need for the feature and how it can be used.

---
 Documentation/admin-guide/hw-vuln/index.rst     |  1 +-
 Documentation/admin-guide/hw-vuln/l1d_flush.rst | 69 ++++++++++++++++-
 Documentation/admin-guide/kernel-parameters.txt | 17 ++++-
 Documentation/userspace-api/spec_ctrl.rst       |  8 ++-
 4 files changed, 95 insertions(+)
 create mode 100644 Documentation/admin-guide/hw-vuln/l1d_flush.rst
Documentation/admin-guide/hw-vuln/index.rst +1 -0
index f12cda5..8cbc711 100644
@@ -16,3 +16,4 @@ are configurable at compile, boot or run time.
    multihit.rst
    special-register-buffer-data-sampling.rst
    core-scheduling.rst
+   l1d_flush.rst
Documentation/admin-guide/hw-vuln/l1d_flush.rst new file +69 -0

69 lines not on this page · raw message

Documentation/admin-guide/kernel-parameters.txt +17 -0
index bdb2200..b105db2 100644
@@ -2421,6 +2421,23 @@
 			feature (tagged TLBs) on capable Intel chips.
 			Default is 1 (enabled)
 
+	l1d_flush=	[X86,INTEL]
+			Control mitigation for L1D based snooping vulnerability.
+
+			Certain CPUs are vulnerable to an exploit against CPU
+			internal buffers which can forward information to a
+			disclosure gadget under certain conditions.
+
+			In vulnerable processors, the speculatively
+			forwarded data can be used in a cache side channel
+			attack, to access data to which the attacker does
+			not have direct access.
+
+			This parameter controls the mitigation. The
+			options are:
+
+			on         - enable the interface for the mitigation
+
 	l1tf=           [X86] Control mitigation of the L1TF vulnerability on
 			      affected CPUs
 
Documentation/userspace-api/spec_ctrl.rst +8 -0
index 7ddd8f6..5e8ed9e 100644
@@ -106,3 +106,11 @@ Speculation misfeature controls
    * prctl(PR_SET_SPECULATION_CTRL, PR_SPEC_INDIRECT_BRANCH, PR_SPEC_ENABLE, 0, 0);
    * prctl(PR_SET_SPECULATION_CTRL, PR_SPEC_INDIRECT_BRANCH, PR_SPEC_DISABLE, 0, 0);
    * prctl(PR_SET_SPECULATION_CTRL, PR_SPEC_INDIRECT_BRANCH, PR_SPEC_FORCE_DISABLE, 0, 0);
+
+- PR_SPEC_L1D_FLUSH: Flush L1D Cache on context switch out of the task
+                        (works only when tasks run on non SMT cores)
+
+  Invocations:
+   * prctl(PR_GET_SPECULATION_CTRL, PR_SPEC_L1D_FLUSH, 0, 0, 0);
+   * prctl(PR_SET_SPECULATION_CTRL, PR_SPEC_L1D_FLUSH, PR_SPEC_ENABLE, 0, 0);
+   * prctl(PR_SET_SPECULATION_CTRL, PR_SPEC_L1D_FLUSH, PR_SPEC_DISABLE, 0, 0);
#11 lore raw
In reply to Balbir Singh
Quoted 79 lines · Balbir Singh

Implement a mechanism that allows tasks to conditionally flush their L1D cache (mitigation mechanism suggested in [2]). The previous posts of these patches were sent for inclusion (see [3]) and were not included due to the concern for the need for additional checks, those checks were:

1. Implement this mechanism only for CPUs affected by the L1TF bug
2. Disable the software fallback
3. Provide an override to enable this mechanism
4. Be SMT aware in the implementation

The patches support a use case where the entire system is not in non SMT mode, but rather a few CPUs can have their SMT turned off and processes that want to opt-in are expected to run on non SMT cores. This gives the administrator complete control over setting up the mitigation for the issue. In addition, the administrator has a boot time override (l1d_flush=on) to turn on the mechanism without which this mechanism will not work.

To implement these efficiently, a new per cpu view of whether the core is in SMT mode or not is implemented in patch 1. The code is refactored in patch 2 so that the existing code can allow for other speculation related checks when switching mm between tasks, this mechanism has not changed since the last post. The ability to flush L1D for tasks if the TIF_SPEC_L1D_FLUSH bit is set and the task has context switched out of a non SMT core is provided by patch 3. Hooks for the user space API, for this feature to be invoked via prctl are provided in patch 4, along with the checks described above (1, 2, and 3). Documentation updates are in patch 5, with updates on l1d_flush, the prctl changes and updates to the kernel-parameters (l1d_flush_out).

The checks for opting into L1D flushing are:
a. If the CPU is affected by L1TF
b. Hardware L1D flush mechanism is available

A task running on a core with SMT enabled and opting into this feature will receive a SIGBUS.

References
[1] https://software.intel.com/security-software-guidance/software-guidance/snoop-assisted-l1-data-sampling
[2] https://software.intel.com/security-software-guidance/insights/deep-dive-snoop-assisted-l1-data-sampling
[3] https://lkml.org/lkml/2020/6/2/1150
[4] https://lore.kernel.org/lkml/[email protected]/
[5] https://lore.kernel.org/lkml/[email protected]/

Reviewers guide to v4
- The key patch in the series and most of the changes to this revision are to patch 4. patches 3 and 5 have been modified to keep them consistent with the changes to patch 4.

Changelog v4:
- Use a static key to enable the mechanism (remove overheads)
- By default have the mechanism turned off, so there are two opt-ins needed, one by the administrator at boot time, second by the application
- Rename l1d_flush_out/L1D_FLUSH_OUT to l1d_flush/L1D_FLUSH
- Implement other review recommendations
Changelog v3:
- Implement the SIGBUS mechansim
- Update and fix the documentation

Balbir Singh (5):
x86/smp: Add a per-cpu view of SMT state x86/mm: Refactor cond_ibpb() to support other use cases x86/mm: Optionally flush L1D on context switch
prctl: Hook L1D flushing in via prctl
Documentation: Add L1D flushing Documentation

 Documentation/admin-guide/hw-vuln/index.rst   |  1 +
 .../admin-guide/hw-vuln/l1d_flush.rst         | 70 +++++++++++++++
 .../admin-guide/kernel-parameters.txt         | 17 ++++
 Documentation/userspace-api/spec_ctrl.rst     |  8 ++
 arch/Kconfig                                  |  4 +
 arch/x86/Kconfig                              |  1 +
 arch/x86/include/asm/cacheflush.h             |  8 ++
 arch/x86/include/asm/nospec-branch.h          |  2 +
 arch/x86/include/asm/processor.h              |  2 +
 arch/x86/include/asm/thread_info.h            |  6 +-
Quoted 9 lines
 arch/x86/include/asm/tlbflush.h               |  2 +-
 arch/x86/kernel/cpu/bugs.c                    | 71 +++++++++++++++
 arch/x86/kernel/smpboot.c                     | 10 ++-
 arch/x86/mm/tlb.c                             | 88 ++++++++++++++-----
 include/linux/sched.h                         | 10 +++
 include/uapi/linux/prctl.h                    |  1 +
 16 files changed, 273 insertions(+), 28 deletions(-)
 create mode 100644 Documentation/admin-guide/hw-vuln/l1d_flush.rst

Ping on any review comments? Suggested refactoring?

Balbir Singh

#12 lore raw

*thread necromancy*
https://lore.kernel.org/lkml/[email protected]/

Quoted 7 lines · Balbir
Quoted 6 lines · Balbir Singh

Implement a mechanism that allows tasks to conditionally flush their L1D cache (mitigation mechanism suggested in [2]). The previous posts of these patches were sent for inclusion (see [3]) and were not included due to the concern for the need for additional checks, those checks were:

Quoted 6 lines
Quoted 5 lines

1. Implement this mechanism only for CPUs affected by the L1TF bug
2. Disable the software fallback
3. Provide an override to enable this mechanism
4. Be SMT aware in the implementation
[...]

Ping on any review comments? Suggested refactoring?

Hi!

I'd still really like to see this -- it's a big hammer, but that's the point for cases where some new flaw appears and we can point to the toolbox and say "you can mitigate it with this while you wait for new kernel/CPU."

Any further thoughts from x86 maintainers? This seems like it addressed all of tglx's review comments.

Signature · 1 line
Kees Cook

A message that is not in the archive (sent off-list or not yet received): [email protected]

#14 lore raw
Quoted 7 lines · Thomas Gleixner
Quoted 6 lines · Kees Cook

I'd still really like to see this -- it's a big hammer, but that's the point for cases where some new flaw appears and we can point to the toolbox and say "you can mitigate it with this while you wait for new kernel/CPU."

Quoted 5 lines
Quoted 2 lines

Any further thoughts from x86 maintainers? This seems like it addressed all of tglx's review comments.

Sorry for dropping the ball on this. It's in my list of things to deal with. Starting to look at it now.

So I went through the pile and for remorse I sat down and made the tweaks I think are necessary myself.

I've pushed out the result to

git://git.kernel.org/pub/scm/linux/kernel/git/tglx/devel.git x86/l1dflush

The only thing I did not address yet is that the documentation lacks any mentioning of the SIGBUS mechanism which is invoked when a task which asked for L1D flush protection ends up on a SMT sibling for whatever reason. That's essential to have because it's part of the contract of that prctl.

Balbir, can you please double check the result and prepare an updated version from there?

If you don't have cycles, please let me know.

Thanks,

tglx

#15 lore raw
Quoted 14 lines · Thomas Gleixner
Quoted 7 lines · Thomas Gleixner
Quoted 6 lines · Kees Cook

I'd still really like to see this -- it's a big hammer, but that's the point for cases where some new flaw appears and we can point to the toolbox and say "you can mitigate it with this while you wait for new kernel/CPU."

Quoted 5 lines
Quoted 2 lines

Any further thoughts from x86 maintainers? This seems like it addressed all of tglx's review comments.

Sorry for dropping the ball on this. It's in my list of things to deal with. Starting to look at it now.

Quoted 6 lines

So I went through the pile and for remorse I sat down and made the tweaks I think are necessary myself.

I've pushed out the result to

git://git.kernel.org/pub/scm/linux/kernel/git/tglx/devel.git x86/l1dflush

Oh excellent; thank you for doing this!

Signature · 1 line
Kees Cook
#16 lore raw
Quoted 15 lines · Kees Cook
Quoted 14 lines · Thomas Gleixner
Quoted 12 lines · Thomas Gleixner
Quoted 8 lines · Kees Cook

I'd still really like to see this -- it's a big hammer, but that's the point for cases where some new flaw appears and we can point to the toolbox and say "you can mitigate it with this while you wait for new kernel/CPU."

Any further thoughts from x86 maintainers? This seems like it addressed all of tglx's review comments.

Sorry for dropping the ball on this. It's in my list of things to deal with. Starting to look at it now.

Quoted 9 lines
Quoted 6 lines

So I went through the pile and for remorse I sat down and made the tweaks I think are necessary myself.

I've pushed out the result to

git://git.kernel.org/pub/scm/linux/kernel/git/tglx/devel.git x86/l1dflush

Oh excellent; thank you for doing this!

Thanks again Thomas!

I no longer have access to the bare metal hardware, but I was able to test this under qemu with some emulation changes. The changes worked as expected.

Folks on the list/cc, appreciate any tested-by or additional reviewed-by tags if you do happen to review/test.

Thanks,
Balbir Singh.

#17 lore raw
Quoted 26 lines · Balbir Singh
Quoted 15 lines · Kees Cook
Quoted 14 lines · Thomas Gleixner
Quoted 12 lines · Thomas Gleixner
Quoted 8 lines · Kees Cook

I'd still really like to see this -- it's a big hammer, but that's the point for cases where some new flaw appears and we can point to the toolbox and say "you can mitigate it with this while you wait for new kernel/CPU."

Any further thoughts from x86 maintainers? This seems like it addressed all of tglx's review comments.

Sorry for dropping the ball on this. It's in my list of things to deal with. Starting to look at it now.

Quoted 9 lines
Quoted 6 lines

So I went through the pile and for remorse I sat down and made the tweaks I think are necessary myself.

I've pushed out the result to

git://git.kernel.org/pub/scm/linux/kernel/git/tglx/devel.git x86/l1dflush

Oh excellent; thank you for doing this!

Quoted 7 lines

Thanks again Thomas!

I no longer have access to the bare metal hardware, but I was able to test this under qemu with some emulation changes. The changes worked as expected.

Folks on the list/cc, appreciate any tested-by or additional reviewed-by tags if you do happen to review/test.

I can't test the behavior (no access to CPU), but I wrote a simple prctl tester. Perhaps this can be expanded on?

tools/testing/selftests/x86/Makefile +1 -1
index 333980375bc7..50c150d35962 100644
@@ -13,7 +13,7 @@ CAN_BUILD_WITH_NOPIE := $(shell ./check_cc.sh $(CC) trivial_program.c -no-pie)
 TARGETS_C_BOTHBITS := single_step_syscall sysret_ss_attrs syscall_nt test_mremap_vdso \
 			check_initial_reg_state sigreturn iopl ioperm \
 			test_vsyscall mov_ss_trap \
-			syscall_arg_fault fsgsbase_restore
+			syscall_arg_fault fsgsbase_restore l1d_flush
 TARGETS_C_32BIT_ONLY := entry_from_vm86 test_syscall_vdso unwind_vdso \
 			test_FCMOV test_FCOMI test_FISTTP \
 			vdso_restorer
tools/testing/selftests/x86/l1d_flush.c new file +66 -0

66 lines not on this page · raw message

Signature · 1 line
Kees Cook
#18 lore raw
In reply to Thomas Gleixner
Quoted 14 lines · Thomas Gleixner
Quoted 7 lines · Thomas Gleixner
Quoted 6 lines · Kees Cook

I'd still really like to see this -- it's a big hammer, but that's the point for cases where some new flaw appears and we can point to the toolbox and say "you can mitigate it with this while you wait for new kernel/CPU."

Quoted 5 lines
Quoted 2 lines

Any further thoughts from x86 maintainers? This seems like it addressed all of tglx's review comments.

Sorry for dropping the ball on this. It's in my list of things to deal with. Starting to look at it now.

Quoted 7 lines

So I went through the pile and for remorse I sat down and made the tweaks I think are necessary myself.

I've pushed out the result to

git://git.kernel.org/pub/scm/linux/kernel/git/tglx/devel.git x86/l1dflush

Thank you I'll take a look and test it.

Quoted 5 lines

The only thing I did not address yet is that the documentation lacks any mentioning of the SIGBUS mechanism which is invoked when a task which asked for L1D flush protection ends up on a SMT sibling for whatever reason. That's essential to have because it's part of the contract of that prctl.

IIRC I documented it, I'll double check.

Quoted 6 lines

Balbir, can you please double check the result and prepare an updated version from there?

If you don't have cycles, please let me know.

I might have some cycles for testing and re-review. Thanks for all the hard work on this

Balbir Singh.

merge bot Applied
#19 lore raw

[tip: x86/cpu] x86/mm: Prepare for opt-in based L1D flush in switch_mm()

In reply to Balbir Singh
The following commit has been merged into the x86/cpu branch of tip:

The following commit has been merged into the x86/cpu branch of tip:

Commit-ID:     b5f06f64e269f9820cd5ad9e9a98afa6c8914b7a
Gitweb:        https://git.kernel.org/tip/b5f06f64e269f9820cd5ad9e9a98afa6c8914b7a
Author:        Balbir Singh <[email protected]>
AuthorDate:    Mon, 26 Apr 2021 21:42:30 +02:00
Committer:     Thomas Gleixner <[email protected]>
CommitterDate: Wed, 28 Jul 2021 11:42:24 +02:00

x86/mm: Prepare for opt-in based L1D flush in switch_mm()

The goal of this is to allow tasks that want to protect sensitive information, against e.g. the recently found snoop assisted data sampling vulnerabilites, to flush their L1D on being switched out. This protects their data from being snooped or leaked via side channels after the task has context switched out.

This could also be used to wipe L1D when an untrusted task is switched in, but that's not a really well defined scenario while the opt-in variant is clearly defined.

The mechanism is default disabled and can be enabled on the kernel command line.

Prepare for the actual prctl based opt-in:

1) Provide the necessary setup functionality similar to the other mitigations and enable the static branch when the command line option is set and the CPU provides support for hardware assisted L1D flushing. Software based L1D flush is not supported because it's CPU model specific and not really well defined.

This does not come with a sysfs file like the other mitigations because it is not bound to any specific vulnerability.

Support has to be queried via the prctl(2) interface.

2) Add TIF_SPEC_L1D_FLUSH next to L1D_SPEC_IB so the two bits can be mangled into the mm pointer in one go which allows to reuse the existing mechanism in switch_mm() for the conditional IBPB speculation barrier efficiently.

3) Add the L1D flush specific functionality which flushes L1D when the outgoing task opted in.

Also check whether the incoming task has requested L1D flush and if so validate that it is not accidentaly running on an SMT sibling as this makes the whole excercise moot because SMT siblings share L1D which opens tons of other attack vectors. If that happens schedule task work which signals the incoming task on return to user/guest with SIGBUS as this is part of the paranoid L1D flush contract.

---
 arch/x86/Kconfig                     |  1 +-
 arch/x86/include/asm/nospec-branch.h |  2 +-
 arch/x86/include/asm/thread_info.h   |  2 +-
 arch/x86/kernel/cpu/bugs.c           | 37 +++++++++++++++++-
 arch/x86/mm/tlb.c                    | 58 ++++++++++++++++++++++++++-
 5 files changed, 98 insertions(+), 2 deletions(-)
arch/x86/Kconfig +1 -0
index 4927065..d8a2c3f 100644
@@ -119,6 +119,7 @@ config X86
 	select ARCH_WANT_HUGE_PMD_SHARE
 	select ARCH_WANT_LD_ORPHAN_WARN
 	select ARCH_WANTS_THP_SWAP		if X86_64
+	select ARCH_HAS_PARANOID_L1D_FLUSH
 	select BUILDTIME_TABLE_SORT
 	select CLKEVT_I8253
 	select CLOCKSOURCE_VALIDATE_LAST_CYCLE
arch/x86/include/asm/nospec-branch.h +2 -0
index 3ad8c6d..ec2d5c8 100644
@@ -252,6 +252,8 @@ DECLARE_STATIC_KEY_FALSE(switch_mm_always_ibpb);
 DECLARE_STATIC_KEY_FALSE(mds_user_clear);
 DECLARE_STATIC_KEY_FALSE(mds_idle_clear);
 
+DECLARE_STATIC_KEY_FALSE(switch_mm_cond_l1d_flush);
+
 #include <asm/segment.h>
 
 /**
arch/x86/include/asm/thread_info.h +2 -0
index d9afd35..cf13266 100644
@@ -81,6 +81,7 @@ struct thread_info {
 #define TIF_SINGLESTEP		4	/* reenable singlestep on user return*/
 #define TIF_SSBD		5	/* Speculative store bypass disable */
 #define TIF_SPEC_IB		9	/* Indirect branch speculation mitigation */
+#define TIF_SPEC_L1D_FLUSH	10	/* Flush L1D on mm switches (processes) */
 #define TIF_USER_RETURN_NOTIFY	11	/* notify kernel of userspace return */
 #define TIF_UPROBE		12	/* breakpointed or singlestepping */
 #define TIF_PATCH_PENDING	13	/* pending live patching update */
@@ -104,6 +105,7 @@ struct thread_info {
 #define _TIF_SINGLESTEP		(1 << TIF_SINGLESTEP)
 #define _TIF_SSBD		(1 << TIF_SSBD)
 #define _TIF_SPEC_IB		(1 << TIF_SPEC_IB)
+#define _TIF_SPEC_L1D_FLUSH	(1 << TIF_SPEC_L1D_FLUSH)
 #define _TIF_USER_RETURN_NOTIFY	(1 << TIF_USER_RETURN_NOTIFY)
 #define _TIF_UPROBE		(1 << TIF_UPROBE)
 #define _TIF_PATCH_PENDING	(1 << TIF_PATCH_PENDING)
arch/x86/kernel/cpu/bugs.c +37 -0
index d41b70f..1a5a1b0 100644
@@ -43,6 +43,7 @@ static void __init mds_select_mitigation(void);
 static void __init mds_print_mitigation(void);
 static void __init taa_select_mitigation(void);
 static void __init srbds_select_mitigation(void);
+static void __init l1d_flush_select_mitigation(void);
 
 /* The base value of the SPEC_CTRL MSR that always has to be preserved. */
 u64 x86_spec_ctrl_base;
@@ -76,6 +77,13 @@ EXPORT_SYMBOL_GPL(mds_user_clear);
 DEFINE_STATIC_KEY_FALSE(mds_idle_clear);
 EXPORT_SYMBOL_GPL(mds_idle_clear);
 
+/*
+ * Controls whether l1d flush based mitigations are enabled,
+ * based on hw features and admin setting via boot parameter
+ * defaults to false
+ */
+DEFINE_STATIC_KEY_FALSE(switch_mm_cond_l1d_flush);
+
 void __init check_bugs(void)
 {
 	identify_boot_cpu();
@@ -111,6 +119,7 @@ void __init check_bugs(void)
 	mds_select_mitigation();
 	taa_select_mitigation();
 	srbds_select_mitigation();
+	l1d_flush_select_mitigation();
 
 	/*
 	 * As MDS and TAA mitigations are inter-related, print MDS
@@ -492,6 +501,34 @@ static int __init srbds_parse_cmdline(char *str)
 early_param("srbds", srbds_parse_cmdline);
 
 #undef pr_fmt
+#define pr_fmt(fmt)     "L1D Flush : " fmt
+
+enum l1d_flush_mitigations {
+	L1D_FLUSH_OFF = 0,
+	L1D_FLUSH_ON,
+};
+
+static enum l1d_flush_mitigations l1d_flush_mitigation __initdata = L1D_FLUSH_OFF;
+
+static void __init l1d_flush_select_mitigation(void)
+{
+	if (!l1d_flush_mitigation || !boot_cpu_has(X86_FEATURE_FLUSH_L1D))
+		return;
+
+	static_branch_enable(&switch_mm_cond_l1d_flush);
+	pr_info("Conditional flush on switch_mm() enabled\n");
+}
+
+static int __init l1d_flush_parse_cmdline(char *str)
+{
+	if (!strcmp(str, "on"))
+		l1d_flush_mitigation = L1D_FLUSH_ON;
+
+	return 0;
+}
+early_param("l1d_flush", l1d_flush_parse_cmdline);
+
+#undef pr_fmt
 #define pr_fmt(fmt)     "Spectre V1 : " fmt
 
 enum spectre_v1_mitigation {
arch/x86/mm/tlb.c +56 -2

95 lines not on this page · raw message

merge bot Applied
#20 lore raw

[tip: x86/cpu] sched: Add task_work callback for paranoid L1D flush

In reply to Balbir Singh
The following commit has been merged into the x86/cpu branch of tip:

The following commit has been merged into the x86/cpu branch of tip:

Commit-ID:     58e106e725eed59896b9141a1c9a917d2f67962a
Gitweb:        https://git.kernel.org/tip/58e106e725eed59896b9141a1c9a917d2f67962a
Author:        Balbir Singh <[email protected]>
AuthorDate:    Mon, 26 Apr 2021 21:59:11 +02:00
Committer:     Thomas Gleixner <[email protected]>
CommitterDate: Wed, 28 Jul 2021 11:42:24 +02:00

sched: Add task_work callback for paranoid L1D flush

The upcoming paranoid L1D flush infrastructure allows to conditionally (opt-in) flush L1D in switch_mm() as a defense against potential new side channels or for paranoia reasons. As the flush makes only sense when a task runs on a non-SMT enabled core, because SMT siblings share L1, the switch_mm() logic will kill a task which is flagged for L1D flush when it is running on a SMT thread.

Add a taskwork callback so switch_mm() can queue a SIG_KILL command which is invoked when the task tries to return to user space.

---
 arch/Kconfig          |  3 +++
 include/linux/sched.h | 10 ++++++++++
 2 files changed, 13 insertions(+)
arch/Kconfig +3 -0
index 129df49..98db634 100644
@@ -1282,6 +1282,9 @@ config ARCH_SPLIT_ARG64
 config ARCH_HAS_ELFCORE_COMPAT
 	bool
 
+config ARCH_HAS_PARANOID_L1D_FLUSH
+	bool
+
 source "kernel/gcov/Kconfig"
 
 source "scripts/gcc-plugins/Kconfig"
include/linux/sched.h +10 -0
index ec8d07d..c048e59 100644
@@ -1400,6 +1400,16 @@ struct task_struct {
 	struct llist_head               kretprobe_instances;
 #endif
 
+#ifdef CONFIG_ARCH_HAS_PARANOID_L1D_FLUSH
+	/*
+	 * If L1D flush is supported on mm context switch
+	 * then we use this callback head to queue kill work
+	 * to kill tasks that are not running on SMT disabled
+	 * cores
+	 */
+	struct callback_head		l1d_flush_kill;
+#endif
+
 	/*
 	 * New fields for task_struct should be added above here, so that
 	 * they are included in the randomized portion of task_struct.
merge bot Applied
#21 lore raw

[tip: x86/cpu] x86/process: Make room for TIF_SPEC_L1D_FLUSH

In reply to Balbir Singh
The following commit has been merged into the x86/cpu branch of tip:

The following commit has been merged into the x86/cpu branch of tip:

Commit-ID:     8aacd1eab53ec853c2d29cdc9b64e9dc87d2a519
Gitweb:        https://git.kernel.org/tip/8aacd1eab53ec853c2d29cdc9b64e9dc87d2a519
Author:        Balbir Singh <[email protected]>
AuthorDate:    Mon, 26 Apr 2021 22:09:43 +02:00
Committer:     Thomas Gleixner <[email protected]>
CommitterDate: Wed, 28 Jul 2021 11:42:24 +02:00

x86/process: Make room for TIF_SPEC_L1D_FLUSH

The upcoming support for paranoid L1D flush in switch_mm() requires that TIF_SPEC_IB and the new TIF_SPEC_L1D_FLUSH are two consecutive bits in thread_info::flags.

Move TIF_SPEC_FORCE_UPDATE to a spare bit to make room for the new one.

---
 arch/x86/include/asm/thread_info.h | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
arch/x86/include/asm/thread_info.h +2 -2
index de406d9..d9afd35 100644
@@ -81,7 +81,6 @@ struct thread_info {
 #define TIF_SINGLESTEP		4	/* reenable singlestep on user return*/
 #define TIF_SSBD		5	/* Speculative store bypass disable */
 #define TIF_SPEC_IB		9	/* Indirect branch speculation mitigation */
-#define TIF_SPEC_FORCE_UPDATE	10	/* Force speculation MSR update in context switch */
 #define TIF_USER_RETURN_NOTIFY	11	/* notify kernel of userspace return */
 #define TIF_UPROBE		12	/* breakpointed or singlestepping */
 #define TIF_PATCH_PENDING	13	/* pending live patching update */
@@ -93,6 +92,7 @@ struct thread_info {
 #define TIF_MEMDIE		20	/* is terminating due to OOM killer */
 #define TIF_POLLING_NRFLAG	21	/* idle is polling for TIF_NEED_RESCHED */
 #define TIF_IO_BITMAP		22	/* uses I/O bitmap */
+#define TIF_SPEC_FORCE_UPDATE	23	/* Force speculation MSR update in context switch */
 #define TIF_FORCED_TF		24	/* true if TF in eflags artificially */
 #define TIF_BLOCKSTEP		25	/* set when we want DEBUGCTLMSR_BTF */
 #define TIF_LAZY_MMU_UPDATES	27	/* task is updating the mmu lazily */
@@ -104,7 +104,6 @@ struct thread_info {
 #define _TIF_SINGLESTEP		(1 << TIF_SINGLESTEP)
 #define _TIF_SSBD		(1 << TIF_SSBD)
 #define _TIF_SPEC_IB		(1 << TIF_SPEC_IB)
-#define _TIF_SPEC_FORCE_UPDATE	(1 << TIF_SPEC_FORCE_UPDATE)
 #define _TIF_USER_RETURN_NOTIFY	(1 << TIF_USER_RETURN_NOTIFY)
 #define _TIF_UPROBE		(1 << TIF_UPROBE)
 #define _TIF_PATCH_PENDING	(1 << TIF_PATCH_PENDING)
@@ -115,6 +114,7 @@ struct thread_info {
 #define _TIF_SLD		(1 << TIF_SLD)
 #define _TIF_POLLING_NRFLAG	(1 << TIF_POLLING_NRFLAG)
 #define _TIF_IO_BITMAP		(1 << TIF_IO_BITMAP)
+#define _TIF_SPEC_FORCE_UPDATE	(1 << TIF_SPEC_FORCE_UPDATE)
 #define _TIF_FORCED_TF		(1 << TIF_FORCED_TF)
 #define _TIF_BLOCKSTEP		(1 << TIF_BLOCKSTEP)
 #define _TIF_LAZY_MMU_UPDATES	(1 << TIF_LAZY_MMU_UPDATES)