Skip to contentSkip to conversation

Core-sched v6+: kernel protection and hotplug fixes

RFC PATCH NAK Joel Fernandes (Google) 3 participants 20 messages Aug 15, 2020 – Aug 22, 2020

Subsystems (8)
lore
Share & tools
lore
#0 lore raw

cover Core-sched v6+: kernel protection and hotplug fixes

Hello!

This series is continuation of main core-sched v6 series [1] and adds support for syscall and IRQ isolation from usermode processes and guests. It is key to safely entering kernel mode in an HT while the other HT is in use by a user or guest. The series also fixes CPU hotplug issues arising because of the cpu_smt_mask changing while the next task is being picked. These hotplug fixes are needed also for kernel protection to work correctly.

The series is based on Thomas's x86/entry tree.

[1] https://lwn.net/Articles/824918/

Background:

Core-scheduling prevents hyperthreads in usermode from attacking each other, but it does not do anything about one of the hyperthreads entering the kernel for any reason. This leaves the door open for MDS and L1TF attacks with concurrent execution sequences between hyperthreads.

This series adds support for protecting all syscall and IRQ kernel mode entries by cleverly tracking when any sibling in a core enter the kernel, and when all the siblings exit the kernel. IPIs are sent to force siblings into the kernel.

Care is taken to avoid waiting in IRQ-disabled sections as Thomas suggested thus avoiding stop_machine deadlocks. Every attempt is made to avoid unnecessary IPIs.

Performance tests:
sysbench is used to test the performance of the patch series. Used a 8 cpu/4 Core VM and ran 2 sysbench tests in parallel. Each sysbench test runs 4 tasks:
sysbench --test=cpu --cpu-max-prime=100000 --num-threads=4 run

Compared the performance results for various combinations as below. The metric below is 'events per second':

1. Coresched disabled

    sysbench-1/sysbench-2 => 175.7/175.6

2. Coreched enabled, both sysbench tagged
sysbench-1/sysbench-2 => 168.8/165.6

3. Coresched enabled, sysbench-1 tagged and sysbench-2 untagged

    sysbench-1/sysbench-2 => 96.4/176.9

4. smt off

    sysbench-1/sysbench-2 => 97.9/98.8

When both sysbench-es are tagged, there is a perf drop of ~4%. With a tagged/untagged case, the tagged one suffers because it always gets stalled when the sibiling enters kernel. But this is no worse than smtoff.

Also a modified rcutorture was used to heavily stress the kernel to make sure there is not crash or instability.

Joel Fernandes (Google) (5):
irq_work: Add support to detect if work is pending entry/idle: Add a common function for activites during idle entry/exit arch/x86: Add a new TIF flag for untrusted tasks kernel/entry: Add support for core-wide protection of kernel-mode entry/idle: Enter and exit kernel protection during idle entry and exit

Vineeth Pillai (7):
entry/kvm: Protect the kernel when entering from guest
bitops: Introduce find_next_or_bit
cpumask: Introduce a new iterator for_each_cpu_wrap_or sched/coresched: Use for_each_cpu(_wrap)_or for pick_next_task sched/coresched: Make core_pick_seq per run-queue sched/coresched: Check for dynamic changes in smt_mask sched/coresched: rq->core should be set only if not previously set

arch/x86/include/asm/thread_info.h |   2 +
arch/x86/kvm/x86.c                 |   3 +
include/asm-generic/bitops/find.h  |  16 ++
include/linux/cpumask.h            |  42 +++++
include/linux/entry-common.h       |  22 +++
include/linux/entry-kvm.h          |  12 ++
include/linux/irq_work.h           |   1 +
include/linux/sched.h              |  12 ++
kernel/entry/common.c              |  88 +++++----
kernel/entry/kvm.c                 |  12 ++
kernel/irq_work.c                  |  11 ++
kernel/sched/core.c                | 281 ++++++++++++++++++++++++++---
kernel/sched/idle.c                |  17 +-
kernel/sched/sched.h               |  11 +-
lib/cpumask.c                      |  53 ++++++
lib/find_bit.c                     |  56 ++++--
16 files changed, 564 insertions(+), 75 deletions(-)

--
2.28.0.220.ged08abb693-goog

#1 lore raw

RFC PATCH 1/12 irq_work: Add support to detect if work is pending

When an unsafe region is entered on an HT, an IPI needs to be sent to siblings to ensure they enter the kernel.

Following are the reasons why we would like to use irq_work to implement forcing of sibling into kernel mode:

1. Existing smp_call infrastructure cannot be used easily since we could end up waiting on CSD lock if previously an smp_call was not yet serviced.

2. I'd like to use generic code, such that there is no need to add an arch-specific IPI.

3. IRQ work already has support to detect that previous work was not yet executed through the IRQ_WORK_PENDING bit.

4. We need to know if the destination of the IPI is not sending more IPIs due to that IPI itself causing an entry into unsafe region.

Support for 4. requires us to be able to detect that irq_work is pending.

This commit therefore adds a way for irq_work users to know if a previous per-HT irq_work is pending. If it is, we need not send new IPIs.

Memory ordering:

I was trying to handle the MP-pattern below. Consider the flag to be the pending bit. P0() is the IRQ work handler. P1() is the code calling irq_work_pending(). P0() already implicitly adds a memory barrier as a part of the atomic_fetch_andnot() before calling work->func(). For P1(), this patch adds the memory barrier as the atomic_read() in this patch's irq_work_pending() is not sufficient.

        P0()
        {
                WRITE_ONCE(buf, 1);
                WRITE_ONCE(flag, 1);
        }

        P1()
        {
                int r1;
                int r2 = 0;

                r1 = READ_ONCE(flag);
                if (r1)
                        r2 = READ_ONCE(buf);
        }
---
 include/linux/irq_work.h |  1 +
 kernel/irq_work.c        | 11 +++++++++++
 2 files changed, 12 insertions(+)
include/linux/irq_work.h +1 -0
index 30823780c192..b26466f95d04 100644
@@ -42,6 +42,7 @@ bool irq_work_queue_on(struct irq_work *work, int cpu);
 
 void irq_work_tick(void);
 void irq_work_sync(struct irq_work *work);
+bool irq_work_pending(struct irq_work *work);
 
 #ifdef CONFIG_IRQ_WORK
 #include <asm/irq_work.h>
kernel/irq_work.c +11 -0
index eca83965b631..2d206d511aa0 100644
@@ -24,6 +24,17 @@
 static DEFINE_PER_CPU(struct llist_head, raised_list);
 static DEFINE_PER_CPU(struct llist_head, lazy_list);
 
+bool irq_work_pending(struct irq_work *work)
+{
+	/*
+	 * Provide ordering to callers who may read other stuff
+	 * after the atomic read (MP-pattern).
+	 */
+	bool ret = atomic_read_acquire(&work->flags) & IRQ_WORK_PENDING;
+
+	return ret;
+}
+
 /*
  * Claim the entry so that no one else will poke at it.
  */
Signature · 1 line
2.28.0.220.ged08abb693-goog
#4 lore raw

RFC PATCH 2/12 entry/idle: Add a common function for activites during idle entry/exit

In reply to Joel Fernandes (Google)

Currently only RCU hooks for idle entry/exit are called. In later patches, kernel-entry protection functionality will be added.

---
 include/linux/entry-common.h | 16 ++++++++++++++++
 kernel/sched/idle.c          | 17 +++++++++--------
 2 files changed, 25 insertions(+), 8 deletions(-)
include/linux/entry-common.h +16 -0
index efebbffcd5cc..2ea0e09b00d5 100644
@@ -369,4 +369,20 @@ void irqentry_exit_cond_resched(void);
  */
 void noinstr irqentry_exit(struct pt_regs *regs, irqentry_state_t state);
 
+/**
+ * generic_idle_enter - Called during entry into idle for housekeeping.
+ */
+static inline void generic_idle_enter(void)
+{
+	rcu_idle_enter();
+}
+
+/**
+ * generic_idle_enter - Called when exiting idle for housekeeping.
+ */
+static inline void generic_idle_exit(void)
+{
+	rcu_idle_exit();
+}
+
 #endif
kernel/sched/idle.c +9 -8
index 3b03bedee280..5db5a946aa3b 100644
@@ -8,6 +8,7 @@
  */
 #include "sched.h"
 
+#include <linux/entry-common.h>
 #include <trace/events/power.h>
 
 /* Linker adds these: start and end of __cpuidle functions */
@@ -54,7 +55,7 @@ __setup("hlt", cpu_idle_nopoll_setup);
 
 static noinline int __cpuidle cpu_idle_poll(void)
 {
-	rcu_idle_enter();
+	generic_idle_enter();
 	trace_cpu_idle_rcuidle(0, smp_processor_id());
 	local_irq_enable();
 	stop_critical_timings();
@@ -64,7 +65,7 @@ static noinline int __cpuidle cpu_idle_poll(void)
 		cpu_relax();
 	start_critical_timings();
 	trace_cpu_idle_rcuidle(PWR_EVENT_EXIT, smp_processor_id());
-	rcu_idle_exit();
+	generic_idle_exit();
 
 	return 1;
 }
@@ -158,7 +159,7 @@ static void cpuidle_idle_call(void)
 
 	if (cpuidle_not_available(drv, dev)) {
 		tick_nohz_idle_stop_tick();
-		rcu_idle_enter();
+		generic_idle_enter();
 
 		default_idle_call();
 		goto exit_idle;
@@ -178,13 +179,13 @@ static void cpuidle_idle_call(void)
 		u64 max_latency_ns;
 
 		if (idle_should_enter_s2idle()) {
-			rcu_idle_enter();
+			generic_idle_enter();
 
 			entered_state = call_cpuidle_s2idle(drv, dev);
 			if (entered_state > 0)
 				goto exit_idle;
 
-			rcu_idle_exit();
+			generic_idle_exit();
 
 			max_latency_ns = U64_MAX;
 		} else {
@@ -192,7 +193,7 @@ static void cpuidle_idle_call(void)
 		}
 
 		tick_nohz_idle_stop_tick();
-		rcu_idle_enter();
+		generic_idle_enter();
 
 		next_state = cpuidle_find_deepest_state(drv, dev, max_latency_ns);
 		call_cpuidle(drv, dev, next_state);
@@ -209,7 +210,7 @@ static void cpuidle_idle_call(void)
 		else
 			tick_nohz_idle_retain_tick();
 
-		rcu_idle_enter();
+		generic_idle_enter();
 
 		entered_state = call_cpuidle(drv, dev, next_state);
 		/*
@@ -227,7 +228,7 @@ static void cpuidle_idle_call(void)
 	if (WARN_ON_ONCE(irqs_disabled()))
 		local_irq_enable();
 
-	rcu_idle_exit();
+	generic_idle_exit();
 }
 
 /*
Signature · 1 line
2.28.0.220.ged08abb693-goog
maintainer NAK
#5 lore raw

Re: [PATCH RFC 02/12] entry/idle: Add a common function for activites during idle entry/exit

Quoted 4 lines · Joel Fernandes (Google)

Currently only RCU hooks for idle entry/exit are called. In later patches, kernel-entry protection functionality will be added.

NAK, rcu_idle_enter() is broken where it is now, it needs to be pushed in deeper:

http://lkml.kernel.org/r/[email protected]

#6 lore raw

Re: [PATCH RFC 02/12] entry/idle: Add a common function for activites during idle entry/exit

Hello Peter,

Quoted 10 lines · On Sat, Aug 15, 2020 at 10:14:41AM +0200, [email protected] wrote:
Quoted 4 lines · Joel Fernandes (Google)

Currently only RCU hooks for idle entry/exit are called. In later patches, kernel-entry protection functionality will be added.

NAK, rcu_idle_enter() is broken where it is now, it needs to be pushed in deeper:

http://lkml.kernel.org/r/[email protected]

Thank you for pointing it out. Not a huge problem, a couple ways I can do it:
1. Move the calls to sched_core_unsafe_{enter,exit}() deeper into the idle loop.
2. Keep the calls to sched_core_unsafe_{enter,exit}() where they are now as in this patch, but leave out the rcu_idle_{enter,exit}() calls alone so they can be moved deeper as you mentioned.

#1 is not necessary for these patches to work and might be overkill.

I'll go the #2 route then. Let me know any other ideas you might have.

What I am trying to do here is to handle a case where task is switching to idle (say it went to sleep) and wakes up later.

usermode -> syscall (kernel mode - so mark unsafe) -> idle (mark safe).

idle -> syscall wakes (kernel mode - so mark unsafe) -> usermode (mark safe).

Thank you,

- Joel

#7 lore raw

RFC PATCH 3/12 arch/x86: Add a new TIF flag for untrusted tasks

In reply to Joel Fernandes (Google)

Add a new TIF flag to indicate whether the kernel needs to be careful and take additional steps to mitigate micro-architectural issues during entry into user or guest mode.

This new flag will be used by the series to determine if waiting is needed or not, during exit to user or guest mode.

---
 arch/x86/include/asm/thread_info.h | 2 ++
 kernel/sched/sched.h               | 6 ++++++
 2 files changed, 8 insertions(+)
arch/x86/include/asm/thread_info.h +2 -0
index 267701ae3d86..42e63969acb3 100644
@@ -98,6 +98,7 @@ struct thread_info {
 #define TIF_IO_BITMAP		22	/* uses I/O bitmap */
 #define TIF_FORCED_TF		24	/* true if TF in eflags artificially */
 #define TIF_BLOCKSTEP		25	/* set when we want DEBUGCTLMSR_BTF */
+#define TIF_UNSAFE_RET   	26	/* On return to process/guest, perform safety checks. */
 #define TIF_LAZY_MMU_UPDATES	27	/* task is updating the mmu lazily */
 #define TIF_SYSCALL_TRACEPOINT	28	/* syscall tracepoint instrumentation */
 #define TIF_ADDR32		29	/* 32-bit address space on 64 bits */
@@ -127,6 +128,7 @@ struct thread_info {
 #define _TIF_IO_BITMAP		(1 << TIF_IO_BITMAP)
 #define _TIF_FORCED_TF		(1 << TIF_FORCED_TF)
 #define _TIF_BLOCKSTEP		(1 << TIF_BLOCKSTEP)
+#define _TIF_UNSAFE_RET 	(1 << TIF_UNSAFE_RET)
 #define _TIF_LAZY_MMU_UPDATES	(1 << TIF_LAZY_MMU_UPDATES)
 #define _TIF_SYSCALL_TRACEPOINT	(1 << TIF_SYSCALL_TRACEPOINT)
 #define _TIF_ADDR32		(1 << TIF_ADDR32)
kernel/sched/sched.h +6 -0
index 3575edc7dc43..1901d11a6f41 100644
@@ -2680,3 +2680,9 @@ static inline bool is_per_cpu_kthread(struct task_struct *p)
 
 void swake_up_all_locked(struct swait_queue_head *q);
 void __prepare_to_swait(struct swait_queue_head *q, struct swait_queue *wait);
+
+#ifdef CONFIG_SCHED_CORE
+#ifndef TIF_UNSAFE_RET
+#define TIF_UNSAFE_RET (0)
+#endif
+#endif
Signature · 1 line
2.28.0.220.ged08abb693-goog
#8 lore raw

RFC PATCH 4/12 kernel/entry: Add support for core-wide protection of kernel-mode

In reply to Joel Fernandes (Google)

Core-scheduling prevents hyperthreads in usermode from attacking each other, but it does not do anything about one of the hyperthreads entering the kernel for any reason. This leaves the door open for MDS and L1TF attacks with concurrent execution sequences between hyperthreads.

This patch therefore adds support for protecting all syscall and IRQ kernel mode entries. Care is taken to track the outermost usermode exit and entry using per-cpu counters. In cases where one of the hyperthreads enter the kernel, no additional IPIs are sent. Further, IPIs are avoided when not needed - example: idle and non-cookie HTs do not need to be forced into kernel mode.

More information about attacks:
For MDS, it is possible for syscalls, IRQ and softirq handlers to leak data to either host or guest attackers. For L1TF, it is possible to leak to guest attackers. There is no possible mitigation involving flushing of buffers to avoid this since the execution of attacker and victims happen concurrently on 2 or more HTs.

---
 include/linux/sched.h |  12 +++
 kernel/entry/common.c |  88 +++++++++++-------
 kernel/sched/core.c   | 205 ++++++++++++++++++++++++++++++++++++++++++
 kernel/sched/sched.h  |   3 +
 4 files changed, 277 insertions(+), 31 deletions(-)
include/linux/sched.h +12 -0
index 295e3258c9bf..910274141ab2 100644
@@ -2054,4 +2054,16 @@ int sched_trace_rq_cpu(struct rq *rq);
 
 const struct cpumask *sched_trace_rd_span(struct root_domain *rd);
 
+#ifdef CONFIG_SCHED_CORE
+void sched_core_unsafe_enter(void);
+void sched_core_unsafe_exit(void);
+void sched_core_unsafe_exit_wait(unsigned long ti_check);
+void sched_core_wait_till_safe(unsigned long ti_check);
+#else
+#define sched_core_unsafe_enter(ignore) do { } while (0)
+#define sched_core_unsafe_exit(ignore) do { } while (0)
+#define sched_core_wait_till_safe(ignore) do { } while (0)
+#define sched_core_unsafe_exit_wait(ignore) do { } while (0)
+#endif
+
 #endif
kernel/entry/common.c +57 -31

122 lines not on this page · raw message

kernel/sched/core.c +205 -0

217 lines not on this page · raw message

kernel/sched/sched.h +3 -0
index 1901d11a6f41..2922e171a1f0 100644
@@ -1040,11 +1040,14 @@ struct rq {
 	unsigned int		core_sched_seq;
 	struct rb_root		core_tree;
 	unsigned char		core_forceidle;
+	struct irq_work		core_irq_work; /* To force HT into kernel */
+	unsigned int		core_this_unsafe_nest;
 
 	/* shared state */
 	unsigned int		core_task_seq;
 	unsigned int		core_pick_seq;
 	unsigned long		core_cookie;
+	unsigned int		core_unsafe_nest;
 #endif
 };
 
Signature · 1 line
2.28.0.220.ged08abb693-goog
#9 lore raw

RFC PATCH 5/12 entry/idle: Enter and exit kernel protection during idle entry and exit

In reply to Joel Fernandes (Google)

Make use of the generic_idle_{enter,exit} helper function added in earlier patches to enter and exit kernel protection.

On exiting idle, protection will be reenabled.

---
 include/linux/entry-common.h | 6 ++++++
 1 file changed, 6 insertions(+)
include/linux/entry-common.h +6 -0
index 2ea0e09b00d5..c833f2fda542 100644
@@ -374,6 +374,9 @@ void noinstr irqentry_exit(struct pt_regs *regs, irqentry_state_t state);
  */
 static inline void generic_idle_enter(void)
 {
+	/* Entering idle ends the protected kernel region. */
+	sched_core_unsafe_exit();
+
 	rcu_idle_enter();
 }
 
@@ -383,6 +386,9 @@ static inline void generic_idle_enter(void)
 static inline void generic_idle_exit(void)
 {
 	rcu_idle_exit();
+
+	/* Exiting idle (re)starts the protected kernel region. */
+	sched_core_unsafe_enter();
 }
 
 #endif
Signature · 1 line
2.28.0.220.ged08abb693-goog
#10 lore raw

RFC PATCH 6/12 entry/kvm: Protect the kernel when entering from guest

In reply to Joel Fernandes (Google)

From: Vineeth Pillai <[email protected]>

Similar to how user to kernel mode transitions are protected in earlier patches, protect the entry into kernel from guest mode as well.

---
 arch/x86/kvm/x86.c        |  3 +++
 include/linux/entry-kvm.h | 12 ++++++++++++
 kernel/entry/kvm.c        | 12 ++++++++++++
 3 files changed, 27 insertions(+)
arch/x86/kvm/x86.c +3 -0
index 82d4a9e88908..b8a6faf78dc6 100644
@@ -8484,6 +8484,8 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)
 	 */
 	smp_mb__after_srcu_read_unlock();
 
+	kvm_exit_to_guest_mode(vcpu);
+
 	/*
 	 * This handles the case where a posted interrupt was
 	 * notified with kvm_vcpu_kick.
@@ -8578,6 +8580,7 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)
 		}
 	}
 
+	kvm_enter_from_guest_mode(vcpu);
 	local_irq_enable();
 	preempt_enable();
 
include/linux/entry-kvm.h +12 -0
index 0cef17afb41a..32aabb7f3e6d 100644
@@ -77,4 +77,16 @@ static inline bool xfer_to_guest_mode_work_pending(void)
 }
 #endif /* CONFIG_KVM_XFER_TO_GUEST_WORK */
 
+/**
+ * kvm_enter_from_guest_mode - Hook called just after entering kernel from guest.
+ * @vcpu:   Pointer to the current VCPU data
+ */
+void kvm_enter_from_guest_mode(struct kvm_vcpu *vcpu);
+
+/**
+ * kvm_exit_to_guest_mode - Hook called just before entering guest from kernel.
+ * @vcpu:   Pointer to the current VCPU data
+ */
+void kvm_exit_to_guest_mode(struct kvm_vcpu *vcpu);
+
 #endif
kernel/entry/kvm.c +12 -0
index eb1a8a4c867c..994af4241646 100644
@@ -49,3 +49,15 @@ int xfer_to_guest_mode_handle_work(struct kvm_vcpu *vcpu)
 	return xfer_to_guest_mode_work(vcpu, ti_work);
 }
 EXPORT_SYMBOL_GPL(xfer_to_guest_mode_handle_work);
+
+void kvm_enter_from_guest_mode(struct kvm_vcpu *vcpu)
+{
+	sched_core_unsafe_enter();
+}
+EXPORT_SYMBOL_GPL(kvm_enter_from_guest_mode);
+
+void kvm_exit_to_guest_mode(struct kvm_vcpu *vcpu)
+{
+	sched_core_unsafe_exit_wait(XFER_TO_GUEST_MODE_WORK);
+}
+EXPORT_SYMBOL_GPL(kvm_exit_to_guest_mode);
Signature · 1 line
2.28.0.220.ged08abb693-goog
#11 lore raw

RFC PATCH 7/12 bitops: Introduce find_next_or_bit

In reply to Joel Fernandes (Google)

From: Vineeth Pillai <[email protected]>

Hotplug fixes to core-scheduling require a new bitops API.

Introduce a new API find_next_or_bit() which returns the bit number of the next set bit in OR-ed bit masks of the given bit masks.

---
 include/asm-generic/bitops/find.h | 16 +++++++++
 lib/find_bit.c                    | 56 +++++++++++++++++++++++++------
 2 files changed, 61 insertions(+), 11 deletions(-)
include/asm-generic/bitops/find.h +16 -0
index 9fdf21302fdf..0b476ca0d665 100644
@@ -32,6 +32,22 @@ extern unsigned long find_next_and_bit(const unsigned long *addr1,
 		unsigned long offset);
 #endif
 
+#ifndef find_next_or_bit
+/**
+ * find_next_or_bit - find the next set bit in any memory regions
+ * @addr1: The first address to base the search on
+ * @addr2: The second address to base the search on
+ * @offset: The bitnumber to start searching at
+ * @size: The bitmap size in bits
+ *
+ * Returns the bit number for the next set bit
+ * If no bits are set, returns @size.
+ */
+extern unsigned long find_next_or_bit(const unsigned long *addr1,
+		const unsigned long *addr2, unsigned long size,
+		unsigned long offset);
+#endif
+
 #ifndef find_next_zero_bit
 /**
  * find_next_zero_bit - find the next cleared bit in a memory region
lib/find_bit.c +45 -11

114 lines not on this page · raw message

Signature · 1 line
2.28.0.220.ged08abb693-goog
#12 lore raw

RFC PATCH 8/12 cpumask: Introduce a new iterator for_each_cpu_wrap_or

In reply to Joel Fernandes (Google)

From: Vineeth Pillai <[email protected]>

Hotplug fixes to core-scheduling require a new cpumask iterator which iterates through all online cpus in both the given cpumasks. This patch introduces it.

---
 include/linux/cpumask.h | 42 ++++++++++++++++++++++++++++++++
 lib/cpumask.c           | 53 +++++++++++++++++++++++++++++++++++++++++
 2 files changed, 95 insertions(+)
include/linux/cpumask.h +42 -0
index f0d895d6ac39..03e8c57c6ca6 100644
@@ -207,6 +207,10 @@ static inline int cpumask_any_and_distribute(const struct cpumask *src1p,
 	for ((cpu) = 0; (cpu) < 1; (cpu)++, (void)mask, (void)(start))
 #define for_each_cpu_and(cpu, mask1, mask2)	\
 	for ((cpu) = 0; (cpu) < 1; (cpu)++, (void)mask1, (void)mask2)
+#define for_each_cpu_or(cpu, mask1, mask2)	\
+	for ((cpu) = 0; (cpu) < 1; (cpu)++, (void)mask1, (void)mask2)
+#define for_each_cpu_wrap_or(cpu, mask1, mask2, start)	\
+	for ((cpu) = 0; (cpu) < 1; (cpu)++, (void)mask1, (void)mask2, (void)(start))
 #else
 /**
  * cpumask_first - get the first cpu in a cpumask
@@ -248,6 +252,7 @@ static inline unsigned int cpumask_next_zero(int n, const struct cpumask *srcp)
 }
 
 int cpumask_next_and(int n, const struct cpumask *, const struct cpumask *);
+int cpumask_next_or(int n, const struct cpumask *, const struct cpumask *);
 int cpumask_any_but(const struct cpumask *mask, unsigned int cpu);
 unsigned int cpumask_local_spread(unsigned int i, int node);
 int cpumask_any_and_distribute(const struct cpumask *src1p,
@@ -278,6 +283,8 @@ int cpumask_any_and_distribute(const struct cpumask *src1p,
 		(cpu) < nr_cpu_ids;)
 
 extern int cpumask_next_wrap(int n, const struct cpumask *mask, int start, bool wrap);
+extern int cpumask_next_wrap_or(int n, const struct cpumask *mask1,
+				 const struct cpumask *mask2, int start, bool wrap);
 
 /**
  * for_each_cpu_wrap - iterate over every cpu in a mask, starting at a specified location
@@ -294,6 +301,22 @@ extern int cpumask_next_wrap(int n, const struct cpumask *mask, int start, bool
 	     (cpu) < nr_cpumask_bits;						\
 	     (cpu) = cpumask_next_wrap((cpu), (mask), (start), true))
 
+/**
+ * for_each_cpu_wrap_or - iterate over every cpu in both masks, starting at a specified location
+ * @cpu: the (optionally unsigned) integer iterator
+ * @mask1: the first cpumask pointer
+ * @mask2: the second cpumask pointer
+ * @start: the start location
+ *
+ * The implementation does not assume any bit both masks are set (including @start).
+ *
+ * After the loop, cpu is >= nr_cpu_ids.
+ */
+#define for_each_cpu_wrap_or(cpu, mask1, mask2, start)					\
+	for ((cpu) = cpumask_next_wrap_or((start)-1, (mask1), (mask2), (start), false);	\
+	     (cpu) < nr_cpumask_bits;						\
+	     (cpu) = cpumask_next_wrap_or((cpu), (mask1), (mask2), (start), true))
+
 /**
  * for_each_cpu_and - iterate over every cpu in both masks
  * @cpu: the (optionally unsigned) integer iterator
@@ -312,6 +335,25 @@ extern int cpumask_next_wrap(int n, const struct cpumask *mask, int start, bool
 	for ((cpu) = -1;						\
 		(cpu) = cpumask_next_and((cpu), (mask1), (mask2)),	\
 		(cpu) < nr_cpu_ids;)
+
+/**
+ * for_each_cpu_or - iterate over every cpu in both masks
+ * @cpu: the (optionally unsigned) integer iterator
+ * @mask1: the first cpumask pointer
+ * @mask2: the second cpumask pointer
+ *
+ * This saves a temporary CPU mask in many places.  It is equivalent to:
+ *	struct cpumask tmp;
+ *	cpumask_and(&tmp, &mask1, &mask2);
+ *	for_each_cpu(cpu, &tmp)
+ *		...
+ *
+ * After the loop, cpu is >= nr_cpu_ids.
+ */
+#define for_each_cpu_or(cpu, mask1, mask2)				\
+	for ((cpu) = -1;						\
+		(cpu) = cpumask_next_or((cpu), (mask1), (mask2)),	\
+		(cpu) < nr_cpu_ids;)
 #endif /* SMP */
 
 #define CPU_BITS_NONE						\
lib/cpumask.c +53 -0

65 lines not on this page · raw message

Signature · 1 line
2.28.0.220.ged08abb693-goog
#13 lore raw

RFC PATCH 9/12 sched/coresched: Use for_each_cpu(_wrap)_or for pick_next_task

In reply to Joel Fernandes (Google)

From: Vineeth Pillai <[email protected]>

During a CPU hotplug event, schedule would be called with the hotplugged CPU not in the cpumask. So use for_each_cpu(_wrap)_or to include the current cpu in the task pick loop.

---
 kernel/sched/core.c | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)
kernel/sched/core.c +6 -3
index ff13254ed317..3e9df8221c62 100644
@@ -4663,7 +4663,7 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
 
 	/* reset state */
 	rq->core->core_cookie = 0UL;
-	for_each_cpu(i, smt_mask) {
+	for_each_cpu_or(i, smt_mask, cpumask_of(cpu)) {
 		struct rq *rq_i = cpu_rq(i);
 
 		trace_printk("CPU %d is in smt_mask, resetting\n", i);
@@ -4685,7 +4685,7 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
 	 */
 	for_each_class(class) {
 again:
-		for_each_cpu_wrap(i, smt_mask, cpu) {
+		for_each_cpu_wrap_or(i, smt_mask, cpumask_of(cpu), cpu) {
 			struct rq *rq_i = cpu_rq(i);
 			struct task_struct *p;
 
@@ -4774,6 +4774,9 @@ next_class:;
 	rq->core->core_pick_seq = rq->core->core_task_seq;
 	next = rq->core_pick;
 	rq->core_sched_seq = rq->core->core_pick_seq;
+
+	/* Something should have been selected for current CPU */
+	WARN_ON_ONCE(!next);
 	trace_printk("picked: %s/%d %lx\n", next->comm, next->pid, next->core_cookie);
 
 	/*
@@ -4784,7 +4787,7 @@ next_class:;
 	 * their task. This ensures there is no inter-sibling overlap between
 	 * non-matching user state.
 	 */
-	for_each_cpu(i, smt_mask) {
+	for_each_cpu_or(i, smt_mask, cpumask_of(cpu)) {
 		struct rq *rq_i = cpu_rq(i);
 
 		WARN_ON_ONCE(!rq_i->core_pick);
Signature · 1 line
2.28.0.220.ged08abb693-goog
#14 lore raw

RFC PATCH 10/12 sched/coresched: Make core_pick_seq per run-queue

In reply to Joel Fernandes (Google)

From: Vineeth Pillai <[email protected]>

core_pick_seq is a core wide counter to identify if a task pick has been made for a CPU by its sibling. But during hotplug scenarios, pick cannot be made for CPUs that are offline and when they come up, they get tricked because the counter is incremented.

So, make core_pick_seq per run-queue.

---
 kernel/sched/core.c  | 19 ++++++++++---------
 kernel/sched/sched.h |  2 +-
 2 files changed, 11 insertions(+), 10 deletions(-)
kernel/sched/core.c +10 -9
index 3e9df8221c62..48a49168e57f 100644
@@ -4623,9 +4623,9 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
 	 * pointers are all still valid), and we haven't scheduled the last
 	 * pick yet, do so now.
 	 */
-	if (rq->core->core_pick_seq == rq->core->core_task_seq &&
-	    rq->core->core_pick_seq != rq->core_sched_seq) {
-		WRITE_ONCE(rq->core_sched_seq, rq->core->core_pick_seq);
+	if (rq->core_pick_seq == rq->core->core_task_seq &&
+	    rq->core_pick_seq != rq->core_sched_seq) {
+		WRITE_ONCE(rq->core_sched_seq, rq->core_pick_seq);
 
 		next = rq->core_pick;
 		if (next != prev) {
@@ -4635,7 +4635,7 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
 
 		trace_printk("pick pre selected (%u %u %u): %s/%d %lx\n",
 			     rq->core->core_task_seq,
-			     rq->core->core_pick_seq,
+			     rq->core_pick_seq,
 			     rq->core_sched_seq,
 			     next->comm, next->pid,
 			     next->core_cookie);
@@ -4649,7 +4649,7 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
 	smt_mask = cpu_smt_mask(cpu);
 
 	/*
-	 * core->core_task_seq, core->core_pick_seq, rq->core_sched_seq
+	 * core->core_task_seq, rq->core_pick_seq, rq->core_sched_seq
 	 *
 	 * @task_seq guards the task state ({en,de}queues)
 	 * @pick_seq is the @task_seq we did a selection on
@@ -4667,8 +4667,10 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
 		struct rq *rq_i = cpu_rq(i);
 
 		trace_printk("CPU %d is in smt_mask, resetting\n", i);
-
-		rq_i->core_pick = NULL;
+		if (rq_i->core_pick) {
+			WRITE_ONCE(rq_i->core_sched_seq, rq_i->core_pick_seq);
+			rq_i->core_pick = NULL;
+		}
 
 		if (rq_i->core_forceidle) {
 			need_sync = true;
@@ -4771,9 +4773,7 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
 next_class:;
 	}
 
-	rq->core->core_pick_seq = rq->core->core_task_seq;
 	next = rq->core_pick;
-	rq->core_sched_seq = rq->core->core_pick_seq;
 
 	/* Something should have been selected for current CPU */
 	WARN_ON_ONCE(!next);
@@ -4801,6 +4801,7 @@ next_class:;
 			continue;
 
 		if (rq_i->curr != rq_i->core_pick) {
+			WRITE_ONCE(rq_i->core_pick_seq, rq->core->core_task_seq);
 			trace_printk("IPI(%d)\n", i);
 			resched_curr(rq_i);
 		}
kernel/sched/sched.h +1 -1
index 2922e171a1f0..c7caece2df6e 100644
@@ -1036,6 +1036,7 @@ struct rq {
 	/* per rq */
 	struct rq		*core;
 	struct task_struct	*core_pick;
+	unsigned int		core_pick_seq;
 	unsigned int		core_enabled;
 	unsigned int		core_sched_seq;
 	struct rb_root		core_tree;
@@ -1045,7 +1046,6 @@ struct rq {
 
 	/* shared state */
 	unsigned int		core_task_seq;
-	unsigned int		core_pick_seq;
 	unsigned long		core_cookie;
 	unsigned int		core_unsafe_nest;
 #endif
Signature · 1 line
2.28.0.220.ged08abb693-goog
#15 lore raw

RFC PATCH 11/12 sched/coresched: Check for dynamic changes in smt_mask

In reply to Joel Fernandes (Google)

From: Vineeth Pillai <[email protected]>

There are multiple loops in pick_next_task that iterate over CPUs in smt_mask. During a hotplug event, sibling could be removed from the smt_mask while pick_next_task is running. So we cannot trust the mask across the different loops. This can confuse the logic.

Add a retry logic if smt_mask changes between the loops.

---
 kernel/sched/core.c | 22 +++++++++++++++++++++-
 1 file changed, 21 insertions(+), 1 deletion(-)
kernel/sched/core.c +21 -1
index 48a49168e57f..5da5b2317b21 100644
@@ -4613,6 +4613,7 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
 	const struct sched_class *class;
 	const struct cpumask *smt_mask;
 	int i, j, cpu, occ = 0;
+	int smt_weight;
 	bool need_sync;
 
 	if (!sched_core_enabled(rq))
@@ -4648,6 +4649,9 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
 	cpu = cpu_of(rq);
 	smt_mask = cpu_smt_mask(cpu);
 
+retry_select:
+	smt_weight = cpumask_weight(smt_mask);
+
 	/*
 	 * core->core_task_seq, rq->core_pick_seq, rq->core_sched_seq
 	 *
@@ -4691,6 +4695,14 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
 			struct rq *rq_i = cpu_rq(i);
 			struct task_struct *p;
 
+			/*
+			 * During hotplug online a sibling can be added in
+			 * the smt_mask * while we are here. If so, we would
+			 * need to restart selection by resetting all over.
+			 */
+			if (unlikely(smt_weight != cpumask_weight(smt_mask)))
+				goto retry_select;
+
 			if (rq_i->core_pick)
 				continue;
 
@@ -4790,7 +4802,15 @@ next_class:;
 	for_each_cpu_or(i, smt_mask, cpumask_of(cpu)) {
 		struct rq *rq_i = cpu_rq(i);
 
-		WARN_ON_ONCE(!rq_i->core_pick);
+		WARN_ON_ONCE(smt_weight == cpumask_weight(smt_mask) && !rq->core_pick);
+
+		/*
+		 * During hotplug online a sibling can be added in the smt_mask
+		 * while we are here. We might have missed picking a task for it.
+		 * Ignore it now as a schedule on that sibling will correct itself.
+		 */
+		if (!rq_i->core_pick)
+			continue;
 
 		if (is_idle_task(rq_i->core_pick) && rq_i->nr_running)
 			rq_i->core_forceidle = true;
Signature · 1 line
2.28.0.220.ged08abb693-goog
#16 lore raw

RFC PATCH 12/12 sched/coresched: rq->core should be set only if not previously set

In reply to Joel Fernandes (Google)

From: Vineeth Pillai <[email protected]>

During hotplug events, smt_mask would not contain all the CPUs in a core and this can cause reassigning of rq->core, which breaks the core-wide counters that are needed for tracking kernel-mode entry/exits.

This patch therefore makes sure that rq->core does not change once it is set.

---
 kernel/sched/core.c | 28 ++++++++++++++++------------
 1 file changed, 16 insertions(+), 12 deletions(-)
kernel/sched/core.c +16 -12
index 5da5b2317b21..464493f3a759 100644
@@ -7519,21 +7519,25 @@ int sched_cpu_starting(unsigned int cpu)
 	struct rq *rq, *core_rq = NULL;
 	int i;
 
-	for_each_cpu(i, smt_mask) {
-		rq = cpu_rq(i);
-		if (rq->core && rq->core == rq)
-			core_rq = rq;
-		init_irq_work(&rq->core_irq_work, sched_core_irq_work);
-	}
+	core_rq = cpu_rq(cpu)->core;
+
+	if (!core_rq) {
+		for_each_cpu(i, smt_mask) {
+			rq = cpu_rq(i);
+			if (rq->core && rq->core == rq)
+				core_rq = rq;
+			init_irq_work(&rq->core_irq_work, sched_core_irq_work);
+		}
 
-	if (!core_rq)
-		core_rq = cpu_rq(cpu);
+		if (!core_rq)
+			core_rq = cpu_rq(cpu);
 
-	for_each_cpu(i, smt_mask) {
-		rq = cpu_rq(i);
+		for_each_cpu(i, smt_mask) {
+			rq = cpu_rq(i);
 
-		WARN_ON_ONCE(rq->core && rq->core != core_rq);
-		rq->core = core_rq;
+			WARN_ON_ONCE(rq->core && rq->core != core_rq);
+			rq->core = core_rq;
+		}
 	}
 
 	printk("core: %d -> %d\n", cpu, cpu_of(core_rq));
Signature · 1 line
2.28.0.220.ged08abb693-goog
#17 lore raw
In reply to Joel Fernandes (Google)
Quoted 1 line · Joel Fernandes (Google)

This series is continuation of main core-sched v6 series [1] and adds support

- Is this really "RFC"? Seeing multiple authors implies this is looking to get merged. :)

- Is this on top of the v6 core-sched series (or instead of)? (That series doesn't seem to be RFC any more either, IMO, if it's at v6.) If this replaces that series, isn't this v7? I don't see the v6 core-sched series in -next anywhere?

- Whatever the case, please include the series version in the "[PATCH vN ...]" portion of the Subject, otherwise things like "b4" don't really know what's happening.

Sorry for the drive-by comments, but I was really confused trying to understand how this fit together. :)

Signature · 1 line
Kees Cook
#18 lore raw
Quoted 5 lines · Kees Cook
Quoted 1 line · Joel Fernandes (Google)

This series is continuation of main core-sched v6 series [1] and adds support

- Is this really "RFC"? Seeing multiple authors implies this is looking to get merged. :)

This mini series was inspired by discussions/concerns mostly by Thomas so that's why I kept it as RFC since I thought more comments will come.

Quoted 4 lines

- Is this on top of the v6 core-sched series (or instead of)? (That series doesn't seem to be RFC any more either, IMO, if it's at v6.) If this replaces that series, isn't this v7? I don't see the v6 core-sched series in -next anywhere?

Yes, it was just a continuation of that. Actually I screwed up, I should have just posted the whole v6 again with these patches with appropriate subject-prefix. That would prevent confusion.

Quoted 3 lines

- Whatever the case, please include the series version in the "[PATCH vN ...]" portion of the Subject, otherwise things like "b4" don't really know what's happening.

Got it.

Quoted 2 lines

Sorry for the drive-by comments, but I was really confused trying to understand how this fit together. :)

No problem, your comments are very valid so they are welcome :)

thanks,

- Joel

Quoted 2 lines
Signature · 1 line
Kees Cook
#19 lore raw
In reply to Joel Fernandes (Google)
Quoted 10 lines · Joel Fernandes (Google)

Hello!

This series is continuation of main core-sched v6 series [1] and adds support for syscall and IRQ isolation from usermode processes and guests. It is key to safely entering kernel mode in an HT while the other HT is in use by a user or guest. The series also fixes CPU hotplug issues arising because of the cpu_smt_mask changing while the next task is being picked. These hotplug fixes are needed also for kernel protection to work correctly.

Quoted 3 lines

The series is based on Thomas's x86/entry tree.

[1] https://lwn.net/Articles/824918/

Hello,
Just wanted to mention that we are talking about this series during the refereed talk on Monday at 16:00 UTC :
https://linuxplumbersconf.org/event/7/contributions/648/

The slides are here with some nice pictures showing the kernel protection stuff:
https://docs.google.com/presentation/d/1VzeQo3AyGTN35DJ3LKoPWBfiZHZJiF8q0NrX9eVYG70/edit?usp=sharing

And Julien has some promising data to share which he just collected with this series (and will add them to the slides).

Looking forward to possibly seeing you there and your participation for these topics both during the refereed talk and the scheduler MC, thanks!

- Joel

Quoted 82 lines

Background:

Core-scheduling prevents hyperthreads in usermode from attacking each other, but it does not do anything about one of the hyperthreads entering the kernel for any reason. This leaves the door open for MDS and L1TF attacks with concurrent execution sequences between hyperthreads.

This series adds support for protecting all syscall and IRQ kernel mode entries by cleverly tracking when any sibling in a core enter the kernel, and when all the siblings exit the kernel. IPIs are sent to force siblings into the kernel.

Care is taken to avoid waiting in IRQ-disabled sections as Thomas suggested thus avoiding stop_machine deadlocks. Every attempt is made to avoid unnecessary IPIs.

Performance tests:
sysbench is used to test the performance of the patch series. Used a 8 cpu/4 Core VM and ran 2 sysbench tests in parallel. Each sysbench test runs 4 tasks:
sysbench --test=cpu --cpu-max-prime=100000 --num-threads=4 run

Compared the performance results for various combinations as below. The metric below is 'events per second':

1. Coresched disabled

    sysbench-1/sysbench-2 => 175.7/175.6

2. Coreched enabled, both sysbench tagged
sysbench-1/sysbench-2 => 168.8/165.6

3. Coresched enabled, sysbench-1 tagged and sysbench-2 untagged

    sysbench-1/sysbench-2 => 96.4/176.9

4. smt off

    sysbench-1/sysbench-2 => 97.9/98.8

When both sysbench-es are tagged, there is a perf drop of ~4%. With a tagged/untagged case, the tagged one suffers because it always gets stalled when the sibiling enters kernel. But this is no worse than smtoff.

Also a modified rcutorture was used to heavily stress the kernel to make sure there is not crash or instability.

Joel Fernandes (Google) (5):
irq_work: Add support to detect if work is pending entry/idle: Add a common function for activites during idle entry/exit arch/x86: Add a new TIF flag for untrusted tasks kernel/entry: Add support for core-wide protection of kernel-mode entry/idle: Enter and exit kernel protection during idle entry and exit

Vineeth Pillai (7):
entry/kvm: Protect the kernel when entering from guest
bitops: Introduce find_next_or_bit
cpumask: Introduce a new iterator for_each_cpu_wrap_or sched/coresched: Use for_each_cpu(_wrap)_or for pick_next_task sched/coresched: Make core_pick_seq per run-queue sched/coresched: Check for dynamic changes in smt_mask sched/coresched: rq->core should be set only if not previously set

arch/x86/include/asm/thread_info.h |   2 +
arch/x86/kvm/x86.c                 |   3 +
include/asm-generic/bitops/find.h  |  16 ++
include/linux/cpumask.h            |  42 +++++
include/linux/entry-common.h       |  22 +++
include/linux/entry-kvm.h          |  12 ++
include/linux/irq_work.h           |   1 +
include/linux/sched.h              |  12 ++
kernel/entry/common.c              |  88 +++++----
kernel/entry/kvm.c                 |  12 ++
kernel/irq_work.c                  |  11 ++
kernel/sched/core.c                | 281 ++++++++++++++++++++++++++---
kernel/sched/idle.c                |  17 +-
kernel/sched/sched.h               |  11 +-
lib/cpumask.c                      |  53 ++++++
lib/find_bit.c                     |  56 ++++--
16 files changed, 564 insertions(+), 75 deletions(-)

--
2.28.0.220.ged08abb693-goog