Core-sched v6+: kernel protection and hotplug fixes
Subsystems (8)
cover Core-sched v6+: kernel protection and hotplug fixes
Hello!
This series is continuation of main core-sched v6 series [1] and adds support for syscall and IRQ isolation from usermode processes and guests. It is key to safely entering kernel mode in an HT while the other HT is in use by a user or guest. The series also fixes CPU hotplug issues arising because of the cpu_smt_mask changing while the next task is being picked. These hotplug fixes are needed also for kernel protection to work correctly.
The series is based on Thomas's x86/entry tree.
[1] https://lwn.net/Articles/824918/
Background:
Core-scheduling prevents hyperthreads in usermode from attacking each other, but it does not do anything about one of the hyperthreads entering the kernel for any reason. This leaves the door open for MDS and L1TF attacks with concurrent execution sequences between hyperthreads.
This series adds support for protecting all syscall and IRQ kernel mode entries by cleverly tracking when any sibling in a core enter the kernel, and when all the siblings exit the kernel. IPIs are sent to force siblings into the kernel.
Care is taken to avoid waiting in IRQ-disabled sections as Thomas suggested thus avoiding stop_machine deadlocks. Every attempt is made to avoid unnecessary IPIs.
Performance tests:
sysbench is used to test the performance of the patch series. Used a 8 cpu/4 Core VM and ran 2 sysbench tests in parallel. Each sysbench test runs 4 tasks:
sysbench --test=cpu --cpu-max-prime=100000 --num-threads=4 run
Compared the performance results for various combinations as below. The metric below is 'events per second':
1. Coresched disabled
sysbench-1/sysbench-2 => 175.7/175.6
2. Coreched enabled, both sysbench tagged
sysbench-1/sysbench-2 => 168.8/165.6
3. Coresched enabled, sysbench-1 tagged and sysbench-2 untagged
sysbench-1/sysbench-2 => 96.4/176.9
4. smt off
sysbench-1/sysbench-2 => 97.9/98.8
When both sysbench-es are tagged, there is a perf drop of ~4%. With a tagged/untagged case, the tagged one suffers because it always gets stalled when the sibiling enters kernel. But this is no worse than smtoff.
Also a modified rcutorture was used to heavily stress the kernel to make sure there is not crash or instability.
Joel Fernandes (Google) (5):irq_work: Add support to detect if work is pending entry/idle: Add a common function for activites during idle entry/exit arch/x86: Add a new TIF flag for untrusted tasks kernel/entry: Add support for core-wide protection of kernel-mode entry/idle: Enter and exit kernel protection during idle entry and exit
Vineeth Pillai (7):
entry/kvm: Protect the kernel when entering from guest
bitops: Introduce find_next_or_bit
cpumask: Introduce a new iterator for_each_cpu_wrap_or sched/coresched: Use for_each_cpu(_wrap)_or for pick_next_task sched/coresched: Make core_pick_seq per run-queue sched/coresched: Check for dynamic changes in smt_mask sched/coresched: rq->core should be set only if not previously set
arch/x86/include/asm/thread_info.h | 2 + arch/x86/kvm/x86.c | 3 + include/asm-generic/bitops/find.h | 16 ++ include/linux/cpumask.h | 42 +++++ include/linux/entry-common.h | 22 +++ include/linux/entry-kvm.h | 12 ++ include/linux/irq_work.h | 1 + include/linux/sched.h | 12 ++ kernel/entry/common.c | 88 +++++---- kernel/entry/kvm.c | 12 ++ kernel/irq_work.c | 11 ++ kernel/sched/core.c | 281 ++++++++++++++++++++++++++--- kernel/sched/idle.c | 17 +- kernel/sched/sched.h | 11 +- lib/cpumask.c | 53 ++++++ lib/find_bit.c | 56 ++++-- 16 files changed, 564 insertions(+), 75 deletions(-)
--
2.28.0.220.ged08abb693-goog
RFC PATCH 1/12 irq_work: Add support to detect if work is pending
When an unsafe region is entered on an HT, an IPI needs to be sent to siblings to ensure they enter the kernel.
Following are the reasons why we would like to use irq_work to implement forcing of sibling into kernel mode:
1. Existing smp_call infrastructure cannot be used easily since we could end up waiting on CSD lock if previously an smp_call was not yet serviced.
2. I'd like to use generic code, such that there is no need to add an arch-specific IPI.
3. IRQ work already has support to detect that previous work was not yet executed through the IRQ_WORK_PENDING bit.
4. We need to know if the destination of the IPI is not sending more IPIs due to that IPI itself causing an entry into unsafe region.
Support for 4. requires us to be able to detect that irq_work is pending.
This commit therefore adds a way for irq_work users to know if a previous per-HT irq_work is pending. If it is, we need not send new IPIs.
Memory ordering:
I was trying to handle the MP-pattern below. Consider the flag to be the pending bit. P0() is the IRQ work handler. P1() is the code calling irq_work_pending(). P0() already implicitly adds a memory barrier as a part of the atomic_fetch_andnot() before calling work->func(). For P1(), this patch adds the memory barrier as the atomic_read() in this patch's irq_work_pending() is not sufficient.
P0()
{
WRITE_ONCE(buf, 1);
WRITE_ONCE(flag, 1);
}
P1()
{
int r1;
int r2 = 0;
r1 = READ_ONCE(flag);
if (r1)
r2 = READ_ONCE(buf);
}--- include/linux/irq_work.h | 1 + kernel/irq_work.c | 11 +++++++++++ 2 files changed, 12 insertions(+)
include/linux/irq_work.h +1 -0
@@ -42,6 +42,7 @@ bool irq_work_queue_on(struct irq_work *work, int cpu);void irq_work_tick (void );void irq_work_sync (struct irq_work *work); +bool irq_work_pending (struct irq_work *work);#ifdef CONFIG_IRQ_WORK #include <asm/irq_work.h>
kernel/irq_work.c +11 -0
@@ -24,6 +24,17 @@ static DEFINE_PER_CPU(struct llist_head, raised_list); static DEFINE_PER_CPU(struct llist_head, lazy_list); +bool irq_work_pending (struct irq_work *work) +{ +/* +* Provide ordering to callers who may read other stuff +* after the atomic read (MP-pattern). +*/ +bool ret =atomic_read_acquire (&work->flags) &IRQ_WORK_PENDING ; + +return ret; +} +/* * Claim the entry so that no one else will poke at it. */
Signature · 1 line
2.28.0.220.ged08abb693-goog
[email protected]
maintainerRe: [PATCH RFC 01/12] irq_work: Add support to detect if work is pending
On Fri, Aug 14, 2020 at 11:18:57PM -0400, Joel Fernandes (Google) wrote:
Re: [PATCH RFC 01/12] irq_work: Add support to detect if work is pending
Quoted 3 lines · On Sat, Aug 15, 2020 at 10:13:30AM +0200, [email protected] wrote:
On Fri, Aug 14, 2020 at 11:18:57PM -0400, Joel Fernandes (Google) wrote:
Thank you, so that means I can drop this patch then.
- Joel
RFC PATCH 2/12 entry/idle: Add a common function for activites during idle entry/exit
In reply to Joel Fernandes (Google)Currently only RCU hooks for idle entry/exit are called. In later patches, kernel-entry protection functionality will be added.
--- include/linux/entry-common.h | 16 ++++++++++++++++ kernel/sched/idle.c | 17 +++++++++-------- 2 files changed, 25 insertions(+), 8 deletions(-)
include/linux/entry-common.h +16 -0
@@ -369,4 +369,20 @@ void irqentry_exit_cond_resched(void);*/ void noinstr irqentry_exit(struct pt_regs *regs, irqentry_state_t state); +/** +* generic_idle_enter - Called during entry into idle for housekeeping. +*/ +static inline void generic_idle_enter (void ) +{ +rcu_idle_enter (); +} + +/** +* generic_idle_enter - Called when exiting idle for housekeeping. +*/ +static inline void generic_idle_exit (void ) +{ +rcu_idle_exit (); +} +#endif
kernel/sched/idle.c +9 -8
@@ -8,6 +8,7 @@*/ #include "sched.h" +#include <linux/entry-common.h> #include <trace/events/power.h> /* Linker adds these: start and end of __cpuidle functions */
@@ -54,7 +55,7 @@ __setup("hlt", cpu_idle_nopoll_setup); static noinline int __cpuidle cpu_idle_poll(void) { -rcu_idle_enter (); +generic_idle_enter ();trace_cpu_idle_rcuidle (0 ,smp_processor_id ());local_irq_enable ();stop_critical_timings ();
@@ -64,7 +65,7 @@ static noinline int __cpuidle cpu_idle_poll(void)cpu_relax ();start_critical_timings ();trace_cpu_idle_rcuidle (PWR_EVENT_EXIT ,smp_processor_id ()); -rcu_idle_exit (); +generic_idle_exit ();return 1 ; }
@@ -158,7 +159,7 @@ static void cpuidle_idle_call(void)if (cpuidle_not_available (drv, dev)) {tick_nohz_idle_stop_tick (); -rcu_idle_enter (); +generic_idle_enter ();default_idle_call (); goto exit_idle;
@@ -178,13 +179,13 @@ static void cpuidle_idle_call(void)u64 max_latency_ns;if (idle_should_enter_s2idle ()) { -rcu_idle_enter (); +generic_idle_enter (); entered_state =call_cpuidle_s2idle (drv, dev);if (entered_state >0 ) goto exit_idle; -rcu_idle_exit (); +generic_idle_exit (); max_latency_ns =U64_MAX ; }else {
@@ -192,7 +193,7 @@ static void cpuidle_idle_call(void) }tick_nohz_idle_stop_tick (); -rcu_idle_enter (); +generic_idle_enter (); next_state =cpuidle_find_deepest_state (drv, dev, max_latency_ns);call_cpuidle (drv, dev, next_state);
@@ -209,7 +210,7 @@ static void cpuidle_idle_call(void)else tick_nohz_idle_retain_tick (); -rcu_idle_enter (); +generic_idle_enter (); entered_state =call_cpuidle (drv, dev, next_state);/*
@@ -227,7 +228,7 @@ static void cpuidle_idle_call(void)if (WARN_ON_ONCE (irqs_disabled ()))local_irq_enable (); -rcu_idle_exit (); +generic_idle_exit (); }/*
Signature · 1 line
2.28.0.220.ged08abb693-goog
Re: [PATCH RFC 02/12] entry/idle: Add a common function for activites during idle entry/exit
Quoted 4 lines · Joel Fernandes (Google)
Currently only RCU hooks for idle entry/exit are called. In later patches, kernel-entry protection functionality will be added.
Signed-off-by Joel Fernandes (Google) <[email protected]>
NAK, rcu_idle_enter() is broken where it is now, it needs to be pushed in deeper:
Re: [PATCH RFC 02/12] entry/idle: Add a common function for activites during idle entry/exit
Hello Peter,
Quoted 10 lines · On Sat, Aug 15, 2020 at 10:14:41AM +0200, [email protected] wrote:
Quoted 4 lines · Joel Fernandes (Google)
Currently only RCU hooks for idle entry/exit are called. In later patches, kernel-entry protection functionality will be added.
Signed-off-by Joel Fernandes (Google) <[email protected]>NAK,
rcu_idle_enter()is broken where it is now, it needs to be pushed in deeper:
Thank you for pointing it out. Not a huge problem, a couple ways I can do it:
1. Move the calls to sched_core_unsafe_{enter,exit}() deeper into the idle loop.
2. Keep the calls to sched_core_unsafe_{enter,exit}() where they are now as in this patch, but leave out the rcu_idle_{enter,exit}() calls alone so they can be moved deeper as you mentioned.
#1 is not necessary for these patches to work and might be overkill.
I'll go the #2 route then. Let me know any other ideas you might have.
What I am trying to do here is to handle a case where task is switching to idle (say it went to sleep) and wakes up later.
usermode -> syscall (kernel mode - so mark unsafe) -> idle (mark safe).
idle -> syscall wakes (kernel mode - so mark unsafe) -> usermode (mark safe).
Thank you,
- Joel
RFC PATCH 3/12 arch/x86: Add a new TIF flag for untrusted tasks
In reply to Joel Fernandes (Google)Add a new TIF flag to indicate whether the kernel needs to be careful and take additional steps to mitigate micro-architectural issues during entry into user or guest mode.
This new flag will be used by the series to determine if waiting is needed or not, during exit to user or guest mode.
--- arch/x86/include/asm/thread_info.h | 2 ++ kernel/sched/sched.h | 6 ++++++ 2 files changed, 8 insertions(+)
arch/x86/include/asm/thread_info.h +2 -0
@@ -98,6 +98,7 @@ struct thread_info {#define TIF_IO_BITMAP 22/* uses I/O bitmap */ #define TIF_FORCED_TF 24/* true if TF in eflags artificially */ #define TIF_BLOCKSTEP 25/* set when we want DEBUGCTLMSR_BTF */ +#define TIF_UNSAFE_RET 26/* On return to process/guest, perform safety checks. */ #define TIF_LAZY_MMU_UPDATES 27/* task is updating the mmu lazily */ #define TIF_SYSCALL_TRACEPOINT 28/* syscall tracepoint instrumentation */ #define TIF_ADDR32 29/* 32-bit address space on 64 bits */
@@ -127,6 +128,7 @@ struct thread_info {#define _TIF_IO_BITMAP (1 << TIF_IO_BITMAP)#define _TIF_FORCED_TF (1 << TIF_FORCED_TF)#define _TIF_BLOCKSTEP (1 << TIF_BLOCKSTEP) +#define _TIF_UNSAFE_RET (1 << TIF_UNSAFE_RET)#define _TIF_LAZY_MMU_UPDATES (1 << TIF_LAZY_MMU_UPDATES)#define _TIF_SYSCALL_TRACEPOINT (1 << TIF_SYSCALL_TRACEPOINT)#define _TIF_ADDR32 (1 << TIF_ADDR32)
kernel/sched/sched.h +6 -0
@@ -2680,3 +2680,9 @@ static inline bool is_per_cpu_kthread(struct task_struct *p)void swake_up_all_locked (struct swait_queue_head *q);void __prepare_to_swait (struct swait_queue_head *q,struct swait_queue *wait); + +#ifdef CONFIG_SCHED_CORE +#ifndef TIF_UNSAFE_RET +#define TIF_UNSAFE_RET (0) +#endif +#endif
Signature · 1 line
2.28.0.220.ged08abb693-goog
RFC PATCH 4/12 kernel/entry: Add support for core-wide protection of kernel-mode
In reply to Joel Fernandes (Google)Core-scheduling prevents hyperthreads in usermode from attacking each other, but it does not do anything about one of the hyperthreads entering the kernel for any reason. This leaves the door open for MDS and L1TF attacks with concurrent execution sequences between hyperthreads.
This patch therefore adds support for protecting all syscall and IRQ kernel mode entries. Care is taken to track the outermost usermode exit and entry using per-cpu counters. In cases where one of the hyperthreads enter the kernel, no additional IPIs are sent. Further, IPIs are avoided when not needed - example: idle and non-cookie HTs do not need to be forced into kernel mode.
More information about attacks:
For MDS, it is possible for syscalls, IRQ and softirq handlers to leak data to either host or guest attackers. For L1TF, it is possible to leak to guest attackers. There is no possible mitigation involving flushing of buffers to avoid this since the execution of attacker and victims happen concurrently on 2 or more HTs.
--- include/linux/sched.h | 12 +++ kernel/entry/common.c | 88 +++++++++++------- kernel/sched/core.c | 205 ++++++++++++++++++++++++++++++++++++++++++ kernel/sched/sched.h | 3 + 4 files changed, 277 insertions(+), 31 deletions(-)
include/linux/sched.h +12 -0
@@ -2054,4 +2054,16 @@ int sched_trace_rq_cpu(struct rq *rq);const struct cpumask *sched_trace_rd_span (struct root_domain *rd); +#ifdef CONFIG_SCHED_CORE +void sched_core_unsafe_enter (void ); +void sched_core_unsafe_exit (void ); +void sched_core_unsafe_exit_wait (unsigned long ti_check); +void sched_core_wait_till_safe (unsigned long ti_check); +#else +#define sched_core_unsafe_enter (ignore) do { } while (0) +#define sched_core_unsafe_exit (ignore) do { } while (0) +#define sched_core_wait_till_safe (ignore) do { } while (0) +#define sched_core_unsafe_exit_wait (ignore) do { } while (0) +#endif +#endif
kernel/entry/common.c +57 -31
122 lines not on this page · raw message
kernel/sched/core.c +205 -0
217 lines not on this page · raw message
kernel/sched/sched.h +3 -0
@@ -1040,11 +1040,14 @@ struct rq {unsigned int core_sched_seq;struct rb_root core_tree;unsigned char core_forceidle; +struct irq_work core_irq_work;/* To force HT into kernel */ +unsigned int core_this_unsafe_nest;/* shared state */ unsigned int core_task_seq;unsigned int core_pick_seq;unsigned long core_cookie; +unsigned int core_unsafe_nest;#endif };
Signature · 1 line
2.28.0.220.ged08abb693-goog
RFC PATCH 5/12 entry/idle: Enter and exit kernel protection during idle entry and exit
In reply to Joel Fernandes (Google)Make use of the generic_idle_{enter,exit} helper function added in earlier patches to enter and exit kernel protection.
On exiting idle, protection will be reenabled.
--- include/linux/entry-common.h | 6 ++++++ 1 file changed, 6 insertions(+)
include/linux/entry-common.h +6 -0
@@ -374,6 +374,9 @@ void noinstr irqentry_exit(struct pt_regs *regs, irqentry_state_t state);*/ static inline void generic_idle_enter (void ) { +/* Entering idle ends the protected kernel region. */ +sched_core_unsafe_exit (); +rcu_idle_enter (); }
@@ -383,6 +386,9 @@ static inline void generic_idle_enter(void)static inline void generic_idle_exit (void ) {rcu_idle_exit (); + +/* Exiting idle (re)starts the protected kernel region. */ +sched_core_unsafe_enter (); }#endif
Signature · 1 line
2.28.0.220.ged08abb693-goog
RFC PATCH 6/12 entry/kvm: Protect the kernel when entering from guest
In reply to Joel Fernandes (Google)From: Vineeth Pillai <[email protected]>
Similar to how user to kernel mode transitions are protected in earlier patches, protect the entry into kernel from guest mode as well.
--- arch/x86/kvm/x86.c | 3 +++ include/linux/entry-kvm.h | 12 ++++++++++++ kernel/entry/kvm.c | 12 ++++++++++++ 3 files changed, 27 insertions(+)
arch/x86/kvm/x86.c +3 -0
@@ -8484,6 +8484,8 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)*/ smp_mb__after_srcu_read_unlock (); +kvm_exit_to_guest_mode (vcpu); +/* * This handles the case where a posted interrupt was * notified with kvm_vcpu_kick.
@@ -8578,6 +8580,7 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu) } } +kvm_enter_from_guest_mode (vcpu);local_irq_enable ();preempt_enable ();
include/linux/entry-kvm.h +12 -0
@@ -77,4 +77,16 @@ static inline bool xfer_to_guest_mode_work_pending(void) }#endif /* CONFIG_KVM_XFER_TO_GUEST_WORK */ +/** +* kvm_enter_from_guest_mode - Hook called just after entering kernel from guest. +* @vcpu: Pointer to the current VCPU data +*/ +void kvm_enter_from_guest_mode (struct kvm_vcpu *vcpu); + +/** +* kvm_exit_to_guest_mode - Hook called just before entering guest from kernel. +* @vcpu: Pointer to the current VCPU data +*/ +void kvm_exit_to_guest_mode (struct kvm_vcpu *vcpu); +#endif
kernel/entry/kvm.c +12 -0
@@ -49,3 +49,15 @@ int xfer_to_guest_mode_handle_work(struct kvm_vcpu *vcpu)return xfer_to_guest_mode_work (vcpu, ti_work); }EXPORT_SYMBOL_GPL (xfer_to_guest_mode_handle_work); + +void kvm_enter_from_guest_mode (struct kvm_vcpu *vcpu) +{ +sched_core_unsafe_enter (); +} +EXPORT_SYMBOL_GPL (kvm_enter_from_guest_mode); + +void kvm_exit_to_guest_mode (struct kvm_vcpu *vcpu) +{ +sched_core_unsafe_exit_wait (XFER_TO_GUEST_MODE_WORK ); +} +EXPORT_SYMBOL_GPL (kvm_exit_to_guest_mode);
Signature · 1 line
2.28.0.220.ged08abb693-goog
RFC PATCH 7/12 bitops: Introduce find_next_or_bit
In reply to Joel Fernandes (Google)From: Vineeth Pillai <[email protected]>
Hotplug fixes to core-scheduling require a new bitops API.
Introduce a new API find_next_or_bit() which returns the bit number of the next set bit in OR-ed bit masks of the given bit masks.
--- include/asm-generic/bitops/find.h | 16 +++++++++ lib/find_bit.c | 56 +++++++++++++++++++++++++------ 2 files changed, 61 insertions(+), 11 deletions(-)
include/asm-generic/bitops/find.h +16 -0
@@ -32,6 +32,22 @@ extern unsigned long find_next_and_bit(const unsigned long *addr1, unsigned long offset);#endif +#ifndef find_next_or_bit +/** +* find_next_or_bit - find the next set bit in any memory regions +* @addr1: The first address to base the search on +* @addr2: The second address to base the search on +* @offset: The bitnumber to start searching at +* @size: The bitmap size in bits +* +* Returns the bit number for the next set bit +* If no bits are set, returns @size. +*/ +extern unsigned long find_next_or_bit (const unsigned long *addr1, +const unsigned long *addr2,unsigned long size, +unsigned long offset); +#endif +#ifndef find_next_zero_bit/** * find_next_zero_bit - find the next cleared bit in a memory region
lib/find_bit.c +45 -11
114 lines not on this page · raw message
Signature · 1 line
2.28.0.220.ged08abb693-goog
RFC PATCH 8/12 cpumask: Introduce a new iterator for_each_cpu_wrap_or
In reply to Joel Fernandes (Google)From: Vineeth Pillai <[email protected]>
Hotplug fixes to core-scheduling require a new cpumask iterator which iterates through all online cpus in both the given cpumasks. This patch introduces it.
--- include/linux/cpumask.h | 42 ++++++++++++++++++++++++++++++++ lib/cpumask.c | 53 +++++++++++++++++++++++++++++++++++++++++ 2 files changed, 95 insertions(+)
include/linux/cpumask.h +42 -0
@@ -207,6 +207,10 @@ static inline int cpumask_any_and_distribute(const struct cpumask *src1p, for ((cpu) = 0; (cpu) < 1; (cpu)++, (void)mask, (void)(start))#define for_each_cpu_and (cpu, mask1, mask2) \ for ((cpu) = 0; (cpu) < 1; (cpu)++, (void)mask1, (void)mask2) +#define for_each_cpu_or (cpu, mask1, mask2) \ + for ((cpu) = 0; (cpu) < 1; (cpu)++, (void)mask1, (void)mask2) +#define for_each_cpu_wrap_or (cpu, mask1, mask2, start) \ + for ((cpu) = 0; (cpu) < 1; (cpu)++, (void)mask1, (void)mask2, (void)(start))#else /** * cpumask_first - get the first cpu in a cpumask
@@ -248,6 +252,7 @@ static inline unsigned int cpumask_next_zero(int n, const struct cpumask *srcp) }int cpumask_next_and (int n,const struct cpumask *,const struct cpumask *); +int cpumask_next_or (int n,const struct cpumask *,const struct cpumask *);int cpumask_any_but (const struct cpumask *mask,unsigned int cpu);unsigned int cpumask_local_spread (unsigned int i,int node); int cpumask_any_and_distribute(const struct cpumask *src1p,
@@ -278,6 +283,8 @@ int cpumask_any_and_distribute(const struct cpumask *src1p, (cpu) < nr_cpu_ids;)extern int cpumask_next_wrap (int n,const struct cpumask *mask,int start,bool wrap); +extern int cpumask_next_wrap_or (int n,const struct cpumask *mask1, +const struct cpumask *mask2,int start,bool wrap);/** * for_each_cpu_wrap - iterate over every cpu in a mask, starting at a specified location
@@ -294,6 +301,22 @@ extern int cpumask_next_wrap(int n, const struct cpumask *mask, int start, bool (cpu) < nr_cpumask_bits; \ (cpu) = cpumask_next_wrap((cpu), (mask), (start), true)) +/** +* for_each_cpu_wrap_or - iterate over every cpu in both masks, starting at a specified location +* @cpu: the (optionally unsigned) integer iterator +* @mask1: the first cpumask pointer +* @mask2: the second cpumask pointer +* @start: the start location +* +* The implementation does not assume any bit both masks are set (including @start). +* +* After the loop, cpu is >= nr_cpu_ids. +*/ +#define for_each_cpu_wrap_or (cpu, mask1, mask2, start) \ + for ((cpu) = cpumask_next_wrap_or((start)-1, (mask1), (mask2), (start), false); \ + (cpu) < nr_cpumask_bits; \ + (cpu) = cpumask_next_wrap_or((cpu), (mask1), (mask2), (start), true)) +/** * for_each_cpu_and - iterate over every cpu in both masks * @cpu: the (optionally unsigned) integer iterator
@@ -312,6 +335,25 @@ extern int cpumask_next_wrap(int n, const struct cpumask *mask, int start, boolfor ((cpu) =-1 ; \ (cpu) =cpumask_next_and ((cpu), (mask1), (mask2)), \ (cpu) < nr_cpu_ids;) + +/** +* for_each_cpu_or - iterate over every cpu in both masks +* @cpu: the (optionally unsigned) integer iterator +* @mask1: the first cpumask pointer +* @mask2: the second cpumask pointer +* +* This saves a temporary CPU mask in many places. It is equivalent to: +* struct cpumask tmp; +* cpumask_and(&tmp, &mask1, &mask2); +* for_each_cpu(cpu, &tmp) +* ... +* +* After the loop, cpu is >= nr_cpu_ids. + */ +#define for_each_cpu_or (cpu, mask1, mask2) \ + for ((cpu) = -1; \ + (cpu) = cpumask_next_or((cpu), (mask1), (mask2)), \ + (cpu) < nr_cpu_ids;)#endif /* SMP */ #define CPU_BITS_NONE \
lib/cpumask.c +53 -0
65 lines not on this page · raw message
Signature · 1 line
2.28.0.220.ged08abb693-goog
RFC PATCH 9/12 sched/coresched: Use for_each_cpu(_wrap)_or for pick_next_task
In reply to Joel Fernandes (Google)From: Vineeth Pillai <[email protected]>
During a CPU hotplug event, schedule would be called with the hotplugged CPU not in the cpumask. So use for_each_cpu(_wrap)_or to include the current cpu in the task pick loop.
--- kernel/sched/core.c | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-)
kernel/sched/core.c +6 -3
@@ -4663,7 +4663,7 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)/* reset state */ rq->core->core_cookie =0UL ; -for_each_cpu (i, smt_mask) { +for_each_cpu_or (i, smt_mask,cpumask_of (cpu)) {struct rq *rq_i =cpu_rq (i);trace_printk ("CPU %d is in smt_mask, resetting\n" , i);
@@ -4685,7 +4685,7 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)*/ for_each_class (class) { again: -for_each_cpu_wrap (i, smt_mask, cpu) { +for_each_cpu_wrap_or (i, smt_mask,cpumask_of (cpu), cpu) {struct rq *rq_i =cpu_rq (i);struct task_struct *p;
@@ -4774,6 +4774,9 @@ next_class:; rq->core->core_pick_seq = rq->core->core_task_seq; next = rq->core_pick; rq->core_sched_seq = rq->core->core_pick_seq; + +/* Something should have been selected for current CPU */ +WARN_ON_ONCE (!next);trace_printk ("picked: %s/%d %lx\n" , next->comm, next->pid, next->core_cookie);/*
@@ -4784,7 +4787,7 @@ next_class:;* their task. This ensures there is no inter-sibling overlap between * non-matching user state. */ -for_each_cpu (i, smt_mask) { +for_each_cpu_or (i, smt_mask,cpumask_of (cpu)) {struct rq *rq_i =cpu_rq (i);WARN_ON_ONCE (!rq_i->core_pick);
Signature · 1 line
2.28.0.220.ged08abb693-goog
RFC PATCH 10/12 sched/coresched: Make core_pick_seq per run-queue
In reply to Joel Fernandes (Google)From: Vineeth Pillai <[email protected]>
core_pick_seq is a core wide counter to identify if a task pick has been made for a CPU by its sibling. But during hotplug scenarios, pick cannot be made for CPUs that are offline and when they come up, they get tricked because the counter is incremented.
So, make core_pick_seq per run-queue.
--- kernel/sched/core.c | 19 ++++++++++--------- kernel/sched/sched.h | 2 +- 2 files changed, 11 insertions(+), 10 deletions(-)
kernel/sched/core.c +10 -9
@@ -4623,9 +4623,9 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)* pointers are all still valid), and we haven't scheduled the last * pick yet, do so now. */ -if (rq->core->core_pick_seq == rq->core->core_task_seq && - rq->core->core_pick_seq != rq->core_sched_seq) { -WRITE_ONCE (rq->core_sched_seq, rq->core->core_pick_seq); +if (rq->core_pick_seq == rq->core->core_task_seq && + rq->core_pick_seq != rq->core_sched_seq) { +WRITE_ONCE (rq->core_sched_seq, rq->core_pick_seq); next = rq->core_pick;if (next != prev) {
@@ -4635,7 +4635,7 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)trace_printk ("pick pre selected (%u %u %u): %s/%d %lx\n" , rq->core->core_task_seq, - rq->core->core_pick_seq, + rq->core_pick_seq, rq->core_sched_seq, next->comm, next->pid, next->core_cookie);
@@ -4649,7 +4649,7 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf) smt_mask =cpu_smt_mask (cpu);/* -* core->core_task_seq, core->core_pick_seq, rq->core_sched_seq +* core->core_task_seq, rq->core_pick_seq, rq->core_sched_seq * * @task_seq guards the task state ({en,de}queues) * @pick_seq is the @task_seq we did a selection on
@@ -4667,8 +4667,10 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)struct rq *rq_i =cpu_rq (i);trace_printk ("CPU %d is in smt_mask, resetting\n" , i); - - rq_i->core_pick =NULL ; +if (rq_i->core_pick) { +WRITE_ONCE (rq_i->core_sched_seq, rq_i->core_pick_seq); + rq_i->core_pick =NULL ; + }if (rq_i->core_forceidle) { need_sync = true;
@@ -4771,9 +4773,7 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf) next_class:; } - rq->core->core_pick_seq = rq->core->core_task_seq; next = rq->core_pick; - rq->core_sched_seq = rq->core->core_pick_seq;/* Something should have been selected for current CPU */ WARN_ON_ONCE (!next);
@@ -4801,6 +4801,7 @@ next_class:;continue ;if (rq_i->curr != rq_i->core_pick) { +WRITE_ONCE (rq_i->core_pick_seq, rq->core->core_task_seq);trace_printk ("IPI(%d)\n" , i);resched_curr (rq_i); }
kernel/sched/sched.h +1 -1
@@ -1036,6 +1036,7 @@ struct rq {/* per rq */ struct rq *core;struct task_struct *core_pick; +unsigned int core_pick_seq;unsigned int core_enabled;unsigned int core_sched_seq;struct rb_root core_tree;
@@ -1045,7 +1046,6 @@ struct rq {/* shared state */ unsigned int core_task_seq; -unsigned int core_pick_seq;unsigned long core_cookie;unsigned int core_unsafe_nest;#endif
Signature · 1 line
2.28.0.220.ged08abb693-goog
RFC PATCH 11/12 sched/coresched: Check for dynamic changes in smt_mask
In reply to Joel Fernandes (Google)From: Vineeth Pillai <[email protected]>
There are multiple loops in pick_next_task that iterate over CPUs in smt_mask. During a hotplug event, sibling could be removed from the smt_mask while pick_next_task is running. So we cannot trust the mask across the different loops. This can confuse the logic.
Add a retry logic if smt_mask changes between the loops.
--- kernel/sched/core.c | 22 +++++++++++++++++++++- 1 file changed, 21 insertions(+), 1 deletion(-)
kernel/sched/core.c +21 -1
@@ -4613,6 +4613,7 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)const struct sched_class *class;const struct cpumask *smt_mask;int i, j, cpu, occ =0 ; +int smt_weight;bool need_sync;if (!sched_core_enabled (rq))
@@ -4648,6 +4649,9 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf) cpu =cpu_of (rq); smt_mask =cpu_smt_mask (cpu); +retry_select: + smt_weight =cpumask_weight (smt_mask); +/* * core->core_task_seq, rq->core_pick_seq, rq->core_sched_seq *
@@ -4691,6 +4695,14 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)struct rq *rq_i =cpu_rq (i);struct task_struct *p; +/* +* During hotplug online a sibling can be added in +* the smt_mask * while we are here. If so, we would +* need to restart selection by resetting all over. +*/ +if (unlikely (smt_weight !=cpumask_weight (smt_mask))) + goto retry_select; +if (rq_i->core_pick)continue ;
@@ -4790,7 +4802,15 @@ next_class:;for_each_cpu_or (i, smt_mask,cpumask_of (cpu)) {struct rq *rq_i =cpu_rq (i); -WARN_ON_ONCE (!rq_i->core_pick); +WARN_ON_ONCE (smt_weight ==cpumask_weight (smt_mask) && !rq->core_pick); + +/* +* During hotplug online a sibling can be added in the smt_mask +* while we are here. We might have missed picking a task for it. +* Ignore it now as a schedule on that sibling will correct itself. +*/ +if (!rq_i->core_pick) +continue ;if (is_idle_task (rq_i->core_pick) && rq_i->nr_running) rq_i->core_forceidle = true;
Signature · 1 line
2.28.0.220.ged08abb693-goog
RFC PATCH 12/12 sched/coresched: rq->core should be set only if not previously set
In reply to Joel Fernandes (Google)From: Vineeth Pillai <[email protected]>
During hotplug events, smt_mask would not contain all the CPUs in a core and this can cause reassigning of rq->core, which breaks the core-wide counters that are needed for tracking kernel-mode entry/exits.
This patch therefore makes sure that rq->core does not change once it is set.
--- kernel/sched/core.c | 28 ++++++++++++++++------------ 1 file changed, 16 insertions(+), 12 deletions(-)
kernel/sched/core.c +16 -12
@@ -7519,21 +7519,25 @@ int sched_cpu_starting(unsigned int cpu)struct rq *rq, *core_rq =NULL ;int i; -for_each_cpu (i, smt_mask) { - rq =cpu_rq (i); -if (rq->core && rq->core == rq) - core_rq = rq; -init_irq_work (&rq->core_irq_work, sched_core_irq_work); - } + core_rq =cpu_rq (cpu)->core; + +if (!core_rq) { +for_each_cpu (i, smt_mask) { + rq =cpu_rq (i); +if (rq->core && rq->core == rq) + core_rq = rq; +init_irq_work (&rq->core_irq_work, sched_core_irq_work); + } -if (!core_rq) - core_rq =cpu_rq (cpu); +if (!core_rq) + core_rq =cpu_rq (cpu); -for_each_cpu (i, smt_mask) { - rq =cpu_rq (i); +for_each_cpu (i, smt_mask) { + rq =cpu_rq (i); -WARN_ON_ONCE (rq->core && rq->core != core_rq); - rq->core = core_rq; +WARN_ON_ONCE (rq->core && rq->core != core_rq); + rq->core = core_rq; + } }printk ("core: %d -> %d\n" , cpu,cpu_of (core_rq));
Signature · 1 line
2.28.0.220.ged08abb693-goog
Quoted 1 line · Joel Fernandes (Google)
This series is continuation of main core-sched v6 series [1] and adds support
- Is this really "RFC"? Seeing multiple authors implies this is looking to get merged. :)
- Is this on top of the v6 core-sched series (or instead of)? (That series doesn't seem to be RFC any more either, IMO, if it's at v6.) If this replaces that series, isn't this v7? I don't see the v6 core-sched series in -next anywhere?
- Whatever the case, please include the series version in the "[PATCH vN ...]" portion of the Subject, otherwise things like "b4" don't really know what's happening.
Sorry for the drive-by comments, but I was really confused trying to understand how this fit together. :)
Signature · 1 line
Kees Cook
Quoted 5 lines · Kees Cook
Quoted 1 line · Joel Fernandes (Google)
This series is continuation of main core-sched v6 series [1] and adds support
- Is this really "RFC"? Seeing multiple authors implies this is looking to get merged. :)
This mini series was inspired by discussions/concerns mostly by Thomas so that's why I kept it as RFC since I thought more comments will come.
Quoted 4 lines
- Is this on top of the v6 core-sched series (or instead of)? (That series doesn't seem to be RFC any more either, IMO, if it's at v6.) If this replaces that series, isn't this v7? I don't see the v6 core-sched series in -next anywhere?
Yes, it was just a continuation of that. Actually I screwed up, I should have just posted the whole v6 again with these patches with appropriate subject-prefix. That would prevent confusion.
Quoted 3 lines
- Whatever the case, please include the series version in the "[PATCH vN ...]" portion of the Subject, otherwise things like "b4" don't really know what's happening.
Got it.
Quoted 2 lines
Sorry for the drive-by comments, but I was really confused trying to understand how this fit together. :)
No problem, your comments are very valid so they are welcome :)
thanks,
- Joel
Quoted 2 lines
Signature · 1 line
Kees Cook
Quoted 10 lines · Joel Fernandes (Google)
Hello!
This series is continuation of main core-sched v6 series [1] and adds support for syscall and IRQ isolation from usermode processes and guests. It is key to safely entering kernel mode in an HT while the other HT is in use by a user or guest. The series also fixes CPU hotplug issues arising because of the
cpu_smt_maskchanging while the next task is being picked. These hotplug fixes are needed also for kernel protection to work correctly.
Quoted 3 lines
The series is based on Thomas's x86/entry tree.
Hello,
Just wanted to mention that we are talking about this series during the refereed talk on Monday at 16:00 UTC :
https://linuxplumbersconf.org/event/7/contributions/648/
The slides are here with some nice pictures showing the kernel protection stuff:
https://docs.google.com/presentation/d/1VzeQo3AyGTN35DJ3LKoPWBfiZHZJiF8q0NrX9eVYG70/edit?usp=sharing
And Julien has some promising data to share which he just collected with this series (and will add them to the slides).
Looking forward to possibly seeing you there and your participation for these topics both during the refereed talk and the scheduler MC, thanks!
- Joel
Quoted 82 lines
Background:
Core-scheduling prevents hyperthreads in usermode from attacking each other, but it does not do anything about one of the hyperthreads entering the kernel for any reason. This leaves the door open for MDS and L1TF attacks with concurrent execution sequences between hyperthreads.
This series adds support for protecting all syscall and IRQ kernel mode entries by cleverly tracking when any sibling in a core enter the kernel, and when all the siblings exit the kernel. IPIs are sent to force siblings into the kernel.
Care is taken to avoid waiting in IRQ-disabled sections as Thomas suggested thus avoiding
stop_machinedeadlocks. Every attempt is made to avoid unnecessary IPIs.Performance tests:
sysbench is used to test the performance of the patch series. Used a 8 cpu/4 Core VM and ran 2 sysbench tests in parallel. Each sysbench test runs 4 tasks:
sysbench --test=cpu --cpu-max-prime=100000 --num-threads=4 runCompared the performance results for various combinations as below. The metric below is 'events per second':
1. Coresched disabled
sysbench-1/sysbench-2 => 175.7/175.62. Coreched enabled, both sysbench tagged
sysbench-1/sysbench-2 => 168.8/165.63. Coresched enabled, sysbench-1 tagged and sysbench-2 untagged
sysbench-1/sysbench-2 => 96.4/176.94. smt off
sysbench-1/sysbench-2 => 97.9/98.8When both sysbench-es are tagged, there is a perf drop of ~4%. With a tagged/untagged case, the tagged one suffers because it always gets stalled when the sibiling enters kernel. But this is no worse than smtoff.
Also a modified rcutorture was used to heavily stress the kernel to make sure there is not crash or instability.
Joel Fernandes (Google) (5):
irq_work: Add support to detect if work is pending entry/idle: Add a common function for activites during idle entry/exitarch/x86: Add a new TIF flag for untrusted tasks kernel/entry: Add support for core-wide protection of kernel-mode entry/idle: Enter and exit kernel protection during idle entry and exitVineeth Pillai (7):
entry/kvm: Protect the kernel when entering from guest
bitops: Introducefind_next_or_bit
cpumask: Introduce a new iteratorfor_each_cpu_wrap_orsched/coresched: Use for_each_cpu(_wrap)_or forpick_next_tasksched/coresched: Makecore_pick_seqper run-queue sched/coresched: Check for dynamic changes insmt_masksched/coresched:rq->coreshould be set only if not previously setarch/x86/include/asm/thread_info.h | 2 + arch/x86/kvm/x86.c | 3 + include/asm-generic/bitops/find.h | 16 ++ include/linux/cpumask.h | 42 +++++ include/linux/entry-common.h | 22 +++ include/linux/entry-kvm.h | 12 ++ include/linux/irq_work.h | 1 + include/linux/sched.h | 12 ++ kernel/entry/common.c | 88 +++++---- kernel/entry/kvm.c | 12 ++ kernel/irq_work.c | 11 ++ kernel/sched/core.c | 281 ++++++++++++++++++++++++++--- kernel/sched/idle.c | 17 +- kernel/sched/sched.h | 11 +- lib/cpumask.c | 53 ++++++ lib/find_bit.c | 56 ++++-- 16 files changed, 564 insertions(+), 75 deletions(-)--
2.28.0.220.ged08abb693-goog