Core scheduling (v9)
Subsystems (19)
cover Core scheduling (v9)
Core-Scheduling
===============
Enclosed is series v9 of core scheduling.
v9 is rebased on tip/master (fe4adf6f92c4 ("Merge branch 'irq/core'")).. I hope that this version is acceptable to be merged (pending any new review comments that arise) as the main issues in the past are all resolved:
1. Vruntime comparison.
2. Documentation updates.
3. CGroup and per-task interface developed by Google and Oracle.
4. Hotplug fixes.
Almost all patches also have Reviewed-by or Acked-by tag. See below for full list of changes in v9.
Introduction of feature
=======================
Core scheduling is a feature that allows only trusted tasks to run concurrently on cpus sharing compute resources (eg: hyperthreads on a core). The goal is to mitigate the core-level side-channel attacks without requiring to disable SMT (which has a significant impact on performance in some situations). Core scheduling (as of v7) mitigates user-space to user-space attacks and user to kernel attack when one of the siblings enters the kernel via interrupts or system call.
By default, the feature doesn't change any of the current scheduler behavior. The user decides which tasks can run simultaneously on the same core (for now by having them in the same tagged cgroup). When a tag is enabled in a cgroup and a task from that cgroup is running on a hardware thread, the scheduler ensures that only idle or trusted tasks run on the other sibling(s). Besides security concerns, this feature can also be beneficial for RT and performance applications where we want to control how tasks make use of SMT dynamically.
Both a CGroup and Per-task interface via prctl(2) are provided for configuring core sharing. More details are provided in documentation patch. Kselftests are provided to verify the correctness/rules of the interface.
Testing
=======
ChromeOS testing shows 300% improvement in keypress latency on a Google docs key press with Google hangout test (the maximum latency drops from 150ms to 50ms for keypresses).
Julien: TPCC tests showed improvements with core-scheduling as below. With kernel protection enabled, it does not show any regression. Possibly ASI will improve the performance for those who choose kernel protection (can be toggled through sched_core_protect_kernel sysctl).
average stdev diff
baseline (SMT on) 1197.272 44.78312824
core sched ( kernel protect) 412.9895 45.42734343 -65.51%
core sched (no kernel protect) 686.6515 71.77756931 -42.65%
nosmt 408.667 39.39042872 -65.87%
(Note these results are from v8).
Vineeth tested sysbench and does not see any regressions.
Hong and Aubrey tested v9 and see results similar to v8. There is a known issue with uperf that does regress. This appears to be because of ksoftirq heavily contending with other tasks on the core. The consensus is this can be improved in the future.
Changes in v9
=============
- Note that the vruntime snapshot change is written in 2 patches to show the progression of the idea and prevent merge conflicts:
sched/fair: Snapshot the min_vruntime of CPUs on force idle
sched: Improve snapshotting of min_vruntime for CGroups
Same with the RT priority inversion change:
sched: Fix priority inversion of cookied task with sibling
sched: Improve snapshotting of min_vruntime for CGroups
- Disable coresched on certain AMD HW.
Changes in v8
=============
- New interface/API implementation
- Joel
- Revised kernel protection patch
- Joel
- Revised Hotplug fixes
- Joel
- Minor bug fixes and address review comments
- Vineeth
Changes in v7
=============
- Kernel protection from untrusted usermode tasks
- Joel, Vineeth
- Fix for hotplug crashes and hangs
- Joel, Vineeth
Changes in v6
=============
- Documentation
- Joel
- Pause siblings on entering nmi/irq/softirq
- Joel, Vineeth
- Fix for RCU crash
- Joel
- Fix for a crash in pick_next_task
- Yu Chen, Vineeth
- Minor re-write of core-wide vruntime comparison
- Aaron Lu
- Cleanup: Address Review comments
- Cleanup: Remove hotplug support (for now)
- Build fixes: 32 bit, SMT=n, AUTOGROUP=n etc
- Joel, Vineeth
Changes in v5
=============
- Fixes for cgroup/process tagging during corner cases like cgroup destroy, task moving across cgroups etc
- Tim Chen
- Coresched aware task migrations
- Aubrey Li
- Other minor stability fixes.
Changes in v4
=============
- Implement a core wide min_vruntime for vruntime comparison of tasks across cpus in a core.
- Aaron Lu
- Fixes a typo bug in setting the forced_idle cpu.
- Aaron Lu
Changes in v3
=============
- Fixes the issue of sibling picking up an incompatible task
- Aaron Lu
- Vineeth Pillai
- Julien Desfossez
- Fixes the issue of starving threads due to forced idle
- Peter Zijlstra
- Fixes the refcounting issue when deleting a cgroup with tag
- Julien Desfossez
- Fixes a crash during cpu offline/online with coresched enabled
- Vineeth Pillai
- Fixes a comparison logic issue in sched_core_find
- Aaron Lu
Changes in v2
=============
- Fixes for couple of NULL pointer dereference crashes
- Subhra Mazumdar
- Tim Chen
- Improves priority comparison logic for process in different cpus
- Peter Zijlstra
- Aaron Lu
- Fixes a hard lockup in rq locking
- Vineeth Pillai
- Julien Desfossez
- Fixes a performance issue seen on IO heavy workloads
- Vineeth Pillai
- Julien Desfossez
- Fix for 32bit build
- Aubrey Li
Future work
===========
- Load balancing/Migration fixes for core scheduling. With v6, Load balancing is partially coresched aware, but has some issues w.r.t process/taskgroup weights:
https://lwn.net/ml/linux-kernel/20200225034438.GA617271@z...
Aubrey Li (1):
sched: migration changes for core scheduling
Joel Fernandes (Google) (16):
sched/fair: Snapshot the min_vruntime of CPUs on force idle
sched: Enqueue task into core queue only after vruntime is updated
sched: Simplify the core pick loop for optimized case
sched: Improve snapshotting of min_vruntime for CGroups arch/x86: Add a new TIF flag for untrusted tasks kernel/entry: Add support for core-wide protection of kernel-mode entry/idle: Enter and exit kernel protection during idle entry and exit
sched: Split the cookie and setup per-task cookie on fork
sched: Add a per-thread core scheduling interface
sched: Release references to the per-task cookie on exit sched/debug: Add CGroup node for printing group cookie if SCHED_DEBUG
kselftest: Add tests for core-sched interface
sched: Move core-scheduler interfacing code to a new file
Documentation: Add core scheduling documentation
sched: Add a coresched command line option
sched: Debug bits...
Josh Don (2):
sched: Refactor core cookie into struct
sched: Add a second-level tag for nested CGroup usecase
Peter Zijlstra (11):
sched: Wrap rq::lock access
sched: Introduce sched_class::pick_task() sched/fair: Fix pick_task_fair crashes due to empty rbtree
sched: Core-wide rq->lock
sched/fair: Add a few assertions
sched: Basic tracking of matching tasks
sched: Add core wide task selection and scheduling.
sched: Fix priority inversion of cookied task with sibling
sched: Trivial forced-newidle balancerirq_work: Cleanup
sched: CGroup tagging interface for core scheduling
Vineeth Pillai (2):
sched/fair: Fix forced idle sibling starvation corner case entry/kvm: Protect the kernel when entering from guest
.../admin-guide/hw-vuln/core-scheduling.rst | 330 +++++ Documentation/admin-guide/hw-vuln/index.rst | 1 + .../admin-guide/kernel-parameters.txt | 25 + arch/x86/include/asm/thread_info.h | 2 + arch/x86/kernel/cpu/bugs.c | 19 + arch/x86/kvm/x86.c | 2 + drivers/gpu/drm/i915/i915_request.c | 4 +- include/linux/cpu.h | 1 + include/linux/entry-common.h | 30 +- include/linux/entry-kvm.h | 12 + include/linux/irq_work.h | 33 +- include/linux/irqflags.h | 4 +- include/linux/sched.h | 28 +- include/linux/sched/smt.h | 4 + include/uapi/linux/prctl.h | 3 + kernel/Kconfig.preempt | 5 + kernel/bpf/stackmap.c | 2 +- kernel/cpu.c | 43 + kernel/entry/common.c | 28 +- kernel/entry/kvm.c | 33 + kernel/fork.c | 1 + kernel/irq_work.c | 18 +- kernel/printk/printk.c | 6 +- kernel/rcu/tree.c | 3 +- kernel/sched/Makefile | 1 + kernel/sched/core.c | 1278 +++++++++++++++-- kernel/sched/coretag.c | 819 +++++++++++ kernel/sched/cpuacct.c | 12 +- kernel/sched/deadline.c | 38 +- kernel/sched/debug.c | 12 +- kernel/sched/fair.c | 313 +++- kernel/sched/idle.c | 24 +- kernel/sched/pelt.h | 2 +- kernel/sched/rt.c | 31 +- kernel/sched/sched.h | 315 +++- kernel/sched/stop_task.c | 14 +- kernel/sched/topology.c | 4 +- kernel/sys.c | 3 + kernel/time/tick-sched.c | 6 +- kernel/trace/bpf_trace.c | 2 +- tools/include/uapi/linux/prctl.h | 3 + tools/testing/selftests/sched/.gitignore | 1 + tools/testing/selftests/sched/Makefile | 14 + tools/testing/selftests/sched/config | 1 + .../testing/selftests/sched/test_coresched.c | 818 +++++++++++ 45 files changed, 4033 insertions(+), 315 deletions(-)
create mode 100644 Documentation/admin-guide/hw-vuln/core-scheduling.rst create mode 100644 kernel/sched/coretag.c
create mode 100644 tools/testing/selftests/sched/.gitignore create mode 100644 tools/testing/selftests/sched/Makefile create mode 100644 tools/testing/selftests/sched/config create mode 100644 tools/testing/selftests/sched/test_coresched.c
--
2.29.2.299.gdc1121823c-goog
PATCH 1/32 sched: Wrap rq::lock access
From: Peter Zijlstra <[email protected]>
In preparation of playing games with rq->lock, abstract the thing using an accessor.
--- kernel/sched/core.c | 68 ++++++++++++------------- kernel/sched/cpuacct.c | 12 ++--- kernel/sched/deadline.c | 22 ++++---- kernel/sched/debug.c | 4 +- kernel/sched/fair.c | 38 +++++++------- kernel/sched/idle.c | 4 +- kernel/sched/pelt.h | 2 +- kernel/sched/rt.c | 16 +++--- kernel/sched/sched.h | 108 +++++++++++++++++++++------------------- kernel/sched/topology.c | 4 +- 10 files changed, 141 insertions(+), 137 deletions(-)
kernel/sched/core.c +34 -34
@@ -186,12 +186,12 @@ struct rq *__task_rq_lock(struct task_struct *p, struct rq_flags *rf)for (;;) { rq =task_rq (p); -raw_spin_lock (&rq->lock); +raw_spin_lock (rq_lockp (rq));if (likely (rq ==task_rq (p) && !task_on_rq_migrating (p))) {rq_pin_lock (rq, rf);return rq; } -raw_spin_unlock (&rq->lock); +raw_spin_unlock (rq_lockp (rq));while (unlikely (task_on_rq_migrating (p)))cpu_relax ();
@@ -210,7 +210,7 @@ struct rq *task_rq_lock(struct task_struct *p, struct rq_flags *rf)for (;;) {raw_spin_lock_irqsave (&p->pi_lock, rf->flags); rq =task_rq (p); -raw_spin_lock (&rq->lock); +raw_spin_lock (rq_lockp (rq));/* * move_queued_task() task_rq_lock() *
@@ -232,7 +232,7 @@ struct rq *task_rq_lock(struct task_struct *p, struct rq_flags *rf)rq_pin_lock (rq, rf);return rq; } -raw_spin_unlock (&rq->lock); +raw_spin_unlock (rq_lockp (rq));raw_spin_unlock_irqrestore (&p->pi_lock, rf->flags);while (unlikely (task_on_rq_migrating (p)))
@@ -302,7 +302,7 @@ void update_rq_clock(struct rq *rq) {s64 delta; -lockdep_assert_held (&rq->lock); +lockdep_assert_held (rq_lockp (rq));if (rq->clock_update_flags &RQCF_ACT_SKIP )return ;
@@ -611,7 +611,7 @@ void resched_curr(struct rq *rq)struct task_struct *curr = rq->curr;int cpu; -lockdep_assert_held (&rq->lock); +lockdep_assert_held (rq_lockp (rq));if (test_tsk_need_resched (curr))return ;
@@ -635,10 +635,10 @@ void resched_cpu(int cpu)struct rq *rq =cpu_rq (cpu);unsigned long flags; -raw_spin_lock_irqsave (&rq->lock, flags); +raw_spin_lock_irqsave (rq_lockp (rq), flags);if (cpu_online (cpu) || cpu ==smp_processor_id ())resched_curr (rq); -raw_spin_unlock_irqrestore (&rq->lock, flags); +raw_spin_unlock_irqrestore (rq_lockp (rq), flags); }#ifdef CONFIG_SMP
@@ -1137,7 +1137,7 @@ static inline void uclamp_rq_inc_id(struct rq *rq, struct task_struct *p,struct uclamp_se *uc_se = &p->uclamp[clamp_id];struct uclamp_bucket *bucket; -lockdep_assert_held (&rq->lock); +lockdep_assert_held (rq_lockp (rq));/* Update task effective clamp */ p->uclamp[clamp_id] =uclamp_eff_get (p, clamp_id);
@@ -1177,7 +1177,7 @@ static inline void uclamp_rq_dec_id(struct rq *rq, struct task_struct *p,unsigned int bkt_clamp;unsigned int rq_clamp; -lockdep_assert_held (&rq->lock); +lockdep_assert_held (rq_lockp (rq));/* * If sched_uclamp_used was enabled after task @p was enqueued,
@@ -1807,7 +1807,7 @@ static inline bool is_cpu_allowed(struct task_struct *p, int cpu)static struct rq *move_queued_task (struct rq *rq,struct rq_flags *rf,struct task_struct *p,int new_cpu) { -lockdep_assert_held (&rq->lock); +lockdep_assert_held (rq_lockp (rq));deactivate_task (rq, p,DEQUEUE_NOCLOCK );set_task_cpu (p, new_cpu);
@@ -1973,7 +1973,7 @@ int push_cpu_stop(void *arg)struct task_struct *p = arg;raw_spin_lock_irq (&p->pi_lock); -raw_spin_lock (&rq->lock); +raw_spin_lock (rq_lockp (rq));if (task_rq (p) != rq) goto out_unlock;
@@ -2003,7 +2003,7 @@ int push_cpu_stop(void *arg) out_unlock: rq->push_busy = false; -raw_spin_unlock (&rq->lock); +raw_spin_unlock (rq_lockp (rq));raw_spin_unlock_irq (&p->pi_lock);put_task_struct (p);
@@ -2056,7 +2056,7 @@ __do_set_cpus_allowed(struct task_struct *p, const struct cpumask *new_mask, u32* Because __kthread_bind() calls this on blocked tasks without * holding rq->lock. */ -lockdep_assert_held (&rq->lock); +lockdep_assert_held (rq_lockp (rq));dequeue_task (rq, p,DEQUEUE_SAVE |DEQUEUE_NOCLOCK ); }if (running)
@@ -2395,7 +2395,7 @@ void set_task_cpu(struct task_struct *p, unsigned int new_cpu)* task_rq_lock(). */ WARN_ON_ONCE (debug_locks && !(lockdep_is_held (&p->pi_lock) || -lockdep_is_held (&task_rq (p)->lock))); +lockdep_is_held (rq_lockp (task_rq (p)))));#endif /* * Clearly, migrating tasks to offline CPUs is a fairly daft thing.
@@ -2941,7 +2941,7 @@ ttwu_do_activate(struct rq *rq, struct task_struct *p, int wake_flags, {int en_flags =ENQUEUE_WAKEUP |ENQUEUE_NOCLOCK ; -lockdep_assert_held (&rq->lock); +lockdep_assert_held (rq_lockp (rq));if (p->sched_contributes_to_load) rq->nr_uninterruptible--;
@@ -3941,7 +3941,7 @@ static void do_balance_callbacks(struct rq *rq, struct callback_head *head)void (*func)(struct rq *rq);struct callback_head *next; -lockdep_assert_held (&rq->lock); +lockdep_assert_held (rq_lockp (rq));while (head) { func = (void (*)(struct rq *))head->func;
@@ -3957,7 +3957,7 @@ static inline struct callback_head *splice_balance_callbacks(struct rq *rq) {struct callback_head *head = rq->balance_callback; -lockdep_assert_held (&rq->lock); +lockdep_assert_held (rq_lockp (rq));if (head) { rq->balance_callback =NULL ; rq->balance_flags &= ~BALANCE_WORK ;
@@ -3976,9 +3976,9 @@ static inline void balance_callbacks(struct rq *rq, struct callback_head *head)unsigned long flags;if (unlikely (head)) { -raw_spin_lock_irqsave (&rq->lock, flags); +raw_spin_lock_irqsave (rq_lockp (rq), flags);do_balance_callbacks (rq, head); -raw_spin_unlock_irqrestore (&rq->lock, flags); +raw_spin_unlock_irqrestore (rq_lockp (rq), flags); } }
@@ -4028,10 +4028,10 @@ prepare_lock_switch(struct rq *rq, struct task_struct *next, struct rq_flags *rf* do an early lockdep release here: */ rq_unpin_lock (rq, rf); -spin_release (&rq->lock.dep_map, _THIS_IP_); +spin_release (&rq_lockp (rq)->dep_map, _THIS_IP_);#ifdef CONFIG_DEBUG_SPINLOCK /* this is a valid case when another task releases the spinlock */ - rq->lock.owner = next; +rq_lockp (rq)->owner = next;#endif }
@@ -4042,9 +4042,9 @@ static inline void finish_lock_switch(struct rq *rq)* fix up the runqueue lock - which gets 'carried over' from * prev into current: */ -spin_acquire (&rq->lock.dep_map,0 ,0 , _THIS_IP_); +spin_acquire (&rq_lockp (rq)->dep_map,0 ,0 , _THIS_IP_);balance_switch (rq); -raw_spin_unlock_irq (&rq->lock); +raw_spin_unlock_irq (rq_lockp (rq)); }/*
@@ -5023,7 +5023,7 @@ static void __sched notrace __schedule(bool preempt)rq_unpin_lock (rq, &rf);__balance_callbacks (rq); -raw_spin_unlock_irq (&rq->lock); +raw_spin_unlock_irq (rq_lockp (rq)); } }
@@ -5438,7 +5438,7 @@ void rt_mutex_setprio(struct task_struct *p, struct task_struct *pi_task)rq_unpin_lock (rq, &rf);__balance_callbacks (rq); -raw_spin_unlock (&rq->lock); +raw_spin_unlock (rq_lockp (rq));preempt_enable (); }
@@ -7020,7 +7020,7 @@ void init_idle(struct task_struct *idle, int cpu)__sched_fork (0 , idle);raw_spin_lock_irqsave (&idle->pi_lock, flags); -raw_spin_lock (&rq->lock); +raw_spin_lock (rq_lockp (rq)); idle->state =TASK_RUNNING ; idle->se.exec_start =sched_clock ();
@@ -7058,7 +7058,7 @@ void init_idle(struct task_struct *idle, int cpu)#ifdef CONFIG_SMP idle->on_cpu =1 ;#endif -raw_spin_unlock (&rq->lock); +raw_spin_unlock (rq_lockp (rq));raw_spin_unlock_irqrestore (&idle->pi_lock, flags);/* Set the preempt count _outside_ the spinlocks! */
@@ -7221,7 +7221,7 @@ static void balance_push(struct rq *rq) {struct task_struct *push_task = rq->curr; -lockdep_assert_held (&rq->lock); +lockdep_assert_held (rq_lockp (rq));SCHED_WARN_ON (rq->cpu !=smp_processor_id ());/*
@@ -7242,9 +7242,9 @@ static void balance_push(struct rq *rq)*/ if (!rq->nr_running && !rq_has_pinned_tasks (rq) &&rcuwait_active (&rq->hotplug_wait)) { -raw_spin_unlock (&rq->lock); +raw_spin_unlock (rq_lockp (rq));rcuwait_wake_up (&rq->hotplug_wait); -raw_spin_lock (&rq->lock); +raw_spin_lock (rq_lockp (rq)); }return ; }
@@ -7254,7 +7254,7 @@ static void balance_push(struct rq *rq)* Temporarily drop rq->lock such that we can wake-up the stop task. * Both preemption and IRQs are still disabled. */ -raw_spin_unlock (&rq->lock); +raw_spin_unlock (rq_lockp (rq));stop_one_cpu_nowait (rq->cpu, __balance_push_cpu_stop, push_task,this_cpu_ptr (&push_work));/*
@@ -7262,7 +7262,7 @@ static void balance_push(struct rq *rq)* schedule(). The next pick is obviously going to be the stop task * which is_per_cpu_kthread() and will push this task away. */ -raw_spin_lock (&rq->lock); +raw_spin_lock (rq_lockp (rq)); }static void balance_push_set (int cpu,bool on)
@@ -7682,7 +7682,7 @@ void __init sched_init(void)struct rq *rq; rq =cpu_rq (i); -raw_spin_lock_init (&rq->lock); +raw_spin_lock_init (&rq->__lock); rq->nr_running =0 ; rq->calc_load_active =0 ; rq->calc_load_update = jiffies +LOAD_FREQ ;
kernel/sched/cpuacct.c +6 -6
@@ -112,7 +112,7 @@ static u64 cpuacct_cpuusage_read(struct cpuacct *ca, int cpu,/* * Take rq->lock to make 64-bit read safe on 32-bit platforms. */ -raw_spin_lock_irq (&cpu_rq (cpu)->lock); +raw_spin_lock_irq (rq_lockp (cpu_rq (cpu)));#endif if (index ==CPUACCT_STAT_NSTATS ) {
@@ -126,7 +126,7 @@ static u64 cpuacct_cpuusage_read(struct cpuacct *ca, int cpu, }#ifndef CONFIG_64BIT -raw_spin_unlock_irq (&cpu_rq (cpu)->lock); +raw_spin_unlock_irq (rq_lockp (cpu_rq (cpu)));#endif return data;
@@ -141,14 +141,14 @@ static void cpuacct_cpuusage_write(struct cpuacct *ca, int cpu, u64 val)/* * Take rq->lock to make 64-bit write safe on 32-bit platforms. */ -raw_spin_lock_irq (&cpu_rq (cpu)->lock); +raw_spin_lock_irq (rq_lockp (cpu_rq (cpu)));#endif for (i =0 ; i <CPUACCT_STAT_NSTATS ; i++) cpuusage->usages[i] = val;#ifndef CONFIG_64BIT -raw_spin_unlock_irq (&cpu_rq (cpu)->lock); +raw_spin_unlock_irq (rq_lockp (cpu_rq (cpu)));#endif }
@@ -253,13 +253,13 @@ static int cpuacct_all_seq_show(struct seq_file *m, void *V)* Take rq->lock to make 64-bit read safe on 32-bit * platforms. */ -raw_spin_lock_irq (&cpu_rq (cpu)->lock); +raw_spin_lock_irq (rq_lockp (cpu_rq (cpu)));#endif seq_printf (m," %llu" , cpuusage->usages[index]);#ifndef CONFIG_64BIT -raw_spin_unlock_irq (&cpu_rq (cpu)->lock); +raw_spin_unlock_irq (rq_lockp (cpu_rq (cpu)));#endif }seq_puts (m,"\n" );
kernel/sched/deadline.c +11 -11
@@ -135,7 +135,7 @@ void __add_running_bw(u64 dl_bw, struct dl_rq *dl_rq) {u64 old = dl_rq->running_bw; -lockdep_assert_held (&(rq_of_dl_rq (dl_rq))->lock); +lockdep_assert_held (rq_lockp (rq_of_dl_rq (dl_rq))); dl_rq->running_bw += dl_bw;SCHED_WARN_ON (dl_rq->running_bw < old);/* overflow */ SCHED_WARN_ON (dl_rq->running_bw > dl_rq->this_bw);
@@ -148,7 +148,7 @@ void __sub_running_bw(u64 dl_bw, struct dl_rq *dl_rq) {u64 old = dl_rq->running_bw; -lockdep_assert_held (&(rq_of_dl_rq (dl_rq))->lock); +lockdep_assert_held (rq_lockp (rq_of_dl_rq (dl_rq))); dl_rq->running_bw -= dl_bw;SCHED_WARN_ON (dl_rq->running_bw > old);/* underflow */ if (dl_rq->running_bw > old)
@@ -162,7 +162,7 @@ void __add_rq_bw(u64 dl_bw, struct dl_rq *dl_rq) {u64 old = dl_rq->this_bw; -lockdep_assert_held (&(rq_of_dl_rq (dl_rq))->lock); +lockdep_assert_held (rq_lockp (rq_of_dl_rq (dl_rq))); dl_rq->this_bw += dl_bw;SCHED_WARN_ON (dl_rq->this_bw < old);/* overflow */ }
@@ -172,7 +172,7 @@ void __sub_rq_bw(u64 dl_bw, struct dl_rq *dl_rq) {u64 old = dl_rq->this_bw; -lockdep_assert_held (&(rq_of_dl_rq (dl_rq))->lock); +lockdep_assert_held (rq_lockp (rq_of_dl_rq (dl_rq))); dl_rq->this_bw -= dl_bw;SCHED_WARN_ON (dl_rq->this_bw > old);/* underflow */ if (dl_rq->this_bw > old)
@@ -982,7 +982,7 @@ static int start_dl_timer(struct task_struct *p)ktime_t now, act;s64 delta; -lockdep_assert_held (&rq->lock); +lockdep_assert_held (rq_lockp (rq));/* * We want the timer to fire at the deadline, but considering
@@ -1092,9 +1092,9 @@ static enum hrtimer_restart dl_task_timer(struct hrtimer *timer)* If the runqueue is no longer available, migrate the * task elsewhere. This necessarily changes rq. */ -lockdep_unpin_lock (&rq->lock, rf.cookie); +lockdep_unpin_lock (rq_lockp (rq), rf.cookie); rq =dl_task_offline_migration (rq, p); - rf.cookie =lockdep_pin_lock (&rq->lock); + rf.cookie =lockdep_pin_lock (rq_lockp (rq));update_rq_clock (rq);/*
@@ -1746,7 +1746,7 @@ static void migrate_task_rq_dl(struct task_struct *p, int new_cpu __maybe_unused* from try_to_wake_up(). Hence, p->pi_lock is locked, but * rq->lock is not... So, lock it */ -raw_spin_lock (&rq->lock); +raw_spin_lock (rq_lockp (rq));if (p->dl.dl_non_contending) {sub_running_bw (&p->dl, &rq->dl); p->dl.dl_non_contending =0 ;
@@ -1761,7 +1761,7 @@ static void migrate_task_rq_dl(struct task_struct *p, int new_cpu __maybe_unusedput_task_struct (p); }sub_rq_bw (&p->dl, &rq->dl); -raw_spin_unlock (&rq->lock); +raw_spin_unlock (rq_lockp (rq)); }static void check_preempt_equal_dl (struct rq *rq,struct task_struct *p)
@@ -2306,10 +2306,10 @@ static void pull_dl_task(struct rq *this_rq)double_unlock_balance (this_rq, src_rq);if (push_task) { -raw_spin_unlock (&this_rq->lock); +raw_spin_unlock (rq_lockp (this_rq));stop_one_cpu_nowait (src_rq->cpu, push_cpu_stop, push_task, &src_rq->push_work); -raw_spin_lock (&this_rq->lock); +raw_spin_lock (rq_lockp (this_rq)); } }
kernel/sched/debug.c +2 -2
@@ -551,7 +551,7 @@ void print_cfs_rq(struct seq_file *m, int cpu, struct cfs_rq *cfs_rq)SEQ_printf (m," .%-30s: %Ld.%06ld\n" ,"exec_clock" ,SPLIT_NS (cfs_rq->exec_clock)); -raw_spin_lock_irqsave (&rq->lock, flags); +raw_spin_lock_irqsave (rq_lockp (rq), flags);if (rb_first_cached (&cfs_rq->tasks_timeline)) MIN_vruntime = (__pick_first_entity (cfs_rq))->vruntime; last =__pick_last_entity (cfs_rq);
@@ -559,7 +559,7 @@ void print_cfs_rq(struct seq_file *m, int cpu, struct cfs_rq *cfs_rq) max_vruntime = last->vruntime; min_vruntime = cfs_rq->min_vruntime; rq0_min_vruntime =cpu_rq (0 )->cfs.min_vruntime; -raw_spin_unlock_irqrestore (&rq->lock, flags); +raw_spin_unlock_irqrestore (rq_lockp (rq), flags);SEQ_printf (m," .%-30s: %Ld.%06ld\n" ,"MIN_vruntime" ,SPLIT_NS (MIN_vruntime)); SEQ_printf(m, " .%-30s: %Ld.%06ld\n", "min_vruntime",
kernel/sched/fair.c +19 -19
@@ -1110,7 +1110,7 @@ struct numa_group {static struct numa_group *deref_task_numa_group (struct task_struct *p) {return rcu_dereference_check (p->numa_group, p == current || - (lockdep_is_held (&task_rq (p)->lock) && !READ_ONCE (p->on_cpu))); + (lockdep_is_held (rq_lockp (task_rq (p))) && !READ_ONCE (p->on_cpu))); }static struct numa_group *deref_curr_numa_group (struct task_struct *p)
@@ -5309,7 +5309,7 @@ static void __maybe_unused update_runtime_enabled(struct rq *rq) {struct task_group *tg; -lockdep_assert_held (&rq->lock); +lockdep_assert_held (rq_lockp (rq));rcu_read_lock ();list_for_each_entry_rcu (tg, &task_groups, list) {
@@ -5328,7 +5328,7 @@ static void __maybe_unused unthrottle_offline_cfs_rqs(struct rq *rq) {struct task_group *tg; -lockdep_assert_held (&rq->lock); +lockdep_assert_held (rq_lockp (rq));rcu_read_lock ();list_for_each_entry_rcu (tg, &task_groups, list) {
@@ -6813,7 +6813,7 @@ static void migrate_task_rq_fair(struct task_struct *p, int new_cpu)* In case of TASK_ON_RQ_MIGRATING we in fact hold the 'old' * rq->lock and can modify state directly. */ -lockdep_assert_held (&task_rq (p)->lock); +lockdep_assert_held (rq_lockp (task_rq (p)));detach_entity_cfs_rq (&p->se); }else {
@@ -7441,7 +7441,7 @@ static int task_hot(struct task_struct *p, struct lb_env *env) {s64 delta; -lockdep_assert_held (&env->src_rq->lock); +lockdep_assert_held (rq_lockp (env->src_rq));if (p->sched_class != &fair_sched_class)return 0 ;
@@ -7539,7 +7539,7 @@ int can_migrate_task(struct task_struct *p, struct lb_env *env) {int tsk_cache_hot; -lockdep_assert_held (&env->src_rq->lock); +lockdep_assert_held (rq_lockp (env->src_rq));/* * We do not migrate tasks that are:
@@ -7617,7 +7617,7 @@ int can_migrate_task(struct task_struct *p, struct lb_env *env)*/ static void detach_task (struct task_struct *p,struct lb_env *env) { -lockdep_assert_held (&env->src_rq->lock); +lockdep_assert_held (rq_lockp (env->src_rq));deactivate_task (env->src_rq, p,DEQUEUE_NOCLOCK );set_task_cpu (p, env->dst_cpu);
@@ -7633,7 +7633,7 @@ static struct task_struct *detach_one_task(struct lb_env *env) {struct task_struct *p; -lockdep_assert_held (&env->src_rq->lock); +lockdep_assert_held (rq_lockp (env->src_rq));list_for_each_entry_reverse (p, &env->src_rq->cfs_tasks, se.group_node) {
@@ -7669,7 +7669,7 @@ static int detach_tasks(struct lb_env *env)struct task_struct *p;int detached =0 ; -lockdep_assert_held (&env->src_rq->lock); +lockdep_assert_held (rq_lockp (env->src_rq));if (env->imbalance <=0 )return 0 ;
@@ -7791,7 +7791,7 @@ static int detach_tasks(struct lb_env *env)*/ static void attach_task (struct rq *rq,struct task_struct *p) { -lockdep_assert_held (&rq->lock); +lockdep_assert_held (rq_lockp (rq));BUG_ON (task_rq (p) != rq);activate_task (rq, p,ENQUEUE_NOCLOCK );
@@ -9726,7 +9726,7 @@ static int load_balance(int this_cpu, struct rq *this_rq,if (need_active_balance (&env)) {unsigned long flags; -raw_spin_lock_irqsave (&busiest->lock, flags); +raw_spin_lock_irqsave (rq_lockp (busiest), flags);/* * Don't kick the active_load_balance_cpu_stop,
@@ -9734,7 +9734,7 @@ static int load_balance(int this_cpu, struct rq *this_rq,* moved to this_cpu: */ if (!cpumask_test_cpu (this_cpu, busiest->curr->cpus_ptr)) { -raw_spin_unlock_irqrestore (&busiest->lock, +raw_spin_unlock_irqrestore (rq_lockp (busiest), flags); env.flags |=LBF_ALL_PINNED ; goto out_one_pinned;
@@ -9750,7 +9750,7 @@ static int load_balance(int this_cpu, struct rq *this_rq, busiest->push_cpu = this_cpu; active_balance =1 ; } -raw_spin_unlock_irqrestore (&busiest->lock, flags); +raw_spin_unlock_irqrestore (rq_lockp (busiest), flags);if (active_balance) { stop_one_cpu_nowait(cpu_of(busiest),
@@ -10506,7 +10506,7 @@ static void nohz_newidle_balance(struct rq *this_rq) time_before(jiffies, READ_ONCE(nohz.next_blocked))) return; -raw_spin_unlock (&this_rq->lock); +raw_spin_unlock (rq_lockp (this_rq));/* * This CPU is going to be idle and blocked load of idle CPUs * need to be updated. Run the ilb locally as it is a good
@@ -10515,7 +10515,7 @@ static void nohz_newidle_balance(struct rq *this_rq)*/ if (!_nohz_idle_balance (this_rq,NOHZ_STATS_KICK ,CPU_NEWLY_IDLE ))kick_ilb (NOHZ_STATS_KICK ); - raw_spin_lock(&this_rq->lock); + raw_spin_lock(rq_lockp(this_rq)); }#else /* !CONFIG_NO_HZ_COMMON */
@@ -10581,7 +10581,7 @@ static int newidle_balance(struct rq *this_rq, struct rq_flags *rf) goto out; } -raw_spin_unlock (&this_rq->lock); +raw_spin_unlock (rq_lockp (this_rq));update_blocked_averages (this_cpu);rcu_read_lock ();
@@ -10619,7 +10619,7 @@ static int newidle_balance(struct rq *this_rq, struct rq_flags *rf) }rcu_read_unlock (); -raw_spin_lock (&this_rq->lock); +raw_spin_lock (rq_lockp (this_rq));if (curr_cost > this_rq->max_idle_balance_cost) this_rq->max_idle_balance_cost = curr_cost;
@@ -11095,9 +11095,9 @@ void unregister_fair_sched_group(struct task_group *tg) rq =cpu_rq (cpu); -raw_spin_lock_irqsave (&rq->lock, flags); +raw_spin_lock_irqsave (rq_lockp (rq), flags);list_del_leaf_cfs_rq (tg->cfs_rq[cpu]); -raw_spin_unlock_irqrestore (&rq->lock, flags); +raw_spin_unlock_irqrestore (rq_lockp (rq), flags); } }
kernel/sched/idle.c +2 -2
@@ -422,10 +422,10 @@ struct task_struct *pick_next_task_idle(struct rq *rq)static void dequeue_task_idle (struct rq *rq,struct task_struct *p,int flags) { -raw_spin_unlock_irq (&rq->lock); +raw_spin_unlock_irq (rq_lockp (rq));printk (KERN_ERR "bad: scheduling from the idle thread!\n" );dump_stack (); -raw_spin_lock_irq (&rq->lock); +raw_spin_lock_irq (rq_lockp (rq)); }/*
kernel/sched/pelt.h +1 -1
@@ -141,7 +141,7 @@ static inline void update_idle_rq_clock_pelt(struct rq *rq)static inline u64 rq_clock_pelt (struct rq *rq) { -lockdep_assert_held (&rq->lock); +lockdep_assert_held (rq_lockp (rq));assert_clock_updated (rq);return rq->clock_pelt - rq->lost_idle_time;
kernel/sched/rt.c +8 -8
@@ -888,7 +888,7 @@ static int do_sched_rt_period_timer(struct rt_bandwidth *rt_b, int overrun)if (skip)continue ; -raw_spin_lock (&rq->lock); +raw_spin_lock (rq_lockp (rq));update_rq_clock (rq);if (rt_rq->rt_time) {
@@ -926,7 +926,7 @@ static int do_sched_rt_period_timer(struct rt_bandwidth *rt_b, int overrun)if (enqueue)sched_rt_rq_enqueue (rt_rq); -raw_spin_unlock (&rq->lock); +raw_spin_unlock (rq_lockp (rq)); }if (!throttled && (!rt_bandwidth_enabled () || rt_b->rt_runtime ==RUNTIME_INF ))
@@ -1894,10 +1894,10 @@ static int push_rt_task(struct rq *rq, bool pull)*/ push_task =get_push_task (rq);if (push_task) { -raw_spin_unlock (&rq->lock); +raw_spin_unlock (rq_lockp (rq));stop_one_cpu_nowait (rq->cpu, push_cpu_stop, push_task, &rq->push_work); -raw_spin_lock (&rq->lock); +raw_spin_lock (rq_lockp (rq)); }return 0 ;
@@ -2122,10 +2122,10 @@ void rto_push_irq_work_func(struct irq_work *work)* When it gets updated, a check is made if a push is possible. */ if (has_pushable_tasks (rq)) { -raw_spin_lock (&rq->lock); +raw_spin_lock (rq_lockp (rq));while (push_rt_task (rq, true)) ; -raw_spin_unlock (&rq->lock); +raw_spin_unlock (rq_lockp (rq)); }raw_spin_lock (&rd->rto_lock);
@@ -2243,10 +2243,10 @@ static void pull_rt_task(struct rq *this_rq)double_unlock_balance (this_rq, src_rq);if (push_task) { -raw_spin_unlock (&this_rq->lock); +raw_spin_unlock (rq_lockp (this_rq));stop_one_cpu_nowait (src_rq->cpu, push_cpu_stop, push_task, &src_rq->push_work); -raw_spin_lock (&this_rq->lock); +raw_spin_lock (rq_lockp (this_rq)); } }
kernel/sched/sched.h +56 -52
278 lines not on this page · raw message
kernel/sched/topology.c +2 -2
@@ -454,7 +454,7 @@ void rq_attach_root(struct rq *rq, struct root_domain *rd)struct root_domain *old_rd =NULL ;unsigned long flags; -raw_spin_lock_irqsave (&rq->lock, flags); +raw_spin_lock_irqsave (rq_lockp (rq), flags);if (rq->rd) { old_rd = rq->rd;
@@ -480,7 +480,7 @@ void rq_attach_root(struct rq *rq, struct root_domain *rd)if (cpumask_test_cpu (rq->cpu, cpu_active_mask))set_rq_online (rq); -raw_spin_unlock_irqrestore (&rq->lock, flags); +raw_spin_unlock_irqrestore (rq_lockp (rq), flags);if (old_rd)call_rcu (&old_rd->rcu, free_rootdomain);
Signature · 1 line
2.29.2.299.gdc1121823c-goog
Re: [PATCH -tip 01/32] sched: Wrap rq::lock access
Quoted 5 lines · Joel Fernandes (Google)
From: Peter Zijlstra <[email protected]>
In preparation of playing games with
rq->lock, abstract the thing using an accessor.
Could you clarify games? I presume the intention is to redefine the scope of the lock based on whether core sched is enabled or not? I presume patch 4/32 has the details.
Balbir Singh
Re: [PATCH -tip 01/32] sched: Wrap rq::lock access
Quoted 10 lines · Balbir
Quoted 5 lines · Joel Fernandes (Google)
From: Peter Zijlstra <[email protected]>
In preparation of playing games with
rq->lock, abstract the thing using an accessor.Could you clarify games? I presume the intention is to redefine the scope of the lock based on whether core sched is enabled or not? I presume patch 4/32 has the details.
Your line wrapping broke, I fixed it.
That is in fact the game. By wrapping it, the nature of the locking is dynamic based on whether core sched is enabled or not (both statically and dynamically).
thanks,
- Joel
Quoted 1 line
Balbir Singh
Re: [PATCH -tip 01/32] sched: Wrap rq::lock access
Quoted 8 lines · Joel Fernandes
Quoted 7 lines · Balbir
Quoted 5 lines · Joel Fernandes (Google)
From: Peter Zijlstra <[email protected]>
In preparation of playing games with
rq->lock, abstract the thing using an accessor.
Quoted 6 lines
Quoted 3 lines
Could you clarify games? I presume the intention is to redefine the scope of the lock based on whether core sched is enabled or not? I presume patch 4/32 has the details.
Your line wrapping broke, I fixed it.
Sorry, I've been using thunderbird from time to time and even though I set the options specified in the Documentation (email-clients), it's not working as expected.
Quoted 4 lines
That is in fact the game. By wrapping it, the nature of the locking is dynamic based on whether core sched is enabled or not (both statically and dynamically).
My point was that the word game does not do justice to the change, some details around how this abstractions helps based on the (re)definition of rq with coresched might help.
Balbir Singh.
PATCH 2/32 sched: Introduce sched_class::pick_task()
In reply to Joel Fernandes (Google)From: Peter Zijlstra <[email protected]>
Because sched_class::pick_next_task() also implies sched_class::set_next_task() (and possibly put_prev_task() and newidle_balance) it is not state invariant. This makes it unsuitable for remote task selection.
--- kernel/sched/deadline.c | 16 ++++++++++++++-- kernel/sched/fair.c | 32 +++++++++++++++++++++++++++++++- kernel/sched/idle.c | 8 ++++++++ kernel/sched/rt.c | 15 +++++++++++++-- kernel/sched/sched.h | 3 +++ kernel/sched/stop_task.c | 14 ++++++++++++-- 6 files changed, 81 insertions(+), 7 deletions(-)
kernel/sched/deadline.c +14 -2
@@ -1867,7 +1867,7 @@ static struct sched_dl_entity *pick_next_dl_entity(struct rq *rq, return rb_entry(left, struct sched_dl_entity, rb_node); } -static struct task_struct *pick_next_task_dl (struct rq *rq) +static struct task_struct *pick_task_dl (struct rq *rq) {struct sched_dl_entity *dl_se;struct dl_rq *dl_rq = &rq->dl;
@@ -1879,7 +1879,18 @@ static struct task_struct *pick_next_task_dl(struct rq *rq) dl_se =pick_next_dl_entity (rq, dl_rq);BUG_ON (!dl_se); p =dl_task_of (dl_se); -set_next_task_dl (rq, p, true); + +return p; +} + +static struct task_struct *pick_next_task_dl (struct rq *rq) +{ +struct task_struct *p; + + p =pick_task_dl (rq); +if (p) +set_next_task_dl (rq, p, true); +return p; }
@@ -2551,6 +2562,7 @@ DEFINE_SCHED_CLASS(dl) = {#ifdef CONFIG_SMP .balance = balance_dl, + .pick_task = pick_task_dl, .select_task_rq = select_task_rq_dl, .migrate_task_rq = migrate_task_rq_dl, .set_cpus_allowed = set_cpus_allowed_dl,
kernel/sched/fair.c +31 -1
@@ -4459,7 +4459,7 @@ pick_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *curr)* Avoid running the skip buddy, if running something else can * be done without getting too unfair. */ -if (cfs_rq->skip == se) { +if (cfs_rq->skip && cfs_rq->skip == se) {struct sched_entity *second;if (se == curr) {
@@ -7017,6 +7017,35 @@ static void check_preempt_wakeup(struct rq *rq, struct task_struct *p, int wake_set_last_buddy (se); } +#ifdef CONFIG_SMP +static struct task_struct *pick_task_fair (struct rq *rq) +{ +struct cfs_rq *cfs_rq = &rq->cfs; +struct sched_entity *se; + +if (!cfs_rq->nr_running) +return NULL ; + +do { +struct sched_entity *curr = cfs_rq->curr; + + se =pick_next_entity (cfs_rq,NULL ); + +if (curr) { +if (se && curr->on_rq) +update_curr (cfs_rq); + +if (!se ||entity_before (curr, se)) + se = curr; + } + + cfs_rq =group_cfs_rq (se); + }while (cfs_rq); + +return task_of (se); +} +#endif +struct task_struct *pick_next_task_fair (struct rq *rq,struct task_struct *prev,struct rq_flags *rf) {
@@ -11219,6 +11248,7 @@ DEFINE_SCHED_CLASS(fair) = {#ifdef CONFIG_SMP .balance = balance_fair, + .pick_task = pick_task_fair, .select_task_rq = select_task_rq_fair, .migrate_task_rq = migrate_task_rq_fair,
kernel/sched/idle.c +8 -0
@@ -406,6 +406,13 @@ static void set_next_task_idle(struct rq *rq, struct task_struct *next, bool firschedstat_inc (rq->sched_goidle); } +#ifdef CONFIG_SMP +static struct task_struct *pick_task_idle (struct rq *rq) +{ +return rq->idle; +} +#endif +struct task_struct *pick_next_task_idle (struct rq *rq) {struct task_struct *next = rq->idle;
@@ -473,6 +480,7 @@ DEFINE_SCHED_CLASS(idle) = {#ifdef CONFIG_SMP .balance = balance_idle, + .pick_task = pick_task_idle, .select_task_rq = select_task_rq_idle, .set_cpus_allowed = set_cpus_allowed_common,#endif
kernel/sched/rt.c +13 -2
@@ -1626,7 +1626,7 @@ static struct task_struct *_pick_next_task_rt(struct rq *rq)return rt_task_of (rt_se); } -static struct task_struct *pick_next_task_rt (struct rq *rq) +static struct task_struct *pick_task_rt (struct rq *rq) {struct task_struct *p;
@@ -1634,7 +1634,17 @@ static struct task_struct *pick_next_task_rt(struct rq *rq)return NULL ; p =_pick_next_task_rt (rq); -set_next_task_rt (rq, p, true); + +return p; +} + +static struct task_struct *pick_next_task_rt (struct rq *rq) +{ +struct task_struct *p =pick_task_rt (rq); + +if (p) +set_next_task_rt (rq, p, true); +return p; }
@@ -2483,6 +2493,7 @@ DEFINE_SCHED_CLASS(rt) = {#ifdef CONFIG_SMP .balance = balance_rt, + .pick_task = pick_task_rt, .select_task_rq = select_task_rq_rt, .set_cpus_allowed = set_cpus_allowed_common, .rq_online = rq_online_rt,
kernel/sched/sched.h +3 -0
@@ -1839,6 +1839,9 @@ struct sched_class {#ifdef CONFIG_SMP int (*balance)(struct rq *rq,struct task_struct *prev,struct rq_flags *rf);int (*select_task_rq)(struct task_struct *p,int task_cpu,int flags); + +struct task_struct * (*pick_task)(struct rq *rq); +void (*migrate_task_rq)(struct task_struct *p,int new_cpu);void (*task_woken)(struct rq *this_rq,struct task_struct *task);
kernel/sched/stop_task.c +12 -2
@@ -34,15 +34,24 @@ static void set_next_task_stop(struct rq *rq, struct task_struct *stop, bool fir stop->se.exec_start =rq_clock_task (rq); } -static struct task_struct *pick_next_task_stop (struct rq *rq) +static struct task_struct *pick_task_stop (struct rq *rq) {if (!sched_stop_runnable (rq))return NULL ; -set_next_task_stop (rq, rq->stop, true);return rq->stop; } +static struct task_struct *pick_next_task_stop (struct rq *rq) +{ +struct task_struct *p =pick_task_stop (rq); + +if (p) +set_next_task_stop (rq, p, true); + +return p; +} +static void enqueue_task_stop (struct rq *rq,struct task_struct *p,int flags) {
@@ -123,6 +132,7 @@ DEFINE_SCHED_CLASS(stop) = {#ifdef CONFIG_SMP .balance = balance_stop, + .pick_task = pick_task_stop, .select_task_rq = select_task_rq_stop, .set_cpus_allowed = set_cpus_allowed_common,#endif
Signature · 1 line
2.29.2.299.gdc1121823c-goog
Re: [PATCH -tip 02/32] sched: Introduce sched_class::pick_task()
Quoted 7 lines · Joel Fernandes (Google)
From: Peter Zijlstra <[email protected]>
Because
sched_class::pick_next_task()also impliessched_class::set_next_task()(and possiblyput_prev_task()andnewidle_balance) it is not state invariant. This makes it unsuitable for remote task selection.
The change makes sense, a small suggestion below
Quoted 83 lines
Tested-by Julien Desfossez <[email protected]>Signed-off-by Peter Zijlstra (Intel) <[email protected]>Signed-off-by Vineeth Remanan Pillai <[email protected]>Signed-off-by Julien Desfossez <[email protected]>Signed-off-by Joel Fernandes (Google) <[email protected]>--- kernel/sched/deadline.c | 16 ++++++++++++++-- kernel/sched/fair.c | 32 +++++++++++++++++++++++++++++++- kernel/sched/idle.c | 8 ++++++++ kernel/sched/rt.c | 15 +++++++++++++-- kernel/sched/sched.h | 3 +++ kernel/sched/stop_task.c | 14 ++++++++++++-- 6 files changed, 81 insertions(+), 7 deletions(-)kernel/sched/deadline.c +14
-2@@ -1867,7 +1867,7 @@ static struct sched_dl_entity *pick_next_dl_entity(struct rq *rq, return rb_entry(left, struct sched_dl_entity, rb_node); } -static struct task_struct *pick_next_task_dl (struct rq *rq) +static struct task_struct *pick_task_dl (struct rq *rq) {struct sched_dl_entity *dl_se;struct dl_rq *dl_rq = &rq->dl;@@ -1879,7 +1879,18 @@ static struct task_struct *pick_next_task_dl(struct rq *rq) dl_se =pick_next_dl_entity (rq, dl_rq);BUG_ON (!dl_se); p =dl_task_of (dl_se); -set_next_task_dl (rq, p, true); + +return p; +} + +static struct task_struct *pick_next_task_dl (struct rq *rq) +{ +struct task_struct *p; + + p =pick_task_dl (rq); +if (p) +set_next_task_dl (rq, p, true); +return p; }@@ -2551,6 +2562,7 @@ DEFINE_SCHED_CLASS(dl) = {#ifdef CONFIG_SMP .balance = balance_dl, + .pick_task = pick_task_dl, .select_task_rq = select_task_rq_dl, .migrate_task_rq = migrate_task_rq_dl, .set_cpus_allowed = set_cpus_allowed_dl,kernel/sched/fair.c +12
-1@@ -4459,7 +4459,7 @@ pick_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *curr)* Avoid running the skip buddy, if running something else can * be done without getting too unfair. */ -if (cfs_rq->skip == se) { +if (cfs_rq->skip && cfs_rq->skip == se) {struct sched_entity *second;if (se == curr) {@@ -7017,6 +7017,35 @@ static void check_preempt_wakeup(struct rq *rq, struct task_struct *p, int wake_set_last_buddy (se); } +#ifdef CONFIG_SMP +static struct task_struct *pick_task_fair (struct rq *rq) +{ +struct cfs_rq *cfs_rq = &rq->cfs; +struct sched_entity *se; + +if (!cfs_rq->nr_running) +return NULL ; + + do { + struct sched_entity *curr = cfs_rq->curr;
kernel/sched/fair.c static void check_preempt_wakeup(struct rq *rq, struct task_struct *p, int wake_ · line 7040
Quoted 10 lines
kernel/sched/fair.c +10
-0+ + se =pick_next_entity (cfs_rq,NULL ); + +if (curr) { +if (se && curr->on_rq) +update_curr (cfs_rq); + +if (!se ||entity_before (curr, se)) + se = curr; + }
Do we want to optimize this a bit
if (curr) {
if (!se || entity_before(curr, se))
se = curr;
if ((se != curr) && curr->on_rq)
update_curr(cfs_rq);}
Balbir
Re: [PATCH -tip 02/32] sched: Introduce sched_class::pick_task()
On Fri, Nov 20, 2020 at 10:56:09AM +1100, Singh, Balbir wrote:
[..]
Quoted 24 lines
Quoted 11 lines
kernel/sched/fair.c +11
-0+#ifdef CONFIG_SMP +static struct task_struct *pick_task_fair (struct rq *rq) +{ +struct cfs_rq *cfs_rq = &rq->cfs; +struct sched_entity *se; + +if (!cfs_rq->nr_running) +return NULL ; + + do { + struct sched_entity *curr = cfs_rq->curr;kernel/sched/fair.c static void check_preempt_wakeup(struct rq *rq, struct task_struct *p, int wake_ · line 7040
Quoted 10 lines
kernel/sched/fair.c +10
-0+ + se =pick_next_entity (cfs_rq,NULL ); + +if (curr) { +if (se && curr->on_rq) +update_curr (cfs_rq); + +if (!se ||entity_before (curr, se)) + se = curr; + }Do we want to optimize this a bit
Quoted 8 lines
if (curr) { if (!se || entity_before(curr, se)) se = curr; if ((se != curr) && curr->on_rq) update_curr(cfs_rq);}
Do you see a difference in codegen? What's the optimization?
Also in later patches this code changes, so it should not matter:
See: 3e0838fa3c51 ("sched/fair: Fix pick_task_fair crashes due to empty rbtree")
thanks,
- Joel
Re: [PATCH -tip 02/32] sched: Introduce sched_class::pick_task()
Quoted 33 lines · Joel Fernandes
On Fri, Nov 20, 2020 at 10:56:09AM +1100, Singh, Balbir wrote:
[..]Quoted 31 lines
Quoted 11 lines
kernel/sched/fair.c +11
-0+#ifdef CONFIG_SMP +static struct task_struct *pick_task_fair (struct rq *rq) +{ +struct cfs_rq *cfs_rq = &rq->cfs; +struct sched_entity *se; + +if (!cfs_rq->nr_running) +return NULL ; + + do { + struct sched_entity *curr = cfs_rq->curr;kernel/sched/fair.c static void check_preempt_wakeup(struct rq *rq, struct task_struct *p, int wake_ · line 7040
Quoted 10 lines
kernel/sched/fair.c +10
-0+ + se =pick_next_entity (cfs_rq,NULL ); + +if (curr) { +if (se && curr->on_rq) +update_curr (cfs_rq); + +if (!se ||entity_before (curr, se)) + se = curr; + }Do we want to optimize this a bit
if (curr) { if (!se || entity_before(curr, se)) se = curr; if ((se != curr) && curr->on_rq) update_curr(cfs_rq);
Quoted 7 lines
Quoted 1 line
}
Do you see a difference in codegen? What's the optimization?
Also in later patches this code changes, so it should not matter:
See:3e0838fa3c51("sched/fair: Fixpick_task_faircrashes due to empty rbtree")
I did see the next patch, but the idea is that we don't need to update_curr() if the picked sched entity is the same as current (se == curr). What are the side-effects of not updating curr when se == curr?
Balbir
Vincent Guittot
maintainerRe: [PATCH -tip 02/32] sched: Introduce sched_class::pick_task()
In reply to Joel Fernandes (Google)Quoted 83 lines · Joel Fernandes (Google)
From: Peter Zijlstra <[email protected]>
Because
sched_class::pick_next_task()also impliessched_class::set_next_task()(and possiblyput_prev_task()andnewidle_balance) it is not state invariant. This makes it unsuitable for remote task selection.Tested-by Julien Desfossez <[email protected]>Signed-off-by Peter Zijlstra (Intel) <[email protected]>Signed-off-by Vineeth Remanan Pillai <[email protected]>Signed-off-by Julien Desfossez <[email protected]>Signed-off-by Joel Fernandes (Google) <[email protected]>--- kernel/sched/deadline.c | 16 ++++++++++++++-- kernel/sched/fair.c | 32 +++++++++++++++++++++++++++++++- kernel/sched/idle.c | 8 ++++++++ kernel/sched/rt.c | 15 +++++++++++++-- kernel/sched/sched.h | 3 +++ kernel/sched/stop_task.c | 14 ++++++++++++-- 6 files changed, 81 insertions(+), 7 deletions(-)kernel/sched/deadline.c +14
-2@@ -1867,7 +1867,7 @@ static struct sched_dl_entity *pick_next_dl_entity(struct rq *rq, return rb_entry(left, struct sched_dl_entity, rb_node); } -static struct task_struct *pick_next_task_dl (struct rq *rq) +static struct task_struct *pick_task_dl (struct rq *rq) {struct sched_dl_entity *dl_se;struct dl_rq *dl_rq = &rq->dl;@@ -1879,7 +1879,18 @@ static struct task_struct *pick_next_task_dl(struct rq *rq) dl_se =pick_next_dl_entity (rq, dl_rq);BUG_ON (!dl_se); p =dl_task_of (dl_se); -set_next_task_dl (rq, p, true); + +return p; +} + +static struct task_struct *pick_next_task_dl (struct rq *rq) +{ +struct task_struct *p; + + p =pick_task_dl (rq); +if (p) +set_next_task_dl (rq, p, true); +return p; }@@ -2551,6 +2562,7 @@ DEFINE_SCHED_CLASS(dl) = {#ifdef CONFIG_SMP .balance = balance_dl, + .pick_task = pick_task_dl, .select_task_rq = select_task_rq_dl, .migrate_task_rq = migrate_task_rq_dl, .set_cpus_allowed = set_cpus_allowed_dl,kernel/sched/fair.c +4
-1@@ -4459,7 +4459,7 @@ pick_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *curr)* Avoid running the skip buddy, if running something else can * be done without getting too unfair. */ -if (cfs_rq->skip == se) { +if (cfs_rq->skip && cfs_rq->skip == se) {struct sched_entity *second;if (se == curr) {@@ -7017,6 +7017,35 @@ static void check_preempt_wakeup(struct rq *rq, struct task_struct *p, int wake_set_last_buddy (se); } +#ifdef CONFIG_SMP +static struct task_struct *pick_task_fair (struct rq *rq) +{
kernel/sched/fair.c static void check_preempt_wakeup(struct rq *rq, struct task_struct *p, int wake_ · line 7032
Quoted 10 lines
kernel/sched/fair.c +10
-0+struct cfs_rq *cfs_rq = &rq->cfs; +struct sched_entity *se; + +if (!cfs_rq->nr_running) +return NULL ; + + do { + struct sched_entity *curr = cfs_rq->curr; + + se = pick_next_entity(cfs_rq, NULL);
Calling pick_next_entity clears buddies. This is fine without coresched because the se will be the next one. But calling pick_task_fair doesn't mean that the se will be used
kernel/sched/fair.c static void check_preempt_wakeup(struct rq *rq, struct task_struct *p, int wake_ · line 7037
Quoted 5 lines
kernel/sched/fair.c +5
-0+ +if (curr) { +if (se && curr->on_rq) +update_curr (cfs_rq); +
Shouldn't you check if cfs_rq is throttled ?
Quoted 144 lines
kernel/sched/fair.c +12
-0+if (!se ||entity_before (curr, se)) + se = curr; + } + + cfs_rq =group_cfs_rq (se); + }while (cfs_rq); + +return task_of (se); +} +#endif +struct task_struct *pick_next_task_fair (struct rq *rq,struct task_struct *prev,struct rq_flags *rf) {@@ -11219,6 +11248,7 @@ DEFINE_SCHED_CLASS(fair) = {#ifdef CONFIG_SMP .balance = balance_fair, + .pick_task = pick_task_fair, .select_task_rq = select_task_rq_fair, .migrate_task_rq = migrate_task_rq_fair,kernel/sched/idle.c +8
-0@@ -406,6 +406,13 @@ static void set_next_task_idle(struct rq *rq, struct task_struct *next, bool firschedstat_inc (rq->sched_goidle); } +#ifdef CONFIG_SMP +static struct task_struct *pick_task_idle (struct rq *rq) +{ +return rq->idle; +} +#endif +struct task_struct *pick_next_task_idle (struct rq *rq) {struct task_struct *next = rq->idle;@@ -473,6 +480,7 @@ DEFINE_SCHED_CLASS(idle) = {#ifdef CONFIG_SMP .balance = balance_idle, + .pick_task = pick_task_idle, .select_task_rq = select_task_rq_idle, .set_cpus_allowed = set_cpus_allowed_common,#endif kernel/sched/rt.c +13
-2@@ -1626,7 +1626,7 @@ static struct task_struct *_pick_next_task_rt(struct rq *rq)return rt_task_of (rt_se); } -static struct task_struct *pick_next_task_rt (struct rq *rq) +static struct task_struct *pick_task_rt (struct rq *rq) {struct task_struct *p;@@ -1634,7 +1634,17 @@ static struct task_struct *pick_next_task_rt(struct rq *rq)return NULL ; p =_pick_next_task_rt (rq); -set_next_task_rt (rq, p, true); + +return p; +} + +static struct task_struct *pick_next_task_rt (struct rq *rq) +{ +struct task_struct *p =pick_task_rt (rq); + +if (p) +set_next_task_rt (rq, p, true); +return p; }@@ -2483,6 +2493,7 @@ DEFINE_SCHED_CLASS(rt) = {#ifdef CONFIG_SMP .balance = balance_rt, + .pick_task = pick_task_rt, .select_task_rq = select_task_rq_rt, .set_cpus_allowed = set_cpus_allowed_common, .rq_online = rq_online_rt,kernel/sched/sched.h +3
-0@@ -1839,6 +1839,9 @@ struct sched_class {#ifdef CONFIG_SMP int (*balance)(struct rq *rq,struct task_struct *prev,struct rq_flags *rf);int (*select_task_rq)(struct task_struct *p,int task_cpu,int flags); + +struct task_struct * (*pick_task)(struct rq *rq); +void (*migrate_task_rq)(struct task_struct *p,int new_cpu);void (*task_woken)(struct rq *this_rq,struct task_struct *task);kernel/sched/stop_task.c +12
-3@@ -34,15 +34,24 @@ static void set_next_task_stop(struct rq *rq, struct task_struct *stop, bool fir stop->se.exec_start =rq_clock_task (rq); } -static struct task_struct *pick_next_task_stop (struct rq *rq) +static struct task_struct *pick_task_stop (struct rq *rq) {if (!sched_stop_runnable (rq))return NULL ; -set_next_task_stop (rq, rq->stop, true);return rq->stop; } +static struct task_struct *pick_next_task_stop (struct rq *rq) +{ +struct task_struct *p =pick_task_stop (rq); + +if (p) +set_next_task_stop (rq, p, true); + +return p; +} +static void enqueue_task_stop (struct rq *rq,struct task_struct *p,int flags) {@@ -123,6 +132,7 @@ DEFINE_SCHED_CLASS(stop) = {#ifdef CONFIG_SMP .balance = balance_stop, + .pick_task = pick_task_stop, .select_task_rq = select_task_rq_stop, .set_cpus_allowed = set_cpus_allowed_common,#endif --2.29.2.299.gdc1121823c-goog
Peter Zijlstra
maintainerRe: [PATCH -tip 02/32] sched: Introduce sched_class::pick_task()
Quoted 11 lines · Vincent Guittot
On Wed, 18 Nov 2020 at 00:20, Joel Fernandes (Google)
Quoted 9 lines
(quoted diff) +9
-0+#ifdef CONFIG_SMP +static struct task_struct *pick_task_fair(struct rq *rq) +{ + struct cfs_rq *cfs_rq = &rq->cfs; + struct sched_entity *se; + + if (!cfs_rq->nr_running) + return NULL; +
Quoted 8 lines
Review
Quoted 4 lines
(quoted diff) +4
-0+ do { + struct sched_entity *curr = cfs_rq->curr; + + se = pick_next_entity(cfs_rq, NULL);Calling
pick_next_entityclears buddies. This is fine without coresched because the se will be the next one. But callingpick_task_fairdoesn't mean that the se will be used
Urgh, nice one :/
Quoted 7 lines
Review
Quoted 5 lines
(quoted diff) +5
-0+ + if (curr) { + if (se && curr->on_rq) + update_curr(cfs_rq); +Shouldn't you check if
cfs_rqis throttled ?
Hmm,... I suppose we do.
Quoted 10 lines
Review
Quoted 10 lines
(quoted diff) +10
-0+ if (!se || entity_before(curr, se)) + se = curr; + } + + cfs_rq = group_cfs_rq(se); + } while (cfs_rq); + + return task_of(se); +} +#endif
Something like so then?
kernel/sched/fair.c +16 -7
@@ -4354,6 +4354,8 @@ check_preempt_tick(struct cfs_rq *cfs_rqstatic void set_next_entity (struct cfs_rq *cfs_rq,struct sched_entity *se) { +clear_buddies (cfs_rq, se); +/* 'current' is not kept within the tree. */ if (se->on_rq) {/*
@@ -4440,8 +4442,6 @@ pick_next_entity(struct cfs_rq *cfs_rq, se = cfs_rq->last; } -clear_buddies (cfs_rq, se); -return se; }
@@ -6982,20 +6982,29 @@ static void check_preempt_wakeup(struct#ifdef CONFIG_SMP static struct task_struct *pick_task_fair (struct rq *rq) { -struct cfs_rq *cfs_rq = &rq->cfs;struct sched_entity *se; - +struct cfs_rq *cfs_rq; + +again: + cfs_rq = &rq->cfs;if (!cfs_rq->nr_running)return NULL ;do {struct sched_entity *curr = cfs_rq->curr; -if (curr && curr->on_rq) -update_curr (cfs_rq); +/* When we pick for a remote RQ, we'll not have done put_prev_entity() */ +if (curr) { +if (curr->on_rq) +update_curr (cfs_rq); +else + curr =NULL ; - se =pick_next_entity (cfs_rq, curr); +if (unlikely (check_cfs_rq_runtime (cfs_rq))) + goto again; + } + se =pick_next_entity (cfs_rq, curr); cfs_rq =group_cfs_rq (se); }while (cfs_rq);
Vincent Guittot
maintainerRe: [PATCH -tip 02/32] sched: Introduce sched_class::pick_task()
Quoted 34 lines · Peter Zijlstra
Quoted 11 lines · Vincent Guittot
On Wed, 18 Nov 2020 at 00:20, Joel Fernandes (Google)
Quoted 9 lines
(quoted diff) +9
-0+#ifdef CONFIG_SMP +static struct task_struct *pick_task_fair(struct rq *rq) +{ + struct cfs_rq *cfs_rq = &rq->cfs; + struct sched_entity *se; + + if (!cfs_rq->nr_running) + return NULL; +Quoted 8 lines
Review
Quoted 4 lines
(quoted diff) +4
-0+ do { + struct sched_entity *curr = cfs_rq->curr; + + se = pick_next_entity(cfs_rq, NULL);Calling
pick_next_entityclears buddies. This is fine without coresched because the se will be the next one. But callingpick_task_fairdoesn't mean that the se will be usedUrgh, nice one :/
Quoted 7 lines
kernel/sched/fair.c static void check_preempt_wakeup(struct
Quoted 5 lines
kernel/sched/fair.c +5
-0+ +if (curr) { +if (se && curr->on_rq) +update_curr (cfs_rq); +Shouldn't you check if
cfs_rqis throttled ?Hmm,... I suppose we do.
Quoted 12 lines
Quoted 10 lines
Review
Quoted 10 lines
(quoted diff) +10
-0+ if (!se || entity_before(curr, se)) + se = curr; + } + + cfs_rq = group_cfs_rq(se); + } while (cfs_rq); + + return task_of(se); +} +#endifSomething like so then?
yes. it seems ok
Quoted 56 lines
kernel/sched/fair.c +16
-7@@ -4354,6 +4354,8 @@ check_preempt_tick(struct cfs_rq *cfs_rqstatic void set_next_entity (struct cfs_rq *cfs_rq,struct sched_entity *se) { +clear_buddies (cfs_rq, se); +/* 'current' is not kept within the tree. */ if (se->on_rq) {/* @@ -4440,8 +4442,6 @@ pick_next_entity(struct cfs_rq *cfs_rq, se = cfs_rq->last; } -clear_buddies (cfs_rq, se); -return se; }@@ -6982,20 +6982,29 @@ static void check_preempt_wakeup(struct#ifdef CONFIG_SMP static struct task_struct *pick_task_fair (struct rq *rq) { -struct cfs_rq *cfs_rq = &rq->cfs;struct sched_entity *se; - +struct cfs_rq *cfs_rq; + +again: + cfs_rq = &rq->cfs;if (!cfs_rq->nr_running)return NULL ;do {struct sched_entity *curr = cfs_rq->curr; -if (curr && curr->on_rq) -update_curr (cfs_rq); +/* When we pick for a remote RQ, we'll not have done put_prev_entity() */ +if (curr) { +if (curr->on_rq) +update_curr (cfs_rq); +else + curr =NULL ; - se =pick_next_entity (cfs_rq, curr); +if (unlikely (check_cfs_rq_runtime (cfs_rq))) + goto again; + } + se =pick_next_entity (cfs_rq, curr); cfs_rq =group_cfs_rq (se); }while (cfs_rq);
Peter Zijlstra
maintainerRe: [PATCH -tip 02/32] sched: Introduce sched_class::pick_task()
Quoted 30 lines · Vincent Guittot
Quoted 1 line
Something like so then?
yes. it seems ok
Quoted 26 lines
kernel/sched/fair.c +10
-0@@ -6982,20 +6982,29 @@ static void check_preempt_wakeup(struct#ifdef CONFIG_SMP static struct task_struct *pick_task_fair (struct rq *rq) {struct sched_entity *se; +struct cfs_rq *cfs_rq; + +again: + cfs_rq = &rq->cfs;if (!cfs_rq->nr_running)return NULL ; do { struct sched_entity *curr = cfs_rq->curr; + /* When we pick for a remote RQ, we'll not have done put_prev_entity() */ + if (curr) { + if (curr->on_rq) + update_curr(cfs_rq); + else + curr = NULL;
Quoted 2 lines
kernel/sched/fair.c static void check_preempt_wakeup(struct · line 7004
Quoted 2 lines
kernel/sched/fair.c +2
-0+if (unlikely (check_cfs_rq_runtime (cfs_rq))) + goto again;
Head-ache though; pick_task() was supposed to be stateless, but now we're modifying a remote runqueue... I suppose it still works, because irrespective of which task we end up picking (even idle), we'll schedule the remote CPU, which would've resulted in the same (and possibly triggered a reschedule if we'd not done it here).
There's a wrinkle through, other than in schedule(), where we dequeue() and keep running with the current task while we release rq->lock, this has preemption enabled as well.
This means that if we do this, the remote CPU could preempt, but the task is then no longer on the runqueue.
I _think_ it all still works, but yuck!
Quoted 6 lines
Quoted 6 lines
kernel/sched/fair.c +2
-0+ } + se =pick_next_entity (cfs_rq, curr); cfs_rq =group_cfs_rq (se); }while (cfs_rq);
PATCH 3/32 sched/fair: Fix pick_task_fair crashes due to empty rbtree
In reply to Joel Fernandes (Google)From: Peter Zijlstra <[email protected]>
pick_next_entity() is passed curr == NULL during core-scheduling. Due to this, if the rbtree is empty, the 'left' variable is set to NULL within the function. This can cause crashes within the function.
This is not an issue if put_prev_task() is invoked on the currently running task before calling pick_next_entity(). However, in core scheduling, it is possible that a sibling CPU picks for another RQ in the core, via pick_task_fair(). This remote sibling would not get any opportunities to do a put_prev_task().
Fix it by refactoring pick_task_fair() such that pick_next_entity() is called with the cfs_rq->curr. This will prevent pick_next_entity() from crashing if its rbtree is empty.
Also this fixes another possible bug where update_curr() would not be called on the cfs_rq hierarchy if the rbtree is empty. This could effect cross-cpu comparison of vruntime.
--- kernel/sched/fair.c | 11 +++-------- 1 file changed, 3 insertions(+), 8 deletions(-)
kernel/sched/fair.c +3 -8
@@ -7029,15 +7029,10 @@ static struct task_struct *pick_task_fair(struct rq *rq)do {struct sched_entity *curr = cfs_rq->curr; - se =pick_next_entity (cfs_rq,NULL ); - -if (curr) { -if (se && curr->on_rq) -update_curr (cfs_rq); +if (curr && curr->on_rq) +update_curr (cfs_rq); -if (!se ||entity_before (curr, se)) - se = curr; - } + se =pick_next_entity (cfs_rq, curr); cfs_rq =group_cfs_rq (se); }while (cfs_rq);
Signature · 1 line
2.29.2.299.gdc1121823c-goog
Re: [PATCH -tip 03/32] sched/fair: Fix pick_task_fair crashes due to empty rbtree
Quoted 12 lines · Joel Fernandes (Google)
From: Peter Zijlstra <[email protected]>
pick_next_entity()is passed curr == NULL during core-scheduling. Due to this, if the rbtree is empty, the 'left' variable is set to NULL within the function. This can cause crashes within the function.This is not an issue if
put_prev_task()is invoked on the currently running task before callingpick_next_entity(). However, in core scheduling, it is possible that a sibling CPU picks for another RQ in the core, viapick_task_fair(). This remote sibling would not get any opportunities to do aput_prev_task().
Quoted 8 lines
Fix it by refactoring
pick_task_fair()such thatpick_next_entity()is called with thecfs_rq->curr. This will preventpick_next_entity()from crashing if its rbtree is empty.Also this fixes another possible bug where
update_curr()would not be called on thecfs_rqhierarchy if the rbtree is empty. This could effect cross-cpu comparison of vruntime.
It is not clear from the changelog as to what does put_prev_task() do to prevent the crash from occuring? Why did we pass NULL as curr in the first place to pick_next_entity?
The patch looks functionally correct as in, it passes curr as the reference to pick_next_entity() for left and entity_before comparisons.
Balbir Singh
Re: [PATCH -tip 03/32] sched/fair: Fix pick_task_fair crashes due to empty rbtree
Hi Balbir,
Quoted 17 lines · Balbir
Quoted 16 lines · Joel Fernandes (Google)
From: Peter Zijlstra <[email protected]>
pick_next_entity()is passed curr == NULL during core-scheduling. Due to this, if the rbtree is empty, the 'left' variable is set to NULL within the function. This can cause crashes within the function.This is not an issue if
put_prev_task()is invoked on the currently running task before callingpick_next_entity(). However, in core scheduling, it is possible that a sibling CPU picks for another RQ in the core, viapick_task_fair(). This remote sibling would not get any opportunities to do aput_prev_task().Fix it by refactoring
pick_task_fair()such thatpick_next_entity()is called with thecfs_rq->curr. This will preventpick_next_entity()from crashing if its rbtree is empty.
Quoted 7 lines
Quoted 4 lines
Also this fixes another possible bug where
update_curr()would not be called on thecfs_rqhierarchy if the rbtree is empty. This could effect cross-cpu comparison of vruntime.It is not clear from the changelog as to what does
put_prev_task()do to prevent the crash from occuring? Why did we pass NULL as curr in the first place topick_next_entity?
A little more context on this crash in v8 is here:
https://lwn.net/ml/linux-kernel/[email protected]/
The issue here arises from the fact that, we try to pick task for a sibling while sibling is running a task. Running tasks are not in the cfs_rq and pick_next_entity can return NULL if there is only one cfs task in the cfs_rq. This would not happen normally because put_prev_task is called before pick_task and put_prev_task adds the task back to cfs_rq. But for coresched, pick_task is called on a remote sibling's cfs_rq without calling put_prev_task and this can lead to pick_next_entity returning NULL.
The initial logic of passing NULL would work fine as long as we call put_prev_task before calling pick_task_fair. But for coresched, we call pick_task_fair on siblings while the task is running and would not be able to call put_prev_task. So this refactor of the code fixes the crash by explicitly passing curr.
Hope this clarifies..
Thanks,
Vineeth
Re: [PATCH -tip 03/32] sched/fair: Fix pick_task_fair crashes due to empty rbtree
Quoted 39 lines · Vineeth Pillai
Hi Balbir,
Quoted 17 lines · Balbir
Quoted 16 lines · Joel Fernandes (Google)
From: Peter Zijlstra <[email protected]>
pick_next_entity()is passed curr == NULL during core-scheduling. Due to this, if the rbtree is empty, the 'left' variable is set to NULL within the function. This can cause crashes within the function.This is not an issue if
put_prev_task()is invoked on the currently running task before callingpick_next_entity(). However, in core scheduling, it is possible that a sibling CPU picks for another RQ in the core, viapick_task_fair(). This remote sibling would not get any opportunities to do aput_prev_task().Fix it by refactoring
pick_task_fair()such thatpick_next_entity()is called with thecfs_rq->curr. This will preventpick_next_entity()from crashing if its rbtree is empty.Quoted 7 lines
Quoted 4 lines
Also this fixes another possible bug where
update_curr()would not be called on thecfs_rqhierarchy if the rbtree is empty. This could effect cross-cpu comparison of vruntime.It is not clear from the changelog as to what does
put_prev_task()do to prevent the crash from occuring? Why did we pass NULL as curr in the first place topick_next_entity?A little more context on this crash in v8 is here:
https://lwn.net/ml/linux-kernel/[email protected]/The issue here arises from the fact that, we try to pick task for a sibling while sibling is running a task. Running tasks are not in the
cfs_rqandpick_next_entitycan return NULL if there is only one cfs task in thecfs_rq. This would not happen normally becauseput_prev_taskis called beforepick_taskandput_prev_taskadds the task back tocfs_rq. But for coresched,pick_taskis called on a remote sibling'scfs_rqwithout callingput_prev_taskand this can lead topick_next_entityreturning NULL.
Quoted 8 lines
The initial logic of passing NULL would work fine as long as we call
put_prev_taskbefore callingpick_task_fair. But for coresched, we callpick_task_fairon siblings while the task is running and would not be able to callput_prev_task. So this refactor of the code fixes the crash by explicitly passing curr.Hope this clarifies..
Yes, it does!
Thanks,
Balbir Singh.
Peter Zijlstra
maintainerRe: [PATCH -tip 03/32] sched/fair: Fix pick_task_fair crashes due to empty rbtree
In reply to Singh, BalbirQuoted 22 lines · Balbir
Quoted 12 lines · Joel Fernandes (Google)
From: Peter Zijlstra <[email protected]>
pick_next_entity()is passed curr == NULL during core-scheduling. Due to this, if the rbtree is empty, the 'left' variable is set to NULL within the function. This can cause crashes within the function.This is not an issue if
put_prev_task()is invoked on the currently running task before callingpick_next_entity(). However, in core scheduling, it is possible that a sibling CPU picks for another RQ in the core, viapick_task_fair(). This remote sibling would not get any opportunities to do aput_prev_task().Quoted 8 lines
Fix it by refactoring
pick_task_fair()such thatpick_next_entity()is called with thecfs_rq->curr. This will preventpick_next_entity()from crashing if its rbtree is empty.Also this fixes another possible bug where
update_curr()would not be called on thecfs_rqhierarchy if the rbtree is empty. This could effect cross-cpu comparison of vruntime.
Quoted 6 lines
It is not clear from the changelog as to what does
put_prev_task()do to prevent the crash from occuring? Why did we pass NULL as curr in the first place topick_next_entity?The patch looks functionally correct as in, it passes curr as the reference to
pick_next_entity()for left andentity_beforecomparisons.
This patch should not exist; it should be smashed into the previous patch. There is no point in preserving a crash.
PATCH 4/32 sched: Core-wide rq->lock
In reply to Joel Fernandes (Google)From: Peter Zijlstra <[email protected]>
Introduce the basic infrastructure to have a core wide rq->lock.
--- kernel/Kconfig.preempt | 5 ++ kernel/sched/core.c | 108 +++++++++++++++++++++++++++++++++++++++++ kernel/sched/sched.h | 31 ++++++++++++ 3 files changed, 144 insertions(+)
kernel/Kconfig.preempt +5 -0
@@ -80,3 +80,8 @@ config PREEMPT_COUNT config PREEMPTION bool select PREEMPT_COUNT + +config SCHED_CORE + bool "Core Scheduling for SMT" + default y + depends on SCHED_SMT
kernel/sched/core.c +108 -0
132 lines not on this page · raw message
kernel/sched/sched.h +31 -0
@@ -1061,6 +1061,12 @@ struct rq {#endif unsigned int push_busy;struct cpu_stop_work push_work; + +#ifdef CONFIG_SCHED_CORE +/* per rq */ +struct rq *core; +unsigned int core_enabled; +#endif };#ifdef CONFIG_FAIR_GROUP_SCHED
@@ -1099,11 +1105,36 @@ static inline bool is_migration_disabled(struct task_struct *p)#endif } +#ifdef CONFIG_SCHED_CORE +DECLARE_STATIC_KEY_FALSE (__sched_core_enabled); + +static inline bool sched_core_enabled (struct rq *rq) +{ +return static_branch_unlikely (&__sched_core_enabled) && rq->core_enabled; +} + +static inline raw_spinlock_t *rq_lockp (struct rq *rq) +{ +if (sched_core_enabled (rq)) +return &rq->core->__lock; + +return &rq->__lock; +} + +#else /* !CONFIG_SCHED_CORE */ + +static inline bool sched_core_enabled (struct rq *rq) +{ +return false; +} +static inline raw_spinlock_t *rq_lockp (struct rq *rq) {return &rq->__lock; } +#endif /* CONFIG_SCHED_CORE */ +#ifdef CONFIG_SCHED_SMT extern void __update_idle_core (struct rq *rq);
Signature · 1 line
2.29.2.299.gdc1121823c-goog
Re: [PATCH -tip 04/32] sched: Core-wide rq->lock
Quoted 4 lines · Joel Fernandes (Google)
From: Peter Zijlstra <[email protected]>
Introduce the basic infrastructure to have a core wide
rq->lock.
Reading through the patch, it seems like all the CPUs have to be running with sched core enabled/disabled? Is it possible to have some cores with core sched disabled? I don't see a strong use case for it, but I am wondering if the design will fall apart if that assumption is broken?
Balbir Singh
Peter Zijlstra
maintainerRe: [PATCH -tip 04/32] sched: Core-wide rq->lock
Quoted 9 lines · Balbir Singh
Quoted 4 lines · Joel Fernandes (Google)
From: Peter Zijlstra <[email protected]>
Introduce the basic infrastructure to have a core wide
rq->lock.Reading through the patch, it seems like all the CPUs have to be running with sched core enabled/disabled? Is it possible to have some cores with core sched disabled?
Yep, patch even says so:
+ * XXX entirely possible to selectively enable cores, don't bother for now.
Quoted 3 lines
I don't see a strong use case for it,
but I am wondering if the design will fall apart if that assumption is broken?
The use-case I have is not using stop-machine. That is, stopping a whole core at a time, instead of the whole sodding machine. It's on the todo list *somewhere*....
Re: [PATCH -tip 04/32] sched: Core-wide rq->lock
Quoted 7 lines · Peter Zijlstra
Quoted 6 lines · Balbir Singh
Quoted 4 lines · Joel Fernandes (Google)
From: Peter Zijlstra <[email protected]>
Introduce the basic infrastructure to have a core wide
rq->lock.
Quoted 7 lines
Quoted 3 lines
Reading through the patch, it seems like all the CPUs have to be running with sched core enabled/disabled? Is it possible to have some cores with core sched disabled?
Yep, patch even says so:
+ * XXX entirely possible to selectively enable cores, don't bother for now.
Yes, it does in the comments, I looked at just the changelog :)
Quoted 10 lines
Quoted 4 lines
I don't see a strong use case for it,
but I am wondering if the design will fall apart if that assumption is broken?The use-case I have is not using stop-machine. That is, stopping a whole core at a time, instead of the whole sodding machine. It's on the todo list *somewhere*....
Good to know, I guess that would need a transition of the entire core to idle and maintenance of a mask that prevents tasks that need core scheduling from getting off of CPUs that enable core scheduling.
Balbir Singh.
PATCH 5/32 sched/fair: Add a few assertions
In reply to Joel Fernandes (Google)From: Peter Zijlstra <[email protected]>
--- kernel/sched/fair.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-)
kernel/sched/fair.c +9 -2
@@ -6245,6 +6245,11 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target) task_util =uclamp_task_util (p); } +/* +* per-cpu select_idle_mask usage +*/ +lockdep_assert_irqs_disabled (); +if ((available_idle_cpu (target) ||sched_idle_cpu (target)) &&asym_fits_capacity (task_util, target))return target;
@@ -6710,8 +6715,6 @@ static int find_energy_efficient_cpu(struct task_struct *p, int prev_cpu)* certain conditions an idle sibling CPU if the domain has SD_WAKE_AFFINE set. * * Returns the target CPU number. -* -* preempt must be disabled. */ static int select_task_rq_fair (struct task_struct *p,int prev_cpu,int wake_flags)
@@ -6724,6 +6727,10 @@ select_task_rq_fair(struct task_struct *p, int prev_cpu, int wake_flags)/* SD_flags and WF_flags share the first nibble */ int sd_flag = wake_flags &0xF ; +/* +* required for stable ->cpus_allowed +*/ +lockdep_assert_held (&p->pi_lock);if (wake_flags &WF_TTWU ) {record_wakee (p);
Signature · 1 line
2.29.2.299.gdc1121823c-goog
PATCH 6/32 sched: Basic tracking of matching tasks
In reply to Joel Fernandes (Google)From: Peter Zijlstra <[email protected]>
Introduce task_struct::core_cookie as an opaque identifier for core scheduling. When enabled; core scheduling will only allow matching task to be on the core; where idle matches everything.
When task_struct::core_cookie is set (and core scheduling is enabled) these tasks are indexed in a second RB-tree, first on cookie value then on scheduling function, such that matching task selection always finds the most elegible match.
NOTE: *shudder* at the overhead...
NOTE: *sigh*, a 3rd copy of the scheduling function; the alternative is per class tracking of cookies and that just duplicates a lot of stuff for no raisin (the 2nd copy lives in the rt-mutex PI code).
--- include/linux/sched.h | 8 ++- kernel/sched/core.c | 146 ++++++++++++++++++++++++++++++++++++++++++ kernel/sched/fair.c | 46 ------------- kernel/sched/sched.h | 55 ++++++++++++++++ 4 files changed, 208 insertions(+), 47 deletions(-)
include/linux/sched.h +7 -1
@@ -683,10 +683,16 @@ struct task_struct {const struct sched_class *sched_class;struct sched_entity se;struct sched_rt_entity rt; +struct sched_dl_entity dl; + +#ifdef CONFIG_SCHED_CORE +struct rb_node core_node; +unsigned long core_cookie; +#endif +#ifdef CONFIG_CGROUP_SCHED struct task_group *sched_task_group;#endif -struct sched_dl_entity dl;#ifdef CONFIG_UCLAMP_TASK /*
kernel/sched/core.c +146 -0
170 lines not on this page · raw message
kernel/sched/fair.c +0 -46
66 lines not on this page · raw message
kernel/sched/sched.h +55 -0
67 lines not on this page · raw message
Signature · 1 line
2.29.2.299.gdc1121823c-goog
PATCH 7/32 sched: Add core wide task selection and scheduling.
In reply to Joel Fernandes (Google)From: Peter Zijlstra <[email protected]>
Instead of only selecting a local task, select a task for all SMT siblings for every reschedule on the core (irrespective which logical CPU does the reschedule).
--- kernel/sched/core.c | 301 ++++++++++++++++++++++++++++++++++++++++++- kernel/sched/sched.h | 6 +- 2 files changed, 305 insertions(+), 2 deletions(-)
kernel/sched/core.c +300 -1
@@ -5029,7 +5029,7 @@ static void put_prev_task_balance(struct rq *rq, struct task_struct *prev,* Pick up the highest-prio task: */ static inline struct task_struct * -pick_next_task (struct rq *rq,struct task_struct *prev,struct rq_flags *rf) +__pick_next_task (struct rq *rq,struct task_struct *prev,struct rq_flags *rf) {const struct sched_class *class;struct task_struct *p;
@@ -5070,6 +5070,294 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf) }#ifdef CONFIG_SCHED_CORE +static inline bool is_task_rq_idle (struct task_struct *t) +{ +return (task_rq (t)->idle == t); +} + +static inline bool cookie_equals (struct task_struct *a,unsigned long cookie) +{ +return is_task_rq_idle (a) || (a->core_cookie == cookie); +} + +static inline bool cookie_match (struct task_struct *a,struct task_struct *b) +{ +if (is_task_rq_idle (a) ||is_task_rq_idle (b)) +return true; + +return a->core_cookie == b->core_cookie; +} + +// XXX fairness/fwd progress conditions +/* +* Returns +* - NULL if there is no runnable task for this class. +* - the highest priority task for this runqueue if it matches +* rq->core->core_cookie or its priority is greater than max. +* - Else returns idle_task. +*/ +static struct task_struct * +pick_task (struct rq *rq,const struct sched_class *class,struct task_struct *max) +{ +struct task_struct *class_pick, *cookie_pick; +unsigned long cookie = rq->core->core_cookie; + + class_pick = class->pick_task (rq); +if (!class_pick) +return NULL ; + +if (!cookie) { +/* +* If class_pick is tagged, return it only if it has +* higher priority than max. +*/ +if (max && class_pick->core_cookie && +prio_less (class_pick, max)) +return idle_sched_class.pick_task (rq); + +return class_pick; + } + +/* +* If class_pick is idle or matches cookie, return early. +*/ +if (cookie_equals (class_pick, cookie)) +return class_pick; + + cookie_pick =sched_core_find (rq, cookie); + +/* +* If class > max && class > cookie, it is the highest priority task on +* the core (so far) and it must be selected, otherwise we must go with +* the cookie pick in order to satisfy the constraint. +*/ +if (prio_less (cookie_pick, class_pick) && + (!max ||prio_less (max, class_pick))) +return class_pick; + +return cookie_pick; +} + +static struct task_struct * +pick_next_task (struct rq *rq,struct task_struct *prev,struct rq_flags *rf) +{ +struct task_struct *next, *max =NULL ; +const struct sched_class *class; +const struct cpumask *smt_mask; +bool need_sync; +int i, j, cpu; + +if (!sched_core_enabled (rq)) +return __pick_next_task (rq, prev, rf); + + cpu =cpu_of (rq); + +/* Stopper task is switching into idle, no need core-wide selection. */ +if (cpu_is_offline (cpu)) { +/* +* Reset core_pick so that we don't enter the fastpath when +* coming online. core_pick would already be migrated to +* another cpu during offline. +*/ + rq->core_pick =NULL ; +return __pick_next_task (rq, prev, rf); + } + +/* +* If there were no {en,de}queues since we picked (IOW, the task +* pointers are all still valid), and we haven't scheduled the last +* pick yet, do so now. +* +* rq->core_pick can be NULL if no selection was made for a CPU because +* it was either offline or went offline during a sibling's core-wide +* selection. In this case, do a core-wide selection. +*/ +if (rq->core->core_pick_seq == rq->core->core_task_seq && + rq->core->core_pick_seq != rq->core_sched_seq && + rq->core_pick) { +WRITE_ONCE (rq->core_sched_seq, rq->core->core_pick_seq); + + next = rq->core_pick; +if (next != prev) { +put_prev_task (rq, prev); +set_next_task (rq, next); + } + + rq->core_pick =NULL ; +return next; + } + +put_prev_task_balance (rq, prev, rf); + + smt_mask =cpu_smt_mask (cpu); + +/* +* core->core_task_seq, core->core_pick_seq, rq->core_sched_seq +* +* @task_seq guards the task state ({en,de}queues) +* @pick_seq is the @task_seq we did a selection on +* @sched_seq is the @pick_seq we scheduled +* +* However, preemptions can cause multiple picks on the same task set. +* 'Fix' this by also increasing @task_seq for every pick. +*/ + rq->core->core_task_seq++; + need_sync = !!rq->core->core_cookie; + +/* reset state */ + rq->core->core_cookie =0UL ; +for_each_cpu (i, smt_mask) { +struct rq *rq_i =cpu_rq (i); + + rq_i->core_pick =NULL ; + +if (rq_i->core_forceidle) { + need_sync = true; + rq_i->core_forceidle = false; + } + +if (i != cpu) +update_rq_clock (rq_i); + } + +/* +* Try and select tasks for each sibling in decending sched_class +* order. +*/ +for_each_class (class) { +again: +for_each_cpu_wrap (i, smt_mask, cpu) { +struct rq *rq_i =cpu_rq (i); +struct task_struct *p; + +if (rq_i->core_pick) +continue ; + +/* +* If this sibling doesn't yet have a suitable task to +* run; ask for the most elegible task, given the +* highest priority task already selected for this +* core. +*/ + p =pick_task (rq_i, class, max); +if (!p) { +/* +* If there weren't no cookies; we don't need to +* bother with the other siblings. +* If the rest of the core is not running a tagged +* task, i.e. need_sync == 0, and the current CPU +* which called into the schedule() loop does not +* have any tasks for this class, skip selecting for +* other siblings since there's no point. We don't skip +* for RT/DL because that could make CFS force-idle RT. +*/ +if (i == cpu && !need_sync && class == &fair_sched_class) + goto next_class; + +continue ; + } + +/* +* Optimize the 'normal' case where there aren't any +* cookies and we don't need to sync up. +*/ +if (i == cpu && !need_sync && !p->core_cookie) { + next = p; + goto done; + } + + rq_i->core_pick = p; + +/* +* If this new candidate is of higher priority than the +* previous; and they're incompatible; we need to wipe +* the slate and start over. pick_task makes sure that +* p's priority is more than max if it doesn't match +* max's cookie. +* +* NOTE: this is a linear max-filter and is thus bounded +* in execution time. +*/ +if (!max || !cookie_match (max, p)) { +struct task_struct *old_max = max; + + rq->core->core_cookie = p->core_cookie; + max = p; + +if (old_max) { +for_each_cpu (j, smt_mask) { +if (j == i) +continue ; + +cpu_rq (j)->core_pick =NULL ; + } + goto again; + }else { +/* +* Once we select a task for a cpu, we +* should not be doing an unconstrained +* pick because it might starve a task +* on a forced idle cpu. +*/ + need_sync = true; + } + + } + } +next_class:; + } + + rq->core->core_pick_seq = rq->core->core_task_seq; + next = rq->core_pick; + rq->core_sched_seq = rq->core->core_pick_seq; + +/* Something should have been selected for current CPU */ +WARN_ON_ONCE (!next); + +/* +* Reschedule siblings +* +* NOTE: L1TF -- at this point we're no longer running the old task and +* sending an IPI (below) ensures the sibling will no longer be running +* their task. This ensures there is no inter-sibling overlap between +* non-matching user state. +*/ +for_each_cpu (i, smt_mask) { +struct rq *rq_i =cpu_rq (i); + +/* +* An online sibling might have gone offline before a task +* could be picked for it, or it might be offline but later +* happen to come online, but its too late and nothing was +* picked for it. That's Ok - it will pick tasks for itself, +* so ignore it. +*/ +if (!rq_i->core_pick) +continue ; + +if (is_task_rq_idle (rq_i->core_pick) && rq_i->nr_running) + rq_i->core_forceidle = true; + +if (i == cpu) { + rq_i->core_pick =NULL ; +continue ; + } + +/* Did we break L1TF mitigation requirements? */ +WARN_ON_ONCE (!cookie_match (next, rq_i->core_pick)); + +if (rq_i->curr == rq_i->core_pick) { + rq_i->core_pick =NULL ; +continue ; + } + +resched_curr (rq_i); + } + +done: +set_next_task (rq, next); +return next; +}static inline void sched_core_cpu_starting (unsigned int cpu) {
@@ -5103,6 +5391,12 @@ static inline void sched_core_cpu_starting(unsigned int cpu)static inline void sched_core_cpu_starting (unsigned int cpu) {} +static struct task_struct * +pick_next_task (struct rq *rq,struct task_struct *prev,struct rq_flags *rf) +{ +return __pick_next_task (rq, prev, rf); +} +#endif /* CONFIG_SCHED_CORE */ /*
@@ -7999,7 +8293,12 @@ void __init sched_init(void)#ifdef CONFIG_SCHED_CORE rq->core =NULL ; + rq->core_pick =NULL ; rq->core_enabled =0 ; + rq->core_tree =RB_ROOT ; + rq->core_forceidle = false; + + rq->core_cookie =0UL ;#endif }
kernel/sched/sched.h +5 -1
@@ -1065,11 +1065,16 @@ struct rq {#ifdef CONFIG_SCHED_CORE /* per rq */ struct rq *core; +struct task_struct *core_pick;unsigned int core_enabled; +unsigned int core_sched_seq;struct rb_root core_tree; +unsigned char core_forceidle;/* shared state */ unsigned int core_task_seq; +unsigned int core_pick_seq; +unsigned long core_cookie;#endif };
@@ -1977,7 +1982,6 @@ static inline void put_prev_task(struct rq *rq, struct task_struct *prev)static inline void set_next_task (struct rq *rq,struct task_struct *next) { -WARN_ON_ONCE (rq->curr != next); next->sched_class->set_next_task (rq, next, false); }
Signature · 1 line
2.29.2.299.gdc1121823c-goog
PATCH 8/32 sched/fair: Fix forced idle sibling starvation corner case
In reply to Joel Fernandes (Google)From: Vineeth Pillai <[email protected]>
If there is only one long running local task and the sibling is forced idle, it might not get a chance to run until a schedule event happens on any cpu in the core.
So we check for this condition during a tick to see if a sibling is starved and then give it a chance to schedule.
--- kernel/sched/core.c | 15 ++++++++------- kernel/sched/fair.c | 40 ++++++++++++++++++++++++++++++++++++++++ kernel/sched/sched.h | 2 +- 3 files changed, 49 insertions(+), 8 deletions(-)
kernel/sched/core.c +8 -7
@@ -5206,16 +5206,15 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)/* reset state */ rq->core->core_cookie =0UL ; +if (rq->core->core_forceidle) { + need_sync = true; + rq->core->core_forceidle = false; + }for_each_cpu (i, smt_mask) {struct rq *rq_i =cpu_rq (i); rq_i->core_pick =NULL ; -if (rq_i->core_forceidle) { - need_sync = true; - rq_i->core_forceidle = false; - } -if (i != cpu)update_rq_clock (rq_i); }
@@ -5335,8 +5334,10 @@ next_class:;if (!rq_i->core_pick)continue ; -if (is_task_rq_idle (rq_i->core_pick) && rq_i->nr_running) - rq_i->core_forceidle = true; +if (is_task_rq_idle (rq_i->core_pick) && rq_i->nr_running && + !rq_i->core->core_forceidle) { + rq_i->core->core_forceidle = true; + }if (i == cpu) { rq_i->core_pick =NULL ;
kernel/sched/fair.c +40 -0
@@ -10692,6 +10692,44 @@ static void rq_offline_fair(struct rq *rq)#endif /* CONFIG_SMP */ +#ifdef CONFIG_SCHED_CORE +static inline bool +__entity_slice_used (struct sched_entity *se,int min_nr_tasks) +{ +u64 slice =sched_slice (cfs_rq_of (se), se); +u64 rtime = se->sum_exec_runtime - se->prev_sum_exec_runtime; + +return (rtime * min_nr_tasks > slice); +} + +#define MIN_NR_TASKS_DURING_FORCEIDLE 2 +static inline void task_tick_core (struct rq *rq,struct task_struct *curr) +{ +if (!sched_core_enabled (rq)) +return ; + +/* +* If runqueue has only one task which used up its slice and +* if the sibling is forced idle, then trigger schedule to +* give forced idle task a chance. +* +* sched_slice() considers only this active rq and it gets the +* whole slice. But during force idle, we have siblings acting +* like a single runqueue and hence we need to consider runnable +* tasks on this cpu and the forced idle cpu. Ideally, we should +* go through the forced idle rq, but that would be a perf hit. +* We can assume that the forced idle cpu has atleast +* MIN_NR_TASKS_DURING_FORCEIDLE - 1 tasks and use that to check +* if we need to give up the cpu. +*/ +if (rq->core->core_forceidle && rq->cfs.nr_running ==1 && +__entity_slice_used (&curr->se,MIN_NR_TASKS_DURING_FORCEIDLE )) +resched_curr (rq); +} +#else +static inline void task_tick_core (struct rq *rq,struct task_struct *curr) {} +#endif +/* * scheduler tick hitting a task of our scheduling class. *
@@ -10715,6 +10753,8 @@ static void task_tick_fair(struct rq *rq, struct task_struct *curr, int queued)update_misfit_status (curr, rq);update_overutilized_status (task_rq (curr)); + +task_tick_core (rq, curr); }/*
kernel/sched/sched.h +1 -1
@@ -1069,12 +1069,12 @@ struct rq {unsigned int core_enabled;unsigned int core_sched_seq;struct rb_root core_tree; -unsigned char core_forceidle;/* shared state */ unsigned int core_task_seq;unsigned int core_pick_seq;unsigned long core_cookie; +unsigned char core_forceidle;#endif };
Signature · 1 line
2.29.2.299.gdc1121823c-goog
Re: [PATCH -tip 08/32] sched/fair: Fix forced idle sibling starvation corner case
Quoted 63 lines · Joel Fernandes (Google)
From: Vineeth Pillai <[email protected]>
If there is only one long running local task and the sibling is forced idle, it might not get a chance to run until a schedule event happens on any cpu in the core.
So we check for this condition during a tick to see if a sibling is starved and then give it a chance to schedule.
Tested-by Julien Desfossez <[email protected]>Reviewed-by Joel Fernandes (Google) <[email protected]>Signed-off-by Vineeth Pillai <[email protected]>Signed-off-by Julien Desfossez <[email protected]>Signed-off-by Joel Fernandes (Google) <[email protected]>--- kernel/sched/core.c | 15 ++++++++------- kernel/sched/fair.c | 40 ++++++++++++++++++++++++++++++++++++++++ kernel/sched/sched.h | 2 +- 3 files changed, 49 insertions(+), 8 deletions(-)kernel/sched/core.c +8
-7@@ -5206,16 +5206,15 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)/* reset state */ rq->core->core_cookie =0UL ; +if (rq->core->core_forceidle) { + need_sync = true; + rq->core->core_forceidle = false; + }for_each_cpu (i, smt_mask) {struct rq *rq_i =cpu_rq (i); rq_i->core_pick =NULL ; -if (rq_i->core_forceidle) { - need_sync = true; - rq_i->core_forceidle = false; - } -if (i != cpu)update_rq_clock (rq_i); }@@ -5335,8 +5334,10 @@ next_class:;if (!rq_i->core_pick)continue ; -if (is_task_rq_idle (rq_i->core_pick) && rq_i->nr_running) - rq_i->core_forceidle = true; +if (is_task_rq_idle (rq_i->core_pick) && rq_i->nr_running && + !rq_i->core->core_forceidle) { + rq_i->core->core_forceidle = true; + }if (i == cpu) { rq_i->core_pick =NULL ;kernel/sched/fair.c +0
-0
kernel/sched/fair.c static void rq_offline_fair(struct rq *rq) · line 10699
Quoted 8 lines
kernel/sched/fair.c +5
-0@@ -10692,6 +10692,44 @@ static void rq_offline_fair(struct rq *rq)#endif /* CONFIG_SMP */ +#ifdef CONFIG_SCHED_CORE +static inline bool +__entity_slice_used (struct sched_entity *se,int min_nr_tasks) +{ +u64 slice =sched_slice (cfs_rq_of (se), se);
I wonder if the definition of sched_slice() should be revisited for core scheduling?
Should we use sched_slice = sched_slice / cpumask_weight(smt_mask)? Would that resolve the issue your seeing? Effectively we need to answer if two sched core siblings should be treated as executing one large slice?
Balbir Singh.
PATCH 9/32 sched/fair: Snapshot the min_vruntime of CPUs on force idle
In reply to Joel Fernandes (Google)During force-idle, we end up doing cross-cpu comparison of vruntimes during pick_next_task. If we simply compare (vruntime-min_vruntime) across CPUs, and if the CPUs only have 1 task each, we will always end up comparing 0 with 0 and pick just one of the tasks all the time. This starves the task that was not picked. To fix this, take a snapshot of the min_vruntime when entering force idle and use it for comparison. This min_vruntime snapshot will only be used for cross-CPU vruntime comparison, and nothing else.
This resolves several performance issues that were seen in ChromeOS audio usecase.
NOTE: Note, this patch will be improved in a later patch. It is just
kept here as the basis for the later patch and to make rebasing easier. Further, it may make reverting the improvement easier in case the improvement causes any regression.
--- kernel/sched/core.c | 33 ++++++++++++++++++++------------- kernel/sched/fair.c | 40 ++++++++++++++++++++++++++++++++++++++++ kernel/sched/sched.h | 5 +++++ 3 files changed, 65 insertions(+), 13 deletions(-)
kernel/sched/core.c +20 -13
@@ -115,19 +115,8 @@ static inline bool prio_less(struct task_struct *a, struct task_struct *b)if (pa ==-1 )/* dl_prio() doesn't work because of stop_class above */ return !dl_time_before (a->dl.deadline, b->dl.deadline); -if (pa ==MAX_RT_PRIO +MAX_NICE ) {/* fair */ -u64 vruntime = b->se.vruntime; - -/* -* Normalize the vruntime if tasks are in different cpus. -*/ -if (task_cpu (a) !=task_cpu (b)) { - vruntime -=task_cfs_rq (b)->min_vruntime; - vruntime +=task_cfs_rq (a)->min_vruntime; - } - -return !((s64)(a->se.vruntime - vruntime) <=0 ); - } +if (pa ==MAX_RT_PRIO +MAX_NICE )/* fair */ +return cfs_prio_less (a, b);return false; }
@@ -5144,6 +5133,7 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)struct task_struct *next, *max =NULL ;const struct sched_class *class;const struct cpumask *smt_mask; +bool fi_before = false;bool need_sync;int i, j, cpu;
@@ -5208,6 +5198,7 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf) rq->core->core_cookie =0UL ;if (rq->core->core_forceidle) { need_sync = true; + fi_before = true; rq->core->core_forceidle = false; }for_each_cpu (i, smt_mask) {
@@ -5219,6 +5210,14 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)update_rq_clock (rq_i); } +/* Reset the snapshot if core is no longer in force-idle. */ +if (!fi_before) { +for_each_cpu (i, smt_mask) { +struct rq *rq_i =cpu_rq (i); + rq_i->cfs.min_vruntime_fi = rq_i->cfs.min_vruntime; + } + } +/* * Try and select tasks for each sibling in decending sched_class * order.
@@ -5355,6 +5354,14 @@ next_class:;resched_curr (rq_i); } +/* Snapshot if core is in force-idle. */ +if (!fi_before && rq->core->core_forceidle) { +for_each_cpu (i, smt_mask) { +struct rq *rq_i =cpu_rq (i); + rq_i->cfs.min_vruntime_fi = rq_i->cfs.min_vruntime; + } + } + done:set_next_task (rq, next);return next;
kernel/sched/fair.c +40 -0
@@ -10726,6 +10726,46 @@ static inline void task_tick_core(struct rq *rq, struct task_struct *curr) __entity_slice_used(&curr->se, MIN_NR_TASKS_DURING_FORCEIDLE))resched_curr (rq); } + +bool cfs_prio_less (struct task_struct *a,struct task_struct *b) +{ +bool samecpu =task_cpu (a) ==task_cpu (b); +struct sched_entity *sea = &a->se; +struct sched_entity *seb = &b->se; +struct cfs_rq *cfs_rqa; +struct cfs_rq *cfs_rqb; +s64 delta; + +if (samecpu) { +/* vruntime is per cfs_rq */ +while (!is_same_group (sea, seb)) { +int sea_depth = sea->depth; +int seb_depth = seb->depth; +if (sea_depth >= seb_depth) + sea =parent_entity (sea); +if (sea_depth <= seb_depth) + seb =parent_entity (seb); + } + + delta = (s64)(sea->vruntime - seb->vruntime); + goto out; + } + +/* crosscpu: compare root level se's vruntime to decide priority */ +while (sea->parent) + sea = sea->parent; +while (seb->parent) + seb = seb->parent; + + cfs_rqa = sea->cfs_rq; + cfs_rqb = seb->cfs_rq; + +/* normalize vruntime WRT their rq's base */ + delta = (s64)(sea->vruntime - seb->vruntime) + + (s64)(cfs_rqb->min_vruntime_fi - cfs_rqa->min_vruntime_fi); +out: +return delta >0 ; +}#else static inline void task_tick_core (struct rq *rq,struct task_struct *curr) {}#endif
kernel/sched/sched.h +5 -0
@@ -517,6 +517,9 @@ struct cfs_rq {u64 exec_clock;u64 min_vruntime; +#ifdef CONFIG_SCHED_CORE +u64 min_vruntime_fi; +#endif #ifndef CONFIG_64BIT u64 min_vruntime_copy;#endif
@@ -1130,6 +1133,8 @@ static inline raw_spinlock_t *rq_lockp(struct rq *rq) return &rq->__lock; } +bool cfs_prio_less (struct task_struct *a,struct task_struct *b); +#else /* !CONFIG_SCHED_CORE */ static inline bool sched_core_enabled (struct rq *rq)
Signature · 1 line
2.29.2.299.gdc1121823c-goog
Re: [PATCH -tip 09/32] sched/fair: Snapshot the min_vruntime of CPUs on force idle
Quoted 9 lines · Joel Fernandes (Google)
During force-idle, we end up doing cross-cpu comparison of vruntimes during
pick_next_task. If we simply compare (vruntime-min_vruntime) across CPUs, and if the CPUs only have 1 task each, we will always end up comparing 0 with 0 and pick just one of the tasks all the time. This starves the task that was not picked. To fix this, take a snapshot of themin_vruntimewhen entering force idle and use it for comparison. Thismin_vruntimesnapshot will only be used for cross-CPU vruntime comparison, and nothing else.
Quoted 8 lines
This resolves several performance issues that were seen in ChromeOS audio usecase.
NOTE: Note, this patch will be improved in a later patch. It is just
kept here as the basis for the later patch and to make rebasing easier. Further, it may make reverting the improvement easier in case the improvement causes any regression.
This seems cumbersome, is there no way to track the min_vruntime via rq->core->min_vruntime?
Balbir Singh.
Re: [PATCH -tip 09/32] sched/fair: Snapshot the min_vruntime of CPUs on force idle
Hi Balbir,
Quoted 3 lines · Balbir Singh
This seems cumbersome, is there no way to track the
min_vruntimeviarq->core->min_vruntime?
Do you mean to have a core wide min_vruntime? We had a similar approach from v3 to v7 and it had major issues which broke the assumptions of cfs. There were some lengthy discussions and Peter explained in-depth about the issues:
https://lwn.net/ml/linux-kernel/[email protected]/
https://lwn.net/ml/linux-kernel/[email protected]/
Thanks,
Vineeth
Re: [PATCH -tip 09/32] sched/fair: Snapshot the min_vruntime of CPUs on force idle
Quoted 6 lines · Vineeth Pillai
Hi Balbir,
Quoted 3 lines · Balbir Singh
This seems cumbersome, is there no way to track the
min_vruntimeviarq->core->min_vruntime?
Quoted 8 lines
Do you mean to have a core wide
min_vruntime? We had a similar approach from v3 to v7 and it had major issues which broke the assumptions of cfs. There were some lengthy discussions and Peter explained in-depth about the issues:https://lwn.net/ml/linux-kernel/[email protected]/
https://lwn.net/ml/linux-kernel/[email protected]/
One of the equations in the link is
">From which immediately follows that:
T_k + T_l
S_k+l = --------- (13)
W_k + W_lOn which we can define a combined lag:
lag_k+l(i) := S_k+l - s_i (14)
And that gives us the tools to compare tasks across a combined runqueue. "
S_k+l reads like rq->core->vruntime, but it sounds like the equivalent of rq->core->vruntime is updated when we enter forced idle as opposed to all the time.
Balbir
Peter Zijlstra
maintainerRe: [PATCH -tip 09/32] sched/fair: Snapshot the min_vruntime of CPUs on force idle
Quoted 26 lines · Balbir Singh
Quoted 6 lines · Vineeth Pillai
Hi Balbir,
Quoted 3 lines · Balbir Singh
This seems cumbersome, is there no way to track the
min_vruntimeviarq->core->min_vruntime?Quoted 8 lines
Do you mean to have a core wide
min_vruntime? We had a similar approach from v3 to v7 and it had major issues which broke the assumptions of cfs. There were some lengthy discussions and Peter explained in-depth about the issues:https://lwn.net/ml/linux-kernel/[email protected]/
https://lwn.net/ml/linux-kernel/[email protected]/One of the equations in the link is
">From which immediately follows that:
T_k + T_l S_k+l = --------- (13) W_k + W_lOn which we can define a combined lag:
Quoted 8 lines
lag_k+l(i) := S_k+l -
s_i(14)And that gives us the tools to compare tasks across a combined runqueue. "
S_k+l reads like
rq->core->vruntime, but it sounds like the equivalent ofrq->core->vruntimeis updated when we enter forced idle as opposed to all the time.
Yes, but actually computing and maintaining it is hella hard. Try it with the very first example in that email (the infeasible weight scenario) and tell me how it works for you ;-)
Also note that the text below (6) mentions dynamic, then look up the EEVDF paper which describes some of the dynamics -- the paper is incomplete and contains a bug, I forget if it ever got updated or if there's another paper that completes it (the BQF I/O scheduler started from that and fixed it).
I'm not saying it cannot be done, I'm just saying it is really rather involved and probably not worth it.
The basic observation the current approach relies on is that al that faffery basically boils down to the fact that vruntime only means something when there is contention. And that only the progression is important not the actual value. That is, this is all fundamentally a differential equation and our integration constant is meaningless (also embodied in (7)).
Also, I think the code as proposed here relies on SMT2 and is buggered for SMT3+. Now, that second link above describes means of making SMT3+ work, but we're not there yet.
Re: [PATCH -tip 09/32] sched/fair: Snapshot the min_vruntime of CPUs on force idle
Quoted 32 lines · Peter Zijlstra
Quoted 31 lines · Balbir Singh
Quoted 6 lines · Vineeth Pillai
Hi Balbir,
Quoted 3 lines · Balbir Singh
This seems cumbersome, is there no way to track the
min_vruntimeviarq->core->min_vruntime?Quoted 8 lines
Do you mean to have a core wide
min_vruntime? We had a similar approach from v3 to v7 and it had major issues which broke the assumptions of cfs. There were some lengthy discussions and Peter explained in-depth about the issues:https://lwn.net/ml/linux-kernel/[email protected]/
https://lwn.net/ml/linux-kernel/[email protected]/One of the equations in the link is
">From which immediately follows that:
T_k + T_l S_k+l = --------- (13) W_k + W_lOn which we can define a combined lag:
lag_k+l(i) := S_k+l -
s_i(14)And that gives us the tools to compare tasks across a combined runqueue. "
Quoted 8 lines
Quoted 3 lines
S_k+l reads like
rq->core->vruntime, but it sounds like the equivalent ofrq->core->vruntimeis updated when we enter forced idle as opposed to all the time.Yes, but actually computing and maintaining it is hella hard. Try it with the very first example in that email (the infeasible weight scenario) and tell me how it works for you ;-)
OK, I was hoping it could be done in the new RBTree's enqueue/dequeue, but yes I've not implemented it and I should go back and take a look at the first example again.
Quoted 5 lines
Also note that the text below (6) mentions dynamic, then look up the EEVDF paper which describes some of the dynamics -- the paper is incomplete and contains a bug, I forget if it ever got updated or if there's another paper that completes it (the BQF I/O scheduler started from that and fixed it).
I see, I am yet to read the EEVDF paper, but now I am out on a tangent :)
Quoted 4 lines
I'm not saying it cannot be done, I'm just saying it is really rather involved and probably not worth it.
Fair enough
Quoted 7 lines
The basic observation the current approach relies on is that al that faffery basically boils down to the fact that vruntime only means something when there is contention. And that only the progression is important not the actual value. That is, this is all fundamentally a differential equation and our integration constant is meaningless (also embodied in (7)).
I'll reread (6) and (7), I am trying to understand forced idle and contention together, from what I understand of the patches, there is
1. two tasks that are core scheduled, in that case vruntime works as expected on each CPU, but we need to compare their combined vrtuntime against other tasks on each CPU respectively for them to be selected/chosen?
2. When one of the tasks selected is a part of the core scheduling group and the other CPU does not select a core scheduled tasks, we need to ask ourselves if that CPU should force idle and that's where this logic comes into play?
Quoted 3 lines
Also, I think the code as proposed here relies on SMT2 and is buggered for SMT3+. Now, that second link above describes means of making SMT3+ work, but we're not there yet.
Thanks,
Balbir Singh.
Peter Zijlstra
maintainerRe: [PATCH -tip 09/32] sched/fair: Snapshot the min_vruntime of CPUs on force idle
Quoted 10 lines · Balbir Singh
Quoted 7 lines · Peter Zijlstra
The basic observation the current approach relies on is that al that faffery basically boils down to the fact that vruntime only means something when there is contention. And that only the progression is important not the actual value. That is, this is all fundamentally a differential equation and our integration constant is meaningless (also embodied in (7)).
Quoted 2 lines
I'll reread (6) and (7), I am trying to understand forced idle and contention together, from what I understand of the patches, there is
When we force-idle there is contention by definition; there's a task that wanted to run, but couldn't.
Quoted 4 lines
1. two tasks that are core scheduled, in that case vruntime works as expected on each CPU, but we need to compare their combined vrtuntime against other tasks on each CPU respectively for them to be selected/chosen?
We need to compare across CPUs when the cookies don't match. This is required to avoid starving one or the other.
Quoted 4 lines
2. When one of the tasks selected is a part of the core scheduling group and the other CPU does not select a core scheduled tasks, we need to ask ourselves if that CPU should force idle and that's where this logic comes into play?
When one CPU selects a cookie task, and the other CPU cannot find a matching task, it must go idle (as idle matches everyone). This is the basic core-scheduling constraint.
So suppose you have two tasks, A and B, both with a cookie, but not matching.
Normal scheduling would run A and B concurrent on the two siblings. Core scheduling obviously cannot do this. When we pick A, the other CPU is not allowed to run B and thus will have to be forced idle and vice-versa.
The next problem is avoiding starvation. Assuming equal weight between the tasks, we'd want to end up running A and B in alternating cycles.
This means having to compare runtimes between A and B, but when they're on different runqueues the actual vruntime values can be wildly divergent and cannot be reasily compared (the integration constant is meaningless but really annoying ;-).
We also cannot use min_vruntime (which is the same as the task vruntime when there is only a single task), because then you cannot observe progress. The difference between min_vruntime and the task runtime is always 0, so you can't tell who just ran and who got starved.
This is where our snapshots come in play, we snapshot vruntime after task selection (before running), such that at the next pick we can tell who made progress and who got starved.
By marking the vruntime of both runqueues at the same point in time we basically normalize away that integration constant. You effectively reset the vruntime to 0 (through (7), but without iterating all the tasks and adjusting it).
Does that make sense?
Once you get this, read that second email linked.
PATCH 10/32 sched: Fix priority inversion of cookied task with sibling
In reply to Joel Fernandes (Google)From: Peter Zijlstra <[email protected]>
The rationale is as follows. In the core-wide pick logic, even if need_sync == false, we need to go look at other CPUs (non-local CPUs) to see if they could be running RT.
Say the RQs in a particular core look like this:
Let CFS1 and CFS2 be 2 tagged CFS tags. Let RT1 be an untagged RT task.
rq0 rq1
CFS1 (tagged) RT1 (not tag)
CFS2 (tagged)
Say schedule() runs on rq0. Now, it will enter the above loop and pick_task(RT) will return NULL for 'p'. It will enter the above if() block and see that need_sync == false and will skip RT entirely.
The end result of the selection will be (say prio(CFS1) > prio(CFS2)):
rq0 rq1
CFS1 IDLE
When it should have selected:
rq0 r1
IDLE RT
Joel saw this issue on real-world usecases in ChromeOS where an RT task gets constantly force-idled and breaks RT. Lets cure it.
NOTE: This problem will be fixed differently in a later patch. It just
kept here for reference purposes about this issue, and to make applying later patches easier.
--- kernel/sched/core.c | 25 ++++++++++++++++--------- 1 file changed, 16 insertions(+), 9 deletions(-)
kernel/sched/core.c +16 -9
@@ -5195,6 +5195,7 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf) need_sync = !!rq->core->core_cookie;/* reset state */ +reset: rq->core->core_cookie =0UL ;if (rq->core->core_forceidle) { need_sync = true;
@@ -5242,14 +5243,8 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)/* * If there weren't no cookies; we don't need to * bother with the other siblings. -* If the rest of the core is not running a tagged -* task, i.e. need_sync == 0, and the current CPU -* which called into the schedule() loop does not -* have any tasks for this class, skip selecting for -* other siblings since there's no point. We don't skip -* for RT/DL because that could make CFS force-idle RT. */ -if (i == cpu && !need_sync && class == &fair_sched_class) +if (i == cpu && !need_sync) goto next_class;continue ;
@@ -5259,7 +5254,20 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)* Optimize the 'normal' case where there aren't any * cookies and we don't need to sync up. */ -if (i == cpu && !need_sync && !p->core_cookie) { +if (i == cpu && !need_sync) { +if (p->core_cookie) { +/* +* This optimization is only valid as +* long as there are no cookies +* involved. We may have skipped +* non-empty higher priority classes on +* siblings, which are empty on this +* CPU, so start over. +*/ + need_sync = true; + goto reset; + } + next = p; goto done; }
@@ -5299,7 +5307,6 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)*/ need_sync = true; } - } } next_class:;
Signature · 1 line
2.29.2.299.gdc1121823c-goog
Re: [PATCH -tip 10/32] sched: Fix priority inversion of cookied task with sibling
Quoted 25 lines · Joel Fernandes (Google)
From: Peter Zijlstra <[email protected]>
The rationale is as follows. In the core-wide pick logic, even if
need_sync== false, we need to go look at other CPUs (non-local CPUs) to see if they could be running RT.Say the RQs in a particular core look like this:
Let CFS1 and CFS2 be 2 tagged CFS tags. Let RT1 be an untagged RT task.rq0 rq1
CFS1 (tagged) RT1 (not tag)
CFS2 (tagged)Say
schedule()runs on rq0. Now, it will enter the above loop andpick_task(RT)will return NULL for 'p'. It will enter the aboveif()block and see thatneed_sync== false and will skip RT entirely.The end result of the selection will be (say prio(CFS1) > prio(CFS2)):
rq0 rq1
CFS1 IDLEWhen it should have selected:
rq0 r1
IDLE RT
Quoted 7 lines
Joel saw this issue on real-world usecases in ChromeOS where an RT task gets constantly force-idled and breaks RT. Lets cure it.
NOTE: This problem will be fixed differently in a later patch. It just
kept here for reference purposes about this issue, and to make applying later patches easier.
The changelog is hard to read, it refers to above if(), whereas there is no code snippet in the changelog. Also, from what I can see following the series, p->core_cookie is not yet set anywhere (unless I missed it), so fixing it in here did not make sense just reading the series.
Balbir
Re: [PATCH -tip 10/32] sched: Fix priority inversion of cookied task with sibling
Quoted 29 lines · Balbir Singh
Quoted 28 lines · Joel Fernandes (Google)
From: Peter Zijlstra <[email protected]>
The rationale is as follows. In the core-wide pick logic, even if
need_sync== false, we need to go look at other CPUs (non-local CPUs) to see if they could be running RT.Say the RQs in a particular core look like this:
Let CFS1 and CFS2 be 2 tagged CFS tags. Let RT1 be an untagged RT task.rq0 rq1
CFS1 (tagged) RT1 (not tag)
CFS2 (tagged)Say
schedule()runs on rq0. Now, it will enter the above loop andpick_task(RT)will return NULL for 'p'. It will enter the aboveif()block and see thatneed_sync== false and will skip RT entirely.The end result of the selection will be (say prio(CFS1) > prio(CFS2)):
rq0 rq1
CFS1 IDLEWhen it should have selected:
rq0 r1
IDLE RTJoel saw this issue on real-world usecases in ChromeOS where an RT task gets constantly force-idled and breaks RT. Lets cure it.
Quoted 7 lines
Quoted 4 lines
NOTE: This problem will be fixed differently in a later patch. It just
kept here for reference purposes about this issue, and to make applying later patches easier.The changelog is hard to read, it refers to above
if(), whereas there is no code snippet in the changelog.
Yeah sorry, it comes from this email where I described the issue:
http://lore.kernel.org/r/[email protected]
I corrected the changelog and appended the patch below. Also pushed it to:
https://git.kernel.org/pub/scm/linux/kernel/git/jfern/linux.git/log/?h=coresched
Quoted 3 lines
Also, from what I can see following
the series,p->core_cookieis not yet set anywhere (unless I missed it), so fixing it in here did not make sense just reading the series.
The interface patches for core_cookie are added later, that's how it is. The infrastructure comes first here. It would also not make sense to add interface first as well so I think the current ordering is fine.
---8<-----------------------
From: Peter Zijlstra <[email protected]>
Subject: [PATCH] sched: Fix priority inversion of cookied task with sibling
The rationale is as follows. In the core-wide pick logic, even if need_sync == false, we need to go look at other CPUs (non-local CPUs) to see if they could be running RT.
Say the RQs in a particular core look like this:
Let CFS1 and CFS2 be 2 tagged CFS tags. Let RT1 be an untagged RT task.
rq0 rq1
CFS1 (tagged) RT1 (not tag)
CFS2 (tagged)
The end result of the selection will be (say prio(CFS1) > prio(CFS2)):
rq0 rq1
CFS1 IDLE
When it should have selected:
rq0 r1
IDLE RT
Fix this issue by forcing need_sync and restarting the search if a cookied task was discovered. This will avoid this optimization from making incorrect picks.
Joel saw this issue on real-world usecases in ChromeOS where an RT task gets constantly force-idled and breaks RT. Lets cure it.
NOTE: This problem will be fixed differently in a later patch. It just
kept here for reference purposes about this issue, and to make applying later patches easier.
--- kernel/sched/core.c | 25 ++++++++++++++++--------- 1 file changed, 16 insertions(+), 9 deletions(-)
kernel/sched/core.c +16 -9
@@ -5195,6 +5195,7 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf) need_sync = !!rq->core->core_cookie;/* reset state */ +reset: rq->core->core_cookie =0UL ;if (rq->core->core_forceidle) { need_sync = true;
@@ -5242,14 +5243,8 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)/* * If there weren't no cookies; we don't need to * bother with the other siblings. -* If the rest of the core is not running a tagged -* task, i.e. need_sync == 0, and the current CPU -* which called into the schedule() loop does not -* have any tasks for this class, skip selecting for -* other siblings since there's no point. We don't skip -* for RT/DL because that could make CFS force-idle RT. */ -if (i == cpu && !need_sync && class == &fair_sched_class) +if (i == cpu && !need_sync) goto next_class;continue ;
@@ -5259,7 +5254,20 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)* Optimize the 'normal' case where there aren't any * cookies and we don't need to sync up. */ -if (i == cpu && !need_sync && !p->core_cookie) { +if (i == cpu && !need_sync) { +if (p->core_cookie) { +/* +* This optimization is only valid as +* long as there are no cookies +* involved. We may have skipped +* non-empty higher priority classes on +* siblings, which are empty on this +* CPU, so start over. +*/ + need_sync = true; + goto reset; + } + next = p; goto done; }
@@ -5299,7 +5307,6 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)*/ need_sync = true; } - } } next_class:;
Signature · 1 line
2.29.2.454.gaff20da3a2-goog
Re: [PATCH -tip 10/32] sched: Fix priority inversion of cookied task with sibling
Quoted 44 lines · Joel Fernandes
Quoted 29 lines · Balbir Singh
Quoted 28 lines · Joel Fernandes (Google)
From: Peter Zijlstra <[email protected]>
The rationale is as follows. In the core-wide pick logic, even if
need_sync== false, we need to go look at other CPUs (non-local CPUs) to see if they could be running RT.Say the RQs in a particular core look like this:
Let CFS1 and CFS2 be 2 tagged CFS tags. Let RT1 be an untagged RT task.rq0 rq1
CFS1 (tagged) RT1 (not tag)
CFS2 (tagged)Say
schedule()runs on rq0. Now, it will enter the above loop andpick_task(RT)will return NULL for 'p'. It will enter the aboveif()block and see thatneed_sync== false and will skip RT entirely.The end result of the selection will be (say prio(CFS1) > prio(CFS2)):
rq0 rq1
CFS1 IDLEWhen it should have selected:
rq0 r1
IDLE RTJoel saw this issue on real-world usecases in ChromeOS where an RT task gets constantly force-idled and breaks RT. Lets cure it.
Quoted 7 lines
Quoted 4 lines
NOTE: This problem will be fixed differently in a later patch. It just
kept here for reference purposes about this issue, and to make applying later patches easier.The changelog is hard to read, it refers to above
if(), whereas there is no code snippet in the changelog.Yeah sorry, it comes from this email where I described the issue:
http://lore.kernel.org/r/[email protected]I corrected the changelog and appended the patch below. Also pushed it to:
https://git.kernel.org/pub/scm/linux/kernel/git/jfern/linux.git/log/?h=coresched
Quoted 8 lines
Quoted 3 lines
Also, from what I can see following
the series,p->core_cookieis not yet set anywhere (unless I missed it), so fixing it in here did not make sense just reading the series.The interface patches for
core_cookieare added later, that's how it is. The infrastructure comes first here. It would also not make sense to add interface first as well so I think the current ordering is fine.
Some comments below to help make the code easier to understand
Quoted 71 lines
---8<-----------------------
From: Peter Zijlstra <[email protected]>
Subject: [PATCH] sched: Fix priority inversion of cookied task with siblingThe rationale is as follows. In the core-wide pick logic, even if
need_sync== false, we need to go look at other CPUs (non-local CPUs) to see if they could be running RT.Say the RQs in a particular core look like this:
Let CFS1 and CFS2 be 2 tagged CFS tags. Let RT1 be an untagged RT task.rq0 rq1
CFS1 (tagged) RT1 (not tag)
CFS2 (tagged)The end result of the selection will be (say prio(CFS1) > prio(CFS2)):
rq0 rq1
CFS1 IDLEWhen it should have selected:
rq0 r1
IDLE RTFix this issue by forcing
need_syncand restarting the search if a cookied task was discovered. This will avoid this optimization from making incorrect picks.Joel saw this issue on real-world usecases in ChromeOS where an RT task gets constantly force-idled and breaks RT. Lets cure it.
NOTE: This problem will be fixed differently in a later patch. It just
kept here for reference purposes about this issue, and to make applying later patches easier.Reported-by Joel Fernandes (Google) <[email protected]>Signed-off-by Peter Zijlstra <[email protected]>Signed-off-by Joel Fernandes (Google) <[email protected]>--- kernel/sched/core.c | 25 ++++++++++++++++--------- 1 file changed, 16 insertions(+), 9 deletions(-)kernel/sched/core.c +2
-7@@ -5195,6 +5195,7 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf) need_sync = !!rq->core->core_cookie;/* reset state */ +reset: rq->core->core_cookie =0UL ;if (rq->core->core_forceidle) { need_sync = true;@@ -5242,14 +5243,8 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)/* * If there weren't no cookies; we don't need to * bother with the other siblings. -* If the rest of the core is not running a tagged -* task, i.e. need_sync == 0, and the current CPU -* which called into the schedule() loop does not -* have any tasks for this class, skip selecting for -* other siblings since there's no point. We don't skip -* for RT/DL because that could make CFS force-idle RT. */ -if (i == cpu && !need_sync && class == &fair_sched_class) +if (i == cpu && !need_sync) goto next_class;continue ;
kernel/sched/core.c pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf) · line 5261
Quoted 9 lines
kernel/sched/core.c +5
-1@@ -5259,7 +5254,20 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)* Optimize the 'normal' case where there aren't any * cookies and we don't need to sync up. */ -if (i == cpu && !need_sync && !p->core_cookie) { +if (i == cpu && !need_sync) { +if (p->core_cookie) { +/* +* This optimization is only valid as +* long as there are no cookies
This is not entirely true, need_sync is a function of core cookies, so I think this needs more clarification, it sounds like we enter this when the core has no cookies, but the task has a core_cookie? The term cookie is quite overloaded when used in the context of core vs task.
Effectively from what I understand this means that p wants to be coscheduled, but the core itself is not coscheduling anything at the moment, so we need to see if we should do a sync and that sync might cause p to get kicked out and a higher priority class to come in?
Balbir Singh.
Peter Zijlstra
maintainerRe: [PATCH -tip 10/32] sched: Fix priority inversion of cookied task with sibling
Quoted 7 lines · Balbir Singh
Quoted 7 lines
kernel/sched/core.c +2
-1@@ -5259,7 +5254,20 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)* Optimize the 'normal' case where there aren't any * cookies and we don't need to sync up. */ -if (i == cpu && !need_sync && !p->core_cookie) { +if (i == cpu && !need_sync) { +if (p->core_cookie) {
Quoted 8 lines
kernel/sched/core.c pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf) · line 5261
Quoted 3 lines
kernel/sched/core.c +3
-0+/* +* This optimization is only valid as +* long as there are no cookies This is not entirely true,
need_syncis a function of core cookies, so I think this needs more clarification, it sounds like we enter this when the core has no cookies, but the task has acore_cookie? The term cookie is quite overloaded when used in the context of core vs task.
Nah, its the same. So each task gets a cookie to identify the 'group' of tasks (possibly just itself) it is allowed to share a core with.
When we to core task selection, the core gets assigned the cookie of the group it will run, same thing.
Quoted 4 lines
Effectively from what I understand this means that p wants to be coscheduled, but the core itself is not coscheduling anything at the moment, so we need to see if we should do a sync and that sync might cause p to get kicked out and a higher priority class to come in?
This whole patch is about eliding code-wide task selection when it is not required. IOW an optimization.
When there wasn't a core cookie (IOW, the previous task selection wasn't core wide and limited) and the task we just selected for our own CPU also didn't have a cookie (IOW it doesn't have to be core-wide) we can skip the core wide task selection and schedule just this CPU and call it a day.
The logic was subtly wrong, this patch fixes it. A next patch completely rewrites it again to make it far simpler again. Don't spend time trying to understand this patch (unless you're _that_ kind of person ;-) but instead apply the whole thing and look at the resulting pick_next_task() function.
Re: [PATCH -tip 10/32] sched: Fix priority inversion of cookied task with sibling
Quoted 37 lines · Peter Zijlstra
Quoted 7 lines · Balbir Singh
Quoted 7 lines
kernel/sched/core.c +2
-1@@ -5259,7 +5254,20 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)* Optimize the 'normal' case where there aren't any * cookies and we don't need to sync up. */ -if (i == cpu && !need_sync && !p->core_cookie) { +if (i == cpu && !need_sync) { +if (p->core_cookie) {Quoted 8 lines
kernel/sched/core.c pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf) · line 5261
Quoted 3 lines
kernel/sched/core.c +3
-0+/* +* This optimization is only valid as +* long as there are no cookies This is not entirely true,
need_syncis a function of core cookies, so I think this needs more clarification, it sounds like we enter this when the core has no cookies, but the task has acore_cookie? The term cookie is quite overloaded when used in the context of core vs task.Nah, its the same. So each task gets a cookie to identify the 'group' of tasks (possibly just itself) it is allowed to share a core with.
When we to core task selection, the core gets assigned the cookie of the group it will run, same thing.
Quoted 4 lines
Effectively from what I understand this means that p wants to be coscheduled, but the core itself is not coscheduling anything at the moment, so we need to see if we should do a sync and that sync might cause p to get kicked out and a higher priority class to come in?
This whole patch is about eliding code-wide task selection when it is not required. IOW an optimization.
When there wasn't a core cookie (IOW, the previous task selection wasn't core wide and limited) and the task we just selected for our own CPU also didn't have a cookie (IOW it doesn't have to be core-wide) we can skip the core wide task selection and schedule just this CPU and call it a day.
Quoted 5 lines
The logic was subtly wrong, this patch fixes it. A next patch completely rewrites it again to make it far simpler again. Don't spend time trying to understand this patch (unless you're _that_ kind of person ;-) but instead apply the whole thing and look at the resulting
pick_next_task()function.
Thanks, I'll look at the git tree and see what the final outcome looks like.
Balbir Singh.
Re: [PATCH -tip 10/32] sched: Fix priority inversion of cookied task with sibling
In reply to Balbir SinghQuoted 142 lines · Balbir Singh
Quoted 44 lines · Joel Fernandes
Quoted 29 lines · Balbir Singh
Quoted 28 lines · Joel Fernandes (Google)
From: Peter Zijlstra <[email protected]>
The rationale is as follows. In the core-wide pick logic, even if
need_sync== false, we need to go look at other CPUs (non-local CPUs) to see if they could be running RT.Say the RQs in a particular core look like this:
Let CFS1 and CFS2 be 2 tagged CFS tags. Let RT1 be an untagged RT task.rq0 rq1
CFS1 (tagged) RT1 (not tag)
CFS2 (tagged)Say
schedule()runs on rq0. Now, it will enter the above loop andpick_task(RT)will return NULL for 'p'. It will enter the aboveif()block and see thatneed_sync== false and will skip RT entirely.The end result of the selection will be (say prio(CFS1) > prio(CFS2)):
rq0 rq1
CFS1 IDLEWhen it should have selected:
rq0 r1
IDLE RTJoel saw this issue on real-world usecases in ChromeOS where an RT task gets constantly force-idled and breaks RT. Lets cure it.
Quoted 7 lines
Quoted 4 lines
NOTE: This problem will be fixed differently in a later patch. It just
kept here for reference purposes about this issue, and to make applying later patches easier.The changelog is hard to read, it refers to above
if(), whereas there is no code snippet in the changelog.Yeah sorry, it comes from this email where I described the issue:
http://lore.kernel.org/r/[email protected]I corrected the changelog and appended the patch below. Also pushed it to:
https://git.kernel.org/pub/scm/linux/kernel/git/jfern/linux.git/log/?h=coreschedQuoted 8 lines
Quoted 3 lines
Also, from what I can see following
the series,p->core_cookieis not yet set anywhere (unless I missed it), so fixing it in here did not make sense just reading the series.The interface patches for
core_cookieare added later, that's how it is. The infrastructure comes first here. It would also not make sense to add interface first as well so I think the current ordering is fine.Some comments below to help make the code easier to understand
Quoted 71 lines
---8<-----------------------
From: Peter Zijlstra <[email protected]>
Subject: [PATCH] sched: Fix priority inversion of cookied task with siblingThe rationale is as follows. In the core-wide pick logic, even if
need_sync== false, we need to go look at other CPUs (non-local CPUs) to see if they could be running RT.Say the RQs in a particular core look like this:
Let CFS1 and CFS2 be 2 tagged CFS tags. Let RT1 be an untagged RT task.rq0 rq1
CFS1 (tagged) RT1 (not tag)
CFS2 (tagged)The end result of the selection will be (say prio(CFS1) > prio(CFS2)):
rq0 rq1
CFS1 IDLEWhen it should have selected:
rq0 r1
IDLE RTFix this issue by forcing
need_syncand restarting the search if a cookied task was discovered. This will avoid this optimization from making incorrect picks.Joel saw this issue on real-world usecases in ChromeOS where an RT task gets constantly force-idled and breaks RT. Lets cure it.
NOTE: This problem will be fixed differently in a later patch. It just
kept here for reference purposes about this issue, and to make applying later patches easier.Reported-by Joel Fernandes (Google) <[email protected]>Signed-off-by Peter Zijlstra <[email protected]>Signed-off-by Joel Fernandes (Google) <[email protected]>--- kernel/sched/core.c | 25 ++++++++++++++++--------- 1 file changed, 16 insertions(+), 9 deletions(-)kernel/sched/core.c +2
-7@@ -5195,6 +5195,7 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf) need_sync = !!rq->core->core_cookie;/* reset state */ +reset: rq->core->core_cookie =0UL ;if (rq->core->core_forceidle) { need_sync = true;@@ -5242,14 +5243,8 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)/* * If there weren't no cookies; we don't need to * bother with the other siblings. -* If the rest of the core is not running a tagged -* task, i.e. need_sync == 0, and the current CPU -* which called into the schedule() loop does not -* have any tasks for this class, skip selecting for -* other siblings since there's no point. We don't skip -* for RT/DL because that could make CFS force-idle RT. */ -if (i == cpu && !need_sync && class == &fair_sched_class) +if (i == cpu && !need_sync) goto next_class;continue ;kernel/sched/core.c pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf) · line 5261
Quoted 9 lines
kernel/sched/core.c +5
-1@@ -5259,7 +5254,20 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)* Optimize the 'normal' case where there aren't any * cookies and we don't need to sync up. */ -if (i == cpu && !need_sync && !p->core_cookie) { +if (i == cpu && !need_sync) { +if (p->core_cookie) { +/* +* This optimization is only valid as +* long as there are no cookies This is not entirely true,
need_syncis a function of core cookies, so I think this needs more clarification, it sounds like we enter this when the core has no cookies, but the task has acore_cookie? The term cookie is quite overloaded when used in the context of core vs task.
Quoted 4 lines
Effectively from what I understand this means that p wants to be coscheduled, but the core itself is not coscheduling anything at the moment, so we need to see if we should do a sync and that sync might cause p to get kicked out and a higher priority class to come in?
Yeah so about need_sync, it is basically a flag that says if the HT running the schedule() loop needs to bother with siblings.
need_sync is true only in following conditions:
- A cookied task is running on any HT on the core.
- Any HT in the core is force idled.
The above code comment you referred to is now reworked. That was for the case where we discovered during local selection that we found a task with a cookie so now we have to do a core-wide scan (need_sync = false before but now it becomes true and we start over). This optimization is done sligtly differently now, we run ->pick_task() on every class of the local CPU until we find something, if we find something with a cookie then we do core-wide selection.
The latest version of this code is now in Peter's branch:
https://git.kernel.org/pub/scm/linux/kernel/git/peterz/queue.git/tree/kernel/sched/core.c?id=6288c0a49631ce6b53eeab7021a43e49c4c4d436
- Joel
PATCH 11/32 sched: Enqueue task into core queue only after vruntime is updated
In reply to Joel Fernandes (Google)A waking task may have its vruntime adjusted. However, the code right now puts it into the core queue without the adjustment. This means the core queue may have a task with incorrect vruntime, potentially a very long one. This may cause a task to get artificially boosted during picking.
Fix it by enqueuing into the core queue only after the class-specific enqueue callback has been called. This ensures that for CFS tasks, the updated vruntime value is used when enqueuing the task into the core rbtree.
--- kernel/sched/core.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
kernel/sched/core.c +3 -3
@@ -1753,9 +1753,6 @@ static inline void init_uclamp(void) { }static inline void enqueue_task (struct rq *rq,struct task_struct *p,int flags) { -if (sched_core_enabled (rq)) -sched_core_enqueue (rq, p); -if (!(flags &ENQUEUE_NOCLOCK ))update_rq_clock (rq);
@@ -1766,6 +1763,9 @@ static inline void enqueue_task(struct rq *rq, struct task_struct *p, int flags)uclamp_rq_inc (rq, p); p->sched_class->enqueue_task (rq, p, flags); + +if (sched_core_enabled (rq)) +sched_core_enqueue (rq, p); }static inline void dequeue_task (struct rq *rq,struct task_struct *p,int flags)
Signature · 1 line
2.29.2.299.gdc1121823c-goog
PATCH 12/32 sched: Simplify the core pick loop for optimized case
In reply to Joel Fernandes (Google)The core pick loop grew a lot of warts over time to support optimizations. Turns out that that directly doing a class pick before entering the core-wide pick is better for readability. Make the changes.
Since this is a relatively new patch, make it a separate patch so that it is easier to revert in case anyone reports an issue with it. Testing shows it to be working for me.
--- kernel/sched/core.c | 73 ++++++++++++++++----------------------------- 1 file changed, 26 insertions(+), 47 deletions(-)
kernel/sched/core.c +26 -47
@@ -5180,6 +5180,15 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)put_prev_task_balance (rq, prev, rf); smt_mask =cpu_smt_mask (cpu); + need_sync = !!rq->core->core_cookie; + +/* reset state */ + rq->core->core_cookie =0UL ; +if (rq->core->core_forceidle) { + need_sync = true; + fi_before = true; + rq->core->core_forceidle = false; + }/* * core->core_task_seq, core->core_pick_seq, rq->core_sched_seq
@@ -5192,16 +5201,25 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)* 'Fix' this by also increasing @task_seq for every pick. */ rq->core->core_task_seq++; - need_sync = !!rq->core->core_cookie; -/* reset state */ -reset: - rq->core->core_cookie =0UL ; -if (rq->core->core_forceidle) { +/* +* Optimize for common case where this CPU has no cookies +* and there are no cookied tasks running on siblings. +*/ +if (!need_sync) { +for_each_class (class) { + next = class->pick_task (rq); +if (next) +break ; + } + +if (!next->core_cookie) { + rq->core_pick =NULL ; + goto done; + } need_sync = true; - fi_before = true; - rq->core->core_forceidle = false; } +for_each_cpu (i, smt_mask) {struct rq *rq_i =cpu_rq (i);
@@ -5239,38 +5257,8 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)* core. */ p =pick_task (rq_i, class, max); -if (!p) { -/* -* If there weren't no cookies; we don't need to -* bother with the other siblings. -*/ -if (i == cpu && !need_sync) - goto next_class; - +if (!p)continue ; - } - -/* -* Optimize the 'normal' case where there aren't any -* cookies and we don't need to sync up. -*/ -if (i == cpu && !need_sync) { -if (p->core_cookie) { -/* -* This optimization is only valid as -* long as there are no cookies -* involved. We may have skipped -* non-empty higher priority classes on -* siblings, which are empty on this -* CPU, so start over. -*/ - need_sync = true; - goto reset; - } - - next = p; - goto done; - } rq_i->core_pick = p;
@@ -5298,18 +5286,9 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)cpu_rq (j)->core_pick =NULL ; } goto again; - }else { -/* -* Once we select a task for a cpu, we -* should not be doing an unconstrained -* pick because it might starve a task -* on a forced idle cpu. -*/ - need_sync = true; } } } -next_class:; } rq->core->core_pick_seq = rq->core->core_task_seq;
Signature · 1 line
2.29.2.299.gdc1121823c-goog
Peter Zijlstra
maintainerRe: [PATCH -tip 12/32] sched: Simplify the core pick loop for optimized case
Quoted 7 lines · Joel Fernandes (Google)
(quoted diff) +7
-0+ /* + * Optimize for common case where this CPU has no cookies + * and there are no cookied tasks running on siblings. + */ + if (!need_sync) { + for_each_class(class) { + next = class->pick_task(rq);
Review
Quoted 10 lines
(quoted diff) +8
-0+ if (next) + break; + } + + if (!next->core_cookie) { + rq->core_pick = NULL; + goto done; + } need_sync = true; }
This isn't what I send you here:
https://lkml.kernel.org/r/[email protected]
Specifically, you've lost the whole cfs-cgroup optimization.
What was wrong/not working with the below?
kernel/sched/core.c +5 -10
@@ -5225,8 +5227,6 @@ pick_next_task(struct rq *rq, struct tasreturn next; } -put_prev_task_balance (rq, prev, rf); - smt_mask =cpu_smt_mask (cpu); need_sync = !!rq->core->core_cookie;
@@ -5255,17 +5255,14 @@ pick_next_task(struct rq *rq, struct tas* and there are no cookied tasks running on siblings. */ if (!need_sync) { -for_each_class (class) { - next = class->pick_task (rq); -if (next) -break ; - } - + next =__pick_next_task (rq, prev, rf);if (!next->core_cookie) { rq->core_pick =NULL ; - goto done; +return next; } - need_sync = true; +put_prev_task (next); + }else { +put_prev_task_balance (rq, prev, rf); }for_each_cpu (i, smt_mask) {
Re: [PATCH -tip 12/32] sched: Simplify the core pick loop for optimized case
Hi Peter,
Quoted 21 lines · Peter Zijlstra
Quoted 7 lines · Joel Fernandes (Google)
(quoted diff) +7
-0+ /* + * Optimize for common case where this CPU has no cookies + * and there are no cookied tasks running on siblings. + */ + if (!need_sync) { + for_each_class(class) { + next = class->pick_task(rq);Review
Quoted 10 lines
(quoted diff) +8
-0+ if (next) + break; + } + + if (!next->core_cookie) { + rq->core_pick = NULL; + goto done; + } need_sync = true; }This isn't what I send you here:
Quoted 1 line
I had replied to it here with concerns about the effects of newly idle balancing not being reverseable, it was only a theoretical concern:
http://lore.kernel.org/r/[email protected]
Also I was trying to keep the logic the same as v8 for unconstrained pick (calling pick_task), considering that has been tested quite a bit.
Quoted 1 line
Specifically, you've lost the whole cfs-cgroup optimization.
Are you referring to this optimization in pick_next_task_fair() ?
/*
* Since we haven't yet done put_prev_entity and if the
* selected task
* is a different task than we started out with, try
* and touch the
* least amount of cfs_rqs.
*/
You are right, we wouldn't get that with just calling pick_task_fair(). We did not have this in v8 series either though.
Also, if the task is a cookied task, then I think you are doing more work with your patch due to the extra put_prev_task().
Quoted 1 line
What was wrong/not working with the below?
Other than the new idle balancing, IIRC it was also causing instability. Maybe we can considering this optimization in the future if that's Ok with you?
thanks,
- Joel
Quoted 34 lines
kernel/sched/core.c +5
-10@@ -5225,8 +5227,6 @@ pick_next_task(struct rq *rq, struct tasreturn next; } -put_prev_task_balance (rq, prev, rf); - smt_mask =cpu_smt_mask (cpu); need_sync = !!rq->core->core_cookie;@@ -5255,17 +5255,14 @@ pick_next_task(struct rq *rq, struct tas* and there are no cookied tasks running on siblings. */ if (!need_sync) { -for_each_class (class) { - next = class->pick_task (rq); -if (next) -break ; - } - + next =__pick_next_task (rq, prev, rf);if (!next->core_cookie) { rq->core_pick =NULL ; - goto done; +return next; } - need_sync = true; +put_prev_task (next); + }else { +put_prev_task_balance (rq, prev, rf); }for_each_cpu (i, smt_mask) {
Peter Zijlstra
maintainerRe: [PATCH -tip 12/32] sched: Simplify the core pick loop for optimized case
Quoted 24 lines · Joel Fernandes
Hi Peter,
Quoted 21 lines · Peter Zijlstra
Quoted 7 lines · Joel Fernandes (Google)
(quoted diff) +7
-0+ /* + * Optimize for common case where this CPU has no cookies + * and there are no cookied tasks running on siblings. + */ + if (!need_sync) { + for_each_class(class) { + next = class->pick_task(rq);Review
Quoted 10 lines
(quoted diff) +8
-0+ if (next) + break; + } + + if (!next->core_cookie) { + rq->core_pick = NULL; + goto done; + } need_sync = true; }This isn't what I send you here:
Quoted 5 lines
Quoted 1 line
I had replied to it here with concerns about the effects of newly idle balancing not being reverseable, it was only a theoretical concern:
http://lore.kernel.org/r/[email protected]
Gah, missed that. I don't think that matters much see:put_prev_task_balance() calling balance_fair().
Quoted 11 lines
Quoted 1 line
Specifically, you've lost the whole cfs-cgroup optimization.
Are you referring to this optimization in
pick_next_task_fair()?/*
* Since we haven't yet doneput_prev_entityand if the
* selected task
* is a different task than we started out with, try
* and touch the
* least amount ofcfs_rqs.
*/
Yep, that. The giant FAIR_GROUP_SCHED hunk. The thing that makes all of pick_next_task() more complicated than it really wants to be.
Quoted 5 lines
You are right, we wouldn't get that with just calling
pick_task_fair(). We did not have this in v8 series either though.Also, if the task is a cookied task, then I think you are doing more work with your patch due to the extra
put_prev_task().
Yes, but only if you mix cookie tasks with non-cookie tasks and schedule two non-cookie tasks back-to-back. I don't think we care overly much about that.
I think it makes more sense to ensure that if you have core-sched enabled on your machine and have a (core-aligned) parition with non-cookie tasks, scheduling has works as 'normal' as possible.
Quoted 5 lines
Quoted 1 line
What was wrong/not working with the below?
Other than the new idle balancing, IIRC it was also causing instability. Maybe we can considering this optimization in the future if that's Ok with you?
Hurmph.. you don't happen to remember what went splat?
PATCH 13/32 sched: Trivial forced-newidle balancer
In reply to Joel Fernandes (Google)From: Peter Zijlstra <[email protected]>
When a sibling is forced-idle to match the core-cookie; search for matching tasks to fill the core.
rcu_read_unlock() can incur an infrequent deadlock in sched_core_balance(). Fix this by using the RCU-sched flavor instead.
--- include/linux/sched.h | 1 + kernel/sched/core.c | 130 +++++++++++++++++++++++++++++++++++++++++- kernel/sched/idle.c | 1 + kernel/sched/sched.h | 6 ++ 4 files changed, 137 insertions(+), 1 deletion(-)
include/linux/sched.h +1 -0
@@ -688,6 +688,7 @@ struct task_struct {#ifdef CONFIG_SCHED_CORE struct rb_node core_node;unsigned long core_cookie; +unsigned int core_occupation;#endif #ifdef CONFIG_CGROUP_SCHED
kernel/sched/core.c +129 -1
167 lines not on this page · raw message
kernel/sched/idle.c +1 -0
@@ -404,6 +404,7 @@ static void set_next_task_idle(struct rq *rq, struct task_struct *next, bool fir {update_idle_core (rq);schedstat_inc (rq->sched_goidle); +queue_core_balance (rq); }#ifdef CONFIG_SMP
kernel/sched/sched.h +6 -0
@@ -1135,6 +1135,8 @@ static inline raw_spinlock_t *rq_lockp(struct rq *rq)bool cfs_prio_less (struct task_struct *a,struct task_struct *b); +extern void queue_core_balance (struct rq *rq); +#else /* !CONFIG_SCHED_CORE */ static inline bool sched_core_enabled (struct rq *rq)
@@ -1147,6 +1149,10 @@ static inline raw_spinlock_t *rq_lockp(struct rq *rq) return &rq->__lock; } +static inline void queue_core_balance (struct rq *rq) +{ +} +#endif /* CONFIG_SCHED_CORE */ #ifdef CONFIG_SCHED_SMT
Signature · 1 line
2.29.2.299.gdc1121823c-goog
Re: [PATCH -tip 13/32] sched: Trivial forced-newidle balancer
Quoted 8 lines · Joel Fernandes (Google)
From: Peter Zijlstra <[email protected]>
When a sibling is forced-idle to match the core-cookie; search for matching tasks to fill the core.
rcu_read_unlock()can incur an infrequent deadlock insched_core_balance(). Fix this by using the RCU-sched flavor instead.
...
kernel/sched/core.c pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf) · line 5407
Quoted 4 lines
kernel/sched/core.c +4
-0+ +if (p->core_occupation > dst->idle->core_occupation) + goto next; +
I am unable to understand this check, a comment or clarification in the changelog will help. I presume we are looking at either one or two cpus to define the core_occupation and we expect to match it against the destination CPU.
Balbir Singh.
Re: [PATCH -tip 13/32] sched: Trivial forced-newidle balancer
Quoted 10 lines · Balbir Singh
Quoted 8 lines · Joel Fernandes (Google)
From: Peter Zijlstra <[email protected]>
When a sibling is forced-idle to match the core-cookie; search for matching tasks to fill the core.
rcu_read_unlock()can incur an infrequent deadlock insched_core_balance(). Fix this by using the RCU-sched flavor instead....
Quoted 9 lines
kernel/sched/core.c pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf) · line 5407
Quoted 4 lines
kernel/sched/core.c +4
-0+ +if (p->core_occupation > dst->idle->core_occupation) + goto next; +I am unable to understand this check, a comment or clarification in the changelog will help. I presume we are looking at either one or two cpus to define the
core_occupationand we expect to match it against the destination CPU.
IIUC, this check prevents a task from keeping jumping among the cores forever.
For example, on a SMT2 platform:
- core0 runs taskA and taskB, core_occupation is 2
- core1 runs taskC, core_occupation is 1
Without this check, taskB could ping-pong between core0 and core1 by core load balance.
Thanks,
-Aubrey
Re: [PATCH -tip 13/32] sched: Trivial forced-newidle balancer
Quoted 21 lines · Aubrey
Quoted 10 lines · Balbir Singh
Quoted 8 lines · Joel Fernandes (Google)
From: Peter Zijlstra <[email protected]>
When a sibling is forced-idle to match the core-cookie; search for matching tasks to fill the core.
rcu_read_unlock()can incur an infrequent deadlock insched_core_balance(). Fix this by using the RCU-sched flavor instead....
Quoted 9 lines
kernel/sched/core.c pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf) · line 5407
Quoted 4 lines
kernel/sched/core.c +4
-0+ +if (p->core_occupation > dst->idle->core_occupation) + goto next; +I am unable to understand this check, a comment or clarification in the changelog will help. I presume we are looking at either one or two cpus to define the
core_occupationand we expect to match it against the destination CPU.
Quoted 8 lines
IIUC, this check prevents a task from keeping jumping among the cores forever.
For example, on a SMT2 platform:
- core0 runs taskA and taskB,core_occupationis 2
- core1 runs taskC,core_occupationis 1Without this check, taskB could ping-pong between core0 and core1 by core load balance.
But the comparison is p->core_occuption (as in tasks core occuptation, not sure what that means, can a task have a core_occupation of > 1?)
Balbir Singh.
Re: [PATCH -tip 13/32] sched: Trivial forced-newidle balancer
Quoted 28 lines · Balbir Singh
Quoted 27 lines · Aubrey
Quoted 10 lines · Balbir Singh
Quoted 8 lines · Joel Fernandes (Google)
From: Peter Zijlstra <[email protected]>
When a sibling is forced-idle to match the core-cookie; search for matching tasks to fill the core.
rcu_read_unlock()can incur an infrequent deadlock insched_core_balance(). Fix this by using the RCU-sched flavor instead....
Quoted 9 lines
kernel/sched/core.c pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf) · line 5407
Quoted 4 lines
kernel/sched/core.c +4
-0+ +if (p->core_occupation > dst->idle->core_occupation) + goto next; +I am unable to understand this check, a comment or clarification in the changelog will help. I presume we are looking at either one or two cpus to define the
core_occupationand we expect to match it against the destination CPU.IIUC, this check prevents a task from keeping jumping among the cores forever.
For example, on a SMT2 platform:
- core0 runs taskA and taskB,core_occupationis 2
- core1 runs taskC,core_occupationis 1
Quoted 6 lines
Quoted 2 lines
Without this check, taskB could ping-pong between core0 and core1 by core load balance.
But the comparison is
p->core_occuption(as in tasks core occuptation, not sure what that means, can a task have acore_occupationof > 1?)
p->core_occupation is assigned to the core occupation in the last pick_next_task. (so yes, it can have a > 1 core_occupation).
Thanks,
-Aubrey
Re: [PATCH -tip 13/32] sched: Trivial forced-newidle balancer
Quoted 32 lines · Aubrey
Quoted 31 lines · Balbir Singh
Quoted 29 lines · Aubrey
Quoted 10 lines · Balbir Singh
Quoted 8 lines · Joel Fernandes (Google)
From: Peter Zijlstra <[email protected]>
When a sibling is forced-idle to match the core-cookie; search for matching tasks to fill the core.
rcu_read_unlock()can incur an infrequent deadlock insched_core_balance(). Fix this by using the RCU-sched flavor instead....
Quoted 9 lines
kernel/sched/core.c pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf) · line 5407
Quoted 4 lines
kernel/sched/core.c +4
-0+ +if (p->core_occupation > dst->idle->core_occupation) + goto next; +I am unable to understand this check, a comment or clarification in the changelog will help. I presume we are looking at either one or two cpus to define the
core_occupationand we expect to match it against the destination CPU.IIUC, this check prevents a task from keeping jumping among the cores forever.
For example, on a SMT2 platform:
- core0 runs taskA and taskB,core_occupationis 2
- core1 runs taskC,core_occupationis 1Without this check, taskB could ping-pong between core0 and core1 by core load balance.
Quoted 7 lines
Quoted 3 lines
But the comparison is
p->core_occuption(as in tasks core occuptation, not sure what that means, can a task have acore_occupationof > 1?)
p->core_occupationis assigned to the core occupation in the lastpick_next_task. (so yes, it can have a > 1core_occupation).
Hmm.. I find that hard to interpret that. But I am happy to re-read the code again.
Balbir Singh.
PATCH 14/32 sched: migration changes for core scheduling
In reply to Joel Fernandes (Google)From: Aubrey Li <[email protected]>
- Don't migrate if there is a cookie mismatch
Load balance tries to move task from busiest CPU to the destination CPU. When core scheduling is enabled, if the task's cookie does not match with the destination CPU's core cookie, this task will be skipped by this CPU. This mitigates the forced idle time on the destination CPU.
- Select cookie matched idle CPU
In the fast path of task wakeup, select the first cookie matched idle CPU instead of the first idle CPU.
- Find cookie matched idlest CPU
In the slow path of task wakeup, find the idlest CPU whose core cookie matches with task's cookie
- Don't migrate task if cookie not match
For the NUMA load balance, don't migrate task to the CPU whose core cookie does not match with task's cookie
--- kernel/sched/fair.c | 64 ++++++++++++++++++++++++++++++++++++++++---- kernel/sched/sched.h | 29 ++++++++++++++++++++ 2 files changed, 88 insertions(+), 5 deletions(-)
kernel/sched/fair.c +59 -5
@@ -1921,6 +1921,15 @@ static void task_numa_find_cpu(struct task_numa_env *env,if (!cpumask_test_cpu (cpu, env->p->cpus_ptr))continue ; +#ifdef CONFIG_SCHED_CORE +/* +* Skip this cpu if source task's cookie does not match +* with CPU's core cookie. +*/ +if (!sched_core_cookie_match (cpu_rq (cpu), env->p)) +continue ; +#endif + env->dst_cpu = cpu;if (task_numa_compare (env, taskimp, groupimp, maymove))break ;
@@ -5867,11 +5876,17 @@ find_idlest_group_cpu(struct sched_group *group, struct task_struct *p, int this/* Traverse only the allowed CPUs */ for_each_cpu_and (i,sched_group_span (group), p->cpus_ptr) { +struct rq *rq =cpu_rq (i); + +#ifdef CONFIG_SCHED_CORE +if (!sched_core_cookie_match (rq, p)) +continue ; +#endif +if (sched_idle_cpu (i))return i;if (available_idle_cpu (i)) { -struct rq *rq =cpu_rq (i);struct cpuidle_state *idle =idle_get_state (rq);if (idle && idle->exit_latency < min_exit_latency) {/*
@@ -6129,8 +6144,18 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, int tfor_each_cpu_wrap (cpu, cpus, target) {if (!--nr)return -1 ; -if (available_idle_cpu (cpu) ||sched_idle_cpu (cpu)) -break ; + +if (available_idle_cpu (cpu) ||sched_idle_cpu (cpu)) { +#ifdef CONFIG_SCHED_CORE +/* +* If Core Scheduling is enabled, select this cpu +* only if the process cookie matches core cookie. +*/ +if (sched_core_enabled (cpu_rq (cpu)) && + p->core_cookie == cpu_rq(cpu)->core->core_cookie) +#endif +break ; + } } time =cpu_clock (this) - time;
@@ -7530,8 +7555,9 @@ int can_migrate_task(struct task_struct *p, struct lb_env *env)* We do not migrate tasks that are: * 1) throttled_lb_pair, or * 2) cannot be migrated to this CPU due to cpus_ptr, or -* 3) running (obviously), or -* 4) are cache-hot on their current CPU. +* 3) task's cookie does not match with this CPU's core cookie +* 4) running (obviously), or +* 5) are cache-hot on their current CPU. */ if (throttled_lb_pair (task_group (p), env->src_cpu, env->dst_cpu))return 0 ;
@@ -7566,6 +7592,15 @@ int can_migrate_task(struct task_struct *p, struct lb_env *env)return 0 ; } +#ifdef CONFIG_SCHED_CORE +/* +* Don't migrate task if the task's cookie does not match +* with the destination CPU's core cookie. +*/ +if (!sched_core_cookie_match (cpu_rq (env->dst_cpu), p)) +return 0 ; +#endif +/* Record that we found atleast one task that could run on dst_cpu */ env->flags &= ~LBF_ALL_PINNED ;
@@ -8792,6 +8827,25 @@ find_idlest_group(struct sched_domain *sd, struct task_struct *p, int this_cpu) p->cpus_ptr))continue ; +#ifdef CONFIG_SCHED_CORE +if (sched_core_enabled (cpu_rq (this_cpu))) { +int i =0 ; +bool cookie_match = false; + +for_each_cpu (i,sched_group_span (group)) { +struct rq *rq =cpu_rq (i); + +if (sched_core_cookie_match (rq, p)) { + cookie_match = true; +break ; + } + } +/* Skip over this group if no cookie matched */ +if (!cookie_match) +continue ; + } +#endif + local_group =cpumask_test_cpu (this_cpu,sched_group_span (group));
kernel/sched/sched.h +29 -0
@@ -1135,6 +1135,35 @@ static inline raw_spinlock_t *rq_lockp(struct rq *rq)bool cfs_prio_less (struct task_struct *a,struct task_struct *b); +/* +* Helper to check if the CPU's core cookie matches with the task's cookie +* when core scheduling is enabled. +* A special case is that the task's cookie always matches with CPU's core +* cookie if the CPU is in an idle core. +*/ +static inline bool sched_core_cookie_match (struct rq *rq,struct task_struct *p) +{ +bool idle_core = true; +int cpu; + +/* Ignore cookie match if core scheduler is not enabled on the CPU. */ +if (!sched_core_enabled (rq)) +return true; + +for_each_cpu (cpu,cpu_smt_mask (cpu_of (rq))) { +if (!available_idle_cpu (cpu)) { + idle_core = false; +break ; + } + } + +/* +* A CPU in an idle core is always the best choice for tasks with +* cookies. +*/ +return idle_core || rq->core->core_cookie == p->core_cookie; +} +extern void queue_core_balance (struct rq *rq);#else /* !CONFIG_SCHED_CORE */
Signature · 1 line
2.29.2.299.gdc1121823c-goog
Re: [PATCH -tip 14/32] sched: migration changes for core scheduling
Quoted 38 lines · Joel Fernandes (Google)
From: Aubrey Li <[email protected]>
- Don't migrate if there is a cookie mismatch
Load balance tries to move task from busiest CPU to the destination CPU. When core scheduling is enabled, if the task's cookie does not match with the destination CPU's core cookie, this task will be skipped by this CPU. This mitigates the forced idle time on the destination CPU.- Select cookie matched idle CPU
In the fast path of task wakeup, select the first cookie matched idle CPU instead of the first idle CPU.- Find cookie matched idlest CPU
In the slow path of task wakeup, find the idlest CPU whose core cookie matches with task's cookie- Don't migrate task if cookie not match
For the NUMA load balance, don't migrate task to the CPU whose core cookie does not match with task's cookieTested-by Julien Desfossez <[email protected]>Signed-off-by Aubrey Li <[email protected]>Signed-off-by Tim Chen <[email protected]>Signed-off-by Vineeth Remanan Pillai <[email protected]>Signed-off-by Joel Fernandes (Google) <[email protected]>--- kernel/sched/fair.c | 64 ++++++++++++++++++++++++++++++++++++++++---- kernel/sched/sched.h | 29 ++++++++++++++++++++ 2 files changed, 88 insertions(+), 5 deletions(-)kernel/sched/fair.c +0
-0@@ -1921,6 +1921,15 @@ static void task_numa_find_cpu(struct task_numa_env *env,if (!cpumask_test_cpu (cpu, env->p->cpus_ptr))continue ;
kernel/sched/fair.c static void task_numa_find_cpu(struct task_numa_env *env, · line 1932
Quoted 10 lines
kernel/sched/fair.c +9
-0+#ifdef CONFIG_SCHED_CORE +/* +* Skip this cpu if source task's cookie does not match +* with CPU's core cookie. +*/ +if (!sched_core_cookie_match (cpu_rq (cpu), env->p)) +continue ; +#endif +
Any reason this is under an #ifdef? In sched_core_cookie_match() won't the check for sched_core_enabled() do the right thing even when CONFIG_SCHED_CORE is not enabed?
Quoted 112 lines
kernel/sched/fair.c +50
-5env->dst_cpu = cpu;if (task_numa_compare (env, taskimp, groupimp, maymove))break ;@@ -5867,11 +5876,17 @@ find_idlest_group_cpu(struct sched_group *group, struct task_struct *p, int this/* Traverse only the allowed CPUs */ for_each_cpu_and (i,sched_group_span (group), p->cpus_ptr) { +struct rq *rq =cpu_rq (i); + +#ifdef CONFIG_SCHED_CORE +if (!sched_core_cookie_match (rq, p)) +continue ; +#endif +if (sched_idle_cpu (i))return i;if (available_idle_cpu (i)) { -struct rq *rq =cpu_rq (i);struct cpuidle_state *idle =idle_get_state (rq);if (idle && idle->exit_latency < min_exit_latency) {/* @@ -6129,8 +6144,18 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, int tfor_each_cpu_wrap (cpu, cpus, target) {if (!--nr)return -1 ; -if (available_idle_cpu (cpu) ||sched_idle_cpu (cpu)) -break ; + +if (available_idle_cpu (cpu) ||sched_idle_cpu (cpu)) { +#ifdef CONFIG_SCHED_CORE +/* +* If Core Scheduling is enabled, select this cpu +* only if the process cookie matches core cookie. +*/ +if (sched_core_enabled (cpu_rq (cpu)) && + p->core_cookie == cpu_rq(cpu)->core->core_cookie) +#endif +break ; + } } time =cpu_clock (this) - time;@@ -7530,8 +7555,9 @@ int can_migrate_task(struct task_struct *p, struct lb_env *env)* We do not migrate tasks that are: * 1) throttled_lb_pair, or * 2) cannot be migrated to this CPU due to cpus_ptr, or -* 3) running (obviously), or -* 4) are cache-hot on their current CPU. +* 3) task's cookie does not match with this CPU's core cookie +* 4) running (obviously), or +* 5) are cache-hot on their current CPU. */ if (throttled_lb_pair (task_group (p), env->src_cpu, env->dst_cpu))return 0 ;@@ -7566,6 +7592,15 @@ int can_migrate_task(struct task_struct *p, struct lb_env *env)return 0 ; } +#ifdef CONFIG_SCHED_CORE +/* +* Don't migrate task if the task's cookie does not match +* with the destination CPU's core cookie. +*/ +if (!sched_core_cookie_match (cpu_rq (env->dst_cpu), p)) +return 0 ; +#endif +/* Record that we found atleast one task that could run on dst_cpu */ env->flags &= ~LBF_ALL_PINNED ;@@ -8792,6 +8827,25 @@ find_idlest_group(struct sched_domain *sd, struct task_struct *p, int this_cpu) p->cpus_ptr))continue ; +#ifdef CONFIG_SCHED_CORE +if (sched_core_enabled (cpu_rq (this_cpu))) { +int i =0 ; +bool cookie_match = false; + +for_each_cpu (i,sched_group_span (group)) { +struct rq *rq =cpu_rq (i); + +if (sched_core_cookie_match (rq, p)) { + cookie_match = true; +break ; + } + } +/* Skip over this group if no cookie matched */ +if (!cookie_match) +continue ; + } +#endif + local_group =cpumask_test_cpu (this_cpu,sched_group_span (group));kernel/sched/sched.h +7
-0@@ -1135,6 +1135,35 @@ static inline raw_spinlock_t *rq_lockp(struct rq *rq)bool cfs_prio_less (struct task_struct *a,struct task_struct *b); +/* +* Helper to check if the CPU's core cookie matches with the task's cookie +* when core scheduling is enabled. +* A special case is that the task's cookie always matches with CPU's core +* cookie if the CPU is in an idle core. +*/ +static inline bool sched_core_cookie_match (struct rq *rq,struct task_struct *p)
kernel/sched/sched.h static inline raw_spinlock_t *rq_lockp(struct rq *rq) · line 1154
Quoted 10 lines
kernel/sched/sched.h +10
-0+{ +bool idle_core = true; +int cpu; + +/* Ignore cookie match if core scheduler is not enabled on the CPU. */ +if (!sched_core_enabled (rq)) +return true; + +for_each_cpu (cpu,cpu_smt_mask (cpu_of (rq))) { +if (!available_idle_cpu (cpu)) {
I was looking at this snippet and comparing this to is_core_idle(), the major difference is the check for vcpu_is_preempted(). Do we want to call the core as non idle if any vcpu was preempted on this CPU?
kernel/sched/sched.h static inline raw_spinlock_t *rq_lockp(struct rq *rq) · line 1166
Quoted 12 lines
kernel/sched/sched.h +12
-0+ idle_core = false; +break ; + } + } + +/* +* A CPU in an idle core is always the best choice for tasks with +* cookies. +*/ +return idle_core || rq->core->core_cookie == p->core_cookie; +} +
Balbir Singh.
Re: [PATCH -tip 14/32] sched: migration changes for core scheduling
Quoted 40 lines · Balbir Singh
Quoted 39 lines · Joel Fernandes (Google)
From: Aubrey Li <[email protected]>
- Don't migrate if there is a cookie mismatch
Load balance tries to move task from busiest CPU to the destination CPU. When core scheduling is enabled, if the task's cookie does not match with the destination CPU's core cookie, this task will be skipped by this CPU. This mitigates the forced idle time on the destination CPU.- Select cookie matched idle CPU
In the fast path of task wakeup, select the first cookie matched idle CPU instead of the first idle CPU.- Find cookie matched idlest CPU
In the slow path of task wakeup, find the idlest CPU whose core cookie matches with task's cookie- Don't migrate task if cookie not match
For the NUMA load balance, don't migrate task to the CPU whose core cookie does not match with task's cookieTested-by Julien Desfossez <[email protected]>Signed-off-by Aubrey Li <[email protected]>Signed-off-by Tim Chen <[email protected]>Signed-off-by Vineeth Remanan Pillai <[email protected]>Signed-off-by Joel Fernandes (Google) <[email protected]>--- kernel/sched/fair.c | 64 ++++++++++++++++++++++++++++++++++++++++---- kernel/sched/sched.h | 29 ++++++++++++++++++++ 2 files changed, 88 insertions(+), 5 deletions(-)kernel/sched/fair.c +0
-0@@ -1921,6 +1921,15 @@ static void task_numa_find_cpu(struct task_numa_env *env,if (!cpumask_test_cpu (cpu, env->p->cpus_ptr))continue ;
Quoted 13 lines
kernel/sched/fair.c static void task_numa_find_cpu(struct task_numa_env *env, · line 1932
Quoted 9 lines
kernel/sched/fair.c +9
-0+#ifdef CONFIG_SCHED_CORE +/* +* Skip this cpu if source task's cookie does not match +* with CPU's core cookie. +*/ +if (!sched_core_cookie_match (cpu_rq (cpu), env->p)) +continue ; +#endif +Any reason this is under an #ifdef? In
sched_core_cookie_match()won't the check forsched_core_enabled()do the right thing even whenCONFIG_SCHED_COREis not enabed?>
Yes, sched_core_enabled works properly when CONFIG_SCHED_CORE is not enabled. But when CONFIG_SCHED_CORE is not enabled, it does not make sense to leave a core scheduler specific function here even at compile time. Also, for the cases in hot path, this saves CPU cycles to avoid a judgment.
Quoted 118 lines
Quoted 118 lines
kernel/sched/fair.c +50
-5env->dst_cpu = cpu;if (task_numa_compare (env, taskimp, groupimp, maymove))break ;@@ -5867,11 +5876,17 @@ find_idlest_group_cpu(struct sched_group *group, struct task_struct *p, int this/* Traverse only the allowed CPUs */ for_each_cpu_and (i,sched_group_span (group), p->cpus_ptr) { +struct rq *rq =cpu_rq (i); + +#ifdef CONFIG_SCHED_CORE +if (!sched_core_cookie_match (rq, p)) +continue ; +#endif +if (sched_idle_cpu (i))return i;if (available_idle_cpu (i)) { -struct rq *rq =cpu_rq (i);struct cpuidle_state *idle =idle_get_state (rq);if (idle && idle->exit_latency < min_exit_latency) {/* @@ -6129,8 +6144,18 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, int tfor_each_cpu_wrap (cpu, cpus, target) {if (!--nr)return -1 ; -if (available_idle_cpu (cpu) ||sched_idle_cpu (cpu)) -break ; + +if (available_idle_cpu (cpu) ||sched_idle_cpu (cpu)) { +#ifdef CONFIG_SCHED_CORE +/* +* If Core Scheduling is enabled, select this cpu +* only if the process cookie matches core cookie. +*/ +if (sched_core_enabled (cpu_rq (cpu)) && + p->core_cookie == cpu_rq(cpu)->core->core_cookie) +#endif +break ; + } } time =cpu_clock (this) - time;@@ -7530,8 +7555,9 @@ int can_migrate_task(struct task_struct *p, struct lb_env *env)* We do not migrate tasks that are: * 1) throttled_lb_pair, or * 2) cannot be migrated to this CPU due to cpus_ptr, or -* 3) running (obviously), or -* 4) are cache-hot on their current CPU. +* 3) task's cookie does not match with this CPU's core cookie +* 4) running (obviously), or +* 5) are cache-hot on their current CPU. */ if (throttled_lb_pair (task_group (p), env->src_cpu, env->dst_cpu))return 0 ;@@ -7566,6 +7592,15 @@ int can_migrate_task(struct task_struct *p, struct lb_env *env)return 0 ; } +#ifdef CONFIG_SCHED_CORE +/* +* Don't migrate task if the task's cookie does not match +* with the destination CPU's core cookie. +*/ +if (!sched_core_cookie_match (cpu_rq (env->dst_cpu), p)) +return 0 ; +#endif +/* Record that we found atleast one task that could run on dst_cpu */ env->flags &= ~LBF_ALL_PINNED ;@@ -8792,6 +8827,25 @@ find_idlest_group(struct sched_domain *sd, struct task_struct *p, int this_cpu) p->cpus_ptr))continue ; +#ifdef CONFIG_SCHED_CORE +if (sched_core_enabled (cpu_rq (this_cpu))) { +int i =0 ; +bool cookie_match = false; + +for_each_cpu (i,sched_group_span (group)) { +struct rq *rq =cpu_rq (i); + +if (sched_core_cookie_match (rq, p)) { + cookie_match = true; +break ; + } + } +/* Skip over this group if no cookie matched */ +if (!cookie_match) +continue ; + } +#endif + local_group =cpumask_test_cpu (this_cpu,sched_group_span (group));kernel/sched/sched.h +13
-0@@ -1135,6 +1135,35 @@ static inline raw_spinlock_t *rq_lockp(struct rq *rq)bool cfs_prio_less (struct task_struct *a,struct task_struct *b); +/* +* Helper to check if the CPU's core cookie matches with the task's cookie +* when core scheduling is enabled. +* A special case is that the task's cookie always matches with CPU's core +* cookie if the CPU is in an idle core. +*/ +static inline bool sched_core_cookie_match (struct rq *rq,struct task_struct *p) +{ +bool idle_core = true; +int cpu; + +/* Ignore cookie match if core scheduler is not enabled on the CPU. */ +if (!sched_core_enabled (rq))
Quoted 8 lines
kernel/sched/sched.h static inline raw_spinlock_t *rq_lockp(struct rq *rq) · line 1154
Quoted 4 lines
kernel/sched/sched.h +4
-0+return true; + +for_each_cpu (cpu,cpu_smt_mask (cpu_of (rq))) { +if (!available_idle_cpu (cpu)) {I was looking at this snippet and comparing this to
is_core_idle(), the major difference is the check forvcpu_is_preempted(). Do we want to call the core as non idle if any vcpu was preempted on this CPU?
Yes, if there is a VCPU was preempted on this CPU, better not place task on this core as the VCPU may be holding a spinlock and wants to be executed again ASAP.
Thanks,
-Aubrey
Peter Zijlstra
maintainerRe: [PATCH -tip 14/32] sched: migration changes for core scheduling
Quoted 13 lines · Aubrey
Quoted 5 lines
Quoted 5 lines
kernel/sched/fair.c +5
-0+#ifdef CONFIG_SCHED_CORE +/* +* Skip this cpu if source task's cookie does not match +* with CPU's core cookie. +*/ Quoted 8 lines
kernel/sched/fair.c static void task_numa_find_cpu(struct task_numa_env *env, · line 1932
Quoted 4 lines
kernel/sched/fair.c +4
-0+if (!sched_core_cookie_match (cpu_rq (cpu), env->p)) +continue ; +#endif +Any reason this is under an #ifdef? In
sched_core_cookie_match()won't the check forsched_core_enabled()do the right thing even whenCONFIG_SCHED_COREis not enabed?>
Quoted 5 lines
Yes,
sched_core_enabledworks properly whenCONFIG_SCHED_COREis not enabled. But whenCONFIG_SCHED_COREis not enabled, it does not make sense to leave a core scheduler specific function here even at compile time. Also, for the cases in hot path, this saves CPU cycles to avoid a judgment.
No, that's nonsense. If it works, remove the #ifdef. Less (#ifdef) is more.
Quoted 16 lines
Quoted 7 lines
Quoted 7 lines
kernel/sched/sched.h +7
-0+static inline bool sched_core_cookie_match (struct rq *rq,struct task_struct *p) +{ +bool idle_core = true; +int cpu; + +/* Ignore cookie match if core scheduler is not enabled on the CPU. */ +if (!sched_core_enabled (rq))Quoted 8 lines
kernel/sched/sched.h static inline raw_spinlock_t *rq_lockp(struct rq *rq) · line 1154
Quoted 4 lines
kernel/sched/sched.h +4
-0+return true; + +for_each_cpu (cpu,cpu_smt_mask (cpu_of (rq))) { +if (!available_idle_cpu (cpu)) {I was looking at this snippet and comparing this to
is_core_idle(), the major difference is the check forvcpu_is_preempted(). Do we want to call the core as non idle if any vcpu was preempted on this CPU?
Quoted 3 lines
Yes, if there is a VCPU was preempted on this CPU, better not place task on this core as the VCPU may be holding a spinlock and wants to be executed again ASAP.
If you're doing core scheduling on vcpus, you deserve all the pain possible.
Li, Aubrey
Tested-by Julien DesfossezRe: [PATCH -tip 14/32] sched: migration changes for core scheduling
Quoted 14 lines · Peter Zijlstra
Quoted 13 lines · Aubrey
Quoted 13 lines
Review
Quoted 9 lines
(quoted diff) +9
-0+#ifdef CONFIG_SCHED_CORE + /* + * Skip this cpu if source task's cookie does not match + * with CPU's core cookie. + */ + if (!sched_core_cookie_match(cpu_rq(cpu), env->p)) + continue; +#endif +Any reason this is under an #ifdef? In
sched_core_cookie_match()won't the check forsched_core_enabled()do the right thing even whenCONFIG_SCHED_COREis not enabed?>
Quoted 9 lines
Quoted 5 lines
Yes,
sched_core_enabledworks properly whenCONFIG_SCHED_COREis not enabled. But whenCONFIG_SCHED_COREis not enabled, it does not make sense to leave a core scheduler specific function here even at compile time. Also, for the cases in hot path, this saves CPU cycles to avoid a judgment.No, that's nonsense. If it works, remove the #ifdef. Less (#ifdef) is more.
Okay, I pasted the refined patch here.
@Joel, please let me know if you want me to send it in a separated thread.
Thanks,
-Aubrey
======================================================================
From 18e4f4592c2a159fcbae637f3a422e37ad24cb5a Mon Sep 17 00:00:00 2001
From: Aubrey Li <[email protected]>
Date: Wed, 25 Nov 2020 02:43:46 +0000
Subject: [PATCH 14/33] sched: migration changes for core scheduling
- Don't migrate if there is a cookie mismatch
Load balance tries to move task from busiest CPU to the destination CPU. When core scheduling is enabled, if the task's cookie does not match with the destination CPU's core cookie, this task will be skipped by this CPU. This mitigates the forced idle time on the destination CPU.
- Select cookie matched idle CPU
In the fast path of task wakeup, select the first cookie matched idle CPU instead of the first idle CPU.
- Find cookie matched idlest CPU
In the slow path of task wakeup, find the idlest CPU whose core cookie matches with task's cookie
- Don't migrate task if cookie not match
For the NUMA load balance, don't migrate task to the CPU whose core cookie does not match with task's cookie
--- kernel/sched/fair.c | 58 ++++++++++++++++++++++++++++++++++++++++---- kernel/sched/sched.h | 33 +++++++++++++++++++++++++ 2 files changed, 86 insertions(+), 5 deletions(-)
kernel/sched/fair.c +53 -5
@@ -1921,6 +1921,13 @@ static void task_numa_find_cpu(struct task_numa_env *env,if (!cpumask_test_cpu (cpu, env->p->cpus_ptr))continue ; +/* +* Skip this cpu if source task's cookie does not match +* with CPU's core cookie. +*/ +if (!sched_core_cookie_match (cpu_rq (cpu), env->p)) +continue ; + env->dst_cpu = cpu;if (task_numa_compare (env, taskimp, groupimp, maymove))break ;
@@ -5867,11 +5874,15 @@ find_idlest_group_cpu(struct sched_group *group, struct task_struct *p, int this/* Traverse only the allowed CPUs */ for_each_cpu_and (i,sched_group_span (group), p->cpus_ptr) { +struct rq *rq =cpu_rq (i); + +if (!sched_core_cookie_match (rq, p)) +continue ; +if (sched_idle_cpu (i))return i;if (available_idle_cpu (i)) { -struct rq *rq =cpu_rq (i);struct cpuidle_state *idle =idle_get_state (rq);if (idle && idle->exit_latency < min_exit_latency) {/*
@@ -6129,8 +6140,18 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, int tfor_each_cpu_wrap (cpu, cpus, target) {if (!--nr)return -1 ; -if (available_idle_cpu (cpu) ||sched_idle_cpu (cpu)) -break ; + +if (available_idle_cpu (cpu) ||sched_idle_cpu (cpu)) { +#ifdef CONFIG_SCHED_CORE +/* +* If Core Scheduling is enabled, select this cpu +* only if the process cookie matches core cookie. +*/ +if (sched_core_enabled (cpu_rq (cpu)) && + p->core_cookie == cpu_rq(cpu)->core->core_cookie) +#endif +break ; + } } time =cpu_clock (this) - time;
@@ -7530,8 +7551,9 @@ int can_migrate_task(struct task_struct *p, struct lb_env *env)* We do not migrate tasks that are: * 1) throttled_lb_pair, or * 2) cannot be migrated to this CPU due to cpus_ptr, or -* 3) running (obviously), or -* 4) are cache-hot on their current CPU. +* 3) task's cookie does not match with this CPU's core cookie +* 4) running (obviously), or +* 5) are cache-hot on their current CPU. */ if (throttled_lb_pair (task_group (p), env->src_cpu, env->dst_cpu))return 0 ;
@@ -7566,6 +7588,13 @@ int can_migrate_task(struct task_struct *p, struct lb_env *env)return 0 ; } +/* +* Don't migrate task if the task's cookie does not match +* with the destination CPU's core cookie. +*/ +if (!sched_core_cookie_match (cpu_rq (env->dst_cpu), p)) +return 0 ; +/* Record that we found atleast one task that could run on dst_cpu */ env->flags &= ~LBF_ALL_PINNED ;
@@ -8792,6 +8821,25 @@ find_idlest_group(struct sched_domain *sd, struct task_struct *p, int this_cpu) p->cpus_ptr))continue ; +#ifdef CONFIG_SCHED_CORE +if (sched_core_enabled (cpu_rq (this_cpu))) { +int i =0 ; +bool cookie_match = false; + +for_each_cpu (i,sched_group_span (group)) { +struct rq *rq =cpu_rq (i); + +if (sched_core_cookie_match (rq, p)) { + cookie_match = true; +break ; + } + } +/* Skip over this group if no cookie matched */ +if (!cookie_match) +continue ; + } +#endif + local_group =cpumask_test_cpu (this_cpu,sched_group_span (group));
kernel/sched/sched.h +33 -0
@@ -1135,6 +1135,35 @@ static inline raw_spinlock_t *rq_lockp(struct rq *rq)bool cfs_prio_less (struct task_struct *a,struct task_struct *b); +/* +* Helper to check if the CPU's core cookie matches with the task's cookie +* when core scheduling is enabled. +* A special case is that the task's cookie always matches with CPU's core +* cookie if the CPU is in an idle core. +*/ +static inline bool sched_core_cookie_match (struct rq *rq,struct task_struct *p) +{ +bool idle_core = true; +int cpu; + +/* Ignore cookie match if core scheduler is not enabled on the CPU. */ +if (!sched_core_enabled (rq)) +return true; + +for_each_cpu (cpu,cpu_smt_mask (cpu_of (rq))) { +if (!available_idle_cpu (cpu)) { + idle_core = false; +break ; + } + } + +/* +* A CPU in an idle core is always the best choice for tasks with +* cookies. +*/ +return idle_core || rq->core->core_cookie == p->core_cookie; +} +extern void queue_core_balance (struct rq *rq);#else /* !CONFIG_SCHED_CORE */
@@ -1153,6 +1182,10 @@ static inline void queue_core_balance(struct rq *rq) { } +static inline bool sched_core_cookie_match (struct rq *rq,struct task_struct *p) +{ +return true; +}#endif /* CONFIG_SCHED_CORE */ #ifdef CONFIG_SCHED_SMT
Signature · 1 line
2.17.1
Re: [PATCH -tip 14/32] sched: migration changes for core scheduling
Quoted 21 lines · Aubrey
Quoted 20 lines · Peter Zijlstra
Quoted 18 lines · Aubrey
Quoted 5 lines
Quoted 5 lines
kernel/sched/fair.c +5
-0+#ifdef CONFIG_SCHED_CORE +/* +* Skip this cpu if source task's cookie does not match +* with CPU's core cookie. +*/ Quoted 8 lines
kernel/sched/fair.c static void task_numa_find_cpu(struct task_numa_env *env, · line 1930
Quoted 4 lines
kernel/sched/fair.c +4
-0+if (!sched_core_cookie_match (cpu_rq (cpu), env->p)) +continue ; +#endif +Any reason this is under an #ifdef? In
sched_core_cookie_match()won't the check forsched_core_enabled()do the right thing even whenCONFIG_SCHED_COREis not enabed?>Yes,
sched_core_enabledworks properly whenCONFIG_SCHED_COREis not enabled. But whenCONFIG_SCHED_COREis not enabled, it does not make sense to leave a core scheduler specific function here even at compile time. Also, for the cases in hot path, this saves CPU cycles to avoid a judgment.
Quoted 7 lines
Quoted 3 lines
No, that's nonsense. If it works, remove the #ifdef. Less (#ifdef) is more.
Okay, I pasted the refined patch here.
@Joel, please let me know if you want me to send it in a separated thread.
You still have a bunch of #ifdefs, can't we just do
#ifndef CONFIG_SCHED_CORE
static inline bool sched_core_enabled(struct rq *rq)
{
return false;
}#endif
and frankly I think even that is not needed because there is a jump label __sched_core_enabled that tells us if sched_core is enabled or not.
Balbir
Li, Aubrey
Tested-by Julien DesfossezRe: [PATCH -tip 14/32] sched: migration changes for core scheduling
Quoted 34 lines · Balbir Singh
Quoted 21 lines · Aubrey
Quoted 20 lines · Peter Zijlstra
Quoted 18 lines · Aubrey
Quoted 5 lines
Quoted 5 lines
(quoted diff) +5
-0+#ifdef CONFIG_SCHED_CORE + /* + * Skip this cpu if source task's cookie does not match + * with CPU's core cookie. + */Quoted 8 lines
Review
Quoted 4 lines
(quoted diff) +4
-0+ if (!sched_core_cookie_match(cpu_rq(cpu), env->p)) + continue; +#endif +Any reason this is under an #ifdef? In
sched_core_cookie_match()won't the check forsched_core_enabled()do the right thing even whenCONFIG_SCHED_COREis not enabed?>Yes,
sched_core_enabledworks properly whenCONFIG_SCHED_COREis not enabled. But whenCONFIG_SCHED_COREis not enabled, it does not make sense to leave a core scheduler specific function here even at compile time. Also, for the cases in hot path, this saves CPU cycles to avoid a judgment.Quoted 7 lines
Quoted 3 lines
No, that's nonsense. If it works, remove the #ifdef. Less (#ifdef) is more.
Okay, I pasted the refined patch here.
@Joel, please let me know if you want me to send it in a separated thread.You still have a bunch of #ifdefs, can't we just do
#ifndef
CONFIG_SCHED_CORE
static inline boolsched_core_enabled(struct rq *rq)
Quoted 8 lines
{ return false; }#endif
and frankly I think even that is not needed because there is a jump label
__sched_core_enabledthat tells us ifsched_coreis enabled or not.
Hmm..., I need another wrapper for CONFIG_SCHED_CORE specific variables. How about this one?
Thanks,
-Aubrey
From 61dac9067e66b5b9ea26c684c8c8235714bab38a Mon Sep 17 00:00:00 2001
From: Aubrey Li <[email protected]>
Date: Thu, 26 Nov 2020 03:08:04 +0000
Subject: [PATCH] sched: migration changes for core scheduling
- Don't migrate if there is a cookie mismatch
Load balance tries to move task from busiest CPU to the destination CPU. When core scheduling is enabled, if the task's cookie does not match with the destination CPU's core cookie, this task will be skipped by this CPU. This mitigates the forced idle time on the destination CPU.
- Select cookie matched idle CPU
In the fast path of task wakeup, select the first cookie matched idle CPU instead of the first idle CPU.
- Find cookie matched idlest CPU
In the slow path of task wakeup, find the idlest CPU whose core cookie matches with task's cookie
- Don't migrate task if cookie not match
For the NUMA load balance, don't migrate task to the CPU whose core cookie does not match with task's cookie
--- kernel/sched/fair.c | 57 ++++++++++++++++++++++++++++++++++++++++---- kernel/sched/sched.h | 43 +++++++++++++++++++++++++++++++++ 2 files changed, 95 insertions(+), 5 deletions(-)
kernel/sched/fair.c +52 -5
@@ -1921,6 +1921,13 @@ static void task_numa_find_cpu(struct task_numa_env *env,if (!cpumask_test_cpu (cpu, env->p->cpus_ptr))continue ; +/* +* Skip this cpu if source task's cookie does not match +* with CPU's core cookie. +*/ +if (!sched_core_cookie_match (cpu_rq (cpu), env->p)) +continue ; + env->dst_cpu = cpu;if (task_numa_compare (env, taskimp, groupimp, maymove))break ;
@@ -5867,11 +5874,15 @@ find_idlest_group_cpu(struct sched_group *group, struct task_struct *p, int this/* Traverse only the allowed CPUs */ for_each_cpu_and (i,sched_group_span (group), p->cpus_ptr) { +struct rq *rq =cpu_rq (i); + +if (!sched_core_cookie_match (rq, p)) +continue ; +if (sched_idle_cpu (i))return i;if (available_idle_cpu (i)) { -struct rq *rq =cpu_rq (i);struct cpuidle_state *idle =idle_get_state (rq);if (idle && idle->exit_latency < min_exit_latency) {/*
@@ -6129,8 +6140,19 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, int tfor_each_cpu_wrap (cpu, cpus, target) {if (!--nr)return -1 ; -if (available_idle_cpu (cpu) ||sched_idle_cpu (cpu)) -break ; + +if (available_idle_cpu (cpu) ||sched_idle_cpu (cpu)) { +/* +* If Core Scheduling is enabled, select this cpu +* only if the process cookie matches core cookie. +*/ +if (sched_core_enabled (cpu_rq (cpu))) { +if (__cookie_match (cpu_rq (cpu), p)) +break ; + }else { +break ; + } + } } time =cpu_clock (this) - time;
@@ -7530,8 +7552,9 @@ int can_migrate_task(struct task_struct *p, struct lb_env *env)* We do not migrate tasks that are: * 1) throttled_lb_pair, or * 2) cannot be migrated to this CPU due to cpus_ptr, or -* 3) running (obviously), or -* 4) are cache-hot on their current CPU. +* 3) task's cookie does not match with this CPU's core cookie +* 4) running (obviously), or +* 5) are cache-hot on their current CPU. */ if (throttled_lb_pair (task_group (p), env->src_cpu, env->dst_cpu))return 0 ;
@@ -7566,6 +7589,13 @@ int can_migrate_task(struct task_struct *p, struct lb_env *env)return 0 ; } +/* +* Don't migrate task if the task's cookie does not match +* with the destination CPU's core cookie. +*/ +if (!sched_core_cookie_match (cpu_rq (env->dst_cpu), p)) +return 0 ; +/* Record that we found atleast one task that could run on dst_cpu */ env->flags &= ~LBF_ALL_PINNED ;
@@ -8792,6 +8822,23 @@ find_idlest_group(struct sched_domain *sd, struct task_struct *p, int this_cpu) p->cpus_ptr))continue ; +if (sched_core_enabled (cpu_rq (this_cpu))) { +int i =0 ; +bool cookie_match = false; + +for_each_cpu (i,sched_group_span (group)) { +struct rq *rq =cpu_rq (i); + +if (sched_core_cookie_match (rq, p)) { + cookie_match = true; +break ; + } + } +/* Skip over this group if no cookie matched */ +if (!cookie_match) +continue ; + } + local_group =cpumask_test_cpu (this_cpu,sched_group_span (group));
kernel/sched/sched.h +43 -0
55 lines not on this page · raw message
Signature · 1 line
2.17.1
Re: [PATCH -tip 14/32] sched: migration changes for core scheduling
Quoted 40 lines · Aubrey
Quoted 39 lines · Balbir Singh
Quoted 21 lines · Aubrey
Quoted 20 lines · Peter Zijlstra
Quoted 18 lines · Aubrey
Quoted 5 lines
Quoted 5 lines
kernel/sched/fair.c +5
-0+#ifdef CONFIG_SCHED_CORE +/* +* Skip this cpu if source task's cookie does not match +* with CPU's core cookie. +*/ Quoted 8 lines
kernel/sched/fair.c static void task_numa_find_cpu(struct task_numa_env *env, · line 1930
Quoted 4 lines
kernel/sched/fair.c +4
-0+if (!sched_core_cookie_match (cpu_rq (cpu), env->p)) +continue ; +#endif +Any reason this is under an #ifdef? In
sched_core_cookie_match()won't the check forsched_core_enabled()do the right thing even whenCONFIG_SCHED_COREis not enabed?>Yes,
sched_core_enabledworks properly whenCONFIG_SCHED_COREis not enabled. But whenCONFIG_SCHED_COREis not enabled, it does not make sense to leave a core scheduler specific function here even at compile time. Also, for the cases in hot path, this saves CPU cycles to avoid a judgment.Quoted 7 lines
Quoted 3 lines
No, that's nonsense. If it works, remove the #ifdef. Less (#ifdef) is more.
Okay, I pasted the refined patch here.
@Joel, please let me know if you want me to send it in a separated thread.You still have a bunch of #ifdefs, can't we just do
#ifndef
CONFIG_SCHED_CORE
static inline boolsched_core_enabled(struct rq *rq){ return false; }#endif
Quoted 7 lines
Quoted 3 lines
and frankly I think even that is not needed because there is a jump label
__sched_core_enabledthat tells us ifsched_coreis enabled or not.Hmm..., I need another wrapper for
CONFIG_SCHED_COREspecific variables. How about this one?
Much better :)
Quoted 81 lines
Thanks,
-AubreyFrom
61dac9067e66b5b9ea26c684c8c8235714bab38aMon Sep 17 00:00:00 2001
From: Aubrey Li <[email protected]>
Date: Thu, 26 Nov 2020 03:08:04 +0000
Subject: [PATCH] sched: migration changes for core scheduling- Don't migrate if there is a cookie mismatch
Load balance tries to move task from busiest CPU to the destination CPU. When core scheduling is enabled, if the task's cookie does not match with the destination CPU's core cookie, this task will be skipped by this CPU. This mitigates the forced idle time on the destination CPU.- Select cookie matched idle CPU
In the fast path of task wakeup, select the first cookie matched idle CPU instead of the first idle CPU.- Find cookie matched idlest CPU
In the slow path of task wakeup, find the idlest CPU whose core cookie matches with task's cookie- Don't migrate task if cookie not match
For the NUMA load balance, don't migrate task to the CPU whose core cookie does not match with task's cookieTested-by Julien Desfossez <[email protected]>Signed-off-by Aubrey Li <[email protected]>Signed-off-by Tim Chen <[email protected]>Signed-off-by Vineeth Remanan Pillai <[email protected]>Signed-off-by Joel Fernandes (Google) <[email protected]>--- kernel/sched/fair.c | 57 ++++++++++++++++++++++++++++++++++++++++---- kernel/sched/sched.h | 43 +++++++++++++++++++++++++++++++++ 2 files changed, 95 insertions(+), 5 deletions(-)kernel/sched/fair.c +15
-3@@ -1921,6 +1921,13 @@ static void task_numa_find_cpu(struct task_numa_env *env,if (!cpumask_test_cpu (cpu, env->p->cpus_ptr))continue ; +/* +* Skip this cpu if source task's cookie does not match +* with CPU's core cookie. +*/ +if (!sched_core_cookie_match (cpu_rq (cpu), env->p)) +continue ; + env->dst_cpu = cpu;if (task_numa_compare (env, taskimp, groupimp, maymove))break ;@@ -5867,11 +5874,15 @@ find_idlest_group_cpu(struct sched_group *group, struct task_struct *p, int this/* Traverse only the allowed CPUs */ for_each_cpu_and (i,sched_group_span (group), p->cpus_ptr) { +struct rq *rq =cpu_rq (i); + +if (!sched_core_cookie_match (rq, p)) +continue ; +if (sched_idle_cpu (i))return i;if (available_idle_cpu (i)) { -struct rq *rq =cpu_rq (i);struct cpuidle_state *idle =idle_get_state (rq);if (idle && idle->exit_latency < min_exit_latency) {/* @@ -6129,8 +6140,19 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, int tfor_each_cpu_wrap (cpu, cpus, target) {if (!--nr)return -1 ; -if (available_idle_cpu (cpu) ||sched_idle_cpu (cpu)) -break ; + +if (available_idle_cpu (cpu) ||sched_idle_cpu (cpu)) { +/*
kernel/sched/fair.c static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, int t · line 6155
Quoted 10 lines
kernel/sched/fair.c +10
-0+* If Core Scheduling is enabled, select this cpu +* only if the process cookie matches core cookie. +*/ +if (sched_core_enabled (cpu_rq (cpu))) { +if (__cookie_match (cpu_rq (cpu), p)) +break ; + }else { +break ; + } + }
Isn't this better and equivalent?
if ((available_idle_cpu(cpu) || sched_idle_cpu(cpu)) && sched_core_cookie_match(cpu_rq(cpu), p)) break;
Quoted 40 lines
kernel/sched/fair.c +17
-2} time =cpu_clock (this) - time;@@ -7530,8 +7552,9 @@ int can_migrate_task(struct task_struct *p, struct lb_env *env)* We do not migrate tasks that are: * 1) throttled_lb_pair, or * 2) cannot be migrated to this CPU due to cpus_ptr, or -* 3) running (obviously), or -* 4) are cache-hot on their current CPU. +* 3) task's cookie does not match with this CPU's core cookie +* 4) running (obviously), or +* 5) are cache-hot on their current CPU. */ if (throttled_lb_pair (task_group (p), env->src_cpu, env->dst_cpu))return 0 ;@@ -7566,6 +7589,13 @@ int can_migrate_task(struct task_struct *p, struct lb_env *env)return 0 ; } +/* +* Don't migrate task if the task's cookie does not match +* with the destination CPU's core cookie. +*/ +if (!sched_core_cookie_match (cpu_rq (env->dst_cpu), p)) +return 0 ; +/* Record that we found atleast one task that could run on dst_cpu */ env->flags &= ~LBF_ALL_PINNED ;@@ -8792,6 +8822,23 @@ find_idlest_group(struct sched_domain *sd, struct task_struct *p, int this_cpu) p->cpus_ptr))continue ; +if (sched_core_enabled (cpu_rq (this_cpu))) { +int i =0 ; +bool cookie_match = false; + +for_each_cpu (i,sched_group_span (group)) { +struct rq *rq =cpu_rq (i); +
kernel/sched/fair.c find_idlest_group(struct sched_domain *sd, struct task_struct *p, int this_cpu) · line 8841
Quoted 10 lines
kernel/sched/fair.c +10
-0+if (sched_core_cookie_match (rq, p)) { + cookie_match = true; +break ; + } + } +/* Skip over this group if no cookie matched */ +if (!cookie_match) +continue ; + } +
Again, I think this can be refactored because sched_core_cookie_match checks for sched_core_enabled()
int i = 0;
bool cookie_match = false;
for_each_cpu(i, sched_group_span(group)) {
if (sched_core_cookie_match(cpu_rq(i), p))
break;
}
if (i >= nr_cpu_ids)
continue;Quoted 72 lines
kernel/sched/fair.c +4
-0+ } +/* Skip over this group if no cookie matched */ +if (!cookie_match) +continue ; local_group =cpumask_test_cpu (this_cpu,sched_group_span (group));kernel/sched/sched.h +43
-0@@ -1135,6 +1135,40 @@ static inline raw_spinlock_t *rq_lockp(struct rq *rq)bool cfs_prio_less (struct task_struct *a,struct task_struct *b); +/* +* Helper to check if the CPU's core cookie matches with the task's cookie +* when core scheduling is enabled. +* A special case is that the task's cookie always matches with CPU's core +* cookie if the CPU is in an idle core. +*/ +static inline bool __cookie_match (struct rq *rq,struct task_struct *p) +{ +return rq->core->core_cookie == p->core_cookie; +} + +static inline bool sched_core_cookie_match (struct rq *rq,struct task_struct *p) +{ +bool idle_core = true; +int cpu; + +/* Ignore cookie match if core scheduler is not enabled on the CPU. */ +if (!sched_core_enabled (rq)) +return true; + +for_each_cpu (cpu,cpu_smt_mask (cpu_of (rq))) { +if (!available_idle_cpu (cpu)) { + idle_core = false; +break ; + } + } + +/* +* A CPU in an idle core is always the best choice for tasks with +* cookies. +*/ +return idle_core ||__cookie_match (rq, p); +} +extern void queue_core_balance (struct rq *rq);#else /* !CONFIG_SCHED_CORE */ @@ -1153,6 +1187,15 @@ static inline void queue_core_balance(struct rq *rq) { } +static inline bool __cookie_match (struct rq *rq,struct task_struct *p) +{ +return true; +} + +static inline bool sched_core_cookie_match (struct rq *rq,struct task_struct *p) +{ +return true; +}#endif /* CONFIG_SCHED_CORE */ #ifdef CONFIG_SCHED_SMT Signature · 1 line
2.17.1
Balbir Singh.
Re: [PATCH -tip 14/32] sched: migration changes for core scheduling
Quoted 145 lines · Balbir Singh
Quoted 40 lines · Aubrey
Quoted 39 lines · Balbir Singh
Quoted 21 lines · Aubrey
Quoted 20 lines · Peter Zijlstra
Quoted 18 lines · Aubrey
Quoted 5 lines
Quoted 5 lines
kernel/sched/fair.c +5
-0+#ifdef CONFIG_SCHED_CORE +/* +* Skip this cpu if source task's cookie does not match +* with CPU's core cookie. +*/ Quoted 8 lines
kernel/sched/fair.c static void task_numa_find_cpu(struct task_numa_env *env, · line 1930
Quoted 4 lines
kernel/sched/fair.c +4
-0+if (!sched_core_cookie_match (cpu_rq (cpu), env->p)) +continue ; +#endif +Any reason this is under an #ifdef? In
sched_core_cookie_match()won't the check forsched_core_enabled()do the right thing even whenCONFIG_SCHED_COREis not enabed?>Yes,
sched_core_enabledworks properly whenCONFIG_SCHED_COREis not enabled. But whenCONFIG_SCHED_COREis not enabled, it does not make sense to leave a core scheduler specific function here even at compile time. Also, for the cases in hot path, this saves CPU cycles to avoid a judgment.Quoted 7 lines
Quoted 3 lines
No, that's nonsense. If it works, remove the #ifdef. Less (#ifdef) is more.
Okay, I pasted the refined patch here.
@Joel, please let me know if you want me to send it in a separated thread.You still have a bunch of #ifdefs, can't we just do
#ifndef
CONFIG_SCHED_CORE
static inline boolsched_core_enabled(struct rq *rq){ return false; }#endif
Quoted 7 lines
Quoted 3 lines
and frankly I think even that is not needed because there is a jump label
__sched_core_enabledthat tells us ifsched_coreis enabled or not.Hmm..., I need another wrapper for
CONFIG_SCHED_COREspecific variables. How about this one?Much better :)
Quoted 81 lines
Thanks,
-AubreyFrom
61dac9067e66b5b9ea26c684c8c8235714bab38aMon Sep 17 00:00:00 2001
From: Aubrey Li <[email protected]>
Date: Thu, 26 Nov 2020 03:08:04 +0000
Subject: [PATCH] sched: migration changes for core scheduling- Don't migrate if there is a cookie mismatch
Load balance tries to move task from busiest CPU to the destination CPU. When core scheduling is enabled, if the task's cookie does not match with the destination CPU's core cookie, this task will be skipped by this CPU. This mitigates the forced idle time on the destination CPU.- Select cookie matched idle CPU
In the fast path of task wakeup, select the first cookie matched idle CPU instead of the first idle CPU.- Find cookie matched idlest CPU
In the slow path of task wakeup, find the idlest CPU whose core cookie matches with task's cookie- Don't migrate task if cookie not match
For the NUMA load balance, don't migrate task to the CPU whose core cookie does not match with task's cookieTested-by Julien Desfossez <[email protected]>Signed-off-by Aubrey Li <[email protected]>Signed-off-by Tim Chen <[email protected]>Signed-off-by Vineeth Remanan Pillai <[email protected]>Signed-off-by Joel Fernandes (Google) <[email protected]>--- kernel/sched/fair.c | 57 ++++++++++++++++++++++++++++++++++++++++---- kernel/sched/sched.h | 43 +++++++++++++++++++++++++++++++++ 2 files changed, 95 insertions(+), 5 deletions(-)kernel/sched/fair.c +15
-3@@ -1921,6 +1921,13 @@ static void task_numa_find_cpu(struct task_numa_env *env,if (!cpumask_test_cpu (cpu, env->p->cpus_ptr))continue ; +/* +* Skip this cpu if source task's cookie does not match +* with CPU's core cookie. +*/ +if (!sched_core_cookie_match (cpu_rq (cpu), env->p)) +continue ; + env->dst_cpu = cpu;if (task_numa_compare (env, taskimp, groupimp, maymove))break ;@@ -5867,11 +5874,15 @@ find_idlest_group_cpu(struct sched_group *group, struct task_struct *p, int this/* Traverse only the allowed CPUs */ for_each_cpu_and (i,sched_group_span (group), p->cpus_ptr) { +struct rq *rq =cpu_rq (i); + +if (!sched_core_cookie_match (rq, p)) +continue ; +if (sched_idle_cpu (i))return i;if (available_idle_cpu (i)) { -struct rq *rq =cpu_rq (i);struct cpuidle_state *idle =idle_get_state (rq);if (idle && idle->exit_latency < min_exit_latency) {/* @@ -6129,8 +6140,19 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, int tfor_each_cpu_wrap (cpu, cpus, target) {if (!--nr)return -1 ; -if (available_idle_cpu (cpu) ||sched_idle_cpu (cpu)) -break ; + +if (available_idle_cpu (cpu) ||sched_idle_cpu (cpu)) { +/* kernel/sched/fair.c static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, int t · line 6155
Quoted 10 lines
kernel/sched/fair.c +10
-0+* If Core Scheduling is enabled, select this cpu +* only if the process cookie matches core cookie. +*/ +if (sched_core_enabled (cpu_rq (cpu))) { +if (__cookie_match (cpu_rq (cpu), p)) +break ; + }else { +break ; + } + }Isn't this better and equivalent?
Quoted 4 lines
if ((available_idle_cpu(cpu) || sched_idle_cpu(cpu)) && sched_core_cookie_match(cpu_rq(cpu), p)) break;
That's my previous implementation in the earlier version. But since here is the hot code path, we want to remove the idle core check in sched_core_cookie_match.
Quoted 54 lines
Quoted 40 lines
kernel/sched/fair.c +17
-2} time =cpu_clock (this) - time;@@ -7530,8 +7552,9 @@ int can_migrate_task(struct task_struct *p, struct lb_env *env)* We do not migrate tasks that are: * 1) throttled_lb_pair, or * 2) cannot be migrated to this CPU due to cpus_ptr, or -* 3) running (obviously), or -* 4) are cache-hot on their current CPU. +* 3) task's cookie does not match with this CPU's core cookie +* 4) running (obviously), or +* 5) are cache-hot on their current CPU. */ if (throttled_lb_pair (task_group (p), env->src_cpu, env->dst_cpu))return 0 ;@@ -7566,6 +7589,13 @@ int can_migrate_task(struct task_struct *p, struct lb_env *env)return 0 ; } +/* +* Don't migrate task if the task's cookie does not match +* with the destination CPU's core cookie. +*/ +if (!sched_core_cookie_match (cpu_rq (env->dst_cpu), p)) +return 0 ; +/* Record that we found atleast one task that could run on dst_cpu */ env->flags &= ~LBF_ALL_PINNED ;@@ -8792,6 +8822,23 @@ find_idlest_group(struct sched_domain *sd, struct task_struct *p, int this_cpu) p->cpus_ptr))continue ; +if (sched_core_enabled (cpu_rq (this_cpu))) { +int i =0 ; +bool cookie_match = false; + +for_each_cpu (i,sched_group_span (group)) { +struct rq *rq =cpu_rq (i); +kernel/sched/fair.c find_idlest_group(struct sched_domain *sd, struct task_struct *p, int this_cpu) · line 8841
Quoted 10 lines
kernel/sched/fair.c +10
-0+if (sched_core_cookie_match (rq, p)) { + cookie_match = true; +break ; + } + } +/* Skip over this group if no cookie matched */ +if (!cookie_match) +continue ; + } +Again, I think this can be refactored because
sched_core_cookie_matchchecks forsched_core_enabled()
Quoted 8 lines
int i = 0;
bool cookie_match = false;
for_each_cpu(i, sched_group_span(group)) {
if (sched_core_cookie_match(cpu_rq(i), p))
break;
}
if (i >= nr_cpu_ids)
continue;There is a loop here when CONFIG_SCHED_CORE=n, which is unwanted I guess.
Thanks,
-Aubrey
Re: [PATCH -tip 14/32] sched: migration changes for core scheduling
Quoted 152 lines · Aubrey
Quoted 145 lines · Balbir Singh
Quoted 40 lines · Aubrey
Quoted 39 lines · Balbir Singh
Quoted 21 lines · Aubrey
Quoted 20 lines · Peter Zijlstra
Quoted 18 lines · Aubrey
Quoted 5 lines
Quoted 5 lines
kernel/sched/fair.c +5
-0+#ifdef CONFIG_SCHED_CORE +/* +* Skip this cpu if source task's cookie does not match +* with CPU's core cookie. +*/ Quoted 8 lines
kernel/sched/fair.c static void task_numa_find_cpu(struct task_numa_env *env, · line 1930
Quoted 4 lines
kernel/sched/fair.c +4
-0+if (!sched_core_cookie_match (cpu_rq (cpu), env->p)) +continue ; +#endif +Any reason this is under an #ifdef? In
sched_core_cookie_match()won't the check forsched_core_enabled()do the right thing even whenCONFIG_SCHED_COREis not enabed?>Yes,
sched_core_enabledworks properly whenCONFIG_SCHED_COREis not enabled. But whenCONFIG_SCHED_COREis not enabled, it does not make sense to leave a core scheduler specific function here even at compile time. Also, for the cases in hot path, this saves CPU cycles to avoid a judgment.Quoted 7 lines
Quoted 3 lines
No, that's nonsense. If it works, remove the #ifdef. Less (#ifdef) is more.
Okay, I pasted the refined patch here.
@Joel, please let me know if you want me to send it in a separated thread.You still have a bunch of #ifdefs, can't we just do
#ifndef
CONFIG_SCHED_CORE
static inline boolsched_core_enabled(struct rq *rq){ return false; }#endif
Quoted 7 lines
Quoted 3 lines
and frankly I think even that is not needed because there is a jump label
__sched_core_enabledthat tells us ifsched_coreis enabled or not.Hmm..., I need another wrapper for
CONFIG_SCHED_COREspecific variables. How about this one?Much better :)
Quoted 81 lines
Thanks,
-AubreyFrom
61dac9067e66b5b9ea26c684c8c8235714bab38aMon Sep 17 00:00:00 2001
From: Aubrey Li <[email protected]>
Date: Thu, 26 Nov 2020 03:08:04 +0000
Subject: [PATCH] sched: migration changes for core scheduling- Don't migrate if there is a cookie mismatch
Load balance tries to move task from busiest CPU to the destination CPU. When core scheduling is enabled, if the task's cookie does not match with the destination CPU's core cookie, this task will be skipped by this CPU. This mitigates the forced idle time on the destination CPU.- Select cookie matched idle CPU
In the fast path of task wakeup, select the first cookie matched idle CPU instead of the first idle CPU.- Find cookie matched idlest CPU
In the slow path of task wakeup, find the idlest CPU whose core cookie matches with task's cookie- Don't migrate task if cookie not match
For the NUMA load balance, don't migrate task to the CPU whose core cookie does not match with task's cookieTested-by Julien Desfossez <[email protected]>Signed-off-by Aubrey Li <[email protected]>Signed-off-by Tim Chen <[email protected]>Signed-off-by Vineeth Remanan Pillai <[email protected]>Signed-off-by Joel Fernandes (Google) <[email protected]>--- kernel/sched/fair.c | 57 ++++++++++++++++++++++++++++++++++++++++---- kernel/sched/sched.h | 43 +++++++++++++++++++++++++++++++++ 2 files changed, 95 insertions(+), 5 deletions(-)kernel/sched/fair.c +15
-3@@ -1921,6 +1921,13 @@ static void task_numa_find_cpu(struct task_numa_env *env,if (!cpumask_test_cpu (cpu, env->p->cpus_ptr))continue ; +/* +* Skip this cpu if source task's cookie does not match +* with CPU's core cookie. +*/ +if (!sched_core_cookie_match (cpu_rq (cpu), env->p)) +continue ; + env->dst_cpu = cpu;if (task_numa_compare (env, taskimp, groupimp, maymove))break ;@@ -5867,11 +5874,15 @@ find_idlest_group_cpu(struct sched_group *group, struct task_struct *p, int this/* Traverse only the allowed CPUs */ for_each_cpu_and (i,sched_group_span (group), p->cpus_ptr) { +struct rq *rq =cpu_rq (i); + +if (!sched_core_cookie_match (rq, p)) +continue ; +if (sched_idle_cpu (i))return i;if (available_idle_cpu (i)) { -struct rq *rq =cpu_rq (i);struct cpuidle_state *idle =idle_get_state (rq);if (idle && idle->exit_latency < min_exit_latency) {/* @@ -6129,8 +6140,19 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, int tfor_each_cpu_wrap (cpu, cpus, target) {if (!--nr)return -1 ; -if (available_idle_cpu (cpu) ||sched_idle_cpu (cpu)) -break ; + +if (available_idle_cpu (cpu) ||sched_idle_cpu (cpu)) { +/* kernel/sched/fair.c static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, int t · line 6155
Quoted 10 lines
kernel/sched/fair.c +10
-0+* If Core Scheduling is enabled, select this cpu +* only if the process cookie matches core cookie. +*/ +if (sched_core_enabled (cpu_rq (cpu))) { +if (__cookie_match (cpu_rq (cpu), p)) +break ; + }else { +break ; + } + }Isn't this better and equivalent?
Quoted 4 lines
if ((available_idle_cpu(cpu) || sched_idle_cpu(cpu)) && sched_core_cookie_match(cpu_rq(cpu), p)) break;
Quoted 3 lines
That's my previous implementation in the earlier version. But since here is the hot code path, we want to remove the idle core check in
sched_core_cookie_match.
I see, so we basically need a jump label, if sched_core_cookie_match can be inlined with a check for sched_core_enabled() upfront, it might solve a lot of the concern, readability of this section of code is not the best.
Quoted 64 lines
Quoted 55 lines
Quoted 41 lines
kernel/sched/fair.c +17
-2} time =cpu_clock (this) - time;@@ -7530,8 +7552,9 @@ int can_migrate_task(struct task_struct *p, struct lb_env *env)* We do not migrate tasks that are: * 1) throttled_lb_pair, or * 2) cannot be migrated to this CPU due to cpus_ptr, or -* 3) running (obviously), or -* 4) are cache-hot on their current CPU. +* 3) task's cookie does not match with this CPU's core cookie +* 4) running (obviously), or +* 5) are cache-hot on their current CPU. */ if (throttled_lb_pair (task_group (p), env->src_cpu, env->dst_cpu))return 0 ;@@ -7566,6 +7589,13 @@ int can_migrate_task(struct task_struct *p, struct lb_env *env)return 0 ; } +/* +* Don't migrate task if the task's cookie does not match +* with the destination CPU's core cookie. +*/ +if (!sched_core_cookie_match (cpu_rq (env->dst_cpu), p)) +return 0 ; +/* Record that we found atleast one task that could run on dst_cpu */ env->flags &= ~LBF_ALL_PINNED ;@@ -8792,6 +8822,23 @@ find_idlest_group(struct sched_domain *sd, struct task_struct *p, int this_cpu) p->cpus_ptr))continue ; +if (sched_core_enabled (cpu_rq (this_cpu))) { +int i =0 ; +bool cookie_match = false; + +for_each_cpu (i,sched_group_span (group)) { +struct rq *rq =cpu_rq (i); +kernel/sched/fair.c find_idlest_group(struct sched_domain *sd, struct task_struct *p, int this_cpu) · line 8841
Quoted 10 lines
kernel/sched/fair.c +10
-0+if (sched_core_cookie_match (rq, p)) { + cookie_match = true; +break ; + } + } +/* Skip over this group if no cookie matched */ +if (!cookie_match) +continue ; + } +Again, I think this can be refactored because
sched_core_cookie_matchchecks forsched_core_enabled()Quoted 8 lines
int i = 0; bool cookie_match = false; for_each_cpu(i, sched_group_span(group)) { if (sched_core_cookie_match(cpu_rq(i), p)) break; } if (i >= nr_cpu_ids) continue;
Quoted 2 lines
There is a loop here when
CONFIG_SCHED_CORE=n, which is unwanted I guess.
Yes, potentially, may be abstract the for_each_cpu into a function and then optimize out the case for SCHED_CORE=n, I feel all the extra checks in the various places make the code hard to read.
Balbir
Re: [PATCH -tip 14/32] sched: migration changes for core scheduling
Quoted 226 lines · Balbir Singh
Quoted 152 lines · Aubrey
Quoted 145 lines · Balbir Singh
Quoted 40 lines · Aubrey
Quoted 39 lines · Balbir Singh
Quoted 21 lines · Aubrey
Quoted 20 lines · Peter Zijlstra
Quoted 18 lines · Aubrey
Quoted 5 lines
Quoted 5 lines
kernel/sched/fair.c +5
-0+#ifdef CONFIG_SCHED_CORE +/* +* Skip this cpu if source task's cookie does not match +* with CPU's core cookie. +*/ Quoted 8 lines
kernel/sched/fair.c static void task_numa_find_cpu(struct task_numa_env *env, · line 1930
Quoted 4 lines
kernel/sched/fair.c +4
-0+if (!sched_core_cookie_match (cpu_rq (cpu), env->p)) +continue ; +#endif +Any reason this is under an #ifdef? In
sched_core_cookie_match()won't the check forsched_core_enabled()do the right thing even whenCONFIG_SCHED_COREis not enabed?>Yes,
sched_core_enabledworks properly whenCONFIG_SCHED_COREis not enabled. But whenCONFIG_SCHED_COREis not enabled, it does not make sense to leave a core scheduler specific function here even at compile time. Also, for the cases in hot path, this saves CPU cycles to avoid a judgment.Quoted 7 lines
Quoted 3 lines
No, that's nonsense. If it works, remove the #ifdef. Less (#ifdef) is more.
Okay, I pasted the refined patch here.
@Joel, please let me know if you want me to send it in a separated thread.You still have a bunch of #ifdefs, can't we just do
#ifndef
CONFIG_SCHED_CORE
static inline boolsched_core_enabled(struct rq *rq){ return false; }#endif
Quoted 7 lines
Quoted 3 lines
and frankly I think even that is not needed because there is a jump label
__sched_core_enabledthat tells us ifsched_coreis enabled or not.Hmm..., I need another wrapper for
CONFIG_SCHED_COREspecific variables. How about this one?Much better :)
Quoted 81 lines
Thanks,
-AubreyFrom
61dac9067e66b5b9ea26c684c8c8235714bab38aMon Sep 17 00:00:00 2001
From: Aubrey Li <[email protected]>
Date: Thu, 26 Nov 2020 03:08:04 +0000
Subject: [PATCH] sched: migration changes for core scheduling- Don't migrate if there is a cookie mismatch
Load balance tries to move task from busiest CPU to the destination CPU. When core scheduling is enabled, if the task's cookie does not match with the destination CPU's core cookie, this task will be skipped by this CPU. This mitigates the forced idle time on the destination CPU.- Select cookie matched idle CPU
In the fast path of task wakeup, select the first cookie matched idle CPU instead of the first idle CPU.- Find cookie matched idlest CPU
In the slow path of task wakeup, find the idlest CPU whose core cookie matches with task's cookie- Don't migrate task if cookie not match
For the NUMA load balance, don't migrate task to the CPU whose core cookie does not match with task's cookieTested-by Julien Desfossez <[email protected]>Signed-off-by Aubrey Li <[email protected]>Signed-off-by Tim Chen <[email protected]>Signed-off-by Vineeth Remanan Pillai <[email protected]>Signed-off-by Joel Fernandes (Google) <[email protected]>--- kernel/sched/fair.c | 57 ++++++++++++++++++++++++++++++++++++++++---- kernel/sched/sched.h | 43 +++++++++++++++++++++++++++++++++ 2 files changed, 95 insertions(+), 5 deletions(-)kernel/sched/fair.c +15
-3@@ -1921,6 +1921,13 @@ static void task_numa_find_cpu(struct task_numa_env *env,if (!cpumask_test_cpu (cpu, env->p->cpus_ptr))continue ; +/* +* Skip this cpu if source task's cookie does not match +* with CPU's core cookie. +*/ +if (!sched_core_cookie_match (cpu_rq (cpu), env->p)) +continue ; + env->dst_cpu = cpu;if (task_numa_compare (env, taskimp, groupimp, maymove))break ;@@ -5867,11 +5874,15 @@ find_idlest_group_cpu(struct sched_group *group, struct task_struct *p, int this/* Traverse only the allowed CPUs */ for_each_cpu_and (i,sched_group_span (group), p->cpus_ptr) { +struct rq *rq =cpu_rq (i); + +if (!sched_core_cookie_match (rq, p)) +continue ; +if (sched_idle_cpu (i))return i;if (available_idle_cpu (i)) { -struct rq *rq =cpu_rq (i);struct cpuidle_state *idle =idle_get_state (rq);if (idle && idle->exit_latency < min_exit_latency) {/* @@ -6129,8 +6140,19 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, int tfor_each_cpu_wrap (cpu, cpus, target) {if (!--nr)return -1 ; -if (available_idle_cpu (cpu) ||sched_idle_cpu (cpu)) -break ; + +if (available_idle_cpu (cpu) ||sched_idle_cpu (cpu)) { +/* kernel/sched/fair.c static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, int t · line 6155
Quoted 10 lines
kernel/sched/fair.c +10
-0+* If Core Scheduling is enabled, select this cpu +* only if the process cookie matches core cookie. +*/ +if (sched_core_enabled (cpu_rq (cpu))) { +if (__cookie_match (cpu_rq (cpu), p)) +break ; + }else { +break ; + } + }Isn't this better and equivalent?
Quoted 4 lines
if ((available_idle_cpu(cpu) || sched_idle_cpu(cpu)) && sched_core_cookie_match(cpu_rq(cpu), p)) break;Quoted 3 lines
That's my previous implementation in the earlier version. But since here is the hot code path, we want to remove the idle core check in
sched_core_cookie_match.I see, so we basically need a jump label, if
sched_core_cookie_matchcan be inlined with a check forsched_core_enabled()upfront, it might solve a lot of the concern, readability of this section of code is not the best.Quoted 64 lines
Quoted 63 lines
Quoted 41 lines
kernel/sched/fair.c +17
-2} time =cpu_clock (this) - time;@@ -7530,8 +7552,9 @@ int can_migrate_task(struct task_struct *p, struct lb_env *env)* We do not migrate tasks that are: * 1) throttled_lb_pair, or * 2) cannot be migrated to this CPU due to cpus_ptr, or -* 3) running (obviously), or -* 4) are cache-hot on their current CPU. +* 3) task's cookie does not match with this CPU's core cookie +* 4) running (obviously), or +* 5) are cache-hot on their current CPU. */ if (throttled_lb_pair (task_group (p), env->src_cpu, env->dst_cpu))return 0 ;@@ -7566,6 +7589,13 @@ int can_migrate_task(struct task_struct *p, struct lb_env *env)return 0 ; } +/* +* Don't migrate task if the task's cookie does not match +* with the destination CPU's core cookie. +*/ +if (!sched_core_cookie_match (cpu_rq (env->dst_cpu), p)) +return 0 ; +/* Record that we found atleast one task that could run on dst_cpu */ env->flags &= ~LBF_ALL_PINNED ;@@ -8792,6 +8822,23 @@ find_idlest_group(struct sched_domain *sd, struct task_struct *p, int this_cpu) p->cpus_ptr))continue ; +if (sched_core_enabled (cpu_rq (this_cpu))) { +int i =0 ; +bool cookie_match = false; + +for_each_cpu (i,sched_group_span (group)) { +struct rq *rq =cpu_rq (i); +kernel/sched/fair.c find_idlest_group(struct sched_domain *sd, struct task_struct *p, int this_cpu) · line 8841
Quoted 10 lines
kernel/sched/fair.c +10
-0+if (sched_core_cookie_match (rq, p)) { + cookie_match = true; +break ; + } + } +/* Skip over this group if no cookie matched */ +if (!cookie_match) +continue ; + } +Again, I think this can be refactored because
sched_core_cookie_matchchecks forsched_core_enabled()int i = 0; bool cookie_match = false; for_each_cpu(i, sched_group_span(group)) { if (sched_core_cookie_match(cpu_rq(i), p)) break; } if (i >= nr_cpu_ids) continue;
Quoted 6 lines
Quoted 2 lines
There is a loop here when
CONFIG_SCHED_CORE=n, which is unwanted I guess.Yes, potentially, may be abstract the
for_each_cpuinto a function and then optimize out the case for SCHED_CORE=n, I feel all the extra checks in the various places make the code hard to read.
Okay, I see your point, let me try if I can make it better.
Thanks,
-Aubrey
Li, Aubrey
Tested-by Julien DesfossezRe: [PATCH -tip 14/32] sched: migration changes for core scheduling
Hi Balbir,
I still placed the patch embedded in this thread, welcome any comments.
Thanks,
-Aubrey
======================================================================
From d64455dcaf47329673903a68a9df1151400cdd7a Mon Sep 17 00:00:00 2001
From: Aubrey Li <[email protected]>
Date: Wed, 2 Dec 2020 13:53:30 +0000
Subject: [PATCH] sched: migration changes for core scheduling
- Don't migrate if there is a cookie mismatch
Load balance tries to move task from busiest CPU to the destination CPU. When core scheduling is enabled, if the task's cookie does not match with the destination CPU's core cookie, this task will be skipped by this CPU. This mitigates the forced idle time on the destination CPU.
- Select cookie matched idle CPU
In the fast path of task wakeup, select the first cookie matched idle CPU instead of the first idle CPU.
- Find cookie matched idlest CPU
In the slow path of task wakeup, find the idlest CPU whose core cookie matches with task's cookie
- Don't migrate task if cookie not match
For the NUMA load balance, don't migrate task to the CPU whose core cookie does not match with task's cookie
--- kernel/sched/fair.c | 33 +++++++++++++++++--- kernel/sched/sched.h | 71 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 100 insertions(+), 4 deletions(-)
kernel/sched/fair.c +29 -4
@@ -1921,6 +1921,13 @@ static void task_numa_find_cpu(struct task_numa_env *env,if (!cpumask_test_cpu (cpu, env->p->cpus_ptr))continue ; +/* +* Skip this cpu if source task's cookie does not match +* with CPU's core cookie. +*/ +if (!sched_core_cookie_match (cpu_rq (cpu), env->p)) +continue ; + env->dst_cpu = cpu;if (task_numa_compare (env, taskimp, groupimp, maymove))break ;
@@ -5867,11 +5874,15 @@ find_idlest_group_cpu(struct sched_group *group, struct task_struct *p, int this/* Traverse only the allowed CPUs */ for_each_cpu_and (i,sched_group_span (group), p->cpus_ptr) { +struct rq *rq =cpu_rq (i); + +if (!sched_core_cookie_match (rq, p)) +continue ; +if (sched_idle_cpu (i))return i;if (available_idle_cpu (i)) { -struct rq *rq =cpu_rq (i);struct cpuidle_state *idle =idle_get_state (rq);if (idle && idle->exit_latency < min_exit_latency) {/*
@@ -6129,7 +6140,9 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, int tfor_each_cpu_wrap (cpu, cpus, target) {if (!--nr)return -1 ; -if (available_idle_cpu (cpu) ||sched_idle_cpu (cpu)) + +if (available_idle_cpu (cpu) ||sched_idle_cpu (cpu) && +sched_cpu_cookie_match (cpu_rq (cpu), p))break ; }
@@ -7530,8 +7543,9 @@ int can_migrate_task(struct task_struct *p, struct lb_env *env)* We do not migrate tasks that are: * 1) throttled_lb_pair, or * 2) cannot be migrated to this CPU due to cpus_ptr, or -* 3) running (obviously), or -* 4) are cache-hot on their current CPU. +* 3) task's cookie does not match with this CPU's core cookie +* 4) running (obviously), or +* 5) are cache-hot on their current CPU. */ if (throttled_lb_pair (task_group (p), env->src_cpu, env->dst_cpu))return 0 ;
@@ -7566,6 +7580,13 @@ int can_migrate_task(struct task_struct *p, struct lb_env *env)return 0 ; } +/* +* Don't migrate task if the task's cookie does not match +* with the destination CPU's core cookie. +*/ +if (!sched_core_cookie_match (cpu_rq (env->dst_cpu), p)) +return 0 ; +/* Record that we found atleast one task that could run on dst_cpu */ env->flags &= ~LBF_ALL_PINNED ;
@@ -8792,6 +8813,10 @@ find_idlest_group(struct sched_domain *sd, struct task_struct *p, int this_cpu) p->cpus_ptr))continue ; +/* Skip over this group if no cookie matched */ +if (!sched_group_cookie_match (cpu_rq (this_cpu), p, group)) +continue ; + local_group =cpumask_test_cpu (this_cpu,sched_group_span (group));
kernel/sched/sched.h +71 -0
83 lines not on this page · raw message
Signature · 1 line
2.17.1
Li, Aubrey
Tested-by Julien DesfossezRe: [PATCH -tip 14/32] sched: migration changes for core scheduling
Quoted 3 lines · Aubrey
Hi Balbir,
I still placed the patch embedded in this thread, welcome any comments.
Sorry, this version needs more work, refined as below, and I realized I should place a version number to the patch, start from v2 now.
Thanks,
-Aubrey
======================================================================
From aff2919889635aa9311d15bac3e949af0300ddc1 Mon Sep 17 00:00:00 2001
From: Aubrey Li <[email protected]>
Date: Thu, 3 Dec 2020 00:51:18 +0000
Subject: [PATCH v2] sched: migration changes for core scheduling
- Don't migrate if there is a cookie mismatch
Load balance tries to move task from busiest CPU to the destination CPU. When core scheduling is enabled, if the task's cookie does not match with the destination CPU's core cookie, this task will be skipped by this CPU. This mitigates the forced idle time on the destination CPU.
- Select cookie matched idle CPU
In the fast path of task wakeup, select the first cookie matched idle CPU instead of the first idle CPU.
- Find cookie matched idlest CPU
In the slow path of task wakeup, find the idlest CPU whose core cookie matches with task's cookie
- Don't migrate task if cookie not match
For the NUMA load balance, don't migrate task to the CPU whose core cookie does not match with task's cookie
--- kernel/sched/fair.c | 33 +++++++++++++++++--- kernel/sched/sched.h | 72 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 101 insertions(+), 4 deletions(-)
kernel/sched/fair.c +29 -4
@@ -1921,6 +1921,13 @@ static void task_numa_find_cpu(struct task_numa_env *env,if (!cpumask_test_cpu (cpu, env->p->cpus_ptr))continue ; +/* +* Skip this cpu if source task's cookie does not match +* with CPU's core cookie. +*/ +if (!sched_core_cookie_match (cpu_rq (cpu), env->p)) +continue ; + env->dst_cpu = cpu;if (task_numa_compare (env, taskimp, groupimp, maymove))break ;
@@ -5867,11 +5874,15 @@ find_idlest_group_cpu(struct sched_group *group, struct task_struct *p, int this/* Traverse only the allowed CPUs */ for_each_cpu_and (i,sched_group_span (group), p->cpus_ptr) { +struct rq *rq =cpu_rq (i); + +if (!sched_core_cookie_match (rq, p)) +continue ; +if (sched_idle_cpu (i))return i;if (available_idle_cpu (i)) { -struct rq *rq =cpu_rq (i);struct cpuidle_state *idle =idle_get_state (rq);if (idle && idle->exit_latency < min_exit_latency) {/*
@@ -6129,7 +6140,9 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, int tfor_each_cpu_wrap (cpu, cpus, target) {if (!--nr)return -1 ; -if (available_idle_cpu (cpu) ||sched_idle_cpu (cpu)) + +if ((available_idle_cpu (cpu) ||sched_idle_cpu (cpu)) && +sched_cpu_cookie_match (cpu_rq (cpu), p))break ; }
@@ -7530,8 +7543,9 @@ int can_migrate_task(struct task_struct *p, struct lb_env *env)* We do not migrate tasks that are: * 1) throttled_lb_pair, or * 2) cannot be migrated to this CPU due to cpus_ptr, or -* 3) running (obviously), or -* 4) are cache-hot on their current CPU. +* 3) task's cookie does not match with this CPU's core cookie +* 4) running (obviously), or +* 5) are cache-hot on their current CPU. */ if (throttled_lb_pair (task_group (p), env->src_cpu, env->dst_cpu))return 0 ;
@@ -7566,6 +7580,13 @@ int can_migrate_task(struct task_struct *p, struct lb_env *env)return 0 ; } +/* +* Don't migrate task if the task's cookie does not match +* with the destination CPU's core cookie. +*/ +if (!sched_core_cookie_match (cpu_rq (env->dst_cpu), p)) +return 0 ; +/* Record that we found atleast one task that could run on dst_cpu */ env->flags &= ~LBF_ALL_PINNED ;
@@ -8792,6 +8813,10 @@ find_idlest_group(struct sched_domain *sd, struct task_struct *p, int this_cpu) p->cpus_ptr))continue ; +/* Skip over this group if no cookie matched */ +if (!sched_group_cookie_match (cpu_rq (this_cpu), p, group)) +continue ; + local_group =cpumask_test_cpu (this_cpu,sched_group_span (group));
kernel/sched/sched.h +72 -0
90 lines not on this page · raw message
Signature · 1 line
2.17.1
Vincent Guittot
maintainerRe: [PATCH -tip 14/32] sched: migration changes for core scheduling
In reply to Joel Fernandes (Google)Quoted 79 lines · Joel Fernandes (Google)
From: Aubrey Li <[email protected]>
- Don't migrate if there is a cookie mismatch
Load balance tries to move task from busiest CPU to the destination CPU. When core scheduling is enabled, if the task's cookie does not match with the destination CPU's core cookie, this task will be skipped by this CPU. This mitigates the forced idle time on the destination CPU.- Select cookie matched idle CPU
In the fast path of task wakeup, select the first cookie matched idle CPU instead of the first idle CPU.- Find cookie matched idlest CPU
In the slow path of task wakeup, find the idlest CPU whose core cookie matches with task's cookie- Don't migrate task if cookie not match
For the NUMA load balance, don't migrate task to the CPU whose core cookie does not match with task's cookieTested-by Julien Desfossez <[email protected]>Signed-off-by Aubrey Li <[email protected]>Signed-off-by Tim Chen <[email protected]>Signed-off-by Vineeth Remanan Pillai <[email protected]>Signed-off-by Joel Fernandes (Google) <[email protected]>--- kernel/sched/fair.c | 64 ++++++++++++++++++++++++++++++++++++++++---- kernel/sched/sched.h | 29 ++++++++++++++++++++ 2 files changed, 88 insertions(+), 5 deletions(-)kernel/sched/fair.c +18
-3@@ -1921,6 +1921,15 @@ static void task_numa_find_cpu(struct task_numa_env *env,if (!cpumask_test_cpu (cpu, env->p->cpus_ptr))continue ; +#ifdef CONFIG_SCHED_CORE +/* +* Skip this cpu if source task's cookie does not match +* with CPU's core cookie. +*/ +if (!sched_core_cookie_match (cpu_rq (cpu), env->p)) +continue ; +#endif + env->dst_cpu = cpu;if (task_numa_compare (env, taskimp, groupimp, maymove))break ;@@ -5867,11 +5876,17 @@ find_idlest_group_cpu(struct sched_group *group, struct task_struct *p, int this/* Traverse only the allowed CPUs */ for_each_cpu_and (i,sched_group_span (group), p->cpus_ptr) { +struct rq *rq =cpu_rq (i); + +#ifdef CONFIG_SCHED_CORE +if (!sched_core_cookie_match (rq, p)) +continue ; +#endif +if (sched_idle_cpu (i))return i;if (available_idle_cpu (i)) { -struct rq *rq =cpu_rq (i);struct cpuidle_state *idle =idle_get_state (rq);if (idle && idle->exit_latency < min_exit_latency) {/* @@ -6129,8 +6144,18 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, int tfor_each_cpu_wrap (cpu, cpus, target) {if (!--nr)return -1 ; -if (available_idle_cpu (cpu) ||sched_idle_cpu (cpu)) -break ; + +if (available_idle_cpu (cpu) ||sched_idle_cpu (cpu)) {
kernel/sched/fair.c static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, int t · line 6158
Quoted 10 lines
kernel/sched/fair.c +10
-0+#ifdef CONFIG_SCHED_CORE +/* +* If Core Scheduling is enabled, select this cpu +* only if the process cookie matches core cookie. +*/ +if (sched_core_enabled (cpu_rq (cpu)) && + p->core_cookie == cpu_rq(cpu)->core->core_cookie) +#endif +break ; + }
This makes code unreadable.
Put this coresched specific stuff in an inline function; You can have a look at what is done with asym_fits_capacity()
Quoted 43 lines
kernel/sched/fair.c +20
-2} time =cpu_clock (this) - time;@@ -7530,8 +7555,9 @@ int can_migrate_task(struct task_struct *p, struct lb_env *env)* We do not migrate tasks that are: * 1) throttled_lb_pair, or * 2) cannot be migrated to this CPU due to cpus_ptr, or -* 3) running (obviously), or -* 4) are cache-hot on their current CPU. +* 3) task's cookie does not match with this CPU's core cookie +* 4) running (obviously), or +* 5) are cache-hot on their current CPU. */ if (throttled_lb_pair (task_group (p), env->src_cpu, env->dst_cpu))return 0 ;@@ -7566,6 +7592,15 @@ int can_migrate_task(struct task_struct *p, struct lb_env *env)return 0 ; } +#ifdef CONFIG_SCHED_CORE +/* +* Don't migrate task if the task's cookie does not match +* with the destination CPU's core cookie. +*/ +if (!sched_core_cookie_match (cpu_rq (env->dst_cpu), p)) +return 0 ; +#endif +/* Record that we found atleast one task that could run on dst_cpu */ env->flags &= ~LBF_ALL_PINNED ;@@ -8792,6 +8827,25 @@ find_idlest_group(struct sched_domain *sd, struct task_struct *p, int this_cpu) p->cpus_ptr))continue ; +#ifdef CONFIG_SCHED_CORE +if (sched_core_enabled (cpu_rq (this_cpu))) { +int i =0 ; +bool cookie_match = false; + +for_each_cpu (i,sched_group_span (group)) { +struct rq *rq =cpu_rq (i); +
kernel/sched/fair.c find_idlest_group(struct sched_domain *sd, struct task_struct *p, int this_cpu) · line 8847
Quoted 10 lines
kernel/sched/fair.c +10
-0+if (sched_core_cookie_match (rq, p)) { + cookie_match = true; +break ; + } + } +/* Skip over this group if no cookie matched */ +if (!cookie_match) +continue ; + } +#endif
same here, encapsulate this to keep find_idlest_group readable
Quoted 47 lines
kernel/sched/fair.c +1
-0+ local_group =cpumask_test_cpu (this_cpu,sched_group_span (group));kernel/sched/sched.h +29
-1@@ -1135,6 +1135,35 @@ static inline raw_spinlock_t *rq_lockp(struct rq *rq)bool cfs_prio_less (struct task_struct *a,struct task_struct *b); +/* +* Helper to check if the CPU's core cookie matches with the task's cookie +* when core scheduling is enabled. +* A special case is that the task's cookie always matches with CPU's core +* cookie if the CPU is in an idle core. +*/ +static inline bool sched_core_cookie_match (struct rq *rq,struct task_struct *p) +{ +bool idle_core = true; +int cpu; + +/* Ignore cookie match if core scheduler is not enabled on the CPU. */ +if (!sched_core_enabled (rq)) +return true; + +for_each_cpu (cpu,cpu_smt_mask (cpu_of (rq))) { +if (!available_idle_cpu (cpu)) { + idle_core = false; +break ; + } + } + +/* +* A CPU in an idle core is always the best choice for tasks with +* cookies. +*/ +return idle_core || rq->core->core_cookie == p->core_cookie; +} +extern void queue_core_balance (struct rq *rq);#else /* !CONFIG_SCHED_CORE */ --2.29.2.299.gdc1121823c-goog
Re: [PATCH -tip 14/32] sched: migration changes for core scheduling
Quoted 87 lines · Vincent Guittot
Quoted 85 lines · Joel Fernandes (Google)
From: Aubrey Li <[email protected]>
- Don't migrate if there is a cookie mismatch
Load balance tries to move task from busiest CPU to the destination CPU. When core scheduling is enabled, if the task's cookie does not match with the destination CPU's core cookie, this task will be skipped by this CPU. This mitigates the forced idle time on the destination CPU.- Select cookie matched idle CPU
In the fast path of task wakeup, select the first cookie matched idle CPU instead of the first idle CPU.- Find cookie matched idlest CPU
In the slow path of task wakeup, find the idlest CPU whose core cookie matches with task's cookie- Don't migrate task if cookie not match
For the NUMA load balance, don't migrate task to the CPU whose core cookie does not match with task's cookieTested-by Julien Desfossez <[email protected]>Signed-off-by Aubrey Li <[email protected]>Signed-off-by Tim Chen <[email protected]>Signed-off-by Vineeth Remanan Pillai <[email protected]>Signed-off-by Joel Fernandes (Google) <[email protected]>--- kernel/sched/fair.c | 64 ++++++++++++++++++++++++++++++++++++++++---- kernel/sched/sched.h | 29 ++++++++++++++++++++ 2 files changed, 88 insertions(+), 5 deletions(-)kernel/sched/fair.c +24
-3@@ -1921,6 +1921,15 @@ static void task_numa_find_cpu(struct task_numa_env *env,if (!cpumask_test_cpu (cpu, env->p->cpus_ptr))continue ; +#ifdef CONFIG_SCHED_CORE +/* +* Skip this cpu if source task's cookie does not match +* with CPU's core cookie. +*/ +if (!sched_core_cookie_match (cpu_rq (cpu), env->p)) +continue ; +#endif + env->dst_cpu = cpu;if (task_numa_compare (env, taskimp, groupimp, maymove))break ;@@ -5867,11 +5876,17 @@ find_idlest_group_cpu(struct sched_group *group, struct task_struct *p, int this/* Traverse only the allowed CPUs */ for_each_cpu_and (i,sched_group_span (group), p->cpus_ptr) { +struct rq *rq =cpu_rq (i); + +#ifdef CONFIG_SCHED_CORE +if (!sched_core_cookie_match (rq, p)) +continue ; +#endif +if (sched_idle_cpu (i))return i;if (available_idle_cpu (i)) { -struct rq *rq =cpu_rq (i);struct cpuidle_state *idle =idle_get_state (rq);if (idle && idle->exit_latency < min_exit_latency) {/* @@ -6129,8 +6144,18 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, int tfor_each_cpu_wrap (cpu, cpus, target) {if (!--nr)return -1 ; -if (available_idle_cpu (cpu) ||sched_idle_cpu (cpu)) -break ; + +if (available_idle_cpu (cpu) ||sched_idle_cpu (cpu)) { +#ifdef CONFIG_SCHED_CORE +/* +* If Core Scheduling is enabled, select this cpu +* only if the process cookie matches core cookie. +*/ + if (sched_core_enabled(cpu_rq(cpu)) &&
Quoted 9 lines
kernel/sched/fair.c static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, int t · line 6158
Quoted 4 lines
kernel/sched/fair.c +4
-0+ p->core_cookie == cpu_rq(cpu)->core->core_cookie) +#endif +break ; + }This makes code unreadable.
Put this coresched specific stuff in an inline function; You can have a look at what is done withasym_fits_capacity()
This is done in a refined version. Sorry I pasted the version embedded in this thread, this is not the latest version.
Quoted 47 lines
Quoted 47 lines
kernel/sched/fair.c +24
-2} time =cpu_clock (this) - time;@@ -7530,8 +7555,9 @@ int can_migrate_task(struct task_struct *p, struct lb_env *env)* We do not migrate tasks that are: * 1) throttled_lb_pair, or * 2) cannot be migrated to this CPU due to cpus_ptr, or -* 3) running (obviously), or -* 4) are cache-hot on their current CPU. +* 3) task's cookie does not match with this CPU's core cookie +* 4) running (obviously), or +* 5) are cache-hot on their current CPU. */ if (throttled_lb_pair (task_group (p), env->src_cpu, env->dst_cpu))return 0 ;@@ -7566,6 +7592,15 @@ int can_migrate_task(struct task_struct *p, struct lb_env *env)return 0 ; } +#ifdef CONFIG_SCHED_CORE +/* +* Don't migrate task if the task's cookie does not match +* with the destination CPU's core cookie. +*/ +if (!sched_core_cookie_match (cpu_rq (env->dst_cpu), p)) +return 0 ; +#endif +/* Record that we found atleast one task that could run on dst_cpu */ env->flags &= ~LBF_ALL_PINNED ;@@ -8792,6 +8827,25 @@ find_idlest_group(struct sched_domain *sd, struct task_struct *p, int this_cpu) p->cpus_ptr))continue ; +#ifdef CONFIG_SCHED_CORE +if (sched_core_enabled (cpu_rq (this_cpu))) { +int i =0 ; +bool cookie_match = false; + +for_each_cpu (i,sched_group_span (group)) { +struct rq *rq =cpu_rq (i); + +if (sched_core_cookie_match (rq, p)) { + cookie_match = true; +break ; + }
Quoted 9 lines
kernel/sched/fair.c find_idlest_group(struct sched_domain *sd, struct task_struct *p, int this_cpu) · line 8847
Quoted 6 lines
kernel/sched/fair.c +6
-0+ } +/* Skip over this group if no cookie matched */ +if (!cookie_match) +continue ; + } +#endif same here, encapsulate this to keep
find_idlest_groupreadable
Okay, I'll try to refine this.
Thanks,
-Aubrey
PATCH 15/32 sched: Improve snapshotting of min_vruntime for CGroups
In reply to Joel Fernandes (Google)A previous patch improved cross-cpu vruntime comparison opertations in pick_next_task(). Improve it further for tasks in CGroups.
In particular, for cross-CPU comparisons, we were previously going to the root-level se(s) for both the task being compared. That was strange. This patch instead finds the se(s) for both tasks that have the same parent (which may be different from root).
A note about the min_vruntime snapshot and force idling:
Abbreviations: fi: force-idled now? ; fib: force-idled before? During selection:
When we're not fi, we need to update snapshot.
when we're fi and we were not fi, we must update snapshot. When we're fi and we were already fi, we must not update snapshot.
Which gives:
fib fi update?
0 0 1
0 1 1
1 0 1
1 1 0So the min_vruntime snapshot needs to be updated when: !(fib && fi).
Also, the cfs_prio_less() function needs to be aware of whether the core is in force idle or not, since it will be use this information to know whether to advance a cfs_rq's min_vruntime_fi in the hierarchy. So pass this information along via pick_task() -> prio_less().
--- kernel/sched/core.c | 61 +++++++++++++++++---------------- kernel/sched/fair.c | 80 ++++++++++++++++++++++++++++++++------------ kernel/sched/sched.h | 7 +++- 3 files changed, 97 insertions(+), 51 deletions(-)
kernel/sched/core.c +32 -29
@@ -101,7 +101,7 @@ static inline int __task_prio(struct task_struct *p)*/ /* real prio, less is less */ -static inline bool prio_less (struct task_struct *a,struct task_struct *b) +static inline bool prio_less (struct task_struct *a,struct task_struct *b,bool in_fi) {int pa =__task_prio (a), pb =__task_prio (b);
@@ -116,7 +116,7 @@ static inline bool prio_less(struct task_struct *a, struct task_struct *b)return !dl_time_before (a->dl.deadline, b->dl.deadline);if (pa ==MAX_RT_PRIO +MAX_NICE )/* fair */ -return cfs_prio_less (a, b); +return cfs_prio_less (a, b, in_fi);return false; }
@@ -130,7 +130,7 @@ static inline bool __sched_core_less(struct task_struct *a, struct task_struct *return false;/* flip prio, so high prio is leftmost */ -if (prio_less (b, a)) +if (prio_less (b, a,task_rq (a)->core->core_forceidle))return true;return false;
@@ -5101,7 +5101,7 @@ static inline bool cookie_match(struct task_struct *a, struct task_struct *b)* - Else returns idle_task. */ static struct task_struct * -pick_task (struct rq *rq,const struct sched_class *class,struct task_struct *max) +pick_task (struct rq *rq,const struct sched_class *class,struct task_struct *max,bool in_fi) {struct task_struct *class_pick, *cookie_pick;unsigned long cookie = rq->core->core_cookie;
@@ -5116,7 +5116,7 @@ pick_task(struct rq *rq, const struct sched_class *class, struct task_struct *ma* higher priority than max. */ if (max && class_pick->core_cookie && -prio_less (class_pick, max)) +prio_less (class_pick, max, in_fi))return idle_sched_class.pick_task (rq);return class_pick;
@@ -5135,13 +5135,15 @@ pick_task(struct rq *rq, const struct sched_class *class, struct task_struct *ma* the core (so far) and it must be selected, otherwise we must go with * the cookie pick in order to satisfy the constraint. */ -if (prio_less (cookie_pick, class_pick) && - (!max ||prio_less (max, class_pick))) +if (prio_less (cookie_pick, class_pick, in_fi) && + (!max ||prio_less (max, class_pick, in_fi)))return class_pick;return cookie_pick; } +extern void task_vruntime_update (struct rq *rq,struct task_struct *p,bool in_fi); +static struct task_struct *pick_next_task (struct rq *rq,struct task_struct *prev,struct rq_flags *rf) {
@@ -5230,9 +5232,14 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)if (!next->core_cookie) { rq->core_pick =NULL ; +/* +* For robustness, update the min_vruntime_fi for +* unconstrained picks as well. +*/ +WARN_ON_ONCE (fi_before); +task_vruntime_update (rq, next, false); goto done; } - need_sync = true; }for_each_cpu (i, smt_mask) {
@@ -5244,14 +5251,6 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)update_rq_clock (rq_i); } -/* Reset the snapshot if core is no longer in force-idle. */ -if (!fi_before) { -for_each_cpu (i, smt_mask) { -struct rq *rq_i =cpu_rq (i); - rq_i->cfs.min_vruntime_fi = rq_i->cfs.min_vruntime; - } - } -/* * Try and select tasks for each sibling in decending sched_class * order.
@@ -5271,7 +5270,7 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)* highest priority task already selected for this * core. */ - p =pick_task (rq_i, class, max); + p =pick_task (rq_i, class, max, fi_before);if (!p)continue ;
@@ -5279,6 +5278,11 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf) occ++; rq_i->core_pick = p; +if (rq_i->idle == p && rq_i->nr_running) { + rq->core->core_forceidle = true; +if (!fi_before) + rq->core->core_forceidle_seq++; + }/* * If this new candidate is of higher priority than the
@@ -5297,6 +5301,7 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf) max = p;if (old_max) { + rq->core->core_forceidle = false;for_each_cpu (j, smt_mask) {if (j == i)continue ;
@@ -5338,10 +5343,16 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)if (!rq_i->core_pick)continue ; -if (is_task_rq_idle (rq_i->core_pick) && rq_i->nr_running && - !rq_i->core->core_forceidle) { - rq_i->core->core_forceidle = true; - } +/* +* Update for new !FI->FI transitions, or if continuing to be in !FI: +* fi_before fi update? +* 0 0 1 +* 0 1 1 +* 1 0 1 +* 1 1 0 +*/ +if (!(fi_before && rq->core->core_forceidle)) +task_vruntime_update (rq_i, rq_i->core_pick, rq->core->core_forceidle); rq_i->core_pick->core_occupation = occ;
@@ -5361,14 +5372,6 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)resched_curr (rq_i); } -/* Snapshot if core is in force-idle. */ -if (!fi_before && rq->core->core_forceidle) { -for_each_cpu (i, smt_mask) { -struct rq *rq_i =cpu_rq (i); - rq_i->cfs.min_vruntime_fi = rq_i->cfs.min_vruntime; - } - } - done:set_next_task (rq, next);return next;
kernel/sched/fair.c +59 -21
102 lines not on this page · raw message
kernel/sched/sched.h +6 -1
@@ -518,8 +518,10 @@ struct cfs_rq {u64 exec_clock;u64 min_vruntime;#ifdef CONFIG_SCHED_CORE +unsigned int forceidle_seq;u64 min_vruntime_fi;#endif +#ifndef CONFIG_64BIT u64 min_vruntime_copy;#endif
@@ -1078,6 +1080,7 @@ struct rq {unsigned int core_pick_seq;unsigned long core_cookie;unsigned char core_forceidle; +unsigned int core_forceidle_seq;#endif };
@@ -1133,7 +1136,7 @@ static inline raw_spinlock_t *rq_lockp(struct rq *rq)return &rq->__lock; } -bool cfs_prio_less (struct task_struct *a,struct task_struct *b); +bool cfs_prio_less (struct task_struct *a,struct task_struct *b,bool fi);/* * Helper to check if the CPU's core cookie matches with the task's cookie
@@ -1166,6 +1169,8 @@ static inline bool sched_core_cookie_match(struct rq *rq, struct task_struct *p)extern void queue_core_balance (struct rq *rq); +bool cfs_prio_less (struct task_struct *a,struct task_struct *b,bool fi); +#else /* !CONFIG_SCHED_CORE */ static inline bool sched_core_enabled (struct rq *rq)
Signature · 1 line
2.29.2.299.gdc1121823c-goog
Peter Zijlstra
maintainerRe: [PATCH -tip 15/32] sched: Improve snapshotting of min_vruntime for CGroups
Quoted 21 lines · Joel Fernandes (Google)
A previous patch improved cross-cpu vruntime comparison opertations in
pick_next_task(). Improve it further for tasks in CGroups.In particular, for cross-CPU comparisons, we were previously going to the root-level se(s) for both the task being compared. That was strange. This patch instead finds the se(s) for both tasks that have the same parent (which may be different from root).
A note about the
min_vruntimesnapshot and force idling:
Abbreviations: fi: force-idled now? ; fib: force-idled before? During selection:
When we're not fi, we need to update snapshot.
when we're fi and we were not fi, we must update snapshot. When we're fi and we were already fi, we must not update snapshot.Which gives:
fib fi update? 0 0 1 0 1 1 1 0 1 1 1 0
Quoted 6 lines
So the
min_vruntimesnapshot needs to be updated when: !(fib && fi).Also, the
cfs_prio_less()function needs to be aware of whether the core is in force idle or not, since it will be use this information to know whether to advance a cfs_rq'smin_vruntime_fiin the hierarchy. So pass this information along viapick_task()->prio_less().
Hurmph.. so I'm tempted to smash a bunch of patches together.
2 <- 3 (already done - bisection crashes are daft) 6 <- 11
7 <- {10, 12}
9 <- 15
I'm thinking that would result in an easier to read series, or do we want to preserve this history?
(fwiw, I pulled 15 before 13,14, as I think that makes more sense anyway).
Hmm?
Re: [PATCH -tip 15/32] sched: Improve snapshotting of min_vruntime for CGroups
Hi Peter,
Quoted 36 lines · Peter Zijlstra
Quoted 21 lines · Joel Fernandes (Google)
A previous patch improved cross-cpu vruntime comparison opertations in
pick_next_task(). Improve it further for tasks in CGroups.In particular, for cross-CPU comparisons, we were previously going to the root-level se(s) for both the task being compared. That was strange. This patch instead finds the se(s) for both tasks that have the same parent (which may be different from root).
A note about the
min_vruntimesnapshot and force idling:
Abbreviations: fi: force-idled now? ; fib: force-idled before? During selection:
When we're not fi, we need to update snapshot.
when we're fi and we were not fi, we must update snapshot. When we're fi and we were already fi, we must not update snapshot.Which gives:
fib fi update? 0 0 1 0 1 1 1 0 1 1 1 0Quoted 6 lines
So the
min_vruntimesnapshot needs to be updated when: !(fib && fi).Also, the
cfs_prio_less()function needs to be aware of whether the core is in force idle or not, since it will be use this information to know whether to advance a cfs_rq'smin_vruntime_fiin the hierarchy. So pass this information along viapick_task()->prio_less().Hurmph.. so I'm tempted to smash a bunch of patches together.
2 <- 3 (already done - bisection crashes are daft) 6 <- 11
7 <- {10, 12}
9 <- 15
Quoted 5 lines
I'm thinking that would result in an easier to read series, or do we want to preserve this history?
(fwiw, I pulled 15 before 13,14, as I think that makes more sense anyway).
Either way would be Ok with me, I would suggest retaining the history though so that the details in the changelog are preserved of the issues we faced, and in the future we can refer back to them.
thanks,
- Joel
Peter Zijlstra
maintainerRe: [PATCH -tip 15/32] sched: Improve snapshotting of min_vruntime for CGroups
Quoted 3 lines · Joel Fernandes
Either way would be Ok with me, I would suggest retaining the history though so that the details in the changelog are preserved of the issues we faced, and in the future we can refer back to them.
Well, in part that's what we have mail archives for. My main concern is that there's a lot of back and forth in the patches as presented and that makes review really awkward.
I've done some of the patch munging proposed, I might do more if I find the motivation.
But first I should probably go read all that API crud you did on top, I've not read that before...
Peter Zijlstra
maintainerRe: [PATCH -tip 15/32] sched: Improve snapshotting of min_vruntime for CGroups
In reply to Joel Fernandes (Google)kernel/sched/fair.c static inline void task_tick_core(struct rq *rq, struct task_struct *curr) · line 10790
Quoted 5 lines · Joel Fernandes (Google)
kernel/sched/fair.c +3
-1+static void se_fi_update (struct sched_entity *se,unsigned int fi_seq,bool forceidle) { -bool samecpu =task_cpu (a) ==task_cpu (b); +bool root = true; +long old, new;
My compiler was not impressed by all those variable definitions.
Quoted 13 lines
kernel/sched/fair.c +13
-0+ +for_each_sched_entity (se) { +struct cfs_rq *cfs_rq =cfs_rq_of (se); + +if (forceidle) { +if (cfs_rq->forceidle_seq == fi_seq) +break ; + cfs_rq->forceidle_seq = fi_seq; + } + + cfs_rq->min_vruntime_fi = cfs_rq->min_vruntime; + } +}
PATCH 16/32 irq_work: Cleanup
In reply to Joel Fernandes (Google)From: Peter Zijlstra <[email protected]>
Get rid of the __call_single_node union and clean up the API a little to avoid external code relying on the structure layout as much.
(Needed for irq_work_is_busy() API in core-scheduling series).
--- drivers/gpu/drm/i915/i915_request.c | 4 ++-- include/linux/irq_work.h | 33 ++++++++++++++++++----------- include/linux/irqflags.h | 4 ++-- kernel/bpf/stackmap.c | 2 +- kernel/irq_work.c | 18 ++++++++-------- kernel/printk/printk.c | 6 ++---- kernel/rcu/tree.c | 3 +-- kernel/time/tick-sched.c | 6 ++---- kernel/trace/bpf_trace.c | 2 +- 9 files changed, 41 insertions(+), 37 deletions(-)
drivers/gpu/drm/i915/i915_request.c +2 -2
@@ -197,7 +197,7 @@ __notify_execute_cb(struct i915_request *rq, bool (*fn)(struct irq_work *wrk))llist_for_each_entry_safe (cb, cn,llist_del_all (&rq->execute_cb), - work.llnode) + work.node.llist)fn (&cb->work); }
@@ -460,7 +460,7 @@ __await_execution(struct i915_request *rq,* callback first, then checking the ACTIVE bit, we serialise with * the completed/retired request. */ -if (llist_add (&cb->work.llnode, &signal->execute_cb)) { +if (llist_add (&cb->work.node.llist, &signal->execute_cb)) {if (i915_request_is_active (signal) ||__request_in_flight (signal))__notify_execute_cb_imm (signal);
include/linux/irq_work.h +21 -12
@@ -14,28 +14,37 @@*/ struct irq_work { -union { -struct __call_single_node node; -struct { -struct llist_node llnode; -atomic_t flags; - }; - }; +struct __call_single_node node;void (*func)(struct irq_work *); }; +#define __IRQ_WORK_INIT (_func, _flags) (struct irq_work){ \ + .node = { .u_flags = (_flags), }, \ + .func = (_func), \ +} + +#define IRQ_WORK_INIT (_func) __IRQ_WORK_INIT(_func, 0) +#define IRQ_WORK_INIT_LAZY (_func) __IRQ_WORK_INIT(_func, IRQ_WORK_LAZY) +#define IRQ_WORK_INIT_HARD (_func) __IRQ_WORK_INIT(_func, IRQ_WORK_HARD_IRQ) + +#define DEFINE_IRQ_WORK (name, _f) \ + struct irq_work name = IRQ_WORK_INIT(_f) +static inline void init_irq_work (struct irq_work *work,void (*func)(struct irq_work *)) { -atomic_set (&work->flags,0 ); - work->func = func; + *work =IRQ_WORK_INIT (func); } -#define DEFINE_IRQ_WORK (name, _f) struct irq_work name = { \ - .flags = ATOMIC_INIT(0), \ - .func = (_f) \ +static inline bool irq_work_is_pending (struct irq_work *work) +{ +return atomic_read (&work->node.a_flags) &IRQ_WORK_PENDING ; } +static inline bool irq_work_is_busy (struct irq_work *work) +{ +return atomic_read (&work->node.a_flags) &IRQ_WORK_BUSY ; +}bool irq_work_queue (struct irq_work *work);bool irq_work_queue_on (struct irq_work *work,int cpu);
include/linux/irqflags.h +2 -2
@@ -109,12 +109,12 @@ do { \# define lockdep_irq_work_enter (__work) \ do { \ - if (!(atomic_read(&__work->flags) & IRQ_WORK_HARD_IRQ))\ + if (!(atomic_read(&__work->node.a_flags) & IRQ_WORK_HARD_IRQ))\ current->irq_config = 1; \ } while (0)# define lockdep_irq_work_exit (__work) \ do { \ - if (!(atomic_read(&__work->flags) & IRQ_WORK_HARD_IRQ))\ + if (!(atomic_read(&__work->node.a_flags) & IRQ_WORK_HARD_IRQ))\ current->irq_config = 0; \ } while (0)
kernel/bpf/stackmap.c +1 -1
@@ -298,7 +298,7 @@ static void stack_map_get_build_id_offset(struct bpf_stack_build_id *id_offs,if (irqs_disabled ()) {if (!IS_ENABLED (CONFIG_PREEMPT_RT )) { work =this_cpu_ptr (&up_read_work); -if (atomic_read (&work->irq_work.flags) &IRQ_WORK_BUSY ) { +if (irq_work_is_busy (&work->irq_work)) {/* cannot queue more up_read, fallback */ irq_work_busy = true; }
kernel/irq_work.c +9 -9
@@ -31,7 +31,7 @@ static bool irq_work_claim(struct irq_work *work) {int oflags; - oflags =atomic_fetch_or (IRQ_WORK_CLAIMED |CSD_TYPE_IRQ_WORK , &work->flags); + oflags =atomic_fetch_or (IRQ_WORK_CLAIMED |CSD_TYPE_IRQ_WORK , &work->node.a_flags);/* * If the work is already pending, no need to raise the IPI. * The pairing atomic_fetch_andnot() in irq_work_run() makes sure
@@ -53,12 +53,12 @@ void __weak arch_irq_work_raise(void)static void __irq_work_queue_local (struct irq_work *work) {/* If the work is "lazy", handle it from next tick if any */ -if (atomic_read (&work->flags) &IRQ_WORK_LAZY ) { -if (llist_add (&work->llnode,this_cpu_ptr (&lazy_list)) && +if (atomic_read (&work->node.a_flags) &IRQ_WORK_LAZY ) { +if (llist_add (&work->node.llist,this_cpu_ptr (&lazy_list)) &&tick_nohz_tick_stopped ())arch_irq_work_raise (); }else { -if (llist_add (&work->llnode,this_cpu_ptr (&raised_list))) +if (llist_add (&work->node.llist,this_cpu_ptr (&raised_list)))arch_irq_work_raise (); } }
@@ -102,7 +102,7 @@ bool irq_work_queue_on(struct irq_work *work, int cpu)if (cpu !=smp_processor_id ()) {/* Arch remote IPI send/receive backend aren't NMI safe */ WARN_ON_ONCE (in_nmi ()); -__smp_call_single_queue (cpu, &work->llnode); +__smp_call_single_queue (cpu, &work->node.llist); }else {__irq_work_queue_local (work); }
@@ -142,7 +142,7 @@ void irq_work_single(void *arg)* to claim that work don't rely on us to handle their data * while we are in the middle of the func. */ - flags =atomic_fetch_andnot (IRQ_WORK_PENDING , &work->flags); + flags =atomic_fetch_andnot (IRQ_WORK_PENDING , &work->node.a_flags);lockdep_irq_work_enter (work); work->func (work);
@@ -152,7 +152,7 @@ void irq_work_single(void *arg)* no-one else claimed it meanwhile. */ flags &= ~IRQ_WORK_PENDING ; - (void )atomic_cmpxchg (&work->flags, flags, flags & ~IRQ_WORK_BUSY ); + (void )atomic_cmpxchg (&work->node.a_flags, flags, flags & ~IRQ_WORK_BUSY ); }static void irq_work_run_list (struct llist_head *list)
@@ -166,7 +166,7 @@ static void irq_work_run_list(struct llist_head *list)return ; llnode =llist_del_all (list); -llist_for_each_entry_safe (work, tmp, llnode, llnode) +llist_for_each_entry_safe (work, tmp, llnode, node.llist)irq_work_single (work); }
@@ -198,7 +198,7 @@ void irq_work_sync(struct irq_work *work) {lockdep_assert_irqs_enabled (); -while (atomic_read (&work->flags) &IRQ_WORK_BUSY ) +while (irq_work_is_busy (work))cpu_relax (); }EXPORT_SYMBOL_GPL (irq_work_sync);
kernel/printk/printk.c +2 -4
@@ -3025,10 +3025,8 @@ static void wake_up_klogd_work_func(struct irq_work *irq_work)wake_up_interruptible (&log_wait); } -static DEFINE_PER_CPU(struct irq_work, wake_up_klogd_work) = { - .func = wake_up_klogd_work_func, - .flags =ATOMIC_INIT (IRQ_WORK_LAZY ), -}; +static DEFINE_PER_CPU(struct irq_work, wake_up_klogd_work) = +IRQ_WORK_INIT_LAZY (wake_up_klogd_work_func);void wake_up_klogd (void ) {
kernel/rcu/tree.c +1 -2
@@ -1311,8 +1311,6 @@ static int rcu_implicit_dynticks_qs(struct rcu_data *rdp)if (IS_ENABLED (CONFIG_IRQ_WORK ) && !rdp->rcu_iw_pending && rdp->rcu_iw_gp_seq != rnp->gp_seq && (rnp->ffmask & rdp->grpmask)) { -init_irq_work (&rdp->rcu_iw, rcu_iw_handler); -atomic_set (&rdp->rcu_iw.flags,IRQ_WORK_HARD_IRQ ); rdp->rcu_iw_pending = true; rdp->rcu_iw_gp_seq = rnp->gp_seq;irq_work_queue_on (&rdp->rcu_iw, rdp->cpu);
@@ -3964,6 +3962,7 @@ int rcutree_prepare_cpu(unsigned int cpu) rdp->cpu_no_qs.b.norm = true; rdp->core_needs_qs = false; rdp->rcu_iw_pending = false; + rdp->rcu_iw =IRQ_WORK_INIT_HARD (rcu_iw_handler); rdp->rcu_iw_gp_seq = rdp->gp_seq -1 ;trace_rcu_grace_period (rcu_state.name, rdp->gp_seq,TPS ("cpuonl" ));raw_spin_unlock_irqrestore_rcu_node (rnp, flags);
kernel/time/tick-sched.c +2 -4
@@ -243,10 +243,8 @@ static void nohz_full_kick_func(struct irq_work *work)/* Empty, the tick restart happens on tick_nohz_irq_exit() */ } -static DEFINE_PER_CPU(struct irq_work, nohz_full_kick_work) = { - .func = nohz_full_kick_func, - .flags =ATOMIC_INIT (IRQ_WORK_HARD_IRQ ), -}; +static DEFINE_PER_CPU(struct irq_work, nohz_full_kick_work) = +IRQ_WORK_INIT_HARD (nohz_full_kick_func);/* * Kick this CPU if it's full dynticks in order to force it to
kernel/trace/bpf_trace.c +1 -1
@@ -1086,7 +1086,7 @@ static int bpf_send_signal_common(u32 sig, enum pid_type type)return -EINVAL ; work =this_cpu_ptr (&send_signal_work); -if (atomic_read (&work->irq_work.flags) &IRQ_WORK_BUSY ) +if (irq_work_is_busy (&work->irq_work))return -EBUSY ;/* Add the current task, which is the target of sending signal,
Signature · 1 line
2.29.2.299.gdc1121823c-goog
PATCH 17/32 arch/x86: Add a new TIF flag for untrusted tasks
In reply to Joel Fernandes (Google)Add a new TIF flag to indicate whether the kernel needs to be careful and take additional steps to mitigate micro-architectural issues during entry into user or guest mode.
This new flag will be used by the series to determine if waiting is needed or not, during exit to user or guest mode.
--- arch/x86/include/asm/thread_info.h | 2 ++ kernel/sched/sched.h | 6 ++++++ 2 files changed, 8 insertions(+)
arch/x86/include/asm/thread_info.h +2 -0
@@ -99,6 +99,7 @@ struct thread_info {#define TIF_SPEC_FORCE_UPDATE 23/* Force speculation MSR update in context switch */ #define TIF_FORCED_TF 24/* true if TF in eflags artificially */ #define TIF_BLOCKSTEP 25/* set when we want DEBUGCTLMSR_BTF */ +#define TIF_UNSAFE_RET 26/* On return to process/guest, perform safety checks. */ #define TIF_LAZY_MMU_UPDATES 27/* task is updating the mmu lazily */ #define TIF_SYSCALL_TRACEPOINT 28/* syscall tracepoint instrumentation */ #define TIF_ADDR32 29/* 32-bit address space on 64 bits */
@@ -127,6 +128,7 @@ struct thread_info {#define _TIF_SPEC_FORCE_UPDATE (1 << TIF_SPEC_FORCE_UPDATE)#define _TIF_FORCED_TF (1 << TIF_FORCED_TF)#define _TIF_BLOCKSTEP (1 << TIF_BLOCKSTEP) +#define _TIF_UNSAFE_RET (1 << TIF_UNSAFE_RET)#define _TIF_LAZY_MMU_UPDATES (1 << TIF_LAZY_MMU_UPDATES)#define _TIF_SYSCALL_TRACEPOINT (1 << TIF_SYSCALL_TRACEPOINT)#define _TIF_ADDR32 (1 << TIF_ADDR32)
kernel/sched/sched.h +6 -0
@@ -2851,3 +2851,9 @@ static inline bool is_per_cpu_kthread(struct task_struct *p)void swake_up_all_locked (struct swait_queue_head *q);void __prepare_to_swait (struct swait_queue_head *q,struct swait_queue *wait); + +#ifdef CONFIG_SCHED_CORE +#ifndef TIF_UNSAFE_RET +#define TIF_UNSAFE_RET (0) +#endif +#endif
Signature · 1 line
2.29.2.299.gdc1121823c-goog
Re: [PATCH -tip 17/32] arch/x86: Add a new TIF flag for untrusted tasks
Quoted 11 lines · Joel Fernandes (Google)
Add a new TIF flag to indicate whether the kernel needs to be careful and take additional steps to mitigate micro-architectural issues during entry into user or guest mode.
This new flag will be used by the series to determine if waiting is needed or not, during exit to user or guest mode.
Tested-by Julien Desfossez <[email protected]>Reviewed-by Aubrey Li <[email protected]>Signed-off-by Joel Fernandes (Google) <[email protected]>---
PATCH 18/32 kernel/entry: Add support for core-wide protection of kernel-mode
In reply to Joel Fernandes (Google)Core-scheduling prevents hyperthreads in usermode from attacking each other, but it does not do anything about one of the hyperthreads entering the kernel for any reason. This leaves the door open for MDS and L1TF attacks with concurrent execution sequences between hyperthreads.
This patch therefore adds support for protecting all syscall and IRQ kernel mode entries. Care is taken to track the outermost usermode exit and entry using per-cpu counters. In cases where one of the hyperthreads enter the kernel, no additional IPIs are sent. Further, IPIs are avoided when not needed - example: idle and non-cookie HTs do not need to be forced into kernel mode.
More information about attacks:
For MDS, it is possible for syscalls, IRQ and softirq handlers to leak data to either host or guest attackers. For L1TF, it is possible to leak to guest attackers. There is no possible mitigation involving flushing of buffers to avoid this since the execution of attacker and victims happen concurrently on 2 or more HTs.
--- .../admin-guide/kernel-parameters.txt | 11 + include/linux/entry-common.h | 12 +- include/linux/sched.h | 12 + kernel/entry/common.c | 28 +- kernel/sched/core.c | 241 ++++++++++++++++++ kernel/sched/sched.h | 3 + 6 files changed, 304 insertions(+), 3 deletions(-)
Documentation/admin-guide/kernel-parameters.txt +11 -0
@@ -4678,6 +4678,17 @@ sbni= [NET] Granch SBNI12 leased line adapter + sched_core_protect_kernel= + [SCHED_CORE] Pause SMT siblings of a core running in + user mode, if at least one of the siblings of the core + is running in kernel mode. This is to guarantee that + kernel data is not leaked to tasks which are not trusted + by the kernel. A value of 0 disables protection, 1 + enables protection. The default is 1. Note that protection + depends on the arch defining the _TIF_UNSAFE_RET flag. + Further, for protecting VMEXIT, arch needs to call + KVM entry/exit hooks. + sched_debug [KNL] Enables verbose scheduler debug messages. schedstats= [KNL,X86] Enable or disable scheduled statistics.
include/linux/entry-common.h +11 -1
@@ -33,6 +33,10 @@# define _TIF_PATCH_PENDING (0)#endif +#ifndef _TIF_UNSAFE_RET +# define _TIF_UNSAFE_RET (0) +#endif +#ifndef _TIF_UPROBE# define _TIF_UPROBE (0)#endif
@@ -74,7 +78,7 @@#define EXIT_TO_USER_MODE_WORK \ (_TIF_SIGPENDING | _TIF_NOTIFY_RESUME | _TIF_UPROBE | \ _TIF_NEED_RESCHED | _TIF_PATCH_PENDING | _TIF_NOTIFY_SIGNAL | \ - ARCH_EXIT_TO_USER_MODE_WORK) + _TIF_UNSAFE_RET | ARCH_EXIT_TO_USER_MODE_WORK)/** * arch_check_user_regs - Architecture specific sanity check for user mode regs
@@ -444,4 +448,10 @@ irqentry_state_t noinstr irqentry_nmi_enter(struct pt_regs *regs);*/ void noinstr irqentry_nmi_exit(struct pt_regs *regs, irqentry_state_t irq_state); +/* entry_kernel_protected - Is kernel protection on entry/exit into kernel supported? */ +static inline bool entry_kernel_protected (void ) +{ +return IS_ENABLED (CONFIG_SCHED_CORE ) &&sched_core_kernel_protected () + && _TIF_UNSAFE_RET !=0 ; +}#endif
include/linux/sched.h +12 -0
@@ -2076,4 +2076,16 @@ int sched_trace_rq_nr_running(struct rq *rq);const struct cpumask *sched_trace_rd_span (struct root_domain *rd); +#ifdef CONFIG_SCHED_CORE +void sched_core_unsafe_enter (void ); +void sched_core_unsafe_exit (void ); +bool sched_core_wait_till_safe (unsigned long ti_check); +bool sched_core_kernel_protected (void ); +#else +#define sched_core_unsafe_enter (ignore) do { } while (0) +#define sched_core_unsafe_exit (ignore) do { } while (0) +#define sched_core_wait_till_safe (ignore) do { } while (0) +#define sched_core_kernel_protected (ignore) do { } while (0) +#endif +#endif
kernel/entry/common.c +26 -2
@@ -28,6 +28,9 @@ static __always_inline void enter_from_user_mode(struct pt_regs *regs)instrumentation_begin ();trace_hardirqs_off_finish (); + +if (entry_kernel_protected ()) +sched_core_unsafe_enter ();instrumentation_end (); }
@@ -145,6 +148,26 @@ static void handle_signal_work(struct pt_regs *regs, unsigned long ti_work)arch_do_signal_or_restart (regs, ti_work & _TIF_SIGPENDING); } +static unsigned long exit_to_user_get_work (void ) +{ +unsigned long ti_work =READ_ONCE (current_thread_info ()->flags); + +if (!entry_kernel_protected ()) +return ti_work; + +#ifdef CONFIG_SCHED_CORE + ti_work &=EXIT_TO_USER_MODE_WORK ; +if ((ti_work & _TIF_UNSAFE_RET) == ti_work) { +sched_core_unsafe_exit (); +if (sched_core_wait_till_safe (EXIT_TO_USER_MODE_WORK )) { +sched_core_unsafe_enter ();/* not exiting to user yet. */ + } + } + +return READ_ONCE (current_thread_info ()->flags); +#endif +} +static unsigned long exit_to_user_mode_loop (struct pt_regs *regs,unsigned long ti_work) {
@@ -182,7 +205,7 @@ static unsigned long exit_to_user_mode_loop(struct pt_regs *regs,* enabled above. */ local_irq_disable_exit_to_user (); - ti_work =READ_ONCE (current_thread_info ()->flags); + ti_work =exit_to_user_get_work (); }/* Return the latest work state for arch_exit_to_user_mode() */
@@ -191,9 +214,10 @@ static unsigned long exit_to_user_mode_loop(struct pt_regs *regs,static void exit_to_user_mode_prepare (struct pt_regs *regs) { -unsigned long ti_work =READ_ONCE (current_thread_info ()->flags); +unsigned long ti_work;lockdep_assert_irqs_disabled (); + ti_work =exit_to_user_get_work ();if (unlikely (ti_work &EXIT_TO_USER_MODE_WORK )) ti_work =exit_to_user_mode_loop (regs, ti_work);
kernel/sched/core.c +241 -0
259 lines not on this page · raw message
kernel/sched/sched.h +3 -0
@@ -1074,6 +1074,8 @@ struct rq {unsigned int core_enabled;unsigned int core_sched_seq;struct rb_root core_tree; +struct irq_work core_irq_work;/* To force HT into kernel */ +unsigned int core_this_unsafe_nest;/* shared state */ unsigned int core_task_seq;
@@ -1081,6 +1083,7 @@ struct rq {unsigned long core_cookie;unsigned char core_forceidle;unsigned int core_forceidle_seq; +unsigned int core_unsafe_nest;#endif };
Signature · 1 line
2.29.2.299.gdc1121823c-goog
Peter Zijlstra
maintainerRe: [PATCH -tip 18/32] kernel/entry: Add support for core-wide protection of kernel-mode
Quoted 13 lines · Joel Fernandes (Google)
Core-scheduling prevents hyperthreads in usermode from attacking each other, but it does not do anything about one of the hyperthreads entering the kernel for any reason. This leaves the door open for MDS and L1TF attacks with concurrent execution sequences between hyperthreads.
This patch therefore adds support for protecting all syscall and IRQ kernel mode entries. Care is taken to track the outermost usermode exit and entry using per-cpu counters. In cases where one of the hyperthreads enter the kernel, no additional IPIs are sent. Further, IPIs are avoided when not needed - example: idle and non-cookie HTs do not need to be forced into kernel mode.
Quoted 6 lines
More information about attacks:
For MDS, it is possible for syscalls, IRQ and softirq handlers to leak data to either host or guest attackers. For L1TF, it is possible to leak to guest attackers. There is no possible mitigation involving flushing of buffers to avoid this since the execution of attacker and victims happen concurrently on 2 or more HTs.
Oh gawd; this is horrible...
Quoted 21 lines
kernel/sched/core.c +21
-0+bool sched_core_wait_till_safe (unsigned long ti_check) +{ +bool restart = false; +struct rq *rq; +int cpu; + +/* We clear the thread flag only at the end, so no need to check for it. */ + ti_check &= ~_TIF_UNSAFE_RET; + + cpu =smp_processor_id (); + rq =cpu_rq (cpu); + +if (!sched_core_enabled (rq)) + goto ret; + +/* Down grade to allow interrupts to prevent stop_machine lockups.. */ +preempt_disable (); +local_irq_enable (); + +/* +* Wait till the core of this HT is not in an unsafe state.
kernel/sched/core.c static inline bool cookie_match(struct task_struct *a, struct task_struct *b) · line 5170
Quoted 10 lines
kernel/sched/core.c +10
-0+* +* Pair with raw_spin_lock/unlock() in sched_core_unsafe_enter/exit(). +*/ +while (smp_load_acquire (&rq->core->core_unsafe_nest) >0 ) { +cpu_relax (); +if (READ_ONCE (current_thread_info ()->flags) & ti_check) { + restart = true; +break ; + } + }
What's that ACQUIRE for?
kernel/sched/core.c static inline bool cookie_match(struct task_struct *a, struct task_struct *b) · line 5181
Quoted 11 lines
kernel/sched/core.c +11
-0+ +/* Upgrade it back to the expectations of entry code. */ +local_irq_disable (); +preempt_enable (); + +ret: +if (!restart) +clear_tsk_thread_flag (current,TIF_UNSAFE_RET ); + +return restart; +}
So if TIF_NEED_RESCHED gets set, we'll break out and reschedule, cute.
Quoted 74 lines
kernel/sched/core.c +74
-0+void sched_core_unsafe_enter (void ) +{ +const struct cpumask *smt_mask; +unsigned long flags; +struct rq *rq; +int i, cpu; + +if (!static_branch_likely (&sched_core_protect_kernel)) +return ; + +local_irq_save (flags); + cpu =smp_processor_id (); + rq =cpu_rq (cpu); +if (!sched_core_enabled (rq)) + goto ret; + +/* Ensure that on return to user/guest, we check whether to wait. */ +if (current->core_cookie) +set_tsk_thread_flag (current,TIF_UNSAFE_RET ); + +/* Count unsafe_enter() calls received without unsafe_exit() on this CPU. */ + rq->core_this_unsafe_nest++; + +/* +* Should not nest: enter() should only pair with exit(). Both are done +* during the first entry into kernel and the last exit from kernel. +* Nested kernel entries (such as nested interrupts) will only trigger +* enter() and exit() on the outer most kernel entry and exit. +*/ +if (WARN_ON_ONCE (rq->core_this_unsafe_nest !=1 )) + goto ret; + +raw_spin_lock (rq_lockp (rq)); + smt_mask =cpu_smt_mask (cpu); + +/* +* Contribute this CPU's unsafe_enter() to the core-wide unsafe_enter() +* count. The raw_spin_unlock() release semantics pairs with the nest +* counter's smp_load_acquire() in sched_core_wait_till_safe(). +*/ +WRITE_ONCE (rq->core->core_unsafe_nest, rq->core->core_unsafe_nest +1 ); + +if (WARN_ON_ONCE (rq->core->core_unsafe_nest ==UINT_MAX )) + goto unlock; + +if (irq_work_is_busy (&rq->core_irq_work)) { +/* +* Do nothing more since we are in an IPI sent from another +* sibling to enforce safety. That sibling would have sent IPIs +* to all of the HTs. +*/ + goto unlock; + } + +/* +* If we are not the first ones on the core to enter core-wide unsafe +* state, do nothing. +*/ +if (rq->core->core_unsafe_nest >1 ) + goto unlock; + +/* Do nothing more if the core is not tagged. */ +if (!rq->core->core_cookie) + goto unlock; + +for_each_cpu (i, smt_mask) { +struct rq *srq =cpu_rq (i); + +if (i == cpu ||cpu_is_offline (i)) +continue ; + +if (!srq->curr->mm ||is_task_rq_idle (srq->curr)) +continue ; +
kernel/sched/core.c static inline bool cookie_match(struct task_struct *a, struct task_struct *b) · line 5272
Quoted 10 lines
kernel/sched/core.c +10
-0+/* Skip if HT is not running a tagged task. */ +if (!srq->curr->core_cookie && !srq->core_pick) +continue ; + +/* +* Force sibling into the kernel by IPI. If work was already +* pending, no new IPIs are sent. This is Ok since the receiver +* would already be in the kernel, or on its way to it. +*/ +irq_work_queue_on (&srq->core_irq_work, i);
Why irq_work though? Why not smp_send_reschedule(i)?
Quoted 6 lines
kernel/sched/core.c +6
-0+ } +unlock: +raw_spin_unlock (rq_lockp (rq)); +ret: +local_irq_restore (flags); +}
Re: [PATCH -tip 18/32] kernel/entry: Add support for core-wide protection of kernel-mode
Hi Peter,
Quoted 14 lines · Peter Zijlstra
Quoted 13 lines · Joel Fernandes (Google)
Core-scheduling prevents hyperthreads in usermode from attacking each other, but it does not do anything about one of the hyperthreads entering the kernel for any reason. This leaves the door open for MDS and L1TF attacks with concurrent execution sequences between hyperthreads.
This patch therefore adds support for protecting all syscall and IRQ kernel mode entries. Care is taken to track the outermost usermode exit and entry using per-cpu counters. In cases where one of the hyperthreads enter the kernel, no additional IPIs are sent. Further, IPIs are avoided when not needed - example: idle and non-cookie HTs do not need to be forced into kernel mode.
Quoted 8 lines
Quoted 6 lines
More information about attacks:
For MDS, it is possible for syscalls, IRQ and softirq handlers to leak data to either host or guest attackers. For L1TF, it is possible to leak to guest attackers. There is no possible mitigation involving flushing of buffers to avoid this since the execution of attacker and victims happen concurrently on 2 or more HTs.Oh gawd; this is horrible...
Yeah, its the same issue we discussed earlier this year :)
Quoted 25 lines
Quoted 25 lines
kernel/sched/core.c +25
-0+bool sched_core_wait_till_safe (unsigned long ti_check) +{ +bool restart = false; +struct rq *rq; +int cpu; + +/* We clear the thread flag only at the end, so no need to check for it. */ + ti_check &= ~_TIF_UNSAFE_RET; + + cpu =smp_processor_id (); + rq =cpu_rq (cpu); + +if (!sched_core_enabled (rq)) + goto ret; + +/* Down grade to allow interrupts to prevent stop_machine lockups.. */ +preempt_disable (); +local_irq_enable (); + +/* +* Wait till the core of this HT is not in an unsafe state. +* +* Pair with raw_spin_lock/unlock() in sched_core_unsafe_enter/exit(). +*/ +while (smp_load_acquire (&rq->core->core_unsafe_nest) >0 ) {
Quoted 8 lines
kernel/sched/core.c static inline bool cookie_match(struct task_struct *a, struct task_struct *b) · line 5170
Quoted 6 lines
kernel/sched/core.c +6
-0+cpu_relax (); +if (READ_ONCE (current_thread_info ()->flags) & ti_check) { + restart = true; +break ; + } + }What's that ACQUIRE for?
I was concerned about something like below for weakly-ordered arch:
Kernel Attacker
------ --------
write unsafe=1
kernel code does stores while (unsafe == 1); (ACQUIRE)
^ ^
| needs to be ordered | needs to be ordered v v write unsafe=0 (RELEASE) Attacker code.
Here, I want the access made by kernel code to be ordered WRT the write to the unsafe nesting counter variable, so that the attacker code does not see those accesses later.
It could be argued its a theoretical concern, but I wanted to play it safe. In particular, the existing uarch buffer flushing before entering the Attacker code might make it sufficiently impossible for Attacker to do anything bad even without the additional memory barriers.
Quoted 5 lines
Quoted 5 lines
kernel/sched/core.c +5
-0+ +/* Upgrade it back to the expectations of entry code. */ +local_irq_disable (); +preempt_enable (); +
Quoted 8 lines
kernel/sched/core.c static inline bool cookie_match(struct task_struct *a, struct task_struct *b) · line 5181
Quoted 6 lines
kernel/sched/core.c +6
-0+ret: +if (!restart) +clear_tsk_thread_flag (current,TIF_UNSAFE_RET ); + +return restart; +}So if
TIF_NEED_RESCHEDgets set, we'll break out and reschedule, cute.
Thanks.
Quoted 17 lines
Quoted 17 lines
kernel/sched/core.c +17
-0+/* Do nothing more if the core is not tagged. */ +if (!rq->core->core_cookie) + goto unlock; + +for_each_cpu (i, smt_mask) { +struct rq *srq =cpu_rq (i); + +if (i == cpu ||cpu_is_offline (i)) +continue ; + +if (!srq->curr->mm ||is_task_rq_idle (srq->curr)) +continue ; + +/* Skip if HT is not running a tagged task. */ +if (!srq->curr->core_cookie && !srq->core_pick) +continue ; +
Quoted 8 lines
kernel/sched/core.c static inline bool cookie_match(struct task_struct *a, struct task_struct *b) · line 5272
Quoted 6 lines
kernel/sched/core.c +6
-0+/* +* Force sibling into the kernel by IPI. If work was already +* pending, no new IPIs are sent. This is Ok since the receiver +* would already be in the kernel, or on its way to it. +*/ +irq_work_queue_on (&srq->core_irq_work, i);Why
irq_workthough? Why notsmp_send_reschedule(i)?
I tried this approach. The main issue is sending IPIs if a previous IPI was already sent. I needed some mechanism (shared variables to detect this) so that an IPI would not be sent if one was just sent.
With irq_work, I get the irq_work_is_busy() which helps with exactly that. With other approaches I tried, I ran into issues with CSD locking and it seems irq_work implements the CSD locking the way I wanted to. So to avoid reinventing the wheel, I stuck to irq_work and it worked well.
thanks,
- Joel
Peter Zijlstra
maintainerRe: [PATCH -tip 18/32] kernel/entry: Add support for core-wide protection of kernel-mode
In reply to Joel Fernandes (Google)Quoted 40 lines · Joel Fernandes (Google)
Core-scheduling prevents hyperthreads in usermode from attacking each other, but it does not do anything about one of the hyperthreads entering the kernel for any reason. This leaves the door open for MDS and L1TF attacks with concurrent execution sequences between hyperthreads.
This patch therefore adds support for protecting all syscall and IRQ kernel mode entries. Care is taken to track the outermost usermode exit and entry using per-cpu counters. In cases where one of the hyperthreads enter the kernel, no additional IPIs are sent. Further, IPIs are avoided when not needed - example: idle and non-cookie HTs do not need to be forced into kernel mode.
More information about attacks:
For MDS, it is possible for syscalls, IRQ and softirq handlers to leak data to either host or guest attackers. For L1TF, it is possible to leak to guest attackers. There is no possible mitigation involving flushing of buffers to avoid this since the execution of attacker and victims happen concurrently on 2 or more HTs..../admin-guide/kernel-parameters.txt | 11 + include/linux/entry-common.h | 12 +- include/linux/sched.h | 12 + kernel/entry/common.c | 28 +- kernel/sched/core.c | 241 ++++++++++++++++++ kernel/sched/sched.h | 3 + 6 files changed, 304 insertions(+), 3 deletions(-)Documentation/admin-guide/kernel-parameters.txt +4
-0@@ -4678,6 +4678,17 @@ sbni= [NET] Granch SBNI12 leased line adapter + sched_core_protect_kernel= + [SCHED_CORE] Pause SMT siblings of a core running in + user mode, if at least one of the siblings of the core + is running in kernel mode. This is to guarantee that
Documentation/admin-guide/kernel-parameters.txt line 4694
Quoted 10 lines
Documentation/admin-guide/kernel-parameters.txt +7
-0+ kernel data is not leaked to tasks which are not trusted + by the kernel. A value of 0 disables protection, 1 + enables protection. The default is 1. Note that protection + depends on the arch defining the _TIF_UNSAFE_RET flag. + Further, for protecting VMEXIT, arch needs to call + KVM entry/exit hooks. + sched_debug [KNL] Enables verbose scheduler debug messages. schedstats= [KNL,X86] Enable or disable scheduled statistics.
So I don't like the parameter name, it's too long. Also I don't like it because its a boolean.
You're adding syscall,irq,kvm under a single knob where they're all due to different flavours of broken. Different hardware might want/need different combinations.
Hardware without MDS but with L1TF wouldn't need the syscall hook, but you're not givng a choice here. And this is generic code, you can't assume stuff like this.
Re: [PATCH -tip 18/32] kernel/entry: Add support for core-wide protection of kernel-mode
Quoted 50 lines · Peter Zijlstra
Quoted 49 lines · Joel Fernandes (Google)
Core-scheduling prevents hyperthreads in usermode from attacking each other, but it does not do anything about one of the hyperthreads entering the kernel for any reason. This leaves the door open for MDS and L1TF attacks with concurrent execution sequences between hyperthreads.
This patch therefore adds support for protecting all syscall and IRQ kernel mode entries. Care is taken to track the outermost usermode exit and entry using per-cpu counters. In cases where one of the hyperthreads enter the kernel, no additional IPIs are sent. Further, IPIs are avoided when not needed - example: idle and non-cookie HTs do not need to be forced into kernel mode.
More information about attacks:
For MDS, it is possible for syscalls, IRQ and softirq handlers to leak data to either host or guest attackers. For L1TF, it is possible to leak to guest attackers. There is no possible mitigation involving flushing of buffers to avoid this since the execution of attacker and victims happen concurrently on 2 or more HTs..../admin-guide/kernel-parameters.txt | 11 + include/linux/entry-common.h | 12 +- include/linux/sched.h | 12 + kernel/entry/common.c | 28 +- kernel/sched/core.c | 241 ++++++++++++++++++ kernel/sched/sched.h | 3 + 6 files changed, 304 insertions(+), 3 deletions(-)Documentation/admin-guide/kernel-parameters.txt +11
-0@@ -4678,6 +4678,17 @@ sbni= [NET] Granch SBNI12 leased line adapter + sched_core_protect_kernel= + [SCHED_CORE] Pause SMT siblings of a core running in + user mode, if at least one of the siblings of the core + is running in kernel mode. This is to guarantee that + kernel data is not leaked to tasks which are not trusted + by the kernel. A value of 0 disables protection, 1 + enables protection. The default is 1. Note that protection + depends on the arch defining the _TIF_UNSAFE_RET flag. + Further, for protecting VMEXIT, arch needs to call + KVM entry/exit hooks. + sched_debug [KNL] Enables verbose scheduler debug messages.
Quoted 4 lines
Documentation/admin-guide/kernel-parameters.txt line 4694
Quoted 1 line
Documentation/admin-guide/kernel-parameters.txt +0
-0schedstats= [KNL,X86] Enable or disable scheduled statistics.So I don't like the parameter name, it's too long. Also I don't like it because its a boolean.
Maybe ht_protect= then?
Quoted 3 lines
You're adding syscall,irq,kvm under a single knob where they're all due to different flavours of broken. Different hardware might want/need different combinations.
Ok, I can try to make it ht_protect=irq,syscall,kvm etc. And conditionally enable the protection. Does that work for you?
Quoted 4 lines
Hardware without MDS but with L1TF wouldn't need the syscall hook, but you're not givng a choice here. And this is generic code, you can't assume stuff like this.
Got it.
thanks,
- Joel
Re: [PATCH -tip 18/32] kernel/entry: Add support for core-wide protection of kernel-mode
In reply to Joel Fernandes (Google)Quoted 134 lines · Joel Fernandes (Google)
Core-scheduling prevents hyperthreads in usermode from attacking each other, but it does not do anything about one of the hyperthreads entering the kernel for any reason. This leaves the door open for MDS and L1TF attacks with concurrent execution sequences between hyperthreads.
This patch therefore adds support for protecting all syscall and IRQ kernel mode entries. Care is taken to track the outermost usermode exit and entry using per-cpu counters. In cases where one of the hyperthreads enter the kernel, no additional IPIs are sent. Further, IPIs are avoided when not needed - example: idle and non-cookie HTs do not need to be forced into kernel mode.
More information about attacks:
For MDS, it is possible for syscalls, IRQ and softirq handlers to leak data to either host or guest attackers. For L1TF, it is possible to leak to guest attackers. There is no possible mitigation involving flushing of buffers to avoid this since the execution of attacker and victims happen concurrently on 2 or more HTs.Reviewed-by Alexandre Chartre <[email protected]>Tested-by Julien Desfossez <[email protected]>Cc Julien Desfossez <[email protected]>Cc Tim Chen <[email protected]>Cc Aaron Lu <[email protected]>Cc Aubrey Li <[email protected]>Cc Tim Chen <[email protected]>Cc Paul E. McKenney <[email protected]>Co-developed-by Vineeth Pillai <[email protected]>Signed-off-by Vineeth Pillai <[email protected]>Signed-off-by Joel Fernandes (Google) <[email protected]>--- .../admin-guide/kernel-parameters.txt | 11 + include/linux/entry-common.h | 12 +- include/linux/sched.h | 12 + kernel/entry/common.c | 28 +- kernel/sched/core.c | 241 ++++++++++++++++++ kernel/sched/sched.h | 3 + 6 files changed, 304 insertions(+), 3 deletions(-)Documentation/admin-guide/kernel-parameters.txt +11
-0@@ -4678,6 +4678,17 @@ sbni= [NET] Granch SBNI12 leased line adapter + sched_core_protect_kernel= + [SCHED_CORE] Pause SMT siblings of a core running in + user mode, if at least one of the siblings of the core + is running in kernel mode. This is to guarantee that + kernel data is not leaked to tasks which are not trusted + by the kernel. A value of 0 disables protection, 1 + enables protection. The default is 1. Note that protection + depends on the arch defining the _TIF_UNSAFE_RET flag. + Further, for protecting VMEXIT, arch needs to call + KVM entry/exit hooks. + sched_debug [KNL] Enables verbose scheduler debug messages. schedstats= [KNL,X86] Enable or disable scheduled statistics.include/linux/entry-common.h +11
-1@@ -33,6 +33,10 @@# define _TIF_PATCH_PENDING (0)#endif +#ifndef _TIF_UNSAFE_RET +# define _TIF_UNSAFE_RET (0) +#endif +#ifndef _TIF_UPROBE# define _TIF_UPROBE (0)#endif @@ -74,7 +78,7 @@#define EXIT_TO_USER_MODE_WORK \ (_TIF_SIGPENDING | _TIF_NOTIFY_RESUME | _TIF_UPROBE | \ _TIF_NEED_RESCHED | _TIF_PATCH_PENDING | _TIF_NOTIFY_SIGNAL | \ - ARCH_EXIT_TO_USER_MODE_WORK) + _TIF_UNSAFE_RET | ARCH_EXIT_TO_USER_MODE_WORK)/** * arch_check_user_regs - Architecture specific sanity check for user mode regs @@ -444,4 +448,10 @@ irqentry_state_t noinstr irqentry_nmi_enter(struct pt_regs *regs);*/ void noinstr irqentry_nmi_exit(struct pt_regs *regs, irqentry_state_t irq_state); +/* entry_kernel_protected - Is kernel protection on entry/exit into kernel supported? */ +static inline bool entry_kernel_protected (void ) +{ +return IS_ENABLED (CONFIG_SCHED_CORE ) &&sched_core_kernel_protected () + && _TIF_UNSAFE_RET !=0 ; +}#endif include/linux/sched.h +12
-0@@ -2076,4 +2076,16 @@ int sched_trace_rq_nr_running(struct rq *rq);const struct cpumask *sched_trace_rd_span (struct root_domain *rd); +#ifdef CONFIG_SCHED_CORE +void sched_core_unsafe_enter (void ); +void sched_core_unsafe_exit (void ); +bool sched_core_wait_till_safe (unsigned long ti_check); +bool sched_core_kernel_protected (void ); +#else +#define sched_core_unsafe_enter (ignore) do { } while (0) +#define sched_core_unsafe_exit (ignore) do { } while (0) +#define sched_core_wait_till_safe (ignore) do { } while (0) +#define sched_core_kernel_protected (ignore) do { } while (0) +#endif +#endif kernel/entry/common.c +3
-0@@ -28,6 +28,9 @@ static __always_inline void enter_from_user_mode(struct pt_regs *regs)instrumentation_begin ();trace_hardirqs_off_finish (); + +if (entry_kernel_protected ()) +sched_core_unsafe_enter ();instrumentation_end (); }@@ -145,6 +148,26 @@ static void handle_signal_work(struct pt_regs *regs, unsigned long ti_work)arch_do_signal_or_restart (regs, ti_work & _TIF_SIGPENDING);
kernel/entry/common.c static void handle_signal_work(struct pt_regs *regs, unsigned long ti_work) · line 158
Quoted 10 lines
kernel/entry/common.c +8
-0} +static unsigned long exit_to_user_get_work (void ) +{ +unsigned long ti_work =READ_ONCE (current_thread_info ()->flags); + +if (!entry_kernel_protected ()) +return ti_work; + +#ifdef CONFIG_SCHED_CORE
This #ifdef is not necessary, entry_kernel_protected() does this check no? The code should also compile anyway from what I can see so far.
Quoted 100 lines
kernel/entry/common.c +15
-2+ ti_work &=EXIT_TO_USER_MODE_WORK ; +if ((ti_work & _TIF_UNSAFE_RET) == ti_work) { +sched_core_unsafe_exit (); +if (sched_core_wait_till_safe (EXIT_TO_USER_MODE_WORK )) { +sched_core_unsafe_enter ();/* not exiting to user yet. */ + } + } + +return READ_ONCE (current_thread_info ()->flags); +#endif +} +static unsigned long exit_to_user_mode_loop (struct pt_regs *regs,unsigned long ti_work) {@@ -182,7 +205,7 @@ static unsigned long exit_to_user_mode_loop(struct pt_regs *regs,* enabled above. */ local_irq_disable_exit_to_user (); - ti_work =READ_ONCE (current_thread_info ()->flags); + ti_work =exit_to_user_get_work (); }/* Return the latest work state for arch_exit_to_user_mode() */ @@ -191,9 +214,10 @@ static unsigned long exit_to_user_mode_loop(struct pt_regs *regs,static void exit_to_user_mode_prepare (struct pt_regs *regs) { -unsigned long ti_work =READ_ONCE (current_thread_info ()->flags); +unsigned long ti_work;lockdep_assert_irqs_disabled (); + ti_work =exit_to_user_get_work ();if (unlikely (ti_work &EXIT_TO_USER_MODE_WORK )) ti_work =exit_to_user_mode_loop (regs, ti_work);kernel/sched/core.c +49
-0@@ -76,6 +76,27 @@ __read_mostly int scheduler_running;#ifdef CONFIG_SCHED_CORE +DEFINE_STATIC_KEY_TRUE (sched_core_protect_kernel); +static int __init set_sched_core_protect_kernel (char *str) +{ +unsigned long val =0 ; + +if (!str) +return 0 ; + +if (!kstrtoul (str,0 , &val) && !val) +static_branch_disable (&sched_core_protect_kernel); + +return 1 ; +} +__setup ("sched_core_protect_kernel=" , set_sched_core_protect_kernel); + +/* Is the kernel protected by core scheduling? */ +bool sched_core_kernel_protected (void ) +{ +return static_branch_likely (&sched_core_protect_kernel); +} +DEFINE_STATIC_KEY_FALSE (__sched_core_enabled);/* kernel prio, less is more */ @@ -5092,6 +5113,225 @@ static inline bool cookie_match(struct task_struct *a, struct task_struct *b)return a->core_cookie == b->core_cookie; } +/* +* Handler to attempt to enter kernel. It does nothing because the exit to +* usermode or guest mode will do the actual work (of waiting if needed). +*/ +static void sched_core_irq_work (struct irq_work *work) +{ +} + +static inline void init_sched_core_irq_work (struct rq *rq) +{ +init_irq_work (&rq->core_irq_work, sched_core_irq_work); +} + +/* +* sched_core_wait_till_safe - Pause the caller's hyperthread until the core +* exits the core-wide unsafe state. Obviously the CPU calling this function +* should not be responsible for the core being in the core-wide unsafe state +* otherwise it will deadlock. +* +* @ti_check: We spin here with IRQ enabled and preempt disabled. Break out of +* the loop if TIF flags are set and notify caller about it. +* +* IRQs should be disabled. +*/ +bool sched_core_wait_till_safe (unsigned long ti_check) +{ +bool restart = false; +struct rq *rq;
kernel/sched/core.c static inline bool cookie_match(struct task_struct *a, struct task_struct *b) · line 5153
Quoted 10 lines
kernel/sched/core.c +10
-0+int cpu; + +/* We clear the thread flag only at the end, so no need to check for it. */ + ti_check &= ~_TIF_UNSAFE_RET; + + cpu =smp_processor_id (); + rq =cpu_rq (cpu); + +if (!sched_core_enabled (rq)) + goto ret;
Why do we need to deal with ti_check if sched_core_enabled() is false (two statements above)
Quoted 40 lines
kernel/sched/core.c +40
-0+ +/* Down grade to allow interrupts to prevent stop_machine lockups.. */ +preempt_disable (); +local_irq_enable (); + +/* +* Wait till the core of this HT is not in an unsafe state. +* +* Pair with raw_spin_lock/unlock() in sched_core_unsafe_enter/exit(). +*/ +while (smp_load_acquire (&rq->core->core_unsafe_nest) >0 ) { +cpu_relax (); +if (READ_ONCE (current_thread_info ()->flags) & ti_check) { + restart = true; +break ; + } + } + +/* Upgrade it back to the expectations of entry code. */ +local_irq_disable (); +preempt_enable (); + +ret: +if (!restart) +clear_tsk_thread_flag (current,TIF_UNSAFE_RET ); + +return restart; +} + +/* +* Enter the core-wide IRQ state. Sibling will be paused if it is running +* 'untrusted' code, until sched_core_unsafe_exit() is called. Every attempt to +* avoid sending useless IPIs is made. Must be called only from hard IRQ +* context. +*/ +void sched_core_unsafe_enter (void ) +{ +const struct cpumask *smt_mask; +unsigned long flags; +struct rq *rq;
kernel/sched/core.c static inline bool cookie_match(struct task_struct *a, struct task_struct *b) · line 5203
Quoted 10 lines
kernel/sched/core.c +10
-0+int i, cpu; + +if (!static_branch_likely (&sched_core_protect_kernel)) +return ; + +local_irq_save (flags); + cpu =smp_processor_id (); + rq =cpu_rq (cpu); +if (!sched_core_enabled (rq)) + goto ret;
I am not sure about the rules of this check, do we have to do this with irq's disabled? Given that sched_core_enabled() can only change under stop_machine, can't we optimize this check?
Quoted 19 lines
kernel/sched/core.c +19
-0+ +/* Ensure that on return to user/guest, we check whether to wait. */ +if (current->core_cookie) +set_tsk_thread_flag (current,TIF_UNSAFE_RET ); + +/* Count unsafe_enter() calls received without unsafe_exit() on this CPU. */ + rq->core_this_unsafe_nest++; + +/* +* Should not nest: enter() should only pair with exit(). Both are done +* during the first entry into kernel and the last exit from kernel. +* Nested kernel entries (such as nested interrupts) will only trigger +* enter() and exit() on the outer most kernel entry and exit. +*/ +if (WARN_ON_ONCE (rq->core_this_unsafe_nest !=1 )) + goto ret; + +raw_spin_lock (rq_lockp (rq)); + smt_mask =cpu_smt_mask (cpu);
kernel/sched/core.c static inline bool cookie_match(struct task_struct *a, struct task_struct *b) · line 5232
Quoted 10 lines
kernel/sched/core.c +10
-0+ +/* +* Contribute this CPU's unsafe_enter() to the core-wide unsafe_enter() +* count. The raw_spin_unlock() release semantics pairs with the nest +* counter's smp_load_acquire() in sched_core_wait_till_safe(). +*/ +WRITE_ONCE (rq->core->core_unsafe_nest, rq->core->core_unsafe_nest +1 ); + +if (WARN_ON_ONCE (rq->core->core_unsafe_nest ==UINT_MAX )) + goto unlock;
I am sure this is quite unlikely unless your concerned about overflows, will this all eventually move to under SCHED_DEBUG?
Quoted 61 lines
kernel/sched/core.c +61
-0+ +if (irq_work_is_busy (&rq->core_irq_work)) { +/* +* Do nothing more since we are in an IPI sent from another +* sibling to enforce safety. That sibling would have sent IPIs +* to all of the HTs. +*/ + goto unlock; + } + +/* +* If we are not the first ones on the core to enter core-wide unsafe +* state, do nothing. +*/ +if (rq->core->core_unsafe_nest >1 ) + goto unlock; + +/* Do nothing more if the core is not tagged. */ +if (!rq->core->core_cookie) + goto unlock; + +for_each_cpu (i, smt_mask) { +struct rq *srq =cpu_rq (i); + +if (i == cpu ||cpu_is_offline (i)) +continue ; + +if (!srq->curr->mm ||is_task_rq_idle (srq->curr)) +continue ; + +/* Skip if HT is not running a tagged task. */ +if (!srq->curr->core_cookie && !srq->core_pick) +continue ; + +/* +* Force sibling into the kernel by IPI. If work was already +* pending, no new IPIs are sent. This is Ok since the receiver +* would already be in the kernel, or on its way to it. +*/ +irq_work_queue_on (&srq->core_irq_work, i); + } +unlock: +raw_spin_unlock (rq_lockp (rq)); +ret: +local_irq_restore (flags); +} + +/* +* Process any work need for either exiting the core-wide unsafe state, or for +* waiting on this hyperthread if the core is still in this state. +* +* @idle: Are we called from the idle loop? +*/ +void sched_core_unsafe_exit (void ) +{ +unsigned long flags; +unsigned int nest; +struct rq *rq; +int cpu; + +if (!static_branch_likely (&sched_core_protect_kernel))
kernel/sched/core.c static inline bool cookie_match(struct task_struct *a, struct task_struct *b) · line 5303
Quoted 10 lines
kernel/sched/core.c +10
-0+return ; + +local_irq_save (flags); + cpu =smp_processor_id (); + rq =cpu_rq (cpu); + +/* Do nothing if core-sched disabled. */ +if (!sched_core_enabled (rq)) + goto ret; +
Same as above
Quoted 56 lines
kernel/sched/core.c +32
-0+/* +* Can happen when a process is forked and the first return to user +* mode is a syscall exit. Either way, there's nothing to do. +*/ +if (rq->core_this_unsafe_nest ==0 ) + goto ret; + + rq->core_this_unsafe_nest--; + +/* enter() should be paired with exit() only. */ +if (WARN_ON_ONCE (rq->core_this_unsafe_nest !=0 )) + goto ret; + +raw_spin_lock (rq_lockp (rq)); +/* +* Core-wide nesting counter can never be 0 because we are +* still in it on this CPU. +*/ + nest = rq->core->core_unsafe_nest; +WARN_ON_ONCE (!nest); + +WRITE_ONCE (rq->core->core_unsafe_nest, nest -1 ); +/* +* The raw_spin_unlock release semantics pairs with the nest counter's +* smp_load_acquire() in sched_core_wait_till_safe(). +*/ +raw_spin_unlock (rq_lockp (rq)); +ret: +local_irq_restore (flags); +} +// XXX fairness/fwd progress conditions /* * Returns @@ -5497,6 +5737,7 @@ static inline void sched_core_cpu_starting(unsigned int cpu) rq =cpu_rq (i);if (rq->core && rq->core == rq) core_rq = rq; +init_sched_core_irq_work (rq); }if (!core_rq)kernel/sched/sched.h +2
-0@@ -1074,6 +1074,8 @@ struct rq {unsigned int core_enabled;unsigned int core_sched_seq;struct rb_root core_tree; +struct irq_work core_irq_work;/* To force HT into kernel */ +unsigned int core_this_unsafe_nest;/* shared state */ unsigned int core_task_seq;
kernel/sched/sched.h struct rq { · line 1089
Quoted 7 lines
kernel/sched/sched.h +1
-0@@ -1081,6 +1083,7 @@ struct rq {unsigned long core_cookie;unsigned char core_forceidle;unsigned int core_forceidle_seq; +unsigned int core_unsafe_nest;#endif };
Balbir Singh.
PATCH 19/32 entry/idle: Enter and exit kernel protection during idle entry and exit
In reply to Joel Fernandes (Google)Add a generic_idle_{enter,exit} helper function to enter and exit kernel protection when entering and exiting idle, respectively.
While at it, remove a stale RCU comment.
--- include/linux/entry-common.h | 18 ++++++++++++++++++ kernel/sched/idle.c | 11 ++++++----- 2 files changed, 24 insertions(+), 5 deletions(-)
include/linux/entry-common.h +18 -0
@@ -454,4 +454,22 @@ static inline bool entry_kernel_protected(void)return IS_ENABLED (CONFIG_SCHED_CORE ) &&sched_core_kernel_protected () && _TIF_UNSAFE_RET != 0; } + +/** +* generic_idle_enter - General tasks to perform during idle entry. +*/ +static inline void generic_idle_enter (void ) +{ +/* Entering idle ends the protected kernel region. */ +sched_core_unsafe_exit (); +} + +/** +* generic_idle_exit - General tasks to perform during idle exit. +*/ +static inline void generic_idle_exit (void ) +{ +/* Exiting idle (re)starts the protected kernel region. */ +sched_core_unsafe_enter (); +}#endif
kernel/sched/idle.c +6 -5
@@ -8,6 +8,7 @@*/ #include "sched.h" +#include <linux/entry-common.h> #include <trace/events/power.h> /* Linker adds these: start and end of __cpuidle functions */
@@ -54,6 +55,7 @@ __setup("hlt", cpu_idle_nopoll_setup); static noinline int __cpuidle cpu_idle_poll(void) { +generic_idle_enter ();trace_cpu_idle (0 ,smp_processor_id ());stop_critical_timings ();rcu_idle_enter ();
@@ -66,6 +68,7 @@ static noinline int __cpuidle cpu_idle_poll(void)rcu_idle_exit ();start_critical_timings ();trace_cpu_idle (PWR_EVENT_EXIT ,smp_processor_id ()); +generic_idle_exit ();return 1 ; }
@@ -156,11 +159,7 @@ static void cpuidle_idle_call(void)return ; } -/* -* The RCU framework needs to be told that we are entering an idle -* section, so no more rcu read side critical sections and one more -* step to the grace period -*/ +generic_idle_enter ();if (cpuidle_not_available (drv, dev)) {tick_nohz_idle_stop_tick ();
@@ -225,6 +224,8 @@ static void cpuidle_idle_call(void)*/ if (WARN_ON_ONCE (irqs_disabled ()))local_irq_enable (); + +generic_idle_exit (); }/*
Signature · 1 line
2.29.2.299.gdc1121823c-goog
Peter Zijlstra
maintainerRe: [PATCH -tip 19/32] entry/idle: Enter and exit kernel protection during idle entry and exit
Quoted 30 lines · Joel Fernandes (Google)
Add a generic_idle_{enter,exit} helper function to enter and exit kernel protection when entering and exiting idle, respectively.
While at it, remove a stale RCU comment.
Reviewed-by Alexandre Chartre <[email protected]>Tested-by Julien Desfossez <[email protected]>Signed-off-by Joel Fernandes (Google) <[email protected]>--- include/linux/entry-common.h | 18 ++++++++++++++++++ kernel/sched/idle.c | 11 ++++++----- 2 files changed, 24 insertions(+), 5 deletions(-)include/linux/entry-common.h +9
-0@@ -454,4 +454,22 @@ static inline bool entry_kernel_protected(void)return IS_ENABLED (CONFIG_SCHED_CORE ) &&sched_core_kernel_protected () && _TIF_UNSAFE_RET !=0 ; } + +/** +* generic_idle_enter - General tasks to perform during idle entry. +*/ +static inline void generic_idle_enter (void ) +{ +/* Entering idle ends the protected kernel region. */ +sched_core_unsafe_exit (); +}
include/linux/entry-common.h static inline bool entry_kernel_protected(void) · line 475
Quoted 10 lines
include/linux/entry-common.h +9
-0+ +/** +* generic_idle_exit - General tasks to perform during idle exit. +*/ +static inline void generic_idle_exit (void ) +{ +/* Exiting idle (re)starts the protected kernel region. */ +sched_core_unsafe_enter (); +}#endif
That naming is terrible..
Quoted 42 lines
kernel/sched/idle.c +4
-5@@ -8,6 +8,7 @@*/ #include "sched.h" +#include <linux/entry-common.h> #include <trace/events/power.h> /* Linker adds these: start and end of __cpuidle functions */ @@ -54,6 +55,7 @@ __setup("hlt", cpu_idle_nopoll_setup); static noinline int __cpuidle cpu_idle_poll(void) { +generic_idle_enter ();trace_cpu_idle (0 ,smp_processor_id ());stop_critical_timings ();rcu_idle_enter ();@@ -66,6 +68,7 @@ static noinline int __cpuidle cpu_idle_poll(void)rcu_idle_exit ();start_critical_timings ();trace_cpu_idle (PWR_EVENT_EXIT ,smp_processor_id ()); +generic_idle_exit ();return 1 ; }@@ -156,11 +159,7 @@ static void cpuidle_idle_call(void)return ; } -/* -* The RCU framework needs to be told that we are entering an idle -* section, so no more rcu read side critical sections and one more -* step to the grace period -*/ +generic_idle_enter ();if (cpuidle_not_available (drv, dev)) {tick_nohz_idle_stop_tick ();
kernel/sched/idle.c static void cpuidle_idle_call(void) · line 229
Quoted 6 lines
kernel/sched/idle.c +2
-0@@ -225,6 +224,8 @@ static void cpuidle_idle_call(void)*/ if (WARN_ON_ONCE (irqs_disabled ()))local_irq_enable (); + +generic_idle_exit (); }
I'm confused.. arch_cpu_idle_{enter,exit}() weren't conveniently placed for you?
Re: [PATCH -tip 19/32] entry/idle: Enter and exit kernel protection during idle entry and exit
Quoted 35 lines · Peter Zijlstra
Quoted 34 lines · Joel Fernandes (Google)
Add a generic_idle_{enter,exit} helper function to enter and exit kernel protection when entering and exiting idle, respectively.
While at it, remove a stale RCU comment.
Reviewed-by Alexandre Chartre <[email protected]>Tested-by Julien Desfossez <[email protected]>Signed-off-by Joel Fernandes (Google) <[email protected]>--- include/linux/entry-common.h | 18 ++++++++++++++++++ kernel/sched/idle.c | 11 ++++++----- 2 files changed, 24 insertions(+), 5 deletions(-)include/linux/entry-common.h +13
-0@@ -454,4 +454,22 @@ static inline bool entry_kernel_protected(void)return IS_ENABLED (CONFIG_SCHED_CORE ) &&sched_core_kernel_protected () && _TIF_UNSAFE_RET !=0 ; } + +/** +* generic_idle_enter - General tasks to perform during idle entry. +*/ +static inline void generic_idle_enter (void ) +{ +/* Entering idle ends the protected kernel region. */ +sched_core_unsafe_exit (); +} + +/** +* generic_idle_exit - General tasks to perform during idle exit. +*/
Quoted 8 lines
include/linux/entry-common.h static inline bool entry_kernel_protected(void) · line 475
Quoted 6 lines
include/linux/entry-common.h +5
-0+static inline void generic_idle_exit (void ) +{ +/* Exiting idle (re)starts the protected kernel region. */ +sched_core_unsafe_enter (); +}#endif That naming is terrible..
Yeah sorry :-\. The naming I chose was to be aligned with the CONFIG_GENERIC_ENTRY naming. I am open to ideas on that.
Quoted 39 lines
Quoted 39 lines
kernel/sched/idle.c +4
-5@@ -8,6 +8,7 @@*/ #include "sched.h" +#include <linux/entry-common.h> #include <trace/events/power.h> /* Linker adds these: start and end of __cpuidle functions */ @@ -54,6 +55,7 @@ __setup("hlt", cpu_idle_nopoll_setup); static noinline int __cpuidle cpu_idle_poll(void) { +generic_idle_enter ();trace_cpu_idle (0 ,smp_processor_id ());stop_critical_timings ();rcu_idle_enter ();@@ -66,6 +68,7 @@ static noinline int __cpuidle cpu_idle_poll(void)rcu_idle_exit ();start_critical_timings ();trace_cpu_idle (PWR_EVENT_EXIT ,smp_processor_id ()); +generic_idle_exit ();return 1 ; }@@ -156,11 +159,7 @@ static void cpuidle_idle_call(void)return ; } -/* -* The RCU framework needs to be told that we are entering an idle -* section, so no more rcu read side critical sections and one more -* step to the grace period -*/ +generic_idle_enter ();
Quoted 12 lines
kernel/sched/idle.c static void cpuidle_idle_call(void) · line 229
Quoted 9 lines
kernel/sched/idle.c +2
-0if (cpuidle_not_available (drv, dev)) {tick_nohz_idle_stop_tick ();@@ -225,6 +224,8 @@ static void cpuidle_idle_call(void)*/ if (WARN_ON_ONCE (irqs_disabled ()))local_irq_enable (); + +generic_idle_exit (); }I'm confused.. arch_cpu_idle_{enter,exit}() weren't conveniently placed for you?
The way this patch series works, it does not depend on arch code as much as possible. Since there are other arch that may need this patchset such as ARM, it may be better to keep it in the generic entry code. Thoughts?
thanks,
- Joel
Peter Zijlstra
maintainerRe: [PATCH -tip 19/32] entry/idle: Enter and exit kernel protection during idle entry and exit
Quoted 10 lines · Joel Fernandes
Quoted 6 lines · Peter Zijlstra
include/linux/entry-common.h static inline bool entry_kernel_protected(void) · line 470
Quoted 2 lines · Joel Fernandes (Google)
include/linux/entry-common.h +2
-0+static inline void generic_idle_enter(void) +static inline void generic_idle_exit(void)That naming is terrible..
Yeah sorry :-\. The naming I chose was to be aligned with the
CONFIG_GENERIC_ENTRYnaming. I am open to ideas on that.
entry_idle_{enter,exit}() ?
Quoted 6 lines
Quoted 2 lines
I'm confused.. arch_cpu_idle_{enter,exit}() weren't conveniently placed for you?
The way this patch series works, it does not depend on arch code as much as possible. Since there are other arch that may need this patchset such as ARM, it may be better to keep it in the generic entry code. Thoughts?
I didn't necessarily mean using those hooks, even placing your new hooks right next to them would've covered the exact same code with less lines modified.
Re: [PATCH -tip 19/32] entry/idle: Enter and exit kernel protection during idle entry and exit
Quoted 9 lines · Peter Zijlstra
Quoted 8 lines · Joel Fernandes
Quoted 6 lines · Peter Zijlstra
include/linux/entry-common.h static inline bool entry_kernel_protected(void) · line 470
Quoted 2 lines · Joel Fernandes (Google)
include/linux/entry-common.h +2
-0+static inline void generic_idle_enter(void) +static inline void generic_idle_exit(void)That naming is terrible..
Quoted 4 lines
Quoted 2 lines
Yeah sorry :-\. The naming I chose was to be aligned with the
CONFIG_GENERIC_ENTRYnaming. I am open to ideas on that.entry_idle_{enter,exit}() ?
Sounds good to me.
Quoted 10 lines
Quoted 6 lines
Quoted 2 lines
I'm confused.. arch_cpu_idle_{enter,exit}() weren't conveniently placed for you?
The way this patch series works, it does not depend on arch code as much as possible. Since there are other arch that may need this patchset such as ARM, it may be better to keep it in the generic entry code. Thoughts?
I didn't necessarily mean using those hooks, even placing your new hooks right next to them would've covered the exact same code with less lines modified.
Ok sure. I will improve it this way for next posting.
thanks,
- Joel
PATCH 20/32 entry/kvm: Protect the kernel when entering from guest
In reply to Joel Fernandes (Google)From: Vineeth Pillai <[email protected]>
Similar to how user to kernel mode transitions are protected in earlier patches, protect the entry into kernel from guest mode as well.
--- arch/x86/kvm/x86.c | 2 ++ include/linux/entry-kvm.h | 12 ++++++++++++ kernel/entry/kvm.c | 33 +++++++++++++++++++++++++++++++++ 3 files changed, 47 insertions(+)
arch/x86/kvm/x86.c +2 -0
@@ -8910,6 +8910,7 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)*/ smp_mb__after_srcu_read_unlock (); +kvm_exit_to_guest_mode ();/* * This handles the case where a posted interrupt was * notified with kvm_vcpu_kick.
@@ -9003,6 +9004,7 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu) } } +kvm_enter_from_guest_mode ();local_irq_enable ();preempt_enable ();
include/linux/entry-kvm.h +12 -0
@@ -77,4 +77,16 @@ static inline bool xfer_to_guest_mode_work_pending(void) }#endif /* CONFIG_KVM_XFER_TO_GUEST_WORK */ +/** +* kvm_enter_from_guest_mode - Hook called just after entering kernel from guest. +* Caller should ensure interrupts are off. +*/ +void kvm_enter_from_guest_mode (void ); + +/** +* kvm_exit_to_guest_mode - Hook called just before entering guest from kernel. +* Caller should ensure interrupts are off. +*/ +void kvm_exit_to_guest_mode (void ); +#endif
kernel/entry/kvm.c +33 -0
@@ -50,3 +50,36 @@ int xfer_to_guest_mode_handle_work(struct kvm_vcpu *vcpu)return xfer_to_guest_mode_work (vcpu, ti_work); }EXPORT_SYMBOL_GPL (xfer_to_guest_mode_handle_work); + +/** +* kvm_enter_from_guest_mode - Hook called just after entering kernel from guest. +* Caller should ensure interrupts are off. +*/ +void kvm_enter_from_guest_mode (void ) +{ +if (!entry_kernel_protected ()) +return ; +sched_core_unsafe_enter (); +} +EXPORT_SYMBOL_GPL (kvm_enter_from_guest_mode); + +/** +* kvm_exit_to_guest_mode - Hook called just before entering guest from kernel. +* Caller should ensure interrupts are off. +*/ +void kvm_exit_to_guest_mode (void ) +{ +if (!entry_kernel_protected ()) +return ; +sched_core_unsafe_exit (); + +/* +* Wait here instead of in xfer_to_guest_mode_handle_work(). The reason +* is because in vcpu_run(), xfer_to_guest_mode_handle_work() is called +* when a vCPU was either runnable or blocked. However, we only care +* about the runnable case (VM entry/exit) which is handled by +* vcpu_enter_guest(). +*/ +sched_core_wait_till_safe (XFER_TO_GUEST_MODE_WORK ); +} +EXPORT_SYMBOL_GPL (kvm_exit_to_guest_mode);
Signature · 1 line
2.29.2.299.gdc1121823c-goog
PATCH 21/32 sched: CGroup tagging interface for core scheduling
In reply to Joel Fernandes (Google)From: Peter Zijlstra <[email protected]>
Marks all tasks in a cgroup as matching for core-scheduling.
A task will need to be moved into the core scheduler queue when the cgroup it belongs to is tagged to run with core scheduling. Similarly the task will need to be moved out of the core scheduler queue when the cgroup is untagged.
Also after we forked a task, its core scheduler queue's presence will need to be updated according to its new cgroup's status.
Use stop machine mechanism to update all tasks in a cgroup to prevent a new task from sneaking into the cgroup, and missed out from the update while we iterates through all the tasks in the cgroup. A more complicated scheme could probably avoid the stop machine. Such scheme will also need to resovle inconsistency between a task's cgroup core scheduling tag and residency in core scheduler queue.
We are opting for the simple stop machine mechanism for now that avoids such complications.
Core scheduler has extra overhead. Enable it only for core with more than one SMT hardware threads.
--- kernel/sched/core.c | 183 +++++++++++++++++++++++++++++++++++++++++-- kernel/sched/sched.h | 4 + 2 files changed, 181 insertions(+), 6 deletions(-)
kernel/sched/core.c +177 -6
@@ -157,6 +157,37 @@ static inline bool __sched_core_less(struct task_struct *a, struct task_struct *return false; } +static bool sched_core_empty (struct rq *rq) +{ +return RB_EMPTY_ROOT (&rq->core_tree); +} + +static bool sched_core_enqueued (struct task_struct *task) +{ +return !RB_EMPTY_NODE (&task->core_node); +} + +static struct task_struct *sched_core_first (struct rq *rq) +{ +struct task_struct *task; + + task = container_of(rb_first(&rq->core_tree), struct task_struct, core_node); +return task; +} + +static void sched_core_flush (int cpu) +{ +struct rq *rq =cpu_rq (cpu); +struct task_struct *task; + +while (!sched_core_empty (rq)) { + task =sched_core_first (rq); +rb_erase (&task->core_node, &rq->core_tree); +RB_CLEAR_NODE (&task->core_node); + } + rq->core->core_task_seq++; +} +static void sched_core_enqueue (struct rq *rq,struct task_struct *p) {struct rb_node *parent, **node;
@@ -188,10 +219,11 @@ static void sched_core_dequeue(struct rq *rq, struct task_struct *p) { rq->core->core_task_seq++; -if (!p->core_cookie) +if (!sched_core_enqueued (p))return ;rb_erase (&p->core_node, &rq->core_tree); +RB_CLEAR_NODE (&p->core_node); }/*
@@ -255,8 +287,24 @@ static int __sched_core_stopper(void *data)bool enabled = !!(unsigned long )data;int cpu; -for_each_possible_cpu (cpu) -cpu_rq (cpu)->core_enabled = enabled; +for_each_possible_cpu (cpu) { +struct rq *rq =cpu_rq (cpu); + +WARN_ON_ONCE (enabled == rq->core_enabled); + +if (!enabled || (enabled &&cpumask_weight (cpu_smt_mask (cpu)) >=2 )) { +/* +* All active and migrating tasks will have already +* been removed from core queue when we clear the +* cgroup tags. However, dying tasks could still be +* left in core queue. Flush them here. +*/ +if (!enabled) +sched_core_flush (cpu); + + rq->core_enabled = enabled; + } + }return 0 ; }
@@ -266,7 +314,11 @@ static int sched_core_count;static void __sched_core_enable (void ) { -// XXX verify there are no cookie tasks (yet) +int cpu; + +/* verify there are no cookie tasks (yet) */ +for_each_online_cpu (cpu) +BUG_ON (!sched_core_empty (cpu_rq (cpu)));static_branch_enable (&__sched_core_enabled);stop_machine (__sched_core_stopper, (void *)true,NULL );
@@ -274,8 +326,6 @@ static void __sched_core_enable(void)static void __sched_core_disable (void ) { -// XXX verify there are no cookie tasks (left) -stop_machine (__sched_core_stopper, (void *)false,NULL );static_branch_disable (&__sched_core_enabled); }
@@ -300,6 +350,7 @@ void sched_core_put(void)static inline void sched_core_enqueue (struct rq *rq,struct task_struct *p) { }static inline void sched_core_dequeue (struct rq *rq,struct task_struct *p) { } +static bool sched_core_enqueued (struct task_struct *task) {return false; }#endif /* CONFIG_SCHED_CORE */
@@ -3978,6 +4029,9 @@ int sched_fork(unsigned long clone_flags, struct task_struct *p)#ifdef CONFIG_SMP plist_node_init (&p->pushable_tasks,MAX_PRIO );RB_CLEAR_NODE (&p->pushable_dl_tasks); +#endif +#ifdef CONFIG_SCHED_CORE +RB_CLEAR_NODE (&p->core_node);#endif return 0 ; }
@@ -7979,6 +8033,9 @@ void init_idle(struct task_struct *idle, int cpu)#ifdef CONFIG_SMP sprintf (idle->comm,"%s/%d" ,INIT_TASK_COMM , cpu);#endif +#ifdef CONFIG_SCHED_CORE +RB_CLEAR_NODE (&idle->core_node); +#endif }#ifdef CONFIG_SMP
@@ -8995,6 +9052,15 @@ static void sched_change_group(struct task_struct *tsk, int type) tg =container_of (task_css_check (tsk, cpu_cgrp_id, true), struct task_group, css); tg =autogroup_task_group (tsk, tg); + +#ifdef CONFIG_SCHED_CORE +if ((unsigned long )tsk->sched_task_group == tsk->core_cookie) + tsk->core_cookie =0UL ; + +if (tg->tagged/* && !tsk->core_cookie ? */ ) + tsk->core_cookie = (unsigned long )tg; +#endif + tsk->sched_task_group = tg;#ifdef CONFIG_FAIR_GROUP_SCHED
@@ -9087,6 +9153,18 @@ static int cpu_cgroup_css_online(struct cgroup_subsys_state *css)return 0 ; } +static void cpu_cgroup_css_offline (struct cgroup_subsys_state *css) +{ +#ifdef CONFIG_SCHED_CORE +struct task_group *tg =css_tg (css); + +if (tg->tagged) { +sched_core_put (); + tg->tagged =0 ; + } +#endif +} +static void cpu_cgroup_css_released (struct cgroup_subsys_state *css) {struct task_group *tg =css_tg (css);
@@ -9652,6 +9730,82 @@ static u64 cpu_rt_period_read_uint(struct cgroup_subsys_state *css, }#endif /* CONFIG_RT_GROUP_SCHED */ +#ifdef CONFIG_SCHED_CORE +static u64 cpu_core_tag_read_u64 (struct cgroup_subsys_state *css,struct cftype *cft) +{ +struct task_group *tg =css_tg (css); + +return !!tg->tagged; +} + +struct write_core_tag { +struct cgroup_subsys_state *css; +int val; +}; + +static int __sched_write_tag (void *data) +{ +struct write_core_tag *tag = (struct write_core_tag *) data; +struct cgroup_subsys_state *css = tag->css; +int val = tag->val; +struct task_group *tg =css_tg (tag->css); +struct css_task_iter it; +struct task_struct *p; + + tg->tagged = !!val; + +css_task_iter_start (css,0 , &it); +/* +* Note: css_task_iter_next will skip dying tasks. +* There could still be dying tasks left in the core queue +* when we set cgroup tag to 0 when the loop is done below. +*/ +while ((p =css_task_iter_next (&it))) { + p->core_cookie = !!val ? (unsigned long )tg :0UL ; + +if (sched_core_enqueued (p)) { +sched_core_dequeue (task_rq (p), p); +if (!p->core_cookie) +continue ; + } + +if (sched_core_enabled (task_rq (p)) && + p->core_cookie &&task_on_rq_queued (p)) +sched_core_enqueue (task_rq (p), p); + + } +css_task_iter_end (&it); + +return 0 ; +} + +static int cpu_core_tag_write_u64 (struct cgroup_subsys_state *css,struct cftype *cft,u64 val) +{ +struct task_group *tg =css_tg (css); +struct write_core_tag wtag; + +if (val >1 ) +return -ERANGE ; + +if (!static_branch_likely (&sched_smt_present)) +return -EINVAL ; + +if (tg->tagged == !!val) +return 0 ; + +if (!!val) +sched_core_get (); + + wtag.css = css; + wtag.val = val; +stop_machine (__sched_write_tag, (void *) &wtag,NULL ); +if (!val) +sched_core_put (); + +return 0 ; +} +#endif + static struct cftype cpu_legacy_files[] = {#ifdef CONFIG_FAIR_GROUP_SCHED {
@@ -9688,6 +9842,14 @@ static struct cftype cpu_legacy_files[] = { .write_u64 = cpu_rt_period_write_uint, },#endif +#ifdef CONFIG_SCHED_CORE + { + .name = "tag", + .flags = CFTYPE_NOT_ON_ROOT, + .read_u64 = cpu_core_tag_read_u64, + .write_u64 = cpu_core_tag_write_u64, + }, +#endif #ifdef CONFIG_UCLAMP_TASK_GROUP { .name = "uclamp.min",
@@ -9861,6 +10023,14 @@ static struct cftype cpu_files[] = { .write_s64 = cpu_weight_nice_write_s64, },#endif +#ifdef CONFIG_SCHED_CORE + { + .name = "tag", + .flags = CFTYPE_NOT_ON_ROOT, + .read_u64 = cpu_core_tag_read_u64, + .write_u64 = cpu_core_tag_write_u64, + }, +#endif #ifdef CONFIG_CFS_BANDWIDTH { .name = "max",
@@ -9889,6 +10059,7 @@ static struct cftype cpu_files[] = {struct cgroup_subsys cpu_cgrp_subsys = { .css_alloc = cpu_cgroup_css_alloc, .css_online = cpu_cgroup_css_online, + .css_offline = cpu_cgroup_css_offline, .css_released = cpu_cgroup_css_released, .css_free = cpu_cgroup_css_free, .css_extra_stat_show = cpu_extra_stat_show,
kernel/sched/sched.h +4 -0
@@ -377,6 +377,10 @@ struct cfs_bandwidth {struct task_group {struct cgroup_subsys_state css; +#ifdef CONFIG_SCHED_CORE +int tagged; +#endif +#ifdef CONFIG_FAIR_GROUP_SCHED /* schedulable entities of this group on each CPU */ struct sched_entity **se;
Signature · 1 line
2.29.2.299.gdc1121823c-goog
PATCH 22/32 sched: Split the cookie and setup per-task cookie on fork
In reply to Joel Fernandes (Google)In order to prevent interference and clearly support both per-task and CGroup APIs, split the cookie into 2 and allow it to be set from either per-task, or CGroup API. The final cookie is the combined value of both and is computed when the stop-machine executes during a change of cookie.
Also, for the per-task cookie, it will get weird if we use pointers of any emphemeral objects. For this reason, introduce a refcounted object who's sole purpose is to assign unique cookie value by way of the object's pointer.
While at it, refactor the CGroup code a bit. Future patches will introduce more APIs and support.
--- include/linux/sched.h | 2 + kernel/sched/core.c | 241 ++++++++++++++++++++++++++++++++++++++++-- kernel/sched/debug.c | 4 + 3 files changed, 236 insertions(+), 11 deletions(-)
include/linux/sched.h +2 -0
@@ -688,6 +688,8 @@ struct task_struct {#ifdef CONFIG_SCHED_CORE struct rb_node core_node;unsigned long core_cookie; +unsigned long core_task_cookie; +unsigned long core_group_cookie;unsigned int core_occupation;#endif
kernel/sched/core.c +230 -11
271 lines not on this page · raw message
kernel/sched/debug.c +4 -0
@@ -1024,6 +1024,10 @@ void proc_sched_show_task(struct task_struct *p, struct pid_namespace *ns,__PS ("clock-delta" , t1-t0); } +#ifdef CONFIG_SCHED_CORE +__PS ("core_cookie" , p->core_cookie); +#endif +sched_show_numa (p, m); }
Signature · 1 line
2.29.2.299.gdc1121823c-goog
Peter Zijlstra
maintainerRe: [PATCH -tip 22/32] sched: Split the cookie and setup per-task cookie on fork
Quoted 3 lines · Joel Fernandes (Google)
Also, for the per-task cookie, it will get weird if we use pointers of any emphemeral objects. For this reason, introduce a refcounted object who's sole purpose is to assign unique cookie value by way of the object's pointer.
Might be useful to explain why exactly none of the many pid_t's are good enough.
Re: [PATCH -tip 22/32] sched: Split the cookie and setup per-task cookie on fork
Quoted 7 lines · Peter Zijlstra
Quoted 3 lines · Joel Fernandes (Google)
Also, for the per-task cookie, it will get weird if we use pointers of any emphemeral objects. For this reason, introduce a refcounted object who's sole purpose is to assign unique cookie value by way of the object's pointer.
Might be useful to explain why exactly none of the many pid_t's are good enough.
I thought about this already and it does not seem a good fit. When 2 processes share, it is possible that more processes are added to that logical group. Then the original processes that share die, but if we hold on to the pid_t or task_struct, that would be awkward. It seemed introducing a new refcounted struct was the right way to go. I can add these details to the change log.
thanks!
- Joel
Peter Zijlstra
maintainerRe: [PATCH -tip 22/32] sched: Split the cookie and setup per-task cookie on fork
In reply to Joel Fernandes (Google)kernel/sched/core.c static u64 cpu_rt_period_read_uint(struct cgroup_subsys_state *css, · line 9779
Quoted 13 lines · Joel Fernandes (Google)
kernel/sched/core.c +13
-0+void sched_core_tag_requeue (struct task_struct *p,unsigned long cookie,bool group) +{ +if (!p) +return ; + +if (group) + p->core_group_cookie = cookie; +else + p->core_task_cookie = cookie; + +/* Use up half of the cookie's bits for task cookie and remaining for group cookie. */ + p->core_cookie = (p->core_task_cookie << + (sizeof (unsigned long ) *4 )) + p->core_group_cookie;
This seems dangerous; afaict there is nothing that prevents cookie collision.
Re: [PATCH -tip 22/32] sched: Split the cookie and setup per-task cookie on fork
Quoted 17 lines · Peter Zijlstra
kernel/sched/core.c static u64 cpu_rt_period_read_uint(struct cgroup_subsys_state *css, · line 9779
Quoted 13 lines · Joel Fernandes (Google)
kernel/sched/core.c +13
-0+void sched_core_tag_requeue (struct task_struct *p,unsigned long cookie,bool group) +{ +if (!p) +return ; + +if (group) + p->core_group_cookie = cookie; +else + p->core_task_cookie = cookie; + +/* Use up half of the cookie's bits for task cookie and remaining for group cookie. */ + p->core_cookie = (p->core_task_cookie << + (sizeof (unsigned long ) *4 )) + p->core_group_cookie;This seems dangerous; afaict there is nothing that prevents cookie collision.
This is fixed in a later patch by Josh "sched: Refactor core cookie into struct" where we are having independent fields for each type of cookie.
I'll squash it next time I post to prevent confusion. Thanks,
- Joel
Peter Zijlstra
maintainerRe: [PATCH -tip 22/32] sched: Split the cookie and setup per-task cookie on fork
Quoted 19 lines · Joel Fernandes
Quoted 17 lines · Peter Zijlstra
kernel/sched/core.c static u64 cpu_rt_period_read_uint(struct cgroup_subsys_state *css, · line 9779
Quoted 13 lines · Joel Fernandes (Google)
kernel/sched/core.c +13
-0+void sched_core_tag_requeue (struct task_struct *p,unsigned long cookie,bool group) +{ +if (!p) +return ; + +if (group) + p->core_group_cookie = cookie; +else + p->core_task_cookie = cookie; + +/* Use up half of the cookie's bits for task cookie and remaining for group cookie. */ + p->core_cookie = (p->core_task_cookie << + (sizeof (unsigned long ) *4 )) + p->core_group_cookie;This seems dangerous; afaict there is nothing that prevents cookie collision.
Quoted 2 lines
This is fixed in a later patch by Josh "sched: Refactor core cookie into struct" where we are having independent fields for each type of cookie.
So I don't think that later patch is right... That is, it works, but afaict it's massive overkill.
COOKIE_CMP_RETURN(task_cookie); COOKIE_CMP_RETURN(group_cookie); COOKIE_CMP_RETURN(color);
So if task_cookie matches, we consider group_cookie, if that matches we consider color.
Now, afaict that's semantically exactly the same as just using the narrowest cookie. That is, use the task cookie if there is, and then, walking the cgroup hierarchy (up) pick the first cgroup cookie.
(I don't understand the color thing, but lets have that discussion in that subthread)
Which means you only need a single active cookie field.
IOW, you're just making things complicated and expensive.
Re: [PATCH -tip 22/32] sched: Split the cookie and setup per-task cookie on fork
Quoted 15 lines · Peter Zijlstra
So I don't think that later patch is right... That is, it works, but afaict it's massive overkill.
COOKIE_CMP_RETURN(task_cookie); COOKIE_CMP_RETURN(group_cookie); COOKIE_CMP_RETURN(color);So if
task_cookiematches, we considergroup_cookie, if that matches we consider color.Now, afaict that's semantically exactly the same as just using the narrowest cookie. That is, use the task cookie if there is, and then, walking the cgroup hierarchy (up) pick the first cgroup cookie.
Quoted 7 lines
(I don't understand the color thing, but lets have that discussion in that subthread)
Which means you only need a single active cookie field.
IOW, you're just making things complicated and expensive.
For the per-task interface, I believe we still want to prevent two tasks that share a task cookie from sharing an overall cookie if they are in two separately tagged groups (Joel please correct me if I'm mistaken there). That's why in Joel's older patch, the overall cookie was a combination of the task and group cookies. My concern about that was the potential cookie collision.
I followed up on the 'color' portion in the other thread.
Thanks,
Josh
Peter Zijlstra
maintainerRe: [PATCH -tip 22/32] sched: Split the cookie and setup per-task cookie on fork
Quoted 24 lines · Josh Don
Quoted 15 lines · Peter Zijlstra
So I don't think that later patch is right... That is, it works, but afaict it's massive overkill.
COOKIE_CMP_RETURN(task_cookie); COOKIE_CMP_RETURN(group_cookie); COOKIE_CMP_RETURN(color);So if
task_cookiematches, we considergroup_cookie, if that matches we consider color.Now, afaict that's semantically exactly the same as just using the narrowest cookie. That is, use the task cookie if there is, and then, walking the cgroup hierarchy (up) pick the first cgroup cookie.
Quoted 7 lines
(I don't understand the color thing, but lets have that discussion in that subthread)
Which means you only need a single active cookie field.
IOW, you're just making things complicated and expensive.
Quoted 6 lines
For the per-task interface, I believe we still want to prevent two tasks that share a task cookie from sharing an overall cookie if they are in two separately tagged groups (Joel please correct me if I'm mistaken there). That's why in Joel's older patch, the overall cookie was a combination of the task and group cookies. My concern about that was the potential cookie collision.
Then disallow sharing a task cookie when the tasks are in different cgroups or disallow cgroup movement when they share a cookie.
Re: [PATCH -tip 22/32] sched: Split the cookie and setup per-task cookie on fork
Quoted 3 lines · Peter Zijlstra
Then disallow sharing a task cookie when the tasks are in different cgroups or disallow cgroup movement when they share a cookie.
Yes, we could restrict task cookie sharing to tasks that are in the same cgroup. Then the cookie easily just becomes a single value; either the task cookie or group cookie.
The advantage of the approach with the cookie struct is that it is easily extensible, and allows for trust models that don't conform exactly to the cgroup hierarchy (ie. our discussion on cookie color). The overhead of the approach seems tolerable, given that updates to a task's cookie are not in fast paths (ie. prctl, setting cgroup cookie, sched_move_task). Are you more concerned with the added complexity of maintaining the RB tree, refcounts, etc?
Re: [PATCH -tip 22/32] sched: Split the cookie and setup per-task cookie on fork
In reply to Peter ZijlstraHi Peter,
Quoted 26 lines · Peter Zijlstra
Quoted 19 lines · Joel Fernandes
Quoted 17 lines · Peter Zijlstra
kernel/sched/core.c static u64 cpu_rt_period_read_uint(struct cgroup_subsys_state *css, · line 9779
Quoted 13 lines · Joel Fernandes (Google)
kernel/sched/core.c +13
-0+void sched_core_tag_requeue (struct task_struct *p,unsigned long cookie,bool group) +{ +if (!p) +return ; + +if (group) + p->core_group_cookie = cookie; +else + p->core_task_cookie = cookie; + +/* Use up half of the cookie's bits for task cookie and remaining for group cookie. */ + p->core_cookie = (p->core_task_cookie << + (sizeof (unsigned long ) *4 )) + p->core_group_cookie;This seems dangerous; afaict there is nothing that prevents cookie collision.
Quoted 2 lines
This is fixed in a later patch by Josh "sched: Refactor core cookie into struct" where we are having independent fields for each type of cookie.
So I don't think that later patch is right... That is, it works, but afaict it's massive overkill.
Quoted 6 lines
COOKIE_CMP_RETURN(task_cookie); COOKIE_CMP_RETURN(group_cookie); COOKIE_CMP_RETURN(color);So if
task_cookiematches, we considergroup_cookie, if that matches we consider color.
Yes, the cookie is a compound one. A cookie structure is created here which has all 3 components. The final cookie value is a pointer to this compound structure.
Quoted 4 lines
Now, afaict that's semantically exactly the same as just using the narrowest cookie. That is, use the task cookie if there is, and then, walking the cgroup hierarchy (up) pick the first cgroup cookie.
[..]
Quoted 1 line
Which means you only need a single active cookie field.
Its not the same. The "compounded" cookie is needed to enforce the CGroup delegation model. This is exactly how both uclamp and cpusets work. For uclamp, if we take uclamp.min as an example, if the CGroup sets the uclamp.min of a group to 0, then even if you use the per-task interface (sched_setattr) for setting the task's clamp - the "effective uclamp.min" of the task as seen in /proc/pid/sched will still be 0.
Similar thing here, if 2 tasks belong to 2 different CGroups and each group is tagged with its own tag, then if you use the per-task interface to make the 2 tasks share a core, the "effective" sharing is still such that the tasks will not share a core -- because the CGroup decided to make it so.
I would like to really maintain this model. Doing it any other way is confusing - we have already discussed doing it this way before. With that way you end up failing one interface if another one was already used. Been there, done that. It sucks a lot.
Quoted 1 line
IOW, you're just making things complicated and expensive.
The cost of the additional comparisons you mentioned is only in the slow path (i.e. when someone joins or leaves a group). Once the task_struct's cookie field is set, the cost is not any more than what it was before.
Any other thoughts?
thanks,
- Joel
Peter Zijlstra
maintainerRe: [PATCH -tip 22/32] sched: Split the cookie and setup per-task cookie on fork
In reply to Joel Fernandes (Google)Quoted 1 line · Joel Fernandes (Google)
+ *
sched_core_tag_requeue- Common helper for all interfaces to set a cookie.
sched_core_set_cookie() would be a saner name, given that description, don't you think?
Re: [PATCH -tip 22/32] sched: Split the cookie and setup per-task cookie on fork
Quoted 6 lines · Peter Zijlstra
Quoted 1 line · Joel Fernandes (Google)
+ *
sched_core_tag_requeue- Common helper for all interfaces to set a cookie.
sched_core_set_cookie()would be a saner name, given that description, don't you think?
Yeah, Josh is better than me at naming so he changed it to sched_core_update_cookie() already :-). Hopefully that's Ok too with you.
thanks,
- Joel
Peter Zijlstra
maintainerRe: [PATCH -tip 22/32] sched: Split the cookie and setup per-task cookie on fork
In reply to Joel Fernandes (Google)Quoted 12 lines · Joel Fernandes (Google)
kernel/sched/core.c +12
-0+/* +* Ensure that the task has been requeued. The stopper ensures that the task cannot +* be migrated to a different CPU while its core scheduler queue state is being updated. +* It also makes sure to requeue a task if it was running actively on another CPU. +*/ +static int sched_core_task_join_stopper (void *data) +{ +struct sched_core_task_write_tag *tag = (struct sched_core_task_write_tag *)data; +int i; + +for (i =0 ; i <2 ; i++) +sched_core_tag_requeue (tag->tasks[i], tag->cookies[i], false/* !group */ );
kernel/sched/core.c static u64 cpu_rt_period_read_uint(struct cgroup_subsys_state *css, · line 9871
Quoted 10 lines
kernel/sched/core.c +8
-0+ +return 0 ; +} + +static int sched_core_share_tasks (struct task_struct *t1,struct task_struct *t2) +{ +stop_machine (sched_core_task_join_stopper, (void *)&wr,NULL ); +}
This is *REALLY* terrible...
Re: [PATCH -tip 22/32] sched: Split the cookie and setup per-task cookie on fork
Quoted 21 lines · Peter Zijlstra
Quoted 18 lines · Joel Fernandes (Google)
kernel/sched/core.c +18
-0+/* +* Ensure that the task has been requeued. The stopper ensures that the task cannot +* be migrated to a different CPU while its core scheduler queue state is being updated. +* It also makes sure to requeue a task if it was running actively on another CPU. +*/ +static int sched_core_task_join_stopper (void *data) +{ +struct sched_core_task_write_tag *tag = (struct sched_core_task_write_tag *)data; +int i; + +for (i =0 ; i <2 ; i++) +sched_core_tag_requeue (tag->tasks[i], tag->cookies[i], false/* !group */ ); + +return 0 ; +} + +static int sched_core_share_tasks (struct task_struct *t1,struct task_struct *t2) +{
Quoted 5 lines
kernel/sched/core.c static u64 cpu_rt_period_read_uint(struct cgroup_subsys_state *css, · line 9959
Quoted 3 lines
kernel/sched/core.c +2
-0+stop_machine (sched_core_task_join_stopper, (void *)&wr,NULL ); +}This is *REALLY* terrible...
I pulled this bit from your original patch. Are you concerned about the stop_machine? Sharing a core is a slow path for our usecases (and as far as I know, for everyone else's). We can probably do something different if that requirement changes.
Peter Zijlstra
maintainerRe: [PATCH -tip 22/32] sched: Split the cookie and setup per-task cookie on fork
Quoted 24 lines · Joel Fernandes
Quoted 23 lines · Peter Zijlstra
Quoted 20 lines · Joel Fernandes (Google)
kernel/sched/core.c +19
-0+/* +* Ensure that the task has been requeued. The stopper ensures that the task cannot +* be migrated to a different CPU while its core scheduler queue state is being updated. +* It also makes sure to requeue a task if it was running actively on another CPU. +*/ +static int sched_core_task_join_stopper (void *data) +{ +struct sched_core_task_write_tag *tag = (struct sched_core_task_write_tag *)data; +int i; + +for (i =0 ; i <2 ; i++) +sched_core_tag_requeue (tag->tasks[i], tag->cookies[i], false/* !group */ ); + +return 0 ; +} + +static int sched_core_share_tasks (struct task_struct *t1,struct task_struct *t2) +{ +stop_machine (sched_core_task_join_stopper, (void *)&wr,NULL );
Quoted 9 lines
Quoted 3 lines
Quoted 1 line
+}
This is *REALLY* terrible...
I pulled this bit from your original patch. Are you concerned about the
stop_machine? Sharing a core is a slow path for our usecases (and as far as I know, for everyone else's). We can probably do something different if that requirement changes.
Yeah.. so I can (and was planning on) remove stop_machine() from sched_core_{dis,en}able() before merging it.
(there's two options, one uses stop_cpus() with the SMT mask, the other RCU)
This though is exposing stop_machine() to joe user. Everybody is allowed to prctl() it's own task and set a cookie on himself. This means you just made giant unpriv DoS vector.
stop_machine is bad, really bad.
Re: [PATCH -tip 22/32] sched: Split the cookie and setup per-task cookie on fork
Quoted 35 lines · Peter Zijlstra
Quoted 24 lines · Joel Fernandes
Quoted 23 lines · Peter Zijlstra
Quoted 20 lines · Joel Fernandes (Google)
kernel/sched/core.c +19
-0+/* +* Ensure that the task has been requeued. The stopper ensures that the task cannot +* be migrated to a different CPU while its core scheduler queue state is being updated. +* It also makes sure to requeue a task if it was running actively on another CPU. +*/ +static int sched_core_task_join_stopper (void *data) +{ +struct sched_core_task_write_tag *tag = (struct sched_core_task_write_tag *)data; +int i; + +for (i =0 ; i <2 ; i++) +sched_core_tag_requeue (tag->tasks[i], tag->cookies[i], false/* !group */ ); + +return 0 ; +} + +static int sched_core_share_tasks (struct task_struct *t1,struct task_struct *t2) +{ +stop_machine (sched_core_task_join_stopper, (void *)&wr,NULL );Quoted 9 lines
Quoted 3 lines
Quoted 1 line
+}
This is *REALLY* terrible...
I pulled this bit from your original patch. Are you concerned about the
stop_machine? Sharing a core is a slow path for our usecases (and as far as I know, for everyone else's). We can probably do something different if that requirement changes.
Quoted 5 lines
Yeah.. so I can (and was planning on) remove
stop_machine()from sched_core_{dis,en}able() before merging it.(there's two options, one uses
stop_cpus()with the SMT mask, the other RCU)
Ok. What about changing the cookie of task T while holding the rq/pi locks, and then doing a resched_curr(rq) for that RQ?
The holding lock ensures no migration of task happens, and resched_curr() ensure that task T's rq will enter the scheduler to consider the task T's new cookie for scheduling. I believe this is analogous to what __sched_setscheduler() does when you switch a task from CFS to RT.
Quoted 5 lines
This though is exposing
stop_machine()to joe user. Everybody is allowed toprctl()it's own task and set a cookie on himself. This means you just made giant unpriv DoS vector.
stop_machineis bad, really bad.
Agreed.
thanks,
- Joel
Peter Zijlstra
maintainerRe: [PATCH -tip 22/32] sched: Split the cookie and setup per-task cookie on fork
In reply to Joel Fernandes (Google)Quoted 34 lines · Joel Fernandes (Google)
kernel/sched/core.c +34
-0+/* Per-task interface */ +static unsigned long sched_core_alloc_task_cookie (void ) +{ +struct sched_core_cookie *ptr = +kmalloc (sizeof (struct sched_core_cookie ),GFP_KERNEL ); + +if (!ptr) +return 0 ; +refcount_set (&ptr->refcnt,1 ); + +/* +* NOTE: sched_core_put() is not done by put_task_cookie(). Instead, it +* is done after the stopper runs. +*/ +sched_core_get (); +return (unsigned long )ptr; +} + +static bool sched_core_get_task_cookie (unsigned long cookie) +{ +struct sched_core_cookie *ptr = (struct sched_core_cookie *)cookie; + +/* +* NOTE: sched_core_put() is not done by put_task_cookie(). Instead, it +* is done after the stopper runs. +*/ +sched_core_get (); +return refcount_inc_not_zero (&ptr->refcnt); +} + +static void sched_core_put_task_cookie (unsigned long cookie) +{ +struct sched_core_cookie *ptr = (struct sched_core_cookie *)cookie; +
kernel/sched/core.c static u64 cpu_rt_period_read_uint(struct cgroup_subsys_state *css, · line 9865
Quoted 10 lines
kernel/sched/core.c +9
-0+if (refcount_dec_and_test (&ptr->refcnt)) +kfree (ptr); +} +/* +* NOTE: sched_core_get() is done by sched_core_alloc_task_cookie() or +* sched_core_put_task_cookie(). However, sched_core_put() is done +* by this function *after* the stopper removes the tasks from the +* core queue, and not before. This is just to play it safe. +*/
So for no reason what so ever you've made the code more difficult?
Re: [PATCH -tip 22/32] sched: Split the cookie and setup per-task cookie on fork
Quoted 39 lines · Peter Zijlstra
Quoted 37 lines · Joel Fernandes (Google)
(quoted diff) +37
-0+/* Per-task interface */ +static unsigned long sched_core_alloc_task_cookie(void) +{ + struct sched_core_cookie *ptr = + kmalloc(sizeof(struct sched_core_cookie), GFP_KERNEL); + + if (!ptr) + return 0; + refcount_set(&ptr->refcnt, 1); + + /* + * NOTE: sched_core_put() is not done by put_task_cookie(). Instead, it + * is done after the stopper runs. + */ + sched_core_get(); + return (unsigned long)ptr; +} + +static bool sched_core_get_task_cookie(unsigned long cookie) +{ + struct sched_core_cookie *ptr = (struct sched_core_cookie *)cookie; + + /* + * NOTE: sched_core_put() is not done by put_task_cookie(). Instead, it + * is done after the stopper runs. + */ + sched_core_get(); + return refcount_inc_not_zero(&ptr->refcnt); +} + +static void sched_core_put_task_cookie(unsigned long cookie) +{ + struct sched_core_cookie *ptr = (struct sched_core_cookie *)cookie; + + if (refcount_dec_and_test(&ptr->refcnt)) + kfree(ptr); +}
Quoted 8 lines
Review
Quoted 6 lines
(quoted diff) +6
-0+ /* + * NOTE: sched_core_get() is done by sched_core_alloc_task_cookie() or + * sched_core_put_task_cookie(). However, sched_core_put() is done + * by this function *after* the stopper removes the tasks from the + * core queue, and not before. This is just to play it safe. + */So for no reason what so ever you've made the code more difficult?
You're right, I could just do sched_core_get() in the caller. I changed it as in the diff below:
---8<-----------------------
kernel/sched/coretag.c +7 -13
@@ -274,6 +274,7 @@ void sched_core_change_group(struct task_struct *p, struct task_group *new_tg)/* Per-task interface: Used by fork(2) and prctl(2). */ static void sched_core_put_cookie_work (struct work_struct *ws); +/* Caller has to call sched_core_get() if non-zero value is returned. */ static unsigned long sched_core_alloc_task_cookie (void ) { struct sched_core_task_cookie *ck =
@@ -284,11 +285,6 @@ static unsigned long sched_core_alloc_task_cookie(void)refcount_set (&ck->refcnt,1 );INIT_WORK (&ck->work, sched_core_put_cookie_work); -/* -* NOTE: sched_core_put() is not done by put_task_cookie(). Instead, it -* is done after the stopper runs. -*/ -sched_core_get ();return (unsigned long )ck; }
@@ -354,12 +350,6 @@ int sched_core_share_tasks(struct task_struct *t1, struct task_struct *t2)mutex_lock (&sched_core_tasks_mutex); -/* -* NOTE: sched_core_get() is done by sched_core_alloc_task_cookie() or -* sched_core_put_task_cookie(). However, sched_core_put() is done -* by this function *after* the stopper removes the tasks from the -* core queue, and not before. This is just to play it safe. -*/ if (!t2) {if (t1->core_task_cookie) {sched_core_put_task_cookie (t1->core_task_cookie);
@@ -370,7 +360,9 @@ int sched_core_share_tasks(struct task_struct *t1, struct task_struct *t2)/* Assign a unique per-task cookie solely for t1. */ cookie =sched_core_alloc_task_cookie (); -if (!cookie) +if (cookie) +sched_core_get (); +else goto out_unlock;if (t1->core_task_cookie) {
@@ -401,7 +393,9 @@ int sched_core_share_tasks(struct task_struct *t1, struct task_struct *t2)/* CASE 1. */ cookie =sched_core_alloc_task_cookie (); -if (!cookie) +if (cookie) +sched_core_get (); +else goto out_unlock;/* Add another reference for the other task. */
Peter Zijlstra
maintainerRe: [PATCH -tip 22/32] sched: Split the cookie and setup per-task cookie on fork
In reply to Joel Fernandes (Google)kernel/sched/core.c static u64 cpu_rt_period_read_uint(struct cgroup_subsys_state *css, · line 9819
Quoted 10 lines · Joel Fernandes (Google)
kernel/sched/core.c +10
-0+static bool sched_core_get_task_cookie (unsigned long cookie) +{ +struct sched_core_cookie *ptr = (struct sched_core_cookie *)cookie; + +/* +* NOTE: sched_core_put() is not done by put_task_cookie(). Instead, it +* is done after the stopper runs. +*/ +sched_core_get (); +return refcount_inc_not_zero (&ptr->refcnt);
See below, but afaict this should be refcount_inc().
Quoted 18 lines
kernel/sched/core.c +17
-0+} +/* +* t1 joining t2 +* CASE 1: +* before 0 0 +* after new cookie new cookie +* +* CASE 2: +* before X (non-zero) 0 +* after 0 0 +* +* CASE 3: +* before 0 X (non-zero) +* after X X +* +* CASE 4: +* before Y (non-zero) X (non-zero)
kernel/sched/core.c static u64 cpu_rt_period_read_uint(struct cgroup_subsys_state *css, · line 9911
Quoted 10 lines
kernel/sched/core.c +10
-0+* after X X +*/ +if (!t1->core_task_cookie && !t2->core_task_cookie) { +/* CASE 1. */ + cookie =sched_core_alloc_task_cookie (); +if (!cookie) + goto out_unlock; + +/* Add another reference for the other task. */ +if (!sched_core_get_task_cookie (cookie)) {
afaict this should be refcount_inc(), as this can never fail and if it does, it's an error.
Quoted 8 lines
kernel/sched/core.c +8
-0+ ret = -EINVAL ; + goto out_unlock; + } + + wr.tasks[0 ] = t1; + wr.tasks[1 ] = t2; + wr.cookies[0 ] = wr.cookies[1 ] = cookie; +
kernel/sched/core.c static u64 cpu_rt_period_read_uint(struct cgroup_subsys_state *css, · line 9929
Quoted 10 lines
kernel/sched/core.c +10
-0+ }else if (t1->core_task_cookie && !t2->core_task_cookie) { +/* CASE 2. */ +sched_core_put_task_cookie (t1->core_task_cookie); + sched_core_put_after_stopper = true; + + wr.tasks[0 ] = t1;/* Reset cookie for t1. */ + + }else if (!t1->core_task_cookie && t2->core_task_cookie) { +/* CASE 3. */ +if (!sched_core_get_task_cookie (t2->core_task_cookie)) {
afaict this can never fail either, because you're calling in here with a reference on t2
kernel/sched/core.c static u64 cpu_rt_period_read_uint(struct cgroup_subsys_state *css, · line 9939
Quoted 10 lines
kernel/sched/core.c +10
-0+ ret = -EINVAL ; + goto out_unlock; + } + + wr.tasks[0 ] = t1; + wr.cookies[0 ] = t2->core_task_cookie; + + }else { +/* CASE 4. */ +if (!sched_core_get_task_cookie (t2->core_task_cookie)) {
Same.
Quoted 9 lines
kernel/sched/core.c +9
-0+ ret = -EINVAL ; + goto out_unlock; + } +sched_core_put_task_cookie (t1->core_task_cookie); + sched_core_put_after_stopper = true; + + wr.tasks[0 ] = t1; + wr.cookies[0 ] = t2->core_task_cookie; + }
Re: [PATCH -tip 22/32] sched: Split the cookie and setup per-task cookie on fork
Quoted 13 lines · Peter Zijlstra
Review
Quoted 10 lines · Joel Fernandes (Google)
(quoted diff) +10
-0+static bool sched_core_get_task_cookie(unsigned long cookie) +{ + struct sched_core_cookie *ptr = (struct sched_core_cookie *)cookie; + + /* + * NOTE: sched_core_put() is not done by put_task_cookie(). Instead, it + * is done after the stopper runs. + */ + sched_core_get(); + return refcount_inc_not_zero(&ptr->refcnt);See below, but afaict this should be
refcount_inc().
Fully agreed with all these. Updated with diff as below. Will test further and post next version soon. Thanks!
---8<-----------------------
kernel/sched/coretag.c +5 -14
@@ -288,12 +288,12 @@ static unsigned long sched_core_alloc_task_cookie(void)return (unsigned long )ck; } -static bool sched_core_get_task_cookie (unsigned long cookie) +static void sched_core_get_task_cookie (unsigned long cookie) {struct sched_core_task_cookie *ptr = (struct sched_core_task_cookie *)cookie; -return refcount_inc_not_zero (&ptr->refcnt); +refcount_inc (&ptr->refcnt); }static void sched_core_put_task_cookie (unsigned long cookie)
@@ -392,10 +392,7 @@ int sched_core_share_tasks(struct task_struct *t1, struct task_struct *t2)sched_core_get ();/* For the alloc. */ /* Add another reference for the other task. */ -if (!sched_core_get_task_cookie (cookie)) { - ret = -EINVAL ; - goto out_unlock; - } +sched_core_get_task_cookie (cookie);sched_core_get ();/* For the other task. */ wr.tasks[0 ] = t1;
@@ -411,10 +408,7 @@ int sched_core_share_tasks(struct task_struct *t1, struct task_struct *t2) }else if (!t1->core_task_cookie && t2->core_task_cookie) {/* CASE 3. */ -if (!sched_core_get_task_cookie (t2->core_task_cookie)) { - ret = -EINVAL ; - goto out_unlock; - } +sched_core_get_task_cookie (t2->core_task_cookie);sched_core_get (); wr.tasks[0 ] = t1;
@@ -422,10 +416,7 @@ int sched_core_share_tasks(struct task_struct *t1, struct task_struct *t2) }else {/* CASE 4. */ -if (!sched_core_get_task_cookie (t2->core_task_cookie)) { - ret = -EINVAL ; - goto out_unlock; - } +sched_core_get_task_cookie (t2->core_task_cookie);sched_core_get ();sched_core_put_task_cookie (t1->core_task_cookie);
Signature · 1 line
2.29.2.454.gaff20da3a2-goog
Re: [PATCH -tip 22/32] sched: Split the cookie and setup per-task cookie on fork
In reply to Joel Fernandes (Google)Quoted 83 lines · Joel Fernandes (Google)
In order to prevent interference and clearly support both per-task and CGroup APIs, split the cookie into 2 and allow it to be set from either per-task, or CGroup API. The final cookie is the combined value of both and is computed when the stop-machine executes during a change of cookie.
Also, for the per-task cookie, it will get weird if we use pointers of any emphemeral objects. For this reason, introduce a refcounted object who's sole purpose is to assign unique cookie value by way of the object's pointer.
While at it, refactor the CGroup code a bit. Future patches will introduce more APIs and support.
Reviewed-by Josh Don <[email protected]>Tested-by Julien Desfossez <[email protected]>Signed-off-by Joel Fernandes (Google) <[email protected]>--- include/linux/sched.h | 2 + kernel/sched/core.c | 241 ++++++++++++++++++++++++++++++++++++++++-- kernel/sched/debug.c | 4 + 3 files changed, 236 insertions(+), 11 deletions(-)include/linux/sched.h +2
-0@@ -688,6 +688,8 @@ struct task_struct {#ifdef CONFIG_SCHED_CORE struct rb_node core_node;unsigned long core_cookie; +unsigned long core_task_cookie; +unsigned long core_group_cookie;unsigned int core_occupation;#endif kernel/sched/core.c +22
-0@@ -346,11 +346,14 @@ void sched_core_put(void) mutex_unlock(&sched_core_mutex); } +static int sched_core_share_tasks (struct task_struct *t1,struct task_struct *t2); +#else /* !CONFIG_SCHED_CORE */ static inline void sched_core_enqueue (struct rq *rq,struct task_struct *p) { }static inline void sched_core_dequeue (struct rq *rq,struct task_struct *p) { }static bool sched_core_enqueued (struct task_struct *task) {return false; } +static int sched_core_share_tasks (struct task_struct *t1,struct task_struct *t2) { }#endif /* CONFIG_SCHED_CORE */ @@ -4032,6 +4035,20 @@ int sched_fork(unsigned long clone_flags, struct task_struct *p)#endif #ifdef CONFIG_SCHED_CORE RB_CLEAR_NODE (&p->core_node); + +/* +* Tag child via per-task cookie only if parent is tagged via per-task +* cookie. This is independent of, but can be additive to the CGroup tagging. +*/ +if (current->core_task_cookie) { + +/* If it is not CLONE_THREAD fork, assign a unique per-task tag. */ +if (!(clone_flags &CLONE_THREAD )) { +return sched_core_share_tasks (p, p); + } +/* Otherwise share the parent's per-task tag. */ +return sched_core_share_tasks (p, current); + }#endif return 0 ; }@@ -9731,6 +9748,217 @@ static u64 cpu_rt_period_read_uint(struct cgroup_subsys_state *css,#endif /* CONFIG_RT_GROUP_SCHED */ #ifdef CONFIG_SCHED_CORE +/* +* A simple wrapper around refcount. An allocated sched_core_cookie's +* address is used to compute the cookie of the task. +*/ +struct sched_core_cookie {
kernel/sched/core.c static u64 cpu_rt_period_read_uint(struct cgroup_subsys_state *css, · line 9765
Quoted 10 lines
kernel/sched/core.c +10
-0+refcount_t refcnt; +}; + +/* +* sched_core_tag_requeue - Common helper for all interfaces to set a cookie. +* @p: The task to assign a cookie to. +* @cookie: The cookie to assign. +* @group: is it a group interface or a per-task interface. +* +* This function is typically called from a stop-machine handler.
Can you clarify if it is typically or always, what are the implications for locking?
Quoted 5 lines
kernel/sched/core.c +5
-0+*/ +void sched_core_tag_requeue (struct task_struct *p,unsigned long cookie,bool group) +{ +if (!p) +return ;
kernel/sched/core.c static u64 cpu_rt_period_read_uint(struct cgroup_subsys_state *css, · line 9780
Quoted 10 lines
kernel/sched/core.c +10
-0+ +if (group) + p->core_group_cookie = cookie; +else + p->core_task_cookie = cookie; + +/* Use up half of the cookie's bits for task cookie and remaining for group cookie. */ + p->core_cookie = (p->core_task_cookie << + (sizeof (unsigned long ) *4 )) + p->core_group_cookie; +
Always use masks to ensure this fits in the space we have, should we be concerned about overflows and the potential for collision of cookie values?
Quoted 43 lines
kernel/sched/core.c +43
-0+if (sched_core_enqueued (p)) { +sched_core_dequeue (task_rq (p), p); +if (!p->core_task_cookie) +return ; + } + +if (sched_core_enabled (task_rq (p)) && + p->core_cookie &&task_on_rq_queued (p)) +sched_core_enqueue (task_rq (p), p); +} + +/* Per-task interface */ +static unsigned long sched_core_alloc_task_cookie (void ) +{ +struct sched_core_cookie *ptr = +kmalloc (sizeof (struct sched_core_cookie ),GFP_KERNEL ); + +if (!ptr) +return 0 ; +refcount_set (&ptr->refcnt,1 ); + +/* +* NOTE: sched_core_put() is not done by put_task_cookie(). Instead, it +* is done after the stopper runs. +*/ +sched_core_get (); +return (unsigned long )ptr; +} + +static bool sched_core_get_task_cookie (unsigned long cookie) +{ +struct sched_core_cookie *ptr = (struct sched_core_cookie *)cookie; + +/* +* NOTE: sched_core_put() is not done by put_task_cookie(). Instead, it +* is done after the stopper runs. +*/ +sched_core_get (); +return refcount_inc_not_zero (&ptr->refcnt); +} + +static void sched_core_put_task_cookie (unsigned long cookie) +{
kernel/sched/core.c static u64 cpu_rt_period_read_uint(struct cgroup_subsys_state *css, · line 9833
Quoted 10 lines
kernel/sched/core.c +10
-0+struct sched_core_cookie *ptr = (struct sched_core_cookie *)cookie; + +if (refcount_dec_and_test (&ptr->refcnt)) +kfree (ptr); +} + +struct sched_core_task_write_tag { +struct task_struct *tasks[2 ]; +unsigned long cookies[2 ]; +};
Use a better name instead of 2?
kernel/sched/core.c static u64 cpu_rt_period_read_uint(struct cgroup_subsys_state *css, · line 9845
Quoted 12 lines
kernel/sched/core.c +12
-0+ +/* +* Ensure that the task has been requeued. The stopper ensures that the task cannot +* be migrated to a different CPU while its core scheduler queue state is being updated. +* It also makes sure to requeue a task if it was running actively on another CPU. +*/ +static int sched_core_task_join_stopper (void *data) +{ +struct sched_core_task_write_tag *tag = (struct sched_core_task_write_tag *)data; +int i; + +for (i =0 ; i <2 ; i++)
Use ARRAY_SIZE(cookies) if you have to or ARRAY_SIZE(tasks)
kernel/sched/core.c static u64 cpu_rt_period_read_uint(struct cgroup_subsys_state *css, · line 9852
Quoted 7 lines
kernel/sched/core.c +7
-0+sched_core_tag_requeue (tag->tasks[i], tag->cookies[i], false/* !group */ ); + +return 0 ; +} + +static int sched_core_share_tasks (struct task_struct *t1,struct task_struct *t2) +{
Can you please explain how t1 and t2 are related, there is a table below, but I don't understand case#2, where the cookies get reset, is t2 the core leader and t2 leads what t1 and t2 collectively get?
May be just called t2 as parent?
Quoted 138 lines
kernel/sched/core.c +111
-11+struct sched_core_task_write_tag wr = {};/* for stop machine. */ +bool sched_core_put_after_stopper = false; +unsigned long cookie; +int ret = -ENOMEM ; + +mutex_lock (&sched_core_mutex); + +/* +* NOTE: sched_core_get() is done by sched_core_alloc_task_cookie() or +* sched_core_put_task_cookie(). However, sched_core_put() is done +* by this function *after* the stopper removes the tasks from the +* core queue, and not before. This is just to play it safe. +*/ +if (t2 ==NULL ) { +if (t1->core_task_cookie) { +sched_core_put_task_cookie (t1->core_task_cookie); + sched_core_put_after_stopper = true; + wr.tasks[0 ] = t1;/* Keep wr.cookies[0] reset for t1. */ + } + }else if (t1 == t2) { +/* Assign a unique per-task cookie solely for t1. */ + + cookie =sched_core_alloc_task_cookie (); +if (!cookie) + goto out_unlock; + +if (t1->core_task_cookie) { +sched_core_put_task_cookie (t1->core_task_cookie); + sched_core_put_after_stopper = true; + } + wr.tasks[0 ] = t1; + wr.cookies[0 ] = cookie; + }else +/* +* t1 joining t2 +* CASE 1: +* before 0 0 +* after new cookie new cookie +* +* CASE 2: +* before X (non-zero) 0 +* after 0 0 +* +* CASE 3: +* before 0 X (non-zero) +* after X X +* +* CASE 4: +* before Y (non-zero) X (non-zero) +* after X X +*/ +if (!t1->core_task_cookie && !t2->core_task_cookie) { +/* CASE 1. */ + cookie =sched_core_alloc_task_cookie (); +if (!cookie) + goto out_unlock; + +/* Add another reference for the other task. */ +if (!sched_core_get_task_cookie (cookie)) { + ret = -EINVAL ; + goto out_unlock; + } + + wr.tasks[0 ] = t1; + wr.tasks[1 ] = t2; + wr.cookies[0 ] = wr.cookies[1 ] = cookie; + + }else if (t1->core_task_cookie && !t2->core_task_cookie) { +/* CASE 2. */ +sched_core_put_task_cookie (t1->core_task_cookie); + sched_core_put_after_stopper = true; + + wr.tasks[0 ] = t1;/* Reset cookie for t1. */ + + }else if (!t1->core_task_cookie && t2->core_task_cookie) { +/* CASE 3. */ +if (!sched_core_get_task_cookie (t2->core_task_cookie)) { + ret = -EINVAL ; + goto out_unlock; + } + + wr.tasks[0 ] = t1; + wr.cookies[0 ] = t2->core_task_cookie; + + }else { +/* CASE 4. */ +if (!sched_core_get_task_cookie (t2->core_task_cookie)) { + ret = -EINVAL ; + goto out_unlock; + } +sched_core_put_task_cookie (t1->core_task_cookie); + sched_core_put_after_stopper = true; + + wr.tasks[0 ] = t1; + wr.cookies[0 ] = t2->core_task_cookie; + } + +stop_machine (sched_core_task_join_stopper, (void *)&wr,NULL ); + +if (sched_core_put_after_stopper) +sched_core_put (); + + ret =0 ; +out_unlock: +mutex_unlock (&sched_core_mutex); +return ret; +} + +/* CGroup interface */ static u64 cpu_core_tag_read_u64 (struct cgroup_subsys_state *css,struct cftype *cft) {struct task_group *tg =css_tg (css);@@ -9761,18 +9989,9 @@ static int __sched_write_tag(void *data)* when we set cgroup tag to 0 when the loop is done below. */ while ((p =css_task_iter_next (&it))) { - p->core_cookie = !!val ? (unsigned long )tg :0UL ; - -if (sched_core_enqueued (p)) { -sched_core_dequeue (task_rq (p), p); -if (!p->core_cookie) -continue ; - } - -if (sched_core_enabled (task_rq (p)) && - p->core_cookie &&task_on_rq_queued (p)) -sched_core_enqueue (task_rq (p), p); +unsigned long cookie = !!val ? (unsigned long )tg :0UL ; +sched_core_tag_requeue (p, cookie, true/* group */ ); }css_task_iter_end (&it);kernel/sched/debug.c +0
-0
kernel/sched/debug.c void proc_sched_show_task(struct task_struct *p, struct pid_namespace *ns, · line 1033
Quoted 10 lines
kernel/sched/debug.c +4
-0@@ -1024,6 +1024,10 @@ void proc_sched_show_task(struct task_struct *p, struct pid_namespace *ns,__PS ("clock-delta" , t1-t0); } +#ifdef CONFIG_SCHED_CORE +__PS ("core_cookie" , p->core_cookie); +#endif +sched_show_numa (p, m); }
Balbir Singh.
PATCH 23/32 sched: Add a per-thread core scheduling interface
In reply to Joel Fernandes (Google)Add a per-thread core scheduling interface which allows a thread to share a core with another thread, or have a core exclusively for itself.
ChromeOS uses core-scheduling to securely enable hyperthreading. This cuts down the keypress latency in Google docs from 150ms to 50ms while improving the camera streaming frame rate by ~3%.
--- include/linux/sched.h | 1 + include/uapi/linux/prctl.h | 3 ++ kernel/sched/core.c | 51 +++++++++++++++++++++++++++++--- kernel/sys.c | 3 ++ tools/include/uapi/linux/prctl.h | 3 ++ 5 files changed, 57 insertions(+), 4 deletions(-)
include/linux/sched.h +1 -0
@@ -2083,6 +2083,7 @@ void sched_core_unsafe_enter(void);void sched_core_unsafe_exit (void );bool sched_core_wait_till_safe (unsigned long ti_check);bool sched_core_kernel_protected (void ); +int sched_core_share_pid (pid_t pid);#else #define sched_core_unsafe_enter (ignore) do { } while (0)#define sched_core_unsafe_exit (ignore) do { } while (0)
include/uapi/linux/prctl.h +3 -0
@@ -248,4 +248,7 @@ struct prctl_mm_map {#define PR_SET_IO_FLUSHER 57#define PR_GET_IO_FLUSHER 58 +/* Request the scheduler to share a core */ +#define PR_SCHED_CORE_SHARE 59 +#endif /* _LINUX_PRCTL_H */
kernel/sched/core.c +47 -4
78 lines not on this page · raw message
kernel/sys.c +3 -0
@@ -2530,6 +2530,9 @@ SYSCALL_DEFINE5(prctl, int, option, unsigned long, arg2, unsigned long, arg3, error = (current->flags &PR_IO_FLUSHER ) ==PR_IO_FLUSHER ;break ; +case PR_SCHED_CORE_SHARE : + error =sched_core_share_pid (arg2); +break ;default : error = -EINVAL ;break ;
tools/include/uapi/linux/prctl.h +3 -0
@@ -247,4 +247,7 @@ struct prctl_mm_map {#define PR_SET_IO_FLUSHER 57#define PR_GET_IO_FLUSHER 58 +/* Request the scheduler to share a core */ +#define PR_SCHED_CORE_SHARE 59 +#endif /* _LINUX_PRCTL_H */
Signature · 1 line
2.29.2.299.gdc1121823c-goog
Peter Zijlstra
maintainerRe: [PATCH -tip 23/32] sched: Add a per-thread core scheduling interface
kernel/sched/core.c static int sched_core_share_tasks(struct task_struct *t1, struct task_struct *t2 · line 9973
Quoted 11 lines · Joel Fernandes (Google)
kernel/sched/core.c +11
-0+/* Called from prctl interface: PR_SCHED_CORE_SHARE */ +int sched_core_share_pid (pid_t pid) +{ +struct task_struct *task; +int err; + +if (pid ==0 ) {/* Recent current task's cookie. */ +/* Resetting a cookie requires privileges. */ +if (current->core_task_cookie) +if (!capable (CAP_SYS_ADMIN )) +return -EPERM ;
Coding-Style fail.
Also, why?!? I realize it is true for your case, because hardware fail. But in general this just isn't true. This wants to be some configurable policy.
Re: [PATCH -tip 23/32] sched: Add a per-thread core scheduling interface
Quoted 15 lines · Peter Zijlstra
Review
Quoted 11 lines · Joel Fernandes (Google)
(quoted diff) +11
-0+/* Called from prctl interface: PR_SCHED_CORE_SHARE */ +int sched_core_share_pid(pid_t pid) +{ + struct task_struct *task; + int err; + + if (pid == 0) { /* Recent current task's cookie. */ + /* Resetting a cookie requires privileges. */ + if (current->core_task_cookie) + if (!capable(CAP_SYS_ADMIN)) + return -EPERM;Coding-Style fail.
Quoted 3 lines
Also, why?!? I realize it is true for your case, because hardware fail. But in general this just isn't true. This wants to be some configurable policy.
True. I think me and you discussed eons ago though that it needs to privileged. For our case, actually we use seccomp so we don't let untrusted task set a cookie anyway, let alone reset it. We do it before we enter the seccomp sandbox. So we don't really need this security check here.
Since you dislike this part of the patch, I am Ok with just dropping it as below:
---8<-----------------------
kernel/sched/coretag.c +1 -5
@@ -443,11 +443,7 @@ int sched_core_share_pid(pid_t pid)struct task_struct *task;int err; -if (pid ==0 ) {/* Recent current task's cookie. */ -/* Resetting a cookie requires privileges. */ -if (current->core_task_cookie) -if (!capable (CAP_SYS_ADMIN )) -return -EPERM ; +if (!pid) {/* Reset current task's cookie. */ task =NULL ; }else {rcu_read_lock ();
Re: [PATCH -tip 23/32] sched: Add a per-thread core scheduling interface
In reply to Joel Fernandes (Google)Quoted 7 lines · Joel Fernandes (Google)
Add a per-thread core scheduling interface which allows a thread to share a core with another thread, or have a core exclusively for itself.
ChromeOS uses core-scheduling to securely enable hyperthreading. This cuts down the keypress latency in Google docs from 150ms to 50ms while improving the camera streaming frame rate by ~3%.
Inline is a patch for comment to extend this interface to make it more useful. This patch would still need to provide doc and selftests updates as well.
-chrish
---8<-----------------------
From ec3d6506fee89022d93789e1ba44d49c1b1b04dd Mon Sep 17 00:00:00 2001
From: chris hyser <[email protected]>
Date: Tue, 10 Nov 2020 15:35:59 -0500
Subject: [PATCH] sched: Provide a more extensive prctl interface for core
scheduling.
The current prctl interface is a "from" only interface allowing a task to join an existing core scheduling group by getting the "cookie" from a specified task/pid.
Additionally, sharing from a task without an existing core scheduling group (cookie == 0) creates a new cookie shared between the two.
"From" functionality works well for programs modified to use the prctl(), but many applications will not be modified simply for core scheduling. Using a wrapper program to share the core scheduling cookie from another task followed by an "exec" can work, but there is no means to assign a cookie for an unmodified running task.
Simply inverting the interface to a "to"-only interface, i.e. the calling task shares it's cookie with the specified task/pid also has limitations, there being no means to enable tasks to join an existing core scheduling group for instance.
The solution is to allow both, or more specifically provide a flags argument to allow various core scheduling commands, currently FROM, TO, and CLEAR.
The combination of FROM and TO allow a helper program to share the core scheduling cookie of one task/core scheduling group, with additional tasks.
if (prctl(PR_SCHED_CORE_SHARE, PR_SCHED_CORE_SHARE_FROM, src_pid, 0, 0) < 0)
handle_error("src_pid sched_core failed");
if (prctl(PR_SCHED_CORE_SHARE, PR_SCHED_CORE_SHARE_TO, dest_pid, 0, 0) < 0)
handle_error("dest_pid sched_core failed");include/linux/sched.h +2 -2
@@ -2084,14 +2084,14 @@ void sched_core_unsafe_enter(void);void sched_core_unsafe_exit (void );bool sched_core_wait_till_safe (unsigned long ti_check);bool sched_core_kernel_protected (void ); -int sched_core_share_pid (pid_t pid); +int sched_core_share_pid (unsigned long flags,pid_t pid);void sched_tsk_free (struct task_struct *tsk);#else #define sched_core_unsafe_enter (ignore) do { } while (0)#define sched_core_unsafe_exit (ignore) do { } while (0)#define sched_core_wait_till_safe (ignore) do { } while (0)#define sched_core_kernel_protected (ignore) do { } while (0) -#define sched_core_share_pid (pid) do { } while (0) +#define sched_core_share_pid (flags, pid) do { } while (0)#define sched_tsk_free (tsk) do { } while (0)#endif
include/uapi/linux/prctl.h +3 -0
@@ -250,5 +250,8 @@ struct prctl_mm_map {/* Request the scheduler to share a core */ #define PR_SCHED_CORE_SHARE 59 +# define PR_SCHED_CORE_CLEAR 0/* clear core_sched cookie of pid */ +# define PR_SCHED_CORE_SHARE_FROM 1/* get core_sched cookie from pid */ +# define PR_SCHED_CORE_SHARE_TO 2/* push core_sched cookie to pid */ #endif /* _LINUX_PRCTL_H */
kernel/sched/coretag.c +33 -27
79 lines not on this page · raw message
kernel/sched/debug.c +1 -0
@@ -1030,6 +1030,7 @@ void proc_sched_show_task(struct task_struct *p, struct pid_namespace *ns,#ifdef CONFIG_SCHED_CORE __PS ("core_cookie" , p->core_cookie); +__PS ("core_task_cookie" , p->core_task_cookie);#endif sched_show_numa (p, m);
kernel/sched/sched.h +1 -1
@@ -1202,7 +1202,7 @@ void sched_core_dequeue(struct rq *rq, struct task_struct *p);void sched_core_get (void );void sched_core_put (void ); -int sched_core_share_pid (pid_t pid); +int sched_core_share_pid (unsigned long flags,pid_t pid);int sched_core_share_tasks (struct task_struct *t1,struct task_struct *t2);#ifdef CONFIG_CGROUP_SCHED
kernel/sys.c +5 -1
@@ -2530,9 +2530,13 @@ SYSCALL_DEFINE5(prctl, int, option, unsigned long, arg2, unsigned long, arg3, error = (current->flags &PR_IO_FLUSHER ) ==PR_IO_FLUSHER ;break ; +#ifdef CONFIG_SCHED_CORE case PR_SCHED_CORE_SHARE : - error =sched_core_share_pid (arg2); +if (arg4 || arg5) +return -EINVAL ; + error =sched_core_share_pid (arg2, arg3);break ; +#endif default : error = -EINVAL ;break ;
Re: [PATCH -tip 23/32] sched: Add a per-thread core scheduling interface
Quoted 13 lines · Chris Hyser
kernel/sched/coretag.c +13
-0+get_task_struct (task); + +/* +* Check if this process has the right to modify the specified +* process. Use the regular "ptrace_may_access()" checks. +*/ +if (!ptrace_may_access (task,PTRACE_MODE_READ_REALCREDS )) { +rcu_read_unlock (); + err = -EPERM ; + goto out; + } +rcu_read_unlock (); +
kernel/sched/coretag.c int sched_core_share_tasks(struct task_struct *t1, struct task_struct *t2) · line 496
Quoted 10 lines
kernel/sched/coretag.c +10
-0+if (flags ==PR_SCHED_CORE_CLEAR ) { + dest = task; + src =NULL ; + }else if (flags ==PR_SCHED_CORE_SHARE_TO ) { + dest = task; + src = current; + }else if (flags ==PR_SCHED_CORE_SHARE_FROM ) { + dest = current; + src = task; + }
I should have put in an else clause to catch bad input.
Quoted 5 lines
kernel/sched/coretag.c +2
-0+ + err =sched_core_share_tasks (dest, src); out:if (task)put_task_struct (task);
Re: [PATCH -tip 23/32] sched: Add a per-thread core scheduling interface
In reply to Chris HyserQuoted 15 lines · Chris Hyser
Quoted 7 lines · Joel Fernandes (Google)
Add a per-thread core scheduling interface which allows a thread to share a core with another thread, or have a core exclusively for itself.
ChromeOS uses core-scheduling to securely enable hyperthreading. This cuts down the keypress latency in Google docs from 150ms to 50ms while improving the camera streaming frame rate by ~3%.
Inline is a patch for comment to extend this interface to make it more useful. This patch would still need to provide doc and selftests updates as well.
-chrish
---8<-----------------------
[..]
Quoted 83 lines
include/uapi/linux/prctl.h +3
-0@@ -250,5 +250,8 @@ struct prctl_mm_map {/* Request the scheduler to share a core */ #define PR_SCHED_CORE_SHARE 59 +# define PR_SCHED_CORE_CLEAR 0/* clear core_sched cookie of pid */ +# define PR_SCHED_CORE_SHARE_FROM 1/* get core_sched cookie from pid */ +# define PR_SCHED_CORE_SHARE_TO 2/* push core_sched cookie to pid */ #endif /* _LINUX_PRCTL_H */ kernel/sched/coretag.c +21
-27@@ -9,6 +9,7 @@*/ #include "sched.h" +#include "linux/prctl.h" /* * Wrapper representing a complete cookie. The address of the cookie is used as @@ -456,40 +457,45 @@ int sched_core_share_tasks(struct task_struct *t1, struct task_struct *t2) }/* Called from prctl interface: PR_SCHED_CORE_SHARE */ -int sched_core_share_pid (pid_t pid) +int sched_core_share_pid (unsigned long flags,pid_t pid) { +struct task_struct *dest; +struct task_struct *src;struct task_struct *task;int err; -if (pid ==0 ) {/* Recent current task's cookie. */ -/* Resetting a cookie requires privileges. */ -if (current->core_task_cookie) -if (!capable (CAP_SYS_ADMIN )) -return -EPERM ; - task =NULL ; - }else { -rcu_read_lock (); - task = pid ?find_task_by_vpid (pid) : current; -if (!task) { -rcu_read_unlock (); -return -ESRCH ; - } - -get_task_struct (task); - -/* -* Check if this process has the right to modify the specified -* process. Use the regular "ptrace_may_access()" checks. -*/ -if (!ptrace_may_access (task,PTRACE_MODE_READ_REALCREDS )) { -rcu_read_unlock (); - err = -EPERM ; - goto out; - } +rcu_read_lock (); + task =find_task_by_vpid (pid); +if (!task) {rcu_read_unlock (); +return -ESRCH ; } - err =sched_core_share_tasks (current, task); +get_task_struct (task); + +/* +* Check if this process has the right to modify the specified +* process. Use the regular "ptrace_may_access()" checks. +*/ +if (!ptrace_may_access (task,PTRACE_MODE_READ_REALCREDS )) { +rcu_read_unlock (); + err = -EPERM ; + goto out; + } +rcu_read_unlock (); +
kernel/sched/coretag.c int sched_core_share_tasks(struct task_struct *t1, struct task_struct *t2) · line 496
Quoted 10 lines
kernel/sched/coretag.c +10
-0+if (flags ==PR_SCHED_CORE_CLEAR ) { + dest = task; + src =NULL ; + }else if (flags ==PR_SCHED_CORE_SHARE_TO ) { + dest = task; + src = current; + }else if (flags ==PR_SCHED_CORE_SHARE_FROM ) { + dest = current; + src = task; + }
Looks ok to me except the missing else { } clause you found. Also, maybe dest/src can be renamed to from/to to make meaning of variables more clear?
Also looking forward to the docs/test updates.
thanks!
- Joel
Quoted 49 lines
kernel/sched/coretag.c +2
-0+ + err =sched_core_share_tasks (dest, src); out:if (task)put_task_struct (task);kernel/sched/debug.c +1
-0@@ -1030,6 +1030,7 @@ void proc_sched_show_task(struct task_struct *p, struct pid_namespace *ns,#ifdef CONFIG_SCHED_CORE __PS ("core_cookie" , p->core_cookie); +__PS ("core_task_cookie" , p->core_task_cookie);#endif sched_show_numa (p, m);kernel/sched/sched.h +1
-1@@ -1202,7 +1202,7 @@ void sched_core_dequeue(struct rq *rq, struct task_struct *p);void sched_core_get (void );void sched_core_put (void ); -int sched_core_share_pid (pid_t pid); +int sched_core_share_pid (unsigned long flags,pid_t pid);int sched_core_share_tasks (struct task_struct *t1,struct task_struct *t2);#ifdef CONFIG_CGROUP_SCHED kernel/sys.c +5
-1@@ -2530,9 +2530,13 @@ SYSCALL_DEFINE5(prctl, int, option, unsigned long, arg2, unsigned long, arg3, error = (current->flags &PR_IO_FLUSHER ) ==PR_IO_FLUSHER ;break ; +#ifdef CONFIG_SCHED_CORE case PR_SCHED_CORE_SHARE : - error =sched_core_share_pid (arg2); +if (arg4 || arg5) +return -EINVAL ; + error =sched_core_share_pid (arg2, arg3);break ; +#endif default : error = -EINVAL ;break ;
Re: [PATCH -tip 23/32] sched: Add a per-thread core scheduling interface
Quoted 3 lines · Joel Fernandes
Looks ok to me except the missing else { } clause you found. Also, maybe dest/src can be renamed to from/to to make meaning of variables more clear?
yes.
Quoted 1 line
Also looking forward to the docs/test updates.
on it.
Thanks.
-chrish
Re: [PATCH -tip 23/32] sched: Add a per-thread core scheduling interface
In reply to Joel FernandesQuoted 1 line · Joel Fernandes
...
Quoted 8 lines
Looks ok to me except the missing else { } clause you found. Also, maybe dest/src can be renamed to from/to to make meaning of variables more clear?
Also looking forward to the docs/test updates.
thanks!
- Joel
I fixed the else clause, made the name changes, added the documentation and fixed the selftests so they pass. Patch inline.
-chrish
.../admin-guide/hw-vuln/core-scheduling.rst | 98 ++++++++++++++++------ include/linux/sched.h | 4 +- include/uapi/linux/prctl.h | 3 + kernel/sched/coretag.c | 59 ++++++++----- kernel/sched/debug.c | 1 + kernel/sched/sched.h | 2 +- kernel/sys.c | 6 +- tools/include/uapi/linux/prctl.h | 3 + tools/testing/selftests/sched/test_coresched.c | 14 +++- 9 files changed, 136 insertions(+), 54 deletions(-)
-- >8 --
From beca9bae6750a66d8c30bbed1d6b8b26b2da05f4 Mon Sep 17 00:00:00 2001
From: chris hyser <[email protected]>
Date: Tue, 10 Nov 2020 15:35:59 -0500
Subject: [PATCH] sched: Provide a more extensive prctl interface for core
scheduling.
The current prctl interface is a "from" only interface allowing a task to join an existing core scheduling group by getting the "cookie" from a specified task/pid.
Additionally, sharing from a task without an existing core scheduling group (cookie == 0) creates a new cookie shared between the two.
"From" functionality works well for programs modified to use the prctl(), but many applications will not be modified simply for core scheduling. Using a wrapper program to share the core scheduling cookie from another task followed by an "exec" can work, but there is no means to assign a cookie for an unmodified running task.
Simply inverting the interface to a "to"-only interface, i.e. the calling task shares it's cookie with the specified task/pid also has limitations, there being no means to enable tasks to join an existing core scheduling group for instance.
The solution is to allow both, or more specifically provide a flags argument to allow various core scheduling commands, currently FROM, TO, and CLEAR.
The combination of FROM and TO allow a helper program to share the core scheduling cookie of one task/core scheduling group, with additional tasks.
if (prctl(PR_SCHED_CORE_SHARE, PR_SCHED_CORE_SHARE_FROM, src_pid, 0, 0) < 0)
handle_error("src_pid sched_core failed");
if (prctl(PR_SCHED_CORE_SHARE, PR_SCHED_CORE_SHARE_TO, dest_pid, 0, 0) < 0)
handle_error("dest_pid sched_core failed");Documentation/admin-guide/hw-vuln/core-scheduling.rst +74 -24
@@ -100,33 +100,83 @@ The color is an 8-bit value allowing for up to 256 unique colors. prctl interface ############### -A ``prtcl(2)`` command ``PR_SCHED_CORE_SHARE`` is available to a process to request -sharing a core with another process. For example, consider 2 processes ``P1`` -and ``P2`` with PIDs 100 and 200. If process ``P1`` calls -``prctl(PR_SCHED_CORE_SHARE, 200)``, the kernel makes ``P1`` share a core with ``P2``. -The kernel performs ptrace access mode checks before granting the request. +A ``prtcl(2)`` command ``PR_SCHED_CORE_SHARE`` provides an interface for the +creation of and admission and removal of tasks from core scheduling groups. -.. note:: This operation is not commutative. P1 calling - ``prctl(PR_SCHED_CORE_SHARE, pidof(P2)`` is not the same as P2 calling the - same for P1. The former case is P1 joining P2's group of processes - (which P2 would have joined with ``prctl(2)`` prior to P1's ``prctl(2)``). +:: -.. note:: The core-sharing granted with prctl(2) will be subject to - core-sharing restrictions specified by the CGroup interface. For example - if P1 and P2 are a part of 2 different tagged CGroups, then they will - not share a core even if a prctl(2) call is made. This is analogous - to how affinities are set using the cpuset interface. + #include <sys/prctl.h> -It is important to note that, on a ``CLONE_THREAD`` ``clone(2)`` syscall, the child -will be assigned the same tag as its parent and thus be allowed to share a core -with them. This design choice is because, for the security usecase, a -``CLONE_THREAD`` child can access its parent's address space anyway, so there's -no point in not allowing them to share a core. If a different behavior is -desired, the child thread can call ``prctl(2)`` as needed. This behavior is -specific to the ``prctl(2)`` interface. For the CGroup interface, the child of a -fork always shares a core with its parent. On the other hand, if a parent -was previously tagged via ``prctl(2)`` and does a regular ``fork(2)`` syscall, the -child will receive a unique tag. + int prctl(int option, unsigned long arg2, unsigned long arg3, + unsigned long arg4, unsigned long arg5); + +option: + ``PR_SCHED_CORE_SHARE`` + +arg2: + - ``PR_SCHED_CORE_CLEAR 0 -- clear core_sched cookie of pid`` + - ``PR_SCHED_CORE_SHARE_FROM 1 -- get core_sched cookie from pid`` + - ``PR_SCHED_CORE_SHARE_TO 2 -- push core_sched cookie to pid`` + +arg3: + ``tid`` of the task for which the operation applies + +arg4 and arg5: + MUST be equal to 0. + +Creation +~~~~~~~~ +Creation is accomplished by sharing a ''cookie'' from a process not currently in +a core scheduling group. + +:: + + if (prctl(PR_SCHED_CORE_SHARE, PR_SCHED_CORE_SHARE_FROM, src_tid, 0, 0) < 0) + handle_error("src_tid sched_core failed"); + +Removal +~~~~~~~ +Removing a task from a core scheduling group is done by: + +:: + + if (prctl(PR_SCHED_CORE_SHARE, PR_SCHED_CORE_SHARE_CLEAR, clr_tid, 0, 0) < 0) + handle_error("clr_tid sched_core failed"); + +Cookie Transferal +~~~~~~~~~~~~~~~~~ +Transferring a cookie between the current and other tasks is possible using +PR_SCHED_CORE_SHARE_FROM and PR_SCHED_CORE_SHARE_TO to inherit a cookie from a +specified task or a share a cookie with a task. In combination this allows a +simple helper program to pull a cookie from a task in an existing core +scheduling group and share it with already running tasks. + +:: + + if (prctl(PR_SCHED_CORE_SHARE, PR_SCHED_CORE_SHARE_FROM, from_tid, 0, 0) < 0) + handle_error("from_tid sched_core failed"); + + if (prctl(PR_SCHED_CORE_SHARE, PR_SCHED_CORE_SHARE_TO, to_tid, 0, 0) < 0) + handle_error("to_tid sched_core failed"); + + +.. note:: The core-sharing granted with ``prctl(2)`` will be subject to + core-sharing restrictions specified by the CGroup interface. For example, + if tasks T1 and T2 are a part of 2 different tagged CGroups, then they will + not share a core even if ``prctl(2)`` is called. This is analogous to how + affinities are set using the cpuset interface. + +It is important to note that, on a ``clone(2)`` syscall with ``CLONE_THREAD`` set, +the child will be assigned the same ''cookie'' as its parent and thus in the +same core scheduling group. In the security usecase, a ``CLONE_THREAD`` child +can access its parent's address space anyway (``CLONE_THREAD`` requires +``CLONE_SIGHAND`` which requires ``CLONE_VM``), so there's no point in not +allowing them to share a core. If a different behavior is desired, the child +thread can call ``prctl(2)`` as needed. This behavior is specific to the +``prctl(2)`` interface. For the CGroup interface, the child of a fork always +shares a core with its parent. On the other hand, if a parent was previously +tagged via ``prctl(2)`` and does a regular ``fork(2)`` syscall, the child will +receive a unique tag. Design/Implementation ---------------------
include/linux/sched.h +2 -2
@@ -2084,14 +2084,14 @@ void sched_core_unsafe_enter(void);void sched_core_unsafe_exit (void );bool sched_core_wait_till_safe (unsigned long ti_check);bool sched_core_kernel_protected (void ); -int sched_core_share_pid (pid_t pid); +int sched_core_share_pid (unsigned long flags,pid_t pid);void sched_tsk_free (struct task_struct *tsk);#else #define sched_core_unsafe_enter (ignore) do { } while (0)#define sched_core_unsafe_exit (ignore) do { } while (0)#define sched_core_wait_till_safe (ignore) do { } while (0)#define sched_core_kernel_protected (ignore) do { } while (0) -#define sched_core_share_pid (pid) do { } while (0) +#define sched_core_share_pid (flags, pid) do { } while (0)#define sched_tsk_free (tsk) do { } while (0)#endif
include/uapi/linux/prctl.h +3 -0
@@ -250,5 +250,8 @@ struct prctl_mm_map {/* Request the scheduler to share a core */ #define PR_SCHED_CORE_SHARE 59 +# define PR_SCHED_CORE_CLEAR 0/* clear core_sched cookie of pid */ +# define PR_SCHED_CORE_SHARE_FROM 1/* get core_sched cookie from pid */ +# define PR_SCHED_CORE_SHARE_TO 2/* push core_sched cookie to pid */ #endif /* _LINUX_PRCTL_H */
kernel/sched/coretag.c +36 -23
78 lines not on this page · raw message
kernel/sched/debug.c +1 -0
@@ -1030,6 +1030,7 @@ void proc_sched_show_task(struct task_struct *p, struct pid_namespace *ns,#ifdef CONFIG_SCHED_CORE __PS ("core_cookie" , p->core_cookie); +__PS ("core_task_cookie" , p->core_task_cookie);#endif sched_show_numa (p, m);
kernel/sched/sched.h +1 -1
@@ -1229,7 +1229,7 @@ void sched_core_dequeue(struct rq *rq, struct task_struct *p);void sched_core_get (void );void sched_core_put (void ); -int sched_core_share_pid (pid_t pid); +int sched_core_share_pid (unsigned long flags,pid_t pid);int sched_core_share_tasks (struct task_struct *t1,struct task_struct *t2);#ifdef CONFIG_CGROUP_SCHED
kernel/sys.c +5 -1
@@ -2530,9 +2530,13 @@ SYSCALL_DEFINE5(prctl, int, option, unsigned long, arg2, unsigned long, arg3, error = (current->flags &PR_IO_FLUSHER ) ==PR_IO_FLUSHER ;break ; +#ifdef CONFIG_SCHED_CORE case PR_SCHED_CORE_SHARE : - error =sched_core_share_pid (arg2); +if (arg4 || arg5) +return -EINVAL ; + error =sched_core_share_pid (arg2, arg3);break ; +#endif default : error = -EINVAL ;break ;
tools/include/uapi/linux/prctl.h +3 -0
@@ -249,5 +249,8 @@ struct prctl_mm_map {/* Request the scheduler to share a core */ #define PR_SCHED_CORE_SHARE 59 +# define PR_SCHED_CORE_CLEAR 0/* clear core_sched cookie of pid */ +# define PR_SCHED_CORE_SHARE_FROM 1/* get core_sched cookie from pid */ +# define PR_SCHED_CORE_SHARE_TO 2/* push core_sched cookie to pid */ #endif /* _LINUX_PRCTL_H */
tools/testing/selftests/sched/test_coresched.c +11 -3
@@ -23,6 +23,9 @@#ifndef PR_SCHED_CORE_SHARE #define PR_SCHED_CORE_SHARE 59 +# define PR_SCHED_CORE_CLEAR 0 +# define PR_SCHED_CORE_SHARE_FROM 1 +# define PR_SCHED_CORE_SHARE_TO 2#endif #ifndef DEBUG_PRINT
@@ -391,9 +394,14 @@ struct task_state *add_task(char *p) pid = mem->pid_share; mem->pid_share =0 ; -if (pid ==-1 ) - pid =0 ; -prctl (PR_SCHED_CORE_SHARE , pid); + +if (pid ==-1 ) { +if (prctl (PR_SCHED_CORE_SHARE ,PR_SCHED_CORE_CLEAR ,getpid (),0 ,0 )) +perror ("prctl() PR_SCHED_CORE_CLEAR failed" ); + }else { +if (prctl (PR_SCHED_CORE_SHARE ,PR_SCHED_CORE_SHARE_FROM , pid,0 ,0 )) +perror ("prctl() PR_SCHED_CORE_SHARE_FROM failed" ); + }pthread_mutex_unlock (&mem->m);pthread_cond_signal (&mem->cond_par); }
Re: [PATCH -tip 23/32] sched: Add a per-thread core scheduling interface
kernel/sched/debug.c void proc_sched_show_task(struct task_struct *p, struct pid_namespace *ns, · line 1034
Quoted 10 lines
kernel/sched/debug.c +1
-0@@ -1030,6 +1030,7 @@ void proc_sched_show_task(struct task_struct *p, struct pid_namespace *ns,#ifdef CONFIG_SCHED_CORE __PS ("core_cookie" , p->core_cookie); +__PS ("core_task_cookie" , p->core_task_cookie);#endif
Hmm, so the final cookie of the task is always p->core_cookie. This is what the scheduler uses. All other fields are ingredients to derive the final cookie value.
I will drop this hunk from your overall diff, but let me know if you disagree!
thanks,
- Joel
Re: [PATCH -tip 23/32] sched: Add a per-thread core scheduling interface
Quoted 15 lines · Joel Fernandes
kernel/sched/debug.c void proc_sched_show_task(struct task_struct *p, struct pid_namespace *ns, · line 1034
Quoted 10 lines
kernel/sched/debug.c +1
-0@@ -1030,6 +1030,7 @@ void proc_sched_show_task(struct task_struct *p, struct pid_namespace *ns,#ifdef CONFIG_SCHED_CORE __PS ("core_cookie" , p->core_cookie); +__PS ("core_task_cookie" , p->core_task_cookie);#endif Hmm, so the final cookie of the task is always
p->core_cookie. This is what the scheduler uses. All other fields are ingredients to derive the final cookie value.
Quoted 2 lines
I will drop this hunk from your overall diff, but let me know if you disagree!
No problem. That was there primarily for debugging.
Thanks.
-chrish
Re: [PATCH -tip 23/32] sched: Add a per-thread core scheduling interface
Quoted 15 lines · chris hyser
Quoted 14 lines · Joel Fernandes
kernel/sched/debug.c void proc_sched_show_task(struct task_struct *p, struct pid_namespace *ns, · line 1033
Quoted 9 lines
kernel/sched/debug.c +1
-0@@ -1030,6 +1030,7 @@ void proc_sched_show_task(struct task_struct *p, struct pid_namespace *ns,#ifdef CONFIG_SCHED_CORE __PS ("core_cookie" , p->core_cookie); +__PS ("core_task_cookie" , p->core_task_cookie);#endif Hmm, so the final cookie of the task is always
p->core_cookie. This is what the scheduler uses. All other fields are ingredients to derive the final cookie value.
Quoted 5 lines
Quoted 2 lines
I will drop this hunk from your overall diff, but let me know if you disagree!
No problem. That was there primarily for debugging.
Ok. I squashed Josh's changes into this patch and several of my fixups. So there'll be 3 patches:
1. CGroup + prctl (single patch as it is hell to split it)
2. Documentation
3. ksefltests
Below is the diff of #1. I still have to squash in the stop_machine removal and some more review changes. But other than that, please take a look and let me know anything that's odd. I will test further as well.
Also next series will only be interface as I want to see if I can get lucky enough to have Peter look at it before he leaves for PTO next week. For the other features, I will post different series as I prepare them. One series for interface, and another for kernel protection / migration changes.
---8<-----------------------
include/linux/sched.h +6 -0
@@ -688,6 +688,8 @@ struct task_struct {#ifdef CONFIG_SCHED_CORE struct rb_node core_node;unsigned long core_cookie; +unsigned long core_task_cookie; +unsigned long core_group_cookie;unsigned int core_occupation;#endif
@@ -2081,11 +2083,15 @@ void sched_core_unsafe_enter(void);void sched_core_unsafe_exit (void );bool sched_core_wait_till_safe (unsigned long ti_check);bool sched_core_kernel_protected (void ); +int sched_core_share_pid (unsigned long flags,pid_t pid); +void sched_tsk_free (struct task_struct *tsk);#else #define sched_core_unsafe_enter (ignore) do { } while (0)#define sched_core_unsafe_exit (ignore) do { } while (0)#define sched_core_wait_till_safe (ignore) do { } while (0)#define sched_core_kernel_protected (ignore) do { } while (0) +#define sched_core_share_pid (flags, pid) do { } while (0) +#define sched_tsk_free (tsk) do { } while (0)#endif #endif
include/uapi/linux/prctl.h +6 -0
@@ -248,4 +248,10 @@ struct prctl_mm_map {#define PR_SET_IO_FLUSHER 57#define PR_GET_IO_FLUSHER 58 +/* Request the scheduler to share a core */ +#define PR_SCHED_CORE_SHARE 59 +#define PR_SCHED_CORE_CLEAR 0/* clear core_sched cookie of pid */ +#define PR_SCHED_CORE_SHARE_FROM 1/* get core_sched cookie from pid */ +#define PR_SCHED_CORE_SHARE_TO 2/* push core_sched cookie to pid */ +#endif /* _LINUX_PRCTL_H */
kernel/fork.c +1 -0
@@ -736,6 +736,7 @@ void __put_task_struct(struct task_struct *tsk)exit_creds (tsk);delayacct_tsk_free (tsk);put_signal_struct (tsk->signal); +sched_tsk_free (tsk);if (!profile_handoff_task (tsk))free_task (tsk);
kernel/sched/Makefile +1 -0
@@ -36,3 +36,4 @@ obj-$(CONFIG_CPU_FREQ_GOV_SCHEDUTIL) += cpufreq_schedutil.o obj-$(CONFIG_MEMBARRIER) += membarrier.o obj-$(CONFIG_CPU_ISOLATION) += isolation.o obj-$(CONFIG_PSI) += psi.o +obj-$(CONFIG_SCHED_CORE) += coretag.o
kernel/sched/core.c +120 -19
247 lines not on this page · raw message
kernel/sched/coretag.c new file +734 -0
734 lines not on this page · raw message
kernel/sched/debug.c +4 -0
@@ -1024,6 +1024,10 @@ void proc_sched_show_task(struct task_struct *p, struct pid_namespace *ns,__PS ("clock-delta" , t1-t0); } +#ifdef CONFIG_SCHED_CORE +__PS ("core_cookie" , p->core_cookie); +#endif +sched_show_numa (p, m); }
kernel/sched/sched.h +57 -4
94 lines not on this page · raw message
kernel/sys.c +7 -0
@@ -2530,6 +2530,13 @@ SYSCALL_DEFINE5(prctl, int, option, unsigned long, arg2, unsigned long, arg3, error = (current->flags &PR_IO_FLUSHER ) ==PR_IO_FLUSHER ;break ; +#ifdef CONFIG_SCHED_CORE +case PR_SCHED_CORE_SHARE : +if (arg4 || arg5) +return -EINVAL ; + error =sched_core_share_pid (arg2, arg3); +break ; +#endif default : error = -EINVAL ;break ;
tools/include/uapi/linux/prctl.h +6 -0
@@ -247,4 +247,10 @@ struct prctl_mm_map {#define PR_SET_IO_FLUSHER 57#define PR_GET_IO_FLUSHER 58 +/* Request the scheduler to share a core */ +#define PR_SCHED_CORE_SHARE 59 +#define PR_SCHED_CORE_CLEAR 0/* clear core_sched cookie of pid */ +#define PR_SCHED_CORE_SHARE_FROM 1/* get core_sched cookie from pid */ +#define PR_SCHED_CORE_SHARE_TO 2/* push core_sched cookie to pid */ +#endif /* _LINUX_PRCTL_H */
Signature · 1 line
2.29.2.684.gfbc64c5ab5-goog
Re: [PATCH -tip 23/32] sched: Add a per-thread core scheduling interface
Quoted 28 lines · Joel Fernandes
Quoted 15 lines · chris hyser
Quoted 14 lines · Joel Fernandes
kernel/sched/debug.c void proc_sched_show_task(struct task_struct *p, struct pid_namespace *ns, · line 1033
Quoted 9 lines
kernel/sched/debug.c +1
-0@@ -1030,6 +1030,7 @@ void proc_sched_show_task(struct task_struct *p, struct pid_namespace *ns,#ifdef CONFIG_SCHED_CORE __PS ("core_cookie" , p->core_cookie); +__PS ("core_task_cookie" , p->core_task_cookie);#endif Hmm, so the final cookie of the task is always
p->core_cookie. This is what the scheduler uses. All other fields are ingredients to derive the final cookie value.Quoted 5 lines
Quoted 2 lines
I will drop this hunk from your overall diff, but let me know if you disagree!
No problem. That was there primarily for debugging.
Ok. I squashed Josh's changes into this patch and several of my fixups. So there'll be 3 patches:
1. CGroup + prctl (single patch as it is hell to split it)
2. Documentation
3. ksefltests
Quoted 3 lines
Below is the diff of #1. I still have to squash in the
stop_machineremoval and some more review changes. But other than that, please take a look and let me know anything that's odd. I will test further as well.
Will do. Looking at it now and just started a build.
-chrish