Core scheduling v3
v3 cover Core scheduling v3
Third iteration of the Core-Scheduling feature.
This version fixes mostly correctness related issues in v2 and addresses performance issues. Also, addressed some crashes related to cgroups and cpu hotplugging.
We have tested and verified that incompatible processes are not selected during schedule. In terms of performance, the impact depends on the workload:
- on CPU intensive applications that use all the logical CPUs with SMT enabled, enabling core scheduling performs better than nosmt.
- on mixed workloads with considerable io compared to cpu usage, nosmt seems to perform better than core scheduling.
Changes in v3
-------------
- Fixes the issue of sibling picking up an incompatible task
- Aaron Lu
- Vineeth Pillai
- Julien Desfossez
- Fixes the issue of starving threads due to forced idle
- Peter Zijlstra
- Fixes the refcounting issue when deleting a cgroup with tag
- Julien Desfossez
- Fixes a crash during cpu offline/online with coresched enabled
- Vineeth Pillai
- Fixes a comparison logic issue in sched_core_find
- Aaron Lu
Changes in v2
-------------
- Fixes for couple of NULL pointer dereference crashes
- Subhra Mazumdar
- Tim Chen
- Improves priority comparison logic for process in different cpus
- Peter Zijlstra
- Aaron Lu
- Fixes a hard lockup in rq locking
- Vineeth Pillai
- Julien Desfossez
- Fixes a performance issue seen on IO heavy workloads
- Vineeth Pillai
- Julien Desfossez
- Fix for 32bit build
- Aubrey Li
Issues
------
- Comparing process priority across cpus is not accurate
TODO
----
- Decide on the API for exposing the feature to userland
---
Peter Zijlstra (16):
stop_machine: Fix stop_cpus_in_progress ordering
sched: Fix kerneldoc comment for ia64_set_curr_task
sched: Wrap rq::lock access
sched/{rt,deadline}: Fix set_next_task vs pick_next_task
sched: Add task_struct pointer to sched_class::set_curr_task sched/fair: Export newidle_balance()
sched: Allow put_prev_task() to drop rq->lock
sched: Rework pick_next_task() slow-path
sched: Introduce sched_class::pick_task()
sched: Core-wide rq->lock
sched: Basic tracking of matching tasks
sched: A quick and dirty cgroup tagging interface
sched: Add core wide task selection and scheduling. sched/fair: Add a few assertions
sched: Trivial forced-newidle balancer
sched: Debug bits...
include/linux/sched.h | 9 +- kernel/Kconfig.preempt | 7 +- kernel/sched/core.c | 858 +++++++++++++++++++++++++++++++++++++-- kernel/sched/cpuacct.c | 12 +- kernel/sched/deadline.c | 99 +++-- kernel/sched/debug.c | 4 +- kernel/sched/fair.c | 180 ++++---- kernel/sched/idle.c | 42 +- kernel/sched/pelt.h | 2 +- kernel/sched/rt.c | 96 ++--- kernel/sched/sched.h | 237 ++++++++--- kernel/sched/stop_task.c | 35 +- kernel/sched/topology.c | 4 +- kernel/stop_machine.c | 2 + 14 files changed, 1250 insertions(+), 337 deletions(-)
Signature · 1 line
2.17.1
RFC PATCH v3 1/16 stop_machine: Fix stop_cpus_in_progress ordering
From: Peter Zijlstra <[email protected]>
Make sure the entire for loop has stop_cpus_in_progress set.
--- kernel/stop_machine.c | 2 ++ 1 file changed, 2 insertions(+)
kernel/stop_machine.c +2 -0
@@ -375,6 +375,7 @@ static bool queue_stop_cpus_work(const struct cpumask *cpumask,*/ preempt_disable (); stop_cpus_in_progress = true; +barrier ();for_each_cpu (cpu, cpumask) { work = &per_cpu (cpu_stopper.stop_work, cpu); work->fn = fn;
@@ -383,6 +384,7 @@ static bool queue_stop_cpus_work(const struct cpumask *cpumask,if (cpu_stop_queue_work (cpu, work)) queued = true; } +barrier (); stop_cpus_in_progress = false;preempt_enable ();
Signature · 1 line
2.17.1
RFC PATCH v3 2/16 sched: Fix kerneldoc comment for ia64_set_curr_task
In reply to Vineeth Remanan PillaiFrom: Peter Zijlstra <[email protected]>
--- kernel/sched/core.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/sched/core.c +1 -1
@@ -6287,7 +6287,7 @@ struct task_struct *curr_task(int cpu)#ifdef CONFIG_IA64 /** -* set_curr_task - set the current task for a given CPU. +* ia64_set_curr_task - set the current task for a given CPU. * @cpu: the processor in question. * @p: the task pointer to set. *
Signature · 1 line
2.17.1
RFC PATCH v3 3/16 sched: Wrap rq::lock access
In reply to Vineeth Remanan PillaiFrom: Peter Zijlstra <[email protected]>
In preparation of playing games with rq->lock, abstract the thing using an accessor.
---
Changes in v2
-------------
- Fixes a deadlock due in double_rq_lock and double_lock_lock
- Vineeth Pillai
- Julien Desfossez
- Fixes 32bit build.
- Aubrey Li
--- kernel/sched/core.c | 46 ++++++++--------- kernel/sched/cpuacct.c | 12 ++--- kernel/sched/deadline.c | 18 +++---- kernel/sched/debug.c | 4 +- kernel/sched/fair.c | 40 +++++++-------- kernel/sched/idle.c | 4 +- kernel/sched/pelt.h | 2 +- kernel/sched/rt.c | 8 +-- kernel/sched/sched.h | 106 ++++++++++++++++++++-------------------- kernel/sched/topology.c | 4 +- 10 files changed, 123 insertions(+), 121 deletions(-)
kernel/sched/core.c +23 -23
@@ -72,12 +72,12 @@ struct rq *__task_rq_lock(struct task_struct *p, struct rq_flags *rf)for (;;) { rq =task_rq (p); -raw_spin_lock (&rq->lock); +raw_spin_lock (rq_lockp (rq));if (likely (rq ==task_rq (p) && !task_on_rq_migrating (p))) {rq_pin_lock (rq, rf);return rq; } -raw_spin_unlock (&rq->lock); +raw_spin_unlock (rq_lockp (rq));while (unlikely (task_on_rq_migrating (p)))cpu_relax ();
@@ -96,7 +96,7 @@ struct rq *task_rq_lock(struct task_struct *p, struct rq_flags *rf)for (;;) {raw_spin_lock_irqsave (&p->pi_lock, rf->flags); rq =task_rq (p); -raw_spin_lock (&rq->lock); +raw_spin_lock (rq_lockp (rq));/* * move_queued_task() task_rq_lock() *
@@ -118,7 +118,7 @@ struct rq *task_rq_lock(struct task_struct *p, struct rq_flags *rf)rq_pin_lock (rq, rf);return rq; } -raw_spin_unlock (&rq->lock); +raw_spin_unlock (rq_lockp (rq));raw_spin_unlock_irqrestore (&p->pi_lock, rf->flags);while (unlikely (task_on_rq_migrating (p)))
@@ -188,7 +188,7 @@ void update_rq_clock(struct rq *rq) {s64 delta; -lockdep_assert_held (&rq->lock); +lockdep_assert_held (rq_lockp (rq));if (rq->clock_update_flags &RQCF_ACT_SKIP )return ;
@@ -497,7 +497,7 @@ void resched_curr(struct rq *rq)struct task_struct *curr = rq->curr;int cpu; -lockdep_assert_held (&rq->lock); +lockdep_assert_held (rq_lockp (rq));if (test_tsk_need_resched (curr))return ;
@@ -521,10 +521,10 @@ void resched_cpu(int cpu)struct rq *rq =cpu_rq (cpu);unsigned long flags; -raw_spin_lock_irqsave (&rq->lock, flags); +raw_spin_lock_irqsave (rq_lockp (rq), flags);if (cpu_online (cpu) || cpu ==smp_processor_id ())resched_curr (rq); -raw_spin_unlock_irqrestore (&rq->lock, flags); +raw_spin_unlock_irqrestore (rq_lockp (rq), flags); }#ifdef CONFIG_SMP
@@ -956,7 +956,7 @@ static inline bool is_cpu_allowed(struct task_struct *p, int cpu)static struct rq *move_queued_task (struct rq *rq,struct rq_flags *rf,struct task_struct *p,int new_cpu) { -lockdep_assert_held (&rq->lock); +lockdep_assert_held (rq_lockp (rq));WRITE_ONCE (p->on_rq,TASK_ON_RQ_MIGRATING );dequeue_task (rq, p,DEQUEUE_NOCLOCK );
@@ -1070,7 +1070,7 @@ void do_set_cpus_allowed(struct task_struct *p, const struct cpumask *new_mask)* Because __kthread_bind() calls this on blocked tasks without * holding rq->lock. */ -lockdep_assert_held (&rq->lock); +lockdep_assert_held (rq_lockp (rq));dequeue_task (rq, p,DEQUEUE_SAVE |DEQUEUE_NOCLOCK ); }if (running)
@@ -1203,7 +1203,7 @@ void set_task_cpu(struct task_struct *p, unsigned int new_cpu)* task_rq_lock(). */ WARN_ON_ONCE (debug_locks && !(lockdep_is_held (&p->pi_lock) || -lockdep_is_held (&task_rq (p)->lock))); +lockdep_is_held (rq_lockp (task_rq (p)))));#endif /* * Clearly, migrating tasks to offline CPUs is a fairly daft thing.
@@ -1732,7 +1732,7 @@ ttwu_do_activate(struct rq *rq, struct task_struct *p, int wake_flags, {int en_flags =ENQUEUE_WAKEUP |ENQUEUE_NOCLOCK ; -lockdep_assert_held (&rq->lock); +lockdep_assert_held (rq_lockp (rq));#ifdef CONFIG_SMP if (p->sched_contributes_to_load)
@@ -2123,7 +2123,7 @@ static void try_to_wake_up_local(struct task_struct *p, struct rq_flags *rf) WARN_ON_ONCE(p == current))return ; -lockdep_assert_held (&rq->lock); +lockdep_assert_held (rq_lockp (rq));if (!raw_spin_trylock (&p->pi_lock)) {/*
@@ -2609,10 +2609,10 @@ prepare_lock_switch(struct rq *rq, struct task_struct *next, struct rq_flags *rf* do an early lockdep release here: */ rq_unpin_lock (rq, rf); -spin_release (&rq->lock.dep_map,1 , _THIS_IP_); +spin_release (&rq_lockp (rq)->dep_map,1 , _THIS_IP_);#ifdef CONFIG_DEBUG_SPINLOCK /* this is a valid case when another task releases the spinlock */ - rq->lock.owner = next; +rq_lockp (rq)->owner = next;#endif }
@@ -2623,8 +2623,8 @@ static inline void finish_lock_switch(struct rq *rq)* fix up the runqueue lock - which gets 'carried over' from * prev into current: */ -spin_acquire (&rq->lock.dep_map,0 ,0 , _THIS_IP_); -raw_spin_unlock_irq (&rq->lock); +spin_acquire (&rq_lockp (rq)->dep_map,0 ,0 , _THIS_IP_); +raw_spin_unlock_irq (rq_lockp (rq)); }/*
@@ -2698,7 +2698,7 @@ static struct rq *finish_task_switch(struct task_struct *prev)* schedule() * preempt_disable(); // 1 * __schedule() -* raw_spin_lock_irq(&rq->lock) // 2 +* raw_spin_lock_irq(rq_lockp(rq)) // 2 * * Also, see FORK_PREEMPT_COUNT. */
@@ -2774,7 +2774,7 @@ static void __balance_callback(struct rq *rq)void (*func)(struct rq *rq);unsigned long flags; -raw_spin_lock_irqsave (&rq->lock, flags); +raw_spin_lock_irqsave (rq_lockp (rq), flags); head = rq->balance_callback; rq->balance_callback =NULL ;while (head) {
@@ -2785,7 +2785,7 @@ static void __balance_callback(struct rq *rq)func (rq); } -raw_spin_unlock_irqrestore (&rq->lock, flags); +raw_spin_unlock_irqrestore (rq_lockp (rq), flags); }static inline void balance_callback (struct rq *rq)
@@ -5414,7 +5414,7 @@ void init_idle(struct task_struct *idle, int cpu)unsigned long flags;raw_spin_lock_irqsave (&idle->pi_lock, flags); -raw_spin_lock (&rq->lock); +raw_spin_lock (rq_lockp (rq));__sched_fork (0 , idle); idle->state =TASK_RUNNING ;
@@ -5451,7 +5451,7 @@ void init_idle(struct task_struct *idle, int cpu)#ifdef CONFIG_SMP idle->on_cpu =1 ;#endif -raw_spin_unlock (&rq->lock); +raw_spin_unlock (rq_lockp (rq));raw_spin_unlock_irqrestore (&idle->pi_lock, flags);/* Set the preempt count _outside_ the spinlocks! */
@@ -6019,7 +6019,7 @@ void __init sched_init(void)struct rq *rq; rq =cpu_rq (i); -raw_spin_lock_init (&rq->lock); +raw_spin_lock_init (&rq->__lock); rq->nr_running =0 ; rq->calc_load_active =0 ; rq->calc_load_update = jiffies +LOAD_FREQ ;
kernel/sched/cpuacct.c +6 -6
@@ -111,7 +111,7 @@ static u64 cpuacct_cpuusage_read(struct cpuacct *ca, int cpu,/* * Take rq->lock to make 64-bit read safe on 32-bit platforms. */ -raw_spin_lock_irq (&cpu_rq (cpu)->lock); +raw_spin_lock_irq (rq_lockp (cpu_rq (cpu)));#endif if (index ==CPUACCT_STAT_NSTATS ) {
@@ -125,7 +125,7 @@ static u64 cpuacct_cpuusage_read(struct cpuacct *ca, int cpu, }#ifndef CONFIG_64BIT -raw_spin_unlock_irq (&cpu_rq (cpu)->lock); +raw_spin_unlock_irq (rq_lockp (cpu_rq (cpu)));#endif return data;
@@ -140,14 +140,14 @@ static void cpuacct_cpuusage_write(struct cpuacct *ca, int cpu, u64 val)/* * Take rq->lock to make 64-bit write safe on 32-bit platforms. */ -raw_spin_lock_irq (&cpu_rq (cpu)->lock); +raw_spin_lock_irq (rq_lockp (cpu_rq (cpu)));#endif for (i =0 ; i <CPUACCT_STAT_NSTATS ; i++) cpuusage->usages[i] = val;#ifndef CONFIG_64BIT -raw_spin_unlock_irq (&cpu_rq (cpu)->lock); +raw_spin_unlock_irq (rq_lockp (cpu_rq (cpu)));#endif }
@@ -252,13 +252,13 @@ static int cpuacct_all_seq_show(struct seq_file *m, void *V)* Take rq->lock to make 64-bit read safe on 32-bit * platforms. */ -raw_spin_lock_irq (&cpu_rq (cpu)->lock); +raw_spin_lock_irq (rq_lockp (cpu_rq (cpu)));#endif seq_printf (m," %llu" , cpuusage->usages[index]);#ifndef CONFIG_64BIT -raw_spin_unlock_irq (&cpu_rq (cpu)->lock); +raw_spin_unlock_irq (rq_lockp (cpu_rq (cpu)));#endif }seq_puts (m,"\n" );
kernel/sched/deadline.c +9 -9
@@ -80,7 +80,7 @@ void __add_running_bw(u64 dl_bw, struct dl_rq *dl_rq) {u64 old = dl_rq->running_bw; -lockdep_assert_held (&(rq_of_dl_rq (dl_rq))->lock); +lockdep_assert_held (rq_lockp ((rq_of_dl_rq (dl_rq)))); dl_rq->running_bw += dl_bw;SCHED_WARN_ON (dl_rq->running_bw < old);/* overflow */ SCHED_WARN_ON (dl_rq->running_bw > dl_rq->this_bw);
@@ -93,7 +93,7 @@ void __sub_running_bw(u64 dl_bw, struct dl_rq *dl_rq) {u64 old = dl_rq->running_bw; -lockdep_assert_held (&(rq_of_dl_rq (dl_rq))->lock); +lockdep_assert_held (rq_lockp ((rq_of_dl_rq (dl_rq)))); dl_rq->running_bw -= dl_bw;SCHED_WARN_ON (dl_rq->running_bw > old);/* underflow */ if (dl_rq->running_bw > old)
@@ -107,7 +107,7 @@ void __add_rq_bw(u64 dl_bw, struct dl_rq *dl_rq) {u64 old = dl_rq->this_bw; -lockdep_assert_held (&(rq_of_dl_rq (dl_rq))->lock); +lockdep_assert_held (rq_lockp ((rq_of_dl_rq (dl_rq)))); dl_rq->this_bw += dl_bw;SCHED_WARN_ON (dl_rq->this_bw < old);/* overflow */ }
@@ -117,7 +117,7 @@ void __sub_rq_bw(u64 dl_bw, struct dl_rq *dl_rq) {u64 old = dl_rq->this_bw; -lockdep_assert_held (&(rq_of_dl_rq (dl_rq))->lock); +lockdep_assert_held (rq_lockp ((rq_of_dl_rq (dl_rq)))); dl_rq->this_bw -= dl_bw;SCHED_WARN_ON (dl_rq->this_bw > old);/* underflow */ if (dl_rq->this_bw > old)
@@ -892,7 +892,7 @@ static int start_dl_timer(struct task_struct *p)ktime_t now, act;s64 delta; -lockdep_assert_held (&rq->lock); +lockdep_assert_held (rq_lockp (rq));/* * We want the timer to fire at the deadline, but considering
@@ -1002,9 +1002,9 @@ static enum hrtimer_restart dl_task_timer(struct hrtimer *timer)* If the runqueue is no longer available, migrate the * task elsewhere. This necessarily changes rq. */ -lockdep_unpin_lock (&rq->lock, rf.cookie); +lockdep_unpin_lock (rq_lockp (rq), rf.cookie); rq =dl_task_offline_migration (rq, p); - rf.cookie =lockdep_pin_lock (&rq->lock); + rf.cookie =lockdep_pin_lock (rq_lockp (rq));update_rq_clock (rq);/*
@@ -1619,7 +1619,7 @@ static void migrate_task_rq_dl(struct task_struct *p, int new_cpu __maybe_unused* from try_to_wake_up(). Hence, p->pi_lock is locked, but * rq->lock is not... So, lock it */ -raw_spin_lock (&rq->lock); +raw_spin_lock (rq_lockp (rq));if (p->dl.dl_non_contending) {sub_running_bw (&p->dl, &rq->dl); p->dl.dl_non_contending =0 ;
@@ -1634,7 +1634,7 @@ static void migrate_task_rq_dl(struct task_struct *p, int new_cpu __maybe_unusedput_task_struct (p); }sub_rq_bw (&p->dl, &rq->dl); -raw_spin_unlock (&rq->lock); +raw_spin_unlock (rq_lockp (rq)); }static void check_preempt_equal_dl (struct rq *rq,struct task_struct *p)
kernel/sched/debug.c +2 -2
@@ -515,7 +515,7 @@ void print_cfs_rq(struct seq_file *m, int cpu, struct cfs_rq *cfs_rq)SEQ_printf (m," .%-30s: %Ld.%06ld\n" ,"exec_clock" ,SPLIT_NS (cfs_rq->exec_clock)); -raw_spin_lock_irqsave (&rq->lock, flags); +raw_spin_lock_irqsave (rq_lockp (rq), flags);if (rb_first_cached (&cfs_rq->tasks_timeline)) MIN_vruntime = (__pick_first_entity (cfs_rq))->vruntime; last =__pick_last_entity (cfs_rq);
@@ -523,7 +523,7 @@ void print_cfs_rq(struct seq_file *m, int cpu, struct cfs_rq *cfs_rq) max_vruntime = last->vruntime; min_vruntime = cfs_rq->min_vruntime; rq0_min_vruntime =cpu_rq (0 )->cfs.min_vruntime; -raw_spin_unlock_irqrestore (&rq->lock, flags); +raw_spin_unlock_irqrestore (rq_lockp (rq), flags);SEQ_printf (m," .%-30s: %Ld.%06ld\n" ,"MIN_vruntime" ,SPLIT_NS (MIN_vruntime)); SEQ_printf(m, " .%-30s: %Ld.%06ld\n", "min_vruntime",
kernel/sched/fair.c +20 -20
@@ -4996,7 +4996,7 @@ static void __maybe_unused update_runtime_enabled(struct rq *rq) {struct task_group *tg; -lockdep_assert_held (&rq->lock); +lockdep_assert_held (rq_lockp (rq));rcu_read_lock ();list_for_each_entry_rcu (tg, &task_groups, list) {
@@ -5015,7 +5015,7 @@ static void __maybe_unused unthrottle_offline_cfs_rqs(struct rq *rq) {struct task_group *tg; -lockdep_assert_held (&rq->lock); +lockdep_assert_held (rq_lockp (rq));rcu_read_lock ();list_for_each_entry_rcu (tg, &task_groups, list) {
@@ -6773,7 +6773,7 @@ static void migrate_task_rq_fair(struct task_struct *p, int new_cpu)* In case of TASK_ON_RQ_MIGRATING we in fact hold the 'old' * rq->lock and can modify state directly. */ -lockdep_assert_held (&task_rq (p)->lock); +lockdep_assert_held (rq_lockp (task_rq (p)));detach_entity_cfs_rq (&p->se); }else {
@@ -7347,7 +7347,7 @@ static int task_hot(struct task_struct *p, struct lb_env *env) {s64 delta; -lockdep_assert_held (&env->src_rq->lock); +lockdep_assert_held (rq_lockp (env->src_rq));if (p->sched_class != &fair_sched_class)return 0 ;
@@ -7441,7 +7441,7 @@ int can_migrate_task(struct task_struct *p, struct lb_env *env) {int tsk_cache_hot; -lockdep_assert_held (&env->src_rq->lock); +lockdep_assert_held (rq_lockp (env->src_rq));/* * We do not migrate tasks that are:
@@ -7519,7 +7519,7 @@ int can_migrate_task(struct task_struct *p, struct lb_env *env)*/ static void detach_task (struct task_struct *p,struct lb_env *env) { -lockdep_assert_held (&env->src_rq->lock); +lockdep_assert_held (rq_lockp (env->src_rq)); p->on_rq =TASK_ON_RQ_MIGRATING ;deactivate_task (env->src_rq, p,DEQUEUE_NOCLOCK );
@@ -7536,7 +7536,7 @@ static struct task_struct *detach_one_task(struct lb_env *env) {struct task_struct *p; -lockdep_assert_held (&env->src_rq->lock); +lockdep_assert_held (rq_lockp (env->src_rq));list_for_each_entry_reverse (p, &env->src_rq->cfs_tasks, se.group_node) {
@@ -7572,7 +7572,7 @@ static int detach_tasks(struct lb_env *env)unsigned long load;int detached =0 ; -lockdep_assert_held (&env->src_rq->lock); +lockdep_assert_held (rq_lockp (env->src_rq));if (env->imbalance <=0 )return 0 ;
@@ -7653,7 +7653,7 @@ static int detach_tasks(struct lb_env *env)*/ static void attach_task (struct rq *rq,struct task_struct *p) { -lockdep_assert_held (&rq->lock); +lockdep_assert_held (rq_lockp (rq));BUG_ON (task_rq (p) != rq);activate_task (rq, p,ENQUEUE_NOCLOCK );
@@ -9206,7 +9206,7 @@ static int load_balance(int this_cpu, struct rq *this_rq,if (need_active_balance (&env)) {unsigned long flags; -raw_spin_lock_irqsave (&busiest->lock, flags); +raw_spin_lock_irqsave (rq_lockp (busiest), flags);/* * Don't kick the active_load_balance_cpu_stop,
@@ -9214,7 +9214,7 @@ static int load_balance(int this_cpu, struct rq *this_rq,* moved to this_cpu: */ if (!cpumask_test_cpu (this_cpu, &busiest->curr->cpus_allowed)) { -raw_spin_unlock_irqrestore (&busiest->lock, +raw_spin_unlock_irqrestore (rq_lockp (busiest), flags); env.flags |=LBF_ALL_PINNED ; goto out_one_pinned;
@@ -9230,7 +9230,7 @@ static int load_balance(int this_cpu, struct rq *this_rq, busiest->push_cpu = this_cpu; active_balance =1 ; } -raw_spin_unlock_irqrestore (&busiest->lock, flags); +raw_spin_unlock_irqrestore (rq_lockp (busiest), flags);if (active_balance) { stop_one_cpu_nowait(cpu_of(busiest),
@@ -9969,7 +9969,7 @@ static void nohz_newidle_balance(struct rq *this_rq) time_before(jiffies, READ_ONCE(nohz.next_blocked))) return; -raw_spin_unlock (&this_rq->lock); +raw_spin_unlock (rq_lockp (this_rq));/* * This CPU is going to be idle and blocked load of idle CPUs * need to be updated. Run the ilb locally as it is a good
@@ -9978,7 +9978,7 @@ static void nohz_newidle_balance(struct rq *this_rq)*/ if (!_nohz_idle_balance (this_rq,NOHZ_STATS_KICK ,CPU_NEWLY_IDLE ))kick_ilb (NOHZ_STATS_KICK ); - raw_spin_lock(&this_rq->lock); + raw_spin_lock(rq_lockp(this_rq)); }#else /* !CONFIG_NO_HZ_COMMON */
@@ -10038,7 +10038,7 @@ static int idle_balance(struct rq *this_rq, struct rq_flags *rf) goto out; } -raw_spin_unlock (&this_rq->lock); +raw_spin_unlock (rq_lockp (this_rq));update_blocked_averages (this_cpu);rcu_read_lock ();
@@ -10079,7 +10079,7 @@ static int idle_balance(struct rq *this_rq, struct rq_flags *rf) }rcu_read_unlock (); -raw_spin_lock (&this_rq->lock); +raw_spin_lock (rq_lockp (this_rq));if (curr_cost > this_rq->max_idle_balance_cost) this_rq->max_idle_balance_cost = curr_cost;
@@ -10515,11 +10515,11 @@ void online_fair_sched_group(struct task_group *tg) rq =cpu_rq (i); se = tg->se[i]; -raw_spin_lock_irq (&rq->lock); +raw_spin_lock_irq (rq_lockp (rq));update_rq_clock (rq);attach_entity_cfs_rq (se);sync_throttle (tg, i); -raw_spin_unlock_irq (&rq->lock); +raw_spin_unlock_irq (rq_lockp (rq)); } }
@@ -10542,9 +10542,9 @@ void unregister_fair_sched_group(struct task_group *tg) rq =cpu_rq (cpu); -raw_spin_lock_irqsave (&rq->lock, flags); +raw_spin_lock_irqsave (rq_lockp (rq), flags);list_del_leaf_cfs_rq (tg->cfs_rq[cpu]); -raw_spin_unlock_irqrestore (&rq->lock, flags); +raw_spin_unlock_irqrestore (rq_lockp (rq), flags); } }
kernel/sched/idle.c +2 -2
@@ -390,10 +390,10 @@ pick_next_task_idle(struct rq *rq, struct task_struct *prev, struct rq_flags *rfstatic void dequeue_task_idle (struct rq *rq,struct task_struct *p,int flags) { -raw_spin_unlock_irq (&rq->lock); +raw_spin_unlock_irq (rq_lockp (rq));printk (KERN_ERR "bad: scheduling from the idle thread!\n" );dump_stack (); -raw_spin_lock_irq (&rq->lock); +raw_spin_lock_irq (rq_lockp (rq)); }static void put_prev_task_idle (struct rq *rq,struct task_struct *prev)
kernel/sched/pelt.h +1 -1
@@ -116,7 +116,7 @@ static inline void update_idle_rq_clock_pelt(struct rq *rq)static inline u64 rq_clock_pelt (struct rq *rq) { -lockdep_assert_held (&rq->lock); +lockdep_assert_held (rq_lockp (rq));assert_clock_updated (rq);return rq->clock_pelt - rq->lost_idle_time;
kernel/sched/rt.c +4 -4
@@ -845,7 +845,7 @@ static int do_sched_rt_period_timer(struct rt_bandwidth *rt_b, int overrun)if (skip)continue ; -raw_spin_lock (&rq->lock); +raw_spin_lock (rq_lockp (rq));update_rq_clock (rq);if (rt_rq->rt_time) {
@@ -883,7 +883,7 @@ static int do_sched_rt_period_timer(struct rt_bandwidth *rt_b, int overrun)if (enqueue)sched_rt_rq_enqueue (rt_rq); -raw_spin_unlock (&rq->lock); +raw_spin_unlock (rq_lockp (rq)); }if (!throttled && (!rt_bandwidth_enabled () || rt_b->rt_runtime ==RUNTIME_INF ))
@@ -2034,9 +2034,9 @@ void rto_push_irq_work_func(struct irq_work *work)* When it gets updated, a check is made if a push is possible. */ if (has_pushable_tasks (rq)) { -raw_spin_lock (&rq->lock); +raw_spin_lock (rq_lockp (rq));push_rt_tasks (rq); -raw_spin_unlock (&rq->lock); +raw_spin_unlock (rq_lockp (rq)); }raw_spin_lock (&rd->rto_lock);
kernel/sched/sched.h +54 -52
270 lines not on this page · raw message
kernel/sched/topology.c +2 -2
@@ -442,7 +442,7 @@ void rq_attach_root(struct rq *rq, struct root_domain *rd)struct root_domain *old_rd =NULL ;unsigned long flags; -raw_spin_lock_irqsave (&rq->lock, flags); +raw_spin_lock_irqsave (rq_lockp (rq), flags);if (rq->rd) { old_rd = rq->rd;
@@ -468,7 +468,7 @@ void rq_attach_root(struct rq *rq, struct root_domain *rd)if (cpumask_test_cpu (rq->cpu, cpu_active_mask))set_rq_online (rq); -raw_spin_unlock_irqrestore (&rq->lock, flags); +raw_spin_unlock_irqrestore (rq_lockp (rq), flags);if (old_rd)call_rcu (&old_rd->rcu, free_rootdomain);
Signature · 1 line
2.17.1
RFC PATCH v3 8/16 sched: Rework pick_next_task() slow-path
In reply to Vineeth Remanan PillaiFrom: Peter Zijlstra <[email protected]>
Avoid the RETRY_TASK case in the pick_next_task() slow path.
By doing the put_prev_task() early, we get the rt/deadline pull done, and by testing rq->nr_running we know if we need newidle_balance().
This then gives a stable state to pick a task from.
Since the fast-path is fair only; it means the other classes will always have pick_next_task(.prev=NULL, .rf=NULL) and we can simplify.
--- kernel/sched/core.c | 19 ++++++++++++------- kernel/sched/deadline.c | 30 ++---------------------------- kernel/sched/fair.c | 9 ++++++--- kernel/sched/idle.c | 4 +++- kernel/sched/rt.c | 29 +---------------------------- kernel/sched/sched.h | 13 ++++++++----- kernel/sched/stop_task.c | 3 ++- 7 files changed, 34 insertions(+), 73 deletions(-)
kernel/sched/core.c +12 -7
@@ -3363,7 +3363,7 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf) p = fair_sched_class.pick_next_task (rq, prev, rf);if (unlikely (p ==RETRY_TASK )) - goto again; + goto restart;/* Assumes fair_sched_class->next == idle_sched_class */ if (unlikely (!p))
@@ -3372,14 +3372,19 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)return p; } -again: +restart: +/* +* Ensure that we put DL/RT tasks before the pick loop, such that they +* can PULL higher prio tasks when we lower the RQ 'priority'. +*/ + prev->sched_class->put_prev_task (rq, prev, rf); +if (!rq->nr_running) +newidle_balance (rq, rf); +for_each_class (class) { - p = class->pick_next_task (rq, prev, rf); -if (p) { -if (unlikely (p ==RETRY_TASK )) - goto again; + p = class->pick_next_task (rq,NULL ,NULL ); +if (p)return p; - } }/* The idle class should always have a runnable task: */
kernel/sched/deadline.c +2 -28
@@ -1729,39 +1729,13 @@ pick_next_task_dl(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)struct task_struct *p;struct dl_rq *dl_rq; - dl_rq = &rq->dl; - -if (need_pull_dl_task (rq, prev)) { -/* -* This is OK, because current is on_cpu, which avoids it being -* picked for load-balance and preemption/IRQs are still -* disabled avoiding further scheduler activity on it and we're -* being very careful to re-start the picking loop. -*/ -rq_unpin_lock (rq, rf); -pull_dl_task (rq); -rq_repin_lock (rq, rf); -/* -* pull_dl_task() can drop (and re-acquire) rq->lock; this -* means a stop task can slip in, in which case we need to -* re-start task selection. -*/ -if (rq->stop &&task_on_rq_queued (rq->stop)) -return RETRY_TASK ; - } +WARN_ON_ONCE (prev || rf); -/* -* When prev is DL, we may throttle it in put_prev_task(). -* So, we update time before we check for dl_nr_running. -*/ -if (prev->sched_class == &dl_sched_class) -update_curr_dl (rq); + dl_rq = &rq->dl;if (unlikely (!dl_rq->dl_nr_running))return NULL ; -put_prev_task (rq, prev); - dl_se =pick_next_dl_entity (rq, dl_rq);BUG_ON (!dl_se);
kernel/sched/fair.c +6 -3
@@ -6979,7 +6979,7 @@ pick_next_task_fair(struct rq *rq, struct task_struct *prev, struct rq_flags *rf goto idle;#ifdef CONFIG_FAIR_GROUP_SCHED -if (prev->sched_class != &fair_sched_class) +if (!prev || prev->sched_class != &fair_sched_class) goto simple;/*
@@ -7056,8 +7056,8 @@ pick_next_task_fair(struct rq *rq, struct task_struct *prev, struct rq_flags *rf goto done; simple: #endif - - put_prev_task(rq, prev); + if (prev) + put_prev_task(rq, prev); do { se = pick_next_entity(cfs_rq, NULL);
@@ -7085,6 +7085,9 @@ done: __maybe_unused;return p; idle: +if (!rf) +return NULL ; + new_tasks =newidle_balance (rq, rf);/*
kernel/sched/idle.c +3 -1
@@ -388,7 +388,9 @@ pick_next_task_idle(struct rq *rq, struct task_struct *prev, struct rq_flags *rf {struct task_struct *next = rq->idle; -put_prev_task (rq, prev); +if (prev) +put_prev_task (rq, prev); +set_next_task_idle (rq, next);return next;
kernel/sched/rt.c +1 -28
@@ -1554,38 +1554,11 @@ pick_next_task_rt(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)struct task_struct *p;struct rt_rq *rt_rq = &rq->rt; -if (need_pull_rt_task (rq, prev)) { -/* -* This is OK, because current is on_cpu, which avoids it being -* picked for load-balance and preemption/IRQs are still -* disabled avoiding further scheduler activity on it and we're -* being very careful to re-start the picking loop. -*/ -rq_unpin_lock (rq, rf); -pull_rt_task (rq); -rq_repin_lock (rq, rf); -/* -* pull_rt_task() can drop (and re-acquire) rq->lock; this -* means a dl or stop task can slip in, in which case we need -* to re-start task selection. -*/ -if (unlikely ((rq->stop &&task_on_rq_queued (rq->stop)) || - rq->dl.dl_nr_running)) -return RETRY_TASK ; - } - -/* -* We may dequeue prev's rt_rq in put_prev_task(). -* So, we update time before rt_queued check. -*/ -if (prev->sched_class == &rt_sched_class) -update_curr_rt (rq); +WARN_ON_ONCE (prev || rf);if (!rt_rq->rt_queued)return NULL ; -put_prev_task (rq, prev); - p =_pick_next_task_rt (rq);set_next_task_rt (rq, p);
kernel/sched/sched.h +8 -5
@@ -1665,12 +1665,15 @@ struct sched_class {void (*check_preempt_curr)(struct rq *rq,struct task_struct *p,int flags);/* -* It is the responsibility of the pick_next_task() method that will -* return the next task to call put_prev_task() on the @prev task or -* something equivalent. +* Both @prev and @rf are optional and may be NULL, in which case the +* caller must already have invoked put_prev_task(rq, prev, rf). * -* May return RETRY_TASK when it finds a higher prio class has runnable -* tasks. +* Otherwise it is the responsibility of the pick_next_task() to call +* put_prev_task() on the @prev task or something equivalent, IFF it +* returns a next task. +* +* In that case (@rf != NULL) it may return RETRY_TASK when it finds a +* higher prio class has runnable tasks. */ struct task_struct * (*pick_next_task)(struct rq *rq, struct task_struct *prev,
kernel/sched/stop_task.c +2 -1
@@ -33,10 +33,11 @@ pick_next_task_stop(struct rq *rq, struct task_struct *prev, struct rq_flags *rf {struct task_struct *stop = rq->stop; +WARN_ON_ONCE (prev || rf); +if (!stop || !task_on_rq_queued (stop))return NULL ; -put_prev_task (rq, prev);set_next_task_stop (rq, stop);return stop;
Signature · 1 line
2.17.1
RFC PATCH v3 11/16 sched: Basic tracking of matching tasks
In reply to Vineeth Remanan PillaiFrom: Peter Zijlstra <[email protected]>
Introduce task_struct::core_cookie as an opaque identifier for core scheduling. When enabled; core scheduling will only allow matching task to be on the core; where idle matches everything.
When task_struct::core_cookie is set (and core scheduling is enabled) these tasks are indexed in a second RB-tree, first on cookie value then on scheduling function, such that matching task selection always finds the most elegible match.
NOTE: *shudder* at the overhead...
NOTE: *sigh*, a 3rd copy of the scheduling function; the alternative is per class tracking of cookies and that just duplicates a lot of stuff for no raisin (the 2nd copy lives in the rt-mutex PI code).
---
Changes in v3
-------------
- Refactored priority comparison code
- Fixed a comparison logic issue in sched_core_find
- Aaron Lu
Changes in v2
-------------
- Improves the priority comparison logic between processes in different cpus.
- Peter Zijlstra
- Aaron Lu
--- include/linux/sched.h | 8 ++- kernel/sched/core.c | 146 ++++++++++++++++++++++++++++++++++++++++++ kernel/sched/fair.c | 46 ------------- kernel/sched/sched.h | 55 ++++++++++++++++ 4 files changed, 208 insertions(+), 47 deletions(-)
include/linux/sched.h +7 -1
@@ -636,10 +636,16 @@ struct task_struct {const struct sched_class *sched_class;struct sched_entity se;struct sched_rt_entity rt; +struct sched_dl_entity dl; + +#ifdef CONFIG_SCHED_CORE +struct rb_node core_node; +unsigned long core_cookie; +#endif +#ifdef CONFIG_CGROUP_SCHED struct task_group *sched_task_group;#endif -struct sched_dl_entity dl;#ifdef CONFIG_PREEMPT_NOTIFIERS /* List of struct preempt_notifier: */
kernel/sched/core.c +146 -0
170 lines not on this page · raw message
kernel/sched/fair.c +0 -46
66 lines not on this page · raw message
kernel/sched/sched.h +55 -0
67 lines not on this page · raw message
Signature · 1 line
2.17.1
RFC PATCH v3 10/16 sched: Core-wide rq->lock
In reply to Vineeth Remanan PillaiFrom: Peter Zijlstra <[email protected]>
Introduce the basic infrastructure to have a core wide rq->lock.
---
Changes in v3
-------------
- Fixes a crash during cpu offline/offline with coresched enabled
- Vineeth Pillai
kernel/Kconfig.preempt | 7 ++- kernel/sched/core.c | 113 ++++++++++++++++++++++++++++++++++++++++- kernel/sched/sched.h | 31 +++++++++++ 3 files changed, 148 insertions(+), 3 deletions(-)
kernel/Kconfig.preempt +6 -1
@@ -57,4 +57,9 @@ config PREEMPT endchoice config PREEMPT_COUNT - bool + bool + +config SCHED_CORE + bool + default y + depends on SCHED_SMT
kernel/sched/core.c +111 -2
151 lines not on this page · raw message
kernel/sched/sched.h +31 -0
@@ -952,6 +952,12 @@ struct rq {/* Must be inspected within a rcu lock section */ struct cpuidle_state *idle_state;#endif + +#ifdef CONFIG_SCHED_CORE +/* per rq */ +struct rq *core; +unsigned int core_enabled; +#endif };#ifdef CONFIG_FAIR_GROUP_SCHED
@@ -979,11 +985,36 @@ static inline int cpu_of(struct rq *rq)#endif } +#ifdef CONFIG_SCHED_CORE +DECLARE_STATIC_KEY_FALSE (__sched_core_enabled); + +static inline bool sched_core_enabled (struct rq *rq) +{ +return static_branch_unlikely (&__sched_core_enabled) && rq->core_enabled; +} + +static inline raw_spinlock_t *rq_lockp (struct rq *rq) +{ +if (sched_core_enabled (rq)) +return &rq->core->__lock; + +return &rq->__lock; +} + +#else /* !CONFIG_SCHED_CORE */ + +static inline bool sched_core_enabled (struct rq *rq) +{ +return false; +} +static inline raw_spinlock_t *rq_lockp (struct rq *rq) {return &rq->__lock; } +#endif /* CONFIG_SCHED_CORE */ +#ifdef CONFIG_SCHED_SMT extern void __update_idle_core (struct rq *rq);
Signature · 1 line
2.17.1
RFC PATCH v3 13/16 sched: Add core wide task selection and scheduling.
In reply to Vineeth Remanan PillaiFrom: Peter Zijlstra <[email protected]>
Instead of only selecting a local task, select a task for all SMT siblings for every reschedule on the core (irrespective which logical CPU does the reschedule).
NOTE: there is still potential for siblings rivalry.
NOTE: this is far too complicated; but thus far I've failed to
simplify it further.
---
Changes in v3
-------------
- Fixes the issue of sibling picking an incompatible task.
- Aaron Lu
- Peter Zijlstra
- Vineeth Pillai
- Julien Desfossez
--- kernel/sched/core.c | 271 ++++++++++++++++++++++++++++++++++++++++++- kernel/sched/sched.h | 6 +- 2 files changed, 274 insertions(+), 3 deletions(-)
kernel/sched/core.c +269 -2
@@ -3556,7 +3556,7 @@ static inline void schedule_debug(struct task_struct *prev)* Pick up the highest-prio task: */ static inline struct task_struct * -pick_next_task (struct rq *rq,struct task_struct *prev,struct rq_flags *rf) +__pick_next_task (struct rq *rq,struct task_struct *prev,struct rq_flags *rf) {const struct sched_class *class;struct task_struct *p;
@@ -3601,6 +3601,268 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)BUG (); } +#ifdef CONFIG_SCHED_CORE + +static inline bool cookie_equals (struct task_struct *a,unsigned long cookie) +{ +return is_idle_task (a) || (a->core_cookie == cookie); +} + +static inline bool cookie_match (struct task_struct *a,struct task_struct *b) +{ +if (is_idle_task (a) ||is_idle_task (b)) +return true; + +return a->core_cookie == b->core_cookie; +} + +// XXX fairness/fwd progress conditions +/* +* Returns +* - NULL if there is no runnable task for this class. +* - the highest priority task for this runqueue if it matches +* rq->core->core_cookie or its priority is greater than max. +* - Else returns idle_task. +*/ +static struct task_struct * +pick_task (struct rq *rq,const struct sched_class *class,struct task_struct *max) +{ +struct task_struct *class_pick, *cookie_pick; +unsigned long cookie = rq->core->core_cookie; + + class_pick = class->pick_task (rq); +if (!class_pick) +return NULL ; + +if (!cookie) { +/* +* If class_pick is tagged, return it only if it has +* higher priority than max. +*/ +if (max && class_pick->core_cookie && +prio_less (class_pick, max)) +return idle_sched_class.pick_task (rq); + +return class_pick; + } + +/* +* If class_pick is idle or matches cookie, return early. +*/ +if (cookie_equals (class_pick, cookie)) +return class_pick; + + cookie_pick =sched_core_find (rq, cookie); + +/* +* If class > max && class > cookie, it is the highest priority task on +* the core (so far) and it must be selected, otherwise we must go with +* the cookie pick in order to satisfy the constraint. +*/ +if (prio_less (cookie_pick, class_pick) && + (!max ||prio_less (max, class_pick))) +return class_pick; + +return cookie_pick; +} + +static struct task_struct * +pick_next_task (struct rq *rq,struct task_struct *prev,struct rq_flags *rf) +{ +struct task_struct *next, *max =NULL ; +const struct sched_class *class; +const struct cpumask *smt_mask; +int i, j, cpu; +bool need_sync = false; + +if (!sched_core_enabled (rq)) +return __pick_next_task (rq, prev, rf); + +/* +* If there were no {en,de}queues since we picked (IOW, the task +* pointers are all still valid), and we haven't scheduled the last +* pick yet, do so now. +*/ +if (rq->core->core_pick_seq == rq->core->core_task_seq && + rq->core->core_pick_seq != rq->core_sched_seq) { +WRITE_ONCE (rq->core_sched_seq, rq->core->core_pick_seq); + + next = rq->core_pick; +if (next != prev) { +put_prev_task (rq, prev); +set_next_task (rq, next); + } +return next; + } + + prev->sched_class->put_prev_task (rq, prev, rf); +if (!rq->nr_running) +newidle_balance (rq, rf); + + cpu =cpu_of (rq); + smt_mask =cpu_smt_mask (cpu); + +/* +* core->core_task_seq, core->core_pick_seq, rq->core_sched_seq +* +* @task_seq guards the task state ({en,de}queues) +* @pick_seq is the @task_seq we did a selection on +* @sched_seq is the @pick_seq we scheduled +* +* However, preemptions can cause multiple picks on the same task set. +* 'Fix' this by also increasing @task_seq for every pick. +*/ + rq->core->core_task_seq++; + need_sync = !!rq->core->core_cookie; + +/* reset state */ + rq->core->core_cookie =0UL ; +for_each_cpu (i, smt_mask) { +struct rq *rq_i =cpu_rq (i); + + rq_i->core_pick =NULL ; + +if (rq_i->core_forceidle) { + need_sync = true; + rq_i->core_forceidle = false; + } + +if (i != cpu) +update_rq_clock (rq_i); + } + +/* +* Try and select tasks for each sibling in decending sched_class +* order. +*/ +for_each_class (class) { +again: +for_each_cpu_wrap (i, smt_mask, cpu) { +struct rq *rq_i =cpu_rq (i); +struct task_struct *p; + +if (cpu_is_offline (i)) +continue ; + +if (rq_i->core_pick) +continue ; + +/* +* If this sibling doesn't yet have a suitable task to +* run; ask for the most elegible task, given the +* highest priority task already selected for this +* core. +*/ + p =pick_task (rq_i, class, max); +if (!p) { +/* +* If there weren't no cookies; we don't need +* to bother with the other siblings. +*/ +if (i == cpu && !need_sync) + goto next_class; + +continue ; + } + +/* +* Optimize the 'normal' case where there aren't any +* cookies and we don't need to sync up. +*/ +if (i == cpu && !need_sync && !p->core_cookie) { + next = p; + goto done; + } + + rq_i->core_pick = p; + +/* +* If this new candidate is of higher priority than the +* previous; and they're incompatible; we need to wipe +* the slate and start over. pick_task makes sure that +* p's priority is more than max if it doesn't match +* max's cookie. +* +* NOTE: this is a linear max-filter and is thus bounded +* in execution time. +*/ +if (!max || !cookie_match (max, p)) { +struct task_struct *old_max = max; + + rq->core->core_cookie = p->core_cookie; + max = p; + +if (old_max) { +for_each_cpu (j, smt_mask) { +if (j == i) +continue ; + +cpu_rq (j)->core_pick =NULL ; + } + goto again; + }else { +/* +* Once we select a task for a cpu, we +* should not be doing an unconstrained +* pick because it might starve a task +* on a forced idle cpu. +*/ + need_sync = true; + } + + } + } +next_class:; + } + + rq->core->core_pick_seq = rq->core->core_task_seq; + next = rq->core_pick; + rq->core_sched_seq = rq->core->core_pick_seq; + +/* +* Reschedule siblings +* +* NOTE: L1TF -- at this point we're no longer running the old task and +* sending an IPI (below) ensures the sibling will no longer be running +* their task. This ensures there is no inter-sibling overlap between +* non-matching user state. +*/ +for_each_cpu (i, smt_mask) { +struct rq *rq_i =cpu_rq (i); + +if (cpu_is_offline (i)) +continue ; + +WARN_ON_ONCE (!rq_i->core_pick); + +if (is_idle_task (rq_i->core_pick) && rq_i->nr_running) + rq->core_forceidle = true; + +if (i == cpu) +continue ; + +if (rq_i->curr != rq_i->core_pick) +resched_curr (rq_i); + +/* Did we break L1TF mitigation requirements? */ +WARN_ON_ONCE (!cookie_match (next, rq_i->core_pick)); + } + +done: +set_next_task (rq, next); +return next; +} + +#else /* !CONFIG_SCHED_CORE */ + +static struct task_struct * +pick_next_task (struct rq *rq,struct task_struct *prev,struct rq_flags *rf) +{ +return __pick_next_task (rq, prev, rf); +} + +#endif /* CONFIG_SCHED_CORE */ +/* * __schedule() is the main scheduler function. *
@@ -5870,7 +6132,7 @@ static void migrate_tasks(struct rq *dead_rq, struct rq_flags *rf)/* * pick_next_task() assumes pinned rq->lock: */ - next =pick_next_task (rq, &fake_task, rf); + next =__pick_next_task (rq, &fake_task, rf);BUG_ON (!next);put_prev_task (rq, next);
@@ -6344,7 +6606,12 @@ void __init sched_init(void)#ifdef CONFIG_SCHED_CORE rq->core =NULL ; + rq->core_pick =NULL ; rq->core_enabled =0 ; + rq->core_tree =RB_ROOT ; + rq->core_forceidle = false; + + rq->core_cookie =0UL ;#endif }
kernel/sched/sched.h +5 -1
@@ -960,11 +960,16 @@ struct rq {#ifdef CONFIG_SCHED_CORE /* per rq */ struct rq *core; +struct task_struct *core_pick;unsigned int core_enabled; +unsigned int core_sched_seq;struct rb_root core_tree; +bool core_forceidle;/* shared state */ unsigned int core_task_seq; +unsigned int core_pick_seq; +unsigned long core_cookie;#endif };
@@ -1821,7 +1826,6 @@ static inline void put_prev_task(struct rq *rq, struct task_struct *prev)static inline void set_next_task (struct rq *rq,struct task_struct *next) { -WARN_ON_ONCE (rq->curr != next); next->sched_class->set_next_task (rq, next); }
Signature · 1 line
2.17.1
RFC PATCH v3 15/16 sched: Trivial forced-newidle balancer
In reply to Vineeth Remanan PillaiFrom: Peter Zijlstra <[email protected]>
When a sibling is forced-idle to match the core-cookie; search for matching tasks to fill the core.
--- include/linux/sched.h | 1 + kernel/sched/core.c | 131 +++++++++++++++++++++++++++++++++++++++++- kernel/sched/idle.c | 1 + kernel/sched/sched.h | 6 ++ 4 files changed, 138 insertions(+), 1 deletion(-)
include/linux/sched.h +1 -0
@@ -641,6 +641,7 @@ struct task_struct {#ifdef CONFIG_SCHED_CORE struct rb_node core_node;unsigned long core_cookie; +unsigned int core_occupation;#endif #ifdef CONFIG_CGROUP_SCHED
kernel/sched/core.c +130 -1
167 lines not on this page · raw message
kernel/sched/idle.c +1 -0
@@ -387,6 +387,7 @@ static void set_next_task_idle(struct rq *rq, struct task_struct *next) {update_idle_core (rq);schedstat_inc (rq->sched_goidle); +queue_core_balance (rq); } static struct task_struct *
kernel/sched/sched.h +6 -0
@@ -1014,6 +1014,8 @@ static inline raw_spinlock_t *rq_lockp(struct rq *rq) return &rq->__lock; } +extern void queue_core_balance (struct rq *rq); +#else /* !CONFIG_SCHED_CORE */ static inline bool sched_core_enabled (struct rq *rq)
@@ -1026,6 +1028,10 @@ static inline raw_spinlock_t *rq_lockp(struct rq *rq) return &rq->__lock; } +static inline void queue_core_balance (struct rq *rq) +{ +} +#endif /* CONFIG_SCHED_CORE */ #ifdef CONFIG_SCHED_SMT
Signature · 1 line
2.17.1
RFC PATCH v3 5/16 sched: Add task_struct pointer to sched_class::set_curr_task
In reply to Vineeth Remanan PillaiFrom: Peter Zijlstra <[email protected]>
In preparation of further separating pick_next_task() and set_curr_task() we have to pass the actual task into it, while there, rename the thing to better pair with put_prev_task().
--- kernel/sched/core.c | 12 ++++++------ kernel/sched/deadline.c | 7 +------ kernel/sched/fair.c | 17 ++++++++++++++--- kernel/sched/idle.c | 27 +++++++++++++++------------ kernel/sched/rt.c | 7 +------ kernel/sched/sched.h | 8 +++++--- kernel/sched/stop_task.c | 17 +++++++---------- 7 files changed, 49 insertions(+), 46 deletions(-)
kernel/sched/core.c +6 -6
@@ -1081,7 +1081,7 @@ void do_set_cpus_allowed(struct task_struct *p, const struct cpumask *new_mask)if (queued)enqueue_task (rq, p,ENQUEUE_RESTORE |ENQUEUE_NOCLOCK );if (running) -set_curr_task (rq, p); +set_next_task (rq, p); }/*
@@ -3890,7 +3890,7 @@ void rt_mutex_setprio(struct task_struct *p, struct task_struct *pi_task)if (queued)enqueue_task (rq, p, queue_flag);if (running) -set_curr_task (rq, p); +set_next_task (rq, p);check_class_changed (rq, p, prev_class, oldprio); out_unlock:
@@ -3957,7 +3957,7 @@ void set_user_nice(struct task_struct *p, long nice)resched_curr (rq); }if (running) -set_curr_task (rq, p); +set_next_task (rq, p); out_unlock:task_rq_unlock (rq, p, &rf); }
@@ -4382,7 +4382,7 @@ static int __sched_setscheduler(struct task_struct *p,enqueue_task (rq, p, queue_flags); }if (running) -set_curr_task (rq, p); +set_next_task (rq, p);check_class_changed (rq, p, prev_class, oldprio);
@@ -5555,7 +5555,7 @@ void sched_setnuma(struct task_struct *p, int nid)if (queued)enqueue_task (rq, p,ENQUEUE_RESTORE |ENQUEUE_NOCLOCK );if (running) -set_curr_task (rq, p); +set_next_task (rq, p); task_rq_unlock(rq, p, &rf); }#endif /* CONFIG_NUMA_BALANCING */
@@ -6438,7 +6438,7 @@ void sched_move_task(struct task_struct *tsk)if (queued)enqueue_task (rq, tsk, queue_flags);if (running) -set_curr_task (rq, tsk); +set_next_task (rq, tsk);task_rq_unlock (rq, tsk, &rf); }
kernel/sched/deadline.c +1 -6
@@ -1812,11 +1812,6 @@ static void task_fork_dl(struct task_struct *p)*/ } -static void set_curr_task_dl (struct rq *rq) -{ -set_next_task_dl (rq, rq->curr); -} -#ifdef CONFIG_SMP /* Only try algorithms three times */
@@ -2404,6 +2399,7 @@ const struct sched_class dl_sched_class = { .pick_next_task = pick_next_task_dl, .put_prev_task = put_prev_task_dl, + .set_next_task = set_next_task_dl, #ifdef CONFIG_SMP .select_task_rq = select_task_rq_dl,
@@ -2414,7 +2410,6 @@ const struct sched_class dl_sched_class = { .task_woken = task_woken_dl, #endif - .set_curr_task = set_curr_task_dl, .task_tick = task_tick_dl, .task_fork = task_fork_dl,
kernel/sched/fair.c +14 -3
@@ -10388,9 +10388,19 @@ static void switched_to_fair(struct rq *rq, struct task_struct *p)* This routine is mostly called to set cfs_rq->curr field when a task * migrates between groups/classes. */ -static void set_curr_task_fair (struct rq *rq) +static void set_next_task_fair (struct rq *rq,struct task_struct *p) { -struct sched_entity *se = &rq->curr->se; +struct sched_entity *se = &p->se; + +#ifdef CONFIG_SMP +if (task_on_rq_queued (p)) { +/* +* Move the next running task to the front of the list, so our +* cfs_tasks list becomes MRU one. +*/ +list_move (&se->group_node, &rq->cfs_tasks); + } +#endif for_each_sched_entity (se) {struct cfs_rq *cfs_rq =cfs_rq_of (se);
@@ -10661,7 +10671,9 @@ const struct sched_class fair_sched_class = { .check_preempt_curr = check_preempt_wakeup, .pick_next_task = pick_next_task_fair, + .put_prev_task = put_prev_task_fair, + .set_next_task = set_next_task_fair, #ifdef CONFIG_SMP .select_task_rq = select_task_rq_fair,
@@ -10674,7 +10686,6 @@ const struct sched_class fair_sched_class = { .set_cpus_allowed = set_cpus_allowed_common, #endif - .set_curr_task = set_curr_task_fair, .task_tick = task_tick_fair, .task_fork = task_fork_fair,
kernel/sched/idle.c +15 -12
@@ -373,14 +373,25 @@ static void check_preempt_curr_idle(struct rq *rq, struct task_struct *p, int flresched_curr (rq); } +static void put_prev_task_idle (struct rq *rq,struct task_struct *prev) +{ +} + +static void set_next_task_idle (struct rq *rq,struct task_struct *next) +{ +update_idle_core (rq); +schedstat_inc (rq->sched_goidle); +} +static struct task_struct *pick_next_task_idle (struct rq *rq,struct task_struct *prev,struct rq_flags *rf) { +struct task_struct *next = rq->idle; +put_prev_task (rq, prev); -update_idle_core (rq); -schedstat_inc (rq->sched_goidle); +set_next_task_idle (rq, next); -return rq->idle; +return next; }/*
@@ -396,10 +407,6 @@ dequeue_task_idle(struct rq *rq, struct task_struct *p, int flags)raw_spin_lock_irq (rq_lockp (rq)); } -static void put_prev_task_idle (struct rq *rq,struct task_struct *prev) -{ -} -/* * scheduler tick hitting a task of our scheduling class. *
@@ -412,10 +419,6 @@ static void task_tick_idle(struct rq *rq, struct task_struct *curr, int queued) { } -static void set_curr_task_idle (struct rq *rq) -{ -} -static void switched_to_idle (struct rq *rq,struct task_struct *p) {BUG ();
@@ -450,13 +453,13 @@ const struct sched_class idle_sched_class = { .pick_next_task = pick_next_task_idle, .put_prev_task = put_prev_task_idle, + .set_next_task = set_next_task_idle, #ifdef CONFIG_SMP .select_task_rq = select_task_rq_idle, .set_cpus_allowed = set_cpus_allowed_common, #endif - .set_curr_task = set_curr_task_idle, .task_tick = task_tick_idle, .get_rr_interval = get_rr_interval_idle,
kernel/sched/rt.c +1 -6
@@ -2355,11 +2355,6 @@ static void task_tick_rt(struct rq *rq, struct task_struct *p, int queued) } } -static void set_curr_task_rt (struct rq *rq) -{ -set_next_task_rt (rq, rq->curr); -} -static unsigned int get_rr_interval_rt (struct rq *rq,struct task_struct *task) {/*
@@ -2381,6 +2376,7 @@ const struct sched_class rt_sched_class = { .pick_next_task = pick_next_task_rt, .put_prev_task = put_prev_task_rt, + .set_next_task = set_next_task_rt, #ifdef CONFIG_SMP .select_task_rq = select_task_rq_rt,
@@ -2392,7 +2388,6 @@ const struct sched_class rt_sched_class = { .switched_from = switched_from_rt, #endif - .set_curr_task = set_curr_task_rt, .task_tick = task_tick_rt, .get_rr_interval = get_rr_interval_rt,
kernel/sched/sched.h +5 -3
@@ -1672,6 +1672,7 @@ struct sched_class {struct task_struct *prev, struct rq_flags *rf);void (*put_prev_task)(struct rq *rq,struct task_struct *p); +void (*set_next_task)(struct rq *rq,struct task_struct *p);#ifdef CONFIG_SMP int (*select_task_rq)(struct task_struct *p,int task_cpu,int sd_flag,int flags);
@@ -1686,7 +1687,6 @@ struct sched_class {void (*rq_offline)(struct rq *rq);#endif -void (*set_curr_task)(struct rq *rq);void (*task_tick)(struct rq *rq,struct task_struct *p,int queued);void (*task_fork)(struct task_struct *p);void (*task_dead)(struct task_struct *p);
@@ -1716,12 +1716,14 @@ struct sched_class {static inline void put_prev_task (struct rq *rq,struct task_struct *prev) { +WARN_ON_ONCE (rq->curr != prev); prev->sched_class->put_prev_task (rq, prev); } -static inline void set_curr_task (struct rq *rq,struct task_struct *curr) +static inline void set_next_task (struct rq *rq,struct task_struct *next) { - curr->sched_class->set_curr_task (rq); +WARN_ON_ONCE (rq->curr != next); + next->sched_class->set_next_task (rq, next); }#ifdef CONFIG_SMP
kernel/sched/stop_task.c +7 -10
@@ -23,6 +23,11 @@ check_preempt_curr_stop(struct rq *rq, struct task_struct *p, int flags)/* we're never preempted */ } +static void set_next_task_stop (struct rq *rq,struct task_struct *stop) +{ + stop->se.exec_start =rq_clock_task (rq); +} +static struct task_struct *pick_next_task_stop (struct rq *rq,struct task_struct *prev,struct rq_flags *rf) {
@@ -32,8 +37,7 @@ pick_next_task_stop(struct rq *rq, struct task_struct *prev, struct rq_flags *rfreturn NULL ;put_prev_task (rq, prev); - - stop->se.exec_start =rq_clock_task (rq); +set_next_task_stop (rq, stop);return stop; }
@@ -86,13 +90,6 @@ static void task_tick_stop(struct rq *rq, struct task_struct *curr, int queued) { } -static void set_curr_task_stop (struct rq *rq) -{ -struct task_struct *stop = rq->stop; - - stop->se.exec_start =rq_clock_task (rq); -} -static void switched_to_stop (struct rq *rq,struct task_struct *p) {BUG ();/* its impossible to change to this class */
@@ -128,13 +125,13 @@ const struct sched_class stop_sched_class = { .pick_next_task = pick_next_task_stop, .put_prev_task = put_prev_task_stop, + .set_next_task = set_next_task_stop, #ifdef CONFIG_SMP .select_task_rq = select_task_rq_stop, .set_cpus_allowed = set_cpus_allowed_common, #endif - .set_curr_task = set_curr_task_stop, .task_tick = task_tick_stop, .get_rr_interval = get_rr_interval_stop,
Signature · 1 line
2.17.1
RFC PATCH v3 16/16 sched: Debug bits...
In reply to Vineeth Remanan PillaiFrom: Peter Zijlstra <[email protected]>
--- kernel/sched/core.c | 44 ++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 42 insertions(+), 2 deletions(-)
kernel/sched/core.c +42 -2
@@ -92,6 +92,10 @@ static inline bool prio_less(struct task_struct *a, struct task_struct *b)int pa =__task_prio (a), pb =__task_prio (b); +trace_printk ("(%s/%d;%d,%Lu,%Lu) ?< (%s/%d;%d,%Lu,%Lu)\n" , + a->comm, a->pid, pa, a->se.vruntime, a->dl.deadline, + b->comm, b->pid, pb, b->se.vruntime, b->dl.deadline); +if (-pa < -pb)return true;
@@ -246,6 +250,8 @@ static void __sched_core_enable(void)static_branch_enable (&__sched_core_enabled);stop_machine (__sched_core_stopper, (void *)true,NULL ); + +printk ("core sched enabled\n" ); }static void __sched_core_disable (void )
@@ -254,6 +260,8 @@ static void __sched_core_disable(void)stop_machine (__sched_core_stopper, (void *)false,NULL );static_branch_disable (&__sched_core_enabled); + +printk ("core sched disabled\n" ); }void sched_core_get (void )
@@ -3707,6 +3715,14 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)put_prev_task (rq, prev);set_next_task (rq, next); } + +trace_printk ("pick pre selected (%u %u %u): %s/%d %lx\n" , + rq->core->core_task_seq, + rq->core->core_pick_seq, + rq->core_sched_seq, + next->comm, next->pid, + next->core_cookie); +return next; }
@@ -3786,6 +3802,9 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)*/ if (i == cpu && !need_sync && !p->core_cookie) { next = p; +trace_printk ("unconstrained pick: %s/%d %lx\n" , + next->comm, next->pid, next->core_cookie); + goto done; }
@@ -3794,6 +3813,9 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf) rq_i->core_pick = p; +trace_printk ("cpu(%d): selected: %s/%d %lx\n" , + i, p->comm, p->pid, p->core_cookie); +/* * If this new candidate is of higher priority than the * previous; and they're incompatible; we need to wipe
@@ -3810,6 +3832,8 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf) rq->core->core_cookie = p->core_cookie; max = p; +trace_printk ("max: %s/%d %lx\n" , max->comm, max->pid, max->core_cookie); +if (old_max) {for_each_cpu (j, smt_mask) {if (j == i)
@@ -3837,6 +3861,7 @@ next_class:; rq->core->core_pick_seq = rq->core->core_task_seq; next = rq->core_pick; rq->core_sched_seq = rq->core->core_pick_seq; +trace_printk ("picked: %s/%d %lx\n" , next->comm, next->pid, next->core_cookie);/* * Reschedule siblings
@@ -3862,11 +3887,20 @@ next_class:;if (i == cpu)continue ; -if (rq_i->curr != rq_i->core_pick) +if (rq_i->curr != rq_i->core_pick) { +trace_printk ("IPI(%d)\n" , i);resched_curr (rq_i); + }/* Did we break L1TF mitigation requirements? */ -WARN_ON_ONCE (!cookie_match (next, rq_i->core_pick)); +if (unlikely (!cookie_match (next, rq_i->core_pick))) { +trace_printk ("[%d]: cookie mismatch. %s/%d/0x%lx/0x%lx\n" , + rq_i->cpu, rq_i->core_pick->comm, + rq_i->core_pick->pid, + rq_i->core_pick->core_cookie, + rq_i->core->core_cookie); +WARN_ON_ONCE (1 ); + } } done:
@@ -3905,6 +3939,10 @@ static bool try_steal_cookie(int this, int that)if (p->core_occupation > dst->idle->core_occupation) goto next; +trace_printk ("core fill: %s/%d (%d->%d) %d %d %lx\n" , + p->comm, p->pid, that, this, + p->core_occupation, dst->idle->core_occupation, cookie); + p->on_rq =TASK_ON_RQ_MIGRATING ;deactivate_task (src, p,0 );set_task_cpu (p, this);
@@ -6501,6 +6539,8 @@ int sched_cpu_starting(unsigned int cpu)WARN_ON_ONCE (rq->core && rq->core != core_rq); rq->core = core_rq; } + +printk ("core: %d -> %d\n" , cpu,cpu_of (core_rq));#endif /* CONFIG_SCHED_CORE */ sched_rq_cpu_starting (cpu);
Signature · 1 line
2.17.1
RFC PATCH v3 12/16 sched: A quick and dirty cgroup tagging interface
In reply to Vineeth Remanan PillaiFrom: Peter Zijlstra <[email protected]>
Marks all tasks in a cgroup as matching for core-scheduling.
---
Changes in v3
-------------
- Fixes the refcount management when deleting a tagged cgroup.
- Julien Desfossez
--- kernel/sched/core.c | 78 ++++++++++++++++++++++++++++++++++++++++++++ kernel/sched/sched.h | 4 +++ 2 files changed, 82 insertions(+)
kernel/sched/core.c +78 -0
@@ -6658,6 +6658,15 @@ static void sched_change_group(struct task_struct *tsk, int type) tg =container_of (task_css_check (tsk, cpu_cgrp_id, true), struct task_group, css); tg =autogroup_task_group (tsk, tg); + +#ifdef CONFIG_SCHED_CORE +if ((unsigned long )tsk->sched_task_group == tsk->core_cookie) + tsk->core_cookie =0UL ; + +if (tg->tagged/* && !tsk->core_cookie ? */ ) + tsk->core_cookie = (unsigned long )tg; +#endif + tsk->sched_task_group = tg;#ifdef CONFIG_FAIR_GROUP_SCHED
@@ -6737,6 +6746,18 @@ static int cpu_cgroup_css_online(struct cgroup_subsys_state *css)return 0 ; } +static void cpu_cgroup_css_offline (struct cgroup_subsys_state *css) +{ +#ifdef CONFIG_SCHED_CORE +struct task_group *tg =css_tg (css); + +if (tg->tagged) { +sched_core_put (); + tg->tagged =0 ; + } +#endif +} +static void cpu_cgroup_css_released (struct cgroup_subsys_state *css) {struct task_group *tg =css_tg (css);
@@ -7117,6 +7138,46 @@ static u64 cpu_rt_period_read_uint(struct cgroup_subsys_state *css, }#endif /* CONFIG_RT_GROUP_SCHED */ +#ifdef CONFIG_SCHED_CORE +static u64 cpu_core_tag_read_u64 (struct cgroup_subsys_state *css,struct cftype *cft) +{ +struct task_group *tg =css_tg (css); + +return !!tg->tagged; +} + +static int cpu_core_tag_write_u64 (struct cgroup_subsys_state *css,struct cftype *cft,u64 val) +{ +struct task_group *tg =css_tg (css); +struct css_task_iter it; +struct task_struct *p; + +if (val >1 ) +return -ERANGE ; + +if (!static_branch_likely (&sched_smt_present)) +return -EINVAL ; + +if (tg->tagged == !!val) +return 0 ; + + tg->tagged = !!val; + +if (!!val) +sched_core_get (); + +css_task_iter_start (css,0 , &it); +while ((p =css_task_iter_next (&it))) + p->core_cookie = !!val ? (unsigned long )tg :0UL ; +css_task_iter_end (&it); + +if (!val) +sched_core_put (); + +return 0 ; +} +#endif + static struct cftype cpu_legacy_files[] = {#ifdef CONFIG_FAIR_GROUP_SCHED {
@@ -7152,6 +7213,14 @@ static struct cftype cpu_legacy_files[] = { .read_u64 = cpu_rt_period_read_uint, .write_u64 = cpu_rt_period_write_uint, }, +#endif +#ifdef CONFIG_SCHED_CORE + { + .name = "tag", + .flags = CFTYPE_NOT_ON_ROOT, + .read_u64 = cpu_core_tag_read_u64, + .write_u64 = cpu_core_tag_write_u64, + }, #endif { }/* Terminate */ };
@@ -7319,6 +7388,14 @@ static struct cftype cpu_files[] = { .seq_show = cpu_max_show, .write = cpu_max_write, }, +#endif +#ifdef CONFIG_SCHED_CORE + { + .name = "tag", + .flags = CFTYPE_NOT_ON_ROOT, + .read_u64 = cpu_core_tag_read_u64, + .write_u64 = cpu_core_tag_write_u64, + }, #endif { }/* terminate */ };
@@ -7326,6 +7403,7 @@ static struct cftype cpu_files[] = {struct cgroup_subsys cpu_cgrp_subsys = { .css_alloc = cpu_cgroup_css_alloc, .css_online = cpu_cgroup_css_online, + .css_offline = cpu_cgroup_css_offline, .css_released = cpu_cgroup_css_released, .css_free = cpu_cgroup_css_free, .css_extra_stat_show = cpu_extra_stat_show,
kernel/sched/sched.h +4 -0
@@ -363,6 +363,10 @@ struct cfs_bandwidth {struct task_group {struct cgroup_subsys_state css; +#ifdef CONFIG_SCHED_CORE +int tagged; +#endif +#ifdef CONFIG_FAIR_GROUP_SCHED /* schedulable entities of this group on each CPU */ struct sched_entity **se;
Signature · 1 line
2.17.1
RFC PATCH v3 14/16 sched/fair: Add a few assertions
In reply to Vineeth Remanan PillaiFrom: Peter Zijlstra <[email protected]>
--- kernel/sched/fair.c | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-)
kernel/sched/fair.c +10 -2
@@ -6192,6 +6192,11 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)struct sched_domain *sd;int i, recent_used_cpu; +/* +* per-cpu select_idle_mask usage +*/ +lockdep_assert_irqs_disabled (); +if (available_idle_cpu (target))return target;
@@ -6619,8 +6624,6 @@ static int find_energy_efficient_cpu(struct task_struct *p, int prev_cpu)* certain conditions an idle sibling CPU if the domain has SD_WAKE_AFFINE set. * * Returns the target CPU number. -* -* preempt must be disabled. */ static int select_task_rq_fair (struct task_struct *p,int prev_cpu,int sd_flag,int wake_flags)
@@ -6631,6 +6634,11 @@ select_task_rq_fair(struct task_struct *p, int prev_cpu, int sd_flag, int wake_fint want_affine =0 ;int sync = (wake_flags &WF_SYNC ) && !(current->flags &PF_EXITING ); +/* +* required for stable ->cpus_allowed +*/ +lockdep_assert_held (&p->pi_lock); +if (sd_flag &SD_BALANCE_WAKE ) {record_wakee (p);
Signature · 1 line
2.17.1
RFC PATCH v3 9/16 sched: Introduce sched_class::pick_task()
In reply to Vineeth Remanan PillaiFrom: Peter Zijlstra <[email protected]>
Because sched_class::pick_next_task() also implies sched_class::set_next_task() (and possibly put_prev_task() and newidle_balance) it is not state invariant. This makes it unsuitable for remote task selection.
---
Chnages in v3
-------------
- Minor refactor to remove redundant NULL checks
Changes in v2
-------------
- Fixes a NULL pointer dereference crash
- Subhra Mazumdar
- Tim Chen
--- kernel/sched/deadline.c | 21 ++++++++++++++++----- kernel/sched/fair.c | 36 +++++++++++++++++++++++++++++++++--- kernel/sched/idle.c | 10 +++++++++- kernel/sched/rt.c | 21 ++++++++++++++++----- kernel/sched/sched.h | 2 ++ kernel/sched/stop_task.c | 21 ++++++++++++++++----- 6 files changed, 92 insertions(+), 19 deletions(-)
kernel/sched/deadline.c +16 -5
@@ -1722,15 +1722,12 @@ static struct sched_dl_entity *pick_next_dl_entity(struct rq *rq, return rb_entry(left, struct sched_dl_entity, rb_node); } -static struct task_struct * -pick_next_task_dl (struct rq *rq,struct task_struct *prev,struct rq_flags *rf) +static struct task_struct *pick_task_dl (struct rq *rq) {struct sched_dl_entity *dl_se;struct task_struct *p;struct dl_rq *dl_rq; -WARN_ON_ONCE (prev || rf); - dl_rq = &rq->dl;if (unlikely (!dl_rq->dl_nr_running))
@@ -1741,7 +1738,19 @@ pick_next_task_dl(struct rq *rq, struct task_struct *prev, struct rq_flags *rf) p =dl_task_of (dl_se); -set_next_task_dl (rq, p); +return p; +} + +static struct task_struct * +pick_next_task_dl (struct rq *rq,struct task_struct *prev,struct rq_flags *rf) +{ +struct task_struct *p; + +WARN_ON_ONCE (prev || rf); + + p =pick_task_dl (rq); +if (p) +set_next_task_dl (rq, p);return p; }
@@ -2388,6 +2397,8 @@ const struct sched_class dl_sched_class = { .set_next_task = set_next_task_dl, #ifdef CONFIG_SMP + .pick_task = pick_task_dl, + .select_task_rq = select_task_rq_dl, .migrate_task_rq = migrate_task_rq_dl, .set_cpus_allowed = set_cpus_allowed_dl,
kernel/sched/fair.c +33 -3
@@ -4136,7 +4136,7 @@ pick_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *curr)* Avoid running the skip buddy, if running something else can * be done without getting too unfair. */ -if (cfs_rq->skip == se) { +if (cfs_rq->skip && cfs_rq->skip == se) {struct sched_entity *second;if (se == curr) {
@@ -4154,13 +4154,13 @@ pick_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *curr)/* * Prefer last buddy, try to return the CPU to a preempted task. */ -if (cfs_rq->last &&wakeup_preempt_entity (cfs_rq->last, left) <1 ) +if (left && cfs_rq->last &&wakeup_preempt_entity (cfs_rq->last, left) <1 ) se = cfs_rq->last;/* * Someone really wants this to run. If it's not unfair, run it. */ -if (cfs_rq->next &&wakeup_preempt_entity (cfs_rq->next, left) <1 ) +if (left && cfs_rq->next &&wakeup_preempt_entity (cfs_rq->next, left) <1 ) se = cfs_rq->next;clear_buddies (cfs_rq, se);
@@ -6966,6 +6966,34 @@ static void check_preempt_wakeup(struct rq *rq, struct task_struct *p, int wake_set_last_buddy (se); } +static struct task_struct * +pick_task_fair (struct rq *rq) +{ +struct cfs_rq *cfs_rq = &rq->cfs; +struct sched_entity *se; + +if (!cfs_rq->nr_running) +return NULL ; + +do { +struct sched_entity *curr = cfs_rq->curr; + + se =pick_next_entity (cfs_rq,NULL ); + +if (curr) { +if (se && curr->on_rq) +update_curr (cfs_rq); + +if (!se ||entity_before (curr, se)) + se = curr; + } + + cfs_rq =group_cfs_rq (se); + }while (cfs_rq); + +return task_of (se); +} +static struct task_struct *pick_next_task_fair (struct rq *rq,struct task_struct *prev,struct rq_flags *rf) {
@@ -10677,6 +10705,8 @@ const struct sched_class fair_sched_class = { .set_next_task = set_next_task_fair, #ifdef CONFIG_SMP + .pick_task = pick_task_fair, + .select_task_rq = select_task_rq_fair, .migrate_task_rq = migrate_task_rq_fair,
kernel/sched/idle.c +9 -1
@@ -373,6 +373,12 @@ static void check_preempt_curr_idle(struct rq *rq, struct task_struct *p, int flresched_curr (rq); } +static struct task_struct * +pick_task_idle (struct rq *rq) +{ +return rq->idle; +} +static void put_prev_task_idle (struct rq *rq,struct task_struct *prev,struct rq_flags *rf) { }
@@ -386,11 +392,12 @@ static void set_next_task_idle(struct rq *rq, struct task_struct *next)static struct task_struct *pick_next_task_idle (struct rq *rq,struct task_struct *prev,struct rq_flags *rf) { -struct task_struct *next = rq->idle; +struct task_struct *next;if (prev)put_prev_task (rq, prev); + next =pick_task_idle (rq);set_next_task_idle (rq, next);return next;
@@ -458,6 +465,7 @@ const struct sched_class idle_sched_class = { .set_next_task = set_next_task_idle,#ifdef CONFIG_SMP + .pick_task = pick_task_idle, .select_task_rq = select_task_rq_idle, .set_cpus_allowed = set_cpus_allowed_common,#endif
kernel/sched/rt.c +16 -5
@@ -1548,20 +1548,29 @@ static struct task_struct *_pick_next_task_rt(struct rq *rq)return rt_task_of (rt_se); } -static struct task_struct * -pick_next_task_rt (struct rq *rq,struct task_struct *prev,struct rq_flags *rf) +static struct task_struct *pick_task_rt (struct rq *rq) {struct task_struct *p;struct rt_rq *rt_rq = &rq->rt; -WARN_ON_ONCE (prev || rf); -if (!rt_rq->rt_queued)return NULL ; p =_pick_next_task_rt (rq); -set_next_task_rt (rq, p); +return p; +} + +static struct task_struct * +pick_next_task_rt (struct rq *rq,struct task_struct *prev,struct rq_flags *rf) +{ +struct task_struct *p; + +WARN_ON_ONCE (prev || rf); + + p =pick_task_rt (rq); +if (p) +set_next_task_rt (rq, p);return p; }
@@ -2364,6 +2373,8 @@ const struct sched_class rt_sched_class = { .set_next_task = set_next_task_rt, #ifdef CONFIG_SMP + .pick_task = pick_task_rt, + .select_task_rq = select_task_rq_rt, .set_cpus_allowed = set_cpus_allowed_common,
kernel/sched/sched.h +2 -0
@@ -1682,6 +1682,8 @@ struct sched_class {void (*set_next_task)(struct rq *rq,struct task_struct *p);#ifdef CONFIG_SMP +struct task_struct * (*pick_task)(struct rq *rq); +int (*select_task_rq)(struct task_struct *p,int task_cpu,int sd_flag,int flags);void (*migrate_task_rq)(struct task_struct *p,int new_cpu);
kernel/sched/stop_task.c +16 -5
@@ -29,20 +29,30 @@ static void set_next_task_stop(struct rq *rq, struct task_struct *stop) }static struct task_struct * -pick_next_task_stop (struct rq *rq,struct task_struct *prev,struct rq_flags *rf) +pick_task_stop (struct rq *rq) {struct task_struct *stop = rq->stop; -WARN_ON_ONCE (prev || rf); -if (!stop || !task_on_rq_queued (stop))return NULL ; -set_next_task_stop (rq, stop); -return stop; } +static struct task_struct * +pick_next_task_stop (struct rq *rq,struct task_struct *prev,struct rq_flags *rf) +{ +struct task_struct *p; + +WARN_ON_ONCE (prev || rf); + + p =pick_task_stop (rq); +if (p) +set_next_task_stop (rq, p); + +return p; +} +static void enqueue_task_stop (struct rq *rq,struct task_struct *p,int flags) {
@@ -129,6 +139,7 @@ const struct sched_class stop_sched_class = { .set_next_task = set_next_task_stop,#ifdef CONFIG_SMP + .pick_task = pick_task_stop, .select_task_rq = select_task_rq_stop, .set_cpus_allowed = set_cpus_allowed_common,#endif
Signature · 1 line
2.17.1
RFC PATCH v3 7/16 sched: Allow put_prev_task() to drop rq->lock
In reply to Vineeth Remanan PillaiFrom: Peter Zijlstra <[email protected]>
Currently the pick_next_task() loop is convoluted and ugly because of how it can drop the rq->lock and needs to restart the picking.
For the RT/Deadline classes, it is put_prev_task() where we do balancing, and we could do this before the picking loop. Make this possible.
--- kernel/sched/core.c | 2 +- kernel/sched/deadline.c | 14 +++++++++++++- kernel/sched/fair.c | 2 +- kernel/sched/idle.c | 2 +- kernel/sched/rt.c | 14 +++++++++++++- kernel/sched/sched.h | 4 ++-- kernel/sched/stop_task.c | 2 +- 7 files changed, 32 insertions(+), 8 deletions(-)
kernel/sched/core.c +1 -1
@@ -5595,7 +5595,7 @@ static void calc_load_migrate(struct rq *rq)atomic_long_add (delta, &calc_load_tasks); } -static void put_prev_task_fake (struct rq *rq,struct task_struct *prev) +static void put_prev_task_fake (struct rq *rq,struct task_struct *prev,struct rq_flags *rf) { }
kernel/sched/deadline.c +13 -1
@@ -1772,13 +1772,25 @@ pick_next_task_dl(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)return p; } -static void put_prev_task_dl (struct rq *rq,struct task_struct *p) +static void put_prev_task_dl (struct rq *rq,struct task_struct *p,struct rq_flags *rf) {update_curr_dl (rq);update_dl_rq_load_avg (rq_clock_pelt (rq), rq,1 );if (on_dl_rq (&p->dl) && p->nr_cpus_allowed >1 )enqueue_pushable_dl_task (rq, p); + +if (rf && !on_dl_rq (&p->dl) &&need_pull_dl_task (rq, p)) { +/* +* This is OK, because current is on_cpu, which avoids it being +* picked for load-balance and preemption/IRQs are still +* disabled avoiding further scheduler activity on it and we've +* not yet started the picking loop. +*/ +rq_unpin_lock (rq, rf); +pull_dl_task (rq); +rq_repin_lock (rq, rf); + } }/*
kernel/sched/fair.c +1 -1
@@ -7110,7 +7110,7 @@ done: __maybe_unused;/* * Account for a descheduled task: */ -static void put_prev_task_fair (struct rq *rq,struct task_struct *prev) +static void put_prev_task_fair (struct rq *rq,struct task_struct *prev,struct rq_flags *rf) {struct sched_entity *se = &prev->se;struct cfs_rq *cfs_rq;
kernel/sched/idle.c +1 -1
@@ -373,7 +373,7 @@ static void check_preempt_curr_idle(struct rq *rq, struct task_struct *p, int flresched_curr (rq); } -static void put_prev_task_idle (struct rq *rq,struct task_struct *prev) +static void put_prev_task_idle (struct rq *rq,struct task_struct *prev,struct rq_flags *rf) { }
kernel/sched/rt.c +13 -1
@@ -1593,7 +1593,7 @@ pick_next_task_rt(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)return p; } -static void put_prev_task_rt (struct rq *rq,struct task_struct *p) +static void put_prev_task_rt (struct rq *rq,struct task_struct *p,struct rq_flags *rf) {update_curr_rt (rq);
@@ -1605,6 +1605,18 @@ static void put_prev_task_rt(struct rq *rq, struct task_struct *p)*/ if (on_rt_rq (&p->rt) && p->nr_cpus_allowed >1 )enqueue_pushable_task (rq, p); + +if (rf && !on_rt_rq (&p->rt) &&need_pull_rt_task (rq, p)) { +/* +* This is OK, because current is on_cpu, which avoids it being +* picked for load-balance and preemption/IRQs are still +* disabled avoiding further scheduler activity on it and we've +* not yet started the picking loop. +*/ +rq_unpin_lock (rq, rf); +pull_rt_task (rq); +rq_repin_lock (rq, rf); + } }#ifdef CONFIG_SMP
kernel/sched/sched.h +2 -2
@@ -1675,7 +1675,7 @@ struct sched_class {struct task_struct * (*pick_next_task)(struct rq *rq,struct task_struct *prev,struct rq_flags *rf); -void (*put_prev_task)(struct rq *rq,struct task_struct *p); +void (*put_prev_task)(struct rq *rq,struct task_struct *p,struct rq_flags *rf);void (*set_next_task)(struct rq *rq,struct task_struct *p);#ifdef CONFIG_SMP
@@ -1721,7 +1721,7 @@ struct sched_class {static inline void put_prev_task (struct rq *rq,struct task_struct *prev) {WARN_ON_ONCE (rq->curr != prev); - prev->sched_class->put_prev_task (rq, prev); + prev->sched_class->put_prev_task (rq, prev,NULL ); }static inline void set_next_task (struct rq *rq,struct task_struct *next)
kernel/sched/stop_task.c +1 -1
@@ -59,7 +59,7 @@ static void yield_task_stop(struct rq *rq)BUG ();/* the stop task should never yield, its pointless. */ } -static void put_prev_task_stop (struct rq *rq,struct task_struct *prev) +static void put_prev_task_stop (struct rq *rq,struct task_struct *prev,struct rq_flags *rf) {struct task_struct *curr = rq->curr;u64 delta_exec;
Signature · 1 line
2.17.1
RFC PATCH v3 6/16 sched/fair: Export newidle_balance()
In reply to Vineeth Remanan PillaiFrom: Peter Zijlstra <[email protected]>
For pick_next_task_fair() it is the newidle balance that requires dropping the rq->lock; provided we do put_prev_task() early, we can also detect the condition for doing newidle early.
--- kernel/sched/fair.c | 18 ++++++++---------- kernel/sched/sched.h | 4 ++++ 2 files changed, 12 insertions(+), 10 deletions(-)
kernel/sched/fair.c +8 -10
@@ -3615,8 +3615,6 @@ static inline unsigned long cfs_rq_load_avg(struct cfs_rq *cfs_rq)return cfs_rq->avg.load_avg; } -static int idle_balance (struct rq *this_rq,struct rq_flags *rf); -static inline unsigned long task_util (struct task_struct *p) {return READ_ONCE (p->se.avg.util_avg);
@@ -7087,11 +7085,10 @@ done: __maybe_unused;return p; idle: -update_misfit_status (NULL , rq); - new_tasks =idle_balance (rq, rf); + new_tasks =newidle_balance (rq, rf);/* -* Because idle_balance() releases (and re-acquires) rq->lock, it is +* Because newidle_balance() releases (and re-acquires) rq->lock, it is * possible for any higher priority task to appear. In that case we * must re-start the pick_next_entity() loop. */
@@ -9286,10 +9283,10 @@ static int load_balance(int this_cpu, struct rq *this_rq, ld_moved =0 ;/* -* idle_balance() disregards balance intervals, so we could repeatedly -* reach this code, which would lead to balance_interval skyrocketting -* in a short amount of time. Skip the balance_interval increase logic -* to avoid that. +* newidle_balance() disregards balance intervals, so we could +* repeatedly reach this code, which would lead to balance_interval +* skyrocketting in a short amount of time. Skip the balance_interval +* increase logic to avoid that. */ if (env.idle ==CPU_NEWLY_IDLE ) goto out;
@@ -9996,7 +9993,7 @@ static inline void nohz_newidle_balance(struct rq *this_rq) { }* idle_balance is called by schedule() if this_cpu is about to become * idle. Attempts to pull tasks from other CPUs. */ -static int idle_balance (struct rq *this_rq,struct rq_flags *rf) +int newidle_balance (struct rq *this_rq,struct rq_flags *rf) {unsigned long next_balance = jiffies +HZ ;int this_cpu = this_rq->cpu;
@@ -10004,6 +10001,7 @@ static int idle_balance(struct rq *this_rq, struct rq_flags *rf)int pulled_task =0 ;u64 curr_cost =0 ; +update_misfit_status (NULL , this_rq);/* * We must set idle_stamp _before_ calling idle_balance(), such that we * measure the duration of idle_balance() as idle time.
kernel/sched/sched.h +4 -0
@@ -1414,10 +1414,14 @@ static inline void unregister_sched_domain_sysctl(void) }#endif +extern int newidle_balance (struct rq *this_rq,struct rq_flags *rf); +#else static inline void sched_ttwu_pending (void ) { } +static inline int newidle_balance (struct rq *this_rq,struct rq_flags *rf) {return 0 ; } +#endif /* CONFIG_SMP */ #include "stats.h"
Signature · 1 line
2.17.1
RFC PATCH v3 4/16 sched/{rt,deadline}: Fix set_next_task vs pick_next_task
In reply to Vineeth Remanan PillaiFrom: Peter Zijlstra <[email protected]>
Because pick_next_task() implies set_curr_task() and some of the details haven't matter too much, some of what _should_ be in set_curr_task() ended up in pick_next_task, correct this.
This prepares the way for a pick_next_task() variant that does not affect the current state; allowing remote picking.
--- kernel/sched/deadline.c | 23 ++++++++++++----------- kernel/sched/rt.c | 27 ++++++++++++++------------- 2 files changed, 26 insertions(+), 24 deletions(-)
kernel/sched/deadline.c +12 -11
@@ -1694,12 +1694,21 @@ static void start_hrtick_dl(struct rq *rq, struct task_struct *p) }#endif -static inline void set_next_task (struct rq *rq,struct task_struct *p) +static void set_next_task_dl (struct rq *rq,struct task_struct *p) { p->se.exec_start =rq_clock_task (rq);/* You can't push away the running task */ dequeue_pushable_dl_task (rq, p); + +if (hrtick_enabled (rq)) +start_hrtick_dl (rq, p); + +if (rq->curr->sched_class != &dl_sched_class) +update_dl_rq_load_avg (rq_clock_pelt (rq), rq,0 ); + +if (rq->curr != p) +deadline_queue_push_tasks (rq); } static struct sched_dl_entity *pick_next_dl_entity(struct rq *rq,
@@ -1758,15 +1767,7 @@ pick_next_task_dl(struct rq *rq, struct task_struct *prev, struct rq_flags *rf) p =dl_task_of (dl_se); -set_next_task (rq, p); - -if (hrtick_enabled (rq)) -start_hrtick_dl (rq, p); - -deadline_queue_push_tasks (rq); - -if (rq->curr->sched_class != &dl_sched_class) -update_dl_rq_load_avg (rq_clock_pelt (rq), rq,0 ); +set_next_task_dl (rq, p);return p; }
@@ -1813,7 +1814,7 @@ static void task_fork_dl(struct task_struct *p)static void set_curr_task_dl (struct rq *rq) { -set_next_task (rq, rq->curr); +set_next_task_dl (rq, rq->curr); }#ifdef CONFIG_SMP
kernel/sched/rt.c +14 -13
@@ -1498,12 +1498,23 @@ static void check_preempt_curr_rt(struct rq *rq, struct task_struct *p, int flag#endif } -static inline void set_next_task (struct rq *rq,struct task_struct *p) +static inline void set_next_task_rt (struct rq *rq,struct task_struct *p) { p->se.exec_start =rq_clock_task (rq);/* The running task is never eligible for pushing */ dequeue_pushable_task (rq, p); + +/* +* If prev task was rt, put_prev_task() has already updated the +* utilization. We only care of the case where we start to schedule a +* rt task +*/ +if (rq->curr->sched_class != &rt_sched_class) +update_rt_rq_load_avg (rq_clock_pelt (rq), rq,0 ); + +if (rq->curr != p) +rt_queue_push_tasks (rq); } static struct sched_rt_entity *pick_next_rt_entity(struct rq *rq,
@@ -1577,17 +1588,7 @@ pick_next_task_rt(struct rq *rq, struct task_struct *prev, struct rq_flags *rf) p =_pick_next_task_rt (rq); -set_next_task (rq, p); - -rt_queue_push_tasks (rq); - -/* -* If prev task was rt, put_prev_task() has already updated the -* utilization. We only care of the case where we start to schedule a -* rt task -*/ -if (rq->curr->sched_class != &rt_sched_class) -update_rt_rq_load_avg (rq_clock_pelt (rq), rq,0 ); +set_next_task_rt (rq, p);return p; }
@@ -2356,7 +2357,7 @@ static void task_tick_rt(struct rq *rq, struct task_struct *p, int queued)static void set_curr_task_rt (struct rq *rq) { -set_next_task (rq, rq->curr); +set_next_task_rt (rq, rq->curr); }static unsigned int get_rr_interval_rt (struct rq *rq,struct task_struct *task)
Signature · 1 line
2.17.1
Re: [RFC PATCH v3 16/16] sched: Debug bits...
In reply to Vineeth Remanan PillaiQuoted 4 lines · Vineeth Remanan Pillai
From: Peter Zijlstra <[email protected]>
Not-Signed-off-by Peter Zijlstra (Intel) <[email protected]>
No commit message, not-signed-off-by...
Quoted 135 lines
--- kernel/sched/core.c | 44 ++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 42 insertions(+), 2 deletions(-)kernel/sched/core.c +42
-3@@ -92,6 +92,10 @@ static inline bool prio_less(struct task_struct *a, struct task_struct *b)int pa =__task_prio (a), pb =__task_prio (b); +trace_printk ("(%s/%d;%d,%Lu,%Lu) ?< (%s/%d;%d,%Lu,%Lu)\n" , + a->comm, a->pid, pa, a->se.vruntime, a->dl.deadline, + b->comm, b->pid, pb, b->se.vruntime, b->dl.deadline); +if (-pa < -pb)return true;@@ -246,6 +250,8 @@ static void __sched_core_enable(void)static_branch_enable (&__sched_core_enabled);stop_machine (__sched_core_stopper, (void *)true,NULL ); + +printk ("core sched enabled\n" ); }static void __sched_core_disable (void )@@ -254,6 +260,8 @@ static void __sched_core_disable(void)stop_machine (__sched_core_stopper, (void *)false,NULL );static_branch_disable (&__sched_core_enabled); + +printk ("core sched disabled\n" ); }void sched_core_get (void )@@ -3707,6 +3715,14 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)put_prev_task (rq, prev);set_next_task (rq, next); } + +trace_printk ("pick pre selected (%u %u %u): %s/%d %lx\n" , + rq->core->core_task_seq, + rq->core->core_pick_seq, + rq->core_sched_seq, + next->comm, next->pid, + next->core_cookie); +return next; }@@ -3786,6 +3802,9 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)*/ if (i == cpu && !need_sync && !p->core_cookie) { next = p; +trace_printk ("unconstrained pick: %s/%d %lx\n" , + next->comm, next->pid, next->core_cookie); + goto done; }@@ -3794,6 +3813,9 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf) rq_i->core_pick = p; +trace_printk ("cpu(%d): selected: %s/%d %lx\n" , + i, p->comm, p->pid, p->core_cookie); +/* * If this new candidate is of higher priority than the * previous; and they're incompatible; we need to wipe @@ -3810,6 +3832,8 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf) rq->core->core_cookie = p->core_cookie; max = p; +trace_printk ("max: %s/%d %lx\n" , max->comm, max->pid, max->core_cookie); +if (old_max) {for_each_cpu (j, smt_mask) {if (j == i)@@ -3837,6 +3861,7 @@ next_class:; rq->core->core_pick_seq = rq->core->core_task_seq; next = rq->core_pick; rq->core_sched_seq = rq->core->core_pick_seq; +trace_printk ("picked: %s/%d %lx\n" , next->comm, next->pid, next->core_cookie);/* * Reschedule siblings @@ -3862,11 +3887,20 @@ next_class:;if (i == cpu)continue ; -if (rq_i->curr != rq_i->core_pick) +if (rq_i->curr != rq_i->core_pick) { +trace_printk ("IPI(%d)\n" , i);resched_curr (rq_i); + }/* Did we break L1TF mitigation requirements? */ -WARN_ON_ONCE (!cookie_match (next, rq_i->core_pick)); +if (unlikely (!cookie_match (next, rq_i->core_pick))) { +trace_printk ("[%d]: cookie mismatch. %s/%d/0x%lx/0x%lx\n" , + rq_i->cpu, rq_i->core_pick->comm, + rq_i->core_pick->pid, + rq_i->core_pick->core_cookie, + rq_i->core->core_cookie); +WARN_ON_ONCE (1 ); + } } done:@@ -3905,6 +3939,10 @@ static bool try_steal_cookie(int this, int that)if (p->core_occupation > dst->idle->core_occupation) goto next; +trace_printk ("core fill: %s/%d (%d->%d) %d %d %lx\n" , + p->comm, p->pid, that, this, + p->core_occupation, dst->idle->core_occupation, cookie); + p->on_rq =TASK_ON_RQ_MIGRATING ;deactivate_task (src, p,0 );set_task_cpu (p, this);@@ -6501,6 +6539,8 @@ int sched_cpu_starting(unsigned int cpu)WARN_ON_ONCE (rq->core && rq->core != core_rq); rq->core = core_rq; } + +printk ("core: %d -> %d\n" , cpu,cpu_of (core_rq));#endif /* CONFIG_SCHED_CORE */ sched_rq_cpu_starting (cpu); --2.17.1
Quoted 7 lines · Vineeth Remanan Pillai
Third iteration of the Core-Scheduling feature.
This version fixes mostly correctness related issues in v2 and addresses performance issues. Also, addressed some crashes related to cgroups and cpu hotplugging.
Quoted 7 lines
We have tested and verified that incompatible processes are not selected during schedule. In terms of performance, the impact depends on the workload:
- on CPU intensive applications that use all the logical CPUs with SMT enabled, enabling core scheduling performs better than nosmt.
- on mixed workloads with considerable io compared to cpu usage, nosmt seems to perform better than core scheduling.
My testing scripts can not be completed on this version. I figured out the number of cpu utilization report entry didn't reach my minimal requirement. Then I wrote a simple script to verify.
====================
$ cat test.sh #!/bin/sh
for i in seq 1 10
do
echo `date`, $i
sleep 1done
====================
Normally it works as below:
Thu May 30 14:13:40 CST 2019, 1
Thu May 30 14:13:41 CST 2019, 2
Thu May 30 14:13:42 CST 2019, 3
Thu May 30 14:13:43 CST 2019, 4
Thu May 30 14:13:44 CST 2019, 5
Thu May 30 14:13:45 CST 2019, 6
Thu May 30 14:13:46 CST 2019, 7
Thu May 30 14:13:47 CST 2019, 8
Thu May 30 14:13:48 CST 2019, 9
Thu May 30 14:13:49 CST 2019, 10
When the system was running 32 sysbench threads and 32 gemmbench threads, it worked as below(the system has ~38% idle time)
Thu May 30 14:14:20 CST 2019, 1
Thu May 30 14:14:21 CST 2019, 2
Thu May 30 14:14:22 CST 2019, 3
Thu May 30 14:14:24 CST 2019, 4 <=======x= Thu May 30 14:14:25 CST 2019, 5
Thu May 30 14:14:26 CST 2019, 6
Thu May 30 14:14:28 CST 2019, 7 <=======x= Thu May 30 14:14:29 CST 2019, 8
Thu May 30 14:14:31 CST 2019, 9 <=======x= Thu May 30 14:14:34 CST 2019, 10 <=======x=
And it got worse when the system was running 64/64 case, the system still had ~3% idle time
Thu May 30 14:26:40 CST 2019, 1
Thu May 30 14:26:46 CST 2019, 2
Thu May 30 14:26:53 CST 2019, 3
Thu May 30 14:27:01 CST 2019, 4
Thu May 30 14:27:03 CST 2019, 5
Thu May 30 14:27:11 CST 2019, 6
Thu May 30 14:27:31 CST 2019, 7
Thu May 30 14:27:32 CST 2019, 8
Thu May 30 14:27:41 CST 2019, 9
Thu May 30 14:27:56 CST 2019, 10
Any thoughts?
Thanks,
-Aubrey
Quoted 71 lines · Aubrey Li
Quoted 7 lines · Vineeth Remanan Pillai
Third iteration of the Core-Scheduling feature.
This version fixes mostly correctness related issues in v2 and addresses performance issues. Also, addressed some crashes related to cgroups and cpu hotplugging.
Quoted 7 lines
We have tested and verified that incompatible processes are not selected during schedule. In terms of performance, the impact depends on the workload:
- on CPU intensive applications that use all the logical CPUs with SMT enabled, enabling core scheduling performs better than nosmt.
- on mixed workloads with considerable io compared to cpu usage, nosmt seems to perform better than core scheduling.My testing scripts can not be completed on this version. I figured out the number of cpu utilization report entry didn't reach my minimal requirement. Then I wrote a simple script to verify.
====================$ cat test.sh #!/bin/shfor i in
seq 1 10
doecho `date`, $i sleep 1done
====================Normally it works as below:
Thu May 30 14:13:40 CST 2019, 1
Thu May 30 14:13:41 CST 2019, 2
Thu May 30 14:13:42 CST 2019, 3
Thu May 30 14:13:43 CST 2019, 4
Thu May 30 14:13:44 CST 2019, 5
Thu May 30 14:13:45 CST 2019, 6
Thu May 30 14:13:46 CST 2019, 7
Thu May 30 14:13:47 CST 2019, 8
Thu May 30 14:13:48 CST 2019, 9
Thu May 30 14:13:49 CST 2019, 10When the system was running 32 sysbench threads and 32 gemmbench threads, it worked as below(the system has ~38% idle time)
Thu May 30 14:14:20 CST 2019, 1
Thu May 30 14:14:21 CST 2019, 2
Thu May 30 14:14:22 CST 2019, 3
Thu May 30 14:14:24 CST 2019, 4 <=======x= Thu May 30 14:14:25 CST 2019, 5
Thu May 30 14:14:26 CST 2019, 6
Thu May 30 14:14:28 CST 2019, 7 <=======x= Thu May 30 14:14:29 CST 2019, 8
Thu May 30 14:14:31 CST 2019, 9 <=======x= Thu May 30 14:14:34 CST 2019, 10 <=======x=And it got worse when the system was running 64/64 case, the system still had ~3% idle time
Thu May 30 14:26:40 CST 2019, 1
Thu May 30 14:26:46 CST 2019, 2
Thu May 30 14:26:53 CST 2019, 3
Thu May 30 14:27:01 CST 2019, 4
Thu May 30 14:27:03 CST 2019, 5
Thu May 30 14:27:11 CST 2019, 6
Thu May 30 14:27:31 CST 2019, 7
Thu May 30 14:27:32 CST 2019, 8
Thu May 30 14:27:41 CST 2019, 9
Thu May 30 14:27:56 CST 2019, 10
Quoted 1 line
Any thoughts?
Interesting, could you detail a bit more your test setup (commands used, type of machine, any cgroup/pinning configuration, etc) ? I would like to reproduce it and investigate.
Thanks,
Julien
Quoted 24 lines · Aubrey Li
Quoted 7 lines · Vineeth Remanan Pillai
Third iteration of the Core-Scheduling feature.
This version fixes mostly correctness related issues in v2 and addresses performance issues. Also, addressed some crashes related to cgroups and cpu hotplugging.
Quoted 7 lines
We have tested and verified that incompatible processes are not selected during schedule. In terms of performance, the impact depends on the workload:
- on CPU intensive applications that use all the logical CPUs with SMT enabled, enabling core scheduling performs better than nosmt.
- on mixed workloads with considerable io compared to cpu usage, nosmt seems to perform better than core scheduling.My testing scripts can not be completed on this version. I figured out the number of cpu utilization report entry didn't reach my minimal requirement. Then I wrote a simple script to verify.
====================$ cat test.sh #!/bin/sh
Quoted 6 lines
for i in
seq 1 10
doecho `date`, $i sleep 1done
====================
Is the shell put to some cgroup and assigned some tag or simply untagged?
Quoted 14 lines
Normally it works as below:
Thu May 30 14:13:40 CST 2019, 1
Thu May 30 14:13:41 CST 2019, 2
Thu May 30 14:13:42 CST 2019, 3
Thu May 30 14:13:43 CST 2019, 4
Thu May 30 14:13:44 CST 2019, 5
Thu May 30 14:13:45 CST 2019, 6
Thu May 30 14:13:46 CST 2019, 7
Thu May 30 14:13:47 CST 2019, 8
Thu May 30 14:13:48 CST 2019, 9
Thu May 30 14:13:49 CST 2019, 10
Quoted 3 lines
When the system was running 32 sysbench threads and 32 gemmbench threads, it worked as below(the system has ~38% idle time)
Are the two workloads assigned different tags?
And how many cores/threads do you have?
Quoted 10 lines
Thu May 30 14:14:20 CST 2019, 1
Thu May 30 14:14:21 CST 2019, 2
Thu May 30 14:14:22 CST 2019, 3
Thu May 30 14:14:24 CST 2019, 4 <=======x=
Thu May 30 14:14:25 CST 2019, 5
Thu May 30 14:14:26 CST 2019, 6
Thu May 30 14:14:28 CST 2019, 7 <=======x=
Thu May 30 14:14:29 CST 2019, 8
Thu May 30 14:14:31 CST 2019, 9 <=======x=
Thu May 30 14:14:34 CST 2019, 10 <=======x=
This feels like "date" failed to schedule on some CPU on time.
Quoted 13 lines
And it got worse when the system was running 64/64 case, the system still had ~3% idle time
Thu May 30 14:26:40 CST 2019, 1
Thu May 30 14:26:46 CST 2019, 2
Thu May 30 14:26:53 CST 2019, 3
Thu May 30 14:27:01 CST 2019, 4
Thu May 30 14:27:03 CST 2019, 5
Thu May 30 14:27:11 CST 2019, 6
Thu May 30 14:27:31 CST 2019, 7
Thu May 30 14:27:32 CST 2019, 8
Thu May 30 14:27:41 CST 2019, 9
Thu May 30 14:27:56 CST 2019, 10
Quoted 1 line
Any thoughts?
My first reaction is: when shell wakes up from sleep, it will fork date. If the script is untagged and those workloads are tagged and all available cores are already running workload threads, the forked date can lose to the running workload threads due to __prio_less() can't properly do vruntime comparison for tasks on different CPUs. So those idle siblings can't run date and are idled instead. See my previous post on this:
https://lore.kernel.org/lkml/20190429033620.GA128241@aaronlu/
(Now that I re-read my post, I see that I didn't make it clear that se_bash and se_hog are assigned different tags(e.g. hog is tagged and bash is untagged).
Siblings being forced idle is expected due to the nature of core scheduling, but when two tasks belonging to two siblings are fighting for schedule, we should let the higher priority one win.
It used to work on v2 is probably due to we mistakenly allow different tagged tasks to schedule on the same core at the same time, but that is fixed in v3.
Quoted 4 lines · Julien Desfossez
Interesting, could you detail a bit more your test setup (commands used, type of machine, any cgroup/pinning configuration, etc) ? I would like to reproduce it and investigate.
Let me see if I can simply my test to reproduce it.
Thanks,
-Aubrey
Quoted 4 lines · Aaron Lu
This feels like "date" failed to schedule on some CPU on time.
Quoted 11 lines
My first reaction is: when shell wakes up from sleep, it will fork date. If the script is untagged and those workloads are tagged and all available cores are already running workload threads, the forked date can lose to the running workload threads due to
__prio_less()can't properly do vruntime comparison for tasks on different CPUs. So those idle siblings can't run date and are idled instead. See my previous post on this:
https://lore.kernel.org/lkml/20190429033620.GA128241@aaronlu/
(Now that I re-read my post, I see that I didn't make it clear thatse_bashandse_hogare assigned different tags(e.g. hog is tagged and bash is untagged).
Yes, script is untagged. This looks like exactly the problem in you previous post. I didn't follow that, does that discussion lead to a solution?
Quoted 8 lines
Siblings being forced idle is expected due to the nature of core scheduling, but when two tasks belonging to two siblings are fighting for schedule, we should let the higher priority one win.
It used to work on v2 is probably due to we mistakenly allow different tagged tasks to schedule on the same core at the same time, but that is fixed in v3.
I have 64 threads running on a 104-CPU server, that is, when the system has ~40% idle time, and "date" is still failed to be picked up onto CPU on time. This may be the nature of core scheduling, but it seems to be far from fairness.
Shouldn't we share the core between (sysbench+gemmbench) and (date)? I mean core level sharing instead of "date" starvation?
Thanks,
-Aubrey
Quoted 17 lines · Aubrey Li
Quoted 4 lines · Aaron Lu
This feels like "date" failed to schedule on some CPU on time.
Quoted 11 lines
My first reaction is: when shell wakes up from sleep, it will fork date. If the script is untagged and those workloads are tagged and all available cores are already running workload threads, the forked date can lose to the running workload threads due to
__prio_less()can't properly do vruntime comparison for tasks on different CPUs. So those idle siblings can't run date and are idled instead. See my previous post on this:
https://lore.kernel.org/lkml/20190429033620.GA128241@aaronlu/
(Now that I re-read my post, I see that I didn't make it clear thatse_bashandse_hogare assigned different tags(e.g. hog is tagged and bash is untagged).
Quoted 2 lines
Yes, script is untagged. This looks like exactly the problem in you previous post. I didn't follow that, does that discussion lead to a solution?
No immediate solution yet.
Quoted 10 lines
Quoted 8 lines
Siblings being forced idle is expected due to the nature of core scheduling, but when two tasks belonging to two siblings are fighting for schedule, we should let the higher priority one win.
It used to work on v2 is probably due to we mistakenly allow different tagged tasks to schedule on the same core at the same time, but that is fixed in v3.
I have 64 threads running on a 104-CPU server, that is, when the
104-CPU means 52 cores I guess.
64 threads may(should?) spread on all the 52 cores and that is enough to make 'date' suffer.
Quoted 3 lines
system has ~40% idle time, and "date" is still failed to be picked up onto CPU on time. This may be the nature of core scheduling, but it seems to be far from fairness.
Exactly.
Quoted 2 lines
Shouldn't we share the core between (sysbench+gemmbench) and (date)? I mean core level sharing instead of "date" starvation?
We need to make core scheduling fair, but due to no immediate solution to vruntime comparison cross CPUs, it's not done yet.
Quoted 33 lines · Aaron Lu
Quoted 17 lines · Aubrey Li
Quoted 4 lines · Aaron Lu
This feels like "date" failed to schedule on some CPU on time.
Quoted 11 lines
My first reaction is: when shell wakes up from sleep, it will fork date. If the script is untagged and those workloads are tagged and all available cores are already running workload threads, the forked date can lose to the running workload threads due to
__prio_less()can't properly do vruntime comparison for tasks on different CPUs. So those idle siblings can't run date and are idled instead. See my previous post on this:
https://lore.kernel.org/lkml/20190429033620.GA128241@aaronlu/
(Now that I re-read my post, I see that I didn't make it clear thatse_bashandse_hogare assigned different tags(e.g. hog is tagged and bash is untagged).Quoted 2 lines
Yes, script is untagged. This looks like exactly the problem in you previous post. I didn't follow that, does that discussion lead to a solution?
No immediate solution yet.
Quoted 9 lines
Quoted 8 lines
Siblings being forced idle is expected due to the nature of core scheduling, but when two tasks belonging to two siblings are fighting for schedule, we should let the higher priority one win.
It used to work on v2 is probably due to we mistakenly allow different tagged tasks to schedule on the same core at the same time, but that is fixed in v3.
Quoted 5 lines
Quoted 1 line
I have 64 threads running on a 104-CPU server, that is, when the
104-CPU means 52 cores I guess.
64 threads may(should?) spread on all the 52 cores and that is enough to make 'date' suffer.
64 threads should spread onto all the 52 cores, but why they can get scheduled while untagged "date" can not? Is it because in the current implementation the task with cookie always has higher priority than the task without a cookie?
Thanks,
-Aubrey
Quoted 34 lines · Aubrey Li
Quoted 33 lines · Aaron Lu
Quoted 17 lines · Aubrey Li
Quoted 4 lines · Aaron Lu
This feels like "date" failed to schedule on some CPU on time.
Quoted 11 lines
My first reaction is: when shell wakes up from sleep, it will fork date. If the script is untagged and those workloads are tagged and all available cores are already running workload threads, the forked date can lose to the running workload threads due to
__prio_less()can't properly do vruntime comparison for tasks on different CPUs. So those idle siblings can't run date and are idled instead. See my previous post on this:
https://lore.kernel.org/lkml/20190429033620.GA128241@aaronlu/
(Now that I re-read my post, I see that I didn't make it clear thatse_bashandse_hogare assigned different tags(e.g. hog is tagged and bash is untagged).Quoted 2 lines
Yes, script is untagged. This looks like exactly the problem in you previous post. I didn't follow that, does that discussion lead to a solution?
No immediate solution yet.
Quoted 9 lines
Quoted 8 lines
Siblings being forced idle is expected due to the nature of core scheduling, but when two tasks belonging to two siblings are fighting for schedule, we should let the higher priority one win.
It used to work on v2 is probably due to we mistakenly allow different tagged tasks to schedule on the same core at the same time, but that is fixed in v3.
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
I have 64 threads running on a 104-CPU server, that is, when the
104-CPU means 52 cores I guess.
64 threads may(should?) spread on all the 52 cores and that is enough to make 'date' suffer.64 threads should spread onto all the 52 cores, but why they can get scheduled while untagged "date" can not? Is it because in the current
If 'date' didn't get scheduled, there will be no output at all unless all those workload threads finished :-)
I guess the workload you used is not entirely CPU intensive, or 'date' can be totally blocked due to START_DEBIT. But note that START_DEBIT isn't the problem here, cross CPU vruntime comparison is.
Quoted 2 lines
implementation the task with cookie always has higher priority than the task without a cookie?
No.
Quoted 42 lines · Aaron Lu
Quoted 40 lines · Aubrey Li
Quoted 38 lines · Aaron Lu
Quoted 17 lines · Aubrey Li
Quoted 4 lines · Aaron Lu
This feels like "date" failed to schedule on some CPU on time.
Quoted 11 lines
My first reaction is: when shell wakes up from sleep, it will fork date. If the script is untagged and those workloads are tagged and all available cores are already running workload threads, the forked date can lose to the running workload threads due to
__prio_less()can't properly do vruntime comparison for tasks on different CPUs. So those idle siblings can't run date and are idled instead. See my previous post on this:
https://lore.kernel.org/lkml/20190429033620.GA128241@aaronlu/
(Now that I re-read my post, I see that I didn't make it clear thatse_bashandse_hogare assigned different tags(e.g. hog is tagged and bash is untagged).Quoted 2 lines
Yes, script is untagged. This looks like exactly the problem in you previous post. I didn't follow that, does that discussion lead to a solution?
No immediate solution yet.
Quoted 10 lines
Quoted 8 lines
Siblings being forced idle is expected due to the nature of core scheduling, but when two tasks belonging to two siblings are fighting for schedule, we should let the higher priority one win.
It used to work on v2 is probably due to we mistakenly allow different tagged tasks to schedule on the same core at the same time, but that is fixed in v3.
I have 64 threads running on a 104-CPU server, that is, when the
104-CPU means 52 cores I guess.
64 threads may(should?) spread on all the 52 cores and that is enough to make 'date' suffer.
Quoted 5 lines
Quoted 2 lines
64 threads should spread onto all the 52 cores, but why they can get scheduled while untagged "date" can not? Is it because in the current
If 'date' didn't get scheduled, there will be no output at all unless all those workload threads finished :-)
Certainly I meant untagged "date" can not be scheduled on time, :)
Quoted 9 lines
I guess the workload you used is not entirely CPU intensive, or 'date' can be totally blocked due to
START_DEBIT. But note thatSTART_DEBITisn't the problem here, cross CPU vruntime comparison is.Quoted 2 lines
implementation the task with cookie always has higher priority than the task without a cookie?
No.
I checked the benchmark log manually, it looks like the data of two benchmarks with cookies are acceptable, but ones without cookies are really bad.
Peter Zijlstra
maintainerRe: [RFC PATCH v3 10/16] sched: Core-wide rq->lock
In reply to Vineeth Remanan PillaiQuoted 32 lines · Vineeth Remanan Pillai
+ * The static-key + stop-machine variable are needed such that:
+ *
+ * spin_lock(rq_lockp(rq));
+ * ...
+ * spin_unlock(rq_lockp(rq));
+ *
+ * ends up locking and unlocking the _same_ lock, and all CPUs
+ * always agree on what rq has what lock.kernel/sched/core.c +9
-2@@ -5790,8 +5854,15 @@ int sched_cpu_activate(unsigned int cpu)/* * When going up, increment the number of cores with SMT present. */ -if (cpumask_weight (cpu_smt_mask (cpu)) ==2 ) +if (cpumask_weight (cpu_smt_mask (cpu)) ==2 ) {static_branch_inc_cpuslocked (&sched_smt_present); +#ifdef CONFIG_SCHED_CORE +if (static_branch_unlikely (&__sched_core_enabled)) { + rq->core_enabled = true; + } +#endif + } +#endif set_cpu_active (cpu, true);@@ -5839,8 +5910,16 @@ int sched_cpu_deactivate(unsigned int cpu)/* * When going down, decrement the number of cores with SMT present. */ -if (cpumask_weight (cpu_smt_mask (cpu)) ==2 ) +if (cpumask_weight (cpu_smt_mask (cpu)) ==2 ) {
kernel/sched/core.c int sched_cpu_deactivate(unsigned int cpu) · line 5923
Quoted 10 lines
kernel/sched/core.c +8
-0+#ifdef CONFIG_SCHED_CORE +struct rq *rq =cpu_rq (cpu); +if (static_branch_unlikely (&__sched_core_enabled)) { + rq->core_enabled = false; + } +#endif static_branch_dec_cpuslocked(&sched_smt_present); + + }#endif
I'm confused, how doesn't this break the invariant above?
That is, all CPUs must at all times agree on the value of rq_lockp(), and I'm not seeing how that is true with the above changes.
Re: [RFC PATCH v3 10/16] sched: Core-wide rq->lock
Quoted 6 lines
I'm confused, how doesn't this break the invariant above?
That is, all CPUs must at all times agree on the value of
rq_lockp(), and I'm not seeing how that is true with the above changes.
While fixing the crash in cpu online/offline, I was focusing on maintaining the invariance
of all online cpus to agree on the value of rq_lockp(). Would it be safe to assume that
rq and rq_lock would be used only after a cpu is onlined(sched:active)?.
To maintain the strict invariance, the sibling should also disable core scheduling, but
we need to empty the rbtree before disabling it. I am trying to see how to empty the
rbtree safely in the offline context.
Thanks,
Vineeth
Quoted 13 lines
My first reaction is: when shell wakes up from sleep, it will fork date. If the script is untagged and those workloads are tagged and all available cores are already running workload threads, the forked date can lose to the running workload threads due to
__prio_less()can't properly do vruntime comparison for tasks on different CPUs. So those idle siblings can't run date and are idled instead. See my previous post on this:https://lore.kernel.org/lkml/20190429033620.GA128241@aaronlu/
(Now that I re-read my post, I see that I didn't make it clear thatse_bashandse_hogare assigned different tags(e.g. hog is tagged and bash is untagged).
Quoted 7 lines
Siblings being forced idle is expected due to the nature of core scheduling, but when two tasks belonging to two siblings are fighting for schedule, we should let the higher priority one win.
It used to work on v2 is probably due to we mistakenly allow different tagged tasks to schedule on the same core at the same time, but that is fixed in v3.
I confirm this is indeed what is happening, we reproduced it with a simple script that only uses one core (cpu 2 and 38 are sibling on this machine):
setup:
cgcreate -g cpu,cpuset:test
cgcreate -g cpu,cpuset:test/set1
cgcreate -g cpu,cpuset:test/set2
echo 2,38 > /sys/fs/cgroup/cpuset/test/cpuset.cpus echo 0 > /sys/fs/cgroup/cpuset/test/cpuset.mems echo 2,38 > /sys/fs/cgroup/cpuset/test/set1/cpuset.cpus echo 2,38 > /sys/fs/cgroup/cpuset/test/set2/cpuset.cpus echo 0 > /sys/fs/cgroup/cpuset/test/set1/cpuset.mems echo 0 > /sys/fs/cgroup/cpuset/test/set2/cpuset.mems echo 1 > /sys/fs/cgroup/cpu,cpuacct/test/set1/cpu.tag
In one terminal:
sudo cgexec -g cpu,cpuset:test/set1 sysbench --threads=1 --time=30 --test=cpu run
In another one:
sudo cgexec -g cpu,cpuset:test/set2 date
It's very clear that 'date' hangs until sysbench is done.
We started experimenting with marking a task on the forced idle sibling if normalized vruntimes are equal. That way, at the next compare, if the normalized vruntimes are still equal, it prefers the task on the forced idle sibling. It still needs more work, but in our early tests it helps.
Thanks,
Julien
Quoted 44 lines · Julien Desfossez
Quoted 13 lines
My first reaction is: when shell wakes up from sleep, it will fork date. If the script is untagged and those workloads are tagged and all available cores are already running workload threads, the forked date can lose to the running workload threads due to
__prio_less()can't properly do vruntime comparison for tasks on different CPUs. So those idle siblings can't run date and are idled instead. See my previous post on this:https://lore.kernel.org/lkml/20190429033620.GA128241@aaronlu/
(Now that I re-read my post, I see that I didn't make it clear thatse_bashandse_hogare assigned different tags(e.g. hog is tagged and bash is untagged).Quoted 7 lines
Siblings being forced idle is expected due to the nature of core scheduling, but when two tasks belonging to two siblings are fighting for schedule, we should let the higher priority one win.
It used to work on v2 is probably due to we mistakenly allow different tagged tasks to schedule on the same core at the same time, but that is fixed in v3.
I confirm this is indeed what is happening, we reproduced it with a simple script that only uses one core (cpu 2 and 38 are sibling on this machine):
setup:
cgcreate -g cpu,cpuset:test
cgcreate -g cpu,cpuset:test/set1
cgcreate -g cpu,cpuset:test/set2
echo 2,38 >/sys/fs/cgroup/cpuset/test/cpuset.cpusecho 0 >/sys/fs/cgroup/cpuset/test/cpuset.memsecho 2,38 >/sys/fs/cgroup/cpuset/test/set1/cpuset.cpusecho 2,38 >/sys/fs/cgroup/cpuset/test/set2/cpuset.cpusecho 0 >/sys/fs/cgroup/cpuset/test/set1/cpuset.memsecho 0 >/sys/fs/cgroup/cpuset/test/set2/cpuset.memsecho 1 >/sys/fs/cgroup/cpu,cpuacct/test/set1/cpu.tagIn one terminal:
sudo cgexec -g cpu,cpuset:test/set1 sysbench --threads=1 --time=30 --test=cpu runIn another one:
sudo cgexec -g cpu,cpuset:test/set2 date
Quoted 6 lines
It's very clear that 'date' hangs until sysbench is done.
We started experimenting with marking a task on the forced idle sibling if normalized vruntimes are equal. That way, at the next compare, if the normalized vruntimes are still equal, it prefers the task on the forced idle sibling. It still needs more work, but in our early tests it helps.
As mentioned above, we have come up with a fix for the long starvation of untagged interactive threads competing for the same core with tagged threads at the same priority. The idea is to detect the stall and boost the stalling threads priority so that it gets a chance next time. Boosting is done by a new counter(min_vruntime_boost) for every task which we subtract from vruntime before comparison. The new logic looks like this:
If we see that normalized runtimes are equal, we check the min_vruntimes of their runqueues and give a chance for the task in the runqueue with less min_vruntime. That will help it to progress its vruntime. While doing this, we boost the priority of the task in the sibling so that, we don’t starve the task in the sibling until the min_vruntime of this runqueue catches up.
If min_vruntimes are also equal, we do as before and consider the task ‘a’ of higher priority. Here we boost the task ‘b’ so that it gets to run next time.
The min_vruntime_boost is reset to zero once the task in on cpu. So only waiting tasks will have a non-zero value if it is starved while matching a task on the other sibling.
The attached patch has a sched_feature to enable the above feature so that you can compare the results with and without this feature.
What we observe with this patch is that it helps for untagged interactive tasks and fairness in general, but this increases the overhead of core scheduling when there is contention for the CPU with tasks of varying cpu usage. The general trend we see is that if there is a cpu intensive thread and multiple relatively idle threads in different tags, the cpu intensive tasks continuously yields to be fair to the relatively idle threads when it becomes runnable. And if the relatively idle threads make up for most of the tasks in a system and are tagged, the cpu intensive tasks sees a considerable drop in performance.
If you have any feedback or creative ideas to help improve, let us know !
Thanks
include/linux/sched.h +1 -0
@@ -642,6 +642,7 @@ struct task_struct {struct rb_node core_node;unsigned long core_cookie;unsigned int core_occupation; +unsigned int core_vruntime_boost;#endif #ifdef CONFIG_CGROUP_SCHED
kernel/sched/core.c +35 -8
86 lines not on this page · raw message
kernel/sched/features.h +6 -0
@@ -90,3 +90,9 @@ SCHED_FEAT(WA_BIAS, true)* UtilEstimation. Use estimated CPU utilization. */ SCHED_FEAT (UTIL_EST , true) + +/* +* Prevent task stall due to vruntime comparison limitation across +* cpus. + */ +SCHED_FEAT (CORESCHED_STALL_FIX , false)
Re: [RFC PATCH v3 13/16] sched: Add core wide task selection and scheduling.
In reply to Vineeth Remanan PillaiQuoted 9 lines · Vineeth Remanan Pillai
From: Peter Zijlstra <[email protected]>
Instead of only selecting a local task, select a task for all SMT siblings for every reschedule on the core (irrespective which logical CPU does the reschedule).
NOTE: there is still potential for siblings rivalry.
NOTE: this is far too complicated; but thus far I've failed to
simplify it further.
Looks like there are still some race conditions while bringing cpu online/offline. I am seeing an easy to reproduce panic when turning SMT on/off in a loop with core scheduling ON. I dont see the panic with core scheduling OFF.
Steps to reproduce:
mkdir /sys/fs/cgroup/cpu/group1
mkdir /sys/fs/cgroup/cpu/group2
echo 1 > /sys/fs/cgroup/cpu/group1/cpu.tag
echo 1 > /sys/fs/cgroup/cpu/group2/cpu.tag
echo $$ > /sys/fs/cgroup/cpu/group1/tasks
while [ 1 ]; do
echo on > /sys/devices/system/cpu/smt/control
echo off > /sys/devices/system/cpu/smt/control
done
Panic logs:
[ 274.629437] BUG: unable to handle kernel NULL pointer dereference at 0000000000000024 [ 274.630366] #PF error: [normal kernel read fault] [ 274.630933] PGD 800000003e52c067 P4D 800000003e52c067 PUD 0 [ 274.631613] Oops: 0000 [#1] SMP PTI [ 274.632016] CPU: 0 PID: 1470 Comm: bash Tainted: G W 5.1.4+ #33 [ 274.632854] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS ?-20180724_192412-buildhw-07.phx2.fedoraproject.org-1.fc29 04/01/2014 [ 274.634248] RIP: 0010:__schedule+0x9d4/0x1350 [ 274.634699] Code: da 0f 83 21 04 00 00 48 8b 35 70 f3 ab 00 48 c7 c7 51 1c a8 81 e8 4c 4e 6b ff 49 8b 85 b8 0b 00 00 48 85 c0 0f 84 2f 09 00 01 [ 274.636648] RSP: 0018:ffffc900008f3ca8 EFLAGS: 00010046 [ 274.637197] RAX: 0000000000000000 RBX: 0000000000000001 RCX: 0000000000000040 [ 274.637941] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffffff82544890 [ 274.638691] RBP: ffffc900008f3d40 R08: 00000000000004c7 R09: 0000000000000030 [ 274.639449] R10: 0000000000000001 R11: ffffc900008f3b28 R12: ffff88803d2d0e80 [ 274.640172] R13: ffff88803eaa0a40 R14: ffff88803ea20a40 R15: ffff88803d2d0e80 [ 274.640915] FS: 0000000000000000(0000) GS:ffff88803ea00000(0063) knlGS:00000000f7f8b780 [ 274.641755] CS: 0010 DS: 002b ES: 002b CR0: 0000000080050033 [ 274.642355] CR2: 0000000000000024 CR3: 000000003c01a005 CR4: 0000000000360ef0 [ 274.643135] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 274.643995] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 274.645023] Call Trace: [ 274.645336] schedule+0x28/0x70 [ 274.645621] native_cpu_up+0x271/0x6d0 [ 274.645959] ? cpus_read_trylock+0x40/0x40 [ 274.646324] bringup_cpu+0x2d/0xe0 [ 274.646631] cpuhp_invoke_callback+0x94/0x550 [ 274.647032] ? ring_buffer_record_is_set_on+0x10/0x10 [ 274.647478] _cpu_up+0xa9/0x140 [ 274.647763] store_smt_control+0x1cb/0x260 [ 274.648132] kernfs_fop_write+0x108/0x190 [ 274.648498] vfs_write+0xa5/0x1a0 [ 274.648794] ksys_write+0x57/0xd0 [ 274.649100] do_fast_syscall_32+0x92/0x220 [ 274.649468] entry_SYSENTER_compat+0x7c/0x8e
NULL pointer exception is triggered when sibling is offline during core task pick in pick_next_task() leaving rq_i->core_pick = NULL and if sibling comes online before the "Reschedule siblings" block in the same function it causes panic in is_idle_task(rq_i->core_pick).
Traces for the scenario:
[ 274.599567] bash-1470 0d... 273921815us : __schedule: cpu(0) is online during core_pick [ 274.600339] bash-1470 0d... 273921816us : __schedule: cpu(1) is offline during core_pick [ 274.601106] bash-1470 0d... 273921816us : __schedule: picked: bash/1470 ffff88803cb9c000 [ 274.602106] bash-1470 0d... 273921816us : __schedule: cpu(0) is online.. during Reschedule siblings [ 274.603219] bash-1470 0d... 273921816us : __schedule: cpu(1) is online.. during Reschedule siblings [ 274.604333] <idle>-0 1d... 273921816us : start_secondary: cpu(1) is online now [ 274.605239] bash-1470 0d... 273922148us : __schedule: rq_i->core_pick on cpu(1) is NULL
I am not able to reproduce the panic after the below change. Not sure if this is the right fix. Maybe we don't have to allow cpus to go online/offline while pick_next_task() is executing.
-------------- 8< ---------------
kernel/sched/core.c +1 -1
@@ -3874,7 +3874,7 @@ next_class:;for_each_cpu (i, smt_mask) {struct rq *rq_i =cpu_rq (i); -if (cpu_is_offline (i)) +if (cpu_is_offline (i) || !rq_i->core_pick)continue ;WARN_ON_ONCE (!rq_i->core_pick);
Quoted 36 lines · Julien Desfossez
As mentioned above, we have come up with a fix for the long starvation of untagged interactive threads competing for the same core with tagged threads at the same priority. The idea is to detect the stall and boost the stalling threads priority so that it gets a chance next time. Boosting is done by a new
counter(min_vruntime_boost)for every task which we subtract from vruntime before comparison. The new logic looks like this:If we see that normalized runtimes are equal, we check the
min_vruntimesof their runqueues and give a chance for the task in the runqueue with lessmin_vruntime. That will help it to progress its vruntime. While doing this, we boost the priority of the task in the sibling so that, we don’t starve the task in the sibling until themin_vruntimeof this runqueue catches up.If
min_vruntimesare also equal, we do as before and consider the task ‘a’ of higher priority. Here we boost the task ‘b’ so that it gets to run next time.The
min_vruntime_boostis reset to zero once the task in on cpu. So only waiting tasks will have a non-zero value if it is starved while matching a task on the other sibling.The attached patch has a
sched_featureto enable the above feature so that you can compare the results with and without this feature.What we observe with this patch is that it helps for untagged interactive tasks and fairness in general, but this increases the overhead of core scheduling when there is contention for the CPU with tasks of varying cpu usage. The general trend we see is that if there is a cpu intensive thread and multiple relatively idle threads in different tags, the cpu intensive tasks continuously yields to be fair to the relatively idle threads when it becomes runnable. And if the relatively idle threads make up for most of the tasks in a system and are tagged, the cpu intensive tasks sees a considerable drop in performance.
Quoted 2 lines
If you have any feedback or creative ideas to help improve, let us know !
The data on my side looks good with CORESCHED_STALL_FIX = true.
Environment setup
--------------------------
Skylake 8170 server, 2 numa nodes, 52 cores, 104 CPUs (HT on) cgroup1 workload, sysbench (CPU mode, non AVX workload) cgroup2 workload, gemmbench (AVX512 workload)
sysbench throughput result:
.--------------------------------------------------------------------------------------------------------------------------------------.
|NA/AVX vanilla-SMT [std% / sem%] cpu% |coresched-SMT [std% / sem%] +/- cpu% | no-SMT [std% / sem%] +/- cpu% | |--------------------------------------------------------------------------------------------------------------------------------------| | 1/1 490.8 [ 0.1%/ 0.0%] 1.9% | 492.6 [ 0.1%/ 0.0%] 0.4% 1.9% | 489.5 [ 0.1%/ 0.0%] -0.3% 3.9% |
'--------------------------------------------------------------------------------------------------------------------------------------'
| 2/2 975.0 [ 0.6%/ 0.1%] 3.9% | 970.4 [ 0.4%/ 0.0%] -0.5% 3.9% | 975.6 [ 0.2%/ 0.0%] 0.1% 7.7% |
'--------------------------------------------------------------------------------------------------------------------------------------'
| 4/4 1856.9 [ 0.2%/ 0.0%] 7.8% | 1854.5 [ 0.3%/ 0.0%] -0.1% 7.8% | 1849.4 [ 0.8%/ 0.1%] -0.4% 14.8% |
'--------------------------------------------------------------------------------------------------------------------------------------'
| 8/8 3622.8 [ 0.2%/ 0.0%] 14.6% | 3618.3 [ 0.1%/ 0.0%] -0.1% 14.7% | 3626.6 [ 0.4%/ 0.0%] 0.1% 30.1% |
'--------------------------------------------------------------------------------------------------------------------------------------'
| 16/16 6976.7 [ 0.2%/ 0.0%] 30.1% | 6959.3 [ 0.3%/ 0.0%] -0.2% 30.1% | 6964.4 [ 0.9%/ 0.1%] -0.2% 60.1% |
'--------------------------------------------------------------------------------------------------------------------------------------'
| 32/32 10347.7 [ 3.8%/ 0.4%] 60.1% | 11525.4 [ 2.8%/ 0.3%] 11.4% 59.5% | 9810.5 [ 9.4%/ 0.8%] -5.2% 97.7% |
'--------------------------------------------------------------------------------------------------------------------------------------'
| 64/64 15284.9 [ 9.0%/ 0.9%] 98.1% | 17022.1 [ 4.5%/ 0.5%] 11.4% 98.2% | 9989.7 [19.3%/ 1.1%] -34.6% 100.0% |
'--------------------------------------------------------------------------------------------------------------------------------------'
|128/128 16211.3 [18.9%/ 1.9%] 100.0% | 16507.9 [ 6.1%/ 0.6%] 1.8% 99.8% | 10379.0 [12.6%/ 0.8%] -36.0% 100.0% |
'--------------------------------------------------------------------------------------------------------------------------------------'
|256/256 16667.1 [ 3.1%/ 0.3%] 100.0% | 16499.1 [ 3.2%/ 0.3%] -1.0% 100.0% | 10540.9 [16.2%/ 1.0%] -36.8% 100.0% |
'--------------------------------------------------------------------------------------------------------------------------------------'
sysbench latency result:
(The story we care about latency is that some customers reported their latency critical job is affected when co-locating a deep learning job (AVX512 task) onto the same core, because when a core executes AVX512 instructions, the core automatically reduces its frequency. This can lead to a significant overall performance loss for a non-AVX512 job on the same core.
And now we have core cookie match mechanism, so if we put AVX512 task and non AVX512 task into the different cgroups, they are supposed not to be co-located. That's why we saw the improvements of 32/32 and 64/64 cases.)
.--------------------------------------------------------------------------------------------------------------------------------------.
|NA/AVX vanilla-SMT [std% / sem%] cpu% |coresched-SMT [std% / sem%] +/- cpu% | no-SMT [std% / sem%] +/- cpu% | |--------------------------------------------------------------------------------------------------------------------------------------| | 1/1 2.1 [ 0.6%/ 0.1%] 1.9% | 2.0 [ 0.2%/ 0.0%] 3.8% 1.9% | 2.1 [ 0.7%/ 0.1%] -0.8% 3.9% |
'--------------------------------------------------------------------------------------------------------------------------------------'
| 2/2 2.1 [ 0.7%/ 0.1%] 3.9% | 2.1 [ 0.3%/ 0.0%] 0.2% 3.9% | 2.1 [ 0.6%/ 0.1%] 0.5% 7.7% |
'--------------------------------------------------------------------------------------------------------------------------------------'
| 4/4 2.2 [ 0.6%/ 0.1%] 7.8% | 2.2 [ 0.4%/ 0.0%] -0.2% 7.8% | 2.2 [ 0.2%/ 0.0%] -0.3% 14.8% |
'--------------------------------------------------------------------------------------------------------------------------------------'
| 8/8 2.2 [ 0.4%/ 0.0%] 14.6% | 2.2 [ 0.0%/ 0.0%] 0.1% 14.7% | 2.2 [ 0.0%/ 0.0%] 0.1% 30.1% |
'--------------------------------------------------------------------------------------------------------------------------------------'
| 16/16 2.4 [ 1.6%/ 0.2%] 30.1% | 2.4 [ 1.6%/ 0.2%] -0.9% 30.1% | 2.4 [ 1.9%/ 0.2%] -0.3% 60.1% |
'--------------------------------------------------------------------------------------------------------------------------------------'
| 32/32 4.9 [ 6.2%/ 0.6%] 60.1% | 3.1 [ 5.0%/ 0.5%] 36.6% 59.5% | 6.7 [17.3%/ 3.7%] -34.5% 97.7% |
'--------------------------------------------------------------------------------------------------------------------------------------'
| 64/64 9.4 [28.3%/ 2.8%] 98.1% | 3.5 [25.6%/ 2.6%] 62.4% 98.2% | 18.5 [ 9.5%/ 5.0%] -97.9% 100.0% |
'--------------------------------------------------------------------------------------------------------------------------------------'
|128/128 21.3 [10.1%/ 1.0%] 100.0% | 24.8 [ 8.1%/ 0.8%] -16.1% 99.8% | 34.5 [ 4.9%/ 0.7%] -62.0% 100.0% |
'--------------------------------------------------------------------------------------------------------------------------------------'
|256/256 35.5 [ 7.8%/ 0.8%] 100.0% | 37.3 [ 5.4%/ 0.5%] -5.1% 100.0% | 40.8 [ 5.9%/ 0.6%] -15.0% 100.0% |
'--------------------------------------------------------------------------------------------------------------------------------------'
Note:
----
64/64: 64 sysbench threads(in one cgroup) and 64 gemmbench threads(in other cgroup) run simultaneously.
Vanilla-SMT: baseline with HT on
coresched-SMT: core scheduling enabled
no-SMT: HT off thru /sys/devices/system/cpu/smt/control
std%: standard deviation
sem%: standard error of the mean
±: improvement/regression against baseline
cpu%: derived by vmstat.idle and vmstat.iowait
Quoted 1 line
The data on my side looks good with
CORESCHED_STALL_FIX= true.
Thank you for testing this fix, I'm glad it works for this use-case as well.
We will be posting another (simpler) version today, stay tuned :-)
Julien
After reading more traces and trying to understand why only untagged tasks are starving when there are cpu-intensive tasks running on the same set of CPUs, we noticed a difference in behavior in ‘pick_task’. In the case where ‘core_cookie’ is 0, we are supposed to only prefer the tagged task if it’s priority is higher, but when the priorities are equal we prefer it as well which causes the starving. ‘pick_task’ is biased toward selecting its first parameter in case of equality which in this case was the ‘class_pick’ instead of ‘max’. Reversing the order of the parameter solves this issue and matches the expected behavior.
So we can get rid of this vruntime_boost concept.
We have tested the fix below and it seems to work well with tagged/untagged tasks.
Here are our initial test results. When core scheduling is enabled, each VM (and associated vhost threads) are in their own cgroup/tag.
1 12-vcpu VM MySQL TPC-C benchmark (IO + CPU) with 96 mostly-idle 1-vcpu VMs on each NUMA node (72 logical CPUs total with SMT on):
+-------------+----------+--------------+------------+--------+ | | baseline | coresched | coresched | nosmt | | | no tag | VMs tagged | VMs tagged | no tag | | | v5.1.5 | no stall fix | stall fix | | +-------------+----------+--------------+------------+--------+ |average TPS | 1474 | 1289 | 1264 | 1339 | |stdev | 48 | 12 | 17 | 24 | |overhead | N/A | -12% | -14% | -9% | +-------------+----------+--------------+------------+--------+
3 12-vcpu VMs running linpack (cpu-intensive), all pinned on the same NUMA node (36 logical CPUs with SMT enabled on that NUMA node):
+---------------+----------+--------------+-----------+--------+ | | baseline | coresched | coresched | nosmt | | | no tag | VMs tagged | VMs tagged| no tag | | | v5.1.5 | no stall fix | stall fix | | +---------------+----------+--------------+-----------+--------+ |average gflops | 177.9 | 171.3 | 172.7 | 81.9 | |stdev | 2.6 | 10.6 | 6.4 | 8.1 | |overhead | N/A | -3.7% | -2.9% | -53.9% | +---------------+----------+--------------+-----------+--------+
This fix can be toggled dynamically with the ‘CORESCHED_STALL_FIX’ sched_feature so it’s easy to test before/after (it is disabled by default).
The up-to-date git tree can also be found here in case it’s easier to follow:
https://github.com/digitalocean/linux-coresched/commits/vpillai/coresched-v3-v5.1.5-test
Feedback welcome !
Thanks,
Julien
kernel/sched/core.c +4 -2
@@ -3668,8 +3668,10 @@ pick_task(struct rq *rq, const struct sched_class *class, struct task_struct *ma* If class_pick is tagged, return it only if it has * higher priority than max. */ -if (max && class_pick->core_cookie && -prio_less (class_pick, max)) +bool max_is_higher =sched_feat (CORESCHED_STALL_FIX ) ? + max && !prio_less (max, class_pick) : + max &&prio_less (class_pick, max); +if (class_pick->core_cookie && max_is_higher)return idle_sched_class.pick_task (rq);return class_pick;
kernel/sched/features.h +6 -0
@@ -90,3 +90,9 @@ SCHED_FEAT(WA_BIAS, true)* UtilEstimation. Use estimated CPU utilization. */ SCHED_FEAT (UTIL_EST , true) + +/* +* Prevent task stall due to vruntime comparison limitation across +* cpus. + */ +SCHED_FEAT (CORESCHED_STALL_FIX , false)
Quoted 12 lines · Subhra Mazumdar
Quoted 10 lines · Julien Desfossez
After reading more traces and trying to understand why only untagged tasks are starving when there are cpu-intensive tasks running on the same set of CPUs, we noticed a difference in behavior in ‘pick_task’. In the case where ‘core_cookie’ is 0, we are supposed to only prefer the tagged task if it’s priority is higher, but when the priorities are equal we prefer it as well which causes the starving. ‘pick_task’ is biased toward selecting its first parameter in case of equality which in this case was the ‘class_pick’ instead of ‘max’. Reversing the order of the parameter solves this issue and matches the expected behavior.
Quoted 8 lines
Quoted 5 lines
So we can get rid of this
vruntime_boostconcept.We have tested the fix below and it seems to work well with tagged/untagged tasks.
My 2 DB instance runs with this patch are better with
CORESCHED_STALL_FIXthanNO_CORESCHED_STALL_FIXin terms of performance, std deviation and idleness. May be enable it by default?
Yes if the fix is approved, we will just remove the option and it will always be enabled.
Thanks,
Julien
Quoted 10 lines · Julien Desfossez
After reading more traces and trying to understand why only untagged tasks are starving when there are cpu-intensive tasks running on the same set of CPUs, we noticed a difference in behavior in ‘pick_task’. In the case where ‘core_cookie’ is 0, we are supposed to only prefer the tagged task if it’s priority is higher, but when the priorities are equal we prefer it as well which causes the starving. ‘pick_task’ is biased toward selecting its first parameter in case of equality which in this case was the ‘class_pick’ instead of ‘max’. Reversing the order of the parameter solves this issue and matches the expected behavior.
Quoted 5 lines
So we can get rid of this
vruntime_boostconcept.We have tested the fix below and it seems to work well with tagged/untagged tasks.
My 2 DB instance runs with this patch are better with CORESCHED_STALL_FIX than NO_CORESCHED_STALL_FIX in terms of performance, std deviation and idleness. May be enable it by default?
NO_CORESCHED_STALL_FIX:
users %stdev %gain %idle
16 25 -42.4 73
24 32 -26.3 67
32 0.2 -48.9 62
CORESCHED_STALL_FIX:
users %stdev %gain %idle
16 6.5 -23 70
24 0.6 -17 60
32 1.5 -30.2 52
Quoted 19 lines · Julien Desfossez
Quoted 17 lines · Subhra Mazumdar
Quoted 15 lines · Julien Desfossez
After reading more traces and trying to understand why only untagged tasks are starving when there are cpu-intensive tasks running on the same set of CPUs, we noticed a difference in behavior in ‘pick_task’. In the case where ‘core_cookie’ is 0, we are supposed to only prefer the tagged task if it’s priority is higher, but when the priorities are equal we prefer it as well which causes the starving. ‘pick_task’ is biased toward selecting its first parameter in case of equality which in this case was the ‘class_pick’ instead of ‘max’. Reversing the order of the parameter solves this issue and matches the expected behavior.
So we can get rid of this
vruntime_boostconcept.We have tested the fix below and it seems to work well with tagged/untagged tasks.
Quoted 7 lines
Quoted 3 lines
My 2 DB instance runs with this patch are better with
CORESCHED_STALL_FIXthanNO_CORESCHED_STALL_FIXin terms of performance, std deviation and idleness. May be enable it by default?Yes if the fix is approved, we will just remove the option and it will always be enabled.
sysbench --report-interval option unveiled something.
benchmark setup
-------------------------
two cgroups, cpuset.cpus = 1, 53(one core, two siblings) sysbench cpu mode, one thread in cgroup1 sysbench memory mode, one thread in cgroup2
no core scheduling
--------------------------
cpu throughput eps: 405.8, std: 0.14% mem bandwidth MB/s: 5785.7, std: 0.11%
cgroup1 enable core scheduling(cpu mode) cgroup2 disable core scheduling(memory mode)
-----------------------------------------------------------------
cpu throughput eps: 8.7, std: 519.2%
mem bandwidth MB/s: 6263.2, std: 9.3%
cgroup1 disable core scheduling(cpu mode) cgroup2 enable core scheduling(memory mode)
-----------------------------------------------------------------
cpu throughput eps: 468.0 , std: 8.7% mem bandwidth MB/S: 311.6 , std: 169.1%
cgroup1 enable core scheduling(cpu mode) cgroup2 enable core scheduling(memory mode)
----------------------------------------------------------------
cpu throughput eps: 76.4 , std: 168.0% mem bandwidth MB/S: 5388.3 , std: 30.9%
The result looks still unfair, and particularly, the variance is too high, ----sysbench cpu log ----
----snip----
[ 10s ] thds: 1 eps: 296.00 lat (ms,95%): 2.03
[ 11s ] thds: 1 eps: 0.00 lat (ms,95%): 1170.65
[ 12s ] thds: 1 eps: 1.00 lat (ms,95%): 0.00
[ 13s ] thds: 1 eps: 0.00 lat (ms,95%): 0.00
[ 14s ] thds: 1 eps: 295.91 lat (ms,95%): 2.03
[ 15s ] thds: 1 eps: 1.00 lat (ms,95%): 170.48
[ 16s ] thds: 1 eps: 0.00 lat (ms,95%): 2009.23
[ 17s ] thds: 1 eps: 1.00 lat (ms,95%): 995.51
[ 18s ] thds: 1 eps: 296.00 lat (ms,95%): 2.03
[ 19s ] thds: 1 eps: 1.00 lat (ms,95%): 170.48
[ 20s ] thds: 1 eps: 0.00 lat (ms,95%): 2009.23
----snip----
Thanks,
-Aubrey
Quoted 1 line · Aubrey Li
The result looks still unfair, and particularly, the variance is too high,
I just want to confirm that I am also seeing the same issue with a similar setup. I also tried with the priority boost fix we previously posted, the results are slightly better, but we are still seeing a very high variance.
On average, the results I get for 10 30-seconds runs are still much better than nosmt (both sysbench pinned on the same sibling) for the memory benchmark, and pretty similar for the CPU benchmark, but the high variance between runs is indeed concerning.
Still digging :-)
Thanks,
Julien
Quoted 8 lines · Julien Desfossez
Quoted 1 line · Aubrey Li
The result looks still unfair, and particularly, the variance is too high,
I just want to confirm that I am also seeing the same issue with a similar setup. I also tried with the priority boost fix we previously posted, the results are slightly better, but we are still seeing a very high variance.
Quoted 4 lines
On average, the results I get for 10 30-seconds runs are still much better than nosmt (both sysbench pinned on the same sibling) for the memory benchmark, and pretty similar for the CPU benchmark, but the high variance between runs is indeed concerning.
I was thinking to use util_avg signal to decide which task win in __prio_less() in the cross cpu case. The reason util_avg is chosen is because it represents how cpu intensive the task is, so the end result is, less cpu intensive task will preempt more cpu intensive task.
Here is the test I have done to see how util_avg works
(on a single node, 16 cores, 32 cpus vm):
1 Start tmux and then start 3 windows with each running bash; 2 Place two shells into two different cgroups and both have cpu.tag set; 3 Switch to the 1st tmux window, start
will-it-scale/page_fault1_processes -t 16 -s 30
in the first tagged shell;
4 Switch to the 2nd tmux window;
5 Start
will-it-scale/page_fault1_processes -t 16 -s 30
in the 2nd tagged shell;
6 Switch to the 3rd tmux window;
7 Do some simple things in the 3rd untagged shell like ls to see if
untagged task is able to proceed;
8 Wait for the two page_fault workloads to finish.
With v3 here, I can not do step 4 and later steps, i.e. the 16 page_fault1 processes started in step 3 will occupy all 16 cores and other tasks do not have a chance to run, including tmux, which made switching tmux window impossible.
With the below patch on top of v3 that makes use of util_avg to decide which task win, I can do all 8 steps and the final scores of the 2 workloads are: 1796191 and 2199586. The score number are not close, suggesting some unfairness, but I can finish the test now...
Here is the diff(consider it as a POC):
--- kernel/sched/core.c | 35 ++--------------------------------- kernel/sched/fair.c | 36 ++++++++++++++++++++++++++++++++++++ kernel/sched/sched.h | 2 ++ 3 files changed, 40 insertions(+), 33 deletions(-)
kernel/sched/core.c +2 -33
@@ -105,25 +105,8 @@ static inline bool prio_less(struct task_struct *a, struct task_struct *b)if (pa ==-1 )/* dl_prio() doesn't work because of stop_class above */ return !dl_time_before (a->dl.deadline, b->dl.deadline); -if (pa ==MAX_RT_PRIO +MAX_NICE ) {/* fair */ -u64 a_vruntime = a->se.vruntime; -u64 b_vruntime = b->se.vruntime; - -/* -* Normalize the vruntime if tasks are in different cpus. -*/ -if (task_cpu (a) !=task_cpu (b)) { - b_vruntime -=task_cfs_rq (b)->min_vruntime; - b_vruntime +=task_cfs_rq (a)->min_vruntime; - -trace_printk ("(%d:%Lu,%Lu,%Lu) <> (%d:%Lu,%Lu,%Lu)\n" , - a->pid, a_vruntime, a->se.vruntime,task_cfs_rq (a)->min_vruntime, - b->pid, b_vruntime, b->se.vruntime,task_cfs_rq (b)->min_vruntime); - - } - -return !((s64)(a_vruntime - b_vruntime) <=0 ); - } +if (pa ==MAX_RT_PRIO +MAX_NICE )/* fair */ +return cfs_prio_less (a, b);return false; }
@@ -3663,20 +3646,6 @@ pick_task(struct rq *rq, const struct sched_class *class, struct task_struct *maif (!class_pick)return NULL ; -if (!cookie) { -/* -* If class_pick is tagged, return it only if it has -* higher priority than max. -*/ -bool max_is_higher =sched_feat (CORESCHED_STALL_FIX ) ? - max && !prio_less (max, class_pick) : - max &&prio_less (class_pick, max); -if (class_pick->core_cookie && max_is_higher) -return idle_sched_class.pick_task (rq); - -return class_pick; - } -/* * If class_pick is idle or matches cookie, return early. */
kernel/sched/fair.c +36 -0
@@ -10740,3 +10740,39 @@ __init void init_sched_fair_class(void)#endif /* SMP */ } + +bool cfs_prio_less (struct task_struct *a,struct task_struct *b) +{ +struct sched_entity *sea = &a->se; +struct sched_entity *seb = &b->se; +bool samecore =task_cpu (a) ==task_cpu (b); +struct task_struct *p; +s64 delta; + +if (samecore) { +/* vruntime is per cfs_rq */ +while (!is_same_group (sea, seb)) { +int sea_depth = sea->depth; +int seb_depth = seb->depth; + +if (sea_depth >= seb_depth) + sea =parent_entity (sea); +if (sea_depth <= seb_depth) + seb =parent_entity (seb); + } + + delta = (s64)(sea->vruntime - seb->vruntime); + } + +/* across cpu: use util_avg to decide which task to run */ + delta = (s64)(sea->avg.util_avg - seb->avg.util_avg); + + p = delta >0 ? b : a; +trace_printk ("picked %s/%d %s: %Lu %Lu %Ld\n" , p->comm, p->pid, + samecore ?"vruntime" :"util_avg" , + samecore ? sea->vruntime : sea->avg.util_avg, + samecore ? seb->vruntime : seb->avg.util_avg, + delta); + +return delta >0 ; +}
kernel/sched/sched.h +2 -0
@@ -2454,3 +2454,5 @@ static inline bool sched_energy_enabled(void)static inline bool sched_energy_enabled (void ) {return false; }#endif /* CONFIG_ENERGY_MODEL && CONFIG_CPU_FREQ_GOV_SCHEDUTIL */ + +bool cfs_prio_less (struct task_struct *a,struct task_struct *b);
Signature · 1 line
2.19.1.3.ge56e4f7
Quoted 7 lines · Aaron Lu
On Wed, Jun 19, 2019 at 02:33:02PM -0400, Julien Desfossez wrote:
With the below patch on top of v3 that makes use of
util_avgto decide which task win, I can do all 8 steps and the final scores of the 2 workloads are: 1796191 and 2199586. The score number are not close, suggesting some unfairness, but I can finish the test now...
Aaron,
Do you still see high variance in terms of workload throughput that was a problem with the previous version?
kernel/sched/fair.c __init void init_sched_fair_class(void) · line 10748
Quoted 9 lines
}
kernel/sched/fair.c +6
-0+ +bool cfs_prio_less (struct task_struct *a,struct task_struct *b) +{ +struct sched_entity *sea = &a->se; +struct sched_entity *seb = &b->se; +bool samecore =task_cpu (a) ==task_cpu (b);
Probably "samecpu" instead of "samecore" will be more accurate. I think task_cpu(a) and task_cpu(b)
can be different, but still belong to the same cpu core.
kernel/sched/fair.c __init void init_sched_fair_class(void) · line 10758
Quoted 10 lines
kernel/sched/fair.c +10
-0+struct task_struct *p; +s64 delta; + +if (samecore) { +/* vruntime is per cfs_rq */ +while (!is_same_group (sea, seb)) { +int sea_depth = sea->depth; +int seb_depth = seb->depth; + +if (sea_depth >= seb_depth)
Should this be strictly ">" instead of ">=" ?
kernel/sched/fair.c __init void init_sched_fair_class(void) · line 10760
Quoted 2 lines
kernel/sched/fair.c +2
-0+ sea =parent_entity (sea); +if (sea_depth <= seb_depth)
Should use "<" ?
kernel/sched/fair.c __init void init_sched_fair_class(void) · line 10766
Quoted 6 lines
kernel/sched/fair.c +6
-0+ seb =parent_entity (seb); + } + + delta = (s64)(sea->vruntime - seb->vruntime); + } +
Thanks.
Tim
Quoted 11 lines · Tim Chen
Quoted 7 lines · Aaron Lu
On Wed, Jun 19, 2019 at 02:33:02PM -0400, Julien Desfossez wrote:
With the below patch on top of v3 that makes use of
util_avgto decide which task win, I can do all 8 steps and the final scores of the 2 workloads are: 1796191 and 2199586. The score number are not close, suggesting some unfairness, but I can finish the test now...
Quoted 4 lines
Aaron,
Do you still see high variance in terms of workload throughput that was a problem with the previous version?
Any suggestion how to measure this?
It's not clear how Aubrey did his test, will need to take a look at sysbench.
Quoted 14 lines
kernel/sched/fair.c __init void init_sched_fair_class(void) · line 10748
Quoted 9 lines
}
kernel/sched/fair.c +6
-0+ +bool cfs_prio_less (struct task_struct *a,struct task_struct *b) +{ +struct sched_entity *sea = &a->se; +struct sched_entity *seb = &b->se; +bool samecore =task_cpu (a) ==task_cpu (b);Probably "samecpu" instead of "samecore" will be more accurate. I think
task_cpu(a)andtask_cpu(b)
can be different, but still belong to the same cpu core.
Right, definitely, guess I'm brain damaged.
Quoted 13 lines
kernel/sched/fair.c __init void init_sched_fair_class(void) · line 10758
Quoted 11 lines
kernel/sched/fair.c +10
-0+struct task_struct *p; +s64 delta; + +if (samecore) { +/* vruntime is per cfs_rq */ +while (!is_same_group (sea, seb)) { +int sea_depth = sea->depth; +int seb_depth = seb->depth; + +if (sea_depth >= seb_depth)Should this be strictly ">" instead of ">=" ?
Same depth doesn't necessarily mean same group while the purpose here is to make sure they are in the same cfs_rq. When they are of the same depth but in different cfs_rqs, we will continue to go up till we reach rq->cfs.
Quoted 5 lines
kernel/sched/fair.c __init void init_sched_fair_class(void) · line 10760
Quoted 3 lines
kernel/sched/fair.c +2
-0+ sea =parent_entity (sea); +if (sea_depth <= seb_depth)Should use "<" ?
Ditto here.
When they are of the same depth but no in the same cfs_rq, both se will move up.
Quoted 6 lines
kernel/sched/fair.c __init void init_sched_fair_class(void) · line 10766
Quoted 6 lines
kernel/sched/fair.c +6
-0+ seb =parent_entity (seb); + } + + delta = (s64)(sea->vruntime - seb->vruntime); + } +
Thanks.
Quoted 13 lines · Aaron Lu
Quoted 11 lines · Tim Chen
Quoted 7 lines · Aaron Lu
On Wed, Jun 19, 2019 at 02:33:02PM -0400, Julien Desfossez wrote:
With the below patch on top of v3 that makes use of
util_avgto decide which task win, I can do all 8 steps and the final scores of the 2 workloads are: 1796191 and 2199586. The score number are not close, suggesting some unfairness, but I can finish the test now...
Quoted 9 lines
Quoted 4 lines
Aaron,
Do you still see high variance in terms of workload throughput that was a problem with the previous version?
Any suggestion how to measure this?
It's not clear how Aubrey did his test, will need to take a look at sysbench.
Well, thanks to post this at the end of my vacation, ;) I'll go back to the office next week and give a shot. I actually have a new setup of co-locating AVX512 tasks with sysbench MYSQL. Both throughput and latency was unacceptable on the top of V3, Looking forward to seeing the difference of patch.
Thanks,
-Aubrey
Quoted 52 lines · Aaron Lu
Quoted 11 lines · Tim Chen
Quoted 7 lines · Aaron Lu
On Wed, Jun 19, 2019 at 02:33:02PM -0400, Julien Desfossez wrote:
With the below patch on top of v3 that makes use of
util_avgto decide which task win, I can do all 8 steps and the final scores of the 2 workloads are: 1796191 and 2199586. The score number are not close, suggesting some unfairness, but I can finish the test now...Quoted 4 lines
Aaron,
Do you still see high variance in terms of workload throughput that was a problem with the previous version?
Any suggestion how to measure this?
It's not clear how Aubrey did his test, will need to take a look at sysbench.Quoted 14 lines
kernel/sched/fair.c __init void init_sched_fair_class(void) · line 10748
Quoted 9 lines
}
kernel/sched/fair.c +6
-0+ +bool cfs_prio_less (struct task_struct *a,struct task_struct *b) +{ +struct sched_entity *sea = &a->se; +struct sched_entity *seb = &b->se; +bool samecore =task_cpu (a) ==task_cpu (b);Probably "samecpu" instead of "samecore" will be more accurate. I think
task_cpu(a)andtask_cpu(b)
can be different, but still belong to the same cpu core.Right, definitely, guess I'm brain damaged.
Quoted 13 lines
kernel/sched/fair.c __init void init_sched_fair_class(void) · line 10758
Quoted 11 lines
kernel/sched/fair.c +10
-0+struct task_struct *p; +s64 delta; + +if (samecore) { +/* vruntime is per cfs_rq */ +while (!is_same_group (sea, seb)) { +int sea_depth = sea->depth; +int seb_depth = seb->depth; + +if (sea_depth >= seb_depth)Should this be strictly ">" instead of ">=" ?
Quoted 4 lines
Same depth doesn't necessarily mean same group while the purpose here is to make sure they are in the same
cfs_rq. When they are of the same depth but in differentcfs_rqs, we will continue to go up till we reachrq->cfs.
Ah, I see what you are doing now. Thanks for the clarification.
Tim
Quoted 132 lines · Aaron Lu
Quoted 8 lines · Julien Desfossez
Quoted 1 line · Aubrey Li
The result looks still unfair, and particularly, the variance is too high,
I just want to confirm that I am also seeing the same issue with a similar setup. I also tried with the priority boost fix we previously posted, the results are slightly better, but we are still seeing a very high variance.
Quoted 4 lines
On average, the results I get for 10 30-seconds runs are still much better than nosmt (both sysbench pinned on the same sibling) for the memory benchmark, and pretty similar for the CPU benchmark, but the high variance between runs is indeed concerning.
I was thinking to use
util_avgsignal to decide which task win in__prio_less()in the cross cpu case. The reasonutil_avgis chosen is because it represents how cpu intensive the task is, so the end result is, less cpu intensive task will preempt more cpu intensive task.Here is the test I have done to see how
util_avgworks
(on a single node, 16 cores, 32 cpus vm):
1 Start tmux and then start 3 windows with each running bash; 2 Place two shells into two different cgroups and both have cpu.tag set; 3 Switch to the 1st tmux window, start
will-it-scale/page_fault1_processes -t 16 -s 30
in the first tagged shell;
4 Switch to the 2nd tmux window;
5 Start
will-it-scale/page_fault1_processes -t 16 -s 30
in the 2nd tagged shell;
6 Switch to the 3rd tmux window;
7 Do some simple things in the 3rd untagged shell like ls to see if
untagged task is able to proceed;
8 Wait for the twopage_faultworkloads to finish.With v3 here, I can not do step 4 and later steps, i.e. the 16
page_fault1processes started in step 3 will occupy all 16 cores and other tasks do not have a chance to run, including tmux, which made switching tmux window impossible.With the below patch on top of v3 that makes use of
util_avgto decide which task win, I can do all 8 steps and the final scores of the 2 workloads are: 1796191 and 2199586. The score number are not close, suggesting some unfairness, but I can finish the test now...Here is the diff(consider it as a POC):
--- kernel/sched/core.c | 35 ++--------------------------------- kernel/sched/fair.c | 36 ++++++++++++++++++++++++++++++++++++ kernel/sched/sched.h | 2 ++ 3 files changed, 40 insertions(+), 33 deletions(-)kernel/sched/core.c +2
-33@@ -105,25 +105,8 @@ static inline bool prio_less(struct task_struct *a, struct task_struct *b)if (pa ==-1 )/* dl_prio() doesn't work because of stop_class above */ return !dl_time_before (a->dl.deadline, b->dl.deadline); -if (pa ==MAX_RT_PRIO +MAX_NICE ) {/* fair */ -u64 a_vruntime = a->se.vruntime; -u64 b_vruntime = b->se.vruntime; - -/* -* Normalize the vruntime if tasks are in different cpus. -*/ -if (task_cpu (a) !=task_cpu (b)) { - b_vruntime -=task_cfs_rq (b)->min_vruntime; - b_vruntime +=task_cfs_rq (a)->min_vruntime; - -trace_printk ("(%d:%Lu,%Lu,%Lu) <> (%d:%Lu,%Lu,%Lu)\n" , - a->pid, a_vruntime, a->se.vruntime,task_cfs_rq (a)->min_vruntime, - b->pid, b_vruntime, b->se.vruntime,task_cfs_rq (b)->min_vruntime); - - } - -return !((s64)(a_vruntime - b_vruntime) <=0 ); - } +if (pa ==MAX_RT_PRIO +MAX_NICE )/* fair */ +return cfs_prio_less (a, b);return false; }@@ -3663,20 +3646,6 @@ pick_task(struct rq *rq, const struct sched_class *class, struct task_struct *maif (!class_pick)return NULL ; -if (!cookie) { -/* -* If class_pick is tagged, return it only if it has -* higher priority than max. -*/ -bool max_is_higher =sched_feat (CORESCHED_STALL_FIX ) ? - max && !prio_less (max, class_pick) : - max &&prio_less (class_pick, max); -if (class_pick->core_cookie && max_is_higher) -return idle_sched_class.pick_task (rq); - -return class_pick; - } -/* * If class_pick is idle or matches cookie, return early. */ kernel/sched/fair.c +16
-0@@ -10740,3 +10740,39 @@ __init void init_sched_fair_class(void)#endif /* SMP */ } + +bool cfs_prio_less (struct task_struct *a,struct task_struct *b) +{ +struct sched_entity *sea = &a->se; +struct sched_entity *seb = &b->se; +bool samecore =task_cpu (a) ==task_cpu (b); +struct task_struct *p; +s64 delta; + +if (samecore) { +/* vruntime is per cfs_rq */ +while (!is_same_group (sea, seb)) { +int sea_depth = sea->depth; +int seb_depth = seb->depth; + +if (sea_depth >= seb_depth)
kernel/sched/fair.c __init void init_sched_fair_class(void) · line 10768
Quoted 10 lines
kernel/sched/fair.c +10
-0+ sea =parent_entity (sea); +if (sea_depth <= seb_depth) + seb =parent_entity (seb); + } + + delta = (s64)(sea->vruntime - seb->vruntime); + } + +/* across cpu: use util_avg to decide which task to run */ + delta = (s64)(sea->avg.util_avg - seb->avg.util_avg);
The granularity period of util_avg seems too large to decide task priority during pick_task(), at least it is in my case, cfs_prio_less() always picked core max task, so pick_task() eventually picked idle, which causes this change not very helpful for my case.
<idle>-0 [057] dN.. 83.716973: __schedule: max: sysbench/2578ffff889050f68600
<idle>-0 [057] dN.. 83.716974: __schedule:
(swapper/5/0;140,0,0) ?< (mysqld/2511;119,1042118143,0)
<idle>-0 [057] dN.. 83.716975: __schedule:
(sysbench/2578;119,96449836,0) ?< (mysqld/2511;119,1042118143,0)
<idle>-0 [057] dN.. 83.716975: cfs_prio_less: picked
sysbench/2578 util_avg: 20 527 -507 <======= here===
<idle>-0 [057] dN.. 83.716976: __schedule: pick_task cookie
pick swapper/5/0 ffff889050f68600
Thanks,
-Aubrey
Quoted 5 lines · Aubrey Li
The granularity period of
util_avgseems too large to decide task priority duringpick_task(), at least it is in my case,cfs_prio_less()always picked core max task, sopick_task()eventually picked idle, which causes this change not very helpful for my case.
Quoted 10 lines
<idle>-0 [057] dN.. 83.716973:
__schedule: max: sysbench/2578ffff889050f68600
<idle>-0 [057] dN.. 83.716974:__schedule:
(swapper/5/0;140,0,0) ?< (mysqld/2511;119,1042118143,0)
<idle>-0 [057] dN.. 83.716975:__schedule:
(sysbench/2578;119,96449836,0) ?< (mysqld/2511;119,1042118143,0)
<idle>-0 [057] dN.. 83.716975:cfs_prio_less: picked
sysbench/2578util_avg: 20 527 -507 <======= here===
<idle>-0 [057] dN.. 83.716976:__schedule:pick_taskcookie
pick swapper/5/0ffff889050f68600
Can you share your setup of the test? I would like to try it locally. Thanks.
Quoted 18 lines · Aaron Lu
Quoted 5 lines · Aubrey Li
The granularity period of
util_avgseems too large to decide task priority duringpick_task(), at least it is in my case,cfs_prio_less()always picked core max task, sopick_task()eventually picked idle, which causes this change not very helpful for my case.Quoted 10 lines
<idle>-0 [057] dN.. 83.716973:
__schedule: max: sysbench/2578ffff889050f68600
<idle>-0 [057] dN.. 83.716974:__schedule:
(swapper/5/0;140,0,0) ?< (mysqld/2511;119,1042118143,0)
<idle>-0 [057] dN.. 83.716975:__schedule:
(sysbench/2578;119,96449836,0) ?< (mysqld/2511;119,1042118143,0)
<idle>-0 [057] dN.. 83.716975:cfs_prio_less: picked
sysbench/2578util_avg: 20 527 -507 <======= here===
<idle>-0 [057] dN.. 83.716976:__schedule:pick_taskcookie
pick swapper/5/0ffff889050f68600
Quoted 1 line
Can you share your setup of the test? I would like to try it locally.
My setup is a co-location of AVX512 tasks(gemmbench) and non-AVX512 tasks (sysbench MYSQL). Let me simply it and send offline.
Thanks,
-Aubrey
Quoted 5 lines · Aubrey Li
The granularity period of
util_avgseems too large to decide task priority duringpick_task(), at least it is in my case,cfs_prio_less()always picked core max task, sopick_task()eventually picked idle, which causes this change not very helpful for my case.
Quoted 10 lines
<idle>-0 [057] dN.. 83.716973:
__schedule: max: sysbench/2578ffff889050f68600
<idle>-0 [057] dN.. 83.716974:__schedule:
(swapper/5/0;140,0,0) ?< (mysqld/2511;119,1042118143,0)
<idle>-0 [057] dN.. 83.716975:__schedule:
(sysbench/2578;119,96449836,0) ?< (mysqld/2511;119,1042118143,0)
<idle>-0 [057] dN.. 83.716975:cfs_prio_less: picked
sysbench/2578util_avg: 20 527 -507 <======= here===
<idle>-0 [057] dN.. 83.716976:__schedule:pick_taskcookie
pick swapper/5/0ffff889050f68600
I tried a different approach based on vruntime with 3 patches following.
When the two tasks are on the same CPU, no change is made, I still route the two sched entities up till they are in the same group(cfs_rq) and then do the vruntime comparison.
When the two tasks are on differen threads of the same core, the root level sched_entities to which the two tasks belong will be used to do the comparison.
An ugly illustration for the cross CPU case:
cpu0 cpu1
/ | \ / | \
se1 se2 se3 se4 se5 se6
/ \ / \
se21 se22 se61 se62
Assume CPU0 and CPU1 are smt siblings and task A's se is se21 while task B's se is se61. To compare priority of task A and B, we compare priority of se2 and se6. The smaller vruntime wins.
To make this work, the root level ses on both CPU should have a common cfs_rq min vuntime, which I call it the core cfs_rq min vruntime.
This is mostly done in patch2/3.
Test:
1 wrote an cpu intensive program that does nothing but while(1) in
main(), let's call it cpuhog;
2 start 2 cgroups, with one cgroup's cpuset binding to CPU2 and the
other binding to cpu3. cpu2 and cpu3 are smt siblings on the test VM;
3 enable cpu.tag for the two cgroups;
4 start one cpuhog task in each cgroup;
5 kill both cpuhog tasks after 10 seconds;
6 check each cgroup's cpu usage.
If the task is scheduled fairly, then each cgroup's cpu usage should be around 5s.
With v3, the cpu usage of both cgroups are sometimes 3s, 7s; sometimes 1s, 9s.
With the 3 patches applied, the numbers are mostly around 5s, 5s.
Another test is starting two cgroups simultaneously with cpu.tag set, with one cgroup running: will-it-scale/page_fault1_processes -t 16 -s 30, the other running: will-it-scale/page_fault2_processes -t 16 -s 30. With v3, like I said last time, the later started page_fault processes can't start running. With the 3 patches applied, both running at the same time with each CPU having a relatively fair score:
output line of 16 page_fault1 processes in 1 second interval:
min:105225 max:131716 total:1872322
output line of 16 page_fault2 processes in 1 second interval:
min:86797 max:110554 total:1581177
Note the value in min and max, the smaller the gap is, the better the faireness is.
Aubrey,
I haven't been able to run your workload yet...
RFC PATCH 1/3 wrapper for cfs_rq->min_vruntime
Add a wrapper function cfs_rq_min_vruntime(cfs_rq) to return cfs_rq->min_vruntime.
It will be used in the following patch, no functionality change.
--- kernel/sched/fair.c | 27 ++++++++++++++++----------- 1 file changed, 16 insertions(+), 11 deletions(-)
kernel/sched/fair.c +16 -11
@@ -431,6 +431,11 @@ find_matching_se(struct sched_entity **se, struct sched_entity **pse)#endif /* CONFIG_FAIR_GROUP_SCHED */ +static inline u64 cfs_rq_min_vruntime (struct cfs_rq *cfs_rq) +{ +return cfs_rq->min_vruntime; +} +static __always_inline void account_cfs_rq_runtime(struct cfs_rq *cfs_rq, u64 delta_exec);
@@ -467,7 +472,7 @@ static void update_min_vruntime(struct cfs_rq *cfs_rq)struct sched_entity *curr = cfs_rq->curr;struct rb_node *leftmost =rb_first_cached (&cfs_rq->tasks_timeline); -u64 vruntime = cfs_rq->min_vruntime; +u64 vruntime =cfs_rq_min_vruntime (cfs_rq);if (curr) {if (curr->on_rq)
@@ -487,7 +492,7 @@ static void update_min_vruntime(struct cfs_rq *cfs_rq) }/* ensure we never gain time by being placed backwards. */ - cfs_rq->min_vruntime =max_vruntime (cfs_rq->min_vruntime, vruntime); + cfs_rq->min_vruntime =max_vruntime (cfs_rq_min_vruntime (cfs_rq), vruntime);#ifndef CONFIG_64BIT smp_wmb (); cfs_rq->min_vruntime_copy = cfs_rq->min_vruntime;
@@ -3742,7 +3747,7 @@ static inline void update_misfit_status(struct task_struct *p, struct rq *rq) {}static void check_spread (struct cfs_rq *cfs_rq,struct sched_entity *se) {#ifdef CONFIG_SCHED_DEBUG -s64 d = se->vruntime - cfs_rq->min_vruntime; +s64 d = se->vruntime -cfs_rq_min_vruntime (cfs_rq);if (d <0 ) d = -d;
@@ -3755,7 +3760,7 @@ static void check_spread(struct cfs_rq *cfs_rq, struct sched_entity *se)static void place_entity (struct cfs_rq *cfs_rq,struct sched_entity *se,int initial) { -u64 vruntime = cfs_rq->min_vruntime; +u64 vruntime =cfs_rq_min_vruntime (cfs_rq);/* * The 'current' period is already promised to the current tasks,
@@ -3848,7 +3853,7 @@ enqueue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)* update_curr(). */ if (renorm && curr) - se->vruntime += cfs_rq->min_vruntime; + se->vruntime +=cfs_rq_min_vruntime (cfs_rq);update_curr (cfs_rq);
@@ -3859,7 +3864,7 @@ enqueue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)* fairness detriment of existing tasks. */ if (renorm && !curr) - se->vruntime += cfs_rq->min_vruntime; + se->vruntime +=cfs_rq_min_vruntime (cfs_rq);/* * When enqueuing a sched_entity, we must:
@@ -3972,7 +3977,7 @@ dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)* can move min_vruntime forward still more. */ if (!(flags &DEQUEUE_SLEEP )) - se->vruntime -= cfs_rq->min_vruntime; + se->vruntime -=cfs_rq_min_vruntime (cfs_rq);/* return excess runtime on last dequeue */ return_cfs_rq_runtime (cfs_rq);
@@ -6722,7 +6727,7 @@ static void migrate_task_rq_fair(struct task_struct *p, int new_cpu) min_vruntime = cfs_rq->min_vruntime; }while (min_vruntime != min_vruntime_copy);#else - min_vruntime = cfs_rq->min_vruntime; + min_vruntime =cfs_rq_min_vruntime (cfs_rq);#endif se->vruntime -= min_vruntime;
@@ -10215,7 +10220,7 @@ static void task_fork_fair(struct task_struct *p)resched_curr (rq); } - se->vruntime -= cfs_rq->min_vruntime; + se->vruntime -=cfs_rq_min_vruntime (cfs_rq);rq_unlock (rq, &rf); }
@@ -10335,7 +10340,7 @@ static void detach_task_cfs_rq(struct task_struct *p)* cause 'unlimited' sleep bonus. */ place_entity (cfs_rq, se,0 ); - se->vruntime -= cfs_rq->min_vruntime; + se->vruntime -=cfs_rq_min_vruntime (cfs_rq); }detach_entity_cfs_rq (se);
@@ -10349,7 +10354,7 @@ static void attach_task_cfs_rq(struct task_struct *p)attach_entity_cfs_rq (se);if (!vruntime_normalized (p)) - se->vruntime += cfs_rq->min_vruntime; + se->vruntime +=cfs_rq_min_vruntime (cfs_rq); }static void switched_from_fair (struct rq *rq,struct task_struct *p)
Signature · 1 line
2.19.1.3.ge56e4f7
PATCH 2/3 core vruntime comparison
In reply to Aaron LuThis patch provides a vruntime based way to compare two cfs task's priority, be it on the same cpu or different threads of the same core.
When the two tasks are on the same CPU, we just need to find a common cfs_rq both sched_entities are on and then do the comparison.
When the two tasks are on differen threads of the same core, the root level sched_entities to which the two tasks belong will be used to do the comparison.
An ugly illustration for the cross CPU case:
cpu0 cpu1
/ | \ / | \
se1 se2 se3 se4 se5 se6
/ \ / \
se21 se22 se61 se62
Assume CPU0 and CPU1 are smt siblings and task A's se is se21 while task B's se is se61. To compare priority of task A and B, we compare priority of se2 and se6. Whose vruntime is smaller, who wins.
To make this work, the root level se should have a common cfs_rq min vuntime, which I call it the core cfs_rq min vruntime.
Potential issues: when core scheduling is enabled, if there are tasks already in some CPU's rq, then new tasks will be queued with the per-core cfs_rq min vruntime while the old tasks are using the original root level cfs_rq's min_vruntime. The two values can differ greatly and can cause tasks with a large vruntime starve. So enable core scheduling early when the system is still kind of idle for the time being to avoid this problem.
--- kernel/sched/core.c | 15 ++------- kernel/sched/fair.c | 79 +++++++++++++++++++++++++++++++++++++++++++- kernel/sched/sched.h | 2 ++ 3 files changed, 82 insertions(+), 14 deletions(-)
kernel/sched/core.c +2 -13
@@ -105,19 +105,8 @@ static inline bool prio_less(struct task_struct *a, struct task_struct *b)if (pa ==-1 )/* dl_prio() doesn't work because of stop_class above */ return !dl_time_before (a->dl.deadline, b->dl.deadline); -if (pa ==MAX_RT_PRIO +MAX_NICE ) {/* fair */ -u64 vruntime = b->se.vruntime; - -/* -* Normalize the vruntime if tasks are in different cpus. -*/ -if (task_cpu (a) !=task_cpu (b)) { - vruntime -=task_cfs_rq (b)->min_vruntime; - vruntime +=task_cfs_rq (a)->min_vruntime; - } - -return !((s64)(a->se.vruntime - vruntime) <=0 ); - } +if (pa ==MAX_RT_PRIO +MAX_NICE )/* fair */ +return cfs_prio_less (a, b);return false; }
kernel/sched/fair.c +78 -1
93 lines not on this page · raw message
kernel/sched/sched.h +2 -0
@@ -2454,3 +2454,5 @@ static inline bool sched_energy_enabled(void)static inline bool sched_energy_enabled (void ) {return false; }#endif /* CONFIG_ENERGY_MODEL && CONFIG_CPU_FREQ_GOV_SCHEDUTIL */ + +bool cfs_prio_less (struct task_struct *a,struct task_struct *b);
Signature · 1 line
2.19.1.3.ge56e4f7
PATCH 3/3 temp hack to make tick based schedule happen
In reply to Aaron LuWhen a hyperthread is forced idle and the other hyperthread has a single CPU intensive task running, the running task can occupy the hyperthread for a long time with no scheduling point and starve the other hyperthread.
Fix this temporarily by always checking if the task has exceed its timeslice and if so, do a schedule.
--- kernel/sched/fair.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
kernel/sched/fair.c +4 -2
@@ -4093,6 +4093,9 @@ check_preempt_tick(struct cfs_rq *cfs_rq, struct sched_entity *curr)return ; } +if (cfs_rq->nr_running <=1 ) +return ; +/* * Ensure that a task that missed wakeup preemption by a * narrow margin doesn't have to wait for a full slice.
@@ -4261,8 +4264,7 @@ entity_tick(struct cfs_rq *cfs_rq, struct sched_entity *curr, int queued)return ;#endif -if (cfs_rq->nr_running >1 ) -check_preempt_tick (cfs_rq, curr); +check_preempt_tick (cfs_rq, curr); }
Signature · 1 line
2.19.1.3.ge56e4f7
Quoted 75 lines · Aaron Lu
Quoted 5 lines · Aubrey Li
The granularity period of
util_avgseems too large to decide task priority duringpick_task(), at least it is in my case,cfs_prio_less()always picked core max task, sopick_task()eventually picked idle, which causes this change not very helpful for my case.Quoted 10 lines
<idle>-0 [057] dN.. 83.716973:
__schedule: max: sysbench/2578ffff889050f68600
<idle>-0 [057] dN.. 83.716974:__schedule:
(swapper/5/0;140,0,0) ?< (mysqld/2511;119,1042118143,0)
<idle>-0 [057] dN.. 83.716975:__schedule:
(sysbench/2578;119,96449836,0) ?< (mysqld/2511;119,1042118143,0)
<idle>-0 [057] dN.. 83.716975:cfs_prio_less: picked
sysbench/2578util_avg: 20 527 -507 <======= here===
<idle>-0 [057] dN.. 83.716976:__schedule:pick_taskcookie
pick swapper/5/0ffff889050f68600I tried a different approach based on vruntime with 3 patches following.
When the two tasks are on the same CPU, no change is made, I still route the two sched entities up till they are in the same
group(cfs_rq)and then do the vruntime comparison.When the two tasks are on differen threads of the same core, the root level
sched_entitiesto which the two tasks belong will be used to do the comparison.An ugly illustration for the cross CPU case:
cpu0 cpu1
/ | \ / | \
se1 se2 se3 se4 se5 se6/ \ / \se21 se22 se61 se62
Assume CPU0 and CPU1 are smt siblings and task A's se is se21 while task B's se is se61. To compare priority of task A and B, we compare priority of se2 and se6. The smaller vruntime wins.
To make this work, the root level ses on both CPU should have a common
cfs_rqmin vuntime, which I call it the corecfs_rqmin vruntime.This is mostly done in patch2/3.
Test:
1 wrote an cpu intensive program that does nothing but while(1) in
main(), let's call it cpuhog;
2 start 2 cgroups, with one cgroup's cpuset binding to CPU2 and the
other binding to cpu3. cpu2 and cpu3 are smt siblings on the test VM;
3 enable cpu.tag for the two cgroups;
4 start one cpuhog task in each cgroup;
5 kill both cpuhog tasks after 10 seconds;
6 check each cgroup's cpu usage.If the task is scheduled fairly, then each cgroup's cpu usage should be around 5s.
With v3, the cpu usage of both cgroups are sometimes 3s, 7s; sometimes 1s, 9s.
With the 3 patches applied, the numbers are mostly around 5s, 5s.
Another test is starting two cgroups simultaneously with cpu.tag set, with one cgroup running: will-it-scale/page_fault1_processes -t 16 -s 30, the other running: will-it-scale/page_fault2_processes -t 16 -s 30. With v3, like I said last time, the later started
page_faultprocesses can't start running. With the 3 patches applied, both running at the same time with each CPU having a relatively fair score:output line of 16
page_fault1processes in 1 second interval:
min:105225 max:131716 total:1872322output line of 16
page_fault2processes in 1 second interval:
min:86797 max:110554 total:1581177
Quoted 7 lines
Note the value in min and max, the smaller the gap is, the better the faireness is.
Aubrey,
I haven't been able to run your workload yet...
No worry, let me try to see how it works.
Thanks,
-Aubrey
Quoted 2 lines · Aaron Lu
I tried a different approach based on vruntime with 3 patches following.
[...]
We have experimented with this new patchset and indeed the fairness is now much better. Interactive tasks with v3 were complete starving when there were cpu-intensive tasks running, now they can run consistently. With my initial test of TPC-C running in large VMs with a lot of background noise VMs, the results are pretty similar to v3, I will run more thorough tests and report the results back here.
Instead of the 3/3 hack patch, we were already working on a different approach to solve the same problem. What we have done so far is create a very low priority per-cpu coresched_idle kernel thread that we use instead of idle when we can't co-schedule tasks. This gives us more control and accounting. It still needs some work, but the initial results are encouraging, I will post more when we have something that works well.
Thanks,
Julien
Quoted 11 lines · Julien Desfossez
Quoted 2 lines · Aaron Lu
I tried a different approach based on vruntime with 3 patches following.
[...]
We have experimented with this new patchset and indeed the fairness is now much better. Interactive tasks with v3 were complete starving when there were cpu-intensive tasks running, now they can run consistently. With my initial test of TPC-C running in large VMs with a lot of background noise VMs, the results are pretty similar to v3, I will run more thorough tests and report the results back here.
Aaron's patch inspired me to experiment with another approach to tackle fairness. The root problem with v3 was we didn't account for the forced idle time when we compare the priority of tasks between two threads.
So what I did here is to account the forced idle time in the top cfs_rq's min_vruntime when we update the runqueue clock. When we are comparing between two cfs runqueues, the task in cpu getting forced idle will now be credited with the forced idle time. The effect should be similar to Aaron's patches. Logic is a bit simpler and we don't need to use one of the sibling's cfs_rq min_vruntime as a time base.
In really limited testing, it seems to have balanced fairness between two tagged cgroups.
Tim
-------patch 1----------
From: Tim Chen <[email protected]>
Date: Wed, 24 Jul 2019 13:58:18 -0700
Subject: [PATCH 1/2] sched: move sched fair prio comparison to fair.c
Move the priority comparison of two tasks in fair class to fair.c. There is no functional change.
--- kernel/sched/core.c | 21 ++------------------- kernel/sched/fair.c | 21 +++++++++++++++++++++ kernel/sched/sched.h | 1 + 3 files changed, 24 insertions(+), 19 deletions(-)
kernel/sched/core.c +2 -19
@@ -105,25 +105,8 @@ static inline bool prio_less(struct task_struct *a, struct task_struct *b)if (pa ==-1 )/* dl_prio() doesn't work because of stop_class above */ return !dl_time_before (a->dl.deadline, b->dl.deadline); -if (pa ==MAX_RT_PRIO +MAX_NICE ) {/* fair */ -u64 a_vruntime = a->se.vruntime; -u64 b_vruntime = b->se.vruntime; - -/* -* Normalize the vruntime if tasks are in different cpus. -*/ -if (task_cpu (a) !=task_cpu (b)) { - b_vruntime -=task_cfs_rq (b)->min_vruntime; - b_vruntime +=task_cfs_rq (a)->min_vruntime; - -trace_printk ("(%d:%Lu,%Lu,%Lu) <> (%d:%Lu,%Lu,%Lu)\n" , - a->pid, a_vruntime, a->se.vruntime,task_cfs_rq (a)->min_vruntime, - b->pid, b_vruntime, b->se.vruntime,task_cfs_rq (b)->min_vruntime); - - } - -return !((s64)(a_vruntime - b_vruntime) <=0 ); - } +if (pa ==MAX_RT_PRIO +MAX_NICE )/* fair */ +return prio_less_fair (a, b);return false; }
kernel/sched/fair.c +21 -0
@@ -602,6 +602,27 @@ static inline u64 calc_delta_fair(u64 delta, struct sched_entity *se)return delta; } +bool prio_less_fair (struct task_struct *a,struct task_struct *b) +{ +u64 a_vruntime = a->se.vruntime; +u64 b_vruntime = b->se.vruntime; + +/* +* Normalize the vruntime if tasks are in different cpus. +*/ +if (task_cpu (a) !=task_cpu (b)) { + b_vruntime -=task_cfs_rq (b)->min_vruntime; + b_vruntime +=task_cfs_rq (a)->min_vruntime; + +trace_printk ("(%d:%Lu,%Lu,%Lu) <> (%d:%Lu,%Lu,%Lu)\n" , + a->pid, a_vruntime, a->se.vruntime,task_cfs_rq (a)->min_vruntime, + b->pid, b_vruntime, b->se.vruntime,task_cfs_rq (b)->min_vruntime); + + } + +return !((s64)(a_vruntime - b_vruntime) <=0 ); +} +/* * The idea is to set a period in which each task runs once. *
kernel/sched/sched.h +1 -0
@@ -1015,6 +1015,7 @@ static inline raw_spinlock_t *rq_lockp(struct rq *rq) }extern void queue_core_balance (struct rq *rq); +extern bool prio_less_fair (struct task_struct *a,struct task_struct *b);#else /* !CONFIG_SCHED_CORE */
Signature · 20 lines
2.20.1 --------------patch 2------------------ From: Tim Chen <[email protected]> Date: Thu, 25 Jul 2019 13:09:21 -0700 Subject: [PATCH 2/2] sched: Account the forced idle time We did not account for the forced idle time when comparing two tasks from different SMT thread in the same core. Account it in root cfs_rq min_vruntime when we update the rq clock. This will allow for fair comparison of which task has higher priority from two different SMT. Signed-off-by: Tim Chen <[email protected]> --- kernel/sched/core.c | 6 ++++++ kernel/sched/fair.c | 22 ++++++++++++++++++---- 2 files changed, 24 insertions(+), 4 deletions(-)
kernel/sched/core.c +6 -0
@@ -393,6 +393,12 @@ static void update_rq_clock_task(struct rq *rq, s64 delta) rq->clock_task += delta; +#ifdef CONFIG_SCHED_CORE +/* Account the forced idle time by sibling */ +if (rq->core_forceidle) + rq->cfs.min_vruntime += delta; +#endif +#ifdef CONFIG_HAVE_SCHED_AVG_IRQ if ((irq_delta + steal) &&sched_feat (NONTASK_CAPACITY ))update_irq_load_avg (rq, irq_delta + steal);
kernel/sched/fair.c +18 -4
@@ -604,20 +604,34 @@ static inline u64 calc_delta_fair(u64 delta, struct sched_entity *se)bool prio_less_fair (struct task_struct *a,struct task_struct *b) { -u64 a_vruntime = a->se.vruntime; -u64 b_vruntime = b->se.vruntime; +u64 a_vruntime; +u64 b_vruntime;/* * Normalize the vruntime if tasks are in different cpus. */ if (task_cpu (a) !=task_cpu (b)) { - b_vruntime -=task_cfs_rq (b)->min_vruntime; - b_vruntime +=task_cfs_rq (a)->min_vruntime; +struct sched_entity *sea = &a->se; +struct sched_entity *seb = &b->se; + +while (sea->parent) + sea = sea->parent; +while (seb->parent) + seb = seb->parent; + + a_vruntime = sea->vruntime; + b_vruntime = seb->vruntime; + + b_vruntime -=task_rq (b)->cfs.min_vruntime; + b_vruntime +=task_rq (a)->cfs.min_vruntime;trace_printk ("(%d:%Lu,%Lu,%Lu) <> (%d:%Lu,%Lu,%Lu)\n" , a->pid, a_vruntime, a->se.vruntime,task_cfs_rq (a)->min_vruntime, b->pid, b_vruntime, b->se.vruntime,task_cfs_rq (b)->min_vruntime); + }else { + a_vruntime = a->se.vruntime; + b_vruntime = b->se.vruntime; }return !((s64)(a_vruntime - b_vruntime) <=0 );
Signature · 1 line
2.20.1
Quoted 8 lines · Julien Desfossez
Quoted 2 lines · Aaron Lu
I tried a different approach based on vruntime with 3 patches following.
[...]
We have experimented with this new patchset and indeed the fairness is now much better. Interactive tasks with v3 were complete starving when there were cpu-intensive tasks running, now they can run consistently.
Yeah, the fairness is much better now.
For two cgroups created, I limited the cpuset to be one core(two siblings) of these two cgroups, I still run gemmbench and sysbench-mysql, and here is the mysql result:
Latency:
.----------------------------------------------------------------------------------------------.
|NA/AVX vanilla-SMT [std% / sem%] cpu% |coresched-SMT [std% / sem%] +/- cpu% | |----------------------------------------------------------------------------------------------| | 1/1 6.7 [13.8%/ 1.4%] 2.1% | 6.4 [14.6%/ 1.5%] 4.0% 2.0% | | 2/2 9.1 [ 5.0%/ 0.5%] 4.0% | 11.4 [ 6.8%/ 0.7%] -24.9% 3.9% |
'----------------------------------------------------------------------------------------------'
Throughput:
.----------------------------------------------------------------------------------------------.
|NA/AVX vanilla-SMT [std% / sem%] cpu% |coresched-SMT [std% / sem%] +/- cpu% | |----------------------------------------------------------------------------------------------| | 1/1 310.2 [ 4.1%/ 0.4%] 2.1% | 296.2 [ 5.0%/ 0.5%] -4.5% 2.0% | | 2/2 547.7 [ 3.6%/ 0.4%] 4.0% | 368.3 [ 4.8%/ 0.5%] -32.8% 3.9% |
'----------------------------------------------------------------------------------------------'
Note: 2/2 case means 4 threads run on one core, which is overloaded.(cpu% is overall system report)
Though the latency/throughput has regressions, but standard deviation is much better now.
Quoted 3 lines
With my initial test of TPC-C running in large VMs with a lot of background noise VMs, the results are pretty similar to v3, I will run more thorough tests and report the results back here.
I see something similar. I guess task placement could be another problem. We don't check cookie matching in load balance and task wakeup, so
- if tasks with different cookie happen to be dispatched onto different cores, The result should be good
- if tasks with different cookie are unfortunately dispatched onto the same core, the result should be bad.
This problem is bypassed in my testing setup above, but may be one cause of my other scenarios, need a while to sort out.
Thanks,
-Aubrey
We tested both Aaron's and Tim's patches and here are our results.
Test setup:
- 2 1-thread sysbench, one running the cpu benchmark, the other one the mem benchmark
- both started at the same time
- both are pinned on the same core (2 hardware threads)
- 10 30-seconds runs
- test script: https://paste.debian.net/plainh/834cf45c
- only showing the CPU events/sec (higher is better)
- tested 4 tag configurations:
- no tag
- sysbench mem untagged, sysbench cpu tagged
- sysbench mem tagged, sysbench cpu untagged
- both tagged with a different tag
- "Alone" is the sysbench CPU running alone on the core, no tag
- "nosmt" is both sysbench pinned on the same hardware thread, no tag
- "Tim's full patchset + sched" is an experiment with Tim's patchset combined with Aaron's "hack patch" to get rid of the remaining deep idle cases
- In all test cases, both tasks can run simultaneously (which was not the case without those patches), but the standard deviation is a pretty good indicator of the fairness/consistency.
No tag
------
Test Average Stdev Alone 1306.90 0.94 nosmt 649.95 1.44 Aaron's full patchset: 828.15 32.45 Aaron's first 2 patches: 832.12 36.53 Aaron's 3rd patch alone: 864.21 3.68 Tim's full patchset: 852.50 4.11 Tim's full patchset + sched: 852.59 8.25
Sysbench mem untagged, sysbench cpu tagged
------------------------------------------
Test Average Stdev Alone 1306.90 0.94 nosmt 649.95 1.44 Aaron's full patchset: 586.06 1.77 Aaron's first 2 patches: 630.08 47.30 Aaron's 3rd patch alone: 1086.65 246.54 Tim's full patchset: 852.50 4.11 Tim's full patchset + sched: 390.49 15.76
Sysbench mem tagged, sysbench cpu untagged
------------------------------------------
Test Average Stdev Alone 1306.90 0.94 nosmt 649.95 1.44 Aaron's full patchset: 583.77 3.52 Aaron's first 2 patches: 513.63 63.09 Aaron's 3rd patch alone: 1171.23 3.35 Tim's full patchset: 564.04 58.05 Tim's full patchset + sched: 1026.16 49.43
Both sysbench tagged
--------------------
Test Average Stdev Alone 1306.90 0.94 nosmt 649.95 1.44 Aaron's full patchset: 582.15 3.75 Aaron's first 2 patches: 561.07 91.61 Aaron's 3rd patch alone: 638.49 231.06 Tim's full patchset: 679.43 70.07 Tim's full patchset + sched: 664.34 210.14
So in terms of fairness, Aaron's full patchset is the most consistent, but only Tim's patchset performs better than nosmt in some conditions.
Of course, this is one of the worst case scenario, as soon as we have multithreaded applications on overcommitted systems, core scheduling performs better than nosmt.
Thanks,
Julien
Quoted 15 lines · Julien Desfossez
We tested both Aaron's and Tim's patches and here are our results.
Test setup:
- 2 1-thread sysbench, one running the cpu benchmark, the other one the mem benchmark
- both started at the same time
- both are pinned on the same core (2 hardware threads)
- 10 30-seconds runs
- test script: https://paste.debian.net/plainh/834cf45c
- only showing the CPU events/sec (higher is better)
- tested 4 tag configurations:
- no tag
- sysbench mem untagged, sysbench cpu tagged
- sysbench mem tagged, sysbench cpu untagged
- both tagged with a different tag
Quoted 8 lines
- "Alone" is the sysbench CPU running alone on the core, no tag
- "nosmt" is both sysbench pinned on the same hardware thread, no tag
- "Tim's full patchset + sched" is an experiment with Tim's patchset combined with Aaron's "hack patch" to get rid of the remaining deep idle cases
- In all test cases, both tasks can run simultaneously (which was not the case without those patches), but the standard deviation is a pretty good indicator of the fairness/consistency.
Thanks for testing the patches and giving such detailed data.
I came to realize that for my scheme, the accumulated deficit of forced idle could be wiped out in one execution of a task on the forced idle cpu, with the update of the min_vruntime, even if the execution time could be far less than the accumulated deficit. That's probably one reason my scheme didn't achieve fairness.
Tim
Hi,
Quoted 59 lines · On Fri, Aug 02, 2019 at 11:37:15AM -0400 Julien Desfossez wrote:
We tested both Aaron's and Tim's patches and here are our results.
Test setup:
- 2 1-thread sysbench, one running the cpu benchmark, the other one the mem benchmark
- both started at the same time
- both are pinned on the same core (2 hardware threads)
- 10 30-seconds runs
- test script: https://paste.debian.net/plainh/834cf45c
- only showing the CPU events/sec (higher is better)
- tested 4 tag configurations:
- no tag
- sysbench mem untagged, sysbench cpu tagged
- sysbench mem tagged, sysbench cpu untagged
- both tagged with a different tag
- "Alone" is the sysbench CPU running alone on the core, no tag
- "nosmt" is both sysbench pinned on the same hardware thread, no tag
- "Tim's full patchset + sched" is an experiment with Tim's patchset combined with Aaron's "hack patch" to get rid of the remaining deep idle cases
- In all test cases, both tasks can run simultaneously (which was not the case without those patches), but the standard deviation is a pretty good indicator of the fairness/consistency.No tag
------Test Average Stdev Alone 1306.90 0.94 nosmt 649.95 1.44 Aaron's full patchset: 828.15 32.45 Aaron's first 2 patches: 832.12 36.53 Aaron's 3rd patch alone: 864.21 3.68 Tim's full patchset: 852.50 4.11 Tim's full patchset + sched: 852.59 8.25Sysbench mem untagged, sysbench cpu tagged
------------------------------------------Test Average Stdev Alone 1306.90 0.94 nosmt 649.95 1.44 Aaron's full patchset: 586.06 1.77 Aaron's first 2 patches: 630.08 47.30 Aaron's 3rd patch alone: 1086.65 246.54 Tim's full patchset: 852.50 4.11 Tim's full patchset + sched: 390.49 15.76Sysbench mem tagged, sysbench cpu untagged
------------------------------------------Test Average Stdev Alone 1306.90 0.94 nosmt 649.95 1.44 Aaron's full patchset: 583.77 3.52 Aaron's first 2 patches: 513.63 63.09 Aaron's 3rd patch alone: 1171.23 3.35 Tim's full patchset: 564.04 58.05 Tim's full patchset + sched: 1026.16 49.43Both sysbench tagged
--------------------
Quoted 9 lines
Test Average Stdev Alone 1306.90 0.94 nosmt 649.95 1.44 Aaron's full patchset: 582.15 3.75 Aaron's first 2 patches: 561.07 91.61 Aaron's 3rd patch alone: 638.49 231.06 Tim's full patchset: 679.43 70.07 Tim's full patchset + sched: 664.34 210.14
Sorry if I'm missing something obvious here but with only 2 processes of interest shouldn't one tagged and one untagged be about the same as both tagged?
In both cases the 2 sysbenches should not be running on the core at the same time.
There will be times when oher non-related threads could share the core with the untagged one. Is that enough to account for this difference?
Thanks,
Phil
Quoted 10 lines
So in terms of fairness, Aaron's full patchset is the most consistent, but only Tim's patchset performs better than nosmt in some conditions.
Of course, this is one of the worst case scenario, as soon as we have multithreaded applications on overcommitted systems, core scheduling performs better than nosmt.
Thanks,
Julien
Quoted 25 lines · Tim Chen
Quoted 15 lines · Julien Desfossez
We tested both Aaron's and Tim's patches and here are our results.
Test setup:
- 2 1-thread sysbench, one running the cpu benchmark, the other one the mem benchmark
- both started at the same time
- both are pinned on the same core (2 hardware threads)
- 10 30-seconds runs
- test script: https://paste.debian.net/plainh/834cf45c
- only showing the CPU events/sec (higher is better)
- tested 4 tag configurations:
- no tag
- sysbench mem untagged, sysbench cpu tagged
- sysbench mem tagged, sysbench cpu untagged
- both tagged with a different tagQuoted 8 lines
- "Alone" is the sysbench CPU running alone on the core, no tag
- "nosmt" is both sysbench pinned on the same hardware thread, no tag
- "Tim's full patchset + sched" is an experiment with Tim's patchset combined with Aaron's "hack patch" to get rid of the remaining deep idle cases
- In all test cases, both tasks can run simultaneously (which was not the case without those patches), but the standard deviation is a pretty good indicator of the fairness/consistency.
Quoted 1 line
Thanks for testing the patches and giving such detailed data.
Thanks Julien.
Quoted 4 lines
I came to realize that for my scheme, the accumulated deficit of forced idle could be wiped out in one execution of a task on the forced idle cpu, with the update of the
min_vruntime, even if the execution time could be far less than the accumulated deficit. That's probably one reason my scheme didn't achieve fairness.
I've been thinking if we should consider core wide tenent fairness?
Let's say there are 3 tasks on 2 threads' rq of the same core, 2 tasks (e.g. A1, A2) belong to tenent A and the 3rd B1 belong to another tenent B. Assume A1 and B1 are queued on the same thread and A2 on the other thread, when we decide priority for A1 and B1, shall we also consider A2's vruntime? i.e. shall we consider A1 and A2 as a whole since they belong to the same tenent? I tend to think we should make fairness per core per tenent, instead of per thread(cpu) per task(sched entity). What do you guys think?
Implemention of the idea is a mess to me, as I feel I'm duplicating the existing per cpu per sched_entity enqueue/update vruntime/dequeue logic for the per core per tenent stuff.
Quoted 38 lines · Aaron Lu
Quoted 25 lines · Tim Chen
Quoted 15 lines · Julien Desfossez
We tested both Aaron's and Tim's patches and here are our results.
Test setup:
- 2 1-thread sysbench, one running the cpu benchmark, the other one the mem benchmark
- both started at the same time
- both are pinned on the same core (2 hardware threads)
- 10 30-seconds runs
- test script: https://paste.debian.net/plainh/834cf45c
- only showing the CPU events/sec (higher is better)
- tested 4 tag configurations:
- no tag
- sysbench mem untagged, sysbench cpu tagged
- sysbench mem tagged, sysbench cpu untagged
- both tagged with a different tagQuoted 8 lines
- "Alone" is the sysbench CPU running alone on the core, no tag
- "nosmt" is both sysbench pinned on the same hardware thread, no tag
- "Tim's full patchset + sched" is an experiment with Tim's patchset combined with Aaron's "hack patch" to get rid of the remaining deep idle cases
- In all test cases, both tasks can run simultaneously (which was not the case without those patches), but the standard deviation is a pretty good indicator of the fairness/consistency.Quoted 1 line
Thanks for testing the patches and giving such detailed data.
Thanks Julien.
Quoted 4 lines
I came to realize that for my scheme, the accumulated deficit of forced idle could be wiped out in one execution of a task on the forced idle cpu, with the update of the
min_vruntime, even if the execution time could be far less than the accumulated deficit. That's probably one reason my scheme didn't achieve fairness.I've been thinking if we should consider core wide tenent fairness?
Quoted 9 lines
Let's say there are 3 tasks on 2 threads' rq of the same core, 2 tasks (e.g. A1, A2) belong to tenent A and the 3rd B1 belong to another tenent B. Assume A1 and B1 are queued on the same thread and A2 on the other thread, when we decide priority for A1 and B1, shall we also consider A2's vruntime? i.e. shall we consider A1 and A2 as a whole since they belong to the same tenent? I tend to think we should make fairness per core per tenent, instead of per thread(cpu) per task(sched entity). What do you guys think?
I also think a way to make fairness per cookie per core, is this what you want to propose?
Thanks,
-Aubrey
Quoted 3 lines
Implemention of the idea is a mess to me, as I feel I'm duplicating the existing per cpu per
sched_entityenqueue/update vruntime/dequeue logic for the per core per tenent stuff.
Quoted 13 lines · Aubrey Li
Quoted 11 lines · Aaron Lu
I've been thinking if we should consider core wide tenent fairness?
Let's say there are 3 tasks on 2 threads' rq of the same core, 2 tasks (e.g. A1, A2) belong to tenent A and the 3rd B1 belong to another tenent B. Assume A1 and B1 are queued on the same thread and A2 on the other thread, when we decide priority for A1 and B1, shall we also consider A2's vruntime? i.e. shall we consider A1 and A2 as a whole since they belong to the same tenent? I tend to think we should make fairness per core per tenent, instead of per thread(cpu) per task(sched entity). What do you guys think?
Quoted 2 lines
I also think a way to make fairness per cookie per core, is this what you want to propose?
Yes, that's what I meant.
Quoted 5 lines
Quoted 3 lines
I also think a way to make fairness per cookie per core, is this what you want to propose?
Yes, that's what I meant.
I think that would hurt some kind of workloads badly, especially if one tenant is
having way more tasks than the other. Tenant with more task on the same core might have immediate requirements from some threads than the other and we would fail to take that into account. With some hierarchical management, we can alleviate this, but as Aaron said, it would be a bit messy.
Peter's rebalance logic actually takes care of most of the runq imbalance caused
due to cookie tagging. What we have found from our testing is, fairness issue is caused mostly due to a Hyperthread going idle and not waking up. Aaron's 3rd patch works around that. As Julien mentioned, we are working on a per thread coresched idle thread concept. The problem that we found was, idle thread causes accounting issues and wakeup issues as it was not designed to be used in this context. So if we can have a low priority thread which looks like any other task to the scheduler, things becomes easy for the scheduler and we achieve security as well. Please share your thoughts on this idea.
The results are encouraging, but we do not yet have the coresched idle to not spin
100%. We will soon post the patch once it is a bit more stable for running the tests
that we all have done so far.
Thanks,
Vineeth
Quoted 4 lines · Vineeth Remanan Pillai
Quoted 4 lines
Quoted 3 lines
I also think a way to make fairness per cookie per core, is this what you want to propose?
Quoted 8 lines
Quoted 1 line
Yes, that's what I meant.
I think that would hurt some kind of workloads badly, especially if one tenant is
having way more tasks than the other. Tenant with more task on the same core might have immediate requirements from some threads than the other and we would fail to take that into account. With some hierarchical management, we can alleviate this, but as Aaron said, it would be a bit messy.
I think tenant will have per core weight, similar to sched entity's per cpu weight. The tenant's per core weight could derive from its corresponding taskgroup's per cpu sched entities' weight(sum them up perhaps). Tenant with higher weight will have its core wide vruntime advance slower than tenant with lower weight. Does this address the issue here?
Quoted 10 lines
Peter's rebalance logic actually takes care of most of the runq imbalance caused
due to cookie tagging. What we have found from our testing is, fairness issue is caused mostly due to a Hyperthread going idle and not waking up. Aaron's 3rd patch works around that. As Julien mentioned, we are working on a per thread coresched idle thread concept. The problem that we found was, idle thread causes accounting issues and wakeup issues as it was not designed to be used in this context. So if we can have a low priority thread which looks like any other task to the scheduler, things becomes easy for the scheduler and we achieve security as well. Please share your thoughts on this idea.
Care to elaborate the idea of coresched idle thread concept? How it solved the hyperthread going idle problem and what the accounting issues and wakeup issues are, etc.
Thanks,
Aaron
Quoted 8 lines
The results are encouraging, but we do not yet have the coresched idle to not spin
100%. We will soon post the patch once it is a bit more stable for running the tests
that we all have done so far.Thanks,
Vineeth
Quoted 72 lines · Phil Auld
Hi,
Quoted 59 lines · On Fri, Aug 02, 2019 at 11:37:15AM -0400 Julien Desfossez wrote:
We tested both Aaron's and Tim's patches and here are our results.
Test setup:
- 2 1-thread sysbench, one running the cpu benchmark, the other one the mem benchmark
- both started at the same time
- both are pinned on the same core (2 hardware threads)
- 10 30-seconds runs
- test script: https://paste.debian.net/plainh/834cf45c
- only showing the CPU events/sec (higher is better)
- tested 4 tag configurations:
- no tag
- sysbench mem untagged, sysbench cpu tagged
- sysbench mem tagged, sysbench cpu untagged
- both tagged with a different tag
- "Alone" is the sysbench CPU running alone on the core, no tag
- "nosmt" is both sysbench pinned on the same hardware thread, no tag
- "Tim's full patchset + sched" is an experiment with Tim's patchset combined with Aaron's "hack patch" to get rid of the remaining deep idle cases
- In all test cases, both tasks can run simultaneously (which was not the case without those patches), but the standard deviation is a pretty good indicator of the fairness/consistency.No tag
------Test Average Stdev Alone 1306.90 0.94 nosmt 649.95 1.44 Aaron's full patchset: 828.15 32.45 Aaron's first 2 patches: 832.12 36.53 Aaron's 3rd patch alone: 864.21 3.68 Tim's full patchset: 852.50 4.11 Tim's full patchset + sched: 852.59 8.25Sysbench mem untagged, sysbench cpu tagged
------------------------------------------Test Average Stdev Alone 1306.90 0.94 nosmt 649.95 1.44 Aaron's full patchset: 586.06 1.77 Aaron's first 2 patches: 630.08 47.30 Aaron's 3rd patch alone: 1086.65 246.54 Tim's full patchset: 852.50 4.11 Tim's full patchset + sched: 390.49 15.76Sysbench mem tagged, sysbench cpu untagged
------------------------------------------Test Average Stdev Alone 1306.90 0.94 nosmt 649.95 1.44 Aaron's full patchset: 583.77 3.52 Aaron's first 2 patches: 513.63 63.09 Aaron's 3rd patch alone: 1171.23 3.35 Tim's full patchset: 564.04 58.05 Tim's full patchset + sched: 1026.16 49.43Both sysbench tagged
--------------------Quoted 9 lines
Test Average Stdev Alone 1306.90 0.94 nosmt 649.95 1.44 Aaron's full patchset: 582.15 3.75 Aaron's first 2 patches: 561.07 91.61 Aaron's 3rd patch alone: 638.49 231.06 Tim's full patchset: 679.43 70.07 Tim's full patchset + sched: 664.34 210.14
Quoted 3 lines
Sorry if I'm missing something obvious here but with only 2 processes of interest shouldn't one tagged and one untagged be about the same as both tagged?
It should.
Quoted 2 lines
In both cases the 2 sysbenches should not be running on the core at the same time.
Agree.
Quoted 2 lines
There will be times when oher non-related threads could share the core with the untagged one. Is that enough to account for this difference?
What difference do you mean?
Thanks,
Aaron
Quoted 12 lines
Quoted 10 lines
So in terms of fairness, Aaron's full patchset is the most consistent, but only Tim's patchset performs better than nosmt in some conditions.
Of course, this is one of the worst case scenario, as soon as we have multithreaded applications on overcommitted systems, core scheduling performs better than nosmt.
Thanks,
Julien
Peter Zijlstra
maintainerQuoted 10 lines · Vineeth Remanan Pillai
Peter's rebalance logic actually takes care of most of the runq imbalance caused
due to cookie tagging. What we have found from our testing is, fairness issue is caused mostly due to a Hyperthread going idle and not waking up. Aaron's 3rd patch works around that. As Julien mentioned, we are working on a per thread coresched idle thread concept. The problem that we found was, idle thread causes accounting issues and wakeup issues as it was not designed to be used in this context. So if we can have a low priority thread which looks like any other task to the scheduler, things becomes easy for the scheduler and we achieve security as well. Please share your thoughts on this idea.
What accounting in particular is upset? Is it things like select_idle_sibling() that thinks the thread is idle and tries to place tasks there?
It should be possible to change idle_cpu() to not report a forced-idle CPU as idle.
(also; it should be possible to optimize select_idle_sibling() for the core-sched case specifically)
Quoted 3 lines
The results are encouraging, but we do not yet have the coresched idle to not spin 100%. We will soon post the patch once it is a bit more stable for running the tests that we all have done so far.
There's play_idle(), which is the entry point for idle injection.
In general, I don't particularly like 'fake' idle threads, please be very specific in describing what issues it works around such that we can look at alternatives.
Peter Zijlstra
maintainerRe: [PATCH 2/3] core vruntime comparison
In reply to Aaron LuQuoted 29 lines · Aaron Lu
kernel/sched/fair.c +29
-0+bool cfs_prio_less (struct task_struct *a,struct task_struct *b) +{ +struct sched_entity *sea = &a->se; +struct sched_entity *seb = &b->se; +bool samecpu =task_cpu (a) ==task_cpu (b); +struct task_struct *p; +s64 delta; + +if (samecpu) { +/* vruntime is per cfs_rq */ +while (!is_same_group (sea, seb)) { +int sea_depth = sea->depth; +int seb_depth = seb->depth; + +if (sea_depth >= seb_depth) + sea =parent_entity (sea); +if (sea_depth <= seb_depth) + seb =parent_entity (seb); + } + + delta = (s64)(sea->vruntime - seb->vruntime); + goto out; + } + +/* crosscpu: compare root level se's vruntime to decide priority */ +while (sea->parent) + sea = sea->parent; +while (seb->parent) + seb = seb->parent;
kernel/sched/fair.c find_matching_se(struct sched_entity **se, struct sched_entity **pse) · line 513
Quoted 10 lines
kernel/sched/fair.c +9
-0+ delta = (s64)(sea->vruntime - seb->vruntime); + +out: + p = delta >0 ? b : a; +trace_printk ("picked %s/%d %s: %Ld %Ld %Ld\n" , p->comm, p->pid, + samecpu ?"samecpu" :"crosscpu" , + sea->vruntime, seb->vruntime, delta); + +return delta >0 ; }
Heh.. I suppose the good news is that Rik is trying very hard to kill the nested runqueues, which would make this _much_ easier again.
Quoted 82 lines · On Tue, Aug 06, 2019 at 09:54:01PM +0800 Aaron Lu wrote:
Quoted 72 lines · Phil Auld
Hi,
Quoted 59 lines · On Fri, Aug 02, 2019 at 11:37:15AM -0400 Julien Desfossez wrote:
We tested both Aaron's and Tim's patches and here are our results.
Test setup:
- 2 1-thread sysbench, one running the cpu benchmark, the other one the mem benchmark
- both started at the same time
- both are pinned on the same core (2 hardware threads)
- 10 30-seconds runs
- test script: https://paste.debian.net/plainh/834cf45c
- only showing the CPU events/sec (higher is better)
- tested 4 tag configurations:
- no tag
- sysbench mem untagged, sysbench cpu tagged
- sysbench mem tagged, sysbench cpu untagged
- both tagged with a different tag
- "Alone" is the sysbench CPU running alone on the core, no tag
- "nosmt" is both sysbench pinned on the same hardware thread, no tag
- "Tim's full patchset + sched" is an experiment with Tim's patchset combined with Aaron's "hack patch" to get rid of the remaining deep idle cases
- In all test cases, both tasks can run simultaneously (which was not the case without those patches), but the standard deviation is a pretty good indicator of the fairness/consistency.No tag
------Test Average Stdev Alone 1306.90 0.94 nosmt 649.95 1.44 Aaron's full patchset: 828.15 32.45 Aaron's first 2 patches: 832.12 36.53 Aaron's 3rd patch alone: 864.21 3.68 Tim's full patchset: 852.50 4.11 Tim's full patchset + sched: 852.59 8.25Sysbench mem untagged, sysbench cpu tagged
------------------------------------------Test Average Stdev Alone 1306.90 0.94 nosmt 649.95 1.44 Aaron's full patchset: 586.06 1.77 Aaron's first 2 patches: 630.08 47.30 Aaron's 3rd patch alone: 1086.65 246.54 Tim's full patchset: 852.50 4.11 Tim's full patchset + sched: 390.49 15.76Sysbench mem tagged, sysbench cpu untagged
------------------------------------------Test Average Stdev Alone 1306.90 0.94 nosmt 649.95 1.44 Aaron's full patchset: 583.77 3.52 Aaron's first 2 patches: 513.63 63.09 Aaron's 3rd patch alone: 1171.23 3.35 Tim's full patchset: 564.04 58.05 Tim's full patchset + sched: 1026.16 49.43Both sysbench tagged
--------------------Quoted 9 lines
Test Average Stdev Alone 1306.90 0.94 nosmt 649.95 1.44 Aaron's full patchset: 582.15 3.75 Aaron's first 2 patches: 561.07 91.61 Aaron's 3rd patch alone: 638.49 231.06 Tim's full patchset: 679.43 70.07 Tim's full patchset + sched: 664.34 210.14Quoted 3 lines
Sorry if I'm missing something obvious here but with only 2 processes of interest shouldn't one tagged and one untagged be about the same as both tagged?
It should.
Quoted 2 lines
In both cases the 2 sysbenches should not be running on the core at the same time.
Quoted 6 lines
Agree.
Quoted 2 lines
There will be times when oher non-related threads could share the core with the untagged one. Is that enough to account for this difference?
What difference do you mean?
I was looking at the above posted numbers. For example:
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Sysbench mem untagged, sysbench cpu tagged
Aaron's 3rd patch alone: 1086.65 246.54Sysbench mem tagged, sysbench cpu untagged
Aaron's 3rd patch alone: 1171.23 3.35Both sysbench tagged
Aaron's 3rd patch alone: 638.49 231.06
Admittedly, there's some high variance on some of those numbers.
Cheers,
Phil
Quoted 16 lines
Thanks,
AaronQuoted 12 lines
Quoted 10 lines
So in terms of fairness, Aaron's full patchset is the most consistent, but only Tim's patchset performs better than nosmt in some conditions.
Of course, this is one of the worst case scenario, as soon as we have multithreaded applications on overcommitted systems, core scheduling performs better than nosmt.
Thanks,
Julien
Quoted 96 lines · Phil Auld
Quoted 82 lines · On Tue, Aug 06, 2019 at 09:54:01PM +0800 Aaron Lu wrote:
Quoted 72 lines · Phil Auld
Hi,
Quoted 59 lines · On Fri, Aug 02, 2019 at 11:37:15AM -0400 Julien Desfossez wrote:
We tested both Aaron's and Tim's patches and here are our results.
Test setup:
- 2 1-thread sysbench, one running the cpu benchmark, the other one the mem benchmark
- both started at the same time
- both are pinned on the same core (2 hardware threads)
- 10 30-seconds runs
- test script: https://paste.debian.net/plainh/834cf45c
- only showing the CPU events/sec (higher is better)
- tested 4 tag configurations:
- no tag
- sysbench mem untagged, sysbench cpu tagged
- sysbench mem tagged, sysbench cpu untagged
- both tagged with a different tag
- "Alone" is the sysbench CPU running alone on the core, no tag
- "nosmt" is both sysbench pinned on the same hardware thread, no tag
- "Tim's full patchset + sched" is an experiment with Tim's patchset combined with Aaron's "hack patch" to get rid of the remaining deep idle cases
- In all test cases, both tasks can run simultaneously (which was not the case without those patches), but the standard deviation is a pretty good indicator of the fairness/consistency.No tag
------Test Average Stdev Alone 1306.90 0.94 nosmt 649.95 1.44 Aaron's full patchset: 828.15 32.45 Aaron's first 2 patches: 832.12 36.53 Aaron's 3rd patch alone: 864.21 3.68 Tim's full patchset: 852.50 4.11 Tim's full patchset + sched: 852.59 8.25Sysbench mem untagged, sysbench cpu tagged
------------------------------------------Test Average Stdev Alone 1306.90 0.94 nosmt 649.95 1.44 Aaron's full patchset: 586.06 1.77 Aaron's first 2 patches: 630.08 47.30 Aaron's 3rd patch alone: 1086.65 246.54 Tim's full patchset: 852.50 4.11 Tim's full patchset + sched: 390.49 15.76Sysbench mem tagged, sysbench cpu untagged
------------------------------------------Test Average Stdev Alone 1306.90 0.94 nosmt 649.95 1.44 Aaron's full patchset: 583.77 3.52 Aaron's first 2 patches: 513.63 63.09 Aaron's 3rd patch alone: 1171.23 3.35 Tim's full patchset: 564.04 58.05 Tim's full patchset + sched: 1026.16 49.43Both sysbench tagged
--------------------Quoted 9 lines
Test Average Stdev Alone 1306.90 0.94 nosmt 649.95 1.44 Aaron's full patchset: 582.15 3.75 Aaron's first 2 patches: 561.07 91.61 Aaron's 3rd patch alone: 638.49 231.06 Tim's full patchset: 679.43 70.07 Tim's full patchset + sched: 664.34 210.14Quoted 3 lines
Sorry if I'm missing something obvious here but with only 2 processes of interest shouldn't one tagged and one untagged be about the same as both tagged?
It should.
Quoted 2 lines
In both cases the 2 sysbenches should not be running on the core at the same time.
Quoted 6 lines
Agree.
Quoted 2 lines
There will be times when oher non-related threads could share the core with the untagged one. Is that enough to account for this difference?
What difference do you mean?
I was looking at the above posted numbers. For example:
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Sysbench mem untagged, sysbench cpu tagged
Aaron's 3rd patch alone: 1086.65 246.54
Quoted 8 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Sysbench mem tagged, sysbench cpu untagged
Aaron's 3rd patch alone: 1171.23 3.35Both sysbench tagged
Aaron's 3rd patch alone: 638.49 231.06Admittedly, there's some high variance on some of those numbers.
The high variance suggests the code having some fairness issues :-)
For the test here, I didn't expect the 3rd patch being used alone since the fairness is solved by patch2 and patch3 together.
Quoted 104 lines · On Tue, Aug 06, 2019 at 10:41:25PM +0800 Aaron Lu wrote:
Quoted 103 lines · Phil Auld
Quoted 82 lines · On Tue, Aug 06, 2019 at 09:54:01PM +0800 Aaron Lu wrote:
Quoted 72 lines · Phil Auld
Hi,
Quoted 59 lines · On Fri, Aug 02, 2019 at 11:37:15AM -0400 Julien Desfossez wrote:
We tested both Aaron's and Tim's patches and here are our results.
Test setup:
- 2 1-thread sysbench, one running the cpu benchmark, the other one the mem benchmark
- both started at the same time
- both are pinned on the same core (2 hardware threads)
- 10 30-seconds runs
- test script: https://paste.debian.net/plainh/834cf45c
- only showing the CPU events/sec (higher is better)
- tested 4 tag configurations:
- no tag
- sysbench mem untagged, sysbench cpu tagged
- sysbench mem tagged, sysbench cpu untagged
- both tagged with a different tag
- "Alone" is the sysbench CPU running alone on the core, no tag
- "nosmt" is both sysbench pinned on the same hardware thread, no tag
- "Tim's full patchset + sched" is an experiment with Tim's patchset combined with Aaron's "hack patch" to get rid of the remaining deep idle cases
- In all test cases, both tasks can run simultaneously (which was not the case without those patches), but the standard deviation is a pretty good indicator of the fairness/consistency.No tag
------Test Average Stdev Alone 1306.90 0.94 nosmt 649.95 1.44 Aaron's full patchset: 828.15 32.45 Aaron's first 2 patches: 832.12 36.53 Aaron's 3rd patch alone: 864.21 3.68 Tim's full patchset: 852.50 4.11 Tim's full patchset + sched: 852.59 8.25Sysbench mem untagged, sysbench cpu tagged
------------------------------------------Test Average Stdev Alone 1306.90 0.94 nosmt 649.95 1.44 Aaron's full patchset: 586.06 1.77 Aaron's first 2 patches: 630.08 47.30 Aaron's 3rd patch alone: 1086.65 246.54 Tim's full patchset: 852.50 4.11 Tim's full patchset + sched: 390.49 15.76Sysbench mem tagged, sysbench cpu untagged
------------------------------------------Test Average Stdev Alone 1306.90 0.94 nosmt 649.95 1.44 Aaron's full patchset: 583.77 3.52 Aaron's first 2 patches: 513.63 63.09 Aaron's 3rd patch alone: 1171.23 3.35 Tim's full patchset: 564.04 58.05 Tim's full patchset + sched: 1026.16 49.43Both sysbench tagged
--------------------Quoted 9 lines
Test Average Stdev Alone 1306.90 0.94 nosmt 649.95 1.44 Aaron's full patchset: 582.15 3.75 Aaron's first 2 patches: 561.07 91.61 Aaron's 3rd patch alone: 638.49 231.06 Tim's full patchset: 679.43 70.07 Tim's full patchset + sched: 664.34 210.14Quoted 3 lines
Sorry if I'm missing something obvious here but with only 2 processes of interest shouldn't one tagged and one untagged be about the same as both tagged?
It should.
Quoted 2 lines
In both cases the 2 sysbenches should not be running on the core at the same time.
Quoted 6 lines
Agree.
Quoted 2 lines
There will be times when oher non-related threads could share the core with the untagged one. Is that enough to account for this difference?
What difference do you mean?
I was looking at the above posted numbers. For example:
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Sysbench mem untagged, sysbench cpu tagged
Aaron's 3rd patch alone: 1086.65 246.54Sysbench mem tagged, sysbench cpu untagged
Aaron's 3rd patch alone: 1171.23 3.35Both sysbench tagged
Aaron's 3rd patch alone: 638.49 231.06
Quoted 6 lines
Quoted 1 line
Admittedly, there's some high variance on some of those numbers.
The high variance suggests the code having some fairness issues :-)
For the test here, I didn't expect the 3rd patch being used alone since the fairness is solved by patch2 and patch3 together.
Makes sense, thanks.
Quoted 5 lines
What accounting in particular is upset? Is it things like
select_idle_sibling()that thinks the thread is idle and tries to place tasks there?
The major issue that we saw was, certain work load causes the idle cpu to never wakeup and schedule again even when there are runnable threads in there. If I remember correctly, this happened when the sibling had only one cpu intensive task and did not enter the pick_next_task for a long time. There were other situations as well which caused this prolonged idle state on the cpu. One was whenpick_next_task was called on the sibling but it always won there because vruntime
was not progressing on the idle cpu.
Having a coresched idle makes sure that the idle thread is not overloaded. Also vruntime moves forward and tsk vruntime comparison across cpus works when we normalize.
Quoted 2 lines
It should be possible to change
idle_cpu()to not report a forced-idle CPU as idle.
I agree. If we can identify all the places the idle thread is considered special and
also account for the vruntime progress for force idle, this should be a better approach compared to coresched idle thread per cpu.
Quoted 4 lines
(also; it should be possible to optimize
select_idle_sibling()for the core-sched case specifically)
We haven't seen this because, most of our micro test cases did not have more threads than the cpus. Thanks for pointing this out, we shall cook some tests to observe this behavior.
Thanks,
Vineeth
Quoted 7 lines
I think tenant will have per core weight, similar to sched entity's per cpu weight. The tenant's per core weight could derive from its corresponding taskgroup's per cpu sched entities' weight(sum them up perhaps). Tenant with higher weight will have its core wide vruntime advance slower than tenant with lower weight. Does this address the issue here?
I think that makes sense. Should work. We should also consider how to classify untagged processes so that they are not starved .
Quoted 5 lines
Care to elaborate the idea of coresched idle thread concept? How it solved the hyperthread going idle problem and what the accounting issues and wakeup issues are, etc.
So we have one coresched_idle thread per cpu and when a sibling cannot find a match, instead of forcing idle, we schedule this new thread. Ideally this thread would be similar to idle, but scheduler doesn't now confuse idle cpu with a forced idle state. This also invokes schedule() as vruntime progresses(alternative to your 3rd patch) and vruntime accounting gets more consistent. There are special cases that need to be handled so that coresched_idle never gets scheduled in the normal scheduling path(without coresched) etc. Hope this clarifies.
But as Peter suggested, if we can differentiate idle from forced idle in the idle thread and account for the vruntime progress, that would be a better approach.
Thanks,
Vineeth
Quoted 11 lines · Aaron Lu
I've been thinking if we should consider core wide tenent fairness?
Let's say there are 3 tasks on 2 threads' rq of the same core, 2 tasks (e.g. A1, A2) belong to tenent A and the 3rd B1 belong to another tenent B. Assume A1 and B1 are queued on the same thread and A2 on the other thread, when we decide priority for A1 and B1, shall we also consider A2's vruntime? i.e. shall we consider A1 and A2 as a whole since they belong to the same tenent? I tend to think we should make fairness per core per tenent, instead of per thread(cpu) per task(sched entity). What do you guys think?
Quoted 3 lines
Implemention of the idea is a mess to me, as I feel I'm duplicating the existing per cpu per
sched_entityenqueue/update vruntime/dequeue logic for the per core per tenent stuff.
I'm wondering if something simpler will work. It is easier to maintain fairness between the CPU threads. A simple scheme may be if the force idle deficit on a CPU thread exceeds a threshold compared to its sibling, we will bias in choosing the task on the suppressed CPU thread.
The fairness among the tenents per run queue is balanced out by cfq fairness, so things should be fair if we maintain fairness in CPU utilization between the two CPU threads.
Tim
Peter Zijlstra
maintainerQuoted 17 lines · Tim Chen
Quoted 11 lines · Aaron Lu
I've been thinking if we should consider core wide tenent fairness?
Let's say there are 3 tasks on 2 threads' rq of the same core, 2 tasks (e.g. A1, A2) belong to tenent A and the 3rd B1 belong to another tenent B. Assume A1 and B1 are queued on the same thread and A2 on the other thread, when we decide priority for A1 and B1, shall we also consider A2's vruntime? i.e. shall we consider A1 and A2 as a whole since they belong to the same tenent? I tend to think we should make fairness per core per tenent, instead of per thread(cpu) per task(sched entity). What do you guys think?
Quoted 3 lines
Implemention of the idea is a mess to me, as I feel I'm duplicating the existing per cpu per
sched_entityenqueue/update vruntime/dequeue logic for the per core per tenent stuff.
Quoted 7 lines
I'm wondering if something simpler will work. It is easier to maintain fairness between the CPU threads. A simple scheme may be if the force idle deficit on a CPU thread exceeds a threshold compared to its sibling, we will bias in choosing the task on the suppressed CPU thread.
The fairness among the tenents per run queue is balanced out by cfq fairness, so things should be fair if we maintain fairness in CPU utilization between the two CPU threads.
IIRC pjt once did a simle 5ms flip flop between siblings.
Quoted 10 lines · Peter Zijlstra
Quoted 7 lines
I'm wondering if something simpler will work. It is easier to maintain fairness between the CPU threads. A simple scheme may be if the force idle deficit on a CPU thread exceeds a threshold compared to its sibling, we will bias in choosing the task on the suppressed CPU thread.
The fairness among the tenents per run queue is balanced out by cfq fairness, so things should be fair if we maintain fairness in CPU utilization between the two CPU threads.IIRC pjt once did a simle 5ms flip flop between siblings.
Trying out Peter's suggestions in the following two patches on v3 to provide fairness between the CPU threads.
The changes is in patch 2 and patch 1 is simply a code reorg.
It is only minimally tested and seems to provide fairness between two will-it-scale cgroups. Haven't tried it yet on something that is less CPU intensive with lots of sleep in between.
Also need to put the account idle time for rt and dl sched classes. Will do that later if this works for the fair class.
Tim
----------------patch 1-----------------------
From ede10309986a6b1bcc82d317f86a5b06459d76bd Mon Sep 17 00:00:00 2001
From: Tim Chen <[email protected]>
Date: Wed, 24 Jul 2019 13:58:18 -0700
Subject: [PATCH 1/2] sched: Move sched fair prio comparison to fair.c
Consolidate the task priority comparison of the fair class
to fair.c. A simple code reorganization and there are no functional changes.
--- kernel/sched/core.c | 21 ++------------------- kernel/sched/fair.c | 21 +++++++++++++++++++++ kernel/sched/sched.h | 1 + 3 files changed, 24 insertions(+), 19 deletions(-)
kernel/sched/core.c +2 -19
@@ -105,25 +105,8 @@ static inline bool prio_less(struct task_struct *a, struct task_struct *b)if (pa ==-1 )/* dl_prio() doesn't work because of stop_class above */ return !dl_time_before (a->dl.deadline, b->dl.deadline); -if (pa ==MAX_RT_PRIO +MAX_NICE ) {/* fair */ -u64 a_vruntime = a->se.vruntime; -u64 b_vruntime = b->se.vruntime; - -/* -* Normalize the vruntime if tasks are in different cpus. -*/ -if (task_cpu (a) !=task_cpu (b)) { - b_vruntime -=task_cfs_rq (b)->min_vruntime; - b_vruntime +=task_cfs_rq (a)->min_vruntime; - -trace_printk ("(%d:%Lu,%Lu,%Lu) <> (%d:%Lu,%Lu,%Lu)\n" , - a->pid, a_vruntime, a->se.vruntime,task_cfs_rq (a)->min_vruntime, - b->pid, b_vruntime, b->se.vruntime,task_cfs_rq (b)->min_vruntime); - - } - -return !((s64)(a_vruntime - b_vruntime) <=0 ); - } +if (pa ==MAX_RT_PRIO +MAX_NICE )/* fair */ +return prio_less_fair (a, b);return false; }
kernel/sched/fair.c +21 -0
@@ -602,6 +602,27 @@ static inline u64 calc_delta_fair(u64 delta, struct sched_entity *se)return delta; } +bool prio_less_fair (struct task_struct *a,struct task_struct *b) +{ +u64 a_vruntime = a->se.vruntime; +u64 b_vruntime = b->se.vruntime; + +/* +* Normalize the vruntime if tasks are in different cpus. +*/ +if (task_cpu (a) !=task_cpu (b)) { + b_vruntime -=task_cfs_rq (b)->min_vruntime; + b_vruntime +=task_cfs_rq (a)->min_vruntime; + +trace_printk ("(%d:%Lu,%Lu,%Lu) <> (%d:%Lu,%Lu,%Lu)\n" , + a->pid, a_vruntime, a->se.vruntime,task_cfs_rq (a)->min_vruntime, + b->pid, b_vruntime, b->se.vruntime,task_cfs_rq (b)->min_vruntime); + + } + +return !((s64)(a_vruntime - b_vruntime) <=0 ); +} +/* * The idea is to set a period in which each task runs once. *
kernel/sched/sched.h +1 -0
@@ -1015,6 +1015,7 @@ static inline raw_spinlock_t *rq_lockp(struct rq *rq) }extern void queue_core_balance (struct rq *rq); +extern bool prio_less_fair (struct task_struct *a,struct task_struct *b);#else /* !CONFIG_SCHED_CORE */
Signature · 20 lines
2.20.1 --------------------patch 2------------------------ From e27305b0042631382bd4f72260d579bf4c971d2f Mon Sep 17 00:00:00 2001 From: Tim Chen <[email protected]> Date: Tue, 6 Aug 2019 12:50:45 -0700 Subject: [PATCH 2/2] sched: Enforce fairness between cpu threads CPU thread could be suppressed by its sibling for extended time. Implement a budget for force idling, making all CPU threads have equal chance to run. Signed-off-by: Tim Chen <[email protected]> --- kernel/sched/core.c | 41 +++++++++++++++++++++++++++++++++++++++++ kernel/sched/fair.c | 12 ++++++++++++ kernel/sched/sched.h | 4 ++++ 3 files changed, 57 insertions(+)
kernel/sched/core.c +41 -0
59 lines not on this page · raw message
kernel/sched/fair.c +12 -0
@@ -611,6 +611,17 @@ bool prio_less_fair(struct task_struct *a, struct task_struct *b)* Normalize the vruntime if tasks are in different cpus. */ if (task_cpu (a) !=task_cpu (b)) { + +if (a->core_cookie && b->core_cookie && + a->core_cookie != b->core_cookie) { +/* +* Will be force idling one thread, +* pick the thread that has more allowance. +*/ +return (task_rq (a)->core_idle_allowance <= +task_rq (b)->core_idle_allowance) ? true : false; + } + b_vruntime -=task_cfs_rq (b)->min_vruntime; b_vruntime +=task_cfs_rq (a)->min_vruntime;
@@ -817,6 +828,7 @@ static void update_curr(struct cfs_rq *cfs_rq)trace_sched_stat_runtime (curtask, delta_exec, curr->vruntime);cgroup_account_cputime (curtask, delta_exec);account_group_exec_runtime (curtask, delta_exec); +account_core_idletime (curtask, delta_exec); }account_cfs_rq_runtime (cfs_rq, delta_exec);
kernel/sched/sched.h +4 -0
@@ -963,6 +963,7 @@ struct rq {struct task_struct *core_pick;unsigned int core_enabled;unsigned int core_sched_seq; +s64 core_idle_allowance;struct rb_root core_tree;bool core_forceidle;
@@ -999,6 +1000,8 @@ static inline int cpu_of(struct rq *rq) }#ifdef CONFIG_SCHED_CORE +#define SCHED_IDLE_ALLOWANCE 5000000/* 5 msec */ +DECLARE_STATIC_KEY_FALSE (__sched_core_enabled);static inline bool sched_core_enabled (struct rq *rq)
@@ -1016,6 +1019,7 @@ static inline raw_spinlock_t *rq_lockp(struct rq *rq)extern void queue_core_balance (struct rq *rq);extern bool prio_less_fair (struct task_struct *a,struct task_struct *b); +extern void account_core_idletime (struct task_struct *p,u64 exec);#else /* !CONFIG_SCHED_CORE */
Signature · 1 line
2.20.1
Hello everyone,
This is Dario, from SUSE. I'm also interesting in core-scheduling, and using it in virtualization use cases.
Just for context, I'm working in virt since a few years, mostly on Xen, but I've done Linux stuff before, and I am getting back at it.
For now, I've been looking at the core-scheduling code, and run some benchmarks myself.
Quoted 16 lines · Julien Desfossez
We tested both Aaron's and Tim's patches and here are our results.
Test setup:
- 2 1-thread sysbench, one running the cpu benchmark, the other one the
mem benchmark
- both started at the same time
- both are pinned on the same core (2 hardware threads)
- 10 30-seconds runs
- test script: https://paste.debian.net/plainh/834cf45c
- only showing the CPU events/sec (higher is better)
- tested 4 tag configurations:
- no tag
- sysbench mem untagged, sysbench cpu tagged
- sysbench mem tagged, sysbench cpu untagged
- both tagged with a different tag
Quoted 9 lines
- "Alone" is the sysbench CPU running alone on the core, no tag
- "nosmt" is both sysbench pinned on the same hardware thread, no tag
- "Tim's full patchset + sched" is an experiment with Tim's patchset combined with Aaron's "hack patch" to get rid of the remaining deep idle cases
- In all test cases, both tasks can run simultaneously (which was not the case without those patches), but the standard deviation is a pretty good indicator of the fairness/consistency.
This, and of course the numbers below too, is very interesting.
So, here comes my question: I've done a benchmarking campaign (yes, I'll post numbers soon) using this branch:
https://github.com/digitalocean/linux-coresched.git vpillai/coresched-v3-v5.1.5-test
https://github.com/digitalocean/linux-coresched/tree/vpillai/coresched-v3-v5.1.5-test
Last commit:7feb1007f274 "Fix stalling of untagged processes competing with tagged processes"
Since I see that, in this thread, there are various patches being proposed and discussed... should I rerun my benchmarks with them applied? If yes, which ones? And is there, by any chance, one (or maybe more than one) updated git branch(es)?
Thanks in advance and Regards
Signature · 6 lines
Dario Faggioli, Ph.D http://about.me/dario.faggioli Virtualization Software Engineer SUSE Labs, SUSE https://www.suse.com/ ------------------------------------------------------------------- <<This happens because _I_ choose it to happen!>> (Raistlin Majere)
Quoted 10 lines · Dario Faggioli
So, here comes my question: I've done a benchmarking campaign (yes, I'll post numbers soon) using this branch:
https://github.com/digitalocean/linux-coresched.git vpillai/coresched-v3-v5.1.5-test
https://github.com/digitalocean/linux-coresched/tree/vpillai/coresched-v3-v5.1.5-testLast commit:
7feb1007f274"Fix stalling of untagged processes competing with tagged processes"
Quoted 7 lines
Since I see that, in this thread, there are various patches being proposed and discussed... should I rerun my benchmarks with them applied? If yes, which ones? And is there, by any chance, one (or maybe more than one) updated git branch(es)?
Thanks in advance and Regards
Hi Dario,
Having an extra set of eyes are certainly welcomed. I'll give my 2 cents on the issues with v3. Others feel free to chime in if my understanding is incorrect or I'm missing something.
1) Unfairness between the sibling threads
-----------------------------------------
One sibling thread could be suppressing and force idling the sibling thread over proportionally. Resulting in the force idled CPU not getting run and stall tasks on suppressed CPU.
Status:
i) Aaron has proposed a patchset here based on using one rq as a base reference for vruntime for task priority comparison between siblings.
https://lore.kernel.org/lkml/20190725143248.GC992@aaronlu/
It works well on fairness but has some initialization issues
ii) Tim has proposed a patchset here to account for forced idle time in rq's min_vruntime
https://lore.kernel.org/lkml/[email protected]/
It improves over v3 with simpler logic compared to Aaron's patch, but does not work as well on fairness
iii) Tim has proposed yet another patch to maintain fairness of forced idle time between CPU threads per Peter's suggestion.
https://lore.kernel.org/lkml/[email protected]/
Its performance has yet to be tested.
2) Not rescheduling forced idled CPU
------------------------------------
The forced idled CPU does not get a chance to re-schedule itself, and will stall for a long time even though it has eligible tasks to run.
Status:
i) Aaron proposed a patch to fix this to check if there are runnable tasks when scheduling tick comes in.
https://lore.kernel.org/lkml/20190725143344.GD992@aaronlu/
ii) Vineeth has patches to this issue and also issue 1, based on scheduling in a new "forced idle task" when getting forced idle, but has yet to post the patches.
3) Load balancing between CPU cores
-----------------------------------
Say if one CPU core's sibling threads get forced idled a lot as it has mostly incompatible tasks between the siblings, moving the incompatible load to other cores and pulling compatible load to the core could help CPU utilization.
So just considering the load of a task is not enough during load balancing, task compatibility also needs to be considered. Peter has put in mechanisms to balance compatible tasks between CPU thread siblings, but not across cores.
Status:
I have not seen patches on this issue. This issue could lead to large variance in workload performance based on your luck in placing the workload among the cores.
Thanks.
Tim
kernel/sched/core.c static struct task_struct *sched_core_next(struct task_struct *p, unsigned long · line 219
Quoted 10 lines · Tim Chen
kernel/sched/core.c +10
-0+void account_core_idletime (struct task_struct *p,u64 exec) +{ +const struct cpumask *smt_mask; +struct rq *rq; +bool force_idle, refill; +int i, cpu; + + rq =task_rq (p); +if (!sched_core_enabled (rq) || !p->core_cookie) +return ;
I don't see why return here for untagged task. Untagged task can also preempt tagged task and force a CPU thread enter idle state. Untagged is just another tag to me, unless we want to allow untagged task to coschedule with a tagged task.
Quoted 26 lines
kernel/sched/core.c +26
-0+ cpu =task_cpu (p); + force_idle = false; + refill = true; + smt_mask =cpu_smt_mask (cpu); + +for_each_cpu (i, smt_mask) { +if (cpu == i) +continue ; + +if (cpu_rq (i)->core_forceidle) + force_idle = true; + +/* Only refill if everyone has run out of allowance */ +if (cpu_rq (i)->core_idle_allowance >0 ) + refill = false; + } + +if (force_idle) + rq->core_idle_allowance -= (s64 ) exec; + +if (rq->core_idle_allowance <0 && refill) { +for_each_cpu (i, smt_mask) { +cpu_rq (i)->core_idle_allowance += (s64 )SCHED_IDLE_ALLOWANCE ; + } + } +}
tip-bot for Peter Zijlstra
merge bot[tip:sched/core] stop_machine: Fix stop_cpus_in_progress ordering
In reply to Vineeth Remanan Pillaistop_machine: Fix stop_cpus_in_progress ordering
Commit-ID: 99d84bf8c65a7a0dbc9e166ca0a58ed949ac4f37 Gitweb: https://git.kernel.org/tip/99d84bf8c65a7a0dbc9e166ca0a58ed949ac4f37 Author: Peter Zijlstra <[email protected]> AuthorDate: Wed, 29 May 2019 20:36:37 +0000 Committer: Peter Zijlstra <[email protected]> CommitDate: Thu, 8 Aug 2019 09:09:30 +0200
stop_machine: Fix stop_cpus_in_progress ordering
Make sure the entire for loop has stop_cpus_in_progress set.
--- kernel/stop_machine.c | 2 ++ 1 file changed, 2 insertions(+)
kernel/stop_machine.c +2 -0
@@ -383,6 +383,7 @@ static bool queue_stop_cpus_work(const struct cpumask *cpumask,*/ preempt_disable (); stop_cpus_in_progress = true; +barrier ();for_each_cpu (cpu, cpumask) { work = &per_cpu (cpu_stopper.stop_work, cpu); work->fn = fn;
@@ -391,6 +392,7 @@ static bool queue_stop_cpus_work(const struct cpumask *cpumask,if (cpu_stop_queue_work (cpu, work)) queued = true; } +barrier (); stop_cpus_in_progress = false;preempt_enable ();
tip-bot for Peter Zijlstra
merge bot[tip:sched/core] sched: Fix kerneldoc comment for ia64_set_curr_task
In reply to Vineeth Remanan Pillaisched: Fix kerneldoc comment for ia64_set_curr_task
Commit-ID: 5feeb7837a448f659e0aaa19fb446b1d9a4b323a Gitweb: https://git.kernel.org/tip/5feeb7837a448f659e0aaa19fb446b1d9a4b323a Author: Peter Zijlstra <[email protected]> AuthorDate: Wed, 29 May 2019 20:36:38 +0000 Committer: Peter Zijlstra <[email protected]> CommitDate: Thu, 8 Aug 2019 09:09:30 +0200
sched: Fix kerneldoc comment for ia64_set_curr_task
--- kernel/sched/core.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/sched/core.c +1 -1
@@ -6772,7 +6772,7 @@ struct task_struct *curr_task(int cpu)#ifdef CONFIG_IA64 /** -* set_curr_task - set the current task for a given CPU. +* ia64_set_curr_task - set the current task for a given CPU. * @cpu: the processor in question. * @p: the task pointer to set. *
tip-bot for Peter Zijlstra
merge bot[tip:sched/core] sched/{rt,deadline}: Fix set_next_task vs pick_next_task
In reply to Vineeth Remanan Pillaisched/{rt,deadline}: Fix set_next_task vs pick_next_task
Commit-ID: f95d4eaee6d0207bff2dc93371133d31227d4cfb Gitweb: https://git.kernel.org/tip/f95d4eaee6d0207bff2dc93371133d31227d4cfb Author: Peter Zijlstra <[email protected]> AuthorDate: Wed, 29 May 2019 20:36:40 +0000 Committer: Peter Zijlstra <[email protected]> CommitDate: Thu, 8 Aug 2019 09:09:30 +0200
sched/{rt,deadline}: Fix set_next_task vs pick_next_task
Because pick_next_task() implies set_curr_task() and some of the details haven't mattered too much, some of what _should_ be in set_curr_task() ended up in pick_next_task, correct this.
This prepares the way for a pick_next_task() variant that does not affect the current state; allowing remote picking.
--- kernel/sched/deadline.c | 22 +++++++++++----------- kernel/sched/rt.c | 26 +++++++++++++------------- 2 files changed, 24 insertions(+), 24 deletions(-)
kernel/sched/deadline.c +11 -11
@@ -1727,12 +1727,20 @@ static void start_hrtick_dl(struct rq *rq, struct task_struct *p) }#endif -static inline void set_next_task (struct rq *rq,struct task_struct *p) +static void set_next_task_dl (struct rq *rq,struct task_struct *p) { p->se.exec_start =rq_clock_task (rq);/* You can't push away the running task */ dequeue_pushable_dl_task (rq, p); + +if (hrtick_enabled (rq)) +start_hrtick_dl (rq, p); + +if (rq->curr->sched_class != &dl_sched_class) +update_dl_rq_load_avg (rq_clock_pelt (rq), rq,0 ); + +deadline_queue_push_tasks (rq); } static struct sched_dl_entity *pick_next_dl_entity(struct rq *rq,
@@ -1791,15 +1799,7 @@ pick_next_task_dl(struct rq *rq, struct task_struct *prev, struct rq_flags *rf) p =dl_task_of (dl_se); -set_next_task (rq, p); - -if (hrtick_enabled (rq)) -start_hrtick_dl (rq, p); - -deadline_queue_push_tasks (rq); - -if (rq->curr->sched_class != &dl_sched_class) -update_dl_rq_load_avg (rq_clock_pelt (rq), rq,0 ); +set_next_task_dl (rq, p);return p; }
@@ -1846,7 +1846,7 @@ static void task_fork_dl(struct task_struct *p)static void set_curr_task_dl (struct rq *rq) { -set_next_task (rq, rq->curr); +set_next_task_dl (rq, rq->curr); }#ifdef CONFIG_SMP
kernel/sched/rt.c +13 -13
@@ -1498,12 +1498,22 @@ static void check_preempt_curr_rt(struct rq *rq, struct task_struct *p, int flag#endif } -static inline void set_next_task (struct rq *rq,struct task_struct *p) +static inline void set_next_task_rt (struct rq *rq,struct task_struct *p) { p->se.exec_start =rq_clock_task (rq);/* The running task is never eligible for pushing */ dequeue_pushable_task (rq, p); + +/* +* If prev task was rt, put_prev_task() has already updated the +* utilization. We only care of the case where we start to schedule a +* rt task +*/ +if (rq->curr->sched_class != &rt_sched_class) +update_rt_rq_load_avg (rq_clock_pelt (rq), rq,0 ); + +rt_queue_push_tasks (rq); } static struct sched_rt_entity *pick_next_rt_entity(struct rq *rq,
@@ -1577,17 +1587,7 @@ pick_next_task_rt(struct rq *rq, struct task_struct *prev, struct rq_flags *rf) p =_pick_next_task_rt (rq); -set_next_task (rq, p); - -rt_queue_push_tasks (rq); - -/* -* If prev task was rt, put_prev_task() has already updated the -* utilization. We only care of the case where we start to schedule a -* rt task -*/ -if (rq->curr->sched_class != &rt_sched_class) -update_rt_rq_load_avg (rq_clock_pelt (rq), rq,0 ); +set_next_task_rt (rq, p);return p; }
@@ -2356,7 +2356,7 @@ static void task_tick_rt(struct rq *rq, struct task_struct *p, int queued)static void set_curr_task_rt (struct rq *rq) { -set_next_task (rq, rq->curr); +set_next_task_rt (rq, rq->curr); }static unsigned int get_rr_interval_rt (struct rq *rq,struct task_struct *task)
tip-bot for Peter Zijlstra
merge bot[tip:sched/core] sched: Add task_struct pointer to sched_class::set_curr_task
In reply to Vineeth Remanan Pillaisched: Add task_struct pointer to sched_class::set_curr_task
Commit-ID: 03b7fad167efca3b7abbbb39733933f9df56e79c Gitweb: https://git.kernel.org/tip/03b7fad167efca3b7abbbb39733933f9df56e79c Author: Peter Zijlstra <[email protected]> AuthorDate: Wed, 29 May 2019 20:36:41 +0000 Committer: Peter Zijlstra <[email protected]> CommitDate: Thu, 8 Aug 2019 09:09:31 +0200
sched: Add task_struct pointer to sched_class::set_curr_task
In preparation of further separating pick_next_task() and set_curr_task() we have to pass the actual task into it, while there, rename the thing to better pair with put_prev_task().
--- kernel/sched/core.c | 12 ++++++------ kernel/sched/deadline.c | 7 +------ kernel/sched/fair.c | 17 ++++++++++++++--- kernel/sched/idle.c | 27 +++++++++++++++------------ kernel/sched/rt.c | 7 +------ kernel/sched/sched.h | 7 ++++--- kernel/sched/stop_task.c | 17 +++++++---------- 7 files changed, 48 insertions(+), 46 deletions(-)
kernel/sched/core.c +6 -6
@@ -1494,7 +1494,7 @@ void do_set_cpus_allowed(struct task_struct *p, const struct cpumask *new_mask)if (queued)enqueue_task (rq, p,ENQUEUE_RESTORE |ENQUEUE_NOCLOCK );if (running) -set_curr_task (rq, p); +set_next_task (rq, p); }/*
@@ -4325,7 +4325,7 @@ void rt_mutex_setprio(struct task_struct *p, struct task_struct *pi_task)if (queued)enqueue_task (rq, p, queue_flag);if (running) -set_curr_task (rq, p); +set_next_task (rq, p);check_class_changed (rq, p, prev_class, oldprio); out_unlock:
@@ -4392,7 +4392,7 @@ void set_user_nice(struct task_struct *p, long nice)resched_curr (rq); }if (running) -set_curr_task (rq, p); +set_next_task (rq, p); out_unlock:task_rq_unlock (rq, p, &rf); }
@@ -4840,7 +4840,7 @@ change:enqueue_task (rq, p, queue_flags); }if (running) -set_curr_task (rq, p); +set_next_task (rq, p);check_class_changed (rq, p, prev_class, oldprio);
@@ -6042,7 +6042,7 @@ void sched_setnuma(struct task_struct *p, int nid)if (queued)enqueue_task (rq, p,ENQUEUE_RESTORE |ENQUEUE_NOCLOCK );if (running) -set_curr_task (rq, p); +set_next_task (rq, p); task_rq_unlock(rq, p, &rf); }#endif /* CONFIG_NUMA_BALANCING */
@@ -6919,7 +6919,7 @@ void sched_move_task(struct task_struct *tsk)if (queued)enqueue_task (rq, tsk, queue_flags);if (running) -set_curr_task (rq, tsk); +set_next_task (rq, tsk);task_rq_unlock (rq, tsk, &rf); }
kernel/sched/deadline.c +1 -6
@@ -1844,11 +1844,6 @@ static void task_fork_dl(struct task_struct *p)*/ } -static void set_curr_task_dl (struct rq *rq) -{ -set_next_task_dl (rq, rq->curr); -} -#ifdef CONFIG_SMP /* Only try algorithms three times */
@@ -2466,6 +2461,7 @@ const struct sched_class dl_sched_class = { .pick_next_task = pick_next_task_dl, .put_prev_task = put_prev_task_dl, + .set_next_task = set_next_task_dl, #ifdef CONFIG_SMP .select_task_rq = select_task_rq_dl,
@@ -2476,7 +2472,6 @@ const struct sched_class dl_sched_class = { .task_woken = task_woken_dl, #endif - .set_curr_task = set_curr_task_dl, .task_tick = task_tick_dl, .task_fork = task_fork_dl,
kernel/sched/fair.c +14 -3
@@ -10150,9 +10150,19 @@ static void switched_to_fair(struct rq *rq, struct task_struct *p)* This routine is mostly called to set cfs_rq->curr field when a task * migrates between groups/classes. */ -static void set_curr_task_fair (struct rq *rq) +static void set_next_task_fair (struct rq *rq,struct task_struct *p) { -struct sched_entity *se = &rq->curr->se; +struct sched_entity *se = &p->se; + +#ifdef CONFIG_SMP +if (task_on_rq_queued (p)) { +/* +* Move the next running task to the front of the list, so our +* cfs_tasks list becomes MRU one. +*/ +list_move (&se->group_node, &rq->cfs_tasks); + } +#endif for_each_sched_entity (se) {struct cfs_rq *cfs_rq =cfs_rq_of (se);
@@ -10423,7 +10433,9 @@ const struct sched_class fair_sched_class = { .check_preempt_curr = check_preempt_wakeup, .pick_next_task = pick_next_task_fair, + .put_prev_task = put_prev_task_fair, + .set_next_task = set_next_task_fair, #ifdef CONFIG_SMP .select_task_rq = select_task_rq_fair,
@@ -10436,7 +10448,6 @@ const struct sched_class fair_sched_class = { .set_cpus_allowed = set_cpus_allowed_common, #endif - .set_curr_task = set_curr_task_fair, .task_tick = task_tick_fair, .task_fork = task_fork_fair,
kernel/sched/idle.c +15 -12
@@ -374,14 +374,25 @@ static void check_preempt_curr_idle(struct rq *rq, struct task_struct *p, int flresched_curr (rq); } +static void put_prev_task_idle (struct rq *rq,struct task_struct *prev) +{ +} + +static void set_next_task_idle (struct rq *rq,struct task_struct *next) +{ +update_idle_core (rq); +schedstat_inc (rq->sched_goidle); +} +static struct task_struct *pick_next_task_idle (struct rq *rq,struct task_struct *prev,struct rq_flags *rf) { +struct task_struct *next = rq->idle; +put_prev_task (rq, prev); -update_idle_core (rq); -schedstat_inc (rq->sched_goidle); +set_next_task_idle (rq, next); -return rq->idle; +return next; }/*
@@ -397,10 +408,6 @@ dequeue_task_idle(struct rq *rq, struct task_struct *p, int flags)raw_spin_lock_irq (&rq->lock); } -static void put_prev_task_idle (struct rq *rq,struct task_struct *prev) -{ -} -/* * scheduler tick hitting a task of our scheduling class. *
@@ -413,10 +420,6 @@ static void task_tick_idle(struct rq *rq, struct task_struct *curr, int queued) { } -static void set_curr_task_idle (struct rq *rq) -{ -} -static void switched_to_idle (struct rq *rq,struct task_struct *p) {BUG ();
@@ -451,13 +454,13 @@ const struct sched_class idle_sched_class = { .pick_next_task = pick_next_task_idle, .put_prev_task = put_prev_task_idle, + .set_next_task = set_next_task_idle, #ifdef CONFIG_SMP .select_task_rq = select_task_rq_idle, .set_cpus_allowed = set_cpus_allowed_common, #endif - .set_curr_task = set_curr_task_idle, .task_tick = task_tick_idle, .get_rr_interval = get_rr_interval_idle,
kernel/sched/rt.c +1 -6
@@ -2354,11 +2354,6 @@ static void task_tick_rt(struct rq *rq, struct task_struct *p, int queued) } } -static void set_curr_task_rt (struct rq *rq) -{ -set_next_task_rt (rq, rq->curr); -} -static unsigned int get_rr_interval_rt (struct rq *rq,struct task_struct *task) {/*
@@ -2380,6 +2375,7 @@ const struct sched_class rt_sched_class = { .pick_next_task = pick_next_task_rt, .put_prev_task = put_prev_task_rt, + .set_next_task = set_next_task_rt, #ifdef CONFIG_SMP .select_task_rq = select_task_rq_rt,
@@ -2391,7 +2387,6 @@ const struct sched_class rt_sched_class = { .switched_from = switched_from_rt, #endif - .set_curr_task = set_curr_task_rt, .task_tick = task_tick_rt, .get_rr_interval = get_rr_interval_rt,
kernel/sched/sched.h +4 -3
@@ -1707,6 +1707,7 @@ struct sched_class {struct task_struct *prev, struct rq_flags *rf);void (*put_prev_task)(struct rq *rq,struct task_struct *p); +void (*set_next_task)(struct rq *rq,struct task_struct *p);#ifdef CONFIG_SMP int (*select_task_rq)(struct task_struct *p,int task_cpu,int sd_flag,int flags);
@@ -1721,7 +1722,6 @@ struct sched_class {void (*rq_offline)(struct rq *rq);#endif -void (*set_curr_task)(struct rq *rq);void (*task_tick)(struct rq *rq,struct task_struct *p,int queued);void (*task_fork)(struct task_struct *p);void (*task_dead)(struct task_struct *p);
@@ -1755,9 +1755,10 @@ static inline void put_prev_task(struct rq *rq, struct task_struct *prev) prev->sched_class->put_prev_task (rq, prev); } -static inline void set_curr_task (struct rq *rq,struct task_struct *curr) +static inline void set_next_task (struct rq *rq,struct task_struct *next) { - curr->sched_class->set_curr_task (rq); +WARN_ON_ONCE (rq->curr != next); + next->sched_class->set_next_task (rq, next); }#ifdef CONFIG_SMP
kernel/sched/stop_task.c +7 -10
@@ -23,6 +23,11 @@ check_preempt_curr_stop(struct rq *rq, struct task_struct *p, int flags)/* we're never preempted */ } +static void set_next_task_stop (struct rq *rq,struct task_struct *stop) +{ + stop->se.exec_start =rq_clock_task (rq); +} +static struct task_struct *pick_next_task_stop (struct rq *rq,struct task_struct *prev,struct rq_flags *rf) {
@@ -32,8 +37,7 @@ pick_next_task_stop(struct rq *rq, struct task_struct *prev, struct rq_flags *rfreturn NULL ;put_prev_task (rq, prev); - - stop->se.exec_start =rq_clock_task (rq); +set_next_task_stop (rq, stop);return stop; }
@@ -86,13 +90,6 @@ static void task_tick_stop(struct rq *rq, struct task_struct *curr, int queued) { } -static void set_curr_task_stop (struct rq *rq) -{ -struct task_struct *stop = rq->stop; - - stop->se.exec_start =rq_clock_task (rq); -} -static void switched_to_stop (struct rq *rq,struct task_struct *p) {BUG ();/* its impossible to change to this class */
@@ -128,13 +125,13 @@ const struct sched_class stop_sched_class = { .pick_next_task = pick_next_task_stop, .put_prev_task = put_prev_task_stop, + .set_next_task = set_next_task_stop, #ifdef CONFIG_SMP .select_task_rq = select_task_rq_stop, .set_cpus_allowed = set_cpus_allowed_common, #endif - .set_curr_task = set_curr_task_stop, .task_tick = task_tick_stop, .get_rr_interval = get_rr_interval_stop,
tip-bot for Peter Zijlstra
merge bot[tip:sched/core] sched/fair: Expose newidle_balance()
In reply to Vineeth Remanan Pillaisched/fair: Expose newidle_balance()
Commit-ID: 5ba553eff0c3a7c099b1e29a740277a82c0c3314 Gitweb: https://git.kernel.org/tip/5ba553eff0c3a7c099b1e29a740277a82c0c3314 Author: Peter Zijlstra <[email protected]> AuthorDate: Wed, 29 May 2019 20:36:42 +0000 Committer: Peter Zijlstra <[email protected]> CommitDate: Thu, 8 Aug 2019 09:09:31 +0200
sched/fair: Expose newidle_balance()
For pick_next_task_fair() it is the newidle balance that requires dropping the rq->lock; provided we do put_prev_task() early, we can also detect the condition for doing newidle early.
--- kernel/sched/fair.c | 18 ++++++++---------- kernel/sched/sched.h | 4 ++++ 2 files changed, 12 insertions(+), 10 deletions(-)
kernel/sched/fair.c +8 -10
@@ -3690,8 +3690,6 @@ static inline unsigned long cfs_rq_load_avg(struct cfs_rq *cfs_rq)return cfs_rq->avg.load_avg; } -static int idle_balance (struct rq *this_rq,struct rq_flags *rf); -static inline unsigned long task_util (struct task_struct *p) {return READ_ONCE (p->se.avg.util_avg);
@@ -6878,11 +6876,10 @@ done: __maybe_unused;return p; idle: -update_misfit_status (NULL , rq); - new_tasks =idle_balance (rq, rf); + new_tasks =newidle_balance (rq, rf);/* -* Because idle_balance() releases (and re-acquires) rq->lock, it is +* Because newidle_balance() releases (and re-acquires) rq->lock, it is * possible for any higher priority task to appear. In that case we * must re-start the pick_next_entity() loop. */
@@ -9045,10 +9042,10 @@ out_one_pinned: ld_moved =0 ;/* -* idle_balance() disregards balance intervals, so we could repeatedly -* reach this code, which would lead to balance_interval skyrocketting -* in a short amount of time. Skip the balance_interval increase logic -* to avoid that. +* newidle_balance() disregards balance intervals, so we could +* repeatedly reach this code, which would lead to balance_interval +* skyrocketting in a short amount of time. Skip the balance_interval +* increase logic to avoid that. */ if (env.idle ==CPU_NEWLY_IDLE ) goto out;
@@ -9758,7 +9755,7 @@ static inline void nohz_newidle_balance(struct rq *this_rq) { }* idle_balance is called by schedule() if this_cpu is about to become * idle. Attempts to pull tasks from other CPUs. */ -static int idle_balance (struct rq *this_rq,struct rq_flags *rf) +int newidle_balance (struct rq *this_rq,struct rq_flags *rf) {unsigned long next_balance = jiffies +HZ ;int this_cpu = this_rq->cpu;
@@ -9766,6 +9763,7 @@ static int idle_balance(struct rq *this_rq, struct rq_flags *rf)int pulled_task =0 ;u64 curr_cost =0 ; +update_misfit_status (NULL , this_rq);/* * We must set idle_stamp _before_ calling idle_balance(), such that we * measure the duration of idle_balance() as idle time.
kernel/sched/sched.h +4 -0
@@ -1445,10 +1445,14 @@ static inline void unregister_sched_domain_sysctl(void) }#endif +extern int newidle_balance (struct rq *this_rq,struct rq_flags *rf); +#else static inline void sched_ttwu_pending (void ) { } +static inline int newidle_balance (struct rq *this_rq,struct rq_flags *rf) {return 0 ; } +#endif /* CONFIG_SMP */ #include "stats.h"
tip-bot for Peter Zijlstra
merge bot[tip:sched/core] sched: Allow put_prev_task() to drop rq->lock
In reply to Vineeth Remanan Pillaisched: Allow put_prev_task() to drop rq->lock
Commit-ID: 5f2a45fc9e89e022233085e6f0f352eb6ff770bb Gitweb: https://git.kernel.org/tip/5f2a45fc9e89e022233085e6f0f352eb6ff770bb Author: Peter Zijlstra <[email protected]> AuthorDate: Wed, 29 May 2019 20:36:43 +0000 Committer: Peter Zijlstra <[email protected]> CommitDate: Thu, 8 Aug 2019 09:09:31 +0200
sched: Allow put_prev_task() to drop rq->lock
Currently the pick_next_task() loop is convoluted and ugly because of how it can drop the rq->lock and needs to restart the picking.
For the RT/Deadline classes, it is put_prev_task() where we do balancing, and we could do this before the picking loop. Make this possible.
--- kernel/sched/core.c | 2 +- kernel/sched/deadline.c | 14 +++++++++++++- kernel/sched/fair.c | 2 +- kernel/sched/idle.c | 2 +- kernel/sched/rt.c | 14 +++++++++++++- kernel/sched/sched.h | 4 ++-- kernel/sched/stop_task.c | 2 +- 7 files changed, 32 insertions(+), 8 deletions(-)
kernel/sched/core.c +1 -1
@@ -6090,7 +6090,7 @@ static struct task_struct *__pick_migrate_task(struct rq *rq)for_each_class (class) { next = class->pick_next_task (rq,NULL ,NULL );if (next) { - next->sched_class->put_prev_task (rq, next); + next->sched_class->put_prev_task (rq, next,NULL );return next; } }
kernel/sched/deadline.c +13 -1
@@ -1804,13 +1804,25 @@ pick_next_task_dl(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)return p; } -static void put_prev_task_dl (struct rq *rq,struct task_struct *p) +static void put_prev_task_dl (struct rq *rq,struct task_struct *p,struct rq_flags *rf) {update_curr_dl (rq);update_dl_rq_load_avg (rq_clock_pelt (rq), rq,1 );if (on_dl_rq (&p->dl) && p->nr_cpus_allowed >1 )enqueue_pushable_dl_task (rq, p); + +if (rf && !on_dl_rq (&p->dl) &&need_pull_dl_task (rq, p)) { +/* +* This is OK, because current is on_cpu, which avoids it being +* picked for load-balance and preemption/IRQs are still +* disabled avoiding further scheduler activity on it and we've +* not yet started the picking loop. +*/ +rq_unpin_lock (rq, rf); +pull_dl_task (rq); +rq_repin_lock (rq, rf); + } }/*
kernel/sched/fair.c +1 -1
@@ -6901,7 +6901,7 @@ idle:/* * Account for a descheduled task: */ -static void put_prev_task_fair (struct rq *rq,struct task_struct *prev) +static void put_prev_task_fair (struct rq *rq,struct task_struct *prev,struct rq_flags *rf) {struct sched_entity *se = &prev->se;struct cfs_rq *cfs_rq;
kernel/sched/idle.c +1 -1
@@ -374,7 +374,7 @@ static void check_preempt_curr_idle(struct rq *rq, struct task_struct *p, int flresched_curr (rq); } -static void put_prev_task_idle (struct rq *rq,struct task_struct *prev) +static void put_prev_task_idle (struct rq *rq,struct task_struct *prev,struct rq_flags *rf) { }
kernel/sched/rt.c +13 -1
@@ -1592,7 +1592,7 @@ pick_next_task_rt(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)return p; } -static void put_prev_task_rt (struct rq *rq,struct task_struct *p) +static void put_prev_task_rt (struct rq *rq,struct task_struct *p,struct rq_flags *rf) {update_curr_rt (rq);
@@ -1604,6 +1604,18 @@ static void put_prev_task_rt(struct rq *rq, struct task_struct *p)*/ if (on_rt_rq (&p->rt) && p->nr_cpus_allowed >1 )enqueue_pushable_task (rq, p); + +if (rf && !on_rt_rq (&p->rt) &&need_pull_rt_task (rq, p)) { +/* +* This is OK, because current is on_cpu, which avoids it being +* picked for load-balance and preemption/IRQs are still +* disabled avoiding further scheduler activity on it and we've +* not yet started the picking loop. +*/ +rq_unpin_lock (rq, rf); +pull_rt_task (rq); +rq_repin_lock (rq, rf); + } }#ifdef CONFIG_SMP
kernel/sched/sched.h +2 -2
@@ -1710,7 +1710,7 @@ struct sched_class {struct task_struct * (*pick_next_task)(struct rq *rq,struct task_struct *prev,struct rq_flags *rf); -void (*put_prev_task)(struct rq *rq,struct task_struct *p); +void (*put_prev_task)(struct rq *rq,struct task_struct *p,struct rq_flags *rf);void (*set_next_task)(struct rq *rq,struct task_struct *p);#ifdef CONFIG_SMP
@@ -1756,7 +1756,7 @@ struct sched_class {static inline void put_prev_task (struct rq *rq,struct task_struct *prev) {WARN_ON_ONCE (rq->curr != prev); - prev->sched_class->put_prev_task (rq, prev); + prev->sched_class->put_prev_task (rq, prev,NULL ); }static inline void set_next_task (struct rq *rq,struct task_struct *next)
kernel/sched/stop_task.c +1 -1
@@ -59,7 +59,7 @@ static void yield_task_stop(struct rq *rq)BUG ();/* the stop task should never yield, its pointless. */ } -static void put_prev_task_stop (struct rq *rq,struct task_struct *prev) +static void put_prev_task_stop (struct rq *rq,struct task_struct *prev,struct rq_flags *rf) {struct task_struct *curr = rq->curr;u64 delta_exec;
tip-bot for Peter Zijlstra
merge bot[tip:sched/core] sched: Rework pick_next_task() slow-path
In reply to Vineeth Remanan Pillaisched: Rework pick_next_task() slow-path
Commit-ID: 67692435c411e5c53a1c588ecca2037aebd81f2e Gitweb: https://git.kernel.org/tip/67692435c411e5c53a1c588ecca2037aebd81f2e Author: Peter Zijlstra <[email protected]> AuthorDate: Wed, 29 May 2019 20:36:44 +0000 Committer: Peter Zijlstra <[email protected]> CommitDate: Thu, 8 Aug 2019 09:09:31 +0200
sched: Rework pick_next_task() slow-path
Avoid the RETRY_TASK case in the pick_next_task() slow path.
By doing the put_prev_task() early, we get the rt/deadline pull done, and by testing rq->nr_running we know if we need newidle_balance().
This then gives a stable state to pick a task from.
Since the fast-path is fair only; it means the other classes will always have pick_next_task(.prev=NULL, .rf=NULL) and we can simplify.
--- kernel/sched/core.c | 19 ++++++++++++------- kernel/sched/deadline.c | 30 ++---------------------------- kernel/sched/fair.c | 9 ++++++--- kernel/sched/idle.c | 4 +++- kernel/sched/rt.c | 29 +---------------------------- kernel/sched/sched.h | 13 ++++++++----- kernel/sched/stop_task.c | 3 ++- 7 files changed, 34 insertions(+), 73 deletions(-)
kernel/sched/core.c +12 -7
@@ -3791,7 +3791,7 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf) p = fair_sched_class.pick_next_task (rq, prev, rf);if (unlikely (p ==RETRY_TASK )) - goto again; + goto restart;/* Assumes fair_sched_class->next == idle_sched_class */ if (unlikely (!p))
@@ -3800,14 +3800,19 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)return p; } -again: +restart: +/* +* Ensure that we put DL/RT tasks before the pick loop, such that they +* can PULL higher prio tasks when we lower the RQ 'priority'. +*/ + prev->sched_class->put_prev_task (rq, prev, rf); +if (!rq->nr_running) +newidle_balance (rq, rf); +for_each_class (class) { - p = class->pick_next_task (rq, prev, rf); -if (p) { -if (unlikely (p ==RETRY_TASK )) - goto again; + p = class->pick_next_task (rq,NULL ,NULL ); +if (p)return p; - } }/* The idle class should always have a runnable task: */
kernel/sched/deadline.c +2 -28
@@ -1761,39 +1761,13 @@ pick_next_task_dl(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)struct task_struct *p;struct dl_rq *dl_rq; - dl_rq = &rq->dl; - -if (need_pull_dl_task (rq, prev)) { -/* -* This is OK, because current is on_cpu, which avoids it being -* picked for load-balance and preemption/IRQs are still -* disabled avoiding further scheduler activity on it and we're -* being very careful to re-start the picking loop. -*/ -rq_unpin_lock (rq, rf); -pull_dl_task (rq); -rq_repin_lock (rq, rf); -/* -* pull_dl_task() can drop (and re-acquire) rq->lock; this -* means a stop task can slip in, in which case we need to -* re-start task selection. -*/ -if (rq->stop &&task_on_rq_queued (rq->stop)) -return RETRY_TASK ; - } +WARN_ON_ONCE (prev || rf); -/* -* When prev is DL, we may throttle it in put_prev_task(). -* So, we update time before we check for dl_nr_running. -*/ -if (prev->sched_class == &dl_sched_class) -update_curr_dl (rq); + dl_rq = &rq->dl;if (unlikely (!dl_rq->dl_nr_running))return NULL ; -put_prev_task (rq, prev); - dl_se =pick_next_dl_entity (rq, dl_rq);BUG_ON (!dl_se);
kernel/sched/fair.c +6 -3
@@ -6770,7 +6770,7 @@ again: goto idle;#ifdef CONFIG_FAIR_GROUP_SCHED -if (prev->sched_class != &fair_sched_class) +if (!prev || prev->sched_class != &fair_sched_class) goto simple;/*
@@ -6847,8 +6847,8 @@ again: goto done; simple: #endif - - put_prev_task(rq, prev); + if (prev) + put_prev_task(rq, prev); do { se = pick_next_entity(cfs_rq, NULL);
@@ -6876,6 +6876,9 @@ done: __maybe_unused;return p; idle: +if (!rf) +return NULL ; + new_tasks =newidle_balance (rq, rf);/*
kernel/sched/idle.c +3 -1
@@ -389,7 +389,9 @@ pick_next_task_idle(struct rq *rq, struct task_struct *prev, struct rq_flags *rf {struct task_struct *next = rq->idle; -put_prev_task (rq, prev); +if (prev) +put_prev_task (rq, prev); +set_next_task_idle (rq, next);return next;
kernel/sched/rt.c +1 -28
@@ -1553,38 +1553,11 @@ pick_next_task_rt(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)struct task_struct *p;struct rt_rq *rt_rq = &rq->rt; -if (need_pull_rt_task (rq, prev)) { -/* -* This is OK, because current is on_cpu, which avoids it being -* picked for load-balance and preemption/IRQs are still -* disabled avoiding further scheduler activity on it and we're -* being very careful to re-start the picking loop. -*/ -rq_unpin_lock (rq, rf); -pull_rt_task (rq); -rq_repin_lock (rq, rf); -/* -* pull_rt_task() can drop (and re-acquire) rq->lock; this -* means a dl or stop task can slip in, in which case we need -* to re-start task selection. -*/ -if (unlikely ((rq->stop &&task_on_rq_queued (rq->stop)) || - rq->dl.dl_nr_running)) -return RETRY_TASK ; - } - -/* -* We may dequeue prev's rt_rq in put_prev_task(). -* So, we update time before rt_queued check. -*/ -if (prev->sched_class == &rt_sched_class) -update_curr_rt (rq); +WARN_ON_ONCE (prev || rf);if (!rt_rq->rt_queued)return NULL ; -put_prev_task (rq, prev); - p =_pick_next_task_rt (rq);set_next_task_rt (rq, p);
kernel/sched/sched.h +8 -5
@@ -1700,12 +1700,15 @@ struct sched_class {void (*check_preempt_curr)(struct rq *rq,struct task_struct *p,int flags);/* -* It is the responsibility of the pick_next_task() method that will -* return the next task to call put_prev_task() on the @prev task or -* something equivalent. +* Both @prev and @rf are optional and may be NULL, in which case the +* caller must already have invoked put_prev_task(rq, prev, rf). * -* May return RETRY_TASK when it finds a higher prio class has runnable -* tasks. +* Otherwise it is the responsibility of the pick_next_task() to call +* put_prev_task() on the @prev task or something equivalent, IFF it +* returns a next task. +* +* In that case (@rf != NULL) it may return RETRY_TASK when it finds a +* higher prio class has runnable tasks. */ struct task_struct * (*pick_next_task)(struct rq *rq, struct task_struct *prev,
kernel/sched/stop_task.c +2 -1
@@ -33,10 +33,11 @@ pick_next_task_stop(struct rq *rq, struct task_struct *prev, struct rq_flags *rf {struct task_struct *stop = rq->stop; +WARN_ON_ONCE (prev || rf); +if (!stop || !task_on_rq_queued (stop))return NULL ; -put_prev_task (rq, prev);set_next_task_stop (rq, stop);return stop;
Quoted 25 lines · Tim Chen
Quoted 15 lines · Julien Desfossez
We tested both Aaron's and Tim's patches and here are our results.
Test setup:
- 2 1-thread sysbench, one running the cpu benchmark, the other one the mem benchmark
- both started at the same time
- both are pinned on the same core (2 hardware threads)
- 10 30-seconds runs
- test script: https://paste.debian.net/plainh/834cf45c
- only showing the CPU events/sec (higher is better)
- tested 4 tag configurations:
- no tag
- sysbench mem untagged, sysbench cpu tagged
- sysbench mem tagged, sysbench cpu untagged
- both tagged with a different tagQuoted 8 lines
- "Alone" is the sysbench CPU running alone on the core, no tag
- "nosmt" is both sysbench pinned on the same hardware thread, no tag
- "Tim's full patchset + sched" is an experiment with Tim's patchset combined with Aaron's "hack patch" to get rid of the remaining deep idle cases
- In all test cases, both tasks can run simultaneously (which was not the case without those patches), but the standard deviation is a pretty good indicator of the fairness/consistency.
Quoted 6 lines
Thanks for testing the patches and giving such detailed data.
I came to realize that for my scheme, the accumulated deficit of forced idle could be wiped out in one execution of a task on the forced idle cpu, with the update of the
min_vruntime, even if the execution time could be far less than the accumulated deficit. That's probably one reason my scheme didn't achieve fairness.
Turns out there is a typo error in v3 when setting rq's core_forceidle:
kernel/sched/core.c +1 -1
@@ -3888,7 +3888,7 @@ next_class:;WARN_ON_ONCE (!rq_i->core_pick);if (is_idle_task (rq_i->core_pick) && rq_i->nr_running) - rq->core_forceidle = true; + rq_i->core_forceidle = true; rq_i->core_pick->core_occupation = occ;
With this fixed and together with the patch to let schedule always happen, your latest 2 patches work well for the 10s cpuhog test I described previously:
https://lore.kernel.org/lkml/20190725143003.GA992@aaronlu/
overloaded workload without any cpu binding doesn't work well though, I haven't taken a closer look yet.
Quoted 10 lines · Aaron Lu
Quoted 2 lines · Tim Chen
Quoted 1 line · Julien Desfossez
We tested both Aaron's and Tim's patches and here are our results.
kernel/sched/core.c +0
-0
kernel/sched/core.c next_class:; · line 3891
Quoted 5 lines
kernel/sched/core.c +1
-1@@ -3888,7 +3888,7 @@ next_class:;WARN_ON_ONCE (!rq_i->core_pick);if (is_idle_task (rq_i->core_pick) && rq_i->nr_running) - rq->core_forceidle = true; + rq_i->core_forceidle = true;
Good catch!
Quoted 7 lines
kernel/sched/core.c +0
-0rq_i->core_pick->core_occupation = occ;With this fixed and together with the patch to let schedule always happen, your latest 2 patches work well for the 10s cpuhog test I described previously:
https://lore.kernel.org/lkml/20190725143003.GA992@aaronlu/
That's encouraging. You are talking about my patches that try to keep the force idle time between sibling threads balanced, right?
Quoted 4 lines
overloaded workload without any cpu binding doesn't work well though, I haven't taken a closer look yet.
I think we need a load balancing scheme among the cores that will try to minimize force idle.
One possible metric to measure load compatibility imbalance that leads to force idle is
Say i, j are sibling threads of a cpu core
imbalanace = \sum_tagged_cgroup abs(Load_cgroup_cpui - Load_cgroup_cpuj)
This gives us a metric to decide if migrating a task will improve load compatability imbalance. As we already track cgroup load on a CPU, it should be doable without adding too much overhead.
Tim
Quoted 16 lines · Aaron Lu
kernel/sched/core.c static struct task_struct *sched_core_next(struct task_struct *p, unsigned long · line 219
Quoted 10 lines · Tim Chen
kernel/sched/core.c +10
-0+void account_core_idletime (struct task_struct *p,u64 exec) +{ +const struct cpumask *smt_mask; +struct rq *rq; +bool force_idle, refill; +int i, cpu; + + rq =task_rq (p); +if (!sched_core_enabled (rq) || !p->core_cookie) +return ;I don't see why return here for untagged task. Untagged task can also preempt tagged task and force a CPU thread enter idle state. Untagged is just another tag to me, unless we want to allow untagged task to coschedule with a tagged task.
You are right. This needs to be fixed.
And the cookie check will also need to be changed in prio_less_fair.
@@ -611,6 +611,17 @@ bool prio_less_fair(struct task_struct *a, struct task_struct *b)
* Normalize the vruntime if tasks are in different cpus.
*/
if (task_cpu(a) != task_cpu(b)) {
+
+ if (a->core_cookie && b->core_cookie &&
+ a->core_cookie != b->core_cookie) {
if (!cookie_match(a, b))
+ /*
+ * Will be force idling one thread,
+ * pick the thread that has more allowance.
+ */
+ return (task_rq(a)->core_idle_allowance <=
+ task_rq(b)->core_idle_allowance) ? true : false;
+ }
+
I'll respin my patches.
Tim
Quoted 18 lines · Tim Chen
Quoted 16 lines · Aaron Lu
Review
Quoted 10 lines · Tim Chen
(quoted diff) +10
-0+void account_core_idletime(struct task_struct *p, u64 exec) +{ + const struct cpumask *smt_mask; + struct rq *rq; + bool force_idle, refill; + int i, cpu; + + rq = task_rq(p); + if (!sched_core_enabled(rq) || !p->core_cookie) + return;I don't see why return here for untagged task. Untagged task can also preempt tagged task and force a CPU thread enter idle state. Untagged is just another tag to me, unless we want to allow untagged task to coschedule with a tagged task.
Quoted 2 lines
You are right. This needs to be fixed.
Here's the updated patchset, including Aaron's fix and also added accounting of force idle time by deadline and rt tasks.
Tim
-----------------patch 1----------------------
From 730dbb125f5f67c75f97f6be154d382767810f8b Mon Sep 17 00:00:00 2001
From: Aaron Lu <[email protected]>
Date: Thu, 8 Aug 2019 08:57:46 -0700
Subject: [PATCH 1/3 v2] sched: Fix incorrect rq tagged as forced idle
Incorrect run queue was tagged as forced idle.
Tag the correct one.
--- kernel/sched/core.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/sched/core.c +1 -1
@@ -3903,7 +3903,7 @@ next_class:;WARN_ON_ONCE (!rq_i->core_pick);if (is_idle_task (rq_i->core_pick) && rq_i->nr_running) - rq->core_forceidle = true; + rq_i->core_forceidle = true; rq_i->core_pick->core_occupation = occ;
Signature · 17 lines
2.20.1 --------------patch 2------------------------ From 263ceeb40843b8ca3a91f1b268bec2b836d4986b Mon Sep 17 00:00:00 2001 From: Tim Chen <[email protected]> Date: Wed, 24 Jul 2019 13:58:18 -0700 Subject: [PATCH 2/3 v2] sched: Move sched fair prio comparison to fair.c Consolidate the task priority comparison of the fair class to fair.c. A simple code reorganization and there are no functional changes. Signed-off-by: Tim Chen <[email protected]> --- kernel/sched/core.c | 21 ++------------------- kernel/sched/fair.c | 21 +++++++++++++++++++++ kernel/sched/sched.h | 1 + 3 files changed, 24 insertions(+), 19 deletions(-)
kernel/sched/core.c +2 -19
@@ -105,25 +105,8 @@ static inline bool prio_less(struct task_struct *a, struct task_struct *b)if (pa ==-1 )/* dl_prio() doesn't work because of stop_class above */ return !dl_time_before (a->dl.deadline, b->dl.deadline); -if (pa ==MAX_RT_PRIO +MAX_NICE ) {/* fair */ -u64 a_vruntime = a->se.vruntime; -u64 b_vruntime = b->se.vruntime; - -/* -* Normalize the vruntime if tasks are in different cpus. -*/ -if (task_cpu (a) !=task_cpu (b)) { - b_vruntime -=task_cfs_rq (b)->min_vruntime; - b_vruntime +=task_cfs_rq (a)->min_vruntime; - -trace_printk ("(%d:%Lu,%Lu,%Lu) <> (%d:%Lu,%Lu,%Lu)\n" , - a->pid, a_vruntime, a->se.vruntime,task_cfs_rq (a)->min_vruntime, - b->pid, b_vruntime, b->se.vruntime,task_cfs_rq (b)->min_vruntime); - - } - -return !((s64)(a_vruntime - b_vruntime) <=0 ); - } +if (pa ==MAX_RT_PRIO +MAX_NICE )/* fair */ +return prio_less_fair (a, b);return false; }
kernel/sched/fair.c +21 -0
@@ -602,6 +602,27 @@ static inline u64 calc_delta_fair(u64 delta, struct sched_entity *se)return delta; } +bool prio_less_fair (struct task_struct *a,struct task_struct *b) +{ +u64 a_vruntime = a->se.vruntime; +u64 b_vruntime = b->se.vruntime; + +/* +* Normalize the vruntime if tasks are in different cpus. +*/ +if (task_cpu (a) !=task_cpu (b)) { + b_vruntime -=task_cfs_rq (b)->min_vruntime; + b_vruntime +=task_cfs_rq (a)->min_vruntime; + +trace_printk ("(%d:%Lu,%Lu,%Lu) <> (%d:%Lu,%Lu,%Lu)\n" , + a->pid, a_vruntime, a->se.vruntime,task_cfs_rq (a)->min_vruntime, + b->pid, b_vruntime, b->se.vruntime,task_cfs_rq (b)->min_vruntime); + + } + +return !((s64)(a_vruntime - b_vruntime) <=0 ); +} +/* * The idea is to set a period in which each task runs once. *
kernel/sched/sched.h +1 -0
@@ -1015,6 +1015,7 @@ static inline raw_spinlock_t *rq_lockp(struct rq *rq) }extern void queue_core_balance (struct rq *rq); +extern bool prio_less_fair (struct task_struct *a,struct task_struct *b);#else /* !CONFIG_SCHED_CORE */
Signature · 20 lines
2.20.1 ------------------------patch 3--------------------------- From 5318e23c741a832140effbaf2f79fdf4b08f883c Mon Sep 17 00:00:00 2001 From: Tim Chen <[email protected]> Date: Tue, 6 Aug 2019 12:50:45 -0700 Subject: [PATCH 3/3 v2] sched: Enforce fairness between cpu threads CPU thread could be suppressed by its sibling for extended time. Implement a budget for force idling, making all CPU threads have equal chance to run. Signed-off-by: Tim Chen <[email protected]> --- kernel/sched/core.c | 43 +++++++++++++++++++++++++++++++++++++++++ kernel/sched/deadline.c | 1 + kernel/sched/fair.c | 11 +++++++++++ kernel/sched/rt.c | 1 + kernel/sched/sched.h | 4 ++++ 5 files changed, 60 insertions(+)
kernel/sched/core.c +43 -0
61 lines not on this page · raw message
kernel/sched/deadline.c +1 -0
@@ -1175,6 +1175,7 @@ static void update_curr_dl(struct rq *rq) curr->se.sum_exec_runtime += delta_exec;account_group_exec_runtime (curr, delta_exec); +account_core_idletime (curr, delta_exec); curr->se.exec_start = now;cgroup_account_cputime (curr, delta_exec);
kernel/sched/fair.c +11 -0
@@ -611,6 +611,16 @@ bool prio_less_fair(struct task_struct *a, struct task_struct *b)* Normalize the vruntime if tasks are in different cpus. */ if (task_cpu (a) !=task_cpu (b)) { + +if (a->core_cookie != b->core_cookie) { +/* +* Will be force idling one thread, +* pick the thread that has more allowance. +*/ +return (task_rq (a)->core_idle_allowance < +task_rq (b)->core_idle_allowance) ? true : false; + } + b_vruntime -=task_cfs_rq (b)->min_vruntime; b_vruntime +=task_cfs_rq (a)->min_vruntime;
@@ -817,6 +827,7 @@ static void update_curr(struct cfs_rq *cfs_rq)trace_sched_stat_runtime (curtask, delta_exec, curr->vruntime);cgroup_account_cputime (curtask, delta_exec);account_group_exec_runtime (curtask, delta_exec); +account_core_idletime (curtask, delta_exec); }account_cfs_rq_runtime (cfs_rq, delta_exec);
kernel/sched/rt.c +1 -0
@@ -971,6 +971,7 @@ static void update_curr_rt(struct rq *rq) curr->se.sum_exec_runtime += delta_exec;account_group_exec_runtime (curr, delta_exec); +account_core_idletime (curr, delta_exec); curr->se.exec_start = now;cgroup_account_cputime (curr, delta_exec);
kernel/sched/sched.h +4 -0
@@ -963,6 +963,7 @@ struct rq {struct task_struct *core_pick;unsigned int core_enabled;unsigned int core_sched_seq; +s64 core_idle_allowance;struct rb_root core_tree;bool core_forceidle;
@@ -999,6 +1000,8 @@ static inline int cpu_of(struct rq *rq) }#ifdef CONFIG_SCHED_CORE +#define SCHED_IDLE_ALLOWANCE 5000000/* 5 msec */ +DECLARE_STATIC_KEY_FALSE (__sched_core_enabled);static inline bool sched_core_enabled (struct rq *rq)
@@ -1016,6 +1019,7 @@ static inline raw_spinlock_t *rq_lockp(struct rq *rq)extern void queue_core_balance (struct rq *rq);extern bool prio_less_fair (struct task_struct *a,struct task_struct *b); +extern void account_core_idletime (struct task_struct *p,u64 exec);#else /* !CONFIG_SCHED_CORE */
Signature · 1 line
2.20.1
Quoted 19 lines · Tim Chen
Quoted 18 lines · Tim Chen
Quoted 16 lines · Aaron Lu
Review
Quoted 10 lines · Tim Chen
(quoted diff) +10
-0+void account_core_idletime(struct task_struct *p, u64 exec) +{ + const struct cpumask *smt_mask; + struct rq *rq; + bool force_idle, refill; + int i, cpu; + + rq = task_rq(p); + if (!sched_core_enabled(rq) || !p->core_cookie) + return;I don't see why return here for untagged task. Untagged task can also preempt tagged task and force a CPU thread enter idle state. Untagged is just another tag to me, unless we want to allow untagged task to coschedule with a tagged task.
Quoted 5 lines
Quoted 2 lines
You are right. This needs to be fixed.
Here's the updated patchset, including Aaron's fix and also added accounting of force idle time by deadline and rt tasks.
I have two other small changes that I think are worth sending out.
The first simplify logic in pick_task() and the 2nd avoid task pick all over again when max is preempted. I also refined the previous hack patch to make schedule always happen only for root cfs rq. Please see below for details, thanks.
patch1:
From cea56db35fe9f393c357cdb1bdcb2ef9b56cfe97 Mon Sep 17 00:00:00 2001
From: Aaron Lu <[email protected]>
Date: Mon, 5 Aug 2019 21:21:25 +0800
Subject: [PATCH 1/3] sched/core: simplify pick_task()
No need to special case !cookie case in pick_task(), we just need to make it possible to return idle in sched_core_find() for !cookie query. And cookie_pick will always have less priority than class_pick, so remove the redundant check of prio_less(cookie_pick, class_pick).
--- kernel/sched/core.c | 19 ++++--------------- 1 file changed, 4 insertions(+), 15 deletions(-)
kernel/sched/core.c +4 -15
@@ -186,6 +186,8 @@ static struct task_struct *sched_core_find(struct rq *rq, unsigned long cookie)* The idle task always matches any cookie! */ match = idle_sched_class.pick_task (rq); +if (!cookie) + goto out;while (node) { node_task = container_of(node, struct task_struct, core_node);
@@ -199,7 +201,7 @@ static struct task_struct *sched_core_find(struct rq *rq, unsigned long cookie) node = node->rb_left; } } - +out:return match; }
@@ -3657,18 +3659,6 @@ pick_task(struct rq *rq, const struct sched_class *class, struct task_struct *maif (!class_pick)return NULL ; -if (!cookie) { -/* -* If class_pick is tagged, return it only if it has -* higher priority than max. -*/ -if (max && class_pick->core_cookie && -prio_less (class_pick, max)) -return idle_sched_class.pick_task (rq); - -return class_pick; - } -/* * If class_pick is idle or matches cookie, return early. */
@@ -3682,8 +3672,7 @@ pick_task(struct rq *rq, const struct sched_class *class, struct task_struct *ma* the core (so far) and it must be selected, otherwise we must go with * the cookie pick in order to satisfy the constraint. */ -if (prio_less (cookie_pick, class_pick) && - (!max ||prio_less (max, class_pick))) +if (!max ||prio_less (max, class_pick))return class_pick;return cookie_pick;
Signature · 16 lines
2.19.1.3.ge56e4f7 patch2: From 487950dc53a40d5c566602f775ce46a0bab7a412 Mon Sep 17 00:00:00 2001 From: Aaron Lu <[email protected]> Date: Fri, 9 Aug 2019 14:48:01 +0800 Subject: [PATCH 2/3] sched/core: no need to pick again after max is preempted When sibling's task preempts current max, there is no need to do the pick all over again - the preempted cpu could just pick idle and done. Signed-off-by: Aaron Lu <[email protected]> --- kernel/sched/core.c | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-)
kernel/sched/core.c +3 -4
@@ -3756,7 +3756,6 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)* order. */ for_each_class (class) { -again:for_each_cpu_wrap (i, smt_mask, cpu) {struct rq *rq_i =cpu_rq (i);struct task_struct *p;
@@ -3828,10 +3827,10 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)if (j == i)continue ; -cpu_rq (j)->core_pick =NULL ; +cpu_rq (j)->core_pick = idle_sched_class.pick_task (cpu_rq (j)); } occ =1 ; - goto again; + goto out; }else {/* * Once we select a task for a cpu, we
@@ -3846,7 +3845,7 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf) } next_class:; } - +out: rq->core->core_pick_seq = rq->core->core_task_seq; next = rq->core_pick; rq->core_sched_seq = rq->core->core_pick_seq;
Signature · 21 lines
2.19.1.3.ge56e4f7 patch3: From 2d396d99e0dd7157b0b4f7a037c8b84ed135ea56 Mon Sep 17 00:00:00 2001 From: Aaron Lu <[email protected]> Date: Thu, 25 Jul 2019 19:57:21 +0800 Subject: [PATCH 3/3] sched/fair: make tick based schedule always happen When a hyperthread is forced idle and the other hyperthread has a single CPU intensive task running, the running task can occupy the hyperthread for a long time with no scheduling point and starve the other hyperthread. Fix this temporarily by always checking if the task has exceed its timeslice and if so, for root cfs_rq, do a schedule. Signed-off-by: Aaron Lu <[email protected]> --- kernel/sched/fair.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
kernel/sched/fair.c +4 -1
@@ -4011,6 +4011,9 @@ check_preempt_tick(struct cfs_rq *cfs_rq, struct sched_entity *curr)return ; } +if (cfs_rq->nr_running <=1 ) +return ; +/* * Ensure that a task that missed wakeup preemption by a * narrow margin doesn't have to wait for a full slice.
@@ -4179,7 +4182,7 @@ entity_tick(struct cfs_rq *cfs_rq, struct sched_entity *curr, int queued)return ;#endif -if (cfs_rq->nr_running >1 ) +if (cfs_rq->nr_running >1 || cfs_rq->tg == &root_task_group)check_preempt_tick (cfs_rq, curr); }
Signature · 1 line
2.19.1.3.ge56e4f7
Quoted 22 lines · Tim Chen
Quoted 10 lines · Aaron Lu
Quoted 2 lines · Tim Chen
Quoted 1 line · Julien Desfossez
We tested both Aaron's and Tim's patches and here are our results.
kernel/sched/core.c +0
-0kernel/sched/core.c next_class:; · line 3891
Quoted 5 lines
kernel/sched/core.c +1
-1@@ -3888,7 +3888,7 @@ next_class:;WARN_ON_ONCE (!rq_i->core_pick);if (is_idle_task (rq_i->core_pick) && rq_i->nr_running) - rq->core_forceidle = true; + rq_i->core_forceidle = true;Good catch!
Quoted 3 lines
kernel/sched/core.c +0
-0rq_i->core_pick->core_occupation = occ;
Quoted 8 lines
Quoted 4 lines
With this fixed and together with the patch to let schedule always happen, your latest 2 patches work well for the 10s cpuhog test I described previously:
https://lore.kernel.org/lkml/20190725143003.GA992@aaronlu/That's encouraging. You are talking about my patches that try to keep the force idle time between sibling threads balanced, right?
Yes.
Quoted 7 lines
Quoted 4 lines
overloaded workload without any cpu binding doesn't work well though, I haven't taken a closer look yet.
I think we need a load balancing scheme among the cores that will try to minimize force idle.
Agree.
Quoted 10 lines
One possible metric to measure load compatibility imbalance that leads to force idle is
Say i, j are sibling threads of a cpu core
imbalanace = \sum_tagged_cgroupabs(Load_cgroup_cpui - Load_cgroup_cpuj)This gives us a metric to decide if migrating a task will improve load compatability imbalance. As we already track cgroup load on a CPU, it should be doable without adding too much overhead.
Quoted 7 lines
I have two other small changes that I think are worth sending out.
The first simplify logic in
pick_task()and the 2nd avoid task pick all over again when max is preempted. I also refined the previous hack patch to make schedule always happen only for root cfs rq. Please see below for details, thanks.
I see a potential issue here. With the simplification in pick_task, you might introduce a livelock where the match logic spins for ever. But you avoid that with the patch 2, by removing the loop if a pick preempts max. The potential problem is that, you miss a case where the newly picked task might have a match in the sibling on which max was selected before. By selecting idle, you ignore the potential match. As of now, the potential match check does not really work because, sched_core_find will always return the same task and we do not check the whole core_tree for a next match. This is in my TODO list to have sched_core_find to return the best next match, if match was preempted. But its a bit complex and needs more thought.
Third patch looks good to me.
Thanks,
Vineeth
Quoted 7 lines · Vineeth Remanan Pillai
Quoted 7 lines
I have two other small changes that I think are worth sending out.
The first simplify logic in
pick_task()and the 2nd avoid task pick all over again when max is preempted. I also refined the previous hack patch to make schedule always happen only for root cfs rq. Please see below for details, thanks.
Quoted 6 lines
I see a potential issue here. With the simplification in
pick_task, you might introduce a livelock where the match logic spins for ever. But you avoid that with the patch 2, by removing the loop if a pick preempts max. The potential problem is that, you miss a case where the newly picked task might have a match in the sibling on which max was selected before. By selecting idle, you ignore the potential match.
Oh that's right, I missed this.
Quoted 5 lines
As of now, the potential match check does not really work because,
sched_core_findwill always return the same task and we do not check the wholecore_treefor a next match. This is in my TODO list to havesched_core_findto return the best next match, if match was preempted. But its a bit complex and needs more thought.
Sounds worth to do :-)
Quoted 10 lines · Tim Chen
Quoted 6 lines · Dario Faggioli
Since I see that, in this thread, there are various patches being proposed and discussed... should I rerun my benchmarks with them applied? If yes, which ones? And is there, by any chance, one (or maybe
more than one) updated git branch(es)?Hi Dario,
Hi Tim!
Quoted 3 lines
Having an extra set of eyes are certainly welcomed. I'll give my 2 cents on the issues with v3.
Ok, and thanks a lot for this.
Quoted 25 lines
1) Unfairness between the sibling threads
-----------------------------------------
One sibling thread could be suppressing and force idling the sibling thread over proportionally. Resulting in the force idled CPU not getting run and stall tasks on suppressed CPU.[...]
2) Not rescheduling forced idled CPU
------------------------------------
The forced idled CPU does not get a chance to re-schedule itself, and will stall for a long time even though it has eligible tasks to run.[...]
3) Load balancing between CPU cores
-----------------------------------
Say if one CPU core's sibling threads get forced idled a lot as it has mostly incompatible tasks between the siblings, moving the incompatible load to other cores and pulling compatible load to the core could help CPU utilization.
Quoted 7 lines
So just considering the load of a task is not enough during load balancing, task compatibility also needs to be considered. Peter has put in mechanisms to balance compatible tasks between CPU thread siblings, but not across cores.
[...]
Ok. Yes, as said, I've been trying to follow the thread, but thanks a lot again for this summary.
As said, I'm about to have numbers for the repo/branch I mentioned.
I was considering whether to also re-run the benchmarking campaign with some of the patches that floated around within this thread. Now, thanks to your summary, I have an even clearer picture about which patch does what, and that is indeed very useful.
I'll see about putting something together. I'm thinking of picking:
https://lore.kernel.org/lkml/[email protected]/
https://lore.kernel.org/lkml/20190725143344.GD992@aaronlu/
And maybe even (part of):
https://lore.kernel.org/lkml/20190810141556.GA73644@aaronlu/#t
If anyone has ideas or suggestions about whether or not this choice makes sense, feel free to share. :-)
Also, I only have another week before leaving, so let's see what I manage to actually run, and then share here, by then.
Thanks and Regards
Signature · 6 lines
Dario Faggioli, Ph.D http://about.me/dario.faggioli Virtualization Software Engineer SUSE Labs, SUSE https://www.suse.com/ ------------------------------------------------------------------- <<This happens because _I_ choose it to happen!>> (Raistlin Majere)
Quoted 62 lines · Dario Faggioli
Quoted 10 lines · Tim Chen
Quoted 6 lines · Dario Faggioli
Since I see that, in this thread, there are various patches being proposed and discussed... should I rerun my benchmarks with them applied? If yes, which ones? And is there, by any chance, one (or maybe
more than one) updated git branch(es)?Hi Dario,
Hi Tim!
Quoted 3 lines
Having an extra set of eyes are certainly welcomed. I'll give my 2 cents on the issues with v3.
Ok, and thanks a lot for this.
Quoted 25 lines
1) Unfairness between the sibling threads
-----------------------------------------
One sibling thread could be suppressing and force idling the sibling thread over proportionally. Resulting in the force idled CPU not getting run and stall tasks on suppressed CPU.[...]
2) Not rescheduling forced idled CPU
------------------------------------
The forced idled CPU does not get a chance to re-schedule itself, and will stall for a long time even though it has eligible tasks to run.[...]
3) Load balancing between CPU cores
-----------------------------------
Say if one CPU core's sibling threads get forced idled a lot as it has mostly incompatible tasks between the siblings, moving the incompatible load to other cores and pulling compatible load to the core could help CPU utilization.Quoted 7 lines
So just considering the load of a task is not enough during load balancing, task compatibility also needs to be considered. Peter has put in mechanisms to balance compatible tasks between CPU thread siblings, but not across cores.
[...]
Ok. Yes, as said, I've been trying to follow the thread, but thanks a lot again for this summary.
As said, I'm about to have numbers for the repo/branch I mentioned.
I was considering whether to also re-run the benchmarking campaign with some of the patches that floated around within this thread. Now, thanks to your summary, I have an even clearer picture about which patch does what, and that is indeed very useful.
I'll see about putting something together. I'm thinking of picking:
Quoted 8 lines
https://lore.kernel.org/lkml/[email protected]/
https://lore.kernel.org/lkml/20190725143344.GD992@aaronlu/And maybe even (part of):
https://lore.kernel.org/lkml/20190810141556.GA73644@aaronlu/#tIf anyone has ideas or suggestions about whether or not this choice makes sense, feel free to share. :-)
Makes sense to me.
patch3 in the last link is slightly better than the one in the 2nd link, so just use that instead.
Thanks,
Aaron
Quoted 12 lines
Also, I only have another week before leaving, so let's see what I manage to actually run, and then share here, by then.
Thanks and Regards
Signature · 6 lines
Dario Faggioli, Ph.D http://about.me/dario.faggioli Virtualization Software Engineer SUSE Labs, SUSE https://www.suse.com/ ------------------------------------------------------------------- <<This happens because _I_ choose it to happen!>> (Raistlin Majere)
Re: [RFC PATCH v3 01/16] stop_machine: Fix stop_cpus_in_progress ordering
In reply to Vineeth Remanan PillaiQuoted 3 lines · Vineeth Remanan Pillai
From: Peter Zijlstra <[email protected]>
Make sure the entire for loop has
stop_cpus_in_progressset.
It is not clear how this commit comment matches the change. Please explain how adding 2 barrier's makes sure stop_cpus_in_progress is set for the entier for loop.
--mark
Quoted 29 lines
Signed-off-by Peter Zijlstra (Intel) <[email protected]>--- kernel/stop_machine.c | 2 ++ 1 file changed, 2 insertions(+)kernel/stop_machine.c +2
-0@@ -375,6 +375,7 @@ static bool queue_stop_cpus_work(const struct cpumask *cpumask,*/ preempt_disable (); stop_cpus_in_progress = true; +barrier ();for_each_cpu (cpu, cpumask) { work = &per_cpu (cpu_stopper.stop_work, cpu); work->fn = fn;@@ -383,6 +384,7 @@ static bool queue_stop_cpus_work(const struct cpumask *cpumask,if (cpu_stop_queue_work (cpu, work)) queued = true; } +barrier (); stop_cpus_in_progress = false;preempt_enable ();Signature · 1 line
2.17.1
Re: [RFC PATCH v3 02/16] sched: Fix kerneldoc comment for ia64_set_curr_task
In reply to Vineeth Remanan PillaiQuoted 1 line · Vineeth Remanan Pillai
From: Peter Zijlstra <[email protected]>
NULL commit comment.
--mark
Quoted 22 lines
Signed-off-by Peter Zijlstra (Intel) <[email protected]>--- kernel/sched/core.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)kernel/sched/core.c +1
-1@@ -6287,7 +6287,7 @@ struct task_struct *curr_task(int cpu)#ifdef CONFIG_IA64 /** -* set_curr_task - set the current task for a given CPU. +* ia64_set_curr_task - set the current task for a given CPU. * @cpu: the processor in question. * @p: the task pointer to set. * Signature · 1 line
2.17.1
Re: [RFC PATCH v3 07/16] sched: Allow put_prev_task() to drop rq->lock
In reply to Vineeth Remanan PillaiQuoted 8 lines · Vineeth Remanan Pillai
From: Peter Zijlstra <[email protected]>
Currently the
pick_next_task()loop is convoluted and ugly because of how it can drop therq->lockand needs to restart the picking.For the RT/Deadline classes, it is
put_prev_task()where we do balancing, and we could do this before the picking loop. Make this possible.
Maybe explain why adding strtu rq_flags pointers to the function call supports the above commit coment.
--mark
Quoted 152 lines
Signed-off-by Peter Zijlstra (Intel) <[email protected]>--- kernel/sched/core.c | 2 +- kernel/sched/deadline.c | 14 +++++++++++++- kernel/sched/fair.c | 2 +- kernel/sched/idle.c | 2 +- kernel/sched/rt.c | 14 +++++++++++++- kernel/sched/sched.h | 4 ++-- kernel/sched/stop_task.c | 2 +- 7 files changed, 32 insertions(+), 8 deletions(-)kernel/sched/core.c +1
-1@@ -5595,7 +5595,7 @@ static void calc_load_migrate(struct rq *rq)atomic_long_add (delta, &calc_load_tasks); } -static void put_prev_task_fake (struct rq *rq,struct task_struct *prev) +static void put_prev_task_fake (struct rq *rq,struct task_struct *prev,struct rq_flags *rf) { }kernel/sched/deadline.c +13
-1@@ -1772,13 +1772,25 @@ pick_next_task_dl(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)return p; } -static void put_prev_task_dl (struct rq *rq,struct task_struct *p) +static void put_prev_task_dl (struct rq *rq,struct task_struct *p,struct rq_flags *rf) {update_curr_dl (rq);update_dl_rq_load_avg (rq_clock_pelt (rq), rq,1 );if (on_dl_rq (&p->dl) && p->nr_cpus_allowed >1 )enqueue_pushable_dl_task (rq, p); + +if (rf && !on_dl_rq (&p->dl) &&need_pull_dl_task (rq, p)) { +/* +* This is OK, because current is on_cpu, which avoids it being +* picked for load-balance and preemption/IRQs are still +* disabled avoiding further scheduler activity on it and we've +* not yet started the picking loop. +*/ +rq_unpin_lock (rq, rf); +pull_dl_task (rq); +rq_repin_lock (rq, rf); + } }/* kernel/sched/fair.c +1
-1@@ -7110,7 +7110,7 @@ done: __maybe_unused;/* * Account for a descheduled task: */ -static void put_prev_task_fair (struct rq *rq,struct task_struct *prev) +static void put_prev_task_fair (struct rq *rq,struct task_struct *prev,struct rq_flags *rf) {struct sched_entity *se = &prev->se;struct cfs_rq *cfs_rq;kernel/sched/idle.c +1
-1@@ -373,7 +373,7 @@ static void check_preempt_curr_idle(struct rq *rq, struct task_struct *p, int flresched_curr (rq); } -static void put_prev_task_idle (struct rq *rq,struct task_struct *prev) +static void put_prev_task_idle (struct rq *rq,struct task_struct *prev,struct rq_flags *rf) { }kernel/sched/rt.c +13
-1@@ -1593,7 +1593,7 @@ pick_next_task_rt(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)return p; } -static void put_prev_task_rt (struct rq *rq,struct task_struct *p) +static void put_prev_task_rt (struct rq *rq,struct task_struct *p,struct rq_flags *rf) {update_curr_rt (rq);@@ -1605,6 +1605,18 @@ static void put_prev_task_rt(struct rq *rq, struct task_struct *p)*/ if (on_rt_rq (&p->rt) && p->nr_cpus_allowed >1 )enqueue_pushable_task (rq, p); + +if (rf && !on_rt_rq (&p->rt) &&need_pull_rt_task (rq, p)) { +/* +* This is OK, because current is on_cpu, which avoids it being +* picked for load-balance and preemption/IRQs are still +* disabled avoiding further scheduler activity on it and we've +* not yet started the picking loop. +*/ +rq_unpin_lock (rq, rf); +pull_rt_task (rq); +rq_repin_lock (rq, rf); + } }#ifdef CONFIG_SMP kernel/sched/sched.h +2
-2@@ -1675,7 +1675,7 @@ struct sched_class {struct task_struct * (*pick_next_task)(struct rq *rq,struct task_struct *prev,struct rq_flags *rf); -void (*put_prev_task)(struct rq *rq,struct task_struct *p); +void (*put_prev_task)(struct rq *rq,struct task_struct *p,struct rq_flags *rf);void (*set_next_task)(struct rq *rq,struct task_struct *p);#ifdef CONFIG_SMP @@ -1721,7 +1721,7 @@ struct sched_class {static inline void put_prev_task (struct rq *rq,struct task_struct *prev) {WARN_ON_ONCE (rq->curr != prev); - prev->sched_class->put_prev_task (rq, prev); + prev->sched_class->put_prev_task (rq, prev,NULL ); }static inline void set_next_task (struct rq *rq,struct task_struct *next)kernel/sched/stop_task.c +1
-1@@ -59,7 +59,7 @@ static void yield_task_stop(struct rq *rq)BUG ();/* the stop task should never yield, its pointless. */ } -static void put_prev_task_stop (struct rq *rq,struct task_struct *prev) +static void put_prev_task_stop (struct rq *rq,struct task_struct *prev,struct rq_flags *rf) {struct task_struct *curr = rq->curr;u64 delta_exec;Signature · 1 line
2.17.1
Peter Zijlstra
maintainerRe: [RFC PATCH v3 01/16] stop_machine: Fix stop_cpus_in_progress ordering
In reply to mark grossQuoted 7 lines · mark gross
Quoted 3 lines · Vineeth Remanan Pillai
From: Peter Zijlstra <[email protected]>
Make sure the entire for loop has
stop_cpus_in_progressset.It is not clear how this commit comment matches the change. Please explain how adding 2 barrier's makes sure
stop_cpus_in_progressis set for the entier for loop.
Without the barrier the compiler is free to move the stores around. It probably doesn't do anything bad, but this makes sure it cannot.
Re: [RFC PATCH v3 08/16] sched: Rework pick_next_task() slow-path
In reply to Vineeth Remanan PillaiQuoted 81 lines · Vineeth Remanan Pillai
From: Peter Zijlstra <[email protected]>
Avoid the
RETRY_TASKcase in thepick_next_task()slow path.By doing the
put_prev_task()early, we get the rt/deadline pull done, and by testingrq->nr_runningwe know if we neednewidle_balance().This then gives a stable state to pick a task from.
Since the fast-path is fair only; it means the other classes will always have
pick_next_task(.prev=NULL, .rf=NULL)and we can simplify.Signed-off-by Peter Zijlstra (Intel) <[email protected]>--- kernel/sched/core.c | 19 ++++++++++++------- kernel/sched/deadline.c | 30 ++---------------------------- kernel/sched/fair.c | 9 ++++++--- kernel/sched/idle.c | 4 +++- kernel/sched/rt.c | 29 +---------------------------- kernel/sched/sched.h | 13 ++++++++----- kernel/sched/stop_task.c | 3 ++- 7 files changed, 34 insertions(+), 73 deletions(-)kernel/sched/core.c +12
-7@@ -3363,7 +3363,7 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf) p = fair_sched_class.pick_next_task (rq, prev, rf);if (unlikely (p ==RETRY_TASK )) - goto again; + goto restart;/* Assumes fair_sched_class->next == idle_sched_class */ if (unlikely (!p))@@ -3372,14 +3372,19 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)return p; } -again: +restart: +/* +* Ensure that we put DL/RT tasks before the pick loop, such that they +* can PULL higher prio tasks when we lower the RQ 'priority'. +*/ + prev->sched_class->put_prev_task (rq, prev, rf); +if (!rq->nr_running) +newidle_balance (rq, rf); +for_each_class (class) { - p = class->pick_next_task (rq, prev, rf); -if (p) { -if (unlikely (p ==RETRY_TASK )) - goto again; + p = class->pick_next_task (rq,NULL ,NULL ); +if (p)return p; - } }/* The idle class should always have a runnable task: */ kernel/sched/deadline.c +0
-11@@ -1729,39 +1729,13 @@ pick_next_task_dl(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)struct task_struct *p;struct dl_rq *dl_rq; - dl_rq = &rq->dl; - -if (need_pull_dl_task (rq, prev)) { -/* -* This is OK, because current is on_cpu, which avoids it being -* picked for load-balance and preemption/IRQs are still -* disabled avoiding further scheduler activity on it and we're -* being very careful to re-start the picking loop. -*/ -rq_unpin_lock (rq, rf); -pull_dl_task (rq);
kernel/sched/deadline.c pick_next_task_dl(struct rq *rq, struct task_struct *prev, struct rq_flags *rf) · line 1732
Quoted 10 lines
kernel/sched/deadline.c +1
-9-rq_repin_lock (rq, rf); -/* -* pull_dl_task() can drop (and re-acquire) rq->lock; this -* means a stop task can slip in, in which case we need to -* re-start task selection. -*/ -if (rq->stop &&task_on_rq_queued (rq->stop)) -return RETRY_TASK ; - } +WARN_ON_ONCE (prev || rf);
should there be a helpful message to go with this warning?
Quoted 140 lines
kernel/sched/deadline.c +1
-8-/* -* When prev is DL, we may throttle it in put_prev_task(). -* So, we update time before we check for dl_nr_running. -*/ -if (prev->sched_class == &dl_sched_class) -update_curr_dl (rq); + dl_rq = &rq->dl;if (unlikely (!dl_rq->dl_nr_running))return NULL ; -put_prev_task (rq, prev); - dl_se =pick_next_dl_entity (rq, dl_rq);BUG_ON (!dl_se);kernel/sched/fair.c +6
-3@@ -6979,7 +6979,7 @@ pick_next_task_fair(struct rq *rq, struct task_struct *prev, struct rq_flags *rf goto idle;#ifdef CONFIG_FAIR_GROUP_SCHED -if (prev->sched_class != &fair_sched_class) +if (!prev || prev->sched_class != &fair_sched_class) goto simple;/* @@ -7056,8 +7056,8 @@ pick_next_task_fair(struct rq *rq, struct task_struct *prev, struct rq_flags *rf goto done; simple: #endif - - put_prev_task(rq, prev); + if (prev) + put_prev_task(rq, prev); do { se = pick_next_entity(cfs_rq, NULL);@@ -7085,6 +7085,9 @@ done: __maybe_unused;return p; idle: +if (!rf) +return NULL ; + new_tasks =newidle_balance (rq, rf);/* kernel/sched/idle.c +3
-1@@ -388,7 +388,9 @@ pick_next_task_idle(struct rq *rq, struct task_struct *prev, struct rq_flags *rf {struct task_struct *next = rq->idle; -put_prev_task (rq, prev); +if (prev) +put_prev_task (rq, prev); +set_next_task_idle (rq, next);return next;kernel/sched/rt.c +1
-28@@ -1554,38 +1554,11 @@ pick_next_task_rt(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)struct task_struct *p;struct rt_rq *rt_rq = &rq->rt; -if (need_pull_rt_task (rq, prev)) { -/* -* This is OK, because current is on_cpu, which avoids it being -* picked for load-balance and preemption/IRQs are still -* disabled avoiding further scheduler activity on it and we're -* being very careful to re-start the picking loop. -*/ -rq_unpin_lock (rq, rf); -pull_rt_task (rq); -rq_repin_lock (rq, rf); -/* -* pull_rt_task() can drop (and re-acquire) rq->lock; this -* means a dl or stop task can slip in, in which case we need -* to re-start task selection. -*/ -if (unlikely ((rq->stop &&task_on_rq_queued (rq->stop)) || - rq->dl.dl_nr_running)) -return RETRY_TASK ; - } - -/* -* We may dequeue prev's rt_rq in put_prev_task(). -* So, we update time before rt_queued check. -*/ -if (prev->sched_class == &rt_sched_class) -update_curr_rt (rq); +WARN_ON_ONCE (prev || rf);if (!rt_rq->rt_queued)return NULL ; -put_prev_task (rq, prev); - p =_pick_next_task_rt (rq);set_next_task_rt (rq, p);kernel/sched/sched.h +8
-5@@ -1665,12 +1665,15 @@ struct sched_class {void (*check_preempt_curr)(struct rq *rq,struct task_struct *p,int flags);/* -* It is the responsibility of the pick_next_task() method that will -* return the next task to call put_prev_task() on the @prev task or -* something equivalent. +* Both @prev and @rf are optional and may be NULL, in which case the +* caller must already have invoked put_prev_task(rq, prev, rf). * -* May return RETRY_TASK when it finds a higher prio class has runnable -* tasks. +* Otherwise it is the responsibility of the pick_next_task() to call +* put_prev_task() on the @prev task or something equivalent, IFF it +* returns a next task. +* +* In that case (@rf != NULL) it may return RETRY_TASK when it finds a +* higher prio class has runnable tasks. */ struct task_struct * (*pick_next_task)(struct rq *rq, struct task_struct *prev,kernel/sched/stop_task.c +0
-0
kernel/sched/stop_task.c pick_next_task_stop(struct rq *rq, struct task_struct *prev, struct rq_flags *rf · line 36
Quoted 4 lines
kernel/sched/stop_task.c +1
-0@@ -33,10 +33,11 @@ pick_next_task_stop(struct rq *rq, struct task_struct *prev, struct rq_flags *rf {struct task_struct *stop = rq->stop; +WARN_ON_ONCE (prev || rf);
should there be a helpful message to go with this warning? --mark
Quoted 11 lines
kernel/sched/stop_task.c +1
-1+if (!stop || !task_on_rq_queued (stop))return NULL ; -put_prev_task (rq, prev);set_next_task_stop (rq, stop);return stop;Signature · 1 line
2.17.1
Re: [RFC PATCH v3 09/16] sched: Introduce sched_class::pick_task()
In reply to Vineeth Remanan PillaiQuoted 6 lines · Vineeth Remanan Pillai
From: Peter Zijlstra <[email protected]>
Because
sched_class::pick_next_task()also impliessched_class::set_next_task()(and possiblyput_prev_task()andnewidle_balance) it is not state invariant. This makes it unsuitable for remote task selection.
It would be cool if the commit comment would explain what the change is going to do about pick_next_task being unsuitable for remote task selection.
Quoted 50 lines
Signed-off-by Peter Zijlstra (Intel) <[email protected]>Signed-off-by Vineeth Remanan Pillai <[email protected]>Signed-off-by Julien Desfossez <[email protected]>---
Chnages in v3
-------------
- Minor refactor to remove redundant NULL checksChanges in v2
-------------
- Fixes a NULL pointer dereference crash
- Subhra Mazumdar
- Tim Chen--- kernel/sched/deadline.c | 21 ++++++++++++++++----- kernel/sched/fair.c | 36 +++++++++++++++++++++++++++++++++--- kernel/sched/idle.c | 10 +++++++++- kernel/sched/rt.c | 21 ++++++++++++++++----- kernel/sched/sched.h | 2 ++ kernel/sched/stop_task.c | 21 ++++++++++++++++----- 6 files changed, 92 insertions(+), 19 deletions(-)kernel/sched/deadline.c +1
-4@@ -1722,15 +1722,12 @@ static struct sched_dl_entity *pick_next_dl_entity(struct rq *rq, return rb_entry(left, struct sched_dl_entity, rb_node); } -static struct task_struct * -pick_next_task_dl (struct rq *rq,struct task_struct *prev,struct rq_flags *rf) +static struct task_struct *pick_task_dl (struct rq *rq) {struct sched_dl_entity *dl_se;struct task_struct *p;struct dl_rq *dl_rq; -WARN_ON_ONCE (prev || rf); - dl_rq = &rq->dl;if (unlikely (!dl_rq->dl_nr_running))@@ -1741,7 +1738,19 @@ pick_next_task_dl(struct rq *rq, struct task_struct *prev, struct rq_flags *rf) p =dl_task_of (dl_se);
kernel/sched/deadline.c pick_next_task_dl(struct rq *rq, struct task_struct *prev, struct rq_flags *rf) · line 1749
Quoted 10 lines
kernel/sched/deadline.c +9
-1-set_next_task_dl (rq, p); +return p; +} + +static struct task_struct * +pick_next_task_dl (struct rq *rq,struct task_struct *prev,struct rq_flags *rf) +{ +struct task_struct *p; + +WARN_ON_ONCE (prev || rf);
What is an admin to do with this warnding if it shows up in there logs? maybe include some text here to help folks that might hit this warn_on.
Quoted 150 lines
kernel/sched/deadline.c +6
-0+ + p =pick_task_dl (rq); +if (p) +set_next_task_dl (rq, p);return p; }@@ -2388,6 +2397,8 @@ const struct sched_class dl_sched_class = { .set_next_task = set_next_task_dl, #ifdef CONFIG_SMP + .pick_task = pick_task_dl, + .select_task_rq = select_task_rq_dl, .migrate_task_rq = migrate_task_rq_dl, .set_cpus_allowed = set_cpus_allowed_dl,kernel/sched/fair.c +33
-3@@ -4136,7 +4136,7 @@ pick_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *curr)* Avoid running the skip buddy, if running something else can * be done without getting too unfair. */ -if (cfs_rq->skip == se) { +if (cfs_rq->skip && cfs_rq->skip == se) {struct sched_entity *second;if (se == curr) {@@ -4154,13 +4154,13 @@ pick_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *curr)/* * Prefer last buddy, try to return the CPU to a preempted task. */ -if (cfs_rq->last &&wakeup_preempt_entity (cfs_rq->last, left) <1 ) +if (left && cfs_rq->last &&wakeup_preempt_entity (cfs_rq->last, left) <1 ) se = cfs_rq->last;/* * Someone really wants this to run. If it's not unfair, run it. */ -if (cfs_rq->next &&wakeup_preempt_entity (cfs_rq->next, left) <1 ) +if (left && cfs_rq->next &&wakeup_preempt_entity (cfs_rq->next, left) <1 ) se = cfs_rq->next;clear_buddies (cfs_rq, se);@@ -6966,6 +6966,34 @@ static void check_preempt_wakeup(struct rq *rq, struct task_struct *p, int wake_set_last_buddy (se); } +static struct task_struct * +pick_task_fair (struct rq *rq) +{ +struct cfs_rq *cfs_rq = &rq->cfs; +struct sched_entity *se; + +if (!cfs_rq->nr_running) +return NULL ; + +do { +struct sched_entity *curr = cfs_rq->curr; + + se =pick_next_entity (cfs_rq,NULL ); + +if (curr) { +if (se && curr->on_rq) +update_curr (cfs_rq); + +if (!se ||entity_before (curr, se)) + se = curr; + } + + cfs_rq =group_cfs_rq (se); + }while (cfs_rq); + +return task_of (se); +} +static struct task_struct *pick_next_task_fair (struct rq *rq,struct task_struct *prev,struct rq_flags *rf) {@@ -10677,6 +10705,8 @@ const struct sched_class fair_sched_class = { .set_next_task = set_next_task_fair, #ifdef CONFIG_SMP + .pick_task = pick_task_fair, + .select_task_rq = select_task_rq_fair, .migrate_task_rq = migrate_task_rq_fair,kernel/sched/idle.c +9
-1@@ -373,6 +373,12 @@ static void check_preempt_curr_idle(struct rq *rq, struct task_struct *p, int flresched_curr (rq); } +static struct task_struct * +pick_task_idle (struct rq *rq) +{ +return rq->idle; +} +static void put_prev_task_idle (struct rq *rq,struct task_struct *prev,struct rq_flags *rf) { }@@ -386,11 +392,12 @@ static void set_next_task_idle(struct rq *rq, struct task_struct *next)static struct task_struct *pick_next_task_idle (struct rq *rq,struct task_struct *prev,struct rq_flags *rf) { -struct task_struct *next = rq->idle; +struct task_struct *next;if (prev)put_prev_task (rq, prev); + next =pick_task_idle (rq);set_next_task_idle (rq, next);return next;@@ -458,6 +465,7 @@ const struct sched_class idle_sched_class = { .set_next_task = set_next_task_idle,#ifdef CONFIG_SMP + .pick_task = pick_task_idle, .select_task_rq = select_task_rq_idle, .set_cpus_allowed = set_cpus_allowed_common,#endif kernel/sched/rt.c +1
-4@@ -1548,20 +1548,29 @@ static struct task_struct *_pick_next_task_rt(struct rq *rq)return rt_task_of (rt_se); } -static struct task_struct * -pick_next_task_rt (struct rq *rq,struct task_struct *prev,struct rq_flags *rf) +static struct task_struct *pick_task_rt (struct rq *rq) {struct task_struct *p;struct rt_rq *rt_rq = &rq->rt; -WARN_ON_ONCE (prev || rf); -if (!rt_rq->rt_queued)return NULL ; p =_pick_next_task_rt (rq);
kernel/sched/rt.c static struct task_struct *_pick_next_task_rt(struct rq *rq) · line 1569
Quoted 10 lines
kernel/sched/rt.c +9
-1-set_next_task_rt (rq, p); +return p; +} + +static struct task_struct * +pick_next_task_rt (struct rq *rq,struct task_struct *prev,struct rq_flags *rf) +{ +struct task_struct *p; + +WARN_ON_ONCE (prev || rf);
what does it mean if this warn on goes off to an admin?
Quoted 80 lines
kernel/sched/rt.c +6
-0+ + p =pick_task_rt (rq); +if (p) +set_next_task_rt (rq, p);return p; }@@ -2364,6 +2373,8 @@ const struct sched_class rt_sched_class = { .set_next_task = set_next_task_rt, #ifdef CONFIG_SMP + .pick_task = pick_task_rt, + .select_task_rq = select_task_rq_rt, .set_cpus_allowed = set_cpus_allowed_common,kernel/sched/sched.h +2
-0@@ -1682,6 +1682,8 @@ struct sched_class {void (*set_next_task)(struct rq *rq,struct task_struct *p);#ifdef CONFIG_SMP +struct task_struct * (*pick_task)(struct rq *rq); +int (*select_task_rq)(struct task_struct *p,int task_cpu,int sd_flag,int flags);void (*migrate_task_rq)(struct task_struct *p,int new_cpu);kernel/sched/stop_task.c +16
-5@@ -29,20 +29,30 @@ static void set_next_task_stop(struct rq *rq, struct task_struct *stop) }static struct task_struct * -pick_next_task_stop (struct rq *rq,struct task_struct *prev,struct rq_flags *rf) +pick_task_stop (struct rq *rq) {struct task_struct *stop = rq->stop; -WARN_ON_ONCE (prev || rf); -if (!stop || !task_on_rq_queued (stop))return NULL ; -set_next_task_stop (rq, stop); -return stop; } +static struct task_struct * +pick_next_task_stop (struct rq *rq,struct task_struct *prev,struct rq_flags *rf) +{ +struct task_struct *p; + +WARN_ON_ONCE (prev || rf); + + p =pick_task_stop (rq); +if (p) +set_next_task_stop (rq, p); + +return p; +} +static void enqueue_task_stop (struct rq *rq,struct task_struct *p,int flags) {@@ -129,6 +139,7 @@ const struct sched_class stop_sched_class = { .set_next_task = set_next_task_stop,#ifdef CONFIG_SMP + .pick_task = pick_task_stop, .select_task_rq = select_task_rq_stop, .set_cpus_allowed = set_cpus_allowed_common,#endif Signature · 1 line
2.17.1
Re: [RFC PATCH v3 11/16] sched: Basic tracking of matching tasks
In reply to Vineeth Remanan PillaiQuoted 11 lines · Vineeth Remanan Pillai
From: Peter Zijlstra <[email protected]>
Introduce
task_struct::core_cookieas an opaque identifier for core scheduling. When enabled; core scheduling will only allow matching task to be on the core; where idle matches everything.When
task_struct::core_cookieis set (and core scheduling is enabled) these tasks are indexed in a second RB-tree, first on cookie value then on scheduling function, such that matching task selection always finds the most elegible match.
Quoted 5 lines
NOTE: *shudder* at the overhead...
NOTE: *sigh*, a 3rd copy of the scheduling function; the alternative is per class tracking of cookies and that just duplicates a lot of stuff for no raisin (the 2nd copy lives in the rt-mutex PI code).
s/raisen/reason
Quoted 66 lines
Signed-off-by Peter Zijlstra (Intel) <[email protected]>Signed-off-by Vineeth Remanan Pillai <[email protected]>Signed-off-by Julien Desfossez <[email protected]>---
Changes in v3
-------------
- Refactored priority comparison code
- Fixed a comparison logic issue insched_core_find
- Aaron LuChanges in v2
-------------
- Improves the priority comparison logic between processes in different cpus.
- Peter Zijlstra
- Aaron Lu--- include/linux/sched.h | 8 ++- kernel/sched/core.c | 146 ++++++++++++++++++++++++++++++++++++++++++ kernel/sched/fair.c | 46 ------------- kernel/sched/sched.h | 55 ++++++++++++++++ 4 files changed, 208 insertions(+), 47 deletions(-)include/linux/sched.h +7
-1@@ -636,10 +636,16 @@ struct task_struct {const struct sched_class *sched_class;struct sched_entity se;struct sched_rt_entity rt; +struct sched_dl_entity dl; + +#ifdef CONFIG_SCHED_CORE +struct rb_node core_node; +unsigned long core_cookie; +#endif +#ifdef CONFIG_CGROUP_SCHED struct task_group *sched_task_group;#endif -struct sched_dl_entity dl;#ifdef CONFIG_PREEMPT_NOTIFIERS /* List of struct preempt_notifier: */ kernel/sched/core.c +10
-0@@ -64,6 +64,141 @@ int sysctl_sched_rt_runtime = 950000;DEFINE_STATIC_KEY_FALSE (__sched_core_enabled); +/* kernel prio, less is more */ +static inline int __task_prio (struct task_struct *p) +{ +if (p->sched_class == &stop_sched_class)/* trumps deadline */ +return -2 ; + +if (rt_prio (p->prio))/* includes deadline */ +return p->prio;/* [-1, 99] */ + +if (p->sched_class == &idle_sched_class)
kernel/sched/core.c int sysctl_sched_rt_runtime = 950000; · line 86
Quoted 10 lines
kernel/sched/core.c +10
-0+return MAX_RT_PRIO +NICE_WIDTH ;/* 140 */ + +return MAX_RT_PRIO +MAX_NICE ;/* 120, squash fair */ +} + +/* +* l(a,b) +* le(a,b) := !l(b,a) +* g(a,b) := l(b,a) +* ge(a,b) := !l(a,b)
why does this truth table comment exist?
maybe inline comments at the confusing inequalities would be better. --mark
Apple have provided a sysctl that allows applications to indicate that specific threads should make use of core isolation while allowing the rest of the system to make use of SMT, and browsers (Safari, Firefox and Chrome, at least) are now making use of this. Trying to do something similar using cgroups seems a bit awkward. Would something like this be reasonable? Having spoken to the Chrome team, I believe that the semantics we want are:
1) A thread to be able to indicate that it should not run on the same core as anything not in posession of the same cookie
2) Descendents of that thread to (by default) have the same cookie
3) No other thread be able to obtain the same cookie
4) Threads not be able to rejoin the global group (ie, threads can segregate themselves from their parent and peers, but can never rejoin that group once segregated)
but don't know if that's what everyone else would want.
include/uapi/linux/prctl.h +1 -0
@@ -229,4 +229,5 @@ struct prctl_mm_map {# define PR_PAC_APDBKEY (1UL << 3)# define PR_PAC_APGAKEY (1UL << 4) +#define PR_CORE_ISOLATE 55#endif /* _LINUX_PRCTL_H */
kernel/sys.c +7 -0
@@ -2486,6 +2486,13 @@ SYSCALL_DEFINE5(prctl, int, option, unsigned long, arg2, unsigned long, arg3,return -EINVAL ; error =PAC_RESET_KEYS (me, arg2);break ; +case PR_CORE_ISOLATE : +#ifdef CONFIG_SCHED_CORE + current->core_cookie = (unsigned long )current; +#else + result = -EINVAL ; +#endif +break ;default : error = -EINVAL ;break ;
Signature · 1 line
Matthew Garrett | [email protected]
Peter Zijlstra
maintainerQuoted 6 lines · Matthew Garrett
Apple have provided a sysctl that allows applications to indicate that specific threads should make use of core isolation while allowing the rest of the system to make use of SMT, and browsers (Safari, Firefox and Chrome, at least) are now making use of this. Trying to do something similar using cgroups seems a bit awkward. Would something like this be reasonable?
Sure; like I wrote earlier; I only did the cgroup thing because I was lazy and it was the easiest interface to hack on in a hurry.
The rest of the ABI nonsense can 'trivially' be done later; if when we decide to actually do this.
And given MDS, I'm still not entirely convinced it all makes sense. If it were just L1TF, then yes, but now...
Quoted 28 lines
Having spoken to the Chrome team, I believe that the semantics we want are:
1) A thread to be able to indicate that it should not run on the same core as anything not in posession of the same cookie
2) Descendents of that thread to (by default) have the same cookie
3) No other thread be able to obtain the same cookie
4) Threads not be able to rejoin the global group (ie, threads can segregate themselves from their parent and peers, but can never rejoin that group once segregated)but don't know if that's what everyone else would want.
include/uapi/linux/prctl.h +1
-0@@ -229,4 +229,5 @@ struct prctl_mm_map {# define PR_PAC_APDBKEY (1UL << 3)# define PR_PAC_APGAKEY (1UL << 4) +#define PR_CORE_ISOLATE 55#endif /* _LINUX_PRCTL_H */ kernel/sys.c +0
-0
kernel/sys.c SYSCALL_DEFINE5(prctl, int, option, unsigned long, arg2, unsigned long, arg3, · line 2491
Quoted 6 lines
kernel/sys.c +3
-0@@ -2486,6 +2486,13 @@ SYSCALL_DEFINE5(prctl, int, option, unsigned long, arg2, unsigned long, arg3,return -EINVAL ; error =PAC_RESET_KEYS (me, arg2);break ; +case PR_CORE_ISOLATE : +#ifdef CONFIG_SCHED_CORE + current->core_cookie = (unsigned long )current;
This needs to then also force a reschedule of current. And there's the little issue of what happens if 'current' dies while its children live on, and current gets re-used for a new process and does this again.
Quoted 11 lines
kernel/sys.c +4
-0+#else + result = -EINVAL ; +#endif +break ;default : error = -EINVAL ;break ;Signature · 2 lines
-- Matthew Garrett | [email protected]
Quoted 34 lines · Matthew Garrett
Apple have provided a sysctl that allows applications to indicate that specific threads should make use of core isolation while allowing the rest of the system to make use of SMT, and browsers (Safari, Firefox and Chrome, at least) are now making use of this. Trying to do something similar using cgroups seems a bit awkward. Would something like this be reasonable? Having spoken to the Chrome team, I believe that the semantics we want are:
1) A thread to be able to indicate that it should not run on the same core as anything not in posession of the same cookie
2) Descendents of that thread to (by default) have the same cookie
3) No other thread be able to obtain the same cookie
4) Threads not be able to rejoin the global group (ie, threads can segregate themselves from their parent and peers, but can never rejoin that group once segregated)but don't know if that's what everyone else would want.
include/uapi/linux/prctl.h +1
-0@@ -229,4 +229,5 @@ struct prctl_mm_map {# define PR_PAC_APDBKEY (1UL << 3)# define PR_PAC_APGAKEY (1UL << 4) +#define PR_CORE_ISOLATE 55#endif /* _LINUX_PRCTL_H */ kernel/sys.c +0
-0
kernel/sys.c SYSCALL_DEFINE5(prctl, int, option, unsigned long, arg2, unsigned long, arg3, · line 2491
Quoted 6 lines
kernel/sys.c +3
-0@@ -2486,6 +2486,13 @@ SYSCALL_DEFINE5(prctl, int, option, unsigned long, arg2, unsigned long, arg3,return -EINVAL ; error =PAC_RESET_KEYS (me, arg2);break ; +case PR_CORE_ISOLATE : +#ifdef CONFIG_SCHED_CORE + current->core_cookie = (unsigned long )current;
Because AVX512 instructions could pull down the core frequency, we also want to give a magic cookie number to all AVX512-using tasks on the system, so they won't affect the performance/latency of any other tasks.
This could be done by putting all AVX512 tasks into a cgroup, or by AVX512 detection the following patch introduced.
Thanks,
-Aubrey
Quoted 8 lines · On Tue, Aug 27, 2019 at 11:50:35PM +0200 Peter Zijlstra wrote:
Quoted 6 lines · Matthew Garrett
Apple have provided a sysctl that allows applications to indicate that specific threads should make use of core isolation while allowing the rest of the system to make use of SMT, and browsers (Safari, Firefox and Chrome, at least) are now making use of this. Trying to do something similar using cgroups seems a bit awkward. Would something like this be reasonable?
Quoted 5 lines
Sure; like I wrote earlier; I only did the cgroup thing because I was lazy and it was the easiest interface to hack on in a hurry.
The rest of the ABI nonsense can 'trivially' be done later; if when we decide to actually do this.
I think something that allows the tag to be set may be needed. One of the use cases for this is virtualization stacks, where you really want to be able to keep the higher CPU count and to set up the isolation from management processes on the host.
The current cgroup interface doesn't work for that because it doesn't apply the tag to children. We've been unable to fully test it in a virt setup because our VMs are made of a child cgroup per vcpu.
Quoted 3 lines
And given MDS, I'm still not entirely convinced it all makes sense. If it were just L1TF, then yes, but now...
I was thinking MDS is really the reason for this. L1TF has mitigations but the only current mitigation for MDS for smt is ... nosmt.
The current core scheduler implementation, I believe, still has (theoretical?) holes involving interrupts, once/if those are closed it may be even less attractive.
Quoted 31 lines
Quoted 31 lines
Having spoken to the Chrome team, I believe that the semantics we want are:
1) A thread to be able to indicate that it should not run on the same core as anything not in posession of the same cookie
2) Descendents of that thread to (by default) have the same cookie
3) No other thread be able to obtain the same cookie
4) Threads not be able to rejoin the global group (ie, threads can segregate themselves from their parent and peers, but can never rejoin that group once segregated)but don't know if that's what everyone else would want.
include/uapi/linux/prctl.h +1
-0@@ -229,4 +229,5 @@ struct prctl_mm_map {# define PR_PAC_APDBKEY (1UL << 3)# define PR_PAC_APGAKEY (1UL << 4) +#define PR_CORE_ISOLATE 55#endif /* _LINUX_PRCTL_H */ kernel/sys.c +0
-0@@ -2486,6 +2486,13 @@ SYSCALL_DEFINE5(prctl, int, option, unsigned long, arg2, unsigned long, arg3,return -EINVAL ; error =PAC_RESET_KEYS (me, arg2);
Quoted 8 lines
kernel/sys.c SYSCALL_DEFINE5(prctl, int, option, unsigned long, arg2, unsigned long, arg3, · line 2491
Quoted 4 lines
kernel/sys.c +3
-0break ; +case PR_CORE_ISOLATE : +#ifdef CONFIG_SCHED_CORE + current->core_cookie = (unsigned long )current;This needs to then also force a reschedule of current. And there's the little issue of what happens if 'current' dies while its children live on, and current gets re-used for a new process and does this again.
sched_core_get() too?
Cheers,
Phil
Quoted 12 lines
Quoted 12 lines
kernel/sys.c +4
-0+#else + result = -EINVAL ; +#endif +break ;default : error = -EINVAL ;break ;Signature · 2 lines
-- Matthew Garrett | [email protected]
Quoted 8 lines · Peter Zijlstra
Quoted 6 lines · Matthew Garrett
Apple have provided a sysctl that allows applications to indicate that specific threads should make use of core isolation while allowing the rest of the system to make use of SMT, and browsers (Safari, Firefox and Chrome, at least) are now making use of this. Trying to do something similar using cgroups seems a bit awkward. Would something like this be reasonable?
Quoted 9 lines
Sure; like I wrote earlier; I only did the cgroup thing because I was lazy and it was the easiest interface to hack on in a hurry.
The rest of the ABI nonsense can 'trivially' be done later; if when we decide to actually do this.
And given MDS, I'm still not entirely convinced it all makes sense. If it were just L1TF, then yes, but now...
For MDS, core-scheduler does prevent thread to thread attack between user space threads running on sibling CPU threads. Yes, it doesn't prevent the user to kernel attack from sibling which will require additional mitigation measure. However, it does block a major attack vector for MDS if HT is enabled.
Tim
Peter Zijlstra
maintainerQuoted 7 lines · Phil Auld
Quoted 2 lines · On Tue, Aug 27, 2019 at 11:50:35PM +0200 Peter Zijlstra wrote:
And given MDS, I'm still not entirely convinced it all makes sense. If it were just L1TF, then yes, but now...
I was thinking MDS is really the reason for this. L1TF has mitigations but the only current mitigation for MDS for smt is ... nosmt.
L1TF has no known mitigation that is SMT safe. The moment you have something in your L1, the other sibling can read it using L1TF.
The nice thing about L1TF is that only (malicious) guests can exploit it, and therefore the synchronizatin context is VMM. And it so happens that VMEXITs are 'rare' (and already expensive and thus lots of effort has already gone into avoiding them).
If you don't use VMs, you're good and SMT is not a problem.
If you do use VMs (and do/can not trust them), _then_ you need core-scheduling; and in that case, the implementation under discussion misses things like synchronization on VMEXITs due to interrupts and things like that.
But under the assumption that VMs don't generate high scheduling rates, it can work.
Quoted 3 lines
The current core scheduler implementation, I believe, still has (theoretical?) holes involving interrupts, once/if those are closed it may be even less attractive.
No; so MDS leaks anything the other sibling (currently) does, this makes _any_ privilidge boundary a synchronization context.
Worse still, the exploit doesn't require a VM at all, any other task can get to it.
That means you get to sync the siblings on lovely things like system call entry and exit, along with VMM and anything else that one would consider a privilidge boundary. Now, system calls are not rare, they are really quite common in fact. Trying to sync up siblings at the rate of system calls is utter madness.
So under MDS, SMT is completely hosed. If you use VMs exclusively, then it _might_ work because a 'pure' host doesn't schedule that often (maybe, same assumption as for L1TF).
Now, there have been proposals of moving the privilidge boundary further into the kernel. Just like PTI exposes the entry stack and code to Meltdown, the thinking is, lets expose more. By moving the priv boundary the hope is that we can do lots of common system calls without having to sync up -- lots of details are 'pending'.
Peter Zijlstra
maintainerQuoted 19 lines · Tim Chen
Quoted 8 lines · Peter Zijlstra
Quoted 6 lines · Matthew Garrett
Apple have provided a sysctl that allows applications to indicate that specific threads should make use of core isolation while allowing the rest of the system to make use of SMT, and browsers (Safari, Firefox and Chrome, at least) are now making use of this. Trying to do something similar using cgroups seems a bit awkward. Would something like this be reasonable?
Quoted 9 lines
Sure; like I wrote earlier; I only did the cgroup thing because I was lazy and it was the easiest interface to hack on in a hurry.
The rest of the ABI nonsense can 'trivially' be done later; if when we decide to actually do this.
And given MDS, I'm still not entirely convinced it all makes sense. If it were just L1TF, then yes, but now...
Quoted 5 lines
For MDS, core-scheduler does prevent thread to thread attack between user space threads running on sibling CPU threads. Yes, it doesn't prevent the user to kernel attack from sibling which will require additional mitigation measure. However, it does block a major attack vector for MDS if HT is enabled.
I'm not sure what your argument is; the dike has two holes; you plug one, you still drown.
Quoted 23 lines · Peter Zijlstra
Quoted 5 lines · Phil Auld
On Tue, Aug 27, 2019 at 11:50:35PM +0200 Peter Zijlstra wrote:
The current core scheduler implementation, I believe, still has (theoretical?) holes involving interrupts, once/if those are closed it may be even less attractive.
No; so MDS leaks anything the other sibling (currently) does, this makes _any_ privilidge boundary a synchronization context.
Worse still, the exploit doesn't require a VM at all, any other task can get to it.
That means you get to sync the siblings on lovely things like system call entry and exit, along with VMM and anything else that one would consider a privilidge boundary. Now, system calls are not rare, they are really quite common in fact. Trying to sync up siblings at the rate of system calls is utter madness.
So under MDS, SMT is completely hosed. If you use VMs exclusively, then it _might_ work because a 'pure' host doesn't schedule that often (maybe, same assumption as for L1TF).
Quoted 6 lines
Now, there have been proposals of moving the privilidge boundary further into the kernel. Just like PTI exposes the entry stack and code to Meltdown, the thinking is, lets expose more. By moving the priv boundary the hope is that we can do lots of common system calls without having to sync up -- lots of details are 'pending'.
If are willing to consider the idea that we will sync with the sibling only if we touch potential user data, then a significant portion of syscalls may not need to sync. Yeah, it still sucks because of the complexity added to audit all the places in kernel that may touch privileged data and require synchronization.
I did a prototype (without core sched), kernel build slow 2.5%. So this use case still seem reasonable.
A worst case scenario is concurrent SMT FIO write to encrypted file, which have a lot of synchronizations due to extended access to privilege data by crypto, we slow by 9%.
Tim
Quoted 47 lines · On Wed, Aug 28, 2019 at 06:01:14PM +0200 Peter Zijlstra wrote:
Quoted 7 lines · Phil Auld
Quoted 2 lines · On Tue, Aug 27, 2019 at 11:50:35PM +0200 Peter Zijlstra wrote:
And given MDS, I'm still not entirely convinced it all makes sense. If it were just L1TF, then yes, but now...
I was thinking MDS is really the reason for this. L1TF has mitigations but the only current mitigation for MDS for smt is ... nosmt.
L1TF has no known mitigation that is SMT safe. The moment you have something in your L1, the other sibling can read it using L1TF.
The nice thing about L1TF is that only (malicious) guests can exploit it, and therefore the synchronizatin context is VMM. And it so happens that VMEXITs are 'rare' (and already expensive and thus lots of effort has already gone into avoiding them).
If you don't use VMs, you're good and SMT is not a problem.
If you do use VMs (and do/can not trust them), _then_ you need core-scheduling; and in that case, the implementation under discussion misses things like synchronization on VMEXITs due to interrupts and things like that.
But under the assumption that VMs don't generate high scheduling rates, it can work.
Quoted 3 lines
The current core scheduler implementation, I believe, still has (theoretical?) holes involving interrupts, once/if those are closed it may be even less attractive.
No; so MDS leaks anything the other sibling (currently) does, this makes _any_ privilidge boundary a synchronization context.
Worse still, the exploit doesn't require a VM at all, any other task can get to it.
That means you get to sync the siblings on lovely things like system call entry and exit, along with VMM and anything else that one would consider a privilidge boundary. Now, system calls are not rare, they are really quite common in fact. Trying to sync up siblings at the rate of system calls is utter madness.
So under MDS, SMT is completely hosed. If you use VMs exclusively, then it _might_ work because a 'pure' host doesn't schedule that often (maybe, same assumption as for L1TF).
Quoted 5 lines
Now, there have been proposals of moving the privilidge boundary further into the kernel. Just like PTI exposes the entry stack and code to Meltdown, the thinking is, lets expose more. By moving the priv boundary the hope is that we can do lots of common system calls without having to sync up -- lots of details are 'pending'.
Thanks for clarifying. My understanding is (somewhat) less fuzzy now. :)
I think, though, that you were basically agreeing with me that the current core scheduler does not close the holes, or am I reading that wrong.
Cheers,
Phil
Peter Zijlstra
maintainerQuoted 55 lines · Phil Auld
Quoted 47 lines · On Wed, Aug 28, 2019 at 06:01:14PM +0200 Peter Zijlstra wrote:
Quoted 7 lines · Phil Auld
Quoted 2 lines · On Tue, Aug 27, 2019 at 11:50:35PM +0200 Peter Zijlstra wrote:
And given MDS, I'm still not entirely convinced it all makes sense. If it were just L1TF, then yes, but now...
I was thinking MDS is really the reason for this. L1TF has mitigations but the only current mitigation for MDS for smt is ... nosmt.
L1TF has no known mitigation that is SMT safe. The moment you have something in your L1, the other sibling can read it using L1TF.
The nice thing about L1TF is that only (malicious) guests can exploit it, and therefore the synchronizatin context is VMM. And it so happens that VMEXITs are 'rare' (and already expensive and thus lots of effort has already gone into avoiding them).
If you don't use VMs, you're good and SMT is not a problem.
If you do use VMs (and do/can not trust them), _then_ you need core-scheduling; and in that case, the implementation under discussion misses things like synchronization on VMEXITs due to interrupts and things like that.
But under the assumption that VMs don't generate high scheduling rates, it can work.
Quoted 3 lines
The current core scheduler implementation, I believe, still has (theoretical?) holes involving interrupts, once/if those are closed it may be even less attractive.
No; so MDS leaks anything the other sibling (currently) does, this makes _any_ privilidge boundary a synchronization context.
Worse still, the exploit doesn't require a VM at all, any other task can get to it.
That means you get to sync the siblings on lovely things like system call entry and exit, along with VMM and anything else that one would consider a privilidge boundary. Now, system calls are not rare, they are really quite common in fact. Trying to sync up siblings at the rate of system calls is utter madness.
So under MDS, SMT is completely hosed. If you use VMs exclusively, then it _might_ work because a 'pure' host doesn't schedule that often (maybe, same assumption as for L1TF).
Quoted 5 lines
Now, there have been proposals of moving the privilidge boundary further into the kernel. Just like PTI exposes the entry stack and code to Meltdown, the thinking is, lets expose more. By moving the priv boundary the hope is that we can do lots of common system calls without having to sync up -- lots of details are 'pending'.
Quoted 4 lines
Thanks for clarifying. My understanding is (somewhat) less fuzzy now. :)
I think, though, that you were basically agreeing with me that the current core scheduler does not close the holes, or am I reading that wrong.
Agreed; the missing bits for L1TF are ugly but doable (I've actually done them before, Tim has that _somewhere_), but I've not seen a 'workable' solution for MDS yet.
Quoted 32 lines
1) Unfairness between the sibling threads
-----------------------------------------
One sibling thread could be suppressing and force idling the sibling thread over proportionally. Resulting in the force idled CPU not getting run and stall tasks on suppressed CPU.Status:
i) Aaron has proposed a patchset here based on using one rq as a base reference for vruntime for task priority comparison between siblings.https://lore.kernel.org/lkml/20190725143248.GC992@aaronlu/
It works well on fairness but has some initialization issuesii) Tim has proposed a patchset here to account for forced idle time in rq's
min_vruntime
https://lore.kernel.org/lkml/[email protected]/
It improves over v3 with simpler logic compared to Aaron's patch, but does not work as well on fairnessiii) Tim has proposed yet another patch to maintain fairness of forced idle time between CPU threads per Peter's suggestion.
https://lore.kernel.org/lkml/[email protected]/
Its performance has yet to be tested.2) Not rescheduling forced idled CPU
------------------------------------
The forced idled CPU does not get a chance to re-schedule itself, and will stall for a long time even though it has eligible tasks to run.
Quoted 8 lines
Status:
i) Aaron proposed a patch to fix this to check if there are runnable tasks when scheduling tick comes in.
https://lore.kernel.org/lkml/20190725143344.GD992@aaronlu/ii) Vineeth has patches to this issue and also issue 1, based on scheduling in a new "forced idle task" when getting forced idle, but has yet to post the patches.
We finished writing and debugging the PoC for the coresched_idle task and here are the results and the code.
Those patches are applied on top of Aaron's patches:
- sched: Fix incorrect rq tagged as forced idle
- wrapper for cfs_rq->min_vruntime
https://lore.kernel.org/lkml/20190725143127.GB992@aaronlu/
- core vruntime comparison
https://lore.kernel.org/lkml/20190725143248.GC992@aaronlu/
For the testing, we used the same strategy as described in
https://lore.kernel.org/lkml/20190802153715.GA18075@sinkpad/
No tag
------
Test Average Stdev Alone 1306.90 0.94 nosmt 649.95 1.44 Aaron's full patchset: 828.15 32.45 Aaron's first 2 patches: 832.12 36.53 Tim's first patchset: 852.50 4.11 Tim's second patchset: 855.11 9.89 coresched_idle 985.67 0.83
Sysbench mem untagged, sysbench cpu tagged
------------------------------------------
Test Average Stdev Alone 1306.90 0.94 nosmt 649.95 1.44 Aaron's full patchset: 586.06 1.77 Tim's first patchset: 852.50 4.11 Tim's second patchset: 663.88 44.43 coresched_idle 653.58 0.49
Sysbench mem tagged, sysbench cpu untagged
------------------------------------------
Test Average Stdev Alone 1306.90 0.94 nosmt 649.95 1.44 Aaron's full patchset: 583.77 3.52 Tim's first patchset: 564.04 58.05 Tim's second patchset: 524.72 55.24 coresched_idle 653.30 0.81
Both sysbench tagged
--------------------
Test Average Stdev Alone 1306.90 0.94 nosmt 649.95 1.44 Aaron's full patchset: 582.15 3.75 Tim's first patchset: 679.43 70.07 Tim's second patchset: 563.10 34.58 coresched_idle 653.12 1.68
As we can see from this stress-test, with the coresched_idle thread being a real process, the fairness is more consistent (low stdev). Also, the performance remains the same regardless of the tagging, and even always slightly better than nosmt.
Thanks,
Julien
From: vpillai <[email protected]>
Date: Wed, 4 Sep 2019 17:41:38 +0000
Subject: [RFC PATCH 1/2] coresched_idle thread
--- kernel/sched/core.c | 46 ++++++++++++++++++++++++++++++++++++++++++++ kernel/sched/sched.h | 1 + 2 files changed, 47 insertions(+)
kernel/sched/core.c +46 -0
@@ -3639,6 +3639,51 @@ static inline bool cookie_match(struct task_struct *a, struct task_struct *b)return a->core_cookie == b->core_cookie; } +static int coresched_idle_worker (void *data) +{ +struct rq *rq = (struct rq *)data; + +/* +* Transition to parked state and dequeue from runqueue. +* pick_task() will select us if needed without enqueueing. +*/ +set_special_state (TASK_PARKED ); +schedule (); + +while (true) { +if (kthread_should_stop ()) +break ; + +play_idle (1 ); + } + +return 0 ; +} + +static void coresched_idle_worker_init (struct rq *rq) +{ + +// XXX core_idle_task needs lock protection? +if (!rq->core_idle_task) { + rq->core_idle_task =kthread_create_on_cpu (coresched_idle_worker, + (void *)rq,cpu_of (rq),"coresched_idle" ); +if (rq->core_idle_task) { +wake_up_process (rq->core_idle_task); + } + + } + +return ; +} + +static void coresched_idle_worker_fini (struct rq *rq) +{ +if (rq->core_idle_task) { +kthread_stop (rq->core_idle_task); + rq->core_idle_task =NULL ; + } +} +// XXX fairness/fwd progress conditions /* * Returns
@@ -6774,6 +6819,7 @@ void __init sched_init(void)atomic_set (&rq->nr_iowait,0 );#ifdef CONFIG_SCHED_CORE + rq->core_idle_task =NULL ; rq->core =NULL ; rq->core_pick =NULL ; rq->core_enabled =0 ;
kernel/sched/sched.h +1 -0
@@ -965,6 +965,7 @@ struct rq {unsigned int core_sched_seq;struct rb_root core_tree;bool core_forceidle; +struct task_struct *core_idle_task;/* shared state */ unsigned int core_task_seq;
Signature · 12 lines
2.17.1 From: vpillai <[email protected]> Date: Wed, 4 Sep 2019 18:22:55 +0000 Subject: [RFC PATCH 2/2] Use coresched_idle to force idle a sibling Currently we use idle thread to force idle on a sibling. Lets use the new coresched_idle thread that scheduler sees a valid task during force idle. --- kernel/sched/core.c | 66 ++++++++++++++++++++++++++++++++++++++------- 1 file changed, 56 insertions(+), 10 deletions(-)
kernel/sched/core.c +56 -10
172 lines not on this page · raw message
Signature · 1 line
2.17.1
Quoted 12 lines · Tim Chen
3) Load balancing between CPU cores
-----------------------------------
Say if one CPU core's sibling threads get forced idled a lot as it has mostly incompatible tasks between the siblings, moving the incompatible load to other cores and pulling compatible load to the core could help CPU utilization.So just considering the load of a task is not enough during load balancing, task compatibility also needs to be considered. Peter has put in mechanisms to balance compatible tasks between CPU thread siblings, but not across cores.
Quoted 5 lines
Status:
I have not seen patches on this issue. This issue could lead to large variance in workload performance based on your luck in placing the workload among the cores.
I've made an attempt in the following two patches to address the load balancing of mismatched load between the siblings.
It is applied on top of Aaron's patches:
- sched: Fix incorrect rq tagged as forced idle
- wrapper for cfs_rq->min_vruntime
https://lore.kernel.org/lkml/20190725143127.GB992@aaronlu/
- core vruntime comparison
https://lore.kernel.org/lkml/20190725143248.GC992@aaronlu/
I will love Julien, Aaron and others to try it out. Suggestions to tune it is welcomed.
Tim
---
From c7b91fb26d787d020f0795c3fbec82914889dc67 Mon Sep 17 00:00:00 2001
From: Tim Chen <[email protected]>
Date: Wed, 21 Aug 2019 15:48:15 -0700
Subject: [PATCH 1/2] sched: scan core sched load mismatch
Calculate the mismatched load imbalance on a core when running the core scheduler when we are updating the load balance statistics. This will guide the load balancer later to move load to another CPU that can reduce the mismatched load.
--- kernel/sched/fair.c | 150 +++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 149 insertions(+), 1 deletion(-)
kernel/sched/fair.c +149 -1
@@ -7507,6 +7507,9 @@ static inline int migrate_degrades_locality(struct task_struct *p, }#endif +static inline s64 core_sched_imbalance_improvement (int src_cpu,int dst_cpu, +struct task_struct *p); +/* * can_migrate_task - may task p from runqueue rq be migrated to this_cpu? */
@@ -7970,6 +7973,11 @@ struct sg_lb_stats {unsigned int nr_numa_running;unsigned int nr_preferred_running;#endif +#ifdef CONFIG_SCHED_CORE +int imbl_cpu; +struct task_group *imbl_tg; +s64 imbl_load; +#endif };/*
@@ -8314,6 +8322,145 @@ static bool update_nohz_stats(struct rq *rq, bool force)#endif } +#ifdef CONFIG_SCHED_CORE +static inline int cpu_sibling (int cpu) +{ +int i; + +for_each_cpu (i,cpu_smt_mask (cpu)) { +if (i == cpu) +continue ; +return i; + } +return -1 ; +} + +static inline s64 core_sched_imbalance_delta (int src_cpu,int dst_cpu, +int src_sibling,int dst_sibling, +struct task_group *tg,u64 task_load) +{ +struct sched_entity *se, *se_sibling, *dst_se, *dst_se_sibling; +s64 excess, deficit, old_mismatch, new_mismatch; + +if (src_cpu == dst_cpu) +return -1 ; + +/* XXX SMT4 will require additional logic */ + + se = tg->se[src_cpu]; + se_sibling = tg->se[src_sibling]; + + excess = se->avg.load_avg - se_sibling->avg.load_avg; +if (src_sibling == dst_cpu) { + old_mismatch =abs (excess); + new_mismatch =abs (excess -2 *task_load); +return old_mismatch - new_mismatch; + } + + dst_se = tg->se[dst_cpu]; + dst_se_sibling = tg->se[dst_sibling]; + deficit = dst_se->avg.load_avg - dst_se_sibling->avg.load_avg; + + old_mismatch =abs (excess) +abs (deficit); + new_mismatch =abs (excess - (s64 ) task_load) + +abs (deficit + (s64 ) task_load); + +if (excess >0 && deficit <0 ) +return old_mismatch - new_mismatch; +else +/* no mismatch improvement */ +return -1 ; +} + +static inline s64 core_sched_imbalance_improvement (int src_cpu,int dst_cpu, +struct task_struct *p) +{ +int src_sibling, dst_sibling; +unsigned long task_load =task_h_load (p); +struct task_group *tg; + +if (!p->se.parent) +return 0 ; + + tg = p->se.parent->cfs_rq->tg; +if (!tg->tagged) +return 0 ; + +/* XXX SMT4 will require additional logic */ + src_sibling =cpu_sibling (src_cpu); + dst_sibling =cpu_sibling (dst_cpu); + +if (src_sibling ==-1 || dst_sibling ==-1 ) +return 0 ; + +return core_sched_imbalance_delta (src_cpu, dst_cpu, + src_sibling, dst_sibling, + tg, task_load); +} + +static inline void core_sched_imbalance_scan (struct sg_lb_stats *sgs, +int src_cpu, +int dst_cpu) +{ +struct rq *rq; +struct cfs_rq *cfs_rq, *pos; +struct task_group *tg; +s64 mismatch; +int src_sibling, dst_sibling; +u64 src_avg_load_task; + +if (!sched_core_enabled (cpu_rq (src_cpu)) || + !sched_core_enabled (cpu_rq (dst_cpu)) || + src_cpu == dst_cpu) +return ; + + rq =cpu_rq (src_cpu); + + src_sibling =cpu_sibling (src_cpu); + dst_sibling =cpu_sibling (dst_cpu); + +if (src_sibling ==-1 || dst_sibling ==-1 ) +return ; + + src_avg_load_task =cpu_avg_load_per_task (src_cpu); + +if (src_avg_load_task ==0 ) +return ; + +/* +* Imbalance in tagged task group's load causes forced +* idle time in sibling, that will be counted as mismatched load +* on the forced idled cpu. Record the source cpu in the sched +* group causing the largest mismatched load. +*/ +for_each_leaf_cfs_rq_safe (rq, cfs_rq, pos) { + + tg = cfs_rq->tg; +if (!tg->tagged) +continue ; + + mismatch =core_sched_imbalance_delta (src_cpu, dst_cpu, + src_sibling, dst_sibling, + tg, src_avg_load_task); + +if (mismatch > sgs->imbl_load && + mismatch > src_avg_load_task) { + sgs->imbl_load = mismatch; + sgs->imbl_tg = tg; + sgs->imbl_cpu = src_cpu; + } + } +} + +#else +#define core_sched_imbalance_scan (sgs, src_cpu, dst_cpu) +static inline s64 core_sched_imbalance_improvement (int src_cpu,int dst_cpu, +struct task_struct *p) +{ +return 0 ; +} +#endif /* CONFIG_SCHED_CORE */ +/** * update_sg_lb_stats - Update sched_group's statistics for load balancing. * @env: The load balancing environment.
@@ -8345,7 +8492,8 @@ static inline void update_sg_lb_stats(struct lb_env *env,else load =source_load (i, load_idx); - sgs->group_load += load; +core_sched_imbalance_scan (sgs, i, env->dst_cpu); + sgs->group_util +=cpu_util (i); sgs->sum_nr_running += rq->cfs.h_nr_running;
Signature · 20 lines
2.20.1 From a11084f84de9c174f36cf2701ba5bbe1546e45f5 Mon Sep 17 00:00:00 2001 From: Tim Chen <[email protected]> Date: Wed, 28 Aug 2019 11:22:43 -0700 Subject: [PATCH 2/2] sched: load balance core imbalanced load If moving mismatched core scheduling load can reduce load imbalance more than regular load balancing, move the mismatched load instead. On regular load balancing, also skip moving a task that could increase load mismatch. Move only one mismatched task at a time to reduce load disturbance. Signed-off-by: Tim Chen <[email protected]> --- kernel/sched/fair.c | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+)
kernel/sched/fair.c +28 -0
@@ -7412,6 +7412,11 @@ struct lb_env {enum fbq_type fbq_type;enum group_type src_grp_type;struct list_head tasks; +#ifdef CONFIG_SCHED_CORE +int imbl_cpu; +struct task_group *imbl_tg; +s64 imbl_load; +#endif };/*
@@ -7560,6 +7565,12 @@ int can_migrate_task(struct task_struct *p, struct lb_env *env)return 0 ; } +#ifdef CONFIG_SCHED_CORE +/* Don't migrate if we increase core imbalance */ +if (core_sched_imbalance_improvement (env->src_cpu, env->dst_cpu, p) <0 ) +return 0 ; +#endif +/* Record that we found atleast one task that could run on dst_cpu */ env->flags &= ~LBF_ALL_PINNED ;
@@ -8533,6 +8544,14 @@ static inline void update_sg_lb_stats(struct lb_env *env, sgs->group_no_capacity =group_is_overloaded (env, sgs); sgs->group_type =group_classify (group, sgs); + +#ifdef CONFIG_SCHED_CORE +if (sgs->imbl_load > env->imbl_load) { + env->imbl_cpu = sgs->imbl_cpu; + env->imbl_tg = sgs->imbl_tg; + env->imbl_load = sgs->imbl_load; + } +#endif }/**
@@ -9066,6 +9085,15 @@ static struct rq *find_busiest_queue(struct lb_env *env,unsigned long busiest_load =0 , busiest_capacity =1 ;int i; +#ifdef CONFIG_SCHED_CORE +if (env->imbl_load > env->imbalance) { + env->imbalance =cpu_avg_load_per_task (env->imbl_cpu); +return cpu_rq (env->imbl_cpu); + }else { + env->imbl_load =0 ; + } +#endif +for_each_cpu_and (i,sched_group_span (group), env->cpus) {unsigned long capacity, wl;enum fbq_type rt;
Signature · 1 line
2.20.1
kernel/sched/core.c pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf) · line 3883
Quoted 6 lines · Julien Desfossez
kernel/sched/core.c +1
-1@@ -3853,7 +3880,7 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf) goto done; } -if (!is_idle_task (p)) +if (!is_force_idle_task (p))
Should this be if (!is_core_idle_task(p))
instead?
Review pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf) · line 3885
Quoted 2 lines
(quoted diff) +0
-0occ++;
Tim
Quoted 7 lines · Peter Zijlstra
Quoted 2 lines · Phil Auld
I think, though, that you were basically agreeing with me that the current core scheduler does not close the holes, or am I reading that wrong.
Agreed; the missing bits for L1TF are ugly but doable (I've actually done them before, Tim has that _somewhere_), but I've not seen a 'workable' solution for MDS yet.
Following the discussion we had yesterday at LPC, after we have agreed on a solution for fixing the current fairness issue, we will post the v4. We will then work on prototyping the other synchronisation points (syscalls, interrupts and VMEXIT) to evaluate the overhead in various use-cases.
Depending on the use-case, we know the performance overhead maybe heavier than just disabling SMT, but the benchmarks we have seen so far indicate that there are valid cases for core scheduling. Core scheduling will continue to be unused by default, but with it, we will have the option to tune the system to be both secure and faster than disabling SMT for those cases.
Thanks,
Julien
P.S: I think the branch that contains the VMEXIT handling is here
https://github.com/pdxChen/gang/commits/sched_1.23-base
Hi Tim & Julien,
Quoted 23 lines · Tim Chen
Quoted 12 lines · Tim Chen
3) Load balancing between CPU cores
-----------------------------------
Say if one CPU core's sibling threads get forced idled a lot as it has mostly incompatible tasks between the siblings, moving the incompatible load to other cores and pulling compatible load to the core could help CPU utilization.So just considering the load of a task is not enough during load balancing, task compatibility also needs to be considered. Peter has put in mechanisms to balance compatible tasks between CPU thread siblings, but not across cores.
Quoted 5 lines
Status:
I have not seen patches on this issue. This issue could lead to large variance in workload performance based on your luck in placing the workload among the cores.I've made an attempt in the following two patches to address the load balancing of mismatched load between the siblings.
Quoted 6 lines
It is applied on top of Aaron's patches:
- sched: Fix incorrect rq tagged as forced idle
- wrapper forcfs_rq->min_vruntime
https://lore.kernel.org/lkml/20190725143127.GB992@aaronlu/
- core vruntime comparison
https://lore.kernel.org/lkml/20190725143248.GC992@aaronlu/
So both of you are working on top of my 2 patches that deal with the fairness issue, but I had the feeling Tim's alternative patches[1] are simpler than mine and achieves the same result(after the force idle tag fix), so unless there is something I missed, I think we should go with the simpler one?
Quoted 33 lines · Aaron Lu
Hi Tim & Julien,
Quoted 23 lines · Tim Chen
Quoted 12 lines · Tim Chen
3) Load balancing between CPU cores
-----------------------------------
Say if one CPU core's sibling threads get forced idled a lot as it has mostly incompatible tasks between the siblings, moving the incompatible load to other cores and pulling compatible load to the core could help CPU utilization.So just considering the load of a task is not enough during load balancing, task compatibility also needs to be considered. Peter has put in mechanisms to balance compatible tasks between CPU thread siblings, but not across cores.
Quoted 5 lines
Status:
I have not seen patches on this issue. This issue could lead to large variance in workload performance based on your luck in placing the workload among the cores.I've made an attempt in the following two patches to address the load balancing of mismatched load between the siblings.
Quoted 6 lines
It is applied on top of Aaron's patches:
- sched: Fix incorrect rq tagged as forced idle
- wrapper forcfs_rq->min_vruntime
https://lore.kernel.org/lkml/20190725143127.GB992@aaronlu/
- core vruntime comparison
https://lore.kernel.org/lkml/20190725143248.GC992@aaronlu/
Quoted 3 lines
So both of you are working on top of my 2 patches that deal with the fairness issue, but I had the feeling Tim's alternative patches[1] are simpler than mine and achieves the same result(after the force idle tag
I think Julien's result show that my patches did not do as well as your patches for fairness. Aubrey did some other testing with the same conclusion. So I think keeping the forced idle time balanced is not enough for maintaining fairness.
Will love to see if my load balancing patches help for your workload.
Tim
Quoted 5 lines
fix), so unless there is something I missed, I think we should go with the simpler one?
Quoted 9 lines
Quoted 3 lines
So both of you are working on top of my 2 patches that deal with the fairness issue, but I had the feeling Tim's alternative patches[1] are simpler than mine and achieves the same result(after the force idle tag
I think Julien's result show that my patches did not do as well as your patches for fairness. Aubrey did some other testing with the same conclusion. So I think keeping the forced idle time balanced is not enough for maintaining fairness.
There are two main issues - vruntime comparison issue and the forced idle issue. coresched_idle thread patch is addressing the forced idle issue as scheduler is no longer overloading idle thread for forcing idle. If I understand correctly, Tim's patch also tries to fix the forced idle issue. On top of fixing forced idle issue, we also need to fix that vruntime comparison issue and I think thats where Aaron's patch helps.
I think comparing parent's runtime also will have issues once the task group has a lot more threads with different running patterns. One example is a task group with lot of active threads and a thread with fairly less activity. So when this less active thread is competing with a thread in another group, there is a chance that it loses continuously for a while until the other group catches up on its vruntime.
As discussed during LPC, probably start thinking along the lines of global vruntime or core wide vruntime to fix the vruntime comparison issue?
Thanks,
Vineeth
Quoted 34 lines · Tim Chen
Quoted 33 lines · Aaron Lu
Hi Tim & Julien,
Quoted 29 lines · Tim Chen
Quoted 12 lines · Tim Chen
3) Load balancing between CPU cores
-----------------------------------
Say if one CPU core's sibling threads get forced idled a lot as it has mostly incompatible tasks between the siblings, moving the incompatible load to other cores and pulling compatible load to the core could help CPU utilization.So just considering the load of a task is not enough during load balancing, task compatibility also needs to be considered. Peter has put in mechanisms to balance compatible tasks between CPU thread siblings, but not across cores.
Quoted 5 lines
Status:
I have not seen patches on this issue. This issue could lead to large variance in workload performance based on your luck in placing the workload among the cores.I've made an attempt in the following two patches to address the load balancing of mismatched load between the siblings.
It is applied on top of Aaron's patches:
- sched: Fix incorrect rq tagged as forced idle
- wrapper forcfs_rq->min_vruntime
https://lore.kernel.org/lkml/20190725143127.GB992@aaronlu/
- core vruntime comparison
https://lore.kernel.org/lkml/20190725143248.GC992@aaronlu/
Quoted 8 lines
Quoted 3 lines
So both of you are working on top of my 2 patches that deal with the fairness issue, but I had the feeling Tim's alternative patches[1] are simpler than mine and achieves the same result(after the force idle tag
I think Julien's result show that my patches did not do as well as your patches for fairness. Aubrey did some other testing with the same conclusion. So I think keeping the forced idle time balanced is not enough for maintaining fairness.
Well, I have done following tests:
1 Julien's test script: https://paste.debian.net/plainh/834cf45c
2 start two tagged will-it-scale/page_fault1, see how each performs; 3 Aubrey's mysql test: https://github.com/aubreyli/coresched_bench.git
They all show your patchset performs equally well...And consider what the patch does, I think they are really doing the same thing in different ways.
Quoted 9 lines · Vineeth Remanan Pillai
Quoted 9 lines
Quoted 3 lines
So both of you are working on top of my 2 patches that deal with the fairness issue, but I had the feeling Tim's alternative patches[1] are simpler than mine and achieves the same result(after the force idle tag
I think Julien's result show that my patches did not do as well as your patches for fairness. Aubrey did some other testing with the same conclusion. So I think keeping the forced idle time balanced is not enough for maintaining fairness.
Quoted 5 lines
There are two main issues - vruntime comparison issue and the forced idle issue.
coresched_idlethread patch is addressing the forced idle issue as scheduler is no longer overloading idle thread for forcing idle. If I understand correctly, Tim's patch also tries to fix the forced idle issue. On top of fixing forced
Er...I don't think so. Tim's patch is meant to solve fairness issue as mine, it doesn't attempt to address the forced idle issue.
Quoted 10 lines
idle issue, we also need to fix that vruntime comparison issue and I think thats where Aaron's patch helps.
I think comparing parent's runtime also will have issues once the task group has a lot more threads with different running patterns. One example is a task group with lot of active threads and a thread with fairly less activity. So when this less active thread is competing with a thread in another group, there is a chance that it loses continuously for a while until the other group catches up on its vruntime.
I actually think this is expected behaviour.
Without core scheduling, when deciding which task to run, we will first decide which "se" to run from the CPU's root level cfs runqueue and then go downwards. Let's call the chosen se on the root level cfs runqueue the winner se. Then with core scheduling, we will also need compare the two winner "se"s of each hyperthread and choose the core wide winner "se".
Quoted 4 lines
As discussed during LPC, probably start thinking along the lines of global vruntime or core wide vruntime to fix the vruntime comparison issue?
core wide vruntime makes sense when there are multiple tasks of different cgroups queued on the same core. e.g. when there are two tasks of cgroupA and one task of cgroupB are queued on the same core, assume cgroupA's one task is on one hyperthread and its other task is on the other hyperthread with cgroupB's task. With my current implementation or Tim's, cgroupA will get more time than cgroupB. If we maintain core wide vruntime for cgroupA and cgroupB, we should be able to maintain fairness between cgroups on this core. Tim propose to solve this problem by doing some kind of load balancing if I'm not mistaken, I haven't taken a look at this yet.
Quoted 9 lines · Aaron Lu
Well, I have done following tests:
1 Julien's test script: https://paste.debian.net/plainh/834cf45c
2 start two tagged will-it-scale/page_fault1, see how each performs; 3 Aubrey's mysql test: https://github.com/aubreyli/coresched_bench.gitThey all show your patchset performs equally well...And consider what the patch does, I think they are really doing the same thing in different ways.
Aaron,
The new feature of my new patches attempt to load balance between cores, and remove imbalance of cgroup load on a core that causes forced idle. Whereas previous patches attempt for fairness of cgroup between sibling threads, so I think the goals are kind of orthogonal and complementary.
The premise is this, say cgroup1 is occupying 50% of cpu on cpu thread 1 and 25% of cpu on cpu thread 2, that means we have a 25% cpu imbalance and cpu is force idled 25% of the time. So ideally we need to remove 12.5% of cgroup 1 load from cpu thread 1 to sibling thread 2, so they both run at 37.5% on both thread for cgroup1 load without causing any force idled time. Otherwise we will try to remove 25% of cgroup1 load from cpu thread 1 to another core that has cgroup1 load to match.
This load balance is done in the regular load balance paths.
Previously for v3, only sched_core_balance made an attempt to pull a cookie task, and only in the idle balance path. So if the cpu is kept busy, the cgroup load imbalance between sibling threads could last a long time. And the thread fairness patches for v3 don't help to balance load for such cases.
The new patches take into actual consideration of the amount of load imbalance of the same group between sibling threads when selecting task to pull, and it also prevent task migration that creates
more load imbalance. So hopefully this feature will help when we have more cores and need load balance across the cores. This tries to help even cgroup workload between threads to minimize forced idle time, and also even out load across cores.
In your test, how many cores are on your machine and how many threads did each page_fault1 spawn off?
Tim
Quoted 9 lines · Aaron Lu
On Wed, Sep 11, 2019 at 12:47:34PM -0400, Vineeth Remanan Pillai wrote:
core wide vruntime makes sense when there are multiple tasks of different cgroups queued on the same core. e.g. when there are two tasks of cgroupA and one task of cgroupB are queued on the same core, assume cgroupA's one task is on one hyperthread and its other task is on the other hyperthread with cgroupB's task. With my current implementation or Tim's, cgroupA will get more time than cgroupB.
I think that's expected because cgroup A has two tasks and cgroup B has one task, so cgroup A should get twice the cpu time than cgroup B to maintain fairness.
Quoted 3 lines
If we
maintain core wide vruntime for cgroupA and cgroupB, we should be able to maintain fairness between cgroups on this core.
I don't think the right thing to do is to give cgroupA and cgroupB equal time on a core. The time they get should still depend on their load weight. The better thing to do is to move one task from cgroupA to another core, that has only one cgroupA task so it can be paired up with that lonely cgroupA task. This will eliminate the forced idle time for cgropuA both on current core and also the migrated core.
Quoted 4 lines
Tim propose to solve
this problem by doing some kind of load balancing if I'm not mistaken, I haven't taken a look at this yet.
My new patchset is trying to solve a different problem. It is not trying to maintain fairness between cgroup on a core, but tries to even out the load of a cgroup between threads, and even out general load between cores. This will minimize the forced idle time.
The fairness between cgroup relies still on
proper vruntime accounting and proper comparison of vruntime between threads. So for now, I am still using Aaron's patchset for this purpose as it has better fairness property than my other proposed patchsets for fairness purpose.
With just Aaron's current patchset we may have a lot of forced idle time due to the uneven distribution of tasks of different cgroup among the threads and cores, even though scheduling fairness is maintained. My new patches try to remove those forced idle time by moving the tasks around, to minimize cgroup unevenness between sibling threads and general load unevenness between the CPUs.
Tim
Quoted 8 lines · Aaron Lu
Quoted 5 lines · Tim Chen
On 9/11/19 7:02 AM, Aaron Lu wrote:
I think Julien's result show that my patches did not do as well as your patches for fairness. Aubrey did some other testing with the same conclusion. So I think keeping the forced idle time balanced is not enough for maintaining fairness.
Quoted 8 lines
Well, I have done following tests:
1 Julien's test script: https://paste.debian.net/plainh/834cf45c
2 start two tagged will-it-scale/page_fault1, see how each performs; 3 Aubrey's mysql test: https://github.com/aubreyli/coresched_bench.gitThey all show your patchset performs equally well...And consider what the patch does, I think they are really doing the same thing in different ways.
It looks like we are not on the same page, if you don't mind, can both of you rebase your patchset onto v5.3-rc8 and provide a public branch so I can fetch and test it at least by my benchmark?
Thanks,
-Aubrey
Quoted 34 lines · Tim Chen
Quoted 9 lines · Aaron Lu
Well, I have done following tests:
1 Julien's test script: https://paste.debian.net/plainh/834cf45c
2 start two tagged will-it-scale/page_fault1, see how each performs; 3 Aubrey's mysql test: https://github.com/aubreyli/coresched_bench.gitThey all show your patchset performs equally well...And consider what the patch does, I think they are really doing the same thing in different ways.
Aaron,
The new feature of my new patches attempt to load balance between cores, and remove imbalance of cgroup load on a core that causes forced idle. Whereas previous patches attempt for fairness of cgroup between sibling threads, so I think the goals are kind of orthogonal and complementary.
The premise is this, say cgroup1 is occupying 50% of cpu on cpu thread 1 and 25% of cpu on cpu thread 2, that means we have a 25% cpu imbalance and cpu is force idled 25% of the time. So ideally we need to remove 12.5% of cgroup 1 load from cpu thread 1 to sibling thread 2, so they both run at 37.5% on both thread for cgroup1 load without causing any force idled time. Otherwise we will try to remove 25% of cgroup1 load from cpu thread 1 to another core that has cgroup1 load to match.
This load balance is done in the regular load balance paths.
Previously for v3, only
sched_core_balancemade an attempt to pull a cookie task, and only in the idle balance path. So if the cpu is kept busy, the cgroup load imbalance between sibling threads could last a long time. And the thread fairness patches for v3 don't help to balance load for such cases.
Quoted 7 lines
The new patches take into actual consideration of the amount of load imbalance of the same group between sibling threads when selecting task to pull, and it also prevent task migration that creates
more load imbalance. So hopefully this feature will help when we have more cores and need load balance across the cores. This tries to help even cgroup workload between threads to minimize forced idle time, and also even out load across cores.
Will take a look at your new patches, thanks for the explanation.
Quoted 2 lines
In your test, how many cores are on your machine and how many threads did each
page_fault1spawn off?
The test VM has 16 cores and 32 threads.
I created 2 tagged cgroups to run page_fault1 and each page_fault1 has 16 processes, like this:
$ ./src/will-it-scale/page_fault1_processes -t 16 -s 60
Quoted 11 lines · Tim Chen
Quoted 9 lines · Aaron Lu
On Wed, Sep 11, 2019 at 12:47:34PM -0400, Vineeth Remanan Pillai wrote:
core wide vruntime makes sense when there are multiple tasks of different cgroups queued on the same core. e.g. when there are two tasks of cgroupA and one task of cgroupB are queued on the same core, assume cgroupA's one task is on one hyperthread and its other task is on the other hyperthread with cgroupB's task. With my current implementation or Tim's, cgroupA will get more time than cgroupB.
Quoted 3 lines
I think that's expected because cgroup A has two tasks and cgroup B has one task, so cgroup A should get twice the cpu time than cgroup B to maintain fairness.
Like you said below, the ideal run time for each cgroup should depend on their individual weight. The fact cgroupA has two tasks doesn't mean it has twice the weight. Both cgroup can have the same cpu.share settings and then, the more task a cgroup has, the less weight it can get for the cgroup's per-cpu se.
I now realized one thing that's different in your idle_allowance implementation and my core_vruntime implementation. In your implementation, the idle_allowance is absolute time while vruntime can be adjusted by the se's weight, that's probably one area your implementation can make things less fair then mine.
Quoted 7 lines
Quoted 3 lines
If we
maintain core wide vruntime for cgroupA and cgroupB, we should be able to maintain fairness between cgroups on this core.I don't think the right thing to do is to give cgroupA and cgroupB equal time on a core. The time they get should still depend on their load weight.
Agree.
Quoted 4 lines
The better thing to do is to move one task from cgroupA to another core, that has only one cgroupA task so it can be paired up
with that lonely cgroupA task. This will eliminate the forced idle time for cgropuA both on current core and also the migrated core.
I'm not sure if this is always possible.
Say on a 16cores/32threads machine, there are 3 cgroups, each has 16 cpu intensive tasks, will it be possible to make things perfectly balanced?
Don't get me wrong, I think this kind of load balancing is good and needed, but I'm not sure if we can always make things perfectly balanced. And if not, do we care those few cores where cgroup tasks are not balanced and then, do we need to implement the core_wide cgoup fairness functionality or we don't care since those cores are supposed to be few and isn't a big deal?
Quoted 9 lines
Quoted 4 lines
Tim propose to solve
this problem by doing some kind of load balancing if I'm not mistaken, I haven't taken a look at this yet.My new patchset is trying to solve a different problem. It is not trying to maintain fairness between cgroup on a core, but tries to even out the load of a cgroup between threads, and even out general load between cores. This will minimize the forced idle time.
Understood.
Quoted 7 lines
The fairness between cgroup relies still on
proper vruntime accounting and proper comparison of vruntime between threads. So for now, I am still using Aaron's patchset for this purpose as it has better fairness property than my other proposed patchsets for fairness purpose.
Quoted 6 lines
With just Aaron's current patchset we may have a lot of forced idle time due to the uneven distribution of tasks of different cgroup among the threads and cores, even though scheduling fairness is maintained. My new patches try to remove those forced idle time by moving the tasks around, to minimize cgroup unevenness between sibling threads and general load unevenness between the CPUs.
Yes I think this is definitely a good thing to do.
Quoted 9 lines · Aaron Lu
Quoted 4 lines · Tim Chen
The better thing to do is to move one task from cgroupA to another core, that has only one cgroupA task so it can be paired up
with that lonely cgroupA task. This will eliminate the forced idle time for cgropuA both on current core and also the migrated core.I'm not sure if this is always possible.
During update_sg_lb_stats, we can scan for opportunities where pulling a task from a source cpu in the sched group to the target dest cpu can reduce the forced idle imbalance. And we also prevent task migrations that increase forced idle imbalance.
With those policies in place, we may not achieve perfect balance, but at least we will load balance in the right direction to lower forced idle imbalance.
Quoted 10 lines
Say on a 16cores/32threads machine, there are 3 cgroups, each has 16 cpu intensive tasks, will it be possible to make things perfectly balanced?
Don't get me wrong, I think this kind of load balancing is good and needed, but I'm not sure if we can always make things perfectly balanced. And if not, do we care those few cores where cgroup tasks are not balanced and then, do we need to implement the
core_widecgoup fairness functionality or we don't care since those cores are supposed to be few and isn't a big deal?
Yes, we still need core wide fairness for tasks. Load balancing is to move tasks around so we will have less imbalance of cgroup tasks in a core that results in forced idle time. Once they are in place, we still need to maintain fairness in a core.
Tim
Quoted 14 lines · Aubrey Li
Quoted 13 lines · Aaron Lu
Quoted 5 lines · Tim Chen
On 9/11/19 7:02 AM, Aaron Lu wrote:
I think Julien's result show that my patches did not do as well as your patches for fairness. Aubrey did some other testing with the same conclusion. So I think keeping the forced idle time balanced is not enough for maintaining fairness.Well, I have done following tests:
1 Julien's test script: https://paste.debian.net/plainh/834cf45c
2 start two tagged will-it-scale/page_fault1, see how each performs; 3 Aubrey's mysql test: https://github.com/aubreyli/coresched_bench.git
Quoted 7 lines
Quoted 3 lines
They all show your patchset performs equally well...And consider what the patch does, I think they are really doing the same thing in different ways.
It looks like we are not on the same page, if you don't mind, can both of you rebase your patchset onto v5.3-rc8 and provide a public branch so I can fetch and test it at least by my benchmark?
I'm using the following branch as base which is v5.1.5 based:
https://github.com/digitalocean/linux-coresched coresched-v3-v5.1.5-test
And I have pushed Tim's branch to:
https://github.com/aaronlu/linux coresched-v3-v5.1.5-test-tim
Mine:
https://github.com/aaronlu/linux coresched-v3-v5.1.5-test-core_vruntime
The two branches both have two patches I have sent previouslly:
https://lore.kernel.org/lkml/20190810141556.GA73644@aaronlu/
Although it has some potential performance loss as pointed out by Vineeth, I haven't got time to rework it yet.
Quoted 30 lines · Aaron Lu
Quoted 14 lines · Aubrey Li
Quoted 13 lines · Aaron Lu
Quoted 5 lines · Tim Chen
On 9/11/19 7:02 AM, Aaron Lu wrote:
I think Julien's result show that my patches did not do as well as your patches for fairness. Aubrey did some other testing with the same conclusion. So I think keeping the forced idle time balanced is not enough for maintaining fairness.Well, I have done following tests:
1 Julien's test script: https://paste.debian.net/plainh/834cf45c
2 start two tagged will-it-scale/page_fault1, see how each performs; 3 Aubrey's mysql test: https://github.com/aubreyli/coresched_bench.gitQuoted 7 lines
Quoted 3 lines
They all show your patchset performs equally well...And consider what the patch does, I think they are really doing the same thing in different ways.
It looks like we are not on the same page, if you don't mind, can both of you rebase your patchset onto v5.3-rc8 and provide a public branch so I can fetch and test it at least by my benchmark?
I'm using the following branch as base which is v5.1.5 based:
https://github.com/digitalocean/linux-coresched coresched-v3-v5.1.5-testAnd I have pushed Tim's branch to:
https://github.com/aaronlu/linux coresched-v3-v5.1.5-test-tim
Quoted 7 lines
Mine:
https://github.com/aaronlu/linux coresched-v3-v5.1.5-test-core_vruntimeThe two branches both have two patches I have sent previouslly:
https://lore.kernel.org/lkml/20190810141556.GA73644@aaronlu/
Although it has some potential performance loss as pointed out by Vineeth, I haven't got time to rework it yet.
In terms of these two branches, we tested two cases:
1) 32 AVX threads and 32 mysql threads on one core(2 HT)
2) 192 AVX threads and 192 mysql threads on 96 cores(192 HTs)
For case 1), we saw two branches is on par
Branch: coresched-v3-v5.1.5-test-core_vruntime
-Avg throughput:: 1865.62 (std: 20.6%)
-Avg latency: 26.43 (std: 8.3%)
Branch: coresched-v3-v5.1.5-test-tim
- Avg throughput: 1804.88 (std: 20.1%)
- Avg latency: 29.78 (std: 11.8%)
For case 2), we saw core vruntime performs better than counting forced idle time
Branch: coresched-v3-v5.1.5-test-core_vruntime
- Avg throughput: 5528.56 (std: 44.2%)
- Avg latency: 165.99 (std: 45.2%)
Branch: coresched-v3-v5.1.5-test-tim
-Avg throughput:: 3842.33 (std: 35.1%)
-Avg latency: 306.99 (std: 72.9%)
As Aaron pointed out, vruntime is with se's weight, which could be a reason for the difference.
So should we go with core vruntime approach?
Or Tim - do you want to improve forced idle time approach?
Thanks,
-Aubrey
Quoted 8 lines · Aubrey Li
Quoted 6 lines · Aaron Lu
And I have pushed Tim's branch to:
https://github.com/aaronlu/linux coresched-v3-v5.1.5-test-timMine:
https://github.com/aaronlu/linux coresched-v3-v5.1.5-test-core_vruntime
Aubrey,
Thanks for testing with your set up.
I think the test that's of interest is to see my load balancing added on top of Aaron's fairness patch, instead of using my previous version of forced idle approach in coresched-v3-v5.1.5-test-tim branch.
I've added my two load balance patches on top of Aaron's patches in coresched-v3-v5.1.5-test-core_vruntime branch and put it in
https://github.com/pdxChen/gang/tree/coresched-v3-v5.1.5-test-core_vruntime-lb
Quoted 7 lines
As Aaron pointed out, vruntime is with se's weight, which could be a reason for the difference.
So should we go with core vruntime approach?
Or Tim - do you want to improve forced idle time approach?
I hope to improve the forced idle time later. But for now let's see if additional load balance logic can help remove cgroup mismatch and improve performance, on top of Aaron's fairness patches.
Thanks.
Tim
Quoted 9 lines · Julien Desfossez
Quoted 7 lines · Peter Zijlstra
Quoted 2 lines · Phil Auld
I think, though, that you were basically agreeing with me that the current core scheduler does not close the holes, or am I reading that wrong.
Agreed; the missing bits for L1TF are ugly but doable (I've actually done them before, Tim has that _somewhere_), but I've not seen a 'workable' solution for MDS yet.
The L1TF problem is a much bigger one for HT than MDS. It is relatively easy for a Rogue VM to sniff L1 cached memory locations. While for MDS, it is quite difficult for the attacker to associate data in the cpu buffers with specific memory to make the sniffed data useful.
Even if we don't have a complete solution yet for MDS HT vulnerability, it is worthwhile to plug the L1TF hole for HT first with core scheduler, as L1TF is much more exploitable.
Tim
Quoted 20 lines
Following the discussion we had yesterday at LPC, after we have agreed on a solution for fixing the current fairness issue, we will post the v4. We will then work on prototyping the other synchronisation points (syscalls, interrupts and VMEXIT) to evaluate the overhead in various use-cases.
Depending on the use-case, we know the performance overhead maybe heavier than just disabling SMT, but the benchmarks we have seen so far indicate that there are valid cases for core scheduling. Core scheduling will continue to be unused by default, but with it, we will have the option to tune the system to be both secure and faster than disabling SMT for those cases.
Thanks,
Julien
P.S: I think the branch that contains the VMEXIT handling is here
https://github.com/pdxChen/gang/commits/sched_1.23-base
kernel/sched/core.c static inline bool cookie_match(struct task_struct *a, struct task_struct *b) · line 3684
Quoted 7 lines · Julien Desfossez
kernel/sched/core.c +7
-0+ +static void coresched_idle_worker_fini (struct rq *rq) +{ +if (rq->core_idle_task) { +kthread_stop (rq->core_idle_task); + rq->core_idle_task =NULL ; + }
During testing, I have found access of rq->core_idle_task as NULL pointer from other cpus (other than the cpu executing the stop_machine function) when you toggle cpu.tag of the cgroup.
Doing locking here is tricky because the rq lock is being transitioned from core lock to per run queue lock. As a fix,
I made coresched_idle_worker_fini a null function, and not
to null out the core_idle_task.
Tim
Quoted 12 lines · Tim Chen
Quoted 8 lines · Aubrey Li
Quoted 6 lines · Aaron Lu
And I have pushed Tim's branch to:
https://github.com/aaronlu/linux coresched-v3-v5.1.5-test-timMine:
https://github.com/aaronlu/linux coresched-v3-v5.1.5-test-core_vruntime
Quoted 8 lines
Aubrey,
Thanks for testing with your set up.
I think the test that's of interest is to see my load balancing added on top of Aaron's fairness patch, instead of using my previous version of forced idle approach in coresched-v3-v5.1.5-test-tim branch.
I'm trying to figure out a way to solve fairness only(not include task placement),
So @Vineeth - if everyone is okay with Aaron's fairness patch, maybe we should have a v4?
Thanks,
-Aubrey
Quoted 19 lines · Tim Chen
kernel/sched/fair.c +19
-0+static inline s64 core_sched_imbalance_delta (int src_cpu,int dst_cpu, +int src_sibling,int dst_sibling, +struct task_group *tg,u64 task_load) +{ +struct sched_entity *se, *se_sibling, *dst_se, *dst_se_sibling; +s64 excess, deficit, old_mismatch, new_mismatch; + +if (src_cpu == dst_cpu) +return -1 ; + +/* XXX SMT4 will require additional logic */ + + se = tg->se[src_cpu]; + se_sibling = tg->se[src_sibling]; + + excess = se->avg.load_avg - se_sibling->avg.load_avg; +if (src_sibling == dst_cpu) { + old_mismatch =abs (excess); + new_mismatch =abs (excess -2 *task_load);
kernel/sched/fair.c static bool update_nohz_stats(struct rq *rq, bool force) · line 8366
Quoted 10 lines
kernel/sched/fair.c +10
-0+return old_mismatch - new_mismatch; + } + + dst_se = tg->se[dst_cpu]; + dst_se_sibling = tg->se[dst_sibling]; + deficit = dst_se->avg.load_avg - dst_se_sibling->avg.load_avg; + + old_mismatch =abs (excess) +abs (deficit); + new_mismatch =abs (excess - (s64 ) task_load) + +abs (deficit + (s64 ) task_load);
If I understood correctly, these formulas made an assumption that the task being moved to the destination is matched the destination's core cookie. so if the task is not matched with dst's core cookie and still have to stay in the runqueue
then the formula becomes not correct.
kernel/sched/fair.c static inline void update_sg_lb_stats(struct lb_env *env, · line 8348
Quoted 8 lines
/**
*update_sg_lb_stats- Update sched_group's statistics for load balancing.
* @env: The load balancing environment.kernel/sched/fair.c +0
-1@@ -8345,7 +8492,8 @@ static inline void update_sg_lb_stats(struct lb_env *env,else load =source_load (i, load_idx); - sgs->group_load += load;
Why is this load update line removed?
kernel/sched/fair.c static inline void update_sg_lb_stats(struct lb_env *env, · line 8499
Quoted 5 lines
kernel/sched/fair.c +2
-0+core_sched_imbalance_scan (sgs, i, env->dst_cpu); + sgs->group_util +=cpu_util (i); sgs->sum_nr_running += rq->cfs.h_nr_running;
Thanks,
-Aubrey
Quoted 25 lines · Aubrey Li
Quoted 24 lines · Tim Chen
kernel/sched/fair.c +24
-0+static inline s64 core_sched_imbalance_delta (int src_cpu,int dst_cpu, +int src_sibling,int dst_sibling, +struct task_group *tg,u64 task_load) +{ +struct sched_entity *se, *se_sibling, *dst_se, *dst_se_sibling; +s64 excess, deficit, old_mismatch, new_mismatch; + +if (src_cpu == dst_cpu) +return -1 ; + +/* XXX SMT4 will require additional logic */ + + se = tg->se[src_cpu]; + se_sibling = tg->se[src_sibling]; + + excess = se->avg.load_avg - se_sibling->avg.load_avg; +if (src_sibling == dst_cpu) { + old_mismatch =abs (excess); + new_mismatch =abs (excess -2 *task_load); +return old_mismatch - new_mismatch; + } + + dst_se = tg->se[dst_cpu]; + dst_se_sibling = tg->se[dst_sibling];
Quoted 8 lines
kernel/sched/fair.c static bool update_nohz_stats(struct rq *rq, bool force) · line 8366
Quoted 5 lines
kernel/sched/fair.c +5
-0+ deficit = dst_se->avg.load_avg - dst_se_sibling->avg.load_avg; + + old_mismatch =abs (excess) +abs (deficit); + new_mismatch =abs (excess - (s64 ) task_load) + +abs (deficit + (s64 ) task_load);If I understood correctly, these formulas made an assumption that the task being moved to the destination is matched the destination's core cookie.
That's not the case. We do not need to match the destination's core cookie, as that may change after context switches. It needs to reduce the load mismatch with the destination CPU's sibling for that cgroup.
Quoted 12 lines
so if
the task is not matched with dst's core cookie and still have to stay in the runqueue
then the formula becomes not correct.Quoted 7 lines
/**
*update_sg_lb_stats- Update sched_group's statistics for load balancing.
* @env: The load balancing environment.kernel/sched/fair.c +0
-0@@ -8345,7 +8492,8 @@ static inline void update_sg_lb_stats(struct lb_env *env,else load =source_load (i, load_idx);
Quoted 3 lines
kernel/sched/fair.c static inline void update_sg_lb_stats(struct lb_env *env, · line 8348
Quoted 1 line
kernel/sched/fair.c +0
-1- sgs->group_load += load;Why is this load update line removed?
This was removed accidentally. Should be restored.
Quoted 7 lines
kernel/sched/fair.c static inline void update_sg_lb_stats(struct lb_env *env, · line 8499
Quoted 6 lines
kernel/sched/fair.c +2
-0+core_sched_imbalance_scan (sgs, i, env->dst_cpu); + sgs->group_util +=cpu_util (i); sgs->sum_nr_running += rq->cfs.h_nr_running;
Thanks.
Tim
Quoted 36 lines · Tim Chen
Quoted 25 lines · Aubrey Li
Quoted 24 lines · Tim Chen
kernel/sched/fair.c +24
-0+static inline s64 core_sched_imbalance_delta (int src_cpu,int dst_cpu, +int src_sibling,int dst_sibling, +struct task_group *tg,u64 task_load) +{ +struct sched_entity *se, *se_sibling, *dst_se, *dst_se_sibling; +s64 excess, deficit, old_mismatch, new_mismatch; + +if (src_cpu == dst_cpu) +return -1 ; + +/* XXX SMT4 will require additional logic */ + + se = tg->se[src_cpu]; + se_sibling = tg->se[src_sibling]; + + excess = se->avg.load_avg - se_sibling->avg.load_avg; +if (src_sibling == dst_cpu) { + old_mismatch =abs (excess); + new_mismatch =abs (excess -2 *task_load); +return old_mismatch - new_mismatch; + } + + dst_se = tg->se[dst_cpu]; + dst_se_sibling = tg->se[dst_sibling];Quoted 8 lines
kernel/sched/fair.c static bool update_nohz_stats(struct rq *rq, bool force) · line 8366
Quoted 5 lines
kernel/sched/fair.c +5
-0+ deficit = dst_se->avg.load_avg - dst_se_sibling->avg.load_avg; + + old_mismatch =abs (excess) +abs (deficit); + new_mismatch =abs (excess - (s64 ) task_load) + +abs (deficit + (s64 ) task_load);If I understood correctly, these formulas made an assumption that the task being moved to the destination is matched the destination's core cookie.
Quoted 1 line
That's not the case. We do not need to match the destination's core cookie,
I actually meant destination core's core cookie.
Quoted 2 lines
as that may change after context switches. It needs to reduce the load mismatch with the destination CPU's sibling for that cgroup.
So the new_mismatch is not always true, especially when there are more cgroups and
more core cookies on the system.
Thanks,
-Aubrey
Quoted 9 lines · Aaron Lu
Quoted 8 lines
I think comparing parent's runtime also will have issues once the task group has a lot more threads with different running patterns. One example is a task group with lot of active threads and a thread with fairly less activity. So when this less active thread is competing with a thread in another group, there is a chance that it loses continuously for a while until the other group catches up on its vruntime.
Quoted 8 lines
I actually think this is expected behaviour.
Without core scheduling, when deciding which task to run, we will first decide which "se" to run from the CPU's root level cfs runqueue and then go downwards. Let's call the chosen se on the root level cfs runqueue the winner se. Then with core scheduling, we will also need compare the two winner "se"s of each hyperthread and choose the core wide winner "se".
Sorry, I misunderstood the fix and I did not initially see the core wide min_vruntime that you tried to maintain in the rq->core. This approach seems reasonable. I think we can fix the potential starvation that you mentioned in the comment by adjusting for the difference in all the children cfs_rq when we set the minvruntime in rq->core. Since we take the lock for both the queues, it should be doable and I am trying to see how we can best do that.
Quoted 5 lines
Quoted 4 lines
As discussed during LPC, probably start thinking along the lines of global vruntime or core wide vruntime to fix the vruntime comparison issue?
Quoted 10 lines
core wide vruntime makes sense when there are multiple tasks of different cgroups queued on the same core. e.g. when there are two tasks of cgroupA and one task of cgroupB are queued on the same core, assume cgroupA's one task is on one hyperthread and its other task is on the other hyperthread with cgroupB's task. With my current implementation or Tim's, cgroupA will get more time than cgroupB. If we maintain core wide vruntime for cgroupA and cgroupB, we should be able to maintain fairness between cgroups on this core. Tim propose to solve this problem by doing some kind of load balancing if I'm not mistaken, I haven't taken a look at this yet.
I think your fix is almost close to maintaining a core wide vruntime as you have a single minvruntime to compare now across the siblings in the core. To make the fix complete, we might need to adjust the whole tree's min_vruntime and I think its doable.
Thanks,
Vineeth
Quoted 23 lines · Aubrey Li
Quoted 12 lines · Tim Chen
Quoted 8 lines · Aubrey Li
Quoted 6 lines · Aaron Lu
And I have pushed Tim's branch to:
https://github.com/aaronlu/linux coresched-v3-v5.1.5-test-timMine:
https://github.com/aaronlu/linux coresched-v3-v5.1.5-test-core_vruntimeQuoted 8 lines
Aubrey,
Thanks for testing with your set up.
I think the test that's of interest is to see my load balancing added on top of Aaron's fairness patch, instead of using my previous version of forced idle approach in coresched-v3-v5.1.5-test-tim branch.
Quoted 5 lines
I'm trying to figure out a way to solve fairness only(not include task placement),
So @Vineeth - if everyone is okay with Aaron's fairness patch, maybe we should have a v4?
Yes, I think we can move to v4 with Aaron's fairness fix and potentially Tim's load balancing fixes. I am working on some improvements to Aaron's fixes and shall post the changes after some testing. Basically, what I am trying to do is to propagate the min_vruntime change down to all the cf_rq and individual se when we update the cfs_rq(rq->core)->min_vrutime. So, we can make sure that the rq stays in sync and starvation do not happen.
If everything goes well, we shall also post the v4 towards the end of this week. We would be testing Tim's load balancing patches in an over-committed VM scenario to observe the effect of the fix.
Thanks,
Vineeth
Quoted 10 lines
I've made an attempt in the following two patches to address the load balancing of mismatched load between the siblings.
It is applied on top of Aaron's patches:
- sched: Fix incorrect rq tagged as forced idle
- wrapper forcfs_rq->min_vruntime
https://lore.kernel.org/lkml/20190725143127.GB992@aaronlu/
- core vruntime comparison
https://lore.kernel.org/lkml/20190725143248.GC992@aaronlu/
Quoted 2 lines
I will love Julien, Aaron and others to try it out. Suggestions to tune it is welcomed.
Just letting you know that I will be testing your load balancing patches this week along with the changes Vineeth is currently doing. I didn't test it before because I was focused on single threaded and pinned micro-benchmarks, but I am back on scaling tests so it will be interesting to see.
Thanks,
Julien
Quoted 10 lines · Vineeth Remanan Pillai
Quoted 2 lines
Sorry, I misunderstood the fix and I did not initially see the core wide
min_vruntimethat you tried to maintain in therq->core. This approach seems reasonable. I think we can fix the potential starvation that you mentioned in the comment by adjusting for the difference in all the childrencfs_rqwhen we set the minvruntime inrq->core. Since we take the lock for both the queues, it should be doable and I am trying to see how we can best do that.
Attaching here with, the 2 patches I was working on in preparation of v4.
Patch 1 is an improvement of patch 2 of Aaron where I am propagating the vruntime changes to the whole tree.
Patch 2 is an improvement for patch 3 of Aaron where we do resched_curr only when the sibling is forced idle.
Micro benchmarks seems good. Will be doing larger set of tests and hopefully posting v4 by end of week. Please let me know what you think of these patches (patch 1 is on top of Aaron's patch 2, patch 2 replaces Aaron's patch 3)
Thanks,
Vineeth
[PATCH 1/2] sched/fair: propagate the min_vruntime change to the whole rq tree
When we adjust the min_vruntime of rq->core, we need to propgate that down the tree so as to not cause starvation of existing tasks based on previous vruntime.
--- kernel/sched/fair.c | 24 ++++++++++++++++++++++-- 1 file changed, 22 insertions(+), 2 deletions(-)
kernel/sched/fair.c +0 -0
@@ -476,6 +476,23 @@ static inline u64 cfs_rq_min_vruntime(struct
cfs_rq *cfs_rq)
return cfs_rq->min_vruntime;
}
+static void coresched_adjust_vruntime(struct cfs_rq *cfs_rq, u64 delta) +{
+ struct sched_entity *se, *next;
+
+ if (!cfs_rq)
+ return;
+
+ cfs_rq->min_vruntime -= delta;
+ rbtree_postorder_for_each_entry_safe(se, next,
+ &cfs_rq->tasks_timeline.rb_root, run_node) {
+ if (se->vruntime > delta)
+ se->vruntime -= delta;
+ if (se->my_q)
+ coresched_adjust_vruntime(se->my_q, delta);
+ }
+}
+
static void update_core_cfs_rq_min_vruntime(struct cfs_rq *cfs_rq)
{
struct cfs_rq *cfs_rq_core;
@@ -487,8 +504,11 @@ static void
update_core_cfs_rq_min_vruntime(struct cfs_rq *cfs_rq)
return;
cfs_rq_core = core_cfs_rq(cfs_rq);- cfs_rq_core->min_vruntime = max(cfs_rq_core->min_vruntime,
- cfs_rq->min_vruntime);
+ if (cfs_rq_core != cfs_rq &&
+ cfs_rq->min_vruntime < cfs_rq_core->min_vruntime) {
+ u64 delta = cfs_rq_core->min_vruntime - cfs_rq->min_vruntime;
+ coresched_adjust_vruntime(cfs_rq_core, delta);
+ }
}
bool cfs_prio_less(struct task_struct *a, struct task_struct *b)
--
2.17.1
[PATCH 2/2] sched/fair : Wake up forced idle siblings if needed
If a cpu has only one task and if it has used up its timeslice, then we should try to wake up the sibling to give the forced idle thread a chance.
We do that by triggering schedule which will IPI the sibling if the task in the sibling wins the priority check.
--- kernel/sched/fair.c | 43 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 43 insertions(+)
kernel/sched/fair.c +0 -0
@@ -4165,6 +4165,13 @@ dequeue_entity(struct cfs_rq *cfs_rq, struct
sched_entity *se, int flags)
update_min_vruntime(cfs_rq);
}
+static inline bool
+__entity_slice_used(struct sched_entity *se)
+{
+ return (se->sum_exec_runtime - se->prev_sum_exec_runtime) >
+ sched_slice(cfs_rq_of(se), se);
+}
+
/*
* Preempt the current task with a newly woken task if needed:
*/
@@ -10052,6 +10059,39 @@ static void rq_offline_fair(struct rq *rq)
#endif /* CONFIG_SMP */
+#ifdef CONFIG_SCHED_CORE
+/*
+ * If runqueue has only one task which used up its slice and
+ * if the sibling is forced idle, then trigger schedule
+ * to give forced idle task a chance.
+ */
+static void resched_forceidle(struct rq *rq, struct sched_entity *se) +{
+ int cpu = cpu_of(rq), sibling_cpu;
+ if (rq->cfs.nr_running > 1 || !__entity_slice_used(se))
+ return;
+
+ for_each_cpu(sibling_cpu, cpu_smt_mask(cpu)) {
+ struct rq *sibling_rq;
+ if (sibling_cpu == cpu)
+ continue;
+ if (cpu_is_offline(sibling_cpu))
+ continue;
+
+ sibling_rq = cpu_rq(sibling_cpu);
+ if (sibling_rq->core_forceidle) {
+ resched_curr(rq);
+ break;
+ }
+ }
+}
+#else
+static inline void resched_forceidle(struct rq *rq, struct sched_entity *se) +{
+}
+#endif
+
+
/*
* scheduler tick hitting a task of our scheduling class.
*
@@ -10075,6 +10115,9 @@ static void task_tick_fair(struct rq *rq, struct task_struct *curr, int queued)
update_misfit_status(curr, rq);
update_overutilized_status(task_rq(curr));+
+ if (sched_core_enabled(rq))
+ resched_forceidle(rq, &curr->se);
}
/*
--
2.17.1
Quoted 12 lines · Vineeth Remanan Pillai
Quoted 10 lines · Vineeth Remanan Pillai
Quoted 2 lines
Sorry, I misunderstood the fix and I did not initially see the core wide
min_vruntimethat you tried to maintain in therq->core. This approach seems reasonable. I think we can fix the potential starvation that you mentioned in the comment by adjusting for the difference in all the childrencfs_rqwhen we set the minvruntime inrq->core. Since we take the lock for both the queues, it should be doable and I am trying to see how we can best do that.
Quoted 4 lines
Attaching here with, the 2 patches I was working on in preparation of v4.
Patch 1 is an improvement of patch 2 of Aaron where I am propagating the vruntime changes to the whole tree.
I didn't see why we need do this.
We only need to have the root level sched entities' vruntime become core wide since we will compare vruntime for them across hyperthreads. For sched entities on sub cfs_rqs, we never(at least, not now) compare their vruntime outside their cfs_rqs.
Thanks,
Aaron
Quoted 2 lines
Patch 2 is an improvement for patch 3 of Aaron where we do
resched_curronly when the sibling is forced idle.
Quoted 7 lines
I didn't see why we need do this.
We only need to have the root level sched entities' vruntime become core wide since we will compare vruntime for them across hyperthreads. For sched entities on sub
cfs_rqs, wenever(at least, not now)compare their vruntime outside theircfs_rqs.
The reason we need to do this is because, new tasks that gets created will have a vruntime based on the new min_vruntime and old tasks will have it based on the old min_vruntime and it can cause starvation based on how you set the min_vruntime. With this new patch, we normalize the whole tree so that new tasks and old tasks compare with the same min_vruntime.
Thanks,
Vineeth
Quoted 10 lines · Vineeth Remanan Pillai
Quoted 7 lines
I didn't see why we need do this.
We only need to have the root level sched entities' vruntime become core wide since we will compare vruntime for them across hyperthreads. For sched entities on sub
cfs_rqs, wenever(at least, not now)compare their vruntime outside theircfs_rqs.The reason we need to do this is because, new tasks that gets created will have a vruntime based on the new
min_vruntimeand old tasks will have it based on the oldmin_vruntime
I think this is expected behaviour.
Quoted 2 lines
and it can cause starvation based on how
you set themin_vruntime.
Care to elaborate the starvation problem?
Quoted 2 lines
With this new patch, we normalize the whole
tree so that new tasks and old tasks compare with the samemin_vruntime.
Again, what's the point of normalizing sched entities' vruntime in sub-cfs_rqs? Their vruntime comparisons only happen inside their own cfs_rq, we don't do cross CPU vruntime comparison for them.
Quoted 6 lines
Quoted 3 lines
The reason we need to do this is because, new tasks that gets created will have a vruntime based on the new
min_vruntimeand old tasks will have it based on the oldmin_vruntimeI think this is expected behaviour.
I don't think this is the expected behavior. If we hadn't changed the root cfs->min_vruntime for the core rq, then it would have been the expected behaviour. But now, we are updating the core rq's root cfs, min_vruntime without changing the the vruntime down to the tree. To explain, consider this example based on your patch. Let cpu 1 and 2 be siblings. And let rq(cpu1) be the core rq. Let rq1->cfs->min_vruntime=1000 and rq2->cfs->min_vruntime=2000. So in update_core_cfs_rq_min_vruntime, you update rq1->cfs->min_vruntime to 2000 because that is the max. So new tasks enqueued on rq1 starts with vruntime of 2000 while the tasks in that runqueue are still based on the old min_vruntime(1000). So the new tasks gets enqueued some where to the right of the tree and has to wait until already existing tasks catch up the vruntime to
2000. This is what I meant by starvation. This happens always when we update the core rq's cfs->min_vruntime. Hope this clarifies.
Quoted 4 lines
Quoted 2 lines
and it can cause starvation based on how
you set themin_vruntime.Care to elaborate the starvation problem?
Explained above.
Quoted 3 lines
Again, what's the point of normalizing sched entities' vruntime in sub-cfs_rqs? Their vruntime comparisons only happen inside their own
cfs_rq, we don't do cross CPU vruntime comparison for them.
As I mentioned above, this is to avoid the starvation case. Even though we are not doing cross cfs_rq comparison, the whole tree's vruntime is based on the root cfs->min_vruntime and we will have an imbalance if we change the root cfs->min_vruntime without updating down the tree.
Thanks,
Vineeth
Quoted 6 lines · Vineeth Remanan Pillai
Quoted 6 lines
Quoted 3 lines
The reason we need to do this is because, new tasks that gets created will have a vruntime based on the new
min_vruntimeand old tasks will have it based on the oldmin_vruntimeI think this is expected behaviour.
Quoted 14 lines
I don't think this is the expected behavior. If we hadn't changed the root
cfs->min_vruntimefor the core rq, then it would have been the expected behaviour. But now, we are updating the core rq's root cfs,min_vruntimewithout changing the the vruntime down to the tree. To explain, consider this example based on your patch. Let cpu 1 and 2 be siblings. And let rq(cpu1) be the core rq. Let rq1->cfs->min_vruntime=1000 and rq2->cfs->min_vruntime=2000. So inupdate_core_cfs_rq_min_vruntime, you updaterq1->cfs->min_vruntimeto 2000 because that is the max. So new tasks enqueued on rq1 starts with vruntime of 2000 while the tasks in that runqueue are still based on the oldmin_vruntime(1000). So the new tasks gets enqueued some where to the right of the tree and has to wait until already existing tasks catch up the vruntime to
2000. This is what I meant by starvation. This happens always when we update the core rq'scfs->min_vruntime. Hope this clarifies.
Thanks for the clarification.
Yes, this is the initialization issue I mentioned before when core scheduling is initially enabled. rq1's vruntime is bumped the first time update_core_cfs_rq_min_vruntime() is called and if there are already some tasks queued, new tasks queued on rq1 will be starved to some extent.
Agree that this needs fix. But we shouldn't need do this afterwards.
So do I understand correctly that patch1 is meant to solve the initialization issue?
Quoted 11 lines
Thanks for the clarification.
Yes, this is the initialization issue I mentioned before when core scheduling is initially enabled. rq1's vruntime is bumped the first time
update_core_cfs_rq_min_vruntime()is called and if there are already some tasks queued, new tasks queued on rq1 will be starved to some extent.Agree that this needs fix. But we shouldn't need do this afterwards.
So do I understand correctly that patch1 is meant to solve the initialization issue?
I think we need this update logic even after initialization. I mean, core runqueue's min_vruntime can get updated every time when the core runqueue's min_vruntime changes with respect to the sibling's min_vruntime. So, whenever this update happens, we would need to propagate the changes down the tree right? Please let me know if I am visualizing it wrong.
Thanks,
Vineeth
Quoted 9 lines · Vineeth Remanan Pillai
Quoted 9 lines
Thanks for the clarification.
Yes, this is the initialization issue I mentioned before when core scheduling is initially enabled. rq1's vruntime is bumped the first time
update_core_cfs_rq_min_vruntime()is called and if there are already some tasks queued, new tasks queued on rq1 will be starved to some extent.Agree that this needs fix. But we shouldn't need do this afterwards.
Quoted 8 lines
Quoted 2 lines
So do I understand correctly that patch1 is meant to solve the initialization issue?
I think we need this update logic even after initialization. I mean, core runqueue's
min_vruntimecan get updated every time when the core runqueue'smin_vruntimechanges with respect to the sibling'smin_vruntime. So, whenever this update happens, we would need to propagate the changes down the tree right? Please let me know if I am visualizing it wrong.
I don't think we need do the normalization afterwrads and it appears we are on the same page regarding core wide vruntime.
The intent of my patch is to treat all the root level sched entities of the two siblings as if they are in a single cfs_rq of the core. With a core wide min_vruntime, the core scheduler can decide which sched entity to run next. And the individual sched entity's vruntime shouldn't be changed based on the change of core wide min_vruntime, or faireness can hurt(if we add or reduce vruntime of a sched entity, its credit will change).
The weird thing about my patch is, the min_vruntime is often increased, it doesn't point to the smallest value as in a traditional cfs_rq. This probabaly can be changed to follow the tradition, I don't quite remember why I did this, will need to check this some time later.
All those sub cfs_rq's sched entities are not interesting. Because once we decided which sched entity in the root level cfs_rq should run next, we can then pick the final next task from there(using the usual way). In other words, to make scheduler choose the correct candidate for the core, we only need worry about sched entities on both CPU's root level cfs_rqs.
Does this make sense?
Quoted 4 lines · Aaron Lu
I don't think we need do the normalization afterwrads and it appears we are on the same page regarding core wide vruntime.
Quoted 8 lines
The intent of my patch is to treat all the root level sched entities of the two siblings as if they are in a single
cfs_rqof the core. With a core widemin_vruntime, the core scheduler can decide which sched entity to run next. And the individual sched entity's vruntime shouldn't be changed based on the change of core widemin_vruntime, or faireness canhurt(if we add or reduce vruntime of a sched entity, its credit will change).
Ok, I think I get it now. I see that your first patch actually wraps all the places
where min_vruntime is accessed. So yes, the tree vruntime updation is needed only one time. From then on, since we use the wrapper cfs_rq_min_vruntime(), both the runqueues would self adjust from then on based on the code wide min_vruntime. Also by the virtue that min_vruntime stays min from there on, the tree updation logic will not be called more than once. So I think the changes are safe.
I will do some profiling to make sure that it is actually called once only.
Quoted 4 lines
The weird thing about my patch is, the
min_vruntimeis often increased, it doesn't point to the smallest value as in a traditionalcfs_rq. This probabaly can be changed to follow the tradition, I don't quite remember why I did this, will need to check this some time later.
Yeah, I noticed this. In my patch, I had already accounted for this and changed to min() instead of max() which is more logical that min_vruntime should be the minimum of both the run queue.
Quoted 6 lines
All those sub cfs_rq's sched entities are not interesting. Because once we decided which sched entity in the root level
cfs_rqshould run next, we can then pick the final next task from there(using the usual way). In other words, to make scheduler choose the correct candidate for the core, we only need worry about sched entities on both CPU's root levelcfs_rqs.
Understood. The only reason I did the normalize is to get both the runqueues under one min_vruntime always. And as long as we use the cfs_rq_min_vruntime from then on, we wouldn't be calling the balancing logic any more.
Quoted 1 line
Does this make sense?
Sure, thanks for the clarification.
Thanks,
Vineeth
Quoted 5 lines · Vineeth Remanan Pillai
Quoted 3 lines · Aaron Lu
I don't think we need do the normalization afterwrads and it appears we are on the same page regarding core wide vruntime.
Should be "we are not on the same page..."
[...]
Quoted 8 lines
Quoted 4 lines
The weird thing about my patch is, the
min_vruntimeis often increased, it doesn't point to the smallest value as in a traditionalcfs_rq. This probabaly can be changed to follow the tradition, I don't quite remember why I did this, will need to check this some time later.Yeah, I noticed this. In my patch, I had already accounted for this and changed to
min()instead ofmax()which is more logical thatmin_vruntimeshould be the minimum of both the run queue.
I now remembered why I used max().
Assume rq1 and rq2's min_vruntime are both at 2000 and the core wide min_vruntime is also 2000. Also assume both runqueues are empty at the moment. Then task t1 is queued to rq1 and runs for a long time while rq2 keeps empty. rq1's min_vruntime will be incremented all the time while the core wide min_vruntime stays at 2000 if min() is used. Then when another task gets queued to rq2, it will get really large unfair boost by using a much smaller min_vruntime as its base.
To fix this, either max() is used as is done in my patch, or adjust rq2's min_vruntime to be the same as rq1's on each update_core_cfs_min_vruntime() when rq2 is found empty and then use min() to get the core wide min_vruntime. Looks not worth the trouble to use min().
Quoted 11 lines · Aaron Lu
I now remembered why I used
max().Assume rq1 and rq2's
min_vruntimeare both at 2000 and the core widemin_vruntimeis also 2000. Also assume both runqueues are empty at the moment. Then task t1 is queued to rq1 and runs for a long time while rq2 keeps empty. rq1'smin_vruntimewill be incremented all the time while the core widemin_vruntimestays at 2000 ifmin()is used. Then when another task gets queued to rq2, it will get really large unfair boost by using a much smallermin_vruntimeas its base.
Quoted 5 lines
To fix this, either
max()is used as is done in my patch, or adjust rq2'smin_vruntimeto be the same as rq1's on eachupdate_core_cfs_min_vruntime()when rq2 is found empty and then usemin()to get the core widemin_vruntime. Looks not worth the trouble to usemin().
Understood. I think this case is a special case where one runqueue is empty and hence the min_vruntime of the core should match the progressing vruntime of the active runqueue. If we use max as the core wide min_vruntime, I think we may hit starvation elsewhere. On quick example I can think of is during force idle. When a sibling is forced idle, and if a new task gets enqueued in the force idled runq, it would inherit the max vruntime and would starve until the other tasks in the forced idle sibling catches up. While this might be okay, we are deviating from the concept that all new tasks inherits the min_vruntime of the cpu(core in our case). I have not tested deeply to see if there are any assumptions which may fail if we use max.
The modified patch actually takes care of syncing the min_vruntime across the siblings so that, core wide min_vruntime and per cpu min_vruntime always stays in sync.
Thanks,
Vineeth
Quoted 11 lines · Aaron Lu
I'm using the following branch as base which is v5.1.5 based:
https://github.com/digitalocean/linux-coresched coresched-v3-v5.1.5- testAnd I have pushed Tim's branch to:
https://github.com/aaronlu/linux coresched-v3-v5.1.5-test-timMine:
https://github.com/aaronlu/linux coresched-v3-v5.1.5-test-core_vruntime
Hello,
As anticipated, I've been trying to follow the development of this feature and, in the meantime, I have done some benchmarks.
I actually have a lot of data (and am planning for more), so I am sending a few emails, each one with a subset of the numbers in it, instead than just one which would be beyond giant! :-)
I'll put, in this first one, some background and some common information, e.g., about the benchmarking platform and configurations, and on how to read and interpreet the data that will follow.
It's quite hard to come up with a concise summary, and sometimes it's even tricky to identify consolidated trends. There are also things that looks weird and, although I double checked my methodology, I can't exclude that of glitches or errors may have occurred. For each of the benchmark, I have at least some information about what the configuration was when it was run, and also some monitorning and perf data. So, if interested, try to ask and we'll see what we can dig out.
And in any case, I have the procedure for running these benchmarks fairly decently (although not completely) automated. So if we see things that looks really really weird, I can rerun (perhaps with different configuration, more monitoring, etc).
For each benchmark, I'll "dump" the results, with just some comments about the things that I find more relevant/interesting. Then, if we want, we can look at them and analyze them together.
For each experiment, I do have some limited amount of tracing and debugging information still available, in case it could be useful. And, as said, I can always rerun.
I can also provide, quite easily, different looking tables. E.g., different set of columns, different baselines, etc. Just as what you thinks it would be the most interesting to see, and, most likely, it will be possible to do it.
Oh, and I'll upload text files whose contents will be identical to the emails in this space:
http://xenbits.xen.org/people/dariof/benchmarks/results/linux/core-sched/mmtests/boxes/wayrath/
In case tables are rendered better in a browser than in a MUA.
Thanks and Regards,
Dario
---
Code: 1) Linux 5.1.5 (commit id 835365932f0dc25468840753e071c05ad6abc76f) 2) https://github.com/digitalocean/linux-coresched/tree/vpillai/coresched-v3-v5.1.5-test 3) https://github.com/aaronlu/linux/tree/coresched-v3-v5.1.5-test-core_vruntime 4) https://github.com/aaronlu/linux/tree/coresched-v3-v5.1.5-test-tim
Benchmarking suite:
- MMTests: https://github.com/gormanm/mmtests
- Tweaked to deal with running benchmarks in VMs. Still working on upstreaming that to Mel (a WIP is available here:
https://github.com/dfaggioli/mmtests/tree/bench-virt )
Benchmarking host:
- CPU: 1 socket, 4 cores, 2 threads
- RAM: 32 GB
- distro: opneSUSE Tumbleweed
- HW Bugs Mitigations: fully disabled
- Filesys: XFS
VMs:
- vCPUs: either 8 or 4
- distro: opneSUSE Tumbleweed
- Kernel: 5.1.16
- HW Bugs Mitigations: fully disabled
- Filesys: XFS
Benchmarks:
- STREAM : pure memory benchmark (various kind of mem-ops done
in parallel). Parallelism is NR_CPUS/2 tasks
- Kernbench : builds a kernel, with varying number of compile
jobs. HT is, in general, known to help, as it let us do "more parallel" builds
- Hackbench : communication (via pipes, in this case) between
group of processes. As we deal with _groups_ of tasks, we're already in saturation with 1 group, hence we expect HyperThreading disabled configurations to suffer
- mutilate : load generator for memcached, with high request
rate;
- netperf-unix : two communicating tasks. Without any pinning
(neither at the host nor at the guest level), we expect HT to play a role. In fact, depending on where the two task are scheduler (i.e., whether on two core of the same thread, or not) performance may vary
- sysbenchcpu : the process-based CPU stressing workload of sysbench
- sysbenchthread : the thread-based CPU stressing workload of sysbench
- sysbench : the database workload
This is kind of a legend for the columns you will see in the tables.
- v-* : vanilla, i.e., benchmarks were run on code _without_ any
core-scheduling patch applied (see 1 in 'Code' section above)
- *BM-* : baremetal, i.e., benchmarks were run on the host, without any VM running or anything
- *VM-* : Virtual Machine, i.e., benchmarks were run inside a VM, with the following haracteristics:
- *VM- : benchmarks were run in a VM with 8 vCPUs. That was the
only VM running in the system
- *VM-v4 : benchmarks were run in a VM with 4 vCPUs. That was the only VM running in the system
- *VMx2 : benchmark were run in a VM with 8 vCPUs, and there was another VM running, also with 8 vCPUS, generating CPU, memory and IO stress load for about 600%
- *-csc-* : benchmarks were run with Core scheduling v3 patch
series (see 2 in 'Code' section above)
- *-csc_stallfix-* : benchmarks were run with Core scheduling v3 and the 'stallfix' feature enabled
- *-csc_vruntime-* : benchmarks were run with Core scheduling v3 + the vruntime patches (see 3 in 'Code' section above)
- *-csc-_tim-* : benchmarks were run with Core scheduling v3 +
Tim's patches (see 4 in 'Code' section above)
- *-noHT : benchmarks were run with HyperThreading Disabled
- *-HT : benchmarks were run with Hyperthreading enabled
So, for instance, the column BM-noHT shows data from a run done on baremetal, with HyperThreading disabled. The column v-VM-HT shows data from a run done in a 8 vCPUs VM, with HyperThreading enabled, and no core-scheduling patches applied. The column VM-csc_vruntime-HT shows data from a run done in a 8 vCPUs VM with core-scheduling v3 patches + the vruntime patches applied. The column VM-v4-HT shows data from a run done in a 4 vCPUs VM, core-scheduling patches were applied but not used (the vCPUs of the VM weren't tagged). The column VMx2-csc_vruntime-HT shows data from a run done in a 8 vCPUs VM, core-scheduling v3 + Tim's patchs were applied and the vCPUs of the VM tagged, while there was another (untagged) VM in the system, trying to introduce ~600% load (CPU, memory and IO, via stress-ng). Etc.
See the 'Appendix' at the bottom of this email, for a comprehensive list of all the combinations (or, at least I think is comprehensive... I hope I haven't missed any :-) ).
In all tables, percent increase and decrease are always relative to the first column. It is already taken care of whether lower or higher values are better.
Basically, when we see -x.yz%, it always means performance are worse than the baseline, and the absolute value of that (i.e., x.yz) tells you by how much.
If, for instance, we want to compare HT and non HT, on baremetal, we check the BM-HT and BM-noHT columns.
If we want to compare v3 + vruntime patches against no HyperThreading, when the system is overloaded, we look at VMx2-noHT and VMx2- csc_vruntime-HT columns and check by how much they deviates from the baseline (i.e., which one regresses more). For comparing, the various core scheduling solutions, we can check by how much each one is either better or worse than baseline. And so on...
The most relevant comparisons, IMO, are:
- the various core scheduling solutions against their respective HT baseline. This, in fact, tells us what people will experience if they start using core scheduling on these workloads
- the various core scheduling solutions against their respective noHT baseline. This, in fact, tells use whether or not core scheduling is effective, for the given workload, or if it would just be better to disable HyperThreading
- the overhead introduced by the core scheduling patches, when they are not used (i.e., v-BM-HT against BM-HT, or v-VM-HT against VM-HT). This, in fact, tells us what happens to *everyone*, including the ones that do not want core scheduling and will keep it disabled, if we merge it
Note that the overhead, so far, has been evaluated only for the -csc case, i.e., when patches from point 2 in 'Code' above are applied, but tasks/vCPUs are not tagged, and hence core scheduling is not really used,
Anyway, let's get to the point where I give you some data already! :-D :-D
STREAM
======
v v BM BM BM BM BM BM
BM-HT BM-noHT HT noHT csc-HT csc_stallfix-HT csc_tim-HT csc_vruntime-HT
MB/sec copy 33827.50 ( 0.00%) 33654.32 ( -0.51%) 33683.34 ( -0.43%) 33819.30 ( -0.02%) 33830.88 ( 0.01%) 33731.02 ( -0.29%) 33573.76 ( -0.75%) 33292.76 ( -1.58%)
MB/sec scale 22762.02 ( 0.00%) 22524.00 ( -1.05%) 22416.54 ( -1.52%) 22444.16 ( -1.40%) 22652.56 ( -0.48%) 22462.80 ( -1.31%) 22461.90 ( -1.32%) 22670.84 ( -0.40%)
MB/sec add 26141.76 ( 0.00%) 26241.42 ( 0.38%) 26559.40 ( 1.60%) 26365.36 ( 0.86%) 26607.10 ( 1.78%) 26384.50 ( 0.93%) 26117.78 ( -0.09%) 26192.12 ( 0.19%)
MB/sec triad 26522.46 ( 0.00%) 26555.26 ( 0.12%) 26499.62 ( -0.09%) 26373.26 ( -0.56%) 26667.32 ( 0.55%) 26642.70 ( 0.45%) 26505.38 ( -0.06%) 26409.60 ( -0.43%)
v v VM VM VM VM VM VM
VM-HT VM-noHT HT noHT csc-HT csc_stallfix-HT csc_tim-HT csc_vruntime-HT
MB/sec copy 34559.32 ( 0.00%) 34153.30 ( -1.17%) 34236.64 ( -0.93%) 33724.38 ( -2.42%) 33535.60 ( -2.96%) 33534.10 ( -2.97%) 33469.70 ( -3.15%) 33873.18 ( -1.99%)
MB/sec scale 22556.18 ( 0.00%) 22834.88 ( 1.24%) 22733.12 ( 0.78%) 23010.46 ( 2.01%) 22480.60 ( -0.34%) 22552.94 ( -0.01%) 22756.50 ( 0.89%) 22434.96 ( -0.54%)
MB/sec add 26209.70 ( 0.00%) 26640.08 ( 1.64%) 26692.54 ( 1.84%) 26747.40 ( 2.05%) 26358.20 ( 0.57%) 26353.50 ( 0.55%) 26686.62 ( 1.82%) 26256.50 ( 0.18%)
MB/sec triad 26521.80 ( 0.00%) 26490.26 ( -0.12%) 26598.66 ( 0.29%) 26466.30 ( -0.21%) 26560.48 ( 0.15%) 26496.30 ( -0.10%) 26609.10 ( 0.33%) 26450.68 ( -0.27%)
v v VM VM VM VM VM VM
VM-v4-HT VM-v4-noHT v4-HT v4-noHT v4-csc-HT v4-csc_stallfix-HT v4-csc_tim-HT v4-csc_vruntime-HT
MB/sec copy 32257.48 ( 0.00%) 32504.18 ( 0.76%) 32375.66 ( 0.37%) 32261.98 ( 0.01%) 31940.84 ( -0.98%) 32070.88 ( -0.58%) 31926.80 ( -1.03%) 31882.18 ( -1.16%)
MB/sec scale 19806.46 ( 0.00%) 20281.18 ( 2.40%) 20266.80 ( 2.32%) 20075.46 ( 1.36%) 19847.66 ( 0.21%) 20119.00 ( 1.58%) 19899.84 ( 0.47%) 20060.48 ( 1.28%)
MB/sec add 22178.58 ( 0.00%) 22426.92 ( 1.12%) 22185.54 ( 0.03%) 22153.52 ( -0.11%) 21975.80 ( -0.91%) 22097.72 ( -0.36%) 21827.66 ( -1.58%) 22068.04 ( -0.50%)
MB/sec triad 22149.10 ( 0.00%) 22200.54 ( 0.23%) 22142.10 ( -0.03%) 21933.04 ( -0.98%) 21898.50 ( -1.13%) 22160.64 ( 0.05%) 22003.40 ( -0.66%) 21951.16 ( -0.89%)
v v VMx2 VMx2 VMx2 VMx2 VMx2 VMx2
VMx2-HT VMx2-noHT HT noHT csc-HT csc_stallfix-HT csc_tim-HT csc_vruntime-HT
MB/sec copy 33514.96 ( 0.00%) 24740.70 ( -26.18%) 30410.96 ( -9.26%) 22157.24 ( -33.89%) 29552.60 ( -11.82%) 29374.78 ( -12.35%) 28717.38 ( -14.31%) 29143.88 ( -13.04%)
MB/sec scale 22605.74 ( 0.00%) 15473.56 ( -31.55%) 19051.76 ( -15.72%) 15278.64 ( -32.41%) 19246.98 ( -14.86%) 19081.04 ( -15.59%) 18747.60 ( -17.07%) 18776.02 ( -16.94%)
MB/sec add 26249.56 ( 0.00%) 18559.92 ( -29.29%) 21143.90 ( -19.45%) 18664.30 ( -28.90%) 21236.00 ( -19.10%) 21067.40 ( -19.74%) 20878.78 ( -20.46%) 21266.92 ( -18.98%)
MB/sec triad 26290.16 ( 0.00%) 19274.10 ( -26.69%) 20573.62 ( -21.74%) 17631.52 ( -32.93%) 21066.94 ( -19.87%) 20975.04 ( -20.22%) 20944.56 ( -20.33%) 20942.18 ( -20.34%)So, STREAM, at least in this configuration, it is not (as it could have been expected) really sensitive to HyperThreading. In fact, in most cases, both when run on baremetal and in VMs, HT and noHT results are pretty much the same. When core scheduling is used, things does not look bad at all to me, although results are, most of the time, only marginally worse.
Do check, however, the overloaded case. There, disabling HT has quite a big impact, and core scheduling does a rather good job in restoring good performance.
From the overhead point of view, the situation does not look too bad either. In fact, in the first three group of measurements, the overhead introduced by having core scheduling patches in, is acceptable (there are actually cases where they seem to do more good than harm! :-P). However, when the system is overloaded, despite there not being any tagged task, numbers look pretty bad. It seems that, for instance, of the 13.04% performance drop between v-VMx2-HT and VMx2-csc_vruntime-HT, 9.26% comes from overhead (as that's there already in VMx2-HT)!!
Something to investigate better, I guess...
Appendix
* v-BM-HT : no coresched patch applied, baremetal, HyperThreading enabled * v-BM-noHT : no coresched patch applied, baremetal, Hyperthreading disabled * v-VM-HT : no coresched patch applied, 8 vCPUs VM, HyperThreading enabled * v-VM-noHT : no coresched patch applied, 8 vCPUs VM, Hyperthreading disabled * v-VM-v4-HT : no coresched patch applied, 4 vCPUs VM, HyperThreading enabled
* v-VM-v4-noHT : no coresched patch applied, 4 vCPUs VM, Hyperthreading disabled
* v-VMx2-HT : no coresched patch applied, 8 vCPUs VM + 600% stress overhead, HyperThreading enabled
* v-VMx2-noHT : no coresched patch applied, 8 vCPUs VM + 600% stress overhead, Hyperthreading disabled
* BM-HT : baremetal, HyperThreading enabled * BM-noHT : baremetal, Hyperthreading disabled * BM-csc-HT : baremetal, coresched-v3 (Hyperthreading enabled, of course)
* BM-csc_stallfix-HT : baremetal, coresched-v3 + stallfix (Hyperthreading enabled, of course)
* BM-csc_tim-HT : baremetal, coresched-v3 + Tim's patches (Hyperthreading enabeld, of course)
* BM-csc_vruntime-HT : baremetal, coresched-v3 + vruntime patches (Hyperthreading enabled, of course)
* VM-HT : 8 vCPUs VM, HyperThreading enabled * VM-noHT : 8 vCPUs VM, Hyperthreading disabled * VM-csc-HT : 8 vCPUs VM, coresched-v3 (Hyperthreading enabled, of course)
* VM-csc_stallfix-HT : 8 vCPUs VM, coresched-v3 + stallfix (Hyperthreading enabled, of course)
* VM-csc_tim-HT : 8 vCPUs VM, coresched-v3 + Tim's patches (Hyperthreading enabeld, of course)
* VM-csc_vruntime-HT : 8 vCPUs VM, coresched-v3 + vruntime patches (Hyperthreading enabled, of course)
* VM-v4-HT : 4 vCPUs VM, HyperThreading enabled * VM-v4-noHT : 4 vCPUs VM, Hyperthreading disabled * VM-v4-csc-HT : 4 vCPUs VM, coresched-v3 (Hyperthreading enabled, of course)
* VM-v4-csc_stallfix-HT : 4 vCPUs VM, coresched-v3 + stallfix (Hyperthreading enabled, of course)
* VM-v4-csc_tim-HT : 4 vCPUs VM, coresched-v3 + Tim's patches (Hyperthreading enabeld, of course)
* VM-v4-csc_vruntime-HT : 4 vCPUs VM, coresched-v3 + vruntime patches (Hyperthreading enabled, of course)
* VMx2-HT : 8 vCPUs VM + 600% stress overhead, HyperThreading enabled * VMx2-noHT : 8 vCPUs VM + 600% stress overhead, Hyperthreading disabled * VMx2-csc-HT : 8 vCPUs VM + 600% stress overhead, coresched-v3 (Hyperthreading enabled, of course)
* VMx2-csc_stallfix-HT : 8 vCPUs VM + 600% stress overhead, coresched-v3 + stallfix (Hyperthreading enabled, of course)
* VMx2-csc_tim-HT : 8 vCPUs VM + 600% stress overhead, coresched-v3 + Tim's patches (Hyperthreading enabeld, of course)
* VMx2-csc_vruntime-HT : 8 vCPUs VM + 600% stress overhead,
coresched-v3 + vruntime patches (Hyperthreading enabled, of course)
Signature · 6 lines
Dario Faggioli, Ph.D http://about.me/dario.faggioli Virtualization Software Engineer SUSE Labs, SUSE https://www.suse.com/ ------------------------------------------------------------------- <<This happens because _I_ choose it to happen!>> (Raistlin Majere)
Quoted 13 lines · Dario Faggioli
Quoted 5 lines · Aaron Lu
I'm using the following branch as base which is v5.1.5 based:
https://github.com/digitalocean/linux-coresched coresched-v3- v5.1.5-
testQuoted 7 lines
And I have pushed Tim's branch to:
https://github.com/aaronlu/linux coresched-v3-v5.1.5-test-timMine:
https://github.com/aaronlu/linux coresched-v3-v5.1.5-test-core_vruntime
Quoted 9 lines
Hello,
As anticipated, I've been trying to follow the development of this feature and, in the meantime, I have done some benchmarks.
I actually have a lot of data (and am planning for more), so I am sending a few emails, each one with a subset of the numbers in it, instead than just one which would be beyond giant! :-)
HACKBENCH-PROCESS-PIPES
=======================
v v BM BM BM BM BM BM
BM-HT BM-noHT HT noHT csc-HT csc_stallfix-HT csc_tim-HT csc_vruntime-HT
Amean 1 0.4372 ( 0.00%) 0.6714 * -53.57%* 0.4242 ( 2.97%) 0.6654 * -52.20%* 1.1748 *-168.71%* 1.1666 *-166.83%* 1.1496 *-162.95%* 1.3696 *-213.27%*
Amean 3 1.1466 ( 0.00%) 2.0284 * -76.91%* 1.1700 ( -2.04%) 2.0320 * -77.22%* 6.8484 *-497.28%* 6.7128 *-485.45%* 6.8068 *-493.65%* 7.9702 *-595.12%*
Amean 5 2.0140 ( 0.00%) 2.7834 * -38.20%* 2.0116 ( 0.12%) 2.6688 * -32.51%* 11.9494 *-493.32%* 11.9450 *-493.10%* 12.0904 *-500.32%* 14.4190 *-615.94%*
Amean 7 2.5396 ( 0.00%) 3.3064 * -30.19%* 2.6002 ( -2.39%) 2.9074 * -14.48%* 16.0142 *-530.58%* 16.3418 *-543.48%* 17.0896 *-572.92%* 20.2050 *-695.60%*
Amean 12 3.2930 ( 0.00%) 2.9830 ( 9.41%) 3.4226 ( -3.94%) 2.8522 ( 13.39%) 28.1482 *-754.79%* 27.3916 *-731.81%* 28.5938 *-768.32%* 34.6184 *-951.27%*
Amean 18 3.9452 ( 0.00%) 4.0806 ( -3.43%) 3.7950 ( 3.81%) 3.8938 ( 1.30%) 41.7120 *-957.28%* 42.1062 *-967.28%* 44.2136 *-1020.69%* 51.0200 *-1193.22%*
Amean 24 4.1618 ( 0.00%) 4.8466 * -16.45%* 4.2258 ( -1.54%) 5.0720 * -21.87%* 56.8598 *-1266.23%* 57.3568 *-1278.17%* 61.2660 *-1372.10%* 68.2538 *-1540.01%*
Amean 30 4.7790 ( 0.00%) 5.9726 * -24.98%* 4.8756 ( -2.02%) 5.9434 * -24.36%* 72.1602 *-1409.94%* 75.8036 *-1486.18%* 80.0066 *-1574.13%* 81.3768 *-1602.80%*
Amean 32 5.1680 ( 0.00%) 6.6000 * -27.71%* 5.2004 ( -0.63%) 6.5490 * -26.72%* 78.1974 *-1413.11%* 82.9812 *-1505.67%* 87.7340 *-1597.64%* 85.6876 *-1558.04%*
Stddev 1 0.0173 ( 0.00%) 0.0624 (-259.99%) 0.0129 ( 25.54%) 0.0451 (-160.36%) 0.0727 (-319.12%) 0.0954 (-450.05%) 0.0545 (-214.16%) 0.0960 (-453.74%)
Stddev 3 0.0471 ( 0.00%) 0.3035 (-544.20%) 0.0547 ( -16.12%) 0.2745 (-482.52%) 0.2029 (-330.60%) 0.1102 (-133.97%) 0.2391 (-407.38%) 0.1864 (-295.62%)
Stddev 5 0.1492 ( 0.00%) 0.4178 (-180.05%) 0.1419 ( 4.88%) 0.2569 ( -72.22%) 0.3878 (-159.96%) 0.2584 ( -73.22%) 0.4259 (-185.47%) 0.3340 (-123.89%)
Stddev 7 0.2077 ( 0.00%) 0.2941 ( -41.59%) 0.2281 ( -9.85%) 0.2049 ( 1.33%) 0.3922 ( -88.86%) 0.8178 (-293.78%) 0.4064 ( -95.67%) 0.6127 (-195.02%)
Stddev 12 0.5560 ( 0.00%) 0.2038 ( 63.34%) 0.2113 ( 62.00%) 0.2490 ( 55.21%) 1.0797 ( -94.21%) 0.7564 ( -36.06%) 1.0225 ( -83.91%) 1.5233 (-174.01%)
Stddev 18 0.3556 ( 0.00%) 0.3110 ( 12.55%) 0.3054 ( 14.11%) 0.1265 ( 64.43%) 1.0258 (-188.47%) 1.4386 (-304.53%) 1.1818 (-232.33%) 2.6710 (-651.08%)
Stddev 24 0.1844 ( 0.00%) 0.1614 ( 12.46%) 0.1135 ( 38.46%) 0.3679 ( -99.54%) 2.0997 (-1038.84%) 1.1493 (-523.36%) 0.5214 (-182.79%) 2.9229 (-1485.35%)
Stddev 30 0.1420 ( 0.00%) 0.0875 ( 38.37%) 0.1799 ( -26.66%) 0.1076 ( 24.26%) 4.5079 (-3073.51%) 1.5704 (-1005.58%) 1.4054 (-889.42%) 4.5743 (-3120.26%)
Stddev 32 0.2184 ( 0.00%) 0.2427 ( -11.11%) 0.3143 ( -43.92%) 0.3517 ( -61.01%) 3.7345 (-1609.81%) 1.3564 (-521.00%) 2.1822 (-899.11%) 4.0896 (-1772.40%)
v v VM VM VM VM VM VM
VM-HT VM-noHT HT noHT csc-HT csc_stallfix-HT csc_tim-HT csc_vruntime-HT
Amean 1 0.6762 ( 0.00%) 1.6824 *-148.80%* 0.6630 ( 1.95%) 1.5706 *-132.27%* 0.6600 ( 2.40%) 0.6514 ( 3.67%) 0.6780 ( -0.27%) 0.6964 * -2.99%*
Amean 3 1.7130 ( 0.00%) 2.2806 * -33.13%* 1.8074 * -5.51%* 2.1882 * -27.74%* 1.7992 ( -5.03%) 1.7896 * -4.47%* 1.7760 ( -3.68%) 1.8264 * -6.62%*
Amean 5 2.8688 ( 0.00%) 2.8048 ( 2.23%) 2.5880 * 9.79%* 2.8474 ( 0.75%) 2.5486 * 11.16%* 2.7896 ( 2.76%) 2.4402 * 14.94%* 2.6020 * 9.30%*
Amean 7 3.3432 ( 0.00%) 3.3434 ( -0.01%) 3.4564 ( -3.39%) 3.4704 ( -3.80%) 3.2424 ( 3.02%) 3.5620 ( -6.54%) 3.0352 ( 9.21%) 3.0874 ( 7.65%)
Amean 12 5.8936 ( 0.00%) 4.9968 * 15.22%* 5.1560 * 12.52%* 5.0670 * 14.03%* 4.2722 * 27.51%* 5.6570 ( 4.01%) 4.8270 * 18.10%* 4.2514 * 27.86%*
Amean 18 6.3938 ( 0.00%) 7.0542 ( -10.33%) 6.4900 ( -1.50%) 6.9682 ( -8.98%) 5.6478 ( 11.67%) 7.4324 ( -16.24%) 6.3160 ( 1.22%) 6.5124 ( -1.85%)
Amean 24 7.6278 ( 0.00%) 9.5096 * -24.67%* 7.0062 * 8.15%* 9.0278 * -18.35%* 7.5650 ( 0.82%) 8.2604 ( -8.29%) 8.5372 ( -11.92%) 7.7008 ( -0.96%)
Amean 30 10.5534 ( 0.00%) 10.9456 ( -3.72%) 11.4470 ( -8.47%) 11.1330 ( -5.49%) 8.8486 ( 16.15%) 10.8508 ( -2.82%) 12.6182 ( -19.57%) 10.5836 ( -0.29%)
Amean 32 11.6024 ( 0.00%) 12.6052 ( -8.64%) 10.8236 ( 6.71%) 11.2156 ( 3.33%) 9.0654 * 21.87%* 11.5074 ( 0.82%) 10.3592 ( 10.72%) 11.7174 ( -0.99%)
Stddev 1 0.0143 ( 0.00%) 0.3700 (-2480.08%) 0.0261 ( -82.29%) 0.1206 (-741.16%) 0.0444 (-209.74%) 0.0270 ( -88.42%) 0.0238 ( -65.73%) 0.0180 ( -25.17%)
Stddev 3 0.0473 ( 0.00%) 0.1384 (-192.63%) 0.0739 ( -56.38%) 0.0714 ( -51.01%) 0.0931 ( -96.94%) 0.0517 ( -9.28%) 0.0960 (-103.02%) 0.0553 ( -17.04%)
Stddev 5 0.1236 ( 0.00%) 0.2251 ( -82.10%) 0.1607 ( -29.98%) 0.1464 ( -18.41%) 0.1319 ( -6.69%) 0.1842 ( -48.96%) 0.0647 ( 47.66%) 0.1977 ( -59.92%)
Stddev 7 0.3597 ( 0.00%) 0.1105 ( 69.27%) 0.2288 ( 36.38%) 0.1335 ( 62.90%) 0.3131 ( 12.94%) 0.2958 ( 17.76%) 0.1581 ( 56.05%) 0.2361 ( 34.37%)
Stddev 12 0.4677 ( 0.00%) 0.0846 ( 81.91%) 0.6319 ( -35.11%) 0.1526 ( 67.37%) 0.3591 ( 23.23%) 0.6898 ( -47.48%) 0.4853 ( -3.76%) 0.2963 ( 36.64%)
Stddev 18 1.2289 ( 0.00%) 0.1849 ( 84.95%) 1.1160 ( 9.18%) 0.2497 ( 79.68%) 0.9843 ( 19.90%) 0.9542 ( 22.35%) 0.6621 ( 46.12%) 0.7668 ( 37.60%)
Stddev 24 0.5202 ( 0.00%) 0.9344 ( -79.60%) 0.1940 ( 62.71%) 0.4706 ( 9.53%) 0.8362 ( -60.73%) 1.0819 (-107.96%) 1.0229 ( -96.63%) 0.8881 ( -70.72%)
Stddev 30 2.1557 ( 0.00%) 0.7984 ( 62.96%) 1.2499 ( 42.02%) 0.9804 ( 54.52%) 0.4846 ( 77.52%) 1.2901 ( 40.15%) 1.5532 ( 27.95%) 1.2932 ( 40.01%)
Stddev 32 2.2255 ( 0.00%) 1.1321 ( 49.13%) 2.2380 ( -0.56%) 0.2127 ( 90.44%) 0.3654 ( 83.58%) 1.5727 ( 29.33%) 2.3291 ( -4.66%) 2.0936 ( 5.93%)
v v VM VM VM VM VM VM
VM-v4-HT VM-v4-noHT v4-HT v4-noHT v4-csc-HT v4-csc_stallfix-HT v4-csc_tim-HT v4-csc_vruntime-HT
Amean 1 1.2194 ( 0.00%) 1.1974 ( 1.80%) 1.0308 * 15.47%* 1.1054 * 9.35%* 1.0522 * 13.71%* 1.2290 ( -0.79%) 1.1392 * 6.58%* 1.0524 * 13.70%*
Amean 3 2.2568 ( 0.00%) 2.0588 ( 8.77%) 2.0708 ( 8.24%) 2.2352 ( 0.96%) 2.1808 ( 3.37%) 2.2820 ( -1.12%) 2.1682 ( 3.93%) 2.2598 ( -0.13%)
Amean 5 2.9848 ( 0.00%) 2.9912 ( -0.21%) 2.4938 * 16.45%* 2.4634 * 17.47%* 2.6890 ( 9.91%) 2.8908 ( 3.15%) 2.8636 ( 4.06%) 2.4158 * 19.06%*
Amean 7 3.4500 ( 0.00%) 3.2538 ( 5.69%) 3.3646 ( 2.48%) 3.2666 ( 5.32%) 3.0800 ( 10.72%) 4.2206 ( -22.34%) 3.1016 ( 10.10%) 3.2186 ( 6.71%)
Amean 12 6.0414 ( 0.00%) 5.0624 ( 16.20%) 5.1276 ( 15.13%) 5.1066 ( 15.47%) 4.7728 * 21.00%* 5.5068 ( 8.85%) 4.7544 * 21.30%* 5.8920 ( 2.47%)
Amean 16 7.5510 ( 0.00%) 7.6888 ( -1.82%) 6.9732 ( 7.65%) 5.9098 * 21.73%* 6.5542 * 13.20%* 6.9492 ( 7.97%) 6.4372 * 14.75%* 6.1968 * 17.93%*
Stddev 1 0.0786 ( 0.00%) 0.1166 ( -48.34%) 0.1762 (-124.09%) 0.0712 ( 9.45%) 0.1541 ( -95.99%) 0.0814 ( -3.55%) 0.0452 ( 42.57%) 0.1817 (-131.05%)
Stddev 3 0.2220 ( 0.00%) 0.1887 ( 15.03%) 0.2174 ( 2.07%) 0.1368 ( 38.37%) 0.1928 ( 13.17%) 0.4342 ( -95.57%) 0.2353 ( -5.97%) 0.1753 ( 21.06%)
Stddev 5 0.4586 ( 0.00%) 0.4689 ( -2.23%) 0.3202 ( 30.19%) 0.2666 ( 41.88%) 0.3108 ( 32.24%) 0.2876 ( 37.29%) 0.4067 ( 11.33%) 0.3010 ( 34.37%)
Stddev 7 0.3769 ( 0.00%) 0.5242 ( -39.09%) 0.2523 ( 33.06%) 0.2498 ( 33.71%) 0.4527 ( -20.13%) 1.5089 (-300.36%) 0.3998 ( -6.09%) 0.4018 ( -6.62%)
Stddev 12 1.0604 ( 0.00%) 0.5194 ( 51.02%) 0.5646 ( 46.75%) 0.5255 ( 50.45%) 0.4872 ( 54.06%) 0.3447 ( 67.49%) 0.5492 ( 48.21%) 0.5487 ( 48.26%)
Stddev 16 0.6245 ( 0.00%) 0.5220 ( 16.40%) 0.6914 ( -10.71%) 0.6984 ( -11.83%) 0.3543 ( 43.27%) 0.5137 ( 17.74%) 0.5083 ( 18.61%) 1.0744 ( -72.04%)
v v VMx2 VMx2 VMx2 VMx2 VMx2 VMx2
VMx2-HT VMx2-noHT HT noHT csc-HT csc_stallfix-HT csc_tim-HT csc_vruntime-HT
Amean 1 0.6780 ( 0.00%) 2.1522 *-217.43%* 1.6790 *-147.64%* 2.2440 *-230.97%* 15.6090 *-2202.21%* 9.1920 *-1255.75%* 2.0732 *-205.78%* 2.2390 *-230.24%*
Amean 3 1.8452 ( 0.00%) 3.2456 * -75.89%* 2.5214 * -36.65%* 3.3944 * -83.96%* 13.7258 *-643.87%* 9.7654 *-429.23%* 2.6668 * -44.53%* 2.9044 * -57.40%*
Amean 5 2.8564 ( 0.00%) 4.2548 * -48.96%* 3.2500 * -13.78%* 4.3958 * -53.89%* 11.6808 *-308.93%* 8.4724 *-196.61%* 3.4000 * -19.03%* 3.5298 * -23.58%*
Amean 7 3.3712 ( 0.00%) 5.1056 * -51.45%* 4.1636 * -23.50%* 4.9874 * -47.94%* 14.0516 (-316.81%) 9.1454 *-171.28%* 4.1396 * -22.79%* 4.3570 * -29.24%*
Amean 12 4.8134 ( 0.00%) 7.4516 * -54.81%* 6.1254 * -27.26%* 7.3488 * -52.67%* 13.7680 *-186.03%* 13.7400 *-185.45%* 6.0424 * -25.53%* 6.3404 * -31.72%*
Amean 18 6.1980 ( 0.00%) 10.4126 * -68.00%* 8.5942 * -38.66%* 9.7554 * -57.40%* 10.7234 * -73.01%* 13.5214 *-118.16%* 8.6628 * -39.77%* 8.5910 * -38.61%*
Amean 24 8.2116 ( 0.00%) 12.7570 * -55.35%* 10.9386 * -33.21%* 12.9032 * -57.13%* 17.8492 *-117.37%* 14.4432 * -75.89%* 10.7488 * -30.90%* 11.1930 * -36.31%*
Amean 30 9.3264 ( 0.00%) 15.3704 * -64.81%* 13.6630 * -46.50%* 15.4806 * -65.99%* 16.9272 * -81.50%* 18.4184 * -97.49%* 13.6174 * -46.01%* 13.5026 * -44.78%*
Amean 32 10.1954 ( 0.00%) 16.5494 * -62.32%* 14.7450 * -44.62%* 16.4650 * -61.49%* 13.9146 * -36.48%* 17.5790 * -72.42%* 14.6072 * -43.27%* 14.0346 * -37.66%*
Stddev 1 0.0129 ( 0.00%) 0.2665 (-1959.36%) 0.2262 (-1647.54%) 0.0703 (-443.26%) 9.5450 (-73651.40%) 6.3226 (-48752.57%) 0.1736 (-1241.41%) 0.3070 (-2272.09%)
Stddev 3 0.1156 ( 0.00%) 0.3653 (-215.89%) 0.1388 ( -20.00%) 0.2067 ( -78.77%) 3.3781 (-2821.39%) 3.2241 (-2688.20%) 0.0970 ( 16.10%) 0.1247 ( -7.87%)
Stddev 5 0.0817 ( 0.00%) 0.2968 (-263.19%) 0.0938 ( -14.79%) 0.1367 ( -67.25%) 3.1273 (-3726.47%) 3.9175 (-4693.34%) 0.1155 ( -41.31%) 0.1864 (-128.10%)
Stddev 7 0.1190 ( 0.00%) 0.1337 ( -12.31%) 0.3136 (-163.48%) 0.2034 ( -70.91%) 17.1856 (-14336.96%) 2.8676 (-2308.94%) 0.0794 ( 33.32%) 0.1362 ( -14.42%)
Stddev 12 0.5237 ( 0.00%) 0.2929 ( 44.08%) 0.1386 ( 73.53%) 0.2330 ( 55.51%) 7.1869 (-1272.26%) 8.0085 (-1429.13%) 0.2535 ( 51.59%) 0.1441 ( 72.49%)
Stddev 18 0.4791 ( 0.00%) 0.1528 ( 68.10%) 0.4254 ( 11.21%) 0.2953 ( 38.37%) 2.8492 (-494.71%) 4.3767 (-813.54%) 0.3205 ( 33.10%) 0.3094 ( 35.43%)
Stddev 24 1.1591 ( 0.00%) 0.2616 ( 77.43%) 0.2446 ( 78.90%) 0.4363 ( 62.36%) 5.3380 (-360.53%) 3.2433 (-179.81%) 0.2704 ( 76.67%) 0.4402 ( 62.02%)
Stddev 30 0.6561 ( 0.00%) 0.4008 ( 38.91%) 0.2670 ( 59.31%) 0.1451 ( 77.88%) 4.5765 (-597.54%) 4.2380 (-545.95%) 0.4768 ( 27.33%) 0.2693 ( 58.96%)
Stddev 32 1.2197 ( 0.00%) 0.4459 ( 63.44%) 0.5496 ( 54.94%) 0.4179 ( 65.74%) 1.3553 ( -11.12%) 4.3537 (-256.96%) 0.5602 ( 54.07%) 0.5312 ( 56.45%)So, situation looks pretty *terrible* on baremetal. Well, something to investigate, I guess.
In VMs, on the other hand, things don't look too bad. Although, it's a bit weird that the benchmark seems to be sensitive to HT when run on baremetal, but not so much when run in VMs. Among the various core scheduling implementations/patches, plain v3 seems to have issues with this workload, while Tim's and vruntime are better.
It is confirmed that, in virtualization scenario, under overcommittment, core scheduling is more effective than disabling HyperThreading, at least either with Tim's or with vruntime patches.
Overhead wise, it is an hard call. Numbers vary a lot, and I think each group of measurements needs to be looked at carefully to come up with a sensible analysis.
Overhead is there, that's for sure, in the baremetal case. Also, it looks more severe when HT is disabled (e.g., comapre v-BM-noHT with BM- noHT).
In virt cases, it's really an hard call. E.g., when a VM with 4 vCPUs is used, core scheduling seems to be able to make the system significantly faster! :-O
Signature · 6 lines
Dario Faggioli, Ph.D http://about.me/dario.faggioli Virtualization Software Engineer SUSE Labs, SUSE https://www.suse.com/ ------------------------------------------------------------------- <<This happens because _I_ choose it to happen!>> (Raistlin Majere)