scheduler changes for v4.9
Subsystems (8)
Linus,
Please pull the latest sched-core-for-linus git tree from:
git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git sched-core-for-linus
# HEAD: 447976ef4fd09b1be88b316d1a81553f1aa7cd07 sched/irqtime: Consolidate irqtime flushing code
The main changes wee:
- irqtime accounting cleanups and enhancements. (Frederic Weisbecker)
- schedstat debugging enhancements, make it more broadly runtime available. (Josh Poimboeuf)
- More work on asymmetric topology/capacity scheduling. (Morten Rasmussen)
- sched/wait fixes and cleanups. (Oleg Nesterov)
- PELT (per entity load tracking) improvements. (Peter Zijlstra)
- Rewrite and enhance select_idle_siblings(). (Peter Zijlstra)
- sched/numa enhancements/fixes (Rik van Riel)
- sched/cputime scalability improvements (Stanislaw Gruszka)
- Load calculation arithmetics fixes. (Dietmar Eggemann)
- sched/deadline enhancements (Tommaso Cucinotta)
- Fix utilization accounting when switching to the SCHED_NORMAL policy. (Vincent Guittot)
- ... plus misc cleanups and enhancements.
Thanks,
Ingo
------------------>
Byungchul Park (1):
sched/fair: Make update_min_vruntime() more readable
Cheng Chao (2):
sched/core: Remove unnecessary initialization in sched_init()
stop_machine: Avoid a sleep and wakeup in stop_one_cpu()
Con Kolivas (1):
sched/core: Do not use smp_processor_id() with preempt enabled in smpboot_thread_fn()
Dietmar Eggemann (3):
sched/core: Store maximum per-CPU capacity in root domain sched/fair: Fix load_above_capacity fixed point arithmetic width sched/fair: Fix fixed point arithmetic width for shares and effective load
Frederic Weisbecker (5):
sched/irqtime: No need for preempt-safe accessors sched/irqtime: Remove needless IRQs disablement on kcpustat update
u64_stats: Introduce IRQs disabled helpers
sched/irqtime: Consolidate accounting synchronization with u64_stats API sched/irqtime: Consolidate irqtime flushing code
Josh Poimboeuf (5):
sched/debug: Rename and move enqueue_sleeper() sched/debug: Clean up schedstat macros
sched/debug: Rename 'schedstat_val()' -> 'schedstat_val_or_zero()' sched/debug: Remove several CONFIG_SCHEDSTATS guards
objtool: Add do_task_dead() to global noreturn list
Juri Lelli (1):
sched/deadline: Remove useless parameter from setup_new_dl_entity()
Leo Yan (2):
sched/fair: Remove 'cpu_busy' parameter from update_next_balance() sched/core: Fix one typo
Luis de Bethencourt (1):
sched/core: Add documentation for 'cookie' argument
Morten Rasmussen (8):
sched/core: Fix power to capacity renaming in comment sched/fair: Make the use of prev_cpu consistent in the wakeup path sched/fair: Optimize find_idlest_cpu() when there is no choice sched/core: Remove unnecessary NULL-pointer check sched/core: Introduce SD_ASYM_CPUCAPACITY sched_domain topology flag sched/core: Pass child domain into sd_init()
sched/core: Enable SD_BALANCE_WAKE for asymmetric capacity systems sched/fair: Let asymmetric CPU configurations balance at wake-up
Oleg Nesterov (4):
sched/wait: Fix abort_exclusive_wait(), it should pass TASK_NORMAL to wake_up() sched/wait: Avoid abort_exclusive_wait() in ___wait_event() sched/wait: Avoid abort_exclusive_wait() in __wait_on_bit_lock() sched/wait: Introduce init_wait_entry()
Peter Zijlstra (17):
sched/fair: Improve PELT stuff some more
sched/core: Clarify SD_flags comment
Revert "sched/fair: Make update_min_vruntime() more readable" sched/core: Optimize __schedule()
sched/core: Avoid _cond_resched() for PREEMPT=y sched/debug: Hide printk() by default
sched/core: Remove unused @cpu argument from destroy_sched_domain*() sched/core: Restructure destroy_sched_domain() sched/core: Introduce 'struct sched_domain_shared' sched/core: Replace sd_busy/nr_busy_cpus with sched_domain_shared sched/core: Rewrite and improve select_idle_siblings() sched/core: Optimize SCHED_SMT
sched/core, ia64: Rename set_curr_task()
sched/fair: Introduce set_curr_task() helper
sched/core: Fix set_user_nice()
sched/debug: Add SCHED_WARN_ON()
sched/fair: Fix min_vruntime tracking
Rik van Riel (2):
sched: Remove struct rq::nohz_stamp
sched/numa, mm: Revert to checking pmd/pte_write instead of VMA flags
Srivatsa Vaddagiri (1):
sched/fair: Fix SCHED_HRTICK bug leading to late preemption of tasks
Stanislaw Gruszka (1):
sched/cputime: Improve scalability by not accounting thread group tasks pending runtime
Tim Chen (1):
sched/core, x86/topology: Fix NUMA in package topology bug
Tommaso Cucinotta (4):
sched/deadline: Refactor CPU heap code
sched/deadline: Make CPU heap faster avoiding real swaps on heapify sched/deadline: Split cpudl_set() into cpudl_set() and cpudl_clear() sched/deadline: Document behavior of sched_yield()
Vegard Nossum (2):
sched/debug: Make the "Preemption disabled at ..." message more useful sched/debug: Add taint on "BUG: Sleeping function called from invalid context"
Vincent Guittot (1):
sched/core: Fix incorrect utilization accounting when switching to fair class
Wanpeng Li (1):
sched/deadline: Fix the intention to re-evalute tick dependency for offline CPU
seokhoon.yoon (1):
schedcore: Remove duplicated init_task's preempt_notifiers init
Documentation/scheduler/sched-deadline.txt | 18 + arch/ia64/kernel/mca.c | 10 +- arch/x86/kernel/smpboot.c | 46 +- include/linux/kernel.h | 9 +- include/linux/sched.h | 30 +- include/linux/u64_stats_sync.h | 45 +- include/linux/wait.h | 17 +- kernel/exit.c | 26 +- kernel/sched/core.c | 322 ++++++++---- kernel/sched/cpudeadline.c | 153 +++--- kernel/sched/cpudeadline.h | 3 +- kernel/sched/cputime.c | 87 ++-- kernel/sched/deadline.c | 78 +-- kernel/sched/debug.c | 103 ++-- kernel/sched/fair.c | 752 +++++++++++++++++++---------- kernel/sched/idle_task.c | 4 +- kernel/sched/sched.h | 92 ++-- kernel/sched/stats.h | 24 +- kernel/sched/wait.c | 113 +++-- kernel/smpboot.c | 2 +- kernel/stop_machine.c | 5 + mm/huge_memory.c | 2 +- mm/memory.c | 2 +- tools/objtool/builtin-check.c | 1 + 24 files changed, 1198 insertions(+), 746 deletions(-)
Documentation/scheduler/sched-deadline.txt +18 -0
@@ -16,6 +16,7 @@ CONTENTS 4.1 System-wide settings 4.2 Task interface 4.3 Default behavior + 4.4 Behavior of sched_yield() 5. Tasks CPU affinity 5.1 SCHED_DEADLINE and cpusets HOWTO 6. Future plans
@@ -426,6 +427,23 @@ CONTENTS Finally, notice that in order not to jeopardize the admission control a -deadline task cannot fork. + +4.4 Behavior of sched_yield() +----------------------------- + + When a SCHED_DEADLINE task calls sched_yield(), it gives up its + remaining runtime and is immediately throttled, until the next + period, when its runtime will be replenished (a special flag + dl_yielded is set and used to handle correctly throttling and runtime + replenishment after a call to sched_yield()). + + This behavior of sched_yield() allows the task to wake-up exactly at + the beginning of the next period. Also, this may be useful in the + future with bandwidth reclaiming mechanisms, where sched_yield() will + make the leftoever runtime available for reclamation by other + SCHED_DEADLINE tasks. + + 5. Tasks CPU affinity =====================
arch/ia64/kernel/mca.c +5 -5
@@ -986,7 +986,7 @@ ia64_mca_modify_original_stack(struct pt_regs *regs,int cpu =smp_processor_id (); previous_current =curr_task (cpu); -set_curr_task (cpu, current); +ia64_set_curr_task (cpu, current);if ((p =strchr (current->comm,' ' ))) *p ='\0' ;
@@ -1360,14 +1360,14 @@ ia64_mca_handler(struct pt_regs *regs, struct switch_stack *sw,cpumask_clear_cpu (i, &mca_cpu);/* wake next cpu */ while (monarch_cpu !=-1 )cpu_relax ();/* spin until last cpu leaves */ -set_curr_task (cpu, previous_current); +ia64_set_curr_task (cpu, previous_current); ia64_mc_info.imi_rendez_checkin[cpu] =IA64_MCA_RENDEZ_CHECKIN_NOTDONE ;return ; } } } -set_curr_task (cpu, previous_current); +ia64_set_curr_task (cpu, previous_current); ia64_mc_info.imi_rendez_checkin[cpu] =IA64_MCA_RENDEZ_CHECKIN_NOTDONE ; monarch_cpu =-1 ;/* This frees the slaves and previous monarchs */ }
@@ -1729,7 +1729,7 @@ ia64_init_handler(struct pt_regs *regs, struct switch_stack *sw,NOTIFY_INIT (DIE_INIT_SLAVE_LEAVE , regs, (long )&nd,1 );mprintk ("Slave on cpu %d returning to normal service.\n" , cpu); -set_curr_task (cpu, previous_current); +ia64_set_curr_task (cpu, previous_current); ia64_mc_info.imi_rendez_checkin[cpu] =IA64_MCA_RENDEZ_CHECKIN_NOTDONE ;atomic_dec (&slaves);return ;
@@ -1756,7 +1756,7 @@ ia64_init_handler(struct pt_regs *regs, struct switch_stack *sw,mprintk ("\nINIT dump complete. Monarch on cpu %d returning to normal service.\n" , cpu);atomic_dec (&monarchs); -set_curr_task (cpu, previous_current); +ia64_set_curr_task (cpu, previous_current); monarch_cpu =-1 ;return ; }
arch/x86/kernel/smpboot.c +30 -16
78 lines not on this page · raw message
include/linux/kernel.h +3 -6
@@ -259,17 +259,14 @@ static inline void might_fault(void) { }extern struct atomic_notifier_head panic_notifier_list;extern long (*panic_blink)(int state); __printf(1, 2) -void panic (const char *fmt, ...) -__noreturn __cold ; +void panic (const char *fmt, ...)__noreturn __cold ;void nmi_panic (struct pt_regs *regs,const char *msg);extern void oops_enter (void );extern void oops_exit (void );void print_oops_end_marker (void );extern int oops_may_print (void ); -void do_exit (long error_code) -__noreturn ; -void complete_and_exit (struct completion *,long ) -__noreturn ; +void do_exit (long error_code)__noreturn ; +void complete_and_exit (struct completion *,long )__noreturn ;/* Internal, do not use. */ int __must_check _kstrtoul (const char *s,unsigned int base,unsigned long *res);
include/linux/sched.h +28 -2
@@ -448,6 +448,8 @@ static inline void io_schedule(void)io_schedule_timeout (MAX_SCHEDULE_TIMEOUT ); } +void __noreturn do_task_dead (void ); +struct nsproxy ;struct user_namespace ;
@@ -1022,7 +1024,8 @@ extern void wake_up_q(struct wake_q_head *head);#define SD_BALANCE_FORK 0x0008/* Balance on fork, clone */ #define SD_BALANCE_WAKE 0x0010/* Balance on wakeup */ #define SD_WAKE_AFFINE 0x0020/* Wake task to waking CPU */ -#define SD_SHARE_CPUCAPACITY 0x0080/* Domain members share cpu power */ +#define SD_ASYM_CPUCAPACITY 0x0040/* Groups have different max cpu capacities */ +#define SD_SHARE_CPUCAPACITY 0x0080/* Domain members share cpu capacity */ #define SD_SHARE_POWERDOMAIN 0x0100/* Domain members share power domain */ #define SD_SHARE_PKG_RESOURCES 0x0200/* Domain members share cpu pkg resources */ #define SD_SERIALIZE 0x0400/* Only a single load balancing instance */
@@ -1064,6 +1067,12 @@ extern int sched_domain_level_max;struct sched_group ; +struct sched_domain_shared { +atomic_t ref; +atomic_t nr_busy_cpus; +int has_idle_cores; +}; +struct sched_domain {/* These fields must be setup */ struct sched_domain *parent;/* top domain must be null terminated */
@@ -1094,6 +1103,8 @@ struct sched_domain {u64 max_newidle_lb_cost;unsigned long next_decay_max_lb_cost; +u64 avg_scan_cost;/* select_idle_sibling */ +#ifdef CONFIG_SCHEDSTATS /* load_balance() stats */ unsigned int lb_count[CPU_MAX_IDLE_TYPES ];
@@ -1132,6 +1143,7 @@ struct sched_domain {void *private;/* used during construction */ struct rcu_head rcu;/* used during destruction */ }; +struct sched_domain_shared *shared;unsigned int span_weight;/*
@@ -1165,6 +1177,7 @@ typedef int (*sched_domain_flags_f)(void);struct sd_data {struct sched_domain **__percpu sd; +struct sched_domain_shared **__percpu sds;struct sched_group **__percpu sg;struct sched_group_capacity **__percpu sgc; };
@@ -2568,7 +2581,7 @@ static inline bool is_idle_task(const struct task_struct *p)return p->pid ==0 ; }extern struct task_struct *curr_task (int cpu); -extern void set_curr_task (int cpu,struct task_struct *p); +extern void ia64_set_curr_task (int cpu,struct task_struct *p);void yield (void );
@@ -3206,7 +3219,11 @@ static inline int signal_pending_state(long state, struct task_struct *p)* cond_resched_lock() will drop the spinlock before scheduling, * cond_resched_softirq() will enable bhs before scheduling. */ +#ifndef CONFIG_PREEMPT extern int _cond_resched (void ); +#else +static inline int _cond_resched (void ) {return 0 ; } +#endif #define cond_resched () ({ \ ___might_sleep(__FILE__, __LINE__, 0); \
@@ -3236,6 +3253,15 @@ static inline void cond_resched_rcu(void)#endif } +static inline unsigned long get_preempt_disable_ip (struct task_struct *p) +{ +#ifdef CONFIG_DEBUG_PREEMPT +return p->preempt_disable_ip; +#else +return 0 ; +#endif +} +/* * Does a critical section need to be broken due to another * task waiting?: (technically does not depend on CONFIG_PREEMPT,
include/linux/u64_stats_sync.h +24 -21
82 lines not on this page · raw message
include/linux/wait.h +4 -13
@@ -248,6 +248,8 @@ wait_queue_head_t *bit_waitqueue(void *, int); (!__builtin_constant_p (state) || \ state == TASK_INTERRUPTIBLE || state == TASK_KILLABLE) \ +extern void init_wait_entry (wait_queue_t *__wait,int flags); +/* * The below macro ___wait_event() has an explicit shadow of the __ret * variable when used from the wait_event_*() macros.
@@ -266,12 +268,7 @@ wait_queue_head_t *bit_waitqueue(void *, int);wait_queue_t __wait; \long __ret = ret;/* explicit shadow */ \ \ -INIT_LIST_HEAD (&__wait.task_list); \ -if (exclusive) \ - __wait.flags =WQ_FLAG_EXCLUSIVE ; \ -else \ - __wait.flags =0 ; \ - \ +init_wait_entry (&__wait, exclusive ?WQ_FLAG_EXCLUSIVE :0 ); \for (;;) { \long __int =prepare_to_wait_event (&wq, &__wait, state);\ \
@@ -280,12 +277,7 @@ wait_queue_head_t *bit_waitqueue(void *, int); \if (___wait_is_interruptible (state) && __int) { \ __ret = __int; \ -if (exclusive) { \ -abort_exclusive_wait (&wq, &__wait, \ - state,NULL ); \ - goto __out; \ - } \ -break ; \ + goto __out; \ } \ \ cmd; \
@@ -989,7 +981,6 @@ void prepare_to_wait(wait_queue_head_t *q, wait_queue_t *wait, int state);void prepare_to_wait_exclusive (wait_queue_head_t *q,wait_queue_t *wait,int state);long prepare_to_wait_event (wait_queue_head_t *q,wait_queue_t *wait,int state);void finish_wait (wait_queue_head_t *q,wait_queue_t *wait); -void abort_exclusive_wait (wait_queue_head_t *q,wait_queue_t *wait,unsigned int mode,void *key);long wait_woken (wait_queue_t *wait,unsigned mode,long timeout);int woken_wake_function (wait_queue_t *wait,unsigned mode,int sync,void *key);int autoremove_wake_function (wait_queue_t *wait,unsigned mode,int sync,void *key);
kernel/exit.c +2 -24
@@ -725,7 +725,7 @@ static void check_stack_usage(void)static inline void check_stack_usage (void ) {}#endif -void do_exit (long code) +void __noreturn do_exit (long code) {struct task_struct *tsk = current;int group_dead;
@@ -882,29 +882,7 @@ void do_exit(long code)exit_rcu ();TASKS_RCU (__srcu_read_unlock (&tasks_rcu_exit_srcu, tasks_rcu_i)); -/* -* The setting of TASK_RUNNING by try_to_wake_up() may be delayed -* when the following two conditions become true. -* - There is race condition of mmap_sem (It is acquired by -* exit_mm()), and -* - SMI occurs before setting TASK_RUNINNG. -* (or hypervisor of virtual machine switches to other guest) -* As a result, we may become TASK_RUNNING after becoming TASK_DEAD -* -* To avoid it, we have to wait for releasing tsk->pi_lock which -* is held by try_to_wake_up() -*/ -smp_mb (); -raw_spin_unlock_wait (&tsk->pi_lock); - -/* causes final put_task_struct in finish_task_switch(). */ - tsk->state =TASK_DEAD ; - tsk->flags |=PF_NOFREEZE ;/* tell freezer to ignore us */ -schedule (); -BUG (); -/* Avoid "noreturn function does return". */ -for (;;) -cpu_relax ();/* For when BUG is null */ +do_task_dead (); }EXPORT_SYMBOL_GPL (do_exit);
kernel/sched/core.c +218 -104
759 lines not on this page · raw message
kernel/sched/cpudeadline.c +95 -58
214 lines not on this page · raw message
kernel/sched/cpudeadline.h +2 -1
@@ -23,7 +23,8 @@ struct cpudl {#ifdef CONFIG_SMP int cpudl_find (struct cpudl *cp,struct task_struct *p,struct cpumask *later_mask); -void cpudl_set (struct cpudl *cp,int cpu,u64 dl,int is_valid); +void cpudl_set (struct cpudl *cp,int cpu,u64 dl); +void cpudl_clear (struct cpudl *cp,int cpu);int cpudl_init (struct cpudl *cp);void cpudl_set_freecpu (struct cpudl *cp,int cpu);void cpudl_clear_freecpu (struct cpudl *cp,int cpu);
kernel/sched/cputime.c +55 -32
158 lines not on this page · raw message
kernel/sched/deadline.c +39 -39
166 lines not on this page · raw message
kernel/sched/debug.c +55 -48
178 lines not on this page · raw message
kernel/sched/fair.c +494 -258
1,102 lines not on this page · raw message
kernel/sched/sched.h +47 -45
92 lines not on this page · raw message
kernel/sched/wait.c +55 -58
113 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
Part of the diff is not shown here. View the raw message.