scheduler updates for v2.6.30
Subsystems (4)
Linus,
Please pull the latest sched-for-linus git tree from:
git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip.git sched-for-linus
Highlights:
- Most of the linecount comes from a clean-up of the balancing code
- Real-time scheduling optimizations
- Performance tweaks (sched-clock speedups for x86)
- Iteractivity tweaks
Risks:
- There were no unusual trouble spots during development.
- There are no open regressions.
- The interactivity/performance tweaks carry performance regression risks:
e52fb7c: sched: prefer wakers
df1c99d: sched: add avg_overlap decay
Thanks,
Ingo
------------------>
Américo Wang (1):
sched: use TASK_NICE for task_struct
Arjan van de Ven (1):
sched, latencytop: incorporate review feedback from Andrew Morton
Frederic Weisbecker (1):
sched: don't rebalance if attached on NULL domain
Gautham R Shenoy (11):
sched: Simple helper functions for find_busiest_group()
sched: Fix indentations in find_busiest_group() using gotos
sched: Define structure to store the sched_group statistics for fbg()
sched: Create a helper function to calculate sched_group stats for fbg()
sched: Define structure to store the sched_domain statistics for fbg()
sched: Create a helper function to calculate sched_domain stats for fbg()
sched: Create helper to calculate small_imbalance in fbg()
sched: Create a helper function to calculate imbalance
sched: Optimize the !power_savings_balance during fbg()
sched: Refactor the power savings balance code
sched: Add comments to find_busiest_group() function
Gregory Haskins (13):
sched: cleanup inc/dec_rt_tasks
sched: track the next-highest priority on each runqueue
sched: use highest_prio.curr for pull threshold
sched: use highest_prio.next to optimize pull operations
sched: only try to push a task on wakeup if it is migratable
sched: pull only one task during NEWIDLE balancing to limit critical section
sched: make double-lock-balance fair
sched: add sched_class->needs_post_schedule() member
plist: fix PLIST_NODE_INIT to work with debug enabled
sched: create "pushable_tasks" list to limit pushing to one attempt
RT: fix push_rt_task() to handle dequeue_pushable properly
sched: de CPP-ify the scheduler code
sched: fix build error in kernel/sched_rt.c when RT_GROUP_SCHED && !SMP
Henrik Austad (1):
sched: idle_at_tick is only used when CONFIG_SMP is set
Ingo Molnar (4):
sched: fix !CONFIG_SCHEDSTATS build failure
sched: allow architectures to specify sched_clock_stable
x86: set X86_FEATURE_TSC_RELIABLE
x86, sched_clock(): mark variables read-mostly
Lai Jiangshan (1):
sched: TIF_NEED_RESCHED -> need_reshed() cleanup
Li Zefan (1):
cpuacct: add a branch prediction
Luis Henriques (4):
sched: fix typos in documentation
sched: small optimisation of can_migrate_task()
sched: jiffies not printed per CPU
sched: remove unused fields from struct rq
Mike Galbraith (1):
sched: add avg_overlap decay
Peter Zijlstra (5):
sched: introduce avg_wakeup
sched: prefer wakers
sched: make plist a library facility
sched_clock: cleanups
sched: optimize ttwu vs group scheduling
Wang Chen (2):
sched, documentation: remove old O(1) scheduler document
sched: kill unused parameter of pick_next_task()
Documentation/scheduler/00-INDEX | 2 - Documentation/scheduler/sched-coding.txt | 126 ---- arch/x86/kernel/cpu/intel.c | 8 +- arch/x86/kernel/tsc.c | 9 +- include/linux/init_task.h | 1 + include/linux/latencytop.h | 10 +- include/linux/plist.h | 9 +- include/linux/sched.h | 17 +- init/Kconfig | 1 - kernel/latencytop.c | 83 +++- kernel/sched.c | 982 +++++++++++++++++++++--------- kernel/sched_clock.c | 30 +- kernel/sched_debug.c | 8 +- kernel/sched_fair.c | 59 ++- kernel/sched_features.h | 3 +- kernel/sched_rt.c | 537 ++++++++++++----- kernel/sched_stats.h | 7 +- lib/Kconfig | 6 - lib/Makefile | 4 +- lib/kernel_lock.c | 2 +- 20 files changed, 1262 insertions(+), 642 deletions(-) delete mode 100644 Documentation/scheduler/sched-coding.txt
Documentation/scheduler/00-INDEX +0 -2
@@ -2,8 +2,6 @@ - this file. sched-arch.txt - CPU Scheduler implementation hints for architecture specific code. -sched-coding.txt - - reference for various scheduler-related methods in the O(1) scheduler. sched-design-CFS.txt - goals, design and implementation of the Complete Fair Scheduler. sched-domains.txt
Documentation/scheduler/sched-coding.txt deleted +0 -126
126 lines not on this page · raw message
arch/x86/kernel/cpu/intel.c +7 -1
@@ -4,6 +4,7 @@#include <linux/string.h> #include <linux/bitops.h> #include <linux/smp.h> +#include <linux/sched.h> #include <linux/thread_info.h> #include <linux/module.h>
@@ -56,11 +57,16 @@ static void __cpuinit early_init_intel(struct cpuinfo_x86 *c)/* * c->x86_power is 8000_0007 edx. Bit 8 is TSC runs at constant rate -* with P/T states and does not stop in deep C-states +* with P/T states and does not stop in deep C-states. +* +* It is also reliable across cores and sockets. (but not across +* cabinets - we turn it off in that case explicitly.) */ if (c->x86_power & (1 <<8 )) {set_cpu_cap (c,X86_FEATURE_CONSTANT_TSC );set_cpu_cap (c,X86_FEATURE_NONSTOP_TSC ); +set_cpu_cap (c,X86_FEATURE_TSC_RELIABLE ); + sched_clock_stable =1 ; } }
arch/x86/kernel/tsc.c +5 -4
@@ -17,20 +17,21 @@#include <asm/delay.h> #include <asm/hypervisor.h> -unsigned int cpu_khz;/* TSC clocks / usec, not used here */ +unsigned int __read_mostly cpu_khz;/* TSC clocks / usec, not used here */ EXPORT_SYMBOL (cpu_khz); -unsigned int tsc_khz; + +unsigned int __read_mostly tsc_khz;EXPORT_SYMBOL (tsc_khz);/* * TSC can be unstable due to cpufreq or due to unsynced TSCs */ -static int tsc_unstable; +static int __read_mostly tsc_unstable;/* native_sched_clock() is called before tsc_init(), so we must start with the TSC soft disabled to prevent erroneous rdtsc usage on !cpu_has_tsc processors */ -static int tsc_disabled =-1 ; +static int __read_mostly tsc_disabled =-1 ;static int tsc_clocksource_reliable;/*
include/linux/init_task.h +1 -0
@@ -147,6 +147,7 @@ extern struct cred init_cred; .nr_cpus_allowed = NR_CPUS, \ }, \ .tasks = LIST_HEAD_INIT(tsk.tasks), \ + .pushable_tasks = PLIST_NODE_INIT(tsk.pushable_tasks, MAX_PRIO), \ .ptraced = LIST_HEAD_INIT(tsk.ptraced), \ .ptrace_entry = LIST_HEAD_INIT(tsk.ptrace_entry), \ .real_parent = &tsk, \
include/linux/latencytop.h +9 -1
@@ -9,6 +9,7 @@#ifndef _INCLUDE_GUARD_LATENCYTOP_H_#define _INCLUDE_GUARD_LATENCYTOP_H_ +#include <linux/compiler.h> #ifdef CONFIG_LATENCYTOP #define LT_SAVECOUNT 32
@@ -24,7 +25,14 @@ struct latency_record {struct task_struct ; -void account_scheduler_latency (struct task_struct *task,int usecs,int inter); +extern int latencytop_enabled; +void __account_scheduler_latency (struct task_struct *task,int usecs,int inter); +static inline void +account_scheduler_latency (struct task_struct *task,int usecs,int inter) +{ +if (unlikely (latencytop_enabled)) +__account_scheduler_latency (task, usecs, inter); +}void clear_all_latency_tracing (struct task_struct *p);
include/linux/plist.h +6 -3
@@ -96,6 +96,10 @@ struct plist_node {# define PLIST_HEAD_LOCK_INIT (_lock)#endif +#define _PLIST_HEAD_INIT (head) \ + .prio_list = LIST_HEAD_INIT((head).prio_list), \ + .node_list = LIST_HEAD_INIT((head).node_list) +/** * PLIST_HEAD_INIT - static struct plist_head initializer * @head: struct plist_head variable name
@@ -103,8 +107,7 @@ struct plist_node {*/ #define PLIST_HEAD_INIT (head, _lock) \ { \ - .prio_list = LIST_HEAD_INIT((head).prio_list), \ - .node_list = LIST_HEAD_INIT((head).node_list), \ + _PLIST_HEAD_INIT(head), \ PLIST_HEAD_LOCK_INIT(&(_lock)) \ }
@@ -116,7 +119,7 @@ struct plist_node {#define PLIST_NODE_INIT (node, __prio) \ { \ .prio = (__prio), \ - .plist = PLIST_HEAD_INIT((node).plist, NULL), \ + .plist = { _PLIST_HEAD_INIT((node).plist) }, \ }/**
include/linux/sched.h +16 -1
@@ -998,6 +998,7 @@ struct sched_class { struct rq *busiest, struct sched_domain *sd, enum cpu_idle_type idle);void (*pre_schedule) (struct rq *this_rq,struct task_struct *task); +int (*needs_post_schedule) (struct rq *this_rq);void (*post_schedule) (struct rq *this_rq);void (*task_wake_up) (struct rq *this_rq,struct task_struct *task);
@@ -1052,6 +1053,10 @@ struct sched_entity {u64 last_wakeup;u64 avg_overlap; +u64 start_runtime; +u64 avg_wakeup; +u64 nr_migrations; +#ifdef CONFIG_SCHEDSTATS u64 wait_start;u64 wait_max;
@@ -1067,7 +1072,6 @@ struct sched_entity {u64 exec_max;u64 slice_max; -u64 nr_migrations;u64 nr_migrations_cold;u64 nr_failed_migrations_affine;u64 nr_failed_migrations_running;
@@ -1164,6 +1168,7 @@ struct task_struct {#endif struct list_head tasks; +struct plist_node pushable_tasks;struct mm_struct *mm, *active_mm;
@@ -1673,6 +1678,16 @@ static inline int set_cpus_allowed(struct task_struct *p, cpumask_t new_mask)return set_cpus_allowed_ptr (p, &new_mask); } +/* +* Architectures can set this to 1 if they have specified +* CONFIG_HAVE_UNSTABLE_SCHED_CLOCK in their arch Kconfig, +* but then during bootup it turns out that sched_clock() +* is reliable after all: +*/ +#ifdef CONFIG_HAVE_UNSTABLE_SCHED_CLOCK +extern int sched_clock_stable; +#endif +extern unsigned long long sched_clock (void );extern void sched_clock_init (void );
init/Kconfig +0 -1
@@ -966,7 +966,6 @@ config SLABINFO config RT_MUTEXES boolean - select PLIST config BASE_SMALL int
kernel/latencytop.c +71 -12
145 lines not on this page · raw message
kernel/sched.c +692 -290
1,297 lines not on this page · raw message
kernel/sched_clock.c +17 -13
@@ -24,11 +24,11 @@* The clock: sched_clock_cpu() is monotonic per cpu, and should be somewhat * consistent between cpus (never more than 2 jiffies difference). */ -#include <linux/sched.h> -#include <linux/percpu.h> #include <linux/spinlock.h> -#include <linux/ktime.h> #include <linux/module.h> +#include <linux/percpu.h> +#include <linux/ktime.h> +#include <linux/sched.h> /* * Scheduler clock - returns current time in nanosec units.
@@ -43,6 +43,7 @@ unsigned long long __attribute__((weak)) sched_clock(void)static __read_mostly int sched_clock_running;#ifdef CONFIG_HAVE_UNSTABLE_SCHED_CLOCK +__read_mostly int sched_clock_stable; struct sched_clock_data {/*
@@ -87,7 +88,7 @@ void sched_clock_init(void) }/* -* min,max except they take wrapping into account +* min, max except they take wrapping into account */ static inline u64 wrap_min (u64 x,u64 y)
@@ -111,15 +112,13 @@ static u64 __update_sched_clock(struct sched_clock_data *scd, u64 now)s64 delta = now - scd->tick_raw;u64 clock, min_clock, max_clock; -WARN_ON_ONCE (!irqs_disabled ()); -if (unlikely (delta <0 )) delta =0 ;/* * scd->clock = clamp(scd->tick_gtod + delta, -* max(scd->tick_gtod, scd->clock), -* scd->tick_gtod + TICK_NSEC); +* max(scd->tick_gtod, scd->clock), +* scd->tick_gtod + TICK_NSEC); */ clock = scd->tick_gtod + delta;
@@ -148,12 +147,13 @@ static void lock_double_clock(struct sched_clock_data *data1,u64 sched_clock_cpu (int cpu) { -struct sched_clock_data *scd =cpu_sdc (cpu);u64 now, clock, this_clock, remote_clock; +struct sched_clock_data *scd; -if (unlikely (!sched_clock_running)) -return 0ull ; +if (sched_clock_stable) +return sched_clock (); + scd =cpu_sdc (cpu);WARN_ON_ONCE (!irqs_disabled ()); now =sched_clock ();
@@ -195,14 +195,18 @@ u64 sched_clock_cpu(int cpu)void sched_clock_tick (void ) { -struct sched_clock_data *scd =this_scd (); +struct sched_clock_data *scd;u64 now, now_gtod; +if (sched_clock_stable) +return ; +if (unlikely (!sched_clock_running))return ;WARN_ON_ONCE (!irqs_disabled ()); + scd =this_scd (); now_gtod =ktime_to_ns (ktime_get ()); now =sched_clock ();
@@ -250,7 +254,7 @@ u64 sched_clock_cpu(int cpu) return sched_clock(); } -#endif +#endif /* CONFIG_HAVE_UNSTABLE_SCHED_CLOCK */ unsigned long long cpu_clock (int cpu) {
kernel/sched_debug.c +3 -5
@@ -272,7 +272,6 @@ static void print_cpu(struct seq_file *m, int cpu)P (nr_switches);P (nr_load_updates);P (nr_uninterruptible); -SEQ_printf (m," .%-30s: %lu\n" ,"jiffies" , jiffies);PN (next_balance);P (curr->pid);PN (clock);
@@ -287,9 +286,6 @@ static void print_cpu(struct seq_file *m, int cpu)#ifdef CONFIG_SCHEDSTATS #define P (n) SEQ_printf(m, " .%-30s: %d\n", #n, rq->n); -P (yld_exp_empty); -P (yld_act_empty); -P (yld_both_empty);P (yld_count);P (sched_switch);
@@ -314,7 +310,7 @@ static int sched_debug_show(struct seq_file *m, void *v)u64 now =ktime_to_ns (ktime_get ());int cpu; -SEQ_printf (m,"Sched Debug Version: v0.08, %s %.*s\n" , +SEQ_printf (m,"Sched Debug Version: v0.09, %s %.*s\n" ,init_utsname ()->release, (int )strcspn (init_utsname ()->version," " ),init_utsname ()->version);
@@ -325,6 +321,7 @@ static int sched_debug_show(struct seq_file *m, void *v) SEQ_printf(m, " .%-40s: %Ld\n", #x, (long long)(x))#define PN (x) \ SEQ_printf(m, " .%-40s: %Ld.%06ld\n", #x, SPLIT_NS(x)) +P (jiffies);PN (sysctl_sched_latency);PN (sysctl_sched_min_granularity);PN (sysctl_sched_wakeup_granularity);
@@ -397,6 +394,7 @@ void proc_sched_show_task(struct task_struct *p, struct seq_file *m)PN (se.vruntime);PN (se.sum_exec_runtime);PN (se.avg_overlap); +PN (se.avg_wakeup); nr_switches = p->nvcsw + p->nivcsw;
kernel/sched_fair.c +53 -6
76 lines not on this page · raw message
kernel/sched_features.h +2 -1
@@ -1,5 +1,6 @@ SCHED_FEAT(NEW_FAIR_SLEEPERS, 1) -SCHED_FEAT(NORMALIZED_SLEEPER, 1) +SCHED_FEAT(NORMALIZED_SLEEPER, 0) +SCHED_FEAT(ADAPTIVE_GRAN, 1) SCHED_FEAT(WAKEUP_PREEMPT, 1) SCHED_FEAT(START_DEBIT, 1)SCHED_FEAT (AFFINE_WAKEUPS ,1 )
kernel/sched_rt.c +374 -163
776 lines not on this page · raw message
kernel/sched_stats.h +3 -4
@@ -4,7 +4,7 @@* bump this up when changing the output format or the meaning of an existing * format, so that tools can adapt (or abort) */ -#define SCHEDSTAT_VERSION 14 +#define SCHEDSTAT_VERSION 15static int show_schedstat (struct seq_file *seq,void *v) {
@@ -26,9 +26,8 @@ static int show_schedstat(struct seq_file *seq, void *v)/* runqueue-specific stats */ seq_printf(seq, - "cpu%d %u %u %u %u %u %u %u %u %u %llu %llu %lu", - cpu, rq->yld_both_empty, - rq->yld_act_empty, rq->yld_exp_empty, rq->yld_count, + "cpu%d %u %u %u %u %u %u %llu %llu %lu", + cpu, rq->yld_count, rq->sched_switch, rq->sched_count, rq->sched_goidle, rq->ttwu_count, rq->ttwu_local, rq->rq_cpu_time,
lib/Kconfig +0 -6
@@ -136,12 +136,6 @@ config TEXTSEARCH_BM config TEXTSEARCH_FSM tristate -# -# plist support is select#ed if needed -# -config PLIST - boolean - config HAS_IOMEM boolean depends on !NO_IOMEM
lib/Makefile +2 -2
@@ -11,7 +11,8 @@ lib-y := ctype.o string.o vsprintf.o cmdline.o \ rbtree.o radix-tree.o dump_stack.o \ idr.o int_sqrt.o extable.o prio_tree.o \ sha1.o irq_regs.o reciprocal_div.o argv_split.o \ - proportions.o prio_heap.o ratelimit.o show_mem.o is_single_threaded.o + proportions.o prio_heap.o ratelimit.o show_mem.o \ + is_single_threaded.o plist.o lib-$(CONFIG_MMU) += ioremap.o lib-$(CONFIG_SMP) += cpumask.o
@@ -40,7 +41,6 @@ lib-$(CONFIG_GENERIC_FIND_NEXT_BIT) += find_next_bit.o lib-$(CONFIG_GENERIC_FIND_LAST_BIT) += find_last_bit.o obj-$(CONFIG_GENERIC_HWEIGHT) += hweight.o obj-$(CONFIG_LOCK_KERNEL) += kernel_lock.o -obj-$(CONFIG_PLIST) += plist.o obj-$(CONFIG_DEBUG_PREEMPT) += smp_processor_id.o obj-$(CONFIG_DEBUG_LIST) += list_debug.o obj-$(CONFIG_DEBUG_OBJECTS) += debugobjects.o
lib/kernel_lock.c +1 -1
@@ -39,7 +39,7 @@ static __cacheline_aligned_in_smp DEFINE_SPINLOCK(kernel_flag); int __lockfunc __reacquire_kernel_lock(void) {while (!_raw_spin_trylock (&kernel_flag)) { -if (test_thread_flag (TIF_NEED_RESCHED )) +if (need_resched ())return -EAGAIN ;cpu_relax (); }