scheduler changes for v3.3
Linus,
Please pull the latest sched-core-for-linus git tree from:
git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git sched-core-for-linus
Thanks,
Ingo
------------------>
Andrew Vagin (2):
sched: Don't call task_group() too many times in set_task_rq() events, sched: Add tracepoint for accounting blocked time
Arun Sharma (1):
sched/tracing: Add a new tracepoint for sleeptime
Daisuke Nishimura (3):
sched: Fix cgroup movement of forking process
sched: Fix cgroup movement of newly created process
sched: Fix cgroup movement of waking process
Dave Jones (1):
sched: Disable scheduler warnings during oopses
Glauber Costa (6):
sched, trivial: Initialize root cgroup's sibling list sched/accounting: Change cpustat fields to an array sched/accounting, cgroups: Reuse cgroup's parent pointer sched/accounting: Re-use scheduler statistics for the root cgroup sched/accounting: Fix user/system tick double accounting sched/accounting: Fix parameter passing in task_group_account_field
Kamalesh Babulal (1):
sched: Remove cfs bandwidth period check in tg_set_cfs_period()
Kees Cook (2):
sched: Mark parent and real_parent as __rcu
sched: Add missing rcu_dereference() around ->real_parent usage
Martin Schwidefsky (2):
[S390] cputime: add sparse checking and cleanup [S390] fix cputime overflow in uptime_proc_show
Mike Galbraith (3):
sched: Use rt.nr_cpus_allowed to recover select_task_rq() cycles
sched: Set skip_clock_update in yield_task_fair()
sched: Save some hrtick_start_fair cycles
Paul Turner (1):
sched: Use jump labels to reduce overhead when bandwidth control is inactive
Peter Zijlstra (8):
sched: Make separate sched*.c translation units
sched: Move all scheduler bits into kernel/sched/
sched: Fix compile error for UP,!NOHZ
sched: Use jump_labels for sched_feat
sched, nohz: Fix missing RCU read lock
sched: Only queue remote wakeups when crossing cache boundaries
sched: Replace all_pinned with a generic flags field
sched: Fix load-balance lock-breaking
Richard Weinberger (1):
sched: Fix comment for requeue_rt_entity
Shan Hai (1):
sched/rt: Code cleanup, remove a redundant function call
Suresh Siddha (8):
sched: Clean up domain traversal in select_idle_sibling()
sched: Fix the sched group node allocation for SD_OVERLAP domains sched, nohz: Introduce nohz_flags in 'struct rq'
sched, nohz: Track nr_busy_cpus in the sched_group_power sched, nohz: Implement sched group, domain aware nohz idle load balancing sched, nohz: Clean up the find_new_ilb() using sched groups nr_busy_cpus sched, nohz: Fix the idle cpu check in nohz_idle_balance sched, nohz: Set the NOHZ_BALANCE_KICK flag for idle load balancer
arch/ia64/include/asm/cputime.h | 69 +-
arch/powerpc/include/asm/cputime.h | 70 +-
arch/s390/appldata/appldata_os.c | 16 +-
arch/s390/include/asm/cputime.h | 140 +-
arch/x86/include/asm/i387.h | 2 +-
drivers/cpufreq/cpufreq_conservative.c | 50 +-
drivers/cpufreq/cpufreq_ondemand.c | 54 +-
drivers/cpufreq/cpufreq_stats.c | 5 +-
drivers/macintosh/rack-meter.c | 14 +-
fs/proc/array.c | 8 +-
fs/proc/stat.c | 63 +-
fs/proc/uptime.c | 11 +-
include/asm-generic/cputime.h | 62 +-
include/linux/kernel_stat.h | 36 +-
include/linux/latencytop.h | 3 +-
include/linux/sched.h | 23 +-
include/trace/events/sched.h | 57 +
kernel/Makefile | 20 +-
kernel/acct.c | 4 +-
kernel/cpu.c | 3 +-
kernel/exit.c | 22 +-
kernel/fork.c | 14 +-
kernel/itimer.c | 15 +-
kernel/posix-cpu-timers.c | 132 +-
kernel/sched/Makefile | 20 +
kernel/{sched_autogroup.c => sched/auto_group.c} | 33 +-
kernel/{sched_autogroup.h => sched/auto_group.h} | 26 +-
kernel/{sched_clock.c => sched/clock.c} | 0
kernel/{sched.c => sched/core.c} | 2247 +++-------------------
kernel/{sched_cpupri.c => sched/cpupri.c} | 4 +-
kernel/{sched_cpupri.h => sched/cpupri.h} | 0
kernel/{sched_debug.c => sched/debug.c} | 6 +-
kernel/{sched_fair.c => sched/fair.c} | 998 ++++++++---
kernel/{sched_features.h => sched/features.h} | 30 +-
kernel/{sched_idletask.c => sched/idle_task.c} | 4 +-
kernel/{sched_rt.c => sched/rt.c} | 218 ++-
kernel/sched/sched.h | 1166 +++++++++++
kernel/sched/stats.c | 111 ++
kernel/{sched_stats.h => sched/stats.h} | 109 +-
kernel/{sched_stoptask.c => sched/stop_task.c} | 4 +-
kernel/signal.c | 6 +-
kernel/sys.c | 6 +-
kernel/time/tick-sched.c | 9 +
kernel/tsacct.c | 2 +-
44 files changed, 3076 insertions(+), 2816 deletions(-)
create mode 100644 kernel/sched/Makefile
rename kernel/{sched_autogroup.c => sched/auto_group.c} (88%)
rename kernel/{sched_autogroup.h => sched/auto_group.h} (66%)
rename kernel/{sched_clock.c => sched/clock.c} (100%)
rename kernel/{sched.c => sched/core.c} (79%)
rename kernel/{sched_cpupri.c => sched/cpupri.c} (99%)
rename kernel/{sched_cpupri.h => sched/cpupri.h} (100%)
rename kernel/{sched_debug.c => sched/debug.c} (99%)
rename kernel/{sched_fair.c => sched/fair.c} (87%)
rename kernel/{sched_features.h => sched/features.h} (75%)
rename kernel/{sched_idletask.c => sched/idle_task.c} (96%)
rename kernel/{sched_rt.c => sched/rt.c} (90%)
create mode 100644 kernel/sched/sched.h
create mode 100644 kernel/sched/stats.c
rename kernel/{sched_stats.h => sched/stats.h} (70%)
rename kernel/{sched_stoptask.c => sched/stop_task.c} (97%)arch/ia64/include/asm/cputime.h +32 -37
@@ -26,59 +26,51 @@#include <linux/jiffies.h> #include <asm/processor.h> -typedef u64 cputime_t ; -typedef u64 cputime64_t ; +typedef u64 __nocast cputime_t ; +typedef u64 __nocast cputime64_t ; -#define cputime_zero ((cputime_t)0)#define cputime_one_jiffy jiffies_to_cputime(1) -#define cputime_max ((~((cputime_t)0) >> 1) - 1) -#define cputime_add (__a, __b) ((__a) + (__b)) -#define cputime_sub (__a, __b) ((__a) - (__b)) -#define cputime_div (__a, __n) ((__a) / (__n)) -#define cputime_halve (__a) ((__a) >> 1) -#define cputime_eq (__a, __b) ((__a) == (__b)) -#define cputime_gt (__a, __b) ((__a) > (__b)) -#define cputime_ge (__a, __b) ((__a) >= (__b)) -#define cputime_lt (__a, __b) ((__a) < (__b)) -#define cputime_le (__a, __b) ((__a) <= (__b)) - -#define cputime64_zero ((cputime64_t)0) -#define cputime64_add (__a, __b) ((__a) + (__b)) -#define cputime64_sub (__a, __b) ((__a) - (__b)) -#define cputime_to_cputime64 (__ct) (__ct)/* * Convert cputime <-> jiffies (HZ) */ -#define cputime_to_jiffies (__ct) ((__ct) / (NSEC_PER_SEC / HZ)) -#define jiffies_to_cputime (__jif) ((__jif) * (NSEC_PER_SEC / HZ)) -#define cputime64_to_jiffies64 (__ct) ((__ct) / (NSEC_PER_SEC / HZ)) -#define jiffies64_to_cputime64 (__jif) ((__jif) * (NSEC_PER_SEC / HZ)) +#define cputime_to_jiffies (__ct) \ + ((__force u64)(__ct) / (NSEC_PER_SEC / HZ)) +#define jiffies_to_cputime (__jif) \ + (__force cputime_t)((__jif) * (NSEC_PER_SEC / HZ)) +#define cputime64_to_jiffies64 (__ct) \ + ((__force u64)(__ct) / (NSEC_PER_SEC / HZ)) +#define jiffies64_to_cputime64 (__jif) \ + (__force cputime64_t)((__jif) * (NSEC_PER_SEC / HZ))/* * Convert cputime <-> microseconds */ -#define cputime_to_usecs (__ct) ((__ct) / NSEC_PER_USEC) -#define usecs_to_cputime (__usecs) ((__usecs) * NSEC_PER_USEC) +#define cputime_to_usecs (__ct) \ + ((__force u64)(__ct) / NSEC_PER_USEC) +#define usecs_to_cputime (__usecs) \ + (__force cputime_t)((__usecs) * NSEC_PER_USEC)/* * Convert cputime <-> seconds */ -#define cputime_to_secs (__ct) ((__ct) / NSEC_PER_SEC) -#define secs_to_cputime (__secs) ((__secs) * NSEC_PER_SEC) +#define cputime_to_secs (__ct) \ + ((__force u64)(__ct) / NSEC_PER_SEC) +#define secs_to_cputime (__secs) \ + (__force cputime_t)((__secs) * NSEC_PER_SEC)/* * Convert cputime <-> timespec (nsec) */ static inline cputime_t timespec_to_cputime (const struct timespec *val) { -cputime_t ret = val->tv_sec *NSEC_PER_SEC ; -return (ret + val->tv_nsec); +u64 ret = val->tv_sec *NSEC_PER_SEC + val->tv_nsec; +return (__force cputime_t ) ret; }static inline void cputime_to_timespec (const cputime_t ct,struct timespec *val) { - val->tv_sec = ct /NSEC_PER_SEC ; - val->tv_nsec = ct %NSEC_PER_SEC ; + val->tv_sec = (__force u64 ) ct /NSEC_PER_SEC ; + val->tv_nsec = (__force u64 ) ct %NSEC_PER_SEC ; }/*
@@ -86,25 +78,28 @@ static inline void cputime_to_timespec(const cputime_t ct, struct timespec *val)*/ static inline cputime_t timeval_to_cputime (struct timeval *val) { -cputime_t ret = val->tv_sec *NSEC_PER_SEC ; -return (ret + val->tv_usec *NSEC_PER_USEC ); +u64 ret = val->tv_sec *NSEC_PER_SEC + val->tv_usec *NSEC_PER_USEC ; +return (__force cputime_t ) ret; }static inline void cputime_to_timeval (const cputime_t ct,struct timeval *val) { - val->tv_sec = ct /NSEC_PER_SEC ; - val->tv_usec = (ct %NSEC_PER_SEC ) /NSEC_PER_USEC ; + val->tv_sec = (__force u64 ) ct /NSEC_PER_SEC ; + val->tv_usec = ((__force u64 ) ct %NSEC_PER_SEC ) /NSEC_PER_USEC ; }/* * Convert cputime <-> clock (USER_HZ) */ -#define cputime_to_clock_t (__ct) ((__ct) / (NSEC_PER_SEC / USER_HZ)) -#define clock_t_to_cputime (__x) ((__x) * (NSEC_PER_SEC / USER_HZ)) +#define cputime_to_clock_t (__ct) \ + ((__force u64)(__ct) / (NSEC_PER_SEC / USER_HZ)) +#define clock_t_to_cputime (__x) \ + (__force cputime_t)((__x) * (NSEC_PER_SEC / USER_HZ))/* * Convert cputime64 to clock. */ -#define cputime64_to_clock_t (__ct) cputime_to_clock_t((cputime_t)__ct) +#define cputime64_to_clock_t (__ct) \ + cputime_to_clock_t((__force cputime_t)__ct)#endif /* CONFIG_VIRT_CPU_ACCOUNTING */ #endif /* __IA64_CPUTIME_H */
arch/powerpc/include/asm/cputime.h +27 -43
187 lines not on this page · raw message
arch/s390/appldata/appldata_os.c +8 -8
@@ -115,21 +115,21 @@ static void appldata_get_os_data(void *data) j =0 ;for_each_online_cpu (i) { os_data->os_cpu[j].per_cpu_user = -cputime_to_jiffies (kstat_cpu (i).cpustat.user); +cputime_to_jiffies (kcpustat_cpu (i).cpustat[CPUTIME_USER ]); os_data->os_cpu[j].per_cpu_nice = -cputime_to_jiffies (kstat_cpu (i).cpustat.nice); +cputime_to_jiffies (kcpustat_cpu (i).cpustat[CPUTIME_NICE ]); os_data->os_cpu[j].per_cpu_system = -cputime_to_jiffies (kstat_cpu (i).cpustat.system); +cputime_to_jiffies (kcpustat_cpu (i).cpustat[CPUTIME_SYSTEM ]); os_data->os_cpu[j].per_cpu_idle = -cputime_to_jiffies (kstat_cpu (i).cpustat.idle); +cputime_to_jiffies (kcpustat_cpu (i).cpustat[CPUTIME_IDLE ]); os_data->os_cpu[j].per_cpu_irq = -cputime_to_jiffies (kstat_cpu (i).cpustat.irq); +cputime_to_jiffies (kcpustat_cpu (i).cpustat[CPUTIME_IRQ ]); os_data->os_cpu[j].per_cpu_softirq = -cputime_to_jiffies (kstat_cpu (i).cpustat.softirq); +cputime_to_jiffies (kcpustat_cpu (i).cpustat[CPUTIME_SOFTIRQ ]); os_data->os_cpu[j].per_cpu_iowait = -cputime_to_jiffies (kstat_cpu (i).cpustat.iowait); +cputime_to_jiffies (kcpustat_cpu (i).cpustat[CPUTIME_IOWAIT ]); os_data->os_cpu[j].per_cpu_steal = -cputime_to_jiffies (kstat_cpu (i).cpustat.steal); +cputime_to_jiffies (kcpustat_cpu (i).cpustat[CPUTIME_STEAL ]); os_data->os_cpu[j].cpu_id = i; j++; }
arch/s390/include/asm/cputime.h +63 -77
227 lines not on this page · raw message
arch/x86/include/asm/i387.h +1 -1
@@ -218,7 +218,7 @@ static inline void fpu_fxsave(struct fpu *fpu)#ifdef CONFIG_SMP #define safe_address (__per_cpu_offset[0])#else -#define safe_address (kstat_cpu(0).cpustat.user) +#define safe_address (__get_cpu_var(kernel_cpustat).cpustat[CPUTIME_USER])#endif /*
drivers/cpufreq/cpufreq_conservative.c +24 -26
95 lines not on this page · raw message
drivers/cpufreq/cpufreq_ondemand.c +26 -28
101 lines not on this page · raw message
drivers/cpufreq/cpufreq_stats.c +2 -3
@@ -61,9 +61,8 @@ static int cpufreq_stats_update(unsigned int cpu)spin_lock (&cpufreq_stats_lock); stat =per_cpu (cpufreq_stats_table, cpu);if (stat->time_in_state) - stat->time_in_state[stat->last_index] = -cputime64_add (stat->time_in_state[stat->last_index], -cputime_sub (cur_time, stat->last_time)); + stat->time_in_state[stat->last_index] += + cur_time - stat->last_time; stat->last_time = cur_time;spin_unlock (&cpufreq_stats_lock);return 0 ;
drivers/macintosh/rack-meter.c +6 -8
@@ -81,13 +81,13 @@ static int rackmeter_ignore_nice;*/ static inline cputime64_t get_cpu_idle_time (unsigned int cpu) { -cputime64_t retval; +u64 retval; - retval =cputime64_add (kstat_cpu (cpu).cpustat.idle, -kstat_cpu (cpu).cpustat.iowait); + retval =kcpustat_cpu (cpu).cpustat[CPUTIME_IDLE ] + +kcpustat_cpu (cpu).cpustat[CPUTIME_IOWAIT ];if (rackmeter_ignore_nice) - retval =cputime64_add (retval,kstat_cpu (cpu).cpustat.nice); + retval +=kcpustat_cpu (cpu).cpustat[CPUTIME_NICE ];return retval; }
@@ -220,13 +220,11 @@ static void rackmeter_do_timer(struct work_struct *work)int i, offset, load, cumm, pause; cur_jiffies =jiffies64_to_cputime64 (get_jiffies_64 ()); - total_ticks = (unsigned int )cputime64_sub (cur_jiffies, - rcpu->prev_wall); + total_ticks = (unsigned int ) (cur_jiffies - rcpu->prev_wall); rcpu->prev_wall = cur_jiffies; total_idle_ticks =get_cpu_idle_time (cpu); - idle_ticks = (unsigned int )cputime64_sub (total_idle_ticks, - rcpu->prev_idle); + idle_ticks = (unsigned int ) (total_idle_ticks - rcpu->prev_idle); rcpu->prev_idle = total_idle_ticks;/* We do a very dumb calculation to update the LEDs for now,
fs/proc/array.c +4 -4
@@ -394,8 +394,8 @@ static int do_task_stat(struct seq_file *m, struct pid_namespace *ns,sigemptyset (&sigign);sigemptyset (&sigcatch); - cutime = cstime = utime = stime = cputime_zero; - cgtime = gtime = cputime_zero; + cutime = cstime = utime = stime =0 ; + cgtime = gtime =0 ;if (lock_task_sighand (task, &flags)) {struct signal_struct *sig = task->signal;
@@ -423,14 +423,14 @@ static int do_task_stat(struct seq_file *m, struct pid_namespace *ns,do { min_flt += t->min_flt; maj_flt += t->maj_flt; - gtime =cputime_add (gtime, t->gtime); + gtime += t->gtime; t =next_thread (t); }while (t != task); min_flt += sig->min_flt; maj_flt += sig->maj_flt;thread_group_times (task, &utime, &stime); - gtime =cputime_add (gtime, sig->gtime); + gtime += sig->gtime; } sid =task_session_nr_ns (task, ns);
fs/proc/stat.c +29 -34
107 lines not on this page · raw message
fs/proc/uptime.c +8 -3
@@ -11,15 +11,20 @@ static int uptime_proc_show(struct seq_file *m, void *v) {struct timespec uptime;struct timespec idle; +u64 idletime; +u64 nsec; +u32 rem;int i; -cputime_t idletime = cputime_zero; + idletime =0 ;for_each_possible_cpu (i) - idletime =cputime64_add (idletime,kstat_cpu (i).cpustat.idle); + idletime += (__force u64 )kcpustat_cpu (i).cpustat[CPUTIME_IDLE ];do_posix_clock_monotonic_gettime (&uptime);monotonic_to_bootbased (&uptime); -cputime_to_timespec (idletime, &idle); + nsec =cputime64_to_jiffies64 (idletime) *TICK_NSEC ; + idle.tv_sec =div_u64_rem (nsec,NSEC_PER_SEC , &rem); + idle.tv_nsec = rem; seq_printf(m, "%lu.%02lu %lu.%02lu\n", (unsigned long) uptime.tv_sec, (uptime.tv_nsec / (NSEC_PER_SEC / 100)),
include/asm-generic/cputime.h +28 -34
98 lines not on this page · raw message
include/linux/kernel_stat.h +22 -14
@@ -6,6 +6,7 @@#include <linux/percpu.h> #include <linux/cpumask.h> #include <linux/interrupt.h> +#include <linux/sched.h> #include <asm/irq.h> #include <asm/cputime.h>
@@ -15,21 +16,25 @@* used by rstatd/perfmeter */ -struct cpu_usage_stat { -cputime64_t user; -cputime64_t nice; -cputime64_t system; -cputime64_t softirq; -cputime64_t irq; -cputime64_t idle; -cputime64_t iowait; -cputime64_t steal; -cputime64_t guest; -cputime64_t guest_nice; +enum cpu_usage_stat { +CPUTIME_USER , +CPUTIME_NICE , +CPUTIME_SYSTEM , +CPUTIME_SOFTIRQ , +CPUTIME_IRQ , +CPUTIME_IDLE , +CPUTIME_IOWAIT , +CPUTIME_STEAL , +CPUTIME_GUEST , +CPUTIME_GUEST_NICE , +NR_STATS , +}; + +struct kernel_cpustat { +u64 cpustat[NR_STATS ]; };struct kernel_stat { -struct cpu_usage_stat cpustat;#ifndef CONFIG_GENERIC_HARDIRQS unsigned int irqs[NR_IRQS ];#endif
@@ -38,10 +43,13 @@ struct kernel_stat { }; DECLARE_PER_CPU(struct kernel_stat, kstat); +DECLARE_PER_CPU(struct kernel_cpustat, kernel_cpustat); -#define kstat_cpu (cpu) per_cpu(kstat, cpu)/* Must have preemption disabled for this to be meaningful. */ -#define kstat_this_cpu __get_cpu_var(kstat) +#define kstat_this_cpu (&__get_cpu_var(kstat)) +#define kcpustat_this_cpu (&__get_cpu_var(kernel_cpustat)) +#define kstat_cpu (cpu) per_cpu(kstat, cpu) +#define kcpustat_cpu (cpu) per_cpu(kernel_cpustat, cpu)extern unsigned long long nr_context_switches (void );
include/linux/latencytop.h +2 -1
@@ -10,6 +10,8 @@#define _INCLUDE_GUARD_LATENCYTOP_H_#include <linux/compiler.h> +struct task_struct ; +#ifdef CONFIG_LATENCYTOP #define LT_SAVECOUNT 32
@@ -23,7 +25,6 @@ struct latency_record { }; -struct task_struct ;extern int latencytop_enabled;void __account_scheduler_latency (struct task_struct *task,int usecs,int inter);
include/linux/sched.h +19 -4
@@ -273,9 +273,11 @@ extern int runqueue_is_locked(int cpu);#if defined(CONFIG_SMP ) && defined(CONFIG_NO_HZ )extern void select_nohz_load_balancer (int stop_tick); +extern void set_cpu_sd_state_idle (void );extern int get_nohz_timer_target (void );#else static inline void select_nohz_load_balancer (int stop_tick) { } +static inline void set_cpu_sd_state_idle (void ) { }#endif /*
@@ -483,8 +485,8 @@ struct task_cputime {#define INIT_CPUTIME \ (struct task_cputime) { \ - .utime = cputime_zero, \ - .stime = cputime_zero, \ + .utime = 0, \ + .stime = 0, \ .sum_exec_runtime = 0, \ }
@@ -901,6 +903,10 @@ struct sched_group_power {* single CPU. */ unsigned int power, power_orig; +/* +* Number of busy cpus in this group. +*/ +atomic_t nr_busy_cpus; };struct sched_group {
@@ -925,6 +931,15 @@ static inline struct cpumask *sched_group_cpus(struct sched_group *sg)return to_cpumask (sg->cpumask); } +/** +* group_first_cpu - Returns the first cpu in the cpumask of a sched_group. +* @group: The group whose first cpu is to be returned. +*/ +static inline unsigned int group_first_cpu (struct sched_group *group) +{ +return cpumask_first (sched_group_cpus (group)); +} +struct sched_domain_attr {int relax_domain_level; };
@@ -1315,8 +1330,8 @@ struct task_struct {* older sibling, respectively. (p->father can be replaced with * p->real_parent->pid) */ -struct task_struct *real_parent;/* real parent process */ -struct task_struct *parent;/* recipient of SIGCHLD, wait4() reports */ +struct task_struct __rcu *real_parent;/* real parent process */ +struct task_struct __rcu *parent;/* recipient of SIGCHLD, wait4() reports */ /* * children/sibling forms the list of my natural children */
include/trace/events/sched.h +57 -0
69 lines not on this page · raw message
kernel/Makefile +5 -15
@@ -2,16 +2,15 @@ # Makefile for the linux kernel. # -obj-y = sched.o fork.o exec_domain.o panic.o printk.o \ +obj-y = fork.o exec_domain.o panic.o printk.o \ cpu.o exit.o itimer.o time.o softirq.o resource.o \ sysctl.o sysctl_binary.o capability.o ptrace.o timer.o user.o \ signal.o sys.o kmod.o workqueue.o pid.o \ rcupdate.o extable.o params.o posix-timers.o \ kthread.o wait.o kfifo.o sys_ni.o posix-cpu-timers.o mutex.o \ hrtimer.o rwsem.o nsproxy.o srcu.o semaphore.o \ - notifier.o ksysfs.o sched_clock.o cred.o \ - async.o range.o -obj-y += groups.o + notifier.o ksysfs.o cred.o \ + async.o range.o groups.o ifdef CONFIG_FUNCTION_TRACER # Do not trace debug files and internal ftrace files
@@ -20,10 +19,11 @@ CFLAGS_REMOVE_lockdep_proc.o = -pg CFLAGS_REMOVE_mutex-debug.o = -pg CFLAGS_REMOVE_rtmutex-debug.o = -pg CFLAGS_REMOVE_cgroup-debug.o = -pg -CFLAGS_REMOVE_sched_clock.o = -pg CFLAGS_REMOVE_irq_work.o = -pg endif +obj-y += sched/ + obj-$(CONFIG_FREEZER) += freezer.o obj-$(CONFIG_PROFILING) += profile.o obj-$(CONFIG_SYSCTL_SYSCALL_CHECK) += sysctl_check.o
@@ -99,7 +99,6 @@ obj-$(CONFIG_TRACING) += trace/ obj-$(CONFIG_X86_DS) += trace/ obj-$(CONFIG_RING_BUFFER) += trace/ obj-$(CONFIG_TRACEPOINTS) += trace/ -obj-$(CONFIG_SMP) += sched_cpupri.o obj-$(CONFIG_IRQ_WORK) += irq_work.o obj-$(CONFIG_CPU_PM) += cpu_pm.o
@@ -110,15 +109,6 @@ obj-$(CONFIG_PADATA) += padata.o obj-$(CONFIG_CRASH_DUMP) += crash_dump.o obj-$(CONFIG_JUMP_LABEL) += jump_label.o -ifneq ($(CONFIG_SCHED_OMIT_FRAME_POINTER),y) -# According to Alan Modra <[email protected]>, the -fno-omit-frame-pointer is -# needed for x86 only. Why this used to be enabled for all architectures is beyond -# me. I suspect most platforms don't need this, but until we know that for sure -# I turn this off for IA-64 only. Andreas Schwab says it's also needed on m68k -# to get a correct value for the wait-channel (WCHAN in ps). --davidm -CFLAGS_sched.o := $(PROFILING) -fno-omit-frame-pointer -endif - $(obj)/configs.o: $(obj)/config_data.h # config_data.h contains the same information as ikconfig.h but gzipped.
kernel/acct.c +2 -2
@@ -613,8 +613,8 @@ void acct_collect(long exitcode, int group_dead) pacct->ac_flag |=ACORE ;if (current->flags &PF_SIGNALED ) pacct->ac_flag |=AXSIG ; - pacct->ac_utime =cputime_add (pacct->ac_utime, current->utime); - pacct->ac_stime =cputime_add (pacct->ac_stime, current->stime); + pacct->ac_utime += current->utime; + pacct->ac_stime += current->stime; pacct->ac_minflt += current->min_flt; pacct->ac_majflt += current->maj_flt;spin_unlock_irq (¤t->sighand->siglock);
kernel/cpu.c +1 -2
@@ -178,8 +178,7 @@ static inline void check_for_tasks(int cpu)write_lock_irq (&tasklist_lock);for_each_process (p) {if (task_cpu (p) == cpu && p->state ==TASK_RUNNING && - (!cputime_eq (p->utime, cputime_zero) || - !cputime_eq (p->stime, cputime_zero))) + (p->utime || p->stime)) printk(KERN_WARNING "Task %s (pid = %d) is on cpu %d " "(state = %ld, flags = %x)\n", p->comm, task_pid_nr(p), cpu,
kernel/exit.c +6 -16
@@ -121,9 +121,9 @@ static void __exit_signal(struct task_struct *tsk)* We won't ever get here for the group leader, since it * will have been the last reference on the signal_struct. */ - sig->utime =cputime_add (sig->utime, tsk->utime); - sig->stime =cputime_add (sig->stime, tsk->stime); - sig->gtime =cputime_add (sig->gtime, tsk->gtime); + sig->utime += tsk->utime; + sig->stime += tsk->stime; + sig->gtime += tsk->gtime; sig->min_flt += tsk->min_flt; sig->maj_flt += tsk->maj_flt; sig->nvcsw += tsk->nvcsw;
@@ -1255,19 +1255,9 @@ static int wait_task_zombie(struct wait_opts *wo, struct task_struct *p)spin_lock_irq (&p->real_parent->sighand->siglock); psig = p->real_parent->signal; sig = p->signal; - psig->cutime = -cputime_add (psig->cutime, -cputime_add (tgutime, - sig->cutime)); - psig->cstime = -cputime_add (psig->cstime, -cputime_add (tgstime, - sig->cstime)); - psig->cgtime = -cputime_add (psig->cgtime, -cputime_add (p->gtime, -cputime_add (sig->gtime, - sig->cgtime))); + psig->cutime += tgutime + sig->cutime; + psig->cstime += tgstime + sig->cstime; + psig->cgtime += p->gtime + sig->gtime + sig->cgtime; psig->cmin_flt += p->min_flt + sig->min_flt + sig->cmin_flt; psig->cmaj_flt +=
kernel/fork.c +5 -9
@@ -1023,8 +1023,8 @@ void mm_init_owner(struct mm_struct *mm, struct task_struct *p)*/ static void posix_cpu_timers_init (struct task_struct *tsk) { - tsk->cputime_expires.prof_exp = cputime_zero; - tsk->cputime_expires.virt_exp = cputime_zero; + tsk->cputime_expires.prof_exp =0 ; + tsk->cputime_expires.virt_exp =0 ; tsk->cputime_expires.sched_exp =0 ;INIT_LIST_HEAD (&tsk->cpu_timers[0 ]);INIT_LIST_HEAD (&tsk->cpu_timers[1 ]);
@@ -1132,14 +1132,10 @@ static struct task_struct *copy_process(unsigned long clone_flags,init_sigpending (&p->pending); - p->utime = cputime_zero; - p->stime = cputime_zero; - p->gtime = cputime_zero; - p->utimescaled = cputime_zero; - p->stimescaled = cputime_zero; + p->utime = p->stime = p->gtime =0 ; + p->utimescaled = p->stimescaled =0 ;#ifndef CONFIG_VIRT_CPU_ACCOUNTING - p->prev_utime = cputime_zero; - p->prev_stime = cputime_zero; + p->prev_utime = p->prev_stime =0 ;#endif #if defined(SPLIT_RSS_COUNTING )memset (&p->rss_stat,0 ,sizeof (p->rss_stat));
kernel/itimer.c +7 -8
@@ -52,22 +52,22 @@ static void get_cpu_itimer(struct task_struct *tsk, unsigned int clock_id, cval = it->expires; cinterval = it->incr; -if (!cputime_eq (cval, cputime_zero)) { +if (cval) {struct task_cputime cputime;cputime_t t;thread_group_cputimer (tsk, &cputime);if (clock_id ==CPUCLOCK_PROF ) - t =cputime_add (cputime.utime, cputime.stime); + t = cputime.utime + cputime.stime;else /* CPUCLOCK_VIRT */ t = cputime.utime; -if (cputime_le (cval, t)) +if (cval < t)/* about to fire */ cval = cputime_one_jiffy;else - cval =cputime_sub (cval, t); + cval = cval - t; }spin_unlock_irq (&tsk->sighand->siglock);
@@ -161,10 +161,9 @@ static void set_cpu_itimer(struct task_struct *tsk, unsigned int clock_id, cval = it->expires; cinterval = it->incr; -if (!cputime_eq (cval, cputime_zero) || - !cputime_eq (nval, cputime_zero)) { -if (cputime_gt (nval, cputime_zero)) - nval =cputime_add (nval, cputime_one_jiffy); +if (cval || nval) { +if (nval >0 ) + nval += cputime_one_jiffy;set_process_cpu_timer (tsk, clock_id, &nval, &cval); } it->expires = nval;
kernel/posix-cpu-timers.c +52 -80
320 lines not on this page · raw message
kernel/sched/Makefile new file +20 -0
@@ -0,0 +1,20 @@ +ifdef CONFIG_FUNCTION_TRACER +CFLAGS_REMOVE_clock.o = -pg +endif + +ifneq ($(CONFIG_SCHED_OMIT_FRAME_POINTER),y) +# According to Alan Modra <[email protected]>, the -fno-omit-frame-pointer is +# needed for x86 only. Why this used to be enabled for all architectures is beyond +# me. I suspect most platforms don't need this, but until we know that for sure +# I turn this off for IA-64 only. Andreas Schwab says it's also needed on m68k +# to get a correct value for the wait-channel (WCHAN in ps). --davidm +CFLAGS_core.o := $(PROFILING) -fno-omit-frame-pointer +endif + +obj-y += core.o clock.o idle_task.o fair.o rt.o stop_task.o +obj-$(CONFIG_SMP) += cpupri.o +obj-$(CONFIG_SCHED_AUTOGROUP) += auto_group.o +obj-$(CONFIG_SCHEDSTATS) += stats.o +obj-$(CONFIG_SCHED_DEBUG) += debug.o + +
kernel/sched/auto_group.c renamed from kernel/sched_autogroup.c +8 -25
@@ -1,15 +1,19 @@#ifdef CONFIG_SCHED_AUTOGROUP +#include "sched.h" +#include <linux/proc_fs.h> #include <linux/seq_file.h> #include <linux/kallsyms.h> #include <linux/utsname.h> +#include <linux/security.h> +#include <linux/export.h> unsigned int __read_mostly sysctl_sched_autogroup_enabled =1 ;static struct autogroup autogroup_default;static atomic_t autogroup_seq_nr; -static void __init autogroup_init (struct task_struct *init_task) +void __init autogroup_init (struct task_struct *init_task) { autogroup_default.tg = &root_task_group;kref_init (&autogroup_default.kref);
@@ -17,7 +21,7 @@ static void __init autogroup_init(struct task_struct *init_task) init_task->signal->autogroup = &autogroup_default; } -static inline void autogroup_free (struct task_group *tg) +void autogroup_free (struct task_group *tg) {kfree (tg->autogroup); }
@@ -59,10 +63,6 @@ static inline struct autogroup *autogroup_task_get(struct task_struct *p)return ag; } -#ifdef CONFIG_RT_GROUP_SCHED -static void free_rt_sched_group (struct task_group *tg); -#endif -static inline struct autogroup *autogroup_create (void ) {struct autogroup *ag =kzalloc (sizeof (*ag),GFP_KERNEL );
@@ -108,8 +108,7 @@ out_fail:return autogroup_kref_get (&autogroup_default); } -static inline bool -task_wants_autogroup (struct task_struct *p,struct task_group *tg) +bool task_wants_autogroup (struct task_struct *p,struct task_group *tg) {if (tg != &root_task_group)return false;
@@ -127,22 +126,6 @@ task_wants_autogroup(struct task_struct *p, struct task_group *tg)return true; } -static inline bool task_group_is_autogroup (struct task_group *tg) -{ -return !!tg->autogroup; -} - -static inline struct task_group * -autogroup_task_group (struct task_struct *p,struct task_group *tg) -{ -int enabled =ACCESS_ONCE (sysctl_sched_autogroup_enabled); - -if (enabled &&task_wants_autogroup (p, tg)) -return p->signal->autogroup->tg; - -return tg; -} -static void autogroup_move_group (struct task_struct *p,struct autogroup *ag) {
@@ -263,7 +246,7 @@ out:#endif /* CONFIG_PROC_FS */ #ifdef CONFIG_SCHED_DEBUG -static inline int autogroup_path (struct task_group *tg,char *buf,int buflen) +int autogroup_path (struct task_group *tg,char *buf,int buflen) {if (!task_group_is_autogroup (tg))return 0 ;
kernel/sched/auto_group.h renamed from kernel/sched_autogroup.h +24 -2
@@ -1,5 +1,8 @@#ifdef CONFIG_SCHED_AUTOGROUP +#include <linux/kref.h> +#include <linux/rwsem.h> + struct autogroup {/* * reference doesn't mean how many thread attach to this
@@ -13,9 +16,28 @@ struct autogroup {int nice; }; -static inline bool task_group_is_autogroup (struct task_group *tg); +extern void autogroup_init (struct task_struct *init_task); +extern void autogroup_free (struct task_group *tg); + +static inline bool task_group_is_autogroup (struct task_group *tg) +{ +return !!tg->autogroup; +} + +extern bool task_wants_autogroup (struct task_struct *p,struct task_group *tg); +static inline struct task_group * -autogroup_task_group (struct task_struct *p,struct task_group *tg); +autogroup_task_group (struct task_struct *p,struct task_group *tg) +{ +int enabled =ACCESS_ONCE (sysctl_sched_autogroup_enabled); + +if (enabled &&task_wants_autogroup (p, tg)) +return p->signal->autogroup->tg; + +return tg; +} + +extern int autogroup_path (struct task_group *tg,char *buf,int buflen);#else /* !CONFIG_SCHED_AUTOGROUP */
kernel/sched/clock.c renamed from kernel/sched_clock.c +0 -0
kernel/sched/core.c renamed from kernel/sched.c +307 -1940
1,114 lines not on this page · raw message
kernel/sched/cpupri.c renamed from kernel/sched_cpupri.c +2 -2
Lines of this file are not shown. View the raw message.
kernel/sched/cpupri.h renamed from kernel/sched_cpupri.h +0 -0
kernel/sched/debug.c renamed from kernel/sched_debug.c +4 -2
Lines of this file are not shown. View the raw message.
kernel/sched/fair.c renamed from kernel/sched_fair.c +751 -247
998 lines not on this page · raw message
kernel/sched/features.h renamed from kernel/sched_features.h +15 -15
Lines of this file are not shown. View the raw message.
kernel/sched/idle_task.c renamed from kernel/sched_idletask.c +3 -1
Lines of this file are not shown. View the raw message.
kernel/sched/rt.c renamed from kernel/sched_rt.c +209 -9
218 lines not on this page · raw message
kernel/sched/sched.h new file +1166 -0
1,166 lines not on this page · raw message
kernel/sched/stats.c new file +111 -0
111 lines not on this page · raw message
kernel/sched/stats.h renamed from kernel/sched_stats.h +2 -107
109 lines not on this page · raw message
kernel/sched/stop_task.c renamed from kernel/sched_stoptask.c +3 -1
Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
Part of the diff is not shown here. View the raw message.
Quoted 4 lines · Ingo Molnar
Please pull the latest sched-core-for-linus git tree from:
git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.gitsched-core-for-linus
Ugh. So I got conflicts on this one. Big deal, fix them up.
But as I was fixing them up, I hit two problems that I wanted people to take a look at:
- some of the code was utter crap. Look at my fixup forinclude/asm-generic/cputime.h, for example. Notice how I had to change __msecs to __usec, and notice the bogus semicolons that I removed?
This came in through commit 648616343cdb ("[S390] cputime: add
sparse checking and cleanup") which is also mis-named: it's not S390-specific. Martin, please take a look.
- I did not do the right thing for the "usecs_to_cputime64()" fixup.
Please take a look at commit 34845636a184 ("procfs: do not confuse jiffies with cputime64_t") which introduced that "usecs_to_cputime64() thing, and which clashes badly with the sparse cleanups.
And by "clashes badly", I mean it. The sparse cleanups added a lot of casts between the types, which means that the mindless
+#define usecs_to_cputime64(us) usecs_to_cputime(us)
introduced by that commit 34845636a184 is now likely *wrong*, because "usecs_to_cputime()" now explicitly casts to a non-64-bit cputime_t.
It may have been wrong before too, but at least on 64-bit architectures it didn't really matter. Now the sparse fixes have made "cputime64_t" a separate type that will complain about casting, so it not only casts the type to a (potentially different and wrong size), but it also breaks the sparse checks.
I tried to fix up the ones that caused actual conflicts (due to changes nearby, like the ones in asm-generic), but the ones that didn't actually cause conflicts (powerpc and s390) I left alone.
Guys, can you take a look? Also double-check my resolve of the set_cpu_sd_state_idle() call in kernel/time/tick-sched.c, please. I've pushed it out, and it seems to work for me, but...
Linus
Quoted 38 lines · Linus Torvalds
Quoted 4 lines · Ingo Molnar
Please pull the latest sched-core-for-linus git tree from:
git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.gitsched-core-for-linusUgh. So I got conflicts on this one. Big deal, fix them up.
But as I was fixing them up, I hit two problems that I wanted people to take a look at:
- some of the code was utter crap. Look at my fixup for
include/asm-generic/cputime.h, for example. Notice how I had to change__msecsto__usec, and notice the bogus semicolons that I removed?This came in through commit
648616343cdb("[S390] cputime: add
sparse checking and cleanup") which is also mis-named: it's not S390-specific. Martin, please take a look.- I did not do the right thing for the "
usecs_to_cputime64()" fixup.
Please take a look at commit34845636a184("procfs: do not confuse jiffies withcputime64_t") which introduced that "usecs_to_cputime64()thing, and which clashes badly with the sparse cleanups.And by "clashes badly", I mean it. The sparse cleanups added a lot of casts between the types, which means that the mindless
+#define
usecs_to_cputime64(us)usecs_to_cputime(us)introduced by that commit
34845636a184is now likely *wrong*, because "usecs_to_cputime()" now explicitly casts to a non-64-bitcputime_t.It may have been wrong before too, but at least on 64-bit architectures it didn't really matter. Now the sparse fixes have made "
cputime64_t" a separate type that will complain about casting, so it not only casts the type to a (potentially different and wrong size), but it also breaks the sparse checks.
Quoted 7 lines
I tried to fix up the ones that caused actual conflicts (due to changes nearby, like the ones in asm-generic), but the ones that didn't actually cause conflicts (powerpc and s390) I left alone.
Guys, can you take a look? Also double-check my resolve of the
set_cpu_sd_state_idle()call inkernel/time/tick-sched.c, please. I've pushed it out, and it seems to work for me, but...
Yes looks good. We want it to be called from idle loop entry but not from idle interrupts. So moving it to tick_nohz_idle_enter() is the right thing.
Quoted 9 lines · Frederic Weisbecker
Quoted 4 lines · Linus Torvalds
Guys, can you take a look? Also double-check my resolve of the
set_cpu_sd_state_idle()call inkernel/time/tick-sched.c, please. I've pushed it out, and it seems to work for me, but...Yes looks good. We want it to be called from idle loop entry but not from idle interrupts. So moving it to
tick_nohz_idle_enter()is the right thing.
So I was pretty sure about that part, but not about *where* in tick_nohz_idle_enter() it needed to be.
In particular, is it ok to be in the irq-enabled area at the top? It looked to me like it was - all it needs is preemption disabled which I *assume* is always true ("enter idle" doesn't seem to make sense unless it is), but quite frankly, I don't know the code that well. So I was going by "this seems to be ok without irqs disabled, and if so, it's better to do it outside the irq-disabled region".
But I'd like somebody who knows the code. Because especially now that I look at it again, I realize that the old tick_nohz_stop_sched_tick() used to have irqs disabled and an interrupt could come in and move us out of idle, so maybe my original thinking when I did the merge was just totally bogus.
Linus
Quoted 7 lines · Linus Torvalds
And by "clashes badly", I mean it. The sparse cleanups added a lot of casts between the types, which means that the mindless
+#define
usecs_to_cputime64(us)usecs_to_cputime(us)introduced by that commit
34845636a184is now likely *wrong*, because "usecs_to_cputime()" now explicitly casts to a non-64-bitcputime_t.
Both s390 and powerpc define cputime_t and cputime64_t as the same types, so this is ok.
Andreas.
Signature · 3 lines
Andreas Schwab, [email protected] GPG Key fingerprint = 58CA 54C7 6D53 942B 1756 01D3 44D5 214B 8276 4ED5 "And now for something completely different."
Quoted 4 lines · Linus Torvalds
[...] Also double-check my resolve of the
set_cpu_sd_state_idle()call inkernel/time/tick-sched.c, please. I've pushed it out, and it seems to work for me, but...
Looks good at first sight - set_cpu_sd_state_idle() is only called from the idle thread when it is going idle - so there should neither be any irq context callers nor any preemption going on - no races.
Thanks,
Ingo
Quoted 6 lines · On Fri, 6 Jan 2012 08:58:55 -0800 Linus Torvalds…
Quoted 4 lines · Ingo Molnar
Please pull the latest sched-core-for-linus git tree from:
git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.gitsched-core-for-linus
Quoted 8 lines
Ugh. So I got conflicts on this one. Big deal, fix them up.
But as I was fixing them up, I hit two problems that I wanted people to take a look at:
- some of the code was utter crap. Look at my fixup for
include/asm-generic/cputime.h, for example. Notice how I had to change__msecsto__usec, and notice the bogus semicolons that I removed?
Yes, the semicolons had been crap and the parameter name suboptimal.
Quoted 3 lines
This came in through commit
648616343cdb("[S390] cputime: add
sparse checking and cleanup") which is also mis-named: it's not S390-specific. Martin, please take a look.
Yeah, I had that patch in the s390 tree before it got moved to a tip tree. I tend to tag everything in my tree with [S390].
Quoted 4 lines
- I did not do the right thing for the "
usecs_to_cputime64()" fixup.
Please take a look at commit34845636a184("procfs: do not confuse jiffies withcputime64_t") which introduced that "usecs_to_cputime64()thing, and which clashes badly with the sparse cleanups.
There had been some fallout from the two changes.
Quoted 7 lines
And by "clashes badly", I mean it. The sparse cleanups added a lot of casts between the types, which means that the mindless
+#define
usecs_to_cputime64(us)usecs_to_cputime(us)introduced by that commit
34845636a184is now likely *wrong*, because "usecs_to_cputime()" now explicitly casts to a non-64-bitcputime_t.
Surprisingly sparse does not give a warning if the base type of two __nowarn types is the same. All three VIRT_CPU_ACCOUNTING architectures use u64 or unsigned long long for cputime_t and cputime64_t. It does work but I agree that an explicit cast to cputime64_t would be better.
Signature · 4 lines
blue skies, Martin. "Reality continues to ruin my life." - Calvin.